2024-04-26T15:53:15.861 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T15:53:15.861 DEBUG:teuthology.run:Teuthology command: teuthology --name phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi --owner scheduled_phlogistonjohn@teuthology --verbose --description orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} --archive /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824 -- /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/orig.config.yaml 2024-04-26T15:53:15.894 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T15:53:15.978 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824 branch: wip-phlogistonjohn-testing-2024-04-25-1915 description: orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} email: jmulligan@redhat.com first_in_suite: false job_id: '7674824' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-phlogistonjohn-testing-2024-04-25-1915 ceph: conf: client: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mgrc: 20 debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon warn on pool no app: false paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 crush_tunables: jewel flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - reached quota - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) log-only-match: - CEPHADM_ sha1: a430f9ed84de3008ca5b632b1ed304bac0e753f6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: a430f9ed84de3008ca5b632b1ed304bac0e753f6 workunit: branch: wip-phlogistonjohn-testing-2024-04-25-1915 sha1: a430f9ed84de3008ca5b632b1ed304bac0e753f6 owner: scheduled_phlogistonjohn@teuthology priority: 89 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4852 sha1: a430f9ed84de3008ca5b632b1ed304bac0e753f6 sleep_before_teardown: 0 subset: 1/30 suite: orch:cephadm suite_branch: wip-phlogistonjohn-testing-2024-04-25-1915 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a430f9ed84de3008ca5b632b1ed304bac0e753f6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a430f9ed84de3008ca5b632b1ed304bac0e753f6 targets: smithi050.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJk6r8VQU9ftJhBaQ5FQ7aAAPKUP3I//8Z9NS040iPizquEHfhWLIxqpJSScHHmYYxS8ndJZpYbWHnpFwK+OmNk= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ8qmCfLhCVaCJLMOQbHbjAABjIA3ovOLeC7gs0G3npyZ45aSWXONpqF6/cHCCLUYyQkCMWL9b0eYdO+H09ndbo= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_15:06:39 tube: smithi user: phlogistonjohn verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T15:53:15.978 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a430f9ed84de3008ca5b632b1ed304bac0e753f6/qa; will attempt to use it 2024-04-26T15:53:15.979 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a430f9ed84de3008ca5b632b1ed304bac0e753f6/qa/tasks 2024-04-26T15:53:15.979 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T15:53:15.980 INFO:teuthology.task.internal:Checking packages... 2024-04-26T15:53:16.008 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a430f9ed84de3008ca5b632b1ed304bac0e753f6' 2024-04-26T15:53:16.008 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T15:53:16.008 INFO:teuthology.packaging:ref: None 2024-04-26T15:53:16.008 INFO:teuthology.packaging:tag: None 2024-04-26T15:53:16.008 INFO:teuthology.packaging:branch: wip-phlogistonjohn-testing-2024-04-25-1915 2024-04-26T15:53:16.008 INFO:teuthology.packaging:sha1: a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T15:53:16.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-phlogistonjohn-testing-2024-04-25-1915 2024-04-26T15:53:16.152 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3305-ga430f9ed-1jammy 2024-04-26T15:53:16.154 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T15:53:16.162 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T15:53:16.162 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T15:53:16.168 INFO:teuthology.task.internal:Saving configuration 2024-04-26T15:53:16.180 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T15:53:16.186 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T15:53:16.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi050.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 15:47:37.811077', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO2GbNsFwn8L74NP1Od6RzS8Ki/ldYaLJ3UNEP32Psr6'} 2024-04-26T15:53:16.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 15:47:37.809691', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPEJmiwAV0q/0P/3GMZW/GidQNVUqv0ao7lwZzYPRUIH'} 2024-04-26T15:53:16.233 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T15:53:16.239 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-04-26T15:53:16.239 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-04-26T15:53:16.239 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T15:53:16.316 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6bb8e9f3a0>, signals=[15]) 2024-04-26T15:53:16.316 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T15:53:16.323 INFO:teuthology.task.internal:Opening connections... 2024-04-26T15:53:16.323 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi050.front.sepia.ceph.com 2024-04-26T15:53:16.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:53:16.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-04-26T15:53:16.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:53:16.563 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T15:53:16.577 DEBUG:teuthology.orchestra.run.smithi050:> uname -m 2024-04-26T15:53:16.584 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64 2024-04-26T15:53:16.585 DEBUG:teuthology.orchestra.run.smithi050:> cat /etc/os-release 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Ubuntu" 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="22.04" 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_CODENAME=jammy 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:ID=ubuntu 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE=debian 2024-04-26T15:53:16.632 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T15:53:16.633 INFO:teuthology.orchestra.run.smithi050.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T15:53:16.633 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T15:53:16.633 INFO:teuthology.orchestra.run.smithi050.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T15:53:16.633 INFO:teuthology.orchestra.run.smithi050.stdout:UBUNTU_CODENAME=jammy 2024-04-26T15:53:16.633 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server 2024-04-26T15:53:16.655 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-04-26T15:53:16.660 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-04-26T15:53:16.660 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Ubuntu" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="22.04" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_CODENAME=jammy 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:ID=ubuntu 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE=debian 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T15:53:16.707 INFO:teuthology.orchestra.run.smithi190.stdout:UBUNTU_CODENAME=jammy 2024-04-26T15:53:16.708 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-04-26T15:53:16.730 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T15:53:16.739 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T15:53:16.744 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T15:53:16.744 DEBUG:teuthology.orchestra.run.smithi050:> test '!' -e /home/ubuntu/cephtest 2024-04-26T15:53:16.749 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-04-26T15:53:16.754 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T15:53:16.760 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T15:53:16.760 DEBUG:teuthology.orchestra.run.smithi050:> test -z $(ls -A /var/lib/ceph) 2024-04-26T15:53:16.796 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-04-26T15:53:16.888 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T15:53:16.943 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T15:53:16.957 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-26T15:53:16.958 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-26T15:53:16.958 DEBUG:teuthology.orchestra.run.smithi050:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T15:53:16.958 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T15:53:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T15:53:16.963 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-26T15:53:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T15:53:16.964 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-04-26T15:53:17.011 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-56-generic 2024-04-26T15:53:17.012 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.15.0-56-generic 2024-04-26T15:53:17.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2024-04-26T15:53:17.015 INFO:teuthology.orchestra.run.smithi050.stdout:5.15.0-56-generic 2024-04-26T15:53:17.015 INFO:teuthology.task.kernel:Running kernel on smithi050: 5.15.0-56-generic 2024-04-26T15:53:17.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get clean 2024-04-26T15:53:17.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-04-26T15:53:17.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2024-04-26T15:53:17.325 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T15:53:17.326 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T15:53:17.326 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T15:53:17.382 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T15:53:17.382 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T15:53:17.382 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T15:53:17.429 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T15:53:17.478 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T15:53:17.559 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T15:53:17.616 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T15:53:17.630 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T15:53:17.690 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T15:53:17.695 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T15:53:17.704 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T15:53:17.708 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T15:53:17.708 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T15:53:17.709 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T15:53:17.713 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T15:53:17.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T15:53:17.757 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T15:53:17.757 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T15:53:17.768 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T15:53:17.781 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T15:53:17.783 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T15:53:17.798 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T15:53:17.801 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T15:53:17.801 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T15:53:17.802 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T15:53:17.802 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T15:53:17.805 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T15:53:17.805 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T15:53:17.807 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T15:53:17.810 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T15:53:17.810 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T15:53:17.811 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T15:53:17.827 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T15:53:17.862 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T15:53:17.863 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T15:53:17.868 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T15:53:17.868 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T15:53:17.873 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T15:53:17.884 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T15:53:17.885 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T15:53:17.885 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T15:53:17.886 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T15:53:17.909 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T15:53:17.918 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T15:53:17.918 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T15:53:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T15:53:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T15:53:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T15:53:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T15:53:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T15:53:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T15:53:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T15:53:17.956 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T15:53:17.968 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T15:53:17.971 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T15:53:17.972 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T15:53:17.972 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T15:53:17.974 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T15:53:17.974 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T15:53:17.974 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T15:53:17.976 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T15:53:17.983 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T15:53:17.984 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T15:53:17.989 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T15:53:17.989 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T15:53:18.002 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T15:53:18.013 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T15:53:18.013 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T15:53:18.014 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T15:53:18.015 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T15:53:18.015 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T15:53:18.022 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T15:53:18.026 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T15:53:18.027 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T15:53:18.027 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T15:53:18.028 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T15:53:18.035 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T15:53:18.036 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T15:53:18.037 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T15:53:18.038 INFO:teuthology.orchestra.run.smithi050.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T15:53:18.057 INFO:teuthology.orchestra.run.smithi050.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T15:53:18.057 INFO:teuthology.orchestra.run.smithi050.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T15:53:18.058 INFO:teuthology.orchestra.run.smithi050.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T15:53:18.059 INFO:teuthology.orchestra.run.smithi050.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T15:53:18.059 INFO:teuthology.orchestra.run.smithi050.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T15:53:23.436 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 13.2 MB in 2s (6,904 kB/s) 2024-04-26T15:53:23.914 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 13.2 MB in 2s (6,605 kB/s) 2024-04-26T15:53:24.838 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-04-26T15:53:24.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T15:53:24.922 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-04-26T15:53:25.123 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-04-26T15:53:25.124 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-04-26T15:53:25.299 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T15:53:25.299 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T15:53:25.299 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T15:53:25.300 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T15:53:25.300 INFO:teuthology.orchestra.run.smithi190.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T15:53:25.300 INFO:teuthology.orchestra.run.smithi190.stdout: python2.7-minimal 2024-04-26T15:53:25.300 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T15:53:25.301 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2024-04-26T15:53:25.301 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T15:53:25.301 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T15:53:25.302 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T15:53:25.303 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2024-04-26T15:53:25.303 INFO:teuthology.orchestra.run.smithi190.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T15:53:25.303 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2024-04-26T15:53:25.303 INFO:teuthology.orchestra.run.smithi190.stdout: thermald 2024-04-26T15:53:25.357 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2024-04-26T15:53:25.357 INFO:teuthology.orchestra.run.smithi190.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T15:53:25.357 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T15:53:25.357 INFO:teuthology.orchestra.run.smithi190.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T15:53:25.359 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages will be upgraded: 2024-04-26T15:53:25.359 INFO:teuthology.orchestra.run.smithi190.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T15:53:25.426 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-26T15:53:25.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T15:53:25.498 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-26T15:53:25.538 INFO:teuthology.orchestra.run.smithi190.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T15:53:25.538 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 113 MB of archives. 2024-04-26T15:53:25.538 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T15:53:25.538 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T15:53:25.700 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2024-04-26T15:53:25.701 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2024-04-26T15:53:25.884 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T15:53:25.884 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T15:53:25.884 INFO:teuthology.orchestra.run.smithi050.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T15:53:25.885 INFO:teuthology.orchestra.run.smithi050.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T15:53:25.885 INFO:teuthology.orchestra.run.smithi050.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T15:53:25.885 INFO:teuthology.orchestra.run.smithi050.stdout: python2.7-minimal 2024-04-26T15:53:25.885 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T15:53:25.886 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2024-04-26T15:53:25.886 INFO:teuthology.orchestra.run.smithi050.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T15:53:25.886 INFO:teuthology.orchestra.run.smithi050.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T15:53:25.887 INFO:teuthology.orchestra.run.smithi050.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T15:53:25.888 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2024-04-26T15:53:25.888 INFO:teuthology.orchestra.run.smithi050.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T15:53:25.888 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2024-04-26T15:53:25.888 INFO:teuthology.orchestra.run.smithi050.stdout: thermald 2024-04-26T15:53:25.945 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2024-04-26T15:53:25.945 INFO:teuthology.orchestra.run.smithi050.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T15:53:25.945 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T15:53:25.945 INFO:teuthology.orchestra.run.smithi050.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T15:53:25.946 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages will be upgraded: 2024-04-26T15:53:25.947 INFO:teuthology.orchestra.run.smithi050.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T15:53:26.005 INFO:teuthology.orchestra.run.smithi050.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T15:53:26.005 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 113 MB of archives. 2024-04-26T15:53:26.005 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T15:53:26.006 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T15:53:26.490 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T15:53:26.610 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T15:53:27.124 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T15:53:27.480 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T15:53:27.533 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T15:53:27.533 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T15:53:27.533 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T15:53:27.780 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T15:53:27.806 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T15:53:28.214 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 113 MB in 2s (61.2 MB/s) 2024-04-26T15:53:28.399 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T15:53:29.993 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T15:53:29.993 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T15:53:29.993 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T15:53:30.352 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T15:53:30.437 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T15:53:30.466 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T15:53:30.471 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:30.615 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:30.826 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 113 MB in 5s (22.3 MB/s) 2024-04-26T15:53:30.969 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T15:53:32.910 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T15:53:32.914 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:32.916 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T15:53:32.937 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:32.974 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:33.027 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:33.611 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T15:53:33.632 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:33.918 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:34.966 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T15:53:34.988 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:35.067 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:35.520 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T15:53:35.540 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:35.577 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:40.627 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T15:53:40.770 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T15:53:41.088 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T15:53:41.207 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T15:53:41.560 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T15:53:41.583 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T15:53:41.626 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T15:53:41.827 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T15:53:41.952 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T15:53:42.285 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T15:53:42.405 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T15:53:42.691 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T15:53:42.702 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T15:53:42.749 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T15:53:47.901 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T15:53:47.938 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:47.975 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:48.308 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T15:53:48.345 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T15:53:48.383 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:50.838 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T15:53:50.853 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T15:53:50.936 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T15:53:50.978 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T15:53:51.235 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T15:53:51.341 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:51.379 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T15:53:51.450 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T15:53:51.500 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:51.579 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:51.651 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T15:53:51.779 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:52.959 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T15:53:52.960 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T15:53:52.960 INFO:teuthology.orchestra.run.smithi190.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T15:53:52.960 INFO:teuthology.orchestra.run.smithi190.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T15:53:53.230 INFO:teuthology.orchestra.run.smithi050.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T15:53:53.231 INFO:teuthology.orchestra.run.smithi050.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T15:53:53.231 INFO:teuthology.orchestra.run.smithi050.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T15:53:53.231 INFO:teuthology.orchestra.run.smithi050.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T15:53:53.381 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:53.790 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:54.799 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:55.267 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:56.142 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T15:53:56.268 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T15:53:56.395 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:56.483 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T15:53:56.483 INFO:teuthology.orchestra.run.smithi190.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T15:53:56.684 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T15:53:56.827 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T15:53:56.953 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T15:53:57.047 INFO:teuthology.orchestra.run.smithi050.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T15:53:57.047 INFO:teuthology.orchestra.run.smithi050.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T15:54:10.015 INFO:teuthology.orchestra.run.smithi190.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T15:54:10.015 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub' 2024-04-26T15:54:10.034 INFO:teuthology.orchestra.run.smithi190.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T15:54:10.061 INFO:teuthology.orchestra.run.smithi190.stdout:Generating grub configuration file ... 2024-04-26T15:54:10.323 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T15:54:10.339 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T15:54:10.610 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T15:54:10.615 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T15:54:10.690 INFO:teuthology.orchestra.run.smithi190.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T15:54:10.695 INFO:teuthology.orchestra.run.smithi190.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T15:54:10.867 INFO:teuthology.orchestra.run.smithi190.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T15:54:10.867 INFO:teuthology.orchestra.run.smithi190.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T15:54:10.867 INFO:teuthology.orchestra.run.smithi190.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T15:54:10.918 INFO:teuthology.orchestra.run.smithi190.stdout:done 2024-04-26T15:54:11.364 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2024-04-26T15:54:11.391 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2024-04-26T15:54:11.391 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2024-04-26T15:54:11.391 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2024-04-26T15:54:11.391 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 21 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.105.102 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2024-04-26T15:54:11.392 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2024-04-26T15:54:11.392 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T15:54:11.392 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T15:54:11.393 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T15:54:11.393 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-26T15:54:11.393 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T15:54:11.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get clean 2024-04-26T15:54:11.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-04-26T15:54:11.695 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T15:54:11.746 INFO:teuthology.orchestra.run.smithi050.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T15:54:11.746 INFO:teuthology.orchestra.run.smithi050.stdout:Sourcing file `/etc/default/grub' 2024-04-26T15:54:11.764 INFO:teuthology.orchestra.run.smithi050.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T15:54:11.806 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T15:54:11.822 INFO:teuthology.orchestra.run.smithi050.stdout:Generating grub configuration file ... 2024-04-26T15:54:11.887 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T15:54:12.192 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T15:54:12.224 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T15:54:12.249 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T15:54:12.453 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 119 kB in 1s (160 kB/s) 2024-04-26T15:54:12.637 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T15:54:12.643 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T15:54:12.716 INFO:teuthology.orchestra.run.smithi050.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T15:54:12.721 INFO:teuthology.orchestra.run.smithi050.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T15:54:12.893 INFO:teuthology.orchestra.run.smithi050.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T15:54:12.894 INFO:teuthology.orchestra.run.smithi050.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T15:54:12.894 INFO:teuthology.orchestra.run.smithi050.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T15:54:12.930 INFO:teuthology.orchestra.run.smithi050.stdout:done 2024-04-26T15:54:13.428 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-04-26T15:54:13.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T15:54:13.469 DEBUG:teuthology.orchestra.run.smithi050:> dpkg -s linux-image-generic 2024-04-26T15:54:13.493 INFO:teuthology.orchestra.run.smithi050.stdout:Package: linux-image-generic 2024-04-26T15:54:13.493 INFO:teuthology.orchestra.run.smithi050.stdout:Status: install ok installed 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Priority: optional 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Section: kernel 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Installed-Size: 21 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: amd64 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Source: linux-meta 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Version: 5.15.0.105.102 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Recommends: thermald 2024-04-26T15:54:13.494 INFO:teuthology.orchestra.run.smithi050.stdout:Description: Generic Linux kernel image 2024-04-26T15:54:13.495 INFO:teuthology.orchestra.run.smithi050.stdout: This package will always depend on the latest generic kernel image 2024-04-26T15:54:13.495 INFO:teuthology.orchestra.run.smithi050.stdout: available. 2024-04-26T15:54:13.495 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T15:54:13.495 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T15:54:13.495 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T15:54:13.495 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T15:54:13.495 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi050.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T15:54:13.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get clean 2024-04-26T15:54:13.503 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-04-26T15:54:13.700 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-04-26T15:54:13.700 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-04-26T15:54:13.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2024-04-26T15:54:13.833 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T15:54:13.871 INFO:teuthology.orchestra.run.smithi190.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T15:54:13.871 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T15:54:13.871 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T15:54:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T15:54:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T15:54:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T15:54:13.872 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T15:54:13.872 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T15:54:13.910 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T15:54:13.912 DEBUG:teuthology.orchestra.run.smithi190:> dpkg -s linux-image-generic 2024-04-26T15:54:13.927 INFO:teuthology.orchestra.run.smithi190.stdout:Package: linux-image-generic 2024-04-26T15:54:13.927 INFO:teuthology.orchestra.run.smithi190.stdout:Status: install ok installed 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Priority: optional 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Section: kernel 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Installed-Size: 21 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: amd64 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Source: linux-meta 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Version: 5.15.0.105.102 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Recommends: thermald 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout:Description: Generic Linux kernel image 2024-04-26T15:54:13.928 INFO:teuthology.orchestra.run.smithi190.stdout: This package will always depend on the latest generic kernel image 2024-04-26T15:54:13.929 INFO:teuthology.orchestra.run.smithi190.stdout: available. 2024-04-26T15:54:13.930 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-04-26T15:54:13.947 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T15:54:13.979 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.kRkGmxKRrP 2024-04-26T15:54:13.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kRkGmxKRrP 2024-04-26T15:54:14.029 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T15:54:14.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.kRkGmxKRrP 2024-04-26T15:54:14.154 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.kRkGmxKRrP is 10KB 2024-04-26T15:54:14.167 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.kRkGmxKRrP 2024-04-26T15:54:14.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T15:54:14.225 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T15:54:14.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T15:54:14.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T15:54:14.289 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T15:54:14.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo update-grub 2024-04-26T15:54:14.384 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T15:54:14.593 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 119 kB in 1s (166 kB/s) 2024-04-26T15:54:15.398 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub' 2024-04-26T15:54:15.399 INFO:teuthology.orchestra.run.smithi190.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T15:54:15.401 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-04-26T15:54:15.563 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-26T15:54:15.575 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T15:54:15.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T15:54:15.582 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T15:54:15.645 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-26T15:54:15.815 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T15:54:15.820 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T15:54:15.859 INFO:teuthology.orchestra.run.smithi190.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T15:54:15.863 INFO:teuthology.orchestra.run.smithi190.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T15:54:15.864 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2024-04-26T15:54:15.864 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2024-04-26T15:54:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T15:54:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T15:54:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T15:54:16.004 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-04-26T15:54:16.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-04-26T15:54:16.048 INFO:teuthology.orchestra.run.smithi050.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T15:54:16.048 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T15:54:16.048 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T15:54:16.048 INFO:teuthology.orchestra.run.smithi050.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T15:54:16.048 INFO:teuthology.orchestra.run.smithi050.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T15:54:16.048 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T15:54:16.049 INFO:teuthology.orchestra.run.smithi050.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T15:54:16.049 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T15:54:16.093 INFO:teuthology.orchestra.run.smithi050.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T15:54:16.095 DEBUG:teuthology.orchestra.run.smithi050:> dpkg -s linux-image-generic 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Package: linux-image-generic 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Status: install ok installed 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Priority: optional 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Section: kernel 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Installed-Size: 21 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T15:54:16.112 INFO:teuthology.orchestra.run.smithi050.stdout:Architecture: amd64 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout:Source: linux-meta 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout:Version: 5.15.0.105.102 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout:Recommends: thermald 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout:Description: Generic Linux kernel image 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout: This package will always depend on the latest generic kernel image 2024-04-26T15:54:16.113 INFO:teuthology.orchestra.run.smithi050.stdout: available. 2024-04-26T15:54:16.113 DEBUG:teuthology.orchestra.run.smithi050:> mktemp 2024-04-26T15:54:16.161 INFO:teuthology.orchestra.run.smithi050.stdout:/tmp/tmp.LUMhvAylrZ 2024-04-26T15:54:16.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo cp /boot/grub/grub.cfg /tmp/tmp.LUMhvAylrZ 2024-04-26T15:54:16.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0666 /tmp/tmp.LUMhvAylrZ 2024-04-26T15:54:16.352 DEBUG:teuthology.orchestra.remote:smithi050:/tmp/tmp.LUMhvAylrZ is 10KB 2024-04-26T15:54:16.365 DEBUG:teuthology.orchestra.run.smithi050:> rm -fr /tmp/tmp.LUMhvAylrZ 2024-04-26T15:54:16.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T15:54:16.426 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T15:54:16.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T15:54:16.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T15:54:16.519 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T15:54:16.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo update-grub 2024-04-26T15:54:17.961 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub' 2024-04-26T15:54:17.962 INFO:teuthology.orchestra.run.smithi050.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T15:54:17.970 INFO:teuthology.orchestra.run.smithi050.stderr:Generating grub configuration file ... 2024-04-26T15:54:18.247 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T15:54:18.254 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T15:54:18.562 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T15:54:18.568 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T15:54:18.612 INFO:teuthology.orchestra.run.smithi050.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T15:54:18.618 INFO:teuthology.orchestra.run.smithi050.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T15:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T15:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T15:54:18.742 INFO:teuthology.orchestra.run.smithi050.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T15:54:18.754 INFO:teuthology.orchestra.run.smithi050.stderr:done 2024-04-26T15:54:18.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo shutdown -r now 2024-04-26T15:54:46.052 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-26T15:54:46.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T15:54:46.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:54:48.816 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T15:54:48.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T15:54:48.817 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:55:04.607 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-04-26T15:55:13.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T15:55:13.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:55:16.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-04-26T15:55:28.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T15:55:28.686 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:55:47.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-04-26T15:55:48.821 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T15:55:57.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T15:55:57.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:56:00.868 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.50 2024-04-26T15:56:02.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T15:56:02.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:56:02.430 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-04-26T15:56:03.377 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T15:56:03.377 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-26T15:56:03.378 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-04-26T15:56:03.424 INFO:teuthology.orchestra.run.smithi190.stdout:5.15.0-105-generic 2024-04-26T15:56:03.424 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T15:56:03.424 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T15:56:03.425 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T15:56:04.425 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-26T15:56:04.425 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T15:56:04.727 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-04-26T15:56:04.754 DEBUG:teuthology.parallel:result is None 2024-04-26T15:56:12.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T15:56:12.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T15:56:13.101 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-26T15:56:13.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T15:56:13.708 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T15:56:13.709 DEBUG:teuthology.orchestra.run.smithi050:> uname -r 2024-04-26T15:56:13.756 INFO:teuthology.orchestra.run.smithi050.stdout:5.15.0-105-generic 2024-04-26T15:56:13.756 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T15:56:13.756 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T15:56:13.756 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T15:56:14.757 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T15:56:14.757 DEBUG:teuthology.orchestra.run.smithi050:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T15:56:14.827 INFO:teuthology.orchestra.run.smithi050.stdout:ttyS1 2024-04-26T15:56:14.841 DEBUG:teuthology.parallel:result is None 2024-04-26T15:56:14.842 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T15:56:14.851 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T15:56:14.851 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T15:56:14.854 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T15:56:14.860 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T15:56:14.866 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T15:56:14.873 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T15:56:14.873 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T15:56:14.901 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T15:56:14.991 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T15:56:14.998 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T15:56:14.998 DEBUG:teuthology.orchestra.run.smithi050:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T15:56:15.001 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T15:56:15.017 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T15:56:15.018 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T15:56:15.026 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T15:56:15.026 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T15:56:15.028 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T15:56:15.035 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T15:56:15.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T15:56:15.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T15:56:15.086 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T15:56:15.095 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T15:56:15.096 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T15:56:15.129 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T15:56:15.134 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T15:56:15.180 DEBUG:teuthology.orchestra.run.smithi050:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T15:56:15.228 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T15:56:15.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T15:56:15.298 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T15:56:15.304 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T15:56:15.353 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T15:56:15.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T15:56:15.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo service rsyslog restart 2024-04-26T15:56:15.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-04-26T15:56:15.489 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T15:56:15.497 INFO:teuthology.task.internal:Starting timer... 2024-04-26T15:56:15.497 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T15:56:15.520 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T15:56:15.529 DEBUG:teuthology.task.selinux:Excluding smithi050: OS 'ubuntu' does not support SELinux 2024-04-26T15:56:15.529 DEBUG:teuthology.task.selinux:Excluding smithi190: OS 'ubuntu' does not support SELinux 2024-04-26T15:56:15.529 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T15:56:15.529 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T15:56:15.529 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T15:56:15.529 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T15:56:15.538 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T15:56:15.555 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T15:56:15.556 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi050.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T16:03:21.630 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi050.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-04-26T16:03:21.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T16:03:21.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi050.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T16:03:21.721 DEBUG:teuthology.orchestra.run.smithi050:> true 2024-04-26T16:03:21.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi050.front.sepia.ceph.com' 2024-04-26T16:03:21.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T16:03:21.807 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T16:03:21.886 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-04-26T16:03:21.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-04-26T16:03:21.962 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T16:03:21.973 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T16:03:21.973 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T16:03:21.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T16:03:21.976 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T16:03:21.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Command line: ntpd -gq 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: ---------------------------------------------------- 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: corporation. Support and training for ntp-4 are 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: available at https://www.nwtime.org/support 2024-04-26T16:03:22.007 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: ---------------------------------------------------- 2024-04-26T16:03:22.009 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: proto: precision = 0.046 usec (-24) 2024-04-26T16:03:22.009 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: basedate set to 2022-02-04 2024-04-26T16:03:22.009 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: gps base set to 2022-02-06 (week 2196) 2024-04-26T16:03:22.009 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T16:03:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T16:03:22.009 INFO:teuthology.orchestra.run.smithi050.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T16:03:22.010 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T16:03:22.010 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T16:03:22.010 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T16:03:22.011 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T16:03:22.011 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listen normally on 3 ens1f0 172.21.15.50:123 2024-04-26T16:03:22.011 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listen normally on 4 lo [::1]:123 2024-04-26T16:03:22.011 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1584%4]:123 2024-04-26T16:03:22.011 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:22 ntpd[17096]: Listening on routing socket on fd #22 for interface updates 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Command line: ntpd -gq 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: ---------------------------------------------------- 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: corporation. Support and training for ntp-4 are 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: available at https://www.nwtime.org/support 2024-04-26T16:03:22.032 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: ---------------------------------------------------- 2024-04-26T16:03:22.033 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: proto: precision = 0.060 usec (-24) 2024-04-26T16:03:22.033 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: basedate set to 2022-02-04 2024-04-26T16:03:22.033 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: gps base set to 2022-02-06 (week 2196) 2024-04-26T16:03:22.033 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T16:03:22.034 INFO:teuthology.orchestra.run.smithi190.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T16:03:22.034 INFO:teuthology.orchestra.run.smithi190.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listen normally on 3 enp3s0f1 172.21.15.190:123 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listen normally on 4 lo [::1]:123 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7763%5]:123 2024-04-26T16:03:22.035 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:22 ntpd[17205]: Listening on routing socket on fd #22 for interface updates 2024-04-26T16:03:23.007 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:23 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:23.007 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:23 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:23.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:23.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:23.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:23 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:24 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:24 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:24 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:24 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:24 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:24.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:24 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:24.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:24 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:24.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:24 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:24.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:24 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:24.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:24 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:24.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:24 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:24.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:24 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:25 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:25 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:25 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:25 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:25 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:25.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:25 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:25.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:25.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:25.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:25.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:25 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:26 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:26 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:26 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:26 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:26 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:26 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:26 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:27 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:27.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:27.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:27 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:28.007 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:28 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:28.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:28 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:28.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:28 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:28.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:28 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:28.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:28 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:28.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:28 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:28 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:28 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:28 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:28 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:28 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:28.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:28 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:29.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:29 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.032 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stdout:26 Apr 16:03:29 ntpd[17205]: ntpd: time slew +0.001077 s 2024-04-26T16:03:29.033 INFO:teuthology.orchestra.run.smithi190.stdout:ntpd: time slew +0.001077s 2024-04-26T16:03:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T16:03:29.034 INFO:teuthology.orchestra.run.smithi190.stderr:26 Apr 16:03:29 ntpd[17205]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T16:03:29.140 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T16:03:29.140 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2024-04-26T16:03:29.140 INFO:teuthology.orchestra.run.smithi190.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:29.140 INFO:teuthology.orchestra.run.smithi190.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:29.140 INFO:teuthology.orchestra.run.smithi190.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:29.140 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stdout:26 Apr 16:03:30 ntpd[17096]: ntpd: time slew -0.002087 s 2024-04-26T16:03:30.008 INFO:teuthology.orchestra.run.smithi050.stdout:ntpd: time slew -0.002087s 2024-04-26T16:03:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T16:03:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:26 Apr 16:03:30 ntpd[17096]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T16:03:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T16:03:30.058 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2024-04-26T16:03:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:30.058 INFO:teuthology.orchestra.run.smithi050.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:30.059 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T16:03:30.060 INFO:teuthology.run_tasks:Running task install... 2024-04-26T16:03:30.068 DEBUG:teuthology.task.install:project ceph 2024-04-26T16:03:30.068 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6'}} 2024-04-26T16:03:30.068 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6'} 2024-04-26T16:03:30.068 INFO:teuthology.task.install:Using flavor: default 2024-04-26T16:03:30.074 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T16:03:30.074 INFO:teuthology.task.install:extra packages: [] 2024-04-26T16:03:30.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-key list | grep Ceph 2024-04-26T16:03:30.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-key list | grep Ceph 2024-04-26T16:03:30.141 INFO:teuthology.orchestra.run.smithi190.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T16:03:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T16:03:30.171 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T16:03:30.171 INFO:teuthology.orchestra.run.smithi190.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T16:03:30.171 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T16:03:30.171 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:03:30.193 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T16:03:30.193 INFO:teuthology.orchestra.run.smithi050.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T16:03:30.193 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T16:03:30.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:03:30.312 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default/ 2024-04-26T16:03:30.312 INFO:teuthology.task.install.deb:Package version is 19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:03:30.412 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default/ 2024-04-26T16:03:30.412 INFO:teuthology.task.install.deb:Package version is 19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:03:30.486 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:03:30.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T16:03:30.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo apt-get update 2024-04-26T16:03:30.524 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:03:30.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T16:03:30.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo apt-get update 2024-04-26T16:03:30.670 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T16:03:30.715 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T16:03:30.717 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T16:03:30.739 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T16:03:30.761 INFO:teuthology.orchestra.run.smithi190.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T16:03:30.762 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T16:03:30.782 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-04-26T16:03:30.783 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T16:03:30.788 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T16:03:30.805 INFO:teuthology.orchestra.run.smithi050.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T16:03:30.809 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,402 B] 2024-04-26T16:03:30.826 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,402 B] 2024-04-26T16:03:30.832 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T16:03:30.853 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-04-26T16:03:30.865 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T16:03:30.911 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T16:03:30.947 INFO:teuthology.orchestra.run.smithi190.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T16:03:30.993 INFO:teuthology.orchestra.run.smithi050.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T16:03:31.090 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 29.2 kB in 0s (72.0 kB/s) 2024-04-26T16:03:31.143 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 29.2 kB in 0s (69.6 kB/s) 2024-04-26T16:03:32.000 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-04-26T16:03:32.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3305-ga430f9ed-1jammy cephadm=19.0.0-3305-ga430f9ed-1jammy ceph-mds=19.0.0-3305-ga430f9ed-1jammy ceph-mgr=19.0.0-3305-ga430f9ed-1jammy ceph-common=19.0.0-3305-ga430f9ed-1jammy ceph-fuse=19.0.0-3305-ga430f9ed-1jammy ceph-test=19.0.0-3305-ga430f9ed-1jammy ceph-volume=19.0.0-3305-ga430f9ed-1jammy radosgw=19.0.0-3305-ga430f9ed-1jammy python3-rados=19.0.0-3305-ga430f9ed-1jammy python3-rgw=19.0.0-3305-ga430f9ed-1jammy python3-cephfs=19.0.0-3305-ga430f9ed-1jammy python3-rbd=19.0.0-3305-ga430f9ed-1jammy libcephfs2=19.0.0-3305-ga430f9ed-1jammy libcephfs-dev=19.0.0-3305-ga430f9ed-1jammy librados2=19.0.0-3305-ga430f9ed-1jammy librbd1=19.0.0-3305-ga430f9ed-1jammy rbd-fuse=19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:03:32.072 INFO:teuthology.orchestra.run.smithi190.stdout:Reading package lists... 2024-04-26T16:03:32.135 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-26T16:03:32.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3305-ga430f9ed-1jammy cephadm=19.0.0-3305-ga430f9ed-1jammy ceph-mds=19.0.0-3305-ga430f9ed-1jammy ceph-mgr=19.0.0-3305-ga430f9ed-1jammy ceph-common=19.0.0-3305-ga430f9ed-1jammy ceph-fuse=19.0.0-3305-ga430f9ed-1jammy ceph-test=19.0.0-3305-ga430f9ed-1jammy ceph-volume=19.0.0-3305-ga430f9ed-1jammy radosgw=19.0.0-3305-ga430f9ed-1jammy python3-rados=19.0.0-3305-ga430f9ed-1jammy python3-rgw=19.0.0-3305-ga430f9ed-1jammy python3-cephfs=19.0.0-3305-ga430f9ed-1jammy python3-rbd=19.0.0-3305-ga430f9ed-1jammy libcephfs2=19.0.0-3305-ga430f9ed-1jammy libcephfs-dev=19.0.0-3305-ga430f9ed-1jammy librados2=19.0.0-3305-ga430f9ed-1jammy librbd1=19.0.0-3305-ga430f9ed-1jammy rbd-fuse=19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:03:32.207 INFO:teuthology.orchestra.run.smithi050.stdout:Reading package lists... 2024-04-26T16:03:32.248 INFO:teuthology.orchestra.run.smithi190.stdout:Building dependency tree... 2024-04-26T16:03:32.249 INFO:teuthology.orchestra.run.smithi190.stdout:Reading state information... 2024-04-26T16:03:32.389 INFO:teuthology.orchestra.run.smithi050.stdout:Building dependency tree... 2024-04-26T16:03:32.390 INFO:teuthology.orchestra.run.smithi050.stdout:Reading state information... 2024-04-26T16:03:32.425 INFO:teuthology.orchestra.run.smithi190.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T16:03:32.425 INFO:teuthology.orchestra.run.smithi190.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T16:03:32.425 INFO:teuthology.orchestra.run.smithi190.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T16:03:32.425 INFO:teuthology.orchestra.run.smithi190.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T16:03:32.425 INFO:teuthology.orchestra.run.smithi190.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T16:03:32.426 INFO:teuthology.orchestra.run.smithi190.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T16:03:32.426 INFO:teuthology.orchestra.run.smithi190.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T16:03:32.427 INFO:teuthology.orchestra.run.smithi190.stdout:The following additional packages will be installed: 2024-04-26T16:03:32.427 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T16:03:32.427 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T16:03:32.427 INFO:teuthology.orchestra.run.smithi190.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T16:03:32.427 INFO:teuthology.orchestra.run.smithi190.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T16:03:32.428 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T16:03:32.429 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet zip 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout:Suggested packages: 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout:Recommended packages: 2024-04-26T16:03:32.430 INFO:teuthology.orchestra.run.smithi190.stdout: btrfs-tools 2024-04-26T16:03:32.528 INFO:teuthology.orchestra.run.smithi190.stdout:The following NEW packages will be installed: 2024-04-26T16:03:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T16:03:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T16:03:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T16:03:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T16:03:32.528 INFO:teuthology.orchestra.run.smithi190.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T16:03:32.529 INFO:teuthology.orchestra.run.smithi190.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T16:03:32.530 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T16:03:32.531 INFO:teuthology.orchestra.run.smithi190.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T16:03:32.568 INFO:teuthology.orchestra.run.smithi050.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T16:03:32.568 INFO:teuthology.orchestra.run.smithi050.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T16:03:32.569 INFO:teuthology.orchestra.run.smithi050.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T16:03:32.569 INFO:teuthology.orchestra.run.smithi050.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T16:03:32.569 INFO:teuthology.orchestra.run.smithi050.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout:The following additional packages will be installed: 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T16:03:32.570 INFO:teuthology.orchestra.run.smithi050.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T16:03:32.571 INFO:teuthology.orchestra.run.smithi050.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T16:03:32.571 INFO:teuthology.orchestra.run.smithi050.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T16:03:32.571 INFO:teuthology.orchestra.run.smithi050.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T16:03:32.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T16:03:32.571 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T16:03:32.572 INFO:teuthology.orchestra.run.smithi050.stdout: xmlstarlet zip 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout:Suggested packages: 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout:Recommended packages: 2024-04-26T16:03:32.573 INFO:teuthology.orchestra.run.smithi050.stdout: btrfs-tools 2024-04-26T16:03:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T16:03:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:Need to get 181 MB of archives. 2024-04-26T16:03:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-26T16:03:32.670 INFO:teuthology.orchestra.run.smithi190.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3305-ga430f9ed-1jammy [3,556 kB] 2024-04-26T16:03:32.674 INFO:teuthology.orchestra.run.smithi050.stdout:The following NEW packages will be installed: 2024-04-26T16:03:32.674 INFO:teuthology.orchestra.run.smithi050.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T16:03:32.674 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T16:03:32.676 INFO:teuthology.orchestra.run.smithi050.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T16:03:32.677 INFO:teuthology.orchestra.run.smithi050.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T16:03:32.717 INFO:teuthology.orchestra.run.smithi190.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T16:03:32.745 INFO:teuthology.orchestra.run.smithi050.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T16:03:32.745 INFO:teuthology.orchestra.run.smithi050.stdout:Need to get 181 MB of archives. 2024-04-26T16:03:32.745 INFO:teuthology.orchestra.run.smithi050.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-26T16:03:32.745 INFO:teuthology.orchestra.run.smithi050.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T16:03:32.851 INFO:teuthology.orchestra.run.smithi050.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T16:03:32.853 INFO:teuthology.orchestra.run.smithi050.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T16:03:32.875 INFO:teuthology.orchestra.run.smithi050.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T16:03:32.883 INFO:teuthology.orchestra.run.smithi190.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3305-ga430f9ed-1jammy [3,224 kB] 2024-04-26T16:03:32.930 INFO:teuthology.orchestra.run.smithi050.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3305-ga430f9ed-1jammy [3,556 kB] 2024-04-26T16:03:32.945 INFO:teuthology.orchestra.run.smithi050.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T16:03:32.950 INFO:teuthology.orchestra.run.smithi050.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T16:03:32.965 INFO:teuthology.orchestra.run.smithi050.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T16:03:32.970 INFO:teuthology.orchestra.run.smithi050.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T16:03:32.971 INFO:teuthology.orchestra.run.smithi050.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T16:03:32.972 INFO:teuthology.orchestra.run.smithi050.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T16:03:32.981 INFO:teuthology.orchestra.run.smithi050.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T16:03:32.983 INFO:teuthology.orchestra.run.smithi190.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3305-ga430f9ed-1jammy [712 kB] 2024-04-26T16:03:32.984 INFO:teuthology.orchestra.run.smithi050.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T16:03:32.986 INFO:teuthology.orchestra.run.smithi050.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T16:03:32.988 INFO:teuthology.orchestra.run.smithi050.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T16:03:32.993 INFO:teuthology.orchestra.run.smithi050.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T16:03:32.996 INFO:teuthology.orchestra.run.smithi050.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T16:03:32.998 INFO:teuthology.orchestra.run.smithi050.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T16:03:33.006 INFO:teuthology.orchestra.run.smithi190.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3305-ga430f9ed-1jammy [356 kB] 2024-04-26T16:03:33.014 INFO:teuthology.orchestra.run.smithi050.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T16:03:33.015 INFO:teuthology.orchestra.run.smithi050.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T16:03:33.015 INFO:teuthology.orchestra.run.smithi050.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T16:03:33.016 INFO:teuthology.orchestra.run.smithi050.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T16:03:33.016 INFO:teuthology.orchestra.run.smithi050.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T16:03:33.017 INFO:teuthology.orchestra.run.smithi050.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T16:03:33.017 INFO:teuthology.orchestra.run.smithi050.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T16:03:33.018 INFO:teuthology.orchestra.run.smithi190.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3305-ga430f9ed-1jammy [32.8 kB] 2024-04-26T16:03:33.019 INFO:teuthology.orchestra.run.smithi190.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3305-ga430f9ed-1jammy [180 kB] 2024-04-26T16:03:33.025 INFO:teuthology.orchestra.run.smithi190.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3305-ga430f9ed-1jammy [67.1 kB] 2024-04-26T16:03:33.027 INFO:teuthology.orchestra.run.smithi190.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3305-ga430f9ed-1jammy [330 kB] 2024-04-26T16:03:33.034 INFO:teuthology.orchestra.run.smithi050.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T16:03:33.035 INFO:teuthology.orchestra.run.smithi050.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T16:03:33.035 INFO:teuthology.orchestra.run.smithi050.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T16:03:33.041 INFO:teuthology.orchestra.run.smithi190.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3305-ga430f9ed-1jammy [6,892 kB] 2024-04-26T16:03:33.054 INFO:teuthology.orchestra.run.smithi050.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T16:03:33.055 INFO:teuthology.orchestra.run.smithi050.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T16:03:33.055 INFO:teuthology.orchestra.run.smithi050.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T16:03:33.056 INFO:teuthology.orchestra.run.smithi050.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T16:03:33.057 INFO:teuthology.orchestra.run.smithi050.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T16:03:33.057 INFO:teuthology.orchestra.run.smithi050.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T16:03:33.058 INFO:teuthology.orchestra.run.smithi050.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T16:03:33.075 INFO:teuthology.orchestra.run.smithi050.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T16:03:33.077 INFO:teuthology.orchestra.run.smithi050.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T16:03:33.077 INFO:teuthology.orchestra.run.smithi050.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T16:03:33.096 INFO:teuthology.orchestra.run.smithi050.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T16:03:33.097 INFO:teuthology.orchestra.run.smithi050.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T16:03:33.097 INFO:teuthology.orchestra.run.smithi050.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T16:03:33.099 INFO:teuthology.orchestra.run.smithi050.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T16:03:33.101 INFO:teuthology.orchestra.run.smithi050.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T16:03:33.104 INFO:teuthology.orchestra.run.smithi050.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T16:03:33.106 INFO:teuthology.orchestra.run.smithi050.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T16:03:33.116 INFO:teuthology.orchestra.run.smithi050.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T16:03:33.118 INFO:teuthology.orchestra.run.smithi050.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T16:03:33.122 INFO:teuthology.orchestra.run.smithi050.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T16:03:33.137 INFO:teuthology.orchestra.run.smithi050.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T16:03:33.138 INFO:teuthology.orchestra.run.smithi050.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T16:03:33.143 INFO:teuthology.orchestra.run.smithi050.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T16:03:33.149 INFO:teuthology.orchestra.run.smithi190.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T16:03:33.162 INFO:teuthology.orchestra.run.smithi190.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T16:03:33.225 INFO:teuthology.orchestra.run.smithi050.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T16:03:33.226 INFO:teuthology.orchestra.run.smithi050.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T16:03:33.228 INFO:teuthology.orchestra.run.smithi050.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T16:03:33.228 INFO:teuthology.orchestra.run.smithi050.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T16:03:33.230 INFO:teuthology.orchestra.run.smithi050.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T16:03:33.244 INFO:teuthology.orchestra.run.smithi190.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T16:03:33.258 INFO:teuthology.orchestra.run.smithi050.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T16:03:33.260 INFO:teuthology.orchestra.run.smithi050.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T16:03:33.260 INFO:teuthology.orchestra.run.smithi050.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T16:03:33.277 INFO:teuthology.orchestra.run.smithi190.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3305-ga430f9ed-1jammy [112 kB] 2024-04-26T16:03:33.280 INFO:teuthology.orchestra.run.smithi050.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T16:03:33.280 INFO:teuthology.orchestra.run.smithi050.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T16:03:33.280 INFO:teuthology.orchestra.run.smithi190.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3305-ga430f9ed-1jammy [466 kB] 2024-04-26T16:03:33.292 INFO:teuthology.orchestra.run.smithi050.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T16:03:33.293 INFO:teuthology.orchestra.run.smithi050.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T16:03:33.293 INFO:teuthology.orchestra.run.smithi050.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T16:03:33.294 INFO:teuthology.orchestra.run.smithi050.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T16:03:33.294 INFO:teuthology.orchestra.run.smithi050.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T16:03:33.295 INFO:teuthology.orchestra.run.smithi050.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T16:03:33.296 INFO:teuthology.orchestra.run.smithi190.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3305-ga430f9ed-1jammy [25.8 MB] 2024-04-26T16:03:33.299 INFO:teuthology.orchestra.run.smithi050.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T16:03:33.313 INFO:teuthology.orchestra.run.smithi050.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T16:03:33.315 INFO:teuthology.orchestra.run.smithi050.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T16:03:33.315 INFO:teuthology.orchestra.run.smithi050.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T16:03:33.334 INFO:teuthology.orchestra.run.smithi050.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T16:03:33.337 INFO:teuthology.orchestra.run.smithi050.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T16:03:33.337 INFO:teuthology.orchestra.run.smithi050.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T16:03:33.338 INFO:teuthology.orchestra.run.smithi050.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T16:03:33.343 INFO:teuthology.orchestra.run.smithi050.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T16:03:33.343 INFO:teuthology.orchestra.run.smithi050.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T16:03:33.346 INFO:teuthology.orchestra.run.smithi050.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T16:03:33.354 INFO:teuthology.orchestra.run.smithi050.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T16:03:33.355 INFO:teuthology.orchestra.run.smithi050.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T16:03:33.356 INFO:teuthology.orchestra.run.smithi050.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T16:03:33.375 INFO:teuthology.orchestra.run.smithi050.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T16:03:33.376 INFO:teuthology.orchestra.run.smithi050.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T16:03:33.383 INFO:teuthology.orchestra.run.smithi050.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T16:03:33.383 INFO:teuthology.orchestra.run.smithi050.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T16:03:33.383 INFO:teuthology.orchestra.run.smithi050.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T16:03:33.386 INFO:teuthology.orchestra.run.smithi050.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T16:03:33.386 INFO:teuthology.orchestra.run.smithi050.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T16:03:33.521 INFO:teuthology.orchestra.run.smithi190.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T16:03:33.529 INFO:teuthology.orchestra.run.smithi190.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T16:03:33.564 INFO:teuthology.orchestra.run.smithi190.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T16:03:33.575 INFO:teuthology.orchestra.run.smithi190.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T16:03:33.576 INFO:teuthology.orchestra.run.smithi190.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T16:03:33.578 INFO:teuthology.orchestra.run.smithi190.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T16:03:33.605 INFO:teuthology.orchestra.run.smithi190.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T16:03:33.789 INFO:teuthology.orchestra.run.smithi190.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T16:03:34.204 INFO:teuthology.orchestra.run.smithi190.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T16:03:34.258 INFO:teuthology.orchestra.run.smithi190.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T16:03:34.259 INFO:teuthology.orchestra.run.smithi190.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T16:03:34.315 INFO:teuthology.orchestra.run.smithi190.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3305-ga430f9ed-1jammy [5,094 kB] 2024-04-26T16:03:34.356 INFO:teuthology.orchestra.run.smithi190.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T16:03:34.394 INFO:teuthology.orchestra.run.smithi190.stdout:Get:30 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3305-ga430f9ed-1jammy [244 kB] 2024-04-26T16:03:34.396 INFO:teuthology.orchestra.run.smithi190.stdout:Get:31 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3305-ga430f9ed-1jammy [124 kB] 2024-04-26T16:03:34.398 INFO:teuthology.orchestra.run.smithi190.stdout:Get:32 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3305-ga430f9ed-1jammy [1,544 kB] 2024-04-26T16:03:34.422 INFO:teuthology.orchestra.run.smithi190.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T16:03:34.442 INFO:teuthology.orchestra.run.smithi190.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3305-ga430f9ed-1jammy [6,257 kB] 2024-04-26T16:03:34.454 INFO:teuthology.orchestra.run.smithi190.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T16:03:34.457 INFO:teuthology.orchestra.run.smithi190.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T16:03:34.461 INFO:teuthology.orchestra.run.smithi190.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T16:03:34.488 INFO:teuthology.orchestra.run.smithi190.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T16:03:34.492 INFO:teuthology.orchestra.run.smithi190.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T16:03:34.657 INFO:teuthology.orchestra.run.smithi190.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3305-ga430f9ed-1jammy [22.8 MB] 2024-04-26T16:03:34.662 INFO:teuthology.orchestra.run.smithi190.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T16:03:34.747 INFO:teuthology.orchestra.run.smithi190.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T16:03:34.776 INFO:teuthology.orchestra.run.smithi190.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T16:03:34.793 INFO:teuthology.orchestra.run.smithi190.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T16:03:34.838 INFO:teuthology.orchestra.run.smithi190.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T16:03:35.052 INFO:teuthology.orchestra.run.smithi190.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T16:03:35.066 INFO:teuthology.orchestra.run.smithi190.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T16:03:35.073 INFO:teuthology.orchestra.run.smithi190.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T16:03:35.078 INFO:teuthology.orchestra.run.smithi190.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T16:03:35.099 INFO:teuthology.orchestra.run.smithi190.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T16:03:35.103 INFO:teuthology.orchestra.run.smithi190.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T16:03:35.270 INFO:teuthology.orchestra.run.smithi050.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3305-ga430f9ed-1jammy [3,224 kB] 2024-04-26T16:03:35.274 INFO:teuthology.orchestra.run.smithi190.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T16:03:35.518 INFO:teuthology.orchestra.run.smithi190.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3305-ga430f9ed-1jammy [14.1 kB] 2024-04-26T16:03:35.518 INFO:teuthology.orchestra.run.smithi190.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3305-ga430f9ed-1jammy [870 kB] 2024-04-26T16:03:35.535 INFO:teuthology.orchestra.run.smithi190.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3305-ga430f9ed-1jammy [2,437 kB] 2024-04-26T16:03:35.575 INFO:teuthology.orchestra.run.smithi190.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3305-ga430f9ed-1jammy [791 kB] 2024-04-26T16:03:35.586 INFO:teuthology.orchestra.run.smithi190.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3305-ga430f9ed-1jammy [148 kB] 2024-04-26T16:03:35.591 INFO:teuthology.orchestra.run.smithi190.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3305-ga430f9ed-1jammy [2,297 kB] 2024-04-26T16:03:35.608 INFO:teuthology.orchestra.run.smithi190.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T16:03:35.639 INFO:teuthology.orchestra.run.smithi190.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T16:03:35.653 INFO:teuthology.orchestra.run.smithi190.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T16:03:35.672 INFO:teuthology.orchestra.run.smithi190.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3305-ga430f9ed-1jammy [8,625 kB] 2024-04-26T16:03:35.894 INFO:teuthology.orchestra.run.smithi190.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T16:03:35.894 INFO:teuthology.orchestra.run.smithi190.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T16:03:35.903 INFO:teuthology.orchestra.run.smithi190.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T16:03:35.907 INFO:teuthology.orchestra.run.smithi190.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T16:03:35.934 INFO:teuthology.orchestra.run.smithi190.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T16:03:35.951 INFO:teuthology.orchestra.run.smithi190.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T16:03:35.956 INFO:teuthology.orchestra.run.smithi190.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T16:03:35.974 INFO:teuthology.orchestra.run.smithi190.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3305-ga430f9ed-1jammy [14.2 kB] 2024-04-26T16:03:35.975 INFO:teuthology.orchestra.run.smithi190.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3305-ga430f9ed-1jammy [51.7 MB] 2024-04-26T16:03:36.138 INFO:teuthology.orchestra.run.smithi190.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T16:03:36.392 INFO:teuthology.orchestra.run.smithi190.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T16:03:36.571 INFO:teuthology.orchestra.run.smithi190.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T16:03:36.585 INFO:teuthology.orchestra.run.smithi190.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T16:03:36.605 INFO:teuthology.orchestra.run.smithi190.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T16:03:36.669 INFO:teuthology.orchestra.run.smithi190.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T16:03:37.099 INFO:teuthology.orchestra.run.smithi050.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3305-ga430f9ed-1jammy [712 kB] 2024-04-26T16:03:37.112 INFO:teuthology.orchestra.run.smithi190.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T16:03:37.114 INFO:teuthology.orchestra.run.smithi190.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T16:03:37.117 INFO:teuthology.orchestra.run.smithi190.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T16:03:37.118 INFO:teuthology.orchestra.run.smithi190.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T16:03:37.121 INFO:teuthology.orchestra.run.smithi190.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T16:03:37.354 INFO:teuthology.orchestra.run.smithi190.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T16:03:37.525 INFO:teuthology.orchestra.run.smithi050.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3305-ga430f9ed-1jammy [356 kB] 2024-04-26T16:03:37.749 INFO:teuthology.orchestra.run.smithi050.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3305-ga430f9ed-1jammy [32.8 kB] 2024-04-26T16:03:37.761 INFO:teuthology.orchestra.run.smithi050.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3305-ga430f9ed-1jammy [180 kB] 2024-04-26T16:03:37.807 INFO:teuthology.orchestra.run.smithi190.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T16:03:37.848 INFO:teuthology.orchestra.run.smithi190.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T16:03:37.856 INFO:teuthology.orchestra.run.smithi050.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3305-ga430f9ed-1jammy [67.1 kB] 2024-04-26T16:03:37.906 INFO:teuthology.orchestra.run.smithi050.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3305-ga430f9ed-1jammy [330 kB] 2024-04-26T16:03:37.979 INFO:teuthology.orchestra.run.smithi050.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3305-ga430f9ed-1jammy [6,892 kB] 2024-04-26T16:03:37.992 INFO:teuthology.orchestra.run.smithi190.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3305-ga430f9ed-1jammy [124 kB] 2024-04-26T16:03:37.995 INFO:teuthology.orchestra.run.smithi190.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3305-ga430f9ed-1jammy [38.9 kB] 2024-04-26T16:03:37.996 INFO:teuthology.orchestra.run.smithi190.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3305-ga430f9ed-1jammy [13.6 MB] 2024-04-26T16:03:38.200 INFO:teuthology.orchestra.run.smithi190.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T16:03:38.201 INFO:teuthology.orchestra.run.smithi190.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T16:03:38.207 INFO:teuthology.orchestra.run.smithi190.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T16:03:38.213 INFO:teuthology.orchestra.run.smithi190.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T16:03:38.218 INFO:teuthology.orchestra.run.smithi190.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T16:03:38.220 INFO:teuthology.orchestra.run.smithi190.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T16:03:38.221 INFO:teuthology.orchestra.run.smithi190.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T16:03:38.222 INFO:teuthology.orchestra.run.smithi190.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T16:03:38.350 INFO:teuthology.orchestra.run.smithi190.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3305-ga430f9ed-1jammy [91.4 kB] 2024-04-26T16:03:38.412 INFO:teuthology.orchestra.run.smithi190.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T16:03:38.819 INFO:teuthology.orchestra.run.smithi190.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T16:03:38.883 INFO:teuthology.orchestra.run.smithi190.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T16:03:38.902 INFO:teuthology.orchestra.run.smithi190.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T16:03:39.002 INFO:teuthology.orchestra.run.smithi190.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T16:03:39.044 INFO:teuthology.orchestra.run.smithi190.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T16:03:39.060 INFO:teuthology.orchestra.run.smithi190.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T16:03:39.067 INFO:teuthology.orchestra.run.smithi190.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T16:03:39.111 INFO:teuthology.orchestra.run.smithi190.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T16:03:39.114 INFO:teuthology.orchestra.run.smithi190.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T16:03:39.134 INFO:teuthology.orchestra.run.smithi190.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T16:03:39.299 INFO:teuthology.orchestra.run.smithi190.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T16:03:39.573 INFO:teuthology.orchestra.run.smithi190.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T16:03:39.591 INFO:teuthology.orchestra.run.smithi190.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T16:03:39.749 INFO:teuthology.orchestra.run.smithi190.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T16:03:39.776 INFO:teuthology.orchestra.run.smithi050.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3305-ga430f9ed-1jammy [112 kB] 2024-04-26T16:03:39.789 INFO:teuthology.orchestra.run.smithi050.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3305-ga430f9ed-1jammy [466 kB] 2024-04-26T16:03:39.789 INFO:teuthology.orchestra.run.smithi190.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T16:03:39.894 INFO:teuthology.orchestra.run.smithi050.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3305-ga430f9ed-1jammy [25.8 MB] 2024-04-26T16:03:39.998 INFO:teuthology.orchestra.run.smithi190.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T16:03:40.000 INFO:teuthology.orchestra.run.smithi190.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T16:03:40.002 INFO:teuthology.orchestra.run.smithi190.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T16:03:40.024 INFO:teuthology.orchestra.run.smithi190.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T16:03:40.031 INFO:teuthology.orchestra.run.smithi190.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T16:03:40.727 INFO:teuthology.orchestra.run.smithi190.stdout:Fetched 181 MB in 8s (23.7 MB/s) 2024-04-26T16:03:40.888 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T16:03:40.988 INFO:teuthology.orchestra.run.smithi190.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T16:03:40.994 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T16:03:41.078 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T16:03:41.440 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T16:03:41.466 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T16:03:41.506 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T16:03:41.818 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T16:03:41.844 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T16:03:41.884 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T16:03:42.329 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T16:03:42.356 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T16:03:42.405 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:03:42.816 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T16:03:42.842 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T16:03:42.882 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:03:43.177 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T16:03:43.204 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T16:03:43.243 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:03:43.605 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T16:03:43.631 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T16:03:43.671 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T16:03:43.974 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librados2. 2024-04-26T16:03:44.000 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../007-librados2_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:44.040 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librados2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:44.602 INFO:teuthology.orchestra.run.smithi050.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3305-ga430f9ed-1jammy [5,094 kB] 2024-04-26T16:03:44.660 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librbd1. 2024-04-26T16:03:44.676 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../008-librbd1_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:44.710 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librbd1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:45.130 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T16:03:45.155 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:45.221 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:45.367 INFO:teuthology.orchestra.run.smithi050.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3305-ga430f9ed-1jammy [244 kB] 2024-04-26T16:03:45.372 INFO:teuthology.orchestra.run.smithi050.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3305-ga430f9ed-1jammy [124 kB] 2024-04-26T16:03:45.375 INFO:teuthology.orchestra.run.smithi050.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3305-ga430f9ed-1jammy [1,544 kB] 2024-04-26T16:03:45.486 INFO:teuthology.orchestra.run.smithi050.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3305-ga430f9ed-1jammy [6,257 kB] 2024-04-26T16:03:45.498 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rados. 2024-04-26T16:03:45.525 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:45.565 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rados (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:45.860 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T16:03:45.886 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:03:45.926 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-argparse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:46.178 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T16:03:46.205 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:46.218 INFO:teuthology.orchestra.run.smithi050.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3305-ga430f9ed-1jammy [22.8 MB] 2024-04-26T16:03:46.245 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cephfs (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:46.556 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T16:03:46.582 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:03:46.622 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:46.976 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T16:03:47.002 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T16:03:47.042 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T16:03:47.295 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T16:03:47.321 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T16:03:47.353 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T16:03:47.614 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T16:03:47.629 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:47.672 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rbd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:48.025 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T16:03:48.051 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T16:03:48.092 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T16:03:48.369 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T16:03:48.384 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T16:03:48.419 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T16:03:48.680 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T16:03:48.706 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T16:03:48.739 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T16:03:48.873 INFO:teuthology.orchestra.run.smithi050.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3305-ga430f9ed-1jammy [14.1 kB] 2024-04-26T16:03:48.907 INFO:teuthology.orchestra.run.smithi050.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3305-ga430f9ed-1jammy [870 kB] 2024-04-26T16:03:48.949 INFO:teuthology.orchestra.run.smithi050.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3305-ga430f9ed-1jammy [2,437 kB] 2024-04-26T16:03:49.100 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua5.1. 2024-04-26T16:03:49.127 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T16:03:49.148 INFO:teuthology.orchestra.run.smithi050.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3305-ga430f9ed-1jammy [791 kB] 2024-04-26T16:03:49.166 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T16:03:49.217 INFO:teuthology.orchestra.run.smithi050.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3305-ga430f9ed-1jammy [148 kB] 2024-04-26T16:03:49.233 INFO:teuthology.orchestra.run.smithi050.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3305-ga430f9ed-1jammy [2,297 kB] 2024-04-26T16:03:49.432 INFO:teuthology.orchestra.run.smithi050.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3305-ga430f9ed-1jammy [8,625 kB] 2024-04-26T16:03:49.486 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-any. 2024-04-26T16:03:49.512 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T16:03:49.678 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T16:03:49.930 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package zip. 2024-04-26T16:03:49.946 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T16:03:49.988 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T16:03:50.154 INFO:teuthology.orchestra.run.smithi050.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3305-ga430f9ed-1jammy [14.2 kB] 2024-04-26T16:03:50.155 INFO:teuthology.orchestra.run.smithi050.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3305-ga430f9ed-1jammy [51.7 MB] 2024-04-26T16:03:50.300 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package unzip. 2024-04-26T16:03:50.327 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T16:03:50.367 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T16:03:50.762 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package luarocks. 2024-04-26T16:03:50.789 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T16:03:50.828 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T16:03:51.161 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package librgw2. 2024-04-26T16:03:51.188 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../025-librgw2_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:51.219 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking librgw2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:51.755 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T16:03:51.781 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:51.821 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:52.158 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T16:03:52.184 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T16:03:52.224 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T16:03:52.544 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T16:03:52.570 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:52.610 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libradosstriper1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:53.022 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-common. 2024-04-26T16:03:53.048 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:53.088 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:54.518 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-base. 2024-04-26T16:03:54.545 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:03:54.594 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-base (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:03:54.903 INFO:teuthology.orchestra.run.smithi050.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3305-ga430f9ed-1jammy [124 kB] 2024-04-26T16:03:54.906 INFO:teuthology.orchestra.run.smithi050.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3305-ga430f9ed-1jammy [38.9 kB] 2024-04-26T16:03:54.907 INFO:teuthology.orchestra.run.smithi050.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3305-ga430f9ed-1jammy [13.6 MB] 2024-04-26T16:03:55.298 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T16:03:55.324 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T16:03:55.364 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T16:03:55.633 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T16:03:55.660 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T16:03:55.700 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T16:03:56.036 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T16:03:56.063 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T16:03:56.103 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T16:03:56.229 INFO:teuthology.orchestra.run.smithi050.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3305-ga430f9ed-1jammy [91.4 kB] 2024-04-26T16:03:56.406 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T16:03:56.432 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T16:03:56.472 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T16:03:56.758 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T16:03:56.776 INFO:teuthology.orchestra.run.smithi050.stdout:Fetched 181 MB in 24s (7,693 kB/s) 2024-04-26T16:03:56.784 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T16:03:56.816 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T16:03:57.057 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T16:03:57.094 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T16:03:57.121 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T16:03:57.161 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T16:03:57.162 INFO:teuthology.orchestra.run.smithi050.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T16:03:57.168 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T16:03:57.256 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T16:03:57.463 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-portend. 2024-04-26T16:03:57.479 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T16:03:57.521 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T16:03:57.593 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T16:03:57.620 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T16:03:57.659 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T16:03:57.824 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T16:03:57.851 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T16:03:57.891 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T16:03:57.954 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T16:03:57.981 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T16:03:58.020 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T16:03:58.144 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T16:03:58.156 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T16:03:58.185 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T16:03:58.457 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T16:03:58.485 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T16:03:58.530 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T16:03:58.532 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:03:58.557 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T16:03:58.597 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T16:03:58.883 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T16:03:58.899 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T16:03:58.927 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T16:03:58.933 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T16:03:58.954 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T16:03:58.994 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:03:59.186 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T16:03:59.212 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T16:03:59.252 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T16:03:59.313 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T16:03:59.340 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T16:03:59.380 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:03:59.539 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-mako. 2024-04-26T16:03:59.565 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T16:03:59.605 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T16:03:59.725 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T16:03:59.751 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T16:03:59.791 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T16:03:59.933 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T16:03:59.959 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T16:03:59.999 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T16:04:00.111 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librados2. 2024-04-26T16:04:00.138 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../007-librados2_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:00.177 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librados2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:00.311 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T16:04:00.337 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T16:04:00.377 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T16:04:00.638 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webob. 2024-04-26T16:04:00.653 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T16:04:00.688 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T16:04:00.722 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librbd1. 2024-04-26T16:04:00.748 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../008-librbd1_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:00.780 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librbd1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:00.957 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T16:04:00.973 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T16:04:01.019 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T16:04:01.233 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T16:04:01.260 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:01.299 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:01.310 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T16:04:01.337 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T16:04:01.377 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T16:04:01.619 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rados. 2024-04-26T16:04:01.646 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:01.680 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-paste. 2024-04-26T16:04:01.685 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rados (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:01.695 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T16:04:01.738 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T16:04:02.005 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T16:04:02.016 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T16:04:02.031 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T16:04:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:02.071 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-argparse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:02.074 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T16:04:02.307 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T16:04:02.323 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:02.352 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T16:04:02.357 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cephfs (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:02.378 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T16:04:02.418 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T16:04:02.651 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T16:04:02.678 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:02.718 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:02.721 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T16:04:02.747 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T16:04:02.788 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T16:04:03.021 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T16:04:03.048 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T16:04:03.057 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T16:04:03.073 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T16:04:03.087 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T16:04:03.115 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T16:04:03.398 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T16:04:03.425 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T16:04:03.454 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T16:04:03.465 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T16:04:03.480 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T16:04:03.512 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T16:04:03.701 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T16:04:03.717 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:03.759 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rbd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:03.823 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-idna. 2024-04-26T16:04:03.849 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T16:04:03.889 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T16:04:04.087 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T16:04:04.114 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T16:04:04.154 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T16:04:04.209 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T16:04:04.235 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T16:04:04.275 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T16:04:04.415 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T16:04:04.431 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T16:04:04.473 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T16:04:04.629 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests. 2024-04-26T16:04:04.655 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T16:04:04.695 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T16:04:04.784 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T16:04:04.804 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T16:04:04.862 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T16:04:05.138 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua5.1. 2024-04-26T16:04:05.165 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T16:04:05.204 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T16:04:05.499 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua-any. 2024-04-26T16:04:05.515 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T16:04:05.557 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T16:04:05.838 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T16:04:05.865 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T16:04:05.998 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T16:04:06.278 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T16:04:06.294 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:06.336 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:06.569 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package zip. 2024-04-26T16:04:06.596 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T16:04:06.635 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T16:04:06.733 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T16:04:06.759 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:06.799 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:06.938 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package unzip. 2024-04-26T16:04:06.965 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T16:04:07.004 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T16:04:07.102 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T16:04:07.129 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T16:04:07.168 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T16:04:07.392 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package luarocks. 2024-04-26T16:04:07.418 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T16:04:07.458 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T16:04:07.471 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T16:04:07.498 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T16:04:07.538 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T16:04:07.816 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package librgw2. 2024-04-26T16:04:07.843 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../025-librgw2_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:07.882 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking librgw2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:07.908 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T16:04:07.934 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:07.982 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:08.418 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T16:04:08.419 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T16:04:08.434 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:08.445 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:08.485 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mon (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:08.485 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:08.838 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T16:04:08.865 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T16:04:08.904 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T16:04:09.080 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T16:04:09.107 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:09.147 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-osd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:09.216 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T16:04:09.243 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:09.282 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libradosstriper1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:09.636 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-common. 2024-04-26T16:04:09.663 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:09.694 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:10.191 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph. 2024-04-26T16:04:10.207 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../066-ceph_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:10.250 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:10.569 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T16:04:10.596 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:10.635 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:11.006 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T16:04:11.021 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:11.072 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mds (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:11.291 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-base. 2024-04-26T16:04:11.318 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:11.367 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-base (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:11.450 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package cephadm. 2024-04-26T16:04:11.476 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../069-cephadm_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:11.516 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:11.837 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T16:04:11.862 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T16:04:11.902 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T16:04:11.937 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T16:04:11.964 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T16:04:12.020 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T16:04:12.247 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T16:04:12.262 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T16:04:12.305 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T16:04:12.323 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T16:04:12.350 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T16:04:12.389 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T16:04:12.701 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T16:04:12.727 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T16:04:12.767 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T16:04:13.053 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T16:04:13.081 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T16:04:13.120 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T16:04:13.310 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-babel. 2024-04-26T16:04:13.337 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T16:04:13.373 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T16:04:13.376 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T16:04:13.400 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T16:04:13.439 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T16:04:13.712 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T16:04:13.739 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T16:04:13.742 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T16:04:13.769 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T16:04:13.779 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T16:04:13.809 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T16:04:14.121 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-portend. 2024-04-26T16:04:14.132 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T16:04:14.147 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T16:04:14.158 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:14.186 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T16:04:14.198 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:14.456 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T16:04:14.483 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T16:04:14.514 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T16:04:14.518 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T16:04:14.545 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T16:04:14.584 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T16:04:14.817 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T16:04:14.844 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T16:04:14.883 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T16:04:14.896 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-routes. 2024-04-26T16:04:14.923 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T16:04:14.962 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T16:04:15.221 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T16:04:15.248 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T16:04:15.287 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T16:04:15.299 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T16:04:15.325 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:15.365 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:15.599 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T16:04:15.625 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T16:04:15.665 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T16:04:15.935 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T16:04:15.961 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T16:04:15.993 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T16:04:16.287 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-mako. 2024-04-26T16:04:16.314 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T16:04:16.354 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T16:04:16.707 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T16:04:16.734 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T16:04:16.770 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T16:04:16.773 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T16:04:16.786 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T16:04:16.828 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T16:04:17.076 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T16:04:17.104 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T16:04:17.143 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T16:04:17.340 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T16:04:17.368 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T16:04:17.408 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T16:04:17.413 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webob. 2024-04-26T16:04:17.440 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T16:04:17.471 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T16:04:17.735 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T16:04:17.762 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T16:04:17.774 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T16:04:17.793 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T16:04:17.801 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T16:04:17.835 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T16:04:18.096 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T16:04:18.110 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T16:04:18.122 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T16:04:18.137 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T16:04:18.162 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T16:04:18.176 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T16:04:18.479 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-paste. 2024-04-26T16:04:18.507 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T16:04:18.546 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T16:04:18.849 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T16:04:18.854 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T16:04:18.876 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T16:04:18.881 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:18.916 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T16:04:18.997 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:19.218 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T16:04:19.246 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T16:04:19.285 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T16:04:19.588 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T16:04:19.617 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T16:04:19.655 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T16:04:19.941 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T16:04:19.967 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T16:04:19.984 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T16:04:20.007 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T16:04:20.009 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T16:04:20.050 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T16:04:20.328 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T16:04:20.355 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T16:04:20.403 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T16:04:20.404 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T16:04:20.431 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T16:04:20.470 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T16:04:20.690 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T16:04:20.717 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T16:04:20.740 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-idna. 2024-04-26T16:04:20.748 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T16:04:20.757 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T16:04:20.790 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T16:04:21.009 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T16:04:21.025 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T16:04:21.067 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T16:04:21.093 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T16:04:21.120 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T16:04:21.159 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T16:04:21.405 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T16:04:21.432 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T16:04:21.463 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-requests. 2024-04-26T16:04:21.471 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T16:04:21.490 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T16:04:21.529 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T16:04:21.791 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T16:04:21.818 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T16:04:21.840 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T16:04:21.857 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T16:04:21.869 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T16:04:21.907 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T16:04:22.169 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T16:04:22.193 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T16:04:22.195 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T16:04:22.209 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:22.235 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T16:04:22.251 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:22.580 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T16:04:22.590 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T16:04:22.610 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T16:04:22.621 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:22.647 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T16:04:22.657 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:22.968 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T16:04:22.995 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T16:04:23.034 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T16:04:23.337 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T16:04:23.343 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T16:04:23.359 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:23.364 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T16:04:23.401 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:23.403 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T16:04:23.706 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T16:04:23.734 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:23.746 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T16:04:23.765 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:23.773 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T16:04:23.812 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T16:04:24.099 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T16:04:24.114 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T16:04:24.149 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T16:04:24.159 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T16:04:24.187 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:24.226 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mon (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:24.401 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package jq. 2024-04-26T16:04:24.417 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T16:04:24.459 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T16:04:24.712 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package socat. 2024-04-26T16:04:24.739 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T16:04:24.787 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T16:04:24.822 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T16:04:24.848 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:24.888 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-osd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:25.090 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T16:04:25.117 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T16:04:25.156 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T16:04:25.527 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-test. 2024-04-26T16:04:25.543 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:25.577 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-test (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:25.974 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph. 2024-04-26T16:04:25.990 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../066-ceph_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:26.032 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:26.302 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T16:04:26.329 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:26.461 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:27.047 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T16:04:27.074 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:27.130 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mds (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:27.525 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package cephadm. 2024-04-26T16:04:27.552 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../069-cephadm_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:27.591 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:27.860 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T16:04:27.888 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T16:04:27.927 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T16:04:28.006 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T16:04:28.034 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:28.073 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking ceph-volume (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:28.238 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T16:04:28.254 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T16:04:28.296 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T16:04:28.376 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T16:04:28.400 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:28.443 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking libcephfs-dev (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:28.763 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T16:04:28.790 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T16:04:28.829 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T16:04:29.108 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T16:04:29.136 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T16:04:29.166 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T16:04:29.284 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-babel. 2024-04-26T16:04:29.312 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T16:04:29.351 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T16:04:29.537 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T16:04:29.564 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T16:04:29.612 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T16:04:29.679 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T16:04:29.706 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T16:04:29.745 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T16:04:29.991 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package pkg-config. 2024-04-26T16:04:30.007 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T16:04:30.050 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T16:04:30.090 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T16:04:30.118 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:30.156 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:30.286 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T16:04:30.302 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T16:04:30.344 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T16:04:30.476 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T16:04:30.504 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T16:04:30.543 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T16:04:30.756 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T16:04:30.783 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T16:04:30.814 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T16:04:30.846 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-routes. 2024-04-26T16:04:30.873 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T16:04:30.912 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T16:04:31.099 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T16:04:31.127 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T16:04:31.166 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T16:04:31.266 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T16:04:31.293 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:31.332 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:31.511 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T16:04:31.538 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T16:04:31.577 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T16:04:31.897 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T16:04:31.925 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T16:04:31.964 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T16:04:32.308 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-py. 2024-04-26T16:04:32.336 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T16:04:32.374 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T16:04:32.678 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T16:04:32.706 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T16:04:32.720 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T16:04:32.737 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T16:04:32.744 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T16:04:32.779 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T16:04:33.165 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T16:04:33.192 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T16:04:33.223 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T16:04:33.282 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T16:04:33.310 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T16:04:33.349 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T16:04:33.525 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-toml. 2024-04-26T16:04:33.553 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T16:04:33.592 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T16:04:33.677 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T16:04:33.705 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T16:04:33.743 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T16:04:33.878 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T16:04:33.905 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T16:04:33.944 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T16:04:34.013 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T16:04:34.041 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T16:04:34.071 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T16:04:34.256 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T16:04:34.284 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T16:04:34.323 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T16:04:34.583 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T16:04:34.599 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T16:04:34.642 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T16:04:34.749 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T16:04:34.764 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:34.890 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:35.421 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package radosgw. 2024-04-26T16:04:35.448 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../116-radosgw_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:35.495 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking radosgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:35.902 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T16:04:35.930 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T16:04:35.968 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T16:04:36.265 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T16:04:36.293 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T16:04:36.298 INFO:teuthology.orchestra.run.smithi190.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T16:04:36.326 INFO:teuthology.orchestra.run.smithi190.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:36.331 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T16:04:36.365 INFO:teuthology.orchestra.run.smithi190.stdout:Unpacking rbd-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:36.651 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T16:04:36.652 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T16:04:36.679 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T16:04:36.718 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T16:04:36.918 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T16:04:37.029 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T16:04:37.057 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T16:04:37.095 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T16:04:37.287 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T16:04:37.399 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T16:04:37.427 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T16:04:37.466 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T16:04:37.542 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T16:04:37.761 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T16:04:37.789 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T16:04:37.827 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T16:04:37.890 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T16:04:38.138 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T16:04:38.166 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T16:04:38.205 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T16:04:38.307 INFO:teuthology.orchestra.run.smithi190.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T16:04:38.372 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:38.517 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T16:04:38.545 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T16:04:38.583 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T16:04:38.880 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user cephadm....done 2024-04-26T16:04:39.061 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T16:04:39.258 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T16:04:39.272 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:39.305 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:39.346 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T16:04:39.572 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T16:04:39.650 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T16:04:39.673 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T16:04:39.678 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T16:04:39.716 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T16:04:39.933 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T16:04:39.994 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T16:04:40.010 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T16:04:40.044 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T16:04:40.192 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T16:04:40.314 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package jq. 2024-04-26T16:04:40.318 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T16:04:40.341 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T16:04:40.380 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T16:04:40.628 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T16:04:40.641 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package socat. 2024-04-26T16:04:40.657 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T16:04:40.699 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T16:04:40.994 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T16:04:41.005 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T16:04:41.010 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T16:04:41.052 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T16:04:41.239 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T16:04:41.407 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-test. 2024-04-26T16:04:41.435 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:41.473 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-test (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:41.507 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T16:04:41.784 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T16:04:42.051 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T16:04:42.319 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T16:04:42.578 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-argparse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:42.896 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T16:04:42.997 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T16:04:43.098 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T16:04:43.216 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T16:04:43.466 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T16:04:43.848 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T16:04:43.869 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T16:04:43.881 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T16:04:43.885 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3305-ga430f9ed-1jammy_all.deb ... 2024-04-26T16:04:43.919 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking ceph-volume (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:43.936 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T16:04:44.071 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T16:04:44.198 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T16:04:44.226 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:44.273 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking libcephfs-dev (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:44.414 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T16:04:44.515 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T16:04:44.575 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T16:04:44.603 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T16:04:44.642 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T16:04:44.765 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T16:04:44.980 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T16:04:45.008 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T16:04:45.033 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T16:04:45.046 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T16:04:45.343 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T16:04:45.350 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T16:04:45.378 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T16:04:45.408 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T16:04:45.469 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T16:04:45.805 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package pkg-config. 2024-04-26T16:04:45.833 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T16:04:45.871 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T16:04:46.133 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T16:04:46.161 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T16:04:46.162 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T16:04:46.199 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T16:04:46.397 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T16:04:46.497 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T16:04:46.607 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T16:04:46.611 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T16:04:46.639 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T16:04:46.677 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T16:04:46.925 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T16:04:46.980 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T16:04:47.008 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T16:04:47.046 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T16:04:47.317 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T16:04:47.392 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T16:04:47.420 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T16:04:47.458 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T16:04:47.568 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T16:04:47.778 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T16:04:47.806 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T16:04:47.844 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T16:04:47.953 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T16:04:48.180 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-py. 2024-04-26T16:04:48.208 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T16:04:48.254 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T16:04:48.255 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T16:04:48.581 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T16:04:48.592 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T16:04:48.621 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T16:04:48.658 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T16:04:48.815 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T16:04:48.941 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:49.079 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T16:04:49.095 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T16:04:49.128 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T16:04:49.242 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T16:04:49.423 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-toml. 2024-04-26T16:04:49.451 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T16:04:49.489 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T16:04:49.560 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T16:04:49.776 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T16:04:49.804 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T16:04:49.842 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T16:04:50.120 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T16:04:50.137 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T16:04:50.178 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T16:04:50.456 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T16:04:50.484 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T16:04:50.523 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T16:04:50.749 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T16:04:51.000 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:04:51.134 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T16:04:51.285 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package radosgw. 2024-04-26T16:04:51.313 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../116-radosgw_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:51.351 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking radosgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:51.402 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T16:04:51.528 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T16:04:51.639 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T16:04:51.907 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T16:04:52.175 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T16:04:52.230 INFO:teuthology.orchestra.run.smithi050.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T16:04:52.246 INFO:teuthology.orchestra.run.smithi050.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3305-ga430f9ed-1jammy_amd64.deb ... 2024-04-26T16:04:52.287 INFO:teuthology.orchestra.run.smithi050.stdout:Unpacking rbd-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:52.426 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:04:52.576 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T16:04:52.811 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T16:04:52.868 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T16:04:52.968 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T16:04:53.112 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T16:04:53.233 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T16:04:53.354 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T16:04:53.593 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T16:04:53.597 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T16:04:53.840 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T16:04:54.040 INFO:teuthology.orchestra.run.smithi050.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T16:04:54.091 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T16:04:54.330 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:54.358 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T16:04:54.485 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T16:04:54.769 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T16:04:54.895 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T16:04:54.913 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user cephadm....done 2024-04-26T16:04:55.061 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T16:04:55.188 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T16:04:55.345 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T16:04:55.472 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T16:04:55.596 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T16:04:55.706 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T16:04:55.715 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T16:04:55.841 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T16:04:55.965 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T16:04:55.966 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T16:04:56.072 INFO:teuthology.orchestra.run.smithi190.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T16:04:56.233 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T16:04:56.302 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T16:04:56.334 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T16:04:56.428 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T16:04:56.628 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T16:04:56.829 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T16:04:56.988 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T16:04:57.072 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T16:04:57.181 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T16:04:57.231 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T16:04:57.432 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:04:57.515 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T16:04:57.558 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T16:04:57.759 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T16:04:57.999 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T16:04:58.018 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T16:04:58.285 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T16:04:58.502 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T16:04:58.553 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T16:04:58.679 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T16:04:58.895 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-argparse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:04:58.955 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T16:04:59.154 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T16:04:59.223 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T16:04:59.256 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T16:04:59.381 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T16:04:59.516 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T16:04:59.568 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T16:04:59.784 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T16:04:59.953 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T16:05:00.155 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T16:05:00.191 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T16:05:00.246 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T16:05:00.253 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T16:05:00.379 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T16:05:00.590 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T16:05:00.716 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librados2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:00.747 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T16:05:00.842 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:00.873 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T16:05:00.968 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T16:05:01.158 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T16:05:01.304 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:01.425 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T16:05:01.430 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libradosstriper1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:01.531 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T16:05:01.727 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T16:05:01.824 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librbd1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:01.828 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T16:05:01.950 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-modules-core (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:02.085 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:02.278 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T16:05:02.278 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T16:05:02.530 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T16:05:02.728 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up libcephfs-dev (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:02.781 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T16:05:02.845 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rados (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:02.915 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T16:05:02.971 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up librgw2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:03.041 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T16:05:03.097 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T16:05:03.351 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T16:05:03.735 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T16:05:03.978 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T16:05:04.359 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rbd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:04.363 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T16:05:04.460 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up rbd-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:04.561 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-rgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:04.656 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T16:05:04.678 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up python3-cephfs (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:04.804 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:05.007 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T16:05:05.267 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T16:05:05.322 INFO:teuthology.orchestra.run.smithi190.stdout:Adding group ceph....done 2024-04-26T16:05:05.393 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:05.685 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T16:05:05.699 INFO:teuthology.orchestra.run.smithi190.stdout:Adding system user ceph....done 2024-04-26T16:05:05.883 INFO:teuthology.orchestra.run.smithi190.stdout:Setting system user ceph properties....done 2024-04-26T16:05:05.898 INFO:teuthology.orchestra.run.smithi190.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T16:05:06.003 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T16:05:06.033 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T16:05:06.330 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T16:05:06.805 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-test (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:06.940 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up radosgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:07.234 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T16:05:07.402 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T16:05:07.402 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T16:05:07.444 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:05:07.570 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T16:05:07.838 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T16:05:07.891 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-base (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:07.980 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T16:05:08.106 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T16:05:08.213 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T16:05:08.441 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T16:05:08.709 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T16:05:08.736 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mds (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:08.951 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:05:08.958 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T16:05:08.958 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T16:05:09.052 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T16:05:09.354 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T16:05:09.438 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:09.597 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T16:05:09.651 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T16:05:09.651 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T16:05:09.873 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T16:05:10.156 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-osd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:10.157 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T16:05:10.410 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T16:05:10.427 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T16:05:10.427 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T16:05:10.676 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T16:05:10.785 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T16:05:10.891 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:11.017 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:11.053 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T16:05:11.154 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T16:05:11.185 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mon (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:11.396 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T16:05:11.396 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T16:05:11.430 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T16:05:11.748 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T16:05:11.886 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:12.016 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T16:05:12.045 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:12.142 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T16:05:12.179 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-mgr-dashboard (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:12.251 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T16:05:12.324 INFO:teuthology.orchestra.run.smithi050.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T16:05:12.339 INFO:teuthology.orchestra.run.smithi190.stdout:Setting up ceph-volume (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:12.578 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T16:05:12.687 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T16:05:12.691 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T16:05:12.800 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T16:05:12.953 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T16:05:13.055 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T16:05:13.323 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T16:05:13.466 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T16:05:13.934 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T16:05:14.068 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T16:05:14.369 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T16:05:14.466 INFO:teuthology.orchestra.run.smithi190.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T16:05:14.678 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T16:05:14.963 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T16:05:15.089 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T16:05:15.140 INFO:teuthology.orchestra.run.smithi190.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T16:05:15.146 DEBUG:teuthology.parallel:result is None 2024-04-26T16:05:15.356 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T16:05:15.649 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T16:05:16.011 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T16:05:16.388 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T16:05:16.681 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T16:05:17.058 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T16:05:17.171 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librados2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:17.269 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:17.370 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T16:05:17.706 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:17.857 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libradosstriper1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:17.983 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T16:05:18.268 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librbd1 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:18.394 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-modules-core (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:18.504 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:18.706 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T16:05:18.706 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T16:05:19.188 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up libcephfs-dev (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:19.289 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rados (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:19.407 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up librgw2 (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:19.533 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T16:05:20.803 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rbd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:20.921 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up rbd-fuse (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:21.022 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-rgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:21.123 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up python3-cephfs (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:21.249 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-common (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:21.825 INFO:teuthology.orchestra.run.smithi050.stdout:Adding group ceph....done 2024-04-26T16:05:22.168 INFO:teuthology.orchestra.run.smithi050.stdout:Adding system user ceph....done 2024-04-26T16:05:22.321 INFO:teuthology.orchestra.run.smithi050.stdout:Setting system user ceph properties....done 2024-04-26T16:05:22.340 INFO:teuthology.orchestra.run.smithi050.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T16:05:22.481 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T16:05:22.808 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T16:05:23.275 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-test (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:23.401 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up radosgw (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:23.873 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T16:05:23.873 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T16:05:24.353 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-base (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:24.685 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T16:05:25.146 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mds (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:25.337 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T16:05:25.337 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T16:05:25.815 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:26.045 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T16:05:26.046 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T16:05:26.527 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-osd (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:26.799 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T16:05:26.799 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T16:05:27.295 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:27.396 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:27.539 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mon (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:27.743 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T16:05:27.743 INFO:teuthology.orchestra.run.smithi050.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T16:05:28.257 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-cephadm (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:28.408 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:28.542 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-mgr-dashboard (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:28.693 INFO:teuthology.orchestra.run.smithi050.stdout:Setting up ceph-volume (19.0.0-3305-ga430f9ed-1jammy) ... 2024-04-26T16:05:29.055 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T16:05:29.198 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T16:05:29.442 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T16:05:30.577 INFO:teuthology.orchestra.run.smithi050.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T16:05:31.275 INFO:teuthology.orchestra.run.smithi050.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T16:05:31.281 DEBUG:teuthology.parallel:result is None 2024-04-26T16:05:31.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:05:31.498 DEBUG:teuthology.orchestra.run.smithi050:> dpkg-query -W -f '${Version}' ceph 2024-04-26T16:05:31.520 INFO:teuthology.orchestra.run.smithi050.stdout:19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:05:31.520 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:05:31.521 INFO:teuthology.task.install:The correct ceph version 19.0.0-3305-ga430f9ed-1jammy is installed. 2024-04-26T16:05:31.523 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:05:31.737 DEBUG:teuthology.orchestra.run.smithi190:> dpkg-query -W -f '${Version}' ceph 2024-04-26T16:05:31.757 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:05:31.757 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3305-ga430f9ed-1jammy 2024-04-26T16:05:31.757 INFO:teuthology.task.install:The correct ceph version 19.0.0-3305-ga430f9ed-1jammy is installed. 2024-04-26T16:05:31.759 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T16:05:31.759 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:05:31.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T16:05:31.774 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:05:31.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T16:05:31.811 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T16:05:31.811 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:05:31.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T16:05:31.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T16:05:31.886 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:05:31.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T16:05:31.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T16:05:31.954 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T16:05:31.954 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:05:31.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T16:05:31.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T16:05:32.023 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:05:32.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T16:05:32.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T16:05:32.093 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T16:05:32.094 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:05:32.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T16:05:32.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T16:05:32.163 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:05:32.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T16:05:32.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T16:05:32.230 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-26T16:05:32.352 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6', 'cephadm_mode': 'root'} 2024-04-26T16:05:32.352 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:05:32.353 INFO:tasks.cephadm:Cluster fsid is cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:05:32.353 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-26T16:05:32.353 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.50', 'mon.c': '[v2:172.21.15.50:3301,v1:172.21.15.50:6790]', 'mon.b': '172.21.15.190'} 2024-04-26T16:05:32.353 INFO:tasks.cephadm:First mon is mon.a on smithi050 2024-04-26T16:05:32.353 INFO:tasks.cephadm:First mgr is y 2024-04-26T16:05:32.353 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-26T16:05:32.353 DEBUG:teuthology.orchestra.run.smithi050:> sudo hostname $(hostname -s) 2024-04-26T16:05:32.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-04-26T16:05:32.384 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-26T16:05:32.385 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:05:32.603 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic/78909/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3305-ga430f9ed', 'node_name': '172.21.5.33+adami03', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jammy,DIST=jammy,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3305-ga430f9ed-1jammy'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default/', 'distro_codename': 'jammy', 'modified': '2024-04-26 13:08:24.875540', 'distro_version': '22.04', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-phlogistonjohn-testing-2024-04-25-1915', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/flavors/default/', 'archs': ['x86_64'], 'distro': 'ubuntu'}] 2024-04-26T16:05:32.817 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-phlogistonjohn-testing-2024-04-25-1915, sha1 a430f9ed84de3008ca5b632b1ed304bac0e753f6 from https://shaman.ceph.com/api/search/?project=ceph&distros=ubuntu%2F22.04%2Fx86_64&flavor=default&sha1=a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:05:32.818 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/x86_64/flavors/default/cephadm 2024-04-26T16:05:32.818 DEBUG:teuthology.orchestra.run.smithi050:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T16:05:33.519 INFO:teuthology.orchestra.run.smithi050.stdout:-rw-rw-r-- 1 ubuntu ubuntu 789040 Apr 26 16:05 /home/ubuntu/cephtest/cephadm 2024-04-26T16:05:33.520 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-phlogistonjohn-testing-2024-04-25-1915/a430f9ed84de3008ca5b632b1ed304bac0e753f6/ubuntu/jammy/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-26T16:05:34.025 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-rw-r-- 1 ubuntu ubuntu 789040 Apr 26 16:05 /home/ubuntu/cephtest/cephadm 2024-04-26T16:05:34.025 DEBUG:teuthology.orchestra.run.smithi050:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T16:05:34.033 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-26T16:05:34.054 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 on all hosts... 2024-04-26T16:05:34.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 pull 2024-04-26T16:05:34.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 pull 2024-04-26T16:05:34.248 INFO:teuthology.orchestra.run.smithi050.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6... 2024-04-26T16:05:34.249 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6... 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout: "ceph_version": "ceph version 19.0.0-3305-ga430f9ed (a430f9ed84de3008ca5b632b1ed304bac0e753f6) squid (dev)", 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout: "image_id": "031ba2747cfdd41e6bd5399209a1ba77775f760ed8b698251b3c2529efe3552f", 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout: "repo_digests": [ 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b71a575c225eecd96353b1d64f11c7118b9683e4e8491c513d9f702bd9a1f45" 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:06:47.869 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 19.0.0-3305-ga430f9ed (a430f9ed84de3008ca5b632b1ed304bac0e753f6) squid (dev)", 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "031ba2747cfdd41e6bd5399209a1ba77775f760ed8b698251b3c2529efe3552f", 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b71a575c225eecd96353b1d64f11c7118b9683e4e8491c513d9f702bd9a1f45" 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:06:51.900 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:06:51.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo mkdir -p /etc/ceph 2024-04-26T16:06:51.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-04-26T16:06:51.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 777 /etc/ceph 2024-04-26T16:06:52.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-04-26T16:06:52.029 INFO:tasks.cephadm:Writing seed config... 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [client] debug objecter = 20 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [client] debug rados = 20 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [global] ms type = async 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [mon] debug mgrc = 20 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-26T16:06:52.031 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-04-26T16:06:52.032 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-04-26T16:06:52.033 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-04-26T16:06:52.033 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-04-26T16:06:52.033 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-04-26T16:06:52.033 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-04-26T16:06:52.033 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-26T16:06:52.033 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-04-26T16:06:52.034 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:06:52.034 DEBUG:teuthology.orchestra.run.smithi050:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-26T16:06:52.065 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = cf170a7a-03e6-11ef-bc93-c7b262605968 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mgrc = 20 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon warn on pool no app = False paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 debug objecter = 20 debug rados = 20 2024-04-26T16:06:52.066 DEBUG:teuthology.orchestra.run.smithi050:mon.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a.service 2024-04-26T16:06:52.110 DEBUG:teuthology.orchestra.run.smithi050:mgr.y> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y.service 2024-04-26T16:06:52.153 INFO:tasks.cephadm:Bootstrapping... 2024-04-26T16:06:52.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 -v bootstrap --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.50 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:06:52.369 INFO:teuthology.orchestra.run.smithi050.stdout:-------------------------------------------------------------------------------- 2024-04-26T16:06:52.369 INFO:teuthology.orchestra.run.smithi050.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6', '-v', 'bootstrap', '--fsid', 'cf170a7a-03e6-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.50', '--skip-admin-label'] 2024-04-26T16:06:52.370 INFO:teuthology.orchestra.run.smithi050.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-26T16:06:52.370 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying podman|docker is present... 2024-04-26T16:06:52.370 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying lvm2 is present... 2024-04-26T16:06:52.370 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying time synchronization is in place... 2024-04-26T16:06:52.377 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T16:06:52.377 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T16:06:52.382 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T16:06:52.382 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.387 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-26T16:06:52.387 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-26T16:06:52.391 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-26T16:06:52.391 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.394 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-26T16:06:52.395 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout masked 2024-04-26T16:06:52.398 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-26T16:06:52.399 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.402 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-26T16:06:52.402 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-26T16:06:52.406 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-26T16:06:52.406 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.410 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout enabled 2024-04-26T16:06:52.414 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout active 2024-04-26T16:06:52.414 INFO:teuthology.orchestra.run.smithi050.stdout:Unit ntp.service is enabled and running 2024-04-26T16:06:52.414 INFO:teuthology.orchestra.run.smithi050.stdout:Repeating the final host check... 2024-04-26T16:06:52.414 INFO:teuthology.orchestra.run.smithi050.stdout:docker (/usr/bin/docker) is present 2024-04-26T16:06:52.414 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl is present 2024-04-26T16:06:52.414 INFO:teuthology.orchestra.run.smithi050.stdout:lvcreate is present 2024-04-26T16:06:52.418 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-26T16:06:52.418 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-26T16:06:52.421 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-26T16:06:52.421 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.425 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled chronyd.service 2024-04-26T16:06:52.425 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for chronyd.service: No such file or directory 2024-04-26T16:06:52.429 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active chronyd.service 2024-04-26T16:06:52.429 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.432 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled systemd-timesyncd.service 2024-04-26T16:06:52.432 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout masked 2024-04-26T16:06:52.436 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active systemd-timesyncd.service 2024-04-26T16:06:52.436 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.439 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl is-enabled ntpd.service 2024-04-26T16:06:52.440 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to get unit file state for ntpd.service: No such file or directory 2024-04-26T16:06:52.443 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 3 from systemctl is-active ntpd.service 2024-04-26T16:06:52.443 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout inactive 2024-04-26T16:06:52.446 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout enabled 2024-04-26T16:06:52.451 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stdout active 2024-04-26T16:06:52.451 INFO:teuthology.orchestra.run.smithi050.stdout:Unit ntp.service is enabled and running 2024-04-26T16:06:52.451 INFO:teuthology.orchestra.run.smithi050.stdout:Host looks OK 2024-04-26T16:06:52.451 INFO:teuthology.orchestra.run.smithi050.stdout:Cluster fsid: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:06:52.451 INFO:teuthology.orchestra.run.smithi050.stdout:Acquiring lock 140444704040720 on /run/cephadm/cf170a7a-03e6-11ef-bc93-c7b262605968.lock 2024-04-26T16:06:52.451 INFO:teuthology.orchestra.run.smithi050.stdout:Lock 140444704040720 acquired on /run/cephadm/cf170a7a-03e6-11ef-bc93-c7b262605968.lock 2024-04-26T16:06:52.452 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying IP 172.21.15.50 port 3300 ... 2024-04-26T16:06:52.452 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying IP 172.21.15.50 port 6789 ... 2024-04-26T16:06:52.452 INFO:teuthology.orchestra.run.smithi050.stdout:Base mon IP(s) is [172.21.15.50:3300, 172.21.15.50:6789], mon addrv is [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2024-04-26T16:06:52.454 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 2024-04-26T16:06:52.454 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2024-04-26T16:06:52.454 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.50 2024-04-26T16:06:52.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-26T16:06:52.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-04-26T16:06:52.457 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-04-26T16:06:52.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-26T16:06:52.460 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1584/64 scope link 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:Mon IP `172.21.15.50` is in CIDR network `172.21.0.0/20` 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:Mon IP `172.21.15.50` is in CIDR network `172.21.0.0/20` 2024-04-26T16:06:52.461 INFO:teuthology.orchestra.run.smithi050.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-26T16:06:52.462 INFO:teuthology.orchestra.run.smithi050.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-26T16:06:52.462 INFO:teuthology.orchestra.run.smithi050.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6... 2024-04-26T16:06:52.737 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout a430f9ed84de3008ca5b632b1ed304bac0e753f6: Pulling from ceph-ci/ceph 2024-04-26T16:06:52.738 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout Digest: sha256:4b71a575c225eecd96353b1d64f11c7118b9683e4e8491c513d9f702bd9a1f45 2024-04-26T16:06:52.738 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout Status: Image is up to date for quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:06:52.738 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/docker: stdout quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:06:54.343 INFO:teuthology.orchestra.run.smithi050.stdout:ceph: stdout ceph version 19.0.0-3305-ga430f9ed (a430f9ed84de3008ca5b632b1ed304bac0e753f6) squid (dev) 2024-04-26T16:06:54.343 INFO:teuthology.orchestra.run.smithi050.stdout:Ceph version: ceph version 19.0.0-3305-ga430f9ed (a430f9ed84de3008ca5b632b1ed304bac0e753f6) squid (dev) 2024-04-26T16:06:54.343 INFO:teuthology.orchestra.run.smithi050.stdout:Extracting ceph user uid/gid from container image... 2024-04-26T16:06:55.677 INFO:teuthology.orchestra.run.smithi050.stdout:stat: stdout 167 167 2024-04-26T16:06:55.677 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial keys... 2024-04-26T16:06:57.007 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAg0StmZlxqGhAAGEmH4HrTpr3e3wcS+Ptttg== 2024-04-26T16:06:58.203 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAh0StmGlEMKRAA4RxTv99GlfC1Irmc6Xg/9Q== 2024-04-26T16:06:59.509 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-authtool: stdout AQAi0StmY4FwOBAA/ueftHZTfE2Q026D2bGU5Q== 2024-04-26T16:06:59.510 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial monmap... 2024-04-26T16:07:00.748 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T16:07:00.748 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-26T16:07:00.748 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout:monmaptool for a [v2:172.21.15.50:3300,v1:172.21.15.50:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout:setting min_mon_release = quincy 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: set fsid to cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:07:00.749 INFO:teuthology.orchestra.run.smithi050.stdout:Creating mon... 2024-04-26T16:07:02.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.427+0000 7fd8e296ec80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-26T16:07:02.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.427+0000 7fd8e296ec80 1 imported monmap: 2024-04-26T16:07:02.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr epoch 0 2024-04-26T16:07:02.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr last_changed 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr min_mon_release 17 (quincy) 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr election_strategy: 1 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.427+0000 7fd8e296ec80 0 /usr/bin/ceph-mon: set fsid to cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: RocksDB version: 7.9.2 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Git sha 0 2024-04-26T16:07:02.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Compile date 2024-04-26 03:22:31 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: DB SUMMARY 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: DB Session ID: 44IKI3TPSYUNQB4ZSKKK 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.error_if_exists: 0 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.create_if_missing: 1 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.paranoid_checks: 1 2024-04-26T16:07:02.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.env: 0x55e2db121cc0 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.info_log: 0x55e2dc4d5020 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.statistics: (nil) 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.use_fsync: 0 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_log_file_size: 0 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T16:07:02.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.allow_fallocate: 1 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.use_direct_reads: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.db_log_dir: 2024-04-26T16:07:02.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.wal_dir: 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.write_buffer_manager: 0x55e2dc4a4aa0 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T16:07:02.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.unordered_write: 0 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T16:07:02.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.row_cache: None 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.wal_filter: None 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.two_write_queues: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.wal_compression: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.atomic_flush: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T16:07:02.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.log_readahead_size: 0 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_background_jobs: 2 2024-04-26T16:07:02.023 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_background_compactions: -1 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_subcompactions: 1 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_open_files: -1 2024-04-26T16:07:02.024 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_background_flushes: -1 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Compression algorithms supported: 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kZSTD supported: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kXpressCompression supported: 0 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kZlibCompression supported: 1 2024-04-26T16:07:02.025 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kSnappyCompression supported: 1 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kLZ4Compression supported: 1 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: kBZip2Compression supported: 0 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:317] Creating manifest 1 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T16:07:02.026 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.merge_operator: 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_filter: None 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e2dc4d51e0) 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks: 1 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T16:07:02.027 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr pin_top_level_index_and_filter: 1 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr index_type: 0 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr data_block_index_type: 0 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr index_shortening: 1 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr data_block_hash_table_util_ratio: 0.750000 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr checksum: 4 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr no_block_cache: 0 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache: 0x55e2dc4d71f0 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache_name: BinnedLRUCache 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache_options: 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr capacity : 536870912 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr num_shard_bits : 4 2024-04-26T16:07:02.028 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr strict_capacity_limit : 0 2024-04-26T16:07:02.029 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr high_pri_pool_ratio: 0.000 2024-04-26T16:07:02.029 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_cache_compressed: (nil) 2024-04-26T16:07:02.029 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr persistent_cache: (nil) 2024-04-26T16:07:02.029 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_size: 4096 2024-04-26T16:07:02.029 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_size_deviation: 10 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_restart_interval: 16 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr index_block_restart_interval: 1 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr metadata_block_size: 4096 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr partition_filters: 0 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr use_delta_encoding: 1 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr filter_policy: bloomfilter 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr whole_key_filtering: 1 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr verify_compression: 0 2024-04-26T16:07:02.030 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr read_amp_bytes_per_bit: 0 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr format_version: 5 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr enable_index_compression: 1 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr block_align: 0 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr max_auto_readahead_size: 262144 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr prepopulate_block_cache: 0 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr initial_auto_readahead_size: 8192 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr num_file_reads_for_auto_readahead: 2 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression: NoCompression 2024-04-26T16:07:02.031 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.num_levels: 7 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T16:07:02.032 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-26T16:07:02.033 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T16:07:02.034 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T16:07:02.035 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T16:07:02.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.inplace_update_support: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.bloom_locality: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.max_successive_merges: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-26T16:07:02.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.ttl: 2592000 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.enable_blob_files: false 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.min_blob_size: 0 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T16:07:02.038 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c4a4517-98e8-4696-a7b0-766a03b05d07 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 5 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e2dc5ce000 2024-04-26T16:07:02.039 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8e296ec80 4 rocksdb: DB pointer 0x55e2dc5be000 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8d2ace700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.431+0000 7fd8d2ace700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** DB Stats ** 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.040 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** Compaction Stats [default] ** 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T16:07:02.041 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(Total Files): cumulative 0, interval 0 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr AddFile(Keys): cumulative 0, interval 0 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Block cache BinnedLRUCache@0x55e2dc4d71f0#7 capacity: 512.00 MB usage: 0.00 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr Block cache entry stats(count,size,portion): Misc(1,0.00 KB,0%) 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr ** File Read Latency Histogram By Level [default] ** 2024-04-26T16:07:02.042 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr 2024-04-26T16:07:02.043 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.435+0000 7fd8e296ec80 4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work 2024-04-26T16:07:02.043 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.435+0000 7fd8e296ec80 4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete 2024-04-26T16:07:02.043 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph-mon: stderr debug 2024-04-26T16:07:01.435+0000 7fd8e296ec80 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2024-04-26T16:07:02.043 INFO:teuthology.orchestra.run.smithi050.stdout:create mon.a on 2024-04-26T16:07:02.250 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-04-26T16:07:02.472 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2024-04-26T16:07:02.697 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968.target -> /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968.target. 2024-04-26T16:07:02.697 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968.target -> /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968.target. 2024-04-26T16:07:03.007 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a 2024-04-26T16:07:03.007 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-26T16:07:03.201 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968.target.wants/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a.service -> /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service. 2024-04-26T16:07:03.211 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-26T16:07:03.211 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T16:07:03.212 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mon to start... 2024-04-26T16:07:03.212 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mon... 2024-04-26T16:07:04.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:04 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.490352+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout cluster: 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout id: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout services: 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.203977s) 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-26T16:07:05.329 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout data: 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout pgs: 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03f5cbb700 1 Processor -- start 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03f5cbb700 1 -- start start 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03f5cbb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f0107d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03f5cbb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f03f01082d0 con 0x7f03f01078e0 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03ef7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f0107d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:05.330 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03ef7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f0107d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53898/0 (socket says 172.21.15.50:53898) 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03ef7fe700 1 -- 172.21.15.50:0/2485703566 learned_addr learned my addr 172.21.15.50:0/2485703566 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03ef7fe700 1 -- 172.21.15.50:0/2485703566 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f0108ae0 con 0x7f03f01078e0 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03ef7fe700 1 --2- 172.21.15.50:0/2485703566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f0107d00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f03e00098a0 tx=0x7f03e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250c096fde534ad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03eeffd700 1 -- 172.21.15.50:0/2485703566 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03e0004160 con 0x7f03f01078e0 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03eeffd700 1 -- 172.21.15.50:0/2485703566 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e0004300 con 0x7f03f01078e0 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.550+0000 7f03eeffd700 1 -- 172.21.15.50:0/2485703566 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03e00036a0 con 0x7f03f01078e0 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/2485703566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 msgr2=0x7f03f0107d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 --2- 172.21.15.50:0/2485703566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f0107d00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f03e00098a0 tx=0x7f03e0009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/2485703566 shutdown_connections 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 --2- 172.21.15.50:0/2485703566 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f0107d00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/2485703566 >> 172.21.15.50:0/2485703566 conn(0x7f03f0102e80 msgr2=0x7f03f01052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/2485703566 shutdown_connections 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/2485703566 wait complete. 2024-04-26T16:07:05.331 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 librados: starting msgr at 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 librados: starting objecter 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 Processor -- start 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- start start 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 librados: setting wanted keys 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 librados: calling monclient init 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f019d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f03f0109010 con 0x7f03f01078e0 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ef7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f019d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ef7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f019d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53900/0 (socket says 172.21.15.50:53900) 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ef7fe700 1 -- 172.21.15.50:0/911942508 learned_addr learned my addr 172.21.15.50:0/911942508 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ef7fe700 1 -- 172.21.15.50:0/911942508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f019d700 con 0x7f03f01078e0 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ef7fe700 1 --2- 172.21.15.50:0/911942508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f019d1c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f03e0000c00 tx=0x7f03e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:05.332 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 10 client.?.objecter ms_handle_connect 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03e0003db0 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03e0011630 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f019d990 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03e0011950 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 10 client.4098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f01a0680 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 librados: init done 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f03e001a460 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f03e001b780 con 0x7f03f01078e0 2024-04-26T16:07:05.333 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 10 client.4098.objecter ms_dispatch 0x7f03f0001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 3 client.4098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 3 client.4098.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03ed7fa700 20 client.4098.objecter dump_active .. 0 homeless 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.554+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03f0064a80 con 0x7f03f01078e0 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.558+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f03e001b980 con 0x7f03f01078e0 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f03f01a09e0 con 0x7f03f01078e0 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03ed7fa700 1 -- 172.21.15.50:0/911942508 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f03e001bb60 con 0x7f03f01078e0 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 10 librados: watch_flush enter 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 10 librados: watch_flush exit 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 20 client.4098.objecter shutdown clearing up homeless session... 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 10 client.4098.objecter successfully canceled tick 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 msgr2=0x7f03f019d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 --2- 172.21.15.50:0/911942508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f019d1c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f03e0000c00 tx=0x7f03e0004750 comp rx=0 tx=0).stop 2024-04-26T16:07:05.334 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 shutdown_connections 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 --2- 172.21.15.50:0/911942508 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03f01078e0 0x7f03f019d1c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 >> 172.21.15.50:0/911942508 conn(0x7f03f0102e80 msgr2=0x7f03f007cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 shutdown_connections 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.694+0000 7f03f5cbb700 1 -- 172.21.15.50:0/911942508 wait complete. 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:04.698+0000 7f03f5cbb700 1 librados: shutdown 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:mon is available 2024-04-26T16:07:05.335 INFO:teuthology.orchestra.run.smithi050.stdout:Assimilating anything we can from ceph.conf... 2024-04-26T16:07:05.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493470+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493470+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493609+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493609+0000 mon.a (mon.0) 3 : cluster [DBG] monmap epoch 1 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493627+0000 mon.a (mon.0) 4 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493627+0000 mon.a (mon.0) 4 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493637+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493637+0000 mon.a (mon.0) 5 : cluster [DBG] last_changed 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493646+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493646+0000 mon.a (mon.0) 6 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:05.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493654+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493654+0000 mon.a (mon.0) 7 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493661+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493661+0000 mon.a (mon.0) 8 : cluster [DBG] election_strategy: 1 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493669+0000 mon.a (mon.0) 9 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.493669+0000 mon.a (mon.0) 9 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.495625+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.495625+0000 mon.a (mon.0) 10 : cluster [DBG] fsmap 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.497846+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.497846+0000 mon.a (mon.0) 11 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.498335+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: cluster 2024-04-26T16:07:04.498335+0000 mon.a (mon.0) 12 : cluster [DBG] mgrmap e1: no daemons active 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: audit 2024-04-26T16:07:04.697407+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.50:0/911942508' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T16:07:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:05 smithi050 bash[20978]: audit 2024-04-26T16:07:04.697407+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.50:0/911942508' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-26T16:07:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:06 smithi050 bash[20978]: audit 2024-04-26T16:07:06.353320+0000 mon.a (mon.0) 14 : audit [INF] from='client.? 172.21.15.50:0/795866702' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T16:07:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:06 smithi050 bash[20978]: audit 2024-04-26T16:07:06.353320+0000 mon.a (mon.0) 14 : audit [INF] from='client.? 172.21.15.50:0/795866702' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T16:07:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:06 smithi050 bash[20978]: audit 2024-04-26T16:07:06.356239+0000 mon.a (mon.0) 15 : audit [INF] from='client.? 172.21.15.50:0/795866702' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T16:07:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:06 smithi050 bash[20978]: audit 2024-04-26T16:07:06.356239+0000 mon.a (mon.0) 15 : audit [INF] from='client.? 172.21.15.50:0/795866702' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout fsid = cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.50:3300,v1:172.21.15.50:6789] 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T16:07:07.036 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T16:07:07.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:07.037 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T16:07:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T16:07:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T16:07:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T16:07:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T16:07:07.044 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 Processor -- start 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- start start 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f8240109ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f824010a4b0 con 0x7f8240109ac0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f8240109ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f8240109ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53914/0 (socket says 172.21.15.50:53914) 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 -- 172.21.15.50:0/1087361477 learned_addr learned my addr 172.21.15.50:0/1087361477 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 -- 172.21.15.50:0/1087361477 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824010acc0 con 0x7f8240109ac0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 --2- 172.21.15.50:0/1087361477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f8240109ee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8238009e80 tx=0x7f823800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6623f2dbebe39b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823dffb700 1 -- 172.21.15.50:0/1087361477 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8238017070 con 0x7f8240109ac0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823dffb700 1 -- 172.21.15.50:0/1087361477 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8238003a20 con 0x7f8240109ac0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823dffb700 1 -- 172.21.15.50:0/1087361477 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8238003d20 con 0x7f8240109ac0 2024-04-26T16:07:07.045 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- 172.21.15.50:0/1087361477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 msgr2=0x7f8240109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 --2- 172.21.15.50:0/1087361477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f8240109ee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8238009e80 tx=0x7f823800b330 comp rx=0 tx=0).stop 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- 172.21.15.50:0/1087361477 shutdown_connections 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 --2- 172.21.15.50:0/1087361477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f8240109ee0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- 172.21.15.50:0/1087361477 >> 172.21.15.50:0/1087361477 conn(0x7f824007d020 msgr2=0x7f824007d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- 172.21.15.50:0/1087361477 shutdown_connections 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- 172.21.15.50:0/1087361477 wait complete. 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 librados: starting msgr at 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 librados: starting objecter 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 Processor -- start 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- start start 2024-04-26T16:07:07.046 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 librados: setting wanted keys 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 librados: calling monclient init 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f82400813f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f8245324700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f824010b1f0 con 0x7f8240109ac0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f82400813f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f82400813f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53920/0 (socket says 172.21.15.50:53920) 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.210+0000 7f823effd700 1 -- 172.21.15.50:0/795866702 learned_addr learned my addr 172.21.15.50:0/795866702 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f823effd700 1 -- 172.21.15.50:0/795866702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8240081930 con 0x7f8240109ac0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f823effd700 1 --2- 172.21.15.50:0/795866702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f82400813f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8238004000 tx=0x7f8238004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 10 client.?.objecter ms_handle_connect 0x7f8240109ac0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8238022020 con 0x7f8240109ac0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8238018040 con 0x7f8240109ac0 2024-04-26T16:07:07.047 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8238017070 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8240081bc0 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8245324700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f824007e180 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8245324700 1 librados: init done 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8238025020 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f823800f050 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 10 client.4100.objecter ms_dispatch 0x7f8240001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8237fff700 20 client.4100.objecter dump_active .. 0 homeless 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.214+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8240064a80 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.218+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f823802c030 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.350+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f8240196710 con 0x7f8240109ac0 2024-04-26T16:07:07.048 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+374 (secure 0 0 0) 0x7f823801e480 con 0x7f8240109ac0 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8237fff700 1 -- 172.21.15.50:0/795866702 <== mon.0 v2:172.21.15.50:3300/0 8 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f8238035070 con 0x7f8240109ac0 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 10 librados: watch_flush enter 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 10 librados: watch_flush exit 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 20 client.4100.objecter shutdown clearing up homeless session... 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 10 client.4100.objecter successfully canceled tick 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 msgr2=0x7f82400813f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 --2- 172.21.15.50:0/795866702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f82400813f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8238004000 tx=0x7f8238004ad0 comp rx=0 tx=0).stop 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 shutdown_connections 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 --2- 172.21.15.50:0/795866702 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8240109ac0 0x7f82400813f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 >> 172.21.15.50:0/795866702 conn(0x7f824007d020 msgr2=0x7f8240193e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 shutdown_connections 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 -- 172.21.15.50:0/795866702 wait complete. 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:06.354+0000 7f8245324700 1 librados: shutdown 2024-04-26T16:07:07.049 INFO:teuthology.orchestra.run.smithi050.stdout:Generating new minimal ceph.conf... 2024-04-26T16:07:08.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:08 smithi050 bash[20978]: audit 2024-04-26T16:07:08.045112+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? 172.21.15.50:0/2790265672' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:07:08.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:08 smithi050 bash[20978]: audit 2024-04-26T16:07:08.045112+0000 mon.a (mon.0) 16 : audit [DBG] from='client.? 172.21.15.50:0/2790265672' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.898+0000 7f8e76b91700 1 Processor -- start 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.898+0000 7f8e76b91700 1 -- start start 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.898+0000 7f8e76b91700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7007bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.898+0000 7f8e76b91700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7007c180 con 0x7f8e7007d750 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7007bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7007bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53930/0 (socket says 172.21.15.50:53930) 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 -- 172.21.15.50:0/732690506 learned_addr learned my addr 172.21.15.50:0/732690506 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:08.621 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 -- 172.21.15.50:0/732690506 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7007c300 con 0x7f8e7007d750 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 --2- 172.21.15.50:0/732690506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7007bbb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009e80 tx=0x7f8e6800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce158c7e993ca27 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/732690506 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e68017070 con 0x7f8e7007d750 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/732690506 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f8e6800b8c0 con 0x7f8e7007d750 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6f7fe700 1 -- 172.21.15.50:0/732690506 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e68004340 con 0x7f8e7007d750 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/732690506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 msgr2=0x7f8e7007bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 --2- 172.21.15.50:0/732690506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7007bbb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009e80 tx=0x7f8e6800b330 comp rx=0 tx=0).stop 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/732690506 shutdown_connections 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 --2- 172.21.15.50:0/732690506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7007bbb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/732690506 >> 172.21.15.50:0/732690506 conn(0x7f8e70102e80 msgr2=0x7f8e701052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/732690506 shutdown_connections 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/732690506 wait complete. 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 librados: starting msgr at 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 librados: starting objecter 2024-04-26T16:07:08.622 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 Processor -- start 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- start start 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 librados: setting wanted keys 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 librados: calling monclient init 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7010dd80 con 0x7f8e7007d750 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53940/0 (socket says 172.21.15.50:53940) 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 -- 172.21.15.50:0/2790265672 learned_addr learned my addr 172.21.15.50:0/2790265672 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 -- 172.21.15.50:0/2790265672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7019de50 con 0x7f8e7007d750 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e7492d700 1 --2- 172.21.15.50:0/2790265672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7019d910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e68004000 tx=0x7f8e68004900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6dffb700 10 client.?.objecter ms_handle_connect 0x7f8e7007d750 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6dffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:08.623 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e68020020 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f8e68003c50 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7019e0e0 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e68017040 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 10 client.4102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e701a0dd0 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.902+0000 7f8e76b91700 1 librados: init done 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e68023020 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e6800f050 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 10 client.4102.objecter ms_dispatch 0x7f8e70001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 3 client.4102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 3 client.4102.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 20 client.4102.objecter dump_active .. 0 homeless 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e70064a80 con 0x7f8e7007d750 2024-04-26T16:07:08.624 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:07.906+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8e6802a020 con 0x7f8e7007d750 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8e701a1170 con 0x7f8e7007d750 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e6dffb700 1 -- 172.21.15.50:0/2790265672 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f8e6800b860 con 0x7f8e7007d750 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 10 librados: watch_flush enter 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 10 librados: watch_flush exit 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 20 client.4102.objecter shutdown clearing up homeless session... 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 10 client.4102.objecter successfully canceled tick 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 msgr2=0x7f8e7019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.042+0000 7f8e76b91700 1 --2- 172.21.15.50:0/2790265672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7019d910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e68004000 tx=0x7f8e68004900 comp rx=0 tx=0).stop 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.046+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 shutdown_connections 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.046+0000 7f8e76b91700 1 --2- 172.21.15.50:0/2790265672 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e7007d750 0x7f8e7019d910 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.046+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 >> 172.21.15.50:0/2790265672 conn(0x7f8e70102e80 msgr2=0x7f8e70103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.046+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 shutdown_connections 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.046+0000 7f8e76b91700 1 -- 172.21.15.50:0/2790265672 wait complete. 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:08.046+0000 7f8e76b91700 1 librados: shutdown 2024-04-26T16:07:08.625 INFO:teuthology.orchestra.run.smithi050.stdout:Restarting the monitor... 2024-04-26T16:07:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:08 smithi050 systemd[1]: Stopping Ceph mon.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:07:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:08 smithi050 bash[20978]: debug 2024-04-26T16:07:08.754+0000 7f91b5728700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-26T16:07:08.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:08 smithi050 bash[20978]: debug 2024-04-26T16:07:08.754+0000 7f91b5728700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-26T16:07:09.558 INFO:teuthology.orchestra.run.smithi050.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-26T16:07:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:09 smithi050 bash[21365]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-mon-a 2024-04-26T16:07:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:09 smithi050 bash[21424]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-mon-a 2024-04-26T16:07:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:09 smithi050 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-26T16:07:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:09 smithi050 systemd[1]: Stopped Ceph mon.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:07:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:09 smithi050 systemd[1]: Started Ceph mon.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.690+0000 7f60d787fc80 0 set uid:gid to 167:167 (ceph:ceph) 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.690+0000 7f60d787fc80 0 ceph version 19.0.0-3305-ga430f9ed (a430f9ed84de3008ca5b632b1ed304bac0e753f6) squid (dev), process ceph-mon, pid 7 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.690+0000 7f60d787fc80 0 pidfile_write: ignore empty --pid-file 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.694+0000 7f60d787fc80 0 load: jerasure load: lrc 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: RocksDB version: 7.9.2 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Git sha 0 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Compile date 2024-04-26 03:22:31 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: DB SUMMARY 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: DB Session ID: 4UB74BXMTTKOBV52R8TF 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: CURRENT file: CURRENT 2024-04-26T16:07:10.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: IDENTITY file: IDENTITY 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 96479 ; 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.error_if_exists: 0 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.create_if_missing: 0 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.paranoid_checks: 1 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.env: 0x55dbe1b0fcc0 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.fs: PosixFileSystem 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.info_log: 0x55dbe3dc5dc0 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_file_opening_threads: 16 2024-04-26T16:07:10.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.statistics: (nil) 2024-04-26T16:07:10.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.use_fsync: 0 2024-04-26T16:07:10.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_log_file_size: 0 2024-04-26T16:07:10.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-26T16:07:10.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.log_file_time_to_roll: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.keep_log_file_num: 1000 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.recycle_log_file_num: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.allow_fallocate: 1 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.allow_mmap_reads: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.allow_mmap_writes: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.use_direct_reads: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.create_missing_column_families: 0 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.db_log_dir: 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.wal_dir: 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.table_cache_numshardbits: 6 2024-04-26T16:07:10.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.advise_random_on_open: 1 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.db_write_buffer_size: 0 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.write_buffer_manager: 0x55dbe3d970e0 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.use_adaptive_mutex: 0 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.rate_limiter: (nil) 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.wal_recovery_mode: 2 2024-04-26T16:07:10.887 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.enable_thread_tracking: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.enable_pipelined_write: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.unordered_write: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.row_cache: None 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.wal_filter: None 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.allow_ingest_behind: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.two_write_queues: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.manual_wal_flush: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.wal_compression: 0 2024-04-26T16:07:10.888 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.atomic_flush: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.persist_stats_to_disk: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.log_readahead_size: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.best_efforts_recovery: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.allow_data_in_errors: 0 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.db_host_id: __hostname__ 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.enforce_single_del_contracts: true 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_background_jobs: 2 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_background_compactions: -1 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_subcompactions: 1 2024-04-26T16:07:10.889 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.delayed_write_rate : 16777216 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_total_wal_size: 0 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.stats_dump_period_sec: 600 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.stats_persist_period_sec: 600 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_open_files: -1 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bytes_per_sync: 0 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_readahead_size: 0 2024-04-26T16:07:10.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_background_flushes: -1 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Compression algorithms supported: 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kZSTD supported: 0 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kXpressCompression supported: 0 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kLZ4HCCompression supported: 1 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kZlibCompression supported: 1 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kSnappyCompression supported: 1 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kLZ4Compression supported: 1 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: kBZip2Compression supported: 0 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2024-04-26T16:07:10.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-26T16:07:10.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.merge_operator: 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_filter: None 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_filter_factory: None 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.sst_partitioner_factory: None 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.table_factory: BlockBasedTable 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dbe3dc5f80) 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cache_index_and_filter_blocks: 1 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: pin_top_level_index_and_filter: 1 2024-04-26T16:07:10.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: index_type: 0 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: data_block_index_type: 0 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: index_shortening: 1 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: data_block_hash_table_util_ratio: 0.750000 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: checksum: 4 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: no_block_cache: 0 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_cache: 0x55dbe3dc71f0 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_cache_name: BinnedLRUCache 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_cache_options: 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: capacity : 536870912 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: num_shard_bits : 4 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: strict_capacity_limit : 0 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: high_pri_pool_ratio: 0.000 2024-04-26T16:07:10.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_cache_compressed: (nil) 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: persistent_cache: (nil) 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_size: 4096 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_size_deviation: 10 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_restart_interval: 16 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: index_block_restart_interval: 1 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: metadata_block_size: 4096 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: partition_filters: 0 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: use_delta_encoding: 1 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: filter_policy: bloomfilter 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: whole_key_filtering: 1 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: verify_compression: 0 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: read_amp_bytes_per_bit: 0 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: format_version: 5 2024-04-26T16:07:10.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: enable_index_compression: 1 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: block_align: 0 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: max_auto_readahead_size: 262144 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: prepopulate_block_cache: 0 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: initial_auto_readahead_size: 8192 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: num_file_reads_for_auto_readahead: 2 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.write_buffer_size: 33554432 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_write_buffer_number: 2 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression: NoCompression 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression: Disabled 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.prefix_extractor: nullptr 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.num_levels: 7 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-26T16:07:10.896 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.window_bits: -14 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.level: 32767 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.strategy: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-26T16:07:10.897 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.enabled: false 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.target_file_size_base: 67108864 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.target_file_size_multiplier: 1 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-26T16:07:10.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.arena_block_size: 1048576 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.disable_auto_compactions: 0 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-26T16:07:10.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.inplace_update_support: 0 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-26T16:07:10.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.memtable_huge_page_size: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.bloom_locality: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.max_successive_merges: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.paranoid_file_checks: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.force_consistency_checks: 1 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.report_bg_io_stats: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.ttl: 2592000 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.enable_blob_files: false 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.min_blob_size: 0 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.blob_file_size: 268435456 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.blob_compression_type: NoCompression 2024-04-26T16:07:10.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.blob_file_starting_level: 0 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4c4a4517-98e8-4696-a7b0-766a03b05d07 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714147630700930, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714147630702144, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92337, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 290, "table_properties": {"data_size": 90378, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12069, "raw_average_key_size": 48, "raw_value_size": 83742, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 249, "num_filter_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714147630, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4c4a4517-98e8-4696-a7b0-766a03b05d07", "db_session_id": "4UB74BXMTTKOBV52R8TF", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714147630702265, "job": 1, "event": "recovery_finished"} 2024-04-26T16:07:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.698+0000 7f60d787fc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dbe3ee8000 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 4 rocksdb: DB pointer 0x55dbe3ed4000 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60c5fd7700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60c5fd7700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: ** DB Stats ** 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: ** Compaction Stats [default] ** 2024-04-26T16:07:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: L0 2/0 92.03 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Sum 2/0 92.03 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 120.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: ** Compaction Stats [default] ** 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 120.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Uptime(secs): 0.0 total, 0.0 interval 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Flush(GB): cumulative 0.000, interval 0.000 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: AddFile(Total Files): cumulative 0, interval 0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: AddFile(L0 Files): cumulative 0, interval 0 2024-04-26T16:07:10.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: AddFile(Keys): cumulative 0, interval 0 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Cumulative compaction: 0.00 GB write, 22.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Interval compaction: 0.00 GB write, 22.08 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Block cache BinnedLRUCache@0x55dbe3dc71f0#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: ** File Read Latency Histogram By Level [default] ** 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] at bind addrs [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 1 mon.a@-1(???) e1 preinit fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 mon.a@-1(???).mds e1 new map 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 mon.a@-1(???).mds e1 print_map 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: e1 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T16:07:10.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: legacy client fscid: -1 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: No filesystems configured 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: debug 2024-04-26T16:07:10.702+0000 7f60d787fc80 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727558+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727558+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727626+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727626+0000 mon.a (mon.0) 2 : cluster [DBG] monmap epoch 1 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727645+0000 mon.a (mon.0) 3 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727645+0000 mon.a (mon.0) 3 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:10.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727669+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727669+0000 mon.a (mon.0) 4 : cluster [DBG] last_changed 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727684+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727684+0000 mon.a (mon.0) 5 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727693+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727693+0000 mon.a (mon.0) 6 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727699+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727699+0000 mon.a (mon.0) 7 : cluster [DBG] election_strategy: 1 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727720+0000 mon.a (mon.0) 8 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.727720+0000 mon.a (mon.0) 8 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.728130+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.728130+0000 mon.a (mon.0) 9 : cluster [DBG] fsmap 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.728158+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.728158+0000 mon.a (mon.0) 10 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.728531+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-04-26T16:07:10.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:10 smithi050 bash[21463]: cluster 2024-04-26T16:07:10.728531+0000 mon.a (mon.0) 11 : cluster [DBG] mgrmap e1: no daemons active 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 Processor -- start 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- start start 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c0109ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f82c010a4b0 con 0x7f82c0109ac0 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82beffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c0109ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82beffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c0109ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56224/0 (socket says 172.21.15.50:56224) 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82beffd700 1 -- 172.21.15.50:0/1677267458 learned_addr learned my addr 172.21.15.50:0/1677267458 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:11.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82beffd700 1 -- 172.21.15.50:0/1677267458 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c010acc0 con 0x7f82c0109ac0 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82beffd700 1 --2- 172.21.15.50:0/1677267458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c0109ee0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009e80 tx=0x7f82b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acddcb0b770cbb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82bdffb700 1 -- 172.21.15.50:0/1677267458 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b0017070 con 0x7f82c0109ac0 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82bdffb700 1 -- 172.21.15.50:0/1677267458 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f82b000b8c0 con 0x7f82c0109ac0 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82bdffb700 1 -- 172.21.15.50:0/1677267458 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b0004340 con 0x7f82c0109ac0 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- 172.21.15.50:0/1677267458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 msgr2=0x7f82c0109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 --2- 172.21.15.50:0/1677267458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c0109ee0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f82b0009e80 tx=0x7f82b000b330 comp rx=0 tx=0).stop 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- 172.21.15.50:0/1677267458 shutdown_connections 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 --2- 172.21.15.50:0/1677267458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c0109ee0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- 172.21.15.50:0/1677267458 >> 172.21.15.50:0/1677267458 conn(0x7f82c007d020 msgr2=0x7f82c007d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- 172.21.15.50:0/1677267458 shutdown_connections 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.790+0000 7f82c5946700 1 -- 172.21.15.50:0/1677267458 wait complete. 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 librados: starting msgr at 2024-04-26T16:07:11.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 librados: starting objecter 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 Processor -- start 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 -- start start 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 librados: setting wanted keys 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 librados: calling monclient init 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f82c010def0 con 0x7f82c0109ac0 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82beffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82beffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56238/0 (socket says 172.21.15.50:56238) 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82beffd700 1 -- 172.21.15.50:0/3597581708 learned_addr learned my addr 172.21.15.50:0/3597581708 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82beffd700 1 -- 172.21.15.50:0/3597581708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c019de70 con 0x7f82c0109ac0 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82beffd700 1 --2- 172.21.15.50:0/3597581708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c019d930 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f82b0004000 tx=0x7f82b0003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 10 client.?.objecter ms_handle_connect 0x7f82c0109ac0 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:11.598 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b0020020 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f82b0003cf0 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c019e100 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 10 client.14098.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82b0017040 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c01a0df0 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 librados: init done 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f82b0023020 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f82b000f050 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 10 client.14098.objecter ms_dispatch 0x7f82c0001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 3 client.14098.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 3 client.14098.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c4944700 20 client.14098.objecter dump_active .. 0 homeless 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.794+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c0064a80 con 0x7f82c0109ac0 2024-04-26T16:07:11.599 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.798+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f82b002a030 con 0x7f82c0109ac0 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.926+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f82c01a1200 con 0x7f82c0109ac0 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.926+0000 7f82c4944700 1 -- 172.21.15.50:0/3597581708 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f82b000b860 con 0x7f82c0109ac0 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 10 librados: watch_flush enter 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 10 librados: watch_flush exit 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 20 client.14098.objecter shutdown clearing up homeless session... 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 10 client.14098.objecter successfully canceled tick 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 msgr2=0x7f82c019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 --2- 172.21.15.50:0/3597581708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c019d930 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f82b0004000 tx=0x7f82b0003a20 comp rx=0 tx=0).stop 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 shutdown_connections 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 --2- 172.21.15.50:0/3597581708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f82c0109ac0 0x7f82c019d930 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 >> 172.21.15.50:0/3597581708 conn(0x7f82c007d020 msgr2=0x7f82c0106cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 shutdown_connections 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 -- 172.21.15.50:0/3597581708 wait complete. 2024-04-26T16:07:11.600 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:10.930+0000 7f82c5946700 1 librados: shutdown 2024-04-26T16:07:11.601 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-26T16:07:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:07:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Creating mgr... 2024-04-26T16:07:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-26T16:07:11.617 INFO:teuthology.orchestra.run.smithi050.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-26T16:07:11.947 INFO:teuthology.orchestra.run.smithi050.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y 2024-04-26T16:07:11.947 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Failed to reset failed state of unit ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-26T16:07:11.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:11 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:07:11.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:11 smithi050 bash[21463]: audit 2024-04-26T16:07:10.930439+0000 mon.a (mon.0) 12 : audit [INF] from='client.? 172.21.15.50:0/3597581708' entity='client.admin' 2024-04-26T16:07:11.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:11 smithi050 bash[21463]: audit 2024-04-26T16:07:10.930439+0000 mon.a (mon.0) 12 : audit [INF] from='client.? 172.21.15.50:0/3597581708' entity='client.admin' 2024-04-26T16:07:11.952 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:11 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:07:12.147 INFO:teuthology.orchestra.run.smithi050.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968.target.wants/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y.service -> /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service. 2024-04-26T16:07:12.154 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-26T16:07:12.155 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to enable service . firewalld.service is not available 2024-04-26T16:07:12.155 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-26T16:07:12.155 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-26T16:07:12.155 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr to start... 2024-04-26T16:07:12.155 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr... 2024-04-26T16:07:12.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:12 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:07:12.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:12 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:07:12.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:12 smithi050 systemd[1]: Started Ceph mgr.y for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:07:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:13 smithi050 bash[21463]: audit 2024-04-26T16:07:13.660064+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.50:0/4047101401' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:13 smithi050 bash[21463]: audit 2024-04-26T16:07:13.660064+0000 mon.a (mon.0) 13 : audit [DBG] from='client.? 172.21.15.50:0/4047101401' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:13.882 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:13 smithi050 bash[21717]: debug 2024-04-26T16:07:13.574+0000 7fea0d9e7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T16:07:14.277 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:14.277 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:07:14.277 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:07:14.277 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T16:07:14.277 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T16:07:14.277 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T16:07:14.278 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T16:07:14.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T16:07:14.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:14.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T16:07:14.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T16:07:14.279 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T16:07:14.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T16:07:14.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T16:07:14.288 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T16:07:14.289 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:14.290 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T16:07:04.494728+0000", 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa99521e700 1 Processor -- start 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa99521e700 1 -- start start 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa99521e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa9901065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa99521e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa99007b680 con 0x7fa9901061a0 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa9901065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa9901065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56276/0 (socket says 172.21.15.50:56276) 2024-04-26T16:07:14.291 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98ed9d700 1 -- 172.21.15.50:0/1939220899 learned_addr learned my addr 172.21.15.50:0/1939220899 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98ed9d700 1 -- 172.21.15.50:0/1939220899 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa990106b00 con 0x7fa9901061a0 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98ed9d700 1 --2- 172.21.15.50:0/1939220899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa9901065c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa980009e80 tx=0x7fa98000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e22a1c5cee7dc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98e59c700 1 -- 172.21.15.50:0/1939220899 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa980017070 con 0x7fa9901061a0 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98e59c700 1 -- 172.21.15.50:0/1939220899 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fa98000b8c0 con 0x7fa9901061a0 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa98e59c700 1 -- 172.21.15.50:0/1939220899 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa980004340 con 0x7fa9901061a0 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa99521e700 1 -- 172.21.15.50:0/1939220899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 msgr2=0x7fa9901065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.514+0000 7fa99521e700 1 --2- 172.21.15.50:0/1939220899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa9901065c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa980009e80 tx=0x7fa98000b330 comp rx=0 tx=0).stop 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/1939220899 shutdown_connections 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 --2- 172.21.15.50:0/1939220899 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa9901065c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/1939220899 >> 172.21.15.50:0/1939220899 conn(0x7fa990101d00 msgr2=0x7fa990104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/1939220899 shutdown_connections 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/1939220899 wait complete. 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 librados: starting msgr at 2024-04-26T16:07:14.292 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 librados: starting objecter 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 Processor -- start 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- start start 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 librados: setting wanted keys 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 librados: calling monclient init 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa99019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa990107a00 con 0x7fa9901061a0 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa99019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa99019d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56280/0 (socket says 172.21.15.50:56280) 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98ed9d700 1 -- 172.21.15.50:0/4047101401 learned_addr learned my addr 172.21.15.50:0/4047101401 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98ed9d700 1 -- 172.21.15.50:0/4047101401 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99019dde0 con 0x7fa9901061a0 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98ed9d700 1 --2- 172.21.15.50:0/4047101401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa99019d8a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa980004000 tx=0x7fa980003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 10 client.?.objecter ms_handle_connect 0x7fa9901061a0 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:14.293 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa980020020 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fa980003cf0 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99019e070 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 10 client.14102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9901a0d60 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 librados: init done 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa980017040 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa980023020 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa98000f050 con 0x7fa9901061a0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 10 client.14102.objecter ms_dispatch 0x7fa990001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 3 client.14102.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 3 client.14102.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa98cd99700 20 client.14102.objecter dump_active .. 0 homeless 2024-04-26T16:07:14.294 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.518+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa970005150 con 0x7fa9901061a0 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.522+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa98002a020 con 0x7fa9901061a0 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa970005710 con 0x7fa9901061a0 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa98cd99700 1 -- 172.21.15.50:0/4047101401 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa98000b860 con 0x7fa9901061a0 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 10 librados: watch_flush enter 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 10 librados: watch_flush exit 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 20 client.14102.objecter shutdown clearing up homeless session... 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 10 client.14102.objecter successfully canceled tick 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 msgr2=0x7fa99019d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 --2- 172.21.15.50:0/4047101401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa99019d8a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa980004000 tx=0x7fa980003a20 comp rx=0 tx=0).stop 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 shutdown_connections 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 --2- 172.21.15.50:0/4047101401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9901061a0 0x7fa99019d8a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 >> 172.21.15.50:0/4047101401 conn(0x7fa990101d00 msgr2=0x7fa990194560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 shutdown_connections 2024-04-26T16:07:14.295 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 -- 172.21.15.50:0/4047101401 wait complete. 2024-04-26T16:07:14.296 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:13.658+0000 7fa99521e700 1 librados: shutdown 2024-04-26T16:07:14.296 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (1/15)... 2024-04-26T16:07:15.563 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:15 smithi050 bash[21717]: debug 2024-04-26T16:07:15.206+0000 7fea0d9e7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T16:07:15.563 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:15 smithi050 bash[21717]: debug 2024-04-26T16:07:15.306+0000 7fea0d9e7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T16:07:15.563 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:15 smithi050 bash[21717]: debug 2024-04-26T16:07:15.562+0000 7fea0d9e7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T16:07:16.614 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:16 smithi050 bash[21717]: debug 2024-04-26T16:07:16.394+0000 7fea0d9e7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T16:07:16.614 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:16 smithi050 bash[21717]: debug 2024-04-26T16:07:16.506+0000 7fea0d9e7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T16:07:16.614 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:16 smithi050 bash[21717]: debug 2024-04-26T16:07:16.614+0000 7fea0d9e7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T16:07:17.378 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:16 smithi050 bash[21717]: debug 2024-04-26T16:07:16.978+0000 7fea0d9e7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T16:07:17.378 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:17 smithi050 bash[21717]: debug 2024-04-26T16:07:17.098+0000 7fea0d9e7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T16:07:17.378 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:17 smithi050 bash[21717]: debug 2024-04-26T16:07:17.202+0000 7fea0d9e7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T16:07:17.631 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:17 smithi050 bash[21717]: debug 2024-04-26T16:07:17.578+0000 7fea0d9e7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T16:07:17.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:17 smithi050 bash[21463]: audit 2024-04-26T16:07:17.367317+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? 172.21.15.50:0/59427280' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:17.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:17 smithi050 bash[21463]: audit 2024-04-26T16:07:17.367317+0000 mon.a (mon.0) 14 : audit [DBG] from='client.? 172.21.15.50:0/59427280' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T16:07:17.939 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.940 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T16:07:17.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:17.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T16:07:17.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T16:07:17.941 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T16:07:17.950 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T16:07:17.950 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T16:07:17.950 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T16:07:17.950 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.950 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T16:07:17.951 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T16:07:04.494728+0000", 2024-04-26T16:07:17.952 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 Processor -- start 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 -- start start 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e28106550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2807adf0 con 0x7f9e28106130 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e28106550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e28106550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56296/0 (socket says 172.21.15.50:56296) 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2659c700 1 -- 172.21.15.50:0/291458640 learned_addr learned my addr 172.21.15.50:0/291458640 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2659c700 1 -- 172.21.15.50:0/291458640 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e28106a90 con 0x7f9e28106130 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2659c700 1 --2- 172.21.15.50:0/291458640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e28106550 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0098a0 tx=0x7f9e1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9ae07871a4f676 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2559a700 1 -- 172.21.15.50:0/291458640 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e1c004160 con 0x7f9e28106130 2024-04-26T16:07:17.953 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2559a700 1 -- 172.21.15.50:0/291458640 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f9e1c004300 con 0x7f9e28106130 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2559a700 1 -- 172.21.15.50:0/291458640 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e1c0118c0 con 0x7f9e28106130 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/291458640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 msgr2=0x7f9e28106550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 --2- 172.21.15.50:0/291458640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e28106550 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c0098a0 tx=0x7f9e1c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/291458640 shutdown_connections 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 --2- 172.21.15.50:0/291458640 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e28106550 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.222+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/291458640 >> 172.21.15.50:0/291458640 conn(0x7f9e28101cd0 msgr2=0x7f9e28104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/291458640 shutdown_connections 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/291458640 wait complete. 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 librados: starting msgr at 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 librados: starting objecter 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 Processor -- start 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- start start 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 librados: setting wanted keys 2024-04-26T16:07:17.954 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 librados: calling monclient init 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e2819d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9e28107990 con 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e2819d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e2819d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56308/0 (socket says 172.21.15.50:56308) 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2659c700 1 -- 172.21.15.50:0/59427280 learned_addr learned my addr 172.21.15.50:0/59427280 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2659c700 1 -- 172.21.15.50:0/59427280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2819ddd0 con 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2659c700 1 --2- 172.21.15.50:0/59427280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e2819d890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c011da0 tx=0x7f9e1c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 10 client.?.objecter ms_handle_connect 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e1c0037d0 con 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e2819e060 con 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f9e1c01a430 con 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e1c024410 con 0x7f9e28106130 2024-04-26T16:07:17.955 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 10 client.14104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e281a0d50 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 librados: init done 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9e1c022020 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9e1c02d930 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 10 client.14104.objecter ms_dispatch 0x7f9e28001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 3 client.14104.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 3 client.14104.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e177fe700 20 client.14104.objecter dump_active .. 0 homeless 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.226+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e28064a80 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.230+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f9e1c004300 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.362+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f9e2802e4c0 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e177fe700 1 -- 172.21.15.50:0/59427280 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f9e1c032030 con 0x7f9e28106130 2024-04-26T16:07:17.956 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 10 librados: watch_flush enter 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 10 librados: watch_flush exit 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 20 client.14104.objecter shutdown clearing up homeless session... 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 10 client.14104.objecter successfully canceled tick 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 msgr2=0x7f9e2819d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 --2- 172.21.15.50:0/59427280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e2819d890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c011da0 tx=0x7f9e1c011e80 comp rx=0 tx=0).stop 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 shutdown_connections 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 --2- 172.21.15.50:0/59427280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9e28106130 0x7f9e2819d890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 >> 172.21.15.50:0/59427280 conn(0x7f9e28101cd0 msgr2=0x7f9e28194550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 shutdown_connections 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 -- 172.21.15.50:0/59427280 wait complete. 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:17.366+0000 7f9e2c9c6700 1 librados: shutdown 2024-04-26T16:07:17.957 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (2/15)... 2024-04-26T16:07:18.093 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:17 smithi050 bash[21717]: debug 2024-04-26T16:07:17.822+0000 7fea0d9e7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T16:07:18.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:18 smithi050 bash[21717]: debug 2024-04-26T16:07:18.094+0000 7fea0d9e7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T16:07:18.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:18 smithi050 bash[21717]: debug 2024-04-26T16:07:18.210+0000 7fea0d9e7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T16:07:18.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:18 smithi050 bash[21717]: debug 2024-04-26T16:07:18.494+0000 7fea0d9e7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T16:07:19.280 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:18 smithi050 bash[21717]: debug 2024-04-26T16:07:18.914+0000 7fea0d9e7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:07:19.281 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:19 smithi050 bash[21717]: debug 2024-04-26T16:07:19.158+0000 7fea0d9e7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T16:07:19.533 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:19 smithi050 bash[21717]: debug 2024-04-26T16:07:19.282+0000 7fea0d9e7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T16:07:20.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:19 smithi050 bash[21717]: debug 2024-04-26T16:07:19.534+0000 7fea0d9e7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:07:20.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:20 smithi050 bash[21717]: debug 2024-04-26T16:07:20.574+0000 7fea0d9e7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T16:07:20.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:20 smithi050 bash[21717]: debug 2024-04-26T16:07:20.702+0000 7fea0d9e7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T16:07:21.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:21 smithi050 bash[21463]: audit 2024-04-26T16:07:20.960402+0000 mon.a (mon.0) 15 : audit [DBG] from='client.? 172.21.15.50:0/805666070' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:21.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:21 smithi050 bash[21463]: audit 2024-04-26T16:07:20.960402+0000 mon.a (mon.0) 15 : audit [DBG] from='client.? 172.21.15.50:0/805666070' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:21.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:21 smithi050 bash[21717]: debug 2024-04-26T16:07:21.006+0000 7fea0d9e7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T16:07:21.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:21 smithi050 bash[21717]: debug 2024-04-26T16:07:21.122+0000 7fea0d9e7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T16:07:21.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T16:07:21.565 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T16:07:21.566 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T16:07:04.494728+0000", 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b3244700 1 Processor -- start 2024-04-26T16:07:21.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b3244700 1 -- start start 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b3244700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac109ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b3244700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac10a4b0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b0fe0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac109ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b0fe0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac109ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46276/0 (socket says 172.21.15.50:46276) 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.814+0000 7ff4b0fe0700 1 -- 172.21.15.50:0/4267997830 learned_addr learned my addr 172.21.15.50:0/4267997830 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 -- 172.21.15.50:0/4267997830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ac10acc0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 --2- 172.21.15.50:0/4267997830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac109ee0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00098a0 tx=0x7ff4a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8db5e37b0a2aeff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4ab7fe700 1 -- 172.21.15.50:0/4267997830 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4a0004160 con 0x7ff4ac109ac0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4ab7fe700 1 -- 172.21.15.50:0/4267997830 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7ff4a0004300 con 0x7ff4ac109ac0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4ab7fe700 1 -- 172.21.15.50:0/4267997830 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4a00118c0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/4267997830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 msgr2=0x7ff4ac109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 --2- 172.21.15.50:0/4267997830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac109ee0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff4a00098a0 tx=0x7ff4a0009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/4267997830 shutdown_connections 2024-04-26T16:07:21.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 --2- 172.21.15.50:0/4267997830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac109ee0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/4267997830 >> 172.21.15.50:0/4267997830 conn(0x7ff4ac07d020 msgr2=0x7ff4ac07d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/4267997830 shutdown_connections 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/4267997830 wait complete. 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 librados: starting msgr at 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 librados: starting objecter 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 Processor -- start 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- start start 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 librados: setting wanted keys 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 librados: calling monclient init 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac10def0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:21.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac19d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46292/0 (socket says 172.21.15.50:46292) 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 -- 172.21.15.50:0/805666070 learned_addr learned my addr 172.21.15.50:0/805666070 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 -- 172.21.15.50:0/805666070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ac19dea0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b0fe0700 1 --2- 172.21.15.50:0/805666070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac19d960 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0011e10 tx=0x7ff4a0011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 10 client.?.objecter ms_handle_connect 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4a00036a0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7ff4a0003c80 con 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ac19e130 con 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff4a001acc0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 10 client.14106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ac1a0e20 con 0x7ff4ac109ac0 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4b3244700 1 librados: init done 2024-04-26T16:07:21.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff4a002d7d0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff4a0024db0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 10 client.14106.objecter ms_dispatch 0x7ff4ac001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 3 client.14106.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 3 client.14106.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.818+0000 7ff4a9ffb700 20 client.14106.objecter dump_active .. 0 homeless 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.822+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4ac064a80 con 0x7ff4ac109ac0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.822+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff4a0003a50 con 0x7ff4ac109ac0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff4ac1a1200 con 0x7ff4ac109ac0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4a9ffb700 1 -- 172.21.15.50:0/805666070 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7ff4a001a5c0 con 0x7ff4ac109ac0 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 10 librados: watch_flush enter 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 10 librados: watch_flush exit 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 20 client.14106.objecter shutdown clearing up homeless session... 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 10 client.14106.objecter successfully canceled tick 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 msgr2=0x7ff4ac19d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:21.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 --2- 172.21.15.50:0/805666070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac19d960 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0011e10 tx=0x7ff4a0011ef0 comp rx=0 tx=0).stop 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 shutdown_connections 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 --2- 172.21.15.50:0/805666070 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4ac109ac0 0x7ff4ac19d960 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 >> 172.21.15.50:0/805666070 conn(0x7ff4ac07d020 msgr2=0x7ff4ac106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 shutdown_connections 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 -- 172.21.15.50:0/805666070 wait complete. 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:20.958+0000 7ff4b3244700 1 librados: shutdown 2024-04-26T16:07:21.572 INFO:teuthology.orchestra.run.smithi050.stdout:mgr not available, waiting (3/15)... 2024-04-26T16:07:21.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:21 smithi050 bash[21717]: debug 2024-04-26T16:07:21.418+0000 7fea0d9e7200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-26T16:07:22.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:21 smithi050 bash[21717]: debug 2024-04-26T16:07:21.906+0000 7fea0d9e7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21717]: debug 2024-04-26T16:07:22.030+0000 7fea0d9e7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: cluster 2024-04-26T16:07:22.053108+0000 mon.a (mon.0) 16 : cluster [INF] Activating manager daemon y 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: cluster 2024-04-26T16:07:22.053108+0000 mon.a (mon.0) 16 : cluster [INF] Activating manager daemon y 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: cluster 2024-04-26T16:07:22.060115+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00730922s) 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: cluster 2024-04-26T16:07:22.060115+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00730922s) 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.061603+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.061603+0000 mon.a (mon.0) 18 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:07:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.061861+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.061861+0000 mon.a (mon.0) 19 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.062113+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.062113+0000 mon.a (mon.0) 20 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.062326+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.062326+0000 mon.a (mon.0) 21 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.063494+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.063494+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: cluster 2024-04-26T16:07:22.089189+0000 mon.a (mon.0) 23 : cluster [INF] Manager daemon y is now available 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: cluster 2024-04-26T16:07:22.089189+0000 mon.a (mon.0) 23 : cluster [INF] Manager daemon y is now available 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.101779+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.101779+0000 mon.a (mon.0) 24 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.103948+0000 mon.a (mon.0) 25 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:07:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:22 smithi050 bash[21463]: audit 2024-04-26T16:07:22.103948+0000 mon.a (mon.0) 25 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: audit 2024-04-26T16:07:22.109091+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: audit 2024-04-26T16:07:22.109091+0000 mon.a (mon.0) 26 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: audit 2024-04-26T16:07:22.110729+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: audit 2024-04-26T16:07:22.110729+0000 mon.a (mon.0) 27 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: audit 2024-04-26T16:07:22.113424+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: audit 2024-04-26T16:07:22.113424+0000 mon.a (mon.0) 28 : audit [INF] from='mgr.14100 172.21.15.50:0/2593971400' entity='mgr.y' 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: cluster 2024-04-26T16:07:23.069156+0000 mon.a (mon.0) 29 : cluster [DBG] mgrmap e3: y(active, since 1.01635s) 2024-04-26T16:07:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:23 smithi050 bash[21463]: cluster 2024-04-26T16:07:23.069156+0000 mon.a (mon.0) 29 : cluster [DBG] mgrmap e3: y(active, since 1.01635s) 2024-04-26T16:07:25.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:25 smithi050 bash[21463]: cluster 2024-04-26T16:07:24.115836+0000 mon.a (mon.0) 30 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-04-26T16:07:25.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:25 smithi050 bash[21463]: cluster 2024-04-26T16:07:24.115836+0000 mon.a (mon.0) 30 : cluster [DBG] mgrmap e4: y(active, since 2s) 2024-04-26T16:07:25.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:25 smithi050 bash[21463]: audit 2024-04-26T16:07:25.057973+0000 mon.a (mon.0) 31 : audit [DBG] from='client.? 172.21.15.50:0/4053110582' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:25.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:25 smithi050 bash[21463]: audit 2024-04-26T16:07:25.057973+0000 mon.a (mon.0) 31 : audit [DBG] from='client.? 172.21.15.50:0/4053110582' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "health": { 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-26T16:07:25.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 0 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "a" 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-26T16:07:25.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-26T16:07:25.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:25.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-26T16:07:25.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-26T16:07:25.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-26T16:07:25.701 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-26T16:07:25.701 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-26T16:07:25.701 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-26T16:07:25.701 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:25.702 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "restful" 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ], 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-26T16:07:25.703 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "modified": "2024-04-26T16:07:04.494728+0000", 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout }, 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34dd05e700 1 Processor -- start 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34dd05e700 1 -- start start 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34dd05e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d8107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34dd05e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f34d81082f0 con 0x7f34d8107900 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d8107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d8107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46378/0 (socket says 172.21.15.50:46378) 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d6d9d700 1 -- 172.21.15.50:0/3509096711 learned_addr learned my addr 172.21.15.50:0/3509096711 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d6d9d700 1 -- 172.21.15.50:0/3509096711 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d8108b00 con 0x7f34d8107900 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d6d9d700 1 --2- 172.21.15.50:0/3509096711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d8107d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34c80098a0 tx=0x7f34c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc6fc168a84971d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d5d9b700 1 -- 172.21.15.50:0/3509096711 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34c8004160 con 0x7f34d8107900 2024-04-26T16:07:25.704 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d5d9b700 1 -- 172.21.15.50:0/3509096711 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f34c8004300 con 0x7f34d8107900 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.478+0000 7f34d5d9b700 1 -- 172.21.15.50:0/3509096711 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34c80118c0 con 0x7f34d8107900 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/3509096711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 msgr2=0x7f34d8107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 --2- 172.21.15.50:0/3509096711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d8107d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34c80098a0 tx=0x7f34c8009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/3509096711 shutdown_connections 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 --2- 172.21.15.50:0/3509096711 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d8107d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/3509096711 >> 172.21.15.50:0/3509096711 conn(0x7f34d8102e60 msgr2=0x7f34d81052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/3509096711 shutdown_connections 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/3509096711 wait complete. 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 librados: starting msgr at 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 librados: starting objecter 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 Processor -- start 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- start start 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 librados: setting wanted keys 2024-04-26T16:07:25.705 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 librados: calling monclient init 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d807d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f34d8109990 con 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34d6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d807d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34d6d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d807d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46394/0 (socket says 172.21.15.50:46394) 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34d6d9d700 1 -- 172.21.15.50:0/4053110582 learned_addr learned my addr 172.21.15.50:0/4053110582 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34d6d9d700 1 -- 172.21.15.50:0/4053110582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d807d920 con 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34d6d9d700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d807d3e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009870 tx=0x7f34c8011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 10 client.?.objecter ms_handle_connect 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34c80036a0 con 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f34c801a430 con 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d807bb10 con 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34c80193c0 con 0x7f34d8107900 2024-04-26T16:07:25.706 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 10 client.14114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d807c050 con 0x7f34d8107900 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 librados: init done 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) ==== 47166+0+0 (secure 0 0 0) 0x7f34c80196c0 con 0x7f34d8107900 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f34c003bbc0 0x7f34c003e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f34c804e7f0 con 0x7f34d8107900 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 10 client.14114.objecter ms_dispatch 0x7f34d8001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 3 client.14114.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 3 client.14114.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34bffff700 20 client.14114.objecter dump_active .. 0 homeless 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d8064a80 con 0x7f34d8107900 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.482+0000 7f34d659c700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f34c003bbc0 0x7f34c003e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.486+0000 7f34d659c700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f34c003bbc0 0x7f34c003e080 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00ad30 tx=0x7f34cc0093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.486+0000 7f34bffff700 10 client.14114.objecter ms_handle_connect 0x7f34c003bbc0 2024-04-26T16:07:25.707 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:24.490+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f34c804e210 con 0x7f34d8107900 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.054+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f34d807c6b0 con 0x7f34d8107900 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.054+0000 7f34bffff700 1 -- 172.21.15.50:0/4053110582 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f34c800fda0 con 0x7f34d8107900 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 10 librados: watch_flush enter 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 10 librados: watch_flush exit 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 20 client.14114.objecter shutdown clearing up homeless session... 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 10 client.14114.objecter successfully canceled tick 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f34c003bbc0 msgr2=0x7f34c003e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f34c003bbc0 0x7f34c003e080 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f34cc00ad30 tx=0x7f34cc0093f0 comp rx=0 tx=0).stop 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 msgr2=0x7f34d807d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d807d3e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f34c8009870 tx=0x7f34c8011660 comp rx=0 tx=0).stop 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 shutdown_connections 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f34c003bbc0 0x7f34c003e080 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 --2- 172.21.15.50:0/4053110582 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34d8107900 0x7f34d807d3e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:25.708 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 >> 172.21.15.50:0/4053110582 conn(0x7f34d8102e60 msgr2=0x7f34d8103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:25.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 shutdown_connections 2024-04-26T16:07:25.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 -- 172.21.15.50:0/4053110582 wait complete. 2024-04-26T16:07:25.709 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:25.058+0000 7f34dd05e700 1 librados: shutdown 2024-04-26T16:07:25.709 INFO:teuthology.orchestra.run.smithi050.stdout:mgr is available 2024-04-26T16:07:27.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:27 smithi050 bash[21463]: audit 2024-04-26T16:07:26.966757+0000 mon.a (mon.0) 32 : audit [INF] from='client.? 172.21.15.50:0/1347535209' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T16:07:27.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:27 smithi050 bash[21463]: audit 2024-04-26T16:07:26.966757+0000 mon.a (mon.0) 32 : audit [INF] from='client.? 172.21.15.50:0/1347535209' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-26T16:07:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [global] 2024-04-26T16:07:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout fsid = cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-26T16:07:27.563 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-26T16:07:27.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-26T16:07:27.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-26T16:07:27.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-26T16:07:27.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:27.564 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout [osd] 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 Processor -- start 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- start start 2024-04-26T16:07:27.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081007bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f081007c120 con 0x7f081007d6f0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f081581a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081007bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f081581a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081007bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46396/0 (socket says 172.21.15.50:46396) 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f081581a700 1 -- 172.21.15.50:0/1512228225 learned_addr learned my addr 172.21.15.50:0/1512228225 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f081581a700 1 -- 172.21.15.50:0/1512228225 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081007c2a0 con 0x7f081007d6f0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f081581a700 1 --2- 172.21.15.50:0/1512228225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081007bb50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f080c0098a0 tx=0x7f080c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1102b5e74e1ae39 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0814818700 1 -- 172.21.15.50:0/1512228225 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f080c004160 con 0x7f081007d6f0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0814818700 1 -- 172.21.15.50:0/1512228225 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f080c004300 con 0x7f081007d6f0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0814818700 1 -- 172.21.15.50:0/1512228225 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f080c0118c0 con 0x7f081007d6f0 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- 172.21.15.50:0/1512228225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 msgr2=0x7f081007bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 --2- 172.21.15.50:0/1512228225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081007bb50 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f080c0098a0 tx=0x7f080c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- 172.21.15.50:0/1512228225 shutdown_connections 2024-04-26T16:07:27.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 --2- 172.21.15.50:0/1512228225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081007bb50 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- 172.21.15.50:0/1512228225 >> 172.21.15.50:0/1512228225 conn(0x7f0810102e60 msgr2=0x7f08101052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- 172.21.15.50:0/1512228225 shutdown_connections 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- 172.21.15.50:0/1512228225 wait complete. 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 librados: starting msgr at 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 librados: starting objecter 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 Processor -- start 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- start start 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 librados: setting wanted keys 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 librados: calling monclient init 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.562+0000 7f0817a7e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f081010dd70 con 0x7f081007d6f0 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f081581a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:27.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f081581a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46398/0 (socket says 172.21.15.50:46398) 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f081581a700 1 -- 172.21.15.50:0/1347535209 learned_addr learned my addr 172.21.15.50:0/1347535209 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f081581a700 1 -- 172.21.15.50:0/1347535209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f081019de50 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f081581a700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081019d910 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f080c011da0 tx=0x7f080c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 10 client.?.objecter ms_handle_connect 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f080c0037d0 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f080c01a430 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f080c024410 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f081019e0e0 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0817a7e700 10 client.14116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08101a0dd0 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0817a7e700 1 librados: init done 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) ==== 47166+0+0 (secure 0 0 0) 0x7f080c022020 con 0x7f081007d6f0 2024-04-26T16:07:27.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f07fc03bc80 0x7f07fc03e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:27.575 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f080c04e0e0 con 0x7f081007d6f0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 10 client.14116.objecter ms_dispatch 0x7f0810001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 3 client.14116.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 3 client.14116.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 20 client.14116.objecter dump_active .. 0 homeless 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0810064a80 con 0x7f081007d6f0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0815019700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f07fc03bc80 0x7f07fc03e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0815019700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f07fc03bc80 0x7f07fc03e140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0800006fd0 tx=0x7f0800006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.566+0000 7f0806ffd700 10 client.14116.objecter ms_handle_connect 0x7f07fc03bc80 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.570+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f080c020c10 con 0x7f081007d6f0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.962+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f08101a2260 con 0x7f081007d6f0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0806ffd700 1 -- 172.21.15.50:0/1347535209 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+318 (secure 0 0 0) 0x7f080c01a5e0 con 0x7f081007d6f0 2024-04-26T16:07:27.576 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 10 librados: watch_flush enter 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 10 librados: watch_flush exit 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 20 client.14116.objecter shutdown clearing up homeless session... 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 10 client.14116.objecter successfully canceled tick 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f07fc03bc80 msgr2=0x7f07fc03e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f07fc03bc80 0x7f07fc03e140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0800006fd0 tx=0x7f0800006e40 comp rx=0 tx=0).stop 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 msgr2=0x7f081019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081019d910 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f080c011da0 tx=0x7f080c011e80 comp rx=0 tx=0).stop 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 shutdown_connections 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f07fc03bc80 0x7f07fc03e140 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 --2- 172.21.15.50:0/1347535209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f081007d6f0 0x7f081019d910 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:27.577 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 >> 172.21.15.50:0/1347535209 conn(0x7f0810102e60 msgr2=0x7f08101039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:27.578 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 shutdown_connections 2024-04-26T16:07:27.578 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 -- 172.21.15.50:0/1347535209 wait complete. 2024-04-26T16:07:27.578 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:26.966+0000 7f0817a7e700 1 librados: shutdown 2024-04-26T16:07:27.578 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling cephadm module... 2024-04-26T16:07:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:29 smithi050 bash[21463]: audit 2024-04-26T16:07:28.974240+0000 mon.a (mon.0) 33 : audit [INF] from='client.? 172.21.15.50:0/3365330534' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T16:07:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:29 smithi050 bash[21463]: audit 2024-04-26T16:07:28.974240+0000 mon.a (mon.0) 33 : audit [INF] from='client.? 172.21.15.50:0/3365330534' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-26T16:07:29.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:29 smithi050 bash[21717]: ignoring --setuser ceph since I am not root 2024-04-26T16:07:29.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:29 smithi050 bash[21717]: ignoring --setgroup ceph since I am not root 2024-04-26T16:07:29.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:29 smithi050 bash[21717]: debug 2024-04-26T16:07:29.214+0000 7fb0d6b7e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T16:07:29.591 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99bbd8b700 1 Processor -- start 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99bbd8b700 1 -- start start 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99bbd8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b407bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99bbd8b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99b407c140 con 0x7f99b407d710 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b9b27700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b407bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b9b27700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b407bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58596/0 (socket says 172.21.15.50:58596) 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b9b27700 1 -- 172.21.15.50:0/2592069978 learned_addr learned my addr 172.21.15.50:0/2592069978 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b9b27700 1 -- 172.21.15.50:0/2592069978 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b407c2c0 con 0x7f99b407d710 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b9b27700 1 --2- 172.21.15.50:0/2592069978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b407bb70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f99b00098a0 tx=0x7f99b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adff2f8ee11cac01 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b8b25700 1 -- 172.21.15.50:0/2592069978 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b0004160 con 0x7f99b407d710 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b8b25700 1 -- 172.21.15.50:0/2592069978 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f99b0004300 con 0x7f99b407d710 2024-04-26T16:07:29.592 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.454+0000 7f99b8b25700 1 -- 172.21.15.50:0/2592069978 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b00118c0 con 0x7f99b407d710 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/2592069978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 msgr2=0x7f99b407bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 --2- 172.21.15.50:0/2592069978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b407bb70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f99b00098a0 tx=0x7f99b0009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/2592069978 shutdown_connections 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 --2- 172.21.15.50:0/2592069978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b407bb70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/2592069978 >> 172.21.15.50:0/2592069978 conn(0x7f99b4102e80 msgr2=0x7f99b41052c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/2592069978 shutdown_connections 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/2592069978 wait complete. 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 librados: starting msgr at 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 librados: starting objecter 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 Processor -- start 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- start start 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 librados: setting wanted keys 2024-04-26T16:07:29.593 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 librados: calling monclient init 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f99b410dd70 con 0x7f99b407d710 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99b9b27700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99b9b27700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b419d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58610/0 (socket says 172.21.15.50:58610) 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99b9b27700 1 -- 172.21.15.50:0/3365330534 learned_addr learned my addr 172.21.15.50:0/3365330534 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99b9b27700 1 -- 172.21.15.50:0/3365330534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b419deb0 con 0x7f99b407d710 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99b9b27700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b419d970 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009870 tx=0x7f99b0011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99aaffd700 10 client.?.objecter ms_handle_connect 0x7f99b407d710 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99aaffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b00036a0 con 0x7f99b407d710 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f99b0003c80 con 0x7f99b407d710 2024-04-26T16:07:29.594 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b419e140 con 0x7f99b407d710 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f99b001acd0 con 0x7f99b407d710 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 10 client.14118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b41a0e30 con 0x7f99b407d710 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 librados: init done 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 4) ==== 47166+0+0 (secure 0 0 0) 0x7f99b002d440 con 0x7f99b407d710 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.458+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b4064a80 con 0x7f99b407d710 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f99a003bc80 0x7f99a003e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f99b004e990 con 0x7f99b407d710 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 10 client.14118.objecter ms_dispatch 0x7f99b4001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 3 client.14118.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 3 client.14118.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 20 client.14118.objecter dump_active .. 0 homeless 2024-04-26T16:07:29.595 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99b9326700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f99a003bc80 0x7f99a003e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99b9326700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f99a003bc80 0x7f99a003e140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f99a400ad30 tx=0x7f99a40093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.462+0000 7f99aaffd700 10 client.14118.objecter ms_handle_connect 0x7f99a003bc80 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.466+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f99b002d730 con 0x7f99b407d710 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:28.970+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f99b41a1290 con 0x7f99b407d710 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.022+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f99b00038f0 con 0x7f99b407d710 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99aaffd700 1 -- 172.21.15.50:0/3365330534 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 5) ==== 47177+0+0 (secure 0 0 0) 0x7f99b000b3e0 con 0x7f99b407d710 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 10 librados: watch_flush enter 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 10 librados: watch_flush exit 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 20 client.14118.objecter shutdown clearing up homeless session... 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 10 client.14118.objecter successfully canceled tick 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f99a003bc80 msgr2=0x7f99a003e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f99a003bc80 0x7f99a003e140 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f99a400ad30 tx=0x7f99a40093f0 comp rx=0 tx=0).stop 2024-04-26T16:07:29.596 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 msgr2=0x7f99b419d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b419d970 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009870 tx=0x7f99b0011660 comp rx=0 tx=0).stop 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 shutdown_connections 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f99a003bc80 0x7f99a003e140 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 --2- 172.21.15.50:0/3365330534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f99b407d710 0x7f99b419d970 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 >> 172.21.15.50:0/3365330534 conn(0x7f99b4102e80 msgr2=0x7f99b4103a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 shutdown_connections 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 -- 172.21.15.50:0/3365330534 wait complete. 2024-04-26T16:07:29.597 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:29.026+0000 7f99bbd8b700 1 librados: shutdown 2024-04-26T16:07:30.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:30 smithi050 bash[21463]: audit 2024-04-26T16:07:29.025850+0000 mon.a (mon.0) 34 : audit [INF] from='client.? 172.21.15.50:0/3365330534' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T16:07:30.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:30 smithi050 bash[21463]: audit 2024-04-26T16:07:29.025850+0000 mon.a (mon.0) 34 : audit [INF] from='client.? 172.21.15.50:0/3365330534' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-26T16:07:30.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:30 smithi050 bash[21463]: cluster 2024-04-26T16:07:29.030483+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e5: y(active, since 6s) 2024-04-26T16:07:30.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:30 smithi050 bash[21463]: cluster 2024-04-26T16:07:29.030483+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e5: y(active, since 6s) 2024-04-26T16:07:30.971 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:30 smithi050 bash[21717]: debug 2024-04-26T16:07:30.866+0000 7fb0d6b7e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T16:07:30.971 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:30 smithi050 bash[21717]: debug 2024-04-26T16:07:30.970+0000 7fb0d6b7e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T16:07:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:31 smithi050 bash[21463]: audit 2024-04-26T16:07:30.914829+0000 mon.a (mon.0) 36 : audit [DBG] from='client.? 172.21.15.50:0/3573521694' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T16:07:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:31 smithi050 bash[21463]: audit 2024-04-26T16:07:30.914829+0000 mon.a (mon.0) 36 : audit [DBG] from='client.? 172.21.15.50:0/3573521694' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T16:07:31.520 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:07:31.520 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-26T16:07:31.520 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.386+0000 7f409cb42700 1 Processor -- start 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.386+0000 7f409cb42700 1 -- start start 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.386+0000 7f409cb42700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f4098106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.386+0000 7f409cb42700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f409807ae10 con 0x7f4098106110 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f4098106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f4098106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58624/0 (socket says 172.21.15.50:58624) 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 -- 172.21.15.50:0/3695432633 learned_addr learned my addr 172.21.15.50:0/3695432633 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 -- 172.21.15.50:0/3695432633 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4098106a70 con 0x7f4098106110 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 --2- 172.21.15.50:0/3695432633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f4098106530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f408c0098a0 tx=0x7f408c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74558641646e1302 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409559a700 1 -- 172.21.15.50:0/3695432633 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f408c004160 con 0x7f4098106110 2024-04-26T16:07:31.521 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409559a700 1 -- 172.21.15.50:0/3695432633 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f408c004300 con 0x7f4098106110 2024-04-26T16:07:31.522 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409559a700 1 -- 172.21.15.50:0/3695432633 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f408c0118c0 con 0x7f4098106110 2024-04-26T16:07:31.522 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- 172.21.15.50:0/3695432633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 msgr2=0x7f4098106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:31.522 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 --2- 172.21.15.50:0/3695432633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f4098106530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f408c0098a0 tx=0x7f408c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- 172.21.15.50:0/3695432633 shutdown_connections 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 --2- 172.21.15.50:0/3695432633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f4098106530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- 172.21.15.50:0/3695432633 >> 172.21.15.50:0/3695432633 conn(0x7f4098101cd0 msgr2=0x7f4098104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- 172.21.15.50:0/3695432633 shutdown_connections 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- 172.21.15.50:0/3695432633 wait complete. 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 librados: starting msgr at 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 librados: starting objecter 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 Processor -- start 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- start start 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 librados: setting wanted keys 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 librados: calling monclient init 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f409819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:31.523 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4098107970 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f409819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f409819d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58640/0 (socket says 172.21.15.50:58640) 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 -- 172.21.15.50:0/3573521694 learned_addr learned my addr 172.21.15.50:0/3573521694 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 -- 172.21.15.50:0/3573521694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409819ddf0 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409659c700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f409819d8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f408c011da0 tx=0x7f408c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f40877fe700 10 client.?.objecter ms_handle_connect 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f40877fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f408c0037d0 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f408c01a430 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409819e080 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f408c024410 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.390+0000 7f409cb42700 10 client.14122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40981a0d70 con 0x7f4098106110 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f409cb42700 1 librados: init done 2024-04-26T16:07:31.524 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) ==== 47177+0+0 (secure 0 0 0) 0x7f408c022020 con 0x7f4098106110 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 0x7f408003e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f408c04e330 con 0x7f4098106110 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 10 client.14122.objecter ms_dispatch 0x7f4098001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 3 client.14122.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 3 client.14122.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f4095d9b700 1 -- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 msgr2=0x7f408003e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f4095d9b700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 0x7f408003e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f40877fe700 20 client.14122.objecter dump_active .. 0 homeless 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.394+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4098064a80 con 0x7f4098106110 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.398+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f408c018e00 con 0x7f4098106110 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.598+0000 7f4095d9b700 1 -- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 msgr2=0x7f408003e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.598+0000 7f4095d9b700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 0x7f408003e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.910+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f4098196d70 con 0x7f4098106110 2024-04-26T16:07:31.525 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.910+0000 7f40877fe700 1 -- 172.21.15.50:0/3573521694 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f408c04e090 con 0x7f4098106110 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 10 librados: watch_flush enter 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 10 librados: watch_flush exit 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 20 client.14122.objecter shutdown clearing up homeless session... 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 10 client.14122.objecter successfully canceled tick 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 msgr2=0x7f408003e190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 0x7f408003e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 msgr2=0x7f409819d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f409819d8b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f408c011da0 tx=0x7f408c011e80 comp rx=0 tx=0).stop 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 shutdown_connections 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f408003bcd0 0x7f408003e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 --2- 172.21.15.50:0/3573521694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4098106110 0x7f409819d8b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 >> 172.21.15.50:0/3573521694 conn(0x7f4098101cd0 msgr2=0x7f40981944d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:31.526 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 shutdown_connections 2024-04-26T16:07:31.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 -- 172.21.15.50:0/3573521694 wait complete. 2024-04-26T16:07:31.527 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:30.914+0000 7f409cb42700 1 librados: shutdown 2024-04-26T16:07:31.527 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for the mgr to restart... 2024-04-26T16:07:31.527 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr epoch 5... 2024-04-26T16:07:31.531 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:31 smithi050 bash[21717]: debug 2024-04-26T16:07:31.226+0000 7fb0d6b7e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T16:07:32.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:32 smithi050 bash[21717]: debug 2024-04-26T16:07:32.054+0000 7fb0d6b7e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T16:07:32.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:32 smithi050 bash[21717]: debug 2024-04-26T16:07:32.170+0000 7fb0d6b7e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T16:07:32.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:32 smithi050 bash[21717]: debug 2024-04-26T16:07:32.278+0000 7fb0d6b7e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T16:07:33.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:32 smithi050 bash[21717]: debug 2024-04-26T16:07:32.650+0000 7fb0d6b7e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T16:07:33.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:32 smithi050 bash[21717]: debug 2024-04-26T16:07:32.766+0000 7fb0d6b7e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T16:07:33.132 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:32 smithi050 bash[21717]: debug 2024-04-26T16:07:32.874+0000 7fb0d6b7e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T16:07:33.632 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:33 smithi050 bash[21717]: debug 2024-04-26T16:07:33.246+0000 7fb0d6b7e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T16:07:33.632 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:33 smithi050 bash[21717]: debug 2024-04-26T16:07:33.490+0000 7fb0d6b7e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T16:07:34.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:33 smithi050 bash[21717]: debug 2024-04-26T16:07:33.762+0000 7fb0d6b7e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T16:07:34.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:33 smithi050 bash[21717]: debug 2024-04-26T16:07:33.878+0000 7fb0d6b7e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T16:07:34.586 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:34 smithi050 bash[21717]: debug 2024-04-26T16:07:34.162+0000 7fb0d6b7e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T16:07:34.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:34 smithi050 bash[21717]: debug 2024-04-26T16:07:34.586+0000 7fb0d6b7e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:07:34.882 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:34 smithi050 bash[21717]: debug 2024-04-26T16:07:34.834+0000 7fb0d6b7e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T16:07:35.214 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:34 smithi050 bash[21717]: debug 2024-04-26T16:07:34.958+0000 7fb0d6b7e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T16:07:35.631 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:35 smithi050 bash[21717]: debug 2024-04-26T16:07:35.214+0000 7fb0d6b7e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:07:36.631 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:36 smithi050 bash[21717]: debug 2024-04-26T16:07:36.254+0000 7fb0d6b7e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T16:07:36.631 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:36 smithi050 bash[21717]: debug 2024-04-26T16:07:36.374+0000 7fb0d6b7e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T16:07:37.086 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:36 smithi050 bash[21717]: debug 2024-04-26T16:07:36.678+0000 7fb0d6b7e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T16:07:37.087 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:36 smithi050 bash[21717]: debug 2024-04-26T16:07:36.794+0000 7fb0d6b7e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T16:07:37.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21717]: debug 2024-04-26T16:07:37.086+0000 7fb0d6b7e200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-26T16:07:37.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21717]: debug 2024-04-26T16:07:37.558+0000 7fb0d6b7e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T16:07:37.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21717]: debug 2024-04-26T16:07:37.682+0000 7fb0d6b7e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.705184+0000 mon.a (mon.0) 37 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.705184+0000 mon.a (mon.0) 37 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.705674+0000 mon.a (mon.0) 38 : cluster [INF] Activating manager daemon y 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.705674+0000 mon.a (mon.0) 38 : cluster [INF] Activating manager daemon y 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.709177+0000 mon.a (mon.0) 39 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.709177+0000 mon.a (mon.0) 39 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.709299+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00381001s) 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.709299+0000 mon.a (mon.0) 40 : cluster [DBG] mgrmap e6: y(active, starting, since 0.00381001s) 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.712823+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.712823+0000 mon.a (mon.0) 41 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:07:37.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.714833+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.714833+0000 mon.a (mon.0) 42 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.715626+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.715626+0000 mon.a (mon.0) 43 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.716193+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.716193+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.716708+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: audit 2024-04-26T16:07:37.716708+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.741458+0000 mon.a (mon.0) 46 : cluster [INF] Manager daemon y is now available 2024-04-26T16:07:37.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:37 smithi050 bash[21463]: cluster 2024-04-26T16:07:37.741458+0000 mon.a (mon.0) 46 : cluster [INF] Manager daemon y is now available 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: cephadm 2024-04-26T16:07:37.760027+0000 mgr.y (mgr.14120) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: cephadm 2024-04-26T16:07:37.760027+0000 mgr.y (mgr.14120) 1 : cephadm [INF] Found migration_current of "None". Setting to last migration. 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.761898+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.761898+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.764416+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.764416+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.769444+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.769444+0000 mon.a (mon.0) 49 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:39.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.772743+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.772743+0000 mon.a (mon.0) 50 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.783057+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.783057+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.785785+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:37.785785+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:38.188022+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:38.188022+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:38.191503+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: audit 2024-04-26T16:07:38.191503+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: cluster 2024-04-26T16:07:38.717793+0000 mon.a (mon.0) 55 : cluster [DBG] mgrmap e7: y(active, since 1.01229s) 2024-04-26T16:07:39.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:38 smithi050 bash[21463]: cluster 2024-04-26T16:07:38.717793+0000 mon.a (mon.0) 55 : cluster [DBG] mgrmap e7: y(active, since 1.01229s) 2024-04-26T16:07:39.394 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:07:39.394 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-26T16:07:39.394 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T16:07:39.394 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:07:39.395 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 Processor -- start 2024-04-26T16:07:39.395 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- start start 2024-04-26T16:07:39.395 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec108130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:39.395 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec108700 con 0x7f17ec107d10 2024-04-26T16:07:39.395 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec108130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:39.395 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec108130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58656/0 (socket says 172.21.15.50:58656) 2024-04-26T16:07:39.396 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17ebfff700 1 -- 172.21.15.50:0/2104932908 learned_addr learned my addr 172.21.15.50:0/2104932908 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:39.396 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17ebfff700 1 -- 172.21.15.50:0/2104932908 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec106920 con 0x7f17ec107d10 2024-04-26T16:07:39.396 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17ebfff700 1 --2- 172.21.15.50:0/2104932908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec108130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0098a0 tx=0x7f17dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1e5e412b0635b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:39.397 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17eaffd700 1 -- 172.21.15.50:0/2104932908 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17dc004160 con 0x7f17ec107d10 2024-04-26T16:07:39.397 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17eaffd700 1 -- 172.21.15.50:0/2104932908 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f17dc004300 con 0x7f17ec107d10 2024-04-26T16:07:39.397 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17eaffd700 1 -- 172.21.15.50:0/2104932908 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17dc0118c0 con 0x7f17ec107d10 2024-04-26T16:07:39.397 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- 172.21.15.50:0/2104932908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 msgr2=0x7f17ec108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:39.397 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 --2- 172.21.15.50:0/2104932908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec108130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f17dc0098a0 tx=0x7f17dc009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.397 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- 172.21.15.50:0/2104932908 shutdown_connections 2024-04-26T16:07:39.398 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 --2- 172.21.15.50:0/2104932908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec108130 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.398 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- 172.21.15.50:0/2104932908 >> 172.21.15.50:0/2104932908 conn(0x7f17ec101450 msgr2=0x7f17ec103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:39.398 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- 172.21.15.50:0/2104932908 shutdown_connections 2024-04-26T16:07:39.398 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.402+0000 7f17f2749700 1 -- 172.21.15.50:0/2104932908 wait complete. 2024-04-26T16:07:39.398 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 librados: starting msgr at 2024-04-26T16:07:39.398 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 librados: starting objecter 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 Processor -- start 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 -- start start 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 librados: setting wanted keys 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 librados: calling monclient init 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec10c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f17ec107760 con 0x7f17ec107d10 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec10c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec10c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58666/0 (socket says 172.21.15.50:58666) 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17ebfff700 1 -- 172.21.15.50:0/1892904340 learned_addr learned my addr 172.21.15.50:0/1892904340 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17ebfff700 1 -- 172.21.15.50:0/1892904340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17ec10cb30 con 0x7f17ec107d10 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17ebfff700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec10c5f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17dc011e10 tx=0x7f17dc011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:39.400 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 10 client.?.objecter ms_handle_connect 0x7f17ec107d10 2024-04-26T16:07:39.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:39.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17dc0036a0 con 0x7f17ec107d10 2024-04-26T16:07:39.401 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17ec10ad20 con 0x7f17ec107d10 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f17dc01a430 con 0x7f17ec107d10 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 10 client.14124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17dc0193c0 con 0x7f17ec107d10 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17ec10b260 con 0x7f17ec107d10 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 librados: init done 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 5) ==== 47177+0+0 (secure 0 0 0) 0x7f17dc019670 con 0x7f17ec107d10 2024-04-26T16:07:39.403 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f17dc052500 con 0x7f17ec107d10 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 10 client.14124.objecter ms_dispatch 0x7f17ec001550 osd_map(1..1 src has 1..1) 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 3 client.14124.objecter handle_osd_map got epochs [1,1] > 0 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 3 client.14124.objecter handle_osd_map decoding full epoch 1 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17eb7fe700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 msgr2=0x7f17d803de10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17e97fa700 20 client.14124.objecter dump_active .. 0 homeless 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.406+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 --> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f17ec064a80 con 0x7f17d803b950 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.610+0000 7f17eb7fe700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 msgr2=0x7f17d803de10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:32.610+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:33.014+0000 7f17eb7fe700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 msgr2=0x7f17d803de10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:33.014+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T16:07:39.404 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:33.818+0000 7f17eb7fe700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 msgr2=0x7f17d803de10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:39.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:33.818+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T16:07:39.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:35.422+0000 7f17eb7fe700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 msgr2=0x7f17d803de10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/127073210 2024-04-26T16:07:39.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:35.422+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T16:07:39.407 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:37.410+0000 7f17f1f48700 10 client.14124.objecter tick 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:37.706+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mgrmap(e 6) ==== 46944+0+0 (secure 0 0 0) 0x7f17dc01a5e0 con 0x7f17ec107d10 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:37.706+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 msgr2=0x7f17d803de10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:37.706+0000 7f17e97fa700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.710+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 7) ==== 47071+0+0 (secure 0 0 0) 0x7f17dc039c30 con 0x7f17ec107d10 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.710+0000 7f17e97fa700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f17d803efc0 0x7f17d80413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.710+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f17ec064a80 con 0x7f17d803efc0 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.710+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f17d803efc0 0x7f17d80413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.714+0000 7f17e97fa700 10 client.14124.objecter ms_handle_connect 0x7f17d803efc0 2024-04-26T16:07:39.408 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.714+0000 7f17eb7fe700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f17d803efc0 0x7f17d80413b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003a10 tx=0x7f17d40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.714+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f17ec064a80 con 0x7f17d803efc0 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f17ec10b4e0 con 0x7f17d803efc0 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17e97fa700 1 -- 172.21.15.50:0/1892904340 <== mgr.14120 v2:172.21.15.50:6800/685732569 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f17ec10b4e0 con 0x7f17d803efc0 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 10 librados: watch_flush enter 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 10 librados: watch_flush exit 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 20 client.14124.objecter shutdown clearing up homeless session... 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 10 client.14124.objecter successfully canceled tick 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f17d803efc0 msgr2=0x7f17d80413b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f17d803efc0 0x7f17d80413b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f17d4003a10 tx=0x7f17d40092b0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 msgr2=0x7f17ec10c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec10c5f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f17dc011e10 tx=0x7f17dc011ef0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.409 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 shutdown_connections 2024-04-26T16:07:39.410 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f17d803efc0 0x7f17d80413b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17ec107d10 0x7f17ec10c5f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 --2- 172.21.15.50:0/1892904340 >> [v2:172.21.15.50:6800/127073210,v1:172.21.15.50:6801/127073210] conn(0x7f17d803b950 0x7f17d803de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 >> 172.21.15.50:0/1892904340 conn(0x7f17ec101450 msgr2=0x7f17ec101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 shutdown_connections 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 -- 172.21.15.50:0/1892904340 wait complete. 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:38.730+0000 7f17f2749700 1 librados: shutdown 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:mgr epoch 5 is available 2024-04-26T16:07:39.411 INFO:teuthology.orchestra.run.smithi050.stdout:Setting orchestrator backend to cephadm... 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: audit 2024-04-26T16:07:38.716978+0000 mgr.y (mgr.14120) 2 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: audit 2024-04-26T16:07:38.716978+0000 mgr.y (mgr.14120) 2 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: audit 2024-04-26T16:07:38.732945+0000 mgr.y (mgr.14120) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: audit 2024-04-26T16:07:38.732945+0000 mgr.y (mgr.14120) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.743723+0000 mgr.y (mgr.14120) 4 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Bus STARTING 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.743723+0000 mgr.y (mgr.14120) 4 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Bus STARTING 2024-04-26T16:07:40.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.862783+0000 mgr.y (mgr.14120) 5 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.862783+0000 mgr.y (mgr.14120) 5 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.964241+0000 mgr.y (mgr.14120) 6 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.964241+0000 mgr.y (mgr.14120) 6 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.964848+0000 mgr.y (mgr.14120) 7 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Bus STARTED 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: cephadm 2024-04-26T16:07:38.964848+0000 mgr.y (mgr.14120) 7 : cephadm [INF] [26/Apr/2024:16:07:38] ENGINE Bus STARTED 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: audit 2024-04-26T16:07:38.991990+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:40.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:39 smithi050 bash[21463]: audit 2024-04-26T16:07:38.991990+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:40 smithi050 bash[21463]: cluster 2024-04-26T16:07:39.775490+0000 mon.a (mon.0) 57 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-04-26T16:07:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:40 smithi050 bash[21463]: cluster 2024-04-26T16:07:39.775490+0000 mon.a (mon.0) 57 : cluster [DBG] mgrmap e8: y(active, since 2s) 2024-04-26T16:07:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:40 smithi050 bash[21463]: audit 2024-04-26T16:07:40.729500+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:40 smithi050 bash[21463]: audit 2024-04-26T16:07:40.729500+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:40 smithi050 bash[21463]: audit 2024-04-26T16:07:40.763546+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:40 smithi050 bash[21463]: audit 2024-04-26T16:07:40.763546+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb25511b700 1 Processor -- start 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb25511b700 1 -- start start 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb25511b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25007bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb25511b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb25007c100 con 0x7fb25007d6d0 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb24ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25007bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb24ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25007bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35518/0 (socket says 172.21.15.50:35518) 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.282+0000 7fb24ed9d700 1 -- 172.21.15.50:0/1361829743 learned_addr learned my addr 172.21.15.50:0/1361829743 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:41.341 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 -- 172.21.15.50:0/1361829743 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25007c280 con 0x7fb25007d6d0 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 --2- 172.21.15.50:0/1361829743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25007bb30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb2400098a0 tx=0x7fb240009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d849cce9c93467fd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24dd9b700 1 -- 172.21.15.50:0/1361829743 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb240004160 con 0x7fb25007d6d0 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24dd9b700 1 -- 172.21.15.50:0/1361829743 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fb240004300 con 0x7fb25007d6d0 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24dd9b700 1 -- 172.21.15.50:0/1361829743 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2400118c0 con 0x7fb25007d6d0 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1361829743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 msgr2=0x7fb25007bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 --2- 172.21.15.50:0/1361829743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25007bb30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb2400098a0 tx=0x7fb240009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1361829743 shutdown_connections 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 --2- 172.21.15.50:0/1361829743 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25007bb30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1361829743 >> 172.21.15.50:0/1361829743 conn(0x7fb250102e60 msgr2=0x7fb2501052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1361829743 shutdown_connections 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1361829743 wait complete. 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 librados: starting msgr at 2024-04-26T16:07:41.342 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 librados: starting objecter 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 Processor -- start 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- start start 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 librados: setting wanted keys 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 librados: calling monclient init 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb25010dd00 con 0x7fb25007d6d0 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25019d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35532/0 (socket says 172.21.15.50:35532) 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 -- 172.21.15.50:0/1957270528 learned_addr learned my addr 172.21.15.50:0/1957270528 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 -- 172.21.15.50:0/1957270528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb25019ddb0 con 0x7fb25007d6d0 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb24ed9d700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25019d870 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb240006af0 tx=0x7fb240011f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb237fff700 10 client.?.objecter ms_handle_connect 0x7fb25007d6d0 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb237fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2400036a0 con 0x7fb25007d6d0 2024-04-26T16:07:41.343 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fb24001a430 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb25019e040 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 10 client.14132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2501a0d30 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb25511b700 1 librados: init done 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2400193c0 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.286+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7fb240019680 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb237fff700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb23803bc20 0x7fb23803e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb250064a80 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb24004e6d0 con 0x7fb25007d6d0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb237fff700 10 client.14132.objecter ms_dispatch 0x7fb250001550 osd_map(2..2 src has 1..2) 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb237fff700 3 client.14132.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb237fff700 3 client.14132.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:07:41.344 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.290+0000 7fb237fff700 20 client.14132.objecter dump_active .. 0 homeless 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.294+0000 7fb24e59c700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb23803bc20 0x7fb23803e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.294+0000 7fb24e59c700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb23803bc20 0x7fb23803e0e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb24400ad30 tx=0x7fb2440093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.294+0000 7fb237fff700 10 client.14132.objecter ms_handle_connect 0x7fb23803bc20 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.294+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb240016260 con 0x7fb25007d6d0 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.718+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fb2501040f0 con 0x7fb23803bc20 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.726+0000 7fb237fff700 1 -- 172.21.15.50:0/1957270528 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb2501040f0 con 0x7fb23803bc20 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 10 librados: watch_flush enter 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 10 librados: watch_flush exit 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 20 client.14132.objecter shutdown clearing up homeless session... 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 10 client.14132.objecter successfully canceled tick 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb23803bc20 msgr2=0x7fb23803e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb23803bc20 0x7fb23803e0e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb24400ad30 tx=0x7fb2440093f0 comp rx=0 tx=0).stop 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 msgr2=0x7fb25019d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25019d870 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb240006af0 tx=0x7fb240011f10 comp rx=0 tx=0).stop 2024-04-26T16:07:41.345 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 shutdown_connections 2024-04-26T16:07:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb23803bc20 0x7fb23803e0e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 --2- 172.21.15.50:0/1957270528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb25007d6d0 0x7fb25019d870 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 >> 172.21.15.50:0/1957270528 conn(0x7fb250102e60 msgr2=0x7fb2501039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 shutdown_connections 2024-04-26T16:07:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 -- 172.21.15.50:0/1957270528 wait complete. 2024-04-26T16:07:41.346 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:40.730+0000 7fb25511b700 1 librados: shutdown 2024-04-26T16:07:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:41 smithi050 bash[21463]: audit 2024-04-26T16:07:40.722058+0000 mgr.y (mgr.14120) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:41 smithi050 bash[21463]: audit 2024-04-26T16:07:40.722058+0000 mgr.y (mgr.14120) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:43.138 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-26T16:07:43.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07f5097700 1 Processor -- start 2024-04-26T16:07:43.141 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07f5097700 1 -- start start 2024-04-26T16:07:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07f5097700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0105990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07f5097700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f07f0105ed0 con 0x7f07f0105570 2024-04-26T16:07:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07eed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0105990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:43.146 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07eed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0105990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35534/0 (socket says 172.21.15.50:35534) 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07eed9d700 1 -- 172.21.15.50:0/3654618210 learned_addr learned my addr 172.21.15.50:0/3654618210 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07eed9d700 1 -- 172.21.15.50:0/3654618210 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f0106050 con 0x7f07f0105570 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07eed9d700 1 --2- 172.21.15.50:0/3654618210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0105990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f07e00098a0 tx=0x7f07e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e82bbc477e19e50 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07edd9b700 1 -- 172.21.15.50:0/3654618210 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e0004160 con 0x7f07f0105570 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07edd9b700 1 -- 172.21.15.50:0/3654618210 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f07e0004300 con 0x7f07f0105570 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07edd9b700 1 -- 172.21.15.50:0/3654618210 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e00118c0 con 0x7f07f0105570 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07f5097700 1 -- 172.21.15.50:0/3654618210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 msgr2=0x7f07f0105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.154+0000 7f07f5097700 1 --2- 172.21.15.50:0/3654618210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0105990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f07e00098a0 tx=0x7f07e0009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/3654618210 shutdown_connections 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 --2- 172.21.15.50:0/3654618210 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0105990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/3654618210 >> 172.21.15.50:0/3654618210 conn(0x7f07f0101010 msgr2=0x7f07f0103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/3654618210 shutdown_connections 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/3654618210 wait complete. 2024-04-26T16:07:43.147 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 librados: starting msgr at 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 librados: starting objecter 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 Processor -- start 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- start start 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 librados: setting wanted keys 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 librados: calling monclient init 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0108740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f07f010ba70 con 0x7f07f0105570 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07eed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0108740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07eed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0108740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35550/0 (socket says 172.21.15.50:35550) 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07eed9d700 1 -- 172.21.15.50:0/2194820699 learned_addr learned my addr 172.21.15.50:0/2194820699 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07eed9d700 1 -- 172.21.15.50:0/2194820699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07f0108c80 con 0x7f07f0105570 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07eed9d700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0108740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f07e0011da0 tx=0x7f07e0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 10 client.?.objecter ms_handle_connect 0x7f07f0105570 2024-04-26T16:07:43.148 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e00037d0 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f07e001a430 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f07e0024410 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07f0106e70 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 10 client.14134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07f01073b0 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 librados: init done 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7f07e0022020 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07f0064a80 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f07d803bcd0 0x7f07d803e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f07e004e5d0 con 0x7f07f0105570 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 10 client.14134.objecter ms_dispatch 0x7f07f0001550 osd_map(2..2 src has 1..2) 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 3 client.14134.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:07:43.149 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 3 client.14134.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:07:43.150 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 20 client.14134.objecter dump_active .. 0 homeless 2024-04-26T16:07:43.150 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07ee59c700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f07d803bcd0 0x7f07d803e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:43.150 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07d7fff700 10 client.14134.objecter ms_handle_connect 0x7f07d803bcd0 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.158+0000 7f07ee59c700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f07d803bcd0 0x7f07d803e190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006fd0 tx=0x7f07e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.166+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f07e00212c0 con 0x7f07f0105570 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.562+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f07f0107be0 con 0x7f07d803bcd0 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.562+0000 7f07d7fff700 1 -- 172.21.15.50:0/2194820699 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f07f0107be0 con 0x7f07d803bcd0 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 10 librados: watch_flush enter 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 10 librados: watch_flush exit 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 20 client.14134.objecter shutdown clearing up homeless session... 2024-04-26T16:07:43.152 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 10 client.14134.objecter successfully canceled tick 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f07d803bcd0 msgr2=0x7f07d803e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f07d803bcd0 0x7f07d803e190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07e4006fd0 tx=0x7f07e4006e40 comp rx=0 tx=0).stop 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 msgr2=0x7f07f0108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0108740 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f07e0011da0 tx=0x7f07e0011e80 comp rx=0 tx=0).stop 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 shutdown_connections 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f07d803bcd0 0x7f07d803e190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 --2- 172.21.15.50:0/2194820699 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07f0105570 0x7f07f0108740 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 >> 172.21.15.50:0/2194820699 conn(0x7f07f0101010 msgr2=0x7f07f0101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 shutdown_connections 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 -- 172.21.15.50:0/2194820699 wait complete. 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:42.566+0000 7f07f5097700 1 librados: shutdown 2024-04-26T16:07:43.153 INFO:teuthology.orchestra.run.smithi050.stdout:Generating ssh key... 2024-04-26T16:07:43.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:43 smithi050 bash[21463]: audit 2024-04-26T16:07:42.566078+0000 mgr.y (mgr.14120) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:43.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:43 smithi050 bash[21463]: audit 2024-04-26T16:07:42.566078+0000 mgr.y (mgr.14120) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21463]: audit 2024-04-26T16:07:44.504869+0000 mgr.y (mgr.14120) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21463]: audit 2024-04-26T16:07:44.504869+0000 mgr.y (mgr.14120) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21463]: cephadm 2024-04-26T16:07:44.505159+0000 mgr.y (mgr.14120) 11 : cephadm [INF] Generating ssh key... 2024-04-26T16:07:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21463]: cephadm 2024-04-26T16:07:44.505159+0000 mgr.y (mgr.14120) 11 : cephadm [INF] Generating ssh key... 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: Generating public/private rsa key pair. 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: Your identification has been saved in /tmp/tmph97qon4k/key. 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: Your public key has been saved in /tmp/tmph97qon4k/key.pub. 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: The key fingerprint is: 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: SHA256:P99Uqej0/STM29fQPxXeFAVf9DS6b7HYS39vf/R7ItA ceph-cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: The key's randomart image is: 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: +---[RSA 3072]----+ 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | .+*| 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | .++| 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | . +| 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | ..o| 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | S . ..=+| 2024-04-26T16:07:45.382 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | .. Eo*oB| 2024-04-26T16:07:45.383 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | o+ o=O*| 2024-04-26T16:07:45.383 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | oooo=*%| 2024-04-26T16:07:45.383 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: | ..oo*^| 2024-04-26T16:07:45.383 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:07:45 smithi050 bash[21717]: +----[SHA256]-----+ 2024-04-26T16:07:46.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 Processor -- start 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- start start 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5007bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca5007c120 con 0x7fca5007d6f0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca55523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5007bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca55523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5007bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35558/0 (socket says 172.21.15.50:35558) 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca55523700 1 -- 172.21.15.50:0/2590080548 learned_addr learned my addr 172.21.15.50:0/2590080548 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca55523700 1 -- 172.21.15.50:0/2590080548 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5007c2a0 con 0x7fca5007d6f0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca55523700 1 --2- 172.21.15.50:0/2590080548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5007bb50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca48009e80 tx=0x7fca4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d636ec5c86d71475 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca47fff700 1 -- 172.21.15.50:0/2590080548 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca48017070 con 0x7fca5007d6f0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca47fff700 1 -- 172.21.15.50:0/2590080548 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fca4800b8c0 con 0x7fca5007d6f0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca47fff700 1 -- 172.21.15.50:0/2590080548 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca48004340 con 0x7fca5007d6f0 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- 172.21.15.50:0/2590080548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 msgr2=0x7fca5007bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 --2- 172.21.15.50:0/2590080548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5007bb50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca48009e80 tx=0x7fca4800b330 comp rx=0 tx=0).stop 2024-04-26T16:07:46.017 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- 172.21.15.50:0/2590080548 shutdown_connections 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 --2- 172.21.15.50:0/2590080548 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5007bb50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- 172.21.15.50:0/2590080548 >> 172.21.15.50:0/2590080548 conn(0x7fca50102df0 msgr2=0x7fca50105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- 172.21.15.50:0/2590080548 shutdown_connections 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- 172.21.15.50:0/2590080548 wait complete. 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 librados: starting msgr at 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 librados: starting objecter 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 Processor -- start 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 -- start start 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 librados: setting wanted keys 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.098+0000 7fca57787700 1 librados: calling monclient init 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5019d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca5010dd40 con 0x7fca5007d6f0 2024-04-26T16:07:46.018 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca55523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5019d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca55523700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5019d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35560/0 (socket says 172.21.15.50:35560) 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca55523700 1 -- 172.21.15.50:0/1537195124 learned_addr learned my addr 172.21.15.50:0/1537195124 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca55523700 1 -- 172.21.15.50:0/1537195124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5019de00 con 0x7fca5007d6f0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca55523700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5019d8c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca48004000 tx=0x7fca48003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 10 client.?.objecter ms_handle_connect 0x7fca5007d6f0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca48020020 con 0x7fca5007d6f0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fca48003cf0 con 0x7fca5007d6f0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fca48017040 con 0x7fca5007d6f0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5019e090 con 0x7fca5007d6f0 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 10 client.14136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:46.019 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca501a0d80 con 0x7fca5007d6f0 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 1 librados: init done 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7fca48023020 con 0x7fca5007d6f0 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fca3c03bc80 0x7fca3c03e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fca4800f050 con 0x7fca5007d6f0 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 10 client.14136.objecter ms_dispatch 0x7fca50001550 osd_map(2..2 src has 1..2) 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 3 client.14136.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 3 client.14136.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 20 client.14136.objecter dump_active .. 0 homeless 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca50064a80 con 0x7fca5007d6f0 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca54d22700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fca3c03bc80 0x7fca3c03e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:46.020 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca54d22700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fca3c03bc80 0x7fca3c03e140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fca40006fd0 tx=0x7fca40006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.102+0000 7fca467fc700 10 client.14136.objecter ms_handle_connect 0x7fca3c03bc80 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.110+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fca4802a020 con 0x7fca5007d6f0 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:44.502+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fca50104090 con 0x7fca3c03bc80 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.382+0000 7fca467fc700 1 -- 172.21.15.50:0/1537195124 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fca50104090 con 0x7fca3c03bc80 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 10 librados: watch_flush enter 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 10 librados: watch_flush exit 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 20 client.14136.objecter shutdown clearing up homeless session... 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 10 client.14136.objecter successfully canceled tick 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fca3c03bc80 msgr2=0x7fca3c03e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:46.021 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fca3c03bc80 0x7fca3c03e140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fca40006fd0 tx=0x7fca40006e40 comp rx=0 tx=0).stop 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 msgr2=0x7fca5019d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5019d8c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca48004000 tx=0x7fca48003a20 comp rx=0 tx=0).stop 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 shutdown_connections 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fca3c03bc80 0x7fca3c03e140 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 --2- 172.21.15.50:0/1537195124 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca5007d6f0 0x7fca5019d8c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 >> 172.21.15.50:0/1537195124 conn(0x7fca50102df0 msgr2=0x7fca50103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 shutdown_connections 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 -- 172.21.15.50:0/1537195124 wait complete. 2024-04-26T16:07:46.022 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:45.386+0000 7fca57787700 1 librados: shutdown 2024-04-26T16:07:46.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:46 smithi050 bash[21463]: audit 2024-04-26T16:07:45.378309+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:46.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:46 smithi050 bash[21463]: audit 2024-04-26T16:07:45.378309+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:46.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:46 smithi050 bash[21463]: audit 2024-04-26T16:07:45.382646+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:46.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:46 smithi050 bash[21463]: audit 2024-04-26T16:07:45.382646+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:48.077 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfdK6ij82KFOTFigIcwNCT/REb6kbuPzHZV5Wgk8bjIYBa7xDfsxuewyv8C+/oziHDOaN+CuhBVjW1PGby2Q9+rXmOvMeHSQs9AtuSHualb5sPTAeARbTPZKaMiKmkrb7fz3lcKS+x/1La9ROw5AXo3IOpxhpNDe9nWp+XSrciu2zH1Z+ZYcFWbQwLC9PyKEVJ/7Ljvo45UMuXVmVxdE2ofTRdw2dnxVWxspEcWdC05nV2VbQogdKEPLw1Qb8Wqb6shZJdWKqIX+j5kJsfsur0t9tu5EU9kUv79OzSVG+xOHHy79LXj5s5auEqdoUUBsvcnggNeGW2Cv4q/9SEzx3sPYv3GN9MG3gerTHJ5hG2drO3/S9eEyAu5qxzQOqPIlO+8z1WrJRxdGKNV0Bv4RS8PRs0A5ggfNLhHQLyABB5Z4656/U3mrD6thvV22T+MzvS+AXKuUuwnn0aqfblWoT5U7YaRH+fyVt26N7+xFoYvP4Vtteqk/JV4Lea6215pkM= ceph-cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:07:48.077 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 Processor -- start 2024-04-26T16:07:48.077 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- start start 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa4106ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa4107170 con 0x7f8aa4106780 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa4106ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa4106ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35574/0 (socket says 172.21.15.50:35574) 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 -- 172.21.15.50:0/1271555097 learned_addr learned my addr 172.21.15.50:0/1271555097 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 -- 172.21.15.50:0/1271555097 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa4107980 con 0x7f8aa4106780 2024-04-26T16:07:48.078 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 --2- 172.21.15.50:0/1271555097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa4106ba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0098a0 tx=0x7f8a8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81c3b2eaa75f939 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa159a700 1 -- 172.21.15.50:0/1271555097 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a8c004160 con 0x7f8aa4106780 2024-04-26T16:07:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa159a700 1 -- 172.21.15.50:0/1271555097 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f8a8c004300 con 0x7f8aa4106780 2024-04-26T16:07:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa159a700 1 -- 172.21.15.50:0/1271555097 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a8c0118c0 con 0x7f8aa4106780 2024-04-26T16:07:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- 172.21.15.50:0/1271555097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 msgr2=0x7f8aa4106ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 --2- 172.21.15.50:0/1271555097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa4106ba0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0098a0 tx=0x7f8a8c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:48.079 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- 172.21.15.50:0/1271555097 shutdown_connections 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 --2- 172.21.15.50:0/1271555097 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa4106ba0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- 172.21.15.50:0/1271555097 >> 172.21.15.50:0/1271555097 conn(0x7f8aa4101ce0 msgr2=0x7f8aa4104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- 172.21.15.50:0/1271555097 shutdown_connections 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- 172.21.15.50:0/1271555097 wait complete. 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 librados: starting msgr at 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 librados: starting objecter 2024-04-26T16:07:48.080 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 Processor -- start 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- start start 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 librados: setting wanted keys 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 librados: calling monclient init 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa41917b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa8c80700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa410abb0 con 0x7f8aa4106780 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa41917b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa41917b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35586/0 (socket says 172.21.15.50:35586) 2024-04-26T16:07:48.081 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.026+0000 7f8aa259c700 1 -- 172.21.15.50:0/2182805258 learned_addr learned my addr 172.21.15.50:0/2182805258 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa259c700 1 -- 172.21.15.50:0/2182805258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa4191cf0 con 0x7f8aa4106780 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa259c700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa41917b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009870 tx=0x7f8a8c011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 10 client.?.objecter ms_handle_connect 0x7f8aa4106780 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a8c0036a0 con 0x7f8aa4106780 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f8a8c01a430 con 0x7f8aa4106780 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a8c0193c0 con 0x7f8aa4106780 2024-04-26T16:07:48.082 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa418fee0 con 0x7f8aa4106780 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa8c80700 10 client.14138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa4190420 con 0x7f8aa4106780 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa8c80700 1 librados: init done 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7f8a8c019670 con 0x7f8aa4106780 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f8a9003bc80 0x7f8a9003e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8a8c04e930 con 0x7f8aa4106780 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 10 client.14138.objecter ms_dispatch 0x7f8aa4001550 osd_map(2..2 src has 1..2) 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 3 client.14138.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 3 client.14138.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa4064a80 con 0x7f8aa4106780 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 20 client.14138.objecter dump_active .. 0 homeless 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa1d9b700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f8a9003bc80 0x7f8a9003e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8aa1d9b700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f8a9003bc80 0x7f8a9003e140 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8a94006fd0 tx=0x7f8a94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:48.083 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.030+0000 7f8a9b7fe700 10 client.14138.objecter ms_handle_connect 0x7f8a9003bc80 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.034+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f8a8c00fda0 con 0x7f8aa4106780 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.434+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f8aa4195340 con 0x7f8a9003bc80 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.434+0000 7f8a9b7fe700 1 -- 172.21.15.50:0/2182805258 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f8aa4195340 con 0x7f8a9003bc80 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 10 librados: watch_flush enter 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 10 librados: watch_flush exit 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 20 client.14138.objecter shutdown clearing up homeless session... 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 10 client.14138.objecter successfully canceled tick 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f8a9003bc80 msgr2=0x7f8a9003e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f8a9003bc80 0x7f8a9003e140 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8a94006fd0 tx=0x7f8a94006e40 comp rx=0 tx=0).stop 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 msgr2=0x7f8aa41917b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa41917b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009870 tx=0x7f8a8c011660 comp rx=0 tx=0).stop 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 shutdown_connections 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f8a9003bc80 0x7f8a9003e140 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:48.084 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 --2- 172.21.15.50:0/2182805258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8aa4106780 0x7f8aa41917b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 >> 172.21.15.50:0/2182805258 conn(0x7f8aa4101ce0 msgr2=0x7f8aa4102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 shutdown_connections 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 -- 172.21.15.50:0/2182805258 wait complete. 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:47.438+0000 7f8aa8c80700 1 librados: shutdown 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:Adding key to root@localhost authorized_keys... 2024-04-26T16:07:48.085 INFO:teuthology.orchestra.run.smithi050.stdout:Adding host smithi050... 2024-04-26T16:07:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:48 smithi050 bash[21463]: audit 2024-04-26T16:07:47.436949+0000 mgr.y (mgr.14120) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:48 smithi050 bash[21463]: audit 2024-04-26T16:07:47.436949+0000 mgr.y (mgr.14120) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:50.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:50 smithi050 bash[21463]: audit 2024-04-26T16:07:49.413789+0000 mgr.y (mgr.14120) 13 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:50.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:50 smithi050 bash[21463]: audit 2024-04-26T16:07:49.413789+0000 mgr.y (mgr.14120) 13 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:50.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:50 smithi050 bash[21463]: cephadm 2024-04-26T16:07:50.014084+0000 mgr.y (mgr.14120) 14 : cephadm [INF] Deploying cephadm binary to smithi050 2024-04-26T16:07:50.325 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:50 smithi050 bash[21463]: cephadm 2024-04-26T16:07:50.014084+0000 mgr.y (mgr.14120) 14 : cephadm [INF] Deploying cephadm binary to smithi050 2024-04-26T16:07:56.009 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Added host 'smithi050' with addr '172.21.15.50' 2024-04-26T16:07:56.009 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75e175a700 1 Processor -- start 2024-04-26T16:07:56.009 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75e175a700 1 -- start start 2024-04-26T16:07:56.009 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75e175a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:56.009 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75e175a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc109920 con 0x7f75dc106f60 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54612/0 (socket says 172.21.15.50:54612) 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75daffd700 1 -- 172.21.15.50:0/655593280 learned_addr learned my addr 172.21.15.50:0/655593280 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:48.998+0000 7f75daffd700 1 -- 172.21.15.50:0/655593280 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc109aa0 con 0x7f75dc106f60 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75daffd700 1 --2- 172.21.15.50:0/655593280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc109350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098a0 tx=0x7f75cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeab8a3dc5567048 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d9ffb700 1 -- 172.21.15.50:0/655593280 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75cc004160 con 0x7f75dc106f60 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d9ffb700 1 -- 172.21.15.50:0/655593280 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f75cc004300 con 0x7f75dc106f60 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d9ffb700 1 -- 172.21.15.50:0/655593280 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75cc0118c0 con 0x7f75dc106f60 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/655593280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 msgr2=0x7f75dc109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 --2- 172.21.15.50:0/655593280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc109350 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098a0 tx=0x7f75cc009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:56.010 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/655593280 shutdown_connections 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 --2- 172.21.15.50:0/655593280 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc109350 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/655593280 >> 172.21.15.50:0/655593280 conn(0x7f75dc101010 msgr2=0x7f75dc103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/655593280 shutdown_connections 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/655593280 wait complete. 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 librados: starting msgr at 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 librados: starting objecter 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 Processor -- start 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- start start 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 librados: setting wanted keys 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 librados: calling monclient init 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc19b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f75dc10a990 con 0x7f75dc106f60 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc19b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:56.011 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc19b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54616/0 (socket says 172.21.15.50:54616) 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75daffd700 1 -- 172.21.15.50:0/1686358828 learned_addr learned my addr 172.21.15.50:0/1686358828 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75daffd700 1 -- 172.21.15.50:0/1686358828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75dc19bc40 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75daffd700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc19b700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75cc006af0 tx=0x7f75cc011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d3fff700 10 client.?.objecter ms_handle_connect 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d3fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75cc0036a0 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7f75cc01a430 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75dc19bed0 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75cc0193c0 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 10 client.14140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75dc19ebc0 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.002+0000 7f75e175a700 1 librados: init done 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7f75cc003840 con 0x7f75dc106f60 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f75c803bc80 0x7f75c803e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:56.012 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75dc064a80 con 0x7f75dc106f60 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f75cc04e820 con 0x7f75dc106f60 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 10 client.14140.objecter ms_dispatch 0x7f75dc001550 osd_map(2..2 src has 1..2) 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 3 client.14140.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 3 client.14140.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 20 client.14140.objecter dump_active .. 0 homeless 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75da7fc700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f75c803bc80 0x7f75c803e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75d3fff700 10 client.14140.objecter ms_handle_connect 0x7f75c803bc80 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.006+0000 7f75da7fc700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f75c803bc80 0x7f75c803e140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f75c4006fd0 tx=0x7f75c4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.010+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f75cc02cc80 con 0x7f75dc106f60 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:49.410+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi050", "addr": "172.21.15.50", "target": ["mon-mgr", ""]}) -- 0x7f75dc02e5d0 con 0x7f75c803bc80 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:54.006+0000 7f75e0f59700 10 client.14140.objecter tick 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.122+0000 7f75d3fff700 1 -- 172.21.15.50:0/1686358828 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f75dc02e5d0 con 0x7f75c803bc80 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.122+0000 7f75e175a700 10 librados: watch_flush enter 2024-04-26T16:07:56.013 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.122+0000 7f75e175a700 10 librados: watch_flush exit 2024-04-26T16:07:56.014 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.122+0000 7f75e175a700 20 client.14140.objecter shutdown clearing up homeless session... 2024-04-26T16:07:56.014 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.122+0000 7f75e175a700 10 client.14140.objecter successfully canceled tick 2024-04-26T16:07:56.014 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f75c803bc80 msgr2=0x7f75c803e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:56.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f75c803bc80 0x7f75c803e140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f75c4006fd0 tx=0x7f75c4006e40 comp rx=0 tx=0).stop 2024-04-26T16:07:56.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 msgr2=0x7f75dc19b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:56.015 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc19b700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75cc006af0 tx=0x7f75cc011660 comp rx=0 tx=0).stop 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 shutdown_connections 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f75c803bc80 0x7f75c803e140 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 --2- 172.21.15.50:0/1686358828 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f75dc106f60 0x7f75dc19b700 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 >> 172.21.15.50:0/1686358828 conn(0x7f75dc101010 msgr2=0x7f75dc101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 shutdown_connections 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 -- 172.21.15.50:0/1686358828 wait complete. 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:55.126+0000 7f75e175a700 1 librados: shutdown 2024-04-26T16:07:56.016 INFO:teuthology.orchestra.run.smithi050.stdout:Deploying unmanaged mon service... 2024-04-26T16:07:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:56 smithi050 bash[21463]: audit 2024-04-26T16:07:55.123876+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:56 smithi050 bash[21463]: audit 2024-04-26T16:07:55.123876+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:56 smithi050 bash[21463]: cephadm 2024-04-26T16:07:55.124313+0000 mgr.y (mgr.14120) 15 : cephadm [INF] Added host smithi050 2024-04-26T16:07:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:56 smithi050 bash[21463]: cephadm 2024-04-26T16:07:55.124313+0000 mgr.y (mgr.14120) 15 : cephadm [INF] Added host smithi050 2024-04-26T16:07:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:56 smithi050 bash[21463]: audit 2024-04-26T16:07:55.172453+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:56.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:56 smithi050 bash[21463]: audit 2024-04-26T16:07:55.172453+0000 mon.a (mon.0) 63 : audit [DBG] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:07:58.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 Processor -- start 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- start start 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd674107cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd674108290 con 0x7fd6741078a0 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd674107cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd674107cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54630/0 (socket says 172.21.15.50:54630) 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 -- 172.21.15.50:0/3763804901 learned_addr learned my addr 172.21.15.50:0/3763804901 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 -- 172.21.15.50:0/3763804901 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd674108aa0 con 0x7fd6741078a0 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 --2- 172.21.15.50:0/3763804901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd674107cc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098a0 tx=0x7fd664009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d69fe0cfda5dfa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd671ffb700 1 -- 172.21.15.50:0/3763804901 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd664004160 con 0x7fd6741078a0 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd671ffb700 1 -- 172.21.15.50:0/3763804901 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fd664004300 con 0x7fd6741078a0 2024-04-26T16:07:58.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd671ffb700 1 -- 172.21.15.50:0/3763804901 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6640118c0 con 0x7fd6741078a0 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- 172.21.15.50:0/3763804901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 msgr2=0x7fd674107cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 --2- 172.21.15.50:0/3763804901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd674107cc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6640098a0 tx=0x7fd664009bb0 comp rx=0 tx=0).stop 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- 172.21.15.50:0/3763804901 shutdown_connections 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 --2- 172.21.15.50:0/3763804901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd674107cc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- 172.21.15.50:0/3763804901 >> 172.21.15.50:0/3763804901 conn(0x7fd674102e20 msgr2=0x7fd674105280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- 172.21.15.50:0/3763804901 shutdown_connections 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- 172.21.15.50:0/3763804901 wait complete. 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 librados: starting msgr at 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 librados: starting objecter 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 Processor -- start 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- start start 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 librados: setting wanted keys 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 librados: calling monclient init 2024-04-26T16:07:58.724 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd67419b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd6797f9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd67410bcd0 con 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd67419b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd67419b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54642/0 (socket says 172.21.15.50:54642) 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.474+0000 7fd672ffd700 1 -- 172.21.15.50:0/3361392399 learned_addr learned my addr 172.21.15.50:0/3361392399 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd672ffd700 1 -- 172.21.15.50:0/3361392399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd67419bca0 con 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd672ffd700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd67419b760 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd664009870 tx=0x7fd664011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 10 client.?.objecter ms_handle_connect 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6640036a0 con 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 985+0+0 (secure 0 0 0) 0x7fd664003c80 con 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd67419bf30 con 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd66401acd0 con 0x7fd6741078a0 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6797f9700 10 client.14142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:07:58.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd67419ec20 con 0x7fd6741078a0 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6797f9700 1 librados: init done 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd674064a80 con 0x7fd6741078a0 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7fd664003840 con 0x7fd6741078a0 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fd65c03bc80 0x7fd65c03e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd66404e560 con 0x7fd6741078a0 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 10 client.14142.objecter ms_dispatch 0x7fd674001550 osd_map(2..2 src has 1..2) 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 3 client.14142.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 3 client.14142.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 20 client.14142.objecter dump_active .. 0 homeless 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6727fc700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fd65c03bc80 0x7fd65c03e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd65bfff700 10 client.14142.objecter ms_handle_connect 0x7fd65c03bc80 2024-04-26T16:07:58.726 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.478+0000 7fd6727fc700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fd65c03bc80 0x7fd65c03e140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd668006fd0 tx=0x7fd668006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.486+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd664024c70 con 0x7fd6741078a0 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.906+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd674104130 con 0x7fd65c03bc80 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd65bfff700 1 -- 172.21.15.50:0/3361392399 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd674104130 con 0x7fd65c03bc80 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 10 librados: watch_flush enter 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 10 librados: watch_flush exit 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 20 client.14142.objecter shutdown clearing up homeless session... 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 10 client.14142.objecter successfully canceled tick 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fd65c03bc80 msgr2=0x7fd65c03e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fd65c03bc80 0x7fd65c03e140 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd668006fd0 tx=0x7fd668006e40 comp rx=0 tx=0).stop 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 msgr2=0x7fd67419b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd67419b760 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd664009870 tx=0x7fd664011660 comp rx=0 tx=0).stop 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 shutdown_connections 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fd65c03bc80 0x7fd65c03e140 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:58.727 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 --2- 172.21.15.50:0/3361392399 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6741078a0 0x7fd67419b760 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:07:58.728 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.918+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 >> 172.21.15.50:0/3361392399 conn(0x7fd674102e20 msgr2=0x7fd674103a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:07:58.728 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.922+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 shutdown_connections 2024-04-26T16:07:58.728 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.922+0000 7fd6797f9700 1 -- 172.21.15.50:0/3361392399 wait complete. 2024-04-26T16:07:58.728 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:57.922+0000 7fd6797f9700 1 librados: shutdown 2024-04-26T16:07:58.728 INFO:teuthology.orchestra.run.smithi050.stdout:Deploying unmanaged mgr service... 2024-04-26T16:07:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: cluster 2024-04-26T16:07:57.717052+0000 mgr.y (mgr.14120) 16 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: cluster 2024-04-26T16:07:57.717052+0000 mgr.y (mgr.14120) 16 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: audit 2024-04-26T16:07:57.913036+0000 mgr.y (mgr.14120) 17 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: audit 2024-04-26T16:07:57.913036+0000 mgr.y (mgr.14120) 17 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: cephadm 2024-04-26T16:07:57.914480+0000 mgr.y (mgr.14120) 18 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: cephadm 2024-04-26T16:07:57.914480+0000 mgr.y (mgr.14120) 18 : cephadm [INF] Saving service mon spec with placement count:5 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: audit 2024-04-26T16:07:57.919238+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: audit 2024-04-26T16:07:57.919238+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: audit 2024-04-26T16:07:58.499910+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:07:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:58 smithi050 bash[21463]: audit 2024-04-26T16:07:58.499910+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:08:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:59 smithi050 bash[21463]: audit 2024-04-26T16:07:58.995956+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:08:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:59 smithi050 bash[21463]: audit 2024-04-26T16:07:58.995956+0000 mon.a (mon.0) 66 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:08:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:59 smithi050 bash[21463]: audit 2024-04-26T16:07:59.989559+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:08:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:07:59 smithi050 bash[21463]: audit 2024-04-26T16:07:59.989559+0000 mon.a (mon.0) 67 : audit [INF] from='mgr.14120 172.21.15.50:0/122528113' entity='mgr.y' 2024-04-26T16:08:00.637 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-26T16:08:00.637 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb364fee700 1 Processor -- start 2024-04-26T16:08:00.637 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb364fee700 1 -- start start 2024-04-26T16:08:00.637 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb364fee700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb360110b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb364fee700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb360111040 con 0x7fb36010e710 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb35e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb360110b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb35e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb360110b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59278/0 (socket says 172.21.15.50:59278) 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb35e59c700 1 -- 172.21.15.50:0/319699572 learned_addr learned my addr 172.21.15.50:0/319699572 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.546+0000 7fb35e59c700 1 -- 172.21.15.50:0/319699572 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3601111c0 con 0x7fb36010e710 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35e59c700 1 --2- 172.21.15.50:0/319699572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb360110b00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb3480098a0 tx=0x7fb348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832efc4776afd546 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35d59a700 1 -- 172.21.15.50:0/319699572 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb348004160 con 0x7fb36010e710 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35d59a700 1 -- 172.21.15.50:0/319699572 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb348004300 con 0x7fb36010e710 2024-04-26T16:08:00.638 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35d59a700 1 -- 172.21.15.50:0/319699572 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb348011870 con 0x7fb36010e710 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/319699572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 msgr2=0x7fb360110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 --2- 172.21.15.50:0/319699572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb360110b00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb3480098a0 tx=0x7fb348009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/319699572 shutdown_connections 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 --2- 172.21.15.50:0/319699572 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb360110b00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/319699572 >> 172.21.15.50:0/319699572 conn(0x7fb36007ce80 msgr2=0x7fb36007f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/319699572 shutdown_connections 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/319699572 wait complete. 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 librados: starting msgr at 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 librados: starting objecter 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 Processor -- start 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- start start 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 librados: setting wanted keys 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 librados: calling monclient init 2024-04-26T16:08:00.639 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb3601a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb3601121c0 con 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb3601a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb3601a1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59292/0 (socket says 172.21.15.50:59292) 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35e59c700 1 -- 172.21.15.50:0/2805099680 learned_addr learned my addr 172.21.15.50:0/2805099680 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35e59c700 1 -- 172.21.15.50:0/2805099680 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3601a2280 con 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb35e59c700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb3601a1d40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb348006af0 tx=0x7fb348011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb3577fe700 10 client.?.objecter ms_handle_connect 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb3577fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3480036a0 con 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb34801a430 con 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3601a2510 con 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb3480193c0 con 0x7fb36010e710 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 10 client.14144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:00.640 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3601a5200 con 0x7fb36010e710 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb364fee700 1 librados: init done 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.550+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7fb348019630 con 0x7fb36010e710 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb34c03bc80 0x7fb34c03e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb360064a80 con 0x7fb36010e710 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb34804e590 con 0x7fb36010e710 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 10 client.14144.objecter ms_dispatch 0x7fb360001550 osd_map(2..2 src has 1..2) 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 3 client.14144.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 3 client.14144.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 20 client.14144.objecter dump_active .. 0 homeless 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb35dd9b700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb34c03bc80 0x7fb34c03e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb3577fe700 10 client.14144.objecter ms_handle_connect 0x7fb34c03bc80 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.554+0000 7fb35dd9b700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb34c03bc80 0x7fb34c03e140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb350006fd0 tx=0x7fb350006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.558+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb34801a5e0 con 0x7fb36010e710 2024-04-26T16:08:00.641 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.978+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fb36007e110 con 0x7fb34c03bc80 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.986+0000 7fb3577fe700 1 -- 172.21.15.50:0/2805099680 <== mgr.14120 v2:172.21.15.50:6800/685732569 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb36007e110 con 0x7fb34c03bc80 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 10 librados: watch_flush enter 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 10 librados: watch_flush exit 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 20 client.14144.objecter shutdown clearing up homeless session... 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 10 client.14144.objecter successfully canceled tick 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb34c03bc80 msgr2=0x7fb34c03e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb34c03bc80 0x7fb34c03e140 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb350006fd0 tx=0x7fb350006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 msgr2=0x7fb3601a1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb3601a1d40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb348006af0 tx=0x7fb348011ea0 comp rx=0 tx=0).stop 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 shutdown_connections 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fb34c03bc80 0x7fb34c03e140 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 --2- 172.21.15.50:0/2805099680 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb36010e710 0x7fb3601a1d40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 >> 172.21.15.50:0/2805099680 conn(0x7fb36007ce80 msgr2=0x7fb36007da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:00.642 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 shutdown_connections 2024-04-26T16:08:00.643 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 -- 172.21.15.50:0/2805099680 wait complete. 2024-04-26T16:08:00.643 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:07:59.990+0000 7fb364fee700 1 librados: shutdown 2024-04-26T16:08:01.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:00 smithi050 bash[21463]: cluster 2024-04-26T16:07:59.717407+0000 mgr.y (mgr.14120) 19 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:01.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:00 smithi050 bash[21463]: cluster 2024-04-26T16:07:59.717407+0000 mgr.y (mgr.14120) 19 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:01.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:00 smithi050 bash[21463]: audit 2024-04-26T16:07:59.983800+0000 mgr.y (mgr.14120) 20 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:01.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:00 smithi050 bash[21463]: audit 2024-04-26T16:07:59.983800+0000 mgr.y (mgr.14120) 20 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:01.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:00 smithi050 bash[21463]: cephadm 2024-04-26T16:07:59.986559+0000 mgr.y (mgr.14120) 21 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-26T16:08:01.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:00 smithi050 bash[21463]: cephadm 2024-04-26T16:07:59.986559+0000 mgr.y (mgr.14120) 21 : cephadm [INF] Saving service mgr spec with placement count:2 2024-04-26T16:08:02.541 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f3bc5700 1 Processor -- start 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f3bc5700 1 -- start start 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f3bc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec109ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f3bc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec10a4b0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f1961700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec109ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f1961700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec109ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59306/0 (socket says 172.21.15.50:59306) 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f1961700 1 -- 172.21.15.50:0/380146416 learned_addr learned my addr 172.21.15.50:0/380146416 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.534+0000 7fa9f1961700 1 -- 172.21.15.50:0/380146416 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec10acc0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f1961700 1 --2- 172.21.15.50:0/380146416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec109ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098a0 tx=0x7fa9e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e99481a8868b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f095f700 1 -- 172.21.15.50:0/380146416 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9e8004160 con 0x7fa9ec109ac0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f095f700 1 -- 172.21.15.50:0/380146416 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa9e8004300 con 0x7fa9ec109ac0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f095f700 1 -- 172.21.15.50:0/380146416 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9e8011870 con 0x7fa9ec109ac0 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/380146416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 msgr2=0x7fa9ec109ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:02.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 --2- 172.21.15.50:0/380146416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec109ee0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9e80098a0 tx=0x7fa9e8009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/380146416 shutdown_connections 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 --2- 172.21.15.50:0/380146416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec109ee0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/380146416 >> 172.21.15.50:0/380146416 conn(0x7fa9ec07ce80 msgr2=0x7fa9ec07d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/380146416 shutdown_connections 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/380146416 wait complete. 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 librados: starting msgr at 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 librados: starting objecter 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 Processor -- start 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- start start 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 librados: setting wanted keys 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 librados: calling monclient init 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ec10def0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f1961700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f1961700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec19d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59310/0 (socket says 172.21.15.50:59310) 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f1961700 1 -- 172.21.15.50:0/1356686746 learned_addr learned my addr 172.21.15.50:0/1356686746 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f1961700 1 -- 172.21.15.50:0/1356686746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ec19df20 con 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f1961700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec19d9e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8003ee0 tx=0x7fa9e8011960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9e2ffd700 10 client.?.objecter ms_handle_connect 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9e2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9e80036a0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ec19e1b0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa9e801a430 con 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa9e80193c0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 10 client.14146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ec1a0ea0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.538+0000 7fa9f3bc5700 1 librados: init done 2024-04-26T16:08:02.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7fa9e80196c0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9ec064a80 con 0x7fa9ec109ac0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fa9d803bc80 0x7fa9d803e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa9e804e820 con 0x7fa9ec109ac0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 10 client.14146.objecter ms_dispatch 0x7fa9ec001550 osd_map(2..2 src has 1..2) 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 3 client.14146.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 3 client.14146.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 20 client.14146.objecter dump_active .. 0 homeless 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9f1160700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fa9d803bc80 0x7fa9d803e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9e2ffd700 10 client.14146.objecter ms_handle_connect 0x7fa9d803bc80 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.542+0000 7fa9f1160700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fa9d803bc80 0x7fa9d803e140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc006fd0 tx=0x7fa9dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.550+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fa9e800f2e0 con 0x7fa9ec109ac0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.942+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fa9ec1a1280 con 0x7fa9ec109ac0 2024-04-26T16:08:02.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.946+0000 7fa9e2ffd700 1 -- 172.21.15.50:0/1356686746 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fa9e804e170 con 0x7fa9ec109ac0 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 10 librados: watch_flush enter 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 10 librados: watch_flush exit 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 20 client.14146.objecter shutdown clearing up homeless session... 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 10 client.14146.objecter successfully canceled tick 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fa9d803bc80 msgr2=0x7fa9d803e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fa9d803bc80 0x7fa9d803e140 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa9dc006fd0 tx=0x7fa9dc006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 msgr2=0x7fa9ec19d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec19d9e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8003ee0 tx=0x7fa9e8011960 comp rx=0 tx=0).stop 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 shutdown_connections 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fa9d803bc80 0x7fa9d803e140 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 --2- 172.21.15.50:0/1356686746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9ec109ac0 0x7fa9ec19d9e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 >> 172.21.15.50:0/1356686746 conn(0x7fa9ec07ce80 msgr2=0x7fa9ec106de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 shutdown_connections 2024-04-26T16:08:02.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 -- 172.21.15.50:0/1356686746 wait complete. 2024-04-26T16:08:02.547 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:01.950+0000 7fa9f3bc5700 1 librados: shutdown 2024-04-26T16:08:03.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:02 smithi050 bash[21463]: cluster 2024-04-26T16:08:01.717717+0000 mgr.y (mgr.14120) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:03.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:02 smithi050 bash[21463]: cluster 2024-04-26T16:08:01.717717+0000 mgr.y (mgr.14120) 22 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:03.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:02 smithi050 bash[21463]: audit 2024-04-26T16:08:01.950342+0000 mon.a (mon.0) 68 : audit [INF] from='client.? 172.21.15.50:0/1356686746' entity='client.admin' 2024-04-26T16:08:03.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:02 smithi050 bash[21463]: audit 2024-04-26T16:08:01.950342+0000 mon.a (mon.0) 68 : audit [INF] from='client.? 172.21.15.50:0/1356686746' entity='client.admin' 2024-04-26T16:08:04.447 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc8177b700 1 Processor -- start 2024-04-26T16:08:04.447 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc8177b700 1 -- start start 2024-04-26T16:08:04.447 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc8177b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c10a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:04.447 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc8177b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c07b6e0 con 0x7fbc7c108330 2024-04-26T16:08:04.447 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c10a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c10a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59326/0 (socket says 172.21.15.50:59326) 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc7affd700 1 -- 172.21.15.50:0/3837813970 learned_addr learned my addr 172.21.15.50:0/3837813970 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc7affd700 1 -- 172.21.15.50:0/3837813970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c10ac60 con 0x7fbc7c108330 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc7affd700 1 --2- 172.21.15.50:0/3837813970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c10a720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0098a0 tx=0x7fbc6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f33a1131063379 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc79ffb700 1 -- 172.21.15.50:0/3837813970 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc6c004160 con 0x7fbc7c108330 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc79ffb700 1 -- 172.21.15.50:0/3837813970 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fbc6c004300 con 0x7fbc7c108330 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.434+0000 7fbc79ffb700 1 -- 172.21.15.50:0/3837813970 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc6c011870 con 0x7fbc7c108330 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/3837813970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 msgr2=0x7fbc7c10a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 --2- 172.21.15.50:0/3837813970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c10a720 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0098a0 tx=0x7fbc6c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/3837813970 shutdown_connections 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 --2- 172.21.15.50:0/3837813970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c10a720 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/3837813970 >> 172.21.15.50:0/3837813970 conn(0x7fbc7c101ce0 msgr2=0x7fbc7c104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/3837813970 shutdown_connections 2024-04-26T16:08:04.448 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/3837813970 wait complete. 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 librados: starting msgr at 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 librados: starting objecter 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 Processor -- start 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- start start 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 librados: setting wanted keys 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 librados: calling monclient init 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c107bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c10bba0 con 0x7fbc7c108330 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c107bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc7affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c107bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59336/0 (socket says 172.21.15.50:59336) 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc7affd700 1 -- 172.21.15.50:0/2029128195 learned_addr learned my addr 172.21.15.50:0/2029128195 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc7affd700 1 -- 172.21.15.50:0/2029128195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c1080f0 con 0x7fbc7c108330 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc7affd700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c107bb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c006af0 tx=0x7fbc6c011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 10 client.?.objecter ms_handle_connect 0x7fbc7c108330 2024-04-26T16:08:04.449 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc6c0036a0 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fbc6c01a430 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc6c0193c0 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c1062e0 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 10 client.14148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c106820 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 librados: init done 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7fbc6c003840 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c064a80 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fbc6803bc80 0x7fbc6803e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbc6c04e5f0 con 0x7fbc7c108330 2024-04-26T16:08:04.450 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 10 client.14148.objecter ms_dispatch 0x7fbc7c001550 osd_map(2..2 src has 1..2) 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 3 client.14148.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.438+0000 7fbc73fff700 3 client.14148.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.442+0000 7fbc73fff700 20 client.14148.objecter dump_active .. 0 homeless 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.442+0000 7fbc7a7fc700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fbc6803bc80 0x7fbc6803e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.442+0000 7fbc7a7fc700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fbc6803bc80 0x7fbc6803e140 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbc64006fd0 tx=0x7fbc64006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.442+0000 7fbc73fff700 10 client.14148.objecter ms_handle_connect 0x7fbc6803bc80 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.446+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fbc6c02a2b0 con 0x7fbc7c108330 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.830+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fbc7c106f80 con 0x7fbc7c108330 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc73fff700 1 -- 172.21.15.50:0/2029128195 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fbc6c003b30 con 0x7fbc7c108330 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 10 librados: watch_flush enter 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 10 librados: watch_flush exit 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 20 client.14148.objecter shutdown clearing up homeless session... 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 10 client.14148.objecter successfully canceled tick 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fbc6803bc80 msgr2=0x7fbc6803e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:04.451 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fbc6803bc80 0x7fbc6803e140 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbc64006fd0 tx=0x7fbc64006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 msgr2=0x7fbc7c107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c107bb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c006af0 tx=0x7fbc6c011ea0 comp rx=0 tx=0).stop 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 shutdown_connections 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7fbc6803bc80 0x7fbc6803e140 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 --2- 172.21.15.50:0/2029128195 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbc7c108330 0x7fbc7c107bb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 >> 172.21.15.50:0/2029128195 conn(0x7fbc7c101ce0 msgr2=0x7fbc7c102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 shutdown_connections 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 -- 172.21.15.50:0/2029128195 wait complete. 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:03.838+0000 7fbc8177b700 1 librados: shutdown 2024-04-26T16:08:04.452 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling the dashboard module... 2024-04-26T16:08:05.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:04 smithi050 bash[21463]: cluster 2024-04-26T16:08:03.718075+0000 mgr.y (mgr.14120) 23 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:05.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:04 smithi050 bash[21463]: cluster 2024-04-26T16:08:03.718075+0000 mgr.y (mgr.14120) 23 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:05.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:04 smithi050 bash[21463]: audit 2024-04-26T16:08:03.839310+0000 mon.a (mon.0) 69 : audit [INF] from='client.? 172.21.15.50:0/2029128195' entity='client.admin' 2024-04-26T16:08:05.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:04 smithi050 bash[21463]: audit 2024-04-26T16:08:03.839310+0000 mon.a (mon.0) 69 : audit [INF] from='client.? 172.21.15.50:0/2029128195' entity='client.admin' 2024-04-26T16:08:06.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:05 smithi050 bash[21463]: audit 2024-04-26T16:08:05.834650+0000 mon.a (mon.0) 70 : audit [INF] from='client.? 172.21.15.50:0/332742340' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T16:08:06.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:05 smithi050 bash[21463]: audit 2024-04-26T16:08:05.834650+0000 mon.a (mon.0) 70 : audit [INF] from='client.? 172.21.15.50:0/332742340' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-26T16:08:06.134 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:05 smithi050 bash[21717]: ignoring --setuser ceph since I am not root 2024-04-26T16:08:06.134 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:05 smithi050 bash[21717]: ignoring --setgroup ceph since I am not root 2024-04-26T16:08:06.134 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:06 smithi050 bash[21717]: debug 2024-04-26T16:08:06.042+0000 7f66bbf07200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.294+0000 7ff52c630700 1 Processor -- start 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- start start 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff524107d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5241082f0 con 0x7ff524107900 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff524107d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff524107d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59338/0 (socket says 172.21.15.50:59338) 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 -- 172.21.15.50:0/621074684 learned_addr learned my addr 172.21.15.50:0/621074684 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:06.477 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 -- 172.21.15.50:0/621074684 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff524108b00 con 0x7ff524107900 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 --2- 172.21.15.50:0/621074684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff524107d20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5200098a0 tx=0x7ff520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42f241e1d1bebaf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff5293ca700 1 -- 172.21.15.50:0/621074684 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff520004160 con 0x7ff524107900 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff5293ca700 1 -- 172.21.15.50:0/621074684 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff520004300 con 0x7ff524107900 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff5293ca700 1 -- 172.21.15.50:0/621074684 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff520011870 con 0x7ff524107900 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- 172.21.15.50:0/621074684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 msgr2=0x7ff524107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 --2- 172.21.15.50:0/621074684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff524107d20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5200098a0 tx=0x7ff520009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- 172.21.15.50:0/621074684 shutdown_connections 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 --2- 172.21.15.50:0/621074684 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff524107d20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- 172.21.15.50:0/621074684 >> 172.21.15.50:0/621074684 conn(0x7ff524102e60 msgr2=0x7ff5241052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- 172.21.15.50:0/621074684 shutdown_connections 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- 172.21.15.50:0/621074684 wait complete. 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 librados: starting msgr at 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 librados: starting objecter 2024-04-26T16:08:06.478 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 Processor -- start 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- start start 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 librados: setting wanted keys 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 librados: calling monclient init 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff52419d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52c630700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff52410bd30 con 0x7ff524107900 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff52419d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff52419d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59342/0 (socket says 172.21.15.50:59342) 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 -- 172.21.15.50:0/332742340 learned_addr learned my addr 172.21.15.50:0/332742340 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 -- 172.21.15.50:0/332742340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff52419de60 con 0x7ff524107900 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff52a3cc700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff52419d920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff520006af0 tx=0x7ff520011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:06.479 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.298+0000 7ff51b7fe700 10 client.?.objecter ms_handle_connect 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff520011970 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff520003b00 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff52001acf0 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff52419e0f0 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff52c630700 10 client.14150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5241a0de0 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff52c630700 1 librados: init done 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 8) ==== 47177+0+0 (secure 0 0 0) 0x7ff52001a5c0 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff524064a80 con 0x7ff524107900 2024-04-26T16:08:06.481 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7ff51003bdb0 0x7ff51003e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff52004e990 con 0x7ff524107900 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 10 client.14150.objecter ms_dispatch 0x7ff524001550 osd_map(2..2 src has 1..2) 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 3 client.14150.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 3 client.14150.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 20 client.14150.objecter dump_active .. 0 homeless 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff529bcb700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7ff51003bdb0 0x7ff51003e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff529bcb700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7ff51003bdb0 0x7ff51003e270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff51400ad30 tx=0x7ff5140093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.302+0000 7ff51b7fe700 10 client.14150.objecter ms_handle_connect 0x7ff51003bdb0 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.306+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff5200172b0 con 0x7ff524107900 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.830+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7ff5241a11c0 con 0x7ff524107900 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.838+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7ff52004e2e0 con 0x7ff524107900 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff51b7fe700 1 -- 172.21.15.50:0/332742340 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mgrmap(e 9) ==== 47190+0+0 (secure 0 0 0) 0x7ff52002d4e0 con 0x7ff524107900 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 10 librados: watch_flush enter 2024-04-26T16:08:06.482 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 10 librados: watch_flush exit 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 20 client.14150.objecter shutdown clearing up homeless session... 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 10 client.14150.objecter successfully canceled tick 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7ff51003bdb0 msgr2=0x7ff51003e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7ff51003bdb0 0x7ff51003e270 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff51400ad30 tx=0x7ff5140093f0 comp rx=0 tx=0).stop 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 msgr2=0x7ff52419d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff52419d920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff520006af0 tx=0x7ff520011f90 comp rx=0 tx=0).stop 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 shutdown_connections 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7ff51003bdb0 0x7ff51003e270 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 --2- 172.21.15.50:0/332742340 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff524107900 0x7ff52419d920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 >> 172.21.15.50:0/332742340 conn(0x7ff524102e60 msgr2=0x7ff524103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 shutdown_connections 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 -- 172.21.15.50:0/332742340 wait complete. 2024-04-26T16:08:06.483 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:05.842+0000 7ff52c630700 1 librados: shutdown 2024-04-26T16:08:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:06 smithi050 bash[21463]: audit 2024-04-26T16:08:05.842787+0000 mon.a (mon.0) 71 : audit [INF] from='client.? 172.21.15.50:0/332742340' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T16:08:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:06 smithi050 bash[21463]: audit 2024-04-26T16:08:05.842787+0000 mon.a (mon.0) 71 : audit [INF] from='client.? 172.21.15.50:0/332742340' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-26T16:08:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:06 smithi050 bash[21463]: cluster 2024-04-26T16:08:05.846095+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e9: y(active, since 28s) 2024-04-26T16:08:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:06 smithi050 bash[21463]: cluster 2024-04-26T16:08:05.846095+0000 mon.a (mon.0) 72 : cluster [DBG] mgrmap e9: y(active, since 28s) 2024-04-26T16:08:08.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:07 smithi050 bash[21463]: audit 2024-04-26T16:08:07.872551+0000 mon.a (mon.0) 73 : audit [DBG] from='client.? 172.21.15.50:0/1107966389' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T16:08:08.048 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:07 smithi050 bash[21463]: audit 2024-04-26T16:08:07.872551+0000 mon.a (mon.0) 73 : audit [DBG] from='client.? 172.21.15.50:0/1107966389' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-26T16:08:08.048 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:07 smithi050 bash[21717]: debug 2024-04-26T16:08:07.690+0000 7f66bbf07200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T16:08:08.049 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:07 smithi050 bash[21717]: debug 2024-04-26T16:08:07.790+0000 7f66bbf07200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T16:08:08.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:08 smithi050 bash[21717]: debug 2024-04-26T16:08:08.050+0000 7f66bbf07200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T16:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:08:08.534 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 Processor -- start 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- start start 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a441065c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4407b6c0 con 0x7f2a441061a0 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a497cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a441065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:08.535 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a497cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a441065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59370/0 (socket says 172.21.15.50:59370) 2024-04-26T16:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a497cc700 1 -- 172.21.15.50:0/1983493487 learned_addr learned my addr 172.21.15.50:0/1983493487 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a497cc700 1 -- 172.21.15.50:0/1983493487 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a44106b00 con 0x7f2a441061a0 2024-04-26T16:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a497cc700 1 --2- 172.21.15.50:0/1983493487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a441065c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2a400098a0 tx=0x7f2a40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7678da3e16d5b42 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a3bfff700 1 -- 172.21.15.50:0/1983493487 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40004160 con 0x7f2a441061a0 2024-04-26T16:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a3bfff700 1 -- 172.21.15.50:0/1983493487 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2a40004300 con 0x7f2a441061a0 2024-04-26T16:08:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a3bfff700 1 -- 172.21.15.50:0/1983493487 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a40011870 con 0x7f2a441061a0 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1983493487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 msgr2=0x7f2a441065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 --2- 172.21.15.50:0/1983493487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a441065c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2a400098a0 tx=0x7f2a40009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1983493487 shutdown_connections 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 --2- 172.21.15.50:0/1983493487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a441065c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1983493487 >> 172.21.15.50:0/1983493487 conn(0x7f2a44101d40 msgr2=0x7f2a441041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1983493487 shutdown_connections 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1983493487 wait complete. 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 librados: starting msgr at 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 librados: starting objecter 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 Processor -- start 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 -- start start 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 librados: setting wanted keys 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.354+0000 7f2a4ba30700 1 librados: calling monclient init 2024-04-26T16:08:08.543 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a4419d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2a44107ab0 con 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a497cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a4419d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a497cc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a4419d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:59384/0 (socket says 172.21.15.50:59384) 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a497cc700 1 -- 172.21.15.50:0/1107966389 learned_addr learned my addr 172.21.15.50:0/1107966389 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a497cc700 1 -- 172.21.15.50:0/1107966389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4419ddc0 con 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a497cc700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a4419d880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a40006af0 tx=0x7f2a40011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 10 client.?.objecter ms_handle_connect 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a400036a0 con 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4419e050 con 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2a4001a430 con 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a400193c0 con 0x7f2a441061a0 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 10 client.14154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:08.544 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a441a0d40 con 0x7f2a441061a0 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 1 librados: init done 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) ==== 47190+0+0 (secure 0 0 0) 0x7f2a40003840 con 0x7f2a441061a0 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a44064a80 con 0x7f2a441061a0 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 0x7f2a3003e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f2a4004e6c0 con 0x7f2a441061a0 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 10 client.14154.objecter ms_dispatch 0x7f2a44001550 osd_map(2..2 src has 1..2) 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 3 client.14154.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 3 client.14154.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a48fcb700 1 -- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 msgr2=0x7f2a3003e140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a48fcb700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 0x7f2a3003e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.358+0000 7f2a3a7fc700 20 client.14154.objecter dump_active .. 0 homeless 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.362+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f2a4000be50 con 0x7f2a441061a0 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.562+0000 7f2a48fcb700 1 -- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 msgr2=0x7f2a3003e140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.562+0000 7f2a48fcb700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 0x7f2a3003e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T16:08:08.545 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.870+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f2a4402e530 con 0x7f2a441061a0 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.870+0000 7f2a3a7fc700 1 -- 172.21.15.50:0/1107966389 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f2a4002ad70 con 0x7f2a441061a0 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.870+0000 7f2a4ba30700 10 librados: watch_flush enter 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 10 librados: watch_flush exit 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 20 client.14154.objecter shutdown clearing up homeless session... 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 10 client.14154.objecter successfully canceled tick 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 msgr2=0x7f2a3003e140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 0x7f2a3003e140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 msgr2=0x7f2a4419d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a4419d880 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a40006af0 tx=0x7f2a40011ea0 comp rx=0 tx=0).stop 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 shutdown_connections 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f2a3003bc80 0x7f2a3003e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 --2- 172.21.15.50:0/1107966389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2a441061a0 0x7f2a4419d880 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 >> 172.21.15.50:0/1107966389 conn(0x7f2a44101d40 msgr2=0x7f2a44194560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:08.546 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 shutdown_connections 2024-04-26T16:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 -- 172.21.15.50:0/1107966389 wait complete. 2024-04-26T16:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:07.874+0000 7f2a4ba30700 1 librados: shutdown 2024-04-26T16:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for the mgr to restart... 2024-04-26T16:08:08.547 INFO:teuthology.orchestra.run.smithi050.stdout:Waiting for mgr epoch 9... 2024-04-26T16:08:09.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:08 smithi050 bash[21717]: debug 2024-04-26T16:08:08.882+0000 7f66bbf07200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T16:08:09.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:08 smithi050 bash[21717]: debug 2024-04-26T16:08:08.994+0000 7f66bbf07200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T16:08:09.131 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:09 smithi050 bash[21717]: debug 2024-04-26T16:08:09.102+0000 7f66bbf07200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T16:08:09.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:09 smithi050 bash[21717]: debug 2024-04-26T16:08:09.478+0000 7f66bbf07200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T16:08:09.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:09 smithi050 bash[21717]: debug 2024-04-26T16:08:09.594+0000 7f66bbf07200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T16:08:09.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:09 smithi050 bash[21717]: debug 2024-04-26T16:08:09.702+0000 7f66bbf07200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T16:08:10.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:10 smithi050 bash[21717]: debug 2024-04-26T16:08:10.082+0000 7f66bbf07200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T16:08:10.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:10 smithi050 bash[21717]: debug 2024-04-26T16:08:10.326+0000 7f66bbf07200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T16:08:10.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:10 smithi050 bash[21717]: debug 2024-04-26T16:08:10.602+0000 7f66bbf07200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T16:08:10.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:10 smithi050 bash[21717]: debug 2024-04-26T16:08:10.722+0000 7f66bbf07200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T16:08:11.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:11 smithi050 bash[21717]: debug 2024-04-26T16:08:11.006+0000 7f66bbf07200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T16:08:11.682 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:11 smithi050 bash[21717]: debug 2024-04-26T16:08:11.430+0000 7f66bbf07200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:08:12.068 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:11 smithi050 bash[21717]: debug 2024-04-26T16:08:11.682+0000 7f66bbf07200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T16:08:12.069 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:11 smithi050 bash[21717]: debug 2024-04-26T16:08:11.806+0000 7f66bbf07200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T16:08:12.069 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:12 smithi050 bash[21717]: debug 2024-04-26T16:08:12.070+0000 7f66bbf07200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:08:13.380 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:13 smithi050 bash[21717]: debug 2024-04-26T16:08:13.118+0000 7f66bbf07200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T16:08:13.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:13 smithi050 bash[21717]: debug 2024-04-26T16:08:13.238+0000 7f66bbf07200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T16:08:13.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:13 smithi050 bash[21717]: debug 2024-04-26T16:08:13.550+0000 7f66bbf07200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T16:08:13.881 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:13 smithi050 bash[21717]: debug 2024-04-26T16:08:13.670+0000 7f66bbf07200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T16:08:14.381 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:13 smithi050 bash[21717]: debug 2024-04-26T16:08:13.966+0000 7f66bbf07200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21717]: debug 2024-04-26T16:08:14.442+0000 7f66bbf07200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21717]: debug 2024-04-26T16:08:14.566+0000 7f66bbf07200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.593090+0000 mon.a (mon.0) 74 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.593090+0000 mon.a (mon.0) 74 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.593568+0000 mon.a (mon.0) 75 : cluster [INF] Activating manager daemon y 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.593568+0000 mon.a (mon.0) 75 : cluster [INF] Activating manager daemon y 2024-04-26T16:08:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.597265+0000 mon.a (mon.0) 76 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.597265+0000 mon.a (mon.0) 76 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.597658+0000 mon.a (mon.0) 77 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00429351s) 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.597658+0000 mon.a (mon.0) 77 : cluster [DBG] mgrmap e10: y(active, starting, since 0.00429351s) 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.599681+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.599681+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.602165+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.602165+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.603383+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.603383+0000 mon.a (mon.0) 80 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.603565+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.603565+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.603746+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:08:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: audit 2024-04-26T16:08:14.603746+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:08:14.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.629352+0000 mon.a (mon.0) 83 : cluster [INF] Manager daemon y is now available 2024-04-26T16:08:14.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:14 smithi050 bash[21463]: cluster 2024-04-26T16:08:14.629352+0000 mon.a (mon.0) 83 : cluster [INF] Manager daemon y is now available 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: audit 2024-04-26T16:08:14.654991+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: audit 2024-04-26T16:08:14.654991+0000 mon.a (mon.0) 84 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: audit 2024-04-26T16:08:14.677609+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: audit 2024-04-26T16:08:14.677609+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: audit 2024-04-26T16:08:14.681714+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: audit 2024-04-26T16:08:14.681714+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: cluster 2024-04-26T16:08:15.605668+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e11: y(active, since 1.0123s) 2024-04-26T16:08:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:15 smithi050 bash[21463]: cluster 2024-04-26T16:08:15.605668+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e11: y(active, since 1.0123s) 2024-04-26T16:08:16.266 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout { 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout } 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 Processor -- start 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- start start 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f8106b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09f8107120 con 0x7f09f8106730 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fd9a2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f8106b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fd9a2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f8106b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52590/0 (socket says 172.21.15.50:52590) 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fd9a2700 1 -- 172.21.15.50:0/523333409 learned_addr learned my addr 172.21.15.50:0/523333409 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fd9a2700 1 -- 172.21.15.50:0/523333409 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f8107930 con 0x7f09f8106730 2024-04-26T16:08:16.267 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fd9a2700 1 --2- 172.21.15.50:0/523333409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f8106b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09f40098a0 tx=0x7f09f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4562e1d9a8fa66 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fc9a0700 1 -- 172.21.15.50:0/523333409 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f4004160 con 0x7f09f8106730 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fc9a0700 1 -- 172.21.15.50:0/523333409 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f09f4004300 con 0x7f09f8106730 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09fc9a0700 1 -- 172.21.15.50:0/523333409 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f4011870 con 0x7f09f8106730 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- 172.21.15.50:0/523333409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 msgr2=0x7f09f8106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 --2- 172.21.15.50:0/523333409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f8106b50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09f40098a0 tx=0x7f09f4009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- 172.21.15.50:0/523333409 shutdown_connections 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 --2- 172.21.15.50:0/523333409 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f8106b50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- 172.21.15.50:0/523333409 >> 172.21.15.50:0/523333409 conn(0x7f09f8101cd0 msgr2=0x7f09f8104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- 172.21.15.50:0/523333409 shutdown_connections 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- 172.21.15.50:0/523333409 wait complete. 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 librados: starting msgr at 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 librados: starting objecter 2024-04-26T16:08:16.268 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 Processor -- start 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 -- start start 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 librados: setting wanted keys 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.402+0000 7f09ffc06700 1 librados: calling monclient init 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09ffc06700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09ffc06700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810ab60 con 0x7f09f8106730 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd9a2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd9a2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f819d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52604/0 (socket says 172.21.15.50:52604) 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd9a2700 1 -- 172.21.15.50:0/3461012331 learned_addr learned my addr 172.21.15.50:0/3461012331 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd9a2700 1 -- 172.21.15.50:0/3461012331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f819de90 con 0x7f09f8106730 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd9a2700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f819d950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f09f4003ee0 tx=0x7f09f4011960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 10 client.?.objecter ms_handle_connect 0x7f09f8106730 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f40036a0 con 0x7f09f8106730 2024-04-26T16:08:16.269 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f09f401a430 con 0x7f09f8106730 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09f40193c0 con 0x7f09f8106730 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f819e120 con 0x7f09f8106730 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09ffc06700 10 client.14156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f81a0e10 con 0x7f09f8106730 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09ffc06700 1 librados: init done 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 9) ==== 47190+0+0 (secure 0 0 0) 0x7f09f401a8e0 con 0x7f09f8106730 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 --> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f09e403e8a0 con 0x7f09e403bcd0 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f09f404e820 con 0x7f09f8106730 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 10 client.14156.objecter ms_dispatch 0x7f09f8001550 osd_map(2..2 src has 1..2) 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 3 client.14156.objecter handle_osd_map got epochs [2,2] > 0 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 3 client.14156.objecter handle_osd_map decoding full epoch 2 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd1a1700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 msgr2=0x7f09e403e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:16.270 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.406+0000 7f09eeffd700 20 client.14156.objecter dump_active .. 0 homeless 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.610+0000 7f09fd1a1700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 msgr2=0x7f09e403e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:09.610+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:10.014+0000 7f09fd1a1700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 msgr2=0x7f09e403e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:10.014+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:10.818+0000 7f09fd1a1700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 msgr2=0x7f09e403e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:10.818+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:12.422+0000 7f09fd1a1700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 msgr2=0x7f09e403e190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/685732569 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:12.422+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:14.410+0000 7f09ff405700 10 client.14156.objecter tick 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:14.594+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mgrmap(e 10) ==== 46957+0+0 (secure 0 0 0) 0x7f09f4003840 con 0x7f09f8106730 2024-04-26T16:08:16.271 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:14.594+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 msgr2=0x7f09e403e190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:14.594+0000 7f09eeffd700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mgrmap(e 11) ==== 47084+0+0 (secure 0 0 0) 0x7f09f40500f0 con 0x7f09f8106730 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09eeffd700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f09e403f750 0x7f09e4041b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f09f402de50 con 0x7f09e403f750 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f09e403f750 0x7f09e4041b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09eeffd700 10 client.14156.objecter ms_handle_connect 0x7f09e403f750 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09fd1a1700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f09e403f750 0x7f09e4041b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09e8003a10 tx=0x7f09e80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.598+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f09f402de50 con 0x7f09e403f750 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f09f8064a80 con 0x7f09e403f750 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09eeffd700 1 -- 172.21.15.50:0/3461012331 <== mgr.14152 v2:172.21.15.50:6800/126446279 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f09f8064a80 con 0x7f09e403f750 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 10 librados: watch_flush enter 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 10 librados: watch_flush exit 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 20 client.14156.objecter shutdown clearing up homeless session... 2024-04-26T16:08:16.272 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 10 client.14156.objecter successfully canceled tick 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f09e403f750 msgr2=0x7f09e4041b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f09e403f750 0x7f09e4041b40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09e8003a10 tx=0x7f09e80092b0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 msgr2=0x7f09f819d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f819d950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f09f4003ee0 tx=0x7f09f4011960 comp rx=0 tx=0).stop 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 shutdown_connections 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f09e403f750 0x7f09e4041b40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f09f8106730 0x7f09f819d950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 --2- 172.21.15.50:0/3461012331 >> [v2:172.21.15.50:6800/685732569,v1:172.21.15.50:6801/685732569] conn(0x7f09e403bcd0 0x7f09e403e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 >> 172.21.15.50:0/3461012331 conn(0x7f09f8101cd0 msgr2=0x7f09f8102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 shutdown_connections 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 -- 172.21.15.50:0/3461012331 wait complete. 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:15.614+0000 7f09ffc06700 1 librados: shutdown 2024-04-26T16:08:16.273 INFO:teuthology.orchestra.run.smithi050.stdout:mgr epoch 9 is available 2024-04-26T16:08:16.274 INFO:teuthology.orchestra.run.smithi050.stdout:Generating a dashboard self-signed certificate... 2024-04-26T16:08:16.943 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.438664+0000 mgr.y (mgr.14152) 1 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Bus STARTING 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.438664+0000 mgr.y (mgr.14152) 1 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Bus STARTING 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.540502+0000 mgr.y (mgr.14152) 2 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.540502+0000 mgr.y (mgr.14152) 2 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: audit 2024-04-26T16:08:15.601883+0000 mgr.y (mgr.14152) 3 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: audit 2024-04-26T16:08:15.601883+0000 mgr.y (mgr.14152) 3 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: audit 2024-04-26T16:08:15.617585+0000 mgr.y (mgr.14152) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: audit 2024-04-26T16:08:15.617585+0000 mgr.y (mgr.14152) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.663137+0000 mgr.y (mgr.14152) 5 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.663137+0000 mgr.y (mgr.14152) 5 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.663290+0000 mgr.y (mgr.14152) 6 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Bus STARTED 2024-04-26T16:08:16.944 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:16 smithi050 bash[21463]: cephadm 2024-04-26T16:08:15.663290+0000 mgr.y (mgr.14152) 6 : cephadm [INF] [26/Apr/2024:16:08:15] ENGINE Bus STARTED 2024-04-26T16:08:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:17 smithi050 bash[21463]: cluster 2024-04-26T16:08:16.654980+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-04-26T16:08:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:17 smithi050 bash[21463]: cluster 2024-04-26T16:08:16.654980+0000 mon.a (mon.0) 88 : cluster [DBG] mgrmap e12: y(active, since 2s) 2024-04-26T16:08:18.259 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.154+0000 7f7390f5e700 1 Processor -- start 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.154+0000 7f7390f5e700 1 -- start start 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.154+0000 7f7390f5e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 0x7f738c108180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.154+0000 7f7390f5e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f738c108750 con 0x7f738c107d60 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 0x7f738c108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 0x7f738c108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52670/0 (socket says 172.21.15.50:52670) 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 -- 172.21.15.50:0/4209024445 learned_addr learned my addr 172.21.15.50:0/4209024445 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:18.280 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 -- 172.21.15.50:0/4209024445 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c1069c0 con 0x7f738c107d60 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 --2- 172.21.15.50:0/4209024445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 0x7f738c108180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f737c0098a0 tx=0x7f737c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5883aeeff896c2c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738959a700 1 -- 172.21.15.50:0/4209024445 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f737c004160 con 0x7f738c107d60 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738959a700 1 -- 172.21.15.50:0/4209024445 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f737c004300 con 0x7f738c107d60 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738959a700 1 -- 172.21.15.50:0/4209024445 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f737c011870 con 0x7f738c107d60 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- 172.21.15.50:0/4209024445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 msgr2=0x7f738c108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 --2- 172.21.15.50:0/4209024445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 0x7f738c108180 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f737c0098a0 tx=0x7f737c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- 172.21.15.50:0/4209024445 shutdown_connections 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 --2- 172.21.15.50:0/4209024445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c107d60 0x7f738c108180 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- 172.21.15.50:0/4209024445 >> 172.21.15.50:0/4209024445 conn(0x7f738c101490 msgr2=0x7f738c1038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- 172.21.15.50:0/4209024445 shutdown_connections 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- 172.21.15.50:0/4209024445 wait complete. 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 librados: starting msgr at 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 librados: starting objecter 2024-04-26T16:08:18.281 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 Processor -- start 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- start start 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 librados: setting wanted keys 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 librados: calling monclient init 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 0x7f738c19df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f738c10ac00 con 0x7f738c19db70 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 0x7f738c19df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 0x7f738c19df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52678/0 (socket says 172.21.15.50:52678) 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 -- 172.21.15.50:0/1195968183 learned_addr learned my addr 172.21.15.50:0/1195968183 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 -- 172.21.15.50:0/1195968183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f738c19e4d0 con 0x7f738c19db70 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f738a59c700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 0x7f738c19df90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f737c003ee0 tx=0x7f737c011910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f737b7fe700 10 client.?.objecter ms_handle_connect 0x7f738c19db70 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f737b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:18.282 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f737c004120 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f737c026480 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f737c01a440 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f738c19e760 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.158+0000 7f7390f5e700 10 client.14164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f738c1a1240 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f7390f5e700 1 librados: init done 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) ==== 47190+0+0 (secure 0 0 0) 0x7f737c0268a0 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f738c064a80 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f737403bcd0 0x7f737403e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f737c04ee90 con 0x7f738c19db70 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 10 client.14164.objecter ms_dispatch 0x7f738c001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 3 client.14164.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 3 client.14164.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:18.283 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 20 client.14164.objecter dump_active .. 0 homeless 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f7389d9b700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f737403bcd0 0x7f737403e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f7389d9b700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f737403bcd0 0x7f737403e190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7380006fd0 tx=0x7f7380006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.162+0000 7f737b7fe700 10 client.14164.objecter ms_handle_connect 0x7f737403bcd0 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.166+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f737c00e260 con 0x7f738c19db70 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.574+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f738c02e640 con 0x7f737403bcd0 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.670+0000 7f737b7fe700 1 -- 172.21.15.50:0/1195968183 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f738c02e640 con 0x7f737403bcd0 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 10 librados: watch_flush enter 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 10 librados: watch_flush exit 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 20 client.14164.objecter shutdown clearing up homeless session... 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 10 client.14164.objecter successfully canceled tick 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f737403bcd0 msgr2=0x7f737403e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f737403bcd0 0x7f737403e190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7380006fd0 tx=0x7f7380006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 msgr2=0x7f738c19df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:18.284 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 0x7f738c19df90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f737c003ee0 tx=0x7f737c011910 comp rx=0 tx=0).stop 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 shutdown_connections 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f737403bcd0 0x7f737403e190 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 --2- 172.21.15.50:0/1195968183 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f738c19db70 0x7f738c19df90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 >> 172.21.15.50:0/1195968183 conn(0x7f738c101490 msgr2=0x7f738c102480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 shutdown_connections 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 -- 172.21.15.50:0/1195968183 wait complete. 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:17.674+0000 7f7390f5e700 1 librados: shutdown 2024-04-26T16:08:18.285 INFO:teuthology.orchestra.run.smithi050.stdout:Creating initial admin user... 2024-04-26T16:08:18.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:18 smithi050 bash[21463]: audit 2024-04-26T16:08:17.576007+0000 mgr.y (mgr.14152) 7 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:18.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:18 smithi050 bash[21463]: audit 2024-04-26T16:08:17.576007+0000 mgr.y (mgr.14152) 7 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:18.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:18 smithi050 bash[21463]: audit 2024-04-26T16:08:17.670617+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:18.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:18 smithi050 bash[21463]: audit 2024-04-26T16:08:17.670617+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:18.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:18 smithi050 bash[21463]: audit 2024-04-26T16:08:17.673397+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:18.941 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:18 smithi050 bash[21463]: audit 2024-04-26T16:08:17.673397+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:20.256 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:19 smithi050 bash[21463]: audit 2024-04-26T16:08:19.600996+0000 mgr.y (mgr.14152) 8 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:20.256 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:19 smithi050 bash[21463]: audit 2024-04-26T16:08:19.600996+0000 mgr.y (mgr.14152) 8 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:20.658 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$16kNuQYSQ1ZM8iZA2uvmBu3mG4zVZ7cLFCvx//VRXueMi9K2rT5ZS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714147700, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-26T16:08:20.658 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e6bfe700 1 Processor -- start 2024-04-26T16:08:20.659 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e6bfe700 1 -- start start 2024-04-26T16:08:20.659 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e6bfe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e01062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:20.659 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e6bfe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68e01067f0 con 0x7f68e0105e90 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e499a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e01062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e499a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e01062b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45362/0 (socket says 172.21.15.50:45362) 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.150+0000 7f68e499a700 1 -- 172.21.15.50:0/1787397063 learned_addr learned my addr 172.21.15.50:0/1787397063 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 -- 172.21.15.50:0/1787397063 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e0107080 con 0x7f68e0105e90 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 --2- 172.21.15.50:0/1787397063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e01062b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00aa40 tx=0x7f68dc00ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71596a48053d4d53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d77fe700 1 -- 172.21.15.50:0/1787397063 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68dc00b040 con 0x7f68e0105e90 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d77fe700 1 -- 172.21.15.50:0/1787397063 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f68dc00c8f0 con 0x7f68e0105e90 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d77fe700 1 -- 172.21.15.50:0/1787397063 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68dc003920 con 0x7f68e0105e90 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/1787397063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 msgr2=0x7f68e01062b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:20.661 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 --2- 172.21.15.50:0/1787397063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e01062b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00aa40 tx=0x7f68dc00ad50 comp rx=0 tx=0).stop 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/1787397063 shutdown_connections 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 --2- 172.21.15.50:0/1787397063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e01062b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/1787397063 >> 172.21.15.50:0/1787397063 conn(0x7f68e0101440 msgr2=0x7f68e01038c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/1787397063 shutdown_connections 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/1787397063 wait complete. 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 librados: starting msgr at 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 librados: starting objecter 2024-04-26T16:08:20.662 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 Processor -- start 2024-04-26T16:08:20.667 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- start start 2024-04-26T16:08:20.667 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 librados: setting wanted keys 2024-04-26T16:08:20.667 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 librados: calling monclient init 2024-04-26T16:08:20.667 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68e0107610 con 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e019d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45368/0 (socket says 172.21.15.50:45368) 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 -- 172.21.15.50:0/263105160 learned_addr learned my addr 172.21.15.50:0/263105160 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 -- 172.21.15.50:0/263105160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e019de70 con 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e499a700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e019d930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00afd0 tx=0x7f68dc003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d5ffb700 10 client.?.objecter ms_handle_connect 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d5ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68dc025070 con 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f68dc004300 con 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e019e100 con 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f68dc0103f0 con 0x7f68e0105e90 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 10 client.14166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:20.668 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e01a0df0 con 0x7f68e0105e90 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.154+0000 7f68e6bfe700 1 librados: init done 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 12) ==== 47190+0+0 (secure 0 0 0) 0x7f68dc028070 con 0x7f68e0105e90 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e0064a80 con 0x7f68e0105e90 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f68c803bcd0 0x7f68c803e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f68dc04e6c0 con 0x7f68e0105e90 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 10 client.14166.objecter ms_dispatch 0x7f68e0001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 3 client.14166.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 3 client.14166.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 20 client.14166.objecter dump_active .. 0 homeless 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d7fff700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f68c803bcd0 0x7f68c803e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d7fff700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f68c803bcd0 0x7f68c803e190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68d0006fd0 tx=0x7f68d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.158+0000 7f68d5ffb700 10 client.14166.objecter ms_handle_connect 0x7f68c803bcd0 2024-04-26T16:08:20.669 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.162+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f68dc014070 con 0x7f68e0105e90 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:19.598+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f68e0195670 con 0x7f68c803bcd0 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.058+0000 7f68d5ffb700 1 -- 172.21.15.50:0/263105160 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f68e0195670 con 0x7f68c803bcd0 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 10 librados: watch_flush enter 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 10 librados: watch_flush exit 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 20 client.14166.objecter shutdown clearing up homeless session... 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 10 client.14166.objecter successfully canceled tick 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f68c803bcd0 msgr2=0x7f68c803e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f68c803bcd0 0x7f68c803e190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68d0006fd0 tx=0x7f68d0006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 msgr2=0x7f68e019d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e019d930 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f68dc00afd0 tx=0x7f68dc003e90 comp rx=0 tx=0).stop 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 shutdown_connections 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f68c803bcd0 0x7f68c803e190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:20.670 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 --2- 172.21.15.50:0/263105160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68e0105e90 0x7f68e019d930 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:20.671 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 >> 172.21.15.50:0/263105160 conn(0x7f68e0101440 msgr2=0x7f68e018eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:20.671 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 shutdown_connections 2024-04-26T16:08:20.671 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 -- 172.21.15.50:0/263105160 wait complete. 2024-04-26T16:08:20.671 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:20.062+0000 7f68e6bfe700 1 librados: shutdown 2024-04-26T16:08:20.671 INFO:teuthology.orchestra.run.smithi050.stdout:Fetching dashboard port number... 2024-04-26T16:08:21.362 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:21 smithi050 bash[21463]: audit 2024-04-26T16:08:20.059274+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:21.363 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:21 smithi050 bash[21463]: audit 2024-04-26T16:08:20.059274+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:22.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:22 smithi050 bash[21463]: cluster 2024-04-26T16:08:21.063869+0000 mon.a (mon.0) 92 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-04-26T16:08:22.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:22 smithi050 bash[21463]: cluster 2024-04-26T16:08:21.063869+0000 mon.a (mon.0) 92 : cluster [DBG] mgrmap e13: y(active, since 6s) 2024-04-26T16:08:22.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:22 smithi050 bash[21463]: audit 2024-04-26T16:08:21.535952+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:22.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:22 smithi050 bash[21463]: audit 2024-04-26T16:08:21.535952+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:22.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:22 smithi050 bash[21463]: audit 2024-04-26T16:08:21.965875+0000 mon.a (mon.0) 94 : audit [DBG] from='client.? 172.21.15.50:0/1257357473' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T16:08:22.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:22 smithi050 bash[21463]: audit 2024-04-26T16:08:21.965875+0000 mon.a (mon.0) 94 : audit [DBG] from='client.? 172.21.15.50:0/1257357473' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-26T16:08:22.685 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stdout 8443 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.550+0000 7ff77241a700 1 Processor -- start 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.550+0000 7ff77241a700 1 -- start start 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c07bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c07c140 con 0x7ff76c07d710 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c07bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c07bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45380/0 (socket says 172.21.15.50:45380) 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 -- 172.21.15.50:0/2032201924 learned_addr learned my addr 172.21.15.50:0/2032201924 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 -- 172.21.15.50:0/2032201924 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c07c2c0 con 0x7ff76c07d710 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 --2- 172.21.15.50:0/2032201924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c07bb70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec572662e9d9862e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76b7fe700 1 -- 172.21.15.50:0/2032201924 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff75c004160 con 0x7ff76c07d710 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76b7fe700 1 -- 172.21.15.50:0/2032201924 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff75c004300 con 0x7ff76c07d710 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76b7fe700 1 -- 172.21.15.50:0/2032201924 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff75c011870 con 0x7ff76c07d710 2024-04-26T16:08:22.686 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- 172.21.15.50:0/2032201924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 msgr2=0x7ff76c07bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 --2- 172.21.15.50:0/2032201924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c07bb70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff75c0098a0 tx=0x7ff75c009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- 172.21.15.50:0/2032201924 shutdown_connections 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 --2- 172.21.15.50:0/2032201924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c07bb70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- 172.21.15.50:0/2032201924 >> 172.21.15.50:0/2032201924 conn(0x7ff76c102df0 msgr2=0x7ff76c105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- 172.21.15.50:0/2032201924 shutdown_connections 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- 172.21.15.50:0/2032201924 wait complete. 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 librados: starting msgr at 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 librados: starting objecter 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 Processor -- start 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- start start 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 librados: setting wanted keys 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 librados: calling monclient init 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:22.687 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff77241a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff76c10ddd0 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c19d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45396/0 (socket says 172.21.15.50:45396) 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 -- 172.21.15.50:0/1257357473 learned_addr learned my addr 172.21.15.50:0/1257357473 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.554+0000 7ff76bfff700 1 -- 172.21.15.50:0/1257357473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff76c19dec0 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff76bfff700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c19d980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009870 tx=0x7ff75c011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 10 client.?.objecter ms_handle_connect 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff75c011930 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff75c003b00 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff75c01ac70 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff76c19e150 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff77241a700 10 client.14168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff76c1a0e40 con 0x7ff76c07d710 2024-04-26T16:08:22.688 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff77241a700 1 librados: init done 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7ff75c02d440 con 0x7ff76c07d710 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff75803bd70 0x7ff75803e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff75c04ec00 con 0x7ff76c07d710 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 10 client.14168.objecter ms_dispatch 0x7ff76c001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 3 client.14168.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 3 client.14168.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 20 client.14168.objecter dump_active .. 0 homeless 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff7637fe700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff75803bd70 0x7ff75803e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff76c064a80 con 0x7ff76c07d710 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff769ffb700 10 client.14168.objecter ms_handle_connect 0x7ff75803bd70 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.558+0000 7ff7637fe700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff75803bd70 0x7ff75803e230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff754006fd0 tx=0x7ff754006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.562+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff75c02dd90 con 0x7ff76c07d710 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.962+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7ff76c1a10e0 con 0x7ff76c07d710 2024-04-26T16:08:22.689 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.962+0000 7ff769ffb700 1 -- 172.21.15.50:0/1257357473 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7ff75c032030 con 0x7ff76c07d710 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 10 librados: watch_flush enter 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 10 librados: watch_flush exit 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 20 client.14168.objecter shutdown clearing up homeless session... 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 10 client.14168.objecter successfully canceled tick 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff75803bd70 msgr2=0x7ff75803e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff75803bd70 0x7ff75803e230 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff754006fd0 tx=0x7ff754006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 msgr2=0x7ff76c19d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c19d980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009870 tx=0x7ff75c011f90 comp rx=0 tx=0).stop 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 shutdown_connections 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff75803bd70 0x7ff75803e230 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 --2- 172.21.15.50:0/1257357473 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff76c07d710 0x7ff76c19d980 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 >> 172.21.15.50:0/1257357473 conn(0x7ff76c102df0 msgr2=0x7ff76c103a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 shutdown_connections 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 -- 172.21.15.50:0/1257357473 wait complete. 2024-04-26T16:08:22.690 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:21.966+0000 7ff77241a700 1 librados: shutdown 2024-04-26T16:08:22.691 INFO:teuthology.orchestra.run.smithi050.stdout:firewalld does not appear to be present 2024-04-26T16:08:22.691 INFO:teuthology.orchestra.run.smithi050.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-26T16:08:22.691 INFO:teuthology.orchestra.run.smithi050.stdout:Ceph Dashboard is now available at: 2024-04-26T16:08:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: URL: https://smithi050.front.sepia.ceph.com:8443/ 2024-04-26T16:08:22.691 INFO:teuthology.orchestra.run.smithi050.stdout: User: admin 2024-04-26T16:08:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: Password: 3x9h71b2f3 2024-04-26T16:08:22.692 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:22.692 INFO:teuthology.orchestra.run.smithi050.stdout:Saving cluster configuration to /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config directory 2024-04-26T16:08:22.692 INFO:teuthology.orchestra.run.smithi050.stdout:Enabling autotune for osd_memory_target 2024-04-26T16:08:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:23 smithi050 bash[21463]: audit 2024-04-26T16:08:22.527560+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:23 smithi050 bash[21463]: audit 2024-04-26T16:08:22.527560+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:24.567 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 Processor -- start 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- start start 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f0078107ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f00781082b0 con 0x7f00781078c0 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007d1d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f0078107ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007d1d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f0078107ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45398/0 (socket says 172.21.15.50:45398) 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007d1d2700 1 -- 172.21.15.50:0/544941424 learned_addr learned my addr 172.21.15.50:0/544941424 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007d1d2700 1 -- 172.21.15.50:0/544941424 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0078108ac0 con 0x7f00781078c0 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007d1d2700 1 --2- 172.21.15.50:0/544941424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f0078107ce0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00740098a0 tx=0x7f0074009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01cb1df97549189 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:24.568 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f006ffff700 1 -- 172.21.15.50:0/544941424 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0074004160 con 0x7f00781078c0 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f006ffff700 1 -- 172.21.15.50:0/544941424 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0074004300 con 0x7f00781078c0 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f006ffff700 1 -- 172.21.15.50:0/544941424 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0074011870 con 0x7f00781078c0 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- 172.21.15.50:0/544941424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 msgr2=0x7f0078107ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 --2- 172.21.15.50:0/544941424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f0078107ce0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f00740098a0 tx=0x7f0074009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- 172.21.15.50:0/544941424 shutdown_connections 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 --2- 172.21.15.50:0/544941424 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f0078107ce0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- 172.21.15.50:0/544941424 >> 172.21.15.50:0/544941424 conn(0x7f0078102e60 msgr2=0x7f00781052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- 172.21.15.50:0/544941424 shutdown_connections 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 -- 172.21.15.50:0/544941424 wait complete. 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 librados: starting msgr at 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.582+0000 7f007f436700 1 librados: starting objecter 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 Processor -- start 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 -- start start 2024-04-26T16:08:24.569 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 librados: setting wanted keys 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 librados: calling monclient init 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f007819b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f007810bcf0 con 0x7f00781078c0 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007d1d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f007819b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007d1d2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f007819b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45410/0 (socket says 172.21.15.50:45410) 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007d1d2700 1 -- 172.21.15.50:0/984604754 learned_addr learned my addr 172.21.15.50:0/984604754 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007d1d2700 1 -- 172.21.15.50:0/984604754 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f007819bce0 con 0x7f00781078c0 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007d1d2700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f007819b7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0074003ee0 tx=0x7f0074011960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:24.570 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 10 client.?.objecter ms_handle_connect 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00740036a0 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f007401a430 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00740193c0 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f007819bf70 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 10 client.14170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f007819ec60 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 librados: init done 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0078064a80 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f007401a8e0 con 0x7f00781078c0 2024-04-26T16:08:24.571 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f006403bdc0 0x7f006403e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f007404e9e0 con 0x7f00781078c0 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 10 client.14170.objecter ms_dispatch 0x7f0078001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 3 client.14170.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 3 client.14170.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 20 client.14170.objecter dump_active .. 0 homeless 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007c9d1700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f006403bdc0 0x7f006403e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f007c9d1700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f006403bdc0 0x7f006403e280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0068006fd0 tx=0x7f0068006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:24.572 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.586+0000 7f006e7fc700 10 client.14170.objecter ms_handle_connect 0x7f006403bdc0 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.590+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f007400fda0 con 0x7f00781078c0 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.982+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f007819f0b0 con 0x7f00781078c0 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.982+0000 7f006e7fc700 1 -- 172.21.15.50:0/984604754 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f0074003840 con 0x7f00781078c0 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 10 librados: watch_flush enter 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 10 librados: watch_flush exit 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 20 client.14170.objecter shutdown clearing up homeless session... 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 10 client.14170.objecter successfully canceled tick 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f006403bdc0 msgr2=0x7f006403e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f006403bdc0 0x7f006403e280 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0068006fd0 tx=0x7f0068006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 msgr2=0x7f007819b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f007819b7a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0074003ee0 tx=0x7f0074011960 comp rx=0 tx=0).stop 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 shutdown_connections 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f006403bdc0 0x7f006403e280 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:24.573 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 --2- 172.21.15.50:0/984604754 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f00781078c0 0x7f007819b7a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:24.574 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 >> 172.21.15.50:0/984604754 conn(0x7f0078102e60 msgr2=0x7f00781039f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:24.574 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 shutdown_connections 2024-04-26T16:08:24.574 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 -- 172.21.15.50:0/984604754 wait complete. 2024-04-26T16:08:24.574 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:23.986+0000 7f007f436700 1 librados: shutdown 2024-04-26T16:08:26.716 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 Processor -- start 2024-04-26T16:08:26.716 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- start start 2024-04-26T16:08:26.716 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc106b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:26.716 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc107120 con 0x7f1acc106730 2024-04-26T16:08:26.716 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad0a24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc106b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad0a24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc106b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45424/0 (socket says 172.21.15.50:45424) 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad0a24700 1 -- 172.21.15.50:0/3930308983 learned_addr learned my addr 172.21.15.50:0/3930308983 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad0a24700 1 -- 172.21.15.50:0/3930308983 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc107930 con 0x7f1acc106730 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad0a24700 1 --2- 172.21.15.50:0/3930308983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc106b50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00098a0 tx=0x7f1ac0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06a62f1cf65e6a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1acb7fe700 1 -- 172.21.15.50:0/3930308983 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ac0004160 con 0x7f1acc106730 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1acb7fe700 1 -- 172.21.15.50:0/3930308983 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1ac0004300 con 0x7f1acc106730 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1acb7fe700 1 -- 172.21.15.50:0/3930308983 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ac0011870 con 0x7f1acc106730 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- 172.21.15.50:0/3930308983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 msgr2=0x7f1acc106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 --2- 172.21.15.50:0/3930308983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc106b50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1ac00098a0 tx=0x7f1ac0009bb0 comp rx=0 tx=0).stop 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- 172.21.15.50:0/3930308983 shutdown_connections 2024-04-26T16:08:26.717 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 --2- 172.21.15.50:0/3930308983 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc106b50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- 172.21.15.50:0/3930308983 >> 172.21.15.50:0/3930308983 conn(0x7f1acc101cd0 msgr2=0x7f1acc104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- 172.21.15.50:0/3930308983 shutdown_connections 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- 172.21.15.50:0/3930308983 wait complete. 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 librados: starting msgr at 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 librados: starting objecter 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 Processor -- start 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 -- start start 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 librados: setting wanted keys 2024-04-26T16:08:26.718 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.434+0000 7f1ad2c88700 1 librados: calling monclient init 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad2c88700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad2c88700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1acc10ab60 con 0x7f1acc106730 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad0a24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad0a24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc19d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45428/0 (socket says 172.21.15.50:45428) 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad0a24700 1 -- 172.21.15.50:0/2392489698 learned_addr learned my addr 172.21.15.50:0/2392489698 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad0a24700 1 -- 172.21.15.50:0/2392489698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1acc19de90 con 0x7f1acc106730 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad0a24700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc19d950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006af0 tx=0x7f1ac0011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 10 client.?.objecter ms_handle_connect 0x7f1acc106730 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ac0011970 con 0x7f1acc106730 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad2c88700 1 -- 172.21.15.50:0/2392489698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1acc19e120 con 0x7f1acc106730 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1ac0003b00 con 0x7f1acc106730 2024-04-26T16:08:26.719 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1ac001acf0 con 0x7f1acc106730 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad2c88700 10 client.14172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad2c88700 1 -- 172.21.15.50:0/2392489698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1acc1a0e10 con 0x7f1acc106730 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ad2c88700 1 librados: init done 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f1ac002d440 con 0x7f1acc106730 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aac005120 con 0x7f1acc106730 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f1abc03bf10 0x7f1abc03e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1ac004ef00 con 0x7f1acc106730 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 10 client.14172.objecter ms_dispatch 0x7f1acc001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:26.720 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 3 client.14172.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 3 client.14172.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 20 client.14172.objecter dump_active .. 0 homeless 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1acbfff700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f1abc03bf10 0x7f1abc03e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1acbfff700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f1abc03bf10 0x7f1abc03e3d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8006fd0 tx=0x7f1ab8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.438+0000 7f1ac9ffb700 10 client.14172.objecter ms_handle_connect 0x7f1abc03bf10 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:25.446+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1ac000b3e0 con 0x7f1acc106730 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.066+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1aac005410 con 0x7f1acc106730 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.066+0000 7f1ac9ffb700 1 -- 172.21.15.50:0/2392489698 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f1ac002d730 con 0x7f1acc106730 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 10 librados: watch_flush enter 2024-04-26T16:08:26.721 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 10 librados: watch_flush exit 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 20 client.14172.objecter shutdown clearing up homeless session... 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 10 client.14172.objecter successfully canceled tick 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f1abc03bf10 msgr2=0x7f1abc03e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f1abc03bf10 0x7f1abc03e3d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8006fd0 tx=0x7f1ab8006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 msgr2=0x7f1acc19d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc19d950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ac0006af0 tx=0x7f1ac0011f90 comp rx=0 tx=0).stop 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 shutdown_connections 2024-04-26T16:08:26.722 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f1abc03bf10 0x7f1abc03e3d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:26.723 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 --2- 172.21.15.50:0/2392489698 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1acc106730 0x7f1acc19d950 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:26.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 >> 172.21.15.50:0/2392489698 conn(0x7f1acc101cd0 msgr2=0x7f1acc102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:26.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 shutdown_connections 2024-04-26T16:08:26.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 -- 172.21.15.50:0/2392489698 wait complete. 2024-04-26T16:08:26.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr 2024-04-26T16:08:26.070+0000 7f1ab77fe700 1 librados: shutdown 2024-04-26T16:08:26.725 INFO:teuthology.orchestra.run.smithi050.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-26T16:08:26.726 INFO:teuthology.orchestra.run.smithi050.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-26T16:08:26.726 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.726 INFO:teuthology.orchestra.run.smithi050.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:08:26.726 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.726 INFO:teuthology.orchestra.run.smithi050.stdout:Or, if you are only running a single cluster on this host: 2024-04-26T16:08:26.726 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-26T16:08:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.727 INFO:teuthology.orchestra.run.smithi050.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-26T16:08:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: ceph telemetry on 2024-04-26T16:08:26.727 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.728 INFO:teuthology.orchestra.run.smithi050.stdout:For more information see: 2024-04-26T16:08:26.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.728 INFO:teuthology.orchestra.run.smithi050.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-26T16:08:26.728 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:08:26.728 INFO:teuthology.orchestra.run.smithi050.stdout:Bootstrap complete. 2024-04-26T16:08:26.759 INFO:tasks.cephadm:Fetching config... 2024-04-26T16:08:26.760 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:08:26.760 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-26T16:08:26.808 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-26T16:08:26.808 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:08:26.808 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-26T16:08:26.859 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-26T16:08:26.860 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:08:26.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-26T16:08:26.916 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-26T16:08:26.916 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:08:26.916 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-26T16:08:26.964 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-26T16:08:26.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfdK6ij82KFOTFigIcwNCT/REb6kbuPzHZV5Wgk8bjIYBa7xDfsxuewyv8C+/oziHDOaN+CuhBVjW1PGby2Q9+rXmOvMeHSQs9AtuSHualb5sPTAeARbTPZKaMiKmkrb7fz3lcKS+x/1La9ROw5AXo3IOpxhpNDe9nWp+XSrciu2zH1Z+ZYcFWbQwLC9PyKEVJ/7Ljvo45UMuXVmVxdE2ofTRdw2dnxVWxspEcWdC05nV2VbQogdKEPLw1Qb8Wqb6shZJdWKqIX+j5kJsfsur0t9tu5EU9kUv79OzSVG+xOHHy79LXj5s5auEqdoUUBsvcnggNeGW2Cv4q/9SEzx3sPYv3GN9MG3gerTHJ5hG2drO3/S9eEyAu5qxzQOqPIlO+8z1WrJRxdGKNV0Bv4RS8PRs0A5ggfNLhHQLyABB5Z4656/U3mrD6thvV22T+MzvS+AXKuUuwnn0aqfblWoT5U7YaRH+fyVt26N7+xFoYvP4Vtteqk/JV4Lea6215pkM= ceph-cf170a7a-03e6-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T16:08:27.030 INFO:teuthology.orchestra.run.smithi050.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfdK6ij82KFOTFigIcwNCT/REb6kbuPzHZV5Wgk8bjIYBa7xDfsxuewyv8C+/oziHDOaN+CuhBVjW1PGby2Q9+rXmOvMeHSQs9AtuSHualb5sPTAeARbTPZKaMiKmkrb7fz3lcKS+x/1La9ROw5AXo3IOpxhpNDe9nWp+XSrciu2zH1Z+ZYcFWbQwLC9PyKEVJ/7Ljvo45UMuXVmVxdE2ofTRdw2dnxVWxspEcWdC05nV2VbQogdKEPLw1Qb8Wqb6shZJdWKqIX+j5kJsfsur0t9tu5EU9kUv79OzSVG+xOHHy79LXj5s5auEqdoUUBsvcnggNeGW2Cv4q/9SEzx3sPYv3GN9MG3gerTHJ5hG2drO3/S9eEyAu5qxzQOqPIlO+8z1WrJRxdGKNV0Bv4RS8PRs0A5ggfNLhHQLyABB5Z4656/U3mrD6thvV22T+MzvS+AXKuUuwnn0aqfblWoT5U7YaRH+fyVt26N7+xFoYvP4Vtteqk/JV4Lea6215pkM= ceph-cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:08:27.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfdK6ij82KFOTFigIcwNCT/REb6kbuPzHZV5Wgk8bjIYBa7xDfsxuewyv8C+/oziHDOaN+CuhBVjW1PGby2Q9+rXmOvMeHSQs9AtuSHualb5sPTAeARbTPZKaMiKmkrb7fz3lcKS+x/1La9ROw5AXo3IOpxhpNDe9nWp+XSrciu2zH1Z+ZYcFWbQwLC9PyKEVJ/7Ljvo45UMuXVmVxdE2ofTRdw2dnxVWxspEcWdC05nV2VbQogdKEPLw1Qb8Wqb6shZJdWKqIX+j5kJsfsur0t9tu5EU9kUv79OzSVG+xOHHy79LXj5s5auEqdoUUBsvcnggNeGW2Cv4q/9SEzx3sPYv3GN9MG3gerTHJ5hG2drO3/S9eEyAu5qxzQOqPIlO+8z1WrJRxdGKNV0Bv4RS8PRs0A5ggfNLhHQLyABB5Z4656/U3mrD6thvV22T+MzvS+AXKuUuwnn0aqfblWoT5U7YaRH+fyVt26N7+xFoYvP4Vtteqk/JV4Lea6215pkM= ceph-cf170a7a-03e6-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-26T16:08:27.065 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCfdK6ij82KFOTFigIcwNCT/REb6kbuPzHZV5Wgk8bjIYBa7xDfsxuewyv8C+/oziHDOaN+CuhBVjW1PGby2Q9+rXmOvMeHSQs9AtuSHualb5sPTAeARbTPZKaMiKmkrb7fz3lcKS+x/1La9ROw5AXo3IOpxhpNDe9nWp+XSrciu2zH1Z+ZYcFWbQwLC9PyKEVJ/7Ljvo45UMuXVmVxdE2ofTRdw2dnxVWxspEcWdC05nV2VbQogdKEPLw1Qb8Wqb6shZJdWKqIX+j5kJsfsur0t9tu5EU9kUv79OzSVG+xOHHy79LXj5s5auEqdoUUBsvcnggNeGW2Cv4q/9SEzx3sPYv3GN9MG3gerTHJ5hG2drO3/S9eEyAu5qxzQOqPIlO+8z1WrJRxdGKNV0Bv4RS8PRs0A5ggfNLhHQLyABB5Z4656/U3mrD6thvV22T+MzvS+AXKuUuwnn0aqfblWoT5U7YaRH+fyVt26N7+xFoYvP4Vtteqk/JV4Lea6215pkM= ceph-cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:08:27.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-26T16:08:27.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:27 smithi050 bash[21463]: audit 2024-04-26T16:08:26.070863+0000 mon.a (mon.0) 96 : audit [INF] from='client.? 172.21.15.50:0/2392489698' entity='client.admin' 2024-04-26T16:08:27.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:27 smithi050 bash[21463]: audit 2024-04-26T16:08:26.070863+0000 mon.a (mon.0) 96 : audit [INF] from='client.? 172.21.15.50:0/2392489698' entity='client.admin' 2024-04-26T16:08:32.231 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:08:33.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.870244+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.870244+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.875441+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.875441+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.876951+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.876951+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi050", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.883574+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.883574+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.918259+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.918259+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.926255+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:32 smithi050 bash[21463]: audit 2024-04-26T16:08:31.926255+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:34.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3497509868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 msgr2=0x7fbd68105d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:34.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 --2- 172.21.15.50:0/3497509868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68105d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbd64009910 tx=0x7fbd64009c20 comp rx=0 tx=0).stop 2024-04-26T16:08:34.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3497509868 shutdown_connections 2024-04-26T16:08:34.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 --2- 172.21.15.50:0/3497509868 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68105d80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:34.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3497509868 >> 172.21.15.50:0/3497509868 conn(0x7fbd680fd360 msgr2=0x7fbd680ff780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:34.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3497509868 shutdown_connections 2024-04-26T16:08:34.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3497509868 wait complete. 2024-04-26T16:08:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 librados: starting msgr at 2024-04-26T16:08:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 librados: starting objecter 2024-04-26T16:08:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 Processor -- start 2024-04-26T16:08:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- start start 2024-04-26T16:08:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 librados: setting wanted keys 2024-04-26T16:08:34.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 librados: calling monclient init 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681082b0 con 0x7fbd68103990 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6da36700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6da36700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38782/0 (socket says 172.21.15.50:38782) 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6da36700 1 -- 172.21.15.50:0/3009077447 learned_addr learned my addr 172.21.15.50:0/3009077447 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6da36700 1 -- 172.21.15.50:0/3009077447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd68103750 con 0x7fbd68103990 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6da36700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68103210 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd64006010 tx=0x7fbd64005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd5affd700 10 client.?.objecter ms_handle_connect 0x7fbd68103990 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd5affd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd64027070 con 0x7fbd68103990 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fbd6400f460 con 0x7fbd68103990 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd64017440 con 0x7fbd68103990 2024-04-26T16:08:34.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd68101940 con 0x7fbd68103990 2024-04-26T16:08:34.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 10 client.14174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:34.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd68101e80 con 0x7fbd68103990 2024-04-26T16:08:34.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd6fc9a700 1 librados: init done 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.658+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7fbd6402a070 con 0x7fbd68103990 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd68108a30 con 0x7fbd68103990 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fbd5403bd70 0x7fbd5403e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbd6404e990 con 0x7fbd68103990 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 10 client.14174.objecter ms_dispatch 0x7fbd68001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 3 client.14174.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:34.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 3 client.14174.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:34.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 20 client.14174.objecter dump_active .. 0 homeless 2024-04-26T16:08:34.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd6d235700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fbd5403bd70 0x7fbd5403e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:34.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd5affd700 10 client.14174.objecter ms_handle_connect 0x7fbd5403bd70 2024-04-26T16:08:34.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.662+0000 7fbd6d235700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fbd5403bd70 0x7fbd5403e230 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c006fd0 tx=0x7fbd5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:34.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.666+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fbd64013070 con 0x7fbd68103990 2024-04-26T16:08:34.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.882+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fbd68191000 con 0x7fbd68103990 2024-04-26T16:08:34.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.886+0000 7fbd5affd700 1 -- 172.21.15.50:0/3009077447 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fbd64031020 con 0x7fbd68103990 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 10 librados: watch_flush enter 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 10 librados: watch_flush exit 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 20 client.14174.objecter shutdown clearing up homeless session... 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 10 client.14174.objecter successfully canceled tick 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fbd5403bd70 msgr2=0x7fbd5403e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fbd5403bd70 0x7fbd5403e230 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c006fd0 tx=0x7fbd5c006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 msgr2=0x7fbd68103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68103210 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd64006010 tx=0x7fbd64005710 comp rx=0 tx=0).stop 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 shutdown_connections 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fbd5403bd70 0x7fbd5403e230 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:34.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 --2- 172.21.15.50:0/3009077447 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbd68103990 0x7fbd68103210 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 >> 172.21.15.50:0/3009077447 conn(0x7fbd680fd360 msgr2=0x7fbd680fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 shutdown_connections 2024-04-26T16:08:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 -- 172.21.15.50:0/3009077447 wait complete. 2024-04-26T16:08:34.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:34.890+0000 7fbd6fc9a700 1 librados: shutdown 2024-04-26T16:08:35.535 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-26T16:08:35.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: cluster 2024-04-26T16:08:34.605160+0000 mgr.y (mgr.14152) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: cluster 2024-04-26T16:08:34.605160+0000 mgr.y (mgr.14152) 9 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: audit 2024-04-26T16:08:34.890226+0000 mon.a (mon.0) 103 : audit [INF] from='client.? 172.21.15.50:0/3009077447' entity='client.admin' 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: audit 2024-04-26T16:08:34.890226+0000 mon.a (mon.0) 103 : audit [INF] from='client.? 172.21.15.50:0/3009077447' entity='client.admin' 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: audit 2024-04-26T16:08:34.932417+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: audit 2024-04-26T16:08:34.932417+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: audit 2024-04-26T16:08:34.937779+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:36.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:35 smithi050 bash[21463]: audit 2024-04-26T16:08:34.937779+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:37 smithi050 bash[21463]: cluster 2024-04-26T16:08:36.605580+0000 mgr.y (mgr.14152) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:37 smithi050 bash[21463]: cluster 2024-04-26T16:08:36.605580+0000 mgr.y (mgr.14152) 10 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:08:40.272 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:39 smithi050 bash[21463]: cluster 2024-04-26T16:08:38.606032+0000 mgr.y (mgr.14152) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:40.272 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:39 smithi050 bash[21463]: cluster 2024-04-26T16:08:38.606032+0000 mgr.y (mgr.14152) 11 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:41 smithi050 bash[21463]: cluster 2024-04-26T16:08:40.606462+0000 mgr.y (mgr.14152) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:41 smithi050 bash[21463]: cluster 2024-04-26T16:08:40.606462+0000 mgr.y (mgr.14152) 12 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/2002731987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 msgr2=0x7f7e88105d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 --2- 172.21.15.50:0/2002731987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88105d70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009910 tx=0x7f7e78009c20 comp rx=0 tx=0).stop 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/2002731987 shutdown_connections 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 --2- 172.21.15.50:0/2002731987 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88105d70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/2002731987 >> 172.21.15.50:0/2002731987 conn(0x7f7e880fd330 msgr2=0x7f7e880ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/2002731987 shutdown_connections 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/2002731987 wait complete. 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 librados: starting msgr at 2024-04-26T16:08:42.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 librados: starting objecter 2024-04-26T16:08:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 Processor -- start 2024-04-26T16:08:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- start start 2024-04-26T16:08:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 librados: setting wanted keys 2024-04-26T16:08:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 librados: calling monclient init 2024-04-26T16:08:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:42.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881082a0 con 0x7f7e88103980 2024-04-26T16:08:42.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:42.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50254/0 (socket says 172.21.15.50:50254) 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8659c700 1 -- 172.21.15.50:0/1565428801 learned_addr learned my addr 172.21.15.50:0/1565428801 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8659c700 1 -- 172.21.15.50:0/1565428801 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e88198040 con 0x7f7e88103980 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8659c700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88197b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800b5c0 tx=0x7f7e78005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e777fe700 10 client.?.objecter ms_handle_connect 0x7f7e88103980 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e777fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e78027070 con 0x7f7e88103980 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7e7800bdc0 con 0x7f7e88103980 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e881982d0 con 0x7f7e88103980 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e7800fcf0 con 0x7f7e88103980 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 10 client.14176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:42.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8819af70 con 0x7f7e88103980 2024-04-26T16:08:42.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.654+0000 7f7e8cdf5700 1 librados: init done 2024-04-26T16:08:42.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f7e7802a070 con 0x7f7e88103980 2024-04-26T16:08:42.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e88108a20 con 0x7f7e88103980 2024-04-26T16:08:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7e7003bd70 0x7f7e7003e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7e7804ea10 con 0x7f7e88103980 2024-04-26T16:08:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 10 client.14176.objecter ms_dispatch 0x7f7e88001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 3 client.14176.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 3 client.14176.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:42.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 20 client.14176.objecter dump_active .. 0 homeless 2024-04-26T16:08:42.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e85d9b700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7e7003bd70 0x7f7e7003e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:42.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e777fe700 10 client.14176.objecter ms_handle_connect 0x7f7e7003bd70 2024-04-26T16:08:42.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.658+0000 7f7e85d9b700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7e7003bd70 0x7f7e7003e230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:42.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.662+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7e78013070 con 0x7f7e88103980 2024-04-26T16:08:42.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.882+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f7e88063860 con 0x7f7e7003bd70 2024-04-26T16:08:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.886+0000 7f7e777fe700 1 -- 172.21.15.50:0/1565428801 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7e88063860 con 0x7f7e7003bd70 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 10 librados: watch_flush enter 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 10 librados: watch_flush exit 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 20 client.14176.objecter shutdown clearing up homeless session... 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 10 client.14176.objecter successfully canceled tick 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7e7003bd70 msgr2=0x7f7e7003e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7e7003bd70 0x7f7e7003e230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 msgr2=0x7f7e88197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88197b00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7e7800b5c0 tx=0x7f7e78005710 comp rx=0 tx=0).stop 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 shutdown_connections 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7e7003bd70 0x7f7e7003e230 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:42.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 --2- 172.21.15.50:0/1565428801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e88103980 0x7f7e88197b00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:42.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 >> 172.21.15.50:0/1565428801 conn(0x7f7e880fd330 msgr2=0x7f7e880fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:42.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 shutdown_connections 2024-04-26T16:08:42.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 -- 172.21.15.50:0/1565428801 wait complete. 2024-04-26T16:08:42.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:42.890+0000 7f7e8cdf5700 1 librados: shutdown 2024-04-26T16:08:43.504 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-04-26T16:08:43.505 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:08:43.505 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-04-26T16:08:43.511 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:08:43.511 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:08:43.564 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-04-26T16:08:43.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch host add smithi190 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: cluster 2024-04-26T16:08:42.606788+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: cluster 2024-04-26T16:08:42.606788+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.885744+0000 mgr.y (mgr.14152) 14 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.885744+0000 mgr.y (mgr.14152) 14 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.889399+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.889399+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.937212+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.937212+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.938676+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:08:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.938676+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.939309+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:42.939309+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:43.652685+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:43.652685+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:43.657010+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:43.657010+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:43.661048+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:44.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:43 smithi050 bash[21463]: audit 2024-04-26T16:08:43.661048+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:42.940350+0000 mgr.y (mgr.14152) 15 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:42.940350+0000 mgr.y (mgr.14152) 15 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:43.054531+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:43.054531+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:43.269511+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:43.269511+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:43.434503+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:08:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:44 smithi050 bash[21463]: cephadm 2024-04-26T16:08:43.434503+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:08:46.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:45 smithi050 bash[21463]: cluster 2024-04-26T16:08:44.607210+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:46.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:45 smithi050 bash[21463]: cluster 2024-04-26T16:08:44.607210+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:48.280 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:08:48.303 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:47 smithi050 bash[21463]: cluster 2024-04-26T16:08:46.607649+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:48.303 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:47 smithi050 bash[21463]: cluster 2024-04-26T16:08:46.607649+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:50.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:49 smithi050 bash[21463]: cluster 2024-04-26T16:08:48.608094+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:50.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:49 smithi050 bash[21463]: cluster 2024-04-26T16:08:48.608094+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:51.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/4100066237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 msgr2=0x7f988c102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 --2- 172.21.15.50:0/4100066237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c102370 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f987c009910 tx=0x7f987c009c20 comp rx=0 tx=0).stop 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/4100066237 shutdown_connections 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 --2- 172.21.15.50:0/4100066237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c102370 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/4100066237 >> 172.21.15.50:0/4100066237 conn(0x7f988c0fd820 msgr2=0x7f988c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/4100066237 shutdown_connections 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/4100066237 wait complete. 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 librados: starting msgr at 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 librados: starting objecter 2024-04-26T16:08:51.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 Processor -- start 2024-04-26T16:08:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- start start 2024-04-26T16:08:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 librados: setting wanted keys 2024-04-26T16:08:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 librados: calling monclient init 2024-04-26T16:08:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f988c10b6d0 con 0x7f988c101f90 2024-04-26T16:08:51.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f988affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f988affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46438/0 (socket says 172.21.15.50:46438) 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f988affd700 1 -- 172.21.15.50:0/2017941661 learned_addr learned my addr 172.21.15.50:0/2017941661 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f988affd700 1 -- 172.21.15.50:0/2017941661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c198410 con 0x7f988c101f90 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f988affd700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c197ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f987c005660 tx=0x7f987c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f9873fff700 10 client.?.objecter ms_handle_connect 0x7f988c101f90 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f9873fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f987c026070 con 0x7f988c101f90 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f987c00b6f0 con 0x7f988c101f90 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c198700 con 0x7f988c101f90 2024-04-26T16:08:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f987c0037d0 con 0x7f988c101f90 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 10 client.14178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c19c7a0 con 0x7f988c101f90 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.117+0000 7f989129a700 1 librados: init done 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f987c029070 con 0x7f988c101f90 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f987403bd70 0x7f987403e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f987c01e070 con 0x7f988c101f90 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 10 client.14178.objecter ms_dispatch 0x7f988c001550 osd_map(3..3 src has 1..3) 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 3 client.14178.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:08:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 3 client.14178.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:08:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f988c103470 con 0x7f988c101f90 2024-04-26T16:08:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 20 client.14178.objecter dump_active .. 0 homeless 2024-04-26T16:08:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f988a7fc700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f987403bd70 0x7f987403e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:08:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f988a7fc700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f987403bd70 0x7f987403e230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9880006fd0 tx=0x7f9880006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:08:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.121+0000 7f9873fff700 10 client.14178.objecter ms_handle_connect 0x7f987403bd70 2024-04-26T16:08:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.125+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f987c030020 con 0x7f988c101f90 2024-04-26T16:08:51.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:51.349+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) -- 0x7f988c064a80 con 0x7f987403bd70 2024-04-26T16:08:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:51 smithi050 bash[21463]: cluster 2024-04-26T16:08:50.608451+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:51 smithi050 bash[21463]: cluster 2024-04-26T16:08:50.608451+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:53.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:52 smithi050 bash[21463]: audit 2024-04-26T16:08:51.355884+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:53.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:52 smithi050 bash[21463]: audit 2024-04-26T16:08:51.355884+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:08:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:53 smithi050 bash[21463]: cephadm 2024-04-26T16:08:52.066385+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi190 2024-04-26T16:08:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:53 smithi050 bash[21463]: cephadm 2024-04-26T16:08:52.066385+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi190 2024-04-26T16:08:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:53 smithi050 bash[21463]: cluster 2024-04-26T16:08:52.608870+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:54.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:53 smithi050 bash[21463]: cluster 2024-04-26T16:08:52.608870+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:56.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:56.121+0000 7f9890a99700 10 client.14178.objecter tick 2024-04-26T16:08:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:55 smithi050 bash[21463]: cluster 2024-04-26T16:08:54.609260+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:55 smithi050 bash[21463]: cluster 2024-04-26T16:08:54.609260+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:57.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.449+0000 7f9873fff700 1 -- 172.21.15.50:0/2017941661 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f988c064a80 con 0x7f987403bd70 2024-04-26T16:08:57.450 INFO:teuthology.orchestra.run.smithi050.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-04-26T16:08:57.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 10 librados: watch_flush enter 2024-04-26T16:08:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 10 librados: watch_flush exit 2024-04-26T16:08:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 20 client.14178.objecter shutdown clearing up homeless session... 2024-04-26T16:08:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 10 client.14178.objecter successfully canceled tick 2024-04-26T16:08:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f987403bd70 msgr2=0x7f987403e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:57.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f987403bd70 0x7f987403e230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9880006fd0 tx=0x7f9880006e40 comp rx=0 tx=0).stop 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 msgr2=0x7f988c197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c197ed0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f987c005660 tx=0x7f987c005740 comp rx=0 tx=0).stop 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 shutdown_connections 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f987403bd70 0x7f987403e230 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 --2- 172.21.15.50:0/2017941661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f988c101f90 0x7f988c197ed0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 >> 172.21.15.50:0/2017941661 conn(0x7f988c0fd820 msgr2=0x7f988c06b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 shutdown_connections 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 -- 172.21.15.50:0/2017941661 wait complete. 2024-04-26T16:08:57.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:08:57.453+0000 7f989129a700 1 librados: shutdown 2024-04-26T16:08:58.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-26T16:08:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:57 smithi050 bash[21463]: cluster 2024-04-26T16:08:56.609528+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:57 smithi050 bash[21463]: cluster 2024-04-26T16:08:56.609528+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:08:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:57 smithi050 bash[21463]: audit 2024-04-26T16:08:57.453076+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:57 smithi050 bash[21463]: audit 2024-04-26T16:08:57.453076+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:57 smithi050 bash[21463]: audit 2024-04-26T16:08:57.499451+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:57 smithi050 bash[21463]: audit 2024-04-26T16:08:57.499451+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:08:59.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:58 smithi050 bash[21463]: cephadm 2024-04-26T16:08:57.453985+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Added host smithi190 2024-04-26T16:08:59.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:58 smithi050 bash[21463]: cephadm 2024-04-26T16:08:57.453985+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Added host smithi190 2024-04-26T16:08:59.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:58 smithi050 bash[21463]: audit 2024-04-26T16:08:57.970999+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:08:59.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:58 smithi050 bash[21463]: audit 2024-04-26T16:08:57.970999+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:00.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:59 smithi050 bash[21463]: cluster 2024-04-26T16:08:58.609925+0000 mgr.y (mgr.14152) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:00.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:59 smithi050 bash[21463]: cluster 2024-04-26T16:08:58.609925+0000 mgr.y (mgr.14152) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:00.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:59 smithi050 bash[21463]: audit 2024-04-26T16:08:59.496700+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:08:59 smithi050 bash[21463]: audit 2024-04-26T16:08:59.496700+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:02.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:01 smithi050 bash[21463]: cluster 2024-04-26T16:09:00.610497+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:02.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:01 smithi050 bash[21463]: cluster 2024-04-26T16:09:00.610497+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:02.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:01 smithi050 bash[21463]: audit 2024-04-26T16:09:00.746952+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:02.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:01 smithi050 bash[21463]: audit 2024-04-26T16:09:00.746952+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:02.842 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:09:04.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:03 smithi050 bash[21463]: cluster 2024-04-26T16:09:02.610921+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:04.034 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:03 smithi050 bash[21463]: cluster 2024-04-26T16:09:02.610921+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/2278446628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 msgr2=0x7fd130075c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 --2- 172.21.15.50:0/2278446628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd130075c30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd120009910 tx=0x7fd120009c20 comp rx=0 tx=0).stop 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/2278446628 shutdown_connections 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 --2- 172.21.15.50:0/2278446628 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd130075c30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/2278446628 >> 172.21.15.50:0/2278446628 conn(0x7fd1300fe960 msgr2=0x7fd130100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/2278446628 shutdown_connections 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/2278446628 wait complete. 2024-04-26T16:09:05.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 librados: starting msgr at 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 librados: starting objecter 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 Processor -- start 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- start start 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 librados: setting wanted keys 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 librados: calling monclient init 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd13019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd13010ea50 con 0x7fd130075850 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd12e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd13019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd12e59c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd13019a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36158/0 (socket says 172.21.15.50:36158) 2024-04-26T16:09:05.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd12e59c700 1 -- 172.21.15.50:0/354098072 learned_addr learned my addr 172.21.15.50:0/354098072 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:09:05.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd12e59c700 1 -- 172.21.15.50:0/354098072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13019a660 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd12e59c700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd13019a120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd120005060 tx=0x7fd120005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd1277fe700 10 client.?.objecter ms_handle_connect 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd1277fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd120026070 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd12000b750 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd13019a8f0 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd1200037b0 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 10 client.14180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd13019ad50 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 librados: init done 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7fd120029070 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.421+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd130076f00 con 0x7fd130075850 2024-04-26T16:09:05.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd11c03bd20 0x7fd11c03e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd12001e070 con 0x7fd130075850 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 10 client.14180.objecter ms_dispatch 0x7fd130001550 osd_map(3..3 src has 1..3) 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 3 client.14180.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 3 client.14180.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 20 client.14180.objecter dump_active .. 0 homeless 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd12dd9b700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd11c03bd20 0x7fd11c03e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd1277fe700 10 client.14180.objecter ms_handle_connect 0x7fd11c03bd20 2024-04-26T16:09:05.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.425+0000 7fd12dd9b700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd11c03bd20 0x7fd11c03e1e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd118006fd0 tx=0x7fd118006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.429+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd120013070 con 0x7fd130075850 2024-04-26T16:09:05.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.661+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd13019b480 con 0x7fd11c03bd20 2024-04-26T16:09:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd1277fe700 1 -- 172.21.15.50:0/354098072 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fd13019b480 con 0x7fd11c03bd20 2024-04-26T16:09:05.663 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:09:05.663 INFO:teuthology.orchestra.run.smithi050.stdout:[{"addr": "172.21.15.50", "hostname": "smithi050", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 10 librados: watch_flush enter 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 10 librados: watch_flush exit 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 20 client.14180.objecter shutdown clearing up homeless session... 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 10 client.14180.objecter successfully canceled tick 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd11c03bd20 msgr2=0x7fd11c03e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd11c03bd20 0x7fd11c03e1e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd118006fd0 tx=0x7fd118006e40 comp rx=0 tx=0).stop 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 msgr2=0x7fd13019a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd13019a120 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd120005060 tx=0x7fd120005710 comp rx=0 tx=0).stop 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 shutdown_connections 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd11c03bd20 0x7fd11c03e1e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:05.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 --2- 172.21.15.50:0/354098072 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd130075850 0x7fd13019a120 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:05.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 >> 172.21.15.50:0/354098072 conn(0x7fd1300fe960 msgr2=0x7fd13010c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:05.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 shutdown_connections 2024-04-26T16:09:05.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 -- 172.21.15.50:0/354098072 wait complete. 2024-04-26T16:09:05.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:05.665+0000 7fd134a5d700 1 librados: shutdown 2024-04-26T16:09:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:05 smithi050 bash[21463]: cluster 2024-04-26T16:09:04.611335+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:05 smithi050 bash[21463]: cluster 2024-04-26T16:09:04.611335+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:06.386 INFO:tasks.cephadm:Setting crush tunables to jewel 2024-04-26T16:09:06.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd crush tunables jewel 2024-04-26T16:09:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:05.667159+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:05.667159+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.233066+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.233066+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.238549+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.238549+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.244838+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.244838+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.250090+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.250090+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.251691+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.251691+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.253816+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.253816+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:07.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.255774+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:07.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:06 smithi050 bash[21463]: audit 2024-04-26T16:09:06.255774+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:08.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.258704+0000 mgr.y (mgr.14152) 34 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:09:08.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.258704+0000 mgr.y (mgr.14152) 34 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.475698+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.475698+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cluster 2024-04-26T16:09:06.611587+0000 mgr.y (mgr.14152) 36 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cluster 2024-04-26T16:09:06.611587+0000 mgr.y (mgr.14152) 36 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.641096+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.641096+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.806126+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: cephadm 2024-04-26T16:09:06.806126+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: audit 2024-04-26T16:09:06.976388+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: audit 2024-04-26T16:09:06.976388+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: audit 2024-04-26T16:09:06.981884+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: audit 2024-04-26T16:09:06.981884+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: audit 2024-04-26T16:09:06.987939+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:07 smithi050 bash[21463]: audit 2024-04-26T16:09:06.987939+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:09 smithi050 bash[21463]: cluster 2024-04-26T16:09:08.611948+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:09 smithi050 bash[21463]: cluster 2024-04-26T16:09:08.611948+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:11.101 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:09:12.008 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:11 smithi050 bash[21463]: cluster 2024-04-26T16:09:10.612354+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:12.008 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:11 smithi050 bash[21463]: cluster 2024-04-26T16:09:10.612354+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:13.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/948748199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 msgr2=0x7fb898075b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:13.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 --2- 172.21.15.50:0/948748199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb898075b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb888009910 tx=0x7fb888009c20 comp rx=0 tx=0).stop 2024-04-26T16:09:13.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/948748199 shutdown_connections 2024-04-26T16:09:13.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 --2- 172.21.15.50:0/948748199 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb898075b20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/948748199 >> 172.21.15.50:0/948748199 conn(0x7fb8980fe960 msgr2=0x7fb898100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/948748199 shutdown_connections 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/948748199 wait complete. 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 librados: starting msgr at 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 librados: starting objecter 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 Processor -- start 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- start start 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 librados: setting wanted keys 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 librados: calling monclient init 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb89819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb89810ea40 con 0x7fb898075740 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb89819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb89819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33514/0 (socket says 172.21.15.50:33514) 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89659c700 1 -- 172.21.15.50:0/688346980 learned_addr learned my addr 172.21.15.50:0/688346980 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89659c700 1 -- 172.21.15.50:0/688346980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89819a670 con 0x7fb898075740 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89659c700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb89819a130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb888005060 tx=0x7fb888005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb88f7fe700 10 client.?.objecter ms_handle_connect 0x7fb898075740 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb88f7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb888026070 con 0x7fb898075740 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb88800b750 con 0x7fb898075740 2024-04-26T16:09:13.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8880037b0 con 0x7fb898075740 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89819a900 con 0x7fb898075740 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89819ad60 con 0x7fb898075740 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.617+0000 7fb89cff5700 1 librados: init done 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7fb888029070 con 0x7fb898075740 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb898076e80 con 0x7fb898075740 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fb88403bd20 0x7fb88403e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb88801e070 con 0x7fb898075740 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 10 client.14182.objecter ms_dispatch 0x7fb898001550 osd_map(3..3 src has 1..3) 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 3 client.14182.objecter handle_osd_map got epochs [3,3] > 0 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 3 client.14182.objecter handle_osd_map decoding full epoch 3 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 20 client.14182.objecter dump_active .. 0 homeless 2024-04-26T16:09:13.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb895d9b700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fb88403bd20 0x7fb88403e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:13.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb895d9b700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fb88403bd20 0x7fb88403e1e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb880006fd0 tx=0x7fb880006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:13.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.621+0000 7fb88f7fe700 10 client.14182.objecter ms_handle_connect 0x7fb88403bd20 2024-04-26T16:09:13.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.625+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb888013070 con 0x7fb898075740 2024-04-26T16:09:13.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:13.853+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "jewel"} v 0) -- 0x7fb89819b490 con 0x7fb898075740 2024-04-26T16:09:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:13 smithi050 bash[21463]: cluster 2024-04-26T16:09:12.612814+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:13 smithi050 bash[21463]: cluster 2024-04-26T16:09:12.612814+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:14.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.757+0000 7fb88f7fe700 1 -- 172.21.15.50:0/688346980 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "jewel"}]=0 adjusted tunables profile to jewel v4) ==== 120+0+0 (secure 0 0 0) 0x7fb888030090 con 0x7fb898075740 2024-04-26T16:09:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 10 librados: watch_flush enter 2024-04-26T16:09:14.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 10 librados: watch_flush exit 2024-04-26T16:09:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 20 client.14182.objecter shutdown clearing up homeless session... 2024-04-26T16:09:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 10 client.14182.objecter successfully canceled tick 2024-04-26T16:09:14.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fb88403bd20 msgr2=0x7fb88403e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fb88403bd20 0x7fb88403e1e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb880006fd0 tx=0x7fb880006e40 comp rx=0 tx=0).stop 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 msgr2=0x7fb89819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb89819a130 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb888005060 tx=0x7fb888005710 comp rx=0 tx=0).stop 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 shutdown_connections 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fb88403bd20 0x7fb88403e1e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 --2- 172.21.15.50:0/688346980 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb898075740 0x7fb89819a130 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 >> 172.21.15.50:0/688346980 conn(0x7fb8980fe960 msgr2=0x7fb89810c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 shutdown_connections 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 -- 172.21.15.50:0/688346980 wait complete. 2024-04-26T16:09:14.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:14.761+0000 7fb89cff5700 1 librados: shutdown 2024-04-26T16:09:14.761 INFO:teuthology.orchestra.run.smithi050.stderr:adjusted tunables profile to jewel 2024-04-26T16:09:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:14 smithi050 bash[21463]: audit 2024-04-26T16:09:13.858013+0000 mon.a (mon.0) 128 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-26T16:09:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:14 smithi050 bash[21463]: audit 2024-04-26T16:09:13.858013+0000 mon.a (mon.0) 128 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-26T16:09:15.413 INFO:tasks.cephadm:Adding mon.a on smithi050 2024-04-26T16:09:15.413 INFO:tasks.cephadm:Adding mon.c on smithi050 2024-04-26T16:09:15.413 INFO:tasks.cephadm:Adding mon.b on smithi190 2024-04-26T16:09:15.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b' 2024-04-26T16:09:16.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:15 smithi050 bash[21463]: cluster 2024-04-26T16:09:14.613242+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:16.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:15 smithi050 bash[21463]: cluster 2024-04-26T16:09:14.613242+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:16.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:15 smithi050 bash[21463]: audit 2024-04-26T16:09:14.760226+0000 mon.a (mon.0) 129 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-26T16:09:16.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:15 smithi050 bash[21463]: audit 2024-04-26T16:09:14.760226+0000 mon.a (mon.0) 129 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-26T16:09:16.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:15 smithi050 bash[21463]: cluster 2024-04-26T16:09:14.764928+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:16.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:15 smithi050 bash[21463]: cluster 2024-04-26T16:09:14.764928+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:16.627 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:18.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:17 smithi050 bash[21463]: cluster 2024-04-26T16:09:16.613698+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:18.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:17 smithi050 bash[21463]: cluster 2024-04-26T16:09:16.613698+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:18.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- 172.21.15.190:0/4052000856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 msgr2=0x7f7a54103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:18.753 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 --2- 172.21.15.190:0/4052000856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54103470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009910 tx=0x7f7a44009c20 comp rx=0 tx=0).stop 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- 172.21.15.190:0/4052000856 shutdown_connections 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 --2- 172.21.15.190:0/4052000856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54103470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- 172.21.15.190:0/4052000856 >> 172.21.15.190:0/4052000856 conn(0x7f7a540fe960 msgr2=0x7f7a54100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- 172.21.15.190:0/4052000856 shutdown_connections 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- 172.21.15.190:0/4052000856 wait complete. 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 librados: starting msgr at 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 librados: starting objecter 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 Processor -- start 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- start start 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 librados: setting wanted keys 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 librados: calling monclient init 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a58c9f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7a5410c7d0 con 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a5259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.190:51954/0 (socket says 172.21.15.190:51954) 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a5259c700 1 -- 172.21.15.190:0/2716924532 learned_addr learned my addr 172.21.15.190:0/2716924532 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a5259c700 1 -- 172.21.15.190:0/2716924532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a54077360 con 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a5259c700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54075720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a44005690 tx=0x7f7a44005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a437fe700 10 client.?.objecter ms_handle_connect 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.749+0000 7f7a437fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a44026070 con 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a54077650 con 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a58c9f700 10 client.14184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7a4400b6f0 con 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a54075e30 con 0x7f7a54103090 2024-04-26T16:09:18.756 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a58c9f700 1 librados: init done 2024-04-26T16:09:18.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7a44003680 con 0x7f7a54103090 2024-04-26T16:09:18.757 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a54104570 con 0x7f7a54103090 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f7a44029070 con 0x7f7a54103090 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a3c03bd70 0x7f7a3c03e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7a4401e070 con 0x7f7a54103090 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 10 client.14184.objecter ms_dispatch 0x7f7a54001550 osd_map(4..4 src has 1..4) 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 3 client.14184.objecter handle_osd_map got epochs [4,4] > 0 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 3 client.14184.objecter handle_osd_map decoding full epoch 4 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 20 client.14184.objecter dump_active .. 0 homeless 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a51d9b700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a3c03bd70 0x7f7a3c03e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a437fe700 10 client.14184.objecter ms_handle_connect 0x7f7a3c03bd70 2024-04-26T16:09:18.758 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.753+0000 7f7a51d9b700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a3c03bd70 0x7f7a3c03e230 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a48006fd0 tx=0x7f7a48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:18.771 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.769+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7a44030020 con 0x7f7a54103090 2024-04-26T16:09:18.995 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:18.989+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}) -- 0x7f7a54064a80 con 0x7f7a3c03bd70 2024-04-26T16:09:19.009 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a437fe700 1 -- 172.21.15.190:0/2716924532 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f7a54064a80 con 0x7f7a3c03bd70 2024-04-26T16:09:19.009 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-04-26T16:09:19.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 10 librados: watch_flush enter 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 10 librados: watch_flush exit 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 20 client.14184.objecter shutdown clearing up homeless session... 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 10 client.14184.objecter successfully canceled tick 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a3c03bd70 msgr2=0x7f7a3c03e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a3c03bd70 0x7f7a3c03e230 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a48006fd0 tx=0x7f7a48006e40 comp rx=0 tx=0).stop 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 msgr2=0x7f7a54075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54075720 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a44005690 tx=0x7f7a44005770 comp rx=0 tx=0).stop 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 shutdown_connections 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a3c03bd70 0x7f7a3c03e230 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 --2- 172.21.15.190:0/2716924532 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a54103090 0x7f7a54075720 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 >> 172.21.15.190:0/2716924532 conn(0x7f7a540fe960 msgr2=0x7f7a54109ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 shutdown_connections 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 -- 172.21.15.190:0/2716924532 wait complete. 2024-04-26T16:09:19.012 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:19.009+0000 7f7a58c9f700 1 librados: shutdown 2024-04-26T16:09:19.545 DEBUG:teuthology.orchestra.run.smithi050:mon.c> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.c.service 2024-04-26T16:09:19.548 DEBUG:teuthology.orchestra.run.smithi190:mon.b> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.b.service 2024-04-26T16:09:19.550 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T16:09:19.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T16:09:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: cluster 2024-04-26T16:09:18.614073+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: cluster 2024-04-26T16:09:18.614073+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.012846+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.012846+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.049770+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.049770+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.053383+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.053383+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.055168+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.055168+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.063396+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.063396+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.067558+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.067558+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.069236+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:19 smithi050 bash[21463]: audit 2024-04-26T16:09:19.069236+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:20.775 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:21.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:20 smithi050 bash[21463]: audit 2024-04-26T16:09:19.002120+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:21.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:20 smithi050 bash[21463]: audit 2024-04-26T16:09:19.002120+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:21.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:20 smithi050 bash[21463]: cephadm 2024-04-26T16:09:19.007878+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b;count:3 2024-04-26T16:09:21.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:20 smithi050 bash[21463]: cephadm 2024-04-26T16:09:19.007878+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b;count:3 2024-04-26T16:09:21.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:20 smithi050 bash[21463]: cephadm 2024-04-26T16:09:19.071418+0000 mgr.y (mgr.14152) 47 : cephadm [INF] Deploying daemon mon.b on smithi190 2024-04-26T16:09:21.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:20 smithi050 bash[21463]: cephadm 2024-04-26T16:09:19.071418+0000 mgr.y (mgr.14152) 47 : cephadm [INF] Deploying daemon mon.b on smithi190 2024-04-26T16:09:22.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:21 smithi050 bash[21463]: cluster 2024-04-26T16:09:20.614540+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:22.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:21 smithi050 bash[21463]: cluster 2024-04-26T16:09:20.614540+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:23.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.853+0000 7f28ca6a5700 1 -- 172.21.15.190:0/2268479172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 msgr2=0x7f28c41023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:23.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.853+0000 7f28ca6a5700 1 --2- 172.21.15.190:0/2268479172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c41023b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009910 tx=0x7f28ac009c20 comp rx=0 tx=0).stop 2024-04-26T16:09:23.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.853+0000 7f28ca6a5700 1 -- 172.21.15.190:0/2268479172 shutdown_connections 2024-04-26T16:09:23.859 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.853+0000 7f28ca6a5700 1 --2- 172.21.15.190:0/2268479172 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c41023b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:23.860 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.853+0000 7f28ca6a5700 1 -- 172.21.15.190:0/2268479172 >> 172.21.15.190:0/2268479172 conn(0x7f28c4077880 msgr2=0x7f28c4077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:23.860 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- 172.21.15.190:0/2268479172 shutdown_connections 2024-04-26T16:09:23.860 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- 172.21.15.190:0/2268479172 wait complete. 2024-04-26T16:09:23.860 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 librados: starting msgr at 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 librados: starting objecter 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 Processor -- start 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- start start 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 librados: setting wanted keys 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 librados: calling monclient init 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c4196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f28c410b6c0 con 0x7f28c4101fd0 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c4196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c3fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c4196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.190:57532/0 (socket says 172.21.15.190:57532) 2024-04-26T16:09:23.861 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c3fff700 1 -- 172.21.15.190:0/826708696 learned_addr learned my addr 172.21.15.190:0/826708696 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:09:23.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c3fff700 1 -- 172.21.15.190:0/826708696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c4197310 con 0x7f28c4101fd0 2024-04-26T16:09:23.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c3fff700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c4196dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28ac005660 tx=0x7f28ac005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:23.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c17fa700 10 client.?.objecter ms_handle_connect 0x7f28c4101fd0 2024-04-26T16:09:23.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c17fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:23.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28ac026070 con 0x7f28c4101fd0 2024-04-26T16:09:23.862 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c4197600 con 0x7f28c4101fd0 2024-04-26T16:09:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 10 client.14186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c419b650 con 0x7f28c4101fd0 2024-04-26T16:09:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 librados: init done 2024-04-26T16:09:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f28ac00b6f0 con 0x7f28c4101fd0 2024-04-26T16:09:23.863 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28ac0037d0 con 0x7f28c4101fd0 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.857+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a4005120 con 0x7f28c4101fd0 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f28ac029070 con 0x7f28c4101fd0 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f28b003bd70 0x7f28b003e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f28ac01e070 con 0x7f28c4101fd0 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 10 client.14186.objecter ms_dispatch 0x7f28c4001550 osd_map(4..4 src has 1..4) 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 3 client.14186.objecter handle_osd_map got epochs [4,4] > 0 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 3 client.14186.objecter handle_osd_map decoding full epoch 4 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 20 client.14186.objecter dump_active .. 0 homeless 2024-04-26T16:09:23.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c37fe700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f28b003bd70 0x7f28b003e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:23.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c17fa700 10 client.14186.objecter ms_handle_connect 0x7f28b003bd70 2024-04-26T16:09:23.865 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.861+0000 7f28c37fe700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f28b003bd70 0x7f28b003e230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f28b4006fd0 tx=0x7f28b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:23.878 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:23.873+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f28ac013070 con 0x7f28c4101fd0 2024-04-26T16:09:24.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:23 smithi050 bash[21463]: cluster 2024-04-26T16:09:22.614984+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:24.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:23 smithi050 bash[21463]: cluster 2024-04-26T16:09:22.614984+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:24.199 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.193+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f28a4005410 con 0x7f28c4101fd0 2024-04-26T16:09:24.200 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28c17fa700 1 -- 172.21.15.190:0/826708696 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f28ac030020 con 0x7f28c4101fd0 2024-04-26T16:09:24.200 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-26T16:09:24.200 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","modified":"2024-04-26T16:07:00.226385Z","created":"2024-04-26T16:07:00.226385Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 10 librados: watch_flush enter 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 10 librados: watch_flush exit 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 20 client.14186.objecter shutdown clearing up homeless session... 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 10 client.14186.objecter successfully canceled tick 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f28b003bd70 msgr2=0x7f28b003e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f28b003bd70 0x7f28b003e230 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f28b4006fd0 tx=0x7f28b4006e40 comp rx=0 tx=0).stop 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 msgr2=0x7f28c4196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c4196dd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f28ac005660 tx=0x7f28ac005740 comp rx=0 tx=0).stop 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 shutdown_connections 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f28b003bd70 0x7f28b003e230 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 --2- 172.21.15.190:0/826708696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f28c4101fd0 0x7f28c4196dd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 >> 172.21.15.190:0/826708696 conn(0x7f28c4077880 msgr2=0x7f28c4108320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 shutdown_connections 2024-04-26T16:09:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 -- 172.21.15.190:0/826708696 wait complete. 2024-04-26T16:09:24.204 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:24.197+0000 7f28ca6a5700 1 librados: shutdown 2024-04-26T16:09:24.204 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-04-26T16:09:25.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.204674+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? 172.21.15.190:0/826708696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.204674+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? 172.21.15.190:0/826708696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.481321+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.481321+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.487140+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.487140+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.507839+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.507839+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.509249+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.509249+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.510936+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:24 smithi050 bash[21463]: audit 2024-04-26T16:09:24.510936+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.641+0000 7f3890ccbc80 1 mon.b@-1(???) e0 preinit fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:25.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3882429700 0 mon.b@-1(synchronizing).mds e1 new map 2024-04-26T16:09:25.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3882429700 0 mon.b@-1(synchronizing).mds e1 print_map 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: e1 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: enable_multiple, ever_enabled_multiple: 1,1 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: legacy client fscid: -1 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: No filesystems configured 2024-04-26T16:09:25.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f387dc20700 4 rocksdb: [db/db_impl/db_impl_write.cc:2098] [default] New memtable created with log file: #13. Immutable memtables: 0. 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3882429700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3886c32700 4 rocksdb: (Original Log Time 2024/04/26-16:09:25.672988) [db/db_impl/db_impl_compaction_flush.cc:2832] Calling FlushMemTableToOutputFile with column family [default], flush slots available 1, compaction slots available 1, flush slots scheduled 1, compaction slots scheduled 0 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3886c32700 4 rocksdb: [db/flush_job.cc:856] [default] [JOB 3] Flushing memtable with next log file: 13 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3886c32700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714147765673072, "job": 3, "event": "flush_started", "num_memtables": 1, "num_entries": 1149, "num_deletes": 24, "total_data_size": 2317332, "memory_usage": 2341424, "flush_reason": "Manual Compaction"} 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.665+0000 7f3886c32700 4 rocksdb: [db/flush_job.cc:885] [default] [JOB 3] Level-0 flush table #14: started 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T16:09:25.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.669+0000 7f3882429700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:42.606788+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:42.606788+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.885744+0000 mgr.y (mgr.14152) 14 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.885744+0000 mgr.y (mgr.14152) 14 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.889399+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.889399+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.937212+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.937212+0000 mon.a (mon.0) 107 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.938676+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.938676+0000 mon.a (mon.0) 108 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.939309+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:42.939309+0000 mon.a (mon.0) 109 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:25.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:43.652685+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:43.652685+0000 mon.a (mon.0) 110 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:43.657010+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:43.657010+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:43.661048+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:43.661048+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:42.940350+0000 mgr.y (mgr.14152) 15 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:42.940350+0000 mgr.y (mgr.14152) 15 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:43.054531+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:43.054531+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:25.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:43.269511+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:25.684 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:43.269511+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:43.434503+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:43.434503+0000 mgr.y (mgr.14152) 18 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:44.607210+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:44.607210+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:46.607649+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:46.607649+0000 mgr.y (mgr.14152) 20 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:48.608094+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:48.608094+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:50.608451+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:50.608451+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:51.355884+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:51.355884+0000 mgr.y (mgr.14152) 23 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:52.066385+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi190 2024-04-26T16:09:25.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:52.066385+0000 mgr.y (mgr.14152) 24 : cephadm [INF] Deploying cephadm binary to smithi190 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:52.608870+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:52.608870+0000 mgr.y (mgr.14152) 25 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:54.609260+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:54.609260+0000 mgr.y (mgr.14152) 26 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:56.609528+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:56.609528+0000 mgr.y (mgr.14152) 27 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:57.453076+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:57.453076+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:57.499451+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:57.499451+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:57.453985+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Added host smithi190 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:08:57.453985+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Added host smithi190 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:57.970999+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:57.970999+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:58.609925+0000 mgr.y (mgr.14152) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:08:58.609925+0000 mgr.y (mgr.14152) 29 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:59.496700+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:08:59.496700+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:00.610497+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:00.610497+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:00.746952+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:00.746952+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:02.610921+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:02.610921+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:04.611335+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:04.611335+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:05.667159+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:09:25.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:05.667159+0000 mgr.y (mgr.14152) 33 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.233066+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.233066+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.238549+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.238549+0000 mon.a (mon.0) 119 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.244838+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.244838+0000 mon.a (mon.0) 120 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.250090+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.250090+0000 mon.a (mon.0) 121 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.251691+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.251691+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.253816+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.253816+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.255774+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.255774+0000 mon.a (mon.0) 124 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.258704+0000 mgr.y (mgr.14152) 34 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.258704+0000 mgr.y (mgr.14152) 34 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.475698+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.475698+0000 mgr.y (mgr.14152) 35 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:06.611587+0000 mgr.y (mgr.14152) 36 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:06.611587+0000 mgr.y (mgr.14152) 36 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.641096+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.641096+0000 mgr.y (mgr.14152) 37 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.806126+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:09:25.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:06.806126+0000 mgr.y (mgr.14152) 38 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.976388+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.976388+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.981884+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.981884+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.987939+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:06.987939+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:08.611948+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:08.611948+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:10.612354+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:10.612354+0000 mgr.y (mgr.14152) 40 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:12.612814+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:12.612814+0000 mgr.y (mgr.14152) 41 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:13.858013+0000 mon.a (mon.0) 128 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:13.858013+0000 mon.a (mon.0) 128 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "jewel"}]: dispatch 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:14.613242+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:14.613242+0000 mgr.y (mgr.14152) 42 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:14.760226+0000 mon.a (mon.0) 129 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:14.760226+0000 mon.a (mon.0) 129 : audit [INF] from='client.? 172.21.15.50:0/688346980' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "jewel"}]': finished 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:14.764928+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:14.764928+0000 mon.a (mon.0) 130 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:16.613698+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:16.613698+0000 mgr.y (mgr.14152) 43 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:18.614073+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:18.614073+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.012846+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.012846+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.049770+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.049770+0000 mon.a (mon.0) 132 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.053383+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.053383+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.055168+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.055168+0000 mon.a (mon.0) 134 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.063396+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.063396+0000 mon.a (mon.0) 135 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.067558+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:25.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.067558+0000 mon.a (mon.0) 136 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.069236+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.069236+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.002120+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:19.002120+0000 mgr.y (mgr.14152) 45 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:19.007878+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b;count:3 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:19.007878+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Saving service mon spec with placement smithi050:172.21.15.50=a;smithi050:[v2:172.21.15.50:3301,v1:172.21.15.50:6790]=c;smithi190:172.21.15.190=b;count:3 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:19.071418+0000 mgr.y (mgr.14152) 47 : cephadm [INF] Deploying daemon mon.b on smithi190 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cephadm 2024-04-26T16:09:19.071418+0000 mgr.y (mgr.14152) 47 : cephadm [INF] Deploying daemon mon.b on smithi190 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:20.614540+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:20.614540+0000 mgr.y (mgr.14152) 48 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:22.614984+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: cluster 2024-04-26T16:09:22.614984+0000 mgr.y (mgr.14152) 49 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.204674+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? 172.21.15.190:0/826708696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.204674+0000 mon.a (mon.0) 138 : audit [DBG] from='client.? 172.21.15.190:0/826708696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.481321+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.481321+0000 mon.a (mon.0) 139 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.487140+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.487140+0000 mon.a (mon.0) 140 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.507839+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.507839+0000 mon.a (mon.0) 141 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.509249+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.509249+0000 mon.a (mon.0) 142 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.510936+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: audit 2024-04-26T16:09:24.510936+0000 mon.a (mon.0) 143 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:25.695 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.673+0000 7f3882429700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-26T16:09:25.695 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.673+0000 7f3886c32700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714147765678423, "cf_name": "default", "job": 3, "event": "table_file_creation", "file_number": 14, "file_size": 2043763, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 6, "largest_seqno": 1154, "table_properties": {"data_size": 2037938, "index_size": 2581, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2181, "raw_key_size": 25583, "raw_average_key_size": 30, "raw_value_size": 2020305, "raw_average_value_size": 2385, "num_data_blocks": 121, "num_entries": 847, "num_filter_entries": 847, "num_deletions": 24, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714147765, "oldest_key_time": 1714147765, "file_creation_time": 1714147765, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "38c4d9bd-390d-47fd-9ffb-6269c5b10956", "db_session_id": "3BZ5S6B776071KM9YNWG", "orig_file_number": 14, "seqno_to_time_mapping": "N/A"}} 2024-04-26T16:09:25.695 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:25 smithi190 bash[23868]: debug 2024-04-26T16:09:25.673+0000 7f3886c32700 4 rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 5447 microseconds, and 4760 cpu microseconds. 2024-04-26T16:09:26.406 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-26T16:09:26.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-26T16:09:28.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:28 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:28.380 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:09:28 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:28 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:28.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:09:28 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:29 smithi050 bash[29234]: debug 2024-04-26T16:09:29.561+0000 7f77f1732700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1714147769564421, "job": 4, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [14, 8], "score": -1, "input_data_size": 2046212, "oldest_snapshot_seqno": -1} 2024-04-26T16:09:30.099 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:09:31.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:25.691950+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:31.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:25.691950+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:31.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:25.692436+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:31.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:25.692436+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:31.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:25.694414+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:25.694414+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:26.615824+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:26.615824+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:26.685800+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:26.685800+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:27.685294+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:27.685294+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:27.686280+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:27.686280+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:28.616205+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:28.616205+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:28.686222+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:28.686222+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:29.567814+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:29.567814+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:29.686221+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:29.686221+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.568312+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.568312+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.686484+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.686484+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.714352+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.714352+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720538+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720538+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720576+0000 mon.a (mon.0) 157 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720576+0000 mon.a (mon.0) 157 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720595+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-26T16:09:25.686298+0000 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720595+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-26T16:09:25.686298+0000 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720616+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720616+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720636+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720636+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720658+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720658+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720678+0000 mon.a (mon.0) 162 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720678+0000 mon.a (mon.0) 162 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720697+0000 mon.a (mon.0) 163 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.720697+0000 mon.a (mon.0) 163 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.743995+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.743995+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-26T16:09:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.744051+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.744051+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.744371+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 76s) 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.744371+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 76s) 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.744609+0000 mon.a (mon.0) 167 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: cluster 2024-04-26T16:09:30.744609+0000 mon.a (mon.0) 167 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.752299+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.752299+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.759996+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:30 smithi190 bash[23868]: audit 2024-04-26T16:09:30.759996+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:25.691950+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:25.691950+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:25.692436+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:25.692436+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:25.694414+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:25.694414+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:26.615824+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:26.615824+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:26.685800+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:26.685800+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:27.685294+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:27.685294+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:27.686280+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:27.686280+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:28.616205+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:28.616205+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:28.686222+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:28.686222+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:29.567814+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:29.567814+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:29.686221+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:29.686221+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.568312+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.568312+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.686484+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.686484+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:31.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.714352+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.714352+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720538+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720538+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720576+0000 mon.a (mon.0) 157 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720576+0000 mon.a (mon.0) 157 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720595+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-26T16:09:25.686298+0000 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720595+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-26T16:09:25.686298+0000 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720616+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720616+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720636+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720636+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720658+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720658+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:31.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720678+0000 mon.a (mon.0) 162 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720678+0000 mon.a (mon.0) 162 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720697+0000 mon.a (mon.0) 163 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.720697+0000 mon.a (mon.0) 163 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.743995+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.743995+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.744051+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.744051+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.744371+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 76s) 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.744371+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 76s) 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.744609+0000 mon.a (mon.0) 167 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: cluster 2024-04-26T16:09:30.744609+0000 mon.a (mon.0) 167 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.752299+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.752299+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.759996+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:31.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:30 smithi050 bash[21463]: audit 2024-04-26T16:09:30.759996+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:32.130 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:09:31 smithi050 bash[21717]: debug 2024-04-26T16:09:31.685+0000 7f6692e65700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 -- 172.21.15.190:0/2998368082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 msgr2=0x7f21cc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 --2- 172.21.15.190:0/2998368082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21cc005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f21e0102e70 tx=0x7f21d0009e80 comp rx=0 tx=0).stop 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 -- 172.21.15.190:0/2998368082 shutdown_connections 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 --2- 172.21.15.190:0/2998368082 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21cc005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 -- 172.21.15.190:0/2998368082 >> 172.21.15.190:0/2998368082 conn(0x7f21e00fea30 msgr2=0x7f21e0100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 -- 172.21.15.190:0/2998368082 shutdown_connections 2024-04-26T16:09:36.879 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.873+0000 7f21e7251700 1 -- 172.21.15.190:0/2998368082 wait complete. 2024-04-26T16:09:36.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 librados: starting msgr at 2024-04-26T16:09:36.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 librados: starting objecter 2024-04-26T16:09:36.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 Processor -- start 2024-04-26T16:09:36.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 -- start start 2024-04-26T16:09:36.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 librados: setting wanted keys 2024-04-26T16:09:36.880 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 librados: calling monclient init 2024-04-26T16:09:36.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21e01980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:36.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:36.882 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f21e019ced0 0x7f21e019d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f21e010a330 con 0x7f21cc0035c0 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f21e010a1b0 con 0x7f21e0198600 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f21e010a4b0 con 0x7f21e019ced0 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21e01980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:47534/0 (socket says 172.21.15.190:47534) 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 -- 172.21.15.190:0/842023703 learned_addr learned my addr 172.21.15.190:0/842023703 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 -- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 msgr2=0x7f21e019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 -- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 msgr2=0x7f21e019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21dffff700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T16:09:36.883 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e57ee700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f21e019ced0 0x7f21e019d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 -- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f21e019ced0 msgr2=0x7f21e019d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f21e019ced0 0x7f21e019d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 -- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 msgr2=0x7f21e019c990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 -- 172.21.15.190:0/842023703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e01a4c40 con 0x7f21cc0035c0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e4fed700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21e01980c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f21d00098a0 tx=0x7f21d00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21ddffb700 10 client.?.objecter ms_handle_connect 0x7f21cc0035c0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21ddffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0004050 con 0x7f21cc0035c0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f21d00041f0 con 0x7f21cc0035c0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21e01a4e10 con 0x7f21cc0035c0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21d0005880 con 0x7f21cc0035c0 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 10 client.14190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:36.884 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21e01a52c0 con 0x7f21cc0035c0 2024-04-26T16:09:36.885 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.877+0000 7f21e7251700 1 librados: init done 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f21d0015040 con 0x7f21cc0035c0 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21e0064a80 con 0x7f21cc0035c0 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f21c003c1c0 0x7f21c003e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f21d004fca0 con 0x7f21cc0035c0 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 10 client.14190.objecter ms_dispatch 0x7f21e0001550 osd_map(4..4 src has 1..4) 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 3 client.14190.objecter handle_osd_map got epochs [4,4] > 0 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 3 client.14190.objecter handle_osd_map decoding full epoch 4 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 20 client.14190.objecter dump_active .. 0 homeless 2024-04-26T16:09:36.886 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21dffff700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f21c003c1c0 0x7f21c003e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:36.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21ddffb700 10 client.14190.objecter ms_handle_connect 0x7f21c003c1c0 2024-04-26T16:09:36.887 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.881+0000 7f21dffff700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f21c003c1c0 0x7f21c003e680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009180 tx=0x7f21c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:36.899 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:36.897+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f21d0018620 con 0x7f21cc0035c0 2024-04-26T16:09:37.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:31.574220+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:37.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:31.574220+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:37.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:31.576840+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:37.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:31.576840+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:37.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:31.577410+0000 mon.a (mon.0) 175 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:31.577410+0000 mon.a (mon.0) 175 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:31.596775+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:31.596775+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:31.597160+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:31.597160+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:32.568864+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:32.568864+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:32.617029+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:32.617029+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:33.568905+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:33.568905+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:34.569100+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:34.569100+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:34.617397+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:34.617397+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:35.569273+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:35.569273+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.569454+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.569454+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.596605+0000 mon.a (mon.0) 183 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.596605+0000 mon.a (mon.0) 183 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603180+0000 mon.a (mon.0) 184 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603180+0000 mon.a (mon.0) 184 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603238+0000 mon.a (mon.0) 185 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603238+0000 mon.a (mon.0) 185 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603284+0000 mon.a (mon.0) 186 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603284+0000 mon.a (mon.0) 186 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603353+0000 mon.a (mon.0) 187 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603353+0000 mon.a (mon.0) 187 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603401+0000 mon.a (mon.0) 188 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603401+0000 mon.a (mon.0) 188 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603444+0000 mon.a (mon.0) 189 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603444+0000 mon.a (mon.0) 189 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:37.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603485+0000 mon.a (mon.0) 190 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603485+0000 mon.a (mon.0) 190 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603529+0000 mon.a (mon.0) 191 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603529+0000 mon.a (mon.0) 191 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603570+0000 mon.a (mon.0) 192 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.603570+0000 mon.a (mon.0) 192 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624096+0000 mon.a (mon.0) 193 : cluster [DBG] fsmap 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624096+0000 mon.a (mon.0) 193 : cluster [DBG] fsmap 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624191+0000 mon.a (mon.0) 194 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624191+0000 mon.a (mon.0) 194 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624541+0000 mon.a (mon.0) 195 : cluster [DBG] mgrmap e13: y(active, since 82s) 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624541+0000 mon.a (mon.0) 195 : cluster [DBG] mgrmap e13: y(active, since 82s) 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624959+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.624959+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-26T16:09:37.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.633856+0000 mon.a (mon.0) 197 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.633856+0000 mon.a (mon.0) 197 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.633908+0000 mon.a (mon.0) 198 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.633908+0000 mon.a (mon.0) 198 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.633934+0000 mon.a (mon.0) 199 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.633934+0000 mon.a (mon.0) 199 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.639172+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.639172+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.645470+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.645470+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.651745+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.651745+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.658007+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.658007+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.664257+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.084 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:36 smithi190 bash[23868]: audit 2024-04-26T16:09:36.664257+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:31.574220+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:31.574220+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:31.576840+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:31.576840+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:31.577410+0000 mon.a (mon.0) 175 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:31.577410+0000 mon.a (mon.0) 175 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:31.596775+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:31.596775+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:31.597160+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:31.597160+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:32.568864+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:32.568864+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:32.617029+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:32.617029+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:33.568905+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:33.568905+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:34.569100+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:34.569100+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:34.617397+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:34.617397+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:35.569273+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:35.569273+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.569454+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.569454+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.596605+0000 mon.a (mon.0) 183 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.596605+0000 mon.a (mon.0) 183 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603180+0000 mon.a (mon.0) 184 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603180+0000 mon.a (mon.0) 184 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603238+0000 mon.a (mon.0) 185 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603238+0000 mon.a (mon.0) 185 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603284+0000 mon.a (mon.0) 186 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603284+0000 mon.a (mon.0) 186 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603353+0000 mon.a (mon.0) 187 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603353+0000 mon.a (mon.0) 187 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603401+0000 mon.a (mon.0) 188 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603401+0000 mon.a (mon.0) 188 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603444+0000 mon.a (mon.0) 189 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603444+0000 mon.a (mon.0) 189 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603485+0000 mon.a (mon.0) 190 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603485+0000 mon.a (mon.0) 190 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603529+0000 mon.a (mon.0) 191 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603529+0000 mon.a (mon.0) 191 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603570+0000 mon.a (mon.0) 192 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.603570+0000 mon.a (mon.0) 192 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:37.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624096+0000 mon.a (mon.0) 193 : cluster [DBG] fsmap 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624096+0000 mon.a (mon.0) 193 : cluster [DBG] fsmap 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624191+0000 mon.a (mon.0) 194 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624191+0000 mon.a (mon.0) 194 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624541+0000 mon.a (mon.0) 195 : cluster [DBG] mgrmap e13: y(active, since 82s) 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624541+0000 mon.a (mon.0) 195 : cluster [DBG] mgrmap e13: y(active, since 82s) 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624959+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.624959+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.633856+0000 mon.a (mon.0) 197 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.633856+0000 mon.a (mon.0) 197 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.633908+0000 mon.a (mon.0) 198 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.633908+0000 mon.a (mon.0) 198 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.633934+0000 mon.a (mon.0) 199 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-26T16:09:37.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.633934+0000 mon.a (mon.0) 199 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.639172+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.639172+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.645470+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.645470+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.651745+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.651745+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.658007+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.658007+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.664257+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:36 smithi050 bash[21463]: audit 2024-04-26T16:09:36.664257+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.220 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f21e01a5520 con 0x7f21cc0035c0 2024-04-26T16:09:37.220 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21ddffb700 1 -- 172.21.15.190:0/842023703 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f21d00183c0 con 0x7f21cc0035c0 2024-04-26T16:09:37.221 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-04-26T16:09:37.221 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":3,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","modified":"2024-04-26T16:09:31.569761Z","created":"2024-04-26T16:07:00.226385Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3300","nonce":0},{"type":"v1","addr":"172.21.15.50:6789","nonce":0}]},"addr":"172.21.15.50:6789/0","public_addr":"172.21.15.50:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:3301","nonce":0},{"type":"v1","addr":"172.21.15.50:6790","nonce":0}]},"addr":"172.21.15.50:6790/0","public_addr":"172.21.15.50:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-26T16:09:37.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 10 librados: watch_flush enter 2024-04-26T16:09:37.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 10 librados: watch_flush exit 2024-04-26T16:09:37.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 20 client.14190.objecter shutdown clearing up homeless session... 2024-04-26T16:09:37.223 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 10 client.14190.objecter successfully canceled tick 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f21c003c1c0 msgr2=0x7f21c003e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.217+0000 7f21e7251700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f21c003c1c0 0x7f21c003e680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009180 tx=0x7f21c8008040 comp rx=0 tx=0).stop 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 msgr2=0x7f21e01980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21e01980c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f21d00098a0 tx=0x7f21d00047c0 comp rx=0 tx=0).stop 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 shutdown_connections 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f21e019ced0 0x7f21e019d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f21cc0035c0 0x7f21e01980c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f21c003c1c0 0x7f21c003e680 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 --2- 172.21.15.190:0/842023703 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21e0198600 0x7f21e019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 >> 172.21.15.190:0/842023703 conn(0x7f21e00fea30 msgr2=0x7f21e0106b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 shutdown_connections 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 -- 172.21.15.190:0/842023703 wait complete. 2024-04-26T16:09:37.224 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:37.221+0000 7f21e7251700 1 librados: shutdown 2024-04-26T16:09:37.225 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 3 2024-04-26T16:09:37.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.617807+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: cluster 2024-04-26T16:09:36.617807+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.225376+0000 mon.a (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/842023703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.225376+0000 mon.a (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/842023703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.337955+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.337955+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.339942+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.339942+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.568861+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.568861+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.677736+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.677736+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.682855+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:37 smithi190 bash[23868]: audit 2024-04-26T16:09:37.682855+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:37.992 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-26T16:09:37.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-26T16:09:38.003 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.617807+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: cluster 2024-04-26T16:09:36.617807+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.225376+0000 mon.a (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/842023703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.225376+0000 mon.a (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/842023703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.337955+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.337955+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.339942+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.339942+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.568861+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.568861+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.677736+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.677736+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.682855+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.004 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:37 smithi050 bash[21463]: audit 2024-04-26T16:09:37.682855+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:25.691950+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:25.691950+0000 mon.a (mon.0) 145 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:25.692436+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:25.692436+0000 mon.a (mon.0) 146 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:25.694414+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:25.694414+0000 mon.a (mon.0) 147 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:26.615824+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:26.615824+0000 mgr.y (mgr.14152) 52 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:26.685800+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:26.685800+0000 mon.a (mon.0) 148 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:27.685294+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:27.685294+0000 mon.b (mon.1) 1 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:27.686280+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:27.686280+0000 mon.a (mon.0) 149 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:28.616205+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:28.616205+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:28.686222+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:28.686222+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:29.567814+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:29.567814+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:33.571618+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:33.571618+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.615208+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.615208+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.617440+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.617440+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.618800+0000 mon.a (mon.0) 222 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.618800+0000 mon.a (mon.0) 222 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.642171+0000 mon.a (mon.0) 223 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.642171+0000 mon.a (mon.0) 223 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-26T16:09:38.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650366+0000 mon.a (mon.0) 224 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650366+0000 mon.a (mon.0) 224 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650404+0000 mon.a (mon.0) 225 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650404+0000 mon.a (mon.0) 225 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650427+0000 mon.a (mon.0) 226 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650427+0000 mon.a (mon.0) 226 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650447+0000 mon.a (mon.0) 227 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650447+0000 mon.a (mon.0) 227 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650475+0000 mon.a (mon.0) 228 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650475+0000 mon.a (mon.0) 228 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650498+0000 mon.a (mon.0) 229 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650498+0000 mon.a (mon.0) 229 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650523+0000 mon.a (mon.0) 230 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650523+0000 mon.a (mon.0) 230 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650547+0000 mon.a (mon.0) 231 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650547+0000 mon.a (mon.0) 231 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650572+0000 mon.a (mon.0) 232 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.650572+0000 mon.a (mon.0) 232 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.651608+0000 mon.a (mon.0) 233 : cluster [DBG] fsmap 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.651608+0000 mon.a (mon.0) 233 : cluster [DBG] fsmap 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.651666+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.651666+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.651958+0000 mon.a (mon.0) 235 : cluster [DBG] mgrmap e13: y(active, since 84s) 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.651958+0000 mon.a (mon.0) 235 : cluster [DBG] mgrmap e13: y(active, since 84s) 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.652235+0000 mon.a (mon.0) 236 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.652235+0000 mon.a (mon.0) 236 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.652282+0000 mon.a (mon.0) 237 : cluster [INF] Cluster is now healthy 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.652282+0000 mon.a (mon.0) 237 : cluster [INF] Cluster is now healthy 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.658716+0000 mon.a (mon.0) 238 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:38.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.658716+0000 mon.a (mon.0) 238 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:38.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:29.686221+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:29.686221+0000 mon.a (mon.0) 152 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.568312+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.568312+0000 mon.a (mon.0) 153 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.686484+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.686484+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.714352+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.714352+0000 mon.a (mon.0) 155 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720538+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720538+0000 mon.a (mon.0) 156 : cluster [DBG] monmap epoch 2 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720576+0000 mon.a (mon.0) 157 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720576+0000 mon.a (mon.0) 157 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720595+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-26T16:09:25.686298+0000 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720595+0000 mon.a (mon.0) 158 : cluster [DBG] last_changed 2024-04-26T16:09:25.686298+0000 2024-04-26T16:09:38.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720616+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720616+0000 mon.a (mon.0) 159 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720636+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720636+0000 mon.a (mon.0) 160 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720658+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720658+0000 mon.a (mon.0) 161 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720678+0000 mon.a (mon.0) 162 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720678+0000 mon.a (mon.0) 162 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720697+0000 mon.a (mon.0) 163 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.720697+0000 mon.a (mon.0) 163 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.743995+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.743995+0000 mon.a (mon.0) 164 : cluster [DBG] fsmap 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.744051+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.744051+0000 mon.a (mon.0) 165 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.744371+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 76s) 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.744371+0000 mon.a (mon.0) 166 : cluster [DBG] mgrmap e13: y(active, since 76s) 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.744609+0000 mon.a (mon.0) 167 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:30.744609+0000 mon.a (mon.0) 167 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.752299+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.752299+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.759996+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:30.759996+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:31.574220+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:31.574220+0000 mon.b (mon.1) 2 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:31.576840+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:31.576840+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:09:38.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:31.577410+0000 mon.a (mon.0) 175 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:31.577410+0000 mon.a (mon.0) 175 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:31.596775+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:31.596775+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:31.597160+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:31.597160+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:32.568864+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:32.568864+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:32.617029+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:32.617029+0000 mgr.y (mgr.14152) 55 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:33.568905+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:33.568905+0000 mon.a (mon.0) 179 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:34.569100+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.890 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:34.569100+0000 mon.a (mon.0) 180 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:34.617397+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:34.617397+0000 mgr.y (mgr.14152) 56 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:35.569273+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:35.569273+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.569454+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.569454+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.596605+0000 mon.a (mon.0) 183 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.596605+0000 mon.a (mon.0) 183 : cluster [INF] mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603180+0000 mon.a (mon.0) 184 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603180+0000 mon.a (mon.0) 184 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603238+0000 mon.a (mon.0) 185 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603238+0000 mon.a (mon.0) 185 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.891 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603284+0000 mon.a (mon.0) 186 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603284+0000 mon.a (mon.0) 186 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603353+0000 mon.a (mon.0) 187 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603353+0000 mon.a (mon.0) 187 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603401+0000 mon.a (mon.0) 188 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603401+0000 mon.a (mon.0) 188 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603444+0000 mon.a (mon.0) 189 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603444+0000 mon.a (mon.0) 189 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603485+0000 mon.a (mon.0) 190 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603485+0000 mon.a (mon.0) 190 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603529+0000 mon.a (mon.0) 191 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603529+0000 mon.a (mon.0) 191 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603570+0000 mon.a (mon.0) 192 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.603570+0000 mon.a (mon.0) 192 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624096+0000 mon.a (mon.0) 193 : cluster [DBG] fsmap 2024-04-26T16:09:38.892 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624096+0000 mon.a (mon.0) 193 : cluster [DBG] fsmap 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624191+0000 mon.a (mon.0) 194 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624191+0000 mon.a (mon.0) 194 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624541+0000 mon.a (mon.0) 195 : cluster [DBG] mgrmap e13: y(active, since 82s) 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624541+0000 mon.a (mon.0) 195 : cluster [DBG] mgrmap e13: y(active, since 82s) 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624959+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.624959+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.633856+0000 mon.a (mon.0) 197 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.633856+0000 mon.a (mon.0) 197 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.633908+0000 mon.a (mon.0) 198 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.633908+0000 mon.a (mon.0) 198 : cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.633934+0000 mon.a (mon.0) 199 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.633934+0000 mon.a (mon.0) 199 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] is down (out of quorum) 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.639172+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.893 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.639172+0000 mon.a (mon.0) 200 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.645470+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.645470+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.651745+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.651745+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.658007+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.658007+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.664257+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:36.664257+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.617807+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:36.617807+0000 mgr.y (mgr.14152) 57 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.225376+0000 mon.a (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/842023703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.225376+0000 mon.a (mon.0) 205 : audit [DBG] from='client.? 172.21.15.190:0/842023703' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.337955+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:38.894 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.337955+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.339942+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.339942+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.568861+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.568861+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.677736+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.677736+0000 mon.a (mon.0) 209 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.682855+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: audit 2024-04-26T16:09:37.682855+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:33.571618+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:33.571618+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.615208+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.615208+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.617440+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.617440+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.618800+0000 mon.a (mon.0) 222 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.618800+0000 mon.a (mon.0) 222 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.642171+0000 mon.a (mon.0) 223 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.642171+0000 mon.a (mon.0) 223 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650366+0000 mon.a (mon.0) 224 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650366+0000 mon.a (mon.0) 224 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650404+0000 mon.a (mon.0) 225 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650404+0000 mon.a (mon.0) 225 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650427+0000 mon.a (mon.0) 226 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650427+0000 mon.a (mon.0) 226 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650447+0000 mon.a (mon.0) 227 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650447+0000 mon.a (mon.0) 227 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:38.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650475+0000 mon.a (mon.0) 228 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650475+0000 mon.a (mon.0) 228 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650498+0000 mon.a (mon.0) 229 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650498+0000 mon.a (mon.0) 229 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650523+0000 mon.a (mon.0) 230 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650523+0000 mon.a (mon.0) 230 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650547+0000 mon.a (mon.0) 231 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650547+0000 mon.a (mon.0) 231 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650572+0000 mon.a (mon.0) 232 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.650572+0000 mon.a (mon.0) 232 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.651608+0000 mon.a (mon.0) 233 : cluster [DBG] fsmap 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.651608+0000 mon.a (mon.0) 233 : cluster [DBG] fsmap 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.651666+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.651666+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.651958+0000 mon.a (mon.0) 235 : cluster [DBG] mgrmap e13: y(active, since 84s) 2024-04-26T16:09:38.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.651958+0000 mon.a (mon.0) 235 : cluster [DBG] mgrmap e13: y(active, since 84s) 2024-04-26T16:09:38.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.652235+0000 mon.a (mon.0) 236 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-26T16:09:38.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.652235+0000 mon.a (mon.0) 236 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-26T16:09:38.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.652282+0000 mon.a (mon.0) 237 : cluster [INF] Cluster is now healthy 2024-04-26T16:09:38.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.652282+0000 mon.a (mon.0) 237 : cluster [INF] Cluster is now healthy 2024-04-26T16:09:38.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.658716+0000 mon.a (mon.0) 238 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:38.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:38 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.658716+0000 mon.a (mon.0) 238 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:39.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:33.571618+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:33.571618+0000 mon.c (mon.2) 1 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.615208+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.615208+0000 mon.b (mon.1) 3 : cluster [INF] mon.b calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.617440+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.617440+0000 mon.c (mon.2) 2 : cluster [INF] mon.c calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.618800+0000 mon.a (mon.0) 222 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.618800+0000 mon.a (mon.0) 222 : cluster [INF] mon.a calling monitor election 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.642171+0000 mon.a (mon.0) 223 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.642171+0000 mon.a (mon.0) 223 : cluster [INF] mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650366+0000 mon.a (mon.0) 224 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650366+0000 mon.a (mon.0) 224 : cluster [DBG] monmap epoch 3 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650404+0000 mon.a (mon.0) 225 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650404+0000 mon.a (mon.0) 225 : cluster [DBG] fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:39.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650427+0000 mon.a (mon.0) 226 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650427+0000 mon.a (mon.0) 226 : cluster [DBG] last_changed 2024-04-26T16:09:31.569761+0000 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650447+0000 mon.a (mon.0) 227 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650447+0000 mon.a (mon.0) 227 : cluster [DBG] created 2024-04-26T16:07:00.226385+0000 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650475+0000 mon.a (mon.0) 228 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650475+0000 mon.a (mon.0) 228 : cluster [DBG] min_mon_release 19 (squid) 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650498+0000 mon.a (mon.0) 229 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650498+0000 mon.a (mon.0) 229 : cluster [DBG] election_strategy: 1 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650523+0000 mon.a (mon.0) 230 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650523+0000 mon.a (mon.0) 230 : cluster [DBG] 0: [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] mon.a 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650547+0000 mon.a (mon.0) 231 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650547+0000 mon.a (mon.0) 231 : cluster [DBG] 1: [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon.b 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650572+0000 mon.a (mon.0) 232 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:39.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.650572+0000 mon.a (mon.0) 232 : cluster [DBG] 2: [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] mon.c 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.651608+0000 mon.a (mon.0) 233 : cluster [DBG] fsmap 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.651608+0000 mon.a (mon.0) 233 : cluster [DBG] fsmap 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.651666+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.651666+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.651958+0000 mon.a (mon.0) 235 : cluster [DBG] mgrmap e13: y(active, since 84s) 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.651958+0000 mon.a (mon.0) 235 : cluster [DBG] mgrmap e13: y(active, since 84s) 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.652235+0000 mon.a (mon.0) 236 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.652235+0000 mon.a (mon.0) 236 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.652282+0000 mon.a (mon.0) 237 : cluster [INF] Cluster is now healthy 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.652282+0000 mon.a (mon.0) 237 : cluster [INF] Cluster is now healthy 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.658716+0000 mon.a (mon.0) 238 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:39.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:38 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.658716+0000 mon.a (mon.0) 238 : cluster [INF] overall HEALTH_OK 2024-04-26T16:09:40.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.618244+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[21463]: cluster 2024-04-26T16:09:38.618244+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[21463]: audit 2024-04-26T16:09:39.569797+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[21463]: audit 2024-04-26T16:09:39.569797+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.618244+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[29234]: cluster 2024-04-26T16:09:38.618244+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[29234]: audit 2024-04-26T16:09:39.569797+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:39 smithi050 bash[29234]: audit 2024-04-26T16:09:39.569797+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:40.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:39 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.618244+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:40.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:39 smithi190 bash[23868]: cluster 2024-04-26T16:09:38.618244+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:40.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:39 smithi190 bash[23868]: audit 2024-04-26T16:09:39.569797+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:40.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:39 smithi190 bash[23868]: audit 2024-04-26T16:09:39.569797+0000 mon.a (mon.0) 239 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.938201+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.938201+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.945231+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.945231+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: cephadm 2024-04-26T16:09:39.962354+0000 mgr.y (mgr.14152) 65 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: cephadm 2024-04-26T16:09:39.962354+0000 mgr.y (mgr.14152) 65 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.963008+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.963008+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:41.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.964791+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.964791+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.966567+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: audit 2024-04-26T16:09:39.966567+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: cephadm 2024-04-26T16:09:39.968690+0000 mgr.y (mgr.14152) 66 : cephadm [INF] Reconfiguring daemon mon.c on smithi050 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: cephadm 2024-04-26T16:09:39.968690+0000 mgr.y (mgr.14152) 66 : cephadm [INF] Reconfiguring daemon mon.c on smithi050 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: cluster 2024-04-26T16:09:40.618508+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[21463]: cluster 2024-04-26T16:09:40.618508+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.938201+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.938201+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.945231+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.945231+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: cephadm 2024-04-26T16:09:39.962354+0000 mgr.y (mgr.14152) 65 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: cephadm 2024-04-26T16:09:39.962354+0000 mgr.y (mgr.14152) 65 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.963008+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.963008+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.964791+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.964791+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.966567+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: audit 2024-04-26T16:09:39.966567+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: cephadm 2024-04-26T16:09:39.968690+0000 mgr.y (mgr.14152) 66 : cephadm [INF] Reconfiguring daemon mon.c on smithi050 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: cephadm 2024-04-26T16:09:39.968690+0000 mgr.y (mgr.14152) 66 : cephadm [INF] Reconfiguring daemon mon.c on smithi050 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: cluster 2024-04-26T16:09:40.618508+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:41.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:40 smithi050 bash[29234]: cluster 2024-04-26T16:09:40.618508+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.938201+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.938201+0000 mon.a (mon.0) 240 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.945231+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.945231+0000 mon.a (mon.0) 241 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: cephadm 2024-04-26T16:09:39.962354+0000 mgr.y (mgr.14152) 65 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: cephadm 2024-04-26T16:09:39.962354+0000 mgr.y (mgr.14152) 65 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.963008+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.963008+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.964791+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:41.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.964791+0000 mon.a (mon.0) 243 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:41.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.966567+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:41.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: audit 2024-04-26T16:09:39.966567+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:41.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: cephadm 2024-04-26T16:09:39.968690+0000 mgr.y (mgr.14152) 66 : cephadm [INF] Reconfiguring daemon mon.c on smithi050 2024-04-26T16:09:41.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: cephadm 2024-04-26T16:09:39.968690+0000 mgr.y (mgr.14152) 66 : cephadm [INF] Reconfiguring daemon mon.c on smithi050 2024-04-26T16:09:41.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: cluster 2024-04-26T16:09:40.618508+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:41.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:40 smithi190 bash[23868]: cluster 2024-04-26T16:09:40.618508+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:42.709 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.907950+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.907950+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.914887+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.914887+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.939204+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.939204+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.940837+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.940837+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.942479+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[21463]: audit 2024-04-26T16:09:41.942479+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.907950+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.907950+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.914887+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.914887+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.939204+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.939204+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.940837+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.940837+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.942479+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:42 smithi050 bash[29234]: audit 2024-04-26T16:09:41.942479+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:43.183 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.907950+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.907950+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.914887+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.914887+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.939204+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.939204+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.940837+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.940837+0000 mon.a (mon.0) 248 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.942479+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:43.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:42 smithi190 bash[23868]: audit 2024-04-26T16:09:41.942479+0000 mon.a (mon.0) 249 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: cephadm 2024-04-26T16:09:41.938516+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-26T16:09:44.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: cephadm 2024-04-26T16:09:41.938516+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-26T16:09:44.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: cephadm 2024-04-26T16:09:41.944453+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.b on smithi190 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: cephadm 2024-04-26T16:09:41.944453+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.b on smithi190 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: cluster 2024-04-26T16:09:42.618885+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: cluster 2024-04-26T16:09:42.618885+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.819308+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.819308+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.825675+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.825675+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.865951+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.865951+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:44.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.869628+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.869628+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.871326+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.871326+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.880938+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[21463]: audit 2024-04-26T16:09:43.880938+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: cephadm 2024-04-26T16:09:41.938516+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: cephadm 2024-04-26T16:09:41.938516+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: cephadm 2024-04-26T16:09:41.944453+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.b on smithi190 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: cephadm 2024-04-26T16:09:41.944453+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.b on smithi190 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: cluster 2024-04-26T16:09:42.618885+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: cluster 2024-04-26T16:09:42.618885+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.819308+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.819308+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.825675+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.825675+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.865951+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.865951+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.869628+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.869628+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.871326+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.871326+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.880938+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:43 smithi050 bash[29234]: audit 2024-04-26T16:09:43.880938+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: cephadm 2024-04-26T16:09:41.938516+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: cephadm 2024-04-26T16:09:41.938516+0000 mgr.y (mgr.14152) 68 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: cephadm 2024-04-26T16:09:41.944453+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.b on smithi190 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: cephadm 2024-04-26T16:09:41.944453+0000 mgr.y (mgr.14152) 69 : cephadm [INF] Reconfiguring daemon mon.b on smithi190 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: cluster 2024-04-26T16:09:42.618885+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: cluster 2024-04-26T16:09:42.618885+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.819308+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.819308+0000 mon.a (mon.0) 250 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.825675+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.825675+0000 mon.a (mon.0) 251 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.865951+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.865951+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.869628+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.869628+0000 mon.a (mon.0) 253 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.871326+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.871326+0000 mon.a (mon.0) 254 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.880938+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:44.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:43 smithi190 bash[23868]: audit 2024-04-26T16:09:43.880938+0000 mon.a (mon.0) 255 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- 172.21.15.50:0/2432566290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 msgr2=0x7fd92810ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/2432566290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92810ba70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd918009990 tx=0x7fd918009ca0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- 172.21.15.50:0/2432566290 shutdown_connections 2024-04-26T16:09:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/2432566290 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd9281124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/2432566290 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92810ba70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/2432566290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd928105600 0x7fd928105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- 172.21.15.50:0/2432566290 >> 172.21.15.50:0/2432566290 conn(0x7fd9281010d0 msgr2=0x7fd9281034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- 172.21.15.50:0/2432566290 shutdown_connections 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- 172.21.15.50:0/2432566290 wait complete. 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 librados: starting msgr at 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 librados: starting objecter 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 Processor -- start 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- start start 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 librados: setting wanted keys 2024-04-26T16:09:45.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 librados: calling monclient init 2024-04-26T16:09:45.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd928105600 0x7fd92819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd92819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd928114d60 con 0x7fd92810b690 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd928114be0 con 0x7fd928105600 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd92dbd9700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd928114ee0 con 0x7fd928105fc0 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd926ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd92819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd926ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd92819b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49404/0 (socket says 172.21.15.50:49404) 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd926ffd700 1 -- 172.21.15.50:0/1680609333 learned_addr learned my addr 172.21.15.50:0/1680609333 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd9277fe700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd928105600 0x7fd92819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 -- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 msgr2=0x7fd92819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd92819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 -- 172.21.15.50:0/1680609333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd928105600 msgr2=0x7fd92819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd928105600 0x7fd92819b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 -- 172.21.15.50:0/1680609333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9281a0370 con 0x7fd92810b690 2024-04-26T16:09:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd926ffd700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd92819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:09:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd927fff700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92819fc30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd920007f10 tx=0x7fd92000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd924ff9700 10 client.?.objecter ms_handle_connect 0x7fd92810b690 2024-04-26T16:09:45.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd924ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.253+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd92000ddd0 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd920004bb0 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9281a0660 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9200043d0 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd92dbd9700 10 client.14196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9281a7ee0 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd92dbd9700 1 librados: init done 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7fd92001c020 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92806a110 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd91003bf30 0x7fd91003e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd92004d2c0 con 0x7fd92810b690 2024-04-26T16:09:45.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 10 client.14196.objecter ms_dispatch 0x7fd928001550 osd_map(4..4 src has 1..4) 2024-04-26T16:09:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 3 client.14196.objecter handle_osd_map got epochs [4,4] > 0 2024-04-26T16:09:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 3 client.14196.objecter handle_osd_map decoding full epoch 4 2024-04-26T16:09:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 20 client.14196.objecter dump_active .. 0 homeless 2024-04-26T16:09:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd9277fe700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd91003bf30 0x7fd91003e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd9277fe700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd91003bf30 0x7fd91003e3f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd918006010 tx=0x7fd91801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:45.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.257+0000 7fd924ff9700 10 client.14196.objecter ms_handle_connect 0x7fd91003bf30 2024-04-26T16:09:45.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.261+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd920010070 con 0x7fd92810b690 2024-04-26T16:09:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.485+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd9281a80b0 con 0x7fd92810b690 2024-04-26T16:09:45.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.485+0000 7fd924ff9700 1 -- 172.21.15.50:0/1680609333 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7fd920014e50 con 0x7fd92810b690 2024-04-26T16:09:45.483 INFO:teuthology.orchestra.run.smithi050.stdout:# minimal ceph.conf for cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:45.483 INFO:teuthology.orchestra.run.smithi050.stdout:[global] 2024-04-26T16:09:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: fsid = cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:09:45.484 INFO:teuthology.orchestra.run.smithi050.stdout: mon_host = [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] 2024-04-26T16:09:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.485+0000 7fd92dbd9700 10 librados: watch_flush enter 2024-04-26T16:09:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 10 librados: watch_flush exit 2024-04-26T16:09:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 20 client.14196.objecter shutdown clearing up homeless session... 2024-04-26T16:09:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 10 client.14196.objecter successfully canceled tick 2024-04-26T16:09:45.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd91003bf30 msgr2=0x7fd91003e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd91003bf30 0x7fd91003e3f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd918006010 tx=0x7fd91801a040 comp rx=0 tx=0).stop 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 msgr2=0x7fd92819fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92819fc30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd920007f10 tx=0x7fd92000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 shutdown_connections 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd91003bf30 0x7fd91003e3f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd928105600 0x7fd92819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd92810b690 0x7fd92819fc30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 --2- 172.21.15.50:0/1680609333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd928105fc0 0x7fd92819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 >> 172.21.15.50:0/1680609333 conn(0x7fd9281010d0 msgr2=0x7fd92810c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 shutdown_connections 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 -- 172.21.15.50:0/1680609333 wait complete. 2024-04-26T16:09:45.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:09:45.489+0000 7fd92dbd9700 1 librados: shutdown 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[21463]: cluster 2024-04-26T16:09:44.619340+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[21463]: cluster 2024-04-26T16:09:44.619340+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[21463]: audit 2024-04-26T16:09:45.488461+0000 mon.a (mon.0) 256 : audit [DBG] from='client.? 172.21.15.50:0/1680609333' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[21463]: audit 2024-04-26T16:09:45.488461+0000 mon.a (mon.0) 256 : audit [DBG] from='client.? 172.21.15.50:0/1680609333' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[29234]: cluster 2024-04-26T16:09:44.619340+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[29234]: cluster 2024-04-26T16:09:44.619340+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[29234]: audit 2024-04-26T16:09:45.488461+0000 mon.a (mon.0) 256 : audit [DBG] from='client.? 172.21.15.50:0/1680609333' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:46.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:45 smithi050 bash[29234]: audit 2024-04-26T16:09:45.488461+0000 mon.a (mon.0) 256 : audit [DBG] from='client.? 172.21.15.50:0/1680609333' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:46.186 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-26T16:09:46.187 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:09:46.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T16:09:46.243 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:09:46.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:46.303 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:09:46.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-04-26T16:09:46.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:45 smithi190 bash[23868]: cluster 2024-04-26T16:09:44.619340+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:46.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:45 smithi190 bash[23868]: cluster 2024-04-26T16:09:44.619340+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:46.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:45 smithi190 bash[23868]: audit 2024-04-26T16:09:45.488461+0000 mon.a (mon.0) 256 : audit [DBG] from='client.? 172.21.15.50:0/1680609333' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:46.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:45 smithi190 bash[23868]: audit 2024-04-26T16:09:45.488461+0000 mon.a (mon.0) 256 : audit [DBG] from='client.? 172.21.15.50:0/1680609333' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:46.320 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:09:46.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:09:46.376 INFO:tasks.cephadm:Adding mgr.y on smithi050 2024-04-26T16:09:46.376 INFO:tasks.cephadm:Adding mgr.x on smithi190 2024-04-26T16:09:46.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi050=y;smithi190=x' 2024-04-26T16:09:48.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:47 smithi190 bash[23868]: cluster 2024-04-26T16:09:46.619793+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:48.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:47 smithi190 bash[23868]: cluster 2024-04-26T16:09:46.619793+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:48.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:47 smithi050 bash[21463]: cluster 2024-04-26T16:09:46.619793+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:48.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:47 smithi050 bash[21463]: cluster 2024-04-26T16:09:46.619793+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:48.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:47 smithi050 bash[29234]: cluster 2024-04-26T16:09:46.619793+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:48.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:47 smithi050 bash[29234]: cluster 2024-04-26T16:09:46.619793+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:49.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:48 smithi190 bash[23868]: cluster 2024-04-26T16:09:48.620225+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:49.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:48 smithi190 bash[23868]: cluster 2024-04-26T16:09:48.620225+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:48 smithi050 bash[29234]: cluster 2024-04-26T16:09:48.620225+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:48 smithi050 bash[29234]: cluster 2024-04-26T16:09:48.620225+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:49.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:48 smithi050 bash[21463]: cluster 2024-04-26T16:09:48.620225+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:49.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:48 smithi050 bash[21463]: cluster 2024-04-26T16:09:48.620225+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:50.116 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:09:52.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:51 smithi190 bash[23868]: cluster 2024-04-26T16:09:50.620658+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:52.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:51 smithi190 bash[23868]: cluster 2024-04-26T16:09:50.620658+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:52.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:51 smithi050 bash[21463]: cluster 2024-04-26T16:09:50.620658+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:52.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:51 smithi050 bash[21463]: cluster 2024-04-26T16:09:50.620658+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:51 smithi050 bash[29234]: cluster 2024-04-26T16:09:50.620658+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:51 smithi050 bash[29234]: cluster 2024-04-26T16:09:50.620658+0000 mgr.y (mgr.14152) 74 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1415555403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884075e80 msgr2=0x7f1884076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1415555403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884075e80 0x7f1884076300 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1880009990 tx=0x7f1880009ca0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1415555403 shutdown_connections 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1415555403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 0x7f188410d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1415555403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884077450 0x7f18840758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1415555403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884075e80 0x7f1884076300 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1415555403 >> 172.21.15.190:0/1415555403 conn(0x7f18840fea20 msgr2=0x7f1884100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1415555403 shutdown_connections 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1415555403 wait complete. 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 librados: starting msgr at 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 librados: starting objecter 2024-04-26T16:09:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 Processor -- start 2024-04-26T16:09:52.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- start start 2024-04-26T16:09:52.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 librados: setting wanted keys 2024-04-26T16:09:52.561 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 librados: calling monclient init 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884075e80 0x7f188419af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 0x7f188419b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 0x7f18841a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f18841106a0 con 0x7f188410b350 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1884110520 con 0x7f1884077450 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1884110820 con 0x7f1884075e80 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 0x7f188419b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 0x7f188419b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:45204/0 (socket says 172.21.15.190:45204) 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 -- 172.21.15.190:0/1841519758 learned_addr learned my addr 172.21.15.190:0/1841519758 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:09:52.563 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188a758700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 0x7f18841a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889f57700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884075e80 0x7f188419af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 -- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884075e80 msgr2=0x7f188419af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884075e80 0x7f188419af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 -- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 msgr2=0x7f18841a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 0x7f18841a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 -- 172.21.15.190:0/1841519758 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18841a2ba0 con 0x7f1884077450 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188a758700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 0x7f18841a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889f57700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884075e80 0x7f188419af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1889756700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 0x7f188419b480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1880005f50 tx=0x7f18800047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 10 client.?.objecter ms_handle_connect 0x7f1884077450 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188000f9d0 con 0x7f1884077450 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1880004830 con 0x7f1884077450 2024-04-26T16:09:52.564 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18841a2e30 con 0x7f1884077450 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1880005430 con 0x7f1884077450 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 10 client.24106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18841a31d0 con 0x7f1884077450 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f188c1bb700 1 librados: init done 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 13) ==== 47233+0+0 (secure 0 0 0) 0x7f1880022470 con 0x7f1884077450 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f187003bf30 0x7f187003e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1880054660 con 0x7f1884077450 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 10 client.24106.objecter ms_dispatch 0x7f1884001550 osd_map(4..4 src has 1..4) 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 3 client.24106.objecter handle_osd_map got epochs [4,4] > 0 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 3 client.24106.objecter handle_osd_map decoding full epoch 4 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.561+0000 7f1876ffd700 20 client.24106.objecter dump_active .. 0 homeless 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.565+0000 7f1889f57700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f187003bf30 0x7f187003e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:09:52.565 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.565+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f188406a110 con 0x7f1884077450 2024-04-26T16:09:52.566 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.565+0000 7f1889f57700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f187003bf30 0x7f187003e3f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1884077830 tx=0x7f1878008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:09:52.566 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.565+0000 7f1876ffd700 10 client.24106.objecter ms_handle_connect 0x7f187003bf30 2024-04-26T16:09:52.568 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.569+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1880013070 con 0x7f1884077450 2024-04-26T16:09:52.803 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.805+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}) -- 0x7f1884063920 con 0x7f187003bf30 2024-04-26T16:09:52.817 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.817+0000 7f1876ffd700 1 -- 172.21.15.190:0/1841519758 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1884063920 con 0x7f187003bf30 2024-04-26T16:09:52.817 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mgr update... 2024-04-26T16:09:52.820 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 10 librados: watch_flush enter 2024-04-26T16:09:52.820 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 10 librados: watch_flush exit 2024-04-26T16:09:52.820 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 20 client.24106.objecter shutdown clearing up homeless session... 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 10 client.24106.objecter successfully canceled tick 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f187003bf30 msgr2=0x7f187003e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f187003bf30 0x7f187003e3f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1884077830 tx=0x7f1878008040 comp rx=0 tx=0).stop 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 msgr2=0x7f188419b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 0x7f188419b480 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1880005f50 tx=0x7f18800047b0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 shutdown_connections 2024-04-26T16:09:52.821 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f188410b350 0x7f18841a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1884075e80 0x7f188419af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f187003bf30 0x7f187003e3f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 --2- 172.21.15.190:0/1841519758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1884077450 0x7f188419b480 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 >> 172.21.15.190:0/1841519758 conn(0x7f18840fea20 msgr2=0x7f1884107250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 shutdown_connections 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 -- 172.21.15.190:0/1841519758 wait complete. 2024-04-26T16:09:52.822 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:09:52.821+0000 7f188c1bb700 1 librados: shutdown 2024-04-26T16:09:53.632 DEBUG:teuthology.orchestra.run.smithi190:mgr.x> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.x.service 2024-04-26T16:09:53.634 INFO:tasks.cephadm:Deploying OSDs... 2024-04-26T16:09:53.634 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:09:53.635 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T16:09:53.641 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T16:09:53.641 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_1 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 16:00:01.137428437 +0000 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 16:00:00.869430827 +0000 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 16:00:00.869430827 +0000 2024-04-26T16:09:53.691 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-26T16:09:53.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T16:09:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T16:09:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T16:09:53.746 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.0002169 s, 2.4 MB/s 2024-04-26T16:09:53.747 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T16:09:53.796 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_2 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 901 Links: 1 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 16:00:01.605424262 +0000 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 16:00:01.349426545 +0000 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 16:00:01.349426545 +0000 2024-04-26T16:09:53.843 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-26T16:09:53.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T16:09:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T16:09:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T16:09:53.897 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.00030447 s, 1.7 MB/s 2024-04-26T16:09:53.898 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T16:09:53.950 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_3 2024-04-26T16:09:53.998 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T16:09:53.998 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:53.998 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-04-26T16:09:53.999 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:53.999 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 16:00:02.085419980 +0000 2024-04-26T16:09:53.999 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 16:00:01.817422371 +0000 2024-04-26T16:09:53.999 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 16:00:01.817422371 +0000 2024-04-26T16:09:53.999 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-26T16:09:53.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T16:09:54.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: cluster 2024-04-26T16:09:52.621077+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:54.051 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: cluster 2024-04-26T16:09:52.621077+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.810771+0000 mgr.y (mgr.14152) 76 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.810771+0000 mgr.y (mgr.14152) 76 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: cephadm 2024-04-26T16:09:52.814409+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Saving service mgr spec with placement smithi050=y;smithi190=x;count:2 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: cephadm 2024-04-26T16:09:52.814409+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Saving service mgr spec with placement smithi050=y;smithi190=x;count:2 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.821123+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.821123+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.840728+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.840728+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.844599+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.844599+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.846567+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.846567+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:54.052 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.856088+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.856088+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.859456+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.859456+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: cluster 2024-04-26T16:09:52.621077+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: cluster 2024-04-26T16:09:52.621077+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.810771+0000 mgr.y (mgr.14152) 76 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.810771+0000 mgr.y (mgr.14152) 76 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: cephadm 2024-04-26T16:09:52.814409+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Saving service mgr spec with placement smithi050=y;smithi190=x;count:2 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: cephadm 2024-04-26T16:09:52.814409+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Saving service mgr spec with placement smithi050=y;smithi190=x;count:2 2024-04-26T16:09:54.053 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.821123+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.821123+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.840728+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.840728+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.844599+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.844599+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.846567+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.846567+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.856088+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.856088+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.859456+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:09:54.054 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.859456+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.862010+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.862010+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.867193+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.867193+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.869365+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: audit 2024-04-26T16:09:52.869365+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: cephadm 2024-04-26T16:09:52.871494+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Deploying daemon mgr.x on smithi190 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[21463]: cephadm 2024-04-26T16:09:52.871494+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Deploying daemon mgr.x on smithi190 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.862010+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T16:09:54.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.862010+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T16:09:54.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.867193+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:09:54.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.867193+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:09:54.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.869365+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: audit 2024-04-26T16:09:52.869365+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: cephadm 2024-04-26T16:09:52.871494+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Deploying daemon mgr.x on smithi190 2024-04-26T16:09:54.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:53 smithi050 bash[29234]: cephadm 2024-04-26T16:09:52.871494+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Deploying daemon mgr.x on smithi190 2024-04-26T16:09:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T16:09:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T16:09:54.057 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000191314 s, 2.7 MB/s 2024-04-26T16:09:54.057 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T16:09:54.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: cluster 2024-04-26T16:09:52.621077+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:54.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: cluster 2024-04-26T16:09:52.621077+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:54.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.810771+0000 mgr.y (mgr.14152) 76 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:54.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.810771+0000 mgr.y (mgr.14152) 76 : audit [DBG] from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi050=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: cephadm 2024-04-26T16:09:52.814409+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Saving service mgr spec with placement smithi050=y;smithi190=x;count:2 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: cephadm 2024-04-26T16:09:52.814409+0000 mgr.y (mgr.14152) 77 : cephadm [INF] Saving service mgr spec with placement smithi050=y;smithi190=x;count:2 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.821123+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.821123+0000 mon.a (mon.0) 257 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.840728+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.840728+0000 mon.a (mon.0) 258 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.844599+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.844599+0000 mon.a (mon.0) 259 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.846567+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.846567+0000 mon.a (mon.0) 260 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.856088+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.856088+0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.859456+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:09:54.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.859456+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.862010+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.862010+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.867193+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.867193+0000 mon.a (mon.0) 264 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.869365+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: audit 2024-04-26T16:09:52.869365+0000 mon.a (mon.0) 265 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: cephadm 2024-04-26T16:09:52.871494+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Deploying daemon mgr.x on smithi190 2024-04-26T16:09:54.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:53 smithi190 bash[23868]: cephadm 2024-04-26T16:09:52.871494+0000 mgr.y (mgr.14152) 78 : cephadm [INF] Deploying daemon mgr.x on smithi190 2024-04-26T16:09:54.107 DEBUG:teuthology.orchestra.run.smithi050:> stat /dev/vg_nvme/lv_4 2024-04-26T16:09:54.154 INFO:teuthology.orchestra.run.smithi050.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout:Access: 2024-04-26 16:00:02.321417875 +0000 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout:Modify: 2024-04-26 16:00:02.317417910 +0000 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout:Change: 2024-04-26 16:00:02.317417910 +0000 2024-04-26T16:09:54.155 INFO:teuthology.orchestra.run.smithi050.stdout: Birth: - 2024-04-26T16:09:54.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T16:09:54.209 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records in 2024-04-26T16:09:54.209 INFO:teuthology.orchestra.run.smithi050.stderr:1+0 records out 2024-04-26T16:09:54.209 INFO:teuthology.orchestra.run.smithi050.stderr:512 bytes copied, 0.000231064 s, 2.2 MB/s 2024-04-26T16:09:54.210 DEBUG:teuthology.orchestra.run.smithi050:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T16:09:54.264 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:09:54.264 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T16:09:54.269 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T16:09:54.269 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-26 15:59:56.488870424 +0000 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-26 15:59:56.236872675 +0000 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-26 15:59:56.236872675 +0000 2024-04-26T16:09:54.315 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-04-26T16:09:54.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T16:09:54.367 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-26T16:09:54.367 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-26T16:09:54.367 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000265999 s, 1.9 MB/s 2024-04-26T16:09:54.368 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T16:09:54.416 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-04-26T16:09:54.462 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T16:09:54.462 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:54.462 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-04-26T16:09:54.462 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:54.462 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-26 15:59:56.972866101 +0000 2024-04-26T16:09:54.463 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-26 15:59:56.716868387 +0000 2024-04-26T16:09:54.463 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-26 15:59:56.716868387 +0000 2024-04-26T16:09:54.463 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-04-26T16:09:54.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T16:09:54.518 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-26T16:09:54.518 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-26T16:09:54.518 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000417092 s, 1.2 MB/s 2024-04-26T16:09:54.519 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T16:09:54.568 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-04-26T16:09:54.616 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T16:09:54.617 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:54.617 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-04-26T16:09:54.617 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:54.618 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-26 15:59:57.504861349 +0000 2024-04-26T16:09:54.618 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-26 15:59:57.236863743 +0000 2024-04-26T16:09:54.618 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-26 15:59:57.236863743 +0000 2024-04-26T16:09:54.618 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-04-26T16:09:54.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T16:09:54.670 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-26T16:09:54.670 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-26T16:09:54.670 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000433961 s, 1.2 MB/s 2024-04-26T16:09:54.672 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T16:09:54.720 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-04-26T16:09:54.766 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-04-26 15:59:57.984857061 +0000 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-04-26 15:59:57.728859348 +0000 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-04-26 15:59:57.728859348 +0000 2024-04-26T16:09:54.767 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2024-04-26T16:09:54.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T16:09:54.820 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-04-26T16:09:54.820 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-04-26T16:09:54.820 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000315361 s, 1.6 MB/s 2024-04-26T16:09:54.821 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T16:09:54.868 INFO:tasks.cephadm:Deploying osd.0 on smithi050 with /dev/vg_nvme/lv_4... 2024-04-26T16:09:54.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T16:09:55.206 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:55.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:55.481 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:55.481 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:55.482 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:55.482 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:09:55.482 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:55 smithi190 systemd[1]: Started Ceph mgr.x for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:09:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: cluster 2024-04-26T16:09:54.621519+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: cluster 2024-04-26T16:09:54.621519+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.489330+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.489330+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.497382+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.497382+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.548561+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.548561+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.555660+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.555660+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.625521+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:55 smithi190 bash[23868]: audit 2024-04-26T16:09:55.625521+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: cluster 2024-04-26T16:09:54.621519+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: cluster 2024-04-26T16:09:54.621519+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.489330+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.489330+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.497382+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.497382+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.548561+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.548561+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.555660+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.555660+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.625521+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[21463]: audit 2024-04-26T16:09:55.625521+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: cluster 2024-04-26T16:09:54.621519+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: cluster 2024-04-26T16:09:54.621519+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.489330+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.489330+0000 mon.a (mon.0) 266 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.497382+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.497382+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.548561+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.548561+0000 mon.a (mon.0) 268 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.555660+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.555660+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.625521+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:55 smithi050 bash[29234]: audit 2024-04-26T16:09:55.625521+0000 mon.a (mon.0) 270 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:09:56.483 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:56 smithi190 bash[24769]: debug 2024-04-26T16:09:56.337+0000 7f6586211700 1 -- 172.21.15.190:0/4260477657 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607c52025a0 con 0x5607c5f82c00 2024-04-26T16:09:56.483 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:56 smithi190 bash[24769]: debug 2024-04-26T16:09:56.485+0000 7f6590423200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T16:09:58.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:57 smithi190 bash[23868]: cluster 2024-04-26T16:09:56.621958+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:58.042 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:57 smithi190 bash[23868]: cluster 2024-04-26T16:09:56.621958+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:58.042 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:57 smithi190 bash[23868]: audit 2024-04-26T16:09:56.647365+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:58.042 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:57 smithi190 bash[23868]: audit 2024-04-26T16:09:56.647365+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:58.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[21463]: cluster 2024-04-26T16:09:56.621958+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:58.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[21463]: cluster 2024-04-26T16:09:56.621958+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[21463]: audit 2024-04-26T16:09:56.647365+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[21463]: audit 2024-04-26T16:09:56.647365+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[29234]: cluster 2024-04-26T16:09:56.621958+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[29234]: cluster 2024-04-26T16:09:56.621958+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[29234]: audit 2024-04-26T16:09:56.647365+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:57 smithi050 bash[29234]: audit 2024-04-26T16:09:56.647365+0000 mon.a (mon.0) 271 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:09:58.327 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:58 smithi190 bash[24769]: debug 2024-04-26T16:09:58.041+0000 7f6590423200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T16:09:58.327 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:58 smithi190 bash[24769]: debug 2024-04-26T16:09:58.137+0000 7f6590423200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T16:09:58.827 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:58 smithi190 bash[24769]: debug 2024-04-26T16:09:58.385+0000 7f6590423200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T16:09:59.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[24769]: debug 2024-04-26T16:09:59.180+0000 7f6590423200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T16:09:59.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[24769]: debug 2024-04-26T16:09:59.288+0000 7f6590423200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T16:09:59.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[24769]: debug 2024-04-26T16:09:59.388+0000 7f6590423200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T16:09:59.594 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:09:59.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:59 smithi050 bash[21463]: cluster 2024-04-26T16:09:58.622373+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:59.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:09:59 smithi050 bash[21463]: cluster 2024-04-26T16:09:58.622373+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:59 smithi050 bash[29234]: cluster 2024-04-26T16:09:58.622373+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:09:59 smithi050 bash[29234]: cluster 2024-04-26T16:09:58.622373+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:59.958 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[24769]: debug 2024-04-26T16:09:59.744+0000 7f6590423200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T16:09:59.958 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[24769]: debug 2024-04-26T16:09:59.856+0000 7f6590423200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T16:09:59.958 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[23868]: cluster 2024-04-26T16:09:58.622373+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:09:59.958 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[23868]: cluster 2024-04-26T16:09:58.622373+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:00.311 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:09:59 smithi190 bash[24769]: debug 2024-04-26T16:09:59.956+0000 7f6590423200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T16:10:00.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:00 smithi190 bash[24769]: debug 2024-04-26T16:10:00.308+0000 7f6590423200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T16:10:00.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:00 smithi190 bash[24769]: debug 2024-04-26T16:10:00.552+0000 7f6590423200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T16:10:00.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:00 smithi190 bash[23868]: cluster 2024-04-26T16:10:00.000125+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-04-26T16:10:00.936 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:00 smithi190 bash[23868]: cluster 2024-04-26T16:10:00.000125+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-04-26T16:10:00.937 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:00 smithi190 bash[24769]: debug 2024-04-26T16:10:00.820+0000 7f6590423200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T16:10:01.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:00 smithi050 bash[21463]: cluster 2024-04-26T16:10:00.000125+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-04-26T16:10:01.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:00 smithi050 bash[21463]: cluster 2024-04-26T16:10:00.000125+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-04-26T16:10:01.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:00 smithi050 bash[29234]: cluster 2024-04-26T16:10:00.000125+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-04-26T16:10:01.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:00 smithi050 bash[29234]: cluster 2024-04-26T16:10:00.000125+0000 mon.a (mon.0) 272 : cluster [INF] overall HEALTH_OK 2024-04-26T16:10:01.217 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:00 smithi190 bash[24769]: debug 2024-04-26T16:10:00.936+0000 7f6590423200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T16:10:01.578 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[24769]: debug 2024-04-26T16:10:01.216+0000 7f6590423200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T16:10:01.989 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[24769]: debug 2024-04-26T16:10:01.636+0000 7f6590423200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:10:01.989 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[24769]: debug 2024-04-26T16:10:01.872+0000 7f6590423200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T16:10:01.989 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[24769]: debug 2024-04-26T16:10:01.988+0000 7f6590423200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T16:10:01.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: cluster 2024-04-26T16:10:00.622778+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:01.990 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: cluster 2024-04-26T16:10:00.622778+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:00.863749+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:00.863749+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:00.871726+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:00.871726+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.497624+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.497624+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.499507+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.499507+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.508705+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.508705+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.578064+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:10:01.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.578064+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:10:01.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.579736+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:10:01.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.579736+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:10:01.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.581171+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:01.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:01 smithi190 bash[23868]: audit 2024-04-26T16:10:01.581171+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: cluster 2024-04-26T16:10:00.622778+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: cluster 2024-04-26T16:10:00.622778+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:00.863749+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:00.863749+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:00.871726+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:00.871726+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.497624+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.497624+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.499507+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.499507+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.508705+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.508705+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.578064+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.578064+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.579736+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.579736+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.581171+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[29234]: audit 2024-04-26T16:10:01.581171+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: cluster 2024-04-26T16:10:00.622778+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: cluster 2024-04-26T16:10:00.622778+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:00.863749+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:00.863749+0000 mon.a (mon.0) 273 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:00.871726+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:00.871726+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.497624+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.497624+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.499507+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.499507+0000 mon.a (mon.0) 276 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.508705+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.508705+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.578064+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.578064+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.579736+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.579736+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.581171+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:01 smithi050 bash[21463]: audit 2024-04-26T16:10:01.581171+0000 mon.a (mon.0) 280 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:02.327 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:02 smithi190 bash[24769]: debug 2024-04-26T16:10:02.232+0000 7f6590423200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:10:02.521 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T16:10:02.521 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T16:10:02.521 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T16:10:02.521 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T16:10:02.521 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013088 s, 801 MB/s 2024-04-26T16:10:02.521 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T16:10:03.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[21463]: cephadm 2024-04-26T16:10:01.577273+0000 mgr.y (mgr.14152) 83 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-04-26T16:10:03.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[21463]: cephadm 2024-04-26T16:10:01.577273+0000 mgr.y (mgr.14152) 83 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-04-26T16:10:03.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[21463]: cephadm 2024-04-26T16:10:01.582622+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring daemon mgr.y on smithi050 2024-04-26T16:10:03.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[21463]: cephadm 2024-04-26T16:10:01.582622+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring daemon mgr.y on smithi050 2024-04-26T16:10:03.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[29234]: cephadm 2024-04-26T16:10:01.577273+0000 mgr.y (mgr.14152) 83 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-04-26T16:10:03.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[29234]: cephadm 2024-04-26T16:10:01.577273+0000 mgr.y (mgr.14152) 83 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-04-26T16:10:03.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[29234]: cephadm 2024-04-26T16:10:01.582622+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring daemon mgr.y on smithi050 2024-04-26T16:10:03.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:02 smithi050 bash[29234]: cephadm 2024-04-26T16:10:01.582622+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring daemon mgr.y on smithi050 2024-04-26T16:10:03.237 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:02 smithi190 bash[23868]: cephadm 2024-04-26T16:10:01.577273+0000 mgr.y (mgr.14152) 83 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-04-26T16:10:03.237 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:02 smithi190 bash[23868]: cephadm 2024-04-26T16:10:01.577273+0000 mgr.y (mgr.14152) 83 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2024-04-26T16:10:03.237 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:02 smithi190 bash[23868]: cephadm 2024-04-26T16:10:01.582622+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring daemon mgr.y on smithi050 2024-04-26T16:10:03.238 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:02 smithi190 bash[23868]: cephadm 2024-04-26T16:10:01.582622+0000 mgr.y (mgr.14152) 84 : cephadm [INF] Reconfiguring daemon mgr.y on smithi050 2024-04-26T16:10:03.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_4 2024-04-26T16:10:03.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:03 smithi190 bash[24769]: debug 2024-04-26T16:10:03.236+0000 7f6590423200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T16:10:03.577 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:03 smithi190 bash[24769]: debug 2024-04-26T16:10:03.348+0000 7f6590423200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T16:10:04.033 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:03 smithi190 bash[24769]: debug 2024-04-26T16:10:03.640+0000 7f6590423200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T16:10:04.034 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:03 smithi190 bash[24769]: debug 2024-04-26T16:10:03.752+0000 7f6590423200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T16:10:04.034 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:03 smithi190 bash[23868]: cluster 2024-04-26T16:10:02.623047+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:04.034 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:03 smithi190 bash[23868]: cluster 2024-04-26T16:10:02.623047+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:03 smithi050 bash[21463]: cluster 2024-04-26T16:10:02.623047+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:03 smithi050 bash[21463]: cluster 2024-04-26T16:10:02.623047+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:04.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:03 smithi050 bash[29234]: cluster 2024-04-26T16:10:02.623047+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:04.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:03 smithi050 bash[29234]: cluster 2024-04-26T16:10:02.623047+0000 mgr.y (mgr.14152) 85 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:04.327 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[24769]: debug 2024-04-26T16:10:04.032+0000 7f6590423200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-26T16:10:04.827 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[24769]: debug 2024-04-26T16:10:04.480+0000 7f6590423200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T16:10:04.827 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[24769]: debug 2024-04-26T16:10:04.600+0000 7f6590423200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T16:10:05.329 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:03.956856+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:03.956856+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:03.965511+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:03.965511+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.004058+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.004058+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.008025+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.008025+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.010093+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.010093+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:05.330 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.021163+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.021163+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: cluster 2024-04-26T16:10:04.623414+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: cluster 2024-04-26T16:10:04.623414+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.626156+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.626156+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.627033+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.627033+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.628273+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.628273+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: cluster 2024-04-26T16:10:04.628456+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: cluster 2024-04-26T16:10:04.628456+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.628752+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:10:05.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:04 smithi190 bash[23868]: audit 2024-04-26T16:10:04.628752+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:03.956856+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:03.956856+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:03.965511+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:03.965511+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.004058+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.004058+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.008025+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.008025+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.010093+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.010093+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.021163+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.021163+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: cluster 2024-04-26T16:10:04.623414+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: cluster 2024-04-26T16:10:04.623414+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.626156+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.626156+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.627033+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.627033+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:10:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.628273+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:03.956856+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:03.956856+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:03.965511+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:03.965511+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.004058+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.004058+0000 mon.a (mon.0) 283 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.008025+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.008025+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.010093+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.010093+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.021163+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.021163+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: cluster 2024-04-26T16:10:04.623414+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: cluster 2024-04-26T16:10:04.623414+0000 mgr.y (mgr.14152) 86 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.626156+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.626156+0000 mon.b (mon.1) 4 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.627033+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.627033+0000 mon.b (mon.1) 5 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.628273+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.628273+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: cluster 2024-04-26T16:10:04.628456+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: cluster 2024-04-26T16:10:04.628456+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.628752+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:10:05.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[21463]: audit 2024-04-26T16:10:04.628752+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:10:05.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.628273+0000 mon.b (mon.1) 6 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:10:05.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: cluster 2024-04-26T16:10:04.628456+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:10:05.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: cluster 2024-04-26T16:10:04.628456+0000 mon.a (mon.0) 287 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:10:05.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.628752+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:10:05.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:04 smithi050 bash[29234]: audit 2024-04-26T16:10:04.628752+0000 mon.b (mon.1) 7 : audit [DBG] from='mgr.? 172.21.15.190:0/3077318319' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:10:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:06 smithi190 bash[23868]: cluster 2024-04-26T16:10:05.037909+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e14: y(active, since 110s), standbys: x 2024-04-26T16:10:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:06 smithi190 bash[23868]: cluster 2024-04-26T16:10:05.037909+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e14: y(active, since 110s), standbys: x 2024-04-26T16:10:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:06 smithi190 bash[23868]: audit 2024-04-26T16:10:05.038223+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:10:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:06 smithi190 bash[23868]: audit 2024-04-26T16:10:05.038223+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:10:06.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[21463]: cluster 2024-04-26T16:10:05.037909+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e14: y(active, since 110s), standbys: x 2024-04-26T16:10:06.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[21463]: cluster 2024-04-26T16:10:05.037909+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e14: y(active, since 110s), standbys: x 2024-04-26T16:10:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[21463]: audit 2024-04-26T16:10:05.038223+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:10:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[21463]: audit 2024-04-26T16:10:05.038223+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:10:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[29234]: cluster 2024-04-26T16:10:05.037909+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e14: y(active, since 110s), standbys: x 2024-04-26T16:10:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[29234]: cluster 2024-04-26T16:10:05.037909+0000 mon.a (mon.0) 288 : cluster [DBG] mgrmap e14: y(active, since 110s), standbys: x 2024-04-26T16:10:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[29234]: audit 2024-04-26T16:10:05.038223+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:10:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:06 smithi050 bash[29234]: audit 2024-04-26T16:10:05.038223+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:10:07.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:07 smithi190 bash[23868]: cluster 2024-04-26T16:10:06.623929+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:07.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:07 smithi190 bash[23868]: cluster 2024-04-26T16:10:06.623929+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:07.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:07 smithi050 bash[21463]: cluster 2024-04-26T16:10:06.623929+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:07.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:07 smithi050 bash[21463]: cluster 2024-04-26T16:10:06.623929+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:07.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:07 smithi050 bash[29234]: cluster 2024-04-26T16:10:06.623929+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:07.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:07 smithi050 bash[29234]: cluster 2024-04-26T16:10:06.623929+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:08.152 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:10:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:09 smithi190 bash[23868]: cluster 2024-04-26T16:10:08.624402+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:09 smithi190 bash[23868]: cluster 2024-04-26T16:10:08.624402+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:10.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:09 smithi050 bash[21463]: cluster 2024-04-26T16:10:08.624402+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:10.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:09 smithi050 bash[21463]: cluster 2024-04-26T16:10:08.624402+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:10.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:09 smithi050 bash[29234]: cluster 2024-04-26T16:10:08.624402+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:10.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:09 smithi050 bash[29234]: cluster 2024-04-26T16:10:08.624402+0000 mgr.y (mgr.14152) 88 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:10.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 -- 172.21.15.50:0/972850331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0104670 msgr2=0x7f3ed0106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:10:10.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- 172.21.15.50:0/972850331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0104670 0x7f3ed0106b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4009990 tx=0x7f3ec4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:10:10.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 -- 172.21.15.50:0/972850331 shutdown_connections 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- 172.21.15.50:0/972850331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ed0107040 0x7f3ed01074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- 172.21.15.50:0/972850331 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed0104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- 172.21.15.50:0/972850331 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0104670 0x7f3ed0106b00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 -- 172.21.15.50:0/972850331 >> 172.21.15.50:0/972850331 conn(0x7f3ed00fd940 msgr2=0x7f3ed00ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 -- 172.21.15.50:0/972850331 shutdown_connections 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 -- 172.21.15.50:0/972850331 wait complete. 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 librados: starting msgr at 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 librados: starting objecter 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 Processor -- start 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 -- start start 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 librados: setting wanted keys 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 librados: calling monclient init 2024-04-26T16:10:10.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ed0104670 0x7f3ed019f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3eceffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.597+0000 7f3ed5748700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0107040 0x7f3ed01a6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed019f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55386/0 (socket says 172.21.15.50:55386) 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed0114940 con 0x7f3ed0101d00 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 -- 172.21.15.50:0/53199856 learned_addr learned my addr 172.21.15.50:0/53199856 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed01147c0 con 0x7f3ed0104670 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed0114ac0 con 0x7f3ed0107040 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ece7fc700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ed0104670 0x7f3ed019f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:10:10.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ecf7fe700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0107040 0x7f3ed01a6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:10:10.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 -- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0107040 msgr2=0x7f3ed01a6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0107040 0x7f3ed01a6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 -- 172.21.15.50:0/53199856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ed0104670 msgr2=0x7f3ed019f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ed0104670 0x7f3ed019f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed01a6dc0 con 0x7f3ed0101d00 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eceffd700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed019f160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000d980 tx=0x7f3ec000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 10 client.?.objecter ms_handle_connect 0x7f3ed0101d00 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0016070 con 0x7f3ed0101d00 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3ec0009d30 con 0x7f3ed0101d00 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed01a70b0 con 0x7f3ed0101d00 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ec0010df0 con 0x7f3ed0101d00 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 10 client.14205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:10:10.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed01a7640 con 0x7f3ed0101d00 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 librados: init done 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ebc005150 con 0x7f3ed0101d00 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 14) ==== 93847+0+0 (secure 0 0 0) 0x7f3ec0019070 con 0x7f3ed0101d00 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f3eb8071df0 0x7f3eb80742b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f3ec008f120 con 0x7f3ed0101d00 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 10 client.14205.objecter ms_dispatch 0x7f3ed0001550 osd_map(4..4 src has 1..4) 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 3 client.14205.objecter handle_osd_map got epochs [4,4] > 0 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 3 client.14205.objecter handle_osd_map decoding full epoch 4 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3ece7fc700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f3eb8071df0 0x7f3eb80742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.601+0000 7f3eb7fff700 20 client.14205.objecter dump_active .. 0 homeless 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.605+0000 7f3eb7fff700 10 client.14205.objecter ms_handle_connect 0x7f3eb8071df0 2024-04-26T16:10:10.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.605+0000 7f3ece7fc700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f3eb8071df0 0x7f3eb80742b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400c010 tx=0x7f3ec400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:10:10.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.609+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3ec005cca0 con 0x7f3ed0101d00 2024-04-26T16:10:10.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:10.841+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3ebc000bf0 con 0x7f3eb8071df0 2024-04-26T16:10:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: cluster 2024-04-26T16:10:10.624640+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: cluster 2024-04-26T16:10:10.624640+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.848225+0000 mgr.y (mgr.14152) 90 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.848225+0000 mgr.y (mgr.14152) 90 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.852571+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.852571+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.858715+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.858715+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.860120+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:12.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:11 smithi190 bash[23868]: audit 2024-04-26T16:10:10.860120+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: cluster 2024-04-26T16:10:10.624640+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: cluster 2024-04-26T16:10:10.624640+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.848225+0000 mgr.y (mgr.14152) 90 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.848225+0000 mgr.y (mgr.14152) 90 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.852571+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.852571+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.858715+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.858715+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.860120+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[29234]: audit 2024-04-26T16:10:10.860120+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: cluster 2024-04-26T16:10:10.624640+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: cluster 2024-04-26T16:10:10.624640+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.848225+0000 mgr.y (mgr.14152) 90 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.848225+0000 mgr.y (mgr.14152) 90 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.852571+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.852571+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.858715+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.858715+0000 mon.a (mon.0) 291 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.860120+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:11 smithi050 bash[21463]: audit 2024-04-26T16:10:10.860120+0000 mon.a (mon.0) 292 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:14.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:13 smithi190 bash[23868]: cluster 2024-04-26T16:10:12.625006+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:14.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:13 smithi190 bash[23868]: cluster 2024-04-26T16:10:12.625006+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:14.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:13 smithi050 bash[21463]: cluster 2024-04-26T16:10:12.625006+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:14.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:13 smithi050 bash[21463]: cluster 2024-04-26T16:10:12.625006+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:14.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:13 smithi050 bash[29234]: cluster 2024-04-26T16:10:12.625006+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:13 smithi050 bash[29234]: cluster 2024-04-26T16:10:12.625006+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:15.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:15.605+0000 7f3ed4f47700 10 client.14205.objecter tick 2024-04-26T16:10:16.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:15 smithi190 bash[23868]: cluster 2024-04-26T16:10:14.625449+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:16.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:15 smithi190 bash[23868]: cluster 2024-04-26T16:10:14.625449+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:16.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:15 smithi050 bash[21463]: cluster 2024-04-26T16:10:14.625449+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:16.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:15 smithi050 bash[21463]: cluster 2024-04-26T16:10:14.625449+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:16.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:15 smithi050 bash[29234]: cluster 2024-04-26T16:10:14.625449+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:15 smithi050 bash[29234]: cluster 2024-04-26T16:10:14.625449+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:18.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:17 smithi190 bash[23868]: cluster 2024-04-26T16:10:16.625771+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:18.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:17 smithi190 bash[23868]: cluster 2024-04-26T16:10:16.625771+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:18.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:17 smithi050 bash[21463]: cluster 2024-04-26T16:10:16.625771+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:18.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:17 smithi050 bash[21463]: cluster 2024-04-26T16:10:16.625771+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:18.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:17 smithi050 bash[29234]: cluster 2024-04-26T16:10:16.625771+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:18.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:17 smithi050 bash[29234]: cluster 2024-04-26T16:10:16.625771+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: cluster 2024-04-26T16:10:18.626099+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: cluster 2024-04-26T16:10:18.626099+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.157715+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.50:0/1619171872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.157715+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.50:0/1619171872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.158368+0000 mon.a (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.158368+0000 mon.a (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.160668+0000 mon.a (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]': finished 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.160668+0000 mon.a (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]': finished 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: cluster 2024-04-26T16:10:19.167676+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-26T16:10:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: cluster 2024-04-26T16:10:19.167676+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-26T16:10:20.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.167978+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:20.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:19 smithi190 bash[23868]: audit 2024-04-26T16:10:19.167978+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:20.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: cluster 2024-04-26T16:10:18.626099+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: cluster 2024-04-26T16:10:18.626099+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.157715+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.50:0/1619171872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.157715+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.50:0/1619171872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.158368+0000 mon.a (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.158368+0000 mon.a (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.160668+0000 mon.a (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]': finished 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.160668+0000 mon.a (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]': finished 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: cluster 2024-04-26T16:10:19.167676+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: cluster 2024-04-26T16:10:19.167676+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.167978+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[21463]: audit 2024-04-26T16:10:19.167978+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: cluster 2024-04-26T16:10:18.626099+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: cluster 2024-04-26T16:10:18.626099+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.157715+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.50:0/1619171872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.157715+0000 mon.c (mon.2) 3 : audit [INF] from='client.? 172.21.15.50:0/1619171872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.158368+0000 mon.a (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.158368+0000 mon.a (mon.0) 293 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]: dispatch 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.160668+0000 mon.a (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]': finished 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.160668+0000 mon.a (mon.0) 294 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1"}]': finished 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: cluster 2024-04-26T16:10:19.167676+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: cluster 2024-04-26T16:10:19.167676+0000 mon.a (mon.0) 295 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.167978+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:20.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:19 smithi050 bash[29234]: audit 2024-04-26T16:10:19.167978+0000 mon.a (mon.0) 296 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:20.609+0000 7f3ed4f47700 10 client.14205.objecter tick 2024-04-26T16:10:21.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:20 smithi190 bash[23868]: audit 2024-04-26T16:10:19.857058+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.50:0/2800041242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:10:21.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:20 smithi190 bash[23868]: audit 2024-04-26T16:10:19.857058+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.50:0/2800041242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:10:21.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:20 smithi050 bash[21463]: audit 2024-04-26T16:10:19.857058+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.50:0/2800041242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:10:21.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:20 smithi050 bash[21463]: audit 2024-04-26T16:10:19.857058+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.50:0/2800041242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:10:21.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:20 smithi050 bash[29234]: audit 2024-04-26T16:10:19.857058+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.50:0/2800041242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:10:21.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:20 smithi050 bash[29234]: audit 2024-04-26T16:10:19.857058+0000 mon.a (mon.0) 297 : audit [DBG] from='client.? 172.21.15.50:0/2800041242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:10:22.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:21 smithi190 bash[23868]: cluster 2024-04-26T16:10:20.626512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:22.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:21 smithi190 bash[23868]: cluster 2024-04-26T16:10:20.626512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:22.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:21 smithi050 bash[21463]: cluster 2024-04-26T16:10:20.626512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:22.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:21 smithi050 bash[21463]: cluster 2024-04-26T16:10:20.626512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:22.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:21 smithi050 bash[29234]: cluster 2024-04-26T16:10:20.626512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:22.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:21 smithi050 bash[29234]: cluster 2024-04-26T16:10:20.626512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:23 smithi190 bash[23868]: cluster 2024-04-26T16:10:22.626890+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:23 smithi190 bash[23868]: cluster 2024-04-26T16:10:22.626890+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:24.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:23 smithi050 bash[21463]: cluster 2024-04-26T16:10:22.626890+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:24.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:23 smithi050 bash[21463]: cluster 2024-04-26T16:10:22.626890+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:24.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:23 smithi050 bash[29234]: cluster 2024-04-26T16:10:22.626890+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:24.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:23 smithi050 bash[29234]: cluster 2024-04-26T16:10:22.626890+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:25.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:25.612+0000 7f3ed4f47700 10 client.14205.objecter tick 2024-04-26T16:10:26.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:25 smithi190 bash[23868]: cluster 2024-04-26T16:10:24.627300+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:26.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:25 smithi190 bash[23868]: cluster 2024-04-26T16:10:24.627300+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:26.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:25 smithi050 bash[21463]: cluster 2024-04-26T16:10:24.627300+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:26.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:25 smithi050 bash[21463]: cluster 2024-04-26T16:10:24.627300+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:26.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:25 smithi050 bash[29234]: cluster 2024-04-26T16:10:24.627300+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:26.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:25 smithi050 bash[29234]: cluster 2024-04-26T16:10:24.627300+0000 mgr.y (mgr.14152) 97 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:27 smithi190 bash[23868]: cluster 2024-04-26T16:10:26.627715+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:27 smithi190 bash[23868]: cluster 2024-04-26T16:10:26.627715+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:28.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:27 smithi050 bash[21463]: cluster 2024-04-26T16:10:26.627715+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:28.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:27 smithi050 bash[21463]: cluster 2024-04-26T16:10:26.627715+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:28.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:27 smithi050 bash[29234]: cluster 2024-04-26T16:10:26.627715+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:28.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:27 smithi050 bash[29234]: cluster 2024-04-26T16:10:26.627715+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:29 smithi190 bash[23868]: cluster 2024-04-26T16:10:28.628198+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:29 smithi190 bash[23868]: cluster 2024-04-26T16:10:28.628198+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:29 smithi050 bash[21463]: cluster 2024-04-26T16:10:28.628198+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:29 smithi050 bash[21463]: cluster 2024-04-26T16:10:28.628198+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:29 smithi050 bash[29234]: cluster 2024-04-26T16:10:28.628198+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:29 smithi050 bash[29234]: cluster 2024-04-26T16:10:28.628198+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:30.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:30.616+0000 7f3ed4f47700 10 client.14205.objecter tick 2024-04-26T16:10:32.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:31 smithi190 bash[23868]: cluster 2024-04-26T16:10:30.628637+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:31 smithi190 bash[23868]: cluster 2024-04-26T16:10:30.628637+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:31 smithi190 bash[23868]: audit 2024-04-26T16:10:31.522025+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T16:10:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:31 smithi190 bash[23868]: audit 2024-04-26T16:10:31.522025+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T16:10:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:31 smithi190 bash[23868]: audit 2024-04-26T16:10:31.523902+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:31 smithi190 bash[23868]: audit 2024-04-26T16:10:31.523902+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:32.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[21463]: cluster 2024-04-26T16:10:30.628637+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:32.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[21463]: cluster 2024-04-26T16:10:30.628637+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:32.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[21463]: audit 2024-04-26T16:10:31.522025+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T16:10:32.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[21463]: audit 2024-04-26T16:10:31.522025+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T16:10:32.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[21463]: audit 2024-04-26T16:10:31.523902+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:32.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[21463]: audit 2024-04-26T16:10:31.523902+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:32.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[29234]: cluster 2024-04-26T16:10:30.628637+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:32.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[29234]: cluster 2024-04-26T16:10:30.628637+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:32.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[29234]: audit 2024-04-26T16:10:31.522025+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T16:10:32.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[29234]: audit 2024-04-26T16:10:31.522025+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-26T16:10:32.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[29234]: audit 2024-04-26T16:10:31.523902+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:32.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:31 smithi050 bash[29234]: audit 2024-04-26T16:10:31.523902+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:33.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:32 smithi050 bash[21463]: cephadm 2024-04-26T16:10:31.525697+0000 mgr.y (mgr.14152) 101 : cephadm [INF] Deploying daemon osd.0 on smithi050 2024-04-26T16:10:33.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:32 smithi050 bash[21463]: cephadm 2024-04-26T16:10:31.525697+0000 mgr.y (mgr.14152) 101 : cephadm [INF] Deploying daemon osd.0 on smithi050 2024-04-26T16:10:33.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:32 smithi050 bash[29234]: cephadm 2024-04-26T16:10:31.525697+0000 mgr.y (mgr.14152) 101 : cephadm [INF] Deploying daemon osd.0 on smithi050 2024-04-26T16:10:33.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:32 smithi050 bash[29234]: cephadm 2024-04-26T16:10:31.525697+0000 mgr.y (mgr.14152) 101 : cephadm [INF] Deploying daemon osd.0 on smithi050 2024-04-26T16:10:33.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:32 smithi190 bash[23868]: cephadm 2024-04-26T16:10:31.525697+0000 mgr.y (mgr.14152) 101 : cephadm [INF] Deploying daemon osd.0 on smithi050 2024-04-26T16:10:33.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:32 smithi190 bash[23868]: cephadm 2024-04-26T16:10:31.525697+0000 mgr.y (mgr.14152) 101 : cephadm [INF] Deploying daemon osd.0 on smithi050 2024-04-26T16:10:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:33 smithi050 bash[21463]: cluster 2024-04-26T16:10:32.629056+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:33 smithi050 bash[21463]: cluster 2024-04-26T16:10:32.629056+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:33 smithi050 bash[29234]: cluster 2024-04-26T16:10:32.629056+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:33 smithi050 bash[29234]: cluster 2024-04-26T16:10:32.629056+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:33 smithi190 bash[23868]: cluster 2024-04-26T16:10:32.629056+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:33 smithi190 bash[23868]: cluster 2024-04-26T16:10:32.629056+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:35.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:35.620+0000 7f3ed4f47700 10 client.14205.objecter tick 2024-04-26T16:10:35.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:35 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:10:35.636 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:10:35 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:10:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:35 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:10:36.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:35 smithi190 bash[23868]: cluster 2024-04-26T16:10:34.629424+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:36.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:35 smithi190 bash[23868]: cluster 2024-04-26T16:10:34.629424+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:36.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:35 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:10:36.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:35 smithi050 bash[21463]: cluster 2024-04-26T16:10:34.629424+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:36.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:35 smithi050 bash[21463]: cluster 2024-04-26T16:10:34.629424+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:36.129 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:10:35 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:10:36.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:35 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:10:36.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:35 smithi050 bash[29234]: cluster 2024-04-26T16:10:34.629424+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:35 smithi050 bash[29234]: cluster 2024-04-26T16:10:34.629424+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[21463]: audit 2024-04-26T16:10:35.928486+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[21463]: audit 2024-04-26T16:10:35.928486+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[21463]: audit 2024-04-26T16:10:35.935512+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[21463]: audit 2024-04-26T16:10:35.935512+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[21463]: audit 2024-04-26T16:10:35.940848+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[21463]: audit 2024-04-26T16:10:35.940848+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[29234]: audit 2024-04-26T16:10:35.928486+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[29234]: audit 2024-04-26T16:10:35.928486+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[29234]: audit 2024-04-26T16:10:35.935512+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[29234]: audit 2024-04-26T16:10:35.935512+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[29234]: audit 2024-04-26T16:10:35.940848+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:36 smithi050 bash[29234]: audit 2024-04-26T16:10:35.940848+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:37.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:36 smithi190 bash[23868]: audit 2024-04-26T16:10:35.928486+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:36 smithi190 bash[23868]: audit 2024-04-26T16:10:35.928486+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:36 smithi190 bash[23868]: audit 2024-04-26T16:10:35.935512+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:36 smithi190 bash[23868]: audit 2024-04-26T16:10:35.935512+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:37.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:36 smithi190 bash[23868]: audit 2024-04-26T16:10:35.940848+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:37.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:36 smithi190 bash[23868]: audit 2024-04-26T16:10:35.940848+0000 mon.a (mon.0) 302 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:38.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:37 smithi190 bash[23868]: cluster 2024-04-26T16:10:36.629851+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:38.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:37 smithi190 bash[23868]: cluster 2024-04-26T16:10:36.629851+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:37 smithi050 bash[21463]: cluster 2024-04-26T16:10:36.629851+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:37 smithi050 bash[21463]: cluster 2024-04-26T16:10:36.629851+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:38.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:37 smithi050 bash[29234]: cluster 2024-04-26T16:10:36.629851+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:38.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:37 smithi050 bash[29234]: cluster 2024-04-26T16:10:36.629851+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:39 smithi190 bash[23868]: cluster 2024-04-26T16:10:38.630298+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:39 smithi190 bash[23868]: cluster 2024-04-26T16:10:38.630298+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:39 smithi050 bash[21463]: cluster 2024-04-26T16:10:38.630298+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:39 smithi050 bash[21463]: cluster 2024-04-26T16:10:38.630298+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:39 smithi050 bash[29234]: cluster 2024-04-26T16:10:38.630298+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:39 smithi050 bash[29234]: cluster 2024-04-26T16:10:38.630298+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:40.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:40.624+0000 7f3ed4f47700 10 client.14205.objecter tick 2024-04-26T16:10:41.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:40 smithi190 bash[23868]: cluster 2024-04-26T16:10:40.630721+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:41.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:40 smithi190 bash[23868]: cluster 2024-04-26T16:10:40.630721+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:41.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:40 smithi050 bash[21463]: cluster 2024-04-26T16:10:40.630721+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:41.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:40 smithi050 bash[21463]: cluster 2024-04-26T16:10:40.630721+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:41.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:40 smithi050 bash[29234]: cluster 2024-04-26T16:10:40.630721+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:41.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:40 smithi050 bash[29234]: cluster 2024-04-26T16:10:40.630721+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.224373+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.224373+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.229704+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.229704+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.846521+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.846521+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.847431+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:42.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.847431+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.852266+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[21463]: audit 2024-04-26T16:10:41.852266+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.224373+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.224373+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.229704+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.229704+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.846521+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.846521+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.847431+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.847431+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:42.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.852266+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:42 smithi050 bash[29234]: audit 2024-04-26T16:10:41.852266+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.224373+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.224373+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.229704+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.229704+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.846521+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.846521+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.847431+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.847431+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.852266+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:42.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:42 smithi190 bash[23868]: audit 2024-04-26T16:10:41.852266+0000 mon.a (mon.0) 307 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:43.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:43 smithi190 bash[23868]: audit 2024-04-26T16:10:42.552839+0000 mon.a (mon.0) 308 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T16:10:43.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:43 smithi190 bash[23868]: audit 2024-04-26T16:10:42.552839+0000 mon.a (mon.0) 308 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T16:10:43.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:43 smithi190 bash[23868]: cluster 2024-04-26T16:10:42.631058+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:43.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:43 smithi190 bash[23868]: cluster 2024-04-26T16:10:42.631058+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[21463]: audit 2024-04-26T16:10:42.552839+0000 mon.a (mon.0) 308 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[21463]: audit 2024-04-26T16:10:42.552839+0000 mon.a (mon.0) 308 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[21463]: cluster 2024-04-26T16:10:42.631058+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[21463]: cluster 2024-04-26T16:10:42.631058+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[29234]: audit 2024-04-26T16:10:42.552839+0000 mon.a (mon.0) 308 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[29234]: audit 2024-04-26T16:10:42.552839+0000 mon.a (mon.0) 308 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[29234]: cluster 2024-04-26T16:10:42.631058+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:43 smithi050 bash[29234]: cluster 2024-04-26T16:10:42.631058+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:44.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: audit 2024-04-26T16:10:43.231932+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: audit 2024-04-26T16:10:43.231932+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: cluster 2024-04-26T16:10:43.238350+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: cluster 2024-04-26T16:10:43.238350+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: audit 2024-04-26T16:10:43.239223+0000 mon.a (mon.0) 311 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: audit 2024-04-26T16:10:43.239223+0000 mon.a (mon.0) 311 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: audit 2024-04-26T16:10:43.239783+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[29234]: audit 2024-04-26T16:10:43.239783+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: audit 2024-04-26T16:10:43.231932+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: audit 2024-04-26T16:10:43.231932+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: cluster 2024-04-26T16:10:43.238350+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-26T16:10:44.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: cluster 2024-04-26T16:10:43.238350+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-26T16:10:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: audit 2024-04-26T16:10:43.239223+0000 mon.a (mon.0) 311 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:10:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: audit 2024-04-26T16:10:43.239223+0000 mon.a (mon.0) 311 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:10:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: audit 2024-04-26T16:10:43.239783+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:44.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:44 smithi050 bash[21463]: audit 2024-04-26T16:10:43.239783+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:44.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: audit 2024-04-26T16:10:43.231932+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T16:10:44.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: audit 2024-04-26T16:10:43.231932+0000 mon.a (mon.0) 309 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-26T16:10:44.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: cluster 2024-04-26T16:10:43.238350+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-26T16:10:44.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: cluster 2024-04-26T16:10:43.238350+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2024-04-26T16:10:44.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: audit 2024-04-26T16:10:43.239223+0000 mon.a (mon.0) 311 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:10:44.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: audit 2024-04-26T16:10:43.239223+0000 mon.a (mon.0) 311 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:10:44.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: audit 2024-04-26T16:10:43.239783+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:44.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:44 smithi190 bash[23868]: audit 2024-04-26T16:10:43.239783+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:44.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.652+0000 7f3eb7fff700 1 -- 172.21.15.50:0/53199856 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ebc000bf0 con 0x7f3eb8071df0 2024-04-26T16:10:44.650 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 0 on host 'smithi050' 2024-04-26T16:10:44.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.652+0000 7f3ed5748700 10 librados: watch_flush enter 2024-04-26T16:10:44.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.652+0000 7f3ed5748700 10 librados: watch_flush exit 2024-04-26T16:10:44.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.652+0000 7f3ed5748700 20 client.14205.objecter shutdown clearing up homeless session... 2024-04-26T16:10:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.652+0000 7f3ed5748700 10 client.14205.objecter successfully canceled tick 2024-04-26T16:10:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f3eb8071df0 msgr2=0x7f3eb80742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:10:44.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f3eb8071df0 0x7f3eb80742b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400c010 tx=0x7f3ec400b540 comp rx=0 tx=0).stop 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 msgr2=0x7f3ed019f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed019f160 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ec000d980 tx=0x7f3ec000dc90 comp rx=0 tx=0).stop 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 shutdown_connections 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ed0107040 0x7f3ed01a6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ed0101d00 0x7f3ed019f160 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f3eb8071df0 0x7f3eb80742b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 --2- 172.21.15.50:0/53199856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ed0104670 0x7f3ed019f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 >> 172.21.15.50:0/53199856 conn(0x7f3ed00fd940 msgr2=0x7f3ed01029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 shutdown_connections 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 -- 172.21.15.50:0/53199856 wait complete. 2024-04-26T16:10:44.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:10:44.656+0000 7f3ed5748700 1 librados: shutdown 2024-04-26T16:10:45.362 DEBUG:teuthology.orchestra.run.smithi050:osd.0> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.0.service 2024-04-26T16:10:45.365 INFO:tasks.cephadm:Deploying osd.1 on smithi050 with /dev/vg_nvme/lv_3... 2024-04-26T16:10:45.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T16:10:45.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.234152+0000 mon.a (mon.0) 313 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:10:45.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.234152+0000 mon.a (mon.0) 313 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:10:45.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: cluster 2024-04-26T16:10:44.239681+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-26T16:10:45.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: cluster 2024-04-26T16:10:44.239681+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.241452+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.241452+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.251387+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.251387+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.564653+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.564653+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' 2024-04-26T16:10:45.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: cluster 2024-04-26T16:10:44.631359+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: cluster 2024-04-26T16:10:44.631359+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.643304+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.643304+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.650230+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.650230+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.672972+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:45.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:45 smithi190 bash[23868]: audit 2024-04-26T16:10:44.672972+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.234152+0000 mon.a (mon.0) 313 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.234152+0000 mon.a (mon.0) 313 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: cluster 2024-04-26T16:10:44.239681+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: cluster 2024-04-26T16:10:44.239681+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.241452+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.241452+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.251387+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.251387+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.564653+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.564653+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: cluster 2024-04-26T16:10:44.631359+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: cluster 2024-04-26T16:10:44.631359+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.643304+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.643304+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.650230+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.650230+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.672972+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:45.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[21463]: audit 2024-04-26T16:10:44.672972+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.234152+0000 mon.a (mon.0) 313 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.234152+0000 mon.a (mon.0) 313 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: cluster 2024-04-26T16:10:44.239681+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: cluster 2024-04-26T16:10:44.239681+0000 mon.a (mon.0) 314 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.241452+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.241452+0000 mon.a (mon.0) 315 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.251387+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.251387+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.564653+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.564653+0000 mon.a (mon.0) 317 : audit [INF] from='osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989]' entity='osd.0' 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: cluster 2024-04-26T16:10:44.631359+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: cluster 2024-04-26T16:10:44.631359+0000 mgr.y (mgr.14152) 108 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.643304+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.643304+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.650230+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.650230+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:45.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.672972+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:45.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:45 smithi050 bash[29234]: audit 2024-04-26T16:10:44.672972+0000 mon.a (mon.0) 320 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:10:46.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:43.582260+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:10:46.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:43.582260+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:10:46.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:43.582395+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:10:46.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:43.582395+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:10:46.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: audit 2024-04-26T16:10:45.253116+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: audit 2024-04-26T16:10:45.253116+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:45.569580+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] boot 2024-04-26T16:10:46.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:45.569580+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] boot 2024-04-26T16:10:46.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:45.569693+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-26T16:10:46.579 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: cluster 2024-04-26T16:10:45.569693+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-26T16:10:46.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: audit 2024-04-26T16:10:45.570518+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.580 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:46 smithi190 bash[23868]: audit 2024-04-26T16:10:45.570518+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:43.582260+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:10:46.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:43.582260+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:10:46.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:43.582395+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:43.582395+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: audit 2024-04-26T16:10:45.253116+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: audit 2024-04-26T16:10:45.253116+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:45.569580+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] boot 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:45.569580+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] boot 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:45.569693+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: cluster 2024-04-26T16:10:45.569693+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: audit 2024-04-26T16:10:45.570518+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[21463]: audit 2024-04-26T16:10:45.570518+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:43.582260+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:43.582260+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:43.582395+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:43.582395+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: audit 2024-04-26T16:10:45.253116+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: audit 2024-04-26T16:10:45.253116+0000 mon.a (mon.0) 321 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:45.569580+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] boot 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:45.569580+0000 mon.a (mon.0) 322 : cluster [INF] osd.0 [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] boot 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:45.569693+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: cluster 2024-04-26T16:10:45.569693+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: audit 2024-04-26T16:10:45.570518+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:46.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:46 smithi050 bash[29234]: audit 2024-04-26T16:10:45.570518+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:10:47.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:47 smithi190 bash[23868]: cluster 2024-04-26T16:10:46.631810+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:47.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:47 smithi190 bash[23868]: cluster 2024-04-26T16:10:46.631810+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:47.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:47 smithi050 bash[29234]: cluster 2024-04-26T16:10:46.631810+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:47.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:47 smithi050 bash[29234]: cluster 2024-04-26T16:10:46.631810+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:47 smithi050 bash[21463]: cluster 2024-04-26T16:10:46.631810+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:47 smithi050 bash[21463]: cluster 2024-04-26T16:10:46.631810+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:48.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:48 smithi190 bash[23868]: cluster 2024-04-26T16:10:47.262816+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-26T16:10:48.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:48 smithi190 bash[23868]: cluster 2024-04-26T16:10:47.262816+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-26T16:10:48.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:48 smithi050 bash[21463]: cluster 2024-04-26T16:10:47.262816+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-26T16:10:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:48 smithi050 bash[21463]: cluster 2024-04-26T16:10:47.262816+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-26T16:10:48.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:48 smithi050 bash[29234]: cluster 2024-04-26T16:10:47.262816+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-26T16:10:48.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:48 smithi050 bash[29234]: cluster 2024-04-26T16:10:47.262816+0000 mon.a (mon.0) 325 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2024-04-26T16:10:49.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:49 smithi190 bash[23868]: cluster 2024-04-26T16:10:48.632274+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:49.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:49 smithi190 bash[23868]: cluster 2024-04-26T16:10:48.632274+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:49.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:49 smithi050 bash[21463]: cluster 2024-04-26T16:10:48.632274+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:49 smithi050 bash[21463]: cluster 2024-04-26T16:10:48.632274+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:49.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:49 smithi050 bash[29234]: cluster 2024-04-26T16:10:48.632274+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:49.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:49 smithi050 bash[29234]: cluster 2024-04-26T16:10:48.632274+0000 mgr.y (mgr.14152) 110 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:10:51.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:51 smithi050 bash[29234]: cluster 2024-04-26T16:10:50.632782+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:51.914 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:51 smithi050 bash[29234]: cluster 2024-04-26T16:10:50.632782+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:51.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:51 smithi050 bash[21463]: cluster 2024-04-26T16:10:50.632782+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:51.914 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:51 smithi050 bash[21463]: cluster 2024-04-26T16:10:50.632782+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:52.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:51 smithi190 bash[23868]: cluster 2024-04-26T16:10:50.632782+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:52.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:51 smithi190 bash[23868]: cluster 2024-04-26T16:10:50.632782+0000 mgr.y (mgr.14152) 111 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T16:10:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T16:10:53.718 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T16:10:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T16:10:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0101034 s, 1.0 GB/s 2024-04-26T16:10:53.718 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T16:10:53.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:53 smithi050 bash[29234]: cluster 2024-04-26T16:10:52.633274+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:53.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:53 smithi050 bash[29234]: cluster 2024-04-26T16:10:52.633274+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:53.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:53 smithi050 bash[21463]: cluster 2024-04-26T16:10:52.633274+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:53.961 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:53 smithi050 bash[21463]: cluster 2024-04-26T16:10:52.633274+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:54.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:53 smithi190 bash[23868]: cluster 2024-04-26T16:10:52.633274+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:54.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:53 smithi190 bash[23868]: cluster 2024-04-26T16:10:52.633274+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:54.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_3 2024-04-26T16:10:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: cluster 2024-04-26T16:10:54.633800+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: cluster 2024-04-26T16:10:54.633800+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.972609+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.972609+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.981307+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.981307+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.983035+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.983035+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.990523+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.990523+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.994834+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.994834+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.996341+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:54.996341+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:55.005066+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:55 smithi190 bash[23868]: audit 2024-04-26T16:10:55.005066+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: cluster 2024-04-26T16:10:54.633800+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: cluster 2024-04-26T16:10:54.633800+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.972609+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.972609+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.981307+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.981307+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.983035+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.983035+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.990523+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.990523+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.994834+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.994834+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.996341+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:54.996341+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:55.005066+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[21463]: audit 2024-04-26T16:10:55.005066+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: cluster 2024-04-26T16:10:54.633800+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:56.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: cluster 2024-04-26T16:10:54.633800+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.972609+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.972609+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.981307+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.981307+0000 mon.a (mon.0) 327 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.983035+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.983035+0000 mon.a (mon.0) 328 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.990523+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.990523+0000 mon.a (mon.0) 329 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.994834+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.994834+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.996341+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:54.996341+0000 mon.a (mon.0) 331 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:55.005066+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:55 smithi050 bash[29234]: audit 2024-04-26T16:10:55.005066+0000 mon.a (mon.0) 332 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:10:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:56 smithi190 bash[23868]: cephadm 2024-04-26T16:10:54.964339+0000 mgr.y (mgr.14152) 114 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:10:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:56 smithi190 bash[23868]: cephadm 2024-04-26T16:10:54.964339+0000 mgr.y (mgr.14152) 114 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:10:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:56 smithi190 bash[23868]: cephadm 2024-04-26T16:10:54.984298+0000 mgr.y (mgr.14152) 115 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 16245M 2024-04-26T16:10:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:56 smithi190 bash[23868]: cephadm 2024-04-26T16:10:54.984298+0000 mgr.y (mgr.14152) 115 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 16245M 2024-04-26T16:10:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[29234]: cephadm 2024-04-26T16:10:54.964339+0000 mgr.y (mgr.14152) 114 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[29234]: cephadm 2024-04-26T16:10:54.964339+0000 mgr.y (mgr.14152) 114 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[29234]: cephadm 2024-04-26T16:10:54.984298+0000 mgr.y (mgr.14152) 115 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 16245M 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[29234]: cephadm 2024-04-26T16:10:54.984298+0000 mgr.y (mgr.14152) 115 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 16245M 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[21463]: cephadm 2024-04-26T16:10:54.964339+0000 mgr.y (mgr.14152) 114 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[21463]: cephadm 2024-04-26T16:10:54.964339+0000 mgr.y (mgr.14152) 114 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[21463]: cephadm 2024-04-26T16:10:54.984298+0000 mgr.y (mgr.14152) 115 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 16245M 2024-04-26T16:10:57.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:56 smithi050 bash[21463]: cephadm 2024-04-26T16:10:54.984298+0000 mgr.y (mgr.14152) 115 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 16245M 2024-04-26T16:10:58.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:57 smithi190 bash[23868]: cluster 2024-04-26T16:10:56.634311+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:58.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:57 smithi190 bash[23868]: cluster 2024-04-26T16:10:56.634311+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:57 smithi050 bash[21463]: cluster 2024-04-26T16:10:56.634311+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:57 smithi050 bash[21463]: cluster 2024-04-26T16:10:56.634311+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:57 smithi050 bash[29234]: cluster 2024-04-26T16:10:56.634311+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:57 smithi050 bash[29234]: cluster 2024-04-26T16:10:56.634311+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:59.241 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:10:59.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:59 smithi050 bash[21463]: cluster 2024-04-26T16:10:58.634781+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:59.994 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:10:59 smithi050 bash[21463]: cluster 2024-04-26T16:10:58.634781+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:59.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:59 smithi050 bash[29234]: cluster 2024-04-26T16:10:58.634781+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:10:59.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:10:59 smithi050 bash[29234]: cluster 2024-04-26T16:10:58.634781+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:00.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:59 smithi190 bash[23868]: cluster 2024-04-26T16:10:58.634781+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:00.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:10:59 smithi190 bash[23868]: cluster 2024-04-26T16:10:58.634781+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- 172.21.15.50:0/1411955786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f81091c0 msgr2=0x7fd7f810b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/1411955786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f81091c0 0x7fd7f810b5d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000b0a0 tx=0x7fd7f000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- 172.21.15.50:0/1411955786 shutdown_connections 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/1411955786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8103f60 0x7fd7f8106350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/1411955786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f81091c0 0x7fd7f810b5d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/1411955786 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f8106890 0x7fd7f8108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- 172.21.15.50:0/1411955786 >> 172.21.15.50:0/1411955786 conn(0x7fd7f80fd910 msgr2=0x7fd7f80ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:11:01.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- 172.21.15.50:0/1411955786 shutdown_connections 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- 172.21.15.50:0/1411955786 wait complete. 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 librados: starting msgr at 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 librados: starting objecter 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 Processor -- start 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- start start 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 librados: setting wanted keys 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 librados: calling monclient init 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8103f60 0x7fd7f8198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8106890 0x7fd7f8199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f81091c0 0x7fd7f81a0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f810e410 con 0x7fd7f8103f60 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f810e290 con 0x7fd7f8106890 2024-04-26T16:11:01.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.720+0000 7fd7fcb99700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f810e590 con 0x7fd7f81091c0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8103f60 0x7fd7f8198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f81091c0 0x7fd7f81a0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8103f60 0x7fd7f8198bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38024/0 (socket says 172.21.15.50:38024) 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f659c700 1 -- 172.21.15.50:0/106554208 learned_addr learned my addr 172.21.15.50:0/106554208 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7effff700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8106890 0x7fd7f8199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 -- 172.21.15.50:0/106554208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8106890 msgr2=0x7fd7f8199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8106890 0x7fd7f8199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 -- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8103f60 msgr2=0x7fd7f8198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8103f60 0x7fd7f8198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 -- 172.21.15.50:0/106554208 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f81a07e0 con 0x7fd7f81091c0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f6d9d700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f81091c0 0x7fd7f81a0140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd7f806ca70 tx=0x7fd7f0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 10 client.?.objecter ms_handle_connect 0x7fd7f81091c0 2024-04-26T16:11:01.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0021030 con 0x7fd7f81091c0 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd7f0007980 con 0x7fd7f81091c0 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f00044a0 con 0x7fd7f81091c0 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f81a0a70 con 0x7fd7f81091c0 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7fcb99700 10 client.24140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f81a0ed0 con 0x7fd7f81091c0 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7fcb99700 1 librados: init done 2024-04-26T16:11:01.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f806a110 con 0x7fd7f81091c0 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 14) ==== 93847+0+0 (secure 0 0 0) 0x7fd7f0024070 con 0x7fd7f81091c0 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd7e4071df0 0x7fd7e40742b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fd7f000f020 con 0x7fd7f81091c0 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 10 client.24140.objecter ms_dispatch 0x7fd7f8001550 osd_map(9..9 src has 1..9) 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 3 client.24140.objecter handle_osd_map got epochs [9,9] > 0 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 3 client.24140.objecter handle_osd_map decoding full epoch 9 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 20 client.24140.objecter dump_active .. 0 homeless 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f659c700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd7e4071df0 0x7fd7e40742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7f659c700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd7e4071df0 0x7fd7e40742b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8007d10 tx=0x7fd7e8007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:11:01.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.724+0000 7fd7ef7fe700 10 client.24140.objecter ms_handle_connect 0x7fd7e4071df0 2024-04-26T16:11:01.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.732+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd7f005de20 con 0x7fd7f81091c0 2024-04-26T16:11:01.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:01 smithi050 bash[29234]: cluster 2024-04-26T16:11:00.635304+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:01.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:01 smithi050 bash[29234]: cluster 2024-04-26T16:11:00.635304+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:01 smithi050 bash[21463]: cluster 2024-04-26T16:11:00.635304+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:01 smithi050 bash[21463]: cluster 2024-04-26T16:11:00.635304+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:01.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:01.964+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fd7f8063860 con 0x7fd7e4071df0 2024-04-26T16:11:02.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:01 smithi190 bash[23868]: cluster 2024-04-26T16:11:00.635304+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:02.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:01 smithi190 bash[23868]: cluster 2024-04-26T16:11:00.635304+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:02 smithi190 bash[23868]: audit 2024-04-26T16:11:01.977647+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:02 smithi190 bash[23868]: audit 2024-04-26T16:11:01.977647+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:02 smithi190 bash[23868]: audit 2024-04-26T16:11:01.980809+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:02 smithi190 bash[23868]: audit 2024-04-26T16:11:01.980809+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:02 smithi190 bash[23868]: audit 2024-04-26T16:11:01.982096+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:02 smithi190 bash[23868]: audit 2024-04-26T16:11:01.982096+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:03.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[29234]: audit 2024-04-26T16:11:01.977647+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[29234]: audit 2024-04-26T16:11:01.977647+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[29234]: audit 2024-04-26T16:11:01.980809+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[29234]: audit 2024-04-26T16:11:01.980809+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[29234]: audit 2024-04-26T16:11:01.982096+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[29234]: audit 2024-04-26T16:11:01.982096+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[21463]: audit 2024-04-26T16:11:01.977647+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[21463]: audit 2024-04-26T16:11:01.977647+0000 mon.a (mon.0) 333 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[21463]: audit 2024-04-26T16:11:01.980809+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[21463]: audit 2024-04-26T16:11:01.980809+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[21463]: audit 2024-04-26T16:11:01.982096+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:03.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:02 smithi050 bash[21463]: audit 2024-04-26T16:11:01.982096+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:04.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:03 smithi190 bash[23868]: audit 2024-04-26T16:11:01.970967+0000 mgr.y (mgr.14152) 119 : audit [DBG] from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:04.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:03 smithi190 bash[23868]: audit 2024-04-26T16:11:01.970967+0000 mgr.y (mgr.14152) 119 : audit [DBG] from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:04.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:03 smithi190 bash[23868]: cluster 2024-04-26T16:11:02.635818+0000 mgr.y (mgr.14152) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:04.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:03 smithi190 bash[23868]: cluster 2024-04-26T16:11:02.635818+0000 mgr.y (mgr.14152) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[21463]: audit 2024-04-26T16:11:01.970967+0000 mgr.y (mgr.14152) 119 : audit [DBG] from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[21463]: audit 2024-04-26T16:11:01.970967+0000 mgr.y (mgr.14152) 119 : audit [DBG] from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[21463]: cluster 2024-04-26T16:11:02.635818+0000 mgr.y (mgr.14152) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[21463]: cluster 2024-04-26T16:11:02.635818+0000 mgr.y (mgr.14152) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[29234]: audit 2024-04-26T16:11:01.970967+0000 mgr.y (mgr.14152) 119 : audit [DBG] from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[29234]: audit 2024-04-26T16:11:01.970967+0000 mgr.y (mgr.14152) 119 : audit [DBG] from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[29234]: cluster 2024-04-26T16:11:02.635818+0000 mgr.y (mgr.14152) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:04.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:03 smithi050 bash[29234]: cluster 2024-04-26T16:11:02.635818+0000 mgr.y (mgr.14152) 120 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:05 smithi190 bash[23868]: cluster 2024-04-26T16:11:04.636303+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:05 smithi190 bash[23868]: cluster 2024-04-26T16:11:04.636303+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:05 smithi050 bash[21463]: cluster 2024-04-26T16:11:04.636303+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:05 smithi050 bash[21463]: cluster 2024-04-26T16:11:04.636303+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:05 smithi050 bash[29234]: cluster 2024-04-26T16:11:04.636303+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:05 smithi050 bash[29234]: cluster 2024-04-26T16:11:04.636303+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:06.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:06.728+0000 7fd7f7fff700 10 client.24140.objecter tick 2024-04-26T16:11:08.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:07 smithi190 bash[23868]: cluster 2024-04-26T16:11:06.636797+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:08.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:07 smithi190 bash[23868]: cluster 2024-04-26T16:11:06.636797+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:08.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:07 smithi050 bash[29234]: cluster 2024-04-26T16:11:06.636797+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:08.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:07 smithi050 bash[29234]: cluster 2024-04-26T16:11:06.636797+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:08.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:07 smithi050 bash[21463]: cluster 2024-04-26T16:11:06.636797+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:08.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:07 smithi050 bash[21463]: cluster 2024-04-26T16:11:06.636797+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:09 smithi190 bash[23868]: cluster 2024-04-26T16:11:08.637161+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:09 smithi190 bash[23868]: cluster 2024-04-26T16:11:08.637161+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:10.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:09 smithi050 bash[21463]: cluster 2024-04-26T16:11:08.637161+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:10.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:09 smithi050 bash[21463]: cluster 2024-04-26T16:11:08.637161+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:10.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:09 smithi050 bash[29234]: cluster 2024-04-26T16:11:08.637161+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:10.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:09 smithi050 bash[29234]: cluster 2024-04-26T16:11:08.637161+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:11.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: audit 2024-04-26T16:11:10.414381+0000 mon.a (mon.0) 336 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]: dispatch 2024-04-26T16:11:11.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: audit 2024-04-26T16:11:10.414381+0000 mon.a (mon.0) 336 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]: dispatch 2024-04-26T16:11:11.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: audit 2024-04-26T16:11:10.416520+0000 mon.a (mon.0) 337 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]': finished 2024-04-26T16:11:11.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: audit 2024-04-26T16:11:10.416520+0000 mon.a (mon.0) 337 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]': finished 2024-04-26T16:11:11.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: cluster 2024-04-26T16:11:10.422630+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-26T16:11:11.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: cluster 2024-04-26T16:11:10.422630+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-26T16:11:11.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: audit 2024-04-26T16:11:10.422995+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:11.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:10 smithi190 bash[23868]: audit 2024-04-26T16:11:10.422995+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: audit 2024-04-26T16:11:10.414381+0000 mon.a (mon.0) 336 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]: dispatch 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: audit 2024-04-26T16:11:10.414381+0000 mon.a (mon.0) 336 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]: dispatch 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: audit 2024-04-26T16:11:10.416520+0000 mon.a (mon.0) 337 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]': finished 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: audit 2024-04-26T16:11:10.416520+0000 mon.a (mon.0) 337 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]': finished 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: cluster 2024-04-26T16:11:10.422630+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: cluster 2024-04-26T16:11:10.422630+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: audit 2024-04-26T16:11:10.422995+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[21463]: audit 2024-04-26T16:11:10.422995+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: audit 2024-04-26T16:11:10.414381+0000 mon.a (mon.0) 336 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]: dispatch 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: audit 2024-04-26T16:11:10.414381+0000 mon.a (mon.0) 336 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]: dispatch 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: audit 2024-04-26T16:11:10.416520+0000 mon.a (mon.0) 337 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]': finished 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: audit 2024-04-26T16:11:10.416520+0000 mon.a (mon.0) 337 : audit [INF] from='client.? 172.21.15.50:0/2587054529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed"}]': finished 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: cluster 2024-04-26T16:11:10.422630+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: cluster 2024-04-26T16:11:10.422630+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: audit 2024-04-26T16:11:10.422995+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:11.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:10 smithi050 bash[29234]: audit 2024-04-26T16:11:10.422995+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:11.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:11.732+0000 7fd7f7fff700 10 client.24140.objecter tick 2024-04-26T16:11:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:11 smithi190 bash[23868]: cluster 2024-04-26T16:11:10.637596+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:11 smithi190 bash[23868]: cluster 2024-04-26T16:11:10.637596+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:11 smithi190 bash[23868]: audit 2024-04-26T16:11:11.171124+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? 172.21.15.50:0/311479842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:11:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:11 smithi190 bash[23868]: audit 2024-04-26T16:11:11.171124+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? 172.21.15.50:0/311479842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:11:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[29234]: cluster 2024-04-26T16:11:10.637596+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[29234]: cluster 2024-04-26T16:11:10.637596+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[29234]: audit 2024-04-26T16:11:11.171124+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? 172.21.15.50:0/311479842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:11:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[29234]: audit 2024-04-26T16:11:11.171124+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? 172.21.15.50:0/311479842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:11:12.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[21463]: cluster 2024-04-26T16:11:10.637596+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:12.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[21463]: cluster 2024-04-26T16:11:10.637596+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:12.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[21463]: audit 2024-04-26T16:11:11.171124+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? 172.21.15.50:0/311479842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:11:12.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:11 smithi050 bash[21463]: audit 2024-04-26T16:11:11.171124+0000 mon.a (mon.0) 340 : audit [DBG] from='client.? 172.21.15.50:0/311479842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:11:13.932 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:13 smithi050 bash[21463]: cluster 2024-04-26T16:11:12.638213+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:13.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:13 smithi050 bash[21463]: cluster 2024-04-26T16:11:12.638213+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:13.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:13 smithi050 bash[29234]: cluster 2024-04-26T16:11:12.638213+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:13.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:13 smithi050 bash[29234]: cluster 2024-04-26T16:11:12.638213+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:14.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:13 smithi190 bash[23868]: cluster 2024-04-26T16:11:12.638213+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:14.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:13 smithi190 bash[23868]: cluster 2024-04-26T16:11:12.638213+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:15 smithi190 bash[23868]: cluster 2024-04-26T16:11:14.638695+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:15 smithi190 bash[23868]: cluster 2024-04-26T16:11:14.638695+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:15 smithi050 bash[21463]: cluster 2024-04-26T16:11:14.638695+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:15 smithi050 bash[21463]: cluster 2024-04-26T16:11:14.638695+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:15 smithi050 bash[29234]: cluster 2024-04-26T16:11:14.638695+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:15 smithi050 bash[29234]: cluster 2024-04-26T16:11:14.638695+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:16.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:16.736+0000 7fd7f7fff700 10 client.24140.objecter tick 2024-04-26T16:11:18.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:17 smithi190 bash[23868]: cluster 2024-04-26T16:11:16.639291+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:18.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:17 smithi190 bash[23868]: cluster 2024-04-26T16:11:16.639291+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:18.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:17 smithi050 bash[21463]: cluster 2024-04-26T16:11:16.639291+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:18.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:17 smithi050 bash[21463]: cluster 2024-04-26T16:11:16.639291+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:18.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:17 smithi050 bash[29234]: cluster 2024-04-26T16:11:16.639291+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:18.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:17 smithi050 bash[29234]: cluster 2024-04-26T16:11:16.639291+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:19 smithi190 bash[23868]: cluster 2024-04-26T16:11:18.639756+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:19 smithi190 bash[23868]: cluster 2024-04-26T16:11:18.639756+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:19 smithi050 bash[21463]: cluster 2024-04-26T16:11:18.639756+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:19 smithi050 bash[21463]: cluster 2024-04-26T16:11:18.639756+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:20.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:19 smithi050 bash[29234]: cluster 2024-04-26T16:11:18.639756+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:20.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:19 smithi050 bash[29234]: cluster 2024-04-26T16:11:18.639756+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:21.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:21.740+0000 7fd7f7fff700 10 client.24140.objecter tick 2024-04-26T16:11:21.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:21 smithi050 bash[21463]: cluster 2024-04-26T16:11:20.640288+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:21.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:21 smithi050 bash[21463]: cluster 2024-04-26T16:11:20.640288+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:21.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:21 smithi050 bash[29234]: cluster 2024-04-26T16:11:20.640288+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:21.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:21 smithi050 bash[29234]: cluster 2024-04-26T16:11:20.640288+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:22.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:21 smithi190 bash[23868]: cluster 2024-04-26T16:11:20.640288+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:22.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:21 smithi190 bash[23868]: cluster 2024-04-26T16:11:20.640288+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:22 smithi190 bash[23868]: audit 2024-04-26T16:11:22.800056+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T16:11:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:22 smithi190 bash[23868]: audit 2024-04-26T16:11:22.800056+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T16:11:23.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:22 smithi050 bash[21463]: audit 2024-04-26T16:11:22.800056+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T16:11:23.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:22 smithi050 bash[21463]: audit 2024-04-26T16:11:22.800056+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T16:11:23.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:22 smithi050 bash[29234]: audit 2024-04-26T16:11:22.800056+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T16:11:23.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:22 smithi050 bash[29234]: audit 2024-04-26T16:11:22.800056+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-26T16:11:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:23 smithi190 bash[23868]: cluster 2024-04-26T16:11:22.640764+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:24.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:23 smithi190 bash[23868]: cluster 2024-04-26T16:11:22.640764+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:24.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:23 smithi190 bash[23868]: audit 2024-04-26T16:11:22.801900+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:24.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:23 smithi190 bash[23868]: audit 2024-04-26T16:11:22.801900+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:24.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:23 smithi190 bash[23868]: cephadm 2024-04-26T16:11:22.803581+0000 mgr.y (mgr.14152) 131 : cephadm [INF] Deploying daemon osd.1 on smithi050 2024-04-26T16:11:24.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:23 smithi190 bash[23868]: cephadm 2024-04-26T16:11:22.803581+0000 mgr.y (mgr.14152) 131 : cephadm [INF] Deploying daemon osd.1 on smithi050 2024-04-26T16:11:24.104 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[21463]: cluster 2024-04-26T16:11:22.640764+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:24.104 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[21463]: cluster 2024-04-26T16:11:22.640764+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:24.104 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[21463]: audit 2024-04-26T16:11:22.801900+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[21463]: audit 2024-04-26T16:11:22.801900+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[21463]: cephadm 2024-04-26T16:11:22.803581+0000 mgr.y (mgr.14152) 131 : cephadm [INF] Deploying daemon osd.1 on smithi050 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[21463]: cephadm 2024-04-26T16:11:22.803581+0000 mgr.y (mgr.14152) 131 : cephadm [INF] Deploying daemon osd.1 on smithi050 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[29234]: cluster 2024-04-26T16:11:22.640764+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[29234]: cluster 2024-04-26T16:11:22.640764+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[29234]: audit 2024-04-26T16:11:22.801900+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[29234]: audit 2024-04-26T16:11:22.801900+0000 mon.a (mon.0) 342 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[29234]: cephadm 2024-04-26T16:11:22.803581+0000 mgr.y (mgr.14152) 131 : cephadm [INF] Deploying daemon osd.1 on smithi050 2024-04-26T16:11:24.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:23 smithi050 bash[29234]: cephadm 2024-04-26T16:11:22.803581+0000 mgr.y (mgr.14152) 131 : cephadm [INF] Deploying daemon osd.1 on smithi050 2024-04-26T16:11:26.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:25 smithi190 bash[23868]: cluster 2024-04-26T16:11:24.641236+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:26.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:25 smithi190 bash[23868]: cluster 2024-04-26T16:11:24.641236+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:26.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:25 smithi050 bash[29234]: cluster 2024-04-26T16:11:24.641236+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:26.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:25 smithi050 bash[29234]: cluster 2024-04-26T16:11:24.641236+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:26.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:25 smithi050 bash[21463]: cluster 2024-04-26T16:11:24.641236+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:26.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:25 smithi050 bash[21463]: cluster 2024-04-26T16:11:24.641236+0000 mgr.y (mgr.14152) 132 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:26.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:26.744+0000 7fd7f7fff700 10 client.24140.objecter tick 2024-04-26T16:11:26.912 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:26 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:26.912 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:11:26 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:26.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:26 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:26.913 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:11:26 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:27.185 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:11:27 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:27.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:27.186 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:11:27 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:27.186 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: cluster 2024-04-26T16:11:26.641790+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: cluster 2024-04-26T16:11:26.641790+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: audit 2024-04-26T16:11:27.184060+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: audit 2024-04-26T16:11:27.184060+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: audit 2024-04-26T16:11:27.188814+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: audit 2024-04-26T16:11:27.188814+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: audit 2024-04-26T16:11:27.221957+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:27 smithi190 bash[23868]: audit 2024-04-26T16:11:27.221957+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: cluster 2024-04-26T16:11:26.641790+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: cluster 2024-04-26T16:11:26.641790+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: audit 2024-04-26T16:11:27.184060+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: audit 2024-04-26T16:11:27.184060+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: audit 2024-04-26T16:11:27.188814+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: audit 2024-04-26T16:11:27.188814+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: audit 2024-04-26T16:11:27.221957+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[21463]: audit 2024-04-26T16:11:27.221957+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: cluster 2024-04-26T16:11:26.641790+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: cluster 2024-04-26T16:11:26.641790+0000 mgr.y (mgr.14152) 133 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: audit 2024-04-26T16:11:27.184060+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: audit 2024-04-26T16:11:27.184060+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: audit 2024-04-26T16:11:27.188814+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: audit 2024-04-26T16:11:27.188814+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: audit 2024-04-26T16:11:27.221957+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:28.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:27 smithi050 bash[29234]: audit 2024-04-26T16:11:27.221957+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:29 smithi190 bash[23868]: cluster 2024-04-26T16:11:28.642345+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:29 smithi190 bash[23868]: cluster 2024-04-26T16:11:28.642345+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:29 smithi050 bash[21463]: cluster 2024-04-26T16:11:28.642345+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:29 smithi050 bash[21463]: cluster 2024-04-26T16:11:28.642345+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:30.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:29 smithi050 bash[29234]: cluster 2024-04-26T16:11:28.642345+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:30.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:29 smithi050 bash[29234]: cluster 2024-04-26T16:11:28.642345+0000 mgr.y (mgr.14152) 134 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:31.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:31.748+0000 7fd7f7fff700 10 client.24140.objecter tick 2024-04-26T16:11:32.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:31 smithi190 bash[23868]: cluster 2024-04-26T16:11:30.642817+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:32.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:31 smithi190 bash[23868]: cluster 2024-04-26T16:11:30.642817+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:32.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:31 smithi050 bash[29234]: cluster 2024-04-26T16:11:30.642817+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:32.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:31 smithi050 bash[29234]: cluster 2024-04-26T16:11:30.642817+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:31 smithi050 bash[21463]: cluster 2024-04-26T16:11:30.642817+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:31 smithi050 bash[21463]: cluster 2024-04-26T16:11:30.642817+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.513247+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.513247+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.517621+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.517621+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.518609+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.518609+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.519488+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.519488+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.524800+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: audit 2024-04-26T16:11:32.524800+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: cluster 2024-04-26T16:11:32.643347+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:33.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:33 smithi190 bash[23868]: cluster 2024-04-26T16:11:32.643347+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.513247+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.513247+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.517621+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.517621+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.518609+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.518609+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.519488+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.519488+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.524800+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: audit 2024-04-26T16:11:32.524800+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: cluster 2024-04-26T16:11:32.643347+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[21463]: cluster 2024-04-26T16:11:32.643347+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.513247+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.513247+0000 mon.a (mon.0) 346 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.517621+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.517621+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.518609+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.518609+0000 mon.a (mon.0) 348 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.519488+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.519488+0000 mon.a (mon.0) 349 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.524800+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: audit 2024-04-26T16:11:32.524800+0000 mon.a (mon.0) 350 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: cluster 2024-04-26T16:11:32.643347+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:33 smithi050 bash[29234]: cluster 2024-04-26T16:11:32.643347+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:34.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[21463]: audit 2024-04-26T16:11:33.540340+0000 mon.c (mon.2) 4 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[21463]: audit 2024-04-26T16:11:33.540340+0000 mon.c (mon.2) 4 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[21463]: audit 2024-04-26T16:11:33.540698+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[21463]: audit 2024-04-26T16:11:33.540698+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[29234]: audit 2024-04-26T16:11:33.540340+0000 mon.c (mon.2) 4 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[29234]: audit 2024-04-26T16:11:33.540340+0000 mon.c (mon.2) 4 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[29234]: audit 2024-04-26T16:11:33.540698+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:34 smithi050 bash[29234]: audit 2024-04-26T16:11:33.540698+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:34 smithi190 bash[23868]: audit 2024-04-26T16:11:33.540340+0000 mon.c (mon.2) 4 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:34 smithi190 bash[23868]: audit 2024-04-26T16:11:33.540340+0000 mon.c (mon.2) 4 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:34 smithi190 bash[23868]: audit 2024-04-26T16:11:33.540698+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:34.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:34 smithi190 bash[23868]: audit 2024-04-26T16:11:33.540698+0000 mon.a (mon.0) 351 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-26T16:11:35.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.533569+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.533569+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.538837+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.538837+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.539198+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.539198+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.540111+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.540111+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.540764+0000 mon.a (mon.0) 355 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: audit 2024-04-26T16:11:34.540764+0000 mon.a (mon.0) 355 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.643782+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:35.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.643782+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.533569+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.533569+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.538837+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.538837+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.539198+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.539198+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.540111+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.540111+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.540764+0000 mon.a (mon.0) 355 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: audit 2024-04-26T16:11:34.540764+0000 mon.a (mon.0) 355 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.643782+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:35 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.643782+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.533569+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.533569+0000 mon.a (mon.0) 352 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.538837+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.538837+0000 mon.a (mon.0) 353 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.539198+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.539198+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.540111+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.540111+0000 mon.c (mon.2) 5 : audit [INF] from='osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.540764+0000 mon.a (mon.0) 355 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: audit 2024-04-26T16:11:34.540764+0000 mon.a (mon.0) 355 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:11:35.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.643782+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:35.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:35 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.643782+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:36.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.136+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/106554208 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd7f8063860 con 0x7fd7e4071df0 2024-04-26T16:11:36.134 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 1 on host 'smithi050' 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 10 librados: watch_flush enter 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 10 librados: watch_flush exit 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 20 client.24140.objecter shutdown clearing up homeless session... 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 10 client.24140.objecter successfully canceled tick 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd7e4071df0 msgr2=0x7fd7e40742b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd7e4071df0 0x7fd7e40742b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8007d10 tx=0x7fd7e8007480 comp rx=0 tx=0).stop 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f81091c0 msgr2=0x7fd7f81a0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:36.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f81091c0 0x7fd7f81a0140 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd7f806ca70 tx=0x7fd7f0004d00 comp rx=0 tx=0).stop 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 shutdown_connections 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f81091c0 0x7fd7f81a0140 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8103f60 0x7fd7f8198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fd7e4071df0 0x7fd7e40742b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 --2- 172.21.15.50:0/106554208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8106890 0x7fd7f8199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 >> 172.21.15.50:0/106554208 conn(0x7fd7f80fd910 msgr2=0x7fd7f8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 shutdown_connections 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 -- 172.21.15.50:0/106554208 wait complete. 2024-04-26T16:11:36.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:36.140+0000 7fd7fcb99700 1 librados: shutdown 2024-04-26T16:11:36.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.560757+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.560757+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.560871+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: cluster 2024-04-26T16:11:34.560871+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:35.535870+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:35.535870+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: cluster 2024-04-26T16:11:35.541491+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: cluster 2024-04-26T16:11:35.541491+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:35.542592+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:35.542592+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:35.550554+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:35.550554+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:36.086362+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:36.086362+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:36.096238+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:36.096238+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:36.117151+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:36.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[21463]: audit 2024-04-26T16:11:36.117151+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:36.814 DEBUG:teuthology.orchestra.run.smithi050:osd.1> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.1.service 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.560757+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.560757+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.560871+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: cluster 2024-04-26T16:11:34.560871+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:35.535870+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:35.535870+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: cluster 2024-04-26T16:11:35.541491+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: cluster 2024-04-26T16:11:35.541491+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:35.542592+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:35.542592+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:35.550554+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:35.550554+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:36.086362+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:36.086362+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:36.096238+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:36.096238+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:36.117151+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:36.816 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:36 smithi050 bash[29234]: audit 2024-04-26T16:11:36.117151+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:36.818 INFO:tasks.cephadm:Deploying osd.2 on smithi050 with /dev/vg_nvme/lv_2... 2024-04-26T16:11:36.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.560757+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.560757+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.560871+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: cluster 2024-04-26T16:11:34.560871+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:35.535870+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:35.535870+0000 mon.a (mon.0) 356 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: cluster 2024-04-26T16:11:35.541491+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: cluster 2024-04-26T16:11:35.541491+0000 mon.a (mon.0) 357 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:35.542592+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:35.542592+0000 mon.a (mon.0) 358 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:35.550554+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:35.550554+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:36.086362+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:36.086362+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:36.096238+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:36.096238+0000 mon.a (mon.0) 361 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:36.117151+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:36.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:36 smithi190 bash[23868]: audit 2024-04-26T16:11:36.117151+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: cluster 2024-04-26T16:11:36.549389+0000 mon.a (mon.0) 363 : cluster [INF] osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] boot 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: cluster 2024-04-26T16:11:36.549389+0000 mon.a (mon.0) 363 : cluster [INF] osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] boot 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: cluster 2024-04-26T16:11:36.549434+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: cluster 2024-04-26T16:11:36.549434+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: audit 2024-04-26T16:11:36.549929+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: audit 2024-04-26T16:11:36.549929+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: cluster 2024-04-26T16:11:36.644236+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:37.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:37 smithi190 bash[23868]: cluster 2024-04-26T16:11:36.644236+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:37.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: cluster 2024-04-26T16:11:36.549389+0000 mon.a (mon.0) 363 : cluster [INF] osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] boot 2024-04-26T16:11:37.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: cluster 2024-04-26T16:11:36.549389+0000 mon.a (mon.0) 363 : cluster [INF] osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] boot 2024-04-26T16:11:37.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: cluster 2024-04-26T16:11:36.549434+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-26T16:11:37.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: cluster 2024-04-26T16:11:36.549434+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-26T16:11:37.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: audit 2024-04-26T16:11:36.549929+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:37.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: audit 2024-04-26T16:11:36.549929+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: cluster 2024-04-26T16:11:36.644236+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[29234]: cluster 2024-04-26T16:11:36.644236+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: cluster 2024-04-26T16:11:36.549389+0000 mon.a (mon.0) 363 : cluster [INF] osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] boot 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: cluster 2024-04-26T16:11:36.549389+0000 mon.a (mon.0) 363 : cluster [INF] osd.1 [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] boot 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: cluster 2024-04-26T16:11:36.549434+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: cluster 2024-04-26T16:11:36.549434+0000 mon.a (mon.0) 364 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: audit 2024-04-26T16:11:36.549929+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: audit 2024-04-26T16:11:36.549929+0000 mon.a (mon.0) 365 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: cluster 2024-04-26T16:11:36.644236+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:37.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:37 smithi050 bash[21463]: cluster 2024-04-26T16:11:36.644236+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-26T16:11:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:38 smithi190 bash[23868]: cluster 2024-04-26T16:11:37.569193+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-26T16:11:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:38 smithi190 bash[23868]: cluster 2024-04-26T16:11:37.569193+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-26T16:11:38.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:38 smithi050 bash[29234]: cluster 2024-04-26T16:11:37.569193+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-26T16:11:38.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:38 smithi050 bash[29234]: cluster 2024-04-26T16:11:37.569193+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-26T16:11:38.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:38 smithi050 bash[21463]: cluster 2024-04-26T16:11:37.569193+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-26T16:11:38.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:38 smithi050 bash[21463]: cluster 2024-04-26T16:11:37.569193+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2024-04-26T16:11:39.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:39 smithi190 bash[23868]: cluster 2024-04-26T16:11:38.644803+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:39.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:39 smithi190 bash[23868]: cluster 2024-04-26T16:11:38.644803+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:39.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:39 smithi050 bash[29234]: cluster 2024-04-26T16:11:38.644803+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:39.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:39 smithi050 bash[29234]: cluster 2024-04-26T16:11:38.644803+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:39.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:39 smithi050 bash[21463]: cluster 2024-04-26T16:11:38.644803+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:39.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:39 smithi050 bash[21463]: cluster 2024-04-26T16:11:38.644803+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:41.115 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:11:42.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:41 smithi190 bash[23868]: cluster 2024-04-26T16:11:40.645341+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:42.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:41 smithi190 bash[23868]: cluster 2024-04-26T16:11:40.645341+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:42.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:41 smithi050 bash[21463]: cluster 2024-04-26T16:11:40.645341+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:42.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:41 smithi050 bash[21463]: cluster 2024-04-26T16:11:40.645341+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:41 smithi050 bash[29234]: cluster 2024-04-26T16:11:40.645341+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:41 smithi050 bash[29234]: cluster 2024-04-26T16:11:40.645341+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:44.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:43 smithi190 bash[23868]: cluster 2024-04-26T16:11:42.645934+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:44.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:43 smithi190 bash[23868]: cluster 2024-04-26T16:11:42.645934+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:44.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:43 smithi050 bash[21463]: cluster 2024-04-26T16:11:42.645934+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:44.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:43 smithi050 bash[21463]: cluster 2024-04-26T16:11:42.645934+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:44.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:43 smithi050 bash[29234]: cluster 2024-04-26T16:11:42.645934+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:44.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:43 smithi050 bash[29234]: cluster 2024-04-26T16:11:42.645934+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:45.247 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T16:11:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T16:11:45.248 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T16:11:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T16:11:45.248 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118611 s, 884 MB/s 2024-04-26T16:11:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T16:11:46.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:45 smithi190 bash[23868]: cluster 2024-04-26T16:11:44.646325+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:46.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:45 smithi190 bash[23868]: cluster 2024-04-26T16:11:44.646325+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:46.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:45 smithi050 bash[21463]: cluster 2024-04-26T16:11:44.646325+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:46.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:45 smithi050 bash[21463]: cluster 2024-04-26T16:11:44.646325+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:46.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:45 smithi050 bash[29234]: cluster 2024-04-26T16:11:44.646325+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:46.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:45 smithi050 bash[29234]: cluster 2024-04-26T16:11:44.646325+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:46.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_2 2024-04-26T16:11:48.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: cluster 2024-04-26T16:11:46.646759+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:48.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: cluster 2024-04-26T16:11:46.646759+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:48.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: cephadm 2024-04-26T16:11:46.712102+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:11:48.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: cephadm 2024-04-26T16:11:46.712102+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.723959+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.723959+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.736041+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.736041+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.738666+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.738666+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.740609+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.740609+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: cephadm 2024-04-26T16:11:46.742063+0000 mgr.y (mgr.14152) 145 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 8122M 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: cephadm 2024-04-26T16:11:46.742063+0000 mgr.y (mgr.14152) 145 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 8122M 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.750426+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.750426+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.756273+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:48.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.756273+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:48.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.757949+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:48.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.757949+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:48.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.770066+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:47 smithi190 bash[23868]: audit 2024-04-26T16:11:46.770066+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: cluster 2024-04-26T16:11:46.646759+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: cluster 2024-04-26T16:11:46.646759+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: cephadm 2024-04-26T16:11:46.712102+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: cephadm 2024-04-26T16:11:46.712102+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.723959+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.723959+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.736041+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.736041+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.738666+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.738666+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.740609+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.740609+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: cephadm 2024-04-26T16:11:46.742063+0000 mgr.y (mgr.14152) 145 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 8122M 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: cephadm 2024-04-26T16:11:46.742063+0000 mgr.y (mgr.14152) 145 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 8122M 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.750426+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.750426+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.756273+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.756273+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.757949+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.757949+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.770066+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[21463]: audit 2024-04-26T16:11:46.770066+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: cluster 2024-04-26T16:11:46.646759+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: cluster 2024-04-26T16:11:46.646759+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: cephadm 2024-04-26T16:11:46.712102+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: cephadm 2024-04-26T16:11:46.712102+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.723959+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.723959+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.736041+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.736041+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.738666+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.738666+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.740609+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.740609+0000 mon.a (mon.0) 370 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: cephadm 2024-04-26T16:11:46.742063+0000 mgr.y (mgr.14152) 145 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 8122M 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: cephadm 2024-04-26T16:11:46.742063+0000 mgr.y (mgr.14152) 145 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 8122M 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.750426+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.750426+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.756273+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.756273+0000 mon.a (mon.0) 372 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.757949+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.757949+0000 mon.a (mon.0) 373 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:11:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.770066+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:47 smithi050 bash[29234]: audit 2024-04-26T16:11:46.770066+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:11:50.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:49 smithi190 bash[23868]: cluster 2024-04-26T16:11:48.647236+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:50.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:49 smithi190 bash[23868]: cluster 2024-04-26T16:11:48.647236+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:50.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:49 smithi050 bash[21463]: cluster 2024-04-26T16:11:48.647236+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:50.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:49 smithi050 bash[21463]: cluster 2024-04-26T16:11:48.647236+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:50.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:49 smithi050 bash[29234]: cluster 2024-04-26T16:11:48.647236+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:50.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:49 smithi050 bash[29234]: cluster 2024-04-26T16:11:48.647236+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:50.994 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:11:52.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:51 smithi190 bash[23868]: cluster 2024-04-26T16:11:50.647631+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:52.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:51 smithi190 bash[23868]: cluster 2024-04-26T16:11:50.647631+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:52.113 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:51 smithi050 bash[29234]: cluster 2024-04-26T16:11:50.647631+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:52.113 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:51 smithi050 bash[29234]: cluster 2024-04-26T16:11:50.647631+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:52.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:51 smithi050 bash[21463]: cluster 2024-04-26T16:11:50.647631+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:52.113 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:51 smithi050 bash[21463]: cluster 2024-04-26T16:11:50.647631+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 -- 172.21.15.50:0/510917109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 msgr2=0x7f7a78104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 --2- 172.21.15.50:0/510917109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78104ae0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7a680099c0 tx=0x7f7a68009cd0 comp rx=0 tx=0).stop 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 -- 172.21.15.50:0/510917109 shutdown_connections 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 --2- 172.21.15.50:0/510917109 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a78103480 0x7f7a781038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 --2- 172.21.15.50:0/510917109 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 0x7f7a7810b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 --2- 172.21.15.50:0/510917109 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78104ae0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 -- 172.21.15.50:0/510917109 >> 172.21.15.50:0/510917109 conn(0x7f7a780fea20 msgr2=0x7f7a78100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 -- 172.21.15.50:0/510917109 shutdown_connections 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 -- 172.21.15.50:0/510917109 wait complete. 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 librados: starting msgr at 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 librados: starting objecter 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 Processor -- start 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 -- start start 2024-04-26T16:11:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 librados: setting wanted keys 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.512+0000 7f7a8052c700 1 librados: calling monclient init 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a78103480 0x7f7a780771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 0x7f7a78075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7810e3e0 con 0x7f7a78104680 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7810e260 con 0x7f7a78103480 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7a7810e560 con 0x7f7a78105020 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7dac7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7dac7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78075810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49874/0 (socket says 172.21.15.50:49874) 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7dac7700 1 -- 172.21.15.50:0/4161889287 learned_addr learned my addr 172.21.15.50:0/4161889287 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 0x7f7a78075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7e2c8700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a78103480 0x7f7a780771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 -- 172.21.15.50:0/4161889287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a78103480 msgr2=0x7f7a780771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a78103480 0x7f7a780771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 -- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 msgr2=0x7f7a78075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 -- 172.21.15.50:0/4161889287 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a78076600 con 0x7f7a78105020 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7dac7700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7eac9700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 0x7f7a78075d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000efc0 tx=0x7f7a7000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 10 client.?.objecter ms_handle_connect 0x7f7a78105020 2024-04-26T16:11:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a700092e0 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7a70004bb0 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781a2c40 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 10 client.24167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a70004020 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a781a3150 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 librados: init done 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a7806a110 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 14) ==== 93847+0+0 (secure 0 0 0) 0x7f7a7001b030 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a64071cc0 0x7f7a64074180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 2199+0+0 (secure 0 0 0) 0x7f7a7008f380 con 0x7f7a78105020 2024-04-26T16:11:53.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 10 client.24167.objecter ms_dispatch 0x7f7a78001550 osd_map(14..14 src has 1..14) 2024-04-26T16:11:53.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 3 client.24167.objecter handle_osd_map got epochs [14,14] > 0 2024-04-26T16:11:53.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 3 client.24167.objecter handle_osd_map decoding full epoch 14 2024-04-26T16:11:53.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7e2c8700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a64071cc0 0x7f7a64074180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:11:53.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 20 client.24167.objecter dump_active .. 0 homeless 2024-04-26T16:11:53.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a6f7fe700 10 client.24167.objecter ms_handle_connect 0x7f7a64071cc0 2024-04-26T16:11:53.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.516+0000 7f7a7e2c8700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a64071cc0 0x7f7a64074180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7a74005fd0 tx=0x7f7a74005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:11:53.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.524+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7a70010040 con 0x7f7a78105020 2024-04-26T16:11:53.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:53.752+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7a7802e720 con 0x7f7a64071cc0 2024-04-26T16:11:54.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:53 smithi190 bash[23868]: cluster 2024-04-26T16:11:52.648200+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:54.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:53 smithi190 bash[23868]: cluster 2024-04-26T16:11:52.648200+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:54.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:53 smithi050 bash[21463]: cluster 2024-04-26T16:11:52.648200+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:54.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:53 smithi050 bash[21463]: cluster 2024-04-26T16:11:52.648200+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:54.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:53 smithi050 bash[29234]: cluster 2024-04-26T16:11:52.648200+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:54.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:53 smithi050 bash[29234]: cluster 2024-04-26T16:11:52.648200+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.758594+0000 mgr.y (mgr.14152) 149 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.758594+0000 mgr.y (mgr.14152) 149 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.762597+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.762597+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.768660+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.768660+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.770101+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:55.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:54 smithi190 bash[23868]: audit 2024-04-26T16:11:53.770101+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.758594+0000 mgr.y (mgr.14152) 149 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.758594+0000 mgr.y (mgr.14152) 149 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.762597+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.762597+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.768660+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.768660+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.770101+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:55.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[29234]: audit 2024-04-26T16:11:53.770101+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.758594+0000 mgr.y (mgr.14152) 149 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.758594+0000 mgr.y (mgr.14152) 149 : audit [DBG] from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.762597+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.762597+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.768660+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.768660+0000 mon.a (mon.0) 376 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.770101+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:55.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:54 smithi050 bash[21463]: audit 2024-04-26T16:11:53.770101+0000 mon.a (mon.0) 377 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:11:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:55 smithi190 bash[23868]: cluster 2024-04-26T16:11:54.648641+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:56.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:55 smithi190 bash[23868]: cluster 2024-04-26T16:11:54.648641+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:56.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:55 smithi050 bash[21463]: cluster 2024-04-26T16:11:54.648641+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:56.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:55 smithi050 bash[21463]: cluster 2024-04-26T16:11:54.648641+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:56.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:55 smithi050 bash[29234]: cluster 2024-04-26T16:11:54.648641+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:56.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:55 smithi050 bash[29234]: cluster 2024-04-26T16:11:54.648641+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:57 smithi190 bash[23868]: cluster 2024-04-26T16:11:56.649207+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:57 smithi190 bash[23868]: cluster 2024-04-26T16:11:56.649207+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:57 smithi050 bash[21463]: cluster 2024-04-26T16:11:56.649207+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:57 smithi050 bash[21463]: cluster 2024-04-26T16:11:56.649207+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:57 smithi050 bash[29234]: cluster 2024-04-26T16:11:56.649207+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:57 smithi050 bash[29234]: cluster 2024-04-26T16:11:56.649207+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:58.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:11:58.520+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:11:59.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:59 smithi050 bash[21463]: cluster 2024-04-26T16:11:58.649836+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:59.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:11:59 smithi050 bash[21463]: cluster 2024-04-26T16:11:58.649836+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:59.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:59 smithi050 bash[29234]: cluster 2024-04-26T16:11:58.649836+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:11:59.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:11:59 smithi050 bash[29234]: cluster 2024-04-26T16:11:58.649836+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:00.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:59 smithi190 bash[23868]: cluster 2024-04-26T16:11:58.649836+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:00.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:11:59 smithi190 bash[23868]: cluster 2024-04-26T16:11:58.649836+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:02.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:01 smithi190 bash[23868]: cluster 2024-04-26T16:12:00.650340+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:02.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:01 smithi190 bash[23868]: cluster 2024-04-26T16:12:00.650340+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:02.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:01 smithi050 bash[21463]: cluster 2024-04-26T16:12:00.650340+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:02.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:01 smithi050 bash[21463]: cluster 2024-04-26T16:12:00.650340+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:02.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:01 smithi050 bash[29234]: cluster 2024-04-26T16:12:00.650340+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:02.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:01 smithi050 bash[29234]: cluster 2024-04-26T16:12:00.650340+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: cluster 2024-04-26T16:12:01.897950+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:03.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: cluster 2024-04-26T16:12:01.897950+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: cluster 2024-04-26T16:12:01.898011+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: cluster 2024-04-26T16:12:01.898011+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: audit 2024-04-26T16:12:02.133117+0000 mon.a (mon.0) 378 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]: dispatch 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: audit 2024-04-26T16:12:02.133117+0000 mon.a (mon.0) 378 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]: dispatch 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: audit 2024-04-26T16:12:02.136009+0000 mon.a (mon.0) 379 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]': finished 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: audit 2024-04-26T16:12:02.136009+0000 mon.a (mon.0) 379 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]': finished 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: cluster 2024-04-26T16:12:02.139746+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: cluster 2024-04-26T16:12:02.139746+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: audit 2024-04-26T16:12:02.140045+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:03.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:02 smithi190 bash[23868]: audit 2024-04-26T16:12:02.140045+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: cluster 2024-04-26T16:12:01.897950+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: cluster 2024-04-26T16:12:01.897950+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: cluster 2024-04-26T16:12:01.898011+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: cluster 2024-04-26T16:12:01.898011+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: audit 2024-04-26T16:12:02.133117+0000 mon.a (mon.0) 378 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]: dispatch 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: audit 2024-04-26T16:12:02.133117+0000 mon.a (mon.0) 378 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]: dispatch 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: audit 2024-04-26T16:12:02.136009+0000 mon.a (mon.0) 379 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]': finished 2024-04-26T16:12:03.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: audit 2024-04-26T16:12:02.136009+0000 mon.a (mon.0) 379 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]': finished 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: cluster 2024-04-26T16:12:02.139746+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: cluster 2024-04-26T16:12:02.139746+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: audit 2024-04-26T16:12:02.140045+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[21463]: audit 2024-04-26T16:12:02.140045+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: cluster 2024-04-26T16:12:01.897950+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: cluster 2024-04-26T16:12:01.897950+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: cluster 2024-04-26T16:12:01.898011+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: cluster 2024-04-26T16:12:01.898011+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:03.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: audit 2024-04-26T16:12:02.133117+0000 mon.a (mon.0) 378 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]: dispatch 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: audit 2024-04-26T16:12:02.133117+0000 mon.a (mon.0) 378 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]: dispatch 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: audit 2024-04-26T16:12:02.136009+0000 mon.a (mon.0) 379 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]': finished 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: audit 2024-04-26T16:12:02.136009+0000 mon.a (mon.0) 379 : audit [INF] from='client.? 172.21.15.50:0/1387627265' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2"}]': finished 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: cluster 2024-04-26T16:12:02.139746+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: cluster 2024-04-26T16:12:02.139746+0000 mon.a (mon.0) 380 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: audit 2024-04-26T16:12:02.140045+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:03.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:02 smithi050 bash[29234]: audit 2024-04-26T16:12:02.140045+0000 mon.a (mon.0) 381 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:03.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:03.523+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:12:04.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:03 smithi190 bash[23868]: cluster 2024-04-26T16:12:02.650772+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:04.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:03 smithi190 bash[23868]: cluster 2024-04-26T16:12:02.650772+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:04.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:03 smithi190 bash[23868]: audit 2024-04-26T16:12:02.850372+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? 172.21.15.50:0/1806027841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:04.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:03 smithi190 bash[23868]: audit 2024-04-26T16:12:02.850372+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? 172.21.15.50:0/1806027841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:04.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[21463]: cluster 2024-04-26T16:12:02.650772+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:04.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[21463]: cluster 2024-04-26T16:12:02.650772+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:04.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[21463]: audit 2024-04-26T16:12:02.850372+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? 172.21.15.50:0/1806027841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:04.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[21463]: audit 2024-04-26T16:12:02.850372+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? 172.21.15.50:0/1806027841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:04.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[29234]: cluster 2024-04-26T16:12:02.650772+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:04.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[29234]: cluster 2024-04-26T16:12:02.650772+0000 mgr.y (mgr.14152) 154 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:04.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[29234]: audit 2024-04-26T16:12:02.850372+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? 172.21.15.50:0/1806027841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:04.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:03 smithi050 bash[29234]: audit 2024-04-26T16:12:02.850372+0000 mon.c (mon.2) 6 : audit [DBG] from='client.? 172.21.15.50:0/1806027841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:06.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:05 smithi190 bash[23868]: cluster 2024-04-26T16:12:04.651272+0000 mgr.y (mgr.14152) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:06.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:05 smithi190 bash[23868]: cluster 2024-04-26T16:12:04.651272+0000 mgr.y (mgr.14152) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:06.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:05 smithi050 bash[29234]: cluster 2024-04-26T16:12:04.651272+0000 mgr.y (mgr.14152) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:06.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:05 smithi050 bash[29234]: cluster 2024-04-26T16:12:04.651272+0000 mgr.y (mgr.14152) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:06.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:05 smithi050 bash[21463]: cluster 2024-04-26T16:12:04.651272+0000 mgr.y (mgr.14152) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:06.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:05 smithi050 bash[21463]: cluster 2024-04-26T16:12:04.651272+0000 mgr.y (mgr.14152) 155 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:07 smithi190 bash[23868]: cluster 2024-04-26T16:12:06.651716+0000 mgr.y (mgr.14152) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:07 smithi190 bash[23868]: cluster 2024-04-26T16:12:06.651716+0000 mgr.y (mgr.14152) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:07 smithi050 bash[21463]: cluster 2024-04-26T16:12:06.651716+0000 mgr.y (mgr.14152) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:07 smithi050 bash[21463]: cluster 2024-04-26T16:12:06.651716+0000 mgr.y (mgr.14152) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:07 smithi050 bash[29234]: cluster 2024-04-26T16:12:06.651716+0000 mgr.y (mgr.14152) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:07 smithi050 bash[29234]: cluster 2024-04-26T16:12:06.651716+0000 mgr.y (mgr.14152) 156 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:08.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:08.527+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:12:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:09 smithi190 bash[23868]: cluster 2024-04-26T16:12:08.652190+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:09 smithi190 bash[23868]: cluster 2024-04-26T16:12:08.652190+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:09 smithi050 bash[21463]: cluster 2024-04-26T16:12:08.652190+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:09 smithi050 bash[21463]: cluster 2024-04-26T16:12:08.652190+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:09 smithi050 bash[29234]: cluster 2024-04-26T16:12:08.652190+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:09 smithi050 bash[29234]: cluster 2024-04-26T16:12:08.652190+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:11 smithi190 bash[23868]: cluster 2024-04-26T16:12:10.652686+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:12.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:11 smithi190 bash[23868]: cluster 2024-04-26T16:12:10.652686+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:11 smithi050 bash[21463]: cluster 2024-04-26T16:12:10.652686+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:11 smithi050 bash[21463]: cluster 2024-04-26T16:12:10.652686+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:11 smithi050 bash[29234]: cluster 2024-04-26T16:12:10.652686+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:11 smithi050 bash[29234]: cluster 2024-04-26T16:12:10.652686+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:13.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:13.531+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:12:14.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:13 smithi190 bash[23868]: cluster 2024-04-26T16:12:12.653154+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:14.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:13 smithi190 bash[23868]: cluster 2024-04-26T16:12:12.653154+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:13 smithi050 bash[21463]: cluster 2024-04-26T16:12:12.653154+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:13 smithi050 bash[21463]: cluster 2024-04-26T16:12:12.653154+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:14.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:13 smithi050 bash[29234]: cluster 2024-04-26T16:12:12.653154+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:13 smithi050 bash[29234]: cluster 2024-04-26T16:12:12.653154+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:15.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:14 smithi190 bash[23868]: audit 2024-04-26T16:12:14.760560+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T16:12:15.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:14 smithi190 bash[23868]: audit 2024-04-26T16:12:14.760560+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T16:12:15.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:14 smithi190 bash[23868]: audit 2024-04-26T16:12:14.762249+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:15.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:14 smithi190 bash[23868]: audit 2024-04-26T16:12:14.762249+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:15.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[21463]: audit 2024-04-26T16:12:14.760560+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[21463]: audit 2024-04-26T16:12:14.760560+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[21463]: audit 2024-04-26T16:12:14.762249+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[21463]: audit 2024-04-26T16:12:14.762249+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[29234]: audit 2024-04-26T16:12:14.760560+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[29234]: audit 2024-04-26T16:12:14.760560+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[29234]: audit 2024-04-26T16:12:14.762249+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:15.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:14 smithi050 bash[29234]: audit 2024-04-26T16:12:14.762249+0000 mon.a (mon.0) 383 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:16.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[21463]: cluster 2024-04-26T16:12:14.653579+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[21463]: cluster 2024-04-26T16:12:14.653579+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[21463]: cephadm 2024-04-26T16:12:14.763710+0000 mgr.y (mgr.14152) 161 : cephadm [INF] Deploying daemon osd.2 on smithi050 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[21463]: cephadm 2024-04-26T16:12:14.763710+0000 mgr.y (mgr.14152) 161 : cephadm [INF] Deploying daemon osd.2 on smithi050 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[29234]: cluster 2024-04-26T16:12:14.653579+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[29234]: cluster 2024-04-26T16:12:14.653579+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[29234]: cephadm 2024-04-26T16:12:14.763710+0000 mgr.y (mgr.14152) 161 : cephadm [INF] Deploying daemon osd.2 on smithi050 2024-04-26T16:12:16.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:15 smithi050 bash[29234]: cephadm 2024-04-26T16:12:14.763710+0000 mgr.y (mgr.14152) 161 : cephadm [INF] Deploying daemon osd.2 on smithi050 2024-04-26T16:12:16.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:15 smithi190 bash[23868]: cluster 2024-04-26T16:12:14.653579+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:16.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:15 smithi190 bash[23868]: cluster 2024-04-26T16:12:14.653579+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:16.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:15 smithi190 bash[23868]: cephadm 2024-04-26T16:12:14.763710+0000 mgr.y (mgr.14152) 161 : cephadm [INF] Deploying daemon osd.2 on smithi050 2024-04-26T16:12:16.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:15 smithi190 bash[23868]: cephadm 2024-04-26T16:12:14.763710+0000 mgr.y (mgr.14152) 161 : cephadm [INF] Deploying daemon osd.2 on smithi050 2024-04-26T16:12:17.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:17 smithi050 bash[21463]: cluster 2024-04-26T16:12:16.654073+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:17.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:17 smithi050 bash[21463]: cluster 2024-04-26T16:12:16.654073+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:17.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:17 smithi050 bash[29234]: cluster 2024-04-26T16:12:16.654073+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:17.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:17 smithi050 bash[29234]: cluster 2024-04-26T16:12:16.654073+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:18.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:17 smithi190 bash[23868]: cluster 2024-04-26T16:12:16.654073+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:18.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:17 smithi190 bash[23868]: cluster 2024-04-26T16:12:16.654073+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:18.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:18.535+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:12:19.081 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:18 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.081 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:18 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.081 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:12:18 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.082 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:12:18 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.082 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:12:18 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.353 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:12:19 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.354 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:12:19 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:19.354 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:12:19 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:12:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: cluster 2024-04-26T16:12:18.654518+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: cluster 2024-04-26T16:12:18.654518+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: audit 2024-04-26T16:12:19.353801+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: audit 2024-04-26T16:12:19.353801+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: audit 2024-04-26T16:12:19.359067+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: audit 2024-04-26T16:12:19.359067+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: audit 2024-04-26T16:12:19.386220+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:20.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:19 smithi190 bash[23868]: audit 2024-04-26T16:12:19.386220+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:20.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: cluster 2024-04-26T16:12:18.654518+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: cluster 2024-04-26T16:12:18.654518+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: audit 2024-04-26T16:12:19.353801+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: audit 2024-04-26T16:12:19.353801+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: audit 2024-04-26T16:12:19.359067+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: audit 2024-04-26T16:12:19.359067+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: audit 2024-04-26T16:12:19.386220+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[21463]: audit 2024-04-26T16:12:19.386220+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: cluster 2024-04-26T16:12:18.654518+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: cluster 2024-04-26T16:12:18.654518+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: audit 2024-04-26T16:12:19.353801+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: audit 2024-04-26T16:12:19.353801+0000 mon.a (mon.0) 384 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: audit 2024-04-26T16:12:19.359067+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: audit 2024-04-26T16:12:19.359067+0000 mon.a (mon.0) 385 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:20.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: audit 2024-04-26T16:12:19.386220+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:20.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:19 smithi050 bash[29234]: audit 2024-04-26T16:12:19.386220+0000 mon.a (mon.0) 386 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:22.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:21 smithi190 bash[23868]: cluster 2024-04-26T16:12:20.654822+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:22.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:21 smithi190 bash[23868]: cluster 2024-04-26T16:12:20.654822+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:22.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:21 smithi050 bash[21463]: cluster 2024-04-26T16:12:20.654822+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:22.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:21 smithi050 bash[21463]: cluster 2024-04-26T16:12:20.654822+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:22.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:21 smithi050 bash[29234]: cluster 2024-04-26T16:12:20.654822+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:22.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:21 smithi050 bash[29234]: cluster 2024-04-26T16:12:20.654822+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:23.539+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:12:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:23 smithi190 bash[23868]: cluster 2024-04-26T16:12:22.655284+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:23 smithi190 bash[23868]: cluster 2024-04-26T16:12:22.655284+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:23 smithi050 bash[21463]: cluster 2024-04-26T16:12:22.655284+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:23 smithi050 bash[21463]: cluster 2024-04-26T16:12:22.655284+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:23 smithi050 bash[29234]: cluster 2024-04-26T16:12:22.655284+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:23 smithi050 bash[29234]: cluster 2024-04-26T16:12:22.655284+0000 mgr.y (mgr.14152) 165 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: cluster 2024-04-26T16:12:24.655588+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: cluster 2024-04-26T16:12:24.655588+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:24.668712+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:24.668712+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.038 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:24.673812+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:24.673812+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:25.241780+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:25.241780+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:25.243607+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:25.243607+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:25.252644+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[29234]: audit 2024-04-26T16:12:25.252644+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: cluster 2024-04-26T16:12:24.655588+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: cluster 2024-04-26T16:12:24.655588+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:24.668712+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:24.668712+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:24.673812+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:24.673812+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:25.241780+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:25.241780+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:25.243607+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:25.243607+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:25.252644+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:25 smithi050 bash[21463]: audit 2024-04-26T16:12:25.252644+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: cluster 2024-04-26T16:12:24.655588+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: cluster 2024-04-26T16:12:24.655588+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:26.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:24.668712+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:24.668712+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:24.673812+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:24.673812+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:25.241780+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:26.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:25.241780+0000 mon.a (mon.0) 389 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:26.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:25.243607+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:26.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:25.243607+0000 mon.a (mon.0) 390 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:26.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:25.252644+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:26.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:25 smithi190 bash[23868]: audit 2024-04-26T16:12:25.252644+0000 mon.a (mon.0) 391 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:27.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:26 smithi190 bash[23868]: audit 2024-04-26T16:12:26.049603+0000 mon.a (mon.0) 392 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T16:12:27.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:26 smithi190 bash[23868]: audit 2024-04-26T16:12:26.049603+0000 mon.a (mon.0) 392 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T16:12:27.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:26 smithi050 bash[21463]: audit 2024-04-26T16:12:26.049603+0000 mon.a (mon.0) 392 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T16:12:27.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:26 smithi050 bash[21463]: audit 2024-04-26T16:12:26.049603+0000 mon.a (mon.0) 392 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T16:12:27.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:26 smithi050 bash[29234]: audit 2024-04-26T16:12:26.049603+0000 mon.a (mon.0) 392 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T16:12:27.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:26 smithi050 bash[29234]: audit 2024-04-26T16:12:26.049603+0000 mon.a (mon.0) 392 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: cluster 2024-04-26T16:12:26.656129+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: cluster 2024-04-26T16:12:26.656129+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: audit 2024-04-26T16:12:26.801965+0000 mon.a (mon.0) 393 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: audit 2024-04-26T16:12:26.801965+0000 mon.a (mon.0) 393 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: cluster 2024-04-26T16:12:26.807790+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: cluster 2024-04-26T16:12:26.807790+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-26T16:12:28.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: audit 2024-04-26T16:12:26.808437+0000 mon.a (mon.0) 395 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:12:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: audit 2024-04-26T16:12:26.808437+0000 mon.a (mon.0) 395 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:12:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: audit 2024-04-26T16:12:26.809030+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:28.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:27 smithi190 bash[23868]: audit 2024-04-26T16:12:26.809030+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:28.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: cluster 2024-04-26T16:12:26.656129+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: cluster 2024-04-26T16:12:26.656129+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: audit 2024-04-26T16:12:26.801965+0000 mon.a (mon.0) 393 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: audit 2024-04-26T16:12:26.801965+0000 mon.a (mon.0) 393 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: cluster 2024-04-26T16:12:26.807790+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: cluster 2024-04-26T16:12:26.807790+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: audit 2024-04-26T16:12:26.808437+0000 mon.a (mon.0) 395 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: audit 2024-04-26T16:12:26.808437+0000 mon.a (mon.0) 395 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: audit 2024-04-26T16:12:26.809030+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:28.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[21463]: audit 2024-04-26T16:12:26.809030+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: cluster 2024-04-26T16:12:26.656129+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: cluster 2024-04-26T16:12:26.656129+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: audit 2024-04-26T16:12:26.801965+0000 mon.a (mon.0) 393 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: audit 2024-04-26T16:12:26.801965+0000 mon.a (mon.0) 393 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: cluster 2024-04-26T16:12:26.807790+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: cluster 2024-04-26T16:12:26.807790+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: audit 2024-04-26T16:12:26.808437+0000 mon.a (mon.0) 395 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: audit 2024-04-26T16:12:26.808437+0000 mon.a (mon.0) 395 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: audit 2024-04-26T16:12:26.809030+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:28.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:27 smithi050 bash[29234]: audit 2024-04-26T16:12:26.809030+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:28.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.543+0000 7f7a7fd2b700 10 client.24167.objecter tick 2024-04-26T16:12:28.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.615+0000 7f7a6f7fe700 1 -- 172.21.15.50:0/4161889287 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7a7802e720 con 0x7f7a64071cc0 2024-04-26T16:12:28.615 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 2 on host 'smithi050' 2024-04-26T16:12:28.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 10 librados: watch_flush enter 2024-04-26T16:12:28.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 10 librados: watch_flush exit 2024-04-26T16:12:28.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 20 client.24167.objecter shutdown clearing up homeless session... 2024-04-26T16:12:28.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 10 client.24167.objecter successfully canceled tick 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a64071cc0 msgr2=0x7f7a64074180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a64071cc0 0x7f7a64074180 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7a74005fd0 tx=0x7f7a74005e50 comp rx=0 tx=0).stop 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 msgr2=0x7f7a78075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 0x7f7a78075d50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7a7000efc0 tx=0x7f7a7000c3a0 comp rx=0 tx=0).stop 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 shutdown_connections 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7a64071cc0 0x7f7a64074180 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a78103480 0x7f7a780771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7a78105020 0x7f7a78075d50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 --2- 172.21.15.50:0/4161889287 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7a78104680 0x7f7a78075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:28.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 >> 172.21.15.50:0/4161889287 conn(0x7f7a780fea20 msgr2=0x7f7a78100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:12:28.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 shutdown_connections 2024-04-26T16:12:28.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 -- 172.21.15.50:0/4161889287 wait complete. 2024-04-26T16:12:28.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:28.623+0000 7f7a8052c700 1 librados: shutdown 2024-04-26T16:12:29.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:27.803528+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:12:29.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:27.803528+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:12:29.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: cluster 2024-04-26T16:12:27.806528+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-26T16:12:29.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: cluster 2024-04-26T16:12:27.806528+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-26T16:12:29.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:27.807456+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:27.807456+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:27.813081+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:27.813081+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:28.561753+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:28.561753+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:28.570996+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:28.570996+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:28.594593+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:29.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:28 smithi190 bash[23868]: audit 2024-04-26T16:12:28.594593+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:29.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:27.803528+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:12:29.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:27.803528+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:12:29.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: cluster 2024-04-26T16:12:27.806528+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-26T16:12:29.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: cluster 2024-04-26T16:12:27.806528+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-26T16:12:29.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:27.807456+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:27.807456+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:27.813081+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:27.813081+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:28.561753+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:28.561753+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:28.570996+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:28.570996+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:28.594593+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[21463]: audit 2024-04-26T16:12:28.594593+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:27.803528+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:27.803528+0000 mon.a (mon.0) 397 : audit [INF] from='osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: cluster 2024-04-26T16:12:27.806528+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-26T16:12:29.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: cluster 2024-04-26T16:12:27.806528+0000 mon.a (mon.0) 398 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:27.807456+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:27.807456+0000 mon.a (mon.0) 399 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:27.813081+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:27.813081+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:28.561753+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:28.561753+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:28.570996+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:28.570996+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:28.594593+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:29.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:28 smithi050 bash[29234]: audit 2024-04-26T16:12:28.594593+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:12:29.390 DEBUG:teuthology.orchestra.run.smithi050:osd.2> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.2.service 2024-04-26T16:12:29.393 INFO:tasks.cephadm:Deploying osd.3 on smithi050 with /dev/vg_nvme/lv_1... 2024-04-26T16:12:29.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:27.026580+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:27.026580+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:27.026667+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:27.026667+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:28.656537+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:28.656537+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:28.814026+0000 mon.a (mon.0) 404 : cluster [INF] osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] boot 2024-04-26T16:12:30.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:28.814026+0000 mon.a (mon.0) 404 : cluster [INF] osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] boot 2024-04-26T16:12:30.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:28.814065+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-04-26T16:12:30.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: cluster 2024-04-26T16:12:28.814065+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-04-26T16:12:30.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: audit 2024-04-26T16:12:28.814556+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:30.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:29 smithi190 bash[23868]: audit 2024-04-26T16:12:28.814556+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:27.026580+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:27.026580+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:27.026667+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:27.026667+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:28.656537+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:28.656537+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:28.814026+0000 mon.a (mon.0) 404 : cluster [INF] osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] boot 2024-04-26T16:12:30.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:28.814026+0000 mon.a (mon.0) 404 : cluster [INF] osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] boot 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:28.814065+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: cluster 2024-04-26T16:12:28.814065+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: audit 2024-04-26T16:12:28.814556+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[21463]: audit 2024-04-26T16:12:28.814556+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:27.026580+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:27.026580+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:27.026667+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:27.026667+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:28.656537+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:28.656537+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:28.814026+0000 mon.a (mon.0) 404 : cluster [INF] osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] boot 2024-04-26T16:12:30.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:28.814026+0000 mon.a (mon.0) 404 : cluster [INF] osd.2 [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] boot 2024-04-26T16:12:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:28.814065+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-04-26T16:12:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: cluster 2024-04-26T16:12:28.814065+0000 mon.a (mon.0) 405 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2024-04-26T16:12:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: audit 2024-04-26T16:12:28.814556+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:29 smithi050 bash[29234]: audit 2024-04-26T16:12:28.814556+0000 mon.a (mon.0) 406 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:12:31.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[21463]: cluster 2024-04-26T16:12:29.836630+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-26T16:12:31.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[21463]: cluster 2024-04-26T16:12:29.836630+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-26T16:12:31.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[21463]: audit 2024-04-26T16:12:30.801174+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:31.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[21463]: audit 2024-04-26T16:12:30.801174+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:31.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[29234]: cluster 2024-04-26T16:12:29.836630+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-26T16:12:31.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[29234]: cluster 2024-04-26T16:12:29.836630+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-26T16:12:31.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[29234]: audit 2024-04-26T16:12:30.801174+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:31.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:30 smithi050 bash[29234]: audit 2024-04-26T16:12:30.801174+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:30 smithi190 bash[23868]: cluster 2024-04-26T16:12:29.836630+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-26T16:12:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:30 smithi190 bash[23868]: cluster 2024-04-26T16:12:29.836630+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2024-04-26T16:12:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:30 smithi190 bash[23868]: audit 2024-04-26T16:12:30.801174+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:30 smithi190 bash[23868]: audit 2024-04-26T16:12:30.801174+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:32.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: cluster 2024-04-26T16:12:30.656978+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:32.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: cluster 2024-04-26T16:12:30.656978+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: audit 2024-04-26T16:12:31.796320+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: audit 2024-04-26T16:12:31.796320+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: cluster 2024-04-26T16:12:31.803008+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: cluster 2024-04-26T16:12:31.803008+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: audit 2024-04-26T16:12:31.804823+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[21463]: audit 2024-04-26T16:12:31.804823+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: cluster 2024-04-26T16:12:30.656978+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: cluster 2024-04-26T16:12:30.656978+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: audit 2024-04-26T16:12:31.796320+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:32.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: audit 2024-04-26T16:12:31.796320+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: cluster 2024-04-26T16:12:31.803008+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-26T16:12:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: cluster 2024-04-26T16:12:31.803008+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-26T16:12:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: audit 2024-04-26T16:12:31.804823+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:31 smithi050 bash[29234]: audit 2024-04-26T16:12:31.804823+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: cluster 2024-04-26T16:12:30.656978+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: cluster 2024-04-26T16:12:30.656978+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: audit 2024-04-26T16:12:31.796320+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: audit 2024-04-26T16:12:31.796320+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: cluster 2024-04-26T16:12:31.803008+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: cluster 2024-04-26T16:12:31.803008+0000 mon.a (mon.0) 410 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: audit 2024-04-26T16:12:31.804823+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:32.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:31 smithi190 bash[23868]: audit 2024-04-26T16:12:31.804823+0000 mon.a (mon.0) 411 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:12:33.602 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: cluster 2024-04-26T16:12:32.657688+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: cluster 2024-04-26T16:12:32.657688+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:32.798191+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:32.798191+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: cluster 2024-04-26T16:12:32.804770+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: cluster 2024-04-26T16:12:32.804770+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: cluster 2024-04-26T16:12:32.841506+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: cluster 2024-04-26T16:12:32.841506+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:12:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.257845+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.257845+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.340863+0000 mon.a (mon.0) 416 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.340863+0000 mon.a (mon.0) 416 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.379068+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.379068+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.380597+0000 mon.a (mon.0) 417 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.380597+0000 mon.a (mon.0) 417 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.380779+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.380779+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.381245+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.381245+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: cluster 2024-04-26T16:12:32.657688+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: cluster 2024-04-26T16:12:32.657688+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:32.798191+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:32.798191+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: cluster 2024-04-26T16:12:32.804770+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: cluster 2024-04-26T16:12:32.804770+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: cluster 2024-04-26T16:12:32.841506+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: cluster 2024-04-26T16:12:32.841506+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.257845+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.257845+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.340863+0000 mon.a (mon.0) 416 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.340863+0000 mon.a (mon.0) 416 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.379068+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.379068+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.380597+0000 mon.a (mon.0) 417 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.380597+0000 mon.a (mon.0) 417 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.380779+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.380779+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.381245+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.381245+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.381397+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.381397+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.383964+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.383964+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.384133+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.384133+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.384308+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.384308+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.434415+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.434415+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.495202+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.495202+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.498378+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.498378+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.498855+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.498855+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.499244+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.499244+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.499534+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.499534+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.550186+0000 mon.c (mon.2) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[21463]: audit 2024-04-26T16:12:33.550186+0000 mon.c (mon.2) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.381397+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.381397+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.383964+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.383964+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.384133+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.384133+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.384308+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.384308+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.434415+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.434415+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.495202+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.495202+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.498378+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.498378+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.498855+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.498855+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.499244+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.499244+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.499534+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.499534+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:33.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.550186+0000 mon.c (mon.2) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:33.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:33 smithi050 bash[29234]: audit 2024-04-26T16:12:33.550186+0000 mon.c (mon.2) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: cluster 2024-04-26T16:12:32.657688+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: cluster 2024-04-26T16:12:32.657688+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v139: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:32.798191+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:32.798191+0000 mon.a (mon.0) 412 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:12:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: cluster 2024-04-26T16:12:32.804770+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: cluster 2024-04-26T16:12:32.804770+0000 mon.a (mon.0) 413 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: cluster 2024-04-26T16:12:32.841506+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: cluster 2024-04-26T16:12:32.841506+0000 mon.a (mon.0) 414 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.257845+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.257845+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.340863+0000 mon.a (mon.0) 416 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.340863+0000 mon.a (mon.0) 416 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.379068+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.379068+0000 mon.b (mon.1) 8 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:34.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.380597+0000 mon.a (mon.0) 417 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.380597+0000 mon.a (mon.0) 417 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.380779+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.380779+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.381245+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.381245+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.381397+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:34.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.381397+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.383964+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.383964+0000 mon.a (mon.0) 421 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.384133+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.384133+0000 mon.a (mon.0) 422 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.384308+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.384308+0000 mon.a (mon.0) 423 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.434415+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.434415+0000 mon.b (mon.1) 9 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.495202+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.495202+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.498378+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.498378+0000 mon.c (mon.2) 7 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-26T16:12:34.080 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.498855+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.498855+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.499244+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.499244+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.499534+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.499534+0000 mon.a (mon.0) 427 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.550186+0000 mon.c (mon.2) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:34.081 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:33 smithi190 bash[23868]: audit 2024-04-26T16:12:33.550186+0000 mon.c (mon.2) 8 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-26T16:12:35.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:34 smithi050 bash[29234]: cluster 2024-04-26T16:12:33.851122+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-26T16:12:35.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:34 smithi050 bash[29234]: cluster 2024-04-26T16:12:33.851122+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-26T16:12:35.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:34 smithi050 bash[21463]: cluster 2024-04-26T16:12:33.851122+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-26T16:12:35.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:34 smithi050 bash[21463]: cluster 2024-04-26T16:12:33.851122+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-26T16:12:35.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:34 smithi190 bash[23868]: cluster 2024-04-26T16:12:33.851122+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-26T16:12:35.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:34 smithi190 bash[23868]: cluster 2024-04-26T16:12:33.851122+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in 2024-04-26T16:12:36.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.658265+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:36.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.658265+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:36.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.846063+0000 mon.a (mon.0) 429 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:12:36.162 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.846063+0000 mon.a (mon.0) 429 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.846124+0000 mon.a (mon.0) 430 : cluster [INF] Cluster is now healthy 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.846124+0000 mon.a (mon.0) 430 : cluster [INF] Cluster is now healthy 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.875524+0000 mon.a (mon.0) 431 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[21463]: cluster 2024-04-26T16:12:34.875524+0000 mon.a (mon.0) 431 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.658265+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.658265+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.846063+0000 mon.a (mon.0) 429 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.846063+0000 mon.a (mon.0) 429 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:12:36.163 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.846124+0000 mon.a (mon.0) 430 : cluster [INF] Cluster is now healthy 2024-04-26T16:12:36.164 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.846124+0000 mon.a (mon.0) 430 : cluster [INF] Cluster is now healthy 2024-04-26T16:12:36.164 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.875524+0000 mon.a (mon.0) 431 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-04-26T16:12:36.164 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:35 smithi050 bash[29234]: cluster 2024-04-26T16:12:34.875524+0000 mon.a (mon.0) 431 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.658265+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.658265+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v142: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.846063+0000 mon.a (mon.0) 429 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.846063+0000 mon.a (mon.0) 429 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.846124+0000 mon.a (mon.0) 430 : cluster [INF] Cluster is now healthy 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.846124+0000 mon.a (mon.0) 430 : cluster [INF] Cluster is now healthy 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.875524+0000 mon.a (mon.0) 431 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-04-26T16:12:36.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:35 smithi190 bash[23868]: cluster 2024-04-26T16:12:34.875524+0000 mon.a (mon.0) 431 : cluster [DBG] mgrmap e15: y(active, since 4m), standbys: x 2024-04-26T16:12:38.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:37 smithi050 bash[21463]: cluster 2024-04-26T16:12:36.658914+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v143: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:38.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:37 smithi050 bash[21463]: cluster 2024-04-26T16:12:36.658914+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v143: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:38.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:37 smithi050 bash[29234]: cluster 2024-04-26T16:12:36.658914+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v143: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:38.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:37 smithi050 bash[29234]: cluster 2024-04-26T16:12:36.658914+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v143: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:38.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:37 smithi190 bash[23868]: cluster 2024-04-26T16:12:36.658914+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v143: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:38.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:37 smithi190 bash[23868]: cluster 2024-04-26T16:12:36.658914+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v143: 1 pgs: 1 creating+peering; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T16:12:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T16:12:38.995 INFO:teuthology.orchestra.run.smithi050.stderr: stderr: 10+0 records in 2024-04-26T16:12:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:10+0 records out 2024-04-26T16:12:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.010039 s, 1.0 GB/s 2024-04-26T16:12:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:--> Zapping successful for: 2024-04-26T16:12:39.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi050:vg_nvme/lv_1 2024-04-26T16:12:40.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:39 smithi050 bash[21463]: cluster 2024-04-26T16:12:38.659354+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:40.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:39 smithi050 bash[21463]: cluster 2024-04-26T16:12:38.659354+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:39 smithi050 bash[29234]: cluster 2024-04-26T16:12:38.659354+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:39 smithi050 bash[29234]: cluster 2024-04-26T16:12:38.659354+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:39 smithi190 bash[23868]: cluster 2024-04-26T16:12:38.659354+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:39 smithi190 bash[23868]: cluster 2024-04-26T16:12:38.659354+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:41.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: cephadm 2024-04-26T16:12:40.461328+0000 mgr.y (mgr.14152) 174 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:12:41.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: cephadm 2024-04-26T16:12:40.461328+0000 mgr.y (mgr.14152) 174 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:12:41.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.470893+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.470893+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.481490+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.481490+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.484289+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.484289+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.486234+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.486234+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.488055+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.488055+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: cephadm 2024-04-26T16:12:40.489544+0000 mgr.y (mgr.14152) 175 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 5415M 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: cephadm 2024-04-26T16:12:40.489544+0000 mgr.y (mgr.14152) 175 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 5415M 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.496291+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.496291+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.501456+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:41.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.501456+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:41.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.503353+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:41.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.503353+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:41.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.513420+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: audit 2024-04-26T16:12:40.513420+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: cluster 2024-04-26T16:12:40.659955+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:41.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:41 smithi190 bash[23868]: cluster 2024-04-26T16:12:40.659955+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: cephadm 2024-04-26T16:12:40.461328+0000 mgr.y (mgr.14152) 174 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: cephadm 2024-04-26T16:12:40.461328+0000 mgr.y (mgr.14152) 174 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.470893+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.470893+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.481490+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.481490+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.484289+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.484289+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.486234+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.486234+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.488055+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.488055+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: cephadm 2024-04-26T16:12:40.489544+0000 mgr.y (mgr.14152) 175 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 5415M 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: cephadm 2024-04-26T16:12:40.489544+0000 mgr.y (mgr.14152) 175 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 5415M 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.496291+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.496291+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.501456+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.501456+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.503353+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:41.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.503353+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: cephadm 2024-04-26T16:12:40.461328+0000 mgr.y (mgr.14152) 174 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: cephadm 2024-04-26T16:12:40.461328+0000 mgr.y (mgr.14152) 174 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.470893+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.470893+0000 mon.a (mon.0) 432 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.481490+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.481490+0000 mon.a (mon.0) 433 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.484289+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.484289+0000 mon.a (mon.0) 434 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.486234+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.486234+0000 mon.a (mon.0) 435 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.488055+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.488055+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: cephadm 2024-04-26T16:12:40.489544+0000 mgr.y (mgr.14152) 175 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 5415M 2024-04-26T16:12:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: cephadm 2024-04-26T16:12:40.489544+0000 mgr.y (mgr.14152) 175 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 5415M 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.496291+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.496291+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.501456+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.501456+0000 mon.a (mon.0) 438 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.503353+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.503353+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.513420+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: audit 2024-04-26T16:12:40.513420+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: cluster 2024-04-26T16:12:40.659955+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[21463]: cluster 2024-04-26T16:12:40.659955+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:41.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.513420+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: audit 2024-04-26T16:12:40.513420+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:12:41.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: cluster 2024-04-26T16:12:40.659955+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:41.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:41 smithi050 bash[29234]: cluster 2024-04-26T16:12:40.659955+0000 mgr.y (mgr.14152) 176 : cluster [DBG] pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:42 smithi190 bash[23868]: cluster 2024-04-26T16:12:40.812621+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:42 smithi190 bash[23868]: cluster 2024-04-26T16:12:40.812621+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:42 smithi190 bash[23868]: cluster 2024-04-26T16:12:40.812738+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:42 smithi190 bash[23868]: cluster 2024-04-26T16:12:40.812738+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:42 smithi190 bash[23868]: cluster 2024-04-26T16:12:41.523330+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-04-26T16:12:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:42 smithi190 bash[23868]: cluster 2024-04-26T16:12:41.523330+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-04-26T16:12:42.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[21463]: cluster 2024-04-26T16:12:40.812621+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:42.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[21463]: cluster 2024-04-26T16:12:40.812621+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:42.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[21463]: cluster 2024-04-26T16:12:40.812738+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:42.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[21463]: cluster 2024-04-26T16:12:40.812738+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:42.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[21463]: cluster 2024-04-26T16:12:41.523330+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-04-26T16:12:42.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[21463]: cluster 2024-04-26T16:12:41.523330+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-04-26T16:12:42.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[29234]: cluster 2024-04-26T16:12:40.812621+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:42.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[29234]: cluster 2024-04-26T16:12:40.812621+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:12:42.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[29234]: cluster 2024-04-26T16:12:40.812738+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:42.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[29234]: cluster 2024-04-26T16:12:40.812738+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:12:42.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[29234]: cluster 2024-04-26T16:12:41.523330+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-04-26T16:12:42.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:42 smithi050 bash[29234]: cluster 2024-04-26T16:12:41.523330+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e23: 3 total, 3 up, 3 in 2024-04-26T16:12:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:43 smithi190 bash[23868]: cluster 2024-04-26T16:12:42.660406+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:43 smithi190 bash[23868]: cluster 2024-04-26T16:12:42.660406+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:43.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:43 smithi050 bash[29234]: cluster 2024-04-26T16:12:42.660406+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:43.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:43 smithi050 bash[29234]: cluster 2024-04-26T16:12:42.660406+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:43.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:43 smithi050 bash[21463]: cluster 2024-04-26T16:12:42.660406+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:43.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:43 smithi050 bash[21463]: cluster 2024-04-26T16:12:42.660406+0000 mgr.y (mgr.14152) 177 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:44.671 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:12:46.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:45 smithi050 bash[21463]: cluster 2024-04-26T16:12:44.660917+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:46.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:45 smithi050 bash[21463]: cluster 2024-04-26T16:12:44.660917+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:46.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:45 smithi050 bash[29234]: cluster 2024-04-26T16:12:44.660917+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:46.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:45 smithi050 bash[29234]: cluster 2024-04-26T16:12:44.660917+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:46.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:45 smithi190 bash[23868]: cluster 2024-04-26T16:12:44.660917+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:46.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:45 smithi190 bash[23868]: cluster 2024-04-26T16:12:44.660917+0000 mgr.y (mgr.14152) 178 : cluster [DBG] pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:47.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- 172.21.15.50:0/2158693688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 msgr2=0x7f2bd4106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:12:47.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/2158693688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4106300 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4009990 tx=0x7f2bc4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:12:47.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- 172.21.15.50:0/2158693688 shutdown_connections 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/2158693688 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4109170 0x7f2bd410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/2158693688 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4106300 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/2158693688 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4106840 0x7f2bd4108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- 172.21.15.50:0/2158693688 >> 172.21.15.50:0/2158693688 conn(0x7f2bd40fd940 msgr2=0x7f2bd40ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- 172.21.15.50:0/2158693688 shutdown_connections 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- 172.21.15.50:0/2158693688 wait complete. 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 librados: starting msgr at 2024-04-26T16:12:47.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 librados: starting objecter 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 Processor -- start 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- start start 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 librados: setting wanted keys 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 librados: calling monclient init 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 0x7f2bd4199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4109170 0x7f2bd41a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:12:47.558 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd410e420 con 0x7f2bd4103f10 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd410e2a0 con 0x7f2bd4109170 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd8f83700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd410e5a0 con 0x7f2bd4106840 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 0x7f2bd4199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 0x7f2bd4199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53784/0 (socket says 172.21.15.50:53784) 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45370/0 (socket says 172.21.15.50:45370) 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 -- 172.21.15.50:0/3241224302 learned_addr learned my addr 172.21.15.50:0/3241224302 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:12:47.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd2d9d700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4109170 0x7f2bd41a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 -- 172.21.15.50:0/3241224302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4109170 msgr2=0x7f2bd41a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4109170 0x7f2bd41a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 -- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 msgr2=0x7f2bd4198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd1d9b700 1 -- 172.21.15.50:0/3241224302 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd41a0890 con 0x7f2bd4106840 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd2d9d700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4109170 0x7f2bd41a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.555+0000 7f2bd259c700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:12:47.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd1d9b700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 0x7f2bd4199180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800d950 tx=0x7f2bc800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 10 client.?.objecter ms_handle_connect 0x7f2bd4106840 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8016070 con 0x7f2bd4106840 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2bc8009da0 con 0x7f2bd4106840 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd41a0b80 con 0x7f2bd4106840 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc8010e60 con 0x7f2bd4106840 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd8f83700 10 client.24203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd41a10c0 con 0x7f2bd4106840 2024-04-26T16:12:47.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd8f83700 1 librados: init done 2024-04-26T16:12:47.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bd406a110 con 0x7f2bd4106840 2024-04-26T16:12:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f2bc8019070 con 0x7f2bd4106840 2024-04-26T16:12:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f2bbc071e90 0x7f2bbc074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:12:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 3017+0+0 (secure 0 0 0) 0x7f2bc808fbd0 con 0x7f2bd4106840 2024-04-26T16:12:47.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 10 client.24203.objecter ms_dispatch 0x7f2bd4001550 osd_map(23..23 src has 1..23) 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 3 client.24203.objecter handle_osd_map got epochs [23,23] > 0 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 3 client.24203.objecter handle_osd_map decoding full epoch 23 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd259c700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f2bbc071e90 0x7f2bbc074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 20 client.24203.objecter dump_active .. 0 homeless 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bd259c700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f2bbc071e90 0x7f2bbc074350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400b2e0 tx=0x7f2bc4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.559+0000 7f2bc37fe700 10 client.24203.objecter ms_handle_connect 0x7f2bbc071e90 2024-04-26T16:12:47.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.563+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f2bc805ceb0 con 0x7f2bd4106840 2024-04-26T16:12:47.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:47.795+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2bd4063860 con 0x7f2bbc071e90 2024-04-26T16:12:48.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:47 smithi190 bash[23868]: cluster 2024-04-26T16:12:46.661447+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:48.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:47 smithi190 bash[23868]: cluster 2024-04-26T16:12:46.661447+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:48.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:47 smithi050 bash[29234]: cluster 2024-04-26T16:12:46.661447+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:48.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:47 smithi050 bash[29234]: cluster 2024-04-26T16:12:46.661447+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:48.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:47 smithi050 bash[21463]: cluster 2024-04-26T16:12:46.661447+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:48.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:47 smithi050 bash[21463]: cluster 2024-04-26T16:12:46.661447+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:49.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.800275+0000 mgr.y (mgr.14152) 180 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.800275+0000 mgr.y (mgr.14152) 180 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.810354+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.810354+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.816168+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.816168+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.817312+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:49.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:48 smithi190 bash[23868]: audit 2024-04-26T16:12:47.817312+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:49.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.800275+0000 mgr.y (mgr.14152) 180 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.800275+0000 mgr.y (mgr.14152) 180 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.810354+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.810354+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.816168+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.816168+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.817312+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[21463]: audit 2024-04-26T16:12:47.817312+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.800275+0000 mgr.y (mgr.14152) 180 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.800275+0000 mgr.y (mgr.14152) 180 : audit [DBG] from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi050:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.810354+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:12:49.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.810354+0000 mon.a (mon.0) 442 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:12:49.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.816168+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:12:49.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.816168+0000 mon.a (mon.0) 443 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:12:49.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.817312+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:49.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:48 smithi050 bash[29234]: audit 2024-04-26T16:12:47.817312+0000 mon.a (mon.0) 444 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:12:50.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:49 smithi190 bash[23868]: cluster 2024-04-26T16:12:48.662117+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:50.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:49 smithi190 bash[23868]: cluster 2024-04-26T16:12:48.662117+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:50.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:49 smithi050 bash[21463]: cluster 2024-04-26T16:12:48.662117+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:50.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:49 smithi050 bash[21463]: cluster 2024-04-26T16:12:48.662117+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:50.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:49 smithi050 bash[29234]: cluster 2024-04-26T16:12:48.662117+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:50.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:49 smithi050 bash[29234]: cluster 2024-04-26T16:12:48.662117+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:51 smithi190 bash[23868]: cluster 2024-04-26T16:12:50.662696+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:51 smithi190 bash[23868]: cluster 2024-04-26T16:12:50.662696+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:51 smithi050 bash[21463]: cluster 2024-04-26T16:12:50.662696+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:51 smithi050 bash[21463]: cluster 2024-04-26T16:12:50.662696+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:51 smithi050 bash[29234]: cluster 2024-04-26T16:12:50.662696+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:51 smithi050 bash[29234]: cluster 2024-04-26T16:12:50.662696+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:52.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:52.563+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:12:54.071 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:53 smithi050 bash[29234]: cluster 2024-04-26T16:12:52.663277+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:54.071 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:53 smithi050 bash[29234]: cluster 2024-04-26T16:12:52.663277+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:54.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:53 smithi050 bash[21463]: cluster 2024-04-26T16:12:52.663277+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:54.072 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:53 smithi050 bash[21463]: cluster 2024-04-26T16:12:52.663277+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:53 smithi190 bash[23868]: cluster 2024-04-26T16:12:52.663277+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:53 smithi190 bash[23868]: cluster 2024-04-26T16:12:52.663277+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:55 smithi190 bash[23868]: cluster 2024-04-26T16:12:54.663796+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:55 smithi190 bash[23868]: cluster 2024-04-26T16:12:54.663796+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:56.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:55 smithi050 bash[21463]: cluster 2024-04-26T16:12:54.663796+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:56.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:55 smithi050 bash[21463]: cluster 2024-04-26T16:12:54.663796+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:56.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:55 smithi050 bash[29234]: cluster 2024-04-26T16:12:54.663796+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:56.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:55 smithi050 bash[29234]: cluster 2024-04-26T16:12:54.663796+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.291065+0000 mon.c (mon.2) 9 : audit [INF] from='client.? 172.21.15.50:0/3661566645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.291065+0000 mon.c (mon.2) 9 : audit [INF] from='client.? 172.21.15.50:0/3661566645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.291642+0000 mon.a (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.291642+0000 mon.a (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.294234+0000 mon.a (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]': finished 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.294234+0000 mon.a (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]': finished 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: cluster 2024-04-26T16:12:56.297585+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: cluster 2024-04-26T16:12:56.297585+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.297889+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:12:57.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:56 smithi190 bash[23868]: audit 2024-04-26T16:12:56.297889+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.291065+0000 mon.c (mon.2) 9 : audit [INF] from='client.? 172.21.15.50:0/3661566645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.291065+0000 mon.c (mon.2) 9 : audit [INF] from='client.? 172.21.15.50:0/3661566645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.291642+0000 mon.a (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.291642+0000 mon.a (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.294234+0000 mon.a (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]': finished 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.294234+0000 mon.a (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]': finished 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: cluster 2024-04-26T16:12:56.297585+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: cluster 2024-04-26T16:12:56.297585+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.297889+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:12:57.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[21463]: audit 2024-04-26T16:12:56.297889+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.291065+0000 mon.c (mon.2) 9 : audit [INF] from='client.? 172.21.15.50:0/3661566645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.291065+0000 mon.c (mon.2) 9 : audit [INF] from='client.? 172.21.15.50:0/3661566645' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.291642+0000 mon.a (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.291642+0000 mon.a (mon.0) 445 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]: dispatch 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.294234+0000 mon.a (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]': finished 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.294234+0000 mon.a (mon.0) 446 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03"}]': finished 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: cluster 2024-04-26T16:12:56.297585+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: cluster 2024-04-26T16:12:56.297585+0000 mon.a (mon.0) 447 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.297889+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:12:57.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:56 smithi050 bash[29234]: audit 2024-04-26T16:12:56.297889+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:12:57.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:12:57.567+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:12:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:57 smithi190 bash[23868]: cluster 2024-04-26T16:12:56.664298+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:57 smithi190 bash[23868]: cluster 2024-04-26T16:12:56.664298+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:57 smithi190 bash[23868]: audit 2024-04-26T16:12:57.056720+0000 mon.a (mon.0) 449 : audit [DBG] from='client.? 172.21.15.50:0/3069719916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:57 smithi190 bash[23868]: audit 2024-04-26T16:12:57.056720+0000 mon.a (mon.0) 449 : audit [DBG] from='client.? 172.21.15.50:0/3069719916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:58.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[29234]: cluster 2024-04-26T16:12:56.664298+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[29234]: cluster 2024-04-26T16:12:56.664298+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[29234]: audit 2024-04-26T16:12:57.056720+0000 mon.a (mon.0) 449 : audit [DBG] from='client.? 172.21.15.50:0/3069719916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[29234]: audit 2024-04-26T16:12:57.056720+0000 mon.a (mon.0) 449 : audit [DBG] from='client.? 172.21.15.50:0/3069719916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[21463]: cluster 2024-04-26T16:12:56.664298+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[21463]: cluster 2024-04-26T16:12:56.664298+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[21463]: audit 2024-04-26T16:12:57.056720+0000 mon.a (mon.0) 449 : audit [DBG] from='client.? 172.21.15.50:0/3069719916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:12:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:57 smithi050 bash[21463]: audit 2024-04-26T16:12:57.056720+0000 mon.a (mon.0) 449 : audit [DBG] from='client.? 172.21.15.50:0/3069719916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:59 smithi190 bash[23868]: cluster 2024-04-26T16:12:58.664860+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:12:59 smithi190 bash[23868]: cluster 2024-04-26T16:12:58.664860+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:00.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:59 smithi050 bash[21463]: cluster 2024-04-26T16:12:58.664860+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:00.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:12:59 smithi050 bash[21463]: cluster 2024-04-26T16:12:58.664860+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:59 smithi050 bash[29234]: cluster 2024-04-26T16:12:58.664860+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:12:59 smithi050 bash[29234]: cluster 2024-04-26T16:12:58.664860+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:01 smithi190 bash[23868]: cluster 2024-04-26T16:13:00.665240+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:01 smithi190 bash[23868]: cluster 2024-04-26T16:13:00.665240+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:01 smithi050 bash[21463]: cluster 2024-04-26T16:13:00.665240+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:01 smithi050 bash[21463]: cluster 2024-04-26T16:13:00.665240+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:01 smithi050 bash[29234]: cluster 2024-04-26T16:13:00.665240+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:01 smithi050 bash[29234]: cluster 2024-04-26T16:13:00.665240+0000 mgr.y (mgr.14152) 187 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:02.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:02.571+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:13:04.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:03 smithi050 bash[21463]: cluster 2024-04-26T16:13:02.665712+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:04.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:03 smithi050 bash[21463]: cluster 2024-04-26T16:13:02.665712+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:04.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:03 smithi050 bash[29234]: cluster 2024-04-26T16:13:02.665712+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:04.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:03 smithi050 bash[29234]: cluster 2024-04-26T16:13:02.665712+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:04.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:03 smithi190 bash[23868]: cluster 2024-04-26T16:13:02.665712+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:04.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:03 smithi190 bash[23868]: cluster 2024-04-26T16:13:02.665712+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:06.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:05 smithi050 bash[21463]: cluster 2024-04-26T16:13:04.666243+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:06.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:05 smithi050 bash[21463]: cluster 2024-04-26T16:13:04.666243+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:06.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:05 smithi050 bash[29234]: cluster 2024-04-26T16:13:04.666243+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:06.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:05 smithi050 bash[29234]: cluster 2024-04-26T16:13:04.666243+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:06.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:05 smithi190 bash[23868]: cluster 2024-04-26T16:13:04.666243+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:06.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:05 smithi190 bash[23868]: cluster 2024-04-26T16:13:04.666243+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:07.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:07.575+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:13:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:07 smithi050 bash[29234]: cluster 2024-04-26T16:13:06.666812+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:08.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:07 smithi050 bash[29234]: cluster 2024-04-26T16:13:06.666812+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:08.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:07 smithi050 bash[21463]: cluster 2024-04-26T16:13:06.666812+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:08.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:07 smithi050 bash[21463]: cluster 2024-04-26T16:13:06.666812+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:08.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:07 smithi190 bash[23868]: cluster 2024-04-26T16:13:06.666812+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:08.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:07 smithi190 bash[23868]: cluster 2024-04-26T16:13:06.666812+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[21463]: cluster 2024-04-26T16:13:08.667392+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[21463]: cluster 2024-04-26T16:13:08.667392+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[21463]: audit 2024-04-26T16:13:09.203044+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[21463]: audit 2024-04-26T16:13:09.203044+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[21463]: audit 2024-04-26T16:13:09.204794+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[21463]: audit 2024-04-26T16:13:09.204794+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:10.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[29234]: cluster 2024-04-26T16:13:08.667392+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[29234]: cluster 2024-04-26T16:13:08.667392+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[29234]: audit 2024-04-26T16:13:09.203044+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T16:13:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[29234]: audit 2024-04-26T16:13:09.203044+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T16:13:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[29234]: audit 2024-04-26T16:13:09.204794+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:10.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:09 smithi050 bash[29234]: audit 2024-04-26T16:13:09.204794+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:10.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:09 smithi190 bash[23868]: cluster 2024-04-26T16:13:08.667392+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:09 smithi190 bash[23868]: cluster 2024-04-26T16:13:08.667392+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:10.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:09 smithi190 bash[23868]: audit 2024-04-26T16:13:09.203044+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T16:13:10.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:09 smithi190 bash[23868]: audit 2024-04-26T16:13:09.203044+0000 mon.a (mon.0) 450 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-26T16:13:10.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:09 smithi190 bash[23868]: audit 2024-04-26T16:13:09.204794+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:10.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:09 smithi190 bash[23868]: audit 2024-04-26T16:13:09.204794+0000 mon.a (mon.0) 451 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cephadm 2024-04-26T16:13:09.206266+0000 mgr.y (mgr.14152) 192 : cephadm [INF] Deploying daemon osd.3 on smithi050 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cephadm 2024-04-26T16:13:09.206266+0000 mgr.y (mgr.14152) 192 : cephadm [INF] Deploying daemon osd.3 on smithi050 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cluster 2024-04-26T16:13:09.845574+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cluster 2024-04-26T16:13:09.845574+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cluster 2024-04-26T16:13:09.845701+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cluster 2024-04-26T16:13:09.845701+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cluster 2024-04-26T16:13:09.904353+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: cluster 2024-04-26T16:13:09.904353+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: audit 2024-04-26T16:13:09.904735+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[29234]: audit 2024-04-26T16:13:09.904735+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:11.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cephadm 2024-04-26T16:13:09.206266+0000 mgr.y (mgr.14152) 192 : cephadm [INF] Deploying daemon osd.3 on smithi050 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cephadm 2024-04-26T16:13:09.206266+0000 mgr.y (mgr.14152) 192 : cephadm [INF] Deploying daemon osd.3 on smithi050 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cluster 2024-04-26T16:13:09.845574+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cluster 2024-04-26T16:13:09.845574+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cluster 2024-04-26T16:13:09.845701+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cluster 2024-04-26T16:13:09.845701+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cluster 2024-04-26T16:13:09.904353+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: cluster 2024-04-26T16:13:09.904353+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: audit 2024-04-26T16:13:09.904735+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:11.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:10 smithi050 bash[21463]: audit 2024-04-26T16:13:09.904735+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cephadm 2024-04-26T16:13:09.206266+0000 mgr.y (mgr.14152) 192 : cephadm [INF] Deploying daemon osd.3 on smithi050 2024-04-26T16:13:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cephadm 2024-04-26T16:13:09.206266+0000 mgr.y (mgr.14152) 192 : cephadm [INF] Deploying daemon osd.3 on smithi050 2024-04-26T16:13:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cluster 2024-04-26T16:13:09.845574+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cluster 2024-04-26T16:13:09.845574+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cluster 2024-04-26T16:13:09.845701+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cluster 2024-04-26T16:13:09.845701+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:11.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cluster 2024-04-26T16:13:09.904353+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-26T16:13:11.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: cluster 2024-04-26T16:13:09.904353+0000 mon.a (mon.0) 452 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2024-04-26T16:13:11.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: audit 2024-04-26T16:13:09.904735+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:11.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:10 smithi190 bash[23868]: audit 2024-04-26T16:13:09.904735+0000 mon.a (mon.0) 453 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:11 smithi050 bash[21463]: cluster 2024-04-26T16:13:10.667960+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:11 smithi050 bash[21463]: cluster 2024-04-26T16:13:10.667960+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:12.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:11 smithi050 bash[29234]: cluster 2024-04-26T16:13:10.667960+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:12.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:11 smithi050 bash[29234]: cluster 2024-04-26T16:13:10.667960+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:12.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:11 smithi190 bash[23868]: cluster 2024-04-26T16:13:10.667960+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:12.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:11 smithi190 bash[23868]: cluster 2024-04-26T16:13:10.667960+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:12.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:12.579+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:13:14.021 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:13:13 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.022 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:13 smithi050 bash[21463]: cluster 2024-04-26T16:13:12.668549+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:14.022 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:13 smithi050 bash[21463]: cluster 2024-04-26T16:13:12.668549+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:14.022 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:13 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:13 smithi050 bash[29234]: cluster 2024-04-26T16:13:12.668549+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:14.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:13 smithi050 bash[29234]: cluster 2024-04-26T16:13:12.668549+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:14.022 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:13 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.023 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:13:13 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.023 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:13:13 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.023 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:13:13 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.301 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:13:14 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.302 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:14 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.302 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:13:14 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.302 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:13:14 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.303 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:13:14 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.303 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:14 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:13:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:13 smithi190 bash[23868]: cluster 2024-04-26T16:13:12.668549+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:14.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:13 smithi190 bash[23868]: cluster 2024-04-26T16:13:12.668549+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: audit 2024-04-26T16:13:14.313198+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: audit 2024-04-26T16:13:14.313198+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: audit 2024-04-26T16:13:14.317795+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: audit 2024-04-26T16:13:14.317795+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: audit 2024-04-26T16:13:14.336574+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:15.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: audit 2024-04-26T16:13:14.336574+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:15.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: cluster 2024-04-26T16:13:14.668885+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:15.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:15 smithi190 bash[23868]: cluster 2024-04-26T16:13:14.668885+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: audit 2024-04-26T16:13:14.313198+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: audit 2024-04-26T16:13:14.313198+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: audit 2024-04-26T16:13:14.317795+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: audit 2024-04-26T16:13:14.317795+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: audit 2024-04-26T16:13:14.336574+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: audit 2024-04-26T16:13:14.336574+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: cluster 2024-04-26T16:13:14.668885+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:15.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[21463]: cluster 2024-04-26T16:13:14.668885+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: audit 2024-04-26T16:13:14.313198+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: audit 2024-04-26T16:13:14.313198+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: audit 2024-04-26T16:13:14.317795+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: audit 2024-04-26T16:13:14.317795+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: audit 2024-04-26T16:13:14.336574+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: audit 2024-04-26T16:13:14.336574+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: cluster 2024-04-26T16:13:14.668885+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:15 smithi050 bash[29234]: cluster 2024-04-26T16:13:14.668885+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:17.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:17.583+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:13:18.073 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:17 smithi050 bash[29234]: cluster 2024-04-26T16:13:16.669207+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:18.074 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:17 smithi050 bash[29234]: cluster 2024-04-26T16:13:16.669207+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:18.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:17 smithi050 bash[21463]: cluster 2024-04-26T16:13:16.669207+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:18.074 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:17 smithi050 bash[21463]: cluster 2024-04-26T16:13:16.669207+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:17 smithi190 bash[23868]: cluster 2024-04-26T16:13:16.669207+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:17 smithi190 bash[23868]: cluster 2024-04-26T16:13:16.669207+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: cluster 2024-04-26T16:13:18.669728+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: cluster 2024-04-26T16:13:18.669728+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.620574+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.620574+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.625558+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.625558+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.626475+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.626475+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.627193+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.627193+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.631373+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:19 smithi190 bash[23868]: audit 2024-04-26T16:13:19.631373+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: cluster 2024-04-26T16:13:18.669728+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: cluster 2024-04-26T16:13:18.669728+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.620574+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.620574+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.625558+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.625558+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.626475+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.626475+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.627193+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.627193+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.631373+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[29234]: audit 2024-04-26T16:13:19.631373+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: cluster 2024-04-26T16:13:18.669728+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: cluster 2024-04-26T16:13:18.669728+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.620574+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.620574+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.625558+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.625558+0000 mon.a (mon.0) 458 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.626475+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.626475+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.627193+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.627193+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.631373+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:20.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:19 smithi050 bash[21463]: audit 2024-04-26T16:13:19.631373+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:21 smithi190 bash[23868]: cluster 2024-04-26T16:13:20.670244+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:21 smithi190 bash[23868]: cluster 2024-04-26T16:13:20.670244+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:21 smithi190 bash[23868]: audit 2024-04-26T16:13:21.087887+0000 mon.a (mon.0) 462 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T16:13:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:21 smithi190 bash[23868]: audit 2024-04-26T16:13:21.087887+0000 mon.a (mon.0) 462 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T16:13:22.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[21463]: cluster 2024-04-26T16:13:20.670244+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[21463]: cluster 2024-04-26T16:13:20.670244+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[21463]: audit 2024-04-26T16:13:21.087887+0000 mon.a (mon.0) 462 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[21463]: audit 2024-04-26T16:13:21.087887+0000 mon.a (mon.0) 462 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[29234]: cluster 2024-04-26T16:13:20.670244+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[29234]: cluster 2024-04-26T16:13:20.670244+0000 mgr.y (mgr.14152) 198 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[29234]: audit 2024-04-26T16:13:21.087887+0000 mon.a (mon.0) 462 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T16:13:22.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:21 smithi050 bash[29234]: audit 2024-04-26T16:13:21.087887+0000 mon.a (mon.0) 462 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-26T16:13:22.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:22.587+0000 7f2bd3fff700 10 client.24203.objecter tick 2024-04-26T16:13:23.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:21.805683+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T16:13:23.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:21.805683+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T16:13:23.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: cluster 2024-04-26T16:13:21.811754+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-04-26T16:13:23.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: cluster 2024-04-26T16:13:21.811754+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-04-26T16:13:23.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:21.812735+0000 mon.a (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:21.812735+0000 mon.a (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:21.813344+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:21.813344+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:22.807633+0000 mon.a (mon.0) 467 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:22.807633+0000 mon.a (mon.0) 467 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.810828+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.810828+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:22.811662+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:22 smithi190 bash[23868]: audit 2024-04-26T16:13:22.811662+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:21.805683+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T16:13:23.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:21.805683+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T16:13:23.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: cluster 2024-04-26T16:13:21.811754+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-04-26T16:13:23.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: cluster 2024-04-26T16:13:21.811754+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:21.812735+0000 mon.a (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:21.812735+0000 mon.a (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:21.813344+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:21.813344+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:22.807633+0000 mon.a (mon.0) 467 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:22.807633+0000 mon.a (mon.0) 467 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.810828+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.810828+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:22.811662+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[29234]: audit 2024-04-26T16:13:22.811662+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.118 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:21.805683+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:21.805683+0000 mon.a (mon.0) 463 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: cluster 2024-04-26T16:13:21.811754+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: cluster 2024-04-26T16:13:21.811754+0000 mon.a (mon.0) 464 : cluster [DBG] osdmap e26: 4 total, 3 up, 4 in 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:21.812735+0000 mon.a (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:21.812735+0000 mon.a (mon.0) 465 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]: dispatch 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:21.813344+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:21.813344+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:22.807633+0000 mon.a (mon.0) 467 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:22.807633+0000 mon.a (mon.0) 467 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi050", "root=default"]}]': finished 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.810828+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.810828+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e27: 4 total, 3 up, 4 in 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:22.811662+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:22 smithi050 bash[21463]: audit 2024-04-26T16:13:22.811662+0000 mon.a (mon.0) 469 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:23.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.711+0000 7f2bc37fe700 1 -- 172.21.15.50:0/3241224302 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2bd4063860 con 0x7f2bbc071e90 2024-04-26T16:13:23.709 INFO:teuthology.orchestra.run.smithi050.stdout:Created osd(s) 3 on host 'smithi050' 2024-04-26T16:13:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 10 librados: watch_flush enter 2024-04-26T16:13:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 10 librados: watch_flush exit 2024-04-26T16:13:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 20 client.24203.objecter shutdown clearing up homeless session... 2024-04-26T16:13:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 10 client.24203.objecter successfully canceled tick 2024-04-26T16:13:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f2bbc071e90 msgr2=0x7f2bbc074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:13:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f2bbc071e90 0x7f2bbc074350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400b2e0 tx=0x7f2bc4005fb0 comp rx=0 tx=0).stop 2024-04-26T16:13:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 msgr2=0x7f2bd4199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:13:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 0x7f2bd4199180 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2bc800d950 tx=0x7f2bc800dc60 comp rx=0 tx=0).stop 2024-04-26T16:13:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 shutdown_connections 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f2bbc071e90 0x7f2bbc074350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2bd4109170 0x7f2bd41a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2bd4103f10 0x7f2bd4198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 --2- 172.21.15.50:0/3241224302 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2bd4106840 0x7f2bd4199180 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 >> 172.21.15.50:0/3241224302 conn(0x7f2bd40fd940 msgr2=0x7f2bd4104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 shutdown_connections 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 -- 172.21.15.50:0/3241224302 wait complete. 2024-04-26T16:13:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:13:23.715+0000 7f2bd8f83700 1 librados: shutdown 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.670588+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.670588+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:22.817542+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:22.817542+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.154923+0000 mon.a (mon.0) 471 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.154923+0000 mon.a (mon.0) 471 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' 2024-04-26T16:13:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.678298+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.678298+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.689309+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.689309+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.705662+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.705662+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.815668+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:23 smithi190 bash[23868]: audit 2024-04-26T16:13:23.815668+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.670588+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.670588+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:22.817542+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:22.817542+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.154923+0000 mon.a (mon.0) 471 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.154923+0000 mon.a (mon.0) 471 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.678298+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.678298+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.689309+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.689309+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.705662+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.705662+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.815668+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[21463]: audit 2024-04-26T16:13:23.815668+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.670588+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:24.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.670588+0000 mgr.y (mgr.14152) 199 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:22.817542+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:22.817542+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.154923+0000 mon.a (mon.0) 471 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.154923+0000 mon.a (mon.0) 471 : audit [INF] from='osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077]' entity='osd.3' 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.678298+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.678298+0000 mon.a (mon.0) 472 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.689309+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.689309+0000 mon.a (mon.0) 473 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.705662+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.705662+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.815668+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:23 smithi050 bash[29234]: audit 2024-04-26T16:13:23.815668+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:24.453 DEBUG:teuthology.orchestra.run.smithi050:osd.3> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.3.service 2024-04-26T16:13:24.458 INFO:tasks.cephadm:Deploying osd.4 on smithi190 with /dev/vg_nvme/lv_4... 2024-04-26T16:13:24.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.066897+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.066897+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.067507+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:22.067507+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:24.157740+0000 mon.a (mon.0) 476 : cluster [INF] osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] boot 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:24.157740+0000 mon.a (mon.0) 476 : cluster [INF] osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] boot 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:24.157785+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: cluster 2024-04-26T16:13:24.157785+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: audit 2024-04-26T16:13:24.158027+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:25.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:24 smithi190 bash[23868]: audit 2024-04-26T16:13:24.158027+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.066897+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.066897+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.067507+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:22.067507+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:24.157740+0000 mon.a (mon.0) 476 : cluster [INF] osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] boot 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:24.157740+0000 mon.a (mon.0) 476 : cluster [INF] osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] boot 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:24.157785+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: cluster 2024-04-26T16:13:24.157785+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: audit 2024-04-26T16:13:24.158027+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:25.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[21463]: audit 2024-04-26T16:13:24.158027+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.066897+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.066897+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.067507+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:22.067507+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:24.157740+0000 mon.a (mon.0) 476 : cluster [INF] osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] boot 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:24.157740+0000 mon.a (mon.0) 476 : cluster [INF] osd.3 [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] boot 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:24.157785+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: cluster 2024-04-26T16:13:24.157785+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e28: 4 total, 4 up, 4 in 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: audit 2024-04-26T16:13:24.158027+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:25.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:24 smithi050 bash[29234]: audit 2024-04-26T16:13:24.158027+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:13:26.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:25 smithi050 bash[21463]: cluster 2024-04-26T16:13:24.671062+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:26.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:25 smithi050 bash[21463]: cluster 2024-04-26T16:13:24.671062+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:26.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:25 smithi050 bash[29234]: cluster 2024-04-26T16:13:24.671062+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:26.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:25 smithi050 bash[29234]: cluster 2024-04-26T16:13:24.671062+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:25 smithi190 bash[23868]: cluster 2024-04-26T16:13:24.671062+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:25 smithi190 bash[23868]: cluster 2024-04-26T16:13:24.671062+0000 mgr.y (mgr.14152) 200 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-04-26T16:13:27.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:26 smithi050 bash[29234]: cluster 2024-04-26T16:13:25.837535+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-04-26T16:13:27.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:26 smithi050 bash[29234]: cluster 2024-04-26T16:13:25.837535+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-04-26T16:13:27.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:26 smithi050 bash[21463]: cluster 2024-04-26T16:13:25.837535+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-04-26T16:13:27.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:26 smithi050 bash[21463]: cluster 2024-04-26T16:13:25.837535+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-04-26T16:13:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:26 smithi190 bash[23868]: cluster 2024-04-26T16:13:25.837535+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-04-26T16:13:27.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:26 smithi190 bash[23868]: cluster 2024-04-26T16:13:25.837535+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e29: 4 total, 4 up, 4 in 2024-04-26T16:13:28.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:27 smithi050 bash[21463]: cluster 2024-04-26T16:13:26.671637+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:28.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:27 smithi050 bash[21463]: cluster 2024-04-26T16:13:26.671637+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:28.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:27 smithi050 bash[29234]: cluster 2024-04-26T16:13:26.671637+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:28.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:27 smithi050 bash[29234]: cluster 2024-04-26T16:13:26.671637+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:28.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:27 smithi190 bash[23868]: cluster 2024-04-26T16:13:26.671637+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:28.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:27 smithi190 bash[23868]: cluster 2024-04-26T16:13:26.671637+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:29.170 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:13:30.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:29 smithi190 bash[23868]: cluster 2024-04-26T16:13:28.672217+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:30.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:29 smithi190 bash[23868]: cluster 2024-04-26T16:13:28.672217+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:30.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:29 smithi050 bash[21463]: cluster 2024-04-26T16:13:28.672217+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:30.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:29 smithi050 bash[21463]: cluster 2024-04-26T16:13:28.672217+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:30.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:29 smithi050 bash[29234]: cluster 2024-04-26T16:13:28.672217+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:30.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:29 smithi050 bash[29234]: cluster 2024-04-26T16:13:28.672217+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-26T16:13:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-26T16:13:31.949 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-26T16:13:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-26T16:13:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138236 s, 759 MB/s 2024-04-26T16:13:31.949 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-26T16:13:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:31 smithi050 bash[21463]: cluster 2024-04-26T16:13:30.672671+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:31 smithi050 bash[21463]: cluster 2024-04-26T16:13:30.672671+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:32.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:31 smithi050 bash[29234]: cluster 2024-04-26T16:13:30.672671+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:32.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:31 smithi050 bash[29234]: cluster 2024-04-26T16:13:30.672671+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:32.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:31 smithi190 bash[23868]: cluster 2024-04-26T16:13:30.672671+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:32.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:31 smithi190 bash[23868]: cluster 2024-04-26T16:13:30.672671+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:32.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-04-26T16:13:34.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: cluster 2024-04-26T16:13:32.673115+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:34.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: cluster 2024-04-26T16:13:32.673115+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.607478+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.607478+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.617425+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.617425+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.620225+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.620225+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.622076+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.622076+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.623917+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.623917+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.625690+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.625690+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.633838+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.633838+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.640338+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.640338+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.642843+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: cluster 2024-04-26T16:13:32.673115+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:34.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: cluster 2024-04-26T16:13:32.673115+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.607478+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.607478+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.617425+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.617425+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.620225+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.620225+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.622076+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.622076+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.623917+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.623917+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.625690+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.625690+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.633838+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.633838+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.640338+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.640338+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.642843+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.642843+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:34.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.652987+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[21463]: audit 2024-04-26T16:13:33.652987+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.642843+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:34.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.652987+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:33 smithi050 bash[29234]: audit 2024-04-26T16:13:33.652987+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: cluster 2024-04-26T16:13:32.673115+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:34.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: cluster 2024-04-26T16:13:32.673115+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:34.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.607478+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.607478+0000 mon.a (mon.0) 480 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.617425+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.617425+0000 mon.a (mon.0) 481 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.620225+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.620225+0000 mon.a (mon.0) 482 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.622076+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.622076+0000 mon.a (mon.0) 483 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.623917+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.623917+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.625690+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.625690+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.633838+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.633838+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.640338+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:34.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.640338+0000 mon.a (mon.0) 487 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:34.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.642843+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:34.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.642843+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:13:34.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.652987+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:34.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:33 smithi190 bash[23868]: audit 2024-04-26T16:13:33.652987+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:13:35.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[21463]: cephadm 2024-04-26T16:13:33.598408+0000 mgr.y (mgr.14152) 205 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:13:35.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[21463]: cephadm 2024-04-26T16:13:33.598408+0000 mgr.y (mgr.14152) 205 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:13:35.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[21463]: cephadm 2024-04-26T16:13:33.627142+0000 mgr.y (mgr.14152) 206 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 4061M 2024-04-26T16:13:35.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[21463]: cephadm 2024-04-26T16:13:33.627142+0000 mgr.y (mgr.14152) 206 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 4061M 2024-04-26T16:13:35.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[29234]: cephadm 2024-04-26T16:13:33.598408+0000 mgr.y (mgr.14152) 205 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:13:35.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[29234]: cephadm 2024-04-26T16:13:33.598408+0000 mgr.y (mgr.14152) 205 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:13:35.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[29234]: cephadm 2024-04-26T16:13:33.627142+0000 mgr.y (mgr.14152) 206 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 4061M 2024-04-26T16:13:35.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:34 smithi050 bash[29234]: cephadm 2024-04-26T16:13:33.627142+0000 mgr.y (mgr.14152) 206 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 4061M 2024-04-26T16:13:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:34 smithi190 bash[23868]: cephadm 2024-04-26T16:13:33.598408+0000 mgr.y (mgr.14152) 205 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:13:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:34 smithi190 bash[23868]: cephadm 2024-04-26T16:13:33.598408+0000 mgr.y (mgr.14152) 205 : cephadm [INF] Detected new or changed devices on smithi050 2024-04-26T16:13:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:34 smithi190 bash[23868]: cephadm 2024-04-26T16:13:33.627142+0000 mgr.y (mgr.14152) 206 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 4061M 2024-04-26T16:13:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:34 smithi190 bash[23868]: cephadm 2024-04-26T16:13:33.627142+0000 mgr.y (mgr.14152) 206 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 4061M 2024-04-26T16:13:36.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:35 smithi050 bash[21463]: cluster 2024-04-26T16:13:34.673757+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:36.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:35 smithi050 bash[21463]: cluster 2024-04-26T16:13:34.673757+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:36.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:35 smithi050 bash[29234]: cluster 2024-04-26T16:13:34.673757+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:36.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:35 smithi050 bash[29234]: cluster 2024-04-26T16:13:34.673757+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:36.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:35 smithi190 bash[23868]: cluster 2024-04-26T16:13:34.673757+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:36.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:35 smithi190 bash[23868]: cluster 2024-04-26T16:13:34.673757+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:37.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[29234]: cluster 2024-04-26T16:13:35.639140+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:13:37.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[29234]: cluster 2024-04-26T16:13:35.639140+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:13:37.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[29234]: cluster 2024-04-26T16:13:35.648645+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:13:37.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[29234]: cluster 2024-04-26T16:13:35.648645+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:13:37.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[21463]: cluster 2024-04-26T16:13:35.639140+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:13:37.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[21463]: cluster 2024-04-26T16:13:35.639140+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:13:37.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[21463]: cluster 2024-04-26T16:13:35.648645+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:13:37.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:36 smithi050 bash[21463]: cluster 2024-04-26T16:13:35.648645+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:13:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:13:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:36 smithi190 bash[23868]: cluster 2024-04-26T16:13:35.639140+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:13:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:36 smithi190 bash[23868]: cluster 2024-04-26T16:13:35.639140+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:13:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:36 smithi190 bash[23868]: cluster 2024-04-26T16:13:35.648645+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:13:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:36 smithi190 bash[23868]: cluster 2024-04-26T16:13:35.648645+0000 osd.1 (osd.1) 6 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:13:38.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:37 smithi050 bash[21463]: cluster 2024-04-26T16:13:36.674303+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:38.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:37 smithi050 bash[21463]: cluster 2024-04-26T16:13:36.674303+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:38.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:37 smithi050 bash[29234]: cluster 2024-04-26T16:13:36.674303+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:38.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:37 smithi050 bash[29234]: cluster 2024-04-26T16:13:36.674303+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:37 smithi190 bash[23868]: cluster 2024-04-26T16:13:36.674303+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:37 smithi190 bash[23868]: cluster 2024-04-26T16:13:36.674303+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:39.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 -- 172.21.15.190:0/111521638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00103f10 msgr2=0x7f7f00106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:13:39.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 --2- 172.21.15.190:0/111521638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00103f10 0x7f7f00106300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8009990 tx=0x7f7ee8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:13:39.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 -- 172.21.15.190:0/111521638 shutdown_connections 2024-04-26T16:13:39.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 --2- 172.21.15.190:0/111521638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00103f10 0x7f7f00106300 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:39.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 --2- 172.21.15.190:0/111521638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00109170 0x7f7f0010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:39.525 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 --2- 172.21.15.190:0/111521638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f00106840 0x7f7f00108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 -- 172.21.15.190:0/111521638 >> 172.21.15.190:0/111521638 conn(0x7f7f000fd940 msgr2=0x7f7f000ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 -- 172.21.15.190:0/111521638 shutdown_connections 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 -- 172.21.15.190:0/111521638 wait complete. 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 librados: starting msgr at 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 librados: starting objecter 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 Processor -- start 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 -- start start 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 librados: setting wanted keys 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.524+0000 7f7f05b98700 1 librados: calling monclient init 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00103f10 0x7f7f00198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f00106840 0x7f7f001991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 0x7f7f001a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:13:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0010e460 con 0x7f7f00103f10 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0010e2e0 con 0x7f7f00109170 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0010e5e0 con 0x7f7f00106840 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 0x7f7f001a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 0x7f7f001a01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:49594/0 (socket says 172.21.15.190:49594) 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 -- 172.21.15.190:0/1506453811 learned_addr learned my addr 172.21.15.190:0/1506453811 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7eff7fe700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00103f10 0x7f7f00198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efeffd700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f00106840 0x7f7f001991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 -- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f00106840 msgr2=0x7f7f001991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f00106840 0x7f7f001991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 -- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00103f10 msgr2=0x7f7f00198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00103f10 0x7f7f00198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 -- 172.21.15.190:0/1506453811 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f001a08c0 con 0x7f7f00109170 2024-04-26T16:13:39.527 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7eff7fe700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00103f10 0x7f7f00198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efffff700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 0x7f7f001a01f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800ada0 tx=0x7f7ef800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 10 client.?.objecter ms_handle_connect 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8017020 con 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7ef8007e30 con 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f001a0bb0 con 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef8005260 con 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 10 client.24202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001a10f0 con 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 librados: init done 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0006a110 con 0x7f7f00109170 2024-04-26T16:13:39.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f7ef800c980 con 0x7f7f00109170 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7eec071e90 0x7f7eec074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3449+0+0 (secure 0 0 0) 0x7f7ef808f8f0 con 0x7f7f00109170 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 10 client.24202.objecter ms_dispatch 0x7f7f00001550 osd_map(29..29 src has 1..29) 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 3 client.24202.objecter handle_osd_map got epochs [29,29] > 0 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 3 client.24202.objecter handle_osd_map decoding full epoch 29 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7efcff9700 20 client.24202.objecter dump_active .. 0 homeless 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.528+0000 7f7eff7fe700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7eec071e90 0x7f7eec074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.532+0000 7f7efcff9700 10 client.24202.objecter ms_handle_connect 0x7f7eec071e90 2024-04-26T16:13:39.530 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.532+0000 7f7eff7fe700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7eec071e90 0x7f7eec074350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ee80098e0 tx=0x7f7ee8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:13:39.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.536+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7ef8021020 con 0x7f7f00109170 2024-04-26T16:13:39.765 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:39.764+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f7f00063860 con 0x7f7eec071e90 2024-04-26T16:13:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: cluster 2024-04-26T16:13:38.674897+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: cluster 2024-04-26T16:13:38.674897+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: audit 2024-04-26T16:13:39.783446+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: audit 2024-04-26T16:13:39.783446+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: audit 2024-04-26T16:13:39.790506+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: audit 2024-04-26T16:13:39.790506+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: audit 2024-04-26T16:13:39.792184+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:40.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:39 smithi190 bash[23868]: audit 2024-04-26T16:13:39.792184+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: cluster 2024-04-26T16:13:38.674897+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: cluster 2024-04-26T16:13:38.674897+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: audit 2024-04-26T16:13:39.783446+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: audit 2024-04-26T16:13:39.783446+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: audit 2024-04-26T16:13:39.790506+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: audit 2024-04-26T16:13:39.790506+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: audit 2024-04-26T16:13:39.792184+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:40.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[29234]: audit 2024-04-26T16:13:39.792184+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: cluster 2024-04-26T16:13:38.674897+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: cluster 2024-04-26T16:13:38.674897+0000 mgr.y (mgr.14152) 209 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: audit 2024-04-26T16:13:39.783446+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: audit 2024-04-26T16:13:39.783446+0000 mon.a (mon.0) 490 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: audit 2024-04-26T16:13:39.790506+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: audit 2024-04-26T16:13:39.790506+0000 mon.a (mon.0) 491 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: audit 2024-04-26T16:13:39.792184+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:40.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:39 smithi050 bash[21463]: audit 2024-04-26T16:13:39.792184+0000 mon.a (mon.0) 492 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:13:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:40 smithi190 bash[23868]: audit 2024-04-26T16:13:39.775606+0000 mgr.y (mgr.14152) 210 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:13:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:40 smithi190 bash[23868]: audit 2024-04-26T16:13:39.775606+0000 mgr.y (mgr.14152) 210 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:13:41.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:40 smithi050 bash[21463]: audit 2024-04-26T16:13:39.775606+0000 mgr.y (mgr.14152) 210 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:13:41.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:40 smithi050 bash[21463]: audit 2024-04-26T16:13:39.775606+0000 mgr.y (mgr.14152) 210 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:13:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:40 smithi050 bash[29234]: audit 2024-04-26T16:13:39.775606+0000 mgr.y (mgr.14152) 210 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:13:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:40 smithi050 bash[29234]: audit 2024-04-26T16:13:39.775606+0000 mgr.y (mgr.14152) 210 : audit [DBG] from='client.24202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:13:42.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:41 smithi190 bash[23868]: cluster 2024-04-26T16:13:40.675441+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:42.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:41 smithi190 bash[23868]: cluster 2024-04-26T16:13:40.675441+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:41 smithi050 bash[21463]: cluster 2024-04-26T16:13:40.675441+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:41 smithi050 bash[21463]: cluster 2024-04-26T16:13:40.675441+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:42.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:41 smithi050 bash[29234]: cluster 2024-04-26T16:13:40.675441+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:42.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:41 smithi050 bash[29234]: cluster 2024-04-26T16:13:40.675441+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:43 smithi190 bash[23868]: cluster 2024-04-26T16:13:42.676005+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:43 smithi190 bash[23868]: cluster 2024-04-26T16:13:42.676005+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:43 smithi050 bash[21463]: cluster 2024-04-26T16:13:42.676005+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:43 smithi050 bash[21463]: cluster 2024-04-26T16:13:42.676005+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:43 smithi050 bash[29234]: cluster 2024-04-26T16:13:42.676005+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:43 smithi050 bash[29234]: cluster 2024-04-26T16:13:42.676005+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:44.529 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:44.532+0000 7f7f05397700 10 client.24202.objecter tick 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.060304+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.060304+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.060420+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.060420+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.676499+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.676499+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.912928+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-04-26T16:13:46.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:45 smithi190 bash[23868]: cluster 2024-04-26T16:13:44.912928+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.060304+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.060304+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.060420+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.060420+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.676499+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.676499+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.912928+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-04-26T16:13:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[29234]: cluster 2024-04-26T16:13:44.912928+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.060304+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.060304+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.060420+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.060420+0000 osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.676499+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.676499+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.912928+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-04-26T16:13:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:45 smithi050 bash[21463]: cluster 2024-04-26T16:13:44.912928+0000 mon.a (mon.0) 493 : cluster [DBG] osdmap e30: 4 total, 4 up, 4 in 2024-04-26T16:13:47.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:46 smithi190 bash[23868]: cluster 2024-04-26T16:13:45.894476+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:47.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:46 smithi190 bash[23868]: cluster 2024-04-26T16:13:45.894476+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:47.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:46 smithi190 bash[23868]: cluster 2024-04-26T16:13:45.894610+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:47.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:46 smithi190 bash[23868]: cluster 2024-04-26T16:13:45.894610+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:47.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:46 smithi190 bash[23868]: cluster 2024-04-26T16:13:46.817816+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-04-26T16:13:47.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:46 smithi190 bash[23868]: cluster 2024-04-26T16:13:46.817816+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-04-26T16:13:47.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[21463]: cluster 2024-04-26T16:13:45.894476+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:47.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[21463]: cluster 2024-04-26T16:13:45.894476+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:47.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[21463]: cluster 2024-04-26T16:13:45.894610+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[21463]: cluster 2024-04-26T16:13:45.894610+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[21463]: cluster 2024-04-26T16:13:46.817816+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[21463]: cluster 2024-04-26T16:13:46.817816+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[29234]: cluster 2024-04-26T16:13:45.894476+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[29234]: cluster 2024-04-26T16:13:45.894476+0000 osd.1 (osd.1) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[29234]: cluster 2024-04-26T16:13:45.894610+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[29234]: cluster 2024-04-26T16:13:45.894610+0000 osd.1 (osd.1) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:13:47.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[29234]: cluster 2024-04-26T16:13:46.817816+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-04-26T16:13:47.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:46 smithi050 bash[29234]: cluster 2024-04-26T16:13:46.817816+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e31: 4 total, 4 up, 4 in 2024-04-26T16:13:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:47 smithi190 bash[23868]: cluster 2024-04-26T16:13:46.677033+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:47 smithi190 bash[23868]: cluster 2024-04-26T16:13:46.677033+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:47 smithi050 bash[21463]: cluster 2024-04-26T16:13:46.677033+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:47 smithi050 bash[21463]: cluster 2024-04-26T16:13:46.677033+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:47 smithi050 bash[29234]: cluster 2024-04-26T16:13:46.677033+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:47 smithi050 bash[29234]: cluster 2024-04-26T16:13:46.677033+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.941428+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.190:0/3037674430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.941428+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.190:0/3037674430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.946736+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.946736+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.950100+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]': finished 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.950100+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]': finished 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: cluster 2024-04-26T16:13:47.956093+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: cluster 2024-04-26T16:13:47.956093+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.956578+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:47.956578+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:48.650749+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.190:0/2459223984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:48 smithi190 bash[23868]: audit 2024-04-26T16:13:48.650749+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.190:0/2459223984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:49.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.941428+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.190:0/3037674430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.941428+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.190:0/3037674430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.946736+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.946736+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.950100+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]': finished 2024-04-26T16:13:49.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.950100+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]': finished 2024-04-26T16:13:49.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: cluster 2024-04-26T16:13:47.956093+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-04-26T16:13:49.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: cluster 2024-04-26T16:13:47.956093+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-04-26T16:13:49.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.956578+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:13:49.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:47.956578+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:13:49.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:48.650749+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.190:0/2459223984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:49.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[21463]: audit 2024-04-26T16:13:48.650749+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.190:0/2459223984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.941428+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.190:0/3037674430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.941428+0000 mon.b (mon.1) 10 : audit [INF] from='client.? 172.21.15.190:0/3037674430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.946736+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.946736+0000 mon.a (mon.0) 495 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]: dispatch 2024-04-26T16:13:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.950100+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]': finished 2024-04-26T16:13:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.950100+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7220491-e9bf-4188-b203-322e9908558c"}]': finished 2024-04-26T16:13:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: cluster 2024-04-26T16:13:47.956093+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-04-26T16:13:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: cluster 2024-04-26T16:13:47.956093+0000 mon.a (mon.0) 497 : cluster [DBG] osdmap e32: 5 total, 4 up, 5 in 2024-04-26T16:13:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.956578+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:13:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:47.956578+0000 mon.a (mon.0) 498 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:13:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:48.650749+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.190:0/2459223984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:49.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:48 smithi050 bash[29234]: audit 2024-04-26T16:13:48.650749+0000 mon.b (mon.1) 11 : audit [DBG] from='client.? 172.21.15.190:0/2459223984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:13:49.533 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:49.536+0000 7f7f05397700 10 client.24202.objecter tick 2024-04-26T16:13:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:49 smithi190 bash[23868]: cluster 2024-04-26T16:13:48.677511+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:49 smithi190 bash[23868]: cluster 2024-04-26T16:13:48.677511+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:50.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:49 smithi050 bash[21463]: cluster 2024-04-26T16:13:48.677511+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:50.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:49 smithi050 bash[21463]: cluster 2024-04-26T16:13:48.677511+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:49 smithi050 bash[29234]: cluster 2024-04-26T16:13:48.677511+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:49 smithi050 bash[29234]: cluster 2024-04-26T16:13:48.677511+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:51 smithi190 bash[23868]: cluster 2024-04-26T16:13:50.678151+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:51 smithi190 bash[23868]: cluster 2024-04-26T16:13:50.678151+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:51 smithi050 bash[21463]: cluster 2024-04-26T16:13:50.678151+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:51 smithi050 bash[21463]: cluster 2024-04-26T16:13:50.678151+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:51 smithi050 bash[29234]: cluster 2024-04-26T16:13:50.678151+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:51 smithi050 bash[29234]: cluster 2024-04-26T16:13:50.678151+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:53 smithi190 bash[23868]: cluster 2024-04-26T16:13:52.678851+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:53 smithi190 bash[23868]: cluster 2024-04-26T16:13:52.678851+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:53 smithi050 bash[21463]: cluster 2024-04-26T16:13:52.678851+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:53 smithi050 bash[21463]: cluster 2024-04-26T16:13:52.678851+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:53 smithi050 bash[29234]: cluster 2024-04-26T16:13:52.678851+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:53 smithi050 bash[29234]: cluster 2024-04-26T16:13:52.678851+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:54.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:54.540+0000 7f7f05397700 10 client.24202.objecter tick 2024-04-26T16:13:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:55 smithi190 bash[23868]: cluster 2024-04-26T16:13:54.679566+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:55 smithi190 bash[23868]: cluster 2024-04-26T16:13:54.679566+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:55 smithi050 bash[21463]: cluster 2024-04-26T16:13:54.679566+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:55 smithi050 bash[21463]: cluster 2024-04-26T16:13:54.679566+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:56.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:55 smithi050 bash[29234]: cluster 2024-04-26T16:13:54.679566+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:56.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:55 smithi050 bash[29234]: cluster 2024-04-26T16:13:54.679566+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:58.301 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:57 smithi190 bash[23868]: cluster 2024-04-26T16:13:56.680213+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:58.301 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:57 smithi190 bash[23868]: cluster 2024-04-26T16:13:56.680213+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:58.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:57 smithi050 bash[21463]: cluster 2024-04-26T16:13:56.680213+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:58.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:57 smithi050 bash[21463]: cluster 2024-04-26T16:13:56.680213+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:57 smithi050 bash[29234]: cluster 2024-04-26T16:13:56.680213+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:57 smithi050 bash[29234]: cluster 2024-04-26T16:13:56.680213+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:13:59.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:13:59.544+0000 7f7f05397700 10 client.24202.objecter tick 2024-04-26T16:14:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:59 smithi190 bash[23868]: cluster 2024-04-26T16:13:58.680919+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:13:59 smithi190 bash[23868]: cluster 2024-04-26T16:13:58.680919+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:59 smithi050 bash[21463]: cluster 2024-04-26T16:13:58.680919+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:13:59 smithi050 bash[21463]: cluster 2024-04-26T16:13:58.680919+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:00.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:59 smithi050 bash[29234]: cluster 2024-04-26T16:13:58.680919+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:00.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:13:59 smithi050 bash[29234]: cluster 2024-04-26T16:13:58.680919+0000 mgr.y (mgr.14152) 220 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:01.229 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: audit 2024-04-26T16:14:00.014264+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: audit 2024-04-26T16:14:00.014264+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: audit 2024-04-26T16:14:00.015832+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: audit 2024-04-26T16:14:00.015832+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: cephadm 2024-04-26T16:14:00.017401+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Deploying daemon osd.4 on smithi190 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: cephadm 2024-04-26T16:14:00.017401+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Deploying daemon osd.4 on smithi190 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: cluster 2024-04-26T16:14:00.681507+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:01.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:00 smithi190 bash[23868]: cluster 2024-04-26T16:14:00.681507+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: audit 2024-04-26T16:14:00.014264+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: audit 2024-04-26T16:14:00.014264+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: audit 2024-04-26T16:14:00.015832+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: audit 2024-04-26T16:14:00.015832+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: cephadm 2024-04-26T16:14:00.017401+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Deploying daemon osd.4 on smithi190 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: cephadm 2024-04-26T16:14:00.017401+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Deploying daemon osd.4 on smithi190 2024-04-26T16:14:01.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: cluster 2024-04-26T16:14:00.681507+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[21463]: cluster 2024-04-26T16:14:00.681507+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: audit 2024-04-26T16:14:00.014264+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: audit 2024-04-26T16:14:00.014264+0000 mon.a (mon.0) 499 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: audit 2024-04-26T16:14:00.015832+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: audit 2024-04-26T16:14:00.015832+0000 mon.a (mon.0) 500 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: cephadm 2024-04-26T16:14:00.017401+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Deploying daemon osd.4 on smithi190 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: cephadm 2024-04-26T16:14:00.017401+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Deploying daemon osd.4 on smithi190 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: cluster 2024-04-26T16:14:00.681507+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:01.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:00 smithi050 bash[29234]: cluster 2024-04-26T16:14:00.681507+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:03.770 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:14:03 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:03.770 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:03 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:03.770 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:03 smithi190 bash[23868]: cluster 2024-04-26T16:14:02.682045+0000 mgr.y (mgr.14152) 223 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:03.770 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:03 smithi190 bash[23868]: cluster 2024-04-26T16:14:02.682045+0000 mgr.y (mgr.14152) 223 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:04.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:03 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:04.042 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:14:03 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:04.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:03 smithi050 bash[29234]: cluster 2024-04-26T16:14:02.682045+0000 mgr.y (mgr.14152) 223 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:04.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:03 smithi050 bash[29234]: cluster 2024-04-26T16:14:02.682045+0000 mgr.y (mgr.14152) 223 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:04.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:03 smithi050 bash[21463]: cluster 2024-04-26T16:14:02.682045+0000 mgr.y (mgr.14152) 223 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:04.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:03 smithi050 bash[21463]: cluster 2024-04-26T16:14:02.682045+0000 mgr.y (mgr.14152) 223 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:04.544 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:04.548+0000 7f7f05397700 10 client.24202.objecter tick 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: audit 2024-04-26T16:14:04.056144+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: audit 2024-04-26T16:14:04.056144+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: audit 2024-04-26T16:14:04.064924+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: audit 2024-04-26T16:14:04.064924+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: audit 2024-04-26T16:14:04.072569+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: audit 2024-04-26T16:14:04.072569+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: cluster 2024-04-26T16:14:04.682477+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:05 smithi190 bash[23868]: cluster 2024-04-26T16:14:04.682477+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: audit 2024-04-26T16:14:04.056144+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: audit 2024-04-26T16:14:04.056144+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: audit 2024-04-26T16:14:04.064924+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: audit 2024-04-26T16:14:04.064924+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: audit 2024-04-26T16:14:04.072569+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: audit 2024-04-26T16:14:04.072569+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: cluster 2024-04-26T16:14:04.682477+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[21463]: cluster 2024-04-26T16:14:04.682477+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:05.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: audit 2024-04-26T16:14:04.056144+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: audit 2024-04-26T16:14:04.056144+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: audit 2024-04-26T16:14:04.064924+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: audit 2024-04-26T16:14:04.064924+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: audit 2024-04-26T16:14:04.072569+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: audit 2024-04-26T16:14:04.072569+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: cluster 2024-04-26T16:14:04.682477+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:05 smithi050 bash[29234]: cluster 2024-04-26T16:14:04.682477+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:07 smithi190 bash[23868]: cluster 2024-04-26T16:14:06.682992+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:07 smithi190 bash[23868]: cluster 2024-04-26T16:14:06.682992+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:08.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:07 smithi050 bash[21463]: cluster 2024-04-26T16:14:06.682992+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:08.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:07 smithi050 bash[21463]: cluster 2024-04-26T16:14:06.682992+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:07 smithi050 bash[29234]: cluster 2024-04-26T16:14:06.682992+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:07 smithi050 bash[29234]: cluster 2024-04-26T16:14:06.682992+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:09.548 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:09.552+0000 7f7f05397700 10 client.24202.objecter tick 2024-04-26T16:14:10.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:09 smithi190 bash[23868]: cluster 2024-04-26T16:14:08.683565+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:09 smithi190 bash[23868]: cluster 2024-04-26T16:14:08.683565+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:10.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:09 smithi050 bash[21463]: cluster 2024-04-26T16:14:08.683565+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:10.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:09 smithi050 bash[21463]: cluster 2024-04-26T16:14:08.683565+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:09 smithi050 bash[29234]: cluster 2024-04-26T16:14:08.683565+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:09 smithi050 bash[29234]: cluster 2024-04-26T16:14:08.683565+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[21463]: audit 2024-04-26T16:14:10.127554+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[21463]: audit 2024-04-26T16:14:10.127554+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[21463]: audit 2024-04-26T16:14:10.132728+0000 mon.a (mon.0) 504 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[21463]: audit 2024-04-26T16:14:10.132728+0000 mon.a (mon.0) 504 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[29234]: audit 2024-04-26T16:14:10.127554+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[29234]: audit 2024-04-26T16:14:10.127554+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[29234]: audit 2024-04-26T16:14:10.132728+0000 mon.a (mon.0) 504 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:10 smithi050 bash[29234]: audit 2024-04-26T16:14:10.132728+0000 mon.a (mon.0) 504 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:10 smithi190 bash[23868]: audit 2024-04-26T16:14:10.127554+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:10 smithi190 bash[23868]: audit 2024-04-26T16:14:10.127554+0000 mon.b (mon.1) 12 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:10 smithi190 bash[23868]: audit 2024-04-26T16:14:10.132728+0000 mon.a (mon.0) 504 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:10 smithi190 bash[23868]: audit 2024-04-26T16:14:10.132728+0000 mon.a (mon.0) 504 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-26T16:14:11.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: cluster 2024-04-26T16:14:10.684154+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:11.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: cluster 2024-04-26T16:14:10.684154+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:11.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.833520+0000 mon.a (mon.0) 505 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.833520+0000 mon.a (mon.0) 505 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.835761+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.835761+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: cluster 2024-04-26T16:14:10.841301+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: cluster 2024-04-26T16:14:10.841301+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.842687+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.842687+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.847273+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.847273+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.848550+0000 mon.a (mon.0) 509 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.848550+0000 mon.a (mon.0) 509 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.856108+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:11.889 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:10.856108+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:11.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:11.539002+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:11.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:11.539002+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:11.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:11.541077+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:11.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:11.541077+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:11.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:11.551827+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:11.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:11 smithi190 bash[23868]: audit 2024-04-26T16:14:11.551827+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: cluster 2024-04-26T16:14:10.684154+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:12.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: cluster 2024-04-26T16:14:10.684154+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.833520+0000 mon.a (mon.0) 505 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.833520+0000 mon.a (mon.0) 505 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.835761+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.835761+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: cluster 2024-04-26T16:14:10.841301+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: cluster 2024-04-26T16:14:10.841301+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.842687+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.842687+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.847273+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.847273+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.848550+0000 mon.a (mon.0) 509 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.848550+0000 mon.a (mon.0) 509 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.856108+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:10.856108+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:11.539002+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:11.539002+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: cluster 2024-04-26T16:14:10.684154+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: cluster 2024-04-26T16:14:10.684154+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.833520+0000 mon.a (mon.0) 505 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.833520+0000 mon.a (mon.0) 505 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.835761+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.835761+0000 mon.b (mon.1) 13 : audit [INF] from='osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: cluster 2024-04-26T16:14:10.841301+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: cluster 2024-04-26T16:14:10.841301+0000 mon.a (mon.0) 506 : cluster [DBG] osdmap e33: 5 total, 4 up, 5 in 2024-04-26T16:14:12.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.842687+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.842687+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.847273+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.847273+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.848550+0000 mon.a (mon.0) 509 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.848550+0000 mon.a (mon.0) 509 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.856108+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:10.856108+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:11.539002+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:11.539002+0000 mon.a (mon.0) 511 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:11.541077+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:12.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:11.541077+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:11.551827+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[21463]: audit 2024-04-26T16:14:11.551827+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:11.541077+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:11.541077+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:11.551827+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:12.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:11 smithi050 bash[29234]: audit 2024-04-26T16:14:11.551827+0000 mon.a (mon.0) 513 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: audit 2024-04-26T16:14:11.849175+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:14:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: audit 2024-04-26T16:14:11.849175+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:14:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: cluster 2024-04-26T16:14:11.856142+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-04-26T16:14:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: cluster 2024-04-26T16:14:11.856142+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-04-26T16:14:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: audit 2024-04-26T16:14:11.856917+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: audit 2024-04-26T16:14:11.856917+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: audit 2024-04-26T16:14:11.862937+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[21463]: audit 2024-04-26T16:14:11.862937+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: audit 2024-04-26T16:14:11.849175+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: audit 2024-04-26T16:14:11.849175+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: cluster 2024-04-26T16:14:11.856142+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: cluster 2024-04-26T16:14:11.856142+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: audit 2024-04-26T16:14:11.856917+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: audit 2024-04-26T16:14:11.856917+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: audit 2024-04-26T16:14:11.862937+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:12 smithi050 bash[29234]: audit 2024-04-26T16:14:11.862937+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: audit 2024-04-26T16:14:11.849175+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: audit 2024-04-26T16:14:11.849175+0000 mon.a (mon.0) 514 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: cluster 2024-04-26T16:14:11.856142+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: cluster 2024-04-26T16:14:11.856142+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e34: 5 total, 4 up, 5 in 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: audit 2024-04-26T16:14:11.856917+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: audit 2024-04-26T16:14:11.856917+0000 mon.a (mon.0) 516 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: audit 2024-04-26T16:14:11.862937+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:12 smithi190 bash[23868]: audit 2024-04-26T16:14:11.862937+0000 mon.a (mon.0) 517 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:11.117466+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:14.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:11.117466+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:14.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:11.117544+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:14.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:11.117544+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:14.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:12.684717+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:14.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:12.684717+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:12.868218+0000 mon.a (mon.0) 518 : cluster [INF] osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] boot 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:12.868218+0000 mon.a (mon.0) 518 : cluster [INF] osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] boot 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:12.868336+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: cluster 2024-04-26T16:14:12.868336+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: audit 2024-04-26T16:14:12.869616+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[21463]: audit 2024-04-26T16:14:12.869616+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:11.117466+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:11.117466+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:11.117544+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:11.117544+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:14.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:12.684717+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:12.684717+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:12.868218+0000 mon.a (mon.0) 518 : cluster [INF] osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] boot 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:12.868218+0000 mon.a (mon.0) 518 : cluster [INF] osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] boot 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:12.868336+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: cluster 2024-04-26T16:14:12.868336+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: audit 2024-04-26T16:14:12.869616+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:13 smithi050 bash[29234]: audit 2024-04-26T16:14:12.869616+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.316 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.316+0000 7f7efcff9700 1 -- 172.21.15.190:0/1506453811 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7f00063860 con 0x7f7eec071e90 2024-04-26T16:14:14.317 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 4 on host 'smithi190' 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 10 librados: watch_flush enter 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 10 librados: watch_flush exit 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 20 client.24202.objecter shutdown clearing up homeless session... 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 10 client.24202.objecter successfully canceled tick 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7eec071e90 msgr2=0x7f7eec074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7eec071e90 0x7f7eec074350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7ee80098e0 tx=0x7f7ee8005fb0 comp rx=0 tx=0).stop 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 msgr2=0x7f7f001a01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 0x7f7f001a01f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800ada0 tx=0x7f7ef800c330 comp rx=0 tx=0).stop 2024-04-26T16:14:14.320 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 shutdown_connections 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7eec071e90 0x7f7eec074350 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f00109170 0x7f7f001a01f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f00103f10 0x7f7f00198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 --2- 172.21.15.190:0/1506453811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f00106840 0x7f7f001991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 >> 172.21.15.190:0/1506453811 conn(0x7f7f000fd940 msgr2=0x7f7f00104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 shutdown_connections 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 -- 172.21.15.190:0/1506453811 wait complete. 2024-04-26T16:14:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:14.320+0000 7f7f05b98700 1 librados: shutdown 2024-04-26T16:14:14.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:11.117466+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:11.117466+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:11.117544+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:11.117544+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:12.684717+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:12.684717+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v203: 1 pgs: 1 peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:12.868218+0000 mon.a (mon.0) 518 : cluster [INF] osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] boot 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:12.868218+0000 mon.a (mon.0) 518 : cluster [INF] osd.4 [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] boot 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:12.868336+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: cluster 2024-04-26T16:14:12.868336+0000 mon.a (mon.0) 519 : cluster [DBG] osdmap e35: 5 total, 5 up, 5 in 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: audit 2024-04-26T16:14:12.869616+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:13 smithi190 bash[23868]: audit 2024-04-26T16:14:12.869616+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:14:14.861 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.4.service 2024-04-26T16:14:14.864 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_3... 2024-04-26T16:14:14.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: cluster 2024-04-26T16:14:13.867079+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: cluster 2024-04-26T16:14:13.867079+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: audit 2024-04-26T16:14:14.291446+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: audit 2024-04-26T16:14:14.291446+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: audit 2024-04-26T16:14:14.296016+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: audit 2024-04-26T16:14:14.296016+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:15.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: audit 2024-04-26T16:14:14.301799+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: audit 2024-04-26T16:14:14.301799+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: cluster 2024-04-26T16:14:14.868420+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[21463]: cluster 2024-04-26T16:14:14.868420+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: cluster 2024-04-26T16:14:13.867079+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: cluster 2024-04-26T16:14:13.867079+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: audit 2024-04-26T16:14:14.291446+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: audit 2024-04-26T16:14:14.291446+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: audit 2024-04-26T16:14:14.296016+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: audit 2024-04-26T16:14:14.296016+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: audit 2024-04-26T16:14:14.301799+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: audit 2024-04-26T16:14:14.301799+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: cluster 2024-04-26T16:14:14.868420+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-04-26T16:14:15.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:14 smithi050 bash[29234]: cluster 2024-04-26T16:14:14.868420+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: cluster 2024-04-26T16:14:13.867079+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: cluster 2024-04-26T16:14:13.867079+0000 mon.a (mon.0) 521 : cluster [DBG] osdmap e36: 5 total, 5 up, 5 in 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: audit 2024-04-26T16:14:14.291446+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: audit 2024-04-26T16:14:14.291446+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: audit 2024-04-26T16:14:14.296016+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: audit 2024-04-26T16:14:14.296016+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: audit 2024-04-26T16:14:14.301799+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: audit 2024-04-26T16:14:14.301799+0000 mon.a (mon.0) 524 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: cluster 2024-04-26T16:14:14.868420+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-04-26T16:14:15.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:14 smithi190 bash[23868]: cluster 2024-04-26T16:14:14.868420+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e37: 5 total, 5 up, 5 in 2024-04-26T16:14:16.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[21463]: cluster 2024-04-26T16:14:14.685350+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:16.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[21463]: cluster 2024-04-26T16:14:14.685350+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:16.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[21463]: cluster 2024-04-26T16:14:15.825140+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-04-26T16:14:16.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[21463]: cluster 2024-04-26T16:14:15.825140+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-04-26T16:14:16.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[29234]: cluster 2024-04-26T16:14:14.685350+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:16.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[29234]: cluster 2024-04-26T16:14:14.685350+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:16.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[29234]: cluster 2024-04-26T16:14:15.825140+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-04-26T16:14:16.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:15 smithi050 bash[29234]: cluster 2024-04-26T16:14:15.825140+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-04-26T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:15 smithi190 bash[23868]: cluster 2024-04-26T16:14:14.685350+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:15 smithi190 bash[23868]: cluster 2024-04-26T16:14:14.685350+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v206: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:15 smithi190 bash[23868]: cluster 2024-04-26T16:14:15.825140+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-04-26T16:14:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:15 smithi190 bash[23868]: cluster 2024-04-26T16:14:15.825140+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e38: 5 total, 5 up, 5 in 2024-04-26T16:14:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:17 smithi190 bash[23868]: cluster 2024-04-26T16:14:16.685943+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:17 smithi190 bash[23868]: cluster 2024-04-26T16:14:16.685943+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:17 smithi190 bash[23868]: cluster 2024-04-26T16:14:16.826156+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-04-26T16:14:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:17 smithi190 bash[23868]: cluster 2024-04-26T16:14:16.826156+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[21463]: cluster 2024-04-26T16:14:16.685943+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[21463]: cluster 2024-04-26T16:14:16.685943+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[21463]: cluster 2024-04-26T16:14:16.826156+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[21463]: cluster 2024-04-26T16:14:16.826156+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[29234]: cluster 2024-04-26T16:14:16.685943+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[29234]: cluster 2024-04-26T16:14:16.685943+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[29234]: cluster 2024-04-26T16:14:16.826156+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-04-26T16:14:18.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:17 smithi050 bash[29234]: cluster 2024-04-26T16:14:16.826156+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e39: 5 total, 5 up, 5 in 2024-04-26T16:14:19.267 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:14:20.089 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:19 smithi190 bash[23868]: cluster 2024-04-26T16:14:18.686516+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T16:14:20.089 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:19 smithi190 bash[23868]: cluster 2024-04-26T16:14:18.686516+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T16:14:20.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:19 smithi050 bash[21463]: cluster 2024-04-26T16:14:18.686516+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T16:14:20.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:19 smithi050 bash[21463]: cluster 2024-04-26T16:14:18.686516+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T16:14:20.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:19 smithi050 bash[29234]: cluster 2024-04-26T16:14:18.686516+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T16:14:20.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:19 smithi050 bash[29234]: cluster 2024-04-26T16:14:18.686516+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.089 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:21 smithi190 bash[23868]: cluster 2024-04-26T16:14:20.687120+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.090 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:21 smithi190 bash[23868]: cluster 2024-04-26T16:14:20.687120+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:21 smithi050 bash[21463]: cluster 2024-04-26T16:14:20.687120+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:21 smithi050 bash[21463]: cluster 2024-04-26T16:14:20.687120+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:21 smithi050 bash[29234]: cluster 2024-04-26T16:14:20.687120+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:21 smithi050 bash[29234]: cluster 2024-04-26T16:14:20.687120+0000 mgr.y (mgr.14152) 232 : cluster [DBG] pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:14:22.906 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-26T16:14:22.906 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-26T16:14:22.906 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-26T16:14:22.907 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-26T16:14:22.907 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145843 s, 719 MB/s 2024-04-26T16:14:22.907 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 2024-04-26T16:14:22.907 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-26T16:14:23.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-04-26T16:14:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:23 smithi190 bash[23868]: cluster 2024-04-26T16:14:22.687643+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T16:14:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:23 smithi190 bash[23868]: cluster 2024-04-26T16:14:22.687643+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T16:14:24.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:23 smithi050 bash[21463]: cluster 2024-04-26T16:14:22.687643+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T16:14:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:23 smithi050 bash[21463]: cluster 2024-04-26T16:14:22.687643+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T16:14:24.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:23 smithi050 bash[29234]: cluster 2024-04-26T16:14:22.687643+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T16:14:24.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:23 smithi050 bash[29234]: cluster 2024-04-26T16:14:22.687643+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-26T16:14:25.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: cephadm 2024-04-26T16:14:24.246173+0000 mgr.y (mgr.14152) 234 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:14:25.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: cephadm 2024-04-26T16:14:24.246173+0000 mgr.y (mgr.14152) 234 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:14:25.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.255750+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.255750+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.265927+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.265927+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.268237+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.268237+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: cephadm 2024-04-26T16:14:24.269760+0000 mgr.y (mgr.14152) 235 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 17265M 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: cephadm 2024-04-26T16:14:24.269760+0000 mgr.y (mgr.14152) 235 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 17265M 2024-04-26T16:14:25.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.278140+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.278140+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.283922+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:25.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.283922+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:25.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.285918+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:25.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.285918+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:25.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.296171+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: audit 2024-04-26T16:14:24.296171+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: cluster 2024-04-26T16:14:24.688148+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:14:25.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:25 smithi190 bash[23868]: cluster 2024-04-26T16:14:24.688148+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: cephadm 2024-04-26T16:14:24.246173+0000 mgr.y (mgr.14152) 234 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: cephadm 2024-04-26T16:14:24.246173+0000 mgr.y (mgr.14152) 234 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.255750+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.255750+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.265927+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.265927+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.268237+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.268237+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: cephadm 2024-04-26T16:14:24.269760+0000 mgr.y (mgr.14152) 235 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 17265M 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: cephadm 2024-04-26T16:14:24.269760+0000 mgr.y (mgr.14152) 235 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 17265M 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.278140+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.278140+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.283922+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.283922+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.285918+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.285918+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.296171+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: audit 2024-04-26T16:14:24.296171+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: cluster 2024-04-26T16:14:24.688148+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:14:25.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[21463]: cluster 2024-04-26T16:14:24.688148+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: cephadm 2024-04-26T16:14:24.246173+0000 mgr.y (mgr.14152) 234 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: cephadm 2024-04-26T16:14:24.246173+0000 mgr.y (mgr.14152) 234 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.255750+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.255750+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.265927+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.265927+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.268237+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.268237+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: cephadm 2024-04-26T16:14:24.269760+0000 mgr.y (mgr.14152) 235 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 17265M 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: cephadm 2024-04-26T16:14:24.269760+0000 mgr.y (mgr.14152) 235 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 17265M 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.278140+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.278140+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.283922+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:25.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.283922+0000 mon.a (mon.0) 532 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.285918+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.285918+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:14:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.296171+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: audit 2024-04-26T16:14:24.296171+0000 mon.a (mon.0) 534 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: cluster 2024-04-26T16:14:24.688148+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:14:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:25 smithi050 bash[29234]: cluster 2024-04-26T16:14:24.688148+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:27 smithi190 bash[23868]: cluster 2024-04-26T16:14:26.688765+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:27 smithi190 bash[23868]: cluster 2024-04-26T16:14:26.688765+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:27 smithi050 bash[21463]: cluster 2024-04-26T16:14:26.688765+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:27 smithi050 bash[21463]: cluster 2024-04-26T16:14:26.688765+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:27 smithi050 bash[29234]: cluster 2024-04-26T16:14:26.688765+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:27 smithi050 bash[29234]: cluster 2024-04-26T16:14:26.688765+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:14:28.437 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:14:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:29 smithi050 bash[21463]: cluster 2024-04-26T16:14:28.689361+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:14:30.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:29 smithi050 bash[21463]: cluster 2024-04-26T16:14:28.689361+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:14:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:29 smithi050 bash[29234]: cluster 2024-04-26T16:14:28.689361+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:14:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:29 smithi050 bash[29234]: cluster 2024-04-26T16:14:28.689361+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:14:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:29 smithi190 bash[23868]: cluster 2024-04-26T16:14:28.689361+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:14:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:29 smithi190 bash[23868]: cluster 2024-04-26T16:14:28.689361+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- 172.21.15.190:0/3281322429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 msgr2=0x7f41ac104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- 172.21.15.190:0/3281322429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac104af0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f41940099c0 tx=0x7f4194009cd0 comp rx=0 tx=0).stop 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- 172.21.15.190:0/3281322429 shutdown_connections 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- 172.21.15.190:0/3281322429 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f41ac105030 0x7f41ac10b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- 172.21.15.190:0/3281322429 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41ac103490 0x7f41ac1038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- 172.21.15.190:0/3281322429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac104af0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- 172.21.15.190:0/3281322429 >> 172.21.15.190:0/3281322429 conn(0x7f41ac0fe9f0 msgr2=0x7f41ac100e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- 172.21.15.190:0/3281322429 shutdown_connections 2024-04-26T16:14:30.847 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- 172.21.15.190:0/3281322429 wait complete. 2024-04-26T16:14:30.849 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 librados: starting msgr at 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 librados: starting objecter 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 Processor -- start 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- start start 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 librados: setting wanted keys 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 librados: calling monclient init 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41ac103490 0x7f41ac198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac1991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f41ac105030 0x7f41ac1a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:14:30.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10e460 con 0x7f41ac103490 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10e2e0 con 0x7f41ac104690 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.848+0000 7f41b13ce700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f41ac10e5e0 con 0x7f41ac105030 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac1991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac1991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:56592/0 (socket says 172.21.15.190:56592) 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 -- 172.21.15.190:0/1983239977 learned_addr learned my addr 172.21.15.190:0/1983239977 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aaffd700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41ac103490 0x7f41ac198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41ab7fe700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f41ac105030 0x7f41ac1a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 -- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f41ac105030 msgr2=0x7f41ac1a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f41ac105030 0x7f41ac1a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 -- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41ac103490 msgr2=0x7f41ac198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41ac103490 0x7f41ac198c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:14:30.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 -- 172.21.15.190:0/1983239977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac1a0910 con 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aa7fc700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac1991d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4194009990 tx=0x7f4194009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 10 client.?.objecter ms_handle_connect 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194026070 con 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f41940040b0 con 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac1a0ba0 con 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419400fcc0 con 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41b13ce700 10 client.24229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac1a1050 con 0x7f41ac104690 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41b13ce700 1 librados: init done 2024-04-26T16:14:30.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ac06a110 con 0x7f41ac104690 2024-04-26T16:14:30.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f4194029070 con 0x7f41ac104690 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f4198071e90 0x7f4198074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 3991+0+0 (secure 0 0 0) 0x7f4194091090 con 0x7f41ac104690 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 10 client.24229.objecter ms_dispatch 0x7f41ac001550 osd_map(40..40 src has 1..40) 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 3 client.24229.objecter handle_osd_map got epochs [40,40] > 0 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 3 client.24229.objecter handle_osd_map decoding full epoch 40 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41a3fff700 20 client.24229.objecter dump_active .. 0 homeless 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.852+0000 7f41aaffd700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f4198071e90 0x7f4198074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.856+0000 7f41a3fff700 10 client.24229.objecter ms_handle_connect 0x7f4198071e90 2024-04-26T16:14:30.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.856+0000 7f41aaffd700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f4198071e90 0x7f4198074350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007720 tx=0x7f41a40076b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:14:30.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:30.860+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4194013070 con 0x7f41ac104690 2024-04-26T16:14:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:30 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.435250+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:30 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.435250+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:30 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.435345+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:30 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.435345+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:31.077 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:31.080+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f41ac063860 con 0x7f4198071e90 2024-04-26T16:14:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.435250+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.435250+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.435345+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:31.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.435345+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:31.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.435250+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:31.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.435250+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:31.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.435345+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:31.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:30 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.435345+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.689900+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.689900+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.855976+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: cluster 2024-04-26T16:14:30.855976+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: audit 2024-04-26T16:14:31.096359+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: audit 2024-04-26T16:14:31.096359+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: audit 2024-04-26T16:14:31.103256+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: audit 2024-04-26T16:14:31.103256+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: audit 2024-04-26T16:14:31.105007+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:32.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[21463]: audit 2024-04-26T16:14:31.105007+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:32.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.689900+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:32.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.689900+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:32.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.855976+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-04-26T16:14:32.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: cluster 2024-04-26T16:14:30.855976+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-04-26T16:14:32.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: audit 2024-04-26T16:14:31.096359+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:14:32.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: audit 2024-04-26T16:14:31.096359+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:14:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: audit 2024-04-26T16:14:31.103256+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:14:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: audit 2024-04-26T16:14:31.103256+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:14:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: audit 2024-04-26T16:14:31.105007+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:32.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:31 smithi050 bash[29234]: audit 2024-04-26T16:14:31.105007+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.689900+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:32.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.689900+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:32.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.855976+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-04-26T16:14:32.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: cluster 2024-04-26T16:14:30.855976+0000 mon.a (mon.0) 535 : cluster [DBG] osdmap e40: 5 total, 5 up, 5 in 2024-04-26T16:14:32.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: audit 2024-04-26T16:14:31.096359+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:14:32.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: audit 2024-04-26T16:14:31.096359+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:14:32.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: audit 2024-04-26T16:14:31.103256+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:14:32.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: audit 2024-04-26T16:14:31.103256+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:14:32.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: audit 2024-04-26T16:14:31.105007+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:32.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:31 smithi190 bash[23868]: audit 2024-04-26T16:14:31.105007+0000 mon.a (mon.0) 538 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:33.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:32 smithi050 bash[21463]: audit 2024-04-26T16:14:31.088544+0000 mgr.y (mgr.14152) 240 : audit [DBG] from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:14:33.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:32 smithi050 bash[21463]: audit 2024-04-26T16:14:31.088544+0000 mgr.y (mgr.14152) 240 : audit [DBG] from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:14:33.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:32 smithi050 bash[29234]: audit 2024-04-26T16:14:31.088544+0000 mgr.y (mgr.14152) 240 : audit [DBG] from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:14:33.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:32 smithi050 bash[29234]: audit 2024-04-26T16:14:31.088544+0000 mgr.y (mgr.14152) 240 : audit [DBG] from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:14:33.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:32 smithi190 bash[23868]: audit 2024-04-26T16:14:31.088544+0000 mgr.y (mgr.14152) 240 : audit [DBG] from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:14:33.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:32 smithi190 bash[23868]: audit 2024-04-26T16:14:31.088544+0000 mgr.y (mgr.14152) 240 : audit [DBG] from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:14:34.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:33 smithi050 bash[21463]: cluster 2024-04-26T16:14:32.690434+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:34.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:33 smithi050 bash[21463]: cluster 2024-04-26T16:14:32.690434+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:34.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:33 smithi050 bash[29234]: cluster 2024-04-26T16:14:32.690434+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:34.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:33 smithi050 bash[29234]: cluster 2024-04-26T16:14:32.690434+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:34.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:33 smithi190 bash[23868]: cluster 2024-04-26T16:14:32.690434+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:34.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:33 smithi190 bash[23868]: cluster 2024-04-26T16:14:32.690434+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:35.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:35.856+0000 7f41b0bcd700 10 client.24229.objecter tick 2024-04-26T16:14:36.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:35 smithi050 bash[21463]: cluster 2024-04-26T16:14:34.690969+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:36.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:35 smithi050 bash[21463]: cluster 2024-04-26T16:14:34.690969+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:36.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:35 smithi050 bash[29234]: cluster 2024-04-26T16:14:34.690969+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:36.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:35 smithi050 bash[29234]: cluster 2024-04-26T16:14:34.690969+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:36.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:35 smithi190 bash[23868]: cluster 2024-04-26T16:14:34.690969+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:36.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:35 smithi190 bash[23868]: cluster 2024-04-26T16:14:34.690969+0000 mgr.y (mgr.14152) 242 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:38.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:37 smithi050 bash[29234]: cluster 2024-04-26T16:14:36.691498+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:38.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:37 smithi050 bash[29234]: cluster 2024-04-26T16:14:36.691498+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:38.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:37 smithi050 bash[21463]: cluster 2024-04-26T16:14:36.691498+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:38.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:37 smithi050 bash[21463]: cluster 2024-04-26T16:14:36.691498+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:38.239 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:37 smithi190 bash[23868]: cluster 2024-04-26T16:14:36.691498+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:38.239 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:37 smithi190 bash[23868]: cluster 2024-04-26T16:14:36.691498+0000 mgr.y (mgr.14152) 243 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: cluster 2024-04-26T16:14:38.692052+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: cluster 2024-04-26T16:14:38.692052+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.169359+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.190:0/2847302385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.169359+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.190:0/2847302385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.174810+0000 mon.a (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.174810+0000 mon.a (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.178748+0000 mon.a (mon.0) 540 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]': finished 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.178748+0000 mon.a (mon.0) 540 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]': finished 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: cluster 2024-04-26T16:14:39.184656+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: cluster 2024-04-26T16:14:39.184656+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.185169+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:40.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[21463]: audit 2024-04-26T16:14:39.185169+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: cluster 2024-04-26T16:14:38.692052+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: cluster 2024-04-26T16:14:38.692052+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.169359+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.190:0/2847302385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.169359+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.190:0/2847302385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.174810+0000 mon.a (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.174810+0000 mon.a (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.178748+0000 mon.a (mon.0) 540 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]': finished 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.178748+0000 mon.a (mon.0) 540 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]': finished 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: cluster 2024-04-26T16:14:39.184656+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: cluster 2024-04-26T16:14:39.184656+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.185169+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:40.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:39 smithi050 bash[29234]: audit 2024-04-26T16:14:39.185169+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: cluster 2024-04-26T16:14:38.692052+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: cluster 2024-04-26T16:14:38.692052+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.169359+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.190:0/2847302385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.169359+0000 mon.b (mon.1) 14 : audit [INF] from='client.? 172.21.15.190:0/2847302385' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.174810+0000 mon.a (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.174810+0000 mon.a (mon.0) 539 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]: dispatch 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.178748+0000 mon.a (mon.0) 540 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]': finished 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.178748+0000 mon.a (mon.0) 540 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587"}]': finished 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: cluster 2024-04-26T16:14:39.184656+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: cluster 2024-04-26T16:14:39.184656+0000 mon.a (mon.0) 541 : cluster [DBG] osdmap e41: 6 total, 5 up, 6 in 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.185169+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:39 smithi190 bash[23868]: audit 2024-04-26T16:14:39.185169+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:40.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:40.860+0000 7f41b0bcd700 10 client.24229.objecter tick 2024-04-26T16:14:41.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[29234]: cluster 2024-04-26T16:14:39.821072+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:14:41.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[29234]: cluster 2024-04-26T16:14:39.821072+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:14:41.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[29234]: cluster 2024-04-26T16:14:39.826009+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:14:41.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[29234]: cluster 2024-04-26T16:14:39.826009+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:14:41.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[29234]: audit 2024-04-26T16:14:39.900666+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.190:0/1488725603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[29234]: audit 2024-04-26T16:14:39.900666+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.190:0/1488725603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[21463]: cluster 2024-04-26T16:14:39.821072+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[21463]: cluster 2024-04-26T16:14:39.821072+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[21463]: cluster 2024-04-26T16:14:39.826009+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[21463]: cluster 2024-04-26T16:14:39.826009+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[21463]: audit 2024-04-26T16:14:39.900666+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.190:0/1488725603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:14:41.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:40 smithi050 bash[21463]: audit 2024-04-26T16:14:39.900666+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.190:0/1488725603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:14:41.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:40 smithi190 bash[23868]: cluster 2024-04-26T16:14:39.821072+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:14:41.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:40 smithi190 bash[23868]: cluster 2024-04-26T16:14:39.821072+0000 osd.1 (osd.1) 9 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:14:41.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:40 smithi190 bash[23868]: cluster 2024-04-26T16:14:39.826009+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:14:41.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:40 smithi190 bash[23868]: cluster 2024-04-26T16:14:39.826009+0000 osd.1 (osd.1) 10 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:14:41.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:40 smithi190 bash[23868]: audit 2024-04-26T16:14:39.900666+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.190:0/1488725603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:14:41.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:40 smithi190 bash[23868]: audit 2024-04-26T16:14:39.900666+0000 mon.b (mon.1) 15 : audit [DBG] from='client.? 172.21.15.190:0/1488725603' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:14:42.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:41 smithi050 bash[21463]: cluster 2024-04-26T16:14:40.692543+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:42.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:41 smithi050 bash[21463]: cluster 2024-04-26T16:14:40.692543+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:41 smithi050 bash[29234]: cluster 2024-04-26T16:14:40.692543+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:41 smithi050 bash[29234]: cluster 2024-04-26T16:14:40.692543+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:42.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:41 smithi190 bash[23868]: cluster 2024-04-26T16:14:40.692543+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:42.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:41 smithi190 bash[23868]: cluster 2024-04-26T16:14:40.692543+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:44.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:43 smithi050 bash[21463]: cluster 2024-04-26T16:14:42.693224+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:44.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:43 smithi050 bash[21463]: cluster 2024-04-26T16:14:42.693224+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:43 smithi050 bash[29234]: cluster 2024-04-26T16:14:42.693224+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:43 smithi050 bash[29234]: cluster 2024-04-26T16:14:42.693224+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:44.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:43 smithi190 bash[23868]: cluster 2024-04-26T16:14:42.693224+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:44.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:43 smithi190 bash[23868]: cluster 2024-04-26T16:14:42.693224+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:45.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:44 smithi190 bash[23868]: cluster 2024-04-26T16:14:43.975514+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:45.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:44 smithi190 bash[23868]: cluster 2024-04-26T16:14:43.975514+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:45.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:44 smithi190 bash[23868]: cluster 2024-04-26T16:14:43.975646+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:45.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:44 smithi190 bash[23868]: cluster 2024-04-26T16:14:43.975646+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[29234]: cluster 2024-04-26T16:14:43.975514+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[29234]: cluster 2024-04-26T16:14:43.975514+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[29234]: cluster 2024-04-26T16:14:43.975646+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[29234]: cluster 2024-04-26T16:14:43.975646+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[21463]: cluster 2024-04-26T16:14:43.975514+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[21463]: cluster 2024-04-26T16:14:43.975514+0000 osd.3 (osd.3) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[21463]: cluster 2024-04-26T16:14:43.975646+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:44 smithi050 bash[21463]: cluster 2024-04-26T16:14:43.975646+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:14:45.860 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:45.864+0000 7f41b0bcd700 10 client.24229.objecter tick 2024-04-26T16:14:46.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:45 smithi190 bash[23868]: cluster 2024-04-26T16:14:44.693777+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:46.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:45 smithi190 bash[23868]: cluster 2024-04-26T16:14:44.693777+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:46.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:45 smithi190 bash[23868]: cluster 2024-04-26T16:14:44.899907+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-04-26T16:14:46.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:45 smithi190 bash[23868]: cluster 2024-04-26T16:14:44.899907+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-04-26T16:14:46.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:45 smithi190 bash[23868]: audit 2024-04-26T16:14:44.900253+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:46.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:45 smithi190 bash[23868]: audit 2024-04-26T16:14:44.900253+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[29234]: cluster 2024-04-26T16:14:44.693777+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[29234]: cluster 2024-04-26T16:14:44.693777+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[29234]: cluster 2024-04-26T16:14:44.899907+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-04-26T16:14:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[29234]: cluster 2024-04-26T16:14:44.899907+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-04-26T16:14:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[29234]: audit 2024-04-26T16:14:44.900253+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[29234]: audit 2024-04-26T16:14:44.900253+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[21463]: cluster 2024-04-26T16:14:44.693777+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[21463]: cluster 2024-04-26T16:14:44.693777+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[21463]: cluster 2024-04-26T16:14:44.899907+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-04-26T16:14:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[21463]: cluster 2024-04-26T16:14:44.899907+0000 mon.a (mon.0) 543 : cluster [DBG] osdmap e42: 6 total, 5 up, 6 in 2024-04-26T16:14:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[21463]: audit 2024-04-26T16:14:44.900253+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:45 smithi050 bash[21463]: audit 2024-04-26T16:14:44.900253+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:14:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:47 smithi190 bash[23868]: cluster 2024-04-26T16:14:46.694275+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:47 smithi190 bash[23868]: cluster 2024-04-26T16:14:46.694275+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:48.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:47 smithi050 bash[21463]: cluster 2024-04-26T16:14:46.694275+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:48.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:47 smithi050 bash[21463]: cluster 2024-04-26T16:14:46.694275+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:47 smithi050 bash[29234]: cluster 2024-04-26T16:14:46.694275+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:47 smithi050 bash[29234]: cluster 2024-04-26T16:14:46.694275+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.221 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:49 smithi190 bash[23868]: cluster 2024-04-26T16:14:48.694956+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.221 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:49 smithi190 bash[23868]: cluster 2024-04-26T16:14:48.694956+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:49 smithi050 bash[21463]: cluster 2024-04-26T16:14:48.694956+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:49 smithi050 bash[21463]: cluster 2024-04-26T16:14:48.694956+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:49 smithi050 bash[29234]: cluster 2024-04-26T16:14:48.694956+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:49 smithi050 bash[29234]: cluster 2024-04-26T16:14:48.694956+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:50.864 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:50.868+0000 7f41b0bcd700 10 client.24229.objecter tick 2024-04-26T16:14:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:51 smithi190 bash[23868]: cluster 2024-04-26T16:14:50.695565+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:51 smithi190 bash[23868]: cluster 2024-04-26T16:14:50.695565+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:51 smithi190 bash[23868]: audit 2024-04-26T16:14:51.333040+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T16:14:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:51 smithi190 bash[23868]: audit 2024-04-26T16:14:51.333040+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T16:14:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:51 smithi190 bash[23868]: audit 2024-04-26T16:14:51.334910+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:51 smithi190 bash[23868]: audit 2024-04-26T16:14:51.334910+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:52.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[21463]: cluster 2024-04-26T16:14:50.695565+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:52.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[21463]: cluster 2024-04-26T16:14:50.695565+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:52.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[21463]: audit 2024-04-26T16:14:51.333040+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T16:14:52.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[21463]: audit 2024-04-26T16:14:51.333040+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[21463]: audit 2024-04-26T16:14:51.334910+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[21463]: audit 2024-04-26T16:14:51.334910+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[29234]: cluster 2024-04-26T16:14:50.695565+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[29234]: cluster 2024-04-26T16:14:50.695565+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[29234]: audit 2024-04-26T16:14:51.333040+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[29234]: audit 2024-04-26T16:14:51.333040+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[29234]: audit 2024-04-26T16:14:51.334910+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:52.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:51 smithi050 bash[29234]: audit 2024-04-26T16:14:51.334910+0000 mon.a (mon.0) 546 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:14:53.309 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:52 smithi190 bash[23868]: cephadm 2024-04-26T16:14:51.336612+0000 mgr.y (mgr.14152) 251 : cephadm [INF] Deploying daemon osd.5 on smithi190 2024-04-26T16:14:53.309 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:52 smithi190 bash[23868]: cephadm 2024-04-26T16:14:51.336612+0000 mgr.y (mgr.14152) 251 : cephadm [INF] Deploying daemon osd.5 on smithi190 2024-04-26T16:14:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:52 smithi050 bash[21463]: cephadm 2024-04-26T16:14:51.336612+0000 mgr.y (mgr.14152) 251 : cephadm [INF] Deploying daemon osd.5 on smithi190 2024-04-26T16:14:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:52 smithi050 bash[21463]: cephadm 2024-04-26T16:14:51.336612+0000 mgr.y (mgr.14152) 251 : cephadm [INF] Deploying daemon osd.5 on smithi190 2024-04-26T16:14:53.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:52 smithi050 bash[29234]: cephadm 2024-04-26T16:14:51.336612+0000 mgr.y (mgr.14152) 251 : cephadm [INF] Deploying daemon osd.5 on smithi190 2024-04-26T16:14:53.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:52 smithi050 bash[29234]: cephadm 2024-04-26T16:14:51.336612+0000 mgr.y (mgr.14152) 251 : cephadm [INF] Deploying daemon osd.5 on smithi190 2024-04-26T16:14:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:53 smithi190 bash[23868]: cluster 2024-04-26T16:14:52.696128+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:53 smithi190 bash[23868]: cluster 2024-04-26T16:14:52.696128+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:53 smithi050 bash[21463]: cluster 2024-04-26T16:14:52.696128+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:53 smithi050 bash[21463]: cluster 2024-04-26T16:14:52.696128+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:54.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:53 smithi050 bash[29234]: cluster 2024-04-26T16:14:52.696128+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:54.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:53 smithi050 bash[29234]: cluster 2024-04-26T16:14:52.696128+0000 mgr.y (mgr.14152) 252 : cluster [DBG] pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:55.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:55.490 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:14:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:55.490 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:14:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:55.761 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:14:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:55.762 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:14:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:14:55.867 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:14:55.871+0000 7f41b0bcd700 10 client.24229.objecter tick 2024-04-26T16:14:56.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: cluster 2024-04-26T16:14:54.696700+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: cluster 2024-04-26T16:14:54.696700+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: audit 2024-04-26T16:14:55.774992+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: audit 2024-04-26T16:14:55.774992+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: audit 2024-04-26T16:14:55.775890+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: audit 2024-04-26T16:14:55.775890+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: audit 2024-04-26T16:14:55.784708+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:55 smithi190 bash[23868]: audit 2024-04-26T16:14:55.784708+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: cluster 2024-04-26T16:14:54.696700+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: cluster 2024-04-26T16:14:54.696700+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: audit 2024-04-26T16:14:55.774992+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: audit 2024-04-26T16:14:55.774992+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: audit 2024-04-26T16:14:55.775890+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: audit 2024-04-26T16:14:55.775890+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: audit 2024-04-26T16:14:55.784708+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[21463]: audit 2024-04-26T16:14:55.784708+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: cluster 2024-04-26T16:14:54.696700+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: cluster 2024-04-26T16:14:54.696700+0000 mgr.y (mgr.14152) 253 : cluster [DBG] pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: audit 2024-04-26T16:14:55.774992+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: audit 2024-04-26T16:14:55.774992+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: audit 2024-04-26T16:14:55.775890+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: audit 2024-04-26T16:14:55.775890+0000 mon.a (mon.0) 548 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:14:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: audit 2024-04-26T16:14:55.784708+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:55 smithi050 bash[29234]: audit 2024-04-26T16:14:55.784708+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:14:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:57 smithi190 bash[23868]: cluster 2024-04-26T16:14:56.697254+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:57 smithi190 bash[23868]: cluster 2024-04-26T16:14:56.697254+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:57 smithi050 bash[21463]: cluster 2024-04-26T16:14:56.697254+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:57 smithi050 bash[21463]: cluster 2024-04-26T16:14:56.697254+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:57 smithi050 bash[29234]: cluster 2024-04-26T16:14:56.697254+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:14:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:57 smithi050 bash[29234]: cluster 2024-04-26T16:14:56.697254+0000 mgr.y (mgr.14152) 254 : cluster [DBG] pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.697810+0000 mgr.y (mgr.14152) 255 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.697810+0000 mgr.y (mgr.14152) 255 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.892674+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.892674+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.892766+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.892766+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.936741+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-04-26T16:15:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: cluster 2024-04-26T16:14:58.936741+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-04-26T16:15:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: audit 2024-04-26T16:14:58.937023+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:14:59 smithi190 bash[23868]: audit 2024-04-26T16:14:58.937023+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:00.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.697810+0000 mgr.y (mgr.14152) 255 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.697810+0000 mgr.y (mgr.14152) 255 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.892674+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.892674+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.892766+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.892766+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:00.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.936741+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-04-26T16:15:00.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: cluster 2024-04-26T16:14:58.936741+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-04-26T16:15:00.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: audit 2024-04-26T16:14:58.937023+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:00.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[21463]: audit 2024-04-26T16:14:58.937023+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:00.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.697810+0000 mgr.y (mgr.14152) 255 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.697810+0000 mgr.y (mgr.14152) 255 : cluster [DBG] pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:00.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.892674+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:00.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.892674+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:00.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.892766+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:00.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.892766+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:00.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.936741+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-04-26T16:15:00.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: cluster 2024-04-26T16:14:58.936741+0000 mon.a (mon.0) 550 : cluster [DBG] osdmap e43: 6 total, 5 up, 6 in 2024-04-26T16:15:00.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: audit 2024-04-26T16:14:58.937023+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:00.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:14:59 smithi050 bash[29234]: audit 2024-04-26T16:14:58.937023+0000 mon.a (mon.0) 551 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:00.871 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:00.875+0000 7f41b0bcd700 10 client.24229.objecter tick 2024-04-26T16:15:02.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.695253+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:02.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.695253+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:02.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.695350+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.695350+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.698374+0000 mgr.y (mgr.14152) 256 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.698374+0000 mgr.y (mgr.14152) 256 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.948232+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: cluster 2024-04-26T16:15:00.948232+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:00.948551+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:00.948551+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.121876+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.121876+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.131223+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.131223+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.133361+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.133361+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:02.307 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.135254+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:02.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.135254+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:02.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.145392+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:01 smithi190 bash[23868]: audit 2024-04-26T16:15:01.145392+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.695253+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.695253+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.695350+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.695350+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.698374+0000 mgr.y (mgr.14152) 256 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.698374+0000 mgr.y (mgr.14152) 256 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.948232+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: cluster 2024-04-26T16:15:00.948232+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:00.948551+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:00.948551+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.121876+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.121876+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.131223+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.131223+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.133361+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.133361+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.135254+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.135254+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.145392+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[21463]: audit 2024-04-26T16:15:01.145392+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.695253+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:02.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.695253+0000 osd.1 (osd.1) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.695350+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.695350+0000 osd.1 (osd.1) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.698374+0000 mgr.y (mgr.14152) 256 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.698374+0000 mgr.y (mgr.14152) 256 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.948232+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: cluster 2024-04-26T16:15:00.948232+0000 mon.a (mon.0) 552 : cluster [DBG] osdmap e44: 6 total, 5 up, 6 in 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:00.948551+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:00.948551+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.121876+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.121876+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.131223+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.131223+0000 mon.a (mon.0) 555 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.133361+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.133361+0000 mon.a (mon.0) 556 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.135254+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.135254+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.145392+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:01 smithi050 bash[29234]: audit 2024-04-26T16:15:01.145392+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:03.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:02 smithi190 bash[23868]: audit 2024-04-26T16:15:02.315020+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:02 smithi190 bash[23868]: audit 2024-04-26T16:15:02.315020+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:02 smithi190 bash[23868]: audit 2024-04-26T16:15:02.319996+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:02 smithi190 bash[23868]: audit 2024-04-26T16:15:02.319996+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[21463]: audit 2024-04-26T16:15:02.315020+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[21463]: audit 2024-04-26T16:15:02.315020+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[21463]: audit 2024-04-26T16:15:02.319996+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[21463]: audit 2024-04-26T16:15:02.319996+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[29234]: audit 2024-04-26T16:15:02.315020+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[29234]: audit 2024-04-26T16:15:02.315020+0000 mon.b (mon.1) 16 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[29234]: audit 2024-04-26T16:15:02.319996+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:03.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:02 smithi050 bash[29234]: audit 2024-04-26T16:15:02.319996+0000 mon.a (mon.0) 559 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-26T16:15:04.301 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.303+0000 7f41a3fff700 1 -- 172.21.15.190:0/1983239977 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f41ac063860 con 0x7f4198071e90 2024-04-26T16:15:04.301 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-04-26T16:15:04.304 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 10 librados: watch_flush enter 2024-04-26T16:15:04.304 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 10 librados: watch_flush exit 2024-04-26T16:15:04.304 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 20 client.24229.objecter shutdown clearing up homeless session... 2024-04-26T16:15:04.304 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 10 client.24229.objecter successfully canceled tick 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f4198071e90 msgr2=0x7f4198074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f4198071e90 0x7f4198074350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f41a4007720 tx=0x7f41a40076b0 comp rx=0 tx=0).stop 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 msgr2=0x7f41ac1991d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac1991d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4194009990 tx=0x7f4194009f80 comp rx=0 tx=0).stop 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 shutdown_connections 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f41ac105030 0x7f41ac1a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f41ac103490 0x7f41ac198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f4198071e90 0x7f4198074350 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 --2- 172.21.15.190:0/1983239977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41ac104690 0x7f41ac1991d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:04.305 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 >> 172.21.15.190:0/1983239977 conn(0x7f41ac0fe9f0 msgr2=0x7f41ac1078c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:15:04.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 shutdown_connections 2024-04-26T16:15:04.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 -- 172.21.15.190:0/1983239977 wait complete. 2024-04-26T16:15:04.306 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:04.307+0000 7f41b13ce700 1 librados: shutdown 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: cluster 2024-04-26T16:15:02.698921+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: cluster 2024-04-26T16:15:02.698921+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.951985+0000 mon.a (mon.0) 560 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.951985+0000 mon.a (mon.0) 560 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.955262+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.955262+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: cluster 2024-04-26T16:15:02.959567+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e45: 6 total, 5 up, 6 in 2024-04-26T16:15:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: cluster 2024-04-26T16:15:02.959567+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e45: 6 total, 5 up, 6 in 2024-04-26T16:15:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.960633+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.960633+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.961562+0000 mon.a (mon.0) 563 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:03 smithi190 bash[23868]: audit 2024-04-26T16:15:02.961562+0000 mon.a (mon.0) 563 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: cluster 2024-04-26T16:15:02.698921+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: cluster 2024-04-26T16:15:02.698921+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.951985+0000 mon.a (mon.0) 560 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T16:15:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.951985+0000 mon.a (mon.0) 560 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T16:15:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.955262+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.955262+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: cluster 2024-04-26T16:15:02.959567+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e45: 6 total, 5 up, 6 in 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: cluster 2024-04-26T16:15:02.959567+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e45: 6 total, 5 up, 6 in 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.960633+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.960633+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.961562+0000 mon.a (mon.0) 563 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[21463]: audit 2024-04-26T16:15:02.961562+0000 mon.a (mon.0) 563 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: cluster 2024-04-26T16:15:02.698921+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: cluster 2024-04-26T16:15:02.698921+0000 mgr.y (mgr.14152) 257 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.951985+0000 mon.a (mon.0) 560 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T16:15:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.951985+0000 mon.a (mon.0) 560 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.955262+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.955262+0000 mon.b (mon.1) 17 : audit [INF] from='osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: cluster 2024-04-26T16:15:02.959567+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e45: 6 total, 5 up, 6 in 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: cluster 2024-04-26T16:15:02.959567+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e45: 6 total, 5 up, 6 in 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.960633+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.960633+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.961562+0000 mon.a (mon.0) 563 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:03 smithi050 bash[29234]: audit 2024-04-26T16:15:02.961562+0000 mon.a (mon.0) 563 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:04.921 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.5.service 2024-04-26T16:15:04.923 INFO:tasks.cephadm:Deploying osd.6 on smithi190 with /dev/vg_nvme/lv_2... 2024-04-26T16:15:04.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:03.954591+0000 mon.a (mon.0) 564 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:03.954591+0000 mon.a (mon.0) 564 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: cluster 2024-04-26T16:15:03.961122+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e46: 6 total, 5 up, 6 in 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: cluster 2024-04-26T16:15:03.961122+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e46: 6 total, 5 up, 6 in 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:03.962292+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:03.962292+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:03.972457+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:03.972457+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:04.275336+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:04.275336+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:04.284781+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:04.284781+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:04.290869+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: audit 2024-04-26T16:15:04.290869+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: cluster 2024-04-26T16:15:04.964782+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] boot 2024-04-26T16:15:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:04 smithi190 bash[23868]: cluster 2024-04-26T16:15:04.964782+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] boot 2024-04-26T16:15:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:03.954591+0000 mon.a (mon.0) 564 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:03.954591+0000 mon.a (mon.0) 564 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: cluster 2024-04-26T16:15:03.961122+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e46: 6 total, 5 up, 6 in 2024-04-26T16:15:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: cluster 2024-04-26T16:15:03.961122+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e46: 6 total, 5 up, 6 in 2024-04-26T16:15:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:03.962292+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:03.962292+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:03.972457+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:03.972457+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:04.275336+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:04.275336+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:04.284781+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:04.284781+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:04.290869+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: audit 2024-04-26T16:15:04.290869+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: cluster 2024-04-26T16:15:04.964782+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] boot 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[21463]: cluster 2024-04-26T16:15:04.964782+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] boot 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:03.954591+0000 mon.a (mon.0) 564 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:03.954591+0000 mon.a (mon.0) 564 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: cluster 2024-04-26T16:15:03.961122+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e46: 6 total, 5 up, 6 in 2024-04-26T16:15:05.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: cluster 2024-04-26T16:15:03.961122+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e46: 6 total, 5 up, 6 in 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:03.962292+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:03.962292+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:03.972457+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:03.972457+0000 mon.a (mon.0) 567 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:04.275336+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:04.275336+0000 mon.a (mon.0) 568 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:04.284781+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:04.284781+0000 mon.a (mon.0) 569 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:04.290869+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: audit 2024-04-26T16:15:04.290869+0000 mon.a (mon.0) 570 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: cluster 2024-04-26T16:15:04.964782+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] boot 2024-04-26T16:15:05.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:04 smithi050 bash[29234]: cluster 2024-04-26T16:15:04.964782+0000 mon.a (mon.0) 571 : cluster [INF] osd.5 [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] boot 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:03.290010+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:03.290010+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:03.290103+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:03.290103+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:04.699463+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:04.699463+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:04.964907+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-26T16:15:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:04.964907+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-26T16:15:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: audit 2024-04-26T16:15:04.965243+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: audit 2024-04-26T16:15:04.965243+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:05.839511+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-26T16:15:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:05 smithi190 bash[23868]: cluster 2024-04-26T16:15:05.839511+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:03.290010+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:03.290010+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:03.290103+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:03.290103+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:04.699463+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:04.699463+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:04.964907+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-26T16:15:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:04.964907+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: audit 2024-04-26T16:15:04.965243+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: audit 2024-04-26T16:15:04.965243+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:05.839511+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[21463]: cluster 2024-04-26T16:15:05.839511+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:03.290010+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:03.290010+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:03.290103+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:03.290103+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:04.699463+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:04.699463+0000 mgr.y (mgr.14152) 258 : cluster [DBG] pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-04-26T16:15:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:04.964907+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-26T16:15:06.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:04.964907+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e47: 6 total, 6 up, 6 in 2024-04-26T16:15:06.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: audit 2024-04-26T16:15:04.965243+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:06.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: audit 2024-04-26T16:15:04.965243+0000 mon.a (mon.0) 573 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:15:06.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:05.839511+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-26T16:15:06.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:05 smithi050 bash[29234]: cluster 2024-04-26T16:15:05.839511+0000 mon.a (mon.0) 574 : cluster [DBG] osdmap e48: 6 total, 6 up, 6 in 2024-04-26T16:15:08.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[21463]: cluster 2024-04-26T16:15:06.700068+0000 mgr.y (mgr.14152) 259 : cluster [DBG] pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:08.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[21463]: cluster 2024-04-26T16:15:06.700068+0000 mgr.y (mgr.14152) 259 : cluster [DBG] pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:08.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[21463]: cluster 2024-04-26T16:15:06.849803+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-26T16:15:08.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[21463]: cluster 2024-04-26T16:15:06.849803+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-26T16:15:08.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[21463]: cluster 2024-04-26T16:15:06.970918+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[21463]: cluster 2024-04-26T16:15:06.970918+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[29234]: cluster 2024-04-26T16:15:06.700068+0000 mgr.y (mgr.14152) 259 : cluster [DBG] pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[29234]: cluster 2024-04-26T16:15:06.700068+0000 mgr.y (mgr.14152) 259 : cluster [DBG] pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[29234]: cluster 2024-04-26T16:15:06.849803+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[29234]: cluster 2024-04-26T16:15:06.849803+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[29234]: cluster 2024-04-26T16:15:06.970918+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:15:08.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:07 smithi050 bash[29234]: cluster 2024-04-26T16:15:06.970918+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:15:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:07 smithi190 bash[23868]: cluster 2024-04-26T16:15:06.700068+0000 mgr.y (mgr.14152) 259 : cluster [DBG] pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:07 smithi190 bash[23868]: cluster 2024-04-26T16:15:06.700068+0000 mgr.y (mgr.14152) 259 : cluster [DBG] pgmap v244: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:07 smithi190 bash[23868]: cluster 2024-04-26T16:15:06.849803+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-26T16:15:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:07 smithi190 bash[23868]: cluster 2024-04-26T16:15:06.849803+0000 mon.a (mon.0) 575 : cluster [DBG] osdmap e49: 6 total, 6 up, 6 in 2024-04-26T16:15:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:07 smithi190 bash[23868]: cluster 2024-04-26T16:15:06.970918+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:15:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:07 smithi190 bash[23868]: cluster 2024-04-26T16:15:06.970918+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:15:09.246 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:15:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:08 smithi190 bash[23868]: cluster 2024-04-26T16:15:07.981250+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-04-26T16:15:09.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:08 smithi190 bash[23868]: cluster 2024-04-26T16:15:07.981250+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-04-26T16:15:09.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:08 smithi050 bash[21463]: cluster 2024-04-26T16:15:07.981250+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-04-26T16:15:09.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:08 smithi050 bash[21463]: cluster 2024-04-26T16:15:07.981250+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-04-26T16:15:09.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:08 smithi050 bash[29234]: cluster 2024-04-26T16:15:07.981250+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-04-26T16:15:09.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:08 smithi050 bash[29234]: cluster 2024-04-26T16:15:07.981250+0000 mon.a (mon.0) 577 : cluster [DBG] osdmap e50: 6 total, 6 up, 6 in 2024-04-26T16:15:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:09 smithi190 bash[23868]: cluster 2024-04-26T16:15:08.700671+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:09 smithi190 bash[23868]: cluster 2024-04-26T16:15:08.700671+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:09 smithi190 bash[23868]: cluster 2024-04-26T16:15:08.991780+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-04-26T16:15:10.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:09 smithi190 bash[23868]: cluster 2024-04-26T16:15:08.991780+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-04-26T16:15:10.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[21463]: cluster 2024-04-26T16:15:08.700671+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:10.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[21463]: cluster 2024-04-26T16:15:08.700671+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:10.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[21463]: cluster 2024-04-26T16:15:08.991780+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-04-26T16:15:10.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[21463]: cluster 2024-04-26T16:15:08.991780+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-04-26T16:15:10.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[29234]: cluster 2024-04-26T16:15:08.700671+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:10.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[29234]: cluster 2024-04-26T16:15:08.700671+0000 mgr.y (mgr.14152) 260 : cluster [DBG] pgmap v247: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:10.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[29234]: cluster 2024-04-26T16:15:08.991780+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-04-26T16:15:10.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:09 smithi050 bash[29234]: cluster 2024-04-26T16:15:08.991780+0000 mon.a (mon.0) 578 : cluster [DBG] osdmap e51: 6 total, 6 up, 6 in 2024-04-26T16:15:11.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:11 smithi190 bash[23868]: cluster 2024-04-26T16:15:10.701231+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:11 smithi190 bash[23868]: cluster 2024-04-26T16:15:10.701231+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:11 smithi190 bash[23868]: cluster 2024-04-26T16:15:10.986645+0000 mon.a (mon.0) 579 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T16:15:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:11 smithi190 bash[23868]: cluster 2024-04-26T16:15:10.986645+0000 mon.a (mon.0) 579 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T16:15:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:11 smithi190 bash[23868]: cluster 2024-04-26T16:15:10.986700+0000 mon.a (mon.0) 580 : cluster [INF] Cluster is now healthy 2024-04-26T16:15:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:11 smithi190 bash[23868]: cluster 2024-04-26T16:15:10.986700+0000 mon.a (mon.0) 580 : cluster [INF] Cluster is now healthy 2024-04-26T16:15:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[21463]: cluster 2024-04-26T16:15:10.701231+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[21463]: cluster 2024-04-26T16:15:10.701231+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[21463]: cluster 2024-04-26T16:15:10.986645+0000 mon.a (mon.0) 579 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T16:15:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[21463]: cluster 2024-04-26T16:15:10.986645+0000 mon.a (mon.0) 579 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T16:15:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[21463]: cluster 2024-04-26T16:15:10.986700+0000 mon.a (mon.0) 580 : cluster [INF] Cluster is now healthy 2024-04-26T16:15:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[21463]: cluster 2024-04-26T16:15:10.986700+0000 mon.a (mon.0) 580 : cluster [INF] Cluster is now healthy 2024-04-26T16:15:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[29234]: cluster 2024-04-26T16:15:10.701231+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[29234]: cluster 2024-04-26T16:15:10.701231+0000 mgr.y (mgr.14152) 261 : cluster [DBG] pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[29234]: cluster 2024-04-26T16:15:10.986645+0000 mon.a (mon.0) 579 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T16:15:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[29234]: cluster 2024-04-26T16:15:10.986645+0000 mon.a (mon.0) 579 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-26T16:15:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[29234]: cluster 2024-04-26T16:15:10.986700+0000 mon.a (mon.0) 580 : cluster [INF] Cluster is now healthy 2024-04-26T16:15:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:11 smithi050 bash[29234]: cluster 2024-04-26T16:15:10.986700+0000 mon.a (mon.0) 580 : cluster [INF] Cluster is now healthy 2024-04-26T16:15:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-26T16:15:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-26T16:15:13.102 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-26T16:15:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-26T16:15:13.102 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013834 s, 758 MB/s 2024-04-26T16:15:13.102 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-26T16:15:13.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-04-26T16:15:14.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:13 smithi190 bash[23868]: cluster 2024-04-26T16:15:12.701822+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:15:14.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:13 smithi190 bash[23868]: cluster 2024-04-26T16:15:12.701822+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:15:14.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:13 smithi050 bash[21463]: cluster 2024-04-26T16:15:12.701822+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:15:14.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:13 smithi050 bash[21463]: cluster 2024-04-26T16:15:12.701822+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:15:14.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:13 smithi050 bash[29234]: cluster 2024-04-26T16:15:12.701822+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:15:14.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:13 smithi050 bash[29234]: cluster 2024-04-26T16:15:12.701822+0000 mgr.y (mgr.14152) 262 : cluster [DBG] pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:15:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: cephadm 2024-04-26T16:15:14.288254+0000 mgr.y (mgr.14152) 263 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:15:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: cephadm 2024-04-26T16:15:14.288254+0000 mgr.y (mgr.14152) 263 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:15:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.297463+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.297463+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.307002+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.307002+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.309281+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.309281+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.311266+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.311266+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: cephadm 2024-04-26T16:15:14.312713+0000 mgr.y (mgr.14152) 264 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 8632M 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: cephadm 2024-04-26T16:15:14.312713+0000 mgr.y (mgr.14152) 264 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 8632M 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.318976+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.318976+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.323395+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.323395+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.325205+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:15.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.325205+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.335024+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: audit 2024-04-26T16:15:14.335024+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: cluster 2024-04-26T16:15:14.702488+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:15:15.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:15 smithi190 bash[23868]: cluster 2024-04-26T16:15:14.702488+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:15:15.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: cephadm 2024-04-26T16:15:14.288254+0000 mgr.y (mgr.14152) 263 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: cephadm 2024-04-26T16:15:14.288254+0000 mgr.y (mgr.14152) 263 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.297463+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.297463+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.307002+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.307002+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.309281+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.309281+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.311266+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.311266+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: cephadm 2024-04-26T16:15:14.312713+0000 mgr.y (mgr.14152) 264 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 8632M 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: cephadm 2024-04-26T16:15:14.312713+0000 mgr.y (mgr.14152) 264 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 8632M 2024-04-26T16:15:15.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.318976+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.318976+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.323395+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.323395+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.325205+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.325205+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.335024+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: audit 2024-04-26T16:15:14.335024+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: cluster 2024-04-26T16:15:14.702488+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[21463]: cluster 2024-04-26T16:15:14.702488+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: cephadm 2024-04-26T16:15:14.288254+0000 mgr.y (mgr.14152) 263 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: cephadm 2024-04-26T16:15:14.288254+0000 mgr.y (mgr.14152) 263 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:15:15.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.297463+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.297463+0000 mon.a (mon.0) 581 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.307002+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.307002+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.309281+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.309281+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.311266+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.311266+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: cephadm 2024-04-26T16:15:14.312713+0000 mgr.y (mgr.14152) 264 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 8632M 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: cephadm 2024-04-26T16:15:14.312713+0000 mgr.y (mgr.14152) 264 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 8632M 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.318976+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.318976+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.323395+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.323395+0000 mon.a (mon.0) 586 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:15.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.325205+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.325205+0000 mon.a (mon.0) 587 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.335024+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: audit 2024-04-26T16:15:14.335024+0000 mon.a (mon.0) 588 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: cluster 2024-04-26T16:15:14.702488+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:15:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:15 smithi050 bash[29234]: cluster 2024-04-26T16:15:14.702488+0000 mgr.y (mgr.14152) 265 : cluster [DBG] pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:17 smithi190 bash[23868]: cluster 2024-04-26T16:15:16.703144+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:17 smithi190 bash[23868]: cluster 2024-04-26T16:15:16.703144+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:17 smithi050 bash[21463]: cluster 2024-04-26T16:15:16.703144+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:17 smithi050 bash[21463]: cluster 2024-04-26T16:15:16.703144+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:17 smithi050 bash[29234]: cluster 2024-04-26T16:15:16.703144+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:17 smithi050 bash[29234]: cluster 2024-04-26T16:15:16.703144+0000 mgr.y (mgr.14152) 266 : cluster [DBG] pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-04-26T16:15:18.587 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:15:20.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:19 smithi190 bash[23868]: cluster 2024-04-26T16:15:18.703661+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:15:20.066 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:19 smithi190 bash[23868]: cluster 2024-04-26T16:15:18.703661+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:15:20.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:19 smithi050 bash[21463]: cluster 2024-04-26T16:15:18.703661+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:15:20.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:19 smithi050 bash[21463]: cluster 2024-04-26T16:15:18.703661+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:15:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:19 smithi050 bash[29234]: cluster 2024-04-26T16:15:18.703661+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:15:20.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:19 smithi050 bash[29234]: cluster 2024-04-26T16:15:18.703661+0000 mgr.y (mgr.14152) 267 : cluster [DBG] pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- 172.21.15.190:0/4265457142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc106830 msgr2=0x7f8acc108c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- 172.21.15.190:0/4265457142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc106830 0x7f8acc108c20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0009990 tx=0x7f8ac0009ca0 comp rx=0 tx=0).stop 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- 172.21.15.190:0/4265457142 shutdown_connections 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- 172.21.15.190:0/4265457142 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8acc109160 0x7f8acc10b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- 172.21.15.190:0/4265457142 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8acc103f00 0x7f8acc1062f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- 172.21.15.190:0/4265457142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc106830 0x7f8acc108c20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- 172.21.15.190:0/4265457142 >> 172.21.15.190:0/4265457142 conn(0x7f8acc0fd8f0 msgr2=0x7f8acc0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:15:20.936 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- 172.21.15.190:0/4265457142 shutdown_connections 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- 172.21.15.190:0/4265457142 wait complete. 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 librados: starting msgr at 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 librados: starting objecter 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 Processor -- start 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- start start 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 librados: setting wanted keys 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 librados: calling monclient init 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 0x7f8acc198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:15:20.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8acc106830 0x7f8acc1991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8acc109160 0x7f8acc1a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc10e4a0 con 0x7f8acc106830 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc10e320 con 0x7f8acc103f00 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad341f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8acc10e620 con 0x7f8acc109160 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad11bb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 0x7f8acc198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad11bb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 0x7f8acc198cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:36936/0 (socket says 172.21.15.190:36936) 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad11bb700 1 -- 172.21.15.190:0/3915291403 learned_addr learned my addr 172.21.15.190:0/3915291403 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad09ba700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8acc106830 0x7f8acc1991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.935+0000 7f8ad19bc700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8acc109160 0x7f8acc1a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad11bb700 1 -- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8acc109160 msgr2=0x7f8acc1a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:20.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad11bb700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8acc109160 0x7f8acc1a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad11bb700 1 -- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8acc106830 msgr2=0x7f8acc1991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad11bb700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8acc106830 0x7f8acc1991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad11bb700 1 -- 172.21.15.190:0/3915291403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8acc1a0930 con 0x7f8acc103f00 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad11bb700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 0x7f8acc198cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400ba50 tx=0x7f8ac400be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 10 client.?.objecter ms_handle_connect 0x7f8acc103f00 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4019070 con 0x7f8acc103f00 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8ac4014440 con 0x7f8acc103f00 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8acc1a0c20 con 0x7f8acc103f00 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4014e60 con 0x7f8acc103f00 2024-04-26T16:15:20.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad341f700 10 client.24256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:15:20.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8acc1a1160 con 0x7f8acc103f00 2024-04-26T16:15:20.940 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad341f700 1 librados: init done 2024-04-26T16:15:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8acc06a110 con 0x7f8acc103f00 2024-04-26T16:15:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f8ac401c070 con 0x7f8acc103f00 2024-04-26T16:15:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f8ab4071e90 0x7f8ab4074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:15:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4423+0+0 (secure 0 0 0) 0x7f8ac408f960 con 0x7f8acc103f00 2024-04-26T16:15:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 10 client.24256.objecter ms_dispatch 0x7f8acc001550 osd_map(51..51 src has 1..51) 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 3 client.24256.objecter handle_osd_map got epochs [51,51] > 0 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 3 client.24256.objecter handle_osd_map decoding full epoch 51 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8aca7fc700 20 client.24256.objecter dump_active .. 0 homeless 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.939+0000 7f8ad09ba700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f8ab4071e90 0x7f8ab4074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.943+0000 7f8aca7fc700 10 client.24256.objecter ms_handle_connect 0x7f8ab4071e90 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.943+0000 7f8ad09ba700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f8ab4071e90 0x7f8ab4074350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000b5c0 tx=0x7f8ac001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:15:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:20.943+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f8ac40145f0 con 0x7f8acc103f00 2024-04-26T16:15:21.168 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:21.171+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f8acc063860 con 0x7f8ab4071e90 2024-04-26T16:15:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: cluster 2024-04-26T16:15:20.704349+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: cluster 2024-04-26T16:15:20.704349+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: audit 2024-04-26T16:15:21.187517+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: audit 2024-04-26T16:15:21.187517+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: audit 2024-04-26T16:15:21.194112+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: audit 2024-04-26T16:15:21.194112+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: audit 2024-04-26T16:15:21.195813+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:21 smithi190 bash[23868]: audit 2024-04-26T16:15:21.195813+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: cluster 2024-04-26T16:15:20.704349+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: cluster 2024-04-26T16:15:20.704349+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: audit 2024-04-26T16:15:21.187517+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: audit 2024-04-26T16:15:21.187517+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: audit 2024-04-26T16:15:21.194112+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: audit 2024-04-26T16:15:21.194112+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: audit 2024-04-26T16:15:21.195813+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:22.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[21463]: audit 2024-04-26T16:15:21.195813+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: cluster 2024-04-26T16:15:20.704349+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: cluster 2024-04-26T16:15:20.704349+0000 mgr.y (mgr.14152) 268 : cluster [DBG] pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: audit 2024-04-26T16:15:21.187517+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: audit 2024-04-26T16:15:21.187517+0000 mon.a (mon.0) 589 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: audit 2024-04-26T16:15:21.194112+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: audit 2024-04-26T16:15:21.194112+0000 mon.a (mon.0) 590 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: audit 2024-04-26T16:15:21.195813+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:22.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:21 smithi050 bash[29234]: audit 2024-04-26T16:15:21.195813+0000 mon.a (mon.0) 591 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:23.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:22 smithi190 bash[23868]: audit 2024-04-26T16:15:21.180326+0000 mgr.y (mgr.14152) 269 : audit [DBG] from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:15:23.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:22 smithi190 bash[23868]: audit 2024-04-26T16:15:21.180326+0000 mgr.y (mgr.14152) 269 : audit [DBG] from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:15:23.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:22 smithi050 bash[21463]: audit 2024-04-26T16:15:21.180326+0000 mgr.y (mgr.14152) 269 : audit [DBG] from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:15:23.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:22 smithi050 bash[21463]: audit 2024-04-26T16:15:21.180326+0000 mgr.y (mgr.14152) 269 : audit [DBG] from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:15:23.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:22 smithi050 bash[29234]: audit 2024-04-26T16:15:21.180326+0000 mgr.y (mgr.14152) 269 : audit [DBG] from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:15:23.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:22 smithi050 bash[29234]: audit 2024-04-26T16:15:21.180326+0000 mgr.y (mgr.14152) 269 : audit [DBG] from='client.24256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:15:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:23 smithi190 bash[23868]: cluster 2024-04-26T16:15:22.704937+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:23 smithi190 bash[23868]: cluster 2024-04-26T16:15:22.704937+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:24.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:23 smithi050 bash[21463]: cluster 2024-04-26T16:15:22.704937+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:24.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:23 smithi050 bash[21463]: cluster 2024-04-26T16:15:22.704937+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:23 smithi050 bash[29234]: cluster 2024-04-26T16:15:22.704937+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:23 smithi050 bash[29234]: cluster 2024-04-26T16:15:22.704937+0000 mgr.y (mgr.14152) 270 : cluster [DBG] pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:25.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:25.943+0000 7f8ad2c1e700 10 client.24256.objecter tick 2024-04-26T16:15:26.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:25 smithi190 bash[23868]: cluster 2024-04-26T16:15:24.705492+0000 mgr.y (mgr.14152) 271 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:26.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:25 smithi190 bash[23868]: cluster 2024-04-26T16:15:24.705492+0000 mgr.y (mgr.14152) 271 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:26.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:25 smithi050 bash[29234]: cluster 2024-04-26T16:15:24.705492+0000 mgr.y (mgr.14152) 271 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:26.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:25 smithi050 bash[29234]: cluster 2024-04-26T16:15:24.705492+0000 mgr.y (mgr.14152) 271 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:26.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:25 smithi050 bash[21463]: cluster 2024-04-26T16:15:24.705492+0000 mgr.y (mgr.14152) 271 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:26.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:25 smithi050 bash[21463]: cluster 2024-04-26T16:15:24.705492+0000 mgr.y (mgr.14152) 271 : cluster [DBG] pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:28.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:27 smithi190 bash[23868]: cluster 2024-04-26T16:15:26.706039+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:28.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:27 smithi190 bash[23868]: cluster 2024-04-26T16:15:26.706039+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:28.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:27 smithi050 bash[21463]: cluster 2024-04-26T16:15:26.706039+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:28.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:27 smithi050 bash[21463]: cluster 2024-04-26T16:15:26.706039+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:28.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:27 smithi050 bash[29234]: cluster 2024-04-26T16:15:26.706039+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:28.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:27 smithi050 bash[29234]: cluster 2024-04-26T16:15:26.706039+0000 mgr.y (mgr.14152) 272 : cluster [DBG] pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: cluster 2024-04-26T16:15:28.706596+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: cluster 2024-04-26T16:15:28.706596+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.270327+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.190:0/3106325450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.270327+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.190:0/3106325450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.275350+0000 mon.a (mon.0) 592 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.275350+0000 mon.a (mon.0) 592 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.278849+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]': finished 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.278849+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]': finished 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: cluster 2024-04-26T16:15:29.284757+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: cluster 2024-04-26T16:15:29.284757+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.285277+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:30.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[21463]: audit 2024-04-26T16:15:29.285277+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: cluster 2024-04-26T16:15:28.706596+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: cluster 2024-04-26T16:15:28.706596+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.270327+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.190:0/3106325450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.270327+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.190:0/3106325450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.275350+0000 mon.a (mon.0) 592 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.275350+0000 mon.a (mon.0) 592 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.278849+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]': finished 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.278849+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]': finished 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: cluster 2024-04-26T16:15:29.284757+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: cluster 2024-04-26T16:15:29.284757+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.285277+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:30.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:29 smithi050 bash[29234]: audit 2024-04-26T16:15:29.285277+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: cluster 2024-04-26T16:15:28.706596+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: cluster 2024-04-26T16:15:28.706596+0000 mgr.y (mgr.14152) 273 : cluster [DBG] pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.270327+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.190:0/3106325450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.270327+0000 mon.b (mon.1) 18 : audit [INF] from='client.? 172.21.15.190:0/3106325450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.275350+0000 mon.a (mon.0) 592 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.275350+0000 mon.a (mon.0) 592 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]: dispatch 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.278849+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]': finished 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.278849+0000 mon.a (mon.0) 593 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94111b7d-7887-4a3a-983b-394294735d0e"}]': finished 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: cluster 2024-04-26T16:15:29.284757+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: cluster 2024-04-26T16:15:29.284757+0000 mon.a (mon.0) 594 : cluster [DBG] osdmap e52: 7 total, 6 up, 7 in 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.285277+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:29 smithi190 bash[23868]: audit 2024-04-26T16:15:29.285277+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:30.943 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:30.947+0000 7f8ad2c1e700 10 client.24256.objecter tick 2024-04-26T16:15:31.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[21463]: cluster 2024-04-26T16:15:29.982094+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[21463]: cluster 2024-04-26T16:15:29.982094+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[21463]: cluster 2024-04-26T16:15:29.982182+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[21463]: cluster 2024-04-26T16:15:29.982182+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[21463]: audit 2024-04-26T16:15:29.990159+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? 172.21.15.190:0/2392250025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[21463]: audit 2024-04-26T16:15:29.990159+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? 172.21.15.190:0/2392250025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[29234]: cluster 2024-04-26T16:15:29.982094+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[29234]: cluster 2024-04-26T16:15:29.982094+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[29234]: cluster 2024-04-26T16:15:29.982182+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[29234]: cluster 2024-04-26T16:15:29.982182+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:31.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[29234]: audit 2024-04-26T16:15:29.990159+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? 172.21.15.190:0/2392250025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:15:31.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:30 smithi050 bash[29234]: audit 2024-04-26T16:15:29.990159+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? 172.21.15.190:0/2392250025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:15:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:30 smithi190 bash[23868]: cluster 2024-04-26T16:15:29.982094+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:30 smithi190 bash[23868]: cluster 2024-04-26T16:15:29.982094+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:30 smithi190 bash[23868]: cluster 2024-04-26T16:15:29.982182+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:30 smithi190 bash[23868]: cluster 2024-04-26T16:15:29.982182+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:30 smithi190 bash[23868]: audit 2024-04-26T16:15:29.990159+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? 172.21.15.190:0/2392250025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:15:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:30 smithi190 bash[23868]: audit 2024-04-26T16:15:29.990159+0000 mon.b (mon.1) 19 : audit [DBG] from='client.? 172.21.15.190:0/2392250025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:15:32.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[21463]: cluster 2024-04-26T16:15:30.707238+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[21463]: cluster 2024-04-26T16:15:30.707238+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[21463]: cluster 2024-04-26T16:15:30.864849+0000 mon.a (mon.0) 596 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[21463]: cluster 2024-04-26T16:15:30.864849+0000 mon.a (mon.0) 596 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[21463]: audit 2024-04-26T16:15:30.865202+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[21463]: audit 2024-04-26T16:15:30.865202+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[29234]: cluster 2024-04-26T16:15:30.707238+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[29234]: cluster 2024-04-26T16:15:30.707238+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[29234]: cluster 2024-04-26T16:15:30.864849+0000 mon.a (mon.0) 596 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[29234]: cluster 2024-04-26T16:15:30.864849+0000 mon.a (mon.0) 596 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[29234]: audit 2024-04-26T16:15:30.865202+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:31 smithi050 bash[29234]: audit 2024-04-26T16:15:30.865202+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:31 smithi190 bash[23868]: cluster 2024-04-26T16:15:30.707238+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:31 smithi190 bash[23868]: cluster 2024-04-26T16:15:30.707238+0000 mgr.y (mgr.14152) 274 : cluster [DBG] pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:31 smithi190 bash[23868]: cluster 2024-04-26T16:15:30.864849+0000 mon.a (mon.0) 596 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-04-26T16:15:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:31 smithi190 bash[23868]: cluster 2024-04-26T16:15:30.864849+0000 mon.a (mon.0) 596 : cluster [DBG] osdmap e53: 7 total, 6 up, 7 in 2024-04-26T16:15:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:31 smithi190 bash[23868]: audit 2024-04-26T16:15:30.865202+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:31 smithi190 bash[23868]: audit 2024-04-26T16:15:30.865202+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:34.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:33 smithi050 bash[21463]: cluster 2024-04-26T16:15:32.707869+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:34.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:33 smithi050 bash[21463]: cluster 2024-04-26T16:15:32.707869+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:34.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:33 smithi050 bash[29234]: cluster 2024-04-26T16:15:32.707869+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:34.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:33 smithi050 bash[29234]: cluster 2024-04-26T16:15:32.707869+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:34.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:33 smithi190 bash[23868]: cluster 2024-04-26T16:15:32.707869+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:34.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:33 smithi190 bash[23868]: cluster 2024-04-26T16:15:32.707869+0000 mgr.y (mgr.14152) 275 : cluster [DBG] pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:35.947 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:35.951+0000 7f8ad2c1e700 10 client.24256.objecter tick 2024-04-26T16:15:36.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:35 smithi050 bash[21463]: cluster 2024-04-26T16:15:34.708448+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:36.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:35 smithi050 bash[21463]: cluster 2024-04-26T16:15:34.708448+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:36.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:35 smithi050 bash[29234]: cluster 2024-04-26T16:15:34.708448+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:36.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:35 smithi050 bash[29234]: cluster 2024-04-26T16:15:34.708448+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:35 smithi190 bash[23868]: cluster 2024-04-26T16:15:34.708448+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:35 smithi190 bash[23868]: cluster 2024-04-26T16:15:34.708448+0000 mgr.y (mgr.14152) 276 : cluster [DBG] pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:38.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:37 smithi050 bash[29234]: cluster 2024-04-26T16:15:36.709130+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:38.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:37 smithi050 bash[29234]: cluster 2024-04-26T16:15:36.709130+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:38.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:37 smithi050 bash[21463]: cluster 2024-04-26T16:15:36.709130+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:38.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:37 smithi050 bash[21463]: cluster 2024-04-26T16:15:36.709130+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:38.228 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:37 smithi190 bash[23868]: cluster 2024-04-26T16:15:36.709130+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:38.228 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:37 smithi190 bash[23868]: cluster 2024-04-26T16:15:36.709130+0000 mgr.y (mgr.14152) 277 : cluster [DBG] pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:39 smithi050 bash[21463]: cluster 2024-04-26T16:15:38.709841+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:39 smithi050 bash[21463]: cluster 2024-04-26T16:15:38.709841+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:39 smithi050 bash[29234]: cluster 2024-04-26T16:15:38.709841+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:39 smithi050 bash[29234]: cluster 2024-04-26T16:15:38.709841+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:39 smithi190 bash[23868]: cluster 2024-04-26T16:15:38.709841+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:39 smithi190 bash[23868]: cluster 2024-04-26T16:15:38.709841+0000 mgr.y (mgr.14152) 278 : cluster [DBG] pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:40.951 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:40.955+0000 7f8ad2c1e700 10 client.24256.objecter tick 2024-04-26T16:15:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[21463]: cluster 2024-04-26T16:15:40.710544+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[21463]: cluster 2024-04-26T16:15:40.710544+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[21463]: audit 2024-04-26T16:15:41.450142+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[21463]: audit 2024-04-26T16:15:41.450142+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[21463]: audit 2024-04-26T16:15:41.452047+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[21463]: audit 2024-04-26T16:15:41.452047+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[29234]: cluster 2024-04-26T16:15:40.710544+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[29234]: cluster 2024-04-26T16:15:40.710544+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[29234]: audit 2024-04-26T16:15:41.450142+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[29234]: audit 2024-04-26T16:15:41.450142+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[29234]: audit 2024-04-26T16:15:41.452047+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:41 smithi050 bash[29234]: audit 2024-04-26T16:15:41.452047+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:42.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:41 smithi190 bash[23868]: cluster 2024-04-26T16:15:40.710544+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:42.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:41 smithi190 bash[23868]: cluster 2024-04-26T16:15:40.710544+0000 mgr.y (mgr.14152) 279 : cluster [DBG] pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:42.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:41 smithi190 bash[23868]: audit 2024-04-26T16:15:41.450142+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T16:15:42.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:41 smithi190 bash[23868]: audit 2024-04-26T16:15:41.450142+0000 mon.a (mon.0) 598 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-26T16:15:42.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:41 smithi190 bash[23868]: audit 2024-04-26T16:15:41.452047+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:42.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:41 smithi190 bash[23868]: audit 2024-04-26T16:15:41.452047+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:43.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:42 smithi190 bash[23868]: cephadm 2024-04-26T16:15:41.453829+0000 mgr.y (mgr.14152) 280 : cephadm [INF] Deploying daemon osd.6 on smithi190 2024-04-26T16:15:43.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:42 smithi190 bash[23868]: cephadm 2024-04-26T16:15:41.453829+0000 mgr.y (mgr.14152) 280 : cephadm [INF] Deploying daemon osd.6 on smithi190 2024-04-26T16:15:43.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:42 smithi050 bash[21463]: cephadm 2024-04-26T16:15:41.453829+0000 mgr.y (mgr.14152) 280 : cephadm [INF] Deploying daemon osd.6 on smithi190 2024-04-26T16:15:43.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:42 smithi050 bash[21463]: cephadm 2024-04-26T16:15:41.453829+0000 mgr.y (mgr.14152) 280 : cephadm [INF] Deploying daemon osd.6 on smithi190 2024-04-26T16:15:43.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:42 smithi050 bash[29234]: cephadm 2024-04-26T16:15:41.453829+0000 mgr.y (mgr.14152) 280 : cephadm [INF] Deploying daemon osd.6 on smithi190 2024-04-26T16:15:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:42 smithi050 bash[29234]: cephadm 2024-04-26T16:15:41.453829+0000 mgr.y (mgr.14152) 280 : cephadm [INF] Deploying daemon osd.6 on smithi190 2024-04-26T16:15:44.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:43 smithi190 bash[23868]: cluster 2024-04-26T16:15:42.711147+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:44.140 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:43 smithi190 bash[23868]: cluster 2024-04-26T16:15:42.711147+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:43 smithi050 bash[21463]: cluster 2024-04-26T16:15:42.711147+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:43 smithi050 bash[21463]: cluster 2024-04-26T16:15:42.711147+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:44.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:43 smithi050 bash[29234]: cluster 2024-04-26T16:15:42.711147+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:44.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:43 smithi050 bash[29234]: cluster 2024-04-26T16:15:42.711147+0000 mgr.y (mgr.14152) 281 : cluster [DBG] pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:45.430 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.431 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.431 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.706 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.707 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:15:45 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:15:45.955 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:45.959+0000 7f8ad2c1e700 10 client.24256.objecter tick 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: cluster 2024-04-26T16:15:44.711635+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: cluster 2024-04-26T16:15:44.711635+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: audit 2024-04-26T16:15:45.718856+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: audit 2024-04-26T16:15:45.718856+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: audit 2024-04-26T16:15:45.724554+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: audit 2024-04-26T16:15:45.724554+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: audit 2024-04-26T16:15:45.734918+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:45 smithi190 bash[23868]: audit 2024-04-26T16:15:45.734918+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: cluster 2024-04-26T16:15:44.711635+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:46.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: cluster 2024-04-26T16:15:44.711635+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:46.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: audit 2024-04-26T16:15:45.718856+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: audit 2024-04-26T16:15:45.718856+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: audit 2024-04-26T16:15:45.724554+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: audit 2024-04-26T16:15:45.724554+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: audit 2024-04-26T16:15:45.734918+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[21463]: audit 2024-04-26T16:15:45.734918+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: cluster 2024-04-26T16:15:44.711635+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: cluster 2024-04-26T16:15:44.711635+0000 mgr.y (mgr.14152) 282 : cluster [DBG] pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: audit 2024-04-26T16:15:45.718856+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: audit 2024-04-26T16:15:45.718856+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: audit 2024-04-26T16:15:45.724554+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: audit 2024-04-26T16:15:45.724554+0000 mon.a (mon.0) 601 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: audit 2024-04-26T16:15:45.734918+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:46.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:45 smithi050 bash[29234]: audit 2024-04-26T16:15:45.734918+0000 mon.a (mon.0) 602 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:47 smithi190 bash[23868]: cluster 2024-04-26T16:15:46.712207+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:47 smithi190 bash[23868]: cluster 2024-04-26T16:15:46.712207+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:47 smithi190 bash[23868]: cluster 2024-04-26T16:15:47.195424+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:15:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:47 smithi190 bash[23868]: cluster 2024-04-26T16:15:47.195424+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:15:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:47 smithi190 bash[23868]: cluster 2024-04-26T16:15:47.198138+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:15:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:47 smithi190 bash[23868]: cluster 2024-04-26T16:15:47.198138+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:15:48.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[21463]: cluster 2024-04-26T16:15:46.712207+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:48.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[21463]: cluster 2024-04-26T16:15:46.712207+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:48.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[21463]: cluster 2024-04-26T16:15:47.195424+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:15:48.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[21463]: cluster 2024-04-26T16:15:47.195424+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[21463]: cluster 2024-04-26T16:15:47.198138+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[21463]: cluster 2024-04-26T16:15:47.198138+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[29234]: cluster 2024-04-26T16:15:46.712207+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[29234]: cluster 2024-04-26T16:15:46.712207+0000 mgr.y (mgr.14152) 283 : cluster [DBG] pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[29234]: cluster 2024-04-26T16:15:47.195424+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[29234]: cluster 2024-04-26T16:15:47.195424+0000 osd.5 (osd.5) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[29234]: cluster 2024-04-26T16:15:47.198138+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:15:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:47 smithi050 bash[29234]: cluster 2024-04-26T16:15:47.198138+0000 osd.5 (osd.5) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:15:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:49 smithi190 bash[23868]: cluster 2024-04-26T16:15:48.712744+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:49 smithi190 bash[23868]: cluster 2024-04-26T16:15:48.712744+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:49 smithi050 bash[21463]: cluster 2024-04-26T16:15:48.712744+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:49 smithi050 bash[21463]: cluster 2024-04-26T16:15:48.712744+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:49 smithi050 bash[29234]: cluster 2024-04-26T16:15:48.712744+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:49 smithi050 bash[29234]: cluster 2024-04-26T16:15:48.712744+0000 mgr.y (mgr.14152) 284 : cluster [DBG] pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:50.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:50.963+0000 7f8ad2c1e700 10 client.24256.objecter tick 2024-04-26T16:15:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:51 smithi190 bash[23868]: cluster 2024-04-26T16:15:50.713299+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:51 smithi190 bash[23868]: cluster 2024-04-26T16:15:50.713299+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:51 smithi050 bash[21463]: cluster 2024-04-26T16:15:50.713299+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:51 smithi050 bash[21463]: cluster 2024-04-26T16:15:50.713299+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:51 smithi050 bash[29234]: cluster 2024-04-26T16:15:50.713299+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:51 smithi050 bash[29234]: cluster 2024-04-26T16:15:50.713299+0000 mgr.y (mgr.14152) 285 : cluster [DBG] pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: cluster 2024-04-26T16:15:51.180430+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: cluster 2024-04-26T16:15:51.180430+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: cluster 2024-04-26T16:15:51.180549+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: cluster 2024-04-26T16:15:51.180549+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: cluster 2024-04-26T16:15:51.908685+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: cluster 2024-04-26T16:15:51.908685+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:51.909279+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:51.909279+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.112767+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.112767+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.117575+0000 mon.a (mon.0) 605 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.117575+0000 mon.a (mon.0) 605 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.682780+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.682780+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.692821+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:52 smithi190 bash[23868]: audit 2024-04-26T16:15:52.692821+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: cluster 2024-04-26T16:15:51.180430+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: cluster 2024-04-26T16:15:51.180430+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: cluster 2024-04-26T16:15:51.180549+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: cluster 2024-04-26T16:15:51.180549+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: cluster 2024-04-26T16:15:51.908685+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: cluster 2024-04-26T16:15:51.908685+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:51.909279+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:51.909279+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.112767+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.112767+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.117575+0000 mon.a (mon.0) 605 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.117575+0000 mon.a (mon.0) 605 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.682780+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.682780+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.692821+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[21463]: audit 2024-04-26T16:15:52.692821+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: cluster 2024-04-26T16:15:51.180430+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: cluster 2024-04-26T16:15:51.180430+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: cluster 2024-04-26T16:15:51.180549+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: cluster 2024-04-26T16:15:51.180549+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: cluster 2024-04-26T16:15:51.908685+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: cluster 2024-04-26T16:15:51.908685+0000 mon.a (mon.0) 603 : cluster [DBG] osdmap e54: 7 total, 6 up, 7 in 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:51.909279+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:51.909279+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.112767+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.112767+0000 mon.b (mon.1) 20 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.117575+0000 mon.a (mon.0) 605 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.117575+0000 mon.a (mon.0) 605 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.682780+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.682780+0000 mon.a (mon.0) 606 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.692821+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:53.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:52 smithi050 bash[29234]: audit 2024-04-26T16:15:52.692821+0000 mon.a (mon.0) 607 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: cluster 2024-04-26T16:15:52.713788+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: cluster 2024-04-26T16:15:52.713788+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.923018+0000 mon.a (mon.0) 608 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:15:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.923018+0000 mon.a (mon.0) 608 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.927005+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.927005+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: cluster 2024-04-26T16:15:52.928830+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: cluster 2024-04-26T16:15:52.928830+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.929357+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.929357+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.931602+0000 mon.a (mon.0) 611 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:52.931602+0000 mon.a (mon.0) 611 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:53.372094+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:53.372094+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:53.374083+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:53.374083+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:53.383821+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:53 smithi190 bash[23868]: audit 2024-04-26T16:15:53.383821+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: cluster 2024-04-26T16:15:52.713788+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: cluster 2024-04-26T16:15:52.713788+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.923018+0000 mon.a (mon.0) 608 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.923018+0000 mon.a (mon.0) 608 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.927005+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.927005+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: cluster 2024-04-26T16:15:52.928830+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: cluster 2024-04-26T16:15:52.928830+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.929357+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.929357+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.931602+0000 mon.a (mon.0) 611 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:52.931602+0000 mon.a (mon.0) 611 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:53.372094+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:53.372094+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:53.374083+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:53.374083+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:53.383821+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[21463]: audit 2024-04-26T16:15:53.383821+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: cluster 2024-04-26T16:15:52.713788+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: cluster 2024-04-26T16:15:52.713788+0000 mgr.y (mgr.14152) 286 : cluster [DBG] pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.923018+0000 mon.a (mon.0) 608 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.923018+0000 mon.a (mon.0) 608 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.927005+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.927005+0000 mon.b (mon.1) 21 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: cluster 2024-04-26T16:15:52.928830+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: cluster 2024-04-26T16:15:52.928830+0000 mon.a (mon.0) 609 : cluster [DBG] osdmap e55: 7 total, 6 up, 7 in 2024-04-26T16:15:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.929357+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.929357+0000 mon.a (mon.0) 610 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.931602+0000 mon.a (mon.0) 611 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:52.931602+0000 mon.a (mon.0) 611 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:53.372094+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:53.372094+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:53.374083+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:53.374083+0000 mon.a (mon.0) 613 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:53.383821+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:53 smithi050 bash[29234]: audit 2024-04-26T16:15:53.383821+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:55.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: audit 2024-04-26T16:15:53.925232+0000 mon.a (mon.0) 615 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:55.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: audit 2024-04-26T16:15:53.925232+0000 mon.a (mon.0) 615 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:55.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: cluster 2024-04-26T16:15:53.931356+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-04-26T16:15:55.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: cluster 2024-04-26T16:15:53.931356+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-04-26T16:15:55.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: audit 2024-04-26T16:15:53.932269+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: audit 2024-04-26T16:15:53.932269+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: audit 2024-04-26T16:15:53.944040+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:54 smithi190 bash[23868]: audit 2024-04-26T16:15:53.944040+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: audit 2024-04-26T16:15:53.925232+0000 mon.a (mon.0) 615 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: audit 2024-04-26T16:15:53.925232+0000 mon.a (mon.0) 615 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: cluster 2024-04-26T16:15:53.931356+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: cluster 2024-04-26T16:15:53.931356+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: audit 2024-04-26T16:15:53.932269+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: audit 2024-04-26T16:15:53.932269+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: audit 2024-04-26T16:15:53.944040+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[21463]: audit 2024-04-26T16:15:53.944040+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: audit 2024-04-26T16:15:53.925232+0000 mon.a (mon.0) 615 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: audit 2024-04-26T16:15:53.925232+0000 mon.a (mon.0) 615 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: cluster 2024-04-26T16:15:53.931356+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: cluster 2024-04-26T16:15:53.931356+0000 mon.a (mon.0) 616 : cluster [DBG] osdmap e56: 7 total, 6 up, 7 in 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: audit 2024-04-26T16:15:53.932269+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: audit 2024-04-26T16:15:53.932269+0000 mon.a (mon.0) 617 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: audit 2024-04-26T16:15:53.944040+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:54 smithi050 bash[29234]: audit 2024-04-26T16:15:53.944040+0000 mon.a (mon.0) 618 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:55.810 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.811+0000 7f8aca7fc700 1 -- 172.21.15.190:0/3915291403 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8acc063860 con 0x7f8ab4071e90 2024-04-26T16:15:55.811 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 6 on host 'smithi190' 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 10 librados: watch_flush enter 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 10 librados: watch_flush exit 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 20 client.24256.objecter shutdown clearing up homeless session... 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 10 client.24256.objecter successfully canceled tick 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f8ab4071e90 msgr2=0x7f8ab4074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f8ab4071e90 0x7f8ab4074350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8ac000b5c0 tx=0x7f8ac001a040 comp rx=0 tx=0).stop 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 msgr2=0x7f8acc198cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 0x7f8acc198cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400ba50 tx=0x7f8ac400be10 comp rx=0 tx=0).stop 2024-04-26T16:15:55.814 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 shutdown_connections 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f8ab4071e90 0x7f8ab4074350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8acc103f00 0x7f8acc198cb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8acc109160 0x7f8acc1a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 --2- 172.21.15.190:0/3915291403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8acc106830 0x7f8acc1991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 >> 172.21.15.190:0/3915291403 conn(0x7f8acc0fd8f0 msgr2=0x7f8acc104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 shutdown_connections 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 -- 172.21.15.190:0/3915291403 wait complete. 2024-04-26T16:15:55.815 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:15:55.815+0000 7f8ad341f700 1 librados: shutdown 2024-04-26T16:15:56.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:53.076413+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:53.076413+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:53.076513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:53.076513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:54.714327+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:54.714327+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:54.935582+0000 mon.a (mon.0) 619 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] boot 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:54.935582+0000 mon.a (mon.0) 619 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] boot 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:54.935716+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:54.935716+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:54.936105+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:54.936105+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:55.097867+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:55.097867+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:55.097959+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:55.097959+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:55.780445+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:55.780445+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:55.781305+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:55.781305+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:55.790308+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: audit 2024-04-26T16:15:55.790308+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:55.853957+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-04-26T16:15:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:55 smithi190 bash[23868]: cluster 2024-04-26T16:15:55.853957+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-04-26T16:15:56.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:53.076413+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:53.076413+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:53.076513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:53.076513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:54.714327+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:54.714327+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:54.935582+0000 mon.a (mon.0) 619 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] boot 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:54.935582+0000 mon.a (mon.0) 619 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] boot 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:54.935716+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:54.935716+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:54.936105+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:54.936105+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:55.097867+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:55.097867+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:55.097959+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:55.097959+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:55.780445+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:55.780445+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:55.781305+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:55.781305+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:55.790308+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: audit 2024-04-26T16:15:55.790308+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:55.853957+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[21463]: cluster 2024-04-26T16:15:55.853957+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:53.076413+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:53.076413+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:53.076513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:53.076513+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:54.714327+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:54.714327+0000 mgr.y (mgr.14152) 287 : cluster [DBG] pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:54.935582+0000 mon.a (mon.0) 619 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] boot 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:54.935582+0000 mon.a (mon.0) 619 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] boot 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:54.935716+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:54.935716+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e57: 7 total, 7 up, 7 in 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:54.936105+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:54.936105+0000 mon.a (mon.0) 621 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:55.097867+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:55.097867+0000 osd.0 (osd.0) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:55.097959+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:55.097959+0000 osd.0 (osd.0) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:55.780445+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:55.780445+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:55.781305+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:55.781305+0000 mon.a (mon.0) 623 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:15:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:55.790308+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: audit 2024-04-26T16:15:55.790308+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:15:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:55.853957+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-04-26T16:15:56.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:55 smithi050 bash[29234]: cluster 2024-04-26T16:15:55.853957+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e58: 7 total, 7 up, 7 in 2024-04-26T16:15:56.473 DEBUG:teuthology.orchestra.run.smithi190:osd.6> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T16:15:56.476 INFO:tasks.cephadm:Deploying osd.7 on smithi190 with /dev/vg_nvme/lv_1... 2024-04-26T16:15:56.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[21463]: cluster 2024-04-26T16:15:56.714873+0000 mgr.y (mgr.14152) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[21463]: cluster 2024-04-26T16:15:56.714873+0000 mgr.y (mgr.14152) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[21463]: cluster 2024-04-26T16:15:56.856172+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[21463]: cluster 2024-04-26T16:15:56.856172+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[29234]: cluster 2024-04-26T16:15:56.714873+0000 mgr.y (mgr.14152) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[29234]: cluster 2024-04-26T16:15:56.714873+0000 mgr.y (mgr.14152) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[29234]: cluster 2024-04-26T16:15:56.856172+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-04-26T16:15:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:57 smithi050 bash[29234]: cluster 2024-04-26T16:15:56.856172+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-04-26T16:15:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:57 smithi190 bash[23868]: cluster 2024-04-26T16:15:56.714873+0000 mgr.y (mgr.14152) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:15:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:57 smithi190 bash[23868]: cluster 2024-04-26T16:15:56.714873+0000 mgr.y (mgr.14152) 288 : cluster [DBG] pgmap v279: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:15:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:57 smithi190 bash[23868]: cluster 2024-04-26T16:15:56.856172+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-04-26T16:15:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:57 smithi190 bash[23868]: cluster 2024-04-26T16:15:56.856172+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e59: 7 total, 7 up, 7 in 2024-04-26T16:15:59.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:58 smithi050 bash[21463]: cluster 2024-04-26T16:15:57.857884+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-04-26T16:15:59.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:58 smithi050 bash[21463]: cluster 2024-04-26T16:15:57.857884+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-04-26T16:15:59.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:58 smithi050 bash[29234]: cluster 2024-04-26T16:15:57.857884+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-04-26T16:15:59.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:58 smithi050 bash[29234]: cluster 2024-04-26T16:15:57.857884+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-04-26T16:15:59.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:58 smithi190 bash[23868]: cluster 2024-04-26T16:15:57.857884+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-04-26T16:15:59.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:58 smithi190 bash[23868]: cluster 2024-04-26T16:15:57.857884+0000 mon.a (mon.0) 627 : cluster [DBG] osdmap e60: 7 total, 7 up, 7 in 2024-04-26T16:16:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[21463]: cluster 2024-04-26T16:15:58.715441+0000 mgr.y (mgr.14152) 289 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[21463]: cluster 2024-04-26T16:15:58.715441+0000 mgr.y (mgr.14152) 289 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[21463]: cluster 2024-04-26T16:15:58.875180+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-04-26T16:16:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[21463]: cluster 2024-04-26T16:15:58.875180+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-04-26T16:16:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[29234]: cluster 2024-04-26T16:15:58.715441+0000 mgr.y (mgr.14152) 289 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:00.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[29234]: cluster 2024-04-26T16:15:58.715441+0000 mgr.y (mgr.14152) 289 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:00.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[29234]: cluster 2024-04-26T16:15:58.875180+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-04-26T16:16:00.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:15:59 smithi050 bash[29234]: cluster 2024-04-26T16:15:58.875180+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-04-26T16:16:00.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:59 smithi190 bash[23868]: cluster 2024-04-26T16:15:58.715441+0000 mgr.y (mgr.14152) 289 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:00.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:59 smithi190 bash[23868]: cluster 2024-04-26T16:15:58.715441+0000 mgr.y (mgr.14152) 289 : cluster [DBG] pgmap v282: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:00.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:59 smithi190 bash[23868]: cluster 2024-04-26T16:15:58.875180+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-04-26T16:16:00.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:15:59 smithi190 bash[23868]: cluster 2024-04-26T16:15:58.875180+0000 mon.a (mon.0) 628 : cluster [DBG] osdmap e61: 7 total, 7 up, 7 in 2024-04-26T16:16:00.769 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:16:02.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:01 smithi190 bash[23868]: cluster 2024-04-26T16:16:00.716037+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-26T16:16:02.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:01 smithi190 bash[23868]: cluster 2024-04-26T16:16:00.716037+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-26T16:16:02.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:01 smithi050 bash[21463]: cluster 2024-04-26T16:16:00.716037+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-26T16:16:02.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:01 smithi050 bash[21463]: cluster 2024-04-26T16:16:00.716037+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-26T16:16:02.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:01 smithi050 bash[29234]: cluster 2024-04-26T16:16:00.716037+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-26T16:16:02.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:01 smithi050 bash[29234]: cluster 2024-04-26T16:16:00.716037+0000 mgr.y (mgr.14152) 290 : cluster [DBG] pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:03 smithi190 bash[23868]: cluster 2024-04-26T16:16:02.716581+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:03 smithi190 bash[23868]: cluster 2024-04-26T16:16:02.716581+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:03 smithi050 bash[21463]: cluster 2024-04-26T16:16:02.716581+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:03 smithi050 bash[21463]: cluster 2024-04-26T16:16:02.716581+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:03 smithi050 bash[29234]: cluster 2024-04-26T16:16:02.716581+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:03 smithi050 bash[29234]: cluster 2024-04-26T16:16:02.716581+0000 mgr.y (mgr.14152) 291 : cluster [DBG] pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:04.506 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-26T16:16:04.506 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-26T16:16:04.506 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-04-26T16:16:04.506 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-04-26T16:16:04.506 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149733 s, 700 MB/s 2024-04-26T16:16:04.506 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-04-26T16:16:05.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2024-04-26T16:16:06.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: cluster 2024-04-26T16:16:04.717152+0000 mgr.y (mgr.14152) 292 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:06.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: cluster 2024-04-26T16:16:04.717152+0000 mgr.y (mgr.14152) 292 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:06.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: cluster 2024-04-26T16:16:04.932048+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: cluster 2024-04-26T16:16:04.932048+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: cluster 2024-04-26T16:16:04.932148+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: cluster 2024-04-26T16:16:04.932148+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.742382+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.742382+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.752496+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.752496+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.754935+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.754935+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.756809+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.756809+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.758570+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.758570+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.766594+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.766594+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.771243+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.771243+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: cluster 2024-04-26T16:16:04.717152+0000 mgr.y (mgr.14152) 292 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: cluster 2024-04-26T16:16:04.717152+0000 mgr.y (mgr.14152) 292 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: cluster 2024-04-26T16:16:04.932048+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: cluster 2024-04-26T16:16:04.932048+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: cluster 2024-04-26T16:16:04.932148+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: cluster 2024-04-26T16:16:04.932148+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.742382+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.742382+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.752496+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.752496+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.754935+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.754935+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.756809+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.756809+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.758570+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.758570+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.766594+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.766594+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.771243+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.771243+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:06.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.773253+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:06.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.773253+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:06.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.783424+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[21463]: audit 2024-04-26T16:16:05.783424+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.773253+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:06.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.773253+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:06.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.783424+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:05 smithi050 bash[29234]: audit 2024-04-26T16:16:05.783424+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: cluster 2024-04-26T16:16:04.717152+0000 mgr.y (mgr.14152) 292 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: cluster 2024-04-26T16:16:04.717152+0000 mgr.y (mgr.14152) 292 : cluster [DBG] pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: cluster 2024-04-26T16:16:04.932048+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: cluster 2024-04-26T16:16:04.932048+0000 osd.1 (osd.1) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: cluster 2024-04-26T16:16:04.932148+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: cluster 2024-04-26T16:16:04.932148+0000 osd.1 (osd.1) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.742382+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.742382+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.752496+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.752496+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.754935+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.754935+0000 mon.a (mon.0) 631 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.756809+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.756809+0000 mon.a (mon.0) 632 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.758570+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.758570+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.766594+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.766594+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.771243+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.771243+0000 mon.a (mon.0) 635 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.773253+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.773253+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.783424+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:05 smithi190 bash[23868]: audit 2024-04-26T16:16:05.783424+0000 mon.a (mon.0) 637 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:07.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:06 smithi190 bash[23868]: cephadm 2024-04-26T16:16:05.732849+0000 mgr.y (mgr.14152) 293 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:07.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:06 smithi190 bash[23868]: cephadm 2024-04-26T16:16:05.732849+0000 mgr.y (mgr.14152) 293 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:07.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:06 smithi190 bash[23868]: cephadm 2024-04-26T16:16:05.760026+0000 mgr.y (mgr.14152) 294 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 5755M 2024-04-26T16:16:07.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:06 smithi190 bash[23868]: cephadm 2024-04-26T16:16:05.760026+0000 mgr.y (mgr.14152) 294 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 5755M 2024-04-26T16:16:07.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:06 smithi190 bash[23868]: cluster 2024-04-26T16:16:05.899756+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-04-26T16:16:07.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:06 smithi190 bash[23868]: cluster 2024-04-26T16:16:05.899756+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-04-26T16:16:07.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[29234]: cephadm 2024-04-26T16:16:05.732849+0000 mgr.y (mgr.14152) 293 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:07.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[29234]: cephadm 2024-04-26T16:16:05.732849+0000 mgr.y (mgr.14152) 293 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:07.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[29234]: cephadm 2024-04-26T16:16:05.760026+0000 mgr.y (mgr.14152) 294 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 5755M 2024-04-26T16:16:07.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[29234]: cephadm 2024-04-26T16:16:05.760026+0000 mgr.y (mgr.14152) 294 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 5755M 2024-04-26T16:16:07.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[29234]: cluster 2024-04-26T16:16:05.899756+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-04-26T16:16:07.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[29234]: cluster 2024-04-26T16:16:05.899756+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-04-26T16:16:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[21463]: cephadm 2024-04-26T16:16:05.732849+0000 mgr.y (mgr.14152) 293 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[21463]: cephadm 2024-04-26T16:16:05.732849+0000 mgr.y (mgr.14152) 293 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[21463]: cephadm 2024-04-26T16:16:05.760026+0000 mgr.y (mgr.14152) 294 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 5755M 2024-04-26T16:16:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[21463]: cephadm 2024-04-26T16:16:05.760026+0000 mgr.y (mgr.14152) 294 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 5755M 2024-04-26T16:16:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[21463]: cluster 2024-04-26T16:16:05.899756+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-04-26T16:16:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:06 smithi050 bash[21463]: cluster 2024-04-26T16:16:05.899756+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e62: 7 total, 7 up, 7 in 2024-04-26T16:16:08.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:07 smithi190 bash[23868]: cluster 2024-04-26T16:16:06.717780+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T16:16:08.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:07 smithi190 bash[23868]: cluster 2024-04-26T16:16:06.717780+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T16:16:08.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:07 smithi050 bash[29234]: cluster 2024-04-26T16:16:06.717780+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T16:16:08.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:07 smithi050 bash[29234]: cluster 2024-04-26T16:16:06.717780+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T16:16:08.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:07 smithi050 bash[21463]: cluster 2024-04-26T16:16:06.717780+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T16:16:08.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:07 smithi050 bash[21463]: cluster 2024-04-26T16:16:06.717780+0000 mgr.y (mgr.14152) 295 : cluster [DBG] pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-26T16:16:10.062 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:16:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:09 smithi190 bash[23868]: cluster 2024-04-26T16:16:08.718487+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T16:16:10.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:09 smithi190 bash[23868]: cluster 2024-04-26T16:16:08.718487+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T16:16:10.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:09 smithi050 bash[21463]: cluster 2024-04-26T16:16:08.718487+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T16:16:10.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:09 smithi050 bash[21463]: cluster 2024-04-26T16:16:08.718487+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T16:16:10.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:09 smithi050 bash[29234]: cluster 2024-04-26T16:16:08.718487+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T16:16:10.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:09 smithi050 bash[29234]: cluster 2024-04-26T16:16:08.718487+0000 mgr.y (mgr.14152) 296 : cluster [DBG] pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-26T16:16:12.299 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:11 smithi190 bash[23868]: cluster 2024-04-26T16:16:10.719262+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:12.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:11 smithi190 bash[23868]: cluster 2024-04-26T16:16:10.719262+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:12.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:11 smithi050 bash[21463]: cluster 2024-04-26T16:16:10.719262+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:12.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:11 smithi050 bash[21463]: cluster 2024-04-26T16:16:10.719262+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:12.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:11 smithi050 bash[29234]: cluster 2024-04-26T16:16:10.719262+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:12.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:11 smithi050 bash[29234]: cluster 2024-04-26T16:16:10.719262+0000 mgr.y (mgr.14152) 297 : cluster [DBG] pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:12.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 -- 172.21.15.190:0/895623975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20106900 msgr2=0x7fae20108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:12.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 --2- 172.21.15.190:0/895623975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20106900 0x7fae20108cf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae1400b0a0 tx=0x7fae1400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:16:12.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 -- 172.21.15.190:0/895623975 shutdown_connections 2024-04-26T16:16:12.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 --2- 172.21.15.190:0/895623975 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae20109230 0x7fae2010b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:12.467 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 --2- 172.21.15.190:0/895623975 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae20103fd0 0x7fae201063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 --2- 172.21.15.190:0/895623975 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20106900 0x7fae20108cf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 -- 172.21.15.190:0/895623975 >> 172.21.15.190:0/895623975 conn(0x7fae200fd940 msgr2=0x7fae200ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 -- 172.21.15.190:0/895623975 shutdown_connections 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 -- 172.21.15.190:0/895623975 wait complete. 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 librados: starting msgr at 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 librados: starting objecter 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 Processor -- start 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 -- start start 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 librados: setting wanted keys 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.467+0000 7fae284fb700 1 librados: calling monclient init 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 0x7fae20198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:12.468 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae20106900 0x7fae201991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae20109230 0x7fae201a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010e450 con 0x7fae20109230 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fae2010e2d0 con 0x7fae20103fd0 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fae2010e5d0 con 0x7fae20106900 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 0x7fae20198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 0x7fae20198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:37818/0 (socket says 172.21.15.190:37818) 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 -- 172.21.15.190:0/2136492021 learned_addr learned my addr 172.21.15.190:0/2136492021 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:16:12.469 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae25a96700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae20106900 0x7fae201991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26a98700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae20109230 0x7fae201a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 -- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae20106900 msgr2=0x7fae201991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae20106900 0x7fae201991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 -- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae20109230 msgr2=0x7fae201a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae20109230 0x7fae201a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 -- 172.21.15.190:0/2136492021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae201a08c0 con 0x7fae20103fd0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae26297700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 0x7fae20198c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fae1000c970 tx=0x7fae1000cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 10 client.?.objecter ms_handle_connect 0x7fae20103fd0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10007aa0 con 0x7fae20103fd0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fae10004c90 con 0x7fae20103fd0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae10005400 con 0x7fae20103fd0 2024-04-26T16:16:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae201a0bb0 con 0x7fae20103fd0 2024-04-26T16:16:12.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 10 client.24283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:16:12.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae201a10f0 con 0x7fae20103fd0 2024-04-26T16:16:12.471 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 librados: init done 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2006a110 con 0x7fae20103fd0 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7fae10004750 con 0x7fae20103fd0 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fae08071e90 0x7fae08074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(62..62 src has 11..62) ==== 4855+0+0 (secure 0 0 0) 0x7fae10090530 con 0x7fae20103fd0 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 10 client.24283.objecter ms_dispatch 0x7fae20001550 osd_map(62..62 src has 11..62) 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 3 client.24283.objecter handle_osd_map got epochs [62,62] > 0 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 3 client.24283.objecter handle_osd_map decoding full epoch 62 2024-04-26T16:16:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae1f7fe700 20 client.24283.objecter dump_active .. 0 homeless 2024-04-26T16:16:12.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.471+0000 7fae25a96700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fae08071e90 0x7fae08074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:12.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.475+0000 7fae1f7fe700 10 client.24283.objecter ms_handle_connect 0x7fae08071e90 2024-04-26T16:16:12.473 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.475+0000 7fae25a96700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fae08071e90 0x7fae08074350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae140062a0 tx=0x7fae14006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:16:12.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.479+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fae1005d160 con 0x7fae20103fd0 2024-04-26T16:16:12.697 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:12.699+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fae20063860 con 0x7fae08071e90 2024-04-26T16:16:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:12 smithi190 bash[23868]: audit 2024-04-26T16:16:12.716989+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:16:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:12 smithi190 bash[23868]: audit 2024-04-26T16:16:12.716989+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:16:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:12 smithi190 bash[23868]: audit 2024-04-26T16:16:12.723606+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:16:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:12 smithi190 bash[23868]: audit 2024-04-26T16:16:12.723606+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:16:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:12 smithi190 bash[23868]: audit 2024-04-26T16:16:12.725414+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:12 smithi190 bash[23868]: audit 2024-04-26T16:16:12.725414+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[21463]: audit 2024-04-26T16:16:12.716989+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[21463]: audit 2024-04-26T16:16:12.716989+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[21463]: audit 2024-04-26T16:16:12.723606+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[21463]: audit 2024-04-26T16:16:12.723606+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[21463]: audit 2024-04-26T16:16:12.725414+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[21463]: audit 2024-04-26T16:16:12.725414+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[29234]: audit 2024-04-26T16:16:12.716989+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[29234]: audit 2024-04-26T16:16:12.716989+0000 mon.a (mon.0) 639 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-26T16:16:13.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[29234]: audit 2024-04-26T16:16:12.723606+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:16:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[29234]: audit 2024-04-26T16:16:12.723606+0000 mon.a (mon.0) 640 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-26T16:16:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[29234]: audit 2024-04-26T16:16:12.725414+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:12 smithi050 bash[29234]: audit 2024-04-26T16:16:12.725414+0000 mon.a (mon.0) 641 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:13 smithi190 bash[23868]: audit 2024-04-26T16:16:12.709883+0000 mgr.y (mgr.14152) 298 : audit [DBG] from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:16:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:13 smithi190 bash[23868]: audit 2024-04-26T16:16:12.709883+0000 mgr.y (mgr.14152) 298 : audit [DBG] from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:16:14.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:13 smithi190 bash[23868]: cluster 2024-04-26T16:16:12.719842+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:14.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:13 smithi190 bash[23868]: cluster 2024-04-26T16:16:12.719842+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:14.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[21463]: audit 2024-04-26T16:16:12.709883+0000 mgr.y (mgr.14152) 298 : audit [DBG] from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[21463]: audit 2024-04-26T16:16:12.709883+0000 mgr.y (mgr.14152) 298 : audit [DBG] from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[21463]: cluster 2024-04-26T16:16:12.719842+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[21463]: cluster 2024-04-26T16:16:12.719842+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[29234]: audit 2024-04-26T16:16:12.709883+0000 mgr.y (mgr.14152) 298 : audit [DBG] from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[29234]: audit 2024-04-26T16:16:12.709883+0000 mgr.y (mgr.14152) 298 : audit [DBG] from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[29234]: cluster 2024-04-26T16:16:12.719842+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:14.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:13 smithi050 bash[29234]: cluster 2024-04-26T16:16:12.719842+0000 mgr.y (mgr.14152) 299 : cluster [DBG] pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:16.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:15 smithi190 bash[23868]: cluster 2024-04-26T16:16:14.720376+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:16.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:15 smithi190 bash[23868]: cluster 2024-04-26T16:16:14.720376+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:16.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:15 smithi050 bash[21463]: cluster 2024-04-26T16:16:14.720376+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:16.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:15 smithi050 bash[21463]: cluster 2024-04-26T16:16:14.720376+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:15 smithi050 bash[29234]: cluster 2024-04-26T16:16:14.720376+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:15 smithi050 bash[29234]: cluster 2024-04-26T16:16:14.720376+0000 mgr.y (mgr.14152) 300 : cluster [DBG] pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:17.470 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:17.475+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:18.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:17 smithi190 bash[23868]: cluster 2024-04-26T16:16:16.720998+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:18.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:17 smithi190 bash[23868]: cluster 2024-04-26T16:16:16.720998+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:18.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:17 smithi050 bash[21463]: cluster 2024-04-26T16:16:16.720998+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:18.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:17 smithi050 bash[21463]: cluster 2024-04-26T16:16:16.720998+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:18.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:17 smithi050 bash[29234]: cluster 2024-04-26T16:16:16.720998+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:18.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:17 smithi050 bash[29234]: cluster 2024-04-26T16:16:16.720998+0000 mgr.y (mgr.14152) 301 : cluster [DBG] pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:19 smithi190 bash[23868]: cluster 2024-04-26T16:16:18.721609+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:19 smithi190 bash[23868]: cluster 2024-04-26T16:16:18.721609+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:20.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:19 smithi050 bash[21463]: cluster 2024-04-26T16:16:18.721609+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:20.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:19 smithi050 bash[21463]: cluster 2024-04-26T16:16:18.721609+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:20.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:19 smithi050 bash[29234]: cluster 2024-04-26T16:16:18.721609+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:20.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:19 smithi050 bash[29234]: cluster 2024-04-26T16:16:18.721609+0000 mgr.y (mgr.14152) 302 : cluster [DBG] pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: cluster 2024-04-26T16:16:20.722190+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: cluster 2024-04-26T16:16:20.722190+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.938925+0000 mon.b (mon.1) 22 : audit [INF] from='client.? 172.21.15.190:0/2448339360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.938925+0000 mon.b (mon.1) 22 : audit [INF] from='client.? 172.21.15.190:0/2448339360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.943655+0000 mon.a (mon.0) 642 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.943655+0000 mon.a (mon.0) 642 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.947388+0000 mon.a (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]': finished 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.947388+0000 mon.a (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]': finished 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: cluster 2024-04-26T16:16:20.952823+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: cluster 2024-04-26T16:16:20.952823+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.953496+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:20.953496+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:21.687754+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? 172.21.15.190:0/449547833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:16:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:21 smithi190 bash[23868]: audit 2024-04-26T16:16:21.687754+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? 172.21.15.190:0/449547833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:16:22.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: cluster 2024-04-26T16:16:20.722190+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: cluster 2024-04-26T16:16:20.722190+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.938925+0000 mon.b (mon.1) 22 : audit [INF] from='client.? 172.21.15.190:0/2448339360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.938925+0000 mon.b (mon.1) 22 : audit [INF] from='client.? 172.21.15.190:0/2448339360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.943655+0000 mon.a (mon.0) 642 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.943655+0000 mon.a (mon.0) 642 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.947388+0000 mon.a (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]': finished 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.947388+0000 mon.a (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]': finished 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: cluster 2024-04-26T16:16:20.952823+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: cluster 2024-04-26T16:16:20.952823+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.953496+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:20.953496+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:21.687754+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? 172.21.15.190:0/449547833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[21463]: audit 2024-04-26T16:16:21.687754+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? 172.21.15.190:0/449547833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: cluster 2024-04-26T16:16:20.722190+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: cluster 2024-04-26T16:16:20.722190+0000 mgr.y (mgr.14152) 303 : cluster [DBG] pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.938925+0000 mon.b (mon.1) 22 : audit [INF] from='client.? 172.21.15.190:0/2448339360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.938925+0000 mon.b (mon.1) 22 : audit [INF] from='client.? 172.21.15.190:0/2448339360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.943655+0000 mon.a (mon.0) 642 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.943655+0000 mon.a (mon.0) 642 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]: dispatch 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.947388+0000 mon.a (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]': finished 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.947388+0000 mon.a (mon.0) 643 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5"}]': finished 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: cluster 2024-04-26T16:16:20.952823+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: cluster 2024-04-26T16:16:20.952823+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e63: 8 total, 7 up, 8 in 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.953496+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:20.953496+0000 mon.a (mon.0) 645 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:21.687754+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? 172.21.15.190:0/449547833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:16:22.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:21 smithi050 bash[29234]: audit 2024-04-26T16:16:21.687754+0000 mon.b (mon.1) 23 : audit [DBG] from='client.? 172.21.15.190:0/449547833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-26T16:16:22.474 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:22.479+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: cluster 2024-04-26T16:16:21.407293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: cluster 2024-04-26T16:16:21.407293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: cluster 2024-04-26T16:16:21.407369+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: cluster 2024-04-26T16:16:21.407369+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: cluster 2024-04-26T16:16:21.954755+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: cluster 2024-04-26T16:16:21.954755+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: audit 2024-04-26T16:16:21.955134+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:23.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:22 smithi190 bash[23868]: audit 2024-04-26T16:16:21.955134+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: cluster 2024-04-26T16:16:21.407293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: cluster 2024-04-26T16:16:21.407293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: cluster 2024-04-26T16:16:21.407369+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: cluster 2024-04-26T16:16:21.407369+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: cluster 2024-04-26T16:16:21.954755+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: cluster 2024-04-26T16:16:21.954755+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: audit 2024-04-26T16:16:21.955134+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:23.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[21463]: audit 2024-04-26T16:16:21.955134+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: cluster 2024-04-26T16:16:21.407293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: cluster 2024-04-26T16:16:21.407293+0000 osd.5 (osd.5) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: cluster 2024-04-26T16:16:21.407369+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: cluster 2024-04-26T16:16:21.407369+0000 osd.5 (osd.5) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: cluster 2024-04-26T16:16:21.954755+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: cluster 2024-04-26T16:16:21.954755+0000 mon.a (mon.0) 646 : cluster [DBG] osdmap e64: 8 total, 7 up, 8 in 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: audit 2024-04-26T16:16:21.955134+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:23.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:22 smithi050 bash[29234]: audit 2024-04-26T16:16:21.955134+0000 mon.a (mon.0) 647 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:24.199 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:23 smithi190 bash[23868]: cluster 2024-04-26T16:16:22.722753+0000 mgr.y (mgr.14152) 304 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:24.199 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:23 smithi190 bash[23868]: cluster 2024-04-26T16:16:22.722753+0000 mgr.y (mgr.14152) 304 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:24.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:23 smithi050 bash[21463]: cluster 2024-04-26T16:16:22.722753+0000 mgr.y (mgr.14152) 304 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:24.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:23 smithi050 bash[21463]: cluster 2024-04-26T16:16:22.722753+0000 mgr.y (mgr.14152) 304 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:24.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:23 smithi050 bash[29234]: cluster 2024-04-26T16:16:22.722753+0000 mgr.y (mgr.14152) 304 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:24.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:23 smithi050 bash[29234]: cluster 2024-04-26T16:16:22.722753+0000 mgr.y (mgr.14152) 304 : cluster [DBG] pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:25 smithi190 bash[23868]: cluster 2024-04-26T16:16:24.723348+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:25 smithi190 bash[23868]: cluster 2024-04-26T16:16:24.723348+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:26.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:25 smithi050 bash[21463]: cluster 2024-04-26T16:16:24.723348+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:26.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:25 smithi050 bash[21463]: cluster 2024-04-26T16:16:24.723348+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:25 smithi050 bash[29234]: cluster 2024-04-26T16:16:24.723348+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:25 smithi050 bash[29234]: cluster 2024-04-26T16:16:24.723348+0000 mgr.y (mgr.14152) 305 : cluster [DBG] pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:27.478 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:27.483+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:27 smithi190 bash[23868]: cluster 2024-04-26T16:16:26.723941+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:27 smithi190 bash[23868]: cluster 2024-04-26T16:16:26.723941+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:28.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:27 smithi050 bash[21463]: cluster 2024-04-26T16:16:26.723941+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:28.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:27 smithi050 bash[21463]: cluster 2024-04-26T16:16:26.723941+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:28.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:27 smithi050 bash[29234]: cluster 2024-04-26T16:16:26.723941+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:28.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:27 smithi050 bash[29234]: cluster 2024-04-26T16:16:26.723941+0000 mgr.y (mgr.14152) 306 : cluster [DBG] pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:29.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: cluster 2024-04-26T16:16:27.876586+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:29.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: cluster 2024-04-26T16:16:27.876586+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:29.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: cluster 2024-04-26T16:16:27.876682+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:29.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: cluster 2024-04-26T16:16:27.876682+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:29.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: cluster 2024-04-26T16:16:27.984494+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-04-26T16:16:29.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: cluster 2024-04-26T16:16:27.984494+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-04-26T16:16:29.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: audit 2024-04-26T16:16:27.984791+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:29.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:28 smithi190 bash[23868]: audit 2024-04-26T16:16:27.984791+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:29.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: cluster 2024-04-26T16:16:27.876586+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:29.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: cluster 2024-04-26T16:16:27.876586+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:29.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: cluster 2024-04-26T16:16:27.876682+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: cluster 2024-04-26T16:16:27.876682+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: cluster 2024-04-26T16:16:27.984494+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: cluster 2024-04-26T16:16:27.984494+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: audit 2024-04-26T16:16:27.984791+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[21463]: audit 2024-04-26T16:16:27.984791+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: cluster 2024-04-26T16:16:27.876586+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: cluster 2024-04-26T16:16:27.876586+0000 osd.2 (osd.2) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: cluster 2024-04-26T16:16:27.876682+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: cluster 2024-04-26T16:16:27.876682+0000 osd.2 (osd.2) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: cluster 2024-04-26T16:16:27.984494+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: cluster 2024-04-26T16:16:27.984494+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e65: 8 total, 7 up, 8 in 2024-04-26T16:16:29.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: audit 2024-04-26T16:16:27.984791+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:28 smithi050 bash[29234]: audit 2024-04-26T16:16:27.984791+0000 mon.a (mon.0) 649 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:29 smithi190 bash[23868]: cluster 2024-04-26T16:16:28.724458+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:29 smithi190 bash[23868]: cluster 2024-04-26T16:16:28.724458+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:30.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:29 smithi050 bash[21463]: cluster 2024-04-26T16:16:28.724458+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:30.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:29 smithi050 bash[21463]: cluster 2024-04-26T16:16:28.724458+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:30.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:29 smithi050 bash[29234]: cluster 2024-04-26T16:16:28.724458+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:30.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:29 smithi050 bash[29234]: cluster 2024-04-26T16:16:28.724458+0000 mgr.y (mgr.14152) 307 : cluster [DBG] pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:31 smithi190 bash[23868]: cluster 2024-04-26T16:16:30.725038+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:31 smithi190 bash[23868]: cluster 2024-04-26T16:16:30.725038+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:31 smithi050 bash[21463]: cluster 2024-04-26T16:16:30.725038+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:31 smithi050 bash[21463]: cluster 2024-04-26T16:16:30.725038+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:31 smithi050 bash[29234]: cluster 2024-04-26T16:16:30.725038+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:31 smithi050 bash[29234]: cluster 2024-04-26T16:16:30.725038+0000 mgr.y (mgr.14152) 308 : cluster [DBG] pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:32.481 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:32.487+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:33 smithi190 bash[23868]: cluster 2024-04-26T16:16:32.725600+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:33 smithi190 bash[23868]: cluster 2024-04-26T16:16:32.725600+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:33 smithi190 bash[23868]: audit 2024-04-26T16:16:33.247777+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T16:16:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:33 smithi190 bash[23868]: audit 2024-04-26T16:16:33.247777+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T16:16:34.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:33 smithi190 bash[23868]: audit 2024-04-26T16:16:33.249934+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:34.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:33 smithi190 bash[23868]: audit 2024-04-26T16:16:33.249934+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:34.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[21463]: cluster 2024-04-26T16:16:32.725600+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[21463]: cluster 2024-04-26T16:16:32.725600+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[21463]: audit 2024-04-26T16:16:33.247777+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[21463]: audit 2024-04-26T16:16:33.247777+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[21463]: audit 2024-04-26T16:16:33.249934+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[21463]: audit 2024-04-26T16:16:33.249934+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[29234]: cluster 2024-04-26T16:16:32.725600+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[29234]: cluster 2024-04-26T16:16:32.725600+0000 mgr.y (mgr.14152) 309 : cluster [DBG] pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[29234]: audit 2024-04-26T16:16:33.247777+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[29234]: audit 2024-04-26T16:16:33.247777+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-26T16:16:34.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[29234]: audit 2024-04-26T16:16:33.249934+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:33 smithi050 bash[29234]: audit 2024-04-26T16:16:33.249934+0000 mon.a (mon.0) 651 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:34 smithi190 bash[23868]: cephadm 2024-04-26T16:16:33.251721+0000 mgr.y (mgr.14152) 310 : cephadm [INF] Deploying daemon osd.7 on smithi190 2024-04-26T16:16:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:34 smithi190 bash[23868]: cephadm 2024-04-26T16:16:33.251721+0000 mgr.y (mgr.14152) 310 : cephadm [INF] Deploying daemon osd.7 on smithi190 2024-04-26T16:16:35.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:34 smithi050 bash[21463]: cephadm 2024-04-26T16:16:33.251721+0000 mgr.y (mgr.14152) 310 : cephadm [INF] Deploying daemon osd.7 on smithi190 2024-04-26T16:16:35.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:34 smithi050 bash[21463]: cephadm 2024-04-26T16:16:33.251721+0000 mgr.y (mgr.14152) 310 : cephadm [INF] Deploying daemon osd.7 on smithi190 2024-04-26T16:16:35.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:34 smithi050 bash[29234]: cephadm 2024-04-26T16:16:33.251721+0000 mgr.y (mgr.14152) 310 : cephadm [INF] Deploying daemon osd.7 on smithi190 2024-04-26T16:16:35.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:34 smithi050 bash[29234]: cephadm 2024-04-26T16:16:33.251721+0000 mgr.y (mgr.14152) 310 : cephadm [INF] Deploying daemon osd.7 on smithi190 2024-04-26T16:16:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:35 smithi190 bash[23868]: cluster 2024-04-26T16:16:34.726165+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:35 smithi190 bash[23868]: cluster 2024-04-26T16:16:34.726165+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:36.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:35 smithi050 bash[21463]: cluster 2024-04-26T16:16:34.726165+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:36.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:35 smithi050 bash[21463]: cluster 2024-04-26T16:16:34.726165+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:36.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:36 smithi050 bash[29234]: cluster 2024-04-26T16:16:34.726165+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:36 smithi050 bash[29234]: cluster 2024-04-26T16:16:34.726165+0000 mgr.y (mgr.14152) 311 : cluster [DBG] pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:37.302 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.302 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.302 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.303 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.485 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:37.491+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:37.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.557 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.558 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:37.558 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:16:37 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: cluster 2024-04-26T16:16:36.726752+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: cluster 2024-04-26T16:16:36.726752+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: audit 2024-04-26T16:16:37.619687+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: audit 2024-04-26T16:16:37.619687+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: audit 2024-04-26T16:16:37.629835+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: audit 2024-04-26T16:16:37.629835+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: audit 2024-04-26T16:16:37.660435+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:38 smithi190 bash[23868]: audit 2024-04-26T16:16:37.660435+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:38.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: cluster 2024-04-26T16:16:36.726752+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: cluster 2024-04-26T16:16:36.726752+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: audit 2024-04-26T16:16:37.619687+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: audit 2024-04-26T16:16:37.619687+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: audit 2024-04-26T16:16:37.629835+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: audit 2024-04-26T16:16:37.629835+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: audit 2024-04-26T16:16:37.660435+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[29234]: audit 2024-04-26T16:16:37.660435+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: cluster 2024-04-26T16:16:36.726752+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: cluster 2024-04-26T16:16:36.726752+0000 mgr.y (mgr.14152) 312 : cluster [DBG] pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: audit 2024-04-26T16:16:37.619687+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: audit 2024-04-26T16:16:37.619687+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: audit 2024-04-26T16:16:37.629835+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: audit 2024-04-26T16:16:37.629835+0000 mon.a (mon.0) 653 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: audit 2024-04-26T16:16:37.660435+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:38.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:38 smithi050 bash[21463]: audit 2024-04-26T16:16:37.660435+0000 mon.a (mon.0) 654 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:39 smithi190 bash[23868]: cluster 2024-04-26T16:16:38.727338+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:39 smithi190 bash[23868]: cluster 2024-04-26T16:16:38.727338+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:39.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:39 smithi050 bash[21463]: cluster 2024-04-26T16:16:38.727338+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:39.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:39 smithi050 bash[21463]: cluster 2024-04-26T16:16:38.727338+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:39.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:39 smithi050 bash[29234]: cluster 2024-04-26T16:16:38.727338+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:39.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:39 smithi050 bash[29234]: cluster 2024-04-26T16:16:38.727338+0000 mgr.y (mgr.14152) 313 : cluster [DBG] pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:41 smithi190 bash[23868]: cluster 2024-04-26T16:16:40.727939+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:41 smithi190 bash[23868]: cluster 2024-04-26T16:16:40.727939+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:41 smithi050 bash[21463]: cluster 2024-04-26T16:16:40.727939+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:41 smithi050 bash[21463]: cluster 2024-04-26T16:16:40.727939+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:41 smithi050 bash[29234]: cluster 2024-04-26T16:16:40.727939+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:41 smithi050 bash[29234]: cluster 2024-04-26T16:16:40.727939+0000 mgr.y (mgr.14152) 314 : cluster [DBG] pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:42.490 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:42.495+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:43 smithi190 bash[23868]: cluster 2024-04-26T16:16:42.728506+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:43 smithi190 bash[23868]: cluster 2024-04-26T16:16:42.728506+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:44.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:43 smithi050 bash[21463]: cluster 2024-04-26T16:16:42.728506+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:44.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:43 smithi050 bash[21463]: cluster 2024-04-26T16:16:42.728506+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:44.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:43 smithi050 bash[29234]: cluster 2024-04-26T16:16:42.728506+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:44.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:43 smithi050 bash[29234]: cluster 2024-04-26T16:16:42.728506+0000 mgr.y (mgr.14152) 315 : cluster [DBG] pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:43.832851+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:43.832851+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:43.837164+0000 mon.a (mon.0) 655 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:43.837164+0000 mon.a (mon.0) 655 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.510540+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.510540+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.520715+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.520715+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.036 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.523131+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:45.036 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.523131+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:45.036 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.525082+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:45.036 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.525082+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:45.036 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.534936+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.036 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:44 smithi190 bash[23868]: audit 2024-04-26T16:16:44.534936+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:43.832851+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:43.832851+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:43.837164+0000 mon.a (mon.0) 655 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:43.837164+0000 mon.a (mon.0) 655 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.510540+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.510540+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.520715+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.520715+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.523131+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:45.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.523131+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.525082+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.525082+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.534936+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[21463]: audit 2024-04-26T16:16:44.534936+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:43.832851+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:43.832851+0000 mon.b (mon.1) 24 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:43.837164+0000 mon.a (mon.0) 655 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:43.837164+0000 mon.a (mon.0) 655 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.510540+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.510540+0000 mon.a (mon.0) 656 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.520715+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.520715+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.523131+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:45.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.523131+0000 mon.a (mon.0) 658 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:45.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.525082+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:45.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.525082+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:45.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.534936+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:45.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:44 smithi050 bash[29234]: audit 2024-04-26T16:16:44.534936+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.729064+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.729064+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.800408+0000 mon.a (mon.0) 661 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.800408+0000 mon.a (mon.0) 661 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.802978+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.802978+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.806167+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-04-26T16:16:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.806167+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-04-26T16:16:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.806707+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.806707+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.807392+0000 mon.a (mon.0) 664 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:45 smithi190 bash[23868]: audit 2024-04-26T16:16:44.807392+0000 mon.a (mon.0) 664 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.729064+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.729064+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.800408+0000 mon.a (mon.0) 661 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.800408+0000 mon.a (mon.0) 661 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.802978+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.802978+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.806167+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.806167+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.806707+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.806707+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.807392+0000 mon.a (mon.0) 664 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[21463]: audit 2024-04-26T16:16:44.807392+0000 mon.a (mon.0) 664 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.729064+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.729064+0000 mgr.y (mgr.14152) 316 : cluster [DBG] pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.800408+0000 mon.a (mon.0) 661 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.800408+0000 mon.a (mon.0) 661 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.802978+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.802978+0000 mon.b (mon.1) 25 : audit [INF] from='osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.806167+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.806167+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e66: 8 total, 7 up, 8 in 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.806707+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.806707+0000 mon.a (mon.0) 663 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.807392+0000 mon.a (mon.0) 664 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:46.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:45 smithi050 bash[29234]: audit 2024-04-26T16:16:44.807392+0000 mon.a (mon.0) 664 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.802761+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.802761+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: cluster 2024-04-26T16:16:45.809713+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e67: 8 total, 7 up, 8 in 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: cluster 2024-04-26T16:16:45.809713+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e67: 8 total, 7 up, 8 in 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.810819+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.810819+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.822478+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.822478+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: cluster 2024-04-26T16:16:45.869876+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e68: 8 total, 7 up, 8 in 2024-04-26T16:16:47.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: cluster 2024-04-26T16:16:45.869876+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e68: 8 total, 7 up, 8 in 2024-04-26T16:16:47.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.870705+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:46 smithi190 bash[23868]: audit 2024-04-26T16:16:45.870705+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.802761+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.802761+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: cluster 2024-04-26T16:16:45.809713+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e67: 8 total, 7 up, 8 in 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: cluster 2024-04-26T16:16:45.809713+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e67: 8 total, 7 up, 8 in 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.810819+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.810819+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.822478+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.822478+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: cluster 2024-04-26T16:16:45.869876+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e68: 8 total, 7 up, 8 in 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: cluster 2024-04-26T16:16:45.869876+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e68: 8 total, 7 up, 8 in 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.870705+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[21463]: audit 2024-04-26T16:16:45.870705+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.802761+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.802761+0000 mon.a (mon.0) 665 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: cluster 2024-04-26T16:16:45.809713+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e67: 8 total, 7 up, 8 in 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: cluster 2024-04-26T16:16:45.809713+0000 mon.a (mon.0) 666 : cluster [DBG] osdmap e67: 8 total, 7 up, 8 in 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.810819+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.810819+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.822478+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.822478+0000 mon.a (mon.0) 668 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: cluster 2024-04-26T16:16:45.869876+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e68: 8 total, 7 up, 8 in 2024-04-26T16:16:47.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: cluster 2024-04-26T16:16:45.869876+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e68: 8 total, 7 up, 8 in 2024-04-26T16:16:47.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.870705+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:46 smithi050 bash[29234]: audit 2024-04-26T16:16:45.870705+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:47.493 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.498+0000 7fae27cfa700 10 client.24283.objecter tick 2024-04-26T16:16:47.934 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae1f7fe700 1 -- 172.21.15.190:0/2136492021 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fae20063860 con 0x7fae08071e90 2024-04-26T16:16:47.934 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 7 on host 'smithi190' 2024-04-26T16:16:47.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 10 librados: watch_flush enter 2024-04-26T16:16:47.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 10 librados: watch_flush exit 2024-04-26T16:16:47.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 20 client.24283.objecter shutdown clearing up homeless session... 2024-04-26T16:16:47.937 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 10 client.24283.objecter successfully canceled tick 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fae08071e90 msgr2=0x7fae08074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fae08071e90 0x7fae08074350 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae140062a0 tx=0x7fae14006210 comp rx=0 tx=0).stop 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 msgr2=0x7fae20198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.938+0000 7fae284fb700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 0x7fae20198c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fae1000c970 tx=0x7fae1000cd30 comp rx=0 tx=0).stop 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 shutdown_connections 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fae08071e90 0x7fae08074350 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae20103fd0 0x7fae20198c70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fae20109230 0x7fae201a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 --2- 172.21.15.190:0/2136492021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fae20106900 0x7fae201991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 >> 172.21.15.190:0/2136492021 conn(0x7fae200fd940 msgr2=0x7fae20104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:16:47.938 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 shutdown_connections 2024-04-26T16:16:47.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 -- 172.21.15.190:0/2136492021 wait complete. 2024-04-26T16:16:47.939 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:16:47.942+0000 7fae284fb700 1 librados: shutdown 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.813142+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.813142+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.813231+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:44.813231+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:46.729653+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:46.729653+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: audit 2024-04-26T16:16:46.822493+0000 mon.a (mon.0) 671 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: audit 2024-04-26T16:16:46.822493+0000 mon.a (mon.0) 671 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:46.872461+0000 mon.a (mon.0) 672 : cluster [INF] osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] boot 2024-04-26T16:16:48.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:46.872461+0000 mon.a (mon.0) 672 : cluster [INF] osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] boot 2024-04-26T16:16:48.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:46.872634+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-04-26T16:16:48.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: cluster 2024-04-26T16:16:46.872634+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-04-26T16:16:48.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: audit 2024-04-26T16:16:46.872982+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:47 smithi190 bash[23868]: audit 2024-04-26T16:16:46.872982+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.813142+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.813142+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.813231+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:44.813231+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:46.729653+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:46.729653+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: audit 2024-04-26T16:16:46.822493+0000 mon.a (mon.0) 671 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: audit 2024-04-26T16:16:46.822493+0000 mon.a (mon.0) 671 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:46.872461+0000 mon.a (mon.0) 672 : cluster [INF] osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] boot 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:46.872461+0000 mon.a (mon.0) 672 : cluster [INF] osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] boot 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:46.872634+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: cluster 2024-04-26T16:16:46.872634+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: audit 2024-04-26T16:16:46.872982+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[21463]: audit 2024-04-26T16:16:46.872982+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.813142+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.813142+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.813231+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:44.813231+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:46.729653+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:46.729653+0000 mgr.y (mgr.14152) 317 : cluster [DBG] pgmap v314: 1 pgs: 1 remapped+peering; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: audit 2024-04-26T16:16:46.822493+0000 mon.a (mon.0) 671 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: audit 2024-04-26T16:16:46.822493+0000 mon.a (mon.0) 671 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:46.872461+0000 mon.a (mon.0) 672 : cluster [INF] osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] boot 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:46.872461+0000 mon.a (mon.0) 672 : cluster [INF] osd.7 [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] boot 2024-04-26T16:16:48.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:46.872634+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-04-26T16:16:48.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: cluster 2024-04-26T16:16:46.872634+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2024-04-26T16:16:48.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: audit 2024-04-26T16:16:46.872982+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:47 smithi050 bash[29234]: audit 2024-04-26T16:16:46.872982+0000 mon.a (mon.0) 674 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:16:48.611 DEBUG:teuthology.orchestra.run.smithi190:osd.7> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.7.service 2024-04-26T16:16:48.613 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-26T16:16:48.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: cluster 2024-04-26T16:16:47.874093+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: cluster 2024-04-26T16:16:47.874093+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: audit 2024-04-26T16:16:47.902315+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: audit 2024-04-26T16:16:47.902315+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: audit 2024-04-26T16:16:47.912611+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: audit 2024-04-26T16:16:47.912611+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: audit 2024-04-26T16:16:47.931915+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[21463]: audit 2024-04-26T16:16:47.931915+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: cluster 2024-04-26T16:16:47.874093+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-04-26T16:16:49.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: cluster 2024-04-26T16:16:47.874093+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-04-26T16:16:49.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: audit 2024-04-26T16:16:47.902315+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: audit 2024-04-26T16:16:47.902315+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: audit 2024-04-26T16:16:47.912611+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: audit 2024-04-26T16:16:47.912611+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: audit 2024-04-26T16:16:47.931915+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:49.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:48 smithi050 bash[29234]: audit 2024-04-26T16:16:47.931915+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: cluster 2024-04-26T16:16:47.874093+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: cluster 2024-04-26T16:16:47.874093+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: audit 2024-04-26T16:16:47.902315+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: audit 2024-04-26T16:16:47.902315+0000 mon.a (mon.0) 676 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: audit 2024-04-26T16:16:47.912611+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: audit 2024-04-26T16:16:47.912611+0000 mon.a (mon.0) 677 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: audit 2024-04-26T16:16:47.931915+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:48 smithi190 bash[23868]: audit 2024-04-26T16:16:47.931915+0000 mon.a (mon.0) 678 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[21463]: cluster 2024-04-26T16:16:48.730200+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v317: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[21463]: cluster 2024-04-26T16:16:48.730200+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v317: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[21463]: cluster 2024-04-26T16:16:48.872552+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[21463]: cluster 2024-04-26T16:16:48.872552+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[29234]: cluster 2024-04-26T16:16:48.730200+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v317: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[29234]: cluster 2024-04-26T16:16:48.730200+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v317: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[29234]: cluster 2024-04-26T16:16:48.872552+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-04-26T16:16:50.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:49 smithi050 bash[29234]: cluster 2024-04-26T16:16:48.872552+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-04-26T16:16:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:49 smithi190 bash[23868]: cluster 2024-04-26T16:16:48.730200+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v317: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:49 smithi190 bash[23868]: cluster 2024-04-26T16:16:48.730200+0000 mgr.y (mgr.14152) 318 : cluster [DBG] pgmap v317: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:49 smithi190 bash[23868]: cluster 2024-04-26T16:16:48.872552+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-04-26T16:16:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:49 smithi190 bash[23868]: cluster 2024-04-26T16:16:48.872552+0000 mon.a (mon.0) 679 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2024-04-26T16:16:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:50 smithi190 bash[23868]: cluster 2024-04-26T16:16:49.918776+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-04-26T16:16:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:50 smithi190 bash[23868]: cluster 2024-04-26T16:16:49.918776+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-04-26T16:16:51.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:50 smithi050 bash[21463]: cluster 2024-04-26T16:16:49.918776+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-04-26T16:16:51.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:50 smithi050 bash[21463]: cluster 2024-04-26T16:16:49.918776+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-04-26T16:16:51.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:50 smithi050 bash[29234]: cluster 2024-04-26T16:16:49.918776+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-04-26T16:16:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:50 smithi050 bash[29234]: cluster 2024-04-26T16:16:49.918776+0000 mon.a (mon.0) 680 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2024-04-26T16:16:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:51 smithi190 bash[23868]: cluster 2024-04-26T16:16:50.730775+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v320: 1 pgs: 1 remapped+peering; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:51 smithi190 bash[23868]: cluster 2024-04-26T16:16:50.730775+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v320: 1 pgs: 1 remapped+peering; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:51 smithi190 bash[23868]: cluster 2024-04-26T16:16:50.925849+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-04-26T16:16:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:51 smithi190 bash[23868]: cluster 2024-04-26T16:16:50.925849+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-04-26T16:16:52.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[29234]: cluster 2024-04-26T16:16:50.730775+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v320: 1 pgs: 1 remapped+peering; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[29234]: cluster 2024-04-26T16:16:50.730775+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v320: 1 pgs: 1 remapped+peering; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[29234]: cluster 2024-04-26T16:16:50.925849+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[29234]: cluster 2024-04-26T16:16:50.925849+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[21463]: cluster 2024-04-26T16:16:50.730775+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v320: 1 pgs: 1 remapped+peering; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[21463]: cluster 2024-04-26T16:16:50.730775+0000 mgr.y (mgr.14152) 319 : cluster [DBG] pgmap v320: 1 pgs: 1 remapped+peering; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[21463]: cluster 2024-04-26T16:16:50.925849+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-04-26T16:16:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:51 smithi050 bash[21463]: cluster 2024-04-26T16:16:50.925849+0000 mon.a (mon.0) 681 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2024-04-26T16:16:53.379 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:16:54.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:53 smithi190 bash[23868]: cluster 2024-04-26T16:16:52.731370+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v322: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:54.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:53 smithi190 bash[23868]: cluster 2024-04-26T16:16:52.731370+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v322: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:53 smithi050 bash[21463]: cluster 2024-04-26T16:16:52.731370+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v322: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:53 smithi050 bash[21463]: cluster 2024-04-26T16:16:52.731370+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v322: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:54.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:53 smithi050 bash[29234]: cluster 2024-04-26T16:16:52.731370+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v322: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:54.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:53 smithi050 bash[29234]: cluster 2024-04-26T16:16:52.731370+0000 mgr.y (mgr.14152) 320 : cluster [DBG] pgmap v322: 1 pgs: 1 remapped+peering; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:16:56.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:55 smithi050 bash[21463]: cluster 2024-04-26T16:16:54.731807+0000 mgr.y (mgr.14152) 321 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:56.257 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:55 smithi050 bash[21463]: cluster 2024-04-26T16:16:54.731807+0000 mgr.y (mgr.14152) 321 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:56.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:55 smithi050 bash[29234]: cluster 2024-04-26T16:16:54.731807+0000 mgr.y (mgr.14152) 321 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:56.258 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:55 smithi050 bash[29234]: cluster 2024-04-26T16:16:54.731807+0000 mgr.y (mgr.14152) 321 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:56.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:55 smithi190 bash[23868]: cluster 2024-04-26T16:16:54.731807+0000 mgr.y (mgr.14152) 321 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:56.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:55 smithi190 bash[23868]: cluster 2024-04-26T16:16:54.731807+0000 mgr.y (mgr.14152) 321 : cluster [DBG] pgmap v323: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-26T16:16:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- 172.21.15.50:0/2012779306 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 msgr2=0x7f5bdc102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- 172.21.15.50:0/2012779306 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc102000 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0098f0 tx=0x7f5bcc009c00 comp rx=0 tx=0).stop 2024-04-26T16:16:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- 172.21.15.50:0/2012779306 shutdown_connections 2024-04-26T16:16:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- 172.21.15.50:0/2012779306 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 0x7f5bdc10a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- 172.21.15.50:0/2012779306 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc102000 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- 172.21.15.50:0/2012779306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bdc102de0 0x7f5bdc103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- 172.21.15.50:0/2012779306 >> 172.21.15.50:0/2012779306 conn(0x7f5bdc0fd180 msgr2=0x7f5bdc0ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:16:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- 172.21.15.50:0/2012779306 shutdown_connections 2024-04-26T16:16:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- 172.21.15.50:0/2012779306 wait complete. 2024-04-26T16:16:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 librados: starting msgr at 2024-04-26T16:16:56.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 librados: starting objecter 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 Processor -- start 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- start start 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 librados: setting wanted keys 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 librados: calling monclient init 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bdc102de0 0x7f5bdc199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 0x7f5bdc1a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc10d3b0 con 0x7f5bdc103780 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc10d230 con 0x7f5bdc102de0 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5be0e91700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5bdc10d530 con 0x7f5bdc101be0 2024-04-26T16:16:56.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5bda59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5bda59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40414/0 (socket says 172.21.15.50:40414) 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5bda59c700 1 -- 172.21.15.50:0/3090552950 learned_addr learned my addr 172.21.15.50:0/3090552950 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5bdad9d700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 0x7f5bdc1a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.424+0000 7f5bd9d9b700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bdc102de0 0x7f5bdc199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bdad9d700 1 -- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 msgr2=0x7f5bdc198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bdad9d700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bdad9d700 1 -- 172.21.15.50:0/3090552950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bdc102de0 msgr2=0x7f5bdc199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bdad9d700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bdc102de0 0x7f5bdc199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bdad9d700 1 -- 172.21.15.50:0/3090552950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc1a0990 con 0x7f5bdc103780 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bda59c700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bdad9d700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 0x7f5bdc1a02c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009cd0 tx=0x7f5bd400c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 10 client.?.objecter ms_handle_connect 0x7f5bdc103780 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:16:56.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd400cd10 con 0x7f5bdc103780 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5bd4003a00 con 0x7f5bdc103780 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc1a0c20 con 0x7f5bdc103780 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4004370 con 0x7f5bdc103780 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5be0e91700 10 client.14358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc1a1160 con 0x7f5bdc103780 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5be0e91700 1 librados: init done 2024-04-26T16:16:56.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bbc005150 con 0x7f5bdc103780 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f5bd401a020 con 0x7f5bdc103780 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f5bc4071e90 0x7f5bc4074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(73..73 src has 22..73) ==== 5287+0+0 (secure 0 0 0) 0x7f5bd4090020 con 0x7f5bdc103780 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 10 client.14358.objecter ms_dispatch 0x7f5bdc001550 osd_map(73..73 src has 22..73) 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 3 client.14358.objecter handle_osd_map got epochs [73,73] > 0 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 3 client.14358.objecter handle_osd_map decoding full epoch 73 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bda59c700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f5bc4071e90 0x7f5bc4074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 20 client.14358.objecter dump_active .. 0 homeless 2024-04-26T16:16:56.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bcb7fe700 10 client.14358.objecter ms_handle_connect 0x7f5bc4071e90 2024-04-26T16:16:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.428+0000 7f5bda59c700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f5bc4071e90 0x7f5bc4074350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0098c0 tx=0x7f5bcc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:16:56.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.436+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5bd4010040 con 0x7f5bdc103780 2024-04-26T16:16:56.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.652+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5bbc005710 con 0x7f5bdc103780 2024-04-26T16:16:56.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.652+0000 7f5bcb7fe700 1 -- 172.21.15.50:0/3090552950 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+130 (secure 0 0 0) 0x7f5bd405caa0 con 0x7f5bdc103780 2024-04-26T16:16:56.646 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:16:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 10 librados: watch_flush enter 2024-04-26T16:16:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 10 librados: watch_flush exit 2024-04-26T16:16:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 20 client.14358.objecter shutdown clearing up homeless session... 2024-04-26T16:16:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 10 client.14358.objecter successfully canceled tick 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f5bc4071e90 msgr2=0x7f5bc4074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f5bc4071e90 0x7f5bc4074350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0098c0 tx=0x7f5bcc019040 comp rx=0 tx=0).stop 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 msgr2=0x7f5bdc1a02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 0x7f5bdc1a02c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009cd0 tx=0x7f5bd400c620 comp rx=0 tx=0).stop 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 shutdown_connections 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bdc103780 0x7f5bdc1a02c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bdc101be0 0x7f5bdc198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f5bc4071e90 0x7f5bc4074350 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 --2- 172.21.15.50:0/3090552950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bdc102de0 0x7f5bdc199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 >> 172.21.15.50:0/3090552950 conn(0x7f5bdc0fd180 msgr2=0x7f5bdc10b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 shutdown_connections 2024-04-26T16:16:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 -- 172.21.15.50:0/3090552950 wait complete. 2024-04-26T16:16:56.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:16:56.656+0000 7f5be0e91700 1 librados: shutdown 2024-04-26T16:16:57.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:56 smithi190 bash[23868]: audit 2024-04-26T16:16:56.656641+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.50:0/3090552950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T16:16:57.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:56 smithi190 bash[23868]: audit 2024-04-26T16:16:56.656641+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.50:0/3090552950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T16:16:57.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:56 smithi050 bash[21463]: audit 2024-04-26T16:16:56.656641+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.50:0/3090552950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T16:16:57.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:56 smithi050 bash[21463]: audit 2024-04-26T16:16:56.656641+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.50:0/3090552950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T16:16:57.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:56 smithi050 bash[29234]: audit 2024-04-26T16:16:56.656641+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.50:0/3090552950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T16:16:57.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:56 smithi050 bash[29234]: audit 2024-04-26T16:16:56.656641+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.50:0/3090552950' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-26T16:16:57.461 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":73,"num_osds":8,"num_up_osds":8,"osd_up_since":1714148206,"num_in_osds":8,"osd_in_since":1714148180,"num_remapped_pgs":0} 2024-04-26T16:16:57.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T16:16:58.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: cluster 2024-04-26T16:16:56.732330+0000 mgr.y (mgr.14152) 322 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: cluster 2024-04-26T16:16:56.732330+0000 mgr.y (mgr.14152) 322 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.205642+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.205642+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.216129+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.216129+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.218794+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.218794+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.220566+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.220566+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.222236+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.222236+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.224017+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.224017+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.229921+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.229921+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.232573+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.232573+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.234119+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.234119+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.241358+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:57 smithi190 bash[23868]: audit 2024-04-26T16:16:57.241358+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: cluster 2024-04-26T16:16:56.732330+0000 mgr.y (mgr.14152) 322 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: cluster 2024-04-26T16:16:56.732330+0000 mgr.y (mgr.14152) 322 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.205642+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.205642+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.216129+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.216129+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.218794+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.218794+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.220566+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.220566+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.222236+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.222236+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.224017+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.224017+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.229921+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.229921+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.232573+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:58.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.232573+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: cluster 2024-04-26T16:16:56.732330+0000 mgr.y (mgr.14152) 322 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: cluster 2024-04-26T16:16:56.732330+0000 mgr.y (mgr.14152) 322 : cluster [DBG] pgmap v324: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.205642+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.205642+0000 mon.a (mon.0) 683 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.216129+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.216129+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.218794+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.218794+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.220566+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.220566+0000 mon.a (mon.0) 686 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.222236+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.222236+0000 mon.a (mon.0) 687 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.224017+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.224017+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.229921+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.229921+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.232573+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.232573+0000 mon.a (mon.0) 690 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.234119+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.234119+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.241358+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[21463]: audit 2024-04-26T16:16:57.241358+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.234119+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:58.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.234119+0000 mon.a (mon.0) 691 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:16:58.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.241358+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:58.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:57 smithi050 bash[29234]: audit 2024-04-26T16:16:57.241358+0000 mon.a (mon.0) 692 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:16:59.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:58 smithi190 bash[23868]: cephadm 2024-04-26T16:16:57.195762+0000 mgr.y (mgr.14152) 323 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:59.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:58 smithi190 bash[23868]: cephadm 2024-04-26T16:16:57.195762+0000 mgr.y (mgr.14152) 323 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:59.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:58 smithi190 bash[23868]: cephadm 2024-04-26T16:16:57.225321+0000 mgr.y (mgr.14152) 324 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 4316M 2024-04-26T16:16:59.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:58 smithi190 bash[23868]: cephadm 2024-04-26T16:16:57.225321+0000 mgr.y (mgr.14152) 324 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 4316M 2024-04-26T16:16:59.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[29234]: cephadm 2024-04-26T16:16:57.195762+0000 mgr.y (mgr.14152) 323 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:59.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[29234]: cephadm 2024-04-26T16:16:57.195762+0000 mgr.y (mgr.14152) 323 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:59.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[29234]: cephadm 2024-04-26T16:16:57.225321+0000 mgr.y (mgr.14152) 324 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 4316M 2024-04-26T16:16:59.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[29234]: cephadm 2024-04-26T16:16:57.225321+0000 mgr.y (mgr.14152) 324 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 4316M 2024-04-26T16:16:59.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[21463]: cephadm 2024-04-26T16:16:57.195762+0000 mgr.y (mgr.14152) 323 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:59.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[21463]: cephadm 2024-04-26T16:16:57.195762+0000 mgr.y (mgr.14152) 323 : cephadm [INF] Detected new or changed devices on smithi190 2024-04-26T16:16:59.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[21463]: cephadm 2024-04-26T16:16:57.225321+0000 mgr.y (mgr.14152) 324 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 4316M 2024-04-26T16:16:59.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:58 smithi050 bash[21463]: cephadm 2024-04-26T16:16:57.225321+0000 mgr.y (mgr.14152) 324 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 4316M 2024-04-26T16:17:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:59 smithi190 bash[23868]: cluster 2024-04-26T16:16:58.732965+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:17:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:16:59 smithi190 bash[23868]: cluster 2024-04-26T16:16:58.732965+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:17:00.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:59 smithi050 bash[21463]: cluster 2024-04-26T16:16:58.732965+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:17:00.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:16:59 smithi050 bash[21463]: cluster 2024-04-26T16:16:58.732965+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:17:00.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:59 smithi050 bash[29234]: cluster 2024-04-26T16:16:58.732965+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:17:00.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:16:59 smithi050 bash[29234]: cluster 2024-04-26T16:16:58.732965+0000 mgr.y (mgr.14152) 325 : cluster [DBG] pgmap v325: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-26T16:17:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:00 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.729873+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:17:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:00 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.729873+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:17:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:00 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.733494+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:17:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:00 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.733494+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:17:01.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.729873+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:17:01.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.729873+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:17:01.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.733494+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:17:01.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.733494+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:17:01.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.729873+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:17:01.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.729873+0000 osd.7 (osd.7) 3 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:17:01.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.733494+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:17:01.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:00 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.733494+0000 osd.7 (osd.7) 4 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:17:02.227 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.233137+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.233137+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.233249+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.233249+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.733564+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.733564+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.968788+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[29234]: cluster 2024-04-26T16:17:00.968788+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.233137+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.233137+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.233249+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:02.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.233249+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:02.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.733564+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:02.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.733564+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:02.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.968788+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-04-26T16:17:02.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:01 smithi050 bash[21463]: cluster 2024-04-26T16:17:00.968788+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.233137+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.233137+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.233249+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.233249+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.733564+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.733564+0000 mgr.y (mgr.14152) 326 : cluster [DBG] pgmap v326: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.968788+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-04-26T16:17:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:01 smithi190 bash[23868]: cluster 2024-04-26T16:17:00.968788+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2024-04-26T16:17:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.177617+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.177617+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.177717+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.177717+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.734255+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:04.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.734255+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:04.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.975520+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-04-26T16:17:04.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:03 smithi190 bash[23868]: cluster 2024-04-26T16:17:02.975520+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-04-26T16:17:04.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.177617+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.177617+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.177717+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.177717+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.734255+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.734255+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.975520+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[21463]: cluster 2024-04-26T16:17:02.975520+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.177617+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:04.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.177617+0000 osd.3 (osd.3) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.177717+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.177717+0000 osd.3 (osd.3) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.734255+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.734255+0000 mgr.y (mgr.14152) 327 : cluster [DBG] pgmap v328: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-26T16:17:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.975520+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-04-26T16:17:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:03 smithi050 bash[29234]: cluster 2024-04-26T16:17:02.975520+0000 mon.a (mon.0) 694 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2024-04-26T16:17:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/1547964382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e7006bbf0 msgr2=0x7f6e7006c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- 172.21.15.50:0/1547964382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e7006bbf0 0x7f6e7006c070 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6e600098f0 tx=0x7f6e60009c00 comp rx=0 tx=0).stop 2024-04-26T16:17:05.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/1547964382 shutdown_connections 2024-04-26T16:17:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- 172.21.15.50:0/1547964382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e70109540 0x7f6e70109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- 172.21.15.50:0/1547964382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e7006c5b0 0x7f6e70110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- 172.21.15.50:0/1547964382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e7006bbf0 0x7f6e7006c070 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/1547964382 >> 172.21.15.50:0/1547964382 conn(0x7f6e70078230 msgr2=0x7f6e70078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/1547964382 shutdown_connections 2024-04-26T16:17:05.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/1547964382 wait complete. 2024-04-26T16:17:05.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 librados: starting msgr at 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 librados: starting objecter 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 Processor -- start 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- start start 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 librados: setting wanted keys 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 librados: calling monclient init 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e7006bbf0 0x7f6e70102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e7006c5b0 0x7f6e70102a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 0x7f6e70102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6e70112b70 con 0x7f6e7006c5b0 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6e701129f0 con 0x7f6e7006bbf0 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6e70112cf0 con 0x7f6e70109540 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 0x7f6e70102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 0x7f6e70102fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39932/0 (socket says 172.21.15.50:39932) 2024-04-26T16:17:05.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 -- 172.21.15.50:0/366779391 learned_addr learned my addr 172.21.15.50:0/366779391 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e67fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e7006c5b0 0x7f6e70102a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e74b48700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e7006bbf0 0x7f6e70102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 -- 172.21.15.50:0/366779391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e7006bbf0 msgr2=0x7f6e70102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e7006bbf0 0x7f6e70102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 -- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e7006c5b0 msgr2=0x7f6e70102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e7006c5b0 0x7f6e70102a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 -- 172.21.15.50:0/366779391 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e701a76e0 con 0x7f6e70109540 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e75349700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 0x7f6e70102fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800e3c0 tx=0x7f6e6800e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e65ffb700 10 client.?.objecter ms_handle_connect 0x7f6e70109540 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e65ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e680092d0 con 0x7f6e70109540 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e701a79d0 con 0x7f6e70109540 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6e6800ee70 con 0x7f6e70109540 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e68007c70 con 0x7f6e70109540 2024-04-26T16:17:05.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 10 client.24278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:17:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e701a7ee0 con 0x7f6e70109540 2024-04-26T16:17:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.200+0000 7f6e76dac700 1 librados: init done 2024-04-26T16:17:05.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e70104bb0 con 0x7f6e70109540 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f6e68012050 con 0x7f6e70109540 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f6e58071e90 0x7f6e58074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(75..75 src has 22..75) ==== 5287+0+0 (secure 0 0 0) 0x7f6e68090490 con 0x7f6e70109540 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 10 client.24278.objecter ms_dispatch 0x7f6e70001550 osd_map(75..75 src has 22..75) 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 3 client.24278.objecter handle_osd_map got epochs [75,75] > 0 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 3 client.24278.objecter handle_osd_map decoding full epoch 75 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e74b48700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f6e58071e90 0x7f6e58074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 20 client.24278.objecter dump_active .. 0 homeless 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e65ffb700 10 client.24278.objecter ms_handle_connect 0x7f6e58071e90 2024-04-26T16:17:05.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.204+0000 7f6e74b48700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f6e58071e90 0x7f6e58074350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c005950 tx=0x7f6e6c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:05.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.208+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6e68022060 con 0x7f6e70109540 2024-04-26T16:17:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.432+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e70103a50 con 0x7f6e70109540 2024-04-26T16:17:05.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.432+0000 7f6e65ffb700 1 -- 172.21.15.50:0/366779391 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+13562 (secure 0 0 0) 0x7f6e68010040 con 0x7f6e70109540 2024-04-26T16:17:05.426 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:17:05.426 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":75,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","created":"2024-04-26T16:07:04.494213+0000","modified":"2024-04-26T16:17:02.962246+0000","last_up_change":"2024-04-26T16:16:46.862501+0000","last_in_change":"2024-04-26T16:16:20.945042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T16:12:30.809607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b46ea8aa-cd97-425b-90e0-1e70af46b8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6803","nonce":310950989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6805","nonce":310950989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6809","nonce":310950989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6807","nonce":310950989}]},"public_addr":"172.21.15.50:6803/310950989","cluster_addr":"172.21.15.50:6805/310950989","heartbeat_back_addr":"172.21.15.50:6809/310950989","heartbeat_front_addr":"172.21.15.50:6807/310950989","state":["exists","up"]},{"osd":1,"uuid":"7042f8e0-f5d0-48d2-9cd6-d9445e326eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6811","nonce":3516275849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6813","nonce":3516275849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6817","nonce":3516275849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6815","nonce":3516275849}]},"public_addr":"172.21.15.50:6811/3516275849","cluster_addr":"172.21.15.50:6813/3516275849","heartbeat_back_addr":"172.21.15.50:6817/3516275849","heartbeat_front_addr":"172.21.15.50:6815/3516275849","state":["exists","up"]},{"osd":2,"uuid":"33069daa-7fd8-4e9c-aafe-89030f84aad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6819","nonce":2835380668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6821","nonce":2835380668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6825","nonce":2835380668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6823","nonce":2835380668}]},"public_addr":"172.21.15.50:6819/2835380668","cluster_addr":"172.21.15.50:6821/2835380668","heartbeat_back_addr":"172.21.15.50:6825/2835380668","heartbeat_front_addr":"172.21.15.50:6823/2835380668","state":["exists","up"]},{"osd":3,"uuid":"43b6aab5-9b42-4f97-a224-b2872de5cb03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6827","nonce":2541695077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6829","nonce":2541695077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6833","nonce":2541695077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6831","nonce":2541695077}]},"public_addr":"172.21.15.50:6827/2541695077","cluster_addr":"172.21.15.50:6829/2541695077","heartbeat_back_addr":"172.21.15.50:6833/2541695077","heartbeat_front_addr":"172.21.15.50:6831/2541695077","state":["exists","up"]},{"osd":4,"uuid":"f7220491-e9bf-4188-b203-322e9908558c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6801","nonce":2207965503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6803","nonce":2207965503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6807","nonce":2207965503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6805","nonce":2207965503}]},"public_addr":"172.21.15.190:6801/2207965503","cluster_addr":"172.21.15.190:6803/2207965503","heartbeat_back_addr":"172.21.15.190:6807/2207965503","heartbeat_front_addr":"172.21.15.190:6805/2207965503","state":["exists","up"]},{"osd":5,"uuid":"ed5c0044-658b-4dfb-8c61-727de6bdd587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6809","nonce":3373187844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6811","nonce":3373187844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6815","nonce":3373187844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6813","nonce":3373187844}]},"public_addr":"172.21.15.190:6809/3373187844","cluster_addr":"172.21.15.190:6811/3373187844","heartbeat_back_addr":"172.21.15.190:6815/3373187844","heartbeat_front_addr":"172.21.15.190:6813/3373187844","state":["exists","up"]},{"osd":6,"uuid":"94111b7d-7887-4a3a-983b-394294735d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6817","nonce":2399532574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6819","nonce":2399532574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6823","nonce":2399532574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6821","nonce":2399532574}]},"public_addr":"172.21.15.190:6817/2399532574","cluster_addr":"172.21.15.190:6819/2399532574","heartbeat_back_addr":"172.21.15.190:6823/2399532574","heartbeat_front_addr":"172.21.15.190:6821/2399532574","state":["exists","up"]},{"osd":7,"uuid":"abbd95e3-07b4-4235-9101-a4dea0f35bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6825","nonce":2856617796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6827","nonce":2856617796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6831","nonce":2856617796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6829","nonce":2856617796}]},"public_addr":"172.21.15.190:6825/2856617796","cluster_addr":"172.21.15.190:6827/2856617796","heartbeat_back_addr":"172.21.15.190:6831/2856617796","heartbeat_front_addr":"172.21.15.190:6829/2856617796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:15:55.097964+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:16:04.932155+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:16:27.876688+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:17:02.177724+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:17:00.233257+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:16:21.407374+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:15:53.076518+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:16:44.813236+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/3973398204":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/71201882":"2024-04-27T16:08:14.593126+0000","172.21.15.50:6801/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2903276422":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2548717623":"2024-04-27T16:07:37.705234+0000","172.21.15.50:0/3454723725":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:0/2324264713":"2024-04-27T16:08:14.593126+0000","172.21.15.50:6801/127073210":"2024-04-27T16:07:37.705234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T16:17:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 10 librados: watch_flush enter 2024-04-26T16:17:05.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 10 librados: watch_flush exit 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 20 client.24278.objecter shutdown clearing up homeless session... 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 10 client.24278.objecter successfully canceled tick 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f6e58071e90 msgr2=0x7f6e58074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f6e58071e90 0x7f6e58074350 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c005950 tx=0x7f6e6c0058e0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 msgr2=0x7f6e70102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 0x7f6e70102fa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800e3c0 tx=0x7f6e6800e6d0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 shutdown_connections 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f6e58071e90 0x7f6e58074350 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6e7006bbf0 0x7f6e70102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6e70109540 0x7f6e70102fa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 --2- 172.21.15.50:0/366779391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6e7006c5b0 0x7f6e70102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 >> 172.21.15.50:0/366779391 conn(0x7f6e70078230 msgr2=0x7f6e70100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 shutdown_connections 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 -- 172.21.15.50:0/366779391 wait complete. 2024-04-26T16:17:05.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:05.436+0000 7f6e76dac700 1 librados: shutdown 2024-04-26T16:17:06.206 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T16:12:30.809607+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T16:17:06.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[21463]: cluster 2024-04-26T16:17:04.734910+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[21463]: cluster 2024-04-26T16:17:04.734910+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[21463]: audit 2024-04-26T16:17:05.436603+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.50:0/366779391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[21463]: audit 2024-04-26T16:17:05.436603+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.50:0/366779391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[29234]: cluster 2024-04-26T16:17:04.734910+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[29234]: cluster 2024-04-26T16:17:04.734910+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:06.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[29234]: audit 2024-04-26T16:17:05.436603+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.50:0/366779391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:17:06.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:05 smithi050 bash[29234]: audit 2024-04-26T16:17:05.436603+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.50:0/366779391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:17:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:05 smithi190 bash[23868]: cluster 2024-04-26T16:17:04.734910+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:05 smithi190 bash[23868]: cluster 2024-04-26T16:17:04.734910+0000 mgr.y (mgr.14152) 328 : cluster [DBG] pgmap v330: 1 pgs: 1 active+clean; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:05 smithi190 bash[23868]: audit 2024-04-26T16:17:05.436603+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.50:0/366779391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:17:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:05 smithi190 bash[23868]: audit 2024-04-26T16:17:05.436603+0000 mon.c (mon.2) 10 : audit [DBG] from='client.? 172.21.15.50:0/366779391' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:17:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:07 smithi190 bash[23868]: cluster 2024-04-26T16:17:06.735513+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:08.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:07 smithi190 bash[23868]: cluster 2024-04-26T16:17:06.735513+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:08.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:07 smithi050 bash[21463]: cluster 2024-04-26T16:17:06.735513+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:08.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:07 smithi050 bash[21463]: cluster 2024-04-26T16:17:06.735513+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:08.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:07 smithi050 bash[29234]: cluster 2024-04-26T16:17:06.735513+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:08.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:07 smithi050 bash[29234]: cluster 2024-04-26T16:17:06.735513+0000 mgr.y (mgr.14152) 329 : cluster [DBG] pgmap v331: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:09 smithi190 bash[23868]: cluster 2024-04-26T16:17:08.736093+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:09 smithi190 bash[23868]: cluster 2024-04-26T16:17:08.736093+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:09 smithi050 bash[21463]: cluster 2024-04-26T16:17:08.736093+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:09 smithi050 bash[21463]: cluster 2024-04-26T16:17:08.736093+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:09 smithi050 bash[29234]: cluster 2024-04-26T16:17:08.736093+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:09 smithi050 bash[29234]: cluster 2024-04-26T16:17:08.736093+0000 mgr.y (mgr.14152) 330 : cluster [DBG] pgmap v332: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:10.970 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:17:12.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:11 smithi050 bash[21463]: cluster 2024-04-26T16:17:10.736703+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:12.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:11 smithi050 bash[21463]: cluster 2024-04-26T16:17:10.736703+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:12.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:11 smithi050 bash[29234]: cluster 2024-04-26T16:17:10.736703+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:12.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:11 smithi050 bash[29234]: cluster 2024-04-26T16:17:10.736703+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:11 smithi190 bash[23868]: cluster 2024-04-26T16:17:10.736703+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:11 smithi190 bash[23868]: cluster 2024-04-26T16:17:10.736703+0000 mgr.y (mgr.14152) 331 : cluster [DBG] pgmap v333: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- 172.21.15.50:0/811811134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 msgr2=0x7fe10c1099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- 172.21.15.50:0/811811134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c1099a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc0098f0 tx=0x7fe0fc009c00 comp rx=0 tx=0).stop 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- 172.21.15.50:0/811811134 shutdown_connections 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- 172.21.15.50:0/811811134 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 0x7fe10c110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- 172.21.15.50:0/811811134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c1099a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- 172.21.15.50:0/811811134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe10c06bb60 0x7fe10c06bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- 172.21.15.50:0/811811134 >> 172.21.15.50:0/811811134 conn(0x7fe10c079120 msgr2=0x7fe10c079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- 172.21.15.50:0/811811134 shutdown_connections 2024-04-26T16:17:13.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- 172.21.15.50:0/811811134 wait complete. 2024-04-26T16:17:13.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 librados: starting msgr at 2024-04-26T16:17:13.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 librados: starting objecter 2024-04-26T16:17:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 Processor -- start 2024-04-26T16:17:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- start start 2024-04-26T16:17:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 librados: setting wanted keys 2024-04-26T16:17:13.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 librados: calling monclient init 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe10c06bb60 0x7fe10c1026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 0x7fe10c102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c112b00 con 0x7fe10c1095c0 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe10c112980 con 0x7fe10c06bb60 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe1110b7700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe10c112c80 con 0x7fe10c06c520 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56750/0 (socket says 172.21.15.50:56750) 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 -- 172.21.15.50:0/1790889591 learned_addr learned my addr 172.21.15.50:0/1790889591 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10a59c700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 0x7fe10c102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10ad9d700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe10c06bb60 0x7fe10c1026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 -- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 msgr2=0x7fe10c102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:13.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 0x7fe10c102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 -- 172.21.15.50:0/1790889591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe10c06bb60 msgr2=0x7fe10c1026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe10c06bb60 0x7fe10c1026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10b59e700 1 -- 172.21.15.50:0/1790889591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe10c1a76d0 con 0x7fe10c1095c0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.784+0000 7fe10a59c700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 0x7fe10c102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe10b59e700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c103170 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c340 tx=0x7fe10400c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 10 client.?.objecter ms_handle_connect 0x7fe10c1095c0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104019070 con 0x7fe10c1095c0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe104003a00 con 0x7fe10c1095c0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104007af0 con 0x7fe10c1095c0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe10c1a79c0 con 0x7fe10c1095c0 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe1110b7700 10 client.14370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:17:13.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe10c1a7f00 con 0x7fe10c1095c0 2024-04-26T16:17:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe1110b7700 1 librados: init done 2024-04-26T16:17:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe10c06a110 con 0x7fe10c1095c0 2024-04-26T16:17:13.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7fe10401c070 con 0x7fe10c1095c0 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fe0f4071e90 0x7fe0f4074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe10ad9d700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fe0f4071e90 0x7fe0f4074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(76..76 src has 22..76) ==== 5287+0+0 (secure 0 0 0) 0x7fe104090270 con 0x7fe10c1095c0 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 10 client.14370.objecter ms_dispatch 0x7fe10c001550 osd_map(76..76 src has 22..76) 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 3 client.14370.objecter handle_osd_map got epochs [76,76] > 0 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 3 client.14370.objecter handle_osd_map decoding full epoch 76 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 20 client.14370.objecter dump_active .. 0 homeless 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe0f3fff700 10 client.14370.objecter ms_handle_connect 0x7fe0f4071e90 2024-04-26T16:17:13.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.788+0000 7fe10ad9d700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fe0f4071e90 0x7fe0f4074350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009fd0 tx=0x7fe0fc005b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:13.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:13.792+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fe104010040 con 0x7fe10c1095c0 2024-04-26T16:17:14.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.012+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe10c1a8190 con 0x7fe10c1095c0 2024-04-26T16:17:14.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.012+0000 7fe0f3fff700 1 -- 172.21.15.50:0/1790889591 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v76) ==== 93+0+10 (secure 0 0 0) 0x7fe10405ccf0 con 0x7fe10c1095c0 2024-04-26T16:17:14.006 INFO:teuthology.orchestra.run.smithi050.stdout:pg_num: 1 2024-04-26T16:17:14.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 10 librados: watch_flush enter 2024-04-26T16:17:14.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 10 librados: watch_flush exit 2024-04-26T16:17:14.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 20 client.14370.objecter shutdown clearing up homeless session... 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 10 client.14370.objecter successfully canceled tick 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fe0f4071e90 msgr2=0x7fe0f4074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fe0f4071e90 0x7fe0f4074350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc009fd0 tx=0x7fe0fc005b40 comp rx=0 tx=0).stop 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 msgr2=0x7fe10c103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c103170 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe10400c340 tx=0x7fe10400c650 comp rx=0 tx=0).stop 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 shutdown_connections 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fe0f4071e90 0x7fe0f4074350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe10c06bb60 0x7fe10c1026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe10c1095c0 0x7fe10c103170 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 --2- 172.21.15.50:0/1790889591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe10c06c520 0x7fe10c102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 >> 172.21.15.50:0/1790889591 conn(0x7fe10c079120 msgr2=0x7fe10c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 shutdown_connections 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 -- 172.21.15.50:0/1790889591 wait complete. 2024-04-26T16:17:14.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:17:14.016+0000 7fe1110b7700 1 librados: shutdown 2024-04-26T16:17:14.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:12.205966+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:14.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:12.205966+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:14.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:12.206083+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:12.206083+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:12.737296+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:12.737296+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:13.000727+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[29234]: cluster 2024-04-26T16:17:13.000727+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:12.205966+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:12.205966+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:12.206083+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:12.206083+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:12.737296+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:14.246 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:12.737296+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:14.247 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:13.000727+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-04-26T16:17:14.247 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:13 smithi050 bash[21463]: cluster 2024-04-26T16:17:13.000727+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:12.205966+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:12.205966+0000 osd.6 (osd.6) 3 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:12.206083+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:12.206083+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:12.737296+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:12.737296+0000 mgr.y (mgr.14152) 332 : cluster [DBG] pgmap v334: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:13.000727+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-04-26T16:17:14.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:13 smithi190 bash[23868]: cluster 2024-04-26T16:17:13.000727+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2024-04-26T16:17:14.800 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi050 2024-04-26T16:17:14.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi050=foo.a' 2024-04-26T16:17:15.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:14 smithi190 bash[23868]: audit 2024-04-26T16:17:14.016982+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.50:0/1790889591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T16:17:15.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:14 smithi190 bash[23868]: audit 2024-04-26T16:17:14.016982+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.50:0/1790889591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T16:17:15.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:14 smithi050 bash[21463]: audit 2024-04-26T16:17:14.016982+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.50:0/1790889591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T16:17:15.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:15 smithi050 bash[21463]: audit 2024-04-26T16:17:14.016982+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.50:0/1790889591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T16:17:15.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:15 smithi050 bash[29234]: audit 2024-04-26T16:17:14.016982+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.50:0/1790889591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T16:17:15.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:15 smithi050 bash[29234]: audit 2024-04-26T16:17:14.016982+0000 mon.a (mon.0) 696 : audit [DBG] from='client.? 172.21.15.50:0/1790889591' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-26T16:17:16.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:16 smithi190 bash[23868]: cluster 2024-04-26T16:17:14.737887+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:16.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:16 smithi190 bash[23868]: cluster 2024-04-26T16:17:14.737887+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:16 smithi050 bash[29234]: cluster 2024-04-26T16:17:14.737887+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:16 smithi050 bash[29234]: cluster 2024-04-26T16:17:14.737887+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:16.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:16 smithi050 bash[21463]: cluster 2024-04-26T16:17:14.737887+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:16.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:16 smithi050 bash[21463]: cluster 2024-04-26T16:17:14.737887+0000 mgr.y (mgr.14152) 333 : cluster [DBG] pgmap v336: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:18 smithi190 bash[23868]: cluster 2024-04-26T16:17:16.738494+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:18.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:18 smithi190 bash[23868]: cluster 2024-04-26T16:17:16.738494+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:18.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:18 smithi050 bash[21463]: cluster 2024-04-26T16:17:16.738494+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:18.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:18 smithi050 bash[21463]: cluster 2024-04-26T16:17:16.738494+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:18.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:18 smithi050 bash[29234]: cluster 2024-04-26T16:17:16.738494+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:18.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:18 smithi050 bash[29234]: cluster 2024-04-26T16:17:16.738494+0000 mgr.y (mgr.14152) 334 : cluster [DBG] pgmap v337: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:19 smithi190 bash[23868]: cluster 2024-04-26T16:17:18.739239+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:19 smithi190 bash[23868]: cluster 2024-04-26T16:17:18.739239+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:19 smithi050 bash[21463]: cluster 2024-04-26T16:17:18.739239+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:19 smithi050 bash[21463]: cluster 2024-04-26T16:17:18.739239+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:19 smithi050 bash[29234]: cluster 2024-04-26T16:17:18.739239+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:19 smithi050 bash[29234]: cluster 2024-04-26T16:17:18.739239+0000 mgr.y (mgr.14152) 335 : cluster [DBG] pgmap v338: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:19.550 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/826811323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c075e80 msgr2=0x7ff55c076300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- 172.21.15.190:0/826811323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c075e80 0x7ff55c076300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff550009990 tx=0x7ff550009ca0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/826811323 shutdown_connections 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- 172.21.15.190:0/826811323 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff55c10fa50 0x7ff55c111e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- 172.21.15.190:0/826811323 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 0x7ff55c0758b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- 172.21.15.190:0/826811323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c075e80 0x7ff55c076300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/826811323 >> 172.21.15.190:0/826811323 conn(0x7ff55c0fea10 msgr2=0x7ff55c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/826811323 shutdown_connections 2024-04-26T16:17:22.067 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/826811323 wait complete. 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 librados: starting msgr at 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 librados: starting objecter 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 Processor -- start 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- start start 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 librados: setting wanted keys 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 librados: calling monclient init 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff55c075e80 0x7ff55c19f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 0x7ff55c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 0x7ff55c1a6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c1149e0 con 0x7ff55c077450 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff55c114860 con 0x7ff55c10fa50 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff55c114b60 con 0x7ff55c075e80 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 0x7ff55c1a6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 0x7ff55c1a6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:57018/0 (socket says 172.21.15.190:57018) 2024-04-26T16:17:22.068 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 -- 172.21.15.190:0/712728519 learned_addr learned my addr 172.21.15.190:0/712728519 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:17:22.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55a59c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff55c075e80 0x7ff55c19f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:22.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff559d9b700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 0x7ff55c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:22.070 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 -- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff55c075e80 msgr2=0x7ff55c19f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff55c075e80 0x7ff55c19f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 -- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 msgr2=0x7ff55c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 0x7ff55c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 -- 172.21.15.190:0/712728519 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff55c1a6e10 con 0x7ff55c10fa50 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff559d9b700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 0x7ff55c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff55ad9d700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 0x7ff55c1a6770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff554007d70 tx=0x7ff55400a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff54b7fe700 10 client.?.objecter ms_handle_connect 0x7ff55c10fa50 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff54b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff554016020 con 0x7ff55c10fa50 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff55c1a7100 con 0x7ff55c10fa50 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff55400ad40 con 0x7ff55c10fa50 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5540115e0 con 0x7ff55c10fa50 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 10 client.24325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:17:22.071 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff55c1a75f0 con 0x7ff55c10fa50 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 librados: init done 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.070+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff55c06a110 con 0x7ff55c10fa50 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7ff554019020 con 0x7ff55c10fa50 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff544071e90 0x7ff544074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(77..77 src has 22..77) ==== 5287+0+0 (secure 0 0 0) 0x7ff5540908e0 con 0x7ff55c10fa50 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 10 client.24325.objecter ms_dispatch 0x7ff55c001550 osd_map(77..77 src has 22..77) 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 3 client.24325.objecter handle_osd_map got epochs [77,77] > 0 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 3 client.24325.objecter handle_osd_map decoding full epoch 77 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 20 client.24325.objecter dump_active .. 0 homeless 2024-04-26T16:17:22.072 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff55a59c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff544071e90 0x7ff544074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:22.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff54b7fe700 10 client.24325.objecter ms_handle_connect 0x7ff544071e90 2024-04-26T16:17:22.073 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.074+0000 7ff55a59c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff544071e90 0x7ff544074350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff54c005950 tx=0x7ff54c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:21 smithi190 bash[23868]: cluster 2024-04-26T16:17:20.739956+0000 mgr.y (mgr.14152) 336 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:21 smithi190 bash[23868]: cluster 2024-04-26T16:17:20.739956+0000 mgr.y (mgr.14152) 336 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:21 smithi190 bash[23868]: cluster 2024-04-26T16:17:21.101299+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:21 smithi190 bash[23868]: cluster 2024-04-26T16:17:21.101299+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:21 smithi190 bash[23868]: cluster 2024-04-26T16:17:21.101411+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:22.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:21 smithi190 bash[23868]: cluster 2024-04-26T16:17:21.101411+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:22.075 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.078+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff554020020 con 0x7ff55c10fa50 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[21463]: cluster 2024-04-26T16:17:20.739956+0000 mgr.y (mgr.14152) 336 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[21463]: cluster 2024-04-26T16:17:20.739956+0000 mgr.y (mgr.14152) 336 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[21463]: cluster 2024-04-26T16:17:21.101299+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[21463]: cluster 2024-04-26T16:17:21.101299+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[21463]: cluster 2024-04-26T16:17:21.101411+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[21463]: cluster 2024-04-26T16:17:21.101411+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:22.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[29234]: cluster 2024-04-26T16:17:20.739956+0000 mgr.y (mgr.14152) 336 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:22.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[29234]: cluster 2024-04-26T16:17:20.739956+0000 mgr.y (mgr.14152) 336 : cluster [DBG] pgmap v339: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:22.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[29234]: cluster 2024-04-26T16:17:21.101299+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:22.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[29234]: cluster 2024-04-26T16:17:21.101299+0000 osd.0 (osd.0) 11 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:22.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[29234]: cluster 2024-04-26T16:17:21.101411+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:22.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:21 smithi050 bash[29234]: cluster 2024-04-26T16:17:21.101411+0000 osd.0 (osd.0) 12 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:22.307 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.310+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}) -- 0x7ff55c063860 con 0x7ff544071e90 2024-04-26T16:17:22.322 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff54b7fe700 1 -- 172.21.15.190:0/712728519 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+30 (secure 0 0 0) 0x7ff55c063860 con 0x7ff544071e90 2024-04-26T16:17:22.322 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled rgw.foo.a update... 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 10 librados: watch_flush enter 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 10 librados: watch_flush exit 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 20 client.24325.objecter shutdown clearing up homeless session... 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 10 client.24325.objecter successfully canceled tick 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff544071e90 msgr2=0x7ff544074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff544071e90 0x7ff544074350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff54c005950 tx=0x7ff54c0058e0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.325 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 msgr2=0x7ff55c1a6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 0x7ff55c1a6770 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff554007d70 tx=0x7ff55400a340 comp rx=0 tx=0).stop 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 shutdown_connections 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff544071e90 0x7ff544074350 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff55c10fa50 0x7ff55c1a6770 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff55c075e80 0x7ff55c19f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 --2- 172.21.15.190:0/712728519 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff55c077450 0x7ff55c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 >> 172.21.15.190:0/712728519 conn(0x7ff55c0fea10 msgr2=0x7ff55c102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.326+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 shutdown_connections 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.330+0000 7ff56092c700 1 -- 172.21.15.190:0/712728519 wait complete. 2024-04-26T16:17:22.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:22.330+0000 7ff56092c700 1 librados: shutdown 2024-04-26T16:17:23.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:22 smithi190 bash[23868]: cluster 2024-04-26T16:17:21.808482+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-04-26T16:17:23.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:22 smithi190 bash[23868]: cluster 2024-04-26T16:17:21.808482+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-04-26T16:17:23.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:22 smithi190 bash[23868]: audit 2024-04-26T16:17:22.332356+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:23.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:22 smithi190 bash[23868]: audit 2024-04-26T16:17:22.332356+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:23.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:22 smithi190 bash[23868]: audit 2024-04-26T16:17:22.346340+0000 mon.a (mon.0) 699 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:23.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:22 smithi190 bash[23868]: audit 2024-04-26T16:17:22.346340+0000 mon.a (mon.0) 699 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:23.058 DEBUG:teuthology.orchestra.run.smithi050:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@rgw.foo.a.service 2024-04-26T16:17:23.061 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi190 2024-04-26T16:17:23.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-04-26T16:17:23.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[21463]: cluster 2024-04-26T16:17:21.808482+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[21463]: cluster 2024-04-26T16:17:21.808482+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[21463]: audit 2024-04-26T16:17:22.332356+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[21463]: audit 2024-04-26T16:17:22.332356+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[21463]: audit 2024-04-26T16:17:22.346340+0000 mon.a (mon.0) 699 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[21463]: audit 2024-04-26T16:17:22.346340+0000 mon.a (mon.0) 699 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[29234]: cluster 2024-04-26T16:17:21.808482+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[29234]: cluster 2024-04-26T16:17:21.808482+0000 mon.a (mon.0) 697 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[29234]: audit 2024-04-26T16:17:22.332356+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:23.070 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[29234]: audit 2024-04-26T16:17:22.332356+0000 mon.a (mon.0) 698 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:23.071 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[29234]: audit 2024-04-26T16:17:22.346340+0000 mon.a (mon.0) 699 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:23.071 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:22 smithi050 bash[29234]: audit 2024-04-26T16:17:22.346340+0000 mon.a (mon.0) 699 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:24.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:22.320717+0000 mgr.y (mgr.14152) 337 : audit [DBG] from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:24.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:22.320717+0000 mgr.y (mgr.14152) 337 : audit [DBG] from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:24.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: cephadm 2024-04-26T16:17:22.324835+0000 mgr.y (mgr.14152) 338 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:24.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: cephadm 2024-04-26T16:17:22.324835+0000 mgr.y (mgr.14152) 338 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:24.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: cluster 2024-04-26T16:17:22.740558+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: cluster 2024-04-26T16:17:22.740558+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.025573+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.025573+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.027721+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.027721+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.038506+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.038506+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.046141+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.046141+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.049623+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.049623+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.063879+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.063879+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.068296+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:23 smithi190 bash[23868]: audit 2024-04-26T16:17:23.068296+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:22.320717+0000 mgr.y (mgr.14152) 337 : audit [DBG] from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:24.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:22.320717+0000 mgr.y (mgr.14152) 337 : audit [DBG] from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:24.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: cephadm 2024-04-26T16:17:22.324835+0000 mgr.y (mgr.14152) 338 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:24.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: cephadm 2024-04-26T16:17:22.324835+0000 mgr.y (mgr.14152) 338 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:24.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: cluster 2024-04-26T16:17:22.740558+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: cluster 2024-04-26T16:17:22.740558+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.025573+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.025573+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.027721+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.027721+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.038506+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.038506+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.046141+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.046141+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.049623+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.049623+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.063879+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.063879+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.068296+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[29234]: audit 2024-04-26T16:17:23.068296+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:22.320717+0000 mgr.y (mgr.14152) 337 : audit [DBG] from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:22.320717+0000 mgr.y (mgr.14152) 337 : audit [DBG] from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi050=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: cephadm 2024-04-26T16:17:22.324835+0000 mgr.y (mgr.14152) 338 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: cephadm 2024-04-26T16:17:22.324835+0000 mgr.y (mgr.14152) 338 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: cluster 2024-04-26T16:17:22.740558+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: cluster 2024-04-26T16:17:22.740558+0000 mgr.y (mgr.14152) 339 : cluster [DBG] pgmap v341: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:24.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.025573+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.025573+0000 mon.a (mon.0) 700 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.027721+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.027721+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.038506+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.038506+0000 mon.a (mon.0) 702 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.046141+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.046141+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.049623+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.049623+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.063879+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.063879+0000 mon.a (mon.0) 705 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:24.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.068296+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:24.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:23 smithi050 bash[21463]: audit 2024-04-26T16:17:23.068296+0000 mon.a (mon.0) 706 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:25.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:24 smithi190 bash[23868]: cephadm 2024-04-26T16:17:23.070436+0000 mgr.y (mgr.14152) 340 : cephadm [INF] Deploying daemon rgw.foo.a on smithi050 2024-04-26T16:17:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:24 smithi190 bash[23868]: cephadm 2024-04-26T16:17:23.070436+0000 mgr.y (mgr.14152) 340 : cephadm [INF] Deploying daemon rgw.foo.a on smithi050 2024-04-26T16:17:25.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:24 smithi050 bash[29234]: cephadm 2024-04-26T16:17:23.070436+0000 mgr.y (mgr.14152) 340 : cephadm [INF] Deploying daemon rgw.foo.a on smithi050 2024-04-26T16:17:25.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:24 smithi050 bash[29234]: cephadm 2024-04-26T16:17:23.070436+0000 mgr.y (mgr.14152) 340 : cephadm [INF] Deploying daemon rgw.foo.a on smithi050 2024-04-26T16:17:25.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:24 smithi050 bash[21463]: cephadm 2024-04-26T16:17:23.070436+0000 mgr.y (mgr.14152) 340 : cephadm [INF] Deploying daemon rgw.foo.a on smithi050 2024-04-26T16:17:25.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:24 smithi050 bash[21463]: cephadm 2024-04-26T16:17:23.070436+0000 mgr.y (mgr.14152) 340 : cephadm [INF] Deploying daemon rgw.foo.a on smithi050 2024-04-26T16:17:25.680 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.681 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.681 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.681 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.682 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:25.683 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:25 smithi190 bash[23868]: cluster 2024-04-26T16:17:24.741150+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:26.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:25 smithi190 bash[23868]: cluster 2024-04-26T16:17:24.741150+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:26.123 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.124 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.124 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.124 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.124 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.124 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: Started Ceph rgw.foo.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:17:26.125 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:25 smithi050 bash[21463]: cluster 2024-04-26T16:17:24.741150+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:26.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:25 smithi050 bash[21463]: cluster 2024-04-26T16:17:24.741150+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:26.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:25 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:26.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:25 smithi050 bash[29234]: cluster 2024-04-26T16:17:24.741150+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:26.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:25 smithi050 bash[29234]: cluster 2024-04-26T16:17:24.741150+0000 mgr.y (mgr.14152) 341 : cluster [DBG] pgmap v342: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.960143+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.960143+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.966398+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.966398+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.994904+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.994904+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: cephadm 2024-04-26T16:17:25.995695+0000 mgr.y (mgr.14152) 342 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: cephadm 2024-04-26T16:17:25.995695+0000 mgr.y (mgr.14152) 342 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.999728+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:25.999728+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:26.006773+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:26.006773+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:26.055282+0000 mon.a (mon.0) 712 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:26.055282+0000 mon.a (mon.0) 712 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:26.671287+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:26 smithi190 bash[23868]: audit 2024-04-26T16:17:26.671287+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.960143+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.960143+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.966398+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.966398+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.994904+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.994904+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: cephadm 2024-04-26T16:17:25.995695+0000 mgr.y (mgr.14152) 342 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: cephadm 2024-04-26T16:17:25.995695+0000 mgr.y (mgr.14152) 342 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.999728+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:25.999728+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:26.006773+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:26.006773+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:26.055282+0000 mon.a (mon.0) 712 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:26.055282+0000 mon.a (mon.0) 712 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:26.671287+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[21463]: audit 2024-04-26T16:17:26.671287+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.960143+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.960143+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.966398+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.966398+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.994904+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.994904+0000 mon.a (mon.0) 709 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: cephadm 2024-04-26T16:17:25.995695+0000 mgr.y (mgr.14152) 342 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: cephadm 2024-04-26T16:17:25.995695+0000 mgr.y (mgr.14152) 342 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi050=foo.a;count:1 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.999728+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:25.999728+0000 mon.a (mon.0) 710 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:26.006773+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:26.006773+0000 mon.a (mon.0) 711 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:26.055282+0000 mon.a (mon.0) 712 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:26.055282+0000 mon.a (mon.0) 712 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:26.671287+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:26 smithi050 bash[29234]: audit 2024-04-26T16:17:26.671287+0000 mon.a (mon.0) 713 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:27.812 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:17:28.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:27 smithi190 bash[23868]: cluster 2024-04-26T16:17:26.741806+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:28.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:27 smithi190 bash[23868]: cluster 2024-04-26T16:17:26.741806+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:28.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:27 smithi050 bash[21463]: cluster 2024-04-26T16:17:26.741806+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:28.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:27 smithi050 bash[21463]: cluster 2024-04-26T16:17:26.741806+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:28.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:27 smithi050 bash[29234]: cluster 2024-04-26T16:17:26.741806+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:28.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:27 smithi050 bash[29234]: cluster 2024-04-26T16:17:26.741806+0000 mgr.y (mgr.14152) 343 : cluster [DBG] pgmap v343: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:29.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:28 smithi190 bash[23868]: cluster 2024-04-26T16:17:27.977649+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-04-26T16:17:29.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:28 smithi190 bash[23868]: cluster 2024-04-26T16:17:27.977649+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-04-26T16:17:29.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:28 smithi190 bash[23868]: audit 2024-04-26T16:17:27.982205+0000 mon.a (mon.0) 715 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-26T16:17:29.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:28 smithi190 bash[23868]: audit 2024-04-26T16:17:27.982205+0000 mon.a (mon.0) 715 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-26T16:17:29.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[21463]: cluster 2024-04-26T16:17:27.977649+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[21463]: cluster 2024-04-26T16:17:27.977649+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[21463]: audit 2024-04-26T16:17:27.982205+0000 mon.a (mon.0) 715 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[21463]: audit 2024-04-26T16:17:27.982205+0000 mon.a (mon.0) 715 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[29234]: cluster 2024-04-26T16:17:27.977649+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[29234]: cluster 2024-04-26T16:17:27.977649+0000 mon.a (mon.0) 714 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[29234]: audit 2024-04-26T16:17:27.982205+0000 mon.a (mon.0) 715 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-26T16:17:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:28 smithi050 bash[29234]: audit 2024-04-26T16:17:27.982205+0000 mon.a (mon.0) 715 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: cluster 2024-04-26T16:17:28.742602+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v345: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: cluster 2024-04-26T16:17:28.742602+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v345: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: cluster 2024-04-26T16:17:28.970790+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: cluster 2024-04-26T16:17:28.970790+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: audit 2024-04-26T16:17:28.980536+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: audit 2024-04-26T16:17:28.980536+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: cluster 2024-04-26T16:17:28.989221+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-04-26T16:17:30.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:29 smithi190 bash[23868]: cluster 2024-04-26T16:17:28.989221+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: cluster 2024-04-26T16:17:28.742602+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v345: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: cluster 2024-04-26T16:17:28.742602+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v345: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: cluster 2024-04-26T16:17:28.970790+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: cluster 2024-04-26T16:17:28.970790+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: audit 2024-04-26T16:17:28.980536+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: audit 2024-04-26T16:17:28.980536+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: cluster 2024-04-26T16:17:28.989221+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-04-26T16:17:30.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[21463]: cluster 2024-04-26T16:17:28.989221+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: cluster 2024-04-26T16:17:28.742602+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v345: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: cluster 2024-04-26T16:17:28.742602+0000 mgr.y (mgr.14152) 344 : cluster [DBG] pgmap v345: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: cluster 2024-04-26T16:17:28.970790+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: cluster 2024-04-26T16:17:28.970790+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: audit 2024-04-26T16:17:28.980536+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: audit 2024-04-26T16:17:28.980536+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.50:0/294328258' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: cluster 2024-04-26T16:17:28.989221+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-04-26T16:17:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:29 smithi050 bash[29234]: cluster 2024-04-26T16:17:28.989221+0000 mon.a (mon.0) 718 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2024-04-26T16:17:30.499 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/96836079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 msgr2=0x7fcc40104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:30.499 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- 172.21.15.190:0/96836079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc40104130 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009990 tx=0x7fcc30009ca0 comp rx=0 tx=0).stop 2024-04-26T16:17:30.499 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/96836079 shutdown_connections 2024-04-26T16:17:30.499 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- 172.21.15.190:0/96836079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc40104130 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- 172.21.15.190:0/96836079 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 0x7fcc401074c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- 172.21.15.190:0/96836079 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc40104670 0x7fcc40106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/96836079 >> 172.21.15.190:0/96836079 conn(0x7fcc400fd940 msgr2=0x7fcc400ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/96836079 shutdown_connections 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/96836079 wait complete. 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 librados: starting msgr at 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 librados: starting objecter 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 Processor -- start 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- start start 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 librados: setting wanted keys 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 librados: calling monclient init 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc4019f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc40104670 0x7fcc4019f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:30.500 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 0x7fcc401a6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcc40114a30 con 0x7fcc40107040 2024-04-26T16:17:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc401148b0 con 0x7fcc40101d00 2024-04-26T16:17:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fcc40114bb0 con 0x7fcc40104670 2024-04-26T16:17:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc4019f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc4019f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:57046/0 (socket says 172.21.15.190:57046) 2024-04-26T16:17:30.501 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 -- 172.21.15.190:0/4215476481 learned_addr learned my addr 172.21.15.190:0/4215476481 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:17:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 -- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc40104670 msgr2=0x7fcc4019f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3dd9b700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc40104670 0x7fcc4019f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:30.502 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc40104670 0x7fcc4019f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3ed9d700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 0x7fcc401a6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 -- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 msgr2=0x7fcc401a6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 0x7fcc401a6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 -- 172.21.15.190:0/4215476481 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc401a6f00 con 0x7fcc40101d00 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3ed9d700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 0x7fcc401a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc3e59c700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc4019f2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc30003760 tx=0x7fcc3000f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc377fe700 10 client.?.objecter ms_handle_connect 0x7fcc40101d00 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc377fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30027070 con 0x7fcc40101d00 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fcc3000fbb0 con 0x7fcc40101d00 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc401a7190 con 0x7fcc40101d00 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30003830 con 0x7fcc40101d00 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 10 client.24343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:17:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc401a7720 con 0x7fcc40101d00 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.501+0000 7fcc44e84700 1 librados: init done 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4006a110 con 0x7fcc40101d00 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7fcc3002a070 con 0x7fcc40101d00 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fcc2c071e90 0x7fcc2c074350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(80..80 src has 22..80) ==== 6000+0+0 (secure 0 0 0) 0x7fcc30091d80 con 0x7fcc40101d00 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 10 client.24343.objecter ms_dispatch 0x7fcc40001550 osd_map(80..80 src has 22..80) 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 3 client.24343.objecter handle_osd_map got epochs [80,80] > 0 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 3 client.24343.objecter handle_osd_map decoding full epoch 80 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 20 client.24343.objecter dump_active .. 0 homeless 2024-04-26T16:17:30.504 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc3dd9b700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fcc2c071e90 0x7fcc2c074350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:30.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc377fe700 10 client.24343.objecter ms_handle_connect 0x7fcc2c071e90 2024-04-26T16:17:30.505 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.505+0000 7fcc3dd9b700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fcc2c071e90 0x7fcc2c074350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc28006fd0 tx=0x7fcc28008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:30.507 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.509+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fcc30013070 con 0x7fcc40101d00 2024-04-26T16:17:30.728 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:30.729+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) -- 0x7fcc4002e490 con 0x7fcc40101d00 2024-04-26T16:17:31.008 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.009+0000 7fcc377fe700 1 -- 172.21.15.190:0/4215476481 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v81) ==== 160+0+0 (secure 0 0 0) 0x7fcc3005e4c0 con 0x7fcc40101d00 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.013+0000 7fcc44e84700 10 librados: watch_flush enter 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 10 librados: watch_flush exit 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 20 client.24343.objecter shutdown clearing up homeless session... 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 10 client.24343.objecter successfully canceled tick 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fcc2c071e90 msgr2=0x7fcc2c074350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fcc2c071e90 0x7fcc2c074350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcc28006fd0 tx=0x7fcc28008040 comp rx=0 tx=0).stop 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 msgr2=0x7fcc4019f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.017+0000 7fcc44e84700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc4019f2a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcc30003760 tx=0x7fcc3000f740 comp rx=0 tx=0).stop 2024-04-26T16:17:31.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 shutdown_connections 2024-04-26T16:17:31.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7fcc2c071e90 0x7fcc2c074350 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:31.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc40101d00 0x7fcc4019f2a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:31.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc40107040 0x7fcc401a6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:31.018 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 --2- 172.21.15.190:0/4215476481 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc40104670 0x7fcc4019f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:31.019 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 >> 172.21.15.190:0/4215476481 conn(0x7fcc400fd940 msgr2=0x7fcc40105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:31.019 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 shutdown_connections 2024-04-26T16:17:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 -- 172.21.15.190:0/4215476481 wait complete. 2024-04-26T16:17:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:31.021+0000 7fcc44e84700 1 librados: shutdown 2024-04-26T16:17:31.021 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'datapool' created 2024-04-26T16:17:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: cluster 2024-04-26T16:17:30.000156+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-04-26T16:17:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: cluster 2024-04-26T16:17:30.000156+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-04-26T16:17:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: audit 2024-04-26T16:17:30.017185+0000 mon.a (mon.0) 720 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-26T16:17:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: audit 2024-04-26T16:17:30.017185+0000 mon.a (mon.0) 720 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-26T16:17:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: audit 2024-04-26T16:17:30.736771+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.190:0/4215476481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: audit 2024-04-26T16:17:30.736771+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.190:0/4215476481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: audit 2024-04-26T16:17:30.740951+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:31 smithi190 bash[23868]: audit 2024-04-26T16:17:30.740951+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: cluster 2024-04-26T16:17:30.000156+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-04-26T16:17:31.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: cluster 2024-04-26T16:17:30.000156+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-04-26T16:17:31.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: audit 2024-04-26T16:17:30.017185+0000 mon.a (mon.0) 720 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: audit 2024-04-26T16:17:30.017185+0000 mon.a (mon.0) 720 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: audit 2024-04-26T16:17:30.736771+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.190:0/4215476481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: audit 2024-04-26T16:17:30.736771+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.190:0/4215476481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: audit 2024-04-26T16:17:30.740951+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[21463]: audit 2024-04-26T16:17:30.740951+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: cluster 2024-04-26T16:17:30.000156+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: cluster 2024-04-26T16:17:30.000156+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: audit 2024-04-26T16:17:30.017185+0000 mon.a (mon.0) 720 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: audit 2024-04-26T16:17:30.017185+0000 mon.a (mon.0) 720 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-04-26T16:17:31.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: audit 2024-04-26T16:17:30.736771+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.190:0/4215476481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: audit 2024-04-26T16:17:30.736771+0000 mon.b (mon.1) 26 : audit [INF] from='client.? 172.21.15.190:0/4215476481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: audit 2024-04-26T16:17:30.740951+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:31 smithi050 bash[29234]: audit 2024-04-26T16:17:30.740951+0000 mon.a (mon.0) 721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-04-26T16:17:31.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- rbd pool init datapool 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: cluster 2024-04-26T16:17:30.743367+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v348: 65 pgs: 1 creating+peering, 18 active+clean, 46 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: cluster 2024-04-26T16:17:30.743367+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v348: 65 pgs: 1 creating+peering, 18 active+clean, 46 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.003117+0000 mon.a (mon.0) 722 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.003117+0000 mon.a (mon.0) 722 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.003366+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.003366+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-26T16:17:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: cluster 2024-04-26T16:17:31.016184+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: cluster 2024-04-26T16:17:31.016184+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.446673+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.446673+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.455388+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.455388+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.456697+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.456697+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.457574+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:32 smithi190 bash[23868]: audit 2024-04-26T16:17:31.457574+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:32.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: cluster 2024-04-26T16:17:30.743367+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v348: 65 pgs: 1 creating+peering, 18 active+clean, 46 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-04-26T16:17:32.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: cluster 2024-04-26T16:17:30.743367+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v348: 65 pgs: 1 creating+peering, 18 active+clean, 46 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-04-26T16:17:32.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.003117+0000 mon.a (mon.0) 722 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-26T16:17:32.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.003117+0000 mon.a (mon.0) 722 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-26T16:17:32.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.003366+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.003366+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: cluster 2024-04-26T16:17:31.016184+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: cluster 2024-04-26T16:17:31.016184+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.446673+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.446673+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.455388+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.455388+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.456697+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.456697+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.457574+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[21463]: audit 2024-04-26T16:17:31.457574+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: cluster 2024-04-26T16:17:30.743367+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v348: 65 pgs: 1 creating+peering, 18 active+clean, 46 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-04-26T16:17:32.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: cluster 2024-04-26T16:17:30.743367+0000 mgr.y (mgr.14152) 345 : cluster [DBG] pgmap v348: 65 pgs: 1 creating+peering, 18 active+clean, 46 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.003117+0000 mon.a (mon.0) 722 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.003117+0000 mon.a (mon.0) 722 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.003366+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.003366+0000 mon.a (mon.0) 723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: cluster 2024-04-26T16:17:31.016184+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: cluster 2024-04-26T16:17:31.016184+0000 mon.a (mon.0) 724 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.446673+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.446673+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.455388+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.455388+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.456697+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.456697+0000 mon.a (mon.0) 727 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.457574+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:32.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:32 smithi050 bash[29234]: audit 2024-04-26T16:17:31.457574+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: cephadm 2024-04-26T16:17:31.467721+0000 mgr.y (mgr.14152) 346 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: cephadm 2024-04-26T16:17:31.467721+0000 mgr.y (mgr.14152) 346 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: cluster 2024-04-26T16:17:32.026323+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: cluster 2024-04-26T16:17:32.026323+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: audit 2024-04-26T16:17:32.033342+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: audit 2024-04-26T16:17:32.033342+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: audit 2024-04-26T16:17:32.044741+0000 mon.a (mon.0) 731 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:33 smithi190 bash[23868]: audit 2024-04-26T16:17:32.044741+0000 mon.a (mon.0) 731 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: cephadm 2024-04-26T16:17:31.467721+0000 mgr.y (mgr.14152) 346 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: cephadm 2024-04-26T16:17:31.467721+0000 mgr.y (mgr.14152) 346 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: cluster 2024-04-26T16:17:32.026323+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: cluster 2024-04-26T16:17:32.026323+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: audit 2024-04-26T16:17:32.033342+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: audit 2024-04-26T16:17:32.033342+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: audit 2024-04-26T16:17:32.044741+0000 mon.a (mon.0) 731 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[21463]: audit 2024-04-26T16:17:32.044741+0000 mon.a (mon.0) 731 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: cephadm 2024-04-26T16:17:31.467721+0000 mgr.y (mgr.14152) 346 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: cephadm 2024-04-26T16:17:31.467721+0000 mgr.y (mgr.14152) 346 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: cluster 2024-04-26T16:17:32.026323+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: cluster 2024-04-26T16:17:32.026323+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: audit 2024-04-26T16:17:32.033342+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: audit 2024-04-26T16:17:32.033342+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: audit 2024-04-26T16:17:32.044741+0000 mon.a (mon.0) 731 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:33 smithi050 bash[29234]: audit 2024-04-26T16:17:32.044741+0000 mon.a (mon.0) 731 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: cluster 2024-04-26T16:17:32.744456+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v351: 100 pgs: 5 creating+peering, 36 active+clean, 59 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: cluster 2024-04-26T16:17:32.744456+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v351: 100 pgs: 5 creating+peering, 36 active+clean, 59 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: audit 2024-04-26T16:17:33.027565+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: audit 2024-04-26T16:17:33.027565+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: audit 2024-04-26T16:17:33.027776+0000 mon.a (mon.0) 733 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: audit 2024-04-26T16:17:33.027776+0000 mon.a (mon.0) 733 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: cluster 2024-04-26T16:17:33.035774+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-04-26T16:17:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:34 smithi190 bash[23868]: cluster 2024-04-26T16:17:33.035774+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: cluster 2024-04-26T16:17:32.744456+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v351: 100 pgs: 5 creating+peering, 36 active+clean, 59 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: cluster 2024-04-26T16:17:32.744456+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v351: 100 pgs: 5 creating+peering, 36 active+clean, 59 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: audit 2024-04-26T16:17:33.027565+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: audit 2024-04-26T16:17:33.027565+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: audit 2024-04-26T16:17:33.027776+0000 mon.a (mon.0) 733 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: audit 2024-04-26T16:17:33.027776+0000 mon.a (mon.0) 733 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: cluster 2024-04-26T16:17:33.035774+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-04-26T16:17:34.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[21463]: cluster 2024-04-26T16:17:33.035774+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: cluster 2024-04-26T16:17:32.744456+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v351: 100 pgs: 5 creating+peering, 36 active+clean, 59 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: cluster 2024-04-26T16:17:32.744456+0000 mgr.y (mgr.14152) 347 : cluster [DBG] pgmap v351: 100 pgs: 5 creating+peering, 36 active+clean, 59 unknown; 577 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: audit 2024-04-26T16:17:33.027565+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: audit 2024-04-26T16:17:33.027565+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: audit 2024-04-26T16:17:33.027776+0000 mon.a (mon.0) 733 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: audit 2024-04-26T16:17:33.027776+0000 mon.a (mon.0) 733 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: cluster 2024-04-26T16:17:33.035774+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-04-26T16:17:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:34 smithi050 bash[29234]: cluster 2024-04-26T16:17:33.035774+0000 mon.a (mon.0) 734 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.038921+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.038921+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: audit 2024-04-26T16:17:34.048070+0000 mon.a (mon.0) 736 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: audit 2024-04-26T16:17:34.048070+0000 mon.a (mon.0) 736 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: audit 2024-04-26T16:17:34.049467+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: audit 2024-04-26T16:17:34.049467+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.745320+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v354: 132 pgs: 8 creating+peering, 58 active+clean, 66 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T16:17:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:35 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.745320+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v354: 132 pgs: 8 creating+peering, 58 active+clean, 66 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.038921+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.038921+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: audit 2024-04-26T16:17:34.048070+0000 mon.a (mon.0) 736 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: audit 2024-04-26T16:17:34.048070+0000 mon.a (mon.0) 736 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: audit 2024-04-26T16:17:34.049467+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: audit 2024-04-26T16:17:34.049467+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.745320+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v354: 132 pgs: 8 creating+peering, 58 active+clean, 66 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.745320+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v354: 132 pgs: 8 creating+peering, 58 active+clean, 66 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.038921+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-04-26T16:17:35.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.038921+0000 mon.a (mon.0) 735 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2024-04-26T16:17:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: audit 2024-04-26T16:17:34.048070+0000 mon.a (mon.0) 736 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: audit 2024-04-26T16:17:34.048070+0000 mon.a (mon.0) 736 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: audit 2024-04-26T16:17:34.049467+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: audit 2024-04-26T16:17:34.049467+0000 mon.a (mon.0) 737 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-04-26T16:17:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.745320+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v354: 132 pgs: 8 creating+peering, 58 active+clean, 66 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T16:17:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:35 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.745320+0000 mgr.y (mgr.14152) 348 : cluster [DBG] pgmap v354: 132 pgs: 8 creating+peering, 58 active+clean, 66 unknown; 578 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.730666+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.730666+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.730753+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:34.730753+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.042656+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.042656+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.042814+0000 mon.a (mon.0) 739 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.042814+0000 mon.a (mon.0) 739 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:35.054920+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-04-26T16:17:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:35.054920+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-04-26T16:17:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.058488+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.058488+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.060582+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: audit 2024-04-26T16:17:35.060582+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:35.876053+0000 mon.a (mon.0) 743 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:36 smithi190 bash[23868]: cluster 2024-04-26T16:17:35.876053+0000 mon.a (mon.0) 743 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:36.373 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[53323]: debug 2024-04-26T16:17:36.155+0000 7fbdc2619ac0 -1 LDAP not started since no server URIs were provided in the configuration. 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.730666+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.730666+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.730753+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:34.730753+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.042656+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.042656+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.042814+0000 mon.a (mon.0) 739 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.042814+0000 mon.a (mon.0) 739 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:35.054920+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-04-26T16:17:36.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:35.054920+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.058488+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.058488+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.060582+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: audit 2024-04-26T16:17:35.060582+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:35.876053+0000 mon.a (mon.0) 743 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[29234]: cluster 2024-04-26T16:17:35.876053+0000 mon.a (mon.0) 743 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.730666+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.730666+0000 osd.1 (osd.1) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.730753+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:34.730753+0000 osd.1 (osd.1) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.042656+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.042656+0000 mon.a (mon.0) 738 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.042814+0000 mon.a (mon.0) 739 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.042814+0000 mon.a (mon.0) 739 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:35.054920+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:35.054920+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.058488+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.058488+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.060582+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: audit 2024-04-26T16:17:35.060582+0000 mon.a (mon.0) 742 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:35.876053+0000 mon.a (mon.0) 743 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:36 smithi050 bash[21463]: cluster 2024-04-26T16:17:35.876053+0000 mon.a (mon.0) 743 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:17:36.503 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:17:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.045292+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.045292+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.045465+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.045465+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: cluster 2024-04-26T16:17:36.052526+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: cluster 2024-04-26T16:17:36.052526+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.340510+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.340510+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.346130+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.346130+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.356443+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.356443+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.446212+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.446212+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.448474+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.448474+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.449381+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:37.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.449381+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:37.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: cephadm 2024-04-26T16:17:36.454807+0000 mgr.y (mgr.14152) 349 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:37.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: cephadm 2024-04-26T16:17:36.454807+0000 mgr.y (mgr.14152) 349 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:37.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: cluster 2024-04-26T16:17:36.746142+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v357: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-04-26T16:17:37.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: cluster 2024-04-26T16:17:36.746142+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v357: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-04-26T16:17:37.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.759593+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:37 smithi190 bash[23868]: audit 2024-04-26T16:17:36.759593+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.045292+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.045292+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.045465+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.045465+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: cluster 2024-04-26T16:17:36.052526+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: cluster 2024-04-26T16:17:36.052526+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.340510+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.340510+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.346130+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.346130+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.356443+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.356443+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.446212+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.446212+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.448474+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.448474+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.449381+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.449381+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: cephadm 2024-04-26T16:17:36.454807+0000 mgr.y (mgr.14152) 349 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: cephadm 2024-04-26T16:17:36.454807+0000 mgr.y (mgr.14152) 349 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: cluster 2024-04-26T16:17:36.746142+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v357: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: cluster 2024-04-26T16:17:36.746142+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v357: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.759593+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[21463]: audit 2024-04-26T16:17:36.759593+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.045292+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.045292+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.50:0/2760378525' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.045465+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.045465+0000 mon.a (mon.0) 745 : audit [INF] from='client.? 172.21.15.50:0/826862182' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: cluster 2024-04-26T16:17:36.052526+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: cluster 2024-04-26T16:17:36.052526+0000 mon.a (mon.0) 746 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.340510+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.340510+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.346130+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.346130+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.356443+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.356443+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.446212+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.446212+0000 mon.a (mon.0) 750 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.448474+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.448474+0000 mon.a (mon.0) 751 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.449381+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.449381+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: cephadm 2024-04-26T16:17:36.454807+0000 mgr.y (mgr.14152) 349 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: cephadm 2024-04-26T16:17:36.454807+0000 mgr.y (mgr.14152) 349 : cephadm [INF] Checking dashboard <-> RGW credentials 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: cluster 2024-04-26T16:17:36.746142+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v357: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: cluster 2024-04-26T16:17:36.746142+0000 mgr.y (mgr.14152) 350 : cluster [DBG] pgmap v357: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 579 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.0 KiB/s wr, 20 op/s 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.759593+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:37 smithi050 bash[29234]: audit 2024-04-26T16:17:36.759593+0000 mon.a (mon.0) 753 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:40.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:39 smithi190 bash[23868]: cluster 2024-04-26T16:17:38.747318+0000 mgr.y (mgr.14152) 351 : cluster [DBG] pgmap v358: 132 pgs: 13 creating+peering, 114 active+clean, 5 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 KiB/s wr, 82 op/s 2024-04-26T16:17:40.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:39 smithi190 bash[23868]: cluster 2024-04-26T16:17:38.747318+0000 mgr.y (mgr.14152) 351 : cluster [DBG] pgmap v358: 132 pgs: 13 creating+peering, 114 active+clean, 5 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 KiB/s wr, 82 op/s 2024-04-26T16:17:40.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:39 smithi190 bash[23868]: audit 2024-04-26T16:17:39.127043+0000 mon.b (mon.1) 27 : audit [INF] from='client.? 172.21.15.190:0/2770429223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:39 smithi190 bash[23868]: audit 2024-04-26T16:17:39.127043+0000 mon.b (mon.1) 27 : audit [INF] from='client.? 172.21.15.190:0/2770429223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:39 smithi190 bash[23868]: audit 2024-04-26T16:17:39.130881+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:39 smithi190 bash[23868]: audit 2024-04-26T16:17:39.130881+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[21463]: cluster 2024-04-26T16:17:38.747318+0000 mgr.y (mgr.14152) 351 : cluster [DBG] pgmap v358: 132 pgs: 13 creating+peering, 114 active+clean, 5 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 KiB/s wr, 82 op/s 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[21463]: cluster 2024-04-26T16:17:38.747318+0000 mgr.y (mgr.14152) 351 : cluster [DBG] pgmap v358: 132 pgs: 13 creating+peering, 114 active+clean, 5 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 KiB/s wr, 82 op/s 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[21463]: audit 2024-04-26T16:17:39.127043+0000 mon.b (mon.1) 27 : audit [INF] from='client.? 172.21.15.190:0/2770429223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[21463]: audit 2024-04-26T16:17:39.127043+0000 mon.b (mon.1) 27 : audit [INF] from='client.? 172.21.15.190:0/2770429223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[21463]: audit 2024-04-26T16:17:39.130881+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[21463]: audit 2024-04-26T16:17:39.130881+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[29234]: cluster 2024-04-26T16:17:38.747318+0000 mgr.y (mgr.14152) 351 : cluster [DBG] pgmap v358: 132 pgs: 13 creating+peering, 114 active+clean, 5 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 KiB/s wr, 82 op/s 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[29234]: cluster 2024-04-26T16:17:38.747318+0000 mgr.y (mgr.14152) 351 : cluster [DBG] pgmap v358: 132 pgs: 13 creating+peering, 114 active+clean, 5 unknown; 580 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 3.5 KiB/s wr, 82 op/s 2024-04-26T16:17:40.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[29234]: audit 2024-04-26T16:17:39.127043+0000 mon.b (mon.1) 27 : audit [INF] from='client.? 172.21.15.190:0/2770429223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[29234]: audit 2024-04-26T16:17:39.127043+0000 mon.b (mon.1) 27 : audit [INF] from='client.? 172.21.15.190:0/2770429223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[29234]: audit 2024-04-26T16:17:39.130881+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:40.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:39 smithi050 bash[29234]: audit 2024-04-26T16:17:39.130881+0000 mon.a (mon.0) 754 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-04-26T16:17:41.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:40 smithi190 bash[23868]: audit 2024-04-26T16:17:39.803810+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-26T16:17:41.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:40 smithi190 bash[23868]: audit 2024-04-26T16:17:39.803810+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-26T16:17:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:40 smithi190 bash[23868]: cluster 2024-04-26T16:17:39.811709+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-04-26T16:17:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:40 smithi190 bash[23868]: cluster 2024-04-26T16:17:39.811709+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[29234]: audit 2024-04-26T16:17:39.803810+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[29234]: audit 2024-04-26T16:17:39.803810+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[29234]: cluster 2024-04-26T16:17:39.811709+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[29234]: cluster 2024-04-26T16:17:39.811709+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[21463]: audit 2024-04-26T16:17:39.803810+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[21463]: audit 2024-04-26T16:17:39.803810+0000 mon.a (mon.0) 755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[21463]: cluster 2024-04-26T16:17:39.811709+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-04-26T16:17:41.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:40 smithi050 bash[21463]: cluster 2024-04-26T16:17:39.811709+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2024-04-26T16:17:41.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.190 --placement '1;smithi190=iscsi.a' 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.748511+0000 mgr.y (mgr.14152) 352 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 173 op/s 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.748511+0000 mgr.y (mgr.14152) 352 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 173 op/s 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.815487+0000 mon.a (mon.0) 757 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.815487+0000 mon.a (mon.0) 757 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.815560+0000 mon.a (mon.0) 758 : cluster [INF] Cluster is now healthy 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.815560+0000 mon.a (mon.0) 758 : cluster [INF] Cluster is now healthy 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.833314+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.833314+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-04-26T16:17:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.887648+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-04-26T16:17:42.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:41 smithi190 bash[23868]: cluster 2024-04-26T16:17:40.887648+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.748511+0000 mgr.y (mgr.14152) 352 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 173 op/s 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.748511+0000 mgr.y (mgr.14152) 352 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 173 op/s 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.815487+0000 mon.a (mon.0) 757 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.815487+0000 mon.a (mon.0) 757 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.815560+0000 mon.a (mon.0) 758 : cluster [INF] Cluster is now healthy 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.815560+0000 mon.a (mon.0) 758 : cluster [INF] Cluster is now healthy 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.833314+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.833314+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.887648+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-04-26T16:17:42.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[21463]: cluster 2024-04-26T16:17:40.887648+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.748511+0000 mgr.y (mgr.14152) 352 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 173 op/s 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.748511+0000 mgr.y (mgr.14152) 352 : cluster [DBG] pgmap v360: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 173 op/s 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.815487+0000 mon.a (mon.0) 757 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.815487+0000 mon.a (mon.0) 757 : cluster [INF] Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.815560+0000 mon.a (mon.0) 758 : cluster [INF] Cluster is now healthy 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.815560+0000 mon.a (mon.0) 758 : cluster [INF] Cluster is now healthy 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.833314+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.833314+0000 mon.a (mon.0) 759 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.887648+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-04-26T16:17:42.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:41 smithi050 bash[29234]: cluster 2024-04-26T16:17:40.887648+0000 mon.a (mon.0) 760 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2024-04-26T16:17:44.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:43 smithi190 bash[23868]: cluster 2024-04-26T16:17:42.749502+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 180 op/s 2024-04-26T16:17:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:43 smithi190 bash[23868]: cluster 2024-04-26T16:17:42.749502+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 180 op/s 2024-04-26T16:17:44.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:43 smithi050 bash[21463]: cluster 2024-04-26T16:17:42.749502+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 180 op/s 2024-04-26T16:17:44.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:43 smithi050 bash[21463]: cluster 2024-04-26T16:17:42.749502+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 180 op/s 2024-04-26T16:17:44.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:43 smithi050 bash[29234]: cluster 2024-04-26T16:17:42.749502+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 180 op/s 2024-04-26T16:17:44.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:43 smithi050 bash[29234]: cluster 2024-04-26T16:17:42.749502+0000 mgr.y (mgr.14152) 353 : cluster [DBG] pgmap v363: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 180 op/s 2024-04-26T16:17:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:45 smithi050 bash[21463]: cluster 2024-04-26T16:17:44.750524+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.3 KiB/s wr, 113 op/s 2024-04-26T16:17:46.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:45 smithi050 bash[21463]: cluster 2024-04-26T16:17:44.750524+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.3 KiB/s wr, 113 op/s 2024-04-26T16:17:46.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:45 smithi050 bash[29234]: cluster 2024-04-26T16:17:44.750524+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.3 KiB/s wr, 113 op/s 2024-04-26T16:17:46.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:45 smithi050 bash[29234]: cluster 2024-04-26T16:17:44.750524+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.3 KiB/s wr, 113 op/s 2024-04-26T16:17:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:45 smithi190 bash[23868]: cluster 2024-04-26T16:17:44.750524+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.3 KiB/s wr, 113 op/s 2024-04-26T16:17:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:45 smithi190 bash[23868]: cluster 2024-04-26T16:17:44.750524+0000 mgr.y (mgr.14152) 354 : cluster [DBG] pgmap v364: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.3 KiB/s wr, 113 op/s 2024-04-26T16:17:46.386 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.393047+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.393047+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.393138+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.393138+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.843988+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.843988+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.889824+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-04-26T16:17:47.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[21463]: cluster 2024-04-26T16:17:45.889824+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.393047+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.393047+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.393138+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.393138+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.843988+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.843988+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.889824+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-04-26T16:17:47.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:46 smithi050 bash[29234]: cluster 2024-04-26T16:17:45.889824+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-04-26T16:17:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.393047+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.393047+0000 osd.2 (osd.2) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.393138+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.393138+0000 osd.2 (osd.2) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:47.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.843988+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-04-26T16:17:47.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.843988+0000 mon.a (mon.0) 761 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2024-04-26T16:17:47.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.889824+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-04-26T16:17:47.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:46 smithi190 bash[23868]: cluster 2024-04-26T16:17:45.889824+0000 mon.a (mon.0) 762 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2024-04-26T16:17:48.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:47 smithi190 bash[23868]: cluster 2024-04-26T16:17:46.751420+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 345 B/s wr, 17 op/s 2024-04-26T16:17:48.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:47 smithi190 bash[23868]: cluster 2024-04-26T16:17:46.751420+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 345 B/s wr, 17 op/s 2024-04-26T16:17:48.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:47 smithi050 bash[29234]: cluster 2024-04-26T16:17:46.751420+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 345 B/s wr, 17 op/s 2024-04-26T16:17:48.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:47 smithi050 bash[29234]: cluster 2024-04-26T16:17:46.751420+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 345 B/s wr, 17 op/s 2024-04-26T16:17:48.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:47 smithi050 bash[21463]: cluster 2024-04-26T16:17:46.751420+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 345 B/s wr, 17 op/s 2024-04-26T16:17:48.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:47 smithi050 bash[21463]: cluster 2024-04-26T16:17:46.751420+0000 mgr.y (mgr.14152) 355 : cluster [DBG] pgmap v367: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 7.3 KiB/s rd, 345 B/s wr, 17 op/s 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/1999431005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 msgr2=0x7f7140108c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- 172.21.15.190:0/1999431005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140108c60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f713c009990 tx=0x7f713c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/1999431005 shutdown_connections 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- 172.21.15.190:0/1999431005 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f71401091a0 0x7f714010b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- 172.21.15.190:0/1999431005 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7140103f40 0x7f7140106330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- 172.21.15.190:0/1999431005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140108c60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/1999431005 >> 172.21.15.190:0/1999431005 conn(0x7f71400fd970 msgr2=0x7f71400ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/1999431005 shutdown_connections 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/1999431005 wait complete. 2024-04-26T16:17:48.956 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 librados: starting msgr at 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 librados: starting objecter 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 Processor -- start 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- start start 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 librados: setting wanted keys 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 librados: calling monclient init 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7140103f40 0x7f71401038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140101ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71401091a0 0x7f7140102430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f714010e490 con 0x7f71401091a0 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f714010e310 con 0x7f7140106870 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f714010e610 con 0x7f7140103f40 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140101ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140101ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:40228/0 (socket says 172.21.15.190:40228) 2024-04-26T16:17:48.957 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 -- 172.21.15.190:0/2541999741 learned_addr learned my addr 172.21.15.190:0/2541999741 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:17:48.958 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7145b9c700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71401091a0 0x7f7140102430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:48.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f714539b700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7140103f40 0x7f71401038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:48.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 -- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7140103f40 msgr2=0x7f71401038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:48.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7140103f40 0x7f71401038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:48.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 -- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71401091a0 msgr2=0x7f7140102430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:48.959 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71401091a0 0x7f7140102430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 -- 172.21.15.190:0/2541999741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7140102ce0 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f714539b700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7140103f40 0x7f71401038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f7144b9a700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140101ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f713c006010 tx=0x7f713c00f740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71367fc700 10 client.?.objecter ms_handle_connect 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71367fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c027070 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f713c00bd70 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71401a2bb0 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f713c022cf0 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 10 client.24385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71401a3000 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.957+0000 7f71475ff700 1 librados: init done 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7140051960 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) ==== 93903+0+0 (secure 0 0 0) 0x7f713c02a070 con 0x7f7140106870 2024-04-26T16:17:48.960 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f712c071f40 0x7f712c074400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(92..92 src has 22..92) ==== 7148+0+0 (secure 0 0 0) 0x7f713c092260 con 0x7f7140106870 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 10 client.24385.objecter ms_dispatch 0x7f7140001550 osd_map(92..92 src has 22..92) 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 3 client.24385.objecter handle_osd_map got epochs [92,92] > 0 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 3 client.24385.objecter handle_osd_map decoding full epoch 92 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 20 client.24385.objecter dump_active .. 0 homeless 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f714539b700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f712c071f40 0x7f712c074400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f71367fc700 10 client.24385.objecter ms_handle_connect 0x7f712c071f40 2024-04-26T16:17:48.961 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.961+0000 7f714539b700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f712c071f40 0x7f712c074400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7130005fd0 tx=0x7f7130005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:48.963 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:48.965+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f713c013070 con 0x7f7140106870 2024-04-26T16:17:49.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:48 smithi190 bash[23868]: cluster 2024-04-26T16:17:47.491275+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:49.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:48 smithi190 bash[23868]: cluster 2024-04-26T16:17:47.491275+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:49.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:48 smithi190 bash[23868]: cluster 2024-04-26T16:17:47.491367+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:49.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:48 smithi190 bash[23868]: cluster 2024-04-26T16:17:47.491367+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:49.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:48 smithi190 bash[23868]: cluster 2024-04-26T16:17:47.913247+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-04-26T16:17:49.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:48 smithi190 bash[23868]: cluster 2024-04-26T16:17:47.913247+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-04-26T16:17:49.195 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.197+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}) -- 0x7f7140063860 con 0x7f712c071f40 2024-04-26T16:17:49.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71367fc700 1 -- 172.21.15.190:0/2541999741 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+35 (secure 0 0 0) 0x7f7140063860 con 0x7f712c071f40 2024-04-26T16:17:49.213 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled iscsi.datapool update... 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 10 librados: watch_flush enter 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 10 librados: watch_flush exit 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 20 client.24385.objecter shutdown clearing up homeless session... 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 10 client.24385.objecter successfully canceled tick 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f712c071f40 msgr2=0x7f712c074400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f712c071f40 0x7f712c074400 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7130005fd0 tx=0x7f7130005ee0 comp rx=0 tx=0).stop 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 msgr2=0x7f7140101ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140101ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f713c006010 tx=0x7f713c00f740 comp rx=0 tx=0).stop 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 shutdown_connections 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f71401091a0 0x7f7140102430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7140103f40 0x7f71401038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:49.216 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f712c071f40 0x7f712c074400 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:49.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 --2- 172.21.15.190:0/2541999741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7140106870 0x7f7140101ef0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:49.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 >> 172.21.15.190:0/2541999741 conn(0x7f71400fd970 msgr2=0x7f71401047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:49.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 shutdown_connections 2024-04-26T16:17:49.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 -- 172.21.15.190:0/2541999741 wait complete. 2024-04-26T16:17:49.217 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:49.217+0000 7f71475ff700 1 librados: shutdown 2024-04-26T16:17:49.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[29234]: cluster 2024-04-26T16:17:47.491275+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:49.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[29234]: cluster 2024-04-26T16:17:47.491275+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:49.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[29234]: cluster 2024-04-26T16:17:47.491367+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[29234]: cluster 2024-04-26T16:17:47.491367+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[29234]: cluster 2024-04-26T16:17:47.913247+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[29234]: cluster 2024-04-26T16:17:47.913247+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[21463]: cluster 2024-04-26T16:17:47.491275+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[21463]: cluster 2024-04-26T16:17:47.491275+0000 osd.5 (osd.5) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[21463]: cluster 2024-04-26T16:17:47.491367+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[21463]: cluster 2024-04-26T16:17:47.491367+0000 osd.5 (osd.5) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[21463]: cluster 2024-04-26T16:17:47.913247+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-04-26T16:17:49.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:48 smithi050 bash[21463]: cluster 2024-04-26T16:17:47.913247+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2024-04-26T16:17:49.925 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-04-26T16:17:49.925 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:17:49.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-26T16:17:49.939 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:17:49.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-04-26T16:17:49.954 DEBUG:teuthology.orchestra.run.smithi190:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@iscsi.iscsi.a.service 2024-04-26T16:17:49.999 INFO:tasks.cephadm:Adding prometheus.a on smithi190 2024-04-26T16:17:49.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi190=a' 2024-04-26T16:17:50.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:49 smithi190 bash[23868]: cluster 2024-04-26T16:17:48.752283+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:49 smithi190 bash[23868]: cluster 2024-04-26T16:17:48.752283+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:49 smithi190 bash[23868]: audit 2024-04-26T16:17:49.223134+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:49 smithi190 bash[23868]: audit 2024-04-26T16:17:49.223134+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:49 smithi190 bash[23868]: audit 2024-04-26T16:17:49.260270+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:49 smithi190 bash[23868]: audit 2024-04-26T16:17:49.260270+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[21463]: cluster 2024-04-26T16:17:48.752283+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[21463]: cluster 2024-04-26T16:17:48.752283+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[21463]: audit 2024-04-26T16:17:49.223134+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[21463]: audit 2024-04-26T16:17:49.223134+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[21463]: audit 2024-04-26T16:17:49.260270+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[21463]: audit 2024-04-26T16:17:49.260270+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[29234]: cluster 2024-04-26T16:17:48.752283+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[29234]: cluster 2024-04-26T16:17:48.752283+0000 mgr.y (mgr.14152) 356 : cluster [DBG] pgmap v369: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[29234]: audit 2024-04-26T16:17:49.223134+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[29234]: audit 2024-04-26T16:17:49.223134+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[29234]: audit 2024-04-26T16:17:49.260270+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:50.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:49 smithi050 bash[29234]: audit 2024-04-26T16:17:49.260270+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.209039+0000 mgr.y (mgr.14152) 357 : audit [DBG] from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.209039+0000 mgr.y (mgr.14152) 357 : audit [DBG] from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: cephadm 2024-04-26T16:17:49.215387+0000 mgr.y (mgr.14152) 358 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: cephadm 2024-04-26T16:17:49.215387+0000 mgr.y (mgr.14152) 358 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.937744+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.937744+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.939797+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.939797+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.950769+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.950769+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.959253+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.959253+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.961891+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.961891+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.992073+0000 mon.a (mon.0) 771 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: audit 2024-04-26T16:17:49.992073+0000 mon.a (mon.0) 771 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: cephadm 2024-04-26T16:17:49.993817+0000 mgr.y (mgr.14152) 359 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi190 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: cephadm 2024-04-26T16:17:49.993817+0000 mgr.y (mgr.14152) 359 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi190 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: cluster 2024-04-26T16:17:50.892761+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-04-26T16:17:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:50 smithi190 bash[23868]: cluster 2024-04-26T16:17:50.892761+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-04-26T16:17:51.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.209039+0000 mgr.y (mgr.14152) 357 : audit [DBG] from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.209039+0000 mgr.y (mgr.14152) 357 : audit [DBG] from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: cephadm 2024-04-26T16:17:49.215387+0000 mgr.y (mgr.14152) 358 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: cephadm 2024-04-26T16:17:49.215387+0000 mgr.y (mgr.14152) 358 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.937744+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.937744+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.939797+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.939797+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.950769+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.950769+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.959253+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.959253+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.961891+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-26T16:17:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.209039+0000 mgr.y (mgr.14152) 357 : audit [DBG] from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.209039+0000 mgr.y (mgr.14152) 357 : audit [DBG] from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: cephadm 2024-04-26T16:17:49.215387+0000 mgr.y (mgr.14152) 358 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: cephadm 2024-04-26T16:17:49.215387+0000 mgr.y (mgr.14152) 358 : cephadm [INF] Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.937744+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.937744+0000 mon.a (mon.0) 766 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.939797+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.939797+0000 mon.a (mon.0) 767 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.950769+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.950769+0000 mon.a (mon.0) 768 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.959253+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.959253+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-26T16:17:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.961891+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.961891+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.992073+0000 mon.a (mon.0) 771 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: audit 2024-04-26T16:17:49.992073+0000 mon.a (mon.0) 771 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: cephadm 2024-04-26T16:17:49.993817+0000 mgr.y (mgr.14152) 359 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi190 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: cephadm 2024-04-26T16:17:49.993817+0000 mgr.y (mgr.14152) 359 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi190 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: cluster 2024-04-26T16:17:50.892761+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[21463]: cluster 2024-04-26T16:17:50.892761+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.961891+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.992073+0000 mon.a (mon.0) 771 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: audit 2024-04-26T16:17:49.992073+0000 mon.a (mon.0) 771 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: cephadm 2024-04-26T16:17:49.993817+0000 mgr.y (mgr.14152) 359 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi190 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: cephadm 2024-04-26T16:17:49.993817+0000 mgr.y (mgr.14152) 359 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi190 2024-04-26T16:17:51.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: cluster 2024-04-26T16:17:50.892761+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-04-26T16:17:51.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:50 smithi050 bash[29234]: cluster 2024-04-26T16:17:50.892761+0000 mon.a (mon.0) 772 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2024-04-26T16:17:52.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:51 smithi190 bash[23868]: cluster 2024-04-26T16:17:50.753530+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:52.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:51 smithi190 bash[23868]: cluster 2024-04-26T16:17:50.753530+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:51 smithi050 bash[21463]: cluster 2024-04-26T16:17:50.753530+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:52.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:51 smithi050 bash[21463]: cluster 2024-04-26T16:17:50.753530+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:51 smithi050 bash[29234]: cluster 2024-04-26T16:17:50.753530+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:51 smithi050 bash[29234]: cluster 2024-04-26T16:17:50.753530+0000 mgr.y (mgr.14152) 360 : cluster [DBG] pgmap v370: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:52.571 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.571 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.572 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.572 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.857 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.858 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.858 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.858 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.859 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.859 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:17:52.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:52 smithi190 systemd[1]: Started Ceph iscsi.iscsi.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: cluster 2024-04-26T16:17:52.754515+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: cluster 2024-04-26T16:17:52.754515+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.924939+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.924939+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.935665+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.935665+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.979292+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.979292+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: cephadm 2024-04-26T16:17:52.981104+0000 mgr.y (mgr.14152) 362 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-04-26T16:17:54.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: cephadm 2024-04-26T16:17:52.981104+0000 mgr.y (mgr.14152) 362 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-04-26T16:17:54.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.997025+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:52.997025+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:53.061796+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:54.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:53 smithi190 bash[23868]: audit 2024-04-26T16:17:53.061796+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: cluster 2024-04-26T16:17:52.754515+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: cluster 2024-04-26T16:17:52.754515+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.924939+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.924939+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.935665+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.935665+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.979292+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.979292+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: cephadm 2024-04-26T16:17:52.981104+0000 mgr.y (mgr.14152) 362 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: cephadm 2024-04-26T16:17:52.981104+0000 mgr.y (mgr.14152) 362 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.997025+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:52.997025+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:53.061796+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[21463]: audit 2024-04-26T16:17:53.061796+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: cluster 2024-04-26T16:17:52.754515+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: cluster 2024-04-26T16:17:52.754515+0000 mgr.y (mgr.14152) 361 : cluster [DBG] pgmap v372: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:54.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.924939+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.924939+0000 mon.a (mon.0) 773 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.935665+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.935665+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.979292+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.979292+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: cephadm 2024-04-26T16:17:52.981104+0000 mgr.y (mgr.14152) 362 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: cephadm 2024-04-26T16:17:52.981104+0000 mgr.y (mgr.14152) 362 : cephadm [INF] Checking pool "datapool" exists for service iscsi.datapool 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.997025+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:52.997025+0000 mon.a (mon.0) 776 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:53.061796+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:53 smithi050 bash[29234]: audit 2024-04-26T16:17:53.061796+0000 mon.a (mon.0) 777 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:17:54.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:54 smithi190 bash[47603]: debug Started the configuration object watcher 2024-04-26T16:17:54.775 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:54 smithi190 bash[47603]: debug Checking for config object changes every 1s 2024-04-26T16:17:54.776 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:54 smithi190 bash[47603]: debug Processing osd blocklist entries for this node 2024-04-26T16:17:54.776 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:54 smithi190 bash[47603]: debug Reading the configuration object to update local LIO configuration 2024-04-26T16:17:54.776 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:54 smithi190 bash[47603]: debug Configuration does not have an entry for this host(smithi190.front.sepia.ceph.com) - nothing to define to LIO 2024-04-26T16:17:55.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:17:54 smithi190 bash[47603]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-04-26T16:17:55.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:55 smithi190 bash[23868]: cluster 2024-04-26T16:17:54.018082+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-04-26T16:17:55.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:55 smithi190 bash[23868]: cluster 2024-04-26T16:17:54.018082+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-04-26T16:17:55.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:55 smithi190 bash[23868]: audit 2024-04-26T16:17:54.769319+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.190:0/2913150664' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T16:17:55.455 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:55 smithi190 bash[23868]: audit 2024-04-26T16:17:54.769319+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.190:0/2913150664' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T16:17:55.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[21463]: cluster 2024-04-26T16:17:54.018082+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-04-26T16:17:55.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[21463]: cluster 2024-04-26T16:17:54.018082+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-04-26T16:17:55.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[21463]: audit 2024-04-26T16:17:54.769319+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.190:0/2913150664' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T16:17:55.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[21463]: audit 2024-04-26T16:17:54.769319+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.190:0/2913150664' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T16:17:55.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[29234]: cluster 2024-04-26T16:17:54.018082+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-04-26T16:17:55.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[29234]: cluster 2024-04-26T16:17:54.018082+0000 mon.a (mon.0) 778 : cluster [DBG] mgrmap e16: y(active, since 9m), standbys: x 2024-04-26T16:17:55.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[29234]: audit 2024-04-26T16:17:54.769319+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.190:0/2913150664' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T16:17:55.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:55 smithi050 bash[29234]: audit 2024-04-26T16:17:54.769319+0000 mon.b (mon.1) 28 : audit [DBG] from='client.? 172.21.15.190:0/2913150664' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-26T16:17:56.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:56 smithi190 bash[23868]: cluster 2024-04-26T16:17:54.755571+0000 mgr.y (mgr.14152) 363 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:56.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:56 smithi190 bash[23868]: cluster 2024-04-26T16:17:54.755571+0000 mgr.y (mgr.14152) 363 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:56.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:56 smithi050 bash[21463]: cluster 2024-04-26T16:17:54.755571+0000 mgr.y (mgr.14152) 363 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:56.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:56 smithi050 bash[21463]: cluster 2024-04-26T16:17:54.755571+0000 mgr.y (mgr.14152) 363 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:56.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:56 smithi050 bash[29234]: cluster 2024-04-26T16:17:54.755571+0000 mgr.y (mgr.14152) 363 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:56.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:56 smithi050 bash[29234]: cluster 2024-04-26T16:17:54.755571+0000 mgr.y (mgr.14152) 363 : cluster [DBG] pgmap v373: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:17:56.812 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:17:58.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:57 smithi190 bash[23868]: audit 2024-04-26T16:17:56.682764+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:58.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:57 smithi190 bash[23868]: audit 2024-04-26T16:17:56.682764+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:58.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:57 smithi190 bash[23868]: cluster 2024-04-26T16:17:56.756816+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v374: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 115 B/s wr, 1 op/s 2024-04-26T16:17:58.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:57 smithi190 bash[23868]: cluster 2024-04-26T16:17:56.756816+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v374: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 115 B/s wr, 1 op/s 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[21463]: audit 2024-04-26T16:17:56.682764+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[21463]: audit 2024-04-26T16:17:56.682764+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[21463]: cluster 2024-04-26T16:17:56.756816+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v374: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 115 B/s wr, 1 op/s 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[21463]: cluster 2024-04-26T16:17:56.756816+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v374: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 115 B/s wr, 1 op/s 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[29234]: audit 2024-04-26T16:17:56.682764+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[29234]: audit 2024-04-26T16:17:56.682764+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[29234]: cluster 2024-04-26T16:17:56.756816+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v374: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 115 B/s wr, 1 op/s 2024-04-26T16:17:58.124 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:57 smithi050 bash[29234]: cluster 2024-04-26T16:17:56.756816+0000 mgr.y (mgr.14152) 364 : cluster [DBG] pgmap v374: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 115 B/s rd, 115 B/s wr, 1 op/s 2024-04-26T16:17:59.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/887415416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 msgr2=0x7f9b8410b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:59.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/887415416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b8410b8f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9b740099c0 tx=0x7f9b74009cd0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/887415416 shutdown_connections 2024-04-26T16:17:59.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/887415416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b8410b8f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.383 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/887415416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b841034a0 0x7f9b841038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/887415416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b841046a0 0x7f9b84104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/887415416 >> 172.21.15.190:0/887415416 conn(0x7f9b840fea40 msgr2=0x7f9b84100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/887415416 shutdown_connections 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/887415416 wait complete. 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 librados: starting msgr at 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 librados: starting objecter 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 Processor -- start 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- start start 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 librados: setting wanted keys 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 librados: calling monclient init 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b841034a0 0x7f9b840771c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b841046a0 0x7f9b84075810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b84075d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:59.384 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8410e420 con 0x7f9b841034a0 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9b8410e2a0 con 0x7f9b84105040 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9b8410e5a0 con 0x7f9b841046a0 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b84075d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b84075d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:54828/0 (socket says 172.21.15.190:54828) 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 -- 172.21.15.190:0/1656711100 learned_addr learned my addr 172.21.15.190:0/1656711100 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8954a700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b841034a0 0x7f9b840771c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b88d49700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b841046a0 0x7f9b84075810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 -- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b841046a0 msgr2=0x7f9b84075810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b841046a0 0x7f9b84075810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 -- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b841034a0 msgr2=0x7f9b840771c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b841034a0 0x7f9b840771c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 -- 172.21.15.190:0/1656711100 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b84076600 con 0x7f9b84105040 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b89d4b700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b84075d50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b740037f0 tx=0x7f9b740057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b827fc700 10 client.?.objecter ms_handle_connect 0x7f9b84105040 2024-04-26T16:17:59.385 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b827fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74027070 con 0x7f9b84105040 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b841a2d10 con 0x7f9b84105040 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9b740040b0 con 0x7f9b84105040 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 10 client.24421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74022d70 con 0x7f9b84105040 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b841a3110 con 0x7f9b84105040 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.389+0000 7f9b8b7ae700 1 librados: init done 2024-04-26T16:17:59.387 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b8406a110 con 0x7f9b84105040 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 93954+0+0 (secure 0 0 0) 0x7f9b7402a070 con 0x7f9b84105040 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f9b6c071de0 0x7f9b6c0742a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(93..93 src has 22..93) ==== 7148+0+0 (secure 0 0 0) 0x7f9b74091d30 con 0x7f9b84105040 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 10 client.24421.objecter ms_dispatch 0x7f9b84001550 osd_map(93..93 src has 22..93) 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 3 client.24421.objecter handle_osd_map got epochs [93,93] > 0 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 3 client.24421.objecter handle_osd_map decoding full epoch 93 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 20 client.24421.objecter dump_active .. 0 homeless 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b8954a700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f9b6c071de0 0x7f9b6c0742a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b827fc700 10 client.24421.objecter ms_handle_connect 0x7f9b6c071de0 2024-04-26T16:17:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.393+0000 7f9b8954a700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f9b6c071de0 0x7f9b6c0742a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9b84104500 tx=0x7f9b7c00c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:17:59.391 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.397+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f9b74013070 con 0x7f9b84105040 2024-04-26T16:17:59.619 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.625+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}) -- 0x7f9b8402e720 con 0x7f9b6c071de0 2024-04-26T16:17:59.634 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b827fc700 1 -- 172.21.15.190:0/1656711100 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+31 (secure 0 0 0) 0x7f9b8402e720 con 0x7f9b6c071de0 2024-04-26T16:17:59.634 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled prometheus update... 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 10 librados: watch_flush enter 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 10 librados: watch_flush exit 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 20 client.24421.objecter shutdown clearing up homeless session... 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 10 client.24421.objecter successfully canceled tick 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f9b6c071de0 msgr2=0x7f9b6c0742a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f9b6c071de0 0x7f9b6c0742a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9b84104500 tx=0x7f9b7c00c040 comp rx=0 tx=0).stop 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 msgr2=0x7f9b84075d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b84075d50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9b740037f0 tx=0x7f9b740057c0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 shutdown_connections 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f9b6c071de0 0x7f9b6c0742a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.637 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b84105040 0x7f9b84075d50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9b841034a0 0x7f9b840771c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 --2- 172.21.15.190:0/1656711100 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9b841046a0 0x7f9b84075810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:17:59.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.641+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 >> 172.21.15.190:0/1656711100 conn(0x7f9b840fea40 msgr2=0x7f9b841078d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:17:59.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.645+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 shutdown_connections 2024-04-26T16:17:59.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.645+0000 7f9b8b7ae700 1 -- 172.21.15.190:0/1656711100 wait complete. 2024-04-26T16:17:59.638 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:17:59.645+0000 7f9b8b7ae700 1 librados: shutdown 2024-04-26T16:18:00.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: cluster 2024-04-26T16:17:58.757842+0000 mgr.y (mgr.14152) 365 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-04-26T16:18:00.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: cluster 2024-04-26T16:17:58.757842+0000 mgr.y (mgr.14152) 365 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-04-26T16:18:00.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.557845+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.557845+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.568689+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.568689+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.571456+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.571456+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.573295+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.573295+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.583626+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.583626+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.644414+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.644414+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.700829+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.700829+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:00.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.705390+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.705390+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.713396+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.713396+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.725234+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.725234+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.735739+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.735739+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.739691+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:17:59 smithi190 bash[23868]: audit 2024-04-26T16:17:59.739691+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: cluster 2024-04-26T16:17:58.757842+0000 mgr.y (mgr.14152) 365 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: cluster 2024-04-26T16:17:58.757842+0000 mgr.y (mgr.14152) 365 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.557845+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.557845+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.568689+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.568689+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.571456+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:00.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.571456+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.573295+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.573295+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.583626+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.583626+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.644414+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.644414+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.700829+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.700829+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.705390+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.705390+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.713396+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.713396+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.725234+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:00.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.725234+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.735739+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.735739+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.739691+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[21463]: audit 2024-04-26T16:17:59.739691+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: cluster 2024-04-26T16:17:58.757842+0000 mgr.y (mgr.14152) 365 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: cluster 2024-04-26T16:17:58.757842+0000 mgr.y (mgr.14152) 365 : cluster [DBG] pgmap v375: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2024-04-26T16:18:00.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.557845+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.557845+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.568689+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.568689+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.571456+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.571456+0000 mon.a (mon.0) 782 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.573295+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.573295+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.583626+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.583626+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.644414+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.644414+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.700829+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.700829+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:00.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.705390+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.705390+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.713396+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.713396+0000 mon.a (mon.0) 788 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.725234+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.725234+0000 mon.a (mon.0) 789 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.735739+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.735739+0000 mon.a (mon.0) 790 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.739691+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:00.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:17:59 smithi050 bash[29234]: audit 2024-04-26T16:17:59.739691+0000 mon.a (mon.0) 791 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:00.358 DEBUG:teuthology.orchestra.run.smithi190:prometheus.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-26T16:18:00.361 INFO:tasks.cephadm:Adding node-exporter.a on smithi050 2024-04-26T16:18:00.361 INFO:tasks.cephadm:Adding node-exporter.b on smithi190 2024-04-26T16:18:00.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi050=a;smithi190=b' 2024-04-26T16:18:01.071 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:00 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.632042+0000 mgr.y (mgr.14152) 366 : audit [DBG] from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.632042+0000 mgr.y (mgr.14152) 366 : audit [DBG] from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: cephadm 2024-04-26T16:17:59.635795+0000 mgr.y (mgr.14152) 367 : cephadm [INF] Saving service prometheus spec with placement smithi190=a;count:1 2024-04-26T16:18:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: cephadm 2024-04-26T16:17:59.635795+0000 mgr.y (mgr.14152) 367 : cephadm [INF] Saving service prometheus spec with placement smithi190=a;count:1 2024-04-26T16:18:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.702260+0000 mgr.y (mgr.14152) 368 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.702260+0000 mgr.y (mgr.14152) 368 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: cephadm 2024-04-26T16:17:59.704687+0000 mgr.y (mgr.14152) 369 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.190:5000 to Dashboard 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: cephadm 2024-04-26T16:17:59.704687+0000 mgr.y (mgr.14152) 369 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.190:5000 to Dashboard 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.706354+0000 mgr.y (mgr.14152) 370 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.706354+0000 mgr.y (mgr.14152) 370 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.726302+0000 mgr.y (mgr.14152) 371 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:17:59.726302+0000 mgr.y (mgr.14152) 371 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.269203+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.269203+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.279725+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.279725+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.281403+0000 mon.a (mon.0) 794 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.281403+0000 mon.a (mon.0) 794 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:01.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.282624+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.282624+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.290862+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:00 smithi190 bash[23868]: audit 2024-04-26T16:18:00.290862+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.632042+0000 mgr.y (mgr.14152) 366 : audit [DBG] from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:01.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.632042+0000 mgr.y (mgr.14152) 366 : audit [DBG] from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:01.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: cephadm 2024-04-26T16:17:59.635795+0000 mgr.y (mgr.14152) 367 : cephadm [INF] Saving service prometheus spec with placement smithi190=a;count:1 2024-04-26T16:18:01.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: cephadm 2024-04-26T16:17:59.635795+0000 mgr.y (mgr.14152) 367 : cephadm [INF] Saving service prometheus spec with placement smithi190=a;count:1 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.702260+0000 mgr.y (mgr.14152) 368 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.702260+0000 mgr.y (mgr.14152) 368 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: cephadm 2024-04-26T16:17:59.704687+0000 mgr.y (mgr.14152) 369 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.190:5000 to Dashboard 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: cephadm 2024-04-26T16:17:59.704687+0000 mgr.y (mgr.14152) 369 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.190:5000 to Dashboard 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.706354+0000 mgr.y (mgr.14152) 370 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.706354+0000 mgr.y (mgr.14152) 370 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.726302+0000 mgr.y (mgr.14152) 371 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:17:59.726302+0000 mgr.y (mgr.14152) 371 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.269203+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.269203+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.279725+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.279725+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.281403+0000 mon.a (mon.0) 794 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.281403+0000 mon.a (mon.0) 794 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:01.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.282624+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.282624+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.290862+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[21463]: audit 2024-04-26T16:18:00.290862+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.632042+0000 mgr.y (mgr.14152) 366 : audit [DBG] from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.632042+0000 mgr.y (mgr.14152) 366 : audit [DBG] from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: cephadm 2024-04-26T16:17:59.635795+0000 mgr.y (mgr.14152) 367 : cephadm [INF] Saving service prometheus spec with placement smithi190=a;count:1 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: cephadm 2024-04-26T16:17:59.635795+0000 mgr.y (mgr.14152) 367 : cephadm [INF] Saving service prometheus spec with placement smithi190=a;count:1 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.702260+0000 mgr.y (mgr.14152) 368 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:01.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.702260+0000 mgr.y (mgr.14152) 368 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: cephadm 2024-04-26T16:17:59.704687+0000 mgr.y (mgr.14152) 369 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.190:5000 to Dashboard 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: cephadm 2024-04-26T16:17:59.704687+0000 mgr.y (mgr.14152) 369 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.190:5000 to Dashboard 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.706354+0000 mgr.y (mgr.14152) 370 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.706354+0000 mgr.y (mgr.14152) 370 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.726302+0000 mgr.y (mgr.14152) 371 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:17:59.726302+0000 mgr.y (mgr.14152) 371 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi190"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.269203+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.269203+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.279725+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.279725+0000 mon.a (mon.0) 793 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.281403+0000 mon.a (mon.0) 794 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.281403+0000 mon.a (mon.0) 794 : audit [DBG] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.282624+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.282624+0000 mon.a (mon.0) 795 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:01.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.290862+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:01.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:00 smithi050 bash[29234]: audit 2024-04-26T16:18:00.290862+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cephadm 2024-04-26T16:18:00.571449+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Deploying daemon prometheus.a on smithi190 2024-04-26T16:18:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cephadm 2024-04-26T16:18:00.571449+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Deploying daemon prometheus.a on smithi190 2024-04-26T16:18:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cluster 2024-04-26T16:18:00.759036+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cluster 2024-04-26T16:18:00.759036+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cluster 2024-04-26T16:18:00.973111+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cluster 2024-04-26T16:18:00.973111+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cluster 2024-04-26T16:18:00.973201+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:01 smithi190 bash[23868]: cluster 2024-04-26T16:18:00.973201+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cephadm 2024-04-26T16:18:00.571449+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Deploying daemon prometheus.a on smithi190 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cephadm 2024-04-26T16:18:00.571449+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Deploying daemon prometheus.a on smithi190 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cluster 2024-04-26T16:18:00.759036+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cluster 2024-04-26T16:18:00.759036+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cluster 2024-04-26T16:18:00.973111+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cluster 2024-04-26T16:18:00.973111+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cluster 2024-04-26T16:18:00.973201+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:02.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[21463]: cluster 2024-04-26T16:18:00.973201+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cephadm 2024-04-26T16:18:00.571449+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Deploying daemon prometheus.a on smithi190 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cephadm 2024-04-26T16:18:00.571449+0000 mgr.y (mgr.14152) 372 : cephadm [INF] Deploying daemon prometheus.a on smithi190 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cluster 2024-04-26T16:18:00.759036+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cluster 2024-04-26T16:18:00.759036+0000 mgr.y (mgr.14152) 373 : cluster [DBG] pgmap v376: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cluster 2024-04-26T16:18:00.973111+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cluster 2024-04-26T16:18:00.973111+0000 osd.7 (osd.7) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cluster 2024-04-26T16:18:00.973201+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:01 smithi050 bash[29234]: cluster 2024-04-26T16:18:00.973201+0000 osd.7 (osd.7) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:03.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:02 smithi190 bash[23868]: cluster 2024-04-26T16:18:01.909958+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-04-26T16:18:03.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:02 smithi190 bash[23868]: cluster 2024-04-26T16:18:01.909958+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-04-26T16:18:03.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:02 smithi050 bash[21463]: cluster 2024-04-26T16:18:01.909958+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-04-26T16:18:03.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:02 smithi050 bash[21463]: cluster 2024-04-26T16:18:01.909958+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-04-26T16:18:03.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:02 smithi050 bash[29234]: cluster 2024-04-26T16:18:01.909958+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-04-26T16:18:03.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:02 smithi050 bash[29234]: cluster 2024-04-26T16:18:01.909958+0000 mon.a (mon.0) 797 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2024-04-26T16:18:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:03 smithi190 bash[23868]: cluster 2024-04-26T16:18:02.759859+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:03 smithi190 bash[23868]: cluster 2024-04-26T16:18:02.759859+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:04.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:03 smithi050 bash[21463]: cluster 2024-04-26T16:18:02.759859+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:04.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:03 smithi050 bash[21463]: cluster 2024-04-26T16:18:02.759859+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:04.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:03 smithi050 bash[29234]: cluster 2024-04-26T16:18:02.759859+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:04.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:03 smithi050 bash[29234]: cluster 2024-04-26T16:18:02.759859+0000 mgr.y (mgr.14152) 374 : cluster [DBG] pgmap v378: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:04.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:04 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:18:05.169 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.145638+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.145638+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.145748+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.145748+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: audit 2024-04-26T16:18:04.452970+0000 mgr.y (mgr.14152) 375 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: audit 2024-04-26T16:18:04.452970+0000 mgr.y (mgr.14152) 375 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.760951+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.760951+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.930769+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-04-26T16:18:06.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:05 smithi190 bash[23868]: cluster 2024-04-26T16:18:04.930769+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-04-26T16:18:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.145638+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.145638+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.145748+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.145748+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: audit 2024-04-26T16:18:04.452970+0000 mgr.y (mgr.14152) 375 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: audit 2024-04-26T16:18:04.452970+0000 mgr.y (mgr.14152) 375 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.760951+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.760951+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.930769+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[21463]: cluster 2024-04-26T16:18:04.930769+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.145638+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.145638+0000 osd.3 (osd.3) 9 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.145748+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.145748+0000 osd.3 (osd.3) 10 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: audit 2024-04-26T16:18:04.452970+0000 mgr.y (mgr.14152) 375 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: audit 2024-04-26T16:18:04.452970+0000 mgr.y (mgr.14152) 375 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.760951+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.760951+0000 mgr.y (mgr.14152) 376 : cluster [DBG] pgmap v379: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 102 B/s wr, 1 op/s 2024-04-26T16:18:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.930769+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-04-26T16:18:06.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:05 smithi050 bash[29234]: cluster 2024-04-26T16:18:04.930769+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2024-04-26T16:18:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:07 smithi190 bash[23868]: cluster 2024-04-26T16:18:06.761997+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:07 smithi190 bash[23868]: cluster 2024-04-26T16:18:06.761997+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:08.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:07 smithi050 bash[21463]: cluster 2024-04-26T16:18:06.761997+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:08.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:07 smithi050 bash[21463]: cluster 2024-04-26T16:18:06.761997+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:08.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:07 smithi050 bash[29234]: cluster 2024-04-26T16:18:06.761997+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:08.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:07 smithi050 bash[29234]: cluster 2024-04-26T16:18:06.761997+0000 mgr.y (mgr.14152) 377 : cluster [DBG] pgmap v381: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:09 smithi190 bash[23868]: cluster 2024-04-26T16:18:08.762988+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:09 smithi190 bash[23868]: cluster 2024-04-26T16:18:08.762988+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:10.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:09 smithi050 bash[21463]: cluster 2024-04-26T16:18:08.762988+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:10.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:09 smithi050 bash[21463]: cluster 2024-04-26T16:18:08.762988+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:10.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:09 smithi050 bash[29234]: cluster 2024-04-26T16:18:08.762988+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:10.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:09 smithi050 bash[29234]: cluster 2024-04-26T16:18:08.762988+0000 mgr.y (mgr.14152) 378 : cluster [DBG] pgmap v382: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- 172.21.15.190:0/2097958237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 msgr2=0x7f7968105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- 172.21.15.190:0/2097958237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f7968105af0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f79580099c0 tx=0x7f7958009cd0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- 172.21.15.190:0/2097958237 shutdown_connections 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- 172.21.15.190:0/2097958237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f7968105af0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- 172.21.15.190:0/2097958237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7968107270 0x7f796810db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- 172.21.15.190:0/2097958237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 0x7f7968106d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- 172.21.15.190:0/2097958237 >> 172.21.15.190:0/2097958237 conn(0x7f7968100c70 msgr2=0x7f79681030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:11.281 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- 172.21.15.190:0/2097958237 shutdown_connections 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- 172.21.15.190:0/2097958237 wait complete. 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 librados: starting msgr at 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 librados: starting objecter 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 Processor -- start 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- start start 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 librados: setting wanted keys 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 librados: calling monclient init 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f796819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 0x7f796819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7968107270 0x7f79681a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7968110680 con 0x7f79681068d0 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7968110500 con 0x7f79681056d0 2024-04-26T16:18:11.282 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f796deca700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7968110800 con 0x7f7968107270 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f79677fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f796819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f79677fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f796819ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:59040/0 (socket says 172.21.15.190:59040) 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.284+0000 7f79677fe700 1 -- 172.21.15.190:0/137558490 learned_addr learned my addr 172.21.15.190:0/137558490 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f795ffff700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 0x7f796819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f7967fff700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7968107270 0x7f79681a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79677fe700 1 -- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7968107270 msgr2=0x7f79681a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79677fe700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7968107270 0x7f79681a23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79677fe700 1 -- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 msgr2=0x7f796819b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79677fe700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 0x7f796819b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79677fe700 1 -- 172.21.15.190:0/137558490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79681a2a70 con 0x7f79681056d0 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f795ffff700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 0x7f796819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79677fe700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f796819ae60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79580047b0 tx=0x7f795800bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 10 client.?.objecter ms_handle_connect 0x7f79681056d0 2024-04-26T16:18:11.283 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7958027070 con 0x7f79681056d0 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7958022470 con 0x7f79681056d0 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795800f6c0 con 0x7f79681056d0 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79681a2d00 con 0x7f79681056d0 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f796deca700 10 client.24427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79681a3100 con 0x7f79681056d0 2024-04-26T16:18:11.285 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f796deca700 1 librados: init done 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796806a110 con 0x7f79681056d0 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) ==== 93954+0+0 (secure 0 0 0) 0x7f795802a070 con 0x7f79681056d0 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7954071f10 0x7f79540743d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(95..95 src has 22..95) ==== 7120+0+0 (secure 0 0 0) 0x7f7958092700 con 0x7f79681056d0 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 10 client.24427.objecter ms_dispatch 0x7f7968001550 osd_map(95..95 src has 22..95) 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 3 client.24427.objecter handle_osd_map got epochs [95,95] > 0 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 3 client.24427.objecter handle_osd_map decoding full epoch 95 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f79657fa700 20 client.24427.objecter dump_active .. 0 homeless 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.288+0000 7f795ffff700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7954071f10 0x7f79540743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.292+0000 7f79657fa700 10 client.24427.objecter ms_handle_connect 0x7f7954071f10 2024-04-26T16:18:11.286 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.292+0000 7f795ffff700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7954071f10 0x7f79540743d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7950005fd0 tx=0x7f7950005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:11.288 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.292+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7958031070 con 0x7f79681056d0 2024-04-26T16:18:11.520 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.524+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}) -- 0x7f7968063860 con 0x7f7954071f10 2024-04-26T16:18:11.537 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f79657fa700 1 -- 172.21.15.190:0/137558490 <== mgr.14152 v2:172.21.15.50:6800/126446279 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+34 (secure 0 0 0) 0x7f7968063860 con 0x7f7954071f10 2024-04-26T16:18:11.537 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled node-exporter update... 2024-04-26T16:18:11.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 10 librados: watch_flush enter 2024-04-26T16:18:11.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 10 librados: watch_flush exit 2024-04-26T16:18:11.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 20 client.24427.objecter shutdown clearing up homeless session... 2024-04-26T16:18:11.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 10 client.24427.objecter successfully canceled tick 2024-04-26T16:18:11.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7954071f10 msgr2=0x7f79540743d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:11.540 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7954071f10 0x7f79540743d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7950005fd0 tx=0x7f7950005e20 comp rx=0 tx=0).stop 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 msgr2=0x7f796819ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f796819ae60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f79580047b0 tx=0x7f795800bfd0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 shutdown_connections 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7f7954071f10 0x7f79540743d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79681056d0 0x7f796819ae60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7968107270 0x7f79681a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 --2- 172.21.15.190:0/137558490 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f79681068d0 0x7f796819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 >> 172.21.15.190:0/137558490 conn(0x7f7968100c70 msgr2=0x7f7968102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 shutdown_connections 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 -- 172.21.15.190:0/137558490 wait complete. 2024-04-26T16:18:11.541 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:11.544+0000 7f796deca700 1 librados: shutdown 2024-04-26T16:18:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:11 smithi190 bash[23868]: cluster 2024-04-26T16:18:10.764197+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:18:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:11 smithi190 bash[23868]: cluster 2024-04-26T16:18:10.764197+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:18:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:11 smithi190 bash[23868]: audit 2024-04-26T16:18:11.547037+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:11 smithi190 bash[23868]: audit 2024-04-26T16:18:11.547037+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[21463]: cluster 2024-04-26T16:18:10.764197+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[21463]: cluster 2024-04-26T16:18:10.764197+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[21463]: audit 2024-04-26T16:18:11.547037+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[21463]: audit 2024-04-26T16:18:11.547037+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[29234]: cluster 2024-04-26T16:18:10.764197+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[29234]: cluster 2024-04-26T16:18:10.764197+0000 mgr.y (mgr.14152) 379 : cluster [DBG] pgmap v383: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[29234]: audit 2024-04-26T16:18:11.547037+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:12.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:11 smithi050 bash[29234]: audit 2024-04-26T16:18:11.547037+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:12.512 DEBUG:teuthology.orchestra.run.smithi050:node-exporter.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-26T16:18:12.515 DEBUG:teuthology.orchestra.run.smithi190:node-exporter.b> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-26T16:18:12.517 INFO:tasks.cephadm:Adding alertmanager.a on smithi050 2024-04-26T16:18:12.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi050=a' 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cluster 2024-04-26T16:18:11.278691+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cluster 2024-04-26T16:18:11.278691+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cluster 2024-04-26T16:18:11.278804+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cluster 2024-04-26T16:18:11.278804+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: audit 2024-04-26T16:18:11.533705+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: audit 2024-04-26T16:18:11.533705+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cephadm 2024-04-26T16:18:11.537544+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service node-exporter spec with placement smithi050=a;smithi190=b;count:2 2024-04-26T16:18:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cephadm 2024-04-26T16:18:11.537544+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service node-exporter spec with placement smithi050=a;smithi190=b;count:2 2024-04-26T16:18:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cluster 2024-04-26T16:18:11.954215+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-04-26T16:18:13.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:12 smithi190 bash[23868]: cluster 2024-04-26T16:18:11.954215+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cluster 2024-04-26T16:18:11.278691+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cluster 2024-04-26T16:18:11.278691+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cluster 2024-04-26T16:18:11.278804+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cluster 2024-04-26T16:18:11.278804+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: audit 2024-04-26T16:18:11.533705+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: audit 2024-04-26T16:18:11.533705+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cephadm 2024-04-26T16:18:11.537544+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service node-exporter spec with placement smithi050=a;smithi190=b;count:2 2024-04-26T16:18:13.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cephadm 2024-04-26T16:18:11.537544+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service node-exporter spec with placement smithi050=a;smithi190=b;count:2 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cluster 2024-04-26T16:18:11.954215+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[21463]: cluster 2024-04-26T16:18:11.954215+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cluster 2024-04-26T16:18:11.278691+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cluster 2024-04-26T16:18:11.278691+0000 osd.4 (osd.4) 7 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cluster 2024-04-26T16:18:11.278804+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cluster 2024-04-26T16:18:11.278804+0000 osd.4 (osd.4) 8 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: audit 2024-04-26T16:18:11.533705+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: audit 2024-04-26T16:18:11.533705+0000 mgr.y (mgr.14152) 380 : audit [DBG] from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi050=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cephadm 2024-04-26T16:18:11.537544+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service node-exporter spec with placement smithi050=a;smithi190=b;count:2 2024-04-26T16:18:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cephadm 2024-04-26T16:18:11.537544+0000 mgr.y (mgr.14152) 381 : cephadm [INF] Saving service node-exporter spec with placement smithi050=a;smithi190=b;count:2 2024-04-26T16:18:13.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cluster 2024-04-26T16:18:11.954215+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-04-26T16:18:13.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:12 smithi050 bash[29234]: cluster 2024-04-26T16:18:11.954215+0000 mon.a (mon.0) 800 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2024-04-26T16:18:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:13 smithi190 bash[23868]: cluster 2024-04-26T16:18:12.765096+0000 mgr.y (mgr.14152) 382 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:13 smithi190 bash[23868]: cluster 2024-04-26T16:18:12.765096+0000 mgr.y (mgr.14152) 382 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:14.373 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:13 smithi050 bash[21463]: cluster 2024-04-26T16:18:12.765096+0000 mgr.y (mgr.14152) 382 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:14.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:13 smithi050 bash[21463]: cluster 2024-04-26T16:18:12.765096+0000 mgr.y (mgr.14152) 382 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:14.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:13 smithi050 bash[29234]: cluster 2024-04-26T16:18:12.765096+0000 mgr.y (mgr.14152) 382 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:14.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:13 smithi050 bash[29234]: cluster 2024-04-26T16:18:12.765096+0000 mgr.y (mgr.14152) 382 : cluster [DBG] pgmap v385: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:14.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:14 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:18:15.163 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.164 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.164 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:14 smithi190 bash[23868]: audit 2024-04-26T16:18:14.690667+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-26T16:18:15.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:14 smithi190 bash[23868]: audit 2024-04-26T16:18:14.690667+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-26T16:18:15.164 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.165 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.165 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:14 smithi050 bash[29234]: audit 2024-04-26T16:18:14.690667+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-26T16:18:15.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:14 smithi050 bash[29234]: audit 2024-04-26T16:18:14.690667+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-26T16:18:15.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:14 smithi050 bash[21463]: audit 2024-04-26T16:18:14.690667+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-26T16:18:15.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:14 smithi050 bash[21463]: audit 2024-04-26T16:18:14.690667+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]: dispatch 2024-04-26T16:18:15.417 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.418 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.418 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.419 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.419 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.419 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.419 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.419 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.419 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:15.821 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:15 smithi190 systemd[1]: Started Ceph prometheus.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:18:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:14.462352+0000 mgr.y (mgr.14152) 383 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:14.462352+0000 mgr.y (mgr.14152) 383 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: cluster 2024-04-26T16:18:14.766080+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-26T16:18:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: cluster 2024-04-26T16:18:14.766080+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-26T16:18:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:14.967632+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-26T16:18:16.254 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:14.967632+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: cluster 2024-04-26T16:18:14.977767+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: cluster 2024-04-26T16:18:14.977767+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.478522+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.478522+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.489701+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.489701+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.501729+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.501729+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.510841+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-26T16:18:16.255 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:15 smithi190 bash[23868]: audit 2024-04-26T16:18:15.510841+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-26T16:18:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:14.462352+0000 mgr.y (mgr.14152) 383 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:14.462352+0000 mgr.y (mgr.14152) 383 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: cluster 2024-04-26T16:18:14.766080+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-26T16:18:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: cluster 2024-04-26T16:18:14.766080+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-26T16:18:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:14.967632+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:14.967632+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: cluster 2024-04-26T16:18:14.977767+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: cluster 2024-04-26T16:18:14.977767+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.478522+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.478522+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.489701+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.489701+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.501729+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.501729+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.510841+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-26T16:18:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[29234]: audit 2024-04-26T16:18:15.510841+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:14.462352+0000 mgr.y (mgr.14152) 383 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:14.462352+0000 mgr.y (mgr.14152) 383 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: cluster 2024-04-26T16:18:14.766080+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: cluster 2024-04-26T16:18:14.766080+0000 mgr.y (mgr.14152) 384 : cluster [DBG] pgmap v386: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:14.967632+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:14.967632+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.9", "id": [7, 2]}]': finished 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: cluster 2024-04-26T16:18:14.977767+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: cluster 2024-04-26T16:18:14.977767+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2024-04-26T16:18:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.478522+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.478522+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.489701+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.489701+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.501729+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.501729+0000 mon.a (mon.0) 806 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.510841+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-26T16:18:16.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:15 smithi050 bash[21463]: audit 2024-04-26T16:18:15.510841+0000 mon.a (mon.0) 807 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-26T16:18:16.538 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.368Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-26T16:18:16.538 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.368Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-26T16:18:16.538 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.368Z caller=main.go:570 level=info host_details="(Linux 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 smithi190 (none))" 2024-04-26T16:18:16.538 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.368Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-26T16:18:16.538 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.369Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-26T16:18:16.538 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.370Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.370Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.372Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.372Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.374Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.374Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.967µs 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.374Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.375Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.375Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.916µs wal_replay_duration=344.989µs wbl_replay_duration=153ns total_replay_duration=387.683µs 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.376Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.376Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.376Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.400Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.56432ms db_storage=1.132µs remote_storage=1.266µs web_handler=343ns query_engine=581ns scrape=291.82µs scrape_sd=78.37µs notify=815ns notify_sd=1.427µs rules=23.866453ms tracing=6.412µs 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.400Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-26T16:18:16.539 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[48704]: ts=2024-04-26T16:18:16.400Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-26T16:18:16.821 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[24769]: ignoring --setuser ceph since I am not root 2024-04-26T16:18:16.821 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[24769]: ignoring --setgroup ceph since I am not root 2024-04-26T16:18:16.821 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[24769]: debug 2024-04-26T16:18:16.572+0000 7f6c884ce700 1 -- 172.21.15.190:0/3099146143 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558114f585a0 con 0x558115cd8400 2024-04-26T16:18:16.822 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[24769]: debug 2024-04-26T16:18:16.716+0000 7f6c922c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T16:18:16.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21717]: ignoring --setuser ceph since I am not root 2024-04-26T16:18:16.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21717]: ignoring --setgroup ceph since I am not root 2024-04-26T16:18:16.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21717]: debug 2024-04-26T16:18:16.722+0000 7f2c9e2a7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T16:18:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[23868]: cluster 2024-04-26T16:18:15.989061+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-04-26T16:18:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[23868]: cluster 2024-04-26T16:18:15.989061+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-04-26T16:18:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[23868]: audit 2024-04-26T16:18:16.502752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-26T16:18:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[23868]: audit 2024-04-26T16:18:16.502752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-26T16:18:17.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[23868]: cluster 2024-04-26T16:18:16.517868+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-04-26T16:18:17.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:16 smithi190 bash[23868]: cluster 2024-04-26T16:18:16.517868+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-04-26T16:18:17.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[29234]: cluster 2024-04-26T16:18:15.989061+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[29234]: cluster 2024-04-26T16:18:15.989061+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[29234]: audit 2024-04-26T16:18:16.502752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[29234]: audit 2024-04-26T16:18:16.502752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[29234]: cluster 2024-04-26T16:18:16.517868+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[29234]: cluster 2024-04-26T16:18:16.517868+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21463]: cluster 2024-04-26T16:18:15.989061+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21463]: cluster 2024-04-26T16:18:15.989061+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21463]: audit 2024-04-26T16:18:16.502752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21463]: audit 2024-04-26T16:18:16.502752+0000 mon.a (mon.0) 809 : audit [INF] from='mgr.14152 172.21.15.50:0/257322837' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-26T16:18:17.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21463]: cluster 2024-04-26T16:18:16.517868+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-04-26T16:18:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:16 smithi050 bash[21463]: cluster 2024-04-26T16:18:16.517868+0000 mon.a (mon.0) 810 : cluster [DBG] mgrmap e17: y(active, since 10m), standbys: x 2024-04-26T16:18:18.325 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:18:18.571 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:18 smithi190 bash[24769]: debug 2024-04-26T16:18:18.308+0000 7f6c922c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T16:18:18.571 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:18 smithi190 bash[24769]: debug 2024-04-26T16:18:18.412+0000 7f6c922c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T16:18:18.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:18 smithi050 bash[21717]: debug 2024-04-26T16:18:18.362+0000 7f2c9e2a7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-26T16:18:18.624 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:18 smithi050 bash[21717]: debug 2024-04-26T16:18:18.466+0000 7f2c9e2a7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T16:18:19.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:18 smithi190 bash[24769]: debug 2024-04-26T16:18:18.660+0000 7f6c922c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T16:18:19.123 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:18 smithi050 bash[21717]: debug 2024-04-26T16:18:18.722+0000 7f2c9e2a7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T16:18:19.571 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:19 smithi190 bash[24769]: debug 2024-04-26T16:18:19.472+0000 7f6c922c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T16:18:19.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:19 smithi050 bash[21717]: debug 2024-04-26T16:18:19.558+0000 7f2c9e2a7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T16:18:19.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:19 smithi050 bash[21717]: debug 2024-04-26T16:18:19.674+0000 7f2c9e2a7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T16:18:19.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:19 smithi050 bash[21717]: debug 2024-04-26T16:18:19.786+0000 7f2c9e2a7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T16:18:20.042 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:19 smithi190 bash[24769]: debug 2024-04-26T16:18:19.584+0000 7f6c922c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T16:18:20.043 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:19 smithi190 bash[24769]: debug 2024-04-26T16:18:19.688+0000 7f6c922c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T16:18:20.043 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:20 smithi190 bash[24769]: debug 2024-04-26T16:18:20.048+0000 7f6c922c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T16:18:20.321 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:20 smithi190 bash[24769]: debug 2024-04-26T16:18:20.168+0000 7f6c922c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T16:18:20.321 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:20 smithi190 bash[24769]: debug 2024-04-26T16:18:20.280+0000 7f6c922c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T16:18:20.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:20 smithi050 bash[21717]: debug 2024-04-26T16:18:20.158+0000 7f2c9e2a7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-26T16:18:20.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:20 smithi050 bash[21717]: debug 2024-04-26T16:18:20.274+0000 7f2c9e2a7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T16:18:20.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:20 smithi050 bash[21717]: debug 2024-04-26T16:18:20.386+0000 7f2c9e2a7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T16:18:21.032 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:20 smithi190 bash[24769]: debug 2024-04-26T16:18:20.636+0000 7f6c922c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T16:18:21.032 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:20 smithi190 bash[24769]: debug 2024-04-26T16:18:20.880+0000 7f6c922c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T16:18:21.123 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:20 smithi050 bash[21717]: debug 2024-04-26T16:18:20.762+0000 7f2c9e2a7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T16:18:21.123 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[21717]: debug 2024-04-26T16:18:21.010+0000 7f2c9e2a7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 -- 172.21.15.190:0/266998678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228109210 msgr2=0x7ff22810b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 --2- 172.21.15.190:0/266998678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228109210 0x7ff22810b600 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff220009990 tx=0x7ff220009ca0 comp rx=0 tx=0).stop 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 -- 172.21.15.190:0/266998678 shutdown_connections 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 --2- 172.21.15.190:0/266998678 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228109210 0x7ff22810b600 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 --2- 172.21.15.190:0/266998678 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff228103fb0 0x7ff2281063a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 --2- 172.21.15.190:0/266998678 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2281068e0 0x7ff228108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 -- 172.21.15.190:0/266998678 >> 172.21.15.190:0/266998678 conn(0x7ff2280fd920 msgr2=0x7ff2280ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 -- 172.21.15.190:0/266998678 shutdown_connections 2024-04-26T16:18:21.205 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.208+0000 7ff230090700 1 -- 172.21.15.190:0/266998678 wait complete. 2024-04-26T16:18:21.207 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 librados: starting msgr at 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 librados: starting objecter 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 Processor -- start 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- start start 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 librados: setting wanted keys 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 librados: calling monclient init 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228103fb0 0x7ff228103910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2281068e0 0x7ff228101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff228109210 0x7ff2281024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff22810e420 con 0x7ff228109210 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff22810e2a0 con 0x7ff228103fb0 2024-04-26T16:18:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff22810e5a0 con 0x7ff2281068e0 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228103fb0 0x7ff228103910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228103fb0 0x7ff228103910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:38148/0 (socket says 172.21.15.190:38148) 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 -- 172.21.15.190:0/1202128521 learned_addr learned my addr 172.21.15.190:0/1202128521 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2281068e0 0x7ff228101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22e62d700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff228109210 0x7ff2281024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2281068e0 msgr2=0x7ff228101f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2281068e0 0x7ff228101f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff228109210 msgr2=0x7ff2281024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff228109210 0x7ff2281024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 -- 172.21.15.190:0/1202128521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff228102d50 con 0x7ff228103fb0 2024-04-26T16:18:21.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22e62d700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff228109210 0x7ff2281024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22de2c700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228103fb0 0x7ff228103910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff2180060b0 tx=0x7ff21800d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 10 client.?.objecter ms_handle_connect 0x7ff228103fb0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff21801a070 con 0x7ff228103fb0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2281a2d60 con 0x7ff228103fb0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff21800ddd0 con 0x7ff228103fb0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff218014600 con 0x7ff228103fb0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 10 client.24442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2281a31c0 con 0x7ff228103fb0 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 librados: init done 2024-04-26T16:18:21.210 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 17) ==== 93968+0+0 (secure 0 0 0) 0x7ff21801d070 con 0x7ff228103fb0 2024-04-26T16:18:21.211 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff22806a110 con 0x7ff228103fb0 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22d62b700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 msgr2=0x7ff210074420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/126446279 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(99..99 src has 22..99) ==== 7149+0+0 (secure 0 0 0) 0x7ff2180909c0 con 0x7ff228103fb0 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 10 client.24442.objecter ms_dispatch 0x7ff228001550 osd_map(99..99 src has 22..99) 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 3 client.24442.objecter handle_osd_map got epochs [99,99] > 0 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff226ffd700 3 client.24442.objecter handle_osd_map decoding full epoch 99 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.212+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T16:18:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.216+0000 7ff226ffd700 20 client.24442.objecter dump_active .. 0 homeless 2024-04-26T16:18:21.213 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.220+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff218010070 con 0x7ff228103fb0 2024-04-26T16:18:21.321 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:21 smithi190 bash[24769]: debug 2024-04-26T16:18:21.152+0000 7f6c922c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T16:18:21.321 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:21 smithi190 bash[24769]: debug 2024-04-26T16:18:21.268+0000 7f6c922c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T16:18:21.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.416+0000 7ff22d62b700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 msgr2=0x7ff210074420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/126446279 2024-04-26T16:18:21.409 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.416+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T16:18:21.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.440+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 --> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}) -- 0x7ff228063860 con 0x7ff210071f60 2024-04-26T16:18:21.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[21717]: debug 2024-04-26T16:18:21.290+0000 7f2c9e2a7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T16:18:21.624 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[21717]: debug 2024-04-26T16:18:21.410+0000 7f2c9e2a7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T16:18:21.813 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.820+0000 7ff22d62b700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 msgr2=0x7ff210074420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/126446279 2024-04-26T16:18:21.813 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:21.820+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T16:18:21.821 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:21 smithi190 bash[24769]: debug 2024-04-26T16:18:21.544+0000 7f6c922c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T16:18:22.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[21463]: cluster 2024-04-26T16:18:20.903668+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-04-26T16:18:22.122 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[21463]: cluster 2024-04-26T16:18:20.903668+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-04-26T16:18:22.122 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[21717]: debug 2024-04-26T16:18:21.698+0000 7f2c9e2a7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T16:18:22.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[29234]: cluster 2024-04-26T16:18:20.903668+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-04-26T16:18:22.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:21 smithi050 bash[29234]: cluster 2024-04-26T16:18:20.903668+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-04-26T16:18:22.190 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:21 smithi190 bash[24769]: debug 2024-04-26T16:18:21.960+0000 7f6c922c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:18:22.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:21 smithi190 bash[23868]: cluster 2024-04-26T16:18:20.903668+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-04-26T16:18:22.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:21 smithi190 bash[23868]: cluster 2024-04-26T16:18:20.903668+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2024-04-26T16:18:22.373 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:22 smithi050 bash[21717]: debug 2024-04-26T16:18:22.130+0000 7f2c9e2a7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:18:22.553 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:22 smithi190 bash[24769]: debug 2024-04-26T16:18:22.196+0000 7f6c922c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T16:18:22.553 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:22 smithi190 bash[24769]: debug 2024-04-26T16:18:22.312+0000 7f6c922c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T16:18:22.553 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:22 smithi190 bash[24769]: debug 2024-04-26T16:18:22.556+0000 7f6c922c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:18:22.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:22.624+0000 7ff22d62b700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 msgr2=0x7ff210074420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/126446279 2024-04-26T16:18:22.617 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:22.624+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T16:18:22.763 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:22 smithi050 bash[21717]: debug 2024-04-26T16:18:22.382+0000 7f2c9e2a7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T16:18:22.763 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:22 smithi050 bash[21717]: debug 2024-04-26T16:18:22.510+0000 7f2c9e2a7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T16:18:23.821 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:23 smithi190 bash[24769]: debug 2024-04-26T16:18:23.564+0000 7f6c922c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T16:18:23.821 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:23 smithi190 bash[24769]: debug 2024-04-26T16:18:23.680+0000 7f6c922c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T16:18:24.123 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:22 smithi050 bash[21717]: debug 2024-04-26T16:18:22.770+0000 7f2c9e2a7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T16:18:24.124 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:23 smithi050 bash[21717]: debug 2024-04-26T16:18:23.826+0000 7f2c9e2a7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T16:18:24.124 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:23 smithi050 bash[21717]: debug 2024-04-26T16:18:23.946+0000 7f2c9e2a7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T16:18:24.221 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:24.228+0000 7ff22d62b700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 msgr2=0x7ff210074420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.50:6800/126446279 2024-04-26T16:18:24.221 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:24.228+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T16:18:24.321 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:23 smithi190 bash[24769]: debug 2024-04-26T16:18:23.976+0000 7f6c922c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T16:18:24.321 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:24 smithi190 bash[24769]: debug 2024-04-26T16:18:24.084+0000 7f6c922c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T16:18:24.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:24 smithi050 bash[21717]: debug 2024-04-26T16:18:24.258+0000 7f2c9e2a7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T16:18:24.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:24 smithi050 bash[21717]: debug 2024-04-26T16:18:24.378+0000 7f2c9e2a7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T16:18:24.811 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:24 smithi190 bash[24769]: debug 2024-04-26T16:18:24.368+0000 7f6c922c2200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-26T16:18:25.011 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:25.016+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mgrmap(e 18) ==== 93968+0+0 (secure 0 0 0) 0x7ff218058ba0 con 0x7ff228103fb0 2024-04-26T16:18:25.016 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:24 smithi050 bash[21717]: debug 2024-04-26T16:18:24.678+0000 7f2c9e2a7200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-04-26T16:18:25.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:24 smithi190 bash[24769]: debug 2024-04-26T16:18:24.816+0000 7f6c922c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T16:18:25.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:24 smithi190 bash[24769]: debug 2024-04-26T16:18:24.936+0000 7f6c922c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T16:18:25.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:24 smithi190 bash[24769]: [26/Apr/2024:16:18:24] ENGINE Bus STARTING 2024-04-26T16:18:25.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[24769]: CherryPy Checker: 2024-04-26T16:18:25.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[24769]: The Application mounted at '' has an empty config. 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[24769]: [26/Apr/2024:16:18:25] ENGINE Serving on http://:::9283 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[24769]: [26/Apr/2024:16:18:25] ENGINE Bus STARTED 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: cluster 2024-04-26T16:18:24.964021+0000 mon.a (mon.0) 812 : cluster [DBG] Standby manager daemon x restarted 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: cluster 2024-04-26T16:18:24.964021+0000 mon.a (mon.0) 812 : cluster [DBG] Standby manager daemon x restarted 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.964059+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.964059+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: cluster 2024-04-26T16:18:24.964201+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: cluster 2024-04-26T16:18:24.964201+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.965462+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.965462+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.967654+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.967654+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.968107+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:18:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:25 smithi190 bash[23868]: audit 2024-04-26T16:18:24.968107+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:18:25.287 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: debug 2024-04-26T16:18:25.166+0000 7f2c9e2a7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: cluster 2024-04-26T16:18:24.964021+0000 mon.a (mon.0) 812 : cluster [DBG] Standby manager daemon x restarted 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: cluster 2024-04-26T16:18:24.964021+0000 mon.a (mon.0) 812 : cluster [DBG] Standby manager daemon x restarted 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.964059+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.964059+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: cluster 2024-04-26T16:18:24.964201+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: cluster 2024-04-26T16:18:24.964201+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.965462+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.965462+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.967654+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.967654+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.968107+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:18:25.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[29234]: audit 2024-04-26T16:18:24.968107+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: cluster 2024-04-26T16:18:24.964021+0000 mon.a (mon.0) 812 : cluster [DBG] Standby manager daemon x restarted 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: cluster 2024-04-26T16:18:24.964021+0000 mon.a (mon.0) 812 : cluster [DBG] Standby manager daemon x restarted 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.964059+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.964059+0000 mon.b (mon.1) 29 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: cluster 2024-04-26T16:18:24.964201+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: cluster 2024-04-26T16:18:24.964201+0000 mon.a (mon.0) 813 : cluster [DBG] Standby manager daemon x started 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.965462+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.965462+0000 mon.b (mon.1) 30 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.967654+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.967654+0000 mon.b (mon.1) 31 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.968107+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:18:25.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21463]: audit 2024-04-26T16:18:24.968107+0000 mon.b (mon.1) 32 : audit [DBG] from='mgr.? 172.21.15.190:0/858561557' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-26T16:18:25.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:25.332+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 8 ==== mgrmap(e 19) ==== 93585+0+0 (secure 0 0 0) 0x7ff21805c790 con 0x7ff228103fb0 2024-04-26T16:18:25.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:25.332+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 msgr2=0x7ff210074420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:18:25.326 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:25.332+0000 7ff226ffd700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/126446279,v1:172.21.15.50:6801/126446279] conn(0x7ff210071f60 0x7ff210074420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:25.563 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: debug 2024-04-26T16:18:25.294+0000 7f2c9e2a7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T16:18:25.564 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: [26/Apr/2024:16:18:25] ENGINE Bus STARTING 2024-04-26T16:18:25.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: CherryPy Checker: 2024-04-26T16:18:25.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: The Application mounted at '' has an empty config. 2024-04-26T16:18:25.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: [26/Apr/2024:16:18:25] ENGINE Serving on http://:::9283 2024-04-26T16:18:25.873 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:25 smithi050 bash[21717]: [26/Apr/2024:16:18:25] ENGINE Bus STARTED 2024-04-26T16:18:26.209 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.216+0000 7ff22f88f700 10 client.24442.objecter tick 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.023840+0000 mon.a (mon.0) 814 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.023840+0000 mon.a (mon.0) 814 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.119287+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.119287+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.119389+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.119389+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.317782+0000 mon.a (mon.0) 815 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.317782+0000 mon.a (mon.0) 815 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.318451+0000 mon.a (mon.0) 816 : cluster [INF] Activating manager daemon y 2024-04-26T16:18:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.318451+0000 mon.a (mon.0) 816 : cluster [INF] Activating manager daemon y 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.326788+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.326788+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.332110+0000 mon.a (mon.0) 818 : cluster [DBG] mgrmap e19: y(active, starting, since 0.0138691s), standbys: x 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.332110+0000 mon.a (mon.0) 818 : cluster [DBG] mgrmap e19: y(active, starting, since 0.0138691s), standbys: x 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.336650+0000 mon.c (mon.2) 11 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.336650+0000 mon.c (mon.2) 11 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.336926+0000 mon.c (mon.2) 12 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.336926+0000 mon.c (mon.2) 12 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.337187+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.337187+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.338621+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.338621+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.338842+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.338842+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:18:26.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.339126+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.339126+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.339455+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.339455+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.339806+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.339806+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.340183+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.340183+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.340488+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.340488+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.340820+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.340820+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.341138+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.341138+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.341456+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.341456+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:18:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.341796+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.341796+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.342034+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.342034+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.342733+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.342733+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.370837+0000 mon.a (mon.0) 819 : cluster [INF] Manager daemon y is now available 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.370837+0000 mon.a (mon.0) 819 : cluster [INF] Manager daemon y is now available 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.410064+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.410064+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:26.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.426183+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.426183+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.436982+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.436982+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.447097+0000 mon.c (mon.2) 29 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.447097+0000 mon.c (mon.2) 29 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.447446+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.447446+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.466779+0000 mon.c (mon.2) 30 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.466779+0000 mon.c (mon.2) 30 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.467191+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: audit 2024-04-26T16:18:25.467191+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.900135+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-04-26T16:18:26.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[23868]: cluster 2024-04-26T16:18:25.900135+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-04-26T16:18:26.332 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.336+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mon.1 v2:172.21.15.190:3300/0 9 ==== mgrmap(e 20) ==== 93712+0+0 (secure 0 0 0) 0x7ff218059470 con 0x7ff228103fb0 2024-04-26T16:18:26.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.340+0000 7ff226ffd700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff2100393d0 0x7ff21007d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:26.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.340+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}) -- 0x7ff218094e50 con 0x7ff2100393d0 2024-04-26T16:18:26.334 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.340+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff2100393d0 0x7ff21007d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:26.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.340+0000 7ff226ffd700 10 client.24442.objecter ms_handle_connect 0x7ff2100393d0 2024-04-26T16:18:26.335 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.340+0000 7ff22d62b700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff2100393d0 0x7ff21007d6f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b700 tx=0x7ff21c00b3c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:26.346 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.352+0000 7ff226ffd700 1 -- 172.21.15.190:0/1202128521 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+33 (secure 0 0 0) 0x7ff218094e50 con 0x7ff2100393d0 2024-04-26T16:18:26.346 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled alertmanager update... 2024-04-26T16:18:26.349 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 10 librados: watch_flush enter 2024-04-26T16:18:26.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 10 librados: watch_flush exit 2024-04-26T16:18:26.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 20 client.24442.objecter shutdown clearing up homeless session... 2024-04-26T16:18:26.350 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 10 client.24442.objecter successfully canceled tick 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff2100393d0 msgr2=0x7ff21007d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff2100393d0 0x7ff21007d6f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff21c00b700 tx=0x7ff21c00b3c0 comp rx=0 tx=0).stop 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228103fb0 msgr2=0x7ff228103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 --2- 172.21.15.190:0/1202128521 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff228103fb0 0x7ff228103910 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff2180060b0 tx=0x7ff21800d3b0 comp rx=0 tx=0).stop 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff22e62d700 1 -- 172.21.15.190:0/1202128521 reap_dead start 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 shutdown_connections 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 >> 172.21.15.190:0/1202128521 conn(0x7ff2280fd920 msgr2=0x7ff2281048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 shutdown_connections 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 -- 172.21.15.190:0/1202128521 wait complete. 2024-04-26T16:18:26.351 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:26.356+0000 7ff230090700 1 librados: shutdown 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.023840+0000 mon.a (mon.0) 814 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.023840+0000 mon.a (mon.0) 814 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.119287+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.119287+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.119389+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.119389+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.317782+0000 mon.a (mon.0) 815 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.317782+0000 mon.a (mon.0) 815 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:18:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.318451+0000 mon.a (mon.0) 816 : cluster [INF] Activating manager daemon y 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.318451+0000 mon.a (mon.0) 816 : cluster [INF] Activating manager daemon y 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.326788+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.326788+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.332110+0000 mon.a (mon.0) 818 : cluster [DBG] mgrmap e19: y(active, starting, since 0.0138691s), standbys: x 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.332110+0000 mon.a (mon.0) 818 : cluster [DBG] mgrmap e19: y(active, starting, since 0.0138691s), standbys: x 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.336650+0000 mon.c (mon.2) 11 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.336650+0000 mon.c (mon.2) 11 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.336926+0000 mon.c (mon.2) 12 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.336926+0000 mon.c (mon.2) 12 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.337187+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.337187+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.338621+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:18:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.338621+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.023840+0000 mon.a (mon.0) 814 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.023840+0000 mon.a (mon.0) 814 : cluster [DBG] mgrmap e18: y(active, since 10m), standbys: x 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.119287+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.119287+0000 osd.0 (osd.0) 13 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.119389+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.119389+0000 osd.0 (osd.0) 14 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.317782+0000 mon.a (mon.0) 815 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.317782+0000 mon.a (mon.0) 815 : cluster [INF] Active manager daemon y restarted 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.318451+0000 mon.a (mon.0) 816 : cluster [INF] Activating manager daemon y 2024-04-26T16:18:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.318451+0000 mon.a (mon.0) 816 : cluster [INF] Activating manager daemon y 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.326788+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.326788+0000 mon.a (mon.0) 817 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.332110+0000 mon.a (mon.0) 818 : cluster [DBG] mgrmap e19: y(active, starting, since 0.0138691s), standbys: x 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.332110+0000 mon.a (mon.0) 818 : cluster [DBG] mgrmap e19: y(active, starting, since 0.0138691s), standbys: x 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.336650+0000 mon.c (mon.2) 11 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.336650+0000 mon.c (mon.2) 11 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.336926+0000 mon.c (mon.2) 12 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.336926+0000 mon.c (mon.2) 12 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.337187+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.337187+0000 mon.c (mon.2) 13 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.338621+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:18:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.338621+0000 mon.c (mon.2) 14 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.338842+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.338842+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.339126+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.339126+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.339455+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.339455+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.339806+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.339806+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.340183+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:18:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.340183+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.340488+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.340488+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.340820+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.340820+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.341138+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.341138+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.341456+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.341456+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:18:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.341796+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.338842+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.338842+0000 mon.c (mon.2) 15 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.339126+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.339126+0000 mon.c (mon.2) 16 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.339455+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.339455+0000 mon.c (mon.2) 17 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.339806+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.339806+0000 mon.c (mon.2) 18 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.340183+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.340183+0000 mon.c (mon.2) 19 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.340488+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.340488+0000 mon.c (mon.2) 20 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-26T16:18:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.340820+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:18:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.340820+0000 mon.c (mon.2) 21 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-26T16:18:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.341138+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:18:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.341138+0000 mon.c (mon.2) 22 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:18:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.341456+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:18:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.341456+0000 mon.c (mon.2) 23 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-26T16:18:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.341796+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.341796+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.342034+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.342034+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.342733+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.342733+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.370837+0000 mon.a (mon.0) 819 : cluster [INF] Manager daemon y is now available 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.370837+0000 mon.a (mon.0) 819 : cluster [INF] Manager daemon y is now available 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.410064+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.410064+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.426183+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.426183+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.436982+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.436982+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.447097+0000 mon.c (mon.2) 29 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.447097+0000 mon.c (mon.2) 29 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.447446+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.447446+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.466779+0000 mon.c (mon.2) 30 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.466779+0000 mon.c (mon.2) 30 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.467191+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: audit 2024-04-26T16:18:25.467191+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.900135+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-04-26T16:18:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[29234]: cluster 2024-04-26T16:18:25.900135+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.341796+0000 mon.c (mon.2) 24 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.342034+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.342034+0000 mon.c (mon.2) 25 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.342733+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.342733+0000 mon.c (mon.2) 26 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.370837+0000 mon.a (mon.0) 819 : cluster [INF] Manager daemon y is now available 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.370837+0000 mon.a (mon.0) 819 : cluster [INF] Manager daemon y is now available 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.410064+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.410064+0000 mon.c (mon.2) 27 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.426183+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.426183+0000 mon.a (mon.0) 820 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.436982+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.436982+0000 mon.c (mon.2) 28 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.447097+0000 mon.c (mon.2) 29 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.447097+0000 mon.c (mon.2) 29 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.447446+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.447446+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.466779+0000 mon.c (mon.2) 30 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.466779+0000 mon.c (mon.2) 30 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.467191+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: audit 2024-04-26T16:18:25.467191+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.900135+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-04-26T16:18:26.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:26 smithi050 bash[21463]: cluster 2024-04-26T16:18:25.900135+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2024-04-26T16:18:26.597 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:18:27.173 DEBUG:teuthology.orchestra.run.smithi050:alertmanager.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-26T16:18:27.176 INFO:tasks.cephadm:Adding grafana.a on smithi190 2024-04-26T16:18:27.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi190=a' 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.163712+0000 mgr.y (mgr.24401) 1 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTING 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.163712+0000 mgr.y (mgr.24401) 1 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTING 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.265834+0000 mgr.y (mgr.24401) 2 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.265834+0000 mgr.y (mgr.24401) 2 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cluster 2024-04-26T16:18:26.333562+0000 mon.a (mon.0) 824 : cluster [DBG] mgrmap e20: y(active, since 1.01533s), standbys: x 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cluster 2024-04-26T16:18:26.333562+0000 mon.a (mon.0) 824 : cluster [DBG] mgrmap e20: y(active, since 1.01533s), standbys: x 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: audit 2024-04-26T16:18:26.344894+0000 mgr.y (mgr.24401) 3 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: audit 2024-04-26T16:18:26.344894+0000 mgr.y (mgr.24401) 3 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cluster 2024-04-26T16:18:26.347543+0000 mgr.y (mgr.24401) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cluster 2024-04-26T16:18:26.347543+0000 mgr.y (mgr.24401) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: audit 2024-04-26T16:18:26.348600+0000 mgr.y (mgr.24401) 5 : audit [DBG] from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: audit 2024-04-26T16:18:26.348600+0000 mgr.y (mgr.24401) 5 : audit [DBG] from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.351143+0000 mgr.y (mgr.24401) 6 : cephadm [INF] Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.351143+0000 mgr.y (mgr.24401) 6 : cephadm [INF] Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: audit 2024-04-26T16:18:26.356363+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: audit 2024-04-26T16:18:26.356363+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.392480+0000 mgr.y (mgr.24401) 7 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.392480+0000 mgr.y (mgr.24401) 7 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.392785+0000 mgr.y (mgr.24401) 8 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTED 2024-04-26T16:18:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:27 smithi190 bash[23868]: cephadm 2024-04-26T16:18:26.392785+0000 mgr.y (mgr.24401) 8 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTED 2024-04-26T16:18:27.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.163712+0000 mgr.y (mgr.24401) 1 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTING 2024-04-26T16:18:27.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.163712+0000 mgr.y (mgr.24401) 1 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTING 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.265834+0000 mgr.y (mgr.24401) 2 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.265834+0000 mgr.y (mgr.24401) 2 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cluster 2024-04-26T16:18:26.333562+0000 mon.a (mon.0) 824 : cluster [DBG] mgrmap e20: y(active, since 1.01533s), standbys: x 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cluster 2024-04-26T16:18:26.333562+0000 mon.a (mon.0) 824 : cluster [DBG] mgrmap e20: y(active, since 1.01533s), standbys: x 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: audit 2024-04-26T16:18:26.344894+0000 mgr.y (mgr.24401) 3 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: audit 2024-04-26T16:18:26.344894+0000 mgr.y (mgr.24401) 3 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cluster 2024-04-26T16:18:26.347543+0000 mgr.y (mgr.24401) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cluster 2024-04-26T16:18:26.347543+0000 mgr.y (mgr.24401) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: audit 2024-04-26T16:18:26.348600+0000 mgr.y (mgr.24401) 5 : audit [DBG] from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: audit 2024-04-26T16:18:26.348600+0000 mgr.y (mgr.24401) 5 : audit [DBG] from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.351143+0000 mgr.y (mgr.24401) 6 : cephadm [INF] Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.351143+0000 mgr.y (mgr.24401) 6 : cephadm [INF] Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-26T16:18:27.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: audit 2024-04-26T16:18:26.356363+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: audit 2024-04-26T16:18:26.356363+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.392480+0000 mgr.y (mgr.24401) 7 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.392480+0000 mgr.y (mgr.24401) 7 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.392785+0000 mgr.y (mgr.24401) 8 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTED 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[21463]: cephadm 2024-04-26T16:18:26.392785+0000 mgr.y (mgr.24401) 8 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTED 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.163712+0000 mgr.y (mgr.24401) 1 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTING 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.163712+0000 mgr.y (mgr.24401) 1 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTING 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.265834+0000 mgr.y (mgr.24401) 2 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.265834+0000 mgr.y (mgr.24401) 2 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on http://172.21.15.50:8765 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cluster 2024-04-26T16:18:26.333562+0000 mon.a (mon.0) 824 : cluster [DBG] mgrmap e20: y(active, since 1.01533s), standbys: x 2024-04-26T16:18:27.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cluster 2024-04-26T16:18:26.333562+0000 mon.a (mon.0) 824 : cluster [DBG] mgrmap e20: y(active, since 1.01533s), standbys: x 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: audit 2024-04-26T16:18:26.344894+0000 mgr.y (mgr.24401) 3 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: audit 2024-04-26T16:18:26.344894+0000 mgr.y (mgr.24401) 3 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cluster 2024-04-26T16:18:26.347543+0000 mgr.y (mgr.24401) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cluster 2024-04-26T16:18:26.347543+0000 mgr.y (mgr.24401) 4 : cluster [DBG] pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: audit 2024-04-26T16:18:26.348600+0000 mgr.y (mgr.24401) 5 : audit [DBG] from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: audit 2024-04-26T16:18:26.348600+0000 mgr.y (mgr.24401) 5 : audit [DBG] from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi050=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.351143+0000 mgr.y (mgr.24401) 6 : cephadm [INF] Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.351143+0000 mgr.y (mgr.24401) 6 : cephadm [INF] Saving service alertmanager spec with placement smithi050=a;count:1 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: audit 2024-04-26T16:18:26.356363+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: audit 2024-04-26T16:18:26.356363+0000 mon.a (mon.0) 825 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.392480+0000 mgr.y (mgr.24401) 7 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.392480+0000 mgr.y (mgr.24401) 7 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Serving on https://172.21.15.50:7150 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.392785+0000 mgr.y (mgr.24401) 8 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTED 2024-04-26T16:18:27.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:27 smithi050 bash[29234]: cephadm 2024-04-26T16:18:26.392785+0000 mgr.y (mgr.24401) 8 : cephadm [INF] [26/Apr/2024:16:18:26] ENGINE Bus STARTED 2024-04-26T16:18:28.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:28 smithi050 bash[29234]: cluster 2024-04-26T16:18:27.340898+0000 mgr.y (mgr.24401) 9 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:28.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:28 smithi050 bash[29234]: cluster 2024-04-26T16:18:27.340898+0000 mgr.y (mgr.24401) 9 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:28 smithi050 bash[21463]: cluster 2024-04-26T16:18:27.340898+0000 mgr.y (mgr.24401) 9 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:28 smithi050 bash[21463]: cluster 2024-04-26T16:18:27.340898+0000 mgr.y (mgr.24401) 9 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:28.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:28 smithi190 bash[23868]: cluster 2024-04-26T16:18:27.340898+0000 mgr.y (mgr.24401) 9 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:28.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:28 smithi190 bash[23868]: cluster 2024-04-26T16:18:27.340898+0000 mgr.y (mgr.24401) 9 : cluster [DBG] pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:29.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:29 smithi050 bash[21463]: cluster 2024-04-26T16:18:28.360676+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e21: y(active, since 3s), standbys: x 2024-04-26T16:18:29.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:29 smithi050 bash[21463]: cluster 2024-04-26T16:18:28.360676+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e21: y(active, since 3s), standbys: x 2024-04-26T16:18:29.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:29 smithi050 bash[29234]: cluster 2024-04-26T16:18:28.360676+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e21: y(active, since 3s), standbys: x 2024-04-26T16:18:29.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:29 smithi050 bash[29234]: cluster 2024-04-26T16:18:28.360676+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e21: y(active, since 3s), standbys: x 2024-04-26T16:18:29.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:29 smithi190 bash[23868]: cluster 2024-04-26T16:18:28.360676+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e21: y(active, since 3s), standbys: x 2024-04-26T16:18:29.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:29 smithi190 bash[23868]: cluster 2024-04-26T16:18:28.360676+0000 mon.a (mon.0) 826 : cluster [DBG] mgrmap e21: y(active, since 3s), standbys: x 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[21463]: cluster 2024-04-26T16:18:29.341671+0000 mgr.y (mgr.24401) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[21463]: cluster 2024-04-26T16:18:29.341671+0000 mgr.y (mgr.24401) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[21463]: cluster 2024-04-26T16:18:29.671745+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[21463]: cluster 2024-04-26T16:18:29.671745+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[21463]: cluster 2024-04-26T16:18:29.674501+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[21463]: cluster 2024-04-26T16:18:29.674501+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[29234]: cluster 2024-04-26T16:18:29.341671+0000 mgr.y (mgr.24401) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:30.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[29234]: cluster 2024-04-26T16:18:29.341671+0000 mgr.y (mgr.24401) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:30.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[29234]: cluster 2024-04-26T16:18:29.671745+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:18:30.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[29234]: cluster 2024-04-26T16:18:29.671745+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:18:30.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[29234]: cluster 2024-04-26T16:18:29.674501+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:18:30.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:30 smithi050 bash[29234]: cluster 2024-04-26T16:18:29.674501+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:18:30.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:30 smithi190 bash[23868]: cluster 2024-04-26T16:18:29.341671+0000 mgr.y (mgr.24401) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:30.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:30 smithi190 bash[23868]: cluster 2024-04-26T16:18:29.341671+0000 mgr.y (mgr.24401) 10 : cluster [DBG] pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:18:30.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:30 smithi190 bash[23868]: cluster 2024-04-26T16:18:29.671745+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:18:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:30 smithi190 bash[23868]: cluster 2024-04-26T16:18:29.671745+0000 osd.7 (osd.7) 7 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:18:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:30 smithi190 bash[23868]: cluster 2024-04-26T16:18:29.674501+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:18:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:30 smithi190 bash[23868]: cluster 2024-04-26T16:18:29.674501+0000 osd.7 (osd.7) 8 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:18:31.127 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:18:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:31 smithi190 bash[23868]: cluster 2024-04-26T16:18:30.380220+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e22: y(active, since 5s), standbys: x 2024-04-26T16:18:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:31 smithi190 bash[23868]: cluster 2024-04-26T16:18:30.380220+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e22: y(active, since 5s), standbys: x 2024-04-26T16:18:31.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:31 smithi050 bash[21463]: cluster 2024-04-26T16:18:30.380220+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e22: y(active, since 5s), standbys: x 2024-04-26T16:18:31.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:31 smithi050 bash[21463]: cluster 2024-04-26T16:18:30.380220+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e22: y(active, since 5s), standbys: x 2024-04-26T16:18:31.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:31 smithi050 bash[29234]: cluster 2024-04-26T16:18:30.380220+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e22: y(active, since 5s), standbys: x 2024-04-26T16:18:31.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:31 smithi050 bash[29234]: cluster 2024-04-26T16:18:30.380220+0000 mon.a (mon.0) 827 : cluster [DBG] mgrmap e22: y(active, since 5s), standbys: x 2024-04-26T16:18:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: cluster 2024-04-26T16:18:31.342246+0000 mgr.y (mgr.24401) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: cluster 2024-04-26T16:18:31.342246+0000 mgr.y (mgr.24401) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.495513+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.495513+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.501613+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.501613+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.878213+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.878213+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.891014+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:32 smithi190 bash[23868]: audit 2024-04-26T16:18:31.891014+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: cluster 2024-04-26T16:18:31.342246+0000 mgr.y (mgr.24401) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: cluster 2024-04-26T16:18:31.342246+0000 mgr.y (mgr.24401) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.495513+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.495513+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.501613+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.501613+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.878213+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.878213+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.891014+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[21463]: audit 2024-04-26T16:18:31.891014+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: cluster 2024-04-26T16:18:31.342246+0000 mgr.y (mgr.24401) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: cluster 2024-04-26T16:18:31.342246+0000 mgr.y (mgr.24401) 11 : cluster [DBG] pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.495513+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.495513+0000 mon.a (mon.0) 828 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.501613+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.501613+0000 mon.a (mon.0) 829 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.878213+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.878213+0000 mon.a (mon.0) 830 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.891014+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:32.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:32 smithi050 bash[29234]: audit 2024-04-26T16:18:31.891014+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/4249565894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 msgr2=0x7f5d4810ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:33.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- 172.21.15.190:0/4249565894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4810ba20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5d38009990 tx=0x7f5d38009ca0 comp rx=0 tx=0).stop 2024-04-26T16:18:33.850 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/4249565894 shutdown_connections 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- 172.21.15.190:0/4249565894 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4810ba20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- 172.21.15.190:0/4249565894 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5d48106000 0x7f5d48112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- 172.21.15.190:0/4249565894 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d48105640 0x7f5d48105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/4249565894 >> 172.21.15.190:0/4249565894 conn(0x7f5d48101130 msgr2=0x7f5d48103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/4249565894 shutdown_connections 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/4249565894 wait complete. 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 librados: starting msgr at 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 librados: starting objecter 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 Processor -- start 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- start start 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 librados: setting wanted keys 2024-04-26T16:18:33.851 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 librados: calling monclient init 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5d48105640 0x7f5d4819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d48106000 0x7f5d4819b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4819fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48114db0 con 0x7f5d48106000 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48114c30 con 0x7f5d4810b640 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5d48114f30 con 0x7f5d48105640 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4819fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d3f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d48106000 0x7f5d4819b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4819fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:41470/0 (socket says 172.21.15.190:41470) 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 -- 172.21.15.190:0/2701702909 learned_addr learned my addr 172.21.15.190:0/2701702909 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d47fff700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5d48105640 0x7f5d4819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 -- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5d48105640 msgr2=0x7f5d4819b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:33.852 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5d48105640 0x7f5d4819b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:33.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 -- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d48106000 msgr2=0x7f5d4819b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:33.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d48106000 0x7f5d4819b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:33.853 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 -- 172.21.15.190:0/2701702909 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d481a0340 con 0x7f5d4810b640 2024-04-26T16:18:33.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4cbe5700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4819fc00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d40007ee0 tx=0x7f5d4000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:33.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d45ffb700 10 client.?.objecter ms_handle_connect 0x7f5d4810b640 2024-04-26T16:18:33.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d45ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:18:33.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40004bb0 con 0x7f5d4810b640 2024-04-26T16:18:33.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5d40004d50 con 0x7f5d4810b640 2024-04-26T16:18:33.854 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d481a0630 con 0x7f5d4810b640 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 10 client.24454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40005720 con 0x7f5d4810b640 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d481a7eb0 con 0x7f5d4810b640 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 librados: init done 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d4810dbd0 con 0x7f5d4810b640 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.856+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5d4001a020 con 0x7f5d4810b640 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5d34071f80 0x7f5d34074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(101..101 src has 22..101) ==== 7450+0+0 (secure 0 0 0) 0x7f5d40090920 con 0x7f5d4810b640 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 10 client.24454.objecter ms_dispatch 0x7f5d48001550 osd_map(101..101 src has 22..101) 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 3 client.24454.objecter handle_osd_map got epochs [101,101] > 0 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 3 client.24454.objecter handle_osd_map decoding full epoch 101 2024-04-26T16:18:33.855 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 20 client.24454.objecter dump_active .. 0 homeless 2024-04-26T16:18:33.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d47fff700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5d34071f80 0x7f5d34074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:33.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d45ffb700 10 client.24454.objecter ms_handle_connect 0x7f5d34071f80 2024-04-26T16:18:33.856 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.860+0000 7f5d47fff700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5d34071f80 0x7f5d34074440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005850 tx=0x7f5d38005fd0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:33.858 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:33.864+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5d40010070 con 0x7f5d4810b640 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.610385+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.610385+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.621529+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.621529+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.624447+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.624447+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.624920+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.624920+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.626383+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.626383+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.626901+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.626901+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cluster 2024-04-26T16:18:32.627919+0000 osd.7 (osd.7) 9 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cluster 2024-04-26T16:18:32.627919+0000 osd.7 (osd.7) 9 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.628421+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.628421+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.628918+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.628918+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cluster 2024-04-26T16:18:32.630084+0000 osd.7 (osd.7) 10 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cluster 2024-04-26T16:18:32.630084+0000 osd.7 (osd.7) 10 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.610385+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.610385+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.621529+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.621529+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.624447+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.624447+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.624920+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.624920+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.626383+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.626383+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.626901+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.626901+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cluster 2024-04-26T16:18:32.627919+0000 osd.7 (osd.7) 9 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cluster 2024-04-26T16:18:32.627919+0000 osd.7 (osd.7) 9 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.628421+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.628421+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.628918+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.628918+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cluster 2024-04-26T16:18:32.630084+0000 osd.7 (osd.7) 10 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cluster 2024-04-26T16:18:32.630084+0000 osd.7 (osd.7) 10 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.630378+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.630378+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.630898+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.630898+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:32.632092+0000 mgr.y (mgr.24401) 12 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3805M 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:32.632092+0000 mgr.y (mgr.24401) 12 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3805M 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.638831+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.638831+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.947804+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.947804+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.961894+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.961894+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.966746+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.966746+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.967570+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.967570+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.969812+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.969812+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.970526+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.970526+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.630378+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.630378+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.630898+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.630898+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:32.632092+0000 mgr.y (mgr.24401) 12 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3805M 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:32.632092+0000 mgr.y (mgr.24401) 12 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3805M 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.638831+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.638831+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.947804+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.947804+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.961894+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.961894+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.966746+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.966746+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.967570+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.967570+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.969812+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.969812+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.970526+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.970526+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.972667+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.972667+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.973327+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.973327+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.975681+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.975681+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.976416+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.976416+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:32.978261+0000 mgr.y (mgr.24401) 13 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3804M 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:32.978261+0000 mgr.y (mgr.24401) 13 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3804M 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.988004+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.988004+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.994579+0000 mon.c (mon.2) 39 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.994579+0000 mon.c (mon.2) 39 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.996636+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: audit 2024-04-26T16:18:32.996636+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:32.999500+0000 mgr.y (mgr.24401) 14 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:32.999500+0000 mgr.y (mgr.24401) 14 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.000707+0000 mgr.y (mgr.24401) 15 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.000707+0000 mgr.y (mgr.24401) 15 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.267276+0000 mgr.y (mgr.24401) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.972667+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.972667+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.973327+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.973327+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.975681+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.975681+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.976416+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.976416+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:32.978261+0000 mgr.y (mgr.24401) 13 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3804M 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:32.978261+0000 mgr.y (mgr.24401) 13 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3804M 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.988004+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.988004+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.994579+0000 mon.c (mon.2) 39 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.994579+0000 mon.c (mon.2) 39 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.996636+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: audit 2024-04-26T16:18:32.996636+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:32.999500+0000 mgr.y (mgr.24401) 14 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:32.999500+0000 mgr.y (mgr.24401) 14 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.000707+0000 mgr.y (mgr.24401) 15 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.000707+0000 mgr.y (mgr.24401) 15 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:18:33.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.267276+0000 mgr.y (mgr.24401) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.267276+0000 mgr.y (mgr.24401) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.269764+0000 mgr.y (mgr.24401) 17 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.269764+0000 mgr.y (mgr.24401) 17 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.343021+0000 mgr.y (mgr.24401) 18 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.343021+0000 mgr.y (mgr.24401) 18 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.431851+0000 mgr.y (mgr.24401) 19 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.431851+0000 mgr.y (mgr.24401) 19 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.435663+0000 mgr.y (mgr.24401) 20 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.435663+0000 mgr.y (mgr.24401) 20 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.267276+0000 mgr.y (mgr.24401) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.269764+0000 mgr.y (mgr.24401) 17 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.269764+0000 mgr.y (mgr.24401) 17 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.343021+0000 mgr.y (mgr.24401) 18 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.343021+0000 mgr.y (mgr.24401) 18 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.431851+0000 mgr.y (mgr.24401) 19 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.431851+0000 mgr.y (mgr.24401) 19 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.435663+0000 mgr.y (mgr.24401) 20 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:33.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:33 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.435663+0000 mgr.y (mgr.24401) 20 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.610385+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.610385+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.621529+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.621529+0000 mon.a (mon.0) 833 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.624447+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.624447+0000 mon.c (mon.2) 31 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.624920+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.624920+0000 mon.a (mon.0) 834 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.626383+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.626383+0000 mon.c (mon.2) 32 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.626901+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.626901+0000 mon.a (mon.0) 835 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cluster 2024-04-26T16:18:32.627919+0000 osd.7 (osd.7) 9 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cluster 2024-04-26T16:18:32.627919+0000 osd.7 (osd.7) 9 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.628421+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.628421+0000 mon.c (mon.2) 33 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.628918+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.628918+0000 mon.a (mon.0) 836 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cluster 2024-04-26T16:18:32.630084+0000 osd.7 (osd.7) 10 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cluster 2024-04-26T16:18:32.630084+0000 osd.7 (osd.7) 10 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.630378+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.630378+0000 mon.c (mon.2) 34 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.630898+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.630898+0000 mon.a (mon.0) 837 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:32.632092+0000 mgr.y (mgr.24401) 12 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3805M 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:32.632092+0000 mgr.y (mgr.24401) 12 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3805M 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.638831+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.638831+0000 mon.a (mon.0) 838 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.947804+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.947804+0000 mon.a (mon.0) 839 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.961894+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.961894+0000 mon.a (mon.0) 840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.966746+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.966746+0000 mon.c (mon.2) 35 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.967570+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.967570+0000 mon.a (mon.0) 841 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.969812+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.969812+0000 mon.c (mon.2) 36 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.970526+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.970526+0000 mon.a (mon.0) 842 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.972667+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.972667+0000 mon.c (mon.2) 37 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.973327+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.973327+0000 mon.a (mon.0) 843 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.975681+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.975681+0000 mon.c (mon.2) 38 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.976416+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.976416+0000 mon.a (mon.0) 844 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:32.978261+0000 mgr.y (mgr.24401) 13 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3804M 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:32.978261+0000 mgr.y (mgr.24401) 13 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3804M 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.988004+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.988004+0000 mon.a (mon.0) 845 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.994579+0000 mon.c (mon.2) 39 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.994579+0000 mon.c (mon.2) 39 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.996636+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: audit 2024-04-26T16:18:32.996636+0000 mon.c (mon.2) 40 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:32.999500+0000 mgr.y (mgr.24401) 14 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:32.999500+0000 mgr.y (mgr.24401) 14 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.conf 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.000707+0000 mgr.y (mgr.24401) 15 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:18:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.000707+0000 mgr.y (mgr.24401) 15 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.conf 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.267276+0000 mgr.y (mgr.24401) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.267276+0000 mgr.y (mgr.24401) 16 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.269764+0000 mgr.y (mgr.24401) 17 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.269764+0000 mgr.y (mgr.24401) 17 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.343021+0000 mgr.y (mgr.24401) 18 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.343021+0000 mgr.y (mgr.24401) 18 : cluster [DBG] pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.431851+0000 mgr.y (mgr.24401) 19 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.431851+0000 mgr.y (mgr.24401) 19 : cephadm [INF] Updating smithi050:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.435663+0000 mgr.y (mgr.24401) 20 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:33 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.435663+0000 mgr.y (mgr.24401) 20 : cephadm [INF] Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-04-26T16:18:34.093 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.096+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}) -- 0x7f5d4806a0f0 con 0x7f5d34071f80 2024-04-26T16:18:34.111 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.116+0000 7f5d45ffb700 1 -- 172.21.15.190:0/2701702909 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f5d4806a0f0 con 0x7f5d34071f80 2024-04-26T16:18:34.111 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled grafana update... 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 10 librados: watch_flush enter 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 10 librados: watch_flush exit 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 20 client.24454.objecter shutdown clearing up homeless session... 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 10 client.24454.objecter successfully canceled tick 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5d34071f80 msgr2=0x7f5d34074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5d34071f80 0x7f5d34074440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d38005850 tx=0x7f5d38005fd0 comp rx=0 tx=0).stop 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 msgr2=0x7f5d4819fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4819fc00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d40007ee0 tx=0x7f5d4000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:18:34.114 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 shutdown_connections 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5d34071f80 0x7f5d34074440 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5d4810b640 0x7f5d4819fc00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5d48105640 0x7f5d4819b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 --2- 172.21.15.190:0/2701702909 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5d48106000 0x7f5d4819b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 >> 172.21.15.190:0/2701702909 conn(0x7f5d48101130 msgr2=0x7f5d48101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 shutdown_connections 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 -- 172.21.15.190:0/2701702909 wait complete. 2024-04-26T16:18:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:34.120+0000 7f5d4e648700 1 librados: shutdown 2024-04-26T16:18:34.517 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:18:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:18:34.518 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.518 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.518 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.519 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.519 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.519 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.519 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.799 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.799 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.799 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.800 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.800 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.800 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: Started Ceph node-exporter.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:18:34.801 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.596781+0000 mgr.y (mgr.24401) 21 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.596781+0000 mgr.y (mgr.24401) 21 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.600323+0000 mgr.y (mgr.24401) 22 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.600323+0000 mgr.y (mgr.24401) 22 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.638463+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.638463+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.640877+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.640877+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.768727+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.768727+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.775444+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.775444+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.779976+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.779976+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.788358+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.788358+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.796467+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.802 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:33.796467+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.801748+0000 mgr.y (mgr.24401) 23 : cephadm [INF] Deploying daemon node-exporter.a on smithi050 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:33.801748+0000 mgr.y (mgr.24401) 23 : cephadm [INF] Deploying daemon node-exporter.a on smithi050 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.887935+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.887935+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.890088+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cluster 2024-04-26T16:18:33.890088+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:34.106289+0000 mgr.y (mgr.24401) 24 : audit [DBG] from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:34.106289+0000 mgr.y (mgr.24401) 24 : audit [DBG] from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:34.111067+0000 mgr.y (mgr.24401) 25 : cephadm [INF] Saving service grafana spec with placement smithi190=a;count:1 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: cephadm 2024-04-26T16:18:34.111067+0000 mgr.y (mgr.24401) 25 : cephadm [INF] Saving service grafana spec with placement smithi190=a;count:1 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:34.120767+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.803 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[29234]: audit 2024-04-26T16:18:34.120767+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.596781+0000 mgr.y (mgr.24401) 21 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.596781+0000 mgr.y (mgr.24401) 21 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.600323+0000 mgr.y (mgr.24401) 22 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.600323+0000 mgr.y (mgr.24401) 22 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.638463+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.638463+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.640877+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.640877+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.768727+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.768727+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.775444+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.775444+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.779976+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.779976+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.788358+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.788358+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.796467+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:33.796467+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.801748+0000 mgr.y (mgr.24401) 23 : cephadm [INF] Deploying daemon node-exporter.a on smithi050 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:33.801748+0000 mgr.y (mgr.24401) 23 : cephadm [INF] Deploying daemon node-exporter.a on smithi050 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.887935+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.887935+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.890088+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:18:34.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cluster 2024-04-26T16:18:33.890088+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:18:34.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:34.106289+0000 mgr.y (mgr.24401) 24 : audit [DBG] from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:34.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:34.106289+0000 mgr.y (mgr.24401) 24 : audit [DBG] from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:34.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:34.111067+0000 mgr.y (mgr.24401) 25 : cephadm [INF] Saving service grafana spec with placement smithi190=a;count:1 2024-04-26T16:18:34.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: cephadm 2024-04-26T16:18:34.111067+0000 mgr.y (mgr.24401) 25 : cephadm [INF] Saving service grafana spec with placement smithi190=a;count:1 2024-04-26T16:18:34.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:34.120767+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[21463]: audit 2024-04-26T16:18:34.120767+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:34.807 DEBUG:teuthology.orchestra.run.smithi190:grafana.a> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@grafana.a.service 2024-04-26T16:18:34.809 INFO:tasks.cephadm:Setting up client nodes... 2024-04-26T16:18:34.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T16:18:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.596781+0000 mgr.y (mgr.24401) 21 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.596781+0000 mgr.y (mgr.24401) 21 : cephadm [INF] Updating smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.600323+0000 mgr.y (mgr.24401) 22 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.600323+0000 mgr.y (mgr.24401) 22 : cephadm [INF] Updating smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-26T16:18:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.638463+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:18:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.638463+0000 osd.7 (osd.7) 11 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.640877+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.640877+0000 osd.7 (osd.7) 12 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.768727+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.768727+0000 mon.a (mon.0) 846 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.775444+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.775444+0000 mon.a (mon.0) 847 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.779976+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.779976+0000 mon.a (mon.0) 848 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.788358+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.788358+0000 mon.a (mon.0) 849 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.796467+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:33.796467+0000 mon.a (mon.0) 850 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.801748+0000 mgr.y (mgr.24401) 23 : cephadm [INF] Deploying daemon node-exporter.a on smithi050 2024-04-26T16:18:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:33.801748+0000 mgr.y (mgr.24401) 23 : cephadm [INF] Deploying daemon node-exporter.a on smithi050 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.887935+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.887935+0000 osd.0 (osd.0) 15 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.890088+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cluster 2024-04-26T16:18:33.890088+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:34.106289+0000 mgr.y (mgr.24401) 24 : audit [DBG] from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:34.106289+0000 mgr.y (mgr.24401) 24 : audit [DBG] from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:34.111067+0000 mgr.y (mgr.24401) 25 : cephadm [INF] Saving service grafana spec with placement smithi190=a;count:1 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: cephadm 2024-04-26T16:18:34.111067+0000 mgr.y (mgr.24401) 25 : cephadm [INF] Saving service grafana spec with placement smithi190=a;count:1 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:34.120767+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:34 smithi190 bash[23868]: audit 2024-04-26T16:18:34.120767+0000 mon.a (mon.0) 851 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:35.123 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:34 smithi050 bash[57836]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-04-26T16:18:35.802 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.803 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.803 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.803 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.804 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.804 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:35.805 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[57836]: v1.5.0: Pulling from prometheus/node-exporter 2024-04-26T16:18:35.805 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[57836]: 22b70bddd3ac: Pulling fs layer 2024-04-26T16:18:35.805 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[57836]: 5c12815fee55: Pulling fs layer 2024-04-26T16:18:35.805 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[57836]: c0e87333d380: Pulling fs layer 2024-04-26T16:18:36.057 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[57836]: 22b70bddd3ac: Verifying Checksum 2024-04-26T16:18:36.057 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[57836]: 22b70bddd3ac: Download complete 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: audit 2024-04-26T16:18:34.813646+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: audit 2024-04-26T16:18:34.813646+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: audit 2024-04-26T16:18:34.821339+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: audit 2024-04-26T16:18:34.821339+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: audit 2024-04-26T16:18:34.873108+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: audit 2024-04-26T16:18:34.873108+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cephadm 2024-04-26T16:18:34.874335+0000 mgr.y (mgr.24401) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi190 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cephadm 2024-04-26T16:18:34.874335+0000 mgr.y (mgr.24401) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi190 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.008930+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:18:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.008930+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.011477+0000 osd.1 (osd.1) 18 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.011477+0000 osd.1 (osd.1) 18 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.343717+0000 mgr.y (mgr.24401) 27 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.343717+0000 mgr.y (mgr.24401) 27 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: audit 2024-04-26T16:18:34.813646+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: audit 2024-04-26T16:18:34.813646+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: audit 2024-04-26T16:18:34.821339+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: audit 2024-04-26T16:18:34.821339+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: audit 2024-04-26T16:18:34.873108+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: audit 2024-04-26T16:18:34.873108+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cephadm 2024-04-26T16:18:34.874335+0000 mgr.y (mgr.24401) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi190 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cephadm 2024-04-26T16:18:34.874335+0000 mgr.y (mgr.24401) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi190 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.008930+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.008930+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.011477+0000 osd.1 (osd.1) 18 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.011477+0000 osd.1 (osd.1) 18 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.343717+0000 mgr.y (mgr.24401) 27 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-26T16:18:36.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:35 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.343717+0000 mgr.y (mgr.24401) 27 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-26T16:18:36.071 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.071 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.072 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.072 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.072 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.072 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.073 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: Started Ceph node-exporter.b for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:18:36.073 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[49714]: Unable to find image 'quay.io/prometheus/node-exporter:v1.5.0' locally 2024-04-26T16:18:36.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: audit 2024-04-26T16:18:34.813646+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: audit 2024-04-26T16:18:34.813646+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: audit 2024-04-26T16:18:34.821339+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: audit 2024-04-26T16:18:34.821339+0000 mon.a (mon.0) 853 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: audit 2024-04-26T16:18:34.873108+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: audit 2024-04-26T16:18:34.873108+0000 mon.a (mon.0) 854 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cephadm 2024-04-26T16:18:34.874335+0000 mgr.y (mgr.24401) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi190 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cephadm 2024-04-26T16:18:34.874335+0000 mgr.y (mgr.24401) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi190 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.008930+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.008930+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.011477+0000 osd.1 (osd.1) 18 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.011477+0000 osd.1 (osd.1) 18 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.343717+0000 mgr.y (mgr.24401) 27 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.343717+0000 mgr.y (mgr.24401) 27 : cluster [DBG] pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-26T16:18:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:35 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:36.373 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[57836]: c0e87333d380: Verifying Checksum 2024-04-26T16:18:36.373 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[57836]: c0e87333d380: Download complete 2024-04-26T16:18:36.373 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[57836]: 22b70bddd3ac: Pull complete 2024-04-26T16:18:36.758 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:18:36.759 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: v1.5.0: Pulling from prometheus/node-exporter 2024-04-26T16:18:36.873 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[57836]: 5c12815fee55: Pull complete 2024-04-26T16:18:36.873 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:36 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:37.070 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: 22b70bddd3ac: Pulling fs layer 2024-04-26T16:18:37.070 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: 5c12815fee55: Pulling fs layer 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: c0e87333d380: Pulling fs layer 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: 22b70bddd3ac: Verifying Checksum 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: 22b70bddd3ac: Download complete 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: 5c12815fee55: Verifying Checksum 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[49714]: 5c12815fee55: Download complete 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[49714]: c0e87333d380: Verifying Checksum 2024-04-26T16:18:37.071 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[49714]: c0e87333d380: Download complete 2024-04-26T16:18:37.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.873701+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:18:37.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.873701+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:18:37.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.877098+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.877098+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.917360+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.917360+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.922667+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:35.922667+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:35.987547+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:35.987547+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:35.996916+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:35.996916+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.016247+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.016247+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.018804+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.018804+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:18:37.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.046057+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.046057+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.051749+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.051749+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.061517+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.061517+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cephadm 2024-04-26T16:18:36.077610+0000 mgr.y (mgr.24401) 28 : cephadm [INF] Deploying daemon alertmanager.a on smithi050 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cephadm 2024-04-26T16:18:36.077610+0000 mgr.y (mgr.24401) 28 : cephadm [INF] Deploying daemon alertmanager.a on smithi050 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.352541+0000 mgr.y (mgr.24401) 29 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: audit 2024-04-26T16:18:36.352541+0000 mgr.y (mgr.24401) 29 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.711685+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.711685+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.713945+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:18:37.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:36 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.713945+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:18:37.321 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[49714]: 22b70bddd3ac: Pull complete 2024-04-26T16:18:37.373 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[57836]: c0e87333d380: Pull complete 2024-04-26T16:18:37.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.873701+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:18:37.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.873701+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:18:37.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.877098+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:18:37.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.877098+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:18:37.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.917360+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.917360+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.922667+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:35.922667+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:35.987547+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:35.987547+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:35.996916+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:35.996916+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.016247+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.016247+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.018804+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.018804+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.046057+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.046057+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.051749+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.051749+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.061517+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.061517+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cephadm 2024-04-26T16:18:36.077610+0000 mgr.y (mgr.24401) 28 : cephadm [INF] Deploying daemon alertmanager.a on smithi050 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cephadm 2024-04-26T16:18:36.077610+0000 mgr.y (mgr.24401) 28 : cephadm [INF] Deploying daemon alertmanager.a on smithi050 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.352541+0000 mgr.y (mgr.24401) 29 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: audit 2024-04-26T16:18:36.352541+0000 mgr.y (mgr.24401) 29 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.711685+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.711685+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.713945+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:18:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.713945+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.873701+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.873701+0000 osd.4 (osd.4) 9 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.877098+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.877098+0000 osd.4 (osd.4) 10 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.917360+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.917360+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.922667+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:35.922667+0000 osd.0 (osd.0) 18 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:35.987547+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:35.987547+0000 mon.a (mon.0) 855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:35.996916+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:35.996916+0000 mon.a (mon.0) 856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.016247+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.016247+0000 osd.3 (osd.3) 11 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.018804+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.018804+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.046057+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.046057+0000 mon.a (mon.0) 857 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.051749+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.051749+0000 mon.a (mon.0) 858 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.061517+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.061517+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cephadm 2024-04-26T16:18:36.077610+0000 mgr.y (mgr.24401) 28 : cephadm [INF] Deploying daemon alertmanager.a on smithi050 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cephadm 2024-04-26T16:18:36.077610+0000 mgr.y (mgr.24401) 28 : cephadm [INF] Deploying daemon alertmanager.a on smithi050 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.352541+0000 mgr.y (mgr.24401) 29 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: audit 2024-04-26T16:18:36.352541+0000 mgr.y (mgr.24401) 29 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:37.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.711685+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:18:37.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.711685+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:18:37.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.713945+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:18:37.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:36 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.713945+0000 osd.7 (osd.7) 14 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:18:37.821 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[49714]: 5c12815fee55: Pull complete 2024-04-26T16:18:37.873 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[57836]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-04-26T16:18:37.873 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[57836]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-04-26T16:18:38.321 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[49714]: c0e87333d380: Pull complete 2024-04-26T16:18:38.321 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:38 smithi190 bash[49714]: Digest: sha256:39c642b2b337e38c18e80266fb14383754178202f40103646337722a594d984c 2024-04-26T16:18:38.321 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:38 smithi190 bash[49714]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.5.0 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.577733+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.577733+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.581468+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.581468+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.977767+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.977767+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.979459+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:36.979459+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:37.345211+0000 mgr.y (mgr.24401) 30 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-26T16:18:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:37 smithi190 bash[23868]: cluster 2024-04-26T16:18:37.345211+0000 mgr.y (mgr.24401) 30 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.577733+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.577733+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.581468+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.581468+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.977767+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.977767+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.979459+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:36.979459+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:37.345211+0000 mgr.y (mgr.24401) 30 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-26T16:18:38.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[21463]: cluster 2024-04-26T16:18:37.345211+0000 mgr.y (mgr.24401) 30 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.577733+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.577733+0000 osd.5 (osd.5) 9 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.581468+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.581468+0000 osd.5 (osd.5) 10 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.977767+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.977767+0000 osd.3 (osd.3) 13 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.979459+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:36.979459+0000 osd.3 (osd.3) 14 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:37.345211+0000 mgr.y (mgr.24401) 30 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-26T16:18:38.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:37 smithi050 bash[29234]: cluster 2024-04-26T16:18:37.345211+0000 mgr.y (mgr.24401) 30 : cluster [DBG] pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-26T16:18:39.574 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.415Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.416Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.416Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.416Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.416Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-26T16:18:39.822 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=arp 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=edac 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-26T16:18:39.823 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-26T16:18:39.824 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=os 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-26T16:18:39.825 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.417Z caller=node_exporter.go:117 level=info collector=stat 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=time 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=uname 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-26T16:18:39.826 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:18:39 smithi190 bash[49714]: ts=2024-04-26T16:18:39.418Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.572654+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.572654+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.574972+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.574972+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.908889+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.908889+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.915192+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.915192+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:18:40.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.974140+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.974140+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.974219+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:38.974219+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.001829+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.001829+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.115428+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.115428+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.117149+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.117149+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.345939+0000 mgr.y (mgr.24401) 31 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-26T16:18:40.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.345939+0000 mgr.y (mgr.24401) 31 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-26T16:18:40.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.572654+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.572654+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.574972+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.574972+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.908889+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.908889+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.915192+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:18:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.915192+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.974140+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.974140+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.974219+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:38.974219+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.001829+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.001829+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.115428+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.115428+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.117149+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.117149+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.345939+0000 mgr.y (mgr.24401) 31 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-26T16:18:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:40 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.345939+0000 mgr.y (mgr.24401) 31 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.572654+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.572654+0000 osd.5 (osd.5) 11 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.574972+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.574972+0000 osd.5 (osd.5) 12 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.908889+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.908889+0000 osd.3 (osd.3) 15 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.915192+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.915192+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:18:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.974140+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.974140+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.974219+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:38.974219+0000 osd.6 (osd.6) 6 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.001829+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.001829+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.115428+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.115428+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.117149+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.117149+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.345939+0000 mgr.y (mgr.24401) 31 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-26T16:18:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:40 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.345939+0000 mgr.y (mgr.24401) 31 : cluster [DBG] pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-26T16:18:41.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.560137+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.560137+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.561523+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: cluster 2024-04-26T16:18:39.561523+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: audit 2024-04-26T16:18:40.445669+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: audit 2024-04-26T16:18:40.445669+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: audit 2024-04-26T16:18:40.462015+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: audit 2024-04-26T16:18:40.462015+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.906431+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-04-26T16:18:41.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:41 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.906431+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.560137+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.560137+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.561523+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: cluster 2024-04-26T16:18:39.561523+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: audit 2024-04-26T16:18:40.445669+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: audit 2024-04-26T16:18:40.445669+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: audit 2024-04-26T16:18:40.462015+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: audit 2024-04-26T16:18:40.462015+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.906431+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-04-26T16:18:41.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.906431+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.560137+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.560137+0000 osd.2 (osd.2) 11 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.561523+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: cluster 2024-04-26T16:18:39.561523+0000 osd.2 (osd.2) 12 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: audit 2024-04-26T16:18:40.445669+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: audit 2024-04-26T16:18:40.445669+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: audit 2024-04-26T16:18:40.462015+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: audit 2024-04-26T16:18:40.462015+0000 mon.c (mon.2) 41 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.906431+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-04-26T16:18:41.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.906431+0000 mon.a (mon.0) 862 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.018Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.018Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.018Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.018Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.018Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=arp 2024-04-26T16:18:41.376 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=bcache 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=bonding 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=btrfs 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=conntrack 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=cpu 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=diskstats 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=dmi 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=edac 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=entropy 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.019Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=filefd 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=filesystem 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=hwmon 2024-04-26T16:18:41.377 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=infiniband 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=ipvs 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=loadavg 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=mdadm 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=meminfo 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=netclass 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=netdev 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=netstat 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=nfs 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=nfsd 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=nvme 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=os 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=pressure 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=rapl 2024-04-26T16:18:41.378 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=schedstat 2024-04-26T16:18:41.379 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=selinux 2024-04-26T16:18:41.379 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=sockstat 2024-04-26T16:18:41.379 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=softnet 2024-04-26T16:18:41.379 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=stat 2024-04-26T16:18:41.382 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=tapestats 2024-04-26T16:18:41.382 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=textfile 2024-04-26T16:18:41.382 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-04-26T16:18:41.382 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=time 2024-04-26T16:18:41.382 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-04-26T16:18:41.382 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=uname 2024-04-26T16:18:41.383 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=vmstat 2024-04-26T16:18:41.383 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=xfs 2024-04-26T16:18:41.383 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=node_exporter.go:117 level=info collector=zfs 2024-04-26T16:18:41.383 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-04-26T16:18:41.383 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:41 smithi050 bash[57836]: ts=2024-04-26T16:18:41.020Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-04-26T16:18:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.659694+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.2 deep-scrub starts 2024-04-26T16:18:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.659694+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.2 deep-scrub starts 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.662138+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 deep-scrub ok 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.662138+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 deep-scrub ok 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.929639+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.929639+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.947085+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:40.947085+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:41.347157+0000 mgr.y (mgr.24401) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:41.347157+0000 mgr.y (mgr.24401) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:41.785695+0000 osd.7 (osd.7) 15 : cluster [DBG] 3.e scrub starts 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:41.785695+0000 osd.7 (osd.7) 15 : cluster [DBG] 3.e scrub starts 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:41.789780+0000 osd.7 (osd.7) 16 : cluster [DBG] 3.e scrub ok 2024-04-26T16:18:42.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:42 smithi190 bash[23868]: cluster 2024-04-26T16:18:41.789780+0000 osd.7 (osd.7) 16 : cluster [DBG] 3.e scrub ok 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.659694+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.2 deep-scrub starts 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.659694+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.2 deep-scrub starts 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.662138+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 deep-scrub ok 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.662138+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 deep-scrub ok 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.929639+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.929639+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.947085+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:18:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:40.947085+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:41.347157+0000 mgr.y (mgr.24401) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:41.347157+0000 mgr.y (mgr.24401) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:41.785695+0000 osd.7 (osd.7) 15 : cluster [DBG] 3.e scrub starts 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:41.785695+0000 osd.7 (osd.7) 15 : cluster [DBG] 3.e scrub starts 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:41.789780+0000 osd.7 (osd.7) 16 : cluster [DBG] 3.e scrub ok 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[29234]: cluster 2024-04-26T16:18:41.789780+0000 osd.7 (osd.7) 16 : cluster [DBG] 3.e scrub ok 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.659694+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.2 deep-scrub starts 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.659694+0000 osd.5 (osd.5) 13 : cluster [DBG] 2.2 deep-scrub starts 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.662138+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 deep-scrub ok 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.662138+0000 osd.5 (osd.5) 14 : cluster [DBG] 2.2 deep-scrub ok 2024-04-26T16:18:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.929639+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.929639+0000 osd.4 (osd.4) 11 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.947085+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:40.947085+0000 osd.4 (osd.4) 12 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:41.347157+0000 mgr.y (mgr.24401) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:41.347157+0000 mgr.y (mgr.24401) 32 : cluster [DBG] pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:41.785695+0000 osd.7 (osd.7) 15 : cluster [DBG] 3.e scrub starts 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:41.785695+0000 osd.7 (osd.7) 15 : cluster [DBG] 3.e scrub starts 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:41.789780+0000 osd.7 (osd.7) 16 : cluster [DBG] 3.e scrub ok 2024-04-26T16:18:42.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:42 smithi050 bash[21463]: cluster 2024-04-26T16:18:41.789780+0000 osd.7 (osd.7) 16 : cluster [DBG] 3.e scrub ok 2024-04-26T16:18:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:43 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.820174+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:18:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:43 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.820174+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:18:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:43 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.822563+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:18:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:43 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.822563+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:18:43.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.820174+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:18:43.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.820174+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:18:43.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.822563+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:18:43.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.822563+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:18:43.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.820174+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:18:43.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.820174+0000 osd.7 (osd.7) 17 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:18:43.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.822563+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:18:43.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:43 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.822563+0000 osd.7 (osd.7) 18 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:18:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:44 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.934250+0000 osd.0 (osd.0) 19 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:18:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:44 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.934250+0000 osd.0 (osd.0) 19 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:18:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:44 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.938245+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:18:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:44 smithi190 bash[23868]: cluster 2024-04-26T16:18:42.938245+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:18:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:44 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.348046+0000 mgr.y (mgr.24401) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:44 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.348046+0000 mgr.y (mgr.24401) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.934250+0000 osd.0 (osd.0) 19 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.934250+0000 osd.0 (osd.0) 19 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.938245+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21463]: cluster 2024-04-26T16:18:42.938245+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.348046+0000 mgr.y (mgr.24401) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.348046+0000 mgr.y (mgr.24401) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.934250+0000 osd.0 (osd.0) 19 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:18:44.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.934250+0000 osd.0 (osd.0) 19 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:18:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.938245+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:18:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[29234]: cluster 2024-04-26T16:18:42.938245+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:18:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.348046+0000 mgr.y (mgr.24401) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.348046+0000 mgr.y (mgr.24401) 33 : cluster [DBG] pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:44.703 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:18:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:18:45.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.136018+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:18:45.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.136018+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.138614+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.138614+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.737318+0000 osd.5 (osd.5) 15 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.737318+0000 osd.5 (osd.5) 15 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.744863+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:43.744863+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.004205+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 deep-scrub starts 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.004205+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 deep-scrub starts 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.006842+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 deep-scrub ok 2024-04-26T16:18:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:45 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.006842+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 deep-scrub ok 2024-04-26T16:18:45.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.136018+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:18:45.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.136018+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:18:45.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.138614+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:18:45.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.138614+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:18:45.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.737318+0000 osd.5 (osd.5) 15 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:18:45.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.737318+0000 osd.5 (osd.5) 15 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.744863+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:43.744863+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.004205+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 deep-scrub starts 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.004205+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 deep-scrub starts 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.006842+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 deep-scrub ok 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.006842+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 deep-scrub ok 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.136018+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.136018+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.138614+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.138614+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:18:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.737318+0000 osd.5 (osd.5) 15 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.737318+0000 osd.5 (osd.5) 15 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.744863+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:43.744863+0000 osd.5 (osd.5) 16 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.004205+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 deep-scrub starts 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.004205+0000 osd.3 (osd.3) 17 : cluster [DBG] 6.11 deep-scrub starts 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.006842+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 deep-scrub ok 2024-04-26T16:18:45.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:45 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.006842+0000 osd.3 (osd.3) 18 : cluster [DBG] 6.11 deep-scrub ok 2024-04-26T16:18:46.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 -- 172.21.15.50:0/265127403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c075650 msgr2=0x7f5e2c075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:46.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 --2- 172.21.15.50:0/265127403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c075650 0x7f5e2c075a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5e28009990 tx=0x7f5e28009ca0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 -- 172.21.15.50:0/265127403 shutdown_connections 2024-04-26T16:18:46.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 --2- 172.21.15.50:0/265127403 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c10fe80 0x7f5e2c112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 --2- 172.21.15.50:0/265127403 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c075650 0x7f5e2c075a30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 --2- 172.21.15.50:0/265127403 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 0x7f5e2c10f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 -- 172.21.15.50:0/265127403 >> 172.21.15.50:0/265127403 conn(0x7f5e2c07a4f0 msgr2=0x7f5e2c07a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 -- 172.21.15.50:0/265127403 shutdown_connections 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.321+0000 7f5e345a9700 1 -- 172.21.15.50:0/265127403 wait complete. 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 librados: starting msgr at 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 librados: starting objecter 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 Processor -- start 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- start start 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 librados: setting wanted keys 2024-04-26T16:18:46.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 librados: calling monclient init 2024-04-26T16:18:46.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c075650 0x7f5e2c19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:46.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 0x7f5e2c19b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 0x7f5e2c19fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c114d50 con 0x7f5e2c075f70 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e2c114bd0 con 0x7f5e2c075650 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5e2c114ed0 con 0x7f5e2c10fe80 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 0x7f5e2c19fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 0x7f5e2c19fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45136/0 (socket says 172.21.15.50:45136) 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e31b44700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 0x7f5e2c19b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 -- 172.21.15.50:0/4270315933 learned_addr learned my addr 172.21.15.50:0/4270315933 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32345700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c075650 0x7f5e2c19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 -- 172.21.15.50:0/4270315933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c075650 msgr2=0x7f5e2c19b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c075650 0x7f5e2c19b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 -- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 msgr2=0x7f5e2c19b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 0x7f5e2c19b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 -- 172.21.15.50:0/4270315933 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e2c1a0260 con 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e31b44700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 0x7f5e2c19b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32345700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c075650 0x7f5e2c19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e32b46700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 0x7f5e2c19fb20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e24007ee0 tx=0x7f5e2400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e237fe700 10 client.?.objecter ms_handle_connect 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e237fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e2400dbd0 con 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e2c1a0550 con 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5e2400dd70 con 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e24005820 con 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 10 client.24446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e2c1a7dd0 con 0x7f5e2c10fe80 2024-04-26T16:18:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 librados: init done 2024-04-26T16:18:46.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e2c051960 con 0x7f5e2c10fe80 2024-04-26T16:18:46.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.325+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5e2401a020 con 0x7f5e2c10fe80 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e18072030 0x7f5e180744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(103..103 src has 22..103) ==== 7450+0+0 (secure 0 0 0) 0x7f5e24090980 con 0x7f5e2c10fe80 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 10 client.24446.objecter ms_dispatch 0x7f5e2c001550 osd_map(103..103 src has 22..103) 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 3 client.24446.objecter handle_osd_map got epochs [103,103] > 0 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 3 client.24446.objecter handle_osd_map decoding full epoch 103 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e32345700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e18072030 0x7f5e180744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 20 client.24446.objecter dump_active .. 0 homeless 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e32345700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e18072030 0x7f5e180744f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800b2e0 tx=0x7f5e2801a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:46.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.329+0000 7f5e237fe700 10 client.24446.objecter ms_handle_connect 0x7f5e18072030 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.904139+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.904139+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.906827+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.906827+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.988247+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.988247+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.990544+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:44.990544+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.348761+0000 mgr.y (mgr.24401) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.348761+0000 mgr.y (mgr.24401) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.848391+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:18:46.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.848391+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:18:46.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.333+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5e24010070 con 0x7f5e2c10fe80 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.904139+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.904139+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.906827+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.906827+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.988247+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.988247+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.990544+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:44.990544+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:18:46.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.348761+0000 mgr.y (mgr.24401) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.348761+0000 mgr.y (mgr.24401) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.848391+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.848391+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.904139+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.904139+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.906827+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.906827+0000 osd.0 (osd.0) 22 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.988247+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.988247+0000 osd.4 (osd.4) 13 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:18:46.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.990544+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:18:46.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:44.990544+0000 osd.4 (osd.4) 14 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:18:46.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.348761+0000 mgr.y (mgr.24401) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:46.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.348761+0000 mgr.y (mgr.24401) 34 : cluster [DBG] pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:46.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.848391+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:18:46.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:46 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.848391+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:18:46.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.665+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f5e2c02e490 con 0x7f5e2c10fe80 2024-04-26T16:18:46.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.669+0000 7f5e237fe700 1 -- 172.21.15.50:0/4270315933 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) ==== 170+0+59 (secure 0 0 0) 0x7f5e240192c0 con 0x7f5e2c10fe80 2024-04-26T16:18:46.663 INFO:teuthology.orchestra.run.smithi050.stdout:[client.0] 2024-04-26T16:18:46.663 INFO:teuthology.orchestra.run.smithi050.stdout: key = AQDm0ytm6pziJxAAHMcWxguR0yKchVYewHsLLw== 2024-04-26T16:18:46.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 10 librados: watch_flush enter 2024-04-26T16:18:46.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 10 librados: watch_flush exit 2024-04-26T16:18:46.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 20 client.24446.objecter shutdown clearing up homeless session... 2024-04-26T16:18:46.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 10 client.24446.objecter successfully canceled tick 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e18072030 msgr2=0x7f5e180744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e18072030 0x7f5e180744f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5e2800b2e0 tx=0x7f5e2801a040 comp rx=0 tx=0).stop 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 msgr2=0x7f5e2c19fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 0x7f5e2c19fb20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e24007ee0 tx=0x7f5e2400d3b0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 shutdown_connections 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e18072030 0x7f5e180744f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e2c075650 0x7f5e2c19b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e2c10fe80 0x7f5e2c19fb20 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 --2- 172.21.15.50:0/4270315933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e2c075f70 0x7f5e2c19b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 >> 172.21.15.50:0/4270315933 conn(0x7f5e2c07a4f0 msgr2=0x7f5e2c100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 shutdown_connections 2024-04-26T16:18:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 -- 172.21.15.50:0/4270315933 wait complete. 2024-04-26T16:18:46.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:18:46.673+0000 7f5e345a9700 1 librados: shutdown 2024-04-26T16:18:46.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.633932+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.633932+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.636569+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.636569+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.852721+0000 osd.7 (osd.7) 20 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.852721+0000 osd.7 (osd.7) 20 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.899458+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.899458+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:18:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.901908+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:45.901908+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.020163+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.020163+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.022532+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.022532+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.165721+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.165721+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.169974+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.169974+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.363648+0000 mgr.y (mgr.24401) 35 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.363648+0000 mgr.y (mgr.24401) 35 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.668188+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.50:0/4270315933' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.668188+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.50:0/4270315933' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.668867+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.668867+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.671678+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:47 smithi190 bash[23868]: audit 2024-04-26T16:18:46.671678+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:47.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.633932+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:18:47.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.633932+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.636569+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.636569+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.852721+0000 osd.7 (osd.7) 20 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.852721+0000 osd.7 (osd.7) 20 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.899458+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.899458+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.901908+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:45.901908+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.020163+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.020163+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.022532+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.022532+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:18:47.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.165721+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.165721+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.169974+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.169974+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.363648+0000 mgr.y (mgr.24401) 35 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.363648+0000 mgr.y (mgr.24401) 35 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.668188+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.50:0/4270315933' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.668188+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.50:0/4270315933' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.668867+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.668867+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.671678+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[21463]: audit 2024-04-26T16:18:46.671678+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:47.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.633932+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.633932+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.636569+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.636569+0000 osd.2 (osd.2) 14 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.852721+0000 osd.7 (osd.7) 20 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.852721+0000 osd.7 (osd.7) 20 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.899458+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.899458+0000 osd.0 (osd.0) 23 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.901908+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:45.901908+0000 osd.0 (osd.0) 24 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.020163+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.020163+0000 osd.4 (osd.4) 15 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.022532+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:18:47.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.022532+0000 osd.4 (osd.4) 16 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.165721+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.165721+0000 osd.1 (osd.1) 19 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.169974+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.169974+0000 osd.1 (osd.1) 20 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.363648+0000 mgr.y (mgr.24401) 35 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.363648+0000 mgr.y (mgr.24401) 35 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.668188+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.50:0/4270315933' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.668188+0000 mon.c (mon.2) 42 : audit [INF] from='client.? 172.21.15.50:0/4270315933' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.668867+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.668867+0000 mon.a (mon.0) 863 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.671678+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:47.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:47 smithi050 bash[29234]: audit 2024-04-26T16:18:46.671678+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:47.936 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:18:47.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-26T16:18:47.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T16:18:47.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-26T16:18:48.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.261154+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:48.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.261154+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:48.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.261284+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:48.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.261284+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:48.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.625265+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.625265+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.628129+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:46.628129+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.057236+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.057236+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.085559+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.2 deep-scrub starts 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.085559+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.2 deep-scrub starts 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.089953+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.2 deep-scrub ok 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.089953+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.2 deep-scrub ok 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.349493+0000 mgr.y (mgr.24401) 36 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:48.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.349493+0000 mgr.y (mgr.24401) 36 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.261154+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.261154+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.261284+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.261284+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.625265+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.625265+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.628129+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:46.628129+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.057236+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.057236+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.085559+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.2 deep-scrub starts 2024-04-26T16:18:48.224 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.085559+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.2 deep-scrub starts 2024-04-26T16:18:48.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.089953+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.2 deep-scrub ok 2024-04-26T16:18:48.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.089953+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.2 deep-scrub ok 2024-04-26T16:18:48.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.349493+0000 mgr.y (mgr.24401) 36 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:48.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:48 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.349493+0000 mgr.y (mgr.24401) 36 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:48.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.261154+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.261154+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.261284+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.261284+0000 osd.2 (osd.2) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.625265+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.625265+0000 osd.5 (osd.5) 17 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.628129+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:46.628129+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.057236+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.057236+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.085559+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.2 deep-scrub starts 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.085559+0000 osd.4 (osd.4) 17 : cluster [DBG] 6.2 deep-scrub starts 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.089953+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.2 deep-scrub ok 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.089953+0000 osd.4 (osd.4) 18 : cluster [DBG] 6.2 deep-scrub ok 2024-04-26T16:18:48.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.349493+0000 mgr.y (mgr.24401) 36 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:48.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:48 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.349493+0000 mgr.y (mgr.24401) 36 : cluster [DBG] pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:49.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.584588+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.584588+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.586851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.586851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.874517+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.874517+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.878434+0000 osd.7 (osd.7) 22 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:47.878434+0000 osd.7 (osd.7) 22 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.079651+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.079651+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.081668+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:18:49.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.081668+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:18:49.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.121713+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:18:49.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.121713+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:18:49.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.126722+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:18:49.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:49 smithi190 bash[23868]: cluster 2024-04-26T16:18:48.126722+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:18:49.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.584588+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:18:49.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.584588+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:18:49.341 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.586851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.586851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.874517+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.874517+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.878434+0000 osd.7 (osd.7) 22 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:47.878434+0000 osd.7 (osd.7) 22 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.079651+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.079651+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.081668+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.081668+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.121713+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.121713+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.126722+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:18:49.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[21463]: cluster 2024-04-26T16:18:48.126722+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.584588+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.584588+0000 osd.5 (osd.5) 19 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.586851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.586851+0000 osd.5 (osd.5) 20 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.874517+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.874517+0000 osd.7 (osd.7) 21 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.878434+0000 osd.7 (osd.7) 22 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:47.878434+0000 osd.7 (osd.7) 22 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.079651+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.079651+0000 osd.4 (osd.4) 19 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.081668+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.081668+0000 osd.4 (osd.4) 20 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.121713+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:18:49.343 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.121713+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:18:49.344 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.126722+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:18:49.344 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:49 smithi050 bash[29234]: cluster 2024-04-26T16:18:48.126722+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:18:50.194 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.194 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.195 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.195 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.195 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.196 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.196 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.079007+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:18:50.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.079007+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:18:50.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.081395+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:18:50.196 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.081395+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.144057+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.144057+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.145584+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.145584+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.350277+0000 mgr.y (mgr.24401) 37 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.350277+0000 mgr.y (mgr.24401) 37 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.197 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.079007+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.079007+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.081395+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.081395+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.144057+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.144057+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.145584+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.145584+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.350277+0000 mgr.y (mgr.24401) 37 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.350277+0000 mgr.y (mgr.24401) 37 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:50.198 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.079007+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.079007+0000 osd.4 (osd.4) 21 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.081395+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.081395+0000 osd.4 (osd.4) 22 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.144057+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.144057+0000 osd.1 (osd.1) 21 : cluster [DBG] 2.a scrub starts 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.145584+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-04-26T16:18:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.145584+0000 osd.1 (osd.1) 22 : cluster [DBG] 2.a scrub ok 2024-04-26T16:18:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.350277+0000 mgr.y (mgr.24401) 37 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:50 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.350277+0000 mgr.y (mgr.24401) 37 : cluster [DBG] pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:50.451 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.452 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.452 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.452 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.452 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.453 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.454 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.454 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.454 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:50.454 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: Started Ceph alertmanager.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:18:50.454 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:18:50 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.547693+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.a scrub starts 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.547693+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.a scrub starts 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.550320+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.a scrub ok 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.550320+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.a scrub ok 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.861292+0000 osd.7 (osd.7) 23 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.861292+0000 osd.7 (osd.7) 23 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.863990+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:18:51.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.863990+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.961851+0000 osd.0 (osd.0) 25 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.961851+0000 osd.0 (osd.0) 25 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.965450+0000 osd.0 (osd.0) 26 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:49.965450+0000 osd.0 (osd.0) 26 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.065459+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.065459+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.067756+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.067756+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.505466+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.505466+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.512423+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.512423+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.528574+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.528574+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.533248+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.533248+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.582099+0000 mon.c (mon.2) 43 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: audit 2024-04-26T16:18:50.582099+0000 mon.c (mon.2) 43 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.913224+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-04-26T16:18:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:51 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.913224+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.547693+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.a scrub starts 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.547693+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.a scrub starts 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.550320+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.a scrub ok 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.550320+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.a scrub ok 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.861292+0000 osd.7 (osd.7) 23 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.861292+0000 osd.7 (osd.7) 23 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.863990+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:18:51.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.863990+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.961851+0000 osd.0 (osd.0) 25 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.961851+0000 osd.0 (osd.0) 25 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.965450+0000 osd.0 (osd.0) 26 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:49.965450+0000 osd.0 (osd.0) 26 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.065459+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.065459+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.067756+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.067756+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.505466+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.505466+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.512423+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.512423+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.528574+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.528574+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.533248+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.533248+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.582099+0000 mon.c (mon.2) 43 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: audit 2024-04-26T16:18:50.582099+0000 mon.c (mon.2) 43 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.913224+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.913224+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.547693+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.a scrub starts 2024-04-26T16:18:51.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.547693+0000 osd.5 (osd.5) 21 : cluster [DBG] 6.a scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.550320+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.a scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.550320+0000 osd.5 (osd.5) 22 : cluster [DBG] 6.a scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.861292+0000 osd.7 (osd.7) 23 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.861292+0000 osd.7 (osd.7) 23 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.863990+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.863990+0000 osd.7 (osd.7) 24 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.961851+0000 osd.0 (osd.0) 25 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.961851+0000 osd.0 (osd.0) 25 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.965450+0000 osd.0 (osd.0) 26 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:49.965450+0000 osd.0 (osd.0) 26 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.065459+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.065459+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.067756+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:18:51.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.067756+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.505466+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.505466+0000 mon.a (mon.0) 866 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.512423+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.512423+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.528574+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.528574+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.533248+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.533248+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.582099+0000 mon.c (mon.2) 43 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: audit 2024-04-26T16:18:50.582099+0000 mon.c (mon.2) 43 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.913224+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-04-26T16:18:51.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.913224+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2024-04-26T16:18:52.067 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.756Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-26T16:18:52.067 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.756Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-26T16:18:52.068 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.758Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.50 port=9094 2024-04-26T16:18:52.068 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.758Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-26T16:18:52.068 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.790Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-26T16:18:52.068 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.790Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-26T16:18:52.068 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.794Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-26T16:18:52.068 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:51 smithi050 bash[58386]: ts=2024-04-26T16:18:51.794Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.541138+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.541138+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.542513+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.542513+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.557904+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.c scrub starts 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.557904+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.c scrub starts 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.560376+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.560376+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.916673+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:18:52.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.916673+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.922087+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:50.922087+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.351781+0000 mgr.y (mgr.24401) 38 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.351781+0000 mgr.y (mgr.24401) 38 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.780456+0000 osd.7 (osd.7) 25 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.780456+0000 osd.7 (osd.7) 25 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.783494+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:18:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:52 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.783494+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:18:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.541138+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:18:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.541138+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:18:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.542513+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:18:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.542513+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:18:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.557904+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.c scrub starts 2024-04-26T16:18:52.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.557904+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.c scrub starts 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.560376+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.560376+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.916673+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.916673+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.922087+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:50.922087+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.351781+0000 mgr.y (mgr.24401) 38 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.351781+0000 mgr.y (mgr.24401) 38 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.780456+0000 osd.7 (osd.7) 25 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.780456+0000 osd.7 (osd.7) 25 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.783494+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.783494+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:18:52.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.541138+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.541138+0000 osd.2 (osd.2) 17 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.542513+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.542513+0000 osd.2 (osd.2) 18 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.557904+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.c scrub starts 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.557904+0000 osd.5 (osd.5) 23 : cluster [DBG] 3.c scrub starts 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.560376+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.560376+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.916673+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.916673+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.922087+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:50.922087+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:18:52.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.351781+0000 mgr.y (mgr.24401) 38 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:52.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.351781+0000 mgr.y (mgr.24401) 38 : cluster [DBG] pgmap v21: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:52.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.780456+0000 osd.7 (osd.7) 25 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:18:52.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.780456+0000 osd.7 (osd.7) 25 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:18:52.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.783494+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:18:52.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:52 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.783494+0000 osd.7 (osd.7) 26 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:18:52.746 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:18:53.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.532847+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:18:53.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.532847+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:18:53.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.535215+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.535215+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.942006+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.942006+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.945814+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:51.945814+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.228484+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.228484+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.230954+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.230954+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.532847+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.532847+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:18:53.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.535215+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.535215+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.942006+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.942006+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.945814+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:51.945814+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.228484+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.228484+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.230954+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:18:53.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.230954+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:18:53.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.532847+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:18:53.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.532847+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:18:53.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.535215+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.535215+0000 osd.5 (osd.5) 26 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.942006+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.942006+0000 osd.3 (osd.3) 21 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.945814+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:51.945814+0000 osd.3 (osd.3) 22 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.228484+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.228484+0000 osd.1 (osd.1) 23 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.230954+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:18:53.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:53 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.230954+0000 osd.1 (osd.1) 24 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:18:54.086 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:18:53 smithi050 bash[58386]: ts=2024-04-26T16:18:53.759Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000672163s 2024-04-26T16:18:54.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.517711+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.b scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.517711+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.b scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.520171+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.b scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:52.520171+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.b scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.051492+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.051492+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.054945+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.054945+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.087395+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.087395+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.089824+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.089824+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.186680+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.186680+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.188968+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:18:54.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.188968+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.352450+0000 mgr.y (mgr.24401) 39 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.352450+0000 mgr.y (mgr.24401) 39 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.790288+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.790288+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.791585+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.791585+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.517711+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.b scrub starts 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.517711+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.b scrub starts 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.520171+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.b scrub ok 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:52.520171+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.b scrub ok 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.051492+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.051492+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:18:54.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.054945+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.054945+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.087395+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.087395+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.089824+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.089824+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.186680+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.186680+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.188968+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.188968+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.352450+0000 mgr.y (mgr.24401) 39 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.352450+0000 mgr.y (mgr.24401) 39 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.790288+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.790288+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.791585+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:18:54.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.791585+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:18:54.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.517711+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.b scrub starts 2024-04-26T16:18:54.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.517711+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.b scrub starts 2024-04-26T16:18:54.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.520171+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.b scrub ok 2024-04-26T16:18:54.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:52.520171+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.b scrub ok 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.051492+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.051492+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.054945+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.054945+0000 osd.4 (osd.4) 24 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.087395+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.087395+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.089824+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.089824+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.186680+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:18:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.186680+0000 osd.1 (osd.1) 25 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.188968+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.188968+0000 osd.1 (osd.1) 26 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.352450+0000 mgr.y (mgr.24401) 39 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.352450+0000 mgr.y (mgr.24401) 39 : cluster [DBG] pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.790288+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.790288+0000 osd.7 (osd.7) 27 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.791585+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:18:54.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:54 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.791585+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:18:54.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:18:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:18:55.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.485246+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.485246+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.489733+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.489733+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.510184+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.510184+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.512687+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.512687+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.982488+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.982488+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.984851+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:53.984851+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:18:55.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.050353+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.050353+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.053648+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.053648+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.142557+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.142557+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.144965+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.144965+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.179863+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.179863+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.179994+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.179994+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.742011+0000 osd.7 (osd.7) 29 : cluster [DBG] 3.d scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.742011+0000 osd.7 (osd.7) 29 : cluster [DBG] 3.d scrub starts 2024-04-26T16:18:55.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.744813+0000 osd.7 (osd.7) 30 : cluster [DBG] 3.d scrub ok 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.744813+0000 osd.7 (osd.7) 30 : cluster [DBG] 3.d scrub ok 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.485246+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.485246+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.489733+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.489733+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.510184+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.510184+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.512687+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.512687+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.982488+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.982488+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:18:55.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.984851+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:53.984851+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.050353+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.050353+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.053648+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.053648+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.142557+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.142557+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.144965+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.144965+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.179863+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.179863+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.179994+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.179994+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.742011+0000 osd.7 (osd.7) 29 : cluster [DBG] 3.d scrub starts 2024-04-26T16:18:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.742011+0000 osd.7 (osd.7) 29 : cluster [DBG] 3.d scrub starts 2024-04-26T16:18:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.744813+0000 osd.7 (osd.7) 30 : cluster [DBG] 3.d scrub ok 2024-04-26T16:18:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:55 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.744813+0000 osd.7 (osd.7) 30 : cluster [DBG] 3.d scrub ok 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 -- 172.21.15.190:0/1440891176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660107650 msgr2=0x7f966010f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 --2- 172.21.15.190:0/1440891176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660107650 0x7f966010f220 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f965800b0a0 tx=0x7f965800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 -- 172.21.15.190:0/1440891176 shutdown_connections 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 --2- 172.21.15.190:0/1440891176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660107650 0x7f966010f220 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 --2- 172.21.15.190:0/1440891176 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9660102580 0x7f9660102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 --2- 172.21.15.190:0/1440891176 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 0x7f9660106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 -- 172.21.15.190:0/1440891176 >> 172.21.15.190:0/1440891176 conn(0x7f96600792c0 msgr2=0x7f96600796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.440+0000 7f9666c24700 1 -- 172.21.15.190:0/1440891176 shutdown_connections 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- 172.21.15.190:0/1440891176 wait complete. 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 librados: starting msgr at 2024-04-26T16:18:55.437 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 librados: starting objecter 2024-04-26T16:18:55.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 Processor -- start 2024-04-26T16:18:55.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- start start 2024-04-26T16:18:55.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 librados: setting wanted keys 2024-04-26T16:18:55.438 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 librados: calling monclient init 2024-04-26T16:18:55.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 0x7f966019b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:55.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 0x7f966019b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:55.439 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9660107650 0x7f966019fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9660111a20 con 0x7f9660102f30 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f96601118a0 con 0x7f9660102580 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9660111ba0 con 0x7f9660107650 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 0x7f966019b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 0x7f966019b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:35974/0 (socket says 172.21.15.190:35974) 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 -- 172.21.15.190:0/4247296996 learned_addr learned my addr 172.21.15.190:0/4247296996 (peer_addr_for_me v2:172.21.15.190:0/0) 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96651c1700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9660107650 0x7f966019fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 -- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9660107650 msgr2=0x7f966019fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9660107650 0x7f966019fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 -- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 msgr2=0x7f966019b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965ffff700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 0x7f966019b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 0x7f966019b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.440 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 -- 172.21.15.190:0/4247296996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96601a02d0 con 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965ffff700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 0x7f966019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96649c0700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 0x7f966019b2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f964800b530 tx=0x7f964800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965dffb700 10 client.?.objecter ms_handle_connect 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965dffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648015020 con 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9648011c90 con 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648010bf0 con 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- 172.21.15.190:0/4247296996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96601a05c0 con 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 10 client.24466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 -- 172.21.15.190:0/4247296996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96601a7e40 con 0x7f9660102580 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f9666c24700 1 librados: init done 2024-04-26T16:18:55.441 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.444+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9644005120 con 0x7f9660102580 2024-04-26T16:18:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9648018020 con 0x7f9660102580 2024-04-26T16:18:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f964c072030 0x7f964c0744f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:18:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(106..106 src has 22..106) ==== 7450+0+0 (secure 0 0 0) 0x7f9648090a80 con 0x7f9660102580 2024-04-26T16:18:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 10 client.24466.objecter ms_dispatch 0x7f9660001550 osd_map(106..106 src has 22..106) 2024-04-26T16:18:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 3 client.24466.objecter handle_osd_map got epochs [106,106] > 0 2024-04-26T16:18:55.442 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 3 client.24466.objecter handle_osd_map decoding full epoch 106 2024-04-26T16:18:55.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 20 client.24466.objecter dump_active .. 0 homeless 2024-04-26T16:18:55.443 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965ffff700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f964c072030 0x7f964c0744f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:18:55.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965ffff700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f964c072030 0x7f964c0744f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9650005fd0 tx=0x7f9650005dc0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:18:55.444 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.448+0000 7f965dffb700 10 client.24466.objecter ms_handle_connect 0x7f964c072030 2024-04-26T16:18:55.450 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.456+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f964801f020 con 0x7f9660102580 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.485246+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.485246+0000 osd.5 (osd.5) 27 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.489733+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.489733+0000 osd.5 (osd.5) 28 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.510184+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.510184+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:18:55.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.512687+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.512687+0000 osd.2 (osd.2) 22 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.982488+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.982488+0000 osd.3 (osd.3) 23 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.984851+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:53.984851+0000 osd.3 (osd.3) 24 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.050353+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.050353+0000 osd.4 (osd.4) 25 : cluster [DBG] 2.f scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.053648+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.053648+0000 osd.4 (osd.4) 26 : cluster [DBG] 2.f scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.142557+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.142557+0000 osd.1 (osd.1) 27 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.144965+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.144965+0000 osd.1 (osd.1) 28 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.179863+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:55.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.179863+0000 osd.1 (osd.1) 29 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:18:55.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.179994+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:55.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.179994+0000 osd.1 (osd.1) 30 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:18:55.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.742011+0000 osd.7 (osd.7) 29 : cluster [DBG] 3.d scrub starts 2024-04-26T16:18:55.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.742011+0000 osd.7 (osd.7) 29 : cluster [DBG] 3.d scrub starts 2024-04-26T16:18:55.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.744813+0000 osd.7 (osd.7) 30 : cluster [DBG] 3.d scrub ok 2024-04-26T16:18:55.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:55 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.744813+0000 osd.7 (osd.7) 30 : cluster [DBG] 3.d scrub ok 2024-04-26T16:18:55.763 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.768+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f9644005410 con 0x7f9660102580 2024-04-26T16:18:55.769 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f965dffb700 1 -- 172.21.15.190:0/4247296996 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) ==== 170+0+59 (secure 0 0 0) 0x7f964805cc10 con 0x7f9660102580 2024-04-26T16:18:55.769 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-04-26T16:18:55.770 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQDv0ytmZXtFLhAAcT0L0fgNYImiy0AkdFhswQ== 2024-04-26T16:18:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 10 librados: watch_flush enter 2024-04-26T16:18:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 10 librados: watch_flush exit 2024-04-26T16:18:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 20 client.24466.objecter shutdown clearing up homeless session... 2024-04-26T16:18:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 10 client.24466.objecter successfully canceled tick 2024-04-26T16:18:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f964c072030 msgr2=0x7f964c0744f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:55.772 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f964c072030 0x7f964c0744f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9650005fd0 tx=0x7f9650005dc0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 msgr2=0x7f966019b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 0x7f966019b2c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f964800b530 tx=0x7f964800b8f0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 shutdown_connections 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f964c072030 0x7f964c0744f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9660102580 0x7f966019b2c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9660107650 0x7f966019fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 --2- 172.21.15.190:0/4247296996 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9660102f30 0x7f966019b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 >> 172.21.15.190:0/4247296996 conn(0x7f96600792c0 msgr2=0x7f966006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 shutdown_connections 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 -- 172.21.15.190:0/4247296996 wait complete. 2024-04-26T16:18:55.773 INFO:teuthology.orchestra.run.smithi190.stderr:2024-04-26T16:18:55.776+0000 7f96577fe700 1 librados: shutdown 2024-04-26T16:18:56.348 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21717]: [26/Apr/2024:16:18:56] ENGINE Bus STOPPING 2024-04-26T16:18:56.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.495406+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.d scrub starts 2024-04-26T16:18:56.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.495406+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.d scrub starts 2024-04-26T16:18:56.348 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.498699+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.d scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:54.498699+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.d scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.026116+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.026116+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.028863+0000 osd.6 (osd.6) 17 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.028863+0000 osd.6 (osd.6) 17 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.030496+0000 osd.3 (osd.3) 26 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.030496+0000 osd.3 (osd.3) 26 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.031487+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.031487+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.082150+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.082150+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.083322+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.083322+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.109266+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-04-26T16:18:56.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.109266+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.353163+0000 mgr.y (mgr.24401) 40 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.353163+0000 mgr.y (mgr.24401) 40 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.466515+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.466515+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.492976+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.492976+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.733858+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.733858+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.734760+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.734760+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:18:56.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.495406+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.d scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.495406+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.d scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.498699+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.d scrub ok 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:54.498699+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.d scrub ok 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.026116+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.026116+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.028863+0000 osd.6 (osd.6) 17 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.028863+0000 osd.6 (osd.6) 17 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.030496+0000 osd.3 (osd.3) 26 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.030496+0000 osd.3 (osd.3) 26 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.031487+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.031487+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.082150+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:18:56.351 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.082150+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.083322+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.083322+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.109266+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.109266+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.353163+0000 mgr.y (mgr.24401) 40 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.353163+0000 mgr.y (mgr.24401) 40 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.466515+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.466515+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.492976+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.492976+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.733858+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.733858+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.734760+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:18:56.352 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.734760+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.772574+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.190:0/4247296996' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.772574+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.190:0/4247296996' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.776120+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.776120+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.778027+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.778027+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.819305+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.819305+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.827521+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: audit 2024-04-26T16:18:55.827521+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.909973+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.909973+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.772574+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.190:0/4247296996' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.772574+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.190:0/4247296996' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.776120+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.776120+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.778027+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.778027+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.819305+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.819305+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.827521+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: audit 2024-04-26T16:18:55.827521+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.909973+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-04-26T16:18:56.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.909973+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-04-26T16:18:56.365 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.495406+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.d scrub starts 2024-04-26T16:18:56.365 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.495406+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.d scrub starts 2024-04-26T16:18:56.365 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.498699+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.d scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:54.498699+0000 osd.2 (osd.2) 24 : cluster [DBG] 5.d scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.026116+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.026116+0000 osd.3 (osd.3) 25 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.028863+0000 osd.6 (osd.6) 17 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.028863+0000 osd.6 (osd.6) 17 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.030496+0000 osd.3 (osd.3) 26 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.030496+0000 osd.3 (osd.3) 26 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.031487+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.031487+0000 osd.6 (osd.6) 18 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.082150+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.082150+0000 osd.4 (osd.4) 27 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.083322+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:18:56.366 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.083322+0000 osd.4 (osd.4) 28 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.109266+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.109266+0000 mon.a (mon.0) 871 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.353163+0000 mgr.y (mgr.24401) 40 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.353163+0000 mgr.y (mgr.24401) 40 : cluster [DBG] pgmap v24: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.466515+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.466515+0000 mon.a (mon.0) 872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.492976+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.492976+0000 mon.c (mon.2) 44 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.733858+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.733858+0000 osd.7 (osd.7) 31 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.734760+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.734760+0000 osd.7 (osd.7) 32 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.772574+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.190:0/4247296996' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.772574+0000 mon.b (mon.1) 33 : audit [INF] from='client.? 172.21.15.190:0/4247296996' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.776120+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.776120+0000 mon.a (mon.0) 873 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.778027+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.778027+0000 mon.a (mon.0) 874 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.819305+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.819305+0000 mon.a (mon.0) 875 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.827521+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: audit 2024-04-26T16:18:55.827521+0000 mon.a (mon.0) 876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.909973+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-04-26T16:18:56.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.909973+0000 mon.a (mon.0) 877 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2024-04-26T16:18:56.470 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-04-26T16:18:56.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-26T16:18:56.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-26T16:18:56.489 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T16:18:56.489 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-26T16:18:56.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-26T16:18:56.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21717]: [26/Apr/2024:16:18:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:18:56.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21717]: [26/Apr/2024:16:18:56] ENGINE Bus STOPPED 2024-04-26T16:18:56.624 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21717]: [26/Apr/2024:16:18:56] ENGINE Bus STARTING 2024-04-26T16:18:56.624 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21717]: [26/Apr/2024:16:18:56] ENGINE Serving on http://:::9283 2024-04-26T16:18:56.624 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:18:56 smithi050 bash[21717]: [26/Apr/2024:16:18:56] ENGINE Bus STARTED 2024-04-26T16:18:56.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:18:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:18:57.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:18:57 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.451089+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.451089+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.454465+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.454465+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.890390+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.890390+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.891969+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:55.891969+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.129831+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:18:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.129831+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.138604+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.138604+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.221642+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.221642+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.228866+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.228866+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.233201+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.233201+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.234451+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.234451+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.240395+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.240395+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cephadm 2024-04-26T16:18:56.259475+0000 mgr.y (mgr.24401) 41 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-04-26T16:18:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cephadm 2024-04-26T16:18:56.259475+0000 mgr.y (mgr.24401) 41 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.311496+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.311496+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.320076+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.320076+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.326750+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.326750+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.328272+0000 mgr.y (mgr.24401) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.328272+0000 mgr.y (mgr.24401) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.333412+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.333412+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cephadm 2024-04-26T16:18:56.364260+0000 mgr.y (mgr.24401) 43 : cephadm [INF] Deploying daemon grafana.a on smithi190 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cephadm 2024-04-26T16:18:56.364260+0000 mgr.y (mgr.24401) 43 : cephadm [INF] Deploying daemon grafana.a on smithi190 2024-04-26T16:18:57.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.374706+0000 mgr.y (mgr.24401) 44 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: audit 2024-04-26T16:18:56.374706+0000 mgr.y (mgr.24401) 44 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.779298+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:18:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.779298+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:18:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.780690+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:18:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:57 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.780690+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:18:57.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.451089+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:18:57.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.451089+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.454465+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.454465+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.890390+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.890390+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.891969+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:55.891969+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.129831+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.129831+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.138604+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.138604+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.221642+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.221642+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.228866+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.228866+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.233201+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.233201+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.234451+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.234451+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.240395+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.240395+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cephadm 2024-04-26T16:18:56.259475+0000 mgr.y (mgr.24401) 41 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cephadm 2024-04-26T16:18:56.259475+0000 mgr.y (mgr.24401) 41 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.311496+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.311496+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.320076+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.451089+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.451089+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.454465+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.454465+0000 osd.5 (osd.5) 30 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.890390+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.890390+0000 osd.0 (osd.0) 27 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.891969+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:55.891969+0000 osd.0 (osd.0) 28 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.129831+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.129831+0000 osd.1 (osd.1) 31 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.138604+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.138604+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.221642+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.221642+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.228866+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.228866+0000 mon.a (mon.0) 879 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.233201+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.233201+0000 mon.c (mon.2) 45 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.234451+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.234451+0000 mon.c (mon.2) 46 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.240395+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.240395+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cephadm 2024-04-26T16:18:56.259475+0000 mgr.y (mgr.24401) 41 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cephadm 2024-04-26T16:18:56.259475+0000 mgr.y (mgr.24401) 41 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.311496+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.311496+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.320076+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.320076+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.326750+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.326750+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.328272+0000 mgr.y (mgr.24401) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.328272+0000 mgr.y (mgr.24401) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.333412+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.333412+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cephadm 2024-04-26T16:18:56.364260+0000 mgr.y (mgr.24401) 43 : cephadm [INF] Deploying daemon grafana.a on smithi190 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cephadm 2024-04-26T16:18:56.364260+0000 mgr.y (mgr.24401) 43 : cephadm [INF] Deploying daemon grafana.a on smithi190 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.374706+0000 mgr.y (mgr.24401) 44 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: audit 2024-04-26T16:18:56.374706+0000 mgr.y (mgr.24401) 44 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.779298+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.779298+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.780690+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.780690+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.320076+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.326750+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.326750+0000 mon.c (mon.2) 47 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.328272+0000 mgr.y (mgr.24401) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.328272+0000 mgr.y (mgr.24401) 42 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.333412+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.333412+0000 mon.a (mon.0) 883 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:18:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cephadm 2024-04-26T16:18:56.364260+0000 mgr.y (mgr.24401) 43 : cephadm [INF] Deploying daemon grafana.a on smithi190 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cephadm 2024-04-26T16:18:56.364260+0000 mgr.y (mgr.24401) 43 : cephadm [INF] Deploying daemon grafana.a on smithi190 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.374706+0000 mgr.y (mgr.24401) 44 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: audit 2024-04-26T16:18:56.374706+0000 mgr.y (mgr.24401) 44 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.779298+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.779298+0000 osd.7 (osd.7) 33 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.780690+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:18:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:57 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.780690+0000 osd.7 (osd.7) 34 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.460071+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.460071+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.461692+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:56.461692+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.128990+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.128990+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.131692+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:18:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.131692+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.354640+0000 mgr.y (mgr.24401) 45 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.354640+0000 mgr.y (mgr.24401) 45 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.492482+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.a scrub starts 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.492482+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.a scrub starts 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.494921+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.494921+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.949117+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub starts 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.949117+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub starts 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.951697+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.c scrub ok 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:57.951697+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.c scrub ok 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.070398+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e scrub starts 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.070398+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e scrub starts 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.073493+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e scrub ok 2024-04-26T16:18:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:58 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.073493+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e scrub ok 2024-04-26T16:18:58.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.460071+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:18:58.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.460071+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:18:58.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.461692+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:18:58.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:56.461692+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:18:58.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.128990+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.128990+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.131692+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.131692+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.354640+0000 mgr.y (mgr.24401) 45 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.354640+0000 mgr.y (mgr.24401) 45 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.492482+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.a scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.492482+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.a scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.494921+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.494921+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.949117+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.949117+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.951697+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.c scrub ok 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:57.951697+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.c scrub ok 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.070398+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e scrub starts 2024-04-26T16:18:58.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.070398+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e scrub starts 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.073493+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e scrub ok 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.073493+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e scrub ok 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.460071+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.460071+0000 osd.5 (osd.5) 31 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.461692+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:56.461692+0000 osd.5 (osd.5) 32 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.128990+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.128990+0000 osd.1 (osd.1) 33 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.131692+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.131692+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:18:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.354640+0000 mgr.y (mgr.24401) 45 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.354640+0000 mgr.y (mgr.24401) 45 : cluster [DBG] pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.492482+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.a scrub starts 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.492482+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.a scrub starts 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.494921+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.494921+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.a scrub ok 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.949117+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub starts 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.949117+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.c scrub starts 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.951697+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.c scrub ok 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:57.951697+0000 osd.3 (osd.3) 28 : cluster [DBG] 6.c scrub ok 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.070398+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e scrub starts 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.070398+0000 osd.4 (osd.4) 29 : cluster [DBG] 5.e scrub starts 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.073493+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e scrub ok 2024-04-26T16:18:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:58 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.073493+0000 osd.4 (osd.4) 30 : cluster [DBG] 5.e scrub ok 2024-04-26T16:18:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:59 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.162553+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:18:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:59 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.162553+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:18:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:59 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.165086+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:18:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:18:59 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.165086+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.162553+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.162553+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.165086+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.165086+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.162553+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.162553+0000 osd.1 (osd.1) 35 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.165086+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:18:59.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:18:59 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.165086+0000 osd.1 (osd.1) 36 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.890583+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.890583+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.892869+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:58.892869+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.355363+0000 mgr.y (mgr.24401) 46 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.355363+0000 mgr.y (mgr.24401) 46 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.516272+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub starts 2024-04-26T16:19:00.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.516272+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub starts 2024-04-26T16:19:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.518874+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.c scrub ok 2024-04-26T16:19:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.518874+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.c scrub ok 2024-04-26T16:19:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.771748+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:19:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.771748+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:19:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.773852+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:19:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:00 smithi190 bash[23868]: cluster 2024-04-26T16:18:59.773852+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:19:00.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.890583+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:19:00.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.890583+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:19:00.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.892869+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:19:00.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:58.892869+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:19:00.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.355363+0000 mgr.y (mgr.24401) 46 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:00.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.355363+0000 mgr.y (mgr.24401) 46 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.516272+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub starts 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.516272+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub starts 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.518874+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.c scrub ok 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.518874+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.c scrub ok 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.771748+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.771748+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.773852+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[21463]: cluster 2024-04-26T16:18:59.773852+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.890583+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.890583+0000 osd.0 (osd.0) 29 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:19:00.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.892869+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:58.892869+0000 osd.0 (osd.0) 30 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.355363+0000 mgr.y (mgr.24401) 46 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.355363+0000 mgr.y (mgr.24401) 46 : cluster [DBG] pgmap v27: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.516272+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub starts 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.516272+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.c scrub starts 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.518874+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.c scrub ok 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.518874+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.c scrub ok 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.771748+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.771748+0000 osd.7 (osd.7) 35 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.773852+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:19:00.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:00 smithi050 bash[29234]: cluster 2024-04-26T16:18:59.773852+0000 osd.7 (osd.7) 36 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:19:01.273 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:01.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.143889+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:19:01.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.143889+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.145588+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.145588+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.162706+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.162706+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.164637+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.164637+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.564940+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.564940+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.567412+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:19:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:01 smithi190 bash[23868]: cluster 2024-04-26T16:19:00.567412+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.143889+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.143889+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.145588+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.145588+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.162706+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.162706+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:19:01.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.164637+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.164637+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.564940+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.564940+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.567412+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[29234]: cluster 2024-04-26T16:19:00.567412+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.143889+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.143889+0000 osd.6 (osd.6) 19 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.145588+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.145588+0000 osd.6 (osd.6) 20 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.162706+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:19:01.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.162706+0000 osd.1 (osd.1) 37 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:19:01.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.164637+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:19:01.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.164637+0000 osd.1 (osd.1) 38 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:19:01.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.564940+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:19:01.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.564940+0000 osd.2 (osd.2) 29 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:19:01.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.567412+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:19:01.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[21463]: cluster 2024-04-26T16:19:00.567412+0000 osd.2 (osd.2) 30 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:19:02.124 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:01 smithi050 bash[58386]: ts=2024-04-26T16:19:01.763Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004141175s 2024-04-26T16:19:02.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.157717+0000 osd.6 (osd.6) 21 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.157717+0000 osd.6 (osd.6) 21 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.162022+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.162022+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.212705+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.212705+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.215031+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.215031+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:19:02.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.356806+0000 mgr.y (mgr.24401) 47 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.356806+0000 mgr.y (mgr.24401) 47 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.943942+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.943942+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.946160+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[29234]: cluster 2024-04-26T16:19:01.946160+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.157717+0000 osd.6 (osd.6) 21 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.157717+0000 osd.6 (osd.6) 21 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:19:02.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.162022+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.162022+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.212705+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.212705+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.215031+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.215031+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.356806+0000 mgr.y (mgr.24401) 47 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.356806+0000 mgr.y (mgr.24401) 47 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.943942+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.943942+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:19:02.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.946160+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:19:02.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:02 smithi050 bash[21463]: cluster 2024-04-26T16:19:01.946160+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.157717+0000 osd.6 (osd.6) 21 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.157717+0000 osd.6 (osd.6) 21 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.162022+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.162022+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.212705+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.212705+0000 osd.1 (osd.1) 39 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.215031+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.215031+0000 osd.1 (osd.1) 40 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.356806+0000 mgr.y (mgr.24401) 47 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:02.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.356806+0000 mgr.y (mgr.24401) 47 : cluster [DBG] pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:02.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.943942+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:19:02.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.943942+0000 osd.3 (osd.3) 29 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:19:02.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.946160+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:19:02.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:02 smithi190 bash[23868]: cluster 2024-04-26T16:19:01.946160+0000 osd.3 (osd.3) 30 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:19:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:03 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.103805+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-04-26T16:19:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:03 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.103805+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-04-26T16:19:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:03 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.105816+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-04-26T16:19:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:03 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.105816+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.103805+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.103805+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.105816+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.105816+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.103805+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.103805+0000 osd.4 (osd.4) 31 : cluster [DBG] 6.e scrub starts 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.105816+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-04-26T16:19:03.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:03 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.105816+0000 osd.4 (osd.4) 32 : cluster [DBG] 6.e scrub ok 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/3089406770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f76601094c0 msgr2=0x7f76601098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- 172.21.15.50:0/3089406770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f76601094c0 0x7f76601098a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f765000b0a0 tx=0x7f765000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/3089406770 shutdown_connections 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- 172.21.15.50:0/3089406770 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7660103e30 0x7f7660110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- 172.21.15.50:0/3089406770 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f76601094c0 0x7f76601098a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- 172.21.15.50:0/3089406770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7660103470 0x7f76601038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/3089406770 >> 172.21.15.50:0/3089406770 conn(0x7f766007a580 msgr2=0x7f766007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/3089406770 shutdown_connections 2024-04-26T16:19:04.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/3089406770 wait complete. 2024-04-26T16:19:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 librados: starting msgr at 2024-04-26T16:19:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 librados: starting objecter 2024-04-26T16:19:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 Processor -- start 2024-04-26T16:19:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- start start 2024-04-26T16:19:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 librados: setting wanted keys 2024-04-26T16:19:04.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 librados: calling monclient init 2024-04-26T16:19:04.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7660103470 0x7f7660075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 0x7f7660075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76601094c0 0x7f76600761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7660112ab0 con 0x7f76601094c0 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7660112930 con 0x7f7660103470 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7660112c30 con 0x7f7660103e30 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 0x7f7660075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76601094c0 0x7f76600761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 0x7f7660075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48540/0 (socket says 172.21.15.50:48540) 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76601094c0 0x7f76600761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57790/0 (socket says 172.21.15.50:57790) 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 -- 172.21.15.50:0/1653783704 learned_addr learned my addr 172.21.15.50:0/1653783704 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765effd700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7660103470 0x7f7660075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 -- 172.21.15.50:0/1653783704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7660103470 msgr2=0x7f7660075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:04.517 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7660103470 0x7f7660075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 -- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76601094c0 msgr2=0x7f76600761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76601094c0 0x7f76600761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 -- 172.21.15.50:0/1653783704 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76601a76e0 con 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f765e7fc700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 0x7f7660075c60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f765400d980 tx=0x7f765400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7647fff700 10 client.?.objecter ms_handle_connect 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7647fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654017070 con 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7654004830 con 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76601a79d0 con 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654005260 con 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 10 client.24455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76601a7ec0 con 0x7f7660103e30 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.521+0000 7f7665493700 1 librados: init done 2024-04-26T16:19:04.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f766006a110 con 0x7f7660103e30 2024-04-26T16:19:04.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7654009b40 con 0x7f7660103e30 2024-04-26T16:19:04.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7648071f80 0x7f7648074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:04.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(107..107 src has 57..107) ==== 7450+0+0 (secure 0 0 0) 0x7f7654091180 con 0x7f7660103e30 2024-04-26T16:19:04.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 10 client.24455.objecter ms_dispatch 0x7f7660001550 osd_map(107..107 src has 57..107) 2024-04-26T16:19:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 3 client.24455.objecter handle_osd_map got epochs [107,107] > 0 2024-04-26T16:19:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 3 client.24455.objecter handle_osd_map decoding full epoch 107 2024-04-26T16:19:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f765effd700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7648071f80 0x7f7648074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 20 client.24455.objecter dump_active .. 0 homeless 2024-04-26T16:19:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f765effd700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7648071f80 0x7f7648074440 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7650006130 tx=0x7f76500060a0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:04.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.525+0000 7f7647fff700 10 client.24455.objecter ms_handle_connect 0x7f7648071f80 2024-04-26T16:19:04.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.529+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7654020020 con 0x7f7660103e30 2024-04-26T16:19:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:04 smithi190 bash[23868]: cluster 2024-04-26T16:19:02.463117+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:19:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:04 smithi190 bash[23868]: cluster 2024-04-26T16:19:02.463117+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:19:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:04 smithi190 bash[23868]: cluster 2024-04-26T16:19:02.464780+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:19:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:04 smithi190 bash[23868]: cluster 2024-04-26T16:19:02.464780+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:19:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:04 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.357503+0000 mgr.y (mgr.24401) 48 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:04 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.357503+0000 mgr.y (mgr.24401) 48 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:04.623 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:19:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:19:04.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21463]: cluster 2024-04-26T16:19:02.463117+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:19:04.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21463]: cluster 2024-04-26T16:19:02.463117+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:19:04.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21463]: cluster 2024-04-26T16:19:02.464780+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:19:04.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21463]: cluster 2024-04-26T16:19:02.464780+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:19:04.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.357503+0000 mgr.y (mgr.24401) 48 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:04.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.357503+0000 mgr.y (mgr.24401) 48 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:04.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[29234]: cluster 2024-04-26T16:19:02.463117+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:19:04.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[29234]: cluster 2024-04-26T16:19:02.463117+0000 osd.5 (osd.5) 33 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:19:04.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[29234]: cluster 2024-04-26T16:19:02.464780+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:19:04.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[29234]: cluster 2024-04-26T16:19:02.464780+0000 osd.5 (osd.5) 34 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:19:04.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.357503+0000 mgr.y (mgr.24401) 48 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:04.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:04 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.357503+0000 mgr.y (mgr.24401) 48 : cluster [DBG] pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:04.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.817+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f76601a8090 con 0x7f7660103e30 2024-04-26T16:19:04.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.821+0000 7f7647fff700 1 -- 172.21.15.50:0/1653783704 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v22) ==== 74+0+179559 (secure 0 0 0) 0x7f765405d310 con 0x7f7660103e30 2024-04-26T16:19:04.816 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:19:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 10 librados: watch_flush enter 2024-04-26T16:19:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 10 librados: watch_flush exit 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 20 client.24455.objecter shutdown clearing up homeless session... 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 10 client.24455.objecter successfully canceled tick 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7648071f80 msgr2=0x7f7648074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7648071f80 0x7f7648074440 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7650006130 tx=0x7f76500060a0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 msgr2=0x7f7660075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 0x7f7660075c60 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f765400d980 tx=0x7f765400dd40 comp rx=0 tx=0).stop 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 shutdown_connections 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7648071f80 0x7f7648074440 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7660103470 0x7f7660075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f76601094c0 0x7f76600761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 --2- 172.21.15.50:0/1653783704 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7660103e30 0x7f7660075c60 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 >> 172.21.15.50:0/1653783704 conn(0x7f766007a580 msgr2=0x7f7660101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:04.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 shutdown_connections 2024-04-26T16:19:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 -- 172.21.15.50:0/1653783704 wait complete. 2024-04-26T16:19:04.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:04.825+0000 7f7665493700 1 librados: shutdown 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.476063+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.476063+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.478577+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:03.478577+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.538133+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.538133+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:19:05.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.539255+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.539255+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: audit 2024-04-26T16:19:04.819879+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? 172.21.15.50:0/1653783704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: audit 2024-04-26T16:19:04.819879+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? 172.21.15.50:0/1653783704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.955473+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.955473+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.959464+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.959464+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.476063+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.476063+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.478577+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:03.478577+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.538133+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:19:05.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.538133+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.539255+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.539255+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: audit 2024-04-26T16:19:04.819879+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? 172.21.15.50:0/1653783704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: audit 2024-04-26T16:19:04.819879+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? 172.21.15.50:0/1653783704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.955473+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.955473+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.959464+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:19:05.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:05 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.959464+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:19:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.476063+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:19:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.476063+0000 osd.5 (osd.5) 35 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:19:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.478577+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:03.478577+0000 osd.5 (osd.5) 36 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.538133+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.538133+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.539255+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.539255+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: audit 2024-04-26T16:19:04.819879+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? 172.21.15.50:0/1653783704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: audit 2024-04-26T16:19:04.819879+0000 mon.c (mon.2) 48 : audit [DBG] from='client.? 172.21.15.50:0/1653783704' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.955473+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.955473+0000 osd.3 (osd.3) 31 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.959464+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:19:05.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:05 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.959464+0000 osd.3 (osd.3) 32 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:19:05.701 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":22,"flags":0,"active_gid":24401,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6800","nonce":584348748},{"type":"v1","addr":"172.21.15.50:6801","nonce":584348748}]},"active_addr":"172.21.15.50:6801/584348748","active_change":"2024-04-26T16:18:25.318215+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24436,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.50:8443/","prometheus":"http://172.21.15.50:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":100,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":2620541526}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":3992753272}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":39578498}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.50:0","nonce":29210100}]}]} 2024-04-26T16:19:05.705 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-26T16:19:05.705 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T16:19:05.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T16:19:06.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:19:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.720896+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:19:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.720896+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:19:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.724241+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:04.724241+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.358240+0000 mgr.y (mgr.24401) 49 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.358240+0000 mgr.y (mgr.24401) 49 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.552911+0000 osd.2 (osd.2) 33 : cluster [DBG] 6.f scrub starts 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.552911+0000 osd.2 (osd.2) 33 : cluster [DBG] 6.f scrub starts 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.555525+0000 osd.2 (osd.2) 34 : cluster [DBG] 6.f scrub ok 2024-04-26T16:19:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:06 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.555525+0000 osd.2 (osd.2) 34 : cluster [DBG] 6.f scrub ok 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.720896+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.720896+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.724241+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:04.724241+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.358240+0000 mgr.y (mgr.24401) 49 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.358240+0000 mgr.y (mgr.24401) 49 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.552911+0000 osd.2 (osd.2) 33 : cluster [DBG] 6.f scrub starts 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.552911+0000 osd.2 (osd.2) 33 : cluster [DBG] 6.f scrub starts 2024-04-26T16:19:06.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.555525+0000 osd.2 (osd.2) 34 : cluster [DBG] 6.f scrub ok 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.555525+0000 osd.2 (osd.2) 34 : cluster [DBG] 6.f scrub ok 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.720896+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.720896+0000 osd.0 (osd.0) 31 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.724241+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:04.724241+0000 osd.0 (osd.0) 32 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.358240+0000 mgr.y (mgr.24401) 49 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.358240+0000 mgr.y (mgr.24401) 49 : cluster [DBG] pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.552911+0000 osd.2 (osd.2) 33 : cluster [DBG] 6.f scrub starts 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.552911+0000 osd.2 (osd.2) 33 : cluster [DBG] 6.f scrub starts 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.555525+0000 osd.2 (osd.2) 34 : cluster [DBG] 6.f scrub ok 2024-04-26T16:19:06.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:06 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.555525+0000 osd.2 (osd.2) 34 : cluster [DBG] 6.f scrub ok 2024-04-26T16:19:07.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.695007+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:19:07.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.695007+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.704261+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:05.704261+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: audit 2024-04-26T16:19:06.377597+0000 mgr.y (mgr.24401) 50 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: audit 2024-04-26T16:19:06.377597+0000 mgr.y (mgr.24401) 50 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.106896+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.106896+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.110977+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:19:07.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:07 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.110977+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:19:07.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.695007+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:19:07.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.695007+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:19:07.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.704261+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:19:07.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:05.704261+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:19:07.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: audit 2024-04-26T16:19:06.377597+0000 mgr.y (mgr.24401) 50 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:07.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: audit 2024-04-26T16:19:06.377597+0000 mgr.y (mgr.24401) 50 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.106896+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.106896+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.110977+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.110977+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.695007+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.695007+0000 osd.0 (osd.0) 33 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.704261+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:05.704261+0000 osd.0 (osd.0) 34 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: audit 2024-04-26T16:19:06.377597+0000 mgr.y (mgr.24401) 50 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: audit 2024-04-26T16:19:06.377597+0000 mgr.y (mgr.24401) 50 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:07.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.106896+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:19:07.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.106896+0000 osd.4 (osd.4) 33 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:19:07.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.110977+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:19:07.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:07 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.110977+0000 osd.4 (osd.4) 34 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:19:08.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:06.657172+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:06.657172+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:06.659777+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:06.659777+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.359731+0000 mgr.y (mgr.24401) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.359731+0000 mgr.y (mgr.24401) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.698473+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.f scrub starts 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.698473+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.f scrub starts 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.702825+0000 osd.7 (osd.7) 38 : cluster [DBG] 3.f scrub ok 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.702825+0000 osd.7 (osd.7) 38 : cluster [DBG] 3.f scrub ok 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.932962+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:19:08.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.932962+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:19:08.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.935377+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:19:08.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:08 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.935377+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:19:08.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:06.657172+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:19:08.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:06.657172+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:19:08.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:06.659777+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:19:08.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:06.659777+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.359731+0000 mgr.y (mgr.24401) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.359731+0000 mgr.y (mgr.24401) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.698473+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.f scrub starts 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.698473+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.f scrub starts 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.702825+0000 osd.7 (osd.7) 38 : cluster [DBG] 3.f scrub ok 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.702825+0000 osd.7 (osd.7) 38 : cluster [DBG] 3.f scrub ok 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.932962+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.932962+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.935377+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.935377+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:06.657172+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:19:08.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:06.657172+0000 osd.0 (osd.0) 35 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:06.659777+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:06.659777+0000 osd.0 (osd.0) 36 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.359731+0000 mgr.y (mgr.24401) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.359731+0000 mgr.y (mgr.24401) 51 : cluster [DBG] pgmap v31: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.698473+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.f scrub starts 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.698473+0000 osd.7 (osd.7) 37 : cluster [DBG] 3.f scrub starts 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.702825+0000 osd.7 (osd.7) 38 : cluster [DBG] 3.f scrub ok 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.702825+0000 osd.7 (osd.7) 38 : cluster [DBG] 3.f scrub ok 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.932962+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.932962+0000 osd.3 (osd.3) 33 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.935377+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:19:08.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:08 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.935377+0000 osd.3 (osd.3) 34 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.520626+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.d scrub starts 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.520626+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.d scrub starts 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.523145+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.d scrub ok 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:07.523145+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.d scrub ok 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:08.928557+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b scrub starts 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:08.928557+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b scrub starts 2024-04-26T16:19:09.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:08.931349+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub ok 2024-04-26T16:19:09.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:09 smithi190 bash[23868]: cluster 2024-04-26T16:19:08.931349+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub ok 2024-04-26T16:19:09.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.520626+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.d scrub starts 2024-04-26T16:19:09.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.520626+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.d scrub starts 2024-04-26T16:19:09.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.523145+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.d scrub ok 2024-04-26T16:19:09.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:07.523145+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.d scrub ok 2024-04-26T16:19:09.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:08.928557+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b scrub starts 2024-04-26T16:19:09.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:08.928557+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b scrub starts 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:08.931349+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub ok 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[21463]: cluster 2024-04-26T16:19:08.931349+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub ok 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.520626+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.d scrub starts 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.520626+0000 osd.5 (osd.5) 37 : cluster [DBG] 6.d scrub starts 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.523145+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.d scrub ok 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:07.523145+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.d scrub ok 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:08.928557+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b scrub starts 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:08.928557+0000 osd.3 (osd.3) 35 : cluster [DBG] 6.b scrub starts 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:08.931349+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub ok 2024-04-26T16:19:09.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:09 smithi050 bash[29234]: cluster 2024-04-26T16:19:08.931349+0000 osd.3 (osd.3) 36 : cluster [DBG] 6.b scrub ok 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.360470+0000 mgr.y (mgr.24401) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.360470+0000 mgr.y (mgr.24401) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.909903+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.909903+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.912771+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.912771+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.157147+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.157147+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:19:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.158727+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:19:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:10 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.158727+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:19:10.506 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.360470+0000 mgr.y (mgr.24401) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.360470+0000 mgr.y (mgr.24401) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.909903+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.909903+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.912771+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.912771+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.157147+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:19:10.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.157147+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.158727+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.158727+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.360470+0000 mgr.y (mgr.24401) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.360470+0000 mgr.y (mgr.24401) 52 : cluster [DBG] pgmap v32: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.909903+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.909903+0000 osd.3 (osd.3) 37 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.912771+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.912771+0000 osd.3 (osd.3) 38 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.157147+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.157147+0000 osd.4 (osd.4) 35 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.158727+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:19:10.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:10 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.158727+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:19:11.184 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.185 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.185 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.185 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.185 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.186 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.186 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.186 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.186 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.187 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.187 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.462 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.463 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.463 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.464 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.464 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: Started Ceph grafana.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:19:11.464 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.464 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.465 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.465 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.567257+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.f scrub starts 2024-04-26T16:19:11.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.567257+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.f scrub starts 2024-04-26T16:19:11.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.569107+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.f scrub ok 2024-04-26T16:19:11.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:09.569107+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.f scrub ok 2024-04-26T16:19:11.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.031736+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.031736+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.031833+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.031833+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.308550+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.308550+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: audit 2024-04-26T16:19:10.523042+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: audit 2024-04-26T16:19:10.523042+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.609928+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.609928+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.614032+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 bash[23868]: cluster 2024-04-26T16:19:10.614032+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:19:11.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:11 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:19:11.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.567257+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.f scrub starts 2024-04-26T16:19:11.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.567257+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.f scrub starts 2024-04-26T16:19:11.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.569107+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.f scrub ok 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:09.569107+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.f scrub ok 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.031736+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.031736+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.031833+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.031833+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.308550+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.308550+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: audit 2024-04-26T16:19:10.523042+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: audit 2024-04-26T16:19:10.523042+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.609928+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.609928+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.614032+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:19:11.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[29234]: cluster 2024-04-26T16:19:10.614032+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.567257+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.f scrub starts 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.567257+0000 osd.5 (osd.5) 39 : cluster [DBG] 5.f scrub starts 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.569107+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.f scrub ok 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:09.569107+0000 osd.5 (osd.5) 40 : cluster [DBG] 5.f scrub ok 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.031736+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.031736+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.031833+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.031833+0000 osd.5 (osd.5) 42 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.308550+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.308550+0000 mon.a (mon.0) 884 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: audit 2024-04-26T16:19:10.523042+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:11.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: audit 2024-04-26T16:19:10.523042+0000 mon.c (mon.2) 49 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:11.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.609928+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:19:11.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.609928+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:19:11.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.614032+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:19:11.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:11 smithi050 bash[21463]: cluster 2024-04-26T16:19:10.614032+0000 osd.0 (osd.0) 38 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:19:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: cluster 2024-04-26T16:19:11.361875+0000 mgr.y (mgr.24401) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: cluster 2024-04-26T16:19:11.361875+0000 mgr.y (mgr.24401) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.482120+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.482120+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.498763+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.498763+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.548462+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.548462+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.554792+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.554792+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.619632+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:11.619632+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:12.146369+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:12.146369+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:12.162988+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[23868]: audit 2024-04-26T16:19:12.162988+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.544849861Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545048478Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545063404Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545070569Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545076055Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545081677Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545087188Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545092876Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545099909Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545108182Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545114487Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545121762Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545127643Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545133467Z level=info msg="Path Home" path=/usr/share/grafana 2024-04-26T16:19:12.573 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545139125Z level=info msg="Path Data" path=/var/lib/grafana 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545144715Z level=info msg="Path Logs" path=/var/log/grafana 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545150336Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545156164Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=settings t=2024-04-26T16:19:12.545163351Z level=info msg="App mode production" 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=sqlstore t=2024-04-26T16:19:12.545222392Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=sqlstore t=2024-04-26T16:19:12.545244335Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.54615296Z level=info msg="Starting DB migrations" 2024-04-26T16:19:12.574 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.546236766Z level=info msg="Executing migration" id="create migration_log table" 2024-04-26T16:19:12.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.593607832Z level=info msg="Executing migration" id="create user table" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.627409836Z level=info msg="Executing migration" id="add unique index user.login" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.677504709Z level=info msg="Executing migration" id="add unique index user.email" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.728074741Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.753341625Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.778565997Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.803911389Z level=info msg="Executing migration" id="create user table v2" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.829111781Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-04-26T16:19:12.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.854319204Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: cluster 2024-04-26T16:19:11.361875+0000 mgr.y (mgr.24401) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: cluster 2024-04-26T16:19:11.361875+0000 mgr.y (mgr.24401) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.482120+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.482120+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.498763+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.498763+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.548462+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.548462+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.554792+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.554792+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.619632+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:11.619632+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:12.146369+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:12.146369+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:12.162988+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[21463]: audit 2024-04-26T16:19:12.162988+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: cluster 2024-04-26T16:19:11.361875+0000 mgr.y (mgr.24401) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: cluster 2024-04-26T16:19:11.361875+0000 mgr.y (mgr.24401) 53 : cluster [DBG] pgmap v34: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.482120+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.482120+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.498763+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.498763+0000 mon.a (mon.0) 886 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.548462+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.548462+0000 mon.a (mon.0) 887 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.554792+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.554792+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.619632+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:11.619632+0000 mon.c (mon.2) 50 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:12.146369+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:12.146369+0000 mon.a (mon.0) 889 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:12.162988+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:12.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:12 smithi050 bash[29234]: audit 2024-04-26T16:19:12.162988+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:13.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.896431278Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-04-26T16:19:13.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.929857589Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-04-26T16:19:13.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:12 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:12.963534574Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-04-26T16:19:13.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.014043909Z level=info msg="Executing migration" id="Update user table charset" 2024-04-26T16:19:13.133 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.05625546Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-04-26T16:19:13.133 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.09826169Z level=info msg="Executing migration" id="Add missing user data" 2024-04-26T16:19:13.133 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.140235797Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-04-26T16:19:13.384 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.182236714Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.20761667Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.249536052Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.291621356Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.316945394Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.342118032Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.367483018Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-04-26T16:19:13.385 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.392588145Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.434611284Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.476515763Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.518739373Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.544010343Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.585854813Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.611225047Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.636481542Z level=info msg="Executing migration" id="create temp_user v2" 2024-04-26T16:19:13.654 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.661762837Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.324284+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.324284+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.326877+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.326877+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.968955+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.968955+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.969066+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:13.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[23868]: cluster 2024-04-26T16:19:12.969066+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- 172.21.15.50:0/3249722355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 msgr2=0x7f4f3c109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- 172.21.15.50:0/3249722355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 0x7f4f3c109880 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c009990 tx=0x7f4f2c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- 172.21.15.50:0/3249722355 shutdown_connections 2024-04-26T16:19:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- 172.21.15.50:0/3249722355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06c510 0x7f4f3c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- 172.21.15.50:0/3249722355 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 0x7f4f3c109880 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- 172.21.15.50:0/3249722355 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06bb70 0x7f4f3c06bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- 172.21.15.50:0/3249722355 >> 172.21.15.50:0/3249722355 conn(0x7f4f3c0792f0 msgr2=0x7f4f3c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- 172.21.15.50:0/3249722355 shutdown_connections 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- 172.21.15.50:0/3249722355 wait complete. 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 librados: starting msgr at 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 librados: starting objecter 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 Processor -- start 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- start start 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 librados: setting wanted keys 2024-04-26T16:19:13.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 librados: calling monclient init 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06bb70 0x7f4f3c1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 0x7f4f3c199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 0x7f4f3c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c112ab0 con 0x7f4f3c06c510 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3c112930 con 0x7f4f3c06bb70 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.708+0000 7f4f42f37700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3c112c30 con 0x7f4f3c1094a0 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 0x7f4f3c199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 0x7f4f3c199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58362/0 (socket says 172.21.15.50:58362) 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 -- 172.21.15.50:0/1785883810 learned_addr learned my addr 172.21.15.50:0/1785883810 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f414d4700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 0x7f4f3c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:13.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f40cd3700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06bb70 0x7f4f3c1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 -- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 msgr2=0x7f4f3c19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 0x7f4f3c19d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 -- 172.21.15.50:0/1785883810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06bb70 msgr2=0x7f4f3c1990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06bb70 0x7f4f3c1990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 -- 172.21.15.50:0/1785883810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3c19e0e0 con 0x7f4f3c06c510 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f40cd3700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06bb70 0x7f4f3c1990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f3bfff700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 0x7f4f3c199610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000eb30 tx=0x7f4f3000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 10 client.?.objecter ms_handle_connect 0x7f4f3c06c510 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3000cc90 con 0x7f4f3c06c510 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4f3000ce30 con 0x7f4f3c06c510 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3c19e3d0 con 0x7f4f3c06c510 2024-04-26T16:19:13.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f30003d80 con 0x7f4f3c06c510 2024-04-26T16:19:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f42f37700 10 client.14571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3c1a5c50 con 0x7f4f3c06c510 2024-04-26T16:19:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f42f37700 1 librados: init done 2024-04-26T16:19:13.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f3c06a110 con 0x7f4f3c06c510 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4f30003f20 con 0x7f4f3c06c510 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4f24071f80 0x7f4f24074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(109..109 src has 57..109) ==== 7450+0+0 (secure 0 0 0) 0x7f4f30091470 con 0x7f4f3c06c510 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 10 client.14571.objecter ms_dispatch 0x7f4f3c001550 osd_map(109..109 src has 57..109) 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 3 client.14571.objecter handle_osd_map got epochs [109,109] > 0 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 3 client.14571.objecter handle_osd_map decoding full epoch 109 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f40cd3700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4f24071f80 0x7f4f24074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 20 client.14571.objecter dump_active .. 0 homeless 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f39ffb700 10 client.14571.objecter ms_handle_connect 0x7f4f24071f80 2024-04-26T16:19:13.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.712+0000 7f4f40cd3700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4f24071f80 0x7f4f24074440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00b2e0 tx=0x7f4f2c01a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:13.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.720+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4f3005d680 con 0x7f4f3c06c510 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.324284+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.324284+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.326877+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.326877+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.968955+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.968955+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:13.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.969066+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[21463]: cluster 2024-04-26T16:19:12.969066+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.324284+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.324284+0000 osd.1 (osd.1) 41 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.326877+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.326877+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.968955+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.968955+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.969066+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:13.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:13 smithi050 bash[29234]: cluster 2024-04-26T16:19:12.969066+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:13.906 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.687035559Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-04-26T16:19:13.906 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.728979675Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-04-26T16:19:13.906 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.754397772Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-04-26T16:19:13.906 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.779748086Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-04-26T16:19:13.906 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.804920667Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-04-26T16:19:13.906 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.830063176Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-04-26T16:19:13.907 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.872201422Z level=info msg="Executing migration" id="create star table" 2024-04-26T16:19:13.907 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.914374998Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-04-26T16:19:13.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.948+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f3c1a5f40 con 0x7f4f3c06c510 2024-04-26T16:19:13.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f39ffb700 1 -- 172.21.15.50:0/1785883810 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) ==== 74+0+23076 (secure 0 0 0) 0x7f4f30061330 con 0x7f4f3c06c510 2024-04-26T16:19:13.945 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:19:13.945 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":109,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","created":"2024-04-26T16:07:04.494213+0000","modified":"2024-04-26T16:19:13.479805+0000","last_up_change":"2024-04-26T16:16:46.862501+0000","last_in_change":"2024-04-26T16:16:20.945042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T16:12:30.809607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-26T16:17:27.029484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-26T16:17:29.124965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-26T16:17:30.749319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-26T16:17:31.064350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-26T16:17:33.192127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b46ea8aa-cd97-425b-90e0-1e70af46b8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6803","nonce":310950989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6805","nonce":310950989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6809","nonce":310950989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6807","nonce":310950989}]},"public_addr":"172.21.15.50:6803/310950989","cluster_addr":"172.21.15.50:6805/310950989","heartbeat_back_addr":"172.21.15.50:6809/310950989","heartbeat_front_addr":"172.21.15.50:6807/310950989","state":["exists","up"]},{"osd":1,"uuid":"7042f8e0-f5d0-48d2-9cd6-d9445e326eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6811","nonce":3516275849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6813","nonce":3516275849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6817","nonce":3516275849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6815","nonce":3516275849}]},"public_addr":"172.21.15.50:6811/3516275849","cluster_addr":"172.21.15.50:6813/3516275849","heartbeat_back_addr":"172.21.15.50:6817/3516275849","heartbeat_front_addr":"172.21.15.50:6815/3516275849","state":["exists","up"]},{"osd":2,"uuid":"33069daa-7fd8-4e9c-aafe-89030f84aad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6819","nonce":2835380668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6821","nonce":2835380668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6825","nonce":2835380668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6823","nonce":2835380668}]},"public_addr":"172.21.15.50:6819/2835380668","cluster_addr":"172.21.15.50:6821/2835380668","heartbeat_back_addr":"172.21.15.50:6825/2835380668","heartbeat_front_addr":"172.21.15.50:6823/2835380668","state":["exists","up"]},{"osd":3,"uuid":"43b6aab5-9b42-4f97-a224-b2872de5cb03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6827","nonce":2541695077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6829","nonce":2541695077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6833","nonce":2541695077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6831","nonce":2541695077}]},"public_addr":"172.21.15.50:6827/2541695077","cluster_addr":"172.21.15.50:6829/2541695077","heartbeat_back_addr":"172.21.15.50:6833/2541695077","heartbeat_front_addr":"172.21.15.50:6831/2541695077","state":["exists","up"]},{"osd":4,"uuid":"f7220491-e9bf-4188-b203-322e9908558c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6801","nonce":2207965503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6803","nonce":2207965503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6807","nonce":2207965503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6805","nonce":2207965503}]},"public_addr":"172.21.15.190:6801/2207965503","cluster_addr":"172.21.15.190:6803/2207965503","heartbeat_back_addr":"172.21.15.190:6807/2207965503","heartbeat_front_addr":"172.21.15.190:6805/2207965503","state":["exists","up"]},{"osd":5,"uuid":"ed5c0044-658b-4dfb-8c61-727de6bdd587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6809","nonce":3373187844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6811","nonce":3373187844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6815","nonce":3373187844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6813","nonce":3373187844}]},"public_addr":"172.21.15.190:6809/3373187844","cluster_addr":"172.21.15.190:6811/3373187844","heartbeat_back_addr":"172.21.15.190:6815/3373187844","heartbeat_front_addr":"172.21.15.190:6813/3373187844","state":["exists","up"]},{"osd":6,"uuid":"94111b7d-7887-4a3a-983b-394294735d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6817","nonce":2399532574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6819","nonce":2399532574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6823","nonce":2399532574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6821","nonce":2399532574}]},"public_addr":"172.21.15.190:6817/2399532574","cluster_addr":"172.21.15.190:6819/2399532574","heartbeat_back_addr":"172.21.15.190:6823/2399532574","heartbeat_front_addr":"172.21.15.190:6821/2399532574","state":["exists","up"]},{"osd":7,"uuid":"abbd95e3-07b4-4235-9101-a4dea0f35bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6825","nonce":2856617796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6827","nonce":2856617796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6831","nonce":2856617796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6829","nonce":2856617796}]},"public_addr":"172.21.15.190:6825/2856617796","cluster_addr":"172.21.15.190:6827/2856617796","heartbeat_back_addr":"172.21.15.190:6831/2856617796","heartbeat_front_addr":"172.21.15.190:6829/2856617796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:25.119394+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:54.180002+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:46.261293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:04.145753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:19:12.969072+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:19:10.031842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:38.974224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:00.973206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.9","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/2324264713":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/542146552":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3454723725":"2024-04-27T16:07:37.705234+0000","172.21.15.50:0/2548717623":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2903276422":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6801/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2979962403":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/71201882":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/3973398204":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/1172169336":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3444006067":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/577039843":"2024-04-27T16:18:25.317826+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 10 librados: watch_flush enter 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 10 librados: watch_flush exit 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 20 client.14571.objecter shutdown clearing up homeless session... 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 10 client.14571.objecter successfully canceled tick 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4f24071f80 msgr2=0x7f4f24074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4f24071f80 0x7f4f24074440 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00b2e0 tx=0x7f4f2c01a040 comp rx=0 tx=0).stop 2024-04-26T16:19:13.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 msgr2=0x7f4f3c199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 0x7f4f3c199610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4f3000eb30 tx=0x7f4f3000ee40 comp rx=0 tx=0).stop 2024-04-26T16:19:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 shutdown_connections 2024-04-26T16:19:13.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4f24071f80 0x7f4f24074440 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4f3c06bb70 0x7f4f3c1990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4f3c1094a0 0x7f4f3c19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 --2- 172.21.15.50:0/1785883810 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4f3c06c510 0x7f4f3c199610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 >> 172.21.15.50:0/1785883810 conn(0x7f4f3c0792f0 msgr2=0x7f4f3c1001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 shutdown_connections 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.952+0000 7f4f42f37700 1 -- 172.21.15.50:0/1785883810 wait complete. 2024-04-26T16:19:13.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:13.956+0000 7f4f42f37700 1 librados: shutdown 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.948033385Z level=info msg="Executing migration" id="create org table v1" 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:13 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:13.981626414Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.031904303Z level=info msg="Executing migration" id="create org_user table v1" 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.065510027Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.107500207Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.160823117Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-04-26T16:19:14.192 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.199921466Z level=info msg="Executing migration" id="Update org table charset" 2024-04-26T16:19:14.461 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.241809315Z level=info msg="Executing migration" id="Update org_user table charset" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.258855846Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.292555344Z level=info msg="Executing migration" id="create dashboard table" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.317849241Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.343081825Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.368352436Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.393900406Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.418856964Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.444060643Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-04-26T16:19:14.462 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.469357742Z level=info msg="Executing migration" id="create dashboard v2" 2024-04-26T16:19:14.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.362595+0000 mgr.y (mgr.24401) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:14.627 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:19:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.51154137Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.536885322Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.56200192Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.60399224Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.66277722Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.704537426Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.721608895Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-04-26T16:19:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.362595+0000 mgr.y (mgr.24401) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.362595+0000 mgr.y (mgr.24401) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.497495+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-04-26T16:19:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.497495+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-04-26T16:19:14.714 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.700593+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.b scrub starts 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.700593+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.b scrub starts 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.705653+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.b scrub ok 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:13.705653+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.b scrub ok 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: audit 2024-04-26T16:19:13.954123+0000 mon.a (mon.0) 892 : audit [DBG] from='client.? 172.21.15.50:0/1785883810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: audit 2024-04-26T16:19:13.954123+0000 mon.a (mon.0) 892 : audit [DBG] from='client.? 172.21.15.50:0/1785883810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.079771+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.079771+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.082057+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:19:14.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.082057+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:19:14.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.362595+0000 mgr.y (mgr.24401) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.497495+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.497495+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.700593+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.b scrub starts 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.700593+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.b scrub starts 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.705653+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.b scrub ok 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:13.705653+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.b scrub ok 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: audit 2024-04-26T16:19:13.954123+0000 mon.a (mon.0) 892 : audit [DBG] from='client.? 172.21.15.50:0/1785883810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: audit 2024-04-26T16:19:13.954123+0000 mon.a (mon.0) 892 : audit [DBG] from='client.? 172.21.15.50:0/1785883810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.079771+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.079771+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.082057+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:19:14.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.082057+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.362595+0000 mgr.y (mgr.24401) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.362595+0000 mgr.y (mgr.24401) 54 : cluster [DBG] pgmap v35: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.497495+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.497495+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.700593+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.b scrub starts 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.700593+0000 osd.7 (osd.7) 39 : cluster [DBG] 2.b scrub starts 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.705653+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.b scrub ok 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:13.705653+0000 osd.7 (osd.7) 40 : cluster [DBG] 2.b scrub ok 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: audit 2024-04-26T16:19:13.954123+0000 mon.a (mon.0) 892 : audit [DBG] from='client.? 172.21.15.50:0/1785883810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: audit 2024-04-26T16:19:13.954123+0000 mon.a (mon.0) 892 : audit [DBG] from='client.? 172.21.15.50:0/1785883810' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:14.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.079771+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:19:14.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.079771+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:19:14.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.082057+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:19:14.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:14 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.082057+0000 osd.4 (osd.4) 40 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:19:14.934 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T16:19:14.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T16:19:14.974 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.738460404Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.755351049Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.772303975Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.797731145Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.823029992Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.848256526Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.890156255Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.915443805Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.957435194Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-04-26T16:19:14.975 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:14 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:14.982738644Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-04-26T16:19:15.268 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.024671414Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-04-26T16:19:15.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.066776726Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-04-26T16:19:15.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.108827504Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-04-26T16:19:15.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.150822029Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-04-26T16:19:15.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.175900034Z level=info msg="Executing migration" id="Update dashboard title length" 2024-04-26T16:19:15.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.201099464Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-26T16:19:15.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.276588841Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-04-26T16:19:15.529 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.301837903Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-26T16:19:15.529 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.327111642Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-04-26T16:19:15.529 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.352249828Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-26T16:19:15.529 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.377615227Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-26T16:19:15.530 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.402895449Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-04-26T16:19:15.530 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.428116099Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-04-26T16:19:15.530 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.469967248Z level=info msg="Executing migration" id="Add check_sum column" 2024-04-26T16:19:15.530 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.51207505Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-04-26T16:19:15.530 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.537384562Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.579354738Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.604660328Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.629939343Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.6551784Z level=info msg="Executing migration" id="create data_source table" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.697305319Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.739498412Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-04-26T16:19:15.782 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.764716433Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-04-26T16:19:15.783 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.789873831Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.8152207Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.840531319Z level=info msg="Executing migration" id="create data_source table v2" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.882459242Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.907728594Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.949771518Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:15 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:15.991729742Z level=info msg="Executing migration" id="Add column with_credentials" 2024-04-26T16:19:16.051 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.016996584Z level=info msg="Executing migration" id="Add secure json data column" 2024-04-26T16:19:16.052 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.059112073Z level=info msg="Executing migration" id="Update data_source table charset" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.084246053Z level=info msg="Executing migration" id="Update initial version to 1" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.109626713Z level=info msg="Executing migration" id="Add read_only data column" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.134781127Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.176782351Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.218831265Z level=info msg="Executing migration" id="Add uid column" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.260820563Z level=info msg="Executing migration" id="Update uid value" 2024-04-26T16:19:16.320 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.302837661Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-04-26T16:19:16.321 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.328063032Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-04-26T16:19:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:19:16.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.378489609Z level=info msg="Executing migration" id="create api_key table" 2024-04-26T16:19:16.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.403778932Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.429615241Z level=info msg="Executing migration" id="add index api_key.key" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.454306087Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.496533621Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.521686678Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.546868664Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.572081729Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-04-26T16:19:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.630581+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:19:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.630581+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:19:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.634264+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:19:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:14.634264+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.227056+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.227056+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.229481+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.229481+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.363337+0000 mgr.y (mgr.24401) 55 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.363337+0000 mgr.y (mgr.24401) 55 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: audit 2024-04-26T16:19:15.480786+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: audit 2024-04-26T16:19:15.480786+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.677372+0000 osd.7 (osd.7) 41 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.677372+0000 osd.7 (osd.7) 41 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.680309+0000 osd.7 (osd.7) 42 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:19:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.680309+0000 osd.7 (osd.7) 42 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:19:16.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.630581+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:19:16.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.630581+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:19:16.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.634264+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:14.634264+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.227056+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.227056+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.229481+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.229481+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.363337+0000 mgr.y (mgr.24401) 55 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.363337+0000 mgr.y (mgr.24401) 55 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: audit 2024-04-26T16:19:15.480786+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: audit 2024-04-26T16:19:15.480786+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.677372+0000 osd.7 (osd.7) 41 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.677372+0000 osd.7 (osd.7) 41 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.680309+0000 osd.7 (osd.7) 42 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:19:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.680309+0000 osd.7 (osd.7) 42 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.614021052Z level=info msg="Executing migration" id="create api_key table v2" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.656326289Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.681633238Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.723691921Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.765674067Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.790878555Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.816075584Z level=info msg="Executing migration" id="Update api_key table charset" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.841336718Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.883212569Z level=info msg="Executing migration" id="Add service account foreign key" 2024-04-26T16:19:16.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.630581+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:19:16.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.630581+0000 osd.0 (osd.0) 39 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:19:16.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.634264+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:14.634264+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.227056+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.227056+0000 osd.1 (osd.1) 43 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.229481+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.229481+0000 osd.1 (osd.1) 44 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.363337+0000 mgr.y (mgr.24401) 55 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.363337+0000 mgr.y (mgr.24401) 55 : cluster [DBG] pgmap v37: 132 pgs: 132 active+clean; 583 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: audit 2024-04-26T16:19:15.480786+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: audit 2024-04-26T16:19:15.480786+0000 mon.a (mon.0) 893 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.677372+0000 osd.7 (osd.7) 41 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.677372+0000 osd.7 (osd.7) 41 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.680309+0000 osd.7 (osd.7) 42 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:19:16.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:16 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.680309+0000 osd.7 (osd.7) 42 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:19:17.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.908485442Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-04-26T16:19:17.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.950460621Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-04-26T16:19:17.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.975859998Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-04-26T16:19:17.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:16 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:16.992706852Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-04-26T16:19:17.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.076511282Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-04-26T16:19:17.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.118788551Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.160702008Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.202895506Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.2280875Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.278171736Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.345418488Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.37065144Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-04-26T16:19:17.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.395861555Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-04-26T16:19:17.699 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.421110361Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-26T16:19:17.699 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.446379307Z level=info msg="Executing migration" id="create quota table v1" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.488234423Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.513752578Z level=info msg="Executing migration" id="Update quota table charset" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.53879799Z level=info msg="Executing migration" id="create plugin_setting table" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.589475509Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.614804497Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.640021338Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.665272212Z level=info msg="Executing migration" id="create session table" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.707367503Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-04-26T16:19:17.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.671432+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:19:17.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.671432+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.673976+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:15.673976+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.146333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.146333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.148761+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.148761+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.242990+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.242990+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.245369+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: cluster 2024-04-26T16:19:16.245369+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: audit 2024-04-26T16:19:16.379434+0000 mgr.y (mgr.24401) 56 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:17.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[23868]: audit 2024-04-26T16:19:16.379434+0000 mgr.y (mgr.24401) 56 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.671432+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.671432+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.673976+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:15.673976+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.146333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.146333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.148761+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.148761+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:19:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.242990+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.242990+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.245369+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: cluster 2024-04-26T16:19:16.245369+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: audit 2024-04-26T16:19:16.379434+0000 mgr.y (mgr.24401) 56 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[21463]: audit 2024-04-26T16:19:16.379434+0000 mgr.y (mgr.24401) 56 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.671432+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.671432+0000 osd.0 (osd.0) 41 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.673976+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:15.673976+0000 osd.0 (osd.0) 42 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.146333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.146333+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:19:17.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.148761+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.148761+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.242990+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.242990+0000 osd.1 (osd.1) 45 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.245369+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: cluster 2024-04-26T16:19:16.245369+0000 osd.1 (osd.1) 46 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: audit 2024-04-26T16:19:16.379434+0000 mgr.y (mgr.24401) 56 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:17 smithi050 bash[29234]: audit 2024-04-26T16:19:16.379434+0000 mgr.y (mgr.24401) 56 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:17.968 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.732591683Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-04-26T16:19:17.968 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.757916218Z level=info msg="Executing migration" id="create playlist table v2" 2024-04-26T16:19:17.968 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.783220393Z level=info msg="Executing migration" id="create playlist item table v2" 2024-04-26T16:19:17.968 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.825187859Z level=info msg="Executing migration" id="Update playlist table charset" 2024-04-26T16:19:17.968 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.867287413Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-04-26T16:19:17.968 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.892613902Z level=info msg="Executing migration" id="drop preferences table v2" 2024-04-26T16:19:17.969 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.93467132Z level=info msg="Executing migration" id="drop preferences table v3" 2024-04-26T16:19:17.969 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:17 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:17.976789542Z level=info msg="Executing migration" id="create preferences table v3" 2024-04-26T16:19:18.254 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.018884487Z level=info msg="Executing migration" id="Update preferences table charset" 2024-04-26T16:19:18.254 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.044220926Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-04-26T16:19:18.254 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.094647589Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-04-26T16:19:18.254 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.128207777Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-04-26T16:19:18.254 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.161895918Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-04-26T16:19:18.255 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.212083056Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-04-26T16:19:18.255 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.262312264Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.304424698Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.329723085Z level=info msg="Executing migration" id="create alert table v1" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.355175517Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.380369159Z level=info msg="Executing migration" id="add index alert state" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.405693138Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.430813732Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.456476385Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.481731189Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-26T16:19:18.516 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.523852932Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.364808+0000 mgr.y (mgr.24401) 57 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.364808+0000 mgr.y (mgr.24401) 57 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.581721+0000 osd.0 (osd.0) 43 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.581721+0000 osd.0 (osd.0) 43 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.582828+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.582828+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.688257+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.688257+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.690563+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:19:18.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: cluster 2024-04-26T16:19:17.690563+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.064578+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.064578+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.076364+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.076364+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.269490+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.269490+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.286072+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.286072+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.290471+0000 mon.c (mon.2) 51 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.290471+0000 mon.c (mon.2) 51 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.292713+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.292713+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.303714+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[23868]: audit 2024-04-26T16:19:18.303714+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.776 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.557438175Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-04-26T16:19:18.776 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.599411106Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-26T16:19:18.777 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.624727055Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-04-26T16:19:18.777 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.649879191Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-04-26T16:19:18.777 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.691875111Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-04-26T16:19:18.777 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.73380483Z level=info msg="Executing migration" id="Add column is_default" 2024-04-26T16:19:18.777 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.759246318Z level=info msg="Executing migration" id="Add column frequency" 2024-04-26T16:19:18.777 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.784441924Z level=info msg="Executing migration" id="Add column send_reminder" 2024-04-26T16:19:18.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.364808+0000 mgr.y (mgr.24401) 57 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:18.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.364808+0000 mgr.y (mgr.24401) 57 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.581721+0000 osd.0 (osd.0) 43 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.581721+0000 osd.0 (osd.0) 43 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.582828+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.582828+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.688257+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.688257+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.690563+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: cluster 2024-04-26T16:19:17.690563+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.064578+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.064578+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.076364+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.076364+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.269490+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.269490+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.840 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.286072+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.286072+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.290471+0000 mon.c (mon.2) 51 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.290471+0000 mon.c (mon.2) 51 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.292713+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.292713+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.303714+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[21463]: audit 2024-04-26T16:19:18.303714+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.364808+0000 mgr.y (mgr.24401) 57 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.364808+0000 mgr.y (mgr.24401) 57 : cluster [DBG] pgmap v38: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.581721+0000 osd.0 (osd.0) 43 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.581721+0000 osd.0 (osd.0) 43 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.582828+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:19:18.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.582828+0000 osd.0 (osd.0) 44 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.688257+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.688257+0000 osd.7 (osd.7) 43 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.690563+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: cluster 2024-04-26T16:19:17.690563+0000 osd.7 (osd.7) 44 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.064578+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.064578+0000 mon.a (mon.0) 894 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.076364+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.076364+0000 mon.a (mon.0) 895 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.269490+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.269490+0000 mon.a (mon.0) 896 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.286072+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.286072+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.290471+0000 mon.c (mon.2) 51 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:18.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.290471+0000 mon.c (mon.2) 51 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:18.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.292713+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:18.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.292713+0000 mon.c (mon.2) 52 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:18.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.303714+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:18.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:18 smithi050 bash[29234]: audit 2024-04-26T16:19:18.303714+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:19.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.809676302Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-04-26T16:19:19.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.834977435Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-04-26T16:19:19.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.860183292Z level=info msg="Executing migration" id="Update alert table charset" 2024-04-26T16:19:19.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.902131001Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-04-26T16:19:19.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.944171333Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-04-26T16:19:19.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:18 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:18.986311269Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-26T16:19:19.072 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.028276775Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.10357795Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.13734384Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.162423676Z level=info msg="Executing migration" id="Add for to alert table" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.204541506Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.229767882Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.271791288Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.297120597Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.322264436Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-04-26T16:19:19.365 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.347556814Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-04-26T16:19:19.366 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.372813232Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-04-26T16:19:19.478 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:19.617 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.398318668Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-04-26T16:19:19.617 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.440335856Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-04-26T16:19:19.618 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.482442596Z level=info msg="Executing migration" id="create annotation table v5" 2024-04-26T16:19:19.618 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.524625958Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-04-26T16:19:19.618 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.549740154Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-04-26T16:19:19.618 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.574988942Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-04-26T16:19:19.618 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.60017064Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cephadm 2024-04-26T16:19:18.383802+0000 mgr.y (mgr.24401) 58 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cephadm 2024-04-26T16:19:18.383802+0000 mgr.y (mgr.24401) 58 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cephadm 2024-04-26T16:19:18.393295+0000 mgr.y (mgr.24401) 59 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi050 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cephadm 2024-04-26T16:19:18.393295+0000 mgr.y (mgr.24401) 59 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi050 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cluster 2024-04-26T16:19:18.948481+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cluster 2024-04-26T16:19:18.948481+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:19:19.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cluster 2024-04-26T16:19:18.953700+0000 osd.3 (osd.3) 40 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:19:19.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[23868]: cluster 2024-04-26T16:19:18.953700+0000 osd.3 (osd.3) 40 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cephadm 2024-04-26T16:19:18.383802+0000 mgr.y (mgr.24401) 58 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cephadm 2024-04-26T16:19:18.383802+0000 mgr.y (mgr.24401) 58 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cephadm 2024-04-26T16:19:18.393295+0000 mgr.y (mgr.24401) 59 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi050 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cephadm 2024-04-26T16:19:18.393295+0000 mgr.y (mgr.24401) 59 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi050 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cluster 2024-04-26T16:19:18.948481+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cluster 2024-04-26T16:19:18.948481+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cluster 2024-04-26T16:19:18.953700+0000 osd.3 (osd.3) 40 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[21463]: cluster 2024-04-26T16:19:18.953700+0000 osd.3 (osd.3) 40 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:19:19.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cephadm 2024-04-26T16:19:18.383802+0000 mgr.y (mgr.24401) 58 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cephadm 2024-04-26T16:19:18.383802+0000 mgr.y (mgr.24401) 58 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cephadm 2024-04-26T16:19:18.393295+0000 mgr.y (mgr.24401) 59 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi050 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cephadm 2024-04-26T16:19:18.393295+0000 mgr.y (mgr.24401) 59 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi050 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cluster 2024-04-26T16:19:18.948481+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cluster 2024-04-26T16:19:18.948481+0000 osd.3 (osd.3) 39 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cluster 2024-04-26T16:19:18.953700+0000 osd.3 (osd.3) 40 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:19:19.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:19 smithi050 bash[29234]: cluster 2024-04-26T16:19:18.953700+0000 osd.3 (osd.3) 40 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:19:19.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.625459537Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.667378135Z level=info msg="Executing migration" id="Update annotation table charset" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.709465919Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.734714455Z level=info msg="Executing migration" id="Drop category_id index" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.75982365Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.785182183Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.810289009Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.835662481Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.860830606Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-26T16:19:19.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.89445048Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-04-26T16:19:20.146 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.91966122Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-26T16:19:20.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:19 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:19.961631626Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-04-26T16:19:20.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.003557014Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-04-26T16:19:20.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.02882938Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-04-26T16:19:20.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.070743148Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-04-26T16:19:20.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.112666929Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-04-26T16:19:20.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.154723882Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-04-26T16:19:20.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.179983882Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.221954501Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.263816707Z level=info msg="Executing migration" id="Add epoch_end column" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.289109536Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.314313661Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.339591268Z level=info msg="Executing migration" id="Move region to single row" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.36504281Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.390307663Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-26T16:19:20.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.415534214Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-26T16:19:20.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.440775708Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-26T16:19:20.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.465909925Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-26T16:19:20.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.491270629Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-04-26T16:19:20.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.533318759Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-04-26T16:19:20.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.558551347Z level=info msg="Executing migration" id="create test_data table" 2024-04-26T16:19:20.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.583848566Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-04-26T16:19:20.686 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.608940788Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-04-26T16:19:20.686 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.651027877Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-26T16:19:20.686 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.693084551Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.240949+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.240949+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.243247+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.243247+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.365541+0000 mgr.y (mgr.24401) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.365541+0000 mgr.y (mgr.24401) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.935532+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.935532+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.939890+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:19:20.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[23868]: cluster 2024-04-26T16:19:19.939890+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:19:20.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.240949+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.240949+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.243247+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.243247+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.365541+0000 mgr.y (mgr.24401) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.365541+0000 mgr.y (mgr.24401) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.935532+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.935532+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.939890+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[21463]: cluster 2024-04-26T16:19:19.939890+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.240949+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.240949+0000 osd.1 (osd.1) 47 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.243247+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:19:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.243247+0000 osd.1 (osd.1) 48 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:19:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.365541+0000 mgr.y (mgr.24401) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.365541+0000 mgr.y (mgr.24401) 60 : cluster [DBG] pgmap v39: 132 pgs: 132 active+clean; 583 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.935532+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:19:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.935532+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:19:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.939890+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:19:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:20 smithi050 bash[29234]: cluster 2024-04-26T16:19:19.939890+0000 osd.3 (osd.3) 42 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.734900944Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.760253588Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.785544994Z level=info msg="Executing migration" id="create team table" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.810776059Z level=info msg="Executing migration" id="add index team.org_id" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.836006457Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.877947827Z level=info msg="Executing migration" id="create team member table" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.903218653Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-04-26T16:19:20.938 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.945662683Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-04-26T16:19:21.189 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:20 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:20.987088423Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.029258272Z level=info msg="Executing migration" id="Add column email to team table" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.054488582Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.079768753Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.105023228Z level=info msg="Executing migration" id="create dashboard acl table" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.147023899Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.172294314Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-26T16:19:21.190 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.19753177Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.222751066Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.264696575Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.281880673Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.298826807Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.315741295Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.332620347Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-04-26T16:19:21.451 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.366189041Z level=info msg="Executing migration" id="create tag table" 2024-04-26T16:19:21.452 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.38332035Z level=info msg="Executing migration" id="add index tag.key_value" 2024-04-26T16:19:21.452 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.408471421Z level=info msg="Executing migration" id="create login attempt table" 2024-04-26T16:19:21.452 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.433730619Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-04-26T16:19:21.452 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.458980131Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-04-26T16:19:21.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.633051+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:19:21.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.633051+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:19:21.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.635359+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:19:21.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.635359+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:19:21.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.980817+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-04-26T16:19:21.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.980817+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.984804+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.984804+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.633051+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.633051+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.635359+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.635359+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.980817+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.980817+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.984804+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-04-26T16:19:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:21 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.984804+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-04-26T16:19:21.703 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.484199367Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-26T16:19:21.703 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.53474001Z level=info msg="Executing migration" id="create login_attempt v2" 2024-04-26T16:19:21.703 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.559877845Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-04-26T16:19:21.703 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.585163069Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-04-26T16:19:21.703 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.643708632Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-04-26T16:19:21.703 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.668903489Z level=info msg="Executing migration" id="create user auth table" 2024-04-26T16:19:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.633051+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:19:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.633051+0000 osd.7 (osd.7) 45 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:19:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.635359+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:19:21.703 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.635359+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:19:21.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.980817+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-04-26T16:19:21.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.980817+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.b scrub starts 2024-04-26T16:19:21.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.984804+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-04-26T16:19:21.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.984804+0000 osd.3 (osd.3) 44 : cluster [DBG] 3.b scrub ok 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.710902927Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.736135555Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.761442543Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.786606061Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.811856399Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.837124471Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.853972354Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.887616438Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-04-26T16:19:21.981 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.904563782Z level=info msg="Executing migration" id="create server_lock table" 2024-04-26T16:19:21.982 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.946518187Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-04-26T16:19:21.982 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:21 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:21.988546158Z level=info msg="Executing migration" id="create user auth token table" 2024-04-26T16:19:22.257 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.013759892Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-04-26T16:19:22.257 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.055752372Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-04-26T16:19:22.257 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.08094241Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-04-26T16:19:22.257 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.12283615Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-04-26T16:19:22.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.265303205Z level=info msg="Executing migration" id="create cache_data table" 2024-04-26T16:19:22.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.307206138Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-04-26T16:19:22.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.357569102Z level=info msg="Executing migration" id="create short_url table v1" 2024-04-26T16:19:22.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.466173589Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-04-26T16:19:22.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.656222+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:19:22.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.656222+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:19:22.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.658678+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:19:22.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:20.658678+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:19:22.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:21.366513+0000 mgr.y (mgr.24401) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:22.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:21.366513+0000 mgr.y (mgr.24401) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:22.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:21.964235+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:19:22.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:21.964235+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:19:22.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:21.966823+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:19:22.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[23868]: cluster 2024-04-26T16:19:21.966823+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:19:22.853 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.600004474Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-04-26T16:19:22.853 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.6419053Z level=info msg="Executing migration" id="delete alert_definition table" 2024-04-26T16:19:22.853 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.675562094Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-04-26T16:19:22.853 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.717541434Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-04-26T16:19:22.853 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.742680934Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-04-26T16:19:22.853 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.76797622Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-04-26T16:19:22.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.793213337Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-04-26T16:19:22.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.818463724Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-04-26T16:19:22.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.860607264Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-04-26T16:19:22.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.656222+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.656222+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.658678+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:20.658678+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:21.366513+0000 mgr.y (mgr.24401) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:21.366513+0000 mgr.y (mgr.24401) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:21.964235+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:21.964235+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:21.966823+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[21463]: cluster 2024-04-26T16:19:21.966823+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:19:22.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.656222+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.656222+0000 osd.0 (osd.0) 45 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.658678+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:20.658678+0000 osd.0 (osd.0) 46 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:21.366513+0000 mgr.y (mgr.24401) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:21.366513+0000 mgr.y (mgr.24401) 61 : cluster [DBG] pgmap v40: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:21.964235+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:21.964235+0000 osd.3 (osd.3) 45 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:21.966823+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:19:22.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:22 smithi050 bash[29234]: cluster 2024-04-26T16:19:21.966823+0000 osd.3 (osd.3) 46 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:19:23.121 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.922191184Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-04-26T16:19:23.122 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.969500684Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-04-26T16:19:23.122 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:22 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:22.994777284Z level=info msg="Executing migration" id="drop alert_definition table" 2024-04-26T16:19:23.122 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.020040805Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-04-26T16:19:23.122 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.061882907Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-04-26T16:19:23.122 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.103921985Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-26T16:19:23.122 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.129261205Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.171536312Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.196518373Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.239106372Z level=info msg="Executing migration" id="create alert_instance table" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.264029905Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.289241634Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.314476467Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-04-26T16:19:23.391 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.33969803Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-26T16:19:23.392 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.364934235Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-04-26T16:19:23.392 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.39885408Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-04-26T16:19:23.684 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.062419+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:19:23.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.062419+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:19:23.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.064626+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:19:23.685 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.064626+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:19:23.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.440765691Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-04-26T16:19:23.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.482701249Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-26T16:19:23.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.524666424Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-04-26T16:19:23.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.575061243Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-04-26T16:19:23.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.608715141Z level=info msg="Executing migration" id="create alert_rule table" 2024-04-26T16:19:23.685 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.642313044Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-04-26T16:19:23.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.062419+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:19:23.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.062419+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:19:23.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.064626+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:19:23.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.064626+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:19:23.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.062419+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:19:23.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.062419+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:19:23.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.064626+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:19:23.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:23 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.064626+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.692594333Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.734644345Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.759794932Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.784997509Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.810313396Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.835570801Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.87745631Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.902743454Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-26T16:19:23.962 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.928009286Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-04-26T16:19:23.963 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.969869555Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:23 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:23.995297922Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.020546329Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.062537338Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.104523Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.129828175Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.171762669Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-26T16:19:24.248 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.213846383Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-26T16:19:24.249 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.256023783Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-26T16:19:24.512 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.281151873Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.306410243Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.33163915Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.373838274Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.399197921Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.424457054Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.44971787Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.466703327Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.483570255Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-26T16:19:24.513 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.500499007Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/2563551569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31581094a0 msgr2=0x7f3158109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- 172.21.15.50:0/2563551569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31581094a0 0x7f3158109880 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3148009f90 tx=0x7f314800b330 comp rx=0 tx=0).stop 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/2563551569 shutdown_connections 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- 172.21.15.50:0/2563551569 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f315806bad0 0x7f31581102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- 172.21.15.50:0/2563551569 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f31581094a0 0x7f3158109880 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- 172.21.15.50:0/2563551569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f315806b130 0x7f315806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/2563551569 >> 172.21.15.50:0/2563551569 conn(0x7f31580792d0 msgr2=0x7f31580796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:24.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/2563551569 shutdown_connections 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/2563551569 wait complete. 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 librados: starting msgr at 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 librados: starting objecter 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 Processor -- start 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- start start 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 librados: setting wanted keys 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 librados: calling monclient init 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f315806b130 0x7f31581991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 0x7f3158199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31581094a0 0x7f315819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:24.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3158112a30 con 0x7f315806bad0 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f31581128b0 con 0x7f315806b130 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3158112bb0 con 0x7f31581094a0 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 0x7f3158199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315d90f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31581094a0 0x7f315819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 0x7f3158199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47676/0 (socket says 172.21.15.50:47676) 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 -- 172.21.15.50:0/4275665686 learned_addr learned my addr 172.21.15.50:0/4275665686 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315d10e700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f315806b130 0x7f31581991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 -- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31581094a0 msgr2=0x7f315819da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31581094a0 0x7f315819da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 -- 172.21.15.50:0/4275665686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f315806b130 msgr2=0x7f31581991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f315806b130 0x7f31581991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 -- 172.21.15.50:0/4275665686 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315819e1b0 con 0x7f315806bad0 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315d90f700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31581094a0 0x7f315819da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:19:24.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315c90d700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 0x7f3158199730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f314c00d950 tx=0x7f314c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f31567fc700 10 client.?.objecter ms_handle_connect 0x7f315806bad0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f31567fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c016070 con 0x7f315806bad0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f314c009da0 con 0x7f315806bad0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315819e4a0 con 0x7f315806bad0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c010e60 con 0x7f315806bad0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 10 client.14577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3158102320 con 0x7f315806bad0 2024-04-26T16:19:24.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 librados: init done 2024-04-26T16:19:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.692+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f315806a110 con 0x7f315806bad0 2024-04-26T16:19:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f314c019070 con 0x7f315806bad0 2024-04-26T16:19:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3140076390 0x7f3140078850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(110..110 src has 57..110) ==== 7450+0+0 (secure 0 0 0) 0x7f314c090e00 con 0x7f315806bad0 2024-04-26T16:19:24.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 10 client.14577.objecter ms_dispatch 0x7f3158001550 osd_map(110..110 src has 57..110) 2024-04-26T16:19:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 3 client.14577.objecter handle_osd_map got epochs [110,110] > 0 2024-04-26T16:19:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 3 client.14577.objecter handle_osd_map decoding full epoch 110 2024-04-26T16:19:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f315d10e700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3140076390 0x7f3140078850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 20 client.14577.objecter dump_active .. 0 homeless 2024-04-26T16:19:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f315d10e700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3140076390 0x7f3140078850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3148009790 tx=0x7f3148009720 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:24.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.696+0000 7f31567fc700 10 client.14577.objecter ms_handle_connect 0x7f3140076390 2024-04-26T16:19:24.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.700+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f314c05d010 con 0x7f315806bad0 2024-04-26T16:19:24.770 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.542748686Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-04-26T16:19:24.770 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.567951247Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.593227216Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.618441831Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.660527453Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.685676786Z level=info msg="Executing migration" id="create provenance_type table" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.727605834Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.753007536Z level=info msg="Executing migration" id="create alert_image table" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.778181331Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-04-26T16:19:24.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.367220+0000 mgr.y (mgr.24401) 62 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:24.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.367220+0000 mgr.y (mgr.24401) 62 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:24.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.664196+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:19:24.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.664196+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.666495+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.666495+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.703312+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.703312+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.706144+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.706144+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.949666+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.949666+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.952290+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:19:24.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[23868]: cluster 2024-04-26T16:19:23.952290+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:19:24.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:19:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:19:24.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.367220+0000 mgr.y (mgr.24401) 62 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:24.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.367220+0000 mgr.y (mgr.24401) 62 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:24.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.664196+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:19:24.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.664196+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:19:24.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.666495+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.666495+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.703312+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.703312+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.706144+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.706144+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.949666+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.949666+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.952290+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[21463]: cluster 2024-04-26T16:19:23.952290+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.367220+0000 mgr.y (mgr.24401) 62 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.367220+0000 mgr.y (mgr.24401) 62 : cluster [DBG] pgmap v41: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:24.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.664196+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.664196+0000 osd.7 (osd.7) 47 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.666495+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.666495+0000 osd.7 (osd.7) 48 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.703312+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.703312+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.706144+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.706144+0000 osd.5 (osd.5) 44 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.949666+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.949666+0000 osd.3 (osd.3) 47 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.952290+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:19:24.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:24 smithi050 bash[29234]: cluster 2024-04-26T16:19:23.952290+0000 osd.3 (osd.3) 48 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:19:24.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.928+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3158102580 con 0x7f315806bad0 2024-04-26T16:19:24.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.932+0000 7f31567fc700 1 -- 172.21.15.50:0/4275665686 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) ==== 74+0+23076 (secure 0 0 0) 0x7f314c060cc0 con 0x7f315806bad0 2024-04-26T16:19:24.927 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:19:24.927 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":110,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","created":"2024-04-26T16:07:04.494213+0000","modified":"2024-04-26T16:19:24.511943+0000","last_up_change":"2024-04-26T16:16:46.862501+0000","last_in_change":"2024-04-26T16:16:20.945042+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T16:12:30.809607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-26T16:17:27.029484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-26T16:17:29.124965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-26T16:17:30.749319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-26T16:17:31.064350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-26T16:17:33.192127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b46ea8aa-cd97-425b-90e0-1e70af46b8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6803","nonce":310950989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6805","nonce":310950989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6809","nonce":310950989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6807","nonce":310950989}]},"public_addr":"172.21.15.50:6803/310950989","cluster_addr":"172.21.15.50:6805/310950989","heartbeat_back_addr":"172.21.15.50:6809/310950989","heartbeat_front_addr":"172.21.15.50:6807/310950989","state":["exists","up"]},{"osd":1,"uuid":"7042f8e0-f5d0-48d2-9cd6-d9445e326eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6811","nonce":3516275849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6813","nonce":3516275849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6817","nonce":3516275849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6815","nonce":3516275849}]},"public_addr":"172.21.15.50:6811/3516275849","cluster_addr":"172.21.15.50:6813/3516275849","heartbeat_back_addr":"172.21.15.50:6817/3516275849","heartbeat_front_addr":"172.21.15.50:6815/3516275849","state":["exists","up"]},{"osd":2,"uuid":"33069daa-7fd8-4e9c-aafe-89030f84aad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6819","nonce":2835380668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6821","nonce":2835380668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6825","nonce":2835380668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6823","nonce":2835380668}]},"public_addr":"172.21.15.50:6819/2835380668","cluster_addr":"172.21.15.50:6821/2835380668","heartbeat_back_addr":"172.21.15.50:6825/2835380668","heartbeat_front_addr":"172.21.15.50:6823/2835380668","state":["exists","up"]},{"osd":3,"uuid":"43b6aab5-9b42-4f97-a224-b2872de5cb03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6827","nonce":2541695077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6829","nonce":2541695077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6833","nonce":2541695077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6831","nonce":2541695077}]},"public_addr":"172.21.15.50:6827/2541695077","cluster_addr":"172.21.15.50:6829/2541695077","heartbeat_back_addr":"172.21.15.50:6833/2541695077","heartbeat_front_addr":"172.21.15.50:6831/2541695077","state":["exists","up"]},{"osd":4,"uuid":"f7220491-e9bf-4188-b203-322e9908558c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6801","nonce":2207965503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6803","nonce":2207965503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6807","nonce":2207965503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6805","nonce":2207965503}]},"public_addr":"172.21.15.190:6801/2207965503","cluster_addr":"172.21.15.190:6803/2207965503","heartbeat_back_addr":"172.21.15.190:6807/2207965503","heartbeat_front_addr":"172.21.15.190:6805/2207965503","state":["exists","up"]},{"osd":5,"uuid":"ed5c0044-658b-4dfb-8c61-727de6bdd587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6809","nonce":3373187844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6811","nonce":3373187844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6815","nonce":3373187844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6813","nonce":3373187844}]},"public_addr":"172.21.15.190:6809/3373187844","cluster_addr":"172.21.15.190:6811/3373187844","heartbeat_back_addr":"172.21.15.190:6815/3373187844","heartbeat_front_addr":"172.21.15.190:6813/3373187844","state":["exists","up"]},{"osd":6,"uuid":"94111b7d-7887-4a3a-983b-394294735d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6817","nonce":2399532574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6819","nonce":2399532574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6823","nonce":2399532574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2399532574},{"type":"v1","addr":"172.21.15.190:6821","nonce":2399532574}]},"public_addr":"172.21.15.190:6817/2399532574","cluster_addr":"172.21.15.190:6819/2399532574","heartbeat_back_addr":"172.21.15.190:6823/2399532574","heartbeat_front_addr":"172.21.15.190:6821/2399532574","state":["exists","up"]},{"osd":7,"uuid":"abbd95e3-07b4-4235-9101-a4dea0f35bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6825","nonce":2856617796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6827","nonce":2856617796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6831","nonce":2856617796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6829","nonce":2856617796}]},"public_addr":"172.21.15.190:6825/2856617796","cluster_addr":"172.21.15.190:6827/2856617796","heartbeat_back_addr":"172.21.15.190:6831/2856617796","heartbeat_front_addr":"172.21.15.190:6829/2856617796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:25.119394+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:54.180002+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:46.261293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:04.145753+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:19:12.969072+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:19:10.031842+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:18:38.974224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:19:24.196501+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.9","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/2324264713":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/542146552":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3454723725":"2024-04-27T16:07:37.705234+0000","172.21.15.50:0/2548717623":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2903276422":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6801/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2979962403":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/71201882":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/3973398204":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/1172169336":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3444006067":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/577039843":"2024-04-27T16:18:25.317826+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T16:19:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 10 librados: watch_flush enter 2024-04-26T16:19:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 10 librados: watch_flush exit 2024-04-26T16:19:24.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 20 client.14577.objecter shutdown clearing up homeless session... 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 10 client.14577.objecter successfully canceled tick 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3140076390 msgr2=0x7f3140078850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3140076390 0x7f3140078850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3148009790 tx=0x7f3148009720 comp rx=0 tx=0).stop 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 msgr2=0x7f3158199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 0x7f3158199730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f314c00d950 tx=0x7f314c00dc60 comp rx=0 tx=0).stop 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 shutdown_connections 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3140076390 0x7f3140078850 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f315806b130 0x7f31581991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f31581094a0 0x7f315819da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 --2- 172.21.15.50:0/4275665686 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f315806bad0 0x7f3158199730 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 >> 172.21.15.50:0/4275665686 conn(0x7f31580792d0 msgr2=0x7f3158100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 shutdown_connections 2024-04-26T16:19:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 -- 172.21.15.50:0/4275665686 wait complete. 2024-04-26T16:19:24.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:24.936+0000 7f315f372700 1 librados: shutdown 2024-04-26T16:19:25.031 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.803470077Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-04-26T16:19:25.031 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.837074392Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-04-26T16:19:25.031 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.887422809Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-04-26T16:19:25.031 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.91265411Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-04-26T16:19:25.031 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.912981158Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-04-26T16:19:25.031 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.937823825Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-04-26T16:19:25.032 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:24 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:24.979755639Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-04-26T16:19:25.032 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.005018762Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-04-26T16:19:25.032 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.005574351Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-04-26T16:19:25.032 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.005981154Z level=info msg="alerts found to migrate" alerts=0 2024-04-26T16:19:25.032 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.021885116Z level=info msg="Executing migration" id="create library_element table v1" 2024-04-26T16:19:25.032 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.038916359Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.07262015Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.106284258Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.148260657Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.173507413Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.215655273Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.265966061Z level=info msg="Executing migration" id="create data_keys table" 2024-04-26T16:19:25.300 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.307983069Z level=info msg="Executing migration" id="create secrets table" 2024-04-26T16:19:25.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.333242167Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-04-26T16:19:25.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.375035684Z level=info msg="Executing migration" id="add name column into data_keys" 2024-04-26T16:19:25.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.400315787Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-04-26T16:19:25.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.425541396Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-04-26T16:19:25.571 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.467460986Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-04-26T16:19:25.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.509420292Z level=info msg="Executing migration" id="create kv_store table v1" 2024-04-26T16:19:25.572 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.551506481Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.153110+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.153110+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.155533+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.155533+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.196372+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.196372+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.196491+0000 osd.7 (osd.7) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.196491+0000 osd.7 (osd.7) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.517737+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-04-26T16:19:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: cluster 2024-04-26T16:19:24.517737+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-04-26T16:19:25.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: audit 2024-04-26T16:19:24.935211+0000 mon.a (mon.0) 900 : audit [DBG] from='client.? 172.21.15.50:0/4275665686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:25.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[23868]: audit 2024-04-26T16:19:24.935211+0000 mon.a (mon.0) 900 : audit [DBG] from='client.? 172.21.15.50:0/4275665686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:25.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.153110+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:19:25.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.153110+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:19:25.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.155533+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.155533+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.196372+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.196372+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.196491+0000 osd.7 (osd.7) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.196491+0000 osd.7 (osd.7) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.517737+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: cluster 2024-04-26T16:19:24.517737+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: audit 2024-04-26T16:19:24.935211+0000 mon.a (mon.0) 900 : audit [DBG] from='client.? 172.21.15.50:0/4275665686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[29234]: audit 2024-04-26T16:19:24.935211+0000 mon.a (mon.0) 900 : audit [DBG] from='client.? 172.21.15.50:0/4275665686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:25.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.153110+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.153110+0000 osd.1 (osd.1) 49 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.155533+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.155533+0000 osd.1 (osd.1) 50 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.196372+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.196372+0000 osd.7 (osd.7) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.196491+0000 osd.7 (osd.7) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.196491+0000 osd.7 (osd.7) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.517737+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: cluster 2024-04-26T16:19:24.517737+0000 mon.a (mon.0) 899 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: audit 2024-04-26T16:19:24.935211+0000 mon.a (mon.0) 900 : audit [DBG] from='client.? 172.21.15.50:0/4275665686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:25.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:25 smithi050 bash[21463]: audit 2024-04-26T16:19:24.935211+0000 mon.a (mon.0) 900 : audit [DBG] from='client.? 172.21.15.50:0/4275665686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.593494453Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.643661215Z level=info msg="Executing migration" id="create permission table" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.685671929Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.72769759Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.753026689Z level=info msg="Executing migration" id="create role table" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.778189123Z level=info msg="Executing migration" id="add column display_name" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.803405928Z level=info msg="Executing migration" id="add column group_name" 2024-04-26T16:19:25.846 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.82866078Z level=info msg="Executing migration" id="add index role.org_id" 2024-04-26T16:19:25.847 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.853834553Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-04-26T16:19:26.124 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:25 smithi050 systemd[1]: Stopping Ceph alertmanager.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:19:26.131 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.895874208Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-04-26T16:19:26.131 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.921084213Z level=info msg="Executing migration" id="create team role table" 2024-04-26T16:19:26.131 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.962968514Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-04-26T16:19:26.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:25 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:25.988374175Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-04-26T16:19:26.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.030327256Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-04-26T16:19:26.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.055591335Z level=info msg="Executing migration" id="create user role table" 2024-04-26T16:19:26.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.097557816Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-04-26T16:19:26.132 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.139548311Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-04-26T16:19:26.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T16:19:26.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T16:19:26.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T16:19:26.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T16:19:26.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T16:19:26.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T16:19:26.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T16:19:26.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T16:19:26.418 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:19:26.418 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.164889768Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-04-26T16:19:26.418 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.181787508Z level=info msg="Executing migration" id="create builtin role table" 2024-04-26T16:19:26.418 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.198681192Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.240875612Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.257750155Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.274657152Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.316653156Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.341844667Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.383841592Z level=info msg="Executing migration" id="add unique index role.uid" 2024-04-26T16:19:26.419 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.425881261Z level=info msg="Executing migration" id="create seed assignment table" 2024-04-26T16:19:26.518 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[58386]: ts=2024-04-26T16:19:26.233Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.193313+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.193313+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.196237+0000 osd.1 (osd.1) 52 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.196237+0000 osd.1 (osd.1) 52 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.367929+0000 mgr.y (mgr.24401) 63 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.367929+0000 mgr.y (mgr.24401) 63 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:26.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: audit 2024-04-26T16:19:25.544942+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: audit 2024-04-26T16:19:25.544942+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.677549+0000 osd.5 (osd.5) 45 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.677549+0000 osd.5 (osd.5) 45 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.680025+0000 osd.5 (osd.5) 46 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:25.680025+0000 osd.5 (osd.5) 46 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.001312+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.001312+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.004506+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.004506+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.193313+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:19:26.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.193313+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.196237+0000 osd.1 (osd.1) 52 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.196237+0000 osd.1 (osd.1) 52 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.367929+0000 mgr.y (mgr.24401) 63 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.367929+0000 mgr.y (mgr.24401) 63 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: audit 2024-04-26T16:19:25.544942+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: audit 2024-04-26T16:19:25.544942+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.677549+0000 osd.5 (osd.5) 45 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.677549+0000 osd.5 (osd.5) 45 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.680025+0000 osd.5 (osd.5) 46 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:25.680025+0000 osd.5 (osd.5) 46 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.001312+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.001312+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:19:26.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.004506+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:19:26.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.004506+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:19:26.687 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.451211112Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-04-26T16:19:26.687 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.493226105Z level=info msg="Executing migration" id="add column hidden to role table" 2024-04-26T16:19:26.687 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.51840695Z level=info msg="Executing migration" id="create query_history table v1" 2024-04-26T16:19:26.687 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.560531927Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-04-26T16:19:26.687 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.585773484Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-04-26T16:19:26.687 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.627673154Z level=info msg="Executing migration" id="teams permissions migration" 2024-04-26T16:19:26.688 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.669704487Z level=info msg="Executing migration" id="dashboard permissions" 2024-04-26T16:19:26.688 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.69496396Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.193313+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.193313+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.196237+0000 osd.1 (osd.1) 52 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.196237+0000 osd.1 (osd.1) 52 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.367929+0000 mgr.y (mgr.24401) 63 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.367929+0000 mgr.y (mgr.24401) 63 : cluster [DBG] pgmap v43: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: audit 2024-04-26T16:19:25.544942+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: audit 2024-04-26T16:19:25.544942+0000 mon.c (mon.2) 53 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:26.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.677549+0000 osd.5 (osd.5) 45 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.677549+0000 osd.5 (osd.5) 45 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.680025+0000 osd.5 (osd.5) 46 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:25.680025+0000 osd.5 (osd.5) 46 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.001312+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.001312+0000 osd.3 (osd.3) 49 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.004506+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:19:26.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.004506+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:19:26.939 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.736942214Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-04-26T16:19:26.939 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.787240966Z level=info msg="Executing migration" id="alerting notification permissions" 2024-04-26T16:19:26.939 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.812447103Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-04-26T16:19:26.939 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.837777131Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-04-26T16:19:26.939 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.862978075Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-04-26T16:19:26.939 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.888202392Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-04-26T16:19:26.940 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.930093912Z level=info msg="Executing migration" id="create correlation table v1" 2024-04-26T16:19:26.940 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.947201074Z level=info msg="Executing migration" id="add index correlations.uid" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:26 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:26.989256263Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.006044063Z level=info msg="Executing migration" id="add correlation config column" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.048071182Z level=info msg="Executing migration" id="create entity_events table" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.089987381Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.14071563Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.141061668Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.182661028Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.182982694Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-26T16:19:27.217 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.224680658Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-04-26T16:19:27.227 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:26 smithi050 bash[59869]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-alertmanager-a 2024-04-26T16:19:27.469 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.275089162Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-04-26T16:19:27.469 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.317116792Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-04-26T16:19:27.469 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.342451854Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-04-26T16:19:27.469 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.367746621Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-04-26T16:19:27.469 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.384598224Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-26T16:19:27.470 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.409823432Z level=info msg="Executing migration" id="Drop public config table" 2024-04-26T16:19:27.470 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.435134738Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-04-26T16:19:27.470 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.460381583Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-04-26T16:19:27.470 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.477104769Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-04-26T16:19:27.530 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[60064]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-alertmanager-a 2024-04-26T16:19:27.531 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:27 smithi050 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-26T16:19:27.531 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:27 smithi050 systemd[1]: Stopped Ceph alertmanager.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:19:27.531 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:27 smithi050 systemd[1]: Started Ceph alertmanager.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:19:27.730 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.493976757Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-04-26T16:19:27.730 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.578036499Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-04-26T16:19:27.730 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.62834917Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-04-26T16:19:27.730 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.653668413Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-04-26T16:19:27.730 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.695702201Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-04-26T16:19:27.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: audit 2024-04-26T16:19:26.388137+0000 mgr.y (mgr.24401) 64 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:27.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: audit 2024-04-26T16:19:26.388137+0000 mgr.y (mgr.24401) 64 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.958051+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.958051+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.960980+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: cluster 2024-04-26T16:19:26.960980+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: audit 2024-04-26T16:19:27.297919+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: audit 2024-04-26T16:19:27.297919+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: audit 2024-04-26T16:19:27.306612+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[23868]: audit 2024-04-26T16:19:27.306612+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: audit 2024-04-26T16:19:26.388137+0000 mgr.y (mgr.24401) 64 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:27.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: audit 2024-04-26T16:19:26.388137+0000 mgr.y (mgr.24401) 64 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.958051+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.958051+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.960980+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: cluster 2024-04-26T16:19:26.960980+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: audit 2024-04-26T16:19:27.297919+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: audit 2024-04-26T16:19:27.297919+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: audit 2024-04-26T16:19:27.306612+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[21463]: audit 2024-04-26T16:19:27.306612+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: audit 2024-04-26T16:19:26.388137+0000 mgr.y (mgr.24401) 64 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: audit 2024-04-26T16:19:26.388137+0000 mgr.y (mgr.24401) 64 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.958051+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.958051+0000 osd.3 (osd.3) 51 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.960980+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: cluster 2024-04-26T16:19:26.960980+0000 osd.3 (osd.3) 52 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: audit 2024-04-26T16:19:27.297919+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: audit 2024-04-26T16:19:27.297919+0000 mon.a (mon.0) 901 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: audit 2024-04-26T16:19:27.306612+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:27 smithi050 bash[29234]: audit 2024-04-26T16:19:27.306612+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.737700301Z level=info msg="Executing migration" id="add share column" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.762912036Z level=info msg="Executing migration" id="create default alerting folders" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.78822299Z level=info msg="Executing migration" id="create file table" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.81351288Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.838715236Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.864027119Z level=info msg="Executing migration" id="create file_meta table" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.905869613Z level=info msg="Executing migration" id="file table idx: path key" 2024-04-26T16:19:27.990 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.931154893Z level=info msg="Executing migration" id="set path collation in file table" 2024-04-26T16:19:27.991 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.9563042Z level=info msg="Executing migration" id="managed permissions migration" 2024-04-26T16:19:27.991 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:27 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:27.998052711Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-04-26T16:19:28.269 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:28.174808773Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-04-26T16:19:28.539 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:28.276884419Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-04-26T16:19:28.821 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:28.602746443Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-04-26T16:19:28.821 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:28.76874222Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-04-26T16:19:28.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.182880+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:28.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.182880+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:28.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.183019+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.183019+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cephadm 2024-04-26T16:19:27.332880+0000 mgr.y (mgr.24401) 65 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cephadm 2024-04-26T16:19:27.332880+0000 mgr.y (mgr.24401) 65 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.368894+0000 mgr.y (mgr.24401) 66 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.368894+0000 mgr.y (mgr.24401) 66 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.551933+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cluster 2024-04-26T16:19:27.551933+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cephadm 2024-04-26T16:19:27.587324+0000 mgr.y (mgr.24401) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi190 2024-04-26T16:19:28.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:28 smithi190 bash[23868]: cephadm 2024-04-26T16:19:27.587324+0000 mgr.y (mgr.24401) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi190 2024-04-26T16:19:28.874 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.631Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-04-26T16:19:28.874 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.632Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-04-26T16:19:28.874 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.634Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.50 port=9094 2024-04-26T16:19:28.874 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.634Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-26T16:19:28.875 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.665Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-26T16:19:28.875 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.666Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-26T16:19:28.875 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.669Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-04-26T16:19:28.875 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[60093]: ts=2024-04-26T16:19:28.669Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.182880+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.182880+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.183019+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.183019+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cephadm 2024-04-26T16:19:27.332880+0000 mgr.y (mgr.24401) 65 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cephadm 2024-04-26T16:19:27.332880+0000 mgr.y (mgr.24401) 65 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-04-26T16:19:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.368894+0000 mgr.y (mgr.24401) 66 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.368894+0000 mgr.y (mgr.24401) 66 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.551933+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cluster 2024-04-26T16:19:27.551933+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cephadm 2024-04-26T16:19:27.587324+0000 mgr.y (mgr.24401) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi190 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[21463]: cephadm 2024-04-26T16:19:27.587324+0000 mgr.y (mgr.24401) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi190 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.182880+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.182880+0000 osd.3 (osd.3) 53 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.183019+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.183019+0000 osd.3 (osd.3) 54 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cephadm 2024-04-26T16:19:27.332880+0000 mgr.y (mgr.24401) 65 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cephadm 2024-04-26T16:19:27.332880+0000 mgr.y (mgr.24401) 65 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2024-04-26T16:19:28.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.368894+0000 mgr.y (mgr.24401) 66 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:28.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.368894+0000 mgr.y (mgr.24401) 66 : cluster [DBG] pgmap v44: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:28.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.551933+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-04-26T16:19:28.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cluster 2024-04-26T16:19:27.551933+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2024-04-26T16:19:28.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cephadm 2024-04-26T16:19:27.587324+0000 mgr.y (mgr.24401) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi190 2024-04-26T16:19:28.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:28 smithi050 bash[29234]: cephadm 2024-04-26T16:19:27.587324+0000 mgr.y (mgr.24401) 67 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi190 2024-04-26T16:19:29.422 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.08631397Z level=info msg="Executing migration" id="update group index for alert rules" 2024-04-26T16:19:29.422 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.253670976Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-04-26T16:19:29.422 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.329100414Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-04-26T16:19:29.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[23868]: cluster 2024-04-26T16:19:28.705128+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:19:29.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[23868]: cluster 2024-04-26T16:19:28.705128+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:19:29.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[23868]: cluster 2024-04-26T16:19:28.707465+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:19:29.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[23868]: cluster 2024-04-26T16:19:28.707465+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:19:29.691 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.429809321Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-04-26T16:19:29.691 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.530350817Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-04-26T16:19:29.691 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.644588773Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-04-26T16:19:29.691 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.698040504Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-04-26T16:19:29.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[21463]: cluster 2024-04-26T16:19:28.705128+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:19:29.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[21463]: cluster 2024-04-26T16:19:28.705128+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:19:29.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[21463]: cluster 2024-04-26T16:19:28.707465+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:19:29.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[21463]: cluster 2024-04-26T16:19:28.707465+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:19:29.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[29234]: cluster 2024-04-26T16:19:28.705128+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:19:29.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[29234]: cluster 2024-04-26T16:19:28.705128+0000 osd.5 (osd.5) 47 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:19:29.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[29234]: cluster 2024-04-26T16:19:28.707465+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:19:29.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:29 smithi050 bash[29234]: cluster 2024-04-26T16:19:28.707465+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:19:29.967 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.825161444Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-04-26T16:19:29.967 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.865918225Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-04-26T16:19:29.967 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.911668896Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-04-26T16:19:30.267 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:29 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:29.975562221Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-04-26T16:19:30.267 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.000939532Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-04-26T16:19:30.267 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.059716407Z level=info msg="Executing migration" id="create folder table" 2024-04-26T16:19:30.267 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.16671178Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-04-26T16:19:30.268 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.275065804Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-04-26T16:19:30.559 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.462274791Z level=info msg="Executing migration" id="Update folder title length" 2024-04-26T16:19:30.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[23868]: cluster 2024-04-26T16:19:29.369618+0000 mgr.y (mgr.24401) 68 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:30.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[23868]: cluster 2024-04-26T16:19:29.369618+0000 mgr.y (mgr.24401) 68 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:30.821 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.735657779Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-04-26T16:19:30.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:30 smithi050 bash[21463]: cluster 2024-04-26T16:19:29.369618+0000 mgr.y (mgr.24401) 68 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:30.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:30 smithi050 bash[21463]: cluster 2024-04-26T16:19:29.369618+0000 mgr.y (mgr.24401) 68 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:30.874 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:30 smithi050 bash[60093]: ts=2024-04-26T16:19:30.635Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00073295s 2024-04-26T16:19:30.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:30 smithi050 bash[29234]: cluster 2024-04-26T16:19:29.369618+0000 mgr.y (mgr.24401) 68 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:30.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:30 smithi050 bash[29234]: cluster 2024-04-26T16:19:29.369618+0000 mgr.y (mgr.24401) 68 : cluster [DBG] pgmap v46: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:19:31.321 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=migrator t=2024-04-26T16:19:30.870771712Z level=info msg="migrations completed" performed=485 skipped=0 duration=18.32455639s 2024-04-26T16:19:31.321 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:30 smithi190 bash[50598]: logger=sqlstore t=2024-04-26T16:19:30.872232339Z level=info msg="Created default organization" 2024-04-26T16:19:31.321 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[50598]: logger=plugin.finder t=2024-04-26T16:19:31.041751786Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-04-26T16:19:31.322 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[50598]: logger=secrets t=2024-04-26T16:19:31.041870559Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-04-26T16:19:31.322 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[50598]: logger=query_data t=2024-04-26T16:19:31.047151751Z level=info msg="Query Service initialization" 2024-04-26T16:19:31.322 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[50598]: logger=live.push_http t=2024-04-26T16:19:31.050313922Z level=info msg="Live Push Gateway initialization" 2024-04-26T16:19:31.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[23868]: cluster 2024-04-26T16:19:30.565900+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:19:31.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[23868]: cluster 2024-04-26T16:19:30.565900+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:19:31.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[23868]: cluster 2024-04-26T16:19:30.568405+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:19:31.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:31 smithi190 bash[23868]: cluster 2024-04-26T16:19:30.568405+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:19:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[21463]: cluster 2024-04-26T16:19:30.565900+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:19:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[21463]: cluster 2024-04-26T16:19:30.565900+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:19:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[21463]: cluster 2024-04-26T16:19:30.568405+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:19:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[21463]: cluster 2024-04-26T16:19:30.568405+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:19:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[29234]: cluster 2024-04-26T16:19:30.565900+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:19:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[29234]: cluster 2024-04-26T16:19:30.565900+0000 osd.7 (osd.7) 51 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:19:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[29234]: cluster 2024-04-26T16:19:30.568405+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:19:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:31 smithi050 bash[29234]: cluster 2024-04-26T16:19:30.568405+0000 osd.7 (osd.7) 52 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:19:32.266 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.267 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.268 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.273 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:32.569 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=infra.usagestats.collector t=2024-04-26T16:19:32.199707433Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-04-26T16:19:32.570 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=provisioning.datasources t=2024-04-26T16:19:32.200951844Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-26T16:19:32.570 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=provisioning.datasources t=2024-04-26T16:19:32.330905653Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-04-26T16:19:32.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:32 smithi050 bash[21463]: cluster 2024-04-26T16:19:31.371027+0000 mgr.y (mgr.24401) 69 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:32.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:32 smithi050 bash[21463]: cluster 2024-04-26T16:19:31.371027+0000 mgr.y (mgr.24401) 69 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[23868]: cluster 2024-04-26T16:19:31.371027+0000 mgr.y (mgr.24401) 69 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[23868]: cluster 2024-04-26T16:19:31.371027+0000 mgr.y (mgr.24401) 69 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:32.821 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=provisioning.alerting t=2024-04-26T16:19:32.627451787Z level=info msg="starting to provision alerting" 2024-04-26T16:19:32.821 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=provisioning.alerting t=2024-04-26T16:19:32.627495029Z level=info msg="finished to provision alerting" 2024-04-26T16:19:32.821 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=grafanaStorageLogger t=2024-04-26T16:19:32.627865667Z level=info msg="storage starting" 2024-04-26T16:19:32.822 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=ngalert.state.manager t=2024-04-26T16:19:32.629225942Z level=info msg="Warming state cache for startup" 2024-04-26T16:19:32.822 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=http.server t=2024-04-26T16:19:32.643315129Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-04-26T16:19:32.822 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:19:32.648314323Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-26T16:19:32.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:32 smithi050 bash[29234]: cluster 2024-04-26T16:19:31.371027+0000 mgr.y (mgr.24401) 69 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:32.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:32 smithi050 bash[29234]: cluster 2024-04-26T16:19:31.371027+0000 mgr.y (mgr.24401) 69 : cluster [DBG] pgmap v47: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:19:33.143 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=ngalert.state.manager t=2024-04-26T16:19:32.872207854Z level=info msg="State cache has been initialized" states=0 duration=243.025635ms 2024-04-26T16:19:33.143 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=ngalert.multiorg.alertmanager t=2024-04-26T16:19:32.872320021Z level=info msg="starting MultiOrg Alertmanager" 2024-04-26T16:19:33.144 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=ticker t=2024-04-26T16:19:32.872676792Z level=info msg=starting first_tick=2024-04-26T16:19:40Z 2024-04-26T16:19:33.144 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:32 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:19:32.969561558Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-26T16:19:33.144 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:32 smithi190 systemd[1]: Stopping Ceph prometheus.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.151Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.151Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.151Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.151Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.151Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.151Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-26T16:19:33.430 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.152Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-26T16:19:33.431 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.152Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-26T16:19:33.431 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.152Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-26T16:19:33.431 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.152Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-26T16:19:33.431 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.152Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-26T16:19:33.431 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[48704]: ts=2024-04-26T16:19:33.152Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-26T16:19:34.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:33 smithi190 bash[51250]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-prometheus-a 2024-04-26T16:19:34.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[51311]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-prometheus-a 2024-04-26T16:19:34.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:34 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-26T16:19:34.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:34 smithi190 systemd[1]: Stopped Ceph prometheus.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:19:34.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:34 smithi190 systemd[1]: Started Ceph prometheus.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:19:34.281 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STOPPING 2024-04-26T16:19:34.281 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:19:34.281 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STOPPED 2024-04-26T16:19:34.282 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STARTING 2024-04-26T16:19:34.533 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Serving on http://:::9283 2024-04-26T16:19:34.533 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STARTED 2024-04-26T16:19:34.533 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STOPPING 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STOPPED 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STARTING 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Serving on http://:::9283 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STARTED 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STOPPING 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STOPPED 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STARTING 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Serving on http://:::9283 2024-04-26T16:19:34.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21717]: [26/Apr/2024:16:19:34] ENGINE Bus STARTED 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: cluster 2024-04-26T16:19:33.371712+0000 mgr.y (mgr.24401) 70 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: cluster 2024-04-26T16:19:33.371712+0000 mgr.y (mgr.24401) 70 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.146403+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.146403+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.153449+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.153449+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.181772+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.181772+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.184625+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.184625+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.190724+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.190724+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.203631+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.203631+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.210268+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.210268+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.216455+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.216455+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.228142+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: cluster 2024-04-26T16:19:33.371712+0000 mgr.y (mgr.24401) 70 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: cluster 2024-04-26T16:19:33.371712+0000 mgr.y (mgr.24401) 70 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.146403+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.146403+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.153449+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.153449+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.181772+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.181772+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.184625+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.184625+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.190724+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.190724+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.203631+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.203631+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:34.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.210268+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.210268+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.216455+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.216455+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.228142+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.228142+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.232067+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.232067+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.237811+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.237811+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.255926+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[29234]: audit 2024-04-26T16:19:34.255926+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.228142+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.232067+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.232067+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.237811+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.237811+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.255926+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:34.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:34 smithi050 bash[21463]: audit 2024-04-26T16:19:34.255926+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: cluster 2024-04-26T16:19:33.371712+0000 mgr.y (mgr.24401) 70 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: cluster 2024-04-26T16:19:33.371712+0000 mgr.y (mgr.24401) 70 : cluster [DBG] pgmap v48: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.146403+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.146403+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.153449+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.153449+0000 mon.a (mon.0) 905 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.181772+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.181772+0000 mon.c (mon.2) 54 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.184625+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.184625+0000 mon.c (mon.2) 55 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.190724+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.190724+0000 mon.a (mon.0) 906 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.203631+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.203631+0000 mon.c (mon.2) 56 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.210268+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.210268+0000 mon.c (mon.2) 57 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.216455+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.216455+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.228142+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.228142+0000 mon.c (mon.2) 58 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.232067+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.232067+0000 mon.c (mon.2) 59 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.237811+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.237811+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.255926+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:34 smithi190 bash[23868]: audit 2024-04-26T16:19:34.255926+0000 mon.c (mon.2) 60 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.436Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.436Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.437Z caller=main.go:570 level=info host_details="(Linux 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 smithi190 (none))" 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.437Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.437Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.438Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.439Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.441Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.441Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.443Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.443Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.306µs 2024-04-26T16:19:35.572 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.443Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.443Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.444Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.444Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.428µs wal_replay_duration=737.557µs wbl_replay_duration=194ns total_replay_duration=819.445µs 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.445Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.445Z caller=main.go:1029 level=info msg="TSDB started" 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.445Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.469Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.914658ms db_storage=992ns remote_storage=1.184µs web_handler=383ns query_engine=766ns scrape=295.854µs scrape_sd=124.34µs notify=14.068µs notify_sd=25.802µs rules=23.051652ms tracing=7.145µs 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.469Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-04-26T16:19:35.573 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[51340]: ts=2024-04-26T16:19:35.469Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-04-26T16:19:35.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.182534+0000 mgr.y (mgr.24401) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.182534+0000 mgr.y (mgr.24401) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.185435+0000 mgr.y (mgr.24401) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.185435+0000 mgr.y (mgr.24401) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.204237+0000 mgr.y (mgr.24401) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.204237+0000 mgr.y (mgr.24401) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.210854+0000 mgr.y (mgr.24401) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.210854+0000 mgr.y (mgr.24401) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.228635+0000 mgr.y (mgr.24401) 75 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.228635+0000 mgr.y (mgr.24401) 75 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.232481+0000 mgr.y (mgr.24401) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: audit 2024-04-26T16:19:34.232481+0000 mgr.y (mgr.24401) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: cluster 2024-04-26T16:19:34.479325+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: cluster 2024-04-26T16:19:34.479325+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: cluster 2024-04-26T16:19:34.482002+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[21463]: cluster 2024-04-26T16:19:34.482002+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.182534+0000 mgr.y (mgr.24401) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.828 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.182534+0000 mgr.y (mgr.24401) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.185435+0000 mgr.y (mgr.24401) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.185435+0000 mgr.y (mgr.24401) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.204237+0000 mgr.y (mgr.24401) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.204237+0000 mgr.y (mgr.24401) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.210854+0000 mgr.y (mgr.24401) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.210854+0000 mgr.y (mgr.24401) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.228635+0000 mgr.y (mgr.24401) 75 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.228635+0000 mgr.y (mgr.24401) 75 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.232481+0000 mgr.y (mgr.24401) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: audit 2024-04-26T16:19:34.232481+0000 mgr.y (mgr.24401) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: cluster 2024-04-26T16:19:34.479325+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: cluster 2024-04-26T16:19:34.479325+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: cluster 2024-04-26T16:19:34.482002+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:19:35.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:35 smithi050 bash[29234]: cluster 2024-04-26T16:19:34.482002+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:19:35.836 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:19:35.767887229Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-26T16:19:35.836 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:19:35.779031483Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-04-26T16:19:35.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.182534+0000 mgr.y (mgr.24401) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.182534+0000 mgr.y (mgr.24401) 71 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-26T16:19:35.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.185435+0000 mgr.y (mgr.24401) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.185435+0000 mgr.y (mgr.24401) 72 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi050.front.sepia.ceph.com:9093"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.204237+0000 mgr.y (mgr.24401) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.204237+0000 mgr.y (mgr.24401) 73 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.210854+0000 mgr.y (mgr.24401) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.210854+0000 mgr.y (mgr.24401) 74 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.228635+0000 mgr.y (mgr.24401) 75 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.228635+0000 mgr.y (mgr.24401) 75 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.232481+0000 mgr.y (mgr.24401) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: audit 2024-04-26T16:19:34.232481+0000 mgr.y (mgr.24401) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: cluster 2024-04-26T16:19:34.479325+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: cluster 2024-04-26T16:19:34.479325+0000 osd.7 (osd.7) 53 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:19:35.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: cluster 2024-04-26T16:19:34.482002+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:19:35.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[23868]: cluster 2024-04-26T16:19:34.482002+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:19:36.321 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:19:35.844142176Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-04-26T16:19:36.321 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:19:35 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:19:35.855576543Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-04-26T16:19:36.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:36 smithi190 bash[23868]: cluster 2024-04-26T16:19:35.372385+0000 mgr.y (mgr.24401) 77 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:36.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:36 smithi190 bash[23868]: cluster 2024-04-26T16:19:35.372385+0000 mgr.y (mgr.24401) 77 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:36.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:19:36.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:36 smithi050 bash[29234]: cluster 2024-04-26T16:19:35.372385+0000 mgr.y (mgr.24401) 77 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:36.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:36 smithi050 bash[29234]: cluster 2024-04-26T16:19:35.372385+0000 mgr.y (mgr.24401) 77 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:36.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:36 smithi050 bash[21463]: cluster 2024-04-26T16:19:35.372385+0000 mgr.y (mgr.24401) 77 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:36.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:36 smithi050 bash[21463]: cluster 2024-04-26T16:19:35.372385+0000 mgr.y (mgr.24401) 77 : cluster [DBG] pgmap v49: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:37.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:37 smithi050 bash[29234]: audit 2024-04-26T16:19:36.399399+0000 mgr.y (mgr.24401) 78 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:37.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:37 smithi050 bash[29234]: audit 2024-04-26T16:19:36.399399+0000 mgr.y (mgr.24401) 78 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:37.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:37 smithi050 bash[21463]: audit 2024-04-26T16:19:36.399399+0000 mgr.y (mgr.24401) 78 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:37.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:37 smithi050 bash[21463]: audit 2024-04-26T16:19:36.399399+0000 mgr.y (mgr.24401) 78 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:38.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:37 smithi190 bash[23868]: audit 2024-04-26T16:19:36.399399+0000 mgr.y (mgr.24401) 78 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:38.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:37 smithi190 bash[23868]: audit 2024-04-26T16:19:36.399399+0000 mgr.y (mgr.24401) 78 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:39.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:38 smithi190 bash[23868]: cluster 2024-04-26T16:19:37.373786+0000 mgr.y (mgr.24401) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:39.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:38 smithi190 bash[23868]: cluster 2024-04-26T16:19:37.373786+0000 mgr.y (mgr.24401) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:39.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:38 smithi190 bash[23868]: cluster 2024-04-26T16:19:37.954640+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:19:39.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:38 smithi190 bash[23868]: cluster 2024-04-26T16:19:37.954640+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:19:39.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:38 smithi190 bash[23868]: cluster 2024-04-26T16:19:37.958663+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:19:39.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:38 smithi190 bash[23868]: cluster 2024-04-26T16:19:37.958663+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:19:39.093 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[60093]: ts=2024-04-26T16:19:38.636Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001723312s 2024-04-26T16:19:39.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[21463]: cluster 2024-04-26T16:19:37.373786+0000 mgr.y (mgr.24401) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[21463]: cluster 2024-04-26T16:19:37.373786+0000 mgr.y (mgr.24401) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[21463]: cluster 2024-04-26T16:19:37.954640+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[21463]: cluster 2024-04-26T16:19:37.954640+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[21463]: cluster 2024-04-26T16:19:37.958663+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[21463]: cluster 2024-04-26T16:19:37.958663+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[29234]: cluster 2024-04-26T16:19:37.373786+0000 mgr.y (mgr.24401) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[29234]: cluster 2024-04-26T16:19:37.373786+0000 mgr.y (mgr.24401) 79 : cluster [DBG] pgmap v50: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[29234]: cluster 2024-04-26T16:19:37.954640+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[29234]: cluster 2024-04-26T16:19:37.954640+0000 osd.4 (osd.4) 41 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:19:39.094 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[29234]: cluster 2024-04-26T16:19:37.958663+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:19:39.095 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:38 smithi050 bash[29234]: cluster 2024-04-26T16:19:37.958663+0000 osd.4 (osd.4) 42 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:19:39.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:39 smithi050 bash[21463]: cluster 2024-04-26T16:19:39.374270+0000 mgr.y (mgr.24401) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-04-26T16:19:39.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:39 smithi050 bash[21463]: cluster 2024-04-26T16:19:39.374270+0000 mgr.y (mgr.24401) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-04-26T16:19:39.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:39 smithi050 bash[29234]: cluster 2024-04-26T16:19:39.374270+0000 mgr.y (mgr.24401) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-04-26T16:19:39.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:39 smithi050 bash[29234]: cluster 2024-04-26T16:19:39.374270+0000 mgr.y (mgr.24401) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-04-26T16:19:40.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:39 smithi190 bash[23868]: cluster 2024-04-26T16:19:39.374270+0000 mgr.y (mgr.24401) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-04-26T16:19:40.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:39 smithi190 bash[23868]: cluster 2024-04-26T16:19:39.374270+0000 mgr.y (mgr.24401) 80 : cluster [DBG] pgmap v51: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 865 B/s rd, 0 op/s 2024-04-26T16:19:40.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: cluster 2024-04-26T16:19:39.899754+0000 osd.3 (osd.3) 55 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: cluster 2024-04-26T16:19:39.899754+0000 osd.3 (osd.3) 55 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: cluster 2024-04-26T16:19:39.904024+0000 osd.3 (osd.3) 56 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: cluster 2024-04-26T16:19:39.904024+0000 osd.3 (osd.3) 56 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: audit 2024-04-26T16:19:40.400968+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: audit 2024-04-26T16:19:40.400968+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: audit 2024-04-26T16:19:40.417686+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: audit 2024-04-26T16:19:40.417686+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: audit 2024-04-26T16:19:40.576763+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:40.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[29234]: audit 2024-04-26T16:19:40.576763+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: cluster 2024-04-26T16:19:39.899754+0000 osd.3 (osd.3) 55 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: cluster 2024-04-26T16:19:39.899754+0000 osd.3 (osd.3) 55 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: cluster 2024-04-26T16:19:39.904024+0000 osd.3 (osd.3) 56 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: cluster 2024-04-26T16:19:39.904024+0000 osd.3 (osd.3) 56 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: audit 2024-04-26T16:19:40.400968+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: audit 2024-04-26T16:19:40.400968+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: audit 2024-04-26T16:19:40.417686+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: audit 2024-04-26T16:19:40.417686+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: audit 2024-04-26T16:19:40.576763+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:40 smithi050 bash[21463]: audit 2024-04-26T16:19:40.576763+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:41.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: cluster 2024-04-26T16:19:39.899754+0000 osd.3 (osd.3) 55 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:19:41.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: cluster 2024-04-26T16:19:39.899754+0000 osd.3 (osd.3) 55 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: cluster 2024-04-26T16:19:39.904024+0000 osd.3 (osd.3) 56 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: cluster 2024-04-26T16:19:39.904024+0000 osd.3 (osd.3) 56 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: audit 2024-04-26T16:19:40.400968+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: audit 2024-04-26T16:19:40.400968+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: audit 2024-04-26T16:19:40.417686+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: audit 2024-04-26T16:19:40.417686+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: audit 2024-04-26T16:19:40.576763+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:41.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:40 smithi190 bash[23868]: audit 2024-04-26T16:19:40.576763+0000 mon.c (mon.2) 61 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:40.304121+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:40.304121+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:40.304204+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:40.304204+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:40.625531+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:40.625531+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:41.375660+0000 mgr.y (mgr.24401) 81 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[29234]: cluster 2024-04-26T16:19:41.375660+0000 mgr.y (mgr.24401) 81 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:40.304121+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:40.304121+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:41.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:40.304204+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:40.304204+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:40.625531+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-04-26T16:19:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:40.625531+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-04-26T16:19:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:41.375660+0000 mgr.y (mgr.24401) 81 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:41 smithi050 bash[21463]: cluster 2024-04-26T16:19:41.375660+0000 mgr.y (mgr.24401) 81 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:42.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:40.304121+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:42.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:40.304121+0000 osd.6 (osd.6) 27 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:42.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:40.304204+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:42.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:40.304204+0000 osd.6 (osd.6) 28 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:42.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:40.625531+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-04-26T16:19:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:40.625531+0000 mon.a (mon.0) 911 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2024-04-26T16:19:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:41.375660+0000 mgr.y (mgr.24401) 81 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:42.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:41 smithi190 bash[23868]: cluster 2024-04-26T16:19:41.375660+0000 mgr.y (mgr.24401) 81 : cluster [DBG] pgmap v53: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:42 smithi190 bash[23868]: cluster 2024-04-26T16:19:42.389723+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:19:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:42 smithi190 bash[23868]: cluster 2024-04-26T16:19:42.389723+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:19:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:42 smithi190 bash[23868]: cluster 2024-04-26T16:19:42.393577+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:19:43.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:42 smithi190 bash[23868]: cluster 2024-04-26T16:19:42.393577+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:19:43.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[29234]: cluster 2024-04-26T16:19:42.389723+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:19:43.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[29234]: cluster 2024-04-26T16:19:42.389723+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:19:43.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[29234]: cluster 2024-04-26T16:19:42.393577+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:19:43.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[29234]: cluster 2024-04-26T16:19:42.393577+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:19:43.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[21463]: cluster 2024-04-26T16:19:42.389723+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:19:43.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[21463]: cluster 2024-04-26T16:19:42.389723+0000 osd.7 (osd.7) 55 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:19:43.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[21463]: cluster 2024-04-26T16:19:42.393577+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:19:43.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:42 smithi050 bash[21463]: cluster 2024-04-26T16:19:42.393577+0000 osd.7 (osd.7) 56 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:19:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.099864+0000 osd.1 (osd.1) 53 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:19:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.099864+0000 osd.1 (osd.1) 53 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:19:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.114671+0000 osd.1 (osd.1) 54 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:19:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.114671+0000 osd.1 (osd.1) 54 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:19:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.375311+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:19:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.375311+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:19:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.376090+0000 mgr.y (mgr.24401) 82 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.376090+0000 mgr.y (mgr.24401) 82 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.379038+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:19:44.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:43 smithi190 bash[23868]: cluster 2024-04-26T16:19:43.379038+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:19:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.099864+0000 osd.1 (osd.1) 53 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:19:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.099864+0000 osd.1 (osd.1) 53 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:19:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.114671+0000 osd.1 (osd.1) 54 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:19:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.114671+0000 osd.1 (osd.1) 54 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.375311+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.375311+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.376090+0000 mgr.y (mgr.24401) 82 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.376090+0000 mgr.y (mgr.24401) 82 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.379038+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[21463]: cluster 2024-04-26T16:19:43.379038+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.099864+0000 osd.1 (osd.1) 53 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.099864+0000 osd.1 (osd.1) 53 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.114671+0000 osd.1 (osd.1) 54 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.114671+0000 osd.1 (osd.1) 54 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.375311+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:19:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.375311+0000 osd.7 (osd.7) 57 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:19:44.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.376090+0000 mgr.y (mgr.24401) 82 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:44.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.376090+0000 mgr.y (mgr.24401) 82 : cluster [DBG] pgmap v54: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:44.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.379038+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:19:44.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:43 smithi050 bash[29234]: cluster 2024-04-26T16:19:43.379038+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:19:44.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:19:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:19:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[29234]: cluster 2024-04-26T16:19:45.373957+0000 osd.7 (osd.7) 59 : cluster [DBG] 3.e scrub starts 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[29234]: cluster 2024-04-26T16:19:45.373957+0000 osd.7 (osd.7) 59 : cluster [DBG] 3.e scrub starts 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[29234]: cluster 2024-04-26T16:19:45.376777+0000 mgr.y (mgr.24401) 83 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[29234]: cluster 2024-04-26T16:19:45.376777+0000 mgr.y (mgr.24401) 83 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[29234]: cluster 2024-04-26T16:19:45.378245+0000 osd.7 (osd.7) 60 : cluster [DBG] 3.e scrub ok 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[29234]: cluster 2024-04-26T16:19:45.378245+0000 osd.7 (osd.7) 60 : cluster [DBG] 3.e scrub ok 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[21463]: cluster 2024-04-26T16:19:45.373957+0000 osd.7 (osd.7) 59 : cluster [DBG] 3.e scrub starts 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[21463]: cluster 2024-04-26T16:19:45.373957+0000 osd.7 (osd.7) 59 : cluster [DBG] 3.e scrub starts 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[21463]: cluster 2024-04-26T16:19:45.376777+0000 mgr.y (mgr.24401) 83 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[21463]: cluster 2024-04-26T16:19:45.376777+0000 mgr.y (mgr.24401) 83 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[21463]: cluster 2024-04-26T16:19:45.378245+0000 osd.7 (osd.7) 60 : cluster [DBG] 3.e scrub ok 2024-04-26T16:19:46.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:46 smithi050 bash[21463]: cluster 2024-04-26T16:19:45.378245+0000 osd.7 (osd.7) 60 : cluster [DBG] 3.e scrub ok 2024-04-26T16:19:46.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:19:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[23868]: cluster 2024-04-26T16:19:45.373957+0000 osd.7 (osd.7) 59 : cluster [DBG] 3.e scrub starts 2024-04-26T16:19:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[23868]: cluster 2024-04-26T16:19:45.373957+0000 osd.7 (osd.7) 59 : cluster [DBG] 3.e scrub starts 2024-04-26T16:19:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[23868]: cluster 2024-04-26T16:19:45.376777+0000 mgr.y (mgr.24401) 83 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[23868]: cluster 2024-04-26T16:19:45.376777+0000 mgr.y (mgr.24401) 83 : cluster [DBG] pgmap v55: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[23868]: cluster 2024-04-26T16:19:45.378245+0000 osd.7 (osd.7) 60 : cluster [DBG] 3.e scrub ok 2024-04-26T16:19:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:46 smithi190 bash[23868]: cluster 2024-04-26T16:19:45.378245+0000 osd.7 (osd.7) 60 : cluster [DBG] 3.e scrub ok 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- 172.21.15.50:0/3611195787 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c1101d0 msgr2=0x7f735c1125c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- 172.21.15.50:0/3611195787 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c1101d0 0x7f735c1125c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f735400b0a0 tx=0x7f735400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- 172.21.15.50:0/3611195787 shutdown_connections 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- 172.21.15.50:0/3611195787 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c1101d0 0x7f735c1125c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- 172.21.15.50:0/3611195787 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c06bbd0 0x7f735c06bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- 172.21.15.50:0/3611195787 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c06c4f0 0x7f735c10fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- 172.21.15.50:0/3611195787 >> 172.21.15.50:0/3611195787 conn(0x7f735c079120 msgr2=0x7f735c079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- 172.21.15.50:0/3611195787 shutdown_connections 2024-04-26T16:19:46.899 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- 172.21.15.50:0/3611195787 wait complete. 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 librados: starting msgr at 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 librados: starting objecter 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 Processor -- start 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- start start 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 librados: setting wanted keys 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 librados: calling monclient init 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c06bbd0 0x7f735c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:46.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c06c4f0 0x7f735c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c1101d0 0x7f735c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f735c114e20 con 0x7f735c1101d0 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f735c114ca0 con 0x7f735c06c4f0 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f7361fbd700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f735c114fa0 con 0x7f735c06bbd0 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f735bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c1101d0 0x7f735c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f735b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c06bbd0 0x7f735c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f735bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c1101d0 0x7f735c19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48024/0 (socket says 172.21.15.50:48024) 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f735b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c06bbd0 0x7f735c19b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48228/0 (socket says 172.21.15.50:48228) 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f735bfff700 1 -- 172.21.15.50:0/2411753625 learned_addr learned my addr 172.21.15.50:0/2411753625 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.903+0000 7f735affd700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c06c4f0 0x7f735c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735b7fe700 1 -- 172.21.15.50:0/2411753625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c06c4f0 msgr2=0x7f735c19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735b7fe700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f735c06c4f0 0x7f735c19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735b7fe700 1 -- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c1101d0 msgr2=0x7f735c19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735b7fe700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c1101d0 0x7f735c19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735b7fe700 1 -- 172.21.15.50:0/2411753625 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f735c1a0350 con 0x7f735c06bbd0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735bfff700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f735c1101d0 0x7f735c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735b7fe700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c06bbd0 0x7f735c19b390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f734c00e9c0 tx=0x7f734c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.?.objecter ms_handle_connect 0x7f735c06bbd0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c00cb70 con 0x7f735c06bbd0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f735c1a0640 con 0x7f735c06bbd0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f734c00ee60 con 0x7f735c06bbd0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734c018420 con 0x7f735c06bbd0 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 10 client.24473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f735c1a7ec0 con 0x7f735c06bbd0 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 1 librados: init done 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 10 client.24473.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 15 client.24473.objecter _session_command_op_assign -1 1 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 20 client.24473.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 10 client.24473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f735c1029f0 con 0x7f735c06bbd0 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f734c01f020 con 0x7f735c06bbd0 2024-04-26T16:19:46.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7344071f80 0x7f7344074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(112..112 src has 57..112) ==== 7450+0+0 (secure 0 0 0) 0x7f734c091930 con 0x7f735c06bbd0 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.24473.objecter ms_dispatch 0x7f735c001550 osd_map(112..112 src has 57..112) 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 3 client.24473.objecter handle_osd_map got epochs [112,112] > 0 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 3 client.24473.objecter handle_osd_map decoding full epoch 112 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735affd700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7344071f80 0x7f7344074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.24473.objecter checking command 1 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f734407bdb0 0x7f734407e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 20 client.24473.objecter _get_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 15 client.24473.objecter _session_command_op_remove -1 1 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 20 client.24473.objecter _get_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 20 client.24473.objecter get_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 15 client.24473.objecter _session_command_op_assign 1 1 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 4 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.24473.objecter _send_command 1 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f734c00be50 con 0x7f734407bdb0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 20 client.24473.objecter dump_active .. 0 homeless 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f734c091dc0 con 0x7f735c06bbd0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735bfff700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f734407bdb0 0x7f734407e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.24473.objecter ms_handle_connect 0x7f7344071f80 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735affd700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7344071f80 0x7f7344074440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f735c102300 tx=0x7f735000b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f7358ff9700 10 client.24473.objecter ms_handle_connect 0x7f734407bdb0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.907+0000 7f735bfff700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f734407bdb0 0x7f734407e210 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.911+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f734c00be50 con 0x7f734407bdb0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.911+0000 7f7358ff9700 10 client.24473.objecter ms_dispatch 0x7f735c001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.911+0000 7f7358ff9700 10 client.24473.objecter _finish_command 1 = system:0 2024-04-26T16:19:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.911+0000 7f7358ff9700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 4 2024-04-26T16:19:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.911+0000 7f7358ff9700 15 client.24473.objecter _session_command_op_remove 1 1 2024-04-26T16:19:46.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 10 client.24473.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 15 client.24473.objecter _session_command_op_assign -1 2 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter _get_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter _get_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 15 client.24473.objecter _session_command_op_remove -1 2 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter get_session s=0x7f734407b6e0 osd=1 3 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 15 client.24473.objecter _session_command_op_assign 1 2 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 4 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 10 client.24473.objecter _send_command 2 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f735c06a110 con 0x7f734407bdb0 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7358ff9700 1 -- 172.21.15.50:0/2411753625 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f735c06a110 con 0x7f734407bdb0 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7358ff9700 10 client.24473.objecter ms_dispatch 0x7f735c001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7358ff9700 10 client.24473.objecter _finish_command 2 = system:0 2024-04-26T16:19:46.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7358ff9700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 4 2024-04-26T16:19:46.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7358ff9700 15 client.24473.objecter _session_command_op_remove 1 2 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 10 librados: watch_flush enter 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 10 librados: watch_flush exit 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 10 client.24473.objecter close_session for osd.1 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f734407bdb0 msgr2=0x7f734407e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f734407bdb0 0x7f734407e210 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter put_session s=0x7f734407b6e0 osd=1 1 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 20 client.24473.objecter shutdown clearing up homeless session... 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 10 client.24473.objecter successfully canceled tick 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7344071f80 msgr2=0x7f7344074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7344071f80 0x7f7344074440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f735c102300 tx=0x7f735000b410 comp rx=0 tx=0).stop 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c06bbd0 msgr2=0x7f735c19b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:46.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 --2- 172.21.15.50:0/2411753625 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f735c06bbd0 0x7f735c19b390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f734c00e9c0 tx=0x7f734c00ecd0 comp rx=0 tx=0).stop 2024-04-26T16:19:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f735bfff700 1 -- 172.21.15.50:0/2411753625 reap_dead start 2024-04-26T16:19:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 shutdown_connections 2024-04-26T16:19:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 >> 172.21.15.50:0/2411753625 conn(0x7f735c079120 msgr2=0x7f735c111a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 shutdown_connections 2024-04-26T16:19:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 -- 172.21.15.50:0/2411753625 wait complete. 2024-04-26T16:19:46.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:46.927+0000 7f7361fbd700 1 librados: shutdown 2024-04-26T16:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/2716513411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806bbd0 msgr2=0x7fb5d806c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- 172.21.15.50:0/2716513411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806bbd0 0x7fb5d806c050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5c80098f0 tx=0x7fb5c8009c00 comp rx=0 tx=0).stop 2024-04-26T16:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/2716513411 shutdown_connections 2024-04-26T16:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- 172.21.15.50:0/2716513411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d806c590 0x7fb5d8110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- 172.21.15.50:0/2716513411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d81094d0 0x7fb5d81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- 172.21.15.50:0/2716513411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806bbd0 0x7fb5d806c050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/2716513411 >> 172.21.15.50:0/2716513411 conn(0x7fb5d80793c0 msgr2=0x7fb5d80797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/2716513411 shutdown_connections 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/2716513411 wait complete. 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 librados: starting msgr at 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 librados: starting objecter 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 Processor -- start 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- start start 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 librados: setting wanted keys 2024-04-26T16:19:47.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 librados: calling monclient init 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d806bbd0 0x7fb5d8199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806c590 0x7fb5d81996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d81094d0 0x7fb5d819da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8112ad0 con 0x7fb5d806c590 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8112950 con 0x7fb5d806bbd0 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb5d8112c50 con 0x7fb5d81094d0 2024-04-26T16:19:47.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806c590 0x7fb5d81996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5de9b2700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d81094d0 0x7fb5d819da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806c590 0x7fb5d81996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48046/0 (socket says 172.21.15.50:48046) 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 -- 172.21.15.50:0/3664233294 learned_addr learned my addr 172.21.15.50:0/3664233294 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5de1b1700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d806bbd0 0x7fb5d8199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 -- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d81094d0 msgr2=0x7fb5d819da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d81094d0 0x7fb5d819da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 -- 172.21.15.50:0/3664233294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d806bbd0 msgr2=0x7fb5d8199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d806bbd0 0x7fb5d8199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 -- 172.21.15.50:0/3664233294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d819e180 con 0x7fb5d806c590 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5dd9b0700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806c590 0x7fb5d81996b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009230 tx=0x7fb5c800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5cf7fe700 10 client.?.objecter ms_handle_connect 0x7fb5d806c590 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5cf7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:47.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c800b9d0 con 0x7fb5d806c590 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb5c800bb70 con 0x7fb5d806c590 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d819e410 con 0x7fb5d806c590 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8004d50 con 0x7fb5d806c590 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 10 client.14589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d81a5cf0 con 0x7fb5d806c590 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 librados: init done 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 10 client.14589.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 15 client.14589.objecter _session_command_op_assign -1 1 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 20 client.14589.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 10 client.14589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:47.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.091+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb5d8104b80 con 0x7fb5d806c590 2024-04-26T16:19:47.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb5c8005040 con 0x7fb5d806c590 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 57..112) ==== 7450+0+0 (secure 0 0 0) 0x7fb5c8092e30 con 0x7fb5d806c590 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter ms_dispatch 0x7fb5d8001550 osd_map(112..112 src has 57..112) 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 3 client.14589.objecter handle_osd_map got epochs [112,112] > 0 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 3 client.14589.objecter handle_osd_map decoding full epoch 112 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5de1b1700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter checking command 1 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb5c407be30 0x7fb5c407e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter _get_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 15 client.14589.objecter _session_command_op_remove -1 1 2024-04-26T16:19:47.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter _get_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter get_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 15 client.14589.objecter _session_command_op_assign 0 1 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 4 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter _send_command 1 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5c8004500 con 0x7fb5c407be30 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter dump_active .. 0 homeless 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fb5c80932c0 con 0x7fb5d806c590 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5de9b2700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb5c407be30 0x7fb5c407e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter ms_handle_connect 0x7fb5c4071f80 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5de1b1700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4005fd0 tx=0x7fb5d4009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter ms_handle_connect 0x7fb5c407be30 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5de9b2700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb5c407be30 0x7fb5c407e290 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5c8004500 con 0x7fb5c407be30 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter ms_dispatch 0x7fb5d8001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 10 client.14589.objecter _finish_command 1 = system:0 2024-04-26T16:19:47.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 4 2024-04-26T16:19:47.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.095+0000 7fb5cf7fe700 15 client.14589.objecter _session_command_op_remove 0 1 2024-04-26T16:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 10 client.14589.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 15 client.14589.objecter _session_command_op_assign -1 2 2024-04-26T16:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter _get_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter _get_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 15 client.14589.objecter _session_command_op_remove -1 2 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter get_session s=0x7fb5c407b760 osd=0 3 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 15 client.14589.objecter _session_command_op_assign 0 2 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 4 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 10 client.14589.objecter _send_command 2 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb5d806a110 con 0x7fb5c407be30 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5cf7fe700 1 -- 172.21.15.50:0/3664233294 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb5d806a110 con 0x7fb5c407be30 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5cf7fe700 10 client.14589.objecter ms_dispatch 0x7fb5d8001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5cf7fe700 10 client.14589.objecter _finish_command 2 = system:0 2024-04-26T16:19:47.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5cf7fe700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 4 2024-04-26T16:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5cf7fe700 15 client.14589.objecter _session_command_op_remove 0 2 2024-04-26T16:19:47.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 10 librados: watch_flush enter 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 10 librados: watch_flush exit 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 10 client.14589.objecter close_session for osd.0 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb5c407be30 msgr2=0x7fb5c407e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb5c407be30 0x7fb5c407e290 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter put_session s=0x7fb5c407b760 osd=0 1 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 20 client.14589.objecter shutdown clearing up homeless session... 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 10 client.14589.objecter successfully canceled tick 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 msgr2=0x7fb5c4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4005fd0 tx=0x7fb5d4009310 comp rx=0 tx=0).stop 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806c590 msgr2=0x7fb5d81996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 --2- 172.21.15.50:0/3664233294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d806c590 0x7fb5d81996b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009230 tx=0x7fb5c800fa00 comp rx=0 tx=0).stop 2024-04-26T16:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5de9b2700 1 -- 172.21.15.50:0/3664233294 reap_dead start 2024-04-26T16:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 shutdown_connections 2024-04-26T16:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 >> 172.21.15.50:0/3664233294 conn(0x7fb5d80793c0 msgr2=0x7fb5d8101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 shutdown_connections 2024-04-26T16:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 -- 172.21.15.50:0/3664233294 wait complete. 2024-04-26T16:19:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.115+0000 7fb5e0415700 1 librados: shutdown 2024-04-26T16:19:47.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:47 smithi050 bash[21463]: audit 2024-04-26T16:19:46.404043+0000 mgr.y (mgr.24401) 84 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:47.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:47 smithi050 bash[21463]: audit 2024-04-26T16:19:46.404043+0000 mgr.y (mgr.24401) 84 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:47.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:47 smithi050 bash[29234]: audit 2024-04-26T16:19:46.404043+0000 mgr.y (mgr.24401) 84 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:47.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:47 smithi050 bash[29234]: audit 2024-04-26T16:19:46.404043+0000 mgr.y (mgr.24401) 84 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1276706872 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006b9c0 msgr2=0x7f87a006bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- 172.21.15.50:0/1276706872 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006b9c0 0x7f87a006bda0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f879c009990 tx=0x7f879c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1276706872 shutdown_connections 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- 172.21.15.50:0/1276706872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a010fec0 0x7f87a01122b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- 172.21.15.50:0/1276706872 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006b9c0 0x7f87a006bda0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- 172.21.15.50:0/1276706872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87a006c2e0 0x7f87a010f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1276706872 >> 172.21.15.50:0/1276706872 conn(0x7f87a0079290 msgr2=0x7f87a00796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1276706872 shutdown_connections 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1276706872 wait complete. 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 librados: starting msgr at 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 librados: starting objecter 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 Processor -- start 2024-04-26T16:19:47.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- start start 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 librados: setting wanted keys 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 librados: calling monclient init 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a006b9c0 0x7f87a019b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006c2e0 0x7f87a019b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87a010fec0 0x7f87a019fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0114db0 con 0x7f87a006b9c0 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0114c30 con 0x7f87a010fec0 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a7a7b700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f87a0114f30 con 0x7f87a006c2e0 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a006b9c0 0x7f87a019b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5016700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006c2e0 0x7f87a019b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a006b9c0 0x7f87a019b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48072/0 (socket says 172.21.15.50:48072) 2024-04-26T16:19:47.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 -- 172.21.15.50:0/1739719088 learned_addr learned my addr 172.21.15.50:0/1739719088 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:47.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:47 smithi190 bash[23868]: audit 2024-04-26T16:19:46.404043+0000 mgr.y (mgr.24401) 84 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:47.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:47 smithi190 bash[23868]: audit 2024-04-26T16:19:46.404043+0000 mgr.y (mgr.24401) 84 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:47.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a6018700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87a010fec0 0x7f87a019fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 -- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006c2e0 msgr2=0x7f87a019b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f87a006c2e0 0x7f87a019b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 -- 172.21.15.50:0/1739719088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87a010fec0 msgr2=0x7f87a019fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87a010fec0 0x7f87a019fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 -- 172.21.15.50:0/1739719088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a01a03c0 con 0x7f87a006b9c0 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a6018700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f87a010fec0 0x7f87a019fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f87a5817700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a006b9c0 0x7f87a019b3b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f879c005910 tx=0x7f879c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.823+0000 7f8796ffd700 10 client.?.objecter ms_handle_connect 0x7f87a006b9c0 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c027070 con 0x7f87a006b9c0 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1739719088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a01a0650 con 0x7f87a006b9c0 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a7a7b700 10 client.14595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a7a7b700 1 -- 172.21.15.50:0/1739719088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a01a7f30 con 0x7f87a006b9c0 2024-04-26T16:19:47.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a7a7b700 1 librados: init done 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f879c022470 con 0x7f87a006b9c0 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c0173f0 con 0x7f87a006b9c0 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8794ff9700 10 client.14595.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8794ff9700 15 client.14595.objecter _session_command_op_assign -1 1 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8794ff9700 20 client.14595.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8794ff9700 10 client.14595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8784002920 con 0x7f87a006b9c0 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f879c02a070 con 0x7f87a006b9c0 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f878c072170 0x7f878c074630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a5016700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f878c072170 0x7f878c074630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 57..112) ==== 7450+0+0 (secure 0 0 0) 0x7f879c022e00 con 0x7f87a006b9c0 2024-04-26T16:19:47.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 10 client.14595.objecter ms_dispatch 0x7f87a0001550 osd_map(112..112 src has 57..112) 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 3 client.14595.objecter handle_osd_map got epochs [112,112] > 0 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 3 client.14595.objecter handle_osd_map decoding full epoch 112 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 10 client.14595.objecter checking command 1 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f878c07cad0 0x7f878c07ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 20 client.14595.objecter _get_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 15 client.14595.objecter _session_command_op_remove -1 1 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 20 client.14595.objecter _get_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 20 client.14595.objecter get_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 15 client.14595.objecter _session_command_op_assign 3 1 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 4 2024-04-26T16:19:47.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 10 client.14595.objecter _send_command 1 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f878c07f640 con 0x7f878c07cad0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 20 client.14595.objecter dump_active .. 0 homeless 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f879c05c200 con 0x7f87a006b9c0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 10 client.14595.objecter ms_handle_connect 0x7f878c072170 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a5016700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f878c072170 0x7f878c074630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87a0101fb0 tx=0x7f879000a300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a6018700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f878c07cad0 0x7f878c07ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f87a6018700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f878c07cad0 0x7f878c07ef70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.827+0000 7f8796ffd700 10 client.14595.objecter ms_handle_connect 0x7f878c07cad0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.831+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f878c07f640 con 0x7f878c07cad0 2024-04-26T16:19:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.831+0000 7f8796ffd700 10 client.14595.objecter ms_dispatch 0x7f87a0001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:47.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.831+0000 7f8796ffd700 10 client.14595.objecter _finish_command 1 = system:0 2024-04-26T16:19:47.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.831+0000 7f8796ffd700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 4 2024-04-26T16:19:47.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.831+0000 7f8796ffd700 15 client.14595.objecter _session_command_op_remove 3 1 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 10 client.14595.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 15 client.14595.objecter _session_command_op_assign -1 2 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 20 client.14595.objecter _get_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 20 client.14595.objecter _get_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 15 client.14595.objecter _session_command_op_remove -1 2 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 20 client.14595.objecter get_session s=0x7f878c07c400 osd=3 3 2024-04-26T16:19:47.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 15 client.14595.objecter _session_command_op_assign 3 2 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 4 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 10 client.14595.objecter _send_command 2 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.847+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8784001240 con 0x7f878c07cad0 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8796ffd700 1 -- 172.21.15.50:0/1739719088 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8784001240 con 0x7f878c07cad0 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8796ffd700 10 client.14595.objecter ms_dispatch 0x7f87a0001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8796ffd700 10 client.14595.objecter _finish_command 2 = system:0 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8796ffd700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 4 2024-04-26T16:19:47.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8796ffd700 15 client.14595.objecter _session_command_op_remove 3 2 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 10 librados: watch_flush enter 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 10 librados: watch_flush exit 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 10 client.14595.objecter close_session for osd.3 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f878c07cad0 msgr2=0x7f878c07ef70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f878c07cad0 0x7f878c07ef70 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 20 client.14595.objecter put_session s=0x7f878c07c400 osd=3 1 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 20 client.14595.objecter shutdown clearing up homeless session... 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 10 client.14595.objecter successfully canceled tick 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f878c072170 msgr2=0x7f878c074630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f878c072170 0x7f878c074630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f87a0101fb0 tx=0x7f879000a300 comp rx=0 tx=0).stop 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a006b9c0 msgr2=0x7f87a019b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 --2- 172.21.15.50:0/1739719088 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f87a006b9c0 0x7f87a019b3b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f879c005910 tx=0x7f879c0047b0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f87a6018700 1 -- 172.21.15.50:0/1739719088 reap_dead start 2024-04-26T16:19:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 shutdown_connections 2024-04-26T16:19:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 >> 172.21.15.50:0/1739719088 conn(0x7f87a0079290 msgr2=0x7f87a0111720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 shutdown_connections 2024-04-26T16:19:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 -- 172.21.15.50:0/1739719088 wait complete. 2024-04-26T16:19:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.851+0000 7f8794ff9700 1 librados: shutdown 2024-04-26T16:19:47.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2403736337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 msgr2=0x7fc61406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2403736337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 0x7fc61406b590 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc6080098f0 tx=0x7fc608009c00 comp rx=0 tx=0).stop 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2403736337 shutdown_connections 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2403736337 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc61406bad0 0x7fc6141102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2403736337 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc6141094a0 0x7fc614109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2403736337 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 0x7fc61406b590 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2403736337 >> 172.21.15.50:0/2403736337 conn(0x7fc6140792d0 msgr2=0x7fc6140796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2403736337 shutdown_connections 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2403736337 wait complete. 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 librados: starting msgr at 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 librados: starting objecter 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 Processor -- start 2024-04-26T16:19:47.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- start start 2024-04-26T16:19:47.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 librados: setting wanted keys 2024-04-26T16:19:47.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 librados: calling monclient init 2024-04-26T16:19:47.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 0x7fc6141023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc61406bad0 0x7fc614102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6141094a0 0x7fc614102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc614112ae0 con 0x7fc61406b130 2024-04-26T16:19:47.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc614112960 con 0x7fc6141094a0 2024-04-26T16:19:47.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fc614112c60 con 0x7fc61406bad0 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc61406bad0 0x7fc614102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc61406bad0 0x7fc614102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48282/0 (socket says 172.21.15.50:48282) 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 -- 172.21.15.50:0/2842259263 learned_addr learned my addr 172.21.15.50:0/2842259263 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc61259c700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 0x7fc6141023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc612d9d700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6141094a0 0x7fc614102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 -- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6141094a0 msgr2=0x7fc614102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6141094a0 0x7fc614102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 -- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 msgr2=0x7fc6141023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 0x7fc6141023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 -- 172.21.15.50:0/2842259263 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6141a76a0 con 0x7fc61406bad0 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc61259c700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc61406b130 0x7fc6141023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc612d9d700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6141094a0 0x7fc614102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:19:47.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc611d9b700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc61406bad0 0x7fc614102930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc608009230 tx=0x7fc60800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc6037fe700 10 client.?.objecter ms_handle_connect 0x7fc61406bad0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc6037fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc60800b9d0 con 0x7fc61406bad0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fc60800bb70 con 0x7fc61406bad0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6141a7930 con 0x7fc61406bad0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc608004d50 con 0x7fc61406bad0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 10 client.24491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6141a7d80 con 0x7fc61406bad0 2024-04-26T16:19:47.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.971+0000 7fc618fa8700 1 librados: init done 2024-04-26T16:19:47.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc618fa8700 10 client.24491.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc618fa8700 15 client.24491.objecter _session_command_op_assign -1 1 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc618fa8700 20 client.24491.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc618fa8700 10 client.24491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc614104b50 con 0x7fc61406bad0 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fc608005040 con 0x7fc61406bad0 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc5fc071f80 0x7fc5fc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(112..112 src has 57..112) ==== 7450+0+0 (secure 0 0 0) 0x7fc6080923c0 con 0x7fc61406bad0 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 10 client.24491.objecter ms_dispatch 0x7fc614001550 osd_map(112..112 src has 57..112) 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 3 client.24491.objecter handle_osd_map got epochs [112,112] > 0 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 3 client.24491.objecter handle_osd_map decoding full epoch 112 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc61259c700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc5fc071f80 0x7fc5fc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 10 client.24491.objecter checking command 1 2024-04-26T16:19:47.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc5fc07bc80 0x7fc5fc07e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 20 client.24491.objecter _get_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 15 client.24491.objecter _session_command_op_remove -1 1 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 20 client.24491.objecter _get_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 20 client.24491.objecter get_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 15 client.24491.objecter _session_command_op_assign 5 1 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 4 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 10 client.24491.objecter _send_command 1 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc60800bd60 con 0x7fc5fc07bc80 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 20 client.24491.objecter dump_active .. 0 homeless 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7fc608099050 con 0x7fc61406bad0 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 10 client.24491.objecter ms_handle_connect 0x7fc5fc071f80 2024-04-26T16:19:47.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc61259c700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc5fc071f80 0x7fc5fc074440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc604005fd0 tx=0x7fc604009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc612d9d700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc5fc07bc80 0x7fc5fc07e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc612d9d700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc5fc07bc80 0x7fc5fc07e0e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.975+0000 7fc6037fe700 10 client.24491.objecter ms_handle_connect 0x7fc5fc07bc80 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.979+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc60800bd60 con 0x7fc5fc07bc80 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.979+0000 7fc6037fe700 10 client.24491.objecter ms_dispatch 0x7fc614001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.979+0000 7fc6037fe700 10 client.24491.objecter _finish_command 1 = system:0 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.979+0000 7fc6037fe700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 4 2024-04-26T16:19:47.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.979+0000 7fc6037fe700 15 client.24491.objecter _session_command_op_remove 5 1 2024-04-26T16:19:47.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 10 client.24491.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:47.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 15 client.24491.objecter _session_command_op_assign -1 2 2024-04-26T16:19:47.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter _get_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter _get_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 15 client.24491.objecter _session_command_op_remove -1 2 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter get_session s=0x7fc5fc07b5b0 osd=5 3 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 15 client.24491.objecter _session_command_op_assign 5 2 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 4 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 10 client.24491.objecter _send_command 2 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc61406a110 con 0x7fc5fc07bc80 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc6037fe700 1 -- 172.21.15.50:0/2842259263 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc61406a110 con 0x7fc5fc07bc80 2024-04-26T16:19:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc6037fe700 10 client.24491.objecter ms_dispatch 0x7fc614001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:47.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc6037fe700 10 client.24491.objecter _finish_command 2 = system:0 2024-04-26T16:19:47.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc6037fe700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 4 2024-04-26T16:19:47.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc6037fe700 15 client.24491.objecter _session_command_op_remove 5 2 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 10 librados: watch_flush enter 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 10 librados: watch_flush exit 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 10 client.24491.objecter close_session for osd.5 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc5fc07bc80 msgr2=0x7fc5fc07e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc5fc07bc80 0x7fc5fc07e0e0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter put_session s=0x7fc5fc07b5b0 osd=5 1 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 20 client.24491.objecter shutdown clearing up homeless session... 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 10 client.24491.objecter successfully canceled tick 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc5fc071f80 msgr2=0x7fc5fc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc5fc071f80 0x7fc5fc074440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc604005fd0 tx=0x7fc604009310 comp rx=0 tx=0).stop 2024-04-26T16:19:47.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc61406bad0 msgr2=0x7fc614102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc618fa8700 1 --2- 172.21.15.50:0/2842259263 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc61406bad0 0x7fc614102930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc608009230 tx=0x7fc60800fa00 comp rx=0 tx=0).stop 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.995+0000 7fc612d9d700 1 -- 172.21.15.50:0/2842259263 reap_dead start 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.999+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 shutdown_connections 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.999+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 >> 172.21.15.50:0/2842259263 conn(0x7fc6140792d0 msgr2=0x7fc614100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.999+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 shutdown_connections 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.999+0000 7fc618fa8700 1 -- 172.21.15.50:0/2842259263 wait complete. 2024-04-26T16:19:47.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:47.999+0000 7fc618fa8700 1 librados: shutdown 2024-04-26T16:19:48.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- 172.21.15.50:0/2474100596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23201095c0 msgr2=0x7f23201099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:48.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- 172.21.15.50:0/2474100596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23201095c0 0x7f23201099a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2310009990 tx=0x7f2310009ca0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- 172.21.15.50:0/2474100596 shutdown_connections 2024-04-26T16:19:48.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- 172.21.15.50:0/2474100596 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f232006c520 0x7f2320110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- 172.21.15.50:0/2474100596 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23201095c0 0x7f23201099a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- 172.21.15.50:0/2474100596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232006bb60 0x7f232006bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- 172.21.15.50:0/2474100596 >> 172.21.15.50:0/2474100596 conn(0x7f2320079120 msgr2=0x7f2320079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:48.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- 172.21.15.50:0/2474100596 shutdown_connections 2024-04-26T16:19:48.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- 172.21.15.50:0/2474100596 wait complete. 2024-04-26T16:19:48.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 librados: starting msgr at 2024-04-26T16:19:48.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 librados: starting objecter 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 Processor -- start 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- start start 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 librados: setting wanted keys 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 librados: calling monclient init 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232006bb60 0x7f23201990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f232006c520 0x7f2320199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23201095c0 0x7f232019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2320112b00 con 0x7f232006c520 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2320112980 con 0x7f232006bb60 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2326248700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f2320112c80 con 0x7f23201095c0 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f232006c520 0x7f2320199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f231ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23201095c0 0x7f232019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:48.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f232006c520 0x7f2320199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50336/0 (socket says 172.21.15.50:50336) 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f231ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23201095c0 0x7f232019d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59574/0 (socket says 172.21.15.50:59574) 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 -- 172.21.15.50:0/917942993 learned_addr learned my addr 172.21.15.50:0/917942993 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f231f7fe700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232006bb60 0x7f23201990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 -- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23201095c0 msgr2=0x7f232019d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23201095c0 0x7f232019d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 -- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232006bb60 msgr2=0x7f23201990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f232006bb60 0x7f23201990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 -- 172.21.15.50:0/917942993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232019e0a0 con 0x7f232006c520 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f2316ffd700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f232006c520 0x7f2320199620 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f230800c9b0 tx=0x7f230800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f231d7fa700 10 client.?.objecter ms_handle_connect 0x7f232006c520 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.055+0000 7f231d7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308007b00 con 0x7f232006c520 2024-04-26T16:19:48.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2308007ca0 con 0x7f232006c520 2024-04-26T16:19:48.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f232019e390 con 0x7f232006c520 2024-04-26T16:19:48.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 10 client.14607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:48.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23201a5c10 con 0x7f232006c520 2024-04-26T16:19:48.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 1 librados: init done 2024-04-26T16:19:48.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2308018400 con 0x7f232006c520 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 10 client.14607.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 15 client.14607.objecter _session_command_op_assign -1 1 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 20 client.14607.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 10 client.14607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2300002920 con 0x7f232006c520 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f230800f450 con 0x7f232006c520 2024-04-26T16:19:48.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f230c071f80 0x7f230c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(112..112 src has 57..112) ==== 7450+0+0 (secure 0 0 0) 0x7f2308091430 con 0x7f232006c520 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231f7fe700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f230c071f80 0x7f230c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 10 client.14607.objecter ms_dispatch 0x7f2320001550 osd_map(112..112 src has 57..112) 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 3 client.14607.objecter handle_osd_map got epochs [112,112] > 0 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 3 client.14607.objecter handle_osd_map decoding full epoch 112 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 10 client.14607.objecter checking command 1 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f230c07be30 0x7f230c07e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 20 client.14607.objecter _get_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 15 client.14607.objecter _session_command_op_remove -1 1 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 20 client.14607.objecter _get_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 20 client.14607.objecter get_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 15 client.14607.objecter _session_command_op_assign 4 1 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 4 2024-04-26T16:19:48.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 10 client.14607.objecter _send_command 1 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f230c07e960 con 0x7f230c07be30 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 20 client.14607.objecter dump_active .. 0 homeless 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) ==== 24+0+0 (secure 0 0 0) 0x7f2308091910 con 0x7f232006c520 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 10 client.14607.objecter ms_handle_connect 0x7f230c071f80 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231f7fe700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f230c071f80 0x7f230c074440 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f231000b730 tx=0x7f231001a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231ffff700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f230c07be30 0x7f230c07e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231ffff700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f230c07be30 0x7f230c07e290 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.059+0000 7f231d7fa700 10 client.14607.objecter ms_handle_connect 0x7f230c07be30 2024-04-26T16:19:48.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.063+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f230c07e960 con 0x7f230c07be30 2024-04-26T16:19:48.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.063+0000 7f231d7fa700 10 client.14607.objecter ms_dispatch 0x7f2320001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:48.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.063+0000 7f231d7fa700 10 client.14607.objecter _finish_command 1 = system:0 2024-04-26T16:19:48.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.063+0000 7f231d7fa700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 4 2024-04-26T16:19:48.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.063+0000 7f231d7fa700 15 client.14607.objecter _session_command_op_remove 4 1 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 10 client.14607.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 15 client.14607.objecter _session_command_op_assign -1 2 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 20 client.14607.objecter _get_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 20 client.14607.objecter _get_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 15 client.14607.objecter _session_command_op_remove -1 2 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 20 client.14607.objecter get_session s=0x7f230c07b760 osd=4 3 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 15 client.14607.objecter _session_command_op_assign 4 2 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 4 2024-04-26T16:19:48.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 10 client.14607.objecter _send_command 2 2024-04-26T16:19:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.083+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2300001240 con 0x7f230c07be30 2024-04-26T16:19:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f231d7fa700 1 -- 172.21.15.50:0/917942993 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2300001240 con 0x7f230c07be30 2024-04-26T16:19:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f231d7fa700 10 client.14607.objecter ms_dispatch 0x7f2320001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f231d7fa700 10 client.14607.objecter _finish_command 2 = system:0 2024-04-26T16:19:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f231d7fa700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 4 2024-04-26T16:19:48.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f231d7fa700 15 client.14607.objecter _session_command_op_remove 4 2 2024-04-26T16:19:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 10 librados: watch_flush enter 2024-04-26T16:19:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 10 librados: watch_flush exit 2024-04-26T16:19:48.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 10 client.14607.objecter close_session for osd.4 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f230c07be30 msgr2=0x7f230c07e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f230c07be30 0x7f230c07e290 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 20 client.14607.objecter put_session s=0x7f230c07b760 osd=4 1 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 20 client.14607.objecter shutdown clearing up homeless session... 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 10 client.14607.objecter successfully canceled tick 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f230c071f80 msgr2=0x7f230c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f230c071f80 0x7f230c074440 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f231000b730 tx=0x7f231001a040 comp rx=0 tx=0).stop 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f232006c520 msgr2=0x7f2320199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 --2- 172.21.15.50:0/917942993 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f232006c520 0x7f2320199620 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f230800c9b0 tx=0x7f230800ccc0 comp rx=0 tx=0).stop 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f231ffff700 1 -- 172.21.15.50:0/917942993 reap_dead start 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 shutdown_connections 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 >> 172.21.15.50:0/917942993 conn(0x7f2320079120 msgr2=0x7f232010a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 shutdown_connections 2024-04-26T16:19:48.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 -- 172.21.15.50:0/917942993 wait complete. 2024-04-26T16:19:48.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:48.087+0000 7f2326248700 1 librados: shutdown 2024-04-26T16:19:48.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.378269+0000 mgr.y (mgr.24401) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.378269+0000 mgr.y (mgr.24401) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.410085+0000 osd.0 (osd.0) 47 : cluster [DBG] 3.1 deep-scrub starts 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.410085+0000 osd.0 (osd.0) 47 : cluster [DBG] 3.1 deep-scrub starts 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.424230+0000 osd.0 (osd.0) 48 : cluster [DBG] 3.1 deep-scrub ok 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.424230+0000 osd.0 (osd.0) 48 : cluster [DBG] 3.1 deep-scrub ok 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.378269+0000 mgr.y (mgr.24401) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.378269+0000 mgr.y (mgr.24401) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.410085+0000 osd.0 (osd.0) 47 : cluster [DBG] 3.1 deep-scrub starts 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.410085+0000 osd.0 (osd.0) 47 : cluster [DBG] 3.1 deep-scrub starts 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.424230+0000 osd.0 (osd.0) 48 : cluster [DBG] 3.1 deep-scrub ok 2024-04-26T16:19:48.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:48 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.424230+0000 osd.0 (osd.0) 48 : cluster [DBG] 3.1 deep-scrub ok 2024-04-26T16:19:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:48 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.378269+0000 mgr.y (mgr.24401) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:48 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.378269+0000 mgr.y (mgr.24401) 85 : cluster [DBG] pgmap v56: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:48 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.410085+0000 osd.0 (osd.0) 47 : cluster [DBG] 3.1 deep-scrub starts 2024-04-26T16:19:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:48 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.410085+0000 osd.0 (osd.0) 47 : cluster [DBG] 3.1 deep-scrub starts 2024-04-26T16:19:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:48 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.424230+0000 osd.0 (osd.0) 48 : cluster [DBG] 3.1 deep-scrub ok 2024-04-26T16:19:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:48 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.424230+0000 osd.0 (osd.0) 48 : cluster [DBG] 3.1 deep-scrub ok 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.970279+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.970279+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.970347+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[29234]: cluster 2024-04-26T16:19:47.970347+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[29234]: cluster 2024-04-26T16:19:48.450890+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[29234]: cluster 2024-04-26T16:19:48.450890+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.970279+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:49.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.970279+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:49.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.970347+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:49.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[21463]: cluster 2024-04-26T16:19:47.970347+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:49.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[21463]: cluster 2024-04-26T16:19:48.450890+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-04-26T16:19:49.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:49 smithi050 bash[21463]: cluster 2024-04-26T16:19:48.450890+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-04-26T16:19:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:49 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.970279+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:49 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.970279+0000 osd.0 (osd.0) 49 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:49 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.970347+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:49 smithi190 bash[23868]: cluster 2024-04-26T16:19:47.970347+0000 osd.0 (osd.0) 50 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:49 smithi190 bash[23868]: cluster 2024-04-26T16:19:48.450890+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-04-26T16:19:49.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:49 smithi190 bash[23868]: cluster 2024-04-26T16:19:48.450890+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2024-04-26T16:19:50.183 INFO:teuthology.orchestra.run.smithi050.stdout:55834574948 2024-04-26T16:19:50.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 -- 172.21.15.50:0/1279992241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 msgr2=0x7f0c04108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1279992241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 0x7f0c04108760 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00b0a0 tx=0x7f0bfc00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 -- 172.21.15.50:0/1279992241 shutdown_connections 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1279992241 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102cd0 0x7f0c0410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1279992241 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 0x7f0c04108760 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1279992241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c04102310 0x7f0c04102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 -- 172.21.15.50:0/1279992241 >> 172.21.15.50:0/1279992241 conn(0x7f0c04079290 msgr2=0x7f0c040796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 -- 172.21.15.50:0/1279992241 shutdown_connections 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.247+0000 7f0c0a766700 1 -- 172.21.15.50:0/1279992241 wait complete. 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 librados: starting msgr at 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 librados: starting objecter 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 Processor -- start 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- start start 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 librados: setting wanted keys 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 librados: calling monclient init 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102310 0x7f0c04199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c04102cd0 0x7f0c041995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 0x7f0c0419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c04111980 con 0x7f0c04108380 2024-04-26T16:19:50.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c04111800 con 0x7f0c04102cd0 2024-04-26T16:19:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0c04111b00 con 0x7f0c04102310 2024-04-26T16:19:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102310 0x7f0c04199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102310 0x7f0c04199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59588/0 (socket says 172.21.15.50:59588) 2024-04-26T16:19:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 -- 172.21.15.50:0/1911090823 learned_addr learned my addr 172.21.15.50:0/1911090823 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:50.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c08d03700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 0x7f0c0419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c037fe700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c04102cd0 0x7f0c041995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 -- 172.21.15.50:0/1911090823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c04102cd0 msgr2=0x7f0c041995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c04102cd0 0x7f0c041995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 -- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 msgr2=0x7f0c0419d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c04108380 0x7f0c0419d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 -- 172.21.15.50:0/1911090823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0419e080 con 0x7f0c04102310 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c03fff700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102310 0x7f0c04199070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0062a0 tx=0x7f0bfc004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 10 client.?.objecter ms_handle_connect 0x7f0c04102310 2024-04-26T16:19:50.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc021070 con 0x7f0c04102310 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0bfc007b80 con 0x7f0c04102310 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0419e310 con 0x7f0c04102310 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc005040 con 0x7f0c04102310 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 10 client.24503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c041a5bf0 con 0x7f0c04102310 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 librados: init done 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 10 client.24503.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 15 client.24503.objecter _session_command_op_assign -1 1 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 20 client.24503.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 10 client.24503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c04103a30 con 0x7f0c04102310 2024-04-26T16:19:50.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0bfc024070 con 0x7f0c04102310 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.251+0000 7f0c017fa700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bf0071f80 0x7f0bf0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(113..113 src has 57..113) ==== 7450+0+0 (secure 0 0 0) 0x7f0bfc00f020 con 0x7f0c04102310 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter ms_dispatch 0x7f0c04001550 osd_map(113..113 src has 57..113) 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 3 client.24503.objecter handle_osd_map got epochs [113,113] > 0 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 3 client.24503.objecter handle_osd_map decoding full epoch 113 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c037fe700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bf0071f80 0x7f0bf0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter checking command 1 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bf007bdb0 0x7f0bf007e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter _get_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 15 client.24503.objecter _session_command_op_remove -1 1 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter _get_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter get_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 15 client.24503.objecter _session_command_op_assign 2 1 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 4 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter _send_command 1 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bfc013e70 con 0x7f0bf007bdb0 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter dump_active .. 0 homeless 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7f0bfc099050 con 0x7f0c04102310 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c08d03700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bf007bdb0 0x7f0bf007e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter ms_handle_connect 0x7f0bf0071f80 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c037fe700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bf0071f80 0x7f0bf0074440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0c04079c60 tx=0x7f0bec005d50 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter ms_handle_connect 0x7f0bf007bdb0 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c08d03700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bf007bdb0 0x7f0bf007e210 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:50.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bfc013e70 con 0x7f0bf007bdb0 2024-04-26T16:19:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter ms_dispatch 0x7f0c04001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 10 client.24503.objecter _finish_command 1 = system:0 2024-04-26T16:19:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 4 2024-04-26T16:19:50.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.255+0000 7f0c017fa700 15 client.24503.objecter _session_command_op_remove 2 1 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 10 client.24503.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 15 client.24503.objecter _session_command_op_assign -1 2 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 20 client.24503.objecter _get_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 20 client.24503.objecter _get_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 15 client.24503.objecter _session_command_op_remove -1 2 2024-04-26T16:19:50.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 20 client.24503.objecter get_session s=0x7f0bf007b6e0 osd=2 3 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 15 client.24503.objecter _session_command_op_assign 2 2 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 4 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 10 client.24503.objecter _send_command 2 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0c0406a110 con 0x7f0bf007bdb0 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c017fa700 1 -- 172.21.15.50:0/1911090823 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0c0406a110 con 0x7f0bf007bdb0 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c017fa700 10 client.24503.objecter ms_dispatch 0x7f0c04001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c017fa700 10 client.24503.objecter _finish_command 2 = system:0 2024-04-26T16:19:50.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c017fa700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 4 2024-04-26T16:19:50.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.271+0000 7f0c017fa700 15 client.24503.objecter _session_command_op_remove 2 2 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 10 librados: watch_flush enter 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 10 librados: watch_flush exit 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 10 client.24503.objecter close_session for osd.2 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bf007bdb0 msgr2=0x7f0bf007e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bf007bdb0 0x7f0bf007e210 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 20 client.24503.objecter put_session s=0x7f0bf007b6e0 osd=2 1 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 20 client.24503.objecter shutdown clearing up homeless session... 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 10 client.24503.objecter successfully canceled tick 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bf0071f80 msgr2=0x7f0bf0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bf0071f80 0x7f0bf0074440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0c04079c60 tx=0x7f0bec005d50 comp rx=0 tx=0).stop 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102310 msgr2=0x7f0c04199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:50.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 --2- 172.21.15.50:0/1911090823 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c04102310 0x7f0c04199070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0062a0 tx=0x7f0bfc004ce0 comp rx=0 tx=0).stop 2024-04-26T16:19:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c08d03700 1 -- 172.21.15.50:0/1911090823 reap_dead start 2024-04-26T16:19:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 shutdown_connections 2024-04-26T16:19:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 >> 172.21.15.50:0/1911090823 conn(0x7f0c04079290 msgr2=0x7f0c041011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 shutdown_connections 2024-04-26T16:19:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 -- 172.21.15.50:0/1911090823 wait complete. 2024-04-26T16:19:50.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:50.275+0000 7f0c0a766700 1 librados: shutdown 2024-04-26T16:19:50.415 INFO:teuthology.orchestra.run.smithi050.stdout:34359738478 2024-04-26T16:19:50.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T16:19:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:50 smithi190 bash[23868]: cluster 2024-04-26T16:19:49.378974+0000 mgr.y (mgr.24401) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:19:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:50 smithi190 bash[23868]: cluster 2024-04-26T16:19:49.378974+0000 mgr.y (mgr.24401) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:19:50.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:50 smithi050 bash[29234]: cluster 2024-04-26T16:19:49.378974+0000 mgr.y (mgr.24401) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:19:50.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:50 smithi050 bash[29234]: cluster 2024-04-26T16:19:49.378974+0000 mgr.y (mgr.24401) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:19:50.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:50 smithi050 bash[21463]: cluster 2024-04-26T16:19:49.378974+0000 mgr.y (mgr.24401) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:19:50.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:50 smithi050 bash[21463]: cluster 2024-04-26T16:19:49.378974+0000 mgr.y (mgr.24401) 86 : cluster [DBG] pgmap v58: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 -- 172.21.15.50:0/216877498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b0105660 msgr2=0x7ff1b0105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 --2- 172.21.15.50:0/216877498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b0105660 0x7ff1b0105ae0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009990 tx=0x7ff1a4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 -- 172.21.15.50:0/216877498 shutdown_connections 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 --2- 172.21.15.50:0/216877498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b0106020 0x7ff1b0112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 --2- 172.21.15.50:0/216877498 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b010b660 0x7ff1b010ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 --2- 172.21.15.50:0/216877498 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b0105660 0x7ff1b0105ae0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 -- 172.21.15.50:0/216877498 >> 172.21.15.50:0/216877498 conn(0x7ff1b0101150 msgr2=0x7ff1b0103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:51.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 -- 172.21.15.50:0/216877498 shutdown_connections 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.099+0000 7ff1b5414700 1 -- 172.21.15.50:0/216877498 wait complete. 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 librados: starting msgr at 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 librados: starting objecter 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 Processor -- start 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- start start 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 librados: setting wanted keys 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 librados: calling monclient init 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b0105660 0x7ff1b019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b0106020 0x7ff1b019b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b010b660 0x7ff1b019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0114dc0 con 0x7ff1b010b660 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff1b0114c40 con 0x7ff1b0105660 2024-04-26T16:19:51.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff1b0114f40 con 0x7ff1b0106020 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1ae7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b0106020 0x7ff1b019b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1ae7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b0106020 0x7ff1b019b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59602/0 (socket says 172.21.15.50:59602) 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1ae7fc700 1 -- 172.21.15.50:0/375097030 learned_addr learned my addr 172.21.15.50:0/375097030 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b010b660 0x7ff1b019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1aeffd700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b0105660 0x7ff1b019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 -- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b0106020 msgr2=0x7ff1b019b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b0106020 0x7ff1b019b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 -- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b0105660 msgr2=0x7ff1b019b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1b0105660 0x7ff1b019b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 -- 172.21.15.50:0/375097030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1b01a0330 con 0x7ff1b010b660 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1ae7fc700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1b0106020 0x7ff1b019b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1af7fe700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b010b660 0x7ff1b019fbf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800be20 tx=0x7ff1a8009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:51.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff197fff700 10 client.?.objecter ms_handle_connect 0x7ff1b010b660 2024-04-26T16:19:51.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff197fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:51.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a8019070 con 0x7ff1b010b660 2024-04-26T16:19:51.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff1a800cc10 con 0x7ff1b010b660 2024-04-26T16:19:51.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1b01a0620 con 0x7ff1b010b660 2024-04-26T16:19:51.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1a80123a0 con 0x7ff1b010b660 2024-04-26T16:19:51.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 10 client.14619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:51.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1b01a7ea0 con 0x7ff1b010b660 2024-04-26T16:19:51.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 librados: init done 2024-04-26T16:19:51.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 10 client.14619.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:51.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 15 client.14619.objecter _session_command_op_assign -1 1 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 20 client.14619.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 10 client.14619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1b0106da0 con 0x7ff1b010b660 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.103+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff1a801c070 con 0x7ff1b010b660 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff198071f80 0x7ff198074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 57..113) ==== 7450+0+0 (secure 0 0 0) 0x7ff1a80910f0 con 0x7ff1b010b660 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter ms_dispatch 0x7ff1b0001550 osd_map(113..113 src has 57..113) 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 3 client.14619.objecter handle_osd_map got epochs [113,113] > 0 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 3 client.14619.objecter handle_osd_map decoding full epoch 113 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff1aeffd700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff198071f80 0x7ff198074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter checking command 1 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff19807be30 0x7ff19807e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter _get_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 15 client.14619.objecter _session_command_op_remove -1 1 2024-04-26T16:19:51.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter _get_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter get_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 15 client.14619.objecter _session_command_op_assign 6 1 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 4 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter _send_command 1 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1a8015e50 con 0x7ff19807be30 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter dump_active .. 0 homeless 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7ff1a80914a0 con 0x7ff1b010b660 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter ms_handle_connect 0x7ff198071f80 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff1aeffd700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff198071f80 0x7ff198074440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009bf0 tx=0x7ff1a0009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff1ae7fc700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff19807be30 0x7ff19807e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter ms_handle_connect 0x7ff19807be30 2024-04-26T16:19:51.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff1ae7fc700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff19807be30 0x7ff19807e290 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:51.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1a8015e50 con 0x7ff19807be30 2024-04-26T16:19:51.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter ms_dispatch 0x7ff1b0001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:51.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 10 client.14619.objecter _finish_command 1 = system:0 2024-04-26T16:19:51.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 4 2024-04-26T16:19:51.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.107+0000 7ff197fff700 15 client.14619.objecter _session_command_op_remove 6 1 2024-04-26T16:19:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 10 client.14619.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 15 client.14619.objecter _session_command_op_assign -1 2 2024-04-26T16:19:51.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter _get_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter _get_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 15 client.14619.objecter _session_command_op_remove -1 2 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter get_session s=0x7ff19807b760 osd=6 3 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 15 client.14619.objecter _session_command_op_assign 6 2 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 4 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 10 client.14619.objecter _send_command 2 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff1b006a110 con 0x7ff19807be30 2024-04-26T16:19:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff197fff700 1 -- 172.21.15.50:0/375097030 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff1b006a110 con 0x7ff19807be30 2024-04-26T16:19:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff197fff700 10 client.14619.objecter ms_dispatch 0x7ff1b0001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff197fff700 10 client.14619.objecter _finish_command 2 = system:0 2024-04-26T16:19:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff197fff700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 4 2024-04-26T16:19:51.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff197fff700 15 client.14619.objecter _session_command_op_remove 6 2 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 10 librados: watch_flush enter 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 10 librados: watch_flush exit 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 10 client.14619.objecter close_session for osd.6 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff19807be30 msgr2=0x7ff19807e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff19807be30 0x7ff19807e290 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter put_session s=0x7ff19807b760 osd=6 1 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 20 client.14619.objecter shutdown clearing up homeless session... 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 10 client.14619.objecter successfully canceled tick 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff198071f80 msgr2=0x7ff198074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff198071f80 0x7ff198074440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff1a0009bf0 tx=0x7ff1a0009260 comp rx=0 tx=0).stop 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b010b660 msgr2=0x7ff1b019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 --2- 172.21.15.50:0/375097030 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1b010b660 0x7ff1b019fbf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff1a800be20 tx=0x7ff1a8009ed0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1af7fe700 1 -- 172.21.15.50:0/375097030 reap_dead start 2024-04-26T16:19:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 shutdown_connections 2024-04-26T16:19:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 >> 172.21.15.50:0/375097030 conn(0x7ff1b0101150 msgr2=0x7ff1b0102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 shutdown_connections 2024-04-26T16:19:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 -- 172.21.15.50:0/375097030 wait complete. 2024-04-26T16:19:51.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.127+0000 7ff1b5414700 1 librados: shutdown 2024-04-26T16:19:51.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 -- 172.21.15.50:0/3062447164 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 msgr2=0x7f935006c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- 172.21.15.50:0/3062447164 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f935006c070 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9344009990 tx=0x7f9344009ca0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 -- 172.21.15.50:0/3062447164 shutdown_connections 2024-04-26T16:19:51.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- 172.21.15.50:0/3062447164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9350109540 0x7f9350109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- 172.21.15.50:0/3062447164 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f935006c5b0 0x7f9350110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- 172.21.15.50:0/3062447164 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f935006c070 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 -- 172.21.15.50:0/3062447164 >> 172.21.15.50:0/3062447164 conn(0x7f9350078230 msgr2=0x7f9350078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:51.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 -- 172.21.15.50:0/3062447164 shutdown_connections 2024-04-26T16:19:51.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 -- 172.21.15.50:0/3062447164 wait complete. 2024-04-26T16:19:51.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 librados: starting msgr at 2024-04-26T16:19:51.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 librados: starting objecter 2024-04-26T16:19:51.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 Processor -- start 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 -- start start 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 librados: setting wanted keys 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 librados: calling monclient init 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f9350102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935006c5b0 0x7f9350102a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.299+0000 7f93565c0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350109540 0x7f9350102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9350112b20 con 0x7f9350109540 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f93501129a0 con 0x7f935006c5b0 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9350112ca0 con 0x7f935006bbf0 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f9350102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f9350102520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59628/0 (socket says 172.21.15.50:59628) 2024-04-26T16:19:51.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350109540 0x7f9350102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934ffff700 1 -- 172.21.15.50:0/1654096073 learned_addr learned my addr 172.21.15.50:0/1654096073 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934f7fe700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935006c5b0 0x7f9350102a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 -- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 msgr2=0x7f9350102520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f9350102520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 -- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935006c5b0 msgr2=0x7f9350102a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935006c5b0 0x7f9350102a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 -- 172.21.15.50:0/1654096073 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93501a7690 con 0x7f9350109540 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934ffff700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f935006bbf0 0x7f9350102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934f7fe700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f935006c5b0 0x7f9350102a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f9354b5d700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350109540 0x7f9350102fa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f934800efb0 tx=0x7f934800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 10 client.?.objecter ms_handle_connect 0x7f9350109540 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934800ca50 con 0x7f9350109540 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93501a7980 con 0x7f9350109540 2024-04-26T16:19:51.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f934800cbf0 con 0x7f9350109540 2024-04-26T16:19:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93480046b0 con 0x7f9350109540 2024-04-26T16:19:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 10 client.14625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93501a7ec0 con 0x7f9350109540 2024-04-26T16:19:51.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 librados: init done 2024-04-26T16:19:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 10 client.14625.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:19:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 15 client.14625.objecter _session_command_op_assign -1 1 2024-04-26T16:19:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 20 client.14625.objecter _get_session osd=-1 returning homeless 2024-04-26T16:19:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 10 client.14625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:19:51.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9350104bb0 con 0x7f9350109540 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9348012050 con 0x7f9350109540 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9338071f80 0x7f9338074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(113..113 src has 57..113) ==== 7450+0+0 (secure 0 0 0) 0x7f9348090ca0 con 0x7f9350109540 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 10 client.14625.objecter ms_dispatch 0x7f9350001550 osd_map(113..113 src has 57..113) 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 3 client.14625.objecter handle_osd_map got epochs [113,113] > 0 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 3 client.14625.objecter handle_osd_map decoding full epoch 113 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934ffff700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9338071f80 0x7f9338074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 10 client.14625.objecter checking command 1 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f933807be30 0x7f933807e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:19:51.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 20 client.14625.objecter _get_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 15 client.14625.objecter _session_command_op_remove -1 1 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 20 client.14625.objecter _get_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 20 client.14625.objecter get_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 15 client.14625.objecter _session_command_op_assign 7 1 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 4 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 10 client.14625.objecter _send_command 1 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f933807e960 con 0x7f933807be30 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 20 client.14625.objecter dump_active .. 0 homeless 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.303+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=113) ==== 24+0+0 (secure 0 0 0) 0x7f93480932c0 con 0x7f9350109540 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934f7fe700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f933807be30 0x7f933807e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 10 client.14625.objecter ms_handle_connect 0x7f9338071f80 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934ffff700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9338071f80 0x7f9338074440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f93400104e0 tx=0x7f934000b480 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 10 client.14625.objecter ms_handle_connect 0x7f933807be30 2024-04-26T16:19:51.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934f7fe700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f933807be30 0x7f933807e290 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:19:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f933807e960 con 0x7f933807be30 2024-04-26T16:19:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 10 client.14625.objecter ms_dispatch 0x7f9350001550 command_reply(tid 1: 0 ) 2024-04-26T16:19:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 10 client.14625.objecter _finish_command 1 = system:0 2024-04-26T16:19:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 4 2024-04-26T16:19:51.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.307+0000 7f934d7fa700 15 client.14625.objecter _session_command_op_remove 7 1 2024-04-26T16:19:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 10 client.14625.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:19:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 15 client.14625.objecter _session_command_op_assign -1 2 2024-04-26T16:19:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter _get_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter _get_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 15 client.14625.objecter _session_command_op_remove -1 2 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter get_session s=0x7f933807b760 osd=7 3 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 15 client.14625.objecter _session_command_op_assign 7 2 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 4 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 10 client.14625.objecter _send_command 2 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f935006a110 con 0x7f933807be30 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f934d7fa700 1 -- 172.21.15.50:0/1654096073 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f935006a110 con 0x7f933807be30 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f934d7fa700 10 client.14625.objecter ms_dispatch 0x7f9350001550 command_reply(tid 2: 0 ) 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f934d7fa700 10 client.14625.objecter _finish_command 2 = system:0 2024-04-26T16:19:51.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f934d7fa700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 4 2024-04-26T16:19:51.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f934d7fa700 15 client.14625.objecter _session_command_op_remove 7 2 2024-04-26T16:19:51.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 10 librados: watch_flush enter 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 10 librados: watch_flush exit 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 10 client.14625.objecter close_session for osd.7 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f933807be30 msgr2=0x7f933807e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f933807be30 0x7f933807e290 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter put_session s=0x7f933807b760 osd=7 1 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 20 client.14625.objecter shutdown clearing up homeless session... 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 10 client.14625.objecter successfully canceled tick 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9338071f80 msgr2=0x7f9338074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9338071f80 0x7f9338074440 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f93400104e0 tx=0x7f934000b480 comp rx=0 tx=0).stop 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350109540 msgr2=0x7f9350102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 --2- 172.21.15.50:0/1654096073 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9350109540 0x7f9350102fa0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f934800efb0 tx=0x7f934800c3a0 comp rx=0 tx=0).stop 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f9354b5d700 1 -- 172.21.15.50:0/1654096073 reap_dead start 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 shutdown_connections 2024-04-26T16:19:51.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 >> 172.21.15.50:0/1654096073 conn(0x7f9350078230 msgr2=0x7f9350107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:19:51.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 shutdown_connections 2024-04-26T16:19:51.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 -- 172.21.15.50:0/1654096073 wait complete. 2024-04-26T16:19:51.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:19:51.327+0000 7f93565c0700 1 librados: shutdown 2024-04-26T16:19:51.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[29234]: cluster 2024-04-26T16:19:50.405365+0000 osd.0 (osd.0) 51 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:19:51.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[29234]: cluster 2024-04-26T16:19:50.405365+0000 osd.0 (osd.0) 51 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:19:51.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[29234]: cluster 2024-04-26T16:19:50.407695+0000 osd.0 (osd.0) 52 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:19:51.469 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[21463]: cluster 2024-04-26T16:19:50.405365+0000 osd.0 (osd.0) 51 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:19:51.469 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[21463]: cluster 2024-04-26T16:19:50.405365+0000 osd.0 (osd.0) 51 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:19:51.469 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[21463]: cluster 2024-04-26T16:19:50.407695+0000 osd.0 (osd.0) 52 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:19:51.469 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[21463]: cluster 2024-04-26T16:19:50.407695+0000 osd.0 (osd.0) 52 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:19:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:51 smithi190 bash[23868]: cluster 2024-04-26T16:19:50.405365+0000 osd.0 (osd.0) 51 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:19:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:51 smithi190 bash[23868]: cluster 2024-04-26T16:19:50.405365+0000 osd.0 (osd.0) 51 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:19:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:51 smithi190 bash[23868]: cluster 2024-04-26T16:19:50.407695+0000 osd.0 (osd.0) 52 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:19:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:51 smithi190 bash[23868]: cluster 2024-04-26T16:19:50.407695+0000 osd.0 (osd.0) 52 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:19:51.856 INFO:teuthology.orchestra.run.smithi050.stdout:201863462970 2024-04-26T16:19:51.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T16:19:51.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:51 smithi050 bash[29234]: cluster 2024-04-26T16:19:50.407695+0000 osd.0 (osd.0) 52 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:19:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:52 smithi190 bash[23868]: cluster 2024-04-26T16:19:51.379718+0000 mgr.y (mgr.24401) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:52 smithi190 bash[23868]: cluster 2024-04-26T16:19:51.379718+0000 mgr.y (mgr.24401) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:52 smithi190 bash[23868]: cluster 2024-04-26T16:19:51.753903+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:19:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:52 smithi190 bash[23868]: cluster 2024-04-26T16:19:51.753903+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:19:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:52 smithi190 bash[23868]: cluster 2024-04-26T16:19:51.756972+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:19:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:52 smithi190 bash[23868]: cluster 2024-04-26T16:19:51.756972+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:19:52.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[21463]: cluster 2024-04-26T16:19:51.379718+0000 mgr.y (mgr.24401) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:52.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[21463]: cluster 2024-04-26T16:19:51.379718+0000 mgr.y (mgr.24401) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:52.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[21463]: cluster 2024-04-26T16:19:51.753903+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:19:52.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[21463]: cluster 2024-04-26T16:19:51.753903+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:19:52.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[21463]: cluster 2024-04-26T16:19:51.756972+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[21463]: cluster 2024-04-26T16:19:51.756972+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[29234]: cluster 2024-04-26T16:19:51.379718+0000 mgr.y (mgr.24401) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[29234]: cluster 2024-04-26T16:19:51.379718+0000 mgr.y (mgr.24401) 87 : cluster [DBG] pgmap v59: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[29234]: cluster 2024-04-26T16:19:51.753903+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[29234]: cluster 2024-04-26T16:19:51.753903+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[29234]: cluster 2024-04-26T16:19:51.756972+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:19:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:52 smithi050 bash[29234]: cluster 2024-04-26T16:19:51.756972+0000 osd.5 (osd.5) 50 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:19:53.084 INFO:teuthology.orchestra.run.smithi050.stdout:120259084366 2024-04-26T16:19:53.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T16:19:53.101 INFO:teuthology.orchestra.run.smithi050.stdout:150323855428 2024-04-26T16:19:53.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T16:19:53.228 INFO:teuthology.orchestra.run.smithi050.stdout:77309411417 2024-04-26T16:19:53.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T16:19:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:53 smithi190 bash[23868]: cluster 2024-04-26T16:19:52.391222+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:19:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:53 smithi190 bash[23868]: cluster 2024-04-26T16:19:52.391222+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:19:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:53 smithi190 bash[23868]: cluster 2024-04-26T16:19:52.393744+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:19:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:53 smithi190 bash[23868]: cluster 2024-04-26T16:19:52.393744+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:19:53.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[29234]: cluster 2024-04-26T16:19:52.391222+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:19:53.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[29234]: cluster 2024-04-26T16:19:52.391222+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:19:53.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[29234]: cluster 2024-04-26T16:19:52.393744+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:19:53.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[29234]: cluster 2024-04-26T16:19:52.393744+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:19:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[21463]: cluster 2024-04-26T16:19:52.391222+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:19:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[21463]: cluster 2024-04-26T16:19:52.391222+0000 osd.2 (osd.2) 35 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:19:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[21463]: cluster 2024-04-26T16:19:52.393744+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:19:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:53 smithi050 bash[21463]: cluster 2024-04-26T16:19:52.393744+0000 osd.2 (osd.2) 36 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:19:54.215 INFO:teuthology.orchestra.run.smithi050.stdout:296352743462 2024-04-26T16:19:54.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T16:19:54.223 INFO:teuthology.orchestra.run.smithi050.stdout:244813135920 2024-04-26T16:19:54.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T16:19:54.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:19:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:19:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:54 smithi190 bash[23868]: cluster 2024-04-26T16:19:53.380085+0000 mgr.y (mgr.24401) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:54 smithi190 bash[23868]: cluster 2024-04-26T16:19:53.380085+0000 mgr.y (mgr.24401) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:54 smithi190 bash[23868]: cluster 2024-04-26T16:19:53.745707+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-04-26T16:19:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:54 smithi190 bash[23868]: cluster 2024-04-26T16:19:53.745707+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-04-26T16:19:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:54 smithi190 bash[23868]: cluster 2024-04-26T16:19:53.748053+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-04-26T16:19:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:54 smithi190 bash[23868]: cluster 2024-04-26T16:19:53.748053+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-04-26T16:19:54.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21463]: cluster 2024-04-26T16:19:53.380085+0000 mgr.y (mgr.24401) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:54.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21463]: cluster 2024-04-26T16:19:53.380085+0000 mgr.y (mgr.24401) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:54.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21463]: cluster 2024-04-26T16:19:53.745707+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-04-26T16:19:54.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21463]: cluster 2024-04-26T16:19:53.745707+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-04-26T16:19:54.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21463]: cluster 2024-04-26T16:19:53.748053+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-04-26T16:19:54.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[21463]: cluster 2024-04-26T16:19:53.748053+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-04-26T16:19:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[29234]: cluster 2024-04-26T16:19:53.380085+0000 mgr.y (mgr.24401) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[29234]: cluster 2024-04-26T16:19:53.380085+0000 mgr.y (mgr.24401) 88 : cluster [DBG] pgmap v60: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[29234]: cluster 2024-04-26T16:19:53.745707+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-04-26T16:19:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[29234]: cluster 2024-04-26T16:19:53.745707+0000 osd.5 (osd.5) 51 : cluster [DBG] 6.a scrub starts 2024-04-26T16:19:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[29234]: cluster 2024-04-26T16:19:53.748053+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-04-26T16:19:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:54 smithi050 bash[29234]: cluster 2024-04-26T16:19:53.748053+0000 osd.5 (osd.5) 52 : cluster [DBG] 6.a scrub ok 2024-04-26T16:19:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: cluster 2024-04-26T16:19:54.347385+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:19:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: cluster 2024-04-26T16:19:54.347385+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:19:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: cluster 2024-04-26T16:19:54.349639+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:19:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: cluster 2024-04-26T16:19:54.349639+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:19:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: audit 2024-04-26T16:19:55.337466+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: audit 2024-04-26T16:19:55.337466+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: audit 2024-04-26T16:19:55.344337+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:55 smithi190 bash[23868]: audit 2024-04-26T16:19:55.344337+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: cluster 2024-04-26T16:19:54.347385+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:19:55.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: cluster 2024-04-26T16:19:54.347385+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:19:55.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: cluster 2024-04-26T16:19:54.349639+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:19:55.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: cluster 2024-04-26T16:19:54.349639+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: audit 2024-04-26T16:19:55.337466+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: audit 2024-04-26T16:19:55.337466+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: audit 2024-04-26T16:19:55.344337+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[21463]: audit 2024-04-26T16:19:55.344337+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: cluster 2024-04-26T16:19:54.347385+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: cluster 2024-04-26T16:19:54.347385+0000 osd.2 (osd.2) 37 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: cluster 2024-04-26T16:19:54.349639+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: cluster 2024-04-26T16:19:54.349639+0000 osd.2 (osd.2) 38 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: audit 2024-04-26T16:19:55.337466+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: audit 2024-04-26T16:19:55.337466+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: audit 2024-04-26T16:19:55.344337+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:55 smithi050 bash[29234]: audit 2024-04-26T16:19:55.344337+0000 mon.a (mon.0) 914 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:56.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:19:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: cluster 2024-04-26T16:19:55.380446+0000 mgr.y (mgr.24401) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: cluster 2024-04-26T16:19:55.380446+0000 mgr.y (mgr.24401) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: cluster 2024-04-26T16:19:55.483832+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: cluster 2024-04-26T16:19:55.483832+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: cluster 2024-04-26T16:19:55.486014+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: cluster 2024-04-26T16:19:55.486014+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.608516+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.608516+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.965332+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.965332+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.967203+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.967203+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.978435+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:56 smithi190 bash[23868]: audit 2024-04-26T16:19:55.978435+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: cluster 2024-04-26T16:19:55.380446+0000 mgr.y (mgr.24401) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: cluster 2024-04-26T16:19:55.380446+0000 mgr.y (mgr.24401) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: cluster 2024-04-26T16:19:55.483832+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: cluster 2024-04-26T16:19:55.483832+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: cluster 2024-04-26T16:19:55.486014+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: cluster 2024-04-26T16:19:55.486014+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.608516+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:56.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.608516+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.965332+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.965332+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.967203+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.967203+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.978435+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[21463]: audit 2024-04-26T16:19:55.978435+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: cluster 2024-04-26T16:19:55.380446+0000 mgr.y (mgr.24401) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: cluster 2024-04-26T16:19:55.380446+0000 mgr.y (mgr.24401) 89 : cluster [DBG] pgmap v61: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:19:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: cluster 2024-04-26T16:19:55.483832+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: cluster 2024-04-26T16:19:55.483832+0000 osd.7 (osd.7) 61 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: cluster 2024-04-26T16:19:55.486014+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: cluster 2024-04-26T16:19:55.486014+0000 osd.7 (osd.7) 62 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.608516+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.608516+0000 mon.c (mon.2) 62 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.965332+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.965332+0000 mon.c (mon.2) 63 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.967203+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.967203+0000 mon.c (mon.2) 64 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.978435+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:56.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:56 smithi050 bash[29234]: audit 2024-04-26T16:19:55.978435+0000 mon.a (mon.0) 915 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:19:57.010 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:57.011 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:57.011 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:57.012 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:57.013 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:57.014 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: cluster 2024-04-26T16:19:56.226614+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: cluster 2024-04-26T16:19:56.226614+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: cluster 2024-04-26T16:19:56.226743+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:57.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: cluster 2024-04-26T16:19:56.226743+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:57.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: audit 2024-04-26T16:19:56.407879+0000 mgr.y (mgr.24401) 90 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:57.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: audit 2024-04-26T16:19:56.407879+0000 mgr.y (mgr.24401) 90 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:57.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: cluster 2024-04-26T16:19:56.982888+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-04-26T16:19:57.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:57 smithi190 bash[23868]: cluster 2024-04-26T16:19:56.982888+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-04-26T16:19:57.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: cluster 2024-04-26T16:19:56.226614+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:57.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: cluster 2024-04-26T16:19:56.226614+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:57.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: cluster 2024-04-26T16:19:56.226743+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:57.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: cluster 2024-04-26T16:19:56.226743+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:57.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: audit 2024-04-26T16:19:56.407879+0000 mgr.y (mgr.24401) 90 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:57.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: audit 2024-04-26T16:19:56.407879+0000 mgr.y (mgr.24401) 90 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: cluster 2024-04-26T16:19:56.982888+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[29234]: cluster 2024-04-26T16:19:56.982888+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: cluster 2024-04-26T16:19:56.226614+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: cluster 2024-04-26T16:19:56.226614+0000 osd.2 (osd.2) 39 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: cluster 2024-04-26T16:19:56.226743+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: cluster 2024-04-26T16:19:56.226743+0000 osd.2 (osd.2) 40 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: audit 2024-04-26T16:19:56.407879+0000 mgr.y (mgr.24401) 90 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: audit 2024-04-26T16:19:56.407879+0000 mgr.y (mgr.24401) 90 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: cluster 2024-04-26T16:19:56.982888+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-04-26T16:19:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:57 smithi050 bash[21463]: cluster 2024-04-26T16:19:56.982888+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2024-04-26T16:19:58.023 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:58.024 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:19:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.381095+0000 mgr.y (mgr.24401) 91 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-26T16:19:58.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.381095+0000 mgr.y (mgr.24401) 91 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-26T16:19:58.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.394707+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.b scrub starts 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.394707+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.b scrub starts 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.396973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.b scrub ok 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.396973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.b scrub ok 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.993472+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.993472+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.996155+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:19:58.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:58 smithi190 bash[23868]: cluster 2024-04-26T16:19:57.996155+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.381095+0000 mgr.y (mgr.24401) 91 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.381095+0000 mgr.y (mgr.24401) 91 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.394707+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.b scrub starts 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.394707+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.b scrub starts 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.396973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.b scrub ok 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.396973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.b scrub ok 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.993472+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.993472+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.996155+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[29234]: cluster 2024-04-26T16:19:57.996155+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:19:58.919 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.381095+0000 mgr.y (mgr.24401) 91 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.381095+0000 mgr.y (mgr.24401) 91 : cluster [DBG] pgmap v63: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.394707+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.b scrub starts 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.394707+0000 osd.2 (osd.2) 41 : cluster [DBG] 5.b scrub starts 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.396973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.b scrub ok 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.396973+0000 osd.2 (osd.2) 42 : cluster [DBG] 5.b scrub ok 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.993472+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.993472+0000 osd.4 (osd.4) 43 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.996155+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:19:58.920 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:58 smithi050 bash[21463]: cluster 2024-04-26T16:19:57.996155+0000 osd.4 (osd.4) 44 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.722480+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.722480+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.724749+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.724749+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.915148+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.915148+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:19:59.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.917470+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:19:59.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:19:59 smithi190 bash[23868]: cluster 2024-04-26T16:19:58.917470+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.722480+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.722480+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.724749+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.724749+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.915148+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.915148+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.917470+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[29234]: cluster 2024-04-26T16:19:58.917470+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.722480+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.722480+0000 osd.5 (osd.5) 53 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:19:59.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.724749+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:19:59.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.724749+0000 osd.5 (osd.5) 54 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:19:59.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.915148+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:19:59.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.915148+0000 osd.6 (osd.6) 29 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:19:59.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.917470+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:19:59.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:19:59 smithi050 bash[21463]: cluster 2024-04-26T16:19:58.917470+0000 osd.6 (osd.6) 30 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:20:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.125840+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.125840+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.129157+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.129157+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.356015+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.356015+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.358771+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.358771+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.381804+0000 mgr.y (mgr.24401) 92 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.381804+0000 mgr.y (mgr.24401) 92 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.874272+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.874272+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.877767+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:19:59.877767+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:20:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:00.000110+0000 mon.a (mon.0) 917 : cluster [INF] overall HEALTH_OK 2024-04-26T16:20:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:00.000110+0000 mon.a (mon.0) 917 : cluster [INF] overall HEALTH_OK 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.125840+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.125840+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.129157+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.129157+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.356015+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.356015+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:20:00.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.358771+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.358771+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.381804+0000 mgr.y (mgr.24401) 92 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.381804+0000 mgr.y (mgr.24401) 92 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.874272+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.874272+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.877767+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:19:59.877767+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:00.000110+0000 mon.a (mon.0) 917 : cluster [INF] overall HEALTH_OK 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:00.000110+0000 mon.a (mon.0) 917 : cluster [INF] overall HEALTH_OK 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.125840+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:20:00.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.125840+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.129157+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.129157+0000 osd.1 (osd.1) 56 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.356015+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.356015+0000 osd.2 (osd.2) 43 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.358771+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.358771+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.381804+0000 mgr.y (mgr.24401) 92 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.381804+0000 mgr.y (mgr.24401) 92 : cluster [DBG] pgmap v64: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.874272+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.874272+0000 osd.6 (osd.6) 31 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.877767+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:19:59.877767+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:00.000110+0000 mon.a (mon.0) 917 : cluster [INF] overall HEALTH_OK 2024-04-26T16:20:00.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:00.000110+0000 mon.a (mon.0) 917 : cluster [INF] overall HEALTH_OK 2024-04-26T16:20:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:01 smithi190 bash[23868]: cluster 2024-04-26T16:20:00.351439+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:20:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:01 smithi190 bash[23868]: cluster 2024-04-26T16:20:00.351439+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:20:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:01 smithi190 bash[23868]: cluster 2024-04-26T16:20:00.355237+0000 osd.0 (osd.0) 54 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:20:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:01 smithi190 bash[23868]: cluster 2024-04-26T16:20:00.355237+0000 osd.0 (osd.0) 54 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:20:01.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[29234]: cluster 2024-04-26T16:20:00.351439+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[29234]: cluster 2024-04-26T16:20:00.351439+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[29234]: cluster 2024-04-26T16:20:00.355237+0000 osd.0 (osd.0) 54 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[29234]: cluster 2024-04-26T16:20:00.355237+0000 osd.0 (osd.0) 54 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[21463]: cluster 2024-04-26T16:20:00.351439+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[21463]: cluster 2024-04-26T16:20:00.351439+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[21463]: cluster 2024-04-26T16:20:00.355237+0000 osd.0 (osd.0) 54 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:20:01.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:01 smithi050 bash[21463]: cluster 2024-04-26T16:20:00.355237+0000 osd.0 (osd.0) 54 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:20:02.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:02 smithi190 bash[23868]: cluster 2024-04-26T16:20:01.383338+0000 mgr.y (mgr.24401) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:02.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:02 smithi190 bash[23868]: cluster 2024-04-26T16:20:01.383338+0000 mgr.y (mgr.24401) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:02.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:02 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.035360+0000 osd.4 (osd.4) 45 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:20:02.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:02 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.035360+0000 osd.4 (osd.4) 45 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:20:02.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:02 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.037888+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:20:02.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:02 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.037888+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:20:02.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[21463]: cluster 2024-04-26T16:20:01.383338+0000 mgr.y (mgr.24401) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:02.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[21463]: cluster 2024-04-26T16:20:01.383338+0000 mgr.y (mgr.24401) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.035360+0000 osd.4 (osd.4) 45 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.035360+0000 osd.4 (osd.4) 45 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.037888+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.037888+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[29234]: cluster 2024-04-26T16:20:01.383338+0000 mgr.y (mgr.24401) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[29234]: cluster 2024-04-26T16:20:01.383338+0000 mgr.y (mgr.24401) 93 : cluster [DBG] pgmap v65: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.035360+0000 osd.4 (osd.4) 45 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.035360+0000 osd.4 (osd.4) 45 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.037888+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:20:02.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:02 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.037888+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:20:03.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.711034+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:20:03.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.711034+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:20:03.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.714453+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:20:03.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.714453+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:20:03.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.931814+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:20:03.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.931814+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:20:03.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.933901+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:20:03.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:03 smithi190 bash[23868]: cluster 2024-04-26T16:20:02.933901+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:20:03.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.711034+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:20:03.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.711034+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:20:03.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.714453+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.714453+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.931814+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.931814+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.933901+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[29234]: cluster 2024-04-26T16:20:02.933901+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.711034+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.711034+0000 osd.5 (osd.5) 55 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.714453+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.714453+0000 osd.5 (osd.5) 56 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.931814+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:20:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.931814+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:20:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.933901+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:20:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:03 smithi050 bash[21463]: cluster 2024-04-26T16:20:02.933901+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:20:04.770 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:20:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:20:04.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.146687+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:20:04.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.146687+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:20:04.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.149588+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:20:04.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.149588+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:20:04.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.384015+0000 mgr.y (mgr.24401) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.384015+0000 mgr.y (mgr.24401) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.388028+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.388028+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.390487+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.390487+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.402139+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.402139+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.405231+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.405231+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.945336+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.945336+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.946945+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:20:04.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[29234]: cluster 2024-04-26T16:20:03.946945+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.146687+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.146687+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.149588+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.149588+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.384015+0000 mgr.y (mgr.24401) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.384015+0000 mgr.y (mgr.24401) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:04.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.388028+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.388028+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.390487+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.390487+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.402139+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.402139+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.405231+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.405231+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.945336+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.945336+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.946945+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:20:04.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:04 smithi050 bash[21463]: cluster 2024-04-26T16:20:03.946945+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:20:04.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.146687+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:20:04.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.146687+0000 osd.1 (osd.1) 57 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:20:04.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.149588+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.149588+0000 osd.1 (osd.1) 58 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.384015+0000 mgr.y (mgr.24401) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.384015+0000 mgr.y (mgr.24401) 94 : cluster [DBG] pgmap v66: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.388028+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.388028+0000 osd.0 (osd.0) 55 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.390487+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.390487+0000 osd.0 (osd.0) 56 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.402139+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.402139+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.d scrub starts 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.405231+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.405231+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.d scrub ok 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.945336+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:20:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.945336+0000 osd.3 (osd.3) 57 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:20:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.946945+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:20:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:04 smithi190 bash[23868]: cluster 2024-04-26T16:20:03.946945+0000 osd.3 (osd.3) 58 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:20:05.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.344610+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:20:05.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.344610+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:20:05.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.348288+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.348288+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.990223+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.990223+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.994511+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:04.994511+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.016698+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.016698+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.018926+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:20:05.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:05 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.018926+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.344610+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.344610+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.348288+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.348288+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.990223+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.990223+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.994511+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:04.994511+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:20:05.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.016698+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.016698+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.018926+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.018926+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.344610+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.344610+0000 osd.0 (osd.0) 57 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.348288+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.348288+0000 osd.0 (osd.0) 58 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.990223+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.990223+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.994511+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:04.994511+0000 osd.3 (osd.3) 60 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:20:05.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.016698+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:20:05.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.016698+0000 osd.4 (osd.4) 47 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:20:05.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.018926+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:20:05.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:05 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.018926+0000 osd.4 (osd.4) 48 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.384722+0000 mgr.y (mgr.24401) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.384722+0000 mgr.y (mgr.24401) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.434307+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.c scrub starts 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.434307+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.c scrub starts 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.436912+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.c scrub ok 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.436912+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.c scrub ok 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.682420+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.682420+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.684610+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:20:06.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[29234]: cluster 2024-04-26T16:20:05.684610+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.384722+0000 mgr.y (mgr.24401) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.384722+0000 mgr.y (mgr.24401) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.434307+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.c scrub starts 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.434307+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.c scrub starts 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.436912+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.c scrub ok 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.436912+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.c scrub ok 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.682420+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.682420+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.684610+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:20:06.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:06 smithi050 bash[21463]: cluster 2024-04-26T16:20:05.684610+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:20:06.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:20:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.384722+0000 mgr.y (mgr.24401) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.384722+0000 mgr.y (mgr.24401) 95 : cluster [DBG] pgmap v67: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.434307+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.c scrub starts 2024-04-26T16:20:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.434307+0000 osd.2 (osd.2) 47 : cluster [DBG] 2.c scrub starts 2024-04-26T16:20:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.436912+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.c scrub ok 2024-04-26T16:20:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.436912+0000 osd.2 (osd.2) 48 : cluster [DBG] 2.c scrub ok 2024-04-26T16:20:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.682420+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:20:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.682420+0000 osd.5 (osd.5) 57 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:20:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.684610+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:20:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:06 smithi190 bash[23868]: cluster 2024-04-26T16:20:05.684610+0000 osd.5 (osd.5) 58 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3579760943 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f68105630 msgr2=0x7f0f68105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3579760943 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f68105630 0x7f0f68105ab0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009990 tx=0x7f0f58009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3579760943 shutdown_connections 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3579760943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f68105ff0 0x7f0f68112500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3579760943 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f6810b630 0x7f0f6810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3579760943 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f68105630 0x7f0f68105ab0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3579760943 >> 172.21.15.50:0/3579760943 conn(0x7f0f68101120 msgr2=0x7f0f68103540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3579760943 shutdown_connections 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3579760943 wait complete. 2024-04-26T16:20:07.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 librados: starting msgr at 2024-04-26T16:20:07.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 librados: starting objecter 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 Processor -- start 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- start start 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 librados: setting wanted keys 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 librados: calling monclient init 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f68105630 0x7f0f6819b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68105ff0 0x7f0f6819b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 0x7f0f6819fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0f68114d90 con 0x7f0f68105ff0 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0f68114c10 con 0x7f0f68105630 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0f68114f10 con 0x7f0f6810b630 2024-04-26T16:20:07.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 0x7f0f6819fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 0x7f0f6819fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55192/0 (socket says 172.21.15.50:55192) 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 -- 172.21.15.50:0/3590310051 learned_addr learned my addr 172.21.15.50:0/3590310051 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6cc57700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68105ff0 0x7f0f6819b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6d458700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f68105630 0x7f0f6819b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 -- 172.21.15.50:0/3590310051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f68105630 msgr2=0x7f0f6819b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f68105630 0x7f0f6819b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 -- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68105ff0 msgr2=0x7f0f6819b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68105ff0 0x7f0f6819b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 -- 172.21.15.50:0/3590310051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f681a0370 con 0x7f0f6810b630 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6dc59700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 0x7f0f6819fc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f60007ee0 tx=0x7f0f6000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f5e7fc700 10 client.?.objecter ms_handle_connect 0x7f0f6810b630 2024-04-26T16:20:07.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f5e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6000dbd0 con 0x7f0f6810b630 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f681a0660 con 0x7f0f6810b630 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0f6000dd70 con 0x7f0f6810b630 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f60005820 con 0x7f0f6810b630 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 10 client.24521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f681a7ee0 con 0x7f0f6810b630 2024-04-26T16:20:07.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.255+0000 7f0f6f6bc700 1 librados: init done 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f6806a110 con 0x7f0f6810b630 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0f6001a020 con 0x7f0f6810b630 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0f54071f80 0x7f0f54074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7f0f60090a40 con 0x7f0f6810b630 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 10 client.24521.objecter ms_dispatch 0x7f0f68001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 3 client.24521.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 3 client.24521.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:07.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f6d458700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0f54071f80 0x7f0f54074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 20 client.24521.objecter dump_active .. 0 homeless 2024-04-26T16:20:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f5e7fc700 10 client.24521.objecter ms_handle_connect 0x7f0f54071f80 2024-04-26T16:20:07.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.259+0000 7f0f6d458700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0f54071f80 0x7f0f54074440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009bd0 tx=0x7f0f64009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:07.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.263+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f0f60010070 con 0x7f0f6810b630 2024-04-26T16:20:07.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0f6802e490 con 0x7f0f6810b630 2024-04-26T16:20:07.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f5e7fc700 1 -- 172.21.15.50:0/3590310051 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0f6005cbd0 con 0x7f0f6810b630 2024-04-26T16:20:07.493 INFO:teuthology.orchestra.run.smithi050.stdout:120259084369 2024-04-26T16:20:07.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f6f6bc700 10 librados: watch_flush enter 2024-04-26T16:20:07.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f6f6bc700 10 librados: watch_flush exit 2024-04-26T16:20:07.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f6f6bc700 20 client.24521.objecter shutdown clearing up homeless session... 2024-04-26T16:20:07.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f6f6bc700 10 client.24521.objecter successfully canceled tick 2024-04-26T16:20:07.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.499+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0f54071f80 msgr2=0x7f0f54074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:07.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0f54071f80 0x7f0f54074440 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009bd0 tx=0x7f0f64009260 comp rx=0 tx=0).stop 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 msgr2=0x7f0f6819fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 0x7f0f6819fc30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0f60007ee0 tx=0x7f0f6000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 shutdown_connections 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0f54071f80 0x7f0f54074440 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f68105630 0x7f0f6819b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0f6810b630 0x7f0f6819fc30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 --2- 172.21.15.50:0/3590310051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0f68105ff0 0x7f0f6819b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:07.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 >> 172.21.15.50:0/3590310051 conn(0x7f0f68101120 msgr2=0x7f0f68101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 shutdown_connections 2024-04-26T16:20:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 -- 172.21.15.50:0/3590310051 wait complete. 2024-04-26T16:20:07.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:07.503+0000 7f0f6f6bc700 1 librados: shutdown 2024-04-26T16:20:07.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: audit 2024-04-26T16:20:06.419202+0000 mgr.y (mgr.24401) 96 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: audit 2024-04-26T16:20:06.419202+0000 mgr.y (mgr.24401) 96 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.713208+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.713208+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.715437+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.715437+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.965840+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.965840+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.968051+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:06.968051+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.026286+0000 osd.3 (osd.3) 61 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.026286+0000 osd.3 (osd.3) 61 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.028714+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.028714+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:20:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: audit 2024-04-26T16:20:07.503737+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? 172.21.15.50:0/3590310051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:20:07.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:07 smithi190 bash[23868]: audit 2024-04-26T16:20:07.503737+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? 172.21.15.50:0/3590310051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: audit 2024-04-26T16:20:06.419202+0000 mgr.y (mgr.24401) 96 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: audit 2024-04-26T16:20:06.419202+0000 mgr.y (mgr.24401) 96 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.713208+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.713208+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.715437+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.715437+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.965840+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:20:07.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.965840+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.968051+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:06.968051+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.026286+0000 osd.3 (osd.3) 61 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.026286+0000 osd.3 (osd.3) 61 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.028714+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.028714+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: audit 2024-04-26T16:20:07.503737+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? 172.21.15.50:0/3590310051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[21463]: audit 2024-04-26T16:20:07.503737+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? 172.21.15.50:0/3590310051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: audit 2024-04-26T16:20:06.419202+0000 mgr.y (mgr.24401) 96 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: audit 2024-04-26T16:20:06.419202+0000 mgr.y (mgr.24401) 96 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.713208+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:20:07.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.713208+0000 osd.5 (osd.5) 59 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.715437+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.715437+0000 osd.5 (osd.5) 60 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.965840+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.965840+0000 osd.4 (osd.4) 49 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.968051+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:06.968051+0000 osd.4 (osd.4) 50 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.026286+0000 osd.3 (osd.3) 61 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.026286+0000 osd.3 (osd.3) 61 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.028714+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.028714+0000 osd.3 (osd.3) 62 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: audit 2024-04-26T16:20:07.503737+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? 172.21.15.50:0/3590310051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:20:07.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:07 smithi050 bash[29234]: audit 2024-04-26T16:20:07.503737+0000 mon.c (mon.2) 65 : audit [DBG] from='client.? 172.21.15.50:0/3590310051' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:20:08.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:08 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.386169+0000 mgr.y (mgr.24401) 97 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-26T16:20:08.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:08 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.386169+0000 mgr.y (mgr.24401) 97 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-26T16:20:08.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:08 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.998249+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:20:08.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:08 smithi190 bash[23868]: cluster 2024-04-26T16:20:07.998249+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:20:08.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:08 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.000825+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:20:08.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:08 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.000825+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.386169+0000 mgr.y (mgr.24401) 97 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.386169+0000 mgr.y (mgr.24401) 97 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.998249+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[29234]: cluster 2024-04-26T16:20:07.998249+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.000825+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.000825+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.386169+0000 mgr.y (mgr.24401) 97 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-26T16:20:08.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.386169+0000 mgr.y (mgr.24401) 97 : cluster [DBG] pgmap v68: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-04-26T16:20:08.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.998249+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:20:08.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[21463]: cluster 2024-04-26T16:20:07.998249+0000 osd.3 (osd.3) 63 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:20:08.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.000825+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:20:08.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:08 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.000825+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:20:09.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.655045+0000 osd.5 (osd.5) 61 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:20:09.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.655045+0000 osd.5 (osd.5) 61 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:20:09.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.659142+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:20:09.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.659142+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:20:09.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.827964+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.827964+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.830435+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:08.830435+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.043536+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.043536+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.045976+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:20:09.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:09 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.045976+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:20:09.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.655045+0000 osd.5 (osd.5) 61 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:20:09.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.655045+0000 osd.5 (osd.5) 61 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:20:09.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.659142+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.659142+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.827964+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.827964+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.830435+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:08.830435+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.043536+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.043536+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.045976+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.045976+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.655045+0000 osd.5 (osd.5) 61 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:20:09.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.655045+0000 osd.5 (osd.5) 61 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.659142+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.659142+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.827964+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.827964+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.830435+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:08.830435+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.043536+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.043536+0000 osd.3 (osd.3) 65 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.045976+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:20:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:09 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.045976+0000 osd.3 (osd.3) 66 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:20:10.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.302001+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:20:10.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.302001+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:20:10.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.305977+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:20:10.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.305977+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.386859+0000 mgr.y (mgr.24401) 98 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.386859+0000 mgr.y (mgr.24401) 98 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.418438+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.a scrub starts 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.418438+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.a scrub starts 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.420756+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.a scrub ok 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:09.420756+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.a scrub ok 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.024115+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.c scrub starts 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.024115+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.c scrub starts 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.026649+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.c scrub ok 2024-04-26T16:20:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:10 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.026649+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.c scrub ok 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.302001+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.302001+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.305977+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.305977+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.386859+0000 mgr.y (mgr.24401) 98 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.386859+0000 mgr.y (mgr.24401) 98 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.418438+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.a scrub starts 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.418438+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.a scrub starts 2024-04-26T16:20:10.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.420756+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.a scrub ok 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:09.420756+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.a scrub ok 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.024115+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.c scrub starts 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.024115+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.c scrub starts 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.026649+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.c scrub ok 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.026649+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.c scrub ok 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.302001+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.302001+0000 osd.7 (osd.7) 63 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.305977+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.305977+0000 osd.7 (osd.7) 64 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.386859+0000 mgr.y (mgr.24401) 98 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.386859+0000 mgr.y (mgr.24401) 98 : cluster [DBG] pgmap v69: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.418438+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.a scrub starts 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.418438+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.a scrub starts 2024-04-26T16:20:10.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.420756+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.a scrub ok 2024-04-26T16:20:10.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:09.420756+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.a scrub ok 2024-04-26T16:20:10.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.024115+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.c scrub starts 2024-04-26T16:20:10.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.024115+0000 osd.3 (osd.3) 67 : cluster [DBG] 6.c scrub starts 2024-04-26T16:20:10.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.026649+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.c scrub ok 2024-04-26T16:20:10.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:10 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.026649+0000 osd.3 (osd.3) 68 : cluster [DBG] 6.c scrub ok 2024-04-26T16:20:11.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.125262+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:20:11.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.125262+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:20:11.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.125262+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:20:11.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.125262+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:20:11.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.127939+0000 osd.1 (osd.1) 60 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.127939+0000 osd.1 (osd.1) 60 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.308586+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.308586+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.311462+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:10.311462+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: audit 2024-04-26T16:20:10.638548+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: audit 2024-04-26T16:20:10.638548+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.070373+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.070373+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.073381+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:20:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:11 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.073381+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.127939+0000 osd.1 (osd.1) 60 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.127939+0000 osd.1 (osd.1) 60 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.308586+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.308586+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.311462+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:10.311462+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: audit 2024-04-26T16:20:10.638548+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: audit 2024-04-26T16:20:10.638548+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:12.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.070373+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:20:12.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.070373+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:20:12.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.073381+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:20:12.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.073381+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:20:12.142 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.125262+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:20:12.142 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.125262+0000 osd.1 (osd.1) 59 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:20:12.142 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.127939+0000 osd.1 (osd.1) 60 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.127939+0000 osd.1 (osd.1) 60 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.308586+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.308586+0000 osd.7 (osd.7) 65 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.311462+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:10.311462+0000 osd.7 (osd.7) 66 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: audit 2024-04-26T16:20:10.638548+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: audit 2024-04-26T16:20:10.638548+0000 mon.c (mon.2) 66 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.070373+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.070373+0000 osd.3 (osd.3) 69 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.073381+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:20:12.143 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:11 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.073381+0000 osd.3 (osd.3) 70 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:20:12.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:12 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.387901+0000 mgr.y (mgr.24401) 99 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:12.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:12 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.387901+0000 mgr.y (mgr.24401) 99 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:12.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:12 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.899075+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:20:12.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:12 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.899075+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:20:12.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:12 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.904278+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:20:12.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:12 smithi190 bash[23868]: cluster 2024-04-26T16:20:11.904278+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:20:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.387901+0000 mgr.y (mgr.24401) 99 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.387901+0000 mgr.y (mgr.24401) 99 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.899075+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:20:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.899075+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:20:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.904278+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:20:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[21463]: cluster 2024-04-26T16:20:11.904278+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:20:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.387901+0000 mgr.y (mgr.24401) 99 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:12.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.387901+0000 mgr.y (mgr.24401) 99 : cluster [DBG] pgmap v70: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:12.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.899075+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:20:12.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.899075+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:20:12.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.904278+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:20:12.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:12 smithi050 bash[29234]: cluster 2024-04-26T16:20:11.904278+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:20:13.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:13 smithi190 bash[23868]: cluster 2024-04-26T16:20:12.896029+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:20:13.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:13 smithi190 bash[23868]: cluster 2024-04-26T16:20:12.896029+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:20:13.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:13 smithi190 bash[23868]: cluster 2024-04-26T16:20:12.898421+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:20:13.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:13 smithi190 bash[23868]: cluster 2024-04-26T16:20:12.898421+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:20:13.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[21463]: cluster 2024-04-26T16:20:12.896029+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:20:13.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[21463]: cluster 2024-04-26T16:20:12.896029+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:20:13.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[21463]: cluster 2024-04-26T16:20:12.898421+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:20:13.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[21463]: cluster 2024-04-26T16:20:12.898421+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:20:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[29234]: cluster 2024-04-26T16:20:12.896029+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:20:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[29234]: cluster 2024-04-26T16:20:12.896029+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:20:13.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[29234]: cluster 2024-04-26T16:20:12.898421+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:20:13.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:13 smithi050 bash[29234]: cluster 2024-04-26T16:20:12.898421+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:20:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 -- 172.21.15.50:0/2595348808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc108340 msgr2=0x7f29cc06b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- 172.21.15.50:0/2595348808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc108340 0x7f29cc06b9c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009990 tx=0x7f29c8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 -- 172.21.15.50:0/2595348808 shutdown_connections 2024-04-26T16:20:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- 172.21.15.50:0/2595348808 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc105e50 0x7f29cc1062e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- 172.21.15.50:0/2595348808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc108340 0x7f29cc06b9c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- 172.21.15.50:0/2595348808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29cc06bf00 0x7f29cc06c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 -- 172.21.15.50:0/2595348808 >> 172.21.15.50:0/2595348808 conn(0x7f29cc079290 msgr2=0x7f29cc0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 -- 172.21.15.50:0/2595348808 shutdown_connections 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 -- 172.21.15.50:0/2595348808 wait complete. 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 librados: starting msgr at 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 librados: starting objecter 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 Processor -- start 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 -- start start 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 librados: setting wanted keys 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 librados: calling monclient init 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 0x7f29cc19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc105e50 0x7f29cc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.678+0000 7f29d374d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29cc108340 0x7f29cc1a3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc119120 con 0x7f29cc105e50 2024-04-26T16:20:14.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f29cc118fa0 con 0x7f29cc108340 2024-04-26T16:20:14.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f29cc1192a0 con 0x7f29cc06bf00 2024-04-26T16:20:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 0x7f29cc19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d0ce8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc105e50 0x7f29cc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 0x7f29cc19f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55058/0 (socket says 172.21.15.50:55058) 2024-04-26T16:20:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 -- 172.21.15.50:0/1597091111 learned_addr learned my addr 172.21.15.50:0/1597091111 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:14.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d1cea700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29cc108340 0x7f29cc1a3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 -- 172.21.15.50:0/1597091111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29cc108340 msgr2=0x7f29cc1a3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29cc108340 0x7f29cc1a3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 -- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc105e50 msgr2=0x7f29cc19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc105e50 0x7f29cc19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 -- 172.21.15.50:0/1597091111 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29cc1a46e0 con 0x7f29cc06bf00 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d0ce8700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc105e50 0x7f29cc19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d14e9700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 0x7f29cc19f6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f29c8000c00 tx=0x7f29c8004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 10 client.?.objecter ms_handle_connect 0x7f29cc06bf00 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8027070 con 0x7f29cc06bf00 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f29c8022470 con 0x7f29cc06bf00 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29cc1a4970 con 0x7f29cc06bf00 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c8017650 con 0x7f29cc06bf00 2024-04-26T16:20:14.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 10 client.24527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29cc1ac250 con 0x7f29cc06bf00 2024-04-26T16:20:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 librados: init done 2024-04-26T16:20:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29cc102930 con 0x7f29cc06bf00 2024-04-26T16:20:14.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f29c802a070 con 0x7f29cc06bf00 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f29b8071f80 0x7f29b8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7f29c80921f0 con 0x7f29cc06bf00 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 10 client.24527.objecter ms_dispatch 0x7f29cc001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 3 client.24527.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 3 client.24527.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29d0ce8700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f29b8071f80 0x7f29b8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.682+0000 7f29c27fc700 20 client.24527.objecter dump_active .. 0 homeless 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.686+0000 7f29c27fc700 10 client.24527.objecter ms_handle_connect 0x7f29b8071f80 2024-04-26T16:20:14.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.686+0000 7f29d0ce8700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f29b8071f80 0x7f29b8074440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f29cc105b80 tx=0x7f29bc00a380 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:14.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.690+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f29c8013070 con 0x7f29cc06bf00 2024-04-26T16:20:14.708 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:20:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:20:14.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.286178+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:20:14.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.286178+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:20:14.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.290606+0000 osd.0 (osd.0) 60 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:20:14.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.290606+0000 osd.0 (osd.0) 60 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:20:14.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.388555+0000 mgr.y (mgr.24401) 100 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:14.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.388555+0000 mgr.y (mgr.24401) 100 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:14.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.523061+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:20:14.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.523061+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:20:14.737 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084366 got 120259084369 for osd.3 2024-04-26T16:20:14.738 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:14.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.286178+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:20:14.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.286178+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:20:14.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.290606+0000 osd.0 (osd.0) 60 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:20:14.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.290606+0000 osd.0 (osd.0) 60 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.388555+0000 mgr.y (mgr.24401) 100 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.388555+0000 mgr.y (mgr.24401) 100 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.523061+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.523061+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.525142+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.525142+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.886882+0000 osd.4 (osd.4) 51 : cluster [DBG] 2.f scrub starts 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.886882+0000 osd.4 (osd.4) 51 : cluster [DBG] 2.f scrub starts 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.890098+0000 osd.4 (osd.4) 52 : cluster [DBG] 2.f scrub ok 2024-04-26T16:20:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:14 smithi190 bash[23868]: cluster 2024-04-26T16:20:13.890098+0000 osd.4 (osd.4) 52 : cluster [DBG] 2.f scrub ok 2024-04-26T16:20:14.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f29cc1a0a30 con 0x7f29cc06bf00 2024-04-26T16:20:14.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29c27fc700 1 -- 172.21.15.50:0/1597091111 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f29c805e380 con 0x7f29cc06bf00 2024-04-26T16:20:14.906 INFO:teuthology.orchestra.run.smithi050.stdout:77309411422 2024-04-26T16:20:14.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 10 librados: watch_flush enter 2024-04-26T16:20:14.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 10 librados: watch_flush exit 2024-04-26T16:20:14.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 20 client.24527.objecter shutdown clearing up homeless session... 2024-04-26T16:20:14.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 10 client.24527.objecter successfully canceled tick 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f29b8071f80 msgr2=0x7f29b8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f29b8071f80 0x7f29b8074440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f29cc105b80 tx=0x7f29bc00a380 comp rx=0 tx=0).stop 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 msgr2=0x7f29cc19f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 0x7f29cc19f6d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f29c8000c00 tx=0x7f29c8004790 comp rx=0 tx=0).stop 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 shutdown_connections 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f29b8071f80 0x7f29b8074440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f29cc108340 0x7f29cc1a3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f29cc06bf00 0x7f29cc19f6d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 --2- 172.21.15.50:0/1597091111 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f29cc105e50 0x7f29cc19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.914+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 >> 172.21.15.50:0/1597091111 conn(0x7f29cc079290 msgr2=0x7f29cc103cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.918+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 shutdown_connections 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.918+0000 7f29d374d700 1 -- 172.21.15.50:0/1597091111 wait complete. 2024-04-26T16:20:14.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:14.918+0000 7f29d374d700 1 librados: shutdown 2024-04-26T16:20:15.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.525142+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:20:15.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.525142+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:20:15.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.886882+0000 osd.4 (osd.4) 51 : cluster [DBG] 2.f scrub starts 2024-04-26T16:20:15.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.886882+0000 osd.4 (osd.4) 51 : cluster [DBG] 2.f scrub starts 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.890098+0000 osd.4 (osd.4) 52 : cluster [DBG] 2.f scrub ok 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[21463]: cluster 2024-04-26T16:20:13.890098+0000 osd.4 (osd.4) 52 : cluster [DBG] 2.f scrub ok 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.286178+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.286178+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.290606+0000 osd.0 (osd.0) 60 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.290606+0000 osd.0 (osd.0) 60 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.388555+0000 mgr.y (mgr.24401) 100 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.388555+0000 mgr.y (mgr.24401) 100 : cluster [DBG] pgmap v71: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:15.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.523061+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.523061+0000 osd.5 (osd.5) 63 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.525142+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.525142+0000 osd.5 (osd.5) 64 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.886882+0000 osd.4 (osd.4) 51 : cluster [DBG] 2.f scrub starts 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.886882+0000 osd.4 (osd.4) 51 : cluster [DBG] 2.f scrub starts 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.890098+0000 osd.4 (osd.4) 52 : cluster [DBG] 2.f scrub ok 2024-04-26T16:20:15.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:14 smithi050 bash[29234]: cluster 2024-04-26T16:20:13.890098+0000 osd.4 (osd.4) 52 : cluster [DBG] 2.f scrub ok 2024-04-26T16:20:15.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.272852+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:20:15.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.272852+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:20:15.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.275169+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:20:15.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.275169+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:20:15.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.277661+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:20:15.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.277661+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.277994+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.277994+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.404718+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.404718+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.406795+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.406795+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.905458+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.905458+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.907537+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: cluster 2024-04-26T16:20:14.907537+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-04-26T16:20:15.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: audit 2024-04-26T16:20:14.916279+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.50:0/1597091111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[29234]: audit 2024-04-26T16:20:14.916279+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.50:0/1597091111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.272852+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.272852+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.275169+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.275169+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.277661+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.277661+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.277994+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.277994+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.404718+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:20:15.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.404718+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.406795+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.406795+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.905458+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.905458+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.907537+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: cluster 2024-04-26T16:20:14.907537+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: audit 2024-04-26T16:20:14.916279+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.50:0/1597091111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:20:15.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:15 smithi050 bash[21463]: audit 2024-04-26T16:20:14.916279+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.50:0/1597091111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:20:16.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.272852+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:20:16.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.272852+0000 osd.7 (osd.7) 67 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.275169+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.275169+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.277661+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.277661+0000 osd.0 (osd.0) 62 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.277994+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.277994+0000 osd.7 (osd.7) 68 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.404718+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.404718+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.406795+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.406795+0000 osd.2 (osd.2) 52 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.905458+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.905458+0000 osd.4 (osd.4) 53 : cluster [DBG] 6.e scrub starts 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.907537+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: cluster 2024-04-26T16:20:14.907537+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.e scrub ok 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: audit 2024-04-26T16:20:14.916279+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.50:0/1597091111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:20:16.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:15 smithi190 bash[23868]: audit 2024-04-26T16:20:14.916279+0000 mon.c (mon.2) 67 : audit [DBG] from='client.? 172.21.15.50:0/1597091111' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:20:16.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 -- 172.21.15.50:0/4157893398 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca341034e0 msgr2=0x7fca34103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 --2- 172.21.15.50:0/4157893398 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca341034e0 0x7fca34103960 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca1c009990 tx=0x7fca1c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 -- 172.21.15.50:0/4157893398 shutdown_connections 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 --2- 172.21.15.50:0/4157893398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca34103ea0 0x7fca341102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 --2- 172.21.15.50:0/4157893398 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 0x7fca34109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 --2- 172.21.15.50:0/4157893398 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca341034e0 0x7fca34103960 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 -- 172.21.15.50:0/4157893398 >> 172.21.15.50:0/4157893398 conn(0x7fca3407a5e0 msgr2=0x7fca3407a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 -- 172.21.15.50:0/4157893398 shutdown_connections 2024-04-26T16:20:16.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 -- 172.21.15.50:0/4157893398 wait complete. 2024-04-26T16:20:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 librados: starting msgr at 2024-04-26T16:20:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 librados: starting objecter 2024-04-26T16:20:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 Processor -- start 2024-04-26T16:20:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 -- start start 2024-04-26T16:20:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 librados: setting wanted keys 2024-04-26T16:20:16.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.330+0000 7fca3b239700 1 librados: calling monclient init 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca341034e0 0x7fca3419b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 0x7fca3419b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 0x7fca3419fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fca34112a80 con 0x7fca34109480 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fca34112900 con 0x7fca341034e0 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fca34112c00 con 0x7fca34103ea0 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca33fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 0x7fca3419b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 0x7fca3419fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca33fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 0x7fca3419b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55084/0 (socket says 172.21.15.50:55084) 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca33fff700 1 -- 172.21.15.50:0/3961289381 learned_addr learned my addr 172.21.15.50:0/3961289381 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:16.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca38fd5700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca341034e0 0x7fca3419b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 -- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 msgr2=0x7fca3419b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 0x7fca3419b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 -- 172.21.15.50:0/3961289381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca341034e0 msgr2=0x7fca3419b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca341034e0 0x7fca3419b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 -- 172.21.15.50:0/3961289381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca341a0430 con 0x7fca34109480 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca38fd5700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca341034e0 0x7fca3419b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca33fff700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 0x7fca3419b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca397d6700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 0x7fca3419fcf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fca2c007f10 tx=0x7fca2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 10 client.?.objecter ms_handle_connect 0x7fca34109480 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c00ddd0 con 0x7fca34109480 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fca2c009e20 con 0x7fca34109480 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca341a0720 con 0x7fca34109480 2024-04-26T16:20:16.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca2c005a40 con 0x7fca34109480 2024-04-26T16:20:16.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 10 client.14643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca341a7fa0 con 0x7fca34109480 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 librados: init done 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3406a110 con 0x7fca34109480 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fca2c005020 con 0x7fca34109480 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fca20071f80 0x7fca20074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7fca2c090b90 con 0x7fca34109480 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 10 client.14643.objecter ms_dispatch 0x7fca34001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 3 client.14643.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 3 client.14643.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca38fd5700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fca20071f80 0x7fca20074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.334+0000 7fca31ffb700 20 client.14643.objecter dump_active .. 0 homeless 2024-04-26T16:20:16.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.338+0000 7fca31ffb700 10 client.14643.objecter ms_handle_connect 0x7fca20071f80 2024-04-26T16:20:16.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.338+0000 7fca38fd5700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fca20071f80 0x7fca20074440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca24009ca0 tx=0x7fca24009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:16.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.342+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fca2c010070 con 0x7fca34109480 2024-04-26T16:20:16.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.566+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fca341a81d0 con 0x7fca34109480 2024-04-26T16:20:16.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.566+0000 7fca31ffb700 1 -- 172.21.15.50:0/3961289381 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fca2c05cda0 con 0x7fca34109480 2024-04-26T16:20:16.560 INFO:teuthology.orchestra.run.smithi050.stdout:55834574953 2024-04-26T16:20:16.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 10 librados: watch_flush enter 2024-04-26T16:20:16.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 10 librados: watch_flush exit 2024-04-26T16:20:16.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 20 client.14643.objecter shutdown clearing up homeless session... 2024-04-26T16:20:16.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 10 client.14643.objecter successfully canceled tick 2024-04-26T16:20:16.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fca20071f80 msgr2=0x7fca20074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fca20071f80 0x7fca20074440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fca24009ca0 tx=0x7fca24009260 comp rx=0 tx=0).stop 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 msgr2=0x7fca3419fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 0x7fca3419fcf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fca2c007f10 tx=0x7fca2c00d3b0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 shutdown_connections 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fca20071f80 0x7fca20074440 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fca341034e0 0x7fca3419b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fca34109480 0x7fca3419fcf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 --2- 172.21.15.50:0/3961289381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fca34103ea0 0x7fca3419b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 >> 172.21.15.50:0/3961289381 conn(0x7fca3407a5e0 msgr2=0x7fca3410e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 shutdown_connections 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 -- 172.21.15.50:0/3961289381 wait complete. 2024-04-26T16:20:16.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.570+0000 7fca3b239700 1 librados: shutdown 2024-04-26T16:20:16.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.726+0000 7f23ef765700 1 -- 172.21.15.50:0/713946123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e806c360 msgr2=0x7f23e81102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.726+0000 7f23ef765700 1 --2- 172.21.15.50:0/713946123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e806c360 0x7f23e81102d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009f90 tx=0x7f23e000b330 comp rx=0 tx=0).stop 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/713946123 shutdown_connections 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 --2- 172.21.15.50:0/713946123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e806c360 0x7f23e81102d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 --2- 172.21.15.50:0/713946123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e8109490 0x7f23e8109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 --2- 172.21.15.50:0/713946123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806b9c0 0x7f23e806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/713946123 >> 172.21.15.50:0/713946123 conn(0x7f23e8079290 msgr2=0x7f23e80796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/713946123 shutdown_connections 2024-04-26T16:20:16.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/713946123 wait complete. 2024-04-26T16:20:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 librados: starting msgr at 2024-04-26T16:20:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 librados: starting objecter 2024-04-26T16:20:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 Processor -- start 2024-04-26T16:20:16.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- start start 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 librados: setting wanted keys 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 librados: calling monclient init 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 0x7f23e81023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806c360 0x7f23e8102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e8109490 0x7f23e8102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f23e8112a70 con 0x7f23e8109490 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f23e81128f0 con 0x7f23e806c360 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f23e8112bf0 con 0x7f23e806b9c0 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ed501700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 0x7f23e81023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e8109490 0x7f23e8102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ed501700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 0x7f23e81023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55104/0 (socket says 172.21.15.50:55104) 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ed501700 1 -- 172.21.15.50:0/3218781479 learned_addr learned my addr 172.21.15.50:0/3218781479 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:16.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ecd00700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806c360 0x7f23e8102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 -- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 msgr2=0x7f23e81023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 0x7f23e81023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 -- 172.21.15.50:0/3218781479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806c360 msgr2=0x7f23e8102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806c360 0x7f23e8102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 -- 172.21.15.50:0/3218781479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e81a7620 con 0x7f23e8109490 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ecd00700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806c360 0x7f23e8102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ed501700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 0x7f23e81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23edd02700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e8109490 0x7f23e8102e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23e000d010 tx=0x7f23e0004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23de7fc700 10 client.?.objecter ms_handle_connect 0x7f23e8109490 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23de7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0021070 con 0x7f23e8109490 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e81a78b0 con 0x7f23e8109490 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f23e0007910 con 0x7f23e8109490 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0005040 con 0x7f23e8109490 2024-04-26T16:20:16.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 10 client.14649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:16.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e81a7db0 con 0x7f23e8109490 2024-04-26T16:20:16.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 librados: init done 2024-04-26T16:20:16.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e810b870 con 0x7f23e8109490 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.730+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f23e0024070 con 0x7f23e8109490 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f23d4071f80 0x7f23d4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7f23e000f020 con 0x7f23e8109490 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 10 client.14649.objecter ms_dispatch 0x7f23e8001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 3 client.14649.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 3 client.14649.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23ed501700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f23d4071f80 0x7f23d4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 20 client.14649.objecter dump_active .. 0 homeless 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23ed501700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f23d4071f80 0x7f23d4074440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23e4005950 tx=0x7f23e4009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:16.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.734+0000 7f23de7fc700 10 client.14649.objecter ms_handle_connect 0x7f23d4071f80 2024-04-26T16:20:16.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.738+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f23e002b020 con 0x7f23e8109490 2024-04-26T16:20:16.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:20:16.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.019610+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:20:16.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.019610+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:20:16.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.021502+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.021502+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.239336+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.239336+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.241010+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.241010+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.389244+0000 mgr.y (mgr.24401) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.389244+0000 mgr.y (mgr.24401) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.933695+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.933695+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.936369+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: cluster 2024-04-26T16:20:15.936369+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:20:16.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: audit 2024-04-26T16:20:16.570003+0000 mon.a (mon.0) 918 : audit [DBG] from='client.? 172.21.15.50:0/3961289381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:20:16.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:16 smithi190 bash[23868]: audit 2024-04-26T16:20:16.570003+0000 mon.a (mon.0) 918 : audit [DBG] from='client.? 172.21.15.50:0/3961289381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:20:16.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.019610+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:20:16.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.019610+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.021502+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.021502+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.239336+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.239336+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.241010+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.241010+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.389244+0000 mgr.y (mgr.24401) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.389244+0000 mgr.y (mgr.24401) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.933695+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.933695+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.936369+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: cluster 2024-04-26T16:20:15.936369+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: audit 2024-04-26T16:20:16.570003+0000 mon.a (mon.0) 918 : audit [DBG] from='client.? 172.21.15.50:0/3961289381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:20:16.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[29234]: audit 2024-04-26T16:20:16.570003+0000 mon.a (mon.0) 918 : audit [DBG] from='client.? 172.21.15.50:0/3961289381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.019610+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.019610+0000 osd.1 (osd.1) 61 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.021502+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.021502+0000 osd.1 (osd.1) 62 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.239336+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.239336+0000 osd.0 (osd.0) 63 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.241010+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.241010+0000 osd.0 (osd.0) 64 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.389244+0000 mgr.y (mgr.24401) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.389244+0000 mgr.y (mgr.24401) 101 : cluster [DBG] pgmap v72: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.933695+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.933695+0000 osd.4 (osd.4) 55 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:20:16.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.936369+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:20:16.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: cluster 2024-04-26T16:20:15.936369+0000 osd.4 (osd.4) 56 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:20:16.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: audit 2024-04-26T16:20:16.570003+0000 mon.a (mon.0) 918 : audit [DBG] from='client.? 172.21.15.50:0/3961289381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:20:16.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:16 smithi050 bash[21463]: audit 2024-04-26T16:20:16.570003+0000 mon.a (mon.0) 918 : audit [DBG] from='client.? 172.21.15.50:0/3961289381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:20:16.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f23e8103910 con 0x7f23e8109490 2024-04-26T16:20:16.945 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23de7fc700 1 -- 172.21.15.50:0/3218781479 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f23e0024360 con 0x7f23e8109490 2024-04-26T16:20:16.945 INFO:teuthology.orchestra.run.smithi050.stdout:150323855433 2024-04-26T16:20:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 10 librados: watch_flush enter 2024-04-26T16:20:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 10 librados: watch_flush exit 2024-04-26T16:20:16.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 20 client.14649.objecter shutdown clearing up homeless session... 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 10 client.14649.objecter successfully canceled tick 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f23d4071f80 msgr2=0x7f23d4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f23d4071f80 0x7f23d4074440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23e4005950 tx=0x7f23e4009310 comp rx=0 tx=0).stop 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e8109490 msgr2=0x7f23e8102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e8109490 0x7f23e8102e60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23e000d010 tx=0x7f23e0004d00 comp rx=0 tx=0).stop 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 shutdown_connections 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f23d4071f80 0x7f23d4074440 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f23e8109490 0x7f23e8102e60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f23e806b9c0 0x7f23e81023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 --2- 172.21.15.50:0/3218781479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23e806c360 0x7f23e8102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:16.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 >> 172.21.15.50:0/3218781479 conn(0x7f23e8079290 msgr2=0x7f23e8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:16.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 shutdown_connections 2024-04-26T16:20:16.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 -- 172.21.15.50:0/3218781479 wait complete. 2024-04-26T16:20:16.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:16.954+0000 7f23ef765700 1 librados: shutdown 2024-04-26T16:20:17.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: cluster 2024-04-26T16:20:16.268019+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:20:17.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: cluster 2024-04-26T16:20:16.268019+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:20:17.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: cluster 2024-04-26T16:20:16.270311+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:20:17.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: cluster 2024-04-26T16:20:16.270311+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: audit 2024-04-26T16:20:16.423343+0000 mgr.y (mgr.24401) 102 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: audit 2024-04-26T16:20:16.423343+0000 mgr.y (mgr.24401) 102 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: audit 2024-04-26T16:20:16.955698+0000 mon.a (mon.0) 919 : audit [DBG] from='client.? 172.21.15.50:0/3218781479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[29234]: audit 2024-04-26T16:20:16.955698+0000 mon.a (mon.0) 919 : audit [DBG] from='client.? 172.21.15.50:0/3218781479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: cluster 2024-04-26T16:20:16.268019+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: cluster 2024-04-26T16:20:16.268019+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: cluster 2024-04-26T16:20:16.270311+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: cluster 2024-04-26T16:20:16.270311+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: audit 2024-04-26T16:20:16.423343+0000 mgr.y (mgr.24401) 102 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: audit 2024-04-26T16:20:16.423343+0000 mgr.y (mgr.24401) 102 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:17.904 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: audit 2024-04-26T16:20:16.955698+0000 mon.a (mon.0) 919 : audit [DBG] from='client.? 172.21.15.50:0/3218781479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:20:17.905 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:17 smithi050 bash[21463]: audit 2024-04-26T16:20:16.955698+0000 mon.a (mon.0) 919 : audit [DBG] from='client.? 172.21.15.50:0/3218781479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:20:18.036 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411417 got 77309411422 for osd.2 2024-04-26T16:20:18.037 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: cluster 2024-04-26T16:20:16.268019+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: cluster 2024-04-26T16:20:16.268019+0000 osd.7 (osd.7) 69 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: cluster 2024-04-26T16:20:16.270311+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: cluster 2024-04-26T16:20:16.270311+0000 osd.7 (osd.7) 70 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: audit 2024-04-26T16:20:16.423343+0000 mgr.y (mgr.24401) 102 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: audit 2024-04-26T16:20:16.423343+0000 mgr.y (mgr.24401) 102 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: audit 2024-04-26T16:20:16.955698+0000 mon.a (mon.0) 919 : audit [DBG] from='client.? 172.21.15.50:0/3218781479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:20:18.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:17 smithi190 bash[23868]: audit 2024-04-26T16:20:16.955698+0000 mon.a (mon.0) 919 : audit [DBG] from='client.? 172.21.15.50:0/3218781479' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:20:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/3856074458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c101a10 msgr2=0x7f1f8c101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- 172.21.15.50:0/3856074458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c101a10 0x7f1f8c101e70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1f78009990 tx=0x7f1f78009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/3856074458 shutdown_connections 2024-04-26T16:20:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- 172.21.15.50:0/3856074458 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 0x7f1f8c109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- 172.21.15.50:0/3856074458 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c1023b0 0x7f1f8c1102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- 172.21.15.50:0/3856074458 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c101a10 0x7f1f8c101e70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/3856074458 >> 172.21.15.50:0/3856074458 conn(0x7f1f8c078990 msgr2=0x7f1f8c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/3856074458 shutdown_connections 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/3856074458 wait complete. 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 librados: starting msgr at 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 librados: starting objecter 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 Processor -- start 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- start start 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 librados: setting wanted keys 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 librados: calling monclient init 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c101a10 0x7f1f8c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 0x7f1f8c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 0x7f1f8c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c112b80 con 0x7f1f8c101a10 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8c112a00 con 0x7f1f8c1094b0 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1f8c112d00 con 0x7f1f8c1023b0 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f90f86700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c101a10 0x7f1f8c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 0x7f1f8c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 0x7f1f8c19b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38778/0 (socket says 172.21.15.50:38778) 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 -- 172.21.15.50:0/2529444237 learned_addr learned my addr 172.21.15.50:0/2529444237 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f90f86700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c101a10 0x7f1f8c19b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51862/0 (socket says 172.21.15.50:51862) 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f91787700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 0x7f1f8c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 -- 172.21.15.50:0/2529444237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 msgr2=0x7f1f8c19fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 0x7f1f8c19fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 -- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c101a10 msgr2=0x7f1f8c19b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c101a10 0x7f1f8c19b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 -- 172.21.15.50:0/2529444237 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f8c1a0330 con 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f91787700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 0x7f1f8c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f8bfff700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 0x7f1f8c19b860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1f78000c00 tx=0x7f1f78004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f89ffb700 10 client.?.objecter ms_handle_connect 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f89ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78027070 con 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1f78022470 con 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f8c1a05c0 con 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f78017650 con 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 10 client.24545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f8c1a7ea0 con 0x7f1f8c1023b0 2024-04-26T16:20:18.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.210+0000 7f1f931ea700 1 librados: init done 2024-04-26T16:20:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f8c104d30 con 0x7f1f8c1023b0 2024-04-26T16:20:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f1f7802a070 con 0x7f1f8c1023b0 2024-04-26T16:20:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1f7c071f80 0x7f1f7c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7f1f780922c0 con 0x7f1f8c1023b0 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 10 client.24545.objecter ms_dispatch 0x7f1f8c001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 3 client.24545.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 3 client.24545.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f90f86700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1f7c071f80 0x7f1f7c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 20 client.24545.objecter dump_active .. 0 homeless 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f89ffb700 10 client.24545.objecter ms_handle_connect 0x7f1f7c071f80 2024-04-26T16:20:18.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.214+0000 7f1f90f86700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1f7c071f80 0x7f1f7c074440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000abd0 tx=0x7f1f8000a380 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:18.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.218+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1f78013070 con 0x7f1f8c1023b0 2024-04-26T16:20:18.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 -- 172.21.15.50:0/1126859857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54102320 msgr2=0x7fec541027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 --2- 172.21.15.50:0/1126859857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54102320 0x7fec541027a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec44009990 tx=0x7fec44009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 -- 172.21.15.50:0/1126859857 shutdown_connections 2024-04-26T16:20:18.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 --2- 172.21.15.50:0/1126859857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec54108390 0x7fec54108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 --2- 172.21.15.50:0/1126859857 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec5410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 --2- 172.21.15.50:0/1126859857 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54102320 0x7fec541027a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 -- 172.21.15.50:0/1126859857 >> 172.21.15.50:0/1126859857 conn(0x7fec540792d0 msgr2=0x7fec540796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 -- 172.21.15.50:0/1126859857 shutdown_connections 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 -- 172.21.15.50:0/1126859857 wait complete. 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 librados: starting msgr at 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 librados: starting objecter 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.434+0000 7fec59d90700 1 Processor -- start 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- start start 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 librados: setting wanted keys 2024-04-26T16:20:18.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 librados: calling monclient init 2024-04-26T16:20:18.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec54102320 0x7fec541990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec541995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54108390 0x7fec5419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fec541119a0 con 0x7fec54102ce0 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fec54111820 con 0x7fec54102320 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fec54111b20 con 0x7fec54108390 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec53fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54108390 0x7fec5419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec541995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec541995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51880/0 (socket says 172.21.15.50:51880) 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec53fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54108390 0x7fec5419d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38796/0 (socket says 172.21.15.50:38796) 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 -- 172.21.15.50:0/3878191036 learned_addr learned my addr 172.21.15.50:0/3878191036 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec537fe700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec54102320 0x7fec541990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 -- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54108390 msgr2=0x7fec5419d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54108390 0x7fec5419d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 -- 172.21.15.50:0/3878191036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec54102320 msgr2=0x7fec541990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec54102320 0x7fec541990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 -- 172.21.15.50:0/3878191036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5419e0b0 con 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec52ffd700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec541995e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fec4400c010 tx=0x7fec44005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 10 client.?.objecter ms_handle_connect 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44027070 con 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5419e340 con 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fec44022470 con 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec44017650 con 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 10 client.14661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec541a5c20 con 0x7fec54102ce0 2024-04-26T16:20:18.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 librados: init done 2024-04-26T16:20:18.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec54103a40 con 0x7fec54102ce0 2024-04-26T16:20:18.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fec4402a070 con 0x7fec54102ce0 2024-04-26T16:20:18.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fec40071f80 0x7fec40074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7fec44092350 con 0x7fec54102ce0 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 10 client.14661.objecter ms_dispatch 0x7fec54001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 3 client.14661.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec50ff9700 3 client.14661.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.438+0000 7fec537fe700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fec40071f80 0x7fec40074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.442+0000 7fec50ff9700 20 client.14661.objecter dump_active .. 0 homeless 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.442+0000 7fec537fe700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fec40071f80 0x7fec40074440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fec3c005fd0 tx=0x7fec3c00af20 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:18.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.442+0000 7fec50ff9700 10 client.14661.objecter ms_handle_connect 0x7fec40071f80 2024-04-26T16:20:18.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.446+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fec44013070 con 0x7fec54102ce0 2024-04-26T16:20:18.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.454+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1f8c19c680 con 0x7f1f8c1023b0 2024-04-26T16:20:18.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.454+0000 7f1f89ffb700 1 -- 172.21.15.50:0/2529444237 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1f7805e450 con 0x7f1f8c1023b0 2024-04-26T16:20:18.446 INFO:teuthology.orchestra.run.smithi050.stdout:201863462976 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 10 librados: watch_flush enter 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 10 librados: watch_flush exit 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 20 client.24545.objecter shutdown clearing up homeless session... 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 10 client.24545.objecter successfully canceled tick 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1f7c071f80 msgr2=0x7f1f7c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1f7c071f80 0x7f1f7c074440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000abd0 tx=0x7f1f8000a380 comp rx=0 tx=0).stop 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 msgr2=0x7f1f8c19b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 0x7f1f8c19b860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1f78000c00 tx=0x7f1f78004790 comp rx=0 tx=0).stop 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 shutdown_connections 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1f7c071f80 0x7f1f7c074440 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1f8c1094b0 0x7f1f8c19fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1f8c101a10 0x7f1f8c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 --2- 172.21.15.50:0/2529444237 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1f8c1023b0 0x7f1f8c19b860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 >> 172.21.15.50:0/2529444237 conn(0x7f1f8c078990 msgr2=0x7f1f8c107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:18.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 shutdown_connections 2024-04-26T16:20:18.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 -- 172.21.15.50:0/2529444237 wait complete. 2024-04-26T16:20:18.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.458+0000 7f1f931ea700 1 librados: shutdown 2024-04-26T16:20:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/2592468487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20108380 msgr2=0x7f3b20108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/2592468487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20108380 0x7f3b20108760 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009990 tx=0x7f3b10009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/2592468487 shutdown_connections 2024-04-26T16:20:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/2592468487 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102cd0 0x7f3b2010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/2592468487 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20108380 0x7f3b20108760 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/2592468487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b20102310 0x7f3b20102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/2592468487 >> 172.21.15.50:0/2592468487 conn(0x7f3b20079290 msgr2=0x7f3b200796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/2592468487 shutdown_connections 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/2592468487 wait complete. 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 librados: starting msgr at 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 librados: starting objecter 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 Processor -- start 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- start start 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 librados: setting wanted keys 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 librados: calling monclient init 2024-04-26T16:20:18.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 0x7f3b20196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 0x7f3b20197480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 0x7f3b20197480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 0x7f3b20196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b20108380 0x7f3b2019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20110940 con 0x7f3b20102cd0 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b201107c0 con 0x7f3b20108380 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3b20110ac0 con 0x7f3b20102310 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 0x7f3b20196f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38818/0 (socket says 172.21.15.50:38818) 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 -- 172.21.15.50:0/1908886767 learned_addr learned my addr 172.21.15.50:0/1908886767 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:18.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 0x7f3b20197480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51886/0 (socket says 172.21.15.50:51886) 2024-04-26T16:20:18.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b24fac700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b20108380 0x7f3b2019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 -- 172.21.15.50:0/1908886767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b20108380 msgr2=0x7f3b2019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b20108380 0x7f3b2019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 -- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 msgr2=0x7f3b20197480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 0x7f3b20197480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 -- 172.21.15.50:0/1908886767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2019bf50 con 0x7f3b20102310 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1f7fe700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 0x7f3b20197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1ffff700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 0x7f3b20196f40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3b10000c00 tx=0x7f3b1000f6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1d7fa700 10 client.?.objecter ms_handle_connect 0x7f3b20102310 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1d7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10027070 con 0x7f3b20102310 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3b1000fc50 con 0x7f3b20102310 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2019c1e0 con 0x7f3b20102310 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b100173f0 con 0x7f3b20102310 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 10 client.24557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:18.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b201a3ac0 con 0x7f3b20102310 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.658+0000 7f3b26a0f700 1 librados: init done 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2006a110 con 0x7f3b20102310 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3b1002a070 con 0x7f3b20102310 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b08074190 0x7f3b08076650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7f3b10092140 con 0x7f3b20102310 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 10 client.24557.objecter ms_dispatch 0x7f3b20001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 3 client.24557.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 3 client.24557.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1f7fe700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b08074190 0x7f3b08076650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 20 client.24557.objecter dump_active .. 0 homeless 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1d7fa700 10 client.24557.objecter ms_handle_connect 0x7f3b08074190 2024-04-26T16:20:18.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.662+0000 7f3b1f7fe700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b08074190 0x7f3b08076650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b20079b30 tx=0x7f3b1400b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:18.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.666+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3b10013070 con 0x7f3b20102310 2024-04-26T16:20:18.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fec5419a400 con 0x7fec54102ce0 2024-04-26T16:20:18.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec50ff9700 1 -- 172.21.15.50:0/3878191036 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fec4405e4e0 con 0x7fec54102ce0 2024-04-26T16:20:18.678 INFO:teuthology.orchestra.run.smithi050.stdout:244813135926 2024-04-26T16:20:18.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 10 librados: watch_flush enter 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 10 librados: watch_flush exit 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 20 client.14661.objecter shutdown clearing up homeless session... 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 10 client.14661.objecter successfully canceled tick 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fec40071f80 msgr2=0x7fec40074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fec40071f80 0x7fec40074440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fec3c005fd0 tx=0x7fec3c00af20 comp rx=0 tx=0).stop 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 msgr2=0x7fec541995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec541995e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fec4400c010 tx=0x7fec44005ef0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 shutdown_connections 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fec40071f80 0x7fec40074440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fec54102320 0x7fec541990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fec54108390 0x7fec5419d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 --2- 172.21.15.50:0/3878191036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fec54102ce0 0x7fec541995e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.686+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 >> 172.21.15.50:0/3878191036 conn(0x7fec540792d0 msgr2=0x7fec5406b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.690+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 shutdown_connections 2024-04-26T16:20:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.690+0000 7fec59d90700 1 -- 172.21.15.50:0/3878191036 wait complete. 2024-04-26T16:20:18.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.690+0000 7fec59d90700 1 librados: shutdown 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.273082+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.273082+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.275516+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.275516+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.302386+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.302386+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.304712+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.304712+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.390654+0000 mgr.y (mgr.24401) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: cluster 2024-04-26T16:20:17.390654+0000 mgr.y (mgr.24401) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:18.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: audit 2024-04-26T16:20:18.456985+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.50:0/2529444237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[29234]: audit 2024-04-26T16:20:18.456985+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.50:0/2529444237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.273082+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.273082+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.275516+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.275516+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.302386+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.302386+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.304712+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.304712+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.390654+0000 mgr.y (mgr.24401) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: cluster 2024-04-26T16:20:17.390654+0000 mgr.y (mgr.24401) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: audit 2024-04-26T16:20:18.456985+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.50:0/2529444237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:20:18.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:18 smithi050 bash[21463]: audit 2024-04-26T16:20:18.456985+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.50:0/2529444237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:20:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.886+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3b2002e490 con 0x7f3b20102310 2024-04-26T16:20:18.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.886+0000 7f3b1d7fa700 1 -- 172.21.15.50:0/1908886767 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3b10062080 con 0x7f3b20102310 2024-04-26T16:20:18.883 INFO:teuthology.orchestra.run.smithi050.stdout:34359738484 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 10 librados: watch_flush enter 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 10 librados: watch_flush exit 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 20 client.24557.objecter shutdown clearing up homeless session... 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 10 client.24557.objecter successfully canceled tick 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b08074190 msgr2=0x7f3b08076650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b08074190 0x7f3b08076650 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b20079b30 tx=0x7f3b1400b410 comp rx=0 tx=0).stop 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 msgr2=0x7f3b20196f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 0x7f3b20196f40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3b10000c00 tx=0x7f3b1000f6e0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 shutdown_connections 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b08074190 0x7f3b08076650 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b20108380 0x7f3b2019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b20102310 0x7f3b20196f40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 --2- 172.21.15.50:0/1908886767 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b20102cd0 0x7f3b20197480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:18.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 >> 172.21.15.50:0/1908886767 conn(0x7f3b20079290 msgr2=0x7f3b20107710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:18.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 shutdown_connections 2024-04-26T16:20:18.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 -- 172.21.15.50:0/1908886767 wait complete. 2024-04-26T16:20:18.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:18.890+0000 7f3b26a0f700 1 librados: shutdown 2024-04-26T16:20:19.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.273082+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:20:19.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.273082+0000 osd.0 (osd.0) 65 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:20:19.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.275516+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:20:19.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.275516+0000 osd.0 (osd.0) 66 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:20:19.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.302386+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:20:19.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.302386+0000 osd.7 (osd.7) 71 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:20:19.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.304712+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:20:19.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.304712+0000 osd.7 (osd.7) 72 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:20:19.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.390654+0000 mgr.y (mgr.24401) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:19.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: cluster 2024-04-26T16:20:17.390654+0000 mgr.y (mgr.24401) 103 : cluster [DBG] pgmap v73: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:19.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: audit 2024-04-26T16:20:18.456985+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.50:0/2529444237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:20:19.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:18 smithi190 bash[23868]: audit 2024-04-26T16:20:18.456985+0000 mon.c (mon.2) 68 : audit [DBG] from='client.? 172.21.15.50:0/2529444237' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.043657+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.043657+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.047042+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.047042+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.395723+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e scrub starts 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.395723+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e scrub starts 2024-04-26T16:20:19.888 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.397688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e scrub ok 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.397688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e scrub ok 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.565494+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.565494+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.567603+0000 osd.5 (osd.5) 66 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.567603+0000 osd.5 (osd.5) 66 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: audit 2024-04-26T16:20:18.688528+0000 mon.a (mon.0) 920 : audit [DBG] from='client.? 172.21.15.50:0/3878191036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: audit 2024-04-26T16:20:18.688528+0000 mon.a (mon.0) 920 : audit [DBG] from='client.? 172.21.15.50:0/3878191036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: audit 2024-04-26T16:20:18.890230+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.50:0/1908886767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: audit 2024-04-26T16:20:18.890230+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.50:0/1908886767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.959800+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.959800+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.961949+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:20:19.889 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[29234]: cluster 2024-04-26T16:20:18.961949+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.043657+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.043657+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.047042+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.047042+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.395723+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e scrub starts 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.395723+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e scrub starts 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.397688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e scrub ok 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.397688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e scrub ok 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.565494+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.565494+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:20:19.890 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.567603+0000 osd.5 (osd.5) 66 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.567603+0000 osd.5 (osd.5) 66 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: audit 2024-04-26T16:20:18.688528+0000 mon.a (mon.0) 920 : audit [DBG] from='client.? 172.21.15.50:0/3878191036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: audit 2024-04-26T16:20:18.688528+0000 mon.a (mon.0) 920 : audit [DBG] from='client.? 172.21.15.50:0/3878191036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: audit 2024-04-26T16:20:18.890230+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.50:0/1908886767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: audit 2024-04-26T16:20:18.890230+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.50:0/1908886767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.959800+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.959800+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.961949+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:20:19.891 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:19 smithi050 bash[21463]: cluster 2024-04-26T16:20:18.961949+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:20:20.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.043657+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.043657+0000 osd.1 (osd.1) 63 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.047042+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.047042+0000 osd.1 (osd.1) 64 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.395723+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e scrub starts 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.395723+0000 osd.2 (osd.2) 53 : cluster [DBG] 2.e scrub starts 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.397688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e scrub ok 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.397688+0000 osd.2 (osd.2) 54 : cluster [DBG] 2.e scrub ok 2024-04-26T16:20:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.565494+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.565494+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.567603+0000 osd.5 (osd.5) 66 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.567603+0000 osd.5 (osd.5) 66 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: audit 2024-04-26T16:20:18.688528+0000 mon.a (mon.0) 920 : audit [DBG] from='client.? 172.21.15.50:0/3878191036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: audit 2024-04-26T16:20:18.688528+0000 mon.a (mon.0) 920 : audit [DBG] from='client.? 172.21.15.50:0/3878191036' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: audit 2024-04-26T16:20:18.890230+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.50:0/1908886767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: audit 2024-04-26T16:20:18.890230+0000 mon.c (mon.2) 69 : audit [DBG] from='client.? 172.21.15.50:0/1908886767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.959800+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:20:20.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.959800+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:20:20.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.961949+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:20:20.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:19 smithi190 bash[23868]: cluster 2024-04-26T16:20:18.961949+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 -- 172.21.15.50:0/1030297555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44241095c0 msgr2=0x7f44241099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 --2- 172.21.15.50:0/1030297555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44241095c0 0x7f44241099a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4420009990 tx=0x7f4420009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 -- 172.21.15.50:0/1030297555 shutdown_connections 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 --2- 172.21.15.50:0/1030297555 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f442406c520 0x7f4424110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 --2- 172.21.15.50:0/1030297555 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f44241095c0 0x7f44241099a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 --2- 172.21.15.50:0/1030297555 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f442406bb60 0x7f442406bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.266+0000 7f442c405700 1 -- 172.21.15.50:0/1030297555 >> 172.21.15.50:0/1030297555 conn(0x7f4424079120 msgr2=0x7f4424079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- 172.21.15.50:0/1030297555 shutdown_connections 2024-04-26T16:20:20.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- 172.21.15.50:0/1030297555 wait complete. 2024-04-26T16:20:20.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 librados: starting msgr at 2024-04-26T16:20:20.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 librados: starting objecter 2024-04-26T16:20:20.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 Processor -- start 2024-04-26T16:20:20.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- start start 2024-04-26T16:20:20.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 librados: setting wanted keys 2024-04-26T16:20:20.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 librados: calling monclient init 2024-04-26T16:20:20.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f442406bb60 0x7f442419b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:20.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 0x7f442419b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:20.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44241095c0 0x7f442419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:20.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4424112b00 con 0x7f442406c520 2024-04-26T16:20:20.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4424112980 con 0x7f44241095c0 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4424112c80 con 0x7f442406bb60 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 0x7f442419b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 0x7f442419b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51900/0 (socket says 172.21.15.50:51900) 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 -- 172.21.15.50:0/351947645 learned_addr learned my addr 172.21.15.50:0/351947645 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442a1a1700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f442406bb60 0x7f442419b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442a9a2700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44241095c0 0x7f442419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 -- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f442406bb60 msgr2=0x7f442419b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f442406bb60 0x7f442419b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 -- 172.21.15.50:0/351947645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44241095c0 msgr2=0x7f442419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44241095c0 0x7f442419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 -- 172.21.15.50:0/351947645 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44241a0350 con 0x7f442406c520 2024-04-26T16:20:20.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442a9a2700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44241095c0 0x7f442419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44299a0700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 0x7f442419b880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f441800e9a0 tx=0x7f441800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44177fe700 10 client.?.objecter ms_handle_connect 0x7f442406c520 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44177fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441800cbd0 con 0x7f442406c520 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f441800cd70 con 0x7f442406c520 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44241a0640 con 0x7f442406c520 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418018420 con 0x7f442406c520 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 10 client.14673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44241a7ec0 con 0x7f442406c520 2024-04-26T16:20:20.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 librados: init done 2024-04-26T16:20:20.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442406a110 con 0x7f442406c520 2024-04-26T16:20:20.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.270+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f44180185c0 con 0x7f442406c520 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4410071f80 0x7f4410074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(114..114 src has 57..114) ==== 7450+0+0 (secure 0 0 0) 0x7f44180914a0 con 0x7f442406c520 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 10 client.14673.objecter ms_dispatch 0x7f4424001550 osd_map(114..114 src has 57..114) 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 3 client.14673.objecter handle_osd_map got epochs [114,114] > 0 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 3 client.14673.objecter handle_osd_map decoding full epoch 114 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f442a1a1700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4410071f80 0x7f4410074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 20 client.14673.objecter dump_active .. 0 homeless 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f442a1a1700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4410071f80 0x7f4410074440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f442000b2e0 tx=0x7f4420005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.274+0000 7f44177fe700 10 client.14673.objecter ms_handle_connect 0x7f4410071f80 2024-04-26T16:20:20.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.278+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f441805d6b0 con 0x7f442406c520 2024-04-26T16:20:20.461 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855428 got 150323855433 for osd.4 2024-04-26T16:20:20.461 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:20.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.506+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f44241a8150 con 0x7f442406c520 2024-04-26T16:20:20.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.506+0000 7f44177fe700 1 -- 172.21.15.50:0/351947645 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4418061360 con 0x7f442406c520 2024-04-26T16:20:20.499 INFO:teuthology.orchestra.run.smithi050.stdout:296352743468 2024-04-26T16:20:20.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 10 librados: watch_flush enter 2024-04-26T16:20:20.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 10 librados: watch_flush exit 2024-04-26T16:20:20.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 20 client.14673.objecter shutdown clearing up homeless session... 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 10 client.14673.objecter successfully canceled tick 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4410071f80 msgr2=0x7f4410074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4410071f80 0x7f4410074440 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f442000b2e0 tx=0x7f4420005960 comp rx=0 tx=0).stop 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 msgr2=0x7f442419b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 0x7f442419b880 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f441800e9a0 tx=0x7f441800ecb0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 shutdown_connections 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4410071f80 0x7f4410074440 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44241095c0 0x7f442419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f442406bb60 0x7f442419b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 --2- 172.21.15.50:0/351947645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f442406c520 0x7f442419b880 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 >> 172.21.15.50:0/351947645 conn(0x7f4424079120 msgr2=0x7f442410e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 shutdown_connections 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 -- 172.21.15.50:0/351947645 wait complete. 2024-04-26T16:20:20.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:20.510+0000 7f442c405700 1 librados: shutdown 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.391361+0000 mgr.y (mgr.24401) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.391361+0000 mgr.y (mgr.24401) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.573553+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.573553+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.575774+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.575774+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.975399+0000 osd.4 (osd.4) 57 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.975399+0000 osd.4 (osd.4) 57 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.982365+0000 osd.4 (osd.4) 58 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: cluster 2024-04-26T16:20:19.982365+0000 osd.4 (osd.4) 58 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:20:20.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: audit 2024-04-26T16:20:20.509162+0000 mon.a (mon.0) 921 : audit [DBG] from='client.? 172.21.15.50:0/351947645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[29234]: audit 2024-04-26T16:20:20.509162+0000 mon.a (mon.0) 921 : audit [DBG] from='client.? 172.21.15.50:0/351947645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.391361+0000 mgr.y (mgr.24401) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.391361+0000 mgr.y (mgr.24401) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.573553+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.573553+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.575774+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.575774+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.975399+0000 osd.4 (osd.4) 57 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.975399+0000 osd.4 (osd.4) 57 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.982365+0000 osd.4 (osd.4) 58 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: cluster 2024-04-26T16:20:19.982365+0000 osd.4 (osd.4) 58 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:20:20.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: audit 2024-04-26T16:20:20.509162+0000 mon.a (mon.0) 921 : audit [DBG] from='client.? 172.21.15.50:0/351947645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:20:20.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:20 smithi050 bash[21463]: audit 2024-04-26T16:20:20.509162+0000 mon.a (mon.0) 921 : audit [DBG] from='client.? 172.21.15.50:0/351947645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:20:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.391361+0000 mgr.y (mgr.24401) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.391361+0000 mgr.y (mgr.24401) 104 : cluster [DBG] pgmap v74: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.573553+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:20:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.573553+0000 osd.5 (osd.5) 67 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:20:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.575774+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:20:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.575774+0000 osd.5 (osd.5) 68 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:20:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.975399+0000 osd.4 (osd.4) 57 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:20:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.975399+0000 osd.4 (osd.4) 57 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:20:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.982365+0000 osd.4 (osd.4) 58 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:20:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: cluster 2024-04-26T16:20:19.982365+0000 osd.4 (osd.4) 58 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:20:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: audit 2024-04-26T16:20:20.509162+0000 mon.a (mon.0) 921 : audit [DBG] from='client.? 172.21.15.50:0/351947645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:20:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:20 smithi190 bash[23868]: audit 2024-04-26T16:20:20.509162+0000 mon.a (mon.0) 921 : audit [DBG] from='client.? 172.21.15.50:0/351947645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:20:21.397 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574948 got 55834574953 for osd.1 2024-04-26T16:20:21.398 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:21.990 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462970 got 201863462976 for osd.5 2024-04-26T16:20:21.990 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:22.074 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738478 got 34359738484 for osd.0 2024-04-26T16:20:22.074 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:21 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.158570+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:20:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:21 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.158570+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:20:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:21 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.161984+0000 osd.0 (osd.0) 68 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:20:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:21 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.161984+0000 osd.0 (osd.0) 68 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:20:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:21 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.392845+0000 mgr.y (mgr.24401) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:21 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.392845+0000 mgr.y (mgr.24401) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.158570+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.158570+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.161984+0000 osd.0 (osd.0) 68 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.161984+0000 osd.0 (osd.0) 68 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.392845+0000 mgr.y (mgr.24401) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.392845+0000 mgr.y (mgr.24401) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.158570+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.158570+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:20:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.161984+0000 osd.0 (osd.0) 68 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:20:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.161984+0000 osd.0 (osd.0) 68 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:20:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.392845+0000 mgr.y (mgr.24401) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:22.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:21 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.392845+0000 mgr.y (mgr.24401) 105 : cluster [DBG] pgmap v75: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:23.045 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135920 got 244813135926 for osd.6 2024-04-26T16:20:23.045 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:23.048 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743462 got 296352743468 for osd.7 2024-04-26T16:20:23.048 DEBUG:teuthology.parallel:result is None 2024-04-26T16:20:23.049 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T16:20:23.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:20:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.522656+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:20:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.522656+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:20:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.524752+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:20:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.524752+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.525219+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.525219+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.529500+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.529500+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.944279+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.944279+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.946984+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:21.946984+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:22.145367+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:22.145367+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:22.150720+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:20:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:22 smithi190 bash[23868]: cluster 2024-04-26T16:20:22.150720+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:20:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.522656+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:20:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.522656+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:20:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.524752+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.524752+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.525219+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.525219+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.529500+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.529500+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.944279+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.944279+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.946984+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:21.946984+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:22.145367+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:22.145367+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:22.150720+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:20:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[21463]: cluster 2024-04-26T16:20:22.150720+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.522656+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.522656+0000 osd.2 (osd.2) 55 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.524752+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.524752+0000 osd.5 (osd.5) 69 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.525219+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.525219+0000 osd.2 (osd.2) 56 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.529500+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.529500+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.944279+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.944279+0000 osd.4 (osd.4) 59 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.946984+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:20:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:21.946984+0000 osd.4 (osd.4) 60 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:20:23.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:22.145367+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:20:23.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:22.145367+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:20:23.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:22.150720+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:20:23.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:22 smithi050 bash[29234]: cluster 2024-04-26T16:20:22.150720+0000 osd.0 (osd.0) 70 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:20:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.022713+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.c scrub starts 2024-04-26T16:20:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.022713+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.c scrub starts 2024-04-26T16:20:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.024871+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.c scrub ok 2024-04-26T16:20:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.024871+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.c scrub ok 2024-04-26T16:20:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.312780+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:20:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.312780+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:20:24.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.315090+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:20:24.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.315090+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:20:24.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.393526+0000 mgr.y (mgr.24401) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:24.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:23 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.393526+0000 mgr.y (mgr.24401) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:24.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.022713+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.c scrub starts 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.022713+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.c scrub starts 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.024871+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.c scrub ok 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.024871+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.c scrub ok 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.312780+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.312780+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.315090+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.315090+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.393526+0000 mgr.y (mgr.24401) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:24.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.393526+0000 mgr.y (mgr.24401) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.022713+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.c scrub starts 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.022713+0000 osd.1 (osd.1) 65 : cluster [DBG] 5.c scrub starts 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.024871+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.c scrub ok 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.024871+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.c scrub ok 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.312780+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.312780+0000 osd.7 (osd.7) 73 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.315090+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.315090+0000 osd.7 (osd.7) 74 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.393526+0000 mgr.y (mgr.24401) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:24.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:23 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.393526+0000 mgr.y (mgr.24401) 106 : cluster [DBG] pgmap v76: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:24.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:20:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:20:25.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.543651+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub starts 2024-04-26T16:20:25.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.543651+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub starts 2024-04-26T16:20:25.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.546372+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.546372+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.991193+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.991193+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.995807+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:23.995807+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.198024+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.198024+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.200438+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.200438+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.305729+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.305729+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:20:25.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.308564+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:20:25.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.308564+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:20:25.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.352396+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:25.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.352396+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:25.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.352507+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:25.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:24 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.352507+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.543651+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub starts 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.543651+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub starts 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.546372+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.546372+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.991193+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.991193+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.995807+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:23.995807+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.198024+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.198024+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.200438+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.200438+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:20:25.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.305729+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.305729+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.308564+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.308564+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.352396+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.352396+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.352507+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.352507+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.543651+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub starts 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.543651+0000 osd.5 (osd.5) 71 : cluster [DBG] 6.d scrub starts 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.546372+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2024-04-26T16:20:25.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.546372+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.d scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.991193+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.991193+0000 osd.4 (osd.4) 61 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.995807+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:23.995807+0000 osd.4 (osd.4) 62 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.198024+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.198024+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.200438+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.200438+0000 osd.0 (osd.0) 72 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.305729+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.305729+0000 osd.7 (osd.7) 75 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.308564+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.308564+0000 osd.7 (osd.7) 76 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.352396+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:25.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.352396+0000 osd.1 (osd.1) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:25.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.352507+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:25.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:24 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.352507+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:26.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.977353+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-04-26T16:20:26.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:24.977353+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-04-26T16:20:26.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.045231+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.045231+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.051367+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.051367+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.234920+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.234920+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.238701+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.238701+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.394175+0000 mgr.y (mgr.24401) 107 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.394175+0000 mgr.y (mgr.24401) 107 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: audit 2024-04-26T16:20:25.669737+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:26.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:25 smithi190 bash[23868]: audit 2024-04-26T16:20:25.669737+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.977353+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:24.977353+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.045231+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.045231+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.051367+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.051367+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.234920+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.234920+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.238701+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.238701+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:20:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.394175+0000 mgr.y (mgr.24401) 107 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.394175+0000 mgr.y (mgr.24401) 107 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: audit 2024-04-26T16:20:25.669737+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[29234]: audit 2024-04-26T16:20:25.669737+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.977353+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:24.977353+0000 mon.a (mon.0) 922 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.045231+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.045231+0000 osd.1 (osd.1) 69 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.051367+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.051367+0000 osd.1 (osd.1) 70 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.234920+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:20:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.234920+0000 osd.0 (osd.0) 73 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:20:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.238701+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:20:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.238701+0000 osd.0 (osd.0) 74 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:20:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.394175+0000 mgr.y (mgr.24401) 107 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.394175+0000 mgr.y (mgr.24401) 107 : cluster [DBG] pgmap v78: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: audit 2024-04-26T16:20:25.669737+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:25 smithi050 bash[21463]: audit 2024-04-26T16:20:25.669737+0000 mon.c (mon.2) 70 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:26.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:20:27.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.506366+0000 osd.5 (osd.5) 73 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:20:27.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.506366+0000 osd.5 (osd.5) 73 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.510930+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.510930+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.969184+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.969184+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.975388+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:25.975388+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.236632+0000 osd.7 (osd.7) 77 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.236632+0000 osd.7 (osd.7) 77 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.239077+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.239077+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: audit 2024-04-26T16:20:26.434809+0000 mgr.y (mgr.24401) 108 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:27.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:26 smithi190 bash[23868]: audit 2024-04-26T16:20:26.434809+0000 mgr.y (mgr.24401) 108 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:27.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.506366+0000 osd.5 (osd.5) 73 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.506366+0000 osd.5 (osd.5) 73 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.510930+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.510930+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.969184+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.969184+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.975388+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:25.975388+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.236632+0000 osd.7 (osd.7) 77 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.236632+0000 osd.7 (osd.7) 77 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.239077+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.239077+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: audit 2024-04-26T16:20:26.434809+0000 mgr.y (mgr.24401) 108 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[21463]: audit 2024-04-26T16:20:26.434809+0000 mgr.y (mgr.24401) 108 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.506366+0000 osd.5 (osd.5) 73 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.506366+0000 osd.5 (osd.5) 73 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.510930+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.510930+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.969184+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.969184+0000 osd.4 (osd.4) 63 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.975388+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:25.975388+0000 osd.4 (osd.4) 64 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.236632+0000 osd.7 (osd.7) 77 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.236632+0000 osd.7 (osd.7) 77 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:20:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.239077+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:20:27.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.239077+0000 osd.7 (osd.7) 78 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:20:27.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: audit 2024-04-26T16:20:26.434809+0000 mgr.y (mgr.24401) 108 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:27.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:26 smithi050 bash[29234]: audit 2024-04-26T16:20:26.434809+0000 mgr.y (mgr.24401) 108 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:27.882 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:20:28.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.896084+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:20:28.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.896084+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:20:28.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.899285+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.899285+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.928844+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.928844+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.931206+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.931206+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.992375+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.992375+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.994621+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:26.994621+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:27.395233+0000 mgr.y (mgr.24401) 109 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:27 smithi190 bash[23868]: cluster 2024-04-26T16:20:27.395233+0000 mgr.y (mgr.24401) 109 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.896084+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.896084+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.899285+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.899285+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.928844+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.928844+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.931206+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.931206+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.992375+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.992375+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.994621+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:20:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:26.994621+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:27.395233+0000 mgr.y (mgr.24401) 109 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[21463]: cluster 2024-04-26T16:20:27.395233+0000 mgr.y (mgr.24401) 109 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.896084+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.896084+0000 osd.6 (osd.6) 43 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.899285+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.899285+0000 osd.6 (osd.6) 44 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.928844+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.928844+0000 osd.4 (osd.4) 65 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.931206+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.931206+0000 osd.4 (osd.4) 66 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.992375+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:20:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.992375+0000 osd.1 (osd.1) 71 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:20:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.994621+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:20:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:26.994621+0000 osd.1 (osd.1) 72 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:20:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:27.395233+0000 mgr.y (mgr.24401) 109 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:27 smithi050 bash[29234]: cluster 2024-04-26T16:20:27.395233+0000 mgr.y (mgr.24401) 109 : cluster [DBG] pgmap v79: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:29.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:27.563247+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:20:29.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:27.563247+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:27.565515+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:27.565515+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.233533+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.f scrub starts 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.233533+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.f scrub starts 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.238402+0000 osd.7 (osd.7) 80 : cluster [DBG] 3.f scrub ok 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.238402+0000 osd.7 (osd.7) 80 : cluster [DBG] 3.f scrub ok 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:27.563247+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:27.563247+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:27.565515+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:27.565515+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.233533+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.f scrub starts 2024-04-26T16:20:29.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.233533+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.f scrub starts 2024-04-26T16:20:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.238402+0000 osd.7 (osd.7) 80 : cluster [DBG] 3.f scrub ok 2024-04-26T16:20:29.240 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:28 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.238402+0000 osd.7 (osd.7) 80 : cluster [DBG] 3.f scrub ok 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:27.563247+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:27.563247+0000 osd.2 (osd.2) 57 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:27.565515+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:27.565515+0000 osd.2 (osd.2) 58 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.233533+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.f scrub starts 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.233533+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.f scrub starts 2024-04-26T16:20:29.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.238402+0000 osd.7 (osd.7) 80 : cluster [DBG] 3.f scrub ok 2024-04-26T16:20:29.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:28 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.238402+0000 osd.7 (osd.7) 80 : cluster [DBG] 3.f scrub ok 2024-04-26T16:20:30.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.437158+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:20:30.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.437158+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.439510+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.439510+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.545388+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.545388+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.545479+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:28.545479+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.010796+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.010796+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.022186+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.a scrub starts 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.022186+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.a scrub starts 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.023710+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub ok 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.023710+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub ok 2024-04-26T16:20:30.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.395832+0000 mgr.y (mgr.24401) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.395832+0000 mgr.y (mgr.24401) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.633046+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.633046+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.633177+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:30 smithi190 bash[23868]: cluster 2024-04-26T16:20:29.633177+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.437158+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.437158+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.439510+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.439510+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.545388+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.545388+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.545479+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:28.545479+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.010796+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.010796+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.022186+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.a scrub starts 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.022186+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.a scrub starts 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.023710+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub ok 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.023710+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub ok 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.395832+0000 mgr.y (mgr.24401) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.395832+0000 mgr.y (mgr.24401) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.633046+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.633046+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.633177+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[21463]: cluster 2024-04-26T16:20:29.633177+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.437158+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.437158+0000 osd.5 (osd.5) 75 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.439510+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.439510+0000 osd.5 (osd.5) 76 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.545388+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.545388+0000 osd.4 (osd.4) 67 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.545479+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:28.545479+0000 osd.4 (osd.4) 68 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.010796+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.010796+0000 mon.a (mon.0) 923 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.022186+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.a scrub starts 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.022186+0000 osd.1 (osd.1) 73 : cluster [DBG] 2.a scrub starts 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.023710+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub ok 2024-04-26T16:20:30.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.023710+0000 osd.1 (osd.1) 74 : cluster [DBG] 2.a scrub ok 2024-04-26T16:20:30.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.395832+0000 mgr.y (mgr.24401) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:30.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.395832+0000 mgr.y (mgr.24401) 110 : cluster [DBG] pgmap v81: 132 pgs: 132 active+clean; 583 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:30.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.633046+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.633046+0000 osd.7 (osd.7) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:30.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.633177+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:30.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:30 smithi050 bash[29234]: cluster 2024-04-26T16:20:29.633177+0000 osd.7 (osd.7) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:31 smithi190 bash[23868]: cluster 2024-04-26T16:20:30.017548+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-04-26T16:20:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:31 smithi190 bash[23868]: cluster 2024-04-26T16:20:30.017548+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-04-26T16:20:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:31 smithi190 bash[23868]: cluster 2024-04-26T16:20:30.222096+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:20:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:31 smithi190 bash[23868]: cluster 2024-04-26T16:20:30.222096+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:20:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:31 smithi190 bash[23868]: cluster 2024-04-26T16:20:30.226081+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:20:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:31 smithi190 bash[23868]: cluster 2024-04-26T16:20:30.226081+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- 172.21.15.50:0/622293913 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e001095c0 msgr2=0x7f1e001099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- 172.21.15.50:0/622293913 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e001095c0 0x7f1e001099a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1df0009f90 tx=0x7f1df000b330 comp rx=0 tx=0).stop 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- 172.21.15.50:0/622293913 shutdown_connections 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- 172.21.15.50:0/622293913 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e0006c520 0x7f1e00110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- 172.21.15.50:0/622293913 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e001095c0 0x7f1e001099a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- 172.21.15.50:0/622293913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e0006bb60 0x7f1e0006bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- 172.21.15.50:0/622293913 >> 172.21.15.50:0/622293913 conn(0x7f1e00079120 msgr2=0x7f1e00079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- 172.21.15.50:0/622293913 shutdown_connections 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- 172.21.15.50:0/622293913 wait complete. 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 librados: starting msgr at 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 librados: starting objecter 2024-04-26T16:20:31.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 Processor -- start 2024-04-26T16:20:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- start start 2024-04-26T16:20:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 librados: setting wanted keys 2024-04-26T16:20:31.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 librados: calling monclient init 2024-04-26T16:20:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 0x7f1e0019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e0006c520 0x7f1e0019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:31.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e001095c0 0x7f1e0019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00112b10 con 0x7f1e001095c0 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e00112990 con 0x7f1e0006c520 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e068e7700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1e00112c90 con 0x7f1e0006bb60 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 0x7f1e0019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e04e84700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e001095c0 0x7f1e0019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 0x7f1e0019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60986/0 (socket says 172.21.15.50:60986) 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 -- 172.21.15.50:0/463011620 learned_addr learned my addr 172.21.15.50:0/463011620 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dff7fe700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e0006c520 0x7f1e0019b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 -- 172.21.15.50:0/463011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e0006c520 msgr2=0x7f1e0019b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e0006c520 0x7f1e0019b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 -- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e001095c0 msgr2=0x7f1e0019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e001095c0 0x7f1e0019fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 -- 172.21.15.50:0/463011620 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e001a03a0 con 0x7f1e0006bb60 2024-04-26T16:20:31.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1e04e84700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e001095c0 0x7f1e0019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfffff700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 0x7f1e0019b390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1df000d010 tx=0x7f1df0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfd7fa700 10 client.?.objecter ms_handle_connect 0x7f1e0006bb60 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfd7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.366+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0021070 con 0x7f1e0006bb60 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e001a0630 con 0x7f1e0006bb60 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1df0007910 con 0x7f1e0006bb60 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df0005040 con 0x7f1e0006bb60 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1e068e7700 10 client.24569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e001a7f10 con 0x7f1e0006bb60 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1e068e7700 1 librados: init done 2024-04-26T16:20:31.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e0006a110 con 0x7f1e0006bb60 2024-04-26T16:20:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f1df0024070 con 0x7f1e0006bb60 2024-04-26T16:20:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1de8071f80 0x7f1de8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(117..117 src has 67..117) ==== 7450+0+0 (secure 0 0 0) 0x7f1df000f020 con 0x7f1e0006bb60 2024-04-26T16:20:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 10 client.24569.objecter ms_dispatch 0x7f1e00001550 osd_map(117..117 src has 67..117) 2024-04-26T16:20:31.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 3 client.24569.objecter handle_osd_map got epochs [117,117] > 0 2024-04-26T16:20:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 3 client.24569.objecter handle_osd_map decoding full epoch 117 2024-04-26T16:20:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dff7fe700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1de8071f80 0x7f1de8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 20 client.24569.objecter dump_active .. 0 homeless 2024-04-26T16:20:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dff7fe700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1de8071f80 0x7f1de8074440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005950 tx=0x7f1df40058e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:31.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.370+0000 7f1dfd7fa700 10 client.24569.objecter ms_handle_connect 0x7f1de8071f80 2024-04-26T16:20:31.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.374+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1df002b020 con 0x7f1e0006bb60 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[21463]: cluster 2024-04-26T16:20:30.017548+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[21463]: cluster 2024-04-26T16:20:30.017548+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[21463]: cluster 2024-04-26T16:20:30.222096+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[21463]: cluster 2024-04-26T16:20:30.222096+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[21463]: cluster 2024-04-26T16:20:30.226081+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[21463]: cluster 2024-04-26T16:20:30.226081+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:20:31.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[29234]: cluster 2024-04-26T16:20:30.017548+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-04-26T16:20:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[29234]: cluster 2024-04-26T16:20:30.017548+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2024-04-26T16:20:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[29234]: cluster 2024-04-26T16:20:30.222096+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:20:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[29234]: cluster 2024-04-26T16:20:30.222096+0000 osd.7 (osd.7) 83 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:20:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[29234]: cluster 2024-04-26T16:20:30.226081+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:20:31.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:31 smithi050 bash[29234]: cluster 2024-04-26T16:20:30.226081+0000 osd.7 (osd.7) 84 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:20:31.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.590+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1e0010edb0 con 0x7f1de8071f80 2024-04-26T16:20:31.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.602+0000 7f1dfd7fa700 1 -- 172.21.15.50:0/463011620 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346099 (secure 0 0 0) 0x7f1e0010edb0 con 0x7f1de8071f80 2024-04-26T16:20:31.601 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:20:31.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 10 librados: watch_flush enter 2024-04-26T16:20:31.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 10 librados: watch_flush exit 2024-04-26T16:20:31.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 20 client.24569.objecter shutdown clearing up homeless session... 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 10 client.24569.objecter successfully canceled tick 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1de8071f80 msgr2=0x7f1de8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1de8071f80 0x7f1de8074440 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1df4005950 tx=0x7f1df40058e0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 msgr2=0x7f1e0019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.606+0000 7f1e068e7700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 0x7f1e0019b390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1df000d010 tx=0x7f1df0004d00 comp rx=0 tx=0).stop 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 shutdown_connections 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1de8071f80 0x7f1de8074440 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e001095c0 0x7f1e0019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e0006bb60 0x7f1e0019b390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 --2- 172.21.15.50:0/463011620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e0006c520 0x7f1e0019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 >> 172.21.15.50:0/463011620 conn(0x7f1e00079120 msgr2=0x7f1e0010e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 shutdown_connections 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 -- 172.21.15.50:0/463011620 wait complete. 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:31.610+0000 7f1e068e7700 1 librados: shutdown 2024-04-26T16:20:31.604 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:20:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:32 smithi190 bash[23868]: cluster 2024-04-26T16:20:31.396592+0000 mgr.y (mgr.24401) 111 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:20:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:32 smithi190 bash[23868]: cluster 2024-04-26T16:20:31.396592+0000 mgr.y (mgr.24401) 111 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:20:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:32 smithi190 bash[23868]: audit 2024-04-26T16:20:31.595080+0000 mgr.y (mgr.24401) 112 : audit [DBG] from='client.24569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:32 smithi190 bash[23868]: audit 2024-04-26T16:20:31.595080+0000 mgr.y (mgr.24401) 112 : audit [DBG] from='client.24569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:32.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[29234]: cluster 2024-04-26T16:20:31.396592+0000 mgr.y (mgr.24401) 111 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[29234]: cluster 2024-04-26T16:20:31.396592+0000 mgr.y (mgr.24401) 111 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[29234]: audit 2024-04-26T16:20:31.595080+0000 mgr.y (mgr.24401) 112 : audit [DBG] from='client.24569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[29234]: audit 2024-04-26T16:20:31.595080+0000 mgr.y (mgr.24401) 112 : audit [DBG] from='client.24569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[21463]: cluster 2024-04-26T16:20:31.396592+0000 mgr.y (mgr.24401) 111 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[21463]: cluster 2024-04-26T16:20:31.396592+0000 mgr.y (mgr.24401) 111 : cluster [DBG] pgmap v83: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[21463]: audit 2024-04-26T16:20:31.595080+0000 mgr.y (mgr.24401) 112 : audit [DBG] from='client.24569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:32.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:32 smithi050 bash[21463]: audit 2024-04-26T16:20:31.595080+0000 mgr.y (mgr.24401) 112 : audit [DBG] from='client.24569 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:32.438 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-26T16:20:31.396090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1112,"num_read_kb":958,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":675,"ondisk_log_size":675,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240988,"kb_used_data":8176,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720228,"statfs":{"total":767960285184,"available":767713513472,"internally_reserved":0,"allocated":8372224,"data_stored":5109693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":238276148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424040"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405939+0000","last_change":"2024-04-26T16:19:21.967066+0000","last_active":"2024-04-26T16:20:30.405939+0000","last_peered":"2024-04-26T16:20:30.405939+0000","last_clean":"2024-04-26T16:20:30.405939+0000","last_became_active":"2024-04-26T16:17:35.677526+0000","last_became_peered":"2024-04-26T16:17:35.677526+0000","last_unstale":"2024-04-26T16:20:30.405939+0000","last_undegraded":"2024-04-26T16:20:30.405939+0000","last_fullsized":"2024-04-26T16:20:30.405939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:21.966835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:21.966835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014491+0000","last_change":"2024-04-26T16:20:05.019229+0000","last_active":"2024-04-26T16:20:30.014491+0000","last_peered":"2024-04-26T16:20:30.014491+0000","last_clean":"2024-04-26T16:20:30.014491+0000","last_became_active":"2024-04-26T16:17:33.070922+0000","last_became_peered":"2024-04-26T16:17:33.070922+0000","last_unstale":"2024-04-26T16:20:30.014491+0000","last_undegraded":"2024-04-26T16:20:30.014491+0000","last_fullsized":"2024-04-26T16:20:30.014491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.018938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.018938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.770490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015664+0000","last_change":"2024-04-26T16:20:17.275735+0000","last_active":"2024-04-26T16:20:30.015664+0000","last_peered":"2024-04-26T16:20:30.015664+0000","last_clean":"2024-04-26T16:20:30.015664+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:20:30.015664+0000","last_undegraded":"2024-04-26T16:20:30.015664+0000","last_fullsized":"2024-04-26T16:20:30.015664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.275530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.275530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:42.183654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"86'10","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406813+0000","last_change":"2024-04-26T16:19:39.904290+0000","last_active":"2024-04-26T16:20:30.406813+0000","last_peered":"2024-04-26T16:20:30.406813+0000","last_clean":"2024-04-26T16:20:30.406813+0000","last_became_active":"2024-04-26T16:17:31.049997+0000","last_became_peered":"2024-04-26T16:17:31.049997+0000","last_unstale":"2024-04-26T16:20:30.406813+0000","last_undegraded":"2024-04-26T16:20:30.406813+0000","last_fullsized":"2024-04-26T16:20:30.406813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:19:39.904035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:39.904035+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.444792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014554+0000","last_change":"2024-04-26T16:20:02.038119+0000","last_active":"2024-04-26T16:20:30.014554+0000","last_peered":"2024-04-26T16:20:30.014554+0000","last_clean":"2024-04-26T16:20:30.014554+0000","last_became_active":"2024-04-26T16:17:35.065560+0000","last_became_peered":"2024-04-26T16:17:35.065560+0000","last_unstale":"2024-04-26T16:20:30.014554+0000","last_undegraded":"2024-04-26T16:20:30.014554+0000","last_fullsized":"2024-04-26T16:20:30.014554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.037900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.037900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.752700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015649+0000","last_change":"2024-04-26T16:20:03.150021+0000","last_active":"2024-04-26T16:20:30.015649+0000","last_peered":"2024-04-26T16:20:30.015649+0000","last_clean":"2024-04-26T16:20:30.015649+0000","last_became_active":"2024-04-26T16:17:33.058899+0000","last_became_peered":"2024-04-26T16:17:33.058899+0000","last_unstale":"2024-04-26T16:20:30.015649+0000","last_undegraded":"2024-04-26T16:20:30.015649+0000","last_fullsized":"2024-04-26T16:20:30.015649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.149607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.149607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.024417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.407247+0000","last_change":"2024-04-26T16:20:03.947061+0000","last_active":"2024-04-26T16:20:30.407247+0000","last_peered":"2024-04-26T16:20:30.407247+0000","last_clean":"2024-04-26T16:20:30.407247+0000","last_became_active":"2024-04-26T16:17:29.016870+0000","last_became_peered":"2024-04-26T16:17:29.016870+0000","last_unstale":"2024-04-26T16:20:30.407247+0000","last_undegraded":"2024-04-26T16:20:30.407247+0000","last_fullsized":"2024-04-26T16:20:30.407247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.946950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.302253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015510+0000","last_change":"2024-04-26T16:20:04.348527+0000","last_active":"2024-04-26T16:20:30.015510+0000","last_peered":"2024-04-26T16:20:30.015510+0000","last_clean":"2024-04-26T16:20:30.015510+0000","last_became_active":"2024-04-26T16:17:31.041819+0000","last_became_peered":"2024-04-26T16:17:31.041819+0000","last_unstale":"2024-04-26T16:20:30.015510+0000","last_undegraded":"2024-04-26T16:20:30.015510+0000","last_fullsized":"2024-04-26T16:20:30.015510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:04.348299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.348299+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.592422+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403406+0000","last_change":"2024-04-26T16:20:02.714702+0000","last_active":"2024-04-26T16:20:30.403406+0000","last_peered":"2024-04-26T16:20:30.403406+0000","last_clean":"2024-04-26T16:20:30.403406+0000","last_became_active":"2024-04-26T16:17:35.087319+0000","last_became_peered":"2024-04-26T16:17:35.087319+0000","last_unstale":"2024-04-26T16:20:30.403406+0000","last_undegraded":"2024-04-26T16:20:30.403406+0000","last_fullsized":"2024-04-26T16:20:30.403406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:20:02.714466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.714466+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.031920+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223031+0000","last_change":"2024-04-26T16:18:55.734849+0000","last_active":"2024-04-26T16:20:30.223031+0000","last_peered":"2024-04-26T16:20:30.223031+0000","last_clean":"2024-04-26T16:20:30.223031+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:20:30.223031+0000","last_undegraded":"2024-04-26T16:20:30.223031+0000","last_fullsized":"2024-04-26T16:20:30.223031+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:18:55.734764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:18:55.734764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224691+0000","last_change":"2024-04-26T16:19:55.486245+0000","last_active":"2024-04-26T16:20:30.224691+0000","last_peered":"2024-04-26T16:20:30.224691+0000","last_clean":"2024-04-26T16:20:30.224691+0000","last_became_active":"2024-04-26T16:17:29.022163+0000","last_became_peered":"2024-04-26T16:17:29.022163+0000","last_unstale":"2024-04-26T16:20:30.224691+0000","last_undegraded":"2024-04-26T16:20:30.224691+0000","last_fullsized":"2024-04-26T16:20:30.224691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:55.486024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:55.486024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:06.433329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"86'15","reported_seq":124,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404825+0000","last_change":"2024-04-26T16:20:25.511190+0000","last_active":"2024-04-26T16:20:30.404825+0000","last_peered":"2024-04-26T16:20:30.404825+0000","last_clean":"2024-04-26T16:20:30.404825+0000","last_became_active":"2024-04-26T16:17:31.022500+0000","last_became_peered":"2024-04-26T16:17:31.022500+0000","last_unstale":"2024-04-26T16:20:30.404825+0000","last_undegraded":"2024-04-26T16:20:30.404825+0000","last_fullsized":"2024-04-26T16:20:30.404825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:25.510941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.510941+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.088665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015243+0000","last_change":"2024-04-26T16:19:20.658911+0000","last_active":"2024-04-26T16:20:30.015243+0000","last_peered":"2024-04-26T16:20:30.015243+0000","last_clean":"2024-04-26T16:20:30.015243+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:20:30.015243+0000","last_undegraded":"2024-04-26T16:20:30.015243+0000","last_fullsized":"2024-04-26T16:20:30.015243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:20.658689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.658689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.930841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403964+0000","last_change":"2024-04-26T16:20:18.567822+0000","last_active":"2024-04-26T16:20:30.403964+0000","last_peered":"2024-04-26T16:20:30.403964+0000","last_clean":"2024-04-26T16:20:30.403964+0000","last_became_active":"2024-04-26T16:17:33.071659+0000","last_became_peered":"2024-04-26T16:17:33.071659+0000","last_unstale":"2024-04-26T16:20:30.403964+0000","last_undegraded":"2024-04-26T16:20:30.403964+0000","last_fullsized":"2024-04-26T16:20:30.403964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.567616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.567616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.293668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223359+0000","last_change":"2024-04-26T16:20:23.315306+0000","last_active":"2024-04-26T16:20:30.223359+0000","last_peered":"2024-04-26T16:20:30.223359+0000","last_clean":"2024-04-26T16:20:30.223359+0000","last_became_active":"2024-04-26T16:17:29.025659+0000","last_became_peered":"2024-04-26T16:17:29.025659+0000","last_unstale":"2024-04-26T16:20:30.223359+0000","last_undegraded":"2024-04-26T16:20:30.223359+0000","last_fullsized":"2024-04-26T16:20:30.223359+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.315099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.315099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.593677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'12","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404861+0000","last_change":"2024-04-26T16:20:08.659388+0000","last_active":"2024-04-26T16:20:30.404861+0000","last_peered":"2024-04-26T16:20:30.404861+0000","last_clean":"2024-04-26T16:20:30.404861+0000","last_became_active":"2024-04-26T16:17:31.019223+0000","last_became_peered":"2024-04-26T16:17:31.019223+0000","last_unstale":"2024-04-26T16:20:30.404861+0000","last_undegraded":"2024-04-26T16:20:30.404861+0000","last_fullsized":"2024-04-26T16:20:30.404861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:08.659153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.659153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.636156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406941+0000","last_change":"2024-04-26T16:19:01.946387+0000","last_active":"2024-04-26T16:20:30.406941+0000","last_peered":"2024-04-26T16:20:30.406941+0000","last_clean":"2024-04-26T16:20:30.406941+0000","last_became_active":"2024-04-26T16:17:35.673178+0000","last_became_peered":"2024-04-26T16:17:35.673178+0000","last_unstale":"2024-04-26T16:20:30.406941+0000","last_undegraded":"2024-04-26T16:20:30.406941+0000","last_fullsized":"2024-04-26T16:20:30.406941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:01.946172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:01.946172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016324+0000","last_change":"2024-04-26T16:20:26.931424+0000","last_active":"2024-04-26T16:20:30.016324+0000","last_peered":"2024-04-26T16:20:30.016324+0000","last_clean":"2024-04-26T16:20:30.016324+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:20:30.016324+0000","last_undegraded":"2024-04-26T16:20:30.016324+0000","last_fullsized":"2024-04-26T16:20:30.016324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.931219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.931219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.617802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406053+0000","last_change":"2024-04-26T16:19:23.952520+0000","last_active":"2024-04-26T16:20:30.406053+0000","last_peered":"2024-04-26T16:20:30.406053+0000","last_clean":"2024-04-26T16:20:30.406053+0000","last_became_active":"2024-04-26T16:17:29.005070+0000","last_became_peered":"2024-04-26T16:17:29.005070+0000","last_unstale":"2024-04-26T16:20:30.406053+0000","last_undegraded":"2024-04-26T16:20:30.406053+0000","last_fullsized":"2024-04-26T16:20:30.406053+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.952306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.952306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"86'9","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014821+0000","last_change":"2024-04-26T16:19:37.958917+0000","last_active":"2024-04-26T16:20:30.014821+0000","last_peered":"2024-04-26T16:20:30.014821+0000","last_clean":"2024-04-26T16:20:30.014821+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:20:30.014821+0000","last_undegraded":"2024-04-26T16:20:30.014821+0000","last_fullsized":"2024-04-26T16:20:30.014821+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:37.958674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:37.958674+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:53.537405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017257+0000","last_change":"2024-04-26T16:20:15.936737+0000","last_active":"2024-04-26T16:20:30.017257+0000","last_peered":"2024-04-26T16:20:30.017257+0000","last_clean":"2024-04-26T16:20:30.017257+0000","last_became_active":"2024-04-26T16:17:35.674743+0000","last_became_peered":"2024-04-26T16:17:35.674743+0000","last_unstale":"2024-04-26T16:20:30.017257+0000","last_undegraded":"2024-04-26T16:20:30.017257+0000","last_fullsized":"2024-04-26T16:20:30.017257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.936392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.456145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015871+0000","last_change":"2024-04-26T16:20:26.994856+0000","last_active":"2024-04-26T16:20:30.015871+0000","last_peered":"2024-04-26T16:20:30.015871+0000","last_clean":"2024-04-26T16:20:30.015871+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:20:30.015871+0000","last_undegraded":"2024-04-26T16:20:30.015871+0000","last_fullsized":"2024-04-26T16:20:30.015871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.994631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.994631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:56.773657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067533+0000","last_change":"2024-04-26T16:20:08.830657+0000","last_active":"2024-04-26T16:20:30.067533+0000","last_peered":"2024-04-26T16:20:30.067533+0000","last_clean":"2024-04-26T16:20:30.067533+0000","last_became_active":"2024-04-26T16:17:29.004486+0000","last_became_peered":"2024-04-26T16:17:29.004486+0000","last_unstale":"2024-04-26T16:20:30.067533+0000","last_undegraded":"2024-04-26T16:20:30.067533+0000","last_fullsized":"2024-04-26T16:20:30.067533+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.830446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.830446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.833491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"114'7","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017270+0000","last_change":"2024-04-26T16:20:22.150870+0000","last_active":"2024-04-26T16:20:30.017270+0000","last_peered":"2024-04-26T16:20:30.017270+0000","last_clean":"2024-04-26T16:20:30.017270+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:20:30.017270+0000","last_undegraded":"2024-04-26T16:20:30.017270+0000","last_fullsized":"2024-04-26T16:20:30.017270+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_clean_scrub_stamp":"2024-04-26T16:20:22.150727+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:47.002699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016084+0000","last_change":"2024-04-26T16:20:10.128158+0000","last_active":"2024-04-26T16:20:30.016084+0000","last_peered":"2024-04-26T16:20:30.016084+0000","last_clean":"2024-04-26T16:20:30.016084+0000","last_became_active":"2024-04-26T16:17:35.080236+0000","last_became_peered":"2024-04-26T16:17:35.080236+0000","last_unstale":"2024-04-26T16:20:30.016084+0000","last_undegraded":"2024-04-26T16:20:30.016084+0000","last_fullsized":"2024-04-26T16:20:30.016084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.127949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.127949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.505883+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014744+0000","last_change":"2024-04-26T16:20:03.390713+0000","last_active":"2024-04-26T16:20:30.014744+0000","last_peered":"2024-04-26T16:20:30.014744+0000","last_clean":"2024-04-26T16:20:30.014744+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:20:30.014744+0000","last_undegraded":"2024-04-26T16:20:30.014744+0000","last_fullsized":"2024-04-26T16:20:30.014744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.390501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.390501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:06.150359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405357+0000","last_change":"2024-04-26T16:19:18.953972+0000","last_active":"2024-04-26T16:20:30.405357+0000","last_peered":"2024-04-26T16:20:30.405357+0000","last_clean":"2024-04-26T16:20:30.405357+0000","last_became_active":"2024-04-26T16:17:28.999524+0000","last_became_peered":"2024-04-26T16:17:28.999524+0000","last_unstale":"2024-04-26T16:20:30.405357+0000","last_undegraded":"2024-04-26T16:20:30.405357+0000","last_fullsized":"2024-04-26T16:20:30.405357+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:19:18.953733+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:19:18.953733+0000","last_clean_scrub_stamp":"2024-04-26T16:19:18.953733+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"86'9","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405595+0000","last_change":"2024-04-26T16:18:51.946113+0000","last_active":"2024-04-26T16:20:30.405595+0000","last_peered":"2024-04-26T16:20:30.405595+0000","last_clean":"2024-04-26T16:20:30.405595+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:20:30.405595+0000","last_undegraded":"2024-04-26T16:20:30.405595+0000","last_fullsized":"2024-04-26T16:20:30.405595+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:18:51.945825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:18:51.945825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224068+0000","last_change":"2024-04-26T16:19:34.482254+0000","last_active":"2024-04-26T16:20:30.224068+0000","last_peered":"2024-04-26T16:20:30.224068+0000","last_clean":"2024-04-26T16:20:30.224068+0000","last_became_active":"2024-04-26T16:17:35.081916+0000","last_became_peered":"2024-04-26T16:17:35.081916+0000","last_unstale":"2024-04-26T16:20:30.224068+0000","last_undegraded":"2024-04-26T16:20:30.224068+0000","last_fullsized":"2024-04-26T16:20:30.224068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:19:34.482014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:34.482014+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:40.383514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"86'11","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068581+0000","last_change":"2024-04-26T16:19:59.878190+0000","last_active":"2024-04-26T16:20:30.068581+0000","last_peered":"2024-04-26T16:20:30.068581+0000","last_clean":"2024-04-26T16:20:30.068581+0000","last_became_active":"2024-04-26T16:17:33.061926+0000","last_became_peered":"2024-04-26T16:17:33.061926+0000","last_unstale":"2024-04-26T16:20:30.068581+0000","last_undegraded":"2024-04-26T16:20:30.068581+0000","last_fullsized":"2024-04-26T16:20:30.068581+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:59.877790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.877790+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.440948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403217+0000","last_change":"2024-04-26T16:19:58.724981+0000","last_active":"2024-04-26T16:20:30.403217+0000","last_peered":"2024-04-26T16:20:30.403217+0000","last_clean":"2024-04-26T16:20:30.403217+0000","last_became_active":"2024-04-26T16:17:29.025319+0000","last_became_peered":"2024-04-26T16:17:29.025319+0000","last_unstale":"2024-04-26T16:20:30.403217+0000","last_undegraded":"2024-04-26T16:20:30.403217+0000","last_fullsized":"2024-04-26T16:20:30.403217+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.724766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.724766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.832669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'15","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015946+0000","last_change":"2024-04-26T16:19:25.196360+0000","last_active":"2024-04-26T16:20:30.015946+0000","last_peered":"2024-04-26T16:20:30.015946+0000","last_clean":"2024-04-26T16:20:30.015946+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:20:30.015946+0000","last_undegraded":"2024-04-26T16:20:30.015946+0000","last_fullsized":"2024-04-26T16:20:30.015946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:19:25.196241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:25.196241+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406885+0000","last_change":"2024-04-26T16:19:07.935631+0000","last_active":"2024-04-26T16:20:30.406885+0000","last_peered":"2024-04-26T16:20:30.406885+0000","last_clean":"2024-04-26T16:20:30.406885+0000","last_became_active":"2024-04-26T16:17:35.677366+0000","last_became_peered":"2024-04-26T16:17:35.677366+0000","last_unstale":"2024-04-26T16:20:30.406885+0000","last_undegraded":"2024-04-26T16:20:30.406885+0000","last_fullsized":"2024-04-26T16:20:30.406885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:07.935387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:07.935387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223713+0000","last_change":"2024-04-26T16:20:26.239452+0000","last_active":"2024-04-26T16:20:30.223713+0000","last_peered":"2024-04-26T16:20:30.223713+0000","last_clean":"2024-04-26T16:20:30.223713+0000","last_became_active":"2024-04-26T16:17:33.075775+0000","last_became_peered":"2024-04-26T16:17:33.075775+0000","last_unstale":"2024-04-26T16:20:30.223713+0000","last_undegraded":"2024-04-26T16:20:30.223713+0000","last_fullsized":"2024-04-26T16:20:30.223713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.239095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.239095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.234785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068716+0000","last_change":"2024-04-26T16:20:02.934191+0000","last_active":"2024-04-26T16:20:30.068716+0000","last_peered":"2024-04-26T16:20:30.068716+0000","last_clean":"2024-04-26T16:20:30.068716+0000","last_became_active":"2024-04-26T16:17:29.018752+0000","last_became_peered":"2024-04-26T16:17:29.018752+0000","last_unstale":"2024-04-26T16:20:30.068716+0000","last_undegraded":"2024-04-26T16:20:30.068716+0000","last_fullsized":"2024-04-26T16:20:30.068716+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.933912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.933912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.658980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"86'9","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404619+0000","last_change":"2024-04-26T16:19:03.478782+0000","last_active":"2024-04-26T16:20:30.404619+0000","last_peered":"2024-04-26T16:20:30.404619+0000","last_clean":"2024-04-26T16:20:30.404619+0000","last_became_active":"2024-04-26T16:17:31.024113+0000","last_became_peered":"2024-04-26T16:17:31.024113+0000","last_unstale":"2024-04-26T16:20:30.404619+0000","last_undegraded":"2024-04-26T16:20:30.404619+0000","last_fullsized":"2024-04-26T16:20:30.404619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:03.478587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:03.478587+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:32.285278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222817+0000","last_change":"2024-04-26T16:19:15.680713+0000","last_active":"2024-04-26T16:20:30.222817+0000","last_peered":"2024-04-26T16:20:30.222817+0000","last_clean":"2024-04-26T16:20:30.222817+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:20:30.222817+0000","last_undegraded":"2024-04-26T16:20:30.222817+0000","last_fullsized":"2024-04-26T16:20:30.222817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:15.680333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:15.680333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067308+0000","last_change":"2024-04-26T16:19:23.064854+0000","last_active":"2024-04-26T16:20:30.067308+0000","last_peered":"2024-04-26T16:20:30.067308+0000","last_clean":"2024-04-26T16:20:30.067308+0000","last_became_active":"2024-04-26T16:17:33.059705+0000","last_became_peered":"2024-04-26T16:17:33.059705+0000","last_unstale":"2024-04-26T16:20:30.067308+0000","last_undegraded":"2024-04-26T16:20:30.067308+0000","last_fullsized":"2024-04-26T16:20:30.067308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.064641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.064641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:48.435806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405333+0000","last_change":"2024-04-26T16:20:19.575996+0000","last_active":"2024-04-26T16:20:30.405333+0000","last_peered":"2024-04-26T16:20:30.405333+0000","last_clean":"2024-04-26T16:20:30.405333+0000","last_became_active":"2024-04-26T16:17:29.022454+0000","last_became_peered":"2024-04-26T16:17:29.022454+0000","last_unstale":"2024-04-26T16:20:30.405333+0000","last_undegraded":"2024-04-26T16:20:30.405333+0000","last_fullsized":"2024-04-26T16:20:30.405333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:19.575787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.575787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.873083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"86'6","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015346+0000","last_change":"2024-04-26T16:20:21.162238+0000","last_active":"2024-04-26T16:20:30.015346+0000","last_peered":"2024-04-26T16:20:30.015346+0000","last_clean":"2024-04-26T16:20:30.015346+0000","last_became_active":"2024-04-26T16:17:31.053359+0000","last_became_peered":"2024-04-26T16:17:31.053359+0000","last_unstale":"2024-04-26T16:20:30.015346+0000","last_undegraded":"2024-04-26T16:20:30.015346+0000","last_fullsized":"2024-04-26T16:20:30.015346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'6","last_scrub_stamp":"2024-04-26T16:20:21.161995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.161995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.190751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405130+0000","last_change":"2024-04-26T16:20:08.001062+0000","last_active":"2024-04-26T16:20:30.405130+0000","last_peered":"2024-04-26T16:20:30.405130+0000","last_clean":"2024-04-26T16:20:30.405130+0000","last_became_active":"2024-04-26T16:17:35.094615+0000","last_became_peered":"2024-04-26T16:17:35.094615+0000","last_unstale":"2024-04-26T16:20:30.405130+0000","last_undegraded":"2024-04-26T16:20:30.405130+0000","last_fullsized":"2024-04-26T16:20:30.405130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.000834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.000834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.457279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017005+0000","last_change":"2024-04-26T16:19:12.327137+0000","last_active":"2024-04-26T16:20:30.017005+0000","last_peered":"2024-04-26T16:20:30.017005+0000","last_clean":"2024-04-26T16:20:30.017005+0000","last_became_active":"2024-04-26T16:17:33.048202+0000","last_became_peered":"2024-04-26T16:17:33.048202+0000","last_unstale":"2024-04-26T16:20:30.017005+0000","last_undegraded":"2024-04-26T16:20:30.017005+0000","last_fullsized":"2024-04-26T16:20:30.017005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:12.326888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:12.326888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.929054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015666+0000","last_change":"2024-04-26T16:19:19.243504+0000","last_active":"2024-04-26T16:20:30.015666+0000","last_peered":"2024-04-26T16:20:30.015666+0000","last_clean":"2024-04-26T16:20:30.015666+0000","last_became_active":"2024-04-26T16:17:29.011346+0000","last_became_peered":"2024-04-26T16:17:29.011346+0000","last_unstale":"2024-04-26T16:20:30.015666+0000","last_undegraded":"2024-04-26T16:20:30.015666+0000","last_fullsized":"2024-04-26T16:20:30.015666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:19.243261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:19.243261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:42.597153+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"86'10","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016663+0000","last_change":"2024-04-26T16:20:23.996060+0000","last_active":"2024-04-26T16:20:30.016663+0000","last_peered":"2024-04-26T16:20:30.016663+0000","last_clean":"2024-04-26T16:20:30.016663+0000","last_became_active":"2024-04-26T16:17:31.021476+0000","last_became_peered":"2024-04-26T16:17:31.021476+0000","last_unstale":"2024-04-26T16:20:30.016663+0000","last_undegraded":"2024-04-26T16:20:30.016663+0000","last_fullsized":"2024-04-26T16:20:30.016663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:23.995817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.995817+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.162295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014994+0000","last_change":"2024-04-26T16:20:25.238939+0000","last_active":"2024-04-26T16:20:30.014994+0000","last_peered":"2024-04-26T16:20:30.014994+0000","last_clean":"2024-04-26T16:20:30.014994+0000","last_became_active":"2024-04-26T16:17:35.078105+0000","last_became_peered":"2024-04-26T16:17:35.078105+0000","last_unstale":"2024-04-26T16:20:30.014994+0000","last_undegraded":"2024-04-26T16:20:30.014994+0000","last_fullsized":"2024-04-26T16:20:30.014994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:25.238710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:06.659791+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.238710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:31.861824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406335+0000","last_change":"2024-04-26T16:20:07.028942+0000","last_active":"2024-04-26T16:20:30.406335+0000","last_peered":"2024-04-26T16:20:30.406335+0000","last_clean":"2024-04-26T16:20:30.406335+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:20:30.406335+0000","last_undegraded":"2024-04-26T16:20:30.406335+0000","last_fullsized":"2024-04-26T16:20:30.406335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:07.028724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:07.028724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.581251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068037+0000","last_change":"2024-04-26T16:20:12.898651+0000","last_active":"2024-04-26T16:20:30.068037+0000","last_peered":"2024-04-26T16:20:30.068037+0000","last_clean":"2024-04-26T16:20:30.068037+0000","last_became_active":"2024-04-26T16:17:29.013639+0000","last_became_peered":"2024-04-26T16:17:29.013639+0000","last_unstale":"2024-04-26T16:20:30.068037+0000","last_undegraded":"2024-04-26T16:20:30.068037+0000","last_fullsized":"2024-04-26T16:20:30.068037+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:20:12.898432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.131422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"86'9","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.225004+0000","last_change":"2024-04-26T16:20:09.306205+0000","last_active":"2024-04-26T16:20:30.225004+0000","last_peered":"2024-04-26T16:20:30.225004+0000","last_clean":"2024-04-26T16:20:30.225004+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:20:30.225004+0000","last_undegraded":"2024-04-26T16:20:30.225004+0000","last_fullsized":"2024-04-26T16:20:30.225004+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:09.305987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.305987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.952773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.018022+0000","last_change":"2024-04-26T16:19:14.082286+0000","last_active":"2024-04-26T16:20:30.018022+0000","last_peered":"2024-04-26T16:20:30.018022+0000","last_clean":"2024-04-26T16:20:30.018022+0000","last_became_active":"2024-04-26T16:17:35.070054+0000","last_became_peered":"2024-04-26T16:17:35.070054+0000","last_unstale":"2024-04-26T16:20:30.018022+0000","last_undegraded":"2024-04-26T16:20:30.018022+0000","last_fullsized":"2024-04-26T16:20:30.018022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:14.082070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:14.082070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:43.454813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406448+0000","last_change":"2024-04-26T16:19:26.961234+0000","last_active":"2024-04-26T16:20:30.406448+0000","last_peered":"2024-04-26T16:20:30.406448+0000","last_clean":"2024-04-26T16:20:30.406448+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:20:30.406448+0000","last_undegraded":"2024-04-26T16:20:30.406448+0000","last_fullsized":"2024-04-26T16:20:30.406448+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:26.960994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:26.960994+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:52.147716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017131+0000","last_change":"2024-04-26T16:20:15.241109+0000","last_active":"2024-04-26T16:20:30.017131+0000","last_peered":"2024-04-26T16:20:30.017131+0000","last_clean":"2024-04-26T16:20:30.017131+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:20:30.017131+0000","last_undegraded":"2024-04-26T16:20:30.017131+0000","last_fullsized":"2024-04-26T16:20:30.017131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.241014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.241014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.603907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014779+0000","last_change":"2024-04-26T16:20:00.355659+0000","last_active":"2024-04-26T16:20:30.014779+0000","last_peered":"2024-04-26T16:20:30.014779+0000","last_clean":"2024-04-26T16:20:30.014779+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:20:30.014779+0000","last_undegraded":"2024-04-26T16:20:30.014779+0000","last_fullsized":"2024-04-26T16:20:30.014779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:00.355248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:00.355248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:00.966364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014490+0000","last_change":"2024-04-26T16:19:10.614262+0000","last_active":"2024-04-26T16:20:30.014490+0000","last_peered":"2024-04-26T16:20:30.014490+0000","last_clean":"2024-04-26T16:20:30.014490+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:20:30.014490+0000","last_undegraded":"2024-04-26T16:20:30.014490+0000","last_fullsized":"2024-04-26T16:20:30.014490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:10.614042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:10.614042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.417468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"101'12","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404894+0000","last_change":"2024-04-26T16:19:51.757222+0000","last_active":"2024-04-26T16:20:30.404894+0000","last_peered":"2024-04-26T16:20:30.404894+0000","last_clean":"2024-04-26T16:20:30.404894+0000","last_became_active":"2024-04-26T16:17:33.062000+0000","last_became_peered":"2024-04-26T16:17:33.062000+0000","last_unstale":"2024-04-26T16:20:30.404894+0000","last_undegraded":"2024-04-26T16:20:30.404894+0000","last_fullsized":"2024-04-26T16:20:30.404894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:19:51.756984+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:57.578136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403615+0000","last_change":"2024-04-26T16:20:05.684861+0000","last_active":"2024-04-26T16:20:30.403615+0000","last_peered":"2024-04-26T16:20:30.403615+0000","last_clean":"2024-04-26T16:20:30.403615+0000","last_became_active":"2024-04-26T16:17:29.025290+0000","last_became_peered":"2024-04-26T16:17:29.025290+0000","last_unstale":"2024-04-26T16:20:30.403615+0000","last_undegraded":"2024-04-26T16:20:30.403615+0000","last_fullsized":"2024-04-26T16:20:30.403615+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.684621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.684621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:12.223118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223712+0000","last_change":"2024-04-26T16:19:43.379294+0000","last_active":"2024-04-26T16:20:30.223712+0000","last_peered":"2024-04-26T16:20:30.223712+0000","last_clean":"2024-04-26T16:20:30.223712+0000","last_became_active":"2024-04-26T16:17:31.034570+0000","last_became_peered":"2024-04-26T16:17:31.034570+0000","last_unstale":"2024-04-26T16:20:30.223712+0000","last_undegraded":"2024-04-26T16:20:30.223712+0000","last_fullsized":"2024-04-26T16:20:30.223712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:43.379052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:43.379052+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:49.859888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223905+0000","last_change":"2024-04-26T16:19:20.635604+0000","last_active":"2024-04-26T16:20:30.223905+0000","last_peered":"2024-04-26T16:20:30.223905+0000","last_clean":"2024-04-26T16:20:30.223905+0000","last_became_active":"2024-04-26T16:17:35.671754+0000","last_became_peered":"2024-04-26T16:17:35.671754+0000","last_unstale":"2024-04-26T16:20:30.223905+0000","last_undegraded":"2024-04-26T16:20:30.223905+0000","last_fullsized":"2024-04-26T16:20:30.223905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:20.635372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.635372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:47.043710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404886+0000","last_change":"2024-04-26T16:19:25.680293+0000","last_active":"2024-04-26T16:20:30.404886+0000","last_peered":"2024-04-26T16:20:30.404886+0000","last_clean":"2024-04-26T16:20:30.404886+0000","last_became_active":"2024-04-26T16:17:33.050011+0000","last_became_peered":"2024-04-26T16:17:33.050011+0000","last_unstale":"2024-04-26T16:20:30.404886+0000","last_undegraded":"2024-04-26T16:20:30.404886+0000","last_fullsized":"2024-04-26T16:20:30.404886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:25.680038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:25.680038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:40.561455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067322+0000","last_change":"2024-04-26T16:19:16.149085+0000","last_active":"2024-04-26T16:20:30.067322+0000","last_peered":"2024-04-26T16:20:30.067322+0000","last_clean":"2024-04-26T16:20:30.067322+0000","last_became_active":"2024-04-26T16:17:28.999634+0000","last_became_peered":"2024-04-26T16:17:28.999634+0000","last_unstale":"2024-04-26T16:20:30.067322+0000","last_undegraded":"2024-04-26T16:20:30.067322+0000","last_fullsized":"2024-04-26T16:20:30.067322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:16.148771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:16.148771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.303316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'4","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067132+0000","last_change":"2024-04-26T16:20:26.899539+0000","last_active":"2024-04-26T16:20:30.067132+0000","last_peered":"2024-04-26T16:20:30.067132+0000","last_clean":"2024-04-26T16:20:30.067132+0000","last_became_active":"2024-04-26T16:17:31.026037+0000","last_became_peered":"2024-04-26T16:17:31.026037+0000","last_unstale":"2024-04-26T16:20:30.067132+0000","last_undegraded":"2024-04-26T16:20:30.067132+0000","last_fullsized":"2024-04-26T16:20:30.067132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'4","last_scrub_stamp":"2024-04-26T16:20:26.899299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.899299+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.728642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345321+0000","last_change":"2024-04-26T16:19:54.349951+0000","last_active":"2024-04-26T16:20:30.345321+0000","last_peered":"2024-04-26T16:20:30.345321+0000","last_clean":"2024-04-26T16:20:30.345321+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:20:30.345321+0000","last_undegraded":"2024-04-26T16:20:30.345321+0000","last_fullsized":"2024-04-26T16:20:30.345321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:54.349649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:54.349649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:55.254477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406930+0000","last_change":"2024-04-26T16:19:09.913165+0000","last_active":"2024-04-26T16:20:30.406930+0000","last_peered":"2024-04-26T16:20:30.406930+0000","last_clean":"2024-04-26T16:20:30.406930+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:20:30.406930+0000","last_undegraded":"2024-04-26T16:20:30.406930+0000","last_fullsized":"2024-04-26T16:20:30.406930+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:09.912789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:09.912789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344460+0000","last_change":"2024-04-26T16:20:27.565849+0000","last_active":"2024-04-26T16:20:30.344460+0000","last_peered":"2024-04-26T16:20:30.344460+0000","last_clean":"2024-04-26T16:20:30.344460+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:20:30.344460+0000","last_undegraded":"2024-04-26T16:20:30.344460+0000","last_fullsized":"2024-04-26T16:20:30.344460+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:27.565529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:27.565529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.357713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222840+0000","last_change":"2024-04-26T16:20:14.278241+0000","last_active":"2024-04-26T16:20:30.222840+0000","last_peered":"2024-04-26T16:20:30.222840+0000","last_clean":"2024-04-26T16:20:30.222840+0000","last_became_active":"2024-04-26T16:17:31.033294+0000","last_became_peered":"2024-04-26T16:17:31.033294+0000","last_unstale":"2024-04-26T16:20:30.222840+0000","last_undegraded":"2024-04-26T16:20:30.222840+0000","last_fullsized":"2024-04-26T16:20:30.222840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:14.278005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.278005+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.458748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406676+0000","last_change":"2024-04-26T16:19:08.931731+0000","last_active":"2024-04-26T16:20:30.406676+0000","last_peered":"2024-04-26T16:20:30.406676+0000","last_clean":"2024-04-26T16:20:30.406676+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:20:30.406676+0000","last_undegraded":"2024-04-26T16:20:30.406676+0000","last_fullsized":"2024-04-26T16:20:30.406676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:08.931368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:08.931368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.434043+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017458+0000","last_change":"2024-04-26T16:20:13.890462+0000","last_active":"2024-04-26T16:20:30.017458+0000","last_peered":"2024-04-26T16:20:30.017458+0000","last_clean":"2024-04-26T16:20:30.017458+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:20:30.017458+0000","last_undegraded":"2024-04-26T16:20:30.017458+0000","last_fullsized":"2024-04-26T16:20:30.017458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:20:13.890111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.890111+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.062559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224305+0000","last_change":"2024-04-26T16:19:45.378650+0000","last_active":"2024-04-26T16:20:30.224305+0000","last_peered":"2024-04-26T16:20:30.224305+0000","last_clean":"2024-04-26T16:20:30.224305+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:20:30.224305+0000","last_undegraded":"2024-04-26T16:20:30.224305+0000","last_fullsized":"2024-04-26T16:20:30.224305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:45.378259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:45.378259+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:54.005675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344161+0000","last_change":"2024-04-26T16:20:14.406999+0000","last_active":"2024-04-26T16:20:30.344161+0000","last_peered":"2024-04-26T16:20:30.344161+0000","last_clean":"2024-04-26T16:20:30.344161+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:20:30.344161+0000","last_undegraded":"2024-04-26T16:20:30.344161+0000","last_fullsized":"2024-04-26T16:20:30.344161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.406805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.406805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.652307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015128+0000","last_change":"2024-04-26T16:19:15.674226+0000","last_active":"2024-04-26T16:20:30.015128+0000","last_peered":"2024-04-26T16:20:30.015128+0000","last_clean":"2024-04-26T16:20:30.015128+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:20:30.015128+0000","last_undegraded":"2024-04-26T16:20:30.015128+0000","last_fullsized":"2024-04-26T16:20:30.015128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:15.673990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:15.673990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016654+0000","last_change":"2024-04-26T16:19:15.229718+0000","last_active":"2024-04-26T16:20:30.016654+0000","last_peered":"2024-04-26T16:20:30.016654+0000","last_clean":"2024-04-26T16:20:30.016654+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:20:30.016654+0000","last_undegraded":"2024-04-26T16:20:30.016654+0000","last_fullsized":"2024-04-26T16:20:30.016654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:15.229495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:15.229495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.459154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"86'16","reported_seq":158,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404522+0000","last_change":"2024-04-26T16:20:21.529761+0000","last_active":"2024-04-26T16:20:30.404522+0000","last_peered":"2024-04-26T16:20:30.404522+0000","last_clean":"2024-04-26T16:20:30.404522+0000","last_became_active":"2024-04-26T16:17:31.040723+0000","last_became_peered":"2024-04-26T16:17:31.040723+0000","last_unstale":"2024-04-26T16:20:30.404522+0000","last_undegraded":"2024-04-26T16:20:30.404522+0000","last_fullsized":"2024-04-26T16:20:30.404522+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'16","last_scrub_stamp":"2024-04-26T16:20:21.529513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.529513+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.582915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016766+0000","last_change":"2024-04-26T16:19:59.129440+0000","last_active":"2024-04-26T16:20:30.016766+0000","last_peered":"2024-04-26T16:20:30.016766+0000","last_clean":"2024-04-26T16:20:30.016766+0000","last_became_active":"2024-04-26T16:17:32.046437+0000","last_became_peered":"2024-04-26T16:17:32.046437+0000","last_unstale":"2024-04-26T16:20:30.016766+0000","last_undegraded":"2024-04-26T16:20:30.016766+0000","last_fullsized":"2024-04-26T16:20:30.016766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:19:59.129167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.129167+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.401811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"86'11","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017595+0000","last_change":"2024-04-26T16:19:50.407803+0000","last_active":"2024-04-26T16:20:30.017595+0000","last_peered":"2024-04-26T16:20:30.017595+0000","last_clean":"2024-04-26T16:20:30.017595+0000","last_became_active":"2024-04-26T16:17:33.069020+0000","last_became_peered":"2024-04-26T16:17:33.069020+0000","last_unstale":"2024-04-26T16:20:30.017595+0000","last_undegraded":"2024-04-26T16:20:30.017595+0000","last_fullsized":"2024-04-26T16:20:30.017595+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:50.407700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:50.407700+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:57.863330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.407147+0000","last_change":"2024-04-26T16:20:09.046364+0000","last_active":"2024-04-26T16:20:30.407147+0000","last_peered":"2024-04-26T16:20:30.407147+0000","last_clean":"2024-04-26T16:20:30.407147+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:20:30.407147+0000","last_undegraded":"2024-04-26T16:20:30.407147+0000","last_fullsized":"2024-04-26T16:20:30.407147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.045989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.045989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.237368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403829+0000","last_change":"2024-04-26T16:20:13.525422+0000","last_active":"2024-04-26T16:20:30.403829+0000","last_peered":"2024-04-26T16:20:30.403829+0000","last_clean":"2024-04-26T16:20:30.403829+0000","last_became_active":"2024-04-26T16:17:29.022734+0000","last_became_peered":"2024-04-26T16:17:29.022734+0000","last_unstale":"2024-04-26T16:20:30.403829+0000","last_undegraded":"2024-04-26T16:20:30.403829+0000","last_fullsized":"2024-04-26T16:20:30.403829+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:13.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.525151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.383572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'27","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015122+0000","last_change":"2024-04-26T16:20:19.982781+0000","last_active":"2024-04-26T16:20:30.015122+0000","last_peered":"2024-04-26T16:20:30.015122+0000","last_clean":"2024-04-26T16:20:30.015122+0000","last_became_active":"2024-04-26T16:17:31.030904+0000","last_became_peered":"2024-04-26T16:17:31.030904+0000","last_unstale":"2024-04-26T16:20:30.015122+0000","last_undegraded":"2024-04-26T16:20:30.015122+0000","last_fullsized":"2024-04-26T16:20:30.015122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'27","last_scrub_stamp":"2024-04-26T16:20:19.982389+0000","last_deep_scrub":"103'27","last_deep_scrub_stamp":"2024-04-26T16:18:40.947097+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.982389+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.646180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016496+0000","last_change":"2024-04-26T16:20:24.200758+0000","last_active":"2024-04-26T16:20:30.016496+0000","last_peered":"2024-04-26T16:20:30.016496+0000","last_clean":"2024-04-26T16:20:30.016496+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:20:30.016496+0000","last_undegraded":"2024-04-26T16:20:30.016496+0000","last_fullsized":"2024-04-26T16:20:30.016496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:24.200448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.200448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.909147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224198+0000","last_change":"2024-04-26T16:20:16.270543+0000","last_active":"2024-04-26T16:20:30.224198+0000","last_peered":"2024-04-26T16:20:30.224198+0000","last_clean":"2024-04-26T16:20:30.224198+0000","last_became_active":"2024-04-26T16:17:35.670585+0000","last_became_peered":"2024-04-26T16:17:35.670585+0000","last_unstale":"2024-04-26T16:20:30.224198+0000","last_undegraded":"2024-04-26T16:20:30.224198+0000","last_fullsized":"2024-04-26T16:20:30.224198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:16.270321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:16.270321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.978435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":140,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017656+0000","last_change":"2024-04-26T16:19:43.114960+0000","last_active":"2024-04-26T16:20:30.017656+0000","last_peered":"2024-04-26T16:20:30.017656+0000","last_clean":"2024-04-26T16:20:30.017656+0000","last_became_active":"2024-04-26T16:17:31.023021+0000","last_became_peered":"2024-04-26T16:17:31.023021+0000","last_unstale":"2024-04-26T16:20:30.017656+0000","last_undegraded":"2024-04-26T16:20:30.017656+0000","last_fullsized":"2024-04-26T16:20:30.017656+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_clean_scrub_stamp":"2024-04-26T16:19:43.114691+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.523228+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344415+0000","last_change":"2024-04-26T16:19:52.393976+0000","last_active":"2024-04-26T16:20:30.344415+0000","last_peered":"2024-04-26T16:20:30.344415+0000","last_clean":"2024-04-26T16:20:30.344415+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:20:30.344415+0000","last_undegraded":"2024-04-26T16:20:30.344415+0000","last_fullsized":"2024-04-26T16:20:30.344415+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:52.393754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:19:52.393754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.078807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344996+0000","last_change":"2024-04-26T16:19:59.359204+0000","last_active":"2024-04-26T16:20:30.344996+0000","last_peered":"2024-04-26T16:20:30.344996+0000","last_clean":"2024-04-26T16:20:30.344996+0000","last_became_active":"2024-04-26T16:17:33.066780+0000","last_became_peered":"2024-04-26T16:17:33.066780+0000","last_unstale":"2024-04-26T16:20:30.344996+0000","last_undegraded":"2024-04-26T16:20:30.344996+0000","last_fullsized":"2024-04-26T16:20:30.344996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:59.358789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.358789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.479227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":83,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405681+0000","last_change":"2024-04-26T16:19:28.707694+0000","last_active":"2024-04-26T16:20:30.405681+0000","last_peered":"2024-04-26T16:20:30.405681+0000","last_clean":"2024-04-26T16:20:30.405681+0000","last_became_active":"2024-04-26T16:17:35.087291+0000","last_became_peered":"2024-04-26T16:17:35.087291+0000","last_unstale":"2024-04-26T16:20:30.405681+0000","last_undegraded":"2024-04-26T16:20:30.405681+0000","last_fullsized":"2024-04-26T16:20:30.405681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:28.707475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:28.707475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:50.575106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'10","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405790+0000","last_change":"2024-04-26T16:20:04.994840+0000","last_active":"2024-04-26T16:20:30.405790+0000","last_peered":"2024-04-26T16:20:30.405790+0000","last_clean":"2024-04-26T16:20:30.405790+0000","last_became_active":"2024-04-26T16:17:31.048785+0000","last_became_peered":"2024-04-26T16:17:31.048785+0000","last_unstale":"2024-04-26T16:20:30.405790+0000","last_undegraded":"2024-04-26T16:20:30.405790+0000","last_fullsized":"2024-04-26T16:20:30.405790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:04.994521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.994521+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:33.785270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403944+0000","last_change":"2024-04-26T16:20:06.715686+0000","last_active":"2024-04-26T16:20:30.403944+0000","last_peered":"2024-04-26T16:20:30.403944+0000","last_clean":"2024-04-26T16:20:30.403944+0000","last_became_active":"2024-04-26T16:17:29.013835+0000","last_became_peered":"2024-04-26T16:17:29.013835+0000","last_unstale":"2024-04-26T16:20:30.403944+0000","last_undegraded":"2024-04-26T16:20:30.403944+0000","last_fullsized":"2024-04-26T16:20:30.403944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:06.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:06.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.617561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":381,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222884+0000","last_change":"2024-04-26T16:19:42.393827+0000","last_active":"2024-04-26T16:20:30.222884+0000","last_peered":"2024-04-26T16:20:30.222884+0000","last_clean":"2024-04-26T16:20:30.222884+0000","last_became_active":"2024-04-26T16:16:50.925983+0000","last_became_peered":"2024-04-26T16:16:50.925983+0000","last_unstale":"2024-04-26T16:20:30.222884+0000","last_undegraded":"2024-04-26T16:20:30.222884+0000","last_fullsized":"2024-04-26T16:20:30.222884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:19:42.393587+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:19:42.393587+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:05.978678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224853+0000","last_change":"2024-04-26T16:19:30.568668+0000","last_active":"2024-04-26T16:20:30.224853+0000","last_peered":"2024-04-26T16:20:30.224853+0000","last_clean":"2024-04-26T16:20:30.224853+0000","last_became_active":"2024-04-26T16:17:33.061816+0000","last_became_peered":"2024-04-26T16:17:33.061816+0000","last_unstale":"2024-04-26T16:20:30.224853+0000","last_undegraded":"2024-04-26T16:20:30.224853+0000","last_fullsized":"2024-04-26T16:20:30.224853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:30.568415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:30.568415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.403532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016278+0000","last_change":"2024-04-26T16:19:24.155748+0000","last_active":"2024-04-26T16:20:30.016278+0000","last_peered":"2024-04-26T16:20:30.016278+0000","last_clean":"2024-04-26T16:20:30.016278+0000","last_became_active":"2024-04-26T16:17:35.090592+0000","last_became_peered":"2024-04-26T16:17:35.090592+0000","last_unstale":"2024-04-26T16:20:30.016278+0000","last_undegraded":"2024-04-26T16:20:30.016278+0000","last_fullsized":"2024-04-26T16:20:30.016278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:24.155543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:24.155543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.047837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"112'21","reported_seq":126,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014384+0000","last_change":"2024-04-26T16:19:47.424498+0000","last_active":"2024-04-26T16:20:30.014384+0000","last_peered":"2024-04-26T16:20:30.014384+0000","last_clean":"2024-04-26T16:20:30.014384+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:20:30.014384+0000","last_undegraded":"2024-04-26T16:20:30.014384+0000","last_fullsized":"2024-04-26T16:20:30.014384+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'21","last_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:19:47.424244+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:56.347516+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.226248+0000","last_change":"2024-04-26T16:20:30.226306+0000","last_active":"2024-04-26T16:20:30.226248+0000","last_peered":"2024-04-26T16:20:30.226248+0000","last_clean":"2024-04-26T16:20:30.226248+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:20:30.226248+0000","last_undegraded":"2024-04-26T16:20:30.226248+0000","last_fullsized":"2024-04-26T16:20:30.226248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:30.226091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:30.226091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.774441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":87,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405535+0000","last_change":"2024-04-26T16:19:23.706442+0000","last_active":"2024-04-26T16:20:30.405535+0000","last_peered":"2024-04-26T16:20:30.405535+0000","last_clean":"2024-04-26T16:20:30.405535+0000","last_became_active":"2024-04-26T16:17:33.046107+0000","last_became_peered":"2024-04-26T16:17:33.046107+0000","last_unstale":"2024-04-26T16:20:30.405535+0000","last_undegraded":"2024-04-26T16:20:30.405535+0000","last_fullsized":"2024-04-26T16:20:30.405535+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.706154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.706154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015504+0000","last_change":"2024-04-26T16:20:15.021647+0000","last_active":"2024-04-26T16:20:30.015504+0000","last_peered":"2024-04-26T16:20:30.015504+0000","last_clean":"2024-04-26T16:20:30.015504+0000","last_became_active":"2024-04-26T16:17:35.673254+0000","last_became_peered":"2024-04-26T16:17:35.673254+0000","last_unstale":"2024-04-26T16:20:30.015504+0000","last_undegraded":"2024-04-26T16:20:30.015504+0000","last_fullsized":"2024-04-26T16:20:30.015504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.021515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.021515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.753340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224480+0000","last_change":"2024-04-26T16:20:24.308841+0000","last_active":"2024-04-26T16:20:30.224480+0000","last_peered":"2024-04-26T16:20:30.224480+0000","last_clean":"2024-04-26T16:20:30.224480+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:20:30.224480+0000","last_undegraded":"2024-04-26T16:20:30.224480+0000","last_fullsized":"2024-04-26T16:20:30.224480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:24.308595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.308595+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:53.172263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"107'40","reported_seq":197,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016807+0000","last_change":"2024-04-26T16:20:25.051639+0000","last_active":"2024-04-26T16:20:30.016807+0000","last_peered":"2024-04-26T16:20:30.016807+0000","last_clean":"2024-04-26T16:20:30.016807+0000","last_became_active":"2024-04-26T16:17:31.024426+0000","last_became_peered":"2024-04-26T16:17:31.024426+0000","last_unstale":"2024-04-26T16:20:30.016807+0000","last_undegraded":"2024-04-26T16:20:30.016807+0000","last_fullsized":"2024-04-26T16:20:30.016807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'40","last_scrub_stamp":"2024-04-26T16:20:25.051382+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:18:56.138611+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.051382+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.857786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067813+0000","last_change":"2024-04-26T16:20:18.962156+0000","last_active":"2024-04-26T16:20:30.067813+0000","last_peered":"2024-04-26T16:20:30.067813+0000","last_clean":"2024-04-26T16:20:30.067813+0000","last_became_active":"2024-04-26T16:17:33.057789+0000","last_became_peered":"2024-04-26T16:17:33.057789+0000","last_unstale":"2024-04-26T16:20:30.067813+0000","last_undegraded":"2024-04-26T16:20:30.067813+0000","last_fullsized":"2024-04-26T16:20:30.067813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.961959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:53.089835+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.961959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.337483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014758+0000","last_change":"2024-04-26T16:19:57.996481+0000","last_active":"2024-04-26T16:20:30.014758+0000","last_peered":"2024-04-26T16:20:30.014758+0000","last_clean":"2024-04-26T16:20:30.014758+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:20:30.014758+0000","last_undegraded":"2024-04-26T16:20:30.014758+0000","last_fullsized":"2024-04-26T16:20:30.014758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:57.996170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:19:57.996170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.461563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015031+0000","last_change":"2024-04-26T16:20:18.047229+0000","last_active":"2024-04-26T16:20:30.015031+0000","last_peered":"2024-04-26T16:20:30.015031+0000","last_clean":"2024-04-26T16:20:30.015031+0000","last_became_active":"2024-04-26T16:17:29.004853+0000","last_became_peered":"2024-04-26T16:17:29.004853+0000","last_unstale":"2024-04-26T16:20:30.015031+0000","last_undegraded":"2024-04-26T16:20:30.015031+0000","last_fullsized":"2024-04-26T16:20:30.015031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.047049+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.615066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"86'13","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.407126+0000","last_change":"2024-04-26T16:19:19.940285+0000","last_active":"2024-04-26T16:20:30.407126+0000","last_peered":"2024-04-26T16:20:30.407126+0000","last_clean":"2024-04-26T16:20:30.407126+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:20:30.407126+0000","last_undegraded":"2024-04-26T16:20:30.407126+0000","last_fullsized":"2024-04-26T16:20:30.407126+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-04-26T16:19:19.939903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:19.939903+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:42.676901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":268,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405337+0000","last_change":"2024-04-26T16:20:11.073651+0000","last_active":"2024-04-26T16:20:30.405337+0000","last_peered":"2024-04-26T16:20:30.405337+0000","last_clean":"2024-04-26T16:20:30.405337+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:20:30.405337+0000","last_undegraded":"2024-04-26T16:20:30.405337+0000","last_fullsized":"2024-04-26T16:20:30.405337+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:20:11.073395+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:19:04.959471+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.073395+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.691968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":162,"num_read_kb":157,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014612+0000","last_change":"2024-04-26T16:20:21.947248+0000","last_active":"2024-04-26T16:20:30.014612+0000","last_peered":"2024-04-26T16:20:30.014612+0000","last_clean":"2024-04-26T16:20:30.014612+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:20:30.014612+0000","last_undegraded":"2024-04-26T16:20:30.014612+0000","last_fullsized":"2024-04-26T16:20:30.014612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.946995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.814368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223266+0000","last_change":"2024-04-26T16:20:17.304956+0000","last_active":"2024-04-26T16:20:30.223266+0000","last_peered":"2024-04-26T16:20:30.223266+0000","last_clean":"2024-04-26T16:20:30.223266+0000","last_became_active":"2024-04-26T16:17:35.670672+0000","last_became_peered":"2024-04-26T16:17:35.670672+0000","last_unstale":"2024-04-26T16:20:30.223266+0000","last_undegraded":"2024-04-26T16:20:30.223266+0000","last_fullsized":"2024-04-26T16:20:30.223266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.304722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.304722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.500725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068570+0000","last_change":"2024-04-26T16:19:58.917701+0000","last_active":"2024-04-26T16:20:30.068570+0000","last_peered":"2024-04-26T16:20:30.068570+0000","last_clean":"2024-04-26T16:20:30.068570+0000","last_became_active":"2024-04-26T16:17:29.006985+0000","last_became_peered":"2024-04-26T16:17:29.006985+0000","last_unstale":"2024-04-26T16:20:30.068570+0000","last_undegraded":"2024-04-26T16:20:30.068570+0000","last_fullsized":"2024-04-26T16:20:30.068570+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.917484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:16.115592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"107'18","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014975+0000","last_change":"2024-04-26T16:20:13.290879+0000","last_active":"2024-04-26T16:20:30.014975+0000","last_peered":"2024-04-26T16:20:30.014975+0000","last_clean":"2024-04-26T16:20:30.014975+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:20:30.014975+0000","last_undegraded":"2024-04-26T16:20:30.014975+0000","last_fullsized":"2024-04-26T16:20:30.014975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'18","last_scrub_stamp":"2024-04-26T16:20:13.290621+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:19:05.704281+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.290621+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.892369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015063+0000","last_change":"2024-04-26T16:20:06.968281+0000","last_active":"2024-04-26T16:20:30.015063+0000","last_peered":"2024-04-26T16:20:30.015063+0000","last_clean":"2024-04-26T16:20:30.015063+0000","last_became_active":"2024-04-26T16:17:32.027388+0000","last_became_peered":"2024-04-26T16:17:32.027388+0000","last_unstale":"2024-04-26T16:20:30.015063+0000","last_undegraded":"2024-04-26T16:20:30.015063+0000","last_fullsized":"2024-04-26T16:20:30.015063+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:06.968062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:20:06.968062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.035273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"86'11","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404967+0000","last_change":"2024-04-26T16:19:26.004779+0000","last_active":"2024-04-26T16:20:30.404967+0000","last_peered":"2024-04-26T16:20:30.404967+0000","last_clean":"2024-04-26T16:20:30.404967+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:20:30.404967+0000","last_undegraded":"2024-04-26T16:20:30.404967+0000","last_fullsized":"2024-04-26T16:20:30.404967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:26.004517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:26.004517+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.533302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406445+0000","last_change":"2024-04-26T16:20:10.026900+0000","last_active":"2024-04-26T16:20:30.406445+0000","last_peered":"2024-04-26T16:20:30.406445+0000","last_clean":"2024-04-26T16:20:30.406445+0000","last_became_active":"2024-04-26T16:17:35.673130+0000","last_became_peered":"2024-04-26T16:17:35.673130+0000","last_unstale":"2024-04-26T16:20:30.406445+0000","last_undegraded":"2024-04-26T16:20:30.406445+0000","last_fullsized":"2024-04-26T16:20:30.406445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.026659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.026659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:39.688645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405648+0000","last_change":"2024-04-26T16:20:28.439735+0000","last_active":"2024-04-26T16:20:30.405648+0000","last_peered":"2024-04-26T16:20:30.405648+0000","last_clean":"2024-04-26T16:20:30.405648+0000","last_became_active":"2024-04-26T16:17:33.068923+0000","last_became_peered":"2024-04-26T16:17:33.068923+0000","last_unstale":"2024-04-26T16:20:30.405648+0000","last_undegraded":"2024-04-26T16:20:30.405648+0000","last_fullsized":"2024-04-26T16:20:30.405648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.439524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.246043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223040+0000","last_change":"2024-04-26T16:20:10.311712+0000","last_active":"2024-04-26T16:20:30.223040+0000","last_peered":"2024-04-26T16:20:30.223040+0000","last_clean":"2024-04-26T16:20:30.223040+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:20:30.223040+0000","last_undegraded":"2024-04-26T16:20:30.223040+0000","last_fullsized":"2024-04-26T16:20:30.223040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:10.311472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.311472+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.794801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"86'12","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014846+0000","last_change":"2024-04-26T16:19:07.111227+0000","last_active":"2024-04-26T16:20:30.014846+0000","last_peered":"2024-04-26T16:20:30.014846+0000","last_clean":"2024-04-26T16:20:30.014846+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:20:30.014846+0000","last_undegraded":"2024-04-26T16:20:30.014846+0000","last_fullsized":"2024-04-26T16:20:30.014846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:19:07.110988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:07.110988+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:33.692258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404370+0000","last_change":"2024-04-26T16:20:23.546630+0000","last_active":"2024-04-26T16:20:30.404370+0000","last_peered":"2024-04-26T16:20:30.404370+0000","last_clean":"2024-04-26T16:20:30.404370+0000","last_became_active":"2024-04-26T16:17:35.077832+0000","last_became_peered":"2024-04-26T16:17:35.077832+0000","last_unstale":"2024-04-26T16:20:30.404370+0000","last_undegraded":"2024-04-26T16:20:30.404370+0000","last_fullsized":"2024-04-26T16:20:30.404370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.546406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.546406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:48.610382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"115'12","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016822+0000","last_change":"2024-04-26T16:20:25.975628+0000","last_active":"2024-04-26T16:20:30.016822+0000","last_peered":"2024-04-26T16:20:30.016822+0000","last_clean":"2024-04-26T16:20:30.016822+0000","last_became_active":"2024-04-26T16:17:33.070130+0000","last_became_peered":"2024-04-26T16:17:33.070130+0000","last_unstale":"2024-04-26T16:20:30.016822+0000","last_undegraded":"2024-04-26T16:20:30.016822+0000","last_fullsized":"2024-04-26T16:20:30.016822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.975399+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.920318+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015806+0000","last_change":"2024-04-26T16:18:54.145187+0000","last_active":"2024-04-26T16:20:30.015806+0000","last_peered":"2024-04-26T16:20:30.015806+0000","last_clean":"2024-04-26T16:20:30.015806+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:20:30.015806+0000","last_undegraded":"2024-04-26T16:20:30.015806+0000","last_fullsized":"2024-04-26T16:20:30.015806+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:18:54.144977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:18:54.144977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"86'15","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405173+0000","last_change":"2024-04-26T16:18:55.030748+0000","last_active":"2024-04-26T16:20:30.405173+0000","last_peered":"2024-04-26T16:20:30.405173+0000","last_clean":"2024-04-26T16:20:30.405173+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:20:30.405173+0000","last_undegraded":"2024-04-26T16:20:30.405173+0000","last_fullsized":"2024-04-26T16:20:30.405173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:18:55.030507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:18:55.030507+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017118+0000","last_change":"2024-04-26T16:20:14.907782+0000","last_active":"2024-04-26T16:20:30.017118+0000","last_peered":"2024-04-26T16:20:30.017118+0000","last_clean":"2024-04-26T16:20:30.017118+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:20:30.017118+0000","last_undegraded":"2024-04-26T16:20:30.017118+0000","last_fullsized":"2024-04-26T16:20:30.017118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.907547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.907547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.001180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"86'11","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344671+0000","last_change":"2024-04-26T16:20:03.405476+0000","last_active":"2024-04-26T16:20:30.344671+0000","last_peered":"2024-04-26T16:20:30.344671+0000","last_clean":"2024-04-26T16:20:30.344671+0000","last_became_active":"2024-04-26T16:17:33.066299+0000","last_became_peered":"2024-04-26T16:17:33.066299+0000","last_unstale":"2024-04-26T16:20:30.344671+0000","last_undegraded":"2024-04-26T16:20:30.344671+0000","last_fullsized":"2024-04-26T16:20:30.344671+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:03.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.405246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.150208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016445+0000","last_change":"2024-04-26T16:20:29.023874+0000","last_active":"2024-04-26T16:20:30.016445+0000","last_peered":"2024-04-26T16:20:30.016445+0000","last_clean":"2024-04-26T16:20:30.016445+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:20:30.016445+0000","last_undegraded":"2024-04-26T16:20:30.016445+0000","last_fullsized":"2024-04-26T16:20:30.016445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:29.023717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:29.023717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.818594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"86'9","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406172+0000","last_change":"2024-04-26T16:19:20.985067+0000","last_active":"2024-04-26T16:20:30.406172+0000","last_peered":"2024-04-26T16:20:30.406172+0000","last_clean":"2024-04-26T16:20:30.406172+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:20:30.406172+0000","last_undegraded":"2024-04-26T16:20:30.406172+0000","last_fullsized":"2024-04-26T16:20:30.406172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:20.984815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.984815+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:50.089853+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345537+0000","last_change":"2024-04-26T16:20:21.525478+0000","last_active":"2024-04-26T16:20:30.345537+0000","last_peered":"2024-04-26T16:20:30.345537+0000","last_clean":"2024-04-26T16:20:30.345537+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:20:30.345537+0000","last_undegraded":"2024-04-26T16:20:30.345537+0000","last_fullsized":"2024-04-26T16:20:30.345537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.525230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.078478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016419+0000","last_change":"2024-04-26T16:20:23.025108+0000","last_active":"2024-04-26T16:20:30.016419+0000","last_peered":"2024-04-26T16:20:30.016419+0000","last_clean":"2024-04-26T16:20:30.016419+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:20:30.016419+0000","last_undegraded":"2024-04-26T16:20:30.016419+0000","last_fullsized":"2024-04-26T16:20:30.016419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.024885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:01.215041+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.024885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.206757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223650+0000","last_change":"2024-04-26T16:19:13.705942+0000","last_active":"2024-04-26T16:20:30.223650+0000","last_peered":"2024-04-26T16:20:30.223650+0000","last_clean":"2024-04-26T16:20:30.223650+0000","last_became_active":"2024-04-26T16:17:29.026793+0000","last_became_peered":"2024-04-26T16:17:29.026793+0000","last_unstale":"2024-04-26T16:20:30.223650+0000","last_undegraded":"2024-04-26T16:20:30.223650+0000","last_fullsized":"2024-04-26T16:20:30.223650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:13.705663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:13.705663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"86'19","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067516+0000","last_change":"2024-04-26T16:20:11.904568+0000","last_active":"2024-04-26T16:20:30.067516+0000","last_peered":"2024-04-26T16:20:30.067516+0000","last_clean":"2024-04-26T16:20:30.067516+0000","last_became_active":"2024-04-26T16:17:31.024436+0000","last_became_peered":"2024-04-26T16:17:31.024436+0000","last_unstale":"2024-04-26T16:20:30.067516+0000","last_undegraded":"2024-04-26T16:20:30.067516+0000","last_fullsized":"2024-04-26T16:20:30.067516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'19","last_scrub_stamp":"2024-04-26T16:20:11.904292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.904292+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:17.775021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222613+0000","last_change":"2024-04-26T16:19:23.666736+0000","last_active":"2024-04-26T16:20:30.222613+0000","last_peered":"2024-04-26T16:20:30.222613+0000","last_clean":"2024-04-26T16:20:30.222613+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:20:30.222613+0000","last_undegraded":"2024-04-26T16:20:30.222613+0000","last_fullsized":"2024-04-26T16:20:30.222613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.666505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.666505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344485+0000","last_change":"2024-04-26T16:19:57.397306+0000","last_active":"2024-04-26T16:20:30.344485+0000","last_peered":"2024-04-26T16:20:30.344485+0000","last_clean":"2024-04-26T16:20:30.344485+0000","last_became_active":"2024-04-26T16:17:33.065671+0000","last_became_peered":"2024-04-26T16:17:33.065671+0000","last_unstale":"2024-04-26T16:20:30.344485+0000","last_undegraded":"2024-04-26T16:20:30.344485+0000","last_fullsized":"2024-04-26T16:20:30.344485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:57.396984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:57.396984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:00.083033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345016+0000","last_change":"2024-04-26T16:20:05.437174+0000","last_active":"2024-04-26T16:20:30.345016+0000","last_peered":"2024-04-26T16:20:30.345016+0000","last_clean":"2024-04-26T16:20:30.345016+0000","last_became_active":"2024-04-26T16:17:29.017152+0000","last_became_peered":"2024-04-26T16:17:29.017152+0000","last_unstale":"2024-04-26T16:20:30.345016+0000","last_undegraded":"2024-04-26T16:20:30.345016+0000","last_fullsized":"2024-04-26T16:20:30.345016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.436941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.436941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:07.446646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'17","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224815+0000","last_change":"2024-04-26T16:18:54.744984+0000","last_active":"2024-04-26T16:20:30.224815+0000","last_peered":"2024-04-26T16:20:30.224815+0000","last_clean":"2024-04-26T16:20:30.224815+0000","last_became_active":"2024-04-26T16:17:31.030244+0000","last_became_peered":"2024-04-26T16:17:31.030244+0000","last_unstale":"2024-04-26T16:20:30.224815+0000","last_undegraded":"2024-04-26T16:20:30.224815+0000","last_fullsized":"2024-04-26T16:20:30.224815+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-04-26T16:18:54.744818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:18:54.744818+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014136+0000","last_change":"2024-04-26T16:20:14.277898+0000","last_active":"2024-04-26T16:20:30.014136+0000","last_peered":"2024-04-26T16:20:30.014136+0000","last_clean":"2024-04-26T16:20:30.014136+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:20:30.014136+0000","last_undegraded":"2024-04-26T16:20:30.014136+0000","last_fullsized":"2024-04-26T16:20:30.014136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:25.766764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344537+0000","last_change":"2024-04-26T16:20:09.420985+0000","last_active":"2024-04-26T16:20:30.344537+0000","last_peered":"2024-04-26T16:20:30.344537+0000","last_clean":"2024-04-26T16:20:30.344537+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:20:30.344537+0000","last_undegraded":"2024-04-26T16:20:30.344537+0000","last_fullsized":"2024-04-26T16:20:30.344537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.420768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.420768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.763275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015646+0000","last_change":"2024-04-26T16:19:16.245606+0000","last_active":"2024-04-26T16:20:30.015646+0000","last_peered":"2024-04-26T16:20:30.015646+0000","last_clean":"2024-04-26T16:20:30.015646+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:20:30.015646+0000","last_undegraded":"2024-04-26T16:20:30.015646+0000","last_fullsized":"2024-04-26T16:20:30.015646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:16.245379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:16.245379+0000","last_clean_scrub_stamp":"2024-04-26T16:19:16.245379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.961824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'10","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404336+0000","last_change":"2024-04-26T16:18:50.560522+0000","last_active":"2024-04-26T16:20:30.404336+0000","last_peered":"2024-04-26T16:20:30.404336+0000","last_clean":"2024-04-26T16:20:30.404336+0000","last_became_active":"2024-04-26T16:17:31.040215+0000","last_became_peered":"2024-04-26T16:17:31.040215+0000","last_unstale":"2024-04-26T16:20:30.404336+0000","last_undegraded":"2024-04-26T16:20:30.404336+0000","last_fullsized":"2024-04-26T16:20:30.404336+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:18:50.560384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:18:50.560384+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403515+0000","last_change":"2024-04-26T16:19:53.748315+0000","last_active":"2024-04-26T16:20:30.403515+0000","last_peered":"2024-04-26T16:20:30.403515+0000","last_clean":"2024-04-26T16:20:30.403515+0000","last_became_active":"2024-04-26T16:17:35.669627+0000","last_became_peered":"2024-04-26T16:17:35.669627+0000","last_unstale":"2024-04-26T16:20:30.403515+0000","last_undegraded":"2024-04-26T16:20:30.403515+0000","last_fullsized":"2024-04-26T16:20:30.403515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:53.748063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:53.748063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:55.439580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345586+0000","last_change":"2024-04-26T16:20:18.397841+0000","last_active":"2024-04-26T16:20:30.345586+0000","last_peered":"2024-04-26T16:20:30.345586+0000","last_clean":"2024-04-26T16:20:30.345586+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:20:30.345586+0000","last_undegraded":"2024-04-26T16:20:30.345586+0000","last_fullsized":"2024-04-26T16:20:30.345586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.397696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:50.542517+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.397696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.381027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"86'15","reported_seq":123,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224801+0000","last_change":"2024-04-26T16:20:28.238717+0000","last_active":"2024-04-26T16:20:30.224801+0000","last_peered":"2024-04-26T16:20:30.224801+0000","last_clean":"2024-04-26T16:20:30.224801+0000","last_became_active":"2024-04-26T16:17:31.033340+0000","last_became_peered":"2024-04-26T16:17:31.033340+0000","last_unstale":"2024-04-26T16:20:30.224801+0000","last_undegraded":"2024-04-26T16:20:30.224801+0000","last_fullsized":"2024-04-26T16:20:30.224801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:28.238415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.238415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.888574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"109'12","reported_seq":134,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223156+0000","last_change":"2024-04-26T16:19:17.690652+0000","last_active":"2024-04-26T16:20:30.223156+0000","last_peered":"2024-04-26T16:20:30.223156+0000","last_clean":"2024-04-26T16:20:30.223156+0000","last_became_active":"2024-04-26T16:17:33.070525+0000","last_became_peered":"2024-04-26T16:17:33.070525+0000","last_unstale":"2024-04-26T16:20:30.223156+0000","last_undegraded":"2024-04-26T16:20:30.223156+0000","last_fullsized":"2024-04-26T16:20:30.223156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_clean_scrub_stamp":"2024-04-26T16:19:17.690568+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.618043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":162,"num_read_kb":157,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":436,"ondisk_log_size":436,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743471,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28424,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716728,"statfs":{"total":95995035648,"available":95965929472,"internally_reserved":0,"allocated":1437696,"data_stored":1010315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135929,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1400832,"data_stored":1007544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":831488,"data_stored":418513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36736,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708416,"statfs":{"total":95995035648,"available":95957417984,"internally_reserved":0,"allocated":1425408,"data_stored":1009628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411426,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":811008,"data_stored":417669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084375,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":811008,"data_stored":417576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855437,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":851968,"data_stored":418557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462979,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717348,"statfs":{"total":95995035648,"available":95966564352,"internally_reserved":0,"allocated":802816,"data_stored":409891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:20:32.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:20:33.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:33 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.254506+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:20:33.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:33 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.254506+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:20:33.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:33 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.256988+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:20:33.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:33 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.256988+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:20:33.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.254506+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.254506+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.256988+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.256988+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.254506+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.254506+0000 osd.0 (osd.0) 75 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.256988+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:20:33.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:33 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.256988+0000 osd.0 (osd.0) 76 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.861907+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.861907+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.864556+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:32.864556+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.239516+0000 osd.7 (osd.7) 85 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.239516+0000 osd.7 (osd.7) 85 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:20:34.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.241727+0000 osd.7 (osd.7) 86 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:20:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.241727+0000 osd.7 (osd.7) 86 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:20:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.397236+0000 mgr.y (mgr.24401) 113 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:34.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:34 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.397236+0000 mgr.y (mgr.24401) 113 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.861907+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.861907+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.864556+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:32.864556+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.239516+0000 osd.7 (osd.7) 85 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.239516+0000 osd.7 (osd.7) 85 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.241727+0000 osd.7 (osd.7) 86 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.241727+0000 osd.7 (osd.7) 86 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.397236+0000 mgr.y (mgr.24401) 113 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:34.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.397236+0000 mgr.y (mgr.24401) 113 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.861907+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.861907+0000 osd.3 (osd.3) 71 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.864556+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:32.864556+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.239516+0000 osd.7 (osd.7) 85 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.239516+0000 osd.7 (osd.7) 85 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.241727+0000 osd.7 (osd.7) 86 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.241727+0000 osd.7 (osd.7) 86 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.397236+0000 mgr.y (mgr.24401) 113 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:34.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.397236+0000 mgr.y (mgr.24401) 113 : cluster [DBG] pgmap v84: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:34.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:20:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.852872+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.852872+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.854210+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.854210+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.855230+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.855230+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.858410+0000 osd.4 (osd.4) 70 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:20:35.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:35 smithi190 bash[23868]: cluster 2024-04-26T16:20:33.858410+0000 osd.4 (osd.4) 70 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:20:35.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.852872+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.852872+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.854210+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.854210+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.855230+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.855230+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.858410+0000 osd.4 (osd.4) 70 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[21463]: cluster 2024-04-26T16:20:33.858410+0000 osd.4 (osd.4) 70 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.852872+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.852872+0000 osd.3 (osd.3) 73 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.854210+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:20:35.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.854210+0000 osd.4 (osd.4) 69 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:20:35.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.855230+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:20:35.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.855230+0000 osd.3 (osd.3) 74 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:20:35.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.858410+0000 osd.4 (osd.4) 70 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:20:35.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:35 smithi050 bash[29234]: cluster 2024-04-26T16:20:33.858410+0000 osd.4 (osd.4) 70 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:20:36.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.522002+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.c scrub starts 2024-04-26T16:20:36.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.522002+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.c scrub starts 2024-04-26T16:20:36.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.526095+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.c scrub ok 2024-04-26T16:20:36.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.526095+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.c scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.895018+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.895018+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.898806+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:34.898806+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.043520+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.043520+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.046837+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.046837+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.284758+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.284758+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.289711+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.289711+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-04-26T16:20:36.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.397950+0000 mgr.y (mgr.24401) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:36.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.397950+0000 mgr.y (mgr.24401) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.522002+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.c scrub starts 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.522002+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.c scrub starts 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.526095+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.c scrub ok 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.526095+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.c scrub ok 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.895018+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.895018+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.898806+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:34.898806+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.043520+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:20:36.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.043520+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.046837+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.046837+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.284758+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.284758+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.289711+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.289711+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.397950+0000 mgr.y (mgr.24401) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.397950+0000 mgr.y (mgr.24401) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.522002+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.c scrub starts 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.522002+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.c scrub starts 2024-04-26T16:20:36.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.526095+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.c scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.526095+0000 osd.5 (osd.5) 78 : cluster [DBG] 3.c scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.895018+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.895018+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.898806+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:34.898806+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.043520+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.043520+0000 osd.1 (osd.1) 75 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.046837+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.046837+0000 osd.1 (osd.1) 76 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.284758+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.284758+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.d scrub starts 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.289711+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.289711+0000 osd.7 (osd.7) 88 : cluster [DBG] 3.d scrub ok 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.397950+0000 mgr.y (mgr.24401) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:36.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:36 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.397950+0000 mgr.y (mgr.24401) 114 : cluster [DBG] pgmap v85: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:20:36.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:20:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:20:37.238 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:20:37.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.549793+0000 osd.5 (osd.5) 79 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:20:37.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.549793+0000 osd.5 (osd.5) 79 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:20:37.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.552485+0000 osd.5 (osd.5) 80 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:20:37.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.552485+0000 osd.5 (osd.5) 80 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.917012+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.917012+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.919449+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:35.919449+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.294928+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.294928+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.297199+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.297199+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: audit 2024-04-26T16:20:36.446085+0000 mgr.y (mgr.24401) 115 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:37.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:37 smithi190 bash[23868]: audit 2024-04-26T16:20:36.446085+0000 mgr.y (mgr.24401) 115 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.549793+0000 osd.5 (osd.5) 79 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.549793+0000 osd.5 (osd.5) 79 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.552485+0000 osd.5 (osd.5) 80 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.552485+0000 osd.5 (osd.5) 80 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.917012+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.917012+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.919449+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:35.919449+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.294928+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:20:37.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.294928+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.297199+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.297199+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: audit 2024-04-26T16:20:36.446085+0000 mgr.y (mgr.24401) 115 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[29234]: audit 2024-04-26T16:20:36.446085+0000 mgr.y (mgr.24401) 115 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.549793+0000 osd.5 (osd.5) 79 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.549793+0000 osd.5 (osd.5) 79 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.552485+0000 osd.5 (osd.5) 80 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.552485+0000 osd.5 (osd.5) 80 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.917012+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.917012+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.919449+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:20:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:35.919449+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:20:37.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.294928+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:20:37.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.294928+0000 osd.7 (osd.7) 89 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:20:37.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.297199+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:20:37.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.297199+0000 osd.7 (osd.7) 90 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:20:37.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: audit 2024-04-26T16:20:36.446085+0000 mgr.y (mgr.24401) 115 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:37.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:37 smithi050 bash[21463]: audit 2024-04-26T16:20:36.446085+0000 mgr.y (mgr.24401) 115 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.805271+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.805271+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.805433+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.805433+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.949878+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.949878+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:20:38.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.954380+0000 osd.3 (osd.3) 80 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:20:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:36.954380+0000 osd.3 (osd.3) 80 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:20:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.072700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-04-26T16:20:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.072700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-04-26T16:20:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.398652+0000 mgr.y (mgr.24401) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:38 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.398652+0000 mgr.y (mgr.24401) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.805271+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.805271+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.805433+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.805433+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.949878+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.949878+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:20:38.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.954380+0000 osd.3 (osd.3) 80 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:36.954380+0000 osd.3 (osd.3) 80 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.072700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.072700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.398652+0000 mgr.y (mgr.24401) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.398652+0000 mgr.y (mgr.24401) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.805271+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.805271+0000 osd.5 (osd.5) 81 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.805433+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.805433+0000 osd.5 (osd.5) 82 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.949878+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:20:38.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.949878+0000 osd.3 (osd.3) 79 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:20:38.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.954380+0000 osd.3 (osd.3) 80 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:20:38.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:36.954380+0000 osd.3 (osd.3) 80 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:20:38.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.072700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-04-26T16:20:38.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.072700+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2024-04-26T16:20:38.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.398652+0000 mgr.y (mgr.24401) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:38.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:38 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.398652+0000 mgr.y (mgr.24401) 116 : cluster [DBG] pgmap v87: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:39.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.945734+0000 osd.1 (osd.1) 77 : cluster [DBG] 3.19 deep-scrub starts 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.945734+0000 osd.1 (osd.1) 77 : cluster [DBG] 3.19 deep-scrub starts 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.959718+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.19 deep-scrub ok 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:37.959718+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.19 deep-scrub ok 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.322139+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.322139+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.324591+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:20:39.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:39 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.324591+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.945734+0000 osd.1 (osd.1) 77 : cluster [DBG] 3.19 deep-scrub starts 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.945734+0000 osd.1 (osd.1) 77 : cluster [DBG] 3.19 deep-scrub starts 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.959718+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.19 deep-scrub ok 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:37.959718+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.19 deep-scrub ok 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.322139+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.322139+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.324591+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.324591+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:20:39.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.945734+0000 osd.1 (osd.1) 77 : cluster [DBG] 3.19 deep-scrub starts 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.945734+0000 osd.1 (osd.1) 77 : cluster [DBG] 3.19 deep-scrub starts 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.959718+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.19 deep-scrub ok 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:37.959718+0000 osd.1 (osd.1) 78 : cluster [DBG] 3.19 deep-scrub ok 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.322139+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.322139+0000 osd.0 (osd.0) 77 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.324591+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:20:39.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:39 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.324591+0000 osd.0 (osd.0) 78 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:20:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.993747+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:20:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.993747+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:20:40.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.996083+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[29234]: cluster 2024-04-26T16:20:38.996083+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[29234]: cluster 2024-04-26T16:20:39.399413+0000 mgr.y (mgr.24401) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[29234]: cluster 2024-04-26T16:20:39.399413+0000 mgr.y (mgr.24401) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.993747+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.993747+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.996083+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[21463]: cluster 2024-04-26T16:20:38.996083+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[21463]: cluster 2024-04-26T16:20:39.399413+0000 mgr.y (mgr.24401) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:20:40.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:40 smithi050 bash[21463]: cluster 2024-04-26T16:20:39.399413+0000 mgr.y (mgr.24401) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:20:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- 172.21.15.50:0/420690844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c80103e60 msgr2=0x7f0c801102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- 172.21.15.50:0/420690844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c80103e60 0x7f0c801102b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c78009990 tx=0x7f0c78009ca0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- 172.21.15.50:0/420690844 shutdown_connections 2024-04-26T16:20:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- 172.21.15.50:0/420690844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c801094e0 0x7f0c801098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- 172.21.15.50:0/420690844 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c80103e60 0x7f0c801102b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- 172.21.15.50:0/420690844 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 0x7f0c80103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- 172.21.15.50:0/420690844 >> 172.21.15.50:0/420690844 conn(0x7f0c8007a590 msgr2=0x7f0c8007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- 172.21.15.50:0/420690844 shutdown_connections 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- 172.21.15.50:0/420690844 wait complete. 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 librados: starting msgr at 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 librados: starting objecter 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 Processor -- start 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- start start 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 librados: setting wanted keys 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 librados: calling monclient init 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 0x7f0c80075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c80103e60 0x7f0c80075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 0x7f0c800761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0c80112ad0 con 0x7f0c801094e0 2024-04-26T16:20:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0c80112950 con 0x7f0c80103e60 2024-04-26T16:20:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c88375700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0c80112c50 con 0x7f0c801034a0 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 0x7f0c800761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86111700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 0x7f0c80075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 0x7f0c800761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42044/0 (socket says 172.21.15.50:42044) 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86111700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 0x7f0c80075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:34546/0 (socket says 172.21.15.50:34546) 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 -- 172.21.15.50:0/2033868558 learned_addr learned my addr 172.21.15.50:0/2033868558 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c85910700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c80103e60 0x7f0c80075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 -- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 msgr2=0x7f0c80075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 0x7f0c80075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 -- 172.21.15.50:0/2033868558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c80103e60 msgr2=0x7f0c80075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c80103e60 0x7f0c80075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 -- 172.21.15.50:0/2033868558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c801a7620 con 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c86912700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 0x7f0c800761a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800c010 tx=0x7f0c78009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c777fe700 10 client.?.objecter ms_handle_connect 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.410+0000 7f0c777fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78027070 con 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0c78022470 con 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c801a78b0 con 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c78017650 con 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c88375700 10 client.14685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c801a7d90 con 0x7f0c801094e0 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c88375700 1 librados: init done 2024-04-26T16:20:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c8006a110 con 0x7f0c801094e0 2024-04-26T16:20:40.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0c7802a070 con 0x7f0c801094e0 2024-04-26T16:20:40.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0c6c071f80 0x7f0c6c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:40.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(118..118 src has 67..118) ==== 7450+0+0 (secure 0 0 0) 0x7f0c78092b10 con 0x7f0c801094e0 2024-04-26T16:20:40.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 10 client.14685.objecter ms_dispatch 0x7f0c80001550 osd_map(118..118 src has 67..118) 2024-04-26T16:20:40.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 3 client.14685.objecter handle_osd_map got epochs [118,118] > 0 2024-04-26T16:20:40.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 3 client.14685.objecter handle_osd_map decoding full epoch 118 2024-04-26T16:20:40.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c86111700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0c6c071f80 0x7f0c6c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:40.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 20 client.14685.objecter dump_active .. 0 homeless 2024-04-26T16:20:40.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c777fe700 10 client.14685.objecter ms_handle_connect 0x7f0c6c071f80 2024-04-26T16:20:40.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.414+0000 7f0c86111700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0c6c071f80 0x7f0c6c074440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c007fd0 tx=0x7f0c7c00c040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:40.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.418+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f0c78031070 con 0x7f0c801094e0 2024-04-26T16:20:40.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:40 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.993747+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:20:40.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:40 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.993747+0000 osd.1 (osd.1) 79 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:20:40.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:40 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.996083+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:20:40.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:40 smithi190 bash[23868]: cluster 2024-04-26T16:20:38.996083+0000 osd.1 (osd.1) 80 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:20:40.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:40 smithi190 bash[23868]: cluster 2024-04-26T16:20:39.399413+0000 mgr.y (mgr.24401) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:20:40.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:40 smithi190 bash[23868]: cluster 2024-04-26T16:20:39.399413+0000 mgr.y (mgr.24401) 117 : cluster [DBG] pgmap v88: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:20:40.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.638+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c80102300 con 0x7f0c6c071f80 2024-04-26T16:20:40.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.650+0000 7f0c777fe700 1 -- 172.21.15.50:0/2033868558 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346562 (secure 0 0 0) 0x7f0c80102300 con 0x7f0c6c071f80 2024-04-26T16:20:40.645 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 10 librados: watch_flush enter 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 10 librados: watch_flush exit 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 20 client.14685.objecter shutdown clearing up homeless session... 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 10 client.14685.objecter successfully canceled tick 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0c6c071f80 msgr2=0x7f0c6c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0c6c071f80 0x7f0c6c074440 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c007fd0 tx=0x7f0c7c00c040 comp rx=0 tx=0).stop 2024-04-26T16:20:40.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 msgr2=0x7f0c800761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 0x7f0c800761a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800c010 tx=0x7f0c78009580 comp rx=0 tx=0).stop 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 shutdown_connections 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0c6c071f80 0x7f0c6c074440 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0c801094e0 0x7f0c800761a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0c801034a0 0x7f0c80075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 --2- 172.21.15.50:0/2033868558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0c80103e60 0x7f0c80075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:40.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.654+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 >> 172.21.15.50:0/2033868558 conn(0x7f0c8007a590 msgr2=0x7f0c80101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:40.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.658+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 shutdown_connections 2024-04-26T16:20:40.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.658+0000 7f0c88375700 1 -- 172.21.15.50:0/2033868558 wait complete. 2024-04-26T16:20:40.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:40.658+0000 7f0c88375700 1 librados: shutdown 2024-04-26T16:20:40.652 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:20:41.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[21463]: cluster 2024-04-26T16:20:40.009319+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:20:41.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[21463]: cluster 2024-04-26T16:20:40.009319+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:20:41.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[21463]: cluster 2024-04-26T16:20:40.011708+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:20:41.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[21463]: cluster 2024-04-26T16:20:40.011708+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:20:41.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[21463]: audit 2024-04-26T16:20:40.699716+0000 mon.c (mon.2) 71 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:41.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[21463]: audit 2024-04-26T16:20:40.699716+0000 mon.c (mon.2) 71 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[29234]: cluster 2024-04-26T16:20:40.009319+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:20:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[29234]: cluster 2024-04-26T16:20:40.009319+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:20:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[29234]: cluster 2024-04-26T16:20:40.011708+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:20:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[29234]: cluster 2024-04-26T16:20:40.011708+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:20:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[29234]: audit 2024-04-26T16:20:40.699716+0000 mon.c (mon.2) 71 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:41.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:41 smithi050 bash[29234]: audit 2024-04-26T16:20:40.699716+0000 mon.c (mon.2) 71 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:41.553 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-04-26T16:20:39.398974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1117,"num_read_kb":963,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":675,"ondisk_log_size":675,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240956,"kb_used_data":8188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720260,"statfs":{"total":767960285184,"available":767713546240,"internally_reserved":0,"allocated":8384512,"data_stored":5117393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":238276148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.385303"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405939+0000","last_change":"2024-04-26T16:19:21.967066+0000","last_active":"2024-04-26T16:20:30.405939+0000","last_peered":"2024-04-26T16:20:30.405939+0000","last_clean":"2024-04-26T16:20:30.405939+0000","last_became_active":"2024-04-26T16:17:35.677526+0000","last_became_peered":"2024-04-26T16:17:35.677526+0000","last_unstale":"2024-04-26T16:20:30.405939+0000","last_undegraded":"2024-04-26T16:20:30.405939+0000","last_fullsized":"2024-04-26T16:20:30.405939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:21.966835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:21.966835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014491+0000","last_change":"2024-04-26T16:20:05.019229+0000","last_active":"2024-04-26T16:20:30.014491+0000","last_peered":"2024-04-26T16:20:30.014491+0000","last_clean":"2024-04-26T16:20:30.014491+0000","last_became_active":"2024-04-26T16:17:33.070922+0000","last_became_peered":"2024-04-26T16:17:33.070922+0000","last_unstale":"2024-04-26T16:20:30.014491+0000","last_undegraded":"2024-04-26T16:20:30.014491+0000","last_fullsized":"2024-04-26T16:20:30.014491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.018938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.018938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.770490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015664+0000","last_change":"2024-04-26T16:20:17.275735+0000","last_active":"2024-04-26T16:20:30.015664+0000","last_peered":"2024-04-26T16:20:30.015664+0000","last_clean":"2024-04-26T16:20:30.015664+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:20:30.015664+0000","last_undegraded":"2024-04-26T16:20:30.015664+0000","last_fullsized":"2024-04-26T16:20:30.015664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.275530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.275530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:42.183654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"86'10","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406813+0000","last_change":"2024-04-26T16:19:39.904290+0000","last_active":"2024-04-26T16:20:30.406813+0000","last_peered":"2024-04-26T16:20:30.406813+0000","last_clean":"2024-04-26T16:20:30.406813+0000","last_became_active":"2024-04-26T16:17:31.049997+0000","last_became_peered":"2024-04-26T16:17:31.049997+0000","last_unstale":"2024-04-26T16:20:30.406813+0000","last_undegraded":"2024-04-26T16:20:30.406813+0000","last_fullsized":"2024-04-26T16:20:30.406813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:19:39.904035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:39.904035+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.444792+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014554+0000","last_change":"2024-04-26T16:20:02.038119+0000","last_active":"2024-04-26T16:20:30.014554+0000","last_peered":"2024-04-26T16:20:30.014554+0000","last_clean":"2024-04-26T16:20:30.014554+0000","last_became_active":"2024-04-26T16:17:35.065560+0000","last_became_peered":"2024-04-26T16:17:35.065560+0000","last_unstale":"2024-04-26T16:20:30.014554+0000","last_undegraded":"2024-04-26T16:20:30.014554+0000","last_fullsized":"2024-04-26T16:20:30.014554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.037900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.037900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.752700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015649+0000","last_change":"2024-04-26T16:20:03.150021+0000","last_active":"2024-04-26T16:20:30.015649+0000","last_peered":"2024-04-26T16:20:30.015649+0000","last_clean":"2024-04-26T16:20:30.015649+0000","last_became_active":"2024-04-26T16:17:33.058899+0000","last_became_peered":"2024-04-26T16:17:33.058899+0000","last_unstale":"2024-04-26T16:20:30.015649+0000","last_undegraded":"2024-04-26T16:20:30.015649+0000","last_fullsized":"2024-04-26T16:20:30.015649+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.149607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.149607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.024417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.407247+0000","last_change":"2024-04-26T16:20:03.947061+0000","last_active":"2024-04-26T16:20:30.407247+0000","last_peered":"2024-04-26T16:20:30.407247+0000","last_clean":"2024-04-26T16:20:30.407247+0000","last_became_active":"2024-04-26T16:17:29.016870+0000","last_became_peered":"2024-04-26T16:17:29.016870+0000","last_unstale":"2024-04-26T16:20:30.407247+0000","last_undegraded":"2024-04-26T16:20:30.407247+0000","last_fullsized":"2024-04-26T16:20:30.407247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.946950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.302253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015510+0000","last_change":"2024-04-26T16:20:04.348527+0000","last_active":"2024-04-26T16:20:30.015510+0000","last_peered":"2024-04-26T16:20:30.015510+0000","last_clean":"2024-04-26T16:20:30.015510+0000","last_became_active":"2024-04-26T16:17:31.041819+0000","last_became_peered":"2024-04-26T16:17:31.041819+0000","last_unstale":"2024-04-26T16:20:30.015510+0000","last_undegraded":"2024-04-26T16:20:30.015510+0000","last_fullsized":"2024-04-26T16:20:30.015510+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:04.348299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.348299+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.592422+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403406+0000","last_change":"2024-04-26T16:20:02.714702+0000","last_active":"2024-04-26T16:20:30.403406+0000","last_peered":"2024-04-26T16:20:30.403406+0000","last_clean":"2024-04-26T16:20:30.403406+0000","last_became_active":"2024-04-26T16:17:35.087319+0000","last_became_peered":"2024-04-26T16:17:35.087319+0000","last_unstale":"2024-04-26T16:20:30.403406+0000","last_undegraded":"2024-04-26T16:20:30.403406+0000","last_fullsized":"2024-04-26T16:20:30.403406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:20:02.714466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.714466+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.031920+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:33.241931+0000","last_change":"2024-04-26T16:20:33.242024+0000","last_active":"2024-04-26T16:20:33.241931+0000","last_peered":"2024-04-26T16:20:33.241931+0000","last_clean":"2024-04-26T16:20:33.241931+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:20:33.241931+0000","last_undegraded":"2024-04-26T16:20:33.241931+0000","last_fullsized":"2024-04-26T16:20:33.241931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:33.241737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.241737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.846027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224691+0000","last_change":"2024-04-26T16:19:55.486245+0000","last_active":"2024-04-26T16:20:30.224691+0000","last_peered":"2024-04-26T16:20:30.224691+0000","last_clean":"2024-04-26T16:20:30.224691+0000","last_became_active":"2024-04-26T16:17:29.022163+0000","last_became_peered":"2024-04-26T16:17:29.022163+0000","last_unstale":"2024-04-26T16:20:30.224691+0000","last_undegraded":"2024-04-26T16:20:30.224691+0000","last_fullsized":"2024-04-26T16:20:30.224691+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:55.486024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:55.486024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:06.433329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"86'15","reported_seq":124,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404825+0000","last_change":"2024-04-26T16:20:25.511190+0000","last_active":"2024-04-26T16:20:30.404825+0000","last_peered":"2024-04-26T16:20:30.404825+0000","last_clean":"2024-04-26T16:20:30.404825+0000","last_became_active":"2024-04-26T16:17:31.022500+0000","last_became_peered":"2024-04-26T16:17:31.022500+0000","last_unstale":"2024-04-26T16:20:30.404825+0000","last_undegraded":"2024-04-26T16:20:30.404825+0000","last_fullsized":"2024-04-26T16:20:30.404825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:25.510941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.510941+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.088665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015243+0000","last_change":"2024-04-26T16:19:20.658911+0000","last_active":"2024-04-26T16:20:30.015243+0000","last_peered":"2024-04-26T16:20:30.015243+0000","last_clean":"2024-04-26T16:20:30.015243+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:20:30.015243+0000","last_undegraded":"2024-04-26T16:20:30.015243+0000","last_fullsized":"2024-04-26T16:20:30.015243+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:20.658689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.658689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.930841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403964+0000","last_change":"2024-04-26T16:20:18.567822+0000","last_active":"2024-04-26T16:20:30.403964+0000","last_peered":"2024-04-26T16:20:30.403964+0000","last_clean":"2024-04-26T16:20:30.403964+0000","last_became_active":"2024-04-26T16:17:33.071659+0000","last_became_peered":"2024-04-26T16:17:33.071659+0000","last_unstale":"2024-04-26T16:20:30.403964+0000","last_undegraded":"2024-04-26T16:20:30.403964+0000","last_fullsized":"2024-04-26T16:20:30.403964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.567616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.567616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.293668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223359+0000","last_change":"2024-04-26T16:20:23.315306+0000","last_active":"2024-04-26T16:20:30.223359+0000","last_peered":"2024-04-26T16:20:30.223359+0000","last_clean":"2024-04-26T16:20:30.223359+0000","last_became_active":"2024-04-26T16:17:29.025659+0000","last_became_peered":"2024-04-26T16:17:29.025659+0000","last_unstale":"2024-04-26T16:20:30.223359+0000","last_undegraded":"2024-04-26T16:20:30.223359+0000","last_fullsized":"2024-04-26T16:20:30.223359+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.315099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.315099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.593677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'12","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404861+0000","last_change":"2024-04-26T16:20:08.659388+0000","last_active":"2024-04-26T16:20:30.404861+0000","last_peered":"2024-04-26T16:20:30.404861+0000","last_clean":"2024-04-26T16:20:30.404861+0000","last_became_active":"2024-04-26T16:17:31.019223+0000","last_became_peered":"2024-04-26T16:17:31.019223+0000","last_unstale":"2024-04-26T16:20:30.404861+0000","last_undegraded":"2024-04-26T16:20:30.404861+0000","last_fullsized":"2024-04-26T16:20:30.404861+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:08.659153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.659153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.636156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:32.864723+0000","last_change":"2024-04-26T16:20:32.864786+0000","last_active":"2024-04-26T16:20:32.864723+0000","last_peered":"2024-04-26T16:20:32.864723+0000","last_clean":"2024-04-26T16:20:32.864723+0000","last_became_active":"2024-04-26T16:17:35.673178+0000","last_became_peered":"2024-04-26T16:17:35.673178+0000","last_unstale":"2024-04-26T16:20:32.864723+0000","last_undegraded":"2024-04-26T16:20:32.864723+0000","last_fullsized":"2024-04-26T16:20:32.864723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:32.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:32.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:00.660802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016324+0000","last_change":"2024-04-26T16:20:26.931424+0000","last_active":"2024-04-26T16:20:30.016324+0000","last_peered":"2024-04-26T16:20:30.016324+0000","last_clean":"2024-04-26T16:20:30.016324+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:20:30.016324+0000","last_undegraded":"2024-04-26T16:20:30.016324+0000","last_fullsized":"2024-04-26T16:20:30.016324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.931219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.931219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.617802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406053+0000","last_change":"2024-04-26T16:19:23.952520+0000","last_active":"2024-04-26T16:20:30.406053+0000","last_peered":"2024-04-26T16:20:30.406053+0000","last_clean":"2024-04-26T16:20:30.406053+0000","last_became_active":"2024-04-26T16:17:29.005070+0000","last_became_peered":"2024-04-26T16:17:29.005070+0000","last_unstale":"2024-04-26T16:20:30.406053+0000","last_undegraded":"2024-04-26T16:20:30.406053+0000","last_fullsized":"2024-04-26T16:20:30.406053+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.952306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.952306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"86'9","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014821+0000","last_change":"2024-04-26T16:19:37.958917+0000","last_active":"2024-04-26T16:20:30.014821+0000","last_peered":"2024-04-26T16:20:30.014821+0000","last_clean":"2024-04-26T16:20:30.014821+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:20:30.014821+0000","last_undegraded":"2024-04-26T16:20:30.014821+0000","last_fullsized":"2024-04-26T16:20:30.014821+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:37.958674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:37.958674+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:53.537405+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017257+0000","last_change":"2024-04-26T16:20:15.936737+0000","last_active":"2024-04-26T16:20:30.017257+0000","last_peered":"2024-04-26T16:20:30.017257+0000","last_clean":"2024-04-26T16:20:30.017257+0000","last_became_active":"2024-04-26T16:17:35.674743+0000","last_became_peered":"2024-04-26T16:17:35.674743+0000","last_unstale":"2024-04-26T16:20:30.017257+0000","last_undegraded":"2024-04-26T16:20:30.017257+0000","last_fullsized":"2024-04-26T16:20:30.017257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.936392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.456145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015871+0000","last_change":"2024-04-26T16:20:26.994856+0000","last_active":"2024-04-26T16:20:30.015871+0000","last_peered":"2024-04-26T16:20:30.015871+0000","last_clean":"2024-04-26T16:20:30.015871+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:20:30.015871+0000","last_undegraded":"2024-04-26T16:20:30.015871+0000","last_fullsized":"2024-04-26T16:20:30.015871+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.994631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.994631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:56.773657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067533+0000","last_change":"2024-04-26T16:20:08.830657+0000","last_active":"2024-04-26T16:20:30.067533+0000","last_peered":"2024-04-26T16:20:30.067533+0000","last_clean":"2024-04-26T16:20:30.067533+0000","last_became_active":"2024-04-26T16:17:29.004486+0000","last_became_peered":"2024-04-26T16:17:29.004486+0000","last_unstale":"2024-04-26T16:20:30.067533+0000","last_undegraded":"2024-04-26T16:20:30.067533+0000","last_fullsized":"2024-04-26T16:20:30.067533+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.830446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.830446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.833491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"114'7","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017270+0000","last_change":"2024-04-26T16:20:22.150870+0000","last_active":"2024-04-26T16:20:30.017270+0000","last_peered":"2024-04-26T16:20:30.017270+0000","last_clean":"2024-04-26T16:20:30.017270+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:20:30.017270+0000","last_undegraded":"2024-04-26T16:20:30.017270+0000","last_fullsized":"2024-04-26T16:20:30.017270+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_clean_scrub_stamp":"2024-04-26T16:20:22.150727+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:47.002699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016084+0000","last_change":"2024-04-26T16:20:10.128158+0000","last_active":"2024-04-26T16:20:30.016084+0000","last_peered":"2024-04-26T16:20:30.016084+0000","last_clean":"2024-04-26T16:20:30.016084+0000","last_became_active":"2024-04-26T16:17:35.080236+0000","last_became_peered":"2024-04-26T16:17:35.080236+0000","last_unstale":"2024-04-26T16:20:30.016084+0000","last_undegraded":"2024-04-26T16:20:30.016084+0000","last_fullsized":"2024-04-26T16:20:30.016084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.127949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.127949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.505883+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014744+0000","last_change":"2024-04-26T16:20:03.390713+0000","last_active":"2024-04-26T16:20:30.014744+0000","last_peered":"2024-04-26T16:20:30.014744+0000","last_clean":"2024-04-26T16:20:30.014744+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:20:30.014744+0000","last_undegraded":"2024-04-26T16:20:30.014744+0000","last_fullsized":"2024-04-26T16:20:30.014744+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.390501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.390501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:06.150359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405357+0000","last_change":"2024-04-26T16:19:18.953972+0000","last_active":"2024-04-26T16:20:30.405357+0000","last_peered":"2024-04-26T16:20:30.405357+0000","last_clean":"2024-04-26T16:20:30.405357+0000","last_became_active":"2024-04-26T16:17:28.999524+0000","last_became_peered":"2024-04-26T16:17:28.999524+0000","last_unstale":"2024-04-26T16:20:30.405357+0000","last_undegraded":"2024-04-26T16:20:30.405357+0000","last_fullsized":"2024-04-26T16:20:30.405357+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:19:18.953733+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:19:18.953733+0000","last_clean_scrub_stamp":"2024-04-26T16:19:18.953733+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"86'9","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:34.898978+0000","last_change":"2024-04-26T16:20:34.899058+0000","last_active":"2024-04-26T16:20:34.898978+0000","last_peered":"2024-04-26T16:20:34.898978+0000","last_clean":"2024-04-26T16:20:34.898978+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:20:34.898978+0000","last_undegraded":"2024-04-26T16:20:34.898978+0000","last_fullsized":"2024-04-26T16:20:34.898978+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:34.898819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:34.898819+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.454975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224068+0000","last_change":"2024-04-26T16:19:34.482254+0000","last_active":"2024-04-26T16:20:30.224068+0000","last_peered":"2024-04-26T16:20:30.224068+0000","last_clean":"2024-04-26T16:20:30.224068+0000","last_became_active":"2024-04-26T16:17:35.081916+0000","last_became_peered":"2024-04-26T16:17:35.081916+0000","last_unstale":"2024-04-26T16:20:30.224068+0000","last_undegraded":"2024-04-26T16:20:30.224068+0000","last_fullsized":"2024-04-26T16:20:30.224068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:19:34.482014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:34.482014+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:40.383514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"86'11","reported_seq":145,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068667+0000","last_change":"2024-04-26T16:19:59.878190+0000","last_active":"2024-04-26T16:20:30.068667+0000","last_peered":"2024-04-26T16:20:30.068667+0000","last_clean":"2024-04-26T16:20:30.068667+0000","last_became_active":"2024-04-26T16:17:33.061926+0000","last_became_peered":"2024-04-26T16:17:33.061926+0000","last_unstale":"2024-04-26T16:20:30.068667+0000","last_undegraded":"2024-04-26T16:20:30.068667+0000","last_fullsized":"2024-04-26T16:20:30.068667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:59.877790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.877790+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.440948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403217+0000","last_change":"2024-04-26T16:19:58.724981+0000","last_active":"2024-04-26T16:20:30.403217+0000","last_peered":"2024-04-26T16:20:30.403217+0000","last_clean":"2024-04-26T16:20:30.403217+0000","last_became_active":"2024-04-26T16:17:29.025319+0000","last_became_peered":"2024-04-26T16:17:29.025319+0000","last_unstale":"2024-04-26T16:20:30.403217+0000","last_undegraded":"2024-04-26T16:20:30.403217+0000","last_fullsized":"2024-04-26T16:20:30.403217+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.724766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.724766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.832669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"86'15","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015946+0000","last_change":"2024-04-26T16:19:25.196360+0000","last_active":"2024-04-26T16:20:30.015946+0000","last_peered":"2024-04-26T16:20:30.015946+0000","last_clean":"2024-04-26T16:20:30.015946+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:20:30.015946+0000","last_undegraded":"2024-04-26T16:20:30.015946+0000","last_fullsized":"2024-04-26T16:20:30.015946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:19:25.196241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:25.196241+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:35.919623+0000","last_change":"2024-04-26T16:20:35.919687+0000","last_active":"2024-04-26T16:20:35.919623+0000","last_peered":"2024-04-26T16:20:35.919623+0000","last_clean":"2024-04-26T16:20:35.919623+0000","last_became_active":"2024-04-26T16:17:35.677366+0000","last_became_peered":"2024-04-26T16:17:35.677366+0000","last_unstale":"2024-04-26T16:20:35.919623+0000","last_undegraded":"2024-04-26T16:20:35.919623+0000","last_fullsized":"2024-04-26T16:20:35.919623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.919459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.919459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.324456+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223713+0000","last_change":"2024-04-26T16:20:26.239452+0000","last_active":"2024-04-26T16:20:30.223713+0000","last_peered":"2024-04-26T16:20:30.223713+0000","last_clean":"2024-04-26T16:20:30.223713+0000","last_became_active":"2024-04-26T16:17:33.075775+0000","last_became_peered":"2024-04-26T16:17:33.075775+0000","last_unstale":"2024-04-26T16:20:30.223713+0000","last_undegraded":"2024-04-26T16:20:30.223713+0000","last_fullsized":"2024-04-26T16:20:30.223713+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.239095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.239095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.234785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068716+0000","last_change":"2024-04-26T16:20:02.934191+0000","last_active":"2024-04-26T16:20:30.068716+0000","last_peered":"2024-04-26T16:20:30.068716+0000","last_clean":"2024-04-26T16:20:30.068716+0000","last_became_active":"2024-04-26T16:17:29.018752+0000","last_became_peered":"2024-04-26T16:17:29.018752+0000","last_unstale":"2024-04-26T16:20:30.068716+0000","last_undegraded":"2024-04-26T16:20:30.068716+0000","last_fullsized":"2024-04-26T16:20:30.068716+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.933912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.933912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.658980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"86'9","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404619+0000","last_change":"2024-04-26T16:19:03.478782+0000","last_active":"2024-04-26T16:20:30.404619+0000","last_peered":"2024-04-26T16:20:30.404619+0000","last_clean":"2024-04-26T16:20:30.404619+0000","last_became_active":"2024-04-26T16:17:31.024113+0000","last_became_peered":"2024-04-26T16:17:31.024113+0000","last_unstale":"2024-04-26T16:20:30.404619+0000","last_undegraded":"2024-04-26T16:20:30.404619+0000","last_fullsized":"2024-04-26T16:20:30.404619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:03.478587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:03.478587+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:32.285278+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222817+0000","last_change":"2024-04-26T16:19:15.680713+0000","last_active":"2024-04-26T16:20:30.222817+0000","last_peered":"2024-04-26T16:20:30.222817+0000","last_clean":"2024-04-26T16:20:30.222817+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:20:30.222817+0000","last_undegraded":"2024-04-26T16:20:30.222817+0000","last_fullsized":"2024-04-26T16:20:30.222817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:15.680333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:15.680333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067308+0000","last_change":"2024-04-26T16:19:23.064854+0000","last_active":"2024-04-26T16:20:30.067308+0000","last_peered":"2024-04-26T16:20:30.067308+0000","last_clean":"2024-04-26T16:20:30.067308+0000","last_became_active":"2024-04-26T16:17:33.059705+0000","last_became_peered":"2024-04-26T16:17:33.059705+0000","last_unstale":"2024-04-26T16:20:30.067308+0000","last_undegraded":"2024-04-26T16:20:30.067308+0000","last_fullsized":"2024-04-26T16:20:30.067308+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.064641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.064641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:48.435806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405333+0000","last_change":"2024-04-26T16:20:19.575996+0000","last_active":"2024-04-26T16:20:30.405333+0000","last_peered":"2024-04-26T16:20:30.405333+0000","last_clean":"2024-04-26T16:20:30.405333+0000","last_became_active":"2024-04-26T16:17:29.022454+0000","last_became_peered":"2024-04-26T16:17:29.022454+0000","last_unstale":"2024-04-26T16:20:30.405333+0000","last_undegraded":"2024-04-26T16:20:30.405333+0000","last_fullsized":"2024-04-26T16:20:30.405333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:19.575787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.575787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.873083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"86'6","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015346+0000","last_change":"2024-04-26T16:20:21.162238+0000","last_active":"2024-04-26T16:20:30.015346+0000","last_peered":"2024-04-26T16:20:30.015346+0000","last_clean":"2024-04-26T16:20:30.015346+0000","last_became_active":"2024-04-26T16:17:31.053359+0000","last_became_peered":"2024-04-26T16:17:31.053359+0000","last_unstale":"2024-04-26T16:20:30.015346+0000","last_undegraded":"2024-04-26T16:20:30.015346+0000","last_fullsized":"2024-04-26T16:20:30.015346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'6","last_scrub_stamp":"2024-04-26T16:20:21.161995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.161995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.190751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405130+0000","last_change":"2024-04-26T16:20:08.001062+0000","last_active":"2024-04-26T16:20:30.405130+0000","last_peered":"2024-04-26T16:20:30.405130+0000","last_clean":"2024-04-26T16:20:30.405130+0000","last_became_active":"2024-04-26T16:17:35.094615+0000","last_became_peered":"2024-04-26T16:17:35.094615+0000","last_unstale":"2024-04-26T16:20:30.405130+0000","last_undegraded":"2024-04-26T16:20:30.405130+0000","last_fullsized":"2024-04-26T16:20:30.405130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.000834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.000834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.457279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017005+0000","last_change":"2024-04-26T16:19:12.327137+0000","last_active":"2024-04-26T16:20:30.017005+0000","last_peered":"2024-04-26T16:20:30.017005+0000","last_clean":"2024-04-26T16:20:30.017005+0000","last_became_active":"2024-04-26T16:17:33.048202+0000","last_became_peered":"2024-04-26T16:17:33.048202+0000","last_unstale":"2024-04-26T16:20:30.017005+0000","last_undegraded":"2024-04-26T16:20:30.017005+0000","last_fullsized":"2024-04-26T16:20:30.017005+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:12.326888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:12.326888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.929054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015666+0000","last_change":"2024-04-26T16:19:19.243504+0000","last_active":"2024-04-26T16:20:30.015666+0000","last_peered":"2024-04-26T16:20:30.015666+0000","last_clean":"2024-04-26T16:20:30.015666+0000","last_became_active":"2024-04-26T16:17:29.011346+0000","last_became_peered":"2024-04-26T16:17:29.011346+0000","last_unstale":"2024-04-26T16:20:30.015666+0000","last_undegraded":"2024-04-26T16:20:30.015666+0000","last_fullsized":"2024-04-26T16:20:30.015666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:19.243261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:19.243261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:42.597153+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"86'10","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016663+0000","last_change":"2024-04-26T16:20:23.996060+0000","last_active":"2024-04-26T16:20:30.016663+0000","last_peered":"2024-04-26T16:20:30.016663+0000","last_clean":"2024-04-26T16:20:30.016663+0000","last_became_active":"2024-04-26T16:17:31.021476+0000","last_became_peered":"2024-04-26T16:17:31.021476+0000","last_unstale":"2024-04-26T16:20:30.016663+0000","last_undegraded":"2024-04-26T16:20:30.016663+0000","last_fullsized":"2024-04-26T16:20:30.016663+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:23.995817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.995817+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.162295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014994+0000","last_change":"2024-04-26T16:20:25.238939+0000","last_active":"2024-04-26T16:20:30.014994+0000","last_peered":"2024-04-26T16:20:30.014994+0000","last_clean":"2024-04-26T16:20:30.014994+0000","last_became_active":"2024-04-26T16:17:35.078105+0000","last_became_peered":"2024-04-26T16:17:35.078105+0000","last_unstale":"2024-04-26T16:20:30.014994+0000","last_undegraded":"2024-04-26T16:20:30.014994+0000","last_fullsized":"2024-04-26T16:20:30.014994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:25.238710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:06.659791+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.238710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:31.861824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406335+0000","last_change":"2024-04-26T16:20:07.028942+0000","last_active":"2024-04-26T16:20:30.406335+0000","last_peered":"2024-04-26T16:20:30.406335+0000","last_clean":"2024-04-26T16:20:30.406335+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:20:30.406335+0000","last_undegraded":"2024-04-26T16:20:30.406335+0000","last_fullsized":"2024-04-26T16:20:30.406335+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:07.028724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:07.028724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.581251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068037+0000","last_change":"2024-04-26T16:20:12.898651+0000","last_active":"2024-04-26T16:20:30.068037+0000","last_peered":"2024-04-26T16:20:30.068037+0000","last_clean":"2024-04-26T16:20:30.068037+0000","last_became_active":"2024-04-26T16:17:29.013639+0000","last_became_peered":"2024-04-26T16:17:29.013639+0000","last_unstale":"2024-04-26T16:20:30.068037+0000","last_undegraded":"2024-04-26T16:20:30.068037+0000","last_fullsized":"2024-04-26T16:20:30.068037+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:20:12.898432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.131422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"86'9","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.225004+0000","last_change":"2024-04-26T16:20:09.306205+0000","last_active":"2024-04-26T16:20:30.225004+0000","last_peered":"2024-04-26T16:20:30.225004+0000","last_clean":"2024-04-26T16:20:30.225004+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:20:30.225004+0000","last_undegraded":"2024-04-26T16:20:30.225004+0000","last_fullsized":"2024-04-26T16:20:30.225004+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:09.305987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.305987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.952773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.018022+0000","last_change":"2024-04-26T16:19:14.082286+0000","last_active":"2024-04-26T16:20:30.018022+0000","last_peered":"2024-04-26T16:20:30.018022+0000","last_clean":"2024-04-26T16:20:30.018022+0000","last_became_active":"2024-04-26T16:17:35.070054+0000","last_became_peered":"2024-04-26T16:17:35.070054+0000","last_unstale":"2024-04-26T16:20:30.018022+0000","last_undegraded":"2024-04-26T16:20:30.018022+0000","last_fullsized":"2024-04-26T16:20:30.018022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:14.082070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:14.082070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:43.454813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":134,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406610+0000","last_change":"2024-04-26T16:19:26.961234+0000","last_active":"2024-04-26T16:20:30.406610+0000","last_peered":"2024-04-26T16:20:30.406610+0000","last_clean":"2024-04-26T16:20:30.406610+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:20:30.406610+0000","last_undegraded":"2024-04-26T16:20:30.406610+0000","last_fullsized":"2024-04-26T16:20:30.406610+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:26.960994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:26.960994+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:52.147716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017131+0000","last_change":"2024-04-26T16:20:15.241109+0000","last_active":"2024-04-26T16:20:30.017131+0000","last_peered":"2024-04-26T16:20:30.017131+0000","last_clean":"2024-04-26T16:20:30.017131+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:20:30.017131+0000","last_undegraded":"2024-04-26T16:20:30.017131+0000","last_fullsized":"2024-04-26T16:20:30.017131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.241014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.241014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.603907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014779+0000","last_change":"2024-04-26T16:20:00.355659+0000","last_active":"2024-04-26T16:20:30.014779+0000","last_peered":"2024-04-26T16:20:30.014779+0000","last_clean":"2024-04-26T16:20:30.014779+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:20:30.014779+0000","last_undegraded":"2024-04-26T16:20:30.014779+0000","last_fullsized":"2024-04-26T16:20:30.014779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:00.355248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:00.355248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:00.966364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014490+0000","last_change":"2024-04-26T16:19:10.614262+0000","last_active":"2024-04-26T16:20:30.014490+0000","last_peered":"2024-04-26T16:20:30.014490+0000","last_clean":"2024-04-26T16:20:30.014490+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:20:30.014490+0000","last_undegraded":"2024-04-26T16:20:30.014490+0000","last_fullsized":"2024-04-26T16:20:30.014490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:10.614042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:10.614042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.417468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"101'12","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405028+0000","last_change":"2024-04-26T16:19:51.757222+0000","last_active":"2024-04-26T16:20:30.405028+0000","last_peered":"2024-04-26T16:20:30.405028+0000","last_clean":"2024-04-26T16:20:30.405028+0000","last_became_active":"2024-04-26T16:17:33.062000+0000","last_became_peered":"2024-04-26T16:17:33.062000+0000","last_unstale":"2024-04-26T16:20:30.405028+0000","last_undegraded":"2024-04-26T16:20:30.405028+0000","last_fullsized":"2024-04-26T16:20:30.405028+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:19:51.756984+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:57.578136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403615+0000","last_change":"2024-04-26T16:20:05.684861+0000","last_active":"2024-04-26T16:20:30.403615+0000","last_peered":"2024-04-26T16:20:30.403615+0000","last_clean":"2024-04-26T16:20:30.403615+0000","last_became_active":"2024-04-26T16:17:29.025290+0000","last_became_peered":"2024-04-26T16:17:29.025290+0000","last_unstale":"2024-04-26T16:20:30.403615+0000","last_undegraded":"2024-04-26T16:20:30.403615+0000","last_fullsized":"2024-04-26T16:20:30.403615+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.684621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.684621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:12.223118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223712+0000","last_change":"2024-04-26T16:19:43.379294+0000","last_active":"2024-04-26T16:20:30.223712+0000","last_peered":"2024-04-26T16:20:30.223712+0000","last_clean":"2024-04-26T16:20:30.223712+0000","last_became_active":"2024-04-26T16:17:31.034570+0000","last_became_peered":"2024-04-26T16:17:31.034570+0000","last_unstale":"2024-04-26T16:20:30.223712+0000","last_undegraded":"2024-04-26T16:20:30.223712+0000","last_fullsized":"2024-04-26T16:20:30.223712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:43.379052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:43.379052+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:49.859888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223905+0000","last_change":"2024-04-26T16:19:20.635604+0000","last_active":"2024-04-26T16:20:30.223905+0000","last_peered":"2024-04-26T16:20:30.223905+0000","last_clean":"2024-04-26T16:20:30.223905+0000","last_became_active":"2024-04-26T16:17:35.671754+0000","last_became_peered":"2024-04-26T16:17:35.671754+0000","last_unstale":"2024-04-26T16:20:30.223905+0000","last_undegraded":"2024-04-26T16:20:30.223905+0000","last_fullsized":"2024-04-26T16:20:30.223905+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:20.635372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.635372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:47.043710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404886+0000","last_change":"2024-04-26T16:19:25.680293+0000","last_active":"2024-04-26T16:20:30.404886+0000","last_peered":"2024-04-26T16:20:30.404886+0000","last_clean":"2024-04-26T16:20:30.404886+0000","last_became_active":"2024-04-26T16:17:33.050011+0000","last_became_peered":"2024-04-26T16:17:33.050011+0000","last_unstale":"2024-04-26T16:20:30.404886+0000","last_undegraded":"2024-04-26T16:20:30.404886+0000","last_fullsized":"2024-04-26T16:20:30.404886+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:25.680038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:25.680038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:40.561455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067322+0000","last_change":"2024-04-26T16:19:16.149085+0000","last_active":"2024-04-26T16:20:30.067322+0000","last_peered":"2024-04-26T16:20:30.067322+0000","last_clean":"2024-04-26T16:20:30.067322+0000","last_became_active":"2024-04-26T16:17:28.999634+0000","last_became_peered":"2024-04-26T16:17:28.999634+0000","last_unstale":"2024-04-26T16:20:30.067322+0000","last_undegraded":"2024-04-26T16:20:30.067322+0000","last_fullsized":"2024-04-26T16:20:30.067322+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:16.148771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:16.148771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.303316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'4","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067132+0000","last_change":"2024-04-26T16:20:26.899539+0000","last_active":"2024-04-26T16:20:30.067132+0000","last_peered":"2024-04-26T16:20:30.067132+0000","last_clean":"2024-04-26T16:20:30.067132+0000","last_became_active":"2024-04-26T16:17:31.026037+0000","last_became_peered":"2024-04-26T16:17:31.026037+0000","last_unstale":"2024-04-26T16:20:30.067132+0000","last_undegraded":"2024-04-26T16:20:30.067132+0000","last_fullsized":"2024-04-26T16:20:30.067132+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'4","last_scrub_stamp":"2024-04-26T16:20:26.899299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.899299+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.728642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345321+0000","last_change":"2024-04-26T16:19:54.349951+0000","last_active":"2024-04-26T16:20:30.345321+0000","last_peered":"2024-04-26T16:20:30.345321+0000","last_clean":"2024-04-26T16:20:30.345321+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:20:30.345321+0000","last_undegraded":"2024-04-26T16:20:30.345321+0000","last_fullsized":"2024-04-26T16:20:30.345321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:54.349649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:54.349649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:55.254477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:33.855397+0000","last_change":"2024-04-26T16:20:33.855455+0000","last_active":"2024-04-26T16:20:33.855397+0000","last_peered":"2024-04-26T16:20:33.855397+0000","last_clean":"2024-04-26T16:20:33.855397+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:20:33.855397+0000","last_undegraded":"2024-04-26T16:20:33.855397+0000","last_fullsized":"2024-04-26T16:20:33.855397+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:33.855240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.855240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.904032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344460+0000","last_change":"2024-04-26T16:20:27.565849+0000","last_active":"2024-04-26T16:20:30.344460+0000","last_peered":"2024-04-26T16:20:30.344460+0000","last_clean":"2024-04-26T16:20:30.344460+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:20:30.344460+0000","last_undegraded":"2024-04-26T16:20:30.344460+0000","last_fullsized":"2024-04-26T16:20:30.344460+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:27.565529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:27.565529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.357713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222840+0000","last_change":"2024-04-26T16:20:14.278241+0000","last_active":"2024-04-26T16:20:30.222840+0000","last_peered":"2024-04-26T16:20:30.222840+0000","last_clean":"2024-04-26T16:20:30.222840+0000","last_became_active":"2024-04-26T16:17:31.033294+0000","last_became_peered":"2024-04-26T16:17:31.033294+0000","last_unstale":"2024-04-26T16:20:30.222840+0000","last_undegraded":"2024-04-26T16:20:30.222840+0000","last_fullsized":"2024-04-26T16:20:30.222840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:14.278005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.278005+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.458748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406676+0000","last_change":"2024-04-26T16:19:08.931731+0000","last_active":"2024-04-26T16:20:30.406676+0000","last_peered":"2024-04-26T16:20:30.406676+0000","last_clean":"2024-04-26T16:20:30.406676+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:20:30.406676+0000","last_undegraded":"2024-04-26T16:20:30.406676+0000","last_fullsized":"2024-04-26T16:20:30.406676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:08.931368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:08.931368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.434043+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017458+0000","last_change":"2024-04-26T16:20:13.890462+0000","last_active":"2024-04-26T16:20:30.017458+0000","last_peered":"2024-04-26T16:20:30.017458+0000","last_clean":"2024-04-26T16:20:30.017458+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:20:30.017458+0000","last_undegraded":"2024-04-26T16:20:30.017458+0000","last_fullsized":"2024-04-26T16:20:30.017458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:20:13.890111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.890111+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.062559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"86'11","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224305+0000","last_change":"2024-04-26T16:19:45.378650+0000","last_active":"2024-04-26T16:20:30.224305+0000","last_peered":"2024-04-26T16:20:30.224305+0000","last_clean":"2024-04-26T16:20:30.224305+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:20:30.224305+0000","last_undegraded":"2024-04-26T16:20:30.224305+0000","last_fullsized":"2024-04-26T16:20:30.224305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:45.378259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:45.378259+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:54.005675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344161+0000","last_change":"2024-04-26T16:20:14.406999+0000","last_active":"2024-04-26T16:20:30.344161+0000","last_peered":"2024-04-26T16:20:30.344161+0000","last_clean":"2024-04-26T16:20:30.344161+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:20:30.344161+0000","last_undegraded":"2024-04-26T16:20:30.344161+0000","last_fullsized":"2024-04-26T16:20:30.344161+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.406805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.406805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.652307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:32.257176+0000","last_change":"2024-04-26T16:20:32.257239+0000","last_active":"2024-04-26T16:20:32.257176+0000","last_peered":"2024-04-26T16:20:32.257176+0000","last_clean":"2024-04-26T16:20:32.257176+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:20:32.257176+0000","last_undegraded":"2024-04-26T16:20:32.257176+0000","last_fullsized":"2024-04-26T16:20:32.257176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:32.256998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:32.256998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.610172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016654+0000","last_change":"2024-04-26T16:19:15.229718+0000","last_active":"2024-04-26T16:20:30.016654+0000","last_peered":"2024-04-26T16:20:30.016654+0000","last_clean":"2024-04-26T16:20:30.016654+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:20:30.016654+0000","last_undegraded":"2024-04-26T16:20:30.016654+0000","last_fullsized":"2024-04-26T16:20:30.016654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:15.229495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:15.229495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.459154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"86'16","reported_seq":159,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404688+0000","last_change":"2024-04-26T16:20:21.529761+0000","last_active":"2024-04-26T16:20:30.404688+0000","last_peered":"2024-04-26T16:20:30.404688+0000","last_clean":"2024-04-26T16:20:30.404688+0000","last_became_active":"2024-04-26T16:17:31.040723+0000","last_became_peered":"2024-04-26T16:17:31.040723+0000","last_unstale":"2024-04-26T16:20:30.404688+0000","last_undegraded":"2024-04-26T16:20:30.404688+0000","last_fullsized":"2024-04-26T16:20:30.404688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'16","last_scrub_stamp":"2024-04-26T16:20:21.529513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.529513+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.582915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016766+0000","last_change":"2024-04-26T16:19:59.129440+0000","last_active":"2024-04-26T16:20:30.016766+0000","last_peered":"2024-04-26T16:20:30.016766+0000","last_clean":"2024-04-26T16:20:30.016766+0000","last_became_active":"2024-04-26T16:17:32.046437+0000","last_became_peered":"2024-04-26T16:17:32.046437+0000","last_unstale":"2024-04-26T16:20:30.016766+0000","last_undegraded":"2024-04-26T16:20:30.016766+0000","last_fullsized":"2024-04-26T16:20:30.016766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:19:59.129167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.129167+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.401811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"86'11","reported_seq":145,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017679+0000","last_change":"2024-04-26T16:19:50.407803+0000","last_active":"2024-04-26T16:20:30.017679+0000","last_peered":"2024-04-26T16:20:30.017679+0000","last_clean":"2024-04-26T16:20:30.017679+0000","last_became_active":"2024-04-26T16:17:33.069020+0000","last_became_peered":"2024-04-26T16:17:33.069020+0000","last_unstale":"2024-04-26T16:20:30.017679+0000","last_undegraded":"2024-04-26T16:20:30.017679+0000","last_fullsized":"2024-04-26T16:20:30.017679+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:50.407700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:50.407700+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:57.863330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.407147+0000","last_change":"2024-04-26T16:20:09.046364+0000","last_active":"2024-04-26T16:20:30.407147+0000","last_peered":"2024-04-26T16:20:30.407147+0000","last_clean":"2024-04-26T16:20:30.407147+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:20:30.407147+0000","last_undegraded":"2024-04-26T16:20:30.407147+0000","last_fullsized":"2024-04-26T16:20:30.407147+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.045989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.045989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.237368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403829+0000","last_change":"2024-04-26T16:20:13.525422+0000","last_active":"2024-04-26T16:20:30.403829+0000","last_peered":"2024-04-26T16:20:30.403829+0000","last_clean":"2024-04-26T16:20:30.403829+0000","last_became_active":"2024-04-26T16:17:29.022734+0000","last_became_peered":"2024-04-26T16:17:29.022734+0000","last_unstale":"2024-04-26T16:20:30.403829+0000","last_undegraded":"2024-04-26T16:20:30.403829+0000","last_fullsized":"2024-04-26T16:20:30.403829+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:13.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.525151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.383572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'27","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015122+0000","last_change":"2024-04-26T16:20:19.982781+0000","last_active":"2024-04-26T16:20:30.015122+0000","last_peered":"2024-04-26T16:20:30.015122+0000","last_clean":"2024-04-26T16:20:30.015122+0000","last_became_active":"2024-04-26T16:17:31.030904+0000","last_became_peered":"2024-04-26T16:17:31.030904+0000","last_unstale":"2024-04-26T16:20:30.015122+0000","last_undegraded":"2024-04-26T16:20:30.015122+0000","last_fullsized":"2024-04-26T16:20:30.015122+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'27","last_scrub_stamp":"2024-04-26T16:20:19.982389+0000","last_deep_scrub":"103'27","last_deep_scrub_stamp":"2024-04-26T16:18:40.947097+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.982389+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.646180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016496+0000","last_change":"2024-04-26T16:20:24.200758+0000","last_active":"2024-04-26T16:20:30.016496+0000","last_peered":"2024-04-26T16:20:30.016496+0000","last_clean":"2024-04-26T16:20:30.016496+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:20:30.016496+0000","last_undegraded":"2024-04-26T16:20:30.016496+0000","last_fullsized":"2024-04-26T16:20:30.016496+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:24.200448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.200448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.909147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224198+0000","last_change":"2024-04-26T16:20:16.270543+0000","last_active":"2024-04-26T16:20:30.224198+0000","last_peered":"2024-04-26T16:20:30.224198+0000","last_clean":"2024-04-26T16:20:30.224198+0000","last_became_active":"2024-04-26T16:17:35.670585+0000","last_became_peered":"2024-04-26T16:17:35.670585+0000","last_unstale":"2024-04-26T16:20:30.224198+0000","last_undegraded":"2024-04-26T16:20:30.224198+0000","last_fullsized":"2024-04-26T16:20:30.224198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:16.270321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:16.270321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.978435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":140,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017656+0000","last_change":"2024-04-26T16:19:43.114960+0000","last_active":"2024-04-26T16:20:30.017656+0000","last_peered":"2024-04-26T16:20:30.017656+0000","last_clean":"2024-04-26T16:20:30.017656+0000","last_became_active":"2024-04-26T16:17:31.023021+0000","last_became_peered":"2024-04-26T16:17:31.023021+0000","last_unstale":"2024-04-26T16:20:30.017656+0000","last_undegraded":"2024-04-26T16:20:30.017656+0000","last_fullsized":"2024-04-26T16:20:30.017656+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_clean_scrub_stamp":"2024-04-26T16:19:43.114691+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.523228+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344415+0000","last_change":"2024-04-26T16:19:52.393976+0000","last_active":"2024-04-26T16:20:30.344415+0000","last_peered":"2024-04-26T16:20:30.344415+0000","last_clean":"2024-04-26T16:20:30.344415+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:20:30.344415+0000","last_undegraded":"2024-04-26T16:20:30.344415+0000","last_fullsized":"2024-04-26T16:20:30.344415+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:52.393754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:19:52.393754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.078807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344996+0000","last_change":"2024-04-26T16:19:59.359204+0000","last_active":"2024-04-26T16:20:30.344996+0000","last_peered":"2024-04-26T16:20:30.344996+0000","last_clean":"2024-04-26T16:20:30.344996+0000","last_became_active":"2024-04-26T16:17:33.066780+0000","last_became_peered":"2024-04-26T16:17:33.066780+0000","last_unstale":"2024-04-26T16:20:30.344996+0000","last_undegraded":"2024-04-26T16:20:30.344996+0000","last_fullsized":"2024-04-26T16:20:30.344996+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:59.358789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.358789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.479227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":83,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405681+0000","last_change":"2024-04-26T16:19:28.707694+0000","last_active":"2024-04-26T16:20:30.405681+0000","last_peered":"2024-04-26T16:20:30.405681+0000","last_clean":"2024-04-26T16:20:30.405681+0000","last_became_active":"2024-04-26T16:17:35.087291+0000","last_became_peered":"2024-04-26T16:17:35.087291+0000","last_unstale":"2024-04-26T16:20:30.405681+0000","last_undegraded":"2024-04-26T16:20:30.405681+0000","last_fullsized":"2024-04-26T16:20:30.405681+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:28.707475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:28.707475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:50.575106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'10","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405790+0000","last_change":"2024-04-26T16:20:04.994840+0000","last_active":"2024-04-26T16:20:30.405790+0000","last_peered":"2024-04-26T16:20:30.405790+0000","last_clean":"2024-04-26T16:20:30.405790+0000","last_became_active":"2024-04-26T16:17:31.048785+0000","last_became_peered":"2024-04-26T16:17:31.048785+0000","last_unstale":"2024-04-26T16:20:30.405790+0000","last_undegraded":"2024-04-26T16:20:30.405790+0000","last_fullsized":"2024-04-26T16:20:30.405790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:04.994521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.994521+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:33.785270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403944+0000","last_change":"2024-04-26T16:20:06.715686+0000","last_active":"2024-04-26T16:20:30.403944+0000","last_peered":"2024-04-26T16:20:30.403944+0000","last_clean":"2024-04-26T16:20:30.403944+0000","last_became_active":"2024-04-26T16:17:29.013835+0000","last_became_peered":"2024-04-26T16:17:29.013835+0000","last_unstale":"2024-04-26T16:20:30.403944+0000","last_undegraded":"2024-04-26T16:20:30.403944+0000","last_fullsized":"2024-04-26T16:20:30.403944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:06.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:06.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.617561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":381,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222884+0000","last_change":"2024-04-26T16:19:42.393827+0000","last_active":"2024-04-26T16:20:30.222884+0000","last_peered":"2024-04-26T16:20:30.222884+0000","last_clean":"2024-04-26T16:20:30.222884+0000","last_became_active":"2024-04-26T16:16:50.925983+0000","last_became_peered":"2024-04-26T16:16:50.925983+0000","last_unstale":"2024-04-26T16:20:30.222884+0000","last_undegraded":"2024-04-26T16:20:30.222884+0000","last_fullsized":"2024-04-26T16:20:30.222884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:19:42.393587+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:19:42.393587+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:05.978678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":86,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224853+0000","last_change":"2024-04-26T16:19:30.568668+0000","last_active":"2024-04-26T16:20:30.224853+0000","last_peered":"2024-04-26T16:20:30.224853+0000","last_clean":"2024-04-26T16:20:30.224853+0000","last_became_active":"2024-04-26T16:17:33.061816+0000","last_became_peered":"2024-04-26T16:17:33.061816+0000","last_unstale":"2024-04-26T16:20:30.224853+0000","last_undegraded":"2024-04-26T16:20:30.224853+0000","last_fullsized":"2024-04-26T16:20:30.224853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:30.568415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:30.568415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.403532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016278+0000","last_change":"2024-04-26T16:19:24.155748+0000","last_active":"2024-04-26T16:20:30.016278+0000","last_peered":"2024-04-26T16:20:30.016278+0000","last_clean":"2024-04-26T16:20:30.016278+0000","last_became_active":"2024-04-26T16:17:35.090592+0000","last_became_peered":"2024-04-26T16:17:35.090592+0000","last_unstale":"2024-04-26T16:20:30.016278+0000","last_undegraded":"2024-04-26T16:20:30.016278+0000","last_fullsized":"2024-04-26T16:20:30.016278+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:24.155543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:24.155543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.047837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"112'21","reported_seq":126,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014384+0000","last_change":"2024-04-26T16:19:47.424498+0000","last_active":"2024-04-26T16:20:30.014384+0000","last_peered":"2024-04-26T16:20:30.014384+0000","last_clean":"2024-04-26T16:20:30.014384+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:20:30.014384+0000","last_undegraded":"2024-04-26T16:20:30.014384+0000","last_fullsized":"2024-04-26T16:20:30.014384+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'21","last_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:19:47.424244+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:56.347516+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.226248+0000","last_change":"2024-04-26T16:20:30.226306+0000","last_active":"2024-04-26T16:20:30.226248+0000","last_peered":"2024-04-26T16:20:30.226248+0000","last_clean":"2024-04-26T16:20:30.226248+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:20:30.226248+0000","last_undegraded":"2024-04-26T16:20:30.226248+0000","last_fullsized":"2024-04-26T16:20:30.226248+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:30.226091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:30.226091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.774441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:35.552647+0000","last_change":"2024-04-26T16:20:35.552710+0000","last_active":"2024-04-26T16:20:35.552647+0000","last_peered":"2024-04-26T16:20:35.552647+0000","last_clean":"2024-04-26T16:20:35.552647+0000","last_became_active":"2024-04-26T16:17:33.046107+0000","last_became_peered":"2024-04-26T16:17:33.046107+0000","last_unstale":"2024-04-26T16:20:35.552647+0000","last_undegraded":"2024-04-26T16:20:35.552647+0000","last_fullsized":"2024-04-26T16:20:35.552647+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.552498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.552498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.056973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015504+0000","last_change":"2024-04-26T16:20:15.021647+0000","last_active":"2024-04-26T16:20:30.015504+0000","last_peered":"2024-04-26T16:20:30.015504+0000","last_clean":"2024-04-26T16:20:30.015504+0000","last_became_active":"2024-04-26T16:17:35.673254+0000","last_became_peered":"2024-04-26T16:17:35.673254+0000","last_unstale":"2024-04-26T16:20:30.015504+0000","last_undegraded":"2024-04-26T16:20:30.015504+0000","last_fullsized":"2024-04-26T16:20:30.015504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.021515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.021515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.753340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224480+0000","last_change":"2024-04-26T16:20:24.308841+0000","last_active":"2024-04-26T16:20:30.224480+0000","last_peered":"2024-04-26T16:20:30.224480+0000","last_clean":"2024-04-26T16:20:30.224480+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:20:30.224480+0000","last_undegraded":"2024-04-26T16:20:30.224480+0000","last_fullsized":"2024-04-26T16:20:30.224480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:24.308595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.308595+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:53.172263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"107'40","reported_seq":198,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016874+0000","last_change":"2024-04-26T16:20:25.051639+0000","last_active":"2024-04-26T16:20:30.016874+0000","last_peered":"2024-04-26T16:20:30.016874+0000","last_clean":"2024-04-26T16:20:30.016874+0000","last_became_active":"2024-04-26T16:17:31.024426+0000","last_became_peered":"2024-04-26T16:17:31.024426+0000","last_unstale":"2024-04-26T16:20:30.016874+0000","last_undegraded":"2024-04-26T16:20:30.016874+0000","last_fullsized":"2024-04-26T16:20:30.016874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'40","last_scrub_stamp":"2024-04-26T16:20:25.051382+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:18:56.138611+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.051382+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.857786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067813+0000","last_change":"2024-04-26T16:20:18.962156+0000","last_active":"2024-04-26T16:20:30.067813+0000","last_peered":"2024-04-26T16:20:30.067813+0000","last_clean":"2024-04-26T16:20:30.067813+0000","last_became_active":"2024-04-26T16:17:33.057789+0000","last_became_peered":"2024-04-26T16:17:33.057789+0000","last_unstale":"2024-04-26T16:20:30.067813+0000","last_undegraded":"2024-04-26T16:20:30.067813+0000","last_fullsized":"2024-04-26T16:20:30.067813+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.961959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:53.089835+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.961959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.337483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014758+0000","last_change":"2024-04-26T16:19:57.996481+0000","last_active":"2024-04-26T16:20:30.014758+0000","last_peered":"2024-04-26T16:20:30.014758+0000","last_clean":"2024-04-26T16:20:30.014758+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:20:30.014758+0000","last_undegraded":"2024-04-26T16:20:30.014758+0000","last_fullsized":"2024-04-26T16:20:30.014758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:57.996170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:19:57.996170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.461563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015031+0000","last_change":"2024-04-26T16:20:18.047229+0000","last_active":"2024-04-26T16:20:30.015031+0000","last_peered":"2024-04-26T16:20:30.015031+0000","last_clean":"2024-04-26T16:20:30.015031+0000","last_became_active":"2024-04-26T16:17:29.004853+0000","last_became_peered":"2024-04-26T16:17:29.004853+0000","last_unstale":"2024-04-26T16:20:30.015031+0000","last_undegraded":"2024-04-26T16:20:30.015031+0000","last_fullsized":"2024-04-26T16:20:30.015031+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.047049+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.615066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"86'13","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.407126+0000","last_change":"2024-04-26T16:19:19.940285+0000","last_active":"2024-04-26T16:20:30.407126+0000","last_peered":"2024-04-26T16:20:30.407126+0000","last_clean":"2024-04-26T16:20:30.407126+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:20:30.407126+0000","last_undegraded":"2024-04-26T16:20:30.407126+0000","last_fullsized":"2024-04-26T16:20:30.407126+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-04-26T16:19:19.939903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:19.939903+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:42.676901+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":273,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:34.807977+0000","last_change":"2024-04-26T16:20:11.073651+0000","last_active":"2024-04-26T16:20:34.807977+0000","last_peered":"2024-04-26T16:20:34.807977+0000","last_clean":"2024-04-26T16:20:34.807977+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:20:34.807977+0000","last_undegraded":"2024-04-26T16:20:34.807977+0000","last_fullsized":"2024-04-26T16:20:34.807977+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:20:11.073395+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:19:04.959471+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.073395+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.691968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":167,"num_read_kb":162,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014612+0000","last_change":"2024-04-26T16:20:21.947248+0000","last_active":"2024-04-26T16:20:30.014612+0000","last_peered":"2024-04-26T16:20:30.014612+0000","last_clean":"2024-04-26T16:20:30.014612+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:20:30.014612+0000","last_undegraded":"2024-04-26T16:20:30.014612+0000","last_fullsized":"2024-04-26T16:20:30.014612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.946995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.814368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223266+0000","last_change":"2024-04-26T16:20:17.304956+0000","last_active":"2024-04-26T16:20:30.223266+0000","last_peered":"2024-04-26T16:20:30.223266+0000","last_clean":"2024-04-26T16:20:30.223266+0000","last_became_active":"2024-04-26T16:17:35.670672+0000","last_became_peered":"2024-04-26T16:17:35.670672+0000","last_unstale":"2024-04-26T16:20:30.223266+0000","last_undegraded":"2024-04-26T16:20:30.223266+0000","last_fullsized":"2024-04-26T16:20:30.223266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.304722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.304722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.500725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.068570+0000","last_change":"2024-04-26T16:19:58.917701+0000","last_active":"2024-04-26T16:20:30.068570+0000","last_peered":"2024-04-26T16:20:30.068570+0000","last_clean":"2024-04-26T16:20:30.068570+0000","last_became_active":"2024-04-26T16:17:29.006985+0000","last_became_peered":"2024-04-26T16:17:29.006985+0000","last_unstale":"2024-04-26T16:20:30.068570+0000","last_undegraded":"2024-04-26T16:20:30.068570+0000","last_fullsized":"2024-04-26T16:20:30.068570+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.917484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:16.115592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"107'18","reported_seq":122,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014975+0000","last_change":"2024-04-26T16:20:13.290879+0000","last_active":"2024-04-26T16:20:30.014975+0000","last_peered":"2024-04-26T16:20:30.014975+0000","last_clean":"2024-04-26T16:20:30.014975+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:20:30.014975+0000","last_undegraded":"2024-04-26T16:20:30.014975+0000","last_fullsized":"2024-04-26T16:20:30.014975+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'18","last_scrub_stamp":"2024-04-26T16:20:13.290621+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:19:05.704281+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.290621+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.892369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015063+0000","last_change":"2024-04-26T16:20:06.968281+0000","last_active":"2024-04-26T16:20:30.015063+0000","last_peered":"2024-04-26T16:20:30.015063+0000","last_clean":"2024-04-26T16:20:30.015063+0000","last_became_active":"2024-04-26T16:17:32.027388+0000","last_became_peered":"2024-04-26T16:17:32.027388+0000","last_unstale":"2024-04-26T16:20:30.015063+0000","last_undegraded":"2024-04-26T16:20:30.015063+0000","last_fullsized":"2024-04-26T16:20:30.015063+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:06.968062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:20:06.968062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.035273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"86'11","reported_seq":134,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405062+0000","last_change":"2024-04-26T16:19:26.004779+0000","last_active":"2024-04-26T16:20:30.405062+0000","last_peered":"2024-04-26T16:20:30.405062+0000","last_clean":"2024-04-26T16:20:30.405062+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:20:30.405062+0000","last_undegraded":"2024-04-26T16:20:30.405062+0000","last_fullsized":"2024-04-26T16:20:30.405062+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:26.004517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:26.004517+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:36.533302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406445+0000","last_change":"2024-04-26T16:20:10.026900+0000","last_active":"2024-04-26T16:20:30.406445+0000","last_peered":"2024-04-26T16:20:30.406445+0000","last_clean":"2024-04-26T16:20:30.406445+0000","last_became_active":"2024-04-26T16:17:35.673130+0000","last_became_peered":"2024-04-26T16:17:35.673130+0000","last_unstale":"2024-04-26T16:20:30.406445+0000","last_undegraded":"2024-04-26T16:20:30.406445+0000","last_fullsized":"2024-04-26T16:20:30.406445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.026659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.026659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:39.688645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405648+0000","last_change":"2024-04-26T16:20:28.439735+0000","last_active":"2024-04-26T16:20:30.405648+0000","last_peered":"2024-04-26T16:20:30.405648+0000","last_clean":"2024-04-26T16:20:30.405648+0000","last_became_active":"2024-04-26T16:17:33.068923+0000","last_became_peered":"2024-04-26T16:17:33.068923+0000","last_unstale":"2024-04-26T16:20:30.405648+0000","last_undegraded":"2024-04-26T16:20:30.405648+0000","last_fullsized":"2024-04-26T16:20:30.405648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.439524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.246043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223040+0000","last_change":"2024-04-26T16:20:10.311712+0000","last_active":"2024-04-26T16:20:30.223040+0000","last_peered":"2024-04-26T16:20:30.223040+0000","last_clean":"2024-04-26T16:20:30.223040+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:20:30.223040+0000","last_undegraded":"2024-04-26T16:20:30.223040+0000","last_fullsized":"2024-04-26T16:20:30.223040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:10.311472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.311472+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.794801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"86'12","reported_seq":123,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:33.858572+0000","last_change":"2024-04-26T16:20:33.858656+0000","last_active":"2024-04-26T16:20:33.858572+0000","last_peered":"2024-04-26T16:20:33.858572+0000","last_clean":"2024-04-26T16:20:33.858572+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:20:33.858572+0000","last_undegraded":"2024-04-26T16:20:33.858572+0000","last_fullsized":"2024-04-26T16:20:33.858572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:33.858421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.858421+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.983533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.404370+0000","last_change":"2024-04-26T16:20:23.546630+0000","last_active":"2024-04-26T16:20:30.404370+0000","last_peered":"2024-04-26T16:20:30.404370+0000","last_clean":"2024-04-26T16:20:30.404370+0000","last_became_active":"2024-04-26T16:17:35.077832+0000","last_became_peered":"2024-04-26T16:17:35.077832+0000","last_unstale":"2024-04-26T16:20:30.404370+0000","last_undegraded":"2024-04-26T16:20:30.404370+0000","last_fullsized":"2024-04-26T16:20:30.404370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.546406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.546406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:48.610382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"115'12","reported_seq":145,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016959+0000","last_change":"2024-04-26T16:20:25.975628+0000","last_active":"2024-04-26T16:20:30.016959+0000","last_peered":"2024-04-26T16:20:30.016959+0000","last_clean":"2024-04-26T16:20:30.016959+0000","last_became_active":"2024-04-26T16:17:33.070130+0000","last_became_peered":"2024-04-26T16:17:33.070130+0000","last_unstale":"2024-04-26T16:20:30.016959+0000","last_undegraded":"2024-04-26T16:20:30.016959+0000","last_fullsized":"2024-04-26T16:20:30.016959+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.975399+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.920318+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:35.047078+0000","last_change":"2024-04-26T16:20:35.047323+0000","last_active":"2024-04-26T16:20:35.047078+0000","last_peered":"2024-04-26T16:20:35.047078+0000","last_clean":"2024-04-26T16:20:35.047078+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:20:35.047078+0000","last_undegraded":"2024-04-26T16:20:35.047078+0000","last_fullsized":"2024-04-26T16:20:35.047078+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.046847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.046847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.436102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"86'15","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.405173+0000","last_change":"2024-04-26T16:18:55.030748+0000","last_active":"2024-04-26T16:20:30.405173+0000","last_peered":"2024-04-26T16:20:30.405173+0000","last_clean":"2024-04-26T16:20:30.405173+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:20:30.405173+0000","last_undegraded":"2024-04-26T16:20:30.405173+0000","last_fullsized":"2024-04-26T16:20:30.405173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:18:55.030507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:18:55.030507+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.017118+0000","last_change":"2024-04-26T16:20:14.907782+0000","last_active":"2024-04-26T16:20:30.017118+0000","last_peered":"2024-04-26T16:20:30.017118+0000","last_clean":"2024-04-26T16:20:30.017118+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:20:30.017118+0000","last_undegraded":"2024-04-26T16:20:30.017118+0000","last_fullsized":"2024-04-26T16:20:30.017118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.907547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.907547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.001180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"86'11","reported_seq":142,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344818+0000","last_change":"2024-04-26T16:20:03.405476+0000","last_active":"2024-04-26T16:20:30.344818+0000","last_peered":"2024-04-26T16:20:30.344818+0000","last_clean":"2024-04-26T16:20:30.344818+0000","last_became_active":"2024-04-26T16:17:33.066299+0000","last_became_peered":"2024-04-26T16:17:33.066299+0000","last_unstale":"2024-04-26T16:20:30.344818+0000","last_undegraded":"2024-04-26T16:20:30.344818+0000","last_fullsized":"2024-04-26T16:20:30.344818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:03.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.405246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.150208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016445+0000","last_change":"2024-04-26T16:20:29.023874+0000","last_active":"2024-04-26T16:20:30.016445+0000","last_peered":"2024-04-26T16:20:30.016445+0000","last_clean":"2024-04-26T16:20:30.016445+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:20:30.016445+0000","last_undegraded":"2024-04-26T16:20:30.016445+0000","last_fullsized":"2024-04-26T16:20:30.016445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:29.023717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:29.023717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.818594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"86'9","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.406172+0000","last_change":"2024-04-26T16:19:20.985067+0000","last_active":"2024-04-26T16:20:30.406172+0000","last_peered":"2024-04-26T16:20:30.406172+0000","last_clean":"2024-04-26T16:20:30.406172+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:20:30.406172+0000","last_undegraded":"2024-04-26T16:20:30.406172+0000","last_fullsized":"2024-04-26T16:20:30.406172+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:20.984815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.984815+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:50.089853+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345537+0000","last_change":"2024-04-26T16:20:21.525478+0000","last_active":"2024-04-26T16:20:30.345537+0000","last_peered":"2024-04-26T16:20:30.345537+0000","last_clean":"2024-04-26T16:20:30.345537+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:20:30.345537+0000","last_undegraded":"2024-04-26T16:20:30.345537+0000","last_fullsized":"2024-04-26T16:20:30.345537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.525230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.078478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.016419+0000","last_change":"2024-04-26T16:20:23.025108+0000","last_active":"2024-04-26T16:20:30.016419+0000","last_peered":"2024-04-26T16:20:30.016419+0000","last_clean":"2024-04-26T16:20:30.016419+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:20:30.016419+0000","last_undegraded":"2024-04-26T16:20:30.016419+0000","last_fullsized":"2024-04-26T16:20:30.016419+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.024885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:01.215041+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.024885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.206757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:36.297411+0000","last_change":"2024-04-26T16:20:36.297478+0000","last_active":"2024-04-26T16:20:36.297411+0000","last_peered":"2024-04-26T16:20:36.297411+0000","last_clean":"2024-04-26T16:20:36.297411+0000","last_became_active":"2024-04-26T16:17:29.026793+0000","last_became_peered":"2024-04-26T16:17:29.026793+0000","last_unstale":"2024-04-26T16:20:36.297411+0000","last_undegraded":"2024-04-26T16:20:36.297411+0000","last_fullsized":"2024-04-26T16:20:36.297411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:20:36.297212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.276670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"86'19","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.067516+0000","last_change":"2024-04-26T16:20:11.904568+0000","last_active":"2024-04-26T16:20:30.067516+0000","last_peered":"2024-04-26T16:20:30.067516+0000","last_clean":"2024-04-26T16:20:30.067516+0000","last_became_active":"2024-04-26T16:17:31.024436+0000","last_became_peered":"2024-04-26T16:17:31.024436+0000","last_unstale":"2024-04-26T16:20:30.067516+0000","last_undegraded":"2024-04-26T16:20:30.067516+0000","last_fullsized":"2024-04-26T16:20:30.067516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'19","last_scrub_stamp":"2024-04-26T16:20:11.904292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.904292+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:17.775021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.222613+0000","last_change":"2024-04-26T16:19:23.666736+0000","last_active":"2024-04-26T16:20:30.222613+0000","last_peered":"2024-04-26T16:20:30.222613+0000","last_clean":"2024-04-26T16:20:30.222613+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:20:30.222613+0000","last_undegraded":"2024-04-26T16:20:30.222613+0000","last_fullsized":"2024-04-26T16:20:30.222613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:23.666505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:23.666505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344485+0000","last_change":"2024-04-26T16:19:57.397306+0000","last_active":"2024-04-26T16:20:30.344485+0000","last_peered":"2024-04-26T16:20:30.344485+0000","last_clean":"2024-04-26T16:20:30.344485+0000","last_became_active":"2024-04-26T16:17:33.065671+0000","last_became_peered":"2024-04-26T16:17:33.065671+0000","last_unstale":"2024-04-26T16:20:30.344485+0000","last_undegraded":"2024-04-26T16:20:30.344485+0000","last_fullsized":"2024-04-26T16:20:30.344485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:57.396984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:57.396984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:00.083033+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345016+0000","last_change":"2024-04-26T16:20:05.437174+0000","last_active":"2024-04-26T16:20:30.345016+0000","last_peered":"2024-04-26T16:20:30.345016+0000","last_clean":"2024-04-26T16:20:30.345016+0000","last_became_active":"2024-04-26T16:17:29.017152+0000","last_became_peered":"2024-04-26T16:17:29.017152+0000","last_unstale":"2024-04-26T16:20:30.345016+0000","last_undegraded":"2024-04-26T16:20:30.345016+0000","last_fullsized":"2024-04-26T16:20:30.345016+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.436941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.436941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:07.446646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'17","reported_seq":126,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:35.289879+0000","last_change":"2024-04-26T16:20:35.289963+0000","last_active":"2024-04-26T16:20:35.289879+0000","last_peered":"2024-04-26T16:20:35.289879+0000","last_clean":"2024-04-26T16:20:35.289879+0000","last_became_active":"2024-04-26T16:17:31.030244+0000","last_became_peered":"2024-04-26T16:17:31.030244+0000","last_unstale":"2024-04-26T16:20:35.289879+0000","last_undegraded":"2024-04-26T16:20:35.289879+0000","last_fullsized":"2024-04-26T16:20:35.289879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-04-26T16:20:35.289727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.289727+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.877432+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.014136+0000","last_change":"2024-04-26T16:20:14.277898+0000","last_active":"2024-04-26T16:20:30.014136+0000","last_peered":"2024-04-26T16:20:30.014136+0000","last_clean":"2024-04-26T16:20:30.014136+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:20:30.014136+0000","last_undegraded":"2024-04-26T16:20:30.014136+0000","last_fullsized":"2024-04-26T16:20:30.014136+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:25.766764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.344537+0000","last_change":"2024-04-26T16:20:09.420985+0000","last_active":"2024-04-26T16:20:30.344537+0000","last_peered":"2024-04-26T16:20:30.344537+0000","last_clean":"2024-04-26T16:20:30.344537+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:20:30.344537+0000","last_undegraded":"2024-04-26T16:20:30.344537+0000","last_fullsized":"2024-04-26T16:20:30.344537+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.420768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.420768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.763275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.015646+0000","last_change":"2024-04-26T16:19:16.245606+0000","last_active":"2024-04-26T16:20:30.015646+0000","last_peered":"2024-04-26T16:20:30.015646+0000","last_clean":"2024-04-26T16:20:30.015646+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:20:30.015646+0000","last_undegraded":"2024-04-26T16:20:30.015646+0000","last_fullsized":"2024-04-26T16:20:30.015646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:16.245379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:16.245379+0000","last_clean_scrub_stamp":"2024-04-26T16:19:16.245379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:39.961824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'10","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:34.526262+0000","last_change":"2024-04-26T16:20:34.526337+0000","last_active":"2024-04-26T16:20:34.526262+0000","last_peered":"2024-04-26T16:20:34.526262+0000","last_clean":"2024-04-26T16:20:34.526262+0000","last_became_active":"2024-04-26T16:17:31.040215+0000","last_became_peered":"2024-04-26T16:17:31.040215+0000","last_unstale":"2024-04-26T16:20:34.526262+0000","last_undegraded":"2024-04-26T16:20:34.526262+0000","last_fullsized":"2024-04-26T16:20:34.526262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:34.526105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:34.526105+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.517534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.403515+0000","last_change":"2024-04-26T16:19:53.748315+0000","last_active":"2024-04-26T16:20:30.403515+0000","last_peered":"2024-04-26T16:20:30.403515+0000","last_clean":"2024-04-26T16:20:30.403515+0000","last_became_active":"2024-04-26T16:17:35.669627+0000","last_became_peered":"2024-04-26T16:17:35.669627+0000","last_unstale":"2024-04-26T16:20:30.403515+0000","last_undegraded":"2024-04-26T16:20:30.403515+0000","last_fullsized":"2024-04-26T16:20:30.403515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:53.748063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:19:53.748063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:55.439580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":102,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.345586+0000","last_change":"2024-04-26T16:20:18.397841+0000","last_active":"2024-04-26T16:20:30.345586+0000","last_peered":"2024-04-26T16:20:30.345586+0000","last_clean":"2024-04-26T16:20:30.345586+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:20:30.345586+0000","last_undegraded":"2024-04-26T16:20:30.345586+0000","last_fullsized":"2024-04-26T16:20:30.345586+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.397696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:50.542517+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.397696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.381027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"86'15","reported_seq":123,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.224801+0000","last_change":"2024-04-26T16:20:28.238717+0000","last_active":"2024-04-26T16:20:30.224801+0000","last_peered":"2024-04-26T16:20:30.224801+0000","last_clean":"2024-04-26T16:20:30.224801+0000","last_became_active":"2024-04-26T16:17:31.033340+0000","last_became_peered":"2024-04-26T16:17:31.033340+0000","last_unstale":"2024-04-26T16:20:30.224801+0000","last_undegraded":"2024-04-26T16:20:30.224801+0000","last_fullsized":"2024-04-26T16:20:30.224801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:28.238415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.238415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.888574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"109'12","reported_seq":135,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-04-26T16:20:30.223287+0000","last_change":"2024-04-26T16:19:17.690652+0000","last_active":"2024-04-26T16:20:30.223287+0000","last_peered":"2024-04-26T16:20:30.223287+0000","last_clean":"2024-04-26T16:20:30.223287+0000","last_became_active":"2024-04-26T16:17:33.070525+0000","last_became_peered":"2024-04-26T16:17:33.070525+0000","last_unstale":"2024-04-26T16:20:30.223287+0000","last_undegraded":"2024-04-26T16:20:30.223287+0000","last_fullsized":"2024-04-26T16:20:30.223287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_clean_scrub_stamp":"2024-04-26T16:19:17.690568+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:44.618043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":167,"num_read_kb":162,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":436,"ondisk_log_size":436,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743472,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28420,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716732,"statfs":{"total":95995035648,"available":95965933568,"internally_reserved":0,"allocated":1437696,"data_stored":1010315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135930,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28384,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716768,"statfs":{"total":95995035648,"available":95965970432,"internally_reserved":0,"allocated":1400832,"data_stored":1007544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31988,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713164,"statfs":{"total":95995035648,"available":95962279936,"internally_reserved":0,"allocated":831488,"data_stored":418513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36728,"kb_used_data":1392,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708424,"statfs":{"total":95995035648,"available":95957426176,"internally_reserved":0,"allocated":1425408,"data_stored":1009628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":417669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084376,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":417576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855438,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":418557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462980,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":417591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:20:41.559 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T16:20:41.559 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T16:20:41.559 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-26T16:20:41.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-26T16:20:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:41 smithi190 bash[23868]: cluster 2024-04-26T16:20:40.009319+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:20:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:41 smithi190 bash[23868]: cluster 2024-04-26T16:20:40.009319+0000 osd.1 (osd.1) 81 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:20:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:41 smithi190 bash[23868]: cluster 2024-04-26T16:20:40.011708+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:20:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:41 smithi190 bash[23868]: cluster 2024-04-26T16:20:40.011708+0000 osd.1 (osd.1) 82 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:20:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:41 smithi190 bash[23868]: audit 2024-04-26T16:20:40.699716+0000 mon.c (mon.2) 71 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:41 smithi190 bash[23868]: audit 2024-04-26T16:20:40.699716+0000 mon.c (mon.2) 71 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:42.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[29234]: audit 2024-04-26T16:20:40.642955+0000 mgr.y (mgr.24401) 118 : audit [DBG] from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[29234]: audit 2024-04-26T16:20:40.642955+0000 mgr.y (mgr.24401) 118 : audit [DBG] from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.400971+0000 mgr.y (mgr.24401) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.400971+0000 mgr.y (mgr.24401) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[21463]: audit 2024-04-26T16:20:40.642955+0000 mgr.y (mgr.24401) 118 : audit [DBG] from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[21463]: audit 2024-04-26T16:20:40.642955+0000 mgr.y (mgr.24401) 118 : audit [DBG] from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.400971+0000 mgr.y (mgr.24401) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:42.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:42 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.400971+0000 mgr.y (mgr.24401) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:42.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:42 smithi190 bash[23868]: audit 2024-04-26T16:20:40.642955+0000 mgr.y (mgr.24401) 118 : audit [DBG] from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:42.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:42 smithi190 bash[23868]: audit 2024-04-26T16:20:40.642955+0000 mgr.y (mgr.24401) 118 : audit [DBG] from='client.14685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:20:42.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:42 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.400971+0000 mgr.y (mgr.24401) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:42.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:42 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.400971+0000 mgr.y (mgr.24401) 119 : cluster [DBG] pgmap v89: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.523330+0000 osd.5 (osd.5) 83 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.523330+0000 osd.5 (osd.5) 83 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.526992+0000 osd.5 (osd.5) 84 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.526992+0000 osd.5 (osd.5) 84 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.758064+0000 osd.6 (osd.6) 45 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.758064+0000 osd.6 (osd.6) 45 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.758154+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:41.758154+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:42.104438+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-04-26T16:20:43.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[29234]: cluster 2024-04-26T16:20:42.104438+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.523330+0000 osd.5 (osd.5) 83 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.523330+0000 osd.5 (osd.5) 83 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.526992+0000 osd.5 (osd.5) 84 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.526992+0000 osd.5 (osd.5) 84 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.758064+0000 osd.6 (osd.6) 45 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.758064+0000 osd.6 (osd.6) 45 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.758154+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:41.758154+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:42.104438+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-04-26T16:20:43.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:43 smithi050 bash[21463]: cluster 2024-04-26T16:20:42.104438+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.523330+0000 osd.5 (osd.5) 83 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.523330+0000 osd.5 (osd.5) 83 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.526992+0000 osd.5 (osd.5) 84 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.526992+0000 osd.5 (osd.5) 84 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.758064+0000 osd.6 (osd.6) 45 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.758064+0000 osd.6 (osd.6) 45 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:43.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.758154+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:43.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:41.758154+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:43.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:42.104438+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-04-26T16:20:43.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:43 smithi190 bash[23868]: cluster 2024-04-26T16:20:42.104438+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2024-04-26T16:20:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.029276+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:20:44.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.029276+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.031624+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.031624+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.401735+0000 mgr.y (mgr.24401) 120 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.401735+0000 mgr.y (mgr.24401) 120 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.029276+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.029276+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.031624+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.031624+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.401735+0000 mgr.y (mgr.24401) 120 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:44.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.401735+0000 mgr.y (mgr.24401) 120 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:44 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.029276+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:20:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:44 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.029276+0000 osd.1 (osd.1) 83 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:20:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:44 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.031624+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:20:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:44 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.031624+0000 osd.1 (osd.1) 84 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:20:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:44 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.401735+0000 mgr.y (mgr.24401) 120 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:44 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.401735+0000 mgr.y (mgr.24401) 120 : cluster [DBG] pgmap v91: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:20:44.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:20:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.858460+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.858460+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.860646+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[21463]: cluster 2024-04-26T16:20:43.860646+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.858460+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.858460+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.860646+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:20:45.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:45 smithi050 bash[29234]: cluster 2024-04-26T16:20:43.860646+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:20:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:45 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.858460+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:20:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:45 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.858460+0000 osd.4 (osd.4) 71 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:20:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:45 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.860646+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:20:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:45 smithi190 bash[23868]: cluster 2024-04-26T16:20:43.860646+0000 osd.4 (osd.4) 72 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.942097+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.942097+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.944445+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.944445+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.973664+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.973664+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:20:46.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.975969+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:44.975969+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:45.402426+0000 mgr.y (mgr.24401) 121 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[21463]: cluster 2024-04-26T16:20:45.402426+0000 mgr.y (mgr.24401) 121 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.942097+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.942097+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.944445+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.944445+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.973664+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.973664+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:20:46.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.975969+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:20:46.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:44.975969+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:20:46.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:45.402426+0000 mgr.y (mgr.24401) 121 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-26T16:20:46.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:46 smithi050 bash[29234]: cluster 2024-04-26T16:20:45.402426+0000 mgr.y (mgr.24401) 121 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-26T16:20:46.392 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.942097+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.942097+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.944445+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.944445+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.973664+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.973664+0000 osd.1 (osd.1) 85 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.975969+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:44.975969+0000 osd.1 (osd.1) 86 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:45.402426+0000 mgr.y (mgr.24401) 121 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-26T16:20:46.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[23868]: cluster 2024-04-26T16:20:45.402426+0000 mgr.y (mgr.24401) 121 : cluster [DBG] pgmap v92: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-04-26T16:20:46.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:20:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[21463]: cluster 2024-04-26T16:20:46.046812+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[21463]: cluster 2024-04-26T16:20:46.046812+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[21463]: cluster 2024-04-26T16:20:46.049305+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[21463]: cluster 2024-04-26T16:20:46.049305+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[21463]: audit 2024-04-26T16:20:46.449459+0000 mgr.y (mgr.24401) 122 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[21463]: audit 2024-04-26T16:20:46.449459+0000 mgr.y (mgr.24401) 122 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[29234]: cluster 2024-04-26T16:20:46.046812+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[29234]: cluster 2024-04-26T16:20:46.046812+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:20:47.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[29234]: cluster 2024-04-26T16:20:46.049305+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:20:47.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[29234]: cluster 2024-04-26T16:20:46.049305+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:20:47.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[29234]: audit 2024-04-26T16:20:46.449459+0000 mgr.y (mgr.24401) 122 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:47.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:47 smithi050 bash[29234]: audit 2024-04-26T16:20:46.449459+0000 mgr.y (mgr.24401) 122 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:47 smithi190 bash[23868]: cluster 2024-04-26T16:20:46.046812+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:20:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:47 smithi190 bash[23868]: cluster 2024-04-26T16:20:46.046812+0000 osd.3 (osd.3) 81 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:20:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:47 smithi190 bash[23868]: cluster 2024-04-26T16:20:46.049305+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:20:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:47 smithi190 bash[23868]: cluster 2024-04-26T16:20:46.049305+0000 osd.3 (osd.3) 82 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:20:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:47 smithi190 bash[23868]: audit 2024-04-26T16:20:46.449459+0000 mgr.y (mgr.24401) 122 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:47 smithi190 bash[23868]: audit 2024-04-26T16:20:46.449459+0000 mgr.y (mgr.24401) 122 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.020264+0000 osd.1 (osd.1) 87 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.020264+0000 osd.1 (osd.1) 87 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.022607+0000 osd.1 (osd.1) 88 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.022607+0000 osd.1 (osd.1) 88 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.166595+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 deep-scrub starts 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.166595+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 deep-scrub starts 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.169224+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 deep-scrub ok 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.169224+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 deep-scrub ok 2024-04-26T16:20:48.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.304249+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.304249+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.306257+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.306257+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.403500+0000 mgr.y (mgr.24401) 123 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[29234]: cluster 2024-04-26T16:20:47.403500+0000 mgr.y (mgr.24401) 123 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.020264+0000 osd.1 (osd.1) 87 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.020264+0000 osd.1 (osd.1) 87 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.022607+0000 osd.1 (osd.1) 88 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.022607+0000 osd.1 (osd.1) 88 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.166595+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 deep-scrub starts 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.166595+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 deep-scrub starts 2024-04-26T16:20:48.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.169224+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 deep-scrub ok 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.169224+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 deep-scrub ok 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.304249+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.304249+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.306257+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.306257+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.403500+0000 mgr.y (mgr.24401) 123 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:48.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:48 smithi050 bash[21463]: cluster 2024-04-26T16:20:47.403500+0000 mgr.y (mgr.24401) 123 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.020264+0000 osd.1 (osd.1) 87 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:20:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.020264+0000 osd.1 (osd.1) 87 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:20:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.022607+0000 osd.1 (osd.1) 88 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:20:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.022607+0000 osd.1 (osd.1) 88 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:20:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.166595+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 deep-scrub starts 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.166595+0000 osd.7 (osd.7) 91 : cluster [DBG] 6.8 deep-scrub starts 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.169224+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 deep-scrub ok 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.169224+0000 osd.7 (osd.7) 92 : cluster [DBG] 6.8 deep-scrub ok 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.304249+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.304249+0000 osd.0 (osd.0) 79 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.306257+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.306257+0000 osd.0 (osd.0) 80 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.403500+0000 mgr.y (mgr.24401) 123 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:48 smithi190 bash[23868]: cluster 2024-04-26T16:20:47.403500+0000 mgr.y (mgr.24401) 123 : cluster [DBG] pgmap v93: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 -- 172.21.15.50:0/1639179301 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d41094c0 msgr2=0x7f03d41098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 --2- 172.21.15.50:0/1639179301 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d41094c0 0x7f03d41098a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f03c40098f0 tx=0x7f03c4009c00 comp rx=0 tx=0).stop 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 -- 172.21.15.50:0/1639179301 shutdown_connections 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 --2- 172.21.15.50:0/1639179301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03d4103e30 0x7f03d4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 --2- 172.21.15.50:0/1639179301 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d41094c0 0x7f03d41098a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 --2- 172.21.15.50:0/1639179301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103470 0x7f03d41038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 -- 172.21.15.50:0/1639179301 >> 172.21.15.50:0/1639179301 conn(0x7f03d407a580 msgr2=0x7f03d407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.921+0000 7f03d988a700 1 -- 172.21.15.50:0/1639179301 shutdown_connections 2024-04-26T16:20:49.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- 172.21.15.50:0/1639179301 wait complete. 2024-04-26T16:20:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 librados: starting msgr at 2024-04-26T16:20:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 librados: starting objecter 2024-04-26T16:20:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 Processor -- start 2024-04-26T16:20:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- start start 2024-04-26T16:20:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 librados: setting wanted keys 2024-04-26T16:20:49.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 librados: calling monclient init 2024-04-26T16:20:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 0x7f03d4199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 0x7f03d4199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03d41094c0 0x7f03d419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4112b00 con 0x7f03d4103e30 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f03d4112980 con 0x7f03d41094c0 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f03d4112c80 con 0x7f03d4103470 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d2ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 0x7f03d4199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 0x7f03d4199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 0x7f03d4199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48018/0 (socket says 172.21.15.50:48018) 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d2ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 0x7f03d4199200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38358/0 (socket says 172.21.15.50:38358) 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 -- 172.21.15.50:0/3437516564 learned_addr learned my addr 172.21.15.50:0/3437516564 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d37fe700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03d41094c0 0x7f03d419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 -- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 msgr2=0x7f03d4199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 0x7f03d4199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:49.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 -- 172.21.15.50:0/3437516564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03d41094c0 msgr2=0x7f03d419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03d41094c0 0x7f03d419dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 -- 172.21.15.50:0/3437516564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d419e210 con 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d2ffd700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 0x7f03d4199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d27fc700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 0x7f03d4199740 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f03c800ec10 tx=0x7f03c800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d8888700 10 client.?.objecter ms_handle_connect 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d8888700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c800cd40 con 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f03c8004480 con 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d419e500 con 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8010430 con 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 10 client.14691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d41a5d80 con 0x7f03d4103e30 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 librados: init done 2024-04-26T16:20:49.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d406a110 con 0x7f03d4103e30 2024-04-26T16:20:49.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.925+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f03c8003ef0 con 0x7f03d4103e30 2024-04-26T16:20:49.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03bc071f80 0x7f03bc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:49.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(119..119 src has 67..119) ==== 7450+0+0 (secure 0 0 0) 0x7f03c8091600 con 0x7f03d4103e30 2024-04-26T16:20:49.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 10 client.14691.objecter ms_dispatch 0x7f03d4001550 osd_map(119..119 src has 67..119) 2024-04-26T16:20:49.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 3 client.14691.objecter handle_osd_map got epochs [119,119] > 0 2024-04-26T16:20:49.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 3 client.14691.objecter handle_osd_map decoding full epoch 119 2024-04-26T16:20:49.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d2ffd700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03bc071f80 0x7f03bc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:49.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 20 client.14691.objecter dump_active .. 0 homeless 2024-04-26T16:20:49.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d2ffd700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03bc071f80 0x7f03bc074440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009810 tx=0x7f03c4019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:49.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.929+0000 7f03d8888700 10 client.14691.objecter ms_handle_connect 0x7f03bc071f80 2024-04-26T16:20:49.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:49.933+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f03c805d790 con 0x7f03d4103e30 2024-04-26T16:20:50.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.205+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f03d40756b0 con 0x7f03d4103e30 2024-04-26T16:20:50.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.205+0000 7f03d8888700 1 -- 172.21.15.50:0/3437516564 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f03c8061440 con 0x7f03d4103e30 2024-04-26T16:20:50.199 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:20:50.199 INFO:teuthology.orchestra.run.smithi050.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 10 librados: watch_flush enter 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 10 librados: watch_flush exit 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 20 client.14691.objecter shutdown clearing up homeless session... 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 10 client.14691.objecter successfully canceled tick 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03bc071f80 msgr2=0x7f03bc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03bc071f80 0x7f03bc074440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f03c4009810 tx=0x7f03c4019040 comp rx=0 tx=0).stop 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 msgr2=0x7f03d4199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 0x7f03d4199740 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f03c800ec10 tx=0x7f03c800ef20 comp rx=0 tx=0).stop 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 shutdown_connections 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03bc071f80 0x7f03bc074440 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03d41094c0 0x7f03d419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:50.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f03d4103470 0x7f03d4199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 --2- 172.21.15.50:0/3437516564 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f03d4103e30 0x7f03d4199740 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 >> 172.21.15.50:0/3437516564 conn(0x7f03d407a580 msgr2=0x7f03d4101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 shutdown_connections 2024-04-26T16:20:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 -- 172.21.15.50:0/3437516564 wait complete. 2024-04-26T16:20:50.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:50.209+0000 7f03d988a700 1 librados: shutdown 2024-04-26T16:20:50.516 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: cluster 2024-04-26T16:20:49.404170+0000 mgr.y (mgr.24401) 124 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:50.516 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: cluster 2024-04-26T16:20:49.404170+0000 mgr.y (mgr.24401) 124 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:50.516 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: cluster 2024-04-26T16:20:49.521209+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:20:50.516 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: cluster 2024-04-26T16:20:49.521209+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:20:50.516 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: cluster 2024-04-26T16:20:49.523861+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: cluster 2024-04-26T16:20:49.523861+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: audit 2024-04-26T16:20:50.209391+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.50:0/3437516564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[21463]: audit 2024-04-26T16:20:50.209391+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.50:0/3437516564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: cluster 2024-04-26T16:20:49.404170+0000 mgr.y (mgr.24401) 124 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: cluster 2024-04-26T16:20:49.404170+0000 mgr.y (mgr.24401) 124 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: cluster 2024-04-26T16:20:49.521209+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: cluster 2024-04-26T16:20:49.521209+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: cluster 2024-04-26T16:20:49.523861+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: cluster 2024-04-26T16:20:49.523861+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: audit 2024-04-26T16:20:50.209391+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.50:0/3437516564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T16:20:50.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:50 smithi050 bash[29234]: audit 2024-04-26T16:20:50.209391+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.50:0/3437516564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: cluster 2024-04-26T16:20:49.404170+0000 mgr.y (mgr.24401) 124 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: cluster 2024-04-26T16:20:49.404170+0000 mgr.y (mgr.24401) 124 : cluster [DBG] pgmap v94: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: cluster 2024-04-26T16:20:49.521209+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: cluster 2024-04-26T16:20:49.521209+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: cluster 2024-04-26T16:20:49.523861+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: cluster 2024-04-26T16:20:49.523861+0000 osd.5 (osd.5) 86 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: audit 2024-04-26T16:20:50.209391+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.50:0/3437516564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T16:20:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:50 smithi190 bash[23868]: audit 2024-04-26T16:20:50.209391+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.50:0/3437516564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-26T16:20:51.100 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-26T16:20:51.101 INFO:tasks.cephadm:Setup complete, yielding 2024-04-26T16:20:51.101 INFO:teuthology.run_tasks:Running task thrashosds... 2024-04-26T16:20:51.117 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-26T16:20:51.117 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-phlogistonjohn-testing-2024-04-25-1915'}, 'ceph': {'conf': {'client': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mgrc': 20, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'crush_tunables': 'jewel', 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'reached quota', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6'}}, 'workunit': {'branch': 'wip-phlogistonjohn-testing-2024-04-25-1915', 'sha1': 'a430f9ed84de3008ca5b632b1ed304bac0e753f6'}} 2024-04-26T16:20:51.117 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-04-26T16:20:51.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-04-26T16:20:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:51 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.015101+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:20:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:51 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.015101+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:20:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:51 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.017382+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:20:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:51 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.017382+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:20:51.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.015101+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.015101+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.017382+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.017382+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.015101+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.015101+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.017382+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:20:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:51 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.017382+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:20:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.058343+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:20:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.058343+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:20:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.060540+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:20:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.060540+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:20:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.405565+0000 mgr.y (mgr.24401) 125 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.405565+0000 mgr.y (mgr.24401) 125 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.459320+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:20:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.459320+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:20:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.461645+0000 osd.5 (osd.5) 88 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:20:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:52 smithi190 bash[23868]: cluster 2024-04-26T16:20:51.461645+0000 osd.5 (osd.5) 88 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.058343+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.058343+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.060540+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.060540+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.405565+0000 mgr.y (mgr.24401) 125 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.405565+0000 mgr.y (mgr.24401) 125 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.459320+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.459320+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.461645+0000 osd.5 (osd.5) 88 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:20:52.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[29234]: cluster 2024-04-26T16:20:51.461645+0000 osd.5 (osd.5) 88 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.058343+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.058343+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.060540+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.060540+0000 osd.7 (osd.7) 94 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.405565+0000 mgr.y (mgr.24401) 125 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.405565+0000 mgr.y (mgr.24401) 125 : cluster [DBG] pgmap v95: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.459320+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.459320+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.461645+0000 osd.5 (osd.5) 88 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:20:52.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:52 smithi050 bash[21463]: cluster 2024-04-26T16:20:51.461645+0000 osd.5 (osd.5) 88 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:20:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:52.078046+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:52.078046+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:52.080643+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:52.080643+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.087482+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.087482+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.090819+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:20:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:53 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.090819+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:20:53.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:52.078046+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:52.078046+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:52.080643+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:52.080643+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.087482+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.087482+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.090819+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.090819+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:52.078046+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:52.078046+0000 osd.7 (osd.7) 95 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:52.080643+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:20:53.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:52.080643+0000 osd.7 (osd.7) 96 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:20:53.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.087482+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:20:53.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.087482+0000 osd.3 (osd.3) 83 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:20:53.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.090819+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:20:53.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:53 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.090819+0000 osd.3 (osd.3) 84 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:20:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:54 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.406391+0000 mgr.y (mgr.24401) 126 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-04-26T16:20:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:54 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.406391+0000 mgr.y (mgr.24401) 126 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-04-26T16:20:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:54 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.793488+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:20:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:54 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.793488+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:20:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:54 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.797418+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:20:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:54 smithi190 bash[23868]: cluster 2024-04-26T16:20:53.797418+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:20:54.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:20:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.406391+0000 mgr.y (mgr.24401) 126 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.406391+0000 mgr.y (mgr.24401) 126 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.793488+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.793488+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.797418+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[21463]: cluster 2024-04-26T16:20:53.797418+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.406391+0000 mgr.y (mgr.24401) 126 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.406391+0000 mgr.y (mgr.24401) 126 : cluster [DBG] pgmap v96: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-04-26T16:20:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.793488+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:20:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.793488+0000 osd.4 (osd.4) 73 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:20:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.797418+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:20:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:54 smithi050 bash[29234]: cluster 2024-04-26T16:20:53.797418+0000 osd.4 (osd.4) 74 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:20:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:55 smithi190 bash[23868]: cluster 2024-04-26T16:20:54.005028+0000 osd.7 (osd.7) 97 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:20:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:55 smithi190 bash[23868]: cluster 2024-04-26T16:20:54.005028+0000 osd.7 (osd.7) 97 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:20:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:55 smithi190 bash[23868]: cluster 2024-04-26T16:20:54.009857+0000 osd.7 (osd.7) 98 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:20:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:55 smithi190 bash[23868]: cluster 2024-04-26T16:20:54.009857+0000 osd.7 (osd.7) 98 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:20:55.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[21463]: cluster 2024-04-26T16:20:54.005028+0000 osd.7 (osd.7) 97 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[21463]: cluster 2024-04-26T16:20:54.005028+0000 osd.7 (osd.7) 97 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[21463]: cluster 2024-04-26T16:20:54.009857+0000 osd.7 (osd.7) 98 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[21463]: cluster 2024-04-26T16:20:54.009857+0000 osd.7 (osd.7) 98 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[29234]: cluster 2024-04-26T16:20:54.005028+0000 osd.7 (osd.7) 97 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[29234]: cluster 2024-04-26T16:20:54.005028+0000 osd.7 (osd.7) 97 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[29234]: cluster 2024-04-26T16:20:54.009857+0000 osd.7 (osd.7) 98 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:20:55.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:55 smithi050 bash[29234]: cluster 2024-04-26T16:20:54.009857+0000 osd.7 (osd.7) 98 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:20:55.916 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:20:56.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:20:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.044144+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:20:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.044144+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:20:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.047095+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:20:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.047095+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:20:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.353790+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.353790+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.353944+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.353944+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.407077+0000 mgr.y (mgr.24401) 127 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.407077+0000 mgr.y (mgr.24401) 127 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.464524+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.a scrub starts 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.464524+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.a scrub starts 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.466817+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.a scrub ok 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.466817+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.a scrub ok 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.500475+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: cluster 2024-04-26T16:20:55.500475+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: audit 2024-04-26T16:20:55.713264+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: audit 2024-04-26T16:20:55.713264+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:56.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: audit 2024-04-26T16:20:56.059466+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:56 smithi190 bash[23868]: audit 2024-04-26T16:20:56.059466+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.044144+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.044144+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.047095+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.047095+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.353790+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.353790+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.353944+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:56.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.353944+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.407077+0000 mgr.y (mgr.24401) 127 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.407077+0000 mgr.y (mgr.24401) 127 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.464524+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.a scrub starts 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.464524+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.a scrub starts 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.466817+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.a scrub ok 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.466817+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.a scrub ok 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.500475+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: cluster 2024-04-26T16:20:55.500475+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: audit 2024-04-26T16:20:55.713264+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: audit 2024-04-26T16:20:55.713264+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: audit 2024-04-26T16:20:56.059466+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[29234]: audit 2024-04-26T16:20:56.059466+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.044144+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.044144+0000 osd.7 (osd.7) 99 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.047095+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.047095+0000 osd.7 (osd.7) 100 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.353790+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.353790+0000 osd.3 (osd.3) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.353944+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.353944+0000 osd.3 (osd.3) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.407077+0000 mgr.y (mgr.24401) 127 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.407077+0000 mgr.y (mgr.24401) 127 : cluster [DBG] pgmap v97: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.464524+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.a scrub starts 2024-04-26T16:20:56.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.464524+0000 osd.5 (osd.5) 89 : cluster [DBG] 6.a scrub starts 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.466817+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.a scrub ok 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.466817+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.a scrub ok 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.500475+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: cluster 2024-04-26T16:20:55.500475+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: audit 2024-04-26T16:20:55.713264+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: audit 2024-04-26T16:20:55.713264+0000 mon.c (mon.2) 72 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: audit 2024-04-26T16:20:56.059466+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:56.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:56 smithi050 bash[21463]: audit 2024-04-26T16:20:56.059466+0000 mon.c (mon.2) 73 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.460716+0000 mgr.y (mgr.24401) 128 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.460716+0000 mgr.y (mgr.24401) 128 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.790358+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.790358+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.792312+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.792312+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.803398+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:20:57.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:57 smithi190 bash[23868]: audit 2024-04-26T16:20:56.803398+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.460716+0000 mgr.y (mgr.24401) 128 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.460716+0000 mgr.y (mgr.24401) 128 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.790358+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.790358+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.792312+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.792312+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.803398+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[29234]: audit 2024-04-26T16:20:56.803398+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:20:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.460716+0000 mgr.y (mgr.24401) 128 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.460716+0000 mgr.y (mgr.24401) 128 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.790358+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.790358+0000 mon.c (mon.2) 74 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.792312+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.792312+0000 mon.c (mon.2) 75 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.803398+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:20:57.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:57 smithi050 bash[21463]: audit 2024-04-26T16:20:56.803398+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:20:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:58 smithi190 bash[23868]: cluster 2024-04-26T16:20:57.408574+0000 mgr.y (mgr.24401) 129 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:58 smithi190 bash[23868]: cluster 2024-04-26T16:20:57.408574+0000 mgr.y (mgr.24401) 129 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:58.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:58 smithi050 bash[21463]: cluster 2024-04-26T16:20:57.408574+0000 mgr.y (mgr.24401) 129 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:58.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:58 smithi050 bash[21463]: cluster 2024-04-26T16:20:57.408574+0000 mgr.y (mgr.24401) 129 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:58.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:58 smithi050 bash[29234]: cluster 2024-04-26T16:20:57.408574+0000 mgr.y (mgr.24401) 129 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:58.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:58 smithi050 bash[29234]: cluster 2024-04-26T16:20:57.408574+0000 mgr.y (mgr.24401) 129 : cluster [DBG] pgmap v99: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:20:59.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- 172.21.15.50:0/1721533299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0010ff10 msgr2=0x7fee00112300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- 172.21.15.50:0/1721533299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0010ff10 0x7fee00112300 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fedf800b0a0 tx=0x7fedf800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- 172.21.15.50:0/1721533299 shutdown_connections 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- 172.21.15.50:0/1721533299 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0006bbb0 0x7fee0006bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- 172.21.15.50:0/1721533299 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0010ff10 0x7fee00112300 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- 172.21.15.50:0/1721533299 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006c4d0 0x7fee0010f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- 172.21.15.50:0/1721533299 >> 172.21.15.50:0/1721533299 conn(0x7fee00079250 msgr2=0x7fee00079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- 172.21.15.50:0/1721533299 shutdown_connections 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- 172.21.15.50:0/1721533299 wait complete. 2024-04-26T16:20:59.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 librados: starting msgr at 2024-04-26T16:20:59.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 librados: starting objecter 2024-04-26T16:20:59.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 Processor -- start 2024-04-26T16:20:59.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- start start 2024-04-26T16:20:59.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 librados: setting wanted keys 2024-04-26T16:20:59.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 librados: calling monclient init 2024-04-26T16:20:59.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 0x7fee0019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:59.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0006c4d0 0x7fee0019b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:59.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0010ff10 0x7fee0019fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:59.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee00114d10 con 0x7fee0006bbb0 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee00114b90 con 0x7fee0010ff10 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee080dc700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fee00114e90 con 0x7fee0006c4d0 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee05e78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 0x7fee0019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee05e78700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 0x7fee0019b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45874/0 (socket says 172.21.15.50:45874) 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee05e78700 1 -- 172.21.15.50:0/1179318205 learned_addr learned my addr 172.21.15.50:0/1179318205 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee05677700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0006c4d0 0x7fee0019b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.357+0000 7fee06679700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0010ff10 0x7fee0019fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05e78700 1 -- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0006c4d0 msgr2=0x7fee0019b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05e78700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0006c4d0 0x7fee0019b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05e78700 1 -- 172.21.15.50:0/1179318205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0010ff10 msgr2=0x7fee0019fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05e78700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0010ff10 0x7fee0019fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05e78700 1 -- 172.21.15.50:0/1179318205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee001a0300 con 0x7fee0006bbb0 2024-04-26T16:20:59.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05e78700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 0x7fee0019b2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00d950 tx=0x7fedfc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 10 client.?.objecter ms_handle_connect 0x7fee0006bbb0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc016070 con 0x7fee0006bbb0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fedfc009da0 con 0x7fee0006bbb0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee001a05f0 con 0x7fee0006bbb0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedfc010e60 con 0x7fee0006bbb0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee080dc700 10 client.14697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee001a7e70 con 0x7fee0006bbb0 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee080dc700 1 librados: init done 2024-04-26T16:20:59.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee00102940 con 0x7fee0006bbb0 2024-04-26T16:20:59.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fedfc019070 con 0x7fee0006bbb0 2024-04-26T16:20:59.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fedec071f80 0x7fedec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:20:59.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(120..120 src has 67..120) ==== 7450+0+0 (secure 0 0 0) 0x7fedfc090d80 con 0x7fee0006bbb0 2024-04-26T16:20:59.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 10 client.14697.objecter ms_dispatch 0x7fee00001550 osd_map(120..120 src has 67..120) 2024-04-26T16:20:59.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 3 client.14697.objecter handle_osd_map got epochs [120,120] > 0 2024-04-26T16:20:59.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 3 client.14697.objecter handle_osd_map decoding full epoch 120 2024-04-26T16:20:59.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05677700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fedec071f80 0x7fedec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:20:59.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 20 client.14697.objecter dump_active .. 0 homeless 2024-04-26T16:20:59.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fee05677700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fedec071f80 0x7fedec074440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009680 tx=0x7fedf0009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:20:59.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.361+0000 7fedf6ffd700 10 client.14697.objecter ms_handle_connect 0x7fedec071f80 2024-04-26T16:20:59.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.369+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fedfc05cf90 con 0x7fee0006bbb0 2024-04-26T16:20:59.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.597+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}] v 0) -- 0x7fee0002e490 con 0x7fee0006bbb0 2024-04-26T16:20:59.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.601+0000 7fedf6ffd700 1 -- 172.21.15.50:0/1179318205 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr_debug_aggressive_pg_num_changes}]=0 v30)=0 v30) ==== 136+0+0 (secure 0 0 0) 0x7fedfc060c40 con 0x7fee0006bbb0 2024-04-26T16:20:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 10 librados: watch_flush enter 2024-04-26T16:20:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 10 librados: watch_flush exit 2024-04-26T16:20:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 20 client.14697.objecter shutdown clearing up homeless session... 2024-04-26T16:20:59.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 10 client.14697.objecter successfully canceled tick 2024-04-26T16:20:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fedec071f80 msgr2=0x7fedec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fedec071f80 0x7fedec074440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009680 tx=0x7fedf0009260 comp rx=0 tx=0).stop 2024-04-26T16:20:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 msgr2=0x7fee0019b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 0x7fee0019b2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00d950 tx=0x7fedfc00dc60 comp rx=0 tx=0).stop 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 shutdown_connections 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fedec071f80 0x7fedec074440 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee0010ff10 0x7fee0019fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee0006bbb0 0x7fee0019b2f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 --2- 172.21.15.50:0/1179318205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee0006c4d0 0x7fee0019b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 >> 172.21.15.50:0/1179318205 conn(0x7fee00079250 msgr2=0x7fee001117d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 shutdown_connections 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 -- 172.21.15.50:0/1179318205 wait complete. 2024-04-26T16:20:59.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:20:59.605+0000 7fee080dc700 1 librados: shutdown 2024-04-26T16:20:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:59 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.109291+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:20:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:59 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.109291+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:20:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:59 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.111289+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:20:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:20:59 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.111289+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:20:59.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.109291+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.109291+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.111289+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.111289+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21717]: [26/Apr/2024:16:20:59] ENGINE Bus STOPPING 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21717]: [26/Apr/2024:16:20:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21717]: [26/Apr/2024:16:20:59] ENGINE Bus STOPPED 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21717]: [26/Apr/2024:16:20:59] ENGINE Bus STARTING 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21717]: [26/Apr/2024:16:20:59] ENGINE Serving on http://:::9283 2024-04-26T16:20:59.875 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[21717]: [26/Apr/2024:16:20:59] ENGINE Bus STARTED 2024-04-26T16:20:59.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.109291+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:20:59.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.109291+0000 osd.3 (osd.3) 87 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:20:59.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.111289+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:20:59.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:20:59 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.111289+0000 osd.3 (osd.3) 88 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:21:00.530 INFO:tasks.thrashosds:Beginning thrashosds... 2024-04-26T16:21:00.530 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-26T16:21:00.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:21:00.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.409030+0000 mgr.y (mgr.24401) 130 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:00.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.409030+0000 mgr.y (mgr.24401) 130 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:00.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.516067+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:21:00.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.516067+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:21:00.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.518888+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: cluster 2024-04-26T16:20:59.518888+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.605965+0000 mon.a (mon.0) 930 : audit [INF] from='client.? 172.21.15.50:0/1179318205' entity='client.admin' 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.605965+0000 mon.a (mon.0) 930 : audit [INF] from='client.? 172.21.15.50:0/1179318205' entity='client.admin' 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.653927+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.653927+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.655902+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.655902+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.656631+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.656631+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.662327+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[21463]: audit 2024-04-26T16:20:59.662327+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:21:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.409030+0000 mgr.y (mgr.24401) 130 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.409030+0000 mgr.y (mgr.24401) 130 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.516067+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.516067+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.518888+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: cluster 2024-04-26T16:20:59.518888+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.605965+0000 mon.a (mon.0) 930 : audit [INF] from='client.? 172.21.15.50:0/1179318205' entity='client.admin' 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.605965+0000 mon.a (mon.0) 930 : audit [INF] from='client.? 172.21.15.50:0/1179318205' entity='client.admin' 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.653927+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.653927+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.655902+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.655902+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.656631+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.656631+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.662327+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:21:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:00 smithi050 bash[29234]: audit 2024-04-26T16:20:59.662327+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:21:01.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.409030+0000 mgr.y (mgr.24401) 130 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:01.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.409030+0000 mgr.y (mgr.24401) 130 : cluster [DBG] pgmap v100: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:01.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.516067+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.516067+0000 osd.5 (osd.5) 91 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.518888+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: cluster 2024-04-26T16:20:59.518888+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.605965+0000 mon.a (mon.0) 930 : audit [INF] from='client.? 172.21.15.50:0/1179318205' entity='client.admin' 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.605965+0000 mon.a (mon.0) 930 : audit [INF] from='client.? 172.21.15.50:0/1179318205' entity='client.admin' 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.653927+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.653927+0000 mon.c (mon.2) 76 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.655902+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.655902+0000 mon.c (mon.2) 77 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.656631+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.656631+0000 mon.c (mon.2) 78 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.662327+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:21:01.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:00 smithi190 bash[23868]: audit 2024-04-26T16:20:59.662327+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:21:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:01 smithi190 bash[23868]: cluster 2024-04-26T16:21:00.994945+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:21:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:01 smithi190 bash[23868]: cluster 2024-04-26T16:21:00.994945+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:21:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:01 smithi190 bash[23868]: cluster 2024-04-26T16:21:00.996580+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:21:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:01 smithi190 bash[23868]: cluster 2024-04-26T16:21:00.996580+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:21:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:01 smithi190 bash[23868]: cluster 2024-04-26T16:21:01.410376+0000 mgr.y (mgr.24401) 131 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:01 smithi190 bash[23868]: cluster 2024-04-26T16:21:01.410376+0000 mgr.y (mgr.24401) 131 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[29234]: cluster 2024-04-26T16:21:00.994945+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[29234]: cluster 2024-04-26T16:21:00.994945+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[29234]: cluster 2024-04-26T16:21:00.996580+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[29234]: cluster 2024-04-26T16:21:00.996580+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[29234]: cluster 2024-04-26T16:21:01.410376+0000 mgr.y (mgr.24401) 131 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[29234]: cluster 2024-04-26T16:21:01.410376+0000 mgr.y (mgr.24401) 131 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:02.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[21463]: cluster 2024-04-26T16:21:00.994945+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:21:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[21463]: cluster 2024-04-26T16:21:00.994945+0000 osd.7 (osd.7) 101 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:21:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[21463]: cluster 2024-04-26T16:21:00.996580+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:21:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[21463]: cluster 2024-04-26T16:21:00.996580+0000 osd.7 (osd.7) 102 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:21:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[21463]: cluster 2024-04-26T16:21:01.410376+0000 mgr.y (mgr.24401) 131 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:02.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:01 smithi050 bash[21463]: cluster 2024-04-26T16:21:01.410376+0000 mgr.y (mgr.24401) 131 : cluster [DBG] pgmap v101: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:03.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:02 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.311220+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:21:03.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:02 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.311220+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:21:03.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:02 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.316327+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:21:03.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:02 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.316327+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:21:03.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.311220+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:21:03.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.311220+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:21:03.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.316327+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:21:03.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.316327+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:21:03.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.311220+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:21:03.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.311220+0000 osd.0 (osd.0) 81 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:21:03.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.316327+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:21:03.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:02 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.316327+0000 osd.0 (osd.0) 82 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:21:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:03 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.456272+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:21:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:03 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.456272+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:21:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:03 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.458768+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:21:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:03 smithi190 bash[23868]: cluster 2024-04-26T16:21:02.458768+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:21:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:03 smithi190 bash[23868]: cluster 2024-04-26T16:21:03.411080+0000 mgr.y (mgr.24401) 132 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:03 smithi190 bash[23868]: cluster 2024-04-26T16:21:03.411080+0000 mgr.y (mgr.24401) 132 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.456272+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.456272+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.458768+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[21463]: cluster 2024-04-26T16:21:02.458768+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[21463]: cluster 2024-04-26T16:21:03.411080+0000 mgr.y (mgr.24401) 132 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[21463]: cluster 2024-04-26T16:21:03.411080+0000 mgr.y (mgr.24401) 132 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:04.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.456272+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:21:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.456272+0000 osd.2 (osd.2) 59 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:21:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.458768+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:21:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[29234]: cluster 2024-04-26T16:21:02.458768+0000 osd.2 (osd.2) 60 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:21:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[29234]: cluster 2024-04-26T16:21:03.411080+0000 mgr.y (mgr.24401) 132 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:04.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:03 smithi050 bash[29234]: cluster 2024-04-26T16:21:03.411080+0000 mgr.y (mgr.24401) 132 : cluster [DBG] pgmap v102: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:04.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:21:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.002737+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.b scrub starts 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.002737+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.b scrub starts 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.005428+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.b scrub ok 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.005428+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.b scrub ok 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.071567+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.071567+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.075896+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:21:05.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:04 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.075896+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:21:05.332 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:05.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.002737+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.b scrub starts 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.002737+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.b scrub starts 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.005428+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.b scrub ok 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.005428+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.b scrub ok 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.071567+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.071567+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.075896+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.075896+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.002737+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.b scrub starts 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.002737+0000 osd.3 (osd.3) 89 : cluster [DBG] 6.b scrub starts 2024-04-26T16:21:05.358 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.005428+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.b scrub ok 2024-04-26T16:21:05.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.005428+0000 osd.3 (osd.3) 90 : cluster [DBG] 6.b scrub ok 2024-04-26T16:21:05.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.071567+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:21:05.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.071567+0000 osd.7 (osd.7) 103 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:21:05.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.075896+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:21:05.359 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:04 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.075896+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:21:06.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:05 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.414582+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-04-26T16:21:06.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:05 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.414582+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-04-26T16:21:06.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:05 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.417169+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-04-26T16:21:06.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:05 smithi190 bash[23868]: cluster 2024-04-26T16:21:04.417169+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-04-26T16:21:06.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:05 smithi190 bash[23868]: cluster 2024-04-26T16:21:05.411667+0000 mgr.y (mgr.24401) 133 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:06.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:05 smithi190 bash[23868]: cluster 2024-04-26T16:21:05.411667+0000 mgr.y (mgr.24401) 133 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:06.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.414582+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.414582+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.417169+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[21463]: cluster 2024-04-26T16:21:04.417169+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[21463]: cluster 2024-04-26T16:21:05.411667+0000 mgr.y (mgr.24401) 133 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[21463]: cluster 2024-04-26T16:21:05.411667+0000 mgr.y (mgr.24401) 133 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.414582+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.414582+0000 osd.2 (osd.2) 61 : cluster [DBG] 5.b scrub starts 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.417169+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[29234]: cluster 2024-04-26T16:21:04.417169+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.b scrub ok 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[29234]: cluster 2024-04-26T16:21:05.411667+0000 mgr.y (mgr.24401) 133 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:06.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:05 smithi050 bash[29234]: cluster 2024-04-26T16:21:05.411667+0000 mgr.y (mgr.24401) 133 : cluster [DBG] pgmap v103: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:06.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:21:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.012665+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:21:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.012665+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:21:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.017698+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:21:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.017698+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:21:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[23868]: audit 2024-04-26T16:21:06.471292+0000 mgr.y (mgr.24401) 134 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:06 smithi190 bash[23868]: audit 2024-04-26T16:21:06.471292+0000 mgr.y (mgr.24401) 134 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.012665+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.012665+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.017698+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.017698+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[21463]: audit 2024-04-26T16:21:06.471292+0000 mgr.y (mgr.24401) 134 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[21463]: audit 2024-04-26T16:21:06.471292+0000 mgr.y (mgr.24401) 134 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.012665+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.012665+0000 osd.3 (osd.3) 91 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:21:07.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.017698+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:21:07.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.017698+0000 osd.3 (osd.3) 92 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:21:07.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[29234]: audit 2024-04-26T16:21:06.471292+0000 mgr.y (mgr.24401) 134 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:07.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:06 smithi050 bash[29234]: audit 2024-04-26T16:21:06.471292+0000 mgr.y (mgr.24401) 134 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:07 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.902050+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:07 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.902050+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:07 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.902199+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:08 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.902199+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:08 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.997896+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:08 smithi190 bash[23868]: cluster 2024-04-26T16:21:06.997896+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:08 smithi190 bash[23868]: cluster 2024-04-26T16:21:07.412330+0000 mgr.y (mgr.24401) 135 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:08 smithi190 bash[23868]: cluster 2024-04-26T16:21:07.412330+0000 mgr.y (mgr.24401) 135 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.902050+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.902050+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.902199+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.902199+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.997896+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:06.997896+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:07.412330+0000 mgr.y (mgr.24401) 135 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[21463]: cluster 2024-04-26T16:21:07.412330+0000 mgr.y (mgr.24401) 135 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:08.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.902050+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.902050+0000 osd.0 (osd.0) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.902199+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.902199+0000 osd.0 (osd.0) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.997896+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:06.997896+0000 mon.a (mon.0) 932 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:07.412330+0000 mgr.y (mgr.24401) 135 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:08.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:08 smithi050 bash[29234]: cluster 2024-04-26T16:21:07.412330+0000 mgr.y (mgr.24401) 135 : cluster [DBG] pgmap v105: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/2751303190 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 msgr2=0x7f95e0102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- 172.21.15.50:0/2751303190 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e0102490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f95c8009990 tx=0x7f95c8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/2751303190 shutdown_connections 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- 172.21.15.50:0/2751303190 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95e010fc30 0x7f95e01120d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- 172.21.15.50:0/2751303190 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e0102490 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- 172.21.15.50:0/2751303190 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e0102a60 0x7f95e010f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/2751303190 >> 172.21.15.50:0/2751303190 conn(0x7f95e0077590 msgr2=0x7f95e00779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/2751303190 shutdown_connections 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/2751303190 wait complete. 2024-04-26T16:21:08.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 librados: starting msgr at 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 librados: starting objecter 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 Processor -- start 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- start start 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 librados: setting wanted keys 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 librados: calling monclient init 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e019b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:08.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95e0102a60 0x7f95e019b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:08.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e010fc30 0x7f95e019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:08.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0114d50 con 0x7f95e010fc30 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f95e0114bd0 con 0x7f95e0102a60 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f95e0114ed0 con 0x7f95e01020b0 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e019b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e019b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42002/0 (socket says 172.21.15.50:42002) 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 -- 172.21.15.50:0/148942495 learned_addr learned my addr 172.21.15.50:0/148942495 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dffff700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e010fc30 0x7f95e019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95deffd700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95e0102a60 0x7f95e019b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 -- 172.21.15.50:0/148942495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95e0102a60 msgr2=0x7f95e019b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95e0102a60 0x7f95e019b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 -- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e010fc30 msgr2=0x7f95e019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e010fc30 0x7f95e019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 -- 172.21.15.50:0/148942495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e01a04c0 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dffff700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e010fc30 0x7f95e019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95df7fe700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e019b4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f95c80057a0 tx=0x7f95c80057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dcff9700 10 client.?.objecter ms_handle_connect 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dcff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c8026070 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f95c8021410 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e01a0750 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c80173f0 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 10 client.24593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e01a8030 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.697+0000 7f95e6160700 1 librados: init done 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e006a110 con 0x7f95e01020b0 2024-04-26T16:21:08.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f95c8029070 con 0x7f95e01020b0 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f95cc071f80 0x7f95cc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(121..121 src has 67..121) ==== 7450+0+0 (secure 0 0 0) 0x7f95c8091f40 con 0x7f95e01020b0 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 10 client.24593.objecter ms_dispatch 0x7f95e0001550 osd_map(121..121 src has 67..121) 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 3 client.24593.objecter handle_osd_map got epochs [121,121] > 0 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 3 client.24593.objecter handle_osd_map decoding full epoch 121 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95deffd700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f95cc071f80 0x7f95cc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 20 client.24593.objecter dump_active .. 0 homeless 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95deffd700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f95cc071f80 0x7f95cc074440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f95e0077fb0 tx=0x7f95d000b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:08.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.701+0000 7f95dcff9700 10 client.24593.objecter ms_handle_connect 0x7f95cc071f80 2024-04-26T16:21:08.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.705+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f95c8013070 con 0x7f95e01020b0 2024-04-26T16:21:08.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.925+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f95e0111cf0 con 0x7f95cc071f80 2024-04-26T16:21:08.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.937+0000 7f95dcff9700 1 -- 172.21.15.50:0/148942495 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346646 (secure 0 0 0) 0x7f95e0111cf0 con 0x7f95cc071f80 2024-04-26T16:21:08.934 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 10 librados: watch_flush enter 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 10 librados: watch_flush exit 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 20 client.24593.objecter shutdown clearing up homeless session... 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 10 client.24593.objecter successfully canceled tick 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f95cc071f80 msgr2=0x7f95cc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f95cc071f80 0x7f95cc074440 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f95e0077fb0 tx=0x7f95d000b040 comp rx=0 tx=0).stop 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 msgr2=0x7f95e019b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:08.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e019b4b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f95c80057a0 tx=0x7f95c80057d0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 shutdown_connections 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f95cc071f80 0x7f95cc074440 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f95e010fc30 0x7f95e019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f95e01020b0 0x7f95e019b4b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 --2- 172.21.15.50:0/148942495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95e0102a60 0x7f95e019b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 >> 172.21.15.50:0/148942495 conn(0x7f95e0077590 msgr2=0x7f95e01115e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 shutdown_connections 2024-04-26T16:21:08.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 -- 172.21.15.50:0/148942495 wait complete. 2024-04-26T16:21:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:08.941+0000 7f95e6160700 1 librados: shutdown 2024-04-26T16:21:08.941 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:21:09.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[21463]: cluster 2024-04-26T16:21:08.079716+0000 osd.7 (osd.7) 105 : cluster [DBG] 3.e scrub starts 2024-04-26T16:21:09.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[21463]: cluster 2024-04-26T16:21:08.079716+0000 osd.7 (osd.7) 105 : cluster [DBG] 3.e scrub starts 2024-04-26T16:21:09.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[21463]: cluster 2024-04-26T16:21:08.083766+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.e scrub ok 2024-04-26T16:21:09.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[21463]: cluster 2024-04-26T16:21:08.083766+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.e scrub ok 2024-04-26T16:21:09.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[29234]: cluster 2024-04-26T16:21:08.079716+0000 osd.7 (osd.7) 105 : cluster [DBG] 3.e scrub starts 2024-04-26T16:21:09.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[29234]: cluster 2024-04-26T16:21:08.079716+0000 osd.7 (osd.7) 105 : cluster [DBG] 3.e scrub starts 2024-04-26T16:21:09.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[29234]: cluster 2024-04-26T16:21:08.083766+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.e scrub ok 2024-04-26T16:21:09.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:09 smithi050 bash[29234]: cluster 2024-04-26T16:21:08.083766+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.e scrub ok 2024-04-26T16:21:09.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:09 smithi190 bash[23868]: cluster 2024-04-26T16:21:08.079716+0000 osd.7 (osd.7) 105 : cluster [DBG] 3.e scrub starts 2024-04-26T16:21:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:09 smithi190 bash[23868]: cluster 2024-04-26T16:21:08.079716+0000 osd.7 (osd.7) 105 : cluster [DBG] 3.e scrub starts 2024-04-26T16:21:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:09 smithi190 bash[23868]: cluster 2024-04-26T16:21:08.083766+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.e scrub ok 2024-04-26T16:21:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:09 smithi190 bash[23868]: cluster 2024-04-26T16:21:08.083766+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.e scrub ok 2024-04-26T16:21:09.811 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-26T16:21:07.411934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1147,"num_read_kb":993,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":683,"ondisk_log_size":683,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244444,"kb_used_data":7516,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716772,"statfs":{"total":767960285184,"available":767709974528,"internally_reserved":0,"allocated":7696384,"data_stored":4879513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":242535988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004533"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.178350+0000","last_change":"2024-04-26T16:20:46.049553+0000","last_active":"2024-04-26T16:20:56.178350+0000","last_peered":"2024-04-26T16:20:56.178350+0000","last_clean":"2024-04-26T16:20:56.178350+0000","last_became_active":"2024-04-26T16:17:35.677526+0000","last_became_peered":"2024-04-26T16:17:35.677526+0000","last_unstale":"2024-04-26T16:20:56.178350+0000","last_undegraded":"2024-04-26T16:20:56.178350+0000","last_fullsized":"2024-04-26T16:20:56.178350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:46.049339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:46.049339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.981921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494011+0000","last_change":"2024-04-26T16:20:05.019229+0000","last_active":"2024-04-26T16:20:55.494011+0000","last_peered":"2024-04-26T16:20:55.494011+0000","last_clean":"2024-04-26T16:20:55.494011+0000","last_became_active":"2024-04-26T16:17:33.070922+0000","last_became_peered":"2024-04-26T16:17:33.070922+0000","last_unstale":"2024-04-26T16:20:55.494011+0000","last_undegraded":"2024-04-26T16:20:55.494011+0000","last_fullsized":"2024-04-26T16:20:55.494011+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.018938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.018938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.770490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.180229+0000","last_change":"2024-04-26T16:20:17.275735+0000","last_active":"2024-04-26T16:20:56.180229+0000","last_peered":"2024-04-26T16:20:56.180229+0000","last_clean":"2024-04-26T16:20:56.180229+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:20:56.180229+0000","last_undegraded":"2024-04-26T16:20:56.180229+0000","last_fullsized":"2024-04-26T16:20:56.180229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.275530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.275530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:42.183654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"86'10","reported_seq":119,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.181094+0000","last_change":"2024-04-26T16:19:39.904290+0000","last_active":"2024-04-26T16:20:56.181094+0000","last_peered":"2024-04-26T16:20:56.181094+0000","last_clean":"2024-04-26T16:20:56.181094+0000","last_became_active":"2024-04-26T16:17:31.049997+0000","last_became_peered":"2024-04-26T16:17:31.049997+0000","last_unstale":"2024-04-26T16:20:56.181094+0000","last_undegraded":"2024-04-26T16:20:56.181094+0000","last_fullsized":"2024-04-26T16:20:56.181094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:19:39.904035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:39.904035+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494143+0000","last_change":"2024-04-26T16:20:02.038119+0000","last_active":"2024-04-26T16:20:55.494143+0000","last_peered":"2024-04-26T16:20:55.494143+0000","last_clean":"2024-04-26T16:20:55.494143+0000","last_became_active":"2024-04-26T16:17:35.065560+0000","last_became_peered":"2024-04-26T16:17:35.065560+0000","last_unstale":"2024-04-26T16:20:55.494143+0000","last_undegraded":"2024-04-26T16:20:55.494143+0000","last_fullsized":"2024-04-26T16:20:55.494143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.037900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.037900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.752700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500018+0000","last_change":"2024-04-26T16:20:03.150021+0000","last_active":"2024-04-26T16:20:55.500018+0000","last_peered":"2024-04-26T16:20:55.500018+0000","last_clean":"2024-04-26T16:20:55.500018+0000","last_became_active":"2024-04-26T16:17:33.058899+0000","last_became_peered":"2024-04-26T16:17:33.058899+0000","last_unstale":"2024-04-26T16:20:55.500018+0000","last_undegraded":"2024-04-26T16:20:55.500018+0000","last_fullsized":"2024-04-26T16:20:55.500018+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.149607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.149607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.024417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.181273+0000","last_change":"2024-04-26T16:20:03.947061+0000","last_active":"2024-04-26T16:20:56.181273+0000","last_peered":"2024-04-26T16:20:56.181273+0000","last_clean":"2024-04-26T16:20:56.181273+0000","last_became_active":"2024-04-26T16:17:29.016870+0000","last_became_peered":"2024-04-26T16:17:29.016870+0000","last_unstale":"2024-04-26T16:20:56.181273+0000","last_undegraded":"2024-04-26T16:20:56.181273+0000","last_fullsized":"2024-04-26T16:20:56.181273+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.946950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.302253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'11","reported_seq":123,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.180098+0000","last_change":"2024-04-26T16:20:04.348527+0000","last_active":"2024-04-26T16:20:56.180098+0000","last_peered":"2024-04-26T16:20:56.180098+0000","last_clean":"2024-04-26T16:20:56.180098+0000","last_became_active":"2024-04-26T16:17:31.041819+0000","last_became_peered":"2024-04-26T16:17:31.041819+0000","last_unstale":"2024-04-26T16:20:56.180098+0000","last_undegraded":"2024-04-26T16:20:56.180098+0000","last_fullsized":"2024-04-26T16:20:56.180098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:04.348299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.348299+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.592422+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174264+0000","last_change":"2024-04-26T16:20:02.714702+0000","last_active":"2024-04-26T16:20:56.174264+0000","last_peered":"2024-04-26T16:20:56.174264+0000","last_clean":"2024-04-26T16:20:56.174264+0000","last_became_active":"2024-04-26T16:17:35.087319+0000","last_became_peered":"2024-04-26T16:17:35.087319+0000","last_unstale":"2024-04-26T16:20:56.174264+0000","last_undegraded":"2024-04-26T16:20:56.174264+0000","last_fullsized":"2024-04-26T16:20:56.174264+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:20:02.714466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.714466+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.031920+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884428+0000","last_change":"2024-04-26T16:20:33.242024+0000","last_active":"2024-04-26T16:20:55.884428+0000","last_peered":"2024-04-26T16:20:55.884428+0000","last_clean":"2024-04-26T16:20:55.884428+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:20:55.884428+0000","last_undegraded":"2024-04-26T16:20:55.884428+0000","last_fullsized":"2024-04-26T16:20:55.884428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:33.241737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.241737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.846027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885701+0000","last_change":"2024-04-26T16:19:55.486245+0000","last_active":"2024-04-26T16:20:55.885701+0000","last_peered":"2024-04-26T16:20:55.885701+0000","last_clean":"2024-04-26T16:20:55.885701+0000","last_became_active":"2024-04-26T16:17:29.022163+0000","last_became_peered":"2024-04-26T16:17:29.022163+0000","last_unstale":"2024-04-26T16:20:55.885701+0000","last_undegraded":"2024-04-26T16:20:55.885701+0000","last_fullsized":"2024-04-26T16:20:55.885701+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:55.486024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:55.486024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:06.433329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"86'15","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.175915+0000","last_change":"2024-04-26T16:20:25.511190+0000","last_active":"2024-04-26T16:20:56.175915+0000","last_peered":"2024-04-26T16:20:56.175915+0000","last_clean":"2024-04-26T16:20:56.175915+0000","last_became_active":"2024-04-26T16:17:31.022500+0000","last_became_peered":"2024-04-26T16:17:31.022500+0000","last_unstale":"2024-04-26T16:20:56.175915+0000","last_undegraded":"2024-04-26T16:20:56.175915+0000","last_fullsized":"2024-04-26T16:20:56.175915+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:25.510941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.510941+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.088665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176276+0000","last_change":"2024-04-26T16:20:47.306391+0000","last_active":"2024-04-26T16:20:56.176276+0000","last_peered":"2024-04-26T16:20:56.176276+0000","last_clean":"2024-04-26T16:20:56.176276+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:20:56.176276+0000","last_undegraded":"2024-04-26T16:20:56.176276+0000","last_fullsized":"2024-04-26T16:20:56.176276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:47.306263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:47.306263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.778374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":101,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174835+0000","last_change":"2024-04-26T16:20:18.567822+0000","last_active":"2024-04-26T16:20:56.174835+0000","last_peered":"2024-04-26T16:20:56.174835+0000","last_clean":"2024-04-26T16:20:56.174835+0000","last_became_active":"2024-04-26T16:17:33.071659+0000","last_became_peered":"2024-04-26T16:17:33.071659+0000","last_unstale":"2024-04-26T16:20:56.174835+0000","last_undegraded":"2024-04-26T16:20:56.174835+0000","last_fullsized":"2024-04-26T16:20:56.174835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.567616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.567616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.293668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885119+0000","last_change":"2024-04-26T16:20:23.315306+0000","last_active":"2024-04-26T16:20:55.885119+0000","last_peered":"2024-04-26T16:20:55.885119+0000","last_clean":"2024-04-26T16:20:55.885119+0000","last_became_active":"2024-04-26T16:17:29.025659+0000","last_became_peered":"2024-04-26T16:17:29.025659+0000","last_unstale":"2024-04-26T16:20:55.885119+0000","last_undegraded":"2024-04-26T16:20:55.885119+0000","last_fullsized":"2024-04-26T16:20:55.885119+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.315099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.315099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.593677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'12","reported_seq":128,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.175473+0000","last_change":"2024-04-26T16:20:08.659388+0000","last_active":"2024-04-26T16:20:56.175473+0000","last_peered":"2024-04-26T16:20:56.175473+0000","last_clean":"2024-04-26T16:20:56.175473+0000","last_became_active":"2024-04-26T16:17:31.019223+0000","last_became_peered":"2024-04-26T16:17:31.019223+0000","last_unstale":"2024-04-26T16:20:56.175473+0000","last_undegraded":"2024-04-26T16:20:56.175473+0000","last_fullsized":"2024-04-26T16:20:56.175473+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:08.659153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.659153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.636156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.181361+0000","last_change":"2024-04-26T16:20:32.864786+0000","last_active":"2024-04-26T16:20:56.181361+0000","last_peered":"2024-04-26T16:20:56.181361+0000","last_clean":"2024-04-26T16:20:56.181361+0000","last_became_active":"2024-04-26T16:17:35.673178+0000","last_became_peered":"2024-04-26T16:17:35.673178+0000","last_unstale":"2024-04-26T16:20:56.181361+0000","last_undegraded":"2024-04-26T16:20:56.181361+0000","last_fullsized":"2024-04-26T16:20:56.181361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:32.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:32.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:00.660802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.495258+0000","last_change":"2024-04-26T16:20:26.931424+0000","last_active":"2024-04-26T16:20:55.495258+0000","last_peered":"2024-04-26T16:20:55.495258+0000","last_clean":"2024-04-26T16:20:55.495258+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:20:55.495258+0000","last_undegraded":"2024-04-26T16:20:55.495258+0000","last_fullsized":"2024-04-26T16:20:55.495258+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.931219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.931219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.617802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.111380+0000","last_change":"2024-04-26T16:20:59.111406+0000","last_active":"2024-04-26T16:20:59.111380+0000","last_peered":"2024-04-26T16:20:59.111380+0000","last_clean":"2024-04-26T16:20:59.111380+0000","last_became_active":"2024-04-26T16:17:29.005070+0000","last_became_peered":"2024-04-26T16:17:29.005070+0000","last_unstale":"2024-04-26T16:20:59.111380+0000","last_undegraded":"2024-04-26T16:20:59.111380+0000","last_fullsized":"2024-04-26T16:20:59.111380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:59.111294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:59.111294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:17.940312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"86'9","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494421+0000","last_change":"2024-04-26T16:20:53.797670+0000","last_active":"2024-04-26T16:20:55.494421+0000","last_peered":"2024-04-26T16:20:55.494421+0000","last_clean":"2024-04-26T16:20:55.494421+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:20:55.494421+0000","last_undegraded":"2024-04-26T16:20:55.494421+0000","last_fullsized":"2024-04-26T16:20:55.494421+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:53.797430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:53.797430+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:06.830917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496116+0000","last_change":"2024-04-26T16:20:15.936737+0000","last_active":"2024-04-26T16:20:55.496116+0000","last_peered":"2024-04-26T16:20:55.496116+0000","last_clean":"2024-04-26T16:20:55.496116+0000","last_became_active":"2024-04-26T16:17:35.674743+0000","last_became_peered":"2024-04-26T16:17:35.674743+0000","last_unstale":"2024-04-26T16:20:55.496116+0000","last_undegraded":"2024-04-26T16:20:55.496116+0000","last_fullsized":"2024-04-26T16:20:55.496116+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.936392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.456145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.501387+0000","last_change":"2024-04-26T16:20:26.994856+0000","last_active":"2024-04-26T16:20:55.501387+0000","last_peered":"2024-04-26T16:20:55.501387+0000","last_clean":"2024-04-26T16:20:55.501387+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:20:55.501387+0000","last_undegraded":"2024-04-26T16:20:55.501387+0000","last_fullsized":"2024-04-26T16:20:55.501387+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.994631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.994631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:56.773657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.089453+0000","last_change":"2024-04-26T16:20:08.830657+0000","last_active":"2024-04-26T16:20:56.089453+0000","last_peered":"2024-04-26T16:20:56.089453+0000","last_clean":"2024-04-26T16:20:56.089453+0000","last_became_active":"2024-04-26T16:17:29.004486+0000","last_became_peered":"2024-04-26T16:17:29.004486+0000","last_unstale":"2024-04-26T16:20:56.089453+0000","last_undegraded":"2024-04-26T16:20:56.089453+0000","last_fullsized":"2024-04-26T16:20:56.089453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.830446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.830446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.833491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"114'7","reported_seq":116,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.177433+0000","last_change":"2024-04-26T16:20:22.150870+0000","last_active":"2024-04-26T16:20:56.177433+0000","last_peered":"2024-04-26T16:20:56.177433+0000","last_clean":"2024-04-26T16:20:56.177433+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:20:56.177433+0000","last_undegraded":"2024-04-26T16:20:56.177433+0000","last_fullsized":"2024-04-26T16:20:56.177433+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_clean_scrub_stamp":"2024-04-26T16:20:22.150727+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:47.002699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.501965+0000","last_change":"2024-04-26T16:20:10.128158+0000","last_active":"2024-04-26T16:20:55.501965+0000","last_peered":"2024-04-26T16:20:55.501965+0000","last_clean":"2024-04-26T16:20:55.501965+0000","last_became_active":"2024-04-26T16:17:35.080236+0000","last_became_peered":"2024-04-26T16:17:35.080236+0000","last_unstale":"2024-04-26T16:20:55.501965+0000","last_undegraded":"2024-04-26T16:20:55.501965+0000","last_fullsized":"2024-04-26T16:20:55.501965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.127949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.127949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.505883+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.179428+0000","last_change":"2024-04-26T16:20:03.390713+0000","last_active":"2024-04-26T16:20:56.179428+0000","last_peered":"2024-04-26T16:20:56.179428+0000","last_clean":"2024-04-26T16:20:56.179428+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:20:56.179428+0000","last_undegraded":"2024-04-26T16:20:56.179428+0000","last_fullsized":"2024-04-26T16:20:56.179428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.390501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.390501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:06.150359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":110,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.177913+0000","last_change":"2024-04-26T16:20:53.091231+0000","last_active":"2024-04-26T16:20:56.177913+0000","last_peered":"2024-04-26T16:20:56.177913+0000","last_clean":"2024-04-26T16:20:56.177913+0000","last_became_active":"2024-04-26T16:17:28.999524+0000","last_became_peered":"2024-04-26T16:17:28.999524+0000","last_unstale":"2024-04-26T16:20:56.177913+0000","last_undegraded":"2024-04-26T16:20:56.177913+0000","last_fullsized":"2024-04-26T16:20:56.177913+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:20:53.090830+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:10.535270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"86'9","reported_seq":120,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176425+0000","last_change":"2024-04-26T16:20:34.899058+0000","last_active":"2024-04-26T16:20:56.176425+0000","last_peered":"2024-04-26T16:20:56.176425+0000","last_clean":"2024-04-26T16:20:56.176425+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:20:56.176425+0000","last_undegraded":"2024-04-26T16:20:56.176425+0000","last_fullsized":"2024-04-26T16:20:56.176425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:34.898819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:34.898819+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.454975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885910+0000","last_change":"2024-04-26T16:20:54.010267+0000","last_active":"2024-04-26T16:20:55.885910+0000","last_peered":"2024-04-26T16:20:55.885910+0000","last_clean":"2024-04-26T16:20:55.885910+0000","last_became_active":"2024-04-26T16:17:35.081916+0000","last_became_peered":"2024-04-26T16:17:35.081916+0000","last_unstale":"2024-04-26T16:20:55.885910+0000","last_undegraded":"2024-04-26T16:20:55.885910+0000","last_fullsized":"2024-04-26T16:20:55.885910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:20:54.009881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:54.009881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:02.400809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"86'11","reported_seq":157,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.247066+0000","last_change":"2024-04-26T16:19:59.878190+0000","last_active":"2024-04-26T16:20:59.247066+0000","last_peered":"2024-04-26T16:20:59.247066+0000","last_clean":"2024-04-26T16:20:59.247066+0000","last_became_active":"2024-04-26T16:17:33.061926+0000","last_became_peered":"2024-04-26T16:17:33.061926+0000","last_unstale":"2024-04-26T16:20:59.247066+0000","last_undegraded":"2024-04-26T16:20:59.247066+0000","last_fullsized":"2024-04-26T16:20:59.247066+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:59.877790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.877790+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.440948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":109,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174071+0000","last_change":"2024-04-26T16:19:58.724981+0000","last_active":"2024-04-26T16:20:56.174071+0000","last_peered":"2024-04-26T16:20:56.174071+0000","last_clean":"2024-04-26T16:20:56.174071+0000","last_became_active":"2024-04-26T16:17:29.025319+0000","last_became_peered":"2024-04-26T16:17:29.025319+0000","last_unstale":"2024-04-26T16:20:56.174071+0000","last_undegraded":"2024-04-26T16:20:56.174071+0000","last_fullsized":"2024-04-26T16:20:56.174071+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.724766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.724766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.832669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"118'22","reported_seq":138,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500319+0000","last_change":"2024-04-26T16:20:37.960013+0000","last_active":"2024-04-26T16:20:55.500319+0000","last_peered":"2024-04-26T16:20:55.500319+0000","last_clean":"2024-04-26T16:20:55.500319+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:20:55.500319+0000","last_undegraded":"2024-04-26T16:20:55.500319+0000","last_fullsized":"2024-04-26T16:20:55.500319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:20:37.959753+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:48.491354+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.180845+0000","last_change":"2024-04-26T16:20:35.919687+0000","last_active":"2024-04-26T16:20:56.180845+0000","last_peered":"2024-04-26T16:20:56.180845+0000","last_clean":"2024-04-26T16:20:56.180845+0000","last_became_active":"2024-04-26T16:17:35.677366+0000","last_became_peered":"2024-04-26T16:17:35.677366+0000","last_unstale":"2024-04-26T16:20:56.180845+0000","last_undegraded":"2024-04-26T16:20:56.180845+0000","last_fullsized":"2024-04-26T16:20:56.180845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.919459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.919459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.324456+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884761+0000","last_change":"2024-04-26T16:20:26.239452+0000","last_active":"2024-04-26T16:20:55.884761+0000","last_peered":"2024-04-26T16:20:55.884761+0000","last_clean":"2024-04-26T16:20:55.884761+0000","last_became_active":"2024-04-26T16:17:33.075775+0000","last_became_peered":"2024-04-26T16:17:33.075775+0000","last_unstale":"2024-04-26T16:20:55.884761+0000","last_undegraded":"2024-04-26T16:20:55.884761+0000","last_fullsized":"2024-04-26T16:20:55.884761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.239095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.239095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.234785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.090559+0000","last_change":"2024-04-26T16:20:02.934191+0000","last_active":"2024-04-26T16:20:56.090559+0000","last_peered":"2024-04-26T16:20:56.090559+0000","last_clean":"2024-04-26T16:20:56.090559+0000","last_became_active":"2024-04-26T16:17:29.018752+0000","last_became_peered":"2024-04-26T16:17:29.018752+0000","last_unstale":"2024-04-26T16:20:56.090559+0000","last_undegraded":"2024-04-26T16:20:56.090559+0000","last_fullsized":"2024-04-26T16:20:56.090559+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.933912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.933912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.658980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"86'9","reported_seq":121,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176147+0000","last_change":"2024-04-26T16:20:41.527249+0000","last_active":"2024-04-26T16:20:56.176147+0000","last_peered":"2024-04-26T16:20:56.176147+0000","last_clean":"2024-04-26T16:20:56.176147+0000","last_became_active":"2024-04-26T16:17:31.024113+0000","last_became_peered":"2024-04-26T16:17:31.024113+0000","last_unstale":"2024-04-26T16:20:56.176147+0000","last_undegraded":"2024-04-26T16:20:56.176147+0000","last_fullsized":"2024-04-26T16:20:56.176147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:41.527003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:41.527003+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:42.238560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884142+0000","last_change":"2024-04-26T16:20:51.060785+0000","last_active":"2024-04-26T16:20:55.884142+0000","last_peered":"2024-04-26T16:20:55.884142+0000","last_clean":"2024-04-26T16:20:55.884142+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:20:55.884142+0000","last_undegraded":"2024-04-26T16:20:55.884142+0000","last_fullsized":"2024-04-26T16:20:55.884142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:51.060550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:51.060550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:11.425128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.089477+0000","last_change":"2024-04-26T16:20:51.017702+0000","last_active":"2024-04-26T16:20:56.089477+0000","last_peered":"2024-04-26T16:20:56.089477+0000","last_clean":"2024-04-26T16:20:56.089477+0000","last_became_active":"2024-04-26T16:17:33.059705+0000","last_became_peered":"2024-04-26T16:17:33.059705+0000","last_unstale":"2024-04-26T16:20:56.089477+0000","last_undegraded":"2024-04-26T16:20:56.089477+0000","last_fullsized":"2024-04-26T16:20:56.089477+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:51.017392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:51.017392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:07.706485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":109,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176536+0000","last_change":"2024-04-26T16:20:19.575996+0000","last_active":"2024-04-26T16:20:56.176536+0000","last_peered":"2024-04-26T16:20:56.176536+0000","last_clean":"2024-04-26T16:20:56.176536+0000","last_became_active":"2024-04-26T16:17:29.022454+0000","last_became_peered":"2024-04-26T16:17:29.022454+0000","last_unstale":"2024-04-26T16:20:56.176536+0000","last_undegraded":"2024-04-26T16:20:56.176536+0000","last_fullsized":"2024-04-26T16:20:56.176536+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:19.575787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.575787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.873083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"86'6","reported_seq":113,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.179958+0000","last_change":"2024-04-26T16:20:21.162238+0000","last_active":"2024-04-26T16:20:56.179958+0000","last_peered":"2024-04-26T16:20:56.179958+0000","last_clean":"2024-04-26T16:20:56.179958+0000","last_became_active":"2024-04-26T16:17:31.053359+0000","last_became_peered":"2024-04-26T16:17:31.053359+0000","last_unstale":"2024-04-26T16:20:56.179958+0000","last_undegraded":"2024-04-26T16:20:56.179958+0000","last_fullsized":"2024-04-26T16:20:56.179958+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'6","last_scrub_stamp":"2024-04-26T16:20:21.161995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.161995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.190751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.177444+0000","last_change":"2024-04-26T16:20:08.001062+0000","last_active":"2024-04-26T16:20:56.177444+0000","last_peered":"2024-04-26T16:20:56.177444+0000","last_clean":"2024-04-26T16:20:56.177444+0000","last_became_active":"2024-04-26T16:17:35.094615+0000","last_became_peered":"2024-04-26T16:17:35.094615+0000","last_unstale":"2024-04-26T16:20:56.177444+0000","last_undegraded":"2024-04-26T16:20:56.177444+0000","last_fullsized":"2024-04-26T16:20:56.177444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.000834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.000834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.457279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.501393+0000","last_change":"2024-04-26T16:20:40.011955+0000","last_active":"2024-04-26T16:20:55.501393+0000","last_peered":"2024-04-26T16:20:55.501393+0000","last_clean":"2024-04-26T16:20:55.501393+0000","last_became_active":"2024-04-26T16:17:33.048202+0000","last_became_peered":"2024-04-26T16:17:33.048202+0000","last_unstale":"2024-04-26T16:20:55.501393+0000","last_undegraded":"2024-04-26T16:20:55.501393+0000","last_fullsized":"2024-04-26T16:20:55.501393+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:40.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:40.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:59.380129+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":110,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500673+0000","last_change":"2024-04-26T16:20:44.976307+0000","last_active":"2024-04-26T16:20:55.500673+0000","last_peered":"2024-04-26T16:20:55.500673+0000","last_clean":"2024-04-26T16:20:55.500673+0000","last_became_active":"2024-04-26T16:17:29.011346+0000","last_became_peered":"2024-04-26T16:17:29.011346+0000","last_unstale":"2024-04-26T16:20:55.500673+0000","last_undegraded":"2024-04-26T16:20:55.500673+0000","last_fullsized":"2024-04-26T16:20:55.500673+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:44.975982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:44.975982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:09.024600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"86'10","reported_seq":121,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.495696+0000","last_change":"2024-04-26T16:20:23.996060+0000","last_active":"2024-04-26T16:20:55.495696+0000","last_peered":"2024-04-26T16:20:55.495696+0000","last_clean":"2024-04-26T16:20:55.495696+0000","last_became_active":"2024-04-26T16:17:31.021476+0000","last_became_peered":"2024-04-26T16:17:31.021476+0000","last_unstale":"2024-04-26T16:20:55.495696+0000","last_undegraded":"2024-04-26T16:20:55.495696+0000","last_fullsized":"2024-04-26T16:20:55.495696+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:23.995817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.995817+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.162295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.175695+0000","last_change":"2024-04-26T16:20:25.238939+0000","last_active":"2024-04-26T16:20:56.175695+0000","last_peered":"2024-04-26T16:20:56.175695+0000","last_clean":"2024-04-26T16:20:56.175695+0000","last_became_active":"2024-04-26T16:17:35.078105+0000","last_became_peered":"2024-04-26T16:17:35.078105+0000","last_unstale":"2024-04-26T16:20:56.175695+0000","last_undegraded":"2024-04-26T16:20:56.175695+0000","last_fullsized":"2024-04-26T16:20:56.175695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:25.238710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:06.659791+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.238710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:31.861824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.180668+0000","last_change":"2024-04-26T16:20:07.028942+0000","last_active":"2024-04-26T16:20:56.180668+0000","last_peered":"2024-04-26T16:20:56.180668+0000","last_clean":"2024-04-26T16:20:56.180668+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:20:56.180668+0000","last_undegraded":"2024-04-26T16:20:56.180668+0000","last_fullsized":"2024-04-26T16:20:56.180668+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:07.028724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:07.028724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.581251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.090154+0000","last_change":"2024-04-26T16:20:12.898651+0000","last_active":"2024-04-26T16:20:56.090154+0000","last_peered":"2024-04-26T16:20:56.090154+0000","last_clean":"2024-04-26T16:20:56.090154+0000","last_became_active":"2024-04-26T16:17:29.013639+0000","last_became_peered":"2024-04-26T16:17:29.013639+0000","last_unstale":"2024-04-26T16:20:56.090154+0000","last_undegraded":"2024-04-26T16:20:56.090154+0000","last_fullsized":"2024-04-26T16:20:56.090154+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:20:12.898432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.131422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"86'9","reported_seq":120,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885935+0000","last_change":"2024-04-26T16:20:09.306205+0000","last_active":"2024-04-26T16:20:55.885935+0000","last_peered":"2024-04-26T16:20:55.885935+0000","last_clean":"2024-04-26T16:20:55.885935+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:20:55.885935+0000","last_undegraded":"2024-04-26T16:20:55.885935+0000","last_fullsized":"2024-04-26T16:20:55.885935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:09.305987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.305987+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.952773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496573+0000","last_change":"2024-04-26T16:20:43.860874+0000","last_active":"2024-04-26T16:20:55.496573+0000","last_peered":"2024-04-26T16:20:55.496573+0000","last_clean":"2024-04-26T16:20:55.496573+0000","last_became_active":"2024-04-26T16:17:35.070054+0000","last_became_peered":"2024-04-26T16:17:35.070054+0000","last_unstale":"2024-04-26T16:20:55.496573+0000","last_undegraded":"2024-04-26T16:20:55.496573+0000","last_fullsized":"2024-04-26T16:20:55.496573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:43.860659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:43.860659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:59.070145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":146,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.248041+0000","last_change":"2024-04-26T16:19:26.961234+0000","last_active":"2024-04-26T16:20:59.248041+0000","last_peered":"2024-04-26T16:20:59.248041+0000","last_clean":"2024-04-26T16:20:59.248041+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:20:59.248041+0000","last_undegraded":"2024-04-26T16:20:59.248041+0000","last_fullsized":"2024-04-26T16:20:59.248041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:26.960994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:26.960994+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.177298+0000","last_change":"2024-04-26T16:20:15.241109+0000","last_active":"2024-04-26T16:20:56.177298+0000","last_peered":"2024-04-26T16:20:56.177298+0000","last_clean":"2024-04-26T16:20:56.177298+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:20:56.177298+0000","last_undegraded":"2024-04-26T16:20:56.177298+0000","last_fullsized":"2024-04-26T16:20:56.177298+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.241014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.241014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.603907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"86'9","reported_seq":120,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176509+0000","last_change":"2024-04-26T16:20:00.355659+0000","last_active":"2024-04-26T16:20:56.176509+0000","last_peered":"2024-04-26T16:20:56.176509+0000","last_clean":"2024-04-26T16:20:56.176509+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:20:56.176509+0000","last_undegraded":"2024-04-26T16:20:56.176509+0000","last_fullsized":"2024-04-26T16:20:56.176509+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:00.355248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:00.355248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:00.966364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.179038+0000","last_change":"2024-04-26T16:20:38.325004+0000","last_active":"2024-04-26T16:20:56.179038+0000","last_peered":"2024-04-26T16:20:56.179038+0000","last_clean":"2024-04-26T16:20:56.179038+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:20:56.179038+0000","last_undegraded":"2024-04-26T16:20:56.179038+0000","last_fullsized":"2024-04-26T16:20:56.179038+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:38.324618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:38.324618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.422515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"101'12","reported_seq":164,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.519147+0000","last_change":"2024-04-26T16:20:59.519147+0000","last_active":"2024-04-26T16:20:59.519147+0000","last_peered":"2024-04-26T16:20:59.519147+0000","last_clean":"2024-04-26T16:20:59.519147+0000","last_became_active":"2024-04-26T16:17:33.062000+0000","last_became_peered":"2024-04-26T16:17:33.062000+0000","last_unstale":"2024-04-26T16:20:59.519147+0000","last_undegraded":"2024-04-26T16:20:59.519147+0000","last_fullsized":"2024-04-26T16:20:59.519147+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-26T16:20:59.518904+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:20:59.518904+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.666407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":109,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174409+0000","last_change":"2024-04-26T16:20:05.684861+0000","last_active":"2024-04-26T16:20:56.174409+0000","last_peered":"2024-04-26T16:20:56.174409+0000","last_clean":"2024-04-26T16:20:56.174409+0000","last_became_active":"2024-04-26T16:17:29.025290+0000","last_became_peered":"2024-04-26T16:17:29.025290+0000","last_unstale":"2024-04-26T16:20:56.174409+0000","last_undegraded":"2024-04-26T16:20:56.174409+0000","last_fullsized":"2024-04-26T16:20:56.174409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.684621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.684621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:12.223118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'11","reported_seq":131,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:04.076069+0000","last_change":"2024-04-26T16:21:04.076152+0000","last_active":"2024-04-26T16:21:04.076069+0000","last_peered":"2024-04-26T16:21:04.076069+0000","last_clean":"2024-04-26T16:21:04.076069+0000","last_became_active":"2024-04-26T16:17:31.034570+0000","last_became_peered":"2024-04-26T16:17:31.034570+0000","last_unstale":"2024-04-26T16:21:04.076069+0000","last_undegraded":"2024-04-26T16:21:04.076069+0000","last_fullsized":"2024-04-26T16:21:04.076069+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:21:04.075909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:21:04.075909+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:23.124588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:00.996774+0000","last_change":"2024-04-26T16:21:00.996835+0000","last_active":"2024-04-26T16:21:00.996774+0000","last_peered":"2024-04-26T16:21:00.996774+0000","last_clean":"2024-04-26T16:21:00.996774+0000","last_became_active":"2024-04-26T16:17:35.671754+0000","last_became_peered":"2024-04-26T16:17:35.671754+0000","last_unstale":"2024-04-26T16:21:00.996774+0000","last_undegraded":"2024-04-26T16:21:00.996774+0000","last_fullsized":"2024-04-26T16:21:00.996774+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:00.996593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:21:00.996593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:11.543878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":101,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176127+0000","last_change":"2024-04-26T16:20:49.524102+0000","last_active":"2024-04-26T16:20:56.176127+0000","last_peered":"2024-04-26T16:20:56.176127+0000","last_clean":"2024-04-26T16:20:56.176127+0000","last_became_active":"2024-04-26T16:17:33.050011+0000","last_became_peered":"2024-04-26T16:17:33.050011+0000","last_unstale":"2024-04-26T16:20:56.176127+0000","last_undegraded":"2024-04-26T16:20:56.176127+0000","last_fullsized":"2024-04-26T16:20:56.176127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:49.523871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:49.523871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:13.761815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.089518+0000","last_change":"2024-04-26T16:20:44.944706+0000","last_active":"2024-04-26T16:20:56.089518+0000","last_peered":"2024-04-26T16:20:56.089518+0000","last_clean":"2024-04-26T16:20:56.089518+0000","last_became_active":"2024-04-26T16:17:28.999634+0000","last_became_peered":"2024-04-26T16:17:28.999634+0000","last_unstale":"2024-04-26T16:20:56.089518+0000","last_undegraded":"2024-04-26T16:20:56.089518+0000","last_fullsized":"2024-04-26T16:20:56.089518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:44.944463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:44.944463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:09.941460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'4","reported_seq":110,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.089197+0000","last_change":"2024-04-26T16:20:26.899539+0000","last_active":"2024-04-26T16:20:56.089197+0000","last_peered":"2024-04-26T16:20:56.089197+0000","last_clean":"2024-04-26T16:20:56.089197+0000","last_became_active":"2024-04-26T16:17:31.026037+0000","last_became_peered":"2024-04-26T16:17:31.026037+0000","last_unstale":"2024-04-26T16:20:56.089197+0000","last_undegraded":"2024-04-26T16:20:56.089197+0000","last_fullsized":"2024-04-26T16:20:56.089197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'4","last_scrub_stamp":"2024-04-26T16:20:26.899299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.899299+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.728642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:02.458943+0000","last_change":"2024-04-26T16:21:02.459002+0000","last_active":"2024-04-26T16:21:02.458943+0000","last_peered":"2024-04-26T16:21:02.458943+0000","last_clean":"2024-04-26T16:21:02.458943+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:21:02.458943+0000","last_undegraded":"2024-04-26T16:21:02.458943+0000","last_fullsized":"2024-04-26T16:21:02.458943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:02.458777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:21:02.458777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:29.849746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.181430+0000","last_change":"2024-04-26T16:20:33.855455+0000","last_active":"2024-04-26T16:20:56.181430+0000","last_peered":"2024-04-26T16:20:56.181430+0000","last_clean":"2024-04-26T16:20:56.181430+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:20:56.181430+0000","last_undegraded":"2024-04-26T16:20:56.181430+0000","last_fullsized":"2024-04-26T16:20:56.181430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:33.855240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.855240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.904032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496169+0000","last_change":"2024-04-26T16:20:27.565849+0000","last_active":"2024-04-26T16:20:55.496169+0000","last_peered":"2024-04-26T16:20:55.496169+0000","last_clean":"2024-04-26T16:20:55.496169+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:20:55.496169+0000","last_undegraded":"2024-04-26T16:20:55.496169+0000","last_fullsized":"2024-04-26T16:20:55.496169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:27.565529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:27.565529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.357713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"86'11","reported_seq":123,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884156+0000","last_change":"2024-04-26T16:20:14.278241+0000","last_active":"2024-04-26T16:20:55.884156+0000","last_peered":"2024-04-26T16:20:55.884156+0000","last_clean":"2024-04-26T16:20:55.884156+0000","last_became_active":"2024-04-26T16:17:31.033294+0000","last_became_peered":"2024-04-26T16:17:31.033294+0000","last_unstale":"2024-04-26T16:20:55.884156+0000","last_undegraded":"2024-04-26T16:20:55.884156+0000","last_fullsized":"2024-04-26T16:20:55.884156+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:14.278005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.278005+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.458748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:04.005592+0000","last_change":"2024-04-26T16:21:04.005652+0000","last_active":"2024-04-26T16:21:04.005592+0000","last_peered":"2024-04-26T16:21:04.005592+0000","last_clean":"2024-04-26T16:21:04.005592+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:21:04.005592+0000","last_undegraded":"2024-04-26T16:21:04.005592+0000","last_fullsized":"2024-04-26T16:21:04.005592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:04.005438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:21:04.005438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:32.525159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496172+0000","last_change":"2024-04-26T16:20:13.890462+0000","last_active":"2024-04-26T16:20:55.496172+0000","last_peered":"2024-04-26T16:20:55.496172+0000","last_clean":"2024-04-26T16:20:55.496172+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:20:55.496172+0000","last_undegraded":"2024-04-26T16:20:55.496172+0000","last_fullsized":"2024-04-26T16:20:55.496172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:20:13.890111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.890111+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.062559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"86'11","reported_seq":123,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885744+0000","last_change":"2024-04-26T16:19:45.378650+0000","last_active":"2024-04-26T16:20:55.885744+0000","last_peered":"2024-04-26T16:20:55.885744+0000","last_clean":"2024-04-26T16:20:55.885744+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:20:55.885744+0000","last_undegraded":"2024-04-26T16:20:55.885744+0000","last_fullsized":"2024-04-26T16:20:55.885744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:45.378259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:45.378259+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.495964+0000","last_change":"2024-04-26T16:20:14.406999+0000","last_active":"2024-04-26T16:20:55.495964+0000","last_peered":"2024-04-26T16:20:55.495964+0000","last_clean":"2024-04-26T16:20:55.495964+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:20:55.495964+0000","last_undegraded":"2024-04-26T16:20:55.495964+0000","last_fullsized":"2024-04-26T16:20:55.495964+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.406805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.406805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.652307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176010+0000","last_change":"2024-04-26T16:20:32.257239+0000","last_active":"2024-04-26T16:20:56.176010+0000","last_peered":"2024-04-26T16:20:56.176010+0000","last_clean":"2024-04-26T16:20:56.176010+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:20:56.176010+0000","last_undegraded":"2024-04-26T16:20:56.176010+0000","last_fullsized":"2024-04-26T16:20:56.176010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:32.256998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:32.256998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.610172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":110,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.501172+0000","last_change":"2024-04-26T16:20:38.996311+0000","last_active":"2024-04-26T16:20:55.501172+0000","last_peered":"2024-04-26T16:20:55.501172+0000","last_clean":"2024-04-26T16:20:55.501172+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:20:55.501172+0000","last_undegraded":"2024-04-26T16:20:55.501172+0000","last_fullsized":"2024-04-26T16:20:55.501172+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:38.996095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:38.996095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:47.489693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"86'16","reported_seq":171,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.247690+0000","last_change":"2024-04-26T16:20:21.529761+0000","last_active":"2024-04-26T16:20:59.247690+0000","last_peered":"2024-04-26T16:20:59.247690+0000","last_clean":"2024-04-26T16:20:59.247690+0000","last_became_active":"2024-04-26T16:17:31.040723+0000","last_became_peered":"2024-04-26T16:17:31.040723+0000","last_unstale":"2024-04-26T16:20:59.247690+0000","last_undegraded":"2024-04-26T16:20:59.247690+0000","last_fullsized":"2024-04-26T16:20:59.247690+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'16","last_scrub_stamp":"2024-04-26T16:20:21.529513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.529513+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.582915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":113,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.501222+0000","last_change":"2024-04-26T16:19:59.129440+0000","last_active":"2024-04-26T16:20:55.501222+0000","last_peered":"2024-04-26T16:20:55.501222+0000","last_clean":"2024-04-26T16:20:55.501222+0000","last_became_active":"2024-04-26T16:17:32.046437+0000","last_became_peered":"2024-04-26T16:17:32.046437+0000","last_unstale":"2024-04-26T16:20:55.501222+0000","last_undegraded":"2024-04-26T16:20:55.501222+0000","last_fullsized":"2024-04-26T16:20:55.501222+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:19:59.129167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.129167+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.401811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"86'11","reported_seq":157,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.247864+0000","last_change":"2024-04-26T16:19:50.407803+0000","last_active":"2024-04-26T16:20:59.247864+0000","last_peered":"2024-04-26T16:20:59.247864+0000","last_clean":"2024-04-26T16:20:59.247864+0000","last_became_active":"2024-04-26T16:17:33.069020+0000","last_became_peered":"2024-04-26T16:17:33.069020+0000","last_unstale":"2024-04-26T16:20:59.247864+0000","last_undegraded":"2024-04-26T16:20:59.247864+0000","last_fullsized":"2024-04-26T16:20:59.247864+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:50.407700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:50.407700+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.181198+0000","last_change":"2024-04-26T16:20:09.046364+0000","last_active":"2024-04-26T16:20:56.181198+0000","last_peered":"2024-04-26T16:20:56.181198+0000","last_clean":"2024-04-26T16:20:56.181198+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:20:56.181198+0000","last_undegraded":"2024-04-26T16:20:56.181198+0000","last_fullsized":"2024-04-26T16:20:56.181198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.045989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.045989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.237368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":109,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174549+0000","last_change":"2024-04-26T16:20:13.525422+0000","last_active":"2024-04-26T16:20:56.174549+0000","last_peered":"2024-04-26T16:20:56.174549+0000","last_clean":"2024-04-26T16:20:56.174549+0000","last_became_active":"2024-04-26T16:17:29.022734+0000","last_became_peered":"2024-04-26T16:17:29.022734+0000","last_unstale":"2024-04-26T16:20:56.174549+0000","last_undegraded":"2024-04-26T16:20:56.174549+0000","last_fullsized":"2024-04-26T16:20:56.174549+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:13.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.525151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.383572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'27","reported_seq":150,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494541+0000","last_change":"2024-04-26T16:20:19.982781+0000","last_active":"2024-04-26T16:20:55.494541+0000","last_peered":"2024-04-26T16:20:55.494541+0000","last_clean":"2024-04-26T16:20:55.494541+0000","last_became_active":"2024-04-26T16:17:31.030904+0000","last_became_peered":"2024-04-26T16:17:31.030904+0000","last_unstale":"2024-04-26T16:20:55.494541+0000","last_undegraded":"2024-04-26T16:20:55.494541+0000","last_fullsized":"2024-04-26T16:20:55.494541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'27","last_scrub_stamp":"2024-04-26T16:20:19.982389+0000","last_deep_scrub":"103'27","last_deep_scrub_stamp":"2024-04-26T16:18:40.947097+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.982389+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.646180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.178948+0000","last_change":"2024-04-26T16:20:24.200758+0000","last_active":"2024-04-26T16:20:56.178948+0000","last_peered":"2024-04-26T16:20:56.178948+0000","last_clean":"2024-04-26T16:20:56.178948+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:20:56.178948+0000","last_undegraded":"2024-04-26T16:20:56.178948+0000","last_fullsized":"2024-04-26T16:20:56.178948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:24.200448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.200448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.909147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885385+0000","last_change":"2024-04-26T16:20:16.270543+0000","last_active":"2024-04-26T16:20:55.885385+0000","last_peered":"2024-04-26T16:20:55.885385+0000","last_clean":"2024-04-26T16:20:55.885385+0000","last_became_active":"2024-04-26T16:17:35.670585+0000","last_became_peered":"2024-04-26T16:17:35.670585+0000","last_unstale":"2024-04-26T16:20:55.885385+0000","last_undegraded":"2024-04-26T16:20:55.885385+0000","last_fullsized":"2024-04-26T16:20:55.885385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:16.270321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:16.270321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.978435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":146,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.499846+0000","last_change":"2024-04-26T16:19:43.114960+0000","last_active":"2024-04-26T16:20:55.499846+0000","last_peered":"2024-04-26T16:20:55.499846+0000","last_clean":"2024-04-26T16:20:55.499846+0000","last_became_active":"2024-04-26T16:17:31.023021+0000","last_became_peered":"2024-04-26T16:17:31.023021+0000","last_unstale":"2024-04-26T16:20:55.499846+0000","last_undegraded":"2024-04-26T16:20:55.499846+0000","last_fullsized":"2024-04-26T16:20:55.499846+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_clean_scrub_stamp":"2024-04-26T16:19:43.114691+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.523228+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496227+0000","last_change":"2024-04-26T16:19:52.393976+0000","last_active":"2024-04-26T16:20:55.496227+0000","last_peered":"2024-04-26T16:20:55.496227+0000","last_clean":"2024-04-26T16:20:55.496227+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:20:55.496227+0000","last_undegraded":"2024-04-26T16:20:55.496227+0000","last_fullsized":"2024-04-26T16:20:55.496227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:52.393754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:19:52.393754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.078807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.497025+0000","last_change":"2024-04-26T16:19:59.359204+0000","last_active":"2024-04-26T16:20:55.497025+0000","last_peered":"2024-04-26T16:20:55.497025+0000","last_clean":"2024-04-26T16:20:55.497025+0000","last_became_active":"2024-04-26T16:17:33.066780+0000","last_became_peered":"2024-04-26T16:17:33.066780+0000","last_unstale":"2024-04-26T16:20:55.497025+0000","last_undegraded":"2024-04-26T16:20:55.497025+0000","last_fullsized":"2024-04-26T16:20:55.497025+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:59.358789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.358789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.479227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":97,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176712+0000","last_change":"2024-04-26T16:20:51.461873+0000","last_active":"2024-04-26T16:20:56.176712+0000","last_peered":"2024-04-26T16:20:56.176712+0000","last_clean":"2024-04-26T16:20:56.176712+0000","last_became_active":"2024-04-26T16:17:35.087291+0000","last_became_peered":"2024-04-26T16:17:35.087291+0000","last_unstale":"2024-04-26T16:20:56.176712+0000","last_undegraded":"2024-04-26T16:20:56.176712+0000","last_fullsized":"2024-04-26T16:20:56.176712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:51.461656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:51.461656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.437880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'10","reported_seq":119,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.178675+0000","last_change":"2024-04-26T16:20:04.994840+0000","last_active":"2024-04-26T16:20:56.178675+0000","last_peered":"2024-04-26T16:20:56.178675+0000","last_clean":"2024-04-26T16:20:56.178675+0000","last_became_active":"2024-04-26T16:17:31.048785+0000","last_became_peered":"2024-04-26T16:17:31.048785+0000","last_unstale":"2024-04-26T16:20:56.178675+0000","last_undegraded":"2024-04-26T16:20:56.178675+0000","last_fullsized":"2024-04-26T16:20:56.178675+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:04.994521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.994521+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:33.785270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":109,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174764+0000","last_change":"2024-04-26T16:20:06.715686+0000","last_active":"2024-04-26T16:20:56.174764+0000","last_peered":"2024-04-26T16:20:56.174764+0000","last_clean":"2024-04-26T16:20:56.174764+0000","last_became_active":"2024-04-26T16:17:29.013835+0000","last_became_peered":"2024-04-26T16:17:29.013835+0000","last_unstale":"2024-04-26T16:20:56.174764+0000","last_undegraded":"2024-04-26T16:20:56.174764+0000","last_fullsized":"2024-04-26T16:20:56.174764+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:06.715447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:06.715447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.617561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":388,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884105+0000","last_change":"2024-04-26T16:19:42.393827+0000","last_active":"2024-04-26T16:20:55.884105+0000","last_peered":"2024-04-26T16:20:55.884105+0000","last_clean":"2024-04-26T16:20:55.884105+0000","last_became_active":"2024-04-26T16:16:50.925983+0000","last_became_peered":"2024-04-26T16:16:50.925983+0000","last_unstale":"2024-04-26T16:20:55.884105+0000","last_undegraded":"2024-04-26T16:20:55.884105+0000","last_fullsized":"2024-04-26T16:20:55.884105+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:19:42.393587+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:19:42.393587+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:05.978678+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885618+0000","last_change":"2024-04-26T16:20:52.080952+0000","last_active":"2024-04-26T16:20:55.885618+0000","last_peered":"2024-04-26T16:20:55.885618+0000","last_clean":"2024-04-26T16:20:55.885618+0000","last_became_active":"2024-04-26T16:17:33.061816+0000","last_became_peered":"2024-04-26T16:17:33.061816+0000","last_unstale":"2024-04-26T16:20:55.885618+0000","last_undegraded":"2024-04-26T16:20:55.885618+0000","last_fullsized":"2024-04-26T16:20:55.885618+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:20:52.080653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.193004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500437+0000","last_change":"2024-04-26T16:20:47.022860+0000","last_active":"2024-04-26T16:20:55.500437+0000","last_peered":"2024-04-26T16:20:55.500437+0000","last_clean":"2024-04-26T16:20:55.500437+0000","last_became_active":"2024-04-26T16:17:35.090592+0000","last_became_peered":"2024-04-26T16:17:35.090592+0000","last_unstale":"2024-04-26T16:20:55.500437+0000","last_undegraded":"2024-04-26T16:20:55.500437+0000","last_fullsized":"2024-04-26T16:20:55.500437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:20:47.022618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:12.217970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"112'21","reported_seq":140,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:02.316506+0000","last_change":"2024-04-26T16:21:02.316590+0000","last_active":"2024-04-26T16:21:02.316506+0000","last_peered":"2024-04-26T16:21:02.316506+0000","last_clean":"2024-04-26T16:21:02.316506+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:21:02.316506+0000","last_undegraded":"2024-04-26T16:21:02.316506+0000","last_fullsized":"2024-04-26T16:21:02.316506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'21","last_scrub_stamp":"2024-04-26T16:21:02.316340+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:21:02.316340+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.820401+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.883947+0000","last_change":"2024-04-26T16:20:30.226306+0000","last_active":"2024-04-26T16:20:55.883947+0000","last_peered":"2024-04-26T16:20:55.883947+0000","last_clean":"2024-04-26T16:20:55.883947+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:20:55.883947+0000","last_undegraded":"2024-04-26T16:20:55.883947+0000","last_fullsized":"2024-04-26T16:20:55.883947+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:30.226091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:30.226091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.774441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":101,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176567+0000","last_change":"2024-04-26T16:20:35.552710+0000","last_active":"2024-04-26T16:20:56.176567+0000","last_peered":"2024-04-26T16:20:56.176567+0000","last_clean":"2024-04-26T16:20:56.176567+0000","last_became_active":"2024-04-26T16:17:33.046107+0000","last_became_peered":"2024-04-26T16:17:33.046107+0000","last_unstale":"2024-04-26T16:20:56.176567+0000","last_undegraded":"2024-04-26T16:20:56.176567+0000","last_fullsized":"2024-04-26T16:20:56.176567+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.552498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.552498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.056973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.499706+0000","last_change":"2024-04-26T16:20:15.021647+0000","last_active":"2024-04-26T16:20:55.499706+0000","last_peered":"2024-04-26T16:20:55.499706+0000","last_clean":"2024-04-26T16:20:55.499706+0000","last_became_active":"2024-04-26T16:17:35.673254+0000","last_became_peered":"2024-04-26T16:17:35.673254+0000","last_unstale":"2024-04-26T16:20:55.499706+0000","last_undegraded":"2024-04-26T16:20:55.499706+0000","last_fullsized":"2024-04-26T16:20:55.499706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.021515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.021515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.753340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":116,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885431+0000","last_change":"2024-04-26T16:20:24.308841+0000","last_active":"2024-04-26T16:20:55.885431+0000","last_peered":"2024-04-26T16:20:55.885431+0000","last_clean":"2024-04-26T16:20:55.885431+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:20:55.885431+0000","last_undegraded":"2024-04-26T16:20:55.885431+0000","last_fullsized":"2024-04-26T16:20:55.885431+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:24.308595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.308595+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:53.172263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"107'40","reported_seq":210,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.248286+0000","last_change":"2024-04-26T16:20:25.051639+0000","last_active":"2024-04-26T16:20:59.248286+0000","last_peered":"2024-04-26T16:20:59.248286+0000","last_clean":"2024-04-26T16:20:59.248286+0000","last_became_active":"2024-04-26T16:17:31.024426+0000","last_became_peered":"2024-04-26T16:17:31.024426+0000","last_unstale":"2024-04-26T16:20:59.248286+0000","last_undegraded":"2024-04-26T16:20:59.248286+0000","last_fullsized":"2024-04-26T16:20:59.248286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'40","last_scrub_stamp":"2024-04-26T16:20:25.051382+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:18:56.138611+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.051382+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.857786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.089482+0000","last_change":"2024-04-26T16:20:18.962156+0000","last_active":"2024-04-26T16:20:56.089482+0000","last_peered":"2024-04-26T16:20:56.089482+0000","last_clean":"2024-04-26T16:20:56.089482+0000","last_became_active":"2024-04-26T16:17:33.057789+0000","last_became_peered":"2024-04-26T16:17:33.057789+0000","last_unstale":"2024-04-26T16:20:56.089482+0000","last_undegraded":"2024-04-26T16:20:56.089482+0000","last_fullsized":"2024-04-26T16:20:56.089482+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.961959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:53.089835+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.961959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.337483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494332+0000","last_change":"2024-04-26T16:19:57.996481+0000","last_active":"2024-04-26T16:20:55.494332+0000","last_peered":"2024-04-26T16:20:55.494332+0000","last_clean":"2024-04-26T16:20:55.494332+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:20:55.494332+0000","last_undegraded":"2024-04-26T16:20:55.494332+0000","last_fullsized":"2024-04-26T16:20:55.494332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:57.996170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:19:57.996170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.461563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":112,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.502087+0000","last_change":"2024-04-26T16:20:18.047229+0000","last_active":"2024-04-26T16:20:55.502087+0000","last_peered":"2024-04-26T16:20:55.502087+0000","last_clean":"2024-04-26T16:20:55.502087+0000","last_became_active":"2024-04-26T16:17:29.004853+0000","last_became_peered":"2024-04-26T16:17:29.004853+0000","last_unstale":"2024-04-26T16:20:55.502087+0000","last_undegraded":"2024-04-26T16:20:55.502087+0000","last_fullsized":"2024-04-26T16:20:55.502087+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.047049+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.615066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"86'13","reported_seq":123,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.181525+0000","last_change":"2024-04-26T16:19:19.940285+0000","last_active":"2024-04-26T16:20:56.181525+0000","last_peered":"2024-04-26T16:20:56.181525+0000","last_clean":"2024-04-26T16:20:56.181525+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:20:56.181525+0000","last_undegraded":"2024-04-26T16:20:56.181525+0000","last_fullsized":"2024-04-26T16:20:56.181525+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'13","last_scrub_stamp":"2024-04-26T16:19:19.939903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:19.939903+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":309,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:04.879624+0000","last_change":"2024-04-26T16:20:11.073651+0000","last_active":"2024-04-26T16:21:04.879624+0000","last_peered":"2024-04-26T16:21:04.879624+0000","last_clean":"2024-04-26T16:21:04.879624+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:21:04.879624+0000","last_undegraded":"2024-04-26T16:21:04.879624+0000","last_fullsized":"2024-04-26T16:21:04.879624+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:20:11.073395+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:19:04.959471+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.073395+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.691968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":197,"num_read_kb":192,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494024+0000","last_change":"2024-04-26T16:20:21.947248+0000","last_active":"2024-04-26T16:20:55.494024+0000","last_peered":"2024-04-26T16:20:55.494024+0000","last_clean":"2024-04-26T16:20:55.494024+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:20:55.494024+0000","last_undegraded":"2024-04-26T16:20:55.494024+0000","last_fullsized":"2024-04-26T16:20:55.494024+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.946995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.814368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.883605+0000","last_change":"2024-04-26T16:20:17.304956+0000","last_active":"2024-04-26T16:20:55.883605+0000","last_peered":"2024-04-26T16:20:55.883605+0000","last_clean":"2024-04-26T16:20:55.883605+0000","last_became_active":"2024-04-26T16:17:35.670672+0000","last_became_peered":"2024-04-26T16:17:35.670672+0000","last_unstale":"2024-04-26T16:20:55.883605+0000","last_undegraded":"2024-04-26T16:20:55.883605+0000","last_fullsized":"2024-04-26T16:20:55.883605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.304722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.304722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.500725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.090578+0000","last_change":"2024-04-26T16:19:58.917701+0000","last_active":"2024-04-26T16:20:56.090578+0000","last_peered":"2024-04-26T16:20:56.090578+0000","last_clean":"2024-04-26T16:20:56.090578+0000","last_became_active":"2024-04-26T16:17:29.006985+0000","last_became_peered":"2024-04-26T16:17:29.006985+0000","last_unstale":"2024-04-26T16:20:56.090578+0000","last_undegraded":"2024-04-26T16:20:56.090578+0000","last_fullsized":"2024-04-26T16:20:56.090578+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.917484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:16.115592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"107'18","reported_seq":128,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.179637+0000","last_change":"2024-04-26T16:20:13.290879+0000","last_active":"2024-04-26T16:20:56.179637+0000","last_peered":"2024-04-26T16:20:56.179637+0000","last_clean":"2024-04-26T16:20:56.179637+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:20:56.179637+0000","last_undegraded":"2024-04-26T16:20:56.179637+0000","last_fullsized":"2024-04-26T16:20:56.179637+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'18","last_scrub_stamp":"2024-04-26T16:20:13.290621+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:19:05.704281+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.290621+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.892369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":111,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494302+0000","last_change":"2024-04-26T16:20:06.968281+0000","last_active":"2024-04-26T16:20:55.494302+0000","last_peered":"2024-04-26T16:20:55.494302+0000","last_clean":"2024-04-26T16:20:55.494302+0000","last_became_active":"2024-04-26T16:17:32.027388+0000","last_became_peered":"2024-04-26T16:17:32.027388+0000","last_unstale":"2024-04-26T16:20:55.494302+0000","last_undegraded":"2024-04-26T16:20:55.494302+0000","last_fullsized":"2024-04-26T16:20:55.494302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:06.968062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:20:06.968062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.035273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":155,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.017903+0000","last_change":"2024-04-26T16:21:06.017992+0000","last_active":"2024-04-26T16:21:06.017903+0000","last_peered":"2024-04-26T16:21:06.017903+0000","last_clean":"2024-04-26T16:21:06.017903+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:21:06.017903+0000","last_undegraded":"2024-04-26T16:21:06.017903+0000","last_fullsized":"2024-04-26T16:21:06.017903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:21:06.017712+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:12.646942+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.178835+0000","last_change":"2024-04-26T16:20:10.026900+0000","last_active":"2024-04-26T16:20:56.178835+0000","last_peered":"2024-04-26T16:20:56.178835+0000","last_clean":"2024-04-26T16:20:56.178835+0000","last_became_active":"2024-04-26T16:17:35.673130+0000","last_became_peered":"2024-04-26T16:17:35.673130+0000","last_unstale":"2024-04-26T16:20:56.178835+0000","last_undegraded":"2024-04-26T16:20:56.178835+0000","last_fullsized":"2024-04-26T16:20:56.178835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.026659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.026659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:39.688645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":101,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176557+0000","last_change":"2024-04-26T16:20:28.439735+0000","last_active":"2024-04-26T16:20:56.176557+0000","last_peered":"2024-04-26T16:20:56.176557+0000","last_clean":"2024-04-26T16:20:56.176557+0000","last_became_active":"2024-04-26T16:17:33.068923+0000","last_became_peered":"2024-04-26T16:17:33.068923+0000","last_unstale":"2024-04-26T16:20:56.176557+0000","last_undegraded":"2024-04-26T16:20:56.176557+0000","last_fullsized":"2024-04-26T16:20:56.176557+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.439524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.246043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":116,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884247+0000","last_change":"2024-04-26T16:20:10.311712+0000","last_active":"2024-04-26T16:20:55.884247+0000","last_peered":"2024-04-26T16:20:55.884247+0000","last_clean":"2024-04-26T16:20:55.884247+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:20:55.884247+0000","last_undegraded":"2024-04-26T16:20:55.884247+0000","last_fullsized":"2024-04-26T16:20:55.884247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:10.311472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.311472+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.794801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"86'12","reported_seq":129,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.494093+0000","last_change":"2024-04-26T16:20:33.858656+0000","last_active":"2024-04-26T16:20:55.494093+0000","last_peered":"2024-04-26T16:20:55.494093+0000","last_clean":"2024-04-26T16:20:55.494093+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:20:55.494093+0000","last_undegraded":"2024-04-26T16:20:55.494093+0000","last_fullsized":"2024-04-26T16:20:55.494093+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:33.858421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.858421+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.983533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":97,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.175072+0000","last_change":"2024-04-26T16:20:23.546630+0000","last_active":"2024-04-26T16:20:56.175072+0000","last_peered":"2024-04-26T16:20:56.175072+0000","last_clean":"2024-04-26T16:20:56.175072+0000","last_became_active":"2024-04-26T16:17:35.077832+0000","last_became_peered":"2024-04-26T16:17:35.077832+0000","last_unstale":"2024-04-26T16:20:56.175072+0000","last_undegraded":"2024-04-26T16:20:56.175072+0000","last_fullsized":"2024-04-26T16:20:56.175072+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.546406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.546406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:48.610382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"115'12","reported_seq":157,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.247103+0000","last_change":"2024-04-26T16:20:25.975628+0000","last_active":"2024-04-26T16:20:59.247103+0000","last_peered":"2024-04-26T16:20:59.247103+0000","last_clean":"2024-04-26T16:20:59.247103+0000","last_became_active":"2024-04-26T16:17:33.070130+0000","last_became_peered":"2024-04-26T16:17:33.070130+0000","last_unstale":"2024-04-26T16:20:59.247103+0000","last_undegraded":"2024-04-26T16:20:59.247103+0000","last_fullsized":"2024-04-26T16:20:59.247103+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.975399+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.920318+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":116,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500323+0000","last_change":"2024-04-26T16:20:35.047323+0000","last_active":"2024-04-26T16:20:55.500323+0000","last_peered":"2024-04-26T16:20:55.500323+0000","last_clean":"2024-04-26T16:20:55.500323+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:20:55.500323+0000","last_undegraded":"2024-04-26T16:20:55.500323+0000","last_fullsized":"2024-04-26T16:20:55.500323+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.046847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.046847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.436102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"86'15","reported_seq":129,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.176168+0000","last_change":"2024-04-26T16:20:36.954635+0000","last_active":"2024-04-26T16:20:56.176168+0000","last_peered":"2024-04-26T16:20:56.176168+0000","last_clean":"2024-04-26T16:20:56.176168+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:20:56.176168+0000","last_undegraded":"2024-04-26T16:20:56.176168+0000","last_fullsized":"2024-04-26T16:20:56.176168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:36.954394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:36.954394+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:05.655723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.495970+0000","last_change":"2024-04-26T16:20:14.907782+0000","last_active":"2024-04-26T16:20:55.495970+0000","last_peered":"2024-04-26T16:20:55.495970+0000","last_clean":"2024-04-26T16:20:55.495970+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:20:55.495970+0000","last_undegraded":"2024-04-26T16:20:55.495970+0000","last_fullsized":"2024-04-26T16:20:55.495970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.907547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.907547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.001180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"86'11","reported_seq":154,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.247880+0000","last_change":"2024-04-26T16:20:03.405476+0000","last_active":"2024-04-26T16:20:59.247880+0000","last_peered":"2024-04-26T16:20:59.247880+0000","last_clean":"2024-04-26T16:20:59.247880+0000","last_became_active":"2024-04-26T16:17:33.066299+0000","last_became_peered":"2024-04-26T16:17:33.066299+0000","last_unstale":"2024-04-26T16:20:59.247880+0000","last_undegraded":"2024-04-26T16:20:59.247880+0000","last_fullsized":"2024-04-26T16:20:59.247880+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:03.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.405246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.150208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":110,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500809+0000","last_change":"2024-04-26T16:20:29.023874+0000","last_active":"2024-04-26T16:20:55.500809+0000","last_peered":"2024-04-26T16:20:55.500809+0000","last_clean":"2024-04-26T16:20:55.500809+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:20:55.500809+0000","last_undegraded":"2024-04-26T16:20:55.500809+0000","last_fullsized":"2024-04-26T16:20:55.500809+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:29.023717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:29.023717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.818594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"86'9","reported_seq":112,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.180635+0000","last_change":"2024-04-26T16:19:20.985067+0000","last_active":"2024-04-26T16:20:56.180635+0000","last_peered":"2024-04-26T16:20:56.180635+0000","last_clean":"2024-04-26T16:20:56.180635+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:20:56.180635+0000","last_undegraded":"2024-04-26T16:20:56.180635+0000","last_fullsized":"2024-04-26T16:20:56.180635+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:20.984815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.984815+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496923+0000","last_change":"2024-04-26T16:20:21.525478+0000","last_active":"2024-04-26T16:20:55.496923+0000","last_peered":"2024-04-26T16:20:55.496923+0000","last_clean":"2024-04-26T16:20:55.496923+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:20:55.496923+0000","last_undegraded":"2024-04-26T16:20:55.496923+0000","last_fullsized":"2024-04-26T16:20:55.496923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.525230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.078478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":102,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.500742+0000","last_change":"2024-04-26T16:20:23.025108+0000","last_active":"2024-04-26T16:20:55.500742+0000","last_peered":"2024-04-26T16:20:55.500742+0000","last_clean":"2024-04-26T16:20:55.500742+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:20:55.500742+0000","last_undegraded":"2024-04-26T16:20:55.500742+0000","last_fullsized":"2024-04-26T16:20:55.500742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.024885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:01.215041+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.024885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.206757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.884605+0000","last_change":"2024-04-26T16:20:36.297478+0000","last_active":"2024-04-26T16:20:55.884605+0000","last_peered":"2024-04-26T16:20:55.884605+0000","last_clean":"2024-04-26T16:20:55.884605+0000","last_became_active":"2024-04-26T16:17:29.026793+0000","last_became_peered":"2024-04-26T16:17:29.026793+0000","last_unstale":"2024-04-26T16:20:55.884605+0000","last_undegraded":"2024-04-26T16:20:55.884605+0000","last_fullsized":"2024-04-26T16:20:55.884605+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:20:36.297212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.276670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"86'19","reported_seq":135,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.089309+0000","last_change":"2024-04-26T16:20:11.904568+0000","last_active":"2024-04-26T16:20:56.089309+0000","last_peered":"2024-04-26T16:20:56.089309+0000","last_clean":"2024-04-26T16:20:56.089309+0000","last_became_active":"2024-04-26T16:17:31.024436+0000","last_became_peered":"2024-04-26T16:17:31.024436+0000","last_unstale":"2024-04-26T16:20:56.089309+0000","last_undegraded":"2024-04-26T16:20:56.089309+0000","last_fullsized":"2024-04-26T16:20:56.089309+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'19","last_scrub_stamp":"2024-04-26T16:20:11.904292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.904292+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:17.775021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.883757+0000","last_change":"2024-04-26T16:20:47.169490+0000","last_active":"2024-04-26T16:20:55.883757+0000","last_peered":"2024-04-26T16:20:55.883757+0000","last_clean":"2024-04-26T16:20:55.883757+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:20:55.883757+0000","last_undegraded":"2024-04-26T16:20:55.883757+0000","last_fullsized":"2024-04-26T16:20:55.883757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:20:47.169235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:12.965763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:21:04.417343+0000","last_change":"2024-04-26T16:21:04.417404+0000","last_active":"2024-04-26T16:21:04.417343+0000","last_peered":"2024-04-26T16:21:04.417343+0000","last_clean":"2024-04-26T16:21:04.417343+0000","last_became_active":"2024-04-26T16:17:33.065671+0000","last_became_peered":"2024-04-26T16:17:33.065671+0000","last_unstale":"2024-04-26T16:21:04.417343+0000","last_undegraded":"2024-04-26T16:21:04.417343+0000","last_fullsized":"2024-04-26T16:21:04.417343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:04.417182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:21:04.417182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:17.533499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.497037+0000","last_change":"2024-04-26T16:20:05.437174+0000","last_active":"2024-04-26T16:20:55.497037+0000","last_peered":"2024-04-26T16:20:55.497037+0000","last_clean":"2024-04-26T16:20:55.497037+0000","last_became_active":"2024-04-26T16:17:29.017152+0000","last_became_peered":"2024-04-26T16:17:29.017152+0000","last_unstale":"2024-04-26T16:20:55.497037+0000","last_undegraded":"2024-04-26T16:20:55.497037+0000","last_fullsized":"2024-04-26T16:20:55.497037+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.436941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.436941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:07.446646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'17","reported_seq":132,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.885752+0000","last_change":"2024-04-26T16:20:35.289963+0000","last_active":"2024-04-26T16:20:55.885752+0000","last_peered":"2024-04-26T16:20:55.885752+0000","last_clean":"2024-04-26T16:20:55.885752+0000","last_became_active":"2024-04-26T16:17:31.030244+0000","last_became_peered":"2024-04-26T16:17:31.030244+0000","last_unstale":"2024-04-26T16:20:55.885752+0000","last_undegraded":"2024-04-26T16:20:55.885752+0000","last_fullsized":"2024-04-26T16:20:55.885752+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-04-26T16:20:35.289727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.289727+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.877432+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":96,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.175960+0000","last_change":"2024-04-26T16:20:14.277898+0000","last_active":"2024-04-26T16:20:56.175960+0000","last_peered":"2024-04-26T16:20:56.175960+0000","last_clean":"2024-04-26T16:20:56.175960+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:20:56.175960+0000","last_undegraded":"2024-04-26T16:20:56.175960+0000","last_fullsized":"2024-04-26T16:20:56.175960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:25.766764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":100,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.496224+0000","last_change":"2024-04-26T16:20:09.420985+0000","last_active":"2024-04-26T16:20:55.496224+0000","last_peered":"2024-04-26T16:20:55.496224+0000","last_clean":"2024-04-26T16:20:55.496224+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:20:55.496224+0000","last_undegraded":"2024-04-26T16:20:55.496224+0000","last_fullsized":"2024-04-26T16:20:55.496224+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.420768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.420768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.763275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":110,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.501120+0000","last_change":"2024-04-26T16:20:43.031893+0000","last_active":"2024-04-26T16:20:55.501120+0000","last_peered":"2024-04-26T16:20:55.501120+0000","last_clean":"2024-04-26T16:20:55.501120+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:20:55.501120+0000","last_undegraded":"2024-04-26T16:20:55.501120+0000","last_fullsized":"2024-04-26T16:20:55.501120+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:20:43.031634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.462599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'10","reported_seq":120,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.175514+0000","last_change":"2024-04-26T16:20:34.526337+0000","last_active":"2024-04-26T16:20:56.175514+0000","last_peered":"2024-04-26T16:20:56.175514+0000","last_clean":"2024-04-26T16:20:56.175514+0000","last_became_active":"2024-04-26T16:17:31.040215+0000","last_became_peered":"2024-04-26T16:17:31.040215+0000","last_unstale":"2024-04-26T16:20:56.175514+0000","last_undegraded":"2024-04-26T16:20:56.175514+0000","last_fullsized":"2024-04-26T16:20:56.175514+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:34.526105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:34.526105+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.517534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":105,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:56.174425+0000","last_change":"2024-04-26T16:20:55.467042+0000","last_active":"2024-04-26T16:20:56.174425+0000","last_peered":"2024-04-26T16:20:56.174425+0000","last_clean":"2024-04-26T16:20:56.174425+0000","last_became_active":"2024-04-26T16:17:35.669627+0000","last_became_peered":"2024-04-26T16:17:35.669627+0000","last_unstale":"2024-04-26T16:20:56.174425+0000","last_undegraded":"2024-04-26T16:20:56.174425+0000","last_fullsized":"2024-04-26T16:20:56.174425+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:55.466831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:55.466831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:23.012141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":108,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.497402+0000","last_change":"2024-04-26T16:20:18.397841+0000","last_active":"2024-04-26T16:20:55.497402+0000","last_peered":"2024-04-26T16:20:55.497402+0000","last_clean":"2024-04-26T16:20:55.497402+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:20:55.497402+0000","last_undegraded":"2024-04-26T16:20:55.497402+0000","last_fullsized":"2024-04-26T16:20:55.497402+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.397696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:50.542517+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.397696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.381027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"86'15","reported_seq":129,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:55.886377+0000","last_change":"2024-04-26T16:20:28.238717+0000","last_active":"2024-04-26T16:20:55.886377+0000","last_peered":"2024-04-26T16:20:55.886377+0000","last_clean":"2024-04-26T16:20:55.886377+0000","last_became_active":"2024-04-26T16:17:31.033340+0000","last_became_peered":"2024-04-26T16:17:31.033340+0000","last_unstale":"2024-04-26T16:20:55.886377+0000","last_undegraded":"2024-04-26T16:20:55.886377+0000","last_fullsized":"2024-04-26T16:20:55.886377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:28.238415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.238415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.888574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"109'12","reported_seq":155,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-04-26T16:20:59.247096+0000","last_change":"2024-04-26T16:20:55.047342+0000","last_active":"2024-04-26T16:20:59.247096+0000","last_peered":"2024-04-26T16:20:59.247096+0000","last_clean":"2024-04-26T16:20:59.247096+0000","last_became_active":"2024-04-26T16:17:33.070525+0000","last_became_peered":"2024-04-26T16:17:33.070525+0000","last_unstale":"2024-04-26T16:20:59.247096+0000","last_undegraded":"2024-04-26T16:20:59.247096+0000","last_fullsized":"2024-04-26T16:20:59.247096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-26T16:20:55.047110+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_clean_scrub_stamp":"2024-04-26T16:20:55.047110+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:01.558186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":197,"num_read_kb":192,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":443,"ondisk_log_size":443,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743478,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28336,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716816,"statfs":{"total":95995035648,"available":95966019584,"internally_reserved":0,"allocated":1351680,"data_stored":980580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135936,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28300,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716852,"statfs":{"total":95995035648,"available":95966056448,"internally_reserved":0,"allocated":1314816,"data_stored":977809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31904,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713248,"statfs":{"total":95995035648,"available":95962365952,"internally_reserved":0,"allocated":745472,"data_stored":388778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36644,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708508,"statfs":{"total":95995035648,"available":95957512192,"internally_reserved":0,"allocated":1339392,"data_stored":979893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":724992,"data_stored":387934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084382,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36044,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709108,"statfs":{"total":95995035648,"available":95958126592,"internally_reserved":0,"allocated":724992,"data_stored":387841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855444,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27764,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":765952,"data_stored":388822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462986,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27728,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":729088,"data_stored":387856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:21:09.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: audit 2024-04-26T16:21:08.929028+0000 mgr.y (mgr.24401) 136 : audit [DBG] from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: audit 2024-04-26T16:21:08.929028+0000 mgr.y (mgr.24401) 136 : audit [DBG] from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: cluster 2024-04-26T16:21:09.375527+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: cluster 2024-04-26T16:21:09.375527+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: cluster 2024-04-26T16:21:09.379533+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: cluster 2024-04-26T16:21:09.379533+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: cluster 2024-04-26T16:21:09.412982+0000 mgr.y (mgr.24401) 137 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:10.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[21463]: cluster 2024-04-26T16:21:09.412982+0000 mgr.y (mgr.24401) 137 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: audit 2024-04-26T16:21:08.929028+0000 mgr.y (mgr.24401) 136 : audit [DBG] from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: audit 2024-04-26T16:21:08.929028+0000 mgr.y (mgr.24401) 136 : audit [DBG] from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: cluster 2024-04-26T16:21:09.375527+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: cluster 2024-04-26T16:21:09.375527+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: cluster 2024-04-26T16:21:09.379533+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: cluster 2024-04-26T16:21:09.379533+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: cluster 2024-04-26T16:21:09.412982+0000 mgr.y (mgr.24401) 137 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:10.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:10 smithi050 bash[29234]: cluster 2024-04-26T16:21:09.412982+0000 mgr.y (mgr.24401) 137 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: audit 2024-04-26T16:21:08.929028+0000 mgr.y (mgr.24401) 136 : audit [DBG] from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: audit 2024-04-26T16:21:08.929028+0000 mgr.y (mgr.24401) 136 : audit [DBG] from='client.24593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: cluster 2024-04-26T16:21:09.375527+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: cluster 2024-04-26T16:21:09.375527+0000 osd.0 (osd.0) 85 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: cluster 2024-04-26T16:21:09.379533+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: cluster 2024-04-26T16:21:09.379533+0000 osd.0 (osd.0) 86 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: cluster 2024-04-26T16:21:09.412982+0000 mgr.y (mgr.24401) 137 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:10 smithi190 bash[23868]: cluster 2024-04-26T16:21:09.412982+0000 mgr.y (mgr.24401) 137 : cluster [DBG] pgmap v106: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:11.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:11 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.040657+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:21:11.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:11 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.040657+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:21:11.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:11 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.043867+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:21:11.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:11 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.043867+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:21:11.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:11 smithi190 bash[23868]: audit 2024-04-26T16:21:10.744647+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:11.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:11 smithi190 bash[23868]: audit 2024-04-26T16:21:10.744647+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:11.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.040657+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.040657+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.043867+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.043867+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[21463]: audit 2024-04-26T16:21:10.744647+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[21463]: audit 2024-04-26T16:21:10.744647+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.040657+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.040657+0000 osd.7 (osd.7) 107 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.043867+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.043867+0000 osd.7 (osd.7) 108 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:21:11.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[29234]: audit 2024-04-26T16:21:10.744647+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:11.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:11 smithi050 bash[29234]: audit 2024-04-26T16:21:10.744647+0000 mon.c (mon.2) 79 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.415325+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.c scrub starts 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.415325+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.c scrub starts 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.417770+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.c scrub ok 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:10.417770+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.c scrub ok 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.045556+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.045556+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.047851+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.047851+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.414466+0000 mgr.y (mgr.24401) 138 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:12 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.414466+0000 mgr.y (mgr.24401) 138 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.415325+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.c scrub starts 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.415325+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.c scrub starts 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.417770+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.c scrub ok 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:10.417770+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.c scrub ok 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.045556+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.045556+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.047851+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.047851+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:21:12.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.414466+0000 mgr.y (mgr.24401) 138 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.414466+0000 mgr.y (mgr.24401) 138 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.415325+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.c scrub starts 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.415325+0000 osd.2 (osd.2) 63 : cluster [DBG] 2.c scrub starts 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.417770+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.c scrub ok 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:10.417770+0000 osd.2 (osd.2) 64 : cluster [DBG] 2.c scrub ok 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.045556+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.045556+0000 osd.7 (osd.7) 109 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.047851+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.047851+0000 osd.7 (osd.7) 110 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.414466+0000 mgr.y (mgr.24401) 138 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:12.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:12 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.414466+0000 mgr.y (mgr.24401) 138 : cluster [DBG] pgmap v107: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:13 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.566311+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:21:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:13 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.566311+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:21:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:13 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.568632+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:21:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:13 smithi190 bash[23868]: cluster 2024-04-26T16:21:11.568632+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.566311+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.566311+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.568632+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[21463]: cluster 2024-04-26T16:21:11.568632+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.566311+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.566311+0000 osd.5 (osd.5) 93 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.568632+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:21:13.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:13 smithi050 bash[29234]: cluster 2024-04-26T16:21:11.568632+0000 osd.5 (osd.5) 94 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:12.193054+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:12.193054+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:12.195425+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:12.195425+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.375511+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.12 deep-scrub starts 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.375511+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.12 deep-scrub starts 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.384527+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.12 deep-scrub ok 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.384527+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.12 deep-scrub ok 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.415213+0000 mgr.y (mgr.24401) 139 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:14 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.415213+0000 mgr.y (mgr.24401) 139 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:12.193054+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:12.193054+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:12.195425+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:12.195425+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.375511+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.12 deep-scrub starts 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.375511+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.12 deep-scrub starts 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.384527+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.12 deep-scrub ok 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.384527+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.12 deep-scrub ok 2024-04-26T16:21:14.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.415213+0000 mgr.y (mgr.24401) 139 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.415213+0000 mgr.y (mgr.24401) 139 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:12.193054+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:12.193054+0000 osd.6 (osd.6) 51 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:12.195425+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:12.195425+0000 osd.6 (osd.6) 52 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.375511+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.12 deep-scrub starts 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.375511+0000 osd.0 (osd.0) 87 : cluster [DBG] 3.12 deep-scrub starts 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.384527+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.12 deep-scrub ok 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.384527+0000 osd.0 (osd.0) 88 : cluster [DBG] 3.12 deep-scrub ok 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.415213+0000 mgr.y (mgr.24401) 139 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:14.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.415213+0000 mgr.y (mgr.24401) 139 : cluster [DBG] pgmap v108: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:14.609 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:14.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:21:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:21:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.384758+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.a scrub starts 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.384758+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.a scrub starts 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.387237+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.a scrub ok 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.387237+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.a scrub ok 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.665336+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.665336+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.667691+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:21:15.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:15 smithi190 bash[23868]: cluster 2024-04-26T16:21:13.667691+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.384758+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.a scrub starts 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.384758+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.a scrub starts 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.387237+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.a scrub ok 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.387237+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.a scrub ok 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.665336+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.665336+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.667691+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[29234]: cluster 2024-04-26T16:21:13.667691+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.384758+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.a scrub starts 2024-04-26T16:21:15.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.384758+0000 osd.2 (osd.2) 65 : cluster [DBG] 5.a scrub starts 2024-04-26T16:21:15.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.387237+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.a scrub ok 2024-04-26T16:21:15.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.387237+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.a scrub ok 2024-04-26T16:21:15.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.665336+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:21:15.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.665336+0000 osd.4 (osd.4) 75 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:21:15.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.667691+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:21:15.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:15 smithi050 bash[21463]: cluster 2024-04-26T16:21:13.667691+0000 osd.4 (osd.4) 76 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:21:16.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[23868]: cluster 2024-04-26T16:21:15.033371+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:21:16.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[23868]: cluster 2024-04-26T16:21:15.033371+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:21:16.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[23868]: cluster 2024-04-26T16:21:15.047752+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:21:16.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[23868]: cluster 2024-04-26T16:21:15.047752+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:21:16.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[23868]: cluster 2024-04-26T16:21:15.415860+0000 mgr.y (mgr.24401) 140 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:16.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[23868]: cluster 2024-04-26T16:21:15.415860+0000 mgr.y (mgr.24401) 140 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[29234]: cluster 2024-04-26T16:21:15.033371+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[29234]: cluster 2024-04-26T16:21:15.033371+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[29234]: cluster 2024-04-26T16:21:15.047752+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[29234]: cluster 2024-04-26T16:21:15.047752+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[29234]: cluster 2024-04-26T16:21:15.415860+0000 mgr.y (mgr.24401) 140 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[29234]: cluster 2024-04-26T16:21:15.415860+0000 mgr.y (mgr.24401) 140 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[21463]: cluster 2024-04-26T16:21:15.033371+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[21463]: cluster 2024-04-26T16:21:15.033371+0000 osd.3 (osd.3) 93 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[21463]: cluster 2024-04-26T16:21:15.047752+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[21463]: cluster 2024-04-26T16:21:15.047752+0000 osd.3 (osd.3) 94 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[21463]: cluster 2024-04-26T16:21:15.415860+0000 mgr.y (mgr.24401) 140 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:16 smithi050 bash[21463]: cluster 2024-04-26T16:21:15.415860+0000 mgr.y (mgr.24401) 140 : cluster [DBG] pgmap v109: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:16.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:21:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.072740+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.072740+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.075349+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.075349+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.083795+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.083795+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.092450+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.092450+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: audit 2024-04-26T16:21:16.482705+0000 mgr.y (mgr.24401) 141 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:17.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:17 smithi190 bash[23868]: audit 2024-04-26T16:21:16.482705+0000 mgr.y (mgr.24401) 141 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.072740+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.072740+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.075349+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.075349+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.083795+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.083795+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.092450+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.092450+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:21:17.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: audit 2024-04-26T16:21:16.482705+0000 mgr.y (mgr.24401) 141 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[21463]: audit 2024-04-26T16:21:16.482705+0000 mgr.y (mgr.24401) 141 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.072740+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.072740+0000 osd.3 (osd.3) 95 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.075349+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.075349+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.083795+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.083795+0000 osd.7 (osd.7) 111 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.092450+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.092450+0000 osd.7 (osd.7) 112 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: audit 2024-04-26T16:21:16.482705+0000 mgr.y (mgr.24401) 141 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:17.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:17 smithi050 bash[29234]: audit 2024-04-26T16:21:16.482705+0000 mgr.y (mgr.24401) 141 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- 172.21.15.50:0/3646196401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103e30 msgr2=0x7fb5d8110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- 172.21.15.50:0/3646196401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103e30 0x7fb5d8110290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb5d000b0a0 tx=0x7fb5d000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:21:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- 172.21.15.50:0/3646196401 shutdown_connections 2024-04-26T16:21:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- 172.21.15.50:0/3646196401 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103e30 0x7fb5d8110290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:17.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- 172.21.15.50:0/3646196401 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d81094c0 0x7fb5d81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- 172.21.15.50:0/3646196401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d8103470 0x7fb5d81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- 172.21.15.50:0/3646196401 >> 172.21.15.50:0/3646196401 conn(0x7fb5d807a580 msgr2=0x7fb5d807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- 172.21.15.50:0/3646196401 shutdown_connections 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- 172.21.15.50:0/3646196401 wait complete. 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 librados: starting msgr at 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 librados: starting objecter 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 Processor -- start 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- start start 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 librados: setting wanted keys 2024-04-26T16:21:17.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 librados: calling monclient init 2024-04-26T16:21:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 0x7fb5d8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d8103e30 0x7fb5d8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d81094c0 0x7fb5d80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:17.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8112ab0 con 0x7fb5d81094c0 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d8112930 con 0x7fb5d8103e30 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5df57e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb5d8112c30 con 0x7fb5d8103470 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 0x7fb5d8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 0x7fb5d8075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42024/0 (socket says 172.21.15.50:42024) 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 -- 172.21.15.50:0/2307982969 learned_addr learned my addr 172.21.15.50:0/2307982969 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5ddb1b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d81094c0 0x7fb5d80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dcb19700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d8103e30 0x7fb5d8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 -- 172.21.15.50:0/2307982969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d8103e30 msgr2=0x7fb5d8075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d8103e30 0x7fb5d8075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 -- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d81094c0 msgr2=0x7fb5d80761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d81094c0 0x7fb5d80761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:17.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 -- 172.21.15.50:0/2307982969 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d81a7640 con 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5dd31a700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 0x7fb5d8075720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400eb30 tx=0x7fb5d400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5ce7fc700 10 client.?.objecter ms_handle_connect 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.885+0000 7fb5ce7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d400cd40 con 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb5d4004480 con 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d81a7930 con 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4003df0 con 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5df57e700 10 client.24599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d81a7ec0 con 0x7fb5d8103470 2024-04-26T16:21:17.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5df57e700 1 librados: init done 2024-04-26T16:21:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d806a110 con 0x7fb5d8103470 2024-04-26T16:21:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb5d4003f90 con 0x7fb5d8103470 2024-04-26T16:21:17.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(121..121 src has 67..121) ==== 7450+0+0 (secure 0 0 0) 0x7fb5d40914f0 con 0x7fb5d8103470 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 10 client.24599.objecter ms_dispatch 0x7fb5d8001550 osd_map(121..121 src has 67..121) 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 3 client.24599.objecter handle_osd_map got epochs [121,121] > 0 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 3 client.24599.objecter handle_osd_map decoding full epoch 121 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5dcb19700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 20 client.24599.objecter dump_active .. 0 homeless 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5dcb19700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8005950 tx=0x7fb5c80058e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:17.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.889+0000 7fb5ce7fc700 10 client.24599.objecter ms_handle_connect 0x7fb5c4071f80 2024-04-26T16:21:17.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:17.897+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb5d4016330 con 0x7fb5d8103470 2024-04-26T16:21:18.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.117+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb5d8102300 con 0x7fb5c4071f80 2024-04-26T16:21:18.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.133+0000 7fb5ce7fc700 1 -- 172.21.15.50:0/2307982969 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+346808 (secure 0 0 0) 0x7fb5d8102300 con 0x7fb5c4071f80 2024-04-26T16:21:18.130 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:21:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 10 librados: watch_flush enter 2024-04-26T16:21:18.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 10 librados: watch_flush exit 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 20 client.24599.objecter shutdown clearing up homeless session... 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 10 client.24599.objecter successfully canceled tick 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 msgr2=0x7fb5c4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8005950 tx=0x7fb5c80058e0 comp rx=0 tx=0).stop 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 msgr2=0x7fb5d8075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 0x7fb5d8075720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb5d400eb30 tx=0x7fb5d400eef0 comp rx=0 tx=0).stop 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 shutdown_connections 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb5c4071f80 0x7fb5c4074440 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb5d81094c0 0x7fb5d80761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb5d8103470 0x7fb5d8075720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 --2- 172.21.15.50:0/2307982969 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb5d8103e30 0x7fb5d8075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 >> 172.21.15.50:0/2307982969 conn(0x7fb5d807a580 msgr2=0x7fb5d8101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:18.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 shutdown_connections 2024-04-26T16:21:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 -- 172.21.15.50:0/2307982969 wait complete. 2024-04-26T16:21:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:18.137+0000 7fb5df57e700 1 librados: shutdown 2024-04-26T16:21:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:21:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:18 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.753200+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:21:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:18 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.753200+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:21:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:18 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.755262+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:21:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:18 smithi190 bash[23868]: cluster 2024-04-26T16:21:16.755262+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:21:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:18 smithi190 bash[23868]: cluster 2024-04-26T16:21:17.417325+0000 mgr.y (mgr.24401) 142 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-04-26T16:21:18.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:18 smithi190 bash[23868]: cluster 2024-04-26T16:21:17.417325+0000 mgr.y (mgr.24401) 142 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.753200+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.753200+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.755262+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[21463]: cluster 2024-04-26T16:21:16.755262+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[21463]: cluster 2024-04-26T16:21:17.417325+0000 mgr.y (mgr.24401) 142 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[21463]: cluster 2024-04-26T16:21:17.417325+0000 mgr.y (mgr.24401) 142 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.753200+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.753200+0000 osd.4 (osd.4) 77 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:21:18.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.755262+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:21:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[29234]: cluster 2024-04-26T16:21:16.755262+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:21:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[29234]: cluster 2024-04-26T16:21:17.417325+0000 mgr.y (mgr.24401) 142 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-04-26T16:21:18.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:18 smithi050 bash[29234]: cluster 2024-04-26T16:21:17.417325+0000 mgr.y (mgr.24401) 142 : cluster [DBG] pgmap v110: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-04-26T16:21:18.999 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-04-26T16:21:17.416156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1157,"num_read_kb":1003,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":696,"ondisk_log_size":696,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244544,"kb_used_data":7612,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716672,"statfs":{"total":767960285184,"available":767709872128,"internally_reserved":0,"allocated":7794688,"data_stored":4941113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":242535988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.423489"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192489+0000","last_change":"2024-04-26T16:20:46.049553+0000","last_active":"2024-04-26T16:21:07.192489+0000","last_peered":"2024-04-26T16:21:07.192489+0000","last_clean":"2024-04-26T16:21:07.192489+0000","last_became_active":"2024-04-26T16:17:35.677526+0000","last_became_peered":"2024-04-26T16:17:35.677526+0000","last_unstale":"2024-04-26T16:21:07.192489+0000","last_undegraded":"2024-04-26T16:21:07.192489+0000","last_fullsized":"2024-04-26T16:21:07.192489+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:46.049339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:46.049339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.981921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.745685+0000","last_change":"2024-04-26T16:20:05.019229+0000","last_active":"2024-04-26T16:21:07.745685+0000","last_peered":"2024-04-26T16:21:07.745685+0000","last_clean":"2024-04-26T16:21:07.745685+0000","last_became_active":"2024-04-26T16:17:33.070922+0000","last_became_peered":"2024-04-26T16:17:33.070922+0000","last_unstale":"2024-04-26T16:21:07.745685+0000","last_undegraded":"2024-04-26T16:21:07.745685+0000","last_fullsized":"2024-04-26T16:21:07.745685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.018938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.018938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.770490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.999560+0000","last_change":"2024-04-26T16:20:17.275735+0000","last_active":"2024-04-26T16:21:06.999560+0000","last_peered":"2024-04-26T16:21:06.999560+0000","last_clean":"2024-04-26T16:21:06.999560+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:21:06.999560+0000","last_undegraded":"2024-04-26T16:21:06.999560+0000","last_fullsized":"2024-04-26T16:21:06.999560+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.275530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.275530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:42.183654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"86'10","reported_seq":129,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:16.075445+0000","last_change":"2024-04-26T16:21:16.075489+0000","last_active":"2024-04-26T16:21:16.075445+0000","last_peered":"2024-04-26T16:21:16.075445+0000","last_clean":"2024-04-26T16:21:16.075445+0000","last_became_active":"2024-04-26T16:17:31.049997+0000","last_became_peered":"2024-04-26T16:17:31.049997+0000","last_unstale":"2024-04-26T16:21:16.075445+0000","last_undegraded":"2024-04-26T16:21:16.075445+0000","last_fullsized":"2024-04-26T16:21:16.075445+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:21:16.075353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:21:16.075353+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:41.789553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.746511+0000","last_change":"2024-04-26T16:20:02.038119+0000","last_active":"2024-04-26T16:21:07.746511+0000","last_peered":"2024-04-26T16:21:07.746511+0000","last_clean":"2024-04-26T16:21:07.746511+0000","last_became_active":"2024-04-26T16:17:35.065560+0000","last_became_peered":"2024-04-26T16:17:35.065560+0000","last_unstale":"2024-04-26T16:21:07.746511+0000","last_undegraded":"2024-04-26T16:21:07.746511+0000","last_fullsized":"2024-04-26T16:21:07.746511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.037900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.037900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.752700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995564+0000","last_change":"2024-04-26T16:20:03.150021+0000","last_active":"2024-04-26T16:21:06.995564+0000","last_peered":"2024-04-26T16:21:06.995564+0000","last_clean":"2024-04-26T16:21:06.995564+0000","last_became_active":"2024-04-26T16:17:33.058899+0000","last_became_peered":"2024-04-26T16:17:33.058899+0000","last_unstale":"2024-04-26T16:21:06.995564+0000","last_undegraded":"2024-04-26T16:21:06.995564+0000","last_fullsized":"2024-04-26T16:21:06.995564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.149607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.149607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:14.024417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.194057+0000","last_change":"2024-04-26T16:20:03.947061+0000","last_active":"2024-04-26T16:21:07.194057+0000","last_peered":"2024-04-26T16:21:07.194057+0000","last_clean":"2024-04-26T16:21:07.194057+0000","last_became_active":"2024-04-26T16:17:29.016870+0000","last_became_peered":"2024-04-26T16:17:29.016870+0000","last_unstale":"2024-04-26T16:21:07.194057+0000","last_undegraded":"2024-04-26T16:21:07.194057+0000","last_fullsized":"2024-04-26T16:21:07.194057+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.946950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.302253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"86'11","reported_seq":125,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.999284+0000","last_change":"2024-04-26T16:20:04.348527+0000","last_active":"2024-04-26T16:21:06.999284+0000","last_peered":"2024-04-26T16:21:06.999284+0000","last_clean":"2024-04-26T16:21:06.999284+0000","last_became_active":"2024-04-26T16:17:31.041819+0000","last_became_peered":"2024-04-26T16:17:31.041819+0000","last_unstale":"2024-04-26T16:21:06.999284+0000","last_undegraded":"2024-04-26T16:21:06.999284+0000","last_fullsized":"2024-04-26T16:21:06.999284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:04.348299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.348299+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:11.592422+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.554421+0000","last_change":"2024-04-26T16:20:02.714702+0000","last_active":"2024-04-26T16:21:07.554421+0000","last_peered":"2024-04-26T16:21:07.554421+0000","last_clean":"2024-04-26T16:21:07.554421+0000","last_became_active":"2024-04-26T16:17:35.087319+0000","last_became_peered":"2024-04-26T16:17:35.087319+0000","last_unstale":"2024-04-26T16:21:07.554421+0000","last_undegraded":"2024-04-26T16:21:07.554421+0000","last_fullsized":"2024-04-26T16:21:07.554421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:20:02.714466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.714466+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.031920+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994167+0000","last_change":"2024-04-26T16:20:33.242024+0000","last_active":"2024-04-26T16:21:06.994167+0000","last_peered":"2024-04-26T16:21:06.994167+0000","last_clean":"2024-04-26T16:21:06.994167+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:21:06.994167+0000","last_undegraded":"2024-04-26T16:21:06.994167+0000","last_fullsized":"2024-04-26T16:21:06.994167+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:33.241737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.241737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.846027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:11.048024+0000","last_change":"2024-04-26T16:21:11.048090+0000","last_active":"2024-04-26T16:21:11.048024+0000","last_peered":"2024-04-26T16:21:11.048024+0000","last_clean":"2024-04-26T16:21:11.048024+0000","last_became_active":"2024-04-26T16:17:29.022163+0000","last_became_peered":"2024-04-26T16:17:29.022163+0000","last_unstale":"2024-04-26T16:21:11.048024+0000","last_undegraded":"2024-04-26T16:21:11.048024+0000","last_fullsized":"2024-04-26T16:21:11.048024+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:11.047860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:21:11.047860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:20.348336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"86'15","reported_seq":132,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.553762+0000","last_change":"2024-04-26T16:20:25.511190+0000","last_active":"2024-04-26T16:21:07.553762+0000","last_peered":"2024-04-26T16:21:07.553762+0000","last_clean":"2024-04-26T16:21:07.553762+0000","last_became_active":"2024-04-26T16:17:31.022500+0000","last_became_peered":"2024-04-26T16:17:31.022500+0000","last_unstale":"2024-04-26T16:21:07.553762+0000","last_undegraded":"2024-04-26T16:21:07.553762+0000","last_fullsized":"2024-04-26T16:21:07.553762+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:25.510941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.510941+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.088665+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994767+0000","last_change":"2024-04-26T16:20:47.306391+0000","last_active":"2024-04-26T16:21:06.994767+0000","last_peered":"2024-04-26T16:21:06.994767+0000","last_clean":"2024-04-26T16:21:06.994767+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:21:06.994767+0000","last_undegraded":"2024-04-26T16:21:06.994767+0000","last_fullsized":"2024-04-26T16:21:06.994767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:47.306263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:47.306263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.778374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":103,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.555001+0000","last_change":"2024-04-26T16:20:18.567822+0000","last_active":"2024-04-26T16:21:07.555001+0000","last_peered":"2024-04-26T16:21:07.555001+0000","last_clean":"2024-04-26T16:21:07.555001+0000","last_became_active":"2024-04-26T16:17:33.071659+0000","last_became_peered":"2024-04-26T16:17:33.071659+0000","last_unstale":"2024-04-26T16:21:07.555001+0000","last_undegraded":"2024-04-26T16:21:07.555001+0000","last_fullsized":"2024-04-26T16:21:07.555001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.567616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.567616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.293668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994864+0000","last_change":"2024-04-26T16:20:23.315306+0000","last_active":"2024-04-26T16:21:06.994864+0000","last_peered":"2024-04-26T16:21:06.994864+0000","last_clean":"2024-04-26T16:21:06.994864+0000","last_became_active":"2024-04-26T16:17:29.025659+0000","last_became_peered":"2024-04-26T16:17:29.025659+0000","last_unstale":"2024-04-26T16:21:06.994864+0000","last_undegraded":"2024-04-26T16:21:06.994864+0000","last_fullsized":"2024-04-26T16:21:06.994864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.315099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.315099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.593677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"86'12","reported_seq":130,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.555751+0000","last_change":"2024-04-26T16:20:08.659388+0000","last_active":"2024-04-26T16:21:07.555751+0000","last_peered":"2024-04-26T16:21:07.555751+0000","last_clean":"2024-04-26T16:21:07.555751+0000","last_became_active":"2024-04-26T16:17:31.019223+0000","last_became_peered":"2024-04-26T16:17:31.019223+0000","last_unstale":"2024-04-26T16:21:07.555751+0000","last_undegraded":"2024-04-26T16:21:07.555751+0000","last_fullsized":"2024-04-26T16:21:07.555751+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:08.659153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.659153+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.636156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.194007+0000","last_change":"2024-04-26T16:20:32.864786+0000","last_active":"2024-04-26T16:21:07.194007+0000","last_peered":"2024-04-26T16:21:07.194007+0000","last_clean":"2024-04-26T16:21:07.194007+0000","last_became_active":"2024-04-26T16:17:35.673178+0000","last_became_peered":"2024-04-26T16:17:35.673178+0000","last_unstale":"2024-04-26T16:21:07.194007+0000","last_undegraded":"2024-04-26T16:21:07.194007+0000","last_fullsized":"2024-04-26T16:21:07.194007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:32.864565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:32.864565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:00.660802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.746958+0000","last_change":"2024-04-26T16:20:26.931424+0000","last_active":"2024-04-26T16:21:07.746958+0000","last_peered":"2024-04-26T16:21:07.746958+0000","last_clean":"2024-04-26T16:21:07.746958+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:21:07.746958+0000","last_undegraded":"2024-04-26T16:21:07.746958+0000","last_fullsized":"2024-04-26T16:21:07.746958+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.931219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.931219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.617802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192865+0000","last_change":"2024-04-26T16:20:59.111406+0000","last_active":"2024-04-26T16:21:07.192865+0000","last_peered":"2024-04-26T16:21:07.192865+0000","last_clean":"2024-04-26T16:21:07.192865+0000","last_became_active":"2024-04-26T16:17:29.005070+0000","last_became_peered":"2024-04-26T16:17:29.005070+0000","last_unstale":"2024-04-26T16:21:07.192865+0000","last_undegraded":"2024-04-26T16:21:07.192865+0000","last_fullsized":"2024-04-26T16:21:07.192865+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:59.111294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:59.111294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:17.940312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"86'9","reported_seq":132,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.744651+0000","last_change":"2024-04-26T16:20:53.797670+0000","last_active":"2024-04-26T16:21:07.744651+0000","last_peered":"2024-04-26T16:21:07.744651+0000","last_clean":"2024-04-26T16:21:07.744651+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:21:07.744651+0000","last_undegraded":"2024-04-26T16:21:07.744651+0000","last_fullsized":"2024-04-26T16:21:07.744651+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:53.797430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:53.797430+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:06.830917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.747896+0000","last_change":"2024-04-26T16:20:15.936737+0000","last_active":"2024-04-26T16:21:07.747896+0000","last_peered":"2024-04-26T16:21:07.747896+0000","last_clean":"2024-04-26T16:21:07.747896+0000","last_became_active":"2024-04-26T16:17:35.674743+0000","last_became_peered":"2024-04-26T16:17:35.674743+0000","last_unstale":"2024-04-26T16:21:07.747896+0000","last_undegraded":"2024-04-26T16:21:07.747896+0000","last_fullsized":"2024-04-26T16:21:07.747896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.936392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.456145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.001453+0000","last_change":"2024-04-26T16:20:26.994856+0000","last_active":"2024-04-26T16:21:07.001453+0000","last_peered":"2024-04-26T16:21:07.001453+0000","last_clean":"2024-04-26T16:21:07.001453+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:21:07.001453+0000","last_undegraded":"2024-04-26T16:21:07.001453+0000","last_fullsized":"2024-04-26T16:21:07.001453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.994631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.994631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:56.773657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:12.195604+0000","last_change":"2024-04-26T16:21:12.195667+0000","last_active":"2024-04-26T16:21:12.195604+0000","last_peered":"2024-04-26T16:21:12.195604+0000","last_clean":"2024-04-26T16:21:12.195604+0000","last_became_active":"2024-04-26T16:17:29.004486+0000","last_became_peered":"2024-04-26T16:17:29.004486+0000","last_unstale":"2024-04-26T16:21:12.195604+0000","last_undegraded":"2024-04-26T16:21:12.195604+0000","last_fullsized":"2024-04-26T16:21:12.195604+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:12.195436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:21:12.195436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:38.017743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"114'7","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.996851+0000","last_change":"2024-04-26T16:20:22.150870+0000","last_active":"2024-04-26T16:21:06.996851+0000","last_peered":"2024-04-26T16:21:06.996851+0000","last_clean":"2024-04-26T16:21:06.996851+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:21:06.996851+0000","last_undegraded":"2024-04-26T16:21:06.996851+0000","last_fullsized":"2024-04-26T16:21:06.996851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:20:22.150727+0000","last_clean_scrub_stamp":"2024-04-26T16:20:22.150727+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:47.002699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.001804+0000","last_change":"2024-04-26T16:20:10.128158+0000","last_active":"2024-04-26T16:21:07.001804+0000","last_peered":"2024-04-26T16:21:07.001804+0000","last_clean":"2024-04-26T16:21:07.001804+0000","last_became_active":"2024-04-26T16:17:35.080236+0000","last_became_peered":"2024-04-26T16:17:35.080236+0000","last_unstale":"2024-04-26T16:21:07.001804+0000","last_undegraded":"2024-04-26T16:21:07.001804+0000","last_fullsized":"2024-04-26T16:21:07.001804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.127949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.127949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.505883+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.999085+0000","last_change":"2024-04-26T16:20:03.390713+0000","last_active":"2024-04-26T16:21:06.999085+0000","last_peered":"2024-04-26T16:21:06.999085+0000","last_clean":"2024-04-26T16:21:06.999085+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:21:06.999085+0000","last_undegraded":"2024-04-26T16:21:06.999085+0000","last_fullsized":"2024-04-26T16:21:06.999085+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:03.390501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.390501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192445+0000","last_change":"2024-04-26T16:20:53.091231+0000","last_active":"2024-04-26T16:21:07.192445+0000","last_peered":"2024-04-26T16:21:07.192445+0000","last_clean":"2024-04-26T16:21:07.192445+0000","last_became_active":"2024-04-26T16:17:28.999524+0000","last_became_peered":"2024-04-26T16:17:28.999524+0000","last_unstale":"2024-04-26T16:21:07.192445+0000","last_undegraded":"2024-04-26T16:21:07.192445+0000","last_fullsized":"2024-04-26T16:21:07.192445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:20:53.090830+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:10.535270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"86'9","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.191764+0000","last_change":"2024-04-26T16:20:34.899058+0000","last_active":"2024-04-26T16:21:07.191764+0000","last_peered":"2024-04-26T16:21:07.191764+0000","last_clean":"2024-04-26T16:21:07.191764+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:21:07.191764+0000","last_undegraded":"2024-04-26T16:21:07.191764+0000","last_fullsized":"2024-04-26T16:21:07.191764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:34.898819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:34.898819+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.454975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995510+0000","last_change":"2024-04-26T16:20:54.010267+0000","last_active":"2024-04-26T16:21:06.995510+0000","last_peered":"2024-04-26T16:21:06.995510+0000","last_clean":"2024-04-26T16:21:06.995510+0000","last_became_active":"2024-04-26T16:17:35.081916+0000","last_became_peered":"2024-04-26T16:17:35.081916+0000","last_unstale":"2024-04-26T16:21:06.995510+0000","last_undegraded":"2024-04-26T16:21:06.995510+0000","last_fullsized":"2024-04-26T16:21:06.995510+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:20:54.009881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:54.009881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:02.400809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"86'11","reported_seq":161,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.256072+0000","last_change":"2024-04-26T16:19:59.878190+0000","last_active":"2024-04-26T16:21:09.256072+0000","last_peered":"2024-04-26T16:21:09.256072+0000","last_clean":"2024-04-26T16:21:09.256072+0000","last_became_active":"2024-04-26T16:17:33.061926+0000","last_became_peered":"2024-04-26T16:17:33.061926+0000","last_unstale":"2024-04-26T16:21:09.256072+0000","last_undegraded":"2024-04-26T16:21:09.256072+0000","last_fullsized":"2024-04-26T16:21:09.256072+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:59.877790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.877790+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.440948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":111,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.553996+0000","last_change":"2024-04-26T16:19:58.724981+0000","last_active":"2024-04-26T16:21:07.553996+0000","last_peered":"2024-04-26T16:21:07.553996+0000","last_clean":"2024-04-26T16:21:07.553996+0000","last_became_active":"2024-04-26T16:17:29.025319+0000","last_became_peered":"2024-04-26T16:17:29.025319+0000","last_unstale":"2024-04-26T16:21:07.553996+0000","last_undegraded":"2024-04-26T16:21:07.553996+0000","last_fullsized":"2024-04-26T16:21:07.553996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.724766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.724766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:10.832669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"118'22","reported_seq":140,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.996062+0000","last_change":"2024-04-26T16:20:37.960013+0000","last_active":"2024-04-26T16:21:06.996062+0000","last_peered":"2024-04-26T16:21:06.996062+0000","last_clean":"2024-04-26T16:21:06.996062+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:21:06.996062+0000","last_undegraded":"2024-04-26T16:21:06.996062+0000","last_fullsized":"2024-04-26T16:21:06.996062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:20:37.959753+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:48.491354+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.193939+0000","last_change":"2024-04-26T16:20:35.919687+0000","last_active":"2024-04-26T16:21:07.193939+0000","last_peered":"2024-04-26T16:21:07.193939+0000","last_clean":"2024-04-26T16:21:07.193939+0000","last_became_active":"2024-04-26T16:17:35.677366+0000","last_became_peered":"2024-04-26T16:17:35.677366+0000","last_unstale":"2024-04-26T16:21:07.193939+0000","last_undegraded":"2024-04-26T16:21:07.193939+0000","last_fullsized":"2024-04-26T16:21:07.193939+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.919459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.919459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.324456+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994119+0000","last_change":"2024-04-26T16:20:26.239452+0000","last_active":"2024-04-26T16:21:06.994119+0000","last_peered":"2024-04-26T16:21:06.994119+0000","last_clean":"2024-04-26T16:21:06.994119+0000","last_became_active":"2024-04-26T16:17:33.075775+0000","last_became_peered":"2024-04-26T16:17:33.075775+0000","last_unstale":"2024-04-26T16:21:06.994119+0000","last_undegraded":"2024-04-26T16:21:06.994119+0000","last_fullsized":"2024-04-26T16:21:06.994119+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:26.239095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.239095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.234785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.133037+0000","last_change":"2024-04-26T16:20:02.934191+0000","last_active":"2024-04-26T16:21:07.133037+0000","last_peered":"2024-04-26T16:21:07.133037+0000","last_clean":"2024-04-26T16:21:07.133037+0000","last_became_active":"2024-04-26T16:17:29.018752+0000","last_became_peered":"2024-04-26T16:17:29.018752+0000","last_unstale":"2024-04-26T16:21:07.133037+0000","last_undegraded":"2024-04-26T16:21:07.133037+0000","last_fullsized":"2024-04-26T16:21:07.133037+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:02.933912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:20:02.933912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.658980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"86'9","reported_seq":123,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.553697+0000","last_change":"2024-04-26T16:20:41.527249+0000","last_active":"2024-04-26T16:21:07.553697+0000","last_peered":"2024-04-26T16:21:07.553697+0000","last_clean":"2024-04-26T16:21:07.553697+0000","last_became_active":"2024-04-26T16:17:31.024113+0000","last_became_peered":"2024-04-26T16:17:31.024113+0000","last_unstale":"2024-04-26T16:21:07.553697+0000","last_undegraded":"2024-04-26T16:21:07.553697+0000","last_fullsized":"2024-04-26T16:21:07.553697+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:20:41.527003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:41.527003+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:42.238560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.993896+0000","last_change":"2024-04-26T16:20:51.060785+0000","last_active":"2024-04-26T16:21:06.993896+0000","last_peered":"2024-04-26T16:21:06.993896+0000","last_clean":"2024-04-26T16:21:06.993896+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:21:06.993896+0000","last_undegraded":"2024-04-26T16:21:06.993896+0000","last_fullsized":"2024-04-26T16:21:06.993896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:51.060550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:51.060550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:11.425128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.131203+0000","last_change":"2024-04-26T16:20:51.017702+0000","last_active":"2024-04-26T16:21:07.131203+0000","last_peered":"2024-04-26T16:21:07.131203+0000","last_clean":"2024-04-26T16:21:07.131203+0000","last_became_active":"2024-04-26T16:17:33.059705+0000","last_became_peered":"2024-04-26T16:17:33.059705+0000","last_unstale":"2024-04-26T16:21:07.131203+0000","last_undegraded":"2024-04-26T16:21:07.131203+0000","last_fullsized":"2024-04-26T16:21:07.131203+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:51.017392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:51.017392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:07.706485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":111,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.556017+0000","last_change":"2024-04-26T16:20:19.575996+0000","last_active":"2024-04-26T16:21:07.556017+0000","last_peered":"2024-04-26T16:21:07.556017+0000","last_clean":"2024-04-26T16:21:07.556017+0000","last_became_active":"2024-04-26T16:17:29.022454+0000","last_became_peered":"2024-04-26T16:17:29.022454+0000","last_unstale":"2024-04-26T16:21:07.556017+0000","last_undegraded":"2024-04-26T16:21:07.556017+0000","last_fullsized":"2024-04-26T16:21:07.556017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:19.575787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.575787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.873083+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"86'6","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.999267+0000","last_change":"2024-04-26T16:20:21.162238+0000","last_active":"2024-04-26T16:21:06.999267+0000","last_peered":"2024-04-26T16:21:06.999267+0000","last_clean":"2024-04-26T16:21:06.999267+0000","last_became_active":"2024-04-26T16:17:31.053359+0000","last_became_peered":"2024-04-26T16:17:31.053359+0000","last_unstale":"2024-04-26T16:21:06.999267+0000","last_undegraded":"2024-04-26T16:21:06.999267+0000","last_fullsized":"2024-04-26T16:21:06.999267+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'6","last_scrub_stamp":"2024-04-26T16:20:21.161995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.161995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.190751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192773+0000","last_change":"2024-04-26T16:20:08.001062+0000","last_active":"2024-04-26T16:21:07.192773+0000","last_peered":"2024-04-26T16:21:07.192773+0000","last_clean":"2024-04-26T16:21:07.192773+0000","last_became_active":"2024-04-26T16:17:35.094615+0000","last_became_peered":"2024-04-26T16:17:35.094615+0000","last_unstale":"2024-04-26T16:21:07.192773+0000","last_undegraded":"2024-04-26T16:21:07.192773+0000","last_fullsized":"2024-04-26T16:21:07.192773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:08.000834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:20:08.000834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.457279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.000895+0000","last_change":"2024-04-26T16:20:40.011955+0000","last_active":"2024-04-26T16:21:07.000895+0000","last_peered":"2024-04-26T16:21:07.000895+0000","last_clean":"2024-04-26T16:21:07.000895+0000","last_became_active":"2024-04-26T16:17:33.048202+0000","last_became_peered":"2024-04-26T16:17:33.048202+0000","last_unstale":"2024-04-26T16:21:07.000895+0000","last_undegraded":"2024-04-26T16:21:07.000895+0000","last_fullsized":"2024-04-26T16:21:07.000895+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:40.011722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:40.011722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:59.380129+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.001085+0000","last_change":"2024-04-26T16:20:44.976307+0000","last_active":"2024-04-26T16:21:07.001085+0000","last_peered":"2024-04-26T16:21:07.001085+0000","last_clean":"2024-04-26T16:21:07.001085+0000","last_became_active":"2024-04-26T16:17:29.011346+0000","last_became_peered":"2024-04-26T16:17:29.011346+0000","last_unstale":"2024-04-26T16:21:07.001085+0000","last_undegraded":"2024-04-26T16:21:07.001085+0000","last_fullsized":"2024-04-26T16:21:07.001085+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:44.975982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:44.975982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:09.024600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"86'10","reported_seq":123,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.744722+0000","last_change":"2024-04-26T16:20:23.996060+0000","last_active":"2024-04-26T16:21:07.744722+0000","last_peered":"2024-04-26T16:21:07.744722+0000","last_clean":"2024-04-26T16:21:07.744722+0000","last_became_active":"2024-04-26T16:17:31.021476+0000","last_became_peered":"2024-04-26T16:17:31.021476+0000","last_unstale":"2024-04-26T16:21:07.744722+0000","last_undegraded":"2024-04-26T16:21:07.744722+0000","last_fullsized":"2024-04-26T16:21:07.744722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:23.995817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.995817+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.162295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994403+0000","last_change":"2024-04-26T16:20:25.238939+0000","last_active":"2024-04-26T16:21:06.994403+0000","last_peered":"2024-04-26T16:21:06.994403+0000","last_clean":"2024-04-26T16:21:06.994403+0000","last_became_active":"2024-04-26T16:17:35.078105+0000","last_became_peered":"2024-04-26T16:17:35.078105+0000","last_unstale":"2024-04-26T16:21:06.994403+0000","last_undegraded":"2024-04-26T16:21:06.994403+0000","last_fullsized":"2024-04-26T16:21:06.994403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:25.238710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:06.659791+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.238710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:31.861824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.193381+0000","last_change":"2024-04-26T16:20:07.028942+0000","last_active":"2024-04-26T16:21:07.193381+0000","last_peered":"2024-04-26T16:21:07.193381+0000","last_clean":"2024-04-26T16:21:07.193381+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:21:07.193381+0000","last_undegraded":"2024-04-26T16:21:07.193381+0000","last_fullsized":"2024-04-26T16:21:07.193381+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:07.028724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:07.028724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.581251+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.132214+0000","last_change":"2024-04-26T16:20:12.898651+0000","last_active":"2024-04-26T16:21:07.132214+0000","last_peered":"2024-04-26T16:21:07.132214+0000","last_clean":"2024-04-26T16:21:07.132214+0000","last_became_active":"2024-04-26T16:17:29.013639+0000","last_became_peered":"2024-04-26T16:17:29.013639+0000","last_unstale":"2024-04-26T16:21:07.132214+0000","last_undegraded":"2024-04-26T16:21:07.132214+0000","last_fullsized":"2024-04-26T16:21:07.132214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:20:12.898432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.131422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"121'13","reported_seq":134,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:16.092634+0000","last_change":"2024-04-26T16:21:16.092712+0000","last_active":"2024-04-26T16:21:16.092634+0000","last_peered":"2024-04-26T16:21:16.092634+0000","last_clean":"2024-04-26T16:21:16.092634+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:21:16.092634+0000","last_undegraded":"2024-04-26T16:21:16.092634+0000","last_fullsized":"2024-04-26T16:21:16.092634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'13","last_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:21:16.092465+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:25.892280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.748153+0000","last_change":"2024-04-26T16:20:43.860874+0000","last_active":"2024-04-26T16:21:07.748153+0000","last_peered":"2024-04-26T16:21:07.748153+0000","last_clean":"2024-04-26T16:21:07.748153+0000","last_became_active":"2024-04-26T16:17:35.070054+0000","last_became_peered":"2024-04-26T16:17:35.070054+0000","last_unstale":"2024-04-26T16:21:07.748153+0000","last_undegraded":"2024-04-26T16:21:07.748153+0000","last_fullsized":"2024-04-26T16:21:07.748153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:43.860659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:43.860659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:59.070145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":150,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.257148+0000","last_change":"2024-04-26T16:19:26.961234+0000","last_active":"2024-04-26T16:21:09.257148+0000","last_peered":"2024-04-26T16:21:09.257148+0000","last_clean":"2024-04-26T16:21:09.257148+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:21:09.257148+0000","last_undegraded":"2024-04-26T16:21:09.257148+0000","last_fullsized":"2024-04-26T16:21:09.257148+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:19:26.960994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:26.960994+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.996713+0000","last_change":"2024-04-26T16:20:15.241109+0000","last_active":"2024-04-26T16:21:06.996713+0000","last_peered":"2024-04-26T16:21:06.996713+0000","last_clean":"2024-04-26T16:21:06.996713+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:21:06.996713+0000","last_undegraded":"2024-04-26T16:21:06.996713+0000","last_fullsized":"2024-04-26T16:21:06.996713+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.241014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.241014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.603907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"121'13","reported_seq":134,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:13.384767+0000","last_change":"2024-04-26T16:21:13.384913+0000","last_active":"2024-04-26T16:21:13.384767+0000","last_peered":"2024-04-26T16:21:13.384767+0000","last_clean":"2024-04-26T16:21:13.384767+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:21:13.384767+0000","last_undegraded":"2024-04-26T16:21:13.384767+0000","last_fullsized":"2024-04-26T16:21:13.384767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'13","last_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:21:13.384551+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:35.904926+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.997664+0000","last_change":"2024-04-26T16:20:38.325004+0000","last_active":"2024-04-26T16:21:06.997664+0000","last_peered":"2024-04-26T16:21:06.997664+0000","last_clean":"2024-04-26T16:21:06.997664+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:21:06.997664+0000","last_undegraded":"2024-04-26T16:21:06.997664+0000","last_fullsized":"2024-04-26T16:21:06.997664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:38.324618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:38.324618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:50.422515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"101'12","reported_seq":168,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.254907+0000","last_change":"2024-04-26T16:20:59.519147+0000","last_active":"2024-04-26T16:21:09.254907+0000","last_peered":"2024-04-26T16:21:09.254907+0000","last_clean":"2024-04-26T16:21:09.254907+0000","last_became_active":"2024-04-26T16:17:33.062000+0000","last_became_peered":"2024-04-26T16:17:33.062000+0000","last_unstale":"2024-04-26T16:21:09.254907+0000","last_undegraded":"2024-04-26T16:21:09.254907+0000","last_fullsized":"2024-04-26T16:21:09.254907+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-26T16:20:59.518904+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:20:59.518904+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.666407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":111,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.554451+0000","last_change":"2024-04-26T16:20:05.684861+0000","last_active":"2024-04-26T16:21:07.554451+0000","last_peered":"2024-04-26T16:21:07.554451+0000","last_clean":"2024-04-26T16:21:07.554451+0000","last_became_active":"2024-04-26T16:17:29.025290+0000","last_became_peered":"2024-04-26T16:17:29.025290+0000","last_unstale":"2024-04-26T16:21:07.554451+0000","last_undegraded":"2024-04-26T16:21:07.554451+0000","last_fullsized":"2024-04-26T16:21:07.554451+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:05.684621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:05.684621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:12.223118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"86'11","reported_seq":133,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995363+0000","last_change":"2024-04-26T16:21:04.076152+0000","last_active":"2024-04-26T16:21:06.995363+0000","last_peered":"2024-04-26T16:21:06.995363+0000","last_clean":"2024-04-26T16:21:06.995363+0000","last_became_active":"2024-04-26T16:17:31.034570+0000","last_became_peered":"2024-04-26T16:17:31.034570+0000","last_unstale":"2024-04-26T16:21:06.995363+0000","last_undegraded":"2024-04-26T16:21:06.995363+0000","last_fullsized":"2024-04-26T16:21:06.995363+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:21:04.075909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:21:04.075909+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:23.124588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994402+0000","last_change":"2024-04-26T16:21:00.996835+0000","last_active":"2024-04-26T16:21:06.994402+0000","last_peered":"2024-04-26T16:21:06.994402+0000","last_clean":"2024-04-26T16:21:06.994402+0000","last_became_active":"2024-04-26T16:17:35.671754+0000","last_became_peered":"2024-04-26T16:17:35.671754+0000","last_unstale":"2024-04-26T16:21:06.994402+0000","last_undegraded":"2024-04-26T16:21:06.994402+0000","last_fullsized":"2024-04-26T16:21:06.994402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:00.996593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:21:00.996593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:11.543878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":103,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.555741+0000","last_change":"2024-04-26T16:20:49.524102+0000","last_active":"2024-04-26T16:21:07.555741+0000","last_peered":"2024-04-26T16:21:07.555741+0000","last_clean":"2024-04-26T16:21:07.555741+0000","last_became_active":"2024-04-26T16:17:33.050011+0000","last_became_peered":"2024-04-26T16:17:33.050011+0000","last_unstale":"2024-04-26T16:21:07.555741+0000","last_undegraded":"2024-04-26T16:21:07.555741+0000","last_fullsized":"2024-04-26T16:21:07.555741+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:49.523871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:49.523871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:13.761815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.131323+0000","last_change":"2024-04-26T16:20:44.944706+0000","last_active":"2024-04-26T16:21:07.131323+0000","last_peered":"2024-04-26T16:21:07.131323+0000","last_clean":"2024-04-26T16:21:07.131323+0000","last_became_active":"2024-04-26T16:17:28.999634+0000","last_became_peered":"2024-04-26T16:17:28.999634+0000","last_unstale":"2024-04-26T16:21:07.131323+0000","last_undegraded":"2024-04-26T16:21:07.131323+0000","last_fullsized":"2024-04-26T16:21:07.131323+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:44.944463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:44.944463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:09.941460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"86'4","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.130653+0000","last_change":"2024-04-26T16:20:26.899539+0000","last_active":"2024-04-26T16:21:07.130653+0000","last_peered":"2024-04-26T16:21:07.130653+0000","last_clean":"2024-04-26T16:21:07.130653+0000","last_became_active":"2024-04-26T16:17:31.026037+0000","last_became_peered":"2024-04-26T16:17:31.026037+0000","last_unstale":"2024-04-26T16:21:07.130653+0000","last_undegraded":"2024-04-26T16:21:07.130653+0000","last_fullsized":"2024-04-26T16:21:07.130653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'4","last_scrub_stamp":"2024-04-26T16:20:26.899299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:26.899299+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.728642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":106,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.429800+0000","last_change":"2024-04-26T16:21:02.459002+0000","last_active":"2024-04-26T16:21:07.429800+0000","last_peered":"2024-04-26T16:21:07.429800+0000","last_clean":"2024-04-26T16:21:07.429800+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:21:07.429800+0000","last_undegraded":"2024-04-26T16:21:07.429800+0000","last_fullsized":"2024-04-26T16:21:07.429800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:02.458777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:21:02.458777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:29.849746+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.194179+0000","last_change":"2024-04-26T16:20:33.855455+0000","last_active":"2024-04-26T16:21:07.194179+0000","last_peered":"2024-04-26T16:21:07.194179+0000","last_clean":"2024-04-26T16:21:07.194179+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:21:07.194179+0000","last_undegraded":"2024-04-26T16:21:07.194179+0000","last_fullsized":"2024-04-26T16:21:07.194179+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:33.855240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.855240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.904032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.428911+0000","last_change":"2024-04-26T16:20:27.565849+0000","last_active":"2024-04-26T16:21:07.428911+0000","last_peered":"2024-04-26T16:21:07.428911+0000","last_clean":"2024-04-26T16:21:07.428911+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:21:07.428911+0000","last_undegraded":"2024-04-26T16:21:07.428911+0000","last_fullsized":"2024-04-26T16:21:07.428911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:27.565529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:27.565529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.357713+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"86'11","reported_seq":125,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.993780+0000","last_change":"2024-04-26T16:20:14.278241+0000","last_active":"2024-04-26T16:21:06.993780+0000","last_peered":"2024-04-26T16:21:06.993780+0000","last_clean":"2024-04-26T16:21:06.993780+0000","last_became_active":"2024-04-26T16:17:31.033294+0000","last_became_peered":"2024-04-26T16:17:31.033294+0000","last_unstale":"2024-04-26T16:21:06.993780+0000","last_undegraded":"2024-04-26T16:21:06.993780+0000","last_fullsized":"2024-04-26T16:21:06.993780+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:14.278005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.278005+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.458748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.193663+0000","last_change":"2024-04-26T16:21:04.005652+0000","last_active":"2024-04-26T16:21:07.193663+0000","last_peered":"2024-04-26T16:21:07.193663+0000","last_clean":"2024-04-26T16:21:07.193663+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:21:07.193663+0000","last_undegraded":"2024-04-26T16:21:07.193663+0000","last_fullsized":"2024-04-26T16:21:07.193663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:04.005438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:21:04.005438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:32.525159+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":128,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.747789+0000","last_change":"2024-04-26T16:20:13.890462+0000","last_active":"2024-04-26T16:21:07.747789+0000","last_peered":"2024-04-26T16:21:07.747789+0000","last_clean":"2024-04-26T16:21:07.747789+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:21:07.747789+0000","last_undegraded":"2024-04-26T16:21:07.747789+0000","last_fullsized":"2024-04-26T16:21:07.747789+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:20:13.890111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.890111+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:23.062559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"86'11","reported_seq":133,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:08.084001+0000","last_change":"2024-04-26T16:21:08.084148+0000","last_active":"2024-04-26T16:21:08.084001+0000","last_peered":"2024-04-26T16:21:08.084001+0000","last_clean":"2024-04-26T16:21:08.084001+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:21:08.084001+0000","last_undegraded":"2024-04-26T16:21:08.084001+0000","last_fullsized":"2024-04-26T16:21:08.084001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:21:08.083777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:21:08.083777+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:09.870357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.428411+0000","last_change":"2024-04-26T16:20:14.406999+0000","last_active":"2024-04-26T16:21:07.428411+0000","last_peered":"2024-04-26T16:21:07.428411+0000","last_clean":"2024-04-26T16:21:07.428411+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:21:07.428411+0000","last_undegraded":"2024-04-26T16:21:07.428411+0000","last_fullsized":"2024-04-26T16:21:07.428411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.406805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.406805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:18.652307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994403+0000","last_change":"2024-04-26T16:20:32.257239+0000","last_active":"2024-04-26T16:21:06.994403+0000","last_peered":"2024-04-26T16:21:06.994403+0000","last_clean":"2024-04-26T16:21:06.994403+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:21:06.994403+0000","last_undegraded":"2024-04-26T16:21:06.994403+0000","last_fullsized":"2024-04-26T16:21:06.994403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:32.256998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:32.256998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:57.610172+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.001195+0000","last_change":"2024-04-26T16:20:38.996311+0000","last_active":"2024-04-26T16:21:07.001195+0000","last_peered":"2024-04-26T16:21:07.001195+0000","last_clean":"2024-04-26T16:21:07.001195+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:21:07.001195+0000","last_undegraded":"2024-04-26T16:21:07.001195+0000","last_fullsized":"2024-04-26T16:21:07.001195+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:38.996095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:38.996095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:47.489693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"86'16","reported_seq":175,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.255110+0000","last_change":"2024-04-26T16:20:21.529761+0000","last_active":"2024-04-26T16:21:09.255110+0000","last_peered":"2024-04-26T16:21:09.255110+0000","last_clean":"2024-04-26T16:21:09.255110+0000","last_became_active":"2024-04-26T16:17:31.040723+0000","last_became_peered":"2024-04-26T16:17:31.040723+0000","last_unstale":"2024-04-26T16:21:09.255110+0000","last_undegraded":"2024-04-26T16:21:09.255110+0000","last_fullsized":"2024-04-26T16:21:09.255110+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'16","last_scrub_stamp":"2024-04-26T16:20:21.529513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.529513+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.582915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.000712+0000","last_change":"2024-04-26T16:19:59.129440+0000","last_active":"2024-04-26T16:21:07.000712+0000","last_peered":"2024-04-26T16:21:07.000712+0000","last_clean":"2024-04-26T16:21:07.000712+0000","last_became_active":"2024-04-26T16:17:32.046437+0000","last_became_peered":"2024-04-26T16:17:32.046437+0000","last_unstale":"2024-04-26T16:21:07.000712+0000","last_undegraded":"2024-04-26T16:21:07.000712+0000","last_fullsized":"2024-04-26T16:21:07.000712+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:19:59.129167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.129167+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:15.401811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"86'11","reported_seq":169,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.379797+0000","last_change":"2024-04-26T16:21:09.379797+0000","last_active":"2024-04-26T16:21:09.379797+0000","last_peered":"2024-04-26T16:21:09.379797+0000","last_clean":"2024-04-26T16:21:09.379797+0000","last_became_active":"2024-04-26T16:17:33.069020+0000","last_became_peered":"2024-04-26T16:17:33.069020+0000","last_unstale":"2024-04-26T16:21:09.379797+0000","last_undegraded":"2024-04-26T16:21:09.379797+0000","last_fullsized":"2024-04-26T16:21:09.379797+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:21:09.379545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:21:09.379545+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:37.047948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.193799+0000","last_change":"2024-04-26T16:20:09.046364+0000","last_active":"2024-04-26T16:21:07.193799+0000","last_peered":"2024-04-26T16:21:07.193799+0000","last_clean":"2024-04-26T16:21:07.193799+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:21:07.193799+0000","last_undegraded":"2024-04-26T16:21:07.193799+0000","last_fullsized":"2024-04-26T16:21:07.193799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:09.045989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:09.045989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:32.237368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":111,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.554795+0000","last_change":"2024-04-26T16:20:13.525422+0000","last_active":"2024-04-26T16:21:07.554795+0000","last_peered":"2024-04-26T16:21:07.554795+0000","last_clean":"2024-04-26T16:21:07.554795+0000","last_became_active":"2024-04-26T16:17:29.022734+0000","last_became_peered":"2024-04-26T16:17:29.022734+0000","last_unstale":"2024-04-26T16:21:07.554795+0000","last_undegraded":"2024-04-26T16:21:07.554795+0000","last_fullsized":"2024-04-26T16:21:07.554795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:13.525151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.525151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.383572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"103'27","reported_seq":152,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.745225+0000","last_change":"2024-04-26T16:20:19.982781+0000","last_active":"2024-04-26T16:21:07.745225+0000","last_peered":"2024-04-26T16:21:07.745225+0000","last_clean":"2024-04-26T16:21:07.745225+0000","last_became_active":"2024-04-26T16:17:31.030904+0000","last_became_peered":"2024-04-26T16:17:31.030904+0000","last_unstale":"2024-04-26T16:21:07.745225+0000","last_undegraded":"2024-04-26T16:21:07.745225+0000","last_fullsized":"2024-04-26T16:21:07.745225+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'27","last_scrub_stamp":"2024-04-26T16:20:19.982389+0000","last_deep_scrub":"103'27","last_deep_scrub_stamp":"2024-04-26T16:18:40.947097+0000","last_clean_scrub_stamp":"2024-04-26T16:20:19.982389+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.646180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.998664+0000","last_change":"2024-04-26T16:20:24.200758+0000","last_active":"2024-04-26T16:21:06.998664+0000","last_peered":"2024-04-26T16:21:06.998664+0000","last_clean":"2024-04-26T16:21:06.998664+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:21:06.998664+0000","last_undegraded":"2024-04-26T16:21:06.998664+0000","last_fullsized":"2024-04-26T16:21:06.998664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:24.200448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.200448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:35.909147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994470+0000","last_change":"2024-04-26T16:20:16.270543+0000","last_active":"2024-04-26T16:21:06.994470+0000","last_peered":"2024-04-26T16:21:06.994470+0000","last_clean":"2024-04-26T16:21:06.994470+0000","last_became_active":"2024-04-26T16:17:35.670585+0000","last_became_peered":"2024-04-26T16:17:35.670585+0000","last_unstale":"2024-04-26T16:21:06.994470+0000","last_undegraded":"2024-04-26T16:21:06.994470+0000","last_fullsized":"2024-04-26T16:21:06.994470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:16.270321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:16.270321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:29.978435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":148,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.996811+0000","last_change":"2024-04-26T16:19:43.114960+0000","last_active":"2024-04-26T16:21:06.996811+0000","last_peered":"2024-04-26T16:21:06.996811+0000","last_clean":"2024-04-26T16:21:06.996811+0000","last_became_active":"2024-04-26T16:17:31.023021+0000","last_became_peered":"2024-04-26T16:17:31.023021+0000","last_unstale":"2024-04-26T16:21:06.996811+0000","last_undegraded":"2024-04-26T16:21:06.996811+0000","last_fullsized":"2024-04-26T16:21:06.996811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_clean_scrub_stamp":"2024-04-26T16:19:43.114691+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:09.523228+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.428660+0000","last_change":"2024-04-26T16:19:52.393976+0000","last_active":"2024-04-26T16:21:07.428660+0000","last_peered":"2024-04-26T16:21:07.428660+0000","last_clean":"2024-04-26T16:21:07.428660+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:21:07.428660+0000","last_undegraded":"2024-04-26T16:21:07.428660+0000","last_fullsized":"2024-04-26T16:21:07.428660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:52.393754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:19:52.393754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.078807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.429569+0000","last_change":"2024-04-26T16:19:59.359204+0000","last_active":"2024-04-26T16:21:07.429569+0000","last_peered":"2024-04-26T16:21:07.429569+0000","last_clean":"2024-04-26T16:21:07.429569+0000","last_became_active":"2024-04-26T16:17:33.066780+0000","last_became_peered":"2024-04-26T16:17:33.066780+0000","last_unstale":"2024-04-26T16:21:07.429569+0000","last_undegraded":"2024-04-26T16:21:07.429569+0000","last_fullsized":"2024-04-26T16:21:07.429569+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:59.358789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:19:59.358789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.479227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.556390+0000","last_change":"2024-04-26T16:20:51.461873+0000","last_active":"2024-04-26T16:21:07.556390+0000","last_peered":"2024-04-26T16:21:07.556390+0000","last_clean":"2024-04-26T16:21:07.556390+0000","last_became_active":"2024-04-26T16:17:35.087291+0000","last_became_peered":"2024-04-26T16:17:35.087291+0000","last_unstale":"2024-04-26T16:21:07.556390+0000","last_undegraded":"2024-04-26T16:21:07.556390+0000","last_fullsized":"2024-04-26T16:21:07.556390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:51.461656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:51.461656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.437880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"86'10","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192790+0000","last_change":"2024-04-26T16:20:04.994840+0000","last_active":"2024-04-26T16:21:07.192790+0000","last_peered":"2024-04-26T16:21:07.192790+0000","last_clean":"2024-04-26T16:21:07.192790+0000","last_became_active":"2024-04-26T16:17:31.048785+0000","last_became_peered":"2024-04-26T16:17:31.048785+0000","last_unstale":"2024-04-26T16:21:07.192790+0000","last_undegraded":"2024-04-26T16:21:07.192790+0000","last_fullsized":"2024-04-26T16:21:07.192790+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:04.994521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:04.994521+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:33.785270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":119,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:11.568788+0000","last_change":"2024-04-26T16:21:11.568846+0000","last_active":"2024-04-26T16:21:11.568788+0000","last_peered":"2024-04-26T16:21:11.568788+0000","last_clean":"2024-04-26T16:21:11.568788+0000","last_became_active":"2024-04-26T16:17:29.013835+0000","last_became_peered":"2024-04-26T16:17:29.013835+0000","last_unstale":"2024-04-26T16:21:11.568788+0000","last_undegraded":"2024-04-26T16:21:11.568788+0000","last_fullsized":"2024-04-26T16:21:11.568788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:11.568645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:21:11.568645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:21.639718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":398,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:10.044066+0000","last_change":"2024-04-26T16:21:10.044149+0000","last_active":"2024-04-26T16:21:10.044066+0000","last_peered":"2024-04-26T16:21:10.044066+0000","last_clean":"2024-04-26T16:21:10.044066+0000","last_became_active":"2024-04-26T16:16:50.925983+0000","last_became_peered":"2024-04-26T16:16:50.925983+0000","last_unstale":"2024-04-26T16:21:10.044066+0000","last_undegraded":"2024-04-26T16:21:10.044066+0000","last_fullsized":"2024-04-26T16:21:10.044066+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:21:10.043878+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:21:10.043878+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:25.567292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995030+0000","last_change":"2024-04-26T16:20:52.080952+0000","last_active":"2024-04-26T16:21:06.995030+0000","last_peered":"2024-04-26T16:21:06.995030+0000","last_clean":"2024-04-26T16:21:06.995030+0000","last_became_active":"2024-04-26T16:17:33.061816+0000","last_became_peered":"2024-04-26T16:17:33.061816+0000","last_unstale":"2024-04-26T16:21:06.995030+0000","last_undegraded":"2024-04-26T16:21:06.995030+0000","last_fullsized":"2024-04-26T16:21:06.995030+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:20:52.080653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.193004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.000876+0000","last_change":"2024-04-26T16:20:47.022860+0000","last_active":"2024-04-26T16:21:07.000876+0000","last_peered":"2024-04-26T16:21:07.000876+0000","last_clean":"2024-04-26T16:21:07.000876+0000","last_became_active":"2024-04-26T16:17:35.090592+0000","last_became_peered":"2024-04-26T16:17:35.090592+0000","last_unstale":"2024-04-26T16:21:07.000876+0000","last_undegraded":"2024-04-26T16:21:07.000876+0000","last_fullsized":"2024-04-26T16:21:07.000876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:20:47.022618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:12.217970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"112'21","reported_seq":142,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995475+0000","last_change":"2024-04-26T16:21:02.316590+0000","last_active":"2024-04-26T16:21:06.995475+0000","last_peered":"2024-04-26T16:21:06.995475+0000","last_clean":"2024-04-26T16:21:06.995475+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:21:06.995475+0000","last_undegraded":"2024-04-26T16:21:06.995475+0000","last_fullsized":"2024-04-26T16:21:06.995475+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'21","last_scrub_stamp":"2024-04-26T16:21:02.316340+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:21:02.316340+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.820401+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.993420+0000","last_change":"2024-04-26T16:20:30.226306+0000","last_active":"2024-04-26T16:21:06.993420+0000","last_peered":"2024-04-26T16:21:06.993420+0000","last_clean":"2024-04-26T16:21:06.993420+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:21:06.993420+0000","last_undegraded":"2024-04-26T16:21:06.993420+0000","last_fullsized":"2024-04-26T16:21:06.993420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:30.226091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:30.226091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.774441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":103,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.556223+0000","last_change":"2024-04-26T16:20:35.552710+0000","last_active":"2024-04-26T16:21:07.556223+0000","last_peered":"2024-04-26T16:21:07.556223+0000","last_clean":"2024-04-26T16:21:07.556223+0000","last_became_active":"2024-04-26T16:17:33.046107+0000","last_became_peered":"2024-04-26T16:17:33.046107+0000","last_unstale":"2024-04-26T16:21:07.556223+0000","last_undegraded":"2024-04-26T16:21:07.556223+0000","last_fullsized":"2024-04-26T16:21:07.556223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.552498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.552498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:44.056973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.996433+0000","last_change":"2024-04-26T16:20:15.021647+0000","last_active":"2024-04-26T16:21:06.996433+0000","last_peered":"2024-04-26T16:21:06.996433+0000","last_clean":"2024-04-26T16:21:06.996433+0000","last_became_active":"2024-04-26T16:17:35.673254+0000","last_became_peered":"2024-04-26T16:17:35.673254+0000","last_unstale":"2024-04-26T16:21:06.996433+0000","last_undegraded":"2024-04-26T16:21:06.996433+0000","last_fullsized":"2024-04-26T16:21:06.996433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:15.021515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:15.021515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:28.753340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994741+0000","last_change":"2024-04-26T16:20:24.308841+0000","last_active":"2024-04-26T16:21:06.994741+0000","last_peered":"2024-04-26T16:21:06.994741+0000","last_clean":"2024-04-26T16:21:06.994741+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:21:06.994741+0000","last_undegraded":"2024-04-26T16:21:06.994741+0000","last_fullsized":"2024-04-26T16:21:06.994741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:24.308595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:24.308595+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:53.172263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"107'40","reported_seq":214,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.256608+0000","last_change":"2024-04-26T16:20:25.051639+0000","last_active":"2024-04-26T16:21:09.256608+0000","last_peered":"2024-04-26T16:21:09.256608+0000","last_clean":"2024-04-26T16:21:09.256608+0000","last_became_active":"2024-04-26T16:17:31.024426+0000","last_became_peered":"2024-04-26T16:17:31.024426+0000","last_unstale":"2024-04-26T16:21:09.256608+0000","last_undegraded":"2024-04-26T16:21:09.256608+0000","last_fullsized":"2024-04-26T16:21:09.256608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'40","last_scrub_stamp":"2024-04-26T16:20:25.051382+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:18:56.138611+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.051382+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.857786+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.131883+0000","last_change":"2024-04-26T16:20:18.962156+0000","last_active":"2024-04-26T16:21:07.131883+0000","last_peered":"2024-04-26T16:21:07.131883+0000","last_clean":"2024-04-26T16:21:07.131883+0000","last_became_active":"2024-04-26T16:17:33.057789+0000","last_became_peered":"2024-04-26T16:17:33.057789+0000","last_unstale":"2024-04-26T16:21:07.131883+0000","last_undegraded":"2024-04-26T16:21:07.131883+0000","last_fullsized":"2024-04-26T16:21:07.131883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.961959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:53.089835+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.961959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:43.337483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.746765+0000","last_change":"2024-04-26T16:19:57.996481+0000","last_active":"2024-04-26T16:21:07.746765+0000","last_peered":"2024-04-26T16:21:07.746765+0000","last_clean":"2024-04-26T16:21:07.746765+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:21:07.746765+0000","last_undegraded":"2024-04-26T16:21:07.746765+0000","last_fullsized":"2024-04-26T16:21:07.746765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:57.996170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:19:57.996170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:19.461563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.001678+0000","last_change":"2024-04-26T16:20:18.047229+0000","last_active":"2024-04-26T16:21:07.001678+0000","last_peered":"2024-04-26T16:21:07.001678+0000","last_clean":"2024-04-26T16:21:07.001678+0000","last_became_active":"2024-04-26T16:17:29.004853+0000","last_became_peered":"2024-04-26T16:17:29.004853+0000","last_unstale":"2024-04-26T16:21:07.001678+0000","last_undegraded":"2024-04-26T16:21:07.001678+0000","last_fullsized":"2024-04-26T16:21:07.001678+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.047049+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:30.615066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"121'18","reported_seq":138,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:15.048044+0000","last_change":"2024-04-26T16:21:15.048231+0000","last_active":"2024-04-26T16:21:15.048044+0000","last_peered":"2024-04-26T16:21:15.048044+0000","last_clean":"2024-04-26T16:21:15.048044+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:21:15.048044+0000","last_undegraded":"2024-04-26T16:21:15.048044+0000","last_fullsized":"2024-04-26T16:21:15.048044+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'18","last_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:21:15.047780+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:25.205823+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":321,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:14.905047+0000","last_change":"2024-04-26T16:20:11.073651+0000","last_active":"2024-04-26T16:21:14.905047+0000","last_peered":"2024-04-26T16:21:14.905047+0000","last_clean":"2024-04-26T16:21:14.905047+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:21:14.905047+0000","last_undegraded":"2024-04-26T16:21:14.905047+0000","last_fullsized":"2024-04-26T16:21:14.905047+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:20:11.073395+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:19:04.959471+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.073395+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:27.691968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":207,"num_read_kb":202,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.745850+0000","last_change":"2024-04-26T16:20:21.947248+0000","last_active":"2024-04-26T16:21:07.745850+0000","last_peered":"2024-04-26T16:21:07.745850+0000","last_clean":"2024-04-26T16:21:07.745850+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:21:07.745850+0000","last_undegraded":"2024-04-26T16:21:07.745850+0000","last_fullsized":"2024-04-26T16:21:07.745850+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.946995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.814368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994516+0000","last_change":"2024-04-26T16:20:17.304956+0000","last_active":"2024-04-26T16:21:06.994516+0000","last_peered":"2024-04-26T16:21:06.994516+0000","last_clean":"2024-04-26T16:21:06.994516+0000","last_became_active":"2024-04-26T16:17:35.670672+0000","last_became_peered":"2024-04-26T16:17:35.670672+0000","last_unstale":"2024-04-26T16:21:06.994516+0000","last_undegraded":"2024-04-26T16:21:06.994516+0000","last_fullsized":"2024-04-26T16:21:06.994516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:17.304722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:17.304722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.500725+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.132932+0000","last_change":"2024-04-26T16:19:58.917701+0000","last_active":"2024-04-26T16:21:07.132932+0000","last_peered":"2024-04-26T16:21:07.132932+0000","last_clean":"2024-04-26T16:21:07.132932+0000","last_became_active":"2024-04-26T16:17:29.006985+0000","last_became_peered":"2024-04-26T16:17:29.006985+0000","last_unstale":"2024-04-26T16:21:07.132932+0000","last_undegraded":"2024-04-26T16:21:07.132932+0000","last_fullsized":"2024-04-26T16:21:07.132932+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:19:58.917484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:16.115592+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"107'18","reported_seq":130,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.999528+0000","last_change":"2024-04-26T16:20:13.290879+0000","last_active":"2024-04-26T16:21:06.999528+0000","last_peered":"2024-04-26T16:21:06.999528+0000","last_clean":"2024-04-26T16:21:06.999528+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:21:06.999528+0000","last_undegraded":"2024-04-26T16:21:06.999528+0000","last_fullsized":"2024-04-26T16:21:06.999528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'18","last_scrub_stamp":"2024-04-26T16:20:13.290621+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:19:05.704281+0000","last_clean_scrub_stamp":"2024-04-26T16:20:13.290621+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:21.892369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":121,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:13.667863+0000","last_change":"2024-04-26T16:21:13.667925+0000","last_active":"2024-04-26T16:21:13.667863+0000","last_peered":"2024-04-26T16:21:13.667863+0000","last_clean":"2024-04-26T16:21:13.667863+0000","last_became_active":"2024-04-26T16:17:32.027388+0000","last_became_peered":"2024-04-26T16:17:32.027388+0000","last_unstale":"2024-04-26T16:21:13.667863+0000","last_undegraded":"2024-04-26T16:21:13.667863+0000","last_fullsized":"2024-04-26T16:21:13.667863+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:13.667701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:13.667701+0000","last_clean_scrub_stamp":"2024-04-26T16:21:13.667701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:28.701807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":159,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.256451+0000","last_change":"2024-04-26T16:21:06.017992+0000","last_active":"2024-04-26T16:21:09.256451+0000","last_peered":"2024-04-26T16:21:09.256451+0000","last_clean":"2024-04-26T16:21:09.256451+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:21:09.256451+0000","last_undegraded":"2024-04-26T16:21:09.256451+0000","last_fullsized":"2024-04-26T16:21:09.256451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:21:06.017712+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:12.646942+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192850+0000","last_change":"2024-04-26T16:20:10.026900+0000","last_active":"2024-04-26T16:21:07.192850+0000","last_peered":"2024-04-26T16:21:07.192850+0000","last_clean":"2024-04-26T16:21:07.192850+0000","last_became_active":"2024-04-26T16:17:35.673130+0000","last_became_peered":"2024-04-26T16:17:35.673130+0000","last_unstale":"2024-04-26T16:21:07.192850+0000","last_undegraded":"2024-04-26T16:21:07.192850+0000","last_fullsized":"2024-04-26T16:21:07.192850+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:10.026659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.026659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:39.688645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":103,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.556290+0000","last_change":"2024-04-26T16:20:28.439735+0000","last_active":"2024-04-26T16:21:07.556290+0000","last_peered":"2024-04-26T16:21:07.556290+0000","last_clean":"2024-04-26T16:21:07.556290+0000","last_became_active":"2024-04-26T16:17:33.068923+0000","last_became_peered":"2024-04-26T16:17:33.068923+0000","last_unstale":"2024-04-26T16:21:07.556290+0000","last_undegraded":"2024-04-26T16:21:07.556290+0000","last_fullsized":"2024-04-26T16:21:07.556290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.439524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.246043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.993651+0000","last_change":"2024-04-26T16:20:10.311712+0000","last_active":"2024-04-26T16:21:06.993651+0000","last_peered":"2024-04-26T16:21:06.993651+0000","last_clean":"2024-04-26T16:21:06.993651+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:21:06.993651+0000","last_undegraded":"2024-04-26T16:21:06.993651+0000","last_fullsized":"2024-04-26T16:21:06.993651+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:20:10.311472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:10.311472+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:20.794801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"86'12","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.745361+0000","last_change":"2024-04-26T16:20:33.858656+0000","last_active":"2024-04-26T16:21:07.745361+0000","last_peered":"2024-04-26T16:21:07.745361+0000","last_clean":"2024-04-26T16:21:07.745361+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:21:07.745361+0000","last_undegraded":"2024-04-26T16:21:07.745361+0000","last_fullsized":"2024-04-26T16:21:07.745361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:20:33.858421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:33.858421+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:54.983533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.555240+0000","last_change":"2024-04-26T16:20:23.546630+0000","last_active":"2024-04-26T16:21:07.555240+0000","last_peered":"2024-04-26T16:21:07.555240+0000","last_clean":"2024-04-26T16:21:07.555240+0000","last_became_active":"2024-04-26T16:17:35.077832+0000","last_became_peered":"2024-04-26T16:17:35.077832+0000","last_unstale":"2024-04-26T16:21:07.555240+0000","last_undegraded":"2024-04-26T16:21:07.555240+0000","last_fullsized":"2024-04-26T16:21:07.555240+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.546406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.546406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:48.610382+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"115'12","reported_seq":161,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.254901+0000","last_change":"2024-04-26T16:20:25.975628+0000","last_active":"2024-04-26T16:21:09.254901+0000","last_peered":"2024-04-26T16:21:09.254901+0000","last_clean":"2024-04-26T16:21:09.254901+0000","last_became_active":"2024-04-26T16:17:33.070130+0000","last_became_peered":"2024-04-26T16:17:33.070130+0000","last_unstale":"2024-04-26T16:21:09.254901+0000","last_undegraded":"2024-04-26T16:21:09.254901+0000","last_fullsized":"2024-04-26T16:21:09.254901+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:20:25.975399+0000","last_clean_scrub_stamp":"2024-04-26T16:20:25.975399+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.920318+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995825+0000","last_change":"2024-04-26T16:20:35.047323+0000","last_active":"2024-04-26T16:21:06.995825+0000","last_peered":"2024-04-26T16:21:06.995825+0000","last_clean":"2024-04-26T16:21:06.995825+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:21:06.995825+0000","last_undegraded":"2024-04-26T16:21:06.995825+0000","last_fullsized":"2024-04-26T16:21:06.995825+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:35.046847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.046847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:51.436102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"86'15","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.192265+0000","last_change":"2024-04-26T16:20:36.954635+0000","last_active":"2024-04-26T16:21:07.192265+0000","last_peered":"2024-04-26T16:21:07.192265+0000","last_clean":"2024-04-26T16:21:07.192265+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:21:07.192265+0000","last_undegraded":"2024-04-26T16:21:07.192265+0000","last_fullsized":"2024-04-26T16:21:07.192265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:36.954394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:36.954394+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:05.655723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.747762+0000","last_change":"2024-04-26T16:20:14.907782+0000","last_active":"2024-04-26T16:21:07.747762+0000","last_peered":"2024-04-26T16:21:07.747762+0000","last_clean":"2024-04-26T16:21:07.747762+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:21:07.747762+0000","last_undegraded":"2024-04-26T16:21:07.747762+0000","last_fullsized":"2024-04-26T16:21:07.747762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.907547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.907547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:34.001180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"86'11","reported_seq":158,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.256459+0000","last_change":"2024-04-26T16:20:03.405476+0000","last_active":"2024-04-26T16:21:09.256459+0000","last_peered":"2024-04-26T16:21:09.256459+0000","last_clean":"2024-04-26T16:21:09.256459+0000","last_became_active":"2024-04-26T16:17:33.066299+0000","last_became_peered":"2024-04-26T16:17:33.066299+0000","last_unstale":"2024-04-26T16:21:09.256459+0000","last_undegraded":"2024-04-26T16:21:09.256459+0000","last_fullsized":"2024-04-26T16:21:09.256459+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:20:03.405246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:20:03.405246+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.150208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.000495+0000","last_change":"2024-04-26T16:20:29.023874+0000","last_active":"2024-04-26T16:21:07.000495+0000","last_peered":"2024-04-26T16:21:07.000495+0000","last_clean":"2024-04-26T16:21:07.000495+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:21:07.000495+0000","last_undegraded":"2024-04-26T16:21:07.000495+0000","last_fullsized":"2024-04-26T16:21:07.000495+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:29.023717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:20:29.023717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.818594+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"86'9","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.191764+0000","last_change":"2024-04-26T16:19:20.985067+0000","last_active":"2024-04-26T16:21:07.191764+0000","last_peered":"2024-04-26T16:21:07.191764+0000","last_clean":"2024-04-26T16:21:07.191764+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:21:07.191764+0000","last_undegraded":"2024-04-26T16:21:07.191764+0000","last_fullsized":"2024-04-26T16:21:07.191764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:19:20.984815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:19:20.984815+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.429944+0000","last_change":"2024-04-26T16:20:21.525478+0000","last_active":"2024-04-26T16:21:07.429944+0000","last_peered":"2024-04-26T16:21:07.429944+0000","last_clean":"2024-04-26T16:21:07.429944+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:21:07.429944+0000","last_undegraded":"2024-04-26T16:21:07.429944+0000","last_fullsized":"2024-04-26T16:21:07.429944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:20:21.525230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.078478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":104,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.000896+0000","last_change":"2024-04-26T16:20:23.025108+0000","last_active":"2024-04-26T16:21:07.000896+0000","last_peered":"2024-04-26T16:21:07.000896+0000","last_clean":"2024-04-26T16:21:07.000896+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:21:07.000896+0000","last_undegraded":"2024-04-26T16:21:07.000896+0000","last_fullsized":"2024-04-26T16:21:07.000896+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:23.024885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:01.215041+0000","last_clean_scrub_stamp":"2024-04-26T16:20:23.024885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.206757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995153+0000","last_change":"2024-04-26T16:20:36.297478+0000","last_active":"2024-04-26T16:21:06.995153+0000","last_peered":"2024-04-26T16:21:06.995153+0000","last_clean":"2024-04-26T16:21:06.995153+0000","last_became_active":"2024-04-26T16:17:29.026793+0000","last_became_peered":"2024-04-26T16:17:29.026793+0000","last_unstale":"2024-04-26T16:21:06.995153+0000","last_undegraded":"2024-04-26T16:21:06.995153+0000","last_fullsized":"2024-04-26T16:21:06.995153+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:20:36.297212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:03.276670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"86'19","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.130723+0000","last_change":"2024-04-26T16:20:11.904568+0000","last_active":"2024-04-26T16:21:07.130723+0000","last_peered":"2024-04-26T16:21:07.130723+0000","last_clean":"2024-04-26T16:21:07.130723+0000","last_became_active":"2024-04-26T16:17:31.024436+0000","last_became_peered":"2024-04-26T16:17:31.024436+0000","last_unstale":"2024-04-26T16:21:07.130723+0000","last_undegraded":"2024-04-26T16:21:07.130723+0000","last_fullsized":"2024-04-26T16:21:07.130723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'19","last_scrub_stamp":"2024-04-26T16:20:11.904292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:11.904292+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:17.775021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.993330+0000","last_change":"2024-04-26T16:20:47.169490+0000","last_active":"2024-04-26T16:21:06.993330+0000","last_peered":"2024-04-26T16:21:06.993330+0000","last_clean":"2024-04-26T16:21:06.993330+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:21:06.993330+0000","last_undegraded":"2024-04-26T16:21:06.993330+0000","last_fullsized":"2024-04-26T16:21:06.993330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:20:47.169235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:12.965763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.428974+0000","last_change":"2024-04-26T16:21:04.417404+0000","last_active":"2024-04-26T16:21:07.428974+0000","last_peered":"2024-04-26T16:21:07.428974+0000","last_clean":"2024-04-26T16:21:07.428974+0000","last_became_active":"2024-04-26T16:17:33.065671+0000","last_became_peered":"2024-04-26T16:17:33.065671+0000","last_unstale":"2024-04-26T16:21:07.428974+0000","last_undegraded":"2024-04-26T16:21:07.428974+0000","last_fullsized":"2024-04-26T16:21:07.428974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:04.417182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:21:04.417182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:17.533499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":118,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:10.417935+0000","last_change":"2024-04-26T16:21:10.417990+0000","last_active":"2024-04-26T16:21:10.417935+0000","last_peered":"2024-04-26T16:21:10.417935+0000","last_clean":"2024-04-26T16:21:10.417935+0000","last_became_active":"2024-04-26T16:17:29.017152+0000","last_became_peered":"2024-04-26T16:17:29.017152+0000","last_unstale":"2024-04-26T16:21:10.417935+0000","last_undegraded":"2024-04-26T16:21:10.417935+0000","last_fullsized":"2024-04-26T16:21:10.417935+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:10.417785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:21:10.417785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:20.521867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"86'17","reported_seq":134,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995469+0000","last_change":"2024-04-26T16:20:35.289963+0000","last_active":"2024-04-26T16:21:06.995469+0000","last_peered":"2024-04-26T16:21:06.995469+0000","last_clean":"2024-04-26T16:21:06.995469+0000","last_became_active":"2024-04-26T16:17:31.030244+0000","last_became_peered":"2024-04-26T16:17:31.030244+0000","last_unstale":"2024-04-26T16:21:06.995469+0000","last_undegraded":"2024-04-26T16:21:06.995469+0000","last_fullsized":"2024-04-26T16:21:06.995469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-04-26T16:20:35.289727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:35.289727+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:46.877432+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.994990+0000","last_change":"2024-04-26T16:20:14.277898+0000","last_active":"2024-04-26T16:21:06.994990+0000","last_peered":"2024-04-26T16:21:06.994990+0000","last_clean":"2024-04-26T16:21:06.994990+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:21:06.994990+0000","last_undegraded":"2024-04-26T16:21:06.994990+0000","last_fullsized":"2024-04-26T16:21:06.994990+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:14.277671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:14.277671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:25.766764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:13.387401+0000","last_change":"2024-04-26T16:21:13.387464+0000","last_active":"2024-04-26T16:21:13.387401+0000","last_peered":"2024-04-26T16:21:13.387401+0000","last_clean":"2024-04-26T16:21:13.387401+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:21:13.387401+0000","last_undegraded":"2024-04-26T16:21:13.387401+0000","last_fullsized":"2024-04-26T16:21:13.387401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:21:13.387247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:21:13.387247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:27.474660+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":112,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.001443+0000","last_change":"2024-04-26T16:20:43.031893+0000","last_active":"2024-04-26T16:21:07.001443+0000","last_peered":"2024-04-26T16:21:07.001443+0000","last_clean":"2024-04-26T16:21:07.001443+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:21:07.001443+0000","last_undegraded":"2024-04-26T16:21:07.001443+0000","last_fullsized":"2024-04-26T16:21:07.001443+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:20:43.031634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:55.462599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"86'10","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.555241+0000","last_change":"2024-04-26T16:20:34.526337+0000","last_active":"2024-04-26T16:21:07.555241+0000","last_peered":"2024-04-26T16:21:07.555241+0000","last_clean":"2024-04-26T16:21:07.555241+0000","last_became_active":"2024-04-26T16:17:31.040215+0000","last_became_peered":"2024-04-26T16:17:31.040215+0000","last_unstale":"2024-04-26T16:21:07.555241+0000","last_undegraded":"2024-04-26T16:21:07.555241+0000","last_fullsized":"2024-04-26T16:21:07.555241+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:20:34.526105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:34.526105+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:49.517534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":107,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.554702+0000","last_change":"2024-04-26T16:20:55.467042+0000","last_active":"2024-04-26T16:21:07.554702+0000","last_peered":"2024-04-26T16:21:07.554702+0000","last_clean":"2024-04-26T16:21:07.554702+0000","last_became_active":"2024-04-26T16:17:35.669627+0000","last_became_peered":"2024-04-26T16:17:35.669627+0000","last_unstale":"2024-04-26T16:21:07.554702+0000","last_undegraded":"2024-04-26T16:21:07.554702+0000","last_fullsized":"2024-04-26T16:21:07.554702+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:55.466831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:20:55.466831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:23.012141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:07.429915+0000","last_change":"2024-04-26T16:20:18.397841+0000","last_active":"2024-04-26T16:21:07.429915+0000","last_peered":"2024-04-26T16:21:07.429915+0000","last_clean":"2024-04-26T16:21:07.429915+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:21:07.429915+0000","last_undegraded":"2024-04-26T16:21:07.429915+0000","last_fullsized":"2024-04-26T16:21:07.429915+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:20:18.397696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:50.542517+0000","last_clean_scrub_stamp":"2024-04-26T16:20:18.397696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:22.381027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"86'15","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:06.995971+0000","last_change":"2024-04-26T16:20:28.238717+0000","last_active":"2024-04-26T16:21:06.995971+0000","last_peered":"2024-04-26T16:21:06.995971+0000","last_clean":"2024-04-26T16:21:06.995971+0000","last_became_active":"2024-04-26T16:17:31.033340+0000","last_became_peered":"2024-04-26T16:17:31.033340+0000","last_unstale":"2024-04-26T16:21:06.995971+0000","last_undegraded":"2024-04-26T16:21:06.995971+0000","last_fullsized":"2024-04-26T16:21:06.995971+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:20:28.238415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:20:28.238415+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:41.888574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"109'12","reported_seq":159,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-04-26T16:21:09.256048+0000","last_change":"2024-04-26T16:20:55.047342+0000","last_active":"2024-04-26T16:21:09.256048+0000","last_peered":"2024-04-26T16:21:09.256048+0000","last_clean":"2024-04-26T16:21:09.256048+0000","last_became_active":"2024-04-26T16:17:33.070525+0000","last_became_peered":"2024-04-26T16:17:33.070525+0000","last_unstale":"2024-04-26T16:21:09.256048+0000","last_undegraded":"2024-04-26T16:21:09.256048+0000","last_fullsized":"2024-04-26T16:21:09.256048+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-26T16:20:55.047110+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_clean_scrub_stamp":"2024-04-26T16:20:55.047110+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:22:01.558186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":207,"num_read_kb":202,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743480,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28348,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716804,"statfs":{"total":95995035648,"available":95966007296,"internally_reserved":0,"allocated":1363968,"data_stored":988280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135938,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28312,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716840,"statfs":{"total":95995035648,"available":95966044160,"internally_reserved":0,"allocated":1327104,"data_stored":985509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":27654601},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":396478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36656,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708496,"statfs":{"total":95995035648,"available":95957499904,"internally_reserved":0,"allocated":1351680,"data_stored":987593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":737280,"data_stored":395634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084384,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36056,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93709096,"statfs":{"total":95995035648,"available":95958114304,"internally_reserved":0,"allocated":737280,"data_stored":395541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855446,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":778240,"data_stored":396522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863462988,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":741376,"data_stored":395556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:21:19.004 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-26T16:21:19.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump 2024-04-26T16:21:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:19 smithi190 bash[23868]: cluster 2024-04-26T16:21:17.468982+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:21:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:19 smithi190 bash[23868]: cluster 2024-04-26T16:21:17.468982+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:21:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:19 smithi190 bash[23868]: cluster 2024-04-26T16:21:17.471370+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:21:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:19 smithi190 bash[23868]: cluster 2024-04-26T16:21:17.471370+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:21:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:19 smithi190 bash[23868]: audit 2024-04-26T16:21:18.120046+0000 mgr.y (mgr.24401) 143 : audit [DBG] from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:19 smithi190 bash[23868]: audit 2024-04-26T16:21:18.120046+0000 mgr.y (mgr.24401) 143 : audit [DBG] from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[21463]: cluster 2024-04-26T16:21:17.468982+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[21463]: cluster 2024-04-26T16:21:17.468982+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[21463]: cluster 2024-04-26T16:21:17.471370+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[21463]: cluster 2024-04-26T16:21:17.471370+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[21463]: audit 2024-04-26T16:21:18.120046+0000 mgr.y (mgr.24401) 143 : audit [DBG] from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[21463]: audit 2024-04-26T16:21:18.120046+0000 mgr.y (mgr.24401) 143 : audit [DBG] from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[29234]: cluster 2024-04-26T16:21:17.468982+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[29234]: cluster 2024-04-26T16:21:17.468982+0000 osd.5 (osd.5) 95 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:21:19.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[29234]: cluster 2024-04-26T16:21:17.471370+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:21:19.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[29234]: cluster 2024-04-26T16:21:17.471370+0000 osd.5 (osd.5) 96 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:21:19.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[29234]: audit 2024-04-26T16:21:18.120046+0000 mgr.y (mgr.24401) 143 : audit [DBG] from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:19.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:19 smithi050 bash[29234]: audit 2024-04-26T16:21:18.120046+0000 mgr.y (mgr.24401) 143 : audit [DBG] from='client.24599 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.237182+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.237182+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.239288+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.239288+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.477964+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.477964+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.480105+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:18.480105+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.418014+0000 mgr.y (mgr.24401) 144 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:21:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:20 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.418014+0000 mgr.y (mgr.24401) 144 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.237182+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.237182+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.239288+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.239288+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.477964+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.477964+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:21:20.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.480105+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:18.480105+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.418014+0000 mgr.y (mgr.24401) 144 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.418014+0000 mgr.y (mgr.24401) 144 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.237182+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.237182+0000 osd.6 (osd.6) 53 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.239288+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.239288+0000 osd.6 (osd.6) 54 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.477964+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.477964+0000 osd.5 (osd.5) 97 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.480105+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:18.480105+0000 osd.5 (osd.5) 98 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:21:20.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.418014+0000 mgr.y (mgr.24401) 144 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:21:20.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:20 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.418014+0000 mgr.y (mgr.24401) 144 : cluster [DBG] pgmap v111: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.218399+0000 osd.6 (osd.6) 55 : cluster [DBG] 3.a scrub starts 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.218399+0000 osd.6 (osd.6) 55 : cluster [DBG] 3.a scrub starts 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.223355+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.a scrub ok 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.223355+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.a scrub ok 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.794398+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.794398+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.796612+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:21:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:21 smithi190 bash[23868]: cluster 2024-04-26T16:21:19.796612+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.218399+0000 osd.6 (osd.6) 55 : cluster [DBG] 3.a scrub starts 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.218399+0000 osd.6 (osd.6) 55 : cluster [DBG] 3.a scrub starts 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.223355+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.a scrub ok 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.223355+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.a scrub ok 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.794398+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.794398+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.796612+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:21:21.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[21463]: cluster 2024-04-26T16:21:19.796612+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.218399+0000 osd.6 (osd.6) 55 : cluster [DBG] 3.a scrub starts 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.218399+0000 osd.6 (osd.6) 55 : cluster [DBG] 3.a scrub starts 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.223355+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.a scrub ok 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.223355+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.a scrub ok 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.794398+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.794398+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.796612+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:21:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:21 smithi050 bash[29234]: cluster 2024-04-26T16:21:19.796612+0000 osd.4 (osd.4) 80 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.332447+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.332447+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.334915+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.334915+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.768957+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.768957+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.771196+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:20.771196+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.419642+0000 mgr.y (mgr.24401) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:22.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.419642+0000 mgr.y (mgr.24401) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.332447+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.332447+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.334915+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.334915+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.768957+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.768957+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.771196+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:20.771196+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.419642+0000 mgr.y (mgr.24401) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:22.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:22 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.419642+0000 mgr.y (mgr.24401) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.332447+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.332447+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.334915+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.334915+0000 osd.2 (osd.2) 68 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.768957+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.768957+0000 osd.4 (osd.4) 81 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.771196+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:20.771196+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.419642+0000 mgr.y (mgr.24401) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:22.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:22 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.419642+0000 mgr.y (mgr.24401) 145 : cluster [DBG] pgmap v112: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.418537+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.418537+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.418651+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.418651+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.503219+0000 osd.5 (osd.5) 99 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.503219+0000 osd.5 (osd.5) 99 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.505483+0000 osd.5 (osd.5) 100 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:21.505483+0000 osd.5 (osd.5) 100 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.086127+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.086127+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.262582+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:21:23.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.262582+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:21:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.267536+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:21:23.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:23 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.267536+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.418537+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.418537+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.418651+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.418651+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.503219+0000 osd.5 (osd.5) 99 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.503219+0000 osd.5 (osd.5) 99 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.505483+0000 osd.5 (osd.5) 100 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:21.505483+0000 osd.5 (osd.5) 100 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:21:23.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.086127+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.086127+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.262582+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.262582+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.267536+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.267536+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.418537+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.418537+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.418651+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.418651+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.503219+0000 osd.5 (osd.5) 99 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.503219+0000 osd.5 (osd.5) 99 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:21:23.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.505483+0000 osd.5 (osd.5) 100 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:21.505483+0000 osd.5 (osd.5) 100 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.086127+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.086127+0000 mon.a (mon.0) 933 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.262582+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.262582+0000 osd.1 (osd.1) 89 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.267536+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:21:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:23 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.267536+0000 osd.1 (osd.1) 90 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:21:23.813 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.252287+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.252287+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.254728+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.254728+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.527843+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.527843+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:21:24.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.531104+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.531104+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.788541+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.788541+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.790993+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:22.790993+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.013121+0000 osd.3 (osd.3) 97 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.013121+0000 osd.3 (osd.3) 97 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.022940+0000 osd.3 (osd.3) 98 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.022940+0000 osd.3 (osd.3) 98 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.420300+0000 mgr.y (mgr.24401) 146 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.420300+0000 mgr.y (mgr.24401) 146 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.252287+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:21:24.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.252287+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.254728+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.254728+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.527843+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.527843+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.531104+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.531104+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.788541+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.788541+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.790993+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:22.790993+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.013121+0000 osd.3 (osd.3) 97 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.013121+0000 osd.3 (osd.3) 97 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.022940+0000 osd.3 (osd.3) 98 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:21:24.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.022940+0000 osd.3 (osd.3) 98 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:21:24.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.420300+0000 mgr.y (mgr.24401) 146 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:24.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.420300+0000 mgr.y (mgr.24401) 146 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.252287+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.252287+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.254728+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.254728+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.527843+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.527843+0000 osd.5 (osd.5) 101 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.531104+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:21:24.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.531104+0000 osd.5 (osd.5) 102 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.788541+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.788541+0000 osd.4 (osd.4) 83 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.790993+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:22.790993+0000 osd.4 (osd.4) 84 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.013121+0000 osd.3 (osd.3) 97 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.013121+0000 osd.3 (osd.3) 97 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.022940+0000 osd.3 (osd.3) 98 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.022940+0000 osd.3 (osd.3) 98 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.420300+0000 mgr.y (mgr.24401) 146 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:24.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:24 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.420300+0000 mgr.y (mgr.24401) 146 : cluster [DBG] pgmap v114: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:24.874 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:21:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:21:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:21:25.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.778182+0000 osd.4 (osd.4) 85 : cluster [DBG] 2.f scrub starts 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.778182+0000 osd.4 (osd.4) 85 : cluster [DBG] 2.f scrub starts 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.781310+0000 osd.4 (osd.4) 86 : cluster [DBG] 2.f scrub ok 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:23.781310+0000 osd.4 (osd.4) 86 : cluster [DBG] 2.f scrub ok 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.384979+0000 osd.0 (osd.0) 89 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.384979+0000 osd.0 (osd.0) 89 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.387487+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.387487+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.778182+0000 osd.4 (osd.4) 85 : cluster [DBG] 2.f scrub starts 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.778182+0000 osd.4 (osd.4) 85 : cluster [DBG] 2.f scrub starts 2024-04-26T16:21:25.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.781310+0000 osd.4 (osd.4) 86 : cluster [DBG] 2.f scrub ok 2024-04-26T16:21:25.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:23.781310+0000 osd.4 (osd.4) 86 : cluster [DBG] 2.f scrub ok 2024-04-26T16:21:25.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.384979+0000 osd.0 (osd.0) 89 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:21:25.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.384979+0000 osd.0 (osd.0) 89 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:21:25.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.387487+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:21:25.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:25 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.387487+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:21:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.778182+0000 osd.4 (osd.4) 85 : cluster [DBG] 2.f scrub starts 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.778182+0000 osd.4 (osd.4) 85 : cluster [DBG] 2.f scrub starts 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.781310+0000 osd.4 (osd.4) 86 : cluster [DBG] 2.f scrub ok 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:23.781310+0000 osd.4 (osd.4) 86 : cluster [DBG] 2.f scrub ok 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.384979+0000 osd.0 (osd.0) 89 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.384979+0000 osd.0 (osd.0) 89 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.387487+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:21:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:25 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.387487+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:21:26.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.195305+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:21:26.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.195305+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:21:26.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.198660+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:21:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: cluster 2024-04-26T16:21:24.198660+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:21:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: cluster 2024-04-26T16:21:25.420759+0000 mgr.y (mgr.24401) 147 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: cluster 2024-04-26T16:21:25.420759+0000 mgr.y (mgr.24401) 147 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: audit 2024-04-26T16:21:25.776667+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[21463]: audit 2024-04-26T16:21:25.776667+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.195305+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.195305+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.198660+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: cluster 2024-04-26T16:21:24.198660+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: cluster 2024-04-26T16:21:25.420759+0000 mgr.y (mgr.24401) 147 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: cluster 2024-04-26T16:21:25.420759+0000 mgr.y (mgr.24401) 147 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: audit 2024-04-26T16:21:25.776667+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:26.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:26 smithi050 bash[29234]: audit 2024-04-26T16:21:25.776667+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.195305+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.195305+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.198660+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: cluster 2024-04-26T16:21:24.198660+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: cluster 2024-04-26T16:21:25.420759+0000 mgr.y (mgr.24401) 147 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: cluster 2024-04-26T16:21:25.420759+0000 mgr.y (mgr.24401) 147 : cluster [DBG] pgmap v115: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: audit 2024-04-26T16:21:25.776667+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:26.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[23868]: audit 2024-04-26T16:21:25.776667+0000 mon.c (mon.2) 80 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:26.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:21:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[21463]: cluster 2024-04-26T16:21:25.365383+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[21463]: cluster 2024-04-26T16:21:25.365383+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[21463]: cluster 2024-04-26T16:21:25.367153+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[21463]: cluster 2024-04-26T16:21:25.367153+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[21463]: audit 2024-04-26T16:21:26.494235+0000 mgr.y (mgr.24401) 148 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[21463]: audit 2024-04-26T16:21:26.494235+0000 mgr.y (mgr.24401) 148 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:27.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[29234]: cluster 2024-04-26T16:21:25.365383+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:21:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[29234]: cluster 2024-04-26T16:21:25.365383+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:21:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[29234]: cluster 2024-04-26T16:21:25.367153+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:21:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[29234]: cluster 2024-04-26T16:21:25.367153+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:21:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[29234]: audit 2024-04-26T16:21:26.494235+0000 mgr.y (mgr.24401) 148 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:27 smithi050 bash[29234]: audit 2024-04-26T16:21:26.494235+0000 mgr.y (mgr.24401) 148 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:27.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:27 smithi190 bash[23868]: cluster 2024-04-26T16:21:25.365383+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:21:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:27 smithi190 bash[23868]: cluster 2024-04-26T16:21:25.365383+0000 osd.2 (osd.2) 71 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:21:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:27 smithi190 bash[23868]: cluster 2024-04-26T16:21:25.367153+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:21:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:27 smithi190 bash[23868]: cluster 2024-04-26T16:21:25.367153+0000 osd.2 (osd.2) 72 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:21:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:27 smithi190 bash[23868]: audit 2024-04-26T16:21:26.494235+0000 mgr.y (mgr.24401) 148 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:27 smithi190 bash[23868]: audit 2024-04-26T16:21:26.494235+0000 mgr.y (mgr.24401) 148 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2320249636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56641094c0 msgr2=0x7f56641098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2320249636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56641094c0 0x7f56641098a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5654009990 tx=0x7f5654009ca0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2320249636 shutdown_connections 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2320249636 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103e30 0x7f5664110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2320249636 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56641094c0 0x7f56641098a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2320249636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5664103470 0x7f56641038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2320249636 >> 172.21.15.50:0/2320249636 conn(0x7f566407a580 msgr2=0x7f566407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:27.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2320249636 shutdown_connections 2024-04-26T16:21:27.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2320249636 wait complete. 2024-04-26T16:21:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 librados: starting msgr at 2024-04-26T16:21:27.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 librados: starting objecter 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 Processor -- start 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- start start 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 librados: setting wanted keys 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 librados: calling monclient init 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 0x7f5664075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5664103e30 0x7f5664075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56641094c0 0x7f56640761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5664112ab0 con 0x7f5664103470 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5664112930 con 0x7f56641094c0 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5664112c30 con 0x7f5664103e30 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 0x7f5664075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:27.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f565b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5664103e30 0x7f5664075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 0x7f5664075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33380/0 (socket says 172.21.15.50:33380) 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 -- 172.21.15.50:0/2553490329 learned_addr learned my addr 172.21.15.50:0/2553490329 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5668b4b700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56641094c0 0x7f56640761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 -- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5664103e30 msgr2=0x7f5664075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5664103e30 0x7f5664075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 -- 172.21.15.50:0/2553490329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56641094c0 msgr2=0x7f56640761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56641094c0 0x7f56640761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 -- 172.21.15.50:0/2553490329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56641a76e0 con 0x7f5664103470 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5663fff700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 0x7f5664075720 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5654000c00 tx=0x7f5654004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5661ffb700 10 client.?.objecter ms_handle_connect 0x7f5664103470 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5661ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654027070 con 0x7f5664103470 2024-04-26T16:21:27.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5654022470 con 0x7f5664103470 2024-04-26T16:21:27.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654017650 con 0x7f5664103470 2024-04-26T16:21:27.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56641a7970 con 0x7f5664103470 2024-04-26T16:21:27.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 10 client.14715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:21:27.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641a7e50 con 0x7f5664103470 2024-04-26T16:21:27.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.588+0000 7f566a5ae700 1 librados: init done 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566406a110 con 0x7f5664103470 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f565402a070 con 0x7f5664103470 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5650071f80 0x7f5650074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(122..122 src has 67..122) ==== 7450+0+0 (secure 0 0 0) 0x7f56540922c0 con 0x7f5664103470 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 10 client.14715.objecter ms_dispatch 0x7f5664001550 osd_map(122..122 src has 67..122) 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 3 client.14715.objecter handle_osd_map got epochs [122,122] > 0 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 3 client.14715.objecter handle_osd_map decoding full epoch 122 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f565b7fe700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5650071f80 0x7f5650074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:27.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 20 client.14715.objecter dump_active .. 0 homeless 2024-04-26T16:21:27.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f565b7fe700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5650071f80 0x7f5650074440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f566407a9a0 tx=0x7f564c00b3c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:27.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.592+0000 7f5661ffb700 10 client.14715.objecter ms_handle_connect 0x7f5650071f80 2024-04-26T16:21:27.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.596+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5654013070 con 0x7f5664103470 2024-04-26T16:21:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.820+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f56641a8140 con 0x7f5664103470 2024-04-26T16:21:27.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f5661ffb700 1 -- 172.21.15.50:0/2553490329 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v122) ==== 56+0+5157 (secure 0 0 0) 0x7f565405e450 con 0x7f5664103470 2024-04-26T16:21:27.817 INFO:teuthology.orchestra.run.smithi050.stdout:epoch 122 2024-04-26T16:21:27.817 INFO:teuthology.orchestra.run.smithi050.stdout:fsid cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:21:27.817 INFO:teuthology.orchestra.run.smithi050.stdout:created 2024-04-26T16:07:04.494213+0000 2024-04-26T16:21:27.817 INFO:teuthology.orchestra.run.smithi050.stdout:modified 2024-04-26T16:21:22.067543+0000 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:crush_version 18 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:full_ratio 0.95 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:backfillfull_ratio 0.9 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:nearfull_ratio 0.85 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:require_min_compat_client luminous 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:min_compat_client luminous 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:require_osd_release squid 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:stretch_mode_enabled false 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:pool 3 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-04-26T16:21:27.818 INFO:teuthology.orchestra.run.smithi050.stdout:pool 4 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 89 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:max_osd 8 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] [v2:172.21.15.50:6804/310950989,v1:172.21.15.50:6805/310950989] exists,up b46ea8aa-cd97-425b-90e0-1e70af46b8f1 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1 up in weight 1 up_from 13 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] [v2:172.21.15.50:6812/3516275849,v1:172.21.15.50:6813/3516275849] exists,up 7042f8e0-f5d0-48d2-9cd6-d9445e326eed 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] [v2:172.21.15.50:6820/2835380668,v1:172.21.15.50:6821/2835380668] exists,up 33069daa-7fd8-4e9c-aafe-89030f84aad2 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3 up in weight 1 up_from 28 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] [v2:172.21.15.50:6828/2541695077,v1:172.21.15.50:6829/2541695077] exists,up 43b6aab5-9b42-4f97-a224-b2872de5cb03 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4 up in weight 1 up_from 35 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] [v2:172.21.15.190:6802/2207965503,v1:172.21.15.190:6803/2207965503] exists,up f7220491-e9bf-4188-b203-322e9908558c 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] [v2:172.21.15.190:6810/3373187844,v1:172.21.15.190:6811/3373187844] exists,up ed5c0044-658b-4dfb-8c61-727de6bdd587 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6 up in weight 1 up_from 57 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] [v2:172.21.15.190:6818/2399532574,v1:172.21.15.190:6819/2399532574] exists,up 94111b7d-7887-4a3a-983b-394294735d0e 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7 up in weight 1 up_from 69 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] [v2:172.21.15.190:6826/2856617796,v1:172.21.15.190:6827/2856617796] exists,up abbd95e3-07b4-4235-9101-a4dea0f35bb5 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:pg_upmap_items 2.9 [7,2] 2024-04-26T16:21:27.819 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2324264713 expires 2024-04-27T16:08:14.593126+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/542146552 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/127073210 expires 2024-04-27T16:07:37.705234+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/127073210 expires 2024-04-27T16:07:37.705234+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/126446279 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/126446279 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/3454723725 expires 2024-04-27T16:07:37.705234+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2548717623 expires 2024-04-27T16:07:37.705234+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6800/685732569 expires 2024-04-27T16:08:14.593126+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2903276422 expires 2024-04-27T16:07:37.705234+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:6801/685732569 expires 2024-04-27T16:08:14.593126+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/2979962403 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/71201882 expires 2024-04-27T16:08:14.593126+0000 2024-04-26T16:21:27.820 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/3973398204 expires 2024-04-27T16:08:14.593126+0000 2024-04-26T16:21:27.821 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/1172169336 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.821 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/3444006067 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.821 INFO:teuthology.orchestra.run.smithi050.stdout:blocklist 172.21.15.50:0/577039843 expires 2024-04-27T16:18:25.317826+0000 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 10 librados: watch_flush enter 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 10 librados: watch_flush exit 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 20 client.14715.objecter shutdown clearing up homeless session... 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 10 client.14715.objecter successfully canceled tick 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5650071f80 msgr2=0x7f5650074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5650071f80 0x7f5650074440 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f566407a9a0 tx=0x7f564c00b3c0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 msgr2=0x7f5664075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 0x7f5664075720 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5654000c00 tx=0x7f5654004790 comp rx=0 tx=0).stop 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 shutdown_connections 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5650071f80 0x7f5650074440 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56641094c0 0x7f56640761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5664103470 0x7f5664075720 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 --2- 172.21.15.50:0/2553490329 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5664103e30 0x7f5664075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:27.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 >> 172.21.15.50:0/2553490329 conn(0x7f566407a580 msgr2=0x7f5664101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:27.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 shutdown_connections 2024-04-26T16:21:27.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 -- 172.21.15.50:0/2553490329 wait complete. 2024-04-26T16:21:27.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:27.824+0000 7f566a5ae700 1 librados: shutdown 2024-04-26T16:21:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:26.318820+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:21:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:26.318820+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:21:28.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:26.320002+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:26.320002+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:27.331053+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:27.331053+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:27.334242+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:27.334242+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:27.421844+0000 mgr.y (mgr.24401) 149 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: cluster 2024-04-26T16:21:27.421844+0000 mgr.y (mgr.24401) 149 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: audit 2024-04-26T16:21:27.826339+0000 mon.a (mon.0) 934 : audit [DBG] from='client.? 172.21.15.50:0/2553490329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[21463]: audit 2024-04-26T16:21:27.826339+0000 mon.a (mon.0) 934 : audit [DBG] from='client.? 172.21.15.50:0/2553490329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:26.318820+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:26.318820+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:21:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:26.320002+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:26.320002+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:27.331053+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:27.331053+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:27.334242+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:27.334242+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:27.421844+0000 mgr.y (mgr.24401) 149 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: cluster 2024-04-26T16:21:27.421844+0000 mgr.y (mgr.24401) 149 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: audit 2024-04-26T16:21:27.826339+0000 mon.a (mon.0) 934 : audit [DBG] from='client.? 172.21.15.50:0/2553490329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:21:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:28 smithi050 bash[29234]: audit 2024-04-26T16:21:27.826339+0000 mon.a (mon.0) 934 : audit [DBG] from='client.? 172.21.15.50:0/2553490329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:26.318820+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:26.318820+0000 osd.2 (osd.2) 73 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:26.320002+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:26.320002+0000 osd.2 (osd.2) 74 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:27.331053+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:27.331053+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:27.334242+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:27.334242+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:27.421844+0000 mgr.y (mgr.24401) 149 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: cluster 2024-04-26T16:21:27.421844+0000 mgr.y (mgr.24401) 149 : cluster [DBG] pgmap v116: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:28.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: audit 2024-04-26T16:21:27.826339+0000 mon.a (mon.0) 934 : audit [DBG] from='client.? 172.21.15.50:0/2553490329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:21:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:28 smithi190 bash[23868]: audit 2024-04-26T16:21:27.826339+0000 mon.a (mon.0) 934 : audit [DBG] from='client.? 172.21.15.50:0/2553490329' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:21:28.649 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] [v2:172.21.15.50:6804/310950989,v1:172.21.15.50:6805/310950989] exists,up b46ea8aa-cd97-425b-90e0-1e70af46b8f1', 'osd.1 up in weight 1 up_from 13 up_thru 97 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] [v2:172.21.15.50:6812/3516275849,v1:172.21.15.50:6813/3516275849] exists,up 7042f8e0-f5d0-48d2-9cd6-d9445e326eed', 'osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] [v2:172.21.15.50:6820/2835380668,v1:172.21.15.50:6821/2835380668] exists,up 33069daa-7fd8-4e9c-aafe-89030f84aad2', 'osd.3 up in weight 1 up_from 28 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] [v2:172.21.15.50:6828/2541695077,v1:172.21.15.50:6829/2541695077] exists,up 43b6aab5-9b42-4f97-a224-b2872de5cb03', 'osd.4 up in weight 1 up_from 35 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] [v2:172.21.15.190:6802/2207965503,v1:172.21.15.190:6803/2207965503] exists,up f7220491-e9bf-4188-b203-322e9908558c', 'osd.5 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] [v2:172.21.15.190:6810/3373187844,v1:172.21.15.190:6811/3373187844] exists,up ed5c0044-658b-4dfb-8c61-727de6bdd587', 'osd.6 up in weight 1 up_from 57 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] [v2:172.21.15.190:6818/2399532574,v1:172.21.15.190:6819/2399532574] exists,up 94111b7d-7887-4a3a-983b-394294735d0e', 'osd.7 up in weight 1 up_from 69 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] [v2:172.21.15.190:6826/2856617796,v1:172.21.15.190:6827/2856617796] exists,up abbd95e3-07b4-4235-9101-a4dea0f35bb5'] 2024-04-26T16:21:28.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon mon.a config show 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.045331+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.045331+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.048167+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.048167+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.068449+0000 osd.6 (osd.6) 61 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.068449+0000 osd.6 (osd.6) 61 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.071734+0000 osd.6 (osd.6) 62 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:21:29.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.071734+0000 osd.6 (osd.6) 62 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.330441+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.330441+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.332896+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.332896+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.510086+0000 osd.1 (osd.1) 91 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.510086+0000 osd.1 (osd.1) 91 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.510183+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.510183+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:29.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.045331+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.045331+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.048167+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.048167+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.068449+0000 osd.6 (osd.6) 61 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.068449+0000 osd.6 (osd.6) 61 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.071734+0000 osd.6 (osd.6) 62 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.071734+0000 osd.6 (osd.6) 62 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.330441+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.330441+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.332896+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.332896+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.510086+0000 osd.1 (osd.1) 91 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.510086+0000 osd.1 (osd.1) 91 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:29.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.510183+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:29.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:29 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.510183+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.045331+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.045331+0000 osd.3 (osd.3) 99 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.048167+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.048167+0000 osd.3 (osd.3) 100 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.068449+0000 osd.6 (osd.6) 61 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.068449+0000 osd.6 (osd.6) 61 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.071734+0000 osd.6 (osd.6) 62 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.071734+0000 osd.6 (osd.6) 62 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:21:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.330441+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.330441+0000 osd.0 (osd.0) 93 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.332896+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.332896+0000 osd.0 (osd.0) 94 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.510086+0000 osd.1 (osd.1) 91 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.510086+0000 osd.1 (osd.1) 91 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.510183+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:29 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.510183+0000 osd.1 (osd.1) 92 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.314451+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.d scrub starts 2024-04-26T16:21:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.314451+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.d scrub starts 2024-04-26T16:21:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.317571+0000 osd.2 (osd.2) 76 : cluster [DBG] 5.d scrub ok 2024-04-26T16:21:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:28.317571+0000 osd.2 (osd.2) 76 : cluster [DBG] 5.d scrub ok 2024-04-26T16:21:30.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.009798+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.009798+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.013041+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.013041+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.120388+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.120388+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.122942+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.122942+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.130022+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.130022+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.422487+0000 mgr.y (mgr.24401) 150 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:30.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.422487+0000 mgr.y (mgr.24401) 150 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.314451+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.d scrub starts 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.314451+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.d scrub starts 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.317571+0000 osd.2 (osd.2) 76 : cluster [DBG] 5.d scrub ok 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:28.317571+0000 osd.2 (osd.2) 76 : cluster [DBG] 5.d scrub ok 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.009798+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.009798+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.013041+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.013041+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.120388+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.120388+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.122942+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.122942+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:21:30.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.130022+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:21:30.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.130022+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:21:30.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.422487+0000 mgr.y (mgr.24401) 150 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:30.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:30 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.422487+0000 mgr.y (mgr.24401) 150 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.314451+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.d scrub starts 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.314451+0000 osd.2 (osd.2) 75 : cluster [DBG] 5.d scrub starts 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.317571+0000 osd.2 (osd.2) 76 : cluster [DBG] 5.d scrub ok 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:28.317571+0000 osd.2 (osd.2) 76 : cluster [DBG] 5.d scrub ok 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.009798+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.009798+0000 osd.3 (osd.3) 101 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.013041+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:21:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.013041+0000 osd.3 (osd.3) 102 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.120388+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.120388+0000 mon.a (mon.0) 935 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.122942+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.122942+0000 osd.7 (osd.7) 113 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.130022+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.130022+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.422487+0000 mgr.y (mgr.24401) 150 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:30 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.422487+0000 mgr.y (mgr.24401) 150 : cluster [DBG] pgmap v118: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.971203+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:21:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.971203+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:21:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.973552+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:21:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:29.973552+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:21:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.039706+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.039706+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.041872+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.041872+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.122498+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.122498+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.127939+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:21:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:31 smithi190 bash[23868]: cluster 2024-04-26T16:21:30.127939+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.971203+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.971203+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.973552+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:29.973552+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.039706+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.039706+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.041872+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:21:31.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.041872+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.122498+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.122498+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.127939+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[21463]: cluster 2024-04-26T16:21:30.127939+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.971203+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.971203+0000 osd.3 (osd.3) 103 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.973552+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:29.973552+0000 osd.3 (osd.3) 104 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:21:31.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.039706+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.039706+0000 osd.6 (osd.6) 63 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.041872+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.041872+0000 osd.6 (osd.6) 64 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.122498+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.122498+0000 osd.7 (osd.7) 115 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.127939+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:21:31.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:31 smithi050 bash[29234]: cluster 2024-04-26T16:21:30.127939+0000 osd.7 (osd.7) 116 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.138016+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.138016+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.141121+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.141121+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.254228+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.254228+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.258935+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.258935+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.358573+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:21:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.358573+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:21:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.360995+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:21:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.360995+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:21:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.423387+0000 mgr.y (mgr.24401) 151 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:32 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.423387+0000 mgr.y (mgr.24401) 151 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.138016+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.138016+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.141121+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.141121+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.254228+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.254228+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:21:32.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.258935+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.258935+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.358573+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.358573+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.360995+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.360995+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.423387+0000 mgr.y (mgr.24401) 151 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.423387+0000 mgr.y (mgr.24401) 151 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.138016+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.138016+0000 osd.7 (osd.7) 117 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.141121+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:21:32.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.141121+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.254228+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.254228+0000 osd.0 (osd.0) 95 : cluster [DBG] 3.6 deep-scrub starts 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.258935+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.258935+0000 osd.0 (osd.0) 96 : cluster [DBG] 3.6 deep-scrub ok 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.358573+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.358573+0000 osd.2 (osd.2) 77 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.360995+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.360995+0000 osd.2 (osd.2) 78 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.423387+0000 mgr.y (mgr.24401) 151 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:32 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.423387+0000 mgr.y (mgr.24401) 151 : cluster [DBG] pgmap v119: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:33.444 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:33.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.523765+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:21:33.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.523765+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:21:33.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.526611+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:21:33.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.526611+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:21:33.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.991348+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:21:33.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.991348+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.994680+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:31.994680+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.090503+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.090503+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.092942+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.092942+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.234748+0000 osd.0 (osd.0) 97 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.234748+0000 osd.0 (osd.0) 97 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.237099+0000 osd.0 (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[29234]: cluster 2024-04-26T16:21:32.237099+0000 osd.0 (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:21:33.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.523765+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.523765+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.526611+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.526611+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.991348+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.991348+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.994680+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:31.994680+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.090503+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.090503+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.092942+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.092942+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.234748+0000 osd.0 (osd.0) 97 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.234748+0000 osd.0 (osd.0) 97 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.237099+0000 osd.0 (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:21:33.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:33 smithi050 bash[21463]: cluster 2024-04-26T16:21:32.237099+0000 osd.0 (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.523765+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.523765+0000 osd.5 (osd.5) 103 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.526611+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.526611+0000 osd.5 (osd.5) 104 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.991348+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.991348+0000 osd.3 (osd.3) 105 : cluster [DBG] 4.0 deep-scrub starts 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.994680+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:31.994680+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.0 deep-scrub ok 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.090503+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.090503+0000 osd.1 (osd.1) 93 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:21:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.092942+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:21:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.092942+0000 osd.1 (osd.1) 94 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:21:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.234748+0000 osd.0 (osd.0) 97 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:21:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.234748+0000 osd.0 (osd.0) 97 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:21:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.237099+0000 osd.0 (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:21:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:33 smithi190 bash[23868]: cluster 2024-04-26T16:21:32.237099+0000 osd.0 (osd.0) 98 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:21:34.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.069405+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.069405+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.073121+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.073121+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.120951+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.120951+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.123087+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.123087+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.424034+0000 mgr.y (mgr.24401) 152 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.424034+0000 mgr.y (mgr.24401) 152 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.069405+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:21:34.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.069405+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.073121+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.073121+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.120951+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.120951+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.123087+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.123087+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.424034+0000 mgr.y (mgr.24401) 152 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:34.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.424034+0000 mgr.y (mgr.24401) 152 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.069405+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.069405+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.073121+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.073121+0000 osd.1 (osd.1) 96 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.120951+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.120951+0000 osd.7 (osd.7) 119 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.123087+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.123087+0000 osd.7 (osd.7) 120 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.424034+0000 mgr.y (mgr.24401) 152 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:34 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.424034+0000 mgr.y (mgr.24401) 152 : cluster [DBG] pgmap v120: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:34.714 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:21:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:21:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.347020+0000 osd.4 (osd.4) 87 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.347020+0000 osd.4 (osd.4) 87 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.347127+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:33.347127+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.027185+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.027185+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.029295+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.029295+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.167528+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-04-26T16:21:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:35 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.167528+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.347020+0000 osd.4 (osd.4) 87 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.347020+0000 osd.4 (osd.4) 87 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.347127+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:33.347127+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.027185+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.027185+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.029295+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.029295+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:21:35.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.167528+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.167528+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.347020+0000 osd.4 (osd.4) 87 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.347020+0000 osd.4 (osd.4) 87 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.347127+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:33.347127+0000 osd.4 (osd.4) 88 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.027185+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.027185+0000 osd.1 (osd.1) 97 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.029295+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:21:35.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.029295+0000 osd.1 (osd.1) 98 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:21:35.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.167528+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-04-26T16:21:35.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:35 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.167528+0000 mon.a (mon.0) 936 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2024-04-26T16:21:36.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.993283+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:21:36.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.993283+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:21:36.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.995838+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:34.995838+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:35.048132+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:35.048132+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:35.050530+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:35.050530+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:35.424733+0000 mgr.y (mgr.24401) 153 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:36.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[23868]: cluster 2024-04-26T16:21:35.424733+0000 mgr.y (mgr.24401) 153 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.993283+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.993283+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.995838+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:34.995838+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:35.048132+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:35.048132+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:35.050530+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:35.050530+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:35.424733+0000 mgr.y (mgr.24401) 153 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:36.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[29234]: cluster 2024-04-26T16:21:35.424733+0000 mgr.y (mgr.24401) 153 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.993283+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.993283+0000 osd.3 (osd.3) 107 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.995838+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:34.995838+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:35.048132+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:35.048132+0000 osd.1 (osd.1) 99 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:35.050530+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:35.050530+0000 osd.1 (osd.1) 100 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:35.424733+0000 mgr.y (mgr.24401) 153 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:36.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:36 smithi050 bash[21463]: cluster 2024-04-26T16:21:35.424733+0000 mgr.y (mgr.24401) 153 : cluster [DBG] pgmap v122: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:36.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:21:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:21:36.952 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:21:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "name": "mon.a", 2024-04-26T16:21:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster": "ceph", 2024-04-26T16:21:36.952 INFO:teuthology.orchestra.run.smithi050.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "admin_socket_mode": "", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_client_required": "cephx, none", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_cluster_required": "cephx", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_debug": "true", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_service_required": "cephx", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_service_ticket_ttl": "240.000000", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "auth_supported": "", 2024-04-26T16:21:36.953 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio": "true", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_max_queue_depth": "1024", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_poll_ms": "250", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_aio_reap_max": "16", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_async_discard_threads": "0", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_block_size": "4096", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio": "false", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_debug_inflight_ios": "false", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_enable_discard": "false", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_flock_retry": "3", 2024-04-26T16:21:36.954 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_flock_retry_interval": "0.100000", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_inject_crash": "0", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_inject_crash_flush_delay": "2", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring": "false", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring_hipri": "false", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_ioring_sqthread_poll": "false", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_read_buffer_alignment": "4096", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bdev_type": "", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_alloc_size": "1048576", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_allocator": "hybrid", 2024-04-26T16:21:36.955 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_buffered_io": "true", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_for_zeros": "false", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_volume_selector_often": "false", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_compact_log_sync": "false", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_compact_min_size": "16777216", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_log_replay_check_allocations": "true", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_max_log_runway": "4194304", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_max_prefetch": "1048576", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_min_flush_size": "524288", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_min_log_runway": "1048576", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_replay_recovery": "false", 2024-04-26T16:21:36.956 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_shared_alloc_size": "65536", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluefs_sync_write": "false", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_allocation_from_file": "true", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_allocator": "hybrid", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-04-26T16:21:36.957 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_blobid_prealloc": "10240", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_create": "true", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_create": "false", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_path": "", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_db_size": "0", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_path": "", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_preallocate_file": "false", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_size": "107374182400", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_create": "false", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_path": "", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_block_wal_size": "100663296", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs": "true", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-04-26T16:21:36.958 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_env_mirror": "false", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_autotune": "true", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size": "0", 2024-04-26T16:21:36.959 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cache_type": "2q", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_clone_cow": "true", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_algorithm": "snappy", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size": "0", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size": "0", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_mode": "none", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-04-26T16:21:36.960 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_csum_type": "crc32c", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_enforce_settings": "default", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_freelist": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_fsck_abort": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_inject_read_err": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_legacy_omap": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_prefill": "0.000000", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-04-26T16:21:36.961 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_small_allocations": "0", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_default_buffered_read": "true", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_default_buffered_write": "false", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops": "0", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_elastic_shared_blobs": "true", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-04-26T16:21:36.962 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fail_eio": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mkfs": "true", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mount": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_umount": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-04-26T16:21:36.963 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_ignore_data_csum": "false", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_kvbackend": "rocksdb", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_log_op_age": "5.000000", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_alloc_size": "0", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size": "0", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_defer_interval": "3.000000", 2024-04-26T16:21:36.964 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_max_deferred_txc": "32", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size": "0", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_nid_prealloc": "1024", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size": "0", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_qfsck_on_mount": "true", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_retry_disk_reads": "3", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_cf": "true", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_rocksdb_options_annex": "", 2024-04-26T16:21:36.965 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_coremask": "0x1", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_io_sleep": "5", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_max_io_completion": "0", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_spdk_mem": "512", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_sync_submit_transaction": "false", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_bytes": "67108864", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io": "0", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_tracing": "false", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_reserved": "0", 2024-04-26T16:21:36.966 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "bluestore_zero_block_detection": "false", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_action_update_interval": "2", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_mount_timeout": "10", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-04-26T16:21:36.967 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_cluster_require_signatures": "false", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_cluster_require_version": "2", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_require_signatures": "false", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_require_version": "2", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_service_require_signatures": "false", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_service_require_version": "2", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "cephx_sign_messages": "true", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "chdir": "", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client_acl_type": "", 2024-04-26T16:21:36.968 INFO:teuthology.orchestra.run.smithi050.stdout: "client_asio_thread_count": "2", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_cache_mid": "0.750000", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_cache_size": "16384", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_caps_release_delay": "5", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_check_pool_perm": "true", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_collect_and_send_global_metrics": "false", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_force_sync_read": "false", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_getattr_caps": "false", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_debug_inject_tick_delay": "0", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_die_on_failed_remount": "false", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_dirsize_rbytes": "true", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_force_lazyio": "false", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_fs": "", 2024-04-26T16:21:36.969 INFO:teuthology.orchestra.run.smithi050.stdout: "client_inject_fixed_oldest_tid": "false", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_inject_release_failure": "false", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_max_inline_size": "4096", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_max_retries_on_remount_failure": "5", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mds_namespace": "", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_metadata": "", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_gid": "-1", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_timeout": "300", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mount_uid": "-1", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_mountpoint": "/", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_notify_timeout": "10", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc": "true", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_dirty": "104857600", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_dirty_age": "5.000000", 2024-04-26T16:21:36.970 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_max_objects": "1000", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_size": "209715200", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_oc_target_dirty": "8388608", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_permissions": "true", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_quota": "true", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_quota_df": "true", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_max_bytes": "0", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_max_periods": "4", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_readahead_min": "131072", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_reconnect_stale": "false", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_shutdown_timeout": "30", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_snapdir": ".snap", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_tick_interval": "1", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_trace": "", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_try_dentry_invalidate": "false", 2024-04-26T16:21:36.971 INFO:teuthology.orchestra.run.smithi050.stdout: "client_use_faked_inos": "false", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "client_use_random_mds": "false", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog": "false", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_graylog_port": "12201", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_monitors": "default=true", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog": "false", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "clog_to_syslog_level": "info", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "-", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_network": "", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_network_interface": "", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_isal": "false", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_level": "5", 2024-04-26T16:21:36.972 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zlib_winsize": "-15", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "compressor_zstd_level": "1", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b71a575c225eecd96353b1d64f11c7118b9683e4e8491c513d9f702bd9a1f45", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_alien_op_num_threads": "6", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_alien_thread_cpu_cores": "", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_obc_lru_size": "512", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_osd_stat_interval": "0", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crimson_seastar_cpu_cores": "", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location": "", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location_hook": "", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_location_hook_timeout": "10", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "daemonize": "false", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "daos_pool": "tank", 2024-04-26T16:21:36.973 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "dbstore_db_name_prefix": "dbstore", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_alienstore": "0/5", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_allow_any_pool_priority": "false", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asok": "1/5", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asok_assert_abort": "false", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_asserts_on_shutdown": "true", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_auth": "1/5", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bdev": "1/3", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bluefs": "1/5", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_bluestore": "1/5", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_buffer": "0/1", 2024-04-26T16:21:36.974 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ceph_dedup": "0/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ceph_exporter": "1/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cephfs_mirror": "0/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cephsqlite": "0/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_client": "0/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_compressor": "1/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_context": "0/1", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crimson_interrupt": "0/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crush": "1/1", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_crypto": "1/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_cyanstore": "0/5", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_deliberately_leak_memory": "false", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_disable_randomized_ping": "false", 2024-04-26T16:21:36.975 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_dpdk": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_eventtrace": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_filer": "0/1", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_filestore": "1/3", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_finisher": "1/1", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_fuse": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_heartbeat_testing_span": "0", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_heartbeatmap": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_immutable_obj_cache": "0/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_javaclient": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_journal": "1/3", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_journaler": "0/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_kstore": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_lockdep": "0/1", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mclock": "1/5", 2024-04-26T16:21:36.976 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_balancer": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_locker": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_log": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_log_expire": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_migrator": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mds_quiesce": "3/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_memstore": "1/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mgr": "2/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mgrc": "20/20", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_mon": "20/20", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_monc": "0/10", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_ms": "1/1", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_none": "0/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objclass": "0/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objectcacher": "0/5", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_objecter": "0/1", 2024-04-26T16:21:36.977 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_optracker": "0/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_osd": "1/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_paxos": "20/20", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_perfcounter": "1/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_prioritycache": "1/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rados": "0/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd": "0/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_mirror": "0/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_pwl": "0/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rbd_replay": "0/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_refs": "0/0", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_reserver": "1/1", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw": "1/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_access": "1/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_datacache": "1/5", 2024-04-26T16:21:36.978 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_dbstore": "1/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_flight": "1/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_lifecycle": "1/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_notification": "1/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rgw_sync": "1/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_rocksdb": "4/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_backref": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_cache": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_cleaner": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_device": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_epm": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_journal": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_lba": "0/5", 2024-04-26T16:21:36.979 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_odata": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_omap": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_onode": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_t": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_seastore_tm": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_striper": "0/1", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_test": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_throttle": "1/1", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_timer": "0/1", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_tp": "0/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "debug_trace": "1/5", 2024-04-26T16:21:36.980 INFO:teuthology.orchestra.run.smithi050.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "device_failure_prediction_mode": "none", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_graylog": "false", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_journald": "false", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_stderr": "true", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "err_to_syslog": "false", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "event_tracing": "false", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_addr": "0.0.0.0", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_http_port": "9926", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_prio_limit": "5", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_sort_metrics": "true", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "exporter_stats_period": "5", 2024-04-26T16:21:36.981 INFO:teuthology.orchestra.run.smithi050.stdout: "fake_statfs_for_testing": "0", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "fatal_signal_handlers": "true", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filer_max_purge_ops": "10", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filer_max_truncate_ops": "128", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_apply_finisher_threads": "1", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_blackhole": "false", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_btrfs_clone_range": "true", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_btrfs_snap": "true", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_caller_concurrency": "10", 2024-04-26T16:21:36.982 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_collect_device_partition_information": "true", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_commit_timeout": "600.000000", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_inject_read_err": "false", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_omap_check": "false", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_debug_verify_split": "false", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_dump_file": "", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fadvise": "true", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fail_eio": "true", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fd_cache_shards": "16", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fd_cache_size": "128", 2024-04-26T16:21:36.983 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fiemap": "false", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fiemap_threshold": "4096", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_index_retry_probability": "0.000000", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_inject_stall": "0", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_parallel": "false", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_trailing": "false", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_journal_writeahead": "false", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_kill_at": "0", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size": "0", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-04-26T16:21:36.984 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs": "0", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_other": "2", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_sync_interval": "5.000000", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size": "0", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_merge_threshold": "-10", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_min_sync_interval": "0.010000", 2024-04-26T16:21:36.985 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_odsync_write": "false", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_backend": "rocksdb", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_backend_path": "", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_omap_header_cache_size": "1024", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_ondisk_finisher_threads": "1", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_thread_timeout": "60", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_op_threads": "2", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_punch_hole": "false", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-04-26T16:21:36.986 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_bytes": "104857600", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_queue_max_ops": "50", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_seek_data_hole": "false", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_sloppy_crc": "false", 2024-04-26T16:21:36.987 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_splice": "false", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_split_multiple": "2", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_split_rand_factor": "20", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_update_to": "1000", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-04-26T16:21:36.988 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_enable": "true", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_xfs_extsize": "false", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "filestore_zfs_snap": "false", 2024-04-26T16:21:36.989 INFO:teuthology.orchestra.run.smithi050.stdout: "fio_dir": "/tmp/fio", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_allow_other": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_atomic_o_trunc": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_big_writes": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_debug": "false", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_default_permissions": "false", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_disable_pagecache": "false", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_max_write": "0", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_multithreaded": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_require_active_mds": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_set_user_groups": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_move": "true", 2024-04-26T16:21:36.990 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_read": "true", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_splice_write": "true", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_syncfs_on_mksnap": "true", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "fuse_use_invalidate_cb": "true", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "gss_target_name": "ceph", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_file": "", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_inject_failure": "0", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_interval": "5", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "host": "smithi050", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_max_size": "1073741824", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-04-26T16:21:36.991 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "immutable_object_cache_watermark": "0.900000", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "inject_early_sigterm": "false", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "jaeger_agent_port": "6799", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "jaeger_tracing_enable": "false", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_aio": "true", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_align_min_size": "65536", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_block_align": "true", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_block_size": "4096", 2024-04-26T16:21:36.992 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_dio": "true", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_discard": "false", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_force_aio": "false", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_ignore_corruption": "false", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_max_write_bytes": "10485760", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_max_write_entries": "100", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_replay_from": "0", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_high_multiple": "0.000000", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_throttle_max_multiple": "0.000000", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_write_header_frequency": "0", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journal_zero_on_create": "false", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_prefetch_periods": "10", 2024-04-26T16:21:36.993 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_prezero_periods": "5", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "journaler_write_head_interval": "15", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "key": "", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "keyfile": "", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_backend": "rocksdb", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_default_stripe_size": "65536", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_fsck_on_mount": "false", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_fsck_on_mount_deep": "true", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_max_bytes": "67108864", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_max_ops": "512", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_nid_prealloc": "1024", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_onode_map_size": "1024", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_sync_submit_transaction": "false", 2024-04-26T16:21:36.994 INFO:teuthology.orchestra.run.smithi050.stdout: "kstore_sync_transaction": "false", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "librados_thread_count": "2", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "lockdep": "false", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "lockdep_force_backtrace": "false", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_coarse_timestamps": "true", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_flush_on_exit": "false", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_graylog_host": "127.0.0.1", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_graylog_port": "12201", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_max_new": "1000", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_max_recent": "10000", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_stderr_prefix": "debug ", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_stop_at_utilization": "0.970000", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_file": "true", 2024-04-26T16:21:36.995 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_graylog": "false", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_journald": "false", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_stderr": "false", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "log_to_syslog": "false", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "max_rotating_auth_attempts": "10", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_action_on_write_error": "1", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_alternate_name_max": "8192", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_asio_thread_count": "2", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_export_pin": "true", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_dirs": "true", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_interval": "5", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_fragment_size_max": "100000", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_idle_threshold": "0.000000", 2024-04-26T16:21:36.996 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_interval": "10", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_max": "-1", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_max_until": "-1", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_merge_size": "50", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_midchunk": "0.300000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_min_rebalance": "0.100000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_min_start": "0.200000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_minchunk": "0.001000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_mode": "0", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_need_max": "1.200000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_need_min": "0.800000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_overload_epochs": "2", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_sample_interval": "3.000000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_bits": "3", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_rd": "25000.000000", 2024-04-26T16:21:36.997 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_size": "10000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_split_wr": "10000.000000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_target_decay": "10.000000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_grace": "15.000000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_interval": "4.000000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_beacon_mon_down_grace": "60", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_memory_limit": "4294967296", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_mid": "0.700000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_delay": "0", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_sleep": "200", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_splitauth": "true", 2024-04-26T16:21:36.998 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_quiesce_threshold": "524288", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_release_free_interval": "10", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_reservation": "0.050000", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_interval": "1", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cache_trim_threshold": "262144", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_delegate_inos_pct": "50", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_prealloc_inos": "1000", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_connect_bootstrapping": "false", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_damage_table_max_entries": "10000", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_auth_pins": "false", 2024-04-26T16:21:36.999 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_frag": "false", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_scatterstat": "false", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_debug_subtrees": "false", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_decay_halflife": "5.000000", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_default_dir_hash": "2", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_defer_session_stale": "true", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_deny_all_reconnect": "false", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_keys_per_op": "16384", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_max_commit_size": "10", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_max_entries": "0", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dir_prefetch": "true", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dirstat_min_interval": "1.000000", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_after_rejoin": "false", 2024-04-26T16:21:37.000 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_on_map": "false", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_threshold_file": "0", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_early_reply": "true", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_enable_op_tracker": "true", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_enforce_unique_name": "true", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_distributed": "true", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_random": "true", 2024-04-26T16:21:37.001 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_forward_all_requests_to_auth": "false", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_health_cache_threshold": "1.500000", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_health_summarize_threshold": "10", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_heartbeat_grace": "15.000000", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_heartbeat_reset_grace": "1000", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_health_dummy": "false", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-04-26T16:21:37.002 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_migrator_session_race": "false", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_join_fs": "", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_journal_format": "1", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_create_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_export_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_import_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_expire_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_journal_replay_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_link_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_mdstable_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_openc_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_rename_at": "0", 2024-04-26T16:21:37.003 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_shutdown_at": "0", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_event_large_threshold": "524288", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_events_per_segment": "1024", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_major_segment_event_ratio": "12", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_max_events": "-1", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_max_segments": "128", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_pause": "false", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_segment_size": "0", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_skip_corrupt_events": "false", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_skip_unbounded_events": "false", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_trim_threshold": "128", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_log_warn_factor": "2.000000", 2024-04-26T16:21:37.004 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_caps_per_client": "1048576", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_completed_flushes": "100000", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_completed_requests": "100000", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_export_size": "20971520", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_file_recover": "32", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_files": "64", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_ops": "8192", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_max_snaps_per_dir": "100", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_metrics_update_interval": "2", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_min_caps_per_client": "100", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_min_caps_working_set": "10000", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_numa_node": "-1", 2024-04-26T16:21:37.005 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_complaint_time": "30.000000", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_duration": "600", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_size": "20", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_slow_op_size": "20", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_history_slow_op_threshold": "10", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_op_log_threshold": "5", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_ping_grace": "15", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_ping_interval": "5", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_caps": "30000", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_max_decay_threshold": "131072", 2024-04-26T16:21:37.006 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_recall_warning_threshold": "262144", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_reconnect_timeout": "45.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_replay_interval": "1.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_root_ino_gid": "0", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_root_ino_uid": "0", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_blocklist_on_evict": "true", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_blocklist_on_timeout": "true", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-04-26T16:21:37.007 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_session_metadata_threshold": "16777216", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_shutdown_check": "0", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_skip_ino": "0", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_sleep_rank_change": "0.000000", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_max_uid": "4294967294", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_min_uid": "0", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_snap_rstat": "false", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_standby_replay_damaged": "false", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_symlink_recovery": "true", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_task_status_update_interval": "2.000000", 2024-04-26T16:21:37.008 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_thrash_exports": "0", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_thrash_fragments": "0", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_tick_interval": "5.000000", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_valgrind_exit": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_verify_backtrace": "1", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_verify_scatter": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_wipe_ino_prealloc": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mds_wipe_sessions": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mempool_debug": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_debug_omit_block_device_write": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_device_bytes": "1073741824", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_page_set": "false", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "memstore_page_size": "65536", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_bytes": "134217728", 2024-04-26T16:21:37.009 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_messages": "512", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_connect_retry_interval": "1.000000", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_disabled_modules": "", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_enable_op_tracker": "true", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_max_pg_creating": "1024", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_max_pg_num_change": "128", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mds_bytes": "134217728", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mds_messages": "128", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mon_bytes": "134217728", 2024-04-26T16:21:37.010 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_mon_messages": "128", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_num_op_tracker_shard": "32", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_op_complaint_time": "30.000000", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_op_history_duration": "600", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_op_history_size": "20", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_op_history_slow_op_size": "20", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_op_log_threshold": "5", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_osd_bytes": "536870912", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_osd_messages": "8192", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_pool": "true", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_service_beacon_grace": "60.000000", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_standby_modules": "true", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_stats_period": "5", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_stats_threshold": "5", 2024-04-26T16:21:37.011 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_tick_period": "2", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_accept_timeout_factor": "2.000000", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_allow_pool_delete": "true", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_allow_pool_size_one": "false", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_auth_validate_all_caps": "true", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_bytes": "104857600", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_directed_command_retry": "2", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval": "3.000000", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_hunt_parallel": "3", 2024-04-26T16:21:37.012 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_log_interval": "1.000000", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_ping_interval": "10.000000", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_ping_timeout": "30.000000", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_client_target_rank": "-1", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clock_drift_allowed": "1.000000", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_level": "debug", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_file": "true", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog": "false", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_journald": "false", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_stderr": "true", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-04-26T16:21:37.013 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_bootstrap": "false", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_start": "false", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_compact_on_trim": "true", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_con_tracker_persist_interval": "10", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_con_tracker_score_halflife": "43200", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_config_key_max_entry_size": "65536", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_cpu_threads": "4", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_crush_min_required_version": "hammer", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_daemon_bytes": "419430400", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_avail_crit": "5", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_avail_warn": "5", 2024-04-26T16:21:37.014 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_data_size_warn": "16106127360", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_block_osdmap_trim": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_json": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_dump_transactions": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_extra_checks": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_reef": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_no_require_squid": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_delta_reset_interval": "10.000000", 2024-04-26T16:21:37.015 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_dns_srv_name": "ceph-mon", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_down_mkfs_grace": "60", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_down_uptime_grace": "60", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_election_default_strategy": "1", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_election_timeout": "5.000000", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ping_divisor": "2", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_elector_ping_timeout": "2.000000", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_enable_op_tracker": "true", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_fake_pool_delete": "false", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_force_quorum_join": "false", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_globalid_prealloc": "10000", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_detail_to_clog": "true", 2024-04-26T16:21:37.016 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_log_update_period": "5", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_max_detail": "50", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog": "true", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog_interval": "600", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_host": "[v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0]", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_host_override": "", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_initial_members": "", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_keyvaluedb": "rocksdb", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease": "5.000000", 2024-04-26T16:21:37.017 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_full_interval": "50", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_max": "10000", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_log_max_summary": "50", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_log_entries_per_event": "4096", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_log_epochs": "500", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_mdsmap_epochs": "500", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_mgrmap_epochs": "500", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_osd": "10000", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_pg_per_osd": "10000", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_pool_pg_num": "65536", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-04-26T16:21:37.018 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_force_trim_to": "0", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mds_skip_sanity": "false", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_memory_autotune": "true", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_memory_target": "2147483648", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_beacon_grace": "30", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_digest_period": "5", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_inactive_grace": "60", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_mkfs_grace": "240", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_min_osdmap_epochs": "50", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_complaint_time": "30", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_duration": "600", 2024-04-26T16:21:37.019 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_size": "20", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_slow_op_size": "20", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_history_slow_op_threshold": "10", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_op_log_threshold": "5", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_in": "false", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_auto_mark_new_in": "true", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_cache_size": "500", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_cache_size_min": "134217728", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_crush_smoke_test": "true", 2024-04-26T16:21:37.020 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_destroyed_out_interval": "600", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_down_out_interval": "600", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_force_trim_to": "0", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_full_ratio": "0.950000", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_halflife": "3600", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_max_interval": "300", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_laggy_weight": "0.300000", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_max_initial_pgs": "1024", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_down_reporters": "2", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-04-26T16:21:37.021 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp": "true", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_report_timeout": "900", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_warn_num_repaired": "10", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osd_warn_op_age": "32.000000", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_interval": "2", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_min": "15", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-04-26T16:21:37.022 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_stuck_threshold": "60", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_objects": "10000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_per_osd": "0", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pool_quota_crit_threshold": "0", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_pool_quota_warn_threshold": "0", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_probe_timeout": "2.000000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_max_change": "0.050000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_max_osds": "4", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-04-26T16:21:37.023 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_interval": "86400", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_max_keys": "100", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_scrub_timeout": "300", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_session_timeout": "300", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_smart_report_timeout": "5", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stat_smooth_intervals": "6", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_pool_min_size": "2", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_pool_size": "4", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_subscribe_interval": "86400.000000", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_debug": "false", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_max_payload_keys": "2000", 2024-04-26T16:21:37.024 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_max_payload_size": "1048576", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_provider_kill_at": "0", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_requester_kill_at": "0", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_sync_timeout": "60.000000", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_target_pg_per_osd": "100", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_tick_interval": "5", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_timecheck_interval": "300.000000", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_use_min_delay_socket": "false", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_older_version_delay": "604800", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-04-26T16:21:37.025 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_filestore_osds": "true", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_misplaced": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_older_version": "true", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_no_app": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_on_too_few_osds": "false", 2024-04-26T16:21:37.026 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "monmap": "", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "motr_tracing_enabled": "false", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_op_threads": "3", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_buffer_size": "131072", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_cm": "false", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_device_name": "", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_dscp": "96", 2024-04-26T16:21:37.027 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_gid_idx": "0", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_local_gid": "", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_polling_us": "1000", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_port_num": "1", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_roce_ver": "1", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_send_buffers": "1024", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_sl": "3", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_support_srq": "true", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_rdma_type": "ib", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_async_reap_threshold": "5", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_before_connect": "false", 2024-04-26T16:21:37.028 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_ipv4": "true", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_ipv6": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_msgr1": "true", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_msgr2": "true", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_port_max": "7568", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_port_min": "6800", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_prefer_ipv4": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_retry_count": "3", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_bind_retry_delay": "5", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_client": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mds": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mgr": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_mon": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_blackhole_osd": "false", 2024-04-26T16:21:37.029 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_client_mode": "crc secure", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_cluster_mode": "crc secure", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_cluster_type": "", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_compress_secure": "false", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_connection_idle_timeout": "900", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_connection_ready_timeout": "10", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_crc_data": "true", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_crc_header": "true", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_bad_msg": "false", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_bug": "true", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_old_message": "true", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_skipped_message": "false", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_die_on_unhandled_msg": "false", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_coremask": "0xF", 2024-04-26T16:21:37.030 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_devs_allowlist": "", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_enable_tso": "true", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hugepages": "", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hw_flow_control": "true", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_lro": "true", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_memory_channel": "4", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_pmd": "", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_port_id": "0", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dump_corrupt_message_level": "1", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_dump_on_send": "false", 2024-04-26T16:21:37.031 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_initial_backoff": "0.200000", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_max": "1.000000", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_probability": "0.000000", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_delay_type": "", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_internal_delays": "0.000000", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_network_congestion": "0", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_inject_socket_failures": "0", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_learn_addr_from_peer": "true", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_max_accept_failures": "4", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_max_backoff": "15.000000", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_client_mode": "secure crc", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_cluster_mode": "secure crc", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_mon_service_mode": "secure crc", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compress_min_size": "1024", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compress_mode": "none", 2024-04-26T16:21:37.032 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_osd_compression_algorithm": "snappy", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_pq_min_cost": "65536", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_public_type": "", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_service_mode": "crc secure", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_listen_backlog": "512", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_nodelay": "true", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_tcp_rcvbuf": "0", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "ms_type": "async+posix", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "no_config_file": "false", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_completion_locks_per_session": "32", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_debug_inject_relock_delay": "false", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inflight_op_bytes": "104857600", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inflight_ops": "1024", 2024-04-26T16:21:37.033 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_inject_no_watch_ping": "false", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_tick_interval": "5.000000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "objecter_timeout": "10.000000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "objectstore_blackhole": "false", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "openssl_engine_opts": "", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_delay_time": "5.000000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_hist_halflife": "1000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_max_low_ops": "2", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_max_ops": "4", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_quantize_effort": "0.100000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_agent_slop": "0.020000", 2024-04-26T16:21:37.034 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_allow_recovery_below_min_size": "true", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_asio_thread_count": "2", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_async_recovery_min_cost": "100", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_auto_mark_unfound_lost": "false", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_retry_interval": "30.000000", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_scan_max": "512", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backfill_scan_min": "64", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_degraded": "false", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_peering": "false", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_backoff_on_unfound": "true", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_beacon_report_interval": "300", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_duration": "30", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-04-26T16:21:37.035 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_max_block_size": "67108864", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_bench_small_size_max_iops": "100", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_blkin_trace_all": "false", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_blocked_scrub_grace_period": "120", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_check_for_log_corruption": "false", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_class_update_on_start": "true", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_message_cap": "256", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_message_size_cap": "524288000", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_op_priority": "63", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_client_watch_timeout": "30", 2024-04-26T16:21:37.036 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_command_max_records": "256", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_compact_on_start": "false", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_chooseleaf_type": "1", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_initial_weight": "-1.000000", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_update_on_start": "true", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_crush_update_weight_set": "true", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_drop_ping_duration": "0", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_feed_pullee": "-1", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-04-26T16:21:37.037 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_misdirected_ops": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_no_acting_change": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_no_purge_strays": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_op_order": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_pg_log_writeout": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_pretend_recovery_active": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_shutdown": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_trim_objects": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_cached_snaps": "false", 2024-04-26T16:21:37.038 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_missing_on_start": "false", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_snaps": "false", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_keys": "1024", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_stride": "524288", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_default_data_pool_replay_window": "45", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_default_notify_timeout": "30", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep": "0.000000", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-04-26T16:21:37.039 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_discard_disconnected_ops": "true", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_enable_op_tracker": "true", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_extblkdev_plugins": "vdo", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_info": "true", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown": "true", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_fast_shutdown_timeout": "15", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-04-26T16:21:37.040 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_function_tracing": "false", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_grace": "20", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_interval": "6", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_peers": "10", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_min_size": "2000", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_stale": "600", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_max_size": "100000", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_min_size": "1000", 2024-04-26T16:21:37.041 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal_flush_on_shutdown": "true", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_journal_size": "5120", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_kill_backfill_at": "0", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_loop_before_reset_tphandle": "64", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_cache_size": "50", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_dedup": "true", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_message_max": "40", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_message_max_bytes": "10485760", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_map_share_max_epochs": "40", 2024-04-26T16:21:37.042 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_attr_name_len": "100", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_attr_size": "0", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_backfills": "1", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_markdown_count": "5", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_markdown_period": "600", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_name_len": "2048", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_namespace_len": "256", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_object_size": "134217728", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_omap_entries_per_request": "1024", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_blocked_by": "16", 2024-04-26T16:21:37.043 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_log_entries": "10000", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_pgls": "1024", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_push_cost": "8388608", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_push_objects": "10", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_scrubs": "3", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_trimming_pgs": "2", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_write_op_reply_len": "64", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_max_write_size": "90", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-04-26T16:21:37.044 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_override_recovery_settings": "false", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_profile": "balanced", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-04-26T16:21:37.045 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mclock_skip_benchmark": "false", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_base": "805306368", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_cache_min": "134217728", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target": "4294967296", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target_autotune": "false", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_min_pg_log_entries": "250", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_min_recovery_priority": "0", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_heartbeat_interval": "30", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_report_interval": "5", 2024-04-26T16:21:37.046 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_report_max_in_flight": "2", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_num_cache_shards": "32", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_num_op_tracker_shard": "32", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_auto_affinity": "true", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_node": "-1", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_numa_prefer_iface": "true", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objecter_finishers": "1", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore": "bluestore", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore_fuse": "false", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_objectstore_tracing": "false", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_complaint_time": "30.000000", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_duration": "600", 2024-04-26T16:21:37.047 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_size": "20", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_slow_op_size": "20", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_log_threshold": "5", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards": "0", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards_hdd": "5", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_shards_ssd": "8", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard": "0", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_pq_min_cost": "65536", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_queue": "mclock_scheduler", 2024-04-26T16:21:37.048 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_queue_cut_off": "high", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_thread_suicide_timeout": "150", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_op_thread_timeout": "15", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_open_classes_on_start": "true", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_os_flags": "0", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_peering_op_priority": "255", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_delete_cost": "1048576", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_delete_priority": "5", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_dups_tracked": "3000", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_trim_max": "10000", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_log_trim_min": "100", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-04-26T16:21:37.049 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_object_context_cache_count": "64", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_crimson": "false", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_crush_rule": "-1", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_ec_fast_read": "false", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_bulk": "false", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-04-26T16:21:37.050 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nodelete": "false", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_flags": "0", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_min_size": "0", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pg_num": "32", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_pgp_num": "0", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_read_ratio": "70", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_size": "3", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_default_type": "replicated", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-04-26T16:21:37.051 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_pool_use_gmt_hitset": "true", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_push_per_object_cost": "1000", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_read_ec_check_for_errors": "false", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recover_clone_overlap": "true", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recover_clone_overlap_limit": "10", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_cost": "20971520", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_delay_start": "0.000000", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active": "0", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active_hdd": "3", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_active_ssd": "10", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_chunk": "8388608", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_max_single_start": "1", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_op_priority": "3", 2024-04-26T16:21:37.052 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_op_warn_multiple": "16", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_priority": "5", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_retry_interval": "30.000000", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep": "0.000000", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_repair_during_recovery": "false", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_requested_scrub_priority": "120", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_rollback_to_cluster_snap": "", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_auto_repair": "false", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-04-26T16:21:37.053 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_begin_hour": "0", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_begin_week_day": "0", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_chunk_max": "25", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_chunk_min": "5", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_cost": "52428800", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_during_recovery": "false", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_end_hour": "0", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_end_week_day": "0", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_event_cost": "4096", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_invalid_stats": "true", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_load_threshold": "0.500000", 2024-04-26T16:21:37.054 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_max_interval": "604800.000000", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_max_preemptions": "5", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_min_interval": "86400.000000", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_priority": "5", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_reservation_timeout": "300000", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_scrub_sleep": "0.000000", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_shutdown_pgref_assert": "false", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_skip_data_digest": "false", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_smart_report_timeout": "5", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_cost": "1048576", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_priority": "5", 2024-04-26T16:21:37.055 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep": "0.000000", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_stats_update_period_scrubbing": "15", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_target_transaction_size": "30", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-04-26T16:21:37.056 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_default_cache_mode": "writeback", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_tracing": "false", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_use_stale_snap": "false", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "osdc_blkin_trace_all": "false", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_kill_at": "0", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_max_join_drift": "10", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_min": "500", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_min_wait": "0.050000", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_propose_interval": "1.000000", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_max": "500", 2024-04-26T16:21:37.057 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_max_multiplier": "20", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_service_trim_min": "10", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_stash_full_interval": "25", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_trim_max": "500", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "paxos_trim_min": "250", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "perf": "true", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "pid_file": "", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "-", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrv": "", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "public_bind_addr": "-", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "public_network": "", 2024-04-26T16:21:37.058 INFO:teuthology.orchestra.run.smithi050.stdout: "public_network_interface": "", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "qat_compressor_enabled": "false", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "qat_compressor_session_max_number": "256", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_mon_op_timeout": "0", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_osd_op_timeout": "0", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rados_tracing": "false", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_atime_update_interval": "60", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_balance_parent_reads": "false", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_balance_snap_reads": "false", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blkin_trace_all": "false", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blocklist_expire_seconds": "0", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_blocklist_on_break_lock": "true", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache": "true", 2024-04-26T16:21:37.059 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_block_writes_upfront": "false", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty": "25165824", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_max_dirty_object": "0", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_policy": "writearound", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_size": "33554432", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_target_dirty": "16777216", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_clone_copy_on_read": "false", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_compression_hint": "none", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_concurrent_management_ops": "10", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_clone_format": "auto", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_data_pool": "", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_features": "61", 2024-04-26T16:21:37.060 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_format": "2", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_map_options": "", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_order": "22", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_pool": "rbd", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_stripe_count": "0", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_default_stripe_unit": "0", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_disable_zero_copy_writes": "true", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_discard_granularity_bytes": "65536", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_enable_alloc_hint": "true", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-04-26T16:21:37.061 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_io_scheduler": "simple", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_commit_age": "5.000000", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_flush_interval": "0", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_order": "24", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_pool": "", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_journal_splay_width": "4", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_localize_parent_reads": "false", 2024-04-26T16:21:37.062 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_localize_snap_reads": "false", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_type": "simple", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-04-26T16:21:37.063 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_autotune": "true", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_base": "805306368", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_memory_target": "4294967296", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_delete_delay": "0", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_replay_delay": "0", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-04-26T16:21:37.064 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_to_trash_on_remove": "false", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_mtime_update_interval": "60", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_non_blocking_aio": "true", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_op_thread_timeout": "60", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_op_threads": "1", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_parent_cache_enabled": "false", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_mode": "disabled", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_path": "/tmp", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_persistent_cache_size": "1073741824", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_plugins": "", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_burst": "0", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_bps_limit": "0", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_exclude_ops": "0", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_burst": "0", 2024-04-26T16:21:37.065 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_iops_limit": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_burst": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_bps_limit": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_burst": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_read_iops_limit": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_schedule_tick_min": "50", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_burst": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_bps_limit": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_burst": "0", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-04-26T16:21:37.066 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_qos_write_iops_limit": "0", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_quiesce_notification_attempts": "10", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_read_from_replica_policy": "default", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_max_bytes": "524288", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_readahead_trigger_requests": "10", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_request_timed_out_seconds": "30", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_skip_partial_discard": "true", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_tracing": "false", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_validate_names": "true", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd_validate_pool": "true", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "restapi_base_url": "", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "restapi_log_level": "", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_account_default_quota_max_size": "-1", 2024-04-26T16:21:37.067 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_acl_grants_max_num": "100", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_admin_entry": "admin", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_backend_store": "rados", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_barbican_url": "", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_beast_enable_async": "true", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_counters_cache": "false", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_index_max_aio": "128", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_quota_ttl": "600", 2024-04-26T16:21:37.068 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_enabled": "true", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_expiry_interval": "900", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cache_lru_size": "25000", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_config_store": "rados", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_content_length_compat": "false", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_copy_obj_progress": "true", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cors_rules_max_num": "100", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_cross_domain_policy": "", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_default_encryption_key": "", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_addr": "", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_ca_path": "", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_client_cert": "", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_client_key": "", 2024-04-26T16:21:37.069 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_password": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_kmip_username": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_require_ssl": "true", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-04-26T16:21:37.070 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_suppress_logs": "true", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_addr": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_auth": "token", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_namespace": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_prefix": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_token_file": "", 2024-04-26T16:21:37.071 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_buffersize": "524288", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_low_speed_limit": "1024", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_low_speed_time": "300", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_tcp_keepalive": "0", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_fadvise": "4", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-04-26T16:21:37.072 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_host": "127.0.0.1", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_l1_fadvise": "4", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_d4n_port": "6379", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_changes_size": "1000", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_num_shards": "128", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_log_window": "30", 2024-04-26T16:21:37.073 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_notify_interval_msec": "0", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_sync_poll_interval": "20", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_data_sync_spawn_window": "20", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_data_log_backing": "fifo", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_region_info_oid": "default.region", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_defer_to_bucket_acls": "", 2024-04-26T16:21:37.074 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_res": "500.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dns_name": "", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dns_s3website_name": "", 2024-04-26T16:21:37.075 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_dynamic_resharding": "true", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_gc_threads": "true", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_lc_threads": "true", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_ops_log": "false", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_quota_threads": "true", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_static_website": "false", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enable_usage_log": "false", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_enforce_swift_acls": "true", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_exit_timeout_secs": "120", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_expose_bucket": "false", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_extended_http_attrs": "", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_filter": "none", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-04-26T16:21:37.076 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_frontends": "beast port=7480", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_concurrent_io": "10", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_deferred": "50", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_objs": "32", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_queue_size": "134213632", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_max_trim_chunk": "16", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_obj_min_wait": "7200", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_processor_max_time": "3600", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_gc_processor_period": "3600", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_get_obj_window_size": "16777216", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_healthcheck_disabling_path": "", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ignore_get_invalid_range": "false", 2024-04-26T16:21:37.077 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_init_timeout": "300", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_kafka_connection_idle": "300", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_kafka_message_timeout": "5000", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_kafka_sleep_timeout": "10", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_domain": "", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_password": "", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_password_path": "", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_project": "", 2024-04-26T16:21:37.078 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_tenant": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_token": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_token_path": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_admin_user": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_api_version": "2", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_domain": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_password": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_project": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_tenant": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_barbican_user": "", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_implicit_tenants": "false", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_service_token_enabled": "false", 2024-04-26T16:21:37.079 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_token_cache_size": "10000", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_url": "", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_keystone_verify_ssl": "true", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_debug_interval": "-1", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_lock_max_time": "90", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_objs": "32", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_rules": "1000", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_worker": "3", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_max_wp_worker": "3", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lc_thread_delay": "0", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_dnattr": "uid", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_searchfilter": "", 2024-04-26T16:21:37.080 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ldap_uri": "", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lfuda_sync_frequency": "60", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_local_cache_address": "127.0.0.1:6379", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_http_headers": "", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_nonexistent_bucket": "false", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_log_object_name_utc": "false", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attr_name_len": "0", 2024-04-26T16:21:37.081 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attr_size": "0", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_attrs_num_in_req": "0", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_chunk_size": "4194304", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_concurrent_requests": "1024", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_dynamic_shards": "1999", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_listing_results": "1000", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_notify_retries": "10", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_objs_per_shard": "100000", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_put_param_size": "1048576", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_put_size": "5368709120", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_max_slo_entries": "1000", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_md_log_max_shards": "64", 2024-04-26T16:21:37.082 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_md_notify_interval_msec": "200", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_meta_sync_poll_interval": "20", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_meta_sync_spawn_window": "20", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_mp_lock_max_time": "600", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multipart_min_part_size": "5242880", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_fhcache_size": "2017", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_lru_lanes": "5", 2024-04-26T16:21:37.083 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_max_gc": "300", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_gc_threads": "false", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_lc_threads": "false", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_quota_threads": "false", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_run_sync_thread": "false", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_num_async_rados_threads": "32", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_num_control_oids": "8", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_numa_node": "-1", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_obj_stripe_size": "4194304", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_chunk_size": "100", 2024-04-26T16:21:37.084 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_gc_interval": "600", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_objexp_hints_num_shards": "127", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_thread_timeout": "600", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_op_tracing": "false", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_token": "", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_url": "", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_opa_verify_ssl": "true", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_rados": "false", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_ops_log_socket_path": "", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-04-26T16:21:37.085 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_push_interval": "2.000000", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_push_interval_max": "30.000000", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_period_root_pool": ".rgw.root", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_lanes": "3", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_max_buckets": "100", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_cache_partitions": "3", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_print_continue": "true", 2024-04-26T16:21:37.086 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_print_prohibited_content_length": "false", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rados_tracing": "false", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm": "", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm_id": "", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_region": "", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_region_root_pool": ".rgw.root", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_relaxed_region_enforcement": "false", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-04-26T16:21:37.087 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_request_uri": "", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_batch_size": "64", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_max_aio": "128", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_num_logs": "16", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_reshard_thread_interval": "600", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_resolve_cname": "false", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_rest_getusage_op_compat": "false", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_run_sync_thread": "true", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_keystone": "false", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_ldap": "false", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_rados": "true", 2024-04-26T16:21:37.088 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_auth_use_sts": "false", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_s3_success_create_obj_status": "0", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_scheduler_type": "throttler", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_script_uri": "", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_service_provider_name": "", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_client_id": "", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_client_secret": "", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_entry": "sts", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_key": "sts", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_max_session_duration": "43200", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_min_session_duration": "900", 2024-04-26T16:21:37.089 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sts_token_introspection_url": "", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_account_in_url": "false", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_auth_entry": "auth", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_auth_url": "", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_custom_header": "", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_enforce_content_length": "false", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_need_stats": "true", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_tenant_name": "", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_token_expiration": "86400", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_url": "", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_url_prefix": "swift", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_swift_versioning_enabled": "false", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-04-26T16:21:37.090 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_lease_period": "120", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_interval": "1200", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_obj_etag_verify": "false", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_history_size": "4096", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_thread_pool_size": "512", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_persistency_max_retries": "0", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-04-26T16:21:37.091 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_topic_require_publish_policy": "false", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_comment": "", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_createby": "", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_encoding": "", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_flag": "false", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_max_size": "5368709120", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_origin": "", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_sha_unit": "524288", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_torrent_tracker": "", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_trust_forwarded_https": "false", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_log_tick_interval": "30", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_max_shards": "32", 2024-04-26T16:21:37.092 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_usage_max_user_shards": "1", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_use_opa_authz": "false", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_counters_cache": "false", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_counters_cache_size": "10000", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_default_quota_max_size": "-1", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_max_buckets": "1000", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_policies_max_num": "100", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_interval": "86400", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_user_unique_email": "true", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_verify_ssl": "true", 2024-04-26T16:21:37.093 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_website_routing_rules_max_num": "50", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone": "", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone_id": "", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup": "", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup_id": "", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgwlc_auto_session_clear": "true", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rgwlc_skip_bucket_step": "false", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_block_size": "4096", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-04-26T16:21:37.094 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_shard_bits": "4", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_size": "536870912", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cache_type": "binned_lru", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_compaction_stats": "false", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_extended_stats": "false", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_collect_memory_stats": "false", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_index_type": "binary_search", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_log_to_ceph_log": "true", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_metadata_block_size": "4096", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_partition_filters": "false", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_perf": "false", 2024-04-26T16:21:37.095 INFO:teuthology.orchestra.run.smithi050.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "rotating_keys_renewal_timeout": "10", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "run_dir": "/var/run/ceph", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_block_create": "true", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_cache_lru_size": "67108864", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_cbjournal_size": "5368709120", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_data_delta_based_overwrite": "0", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_default_max_object_size": "16777216", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_device_size": "53687091200", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_capacity": "16", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_journal_iodepth_limit": "5", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_main_device_type": "SSD", 2024-04-26T16:21:37.096 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_max_concurrent_transactions": "8", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "seastore_segment_size": "67108864", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "set_keepcaps": "false", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "setgroup": "ceph", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "setuser": "ceph", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "setuser_match_path": "", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_misplaced_ratio": "0.050000", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "thp": "false", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "threadpool_default_timeout": "60", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "threadpool_empty_queue_max_wait": "2", 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout: "throttler_perf_counter": "true" 2024-04-26T16:21:37.097 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.019349+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.019349+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.021677+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.021677+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.213492+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.213492+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:21:37.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.215529+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.215529+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: audit 2024-04-26T16:21:36.496157+0000 mgr.y (mgr.24401) 154 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: audit 2024-04-26T16:21:36.496157+0000 mgr.y (mgr.24401) 154 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.917352+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.2 deep-scrub starts 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.917352+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.2 deep-scrub starts 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.925476+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.2 deep-scrub ok 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[21463]: cluster 2024-04-26T16:21:36.925476+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.2 deep-scrub ok 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.019349+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.019349+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:21:37.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.021677+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.021677+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.213492+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.213492+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.215529+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.215529+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: audit 2024-04-26T16:21:36.496157+0000 mgr.y (mgr.24401) 154 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: audit 2024-04-26T16:21:36.496157+0000 mgr.y (mgr.24401) 154 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.917352+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.2 deep-scrub starts 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.917352+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.2 deep-scrub starts 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.925476+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.2 deep-scrub ok 2024-04-26T16:21:37.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:37 smithi050 bash[29234]: cluster 2024-04-26T16:21:36.925476+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.2 deep-scrub ok 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.019349+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.019349+0000 osd.1 (osd.1) 101 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.021677+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.021677+0000 osd.1 (osd.1) 102 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.213492+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.213492+0000 osd.7 (osd.7) 121 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.215529+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.215529+0000 osd.7 (osd.7) 122 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: audit 2024-04-26T16:21:36.496157+0000 mgr.y (mgr.24401) 154 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: audit 2024-04-26T16:21:36.496157+0000 mgr.y (mgr.24401) 154 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.917352+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.2 deep-scrub starts 2024-04-26T16:21:37.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.917352+0000 osd.3 (osd.3) 109 : cluster [DBG] 3.2 deep-scrub starts 2024-04-26T16:21:37.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.925476+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.2 deep-scrub ok 2024-04-26T16:21:37.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:37 smithi190 bash[23868]: cluster 2024-04-26T16:21:36.925476+0000 osd.3 (osd.3) 110 : cluster [DBG] 3.2 deep-scrub ok 2024-04-26T16:21:37.816 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-04-26T16:21:38.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:38 smithi190 bash[23868]: cluster 2024-04-26T16:21:37.426207+0000 mgr.y (mgr.24401) 155 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:38.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:38 smithi190 bash[23868]: cluster 2024-04-26T16:21:37.426207+0000 mgr.y (mgr.24401) 155 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:38.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:38 smithi190 bash[23868]: cluster 2024-04-26T16:21:37.941985+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:21:38.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:38 smithi190 bash[23868]: cluster 2024-04-26T16:21:37.941985+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:21:38.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:38 smithi190 bash[23868]: cluster 2024-04-26T16:21:37.944327+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:21:38.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:38 smithi190 bash[23868]: cluster 2024-04-26T16:21:37.944327+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[21463]: cluster 2024-04-26T16:21:37.426207+0000 mgr.y (mgr.24401) 155 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[21463]: cluster 2024-04-26T16:21:37.426207+0000 mgr.y (mgr.24401) 155 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[21463]: cluster 2024-04-26T16:21:37.941985+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[21463]: cluster 2024-04-26T16:21:37.941985+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[21463]: cluster 2024-04-26T16:21:37.944327+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[21463]: cluster 2024-04-26T16:21:37.944327+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[29234]: cluster 2024-04-26T16:21:37.426207+0000 mgr.y (mgr.24401) 155 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[29234]: cluster 2024-04-26T16:21:37.426207+0000 mgr.y (mgr.24401) 155 : cluster [DBG] pgmap v123: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:38.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[29234]: cluster 2024-04-26T16:21:37.941985+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:21:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[29234]: cluster 2024-04-26T16:21:37.941985+0000 osd.3 (osd.3) 111 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:21:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[29234]: cluster 2024-04-26T16:21:37.944327+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:21:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:38 smithi050 bash[29234]: cluster 2024-04-26T16:21:37.944327+0000 osd.3 (osd.3) 112 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:38.970702+0000 osd.1 (osd.1) 103 : cluster [DBG] 5.c scrub starts 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:38.970702+0000 osd.1 (osd.1) 103 : cluster [DBG] 5.c scrub starts 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:38.973181+0000 osd.1 (osd.1) 104 : cluster [DBG] 5.c scrub ok 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:38.973181+0000 osd.1 (osd.1) 104 : cluster [DBG] 5.c scrub ok 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:39.060247+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:39.060247+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:39.061980+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:39.061980+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:39.426920+0000 mgr.y (mgr.24401) 156 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:39.426920+0000 mgr.y (mgr.24401) 156 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.029858+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:21:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.029858+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:21:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.032556+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:21:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:40 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.032556+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:38.970702+0000 osd.1 (osd.1) 103 : cluster [DBG] 5.c scrub starts 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:38.970702+0000 osd.1 (osd.1) 103 : cluster [DBG] 5.c scrub starts 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:38.973181+0000 osd.1 (osd.1) 104 : cluster [DBG] 5.c scrub ok 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:38.973181+0000 osd.1 (osd.1) 104 : cluster [DBG] 5.c scrub ok 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:39.060247+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:39.060247+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:39.061980+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:39.061980+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:39.426920+0000 mgr.y (mgr.24401) 156 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:40.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:39.426920+0000 mgr.y (mgr.24401) 156 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.029858+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.029858+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.032556+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.032556+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:38.970702+0000 osd.1 (osd.1) 103 : cluster [DBG] 5.c scrub starts 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:38.970702+0000 osd.1 (osd.1) 103 : cluster [DBG] 5.c scrub starts 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:38.973181+0000 osd.1 (osd.1) 104 : cluster [DBG] 5.c scrub ok 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:38.973181+0000 osd.1 (osd.1) 104 : cluster [DBG] 5.c scrub ok 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:39.060247+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:39.060247+0000 osd.0 (osd.0) 99 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:39.061980+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:39.061980+0000 osd.0 (osd.0) 100 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:21:40.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:39.426920+0000 mgr.y (mgr.24401) 156 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:40.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:39.426920+0000 mgr.y (mgr.24401) 156 : cluster [DBG] pgmap v124: 132 pgs: 132 active+clean; 583 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:40.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.029858+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:21:40.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.029858+0000 osd.3 (osd.3) 113 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:21:40.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.032556+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:21:40.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:40 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.032556+0000 osd.3 (osd.3) 114 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:21:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:41 smithi190 bash[23868]: audit 2024-04-26T16:21:40.808223+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:41 smithi190 bash[23868]: audit 2024-04-26T16:21:40.808223+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:41 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.990476+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.c scrub starts 2024-04-26T16:21:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:41 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.990476+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.c scrub starts 2024-04-26T16:21:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:41 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.993060+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.c scrub ok 2024-04-26T16:21:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:41 smithi190 bash[23868]: cluster 2024-04-26T16:21:40.993060+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.c scrub ok 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[29234]: audit 2024-04-26T16:21:40.808223+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[29234]: audit 2024-04-26T16:21:40.808223+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.990476+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.c scrub starts 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.990476+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.c scrub starts 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.993060+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.c scrub ok 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[29234]: cluster 2024-04-26T16:21:40.993060+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.c scrub ok 2024-04-26T16:21:41.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[21463]: audit 2024-04-26T16:21:40.808223+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[21463]: audit 2024-04-26T16:21:40.808223+0000 mon.c (mon.2) 81 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.990476+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.c scrub starts 2024-04-26T16:21:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.990476+0000 osd.3 (osd.3) 115 : cluster [DBG] 6.c scrub starts 2024-04-26T16:21:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.993060+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.c scrub ok 2024-04-26T16:21:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:41 smithi050 bash[21463]: cluster 2024-04-26T16:21:40.993060+0000 osd.3 (osd.3) 116 : cluster [DBG] 6.c scrub ok 2024-04-26T16:21:42.618 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.180405+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.180405+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.182830+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.182830+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.428321+0000 mgr.y (mgr.24401) 157 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.428321+0000 mgr.y (mgr.24401) 157 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.605700+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.605700+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.605811+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:42 smithi190 bash[23868]: cluster 2024-04-26T16:21:41.605811+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:42.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.180405+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.180405+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.182830+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.182830+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.428321+0000 mgr.y (mgr.24401) 157 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.428321+0000 mgr.y (mgr.24401) 157 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.605700+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.605700+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.605811+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[21463]: cluster 2024-04-26T16:21:41.605811+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:42.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.180405+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.180405+0000 osd.7 (osd.7) 123 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.182830+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.182830+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.428321+0000 mgr.y (mgr.24401) 157 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.428321+0000 mgr.y (mgr.24401) 157 : cluster [DBG] pgmap v125: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.605700+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.605700+0000 osd.5 (osd.5) 105 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.605811+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:42.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:42 smithi050 bash[29234]: cluster 2024-04-26T16:21:41.605811+0000 osd.5 (osd.5) 106 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:43.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[21463]: cluster 2024-04-26T16:21:42.504644+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-04-26T16:21:43.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[21463]: cluster 2024-04-26T16:21:42.504644+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-04-26T16:21:43.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[21463]: cluster 2024-04-26T16:21:42.881630+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:43.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[21463]: cluster 2024-04-26T16:21:42.881630+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[21463]: cluster 2024-04-26T16:21:42.881741+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[21463]: cluster 2024-04-26T16:21:42.881741+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[29234]: cluster 2024-04-26T16:21:42.504644+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[29234]: cluster 2024-04-26T16:21:42.504644+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[29234]: cluster 2024-04-26T16:21:42.881630+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[29234]: cluster 2024-04-26T16:21:42.881630+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[29234]: cluster 2024-04-26T16:21:42.881741+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:43.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:43 smithi050 bash[29234]: cluster 2024-04-26T16:21:42.881741+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:43.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:43 smithi190 bash[23868]: cluster 2024-04-26T16:21:42.504644+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-04-26T16:21:43.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:43 smithi190 bash[23868]: cluster 2024-04-26T16:21:42.504644+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2024-04-26T16:21:43.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:43 smithi190 bash[23868]: cluster 2024-04-26T16:21:42.881630+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:43.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:43 smithi190 bash[23868]: cluster 2024-04-26T16:21:42.881630+0000 osd.6 (osd.6) 65 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:43.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:43 smithi190 bash[23868]: cluster 2024-04-26T16:21:42.881741+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:43.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:43 smithi190 bash[23868]: cluster 2024-04-26T16:21:42.881741+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:44.676 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:21:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:21:44.676 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.042086+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.a scrub starts 2024-04-26T16:21:44.676 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.042086+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.a scrub starts 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.044525+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.a scrub ok 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.044525+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.a scrub ok 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.429008+0000 mgr.y (mgr.24401) 158 : cluster [DBG] pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.429008+0000 mgr.y (mgr.24401) 158 : cluster [DBG] pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.514531+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.514531+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.599814+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.599814+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.602241+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[29234]: cluster 2024-04-26T16:21:43.602241+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.042086+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.a scrub starts 2024-04-26T16:21:44.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.042086+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.a scrub starts 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.044525+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.a scrub ok 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.044525+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.a scrub ok 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.429008+0000 mgr.y (mgr.24401) 158 : cluster [DBG] pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.429008+0000 mgr.y (mgr.24401) 158 : cluster [DBG] pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.514531+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.514531+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.599814+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.599814+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.602241+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:21:44.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:44 smithi050 bash[21463]: cluster 2024-04-26T16:21:43.602241+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:21:44.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.042086+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.a scrub starts 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.042086+0000 osd.1 (osd.1) 105 : cluster [DBG] 2.a scrub starts 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.044525+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.a scrub ok 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.044525+0000 osd.1 (osd.1) 106 : cluster [DBG] 2.a scrub ok 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.429008+0000 mgr.y (mgr.24401) 158 : cluster [DBG] pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.429008+0000 mgr.y (mgr.24401) 158 : cluster [DBG] pgmap v127: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.514531+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.514531+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.599814+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.599814+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.602241+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:21:44.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:44 smithi190 bash[23868]: cluster 2024-04-26T16:21:43.602241+0000 osd.4 (osd.4) 90 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:44.413273+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:44.413273+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:44.415132+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:44.415132+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.089622+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.089622+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:21:45.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.092124+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.092124+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:44.413273+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:44.413273+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:44.415132+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:44.415132+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.089622+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.089622+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.092124+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:21:45.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:45 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.092124+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:21:45.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:44.413273+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:21:45.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:44.413273+0000 osd.2 (osd.2) 79 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:21:45.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:44.415132+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:21:45.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:44.415132+0000 osd.2 (osd.2) 80 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:21:45.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.089622+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:21:45.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.089622+0000 osd.6 (osd.6) 67 : cluster [DBG] 5.2 deep-scrub starts 2024-04-26T16:21:45.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.092124+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:21:45.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:45 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.092124+0000 osd.6 (osd.6) 68 : cluster [DBG] 5.2 deep-scrub ok 2024-04-26T16:21:45.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/1355286040 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa234101a10 msgr2=0x7fa234101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- 172.21.15.50:0/1355286040 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa234101a10 0x7fa234101e70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa2280098f0 tx=0x7fa228009c00 comp rx=0 tx=0).stop 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/1355286040 shutdown_connections 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- 172.21.15.50:0/1355286040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341094b0 0x7fa234109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- 172.21.15.50:0/1355286040 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 0x7fa2341102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- 172.21.15.50:0/1355286040 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa234101a10 0x7fa234101e70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/1355286040 >> 172.21.15.50:0/1355286040 conn(0x7fa234078990 msgr2=0x7fa2340ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/1355286040 shutdown_connections 2024-04-26T16:21:45.922 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/1355286040 wait complete. 2024-04-26T16:21:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 librados: starting msgr at 2024-04-26T16:21:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 librados: starting objecter 2024-04-26T16:21:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 Processor -- start 2024-04-26T16:21:45.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- start start 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 librados: setting wanted keys 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 librados: calling monclient init 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa234101a10 0x7fa2341992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 0x7fa234199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341094b0 0x7fa23419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa234112b40 con 0x7fa2341023b0 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa2341129c0 con 0x7fa234101a10 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa234112cc0 con 0x7fa2341094b0 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa231d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 0x7fa234199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341094b0 0x7fa23419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa231d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 0x7fa234199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57202/0 (socket says 172.21.15.50:57202) 2024-04-26T16:21:45.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 learned_addr learned my addr 172.21.15.50:0/4242004661 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa23259c700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa234101a10 0x7fa2341992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa234101a10 msgr2=0x7fa2341992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa234101a10 0x7fa2341992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 msgr2=0x7fa234199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 0x7fa234199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23419e280 con 0x7fa2341094b0 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa2341023b0 0x7fa234199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa23259c700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa234101a10 0x7fa2341992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa232d9d700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341094b0 0x7fa23419db40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009cd0 tx=0x7fa22c00c620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa2237fe700 10 client.?.objecter ms_handle_connect 0x7fa2341094b0 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa2237fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c00cdc0 con 0x7fa2341094b0 2024-04-26T16:21:45.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa22c003a00 con 0x7fa2341094b0 2024-04-26T16:21:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa23419e510 con 0x7fa2341094b0 2024-04-26T16:21:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c004450 con 0x7fa2341094b0 2024-04-26T16:21:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 10 client.24611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:21:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2341a5df0 con 0x7fa2341094b0 2024-04-26T16:21:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 librados: init done 2024-04-26T16:21:45.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.928+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fa23410b890 con 0x7fa2341094b0 2024-04-26T16:21:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa22c01a020 con 0x7fa2341094b0 2024-04-26T16:21:45.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa21c071f80 0x7fa21c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(126..126 src has 67..126) ==== 7450+0+0 (secure 0 0 0) 0x7fa22c090b50 con 0x7fa2341094b0 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 10 client.24611.objecter ms_dispatch 0x7fa234001550 osd_map(126..126 src has 67..126) 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 3 client.24611.objecter handle_osd_map got epochs [126,126] > 0 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 3 client.24611.objecter handle_osd_map decoding full epoch 126 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa23259c700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa21c071f80 0x7fa21c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 20 client.24611.objecter dump_active .. 0 homeless 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7fa22c010040 con 0x7fa2341094b0 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 0x7fa234064030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa23419a5d0 con 0x7fa23410b270 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa21c071f80 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa23259c700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa21c071f80 0x7fa21c074440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa224005e10 tx=0x7fa224005d60 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 0x7fa234064030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7fa23406a110 con 0x7fa23410b270 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa23410b270 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 10 client.24611.objecter resend_mon_ops 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 0x7fa234064030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa228009fd0 tx=0x7fa228005690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228004ac0 con 0x7fa23410b270 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa228028070 con 0x7fa23410b270 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 msgr2=0x7fa234064030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 0x7fa234064030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa228009fd0 tx=0x7fa228005690 comp rx=0 tx=0).stop 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 msgr2=0x7fa234064030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:21:45.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.932+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa23410b270 0x7fa234064030 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa228009fd0 tx=0x7fa228005690 comp rx=0 tx=0).stop 2024-04-26T16:21:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 0x7fa2341a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa23410b650 con 0x7fa234064570 2024-04-26T16:21:45.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 0x7fa2341a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7fa2341aa520 con 0x7fa234064570 2024-04-26T16:21:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa234064570 2024-04-26T16:21:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa2237fe700 10 client.24611.objecter resend_mon_ops 2024-04-26T16:21:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 0x7fa2341a9fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa228024000 tx=0x7fa228009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.940+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22802d9c0 con 0x7fa234064570 2024-04-26T16:21:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7fa22802d9c0 con 0x7fa234064570 2024-04-26T16:21:45.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 msgr2=0x7fa2341a9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 0x7fa2341a9fe0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa228024000 tx=0x7fa228009260 comp rx=0 tx=0).stop 2024-04-26T16:21:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 msgr2=0x7fa2341a9fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:21:45.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa234064570 0x7fa2341a9fe0 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa228024000 tx=0x7fa228009260 comp rx=0 tx=0).stop 2024-04-26T16:21:45.938 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a: {} 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 0x7fa23402d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa23402dc60 con 0x7fa23402d310 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 0x7fa23402d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7fa23402de10 con 0x7fa23402d310 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 0x7fa23402d720 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa228009ff0 tx=0x7fa228019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa23402d310 2024-04-26T16:21:45.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 10 client.24611.objecter resend_mon_ops 2024-04-26T16:21:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228037870 con 0x7fa23402d310 2024-04-26T16:21:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.944+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa228037870 con 0x7fa23402d310 2024-04-26T16:21:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.948+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 msgr2=0x7fa23402d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.948+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 0x7fa23402d720 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa228009ff0 tx=0x7fa228019040 comp rx=0 tx=0).stop 2024-04-26T16:21:45.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.948+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 msgr2=0x7fa23402d720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:21:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.948+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa23402d310 0x7fa23402d720 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa228009ff0 tx=0x7fa228019040 comp rx=0 tx=0).stop 2024-04-26T16:21:45.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.948+0000 7fa232d9d700 1 -- 172.21.15.50:0/4242004661 reap_dead start 2024-04-26T16:21:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.952+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 0x7fa2341996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.952+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa2340770d0 con 0x7fa2341992c0 2024-04-26T16:21:45.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.952+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 0x7fa2341996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.956+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7fa2341a3a00 con 0x7fa2341992c0 2024-04-26T16:21:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.956+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 0x7fa2341996a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa23402dde0 tx=0x7fa228035000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.956+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa2341992c0 2024-04-26T16:21:45.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.956+0000 7fa2237fe700 10 client.24611.objecter resend_mon_ops 2024-04-26T16:21:45.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.956+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22804e070 con 0x7fa2341992c0 2024-04-26T16:21:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.960+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7fa228010ea0 con 0x7fa2341992c0 2024-04-26T16:21:45.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.960+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 msgr2=0x7fa2341996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.960+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 0x7fa2341996a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa23402dde0 tx=0x7fa228035000 comp rx=0 tx=0).stop 2024-04-26T16:21:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.960+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 msgr2=0x7fa2341996a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:21:45.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.960+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa2341992c0 0x7fa2341996a0 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa23402dde0 tx=0x7fa228035000 comp rx=0 tx=0).stop 2024-04-26T16:21:45.956 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b: {} 2024-04-26T16:21:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 0x7fa23402e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa2341a3c10 con 0x7fa23402e020 2024-04-26T16:21:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 0x7fa23402e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7fa2340770d0 con 0x7fa23402e020 2024-04-26T16:21:45.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa23402e020 2024-04-26T16:21:45.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 10 client.24611.objecter resend_mon_ops 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 0x7fa23402e400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2280092a0 tx=0x7fa228033000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228059070 con 0x7fa23402e020 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa22800bb50 con 0x7fa23402e020 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 msgr2=0x7fa23402e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 0x7fa23402e400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2280092a0 tx=0x7fa228033000 comp rx=0 tx=0).stop 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 msgr2=0x7fa23402e400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:21:45.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.964+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa23402e020 0x7fa23402e400 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2280092a0 tx=0x7fa228033000 comp rx=0 tx=0).stop 2024-04-26T16:21:45.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 0x7fa2341b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:21:45.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa2341a3c10 con 0x7fa2341a1dd0 2024-04-26T16:21:45.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 0x7fa2341b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:21:45.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa231d9b700 1 -- 172.21.15.50:0/4242004661 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=0"]}) -- 0x7fa2341b0cd0 con 0x7fa2341a1dd0 2024-04-26T16:21:45.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 10 client.24611.objecter ms_handle_connect 0x7fa2341a1dd0 2024-04-26T16:21:45.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa231d9b700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 0x7fa2341b0790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa228005380 tx=0x7fa228051000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:21:45.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 10 client.24611.objecter resend_mon_ops 2024-04-26T16:21:45.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa228061020 con 0x7fa2341a1dd0 2024-04-26T16:21:45.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7fa2280039a0 con 0x7fa2341a1dd0 2024-04-26T16:21:45.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 msgr2=0x7fa2341b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 0x7fa2341b0790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa228005380 tx=0x7fa228051000 comp rx=0 tx=0).stop 2024-04-26T16:21:45.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 msgr2=0x7fa2341b0790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:21:45.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa2237fe700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341a1dd0 0x7fa2341b0790 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa228005380 tx=0x7fa228051000 comp rx=0 tx=0).stop 2024-04-26T16:21:45.976 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c: {} 2024-04-26T16:21:45.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 10 librados: watch_flush enter 2024-04-26T16:21:45.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 10 librados: watch_flush exit 2024-04-26T16:21:45.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 20 client.24611.objecter shutdown clearing up homeless session... 2024-04-26T16:21:45.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 10 client.24611.objecter successfully canceled tick 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa21c071f80 msgr2=0x7fa21c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa21c071f80 0x7fa21c074440 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa224005e10 tx=0x7fa224005d60 comp rx=0 tx=0).stop 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341094b0 msgr2=0x7fa23419db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa238f08700 1 --2- 172.21.15.50:0/4242004661 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa2341094b0 0x7fa23419db40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009cd0 tx=0x7fa22c00c620 comp rx=0 tx=0).stop 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.980+0000 7fa232d9d700 1 -- 172.21.15.50:0/4242004661 reap_dead start 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.984+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 shutdown_connections 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.984+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 >> 172.21.15.50:0/4242004661 conn(0x7fa234078990 msgr2=0x7fa2341062e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.984+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 shutdown_connections 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.984+0000 7fa238f08700 1 -- 172.21.15.50:0/4242004661 wait complete. 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:21:45.984+0000 7fa238f08700 1 librados: shutdown 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T16:21:45.978 INFO:teuthology.orchestra.run.smithi050.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.429693+0000 mgr.y (mgr.24401) 159 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.429693+0000 mgr.y (mgr.24401) 159 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.599829+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.599829+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.602120+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: cluster 2024-04-26T16:21:45.602120+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: audit 2024-04-26T16:21:45.936008+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.50:0/4242004661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:21:46.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[21463]: audit 2024-04-26T16:21:45.936008+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.50:0/4242004661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.429693+0000 mgr.y (mgr.24401) 159 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.429693+0000 mgr.y (mgr.24401) 159 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.599829+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.599829+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.602120+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: cluster 2024-04-26T16:21:45.602120+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: audit 2024-04-26T16:21:45.936008+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.50:0/4242004661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:21:46.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:46 smithi050 bash[29234]: audit 2024-04-26T16:21:45.936008+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.50:0/4242004661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:21:46.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.429693+0000 mgr.y (mgr.24401) 159 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.429693+0000 mgr.y (mgr.24401) 159 : cluster [DBG] pgmap v129: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.599829+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.599829+0000 osd.4 (osd.4) 91 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.602120+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: cluster 2024-04-26T16:21:45.602120+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: audit 2024-04-26T16:21:45.936008+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.50:0/4242004661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:21:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:46 smithi190 bash[23868]: audit 2024-04-26T16:21:45.936008+0000 mon.c (mon.2) 82 : audit [DBG] from='client.? 172.21.15.50:0/4242004661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:21:46.890 INFO:teuthology.run_tasks:Running task workunit... 2024-04-26T16:21:46.906 INFO:tasks.workunit:Pulling workunits from ref a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:21:46.906 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-26T16:21:46.907 DEBUG:teuthology.orchestra.run.smithi050:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-26T16:21:46.907 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-04-26T16:21:46.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:21:46.907 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:21:46.907 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-04-26T16:21:46.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-04-26T16:21:46.908 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-04-26T16:21:46.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:46.909 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-04-26T16:21:46.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:21:46.909 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-04-26T16:21:46.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-26T16:21:46.910 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-04-26T16:21:46.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:21:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:46.918 INFO:teuthology.orchestra.run.smithi050.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-26T16:21:46.918 DEBUG:teuthology.orchestra.run.smithi050:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-26T16:21:46.938 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:46.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:46.966 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-26T16:21:46.966 DEBUG:teuthology.orchestra.run.smithi050:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-26T16:21:47.013 INFO:tasks.workunit:timeout=3h 2024-04-26T16:21:47.013 INFO:tasks.workunit:cleanup=True 2024-04-26T16:21:47.014 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a430f9ed84de3008ca5b632b1ed304bac0e753f6 2024-04-26T16:21:47.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:47.062 INFO:tasks.workunit.client.0.smithi050.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-26T16:21:47.069 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:47.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:47.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:47.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:47.330 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:47.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:47.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:47.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:47.592 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:47.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:47.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:47.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: audit 2024-04-26T16:21:46.501775+0000 mgr.y (mgr.24401) 160 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: audit 2024-04-26T16:21:46.501775+0000 mgr.y (mgr.24401) 160 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:46.575649+0000 osd.4 (osd.4) 93 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:46.575649+0000 osd.4 (osd.4) 93 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:46.578462+0000 osd.4 (osd.4) 94 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:46.578462+0000 osd.4 (osd.4) 94 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.006442+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.006442+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.008167+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:21:47.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:47 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.008167+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:21:47.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:47.854 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: audit 2024-04-26T16:21:46.501775+0000 mgr.y (mgr.24401) 160 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: audit 2024-04-26T16:21:46.501775+0000 mgr.y (mgr.24401) 160 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:46.575649+0000 osd.4 (osd.4) 93 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:46.575649+0000 osd.4 (osd.4) 93 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:46.578462+0000 osd.4 (osd.4) 94 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:46.578462+0000 osd.4 (osd.4) 94 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.006442+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:21:47.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.006442+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.008167+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.008167+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: audit 2024-04-26T16:21:46.501775+0000 mgr.y (mgr.24401) 160 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: audit 2024-04-26T16:21:46.501775+0000 mgr.y (mgr.24401) 160 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:46.575649+0000 osd.4 (osd.4) 93 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:46.575649+0000 osd.4 (osd.4) 93 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:46.578462+0000 osd.4 (osd.4) 94 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:46.578462+0000 osd.4 (osd.4) 94 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.006442+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.006442+0000 osd.3 (osd.3) 117 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.008167+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:21:47.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:47 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.008167+0000 osd.3 (osd.3) 118 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:21:47.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:47.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:47.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:48.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:48.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:48.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:48.252 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:48.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:48.381 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:48.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:48.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:48.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:48.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.073015+0000 osd.0 (osd.0) 101 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.073015+0000 osd.0 (osd.0) 101 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.074378+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.074378+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.080363+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.080363+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.081632+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:21:48.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.081632+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:21:48.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.431324+0000 mgr.y (mgr.24401) 161 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:48.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.431324+0000 mgr.y (mgr.24401) 161 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:48.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.542762+0000 osd.4 (osd.4) 95 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:21:48.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.542762+0000 osd.4 (osd.4) 95 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:21:48.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.545986+0000 osd.4 (osd.4) 96 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:21:48.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:48 smithi190 bash[23868]: cluster 2024-04-26T16:21:47.545986+0000 osd.4 (osd.4) 96 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:21:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:48.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.073015+0000 osd.0 (osd.0) 101 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.073015+0000 osd.0 (osd.0) 101 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.074378+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.074378+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.080363+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.080363+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.081632+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.081632+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.431324+0000 mgr.y (mgr.24401) 161 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.431324+0000 mgr.y (mgr.24401) 161 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:48.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.542762+0000 osd.4 (osd.4) 95 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.542762+0000 osd.4 (osd.4) 95 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.545986+0000 osd.4 (osd.4) 96 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[29234]: cluster 2024-04-26T16:21:47.545986+0000 osd.4 (osd.4) 96 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.073015+0000 osd.0 (osd.0) 101 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.073015+0000 osd.0 (osd.0) 101 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.074378+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.074378+0000 osd.0 (osd.0) 102 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.080363+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.080363+0000 osd.7 (osd.7) 125 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.081632+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.081632+0000 osd.7 (osd.7) 126 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:21:48.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.431324+0000 mgr.y (mgr.24401) 161 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:48.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.431324+0000 mgr.y (mgr.24401) 161 : cluster [DBG] pgmap v130: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:21:48.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.542762+0000 osd.4 (osd.4) 95 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:21:48.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.542762+0000 osd.4 (osd.4) 95 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:21:48.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.545986+0000 osd.4 (osd.4) 96 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:21:48.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:48 smithi050 bash[21463]: cluster 2024-04-26T16:21:47.545986+0000 osd.4 (osd.4) 96 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:21:48.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:48.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:48.900 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:48.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:49.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:49.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:49.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:49.161 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:49.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:49.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:49.422 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:49.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:49.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:49.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:49.685 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:49.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:49.816 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:49.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:49.944 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:49.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:50.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:50.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:50.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:50.204 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:50.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:50.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:50.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:50.466 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:50.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:50.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.025037+0000 osd.7 (osd.7) 127 : cluster [DBG] 3.f scrub starts 2024-04-26T16:21:50.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.025037+0000 osd.7 (osd.7) 127 : cluster [DBG] 3.f scrub starts 2024-04-26T16:21:50.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.030057+0000 osd.7 (osd.7) 128 : cluster [DBG] 3.f scrub ok 2024-04-26T16:21:50.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.030057+0000 osd.7 (osd.7) 128 : cluster [DBG] 3.f scrub ok 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.110740+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.110740+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.113135+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.113135+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.432042+0000 mgr.y (mgr.24401) 162 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[21463]: cluster 2024-04-26T16:21:49.432042+0000 mgr.y (mgr.24401) 162 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.025037+0000 osd.7 (osd.7) 127 : cluster [DBG] 3.f scrub starts 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.025037+0000 osd.7 (osd.7) 127 : cluster [DBG] 3.f scrub starts 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.030057+0000 osd.7 (osd.7) 128 : cluster [DBG] 3.f scrub ok 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.030057+0000 osd.7 (osd.7) 128 : cluster [DBG] 3.f scrub ok 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.110740+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:21:50.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.110740+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:21:50.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.113135+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:21:50.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.113135+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:21:50.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.432042+0000 mgr.y (mgr.24401) 162 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:50.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:50 smithi050 bash[29234]: cluster 2024-04-26T16:21:49.432042+0000 mgr.y (mgr.24401) 162 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:50.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:50.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:50.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.025037+0000 osd.7 (osd.7) 127 : cluster [DBG] 3.f scrub starts 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.025037+0000 osd.7 (osd.7) 127 : cluster [DBG] 3.f scrub starts 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.030057+0000 osd.7 (osd.7) 128 : cluster [DBG] 3.f scrub ok 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.030057+0000 osd.7 (osd.7) 128 : cluster [DBG] 3.f scrub ok 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.110740+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.110740+0000 osd.1 (osd.1) 107 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.113135+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.113135+0000 osd.1 (osd.1) 108 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:21:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.432042+0000 mgr.y (mgr.24401) 162 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:50 smithi190 bash[23868]: cluster 2024-04-26T16:21:49.432042+0000 mgr.y (mgr.24401) 162 : cluster [DBG] pgmap v131: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:21:50.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:50.861 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:50.964 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:50.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:50.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:51.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:51.121 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:51.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:51.252 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:51.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:51.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:51.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:51.512 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:51.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:51.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.088217+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:21:51.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.088217+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:21:51.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.089877+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:21:51.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.089877+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:21:51.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.398449+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:21:51.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.398449+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.402709+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:50.402709+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.352432+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1d deep-scrub starts 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.352432+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1d deep-scrub starts 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.362086+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1d deep-scrub ok 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.362086+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1d deep-scrub ok 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.088217+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.088217+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.089877+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.089877+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.398449+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:21:51.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.398449+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:21:51.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.402709+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:21:51.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:50.402709+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:21:51.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.352432+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1d deep-scrub starts 2024-04-26T16:21:51.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.352432+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1d deep-scrub starts 2024-04-26T16:21:51.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.362086+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1d deep-scrub ok 2024-04-26T16:21:51.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:51 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.362086+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1d deep-scrub ok 2024-04-26T16:21:51.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:51.724 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:51.724 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:51.724 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:21:51.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:51.779 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.088217+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.088217+0000 osd.0 (osd.0) 103 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.089877+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.089877+0000 osd.0 (osd.0) 104 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.398449+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.398449+0000 osd.5 (osd.5) 107 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.402709+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:50.402709+0000 osd.5 (osd.5) 108 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.352432+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1d deep-scrub starts 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.352432+0000 osd.5 (osd.5) 109 : cluster [DBG] 3.1d deep-scrub starts 2024-04-26T16:21:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.362086+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1d deep-scrub ok 2024-04-26T16:21:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:51 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.362086+0000 osd.5 (osd.5) 110 : cluster [DBG] 3.1d deep-scrub ok 2024-04-26T16:21:51.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:51.909 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:51.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:52.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:52.040 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:52.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:52.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:52.170 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:52.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:52.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:52.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:52.429 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:52.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:52.562 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:52.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:52.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.042113+0000 osd.7 (osd.7) 129 : cluster [DBG] 3.d scrub starts 2024-04-26T16:21:52.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.042113+0000 osd.7 (osd.7) 129 : cluster [DBG] 3.d scrub starts 2024-04-26T16:21:52.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.046690+0000 osd.7 (osd.7) 130 : cluster [DBG] 3.d scrub ok 2024-04-26T16:21:52.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.046690+0000 osd.7 (osd.7) 130 : cluster [DBG] 3.d scrub ok 2024-04-26T16:21:52.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.432927+0000 mgr.y (mgr.24401) 163 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:52.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:51.432927+0000 mgr.y (mgr.24401) 163 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:52.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:52.070570+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:21:52.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:52.070570+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:21:52.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:52.074389+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:21:52.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:52 smithi190 bash[23868]: cluster 2024-04-26T16:21:52.074389+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:21:52.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:52.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:52.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.042113+0000 osd.7 (osd.7) 129 : cluster [DBG] 3.d scrub starts 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.042113+0000 osd.7 (osd.7) 129 : cluster [DBG] 3.d scrub starts 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.046690+0000 osd.7 (osd.7) 130 : cluster [DBG] 3.d scrub ok 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.046690+0000 osd.7 (osd.7) 130 : cluster [DBG] 3.d scrub ok 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.432927+0000 mgr.y (mgr.24401) 163 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:51.432927+0000 mgr.y (mgr.24401) 163 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:52.070570+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:21:52.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:52.070570+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:52.074389+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[21463]: cluster 2024-04-26T16:21:52.074389+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.042113+0000 osd.7 (osd.7) 129 : cluster [DBG] 3.d scrub starts 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.042113+0000 osd.7 (osd.7) 129 : cluster [DBG] 3.d scrub starts 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.046690+0000 osd.7 (osd.7) 130 : cluster [DBG] 3.d scrub ok 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.046690+0000 osd.7 (osd.7) 130 : cluster [DBG] 3.d scrub ok 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.432927+0000 mgr.y (mgr.24401) 163 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:52.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:51.432927+0000 mgr.y (mgr.24401) 163 : cluster [DBG] pgmap v132: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:21:52.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:52.070570+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:21:52.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:52.070570+0000 osd.3 (osd.3) 119 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:21:52.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:52.074389+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:21:52.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:52 smithi050 bash[29234]: cluster 2024-04-26T16:21:52.074389+0000 osd.3 (osd.3) 120 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:21:52.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:52.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:52.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:52.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:52.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:53.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:53.086 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:53.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:53.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:53.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:53.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:53.349 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:53.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:53.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:53.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:53.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:53.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:53.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:53.741 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:53.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:53.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:53.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:53.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:54.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:54.127 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:54.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:54.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:54.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:54.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:54.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:54.497 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:21:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:21:54.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:54.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:54.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:54.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:54.774 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.053165+0000 osd.1 (osd.1) 109 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.053165+0000 osd.1 (osd.1) 109 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.056739+0000 osd.1 (osd.1) 110 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.056739+0000 osd.1 (osd.1) 110 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.096973+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.096973+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.099045+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.099045+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.433692+0000 mgr.y (mgr.24401) 164 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:54.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:53.433692+0000 mgr.y (mgr.24401) 164 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:54.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.338581+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:21:54.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.338581+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:21:54.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.340836+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:21:54.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:54 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.340836+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.053165+0000 osd.1 (osd.1) 109 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.053165+0000 osd.1 (osd.1) 109 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.056739+0000 osd.1 (osd.1) 110 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.056739+0000 osd.1 (osd.1) 110 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.096973+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.096973+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.099045+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:21:54.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.099045+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.433692+0000 mgr.y (mgr.24401) 164 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:53.433692+0000 mgr.y (mgr.24401) 164 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.338581+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.338581+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.340836+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.340836+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.053165+0000 osd.1 (osd.1) 109 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.053165+0000 osd.1 (osd.1) 109 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.056739+0000 osd.1 (osd.1) 110 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.056739+0000 osd.1 (osd.1) 110 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.096973+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:21:54.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.096973+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.099045+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.099045+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.433692+0000 mgr.y (mgr.24401) 164 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:53.433692+0000 mgr.y (mgr.24401) 164 : cluster [DBG] pgmap v133: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.338581+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.338581+0000 osd.5 (osd.5) 111 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.340836+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:21:54.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:54 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.340836+0000 osd.5 (osd.5) 112 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:21:54.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:54.906 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:54.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:54.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:55.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:55.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.172 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:55.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:55.300 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:55.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:55.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.432 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:55.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:55.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:55.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.096479+0000 osd.1 (osd.1) 111 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.096479+0000 osd.1 (osd.1) 111 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.097488+0000 osd.1 (osd.1) 112 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.097488+0000 osd.1 (osd.1) 112 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.581493+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.581493+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.581602+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:54.581602+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.324264+0000 osd.5 (osd.5) 113 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.324264+0000 osd.5 (osd.5) 113 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.328139+0000 osd.5 (osd.5) 114 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:21:55.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.328139+0000 osd.5 (osd.5) 114 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.096479+0000 osd.1 (osd.1) 111 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.096479+0000 osd.1 (osd.1) 111 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.097488+0000 osd.1 (osd.1) 112 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.097488+0000 osd.1 (osd.1) 112 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.581493+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.581493+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.581602+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:55.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:54.581602+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:55.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.324264+0000 osd.5 (osd.5) 113 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:21:55.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.324264+0000 osd.5 (osd.5) 113 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:21:55.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.328139+0000 osd.5 (osd.5) 114 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:21:55.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:55 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.328139+0000 osd.5 (osd.5) 114 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:21:55.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.096479+0000 osd.1 (osd.1) 111 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:21:55.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.096479+0000 osd.1 (osd.1) 111 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:21:55.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.097488+0000 osd.1 (osd.1) 112 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:21:55.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.097488+0000 osd.1 (osd.1) 112 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.581493+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.581493+0000 osd.7 (osd.7) 131 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.581602+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:54.581602+0000 osd.7 (osd.7) 132 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.324264+0000 osd.5 (osd.5) 113 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.324264+0000 osd.5 (osd.5) 113 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.328139+0000 osd.5 (osd.5) 114 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:21:55.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:55 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.328139+0000 osd.5 (osd.5) 114 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:21:55.699 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:55.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.829 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:21:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:21:55.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:55.963 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:55.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:56.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:56.092 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:56.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:56.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:56.222 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:56.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:56.353 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:56.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:56.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:56.617 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:56.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:56.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.434527+0000 mgr.y (mgr.24401) 165 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-04-26T16:21:56.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.434527+0000 mgr.y (mgr.24401) 165 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-04-26T16:21:56.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.568047+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-04-26T16:21:56.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[21463]: cluster 2024-04-26T16:21:55.568047+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-04-26T16:21:56.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[21463]: audit 2024-04-26T16:21:55.839218+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:56.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[21463]: audit 2024-04-26T16:21:55.839218+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:56.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.434527+0000 mgr.y (mgr.24401) 165 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-04-26T16:21:56.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.434527+0000 mgr.y (mgr.24401) 165 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-04-26T16:21:56.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.568047+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-04-26T16:21:56.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[29234]: cluster 2024-04-26T16:21:55.568047+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-04-26T16:21:56.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[29234]: audit 2024-04-26T16:21:55.839218+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:56.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:56 smithi050 bash[29234]: audit 2024-04-26T16:21:55.839218+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:56.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:56.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:21:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.434527+0000 mgr.y (mgr.24401) 165 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-04-26T16:21:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.434527+0000 mgr.y (mgr.24401) 165 : cluster [DBG] pgmap v134: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-04-26T16:21:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.568047+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-04-26T16:21:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[23868]: cluster 2024-04-26T16:21:55.568047+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2024-04-26T16:21:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[23868]: audit 2024-04-26T16:21:55.839218+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:56.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:56 smithi190 bash[23868]: audit 2024-04-26T16:21:55.839218+0000 mon.c (mon.2) 83 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:21:56.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:56.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:56.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:56.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:57.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:57.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:57.141 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:57.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:57.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:57.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:57.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:57.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:57.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:57.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:57.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:57.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:57.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:57.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:57 smithi190 bash[23868]: audit 2024-04-26T16:21:56.513277+0000 mgr.y (mgr.24401) 166 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:57.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:57 smithi190 bash[23868]: audit 2024-04-26T16:21:56.513277+0000 mgr.y (mgr.24401) 166 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:57.798 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:21:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:21:57.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:57 smithi050 bash[21463]: audit 2024-04-26T16:21:56.513277+0000 mgr.y (mgr.24401) 166 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:57.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:57 smithi050 bash[21463]: audit 2024-04-26T16:21:56.513277+0000 mgr.y (mgr.24401) 166 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:57 smithi050 bash[29234]: audit 2024-04-26T16:21:56.513277+0000 mgr.y (mgr.24401) 166 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:57.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:57 smithi050 bash[29234]: audit 2024-04-26T16:21:56.513277+0000 mgr.y (mgr.24401) 166 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:21:57.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:57.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:57.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:58.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:58.059 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:58.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:58.190 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:58.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:58.320 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:58.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:58.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:58.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:58.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:58.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.008925+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.008925+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.011413+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.011413+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.090326+0000 osd.1 (osd.1) 113 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.090326+0000 osd.1 (osd.1) 113 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:21:58.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.094566+0000 osd.1 (osd.1) 114 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.094566+0000 osd.1 (osd.1) 114 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.416203+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f scrub starts 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.416203+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f scrub starts 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.417865+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f scrub ok 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.417865+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f scrub ok 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.435610+0000 mgr.y (mgr.24401) 167 : cluster [DBG] pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.435610+0000 mgr.y (mgr.24401) 167 : cluster [DBG] pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.522140+0000 osd.4 (osd.4) 97 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.522140+0000 osd.4 (osd.4) 97 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.526130+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:21:58.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[21463]: cluster 2024-04-26T16:21:57.526130+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.008925+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.008925+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.011413+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.011413+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.090326+0000 osd.1 (osd.1) 113 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.090326+0000 osd.1 (osd.1) 113 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.094566+0000 osd.1 (osd.1) 114 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.094566+0000 osd.1 (osd.1) 114 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.416203+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f scrub starts 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.416203+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f scrub starts 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.417865+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.417865+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f scrub ok 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.435610+0000 mgr.y (mgr.24401) 167 : cluster [DBG] pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:58.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.435610+0000 mgr.y (mgr.24401) 167 : cluster [DBG] pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:58.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.522140+0000 osd.4 (osd.4) 97 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:21:58.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.522140+0000 osd.4 (osd.4) 97 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:21:58.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.526130+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:21:58.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:58 smithi050 bash[29234]: cluster 2024-04-26T16:21:57.526130+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:21:58.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.008925+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:21:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.008925+0000 osd.0 (osd.0) 107 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:21:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.011413+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:21:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.011413+0000 osd.0 (osd.0) 108 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:21:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.090326+0000 osd.1 (osd.1) 113 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.090326+0000 osd.1 (osd.1) 113 : cluster [DBG] 3.4 deep-scrub starts 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.094566+0000 osd.1 (osd.1) 114 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.094566+0000 osd.1 (osd.1) 114 : cluster [DBG] 3.4 deep-scrub ok 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.416203+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f scrub starts 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.416203+0000 osd.2 (osd.2) 81 : cluster [DBG] 6.f scrub starts 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.417865+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f scrub ok 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.417865+0000 osd.2 (osd.2) 82 : cluster [DBG] 6.f scrub ok 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.435610+0000 mgr.y (mgr.24401) 167 : cluster [DBG] pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.435610+0000 mgr.y (mgr.24401) 167 : cluster [DBG] pgmap v136: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.522140+0000 osd.4 (osd.4) 97 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.522140+0000 osd.4 (osd.4) 97 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.526130+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:21:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:58 smithi190 bash[23868]: cluster 2024-04-26T16:21:57.526130+0000 osd.4 (osd.4) 98 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:21:58.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:58.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:58.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:58.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:21:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:58.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:21:59.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:59.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:59.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:59.244 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:59.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:59.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:21:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:21:59.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:59.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:21:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:21:59.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:59.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.042604+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.042604+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.045064+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.045064+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.058267+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.058267+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.060710+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.060710+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.227458+0000 osd.3 (osd.3) 121 : cluster [DBG] 6.1b deep-scrub starts 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.227458+0000 osd.3 (osd.3) 121 : cluster [DBG] 6.1b deep-scrub starts 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.228922+0000 osd.3 (osd.3) 122 : cluster [DBG] 6.1b deep-scrub ok 2024-04-26T16:21:59.623 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[21463]: cluster 2024-04-26T16:21:58.228922+0000 osd.3 (osd.3) 122 : cluster [DBG] 6.1b deep-scrub ok 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.042604+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.042604+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.045064+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.045064+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.058267+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.058267+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.060710+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.060710+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.227458+0000 osd.3 (osd.3) 121 : cluster [DBG] 6.1b deep-scrub starts 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.227458+0000 osd.3 (osd.3) 121 : cluster [DBG] 6.1b deep-scrub starts 2024-04-26T16:21:59.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.228922+0000 osd.3 (osd.3) 122 : cluster [DBG] 6.1b deep-scrub ok 2024-04-26T16:21:59.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:21:59 smithi050 bash[29234]: cluster 2024-04-26T16:21:58.228922+0000 osd.3 (osd.3) 122 : cluster [DBG] 6.1b deep-scrub ok 2024-04-26T16:21:59.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:21:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:21:59.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:59.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:21:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:21:59.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:59.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.042604+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-04-26T16:21:59.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.042604+0000 osd.1 (osd.1) 115 : cluster [DBG] 2.d scrub starts 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.045064+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.045064+0000 osd.1 (osd.1) 116 : cluster [DBG] 2.d scrub ok 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.058267+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.058267+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.060710+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.060710+0000 osd.0 (osd.0) 110 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.227458+0000 osd.3 (osd.3) 121 : cluster [DBG] 6.1b deep-scrub starts 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.227458+0000 osd.3 (osd.3) 121 : cluster [DBG] 6.1b deep-scrub starts 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.228922+0000 osd.3 (osd.3) 122 : cluster [DBG] 6.1b deep-scrub ok 2024-04-26T16:21:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:21:59 smithi190 bash[23868]: cluster 2024-04-26T16:21:58.228922+0000 osd.3 (osd.3) 122 : cluster [DBG] 6.1b deep-scrub ok 2024-04-26T16:21:59.895 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:21:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:21:59.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:21:59.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:00.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- 172.21.15.50:0/1746964824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e001023b0 msgr2=0x7f7e001102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- 172.21.15.50:0/1746964824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e001023b0 0x7f7e001102f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7df8009f90 tx=0x7f7df800b330 comp rx=0 tx=0).stop 2024-04-26T16:22:00.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- 172.21.15.50:0/1746964824 shutdown_connections 2024-04-26T16:22:00.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- 172.21.15.50:0/1746964824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e001023b0 0x7f7e001102f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- 172.21.15.50:0/1746964824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e001094b0 0x7f7e00109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- 172.21.15.50:0/1746964824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e00101a10 0x7f7e00101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- 172.21.15.50:0/1746964824 >> 172.21.15.50:0/1746964824 conn(0x7f7e00078990 msgr2=0x7f7e000ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:00.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- 172.21.15.50:0/1746964824 shutdown_connections 2024-04-26T16:22:00.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- 172.21.15.50:0/1746964824 wait complete. 2024-04-26T16:22:00.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 librados: starting msgr at 2024-04-26T16:22:00.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 librados: starting objecter 2024-04-26T16:22:00.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 Processor -- start 2024-04-26T16:22:00.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- start start 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 librados: setting wanted keys 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 librados: calling monclient init 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e00101a10 0x7f7e00102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e001023b0 0x7f7e00103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e001094b0 0x7f7e00103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7e00112b30 con 0x7f7e001023b0 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001129b0 con 0x7f7e001094b0 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.008+0000 7f7e07d19700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7e00112cb0 con 0x7f7e00101a10 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e05ab5700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e00101a10 0x7f7e00102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e001023b0 0x7f7e00103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e05ab5700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e00101a10 0x7f7e00102bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51756/0 (socket says 172.21.15.50:51756) 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e05ab5700 1 -- 172.21.15.50:0/409522712 learned_addr learned my addr 172.21.15.50:0/409522712 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:00.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e00101a10 msgr2=0x7f7e00102bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e00101a10 0x7f7e00102bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e001094b0 msgr2=0x7f7e00103670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7e001094b0 0x7f7e00103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e001a7660 con 0x7f7e001023b0 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7e00101a10 0x7f7e00102bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e001023b0 0x7f7e00103130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7df000b450 tx=0x7f7df000b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 10 client.?.objecter ms_handle_connect 0x7f7e001023b0 2024-04-26T16:22:00.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0015020 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e001a7950 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7df0011c90 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0010ae0 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e07d19700 10 client.14733.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e001a7e90 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e07d19700 1 librados: init done 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7e0006a110 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7df0018020 con 0x7f7e001023b0 2024-04-26T16:22:00.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7dec071f80 0x7f7dec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(127..127 src has 77..127) ==== 7450+0+0 (secure 0 0 0) 0x7f7df0090940 con 0x7f7e001023b0 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 osd_map(127..127 src has 77..127) 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 3 client.14733.objecter handle_osd_map got epochs [127,127] > 0 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 3 client.14733.objecter handle_osd_map decoding full epoch 127 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7dec071f80 0x7f7dec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 20 client.14733.objecter dump_active .. 0 homeless 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.012+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v127) ==== 54+0+16 (secure 0 0 0) 0x7f7df001f020 con 0x7f7e001023b0 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 10 client.14733.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 1 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7e00064280 0x7f7e0010a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 1 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 0 1 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 4 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 10 client.14733.objecter _send_command 1 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e00104d10 con 0x7f7e00064280 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7dec071f80 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7dec071f80 0x7f7dec074440 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc009750 tx=0x7f7dfc009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e062b6700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7e00064280 0x7f7e0010a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e00064280 2024-04-26T16:22:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7e062b6700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7e00064280 0x7f7e0010a110 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e00104d10 con 0x7f7e00064280 2024-04-26T16:22:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 1 = system:0 2024-04-26T16:22:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 4 2024-04-26T16:22:00.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.016+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 0 1 2024-04-26T16:22:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 10 client.14733.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 2 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 2 2024-04-26T16:22:00.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001040d0 osd=0 3 2024-04-26T16:22:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 0 2 2024-04-26T16:22:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 4 2024-04-26T16:22:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 10 client.14733.objecter _send_command 2 2024-04-26T16:22:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e0010a650 con 0x7f7e00064280 2024-04-26T16:22:00.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e0010a650 con 0x7f7e00064280 2024-04-26T16:22:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 4 2024-04-26T16:22:00.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 0 2 2024-04-26T16:22:00.051 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 10 client.14733.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7e00063e70 0x7f7e001ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 1 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 4 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 10 client.14733.objecter _send_command 3 2024-04-26T16:22:00.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7e0002dff0 con 0x7f7e00063e70 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7e00063e70 0x7f7e001ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e00063e70 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7e00063e70 0x7f7e001ac0e0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e0002dff0 con 0x7f7e00063e70 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 3: 0 ) 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 3 = system:0 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 4 2024-04-26T16:22:00.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.056+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 1 3 2024-04-26T16:22:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 10 client.14733.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 4 2024-04-26T16:22:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 4 2024-04-26T16:22:00.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e00063d00 osd=1 3 2024-04-26T16:22:00.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 1 4 2024-04-26T16:22:00.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 4 2024-04-26T16:22:00.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 10 client.14733.objecter _send_command 4 2024-04-26T16:22:00.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.076+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e001ac620 con 0x7f7e00063e70 2024-04-26T16:22:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e001ac620 con 0x7f7e00063e70 2024-04-26T16:22:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 4 2024-04-26T16:22:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 1 4 2024-04-26T16:22:00.073 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:22:00.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 10 client.14733.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 5 2024-04-26T16:22:00.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7e0002d6d0 0x7f7e001af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 5 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 2 5 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 4 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 10 client.14733.objecter _send_command 5 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7e0010b6f0 con 0x7f7e0002d6d0 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7e0002d6d0 0x7f7e001af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e0002d6d0 2024-04-26T16:22:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7e0002d6d0 0x7f7e001af960 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e0010b6f0 con 0x7f7e0002d6d0 2024-04-26T16:22:00.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 5: 0 ) 2024-04-26T16:22:00.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 5 = system:0 2024-04-26T16:22:00.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 4 2024-04-26T16:22:00.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.080+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 2 5 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 10 client.14733.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 6 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 6 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e0002d560 osd=2 3 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 2 6 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 4 2024-04-26T16:22:00.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 10 client.14733.objecter _send_command 6 2024-04-26T16:22:00.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e0010aa50 con 0x7f7e0002d6d0 2024-04-26T16:22:00.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e0010aa50 con 0x7f7e0002d6d0 2024-04-26T16:22:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 4 2024-04-26T16:22:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 2 6 2024-04-26T16:22:00.095 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 10 client.14733.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 7 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7e001b1020 0x7f7e001b3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 7 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 3 7 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 4 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 10 client.14733.objecter _send_command 7 2024-04-26T16:22:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7e001ad750 con 0x7f7e001b1020 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e062b6700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7e001b1020 0x7f7e001b3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e001b1020 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.100+0000 7f7e062b6700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7e001b1020 0x7f7e001b3410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.104+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e001ad750 con 0x7f7e001b1020 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.104+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 7: 0 ) 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.104+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 7 = system:0 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.104+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 4 2024-04-26T16:22:00.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.104+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 3 7 2024-04-26T16:22:00.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 10 client.14733.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 8 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 8 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001acfa0 osd=3 3 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 3 8 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 4 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 10 client.14733.objecter _send_command 8 2024-04-26T16:22:00.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.140+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e001acae0 con 0x7f7e001b1020 2024-04-26T16:22:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e001acae0 con 0x7f7e001b1020 2024-04-26T16:22:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 4 2024-04-26T16:22:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 3 8 2024-04-26T16:22:00.137 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 10 client.14733.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 9 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7e001b0b80 0x7f7e001b4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 9 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 4 9 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 4 2024-04-26T16:22:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 10 client.14733.objecter _send_command 9 2024-04-26T16:22:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7e001b9ad0 con 0x7f7e001b0b80 2024-04-26T16:22:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7e001b0b80 0x7f7e001b4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e001b0b80 2024-04-26T16:22:00.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7e001b0b80 0x7f7e001b4ad0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e001b9ad0 con 0x7f7e001b0b80 2024-04-26T16:22:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 9: 0 ) 2024-04-26T16:22:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 9 = system:0 2024-04-26T16:22:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 4 2024-04-26T16:22:00.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.144+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 4 9 2024-04-26T16:22:00.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:00.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 10 client.14733.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 10 2024-04-26T16:22:00.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 10 2024-04-26T16:22:00.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001acae0 osd=4 3 2024-04-26T16:22:00.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 4 10 2024-04-26T16:22:00.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 4 2024-04-26T16:22:00.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 10 client.14733.objecter _send_command 10 2024-04-26T16:22:00.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.164+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e001b9ad0 con 0x7f7e001b0b80 2024-04-26T16:22:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.168+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e001b9ad0 con 0x7f7e001b0b80 2024-04-26T16:22:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.168+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.168+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.168+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 4 2024-04-26T16:22:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.168+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 4 10 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 10 client.14733.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 11 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7e001b5690 0x7f7e001b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 11 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 5 11 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 4 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 10 client.14733.objecter _send_command 11 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7e001bce40 con 0x7f7e001b5690 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7e001b5690 0x7f7e001b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e001b5690 2024-04-26T16:22:00.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7e052b4700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7e001b5690 0x7f7e001b5af0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e001bce40 con 0x7f7e001b5690 2024-04-26T16:22:00.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 11: 0 ) 2024-04-26T16:22:00.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 11 = system:0 2024-04-26T16:22:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 4 2024-04-26T16:22:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.172+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 5 11 2024-04-26T16:22:00.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 10 client.14733.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 12 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 12 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001b9ad0 osd=5 3 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 5 12 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 4 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 10 client.14733.objecter _send_command 12 2024-04-26T16:22:00.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.192+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e001b82e0 con 0x7f7e001b5690 2024-04-26T16:22:00.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e001b82e0 con 0x7f7e001b5690 2024-04-26T16:22:00.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 4 2024-04-26T16:22:00.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 5 12 2024-04-26T16:22:00.190 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:22:00.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 10 client.14733.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 13 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7e001b8a00 0x7f7e001b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 13 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 6 13 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 4 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 10 client.14733.objecter _send_command 13 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7e001b9500 con 0x7f7e001b8a00 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e062b6700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7e001b8a00 0x7f7e001b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e001b8a00 2024-04-26T16:22:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7e062b6700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7e001b8a00 0x7f7e001b8e60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e001b9500 con 0x7f7e001b8a00 2024-04-26T16:22:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 13: 0 ) 2024-04-26T16:22:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 13 = system:0 2024-04-26T16:22:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 4 2024-04-26T16:22:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.196+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 6 13 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 10 client.14733.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 14 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 14 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001b82e0 osd=6 3 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 6 14 2024-04-26T16:22:00.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 4 2024-04-26T16:22:00.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 10 client.14733.objecter _send_command 14 2024-04-26T16:22:00.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.216+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e001b9500 con 0x7f7e001b8a00 2024-04-26T16:22:00.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e001b9500 con 0x7f7e001b8a00 2024-04-26T16:22:00.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 4 2024-04-26T16:22:00.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 6 14 2024-04-26T16:22:00.214 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 10 client.14733.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 15 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7e001be1e0 0x7f7e001c05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 15 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 7 15 2024-04-26T16:22:00.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 4 2024-04-26T16:22:00.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 10 client.14733.objecter _send_command 15 2024-04-26T16:22:00.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7e001c3430 con 0x7f7e001be1e0 2024-04-26T16:22:00.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7e001be1e0 0x7f7e001c05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:00.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7df6ffd700 10 client.14733.objecter ms_handle_connect 0x7f7e001be1e0 2024-04-26T16:22:00.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.220+0000 7f7e05ab5700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7e001be1e0 0x7f7e001c05d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:00.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.224+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e001c3430 con 0x7f7e001be1e0 2024-04-26T16:22:00.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.224+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 15: 0 ) 2024-04-26T16:22:00.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.224+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 15 = system:0 2024-04-26T16:22:00.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.224+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 4 2024-04-26T16:22:00.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.224+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 7 15 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 10 client.14733.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign -1 16 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 20 client.14733.objecter _get_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_remove -1 16 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 20 client.14733.objecter get_session s=0x7f7e001b9500 osd=7 3 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 15 client.14733.objecter _session_command_op_assign 7 16 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 4 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 10 client.14733.objecter _send_command 16 2024-04-26T16:22:00.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.240+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f7e001c0b10 con 0x7f7e001be1e0 2024-04-26T16:22:00.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7df6ffd700 1 -- 172.21.15.50:0/409522712 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7e001c0b10 con 0x7f7e001be1e0 2024-04-26T16:22:00.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7df6ffd700 10 client.14733.objecter ms_dispatch 0x7f7e00001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:00.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7df6ffd700 10 client.14733.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:00.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7df6ffd700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 4 2024-04-26T16:22:00.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7df6ffd700 15 client.14733.objecter _session_command_op_remove 7 16 2024-04-26T16:22:00.239 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 librados: watch_flush enter 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 librados: watch_flush exit 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.0 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7e00064280 msgr2=0x7f7e0010a110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7e00064280 0x7f7e0010a110 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001040d0 osd=0 1 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.1 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7e00063e70 msgr2=0x7f7e001ac0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7e00063e70 0x7f7e001ac0e0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e00063d00 osd=1 1 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.2 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7e0002d6d0 msgr2=0x7f7e001af960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7e0002d6d0 0x7f7e001af960 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e0002d560 osd=2 1 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.3 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7e001b1020 msgr2=0x7f7e001b3410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7e001b1020 0x7f7e001b3410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acfa0 osd=3 1 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.4 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7e001b0b80 msgr2=0x7f7e001b4ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7e001b0b80 0x7f7e001b4ad0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001acae0 osd=4 1 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.5 2024-04-26T16:22:00.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7e001b5690 msgr2=0x7f7e001b5af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7e001b5690 0x7f7e001b5af0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9ad0 osd=5 1 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.6 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7e001b8a00 msgr2=0x7f7e001b8e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7e001b8a00 0x7f7e001b8e60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b82e0 osd=6 1 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter close_session for osd.7 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7e001be1e0 msgr2=0x7f7e001c05d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7e001be1e0 0x7f7e001c05d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter put_session s=0x7f7e001b9500 osd=7 1 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 20 client.14733.objecter shutdown clearing up homeless session... 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 10 client.14733.objecter successfully canceled tick 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e062b6700 1 -- 172.21.15.50:0/409522712 reap_dead start 2024-04-26T16:22:00.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e062b6700 1 -- 172.21.15.50:0/409522712 reap_dead start 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7dec071f80 msgr2=0x7f7dec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7dec071f80 0x7f7dec074440 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc009750 tx=0x7f7dfc009310 comp rx=0 tx=0).stop 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e001023b0 msgr2=0x7f7e00103130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 --2- 172.21.15.50:0/409522712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7e001023b0 0x7f7e00103130 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7df000b450 tx=0x7f7df000b760 comp rx=0 tx=0).stop 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e062b6700 1 -- 172.21.15.50:0/409522712 reap_dead start 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e062b6700 1 -- 172.21.15.50:0/409522712 reap_dead start 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e062b6700 1 -- 172.21.15.50:0/409522712 reap_dead start 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 shutdown_connections 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 >> 172.21.15.50:0/409522712 conn(0x7f7e00078990 msgr2=0x7f7e000ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 shutdown_connections 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 -- 172.21.15.50:0/409522712 wait complete. 2024-04-26T16:22:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:00.244+0000 7f7e07d19700 1 librados: shutdown 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:22:00.245 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:22:00.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:00.270 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:00.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:00.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:00.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:00.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:00.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:00.646 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:00.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:00.777 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:00.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: cluster 2024-04-26T16:21:59.104995+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:22:00.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: cluster 2024-04-26T16:21:59.104995+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:22:00.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: cluster 2024-04-26T16:21:59.107295+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:22:00.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: cluster 2024-04-26T16:21:59.107295+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:22:00.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: cluster 2024-04-26T16:21:59.436266+0000 mgr.y (mgr.24401) 168 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:00.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: cluster 2024-04-26T16:21:59.436266+0000 mgr.y (mgr.24401) 168 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:21:59.721459+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:21:59.721459+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.018337+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? 172.21.15.50:0/409522712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.018337+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? 172.21.15.50:0/409522712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.446250+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.446250+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.447123+0000 mon.c (mon.2) 86 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.447123+0000 mon.c (mon.2) 86 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.453135+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[29234]: audit 2024-04-26T16:22:00.453135+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:00.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: cluster 2024-04-26T16:21:59.104995+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: cluster 2024-04-26T16:21:59.104995+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: cluster 2024-04-26T16:21:59.107295+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: cluster 2024-04-26T16:21:59.107295+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: cluster 2024-04-26T16:21:59.436266+0000 mgr.y (mgr.24401) 168 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: cluster 2024-04-26T16:21:59.436266+0000 mgr.y (mgr.24401) 168 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:21:59.721459+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:21:59.721459+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.018337+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? 172.21.15.50:0/409522712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.018337+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? 172.21.15.50:0/409522712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.446250+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.446250+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.447123+0000 mon.c (mon.2) 86 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.447123+0000 mon.c (mon.2) 86 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.453135+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:00 smithi050 bash[21463]: audit 2024-04-26T16:22:00.453135+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:00.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: cluster 2024-04-26T16:21:59.104995+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: cluster 2024-04-26T16:21:59.104995+0000 osd.0 (osd.0) 111 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: cluster 2024-04-26T16:21:59.107295+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: cluster 2024-04-26T16:21:59.107295+0000 osd.0 (osd.0) 112 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: cluster 2024-04-26T16:21:59.436266+0000 mgr.y (mgr.24401) 168 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: cluster 2024-04-26T16:21:59.436266+0000 mgr.y (mgr.24401) 168 : cluster [DBG] pgmap v137: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:21:59.721459+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:21:59.721459+0000 mon.c (mon.2) 84 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.018337+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? 172.21.15.50:0/409522712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.018337+0000 mon.a (mon.0) 940 : audit [DBG] from='client.? 172.21.15.50:0/409522712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.446250+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:00.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.446250+0000 mon.c (mon.2) 85 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:22:00.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.447123+0000 mon.c (mon.2) 86 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:00.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.447123+0000 mon.c (mon.2) 86 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:22:00.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.453135+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:00.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:00 smithi190 bash[23868]: audit 2024-04-26T16:22:00.453135+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:00.906 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:00.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:00.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:01.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:01.032 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:01.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:01.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:01.162 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:01.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:01.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:01.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:01.425 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:01.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:01.556 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:01.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:01.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:01.728 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:22:01.728 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:22:01.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:01.835 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:00.085613+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:22:01.835 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:00.085613+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:22:01.835 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:00.087561+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:22:01.835 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:00.087561+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:22:01.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.348044+0000 osd.5 (osd.5) 115 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:22:01.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.348044+0000 osd.5 (osd.5) 115 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:22:01.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.352682+0000 osd.5 (osd.5) 116 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:22:01.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:01 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.352682+0000 osd.5 (osd.5) 116 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:22:01.863 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:00.085613+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:22:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:00.085613+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:22:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:00.087561+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:22:01.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:00.087561+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:22:01.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.348044+0000 osd.5 (osd.5) 115 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:22:01.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.348044+0000 osd.5 (osd.5) 115 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:22:01.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.352682+0000 osd.5 (osd.5) 116 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:22:01.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.352682+0000 osd.5 (osd.5) 116 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:22:01.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:00.085613+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:00.085613+0000 osd.1 (osd.1) 117 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:00.087561+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:00.087561+0000 osd.1 (osd.1) 118 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.348044+0000 osd.5 (osd.5) 115 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.348044+0000 osd.5 (osd.5) 115 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.352682+0000 osd.5 (osd.5) 116 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:22:01.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:01 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.352682+0000 osd.5 (osd.5) 116 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:22:01.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:01.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:01.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:02.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:02.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:02.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:02.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:02.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:02.386 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:02.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:02.517 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:02.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:02.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:02.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/747217416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc281094e0 msgr2=0x7fcc281098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/747217416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc281094e0 0x7fcc281098c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcc18009f90 tx=0x7fcc1800b330 comp rx=0 tx=0).stop 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/747217416 shutdown_connections 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/747217416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc28103e60 0x7fcc281102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/747217416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc281094e0 0x7fcc281098c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/747217416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc281034a0 0x7fcc28103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/747217416 >> 172.21.15.50:0/747217416 conn(0x7fcc2807a590 msgr2=0x7fcc2807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/747217416 shutdown_connections 2024-04-26T16:22:02.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/747217416 wait complete. 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 librados: starting msgr at 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 librados: starting objecter 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 Processor -- start 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- start start 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 librados: setting wanted keys 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 librados: calling monclient init 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 0x7fcc28199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc28103e60 0x7fcc281996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:02.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc281094e0 0x7fcc2819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28112ac0 con 0x7fcc281034a0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc28112940 con 0x7fcc281094e0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fcc28112c40 con 0x7fcc28103e60 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 0x7fcc28199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 0x7fcc28199180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39854/0 (socket says 172.21.15.50:39854) 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 -- 172.21.15.50:0/3013016491 learned_addr learned my addr 172.21.15.50:0/3013016491 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc27fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc28103e60 0x7fcc281996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2d483700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc281094e0 0x7fcc2819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 -- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc28103e60 msgr2=0x7fcc281996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc28103e60 0x7fcc281996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 -- 172.21.15.50:0/3013016491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc281094e0 msgr2=0x7fcc2819da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc281094e0 0x7fcc2819da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 -- 172.21.15.50:0/3013016491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2819e190 con 0x7fcc281034a0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2cc82700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 0x7fcc28199180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800d010 tx=0x7fcc18004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:02.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc25ffb700 10 client.?.objecter ms_handle_connect 0x7fcc281034a0 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc25ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18021070 con 0x7fcc281034a0 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2819e420 con 0x7fcc281034a0 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fcc18007910 con 0x7fcc281034a0 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18005040 con 0x7fcc281034a0 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 10 client.14739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc281a5d00 con 0x7fcc281034a0 2024-04-26T16:22:02.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.720+0000 7fcc2eee6700 1 librados: init done 2024-04-26T16:22:02.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc2806a110 con 0x7fcc281034a0 2024-04-26T16:22:02.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fcc18024070 con 0x7fcc281034a0 2024-04-26T16:22:02.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc10071f80 0x7fcc10074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:02.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(127..127 src has 77..127) ==== 7450+0+0 (secure 0 0 0) 0x7fcc1800f020 con 0x7fcc281034a0 2024-04-26T16:22:02.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 10 client.14739.objecter ms_dispatch 0x7fcc28001550 osd_map(127..127 src has 77..127) 2024-04-26T16:22:02.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 3 client.14739.objecter handle_osd_map got epochs [127,127] > 0 2024-04-26T16:22:02.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 3 client.14739.objecter handle_osd_map decoding full epoch 127 2024-04-26T16:22:02.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc27fff700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc10071f80 0x7fcc10074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:02.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 20 client.14739.objecter dump_active .. 0 homeless 2024-04-26T16:22:02.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc25ffb700 10 client.14739.objecter ms_handle_connect 0x7fcc10071f80 2024-04-26T16:22:02.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.724+0000 7fcc27fff700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc10071f80 0x7fcc10074440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc2807a9b0 tx=0x7fcc1c00a300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:02.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.728+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fcc1802b030 con 0x7fcc281034a0 2024-04-26T16:22:02.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:02.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.163619+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:22:02.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.163619+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:22:02.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.165702+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:22:02.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.165702+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:22:02.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.437805+0000 mgr.y (mgr.24401) 169 : cluster [DBG] pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:02.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.437805+0000 mgr.y (mgr.24401) 169 : cluster [DBG] pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:02.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.603268+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:22:02.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.603268+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:22:02.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.604601+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:22:02.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:01.604601+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:22:02.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.350893+0000 osd.5 (osd.5) 117 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:22:02.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.350893+0000 osd.5 (osd.5) 117 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:22:02.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.352992+0000 osd.5 (osd.5) 118 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:22:02.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:02 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.352992+0000 osd.5 (osd.5) 118 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:22:02.770 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:02.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.163619+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:22:02.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.163619+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:22:02.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.165702+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.165702+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.437805+0000 mgr.y (mgr.24401) 169 : cluster [DBG] pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.437805+0000 mgr.y (mgr.24401) 169 : cluster [DBG] pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.603268+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.603268+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.604601+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:01.604601+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.350893+0000 osd.5 (osd.5) 117 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.350893+0000 osd.5 (osd.5) 117 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.352992+0000 osd.5 (osd.5) 118 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:22:02.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.352992+0000 osd.5 (osd.5) 118 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.163619+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.163619+0000 osd.0 (osd.0) 113 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.165702+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.165702+0000 osd.0 (osd.0) 114 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.437805+0000 mgr.y (mgr.24401) 169 : cluster [DBG] pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.437805+0000 mgr.y (mgr.24401) 169 : cluster [DBG] pgmap v138: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.603268+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.603268+0000 osd.4 (osd.4) 99 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.604601+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:01.604601+0000 osd.4 (osd.4) 100 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.350893+0000 osd.5 (osd.5) 117 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:22:02.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.350893+0000 osd.5 (osd.5) 117 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:22:02.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.352992+0000 osd.5 (osd.5) 118 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:22:02.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:02 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.352992+0000 osd.5 (osd.5) 118 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:22:02.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:02.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:02.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:02.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:02.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:02.972+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fcc281a5ed0 con 0x7fcc281034a0 2024-04-26T16:22:02.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:03.022 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- 172.21.15.50:0/4087102592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb13406bad0 msgr2=0x7fb1341102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- 172.21.15.50:0/4087102592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb13406bad0 0x7fb1341102e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb12c00b0a0 tx=0x7fb12c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- 172.21.15.50:0/4087102592 shutdown_connections 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- 172.21.15.50:0/4087102592 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb13406bad0 0x7fb1341102e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- 172.21.15.50:0/4087102592 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb1341094a0 0x7fb134109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- 172.21.15.50:0/4087102592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13406b130 0x7fb13406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- 172.21.15.50:0/4087102592 >> 172.21.15.50:0/4087102592 conn(0x7fb1340792d0 msgr2=0x7fb1340796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- 172.21.15.50:0/4087102592 shutdown_connections 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- 172.21.15.50:0/4087102592 wait complete. 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 librados: starting msgr at 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 librados: starting objecter 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 Processor -- start 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- start start 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 librados: setting wanted keys 2024-04-26T16:22:03.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 librados: calling monclient init 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13406b130 0x7fb1341023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb13406bad0 0x7fb134102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1341094a0 0x7fb134102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb134112b20 con 0x7fb1341094a0 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb1341129a0 con 0x7fb13406b130 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13bd95700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb134112ca0 con 0x7fb13406bad0 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb13406bad0 0x7fb134102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb13406bad0 0x7fb134102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51804/0 (socket says 172.21.15.50:51804) 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13a332700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1341094a0 0x7fb134102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 -- 172.21.15.50:0/3303335807 learned_addr learned my addr 172.21.15.50:0/3303335807 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:03.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139b31700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13406b130 0x7fb1341023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 -- 172.21.15.50:0/3303335807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13406b130 msgr2=0x7fb1341023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb13406b130 0x7fb1341023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 -- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1341094a0 msgr2=0x7fb134102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1341094a0 0x7fb134102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 -- 172.21.15.50:0/3303335807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1341abb90 con 0x7fb13406bad0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb13a332700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb1341094a0 0x7fb134102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb139330700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb13406bad0 0x7fb134102930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb12400d950 tx=0x7fb12400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.036+0000 7fb12affd700 10 client.?.objecter ms_handle_connect 0x7fb13406bad0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124016070 con 0x7fb13406bad0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb12400fcf0 con 0x7fb13406bad0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1341abe20 con 0x7fb13406bad0 2024-04-26T16:22:03.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 10 client.24635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb124010df0 con 0x7fb13406bad0 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1341ac280 con 0x7fb13406bad0 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 1 librados: init done 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 10 client.24635.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 15 client.24635.objecter _session_command_op_assign -1 1 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 20 client.24635.objecter _get_session osd=-1 returning homeless 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 10 client.24635.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:03.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb134104b50 con 0x7fb13406bad0 2024-04-26T16:22:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb124019070 con 0x7fb13406bad0 2024-04-26T16:22:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb120071f80 0x7fb120074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:03.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(127..127 src has 77..127) ==== 7450+0+0 (secure 0 0 0) 0x7fb1240910d0 con 0x7fb13406bad0 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 10 client.24635.objecter ms_dispatch 0x7fb134001550 osd_map(127..127 src has 77..127) 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 3 client.24635.objecter handle_osd_map got epochs [127,127] > 0 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 3 client.24635.objecter handle_osd_map decoding full epoch 127 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb139b31700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb120071f80 0x7fb120074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 10 client.24635.objecter checking command 1 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb12007be30 0x7fb12007e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 20 client.24635.objecter _get_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 15 client.24635.objecter _session_command_op_remove -1 1 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 20 client.24635.objecter _get_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 20 client.24635.objecter get_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 15 client.24635.objecter _session_command_op_assign 0 1 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 4 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 10 client.24635.objecter _send_command 1 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb124004500 con 0x7fb12007be30 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 20 client.24635.objecter dump_active .. 0 homeless 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=127) ==== 24+0+0 (secure 0 0 0) 0x7fb124091560 con 0x7fb13406bad0 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13a332700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb12007be30 0x7fb12007e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 10 client.24635.objecter ms_handle_connect 0x7fb120071f80 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb139b31700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb120071f80 0x7fb120074440 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb1300045b0 tx=0x7fb13000b3c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb12affd700 10 client.24635.objecter ms_handle_connect 0x7fb12007be30 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.040+0000 7fb13a332700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb12007be30 0x7fb12007e290 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.044+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb124004500 con 0x7fb12007be30 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.044+0000 7fb12affd700 10 client.24635.objecter ms_dispatch 0x7fb134001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.044+0000 7fb12affd700 10 client.24635.objecter _finish_command 1 = system:0 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.044+0000 7fb12affd700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 4 2024-04-26T16:22:03.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.044+0000 7fb12affd700 15 client.24635.objecter _session_command_op_remove 0 1 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 10 client.24635.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 15 client.24635.objecter _session_command_op_assign -1 2 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter _get_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter _get_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 15 client.24635.objecter _session_command_op_remove -1 2 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter get_session s=0x7fb12007b760 osd=0 3 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 15 client.24635.objecter _session_command_op_assign 0 2 2024-04-26T16:22:03.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 4 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 10 client.24635.objecter _send_command 2 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fb13406a110 con 0x7fb12007be30 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb12affd700 1 -- 172.21.15.50:0/3303335807 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fb13406a110 con 0x7fb12007be30 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb12affd700 10 client.24635.objecter ms_dispatch 0x7fb134001550 command_reply(tid 2: 0 ) 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb12affd700 10 client.24635.objecter _finish_command 2 = system:0 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb12affd700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 4 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb12affd700 15 client.24635.objecter _session_command_op_remove 0 2 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 10 librados: watch_flush enter 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 10 librados: watch_flush exit 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 10 client.24635.objecter close_session for osd.0 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb12007be30 msgr2=0x7fb12007e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb12007be30 0x7fb12007e290 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter put_session s=0x7fb12007b760 osd=0 1 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 20 client.24635.objecter shutdown clearing up homeless session... 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 10 client.24635.objecter successfully canceled tick 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb120071f80 msgr2=0x7fb120074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb120071f80 0x7fb120074440 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb1300045b0 tx=0x7fb13000b3c0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb13406bad0 msgr2=0x7fb134102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 --2- 172.21.15.50:0/3303335807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb13406bad0 0x7fb134102930 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb12400d950 tx=0x7fb12400dc60 comp rx=0 tx=0).stop 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13a332700 1 -- 172.21.15.50:0/3303335807 reap_dead start 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 shutdown_connections 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 >> 172.21.15.50:0/3303335807 conn(0x7fb1340792d0 msgr2=0x7fb134100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 shutdown_connections 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 -- 172.21.15.50:0/3303335807 wait complete. 2024-04-26T16:22:03.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.064+0000 7fb13bd95700 1 librados: shutdown 2024-04-26T16:22:03.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:03.153 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:03.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:03.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:03.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:03.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:03.405 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:03.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:03.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.595+0000 7fcc25ffb700 1 -- 172.21.15.50:0/3013016491 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v128) ==== 87+0+0 (secure 0 0 0) 0x7fcc1805e7d0 con 0x7fcc281034a0 2024-04-26T16:22:03.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.599+0000 7fcc2eee6700 10 librados: watch_flush enter 2024-04-26T16:22:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 10 librados: watch_flush exit 2024-04-26T16:22:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 20 client.14739.objecter shutdown clearing up homeless session... 2024-04-26T16:22:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 10 client.14739.objecter successfully canceled tick 2024-04-26T16:22:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc10071f80 msgr2=0x7fcc10074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc10071f80 0x7fcc10074440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fcc2807a9b0 tx=0x7fcc1c00a300 comp rx=0 tx=0).stop 2024-04-26T16:22:03.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 msgr2=0x7fcc28199180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 0x7fcc28199180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800d010 tx=0x7fcc18004d00 comp rx=0 tx=0).stop 2024-04-26T16:22:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 shutdown_connections 2024-04-26T16:22:03.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc10071f80 0x7fcc10074440 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc281094e0 0x7fcc2819da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc281034a0 0x7fcc28199180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 --2- 172.21.15.50:0/3013016491 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc28103e60 0x7fcc281996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:03.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.603+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 >> 172.21.15.50:0/3013016491 conn(0x7fcc2807a590 msgr2=0x7fcc28101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:03.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.607+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 shutdown_connections 2024-04-26T16:22:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.607+0000 7fcc2eee6700 1 -- 172.21.15.50:0/3013016491 wait complete. 2024-04-26T16:22:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:03.607+0000 7fcc2eee6700 1 librados: shutdown 2024-04-26T16:22:03.603 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:22:03.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:03.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.129202+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.129202+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.131463+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.131463+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.217786+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.217786+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.220426+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:02.220426+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: audit 2024-04-26T16:22:02.975904+0000 mon.a (mon.0) 942 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: audit 2024-04-26T16:22:02.975904+0000 mon.a (mon.0) 942 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.328404+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.328404+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.330953+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:22:03.652 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:03 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.330953+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:22:03.669 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:03.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:03.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:03.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.129202+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.129202+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.131463+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.131463+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.217786+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.217786+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.220426+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:02.220426+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: audit 2024-04-26T16:22:02.975904+0000 mon.a (mon.0) 942 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: audit 2024-04-26T16:22:02.975904+0000 mon.a (mon.0) 942 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.328404+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.328404+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.330953+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:22:03.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.330953+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.129202+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.129202+0000 osd.7 (osd.7) 133 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.131463+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.131463+0000 osd.7 (osd.7) 134 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.217786+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.217786+0000 osd.3 (osd.3) 123 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.220426+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:02.220426+0000 osd.3 (osd.3) 124 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: audit 2024-04-26T16:22:02.975904+0000 mon.a (mon.0) 942 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: audit 2024-04-26T16:22:02.975904+0000 mon.a (mon.0) 942 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:22:03.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.328404+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:22:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.328404+0000 osd.5 (osd.5) 119 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:22:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.330953+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:22:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:03 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.330953+0000 osd.5 (osd.5) 120 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:22:03.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:03.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:03.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:04.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:04.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:04.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:04.193 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:04.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:04.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:04.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:04.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-26T16:22:04.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:04.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:04.480 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:22:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:22:04.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:04.576 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:04.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:04.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:04.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:04.836 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.438517+0000 mgr.y (mgr.24401) 170 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.438517+0000 mgr.y (mgr.24401) 170 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.590346+0000 mon.a (mon.0) 943 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.590346+0000 mon.a (mon.0) 943 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: audit 2024-04-26T16:22:03.600666+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: audit 2024-04-26T16:22:03.600666+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.608593+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[21463]: cluster 2024-04-26T16:22:03.608593+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-04-26T16:22:04.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.438517+0000 mgr.y (mgr.24401) 170 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.438517+0000 mgr.y (mgr.24401) 170 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.590346+0000 mon.a (mon.0) 943 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.590346+0000 mon.a (mon.0) 943 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: audit 2024-04-26T16:22:03.600666+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: audit 2024-04-26T16:22:03.600666+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.608593+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-04-26T16:22:04.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:04 smithi050 bash[29234]: cluster 2024-04-26T16:22:03.608593+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-04-26T16:22:04.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.438517+0000 mgr.y (mgr.24401) 170 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.438517+0000 mgr.y (mgr.24401) 170 : cluster [DBG] pgmap v139: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.590346+0000 mon.a (mon.0) 943 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.590346+0000 mon.a (mon.0) 943 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: audit 2024-04-26T16:22:03.600666+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: audit 2024-04-26T16:22:03.600666+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.50:0/3013016491' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.608593+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-04-26T16:22:04.951 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:04 smithi190 bash[23868]: cluster 2024-04-26T16:22:03.608593+0000 mon.a (mon.0) 945 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2024-04-26T16:22:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:04.967 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:04.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:05.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:22:05.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:05.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:05.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:05.224 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:05.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:05.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:05.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:05.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:05.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:05.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:05.613 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:05.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-04-26T16:22:05.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:05.744 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:05.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:05.876 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:05.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:05.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:05.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:06.006 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:06.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:06.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:06.139 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:06.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:06.269 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:06.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:06.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:06.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:06.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:06.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:06.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:06 smithi050 bash[21463]: cluster 2024-04-26T16:22:05.439282+0000 mgr.y (mgr.24401) 171 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:06.644 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:06 smithi050 bash[21463]: cluster 2024-04-26T16:22:05.439282+0000 mgr.y (mgr.24401) 171 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:06.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:06 smithi050 bash[29234]: cluster 2024-04-26T16:22:05.439282+0000 mgr.y (mgr.24401) 171 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:06.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:06 smithi050 bash[29234]: cluster 2024-04-26T16:22:05.439282+0000 mgr.y (mgr.24401) 171 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:06.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:06.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:06.773 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:06 smithi190 bash[23868]: cluster 2024-04-26T16:22:05.439282+0000 mgr.y (mgr.24401) 171 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:06.773 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:06 smithi190 bash[23868]: cluster 2024-04-26T16:22:05.439282+0000 mgr.y (mgr.24401) 171 : cluster [DBG] pgmap v141: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:06.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:22:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:22:06.789 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:06.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:06.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:06.922 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:06.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:07.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:07.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:07.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:07.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:07.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:07.316 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:07.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:07.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:07.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:07.578 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:07.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:22:07.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:07.709 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:07.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:07 smithi190 bash[23868]: audit 2024-04-26T16:22:06.524675+0000 mgr.y (mgr.24401) 172 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:07 smithi190 bash[23868]: audit 2024-04-26T16:22:06.524675+0000 mgr.y (mgr.24401) 172 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:07.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:07 smithi050 bash[29234]: audit 2024-04-26T16:22:06.524675+0000 mgr.y (mgr.24401) 172 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:07.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:07 smithi050 bash[29234]: audit 2024-04-26T16:22:06.524675+0000 mgr.y (mgr.24401) 172 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:07.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:07 smithi050 bash[21463]: audit 2024-04-26T16:22:06.524675+0000 mgr.y (mgr.24401) 172 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:07.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:07 smithi050 bash[21463]: audit 2024-04-26T16:22:06.524675+0000 mgr.y (mgr.24401) 172 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:07.837 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:07.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:07.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:07.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:08.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:08.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:08.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:08.224 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:08.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:08.354 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:08.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:08.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:08.614 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:08.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:08.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:08.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:08.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:08.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:08 smithi050 bash[21463]: cluster 2024-04-26T16:22:07.440723+0000 mgr.y (mgr.24401) 173 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:08.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:08 smithi050 bash[21463]: cluster 2024-04-26T16:22:07.440723+0000 mgr.y (mgr.24401) 173 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:08.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:08 smithi050 bash[29234]: cluster 2024-04-26T16:22:07.440723+0000 mgr.y (mgr.24401) 173 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:08.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:08 smithi050 bash[29234]: cluster 2024-04-26T16:22:07.440723+0000 mgr.y (mgr.24401) 173 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:08.879 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:08.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:08.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:09.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:08 smithi190 bash[23868]: cluster 2024-04-26T16:22:07.440723+0000 mgr.y (mgr.24401) 173 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:09.001 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:08 smithi190 bash[23868]: cluster 2024-04-26T16:22:07.440723+0000 mgr.y (mgr.24401) 173 : cluster [DBG] pgmap v142: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:09.010 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:09.105 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:09.107 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:09.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:09.141 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:09.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:09.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:09.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:09.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:09.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:09.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:09.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:09.664 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:09.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:09.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:09.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:09.923 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:09.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:09.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:10.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:10.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:10.120 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:10.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:10.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:10.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:10.316 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:10.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:10.446 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:10.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:10.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:10.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:10.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:10.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:10.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:10 smithi190 bash[23868]: cluster 2024-04-26T16:22:09.441382+0000 mgr.y (mgr.24401) 174 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:10.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:10 smithi190 bash[23868]: cluster 2024-04-26T16:22:09.441382+0000 mgr.y (mgr.24401) 174 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:10 smithi050 bash[29234]: cluster 2024-04-26T16:22:09.441382+0000 mgr.y (mgr.24401) 174 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:10.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:10 smithi050 bash[29234]: cluster 2024-04-26T16:22:09.441382+0000 mgr.y (mgr.24401) 174 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:10.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:10 smithi050 bash[21463]: cluster 2024-04-26T16:22:09.441382+0000 mgr.y (mgr.24401) 174 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:10.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:10 smithi050 bash[21463]: cluster 2024-04-26T16:22:09.441382+0000 mgr.y (mgr.24401) 174 : cluster [DBG] pgmap v143: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:10.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:10.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:10.976 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:10.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:10.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:11.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:11.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:11.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:11.369 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:11.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:11.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:11.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:11.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:11.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:11.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:11.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:11.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[21463]: audit 2024-04-26T16:22:10.875861+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:11.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[21463]: audit 2024-04-26T16:22:10.875861+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:11.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[21463]: audit 2024-04-26T16:22:10.877190+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:11.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[21463]: audit 2024-04-26T16:22:10.877190+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:11.921 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[21463]: cluster 2024-04-26T16:22:11.442719+0000 mgr.y (mgr.24401) 175 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[21463]: cluster 2024-04-26T16:22:11.442719+0000 mgr.y (mgr.24401) 175 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[29234]: audit 2024-04-26T16:22:10.875861+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[29234]: audit 2024-04-26T16:22:10.875861+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[29234]: audit 2024-04-26T16:22:10.877190+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[29234]: audit 2024-04-26T16:22:10.877190+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[29234]: cluster 2024-04-26T16:22:11.442719+0000 mgr.y (mgr.24401) 175 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:11.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:11 smithi050 bash[29234]: cluster 2024-04-26T16:22:11.442719+0000 mgr.y (mgr.24401) 175 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:11.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:12.018 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:12.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:12.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:11 smithi190 bash[23868]: audit 2024-04-26T16:22:10.875861+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:12.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:11 smithi190 bash[23868]: audit 2024-04-26T16:22:10.875861+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:22:12.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:11 smithi190 bash[23868]: audit 2024-04-26T16:22:10.877190+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:12.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:11 smithi190 bash[23868]: audit 2024-04-26T16:22:10.877190+0000 mon.c (mon.2) 87 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:12.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:11 smithi190 bash[23868]: cluster 2024-04-26T16:22:11.442719+0000 mgr.y (mgr.24401) 175 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:12.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:11 smithi190 bash[23868]: cluster 2024-04-26T16:22:11.442719+0000 mgr.y (mgr.24401) 175 : cluster [DBG] pgmap v144: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:12.146 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:12.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:12.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:12.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:12.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:12.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:12.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:12.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:12.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:12.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:12.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:12.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:12.672 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:12.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:12.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:12.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:12.936 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:12.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:13.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:13.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:13.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:13.194 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:13.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:13.324 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:13.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:13.454 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:13.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:13.584 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:13.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:13.714 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:13.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:13.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:13.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:13.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:13.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:14.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:14.104 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:14.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:14.236 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:14.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:14.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:14.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:14.483 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:22:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:22:14.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:14.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:14.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:14.764 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:14 smithi190 bash[23868]: cluster 2024-04-26T16:22:13.443491+0000 mgr.y (mgr.24401) 176 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:14 smithi190 bash[23868]: cluster 2024-04-26T16:22:13.443491+0000 mgr.y (mgr.24401) 176 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:14 smithi190 bash[23868]: cluster 2024-04-26T16:22:14.101018+0000 osd.3 (osd.3) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:14 smithi190 bash[23868]: cluster 2024-04-26T16:22:14.101018+0000 osd.3 (osd.3) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:14 smithi190 bash[23868]: cluster 2024-04-26T16:22:14.101171+0000 osd.3 (osd.3) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:14.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:14 smithi190 bash[23868]: cluster 2024-04-26T16:22:14.101171+0000 osd.3 (osd.3) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:14.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[29234]: cluster 2024-04-26T16:22:13.443491+0000 mgr.y (mgr.24401) 176 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:14.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[29234]: cluster 2024-04-26T16:22:13.443491+0000 mgr.y (mgr.24401) 176 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:14.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[29234]: cluster 2024-04-26T16:22:14.101018+0000 osd.3 (osd.3) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:14.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[29234]: cluster 2024-04-26T16:22:14.101018+0000 osd.3 (osd.3) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:14.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[29234]: cluster 2024-04-26T16:22:14.101171+0000 osd.3 (osd.3) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:14.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[29234]: cluster 2024-04-26T16:22:14.101171+0000 osd.3 (osd.3) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:14.771 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21463]: cluster 2024-04-26T16:22:13.443491+0000 mgr.y (mgr.24401) 176 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:14.771 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21463]: cluster 2024-04-26T16:22:13.443491+0000 mgr.y (mgr.24401) 176 : cluster [DBG] pgmap v145: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:14.771 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21463]: cluster 2024-04-26T16:22:14.101018+0000 osd.3 (osd.3) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:14.771 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21463]: cluster 2024-04-26T16:22:14.101018+0000 osd.3 (osd.3) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:14.771 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21463]: cluster 2024-04-26T16:22:14.101171+0000 osd.3 (osd.3) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:14.771 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:14 smithi050 bash[21463]: cluster 2024-04-26T16:22:14.101171+0000 osd.3 (osd.3) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:14.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:14.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:14.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:14.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:15.023 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:15.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:15.154 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:15.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:15.285 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:15.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:15.414 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:15.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:15.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:15 smithi190 bash[23868]: cluster 2024-04-26T16:22:14.504253+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-04-26T16:22:15.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:15 smithi190 bash[23868]: cluster 2024-04-26T16:22:14.504253+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-04-26T16:22:15.545 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:15.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:15.677 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:15.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:15.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:15 smithi050 bash[29234]: cluster 2024-04-26T16:22:14.504253+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-04-26T16:22:15.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:15 smithi050 bash[29234]: cluster 2024-04-26T16:22:14.504253+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-04-26T16:22:15.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:15 smithi050 bash[21463]: cluster 2024-04-26T16:22:14.504253+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-04-26T16:22:15.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:15 smithi050 bash[21463]: cluster 2024-04-26T16:22:14.504253+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2024-04-26T16:22:15.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:15.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:15.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:15.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:16.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:16.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:16.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:16.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:16.200 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:16.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:16.330 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:16.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:16.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:16.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:16.593 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:16.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:16.724 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:16.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:16 smithi190 bash[23868]: cluster 2024-04-26T16:22:15.444149+0000 mgr.y (mgr.24401) 177 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:16.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:16 smithi190 bash[23868]: cluster 2024-04-26T16:22:15.444149+0000 mgr.y (mgr.24401) 177 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:16.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:22:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:22:16.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:16.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:16 smithi050 bash[29234]: cluster 2024-04-26T16:22:15.444149+0000 mgr.y (mgr.24401) 177 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:16.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:16 smithi050 bash[29234]: cluster 2024-04-26T16:22:15.444149+0000 mgr.y (mgr.24401) 177 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:16.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:16 smithi050 bash[21463]: cluster 2024-04-26T16:22:15.444149+0000 mgr.y (mgr.24401) 177 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:16.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:16 smithi050 bash[21463]: cluster 2024-04-26T16:22:15.444149+0000 mgr.y (mgr.24401) 177 : cluster [DBG] pgmap v147: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:16.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:16.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:16.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:16.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:17.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:17.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:17.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:17.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:17.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:17.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:17.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:17.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:17.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:17 smithi050 bash[29234]: audit 2024-04-26T16:22:16.534511+0000 mgr.y (mgr.24401) 178 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:17.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:17 smithi050 bash[29234]: audit 2024-04-26T16:22:16.534511+0000 mgr.y (mgr.24401) 178 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:17.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:17 smithi050 bash[21463]: audit 2024-04-26T16:22:16.534511+0000 mgr.y (mgr.24401) 178 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:17.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:17 smithi050 bash[21463]: audit 2024-04-26T16:22:16.534511+0000 mgr.y (mgr.24401) 178 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:17.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:17.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:17.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:17 smithi190 bash[23868]: audit 2024-04-26T16:22:16.534511+0000 mgr.y (mgr.24401) 178 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:17.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:17 smithi190 bash[23868]: audit 2024-04-26T16:22:16.534511+0000 mgr.y (mgr.24401) 178 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:17.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:17.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:17.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:17.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:18.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:18.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:18.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:18.161 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:18.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:18.289 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:18.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:18.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:18.420 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:18.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:18.549 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:18.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:18.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:18 smithi190 bash[23868]: cluster 2024-04-26T16:22:17.445211+0000 mgr.y (mgr.24401) 179 : cluster [DBG] pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:18.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:18 smithi190 bash[23868]: cluster 2024-04-26T16:22:17.445211+0000 mgr.y (mgr.24401) 179 : cluster [DBG] pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:18.678 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:18.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:18.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:18 smithi050 bash[29234]: cluster 2024-04-26T16:22:17.445211+0000 mgr.y (mgr.24401) 179 : cluster [DBG] pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:18.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:18 smithi050 bash[29234]: cluster 2024-04-26T16:22:17.445211+0000 mgr.y (mgr.24401) 179 : cluster [DBG] pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:18.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:18 smithi050 bash[21463]: cluster 2024-04-26T16:22:17.445211+0000 mgr.y (mgr.24401) 179 : cluster [DBG] pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:18.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:18 smithi050 bash[21463]: cluster 2024-04-26T16:22:17.445211+0000 mgr.y (mgr.24401) 179 : cluster [DBG] pgmap v148: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:18.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:18.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:18.940 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:18.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:19.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:19.069 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:19.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:19.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:19.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:19.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:19.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:19.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:19.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:19.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:19.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:19.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:19.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- 172.21.15.50:0/2058854835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc103e90 msgr2=0x7f39dc110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- 172.21.15.50:0/2058854835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc103e90 0x7f39dc110300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f39d4009f90 tx=0x7f39d400b330 comp rx=0 tx=0).stop 2024-04-26T16:22:19.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- 172.21.15.50:0/2058854835 shutdown_connections 2024-04-26T16:22:19.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- 172.21.15.50:0/2058854835 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39dc1083b0 0x7f39dc108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- 172.21.15.50:0/2058854835 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc103e90 0x7f39dc110300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- 172.21.15.50:0/2058854835 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc1034d0 0x7f39dc103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- 172.21.15.50:0/2058854835 >> 172.21.15.50:0/2058854835 conn(0x7f39dc07a580 msgr2=0x7f39dc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:19.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- 172.21.15.50:0/2058854835 shutdown_connections 2024-04-26T16:22:19.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- 172.21.15.50:0/2058854835 wait complete. 2024-04-26T16:22:19.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 librados: starting msgr at 2024-04-26T16:22:19.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 librados: starting objecter 2024-04-26T16:22:19.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 Processor -- start 2024-04-26T16:22:19.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- start start 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 librados: setting wanted keys 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 librados: calling monclient init 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39dc1034d0 0x7f39dc199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc103e90 0x7f39dc199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 0x7f39dc19da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc112b00 con 0x7f39dc103e90 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f39dc112980 con 0x7f39dc1034d0 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39e2054700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f39dc112c80 con 0x7f39dc1083b0 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39dbfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 0x7f39dc19da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc103e90 0x7f39dc199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39dbfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 0x7f39dc19da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:58788/0 (socket says 172.21.15.50:58788) 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39daffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc103e90 0x7f39dc199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39764/0 (socket says 172.21.15.50:39764) 2024-04-26T16:22:19.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39dbfff700 1 -- 172.21.15.50:0/948685979 learned_addr learned my addr 172.21.15.50:0/948685979 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.803+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39dc1034d0 0x7f39dc199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39dbfff700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39dc1034d0 msgr2=0x7f39dc199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f39dc1034d0 0x7f39dc199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39dbfff700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc103e90 msgr2=0x7f39dc199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc103e90 0x7f39dc199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39dbfff700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39dc19e150 con 0x7f39dc1083b0 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f39dc103e90 0x7f39dc199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 0x7f39dc19da10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f39d400d010 tx=0x7f39d4007f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 10 client.?.objecter ms_handle_connect 0x7f39dc1083b0 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d4021070 con 0x7f39dc1083b0 2024-04-26T16:22:19.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f39d4004e40 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39dc19e3e0 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d4005040 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 10 client.24641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39dc1a5cc0 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 1 librados: init done 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f39dc06a110 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f39d4024070 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f39c4071f80 0x7f39c4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(129..129 src has 77..129) ==== 7450+0+0 (secure 0 0 0) 0x7f39d400f020 con 0x7f39dc1083b0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 osd_map(129..129 src has 77..129) 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 3 client.24641.objecter handle_osd_map got epochs [129,129] > 0 2024-04-26T16:22:19.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 3 client.24641.objecter handle_osd_map decoding full epoch 129 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f39c4071f80 0x7f39c4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 20 client.24641.objecter dump_active .. 0 homeless 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v129) ==== 54+0+16 (secure 0 0 0) 0x7f39d402b020 con 0x7f39dc1083b0 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 10 client.24641.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 1 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f39dc064030 0x7f39dc064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 1 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 0 1 2024-04-26T16:22:19.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 4 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 10 client.24641.objecter _send_command 1 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1a9020 con 0x7f39dc064030 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39c4071f80 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f39c4071f80 0x7f39c4074440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39cc005fd0 tx=0x7f39cc005d90 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.807+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f39dc064030 0x7f39dc064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc064030 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f39dc064030 0x7f39dc064410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1a9020 con 0x7f39dc064030 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:19.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 1 = system:0 2024-04-26T16:22:19.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 4 2024-04-26T16:22:19.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.811+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 0 1 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 10 client.24641.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 2 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 2 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc19a4a0 osd=0 3 2024-04-26T16:22:19.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 0 2 2024-04-26T16:22:19.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 4 2024-04-26T16:22:19.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 10 client.24641.objecter _send_command 2 2024-04-26T16:22:19.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.827+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc063ab0 con 0x7f39dc064030 2024-04-26T16:22:19.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:19.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc063ab0 con 0x7f39dc064030 2024-04-26T16:22:19.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 4 2024-04-26T16:22:19.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 0 2 2024-04-26T16:22:19.827 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 10 client.24641.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f39dc02d310 0x7f39dc02d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 1 3 2024-04-26T16:22:19.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 4 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 10 client.24641.objecter _send_command 3 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f39dc10b120 con 0x7f39dc02d310 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f39dc02d310 0x7f39dc02d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc02d310 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f39dc02d310 0x7f39dc02d6f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc10b120 con 0x7f39dc02d310 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 3: 0 ) 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 3 = system:0 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 4 2024-04-26T16:22:19.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.831+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 1 3 2024-04-26T16:22:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:19.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 10 client.24641.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 4 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 4 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc10aaa0 osd=1 3 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 1 4 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 4 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 10 client.24641.objecter _send_command 4 2024-04-26T16:22:19.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc10a520 con 0x7f39dc02d310 2024-04-26T16:22:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc10a520 con 0x7f39dc02d310 2024-04-26T16:22:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 4 2024-04-26T16:22:19.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 1 4 2024-04-26T16:22:19.848 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 10 client.24641.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 5 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f39dc1ab1e0 0x7f39dc1ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 5 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 2 5 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 4 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 10 client.24641.objecter _send_command 5 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1b0430 con 0x7f39dc1ab1e0 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f39dc1ab1e0 0x7f39dc1ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc1ab1e0 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.851+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f39dc1ab1e0 0x7f39dc1ad5d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.855+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1b0430 con 0x7f39dc1ab1e0 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.855+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 5: 0 ) 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.855+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 5 = system:0 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.855+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 4 2024-04-26T16:22:19.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.855+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 2 5 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 10 client.24641.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 6 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 6 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc02e400 osd=2 3 2024-04-26T16:22:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 2 6 2024-04-26T16:22:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 4 2024-04-26T16:22:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 10 client.24641.objecter _send_command 6 2024-04-26T16:22:19.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.871+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc02de80 con 0x7f39dc1ab1e0 2024-04-26T16:22:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc02de80 con 0x7f39dc1ab1e0 2024-04-26T16:22:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 4 2024-04-26T16:22:19.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 2 6 2024-04-26T16:22:19.869 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 10 client.24641.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 7 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f39dc1aec90 0x7f39dc1af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 7 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 3 7 2024-04-26T16:22:19.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 4 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 10 client.24641.objecter _send_command 7 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1a8cf0 con 0x7f39dc1aec90 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f39dc1aec90 0x7f39dc1af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc1aec90 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f39dc1aec90 0x7f39dc1af0f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1a8cf0 con 0x7f39dc1aec90 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 7: 0 ) 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 7 = system:0 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 4 2024-04-26T16:22:19.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.875+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 3 7 2024-04-26T16:22:19.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 10 client.24641.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 8 2024-04-26T16:22:19.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 8 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1a8670 osd=3 3 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 3 8 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 4 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 10 client.24641.objecter _send_command 8 2024-04-26T16:22:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.891+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc1a8cf0 con 0x7f39dc1aec90 2024-04-26T16:22:19.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc1a8cf0 con 0x7f39dc1aec90 2024-04-26T16:22:19.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 4 2024-04-26T16:22:19.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 3 8 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 10 client.24641.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 9 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f39dc1afe00 0x7f39dc1b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 9 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 4 9 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 4 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 10 client.24641.objecter _send_command 9 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1b0260 con 0x7f39dc1afe00 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f39dc1afe00 0x7f39dc1b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f39dc1afe00 0x7f39dc1b4600 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.895+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc1afe00 2024-04-26T16:22:19.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.899+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1b0260 con 0x7f39dc1afe00 2024-04-26T16:22:19.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.899+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 9: 0 ) 2024-04-26T16:22:19.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.899+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 9 = system:0 2024-04-26T16:22:19.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.899+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 4 2024-04-26T16:22:19.894 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.899+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 4 9 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 10 client.24641.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 10 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 10 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1a8cf0 osd=4 3 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 4 10 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 4 2024-04-26T16:22:19.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 10 client.24641.objecter _send_command 10 2024-04-26T16:22:19.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.915+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc1b4b40 con 0x7f39dc1afe00 2024-04-26T16:22:19.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc1b4b40 con 0x7f39dc1afe00 2024-04-26T16:22:19.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 4 2024-04-26T16:22:19.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 4 10 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 10 client.24641.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 11 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f39dc1b5260 0x7f39dc1b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 11 2024-04-26T16:22:19.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 5 11 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 4 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 10 client.24641.objecter _send_command 11 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1ba650 con 0x7f39dc1b5260 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.919+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f39dc1b5260 0x7f39dc1b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc1b5260 2024-04-26T16:22:19.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39daffd700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f39dc1b5260 0x7f39dc1b56a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1ba650 con 0x7f39dc1b5260 2024-04-26T16:22:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 11: 0 ) 2024-04-26T16:22:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 11 = system:0 2024-04-26T16:22:19.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 4 2024-04-26T16:22:19.919 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.923+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 5 11 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 10 client.24641.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 12 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 12 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1b4b40 osd=5 3 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 5 12 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 4 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 10 client.24641.objecter _send_command 12 2024-04-26T16:22:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.939+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc1b5be0 con 0x7f39dc1b5260 2024-04-26T16:22:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc1b5be0 con 0x7f39dc1b5260 2024-04-26T16:22:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 4 2024-04-26T16:22:19.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 5 12 2024-04-26T16:22:19.941 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 10 client.24641.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 13 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f39dc1b6fc0 0x7f39dc1b6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 13 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 6 13 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 4 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 10 client.24641.objecter _send_command 13 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.943+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1b6880 con 0x7f39dc1b6fc0 2024-04-26T16:22:19.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f39dc1b6fc0 0x7f39dc1b6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc1b6fc0 2024-04-26T16:22:19.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39dbfff700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f39dc1b6fc0 0x7f39dc1b6340 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1b6880 con 0x7f39dc1b6fc0 2024-04-26T16:22:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 13: 0 ) 2024-04-26T16:22:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 13 = system:0 2024-04-26T16:22:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 4 2024-04-26T16:22:19.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.947+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 6 13 2024-04-26T16:22:19.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 10 client.24641.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 14 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 14 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1b5be0 osd=6 3 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 6 14 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 4 2024-04-26T16:22:19.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 10 client.24641.objecter _send_command 14 2024-04-26T16:22:19.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.967+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc1b6880 con 0x7f39dc1b6fc0 2024-04-26T16:22:19.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc1b6880 con 0x7f39dc1b6fc0 2024-04-26T16:22:19.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 4 2024-04-26T16:22:19.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 6 14 2024-04-26T16:22:19.966 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 10 client.24641.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 15 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f39dc1bd8e0 0x7f39dc1bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 15 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 7 15 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 4 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 10 client.24641.objecter _send_command 15 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f39dc1c2df0 con 0x7f39dc1bd8e0 2024-04-26T16:22:19.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f39dc1bd8e0 0x7f39dc1bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 10 client.24641.objecter ms_handle_connect 0x7f39dc1bd8e0 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39db7fe700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f39dc1bd8e0 0x7f39dc1bfd30 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f39dc1c2df0 con 0x7f39dc1bd8e0 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 15: 0 ) 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 15 = system:0 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 4 2024-04-26T16:22:19.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.971+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 7 15 2024-04-26T16:22:19.974 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:19.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:22:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign -1 16 2024-04-26T16:22:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter _get_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 15 client.24641.objecter _session_command_op_remove -1 16 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter get_session s=0x7f39dc1b6880 osd=7 3 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 15 client.24641.objecter _session_command_op_assign 7 16 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 4 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter _send_command 16 2024-04-26T16:22:19.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f39dc1a7f00 con 0x7f39dc1bd8e0 2024-04-26T16:22:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39d8ff9700 1 -- 172.21.15.50:0/948685979 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f39dc1a7f00 con 0x7f39dc1bd8e0 2024-04-26T16:22:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39d8ff9700 10 client.24641.objecter ms_dispatch 0x7f39dc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39d8ff9700 10 client.24641.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39d8ff9700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 4 2024-04-26T16:22:19.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39d8ff9700 15 client.24641.objecter _session_command_op_remove 7 16 2024-04-26T16:22:19.989 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:22:19.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 librados: watch_flush enter 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 librados: watch_flush exit 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.0 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f39dc064030 msgr2=0x7f39dc064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f39dc064030 0x7f39dc064410 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc19a4a0 osd=0 1 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.1 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f39dc02d310 msgr2=0x7f39dc02d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f39dc02d310 0x7f39dc02d6f0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc10aaa0 osd=1 1 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.2 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f39dc1ab1e0 msgr2=0x7f39dc1ad5d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f39dc1ab1e0 0x7f39dc1ad5d0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc02e400 osd=2 1 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.3 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39dbfff700 1 -- 172.21.15.50:0/948685979 reap_dead start 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f39dc1aec90 msgr2=0x7f39dc1af0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f39dc1aec90 0x7f39dc1af0f0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8670 osd=3 1 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.4 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f39dc1afe00 msgr2=0x7f39dc1b4600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f39dc1afe00 0x7f39dc1b4600 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1a8cf0 osd=4 1 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.5 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f39dc1b5260 msgr2=0x7f39dc1b56a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f39dc1b5260 0x7f39dc1b56a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b4b40 osd=5 1 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.6 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f39dc1b6fc0 msgr2=0x7f39dc1b6340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f39dc1b6fc0 0x7f39dc1b6340 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b5be0 osd=6 1 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 10 client.24641.objecter close_session for osd.7 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f39dc1bd8e0 msgr2=0x7f39dc1bfd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.991+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f39dc1bd8e0 0x7f39dc1bfd30 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 20 client.24641.objecter put_session s=0x7f39dc1b6880 osd=7 1 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 20 client.24641.objecter shutdown clearing up homeless session... 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 10 client.24641.objecter successfully canceled tick 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39dbfff700 1 -- 172.21.15.50:0/948685979 reap_dead start 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f39c4071f80 msgr2=0x7f39c4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f39c4071f80 0x7f39c4074440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39cc005fd0 tx=0x7f39cc005d90 comp rx=0 tx=0).stop 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 msgr2=0x7f39dc19da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:19.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 0x7f39dc19da10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f39d400d010 tx=0x7f39d4007f10 comp rx=0 tx=0).stop 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 shutdown_connections 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f39dc1083b0 0x7f39dc19da10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 --2- 172.21.15.50:0/948685979 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f39c4071f80 0x7f39c4074440 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 >> 172.21.15.50:0/948685979 conn(0x7f39dc07a580 msgr2=0x7f39dc101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 shutdown_connections 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 -- 172.21.15.50:0/948685979 wait complete. 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:19.995+0000 7f39e2054700 1 librados: shutdown 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:22:19.993 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:22:19.994 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:22:19.994 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:22:19.994 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:22:19.994 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:22:19.994 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:22:20.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:20.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:20.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:20.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:20.231 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:22:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:22:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-26T16:22:20.231 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-26T16:22:20.231 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:22:20.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:20.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:20.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:20.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:20.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:20.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:20.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:20.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[21463]: cluster 2024-04-26T16:22:19.445853+0000 mgr.y (mgr.24401) 180 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:20.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[21463]: cluster 2024-04-26T16:22:19.445853+0000 mgr.y (mgr.24401) 180 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:20.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[21463]: audit 2024-04-26T16:22:19.812899+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.50:0/948685979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:20.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[21463]: audit 2024-04-26T16:22:19.812899+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.50:0/948685979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:20.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[29234]: cluster 2024-04-26T16:22:19.445853+0000 mgr.y (mgr.24401) 180 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:20.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[29234]: cluster 2024-04-26T16:22:19.445853+0000 mgr.y (mgr.24401) 180 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:20.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[29234]: audit 2024-04-26T16:22:19.812899+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.50:0/948685979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:20.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:20 smithi050 bash[29234]: audit 2024-04-26T16:22:19.812899+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.50:0/948685979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:20.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 -- 172.21.15.50:0/1850952264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 msgr2=0x7fc24c1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:20.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 --2- 172.21.15.50:0/1850952264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 0x7fc24c1098c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc24400b0a0 tx=0x7fc24400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 -- 172.21.15.50:0/1850952264 shutdown_connections 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 --2- 172.21.15.50:0/1850952264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc24c103e60 0x7fc24c1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 --2- 172.21.15.50:0/1850952264 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 0x7fc24c1098c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 --2- 172.21.15.50:0/1850952264 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc24c1034a0 0x7fc24c103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 -- 172.21.15.50:0/1850952264 >> 172.21.15.50:0/1850952264 conn(0x7fc24c07a590 msgr2=0x7fc24c07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 -- 172.21.15.50:0/1850952264 shutdown_connections 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.683+0000 7fc25119e700 1 -- 172.21.15.50:0/1850952264 wait complete. 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 librados: starting msgr at 2024-04-26T16:22:20.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 librados: starting objecter 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 Processor -- start 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- start start 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 librados: setting wanted keys 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 librados: calling monclient init 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc24c1034a0 0x7fc24c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc24c103e60 0x7fc24c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 0x7fc24c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c112b30 con 0x7fc24c1094e0 2024-04-26T16:22:20.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1129b0 con 0x7fc24c1034a0 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fc24c112cb0 con 0x7fc24c103e60 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24a59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc24c103e60 0x7fc24c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 0x7fc24c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24a59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc24c103e60 0x7fc24c19b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:58806/0 (socket says 172.21.15.50:58806) 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24a59c700 1 -- 172.21.15.50:0/1112630503 learned_addr learned my addr 172.21.15.50:0/1112630503 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24ad9d700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc24c1034a0 0x7fc24c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 -- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc24c103e60 msgr2=0x7fc24c19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:20.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc24c103e60 0x7fc24c19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 -- 172.21.15.50:0/1112630503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc24c1034a0 msgr2=0x7fc24c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc24c1034a0 0x7fc24c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 -- 172.21.15.50:0/1112630503 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c1a03a0 con 0x7fc24c1094e0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc24b59e700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 0x7fc24c19fc60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00b3f0 tx=0x7fc23c00b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 10 client.?.objecter ms_handle_connect 0x7fc24c1094e0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c015020 con 0x7fc24c1094e0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fc23c011c90 con 0x7fc24c1094e0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c1a0690 con 0x7fc24c1094e0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc23c010a30 con 0x7fc24c1094e0 2024-04-26T16:22:20.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1a7f10 con 0x7fc24c1094e0 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 librados: init done 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 10 client.14757.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 15 client.14757.objecter _session_command_op_assign -1 1 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 20 client.14757.objecter _get_session osd=-1 returning homeless 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 10 client.14757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:20.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc24c104b80 con 0x7fc24c1094e0 2024-04-26T16:22:20.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fc23c018020 con 0x7fc24c1094e0 2024-04-26T16:22:20.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.687+0000 7fc233fff700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc234071f80 0x7fc234074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:20.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc24ad9d700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc234071f80 0x7fc234074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:20.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 77..130) ==== 7450+0+0 (secure 0 0 0) 0x7fc23c090860 con 0x7fc24c1094e0 2024-04-26T16:22:20.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter ms_dispatch 0x7fc24c001550 osd_map(130..130 src has 77..130) 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 3 client.14757.objecter handle_osd_map got epochs [130,130] > 0 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 3 client.14757.objecter handle_osd_map decoding full epoch 130 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter checking command 1 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc23407be30 0x7fc23407e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter _get_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 15 client.14757.objecter _session_command_op_remove -1 1 2024-04-26T16:22:20.691 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter _get_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter get_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 15 client.14757.objecter _session_command_op_assign 1 1 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 4 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter _send_command 1 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc23407e960 con 0x7fc23407be30 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter dump_active .. 0 homeless 2024-04-26T16:22:20.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=130) ==== 24+0+0 (secure 0 0 0) 0x7fc23c0952c0 con 0x7fc24c1094e0 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc24a59c700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc23407be30 0x7fc23407e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc24ad9d700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc234071f80 0x7fc234074440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc244006130 tx=0x7fc2440060a0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter ms_handle_connect 0x7fc234071f80 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter ms_handle_connect 0x7fc23407be30 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc24a59c700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc23407be30 0x7fc23407e290 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc23407e960 con 0x7fc23407be30 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter ms_dispatch 0x7fc24c001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:20.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 10 client.14757.objecter _finish_command 1 = system:0 2024-04-26T16:22:20.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 4 2024-04-26T16:22:20.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.691+0000 7fc233fff700 15 client.14757.objecter _session_command_op_remove 1 1 2024-04-26T16:22:20.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:20.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 10 client.14757.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 15 client.14757.objecter _session_command_op_assign -1 2 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter _get_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter _get_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 15 client.14757.objecter _session_command_op_remove -1 2 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter get_session s=0x7fc23407b760 osd=1 3 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 15 client.14757.objecter _session_command_op_assign 1 2 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 4 2024-04-26T16:22:20.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 10 client.14757.objecter _send_command 2 2024-04-26T16:22:20.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fc24c06a110 con 0x7fc23407be30 2024-04-26T16:22:20.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc233fff700 1 -- 172.21.15.50:0/1112630503 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fc24c06a110 con 0x7fc23407be30 2024-04-26T16:22:20.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc233fff700 10 client.14757.objecter ms_dispatch 0x7fc24c001550 command_reply(tid 2: 0 ) 2024-04-26T16:22:20.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc233fff700 10 client.14757.objecter _finish_command 2 = system:0 2024-04-26T16:22:20.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc233fff700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 4 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc233fff700 15 client.14757.objecter _session_command_op_remove 1 2 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 10 librados: watch_flush enter 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 10 librados: watch_flush exit 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 10 client.14757.objecter close_session for osd.1 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc23407be30 msgr2=0x7fc23407e290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc23407be30 0x7fc23407e290 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter put_session s=0x7fc23407b760 osd=1 1 2024-04-26T16:22:20.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 20 client.14757.objecter shutdown clearing up homeless session... 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 10 client.14757.objecter successfully canceled tick 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc234071f80 msgr2=0x7fc234074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc234071f80 0x7fc234074440 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc244006130 tx=0x7fc2440060a0 comp rx=0 tx=0).stop 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 msgr2=0x7fc24c19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 --2- 172.21.15.50:0/1112630503 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc24c1094e0 0x7fc24c19fc60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00b3f0 tx=0x7fc23c00b700 comp rx=0 tx=0).stop 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc24b59e700 1 -- 172.21.15.50:0/1112630503 reap_dead start 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 shutdown_connections 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 >> 172.21.15.50:0/1112630503 conn(0x7fc24c07a590 msgr2=0x7fc24c075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 shutdown_connections 2024-04-26T16:22:20.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 -- 172.21.15.50:0/1112630503 wait complete. 2024-04-26T16:22:20.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:20.711+0000 7fc25119e700 1 librados: shutdown 2024-04-26T16:22:20.729 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:20.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:20 smithi190 bash[23868]: cluster 2024-04-26T16:22:19.445853+0000 mgr.y (mgr.24401) 180 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:20.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:20 smithi190 bash[23868]: cluster 2024-04-26T16:22:19.445853+0000 mgr.y (mgr.24401) 180 : cluster [DBG] pgmap v149: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:20.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:20 smithi190 bash[23868]: audit 2024-04-26T16:22:19.812899+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.50:0/948685979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:20.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:20 smithi190 bash[23868]: audit 2024-04-26T16:22:19.812899+0000 mon.c (mon.2) 88 : audit [DBG] from='client.? 172.21.15.50:0/948685979' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:20.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:20.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:20.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:20.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:20.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:21.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:21.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:21.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:21.248 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:21.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:21.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:21.377 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:21.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:21.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:21.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:21.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:21.639 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:21.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:21.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:21.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:21 smithi190 bash[23868]: cluster 2024-04-26T16:22:19.932691+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:21.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:21 smithi190 bash[23868]: cluster 2024-04-26T16:22:19.932691+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:21.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:21 smithi190 bash[23868]: cluster 2024-04-26T16:22:19.932759+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:21.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:21 smithi190 bash[23868]: cluster 2024-04-26T16:22:19.932759+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:21.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:21 smithi190 bash[23868]: cluster 2024-04-26T16:22:20.540235+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-04-26T16:22:21.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:21 smithi190 bash[23868]: cluster 2024-04-26T16:22:20.540235+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-04-26T16:22:21.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[21463]: cluster 2024-04-26T16:22:19.932691+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:21.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[21463]: cluster 2024-04-26T16:22:19.932691+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[21463]: cluster 2024-04-26T16:22:19.932759+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[21463]: cluster 2024-04-26T16:22:19.932759+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[21463]: cluster 2024-04-26T16:22:20.540235+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[21463]: cluster 2024-04-26T16:22:20.540235+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[29234]: cluster 2024-04-26T16:22:19.932691+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[29234]: cluster 2024-04-26T16:22:19.932691+0000 osd.0 (osd.0) 115 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[29234]: cluster 2024-04-26T16:22:19.932759+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[29234]: cluster 2024-04-26T16:22:19.932759+0000 osd.0 (osd.0) 116 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[29234]: cluster 2024-04-26T16:22:20.540235+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-04-26T16:22:21.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:21 smithi050 bash[29234]: cluster 2024-04-26T16:22:20.540235+0000 mon.a (mon.0) 948 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2024-04-26T16:22:21.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:21.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:21.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:22.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:22.035 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:22.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:22.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:22.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:22.297 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:22.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:22.427 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:22.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:22.557 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:22.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:22.688 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:22.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:22.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:22 smithi190 bash[23868]: cluster 2024-04-26T16:22:21.447294+0000 mgr.y (mgr.24401) 181 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:22.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:22 smithi190 bash[23868]: cluster 2024-04-26T16:22:21.447294+0000 mgr.y (mgr.24401) 181 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:22.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:22.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:22 smithi050 bash[21463]: cluster 2024-04-26T16:22:21.447294+0000 mgr.y (mgr.24401) 181 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:22.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:22 smithi050 bash[21463]: cluster 2024-04-26T16:22:21.447294+0000 mgr.y (mgr.24401) 181 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:22.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:22 smithi050 bash[29234]: cluster 2024-04-26T16:22:21.447294+0000 mgr.y (mgr.24401) 181 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:22.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:22 smithi050 bash[29234]: cluster 2024-04-26T16:22:21.447294+0000 mgr.y (mgr.24401) 181 : cluster [DBG] pgmap v151: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:22.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:22.945 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:22.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:22.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:23.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:23.076 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:23.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-26T16:22:23.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:23.207 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:23.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:23.337 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:23.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.2 reset_purged_snaps_last 2024-04-26T16:22:23.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:23.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:23.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:23.638 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:23.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:23.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:23.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 -- 172.21.15.50:0/3811932695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954102810 msgr2=0x7fb954110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 --2- 172.21.15.50:0/3811932695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954102810 0x7fb954110310 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb94c00b0a0 tx=0x7fb94c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 -- 172.21.15.50:0/3811932695 shutdown_connections 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 --2- 172.21.15.50:0/3811932695 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954102810 0x7fb954110310 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 --2- 172.21.15.50:0/3811932695 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954109600 0x7fb9541099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 --2- 172.21.15.50:0/3811932695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb954101e50 0x7fb9541022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 -- 172.21.15.50:0/3811932695 >> 172.21.15.50:0/3811932695 conn(0x7fb954077590 msgr2=0x7fb9540779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 -- 172.21.15.50:0/3811932695 shutdown_connections 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.847+0000 7fb959b3c700 1 -- 172.21.15.50:0/3811932695 wait complete. 2024-04-26T16:22:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 librados: starting msgr at 2024-04-26T16:22:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 librados: starting objecter 2024-04-26T16:22:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 Processor -- start 2024-04-26T16:22:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- start start 2024-04-26T16:22:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 librados: setting wanted keys 2024-04-26T16:22:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 librados: calling monclient init 2024-04-26T16:22:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 0x7fb954103030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 0x7fb954103570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb954109600 0x7fb954103ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb954112b50 con 0x7fb954102810 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb9541129d0 con 0x7fb954109600 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb954112cd0 con 0x7fb954101e50 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9537fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 0x7fb954103030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 0x7fb954103570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9537fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 0x7fb954103030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:58830/0 (socket says 172.21.15.50:58830) 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 0x7fb954103570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39800/0 (socket says 172.21.15.50:39800) 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9537fe700 1 -- 172.21.15.50:0/218122433 learned_addr learned my addr 172.21.15.50:0/218122433 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb953fff700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb954109600 0x7fb954103ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 -- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 msgr2=0x7fb954103030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 0x7fb954103030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 -- 172.21.15.50:0/218122433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb954109600 msgr2=0x7fb954103ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb954109600 0x7fb954103ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 -- 172.21.15.50:0/218122433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9541a7680 con 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9537fe700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 0x7fb954103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb94affd700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 0x7fb954103570 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b450 tx=0x7fb93c00b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 10 client.?.objecter ms_handle_connect 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c015020 con 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb93c011c90 con 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9541a7970 con 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c010ae0 con 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 10 client.14763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9541a7f00 con 0x7fb954102810 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 librados: init done 2024-04-26T16:22:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95406a110 con 0x7fb954102810 2024-04-26T16:22:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb93c018020 con 0x7fb954102810 2024-04-26T16:22:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb940071f80 0x7fb940074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:23.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.851+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(130..130 src has 77..130) ==== 7450+0+0 (secure 0 0 0) 0x7fb93c090920 con 0x7fb954102810 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9517fa700 10 client.14763.objecter ms_dispatch 0x7fb954001550 osd_map(130..130 src has 77..130) 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9517fa700 3 client.14763.objecter handle_osd_map got epochs [130,130] > 0 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9517fa700 3 client.14763.objecter handle_osd_map decoding full epoch 130 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9537fe700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb940071f80 0x7fb940074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9517fa700 20 client.14763.objecter dump_active .. 0 homeless 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9537fe700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb940071f80 0x7fb940074440 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb944011550 tx=0x7fb944009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:23.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.855+0000 7fb9517fa700 10 client.14763.objecter ms_handle_connect 0x7fb940071f80 2024-04-26T16:22:23.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:23.859+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb93c01f020 con 0x7fb954102810 2024-04-26T16:22:23.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:23.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:23.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:23.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:24.022 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.095+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fb9541a8130 con 0x7fb954102810 2024-04-26T16:22:24.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:24.157 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:24.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:24.288 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:24.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:24.418 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:24.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:24.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:24.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.547+0000 7fb9517fa700 1 -- 172.21.15.50:0/218122433 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v131) ==== 97+0+0 (secure 0 0 0) 0x7fb93c05cb30 con 0x7fb954102810 2024-04-26T16:22:24.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 10 librados: watch_flush enter 2024-04-26T16:22:24.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 10 librados: watch_flush exit 2024-04-26T16:22:24.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 20 client.14763.objecter shutdown clearing up homeless session... 2024-04-26T16:22:24.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 10 client.14763.objecter successfully canceled tick 2024-04-26T16:22:24.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb940071f80 msgr2=0x7fb940074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:24.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb940071f80 0x7fb940074440 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb944011550 tx=0x7fb944009260 comp rx=0 tx=0).stop 2024-04-26T16:22:24.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 msgr2=0x7fb954103570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:24.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 0x7fb954103570 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b450 tx=0x7fb93c00b760 comp rx=0 tx=0).stop 2024-04-26T16:22:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 shutdown_connections 2024-04-26T16:22:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb940071f80 0x7fb940074440 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb954109600 0x7fb954103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb954101e50 0x7fb954103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 --2- 172.21.15.50:0/218122433 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb954102810 0x7fb954103570 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb93c00b450 tx=0x7fb93c00b760 comp rx=0 tx=0).stop 2024-04-26T16:22:24.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 >> 172.21.15.50:0/218122433 conn(0x7fb954077590 msgr2=0x7fb954100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:24.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 shutdown_connections 2024-04-26T16:22:24.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.551+0000 7fb959b3c700 1 -- 172.21.15.50:0/218122433 wait complete. 2024-04-26T16:22:24.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:24.555+0000 7fb959b3c700 1 librados: shutdown 2024-04-26T16:22:24.551 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:22:24.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:24.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:24.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:24.804 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:22:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:22:24.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[21463]: cluster 2024-04-26T16:22:23.448025+0000 mgr.y (mgr.24401) 182 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:24.804 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[21463]: cluster 2024-04-26T16:22:23.448025+0000 mgr.y (mgr.24401) 182 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:24.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[21463]: audit 2024-04-26T16:22:24.101854+0000 mon.a (mon.0) 949 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:22:24.805 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[21463]: audit 2024-04-26T16:22:24.101854+0000 mon.a (mon.0) 949 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:22:24.805 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[29234]: cluster 2024-04-26T16:22:23.448025+0000 mgr.y (mgr.24401) 182 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:24.805 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[29234]: cluster 2024-04-26T16:22:23.448025+0000 mgr.y (mgr.24401) 182 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:24.805 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[29234]: audit 2024-04-26T16:22:24.101854+0000 mon.a (mon.0) 949 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:22:24.805 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:24 smithi050 bash[29234]: audit 2024-04-26T16:22:24.101854+0000 mon.a (mon.0) 949 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:22:24.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:24 smithi190 bash[23868]: cluster 2024-04-26T16:22:23.448025+0000 mgr.y (mgr.24401) 182 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:24.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:24 smithi190 bash[23868]: cluster 2024-04-26T16:22:23.448025+0000 mgr.y (mgr.24401) 182 : cluster [DBG] pgmap v152: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:24.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:24 smithi190 bash[23868]: audit 2024-04-26T16:22:24.101854+0000 mon.a (mon.0) 949 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:22:24.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:24 smithi190 bash[23868]: audit 2024-04-26T16:22:24.101854+0000 mon.a (mon.0) 949 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:22:24.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:24.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:24.953 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:24.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:25.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:25.084 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:25.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:22:25.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:25.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:25.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:25.344 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:25.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:25.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:25.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:25.587 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:25 smithi190 bash[23868]: cluster 2024-04-26T16:22:24.550334+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:25.588 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:25 smithi190 bash[23868]: cluster 2024-04-26T16:22:24.550334+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:25.588 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:25 smithi190 bash[23868]: audit 2024-04-26T16:22:24.552218+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:22:25.588 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:25 smithi190 bash[23868]: audit 2024-04-26T16:22:24.552218+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:22:25.588 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:25 smithi190 bash[23868]: cluster 2024-04-26T16:22:24.557551+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-04-26T16:22:25.588 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:25 smithi190 bash[23868]: cluster 2024-04-26T16:22:24.557551+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-04-26T16:22:25.604 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:25.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:25.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:25.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[21463]: cluster 2024-04-26T16:22:24.550334+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:25.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[21463]: cluster 2024-04-26T16:22:24.550334+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:25.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[21463]: audit 2024-04-26T16:22:24.552218+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[21463]: audit 2024-04-26T16:22:24.552218+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[21463]: cluster 2024-04-26T16:22:24.557551+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[21463]: cluster 2024-04-26T16:22:24.557551+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[29234]: cluster 2024-04-26T16:22:24.550334+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[29234]: cluster 2024-04-26T16:22:24.550334+0000 mon.a (mon.0) 950 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[29234]: audit 2024-04-26T16:22:24.552218+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[29234]: audit 2024-04-26T16:22:24.552218+0000 mon.a (mon.0) 951 : audit [INF] from='client.? 172.21.15.50:0/218122433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[29234]: cluster 2024-04-26T16:22:24.557551+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-04-26T16:22:25.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:25 smithi050 bash[29234]: cluster 2024-04-26T16:22:24.557551+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2024-04-26T16:22:25.736 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:25.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:25.866 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:25.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:25.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:25.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:25.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:26.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:26.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:26.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:26.250 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:26.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:26.379 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:26.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:26.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:26.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:26.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:26.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:26.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:26.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:22:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:22:26.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:26 smithi190 bash[23868]: cluster 2024-04-26T16:22:25.448645+0000 mgr.y (mgr.24401) 183 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:26.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:26 smithi190 bash[23868]: cluster 2024-04-26T16:22:25.448645+0000 mgr.y (mgr.24401) 183 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:26.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:26 smithi190 bash[23868]: audit 2024-04-26T16:22:25.892552+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:26.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:26 smithi190 bash[23868]: audit 2024-04-26T16:22:25.892552+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:26.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[29234]: cluster 2024-04-26T16:22:25.448645+0000 mgr.y (mgr.24401) 183 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[29234]: cluster 2024-04-26T16:22:25.448645+0000 mgr.y (mgr.24401) 183 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[29234]: audit 2024-04-26T16:22:25.892552+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[29234]: audit 2024-04-26T16:22:25.892552+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[21463]: cluster 2024-04-26T16:22:25.448645+0000 mgr.y (mgr.24401) 183 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[21463]: cluster 2024-04-26T16:22:25.448645+0000 mgr.y (mgr.24401) 183 : cluster [DBG] pgmap v154: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[21463]: audit 2024-04-26T16:22:25.892552+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:26.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:26 smithi050 bash[21463]: audit 2024-04-26T16:22:25.892552+0000 mon.c (mon.2) 89 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:26.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:26.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:27.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:27.028 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:27.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:27.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:27.157 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:27.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:27.288 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:27.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:27.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:27.419 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:27.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:27.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:22:27.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:27.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:27.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:27.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:27 smithi190 bash[23868]: audit 2024-04-26T16:22:26.545508+0000 mgr.y (mgr.24401) 184 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:27.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:27 smithi190 bash[23868]: audit 2024-04-26T16:22:26.545508+0000 mgr.y (mgr.24401) 184 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:27.809 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:27 smithi050 bash[21463]: audit 2024-04-26T16:22:26.545508+0000 mgr.y (mgr.24401) 184 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:27.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:27 smithi050 bash[21463]: audit 2024-04-26T16:22:26.545508+0000 mgr.y (mgr.24401) 184 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:27.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:27 smithi050 bash[29234]: audit 2024-04-26T16:22:26.545508+0000 mgr.y (mgr.24401) 184 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:27.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:27 smithi050 bash[29234]: audit 2024-04-26T16:22:26.545508+0000 mgr.y (mgr.24401) 184 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:27.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:27.940 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:27.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:28.041 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:28.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:28.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:28.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:28.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.198 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:28.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:28.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:28.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:28.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.589 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:28.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:28.720 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:28.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:28 smithi190 bash[23868]: cluster 2024-04-26T16:22:27.450126+0000 mgr.y (mgr.24401) 185 : cluster [DBG] pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:28.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:28 smithi190 bash[23868]: cluster 2024-04-26T16:22:27.450126+0000 mgr.y (mgr.24401) 185 : cluster [DBG] pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:28.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:28.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:28 smithi050 bash[21463]: cluster 2024-04-26T16:22:27.450126+0000 mgr.y (mgr.24401) 185 : cluster [DBG] pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:28.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:28 smithi050 bash[21463]: cluster 2024-04-26T16:22:27.450126+0000 mgr.y (mgr.24401) 185 : cluster [DBG] pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:28.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:28 smithi050 bash[29234]: cluster 2024-04-26T16:22:27.450126+0000 mgr.y (mgr.24401) 185 : cluster [DBG] pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:28.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:28 smithi050 bash[29234]: cluster 2024-04-26T16:22:27.450126+0000 mgr.y (mgr.24401) 185 : cluster [DBG] pgmap v155: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:28.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.979 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:28.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:28.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:29.065 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:29.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:29.107 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:29.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:29.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:29.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:29.371 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:29.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:29.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:29.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:29.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:29.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:29.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:29.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:29.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:29.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:30.020 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:30.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:30.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:30.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:30.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:30.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:30.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:30.540 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:30.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:30.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:30 smithi190 bash[23868]: cluster 2024-04-26T16:22:29.450791+0000 mgr.y (mgr.24401) 186 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:30.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:30 smithi190 bash[23868]: cluster 2024-04-26T16:22:29.450791+0000 mgr.y (mgr.24401) 186 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:30.672 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:30.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:30 smithi050 bash[29234]: cluster 2024-04-26T16:22:29.450791+0000 mgr.y (mgr.24401) 186 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:30.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:30 smithi050 bash[29234]: cluster 2024-04-26T16:22:29.450791+0000 mgr.y (mgr.24401) 186 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:30.698 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:30 smithi050 bash[21463]: cluster 2024-04-26T16:22:29.450791+0000 mgr.y (mgr.24401) 186 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:30.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:30 smithi050 bash[21463]: cluster 2024-04-26T16:22:29.450791+0000 mgr.y (mgr.24401) 186 : cluster [DBG] pgmap v156: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:30.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:30.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:30.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:30.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:30.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:31.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:31.064 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:31.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:31.194 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:31.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:31.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:31.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:31.457 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:31.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:31.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:31.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:31.718 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:31.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:31.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:31.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:31.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:31.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:32.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:32.112 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:32 smithi050 bash[29234]: cluster 2024-04-26T16:22:31.452259+0000 mgr.y (mgr.24401) 187 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:32.125 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:32 smithi050 bash[29234]: cluster 2024-04-26T16:22:31.452259+0000 mgr.y (mgr.24401) 187 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:32 smithi050 bash[21463]: cluster 2024-04-26T16:22:31.452259+0000 mgr.y (mgr.24401) 187 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:32.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:32 smithi050 bash[21463]: cluster 2024-04-26T16:22:31.452259+0000 mgr.y (mgr.24401) 187 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:32.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:32.227 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:31 smithi190 bash[23868]: cluster 2024-04-26T16:22:31.452259+0000 mgr.y (mgr.24401) 187 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:32.227 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:31 smithi190 bash[23868]: cluster 2024-04-26T16:22:31.452259+0000 mgr.y (mgr.24401) 187 : cluster [DBG] pgmap v157: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:32.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:32.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:32.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:32.376 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:32.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:32.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:32.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:32.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:32.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:32.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:32.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:32.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:32.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:32.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:33.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:33.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:33.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:33.199 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:33.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:33.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:33.424 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:33.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:33.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:33.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:33.584 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:33.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:33.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:33.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:33.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:33.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:33.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:33.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:34.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:34.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:34.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:34.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:34.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:34.367 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:34.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:34.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.234876+0000 osd.2 (osd.2) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.234876+0000 osd.2 (osd.2) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.235001+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.235001+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.296602+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-04-26T16:22:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.296602+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-04-26T16:22:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.452729+0000 mgr.y (mgr.24401) 188 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:34 smithi190 bash[23868]: cluster 2024-04-26T16:22:33.452729+0000 mgr.y (mgr.24401) 188 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:34.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:34.625 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:22:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:22:34.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.234876+0000 osd.2 (osd.2) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:34.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.234876+0000 osd.2 (osd.2) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.235001+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.235001+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.296602+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.296602+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.452729+0000 mgr.y (mgr.24401) 188 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[29234]: cluster 2024-04-26T16:22:33.452729+0000 mgr.y (mgr.24401) 188 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.234876+0000 osd.2 (osd.2) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.234876+0000 osd.2 (osd.2) 83 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.235001+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.235001+0000 osd.2 (osd.2) 84 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:34.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.296602+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-04-26T16:22:34.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.296602+0000 mon.a (mon.0) 953 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2024-04-26T16:22:34.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.452729+0000 mgr.y (mgr.24401) 188 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:34.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:34 smithi050 bash[21463]: cluster 2024-04-26T16:22:33.452729+0000 mgr.y (mgr.24401) 188 : cluster [DBG] pgmap v159: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:22:34.630 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:34.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:34.760 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:34.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:34.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:34.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:34.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:35.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:35.145 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:35.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:35.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:35.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:35.404 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:35.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:35.533 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:35.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:35.665 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:35.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:35.796 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:35.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:35.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:35.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:35.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:36.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:36.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:36.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:36.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:36.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:36.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:36.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:36.433 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:36.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:36.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:36 smithi190 bash[23868]: cluster 2024-04-26T16:22:35.453175+0000 mgr.y (mgr.24401) 189 : cluster [DBG] pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:36.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:36 smithi190 bash[23868]: cluster 2024-04-26T16:22:35.453175+0000 mgr.y (mgr.24401) 189 : cluster [DBG] pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:36.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:36.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/3194798746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc109490 msgr2=0x7f1cbc109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/3194798746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc109490 0x7f1cbc109870 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1cac009990 tx=0x7f1cac009ca0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/3194798746 shutdown_connections 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/3194798746 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cbc06c360 0x7f1cbc1102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/3194798746 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc109490 0x7f1cbc109870 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/3194798746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cbc06b9c0 0x7f1cbc06be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/3194798746 >> 172.21.15.50:0/3194798746 conn(0x7f1cbc079290 msgr2=0x7f1cbc0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/3194798746 shutdown_connections 2024-04-26T16:22:36.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/3194798746 wait complete. 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 librados: starting msgr at 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 librados: starting objecter 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 Processor -- start 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- start start 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 librados: setting wanted keys 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 librados: calling monclient init 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cbc06b9c0 0x7f1cbc199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc06c360 0x7f1cbc199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cbc109490 0x7f1cbc19db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc112ad0 con 0x7f1cbc06c360 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1cbc112950 con 0x7f1cbc06b9c0 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1cbc112c50 con 0x7f1cbc109490 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc06c360 0x7f1cbc199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc06c360 0x7f1cbc199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35288/0 (socket says 172.21.15.50:35288) 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 -- 172.21.15.50:0/517851963 learned_addr learned my addr 172.21.15.50:0/517851963 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cbc109490 0x7f1cbc19db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cbc06b9c0 0x7f1cbc199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cbc109490 msgr2=0x7f1cbc19db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1cbc109490 0x7f1cbc19db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cbc06b9c0 msgr2=0x7f1cbc199250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cbc06b9c0 0x7f1cbc199250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cbc19e260 con 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1cbc06b9c0 0x7f1cbc199250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc06c360 0x7f1cbc199790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000b530 tx=0x7f1cb000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cb8ff9700 10 client.?.objecter ms_handle_connect 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cb8ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0015020 con 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cbc19e550 con 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1cb0011c90 con 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 10 client.14769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0010bf0 con 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cbc1a5dd0 con 0x7f1cbc06c360 2024-04-26T16:22:36.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 librados: init done 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.626+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f1cbc104b40 con 0x7f1cbc06c360 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f1cb0018020 con 0x7f1cbc06c360 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1ca4071e50 0x7f1ca4074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(132..132 src has 77..132) ==== 7450+0+0 (secure 0 0 0) 0x7f1cb0090810 con 0x7f1cbc06c360 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1ca4071e50 0x7f1ca4074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 osd_map(132..132 src has 77..132) 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 3 client.14769.objecter handle_osd_map got epochs [132,132] > 0 2024-04-26T16:22:36.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 3 client.14769.objecter handle_osd_map decoding full epoch 132 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 20 client.14769.objecter dump_active .. 0 homeless 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v132) ==== 54+0+16 (secure 0 0 0) 0x7f1cb001f030 con 0x7f1cbc06c360 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 1 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1cbc064030 0x7f1cbc064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 1 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 0 1 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 4 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 10 client.14769.objecter _send_command 1 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc19a5b0 con 0x7f1cbc064030 2024-04-26T16:22:36.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1ca4071e50 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1ca4071e50 0x7f1ca4074310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00b730 tx=0x7f1cac005e80 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1cbc064030 0x7f1cbc064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc064030 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1cbc064030 0x7f1cbc064410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc19a5b0 con 0x7f1cbc064030 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 1 = system:0 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 4 2024-04-26T16:22:36.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.630+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 0 1 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 2 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 2 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc06a110 osd=0 3 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 0 2 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 4 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 10 client.14769.objecter _send_command 2 2024-04-26T16:22:36.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc063ab0 con 0x7f1cbc064030 2024-04-26T16:22:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc063ab0 con 0x7f1cbc064030 2024-04-26T16:22:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 4 2024-04-26T16:22:36.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 0 2 2024-04-26T16:22:36.643 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 3 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f1cbc10aef0 0x7f1cbc1a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 3 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 1 3 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 4 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 10 client.14769.objecter _send_command 3 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc02e600 con 0x7f1cbc10aef0 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f1cbc10aef0 0x7f1cbc1a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc10aef0 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f1cbc10aef0 0x7f1cbc1a9fc0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc02e600 con 0x7f1cbc10aef0 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 3: 0 ) 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 3 = system:0 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 4 2024-04-26T16:22:36.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.650+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 1 3 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 4 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 4 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc02e490 osd=1 3 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 1 4 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 4 2024-04-26T16:22:36.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 10 client.14769.objecter _send_command 4 2024-04-26T16:22:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc1aa500 con 0x7f1cbc10aef0 2024-04-26T16:22:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc1aa500 con 0x7f1cbc10aef0 2024-04-26T16:22:36.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 4 2024-04-26T16:22:36.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 1 4 2024-04-26T16:22:36.664 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 5 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f1cbc10a5d0 0x7f1cbc1ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 5 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 2 5 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 4 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 10 client.14769.objecter _send_command 5 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc02dff0 con 0x7f1cbc10a5d0 2024-04-26T16:22:36.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f1cbc10a5d0 0x7f1cbc1ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc10a5d0 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f1cbc10a5d0 0x7f1cbc1ad840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc02dff0 con 0x7f1cbc10a5d0 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 5: 0 ) 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 5 = system:0 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 4 2024-04-26T16:22:36.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.670+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 2 5 2024-04-26T16:22:36.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 6 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 6 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc10a460 osd=2 3 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 2 6 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 4 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 10 client.14769.objecter _send_command 6 2024-04-26T16:22:36.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.690+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc02e240 con 0x7f1cbc10a5d0 2024-04-26T16:22:36.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc02e240 con 0x7f1cbc10a5d0 2024-04-26T16:22:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 4 2024-04-26T16:22:36.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 2 6 2024-04-26T16:22:36.687 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 7 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f1cbc1aae80 0x7f1cbc1ab260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 7 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 3 7 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 4 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 10 client.14769.objecter _send_command 7 2024-04-26T16:22:36.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc1afd90 con 0x7f1cbc1aae80 2024-04-26T16:22:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f1cbc1aae80 0x7f1cbc1ab260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc1aae80 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f1cbc1aae80 0x7f1cbc1ab260 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc1afd90 con 0x7f1cbc1aae80 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 7: 0 ) 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 7 = system:0 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 4 2024-04-26T16:22:36.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.694+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 3 7 2024-04-26T16:22:36.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:36 smithi050 bash[21463]: cluster 2024-04-26T16:22:35.453175+0000 mgr.y (mgr.24401) 189 : cluster [DBG] pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:36.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:36 smithi050 bash[21463]: cluster 2024-04-26T16:22:35.453175+0000 mgr.y (mgr.24401) 189 : cluster [DBG] pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:36.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:36 smithi050 bash[29234]: cluster 2024-04-26T16:22:35.453175+0000 mgr.y (mgr.24401) 189 : cluster [DBG] pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:36.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:36 smithi050 bash[29234]: cluster 2024-04-26T16:22:35.453175+0000 mgr.y (mgr.24401) 189 : cluster [DBG] pgmap v160: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 8 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 8 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc02e240 osd=3 3 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 3 8 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 4 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 10 client.14769.objecter _send_command 8 2024-04-26T16:22:36.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.714+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc1b0220 con 0x7f1cbc1aae80 2024-04-26T16:22:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc1b0220 con 0x7f1cbc1aae80 2024-04-26T16:22:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 4 2024-04-26T16:22:36.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 3 8 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 9 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1cbc1039a0 0x7f1cbc103e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 9 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 4 9 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 4 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 10 client.14769.objecter _send_command 9 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.718+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc1b0390 con 0x7f1cbc1039a0 2024-04-26T16:22:36.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1cbc1039a0 0x7f1cbc103e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc1039a0 2024-04-26T16:22:36.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1cbc1039a0 0x7f1cbc103e00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc1b0390 con 0x7f1cbc1039a0 2024-04-26T16:22:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 9: 0 ) 2024-04-26T16:22:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 9 = system:0 2024-04-26T16:22:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 4 2024-04-26T16:22:36.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.722+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 4 9 2024-04-26T16:22:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 10 2024-04-26T16:22:36.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 10 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1b0220 osd=4 3 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 4 10 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 4 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 10 client.14769.objecter _send_command 10 2024-04-26T16:22:36.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.746+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc1b0390 con 0x7f1cbc1039a0 2024-04-26T16:22:36.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc1b0390 con 0x7f1cbc1039a0 2024-04-26T16:22:36.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 4 2024-04-26T16:22:36.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 4 10 2024-04-26T16:22:36.745 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 11 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1cbc1b6fd0 0x7f1cbc1b93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 11 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 5 11 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 4 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 10 client.14769.objecter _send_command 11 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.750+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc1bc2b0 con 0x7f1cbc1b6fd0 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1cbc1b6fd0 0x7f1cbc1b93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc1b6fd0 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cbaffd700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1cbc1b6fd0 0x7f1cbc1b93c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cb004da50 con 0x7f1cbc1b6fd0 2024-04-26T16:22:36.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 11: 0 ) 2024-04-26T16:22:36.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 11 = system:0 2024-04-26T16:22:36.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 4 2024-04-26T16:22:36.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.754+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 5 11 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 12 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 12 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1b0390 osd=5 3 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 5 12 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 4 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 10 client.14769.objecter _send_command 12 2024-04-26T16:22:36.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.774+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc1b9900 con 0x7f1cbc1b6fd0 2024-04-26T16:22:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.778+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc1b9900 con 0x7f1cbc1b6fd0 2024-04-26T16:22:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.778+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.778+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.778+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 4 2024-04-26T16:22:36.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.778+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 5 12 2024-04-26T16:22:36.773 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 13 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f1cbc1bb810 0x7f1cbc1bbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 13 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 6 13 2024-04-26T16:22:36.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 4 2024-04-26T16:22:36.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 10 client.14769.objecter _send_command 13 2024-04-26T16:22:36.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc1badf0 con 0x7f1cbc1bb810 2024-04-26T16:22:36.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f1cbc1bb810 0x7f1cbc1bbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc1bb810 2024-04-26T16:22:36.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cbbfff700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f1cbc1bb810 0x7f1cbc1bbc50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc1badf0 con 0x7f1cbc1bb810 2024-04-26T16:22:36.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 13: 0 ) 2024-04-26T16:22:36.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 13 = system:0 2024-04-26T16:22:36.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 4 2024-04-26T16:22:36.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.782+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 6 13 2024-04-26T16:22:36.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 14 2024-04-26T16:22:36.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 14 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1b9900 osd=6 3 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 6 14 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 4 2024-04-26T16:22:36.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 10 client.14769.objecter _send_command 14 2024-04-26T16:22:36.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.802+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc1badf0 con 0x7f1cbc1bb810 2024-04-26T16:22:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc1badf0 con 0x7f1cbc1bb810 2024-04-26T16:22:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 4 2024-04-26T16:22:36.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 6 14 2024-04-26T16:22:36.800 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:22:36.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:36.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 15 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f1cbc1ba130 0x7f1cbc1ba590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 15 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 7 15 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 4 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 10 client.14769.objecter _send_command 15 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f1cbc1c2e30 con 0x7f1cbc1ba130 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f1cbc1ba130 0x7f1cbc1ba590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cb8ff9700 10 client.14769.objecter ms_handle_connect 0x7f1cbc1ba130 2024-04-26T16:22:36.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.806+0000 7f1cbb7fe700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f1cbc1ba130 0x7f1cbc1ba590 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:36.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:22:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:22:36.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.810+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1cbc1c2e30 con 0x7f1cbc1ba130 2024-04-26T16:22:36.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.810+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 15: 0 ) 2024-04-26T16:22:36.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.810+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 15 = system:0 2024-04-26T16:22:36.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.810+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 4 2024-04-26T16:22:36.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.810+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 7 15 2024-04-26T16:22:36.817 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:36.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 10 client.14769.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:22:36.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign -1 16 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 20 client.14769.objecter _get_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_remove -1 16 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 20 client.14769.objecter get_session s=0x7f1cbc1badf0 osd=7 3 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 15 client.14769.objecter _session_command_op_assign 7 16 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 4 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 10 client.14769.objecter _send_command 16 2024-04-26T16:22:36.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.826+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f1cbc1c0510 con 0x7f1cbc1ba130 2024-04-26T16:22:36.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cb8ff9700 1 -- 172.21.15.50:0/517851963 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1cbc1c0510 con 0x7f1cbc1ba130 2024-04-26T16:22:36.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cb8ff9700 10 client.14769.objecter ms_dispatch 0x7f1cbc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:22:36.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cb8ff9700 10 client.14769.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:22:36.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cb8ff9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 4 2024-04-26T16:22:36.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cb8ff9700 15 client.14769.objecter _session_command_op_remove 7 16 2024-04-26T16:22:36.823 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 librados: watch_flush enter 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 librados: watch_flush exit 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.0 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1cbc064030 msgr2=0x7f1cbc064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1cbc064030 0x7f1cbc064410 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc06a110 osd=0 1 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.1 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f1cbc10aef0 msgr2=0x7f1cbc1a9fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f1cbc10aef0 0x7f1cbc1a9fc0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e490 osd=1 1 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.2 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f1cbc10a5d0 msgr2=0x7f1cbc1ad840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f1cbc10a5d0 0x7f1cbc1ad840 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cbbfff700 1 -- 172.21.15.50:0/517851963 reap_dead start 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc10a460 osd=2 1 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.3 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f1cbc1aae80 msgr2=0x7f1cbc1ab260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f1cbc1aae80 0x7f1cbc1ab260 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc02e240 osd=3 1 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.4 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1cbc1039a0 msgr2=0x7f1cbc103e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1cbc1039a0 0x7f1cbc103e00 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0220 osd=4 1 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.5 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1cbc1b6fd0 msgr2=0x7f1cbc1b93c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1cbc1b6fd0 0x7f1cbc1b93c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b0390 osd=5 1 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.6 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f1cbc1bb810 msgr2=0x7f1cbc1bbc50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f1cbc1bb810 0x7f1cbc1bbc50 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1b9900 osd=6 1 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter close_session for osd.7 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f1cbc1ba130 msgr2=0x7f1cbc1ba590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f1cbc1ba130 0x7f1cbc1ba590 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter put_session s=0x7f1cbc1badf0 osd=7 1 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 20 client.14769.objecter shutdown clearing up homeless session... 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 10 client.14769.objecter successfully canceled tick 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1ca4071e50 msgr2=0x7f1ca4074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1ca4071e50 0x7f1ca4074310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00b730 tx=0x7f1cac005e80 comp rx=0 tx=0).stop 2024-04-26T16:22:36.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc06c360 msgr2=0x7f1cbc199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 --2- 172.21.15.50:0/517851963 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1cbc06c360 0x7f1cbc199790 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000b530 tx=0x7f1cb000b8f0 comp rx=0 tx=0).stop 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cbbfff700 1 -- 172.21.15.50:0/517851963 reap_dead start 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cbbfff700 1 -- 172.21.15.50:0/517851963 reap_dead start 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 shutdown_connections 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 >> 172.21.15.50:0/517851963 conn(0x7f1cbc079290 msgr2=0x7f1cbc100240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 shutdown_connections 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 -- 172.21.15.50:0/517851963 wait complete. 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:36.830+0000 7f1cc21f9700 1 librados: shutdown 2024-04-26T16:22:36.828 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:22:36.829 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:22:36.868 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:22:36.869 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:22:36.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:36.949 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:36.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:37.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 -- 172.21.15.50:0/3585695063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898103e80 msgr2=0x7f3898110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 --2- 172.21.15.50:0/3585695063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898103e80 0x7f3898110330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3890009f90 tx=0x7f389000b330 comp rx=0 tx=0).stop 2024-04-26T16:22:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 -- 172.21.15.50:0/3585695063 shutdown_connections 2024-04-26T16:22:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 --2- 172.21.15.50:0/3585695063 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898109560 0x7f3898109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 --2- 172.21.15.50:0/3585695063 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898103e80 0x7f3898110330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 --2- 172.21.15.50:0/3585695063 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38981034c0 0x7f3898103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 -- 172.21.15.50:0/3585695063 >> 172.21.15.50:0/3585695063 conn(0x7f389807a5d0 msgr2=0x7f389807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 -- 172.21.15.50:0/3585695063 shutdown_connections 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 -- 172.21.15.50:0/3585695063 wait complete. 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 librados: starting msgr at 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.058+0000 7f389fe4d700 1 librados: starting objecter 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 Processor -- start 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- start start 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 librados: setting wanted keys 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 librados: calling monclient init 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38981034c0 0x7f3898199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898103e80 0x7f3898199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:37.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898109560 0x7f389819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3898112b40 con 0x7f3898103e80 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f38981129c0 con 0x7f3898109560 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3898112cc0 con 0x7f38981034c0 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898103e80 0x7f3898199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898103e80 0x7f3898199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35316/0 (socket says 172.21.15.50:35316) 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 -- 172.21.15.50:0/224281057 learned_addr learned my addr 172.21.15.50:0/224281057 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389dbe9700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38981034c0 0x7f3898199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389e3ea700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898109560 0x7f389819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 -- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38981034c0 msgr2=0x7f3898199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38981034c0 0x7f3898199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 -- 172.21.15.50:0/224281057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898109560 msgr2=0x7f389819d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898109560 0x7f389819d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 -- 172.21.15.50:0/224281057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389819e110 con 0x7f3898103e80 2024-04-26T16:22:37.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389e3ea700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898109560 0x7f389819d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389d3e8700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898103e80 0x7f3898199640 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f388800ec10 tx=0x7f388800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 10 client.?.objecter ms_handle_connect 0x7f3898103e80 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f388800cd40 con 0x7f3898103e80 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3888004480 con 0x7f3898103e80 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888010430 con 0x7f3898103e80 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389819e400 con 0x7f3898103e80 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 10 client.14775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3898075650 con 0x7f3898103e80 2024-04-26T16:22:37.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 librados: init done 2024-04-26T16:22:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 10 client.14775.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 15 client.14775.objecter _session_command_op_assign -1 1 2024-04-26T16:22:37.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 20 client.14775.objecter _get_session osd=-1 returning homeless 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 10 client.14775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3898104bd0 con 0x7f3898103e80 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3888003ef0 con 0x7f3898103e80 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(132..132 src has 77..132) ==== 7450+0+0 (secure 0 0 0) 0x7f3888091600 con 0x7f3898103e80 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 10 client.14775.objecter ms_dispatch 0x7f3898001550 osd_map(132..132 src has 77..132) 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 3 client.14775.objecter handle_osd_map got epochs [132,132] > 0 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.062+0000 7f388effd700 3 client.14775.objecter handle_osd_map decoding full epoch 132 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 10 client.14775.objecter checking command 1 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f388407bdb0 0x7f388407e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter _get_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 15 client.14775.objecter _session_command_op_remove -1 1 2024-04-26T16:22:37.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter _get_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter get_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 15 client.14775.objecter _session_command_op_assign 2 1 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 4 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 10 client.14775.objecter _send_command 1 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f388407e8e0 con 0x7f388407bdb0 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter dump_active .. 0 homeless 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=132) ==== 24+0+0 (secure 0 0 0) 0x7f3888098050 con 0x7f3898103e80 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f389dbe9700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f389e3ea700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f388407bdb0 0x7f388407e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 10 client.14775.objecter ms_handle_connect 0x7f388407bdb0 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f389e3ea700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f388407bdb0 0x7f388407e210 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f389dbe9700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38940045b0 tx=0x7f389400b3c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:37.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 10 client.14775.objecter ms_handle_connect 0x7f3884071f80 2024-04-26T16:22:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f388407e8e0 con 0x7f388407bdb0 2024-04-26T16:22:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 10 client.14775.objecter ms_dispatch 0x7f3898001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 10 client.14775.objecter _finish_command 1 = system:0 2024-04-26T16:22:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 4 2024-04-26T16:22:37.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.066+0000 7f388effd700 15 client.14775.objecter _session_command_op_remove 2 1 2024-04-26T16:22:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 10 client.14775.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:22:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 15 client.14775.objecter _session_command_op_assign -1 2 2024-04-26T16:22:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 20 client.14775.objecter _get_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 20 client.14775.objecter _get_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 15 client.14775.objecter _session_command_op_remove -1 2 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 20 client.14775.objecter get_session s=0x7f388407b6e0 osd=2 3 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 15 client.14775.objecter _session_command_op_assign 2 2 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 4 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 10 client.14775.objecter _send_command 2 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.086+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f389806a110 con 0x7f388407bdb0 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f388effd700 1 -- 172.21.15.50:0/224281057 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f389806a110 con 0x7f388407bdb0 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f388effd700 10 client.14775.objecter ms_dispatch 0x7f3898001550 command_reply(tid 2: 0 ) 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f388effd700 10 client.14775.objecter _finish_command 2 = system:0 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f388effd700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 4 2024-04-26T16:22:37.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f388effd700 15 client.14775.objecter _session_command_op_remove 2 2 2024-04-26T16:22:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 10 librados: watch_flush enter 2024-04-26T16:22:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 10 librados: watch_flush exit 2024-04-26T16:22:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 10 client.14775.objecter close_session for osd.2 2024-04-26T16:22:37.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f388407bdb0 msgr2=0x7f388407e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f388407bdb0 0x7f388407e210 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 20 client.14775.objecter put_session s=0x7f388407b6e0 osd=2 1 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 20 client.14775.objecter shutdown clearing up homeless session... 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 10 client.14775.objecter successfully canceled tick 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 msgr2=0x7f3884074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38940045b0 tx=0x7f389400b3c0 comp rx=0 tx=0).stop 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898103e80 msgr2=0x7f3898199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 --2- 172.21.15.50:0/224281057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898103e80 0x7f3898199640 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f388800ec10 tx=0x7f388800ef20 comp rx=0 tx=0).stop 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389e3ea700 1 -- 172.21.15.50:0/224281057 reap_dead start 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 shutdown_connections 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 >> 172.21.15.50:0/224281057 conn(0x7f389807a5d0 msgr2=0x7f3898101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:37.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 shutdown_connections 2024-04-26T16:22:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 -- 172.21.15.50:0/224281057 wait complete. 2024-04-26T16:22:37.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:37.090+0000 7f389fe4d700 1 librados: shutdown 2024-04-26T16:22:37.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:37.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:37.216 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:37.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:37.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:37.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:37.468 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:37.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:37.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[29234]: audit 2024-04-26T16:22:36.556534+0000 mgr.y (mgr.24401) 190 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:37.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[29234]: audit 2024-04-26T16:22:36.556534+0000 mgr.y (mgr.24401) 190 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:37.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[29234]: audit 2024-04-26T16:22:36.631614+0000 mon.a (mon.0) 954 : audit [DBG] from='client.? 172.21.15.50:0/517851963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:37.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[29234]: audit 2024-04-26T16:22:36.631614+0000 mon.a (mon.0) 954 : audit [DBG] from='client.? 172.21.15.50:0/517851963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:37.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[21463]: audit 2024-04-26T16:22:36.556534+0000 mgr.y (mgr.24401) 190 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:37.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[21463]: audit 2024-04-26T16:22:36.556534+0000 mgr.y (mgr.24401) 190 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:37.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[21463]: audit 2024-04-26T16:22:36.631614+0000 mon.a (mon.0) 954 : audit [DBG] from='client.? 172.21.15.50:0/517851963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:37.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:37 smithi050 bash[21463]: audit 2024-04-26T16:22:36.631614+0000 mon.a (mon.0) 954 : audit [DBG] from='client.? 172.21.15.50:0/517851963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:37.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:37.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:37.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:37 smithi190 bash[23868]: audit 2024-04-26T16:22:36.556534+0000 mgr.y (mgr.24401) 190 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:37.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:37 smithi190 bash[23868]: audit 2024-04-26T16:22:36.556534+0000 mgr.y (mgr.24401) 190 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:37.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:37 smithi190 bash[23868]: audit 2024-04-26T16:22:36.631614+0000 mon.a (mon.0) 954 : audit [DBG] from='client.? 172.21.15.50:0/517851963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:37.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:37 smithi190 bash[23868]: audit 2024-04-26T16:22:36.631614+0000 mon.a (mon.0) 954 : audit [DBG] from='client.? 172.21.15.50:0/517851963' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:22:37.726 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:37.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:37.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:37.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:37.984 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:37.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:38.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:38.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:38.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:38.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:38.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:38.369 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:38.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:38.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:38.500 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:38.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:38.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:38.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:38.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:38.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:38.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:38 smithi050 bash[29234]: cluster 2024-04-26T16:22:37.453984+0000 mgr.y (mgr.24401) 191 : cluster [DBG] pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:38.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:38 smithi050 bash[29234]: cluster 2024-04-26T16:22:37.453984+0000 mgr.y (mgr.24401) 191 : cluster [DBG] pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:38.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:38 smithi050 bash[21463]: cluster 2024-04-26T16:22:37.453984+0000 mgr.y (mgr.24401) 191 : cluster [DBG] pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:38.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:38 smithi050 bash[21463]: cluster 2024-04-26T16:22:37.453984+0000 mgr.y (mgr.24401) 191 : cluster [DBG] pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:38.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:38 smithi190 bash[23868]: cluster 2024-04-26T16:22:37.453984+0000 mgr.y (mgr.24401) 191 : cluster [DBG] pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:38.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:38 smithi190 bash[23868]: cluster 2024-04-26T16:22:37.453984+0000 mgr.y (mgr.24401) 191 : cluster [DBG] pgmap v161: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:38.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:38.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:38.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:38.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:39.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:39.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:39.137 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:39.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:39.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:39.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:39.396 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:39.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:39.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:39.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:39.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:39.781 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:39.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:39.912 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:39.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:40.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:40.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:40.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:40.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:40.304 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:40.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:40.433 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:40.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:40 smithi050 bash[21463]: cluster 2024-04-26T16:22:39.454609+0000 mgr.y (mgr.24401) 192 : cluster [DBG] pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:40.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:40 smithi050 bash[21463]: cluster 2024-04-26T16:22:39.454609+0000 mgr.y (mgr.24401) 192 : cluster [DBG] pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:40.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:40.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:40 smithi190 bash[23868]: cluster 2024-04-26T16:22:39.454609+0000 mgr.y (mgr.24401) 192 : cluster [DBG] pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:40.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:40 smithi190 bash[23868]: cluster 2024-04-26T16:22:39.454609+0000 mgr.y (mgr.24401) 192 : cluster [DBG] pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:40.564 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:40.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:40.693 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:40.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:40.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:40 smithi050 bash[29234]: cluster 2024-04-26T16:22:39.454609+0000 mgr.y (mgr.24401) 192 : cluster [DBG] pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:40.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:40 smithi050 bash[29234]: cluster 2024-04-26T16:22:39.454609+0000 mgr.y (mgr.24401) 192 : cluster [DBG] pgmap v162: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:40.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:40.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:40.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:41.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:41.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:41.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:41.200 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:41.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:41.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:41.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:41.462 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:41.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:41.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:41 smithi190 bash[23868]: audit 2024-04-26T16:22:40.923213+0000 mon.c (mon.2) 90 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:41.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:41 smithi190 bash[23868]: audit 2024-04-26T16:22:40.923213+0000 mon.c (mon.2) 90 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:41.592 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:41.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:41.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:41.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:41.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:41.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:41 smithi050 bash[29234]: audit 2024-04-26T16:22:40.923213+0000 mon.c (mon.2) 90 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:41 smithi050 bash[29234]: audit 2024-04-26T16:22:40.923213+0000 mon.c (mon.2) 90 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:41 smithi050 bash[21463]: audit 2024-04-26T16:22:40.923213+0000 mon.c (mon.2) 90 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:41 smithi050 bash[21463]: audit 2024-04-26T16:22:40.923213+0000 mon.c (mon.2) 90 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:41.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:41.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:41.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:42.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:42.114 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:42.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:42.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:42.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-26T16:22:42.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-04-26T16:22:42.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:42.372 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:42.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:42.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:42.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:42.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:42.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:42.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:42.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:42 smithi190 bash[23868]: cluster 2024-04-26T16:22:41.456017+0000 mgr.y (mgr.24401) 193 : cluster [DBG] pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:42.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:42 smithi190 bash[23868]: cluster 2024-04-26T16:22:41.456017+0000 mgr.y (mgr.24401) 193 : cluster [DBG] pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:42.757 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:42.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:42.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:42 smithi050 bash[21463]: cluster 2024-04-26T16:22:41.456017+0000 mgr.y (mgr.24401) 193 : cluster [DBG] pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:42.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:42 smithi050 bash[21463]: cluster 2024-04-26T16:22:41.456017+0000 mgr.y (mgr.24401) 193 : cluster [DBG] pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:42.871 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:42 smithi050 bash[29234]: cluster 2024-04-26T16:22:41.456017+0000 mgr.y (mgr.24401) 193 : cluster [DBG] pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:42.871 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:42 smithi050 bash[29234]: cluster 2024-04-26T16:22:41.456017+0000 mgr.y (mgr.24401) 193 : cluster [DBG] pgmap v163: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:42.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:42.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:42.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:43.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:43.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:43.140 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:43.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:43.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:43.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:43.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:43.397 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:43.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:43.529 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:43.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:43.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:43.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:43.779 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:43.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:43.908 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:43.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:44.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:44.038 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:44.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:44.169 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:44.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:44.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:22:44.301 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:44.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:44.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:44.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:44.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:44.538 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:22:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:22:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:22:44.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:44 smithi050 bash[29234]: cluster 2024-04-26T16:22:43.456816+0000 mgr.y (mgr.24401) 194 : cluster [DBG] pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-04-26T16:22:44.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:44 smithi050 bash[29234]: cluster 2024-04-26T16:22:43.456816+0000 mgr.y (mgr.24401) 194 : cluster [DBG] pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-04-26T16:22:44.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:44 smithi050 bash[21463]: cluster 2024-04-26T16:22:43.456816+0000 mgr.y (mgr.24401) 194 : cluster [DBG] pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-04-26T16:22:44.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:44 smithi050 bash[21463]: cluster 2024-04-26T16:22:43.456816+0000 mgr.y (mgr.24401) 194 : cluster [DBG] pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-04-26T16:22:44.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:44.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:44 smithi190 bash[23868]: cluster 2024-04-26T16:22:43.456816+0000 mgr.y (mgr.24401) 194 : cluster [DBG] pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-04-26T16:22:44.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:44 smithi190 bash[23868]: cluster 2024-04-26T16:22:43.456816+0000 mgr.y (mgr.24401) 194 : cluster [DBG] pgmap v164: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-04-26T16:22:44.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:44.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:44.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 -- 172.21.15.50:0/353341685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4108380 msgr2=0x7f5bb4108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:44.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 --2- 172.21.15.50:0/353341685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4108380 0x7f5bb4108760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009950 tx=0x7f5bb0009c60 comp rx=0 tx=0).stop 2024-04-26T16:22:44.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 -- 172.21.15.50:0/353341685 shutdown_connections 2024-04-26T16:22:44.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 --2- 172.21.15.50:0/353341685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4102cd0 0x7f5bb410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:44.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 --2- 172.21.15.50:0/353341685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4108380 0x7f5bb4108760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 --2- 172.21.15.50:0/353341685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 0x7f5bb4102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 -- 172.21.15.50:0/353341685 >> 172.21.15.50:0/353341685 conn(0x7f5bb4079290 msgr2=0x7f5bb40796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 -- 172.21.15.50:0/353341685 shutdown_connections 2024-04-26T16:22:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 -- 172.21.15.50:0/353341685 wait complete. 2024-04-26T16:22:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 librados: starting msgr at 2024-04-26T16:22:44.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 librados: starting objecter 2024-04-26T16:22:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 Processor -- start 2024-04-26T16:22:44.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 -- start start 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 librados: setting wanted keys 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.682+0000 7f5bbc40e700 1 librados: calling monclient init 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 0x7f5bb419b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 0x7f5bb419b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4108380 0x7f5bb419fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb41108e0 con 0x7f5bb4108380 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb4110760 con 0x7f5bb4102310 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb4110a60 con 0x7f5bb4102cd0 2024-04-26T16:22:44.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bb99a9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 0x7f5bb419b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4108380 0x7f5bb419fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bb99a9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 0x7f5bb419b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35036/0 (socket says 172.21.15.50:35036) 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bb99a9700 1 -- 172.21.15.50:0/345500985 learned_addr learned my addr 172.21.15.50:0/345500985 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba1aa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 0x7f5bb419b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 -- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 msgr2=0x7f5bb419b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 0x7f5bb419b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 -- 172.21.15.50:0/345500985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 msgr2=0x7f5bb419b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 0x7f5bb419b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 -- 172.21.15.50:0/345500985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb41a0340 con 0x7f5bb4108380 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bb99a9700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 0x7f5bb419b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:22:44.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba1aa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 0x7f5bb419b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba9ab700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4108380 0x7f5bb419fc00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00efc0 tx=0x7f5bac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 10 client.?.objecter ms_handle_connect 0x7f5bb4108380 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0092e0 con 0x7f5bb4108380 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 -- 172.21.15.50:0/345500985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb41a05d0 con 0x7f5bb4108380 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5bac004bb0 con 0x7f5bb4108380 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bac004020 con 0x7f5bb4108380 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 10 client.14781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 -- 172.21.15.50:0/345500985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb41a7eb0 con 0x7f5bb4108380 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bbc40e700 1 librados: init done 2024-04-26T16:22:44.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b9c005120 con 0x7f5bb4108380 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5bac01b030 con 0x7f5bb4108380 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ba0071f80 0x7f5ba0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(132..132 src has 77..132) ==== 7450+0+0 (secure 0 0 0) 0x7f5bac090de0 con 0x7f5bb4108380 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 10 client.14781.objecter ms_dispatch 0x7f5bb4001550 osd_map(132..132 src has 77..132) 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 3 client.14781.objecter handle_osd_map got epochs [132,132] > 0 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bab7fe700 3 client.14781.objecter handle_osd_map decoding full epoch 132 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.686+0000 7f5bba1aa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ba0071f80 0x7f5ba0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.690+0000 7f5bab7fe700 20 client.14781.objecter dump_active .. 0 homeless 2024-04-26T16:22:44.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.690+0000 7f5bab7fe700 10 client.14781.objecter ms_handle_connect 0x7f5ba0071f80 2024-04-26T16:22:44.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.690+0000 7f5bba1aa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ba0071f80 0x7f5ba0074440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0018040 tx=0x7f5bb0005fa0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:44.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.694+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5bac022020 con 0x7f5bb4108380 2024-04-26T16:22:44.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:44.806 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:44.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:44.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:44.934+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f5b9c0056e0 con 0x7f5bb4108380 2024-04-26T16:22:44.938 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:44.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:45.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:45.068 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:45.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:45.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:45.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:45.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:45.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:45.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:45.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:45.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5bab7fe700 1 -- 172.21.15.50:0/345500985 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v133) ==== 91+0+0 (secure 0 0 0) 0x7f5bac010040 con 0x7f5bb4108380 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5ba97fa700 10 librados: watch_flush enter 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5ba97fa700 10 librados: watch_flush exit 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5ba97fa700 20 client.14781.objecter shutdown clearing up homeless session... 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5ba97fa700 10 client.14781.objecter successfully canceled tick 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ba0071f80 msgr2=0x7f5ba0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.534+0000 7f5ba97fa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ba0071f80 0x7f5ba0074440 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0018040 tx=0x7f5bb0005fa0 comp rx=0 tx=0).stop 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.538+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4108380 msgr2=0x7f5bb419fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.538+0000 7f5ba97fa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4108380 0x7f5bb419fc00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00efc0 tx=0x7f5bac00c3a0 comp rx=0 tx=0).stop 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 shutdown_connections 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ba0071f80 0x7f5ba0074440 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bb4102310 0x7f5bb419b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5bb4108380 0x7f5bb419fc00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:45.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 --2- 172.21.15.50:0/345500985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5bb4102cd0 0x7f5bb419b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:45.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 >> 172.21.15.50:0/345500985 conn(0x7f5bb4079290 msgr2=0x7f5bb4100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:45.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 shutdown_connections 2024-04-26T16:22:45.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 -- 172.21.15.50:0/345500985 wait complete. 2024-04-26T16:22:45.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:45.542+0000 7f5ba97fa700 1 librados: shutdown 2024-04-26T16:22:45.536 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:22:45.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:45.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:45.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:45.709 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:45.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:45 smithi050 bash[21463]: audit 2024-04-26T16:22:44.936406+0000 mon.a (mon.0) 955 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:22:45.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:45 smithi050 bash[21463]: audit 2024-04-26T16:22:44.936406+0000 mon.a (mon.0) 955 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:22:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:45 smithi050 bash[29234]: audit 2024-04-26T16:22:44.936406+0000 mon.a (mon.0) 955 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:22:45.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:45 smithi050 bash[29234]: audit 2024-04-26T16:22:44.936406+0000 mon.a (mon.0) 955 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:22:45.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:45.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:45 smithi190 bash[23868]: audit 2024-04-26T16:22:44.936406+0000 mon.a (mon.0) 955 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:22:45.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:45 smithi190 bash[23868]: audit 2024-04-26T16:22:44.936406+0000 mon.a (mon.0) 955 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:22:45.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:45.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:45.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:45.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:46.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:46.096 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:46.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:46.226 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:46.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:46.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:46.358 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:46.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:46.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:46.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:46.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:46.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:46.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: cluster 2024-04-26T16:22:45.457342+0000 mgr.y (mgr.24401) 195 : cluster [DBG] pgmap v165: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: cluster 2024-04-26T16:22:45.457342+0000 mgr.y (mgr.24401) 195 : cluster [DBG] pgmap v165: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: cluster 2024-04-26T16:22:45.527838+0000 mon.a (mon.0) 956 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: cluster 2024-04-26T16:22:45.527838+0000 mon.a (mon.0) 956 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: audit 2024-04-26T16:22:45.535456+0000 mon.a (mon.0) 957 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: audit 2024-04-26T16:22:45.535456+0000 mon.a (mon.0) 957 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: cluster 2024-04-26T16:22:45.539136+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[29234]: cluster 2024-04-26T16:22:45.539136+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-04-26T16:22:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: cluster 2024-04-26T16:22:45.457342+0000 mgr.y (mgr.24401) 195 : cluster [DBG] pgmap v165: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: cluster 2024-04-26T16:22:45.457342+0000 mgr.y (mgr.24401) 195 : cluster [DBG] pgmap v165: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: cluster 2024-04-26T16:22:45.527838+0000 mon.a (mon.0) 956 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: cluster 2024-04-26T16:22:45.527838+0000 mon.a (mon.0) 956 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: audit 2024-04-26T16:22:45.535456+0000 mon.a (mon.0) 957 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: audit 2024-04-26T16:22:45.535456+0000 mon.a (mon.0) 957 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: cluster 2024-04-26T16:22:45.539136+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-04-26T16:22:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:46 smithi050 bash[21463]: cluster 2024-04-26T16:22:45.539136+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-04-26T16:22:46.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:46.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:22:46.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: cluster 2024-04-26T16:22:45.457342+0000 mgr.y (mgr.24401) 195 : cluster [DBG] pgmap v165: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:22:46.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: cluster 2024-04-26T16:22:45.457342+0000 mgr.y (mgr.24401) 195 : cluster [DBG] pgmap v165: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:22:46.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: cluster 2024-04-26T16:22:45.527838+0000 mon.a (mon.0) 956 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:46.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: cluster 2024-04-26T16:22:45.527838+0000 mon.a (mon.0) 956 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:22:46.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: audit 2024-04-26T16:22:45.535456+0000 mon.a (mon.0) 957 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:22:46.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: audit 2024-04-26T16:22:45.535456+0000 mon.a (mon.0) 957 : audit [INF] from='client.? 172.21.15.50:0/345500985' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:22:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: cluster 2024-04-26T16:22:45.539136+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-04-26T16:22:46.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:46 smithi190 bash[23868]: cluster 2024-04-26T16:22:45.539136+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2024-04-26T16:22:46.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:46.877 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:46.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:46.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:47.010 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:47.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:47.139 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:47.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:47.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:47.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:47.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:47.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:47.511 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:47.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:47 smithi050 bash[21463]: audit 2024-04-26T16:22:46.567508+0000 mgr.y (mgr.24401) 196 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:47.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:47 smithi050 bash[21463]: audit 2024-04-26T16:22:46.567508+0000 mgr.y (mgr.24401) 196 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:47.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:47.636 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:47.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:47.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:47.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:47 smithi190 bash[23868]: audit 2024-04-26T16:22:46.567508+0000 mgr.y (mgr.24401) 196 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:47.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:47 smithi190 bash[23868]: audit 2024-04-26T16:22:46.567508+0000 mgr.y (mgr.24401) 196 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:47.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:47.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:47 smithi050 bash[29234]: audit 2024-04-26T16:22:46.567508+0000 mgr.y (mgr.24401) 196 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:47.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:47 smithi050 bash[29234]: audit 2024-04-26T16:22:46.567508+0000 mgr.y (mgr.24401) 196 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:47.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:47.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:47.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:48.007 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:48.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:48.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:48.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:48.239 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:48.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:48.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:48.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:48.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:48.530 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:48.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:48.661 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:48.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:46.942992+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:46.942992+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:46.944167+0000 osd.1 (osd.1) 120 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:46.944167+0000 osd.1 (osd.1) 120 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.241009+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.241009+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.242540+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.242540+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.304203+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:22:48.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.304203+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.305773+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.305773+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.333314+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.333314+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.335299+0000 osd.7 (osd.7) 136 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.335299+0000 osd.7 (osd.7) 136 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.457987+0000 mgr.y (mgr.24401) 197 : cluster [DBG] pgmap v167: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.457987+0000 mgr.y (mgr.24401) 197 : cluster [DBG] pgmap v167: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:46.942992+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:46.942992+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:46.944167+0000 osd.1 (osd.1) 120 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:22:48.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:46.944167+0000 osd.1 (osd.1) 120 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.241009+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.241009+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.242540+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.242540+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.304203+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.304203+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.305773+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.305773+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.333314+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.333314+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.335299+0000 osd.7 (osd.7) 136 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.335299+0000 osd.7 (osd.7) 136 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.457987+0000 mgr.y (mgr.24401) 197 : cluster [DBG] pgmap v167: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:48.775 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:48 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.457987+0000 mgr.y (mgr.24401) 197 : cluster [DBG] pgmap v167: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:48.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:48.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:46.942992+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:22:48.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:46.942992+0000 osd.1 (osd.1) 119 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:22:48.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:46.944167+0000 osd.1 (osd.1) 120 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:46.944167+0000 osd.1 (osd.1) 120 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.241009+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.241009+0000 osd.0 (osd.0) 117 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.242540+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.242540+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.304203+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.304203+0000 osd.3 (osd.3) 127 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.305773+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.305773+0000 osd.3 (osd.3) 128 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.333314+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.333314+0000 osd.7 (osd.7) 135 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.335299+0000 osd.7 (osd.7) 136 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.335299+0000 osd.7 (osd.7) 136 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:22:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.457987+0000 mgr.y (mgr.24401) 197 : cluster [DBG] pgmap v167: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:48 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.457987+0000 mgr.y (mgr.24401) 197 : cluster [DBG] pgmap v167: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:48.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:48.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:48.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:49.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:49.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:49.164 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:49.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:49.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:49.421 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:49.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:49.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:49.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:49.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:49.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.968627+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.968627+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.970211+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:47.970211+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.249379+0000 osd.1 (osd.1) 123 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.249379+0000 osd.1 (osd.1) 123 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.249467+0000 osd.1 (osd.1) 124 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.249467+0000 osd.1 (osd.1) 124 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.426937+0000 osd.4 (osd.4) 101 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.426937+0000 osd.4 (osd.4) 101 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.429456+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.429456+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:22:49.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.545439+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[21463]: cluster 2024-04-26T16:22:48.545439+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.968627+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.968627+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.970211+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:47.970211+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.249379+0000 osd.1 (osd.1) 123 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.249379+0000 osd.1 (osd.1) 123 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.249467+0000 osd.1 (osd.1) 124 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.249467+0000 osd.1 (osd.1) 124 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.426937+0000 osd.4 (osd.4) 101 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.426937+0000 osd.4 (osd.4) 101 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:22:49.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.429456+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:22:49.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.429456+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:22:49.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.545439+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-04-26T16:22:49.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:49 smithi050 bash[29234]: cluster 2024-04-26T16:22:48.545439+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-04-26T16:22:49.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:49.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.968627+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.968627+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.970211+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:47.970211+0000 osd.1 (osd.1) 122 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.249379+0000 osd.1 (osd.1) 123 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.249379+0000 osd.1 (osd.1) 123 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.249467+0000 osd.1 (osd.1) 124 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:49.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.249467+0000 osd.1 (osd.1) 124 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.426937+0000 osd.4 (osd.4) 101 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:22:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.426937+0000 osd.4 (osd.4) 101 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:22:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.429456+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:22:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.429456+0000 osd.4 (osd.4) 102 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:22:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.545439+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-04-26T16:22:49.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:49 smithi190 bash[23868]: cluster 2024-04-26T16:22:48.545439+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2024-04-26T16:22:49.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:49.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:50.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:50.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:50.262 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:50.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.295 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:50.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:50.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:50.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:50.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.003304+0000 osd.1 (osd.1) 125 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.003304+0000 osd.1 (osd.1) 125 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.005016+0000 osd.1 (osd.1) 126 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.005016+0000 osd.1 (osd.1) 126 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.268637+0000 osd.3 (osd.3) 129 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.268637+0000 osd.3 (osd.3) 129 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.271855+0000 osd.3 (osd.3) 130 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.271855+0000 osd.3 (osd.3) 130 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.282449+0000 osd.2 (osd.2) 85 : cluster [DBG] 5.b scrub starts 2024-04-26T16:22:50.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.282449+0000 osd.2 (osd.2) 85 : cluster [DBG] 5.b scrub starts 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.284159+0000 osd.2 (osd.2) 86 : cluster [DBG] 5.b scrub ok 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.284159+0000 osd.2 (osd.2) 86 : cluster [DBG] 5.b scrub ok 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.458382+0000 mgr.y (mgr.24401) 198 : cluster [DBG] pgmap v169: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[29234]: cluster 2024-04-26T16:22:49.458382+0000 mgr.y (mgr.24401) 198 : cluster [DBG] pgmap v169: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.003304+0000 osd.1 (osd.1) 125 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.003304+0000 osd.1 (osd.1) 125 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.005016+0000 osd.1 (osd.1) 126 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.005016+0000 osd.1 (osd.1) 126 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.268637+0000 osd.3 (osd.3) 129 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.268637+0000 osd.3 (osd.3) 129 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.271855+0000 osd.3 (osd.3) 130 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:22:50.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.271855+0000 osd.3 (osd.3) 130 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:22:50.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.282449+0000 osd.2 (osd.2) 85 : cluster [DBG] 5.b scrub starts 2024-04-26T16:22:50.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.282449+0000 osd.2 (osd.2) 85 : cluster [DBG] 5.b scrub starts 2024-04-26T16:22:50.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.284159+0000 osd.2 (osd.2) 86 : cluster [DBG] 5.b scrub ok 2024-04-26T16:22:50.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.284159+0000 osd.2 (osd.2) 86 : cluster [DBG] 5.b scrub ok 2024-04-26T16:22:50.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.458382+0000 mgr.y (mgr.24401) 198 : cluster [DBG] pgmap v169: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:50.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:50 smithi050 bash[21463]: cluster 2024-04-26T16:22:49.458382+0000 mgr.y (mgr.24401) 198 : cluster [DBG] pgmap v169: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:50.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:50.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:50.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.003304+0000 osd.1 (osd.1) 125 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.003304+0000 osd.1 (osd.1) 125 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.005016+0000 osd.1 (osd.1) 126 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.005016+0000 osd.1 (osd.1) 126 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.268637+0000 osd.3 (osd.3) 129 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.268637+0000 osd.3 (osd.3) 129 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.271855+0000 osd.3 (osd.3) 130 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:22:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.271855+0000 osd.3 (osd.3) 130 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:22:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.282449+0000 osd.2 (osd.2) 85 : cluster [DBG] 5.b scrub starts 2024-04-26T16:22:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.282449+0000 osd.2 (osd.2) 85 : cluster [DBG] 5.b scrub starts 2024-04-26T16:22:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.284159+0000 osd.2 (osd.2) 86 : cluster [DBG] 5.b scrub ok 2024-04-26T16:22:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.284159+0000 osd.2 (osd.2) 86 : cluster [DBG] 5.b scrub ok 2024-04-26T16:22:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.458382+0000 mgr.y (mgr.24401) 198 : cluster [DBG] pgmap v169: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:50 smithi190 bash[23868]: cluster 2024-04-26T16:22:49.458382+0000 mgr.y (mgr.24401) 198 : cluster [DBG] pgmap v169: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:22:50.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:50.923 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:50.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:51.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:51.047 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:51.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:51.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:51.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:51.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:51.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:51.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:51.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:51.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:51.544 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:51.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.149935+0000 osd.0 (osd.0) 119 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.149935+0000 osd.0 (osd.0) 119 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.152122+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.152122+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.267329+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.c scrub starts 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.267329+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.c scrub starts 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.268840+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.c scrub ok 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.268840+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.c scrub ok 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.390165+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:22:51.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.390165+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:22:51.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:51.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:51.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:51.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.149935+0000 osd.0 (osd.0) 119 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:22:51.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.149935+0000 osd.0 (osd.0) 119 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.152122+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.152122+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.267329+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.c scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.267329+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.c scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.268840+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.c scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.268840+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.c scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.390165+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.390165+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.394260+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:50.394260+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.036810+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.036810+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.039401+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:22:51.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:51 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.039401+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:22:51.800 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.394260+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:22:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:50.394260+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:22:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.036810+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:22:51.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.036810+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.039401+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.039401+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.149935+0000 osd.0 (osd.0) 119 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.149935+0000 osd.0 (osd.0) 119 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.152122+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.152122+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.267329+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.c scrub starts 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.267329+0000 osd.2 (osd.2) 87 : cluster [DBG] 2.c scrub starts 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.268840+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.c scrub ok 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.268840+0000 osd.2 (osd.2) 88 : cluster [DBG] 2.c scrub ok 2024-04-26T16:22:51.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.390165+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.390165+0000 osd.4 (osd.4) 103 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.394260+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:50.394260+0000 osd.4 (osd.4) 104 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.036810+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.036810+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.039401+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:22:51.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:51 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.039401+0000 osd.6 (osd.6) 70 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:22:51.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:51.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:51.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:52.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:52.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:52.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:52.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:52.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:52.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:52.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:52.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:52.429 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:52.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:52.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.143598+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:22:52.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.143598+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:22:52.555 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.145725+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.145725+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.296863+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.296863+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.298666+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.298666+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.393245+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.393245+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.395435+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.395435+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.459248+0000 mgr.y (mgr.24401) 199 : cluster [DBG] pgmap v170: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.459248+0000 mgr.y (mgr.24401) 199 : cluster [DBG] pgmap v170: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:52.556 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.075063+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.075063+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.077303+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.077303+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.143598+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.143598+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.145725+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.145725+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.296863+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.296863+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-04-26T16:22:52.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.298666+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.298666+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.393245+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.393245+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.395435+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.395435+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.459248+0000 mgr.y (mgr.24401) 199 : cluster [DBG] pgmap v170: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.459248+0000 mgr.y (mgr.24401) 199 : cluster [DBG] pgmap v170: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.075063+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.075063+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.077303+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:22:52.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:52 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.077303+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:22:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.560 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:52.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:52.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:52.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:52.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:52.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.143598+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.143598+0000 osd.0 (osd.0) 121 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.145725+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.145725+0000 osd.0 (osd.0) 122 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.296863+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.296863+0000 osd.2 (osd.2) 89 : cluster [DBG] 5.a scrub starts 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.298666+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-04-26T16:22:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.298666+0000 osd.2 (osd.2) 90 : cluster [DBG] 5.a scrub ok 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.393245+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.393245+0000 osd.4 (osd.4) 105 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.395435+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.395435+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.459248+0000 mgr.y (mgr.24401) 199 : cluster [DBG] pgmap v170: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.459248+0000 mgr.y (mgr.24401) 199 : cluster [DBG] pgmap v170: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.075063+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.075063+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.077303+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:22:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:52 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.077303+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:22:52.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:52.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:52.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:52.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:53.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:53.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:53.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:53.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:53.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:53.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:53.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:53.456 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:53.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.913992+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:22:53.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.913992+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:22:53.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.917121+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:22:53.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:51.917121+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:22:53.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.913992+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:22:53.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.913992+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:22:53.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.917121+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:22:53.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:51.917121+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:22:53.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.913992+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.913992+0000 osd.1 (osd.1) 127 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.917121+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:51.917121+0000 osd.1 (osd.1) 128 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.250964+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.250964+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.253284+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:22:53.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:53 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.253284+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:22:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:53.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:53.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:53.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:53.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:53.912 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.250964+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.250964+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.253284+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.253284+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.250964+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.250964+0000 osd.3 (osd.3) 131 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.253284+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:22:53.913 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:53 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.253284+0000 osd.3 (osd.3) 132 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:22:53.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:53.974 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:53.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:54.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:54.105 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:54.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:54.234 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:54.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:54.364 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:54.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:54.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:54.596 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:54.656 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:22:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:22:54.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.864805+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.c scrub starts 2024-04-26T16:22:54.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.864805+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.c scrub starts 2024-04-26T16:22:54.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.866247+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.c scrub ok 2024-04-26T16:22:54.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:52.866247+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.c scrub ok 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.243633+0000 osd.2 (osd.2) 91 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.243633+0000 osd.2 (osd.2) 91 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.245323+0000 osd.2 (osd.2) 92 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.245323+0000 osd.2 (osd.2) 92 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.459681+0000 mgr.y (mgr.24401) 200 : cluster [DBG] pgmap v171: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.459681+0000 mgr.y (mgr.24401) 200 : cluster [DBG] pgmap v171: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.460943+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.460943+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.463037+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.463037+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.087655+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.087655+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.089754+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:22:54.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.089754+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.864805+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.c scrub starts 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.864805+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.c scrub starts 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.866247+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.c scrub ok 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:52.866247+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.c scrub ok 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.243633+0000 osd.2 (osd.2) 91 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.243633+0000 osd.2 (osd.2) 91 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.245323+0000 osd.2 (osd.2) 92 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.245323+0000 osd.2 (osd.2) 92 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.459681+0000 mgr.y (mgr.24401) 200 : cluster [DBG] pgmap v171: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.459681+0000 mgr.y (mgr.24401) 200 : cluster [DBG] pgmap v171: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:54.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.460943+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.460943+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.463037+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.463037+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.087655+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.087655+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.089754+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:22:54.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:54 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.089754+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:22:54.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:54.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.864805+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.c scrub starts 2024-04-26T16:22:54.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.864805+0000 osd.1 (osd.1) 129 : cluster [DBG] 5.c scrub starts 2024-04-26T16:22:54.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.866247+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.c scrub ok 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:52.866247+0000 osd.1 (osd.1) 130 : cluster [DBG] 5.c scrub ok 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.243633+0000 osd.2 (osd.2) 91 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.243633+0000 osd.2 (osd.2) 91 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.245323+0000 osd.2 (osd.2) 92 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.245323+0000 osd.2 (osd.2) 92 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.459681+0000 mgr.y (mgr.24401) 200 : cluster [DBG] pgmap v171: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.459681+0000 mgr.y (mgr.24401) 200 : cluster [DBG] pgmap v171: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.460943+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.460943+0000 osd.4 (osd.4) 107 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.463037+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.463037+0000 osd.4 (osd.4) 108 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.087655+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.087655+0000 osd.6 (osd.6) 73 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:22:54.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.089754+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:22:54.741 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:54 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.089754+0000 osd.6 (osd.6) 74 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:22:54.755 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:54.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:54.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:54.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:54.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:55.014 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:22:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:22:55.117 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:55.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:55.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:55.273 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:55.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:55.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:55.528 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:22:55.528 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:22:55.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:55.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:55.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:55.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.909102+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.909102+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.910573+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:53.910573+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.218162+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.218162+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.221233+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.221233+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.290725+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.290725+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.292584+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.292584+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.409844+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:22:55.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.409844+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:22:55.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.410960+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:22:55.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:54.410960+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:22:55.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.076016+0000 osd.5 (osd.5) 121 : cluster [DBG] 3.c scrub starts 2024-04-26T16:22:55.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.076016+0000 osd.5 (osd.5) 121 : cluster [DBG] 3.c scrub starts 2024-04-26T16:22:55.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.079788+0000 osd.5 (osd.5) 122 : cluster [DBG] 3.c scrub ok 2024-04-26T16:22:55.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.079788+0000 osd.5 (osd.5) 122 : cluster [DBG] 3.c scrub ok 2024-04-26T16:22:55.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.909102+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.909102+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.910573+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:53.910573+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.218162+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.218162+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.221233+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.221233+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.290725+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.290725+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:22:55.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.292584+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.292584+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.409844+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.409844+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.410960+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:54.410960+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.076016+0000 osd.5 (osd.5) 121 : cluster [DBG] 3.c scrub starts 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.076016+0000 osd.5 (osd.5) 121 : cluster [DBG] 3.c scrub starts 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.079788+0000 osd.5 (osd.5) 122 : cluster [DBG] 3.c scrub ok 2024-04-26T16:22:55.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:55 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.079788+0000 osd.5 (osd.5) 122 : cluster [DBG] 3.c scrub ok 2024-04-26T16:22:55.792 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.909102+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.909102+0000 osd.1 (osd.1) 131 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.910573+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:53.910573+0000 osd.1 (osd.1) 132 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.218162+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.218162+0000 osd.0 (osd.0) 123 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.221233+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.221233+0000 osd.0 (osd.0) 124 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.290725+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.290725+0000 osd.3 (osd.3) 133 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:22:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.292584+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.292584+0000 osd.3 (osd.3) 134 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.409844+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.409844+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.410960+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:54.410960+0000 osd.7 (osd.7) 138 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.076016+0000 osd.5 (osd.5) 121 : cluster [DBG] 3.c scrub starts 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.076016+0000 osd.5 (osd.5) 121 : cluster [DBG] 3.c scrub starts 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.079788+0000 osd.5 (osd.5) 122 : cluster [DBG] 3.c scrub ok 2024-04-26T16:22:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:55 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.079788+0000 osd.5 (osd.5) 122 : cluster [DBG] 3.c scrub ok 2024-04-26T16:22:55.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:55.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:55.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:56.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:56.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:56.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:56.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:56.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:56.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:56.309 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:56.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:56.440 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:56.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:56.572 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:56.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 -- 172.21.15.50:0/1108809723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 msgr2=0x7fd354101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 --2- 172.21.15.50:0/1108809723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 0x7fd354101e70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd340009990 tx=0x7fd340009ca0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 -- 172.21.15.50:0/1108809723 shutdown_connections 2024-04-26T16:22:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 --2- 172.21.15.50:0/1108809723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3541023b0 0x7fd3541102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 --2- 172.21.15.50:0/1108809723 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3541094b0 0x7fd354109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 --2- 172.21.15.50:0/1108809723 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 0x7fd354101e70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.669+0000 7fd35b14f700 1 -- 172.21.15.50:0/1108809723 >> 172.21.15.50:0/1108809723 conn(0x7fd354078990 msgr2=0x7fd3540ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- 172.21.15.50:0/1108809723 shutdown_connections 2024-04-26T16:22:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- 172.21.15.50:0/1108809723 wait complete. 2024-04-26T16:22:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 librados: starting msgr at 2024-04-26T16:22:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 librados: starting objecter 2024-04-26T16:22:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 Processor -- start 2024-04-26T16:22:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- start start 2024-04-26T16:22:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 librados: setting wanted keys 2024-04-26T16:22:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 librados: calling monclient init 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 0x7fd3541991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3541023b0 0x7fd354199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3541094b0 0x7fd35419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd354112af0 con 0x7fd354101a10 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd354112970 con 0x7fd3541094b0 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd354112c70 con 0x7fd3541023b0 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd353fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3541023b0 0x7fd354199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 0x7fd3541991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd353fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3541023b0 0x7fd354199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59584/0 (socket says 172.21.15.50:59584) 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd353fff700 1 -- 172.21.15.50:0/906079261 learned_addr learned my addr 172.21.15.50:0/906079261 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 -- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3541023b0 msgr2=0x7fd354199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd3541023b0 0x7fd354199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 -- 172.21.15.50:0/906079261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3541094b0 msgr2=0x7fd35419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd3541094b0 0x7fd35419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 -- 172.21.15.50:0/906079261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35419e1b0 con 0x7fd354101a10 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd358eeb700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 0x7fd3541991f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd348009ef0 tx=0x7fd34800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd351ffb700 10 client.?.objecter ms_handle_connect 0x7fd354101a10 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd351ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348020070 con 0x7fd354101a10 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd34800cc20 con 0x7fd354101a10 2024-04-26T16:22:56.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.677+0000 7fd35b14f700 1 -- 172.21.15.50:0/906079261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35419e4a0 con 0x7fd354101a10 2024-04-26T16:22:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34801b6a0 con 0x7fd354101a10 2024-04-26T16:22:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd35b14f700 10 client.14787.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:22:56.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd35b14f700 1 -- 172.21.15.50:0/906079261 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd354102b20 con 0x7fd354101a10 2024-04-26T16:22:56.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd35b14f700 1 librados: init done 2024-04-26T16:22:56.675 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd348023070 con 0x7fd354101a10 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd344071fd0 0x7fd344074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd353fff700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd344071fd0 0x7fd344074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(134..134 src has 77..134) ==== 7450+0+0 (secure 0 0 0) 0x7fd34800cdd0 con 0x7fd354101a10 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 10 client.14787.objecter ms_dispatch 0x7fd354001550 osd_map(134..134 src has 77..134) 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 3 client.14787.objecter handle_osd_map got epochs [134,134] > 0 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 3 client.14787.objecter handle_osd_map decoding full epoch 134 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd353fff700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd344071fd0 0x7fd344074490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd354108960 tx=0x7fd34000b610 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 20 client.14787.objecter dump_active .. 0 homeless 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 10 client.14787.objecter ms_handle_connect 0x7fd344071fd0 2024-04-26T16:22:56.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 10 client.14787.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 15 client.14787.objecter _session_command_op_assign -1 1 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fd334002ff0 0x7fd3340054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 20 client.14787.objecter _get_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 20 client.14787.objecter _get_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 15 client.14787.objecter _session_command_op_remove -1 1 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 20 client.14787.objecter get_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 15 client.14787.objecter _session_command_op_assign 3 1 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 4 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 10 client.14787.objecter _send_command 1 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd334008530 con 0x7fd334002ff0 2024-04-26T16:22:56.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd3596ec700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fd334002ff0 0x7fd3340054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:22:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd3596ec700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fd334002ff0 0x7fd3340054b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:22:56.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.681+0000 7fd351ffb700 10 client.14787.objecter ms_handle_connect 0x7fd334002ff0 2024-04-26T16:22:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.685+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd334008530 con 0x7fd334002ff0 2024-04-26T16:22:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.685+0000 7fd351ffb700 10 client.14787.objecter ms_dispatch 0x7fd354001550 command_reply(tid 1: 0 ) 2024-04-26T16:22:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.685+0000 7fd351ffb700 10 client.14787.objecter _finish_command 1 = system:0 2024-04-26T16:22:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.685+0000 7fd351ffb700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 4 2024-04-26T16:22:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.685+0000 7fd351ffb700 15 client.14787.objecter _session_command_op_remove 3 1 2024-04-26T16:22:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:56.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.295301+0000 osd.3 (osd.3) 135 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:22:56.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.295301+0000 osd.3 (osd.3) 135 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:22:56.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.296601+0000 osd.3 (osd.3) 136 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.296601+0000 osd.3 (osd.3) 136 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.460111+0000 mgr.y (mgr.24401) 201 : cluster [DBG] pgmap v172: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:55.460111+0000 mgr.y (mgr.24401) 201 : cluster [DBG] pgmap v172: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: audit 2024-04-26T16:22:55.956863+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: audit 2024-04-26T16:22:55.956863+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.057016+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.f scrub starts 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.057016+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.f scrub starts 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.058570+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f scrub ok 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.058570+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f scrub ok 2024-04-26T16:22:56.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.295301+0000 osd.3 (osd.3) 135 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.295301+0000 osd.3 (osd.3) 135 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.296601+0000 osd.3 (osd.3) 136 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.296601+0000 osd.3 (osd.3) 136 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.460111+0000 mgr.y (mgr.24401) 201 : cluster [DBG] pgmap v172: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:55.460111+0000 mgr.y (mgr.24401) 201 : cluster [DBG] pgmap v172: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: audit 2024-04-26T16:22:55.956863+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: audit 2024-04-26T16:22:55.956863+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.057016+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.f scrub starts 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.057016+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.f scrub starts 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.058570+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f scrub ok 2024-04-26T16:22:56.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:56 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.058570+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f scrub ok 2024-04-26T16:22:56.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 10 client.14787.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 15 client.14787.objecter _session_command_op_assign -1 2 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter _get_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter _get_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 15 client.14787.objecter _session_command_op_remove -1 2 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter get_session s=0x7fd3340028e0 osd=3 3 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 15 client.14787.objecter _session_command_op_assign 3 2 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 4 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 10 client.14787.objecter _send_command 2 2024-04-26T16:22:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fd334001210 con 0x7fd334002ff0 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd351ffb700 1 -- 172.21.15.50:0/906079261 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fd334001210 con 0x7fd334002ff0 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd351ffb700 10 client.14787.objecter ms_dispatch 0x7fd354001550 command_reply(tid 2: 0 ) 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd351ffb700 10 client.14787.objecter _finish_command 2 = system:0 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd351ffb700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 4 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd351ffb700 15 client.14787.objecter _session_command_op_remove 3 2 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 10 librados: watch_flush enter 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 10 librados: watch_flush exit 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 10 client.14787.objecter close_session for osd.3 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fd334002ff0 msgr2=0x7fd3340054b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:56.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fd334002ff0 0x7fd3340054b0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter put_session s=0x7fd3340028e0 osd=3 1 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 20 client.14787.objecter shutdown clearing up homeless session... 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 10 client.14787.objecter successfully canceled tick 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd344071fd0 msgr2=0x7fd344074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd344071fd0 0x7fd344074490 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd354108960 tx=0x7fd34000b610 comp rx=0 tx=0).stop 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 msgr2=0x7fd3541991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 --2- 172.21.15.50:0/906079261 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd354101a10 0x7fd3541991f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd348009ef0 tx=0x7fd34800ecd0 comp rx=0 tx=0).stop 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd3596ec700 1 -- 172.21.15.50:0/906079261 reap_dead start 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 shutdown_connections 2024-04-26T16:22:56.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 >> 172.21.15.50:0/906079261 conn(0x7fd354078990 msgr2=0x7fd354105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:22:56.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 shutdown_connections 2024-04-26T16:22:56.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 -- 172.21.15.50:0/906079261 wait complete. 2024-04-26T16:22:56.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:22:56.725+0000 7fd33f7fe700 1 librados: shutdown 2024-04-26T16:22:56.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:56.817 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.295301+0000 osd.3 (osd.3) 135 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.295301+0000 osd.3 (osd.3) 135 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.296601+0000 osd.3 (osd.3) 136 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.296601+0000 osd.3 (osd.3) 136 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.460111+0000 mgr.y (mgr.24401) 201 : cluster [DBG] pgmap v172: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:55.460111+0000 mgr.y (mgr.24401) 201 : cluster [DBG] pgmap v172: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: audit 2024-04-26T16:22:55.956863+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:56.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: audit 2024-04-26T16:22:55.956863+0000 mon.c (mon.2) 91 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:22:56.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.057016+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.f scrub starts 2024-04-26T16:22:56.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.057016+0000 osd.5 (osd.5) 123 : cluster [DBG] 5.f scrub starts 2024-04-26T16:22:56.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.058570+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f scrub ok 2024-04-26T16:22:56.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:56 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.058570+0000 osd.5 (osd.5) 124 : cluster [DBG] 5.f scrub ok 2024-04-26T16:22:56.832 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:56.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:56.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:56.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:56.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:57.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:57.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:57.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:57.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:57.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:57.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:57.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:57.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:57.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:57.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:57.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:57.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:57.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.297068+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:22:57.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.297068+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:22:57.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.298299+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:22:57.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.298299+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:22:57.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: audit 2024-04-26T16:22:56.577109+0000 mgr.y (mgr.24401) 202 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: audit 2024-04-26T16:22:56.577109+0000 mgr.y (mgr.24401) 202 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:57.070374+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:57.070374+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:57.072484+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[29234]: cluster 2024-04-26T16:22:57.072484+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.297068+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.297068+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.298299+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.298299+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: audit 2024-04-26T16:22:56.577109+0000 mgr.y (mgr.24401) 202 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: audit 2024-04-26T16:22:56.577109+0000 mgr.y (mgr.24401) 202 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:57.719 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:57.070374+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:22:57.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:57.070374+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:22:57.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:57.072484+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:22:57.720 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:57 smithi050 bash[21463]: cluster 2024-04-26T16:22:57.072484+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:22:57.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:22:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:22:57.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:57.848 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.297068+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.297068+0000 osd.2 (osd.2) 93 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.298299+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.298299+0000 osd.2 (osd.2) 94 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: audit 2024-04-26T16:22:56.577109+0000 mgr.y (mgr.24401) 202 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: audit 2024-04-26T16:22:56.577109+0000 mgr.y (mgr.24401) 202 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:57.070374+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:57.070374+0000 osd.5 (osd.5) 125 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:57.072484+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:22:57.849 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:57 smithi190 bash[23868]: cluster 2024-04-26T16:22:57.072484+0000 osd.5 (osd.5) 126 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:22:57.865 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:57.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:57.993 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:57.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:58.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:58.123 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:58.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:58.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:58.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:58.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:58.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:58.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:58.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.992497+0000 osd.1 (osd.1) 133 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:22:58.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.992497+0000 osd.1 (osd.1) 133 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:22:58.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.994275+0000 osd.1 (osd.1) 134 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:56.994275+0000 osd.1 (osd.1) 134 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:57.460900+0000 mgr.y (mgr.24401) 203 : cluster [DBG] pgmap v173: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:57.460900+0000 mgr.y (mgr.24401) 203 : cluster [DBG] pgmap v173: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.100509+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.100509+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.103053+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.103053+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.255312+0000 osd.5 (osd.5) 129 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.255312+0000 osd.5 (osd.5) 129 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.255454+0000 osd.5 (osd.5) 130 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.255454+0000 osd.5 (osd.5) 130 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.992497+0000 osd.1 (osd.1) 133 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:22:58.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.992497+0000 osd.1 (osd.1) 133 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.994275+0000 osd.1 (osd.1) 134 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:56.994275+0000 osd.1 (osd.1) 134 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:57.460900+0000 mgr.y (mgr.24401) 203 : cluster [DBG] pgmap v173: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:57.460900+0000 mgr.y (mgr.24401) 203 : cluster [DBG] pgmap v173: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.100509+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.100509+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.103053+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.103053+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.255312+0000 osd.5 (osd.5) 129 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.255312+0000 osd.5 (osd.5) 129 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.255454+0000 osd.5 (osd.5) 130 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:58.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:58 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.255454+0000 osd.5 (osd.5) 130 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:58.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:58.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:58.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:59.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:59.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:22:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:22:59.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.992497+0000 osd.1 (osd.1) 133 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:22:59.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.992497+0000 osd.1 (osd.1) 133 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:22:59.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.994275+0000 osd.1 (osd.1) 134 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:22:59.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:56.994275+0000 osd.1 (osd.1) 134 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:57.460900+0000 mgr.y (mgr.24401) 203 : cluster [DBG] pgmap v173: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:57.460900+0000 mgr.y (mgr.24401) 203 : cluster [DBG] pgmap v173: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.100509+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.100509+0000 osd.5 (osd.5) 127 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.103053+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.103053+0000 osd.5 (osd.5) 128 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.255312+0000 osd.5 (osd.5) 129 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.255312+0000 osd.5 (osd.5) 129 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.255454+0000 osd.5 (osd.5) 130 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:58 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.255454+0000 osd.5 (osd.5) 130 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:22:59.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:59.156 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:59.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:59.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:22:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:22:59.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:59.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:22:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:22:59.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-04-26T16:22:59.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-26T16:22:59.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:59.549 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:22:59.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:59.678 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:22:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:22:59.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.301194+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.301194+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.302538+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.302538+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.399449+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.399449+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.402622+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:22:59.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.402622+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:22:59.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.596781+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-04-26T16:22:59.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:58.596781+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-04-26T16:22:59.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.052584+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:22:59.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.052584+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:22:59.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.055249+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:22:59.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:22:59 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.055249+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:22:59.809 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:22:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.301194+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.301194+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.302538+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.302538+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.399449+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.399449+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.402622+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.402622+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.596781+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:58.596781+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-04-26T16:22:59.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.052584+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.052584+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.055249+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.055249+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.301194+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.301194+0000 osd.2 (osd.2) 95 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.302538+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.302538+0000 osd.2 (osd.2) 96 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.399449+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.399449+0000 osd.7 (osd.7) 139 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.402622+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.402622+0000 osd.7 (osd.7) 140 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:22:59.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.596781+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-04-26T16:22:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:58.596781+0000 mon.a (mon.0) 960 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2024-04-26T16:22:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.052584+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:22:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.052584+0000 osd.6 (osd.6) 75 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:22:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.055249+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:22:59.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:22:59 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.055249+0000 osd.6 (osd.6) 76 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:22:59.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:59.916 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:22:59.940 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:22:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:22:59.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/2326336756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806bb50 msgr2=0x7f01b806bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/2326336756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806bb50 0x7f01b806bfb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0099c0 tx=0x7f01ac009cd0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/2326336756 shutdown_connections 2024-04-26T16:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/2326336756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01b806c4f0 0x7f01b81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/2326336756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f01b81094a0 0x7f01b8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/2326336756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806bb50 0x7f01b806bfb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/2326336756 >> 172.21.15.50:0/2326336756 conn(0x7f01b80792c0 msgr2=0x7f01b80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/2326336756 shutdown_connections 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/2326336756 wait complete. 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 librados: starting msgr at 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 librados: starting objecter 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 Processor -- start 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- start start 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 librados: setting wanted keys 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 librados: calling monclient init 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01b806bb50 0x7f01b8102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 0x7f01b8102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f01b81094a0 0x7f01b8102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 0x7f01b8102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 0x7f01b8102960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43946/0 (socket says 172.21.15.50:43946) 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 -- 172.21.15.50:0/1403248774 learned_addr learned my addr 172.21.15.50:0/1403248774 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f01b81094a0 0x7f01b8102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f01b8112b70 con 0x7f01b806c4f0 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01b806bb50 0x7f01b8102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f01b81129f0 con 0x7f01b806bb50 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f01b8112cf0 con 0x7f01b81094a0 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f01b81094a0 msgr2=0x7f01b8102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f01b81094a0 0x7f01b8102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01b806bb50 msgr2=0x7f01b8102420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f01b806bb50 0x7f01b8102420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b81a7720 con 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f01b81094a0 0x7f01b8102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 0x7f01b8102960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f01ac005b40 tx=0x7f01ac0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f019ffff700 10 client.?.objecter ms_handle_connect 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f019ffff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ac027070 con 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f01ac00fad0 con 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01b81a79b0 con 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ac022a40 con 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 10 client.14793.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01b81a7e60 con 0x7f01b806c4f0 2024-04-26T16:23:00.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.021+0000 7f01bd7b3700 1 librados: init done 2024-04-26T16:23:00.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f01b8104b50 con 0x7f01b806c4f0 2024-04-26T16:23:00.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f01ac02a070 con 0x7f01b806c4f0 2024-04-26T16:23:00.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f01a0071f80 0x7f01a0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(135..135 src has 77..135) ==== 7450+0+0 (secure 0 0 0) 0x7f01ac092290 con 0x7f01b806c4f0 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 osd_map(135..135 src has 77..135) 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 3 client.14793.objecter handle_osd_map got epochs [135,135] > 0 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 3 client.14793.objecter handle_osd_map decoding full epoch 135 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f01a0071f80 0x7f01a0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 20 client.14793.objecter dump_active .. 0 homeless 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v135) ==== 54+0+16 (secure 0 0 0) 0x7f01ac013070 con 0x7f01b806c4f0 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01a0071f80 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f01a0071f80 0x7f01a0074440 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f01a8005950 tx=0x7f01a800b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 1 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f01b8064030 0x7f01b8064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 1 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 0 1 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 4 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 10 client.14793.objecter _send_command 1 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f01b8064030 0x7f01b8064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01b8103950 con 0x7f01b8064030 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b8064030 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.025+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f01b8064030 0x7f01b8064410 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.029+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b8103950 con 0x7f01b8064030 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.029+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 1: 0 ) 2024-04-26T16:23:00.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.029+0000 7f019ffff700 10 client.14793.objecter _finish_command 1 = system:0 2024-04-26T16:23:00.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.029+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 4 2024-04-26T16:23:00.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.029+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 0 1 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 2 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 2 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b806a110 osd=0 3 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 0 2 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 4 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 10 client.14793.objecter _send_command 2 2024-04-26T16:23:00.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b8063ab0 con 0x7f01b8064030 2024-04-26T16:23:00.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b8063ab0 con 0x7f01b8064030 2024-04-26T16:23:00.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f019ffff700 10 client.14793.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 4 2024-04-26T16:23:00.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.045+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 0 2 2024-04-26T16:23:00.042 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f01b810af00 0x7f01b81ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 1 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 4 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 10 client.14793.objecter _send_command 3 2024-04-26T16:23:00.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f01b802e600 con 0x7f01b810af00 2024-04-26T16:23:00.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f01b810af00 0x7f01b81ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b810af00 2024-04-26T16:23:00.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f01b810af00 0x7f01b81ac050 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b802e600 con 0x7f01b810af00 2024-04-26T16:23:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 3: 0 ) 2024-04-26T16:23:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f019ffff700 10 client.14793.objecter _finish_command 3 = system:0 2024-04-26T16:23:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 4 2024-04-26T16:23:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.049+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 1 3 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 4 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 4 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b802e490 osd=1 3 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 1 4 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 4 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 10 client.14793.objecter _send_command 4 2024-04-26T16:23:00.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b81ac590 con 0x7f01b810af00 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b81ac590 con 0x7f01b810af00 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f019ffff700 10 client.14793.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 4 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 1 4 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 5 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f01b810a5e0 0x7f01b81af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 5 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 2 5 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 4 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 10 client.14793.objecter _send_command 5 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f01b802e0d0 con 0x7f01b810a5e0 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.069+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f01b810a5e0 0x7f01b81af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b810a5e0 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f01b810a5e0 0x7f01b81af8d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b802e0d0 con 0x7f01b810a5e0 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 5: 0 ) 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f019ffff700 10 client.14793.objecter _finish_command 5 = system:0 2024-04-26T16:23:00.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 4 2024-04-26T16:23:00.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.073+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 2 5 2024-04-26T16:23:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:00.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:00.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 6 2024-04-26T16:23:00.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 6 2024-04-26T16:23:00.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b810a470 osd=2 3 2024-04-26T16:23:00.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 2 6 2024-04-26T16:23:00.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 4 2024-04-26T16:23:00.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 10 client.14793.objecter _send_command 6 2024-04-26T16:23:00.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b802d560 con 0x7f01b810a5e0 2024-04-26T16:23:00.089 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b802d560 con 0x7f01b810a5e0 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f019ffff700 10 client.14793.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 4 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 2 6 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 7 2024-04-26T16:23:00.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f01b81ad420 0x7f01b81b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 7 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 3 7 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 4 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 10 client.14793.objecter _send_command 7 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f01b81b5ef0 con 0x7f01b81ad420 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f01b81ad420 0x7f01b81b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b81ad420 2024-04-26T16:23:00.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.093+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f01b81ad420 0x7f01b81b2fa0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.097+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b81b5ef0 con 0x7f01b81ad420 2024-04-26T16:23:00.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.097+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 7: 0 ) 2024-04-26T16:23:00.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.097+0000 7f019ffff700 10 client.14793.objecter _finish_command 7 = system:0 2024-04-26T16:23:00.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.097+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 4 2024-04-26T16:23:00.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.097+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 3 7 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 8 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 8 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b802d960 osd=3 3 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 3 8 2024-04-26T16:23:00.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 4 2024-04-26T16:23:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 10 client.14793.objecter _send_command 8 2024-04-26T16:23:00.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.129+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b81aca50 con 0x7f01b81ad420 2024-04-26T16:23:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b81aca50 con 0x7f01b81ad420 2024-04-26T16:23:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f019ffff700 10 client.14793.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 4 2024-04-26T16:23:00.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 3 8 2024-04-26T16:23:00.128 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 9 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f01b81b0af0 0x7f01b81b4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 9 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 4 9 2024-04-26T16:23:00.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 4 2024-04-26T16:23:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 10 client.14793.objecter _send_command 9 2024-04-26T16:23:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f01b81b98e0 con 0x7f01b81b0af0 2024-04-26T16:23:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f01b81b0af0 0x7f01b81b4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b81b0af0 2024-04-26T16:23:00.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.133+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f01b81b0af0 0x7f01b81b4700 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.137+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b81b98e0 con 0x7f01b81b0af0 2024-04-26T16:23:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.137+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 9: 0 ) 2024-04-26T16:23:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.137+0000 7f019ffff700 10 client.14793.objecter _finish_command 9 = system:0 2024-04-26T16:23:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.137+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 4 2024-04-26T16:23:00.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.137+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 4 9 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 10 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 10 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81aca50 osd=4 3 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 4 10 2024-04-26T16:23:00.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 4 2024-04-26T16:23:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 10 client.14793.objecter _send_command 10 2024-04-26T16:23:00.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.169+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b81b4c40 con 0x7f01b81b0af0 2024-04-26T16:23:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b81b4c40 con 0x7f01b81b0af0 2024-04-26T16:23:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f019ffff700 10 client.14793.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 4 2024-04-26T16:23:00.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 4 10 2024-04-26T16:23:00.169 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:23:00.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 11 2024-04-26T16:23:00.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f01b81b5360 0x7f01b81b57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 11 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 5 11 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 4 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 10 client.14793.objecter _send_command 11 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f01b81bcac0 con 0x7f01b81b5360 2024-04-26T16:23:00.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.173+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f01b81b5360 0x7f01b81b57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b81b5360 2024-04-26T16:23:00.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f01b67fc700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f01b81b5360 0x7f01b81b57c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b81bcac0 con 0x7f01b81b5360 2024-04-26T16:23:00.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 11: 0 ) 2024-04-26T16:23:00.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f019ffff700 10 client.14793.objecter _finish_command 11 = system:0 2024-04-26T16:23:00.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 4 2024-04-26T16:23:00.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.177+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 5 11 2024-04-26T16:23:00.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 12 2024-04-26T16:23:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 12 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81b4c40 osd=5 3 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 5 12 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 4 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 10 client.14793.objecter _send_command 12 2024-04-26T16:23:00.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.197+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b81bcac0 con 0x7f01b81b5360 2024-04-26T16:23:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b81bcac0 con 0x7f01b81b5360 2024-04-26T16:23:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f019ffff700 10 client.14793.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 4 2024-04-26T16:23:00.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 5 12 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 13 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f01b81bbeb0 0x7f01b81bc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 13 2024-04-26T16:23:00.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 6 13 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 4 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 10 client.14793.objecter _send_command 13 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f01b81bfe30 con 0x7f01b81bbeb0 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f01b81bbeb0 0x7f01b81bc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b81bbeb0 2024-04-26T16:23:00.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.201+0000 7f01b77fe700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f01b81bbeb0 0x7f01b81bc2f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.205+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b81bfe30 con 0x7f01b81bbeb0 2024-04-26T16:23:00.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.205+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 13: 0 ) 2024-04-26T16:23:00.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.205+0000 7f019ffff700 10 client.14793.objecter _finish_command 13 = system:0 2024-04-26T16:23:00.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.205+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 4 2024-04-26T16:23:00.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.205+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 6 13 2024-04-26T16:23:00.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:00.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 14 2024-04-26T16:23:00.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 14 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81bcac0 osd=6 3 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 6 14 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 4 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 10 client.14793.objecter _send_command 14 2024-04-26T16:23:00.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.225+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b81ba1b0 con 0x7f01b81bbeb0 2024-04-26T16:23:00.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b81ba1b0 con 0x7f01b81bbeb0 2024-04-26T16:23:00.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f019ffff700 10 client.14793.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 4 2024-04-26T16:23:00.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 6 14 2024-04-26T16:23:00.223 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:23:00.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 15 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f01b81ba8d0 0x7f01b81bad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 15 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 7 15 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 4 2024-04-26T16:23:00.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 10 client.14793.objecter _send_command 15 2024-04-26T16:23:00.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f01b81bb3e0 con 0x7f01b81ba8d0 2024-04-26T16:23:00.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f01b81ba8d0 0x7f01b81bad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:00.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f019ffff700 10 client.14793.objecter ms_handle_connect 0x7f01b81ba8d0 2024-04-26T16:23:00.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.229+0000 7f01b6ffd700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f01b81ba8d0 0x7f01b81bad30 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:00.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.233+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f01b81bb3e0 con 0x7f01b81ba8d0 2024-04-26T16:23:00.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.233+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 15: 0 ) 2024-04-26T16:23:00.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.233+0000 7f019ffff700 10 client.14793.objecter _finish_command 15 = system:0 2024-04-26T16:23:00.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.233+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 4 2024-04-26T16:23:00.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.233+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 7 15 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 10 client.14793.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign -1 16 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 20 client.14793.objecter _get_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_remove -1 16 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 20 client.14793.objecter get_session s=0x7f01b81ba1b0 osd=7 3 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 15 client.14793.objecter _session_command_op_assign 7 16 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 4 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 10 client.14793.objecter _send_command 16 2024-04-26T16:23:00.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.249+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f01b81bb3e0 con 0x7f01b81ba8d0 2024-04-26T16:23:00.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f019ffff700 1 -- 172.21.15.50:0/1403248774 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f01b81bb3e0 con 0x7f01b81ba8d0 2024-04-26T16:23:00.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f019ffff700 10 client.14793.objecter ms_dispatch 0x7f01b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:00.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f019ffff700 10 client.14793.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:00.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f019ffff700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 4 2024-04-26T16:23:00.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f019ffff700 15 client.14793.objecter _session_command_op_remove 7 16 2024-04-26T16:23:00.247 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 librados: watch_flush enter 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 librados: watch_flush exit 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.0 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f01b8064030 msgr2=0x7f01b8064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f01b8064030 0x7f01b8064410 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b806a110 osd=0 1 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.1 2024-04-26T16:23:00.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f01b810af00 msgr2=0x7f01b81ac050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f01b810af00 0x7f01b81ac050 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802e490 osd=1 1 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.2 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f01b810a5e0 msgr2=0x7f01b81af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f01b810a5e0 0x7f01b81af8d0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b810a470 osd=2 1 2024-04-26T16:23:00.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.3 2024-04-26T16:23:00.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f01b81ad420 msgr2=0x7f01b81b2fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f01b81ad420 0x7f01b81b2fa0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b802d960 osd=3 1 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.4 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f01b81b0af0 msgr2=0x7f01b81b4700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f01b81b0af0 0x7f01b81b4700 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81aca50 osd=4 1 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.5 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f01b81b5360 msgr2=0x7f01b81b57c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f01b81b5360 0x7f01b81b57c0 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01b77fe700 1 -- 172.21.15.50:0/1403248774 reap_dead start 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81b4c40 osd=5 1 2024-04-26T16:23:00.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.6 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f01b81bbeb0 msgr2=0x7f01b81bc2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f01b81bbeb0 0x7f01b81bc2f0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01b77fe700 1 -- 172.21.15.50:0/1403248774 reap_dead start 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81bcac0 osd=6 1 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter close_session for osd.7 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f01b81ba8d0 msgr2=0x7f01b81bad30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f01b81ba8d0 0x7f01b81bad30 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter put_session s=0x7f01b81ba1b0 osd=7 1 2024-04-26T16:23:00.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 20 client.14793.objecter shutdown clearing up homeless session... 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 10 client.14793.objecter successfully canceled tick 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01b77fe700 1 -- 172.21.15.50:0/1403248774 reap_dead start 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f01a0071f80 msgr2=0x7f01a0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f01a0071f80 0x7f01a0074440 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f01a8005950 tx=0x7f01a800b410 comp rx=0 tx=0).stop 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 msgr2=0x7f01b8102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 0x7f01b8102960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f01ac005b40 tx=0x7f01ac0057c0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 shutdown_connections 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f01b806c4f0 0x7f01b8102960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 --2- 172.21.15.50:0/1403248774 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f01a0071f80 0x7f01a0074440 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 >> 172.21.15.50:0/1403248774 conn(0x7f01b80792c0 msgr2=0x7f01b81001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 shutdown_connections 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 -- 172.21.15.50:0/1403248774 wait complete. 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:00.253+0000 7f01bd7b3700 1 librados: shutdown 2024-04-26T16:23:00.253 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:23:00.254 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:23:00.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:00.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:00.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:00.466 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:00.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:00.595 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:00.598 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.000812+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:23:00.598 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.000812+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:23:00.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:00.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.000812+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.000812+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.002240+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.002240+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.301210+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.d scrub starts 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.301210+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.d scrub starts 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.303514+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.d scrub ok 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.303514+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.d scrub ok 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.360033+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:23:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.360033+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.363240+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.363240+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.461331+0000 mgr.y (mgr.24401) 204 : cluster [DBG] pgmap v175: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: cluster 2024-04-26T16:22:59.461331+0000 mgr.y (mgr.24401) 204 : cluster [DBG] pgmap v175: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: audit 2024-04-26T16:23:00.028386+0000 mon.a (mon.0) 961 : audit [DBG] from='client.? 172.21.15.50:0/1403248774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: audit 2024-04-26T16:23:00.028386+0000 mon.a (mon.0) 961 : audit [DBG] from='client.? 172.21.15.50:0/1403248774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: audit 2024-04-26T16:23:00.512286+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:00 smithi190 bash[23868]: audit 2024-04-26T16:23:00.512286+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:00.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:00.854 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:00.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.000812+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.000812+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.002240+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.002240+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.301210+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.d scrub starts 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.301210+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.d scrub starts 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.303514+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.d scrub ok 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.303514+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.d scrub ok 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.360033+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:23:00.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.360033+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.363240+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.363240+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.461331+0000 mgr.y (mgr.24401) 204 : cluster [DBG] pgmap v175: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: cluster 2024-04-26T16:22:59.461331+0000 mgr.y (mgr.24401) 204 : cluster [DBG] pgmap v175: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: audit 2024-04-26T16:23:00.028386+0000 mon.a (mon.0) 961 : audit [DBG] from='client.? 172.21.15.50:0/1403248774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: audit 2024-04-26T16:23:00.028386+0000 mon.a (mon.0) 961 : audit [DBG] from='client.? 172.21.15.50:0/1403248774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: audit 2024-04-26T16:23:00.512286+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[29234]: audit 2024-04-26T16:23:00.512286+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.002240+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.002240+0000 osd.1 (osd.1) 136 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.301210+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.d scrub starts 2024-04-26T16:23:00.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.301210+0000 osd.2 (osd.2) 97 : cluster [DBG] 5.d scrub starts 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.303514+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.d scrub ok 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.303514+0000 osd.2 (osd.2) 98 : cluster [DBG] 5.d scrub ok 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.360033+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.360033+0000 osd.7 (osd.7) 141 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.363240+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.363240+0000 osd.7 (osd.7) 142 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.461331+0000 mgr.y (mgr.24401) 204 : cluster [DBG] pgmap v175: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: cluster 2024-04-26T16:22:59.461331+0000 mgr.y (mgr.24401) 204 : cluster [DBG] pgmap v175: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: audit 2024-04-26T16:23:00.028386+0000 mon.a (mon.0) 961 : audit [DBG] from='client.? 172.21.15.50:0/1403248774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: audit 2024-04-26T16:23:00.028386+0000 mon.a (mon.0) 961 : audit [DBG] from='client.? 172.21.15.50:0/1403248774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: audit 2024-04-26T16:23:00.512286+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:00.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:00 smithi050 bash[21463]: audit 2024-04-26T16:23:00.512286+0000 mon.c (mon.2) 92 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:23:00.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:00.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:00.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:01.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:01.124 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:01.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:01.254 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:01.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:01.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:01.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:01.514 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:01.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.003972+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:23:01.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.003972+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:23:01.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.006416+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:23:01.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.006416+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.261923+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.261923+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.263765+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.263765+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.273514+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.273514+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.275411+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.275411+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.389937+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.b scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.389937+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.b scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.392102+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.b scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:00.392102+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.b scrub ok 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.028928+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:23:01.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.028928+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.030857+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.030857+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.090120+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.090120+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.090182+0000 osd.4 (osd.4) 110 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.090182+0000 osd.4 (osd.4) 110 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: audit 2024-04-26T16:23:01.246147+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: audit 2024-04-26T16:23:01.246147+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: audit 2024-04-26T16:23:01.247249+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: audit 2024-04-26T16:23:01.247249+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: audit 2024-04-26T16:23:01.255283+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:01.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[21463]: audit 2024-04-26T16:23:01.255283+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:01.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:01.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.003972+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:23:01.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.003972+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:23:01.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.006416+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:23:01.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.006416+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:23:01.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.261923+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.261923+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.263765+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.263765+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.273514+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.273514+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.275411+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.275411+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.389937+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.b scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.389937+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.b scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.392102+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.b scrub ok 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:00.392102+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.b scrub ok 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.028928+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.028928+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:23:01.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.030857+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.030857+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.090120+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.090120+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.090182+0000 osd.4 (osd.4) 110 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.090182+0000 osd.4 (osd.4) 110 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: audit 2024-04-26T16:23:01.246147+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: audit 2024-04-26T16:23:01.246147+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: audit 2024-04-26T16:23:01.247249+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: audit 2024-04-26T16:23:01.247249+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: audit 2024-04-26T16:23:01.255283+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:01.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:01 smithi050 bash[29234]: audit 2024-04-26T16:23:01.255283+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:01.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:01.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:01.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.003972+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:23:01.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.003972+0000 osd.1 (osd.1) 137 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:23:01.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.006416+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:23:01.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.006416+0000 osd.1 (osd.1) 138 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.261923+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.261923+0000 osd.2 (osd.2) 99 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.263765+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.263765+0000 osd.2 (osd.2) 100 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.273514+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.273514+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.275411+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.275411+0000 osd.0 (osd.0) 126 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.389937+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.b scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.389937+0000 osd.7 (osd.7) 143 : cluster [DBG] 2.b scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.392102+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.b scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:00.392102+0000 osd.7 (osd.7) 144 : cluster [DBG] 2.b scrub ok 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.028928+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.028928+0000 osd.6 (osd.6) 77 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:23:01.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.030857+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.030857+0000 osd.6 (osd.6) 78 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.090120+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.090120+0000 osd.4 (osd.4) 109 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.090182+0000 osd.4 (osd.4) 110 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.090182+0000 osd.4 (osd.4) 110 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: audit 2024-04-26T16:23:01.246147+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: audit 2024-04-26T16:23:01.246147+0000 mon.c (mon.2) 93 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: audit 2024-04-26T16:23:01.247249+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: audit 2024-04-26T16:23:01.247249+0000 mon.c (mon.2) 94 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: audit 2024-04-26T16:23:01.255283+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:01.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:01 smithi190 bash[23868]: audit 2024-04-26T16:23:01.255283+0000 mon.a (mon.0) 962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:01.772 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:01.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:01.903 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:01.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:02.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:02.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:02.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:02.165 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:02.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:02.296 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:02.399 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:02.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:02.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:02.557 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:02.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:02.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.299258+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:23:02.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.299258+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.300800+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.300800+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.361220+0000 osd.7 (osd.7) 145 : cluster [DBG] 3.e scrub starts 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.361220+0000 osd.7 (osd.7) 145 : cluster [DBG] 3.e scrub starts 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.365093+0000 osd.7 (osd.7) 146 : cluster [DBG] 3.e scrub ok 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.365093+0000 osd.7 (osd.7) 146 : cluster [DBG] 3.e scrub ok 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.407422+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.407422+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.409160+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.409160+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.462135+0000 mgr.y (mgr.24401) 205 : cluster [DBG] pgmap v176: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.462135+0000 mgr.y (mgr.24401) 205 : cluster [DBG] pgmap v176: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:02.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.993210+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:01.993210+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.046929+0000 osd.6 (osd.6) 79 : cluster [DBG] 3.a scrub starts 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.046929+0000 osd.6 (osd.6) 79 : cluster [DBG] 3.a scrub starts 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.052460+0000 osd.6 (osd.6) 80 : cluster [DBG] 3.a scrub ok 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.052460+0000 osd.6 (osd.6) 80 : cluster [DBG] 3.a scrub ok 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.299258+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.299258+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.300800+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.300800+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.361220+0000 osd.7 (osd.7) 145 : cluster [DBG] 3.e scrub starts 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.361220+0000 osd.7 (osd.7) 145 : cluster [DBG] 3.e scrub starts 2024-04-26T16:23:02.672 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.365093+0000 osd.7 (osd.7) 146 : cluster [DBG] 3.e scrub ok 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.365093+0000 osd.7 (osd.7) 146 : cluster [DBG] 3.e scrub ok 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.407422+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.407422+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.409160+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.409160+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.462135+0000 mgr.y (mgr.24401) 205 : cluster [DBG] pgmap v176: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.462135+0000 mgr.y (mgr.24401) 205 : cluster [DBG] pgmap v176: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.993210+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:01.993210+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.046929+0000 osd.6 (osd.6) 79 : cluster [DBG] 3.a scrub starts 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.046929+0000 osd.6 (osd.6) 79 : cluster [DBG] 3.a scrub starts 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.052460+0000 osd.6 (osd.6) 80 : cluster [DBG] 3.a scrub ok 2024-04-26T16:23:02.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:02 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.052460+0000 osd.6 (osd.6) 80 : cluster [DBG] 3.a scrub ok 2024-04-26T16:23:02.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:02.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.299258+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.299258+0000 osd.0 (osd.0) 127 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.300800+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.300800+0000 osd.0 (osd.0) 128 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.361220+0000 osd.7 (osd.7) 145 : cluster [DBG] 3.e scrub starts 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.361220+0000 osd.7 (osd.7) 145 : cluster [DBG] 3.e scrub starts 2024-04-26T16:23:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.365093+0000 osd.7 (osd.7) 146 : cluster [DBG] 3.e scrub ok 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.365093+0000 osd.7 (osd.7) 146 : cluster [DBG] 3.e scrub ok 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.407422+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.407422+0000 osd.3 (osd.3) 137 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.409160+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.409160+0000 osd.3 (osd.3) 138 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.462135+0000 mgr.y (mgr.24401) 205 : cluster [DBG] pgmap v176: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.462135+0000 mgr.y (mgr.24401) 205 : cluster [DBG] pgmap v176: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.993210+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:01.993210+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.046929+0000 osd.6 (osd.6) 79 : cluster [DBG] 3.a scrub starts 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.046929+0000 osd.6 (osd.6) 79 : cluster [DBG] 3.a scrub starts 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.052460+0000 osd.6 (osd.6) 80 : cluster [DBG] 3.a scrub ok 2024-04-26T16:23:02.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:02 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.052460+0000 osd.6 (osd.6) 80 : cluster [DBG] 3.a scrub ok 2024-04-26T16:23:02.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:02.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:02.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:02.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:03.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:03.073 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:03.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:03.204 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:03.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:03.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:03.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:03.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:03.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:03.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:03.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:03.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:03.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.257374+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.257374+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.260870+0000 osd.0 (osd.0) 130 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.260870+0000 osd.0 (osd.0) 130 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.417351+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.b scrub starts 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.417351+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.b scrub starts 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.419899+0000 osd.3 (osd.3) 140 : cluster [DBG] 3.b scrub ok 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.419899+0000 osd.3 (osd.3) 140 : cluster [DBG] 3.b scrub ok 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.477587+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.477587+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.479880+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:23:03.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[21463]: cluster 2024-04-26T16:23:02.479880+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.257374+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.257374+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.260870+0000 osd.0 (osd.0) 130 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.260870+0000 osd.0 (osd.0) 130 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.417351+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.b scrub starts 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.417351+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.b scrub starts 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.419899+0000 osd.3 (osd.3) 140 : cluster [DBG] 3.b scrub ok 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.419899+0000 osd.3 (osd.3) 140 : cluster [DBG] 3.b scrub ok 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.477587+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.477587+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.479880+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:23:03.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:03 smithi050 bash[29234]: cluster 2024-04-26T16:23:02.479880+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:23:03.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.257374+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:23:03.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.257374+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:23:03.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.260870+0000 osd.0 (osd.0) 130 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.260870+0000 osd.0 (osd.0) 130 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.417351+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.b scrub starts 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.417351+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.b scrub starts 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.419899+0000 osd.3 (osd.3) 140 : cluster [DBG] 3.b scrub ok 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.419899+0000 osd.3 (osd.3) 140 : cluster [DBG] 3.b scrub ok 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.477587+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.477587+0000 osd.4 (osd.4) 111 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.479880+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:23:03.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:03 smithi190 bash[23868]: cluster 2024-04-26T16:23:02.479880+0000 osd.4 (osd.4) 112 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:23:03.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:03.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:03.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:04.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:04.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:04.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:04.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:04.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:04.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:23:04.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:04.287 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:04.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:04.406 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:23:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:23:04.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:04.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:04.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:04.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:04.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:04.765 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.346070+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.346070+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.347612+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.347612+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.378629+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.b scrub starts 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.378629+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.b scrub starts 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.380132+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.b scrub ok 2024-04-26T16:23:04.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.380132+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.b scrub ok 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.462562+0000 mgr.y (mgr.24401) 206 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[29234]: cluster 2024-04-26T16:23:03.462562+0000 mgr.y (mgr.24401) 206 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.346070+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.346070+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.347612+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.347612+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.378629+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.b scrub starts 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.378629+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.b scrub starts 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.380132+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.b scrub ok 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.380132+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.b scrub ok 2024-04-26T16:23:04.767 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.462562+0000 mgr.y (mgr.24401) 206 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:04.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:04 smithi050 bash[21463]: cluster 2024-04-26T16:23:03.462562+0000 mgr.y (mgr.24401) 206 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:04.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:04.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:04.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:04.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.346070+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.346070+0000 osd.2 (osd.2) 101 : cluster [DBG] 2.e scrub starts 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.347612+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.347612+0000 osd.2 (osd.2) 102 : cluster [DBG] 2.e scrub ok 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.378629+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.b scrub starts 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.378629+0000 osd.3 (osd.3) 141 : cluster [DBG] 6.b scrub starts 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.380132+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.b scrub ok 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.380132+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.b scrub ok 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.462562+0000 mgr.y (mgr.24401) 206 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:04.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:04 smithi190 bash[23868]: cluster 2024-04-26T16:23:03.462562+0000 mgr.y (mgr.24401) 206 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- 172.21.15.50:0/3882240978 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 msgr2=0x7fdef810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3882240978 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef810f250 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdef0009fc0 tx=0x7fdef000b330 comp rx=0 tx=0).stop 2024-04-26T16:23:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- 172.21.15.50:0/3882240978 shutdown_connections 2024-04-26T16:23:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3882240978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8102960 0x7fdef8102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3882240978 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef810f250 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:04.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3882240978 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8103310 0x7fdef8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- 172.21.15.50:0/3882240978 >> 172.21.15.50:0/3882240978 conn(0x7fdef80792d0 msgr2=0x7fdef80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- 172.21.15.50:0/3882240978 shutdown_connections 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- 172.21.15.50:0/3882240978 wait complete. 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 librados: starting msgr at 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 librados: starting objecter 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 Processor -- start 2024-04-26T16:23:04.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- start start 2024-04-26T16:23:04.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 librados: setting wanted keys 2024-04-26T16:23:04.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 librados: calling monclient init 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8102960 0x7fdef8199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8103310 0x7fdef8199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdef81119d0 con 0x7fdef8102960 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdef8111850 con 0x7fdef8103310 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdefe10f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fdef8111b50 con 0x7fdef81076b0 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef819d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44748/0 (socket says 172.21.15.50:44748) 2024-04-26T16:23:04.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 -- 172.21.15.50:0/3339852742 learned_addr learned my addr 172.21.15.50:0/3339852742 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef77fe700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8102960 0x7fdef8199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef6ffd700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8103310 0x7fdef8199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 -- 172.21.15.50:0/3339852742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8103310 msgr2=0x7fdef8199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8103310 0x7fdef8199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 -- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8102960 msgr2=0x7fdef8199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8102960 0x7fdef8199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 -- 172.21.15.50:0/3339852742 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef819e130 con 0x7fdef81076b0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef6ffd700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8103310 0x7fdef8199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef77fe700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8102960 0x7fdef8199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef7fff700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef819d9f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdef0006070 tx=0x7fdef0004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef4ff9700 10 client.?.objecter ms_handle_connect 0x7fdef81076b0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef4ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef0022070 con 0x7fdef81076b0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fdef0007910 con 0x7fdef81076b0 2024-04-26T16:23:04.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.929+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00052a0 con 0x7fdef81076b0 2024-04-26T16:23:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef819e3c0 con 0x7fdef81076b0 2024-04-26T16:23:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdefe10f700 10 client.24689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef819e820 con 0x7fdef81076b0 2024-04-26T16:23:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdefe10f700 1 librados: init done 2024-04-26T16:23:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef8103a60 con 0x7fdef81076b0 2024-04-26T16:23:04.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fdef001c6c0 con 0x7fdef81076b0 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdee0071fd0 0x7fdee0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(136..136 src has 77..136) ==== 7450+0+0 (secure 0 0 0) 0x7fdef000f020 con 0x7fdef81076b0 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 10 client.24689.objecter ms_dispatch 0x7fdef8001550 osd_map(136..136 src has 77..136) 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 3 client.24689.objecter handle_osd_map got epochs [136,136] > 0 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 3 client.24689.objecter handle_osd_map decoding full epoch 136 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef77fe700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdee0071fd0 0x7fdee0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:04.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.933+0000 7fdef4ff9700 20 client.24689.objecter dump_active .. 0 homeless 2024-04-26T16:23:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.937+0000 7fdef4ff9700 10 client.24689.objecter ms_handle_connect 0x7fdee0071fd0 2024-04-26T16:23:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.937+0000 7fdef77fe700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdee0071fd0 0x7fdee0074490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdee80104e0 tx=0x7fdee800b3c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:04.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:04.941+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fdef819e820 con 0x7fdef81076b0 2024-04-26T16:23:04.936 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:04.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:05.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:05.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:05.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.169+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fdef8103fb0 con 0x7fdef81076b0 2024-04-26T16:23:05.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:05.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:05.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:05.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:05.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:05.460 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:05.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:05.590 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:05.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.625+0000 7fdef4ff9700 1 -- 172.21.15.50:0/3339852742 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v137) ==== 101+0+0 (secure 0 0 0) 0x7fdef0062150 con 0x7fdef81076b0 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 10 librados: watch_flush enter 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 10 librados: watch_flush exit 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 20 client.24689.objecter shutdown clearing up homeless session... 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 10 client.24689.objecter successfully canceled tick 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdee0071fd0 msgr2=0x7fdee0074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdee0071fd0 0x7fdee0074490 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdee80104e0 tx=0x7fdee800b3c0 comp rx=0 tx=0).stop 2024-04-26T16:23:05.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 msgr2=0x7fdef819d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef819d9f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdef0006070 tx=0x7fdef0004d00 comp rx=0 tx=0).stop 2024-04-26T16:23:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 shutdown_connections 2024-04-26T16:23:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdee0071fd0 0x7fdee0074490 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdef81076b0 0x7fdef819d9f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:05.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdef8102960 0x7fdef8199120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 --2- 172.21.15.50:0/3339852742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdef8103310 0x7fdef8199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 >> 172.21.15.50:0/3339852742 conn(0x7fdef80792d0 msgr2=0x7fdef806baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 shutdown_connections 2024-04-26T16:23:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 -- 172.21.15.50:0/3339852742 wait complete. 2024-04-26T16:23:05.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:05.629+0000 7fdefe10f700 1 librados: shutdown 2024-04-26T16:23:05.627 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:23:05.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:05.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.123895+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:23:05.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.123895+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.126055+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.126055+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.331021+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.331021+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.332207+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:04.332207+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.088384+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.088384+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.089782+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.089782+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: audit 2024-04-26T16:23:05.175254+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.50:0/3339852742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: audit 2024-04-26T16:23:05.175254+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.50:0/3339852742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: audit 2024-04-26T16:23:05.175605+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:05 smithi190 bash[23868]: audit 2024-04-26T16:23:05.175605+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:05.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:05.852 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.123895+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.123895+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.126055+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.126055+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.331021+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.331021+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.332207+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:04.332207+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.088384+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.088384+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.089782+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:23:05.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.089782+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: audit 2024-04-26T16:23:05.175254+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.50:0/3339852742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: audit 2024-04-26T16:23:05.175254+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.50:0/3339852742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: audit 2024-04-26T16:23:05.175605+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[21463]: audit 2024-04-26T16:23:05.175605+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.123895+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.123895+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.126055+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.126055+0000 osd.1 (osd.1) 140 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.331021+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.331021+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:23:05.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.332207+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:04.332207+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.088384+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.088384+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.089782+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.089782+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: audit 2024-04-26T16:23:05.175254+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.50:0/3339852742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: audit 2024-04-26T16:23:05.175254+0000 mon.c (mon.2) 95 : audit [INF] from='client.? 172.21.15.50:0/3339852742' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: audit 2024-04-26T16:23:05.175605+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:05 smithi050 bash[29234]: audit 2024-04-26T16:23:05.175605+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:23:05.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:05.980 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:05.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:06.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:06.113 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:06.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:06.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:06.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:06.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:06.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:06.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:06.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:06.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:06.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:06.630 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:06.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:06.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:06.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:23:06.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.324794+0000 osd.2 (osd.2) 103 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:23:06.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.324794+0000 osd.2 (osd.2) 103 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:23:06.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.326026+0000 osd.2 (osd.2) 104 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:23:06.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.326026+0000 osd.2 (osd.2) 104 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:23:06.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.462982+0000 mgr.y (mgr.24401) 207 : cluster [DBG] pgmap v179: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.462982+0000 mgr.y (mgr.24401) 207 : cluster [DBG] pgmap v179: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.465467+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.465467+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.466555+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.466555+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.618783+0000 mon.a (mon.0) 965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.618783+0000 mon.a (mon.0) 965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:23:06.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.618819+0000 mon.a (mon.0) 966 : cluster [INF] Cluster is now healthy 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.618819+0000 mon.a (mon.0) 966 : cluster [INF] Cluster is now healthy 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: audit 2024-04-26T16:23:05.623548+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: audit 2024-04-26T16:23:05.623548+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.628307+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.628307+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.966507+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.966507+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:23:06.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.968653+0000 osd.5 (osd.5) 132 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:23:06.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:06 smithi190 bash[23868]: cluster 2024-04-26T16:23:05.968653+0000 osd.5 (osd.5) 132 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:23:06.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:06.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.324794+0000 osd.2 (osd.2) 103 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:23:06.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.324794+0000 osd.2 (osd.2) 103 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:23:06.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.326026+0000 osd.2 (osd.2) 104 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:23:06.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.326026+0000 osd.2 (osd.2) 104 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:23:06.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.462982+0000 mgr.y (mgr.24401) 207 : cluster [DBG] pgmap v179: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:06.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.462982+0000 mgr.y (mgr.24401) 207 : cluster [DBG] pgmap v179: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.465467+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.465467+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.466555+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.466555+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.618783+0000 mon.a (mon.0) 965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.618783+0000 mon.a (mon.0) 965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.618819+0000 mon.a (mon.0) 966 : cluster [INF] Cluster is now healthy 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.618819+0000 mon.a (mon.0) 966 : cluster [INF] Cluster is now healthy 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: audit 2024-04-26T16:23:05.623548+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: audit 2024-04-26T16:23:05.623548+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.628307+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.628307+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-04-26T16:23:06.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.966507+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.966507+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.968653+0000 osd.5 (osd.5) 132 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[21463]: cluster 2024-04-26T16:23:05.968653+0000 osd.5 (osd.5) 132 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.324794+0000 osd.2 (osd.2) 103 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.324794+0000 osd.2 (osd.2) 103 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.326026+0000 osd.2 (osd.2) 104 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.326026+0000 osd.2 (osd.2) 104 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.462982+0000 mgr.y (mgr.24401) 207 : cluster [DBG] pgmap v179: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.462982+0000 mgr.y (mgr.24401) 207 : cluster [DBG] pgmap v179: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.465467+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:23:06.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.465467+0000 osd.4 (osd.4) 113 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.466555+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.466555+0000 osd.4 (osd.4) 114 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.618783+0000 mon.a (mon.0) 965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.618783+0000 mon.a (mon.0) 965 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.618819+0000 mon.a (mon.0) 966 : cluster [INF] Cluster is now healthy 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.618819+0000 mon.a (mon.0) 966 : cluster [INF] Cluster is now healthy 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: audit 2024-04-26T16:23:05.623548+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: audit 2024-04-26T16:23:05.623548+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.628307+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.628307+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.966507+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.966507+0000 osd.5 (osd.5) 131 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.968653+0000 osd.5 (osd.5) 132 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:23:06.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:06 smithi050 bash[29234]: cluster 2024-04-26T16:23:05.968653+0000 osd.5 (osd.5) 132 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:23:06.888 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:06.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:06.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:06.992 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:06.992 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:07.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:07.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:07.163 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:07.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:07.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:07.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:07.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:07.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:07.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:07.552 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:07.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:07.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:06.338168+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:23:07.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:06.338168+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:23:07.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:06.340107+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:23:07.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:06.340107+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:23:07.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: audit 2024-04-26T16:23:06.587896+0000 mgr.y (mgr.24401) 208 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: audit 2024-04-26T16:23:06.587896+0000 mgr.y (mgr.24401) 208 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.066077+0000 osd.6 (osd.6) 83 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.066077+0000 osd.6 (osd.6) 83 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.069379+0000 osd.6 (osd.6) 84 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.069379+0000 osd.6 (osd.6) 84 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:06.338168+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:06.338168+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:06.340107+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:06.340107+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:23:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: audit 2024-04-26T16:23:06.587896+0000 mgr.y (mgr.24401) 208 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: audit 2024-04-26T16:23:06.587896+0000 mgr.y (mgr.24401) 208 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.066077+0000 osd.6 (osd.6) 83 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:23:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.066077+0000 osd.6 (osd.6) 83 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:23:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.069379+0000 osd.6 (osd.6) 84 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:23:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:07 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.069379+0000 osd.6 (osd.6) 84 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:23:07.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:07.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:07.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:07.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:07.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:06.338168+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:06.338168+0000 osd.3 (osd.3) 143 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:06.340107+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:06.340107+0000 osd.3 (osd.3) 144 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: audit 2024-04-26T16:23:06.587896+0000 mgr.y (mgr.24401) 208 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: audit 2024-04-26T16:23:06.587896+0000 mgr.y (mgr.24401) 208 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.066077+0000 osd.6 (osd.6) 83 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.066077+0000 osd.6 (osd.6) 83 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.069379+0000 osd.6 (osd.6) 84 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:23:07.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:07 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.069379+0000 osd.6 (osd.6) 84 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:23:07.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:07.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:07.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:08.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:08.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:08.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:08.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:08.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:08.326 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:08.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:08.456 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:08.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:08.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:23:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:08.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:08.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:08.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.357848+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.357848+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.360641+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.360641+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.463771+0000 mgr.y (mgr.24401) 209 : cluster [DBG] pgmap v181: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[21463]: cluster 2024-04-26T16:23:07.463771+0000 mgr.y (mgr.24401) 209 : cluster [DBG] pgmap v181: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.357848+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.357848+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:23:08.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.360641+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:23:08.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.360641+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:23:08.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.463771+0000 mgr.y (mgr.24401) 209 : cluster [DBG] pgmap v181: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:08.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:08 smithi050 bash[29234]: cluster 2024-04-26T16:23:07.463771+0000 mgr.y (mgr.24401) 209 : cluster [DBG] pgmap v181: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:08.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:08.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:08.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:08.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:08.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:08.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:09.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:08 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.357848+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:23:09.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:08 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.357848+0000 osd.3 (osd.3) 145 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:23:09.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:08 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.360641+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:23:09.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:08 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.360641+0000 osd.3 (osd.3) 146 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:23:09.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:08 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.463771+0000 mgr.y (mgr.24401) 209 : cluster [DBG] pgmap v181: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:09.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:08 smithi190 bash[23868]: cluster 2024-04-26T16:23:07.463771+0000 mgr.y (mgr.24401) 209 : cluster [DBG] pgmap v181: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:09.074 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:09.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:09.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:09.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:09.241 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:09.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:09.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:09.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:09.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:09.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:09.617 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:09.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:09.749 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:09.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.124249+0000 osd.1 (osd.1) 141 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.124249+0000 osd.1 (osd.1) 141 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.126392+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.126392+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.259052+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.259052+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.262417+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.262417+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.990203+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:23:09.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.990203+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.992355+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[21463]: cluster 2024-04-26T16:23:08.992355+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.124249+0000 osd.1 (osd.1) 141 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.124249+0000 osd.1 (osd.1) 141 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.126392+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.126392+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.259052+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:23:09.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.259052+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:23:09.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.262417+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:23:09.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.262417+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:23:09.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.990203+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:23:09.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.990203+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:23:09.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.992355+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:23:09.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:09 smithi050 bash[29234]: cluster 2024-04-26T16:23:08.992355+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:23:09.880 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:09.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:09.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.124249+0000 osd.1 (osd.1) 141 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.124249+0000 osd.1 (osd.1) 141 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.126392+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.126392+0000 osd.1 (osd.1) 142 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.259052+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.259052+0000 osd.0 (osd.0) 131 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.262417+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.262417+0000 osd.0 (osd.0) 132 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:23:09.995 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.990203+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:23:09.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.990203+0000 osd.5 (osd.5) 133 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:23:09.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.992355+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:23:09.996 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:09 smithi190 bash[23868]: cluster 2024-04-26T16:23:08.992355+0000 osd.5 (osd.5) 134 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:23:10.008 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:10.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:10.137 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:10.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:10.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:10.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:10.398 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:10.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:10.529 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:10.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:10.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:10.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:10.789 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:10.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.109094+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.a scrub starts 2024-04-26T16:23:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.109094+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.a scrub starts 2024-04-26T16:23:10.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.110549+0000 osd.1 (osd.1) 144 : cluster [DBG] 2.a scrub ok 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.110549+0000 osd.1 (osd.1) 144 : cluster [DBG] 2.a scrub ok 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.359723+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.359723+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.361561+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.361561+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.464189+0000 mgr.y (mgr.24401) 210 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[21463]: cluster 2024-04-26T16:23:09.464189+0000 mgr.y (mgr.24401) 210 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.109094+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.a scrub starts 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.109094+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.a scrub starts 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.110549+0000 osd.1 (osd.1) 144 : cluster [DBG] 2.a scrub ok 2024-04-26T16:23:10.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.110549+0000 osd.1 (osd.1) 144 : cluster [DBG] 2.a scrub ok 2024-04-26T16:23:10.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.359723+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:23:10.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.359723+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:23:10.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.361561+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:23:10.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.361561+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:23:10.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.464189+0000 mgr.y (mgr.24401) 210 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:10.904 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:10 smithi050 bash[29234]: cluster 2024-04-26T16:23:09.464189+0000 mgr.y (mgr.24401) 210 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:10.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:10.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:11.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:11.047 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.109094+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.a scrub starts 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.109094+0000 osd.1 (osd.1) 143 : cluster [DBG] 2.a scrub starts 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.110549+0000 osd.1 (osd.1) 144 : cluster [DBG] 2.a scrub ok 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.110549+0000 osd.1 (osd.1) 144 : cluster [DBG] 2.a scrub ok 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.359723+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.359723+0000 osd.3 (osd.3) 147 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.361561+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:23:11.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.361561+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:23:11.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.464189+0000 mgr.y (mgr.24401) 210 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:11.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:10 smithi190 bash[23868]: cluster 2024-04-26T16:23:09.464189+0000 mgr.y (mgr.24401) 210 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:11.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:11.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:11.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:11.308 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:11.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:11.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:11.438 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:11.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:11.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:11.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:11.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:11.829 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:11.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:11.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:11.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:12.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:12.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: cluster 2024-04-26T16:23:10.455770+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: cluster 2024-04-26T16:23:10.455770+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: cluster 2024-04-26T16:23:10.457853+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: cluster 2024-04-26T16:23:10.457853+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: audit 2024-04-26T16:23:10.977302+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: audit 2024-04-26T16:23:10.977302+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: audit 2024-04-26T16:23:10.978795+0000 mon.c (mon.2) 96 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: audit 2024-04-26T16:23:10.978795+0000 mon.c (mon.2) 96 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:12.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.465004+0000 mgr.y (mgr.24401) 211 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:12.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:11 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.465004+0000 mgr.y (mgr.24401) 211 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:12.090 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:12.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:12.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: cluster 2024-04-26T16:23:10.455770+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:23:12.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: cluster 2024-04-26T16:23:10.455770+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:23:12.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: cluster 2024-04-26T16:23:10.457853+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:23:12.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: cluster 2024-04-26T16:23:10.457853+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:23:12.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: audit 2024-04-26T16:23:10.977302+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:12.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: audit 2024-04-26T16:23:10.977302+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: audit 2024-04-26T16:23:10.978795+0000 mon.c (mon.2) 96 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: audit 2024-04-26T16:23:10.978795+0000 mon.c (mon.2) 96 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.465004+0000 mgr.y (mgr.24401) 211 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.465004+0000 mgr.y (mgr.24401) 211 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: cluster 2024-04-26T16:23:10.455770+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: cluster 2024-04-26T16:23:10.455770+0000 osd.4 (osd.4) 115 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:23:12.206 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: cluster 2024-04-26T16:23:10.457853+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:23:12.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: cluster 2024-04-26T16:23:10.457853+0000 osd.4 (osd.4) 116 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:23:12.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: audit 2024-04-26T16:23:10.977302+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:12.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: audit 2024-04-26T16:23:10.977302+0000 mon.a (mon.0) 969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:12.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: audit 2024-04-26T16:23:10.978795+0000 mon.c (mon.2) 96 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:12.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: audit 2024-04-26T16:23:10.978795+0000 mon.c (mon.2) 96 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:12.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.465004+0000 mgr.y (mgr.24401) 211 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:12.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:11 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.465004+0000 mgr.y (mgr.24401) 211 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:12.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:12.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:12.349 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:12.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:12.478 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:12.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:12.609 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:12.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:12.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:12.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:12.882 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:12.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:12.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:12.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:13.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:13.116 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.829708+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.829708+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.829849+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.829849+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.962173+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.a scrub starts 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.962173+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.a scrub starts 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.964440+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.a scrub ok 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.964440+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.a scrub ok 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.986883+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-04-26T16:23:13.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:11.986883+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.111190+0000 osd.1 (osd.1) 145 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.111190+0000 osd.1 (osd.1) 145 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.114294+0000 osd.1 (osd.1) 146 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.114294+0000 osd.1 (osd.1) 146 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.829708+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.829708+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.829849+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.829849+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.962173+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.a scrub starts 2024-04-26T16:23:13.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.962173+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.a scrub starts 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.964440+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.a scrub ok 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.964440+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.a scrub ok 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.986883+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:11.986883+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.111190+0000 osd.1 (osd.1) 145 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.111190+0000 osd.1 (osd.1) 145 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.114294+0000 osd.1 (osd.1) 146 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:23:13.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:12 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.114294+0000 osd.1 (osd.1) 146 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:23:13.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:13.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:13.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.829708+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.829708+0000 osd.6 (osd.6) 85 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.829849+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.829849+0000 osd.6 (osd.6) 86 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.962173+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.a scrub starts 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.962173+0000 osd.5 (osd.5) 135 : cluster [DBG] 6.a scrub starts 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.964440+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.a scrub ok 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.964440+0000 osd.5 (osd.5) 136 : cluster [DBG] 6.a scrub ok 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.986883+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:11.986883+0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.111190+0000 osd.1 (osd.1) 145 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.111190+0000 osd.1 (osd.1) 145 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.114294+0000 osd.1 (osd.1) 146 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:23:13.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:12 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.114294+0000 osd.1 (osd.1) 146 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:23:13.272 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:13.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:13.402 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:13.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:13.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:13.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:13.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:13.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:13.791 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:13.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:13.920 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:13.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:14.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:14.049 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:14.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.375382+0000 osd.3 (osd.3) 149 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.375382+0000 osd.3 (osd.3) 149 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.377705+0000 osd.3 (osd.3) 150 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.377705+0000 osd.3 (osd.3) 150 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.975053+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.975053+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.976529+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:12.976529+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:13.465427+0000 mgr.y (mgr.24401) 212 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:14.070 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:13 smithi190 bash[23868]: cluster 2024-04-26T16:23:13.465427+0000 mgr.y (mgr.24401) 212 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:14.152 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:14.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:14.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.375382+0000 osd.3 (osd.3) 149 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:23:14.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.375382+0000 osd.3 (osd.3) 149 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:23:14.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.377705+0000 osd.3 (osd.3) 150 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:23:14.278 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.377705+0000 osd.3 (osd.3) 150 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.975053+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.975053+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.976529+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:12.976529+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:13.465427+0000 mgr.y (mgr.24401) 212 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[29234]: cluster 2024-04-26T16:23:13.465427+0000 mgr.y (mgr.24401) 212 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.375382+0000 osd.3 (osd.3) 149 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.375382+0000 osd.3 (osd.3) 149 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.377705+0000 osd.3 (osd.3) 150 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.377705+0000 osd.3 (osd.3) 150 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.975053+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.975053+0000 osd.5 (osd.5) 137 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:23:14.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.976529+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:23:14.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:12.976529+0000 osd.5 (osd.5) 138 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:23:14.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:13.465427+0000 mgr.y (mgr.24401) 212 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:14.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:13 smithi050 bash[21463]: cluster 2024-04-26T16:23:13.465427+0000 mgr.y (mgr.24401) 212 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:23:14.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.308 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:14.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.440 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:14.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:14.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:14.626 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:23:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:23:14.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:14.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:14.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.824 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:14.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.955 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:14.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:15.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:15.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:14 smithi190 bash[23868]: cluster 2024-04-26T16:23:13.549920+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:23:15.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:14 smithi190 bash[23868]: cluster 2024-04-26T16:23:13.549920+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:23:15.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:14 smithi190 bash[23868]: cluster 2024-04-26T16:23:13.552482+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:23:15.069 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:14 smithi190 bash[23868]: cluster 2024-04-26T16:23:13.552482+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:23:15.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:15.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:15.216 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:15.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:15.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[21463]: cluster 2024-04-26T16:23:13.549920+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:23:15.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[21463]: cluster 2024-04-26T16:23:13.549920+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:23:15.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[21463]: cluster 2024-04-26T16:23:13.552482+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:23:15.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[21463]: cluster 2024-04-26T16:23:13.552482+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:23:15.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[29234]: cluster 2024-04-26T16:23:13.549920+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:23:15.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[29234]: cluster 2024-04-26T16:23:13.549920+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:23:15.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[29234]: cluster 2024-04-26T16:23:13.552482+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:23:15.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:14 smithi050 bash[29234]: cluster 2024-04-26T16:23:13.552482+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:23:15.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:15.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:15.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:15.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:15.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:15.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:15.781 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:15.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:15.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:15.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:15.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:15.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:16.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:16.124 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:16.227 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:16.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:16.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[23868]: cluster 2024-04-26T16:23:15.409185+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:23:16.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[23868]: cluster 2024-04-26T16:23:15.409185+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:23:16.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[23868]: cluster 2024-04-26T16:23:15.410739+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:23:16.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[23868]: cluster 2024-04-26T16:23:15.410739+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:23:16.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[23868]: cluster 2024-04-26T16:23:15.466052+0000 mgr.y (mgr.24401) 213 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:23:16.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[23868]: cluster 2024-04-26T16:23:15.466052+0000 mgr.y (mgr.24401) 213 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:23:16.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[21463]: cluster 2024-04-26T16:23:15.409185+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[21463]: cluster 2024-04-26T16:23:15.409185+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[21463]: cluster 2024-04-26T16:23:15.410739+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[21463]: cluster 2024-04-26T16:23:15.410739+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[21463]: cluster 2024-04-26T16:23:15.466052+0000 mgr.y (mgr.24401) 213 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[21463]: cluster 2024-04-26T16:23:15.466052+0000 mgr.y (mgr.24401) 213 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[29234]: cluster 2024-04-26T16:23:15.409185+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[29234]: cluster 2024-04-26T16:23:15.409185+0000 osd.0 (osd.0) 133 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:23:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[29234]: cluster 2024-04-26T16:23:15.410739+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:23:16.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[29234]: cluster 2024-04-26T16:23:15.410739+0000 osd.0 (osd.0) 134 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:23:16.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[29234]: cluster 2024-04-26T16:23:15.466052+0000 mgr.y (mgr.24401) 213 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:23:16.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:16 smithi050 bash[29234]: cluster 2024-04-26T16:23:15.466052+0000 mgr.y (mgr.24401) 213 : cluster [DBG] pgmap v186: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:23:16.386 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:16.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:16.513 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:16.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:16.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:16.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:16.774 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:16.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:23:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:23:16.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:16.904 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:16.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:17.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:17.033 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:17.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:17.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:17.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:17.276 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:17 smithi190 bash[23868]: cluster 2024-04-26T16:23:16.416137+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:23:17.276 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:17 smithi190 bash[23868]: cluster 2024-04-26T16:23:16.416137+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:23:17.276 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:17 smithi190 bash[23868]: cluster 2024-04-26T16:23:16.418461+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:23:17.276 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:17 smithi190 bash[23868]: cluster 2024-04-26T16:23:16.418461+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:23:17.276 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:17 smithi190 bash[23868]: audit 2024-04-26T16:23:16.592706+0000 mgr.y (mgr.24401) 214 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:17.276 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:17 smithi190 bash[23868]: audit 2024-04-26T16:23:16.592706+0000 mgr.y (mgr.24401) 214 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:17.292 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:17.324 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:23:17.324 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:23:17.324 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-26T16:23:17.324 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-26T16:23:17.324 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:23:17.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[29234]: cluster 2024-04-26T16:23:16.416137+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[29234]: cluster 2024-04-26T16:23:16.416137+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[29234]: cluster 2024-04-26T16:23:16.418461+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[29234]: cluster 2024-04-26T16:23:16.418461+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[29234]: audit 2024-04-26T16:23:16.592706+0000 mgr.y (mgr.24401) 214 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[29234]: audit 2024-04-26T16:23:16.592706+0000 mgr.y (mgr.24401) 214 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[21463]: cluster 2024-04-26T16:23:16.416137+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[21463]: cluster 2024-04-26T16:23:16.416137+0000 osd.0 (osd.0) 135 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[21463]: cluster 2024-04-26T16:23:16.418461+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:23:17.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[21463]: cluster 2024-04-26T16:23:16.418461+0000 osd.0 (osd.0) 136 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:23:17.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[21463]: audit 2024-04-26T16:23:16.592706+0000 mgr.y (mgr.24401) 214 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:17.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:17 smithi050 bash[21463]: audit 2024-04-26T16:23:16.592706+0000 mgr.y (mgr.24401) 214 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:17.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:17.423 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:17.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:17.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:17.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:17.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:17.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 -- 172.21.15.50:0/3679363820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498102ce0 msgr2=0x7fb49810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 --2- 172.21.15.50:0/3679363820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498102ce0 0x7fb49810f1d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b0a0 tx=0x7fb49000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 -- 172.21.15.50:0/3679363820 shutdown_connections 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 --2- 172.21.15.50:0/3679363820 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb498108390 0x7fb498108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 --2- 172.21.15.50:0/3679363820 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498102ce0 0x7fb49810f1d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 --2- 172.21.15.50:0/3679363820 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 0x7fb4981027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.792+0000 7fb49f237700 1 -- 172.21.15.50:0/3679363820 >> 172.21.15.50:0/3679363820 conn(0x7fb4980792d0 msgr2=0x7fb4980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- 172.21.15.50:0/3679363820 shutdown_connections 2024-04-26T16:23:17.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- 172.21.15.50:0/3679363820 wait complete. 2024-04-26T16:23:17.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 librados: starting msgr at 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 librados: starting objecter 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 Processor -- start 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- start start 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 librados: setting wanted keys 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 librados: calling monclient init 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 0x7fb498197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb498102ce0 0x7fb498198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498108390 0x7fb498198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb4981119f0 con 0x7fb498102320 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb498111870 con 0x7fb498102ce0 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb498111b70 con 0x7fb498108390 2024-04-26T16:23:17.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49cfd3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 0x7fb498197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49cfd3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 0x7fb498197d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52346/0 (socket says 172.21.15.50:52346) 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49cfd3700 1 -- 172.21.15.50:0/1324104873 learned_addr learned my addr 172.21.15.50:0/1324104873 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498108390 0x7fb498198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb497fff700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb498102ce0 0x7fb498198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 -- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb498102ce0 msgr2=0x7fb498198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb498102ce0 0x7fb498198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 -- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 msgr2=0x7fb498197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:17.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 0x7fb498197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 -- 172.21.15.50:0/1324104873 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4981a7580 con 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb497fff700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb498102ce0 0x7fb498198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49cfd3700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb498102320 0x7fb498197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49d7d4700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498108390 0x7fb498198790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b070 tx=0x7fb490004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb495ffb700 10 client.?.objecter ms_handle_connect 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb495ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb490021070 con 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- 172.21.15.50:0/1324104873 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4981a7810 con 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb490007b80 con 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 10 client.24695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 -- 172.21.15.50:0/1324104873 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4981a7cc0 con 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb49f237700 1 librados: init done 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb490005040 con 0x7fb498108390 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb4837fe700 10 client.24695.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:17.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb4837fe700 15 client.24695.objecter _session_command_op_assign -1 1 2024-04-26T16:23:17.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb4837fe700 20 client.24695.objecter _get_session osd=-1 returning homeless 2024-04-26T16:23:17.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb4837fe700 10 client.24695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:17.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.796+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb478002920 con 0x7fb498108390 2024-04-26T16:23:17.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb490024070 con 0x7fb498108390 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb488071f80 0x7fb488074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(138..138 src has 87..138) ==== 7450+0+0 (secure 0 0 0) 0x7fb49000f020 con 0x7fb498108390 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 10 client.24695.objecter ms_dispatch 0x7fb498001550 osd_map(138..138 src has 87..138) 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 3 client.24695.objecter handle_osd_map got epochs [138,138] > 0 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 3 client.24695.objecter handle_osd_map decoding full epoch 138 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb49cfd3700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb488071f80 0x7fb488074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 10 client.24695.objecter checking command 1 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb48807bc80 0x7fb48807e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 20 client.24695.objecter _get_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 15 client.24695.objecter _session_command_op_remove -1 1 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 20 client.24695.objecter _get_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 20 client.24695.objecter get_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 15 client.24695.objecter _session_command_op_assign 4 1 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 4 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 10 client.24695.objecter _send_command 1 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb48807e7b0 con 0x7fb48807bc80 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 20 client.24695.objecter dump_active .. 0 homeless 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=138) ==== 24+0+0 (secure 0 0 0) 0x7fb490099050 con 0x7fb498108390 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb497fff700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb48807bc80 0x7fb48807e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb49cfd3700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb488071f80 0x7fb488074440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0104e0 tx=0x7fb48c00b480 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 10 client.24695.objecter ms_handle_connect 0x7fb488071f80 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb497fff700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb48807bc80 0x7fb48807e0e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:17.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.800+0000 7fb495ffb700 10 client.24695.objecter ms_handle_connect 0x7fb48807bc80 2024-04-26T16:23:17.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.804+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb48807e7b0 con 0x7fb48807bc80 2024-04-26T16:23:17.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.804+0000 7fb495ffb700 10 client.24695.objecter ms_dispatch 0x7fb498001550 command_reply(tid 1: 0 ) 2024-04-26T16:23:17.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.804+0000 7fb495ffb700 10 client.24695.objecter _finish_command 1 = system:0 2024-04-26T16:23:17.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.804+0000 7fb495ffb700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 4 2024-04-26T16:23:17.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.804+0000 7fb495ffb700 15 client.24695.objecter _session_command_op_remove 4 1 2024-04-26T16:23:17.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:17.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 10 client.24695.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 15 client.24695.objecter _session_command_op_assign -1 2 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter _get_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter _get_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 15 client.24695.objecter _session_command_op_remove -1 2 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter get_session s=0x7fb48807b5b0 osd=4 3 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 15 client.24695.objecter _session_command_op_assign 4 2 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 4 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 10 client.24695.objecter _send_command 2 2024-04-26T16:23:17.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7fb478001220 con 0x7fb48807bc80 2024-04-26T16:23:17.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb495ffb700 1 -- 172.21.15.50:0/1324104873 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7fb478001220 con 0x7fb48807bc80 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb495ffb700 10 client.24695.objecter ms_dispatch 0x7fb498001550 command_reply(tid 2: 0 ) 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb495ffb700 10 client.24695.objecter _finish_command 2 = system:0 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb495ffb700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 4 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb495ffb700 15 client.24695.objecter _session_command_op_remove 4 2 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 10 librados: watch_flush enter 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 10 librados: watch_flush exit 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 10 client.24695.objecter close_session for osd.4 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb48807bc80 msgr2=0x7fb48807e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb48807bc80 0x7fb48807e0e0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter put_session s=0x7fb48807b5b0 osd=4 1 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 20 client.24695.objecter shutdown clearing up homeless session... 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 10 client.24695.objecter successfully canceled tick 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb488071f80 msgr2=0x7fb488074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:17.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb488071f80 0x7fb488074440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb48c0104e0 tx=0x7fb48c00b480 comp rx=0 tx=0).stop 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498108390 msgr2=0x7fb498198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 --2- 172.21.15.50:0/1324104873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb498108390 0x7fb498198790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb49000b070 tx=0x7fb490004c80 comp rx=0 tx=0).stop 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb49d7d4700 1 -- 172.21.15.50:0/1324104873 reap_dead start 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 shutdown_connections 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 >> 172.21.15.50:0/1324104873 conn(0x7fb4980792d0 msgr2=0x7fb498105f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 shutdown_connections 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 -- 172.21.15.50:0/1324104873 wait complete. 2024-04-26T16:23:17.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:17.824+0000 7fb4837fe700 1 librados: shutdown 2024-04-26T16:23:17.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:17.931 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:17.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:17.953 INFO:tasks.workunit.client.0.smithi050.stderr:Updating files: 78% (9630/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-26T16:23:18.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:18.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:18.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.038831+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:23:18.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.038831+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:23:18.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.040394+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:23:18.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.040394+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:23:18.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.310676+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.310676+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.311729+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.311729+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.388851+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.388851+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.390008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.390008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.467500+0000 mgr.y (mgr.24401) 215 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:18.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:18 smithi190 bash[23868]: cluster 2024-04-26T16:23:17.467500+0000 mgr.y (mgr.24401) 215 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:18.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:18.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.038831+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:23:18.298 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.038831+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.040394+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.040394+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.310676+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.310676+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.311729+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.311729+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.388851+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.388851+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.390008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.390008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.467500+0000 mgr.y (mgr.24401) 215 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:18.299 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[29234]: cluster 2024-04-26T16:23:17.467500+0000 mgr.y (mgr.24401) 215 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.038831+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.038831+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.040394+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.040394+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.310676+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.310676+0000 osd.7 (osd.7) 149 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.311729+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.311729+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:23:18.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.388851+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:23:18.301 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.388851+0000 osd.0 (osd.0) 137 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:23:18.301 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.390008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:23:18.301 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.390008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:23:18.301 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.467500+0000 mgr.y (mgr.24401) 215 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:18.301 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:18 smithi050 bash[21463]: cluster 2024-04-26T16:23:17.467500+0000 mgr.y (mgr.24401) 215 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:18.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:18.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:18.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:18.567 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:18.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:18.697 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:18.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:18.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:18.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:19.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.032859+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.032859+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.034273+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.034273+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.279450+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.279450+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.281172+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.281172+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.436025+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.436025+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.438601+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:23:19.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.438601+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.032859+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.032859+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.034273+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.034273+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.279450+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.279450+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.281172+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.281172+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.436025+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:23:19.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.436025+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:23:19.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.438601+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:23:19.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:19 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.438601+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:23:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:19.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:19.206 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:19.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:19.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.032859+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:23:19.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.032859+0000 osd.1 (osd.1) 147 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.034273+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.034273+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.279450+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.279450+0000 osd.7 (osd.7) 151 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.281172+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.281172+0000 osd.7 (osd.7) 152 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.436025+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.436025+0000 osd.0 (osd.0) 139 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.438601+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:23:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:19 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.438601+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:23:19.336 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr:Note: switching to 'a430f9ed84de3008ca5b632b1ed304bac0e753f6'. 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr:state without impacting any branches by switching back to a branch. 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.397 INFO:tasks.workunit.client.0.smithi050.stderr: git switch -c 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr:Or undo this operation with: 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr: git switch - 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr: 2024-04-26T16:23:19.398 INFO:tasks.workunit.client.0.smithi050.stderr:HEAD is now at a430f9ed84d Merge branch 'jjm-mgr-smb-module' of https://github.com/phlogistonjohn/ceph into wip-phlogistonjohn-testing-2024-04-25-1915 2024-04-26T16:23:19.406 DEBUG:teuthology.orchestra.run.smithi050:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-26T16:23:19.413 INFO:tasks.workunit.client.0.smithi050.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-26T16:23:19.416 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T16:23:19.416 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-26T16:23:19.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:19.466 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:19.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:19.600 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:19.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:19.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:19.734 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:19.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:19.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:19.864 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:19.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:19.994 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:19.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:19.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:20.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:20.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:20.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:20.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.545483+0000 osd.2 (osd.2) 105 : cluster [DBG] 6.f scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.545483+0000 osd.2 (osd.2) 105 : cluster [DBG] 6.f scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.547847+0000 osd.2 (osd.2) 106 : cluster [DBG] 6.f scrub ok 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:18.547847+0000 osd.2 (osd.2) 106 : cluster [DBG] 6.f scrub ok 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.003343+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.003343+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.005629+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.005629+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.107666+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.107666+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.109902+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.109902+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.169610+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.169610+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.169707+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.169707+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.272728+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.272728+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.274029+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.274029+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.468076+0000 mgr.y (mgr.24401) 216 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.468076+0000 mgr.y (mgr.24401) 216 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.832284+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.832284+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.832440+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:20 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.832440+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:20.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.545483+0000 osd.2 (osd.2) 105 : cluster [DBG] 6.f scrub starts 2024-04-26T16:23:20.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.545483+0000 osd.2 (osd.2) 105 : cluster [DBG] 6.f scrub starts 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.547847+0000 osd.2 (osd.2) 106 : cluster [DBG] 6.f scrub ok 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:18.547847+0000 osd.2 (osd.2) 106 : cluster [DBG] 6.f scrub ok 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.003343+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.003343+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.005629+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.005629+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.107666+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.107666+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.109902+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.109902+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:23:20.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.169610+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.169610+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.169707+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.169707+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.272728+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.272728+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.274029+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.274029+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.468076+0000 mgr.y (mgr.24401) 216 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.468076+0000 mgr.y (mgr.24401) 216 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.832284+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.832284+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.832440+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.832440+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.545483+0000 osd.2 (osd.2) 105 : cluster [DBG] 6.f scrub starts 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.545483+0000 osd.2 (osd.2) 105 : cluster [DBG] 6.f scrub starts 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.547847+0000 osd.2 (osd.2) 106 : cluster [DBG] 6.f scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:18.547847+0000 osd.2 (osd.2) 106 : cluster [DBG] 6.f scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.003343+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.003343+0000 osd.6 (osd.6) 89 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.005629+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.005629+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.107666+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.107666+0000 osd.5 (osd.5) 139 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.109902+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.109902+0000 osd.5 (osd.5) 140 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:23:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.169610+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.169610+0000 osd.3 (osd.3) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.169707+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.169707+0000 osd.3 (osd.3) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.272728+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.272728+0000 osd.7 (osd.7) 153 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.274029+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.274029+0000 osd.7 (osd.7) 154 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.468076+0000 mgr.y (mgr.24401) 216 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.468076+0000 mgr.y (mgr.24401) 216 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 583 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.832284+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.832284+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.832440+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:20 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.832440+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:20.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:20.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:20.516 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:20.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:20.645 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:20.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:20.777 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:20.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:20.908 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:20.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:21.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:21.037 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:21.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:21.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:21.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:21.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:21 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.292918+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:23:21.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:21 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.292918+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:23:21.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:21 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.294893+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:23:21.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:21 smithi190 bash[23868]: cluster 2024-04-26T16:23:19.294893+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:23:21.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:21 smithi190 bash[23868]: cluster 2024-04-26T16:23:20.026426+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-04-26T16:23:21.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:21 smithi190 bash[23868]: cluster 2024-04-26T16:23:20.026426+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-04-26T16:23:21.300 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:21.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.292918+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:23:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.292918+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:23:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.294893+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:23:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[29234]: cluster 2024-04-26T16:23:19.294893+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:23:21.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[29234]: cluster 2024-04-26T16:23:20.026426+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[29234]: cluster 2024-04-26T16:23:20.026426+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.292918+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.292918+0000 osd.3 (osd.3) 153 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.294893+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[21463]: cluster 2024-04-26T16:23:19.294893+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[21463]: cluster 2024-04-26T16:23:20.026426+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-04-26T16:23:21.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:21 smithi050 bash[21463]: cluster 2024-04-26T16:23:20.026426+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2024-04-26T16:23:21.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:21.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:21.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:21.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:21.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:21.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:21.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:21.932 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:21.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:21.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:22.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:22.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:22.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:22.204 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.056289+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.056289+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.058547+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.058547+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.122166+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.122166+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.124338+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.124338+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.469027+0000 mgr.y (mgr.24401) 217 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[29234]: cluster 2024-04-26T16:23:21.469027+0000 mgr.y (mgr.24401) 217 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:22.205 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.056289+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.056289+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.058547+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.058547+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.122166+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.122166+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.124338+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.124338+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.469027+0000 mgr.y (mgr.24401) 217 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:22.206 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:22 smithi050 bash[21463]: cluster 2024-04-26T16:23:21.469027+0000 mgr.y (mgr.24401) 217 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:22.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.056289+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.056289+0000 osd.1 (osd.1) 149 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.058547+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.058547+0000 osd.1 (osd.1) 150 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.122166+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.122166+0000 osd.5 (osd.5) 141 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.124338+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.124338+0000 osd.5 (osd.5) 142 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.469027+0000 mgr.y (mgr.24401) 217 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:22 smithi190 bash[23868]: cluster 2024-04-26T16:23:21.469027+0000 mgr.y (mgr.24401) 217 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:23:22.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:22.348 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:22.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:22.478 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:22.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:22.608 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:22.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:22.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:22.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:22.872 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:22.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:22.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:23.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:23.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:23.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:23.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:23.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:23 smithi190 bash[23868]: cluster 2024-04-26T16:23:22.124150+0000 osd.5 (osd.5) 143 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:23:23.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:23 smithi190 bash[23868]: cluster 2024-04-26T16:23:22.124150+0000 osd.5 (osd.5) 143 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:23:23.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:23 smithi190 bash[23868]: cluster 2024-04-26T16:23:22.126426+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:23:23.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:23 smithi190 bash[23868]: cluster 2024-04-26T16:23:22.126426+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:23:23.263 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:23.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[21463]: cluster 2024-04-26T16:23:22.124150+0000 osd.5 (osd.5) 143 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:23:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[21463]: cluster 2024-04-26T16:23:22.124150+0000 osd.5 (osd.5) 143 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:23:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[21463]: cluster 2024-04-26T16:23:22.126426+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:23:23.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[21463]: cluster 2024-04-26T16:23:22.126426+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:23:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[29234]: cluster 2024-04-26T16:23:22.124150+0000 osd.5 (osd.5) 143 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:23:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[29234]: cluster 2024-04-26T16:23:22.124150+0000 osd.5 (osd.5) 143 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:23:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[29234]: cluster 2024-04-26T16:23:22.126426+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:23:23.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:23 smithi050 bash[29234]: cluster 2024-04-26T16:23:22.126426+0000 osd.5 (osd.5) 144 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:23:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:23.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.523 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:23.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.655 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:23.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:23.788 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:23.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.919 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:23.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:24.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:24.037 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.105513+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.4 deep-scrub starts 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.105513+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.4 deep-scrub starts 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.108332+0000 osd.1 (osd.1) 152 : cluster [DBG] 2.4 deep-scrub ok 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.108332+0000 osd.1 (osd.1) 152 : cluster [DBG] 2.4 deep-scrub ok 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.129786+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.129786+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.132100+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.132100+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.246798+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.246798+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.251287+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.251287+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.469815+0000 mgr.y (mgr.24401) 218 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:24.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.469815+0000 mgr.y (mgr.24401) 218 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:24.050 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:24.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.105513+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.4 deep-scrub starts 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.105513+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.4 deep-scrub starts 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.108332+0000 osd.1 (osd.1) 152 : cluster [DBG] 2.4 deep-scrub ok 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.108332+0000 osd.1 (osd.1) 152 : cluster [DBG] 2.4 deep-scrub ok 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.129786+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.129786+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.132100+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.132100+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.246798+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.246798+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.251287+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:23:24.165 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.251287+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:23:24.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.469815+0000 mgr.y (mgr.24401) 218 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:24.166 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:24 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.469815+0000 mgr.y (mgr.24401) 218 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:24.181 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:24.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:24.312 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:24.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:24.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.105513+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.4 deep-scrub starts 2024-04-26T16:23:24.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.105513+0000 osd.1 (osd.1) 151 : cluster [DBG] 2.4 deep-scrub starts 2024-04-26T16:23:24.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.108332+0000 osd.1 (osd.1) 152 : cluster [DBG] 2.4 deep-scrub ok 2024-04-26T16:23:24.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.108332+0000 osd.1 (osd.1) 152 : cluster [DBG] 2.4 deep-scrub ok 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.129786+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.129786+0000 osd.5 (osd.5) 145 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.132100+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.132100+0000 osd.5 (osd.5) 146 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.246798+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.246798+0000 osd.7 (osd.7) 157 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.251287+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.251287+0000 osd.7 (osd.7) 158 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.469815+0000 mgr.y (mgr.24401) 218 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.469815+0000 mgr.y (mgr.24401) 218 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:24.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:24.444 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:24.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:24.573 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:24.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-04-26T16:23:24.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-26T16:23:24.671 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:23:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:23:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:23:24.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:24.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:24.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:24.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:24.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:24.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:24.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:25.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:25.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:25.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.628540+0000 osd.4 (osd.4) 119 : cluster [DBG] 2.f scrub starts 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.628540+0000 osd.4 (osd.4) 119 : cluster [DBG] 2.f scrub starts 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.630678+0000 osd.4 (osd.4) 120 : cluster [DBG] 2.f scrub ok 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[29234]: cluster 2024-04-26T16:23:23.630678+0000 osd.4 (osd.4) 120 : cluster [DBG] 2.f scrub ok 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.628540+0000 osd.4 (osd.4) 119 : cluster [DBG] 2.f scrub starts 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.628540+0000 osd.4 (osd.4) 119 : cluster [DBG] 2.f scrub starts 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.630678+0000 osd.4 (osd.4) 120 : cluster [DBG] 2.f scrub ok 2024-04-26T16:23:25.208 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:25 smithi050 bash[21463]: cluster 2024-04-26T16:23:23.630678+0000 osd.4 (osd.4) 120 : cluster [DBG] 2.f scrub ok 2024-04-26T16:23:25.212 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:25.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:25.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:25 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.628540+0000 osd.4 (osd.4) 119 : cluster [DBG] 2.f scrub starts 2024-04-26T16:23:25.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:25 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.628540+0000 osd.4 (osd.4) 119 : cluster [DBG] 2.f scrub starts 2024-04-26T16:23:25.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:25 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.630678+0000 osd.4 (osd.4) 120 : cluster [DBG] 2.f scrub ok 2024-04-26T16:23:25.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:25 smithi190 bash[23868]: cluster 2024-04-26T16:23:23.630678+0000 osd.4 (osd.4) 120 : cluster [DBG] 2.f scrub ok 2024-04-26T16:23:25.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/2271764901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac1094a0 msgr2=0x7f53ac109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- 172.21.15.50:0/2271764901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac1094a0 0x7f53ac109880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f53a80098f0 tx=0x7f53a8009c00 comp rx=0 tx=0).stop 2024-04-26T16:23:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/2271764901 shutdown_connections 2024-04-26T16:23:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- 172.21.15.50:0/2271764901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53ac06bad0 0x7f53ac1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- 172.21.15.50:0/2271764901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac1094a0 0x7f53ac109880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- 172.21.15.50:0/2271764901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac06b130 0x7f53ac06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/2271764901 >> 172.21.15.50:0/2271764901 conn(0x7f53ac0792d0 msgr2=0x7f53ac0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/2271764901 shutdown_connections 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/2271764901 wait complete. 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 librados: starting msgr at 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 librados: starting objecter 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 Processor -- start 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- start start 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 librados: setting wanted keys 2024-04-26T16:23:25.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 librados: calling monclient init 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53ac06b130 0x7f53ac1023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac06bad0 0x7f53ac102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac1094a0 0x7f53ac102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac112ab0 con 0x7f53ac1094a0 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac112930 con 0x7f53ac06b130 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac112c30 con 0x7f53ac06bad0 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b0b8b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac06bad0 0x7f53ac102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac1094a0 0x7f53ac102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b0b8b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac06bad0 0x7f53ac102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38650/0 (socket says 172.21.15.50:38650) 2024-04-26T16:23:25.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b0b8b700 1 -- 172.21.15.50:0/3135889021 learned_addr learned my addr 172.21.15.50:0/3135889021 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53ac06b130 0x7f53ac1023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac06bad0 msgr2=0x7f53ac102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac06bad0 0x7f53ac102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53ac06b130 msgr2=0x7f53ac1023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53ac06b130 0x7f53ac1023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac1a76e0 con 0x7f53ac1094a0 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f53ac06bad0 0x7f53ac102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac1094a0 0x7f53ac102e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53a4007f10 tx=0x7f53a400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f539e7fc700 10 client.?.objecter ms_handle_connect 0x7f53ac1094a0 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f539e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a400ddd0 con 0x7f53ac1094a0 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f53a4009e20 con 0x7f53ac1094a0 2024-04-26T16:23:25.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac1a79d0 con 0x7f53ac1094a0 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4005a40 con 0x7f53ac1094a0 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 10 client.14811.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac1a7f60 con 0x7f53ac1094a0 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.368+0000 7f53b35f0700 1 librados: init done 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f53ac06a110 con 0x7f53ac1094a0 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f53a4005000 con 0x7f53ac1094a0 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5398071f80 0x7f5398074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(139..139 src has 87..139) ==== 7450+0+0 (secure 0 0 0) 0x7f53a4091640 con 0x7f53ac1094a0 2024-04-26T16:23:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 osd_map(139..139 src has 87..139) 2024-04-26T16:23:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 3 client.14811.objecter handle_osd_map got epochs [139,139] > 0 2024-04-26T16:23:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 3 client.14811.objecter handle_osd_map decoding full epoch 139 2024-04-26T16:23:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5398071f80 0x7f5398074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 20 client.14811.objecter dump_active .. 0 homeless 2024-04-26T16:23:25.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v139) ==== 54+0+16 (secure 0 0 0) 0x7f53a4010070 con 0x7f53ac1094a0 2024-04-26T16:23:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 10 client.14811.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 1 2024-04-26T16:23:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f53ac064030 0x7f53ac064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 1 2024-04-26T16:23:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 0 1 2024-04-26T16:23:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 4 2024-04-26T16:23:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 10 client.14811.objecter _send_command 1 2024-04-26T16:23:25.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53ac1ab320 con 0x7f53ac064030 2024-04-26T16:23:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f5398071f80 2024-04-26T16:23:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5398071f80 0x7f5398074440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009810 tx=0x7f53a8019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f53ac064030 0x7f53ac064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac064030 2024-04-26T16:23:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f53ac064030 0x7f53ac064410 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac1ab320 con 0x7f53ac064030 2024-04-26T16:23:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 1: 0 ) 2024-04-26T16:23:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 10 client.14811.objecter _finish_command 1 = system:0 2024-04-26T16:23:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 4 2024-04-26T16:23:25.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.372+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 0 1 2024-04-26T16:23:25.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 10 client.14811.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 2 2024-04-26T16:23:25.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 2 2024-04-26T16:23:25.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac103920 osd=0 3 2024-04-26T16:23:25.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 0 2 2024-04-26T16:23:25.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 4 2024-04-26T16:23:25.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 10 client.14811.objecter _send_command 2 2024-04-26T16:23:25.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac063ab0 con 0x7f53ac064030 2024-04-26T16:23:25.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac063ab0 con 0x7f53ac064030 2024-04-26T16:23:25.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f539e7fc700 10 client.14811.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 4 2024-04-26T16:23:25.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 0 2 2024-04-26T16:23:25.392 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:23:25.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 10 client.14811.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f53ac02d310 0x7f53ac02d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 1 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 4 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 10 client.14811.objecter _send_command 3 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f53ac10b050 con 0x7f53ac02d310 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.392+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f53ac02d310 0x7f53ac02d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac02d310 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f53ac02d310 0x7f53ac02d6f0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac10b050 con 0x7f53ac02d310 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 3: 0 ) 2024-04-26T16:23:25.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f539e7fc700 10 client.14811.objecter _finish_command 3 = system:0 2024-04-26T16:23:25.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 4 2024-04-26T16:23:25.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.396+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 1 3 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 10 client.14811.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 4 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 4 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac10a9d0 osd=1 3 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 1 4 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 4 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 10 client.14811.objecter _send_command 4 2024-04-26T16:23:25.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.412+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac10a450 con 0x7f53ac02d310 2024-04-26T16:23:25.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac10a450 con 0x7f53ac02d310 2024-04-26T16:23:25.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 10 client.14811.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 4 2024-04-26T16:23:25.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 1 4 2024-04-26T16:23:25.411 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 10 client.14811.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 5 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f53ac1ad4e0 0x7f53ac1af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 5 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 2 5 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 4 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 10 client.14811.objecter _send_command 5 2024-04-26T16:23:25.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f53ac1b2730 con 0x7f53ac1ad4e0 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f53ac1ad4e0 0x7f53ac1af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac1ad4e0 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f53ac1ad4e0 0x7f53ac1af8d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac1b2730 con 0x7f53ac1ad4e0 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 5: 0 ) 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 10 client.14811.objecter _finish_command 5 = system:0 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 4 2024-04-26T16:23:25.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.416+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 2 5 2024-04-26T16:23:25.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 10 client.14811.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 6 2024-04-26T16:23:25.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 6 2024-04-26T16:23:25.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac10b200 osd=2 3 2024-04-26T16:23:25.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 2 6 2024-04-26T16:23:25.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 4 2024-04-26T16:23:25.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 10 client.14811.objecter _send_command 6 2024-04-26T16:23:25.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac02de80 con 0x7f53ac1ad4e0 2024-04-26T16:23:25.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac02de80 con 0x7f53ac1ad4e0 2024-04-26T16:23:25.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 10 client.14811.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 4 2024-04-26T16:23:25.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 2 6 2024-04-26T16:23:25.432 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 10 client.14811.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 7 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f53ac1aae80 0x7f53ac1b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 7 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 3 7 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 4 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 10 client.14811.objecter _send_command 7 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f53ac02de80 con 0x7f53ac1aae80 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f53ac1aae80 0x7f53ac1b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac1aae80 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f53ac1aae80 0x7f53ac1b0f90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac02de80 con 0x7f53ac1aae80 2024-04-26T16:23:25.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 7: 0 ) 2024-04-26T16:23:25.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 10 client.14811.objecter _finish_command 7 = system:0 2024-04-26T16:23:25.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 4 2024-04-26T16:23:25.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.436+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 3 7 2024-04-26T16:23:25.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 10 client.14811.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 8 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 8 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac02e910 osd=3 3 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 3 8 2024-04-26T16:23:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 4 2024-04-26T16:23:25.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 10 client.14811.objecter _send_command 8 2024-04-26T16:23:25.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac02de80 con 0x7f53ac1aae80 2024-04-26T16:23:25.458 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:23:25.458 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac02de80 con 0x7f53ac1aae80 2024-04-26T16:23:25.458 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f539e7fc700 10 client.14811.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 4 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.460+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 3 8 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 10 client.14811.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 9 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f53ac1b05e0 0x7f53ac1b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 9 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 4 9 2024-04-26T16:23:25.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 4 2024-04-26T16:23:25.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 10 client.14811.objecter _send_command 9 2024-04-26T16:23:25.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f53ac1b0c00 con 0x7f53ac1b05e0 2024-04-26T16:23:25.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f53ac1b05e0 0x7f53ac1b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f53ac1b05e0 0x7f53ac1b6900 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac1b05e0 2024-04-26T16:23:25.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:25.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac1b0c00 con 0x7f53ac1b05e0 2024-04-26T16:23:25.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 9: 0 ) 2024-04-26T16:23:25.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f539e7fc700 10 client.14811.objecter _finish_command 9 = system:0 2024-04-26T16:23:25.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 4 2024-04-26T16:23:25.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.464+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 4 9 2024-04-26T16:23:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:25.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 10 client.14811.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 10 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 10 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac02de80 osd=4 3 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 4 10 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 4 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 10 client.14811.objecter _send_command 10 2024-04-26T16:23:25.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.484+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac1b0c00 con 0x7f53ac1b05e0 2024-04-26T16:23:25.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac1b0c00 con 0x7f53ac1b05e0 2024-04-26T16:23:25.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f539e7fc700 10 client.14811.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 4 2024-04-26T16:23:25.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 4 10 2024-04-26T16:23:25.485 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 10 client.14811.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 11 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f53ac1b7350 0x7f53ac1b9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 11 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 5 11 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 4 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 10 client.14811.objecter _send_command 11 2024-04-26T16:23:25.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f53ac1bc5f0 con 0x7f53ac1b7350 2024-04-26T16:23:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.488+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f53ac1b7350 0x7f53ac1b9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f53b0b8b700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f53ac1b7350 0x7f53ac1b9740 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac1b7350 2024-04-26T16:23:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac1bc5f0 con 0x7f53ac1b7350 2024-04-26T16:23:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 11: 0 ) 2024-04-26T16:23:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f539e7fc700 10 client.14811.objecter _finish_command 11 = system:0 2024-04-26T16:23:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 4 2024-04-26T16:23:25.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.492+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 5 11 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.508+0000 7f53b35f0700 10 client.14811.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 12 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 12 2024-04-26T16:23:25.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac1aa1a0 osd=5 3 2024-04-26T16:23:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 5 12 2024-04-26T16:23:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 4 2024-04-26T16:23:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 10 client.14811.objecter _send_command 12 2024-04-26T16:23:25.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.512+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac1b9c80 con 0x7f53ac1b7350 2024-04-26T16:23:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac1b9c80 con 0x7f53ac1b7350 2024-04-26T16:23:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 10 client.14811.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 4 2024-04-26T16:23:25.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 5 12 2024-04-26T16:23:25.511 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:23:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 10 client.14811.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 13 2024-04-26T16:23:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f53ac1ba3a0 0x7f53ac1ba7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 13 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 6 13 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 4 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 10 client.14811.objecter _send_command 13 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f53ac1bf9e0 con 0x7f53ac1ba3a0 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f53ac1ba3a0 0x7f53ac1ba7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac1ba3a0 2024-04-26T16:23:25.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f53b1b8d700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f53ac1ba3a0 0x7f53ac1ba7e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac1bf9e0 con 0x7f53ac1ba3a0 2024-04-26T16:23:25.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 13: 0 ) 2024-04-26T16:23:25.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 10 client.14811.objecter _finish_command 13 = system:0 2024-04-26T16:23:25.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 4 2024-04-26T16:23:25.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.516+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 6 13 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 10 client.14811.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 14 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 14 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac1b9c80 osd=6 3 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 6 14 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 4 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 10 client.14811.objecter _send_command 14 2024-04-26T16:23:25.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.536+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac1bf9e0 con 0x7f53ac1ba3a0 2024-04-26T16:23:25.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac1bf9e0 con 0x7f53ac1ba3a0 2024-04-26T16:23:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f539e7fc700 10 client.14811.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 4 2024-04-26T16:23:25.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 6 14 2024-04-26T16:23:25.536 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 10 client.14811.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 15 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f53ac1bedd0 0x7f53ac1bf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 15 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 7 15 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 4 2024-04-26T16:23:25.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 10 client.14811.objecter _send_command 15 2024-04-26T16:23:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f53ac1c2d50 con 0x7f53ac1bedd0 2024-04-26T16:23:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f53ac1bedd0 0x7f53ac1bf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f539e7fc700 10 client.14811.objecter ms_handle_connect 0x7f53ac1bedd0 2024-04-26T16:23:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.540+0000 7f53b138c700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f53ac1bedd0 0x7f53ac1bf250 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.544+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53ac1c2d50 con 0x7f53ac1bedd0 2024-04-26T16:23:25.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.544+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 15: 0 ) 2024-04-26T16:23:25.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.544+0000 7f539e7fc700 10 client.14811.objecter _finish_command 15 = system:0 2024-04-26T16:23:25.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.544+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 4 2024-04-26T16:23:25.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.544+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 7 15 2024-04-26T16:23:25.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 10 client.14811.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign -1 16 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 20 client.14811.objecter _get_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_remove -1 16 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 20 client.14811.objecter get_session s=0x7f53ac1bf9e0 osd=7 3 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 15 client.14811.objecter _session_command_op_assign 7 16 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 4 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 10 client.14811.objecter _send_command 16 2024-04-26T16:23:25.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.572+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f53ac1bad70 con 0x7f53ac1bedd0 2024-04-26T16:23:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f539e7fc700 1 -- 172.21.15.50:0/3135889021 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f53ac1bad70 con 0x7f53ac1bedd0 2024-04-26T16:23:25.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f539e7fc700 10 client.14811.objecter ms_dispatch 0x7f53ac001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f539e7fc700 10 client.14811.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f539e7fc700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 4 2024-04-26T16:23:25.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f539e7fc700 15 client.14811.objecter _session_command_op_remove 7 16 2024-04-26T16:23:25.572 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:23:25.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 librados: watch_flush enter 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 librados: watch_flush exit 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.0 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f53ac064030 msgr2=0x7f53ac064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f53ac064030 0x7f53ac064410 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac103920 osd=0 1 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.1 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f53ac02d310 msgr2=0x7f53ac02d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f53ac02d310 0x7f53ac02d6f0 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10a9d0 osd=1 1 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.2 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f53ac1ad4e0 msgr2=0x7f53ac1af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f53ac1ad4e0 0x7f53ac1af8d0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac10b200 osd=2 1 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.3 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f53ac1aae80 msgr2=0x7f53ac1b0f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b1b8d700 1 -- 172.21.15.50:0/3135889021 reap_dead start 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f53ac1aae80 0x7f53ac1b0f90 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02e910 osd=3 1 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.4 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f53ac1b05e0 msgr2=0x7f53ac1b6900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f53ac1b05e0 0x7f53ac1b6900 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac02de80 osd=4 1 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.5 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f53ac1b7350 msgr2=0x7f53ac1b9740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f53ac1b7350 0x7f53ac1b9740 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1aa1a0 osd=5 1 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.6 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f53ac1ba3a0 msgr2=0x7f53ac1ba7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f53ac1ba3a0 0x7f53ac1ba7e0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1b9c80 osd=6 1 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter close_session for osd.7 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f53ac1bedd0 msgr2=0x7f53ac1bf250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f53ac1bedd0 0x7f53ac1bf250 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter put_session s=0x7f53ac1bf9e0 osd=7 1 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 20 client.14811.objecter shutdown clearing up homeless session... 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 10 client.14811.objecter successfully canceled tick 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b1b8d700 1 -- 172.21.15.50:0/3135889021 reap_dead start 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5398071f80 msgr2=0x7f5398074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5398071f80 0x7f5398074440 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009810 tx=0x7f53a8019040 comp rx=0 tx=0).stop 2024-04-26T16:23:25.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac1094a0 msgr2=0x7f53ac102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac1094a0 0x7f53ac102e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53a4007f10 tx=0x7f53a400d3b0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 shutdown_connections 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f53ac1094a0 0x7f53ac102e70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 --2- 172.21.15.50:0/3135889021 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5398071f80 0x7f5398074440 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 >> 172.21.15.50:0/3135889021 conn(0x7f53ac0792d0 msgr2=0x7f53ac100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 shutdown_connections 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 -- 172.21.15.50:0/3135889021 wait complete. 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:25.576+0000 7f53b35f0700 1 librados: shutdown 2024-04-26T16:23:25.577 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:23:25.578 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:23:25.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:25.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:25.715 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:25.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:25.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:25.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:25.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:25.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:25.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:26.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:26.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:24.328275+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:24.328275+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:24.329912+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:24.329912+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: audit 2024-04-26T16:23:25.375645+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? 172.21.15.50:0/3135889021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: audit 2024-04-26T16:23:25.375645+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? 172.21.15.50:0/3135889021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:25.462967+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:25.462967+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:25.467383+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:25.467383+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:23:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:25.470280+0000 mgr.y (mgr.24401) 219 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: cluster 2024-04-26T16:23:25.470280+0000 mgr.y (mgr.24401) 219 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: audit 2024-04-26T16:23:25.995451+0000 mon.c (mon.2) 97 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[29234]: audit 2024-04-26T16:23:25.995451+0000 mon.c (mon.2) 97 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:24.328275+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:24.328275+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:24.329912+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:24.329912+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: audit 2024-04-26T16:23:25.375645+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? 172.21.15.50:0/3135889021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: audit 2024-04-26T16:23:25.375645+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? 172.21.15.50:0/3135889021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:25.462967+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:25.462967+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:23:26.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:25.467383+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:23:26.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:25.467383+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:23:26.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:25.470280+0000 mgr.y (mgr.24401) 219 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:26.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: cluster 2024-04-26T16:23:25.470280+0000 mgr.y (mgr.24401) 219 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:26.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: audit 2024-04-26T16:23:25.995451+0000 mon.c (mon.2) 97 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:26.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:26 smithi050 bash[21463]: audit 2024-04-26T16:23:25.995451+0000 mon.c (mon.2) 97 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:26.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:26.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:24.328275+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:24.328275+0000 osd.3 (osd.3) 155 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:24.329912+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:24.329912+0000 osd.3 (osd.3) 156 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: audit 2024-04-26T16:23:25.375645+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? 172.21.15.50:0/3135889021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: audit 2024-04-26T16:23:25.375645+0000 mon.a (mon.0) 972 : audit [DBG] from='client.? 172.21.15.50:0/3135889021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:25.462967+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:25.462967+0000 osd.0 (osd.0) 141 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:25.467383+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:25.467383+0000 osd.0 (osd.0) 142 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:25.470280+0000 mgr.y (mgr.24401) 219 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:26.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: cluster 2024-04-26T16:23:25.470280+0000 mgr.y (mgr.24401) 219 : cluster [DBG] pgmap v192: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:26.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: audit 2024-04-26T16:23:25.995451+0000 mon.c (mon.2) 97 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:26.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[23868]: audit 2024-04-26T16:23:25.995451+0000 mon.c (mon.2) 97 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:26.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:26.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:26.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:26.498 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:26.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:26.629 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:26.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:26.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:26.743 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:23:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:23:26.759 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:26.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:26.891 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:26.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:26.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:27.021 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:27.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:27.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:27.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:27.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:27 smithi190 bash[23868]: audit 2024-04-26T16:23:26.603986+0000 mgr.y (mgr.24401) 220 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:27.267 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:27 smithi190 bash[23868]: audit 2024-04-26T16:23:26.603986+0000 mgr.y (mgr.24401) 220 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:27.284 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:27.357 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-26T16:23:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:27 smithi050 bash[29234]: audit 2024-04-26T16:23:26.603986+0000 mgr.y (mgr.24401) 220 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:27.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:27 smithi050 bash[29234]: audit 2024-04-26T16:23:26.603986+0000 mgr.y (mgr.24401) 220 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:27.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:27 smithi050 bash[21463]: audit 2024-04-26T16:23:26.603986+0000 mgr.y (mgr.24401) 220 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:27.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:27 smithi050 bash[21463]: audit 2024-04-26T16:23:26.603986+0000 mgr.y (mgr.24401) 220 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:27.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:27.413 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:27.471 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-26T16:23:27.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:27.543 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:27.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:27.672 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:27.762 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-26T16:23:27.762 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T16:23:27.762 INFO:tasks.workunit.client.0.smithi050.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-26T16:23:27.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:27.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:27.817 INFO:tasks.workunit.client.0.smithi050.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-26T16:23:27.820 DEBUG:teuthology.orchestra.run.smithi050:> set -ex 2024-04-26T16:23:27.820 DEBUG:teuthology.orchestra.run.smithi050:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-26T16:23:27.867 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-04-26T16:23:27.868 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-04-26T16:23:27.868 DEBUG:teuthology.orchestra.run.smithi050:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a430f9ed84de3008ca5b632b1ed304bac0e753f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-04-26T16:23:27.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:27.919 INFO:tasks.workunit.client.0.smithi050.stderr:+ parallel=1 2024-04-26T16:23:27.919 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' '' = --serial ']' 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ crimson=0 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' '' = --crimson ']' 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ color= 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -t 1 ']' 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-26T16:23:27.920 INFO:tasks.workunit.client.0.smithi050.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-04-26T16:23:27.921 INFO:tasks.workunit.client.0.smithi050.stderr:+ declare -A pids 2024-04-26T16:23:27.921 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.921 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.921 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_aio 2024-04-26T16:23:27.921 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_aio' 2024-04-26T16:23:27.922 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_aio 2024-04-26T16:23:27.922 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.923 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_aio 2024-04-26T16:23:27.923 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73188 2024-04-26T16:23:27.924 INFO:tasks.workunit.client.0.smithi050.stdout:test api_aio on pid 73188 2024-04-26T16:23:27.925 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_aio on pid 73188' 2024-04-26T16:23:27.925 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73188 2024-04-26T16:23:27.925 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.925 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.925 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-04-26T16:23:27.925 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_aio_pp 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_aio_pp' 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_aio_pp 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_aio.log 2024-04-26T16:23:27.926 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_aio: /' 2024-04-26T16:23:27.927 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_aio_pp 2024-04-26T16:23:27.927 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73196 2024-04-26T16:23:27.927 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_aio_pp on pid 73196' 2024-04-26T16:23:27.928 INFO:tasks.workunit.client.0.smithi050.stdout:test api_aio_pp on pid 73196 2024-04-26T16:23:27.929 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73196 2024-04-26T16:23:27.929 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.929 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.929 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-04-26T16:23:27.929 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_io 2024-04-26T16:23:27.929 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_io' 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_io 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-04-26T16:23:27.930 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_aio_pp: /' 2024-04-26T16:23:27.931 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_io 2024-04-26T16:23:27.931 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73204 2024-04-26T16:23:27.931 INFO:tasks.workunit.client.0.smithi050.stdout:test api_io on pid 73204 2024-04-26T16:23:27.932 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_io on pid 73204' 2024-04-26T16:23:27.932 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73204 2024-04-26T16:23:27.932 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_io_pp 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_io_pp' 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_io_pp 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_io.log 2024-04-26T16:23:27.933 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_io: /' 2024-04-26T16:23:27.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:27.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:27.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:27.935 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_io_pp 2024-04-26T16:23:27.935 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73212 2024-04-26T16:23:27.935 INFO:tasks.workunit.client.0.smithi050.stdout:test api_io_pp on pid 73212 2024-04-26T16:23:27.936 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_io_pp on pid 73212' 2024-04-26T16:23:27.936 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73212 2024-04-26T16:23:27.936 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_asio 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_asio' 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_asio 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-04-26T16:23:27.937 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_io_pp: /' 2024-04-26T16:23:27.938 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_asio 2024-04-26T16:23:27.939 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73220 2024-04-26T16:23:27.939 INFO:tasks.workunit.client.0.smithi050.stdout:test api_asio on pid 73220 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_asio on pid 73220' 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73220 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_list 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_list' 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_list 2024-04-26T16:23:27.940 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.941 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.941 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.941 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.941 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.941 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-04-26T16:23:27.941 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_asio.log 2024-04-26T16:23:27.942 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_asio: /' 2024-04-26T16:23:27.942 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_list 2024-04-26T16:23:27.942 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73228 2024-04-26T16:23:27.942 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_list on pid 73228' 2024-04-26T16:23:27.943 INFO:tasks.workunit.client.0.smithi050.stdout:test api_list on pid 73228 2024-04-26T16:23:27.943 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73228 2024-04-26T16:23:27.943 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.943 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.943 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-04-26T16:23:27.944 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.944 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.944 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.944 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.945 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-04-26T16:23:27.945 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_list.log 2024-04-26T16:23:27.945 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_lock 2024-04-26T16:23:27.945 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_lock' 2024-04-26T16:23:27.946 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_list: /' 2024-04-26T16:23:27.946 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_lock 2024-04-26T16:23:27.947 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.950 INFO:tasks.workunit.client.0.smithi050.stdout:test api_lock on pid 73236 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_lock 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73236 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_lock on pid 73236' 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73236 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_lock_pp 2024-04-26T16:23:27.951 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_lock_pp' 2024-04-26T16:23:27.952 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.952 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.952 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.952 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.953 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-04-26T16:23:27.953 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_lock_pp 2024-04-26T16:23:27.953 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.955 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_lock: /' 2024-04-26T16:23:27.955 INFO:tasks.workunit.client.0.smithi050.stdout:test api_lock_pp on pid 73244 2024-04-26T16:23:27.955 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_lock_pp 2024-04-26T16:23:27.955 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73244 2024-04-26T16:23:27.955 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_lock_pp on pid 73244' 2024-04-26T16:23:27.955 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73244 2024-04-26T16:23:27.956 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.956 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.956 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-04-26T16:23:27.958 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.958 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.958 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.958 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.958 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_lock_pp: /' 2024-04-26T16:23:27.960 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-04-26T16:23:27.962 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_misc 2024-04-26T16:23:27.962 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_misc' 2024-04-26T16:23:27.965 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_lock.log 2024-04-26T16:23:27.969 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_misc 2024-04-26T16:23:27.969 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-04-26T16:23:27.972 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.980 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_misc 2024-04-26T16:23:27.981 INFO:tasks.workunit.client.0.smithi050.stdout:test api_misc on pid 73308 2024-04-26T16:23:27.981 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73308 2024-04-26T16:23:27.981 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_misc on pid 73308' 2024-04-26T16:23:27.982 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73308 2024-04-26T16:23:27.982 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.982 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.982 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_misc_pp 2024-04-26T16:23:27.982 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_misc_pp' 2024-04-26T16:23:27.988 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-04-26T16:23:27.989 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_misc_pp 2024-04-26T16:23:27.991 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:27.993 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:27.994 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.994 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:27.994 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:27.994 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stdout:test api_misc_pp on pid 73340 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_misc_pp 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73340 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_misc_pp on pid 73340' 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73340 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:27.995 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_misc.log 2024-04-26T16:23:27.996 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_tier_pp 2024-04-26T16:23:27.996 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_misc: /' 2024-04-26T16:23:27.997 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_tier_pp' 2024-04-26T16:23:27.997 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-04-26T16:23:27.998 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_tier_pp 2024-04-26T16:23:27.999 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.000 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.000 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.000 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.000 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.001 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_misc_pp: /' 2024-04-26T16:23:28.001 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-04-26T16:23:28.001 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-04-26T16:23:28.002 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_tier_pp 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stdout:test api_tier_pp on pid 73367 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73367 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_tier_pp on pid 73367' 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73367 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.003 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-04-26T16:23:28.004 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.004 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.005 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.005 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.005 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-04-26T16:23:28.009 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-04-26T16:23:28.010 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_pool 2024-04-26T16:23:28.011 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_pool' 2024-04-26T16:23:28.011 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_tier_pp: /' 2024-04-26T16:23:28.014 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.014 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_pool 2024-04-26T16:23:28.017 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_pool 2024-04-26T16:23:28.017 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73400 2024-04-26T16:23:28.017 INFO:tasks.workunit.client.0.smithi050.stdout:test api_pool on pid 73400 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_pool on pid 73400' 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73400 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_snapshots 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_snapshots' 2024-04-26T16:23:28.018 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_snapshots 2024-04-26T16:23:28.019 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.019 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.019 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.019 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.019 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-04-26T16:23:28.019 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_pool.log 2024-04-26T16:23:28.022 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_snapshots 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73418 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_snapshots on pid 73418' 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stdout:test api_snapshots on pid 73418 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73418 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.025 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.026 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-04-26T16:23:28.026 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_snapshots: /' 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_snapshots_pp 2024-04-26T16:23:28.027 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_snapshots_pp' 2024-04-26T16:23:28.028 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_pool: /' 2024-04-26T16:23:28.028 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_snapshots_pp 2024-04-26T16:23:28.029 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.032 INFO:tasks.workunit.client.0.smithi050.stdout:test api_snapshots_pp on pid 73426 2024-04-26T16:23:28.032 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_snapshots_pp 2024-04-26T16:23:28.032 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73426 2024-04-26T16:23:28.032 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_snapshots_pp on pid 73426' 2024-04-26T16:23:28.032 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73426 2024-04-26T16:23:28.032 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.033 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.033 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-04-26T16:23:28.033 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_stat 2024-04-26T16:23:28.033 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_stat' 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-04-26T16:23:28.034 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:28.035 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-04-26T16:23:28.038 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_stat 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stdout:test api_stat on pid 73441 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_stat 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73441 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_stat on pid 73441' 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73441 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.040 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.046 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_stat_pp 2024-04-26T16:23:28.047 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_stat_pp' 2024-04-26T16:23:28.057 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-04-26T16:23:28.058 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.059 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.059 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.060 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.060 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.061 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-04-26T16:23:28.061 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_stat.log 2024-04-26T16:23:28.063 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_stat_pp 2024-04-26T16:23:28.063 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_stat: /' 2024-04-26T16:23:28.064 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_stat_pp 2024-04-26T16:23:28.064 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73479 2024-04-26T16:23:28.064 INFO:tasks.workunit.client.0.smithi050.stdout:test api_stat_pp on pid 73479 2024-04-26T16:23:28.065 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_stat_pp on pid 73479' 2024-04-26T16:23:28.065 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73479 2024-04-26T16:23:28.065 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.065 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.065 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-04-26T16:23:28.067 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.067 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.067 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.067 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.069 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_stat_pp: /' 2024-04-26T16:23:28.070 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-04-26T16:23:28.071 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-04-26T16:23:28.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:28.072 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_watch_notify 2024-04-26T16:23:28.073 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_watch_notify' 2024-04-26T16:23:28.074 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.075 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_watch_notify 2024-04-26T16:23:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:28.078 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_watch_notify 2024-04-26T16:23:28.078 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73527 2024-04-26T16:23:28.079 INFO:tasks.workunit.client.0.smithi050.stdout:test api_watch_notify on pid 73527 2024-04-26T16:23:28.079 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_watch_notify on pid 73527' 2024-04-26T16:23:28.079 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73527 2024-04-26T16:23:28.080 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.080 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.080 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-04-26T16:23:28.081 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.082 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_watch_notify_pp 2024-04-26T16:23:28.082 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.082 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.082 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.082 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_watch_notify_pp' 2024-04-26T16:23:28.082 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-04-26T16:23:28.087 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.088 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_watch_notify: /' 2024-04-26T16:23:28.088 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-04-26T16:23:28.095 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_watch_notify_pp 2024-04-26T16:23:28.097 INFO:tasks.workunit.client.0.smithi050.stdout:test api_watch_notify_pp on pid 73568 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_watch_notify_pp 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73568 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_watch_notify_pp on pid 73568' 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73568 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-04-26T16:23:28.098 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_cmd 2024-04-26T16:23:28.099 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_cmd' 2024-04-26T16:23:28.099 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.099 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.099 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.099 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.105 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.110 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-04-26T16:23:28.110 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_cmd 2024-04-26T16:23:28.113 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-04-26T16:23:28.115 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-04-26T16:23:28.116 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_cmd 2024-04-26T16:23:28.116 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73580 2024-04-26T16:23:28.116 INFO:tasks.workunit.client.0.smithi050.stdout:test api_cmd on pid 73580 2024-04-26T16:23:28.117 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_cmd on pid 73580' 2024-04-26T16:23:28.117 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73580 2024-04-26T16:23:28.117 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.117 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.118 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-04-26T16:23:28.126 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_cmd_pp 2024-04-26T16:23:28.126 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_cmd_pp' 2024-04-26T16:23:28.128 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.128 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.128 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.128 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.134 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.135 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_cmd.log 2024-04-26T16:23:28.138 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-04-26T16:23:28.138 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_cmd: /' 2024-04-26T16:23:28.148 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_cmd_pp 2024-04-26T16:23:28.150 INFO:tasks.workunit.client.0.smithi050.stdout:test api_cmd_pp on pid 73606 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_cmd_pp 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73606 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_cmd_pp on pid 73606' 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73606 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_service 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-04-26T16:23:28.151 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_service' 2024-04-26T16:23:28.152 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.153 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.153 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.153 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.153 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.154 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_service 2024-04-26T16:23:28.154 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-04-26T16:23:28.154 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-04-26T16:23:28.155 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-04-26T16:23:28.155 INFO:tasks.workunit.client.0.smithi050.stdout:test api_service on pid 73618 2024-04-26T16:23:28.155 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_service 2024-04-26T16:23:28.155 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73618 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_service on pid 73618' 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73618 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_service_pp 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_service_pp' 2024-04-26T16:23:28.156 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-04-26T16:23:28.157 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.157 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.157 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.157 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.158 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_service_pp 2024-04-26T16:23:28.158 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-04-26T16:23:28.158 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_service.log 2024-04-26T16:23:28.159 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.160 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_service: /' 2024-04-26T16:23:28.163 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_service_pp 2024-04-26T16:23:28.163 INFO:tasks.workunit.client.0.smithi050.stdout:test api_service_pp on pid 73644 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73644 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_service_pp on pid 73644' 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73644 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.164 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_c_write_operations 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_c_write_operations' 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-04-26T16:23:28.165 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-04-26T16:23:28.166 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_service_pp: /' 2024-04-26T16:23:28.170 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_c_write_operations 2024-04-26T16:23:28.170 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.172 INFO:tasks.workunit.client.0.smithi050.stdout:test api_c_write_operations on pid 73661 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_c_write_operations 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73661 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_c_write_operations on pid 73661' 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73661 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_c_read_operations 2024-04-26T16:23:28.173 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_c_read_operations' 2024-04-26T16:23:28.174 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.174 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.174 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.174 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.174 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-04-26T16:23:28.174 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-04-26T16:23:28.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:28.179 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-04-26T16:23:28.183 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_c_read_operations 2024-04-26T16:23:28.184 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.186 INFO:tasks.workunit.client.0.smithi050.stdout:test api_c_read_operations on pid 73683 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_c_read_operations 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73683 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_c_read_operations on pid 73683' 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73683 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s api_cls_remote_reads 2024-04-26T16:23:28.187 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' api_cls_remote_reads' 2024-04-26T16:23:28.188 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-04-26T16:23:28.189 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.189 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.189 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.189 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.189 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-04-26T16:23:28.190 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-04-26T16:23:28.191 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.193 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo api_cls_remote_reads 2024-04-26T16:23:28.194 INFO:tasks.workunit.client.0.smithi050.stdout:test api_cls_remote_reads on pid 73694 2024-04-26T16:23:28.194 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=api_cls_remote_reads 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73694 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test api_cls_remote_reads on pid 73694' 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73694 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s list_parallel 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' list_parallel' 2024-04-26T16:23:28.195 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-04-26T16:23:28.197 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.197 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.197 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.197 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.197 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-04-26T16:23:28.202 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo list_parallel 2024-04-26T16:23:28.204 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.211 INFO:tasks.workunit.client.0.smithi050.stdout:test list_parallel on pid 73740 2024-04-26T16:23:28.211 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=list_parallel 2024-04-26T16:23:28.211 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73740 2024-04-26T16:23:28.212 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test list_parallel on pid 73740' 2024-04-26T16:23:28.212 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73740 2024-04-26T16:23:28.212 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.212 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.212 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-04-26T16:23:28.212 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s open_pools_parallel 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' open_pools_parallel' 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo open_pools_parallel 2024-04-26T16:23:28.213 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ list_parallel: /' 2024-04-26T16:23:28.214 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-04-26T16:23:28.214 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_list_parallel.log 2024-04-26T16:23:28.214 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.217 INFO:tasks.workunit.client.0.smithi050.stdout:test open_pools_parallel on pid 73758 2024-04-26T16:23:28.217 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=open_pools_parallel 2024-04-26T16:23:28.218 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73758 2024-04-26T16:23:28.218 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test open_pools_parallel on pid 73758' 2024-04-26T16:23:28.218 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73758 2024-04-26T16:23:28.218 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-04-26T16:23:28.218 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.218 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.219 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s delete_pools_parallel 2024-04-26T16:23:28.220 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-04-26T16:23:28.220 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:28.223 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' delete_pools_parallel' 2024-04-26T16:23:28.223 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-04-26T16:23:28.225 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.225 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo delete_pools_parallel 2024-04-26T16:23:28.225 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-04-26T16:23:28.228 INFO:tasks.workunit.client.0.smithi050.stdout:test delete_pools_parallel on pid 73781 2024-04-26T16:23:28.228 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=delete_pools_parallel 2024-04-26T16:23:28.228 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73781 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test delete_pools_parallel on pid 73781' 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73781 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s cls 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' cls' 2024-04-26T16:23:28.229 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-04-26T16:23:28.232 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.232 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.232 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.232 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.235 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-04-26T16:23:28.236 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo cls 2024-04-26T16:23:28.236 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-04-26T16:23:28.236 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-04-26T16:23:28.236 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.243 INFO:tasks.workunit.client.0.smithi050.stdout:test cls on pid 73824 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=cls 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73824 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test cls on pid 73824' 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73824 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.244 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s cmd 2024-04-26T16:23:28.245 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' cmd' 2024-04-26T16:23:28.245 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.247 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo cmd 2024-04-26T16:23:28.248 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-04-26T16:23:28.248 INFO:tasks.workunit.client.0.smithi050.stdout:test cmd on pid 73830 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=cmd 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73830 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test cmd on pid 73830' 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73830 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-04-26T16:23:28.249 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.251 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.251 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.252 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.252 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_cls 2024-04-26T16:23:28.254 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.254 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.254 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.254 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.257 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_cls.log 2024-04-26T16:23:28.258 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ cls: /' 2024-04-26T16:23:28.259 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_cmd 2024-04-26T16:23:28.262 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s handler_error 2024-04-26T16:23:28.262 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' handler_error' 2024-04-26T16:23:28.274 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_cmd.log 2024-04-26T16:23:28.275 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ cmd: /' 2024-04-26T16:23:28.275 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo handler_error 2024-04-26T16:23:28.275 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.279 INFO:tasks.workunit.client.0.smithi050.stdout:test handler_error on pid 73893 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=handler_error 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73893 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test handler_error on pid 73893' 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73893 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-04-26T16:23:28.280 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s io 2024-04-26T16:23:28.281 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' io' 2024-04-26T16:23:28.282 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.282 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.283 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.283 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.287 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_handler_error.log 2024-04-26T16:23:28.289 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ handler_error: /' 2024-04-26T16:23:28.289 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_handler_error 2024-04-26T16:23:28.299 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.306 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo io 2024-04-26T16:23:28.309 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=io 2024-04-26T16:23:28.309 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73980 2024-04-26T16:23:28.309 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test io on pid 73980' 2024-04-26T16:23:28.309 INFO:tasks.workunit.client.0.smithi050.stdout:test io on pid 73980 2024-04-26T16:23:28.310 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73980 2024-04-26T16:23:28.310 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.310 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.310 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s ec_io 2024-04-26T16:23:28.310 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' ec_io' 2024-04-26T16:23:28.311 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.311 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo ec_io 2024-04-26T16:23:28.313 INFO:tasks.workunit.client.0.smithi050.stdout:test ec_io on pid 73993 2024-04-26T16:23:28.313 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=ec_io 2024-04-26T16:23:28.313 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73993 2024-04-26T16:23:28.313 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test ec_io on pid 73993' 2024-04-26T16:23:28.313 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=73993 2024-04-26T16:23:28.313 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.314 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.314 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s list 2024-04-26T16:23:28.314 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' list' 2024-04-26T16:23:28.314 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-04-26T16:23:28.315 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-04-26T16:23:28.315 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.315 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.315 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.315 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.316 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_ec_io 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_ec_io.log 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo list 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_io 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ io: /' 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_io.log 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ ec_io: /' 2024-04-26T16:23:28.317 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:28 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.210048+0000 osd.1 (osd.1) 153 : cluster [DBG] 2.d scrub starts 2024-04-26T16:23:28.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:28 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.210048+0000 osd.1 (osd.1) 153 : cluster [DBG] 2.d scrub starts 2024-04-26T16:23:28.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:28 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.211542+0000 osd.1 (osd.1) 154 : cluster [DBG] 2.d scrub ok 2024-04-26T16:23:28.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:28 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.211542+0000 osd.1 (osd.1) 154 : cluster [DBG] 2.d scrub ok 2024-04-26T16:23:28.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:28 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.471065+0000 mgr.y (mgr.24401) 221 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:28.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:28 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.471065+0000 mgr.y (mgr.24401) 221 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:28.322 INFO:tasks.workunit.client.0.smithi050.stdout:test list on pid 74033 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=list 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74033 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test list on pid 74033' 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74033 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.323 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-04-26T16:23:28.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:28.325 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s ec_list 2024-04-26T16:23:28.326 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' ec_list' 2024-04-26T16:23:28.326 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.326 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.326 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.326 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.327 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ list: /' 2024-04-26T16:23:28.327 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_list 2024-04-26T16:23:28.327 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo ec_list 2024-04-26T16:23:28.328 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_list.log 2024-04-26T16:23:28.334 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.339 INFO:tasks.workunit.client.0.smithi050.stdout:test ec_list on pid 74078 2024-04-26T16:23:28.339 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=ec_list 2024-04-26T16:23:28.339 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74078 2024-04-26T16:23:28.339 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test ec_list on pid 74078' 2024-04-26T16:23:28.339 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74078 2024-04-26T16:23:28.340 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.340 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.340 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s misc 2024-04-26T16:23:28.340 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' misc' 2024-04-26T16:23:28.340 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.340 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo misc 2024-04-26T16:23:28.341 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-04-26T16:23:28.343 INFO:tasks.workunit.client.0.smithi050.stdout:test misc on pid 74086 2024-04-26T16:23:28.344 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=misc 2024-04-26T16:23:28.344 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74086 2024-04-26T16:23:28.344 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test misc on pid 74086' 2024-04-26T16:23:28.344 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74086 2024-04-26T16:23:28.344 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.344 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s pool 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' pool' 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ ec_list: /' 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo pool 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_ec_list 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.345 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.346 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ misc: /' 2024-04-26T16:23:28.346 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_misc.log 2024-04-26T16:23:28.350 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:28.351 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=pool 2024-04-26T16:23:28.351 INFO:tasks.workunit.client.0.smithi050.stdout:test pool on pid 74114 2024-04-26T16:23:28.351 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74114 2024-04-26T16:23:28.351 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test pool on pid 74114' 2024-04-26T16:23:28.352 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74114 2024-04-26T16:23:28.352 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.352 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.352 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-04-26T16:23:28.352 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s read_operations 2024-04-26T16:23:28.352 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' read_operations' 2024-04-26T16:23:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_ec_list.log 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_pool.log 2024-04-26T16:23:28.354 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_misc 2024-04-26T16:23:28.355 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo read_operations 2024-04-26T16:23:28.355 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.355 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ pool: /' 2024-04-26T16:23:28.359 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_pool 2024-04-26T16:23:28.367 INFO:tasks.workunit.client.0.smithi050.stdout:test read_operations on pid 74143 2024-04-26T16:23:28.367 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=read_operations 2024-04-26T16:23:28.368 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74143 2024-04-26T16:23:28.368 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test read_operations on pid 74143' 2024-04-26T16:23:28.368 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74143 2024-04-26T16:23:28.368 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.368 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.369 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-04-26T16:23:28.375 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.375 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.375 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.375 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.376 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_read_operations 2024-04-26T16:23:28.376 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ read_operations: /' 2024-04-26T16:23:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.210048+0000 osd.1 (osd.1) 153 : cluster [DBG] 2.d scrub starts 2024-04-26T16:23:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.210048+0000 osd.1 (osd.1) 153 : cluster [DBG] 2.d scrub starts 2024-04-26T16:23:28.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.211542+0000 osd.1 (osd.1) 154 : cluster [DBG] 2.d scrub ok 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.211542+0000 osd.1 (osd.1) 154 : cluster [DBG] 2.d scrub ok 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.471065+0000 mgr.y (mgr.24401) 221 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.471065+0000 mgr.y (mgr.24401) 221 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.210048+0000 osd.1 (osd.1) 153 : cluster [DBG] 2.d scrub starts 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.210048+0000 osd.1 (osd.1) 153 : cluster [DBG] 2.d scrub starts 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.211542+0000 osd.1 (osd.1) 154 : cluster [DBG] 2.d scrub ok 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.211542+0000 osd.1 (osd.1) 154 : cluster [DBG] 2.d scrub ok 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.471065+0000 mgr.y (mgr.24401) 221 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:28.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:28 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.471065+0000 mgr.y (mgr.24401) 221 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:23:28.378 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_read_operations.log 2024-04-26T16:23:28.378 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s snapshots 2024-04-26T16:23:28.378 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' snapshots' 2024-04-26T16:23:28.379 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo snapshots 2024-04-26T16:23:28.379 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.382 INFO:tasks.workunit.client.0.smithi050.stdout:test snapshots on pid 74192 2024-04-26T16:23:28.382 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=snapshots 2024-04-26T16:23:28.383 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74192 2024-04-26T16:23:28.383 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test snapshots on pid 74192' 2024-04-26T16:23:28.383 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74192 2024-04-26T16:23:28.383 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.383 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.383 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s watch_notify 2024-04-26T16:23:28.384 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' watch_notify' 2024-04-26T16:23:28.384 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo watch_notify 2024-04-26T16:23:28.384 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.385 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=watch_notify 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stdout:test watch_notify on pid 74210 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74210 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test watch_notify on pid 74210' 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74210 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-04-26T16:23:28.386 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.389 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.389 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.389 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.389 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.390 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_watch_notify 2024-04-26T16:23:28.390 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-04-26T16:23:28.390 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ watch_notify: /' 2024-04-26T16:23:28.390 INFO:tasks.workunit.client.0.smithi050.stderr:++ printf %25s write_operations 2024-04-26T16:23:28.390 INFO:tasks.workunit.client.0.smithi050.stderr:+ r=' write_operations' 2024-04-26T16:23:28.390 INFO:tasks.workunit.client.0.smithi050.stderr:++ echo write_operations 2024-04-26T16:23:28.391 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-04-26T16:23:28.393 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.393 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.393 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.393 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.393 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_snapshots.log 2024-04-26T16:23:28.398 INFO:tasks.workunit.client.0.smithi050.stderr:++ awk '{print $1}' 2024-04-26T16:23:28.401 INFO:tasks.workunit.client.0.smithi050.stdout:test write_operations on pid 74225 2024-04-26T16:23:28.401 INFO:tasks.workunit.client.0.smithi050.stderr:+ ff=write_operations 2024-04-26T16:23:28.401 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74225 2024-04-26T16:23:28.401 INFO:tasks.workunit.client.0.smithi050.stderr:+ echo 'test write_operations on pid 74225' 2024-04-26T16:23:28.401 INFO:tasks.workunit.client.0.smithi050.stderr:+ pids[$f]=74225 2024-04-26T16:23:28.401 INFO:tasks.workunit.client.0.smithi050.stderr:+ ret=0 2024-04-26T16:23:28.402 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' 1 -eq 1 ']' 2024-04-26T16:23:28.402 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:23:28.402 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73367 2024-04-26T16:23:28.402 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73367 2024-04-26T16:23:28.403 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ snapshots: /' 2024-04-26T16:23:28.403 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_snapshots 2024-04-26T16:23:28.408 INFO:tasks.workunit.client.0.smithi050.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-04-26T16:23:28.411 INFO:tasks.workunit.client.0.smithi050.stderr:+ '[' -z '' ']' 2024-04-26T16:23:28.411 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.411 INFO:tasks.workunit.client.0.smithi050.stderr:+ case $- in 2024-04-26T16:23:28.412 INFO:tasks.workunit.client.0.smithi050.stderr:+ return 2024-04-26T16:23:28.412 INFO:tasks.workunit.client.0.smithi050.stderr:+ tee ceph_test_neorados_write_operations.log 2024-04-26T16:23:28.412 INFO:tasks.workunit.client.0.smithi050.stderr:+ ceph_test_neorados_write_operations 2024-04-26T16:23:28.414 INFO:tasks.workunit.client.0.smithi050.stderr:+ sed 's/^/ write_operations: /' 2024-04-26T16:23:28.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:28.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:28.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:28.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:28.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:28.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:28.735 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:28.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:28.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:28.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/3358130895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb22806bad0 msgr2=0x7fb2281102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- 172.21.15.50:0/3358130895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb22806bad0 0x7fb2281102e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb22000b0a0 tx=0x7fb22000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:23:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/3358130895 shutdown_connections 2024-04-26T16:23:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- 172.21.15.50:0/3358130895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb2281094a0 0x7fb228109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- 172.21.15.50:0/3358130895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb22806bad0 0x7fb2281102e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- 172.21.15.50:0/3358130895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb22806b130 0x7fb22806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/3358130895 >> 172.21.15.50:0/3358130895 conn(0x7fb2280792d0 msgr2=0x7fb2280796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/3358130895 shutdown_connections 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/3358130895 wait complete. 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 librados: starting msgr at 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 librados: starting objecter 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 Processor -- start 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- start start 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 librados: setting wanted keys 2024-04-26T16:23:28.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 librados: calling monclient init 2024-04-26T16:23:28.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb22806b130 0x7fb2281991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb22806bad0 0x7fb2281996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 0x7fb22819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb228112ad0 con 0x7fb22806b130 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb228112950 con 0x7fb22806bad0 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb228112c50 con 0x7fb2281094a0 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 0x7fb22819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb226ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb22806b130 0x7fb2281991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 0x7fb22819da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:42492/0 (socket says 172.21.15.50:42492) 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 -- 172.21.15.50:0/1769465317 learned_addr learned my addr 172.21.15.50:0/1769465317 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2267fc700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb22806bad0 0x7fb2281996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 -- 172.21.15.50:0/1769465317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb22806bad0 msgr2=0x7fb2281996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:28.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb22806bad0 0x7fb2281996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 -- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb22806b130 msgr2=0x7fb2281991b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb22806b130 0x7fb2281991b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 -- 172.21.15.50:0/1769465317 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22819e170 con 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb2277fe700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 0x7fb22819da30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb22000b780 tx=0x7fb220004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb20ffff700 10 client.?.objecter ms_handle_connect 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb20ffff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220021070 con 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22819e400 con 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb220007b80 con 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb220005040 con 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 10 client.24938.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2281a5ce0 con 0x7fb2281094a0 2024-04-26T16:23:28.971 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.972+0000 7fb22d578700 1 librados: init done 2024-04-26T16:23:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb22810b880 con 0x7fb2281094a0 2024-04-26T16:23:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb220024070 con 0x7fb2281094a0 2024-04-26T16:23:28.972 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb210071f80 0x7fb210074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(140..140 src has 87..140) ==== 10386+0+0 (secure 0 0 0) 0x7fb22000f020 con 0x7fb2281094a0 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 10 client.24938.objecter ms_dispatch 0x7fb228001550 osd_map(140..140 src has 87..140) 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 3 client.24938.objecter handle_osd_map got epochs [140,140] > 0 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 3 client.24938.objecter handle_osd_map decoding full epoch 140 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb226ffd700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb210071f80 0x7fb210074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 20 client.24938.objecter dump_active .. 0 homeless 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb20ffff700 10 client.24938.objecter ms_handle_connect 0x7fb210071f80 2024-04-26T16:23:28.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.976+0000 7fb226ffd700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb210071f80 0x7fb210074440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb218005950 tx=0x7fb21800a300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:28.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:28.980+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb22002b070 con 0x7fb2281094a0 2024-04-26T16:23:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:28.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:29.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:29.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:29.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:29.270 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:29.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:29.352+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fb22819a4c0 con 0x7fb2281094a0 2024-04-26T16:23:29.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:29.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.380002+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.380002+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.383213+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:27.383213+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:28.109927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:28.109927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.117387+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.50:0/828319541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.117387+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.50:0/828319541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.122122+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.122122+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.123561+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.123561+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.124307+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.124307+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.124577+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.50:0/2534252021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.124577+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.50:0/2534252021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.126279+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.50:0/551282432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.126279+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.50:0/551282432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.136948+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.50:0/182122840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.136948+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.50:0/182122840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.165901+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.165901+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.166253+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.166253+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.166529+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.166529+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.184110+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.184110+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.210903+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.210903+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.380002+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.380002+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:23:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.383213+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:27.383213+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:28.109927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:28.109927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.117387+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.50:0/828319541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.117387+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.50:0/828319541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.122122+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.122122+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.123561+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.123561+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.124307+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.124307+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.124577+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.50:0/2534252021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.124577+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.50:0/2534252021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.126279+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.50:0/551282432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.126279+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.50:0/551282432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.136948+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.50:0/182122840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.136948+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.50:0/182122840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.165901+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.165901+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.166253+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.166253+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.166529+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.166529+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.184110+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.184110+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.210903+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.210903+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:28.215688+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:28.215688+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:28.218451+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: cluster 2024-04-26T16:23:28.218451+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.221146+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.221146+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.225210+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.225210+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.236326+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.236326+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.247331+0000 mon.b (mon.1) 38 : audit [DBG] from='client.? 172.21.15.50:0/2529552871' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.247331+0000 mon.b (mon.1) 38 : audit [DBG] from='client.? 172.21.15.50:0/2529552871' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.255621+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.255621+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.258263+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.258263+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.273651+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.273651+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.390896+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.390896+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.406729+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.406729+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.413987+0000 mon.a (mon.0) 985 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[29234]: audit 2024-04-26T16:23:28.413987+0000 mon.a (mon.0) 985 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:28.215688+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:28.215688+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:28.218451+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: cluster 2024-04-26T16:23:28.218451+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.221146+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.221146+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.225210+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.225210+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.236326+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.236326+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.247331+0000 mon.b (mon.1) 38 : audit [DBG] from='client.? 172.21.15.50:0/2529552871' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-26T16:23:29.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.247331+0000 mon.b (mon.1) 38 : audit [DBG] from='client.? 172.21.15.50:0/2529552871' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.255621+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.255621+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.258263+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.258263+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.273651+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.273651+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.390896+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.390896+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.406729+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.406729+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.413987+0000 mon.a (mon.0) 985 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:29 smithi050 bash[21463]: audit 2024-04-26T16:23:28.413987+0000 mon.a (mon.0) 985 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [----------] Global test environment set-up. 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [----------] 12 tests from AsioRados 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-04-26T16:23:29.388 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (2 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (2 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-04-26T16:23:29.389 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [----------] 12 tests from AsioRados (14 ms total) 2024-04-26T16:23:29.390 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: 2024-04-26T16:23:29.390 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [----------] Global test environment tear-down 2024-04-26T16:23:29.390 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1392 ms total) 2024-04-26T16:23:29.390 INFO:tasks.workunit.client.0.smithi050.stdout: api_asio: [ PASSED ] 12 tests. 2024-04-26T16:23:29.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:29.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:23:29.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:29.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.380002+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:23:29.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.380002+0000 osd.3 (osd.3) 157 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:23:29.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.383213+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:23:29.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:27.383213+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:28.109927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:28.109927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.117387+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.50:0/828319541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.117387+0000 mon.c (mon.2) 98 : audit [INF] from='client.? 172.21.15.50:0/828319541' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.122122+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.122122+0000 mon.c (mon.2) 99 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.123561+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.123561+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.124307+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.124307+0000 mon.a (mon.0) 975 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.124577+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.50:0/2534252021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.124577+0000 mon.b (mon.1) 34 : audit [INF] from='client.? 172.21.15.50:0/2534252021' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.126279+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.50:0/551282432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.126279+0000 mon.b (mon.1) 35 : audit [INF] from='client.? 172.21.15.50:0/551282432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.136948+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.50:0/182122840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.136948+0000 mon.c (mon.2) 100 : audit [INF] from='client.? 172.21.15.50:0/182122840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.165901+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.165901+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.166253+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.166253+0000 mon.a (mon.0) 977 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.166529+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.166529+0000 mon.a (mon.0) 978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.184110+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.184110+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.210903+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.210903+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:28.215688+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:28.215688+0000 osd.5 (osd.5) 147 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:28.218451+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: cluster 2024-04-26T16:23:28.218451+0000 osd.5 (osd.5) 148 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.221146+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.221146+0000 mon.b (mon.1) 36 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.225210+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.225210+0000 mon.a (mon.0) 981 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.236326+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.236326+0000 mon.b (mon.1) 37 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.247331+0000 mon.b (mon.1) 38 : audit [DBG] from='client.? 172.21.15.50:0/2529552871' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.247331+0000 mon.b (mon.1) 38 : audit [DBG] from='client.? 172.21.15.50:0/2529552871' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.255621+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.255621+0000 mon.a (mon.0) 982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.258263+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.258263+0000 mon.b (mon.1) 39 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.273651+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.273651+0000 mon.a (mon.0) 983 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.390896+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.390896+0000 mon.b (mon.1) 40 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.406729+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.406729+0000 mon.a (mon.0) 984 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.413987+0000 mon.a (mon.0) 985 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:29 smithi190 bash[23868]: audit 2024-04-26T16:23:28.413987+0000 mon.a (mon.0) 985 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:29.523 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:29.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:29.649 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:29.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:29.690 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-04-26T16:23:29.690 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] Global test environment set-up. 2024-04-26T16:23:29.690 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] 7 tests from LibRadosList 2024-04-26T16:23:29.690 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-04-26T16:23:29.690 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.ListObjects (267 ms) 2024-04-26T16:23:29.690 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (26 ms) 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset foo1,foo2,foo3 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo1 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo2 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo3 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset foo1,foo4,foo5 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo4 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo5 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo1 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset foo6,foo7 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo7 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo6 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-26T16:23:29.691 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns1:foo4 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns1:foo5 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns2:foo7 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns2:foo6 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns1:foo1 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: :foo1 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: :foo2 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: :foo3 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (89 ms) 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 1 0 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 10 0 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 13 0 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 7 0 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 14 0 2024-04-26T16:23:29.692 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 0 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 15 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 11 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 5 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 8 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 6 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 3 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 4 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 12 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 9 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 2 0 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (84 ms) 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-04-26T16:23:29.693 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: x cursor=MIN 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=1 cursor=12:02547ec2:::1:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=10 cursor=12:52ea6a34:::10:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=13 cursor=12:566253c9:::13:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=7 cursor=12:5c6b0b28:::7:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=14 cursor=12:62a1935d:::14:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=0 cursor=12:6cac518f:::0:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=15 cursor=12:863748b0:::15:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=11 cursor=12:89d3ae78:::11:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=5 cursor=12:b29083e3:::5:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=8 cursor=12:bd63b0f1:::8:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=6 cursor=12:c4fdafeb:::6:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=3 cursor=12:cfc208b3:::3:head 2024-04-26T16:23:29.694 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=4 cursor=12:d83876eb:::4:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=12 cursor=12:de5d7c5f:::12:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=9 cursor=12:e960b815:::9:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > oid=2 cursor=12:f905c69b:::2:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: FIRST> seek to MIN oid=1 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=1 cursor=12:02547ec2:::1:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=10 cursor=12:52ea6a34:::10:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=13 cursor=12:566253c9:::13:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:566253c9:::13:head 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-04-26T16:23:29.695 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=7 cursor=12:5c6b0b28:::7:head 2024-04-26T16:23:29.696 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-04-26T16:23:29.696 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-04-26T16:23:29.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:29.764 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=14 cursor=12:62a1935d:::14:head 2024-04-26T16:23:29.764 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=0 cursor=12:6cac518f:::0:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=15 cursor=12:863748b0:::15:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:863748b0:::15:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=11 cursor=12:89d3ae78:::11:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=5 cursor=12:b29083e3:::5:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-04-26T16:23:29.765 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=8 cursor=12:bd63b0f1:::8:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=6 cursor=12:c4fdafeb:::6:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=3 cursor=12:cfc208b3:::3:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=4 cursor=12:d83876eb:::4:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=12 cursor=12:de5d7c5f:::12:head 2024-04-26T16:23:29.766 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=9 cursor=12:e960b815:::9:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:e960b815:::9:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : oid=2 cursor=12:f905c69b:::2:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:62a1935d:::14:head expected=12:62a1935d:::14:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=14 expected=14 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:de5d7c5f:::12:head expected=12:de5d7c5f:::12:head 2024-04-26T16:23:29.767 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=12 expected=12 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:d83876eb:::4:head expected=12:d83876eb:::4:head 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=4 expected=4 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:cfc208b3:::3:head expected=12:cfc208b3:::3:head 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=3 expected=3 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:c4fdafeb:::6:head expected=12:c4fdafeb:::6:head 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=6 expected=6 2024-04-26T16:23:29.768 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:b29083e3:::5:head expected=12:b29083e3:::5:head 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=5 expected=5 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:566253c9:::13:head 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:566253c9:::13:head expected=12:566253c9:::13:head 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=13 expected=13 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:5c6b0b28:::7:head expected=12:5c6b0b28:::7:head 2024-04-26T16:23:29.769 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-04-26T16:23:29.774 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:29.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:29.900 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:29.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:30.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:30.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:30.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.136+0000 7fb20ffff700 1 -- 172.21.15.50:0/1769465317 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v142) ==== 87+0+0 (secure 0 0 0) 0x7fb22005e1c0 con 0x7fb2281094a0 2024-04-26T16:23:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 10 librados: watch_flush enter 2024-04-26T16:23:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 10 librados: watch_flush exit 2024-04-26T16:23:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 20 client.24938.objecter shutdown clearing up homeless session... 2024-04-26T16:23:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 10 client.24938.objecter successfully canceled tick 2024-04-26T16:23:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb210071f80 msgr2=0x7fb210074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb210071f80 0x7fb210074440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb218005950 tx=0x7fb21800a300 comp rx=0 tx=0).stop 2024-04-26T16:23:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 msgr2=0x7fb22819da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.144+0000 7fb22d578700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 0x7fb22819da30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb22000b780 tx=0x7fb220004d00 comp rx=0 tx=0).stop 2024-04-26T16:23:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.148+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 shutdown_connections 2024-04-26T16:23:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.148+0000 7fb22d578700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb210071f80 0x7fb210074440 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.148+0000 7fb22d578700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb2281094a0 0x7fb22819da30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.148+0000 7fb22d578700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb22806b130 0x7fb2281991b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.148+0000 7fb22d578700 1 --2- 172.21.15.50:0/1769465317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb22806bad0 0x7fb2281996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.148+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 >> 172.21.15.50:0/1769465317 conn(0x7fb2280792d0 msgr2=0x7fb2281002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:30.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.152+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 shutdown_connections 2024-04-26T16:23:30.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.152+0000 7fb22d578700 1 -- 172.21.15.50:0/1769465317 wait complete. 2024-04-26T16:23:30.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:30.160+0000 7fb22d578700 1 librados: shutdown 2024-04-26T16:23:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:23:30.174 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:30.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:30.306 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116628+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.306 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116628+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116731+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116731+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116823+0000 mon.a (mon.0) 988 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116823+0000 mon.a (mon.0) 988 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116941+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.116941+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117065+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117065+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117200+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117200+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117375+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117375+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117473+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117473+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117557+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117557+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117643+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.117643+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.130467+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.130467+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.134081+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.50:0/2828827295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.134081+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.50:0/2828827295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.134421+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.50:0/3409971702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.134421+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.50:0/3409971702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.135523+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.135523+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.136437+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.50:0/779541252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.136437+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.50:0/779541252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.136775+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.50:0/505282512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.136775+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.50:0/505282512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.137237+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.137237+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.141164+0000 mon.c (mon.2) 106 : audit [INF] from='client.? 172.21.15.50:0/3136352124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.141164+0000 mon.c (mon.2) 106 : audit [INF] from='client.? 172.21.15.50:0/3136352124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.146638+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.146638+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.148206+0000 mon.a (mon.0) 998 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.148206+0000 mon.a (mon.0) 998 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.148878+0000 mon.a (mon.0) 999 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.148878+0000 mon.a (mon.0) 999 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.149078+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.149078+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.149286+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.149286+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.157576+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.157576+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.159445+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.50:0/104700156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.159445+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.50:0/104700156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.160496+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.50:0/3394545011' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.160496+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.50:0/3394545011' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.161662+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.50:0/2135820934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.161662+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.50:0/2135820934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.162164+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.50:0/1724061313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.162164+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.50:0/1724061313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.168460+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.168460+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.192886+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.192886+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.193285+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.193285+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.193661+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.193661+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.194065+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.194065+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.194394+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.311 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.194394+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.194692+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.194692+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.195181+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.195181+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.195620+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.195620+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.196100+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.196100+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.196739+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.196739+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.339910+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.339910+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:23:30.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.354179+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.354179+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.358925+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.50:0/1769465317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.358925+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.50:0/1769465317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.359631+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.359631+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.375041+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.375041+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.377625+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.377625+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.472022+0000 mgr.y (mgr.24401) 222 : cluster [DBG] pgmap v196: 1252 pgs: 1120 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:29.472022+0000 mgr.y (mgr.24401) 222 : cluster [DBG] pgmap v196: 1252 pgs: 1120 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.879497+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.879497+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.880076+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.880076+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.880379+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:29.880379+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:30.120535+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:23:30.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: cluster 2024-04-26T16:23:30.120535+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:23:30.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:30.124956+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[29234]: audit 2024-04-26T16:23:30.124956+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116628+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116628+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116731+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116731+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116823+0000 mon.a (mon.0) 988 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116823+0000 mon.a (mon.0) 988 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116941+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.116941+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117065+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117065+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117200+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117200+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117375+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117375+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117473+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117473+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117557+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117557+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117643+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.117643+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.130467+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.130467+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.134081+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.50:0/2828827295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.134081+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.50:0/2828827295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.134421+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.50:0/3409971702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.134421+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.50:0/3409971702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.135523+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.135523+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.136437+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.50:0/779541252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.136437+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.50:0/779541252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.136775+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.50:0/505282512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.136775+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.50:0/505282512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.137237+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.137237+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.141164+0000 mon.c (mon.2) 106 : audit [INF] from='client.? 172.21.15.50:0/3136352124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.141164+0000 mon.c (mon.2) 106 : audit [INF] from='client.? 172.21.15.50:0/3136352124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.146638+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.146638+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.148206+0000 mon.a (mon.0) 998 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.148206+0000 mon.a (mon.0) 998 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.148878+0000 mon.a (mon.0) 999 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.148878+0000 mon.a (mon.0) 999 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.149078+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.149078+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.149286+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.149286+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.157576+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.157576+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.159445+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.50:0/104700156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.159445+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.50:0/104700156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.160496+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.50:0/3394545011' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.160496+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.50:0/3394545011' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.161662+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.50:0/2135820934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.161662+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.50:0/2135820934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.162164+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.50:0/1724061313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.162164+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.50:0/1724061313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.168460+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.168460+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.192886+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.192886+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.193285+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.193285+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.193661+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.193661+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.194065+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.194065+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.194394+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.194394+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.194692+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.194692+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.195181+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.195181+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.195620+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.195620+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.196100+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.196100+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.196739+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.320 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.196739+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.339910+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.339910+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.354179+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.354179+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.358925+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.50:0/1769465317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.358925+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.50:0/1769465317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.359631+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.359631+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.375041+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.375041+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.377625+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.377625+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.472022+0000 mgr.y (mgr.24401) 222 : cluster [DBG] pgmap v196: 1252 pgs: 1120 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:30.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:29.472022+0000 mgr.y (mgr.24401) 222 : cluster [DBG] pgmap v196: 1252 pgs: 1120 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.879497+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.879497+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.880076+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.880076+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.880379+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:29.880379+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:30.120535+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: cluster 2024-04-26T16:23:30.120535+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:30.124956+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:30 smithi050 bash[21463]: audit 2024-04-26T16:23:30.124956+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:30.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116628+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116628+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi050-73193-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116731+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116731+0000 mon.a (mon.0) 987 : audit [INF] from='client.? 172.21.15.50:0/1486621511' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi050-73201-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116823+0000 mon.a (mon.0) 988 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116823+0000 mon.a (mon.0) 988 : audit [INF] from='client.? 172.21.15.50:0/3001143320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi050-73209-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116941+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.116941+0000 mon.a (mon.0) 989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi050-73217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117065+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117065+0000 mon.a (mon.0) 990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi050-73230-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117200+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117200+0000 mon.a (mon.0) 991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi050-73241-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117375+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117375+0000 mon.a (mon.0) 992 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi050-73246-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117473+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117473+0000 mon.a (mon.0) 993 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117557+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117557+0000 mon.a (mon.0) 994 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi050-74008-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117643+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.117643+0000 mon.a (mon.0) 995 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi050-74090-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.130467+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.130467+0000 mon.a (mon.0) 996 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.134081+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.50:0/2828827295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.134081+0000 mon.c (mon.2) 101 : audit [INF] from='client.? 172.21.15.50:0/2828827295' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.134421+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.50:0/3409971702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.134421+0000 mon.c (mon.2) 102 : audit [INF] from='client.? 172.21.15.50:0/3409971702' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.135523+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.135523+0000 mon.b (mon.1) 41 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.136437+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.50:0/779541252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.136437+0000 mon.c (mon.2) 103 : audit [INF] from='client.? 172.21.15.50:0/779541252' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.136775+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.50:0/505282512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.136775+0000 mon.c (mon.2) 104 : audit [INF] from='client.? 172.21.15.50:0/505282512' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.137237+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.137237+0000 mon.c (mon.2) 105 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.141164+0000 mon.c (mon.2) 106 : audit [INF] from='client.? 172.21.15.50:0/3136352124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.141164+0000 mon.c (mon.2) 106 : audit [INF] from='client.? 172.21.15.50:0/3136352124' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.146638+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.146638+0000 mon.a (mon.0) 997 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.148206+0000 mon.a (mon.0) 998 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.148206+0000 mon.a (mon.0) 998 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.148878+0000 mon.a (mon.0) 999 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.148878+0000 mon.a (mon.0) 999 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.149078+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.149078+0000 mon.a (mon.0) 1000 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.149286+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.149286+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.157576+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.157576+0000 mon.b (mon.1) 42 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.159445+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.50:0/104700156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.159445+0000 mon.b (mon.1) 43 : audit [INF] from='client.? 172.21.15.50:0/104700156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.160496+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.50:0/3394545011' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.160496+0000 mon.b (mon.1) 44 : audit [INF] from='client.? 172.21.15.50:0/3394545011' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.161662+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.50:0/2135820934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.161662+0000 mon.b (mon.1) 45 : audit [INF] from='client.? 172.21.15.50:0/2135820934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.162164+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.50:0/1724061313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.162164+0000 mon.b (mon.1) 46 : audit [INF] from='client.? 172.21.15.50:0/1724061313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.168460+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.168460+0000 mon.a (mon.0) 1002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.192886+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.192886+0000 mon.a (mon.0) 1003 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.193285+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.193285+0000 mon.a (mon.0) 1004 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.193661+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.193661+0000 mon.a (mon.0) 1005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.194065+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.194065+0000 mon.a (mon.0) 1006 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.194394+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.194394+0000 mon.a (mon.0) 1007 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.194692+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.194692+0000 mon.a (mon.0) 1008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.195181+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.195181+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.195620+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.195620+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.196100+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.196100+0000 mon.a (mon.0) 1011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.196739+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.196739+0000 mon.a (mon.0) 1012 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.339910+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.339910+0000 osd.5 (osd.5) 149 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.354179+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.354179+0000 osd.5 (osd.5) 150 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.358925+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.50:0/1769465317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.358925+0000 mon.c (mon.2) 107 : audit [INF] from='client.? 172.21.15.50:0/1769465317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.359631+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.359631+0000 mon.a (mon.0) 1013 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.375041+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.375041+0000 osd.7 (osd.7) 159 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.377625+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.377625+0000 osd.7 (osd.7) 160 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.472022+0000 mgr.y (mgr.24401) 222 : cluster [DBG] pgmap v196: 1252 pgs: 1120 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:29.472022+0000 mgr.y (mgr.24401) 222 : cluster [DBG] pgmap v196: 1252 pgs: 1120 unknown, 132 active+clean; 583 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.879497+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.879497+0000 mon.c (mon.2) 108 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.880076+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.880076+0000 mon.c (mon.2) 109 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.880379+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:29.880379+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:23:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:30.120535+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:23:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: cluster 2024-04-26T16:23:30.120535+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:23:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:30.124956+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:30 smithi190 bash[23868]: audit 2024-04-26T16:23:30.124956+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? 172.21.15.50:0/268206588' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi050-73536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:30.464 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:30.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:30.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:30.720 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:30.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:30.843 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:30.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:30.966 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:30.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:30.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:31.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:31.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:31.165 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : e api_cmd_pp: Running main() from gmock_main.cc 2024-04-26T16:23:31.165 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-04-26T16:23:31.165 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-04-26T16:23:31.165 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-04-26T16:23:31.165 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (98 ms) 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (57 ms) 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2822 ms) 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (2977 ms total) 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (2977 ms total) 2024-04-26T16:23:31.166 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (2904 ms) 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (2904 ms total) 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: 2024-04-26T16:23:31.174 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-04-26T16:23:31.175 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (2904 ms total) 2024-04-26T16:23:31.175 INFO:tasks.workunit.client.0.smithi050.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: Running main() from gmock_main.cc 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [----------] Global test environment set-up. 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2838 ms) 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [----------] 1 test from neocls_handler_error (2838 ms total) 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: 2024-04-26T16:23:31.180 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [----------] Global test environment tear-down 2024-04-26T16:23:31.181 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2838 ms total) 2024-04-26T16:23:31.181 INFO:tasks.workunit.client.0.smithi050.stdout: handler_error: [ PASSED ] 1 test. 2024-04-26T16:23:31.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-04-26T16:23:31.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (413 ms) 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (19 ms) 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (15 ms) 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (28 ms) 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10 ms) 2024-04-26T16:23:31.183 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (10 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (6 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (6 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (7 ms) 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-04-26T16:23:31.184 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (18 ms) 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (20 ms) 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (9 ms) 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (7 ms) 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (586 ms total) 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2971 ms total) 2024-04-26T16:23:31.185 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-04-26T16:23:31.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:31.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:31.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125045+0000 mon.a (mon.0) 1017 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125045+0000 mon.a (mon.0) 1017 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125122+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125122+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125219+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125219+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125326+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125326+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125416+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125416+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125517+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125517+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125624+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125624+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125745+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125745+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125910+0000 mon.a (mon.0) 1025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125910+0000 mon.a (mon.0) 1025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125963+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.125963+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126005+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126005+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126066+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126066+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126128+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126128+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126203+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.126203+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.142167+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.142167+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.150029+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.150029+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.156045+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.156045+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: cluster 2024-04-26T16:23:30.173087+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: cluster 2024-04-26T16:23:30.173087+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: cluster 2024-04-26T16:23:30.179677+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: cluster 2024-04-26T16:23:30.179677+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.183590+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.183590+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.259091+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.259091+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.310537+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.310537+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.313265+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.313265+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.316755+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.316755+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.360549+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.360549+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.361118+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.361118+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.368459+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.368459+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.369569+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.369569+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.396254+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.396254+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.398786+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.398786+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.408517+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.408517+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.414845+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:30.414845+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129128+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129128+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129193+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129193+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129240+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129240+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129278+0000 mon.a (mon.0) 1043 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129278+0000 mon.a (mon.0) 1043 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129318+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129318+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129364+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.129364+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: cluster 2024-04-26T16:23:31.134608+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: cluster 2024-04-26T16:23:31.134608+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.139977+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.139977+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.140513+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.140513+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.140875+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.140875+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.141192+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.141192+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.151285+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.151285+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.154806+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.154806+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.157501+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.157501+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.158150+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.158150+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.169512+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.169512+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.169746+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.169746+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.169941+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:31 smithi190 bash[23868]: audit 2024-04-26T16:23:31.169941+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.351 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: Running main() from gmock_main.cc 2024-04-26T16:23:31.351 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-04-26T16:23:31.351 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [----------] Global test environment set-up. 2024-04-26T16:23:31.351 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-04-26T16:23:31.351 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (69 ms) 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (112 ms) 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2783 ms) 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125045+0000 mon.a [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125122+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125219+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125326+0000 mon.a [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125416+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125517+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125624+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.352 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125745+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125910+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.125963+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.126005+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.126066+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.126128+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.126203+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.353 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.156045+0000 mon.c [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.179677+0000 mon.a [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.183590+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.259091+0000 mon.b [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.310537+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.313265+0000 mon.b [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.316755+0000 mon.b [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.360549+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.361118+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.368459+0000 mon.b [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.369569+0000 mon.b [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.396254+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.354 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.398786+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.408517+0000 mon.b [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:30.414845+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.129128+0000 mon.a [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.129193+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.129240+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.129278+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.129318+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.129364+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.139977+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.140513+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.140875+0000 mon.b [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.141192+0000 mon.b [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.355 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.154806+0000 mon.b [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.356 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.157501+0000 mon.b [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.356 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.158150+0000 mon.c [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.356 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.169512+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125045+0000 mon.a (mon.0) 1017 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125045+0000 mon.a (mon.0) 1017 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125122+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125122+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125219+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125219+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125326+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125326+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125416+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125416+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125517+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125517+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125624+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125624+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125745+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125745+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125910+0000 mon.a (mon.0) 1025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125910+0000 mon.a (mon.0) 1025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125963+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.125963+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126005+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126005+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126066+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126066+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126128+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126128+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126203+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.126203+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.142167+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.142167+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.150029+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.150029+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.156045+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.156045+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: cluster 2024-04-26T16:23:30.173087+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: cluster 2024-04-26T16:23:30.173087+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: cluster 2024-04-26T16:23:30.179677+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: cluster 2024-04-26T16:23:30.179677+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.183590+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.183590+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.259091+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.259091+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.310537+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.310537+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.313265+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.313265+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.316755+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.316755+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.360549+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.360549+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.361118+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.361118+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.368459+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.368459+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.369569+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.369569+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.396254+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.396254+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.398786+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.398786+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.408517+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.408517+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.414845+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:30.414845+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129128+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129128+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129193+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129193+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129240+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129240+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129278+0000 mon.a (mon.0) 1043 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129278+0000 mon.a (mon.0) 1043 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129318+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129318+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129364+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.129364+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: cluster 2024-04-26T16:23:31.134608+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: cluster 2024-04-26T16:23:31.134608+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.139977+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.139977+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.140513+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.140513+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.140875+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.140875+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.141192+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125045+0000 mon.a (mon.0) 1017 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125045+0000 mon.a (mon.0) 1017 : audit [INF] from='client.? 172.21.15.50:0/469390519' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73615-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125122+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125122+0000 mon.a (mon.0) 1018 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125219+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125219+0000 mon.a (mon.0) 1019 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi050-73420-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125326+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125326+0000 mon.a (mon.0) 1020 : audit [INF] from='client.? 172.21.15.50:0/4165092570' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73583-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125416+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125416+0000 mon.a (mon.0) 1021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi050-73432-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125517+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125517+0000 mon.a (mon.0) 1022 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125624+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125624+0000 mon.a (mon.0) 1023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi050-73489-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125745+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125745+0000 mon.a (mon.0) 1024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73708-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125910+0000 mon.a (mon.0) 1025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125910+0000 mon.a (mon.0) 1025 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi050-73469-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125963+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.125963+0000 mon.a (mon.0) 1026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126005+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126005+0000 mon.a (mon.0) 1027 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi050-73372-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126066+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126066+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi050-73686-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126128+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126128+0000 mon.a (mon.0) 1029 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126203+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.126203+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.142167+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.142167+0000 mon.c (mon.2) 110 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.150029+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.150029+0000 mon.c (mon.2) 111 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.156045+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.156045+0000 mon.c (mon.2) 112 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: cluster 2024-04-26T16:23:30.173087+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: cluster 2024-04-26T16:23:30.173087+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: cluster 2024-04-26T16:23:30.179677+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: cluster 2024-04-26T16:23:30.179677+0000 mon.a (mon.0) 1032 : cluster [WRN] Health check failed: 28 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.183590+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.183590+0000 mon.a (mon.0) 1033 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.259091+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.259091+0000 mon.b (mon.1) 47 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.310537+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.310537+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.313265+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.313265+0000 mon.b (mon.1) 48 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.316755+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.316755+0000 mon.b (mon.1) 49 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.360549+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.360549+0000 mon.a (mon.0) 1035 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.361118+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.361118+0000 mon.a (mon.0) 1036 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.368459+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.368459+0000 mon.b (mon.1) 50 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.369569+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.369569+0000 mon.b (mon.1) 51 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.396254+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.396254+0000 mon.a (mon.0) 1037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.398786+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.398786+0000 mon.a (mon.0) 1038 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.408517+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.408517+0000 mon.b (mon.1) 52 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.414845+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:30.414845+0000 mon.a (mon.0) 1039 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129128+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129128+0000 mon.a (mon.0) 1040 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi050-74090-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129193+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129193+0000 mon.a (mon.0) 1041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi050-74008-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129240+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129240+0000 mon.a (mon.0) 1042 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi050-73572-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129278+0000 mon.a (mon.0) 1043 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129278+0000 mon.a (mon.0) 1043 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129318+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129318+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi050-73209-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129364+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.129364+0000 mon.a (mon.0) 1045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: cluster 2024-04-26T16:23:31.134608+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: cluster 2024-04-26T16:23:31.134608+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.139977+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.139977+0000 mon.a (mon.0) 1047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.140513+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.140513+0000 mon.a (mon.0) 1048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.140875+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.140875+0000 mon.b (mon.1) 53 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.141192+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.141192+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.151285+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.151285+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.154806+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.154806+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.157501+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.157501+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.158150+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.158150+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.169512+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.169512+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.169746+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.169746+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.169941+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[29234]: audit 2024-04-26T16:23:31.169941+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.141192+0000 mon.b (mon.1) 54 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.151285+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.151285+0000 mon.c (mon.2) 113 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:31.392 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.154806+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.154806+0000 mon.b (mon.1) 55 : audit [INF] from='client.? 172.21.15.50:0/450862114' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.157501+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.157501+0000 mon.b (mon.1) 56 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.158150+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.158150+0000 mon.c (mon.2) 114 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.169512+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.169512+0000 mon.a (mon.0) 1049 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.169746+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.169746+0000 mon.a (mon.0) 1050 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.169941+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.393 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:31 smithi050 bash[21463]: audit 2024-04-26T16:23:31.169941+0000 mon.a (mon.0) 1051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:31.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:31.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:2 list_parallel: process_1_[73787]: starting. 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_1_[73787]: creating pool ceph_test_rados_list_parallel.smithi050-73746 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_1_[73787]: created object 0... 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_1_[73787]: created object 25... 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_1_[73787]: created object 49... 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_1_[73787]: finishing. 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_1_[73787]: shutting down. 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_2_[73789]: starting. 2024-04-26T16:23:31.475 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_2_[73789]: listing objects. 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_2_[73789]: listed object 0... 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_2_[73789]: listed object 25... 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_2_[73789]: saw 50 objects 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_2_[73789]: shutting down. 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: starting. 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: creating pool ceph_test_rados_list_parallel.smithi050-73746 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: created object 0... 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: created object 25... 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: created object 49... 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: finishing. 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_3_[74717]: shutting down. 2024-04-26T16:23:31.476 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_4_[74718]: starting. 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_4_[74718]: listing objects. 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_4_[74718]: listed object 0... 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_4_[74718]: listed object 25... 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_4_[74718]: saw 43 objects 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_4_[74718]: shutting down. 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_5_[74719]: starting. 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_5_[74719]: removed 25 objects... 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_5_[74719]: removed half of the objects 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_5_[74719]: removed 50 objects... 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_5_[74719]: removed 50 objects 2024-04-26T16:23:31.477 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_5_[74719]: shutting down. 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: starting. 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: creating pool ceph_test_rados_list_parallel.smithi050-73746 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: created object 0... 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: created object 25... 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: created object 49... 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: finishing. 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_6_[74765]: shutting down. 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: starting. 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: listing objects. 2024-04-26T16:23:31.478 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: listed object 0... 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: listed object 25... 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: listed object 50... 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: saw 57 objects 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_7_[74766]: shutting down. 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_8_[74767]: starting. 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_8_[74767]: added 25 objects... 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_8_[74767]: added half of the objects 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_8_[74767]: added 50 objects... 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_8_[74767]: added 50 objects 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_8_[74767]: shutting down. 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.479 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:31.559 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:31.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:31.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: starting. 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: creating pool ceph_test_rados_list_parallel.smithi050-73746 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: created object 0... 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: created object 25... 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: created object 49... 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: finishing. 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_9_[74813]: shutting down. 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.712 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: starting. 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: listing objects. 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: listed object 0... 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: listed object 25... 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: listed object 50... 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: listed object 75... 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: saw 98 objects 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_10_[74814]: shutting down. 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_13_[74817]: starting. 2024-04-26T16:23:31.713 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_13_[74817]: removed 25 objects... 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_13_[74817]: removed half of the objects 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_13_[74817]: removed 50 objects... 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_13_[74817]: removed 50 objects 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_13_[74817]: shutting down. 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_11_[74815]: starting. 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_11_[74815]: added 25 objects... 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_11_[74815]: added half of the objects 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_11_[74815]: added 50 objects... 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_11_[74815]: added 50 objects 2024-04-26T16:23:31.714 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_11_[74815]: shutting down. 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_12_[74816]: starting. 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_12_[74816]: added 25 objects... 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_12_[74816]: added half of the objects 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_12_[74816]: added 50 objects... 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_12_[74816]: added 50 objects 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_12_[74816]: shutting down. 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.715 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: starting. 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: creating pool ceph_test_rados_list_parallel.smithi050-73746 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: created object 0... 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: created object 25... 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: created object 49... 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: finishing. 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_14_[75175]: shutting down. 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: starting. 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listing objects. 2024-04-26T16:23:31.716 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listed object 0... 2024-04-26T16:23:31.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:31.806 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:31.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:31.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:31.930 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:31.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:32.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.056 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:32.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:32.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:32.229 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listed object 25... 2024-04-26T16:23:32.229 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listed object 50... 2024-04-26T16:23:32.229 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listed object 75... 2024-04-26T16:23:32.229 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listed object 100... 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: listed object 125... 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: saw 150 objects 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_15_[75176]: shutting down. 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_16_[75177]: starting. 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_16_[75177]: added 25 objects... 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_16_[75177]: added half of the objects 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_16_[75177]: added 50 objects... 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_16_[75177]: added 50 objects 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: process_16_[75177]: shutting down. 2024-04-26T16:23:32.230 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.231 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.231 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.231 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.231 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******************************* 2024-04-26T16:23:32.231 INFO:tasks.workunit.client.0.smithi050.stdout: list_parallel: ******* SUCCESS ********** 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout:3:31.169746+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.169941+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.265887+0000 mon.b [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.266144+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.281709+0000 mon.b [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.327558+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.337200+0000 mon.b [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.355621+0000 mon.c [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.266 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.357192+0000 client.admin [INF] onexx 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.371328+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.383940+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.395833+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.396080+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.416732+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.474563+0000 mon.c [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:31.476896+0000 mon.a [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.267 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.140145+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.265887+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.265887+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.266144+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.266144+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.281709+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.281709+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.327558+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.327558+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.337200+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.337200+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.355621+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.355621+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: cluster 2024-04-26T16:23:31.357192+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: cluster 2024-04-26T16:23:31.357192+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.371328+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.371328+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.383940+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.383940+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.395833+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.395833+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.396080+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.396080+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.416732+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.416732+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: cluster 2024-04-26T16:23:31.474176+0000 mgr.y (mgr.24401) 223 : cluster [DBG] pgmap v199: 732 pgs: 248 creating+peering, 484 active+clean; 599 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 76 KiB/s wr, 245 op/s 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: cluster 2024-04-26T16:23:31.474176+0000 mgr.y (mgr.24401) 223 : cluster [DBG] pgmap v199: 732 pgs: 248 creating+peering, 484 active+clean; 599 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 76 KiB/s wr, 245 op/s 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.474563+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.474563+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.476896+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:31.476896+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.140145+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.140145+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.140911+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.140911+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141124+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141124+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141389+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141389+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141605+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141605+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141790+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.141790+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.153034+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.153034+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.156786+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.156786+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: cluster 2024-04-26T16:23:32.182527+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: cluster 2024-04-26T16:23:32.182527+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.193285+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.193285+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.193793+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.193793+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.199078+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.199078+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.201159+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.201159+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.201500+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:32 smithi190 bash[23868]: audit 2024-04-26T16:23:32.201500+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:32.350 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.265887+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.265887+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.266144+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.266144+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.281709+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.281709+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.327558+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.327558+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.265887+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.265887+0000 mon.b (mon.1) 57 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.266144+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.266144+0000 mon.a (mon.0) 1052 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.281709+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.281709+0000 mon.b (mon.1) 58 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.327558+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.327558+0000 mon.a (mon.0) 1053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.337200+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.337200+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.355621+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.355621+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: cluster 2024-04-26T16:23:31.357192+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: cluster 2024-04-26T16:23:31.357192+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-04-26T16:23:32.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.371328+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.371328+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:32.473 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:32.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:32.594 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.337200+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.337200+0000 mon.b (mon.1) 59 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.355621+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.355621+0000 mon.c (mon.2) 115 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: cluster 2024-04-26T16:23:31.357192+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: cluster 2024-04-26T16:23:31.357192+0000 client.admin (client.?) 0 : cluster [INF] onexx 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.371328+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.371328+0000 mon.a (mon.0) 1054 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.383940+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.383940+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.395833+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.395833+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.396080+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.396080+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.416732+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.383940+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.383940+0000 mon.a (mon.0) 1055 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.395833+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.395833+0000 mon.a (mon.0) 1056 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.396080+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.396080+0000 mon.a (mon.0) 1057 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.416732+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.416732+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: cluster 2024-04-26T16:23:31.474176+0000 mgr.y (mgr.24401) 223 : cluster [DBG] pgmap v199: 732 pgs: 248 creating+peering, 484 active+clean; 599 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 76 KiB/s wr, 245 op/s 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: cluster 2024-04-26T16:23:31.474176+0000 mgr.y (mgr.24401) 223 : cluster [DBG] pgmap v199: 732 pgs: 248 creating+peering, 484 active+clean; 599 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 76 KiB/s wr, 245 op/s 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.474563+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.474563+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.476896+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:31.476896+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.416732+0000 mon.a (mon.0) 1058 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: cluster 2024-04-26T16:23:31.474176+0000 mgr.y (mgr.24401) 223 : cluster [DBG] pgmap v199: 732 pgs: 248 creating+peering, 484 active+clean; 599 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 76 KiB/s wr, 245 op/s 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: cluster 2024-04-26T16:23:31.474176+0000 mgr.y (mgr.24401) 223 : cluster [DBG] pgmap v199: 732 pgs: 248 creating+peering, 484 active+clean; 599 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 76 KiB/s wr, 245 op/s 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.474563+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.474563+0000 mon.c (mon.2) 116 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.476896+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:31.476896+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.140145+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.140145+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.140911+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.140911+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141124+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141124+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141389+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.140145+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.140145+0000 mon.a (mon.0) 1060 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi050-73193-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.140911+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.140911+0000 mon.a (mon.0) 1061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141124+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141124+0000 mon.a (mon.0) 1062 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141389+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141389+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141605+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141605+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141790+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.141790+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.153034+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.153034+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141389+0000 mon.a (mon.0) 1063 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141605+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141605+0000 mon.a (mon.0) 1064 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141790+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.141790+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.153034+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.153034+0000 mon.c (mon.2) 117 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.156786+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.156786+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: cluster 2024-04-26T16:23:32.182527+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: cluster 2024-04-26T16:23:32.182527+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.193285+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.193285+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.193793+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.193793+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.199078+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.199078+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.201159+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.201159+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.201500+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[29234]: audit 2024-04-26T16:23:32.201500+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.156786+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.156786+0000 mon.b (mon.1) 60 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: cluster 2024-04-26T16:23:32.182527+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: cluster 2024-04-26T16:23:32.182527+0000 mon.a (mon.0) 1066 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2024-04-26T16:23:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.193285+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.193285+0000 mon.a (mon.0) 1067 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.193793+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.193793+0000 mon.a (mon.0) 1068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.199078+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.199078+0000 mon.b (mon.1) 61 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.201159+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.201159+0000 mon.b (mon.1) 62 : audit [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.201500+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:32 smithi050 bash[21463]: audit 2024-04-26T16:23:32.201500+0000 mon.b (mon.1) 63 : audit [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:32.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.718 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:32.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:32.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.965 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:32.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:33.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:33.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:33.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:33.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:33.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:33.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.247068+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.247068+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.247758+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.247758+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.248006+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.248006+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.267977+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-26T16:23:33.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.267977+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.282861+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.282861+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.284996+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.284996+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.285614+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.285614+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: cluster 2024-04-26T16:23:32.328497+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: cluster 2024-04-26T16:23:32.328497+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-04-26T16:23:33.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.339186+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.339186+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.342258+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.342258+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.348608+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.348608+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.352005+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.352005+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.354644+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.354644+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.380253+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.380253+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.383643+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.383643+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.393454+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.393454+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.395767+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.395767+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.410826+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.410826+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.413288+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:32.413288+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149071+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149071+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149240+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149240+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149339+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149339+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149450+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149450+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149551+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149551+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149652+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149652+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149746+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.149746+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.164544+0000 mon.c (mon.2) 125 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.164544+0000 mon.c (mon.2) 125 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.165881+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.165881+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.166300+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.166300+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.168731+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.168731+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: cluster 2024-04-26T16:23:33.182435+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: cluster 2024-04-26T16:23:33.182435+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.185378+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.185378+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.187592+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.187592+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.188116+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.188116+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.191379+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.191379+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.192910+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.192910+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.193278+0000 mon.a (mon.0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.193278+0000 mon.a (mon.0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.193600+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.193600+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.194158+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.194158+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.194370+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.194370+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.194734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[21463]: audit 2024-04-26T16:23:33.194734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.247068+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.247068+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.247758+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.247758+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.248006+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.248006+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.267977+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.267977+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.282861+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.282861+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.284996+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.284996+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.285614+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.285614+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: cluster 2024-04-26T16:23:32.328497+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: cluster 2024-04-26T16:23:32.328497+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.339186+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.339186+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.342258+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.342258+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.348608+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.348608+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.352005+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.352005+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.354644+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.354644+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.380253+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.380253+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.383643+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.383643+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.393454+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.393454+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.395767+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.395767+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.410826+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.410826+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.413288+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:32.413288+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149071+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149071+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149240+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149240+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149339+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149339+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:33.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149450+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149450+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149551+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149551+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149652+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149652+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149746+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.149746+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.164544+0000 mon.c (mon.2) 125 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.164544+0000 mon.c (mon.2) 125 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.165881+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.165881+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.166300+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.390 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.166300+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.168731+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.168731+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: cluster 2024-04-26T16:23:33.182435+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: cluster 2024-04-26T16:23:33.182435+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.185378+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.185378+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.187592+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.187592+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.188116+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.391 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.188116+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.191379+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.191379+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.192910+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.192910+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.193278+0000 mon.a (mon.0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.193278+0000 mon.a (mon.0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.193600+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.193600+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.194158+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.194158+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.194370+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.194370+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.194734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.392 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:33 smithi050 bash[29234]: audit 2024-04-26T16:23:33.194734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.393 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:33.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:33.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:33.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.247068+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.247068+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.247758+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.247758+0000 mon.a (mon.0) 1070 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.248006+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.248006+0000 mon.a (mon.0) 1071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.267977+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.267977+0000 mon.a (mon.0) 1072 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.282861+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.282861+0000 mon.c (mon.2) 118 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.284996+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.284996+0000 mon.c (mon.2) 119 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.285614+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.285614+0000 mon.c (mon.2) 120 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: cluster 2024-04-26T16:23:32.328497+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-04-26T16:23:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: cluster 2024-04-26T16:23:32.328497+0000 client.admin (client.?) 0 : cluster [INF] twoxx 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.339186+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.339186+0000 mon.a (mon.0) 1073 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.342258+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.342258+0000 mon.c (mon.2) 121 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.348608+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.348608+0000 mon.a (mon.0) 1074 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.352005+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.352005+0000 mon.a (mon.0) 1075 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.354644+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.354644+0000 mon.c (mon.2) 122 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.380253+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.380253+0000 mon.a (mon.0) 1076 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.383643+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.383643+0000 mon.c (mon.2) 123 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.393454+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.393454+0000 mon.a (mon.0) 1077 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.395767+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.395767+0000 mon.c (mon.2) 124 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.410826+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.410826+0000 mon.a (mon.0) 1078 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.413288+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:32.413288+0000 mon.a (mon.0) 1079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149071+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149071+0000 mon.a (mon.0) 1080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi050-73209-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149240+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149240+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi050-73217-23", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149339+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149339+0000 mon.a (mon.0) 1082 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149450+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149450+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149551+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149551+0000 mon.a (mon.0) 1084 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149652+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149652+0000 mon.a (mon.0) 1085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149746+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.149746+0000 mon.a (mon.0) 1086 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi050-73241-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.164544+0000 mon.c (mon.2) 125 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.164544+0000 mon.c (mon.2) 125 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.165881+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.165881+0000 mon.c (mon.2) 126 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.166300+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.166300+0000 mon.c (mon.2) 127 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.168731+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.168731+0000 mon.b (mon.1) 64 : audit [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: cluster 2024-04-26T16:23:33.182435+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: cluster 2024-04-26T16:23:33.182435+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2024-04-26T16:23:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.185378+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.185378+0000 mon.c (mon.2) 128 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.187592+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.187592+0000 mon.b (mon.1) 65 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.188116+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.188116+0000 mon.b (mon.1) 66 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.191379+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.191379+0000 mon.a (mon.0) 1088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.192910+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.192910+0000 mon.a (mon.0) 1089 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.193278+0000 mon.a (mon.0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.193278+0000 mon.a (mon.0) 1090 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.193600+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.193600+0000 mon.a (mon.0) 1091 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.194158+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.194158+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.194370+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.194370+0000 mon.a (mon.0) 1093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.194734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:33 smithi190 bash[23868]: audit 2024-04-26T16:23:33.194734+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:33.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:33.658 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:33.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:33.785 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:33.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:33.903 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:33.903 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:33.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:33.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:34.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:34.044 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:34.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:34.184 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:34.210 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.140911+0000 mon.a [INF] from='client.? ' entity=' cls: Running main() from gmock_main.cc 2024-04-26T16:23:34.210 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [----------] Global test environment set-up. 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [ OK ] NeoRadosCls.DNE (2884 ms) 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (3050 ms) 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [----------] 2 tests from NeoRadosCls (5934 ms total) 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [----------] Global test environment tear-down 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [==========] 2 tests from 1 test suite ran. (5934 ms total) 2024-04-26T16:23:34.211 INFO:tasks.workunit.client.0.smithi050.stdout: cls: [ PASSED ] 2 tests. 2024-04-26T16:23:34.230 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: starting. 2024-04-26T16:23:34.230 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: creating pool ceph_test_rados_delete_pools_parallel.smithi050-73800 2024-04-26T16:23:34.230 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: created object 0... 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: created object 25... 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: created object 49... 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: finishing. 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_1_[73876]: shutting down. 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_2_[73877]: starting. 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_2_[73877]: deleting pool ceph_test_rados_delete_pools_parallel.smithi050-73800 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_2_[73877]: shutting down. 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: ******************************* 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: starting. 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: creating pool ceph_test_rados_delete_pools_parallel.smithi050-73800 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: created object 0... 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: created object 25... 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: created object 49... 2024-04-26T16:23:34.231 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: finishing. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_3_[74903]: shutting down. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: ******************************* 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_5_[74905]: starting. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_5_[74905]: listing objects. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_5_[74905]: listed object 0... 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_5_[74905]: listed object 25... 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_5_[74905]: saw 50 objects 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_5_[74905]: shutting down. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: ******************************* 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_4_[74904]: starting. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_4_[74904]: deleting pool ceph_test_rados_delete_pools_parallel.smithi050-73800 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: process_4_[74904]: shutting down. 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: ******************************* 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: ******************************* 2024-04-26T16:23:34.232 INFO:tasks.workunit.client.0.smithi050.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: starting. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: creating pool ceph_test_rados_open_pools_parallel.smithi050-73763 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: created object 0... 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: created object 25... 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: created object 49... 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: finishing. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_1_[73804]: shutting down. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_2_[73805]: starting. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_2_[73805]: rados_pool_create. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_2_[73805]: rados_ioctx_create. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_2_[73805]: shutting down. 2024-04-26T16:23:34.242 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: ******************************* 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: starting. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: creating pool ceph_test_rados_open_pools_parallel.smithi050-73763 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: created object 0... 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: created object 25... 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: created object 49... 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: finishing. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_3_[74906]: shutting down. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: ******************************* 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_4_[74907]: starting. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_4_[74907]: rados_pool_create. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_4_[74907]: rados_ioctx_create. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: process_4_[74907]: shutting down. 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: ******************************* 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: ******************************* 2024-04-26T16:23:34.243 INFO:tasks.workunit.client.0.smithi050.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-04-26T16:23:34.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:34.295 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:33.279579+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:34.295 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:33.279579+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:34.295 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: cluster 2024-04-26T16:23:33.475373+0000 mgr.y (mgr.24401) 224 : cluster [DBG] pgmap v202: 980 pgs: 528 unknown, 96 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 173 op/s 2024-04-26T16:23:34.295 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: cluster 2024-04-26T16:23:33.475373+0000 mgr.y (mgr.24401) 224 : cluster [DBG] pgmap v202: 980 pgs: 528 unknown, 96 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 173 op/s 2024-04-26T16:23:34.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:34.448 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: Running main() from gmock_main.cc 2024-04-26T16:23:34.448 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [==========] Running 38 tests from 2 test suites. 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [----------] Global test environment set-up. 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: seed 73217 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (216 ms) 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (7 ms) 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (3 ms) 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (297 ms) 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-04-26T16:23:34.449 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (3 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (2 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2 ms) 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-04-26T16:23:34.450 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (3 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (9 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-04-26T16:23:34.451 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3 ms) 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [----------] 21 tests from LibRadosIoPP (578 ms total) 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1258 ms) 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (17 ms) 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-04-26T16:23:34.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (8 ms) 2024-04-26T16:23:34.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:34.474 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:33.279579+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:33.279579+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: cluster 2024-04-26T16:23:33.475373+0000 mgr.y (mgr.24401) 224 : cluster [DBG] pgmap v202: 980 pgs: 528 unknown, 96 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 173 op/s 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: cluster 2024-04-26T16:23:33.475373+0000 mgr.y (mgr.24401) 224 : cluster [DBG] pgmap v202: 980 pgs: 528 unknown, 96 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 173 op/s 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:33.710035+0000 mon.a (mon.0) 1096 : audit [DBG] from='client.? 172.21.15.50:0/3790656342' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:33.710035+0000 mon.a (mon.0) 1096 : audit [DBG] from='client.? 172.21.15.50:0/3790656342' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157128+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157128+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157410+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157410+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157692+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157692+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157865+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.157865+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.158048+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.158048+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.158253+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.158253+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.158464+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.158464+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.168813+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.168813+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.172109+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.172109+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.179563+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.179563+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.182347+0000 mon.c (mon.2) 130 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.182347+0000 mon.c (mon.2) 130 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.183329+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.183329+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.202724+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.202724+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.203912+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.203912+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: cluster 2024-04-26T16:23:34.204882+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: cluster 2024-04-26T16:23:34.204882+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.213348+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.213348+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.213714+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.213714+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.213928+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.213928+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.214619+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.214619+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.215596+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.215596+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.232535+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.232535+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.236153+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.236153+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.239740+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.239740+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.248571+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.248571+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.251938+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.251938+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.256249+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.256249+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.262417+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:34 smithi190 bash[23868]: audit 2024-04-26T16:23:34.262417+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:34.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:34.600 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:34.626 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:23:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:23:34.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:33.710035+0000 mon.a (mon.0) 1096 : audit [DBG] from='client.? 172.21.15.50:0/3790656342' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:34.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:33.710035+0000 mon.a (mon.0) 1096 : audit [DBG] from='client.? 172.21.15.50:0/3790656342' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:34.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157128+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157128+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157410+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157410+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157692+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157692+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157865+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:34.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.157865+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:34.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.158048+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.158048+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.158253+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:33.279579+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:33.279579+0000 mon.a (mon.0) 1095 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: cluster 2024-04-26T16:23:33.475373+0000 mgr.y (mgr.24401) 224 : cluster [DBG] pgmap v202: 980 pgs: 528 unknown, 96 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 173 op/s 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: cluster 2024-04-26T16:23:33.475373+0000 mgr.y (mgr.24401) 224 : cluster [DBG] pgmap v202: 980 pgs: 528 unknown, 96 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 21 KiB/s wr, 173 op/s 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:33.710035+0000 mon.a (mon.0) 1096 : audit [DBG] from='client.? 172.21.15.50:0/3790656342' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:33.710035+0000 mon.a (mon.0) 1096 : audit [DBG] from='client.? 172.21.15.50:0/3790656342' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157128+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157128+0000 mon.a (mon.0) 1097 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157410+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157410+0000 mon.a (mon.0) 1098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157692+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:34.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157692+0000 mon.a (mon.0) 1099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:34.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157865+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:34.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.157865+0000 mon.a (mon.0) 1100 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.158253+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.158464+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.158464+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.168813+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.168813+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.172109+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.172109+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.179563+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.179563+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.182347+0000 mon.c (mon.2) 130 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.182347+0000 mon.c (mon.2) 130 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.183329+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.183329+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.202724+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.202724+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:34.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.203912+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.158048+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.158048+0000 mon.a (mon.0) 1101 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.158253+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.158253+0000 mon.a (mon.0) 1102 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.158464+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.158464+0000 mon.a (mon.0) 1103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.168813+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.168813+0000 mon.b (mon.1) 67 : audit [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.172109+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:34.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.172109+0000 mon.c (mon.2) 129 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.179563+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.179563+0000 mon.b (mon.1) 68 : audit [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.182347+0000 mon.c (mon.2) 130 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.182347+0000 mon.c (mon.2) 130 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.183329+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.203912+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: cluster 2024-04-26T16:23:34.204882+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: cluster 2024-04-26T16:23:34.204882+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-04-26T16:23:34.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.213348+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.213348+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.213714+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.213714+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.213928+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.213928+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.214619+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.214619+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.215596+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.215596+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.232535+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.232535+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.236153+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.183329+0000 mon.c (mon.2) 131 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.202724+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.202724+0000 mon.c (mon.2) 132 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.203912+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.203912+0000 mon.c (mon.2) 133 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: cluster 2024-04-26T16:23:34.204882+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: cluster 2024-04-26T16:23:34.204882+0000 mon.a (mon.0) 1104 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.213348+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.213348+0000 mon.a (mon.0) 1105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:34.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.213714+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.213714+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.213928+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.213928+0000 mon.a (mon.0) 1107 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.214619+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.214619+0000 mon.a (mon.0) 1108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.215596+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:34.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.215596+0000 mon.c (mon.2) 134 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.236153+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.239740+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.239740+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.248571+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.248571+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.251938+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:34.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.251938+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.256249+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.256249+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.262417+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[21463]: audit 2024-04-26T16:23:34.262417+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.232535+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.232535+0000 mon.c (mon.2) 135 : audit [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.236153+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.236153+0000 mon.a (mon.0) 1109 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.239740+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.239740+0000 mon.c (mon.2) 136 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:34.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.248571+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.248571+0000 mon.c (mon.2) 137 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.251938+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.251938+0000 mon.c (mon.2) 138 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.256249+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.256249+0000 mon.c (mon.2) 139 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.262417+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:34.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:34 smithi050 bash[29234]: audit 2024-04-26T16:23:34.262417+0000 mon.c (mon.2) 140 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:34.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:34.732 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:34.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:34.862 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:34.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:35.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:35.190 INFO:tasks.workunit.client.0.smithi050.stdout: a cmd: Running main() from gmock_main.cc 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [----------] Global test environment set-up. 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1901 ms) 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2025 ms) 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (2976 ms) 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6902 ms total) 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [----------] Global test environment tear-down 2024-04-26T16:23:35.191 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6903 ms total) 2024-04-26T16:23:35.192 INFO:tasks.workunit.client.0.smithi050.stdout: cmd: [ PASSED ] 3 tests. 2024-04-26T16:23:35.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:35.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:35.287 INFO:tasks.workunit.client.0.smithi050.stdout:client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.287 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.141124+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.141389+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi050-73469-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.141605+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.141790+0000 mon.a [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.156786+0000 mon.b [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.193285+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.193793+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.199078+0000 mon.b [INF] from='client.? 172.21.15.50:0/1010670023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi050-74008-1"}]: dispatch 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.201159+0000 mon.b [INF] from='client.? 172.21.15.50:0/4018775550' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:32.201500+0000 mon.b [INF] from='client.? 172.21.15.50:0/522221863' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:33.279579+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.157128+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi050-73469-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.157410+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi050-73489-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:35.288 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.157692+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi050-74008-1"}]': finished 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.157865+0000 mon.a [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.158048+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.158253+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi050-73201-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.158464+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.168813+0000 mon.b [INF] from='client.? 172.21.15.50:0/2896264396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.179563+0000 mon.b [INF] from='client.? 172.21.15.50:0/1643474775' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.182347+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.183329+0000 mon.c [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.202724+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.203912+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.213348+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.213714+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.289 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.213928+0000 mon.a [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]: dispatch 2024-04-26T16:23:35.290 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.214619+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-04-26T16:23:35.290 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.215596+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.290 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.232535+0000 mon.c [INF] from='client.? 172.21.15.50:0/2543245543' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.236153+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.239740+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.248571+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.251938+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.256249+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.262417+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.274983+0000 mon.c [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.278929+0000 mon.c [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:34.381315+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.167531+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.167815+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.168081+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.168229+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.168374+0000 mon.a [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.168557+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.168788+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.168984+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:35.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.183637+0000 mgr.y (mgr.24401) 225 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.183637+0000 mgr.y (mgr.24401) 225 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.203204+0000 mgr.y (mgr.24401) 226 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.203204+0000 mgr.y (mgr.24401) 226 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.204067+0000 mgr.y (mgr.24401) 227 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.204067+0000 mgr.y (mgr.24401) 227 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.216208+0000 mgr.y (mgr.24401) 228 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.216208+0000 mgr.y (mgr.24401) 228 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.242479+0000 mgr.y (mgr.24401) 229 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.242479+0000 mgr.y (mgr.24401) 229 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.248911+0000 mgr.y (mgr.24401) 230 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.248911+0000 mgr.y (mgr.24401) 230 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.253855+0000 mgr.y (mgr.24401) 231 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.253855+0000 mgr.y (mgr.24401) 231 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.256760+0000 mgr.y (mgr.24401) 232 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.256760+0000 mgr.y (mgr.24401) 232 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.262680+0000 mgr.y (mgr.24401) 233 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.262680+0000 mgr.y (mgr.24401) 233 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.274983+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.274983+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.276830+0000 mgr.y (mgr.24401) 234 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.276830+0000 mgr.y (mgr.24401) 234 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.278929+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.278929+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.381315+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:34.381315+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: cluster 2024-04-26T16:23:34.810345+0000 osd.4 (osd.4) 121 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: cluster 2024-04-26T16:23:34.810345+0000 osd.4 (osd.4) 121 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: cluster 2024-04-26T16:23:34.812885+0000 osd.4 (osd.4) 122 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: cluster 2024-04-26T16:23:34.812885+0000 osd.4 (osd.4) 122 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:23:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.167531+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.167531+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.167815+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.167815+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168081+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168081+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168229+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168229+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168374+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168374+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168557+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168557+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168788+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168788+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168984+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.168984+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.180948+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.180948+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.182992+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.182992+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.185628+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.185628+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.190792+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.190792+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.193753+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.193753+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.194194+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.194194+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.218585+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.218585+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.223335+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.223335+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.224346+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.224346+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.225066+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.225066+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.225922+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.225922+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.226385+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.226385+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.226717+0000 mon.a (mon.0) 1124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.226717+0000 mon.a (mon.0) 1124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.229991+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.229991+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.246519+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.246519+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.248157+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.248157+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.295787+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:35.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:35 smithi190 bash[23868]: audit 2024-04-26T16:23:35.295787+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:35.392 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:35.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:23:35.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:35.521 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:35.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:35.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.183637+0000 mgr.y (mgr.24401) 225 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.183637+0000 mgr.y (mgr.24401) 225 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.203204+0000 mgr.y (mgr.24401) 226 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.203204+0000 mgr.y (mgr.24401) 226 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.204067+0000 mgr.y (mgr.24401) 227 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.204067+0000 mgr.y (mgr.24401) 227 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.216208+0000 mgr.y (mgr.24401) 228 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.216208+0000 mgr.y (mgr.24401) 228 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.242479+0000 mgr.y (mgr.24401) 229 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.242479+0000 mgr.y (mgr.24401) 229 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.248911+0000 mgr.y (mgr.24401) 230 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.248911+0000 mgr.y (mgr.24401) 230 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.253855+0000 mgr.y (mgr.24401) 231 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.253855+0000 mgr.y (mgr.24401) 231 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.256760+0000 mgr.y (mgr.24401) 232 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.256760+0000 mgr.y (mgr.24401) 232 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.262680+0000 mgr.y (mgr.24401) 233 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.262680+0000 mgr.y (mgr.24401) 233 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.274983+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.183637+0000 mgr.y (mgr.24401) 225 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.183637+0000 mgr.y (mgr.24401) 225 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.203204+0000 mgr.y (mgr.24401) 226 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.203204+0000 mgr.y (mgr.24401) 226 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.204067+0000 mgr.y (mgr.24401) 227 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.204067+0000 mgr.y (mgr.24401) 227 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.216208+0000 mgr.y (mgr.24401) 228 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.216208+0000 mgr.y (mgr.24401) 228 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.242479+0000 mgr.y (mgr.24401) 229 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.242479+0000 mgr.y (mgr.24401) 229 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.248911+0000 mgr.y (mgr.24401) 230 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.248911+0000 mgr.y (mgr.24401) 230 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.253855+0000 mgr.y (mgr.24401) 231 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.253855+0000 mgr.y (mgr.24401) 231 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.256760+0000 mgr.y (mgr.24401) 232 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.256760+0000 mgr.y (mgr.24401) 232 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.262680+0000 mgr.y (mgr.24401) 233 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.262680+0000 mgr.y (mgr.24401) 233 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.274983+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.274983+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.276830+0000 mgr.y (mgr.24401) 234 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.276830+0000 mgr.y (mgr.24401) 234 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.278929+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.278929+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.381315+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:34.381315+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: cluster 2024-04-26T16:23:34.810345+0000 osd.4 (osd.4) 121 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: cluster 2024-04-26T16:23:34.810345+0000 osd.4 (osd.4) 121 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: cluster 2024-04-26T16:23:34.812885+0000 osd.4 (osd.4) 122 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: cluster 2024-04-26T16:23:34.812885+0000 osd.4 (osd.4) 122 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.167531+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.167531+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.167815+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.167815+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168081+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.274983+0000 mon.c (mon.2) 141 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.276830+0000 mgr.y (mgr.24401) 234 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.276830+0000 mgr.y (mgr.24401) 234 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.278929+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.278929+0000 mon.c (mon.2) 142 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.381315+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:34.381315+0000 mon.a (mon.0) 1110 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: cluster 2024-04-26T16:23:34.810345+0000 osd.4 (osd.4) 121 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: cluster 2024-04-26T16:23:34.810345+0000 osd.4 (osd.4) 121 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: cluster 2024-04-26T16:23:34.812885+0000 osd.4 (osd.4) 122 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:23:35.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: cluster 2024-04-26T16:23:34.812885+0000 osd.4 (osd.4) 122 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.167531+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.167531+0000 mon.a (mon.0) 1111 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi050-73241-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.167815+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.167815+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi050-73246-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168081+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168081+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168229+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168229+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168374+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168374+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168557+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168557+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168788+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168788+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168984+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.168984+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.180948+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.180948+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.182992+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168081+0000 mon.a (mon.0) 1113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi050-73572-1"}]': finished 2024-04-26T16:23:35.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168229+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168229+0000 mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168374+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168374+0000 mon.a (mon.0) 1115 : audit [INF] from='client.? 172.21.15.50:0/2446911789' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi050-74090-1"}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168557+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168557+0000 mon.a (mon.0) 1116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi050-73201-3", "field": "max_bytes", "val": "4096"}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168788+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168788+0000 mon.a (mon.0) 1117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi050-73193-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168984+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.168984+0000 mon.a (mon.0) 1118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi050-74008-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.180948+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.180948+0000 mon.c (mon.2) 143 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.182992+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.182992+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.185628+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.185628+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.190792+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.190792+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.193753+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.193753+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.194194+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.194194+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.218585+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.218585+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.223335+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.223335+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.224346+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.224346+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.225066+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.225066+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.225922+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.225922+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.226385+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.226385+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.226717+0000 mon.a (mon.0) 1124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.226717+0000 mon.a (mon.0) 1124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.229991+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.229991+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.246519+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.246519+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.248157+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.248157+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.295787+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:35.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[29234]: audit 2024-04-26T16:23:35.295787+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.182992+0000 mon.c (mon.2) 144 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.185628+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.185628+0000 mon.a (mon.0) 1119 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.190792+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.190792+0000 mon.c (mon.2) 145 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.193753+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.193753+0000 mon.b (mon.1) 69 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.194194+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.194194+0000 mon.b (mon.1) 70 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.218585+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.218585+0000 mon.c (mon.2) 146 : audit [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.223335+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.223335+0000 mon.c (mon.2) 147 : audit [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.224346+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.224346+0000 mon.a (mon.0) 1120 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.225066+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.225066+0000 mon.a (mon.0) 1121 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.225922+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.225922+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.226385+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.226385+0000 mon.a (mon.0) 1123 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.226717+0000 mon.a (mon.0) 1124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.226717+0000 mon.a (mon.0) 1124 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.229991+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.229991+0000 mon.a (mon.0) 1125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.246519+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.246519+0000 mon.c (mon.2) 148 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.248157+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.248157+0000 mon.a (mon.0) 1126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.295787+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:35.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:35 smithi050 bash[21463]: audit 2024-04-26T16:23:35.295787+0000 mon.c (mon.2) 149 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:35.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:35.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:35.781 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:35.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:35.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:35.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:36.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:36.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:36.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:36.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:36.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:36.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.180948+0000 mon.c [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:36.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.182992+0000 mon.c [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:36.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.193753+0000 mon.b [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.194194+0000 mon.b [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.218585+0000 mon.c [INF] from='client.? 172.21.15.50:0/2447682240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.310 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.223335+0000 mon.c [INF] from='client.? 172.21.15.50:0/2561118861' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.224346+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.225066+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.225922+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.226385+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.226717+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.229991+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.246519+0000 mon.c [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.248157+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.295787+0000 mon.c [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.298486+0000 client.admin [INF] threexx 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.302733+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.311 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:35.992054+0000 mon.a [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.176909+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.176992+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.177056+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.177111+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.177207+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.177316+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.186627+0000 mon.c [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.194836+0000 mon.b [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.199539+0000 mon.c [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.200320+0000 mon.c [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.200589+0000 mon.b [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.312 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.202537+0000 mon.b [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.313 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.211333+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.313 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.212116+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.342 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:36.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:36.472 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.298486+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.298486+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:35.302733+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:35.302733+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.476431+0000 mgr.y (mgr.24401) 235 : cluster [DBG] pgmap v205: 868 pgs: 448 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.476431+0000 mgr.y (mgr.24401) 235 : cluster [DBG] pgmap v205: 868 pgs: 448 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:35.746864+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? 172.21.15.50:0/373569517' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:35.746864+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? 172.21.15.50:0/373569517' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.778785+0000 osd.4 (osd.4) 123 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.778785+0000 osd.4 (osd.4) 123 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.780750+0000 osd.4 (osd.4) 124 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.780750+0000 osd.4 (osd.4) 124 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.992054+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:35.992054+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.176909+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.176909+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.176992+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.176992+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177056+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177056+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177111+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177111+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177207+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177207+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177316+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.177316+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.186627+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.186627+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.194836+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.194836+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.197176+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.197176+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:36.197496+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: cluster 2024-04-26T16:23:36.197496+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-04-26T16:23:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.199539+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.199539+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.200320+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.200320+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.200589+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.200589+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.202537+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.202537+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.211333+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.211333+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.212116+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.212116+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.212602+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.212602+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.212946+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.212946+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.213202+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:36.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.213202+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:36.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.231842+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.231842+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.233015+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[23868]: audit 2024-04-26T16:23:36.233015+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:36.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.298486+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.298486+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:35.302733+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:35.302733+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.476431+0000 mgr.y (mgr.24401) 235 : cluster [DBG] pgmap v205: 868 pgs: 448 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.476431+0000 mgr.y (mgr.24401) 235 : cluster [DBG] pgmap v205: 868 pgs: 448 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:35.746864+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? 172.21.15.50:0/373569517' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:35.746864+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? 172.21.15.50:0/373569517' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.778785+0000 osd.4 (osd.4) 123 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.778785+0000 osd.4 (osd.4) 123 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.780750+0000 osd.4 (osd.4) 124 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.780750+0000 osd.4 (osd.4) 124 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:23:36.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.992054+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:35.992054+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.176909+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.176909+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.176992+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.176992+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177056+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.298486+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.298486+0000 client.admin (client.?) 0 : cluster [INF] threexx 2024-04-26T16:23:36.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:35.302733+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:35.302733+0000 mon.a (mon.0) 1127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.476431+0000 mgr.y (mgr.24401) 235 : cluster [DBG] pgmap v205: 868 pgs: 448 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.476431+0000 mgr.y (mgr.24401) 235 : cluster [DBG] pgmap v205: 868 pgs: 448 unknown, 64 creating+peering, 356 active+clean; 586 KiB data, 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:35.746864+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? 172.21.15.50:0/373569517' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:35.746864+0000 mon.c (mon.2) 150 : audit [DBG] from='client.? 172.21.15.50:0/373569517' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.778785+0000 osd.4 (osd.4) 123 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.778785+0000 osd.4 (osd.4) 123 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.780750+0000 osd.4 (osd.4) 124 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.780750+0000 osd.4 (osd.4) 124 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.992054+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:35.992054+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check update: 18 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.176909+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.176909+0000 mon.a (mon.0) 1129 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-4", "overlaypool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.176992+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.176992+0000 mon.a (mon.0) 1130 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177056+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177056+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177111+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177111+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177207+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177207+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177316+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.177316+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.589 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.186627+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.186627+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.194836+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.194836+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.197176+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.197176+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:36.197496+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: cluster 2024-04-26T16:23:36.197496+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177056+0000 mon.a (mon.0) 1131 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177111+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177111+0000 mon.a (mon.0) 1132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:36.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177207+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177207+0000 mon.a (mon.0) 1133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177316+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.177316+0000 mon.a (mon.0) 1134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi050-74090-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.186627+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.186627+0000 mon.c (mon.2) 151 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.194836+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.194836+0000 mon.b (mon.1) 71 : audit [INF] from='client.? 172.21.15.50:0/332266018' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.197176+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.197176+0000 mon.c (mon.2) 152 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:36.197496+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-04-26T16:23:36.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: cluster 2024-04-26T16:23:36.197496+0000 mon.a (mon.0) 1135 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.199539+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.199539+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.200320+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.200320+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.200589+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.200589+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.202537+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.202537+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.211333+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.211333+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.212116+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.212116+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.212602+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.212602+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.199539+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.199539+0000 mon.c (mon.2) 153 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.200320+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.200320+0000 mon.c (mon.2) 154 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.200589+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.200589+0000 mon.b (mon.1) 72 : audit [INF] from='client.? 172.21.15.50:0/402651857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.202537+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.202537+0000 mon.b (mon.1) 73 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.211333+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.211333+0000 mon.a (mon.0) 1136 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.212116+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.212116+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.212602+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.212602+0000 mon.a (mon.0) 1138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.212946+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.212946+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.213202+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.213202+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.231842+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.231842+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.233015+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[21463]: audit 2024-04-26T16:23:36.233015+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.212946+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.212946+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:36.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.213202+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:36.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.213202+0000 mon.a (mon.0) 1140 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:36.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.231842+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.231842+0000 mon.a (mon.0) 1141 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:36.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.233015+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:36 smithi050 bash[29234]: audit 2024-04-26T16:23:36.233015+0000 mon.a (mon.0) 1142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:36.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:36.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:36.722 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:36.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:36.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:36.948 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:36.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:36.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:36.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:37.069 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:23:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:23:37.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:37.101 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:37.199 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.212602+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefi api_io: Running main() from gmock_main.cc 2024-04-26T16:23:37.199 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-04-26T16:23:37.199 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [----------] Global test environment set-up. 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (229 ms) 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: no timeout :/ 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: no timeout :/ 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: no timeout :/ 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: no timeout :/ 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: no timeout :/ 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (32 ms) 2024-04-26T16:23:37.200 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (5 ms) 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (4 ms) 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-04-26T16:23:37.201 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4 ms) 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (3 ms) 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.TruncTest (3 ms) 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (5 ms) 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (9 ms) 2024-04-26T16:23:37.202 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.RmXattr (11 ms) 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIo.XattrIter (7 ms) 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [----------] 14 tests from LibRadosIo (324 ms total) 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-04-26T16:23:37.203 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1263 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (5 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (10 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (9 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (5 ms) 2024-04-26T16:23:37.204 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5 ms) 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (11 ms) 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1323 ms total) 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [----------] Global test environment tear-down 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9250 ms total) 2024-04-26T16:23:37.205 INFO:tasks.workunit.client.0.smithi050.stdout: api_io: [ PASSED ] 24 tests. 2024-04-26T16:23:37.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:37.206 INFO:tasks.workunit.client.0.smithi050.stdout:pi_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-04-26T16:23:37.206 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (7 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (5 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-04-26T16:23:37.207 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9 ms) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (5 ms) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1351 ms total) 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: 2024-04-26T16:23:37.208 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [----------] Global test environment tear-down 2024-04-26T16:23:37.209 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [==========] 38 tests from 2 test suites ran. (9240 ms total) 2024-04-26T16:23:37.209 INFO:tasks.workunit.client.0.smithi050.stdout: api_io_pp: [ PASSED ] 38 tests. 2024-04-26T16:23:37.230 INFO:tasks.workunit.client.0.smithi050.stdout: list: Running main() from gmock_main.cc 2024-04-26T16:23:37.230 INFO:tasks.workunit.client.0.smithi050.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-04-26T16:23:37.230 INFO:tasks.workunit.client.0.smithi050.stdout: list: [----------] Global test environment set-up. 2024-04-26T16:23:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: list: [----------] 3 tests from NeoradosList 2024-04-26T16:23:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-04-26T16:23:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ OK ] NeoradosList.ListObjects (2808 ms) 2024-04-26T16:23:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-04-26T16:23:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3048 ms) 2024-04-26T16:23:37.231 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-04-26T16:23:37.232 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3022 ms) 2024-04-26T16:23:37.232 INFO:tasks.workunit.client.0.smithi050.stdout: list: [----------] 3 tests from NeoradosList (8878 ms total) 2024-04-26T16:23:37.232 INFO:tasks.workunit.client.0.smithi050.stdout: list: 2024-04-26T16:23:37.232 INFO:tasks.workunit.client.0.smithi050.stdout: list: [----------] Global test environment tear-down 2024-04-26T16:23:37.232 INFO:tasks.workunit.client.0.smithi050.stdout: list: [==========] 3 tests from 1 test suite ran. (8879 ms total) 2024-04-26T16:23:37.233 INFO:tasks.workunit.client.0.smithi050.stdout: list: [ PASSED ] 3 tests. 2024-04-26T16:23:37.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:37.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:37.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:37.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:37.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.314670+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-26T16:23:37.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.314670+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-26T16:23:37.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.326028+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.326028+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: cluster 2024-04-26T16:23:36.326625+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-04-26T16:23:37.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: cluster 2024-04-26T16:23:36.326625+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.349956+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.349956+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.611894+0000 mgr.y (mgr.24401) 236 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:36.611894+0000 mgr.y (mgr.24401) 236 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: cluster 2024-04-26T16:23:37.181201+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: cluster 2024-04-26T16:23:37.181201+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.184633+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.184633+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.184826+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.184826+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185004+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:37.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185004+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185146+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185146+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185277+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185277+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185461+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185461+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185659+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:37.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.185659+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.196816+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.196816+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.202089+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.202089+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: cluster 2024-04-26T16:23:37.210782+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: cluster 2024-04-26T16:23:37.210782+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.214111+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.214111+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.216693+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.216693+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.216748+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.216748+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.217476+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.217476+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.222678+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:37 smithi190 bash[23868]: audit 2024-04-26T16:23:37.222678+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:37.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:37.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:37.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.314670+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-26T16:23:37.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.314670+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-26T16:23:37.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.326028+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.326028+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: cluster 2024-04-26T16:23:36.326625+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: cluster 2024-04-26T16:23:36.326625+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.349956+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.349956+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.611894+0000 mgr.y (mgr.24401) 236 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:36.611894+0000 mgr.y (mgr.24401) 236 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: cluster 2024-04-26T16:23:37.181201+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: cluster 2024-04-26T16:23:37.181201+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.184633+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.184633+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.184826+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.184826+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:37.639 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185004+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185004+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.314670+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.314670+0000 mon.a (mon.0) 1143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.326028+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.326028+0000 mon.c (mon.2) 155 : audit [INF] from='client.? 172.21.15.50:0/1787718292' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: cluster 2024-04-26T16:23:36.326625+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: cluster 2024-04-26T16:23:36.326625+0000 client.admin (client.?) 0 : cluster [INF] fourxx 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.349956+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.349956+0000 mon.a (mon.0) 1144 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.611894+0000 mgr.y (mgr.24401) 236 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:36.611894+0000 mgr.y (mgr.24401) 236 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: cluster 2024-04-26T16:23:37.181201+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: cluster 2024-04-26T16:23:37.181201+0000 mon.a (mon.0) 1145 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.184633+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.184633+0000 mon.a (mon.0) 1146 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi050-74008-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.184826+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.184826+0000 mon.a (mon.0) 1147 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi050-73217-23"}]': finished 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185004+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:37.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185004+0000 mon.a (mon.0) 1148 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185146+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185146+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185277+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185277+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185461+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185461+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185659+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.185659+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.196816+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.196816+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.642 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.202089+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.202089+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: cluster 2024-04-26T16:23:37.210782+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: cluster 2024-04-26T16:23:37.210782+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.214111+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.214111+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.216693+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.216693+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.216748+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.216748+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.217476+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.217476+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.222678+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.643 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[21463]: audit 2024-04-26T16:23:37.222678+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185146+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]': finished 2024-04-26T16:23:37.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185146+0000 mon.a (mon.0) 1149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]': finished 2024-04-26T16:23:37.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185277+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:37.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185277+0000 mon.a (mon.0) 1150 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:37.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185461+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:37.644 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185461+0000 mon.a (mon.0) 1151 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185659+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.185659+0000 mon.a (mon.0) 1152 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.196816+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.196816+0000 mon.b (mon.1) 74 : audit [INF] from='client.? 172.21.15.50:0/1818626867' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.202089+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.202089+0000 mon.c (mon.2) 156 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: cluster 2024-04-26T16:23:37.210782+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: cluster 2024-04-26T16:23:37.210782+0000 mon.a (mon.0) 1153 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.214111+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.214111+0000 mon.a (mon.0) 1154 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.216693+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.645 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.216693+0000 mon.b (mon.1) 75 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.216748+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.216748+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:37.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.217476+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.217476+0000 mon.a (mon.0) 1156 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.222678+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.646 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:37 smithi050 bash[29234]: audit 2024-04-26T16:23:37.222678+0000 mon.a (mon.0) 1157 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:37.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:37.757 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-04-26T16:23:37.757 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [----------] Global test environment set-up. 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [----------] 4 tests from LibRadosService 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5115 ms) 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ OK ] LibRadosService.RegisterLate (42 ms) 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: cluster: 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: id: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: health: HEALTH_WARN 2024-04-26T16:23:37.758 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: noscrub flag(s) set 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 13 pool(s) do not have an application enabled 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: services: 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: mgr: y(active, since 5m), standbys: x 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 7m) 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: flags noscrub 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.759 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: data: 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: pools: 27 pools, 732 pgs 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: objects: 347 objects, 599 KiB 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: usage: 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: pgs: 33.880% pgs not active 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 484 active+clean 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 248 creating+peering 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: io: 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: client: 3.5 KiB/s rd, 76 KiB/s wr, 73 op/s rd, 172 op/s wr 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: cluster: 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: id: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:23:37.760 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: health: HEALTH_WARN 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: noscrub flag(s) set 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 18 pool(s) do not have an application enabled 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: services: 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 13m) 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: mgr: y(active, since 5m), standbys: x 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 7m) 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: flags noscrub 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: data: 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: pools: 34 pools, 980 pgs 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: objects: 238 objects, 586 KiB 2024-04-26T16:23:37.761 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: usage: 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: pgs: 53.878% pgs unknown 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 9.796% pgs not active 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 528 unknown 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 356 active+clean 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 96 creating+peering 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: io: 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: client: 3.0 KiB/s rd, 21 KiB/s wr, 71 op/s rd, 102 op/s wr 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: cluster: 2024-04-26T16:23:37.762 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: id: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:23:37.763 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: health: HEALTH_WARN 2024-04-26T16:23:37.763 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 1 cache pools are missing hit_sets 2024-04-26T16:23:37.763 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: noscrub flag(s) set 2024-04-26T16:23:37.763 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 15 pool(s) do not have an application enabled 2024-04-26T16:23:37.763 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:37.763 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: services: 2024-04-26T16:23:37.775 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:37.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:37.903 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:37.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:38.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:38.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:38.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:38.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:38.161 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:38.218 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: mon: 3 daem api_stat: Running main() from gmock_main.cc 2024-04-26T16:23:38.218 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-04-26T16:23:38.218 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [----------] Global test environment set-up. 2024-04-26T16:23:38.218 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-04-26T16:23:38.218 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStat.Stat (371 ms) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (80 ms) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStat.StatNS (24 ms) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (2 ms) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [----------] 5 tests from LibRadosStat (478 ms total) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1086 ms) 2024-04-26T16:23:38.219 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (49 ms) 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (5 ms) 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3 ms) 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1143 ms total) 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [----------] Global test environment tear-down 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10122 ms total) 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat: [ PASSED ] 9 tests. 2024-04-26T16:23:38.220 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: Running main() from gmock_main.cc 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [----------] Global test environment set-up. 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: handle_notify cookie 94105806419568 notify_id 609885356033 notifier_gid 24920 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2756 ms) 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: Trying... 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: handle_notify cookie 94105819834896 notify_id 622770257921 notifier_gid 15084 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: Waiting for 3.000000000s 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: Timed out. 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: Flushing... 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: Flushed... 2024-04-26T16:23:38.221 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7057 ms) 2024-04-26T16:23:38.222 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9813 ms total) 2024-04-26T16:23:38.222 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: 2024-04-26T16:23:38.222 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [----------] Global test environment tear-down 2024-04-26T16:23:38.222 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9813 ms total) 2024-04-26T16:23:38.222 INFO:tasks.workunit.client.0.smithi050.stdout: watch_notify: [ PASSED ] 2 tests. 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: seed 73489 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-04-26T16:23:38.246 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (287 ms) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (50 ms) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (10 ms) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (84 ms) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (431 ms total) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1157 ms) 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-04-26T16:23:38.247 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (21 ms) 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (7 ms) 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (8 ms) 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1193 ms total) 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10136 ms total) 2024-04-26T16:23:38.248 INFO:tasks.workunit.client.0.smithi050.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-04-26T16:23:38.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:38.298 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:38.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:38.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.325304+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.325304+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: cluster 2024-04-26T16:23:37.477177+0000 mgr.y (mgr.24401) 237 : cluster [DBG] pgmap v208: 732 pgs: 64 creating+peering, 1 active+clean+snaptrim, 104 unknown, 563 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 381 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: cluster 2024-04-26T16:23:37.477177+0000 mgr.y (mgr.24401) 237 : cluster [DBG] pgmap v208: 732 pgs: 64 creating+peering, 1 active+clean+snaptrim, 104 unknown, 563 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 381 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.477876+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.477876+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.479751+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.479751+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.693398+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.693398+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.694099+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.694099+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.700949+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.700949+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.701343+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.701343+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.765430+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? 172.21.15.50:0/599270807' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:37.765430+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? 172.21.15.50:0/599270807' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.193668+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.193668+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.193983+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.193983+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194183+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194183+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194429+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194429+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194664+0000 mon.a (mon.0) 1166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194664+0000 mon.a (mon.0) 1166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194912+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.194912+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-26T16:23:38.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.195173+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]': finished 2024-04-26T16:23:38.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.195173+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]': finished 2024-04-26T16:23:38.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.195399+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]': finished 2024-04-26T16:23:38.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.195399+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]': finished 2024-04-26T16:23:38.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: cluster 2024-04-26T16:23:38.205890+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-04-26T16:23:38.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: cluster 2024-04-26T16:23:38.205890+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-04-26T16:23:38.416 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.217765+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.217765+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.220476+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.220476+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.222839+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.222839+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.223314+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:38.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.223314+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:38.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.230232+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.230232+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.230639+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.230639+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.231048+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.418 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:38 smithi190 bash[23868]: audit 2024-04-26T16:23:38.231048+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.423 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:38.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:38.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:38.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:38.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.325304+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-26T16:23:38.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.325304+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-26T16:23:38.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: cluster 2024-04-26T16:23:37.477177+0000 mgr.y (mgr.24401) 237 : cluster [DBG] pgmap v208: 732 pgs: 64 creating+peering, 1 active+clean+snaptrim, 104 unknown, 563 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 381 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:38.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: cluster 2024-04-26T16:23:37.477177+0000 mgr.y (mgr.24401) 237 : cluster [DBG] pgmap v208: 732 pgs: 64 creating+peering, 1 active+clean+snaptrim, 104 unknown, 563 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 381 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:38.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.477876+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.477876+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.479751+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.479751+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.693398+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.693398+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.694099+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.694099+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.700949+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.700949+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.701343+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.701343+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.765430+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? 172.21.15.50:0/599270807' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:37.765430+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? 172.21.15.50:0/599270807' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.193668+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.193668+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.193983+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.193983+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194183+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194183+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:38.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194429+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194429+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194664+0000 mon.a (mon.0) 1166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194664+0000 mon.a (mon.0) 1166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194912+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.194912+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.325304+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.325304+0000 mon.a (mon.0) 1158 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: cluster 2024-04-26T16:23:37.477177+0000 mgr.y (mgr.24401) 237 : cluster [DBG] pgmap v208: 732 pgs: 64 creating+peering, 1 active+clean+snaptrim, 104 unknown, 563 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 381 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:38.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: cluster 2024-04-26T16:23:37.477177+0000 mgr.y (mgr.24401) 237 : cluster [DBG] pgmap v208: 732 pgs: 64 creating+peering, 1 active+clean+snaptrim, 104 unknown, 563 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 381 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.477876+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.477876+0000 mon.c (mon.2) 157 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.479751+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.479751+0000 mon.a (mon.0) 1159 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.693398+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.693398+0000 mon.c (mon.2) 158 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.694099+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.694099+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.700949+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.700949+0000 mon.c (mon.2) 159 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.701343+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.195173+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]': finished 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.195173+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]': finished 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.195399+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]': finished 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.195399+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]': finished 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: cluster 2024-04-26T16:23:38.205890+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: cluster 2024-04-26T16:23:38.205890+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.217765+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.217765+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.220476+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.220476+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.222839+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.222839+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.223314+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.223314+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.230232+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.230232+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.230639+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.230639+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.231048+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[21463]: audit 2024-04-26T16:23:38.231048+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:38.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.701343+0000 mon.a (mon.0) 1161 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]: dispatch 2024-04-26T16:23:38.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.765430+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? 172.21.15.50:0/599270807' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:38.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:37.765430+0000 mon.c (mon.2) 160 : audit [DBG] from='client.? 172.21.15.50:0/599270807' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-04-26T16:23:38.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.193668+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:38.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.193668+0000 mon.a (mon.0) 1162 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi050-74090-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.193983+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.193983+0000 mon.a (mon.0) 1163 : audit [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi050-73469-7"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194183+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194183+0000 mon.a (mon.0) 1164 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi050-73489-7"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194429+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194429+0000 mon.a (mon.0) 1165 : audit [INF] from='client.? 172.21.15.50:0/2833113482' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi050-73193-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194664+0000 mon.a (mon.0) 1166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194664+0000 mon.a (mon.0) 1166 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194912+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.194912+0000 mon.a (mon.0) 1167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.195173+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.195173+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73572-6", "pg_num": 4}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.195399+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]': finished 2024-04-26T16:23:38.688 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.195399+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-4"}]': finished 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: cluster 2024-04-26T16:23:38.205890+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: cluster 2024-04-26T16:23:38.205890+0000 mon.a (mon.0) 1170 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.217765+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.217765+0000 mon.c (mon.2) 161 : audit [INF] from='client.? 172.21.15.50:0/1499025373' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.220476+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.220476+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.222839+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.222839+0000 mon.c (mon.2) 162 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.223314+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.223314+0000 mon.c (mon.2) 163 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:38.689 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.230232+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.230232+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:38.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.230639+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.230639+0000 mon.a (mon.0) 1173 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.231048+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.690 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:38 smithi050 bash[29234]: audit 2024-04-26T16:23:38.231048+0000 mon.a (mon.0) 1174 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:38.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:38.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:38.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:38.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:38.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:38.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:39.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:39.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:39.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:39.220 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:39.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout:x": "osd crush rule rm", "name":"LibRadosIoEC_smithi050-73209-16"}]: dispatch 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.212946+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-4-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.213202+0000 mon.a [INF] from='client.? 172.21.15.50:0/1420668021' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi050-73469-7"}]: dispatch 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.231842+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: got: 2024-04-26T16:23:36.233015+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi050-73489-7"}]: dispatch 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8206 ms) 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (11170 ms total) 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [----------] Global test environment tear-down 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (11184 ms total) 2024-04-26T16:23:39.420 INFO:tasks.workunit.client.0.smithi050.stdout: api_cmd: [ PASSED ] 4 tests. 2024-04-26T16:23:39.440 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:39.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:39.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:38.509675+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-26T16:23:39.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:38.509675+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-26T16:23:39.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:38.512954+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:38.512954+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:38.514318+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:38.514318+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.197099+0000 mon.a (mon.0) 1176 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.197099+0000 mon.a (mon.0) 1176 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.200856+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.200856+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201082+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201082+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201268+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201268+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201436+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201436+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201565+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.201565+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.211780+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.211780+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-04-26T16:23:39.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.215686+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.215686+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.218741+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.218741+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.220438+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.220438+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.226605+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.226605+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.251907+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.251907+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.252725+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.252725+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.254074+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.254074+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.254369+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.254369+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.254823+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.254823+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.255426+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.255426+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.256814+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.256814+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.266122+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:39 smithi190 bash[23868]: audit 2024-04-26T16:23:39.266122+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:39.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:38.509675+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:38.509675+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:38.512954+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:38.512954+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:38.514318+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:38.514318+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.197099+0000 mon.a (mon.0) 1176 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.197099+0000 mon.a (mon.0) 1176 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.200856+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.200856+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201082+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:39.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201082+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201268+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201268+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:38.509675+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:38.509675+0000 mon.b (mon.1) 76 : audit [DBG] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:38.512954+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:38.512954+0000 mon.a (mon.0) 1175 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:38.514318+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:38.514318+0000 mon.b (mon.1) 77 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.197099+0000 mon.a (mon.0) 1176 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.197099+0000 mon.a (mon.0) 1176 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.200856+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.200856+0000 mon.a (mon.0) 1177 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-4", "tierpool": "test-rados-api-smithi050-73372-4-cache"}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201082+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201082+0000 mon.a (mon.0) 1178 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201268+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201268+0000 mon.a (mon.0) 1179 : audit [INF] from='client.? 172.21.15.50:0/2923522450' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201436+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201436+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201565+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.201565+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.211780+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.211780+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.215686+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2"}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.215686+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2"}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.218741+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.218741+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.220438+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.220438+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.226605+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.226605+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.251907+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201436+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201436+0000 mon.a (mon.0) 1180 : audit [INF] from='client.? 172.21.15.50:0/4012015274' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201565+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.201565+0000 mon.a (mon.0) 1181 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.211780+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.211780+0000 mon.a (mon.0) 1182 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.215686+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2"}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.215686+0000 mon.b (mon.1) 78 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2"}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.218741+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.218741+0000 mon.b (mon.1) 79 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.220438+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.220438+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:39.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.226605+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.226605+0000 mon.c (mon.2) 164 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.251907+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.251907+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.252725+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.252725+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.254074+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.254074+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.254369+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.254369+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.254823+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.254823+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.255426+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.255426+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.256814+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.256814+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.266122+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[21463]: audit 2024-04-26T16:23:39.266122+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.251907+0000 mon.c (mon.2) 165 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.252725+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.252725+0000 mon.c (mon.2) 166 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.254074+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.254074+0000 mon.c (mon.2) 167 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.254369+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.254369+0000 mon.a (mon.0) 1184 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.254823+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.254823+0000 mon.a (mon.0) 1185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.255426+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.255426+0000 mon.a (mon.0) 1186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.256814+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.256814+0000 mon.c (mon.2) 168 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.266122+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:39 smithi050 bash[29234]: audit 2024-04-26T16:23:39.266122+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:39.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:39.698 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:39.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:39.828 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:39.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:39.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:39.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:39.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:40.017 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:23:40.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:40.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:40.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:40.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:40.366 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:40.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:40.525 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:40.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:40.654 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:40.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:40.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.375637+0000 osd.7 (osd.7) 161 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:23:40.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.375637+0000 osd.7 (osd.7) 161 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.378485+0000 osd.7 (osd.7) 162 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.378485+0000 osd.7 (osd.7) 162 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.478574+0000 mgr.y (mgr.24401) 238 : cluster [DBG] pgmap v211: 784 pgs: 32 creating+peering, 1 active+clean+snaptrim, 300 unknown, 451 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 376 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.478574+0000 mgr.y (mgr.24401) 238 : cluster [DBG] pgmap v211: 784 pgs: 32 creating+peering, 1 active+clean+snaptrim, 300 unknown, 451 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 376 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.555361+0000 osd.0 (osd.0) 143 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.555361+0000 osd.0 (osd.0) 143 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.558178+0000 osd.0 (osd.0) 144 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:39.558178+0000 osd.0 (osd.0) 144 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207053+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207053+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207377+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207377+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:40.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207644+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207644+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207890+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.207890+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.208124+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.208124+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.218693+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.218693+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-04-26T16:23:40.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.220950+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.220950+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.221482+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.221482+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.221897+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.221897+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.225050+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.225050+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.225371+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.225371+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.225816+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.375637+0000 osd.7 (osd.7) 161 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.375637+0000 osd.7 (osd.7) 161 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.378485+0000 osd.7 (osd.7) 162 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.378485+0000 osd.7 (osd.7) 162 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.478574+0000 mgr.y (mgr.24401) 238 : cluster [DBG] pgmap v211: 784 pgs: 32 creating+peering, 1 active+clean+snaptrim, 300 unknown, 451 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 376 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.478574+0000 mgr.y (mgr.24401) 238 : cluster [DBG] pgmap v211: 784 pgs: 32 creating+peering, 1 active+clean+snaptrim, 300 unknown, 451 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 376 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.555361+0000 osd.0 (osd.0) 143 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.555361+0000 osd.0 (osd.0) 143 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.558178+0000 osd.0 (osd.0) 144 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:39.558178+0000 osd.0 (osd.0) 144 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207053+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:40.773 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207053+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207377+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207377+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207644+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207644+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207890+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.207890+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.208124+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.208124+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.218693+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.218693+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.220950+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.220950+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.221482+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.221482+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.221897+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.221897+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.225050+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.225050+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.225371+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.225371+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.225816+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.775 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.225816+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.226668+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.50:0/4267792152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.226668+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.50:0/4267792152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.228462+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.228462+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.230715+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.230715+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.235639+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.235639+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.235796+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.235796+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.239280+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.239280+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.239952+0000 mon.a (mon.0) 1197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.239952+0000 mon.a (mon.0) 1197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.240548+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.240548+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.241129+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.241129+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.241644+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.241644+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.242114+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[29234]: audit 2024-04-26T16:23:40.242114+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.225816+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.226668+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.50:0/4267792152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.226668+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.50:0/4267792152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.228462+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.228462+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.230715+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.230715+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.235639+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.235639+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.235796+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.235796+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.239280+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.239280+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.239952+0000 mon.a (mon.0) 1197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.239952+0000 mon.a (mon.0) 1197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.240548+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.240548+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.241129+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.241129+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.241644+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.241644+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.242114+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.779 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:40 smithi050 bash[21463]: audit 2024-04-26T16:23:40.242114+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.375637+0000 osd.7 (osd.7) 161 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.375637+0000 osd.7 (osd.7) 161 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.378485+0000 osd.7 (osd.7) 162 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.378485+0000 osd.7 (osd.7) 162 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.478574+0000 mgr.y (mgr.24401) 238 : cluster [DBG] pgmap v211: 784 pgs: 32 creating+peering, 1 active+clean+snaptrim, 300 unknown, 451 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 376 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.478574+0000 mgr.y (mgr.24401) 238 : cluster [DBG] pgmap v211: 784 pgs: 32 creating+peering, 1 active+clean+snaptrim, 300 unknown, 451 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 376 op/s; 319 B/s, 4 objects/s recovering 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.555361+0000 osd.0 (osd.0) 143 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:23:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.555361+0000 osd.0 (osd.0) 143 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.558178+0000 osd.0 (osd.0) 144 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:39.558178+0000 osd.0 (osd.0) 144 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207053+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207053+0000 mon.a (mon.0) 1188 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi050-73337-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207377+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207377+0000 mon.a (mon.0) 1189 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "overlaypool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207644+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207644+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207890+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.207890+0000 mon.a (mon.0) 1191 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.208124+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.208124+0000 mon.a (mon.0) 1192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.218693+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.218693+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.220950+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.220950+0000 mon.c (mon.2) 169 : audit [INF] from='client.? 172.21.15.50:0/3771902268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.221482+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.221482+0000 mon.c (mon.2) 170 : audit [INF] from='client.? 172.21.15.50:0/830244741' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.221897+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.221897+0000 mon.c (mon.2) 171 : audit [INF] from='client.? 172.21.15.50:0/954242931' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.225050+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.225050+0000 mon.c (mon.2) 172 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.225371+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.225371+0000 mon.a (mon.0) 1194 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.225816+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.225816+0000 mon.c (mon.2) 173 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.226668+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.50:0/4267792152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.226668+0000 mon.c (mon.2) 174 : audit [INF] from='client.? 172.21.15.50:0/4267792152' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.228462+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.228462+0000 mon.b (mon.1) 80 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.230715+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.230715+0000 mon.c (mon.2) 175 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.235639+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.235639+0000 mon.a (mon.0) 1195 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.235796+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.235796+0000 mon.b (mon.1) 81 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.239280+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.239280+0000 mon.a (mon.0) 1196 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.239952+0000 mon.a (mon.0) 1197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.239952+0000 mon.a (mon.0) 1197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.240548+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.240548+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.241129+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.241129+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.241644+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.241644+0000 mon.a (mon.0) 1200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.242114+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:40 smithi190 bash[23868]: audit 2024-04-26T16:23:40.242114+0000 mon.a (mon.0) 1201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:40.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:40.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:40.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:41.009 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: Running main() from gmock_main.cc 2024-04-26T16:23:41.009 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-04-26T16:23:41.009 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [----------] Global test environment set-up. 2024-04-26T16:23:41.009 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-04-26T16:23:41.009 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-04-26T16:23:41.009 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5828 ms) 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6844 ms) 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12672 ms total) 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [----------] Global test environment tear-down 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12672 ms total) 2024-04-26T16:23:41.010 INFO:tasks.workunit.client.0.smithi050.stdout: ec_io: [ PASSED ] 2 tests. 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: Running main() from gmock_main.cc 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [----------] Global test environment set-up. 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (179 ms) 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.LockShared (4 ms) 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1131 ms) 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-04-26T16:23:41.012 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1050 ms) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (8 ms) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (6 ms) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (2 ms) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [----------] 8 tests from LibRadosLock (2385 ms total) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1173 ms) 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-04-26T16:23:41.013 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (13 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1064 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1007 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (17 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (12 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (11 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (4 ms) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3301 ms total) 2024-04-26T16:23:41.014 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: 2024-04-26T16:23:41.015 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [----------] Global test environment tear-down 2024-04-26T16:23:41.015 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13037 ms total) 2024-04-26T16:23:41.015 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock: [ PASSED ] 16 tests. 2024-04-26T16:23:41.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: seed 73246 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (224 ms) 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (13 ms) 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1195 ms) 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1015 ms) 2024-04-26T16:23:41.016 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (6 ms) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (3 ms) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (10 ms) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (5 ms) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2471 ms total) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1146 ms) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (9 ms) 2024-04-26T16:23:41.017 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1135 ms) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1056 ms) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (5 ms) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (5 ms) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (6 ms) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3365 ms total) 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: 2024-04-26T16:23:41.018 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-04-26T16:23:41.019 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13033 ms total) 2024-04-26T16:23:41.019 INFO:tasks.workunit.client.0.smithi050.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-04-26T16:23:41.043 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:41.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:41.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:41.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:41.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:41.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:41.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:41.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:41.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:41.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:41.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:41.686 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:41.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:41.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.409379+0000 osd.7 (osd.7) 163 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:23:41.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.409379+0000 osd.7 (osd.7) 163 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:23:41.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.582465+0000 osd.7 (osd.7) 164 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.582465+0000 osd.7 (osd.7) 164 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.993579+0000 mon.a (mon.0) 1202 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.993579+0000 mon.a (mon.0) 1202 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.993998+0000 mon.a (mon.0) 1203 : cluster [WRN] pool 'PoolQuotaPP_smithi050-73201-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.993998+0000 mon.a (mon.0) 1203 : cluster [WRN] pool 'PoolQuotaPP_smithi050-73201-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.996569+0000 mon.a (mon.0) 1204 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.996569+0000 mon.a (mon.0) 1204 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.996595+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:40.996595+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999383+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999383+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999463+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999463+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:23:41.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999590+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:41.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999590+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:41.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999680+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:41.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999680+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:41.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999786+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:41.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999786+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999888+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999888+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999972+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:40.999972+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.000042+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.000042+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:41.005658+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: cluster 2024-04-26T16:23:41.005658+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.006389+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.006389+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.010009+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.50:0/2384347464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.010009+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.50:0/2384347464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.012117+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.50:0/1244504156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.012117+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.50:0/1244504156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.013290+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.013290+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.023664+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.023664+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.024082+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.024082+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.048965+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.048965+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.066704+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.066704+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.067075+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.067075+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.069181+0000 mon.c (mon.2) 180 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:41.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.069181+0000 mon.c (mon.2) 180 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.079492+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.079492+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.080932+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.080932+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.081706+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.081706+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.082318+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.082318+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.082703+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.082703+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.083120+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.083120+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:41.804 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.246072+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:41.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.246072+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:41.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.481711+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:41.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.481711+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:41.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.483716+0000 mon.a (mon.0) 1223 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:41.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:41 smithi190 bash[23868]: audit 2024-04-26T16:23:41.483716+0000 mon.a (mon.0) 1223 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:41.814 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:41.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:41.946 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:41.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:42.032 INFO:tasks.workunit.client.0.smithi050.stdout: pool: Running main() from gmock_main.cc 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [----------] Global test environment set-up. 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ OK ] NeoRadosPools.PoolList (1802 ms) 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2049 ms) 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (1986 ms) 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3993 ms) 2024-04-26T16:23:42.033 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2013 ms) 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (1810 ms) 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [----------] 6 tests from NeoRadosPools (13653 ms total) 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [----------] Global test environment tear-down 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [==========] 6 tests from 1 test suite ran. (13653 ms total) 2024-04-26T16:23:42.034 INFO:tasks.workunit.client.0.smithi050.stdout: pool: [ PASSED ] 6 tests. 2024-04-26T16:23:42.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.409379+0000 osd.7 (osd.7) 163 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.409379+0000 osd.7 (osd.7) 163 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.582465+0000 osd.7 (osd.7) 164 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.582465+0000 osd.7 (osd.7) 164 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.993579+0000 mon.a (mon.0) 1202 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.993579+0000 mon.a (mon.0) 1202 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.993998+0000 mon.a (mon.0) 1203 : cluster [WRN] pool 'PoolQuotaPP_smithi050-73201-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.993998+0000 mon.a (mon.0) 1203 : cluster [WRN] pool 'PoolQuotaPP_smithi050-73201-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-26T16:23:42.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.996569+0000 mon.a (mon.0) 1204 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.996569+0000 mon.a (mon.0) 1204 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.996595+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:40.996595+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999383+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999383+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999463+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999463+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999590+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999590+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999680+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999680+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999786+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:42.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999786+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999888+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999888+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999972+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:40.999972+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.000042+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.000042+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:41.005658+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: cluster 2024-04-26T16:23:41.005658+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.006389+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.006389+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.010009+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.50:0/2384347464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.010009+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.50:0/2384347464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.012117+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.50:0/1244504156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.012117+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.50:0/1244504156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.013290+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.013290+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.023664+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.023664+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.024082+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.024082+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.048965+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.048965+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.066704+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.066704+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.067075+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.067075+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.069181+0000 mon.c (mon.2) 180 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.069181+0000 mon.c (mon.2) 180 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.079492+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.079492+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.080932+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.080932+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.081706+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.081706+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.082318+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.082318+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.082703+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.082703+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.083120+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.083120+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.246072+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.246072+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.481711+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.481711+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.483716+0000 mon.a (mon.0) 1223 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[29234]: audit 2024-04-26T16:23:41.483716+0000 mon.a (mon.0) 1223 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.409379+0000 osd.7 (osd.7) 163 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.409379+0000 osd.7 (osd.7) 163 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:23:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.582465+0000 osd.7 (osd.7) 164 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.582465+0000 osd.7 (osd.7) 164 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.993579+0000 mon.a (mon.0) 1202 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.993579+0000 mon.a (mon.0) 1202 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.993998+0000 mon.a (mon.0) 1203 : cluster [WRN] pool 'PoolQuotaPP_smithi050-73201-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.993998+0000 mon.a (mon.0) 1203 : cluster [WRN] pool 'PoolQuotaPP_smithi050-73201-3' is full (reached quota's max_bytes: 4 KiB) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.996569+0000 mon.a (mon.0) 1204 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.996569+0000 mon.a (mon.0) 1204 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.996595+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:40.996595+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL) 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999383+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999383+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? 172.21.15.50:0/3241572988' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi050-73193-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999463+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999463+0000 mon.a (mon.0) 1207 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999590+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999590+0000 mon.a (mon.0) 1208 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi050-73241-10"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999680+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999680+0000 mon.a (mon.0) 1209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi050-73246-10"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999786+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999786+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi050-74008-2"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999888+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999888+0000 mon.a (mon.0) 1211 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73572-6", "mode": "writeback"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999972+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:40.999972+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.000042+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.000042+0000 mon.a (mon.0) 1213 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi050-73420-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:41.005658+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: cluster 2024-04-26T16:23:41.005658+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.006389+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.006389+0000 mon.c (mon.2) 176 : audit [INF] from='client.? 172.21.15.50:0/4139276561' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.010009+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.50:0/2384347464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.010009+0000 mon.c (mon.2) 177 : audit [INF] from='client.? 172.21.15.50:0/2384347464' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.012117+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.50:0/1244504156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.012117+0000 mon.c (mon.2) 178 : audit [INF] from='client.? 172.21.15.50:0/1244504156' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.013290+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.013290+0000 mon.a (mon.0) 1215 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.023664+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.023664+0000 mon.a (mon.0) 1216 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.024082+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.024082+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.048965+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.048965+0000 mon.c (mon.2) 179 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.066704+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.066704+0000 mon.b (mon.1) 82 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.067075+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.067075+0000 mon.a (mon.0) 1218 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.069181+0000 mon.c (mon.2) 180 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.069181+0000 mon.c (mon.2) 180 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.079492+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.079492+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.080932+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.080932+0000 mon.c (mon.2) 181 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.081706+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.081706+0000 mon.a (mon.0) 1220 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.082318+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.082318+0000 mon.a (mon.0) 1221 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.082703+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.082703+0000 mon.c (mon.2) 182 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.083120+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.083120+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.246072+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.246072+0000 mon.c (mon.2) 183 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.481711+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.481711+0000 mon.c (mon.2) 184 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.483716+0000 mon.a (mon.0) 1223 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:41 smithi050 bash[21463]: audit 2024-04-26T16:23:41.483716+0000 mon.a (mon.0) 1223 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]: dispatch 2024-04-26T16:23:42.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:42.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:42.272 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:42.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:42.397 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:42.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:42.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:42.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:42.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:42.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:42.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:42.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: cluster 2024-04-26T16:23:41.481179+0000 mgr.y (mgr.24401) 239 : cluster [DBG] pgmap v214: 711 pgs: 192 creating+peering, 519 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-04-26T16:23:42.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: cluster 2024-04-26T16:23:41.481179+0000 mgr.y (mgr.24401) 239 : cluster [DBG] pgmap v214: 711 pgs: 192 creating+peering, 519 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-04-26T16:23:42.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006402+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006402+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006588+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006588+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006764+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006764+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006909+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.006909+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.007085+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.007085+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.007315+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.007315+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.022443+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.022443+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: cluster 2024-04-26T16:23:42.024003+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: cluster 2024-04-26T16:23:42.024003+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.043020+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.043020+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.085610+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.085610+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.129703+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.129703+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.134552+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.134552+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.135386+0000 mon.a (mon.0) 1233 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.135386+0000 mon.a (mon.0) 1233 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.247013+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[29234]: audit 2024-04-26T16:23:42.247013+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: cluster 2024-04-26T16:23:41.481179+0000 mgr.y (mgr.24401) 239 : cluster [DBG] pgmap v214: 711 pgs: 192 creating+peering, 519 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: cluster 2024-04-26T16:23:41.481179+0000 mgr.y (mgr.24401) 239 : cluster [DBG] pgmap v214: 711 pgs: 192 creating+peering, 519 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006402+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006402+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006588+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006588+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006764+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006764+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006909+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.006909+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.007085+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.007085+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.007315+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.007315+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.022443+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.022443+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: cluster 2024-04-26T16:23:42.024003+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: cluster 2024-04-26T16:23:42.024003+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.043020+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.043020+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.085610+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.085610+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.129703+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.129703+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.134552+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.134552+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.135386+0000 mon.a (mon.0) 1233 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.135386+0000 mon.a (mon.0) 1233 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.247013+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:42 smithi050 bash[21463]: audit 2024-04-26T16:23:42.247013+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:42.783 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:42.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:42.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: cluster 2024-04-26T16:23:41.481179+0000 mgr.y (mgr.24401) 239 : cluster [DBG] pgmap v214: 711 pgs: 192 creating+peering, 519 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: cluster 2024-04-26T16:23:41.481179+0000 mgr.y (mgr.24401) 239 : cluster [DBG] pgmap v214: 711 pgs: 192 creating+peering, 519 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 5.5 KiB/s wr, 3 op/s 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006402+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006402+0000 mon.a (mon.0) 1224 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi050-74090-2"}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006588+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006588+0000 mon.a (mon.0) 1225 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006764+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006764+0000 mon.a (mon.0) 1226 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006909+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.006909+0000 mon.a (mon.0) 1227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.007085+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.007085+0000 mon.a (mon.0) 1228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.007315+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-26T16:23:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.007315+0000 mon.a (mon.0) 1229 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "30"}]': finished 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.022443+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.022443+0000 mon.b (mon.1) 83 : audit [INF] from='client.? 172.21.15.50:0/686778234' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: cluster 2024-04-26T16:23:42.024003+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: cluster 2024-04-26T16:23:42.024003+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.043020+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.043020+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.085610+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.085610+0000 mon.b (mon.1) 84 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.129703+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.129703+0000 mon.c (mon.2) 185 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.134552+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.134552+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.135386+0000 mon.a (mon.0) 1233 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.135386+0000 mon.a (mon.0) 1233 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.247013+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:42 smithi190 bash[23868]: audit 2024-04-26T16:23:42.247013+0000 mon.c (mon.2) 186 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:42.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:42.910 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:42.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:43.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:43.073 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:43.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:43.199 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:43.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:43.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:43.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:43.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:43.460 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:43.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:43.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:43.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:43.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:43.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:43.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:43.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:43.988 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:43.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:43.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:44.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.013048+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.013048+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.013431+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.013431+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.029415+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.029415+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: cluster 2024-04-26T16:23:43.048996+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: cluster 2024-04-26T16:23:43.048996+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.051629+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.051629+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.053481+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.50:0/1473183911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.053481+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.50:0/1473183911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.054961+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.054961+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.059797+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.013048+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.013048+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.013431+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.013431+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.029415+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.029415+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: cluster 2024-04-26T16:23:43.048996+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-04-26T16:23:44.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: cluster 2024-04-26T16:23:43.048996+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-04-26T16:23:44.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.051629+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.059797+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.064783+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.064783+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.248046+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:44.126 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: audit 2024-04-26T16:23:43.248046+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:44.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: cluster 2024-04-26T16:23:43.481846+0000 mgr.y (mgr.24401) 240 : cluster [DBG] pgmap v217: 519 pgs: 128 unknown, 391 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-04-26T16:23:44.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[29234]: cluster 2024-04-26T16:23:43.481846+0000 mgr.y (mgr.24401) 240 : cluster [DBG] pgmap v217: 519 pgs: 128 unknown, 391 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-04-26T16:23:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.051629+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.053481+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.50:0/1473183911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.053481+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.50:0/1473183911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.054961+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.054961+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.059797+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.059797+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.064783+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.064783+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.248046+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: audit 2024-04-26T16:23:43.248046+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: cluster 2024-04-26T16:23:43.481846+0000 mgr.y (mgr.24401) 240 : cluster [DBG] pgmap v217: 519 pgs: 128 unknown, 391 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-04-26T16:23:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21463]: cluster 2024-04-26T16:23:43.481846+0000 mgr.y (mgr.24401) 240 : cluster [DBG] pgmap v217: 519 pgs: 128 unknown, 391 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-04-26T16:23:44.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:44.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1259 ms) 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (6 ms) 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1265 ms total) 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-04-26T16:23:44.270 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (486 ms) 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3534 ms) 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (5 ms) 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526441529376 notify_id 657129996289 notifier_gid 25307 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-04-26T16:23:44.271 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526441529376 notify_id 657129996289 notifier_gid 25307 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (7 ms) 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442320864 notify_id 657129996292 notifier_gid 25307 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (6 ms) 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442320864 notify_id 657129996290 notifier_gid 25307 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (6 ms) 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442324512 notify_id 657129996293 notifier_gid 25307 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442324512 notify_id 657129996288 notifier_gid 25307 2024-04-26T16:23:44.272 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (6 ms) 2024-04-26T16:23:44.273 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-04-26T16:23:44.273 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: trying... 2024-04-26T16:23:44.273 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442324512 notify_id 657129996291 notifier_gid 25307 2024-04-26T16:23:44.273 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: timed out 2024-04-26T16:23:44.273 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: flushing 2024-04-26T16:23:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:44.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.013048+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:23:44.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.013048+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi050-73337-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:23:44.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.013431+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:44.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.013431+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi050-74090-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.029415+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.029415+0000 mon.b (mon.1) 85 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: cluster 2024-04-26T16:23:43.048996+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: cluster 2024-04-26T16:23:43.048996+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.051629+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.051629+0000 mon.a (mon.0) 1237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.053481+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.50:0/1473183911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.053481+0000 mon.b (mon.1) 86 : audit [INF] from='client.? 172.21.15.50:0/1473183911' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.054961+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.054961+0000 mon.a (mon.0) 1238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.059797+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.059797+0000 mon.b (mon.1) 87 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.064783+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.064783+0000 mon.a (mon.0) 1239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.248046+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: audit 2024-04-26T16:23:43.248046+0000 mon.c (mon.2) 187 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: cluster 2024-04-26T16:23:43.481846+0000 mgr.y (mgr.24401) 240 : cluster [DBG] pgmap v217: 519 pgs: 128 unknown, 391 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-04-26T16:23:44.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:44 smithi190 bash[23868]: cluster 2024-04-26T16:23:43.481846+0000 mgr.y (mgr.24401) 240 : cluster [DBG] pgmap v217: 519 pgs: 128 unknown, 391 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 3 op/s 2024-04-26T16:23:44.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:44.404 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:44.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:44.532 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:44.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:44.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:44.680 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:23:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:23:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:23:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:44.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:44.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:44.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:44.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:44.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:44.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:45.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:45.138 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:45.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:45.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:45.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:45.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:45.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.025737+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.025737+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.025834+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.025834+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.025960+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.025960+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: cluster 2024-04-26T16:23:44.044104+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-04-26T16:23:45.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: cluster 2024-04-26T16:23:44.044104+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.055399+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.50:0/434261661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.055399+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.50:0/434261661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.055763+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.055763+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.056144+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.056144+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.066006+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.50:0/3763106952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.066006+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.50:0/3763106952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.073983+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.073983+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.249450+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:45.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:45 smithi190 bash[23868]: audit 2024-04-26T16:23:44.249450+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.025737+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.025737+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.025834+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.025834+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.025960+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.025960+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: cluster 2024-04-26T16:23:44.044104+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-04-26T16:23:45.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: cluster 2024-04-26T16:23:44.044104+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.055399+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.50:0/434261661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.055399+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.50:0/434261661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.055763+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.055763+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.056144+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.056144+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.066006+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.50:0/3763106952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.066006+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.50:0/3763106952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.073983+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.073983+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.249450+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:45.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[29234]: audit 2024-04-26T16:23:44.249450+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.025737+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.025737+0000 mon.a (mon.0) 1240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi050-73536-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.025834+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.025834+0000 mon.a (mon.0) 1241 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi050-73193-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.025960+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.025960+0000 mon.a (mon.0) 1242 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: cluster 2024-04-26T16:23:44.044104+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-04-26T16:23:45.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: cluster 2024-04-26T16:23:44.044104+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.055399+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.50:0/434261661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.055399+0000 mon.c (mon.2) 188 : audit [INF] from='client.? 172.21.15.50:0/434261661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.055763+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.055763+0000 mon.a (mon.0) 1244 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.056144+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.056144+0000 mon.a (mon.0) 1245 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.066006+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.50:0/3763106952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.066006+0000 mon.b (mon.1) 88 : audit [INF] from='client.? 172.21.15.50:0/3763106952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.073983+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.073983+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.249450+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:45.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:45 smithi050 bash[21463]: audit 2024-04-26T16:23:44.249450+0000 mon.c (mon.2) 189 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:45.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:45.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:45.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:45.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:45.553 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:45.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:45.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:45.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:45.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:45.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:45.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:45.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:45.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:45.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:46.035 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:46.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031567+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031567+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031673+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031673+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031762+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031567+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031567+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031673+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031673+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031762+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031762+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031887+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.031887+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.035952+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.035952+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.061246+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.061246+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.089816+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.089816+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.253357+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.253357+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.482623+0000 mgr.y (mgr.24401) 241 : cluster [DBG] pgmap v220: 726 pgs: 336 unknown, 390 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.482623+0000 mgr.y (mgr.24401) 241 : cluster [DBG] pgmap v220: 726 pgs: 336 unknown, 390 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.483065+0000 mon.c (mon.2) 191 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.483065+0000 mon.c (mon.2) 191 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.486256+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:45.486256+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.633299+0000 osd.0 (osd.0) 145 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.633299+0000 osd.0 (osd.0) 145 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.635261+0000 osd.0 (osd.0) 146 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.635261+0000 osd.0 (osd.0) 146 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.998030+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.998030+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.001826+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.001826+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.001910+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.001910+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.015899+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.015899+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.032343+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.032343+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.033954+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.50:0/2203909848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.033954+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.50:0/2203909848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.077130+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[29234]: audit 2024-04-26T16:23:46.077130+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:46.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:46.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:46.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031567+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:46.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031567+0000 mon.a (mon.0) 1247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi050-74090-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031673+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031673+0000 mon.a (mon.0) 1248 : audit [INF] from='client.? 172.21.15.50:0/3301772666' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi050-73432-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031762+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031762+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031887+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.031887+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.035952+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.035952+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.061246+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.061246+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.089816+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.089816+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.253357+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.253357+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.482623+0000 mgr.y (mgr.24401) 241 : cluster [DBG] pgmap v220: 726 pgs: 336 unknown, 390 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.482623+0000 mgr.y (mgr.24401) 241 : cluster [DBG] pgmap v220: 726 pgs: 336 unknown, 390 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.483065+0000 mon.c (mon.2) 191 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.483065+0000 mon.c (mon.2) 191 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.486256+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:45.486256+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.633299+0000 osd.0 (osd.0) 145 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.633299+0000 osd.0 (osd.0) 145 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.635261+0000 osd.0 (osd.0) 146 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.635261+0000 osd.0 (osd.0) 146 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.998030+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.998030+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.001826+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.001826+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.001910+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.001910+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.015899+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.015899+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.032343+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.032343+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.033954+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.50:0/2203909848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.033954+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.50:0/2203909848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.077130+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[23868]: audit 2024-04-26T16:23:46.077130+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:46.366 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:46.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031762+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031887+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.031887+0000 mon.a (mon.0) 1250 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.035952+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.035952+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.061246+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.061246+0000 mon.b (mon.1) 89 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.089816+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.089816+0000 mon.a (mon.0) 1252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.253357+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.253357+0000 mon.c (mon.2) 190 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.482623+0000 mgr.y (mgr.24401) 241 : cluster [DBG] pgmap v220: 726 pgs: 336 unknown, 390 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.482623+0000 mgr.y (mgr.24401) 241 : cluster [DBG] pgmap v220: 726 pgs: 336 unknown, 390 active+clean; 145 MiB data, 792 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.483065+0000 mon.c (mon.2) 191 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.483065+0000 mon.c (mon.2) 191 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.486256+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:45.486256+0000 mon.a (mon.0) 1253 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]: dispatch 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.633299+0000 osd.0 (osd.0) 145 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.633299+0000 osd.0 (osd.0) 145 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.635261+0000 osd.0 (osd.0) 146 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.635261+0000 osd.0 (osd.0) 146 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:23:46.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.998030+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.998030+0000 mon.a (mon.0) 1254 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.001826+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.001826+0000 mon.a (mon.0) 1255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.001910+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.001910+0000 mon.a (mon.0) 1256 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "29"}]': finished 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.015899+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.015899+0000 mon.a (mon.0) 1257 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.032343+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.032343+0000 mon.c (mon.2) 192 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.033954+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.50:0/2203909848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.033954+0000 mon.b (mon.1) 90 : audit [INF] from='client.? 172.21.15.50:0/2203909848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.077130+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:46 smithi050 bash[21463]: audit 2024-04-26T16:23:46.077130+0000 mon.a (mon.0) 1258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:46.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:46.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:46.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:46.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:46.654 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:46.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:46.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:46.776 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:46.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:46.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:46.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:47.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:47.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:47.069 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:23:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:23:47.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:47.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:47.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:47.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:47.348 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:47.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:47.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:47.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.286745+0000 osd.6 (osd.6) 91 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.286745+0000 osd.6 (osd.6) 91 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.289476+0000 osd.6 (osd.6) 92 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.289476+0000 osd.6 (osd.6) 92 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.629535+0000 osd.3 (osd.3) 159 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.629535+0000 osd.3 (osd.3) 159 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.631718+0000 osd.3 (osd.3) 160 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:45.631718+0000 osd.3 (osd.3) 160 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:23:47.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.080647+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.080647+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.184727+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.184727+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.212511+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.212511+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.256866+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:47.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.256866+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.619618+0000 mgr.y (mgr.24401) 242 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:46.619618+0000 mgr.y (mgr.24401) 242 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.641040+0000 osd.0 (osd.0) 147 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.641040+0000 osd.0 (osd.0) 147 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:47.004745+0000 mon.a (mon.0) 1261 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:47.004745+0000 mon.a (mon.0) 1261 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-26T16:23:47.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.008265+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.008265+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.008354+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.008354+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.008461+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.008461+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:47.013801+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: cluster 2024-04-26T16:23:47.013801+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.022353+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.022353+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.022799+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.022799+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.040680+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.040680+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.046287+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.50:0/2128397574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.046287+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.50:0/2128397574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.103533+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.103533+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.103842+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[29234]: audit 2024-04-26T16:23:47.103842+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.286745+0000 osd.6 (osd.6) 91 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:23:47.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.286745+0000 osd.6 (osd.6) 91 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.289476+0000 osd.6 (osd.6) 92 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.289476+0000 osd.6 (osd.6) 92 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.629535+0000 osd.3 (osd.3) 159 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.629535+0000 osd.3 (osd.3) 159 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.631718+0000 osd.3 (osd.3) 160 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:45.631718+0000 osd.3 (osd.3) 160 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.080647+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.080647+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.184727+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.184727+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.212511+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.212511+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.256866+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:47.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.256866+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.619618+0000 mgr.y (mgr.24401) 242 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:46.619618+0000 mgr.y (mgr.24401) 242 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.641040+0000 osd.0 (osd.0) 147 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.641040+0000 osd.0 (osd.0) 147 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:47.004745+0000 mon.a (mon.0) 1261 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:47.004745+0000 mon.a (mon.0) 1261 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.008265+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.008265+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.008354+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.008354+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.008461+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.008461+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:47.013801+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-04-26T16:23:47.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: cluster 2024-04-26T16:23:47.013801+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.022353+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.022353+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.022799+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.022799+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.040680+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.040680+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.046287+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.50:0/2128397574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.046287+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.50:0/2128397574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.103533+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.103533+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.522 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.103842+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.522 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:47 smithi050 bash[21463]: audit 2024-04-26T16:23:47.103842+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.286745+0000 osd.6 (osd.6) 91 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.286745+0000 osd.6 (osd.6) 91 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.289476+0000 osd.6 (osd.6) 92 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.289476+0000 osd.6 (osd.6) 92 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.629535+0000 osd.3 (osd.3) 159 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.629535+0000 osd.3 (osd.3) 159 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.631718+0000 osd.3 (osd.3) 160 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:45.631718+0000 osd.3 (osd.3) 160 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.080647+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.080647+0000 mon.a (mon.0) 1259 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.184727+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.184727+0000 mon.b (mon.1) 91 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.212511+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.212511+0000 mon.a (mon.0) 1260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.256866+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.256866+0000 mon.c (mon.2) 193 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.619618+0000 mgr.y (mgr.24401) 242 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:46.619618+0000 mgr.y (mgr.24401) 242 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.641040+0000 osd.0 (osd.0) 147 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.641040+0000 osd.0 (osd.0) 147 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:47.004745+0000 mon.a (mon.0) 1261 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:47.004745+0000 mon.a (mon.0) 1261 : cluster [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.008265+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:47.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.008265+0000 mon.a (mon.0) 1262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi050-73193-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.008354+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.008354+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.008461+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.008461+0000 mon.a (mon.0) 1264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:47.013801+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: cluster 2024-04-26T16:23:47.013801+0000 mon.a (mon.0) 1265 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.022353+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.022353+0000 mon.c (mon.2) 194 : audit [INF] from='client.? 172.21.15.50:0/3513692793' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.022799+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.022799+0000 mon.a (mon.0) 1266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]: dispatch 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.040680+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.040680+0000 mon.b (mon.1) 92 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.046287+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.50:0/2128397574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.046287+0000 mon.b (mon.1) 93 : audit [INF] from='client.? 172.21.15.50:0/2128397574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.103533+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.103533+0000 mon.a (mon.0) 1267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.103842+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:47 smithi190 bash[23868]: audit 2024-04-26T16:23:47.103842+0000 mon.a (mon.0) 1268 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:47.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:47.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:47.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:47.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:47.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:47.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:47.885 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:47.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:47.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watc api_watch_notify: Running main() from gmock_main.cc 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify_test_cb 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (365 ms) 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94132027830288 err -107 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-26T16:23:48.042 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (19 ms) 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94132027841824 err -107 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1013 ms) 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 614180323330 cookie 94132027862032 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (6 ms) 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 614180323329 cookie 94132027862032 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (7 ms) 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 614180323330 cookie 94132027862032 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (5 ms) 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-04-26T16:23:48.043 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 614180323331 cookie 94132027862032 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 614180323331 cookie 94132027880784 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (6 ms) 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 614180323328 cookie 94132027880784 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 618475290625 cookie 94132027880784 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3009 ms) 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94132027880784 err -107 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_cb from 14877 notify_id 648540061699 cookie 94132027880784 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5014 ms) 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94132027875616 err -107 2024-04-26T16:23:48.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1010 ms) 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10454 ms total) 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: watch_notify_test_cb 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1150 ms) 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1150 ms total) 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-04-26T16:23:48.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (19918 ms total) 2024-04-26T16:23:48.046 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-04-26T16:23:48.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:48.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:48.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:48.312 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:48.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:48.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.319696+0000 osd.6 (osd.6) 93 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.319696+0000 osd.6 (osd.6) 93 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.322087+0000 osd.6 (osd.6) 94 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.322087+0000 osd.6 (osd.6) 94 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.826135+0000 osd.0 (osd.0) 148 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.826135+0000 osd.0 (osd.0) 148 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.946719+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.946719+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.946796+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:48.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:46.946796+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:47.258889+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:47.258889+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:47.484176+0000 mgr.y (mgr.24401) 243 : cluster [DBG] pgmap v223: 654 pgs: 32 creating+peering, 2 peering, 160 unknown, 460 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:47.484176+0000 mgr.y (mgr.24401) 243 : cluster [DBG] pgmap v223: 654 pgs: 32 creating+peering, 2 peering, 160 unknown, 460 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:48.018370+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:48.018370+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:48.018432+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:48.018432+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:48.018483+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: audit 2024-04-26T16:23:48.018483+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:48.022760+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-04-26T16:23:48.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[21463]: cluster 2024-04-26T16:23:48.022760+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.319696+0000 osd.6 (osd.6) 93 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.319696+0000 osd.6 (osd.6) 93 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.322087+0000 osd.6 (osd.6) 94 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.322087+0000 osd.6 (osd.6) 94 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.826135+0000 osd.0 (osd.0) 148 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.826135+0000 osd.0 (osd.0) 148 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.946719+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.946719+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.946796+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:46.946796+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:47.258889+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:47.258889+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:47.484176+0000 mgr.y (mgr.24401) 243 : cluster [DBG] pgmap v223: 654 pgs: 32 creating+peering, 2 peering, 160 unknown, 460 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:23:48.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:47.484176+0000 mgr.y (mgr.24401) 243 : cluster [DBG] pgmap v223: 654 pgs: 32 creating+peering, 2 peering, 160 unknown, 460 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:48.018370+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:48.018370+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:48.018432+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:48.018432+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:48.018483+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: audit 2024-04-26T16:23:48.018483+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:48.022760+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-04-26T16:23:48.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:48 smithi050 bash[29234]: cluster 2024-04-26T16:23:48.022760+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-04-26T16:23:48.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:48.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:48.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.319696+0000 osd.6 (osd.6) 93 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.319696+0000 osd.6 (osd.6) 93 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.322087+0000 osd.6 (osd.6) 94 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.322087+0000 osd.6 (osd.6) 94 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.826135+0000 osd.0 (osd.0) 148 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.826135+0000 osd.0 (osd.0) 148 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.946719+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:48.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.946719+0000 osd.0 (osd.0) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.946796+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:46.946796+0000 osd.0 (osd.0) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:47.258889+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:47.258889+0000 mon.c (mon.2) 195 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:47.484176+0000 mgr.y (mgr.24401) 243 : cluster [DBG] pgmap v223: 654 pgs: 32 creating+peering, 2 peering, 160 unknown, 460 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:47.484176+0000 mgr.y (mgr.24401) 243 : cluster [DBG] pgmap v223: 654 pgs: 32 creating+peering, 2 peering, 160 unknown, 460 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:48.018370+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:48.018370+0000 mon.a (mon.0) 1269 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi050-73536-12"}]': finished 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:48.018432+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:48.018432+0000 mon.a (mon.0) 1270 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:48.018483+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: audit 2024-04-26T16:23:48.018483+0000 mon.a (mon.0) 1271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73664-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:48.022760+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-04-26T16:23:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:48 smithi190 bash[23868]: cluster 2024-04-26T16:23:48.022760+0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2024-04-26T16:23:48.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:48.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:48.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:48.813 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:48.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:48.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:49.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:49.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:49.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:49.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:49.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:49.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:49.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:49.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:49.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:49.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:49.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:49.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:48.235412+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:48.235412+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:48.254086+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:48.254086+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:48.263266+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:48.263266+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.029302+0000 mon.a (mon.0) 1274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.029302+0000 mon.a (mon.0) 1274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: cluster 2024-04-26T16:23:49.035425+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: cluster 2024-04-26T16:23:49.035425+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.036712+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.036712+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.038734+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.50:0/3238677290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.038734+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.50:0/3238677290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.039020+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.50:0/1995497020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.039020+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.50:0/1995497020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.040978+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.040978+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.041355+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.041355+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.046280+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.50:0/2514673820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.046280+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.50:0/2514673820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.053339+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.053339+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.054813+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:49 smithi190 bash[23868]: audit 2024-04-26T16:23:49.054813+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:48.235412+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:48.235412+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:48.254086+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:48.254086+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:48.263266+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:48.263266+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.029302+0000 mon.a (mon.0) 1274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.029302+0000 mon.a (mon.0) 1274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:49.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: cluster 2024-04-26T16:23:49.035425+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: cluster 2024-04-26T16:23:49.035425+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.036712+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.036712+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.038734+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.50:0/3238677290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.038734+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.50:0/3238677290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.039020+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.50:0/1995497020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.039020+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.50:0/1995497020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.040978+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.040978+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.041355+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.041355+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.046280+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.50:0/2514673820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.046280+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.50:0/2514673820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.053339+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.053339+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.054813+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[21463]: audit 2024-04-26T16:23:49.054813+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:48.235412+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:48.235412+0000 mon.b (mon.1) 94 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:48.254086+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:48.254086+0000 mon.a (mon.0) 1273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:48.263266+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:48.263266+0000 mon.c (mon.2) 196 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.029302+0000 mon.a (mon.0) 1274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.029302+0000 mon.a (mon.0) 1274 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: cluster 2024-04-26T16:23:49.035425+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-04-26T16:23:49.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: cluster 2024-04-26T16:23:49.035425+0000 mon.a (mon.0) 1275 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.036712+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.036712+0000 mon.b (mon.1) 95 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.038734+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.50:0/3238677290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.038734+0000 mon.b (mon.1) 96 : audit [INF] from='client.? 172.21.15.50:0/3238677290' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.039020+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.50:0/1995497020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.039020+0000 mon.b (mon.1) 97 : audit [INF] from='client.? 172.21.15.50:0/1995497020' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.040978+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.040978+0000 mon.a (mon.0) 1276 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.041355+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.041355+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.046280+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.50:0/2514673820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.046280+0000 mon.c (mon.2) 197 : audit [INF] from='client.? 172.21.15.50:0/2514673820' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.053339+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.053339+0000 mon.a (mon.0) 1278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.054813+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:49 smithi050 bash[29234]: audit 2024-04-26T16:23:49.054813+0000 mon.a (mon.0) 1279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:49.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:49.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:49.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:50.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:50.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:50.225 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:50.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:50.360 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:50.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:50.485 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:50.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:50.607 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:49.388481+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:49.388481+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: cluster 2024-04-26T16:23:49.484812+0000 mgr.y (mgr.24401) 244 : cluster [DBG] pgmap v226: 653 pgs: 1 peering, 224 unknown, 428 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: cluster 2024-04-26T16:23:49.484812+0000 mgr.y (mgr.24401) 244 : cluster [DBG] pgmap v226: 653 pgs: 1 peering, 224 unknown, 428 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:49.485155+0000 mon.c (mon.2) 199 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:49.485155+0000 mon.c (mon.2) 199 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:49.489823+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:49.489823+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033480+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033480+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033558+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033558+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033621+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033621+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033677+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033677+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033740+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: audit 2024-04-26T16:23:50.033740+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: cluster 2024-04-26T16:23:50.038092+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[29234]: cluster 2024-04-26T16:23:50.038092+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:49.388481+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:50.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:49.388481+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: cluster 2024-04-26T16:23:49.484812+0000 mgr.y (mgr.24401) 244 : cluster [DBG] pgmap v226: 653 pgs: 1 peering, 224 unknown, 428 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: cluster 2024-04-26T16:23:49.484812+0000 mgr.y (mgr.24401) 244 : cluster [DBG] pgmap v226: 653 pgs: 1 peering, 224 unknown, 428 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:49.485155+0000 mon.c (mon.2) 199 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:49.485155+0000 mon.c (mon.2) 199 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:49.489823+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:49.489823+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033480+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033480+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033558+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033558+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033621+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033621+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033677+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033677+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033740+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-26T16:23:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: audit 2024-04-26T16:23:50.033740+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-26T16:23:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: cluster 2024-04-26T16:23:50.038092+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-04-26T16:23:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:50 smithi050 bash[21463]: cluster 2024-04-26T16:23:50.038092+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-04-26T16:23:50.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:50.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:50.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:49.388481+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:50.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:49.388481+0000 mon.c (mon.2) 198 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:50.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: cluster 2024-04-26T16:23:49.484812+0000 mgr.y (mgr.24401) 244 : cluster [DBG] pgmap v226: 653 pgs: 1 peering, 224 unknown, 428 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:23:50.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: cluster 2024-04-26T16:23:49.484812+0000 mgr.y (mgr.24401) 244 : cluster [DBG] pgmap v226: 653 pgs: 1 peering, 224 unknown, 428 active+clean; 145 MiB data, 785 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:23:50.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:49.485155+0000 mon.c (mon.2) 199 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:49.485155+0000 mon.c (mon.2) 199 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:49.489823+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:49.489823+0000 mon.a (mon.0) 1280 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]: dispatch 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033480+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033480+0000 mon.a (mon.0) 1281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-7"}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033558+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033558+0000 mon.a (mon.0) 1282 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033621+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033621+0000 mon.a (mon.0) 1283 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi050-73193-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033677+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033677+0000 mon.a (mon.0) 1284 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033740+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: audit 2024-04-26T16:23:50.033740+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "28"}]': finished 2024-04-26T16:23:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: cluster 2024-04-26T16:23:50.038092+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-04-26T16:23:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:50 smithi190 bash[23868]: cluster 2024-04-26T16:23:50.038092+0000 mon.a (mon.0) 1286 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2024-04-26T16:23:50.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:50.908 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:50.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:51.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:51.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:51.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:51.195 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:51.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:51.318 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:51.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:51.438 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:51.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: audit 2024-04-26T16:23:50.396934+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:51.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: audit 2024-04-26T16:23:50.396934+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: cluster 2024-04-26T16:23:51.001240+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: cluster 2024-04-26T16:23:51.001240+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: cluster 2024-04-26T16:23:51.009232+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: cluster 2024-04-26T16:23:51.009232+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: audit 2024-04-26T16:23:51.029104+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: audit 2024-04-26T16:23:51.029104+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: audit 2024-04-26T16:23:51.030401+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:51 smithi190 bash[23868]: audit 2024-04-26T16:23:51.030401+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:51.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:51.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:51.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:51.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: audit 2024-04-26T16:23:50.396934+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:51.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: audit 2024-04-26T16:23:50.396934+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:51.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: cluster 2024-04-26T16:23:51.001240+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:51.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: cluster 2024-04-26T16:23:51.001240+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: cluster 2024-04-26T16:23:51.009232+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: cluster 2024-04-26T16:23:51.009232+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: audit 2024-04-26T16:23:51.029104+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: audit 2024-04-26T16:23:51.029104+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: audit 2024-04-26T16:23:51.030401+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[21463]: audit 2024-04-26T16:23:51.030401+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: audit 2024-04-26T16:23:50.396934+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: audit 2024-04-26T16:23:50.396934+0000 mon.c (mon.2) 200 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: cluster 2024-04-26T16:23:51.001240+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: cluster 2024-04-26T16:23:51.001240+0000 mon.a (mon.0) 1287 : cluster [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:51.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: cluster 2024-04-26T16:23:51.009232+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-04-26T16:23:51.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: cluster 2024-04-26T16:23:51.009232+0000 mon.a (mon.0) 1288 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2024-04-26T16:23:51.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: audit 2024-04-26T16:23:51.029104+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: audit 2024-04-26T16:23:51.029104+0000 mon.a (mon.0) 1289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: audit 2024-04-26T16:23:51.030401+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.714 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:51 smithi050 bash[29234]: audit 2024-04-26T16:23:51.030401+0000 mon.b (mon.1) 98 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:51.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:51.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:51.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:51.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:52.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:52.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:52.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:52.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:52.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:52.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:52.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:52.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:52.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:52.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:52.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:52.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:52.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:52.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:51.401587+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:52.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:51.401587+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:52.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: cluster 2024-04-26T16:23:51.487548+0000 mgr.y (mgr.24401) 245 : cluster [DBG] pgmap v229: 645 pgs: 64 creating+peering, 2 clean+premerge+peered, 579 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-04-26T16:23:52.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: cluster 2024-04-26T16:23:51.487548+0000 mgr.y (mgr.24401) 245 : cluster [DBG] pgmap v229: 645 pgs: 64 creating+peering, 2 clean+premerge+peered, 579 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.008469+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.008469+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: cluster 2024-04-26T16:23:52.013236+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: cluster 2024-04-26T16:23:52.013236+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.040711+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.040711+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.041904+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.50:0/3766472482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.041904+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.50:0/3766472482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.042550+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.042550+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.043233+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.043233+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.043927+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.50:0/213029700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.043927+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.50:0/213029700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.048513+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.048513+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.048982+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.048982+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.049353+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:52 smithi190 bash[23868]: audit 2024-04-26T16:23:52.049353+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:51.401587+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:51.401587+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: cluster 2024-04-26T16:23:51.487548+0000 mgr.y (mgr.24401) 245 : cluster [DBG] pgmap v229: 645 pgs: 64 creating+peering, 2 clean+premerge+peered, 579 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: cluster 2024-04-26T16:23:51.487548+0000 mgr.y (mgr.24401) 245 : cluster [DBG] pgmap v229: 645 pgs: 64 creating+peering, 2 clean+premerge+peered, 579 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.008469+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.008469+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: cluster 2024-04-26T16:23:52.013236+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: cluster 2024-04-26T16:23:52.013236+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.040711+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.040711+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.041904+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.50:0/3766472482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.041904+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.50:0/3766472482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.042550+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.042550+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.043233+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.043233+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.043927+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.50:0/213029700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.043927+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.50:0/213029700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.048513+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.048513+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.048982+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.048982+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.049353+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[21463]: audit 2024-04-26T16:23:52.049353+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:51.401587+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:51.401587+0000 mon.c (mon.2) 201 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: cluster 2024-04-26T16:23:51.487548+0000 mgr.y (mgr.24401) 245 : cluster [DBG] pgmap v229: 645 pgs: 64 creating+peering, 2 clean+premerge+peered, 579 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: cluster 2024-04-26T16:23:51.487548+0000 mgr.y (mgr.24401) 245 : cluster [DBG] pgmap v229: 645 pgs: 64 creating+peering, 2 clean+premerge+peered, 579 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.7 KiB/s wr, 12 op/s 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.008469+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.008469+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: cluster 2024-04-26T16:23:52.013236+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: cluster 2024-04-26T16:23:52.013236+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.040711+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.040711+0000 mon.a (mon.0) 1292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.041904+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.50:0/3766472482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.041904+0000 mon.b (mon.1) 99 : audit [INF] from='client.? 172.21.15.50:0/3766472482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.042550+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.042550+0000 mon.b (mon.1) 100 : audit [INF] from='client.? 172.21.15.50:0/3219310439' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.043233+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.043233+0000 mon.b (mon.1) 101 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.043927+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.50:0/213029700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.043927+0000 mon.b (mon.1) 102 : audit [INF] from='client.? 172.21.15.50:0/213029700' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.048513+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.048513+0000 mon.a (mon.0) 1293 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.048982+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.048982+0000 mon.a (mon.0) 1294 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.049353+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:52 smithi050 bash[29234]: audit 2024-04-26T16:23:52.049353+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:52.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:52.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:53.054 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: Running main() from gmock_main.cc 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [----------] Global test environment set-up. 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6800 ms) 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6864 ms) 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (11012 ms) 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [----------] 3 tests from NeoradosECList (24676 ms total) 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [----------] Global test environment tear-down 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (24677 ms total) 2024-04-26T16:23:53.055 INFO:tasks.workunit.client.0.smithi050.stdout: ec_list: [ PASSED ] 3 tests. 2024-04-26T16:23:53.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:53.108 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:53.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:53.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:53.231 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:53.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:53.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:53.353 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:53.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:53.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:53.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:53.480 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:53.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:53.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:53.617 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:53.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:53.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:53.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:53.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:52.407793+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:52.407793+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019563+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019563+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019671+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019671+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019742+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019742+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019810+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.019810+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.024106+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-04-26T16:23:53.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.024106+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.169786+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.169786+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.171596+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.171596+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.313082+0000 mon.c (mon.2) 203 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.313082+0000 mon.c (mon.2) 203 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.314634+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[29234]: audit 2024-04-26T16:23:53.314634+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:52.407793+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:52.407793+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019563+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019563+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019671+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019671+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019742+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019742+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019810+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.019810+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.024106+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.024106+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-04-26T16:23:53.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.169786+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.169786+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.171596+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.171596+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.313082+0000 mon.c (mon.2) 203 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.313082+0000 mon.c (mon.2) 203 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.314634+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:53 smithi050 bash[21463]: audit 2024-04-26T16:23:53.314634+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:52.407793+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:52.407793+0000 mon.c (mon.2) 202 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019563+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019563+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi050-73193-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019671+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019671+0000 mon.a (mon.0) 1297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi050-74090-3"}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019742+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019742+0000 mon.a (mon.0) 1298 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019810+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.019810+0000 mon.a (mon.0) 1299 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi050-73201-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.024106+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.024106+0000 mon.a (mon.0) 1300 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.169786+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.169786+0000 mon.b (mon.1) 103 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.171596+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.171596+0000 mon.a (mon.0) 1301 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.313082+0000 mon.c (mon.2) 203 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.313082+0000 mon.c (mon.2) 203 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.314634+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:53 smithi190 bash[23868]: audit 2024-04-26T16:23:53.314634+0000 mon.a (mon.0) 1302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]: dispatch 2024-04-26T16:23:53.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:54.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:54.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:54.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:54.261 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:54.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:54.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:54.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:54.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:54.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:54.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:53.425607+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:54.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:53.425607+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:54.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.488803+0000 mgr.y (mgr.24401) 246 : cluster [DBG] pgmap v232: 644 pgs: 160 unknown, 1 clean+premerge+peered, 483 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:23:54.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.488803+0000 mgr.y (mgr.24401) 246 : cluster [DBG] pgmap v232: 644 pgs: 160 unknown, 1 clean+premerge+peered, 483 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:23:54.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:53.489382+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:53.489382+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:53.494701+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:53.494701+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.883408+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.883408+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.883486+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:53.883486+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.024070+0000 mon.a (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.024070+0000 mon.a (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.024163+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]': finished 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.024163+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]': finished 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.024316+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.024316+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.040899+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.040899+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:54.043710+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: cluster 2024-04-26T16:23:54.043710+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.055024+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.055024+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.059184+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.059184+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.062815+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.062815+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.063100+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:54.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:54 smithi190 bash[23868]: audit 2024-04-26T16:23:54.063100+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:54.544 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:54.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:54.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:53.425607+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:53.425607+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.488803+0000 mgr.y (mgr.24401) 246 : cluster [DBG] pgmap v232: 644 pgs: 160 unknown, 1 clean+premerge+peered, 483 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.488803+0000 mgr.y (mgr.24401) 246 : cluster [DBG] pgmap v232: 644 pgs: 160 unknown, 1 clean+premerge+peered, 483 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:53.489382+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:53.489382+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:53.494701+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.651 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:53.494701+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.883408+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.883408+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.883486+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:53.883486+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.024070+0000 mon.a (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.024070+0000 mon.a (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.024163+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]': finished 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.024163+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]': finished 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.024316+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.024316+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.040899+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.040899+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.652 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:54.043710+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: cluster 2024-04-26T16:23:54.043710+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.055024+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.055024+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.059184+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.059184+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.062815+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.062815+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.063100+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21463]: audit 2024-04-26T16:23:54.063100+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:54.653 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:23:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:53.425607+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:53.425607+0000 mon.c (mon.2) 204 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.488803+0000 mgr.y (mgr.24401) 246 : cluster [DBG] pgmap v232: 644 pgs: 160 unknown, 1 clean+premerge+peered, 483 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.488803+0000 mgr.y (mgr.24401) 246 : cluster [DBG] pgmap v232: 644 pgs: 160 unknown, 1 clean+premerge+peered, 483 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.7 KiB/s wr, 11 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:53.489382+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:53.489382+0000 mon.c (mon.2) 205 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:53.494701+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:53.494701+0000 mon.a (mon.0) 1303 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]: dispatch 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.883408+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.883408+0000 osd.2 (osd.2) 107 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:23:54.654 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.883486+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:53.883486+0000 osd.2 (osd.2) 108 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.024070+0000 mon.a (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.024070+0000 mon.a (mon.0) 1304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.024163+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]': finished 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.024163+0000 mon.a (mon.0) 1305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4"}]': finished 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.024316+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.024316+0000 mon.a (mon.0) 1306 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "27"}]': finished 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.040899+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.040899+0000 mon.b (mon.1) 104 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:54.043710+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: cluster 2024-04-26T16:23:54.043710+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.055024+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.055024+0000 mon.a (mon.0) 1308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.059184+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.059184+0000 mon.c (mon.2) 206 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.062815+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.062815+0000 mon.a (mon.0) 1309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]: dispatch 2024-04-26T16:23:54.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.063100+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:54.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:54 smithi050 bash[29234]: audit 2024-04-26T16:23:54.063100+0000 mon.a (mon.0) 1310 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:54.667 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:54.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:54.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:54.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:54.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:55.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:55.113 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:55.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:55.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:55.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:55.358 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:55.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:55.480 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:55.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:55.645 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:55.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:55.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:54.427748+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:54.427748+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: cluster 2024-04-26T16:23:55.025979+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: cluster 2024-04-26T16:23:55.025979+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.028488+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:55.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.028488+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.028563+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.028563+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.028640+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.028640+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: cluster 2024-04-26T16:23:55.032566+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: cluster 2024-04-26T16:23:55.032566+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.034617+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.034617+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.035724+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.035724+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.036562+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.036562+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.760 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.037941+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.037941+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.040149+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.50:0/3876389472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.040149+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.50:0/3876389472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.040974+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.040974+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.063797+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.063797+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.065555+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.50:0/2853974335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.065555+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.50:0/2853974335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.132808+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.132808+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.134244+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.134244+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.138801+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.138801+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.139886+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.139886+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.140642+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.140642+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.141874+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.141874+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.428884+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:55 smithi190 bash[23868]: audit 2024-04-26T16:23:55.428884+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.771 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:55.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:54.427748+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:54.427748+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: cluster 2024-04-26T16:23:55.025979+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: cluster 2024-04-26T16:23:55.025979+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.028488+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.028488+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.028563+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:55.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.028563+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.028640+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.028640+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: cluster 2024-04-26T16:23:55.032566+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: cluster 2024-04-26T16:23:55.032566+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.034617+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.034617+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.035724+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.035724+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.036562+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.036562+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.037941+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.037941+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.040149+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.50:0/3876389472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.040149+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.50:0/3876389472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.040974+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.040974+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.063797+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.063797+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.065555+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.50:0/2853974335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.065555+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.50:0/2853974335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.132808+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.132808+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.134244+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.134244+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.138801+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.138801+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.139886+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.139886+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.140642+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.140642+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.141874+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.141874+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.428884+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[29234]: audit 2024-04-26T16:23:55.428884+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:54.427748+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:54.427748+0000 mon.c (mon.2) 207 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: cluster 2024-04-26T16:23:55.025979+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: cluster 2024-04-26T16:23:55.025979+0000 mon.a (mon.0) 1311 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.028488+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.028488+0000 mon.a (mon.0) 1312 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.028563+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.028563+0000 mon.a (mon.0) 1313 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi050-73572-4", "tierpool": "test-rados-api-smithi050-73572-6"}]': finished 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.028640+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.028640+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.50:0/156308417' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: cluster 2024-04-26T16:23:55.032566+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: cluster 2024-04-26T16:23:55.032566+0000 mon.a (mon.0) 1315 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.034617+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.034617+0000 mon.b (mon.1) 105 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.035724+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.035724+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.036562+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.036562+0000 mon.c (mon.2) 208 : audit [INF] from='client.? 172.21.15.50:0/1424514617' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.037941+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.037941+0000 mon.a (mon.0) 1317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.040149+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.50:0/3876389472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.040149+0000 mon.c (mon.2) 209 : audit [INF] from='client.? 172.21.15.50:0/3876389472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.040974+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.040974+0000 mon.a (mon.0) 1318 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.063797+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.063797+0000 mon.a (mon.0) 1319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.065555+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.50:0/2853974335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.065555+0000 mon.b (mon.1) 106 : audit [INF] from='client.? 172.21.15.50:0/2853974335' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.132808+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.132808+0000 mon.a (mon.0) 1320 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.134244+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.134244+0000 mon.b (mon.1) 107 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.138801+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.138801+0000 mon.a (mon.0) 1321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.139886+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.139886+0000 mon.a (mon.0) 1322 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.140642+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.140642+0000 mon.b (mon.1) 108 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.141874+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.141874+0000 mon.b (mon.1) 109 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.428884+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:55 smithi050 bash[21463]: audit 2024-04-26T16:23:55.428884+0000 mon.c (mon.2) 210 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:55.896 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:55.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:55.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:56.028 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-04-26T16:23:56.028 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-04-26T16:23:56.028 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2978 ms) 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3037 ms) 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3013 ms) 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-04-26T16:23:56.029 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3006 ms) 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2829 ms) 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3001 ms) 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (9973 ms) 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (27837 ms total) 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (27837 ms total) 2024-04-26T16:23:56.030 INFO:tasks.workunit.client.0.smithi050.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-04-26T16:23:56.067 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: Running main() from gmock_main.cc 2024-04-26T16:23:56.067 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [----------] Global test environment set-up. 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2734 ms) 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3033 ms) 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3005 ms) 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3011 ms) 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2828 ms) 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2985 ms) 2024-04-26T16:23:56.068 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-04-26T16:23:56.069 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (10014 ms) 2024-04-26T16:23:56.069 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (27610 ms total) 2024-04-26T16:23:56.069 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: 2024-04-26T16:23:56.069 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [----------] Global test environment tear-down 2024-04-26T16:23:56.069 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (27610 ms total) 2024-04-26T16:23:56.069 INFO:tasks.workunit.client.0.smithi050.stdout: write_operations: [ PASSED ] 7 tests. 2024-04-26T16:23:56.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:56.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:56.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:56.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:56.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:56.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:56.498 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:56.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:56.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:56.619 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:56.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:56.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:56.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:56.743 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:23:56.743 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:55.494181+0000 mgr.y (mgr.24401) 247 : cluster [DBG] pgmap v235: 612 pgs: 192 unknown, 1 clean+premerge+peered, 419 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:56.743 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:55.494181+0000 mgr.y (mgr.24401) 247 : cluster [DBG] pgmap v235: 612 pgs: 192 unknown, 1 clean+premerge+peered, 419 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:56.743 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:56.004162+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:56.743 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:56.004162+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:56.006346+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:56.006346+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.008981+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.008981+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009089+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009089+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009213+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009213+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009321+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009321+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.744 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009426+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.009426+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.020675+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.020675+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:56.021585+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: cluster 2024-04-26T16:23:56.021585+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.026393+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.026393+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.178219+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.178219+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.190101+0000 mon.c (mon.2) 211 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.190101+0000 mon.c (mon.2) 211 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:56.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.406760+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:56.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.406760+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:56.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.408534+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:56.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.408534+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:56.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.431268+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:56.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:56 smithi190 bash[23868]: audit 2024-04-26T16:23:56.431268+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:56.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:56.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:56.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:56.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:55.494181+0000 mgr.y (mgr.24401) 247 : cluster [DBG] pgmap v235: 612 pgs: 192 unknown, 1 clean+premerge+peered, 419 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:55.494181+0000 mgr.y (mgr.24401) 247 : cluster [DBG] pgmap v235: 612 pgs: 192 unknown, 1 clean+premerge+peered, 419 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:56.004162+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:56.004162+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:56.006346+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:56.006346+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.008981+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]': finished 2024-04-26T16:23:56.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.008981+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009089+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009089+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009213+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009213+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009321+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009321+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009426+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.009426+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.020675+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:55.494181+0000 mgr.y (mgr.24401) 247 : cluster [DBG] pgmap v235: 612 pgs: 192 unknown, 1 clean+premerge+peered, 419 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:56.975 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:55.494181+0000 mgr.y (mgr.24401) 247 : cluster [DBG] pgmap v235: 612 pgs: 192 unknown, 1 clean+premerge+peered, 419 active+clean; 145 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:56.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:56.004162+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:56.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:56.004162+0000 mon.a (mon.0) 1323 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:23:56.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:56.006346+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:56.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:56.006346+0000 mon.a (mon.0) 1324 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:23:56.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.008981+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]': finished 2024-04-26T16:23:57.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:57.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:57.054 INFO:tasks.workunit.client.0.smithi050.stdout:h_notify_pp: flushed 2024-04-26T16:23:57.054 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: trying... 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442324512 notify_id 670014898180 notifier_gid 25307 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: timed out 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: flushing 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: flushed 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3011 ms) 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: List watches 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify2 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442337648 notify_id 682899800070 notifier_gid 25307 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify2 done 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: watch_check 2024-04-26T16:23:57.055 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: unwatch2 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: flushing 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: done 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3014 ms) 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: List watches 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify2 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: handle_notify cookie 94526442337648 notify_id 695784701957 notifier_gid 25307 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: notify2 done 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: watch_check 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: unwatch2 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: flushing 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: done 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3009 ms) 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16105 ms total) 2024-04-26T16:23:57.056 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: 2024-04-26T16:23:57.057 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-04-26T16:23:57.057 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28900 ms total) 2024-04-26T16:23:57.057 INFO:tasks.workunit.client.0.smithi050.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-04-26T16:23:57.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:57.172 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:57.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.020675+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:56.021585+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: cluster 2024-04-26T16:23:56.021585+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.026393+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.026393+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.178219+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.178219+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.190101+0000 mon.c (mon.2) 211 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:57.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.190101+0000 mon.c (mon.2) 211 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.406760+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.406760+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.408534+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.408534+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.431268+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[21463]: audit 2024-04-26T16:23:56.431268+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.008981+0000 mon.a (mon.0) 1325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-9", "mode": "writeback"}]': finished 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009089+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009089+0000 mon.a (mon.0) 1326 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi050-73572-6", "pool2": "test-rados-api-smithi050-73572-6", "yes_i_really_really_mean_it": true}]': finished 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009213+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:57.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009213+0000 mon.a (mon.0) 1327 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi050-73201-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009321+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009321+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi050-73193-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009426+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.009426+0000 mon.a (mon.0) 1329 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.020675+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.020675+0000 mon.b (mon.1) 110 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:56.021585+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: cluster 2024-04-26T16:23:56.021585+0000 mon.a (mon.0) 1330 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.026393+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.026393+0000 mon.a (mon.0) 1331 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.178219+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.178219+0000 mon.a (mon.0) 1332 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.190101+0000 mon.c (mon.2) 211 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:57.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.190101+0000 mon.c (mon.2) 211 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:23:57.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.406760+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.406760+0000 mon.a (mon.0) 1333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.408534+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.408534+0000 mon.b (mon.1) 111 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:23:57.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.431268+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:56 smithi050 bash[29234]: audit 2024-04-26T16:23:56.431268+0000 mon.c (mon.2) 212 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:57.431 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:23:57.431 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:23:57.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:57.492 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:57.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:57.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:56.631044+0000 mgr.y (mgr.24401) 248 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:57.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:56.631044+0000 mgr.y (mgr.24401) 248 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:57.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.013466+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:57.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.013466+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:57.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: cluster 2024-04-26T16:23:57.050372+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: cluster 2024-04-26T16:23:57.050372+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.050810+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.050810+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.060544+0000 mon.a (mon.0) 1336 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.060544+0000 mon.a (mon.0) 1336 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.437039+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.437039+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.495292+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.495292+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.501308+0000 mon.a (mon.0) 1337 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[29234]: audit 2024-04-26T16:23:57.501308+0000 mon.a (mon.0) 1337 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:56.631044+0000 mgr.y (mgr.24401) 248 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:56.631044+0000 mgr.y (mgr.24401) 248 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.013466+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.013466+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: cluster 2024-04-26T16:23:57.050372+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: cluster 2024-04-26T16:23:57.050372+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.050810+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.050810+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.060544+0000 mon.a (mon.0) 1336 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.060544+0000 mon.a (mon.0) 1336 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.437039+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.437039+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.495292+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.495292+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.501308+0000 mon.a (mon.0) 1337 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.614 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:57 smithi050 bash[21463]: audit 2024-04-26T16:23:57.501308+0000 mon.a (mon.0) 1337 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.617 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:57.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:57.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:56.631044+0000 mgr.y (mgr.24401) 248 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:57.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:56.631044+0000 mgr.y (mgr.24401) 248 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.013466+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.013466+0000 mon.a (mon.0) 1334 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: cluster 2024-04-26T16:23:57.050372+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: cluster 2024-04-26T16:23:57.050372+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.050810+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.050810+0000 mon.b (mon.1) 112 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.060544+0000 mon.a (mon.0) 1336 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.060544+0000 mon.a (mon.0) 1336 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.437039+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.437039+0000 mon.c (mon.2) 213 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.495292+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.495292+0000 mon.c (mon.2) 214 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.501308+0000 mon.a (mon.0) 1337 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:57 smithi190 bash[23868]: audit 2024-04-26T16:23:57.501308+0000 mon.a (mon.0) 1337 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-04-26T16:23:57.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:57.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:57.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:57.862 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:57.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:57.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 -- 172.21.15.50:0/1338287283 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc06ace0 msgr2=0x7f90dc107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 --2- 172.21.15.50:0/1338287283 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc06ace0 0x7f90dc107fd0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f90d8009990 tx=0x7f90d8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 -- 172.21.15.50:0/1338287283 shutdown_connections 2024-04-26T16:23:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 --2- 172.21.15.50:0/1338287283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90dc06a870 0x7f90dc107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 --2- 172.21.15.50:0/1338287283 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90dc101ae0 0x7f90dc101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 --2- 172.21.15.50:0/1338287283 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc06ace0 0x7f90dc107fd0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 -- 172.21.15.50:0/1338287283 >> 172.21.15.50:0/1338287283 conn(0x7f90dc078990 msgr2=0x7f90dc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 -- 172.21.15.50:0/1338287283 shutdown_connections 2024-04-26T16:23:57.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.875+0000 7f90e454c700 1 -- 172.21.15.50:0/1338287283 wait complete. 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 librados: starting msgr at 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 librados: starting objecter 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 Processor -- start 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 -- start start 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 librados: setting wanted keys 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 librados: calling monclient init 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90dc06a870 0x7f90dc10f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:57.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90dc06ace0 0x7f90dc10b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc101ae0 0x7f90dc074e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f90dc1109d0 con 0x7f90dc06ace0 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f90dc110850 con 0x7f90dc06a870 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f90dc110b50 con 0x7f90dc101ae0 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc101ae0 0x7f90dc074e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc101ae0 0x7f90dc074e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55806/0 (socket says 172.21.15.50:55806) 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 -- 172.21.15.50:0/2945259574 learned_addr learned my addr 172.21.15.50:0/2945259574 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 -- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90dc06a870 msgr2=0x7f90dc10f1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90dc06a870 0x7f90dc10f1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 -- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90dc06ace0 msgr2=0x7f90dc10b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f90dc06ace0 0x7f90dc10b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 -- 172.21.15.50:0/2945259574 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90dc10b7a0 con 0x7f90dc101ae0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e2ae9700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc101ae0 0x7f90dc074e90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f90d400b970 tx=0x7f90d400bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90d37fe700 10 client.?.objecter ms_handle_connect 0x7f90dc101ae0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90d37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4019070 con 0x7f90dc101ae0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f90d400ce10 con 0x7f90dc101ae0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 -- 172.21.15.50:0/2945259574 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90dc10ba90 con 0x7f90dc101ae0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 10 client.34127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 -- 172.21.15.50:0/2945259574 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90dc10bfd0 con 0x7f90dc101ae0 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.879+0000 7f90e454c700 1 librados: init done 2024-04-26T16:23:57.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.883+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d4014e00 con 0x7f90dc101ae0 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout:ons, quorum a,b,c (age 13m) 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: mgr: y(active, since 5m), standbys: x 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: osd: 8 osds: 8 up (since 6m), 8 in (since 7m) 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: flags noscrub 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: data: 2024-04-26T16:23:57.883 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: pools: 32 pools, 868 pgs 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: objects: 238 objects, 586 KiB 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: usage: 271 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: pgs: 51.613% pgs unknown 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 7.373% pgs not active 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 448 unknown 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 356 active+clean 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 64 creating+peering 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4428 ms) 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ RUN ] LibRadosService.Status 2024-04-26T16:23:57.884 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ OK ] LibRadosService.Status (20053 ms) 2024-04-26T16:23:57.885 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [----------] 4 tests from LibRadosService (29638 ms total) 2024-04-26T16:23:57.885 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: 2024-04-26T16:23:57.885 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [----------] Global test environment tear-down 2024-04-26T16:23:57.885 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29639 ms total) 2024-04-26T16:23:57.885 INFO:tasks.workunit.client.0.smithi050.stdout: api_service: [ PASSED ] 4 tests. 2024-04-26T16:23:57.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d17fa700 10 client.34127.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d17fa700 15 client.34127.objecter _session_command_op_assign -1 1 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d17fa700 20 client.34127.objecter _get_session osd=-1 returning homeless 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d17fa700 10 client.34127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f90c4002920 con 0x7f90dc101ae0 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f90d400c9b0 con 0x7f90dc101ae0 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d37fe700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f90c8072090 0x7f90c8074550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(169..169 src has 87..169) ==== 12774+0+0 (secure 0 0 0) 0x7f90d4091bc0 con 0x7f90dc101ae0 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d37fe700 10 client.34127.objecter ms_dispatch 0x7f90dc001550 osd_map(169..169 src has 87..169) 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d37fe700 3 client.34127.objecter handle_osd_map got epochs [169,169] > 0 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.887+0000 7f90d37fe700 3 client.34127.objecter handle_osd_map decoding full epoch 169 2024-04-26T16:23:57.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90e22e8700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f90c8072090 0x7f90c8074550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90e22e8700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f90c8072090 0x7f90c8074550 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009710 tx=0x7f90cc006c60 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 10 client.34127.objecter checking command 1 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f90c8086ca0 0x7f90c8089140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 20 client.34127.objecter _get_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 15 client.34127.objecter _session_command_op_remove -1 1 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 20 client.34127.objecter _get_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 20 client.34127.objecter get_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 15 client.34127.objecter _session_command_op_assign 5 1 2024-04-26T16:23:57.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 4 2024-04-26T16:23:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 10 client.34127.objecter _send_command 1 2024-04-26T16:23:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90c8089810 con 0x7f90c8086ca0 2024-04-26T16:23:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 20 client.34127.objecter dump_active .. 0 homeless 2024-04-26T16:23:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 10 client.34127.objecter ms_handle_connect 0x7f90c8072090 2024-04-26T16:23:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=169) ==== 24+0+0 (secure 0 0 0) 0x7f90d4098050 con 0x7f90dc101ae0 2024-04-26T16:23:57.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.891+0000 7f90e1ae7700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f90c8086ca0 0x7f90c8089140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:57.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.903+0000 7f90e1ae7700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f90c8086ca0 0x7f90c8089140 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:57.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.903+0000 7f90d37fe700 10 client.34127.objecter ms_handle_connect 0x7f90c8086ca0 2024-04-26T16:23:57.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.907+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f90c8089810 con 0x7f90c8086ca0 2024-04-26T16:23:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.907+0000 7f90d37fe700 10 client.34127.objecter ms_dispatch 0x7f90dc001550 command_reply(tid 1: 0 ) 2024-04-26T16:23:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.907+0000 7f90d37fe700 10 client.34127.objecter _finish_command 1 = system:0 2024-04-26T16:23:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.907+0000 7f90d37fe700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 4 2024-04-26T16:23:57.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.907+0000 7f90d37fe700 15 client.34127.objecter _session_command_op_remove 5 1 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 10 client.34127.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 15 client.34127.objecter _session_command_op_assign -1 2 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter _get_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter _get_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 15 client.34127.objecter _session_command_op_remove -1 2 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter get_session s=0x7f90c80865d0 osd=5 3 2024-04-26T16:23:57.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 15 client.34127.objecter _session_command_op_assign 5 2 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 4 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 10 client.34127.objecter _send_command 2 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f90c4001220 con 0x7f90c8086ca0 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d37fe700 1 -- 172.21.15.50:0/2945259574 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f90c4001220 con 0x7f90c8086ca0 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d37fe700 10 client.34127.objecter ms_dispatch 0x7f90dc001550 command_reply(tid 2: 0 ) 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d37fe700 10 client.34127.objecter _finish_command 2 = system:0 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d37fe700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 4 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d37fe700 15 client.34127.objecter _session_command_op_remove 5 2 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 10 librados: watch_flush enter 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 10 librados: watch_flush exit 2024-04-26T16:23:57.950 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 10 client.34127.objecter close_session for osd.5 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f90c8086ca0 msgr2=0x7f90c8089140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f90c8086ca0 0x7f90c8089140 crc :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter put_session s=0x7f90c80865d0 osd=5 1 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 20 client.34127.objecter shutdown clearing up homeless session... 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 10 client.34127.objecter successfully canceled tick 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f90c8072090 msgr2=0x7f90c8074550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f90c8072090 0x7f90c8074550 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009710 tx=0x7f90cc006c60 comp rx=0 tx=0).stop 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc101ae0 msgr2=0x7f90dc074e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:57.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.951+0000 7f90d17fa700 1 --2- 172.21.15.50:0/2945259574 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f90dc101ae0 0x7f90dc074e90 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f90d400b970 tx=0x7f90d400bd30 comp rx=0 tx=0).stop 2024-04-26T16:23:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.955+0000 7f90e2ae9700 1 -- 172.21.15.50:0/2945259574 reap_dead start 2024-04-26T16:23:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.955+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 shutdown_connections 2024-04-26T16:23:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.955+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 >> 172.21.15.50:0/2945259574 conn(0x7f90dc078990 msgr2=0x7f90dc105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.955+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 shutdown_connections 2024-04-26T16:23:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.955+0000 7f90d17fa700 1 -- 172.21.15.50:0/2945259574 wait complete. 2024-04-26T16:23:57.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:57.955+0000 7f90d17fa700 1 librados: shutdown 2024-04-26T16:23:57.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:58.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:23:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:23:58.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:58.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:58.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/2071355232 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 msgr2=0x7f7b08108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 --2- 172.21.15.50:0/2071355232 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b08108770 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b970 tx=0x7f7af000bc80 comp rx=0 tx=0).stop 2024-04-26T16:23:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/2071355232 shutdown_connections 2024-04-26T16:23:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 --2- 172.21.15.50:0/2071355232 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b08102ce0 0x7f7b0810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 --2- 172.21.15.50:0/2071355232 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b08108770 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 --2- 172.21.15.50:0/2071355232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08102320 0x7f7b081027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.279+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/2071355232 >> 172.21.15.50:0/2071355232 conn(0x7f7b080792d0 msgr2=0x7f7b080796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/2071355232 shutdown_connections 2024-04-26T16:23:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/2071355232 wait complete. 2024-04-26T16:23:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 librados: starting msgr at 2024-04-26T16:23:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 librados: starting objecter 2024-04-26T16:23:58.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 Processor -- start 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 -- start start 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 librados: setting wanted keys 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 librados: calling monclient init 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b08102320 0x7f7b08075090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08102ce0 0x7f7b0806f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b0806f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0806da00 con 0x7f7b08102320 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0806d880 con 0x7f7b08102ce0 2024-04-26T16:23:58.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0cc7e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0806db80 con 0x7f7b08108390 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b0806f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b0806f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56754/0 (socket says 172.21.15.50:56754) 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 learned_addr learned my addr 172.21.15.50:0/309386371 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b08102320 0x7f7b08075090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08102ce0 msgr2=0x7f7b0806f160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7b08102ce0 0x7f7b0806f160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b08102320 msgr2=0x7f7b08075090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b08102320 0x7f7b08075090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0819ab60 con 0x7f7b08108390 2024-04-26T16:23:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7b08102320 0x7f7b08075090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:23:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b0806f880 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000b970 tx=0x7f7b0000bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7aff7fe700 10 client.?.objecter ms_handle_connect 0x7f7b08108390 2024-04-26T16:23:58.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.287+0000 7f7aff7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:23:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00019070 con 0x7f7b08108390 2024-04-26T16:23:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7b0000ce10 con 0x7f7b08108390 2024-04-26T16:23:58.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b00014e00 con 0x7f7b08108390 2024-04-26T16:23:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0819ae50 con 0x7f7b08108390 2024-04-26T16:23:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7b0cc7e700 10 client.34376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:23:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b0819b310 con 0x7f7b08108390 2024-04-26T16:23:58.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7b0cc7e700 1 librados: init done 2024-04-26T16:23:58.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.295+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f7b08103a40 con 0x7f7b08108390 2024-04-26T16:23:58.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7b0000c9b0 con 0x7f7b08108390 2024-04-26T16:23:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7af4071f70 0x7f7af4074430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(170..170 src has 87..170) ==== 12513+0+0 (secure 0 0 0) 0x7f7b00092950 con 0x7f7b08108390 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 osd_map(170..170 src has 87..170) 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 3 client.34376.objecter handle_osd_map got epochs [170,170] > 0 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 3 client.34376.objecter handle_osd_map decoding full epoch 170 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 20 client.34376.objecter dump_active .. 0 homeless 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v170) ==== 54+0+16 (secure 0 0 0) 0x7f7b0005d760 con 0x7f7b08108390 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 10 client.34376.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 15 client.34376.objecter _session_command_op_assign -1 1 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7b08064030 0x7f7b08064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 20 client.34376.objecter _get_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 20 client.34376.objecter _get_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 15 client.34376.objecter _session_command_op_remove -1 1 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 20 client.34376.objecter get_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 15 client.34376.objecter _session_command_op_assign 0 1 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 4 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 10 client.34376.objecter _send_command 1 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b0806a110 con 0x7f7b08064030 2024-04-26T16:23:58.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b05d9b700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7b08064030 0x7f7b08064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b08064030 2024-04-26T16:23:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.299+0000 7f7b05d9b700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7b08064030 0x7f7b08064410 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7af4071f70 0x7f7af4074430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b0806a110 con 0x7f7b08064030 2024-04-26T16:23:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 1: 0 ) 2024-04-26T16:23:58.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 1 = system:0 2024-04-26T16:23:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 4 2024-04-26T16:23:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 0 1 2024-04-26T16:23:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7af4071f70 2024-04-26T16:23:58.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.303+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7af4071f70 0x7f7af4074430 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b890 tx=0x7f7af000d040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.320 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 10 client.34376.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 15 client.34376.objecter _session_command_op_assign -1 2 2024-04-26T16:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 20 client.34376.objecter _get_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 20 client.34376.objecter _get_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 15 client.34376.objecter _session_command_op_remove -1 2 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 20 client.34376.objecter get_session s=0x7f7b08070640 osd=0 3 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 15 client.34376.objecter _session_command_op_assign 0 2 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 4 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 10 client.34376.objecter _send_command 2 2024-04-26T16:23:58.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.335+0000 7f7b0cc7e700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b08063ab0 con 0x7f7b08064030 2024-04-26T16:23:58.344 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.339+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b08063ab0 con 0x7f7b08064030 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.339+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.339+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.339+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 4 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.339+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 0 2 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 3 2024-04-26T16:23:58.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7b0802e490 0x7f7b081b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 1 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 4 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 10 client.34376.objecter _send_command 3 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.343+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f7b0802e870 con 0x7f7b0802e490 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7b0802e490 0x7f7b081b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7b0802e490 0x7f7b081b2370 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b0802e490 2024-04-26T16:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b0802e870 con 0x7f7b0802e490 2024-04-26T16:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 3: 0 ) 2024-04-26T16:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 3 = system:0 2024-04-26T16:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 4 2024-04-26T16:23:58.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.347+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 1 3 2024-04-26T16:23:58.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 4 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 4 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b0810a030 osd=1 3 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 1 4 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 4 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 10 client.34376.objecter _send_command 4 2024-04-26T16:23:58.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.379+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b08109ab0 con 0x7f7b0802e490 2024-04-26T16:23:58.380 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:23:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.383+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b08109ab0 con 0x7f7b0802e490 2024-04-26T16:23:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.383+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.383+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.383+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 4 2024-04-26T16:23:58.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.383+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 1 4 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 5 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7b0802dff0 0x7f7b081b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7b05d9b700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7b0802dff0 0x7f7b081b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7b05d9b700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7b0802dff0 0x7f7b081b5a40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b0802dff0 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 5 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 2 5 2024-04-26T16:23:58.392 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 4 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 10 client.34376.objecter _send_command 5 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f7b081b88a0 con 0x7f7b0802dff0 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b081b88a0 con 0x7f7b0802dff0 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 5: 0 ) 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 5 = system:0 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 4 2024-04-26T16:23:58.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.387+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 2 5 2024-04-26T16:23:58.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 6 2024-04-26T16:23:58.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 6 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b08109eb0 osd=2 3 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 2 6 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 4 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 10 client.34376.objecter _send_command 6 2024-04-26T16:23:58.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.419+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b0802d560 con 0x7f7b0802dff0 2024-04-26T16:23:58.422 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:23:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b0802d560 con 0x7f7b0802dff0 2024-04-26T16:23:58.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 4 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 2 6 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 7 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7b081b3590 0x7f7b081b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 7 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 3 7 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 4 2024-04-26T16:23:58.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 10 client.34376.objecter _send_command 7 2024-04-26T16:23:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f7b081bc200 con 0x7f7b081b3590 2024-04-26T16:23:58.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7b081b3590 0x7f7b081b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.423+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7b081b3590 0x7f7b081b7100 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.427+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b081b3590 2024-04-26T16:23:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.427+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b081bc200 con 0x7f7b081b3590 2024-04-26T16:23:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.427+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 7: 0 ) 2024-04-26T16:23:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.427+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 7 = system:0 2024-04-26T16:23:58.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.427+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 4 2024-04-26T16:23:58.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.427+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 3 7 2024-04-26T16:23:58.444 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:58.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 8 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 8 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b0802d560 osd=3 3 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 3 8 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 4 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 10 client.34376.objecter _send_command 8 2024-04-26T16:23:58.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.463+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b081b2b00 con 0x7f7b081b3590 2024-04-26T16:23:58.464 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b081b2b00 con 0x7f7b081b3590 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 4 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 3 8 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 9 2024-04-26T16:23:58.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7b081b6c60 0x7f7b081bca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 9 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 4 9 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 4 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 10 client.34376.objecter _send_command 9 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f7b081b2a20 con 0x7f7b081b6c60 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7b081b6c60 0x7f7b081bca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7b081b6c60 0x7f7b081bca70 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b081b6c60 2024-04-26T16:23:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b081b2a20 con 0x7f7b081b6c60 2024-04-26T16:23:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 9: 0 ) 2024-04-26T16:23:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 9 = system:0 2024-04-26T16:23:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 4 2024-04-26T16:23:58.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.467+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 4 9 2024-04-26T16:23:58.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 10 2024-04-26T16:23:58.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 10 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081b28b0 osd=4 3 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 4 10 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 4 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 10 client.34376.objecter _send_command 10 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.495+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b081b2a20 con 0x7f7b081b6c60 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.499+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b081b2a20 con 0x7f7b081b6c60 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.499+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.499+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.499+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 4 2024-04-26T16:23:58.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.499+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 4 10 2024-04-26T16:23:58.499 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 11 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7b081bd4c0 0x7f7b081bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 11 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 5 11 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 4 2024-04-26T16:23:58.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 10 client.34376.objecter _send_command 11 2024-04-26T16:23:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f7b081c2710 con 0x7f7b081bd4c0 2024-04-26T16:23:58.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7b05d9b700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7b081bd4c0 0x7f7b081bf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b081bd4c0 2024-04-26T16:23:58.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.503+0000 7f7b05d9b700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7b081bd4c0 0x7f7b081bf8b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.507+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b081c2710 con 0x7f7b081bd4c0 2024-04-26T16:23:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.507+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 11: 0 ) 2024-04-26T16:23:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.507+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 11 = system:0 2024-04-26T16:23:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.507+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 4 2024-04-26T16:23:58.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.507+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 5 11 2024-04-26T16:23:58.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 12 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 12 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081b2c70 osd=5 3 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 5 12 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 4 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 10 client.34376.objecter _send_command 12 2024-04-26T16:23:58.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b081b61d0 con 0x7f7b081bd4c0 2024-04-26T16:23:58.534 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b081b61d0 con 0x7f7b081bd4c0 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 4 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.535+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 5 12 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 13 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7b081b5f80 0x7f7b081c0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 13 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 6 13 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 4 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 10 client.34376.objecter _send_command 13 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f7b081c2680 con 0x7f7b081b5f80 2024-04-26T16:23:58.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7b081b5f80 0x7f7b081c0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7b06d9d700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7b081b5f80 0x7f7b081c0f70 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.539+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b081b5f80 2024-04-26T16:23:58.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.543+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b081c2680 con 0x7f7b081b5f80 2024-04-26T16:23:58.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.543+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 13: 0 ) 2024-04-26T16:23:58.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.543+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 13 = system:0 2024-04-26T16:23:58.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.543+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 4 2024-04-26T16:23:58.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.543+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 6 13 2024-04-26T16:23:58.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.559+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.559+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 14 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.559+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.559+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.559+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.559+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 14 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081b2ae0 osd=6 3 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 6 14 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 4 2024-04-26T16:23:58.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 10 client.34376.objecter _send_command 14 2024-04-26T16:23:58.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b081c2680 con 0x7f7b081b5f80 2024-04-26T16:23:58.561 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:23:58.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b081c2680 con 0x7f7b081b5f80 2024-04-26T16:23:58.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 4 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 6 14 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 15 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7b081c19c0 0x7f7b081c1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 15 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 7 15 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 4 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 10 client.34376.objecter _send_command 15 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f7b081b8430 con 0x7f7b081c19c0 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7b081c19c0 0x7f7b081c1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7b0659c700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7b081c19c0 0x7f7b081c1e20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:23:58.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.563+0000 7f7aff7fe700 10 client.34376.objecter ms_handle_connect 0x7f7b081c19c0 2024-04-26T16:23:58.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.567+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b081b8430 con 0x7f7b081c19c0 2024-04-26T16:23:58.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.567+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 15: 0 ) 2024-04-26T16:23:58.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.567+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 15 = system:0 2024-04-26T16:23:58.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.567+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 4 2024-04-26T16:23:58.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.567+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 7 15 2024-04-26T16:23:58.567 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:23:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 10 client.34376.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign -1 16 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 20 client.34376.objecter _get_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_remove -1 16 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 20 client.34376.objecter get_session s=0x7f7b081c2680 osd=7 3 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 15 client.34376.objecter _session_command_op_assign 7 16 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 4 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 10 client.34376.objecter _send_command 16 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.587+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f7b081b8430 con 0x7f7b081c19c0 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7aff7fe700 1 -- 172.21.15.50:0/309386371 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f7b081b8430 con 0x7f7b081c19c0 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7aff7fe700 10 client.34376.objecter ms_dispatch 0x7f7b08001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7aff7fe700 10 client.34376.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7aff7fe700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 4 2024-04-26T16:23:58.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7aff7fe700 15 client.34376.objecter _session_command_op_remove 7 16 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 librados: watch_flush enter 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 librados: watch_flush exit 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.0 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7b08064030 msgr2=0x7f7b08064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f7b08064030 0x7f7b08064410 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b08070640 osd=0 1 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.1 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7b0802e490 msgr2=0x7f7b081b2370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f7b0802e490 0x7f7b081b2370 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0810a030 osd=1 1 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.2 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7b0802dff0 msgr2=0x7f7b081b5a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f7b0802dff0 0x7f7b081b5a40 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b08109eb0 osd=2 1 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.3 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7b081b3590 msgr2=0x7f7b081b7100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f7b081b3590 0x7f7b081b7100 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b0802d560 osd=3 1 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.4 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7b081b6c60 msgr2=0x7f7b081bca70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f7b081b6c60 0x7f7b081bca70 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b28b0 osd=4 1 2024-04-26T16:23:58.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.5 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7b081bd4c0 msgr2=0x7f7b081bf8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f7b081bd4c0 0x7f7b081bf8b0 crc :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2c70 osd=5 1 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.6 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7b081b5f80 msgr2=0x7f7b081c0f70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f7b081b5f80 0x7f7b081c0f70 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081b2ae0 osd=6 1 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter close_session for osd.7 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7b081c19c0 msgr2=0x7f7b081c1e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f7b081c19c0 0x7f7b081c1e20 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter put_session s=0x7f7b081c2680 osd=7 1 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 20 client.34376.objecter shutdown clearing up homeless session... 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 10 client.34376.objecter successfully canceled tick 2024-04-26T16:23:58.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 reap_dead start 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 reap_dead start 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 reap_dead start 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 reap_dead start 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7af4071f70 msgr2=0x7f7af4074430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7af4071f70 0x7f7af4074430 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b890 tx=0x7f7af000d040 comp rx=0 tx=0).stop 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7b06d9d700 1 -- 172.21.15.50:0/309386371 reap_dead start 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 msgr2=0x7f7b0806f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b0806f880 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f7b0000b970 tx=0x7f7b0000bd30 comp rx=0 tx=0).stop 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 shutdown_connections 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 --2- 172.21.15.50:0/309386371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7b08108390 0x7f7b0806f880 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 >> 172.21.15.50:0/309386371 conn(0x7f7b080792d0 msgr2=0x7f7b08100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:23:58.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.591+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 shutdown_connections 2024-04-26T16:23:58.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.599+0000 7f7afd7fa700 1 -- 172.21.15.50:0/309386371 wait complete. 2024-04-26T16:23:58.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:23:58.599+0000 7f7afd7fa700 1 librados: shutdown 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:23:58.603 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [----------] Global test environment set-up. 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5077 ms) 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (86 ms) 2024-04-26T16:23:58.606 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20113 ms) 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: attempt 0 of 20 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5130 ms) 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30406 ms total) 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [----------] Global test environment tear-down 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30406 ms total) 2024-04-26T16:23:58.607 INFO:tasks.workunit.client.0.smithi050.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-04-26T16:23:58.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:58.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:58.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: cluster 2024-04-26T16:23:57.494905+0000 mgr.y (mgr.24401) 249 : cluster [DBG] pgmap v238: 511 pgs: 64 creating+peering, 32 unknown, 1 peering, 414 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:58.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: cluster 2024-04-26T16:23:57.494905+0000 mgr.y (mgr.24401) 249 : cluster [DBG] pgmap v238: 511 pgs: 64 creating+peering, 32 unknown, 1 peering, 414 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: cluster 2024-04-26T16:23:58.014495+0000 mon.a (mon.0) 1338 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: cluster 2024-04-26T16:23:58.014495+0000 mon.a (mon.0) 1338 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.026311+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.026311+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.026463+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.026463+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.026598+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.026598+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: cluster 2024-04-26T16:23:58.031688+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: cluster 2024-04-26T16:23:58.031688+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.044845+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.044845+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.045147+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.045147+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.305774+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.50:0/309386371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.305774+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.50:0/309386371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.439373+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:58.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:58 smithi190 bash[23868]: audit 2024-04-26T16:23:58.439373+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:58.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:58.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: cluster 2024-04-26T16:23:57.494905+0000 mgr.y (mgr.24401) 249 : cluster [DBG] pgmap v238: 511 pgs: 64 creating+peering, 32 unknown, 1 peering, 414 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: cluster 2024-04-26T16:23:57.494905+0000 mgr.y (mgr.24401) 249 : cluster [DBG] pgmap v238: 511 pgs: 64 creating+peering, 32 unknown, 1 peering, 414 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: cluster 2024-04-26T16:23:58.014495+0000 mon.a (mon.0) 1338 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: cluster 2024-04-26T16:23:58.014495+0000 mon.a (mon.0) 1338 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.026311+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.026311+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.026463+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.026463+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.026598+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.026598+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: cluster 2024-04-26T16:23:58.031688+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: cluster 2024-04-26T16:23:58.031688+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.044845+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.044845+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.045147+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.045147+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.305774+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.50:0/309386371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.305774+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.50:0/309386371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.439373+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[21463]: audit 2024-04-26T16:23:58.439373+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: cluster 2024-04-26T16:23:57.494905+0000 mgr.y (mgr.24401) 249 : cluster [DBG] pgmap v238: 511 pgs: 64 creating+peering, 32 unknown, 1 peering, 414 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: cluster 2024-04-26T16:23:57.494905+0000 mgr.y (mgr.24401) 249 : cluster [DBG] pgmap v238: 511 pgs: 64 creating+peering, 32 unknown, 1 peering, 414 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: cluster 2024-04-26T16:23:58.014495+0000 mon.a (mon.0) 1338 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: cluster 2024-04-26T16:23:58.014495+0000 mon.a (mon.0) 1338 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.026311+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:23:58.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.026311+0000 mon.a (mon.0) 1339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi050-73420-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.026463+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.026463+0000 mon.a (mon.0) 1340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-9"}]': finished 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.026598+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.026598+0000 mon.a (mon.0) 1341 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "26"}]': finished 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: cluster 2024-04-26T16:23:58.031688+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: cluster 2024-04-26T16:23:58.031688+0000 mon.a (mon.0) 1342 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.044845+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.044845+0000 mon.a (mon.0) 1343 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.045147+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.045147+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.305774+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.50:0/309386371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.305774+0000 mon.c (mon.2) 215 : audit [DBG] from='client.? 172.21.15.50:0/309386371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.439373+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:58.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:58 smithi050 bash[29234]: audit 2024-04-26T16:23:58.439373+0000 mon.c (mon.2) 216 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:58.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:58.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:58.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:59.108 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:23:59.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:23:59.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:59.242 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:23:59.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:59.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:23:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:23:59.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:59.515 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:23:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:23:59.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:59.644 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:23:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:23:59.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:58.607327+0000 mgr.y (mgr.24401) 250 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:58.607327+0000 mgr.y (mgr.24401) 250 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:58.609466+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? 172.21.15.50:0/346577753' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:58.609466+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? 172.21.15.50:0/346577753' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.030117+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.030117+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.030218+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.030218+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: cluster 2024-04-26T16:23:59.038895+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: cluster 2024-04-26T16:23:59.038895+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.047568+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.047568+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.052749+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.052749+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.440598+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[21463]: audit 2024-04-26T16:23:59.440598+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:58.607327+0000 mgr.y (mgr.24401) 250 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:58.607327+0000 mgr.y (mgr.24401) 250 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:58.609466+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? 172.21.15.50:0/346577753' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:58.609466+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? 172.21.15.50:0/346577753' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.030117+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.030117+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.030218+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.030218+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: cluster 2024-04-26T16:23:59.038895+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: cluster 2024-04-26T16:23:59.038895+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.047568+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.047568+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.052749+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.052749+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.440598+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:59.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:23:59 smithi050 bash[29234]: audit 2024-04-26T16:23:59.440598+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:58.607327+0000 mgr.y (mgr.24401) 250 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:58.607327+0000 mgr.y (mgr.24401) 250 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:58.609466+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? 172.21.15.50:0/346577753' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:58.609466+0000 mon.b (mon.1) 113 : audit [DBG] from='client.? 172.21.15.50:0/346577753' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.030117+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.030117+0000 mon.a (mon.0) 1345 : audit [INF] from='client.? 172.21.15.50:0/2196624441' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi050-73201-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.030218+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.030218+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.50:0/1150997960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi050-73193-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: cluster 2024-04-26T16:23:59.038895+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: cluster 2024-04-26T16:23:59.038895+0000 mon.a (mon.0) 1347 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.047568+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.047568+0000 mon.b (mon.1) 114 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.052749+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.052749+0000 mon.a (mon.0) 1348 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:23:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.440598+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:59.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:23:59 smithi190 bash[23868]: audit 2024-04-26T16:23:59.440598+0000 mon.c (mon.2) 217 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:23:59.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:23:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:23:59.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:59.988 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:23:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:23:59.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- 172.21.15.50:0/1518881389 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c111c00 msgr2=0x7fcb6c10a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- 172.21.15.50:0/1518881389 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c111c00 0x7fcb6c10a420 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fcb64009f90 tx=0x7fcb6400b330 comp rx=0 tx=0).stop 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- 172.21.15.50:0/1518881389 shutdown_connections 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- 172.21.15.50:0/1518881389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c11cc00 0x7fcb6c11eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- 172.21.15.50:0/1518881389 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c111c00 0x7fcb6c10a420 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- 172.21.15.50:0/1518881389 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c10a960 0x7fcb6c10ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- 172.21.15.50:0/1518881389 >> 172.21.15.50:0/1518881389 conn(0x7fcb6c06e590 msgr2=0x7fcb6c06e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- 172.21.15.50:0/1518881389 shutdown_connections 2024-04-26T16:24:00.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- 172.21.15.50:0/1518881389 wait complete. 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 librados: starting msgr at 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 librados: starting objecter 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 Processor -- start 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- start start 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 librados: setting wanted keys 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 librados: calling monclient init 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 0x7fcb6c11c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c111c00 0x7fcb6c116310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c11cc00 0x7fcb6c116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c112a60 con 0x7fcb6c111c00 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6c1128e0 con 0x7fcb6c10a960 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb716cb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fcb6c112be0 con 0x7fcb6c11cc00 2024-04-26T16:24:00.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb6b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c11cc00 0x7fcb6c116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb6affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 0x7fcb6c11c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb6affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 0x7fcb6c11c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.50:48782/0 (socket says 172.21.15.50:48782) 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb6affd700 1 -- 172.21.15.50:0/47561067 learned_addr learned my addr 172.21.15.50:0/47561067 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.043+0000 7fcb6a7fc700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c111c00 0x7fcb6c116310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6b7fe700 1 -- 172.21.15.50:0/47561067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 msgr2=0x7fcb6c11c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6b7fe700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 0x7fcb6c11c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6b7fe700 1 -- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c111c00 msgr2=0x7fcb6c116310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6b7fe700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c111c00 0x7fcb6c116310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6b7fe700 1 -- 172.21.15.50:0/47561067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6c116ec0 con 0x7fcb6c11cc00 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6affd700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 0x7fcb6c11c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6a7fc700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c111c00 0x7fcb6c116310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb6b7fe700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c11cc00 0x7fcb6c116850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06cb90 tx=0x7fcb6000da00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb53fff700 10 client.?.objecter ms_handle_connect 0x7fcb6c11cc00 2024-04-26T16:24:00.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.047+0000 7fcb53fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:00.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.063+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60016070 con 0x7fcb6c11cc00 2024-04-26T16:24:00.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.063+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fcb6000f690 con 0x7fcb6c11cc00 2024-04-26T16:24:00.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.063+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb600038c0 con 0x7fcb6c11cc00 2024-04-26T16:24:00.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.067+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6c1bd000 con 0x7fcb6c11cc00 2024-04-26T16:24:00.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.071+0000 7fcb716cb700 10 client.35477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:00.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.079+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fcb60004220 con 0x7fcb6c11cc00 2024-04-26T16:24:00.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.079+0000 7fcb53fff700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcb54072000 0x7fcb540744c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.083+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb6c1bd3a0 con 0x7fcb6c11cc00 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.083+0000 7fcb716cb700 1 librados: init done 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.087+0000 7fcb6affd700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcb54072000 0x7fcb540744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.087+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(172..172 src has 87..172) ==== 13212+0+0 (secure 0 0 0) 0x7fcb60059be0 con 0x7fcb6c11cc00 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.087+0000 7fcb53fff700 10 client.35477.objecter ms_dispatch 0x7fcb6c001550 osd_map(172..172 src has 87..172) 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.087+0000 7fcb53fff700 3 client.35477.objecter handle_osd_map got epochs [172,172] > 0 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.087+0000 7fcb53fff700 3 client.35477.objecter handle_osd_map decoding full epoch 172 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.087+0000 7fcb53fff700 20 client.35477.objecter dump_active .. 0 homeless 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.091+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb6c06fb20 con 0x7fcb6c11cc00 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.091+0000 7fcb6affd700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcb54072000 0x7fcb540744c0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400b6c0 tx=0x7fcb64009b00 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:00.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.091+0000 7fcb53fff700 10 client.35477.objecter ms_handle_connect 0x7fcb54072000 2024-04-26T16:24:00.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.111+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fcb6005c240 con 0x7fcb6c11cc00 2024-04-26T16:24:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:00.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:00.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:00.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:00.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:00.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:00.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:00.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:00.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:00.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:00.603+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fcb6c1bd570 con 0x7fcb6c11cc00 2024-04-26T16:24:00.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: cluster 2024-04-26T16:23:59.499687+0000 mgr.y (mgr.24401) 251 : cluster [DBG] pgmap v241: 519 pgs: 200 unknown, 1 peering, 318 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: cluster 2024-04-26T16:23:59.499687+0000 mgr.y (mgr.24401) 251 : cluster [DBG] pgmap v241: 519 pgs: 200 unknown, 1 peering, 318 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.045317+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.045317+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:00.050470+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:00.050470+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.079550+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.079550+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.080469+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.080469+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.443577+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[21463]: audit 2024-04-26T16:24:00.443577+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: cluster 2024-04-26T16:23:59.499687+0000 mgr.y (mgr.24401) 251 : cluster [DBG] pgmap v241: 519 pgs: 200 unknown, 1 peering, 318 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: cluster 2024-04-26T16:23:59.499687+0000 mgr.y (mgr.24401) 251 : cluster [DBG] pgmap v241: 519 pgs: 200 unknown, 1 peering, 318 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.045317+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.045317+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:00.050470+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:00.050470+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.079550+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.079550+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.080469+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.718 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.080469+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.443577+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:00 smithi050 bash[29234]: audit 2024-04-26T16:24:00.443577+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:00.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:00.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:00.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: cluster 2024-04-26T16:23:59.499687+0000 mgr.y (mgr.24401) 251 : cluster [DBG] pgmap v241: 519 pgs: 200 unknown, 1 peering, 318 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:00.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: cluster 2024-04-26T16:23:59.499687+0000 mgr.y (mgr.24401) 251 : cluster [DBG] pgmap v241: 519 pgs: 200 unknown, 1 peering, 318 active+clean; 145 MiB data, 780 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:00.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.045317+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:00.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.045317+0000 mon.a (mon.0) 1349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73405-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:00.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:00.050470+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-04-26T16:24:00.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:00.050470+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2024-04-26T16:24:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.079550+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.079550+0000 mon.a (mon.0) 1351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.080469+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.080469+0000 mon.b (mon.1) 115 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.443577+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:00.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:00 smithi190 bash[23868]: audit 2024-04-26T16:24:00.443577+0000 mon.c (mon.2) 218 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:00.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:00.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:01.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.031+0000 7fcb53fff700 1 -- 172.21.15.50:0/47561067 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v173) ==== 97+0+0 (secure 0 0 0) 0x7fcb6005fef0 con 0x7fcb6c11cc00 2024-04-26T16:24:01.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 10 librados: watch_flush enter 2024-04-26T16:24:01.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 10 librados: watch_flush exit 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 20 client.35477.objecter shutdown clearing up homeless session... 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 10 client.35477.objecter successfully canceled tick 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcb54072000 msgr2=0x7fcb540744c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcb54072000 0x7fcb540744c0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400b6c0 tx=0x7fcb64009b00 comp rx=0 tx=0).stop 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c11cc00 msgr2=0x7fcb6c116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.035+0000 7fcb716cb700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c11cc00 0x7fcb6c116850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06cb90 tx=0x7fcb6000da00 comp rx=0 tx=0).stop 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.039+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 shutdown_connections 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.039+0000 7fcb716cb700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcb54072000 0x7fcb540744c0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.039+0000 7fcb716cb700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb6c10a960 0x7fcb6c11c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:01.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.039+0000 7fcb716cb700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcb6c11cc00 0x7fcb6c116850 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:01.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.039+0000 7fcb716cb700 1 --2- 172.21.15.50:0/47561067 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcb6c111c00 0x7fcb6c116310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:01.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.039+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 >> 172.21.15.50:0/47561067 conn(0x7fcb6c06e590 msgr2=0x7fcb6c11e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:01.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.047+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 shutdown_connections 2024-04-26T16:24:01.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.047+0000 7fcb716cb700 1 -- 172.21.15.50:0/47561067 wait complete. 2024-04-26T16:24:01.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:01.047+0000 7fcb716cb700 1 librados: shutdown 2024-04-26T16:24:01.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:01.054 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: Running main() from gmock_main.cc 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [----------] Global test environment set-up. 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.Version (1764 ms) 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2041 ms) 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-04-26T16:24:01.077 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.LongName (2990 ms) 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3037 ms) 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2788 ms) 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3030 ms) 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.Exec (2973 ms) 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3039 ms) 2024-04-26T16:24:01.078 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2978 ms) 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2997 ms) 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2028 ms) 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3000 ms) 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [----------] 12 tests from NeoRadosMisc (32665 ms total) 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [----------] Global test environment tear-down 2024-04-26T16:24:01.079 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [==========] 12 tests from 1 test suite ran. (32665 ms total) 2024-04-26T16:24:01.080 INFO:tasks.workunit.client.0.smithi050.stdout: misc: [ PASSED ] 12 tests. 2024-04-26T16:24:01.092 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:01.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:01.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:01.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:01.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:01.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:01.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:01.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:01.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:01.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:01.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:01.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:01.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:01.772 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:01.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:01.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:01.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:00.611251+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.50:0/47561067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:00.611251+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.50:0/47561067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:00.611915+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:00.611915+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.819 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.007957+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.007957+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.010468+0000 mon.a (mon.0) 1354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.010468+0000 mon.a (mon.0) 1354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.012900+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.012900+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.012997+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.012997+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.023047+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.023047+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.054651+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.054651+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.058894+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.058894+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.059317+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.50:0/438790398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.059317+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.50:0/438790398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.065842+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.065842+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.072854+0000 mon.a (mon.0) 1360 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.072854+0000 mon.a (mon.0) 1360 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.324318+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.324318+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.446110+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.446110+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.507609+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.507609+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.510431+0000 mon.a (mon.0) 1361 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:01 smithi190 bash[23868]: audit 2024-04-26T16:24:01.510431+0000 mon.a (mon.0) 1361 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:00.611251+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.50:0/47561067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:00.611251+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.50:0/47561067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:00.611915+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:00.611915+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.007957+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:01.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.007957+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.010468+0000 mon.a (mon.0) 1354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.010468+0000 mon.a (mon.0) 1354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.012900+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.012900+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.012997+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.012997+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.023047+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.023047+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.054651+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.054651+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.058894+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:00.611251+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.50:0/47561067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:00.611251+0000 mon.c (mon.2) 219 : audit [INF] from='client.? 172.21.15.50:0/47561067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:00.611915+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:00.611915+0000 mon.a (mon.0) 1352 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:01.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.007957+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.007957+0000 mon.a (mon.0) 1353 : cluster [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.010468+0000 mon.a (mon.0) 1354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.010468+0000 mon.a (mon.0) 1354 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.012900+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.012900+0000 mon.a (mon.0) 1355 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.012997+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.012997+0000 mon.a (mon.0) 1356 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.023047+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.023047+0000 mon.a (mon.0) 1357 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.054651+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.054651+0000 mon.b (mon.1) 116 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.058894+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.058894+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.059317+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.50:0/438790398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.059317+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.50:0/438790398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.065842+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.065842+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.072854+0000 mon.a (mon.0) 1360 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.072854+0000 mon.a (mon.0) 1360 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.324318+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.324318+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.446110+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.446110+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.507609+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.507609+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.510431+0000 mon.a (mon.0) 1361 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[29234]: audit 2024-04-26T16:24:01.510431+0000 mon.a (mon.0) 1361 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.058894+0000 mon.a (mon.0) 1358 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.059317+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.50:0/438790398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.059317+0000 mon.c (mon.2) 220 : audit [INF] from='client.? 172.21.15.50:0/438790398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.065842+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.065842+0000 mon.a (mon.0) 1359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.072854+0000 mon.a (mon.0) 1360 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.072854+0000 mon.a (mon.0) 1360 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.324318+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.324318+0000 mon.c (mon.2) 221 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:24:01.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.446110+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:01.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.446110+0000 mon.c (mon.2) 222 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:01.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.507609+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.507609+0000 mon.c (mon.2) 223 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.510431+0000 mon.a (mon.0) 1361 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:01 smithi050 bash[21463]: audit 2024-04-26T16:24:01.510431+0000 mon.a (mon.0) 1361 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-04-26T16:24:01.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:01.908 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:01.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:02.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.048 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:02.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.174 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:02.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-26T16:24:02.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.299 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:02.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:02.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-04-26T16:24:02.425 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:02.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:02.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:02.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:02.679 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.501988+0000 mgr.y (mgr.24401) 252 : cluster [DBG] pgmap v244: 518 pgs: 96 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 258 KiB/s wr, 3 op/s 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: cluster 2024-04-26T16:24:01.501988+0000 mgr.y (mgr.24401) 252 : cluster [DBG] pgmap v244: 518 pgs: 96 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 258 KiB/s wr, 3 op/s 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016728+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016728+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016814+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016814+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016886+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016886+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016958+0000 mon.a (mon.0) 1365 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-26T16:24:02.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.016958+0000 mon.a (mon.0) 1365 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: cluster 2024-04-26T16:24:02.021292+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: cluster 2024-04-26T16:24:02.021292+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.042802+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.042802+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.045450+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.045450+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.158591+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.158591+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.160299+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.160299+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.447671+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:02.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:02 smithi190 bash[23868]: audit 2024-04-26T16:24:02.447671+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:02.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.810 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:02.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.501988+0000 mgr.y (mgr.24401) 252 : cluster [DBG] pgmap v244: 518 pgs: 96 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 258 KiB/s wr, 3 op/s 2024-04-26T16:24:02.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: cluster 2024-04-26T16:24:01.501988+0000 mgr.y (mgr.24401) 252 : cluster [DBG] pgmap v244: 518 pgs: 96 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 258 KiB/s wr, 3 op/s 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016728+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016728+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016814+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016814+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016886+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016886+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016958+0000 mon.a (mon.0) 1365 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.016958+0000 mon.a (mon.0) 1365 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: cluster 2024-04-26T16:24:02.021292+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: cluster 2024-04-26T16:24:02.021292+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.042802+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.042802+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.045450+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.045450+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.158591+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.158591+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.160299+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.160299+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.447671+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[21463]: audit 2024-04-26T16:24:02.447671+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.501988+0000 mgr.y (mgr.24401) 252 : cluster [DBG] pgmap v244: 518 pgs: 96 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 258 KiB/s wr, 3 op/s 2024-04-26T16:24:02.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: cluster 2024-04-26T16:24:01.501988+0000 mgr.y (mgr.24401) 252 : cluster [DBG] pgmap v244: 518 pgs: 96 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 258 KiB/s wr, 3 op/s 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016728+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016728+0000 mon.a (mon.0) 1362 : audit [INF] from='client.? 172.21.15.50:0/3226602972' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi050-73193-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016814+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016814+0000 mon.a (mon.0) 1363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi050-73201-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016886+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016886+0000 mon.a (mon.0) 1364 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache", "force_nonempty":""}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016958+0000 mon.a (mon.0) 1365 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.016958+0000 mon.a (mon.0) 1365 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "25"}]': finished 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: cluster 2024-04-26T16:24:02.021292+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: cluster 2024-04-26T16:24:02.021292+0000 mon.a (mon.0) 1366 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.042802+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.042802+0000 mon.b (mon.1) 117 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.045450+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.045450+0000 mon.a (mon.0) 1367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.158591+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.158591+0000 mon.b (mon.1) 118 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.160299+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.160299+0000 mon.a (mon.0) 1368 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.447671+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:02 smithi050 bash[29234]: audit 2024-04-26T16:24:02.447671+0000 mon.c (mon.2) 224 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:02.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:02.935 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:02.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:03.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:03.076 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:03.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:03.198 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:03.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:03.323 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:03.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:03.446 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:03.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:03.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:03.567 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:03.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:03.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:03.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:03.832 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:03.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:03.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:04.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:04.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:04.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:04.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.019204+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.019204+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.019317+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.019317+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.023803+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.023803+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.044110+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.044110+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.046368+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.046368+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.057449+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.057449+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.058597+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.058597+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.104346+0000 mon.c (mon.2) 225 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.104346+0000 mon.c (mon.2) 225 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.118195+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.118195+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.171775+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.171775+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.448664+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: audit 2024-04-26T16:24:03.448664+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.503297+0000 mgr.y (mgr.24401) 253 : cluster [DBG] pgmap v247: 518 pgs: 64 unknown, 32 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-04-26T16:24:04.281 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:04 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.503297+0000 mgr.y (mgr.24401) 253 : cluster [DBG] pgmap v247: 518 pgs: 64 unknown, 32 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-04-26T16:24:04.293 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:04.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:04.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.019204+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.019204+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.019317+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.019317+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.023803+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.023803+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.044110+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.044110+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.046368+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.046368+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.057449+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.057449+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.058597+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.058597+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.104346+0000 mon.c (mon.2) 225 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.104346+0000 mon.c (mon.2) 225 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.118195+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.118195+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.171775+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.171775+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.448664+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: audit 2024-04-26T16:24:03.448664+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.503297+0000 mgr.y (mgr.24401) 253 : cluster [DBG] pgmap v247: 518 pgs: 64 unknown, 32 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-04-26T16:24:04.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.503297+0000 mgr.y (mgr.24401) 253 : cluster [DBG] pgmap v247: 518 pgs: 64 unknown, 32 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.019204+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.019204+0000 mon.a (mon.0) 1369 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73405-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.019317+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.019317+0000 mon.a (mon.0) 1370 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.023803+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.023803+0000 mon.a (mon.0) 1371 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.044110+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.044110+0000 mon.b (mon.1) 119 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.046368+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.046368+0000 mon.a (mon.0) 1372 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.057449+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.057449+0000 mon.b (mon.1) 120 : audit [INF] from='client.? 172.21.15.50:0/3061450153' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.058597+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.058597+0000 mon.a (mon.0) 1373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.104346+0000 mon.c (mon.2) 225 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.104346+0000 mon.c (mon.2) 225 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.118195+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.118195+0000 mon.c (mon.2) 226 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.171775+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.171775+0000 mon.a (mon.0) 1374 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.448664+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: audit 2024-04-26T16:24:03.448664+0000 mon.c (mon.2) 227 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.503297+0000 mgr.y (mgr.24401) 253 : cluster [DBG] pgmap v247: 518 pgs: 64 unknown, 32 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-04-26T16:24:04.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.503297+0000 mgr.y (mgr.24401) 253 : cluster [DBG] pgmap v247: 518 pgs: 64 unknown, 32 creating+peering, 422 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-04-26T16:24:04.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:04.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:04.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:04.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:04.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:04.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:04.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:04.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:04.757 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:24:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:24:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:24:04.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:04.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:04.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:05.051 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:05.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:05.178 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:05.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:05.290 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.303722+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.303722+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.303866+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:03.303866+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.023350+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.023350+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.023489+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]': finished 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.023489+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]': finished 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.028833+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.028833+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.068095+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.068095+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.071203+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.071203+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.074353+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.50:0/1025564474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.074353+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.50:0/1025564474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.074605+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.50:0/1824440060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.074605+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.50:0/1824440060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.079192+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.079192+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.079483+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.079483+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.451647+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.451647+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.638257+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.638257+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.638601+0000 mgr.y (mgr.24401) 254 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.638601+0000 mgr.y (mgr.24401) 254 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.639452+0000 mon.c (mon.2) 232 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.639452+0000 mon.c (mon.2) 232 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.639726+0000 mgr.y (mgr.24401) 255 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.639726+0000 mgr.y (mgr.24401) 255 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.640331+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.640331+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.640662+0000 mgr.y (mgr.24401) 256 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.640662+0000 mgr.y (mgr.24401) 256 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.641192+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.641192+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.641433+0000 mgr.y (mgr.24401) 257 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.641433+0000 mgr.y (mgr.24401) 257 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.643373+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.643373+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.644139+0000 mgr.y (mgr.24401) 258 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.644139+0000 mgr.y (mgr.24401) 258 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.645642+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.645642+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.646368+0000 mgr.y (mgr.24401) 259 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.646368+0000 mgr.y (mgr.24401) 259 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.647353+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.647353+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.294 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.648815+0000 mgr.y (mgr.24401) 260 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.648815+0000 mgr.y (mgr.24401) 260 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.650578+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.650578+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.653927+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.653927+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.657861+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:04.657861+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.794820+0000 osd.0 (osd.0) 151 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.794820+0000 osd.0 (osd.0) 151 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.800796+0000 osd.0 (osd.0) 152 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.800796+0000 osd.0 (osd.0) 152 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.024234+0000 mon.a (mon.0) 1381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.024234+0000 mon.a (mon.0) 1381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:05.026312+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]': finished 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:05.026312+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]': finished 2024-04-26T16:24:05.295 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:05.026423+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:05.026423+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:05.026537+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: audit 2024-04-26T16:24:05.026537+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.032028+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-04-26T16:24:05.296 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:05 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.032028+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-04-26T16:24:05.304 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:05.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:24:05.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.303722+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:05.321 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.303722+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.303866+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:03.303866+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.023350+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.023350+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.023489+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]': finished 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.023489+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]': finished 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.028833+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.028833+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.068095+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.068095+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.071203+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.071203+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.074353+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.50:0/1025564474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.074353+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.50:0/1025564474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.074605+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.50:0/1824440060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.074605+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.50:0/1824440060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.079192+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.079192+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.079483+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.079483+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.451647+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:05.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.451647+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.638257+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.638257+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.638601+0000 mgr.y (mgr.24401) 254 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.638601+0000 mgr.y (mgr.24401) 254 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.639452+0000 mon.c (mon.2) 232 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.639452+0000 mon.c (mon.2) 232 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.639726+0000 mgr.y (mgr.24401) 255 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.639726+0000 mgr.y (mgr.24401) 255 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.640331+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.303722+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.303722+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.303866+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:03.303866+0000 osd.4 (osd.4) 126 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.023350+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.023350+0000 mon.a (mon.0) 1375 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.023489+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]': finished 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.023489+0000 mon.a (mon.0) 1376 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73405-10", "tierpool":"test-rados-api-smithi050-73405-10-cache"}]': finished 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.028833+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.028833+0000 mon.a (mon.0) 1377 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.068095+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.068095+0000 mon.b (mon.1) 121 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.071203+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.071203+0000 mon.a (mon.0) 1378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.074353+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.50:0/1025564474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.074353+0000 mon.c (mon.2) 228 : audit [INF] from='client.? 172.21.15.50:0/1025564474' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.074605+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.50:0/1824440060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.074605+0000 mon.c (mon.2) 229 : audit [INF] from='client.? 172.21.15.50:0/1824440060' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.079192+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.079192+0000 mon.a (mon.0) 1379 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.079483+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.079483+0000 mon.a (mon.0) 1380 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.451647+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.451647+0000 mon.c (mon.2) 230 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.638257+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.326 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.638257+0000 mon.c (mon.2) 231 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.638601+0000 mgr.y (mgr.24401) 254 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.638601+0000 mgr.y (mgr.24401) 254 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.0"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.639452+0000 mon.c (mon.2) 232 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.639452+0000 mon.c (mon.2) 232 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.639726+0000 mgr.y (mgr.24401) 255 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.639726+0000 mgr.y (mgr.24401) 255 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.1"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.640331+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.640331+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.640662+0000 mgr.y (mgr.24401) 256 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.640662+0000 mgr.y (mgr.24401) 256 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.641192+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.641192+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.641433+0000 mgr.y (mgr.24401) 257 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.641433+0000 mgr.y (mgr.24401) 257 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.643373+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.643373+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.644139+0000 mgr.y (mgr.24401) 258 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.644139+0000 mgr.y (mgr.24401) 258 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.645642+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.645642+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.646368+0000 mgr.y (mgr.24401) 259 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.646368+0000 mgr.y (mgr.24401) 259 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.647353+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.647353+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.648815+0000 mgr.y (mgr.24401) 260 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.648815+0000 mgr.y (mgr.24401) 260 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.650578+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.650578+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.653927+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.653927+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.657861+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:04.657861+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.794820+0000 osd.0 (osd.0) 151 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.794820+0000 osd.0 (osd.0) 151 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.800796+0000 osd.0 (osd.0) 152 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.800796+0000 osd.0 (osd.0) 152 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.024234+0000 mon.a (mon.0) 1381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.024234+0000 mon.a (mon.0) 1381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:05.026312+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]': finished 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:05.026312+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]': finished 2024-04-26T16:24:05.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:05.026423+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:05.026423+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:05.026537+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: audit 2024-04-26T16:24:05.026537+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.032028+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.032028+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.640331+0000 mon.c (mon.2) 233 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.640662+0000 mgr.y (mgr.24401) 256 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.640662+0000 mgr.y (mgr.24401) 256 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.2"}]: dispatch 2024-04-26T16:24:05.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.641192+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.641192+0000 mon.c (mon.2) 234 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.641433+0000 mgr.y (mgr.24401) 257 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.641433+0000 mgr.y (mgr.24401) 257 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.3"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.643373+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.643373+0000 mon.c (mon.2) 235 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.644139+0000 mgr.y (mgr.24401) 258 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.644139+0000 mgr.y (mgr.24401) 258 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.4"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.645642+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.645642+0000 mon.c (mon.2) 236 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.646368+0000 mgr.y (mgr.24401) 259 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.646368+0000 mgr.y (mgr.24401) 259 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.5"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.647353+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.647353+0000 mon.c (mon.2) 237 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.648815+0000 mgr.y (mgr.24401) 260 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.648815+0000 mgr.y (mgr.24401) 260 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.6"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.650578+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.650578+0000 mon.c (mon.2) 238 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.653927+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.653927+0000 mon.c (mon.2) 239 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.657861+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:04.657861+0000 mon.c (mon.2) 240 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.794820+0000 osd.0 (osd.0) 151 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.794820+0000 osd.0 (osd.0) 151 : cluster [DBG] 17.4 deep-scrub starts 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.800796+0000 osd.0 (osd.0) 152 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.800796+0000 osd.0 (osd.0) 152 : cluster [DBG] 17.4 deep-scrub ok 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.024234+0000 mon.a (mon.0) 1381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.024234+0000 mon.a (mon.0) 1381 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:05.026312+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]': finished 2024-04-26T16:24:05.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:05.026312+0000 mon.a (mon.0) 1382 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-11", "mode": "writeback"}]': finished 2024-04-26T16:24:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:05.026423+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:05.026423+0000 mon.a (mon.0) 1383 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:05.026537+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: audit 2024-04-26T16:24:05.026537+0000 mon.a (mon.0) 1384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi050-73193-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.032028+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-04-26T16:24:05.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:05 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.032028+0000 mon.a (mon.0) 1385 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2024-04-26T16:24:05.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:05.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:05.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:05.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:05.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:05.826 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:05.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:05.954 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:05.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: Running main() from gmock_main.cc 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [----------] Global test environment set-up. 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3138 ms) 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3045 ms) 2024-04-26T16:24:06.044 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2991 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3027 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2838 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4981 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5003 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5016 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7955 ms) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [----------] 9 tests from LibRadosPools (37994 ms total) 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [----------] Global test environment tear-down 2024-04-26T16:24:06.045 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (37994 ms total) 2024-04-26T16:24:06.046 INFO:tasks.workunit.client.0.smithi050.stdout: api_pool: [ PASSED ] 9 tests. 2024-04-26T16:24:06.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:06.089 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:06.095 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:04.652721+0000 mgr.y (mgr.24401) 261 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:06.095 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:04.652721+0000 mgr.y (mgr.24401) 261 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:06.095 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:04.655560+0000 mgr.y (mgr.24401) 262 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:06.095 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:04.655560+0000 mgr.y (mgr.24401) 262 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:04.659360+0000 mgr.y (mgr.24401) 263 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:04.659360+0000 mgr.y (mgr.24401) 263 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.865495+0000 osd.4 (osd.4) 127 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.865495+0000 osd.4 (osd.4) 127 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.868399+0000 osd.4 (osd.4) 128 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.868399+0000 osd.4 (osd.4) 128 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.885227+0000 osd.3 (osd.3) 161 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.885227+0000 osd.3 (osd.3) 161 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.920254+0000 osd.3 (osd.3) 162 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:04.920254+0000 osd.3 (osd.3) 162 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:05.452653+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:05.452653+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.503990+0000 mgr.y (mgr.24401) 264 : cluster [DBG] pgmap v250: 517 pgs: 128 unknown, 389 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.503990+0000 mgr.y (mgr.24401) 264 : cluster [DBG] pgmap v250: 517 pgs: 128 unknown, 389 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:06.096 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:05.504504+0000 mon.c (mon.2) 242 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:05.504504+0000 mon.c (mon.2) 242 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:05.506733+0000 mon.a (mon.0) 1386 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:05.506733+0000 mon.a (mon.0) 1386 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.506909+0000 osd.7 (osd.7) 165 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.506909+0000 osd.7 (osd.7) 165 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.510963+0000 osd.7 (osd.7) 166 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.510963+0000 osd.7 (osd.7) 166 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.011915+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.011915+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:06.015740+0000 mon.a (mon.0) 1388 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: audit 2024-04-26T16:24:06.015740+0000 mon.a (mon.0) 1388 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.020032+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-04-26T16:24:06.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.020032+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:04.652721+0000 mgr.y (mgr.24401) 261 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:04.652721+0000 mgr.y (mgr.24401) 261 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:04.655560+0000 mgr.y (mgr.24401) 262 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:04.655560+0000 mgr.y (mgr.24401) 262 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:04.659360+0000 mgr.y (mgr.24401) 263 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:04.659360+0000 mgr.y (mgr.24401) 263 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.865495+0000 osd.4 (osd.4) 127 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.865495+0000 osd.4 (osd.4) 127 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:24:06.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.868399+0000 osd.4 (osd.4) 128 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.868399+0000 osd.4 (osd.4) 128 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.885227+0000 osd.3 (osd.3) 161 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.885227+0000 osd.3 (osd.3) 161 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.920254+0000 osd.3 (osd.3) 162 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:04.920254+0000 osd.3 (osd.3) 162 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:05.452653+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:05.452653+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.503990+0000 mgr.y (mgr.24401) 264 : cluster [DBG] pgmap v250: 517 pgs: 128 unknown, 389 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.503990+0000 mgr.y (mgr.24401) 264 : cluster [DBG] pgmap v250: 517 pgs: 128 unknown, 389 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:05.504504+0000 mon.c (mon.2) 242 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:05.504504+0000 mon.c (mon.2) 242 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:05.506733+0000 mon.a (mon.0) 1386 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:05.506733+0000 mon.a (mon.0) 1386 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.506909+0000 osd.7 (osd.7) 165 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.506909+0000 osd.7 (osd.7) 165 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.510963+0000 osd.7 (osd.7) 166 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.510963+0000 osd.7 (osd.7) 166 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.011915+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.011915+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:06.015740+0000 mon.a (mon.0) 1388 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: audit 2024-04-26T16:24:06.015740+0000 mon.a (mon.0) 1388 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.020032+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-04-26T16:24:06.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:06 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.020032+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-04-26T16:24:06.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:06.214 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:06.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:06.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:04.652721+0000 mgr.y (mgr.24401) 261 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:06.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:04.652721+0000 mgr.y (mgr.24401) 261 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.7"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:04.655560+0000 mgr.y (mgr.24401) 262 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:04.655560+0000 mgr.y (mgr.24401) 262 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.8"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:04.659360+0000 mgr.y (mgr.24401) 263 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:04.659360+0000 mgr.y (mgr.24401) 263 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "17.9"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.865495+0000 osd.4 (osd.4) 127 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.865495+0000 osd.4 (osd.4) 127 : cluster [DBG] 17.3 deep-scrub starts 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.868399+0000 osd.4 (osd.4) 128 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.868399+0000 osd.4 (osd.4) 128 : cluster [DBG] 17.3 deep-scrub ok 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.885227+0000 osd.3 (osd.3) 161 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.885227+0000 osd.3 (osd.3) 161 : cluster [DBG] 17.9 deep-scrub starts 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.920254+0000 osd.3 (osd.3) 162 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:04.920254+0000 osd.3 (osd.3) 162 : cluster [DBG] 17.9 deep-scrub ok 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:05.452653+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:05.452653+0000 mon.c (mon.2) 241 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:06.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.503990+0000 mgr.y (mgr.24401) 264 : cluster [DBG] pgmap v250: 517 pgs: 128 unknown, 389 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.503990+0000 mgr.y (mgr.24401) 264 : cluster [DBG] pgmap v250: 517 pgs: 128 unknown, 389 active+clean; 146 MiB data, 814 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:05.504504+0000 mon.c (mon.2) 242 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:05.504504+0000 mon.c (mon.2) 242 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:05.506733+0000 mon.a (mon.0) 1386 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:05.506733+0000 mon.a (mon.0) 1386 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.506909+0000 osd.7 (osd.7) 165 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.506909+0000 osd.7 (osd.7) 165 : cluster [DBG] 17.0 deep-scrub starts 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.510963+0000 osd.7 (osd.7) 166 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.510963+0000 osd.7 (osd.7) 166 : cluster [DBG] 17.0 deep-scrub ok 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.011915+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.011915+0000 mon.a (mon.0) 1387 : cluster [WRN] Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:06.015740+0000 mon.a (mon.0) 1388 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: audit 2024-04-26T16:24:06.015740+0000 mon.a (mon.0) 1388 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "24"}]': finished 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.020032+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-04-26T16:24:06.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.020032+0000 mon.a (mon.0) 1389 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2024-04-26T16:24:06.339 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:06.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:06.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:06.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:06.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:06.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:06.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:06.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:06.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:06.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:06.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:07.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:07.069 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:24:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:24:07.101 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:07.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:07.227 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:07.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:07.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:07.352 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:07.353 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:07.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.328285+0000 osd.6 (osd.6) 95 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:24:07.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.328285+0000 osd.6 (osd.6) 95 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:24:07.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.329579+0000 osd.6 (osd.6) 96 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:24:07.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.329579+0000 osd.6 (osd.6) 96 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.826542+0000 osd.0 (osd.0) 153 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.826542+0000 osd.0 (osd.0) 153 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.830950+0000 osd.0 (osd.0) 154 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.830950+0000 osd.0 (osd.0) 154 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.833941+0000 osd.4 (osd.4) 129 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.833941+0000 osd.4 (osd.4) 129 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.835334+0000 osd.4 (osd.4) 130 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:05.835334+0000 osd.4 (osd.4) 130 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.227503+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.227503+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.229113+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.229113+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.454095+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:07.359 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.454095+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.545340+0000 osd.7 (osd.7) 167 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.545340+0000 osd.7 (osd.7) 167 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.546497+0000 osd.7 (osd.7) 168 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.546497+0000 osd.7 (osd.7) 168 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.643255+0000 mgr.y (mgr.24401) 265 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:06.643255+0000 mgr.y (mgr.24401) 265 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.018607+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.018607+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:07.023405+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: cluster 2024-04-26T16:24:07.023405+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.047047+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.047047+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.048739+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.048739+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.051252+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.50:0/3796988213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.051252+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.50:0/3796988213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.052146+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.50:0/3241106947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.052146+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.50:0/3241106947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.056054+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.056054+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.056928+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[21463]: audit 2024-04-26T16:24:07.056928+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.328285+0000 osd.6 (osd.6) 95 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.328285+0000 osd.6 (osd.6) 95 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.329579+0000 osd.6 (osd.6) 96 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.329579+0000 osd.6 (osd.6) 96 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:24:07.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.826542+0000 osd.0 (osd.0) 153 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.826542+0000 osd.0 (osd.0) 153 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.830950+0000 osd.0 (osd.0) 154 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.830950+0000 osd.0 (osd.0) 154 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.833941+0000 osd.4 (osd.4) 129 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.833941+0000 osd.4 (osd.4) 129 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.835334+0000 osd.4 (osd.4) 130 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:05.835334+0000 osd.4 (osd.4) 130 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.227503+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.227503+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.229113+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.229113+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.454095+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.454095+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.545340+0000 osd.7 (osd.7) 167 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:24:07.362 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.545340+0000 osd.7 (osd.7) 167 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.546497+0000 osd.7 (osd.7) 168 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.546497+0000 osd.7 (osd.7) 168 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.643255+0000 mgr.y (mgr.24401) 265 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:06.643255+0000 mgr.y (mgr.24401) 265 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.018607+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.018607+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:07.023405+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: cluster 2024-04-26T16:24:07.023405+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.047047+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.047047+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.048739+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.048739+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.051252+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.50:0/3796988213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.051252+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.50:0/3796988213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.363 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.052146+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.50:0/3241106947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.052146+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.50:0/3241106947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.056054+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.056054+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.056928+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:07 smithi050 bash[29234]: audit 2024-04-26T16:24:07.056928+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:07.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:07.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.328285+0000 osd.6 (osd.6) 95 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.328285+0000 osd.6 (osd.6) 95 : cluster [DBG] 17.5 deep-scrub starts 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.329579+0000 osd.6 (osd.6) 96 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.329579+0000 osd.6 (osd.6) 96 : cluster [DBG] 17.5 deep-scrub ok 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.826542+0000 osd.0 (osd.0) 153 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.826542+0000 osd.0 (osd.0) 153 : cluster [DBG] 17.7 deep-scrub starts 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.830950+0000 osd.0 (osd.0) 154 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.830950+0000 osd.0 (osd.0) 154 : cluster [DBG] 17.7 deep-scrub ok 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.833941+0000 osd.4 (osd.4) 129 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.833941+0000 osd.4 (osd.4) 129 : cluster [DBG] 17.2 deep-scrub starts 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.835334+0000 osd.4 (osd.4) 130 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:05.835334+0000 osd.4 (osd.4) 130 : cluster [DBG] 17.2 deep-scrub ok 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.227503+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.227503+0000 mon.a (mon.0) 1390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.229113+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.229113+0000 mon.b (mon.1) 122 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.454095+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.454095+0000 mon.c (mon.2) 243 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.545340+0000 osd.7 (osd.7) 167 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.545340+0000 osd.7 (osd.7) 167 : cluster [DBG] 17.8 deep-scrub starts 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.546497+0000 osd.7 (osd.7) 168 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.546497+0000 osd.7 (osd.7) 168 : cluster [DBG] 17.8 deep-scrub ok 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.643255+0000 mgr.y (mgr.24401) 265 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:06.643255+0000 mgr.y (mgr.24401) 265 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.018607+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.018607+0000 mon.a (mon.0) 1391 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:07.494 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:07.023405+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: cluster 2024-04-26T16:24:07.023405+0000 mon.a (mon.0) 1392 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.047047+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.047047+0000 mon.a (mon.0) 1393 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.048739+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.048739+0000 mon.b (mon.1) 123 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.051252+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.50:0/3796988213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.051252+0000 mon.b (mon.1) 124 : audit [INF] from='client.? 172.21.15.50:0/3796988213' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.052146+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.50:0/3241106947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.052146+0000 mon.b (mon.1) 125 : audit [INF] from='client.? 172.21.15.50:0/3241106947' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.056054+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.056054+0000 mon.a (mon.0) 1394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.056928+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:07 smithi190 bash[23868]: audit 2024-04-26T16:24:07.056928+0000 mon.a (mon.0) 1395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:07.507 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:07.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:07.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:07.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:07.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:07.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:07.897 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:07.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:08.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:08.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] Global test environment set-up. 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-04-26T16:24:08.073 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: 2024-04-26T16:23:28.116+0000 7f5541592880 -1 failed for service _ceph-mon._tcp 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: 2024-04-26T16:23:28.116+0000 7f5541592880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (79 ms) 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2029 ms) 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2108 ms total) 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: started 0x558ae5cd3300 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: started 0x558ae5f974b0 2024-04-26T16:24:08.074 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: started 2 aios 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: waiting 0x558ae5cd3300 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: waiting 0x558ae5f974b0 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: done. 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (6053 ms) 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (6053 ms total) 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.Exec (259 ms) 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (13 ms) 2024-04-26T16:24:08.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (25 ms) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4522 ms) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (24512 ms) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] 8 tests from LibRadosMisc (29331 ms total) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [----------] Global test environment tear-down 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (40021 ms total) 2024-04-26T16:24:08.076 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc: [ PASSED ] 12 tests. 2024-04-26T16:24:08.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:08.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.310249+0000 osd.6 (osd.6) 97 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:24:08.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.310249+0000 osd.6 (osd.6) 97 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.311655+0000 osd.6 (osd.6) 98 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.311655+0000 osd.6 (osd.6) 98 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.868931+0000 osd.0 (osd.0) 155 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.868931+0000 osd.0 (osd.0) 155 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.908060+0000 osd.0 (osd.0) 156 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:06.908060+0000 osd.0 (osd.0) 156 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:07.455318+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:07.455318+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:07.504606+0000 mgr.y (mgr.24401) 266 : cluster [DBG] pgmap v253: 453 pgs: 32 creating+peering, 6 active+clean+snaptrim, 2 peering, 3 active+clean+snaptrim_wait, 64 unknown, 346 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:07.504606+0000 mgr.y (mgr.24401) 266 : cluster [DBG] pgmap v253: 453 pgs: 32 creating+peering, 6 active+clean+snaptrim, 2 peering, 3 active+clean+snaptrim_wait, 64 unknown, 346 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.022461+0000 mon.a (mon.0) 1396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.022461+0000 mon.a (mon.0) 1396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:08.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.025228+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.025228+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.025353+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.025353+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.025493+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.025493+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.031891+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.031891+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.043652+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.043652+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.054314+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.281 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[21463]: audit 2024-04-26T16:24:08.054314+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.310249+0000 osd.6 (osd.6) 97 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.310249+0000 osd.6 (osd.6) 97 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.311655+0000 osd.6 (osd.6) 98 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.311655+0000 osd.6 (osd.6) 98 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.868931+0000 osd.0 (osd.0) 155 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.868931+0000 osd.0 (osd.0) 155 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.908060+0000 osd.0 (osd.0) 156 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:06.908060+0000 osd.0 (osd.0) 156 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:24:08.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:07.455318+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:07.455318+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:07.504606+0000 mgr.y (mgr.24401) 266 : cluster [DBG] pgmap v253: 453 pgs: 32 creating+peering, 6 active+clean+snaptrim, 2 peering, 3 active+clean+snaptrim_wait, 64 unknown, 346 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:07.504606+0000 mgr.y (mgr.24401) 266 : cluster [DBG] pgmap v253: 453 pgs: 32 creating+peering, 6 active+clean+snaptrim, 2 peering, 3 active+clean+snaptrim_wait, 64 unknown, 346 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.022461+0000 mon.a (mon.0) 1396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.022461+0000 mon.a (mon.0) 1396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.025228+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.025228+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.025353+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.025353+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.025493+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.025493+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.031891+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.031891+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-04-26T16:24:08.283 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.043652+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.043652+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.054314+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:08 smithi050 bash[29234]: audit 2024-04-26T16:24:08.054314+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:08.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.310249+0000 osd.6 (osd.6) 97 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:24:08.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.310249+0000 osd.6 (osd.6) 97 : cluster [DBG] 17.6 deep-scrub starts 2024-04-26T16:24:08.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.311655+0000 osd.6 (osd.6) 98 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.311655+0000 osd.6 (osd.6) 98 : cluster [DBG] 17.6 deep-scrub ok 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.868931+0000 osd.0 (osd.0) 155 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.868931+0000 osd.0 (osd.0) 155 : cluster [DBG] 17.1 deep-scrub starts 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.908060+0000 osd.0 (osd.0) 156 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:06.908060+0000 osd.0 (osd.0) 156 : cluster [DBG] 17.1 deep-scrub ok 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:07.455318+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:07.455318+0000 mon.c (mon.2) 244 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:07.504606+0000 mgr.y (mgr.24401) 266 : cluster [DBG] pgmap v253: 453 pgs: 32 creating+peering, 6 active+clean+snaptrim, 2 peering, 3 active+clean+snaptrim_wait, 64 unknown, 346 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:07.504606+0000 mgr.y (mgr.24401) 266 : cluster [DBG] pgmap v253: 453 pgs: 32 creating+peering, 6 active+clean+snaptrim, 2 peering, 3 active+clean+snaptrim_wait, 64 unknown, 346 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 18 MiB/s wr, 7 op/s 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.022461+0000 mon.a (mon.0) 1396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.022461+0000 mon.a (mon.0) 1396 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.025228+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.025228+0000 mon.a (mon.0) 1397 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-11"}]': finished 2024-04-26T16:24:08.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.025353+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.025353+0000 mon.a (mon.0) 1398 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi050-73201-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.025493+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.025493+0000 mon.a (mon.0) 1399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi050-73193-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.031891+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.031891+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.043652+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.043652+0000 mon.b (mon.1) 126 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.054314+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:08 smithi190 bash[23868]: audit 2024-04-26T16:24:08.054314+0000 mon.a (mon.0) 1401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:08.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:08.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:08.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:08.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:08.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:08.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:08.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:08.928 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:08.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:09.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:09.075 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:09.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.260797+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.260797+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.260985+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: cluster 2024-04-26T16:24:08.260985+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:08.456182+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:08.456182+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:09.029126+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:09.029126+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.037487+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.037487+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:09.041228+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:09.041228+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.188 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:09.051791+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:09 smithi190 bash[23868]: audit 2024-04-26T16:24:09.051791+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.199 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:09.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:09.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.260797+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.260797+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.260985+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: cluster 2024-04-26T16:24:08.260985+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:08.456182+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:08.456182+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:09.029126+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:09.029126+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.037487+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.037487+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:09.041228+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:09.041228+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:09.051791+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[29234]: audit 2024-04-26T16:24:09.051791+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.260797+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.260797+0000 osd.5 (osd.5) 151 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.260985+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: cluster 2024-04-26T16:24:08.260985+0000 osd.5 (osd.5) 152 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:08.456182+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:08.456182+0000 mon.c (mon.2) 245 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:09.029126+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:09.029126+0000 mon.a (mon.0) 1402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:09.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.037487+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-04-26T16:24:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.037487+0000 mon.a (mon.0) 1403 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2024-04-26T16:24:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:09.041228+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:09.041228+0000 mon.b (mon.1) 127 : audit [INF] from='client.? 172.21.15.50:0/1108064423' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:09.051791+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:09 smithi050 bash[21463]: audit 2024-04-26T16:24:09.051791+0000 mon.a (mon.0) 1404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]: dispatch 2024-04-26T16:24:09.388 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:09.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:09.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:09.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:09.717 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:09.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:09.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:09.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:09.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:09.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:10.055 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: Running main() from gmock_main.cc 2024-04-26T16:24:10.055 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-04-26T16:24:10.055 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [----------] Global test environment set-up. 2024-04-26T16:24:10.055 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-04-26T16:24:10.055 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2770 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3046 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2998 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3008 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2835 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2974 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3020 ms) 2024-04-26T16:24:10.056 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2984 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3030 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2996 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3001 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2996 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2983 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3006 ms) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41647 ms total) 2024-04-26T16:24:10.057 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: 2024-04-26T16:24:10.058 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [----------] Global test environment tear-down 2024-04-26T16:24:10.058 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41648 ms total) 2024-04-26T16:24:10.058 INFO:tasks.workunit.client.0.smithi050.stdout: read_operations: [ PASSED ] 14 tests. 2024-04-26T16:24:10.064 INFO:tasks.workunit.client.0.smithi050.stdout: io: Running main() from gmock_main.cc 2024-04-26T16:24:10.064 INFO:tasks.workunit.client.0.smithi050.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [----------] Global test environment set-up. 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [----------] 14 tests from NeoRadosIo 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.Limits (2829 ms) 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3041 ms) 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.ReadOp (3005 ms) 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.SparseRead (3018 ms) 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2849 ms) 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2955 ms) 2024-04-26T16:24:10.065 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3023 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2996 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2996 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.Trunc (3027 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.Remove (2974 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3020 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.RmXattr (2967 ms) 2024-04-26T16:24:10.066 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-04-26T16:24:10.067 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3024 ms) 2024-04-26T16:24:10.067 INFO:tasks.workunit.client.0.smithi050.stdout: io: [----------] 14 tests from NeoRadosIo (41724 ms total) 2024-04-26T16:24:10.067 INFO:tasks.workunit.client.0.smithi050.stdout: io: 2024-04-26T16:24:10.067 INFO:tasks.workunit.client.0.smithi050.stdout: io: [----------] Global test environment tear-down 2024-04-26T16:24:10.067 INFO:tasks.workunit.client.0.smithi050.stdout: io: [==========] 14 tests from 1 test suite ran. (41724 ms total) 2024-04-26T16:24:10.067 INFO:tasks.workunit.client.0.smithi050.stdout: io: [ PASSED ] 14 tests. 2024-04-26T16:24:10.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:10.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:10.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:10.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:10.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:10.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:10.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:10.401 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:10.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:10.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:09.457381+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:10.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:09.457381+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:10.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.505494+0000 mgr.y (mgr.24401) 267 : cluster [DBG] pgmap v256: 380 pgs: 32 creating+peering, 2 active+clean+snaptrim, 1 peering, 2 active+clean+snaptrim_wait, 64 unknown, 279 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 2 op/s 2024-04-26T16:24:10.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.505494+0000 mgr.y (mgr.24401) 267 : cluster [DBG] pgmap v256: 380 pgs: 32 creating+peering, 2 active+clean+snaptrim, 1 peering, 2 active+clean+snaptrim_wait, 64 unknown, 279 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 2 op/s 2024-04-26T16:24:10.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:09.506015+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:09.506015+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:09.507342+0000 mon.a (mon.0) 1405 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:09.507342+0000 mon.a (mon.0) 1405 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.595943+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:10.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.595943+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:10.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.596084+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:10.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:09.596084+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:10.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.033419+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:10.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.033419+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:10.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.033512+0000 mon.a (mon.0) 1407 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-26T16:24:10.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.033512+0000 mon.a (mon.0) 1407 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-26T16:24:10.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:10.039166+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-04-26T16:24:10.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: cluster 2024-04-26T16:24:10.039166+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-04-26T16:24:10.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.043359+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.50:0/4068896200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.043359+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.50:0/4068896200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.045433+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.045433+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.046931+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.50:0/826194399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.046931+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.50:0/826194399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.062398+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.062398+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.062925+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.062925+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.063552+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.063552+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.098563+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.098563+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.099800+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.099800+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.100159+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:10.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:10 smithi190 bash[23868]: audit 2024-04-26T16:24:10.100159+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:10.510 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:10.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:09.457381+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:09.457381+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.505494+0000 mgr.y (mgr.24401) 267 : cluster [DBG] pgmap v256: 380 pgs: 32 creating+peering, 2 active+clean+snaptrim, 1 peering, 2 active+clean+snaptrim_wait, 64 unknown, 279 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 2 op/s 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.505494+0000 mgr.y (mgr.24401) 267 : cluster [DBG] pgmap v256: 380 pgs: 32 creating+peering, 2 active+clean+snaptrim, 1 peering, 2 active+clean+snaptrim_wait, 64 unknown, 279 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 2 op/s 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:09.506015+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:09.506015+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:09.507342+0000 mon.a (mon.0) 1405 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:09.507342+0000 mon.a (mon.0) 1405 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.595943+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:10.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.595943+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.596084+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:09.596084+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.033419+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.033419+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.033512+0000 mon.a (mon.0) 1407 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.033512+0000 mon.a (mon.0) 1407 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:10.039166+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-04-26T16:24:10.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:09.457381+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:09.457381+0000 mon.c (mon.2) 246 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.505494+0000 mgr.y (mgr.24401) 267 : cluster [DBG] pgmap v256: 380 pgs: 32 creating+peering, 2 active+clean+snaptrim, 1 peering, 2 active+clean+snaptrim_wait, 64 unknown, 279 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 2 op/s 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.505494+0000 mgr.y (mgr.24401) 267 : cluster [DBG] pgmap v256: 380 pgs: 32 creating+peering, 2 active+clean+snaptrim, 1 peering, 2 active+clean+snaptrim_wait, 64 unknown, 279 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 18 MiB/s wr, 2 op/s 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:09.506015+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:09.506015+0000 mon.c (mon.2) 247 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:09.507342+0000 mon.a (mon.0) 1405 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:09.507342+0000 mon.a (mon.0) 1405 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.595943+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.595943+0000 osd.1 (osd.1) 155 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.596084+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:10.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:09.596084+0000 osd.1 (osd.1) 156 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.033419+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.033419+0000 mon.a (mon.0) 1406 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi050-73420-10"}]': finished 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.033512+0000 mon.a (mon.0) 1407 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.033512+0000 mon.a (mon.0) 1407 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "23"}]': finished 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:10.039166+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: cluster 2024-04-26T16:24:10.039166+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.043359+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.50:0/4068896200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.043359+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.50:0/4068896200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.045433+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.045433+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.046931+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.50:0/826194399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.046931+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.50:0/826194399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.062398+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.062398+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.062925+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.062925+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.063552+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.063552+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.098563+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: cluster 2024-04-26T16:24:10.039166+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.043359+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.50:0/4068896200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.043359+0000 mon.c (mon.2) 248 : audit [INF] from='client.? 172.21.15.50:0/4068896200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.045433+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.045433+0000 mon.b (mon.1) 128 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.046931+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.50:0/826194399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.046931+0000 mon.b (mon.1) 129 : audit [INF] from='client.? 172.21.15.50:0/826194399' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.062398+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.062398+0000 mon.a (mon.0) 1409 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.062925+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.062925+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.063552+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.063552+0000 mon.a (mon.0) 1411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.098563+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.098563+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.099800+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.099800+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.100159+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[29234]: audit 2024-04-26T16:24:10.100159+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.098563+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.099800+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.099800+0000 mon.a (mon.0) 1413 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.100159+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:10.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:10 smithi050 bash[21463]: audit 2024-04-26T16:24:10.100159+0000 mon.a (mon.0) 1414 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:10.640 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:10.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:10.772 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:10.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:10.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:10.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:11.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:11.035 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:11.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:11.148 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:10.458692+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:11.148 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:10.458692+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:11.148 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: cluster 2024-04-26T16:24:11.015007+0000 mon.a (mon.0) 1415 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:11.148 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: cluster 2024-04-26T16:24:11.015007+0000 mon.a (mon.0) 1415 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:11.148 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.018598+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.148 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.018598+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.149 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.018939+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.149 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.018939+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.149 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.019077+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.149 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.019077+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.149 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.019324+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:11.149 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.019324+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:11.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: cluster 2024-04-26T16:24:11.026701+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-04-26T16:24:11.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: cluster 2024-04-26T16:24:11.026701+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-04-26T16:24:11.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.029665+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:11.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[29234]: audit 2024-04-26T16:24:11.029665+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:11.150 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:10.458692+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:10.458692+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: cluster 2024-04-26T16:24:11.015007+0000 mon.a (mon.0) 1415 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: cluster 2024-04-26T16:24:11.015007+0000 mon.a (mon.0) 1415 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.018598+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.018598+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.018939+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.018939+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.019077+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.019077+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.019324+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.019324+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: cluster 2024-04-26T16:24:11.026701+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: cluster 2024-04-26T16:24:11.026701+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-04-26T16:24:11.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.029665+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:11.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:11 smithi050 bash[21463]: audit 2024-04-26T16:24:11.029665+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:11.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:11.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:11.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:11.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:10.458692+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:10.458692+0000 mon.c (mon.2) 249 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: cluster 2024-04-26T16:24:11.015007+0000 mon.a (mon.0) 1415 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: cluster 2024-04-26T16:24:11.015007+0000 mon.a (mon.0) 1415 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.018598+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.018598+0000 mon.a (mon.0) 1416 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.018939+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.018939+0000 mon.a (mon.0) 1417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi050-73193-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.019077+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.019077+0000 mon.a (mon.0) 1418 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi050-73201-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.019324+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.019324+0000 mon.a (mon.0) 1419 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: cluster 2024-04-26T16:24:11.026701+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: cluster 2024-04-26T16:24:11.026701+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.029665+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:11.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:11 smithi190 bash[23868]: audit 2024-04-26T16:24:11.029665+0000 mon.a (mon.0) 1421 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:11.416 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:11.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:11.544 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:11.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:11.675 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:11.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:11.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:11.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:11.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:12.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:12.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:12.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:12.217 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:12.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:12.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: audit 2024-04-26T16:24:11.237986+0000 mon.a (mon.0) 1422 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: audit 2024-04-26T16:24:11.237986+0000 mon.a (mon.0) 1422 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: audit 2024-04-26T16:24:11.239797+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: audit 2024-04-26T16:24:11.239797+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: audit 2024-04-26T16:24:11.460308+0000 mon.c (mon.2) 251 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: audit 2024-04-26T16:24:11.460308+0000 mon.c (mon.2) 251 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: cluster 2024-04-26T16:24:11.509287+0000 mgr.y (mgr.24401) 268 : cluster [DBG] pgmap v259: 412 pgs: 1 active+clean+snaptrim, 2 clean+premerge+peered, 409 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s wr, 25 op/s 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: cluster 2024-04-26T16:24:11.509287+0000 mgr.y (mgr.24401) 268 : cluster [DBG] pgmap v259: 412 pgs: 1 active+clean+snaptrim, 2 clean+premerge+peered, 409 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s wr, 25 op/s 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: cluster 2024-04-26T16:24:12.050522+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[29234]: cluster 2024-04-26T16:24:12.050522+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: audit 2024-04-26T16:24:11.237986+0000 mon.a (mon.0) 1422 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:12.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: audit 2024-04-26T16:24:11.237986+0000 mon.a (mon.0) 1422 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: audit 2024-04-26T16:24:11.239797+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: audit 2024-04-26T16:24:11.239797+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: audit 2024-04-26T16:24:11.460308+0000 mon.c (mon.2) 251 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: audit 2024-04-26T16:24:11.460308+0000 mon.c (mon.2) 251 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: cluster 2024-04-26T16:24:11.509287+0000 mgr.y (mgr.24401) 268 : cluster [DBG] pgmap v259: 412 pgs: 1 active+clean+snaptrim, 2 clean+premerge+peered, 409 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s wr, 25 op/s 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: cluster 2024-04-26T16:24:11.509287+0000 mgr.y (mgr.24401) 268 : cluster [DBG] pgmap v259: 412 pgs: 1 active+clean+snaptrim, 2 clean+premerge+peered, 409 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s wr, 25 op/s 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: cluster 2024-04-26T16:24:12.050522+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-04-26T16:24:12.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:12 smithi050 bash[21463]: cluster 2024-04-26T16:24:12.050522+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-04-26T16:24:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:12.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: audit 2024-04-26T16:24:11.237986+0000 mon.a (mon.0) 1422 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: audit 2024-04-26T16:24:11.237986+0000 mon.a (mon.0) 1422 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: audit 2024-04-26T16:24:11.239797+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: audit 2024-04-26T16:24:11.239797+0000 mon.c (mon.2) 250 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: audit 2024-04-26T16:24:11.460308+0000 mon.c (mon.2) 251 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: audit 2024-04-26T16:24:11.460308+0000 mon.c (mon.2) 251 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: cluster 2024-04-26T16:24:11.509287+0000 mgr.y (mgr.24401) 268 : cluster [DBG] pgmap v259: 412 pgs: 1 active+clean+snaptrim, 2 clean+premerge+peered, 409 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s wr, 25 op/s 2024-04-26T16:24:12.458 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: cluster 2024-04-26T16:24:11.509287+0000 mgr.y (mgr.24401) 268 : cluster [DBG] pgmap v259: 412 pgs: 1 active+clean+snaptrim, 2 clean+premerge+peered, 409 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s wr, 25 op/s 2024-04-26T16:24:12.459 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: cluster 2024-04-26T16:24:12.050522+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-04-26T16:24:12.459 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:12 smithi190 bash[23868]: cluster 2024-04-26T16:24:12.050522+0000 mon.a (mon.0) 1423 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2024-04-26T16:24:12.475 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:12.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:12.601 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:12.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:12.732 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:12.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:12.862 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:12.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:12.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:12.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:12.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:13.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:13.118 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:13.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:13.250 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:13.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:13.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:13.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:12.462142+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:12.462142+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.027590+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.027590+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: cluster 2024-04-26T16:24:13.036426+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: cluster 2024-04-26T16:24:13.036426+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.038910+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.50:0/2111160627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.038910+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.50:0/2111160627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.039946+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.039946+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.040309+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.498 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[29234]: audit 2024-04-26T16:24:13.040309+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:12.462142+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:12.462142+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.027590+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.027590+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: cluster 2024-04-26T16:24:13.036426+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: cluster 2024-04-26T16:24:13.036426+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.038910+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.50:0/2111160627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.038910+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.50:0/2111160627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.039946+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.039946+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.040309+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.500 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:13 smithi050 bash[21463]: audit 2024-04-26T16:24:13.040309+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:13.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:13.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:12.462142+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:13.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:12.462142+0000 mon.c (mon.2) 252 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:13.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.027590+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.027590+0000 mon.a (mon.0) 1424 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi050-73420-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: cluster 2024-04-26T16:24:13.036426+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: cluster 2024-04-26T16:24:13.036426+0000 mon.a (mon.0) 1425 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.038910+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.50:0/2111160627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.038910+0000 mon.c (mon.2) 253 : audit [INF] from='client.? 172.21.15.50:0/2111160627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.039946+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.039946+0000 mon.a (mon.0) 1426 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.040309+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:13 smithi190 bash[23868]: audit 2024-04-26T16:24:13.040309+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:13.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:13.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:13.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:13.766 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:13.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:13.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:13.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:13.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:13.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:14.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:14.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:14.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:14.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:14.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:13.463792+0000 mon.c (mon.2) 254 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:13.463792+0000 mon.c (mon.2) 254 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: cluster 2024-04-26T16:24:13.510224+0000 mgr.y (mgr.24401) 269 : cluster [DBG] pgmap v262: 387 pgs: 72 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: cluster 2024-04-26T16:24:13.510224+0000 mgr.y (mgr.24401) 269 : cluster [DBG] pgmap v262: 387 pgs: 72 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:13.510740+0000 mon.c (mon.2) 255 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:13.510740+0000 mon.c (mon.2) 255 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:13.513126+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:13.513126+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:14.030853+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:14.030853+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:14.031072+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:14.031072+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:14.031228+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-26T16:24:14.365 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: audit 2024-04-26T16:24:14.031228+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: cluster 2024-04-26T16:24:14.038756+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21463]: cluster 2024-04-26T16:24:14.038756+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:13.463792+0000 mon.c (mon.2) 254 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:13.463792+0000 mon.c (mon.2) 254 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: cluster 2024-04-26T16:24:13.510224+0000 mgr.y (mgr.24401) 269 : cluster [DBG] pgmap v262: 387 pgs: 72 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: cluster 2024-04-26T16:24:13.510224+0000 mgr.y (mgr.24401) 269 : cluster [DBG] pgmap v262: 387 pgs: 72 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:13.510740+0000 mon.c (mon.2) 255 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:13.510740+0000 mon.c (mon.2) 255 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:13.513126+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:13.513126+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:14.030853+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:14.030853+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:14.031072+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:14.031072+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:14.031228+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-26T16:24:14.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: audit 2024-04-26T16:24:14.031228+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-26T16:24:14.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: cluster 2024-04-26T16:24:14.038756+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-04-26T16:24:14.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[29234]: cluster 2024-04-26T16:24:14.038756+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-04-26T16:24:14.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:14.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:14.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:14.539 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:13.463792+0000 mon.c (mon.2) 254 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:13.463792+0000 mon.c (mon.2) 254 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: cluster 2024-04-26T16:24:13.510224+0000 mgr.y (mgr.24401) 269 : cluster [DBG] pgmap v262: 387 pgs: 72 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: cluster 2024-04-26T16:24:13.510224+0000 mgr.y (mgr.24401) 269 : cluster [DBG] pgmap v262: 387 pgs: 72 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:13.510740+0000 mon.c (mon.2) 255 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:13.510740+0000 mon.c (mon.2) 255 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.594 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:13.513126+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:13.513126+0000 mon.a (mon.0) 1428 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:14.030853+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:14.030853+0000 mon.a (mon.0) 1429 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi050-73201-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:14.031072+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:14.031072+0000 mon.a (mon.0) 1430 : audit [INF] from='client.? 172.21.15.50:0/2703831585' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi050-73193-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:14.031228+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: audit 2024-04-26T16:24:14.031228+0000 mon.a (mon.0) 1431 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "22"}]': finished 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: cluster 2024-04-26T16:24:14.038756+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-04-26T16:24:14.595 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:14 smithi190 bash[23868]: cluster 2024-04-26T16:24:14.038756+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2024-04-26T16:24:14.626 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:24:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:24:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:24:14.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:14.669 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:14.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:14.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:14.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:14.927 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:14.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:15.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:15.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:15.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:15.210 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:15.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:15.338 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:15.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[29234]: audit 2024-04-26T16:24:14.465355+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:15.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[29234]: audit 2024-04-26T16:24:14.465355+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:15.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[29234]: cluster 2024-04-26T16:24:15.043655+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-04-26T16:24:15.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[29234]: cluster 2024-04-26T16:24:15.043655+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-04-26T16:24:15.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[21463]: audit 2024-04-26T16:24:14.465355+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:15.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[21463]: audit 2024-04-26T16:24:14.465355+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:15.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[21463]: cluster 2024-04-26T16:24:15.043655+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-04-26T16:24:15.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:15 smithi050 bash[21463]: cluster 2024-04-26T16:24:15.043655+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-04-26T16:24:15.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:15.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:15 smithi190 bash[23868]: audit 2024-04-26T16:24:14.465355+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:15.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:15 smithi190 bash[23868]: audit 2024-04-26T16:24:14.465355+0000 mon.c (mon.2) 256 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:15.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:15 smithi190 bash[23868]: cluster 2024-04-26T16:24:15.043655+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-04-26T16:24:15.453 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:15 smithi190 bash[23868]: cluster 2024-04-26T16:24:15.043655+0000 mon.a (mon.0) 1433 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2024-04-26T16:24:15.467 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:15.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:15.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:15.732 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:15.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:15.863 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:15.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:15.992 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:15.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:16.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:16.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:16.248 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:16.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:16.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:15.281928+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:15.281928+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:15.284102+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:15.284102+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:15.467048+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:15.467048+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: cluster 2024-04-26T16:24:15.511323+0000 mgr.y (mgr.24401) 270 : cluster [DBG] pgmap v265: 355 pgs: 40 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: cluster 2024-04-26T16:24:15.511323+0000 mgr.y (mgr.24401) 270 : cluster [DBG] pgmap v265: 355 pgs: 40 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.024084+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.024084+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: cluster 2024-04-26T16:24:16.033651+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: cluster 2024-04-26T16:24:16.033651+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.033715+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.033715+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:15.281928+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:15.281928+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:15.284102+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:15.284102+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:15.467048+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:15.467048+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: cluster 2024-04-26T16:24:15.511323+0000 mgr.y (mgr.24401) 270 : cluster [DBG] pgmap v265: 355 pgs: 40 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: cluster 2024-04-26T16:24:15.511323+0000 mgr.y (mgr.24401) 270 : cluster [DBG] pgmap v265: 355 pgs: 40 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.024084+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.024084+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: cluster 2024-04-26T16:24:16.033651+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: cluster 2024-04-26T16:24:16.033651+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.033715+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.033715+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.042459+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.50:0/3767464243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.042459+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.50:0/3767464243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.048542+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.048542+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.050738+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.050738+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.053106+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[29234]: audit 2024-04-26T16:24:16.053106+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.042459+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.50:0/3767464243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.042459+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.50:0/3767464243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.048542+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.048542+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.050738+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.050738+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.053106+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:16 smithi050 bash[21463]: audit 2024-04-26T16:24:16.053106+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.382 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:16.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:16.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:15.281928+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:15.281928+0000 mon.a (mon.0) 1434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:15.284102+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:15.284102+0000 mon.b (mon.1) 130 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:15.467048+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:15.467048+0000 mon.c (mon.2) 257 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: cluster 2024-04-26T16:24:15.511323+0000 mgr.y (mgr.24401) 270 : cluster [DBG] pgmap v265: 355 pgs: 40 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:16.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: cluster 2024-04-26T16:24:15.511323+0000 mgr.y (mgr.24401) 270 : cluster [DBG] pgmap v265: 355 pgs: 40 unknown, 1 clean+premerge+peered, 314 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.024084+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.024084+0000 mon.a (mon.0) 1435 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: cluster 2024-04-26T16:24:16.033651+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: cluster 2024-04-26T16:24:16.033651+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.033715+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.033715+0000 mon.b (mon.1) 131 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.042459+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.50:0/3767464243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.042459+0000 mon.b (mon.1) 132 : audit [INF] from='client.? 172.21.15.50:0/3767464243' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.048542+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.048542+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.050738+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.050738+0000 mon.a (mon.0) 1438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.053106+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[23868]: audit 2024-04-26T16:24:16.053106+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:16.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:16.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:16.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:16.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:16.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:16.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:16.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:16.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:24:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:24:16.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:16.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:16.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:17.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:17.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:17.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:17.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:17.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: cluster 2024-04-26T16:24:16.255258+0000 mon.a (mon.0) 1440 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: cluster 2024-04-26T16:24:16.255258+0000 mon.a (mon.0) 1440 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:16.468953+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:16.468953+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.028234+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.028234+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.028380+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.028380+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.028537+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.028537+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: cluster 2024-04-26T16:24:17.034111+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: cluster 2024-04-26T16:24:17.034111+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.041883+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.041883+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.042973+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[21463]: audit 2024-04-26T16:24:17.042973+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: cluster 2024-04-26T16:24:16.255258+0000 mon.a (mon.0) 1440 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:17.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: cluster 2024-04-26T16:24:16.255258+0000 mon.a (mon.0) 1440 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:17.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:16.468953+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:17.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:16.468953+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.028234+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.028234+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.028380+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.028380+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.028537+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.028537+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.452 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: cluster 2024-04-26T16:24:17.034111+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-04-26T16:24:17.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: cluster 2024-04-26T16:24:17.034111+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-04-26T16:24:17.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.041883+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.041883+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.042973+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:17 smithi050 bash[29234]: audit 2024-04-26T16:24:17.042973+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:17.566 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:17.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: cluster 2024-04-26T16:24:16.255258+0000 mon.a (mon.0) 1440 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:17.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: cluster 2024-04-26T16:24:16.255258+0000 mon.a (mon.0) 1440 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:16.468953+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:16.468953+0000 mon.c (mon.2) 258 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.028234+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.028234+0000 mon.a (mon.0) 1441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.028380+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.028380+0000 mon.a (mon.0) 1442 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi050-73193-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.028537+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.028537+0000 mon.a (mon.0) 1443 : audit [INF] from='client.? 172.21.15.50:0/1209262537' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi050-73201-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: cluster 2024-04-26T16:24:17.034111+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: cluster 2024-04-26T16:24:17.034111+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.041883+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.041883+0000 mon.a (mon.0) 1445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.042973+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:17 smithi190 bash[23868]: audit 2024-04-26T16:24:17.042973+0000 mon.b (mon.1) 133 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]: dispatch 2024-04-26T16:24:17.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:17.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:17.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.724 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:17.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:17.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:17.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:17.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.984 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:17.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:18.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:18.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:18.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:18.238 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:18.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:18.366 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:18.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:16.654221+0000 mgr.y (mgr.24401) 271 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:16.654221+0000 mgr.y (mgr.24401) 271 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:17.470583+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:17.470583+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: cluster 2024-04-26T16:24:17.511983+0000 mgr.y (mgr.24401) 272 : cluster [DBG] pgmap v268: 418 pgs: 1 active+clean+snaptrim, 1 peering, 64 creating+peering, 352 active+clean; 218 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 361 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: cluster 2024-04-26T16:24:17.511983+0000 mgr.y (mgr.24401) 272 : cluster [DBG] pgmap v268: 418 pgs: 1 active+clean+snaptrim, 1 peering, 64 creating+peering, 352 active+clean; 218 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 361 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:17.512455+0000 mon.c (mon.2) 260 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:17.512455+0000 mon.c (mon.2) 260 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:17.513777+0000 mon.a (mon.0) 1446 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:17.513777+0000 mon.a (mon.0) 1446 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.028948+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.028948+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:18.031464+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]': finished 2024-04-26T16:24:18.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:18.031464+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]': finished 2024-04-26T16:24:18.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:18.031579+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-26T16:24:18.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: audit 2024-04-26T16:24:18.031579+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-26T16:24:18.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.036877+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-04-26T16:24:18.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:18 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.036877+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-04-26T16:24:18.492 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:18.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:18.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:16.654221+0000 mgr.y (mgr.24401) 271 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:18.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:16.654221+0000 mgr.y (mgr.24401) 271 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:18.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:17.470583+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:18.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:17.470583+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:18.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: cluster 2024-04-26T16:24:17.511983+0000 mgr.y (mgr.24401) 272 : cluster [DBG] pgmap v268: 418 pgs: 1 active+clean+snaptrim, 1 peering, 64 creating+peering, 352 active+clean; 218 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 361 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: cluster 2024-04-26T16:24:17.511983+0000 mgr.y (mgr.24401) 272 : cluster [DBG] pgmap v268: 418 pgs: 1 active+clean+snaptrim, 1 peering, 64 creating+peering, 352 active+clean; 218 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 361 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:17.512455+0000 mon.c (mon.2) 260 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:17.512455+0000 mon.c (mon.2) 260 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:17.513777+0000 mon.a (mon.0) 1446 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:17.513777+0000 mon.a (mon.0) 1446 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.028948+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.028948+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:18.031464+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]': finished 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:18.031464+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]': finished 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:18.031579+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: audit 2024-04-26T16:24:18.031579+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.036877+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-04-26T16:24:18.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.036877+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:16.654221+0000 mgr.y (mgr.24401) 271 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:16.654221+0000 mgr.y (mgr.24401) 271 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:17.470583+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:17.470583+0000 mon.c (mon.2) 259 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: cluster 2024-04-26T16:24:17.511983+0000 mgr.y (mgr.24401) 272 : cluster [DBG] pgmap v268: 418 pgs: 1 active+clean+snaptrim, 1 peering, 64 creating+peering, 352 active+clean; 218 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 361 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: cluster 2024-04-26T16:24:17.511983+0000 mgr.y (mgr.24401) 272 : cluster [DBG] pgmap v268: 418 pgs: 1 active+clean+snaptrim, 1 peering, 64 creating+peering, 352 active+clean; 218 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 361 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:17.512455+0000 mon.c (mon.2) 260 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:17.512455+0000 mon.c (mon.2) 260 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:17.513777+0000 mon.a (mon.0) 1446 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:17.513777+0000 mon.a (mon.0) 1446 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.028948+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.028948+0000 mon.a (mon.0) 1447 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:18.031464+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]': finished 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:18.031464+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-13", "mode": "writeback"}]': finished 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:18.031579+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: audit 2024-04-26T16:24:18.031579+0000 mon.a (mon.0) 1449 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "21"}]': finished 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.036877+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-04-26T16:24:18.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:18 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.036877+0000 mon.a (mon.0) 1450 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2024-04-26T16:24:18.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:18.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:18.747 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:18.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:18.874 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:18.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:18.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:19.004 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:19.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:19.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:19.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:19.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:19.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:19.393 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:19.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:19.524 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.471519+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.471519+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.765103+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.765103+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766200+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766200+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766947+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766947+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:19.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.767567+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.767567+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.768143+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.768143+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.768691+0000 mon.b (mon.1) 139 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.768691+0000 mon.b (mon.1) 139 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.769227+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.769227+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.769807+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.769807+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.770336+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.770336+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.770927+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:18.770927+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.877737+0000 osd.4 (osd.4) 131 : cluster [DBG] 173.0 scrub starts 2024-04-26T16:24:19.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.877737+0000 osd.4 (osd.4) 131 : cluster [DBG] 173.0 scrub starts 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.883207+0000 osd.4 (osd.4) 132 : cluster [DBG] 173.0 scrub ok 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.883207+0000 osd.4 (osd.4) 132 : cluster [DBG] 173.0 scrub ok 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.042170+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.042170+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:19.050213+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:19.050213+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:19.050987+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:19 smithi190 bash[23868]: audit 2024-04-26T16:24:19.050987+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.471519+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.471519+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.765103+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.765103+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766200+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766200+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766947+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766947+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.767567+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:19.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.767567+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.768143+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.768143+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.768691+0000 mon.b (mon.1) 139 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.768691+0000 mon.b (mon.1) 139 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.769227+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.769227+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.769807+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.471519+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.471519+0000 mon.c (mon.2) 261 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.765103+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.765103+0000 mon.b (mon.1) 134 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766200+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766200+0000 mon.b (mon.1) 135 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766947+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766947+0000 mon.b (mon.1) 136 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.767567+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.767567+0000 mon.b (mon.1) 137 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.768143+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.768143+0000 mon.b (mon.1) 138 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.768691+0000 mon.b (mon.1) 139 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.768691+0000 mon.b (mon.1) 139 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.769227+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.769227+0000 mon.b (mon.1) 140 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.769807+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.769807+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.770336+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.770336+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.770927+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:18.770927+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.877737+0000 osd.4 (osd.4) 131 : cluster [DBG] 173.0 scrub starts 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.877737+0000 osd.4 (osd.4) 131 : cluster [DBG] 173.0 scrub starts 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.883207+0000 osd.4 (osd.4) 132 : cluster [DBG] 173.0 scrub ok 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.883207+0000 osd.4 (osd.4) 132 : cluster [DBG] 173.0 scrub ok 2024-04-26T16:24:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.042170+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.042170+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:19.050213+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:19.050213+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:19.050987+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[21463]: audit 2024-04-26T16:24:19.050987+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.769807+0000 mon.b (mon.1) 141 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.770336+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.770336+0000 mon.b (mon.1) 142 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:19.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.770927+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:18.770927+0000 mon.b (mon.1) 143 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.877737+0000 osd.4 (osd.4) 131 : cluster [DBG] 173.0 scrub starts 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.877737+0000 osd.4 (osd.4) 131 : cluster [DBG] 173.0 scrub starts 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.883207+0000 osd.4 (osd.4) 132 : cluster [DBG] 173.0 scrub ok 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.883207+0000 osd.4 (osd.4) 132 : cluster [DBG] 173.0 scrub ok 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.042170+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.042170+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:19.050213+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:19.050213+0000 mon.a (mon.0) 1452 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:19.050987+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:19 smithi050 bash[29234]: audit 2024-04-26T16:24:19.050987+0000 mon.a (mon.0) 1453 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:19.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:19.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:19.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:19.793 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:19.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:19.924 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:19.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:20.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.076 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:20.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:20.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:20.209 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:20.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:20.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.762850+0000 mgr.y (mgr.24401) 273 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:20.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.762850+0000 mgr.y (mgr.24401) 273 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:20.319 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.763788+0000 mgr.y (mgr.24401) 274 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.763788+0000 mgr.y (mgr.24401) 274 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.764478+0000 mgr.y (mgr.24401) 275 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.764478+0000 mgr.y (mgr.24401) 275 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.765101+0000 mgr.y (mgr.24401) 276 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.765101+0000 mgr.y (mgr.24401) 276 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.765676+0000 mgr.y (mgr.24401) 277 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.765676+0000 mgr.y (mgr.24401) 277 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766212+0000 mgr.y (mgr.24401) 278 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766212+0000 mgr.y (mgr.24401) 278 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766789+0000 mgr.y (mgr.24401) 279 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.766789+0000 mgr.y (mgr.24401) 279 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.767336+0000 mgr.y (mgr.24401) 280 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:20.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.767336+0000 mgr.y (mgr.24401) 280 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.767876+0000 mgr.y (mgr.24401) 281 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.767876+0000 mgr.y (mgr.24401) 281 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.768467+0000 mgr.y (mgr.24401) 282 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:18.768467+0000 mgr.y (mgr.24401) 282 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.891698+0000 osd.2 (osd.2) 109 : cluster [DBG] 173.8 scrub starts 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.891698+0000 osd.2 (osd.2) 109 : cluster [DBG] 173.8 scrub starts 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.894656+0000 osd.2 (osd.2) 110 : cluster [DBG] 173.8 scrub ok 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.894656+0000 osd.2 (osd.2) 110 : cluster [DBG] 173.8 scrub ok 2024-04-26T16:24:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.955009+0000 osd.3 (osd.3) 163 : cluster [DBG] 173.6 scrub starts 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.955009+0000 osd.3 (osd.3) 163 : cluster [DBG] 173.6 scrub starts 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.959276+0000 osd.3 (osd.3) 164 : cluster [DBG] 173.6 scrub ok 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:18.959276+0000 osd.3 (osd.3) 164 : cluster [DBG] 173.6 scrub ok 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.287082+0000 osd.0 (osd.0) 157 : cluster [DBG] 173.9 scrub starts 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.287082+0000 osd.0 (osd.0) 157 : cluster [DBG] 173.9 scrub starts 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.289494+0000 osd.0 (osd.0) 158 : cluster [DBG] 173.9 scrub ok 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.289494+0000 osd.0 (osd.0) 158 : cluster [DBG] 173.9 scrub ok 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:19.472441+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:19.472441+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.512640+0000 mgr.y (mgr.24401) 283 : cluster [DBG] pgmap v271: 386 pgs: 64 unknown, 1 active+clean+snaptrim, 1 peering, 320 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 105 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.512640+0000 mgr.y (mgr.24401) 283 : cluster [DBG] pgmap v271: 386 pgs: 64 unknown, 1 active+clean+snaptrim, 1 peering, 320 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 105 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.719142+0000 osd.7 (osd.7) 169 : cluster [DBG] 173.4 scrub starts 2024-04-26T16:24:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.719142+0000 osd.7 (osd.7) 169 : cluster [DBG] 173.4 scrub starts 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.726204+0000 osd.7 (osd.7) 170 : cluster [DBG] 173.4 scrub ok 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.726204+0000 osd.7 (osd.7) 170 : cluster [DBG] 173.4 scrub ok 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.901847+0000 osd.4 (osd.4) 133 : cluster [DBG] 173.3 scrub starts 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.901847+0000 osd.4 (osd.4) 133 : cluster [DBG] 173.3 scrub starts 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.906796+0000 osd.4 (osd.4) 134 : cluster [DBG] 173.3 scrub ok 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:19.906796+0000 osd.4 (osd.4) 134 : cluster [DBG] 173.3 scrub ok 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:20.041476+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:20.041476+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:20.041719+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: audit 2024-04-26T16:24:20.041719+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.047921+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-04-26T16:24:20.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:20 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.047921+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-04-26T16:24:20.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:20.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:20.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:20.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.762850+0000 mgr.y (mgr.24401) 273 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.762850+0000 mgr.y (mgr.24401) 273 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.763788+0000 mgr.y (mgr.24401) 274 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.763788+0000 mgr.y (mgr.24401) 274 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.764478+0000 mgr.y (mgr.24401) 275 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.764478+0000 mgr.y (mgr.24401) 275 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.765101+0000 mgr.y (mgr.24401) 276 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.765101+0000 mgr.y (mgr.24401) 276 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.765676+0000 mgr.y (mgr.24401) 277 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.765676+0000 mgr.y (mgr.24401) 277 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766212+0000 mgr.y (mgr.24401) 278 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766212+0000 mgr.y (mgr.24401) 278 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766789+0000 mgr.y (mgr.24401) 279 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.766789+0000 mgr.y (mgr.24401) 279 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:20.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.767336+0000 mgr.y (mgr.24401) 280 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.767336+0000 mgr.y (mgr.24401) 280 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.767876+0000 mgr.y (mgr.24401) 281 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.767876+0000 mgr.y (mgr.24401) 281 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.768467+0000 mgr.y (mgr.24401) 282 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:18.768467+0000 mgr.y (mgr.24401) 282 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.891698+0000 osd.2 (osd.2) 109 : cluster [DBG] 173.8 scrub starts 2024-04-26T16:24:20.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.762850+0000 mgr.y (mgr.24401) 273 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.762850+0000 mgr.y (mgr.24401) 273 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.0"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.763788+0000 mgr.y (mgr.24401) 274 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.763788+0000 mgr.y (mgr.24401) 274 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.1"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.764478+0000 mgr.y (mgr.24401) 275 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.764478+0000 mgr.y (mgr.24401) 275 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.2"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.765101+0000 mgr.y (mgr.24401) 276 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.765101+0000 mgr.y (mgr.24401) 276 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.3"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.765676+0000 mgr.y (mgr.24401) 277 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.765676+0000 mgr.y (mgr.24401) 277 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.4"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766212+0000 mgr.y (mgr.24401) 278 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766212+0000 mgr.y (mgr.24401) 278 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.5"}]: dispatch 2024-04-26T16:24:20.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766789+0000 mgr.y (mgr.24401) 279 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.766789+0000 mgr.y (mgr.24401) 279 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.6"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.767336+0000 mgr.y (mgr.24401) 280 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.767336+0000 mgr.y (mgr.24401) 280 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.7"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.767876+0000 mgr.y (mgr.24401) 281 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.767876+0000 mgr.y (mgr.24401) 281 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.8"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.768467+0000 mgr.y (mgr.24401) 282 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:18.768467+0000 mgr.y (mgr.24401) 282 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "173.9"}]: dispatch 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.891698+0000 osd.2 (osd.2) 109 : cluster [DBG] 173.8 scrub starts 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.891698+0000 osd.2 (osd.2) 109 : cluster [DBG] 173.8 scrub starts 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.894656+0000 osd.2 (osd.2) 110 : cluster [DBG] 173.8 scrub ok 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.894656+0000 osd.2 (osd.2) 110 : cluster [DBG] 173.8 scrub ok 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.955009+0000 osd.3 (osd.3) 163 : cluster [DBG] 173.6 scrub starts 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.955009+0000 osd.3 (osd.3) 163 : cluster [DBG] 173.6 scrub starts 2024-04-26T16:24:20.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.959276+0000 osd.3 (osd.3) 164 : cluster [DBG] 173.6 scrub ok 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:18.959276+0000 osd.3 (osd.3) 164 : cluster [DBG] 173.6 scrub ok 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.287082+0000 osd.0 (osd.0) 157 : cluster [DBG] 173.9 scrub starts 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.287082+0000 osd.0 (osd.0) 157 : cluster [DBG] 173.9 scrub starts 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.289494+0000 osd.0 (osd.0) 158 : cluster [DBG] 173.9 scrub ok 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.289494+0000 osd.0 (osd.0) 158 : cluster [DBG] 173.9 scrub ok 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:19.472441+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:19.472441+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.512640+0000 mgr.y (mgr.24401) 283 : cluster [DBG] pgmap v271: 386 pgs: 64 unknown, 1 active+clean+snaptrim, 1 peering, 320 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 105 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.512640+0000 mgr.y (mgr.24401) 283 : cluster [DBG] pgmap v271: 386 pgs: 64 unknown, 1 active+clean+snaptrim, 1 peering, 320 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 105 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.719142+0000 osd.7 (osd.7) 169 : cluster [DBG] 173.4 scrub starts 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.719142+0000 osd.7 (osd.7) 169 : cluster [DBG] 173.4 scrub starts 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.726204+0000 osd.7 (osd.7) 170 : cluster [DBG] 173.4 scrub ok 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.726204+0000 osd.7 (osd.7) 170 : cluster [DBG] 173.4 scrub ok 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.901847+0000 osd.4 (osd.4) 133 : cluster [DBG] 173.3 scrub starts 2024-04-26T16:24:20.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.901847+0000 osd.4 (osd.4) 133 : cluster [DBG] 173.3 scrub starts 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.906796+0000 osd.4 (osd.4) 134 : cluster [DBG] 173.3 scrub ok 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:19.906796+0000 osd.4 (osd.4) 134 : cluster [DBG] 173.3 scrub ok 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:20.041476+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.891698+0000 osd.2 (osd.2) 109 : cluster [DBG] 173.8 scrub starts 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.894656+0000 osd.2 (osd.2) 110 : cluster [DBG] 173.8 scrub ok 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.894656+0000 osd.2 (osd.2) 110 : cluster [DBG] 173.8 scrub ok 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.955009+0000 osd.3 (osd.3) 163 : cluster [DBG] 173.6 scrub starts 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.955009+0000 osd.3 (osd.3) 163 : cluster [DBG] 173.6 scrub starts 2024-04-26T16:24:20.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.959276+0000 osd.3 (osd.3) 164 : cluster [DBG] 173.6 scrub ok 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:18.959276+0000 osd.3 (osd.3) 164 : cluster [DBG] 173.6 scrub ok 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.287082+0000 osd.0 (osd.0) 157 : cluster [DBG] 173.9 scrub starts 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.287082+0000 osd.0 (osd.0) 157 : cluster [DBG] 173.9 scrub starts 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.289494+0000 osd.0 (osd.0) 158 : cluster [DBG] 173.9 scrub ok 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.289494+0000 osd.0 (osd.0) 158 : cluster [DBG] 173.9 scrub ok 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:19.472441+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:19.472441+0000 mon.c (mon.2) 262 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.512640+0000 mgr.y (mgr.24401) 283 : cluster [DBG] pgmap v271: 386 pgs: 64 unknown, 1 active+clean+snaptrim, 1 peering, 320 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 105 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.512640+0000 mgr.y (mgr.24401) 283 : cluster [DBG] pgmap v271: 386 pgs: 64 unknown, 1 active+clean+snaptrim, 1 peering, 320 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 105 KiB/s wr, 103 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.719142+0000 osd.7 (osd.7) 169 : cluster [DBG] 173.4 scrub starts 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.719142+0000 osd.7 (osd.7) 169 : cluster [DBG] 173.4 scrub starts 2024-04-26T16:24:20.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.726204+0000 osd.7 (osd.7) 170 : cluster [DBG] 173.4 scrub ok 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.726204+0000 osd.7 (osd.7) 170 : cluster [DBG] 173.4 scrub ok 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.901847+0000 osd.4 (osd.4) 133 : cluster [DBG] 173.3 scrub starts 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.901847+0000 osd.4 (osd.4) 133 : cluster [DBG] 173.3 scrub starts 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.906796+0000 osd.4 (osd.4) 134 : cluster [DBG] 173.3 scrub ok 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:19.906796+0000 osd.4 (osd.4) 134 : cluster [DBG] 173.3 scrub ok 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:20.041476+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:20.041476+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:20.041719+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: audit 2024-04-26T16:24:20.041719+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.047921+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-04-26T16:24:20.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.047921+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-04-26T16:24:20.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:20.041476+0000 mon.a (mon.0) 1454 : audit [INF] from='client.? 172.21.15.50:0/2347486620' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi050-73201-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:20.041719+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: audit 2024-04-26T16:24:20.041719+0000 mon.a (mon.0) 1455 : audit [INF] from='client.? 172.21.15.50:0/626245122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi050-73193-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:20.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.047921+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-04-26T16:24:20.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:20 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.047921+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2024-04-26T16:24:20.576 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:20.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:20.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:20.736 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:20.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:20.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:20.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:21.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:21.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:21.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:21.256 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:21.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: audit 2024-04-26T16:24:20.473519+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:21.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: audit 2024-04-26T16:24:20.473519+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:21.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.740297+0000 osd.7 (osd.7) 171 : cluster [DBG] 173.2 scrub starts 2024-04-26T16:24:21.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.740297+0000 osd.7 (osd.7) 171 : cluster [DBG] 173.2 scrub starts 2024-04-26T16:24:21.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.743387+0000 osd.7 (osd.7) 172 : cluster [DBG] 173.2 scrub ok 2024-04-26T16:24:21.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.743387+0000 osd.7 (osd.7) 172 : cluster [DBG] 173.2 scrub ok 2024-04-26T16:24:21.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: audit 2024-04-26T16:24:20.473519+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:21.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: audit 2024-04-26T16:24:20.473519+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:21.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.740297+0000 osd.7 (osd.7) 171 : cluster [DBG] 173.2 scrub starts 2024-04-26T16:24:21.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:21.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:21.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:21.520 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: audit 2024-04-26T16:24:20.473519+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: audit 2024-04-26T16:24:20.473519+0000 mon.c (mon.2) 263 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.740297+0000 osd.7 (osd.7) 171 : cluster [DBG] 173.2 scrub starts 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.740297+0000 osd.7 (osd.7) 171 : cluster [DBG] 173.2 scrub starts 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.743387+0000 osd.7 (osd.7) 172 : cluster [DBG] 173.2 scrub ok 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.743387+0000 osd.7 (osd.7) 172 : cluster [DBG] 173.2 scrub ok 2024-04-26T16:24:21.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.872116+0000 osd.4 (osd.4) 135 : cluster [DBG] 173.1 scrub starts 2024-04-26T16:24:21.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.872116+0000 osd.4 (osd.4) 135 : cluster [DBG] 173.1 scrub starts 2024-04-26T16:24:21.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.880567+0000 osd.4 (osd.4) 136 : cluster [DBG] 173.1 scrub ok 2024-04-26T16:24:21.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:20.880567+0000 osd.4 (osd.4) 136 : cluster [DBG] 173.1 scrub ok 2024-04-26T16:24:21.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.048924+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-04-26T16:24:21.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:21 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.048924+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-04-26T16:24:21.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.872116+0000 osd.4 (osd.4) 135 : cluster [DBG] 173.1 scrub starts 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.872116+0000 osd.4 (osd.4) 135 : cluster [DBG] 173.1 scrub starts 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.880567+0000 osd.4 (osd.4) 136 : cluster [DBG] 173.1 scrub ok 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:20.880567+0000 osd.4 (osd.4) 136 : cluster [DBG] 173.1 scrub ok 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.048924+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.048924+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.740297+0000 osd.7 (osd.7) 171 : cluster [DBG] 173.2 scrub starts 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.743387+0000 osd.7 (osd.7) 172 : cluster [DBG] 173.2 scrub ok 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.743387+0000 osd.7 (osd.7) 172 : cluster [DBG] 173.2 scrub ok 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.872116+0000 osd.4 (osd.4) 135 : cluster [DBG] 173.1 scrub starts 2024-04-26T16:24:21.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.872116+0000 osd.4 (osd.4) 135 : cluster [DBG] 173.1 scrub starts 2024-04-26T16:24:21.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.880567+0000 osd.4 (osd.4) 136 : cluster [DBG] 173.1 scrub ok 2024-04-26T16:24:21.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:20.880567+0000 osd.4 (osd.4) 136 : cluster [DBG] 173.1 scrub ok 2024-04-26T16:24:21.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.048924+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-04-26T16:24:21.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:21 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.048924+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2024-04-26T16:24:21.651 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:21.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:21.780 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:21.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:21.910 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:21.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:21.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/444284647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4103ea0 msgr2=0x7f6bf41102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:21.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/444284647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4103ea0 0x7f6bf41102c0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00b0a0 tx=0x7f6bec00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/444284647 shutdown_connections 2024-04-26T16:24:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/444284647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf4109480 0x7f6bf4109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/444284647 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4103ea0 0x7f6bf41102c0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/444284647 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bf41034e0 0x7f6bf4103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/444284647 >> 172.21.15.50:0/444284647 conn(0x7f6bf407a5e0 msgr2=0x7f6bf407a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:21.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/444284647 shutdown_connections 2024-04-26T16:24:21.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/444284647 wait complete. 2024-04-26T16:24:21.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 librados: starting msgr at 2024-04-26T16:24:21.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 librados: starting objecter 2024-04-26T16:24:21.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 Processor -- start 2024-04-26T16:24:21.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- start start 2024-04-26T16:24:21.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 librados: setting wanted keys 2024-04-26T16:24:21.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 librados: calling monclient init 2024-04-26T16:24:21.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf41034e0 0x7f6bf419b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:21.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bf4103ea0 0x7f6bf419b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:21.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4109480 0x7f6bf419fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:21.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf4112ae0 con 0x7f6bf4109480 2024-04-26T16:24:21.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf4112960 con 0x7f6bf41034e0 2024-04-26T16:24:21.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf4112c60 con 0x7f6bf4103ea0 2024-04-26T16:24:21.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4109480 0x7f6bf419fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:21.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bf952f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bf4103ea0 0x7f6bf419b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:21.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4109480 0x7f6bf419fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58586/0 (socket says 172.21.15.50:58586) 2024-04-26T16:24:21.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 -- 172.21.15.50:0/138974387 learned_addr learned my addr 172.21.15.50:0/138974387 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:21.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bf9d30700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf41034e0 0x7f6bf419b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:21.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 -- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bf4103ea0 msgr2=0x7f6bf419b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:21.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bf4103ea0 0x7f6bf419b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 -- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf41034e0 msgr2=0x7f6bf419b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:21.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf41034e0 0x7f6bf419b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 -- 172.21.15.50:0/138974387 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf41a04b0 con 0x7f6bf4109480 2024-04-26T16:24:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bf9d30700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bf41034e0 0x7f6bf419b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:24:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bf952f700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bf4103ea0 0x7f6bf419b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:24:21.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfa531700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4109480 0x7f6bf419fd70 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00b780 tx=0x7f6bec004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6be6ffd700 10 client.?.objecter ms_handle_connect 0x7f6bf4109480 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6be6ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec021070 con 0x7f6bf4109480 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf41a0740 con 0x7f6bf4109480 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6bec007b80 con 0x7f6bf4109480 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bec005040 con 0x7f6bf4109480 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 10 client.29913.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:21.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf41a8020 con 0x7f6bf4109480 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 librados: init done 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 10 client.29913.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 15 client.29913.objecter _session_command_op_assign -1 1 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 20 client.29913.objecter _get_session osd=-1 returning homeless 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 10 client.29913.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.938+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6bf4104b30 con 0x7f6bf4109480 2024-04-26T16:24:21.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6bec024070 con 0x7f6bf4109480 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6be0071f80 0x7f6be0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(193..193 src has 87..193) ==== 11509+0+0 (secure 0 0 0) 0x7f6bec00f020 con 0x7f6bf4109480 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter ms_dispatch 0x7f6bf4001550 osd_map(193..193 src has 87..193) 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 3 client.29913.objecter handle_osd_map got epochs [193,193] > 0 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 3 client.29913.objecter handle_osd_map decoding full epoch 193 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6bf9d30700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6be0071f80 0x7f6be0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter checking command 1 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f6be0082db0 0x7f6be0085250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter _get_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 15 client.29913.objecter _session_command_op_remove -1 1 2024-04-26T16:24:21.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter _get_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter get_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 15 client.29913.objecter _session_command_op_assign 6 1 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 4 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter _send_command 1 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6bec00aea0 con 0x7f6be0082db0 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter dump_active .. 0 homeless 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=193) ==== 24+0+0 (secure 0 0 0) 0x7f6bec093f50 con 0x7f6bf4109480 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6bf952f700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f6be0082db0 0x7f6be0085250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter ms_handle_connect 0x7f6be0071f80 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6bf9d30700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6be0071f80 0x7f6be0074440 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00099d0 tx=0x7f6bf0008040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter ms_handle_connect 0x7f6be0082db0 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6bf952f700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f6be0082db0 0x7f6be0085250 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6bec00aea0 con 0x7f6be0082db0 2024-04-26T16:24:21.947 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter ms_dispatch 0x7f6bf4001550 command_reply(tid 1: 0 ) 2024-04-26T16:24:21.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 10 client.29913.objecter _finish_command 1 = system:0 2024-04-26T16:24:21.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 4 2024-04-26T16:24:21.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.942+0000 7f6be6ffd700 15 client.29913.objecter _session_command_op_remove 6 1 2024-04-26T16:24:21.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 10 client.29913.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 15 client.29913.objecter _session_command_op_assign -1 2 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter _get_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter _get_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 15 client.29913.objecter _session_command_op_remove -1 2 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter get_session s=0x7f6be00826e0 osd=6 3 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 15 client.29913.objecter _session_command_op_assign 6 2 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 4 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 10 client.29913.objecter _send_command 2 2024-04-26T16:24:21.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f6bf406a110 con 0x7f6be0082db0 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6be6ffd700 1 -- 172.21.15.50:0/138974387 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f6bf406a110 con 0x7f6be0082db0 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6be6ffd700 10 client.29913.objecter ms_dispatch 0x7f6bf4001550 command_reply(tid 2: 0 ) 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6be6ffd700 10 client.29913.objecter _finish_command 2 = system:0 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6be6ffd700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 4 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6be6ffd700 15 client.29913.objecter _session_command_op_remove 6 2 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 10 librados: watch_flush enter 2024-04-26T16:24:21.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 10 librados: watch_flush exit 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 10 client.29913.objecter close_session for osd.6 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f6be0082db0 msgr2=0x7f6be0085250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f6be0082db0 0x7f6be0085250 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter put_session s=0x7f6be00826e0 osd=6 1 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 20 client.29913.objecter shutdown clearing up homeless session... 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 10 client.29913.objecter successfully canceled tick 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6be0071f80 msgr2=0x7f6be0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6be0071f80 0x7f6be0074440 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f6bf00099d0 tx=0x7f6bf0008040 comp rx=0 tx=0).stop 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4109480 msgr2=0x7f6bf419fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 --2- 172.21.15.50:0/138974387 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bf4109480 0x7f6bf419fd70 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f6bec00b780 tx=0x7f6bec004d00 comp rx=0 tx=0).stop 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfa531700 1 -- 172.21.15.50:0/138974387 reap_dead start 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 shutdown_connections 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 >> 172.21.15.50:0/138974387 conn(0x7f6bf407a5e0 msgr2=0x7f6bf410e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 shutdown_connections 2024-04-26T16:24:21.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 -- 172.21.15.50:0/138974387 wait complete. 2024-04-26T16:24:21.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:21.966+0000 7f6bfbf94700 1 librados: shutdown 2024-04-26T16:24:22.014 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:22.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:22.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:22.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/1207577623 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 msgr2=0x7ff5fc116a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- 172.21.15.50:0/1207577623 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc116a10 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7ff5f400b0a0 tx=0x7ff5f400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/1207577623 shutdown_connections 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- 172.21.15.50:0/1207577623 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5fc075650 0x7ff5fc075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- 172.21.15.50:0/1207577623 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc116a10 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- 172.21.15.50:0/1207577623 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075f70 0x7ff5fc1140e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/1207577623 >> 172.21.15.50:0/1207577623 conn(0x7ff5fc07a590 msgr2=0x7ff5fc07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/1207577623 shutdown_connections 2024-04-26T16:24:22.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/1207577623 wait complete. 2024-04-26T16:24:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 librados: starting msgr at 2024-04-26T16:24:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 librados: starting objecter 2024-04-26T16:24:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 Processor -- start 2024-04-26T16:24:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- start start 2024-04-26T16:24:22.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 librados: setting wanted keys 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 librados: calling monclient init 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 0x7ff5fc19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5fc075f70 0x7ff5fc19fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc1a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc119180 con 0x7ff5fc075650 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc119000 con 0x7ff5fc075f70 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff5fc119300 con 0x7ff5fc114620 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 0x7ff5fc19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff60274e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc1a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 0x7ff5fc19f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58602/0 (socket says 172.21.15.50:58602) 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff60274e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc1a3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47648/0 (socket says 172.21.15.50:47648) 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 -- 172.21.15.50:0/743832826 learned_addr learned my addr 172.21.15.50:0/743832826 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:22.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5fc075f70 0x7ff5fc19fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 msgr2=0x7ff5fc1a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc1a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5fc075f70 msgr2=0x7ff5fc19fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5fc075f70 0x7ff5fc19fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc1a4720 con 0x7ff5fc075650 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5fc114620 0x7ff5fc1a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5fc075f70 0x7ff5fc19fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 0x7ff5fc19f710 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800d980 tx=0x7ff5f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff5f2ffd700 10 client.?.objecter ms_handle_connect 0x7ff5fc075650 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff5f2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8017070 con 0x7ff5fc075650 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5fc1a4a10 con 0x7ff5fc075650 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff5f8005020 con 0x7ff5fc075650 2024-04-26T16:24:22.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8009990 con 0x7ff5fc075650 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 10 client.29919.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5fc1ac290 con 0x7ff5fc075650 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 librados: init done 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.154+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff5fc06a110 con 0x7ff5fc075650 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff5f8004a30 con 0x7ff5fc075650 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff5e8071f80 0x7ff5e8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(194..194 src has 87..194) ==== 11770+0+0 (secure 0 0 0) 0x7ff5f80927b0 con 0x7ff5fc075650 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 osd_map(194..194 src has 87..194) 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 3 client.29919.objecter handle_osd_map got epochs [194,194] > 0 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 3 client.29919.objecter handle_osd_map decoding full epoch 194 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff5e8071f80 0x7ff5e8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 20 client.29919.objecter dump_active .. 0 homeless 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v194) ==== 54+0+16 (secure 0 0 0) 0x7ff5f8020020 con 0x7ff5fc075650 2024-04-26T16:24:22.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 10 client.29919.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 1 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff5fc064030 0x7ff5fc064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 1 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 0 1 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 4 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 10 client.29919.objecter _send_command 1 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5e8071f80 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff5e8071f80 0x7ff5e8074440 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc101ba0 tx=0x7ff5ec009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1af6b0 con 0x7ff5fc064030 2024-04-26T16:24:22.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff5fc064030 0x7ff5fc064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc064030 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff5fc064030 0x7ff5fc064410 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1af6b0 con 0x7ff5fc064030 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 1: 0 ) 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 1 = system:0 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 4 2024-04-26T16:24:22.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.158+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 0 1 2024-04-26T16:24:22.165 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:22.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 10 client.29919.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 2 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 2 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1a0a70 osd=0 3 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 0 2 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 4 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 10 client.29919.objecter _send_command 2 2024-04-26T16:24:22.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc063ab0 con 0x7ff5fc064030 2024-04-26T16:24:22.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc063ab0 con 0x7ff5fc064030 2024-04-26T16:24:22.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 4 2024-04-26T16:24:22.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 0 2 2024-04-26T16:24:22.175 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 10 client.29919.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff5fc02d310 0x7ff5fc02d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.178+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 1 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 4 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 10 client.29919.objecter _send_command 3 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1033f0 con 0x7ff5fc02d310 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff5fc02d310 0x7ff5fc02d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc02d310 2024-04-26T16:24:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff5fc02d310 0x7ff5fc02d710 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1033f0 con 0x7ff5fc02d310 2024-04-26T16:24:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 3: 0 ) 2024-04-26T16:24:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 3 = system:0 2024-04-26T16:24:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 4 2024-04-26T16:24:22.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.182+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 1 3 2024-04-26T16:24:22.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 10 client.29919.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 4 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 4 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc102d70 osd=1 3 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 1 4 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 4 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 10 client.29919.objecter _send_command 4 2024-04-26T16:24:22.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.198+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc1027f0 con 0x7ff5fc02d310 2024-04-26T16:24:22.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc1027f0 con 0x7ff5fc02d310 2024-04-26T16:24:22.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 4 2024-04-26T16:24:22.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 1 4 2024-04-26T16:24:22.196 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:24:22.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 10 client.29919.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 5 2024-04-26T16:24:22.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff5fc1b1870 0x7ff5fc1b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 5 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 2 5 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 4 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 10 client.29919.objecter _send_command 5 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1b6ac0 con 0x7ff5fc1b1870 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff5fc1b1870 0x7ff5fc1b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc1b1870 2024-04-26T16:24:22.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff5fc1b1870 0x7ff5fc1b3c60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1b6ac0 con 0x7ff5fc1b1870 2024-04-26T16:24:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 5: 0 ) 2024-04-26T16:24:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 5 = system:0 2024-04-26T16:24:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 4 2024-04-26T16:24:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.202+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 2 5 2024-04-26T16:24:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 10 client.29919.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 6 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 6 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1035a0 osd=2 3 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 2 6 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 4 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 10 client.29919.objecter _send_command 6 2024-04-26T16:24:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.218+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc02dea0 con 0x7ff5fc1b1870 2024-04-26T16:24:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc02dea0 con 0x7ff5fc1b1870 2024-04-26T16:24:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 4 2024-04-26T16:24:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 2 6 2024-04-26T16:24:22.217 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:24:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 10 client.29919.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 7 2024-04-26T16:24:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff5fc1af210 0x7ff5fc1b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 7 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 3 7 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 4 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 10 client.29919.objecter _send_command 7 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1ba420 con 0x7ff5fc1af210 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff5fc1af210 0x7ff5fc1b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc1af210 2024-04-26T16:24:22.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff5fc1af210 0x7ff5fc1b5320 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1ba420 con 0x7ff5fc1af210 2024-04-26T16:24:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 7: 0 ) 2024-04-26T16:24:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 7 = system:0 2024-04-26T16:24:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 4 2024-04-26T16:24:22.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.222+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 3 7 2024-04-26T16:24:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 10 client.29919.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 8 2024-04-26T16:24:22.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 8 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc02e930 osd=3 3 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 3 8 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 4 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 10 client.29919.objecter _send_command 8 2024-04-26T16:24:22.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc1ae7e0 con 0x7ff5fc1af210 2024-04-26T16:24:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc1ae7e0 con 0x7ff5fc1af210 2024-04-26T16:24:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 4 2024-04-26T16:24:22.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 3 8 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 10 client.29919.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 9 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff5fc1b4e80 0x7ff5fc1b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 9 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 4 9 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 4 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 10 client.29919.objecter _send_command 9 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1bde10 con 0x7ff5fc1b4e80 2024-04-26T16:24:22.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.242+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff5fc1b4e80 0x7ff5fc1b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc1b4e80 2024-04-26T16:24:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff5fc1b4e80 0x7ff5fc1b8c80 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1bde10 con 0x7ff5fc1b4e80 2024-04-26T16:24:22.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 9: 0 ) 2024-04-26T16:24:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 9 = system:0 2024-04-26T16:24:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 4 2024-04-26T16:24:22.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.246+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 4 9 2024-04-26T16:24:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 10 client.29919.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 10 2024-04-26T16:24:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 10 2024-04-26T16:24:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1ae590 osd=4 3 2024-04-26T16:24:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 4 10 2024-04-26T16:24:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 4 2024-04-26T16:24:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 10 client.29919.objecter _send_command 10 2024-04-26T16:24:22.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.262+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc1b91c0 con 0x7ff5fc1b4e80 2024-04-26T16:24:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc1b91c0 con 0x7ff5fc1b4e80 2024-04-26T16:24:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 4 2024-04-26T16:24:22.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 4 10 2024-04-26T16:24:22.263 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:24:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 10 client.29919.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 11 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff5fc1b98e0 0x7ff5fc1b9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 11 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 5 11 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 4 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 10 client.29919.objecter _send_command 11 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.266+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1c0fa0 con 0x7ff5fc1b98e0 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff5fc1b98e0 0x7ff5fc1b9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff60174c700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff5fc1b98e0 0x7ff5fc1b9d40 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc1b98e0 2024-04-26T16:24:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1c0fa0 con 0x7ff5fc1b98e0 2024-04-26T16:24:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 11: 0 ) 2024-04-26T16:24:22.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 11 = system:0 2024-04-26T16:24:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 4 2024-04-26T16:24:22.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.270+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 5 11 2024-04-26T16:24:22.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:22.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:21.474912+0000 mon.c (mon.2) 264 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:22.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:21.474912+0000 mon.c (mon.2) 264 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:22.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.517063+0000 mgr.y (mgr.24401) 284 : cluster [DBG] pgmap v274: 353 pgs: 1 active+clean+snaptrim, 352 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-26T16:24:22.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.517063+0000 mgr.y (mgr.24401) 284 : cluster [DBG] pgmap v274: 353 pgs: 1 active+clean+snaptrim, 352 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-26T16:24:22.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:21.517855+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:21.517855+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:21.522194+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:21.522194+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.744009+0000 osd.7 (osd.7) 173 : cluster [DBG] 173.7 scrub starts 2024-04-26T16:24:22.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.744009+0000 osd.7 (osd.7) 173 : cluster [DBG] 173.7 scrub starts 2024-04-26T16:24:22.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.748671+0000 osd.7 (osd.7) 174 : cluster [DBG] 173.7 scrub ok 2024-04-26T16:24:22.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:21.748671+0000 osd.7 (osd.7) 174 : cluster [DBG] 173.7 scrub ok 2024-04-26T16:24:22.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 10 client.29919.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 12 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 12 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1ae810 osd=5 3 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 5 12 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 4 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 10 client.29919.objecter _send_command 12 2024-04-26T16:24:22.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.294+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc1be680 con 0x7ff5fc1b98e0 2024-04-26T16:24:22.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc1be680 con 0x7ff5fc1b98e0 2024-04-26T16:24:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 4 2024-04-26T16:24:22.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 5 12 2024-04-26T16:24:22.293 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 10 client.29919.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 13 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff5fc1bed00 0x7ff5fc1c10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 13 2024-04-26T16:24:22.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 6 13 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 4 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 10 client.29919.objecter _send_command 13 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1c4190 con 0x7ff5fc1bed00 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff5fc1bed00 0x7ff5fc1c10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc1bed00 2024-04-26T16:24:22.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.298+0000 7ff60274e700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff5fc1bed00 0x7ff5fc1c10f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.302+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1c4190 con 0x7ff5fc1bed00 2024-04-26T16:24:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.302+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 13: 0 ) 2024-04-26T16:24:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.302+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 13 = system:0 2024-04-26T16:24:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.302+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 4 2024-04-26T16:24:22.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.302+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 6 13 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 10 client.29919.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 14 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 14 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1be680 osd=6 3 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 6 14 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 4 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 10 client.29919.objecter _send_command 14 2024-04-26T16:24:22.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.318+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc1c2f20 con 0x7ff5fc1bed00 2024-04-26T16:24:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc1c2f20 con 0x7ff5fc1bed00 2024-04-26T16:24:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 4 2024-04-26T16:24:22.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 6 14 2024-04-26T16:24:22.318 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 10 client.29919.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 15 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff5fc1c3640 0x7ff5fc1c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 15 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 7 15 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 4 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 10 client.29919.objecter _send_command 15 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff5fc1c7560 con 0x7ff5fc1c3640 2024-04-26T16:24:22.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.322+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff5fc1c3640 0x7ff5fc1c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff5f2ffd700 10 client.29919.objecter ms_handle_connect 0x7ff5fc1c3640 2024-04-26T16:24:22.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff601f4d700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff5fc1c3640 0x7ff5fc1c3a80 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5fc1c7560 con 0x7ff5fc1c3640 2024-04-26T16:24:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 15: 0 ) 2024-04-26T16:24:22.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 15 = system:0 2024-04-26T16:24:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 4 2024-04-26T16:24:22.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.326+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 7 15 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 10 client.29919.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign -1 16 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 20 client.29919.objecter _get_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_remove -1 16 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 20 client.29919.objecter get_session s=0x7ff5fc1c2f20 osd=7 3 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.350+0000 7ff6041b1700 15 client.29919.objecter _session_command_op_assign 7 16 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.354+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 4 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.354+0000 7ff6041b1700 10 client.29919.objecter _send_command 16 2024-04-26T16:24:22.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.354+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff5fc1c1750 con 0x7ff5fc1c3640 2024-04-26T16:24:22.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff5f2ffd700 1 -- 172.21.15.50:0/743832826 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff5fc1c1750 con 0x7ff5fc1c3640 2024-04-26T16:24:22.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff5f2ffd700 10 client.29919.objecter ms_dispatch 0x7ff5fc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:22.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff5f2ffd700 10 client.29919.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:22.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff5f2ffd700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 4 2024-04-26T16:24:22.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff5f2ffd700 15 client.29919.objecter _session_command_op_remove 7 16 2024-04-26T16:24:22.351 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:24:22.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 librados: watch_flush enter 2024-04-26T16:24:22.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 librados: watch_flush exit 2024-04-26T16:24:22.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.0 2024-04-26T16:24:22.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff5fc064030 msgr2=0x7ff5fc064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff5fc064030 0x7ff5fc064410 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1a0a70 osd=0 1 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.1 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff5fc02d310 msgr2=0x7ff5fc02d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff5fc02d310 0x7ff5fc02d710 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc102d70 osd=1 1 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.2 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff5fc1b1870 msgr2=0x7ff5fc1b3c60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff5fc1b1870 0x7ff5fc1b3c60 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1035a0 osd=2 1 2024-04-26T16:24:22.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.3 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff5fc1af210 msgr2=0x7ff5fc1b5320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff5fc1af210 0x7ff5fc1b5320 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc02e930 osd=3 1 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.4 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff5fc1b4e80 msgr2=0x7ff5fc1b8c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff5fc1b4e80 0x7ff5fc1b8c80 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae590 osd=4 1 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.5 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff5fc1b98e0 msgr2=0x7ff5fc1b9d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff5fc1b98e0 0x7ff5fc1b9d40 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1ae810 osd=5 1 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.6 2024-04-26T16:24:22.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff5fc1bed00 msgr2=0x7ff5fc1c10f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7ff5fc1bed00 0x7ff5fc1c10f0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1be680 osd=6 1 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter close_session for osd.7 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff5fc1c3640 msgr2=0x7ff5fc1c3a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff5fc1c3640 0x7ff5fc1c3a80 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter put_session s=0x7ff5fc1c2f20 osd=7 1 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 20 client.29919.objecter shutdown clearing up homeless session... 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 10 client.29919.objecter successfully canceled tick 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff60274e700 1 -- 172.21.15.50:0/743832826 reap_dead start 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff60274e700 1 -- 172.21.15.50:0/743832826 reap_dead start 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff5e8071f80 msgr2=0x7ff5e8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff5e8071f80 0x7ff5e8074440 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc101ba0 tx=0x7ff5ec009260 comp rx=0 tx=0).stop 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff60274e700 1 -- 172.21.15.50:0/743832826 reap_dead start 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff60274e700 1 -- 172.21.15.50:0/743832826 reap_dead start 2024-04-26T16:24:22.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 msgr2=0x7ff5fc19f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff60274e700 1 -- 172.21.15.50:0/743832826 reap_dead start 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 0x7ff5fc19f710 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800d980 tx=0x7ff5f800dc90 comp rx=0 tx=0).stop 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 shutdown_connections 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 --2- 172.21.15.50:0/743832826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5fc075650 0x7ff5fc19f710 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 >> 172.21.15.50:0/743832826 conn(0x7ff5fc07a590 msgr2=0x7ff5fc1100c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 shutdown_connections 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 -- 172.21.15.50:0/743832826 wait complete. 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:22.358+0000 7ff6041b1700 1 librados: shutdown 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:24:22.356 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:24:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:24:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:24:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:24:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:24:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:24:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:24:22.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:22.420 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:22.479 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:24:22.479 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:24:22.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:22.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:21.474912+0000 mon.c (mon.2) 264 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:21.474912+0000 mon.c (mon.2) 264 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.517063+0000 mgr.y (mgr.24401) 284 : cluster [DBG] pgmap v274: 353 pgs: 1 active+clean+snaptrim, 352 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.517063+0000 mgr.y (mgr.24401) 284 : cluster [DBG] pgmap v274: 353 pgs: 1 active+clean+snaptrim, 352 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:21.517855+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:21.517855+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:21.522194+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:21.522194+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.744009+0000 osd.7 (osd.7) 173 : cluster [DBG] 173.7 scrub starts 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.744009+0000 osd.7 (osd.7) 173 : cluster [DBG] 173.7 scrub starts 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.748671+0000 osd.7 (osd.7) 174 : cluster [DBG] 173.7 scrub ok 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:21.748671+0000 osd.7 (osd.7) 174 : cluster [DBG] 173.7 scrub ok 2024-04-26T16:24:22.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.046651+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.046651+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.051520+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.051520+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.055348+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.055348+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.055656+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.055656+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.060386+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.50:0/169579122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.060386+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.50:0/169579122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.061249+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.061249+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.159331+0000 mon.a (mon.0) 1464 : audit [DBG] from='client.? 172.21.15.50:0/743832826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:22.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:22 smithi190 bash[23868]: audit 2024-04-26T16:24:22.159331+0000 mon.a (mon.0) 1464 : audit [DBG] from='client.? 172.21.15.50:0/743832826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:22.551 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:22.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.046651+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-26T16:24:22.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.046651+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.051520+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.051520+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.055348+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.055348+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.055656+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.055656+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.060386+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.50:0/169579122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.060386+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.50:0/169579122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.061249+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.061249+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.159331+0000 mon.a (mon.0) 1464 : audit [DBG] from='client.? 172.21.15.50:0/743832826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:22.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[21463]: audit 2024-04-26T16:24:22.159331+0000 mon.a (mon.0) 1464 : audit [DBG] from='client.? 172.21.15.50:0/743832826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:21.474912+0000 mon.c (mon.2) 264 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:21.474912+0000 mon.c (mon.2) 264 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.517063+0000 mgr.y (mgr.24401) 284 : cluster [DBG] pgmap v274: 353 pgs: 1 active+clean+snaptrim, 352 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.517063+0000 mgr.y (mgr.24401) 284 : cluster [DBG] pgmap v274: 353 pgs: 1 active+clean+snaptrim, 352 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:21.517855+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:21.517855+0000 mon.c (mon.2) 265 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:21.522194+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:21.522194+0000 mon.a (mon.0) 1458 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.744009+0000 osd.7 (osd.7) 173 : cluster [DBG] 173.7 scrub starts 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.744009+0000 osd.7 (osd.7) 173 : cluster [DBG] 173.7 scrub starts 2024-04-26T16:24:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.748671+0000 osd.7 (osd.7) 174 : cluster [DBG] 173.7 scrub ok 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:21.748671+0000 osd.7 (osd.7) 174 : cluster [DBG] 173.7 scrub ok 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.046651+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.046651+0000 mon.a (mon.0) 1459 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "20"}]': finished 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.051520+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.051520+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.055348+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.055348+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.055656+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.055656+0000 mon.a (mon.0) 1462 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.060386+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.50:0/169579122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.060386+0000 mon.b (mon.1) 144 : audit [INF] from='client.? 172.21.15.50:0/169579122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.061249+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.061249+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.159331+0000 mon.a (mon.0) 1464 : audit [DBG] from='client.? 172.21.15.50:0/743832826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:22.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:22 smithi050 bash[29234]: audit 2024-04-26T16:24:22.159331+0000 mon.a (mon.0) 1464 : audit [DBG] from='client.? 172.21.15.50:0/743832826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:22.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:22.683 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:22.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:22.811 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:22.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:22.941 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:22.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:22.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:23.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:23.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:23.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 -- 172.21.15.50:0/1820317786 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 msgr2=0x7fd46c06b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 --2- 172.21.15.50:0/1820317786 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c06b510 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009990 tx=0x7fd45c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 -- 172.21.15.50:0/1820317786 shutdown_connections 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 --2- 172.21.15.50:0/1820317786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd46c10fed0 0x7fd46c1122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 --2- 172.21.15.50:0/1820317786 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c06b510 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 --2- 172.21.15.50:0/1820317786 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 0x7fd46c10f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:23.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 -- 172.21.15.50:0/1820317786 >> 172.21.15.50:0/1820317786 conn(0x7fd46c0792d0 msgr2=0x7fd46c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:23.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 -- 172.21.15.50:0/1820317786 shutdown_connections 2024-04-26T16:24:23.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 -- 172.21.15.50:0/1820317786 wait complete. 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 librados: starting msgr at 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 librados: starting objecter 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 Processor -- start 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 -- start start 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 librados: setting wanted keys 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.106+0000 7fd472a23700 1 librados: calling monclient init 2024-04-26T16:24:23.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c19b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 0x7fd46c19b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd46c10fed0 0x7fd46c19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c114d70 con 0x7fd46c06ba50 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd46c114bf0 con 0x7fd46c10fed0 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd46c114ef0 con 0x7fd46c06b130 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c19b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 0x7fd46c19b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c19b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47678/0 (socket says 172.21.15.50:47678) 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 -- 172.21.15.50:0/3456170994 learned_addr learned my addr 172.21.15.50:0/3456170994 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 -- 172.21.15.50:0/3456170994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd46c10fed0 msgr2=0x7fd46c19fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd470fc0700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd46c10fed0 0x7fd46c19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd46c10fed0 0x7fd46c19fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 -- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 msgr2=0x7fd46c19b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:23.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 0x7fd46c19b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 -- 172.21.15.50:0/3456170994 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd46c1a0320 con 0x7fd46c06b130 2024-04-26T16:24:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46b7fe700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 0x7fd46c19b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:24:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46bfff700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c19b310 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fd45c000c00 tx=0x7fd45c004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 10 client.?.objecter ms_handle_connect 0x7fd46c06b130 2024-04-26T16:24:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:23.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c027070 con 0x7fd46c06b130 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd45c022470 con 0x7fd46c06b130 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd46c1a05b0 con 0x7fd46c06b130 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd45c017650 con 0x7fd46c06b130 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 10 client.40523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd46c1a7e90 con 0x7fd46c06b130 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 librados: init done 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd45c02a070 con 0x7fd46c06b130 2024-04-26T16:24:23.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd46c06a110 con 0x7fd46c06b130 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd454071f80 0x7fd454074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(195..195 src has 87..195) ==== 11085+0+0 (secure 0 0 0) 0x7fd45c093100 con 0x7fd46c06b130 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 10 client.40523.objecter ms_dispatch 0x7fd46c001550 osd_map(195..195 src has 87..195) 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 3 client.40523.objecter handle_osd_map got epochs [195,195] > 0 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 3 client.40523.objecter handle_osd_map decoding full epoch 195 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd46b7fe700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd454071f80 0x7fd454074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.110+0000 7fd4697fa700 20 client.40523.objecter dump_active .. 0 homeless 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.114+0000 7fd4697fa700 10 client.40523.objecter ms_handle_connect 0x7fd454071f80 2024-04-26T16:24:23.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.114+0000 7fd46b7fe700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd454071f80 0x7fd454074440 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fd46c101840 tx=0x7fd460009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:23.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.118+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd45c013070 con 0x7fd46c06b130 2024-04-26T16:24:23.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:23.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:23.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:23.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.206873+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.206873+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.208515+0000 osd.7 (osd.7) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.208515+0000 osd.7 (osd.7) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.279107+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.279107+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:22.476498+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:22.476498+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.783081+0000 osd.7 (osd.7) 177 : cluster [DBG] 173.5 scrub starts 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.783081+0000 osd.7 (osd.7) 177 : cluster [DBG] 173.5 scrub starts 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.787641+0000 osd.7 (osd.7) 178 : cluster [DBG] 173.5 scrub ok 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:22.787641+0000 osd.7 (osd.7) 178 : cluster [DBG] 173.5 scrub ok 2024-04-26T16:24:23.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.049694+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.049694+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.049788+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.049788+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.049837+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.049837+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:23.053439+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: cluster 2024-04-26T16:24:23.053439+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.054893+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:23.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[29234]: audit 2024-04-26T16:24:23.054893+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.206873+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.206873+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.208515+0000 osd.7 (osd.7) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.208515+0000 osd.7 (osd.7) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.279107+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.279107+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:22.476498+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:22.476498+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.783081+0000 osd.7 (osd.7) 177 : cluster [DBG] 173.5 scrub starts 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.783081+0000 osd.7 (osd.7) 177 : cluster [DBG] 173.5 scrub starts 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.787641+0000 osd.7 (osd.7) 178 : cluster [DBG] 173.5 scrub ok 2024-04-26T16:24:23.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:22.787641+0000 osd.7 (osd.7) 178 : cluster [DBG] 173.5 scrub ok 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.049694+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.049694+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.049788+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.049788+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.049837+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.049837+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:23.053439+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: cluster 2024-04-26T16:24:23.053439+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.054893+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:23.333 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:23 smithi050 bash[21463]: audit 2024-04-26T16:24:23.054893+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:23.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:23.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:23.370+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd46c02e490 con 0x7fd46c06b130 2024-04-26T16:24:23.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:23.472 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:23.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.206873+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.206873+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.208515+0000 osd.7 (osd.7) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.208515+0000 osd.7 (osd.7) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.279107+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.279107+0000 mon.a (mon.0) 1465 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:22.476498+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:22.476498+0000 mon.c (mon.2) 266 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.783081+0000 osd.7 (osd.7) 177 : cluster [DBG] 173.5 scrub starts 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.783081+0000 osd.7 (osd.7) 177 : cluster [DBG] 173.5 scrub starts 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.787641+0000 osd.7 (osd.7) 178 : cluster [DBG] 173.5 scrub ok 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:22.787641+0000 osd.7 (osd.7) 178 : cluster [DBG] 173.5 scrub ok 2024-04-26T16:24:23.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.049694+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.049694+0000 mon.a (mon.0) 1466 : audit [INF] from='client.? 172.21.15.50:0/3381001404' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi050-73201-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.049788+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.049788+0000 mon.a (mon.0) 1467 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.049837+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.049837+0000 mon.a (mon.0) 1468 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi050-73193-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:23.053439+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: cluster 2024-04-26T16:24:23.053439+0000 mon.a (mon.0) 1469 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.054893+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:23.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:23 smithi190 bash[23868]: audit 2024-04-26T16:24:23.054893+0000 mon.a (mon.0) 1470 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]: dispatch 2024-04-26T16:24:23.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:23.601 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:23.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:23.731 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:23.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:23.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:23.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:23.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:23.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:24.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.058+0000 7fd4697fa700 1 -- 172.21.15.50:0/3456170994 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v196) ==== 91+0+0 (secure 0 0 0) 0x7fd45c05e490 con 0x7fd46c06b130 2024-04-26T16:24:24.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: Running main() from gmock_main.cc 2024-04-26T16:24:24.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-04-26T16:24:24.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] Global test environment set-up. 2024-04-26T16:24:24.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-04-26T16:24:24.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2040 ms) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2011 ms) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1994 ms) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2018 ms) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8063 ms total) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4138 ms) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4323 ms) 2024-04-26T16:24:24.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4699 ms) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13160 ms total) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2993 ms) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1992 ms) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2012 ms) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2004 ms) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9001 ms total) 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: 2024-04-26T16:24:24.068 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4066 ms) 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4026 ms) 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8092 ms total) 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [----------] Global test environment tear-down 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (56016 ms total) 2024-04-26T16:24:24.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-04-26T16:24:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.070+0000 7fd472a23700 10 librados: watch_flush enter 2024-04-26T16:24:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.070+0000 7fd472a23700 10 librados: watch_flush exit 2024-04-26T16:24:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.070+0000 7fd472a23700 20 client.40523.objecter shutdown clearing up homeless session... 2024-04-26T16:24:24.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.070+0000 7fd472a23700 10 client.40523.objecter successfully canceled tick 2024-04-26T16:24:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd454071f80 msgr2=0x7fd454074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd454071f80 0x7fd454074440 secure :-1 s=READY pgs=3629 cs=0 l=1 rev1=1 crypto rx=0x7fd46c101840 tx=0x7fd460009260 comp rx=0 tx=0).stop 2024-04-26T16:24:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 msgr2=0x7fd46c19b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c19b310 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7fd45c000c00 tx=0x7fd45c004790 comp rx=0 tx=0).stop 2024-04-26T16:24:24.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 shutdown_connections 2024-04-26T16:24:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd454071f80 0x7fd454074440 unknown :-1 s=CLOSED pgs=3629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd46c10fed0 0x7fd46c19fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd46c06b130 0x7fd46c19b310 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 --2- 172.21.15.50:0/3456170994 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd46c06ba50 0x7fd46c19b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:24.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.086+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 >> 172.21.15.50:0/3456170994 conn(0x7fd46c0792d0 msgr2=0x7fd46c1116a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.090+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 shutdown_connections 2024-04-26T16:24:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.090+0000 7fd472a23700 1 -- 172.21.15.50:0/3456170994 wait complete. 2024-04-26T16:24:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:24.090+0000 7fd472a23700 1 librados: shutdown 2024-04-26T16:24:24.093 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:24:24.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:24.137 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:24.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:24.267 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:24.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:23.373175+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.50:0/3456170994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:23.373175+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.50:0/3456170994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:23.373524+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.377 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:23.373524+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:23.477588+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:23.477588+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: cluster 2024-04-26T16:24:23.518336+0000 mgr.y (mgr.24401) 285 : cluster [DBG] pgmap v277: 377 pgs: 64 unknown, 313 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: cluster 2024-04-26T16:24:23.518336+0000 mgr.y (mgr.24401) 285 : cluster [DBG] pgmap v277: 377 pgs: 64 unknown, 313 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.051055+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.051055+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:24.053363+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:24.053363+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:24.053567+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: audit 2024-04-26T16:24:24.053567+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.058249+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-04-26T16:24:24.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.058249+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:23.373175+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.50:0/3456170994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:23.373175+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.50:0/3456170994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:23.373524+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:23.373524+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:23.477588+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:23.477588+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: cluster 2024-04-26T16:24:23.518336+0000 mgr.y (mgr.24401) 285 : cluster [DBG] pgmap v277: 377 pgs: 64 unknown, 313 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: cluster 2024-04-26T16:24:23.518336+0000 mgr.y (mgr.24401) 285 : cluster [DBG] pgmap v277: 377 pgs: 64 unknown, 313 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.051055+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.051055+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:24.053363+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:24.053363+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:24.053567+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:24:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: audit 2024-04-26T16:24:24.053567+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:24:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.058249+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-04-26T16:24:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.058249+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-04-26T16:24:24.399 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:24.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:24.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:23.373175+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.50:0/3456170994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:23.373175+0000 mon.c (mon.2) 267 : audit [INF] from='client.? 172.21.15.50:0/3456170994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:23.373524+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:23.373524+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:23.477588+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:23.477588+0000 mon.c (mon.2) 268 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: cluster 2024-04-26T16:24:23.518336+0000 mgr.y (mgr.24401) 285 : cluster [DBG] pgmap v277: 377 pgs: 64 unknown, 313 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: cluster 2024-04-26T16:24:23.518336+0000 mgr.y (mgr.24401) 285 : cluster [DBG] pgmap v277: 377 pgs: 64 unknown, 313 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.051055+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.051055+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:24.053363+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:24.053363+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? 172.21.15.50:0/45669569' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi050-73420-15"}]': finished 2024-04-26T16:24:24.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:24.053567+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:24:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: audit 2024-04-26T16:24:24.053567+0000 mon.a (mon.0) 1474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:24:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.058249+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-04-26T16:24:24.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:24 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.058249+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2024-04-26T16:24:24.527 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:24.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:24.655 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:24.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:24.784 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:24.877 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:24:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:24:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:24:24.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:24.913 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:24.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:25.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:25.043 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:25.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:25.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:25.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:25.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:25.317 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:25.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:25.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.267747+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.267747+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.269035+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: cluster 2024-04-26T16:24:24.269035+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:24.479254+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:24.479254+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.067795+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.067795+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.072074+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.072074+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.073441+0000 mon.c (mon.2) 270 : audit [INF] from='client.? 172.21.15.50:0/1086462146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.073441+0000 mon.c (mon.2) 270 : audit [INF] from='client.? 172.21.15.50:0/1086462146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.073749+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.073749+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.074308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[21463]: audit 2024-04-26T16:24:25.074308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.267747+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:24:25.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.267747+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:24:25.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.269035+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:24:25.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: cluster 2024-04-26T16:24:24.269035+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:24:25.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:24.479254+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:25.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:24.479254+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:25.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.067795+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-04-26T16:24:25.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.067795+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.072074+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.072074+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.073441+0000 mon.c (mon.2) 270 : audit [INF] from='client.? 172.21.15.50:0/1086462146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.073441+0000 mon.c (mon.2) 270 : audit [INF] from='client.? 172.21.15.50:0/1086462146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.073749+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.073749+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.074308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:25 smithi050 bash[29234]: audit 2024-04-26T16:24:25.074308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:25.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:25.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.267747+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:24:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.267747+0000 osd.0 (osd.0) 159 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:24:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.269035+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:24:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: cluster 2024-04-26T16:24:24.269035+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:24:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:24.479254+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:24.479254+0000 mon.c (mon.2) 269 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.067795+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.067795+0000 mon.a (mon.0) 1476 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.072074+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.072074+0000 mon.a (mon.0) 1477 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.073441+0000 mon.c (mon.2) 270 : audit [INF] from='client.? 172.21.15.50:0/1086462146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.073441+0000 mon.c (mon.2) 270 : audit [INF] from='client.? 172.21.15.50:0/1086462146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.073749+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.073749+0000 mon.a (mon.0) 1478 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.074308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:25 smithi190 bash[23868]: audit 2024-04-26T16:24:25.074308+0000 mon.a (mon.0) 1479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:25.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:25.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:25.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:25.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-04-26T16:24:25.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:25.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:25.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:25.974 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:25.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:26.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:26.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:26.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:26.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:26.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:26.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:25.480751+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:25.480751+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.519206+0000 mgr.y (mgr.24401) 286 : cluster [DBG] pgmap v280: 408 pgs: 128 unknown, 280 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.519206+0000 mgr.y (mgr.24401) 286 : cluster [DBG] pgmap v280: 408 pgs: 128 unknown, 280 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:25.519625+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:25.519625+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:25.521030+0000 mon.a (mon.0) 1480 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:25.521030+0000 mon.a (mon.0) 1480 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029039+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.353 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029039+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029229+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029229+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029379+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029379+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:25.480751+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:25.480751+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.519206+0000 mgr.y (mgr.24401) 286 : cluster [DBG] pgmap v280: 408 pgs: 128 unknown, 280 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.519206+0000 mgr.y (mgr.24401) 286 : cluster [DBG] pgmap v280: 408 pgs: 128 unknown, 280 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:25.519625+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:25.519625+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:25.521030+0000 mon.a (mon.0) 1480 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:25.521030+0000 mon.a (mon.0) 1480 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029039+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029039+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029229+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029229+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029379+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029379+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029599+0000 mon.a (mon.0) 1484 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-26T16:24:26.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.029599+0000 mon.a (mon.0) 1484 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.037897+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.037897+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.262274+0000 mon.a (mon.0) 1486 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.262274+0000 mon.a (mon.0) 1486 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.263773+0000 mon.c (mon.2) 273 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[21463]: audit 2024-04-26T16:24:26.263773+0000 mon.c (mon.2) 273 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029599+0000 mon.a (mon.0) 1484 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.029599+0000 mon.a (mon.0) 1484 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.037897+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.037897+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.262274+0000 mon.a (mon.0) 1486 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.262274+0000 mon.a (mon.0) 1486 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:26.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.263773+0000 mon.c (mon.2) 273 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:26.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:26 smithi050 bash[29234]: audit 2024-04-26T16:24:26.263773+0000 mon.c (mon.2) 273 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:26.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:26.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:26.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:26.496 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:26.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-26T16:24:26.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:25.480751+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:26.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:25.480751+0000 mon.c (mon.2) 271 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:26.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.519206+0000 mgr.y (mgr.24401) 286 : cluster [DBG] pgmap v280: 408 pgs: 128 unknown, 280 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:26.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.519206+0000 mgr.y (mgr.24401) 286 : cluster [DBG] pgmap v280: 408 pgs: 128 unknown, 280 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:25.519625+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:25.519625+0000 mon.c (mon.2) 272 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:25.521030+0000 mon.a (mon.0) 1480 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:25.521030+0000 mon.a (mon.0) 1480 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029039+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029039+0000 mon.a (mon.0) 1481 : audit [INF] from='client.? 172.21.15.50:0/3832529375' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi050-73201-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029229+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029229+0000 mon.a (mon.0) 1482 : audit [INF] from='client.? 172.21.15.50:0/3436258914' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73432-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029379+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029379+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73193-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029599+0000 mon.a (mon.0) 1484 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.029599+0000 mon.a (mon.0) 1484 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "19"}]': finished 2024-04-26T16:24:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.037897+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-04-26T16:24:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.037897+0000 mon.a (mon.0) 1485 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2024-04-26T16:24:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.262274+0000 mon.a (mon.0) 1486 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.262274+0000 mon.a (mon.0) 1486 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.263773+0000 mon.c (mon.2) 273 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[23868]: audit 2024-04-26T16:24:26.263773+0000 mon.c (mon.2) 273 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:26.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:26.627 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:26.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:26.741 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:24:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:24:26.757 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:26.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:26.886 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:26.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:26.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:27.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:27.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:27.143 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:27.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:27.273 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:27.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.958361+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.c scrub starts 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.958361+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.c scrub starts 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.959784+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.c scrub ok 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:25.959784+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.c scrub ok 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.306567+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.306567+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.307694+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.307694+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.365901+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:24:27.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.365901+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.368211+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.368211+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.465332+0000 osd.1 (osd.1) 157 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.465332+0000 osd.1 (osd.1) 157 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.466976+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:26.466976+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: audit 2024-04-26T16:24:26.481953+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: audit 2024-04-26T16:24:26.481953+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.035473+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-04-26T16:24:27.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:27 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.035473+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-04-26T16:24:27.400 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:27.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:27.531 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:27.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.958361+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.c scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.958361+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.c scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.959784+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.c scrub ok 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:25.959784+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.c scrub ok 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.306567+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.306567+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.307694+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.307694+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.365901+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.365901+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.368211+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.368211+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.465332+0000 osd.1 (osd.1) 157 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.465332+0000 osd.1 (osd.1) 157 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:24:27.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.466976+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:26.466976+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: audit 2024-04-26T16:24:26.481953+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: audit 2024-04-26T16:24:26.481953+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.035473+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.035473+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.958361+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.c scrub starts 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.958361+0000 osd.3 (osd.3) 165 : cluster [DBG] 6.c scrub starts 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.959784+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.c scrub ok 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:25.959784+0000 osd.3 (osd.3) 166 : cluster [DBG] 6.c scrub ok 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.306567+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.306567+0000 osd.0 (osd.0) 161 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:24:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.307694+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.307694+0000 osd.0 (osd.0) 162 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.365901+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.365901+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.368211+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.368211+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.465332+0000 osd.1 (osd.1) 157 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.465332+0000 osd.1 (osd.1) 157 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.466976+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:26.466976+0000 osd.1 (osd.1) 158 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: audit 2024-04-26T16:24:26.481953+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: audit 2024-04-26T16:24:26.481953+0000 mon.c (mon.2) 274 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.035473+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-04-26T16:24:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:27 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.035473+0000 mon.a (mon.0) 1487 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2024-04-26T16:24:27.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:27.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:27.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:27.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:24:27.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:27.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:27.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:27.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:28.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:28.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:28.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:28.183 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:28.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:28.312 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:26.663321+0000 mgr.y (mgr.24401) 287 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:26.663321+0000 mgr.y (mgr.24401) 287 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.341729+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.d scrub starts 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.341729+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.d scrub starts 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.349529+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub ok 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.349529+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub ok 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:27.483694+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:27.483694+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:28.313 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.486372+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:28.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:28.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:26.663321+0000 mgr.y (mgr.24401) 287 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:26.663321+0000 mgr.y (mgr.24401) 287 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.341729+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.d scrub starts 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.341729+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.d scrub starts 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.349529+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub ok 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.349529+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub ok 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:27.483694+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:27.483694+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:28.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.486372+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.486372+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.487354+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.487354+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.520111+0000 mgr.y (mgr.24401) 288 : cluster [DBG] pgmap v283: 344 pgs: 2 peering, 342 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:27.520111+0000 mgr.y (mgr.24401) 288 : cluster [DBG] pgmap v283: 344 pgs: 2 peering, 342 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.040640+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.040640+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:28.050774+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:28.050774+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:28.053296+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:28 smithi190 bash[23868]: audit 2024-04-26T16:24:28.053296+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.445 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:28.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:28.575 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:28.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:28.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.486372+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.487354+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.487354+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.520111+0000 mgr.y (mgr.24401) 288 : cluster [DBG] pgmap v283: 344 pgs: 2 peering, 342 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:27.520111+0000 mgr.y (mgr.24401) 288 : cluster [DBG] pgmap v283: 344 pgs: 2 peering, 342 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.040640+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.040640+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:28.050774+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:28.050774+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:28.053296+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[29234]: audit 2024-04-26T16:24:28.053296+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:26.663321+0000 mgr.y (mgr.24401) 287 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:26.663321+0000 mgr.y (mgr.24401) 287 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.341729+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.d scrub starts 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.341729+0000 osd.5 (osd.5) 153 : cluster [DBG] 6.d scrub starts 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.349529+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub ok 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.349529+0000 osd.5 (osd.5) 154 : cluster [DBG] 6.d scrub ok 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:27.483694+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:27.483694+0000 mon.c (mon.2) 275 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.486372+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.486372+0000 osd.6 (osd.6) 101 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.487354+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.487354+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.520111+0000 mgr.y (mgr.24401) 288 : cluster [DBG] pgmap v283: 344 pgs: 2 peering, 342 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:27.520111+0000 mgr.y (mgr.24401) 288 : cluster [DBG] pgmap v283: 344 pgs: 2 peering, 342 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.040640+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.040640+0000 mon.a (mon.0) 1488 : cluster [DBG] osdmap e200: 8 total, 8 up, 8 in 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:28.050774+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:28.050774+0000 mon.a (mon.0) 1489 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:28.053296+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:28 smithi050 bash[21463]: audit 2024-04-26T16:24:28.053296+0000 mon.a (mon.0) 1490 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:28.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:28.704 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:28.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:28.833 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:28.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:28.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:28.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:28.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:29.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:29.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:29.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:29.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:29.251 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:29.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:29.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.303263+0000 mon.a (mon.0) 1491 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:29.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.303263+0000 mon.a (mon.0) 1491 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:29.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.381702+0000 osd.5 (osd.5) 155 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:24:29.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.381702+0000 osd.5 (osd.5) 155 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:24:29.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.384884+0000 osd.5 (osd.5) 156 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.384884+0000 osd.5 (osd.5) 156 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:28.485384+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:28.485384+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.037932+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.037932+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.038223+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.038223+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.045447+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.045447+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.145622+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.145622+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.146877+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.146877+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.147284+0000 mon.a (mon.0) 1497 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[21463]: audit 2024-04-26T16:24:29.147284+0000 mon.a (mon.0) 1497 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.303263+0000 mon.a (mon.0) 1491 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.303263+0000 mon.a (mon.0) 1491 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.381702+0000 osd.5 (osd.5) 155 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.381702+0000 osd.5 (osd.5) 155 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.384884+0000 osd.5 (osd.5) 156 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.384884+0000 osd.5 (osd.5) 156 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:28.485384+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:29.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:28.485384+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.037932+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.037932+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.038223+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.038223+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.045447+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.045447+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.145622+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.145622+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.146877+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.146877+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.147284+0000 mon.a (mon.0) 1497 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:29.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:29 smithi050 bash[29234]: audit 2024-04-26T16:24:29.147284+0000 mon.a (mon.0) 1497 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:29.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:29.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:29.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.303263+0000 mon.a (mon.0) 1491 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:29.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.303263+0000 mon.a (mon.0) 1491 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:29.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.381702+0000 osd.5 (osd.5) 155 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:24:29.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.381702+0000 osd.5 (osd.5) 155 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:24:29.495 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.384884+0000 osd.5 (osd.5) 156 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.384884+0000 osd.5 (osd.5) 156 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:28.485384+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:28.485384+0000 mon.c (mon.2) 276 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.037932+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.037932+0000 mon.a (mon.0) 1492 : audit [INF] from='client.? 172.21.15.50:0/2201509339' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.038223+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.038223+0000 mon.a (mon.0) 1493 : audit [INF] from='client.? 172.21.15.50:0/120001074' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi050-73193-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.045447+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.045447+0000 mon.a (mon.0) 1494 : cluster [DBG] osdmap e201: 8 total, 8 up, 8 in 2024-04-26T16:24:29.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.145622+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.145622+0000 mon.a (mon.0) 1495 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.146877+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.146877+0000 mon.a (mon.0) 1496 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:29.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.147284+0000 mon.a (mon.0) 1497 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:29.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:29 smithi190 bash[23868]: audit 2024-04-26T16:24:29.147284+0000 mon.a (mon.0) 1497 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:29.510 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:29.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:29.641 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:29.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:29.772 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:29.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:29.902 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:29.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:30.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:30.031 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:30.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:30.160 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:30.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:30.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:30.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:30.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:30.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:30.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:30.556 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:30.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:30.559 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:30.559 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:30.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.926224+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.c scrub starts 2024-04-26T16:24:30.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.926224+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.c scrub starts 2024-04-26T16:24:30.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.928420+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.c scrub ok 2024-04-26T16:24:30.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.928420+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.c scrub ok 2024-04-26T16:24:30.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.984842+0000 osd.3 (osd.3) 167 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.984842+0000 osd.3 (osd.3) 167 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.987065+0000 osd.3 (osd.3) 168 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:28.987065+0000 osd.3 (osd.3) 168 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.306813+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.306813+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.309286+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.309286+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.334073+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.334073+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:24:30.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.336136+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.336136+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:29.487214+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:29.487214+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.521230+0000 mgr.y (mgr.24401) 289 : cluster [DBG] pgmap v286: 375 pgs: 64 unknown, 1 peering, 310 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.521230+0000 mgr.y (mgr.24401) 289 : cluster [DBG] pgmap v286: 375 pgs: 64 unknown, 1 peering, 310 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:29.521883+0000 mon.c (mon.2) 278 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:29.521883+0000 mon.c (mon.2) 278 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:29.525210+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:29.525210+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:30.040107+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:30.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:30.040107+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:30.040293+0000 mon.a (mon.0) 1500 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-26T16:24:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:30.040293+0000 mon.a (mon.0) 1500 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-26T16:24:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.046059+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-04-26T16:24:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.046059+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-04-26T16:24:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:30.047436+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:30.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:30 smithi190 bash[23868]: audit 2024-04-26T16:24:30.047436+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:30.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.926224+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.c scrub starts 2024-04-26T16:24:30.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.926224+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.c scrub starts 2024-04-26T16:24:30.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.928420+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.c scrub ok 2024-04-26T16:24:30.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.928420+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.c scrub ok 2024-04-26T16:24:30.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.984842+0000 osd.3 (osd.3) 167 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.984842+0000 osd.3 (osd.3) 167 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.987065+0000 osd.3 (osd.3) 168 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:28.987065+0000 osd.3 (osd.3) 168 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.306813+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.306813+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.309286+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.309286+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.334073+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:24:30.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.334073+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.336136+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.336136+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:29.487214+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:29.487214+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.521230+0000 mgr.y (mgr.24401) 289 : cluster [DBG] pgmap v286: 375 pgs: 64 unknown, 1 peering, 310 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.521230+0000 mgr.y (mgr.24401) 289 : cluster [DBG] pgmap v286: 375 pgs: 64 unknown, 1 peering, 310 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:29.521883+0000 mon.c (mon.2) 278 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:29.521883+0000 mon.c (mon.2) 278 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:29.525210+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:29.525210+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:30.040107+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:30.040107+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:30.040293+0000 mon.a (mon.0) 1500 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:30.040293+0000 mon.a (mon.0) 1500 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-26T16:24:30.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.046059+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.046059+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:30.047436+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[29234]: audit 2024-04-26T16:24:30.047436+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.926224+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.c scrub starts 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.926224+0000 osd.2 (osd.2) 111 : cluster [DBG] 2.c scrub starts 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.928420+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.c scrub ok 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.928420+0000 osd.2 (osd.2) 112 : cluster [DBG] 2.c scrub ok 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.984842+0000 osd.3 (osd.3) 167 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:24:30.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.984842+0000 osd.3 (osd.3) 167 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.987065+0000 osd.3 (osd.3) 168 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:28.987065+0000 osd.3 (osd.3) 168 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.306813+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.306813+0000 osd.0 (osd.0) 163 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.309286+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.309286+0000 osd.0 (osd.0) 164 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.334073+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.334073+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.336136+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.336136+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:29.487214+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:29.487214+0000 mon.c (mon.2) 277 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.521230+0000 mgr.y (mgr.24401) 289 : cluster [DBG] pgmap v286: 375 pgs: 64 unknown, 1 peering, 310 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:30.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.521230+0000 mgr.y (mgr.24401) 289 : cluster [DBG] pgmap v286: 375 pgs: 64 unknown, 1 peering, 310 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:29.521883+0000 mon.c (mon.2) 278 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:29.521883+0000 mon.c (mon.2) 278 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:29.525210+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:29.525210+0000 mon.a (mon.0) 1498 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:30.040107+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:30.040107+0000 mon.a (mon.0) 1499 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:30.040293+0000 mon.a (mon.0) 1500 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:30.040293+0000 mon.a (mon.0) 1500 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "18"}]': finished 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.046059+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.046059+0000 mon.a (mon.0) 1501 : cluster [DBG] osdmap e202: 8 total, 8 up, 8 in 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:30.047436+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:30.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:30 smithi050 bash[21463]: audit 2024-04-26T16:24:30.047436+0000 mon.a (mon.0) 1502 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:30.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:30.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:30.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:30.819 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:30.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:30.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:30.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:31.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:31.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:31.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:31.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:31.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:31.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:31.482 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:31.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:31.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:31.589 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:31.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.962012+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:24:31.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.962012+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:24:31.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.964466+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:24:31.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.964466+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:24:31.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.988946+0000 osd.3 (osd.3) 169 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.988946+0000 osd.3 (osd.3) 169 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.992313+0000 osd.3 (osd.3) 170 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:29.992313+0000 osd.3 (osd.3) 170 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:30.488893+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:30.488893+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.580444+0000 osd.7 (osd.7) 179 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.580444+0000 osd.7 (osd.7) 179 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.582558+0000 osd.7 (osd.7) 180 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.582558+0000 osd.7 (osd.7) 180 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.086439+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.086439+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.089664+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.089664+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.091010+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.50:0/3330541584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.091010+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.50:0/3330541584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.091342+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.50:0/1064514980' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.091342+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.50:0/1064514980' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.091344+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[29234]: audit 2024-04-26T16:24:31.091344+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.962012+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.962012+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.964466+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.964466+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.988946+0000 osd.3 (osd.3) 169 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.988946+0000 osd.3 (osd.3) 169 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.992313+0000 osd.3 (osd.3) 170 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:29.992313+0000 osd.3 (osd.3) 170 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:24:31.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:30.488893+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:31.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:30.488893+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:31.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.580444+0000 osd.7 (osd.7) 179 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:24:31.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.580444+0000 osd.7 (osd.7) 179 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:24:31.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.582558+0000 osd.7 (osd.7) 180 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:24:31.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.582558+0000 osd.7 (osd.7) 180 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:24:31.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.086439+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.086439+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.089664+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.089664+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.091010+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.50:0/3330541584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.091010+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.50:0/3330541584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.091342+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.50:0/1064514980' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.091342+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.50:0/1064514980' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.091344+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:31 smithi050 bash[21463]: audit 2024-04-26T16:24:31.091344+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:31.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.962012+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.962012+0000 osd.2 (osd.2) 113 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.964466+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.964466+0000 osd.2 (osd.2) 114 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.988946+0000 osd.3 (osd.3) 169 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.988946+0000 osd.3 (osd.3) 169 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:24:31.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.992313+0000 osd.3 (osd.3) 170 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:29.992313+0000 osd.3 (osd.3) 170 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:30.488893+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:30.488893+0000 mon.c (mon.2) 279 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.580444+0000 osd.7 (osd.7) 179 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.580444+0000 osd.7 (osd.7) 179 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.582558+0000 osd.7 (osd.7) 180 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.582558+0000 osd.7 (osd.7) 180 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.086439+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.086439+0000 mon.a (mon.0) 1503 : cluster [DBG] osdmap e203: 8 total, 8 up, 8 in 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.089664+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.089664+0000 mon.a (mon.0) 1504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.091010+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.50:0/3330541584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.091010+0000 mon.b (mon.1) 145 : audit [INF] from='client.? 172.21.15.50:0/3330541584' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.091342+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.50:0/1064514980' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.091342+0000 mon.b (mon.1) 146 : audit [INF] from='client.? 172.21.15.50:0/1064514980' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.091344+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:31 smithi190 bash[23868]: audit 2024-04-26T16:24:31.091344+0000 mon.a (mon.0) 1505 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:31.744 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:31.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:31.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:31.876 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:31.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:31.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:31.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:32.005 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:32.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:32.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:32.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:32.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:32.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:32.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.987376+0000 osd.3 (osd.3) 171 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.987376+0000 osd.3 (osd.3) 171 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.989281+0000 osd.3 (osd.3) 172 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:30.989281+0000 osd.3 (osd.3) 172 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.305915+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.305915+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.308243+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.308243+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.375242+0000 osd.6 (osd.6) 105 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.375242+0000 osd.6 (osd.6) 105 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:24:32.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.376369+0000 osd.6 (osd.6) 106 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.376369+0000 osd.6 (osd.6) 106 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:31.490212+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:31.490212+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.524610+0000 mgr.y (mgr.24401) 290 : cluster [DBG] pgmap v289: 375 pgs: 2 clean+premerge+peered, 64 creating+peering, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.524610+0000 mgr.y (mgr.24401) 290 : cluster [DBG] pgmap v289: 375 pgs: 2 clean+premerge+peered, 64 creating+peering, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.594398+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.594398+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.596741+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:31.596741+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:32.049008+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:32.049008+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:32.506 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:32.049169+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:32.049169+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:32.049349+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: audit 2024-04-26T16:24:32.049349+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.059546+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.059546+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.987376+0000 osd.3 (osd.3) 171 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:24:32.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.987376+0000 osd.3 (osd.3) 171 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.989281+0000 osd.3 (osd.3) 172 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:30.989281+0000 osd.3 (osd.3) 172 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.305915+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.305915+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.308243+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.308243+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.375242+0000 osd.6 (osd.6) 105 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.375242+0000 osd.6 (osd.6) 105 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.376369+0000 osd.6 (osd.6) 106 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.376369+0000 osd.6 (osd.6) 106 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:31.490212+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:31.490212+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.524610+0000 mgr.y (mgr.24401) 290 : cluster [DBG] pgmap v289: 375 pgs: 2 clean+premerge+peered, 64 creating+peering, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:32.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.524610+0000 mgr.y (mgr.24401) 290 : cluster [DBG] pgmap v289: 375 pgs: 2 clean+premerge+peered, 64 creating+peering, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.594398+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.594398+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.596741+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:31.596741+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:32.049008+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:32.049008+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:32.049169+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:32.049169+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:32.049349+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: audit 2024-04-26T16:24:32.049349+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.059546+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-04-26T16:24:32.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:32 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.059546+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-04-26T16:24:32.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:32.614 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:32.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:32.647 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:32.750 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:32.779 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:32.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.987376+0000 osd.3 (osd.3) 171 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:24:32.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.987376+0000 osd.3 (osd.3) 171 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:24:32.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.989281+0000 osd.3 (osd.3) 172 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:24:32.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:30.989281+0000 osd.3 (osd.3) 172 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:24:32.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.305915+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:24:32.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.305915+0000 osd.0 (osd.0) 165 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.308243+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.308243+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.375242+0000 osd.6 (osd.6) 105 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.375242+0000 osd.6 (osd.6) 105 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.376369+0000 osd.6 (osd.6) 106 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.376369+0000 osd.6 (osd.6) 106 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:31.490212+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:31.490212+0000 mon.c (mon.2) 280 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.524610+0000 mgr.y (mgr.24401) 290 : cluster [DBG] pgmap v289: 375 pgs: 2 clean+premerge+peered, 64 creating+peering, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.524610+0000 mgr.y (mgr.24401) 290 : cluster [DBG] pgmap v289: 375 pgs: 2 clean+premerge+peered, 64 creating+peering, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.594398+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:24:32.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.594398+0000 osd.7 (osd.7) 181 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.596741+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:31.596741+0000 osd.7 (osd.7) 182 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:32.049008+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:32.049008+0000 mon.a (mon.0) 1506 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi050-73432-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:32.049169+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:32.049169+0000 mon.a (mon.0) 1507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi050-73193-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:32.049349+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: audit 2024-04-26T16:24:32.049349+0000 mon.a (mon.0) 1508 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi050-73201-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.059546+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-04-26T16:24:32.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:32 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.059546+0000 mon.a (mon.0) 1509 : cluster [DBG] osdmap e204: 8 total, 8 up, 8 in 2024-04-26T16:24:32.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:32.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:32.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:33.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:33.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:33.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:33.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:33.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:33.301 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:33.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:33.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:33.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:33.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:33.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.012142+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.012142+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.014390+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.014390+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: audit 2024-04-26T16:24:32.492176+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: audit 2024-04-26T16:24:32.492176+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.641099+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.641099+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.643929+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.643929+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.792332+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.e scrub starts 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.792332+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.e scrub starts 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.795327+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.e scrub ok 2024-04-26T16:24:33.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.795327+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.e scrub ok 2024-04-26T16:24:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.055444+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-04-26T16:24:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:33 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.055444+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-04-26T16:24:33.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.012142+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:24:33.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.012142+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:24:33.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.014390+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:24:33.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.014390+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:24:33.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: audit 2024-04-26T16:24:32.492176+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: audit 2024-04-26T16:24:32.492176+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.641099+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.641099+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.643929+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.643929+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.792332+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.e scrub starts 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.792332+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.e scrub starts 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.795327+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.e scrub ok 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.795327+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.e scrub ok 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.055444+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-04-26T16:24:33.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.055444+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.012142+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.012142+0000 osd.3 (osd.3) 173 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.014390+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.014390+0000 osd.3 (osd.3) 174 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: audit 2024-04-26T16:24:32.492176+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: audit 2024-04-26T16:24:32.492176+0000 mon.c (mon.2) 281 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.641099+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.641099+0000 osd.7 (osd.7) 183 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.643929+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.643929+0000 osd.7 (osd.7) 184 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.792332+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.e scrub starts 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.792332+0000 osd.4 (osd.4) 137 : cluster [DBG] 5.e scrub starts 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.795327+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.e scrub ok 2024-04-26T16:24:33.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.795327+0000 osd.4 (osd.4) 138 : cluster [DBG] 5.e scrub ok 2024-04-26T16:24:33.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.055444+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-04-26T16:24:33.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:33 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.055444+0000 mon.a (mon.0) 1510 : cluster [DBG] osdmap e205: 8 total, 8 up, 8 in 2024-04-26T16:24:33.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:33.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:33.794 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:33.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:33.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:33.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:33.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:33.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:34.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:34.094 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:34.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:34.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:34.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:34.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:34.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:34.489 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:34.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:34.604 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:24:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:24:34.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.938718+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.a scrub starts 2024-04-26T16:24:34.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.938718+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.a scrub starts 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.940500+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.a scrub ok 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:32.940500+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.a scrub ok 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:33.493817+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:33.493817+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.525332+0000 mgr.y (mgr.24401) 291 : cluster [DBG] pgmap v292: 318 pgs: 1 clean+premerge+peered, 8 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.525332+0000 mgr.y (mgr.24401) 291 : cluster [DBG] pgmap v292: 318 pgs: 1 clean+premerge+peered, 8 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:33.525866+0000 mon.c (mon.2) 283 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:33.525866+0000 mon.c (mon.2) 283 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:33.528971+0000 mon.a (mon.0) 1511 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:33.528971+0000 mon.a (mon.0) 1511 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.813529+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.e scrub starts 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.813529+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.e scrub starts 2024-04-26T16:24:34.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.816063+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.e scrub ok 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.816063+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.e scrub ok 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.054184+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.054184+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.058880+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.058880+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.063587+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.50:0/2324850962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.063587+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.50:0/2324850962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.064243+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.064243+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.067184+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.50:0/3576143342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.067184+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.50:0/3576143342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.068015+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[29234]: audit 2024-04-26T16:24:34.068015+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.938718+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.a scrub starts 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.938718+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.a scrub starts 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.940500+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.a scrub ok 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:32.940500+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.a scrub ok 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:33.493817+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:33.493817+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.525332+0000 mgr.y (mgr.24401) 291 : cluster [DBG] pgmap v292: 318 pgs: 1 clean+premerge+peered, 8 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.525332+0000 mgr.y (mgr.24401) 291 : cluster [DBG] pgmap v292: 318 pgs: 1 clean+premerge+peered, 8 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:34.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:33.525866+0000 mon.c (mon.2) 283 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:33.525866+0000 mon.c (mon.2) 283 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:33.528971+0000 mon.a (mon.0) 1511 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:33.528971+0000 mon.a (mon.0) 1511 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.813529+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.e scrub starts 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.813529+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.e scrub starts 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.816063+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.e scrub ok 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.816063+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.e scrub ok 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.054184+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.054184+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.058880+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.058880+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.063587+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.50:0/2324850962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.063587+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.50:0/2324850962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.064243+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.064243+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.609 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.067184+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.50:0/3576143342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.609 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.067184+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.50:0/3576143342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.609 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.068015+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.609 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:34 smithi050 bash[21463]: audit 2024-04-26T16:24:34.068015+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:34.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:34.734 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.938718+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.a scrub starts 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.938718+0000 osd.2 (osd.2) 115 : cluster [DBG] 5.a scrub starts 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.940500+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.a scrub ok 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:32.940500+0000 osd.2 (osd.2) 116 : cluster [DBG] 5.a scrub ok 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:33.493817+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:33.493817+0000 mon.c (mon.2) 282 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.525332+0000 mgr.y (mgr.24401) 291 : cluster [DBG] pgmap v292: 318 pgs: 1 clean+premerge+peered, 8 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.525332+0000 mgr.y (mgr.24401) 291 : cluster [DBG] pgmap v292: 318 pgs: 1 clean+premerge+peered, 8 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 3 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:33.525866+0000 mon.c (mon.2) 283 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:33.525866+0000 mon.c (mon.2) 283 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:33.528971+0000 mon.a (mon.0) 1511 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:33.528971+0000 mon.a (mon.0) 1511 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.813529+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.e scrub starts 2024-04-26T16:24:34.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.813529+0000 osd.4 (osd.4) 139 : cluster [DBG] 6.e scrub starts 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.816063+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.e scrub ok 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.816063+0000 osd.4 (osd.4) 140 : cluster [DBG] 6.e scrub ok 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.054184+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.054184+0000 mon.a (mon.0) 1512 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "17"}]': finished 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.058880+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.058880+0000 mon.a (mon.0) 1513 : cluster [DBG] osdmap e206: 8 total, 8 up, 8 in 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.063587+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.50:0/2324850962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.063587+0000 mon.c (mon.2) 284 : audit [INF] from='client.? 172.21.15.50:0/2324850962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.064243+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.064243+0000 mon.a (mon.0) 1514 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.067184+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.50:0/3576143342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.067184+0000 mon.c (mon.2) 285 : audit [INF] from='client.? 172.21.15.50:0/3576143342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.737 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.068015+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.737 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:34 smithi190 bash[23868]: audit 2024-04-26T16:24:34.068015+0000 mon.a (mon.0) 1515 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:34.749 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:34.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:34.881 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:34.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:34.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:35.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:35.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:35.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:35.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:35.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:35.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:35.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:35.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.431507+0000 osd.5 (osd.5) 157 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.431507+0000 osd.5 (osd.5) 157 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.433594+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.433594+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.967770+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.967770+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.969032+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:33.969032+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.340533+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.340533+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.460229+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:24:35.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.460229+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.463441+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.463441+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.495215+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.495215+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.675590+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.675590+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.679006+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.679006+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.679433+0000 mon.a (mon.0) 1517 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:34.679433+0000 mon.a (mon.0) 1517 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.829506+0000 osd.4 (osd.4) 141 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.829506+0000 osd.4 (osd.4) 141 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.832770+0000 osd.4 (osd.4) 142 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.832770+0000 osd.4 (osd.4) 142 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.056822+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.056822+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.056964+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.056964+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.057114+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.057114+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.061992+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-04-26T16:24:35.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.061992+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-04-26T16:24:35.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.070016+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2"}]: dispatch 2024-04-26T16:24:35.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.070016+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2"}]: dispatch 2024-04-26T16:24:35.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.071030+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.071030+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.071714+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.356 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:35 smithi190 bash[23868]: audit 2024-04-26T16:24:35.071714+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:35.397 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:35.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:35.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:35.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.431507+0000 osd.5 (osd.5) 157 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:24:35.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.431507+0000 osd.5 (osd.5) 157 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.433594+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.433594+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.967770+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.967770+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.969032+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:33.969032+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.340533+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.340533+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.460229+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.460229+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.463441+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.463441+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:24:35.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.495215+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.495215+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.675590+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.675590+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.679006+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.679006+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.679433+0000 mon.a (mon.0) 1517 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:34.679433+0000 mon.a (mon.0) 1517 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.431507+0000 osd.5 (osd.5) 157 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.431507+0000 osd.5 (osd.5) 157 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:24:35.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.433594+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.433594+0000 osd.5 (osd.5) 158 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.967770+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.967770+0000 osd.2 (osd.2) 117 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.969032+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:33.969032+0000 osd.2 (osd.2) 118 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.340533+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.340533+0000 mon.a (mon.0) 1516 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.460229+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.460229+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.463441+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.463441+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.495215+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.495215+0000 mon.c (mon.2) 286 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:35.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.675590+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.675590+0000 mon.c (mon.2) 287 : audit [DBG] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.679006+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.679006+0000 mon.c (mon.2) 288 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.679433+0000 mon.a (mon.0) 1517 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:34.679433+0000 mon.a (mon.0) 1517 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.829506+0000 osd.4 (osd.4) 141 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.829506+0000 osd.4 (osd.4) 141 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.832770+0000 osd.4 (osd.4) 142 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.832770+0000 osd.4 (osd.4) 142 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.056822+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.056822+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.056964+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.056964+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.057114+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.057114+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.061992+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.061992+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.070016+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2"}]: dispatch 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.070016+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2"}]: dispatch 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.071030+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.071030+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.071714+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[21463]: audit 2024-04-26T16:24:35.071714+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.829506+0000 osd.4 (osd.4) 141 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.829506+0000 osd.4 (osd.4) 141 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.832770+0000 osd.4 (osd.4) 142 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.832770+0000 osd.4 (osd.4) 142 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.056822+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.056822+0000 mon.a (mon.0) 1518 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi050-73193-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.056964+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.056964+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi050-73201-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.057114+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.057114+0000 mon.a (mon.0) 1520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.061992+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.061992+0000 mon.a (mon.0) 1521 : cluster [DBG] osdmap e207: 8 total, 8 up, 8 in 2024-04-26T16:24:35.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.070016+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2"}]: dispatch 2024-04-26T16:24:35.512 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.070016+0000 mon.c (mon.2) 289 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2"}]: dispatch 2024-04-26T16:24:35.512 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.071030+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.512 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.071030+0000 mon.c (mon.2) 290 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.512 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.071714+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.512 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:35 smithi050 bash[29234]: audit 2024-04-26T16:24:35.071714+0000 mon.a (mon.0) 1522 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:35.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:35.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:35.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:35.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:35.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:35.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:36.024 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:36.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:36.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:36.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:36.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:36.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:36.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:36.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:36.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:36.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.471260+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.471260+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.475080+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.475080+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.971233+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.b scrub starts 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.971233+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.b scrub starts 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.973862+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.b scrub ok 2024-04-26T16:24:36.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:34.973862+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.b scrub ok 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.393771+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.393771+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.397601+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.397601+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:35.496804+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:35.496804+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.526041+0000 mgr.y (mgr.24401) 292 : cluster [DBG] pgmap v295: 382 pgs: 1 clean+premerge+peered, 72 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.526041+0000 mgr.y (mgr.24401) 292 : cluster [DBG] pgmap v295: 382 pgs: 1 clean+premerge+peered, 72 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.790148+0000 osd.4 (osd.4) 143 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.790148+0000 osd.4 (osd.4) 143 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.794159+0000 osd.4 (osd.4) 144 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.794159+0000 osd.4 (osd.4) 144 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:24:36.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.032778+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.032778+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.036918+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.036918+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.044141+0000 mon.c (mon.2) 292 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.044141+0000 mon.c (mon.2) 292 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.045414+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.045414+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.055426+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[29234]: audit 2024-04-26T16:24:36.055426+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.471260+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.471260+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.475080+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.475080+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.971233+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.b scrub starts 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.971233+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.b scrub starts 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.973862+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.b scrub ok 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:34.973862+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.b scrub ok 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.393771+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.393771+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.397601+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.397601+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:35.496804+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:36.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:35.496804+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.526041+0000 mgr.y (mgr.24401) 292 : cluster [DBG] pgmap v295: 382 pgs: 1 clean+premerge+peered, 72 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.526041+0000 mgr.y (mgr.24401) 292 : cluster [DBG] pgmap v295: 382 pgs: 1 clean+premerge+peered, 72 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.790148+0000 osd.4 (osd.4) 143 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.790148+0000 osd.4 (osd.4) 143 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.794159+0000 osd.4 (osd.4) 144 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.794159+0000 osd.4 (osd.4) 144 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.032778+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.032778+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.036918+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.036918+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.044141+0000 mon.c (mon.2) 292 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.044141+0000 mon.c (mon.2) 292 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.045414+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.045414+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.055426+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:36 smithi050 bash[21463]: audit 2024-04-26T16:24:36.055426+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.471260+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:24:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.471260+0000 osd.5 (osd.5) 159 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:24:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.475080+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:24:36.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.475080+0000 osd.5 (osd.5) 160 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.971233+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.b scrub starts 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.971233+0000 osd.2 (osd.2) 119 : cluster [DBG] 5.b scrub starts 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.973862+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.b scrub ok 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:34.973862+0000 osd.2 (osd.2) 120 : cluster [DBG] 5.b scrub ok 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.393771+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.393771+0000 osd.6 (osd.6) 107 : cluster [DBG] 3.a scrub starts 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.397601+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.397601+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.a scrub ok 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:35.496804+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:35.496804+0000 mon.c (mon.2) 291 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.526041+0000 mgr.y (mgr.24401) 292 : cluster [DBG] pgmap v295: 382 pgs: 1 clean+premerge+peered, 72 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.526041+0000 mgr.y (mgr.24401) 292 : cluster [DBG] pgmap v295: 382 pgs: 1 clean+premerge+peered, 72 unknown, 309 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.790148+0000 osd.4 (osd.4) 143 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:24:36.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.790148+0000 osd.4 (osd.4) 143 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.794159+0000 osd.4 (osd.4) 144 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.794159+0000 osd.4 (osd.4) 144 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.032778+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.032778+0000 mon.a (mon.0) 1523 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi050-73363-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.036918+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.036918+0000 mon.a (mon.0) 1524 : cluster [DBG] osdmap e208: 8 total, 8 up, 8 in 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.044141+0000 mon.c (mon.2) 292 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.044141+0000 mon.c (mon.2) 292 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.045414+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.045414+0000 mon.c (mon.2) 293 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.055426+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[23868]: audit 2024-04-26T16:24:36.055426+0000 mon.a (mon.0) 1525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-04-26T16:24:36.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:36.667 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:36.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:36.797 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:36.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:36.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:24:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:24:36.928 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:36.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:37.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:37.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:37.175 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:37.196 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:37.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:37.323 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:37.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:37.453 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:37.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:37.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.441411+0000 osd.5 (osd.5) 161 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:24:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.441411+0000 osd.5 (osd.5) 161 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:24:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.443112+0000 osd.5 (osd.5) 162 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:35.443112+0000 osd.5 (osd.5) 162 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.080300+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.080300+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.099622+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.099622+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:36.498068+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:36.498068+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.577731+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.577731+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.579502+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.579502+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.927687+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:24:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.927687+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.928878+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:36.928878+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.037427+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.037427+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.047395+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.047395+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.054822+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.50:0/3713917263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.054822+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.50:0/3713917263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.056320+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.056320+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.056876+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.50:0/1480067862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.056876+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.50:0/1480067862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.057976+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.057976+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.058325+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.058325+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.058611+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:37 smithi190 bash[23868]: audit 2024-04-26T16:24:37.058611+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.584 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:37.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.441411+0000 osd.5 (osd.5) 161 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:24:37.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.441411+0000 osd.5 (osd.5) 161 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:24:37.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.443112+0000 osd.5 (osd.5) 162 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:24:37.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:35.443112+0000 osd.5 (osd.5) 162 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:24:37.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.080300+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:24:37.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.080300+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.099622+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.099622+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:36.498068+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:36.498068+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.577731+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.577731+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.579502+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.579502+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.927687+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.927687+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.928878+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:36.928878+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:24:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.037427+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.037427+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.047395+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.047395+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.054822+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.50:0/3713917263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.054822+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.50:0/3713917263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.441411+0000 osd.5 (osd.5) 161 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.441411+0000 osd.5 (osd.5) 161 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:24:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.443112+0000 osd.5 (osd.5) 162 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:35.443112+0000 osd.5 (osd.5) 162 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.080300+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.080300+0000 osd.3 (osd.3) 175 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.099622+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.099622+0000 osd.3 (osd.3) 176 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:36.498068+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:36.498068+0000 mon.c (mon.2) 294 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.577731+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.577731+0000 osd.1 (osd.1) 161 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.579502+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.579502+0000 osd.1 (osd.1) 162 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.927687+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.927687+0000 osd.2 (osd.2) 121 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.928878+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:24:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:36.928878+0000 osd.2 (osd.2) 122 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.037427+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.037427+0000 mon.a (mon.0) 1526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.047395+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.047395+0000 mon.a (mon.0) 1527 : cluster [DBG] osdmap e209: 8 total, 8 up, 8 in 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.054822+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.50:0/3713917263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.054822+0000 mon.c (mon.2) 295 : audit [INF] from='client.? 172.21.15.50:0/3713917263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.056320+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.056320+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.056876+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.50:0/1480067862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.056876+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.50:0/1480067862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.057976+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.057976+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.058325+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.058325+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.058611+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[29234]: audit 2024-04-26T16:24:37.058611+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.056320+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.056320+0000 mon.c (mon.2) 296 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.056876+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.50:0/1480067862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.056876+0000 mon.c (mon.2) 297 : audit [INF] from='client.? 172.21.15.50:0/1480067862' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.057976+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.057976+0000 mon.a (mon.0) 1528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.058325+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.058325+0000 mon.a (mon.0) 1529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-04-26T16:24:37.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.058611+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:37 smithi050 bash[21463]: audit 2024-04-26T16:24:37.058611+0000 mon.a (mon.0) 1530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:37.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:37.712 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:37.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:37.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:37.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:37.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:37.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:38.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:38.116 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:38.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:38.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:38.246 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:38.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:38.376 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:38.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:38.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:38.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:38.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:36.671258+0000 mgr.y (mgr.24401) 293 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:38.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:36.671258+0000 mgr.y (mgr.24401) 293 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:38.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.083207+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:24:38.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.083207+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:24:38.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.085068+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:24:38.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.085068+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.351037+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.351037+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.355689+0000 osd.0 (osd.0) 168 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.355689+0000 osd.0 (osd.0) 168 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:37.499392+0000 mon.c (mon.2) 298 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:37.499392+0000 mon.c (mon.2) 298 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.527010+0000 mgr.y (mgr.24401) 294 : cluster [DBG] pgmap v298: 381 pgs: 64 unknown, 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.527010+0000 mgr.y (mgr.24401) 294 : cluster [DBG] pgmap v298: 381 pgs: 64 unknown, 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:37.527706+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:37.527706+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:37.530899+0000 mon.a (mon.0) 1531 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:37.530899+0000 mon.a (mon.0) 1531 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.706909+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.706909+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.707035+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:36.671258+0000 mgr.y (mgr.24401) 293 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:36.671258+0000 mgr.y (mgr.24401) 293 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.083207+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.083207+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.085068+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:24:38.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.085068+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.351037+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.351037+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.355689+0000 osd.0 (osd.0) 168 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.355689+0000 osd.0 (osd.0) 168 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:37.499392+0000 mon.c (mon.2) 298 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:37.499392+0000 mon.c (mon.2) 298 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.527010+0000 mgr.y (mgr.24401) 294 : cluster [DBG] pgmap v298: 381 pgs: 64 unknown, 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.527010+0000 mgr.y (mgr.24401) 294 : cluster [DBG] pgmap v298: 381 pgs: 64 unknown, 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:37.527706+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:37.527706+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:37.530899+0000 mon.a (mon.0) 1531 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:37.530899+0000 mon.a (mon.0) 1531 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.706909+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.706909+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.707035+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:37.707035+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041091+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041091+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041164+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041164+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041217+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041217+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041306+0000 mon.a (mon.0) 1535 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.041306+0000 mon.a (mon.0) 1535 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.046361+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-04-26T16:24:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.046361+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.058069+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.058069+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.058894+0000 mon.a (mon.0) 1537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[29234]: audit 2024-04-26T16:24:38.058894+0000 mon.a (mon.0) 1537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:37.707035+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041091+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041091+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041164+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041164+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041217+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041217+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041306+0000 mon.a (mon.0) 1535 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.041306+0000 mon.a (mon.0) 1535 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.046361+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.046361+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.058069+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.058069+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.058894+0000 mon.a (mon.0) 1537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:38 smithi050 bash[21463]: audit 2024-04-26T16:24:38.058894+0000 mon.a (mon.0) 1537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:38.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:36.671258+0000 mgr.y (mgr.24401) 293 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:36.671258+0000 mgr.y (mgr.24401) 293 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.083207+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.083207+0000 osd.3 (osd.3) 177 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.085068+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.085068+0000 osd.3 (osd.3) 178 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.351037+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.351037+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.355689+0000 osd.0 (osd.0) 168 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:24:38.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.355689+0000 osd.0 (osd.0) 168 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:37.499392+0000 mon.c (mon.2) 298 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:37.499392+0000 mon.c (mon.2) 298 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.527010+0000 mgr.y (mgr.24401) 294 : cluster [DBG] pgmap v298: 381 pgs: 64 unknown, 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.527010+0000 mgr.y (mgr.24401) 294 : cluster [DBG] pgmap v298: 381 pgs: 64 unknown, 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:37.527706+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:37.527706+0000 mon.c (mon.2) 299 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:37.530899+0000 mon.a (mon.0) 1531 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:37.530899+0000 mon.a (mon.0) 1531 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.706909+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.706909+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.707035+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:37.707035+0000 osd.3 (osd.3) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041091+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.758 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041091+0000 mon.a (mon.0) 1532 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi050-73201-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041164+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041164+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041217+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041217+0000 mon.a (mon.0) 1534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi050-73193-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041306+0000 mon.a (mon.0) 1535 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.041306+0000 mon.a (mon.0) 1535 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "16"}]': finished 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.046361+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.046361+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e210: 8 total, 8 up, 8 in 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.058069+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.058069+0000 mon.c (mon.2) 300 : audit [INF] from='client.? 172.21.15.50:0/2506934054' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.058894+0000 mon.a (mon.0) 1537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.759 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:38 smithi190 bash[23868]: audit 2024-04-26T16:24:38.058894+0000 mon.a (mon.0) 1537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]: dispatch 2024-04-26T16:24:38.773 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:38.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:38.905 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:38.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:39.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:39.036 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:39.049 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-04-26T16:24:39.049 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: seed 73432 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2048 ms) 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2020 ms) 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1978 ms) 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2019 ms) 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-04-26T16:24:39.050 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3812 ms) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11877 ms total) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4375 ms) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3582 ms) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6015 ms) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3991 ms) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2095 ms) 2024-04-26T16:24:39.051 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:461: Skipped 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi050-73432-7 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: waiting for snaps to purge 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18019 ms) 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38077 ms total) 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (11 ms) 2024-04-26T16:24:39.052 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5047 ms) 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5058 ms total) 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3006 ms) 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1985 ms) 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-04-26T16:24:39.053 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2003 ms) 2024-04-26T16:24:39.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:39.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:39.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:39.286 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:39.390 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.084538+0000 osd.3 (osd.3) 181 : cluster [DBG] 3.b scrub starts 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.084538+0000 osd.3 (osd.3) 181 : cluster [DBG] 3.b scrub starts 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.091829+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.b scrub ok 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.091829+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.b scrub ok 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: audit 2024-04-26T16:24:38.501005+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: audit 2024-04-26T16:24:38.501005+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.944436+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.944436+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:24:39.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.945633+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.945633+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: audit 2024-04-26T16:24:39.044264+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: audit 2024-04-26T16:24:39.044264+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.053384+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.053384+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.084538+0000 osd.3 (osd.3) 181 : cluster [DBG] 3.b scrub starts 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.084538+0000 osd.3 (osd.3) 181 : cluster [DBG] 3.b scrub starts 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.091829+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.b scrub ok 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.091829+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.b scrub ok 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: audit 2024-04-26T16:24:38.501005+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:39.401 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: audit 2024-04-26T16:24:38.501005+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.944436+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.944436+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.945633+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.945633+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: audit 2024-04-26T16:24:39.044264+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: audit 2024-04-26T16:24:39.044264+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.053384+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-04-26T16:24:39.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:39 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.053384+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-04-26T16:24:39.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:39.517 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:39.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:39.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:39.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.084538+0000 osd.3 (osd.3) 181 : cluster [DBG] 3.b scrub starts 2024-04-26T16:24:39.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.084538+0000 osd.3 (osd.3) 181 : cluster [DBG] 3.b scrub starts 2024-04-26T16:24:39.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.091829+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.b scrub ok 2024-04-26T16:24:39.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.091829+0000 osd.3 (osd.3) 182 : cluster [DBG] 3.b scrub ok 2024-04-26T16:24:39.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: audit 2024-04-26T16:24:38.501005+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:39.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: audit 2024-04-26T16:24:38.501005+0000 mon.c (mon.2) 301 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.944436+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.944436+0000 osd.2 (osd.2) 123 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.945633+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.945633+0000 osd.2 (osd.2) 124 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: audit 2024-04-26T16:24:39.044264+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: audit 2024-04-26T16:24:39.044264+0000 mon.a (mon.0) 1538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi050-73363-1","app":"app1","key":"key1"}]': finished 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.053384+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-04-26T16:24:39.657 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:39 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.053384+0000 mon.a (mon.0) 1539 : cluster [DBG] osdmap e211: 8 total, 8 up, 8 in 2024-04-26T16:24:39.671 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:39.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:39.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:39.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:39.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:39.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:39.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:40.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:40.061 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_ api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-04-26T16:24:40.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: seed 73363 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (466 ms) 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (11 ms) 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (12 ms) 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (10 ms) 2024-04-26T16:24:40.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (10 ms) 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-04-26T16:24:40.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (6 ms) 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: osd_max_attr_size = 0 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (2385 ms) 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (716 ms) 2024-04-26T16:24:40.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: done waiting, doing copies 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: waiting for final deep scrubs... 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: done waiting 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60871 ms) 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (11 ms) 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (4 ms) 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-04-26T16:24:40.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4383 ms) 2024-04-26T16:24:40.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-04-26T16:24:40.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-04-26T16:24:40.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-04-26T16:24:40.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-04-26T16:24:40.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-04-26T16:24:40.067 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-04-26T16:24:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:40.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:40.210 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:40.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:40.339 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:40.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:40.468 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:40.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:40.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:40.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.426244+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:24:40.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.426244+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.428572+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:38.428572+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.043956+0000 osd.3 (osd.3) 183 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.043956+0000 osd.3 (osd.3) 183 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.048137+0000 osd.3 (osd.3) 184 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.048137+0000 osd.3 (osd.3) 184 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.339300+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.339300+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.341912+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.341912+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:39.502600+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:39.502600+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.527774+0000 mgr.y (mgr.24401) 295 : cluster [DBG] pgmap v301: 317 pgs: 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:40.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.527774+0000 mgr.y (mgr.24401) 295 : cluster [DBG] pgmap v301: 317 pgs: 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.548442+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.548442+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.549785+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.549785+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.985612+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.f scrub starts 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.985612+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.f scrub starts 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.988062+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.f scrub ok 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:39.988062+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.f scrub ok 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.056629+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-04-26T16:24:40.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.056629+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.426244+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.426244+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.428572+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:38.428572+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.043956+0000 osd.3 (osd.3) 183 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.043956+0000 osd.3 (osd.3) 183 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.048137+0000 osd.3 (osd.3) 184 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.048137+0000 osd.3 (osd.3) 184 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.339300+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.339300+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.341912+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:24:40.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.341912+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:39.502600+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:39.502600+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.527774+0000 mgr.y (mgr.24401) 295 : cluster [DBG] pgmap v301: 317 pgs: 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.527774+0000 mgr.y (mgr.24401) 295 : cluster [DBG] pgmap v301: 317 pgs: 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.548442+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.548442+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.549785+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.549785+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.985612+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.f scrub starts 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.985612+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.f scrub starts 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.988062+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.f scrub ok 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:39.988062+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.f scrub ok 2024-04-26T16:24:40.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.056629+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.056629+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.060425+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.060425+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.064377+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.50:0/995603534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.064377+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.50:0/995603534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.066096+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.066096+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.112759+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.112759+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.114234+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.114234+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.114801+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:40.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[29234]: audit 2024-04-26T16:24:40.114801+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.060425+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.060425+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.064377+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.50:0/995603534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.064377+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.50:0/995603534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.066096+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.066096+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.112759+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.112759+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.114234+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.114234+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.114801+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:40.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:40 smithi050 bash[21463]: audit 2024-04-26T16:24:40.114801+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:40.599 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:40.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:40.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.426244+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:24:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.426244+0000 osd.5 (osd.5) 163 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:24:40.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.428572+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:38.428572+0000 osd.5 (osd.5) 164 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.043956+0000 osd.3 (osd.3) 183 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.043956+0000 osd.3 (osd.3) 183 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.048137+0000 osd.3 (osd.3) 184 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.048137+0000 osd.3 (osd.3) 184 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.339300+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.339300+0000 osd.0 (osd.0) 169 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.341912+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.341912+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:39.502600+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:39.502600+0000 mon.c (mon.2) 302 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.527774+0000 mgr.y (mgr.24401) 295 : cluster [DBG] pgmap v301: 317 pgs: 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.527774+0000 mgr.y (mgr.24401) 295 : cluster [DBG] pgmap v301: 317 pgs: 1 peering, 316 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:24:40.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.548442+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.548442+0000 osd.1 (osd.1) 163 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.549785+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.549785+0000 osd.1 (osd.1) 164 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.985612+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.f scrub starts 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.985612+0000 osd.2 (osd.2) 125 : cluster [DBG] 6.f scrub starts 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.988062+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.f scrub ok 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:39.988062+0000 osd.2 (osd.2) 126 : cluster [DBG] 6.f scrub ok 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.056629+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.056629+0000 mon.a (mon.0) 1540 : cluster [DBG] osdmap e212: 8 total, 8 up, 8 in 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.060425+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.060425+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.064377+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.50:0/995603534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.064377+0000 mon.b (mon.1) 147 : audit [INF] from='client.? 172.21.15.50:0/995603534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.066096+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.066096+0000 mon.a (mon.0) 1542 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.112759+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.112759+0000 mon.a (mon.0) 1543 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.114234+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.114234+0000 mon.a (mon.0) 1544 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.114801+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:40 smithi190 bash[23868]: audit 2024-04-26T16:24:40.114801+0000 mon.a (mon.0) 1545 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:40.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:40.861 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:40.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:40.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:40.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:40.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:41.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:41.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:41.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:41.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:41.376 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:41.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:41.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:41.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.375514+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.375514+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:40.504205+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:40.504205+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.511015+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.c scrub starts 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.511015+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.c scrub starts 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.512764+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.512764+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.037128+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.037128+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.037289+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.375514+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.375514+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:40.504205+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:40.504205+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.511015+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.c scrub starts 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.511015+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.c scrub starts 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.512764+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.512764+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.037128+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.037128+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.037289+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:41.625 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.375514+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.375514+0000 mon.a (mon.0) 1546 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:40.504205+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:40.504205+0000 mon.c (mon.2) 303 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.511015+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.c scrub starts 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.511015+0000 osd.1 (osd.1) 165 : cluster [DBG] 5.c scrub starts 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.512764+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.512764+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.037128+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.037128+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.50:0/4079268492' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi050-73193-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.037289+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.037289+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.037460+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.037460+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:41.045368+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: cluster 2024-04-26T16:24:41.045368+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.049646+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.049646+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.279586+0000 mon.c (mon.2) 304 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:41.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:41 smithi190 bash[23868]: audit 2024-04-26T16:24:41.279586+0000 mon.c (mon.2) 304 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:41.640 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:41.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.037289+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.037460+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:41.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.037460+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:41.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:41.045368+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-04-26T16:24:41.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: cluster 2024-04-26T16:24:41.045368+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-04-26T16:24:41.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.049646+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:41.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.049646+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:41.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.279586+0000 mon.c (mon.2) 304 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:41.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:41.774 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[21463]: audit 2024-04-26T16:24:41.279586+0000 mon.c (mon.2) 304 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:41.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.037289+0000 mon.a (mon.0) 1548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi050-73201-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.037460+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.037460+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73363-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:41.045368+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: cluster 2024-04-26T16:24:41.045368+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e213: 8 total, 8 up, 8 in 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.049646+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.049646+0000 mon.a (mon.0) 1551 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.279586+0000 mon.c (mon.2) 304 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:41.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:41 smithi050 bash[29234]: audit 2024-04-26T16:24:41.279586+0000 mon.c (mon.2) 304 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:41.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:41.908 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:41.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:42.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:42.041 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:42.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:42.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:42.296 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:42.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:42.426 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:42.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.980070+0000 osd.3 (osd.3) 185 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.980070+0000 osd.3 (osd.3) 185 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.982519+0000 osd.3 (osd.3) 186 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:40.982519+0000 osd.3 (osd.3) 186 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:41.505856+0000 mon.c (mon.2) 305 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:41.505856+0000 mon.c (mon.2) 305 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:41.529567+0000 mgr.y (mgr.24401) 296 : cluster [DBG] pgmap v304: 348 pgs: 348 active+clean; 593 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:41.529567+0000 mgr.y (mgr.24401) 296 : cluster [DBG] pgmap v304: 348 pgs: 348 active+clean; 593 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:41.530062+0000 mon.c (mon.2) 306 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:41.530062+0000 mon.c (mon.2) 306 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:41.531650+0000 mon.a (mon.0) 1552 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:41.531650+0000 mon.a (mon.0) 1552 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.019039+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.019039+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.021581+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.021581+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:42.040879+0000 mon.a (mon.0) 1553 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:42.040879+0000 mon.a (mon.0) 1553 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.074708+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.074708+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:42.078751+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:42.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:42 smithi190 bash[23868]: audit 2024-04-26T16:24:42.078751+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:42.557 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:42.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.980070+0000 osd.3 (osd.3) 185 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.980070+0000 osd.3 (osd.3) 185 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.982519+0000 osd.3 (osd.3) 186 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:40.982519+0000 osd.3 (osd.3) 186 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:41.505856+0000 mon.c (mon.2) 305 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:41.505856+0000 mon.c (mon.2) 305 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:41.529567+0000 mgr.y (mgr.24401) 296 : cluster [DBG] pgmap v304: 348 pgs: 348 active+clean; 593 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:41.529567+0000 mgr.y (mgr.24401) 296 : cluster [DBG] pgmap v304: 348 pgs: 348 active+clean; 593 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:41.530062+0000 mon.c (mon.2) 306 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:41.530062+0000 mon.c (mon.2) 306 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:41.531650+0000 mon.a (mon.0) 1552 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:41.531650+0000 mon.a (mon.0) 1552 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.019039+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.019039+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.021581+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.021581+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:42.040879+0000 mon.a (mon.0) 1553 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:42.040879+0000 mon.a (mon.0) 1553 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.074708+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.074708+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.980070+0000 osd.3 (osd.3) 185 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:24:42.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.980070+0000 osd.3 (osd.3) 185 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.982519+0000 osd.3 (osd.3) 186 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:40.982519+0000 osd.3 (osd.3) 186 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:41.505856+0000 mon.c (mon.2) 305 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:41.505856+0000 mon.c (mon.2) 305 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:41.529567+0000 mgr.y (mgr.24401) 296 : cluster [DBG] pgmap v304: 348 pgs: 348 active+clean; 593 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:41.529567+0000 mgr.y (mgr.24401) 296 : cluster [DBG] pgmap v304: 348 pgs: 348 active+clean; 593 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:41.530062+0000 mon.c (mon.2) 306 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:41.530062+0000 mon.c (mon.2) 306 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:41.531650+0000 mon.a (mon.0) 1552 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:41.531650+0000 mon.a (mon.0) 1552 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.019039+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.019039+0000 osd.2 (osd.2) 127 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.021581+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:24:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.021581+0000 osd.2 (osd.2) 128 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:42.040879+0000 mon.a (mon.0) 1553 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:42.040879+0000 mon.a (mon.0) 1553 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "15"}]': finished 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.074708+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.074708+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e214: 8 total, 8 up, 8 in 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:42.078751+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[29234]: audit 2024-04-26T16:24:42.078751+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:42.078751+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:42.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:42 smithi050 bash[21463]: audit 2024-04-26T16:24:42.078751+0000 mon.a (mon.0) 1555 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:42.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:42.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:42.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:42.820 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:42.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:42.949 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:42.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:43.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:43.075 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscP snapshots: Running main() from gmock_main.cc 2024-04-26T16:24:43.075 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-04-26T16:24:43.075 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [----------] Global test environment set-up. 2024-04-26T16:24:43.075 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4753 ms) 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5037 ms) 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3818 ms) 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (4998 ms) 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7032 ms) 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25638 ms total) 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-04-26T16:24:43.076 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (4990 ms) 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5981 ms) 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7997 ms) 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6009 ms) 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4009 ms) 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi050-74220-11. 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: Waiting for snaps to purge. 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20015 ms) 2024-04-26T16:24:43.077 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (49002 ms total) 2024-04-26T16:24:43.078 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: 2024-04-26T16:24:43.078 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [----------] Global test environment tear-down 2024-04-26T16:24:43.078 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (74640 ms total) 2024-04-26T16:24:43.078 INFO:tasks.workunit.client.0.smithi050.stdout: snapshots: [ PASSED ] 11 tests. 2024-04-26T16:24:43.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:43.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:43.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:43.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:43.351 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:43.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:43.479 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:43.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:43.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:43.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:43.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:43.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.391014+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:24:43.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.391014+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.399585+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.399585+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:42.507322+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:42.507322+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.748696+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.748696+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.751162+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:42.751162+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:24:43.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.045207+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.045207+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.045450+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.045450+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.055124+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.055124+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.056650+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.056650+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.057646+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.057646+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.072932+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.50:0/1914133938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.072932+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.50:0/1914133938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.075012+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:43 smithi190 bash[23868]: audit 2024-04-26T16:24:43.075012+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:43.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:43.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.391014+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.391014+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.399585+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.399585+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:42.507322+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:42.507322+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.748696+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.748696+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.751162+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:42.751162+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:24:43.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.045207+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.045207+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.045450+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.045450+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.055124+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.055124+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.056650+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.056650+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.057646+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.057646+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.072932+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.50:0/1914133938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.072932+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.50:0/1914133938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.075012+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[21463]: audit 2024-04-26T16:24:43.075012+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.391014+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:24:43.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.391014+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:24:43.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.399585+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:24:43.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.399585+0000 osd.0 (osd.0) 172 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:24:43.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:42.507322+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:43.860 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:42.507322+0000 mon.c (mon.2) 307 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.748696+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.748696+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.751162+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:42.751162+0000 osd.7 (osd.7) 186 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.045207+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.045207+0000 mon.a (mon.0) 1556 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73363-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.045450+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.045450+0000 mon.a (mon.0) 1557 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.055124+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.055124+0000 mon.a (mon.0) 1558 : cluster [DBG] osdmap e215: 8 total, 8 up, 8 in 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.056650+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.056650+0000 mon.a (mon.0) 1559 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]: dispatch 2024-04-26T16:24:43.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.057646+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.057646+0000 mon.a (mon.0) 1560 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.072932+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.50:0/1914133938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.072932+0000 mon.c (mon.2) 308 : audit [INF] from='client.? 172.21.15.50:0/1914133938' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.075012+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:43 smithi050 bash[29234]: audit 2024-04-26T16:24:43.075012+0000 mon.a (mon.0) 1561 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:43.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:43.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:43.995 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:43.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:44.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:44.203 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:24:44.203 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:24:44.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:44.258 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:44.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:44.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:44.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:44.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:44.520 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:44.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.397791+0000 osd.0 (osd.0) 173 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:24:44.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.397791+0000 osd.0 (osd.0) 173 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:24:44.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.402733+0000 osd.0 (osd.0) 174 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.402733+0000 osd.0 (osd.0) 174 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.421234+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.421234+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.423092+0000 osd.6 (osd.6) 110 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.423092+0000 osd.6 (osd.6) 110 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.530482+0000 mgr.y (mgr.24401) 297 : cluster [DBG] pgmap v307: 316 pgs: 72 unknown, 244 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.530482+0000 mgr.y (mgr.24401) 297 : cluster [DBG] pgmap v307: 316 pgs: 72 unknown, 244 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.707952+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.707952+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.710368+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.710368+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.048304+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.048304+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.048365+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.048365+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.048436+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.048436+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.053611+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.053611+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.064767+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.064767+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.109669+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.109669+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.110274+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.110274+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.112224+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.112224+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.112576+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.112576+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.113391+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.113391+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.115526+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: audit 2024-04-26T16:24:44.115526+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.122117+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.d scrub starts 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.122117+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.d scrub starts 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.127030+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.d scrub ok 2024-04-26T16:24:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:44 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.127030+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.d scrub ok 2024-04-26T16:24:44.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:44.650 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:44.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:44.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:44.781 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:44.877 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:24:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.397791+0000 osd.0 (osd.0) 173 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.397791+0000 osd.0 (osd.0) 173 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.402733+0000 osd.0 (osd.0) 174 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.402733+0000 osd.0 (osd.0) 174 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.421234+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.421234+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.423092+0000 osd.6 (osd.6) 110 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.423092+0000 osd.6 (osd.6) 110 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.530482+0000 mgr.y (mgr.24401) 297 : cluster [DBG] pgmap v307: 316 pgs: 72 unknown, 244 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.530482+0000 mgr.y (mgr.24401) 297 : cluster [DBG] pgmap v307: 316 pgs: 72 unknown, 244 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.707952+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.707952+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:24:44.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.710368+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.710368+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.048304+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.048304+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.048365+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.048365+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.048436+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.048436+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.053611+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.053611+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-04-26T16:24:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.397791+0000 osd.0 (osd.0) 173 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.397791+0000 osd.0 (osd.0) 173 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.402733+0000 osd.0 (osd.0) 174 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.402733+0000 osd.0 (osd.0) 174 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.421234+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.421234+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.423092+0000 osd.6 (osd.6) 110 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.423092+0000 osd.6 (osd.6) 110 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.530482+0000 mgr.y (mgr.24401) 297 : cluster [DBG] pgmap v307: 316 pgs: 72 unknown, 244 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.530482+0000 mgr.y (mgr.24401) 297 : cluster [DBG] pgmap v307: 316 pgs: 72 unknown, 244 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.707952+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.707952+0000 osd.7 (osd.7) 187 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.710368+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:24:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.710368+0000 osd.7 (osd.7) 188 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.048304+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.048304+0000 mon.a (mon.0) 1562 : audit [INF] from='client.? 172.21.15.50:0/3899662302' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi050-73432-16"}]': finished 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.048365+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.048365+0000 mon.a (mon.0) 1563 : audit [INF] from='client.? 172.21.15.50:0/584314985' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi050-73201-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.048436+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.048436+0000 mon.a (mon.0) 1564 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi050-73193-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.053611+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.053611+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e216: 8 total, 8 up, 8 in 2024-04-26T16:24:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.064767+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.064767+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.109669+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.109669+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.110274+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.110274+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.112224+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.112224+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.112576+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.112576+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.113391+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.113391+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.115526+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.064767+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.064767+0000 mon.a (mon.0) 1566 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.109669+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.109669+0000 mon.c (mon.2) 309 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.110274+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.110274+0000 mon.a (mon.0) 1567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.112224+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.112224+0000 mon.c (mon.2) 310 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.112576+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.112576+0000 mon.a (mon.0) 1568 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:44.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.113391+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.113391+0000 mon.c (mon.2) 311 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.115526+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: audit 2024-04-26T16:24:44.115526+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.122117+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.d scrub starts 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.122117+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.d scrub starts 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.127030+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.d scrub ok 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.127030+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.d scrub ok 2024-04-26T16:24:44.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: audit 2024-04-26T16:24:44.115526+0000 mon.a (mon.0) 1569 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:44.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.122117+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.d scrub starts 2024-04-26T16:24:44.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.122117+0000 osd.2 (osd.2) 129 : cluster [DBG] 5.d scrub starts 2024-04-26T16:24:44.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.127030+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.d scrub ok 2024-04-26T16:24:44.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:44 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.127030+0000 osd.2 (osd.2) 130 : cluster [DBG] 5.d scrub ok 2024-04-26T16:24:44.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:44.912 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:44.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:45.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:45.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:45.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:45.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:45.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:45.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:45.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:45.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:45.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:45.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:45.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:45.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.900367+0000 osd.3 (osd.3) 187 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.900367+0000 osd.3 (osd.3) 187 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.903022+0000 osd.3 (osd.3) 188 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:43.903022+0000 osd.3 (osd.3) 188 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.368600+0000 osd.0 (osd.0) 175 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.368600+0000 osd.0 (osd.0) 175 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.373995+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.373995+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:24:45.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.419159+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.f scrub starts 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.419159+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.f scrub starts 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.421400+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.421400+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.445208+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.445208+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.447593+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.447593+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.051236+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.051236+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.051328+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.051328+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:45.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.055578+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-04-26T16:24:45.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.055578+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-04-26T16:24:45.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.057020+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.057020+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.057611+0000 mon.a (mon.0) 1573 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.057611+0000 mon.a (mon.0) 1573 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.058431+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.058431+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.107897+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.e scrub starts 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.107897+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.e scrub starts 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.108503+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.108503+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.109861+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.109861+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.112227+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.112227+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.113779+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.113779+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.114930+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.114930+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.121518+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.e scrub ok 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.121518+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.e scrub ok 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.123430+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.123430+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.186446+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:45.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:45 smithi190 bash[23868]: audit 2024-04-26T16:24:45.186446+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:45.683 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:45.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 -- 172.21.15.50:0/3950025795 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 msgr2=0x7f98241166a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:45.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 --2- 172.21.15.50:0/3950025795 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241166a0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f981c0098f0 tx=0x7f981c009c00 comp rx=0 tx=0).stop 2024-04-26T16:24:45.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 -- 172.21.15.50:0/3950025795 shutdown_connections 2024-04-26T16:24:45.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 --2- 172.21.15.50:0/3950025795 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9824077950 0x7f9824077d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:45.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 --2- 172.21.15.50:0/3950025795 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241166a0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 --2- 172.21.15.50:0/3950025795 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9824078270 0x7f9824113d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 -- 172.21.15.50:0/3950025795 >> 172.21.15.50:0/3950025795 conn(0x7f9824101100 msgr2=0x7f9824103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 -- 172.21.15.50:0/3950025795 shutdown_connections 2024-04-26T16:24:45.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 -- 172.21.15.50:0/3950025795 wait complete. 2024-04-26T16:24:45.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 librados: starting msgr at 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 librados: starting objecter 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 Processor -- start 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 -- start start 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 librados: setting wanted keys 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.717+0000 7f9828e8f700 1 librados: calling monclient init 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9824077950 0x7f982419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9824078270 0x7f982419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241a4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9824119180 con 0x7f9824077950 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9824119000 con 0x7f9824078270 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9824119300 con 0x7f98241142b0 2024-04-26T16:24:45.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f982259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9824077950 0x7f982419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241a4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60626/0 (socket says 172.21.15.50:60626) 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 -- 172.21.15.50:0/998203431 learned_addr learned my addr 172.21.15.50:0/998203431 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9821d9b700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9824078270 0x7f982419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 -- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9824078270 msgr2=0x7f982419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9824078270 0x7f982419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 -- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9824077950 msgr2=0x7f982419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9824077950 0x7f982419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 -- 172.21.15.50:0/998203431 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98241a4740 con 0x7f98241142b0 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f982259c700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9824077950 0x7f982419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9822d9d700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241a4000 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f981c009230 tx=0x7f981c00fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:45.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 10 client.?.objecter ms_handle_connect 0x7f98241142b0 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c025070 con 0x7f98241142b0 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f981c005040 con 0x7f98241142b0 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98241a49d0 con 0x7f98241142b0 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c0039a0 con 0x7f98241142b0 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 10 client.40661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f98241ac2b0 con 0x7f98241142b0 2024-04-26T16:24:45.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 librados: init done 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 10 client.40661.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 15 client.40661.objecter _session_command_op_assign -1 1 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 20 client.40661.objecter _get_session osd=-1 returning homeless 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 10 client.40661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9824079090 con 0x7f98241142b0 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f981c028070 con 0x7f98241142b0 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9810071f80 0x7f9810074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(217..217 src has 87..217) ==== 11055+0+0 (secure 0 0 0) 0x7f981c092fd0 con 0x7f98241142b0 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 10 client.40661.objecter ms_dispatch 0x7f9824001550 osd_map(217..217 src has 87..217) 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 3 client.40661.objecter handle_osd_map got epochs [217,217] > 0 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 3 client.40661.objecter handle_osd_map decoding full epoch 217 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f982259c700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9810071f80 0x7f9810074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 10 client.40661.objecter checking command 1 2024-04-26T16:24:45.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9810081870 0x7f9810083d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 20 client.40661.objecter _get_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 15 client.40661.objecter _session_command_op_remove -1 1 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 20 client.40661.objecter _get_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 20 client.40661.objecter get_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 15 client.40661.objecter _session_command_op_assign 7 1 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 4 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 10 client.40661.objecter _send_command 1 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98100843e0 con 0x7f9810081870 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 20 client.40661.objecter dump_active .. 0 homeless 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=217) ==== 24+0+0 (secure 0 0 0) 0x7f981c093420 con 0x7f98241142b0 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.721+0000 7f9821d9b700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9810081870 0x7f9810083d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 10 client.40661.objecter ms_handle_connect 0x7f9810071f80 2024-04-26T16:24:45.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f982259c700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9810071f80 0x7f9810074440 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f9814009fd0 tx=0x7f9814009180 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f9821d9b700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9810081870 0x7f9810083d10 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 10 client.40661.objecter ms_handle_connect 0x7f9810081870 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98100843e0 con 0x7f9810081870 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 10 client.40661.objecter ms_dispatch 0x7f9824001550 command_reply(tid 1: 0 ) 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 10 client.40661.objecter _finish_command 1 = system:0 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 4 2024-04-26T16:24:45.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.725+0000 7f981b7fe700 15 client.40661.objecter _session_command_op_remove 7 1 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 10 client.40661.objecter _submit_command 2 [{"prefix": "reset_purged_snaps_last"}] 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 15 client.40661.objecter _session_command_op_assign -1 2 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter _get_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter _get_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 15 client.40661.objecter _session_command_op_remove -1 2 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter get_session s=0x7f98100811a0 osd=7 3 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 15 client.40661.objecter _session_command_op_assign 7 2 2024-04-26T16:24:45.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 4 2024-04-26T16:24:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 10 client.40661.objecter _send_command 2 2024-04-26T16:24:45.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 2: {"prefix": "reset_purged_snaps_last"}) -- 0x7f982406a110 con 0x7f9810081870 2024-04-26T16:24:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f981b7fe700 1 -- 172.21.15.50:0/998203431 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+0 (crc 0 0 0) 0x7f982406a110 con 0x7f9810081870 2024-04-26T16:24:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f981b7fe700 10 client.40661.objecter ms_dispatch 0x7f9824001550 command_reply(tid 2: 0 ) 2024-04-26T16:24:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f981b7fe700 10 client.40661.objecter _finish_command 2 = system:0 2024-04-26T16:24:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f981b7fe700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 4 2024-04-26T16:24:45.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f981b7fe700 15 client.40661.objecter _session_command_op_remove 7 2 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 10 librados: watch_flush enter 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 10 librados: watch_flush exit 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 10 client.40661.objecter close_session for osd.7 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9810081870 msgr2=0x7f9810083d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9810081870 0x7f9810083d10 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter put_session s=0x7f98100811a0 osd=7 1 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 20 client.40661.objecter shutdown clearing up homeless session... 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 10 client.40661.objecter successfully canceled tick 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9810071f80 msgr2=0x7f9810074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9810071f80 0x7f9810074440 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f9814009fd0 tx=0x7f9814009180 comp rx=0 tx=0).stop 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 msgr2=0x7f98241a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 --2- 172.21.15.50:0/998203431 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98241142b0 0x7f98241a4000 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f981c009230 tx=0x7f981c00fa00 comp rx=0 tx=0).stop 2024-04-26T16:24:45.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9822d9d700 1 -- 172.21.15.50:0/998203431 reap_dead start 2024-04-26T16:24:45.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 shutdown_connections 2024-04-26T16:24:45.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 >> 172.21.15.50:0/998203431 conn(0x7f9824101100 msgr2=0x7f98241034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:45.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 shutdown_connections 2024-04-26T16:24:45.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 -- 172.21.15.50:0/998203431 wait complete. 2024-04-26T16:24:45.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:45.745+0000 7f9828e8f700 1 librados: shutdown 2024-04-26T16:24:45.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:45.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.900367+0000 osd.3 (osd.3) 187 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.900367+0000 osd.3 (osd.3) 187 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.903022+0000 osd.3 (osd.3) 188 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:43.903022+0000 osd.3 (osd.3) 188 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.368600+0000 osd.0 (osd.0) 175 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.368600+0000 osd.0 (osd.0) 175 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.373995+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.373995+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.419159+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.f scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.419159+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.f scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.421400+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.421400+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.445208+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:24:45.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.445208+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.447593+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.447593+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.051236+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.051236+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.051328+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.051328+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.055578+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.055578+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.057020+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.057020+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.057611+0000 mon.a (mon.0) 1573 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.057611+0000 mon.a (mon.0) 1573 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.058431+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.058431+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.107897+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.e scrub starts 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.107897+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.e scrub starts 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.108503+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.108503+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.109861+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.109861+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.112227+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.112227+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.113779+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.900367+0000 osd.3 (osd.3) 187 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.900367+0000 osd.3 (osd.3) 187 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.903022+0000 osd.3 (osd.3) 188 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:43.903022+0000 osd.3 (osd.3) 188 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.368600+0000 osd.0 (osd.0) 175 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.368600+0000 osd.0 (osd.0) 175 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.373995+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.373995+0000 osd.0 (osd.0) 176 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.419159+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.f scrub starts 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.419159+0000 osd.5 (osd.5) 165 : cluster [DBG] 5.f scrub starts 2024-04-26T16:24:45.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.421400+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.421400+0000 osd.5 (osd.5) 166 : cluster [DBG] 5.f scrub ok 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.445208+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.445208+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.447593+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.447593+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.051236+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.051236+0000 mon.a (mon.0) 1570 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.051328+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.051328+0000 mon.a (mon.0) 1571 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.055578+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.055578+0000 mon.a (mon.0) 1572 : cluster [DBG] osdmap e217: 8 total, 8 up, 8 in 2024-04-26T16:24:45.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.057020+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.113779+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.114930+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.114930+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.121518+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.e scrub ok 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.121518+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.e scrub ok 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.123430+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.123430+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.186446+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:45.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[29234]: audit 2024-04-26T16:24:45.186446+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.057020+0000 mon.c (mon.2) 312 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.057611+0000 mon.a (mon.0) 1573 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.057611+0000 mon.a (mon.0) 1573 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.058431+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.058431+0000 mon.a (mon.0) 1574 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.107897+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.e scrub starts 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.107897+0000 osd.2 (osd.2) 131 : cluster [DBG] 2.e scrub starts 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.108503+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.108503+0000 mon.c (mon.2) 313 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.109861+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.109861+0000 mon.a (mon.0) 1575 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.112227+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.112227+0000 mon.c (mon.2) 314 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.113779+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.113779+0000 mon.a (mon.0) 1576 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.114930+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.114930+0000 mon.c (mon.2) 315 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.121518+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.e scrub ok 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.121518+0000 osd.2 (osd.2) 132 : cluster [DBG] 2.e scrub ok 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.123430+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.123430+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:45.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.186446+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:45.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:45 smithi050 bash[21463]: audit 2024-04-26T16:24:45.186446+0000 mon.c (mon.2) 316 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:45.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:45.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:45.938 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:45.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:45.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:46.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:46.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:46.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:46.200 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:46.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:46.330 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:46.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:46.459 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:46.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.948965+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.948965+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.951221+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:44.951221+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.531048+0000 mgr.y (mgr.24401) 298 : cluster [DBG] pgmap v310: 283 pgs: 40 unknown, 243 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.531048+0000 mgr.y (mgr.24401) 298 : cluster [DBG] pgmap v310: 283 pgs: 40 unknown, 243 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:45.531708+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:45.531708+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:45.533063+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:45.533063+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.663722+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.663722+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.665731+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.665731+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.764575+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.764575+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.765664+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.765664+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.036264+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.036264+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.039819+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.039819+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.039907+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.039907+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.039981+0000 mon.a (mon.0) 1582 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.039981+0000 mon.a (mon.0) 1582 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.045175+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.045175+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.059520+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.059520+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-04-26T16:24:46.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.065212+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.065212+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.066729+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.066729+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.107949+0000 osd.2 (osd.2) 133 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.107949+0000 osd.2 (osd.2) 133 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.109591+0000 osd.2 (osd.2) 134 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.109591+0000 osd.2 (osd.2) 134 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.187376+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:46.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[23868]: audit 2024-04-26T16:24:46.187376+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:46.588 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:46.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.948965+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.948965+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.951221+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:44.951221+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.531048+0000 mgr.y (mgr.24401) 298 : cluster [DBG] pgmap v310: 283 pgs: 40 unknown, 243 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.531048+0000 mgr.y (mgr.24401) 298 : cluster [DBG] pgmap v310: 283 pgs: 40 unknown, 243 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:45.531708+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.705 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:45.531708+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:45.533063+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:45.533063+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.663722+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.663722+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.665731+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.665731+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.764575+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.764575+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.765664+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.765664+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.036264+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.036264+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:46.706 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.039819+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.948965+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.948965+0000 osd.3 (osd.3) 189 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.951221+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:44.951221+0000 osd.3 (osd.3) 190 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.531048+0000 mgr.y (mgr.24401) 298 : cluster [DBG] pgmap v310: 283 pgs: 40 unknown, 243 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.531048+0000 mgr.y (mgr.24401) 298 : cluster [DBG] pgmap v310: 283 pgs: 40 unknown, 243 active+clean; 592 KiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:45.531708+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:45.531708+0000 mon.c (mon.2) 317 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:45.533063+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:45.533063+0000 mon.a (mon.0) 1578 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-04-26T16:24:46.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.663722+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:24:46.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.663722+0000 osd.7 (osd.7) 189 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:24:46.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.665731+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.665731+0000 osd.7 (osd.7) 190 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.764575+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.764575+0000 osd.4 (osd.4) 145 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.765664+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.765664+0000 osd.4 (osd.4) 146 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.036264+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.036264+0000 mon.a (mon.0) 1579 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:46.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.039819+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.039819+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.039907+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.039907+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.039981+0000 mon.a (mon.0) 1582 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.039981+0000 mon.a (mon.0) 1582 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.045175+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.045175+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.059520+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.059520+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-04-26T16:24:46.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.065212+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.065212+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.066729+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.066729+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.107949+0000 osd.2 (osd.2) 133 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.107949+0000 osd.2 (osd.2) 133 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.109591+0000 osd.2 (osd.2) 134 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.039819+0000 mon.a (mon.0) 1580 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73363-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.039907+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.039907+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi050-73193-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.039981+0000 mon.a (mon.0) 1582 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-26T16:24:46.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.039981+0000 mon.a (mon.0) 1582 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "14"}]': finished 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.045175+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.045175+0000 mon.c (mon.2) 318 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.059520+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.059520+0000 mon.a (mon.0) 1583 : cluster [DBG] osdmap e218: 8 total, 8 up, 8 in 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.065212+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.065212+0000 mon.a (mon.0) 1584 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.066729+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.066729+0000 mon.a (mon.0) 1585 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.107949+0000 osd.2 (osd.2) 133 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.107949+0000 osd.2 (osd.2) 133 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:24:46.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.109591+0000 osd.2 (osd.2) 134 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:24:46.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.109591+0000 osd.2 (osd.2) 134 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:24:46.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.187376+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:46.713 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[21463]: audit 2024-04-26T16:24:46.187376+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:46.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.109591+0000 osd.2 (osd.2) 134 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:24:46.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.187376+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:46.713 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:46 smithi050 bash[29234]: audit 2024-04-26T16:24:46.187376+0000 mon.c (mon.2) 319 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:46.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:46.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:46.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:46.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:46.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:46.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:24:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:24:46.984 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:46.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:47.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:47.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:47.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:47.241 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:47.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:47.372 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:47.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:47.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:47.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:47.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-26T16:24:47.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:47.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.900278+0000 osd.3 (osd.3) 191 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.900278+0000 osd.3 (osd.3) 191 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.903310+0000 osd.3 (osd.3) 192 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:45.903310+0000 osd.3 (osd.3) 192 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.343812+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.343812+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.346553+0000 osd.0 (osd.0) 178 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.346553+0000 osd.0 (osd.0) 178 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.454249+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.454249+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.456431+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.456431+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:24:47.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.492576+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:24:47.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.492576+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:24:47.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.494444+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:24:47.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.494444+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: audit 2024-04-26T16:24:47.044412+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: audit 2024-04-26T16:24:47.044412+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: audit 2024-04-26T16:24:47.044698+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: audit 2024-04-26T16:24:47.044698+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.072677+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.072677+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: audit 2024-04-26T16:24:47.188713+0000 mon.c (mon.2) 320 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[21463]: audit 2024-04-26T16:24:47.188713+0000 mon.c (mon.2) 320 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.900278+0000 osd.3 (osd.3) 191 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.900278+0000 osd.3 (osd.3) 191 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:24:47.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.903310+0000 osd.3 (osd.3) 192 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:45.903310+0000 osd.3 (osd.3) 192 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.343812+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.343812+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.346553+0000 osd.0 (osd.0) 178 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.346553+0000 osd.0 (osd.0) 178 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.454249+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.454249+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.456431+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.456431+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.492576+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.492576+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.494444+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.494444+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:24:47.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: audit 2024-04-26T16:24:47.044412+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: audit 2024-04-26T16:24:47.044412+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: audit 2024-04-26T16:24:47.044698+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: audit 2024-04-26T16:24:47.044698+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.072677+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.072677+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: audit 2024-04-26T16:24:47.188713+0000 mon.c (mon.2) 320 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:47.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:47 smithi050 bash[29234]: audit 2024-04-26T16:24:47.188713+0000 mon.c (mon.2) 320 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:47.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.900278+0000 osd.3 (osd.3) 191 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.900278+0000 osd.3 (osd.3) 191 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.903310+0000 osd.3 (osd.3) 192 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:45.903310+0000 osd.3 (osd.3) 192 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.343812+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.343812+0000 osd.0 (osd.0) 177 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.346553+0000 osd.0 (osd.0) 178 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.346553+0000 osd.0 (osd.0) 178 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:24:47.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.454249+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.454249+0000 osd.5 (osd.5) 167 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.456431+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.456431+0000 osd.5 (osd.5) 168 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.492576+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.492576+0000 osd.6 (osd.6) 113 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.494444+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.494444+0000 osd.6 (osd.6) 114 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: audit 2024-04-26T16:24:47.044412+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: audit 2024-04-26T16:24:47.044412+0000 mon.a (mon.0) 1586 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: audit 2024-04-26T16:24:47.044698+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: audit 2024-04-26T16:24:47.044698+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.50:0/3456335653' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi050-73201-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.072677+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.072677+0000 mon.a (mon.0) 1588 : cluster [DBG] osdmap e219: 8 total, 8 up, 8 in 2024-04-26T16:24:47.749 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: audit 2024-04-26T16:24:47.188713+0000 mon.c (mon.2) 320 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:47.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:47 smithi190 bash[23868]: audit 2024-04-26T16:24:47.188713+0000 mon.c (mon.2) 320 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:47.764 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 -- 172.21.15.50:0/4127875696 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 msgr2=0x7f169c10ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 --2- 172.21.15.50:0/4127875696 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c10ba80 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f16980098f0 tx=0x7f1698009c00 comp rx=0 tx=0).stop 2024-04-26T16:24:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 -- 172.21.15.50:0/4127875696 shutdown_connections 2024-04-26T16:24:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 --2- 172.21.15.50:0/4127875696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f169c105fd0 0x7f169c1124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 --2- 172.21.15.50:0/4127875696 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c10ba80 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:47.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 --2- 172.21.15.50:0/4127875696 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105610 0x7f169c105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 -- 172.21.15.50:0/4127875696 >> 172.21.15.50:0/4127875696 conn(0x7f169c101100 msgr2=0x7f169c103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 -- 172.21.15.50:0/4127875696 shutdown_connections 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 -- 172.21.15.50:0/4127875696 wait complete. 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 librados: starting msgr at 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 librados: starting objecter 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.849+0000 7f16a344e700 1 Processor -- start 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- start start 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 librados: setting wanted keys 2024-04-26T16:24:47.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 librados: calling monclient init 2024-04-26T16:24:47.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f169c105610 0x7f169c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 0x7f169c19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f169c114d10 con 0x7f169c105fd0 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f169c114b90 con 0x7f169c105610 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f169c114e90 con 0x7f169c10b6a0 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 0x7f169c19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 0x7f169c19b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:35562/0 (socket says 172.21.15.50:35562) 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a19eb700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 -- 172.21.15.50:0/2078527475 learned_addr learned my addr 172.21.15.50:0/2078527475 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a11ea700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f169c105610 0x7f169c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 -- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 msgr2=0x7f169c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:47.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 -- 172.21.15.50:0/2078527475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f169c105610 msgr2=0x7f169c19b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f169c105610 0x7f169c19b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 -- 172.21.15.50:0/2078527475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c1a02f0 con 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a19eb700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a09e9700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 0x7f169c19b820 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f168c00ec10 tx=0x7f168c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16927fc700 10 client.?.objecter ms_handle_connect 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16927fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c00cd40 con 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c1a05e0 con 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f168c004480 con 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c010430 con 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 10 client.30066.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c1a7e60 con 0x7f169c105fd0 2024-04-26T16:24:47.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 librados: init done 2024-04-26T16:24:47.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c06a110 con 0x7f169c105fd0 2024-04-26T16:24:47.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.853+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f168c0105d0 con 0x7f169c105fd0 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1688071f80 0x7f1688074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(219..219 src has 87..219) ==== 12079+0+0 (secure 0 0 0) 0x7f168c092870 con 0x7f169c105fd0 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 10 client.30066.objecter ms_dispatch 0x7f169c001550 osd_map(219..219 src has 87..219) 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 3 client.30066.objecter handle_osd_map got epochs [219,219] > 0 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 3 client.30066.objecter handle_osd_map decoding full epoch 219 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16a11ea700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1688071f80 0x7f1688074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 20 client.30066.objecter dump_active .. 0 homeless 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16a11ea700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1688071f80 0x7f1688074440 secure :-1 s=READY pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7f16980098c0 tx=0x7f1698019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:47.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.857+0000 7f16927fc700 10 client.30066.objecter ms_handle_connect 0x7f1688071f80 2024-04-26T16:24:47.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:47.861+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f168c05d7a0 con 0x7f169c105fd0 2024-04-26T16:24:47.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:47.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:47.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:47.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:48.023 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:48.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.121+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f169c19c640 con 0x7f169c105fd0 2024-04-26T16:24:48.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:48.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:48.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:48.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:48.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:48.411 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:48.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:48.539 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- 172.21.15.50:0/592730872 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102ce0 msgr2=0x7fc82410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/592730872 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102ce0 0x7fc82410f1d0 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00b0a0 tx=0x7fc81c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- 172.21.15.50:0/592730872 shutdown_connections 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/592730872 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824108390 0x7fc824108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/592730872 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102ce0 0x7fc82410f1d0 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/592730872 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824102320 0x7fc8241027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- 172.21.15.50:0/592730872 >> 172.21.15.50:0/592730872 conn(0x7fc8240792d0 msgr2=0x7fc8240796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- 172.21.15.50:0/592730872 shutdown_connections 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- 172.21.15.50:0/592730872 wait complete. 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 librados: starting msgr at 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 librados: starting objecter 2024-04-26T16:24:48.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 Processor -- start 2024-04-26T16:24:48.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- start start 2024-04-26T16:24:48.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 librados: setting wanted keys 2024-04-26T16:24:48.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 librados: calling monclient init 2024-04-26T16:24:48.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 0x7fc82410fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824102ce0 0x7fc824110430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824108390 0x7fc824110970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc824112a90 con 0x7fc824102ce0 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc824112910 con 0x7fc824108390 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82bfe2700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fc824112c10 con 0x7fc824102320 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82957d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824102ce0 0x7fc824110430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 0x7fc82410fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 0x7fc82410fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:47000/0 (socket says 172.21.15.50:47000) 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82957d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824102ce0 0x7fc824110430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33326/0 (socket says 172.21.15.50:33326) 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 -- 172.21.15.50:0/879267744 learned_addr learned my addr 172.21.15.50:0/879267744 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824108390 0x7fc824110970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824108390 msgr2=0x7fc824110970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc824108390 0x7fc824110970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824102ce0 msgr2=0x7fc824110430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc824102ce0 0x7fc824110430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8241a7610 con 0x7fc824102320 2024-04-26T16:24:48.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 0x7fc82410fef0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fc82000d950 tx=0x7fc82000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc81affd700 10 client.?.objecter ms_handle_connect 0x7fc824102320 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.549+0000 7fc81affd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820016070 con 0x7fc824102320 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fc82000fcf0 con 0x7fc824102320 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc820010df0 con 0x7fc824102320 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8241a7900 con 0x7fc824102320 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 10 client.40676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8241a7df0 con 0x7fc824102320 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 1 librados: init done 2024-04-26T16:24:48.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc82406a110 con 0x7fc824102320 2024-04-26T16:24:48.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fc820019070 con 0x7fc824102320 2024-04-26T16:24:48.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc810071f80 0x7fc810074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(220..220 src has 87..220) ==== 11676+0+0 (secure 0 0 0) 0x7fc820091d60 con 0x7fc824102320 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 osd_map(220..220 src has 87..220) 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82957d700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc810071f80 0x7fc810074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 3 client.40676.objecter handle_osd_map got epochs [220,220] > 0 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 3 client.40676.objecter handle_osd_map decoding full epoch 220 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 20 client.40676.objecter dump_active .. 0 homeless 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v220) ==== 54+0+16 (secure 0 0 0) 0x7fc820093020 con 0x7fc824102320 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 1 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc824064030 0x7fc824064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 1 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 0 1 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 4 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 10 client.40676.objecter _send_command 1 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8241ab180 con 0x7fc824064030 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82957d700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc810071f80 0x7fc810074440 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7fc814009680 tx=0x7fc814009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc810071f80 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc824064030 0x7fc824064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc824064030 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.553+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc824064030 0x7fc824064410 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.557+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241ab180 con 0x7fc824064030 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.557+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 1: 0 ) 2024-04-26T16:24:48.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.557+0000 7fc81affd700 10 client.40676.objecter _finish_command 1 = system:0 2024-04-26T16:24:48.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.557+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 4 2024-04-26T16:24:48.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.557+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 0 1 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 2 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 2 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241113d0 osd=0 3 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 0 2 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 4 2024-04-26T16:24:48.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 10 client.40676.objecter _send_command 2 2024-04-26T16:24:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.573+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc824063ab0 con 0x7fc824064030 2024-04-26T16:24:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc824063ab0 con 0x7fc824064030 2024-04-26T16:24:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 10 client.40676.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 4 2024-04-26T16:24:48.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 0 2 2024-04-26T16:24:48.572 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:24:48.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc82402d310 0x7fc82402d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 1 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 4 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 10 client.40676.objecter _send_command 3 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc824109e10 con 0x7fc82402d310 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc82402d310 0x7fc82402d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc82402d310 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc82402d310 0x7fc82402d710 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc824109e10 con 0x7fc82402d310 2024-04-26T16:24:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 3: 0 ) 2024-04-26T16:24:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 10 client.40676.objecter _finish_command 3 = system:0 2024-04-26T16:24:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 4 2024-04-26T16:24:48.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.577+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 1 3 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 4 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 4 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc824109790 osd=1 3 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 1 4 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 4 2024-04-26T16:24:48.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 10 client.40676.objecter _send_command 4 2024-04-26T16:24:48.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.593+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc824109210 con 0x7fc82402d310 2024-04-26T16:24:48.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc824109210 con 0x7fc82402d310 2024-04-26T16:24:48.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 10 client.40676.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 4 2024-04-26T16:24:48.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 1 4 2024-04-26T16:24:48.593 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:24:48.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 5 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc8241ad340 0x7fc8241af730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 5 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 2 5 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 4 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 10 client.40676.objecter _send_command 5 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc8241b2590 con 0x7fc8241ad340 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82957d700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc8241ad340 0x7fc8241af730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc8241ad340 2024-04-26T16:24:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc82957d700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc8241ad340 0x7fc8241af730 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241b2590 con 0x7fc8241ad340 2024-04-26T16:24:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 5: 0 ) 2024-04-26T16:24:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 10 client.40676.objecter _finish_command 5 = system:0 2024-04-26T16:24:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 4 2024-04-26T16:24:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.597+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 2 5 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 6 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 6 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc824109fc0 osd=2 3 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 2 6 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 4 2024-04-26T16:24:48.612 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 10 client.40676.objecter _send_command 6 2024-04-26T16:24:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc82402dea0 con 0x7fc8241ad340 2024-04-26T16:24:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc82402dea0 con 0x7fc8241ad340 2024-04-26T16:24:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 10 client.40676.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 4 2024-04-26T16:24:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 2 6 2024-04-26T16:24:48.614 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:24:48.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 7 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc8241b0df0 0x7fc8241b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 7 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 3 7 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 4 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 10 client.40676.objecter _send_command 7 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc8241b1770 con 0x7fc8241b0df0 2024-04-26T16:24:48.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc8241b0df0 0x7fc8241b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc8241b0df0 0x7fc8241b1230 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc8241b0df0 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241b1770 con 0x7fc8241b0df0 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 7: 0 ) 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 10 client.40676.objecter _finish_command 7 = system:0 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 4 2024-04-26T16:24:48.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.617+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 3 7 2024-04-26T16:24:48.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:46.682436+0000 mgr.y (mgr.24401) 299 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:48.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:46.682436+0000 mgr.y (mgr.24401) 299 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:48.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.911587+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:24:48.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.911587+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.914301+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:46.914301+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.349597+0000 osd.0 (osd.0) 179 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.349597+0000 osd.0 (osd.0) 179 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.352015+0000 osd.0 (osd.0) 180 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.352015+0000 osd.0 (osd.0) 180 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.531854+0000 mgr.y (mgr.24401) 300 : cluster [DBG] pgmap v313: 291 pgs: 2 peering, 32 creating+peering, 8 unknown, 249 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.531854+0000 mgr.y (mgr.24401) 300 : cluster [DBG] pgmap v313: 291 pgs: 2 peering, 32 creating+peering, 8 unknown, 249 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.705723+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.d scrub starts 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.705723+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.d scrub starts 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.710541+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.d scrub ok 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:47.710541+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.d scrub ok 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.047528+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:48.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.047528+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.056026+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.056026+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.062860+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:46.682436+0000 mgr.y (mgr.24401) 299 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:46.682436+0000 mgr.y (mgr.24401) 299 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.911587+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:24:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.911587+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.914301+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:46.914301+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.349597+0000 osd.0 (osd.0) 179 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.349597+0000 osd.0 (osd.0) 179 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.352015+0000 osd.0 (osd.0) 180 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.352015+0000 osd.0 (osd.0) 180 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.531854+0000 mgr.y (mgr.24401) 300 : cluster [DBG] pgmap v313: 291 pgs: 2 peering, 32 creating+peering, 8 unknown, 249 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.531854+0000 mgr.y (mgr.24401) 300 : cluster [DBG] pgmap v313: 291 pgs: 2 peering, 32 creating+peering, 8 unknown, 249 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.705723+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.d scrub starts 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.705723+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.d scrub starts 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.710541+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.d scrub ok 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:47.710541+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.d scrub ok 2024-04-26T16:24:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.047528+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.047528+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.056026+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.056026+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.062860+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.062860+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.124956+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.124956+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.189991+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[21463]: audit 2024-04-26T16:24:48.189991+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.062860+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.124956+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.124956+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:48.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.189991+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:48.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:48 smithi050 bash[29234]: audit 2024-04-26T16:24:48.189991+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 8 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 8 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241aa7d0 osd=3 3 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 3 8 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 4 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 10 client.40676.objecter _send_command 8 2024-04-26T16:24:48.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.637+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc8241b2180 con 0x7fc8241b0df0 2024-04-26T16:24:48.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8241b2180 con 0x7fc8241b0df0 2024-04-26T16:24:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 10 client.40676.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 4 2024-04-26T16:24:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 3 8 2024-04-26T16:24:48.636 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:24:48.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 9 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc8241b4750 0x7fc8241b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 9 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 4 9 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 4 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 10 client.40676.objecter _send_command 9 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc8241b99a0 con 0x7fc8241b4750 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc8241b4750 0x7fc8241b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc8241b4750 2024-04-26T16:24:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc8241b4750 0x7fc8241b6b40 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241b99a0 con 0x7fc8241b4750 2024-04-26T16:24:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 9: 0 ) 2024-04-26T16:24:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 10 client.40676.objecter _finish_command 9 = system:0 2024-04-26T16:24:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 4 2024-04-26T16:24:48.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.641+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 4 9 2024-04-26T16:24:48.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:48.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 10 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 10 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241b2150 osd=4 3 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 4 10 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 4 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 10 client.40676.objecter _send_command 10 2024-04-26T16:24:48.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.661+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc8241b7080 con 0x7fc8241b4750 2024-04-26T16:24:48.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:48.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8241b7080 con 0x7fc8241b4750 2024-04-26T16:24:48.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc81affd700 10 client.40676.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 4 2024-04-26T16:24:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 4 10 2024-04-26T16:24:48.661 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:24:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 11 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc8241b77a0 0x7fc8241b9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 11 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 5 11 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 4 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 10 client.40676.objecter _send_command 11 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc8241bcbe0 con 0x7fc8241b77a0 2024-04-26T16:24:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82957d700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc8241b77a0 0x7fc8241b9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc8241b77a0 2024-04-26T16:24:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.665+0000 7fc82957d700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc8241b77a0 0x7fc8241b9b90 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.669+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241bcbe0 con 0x7fc8241b77a0 2024-04-26T16:24:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.669+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 11: 0 ) 2024-04-26T16:24:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.669+0000 7fc81affd700 10 client.40676.objecter _finish_command 11 = system:0 2024-04-26T16:24:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.669+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 4 2024-04-26T16:24:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.669+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 5 11 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 12 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 12 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241b7080 osd=5 3 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 5 12 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 4 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 10 client.40676.objecter _send_command 12 2024-04-26T16:24:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.685+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc8241bcbe0 con 0x7fc8241b77a0 2024-04-26T16:24:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8241bcbe0 con 0x7fc8241b77a0 2024-04-26T16:24:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc81affd700 10 client.40676.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 4 2024-04-26T16:24:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 5 12 2024-04-26T16:24:48.686 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 13 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fc8241bbf40 0x7fc8241bc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 13 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 6 13 2024-04-26T16:24:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 4 2024-04-26T16:24:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 10 client.40676.objecter _send_command 13 2024-04-26T16:24:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc8241bff80 con 0x7fc8241bbf40 2024-04-26T16:24:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.689+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fc8241bbf40 0x7fc8241bc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc8241bbf40 2024-04-26T16:24:48.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc82a57f700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fc8241bbf40 0x7fc8241bc3a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241bff80 con 0x7fc8241bbf40 2024-04-26T16:24:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 13: 0 ) 2024-04-26T16:24:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc81affd700 10 client.40676.objecter _finish_command 13 = system:0 2024-04-26T16:24:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 4 2024-04-26T16:24:48.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.693+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 6 13 2024-04-26T16:24:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 14 2024-04-26T16:24:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.707 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 14 2024-04-26T16:24:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241bcbe0 osd=6 3 2024-04-26T16:24:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 6 14 2024-04-26T16:24:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 4 2024-04-26T16:24:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 10 client.40676.objecter _send_command 14 2024-04-26T16:24:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.709+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc8241ba1e0 con 0x7fc8241bbf40 2024-04-26T16:24:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.713+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8241ba1e0 con 0x7fc8241bbf40 2024-04-26T16:24:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.713+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.713+0000 7fc81affd700 10 client.40676.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.713+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 4 2024-04-26T16:24:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.713+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 6 14 2024-04-26T16:24:48.712 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:24:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:24:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 15 2024-04-26T16:24:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc8241ba900 0x7fc8241bad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:24:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 15 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 7 15 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 4 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 10 client.40676.objecter _send_command 15 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc8241bb3c0 con 0x7fc8241ba900 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc8241ba900 0x7fc8241bad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc81affd700 10 client.40676.objecter ms_handle_connect 0x7fc8241ba900 2024-04-26T16:24:48.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc829d7e700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc8241ba900 0x7fc8241bad20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:24:48.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8241bb3c0 con 0x7fc8241ba900 2024-04-26T16:24:48.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 15: 0 ) 2024-04-26T16:24:48.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc81affd700 10 client.40676.objecter _finish_command 15 = system:0 2024-04-26T16:24:48.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 4 2024-04-26T16:24:48.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.717+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 7 15 2024-04-26T16:24:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 10 client.40676.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:24:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign -1 16 2024-04-26T16:24:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 20 client.40676.objecter _get_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_remove -1 16 2024-04-26T16:24:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 20 client.40676.objecter get_session s=0x7fc8241ba1e0 osd=7 3 2024-04-26T16:24:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 15 client.40676.objecter _session_command_op_assign 7 16 2024-04-26T16:24:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 4 2024-04-26T16:24:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 10 client.40676.objecter _send_command 16 2024-04-26T16:24:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.737+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fc8241bb3c0 con 0x7fc8241ba900 2024-04-26T16:24:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc81affd700 1 -- 172.21.15.50:0/879267744 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc8241bb3c0 con 0x7fc8241ba900 2024-04-26T16:24:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc81affd700 10 client.40676.objecter ms_dispatch 0x7fc824001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:24:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc81affd700 10 client.40676.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:24:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc81affd700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 4 2024-04-26T16:24:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc81affd700 15 client.40676.objecter _session_command_op_remove 7 16 2024-04-26T16:24:48.737 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:24:48.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 librados: watch_flush enter 2024-04-26T16:24:48.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 librados: watch_flush exit 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.0 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc824064030 msgr2=0x7fc824064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc824064030 0x7fc824064410 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241113d0 osd=0 1 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.1 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc82402d310 msgr2=0x7fc82402d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc82402d310 0x7fc82402d710 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109790 osd=1 1 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.2 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc8241ad340 msgr2=0x7fc8241af730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc8241ad340 0x7fc8241af730 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82a57f700 1 -- 172.21.15.50:0/879267744 reap_dead start 2024-04-26T16:24:48.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc824109fc0 osd=2 1 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.3 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc8241b0df0 msgr2=0x7fc8241b1230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc8241b0df0 0x7fc8241b1230 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241aa7d0 osd=3 1 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.4 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc8241b4750 msgr2=0x7fc8241b6b40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc8241b4750 0x7fc8241b6b40 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b2150 osd=4 1 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.5 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc8241b77a0 msgr2=0x7fc8241b9b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc8241b77a0 0x7fc8241b9b90 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241b7080 osd=5 1 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.6 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fc8241bbf40 msgr2=0x7fc8241bc3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fc8241bbf40 0x7fc8241bc3a0 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241bcbe0 osd=6 1 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter close_session for osd.7 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc8241ba900 msgr2=0x7fc8241bad20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc8241ba900 0x7fc8241bad20 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter put_session s=0x7fc8241ba1e0 osd=7 1 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 20 client.40676.objecter shutdown clearing up homeless session... 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 10 client.40676.objecter successfully canceled tick 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82a57f700 1 -- 172.21.15.50:0/879267744 reap_dead start 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc810071f80 msgr2=0x7fc810074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc810071f80 0x7fc810074440 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7fc814009680 tx=0x7fc814009260 comp rx=0 tx=0).stop 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 msgr2=0x7fc82410fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 0x7fc82410fef0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7fc82000d950 tx=0x7fc82000dc60 comp rx=0 tx=0).stop 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 shutdown_connections 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc824102320 0x7fc82410fef0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 --2- 172.21.15.50:0/879267744 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc810071f80 0x7fc810074440 unknown :-1 s=CLOSED pgs=3653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 >> 172.21.15.50:0/879267744 conn(0x7fc8240792d0 msgr2=0x7fc824100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 shutdown_connections 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 -- 172.21.15.50:0/879267744 wait complete. 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:48.741+0000 7fc82bfe2700 1 librados: shutdown 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:24:48.742 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:24:48.743 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:24:48.743 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:24:48.743 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:24:48.743 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:24:48.743 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:24:48.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:48.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:46.682436+0000 mgr.y (mgr.24401) 299 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:48.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:46.682436+0000 mgr.y (mgr.24401) 299 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:48.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.911587+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:24:48.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.911587+0000 osd.3 (osd.3) 193 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.914301+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:46.914301+0000 osd.3 (osd.3) 194 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.349597+0000 osd.0 (osd.0) 179 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.349597+0000 osd.0 (osd.0) 179 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.352015+0000 osd.0 (osd.0) 180 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.352015+0000 osd.0 (osd.0) 180 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.531854+0000 mgr.y (mgr.24401) 300 : cluster [DBG] pgmap v313: 291 pgs: 2 peering, 32 creating+peering, 8 unknown, 249 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.531854+0000 mgr.y (mgr.24401) 300 : cluster [DBG] pgmap v313: 291 pgs: 2 peering, 32 creating+peering, 8 unknown, 249 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.705723+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.d scrub starts 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.705723+0000 osd.7 (osd.7) 191 : cluster [DBG] 3.d scrub starts 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.710541+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.d scrub ok 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:47.710541+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.d scrub ok 2024-04-26T16:24:48.777 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.047528+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.047528+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi050-73193-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.056026+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.056026+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e220: 8 total, 8 up, 8 in 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.062860+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.062860+0000 mon.a (mon.0) 1591 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.124956+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.124956+0000 mon.a (mon.0) 1592 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.189991+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:48.778 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:48 smithi190 bash[23868]: audit 2024-04-26T16:24:48.189991+0000 mon.c (mon.2) 321 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:48.792 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:48.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:48.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:48.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:48.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:49.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.049+0000 7f16927fc700 1 -- 172.21.15.50:0/2078527475 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v221) ==== 101+0+0 (secure 0 0 0) 0x7f168c061450 con 0x7f169c105fd0 2024-04-26T16:24:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.053+0000 7f16a344e700 10 librados: watch_flush enter 2024-04-26T16:24:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.053+0000 7f16a344e700 10 librados: watch_flush exit 2024-04-26T16:24:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.053+0000 7f16a344e700 20 client.30066.objecter shutdown clearing up homeless session... 2024-04-26T16:24:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.053+0000 7f16a344e700 10 client.30066.objecter successfully canceled tick 2024-04-26T16:24:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.067 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:49.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1688071f80 msgr2=0x7f1688074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1688071f80 0x7f1688074440 secure :-1 s=READY pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7f16980098c0 tx=0x7f1698019040 comp rx=0 tx=0).stop 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 msgr2=0x7f169c19b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 0x7f169c19b820 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f168c00ec10 tx=0x7f168c00ef20 comp rx=0 tx=0).stop 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 shutdown_connections 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1688071f80 0x7f1688074440 secure :-1 s=CLOSED pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7f16980098c0 tx=0x7f1698019040 comp rx=0 tx=0).stop 2024-04-26T16:24:49.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f169c105610 0x7f169c19b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:49.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f169c10b6a0 0x7f169c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:49.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 --2- 172.21.15.50:0/2078527475 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f169c105fd0 0x7f169c19b820 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:24:49.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.057+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 >> 172.21.15.50:0/2078527475 conn(0x7f169c101100 msgr2=0x7f169c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:24:49.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.065+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 shutdown_connections 2024-04-26T16:24:49.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.065+0000 7f16a344e700 1 -- 172.21.15.50:0/2078527475 wait complete. 2024-04-26T16:24:49.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:24:49.065+0000 7f16a344e700 1 librados: shutdown 2024-04-26T16:24:49.072 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:24:49.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:49.194 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:49.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:49.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:49.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:49.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:49.451 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:49.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:49.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.382784+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:24:49.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.382784+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:24:49.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.386333+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:24:49.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.386333+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:24:49.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.447067+0000 osd.5 (osd.5) 169 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:24:49.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.447067+0000 osd.5 (osd.5) 169 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.448230+0000 osd.5 (osd.5) 170 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.448230+0000 osd.5 (osd.5) 170 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.501751+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.501751+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.504871+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.504871+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:48.557164+0000 mon.c (mon.2) 322 : audit [DBG] from='client.? 172.21.15.50:0/879267744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:48.557164+0000 mon.c (mon.2) 322 : audit [DBG] from='client.? 172.21.15.50:0/879267744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.743355+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.743355+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.747780+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:48.747780+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:24:49.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.048792+0000 mon.a (mon.0) 1593 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.048792+0000 mon.a (mon.0) 1593 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.051243+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.051243+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.051471+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.051471+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.059473+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.059473+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.063722+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.063722+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.065601+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.065601+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.191201+0000 mon.c (mon.2) 323 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:49.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[29234]: audit 2024-04-26T16:24:49.191201+0000 mon.c (mon.2) 323 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.382784+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.382784+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.386333+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.386333+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.447067+0000 osd.5 (osd.5) 169 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.447067+0000 osd.5 (osd.5) 169 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.448230+0000 osd.5 (osd.5) 170 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:24:49.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.448230+0000 osd.5 (osd.5) 170 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.501751+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.501751+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.504871+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.504871+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:48.557164+0000 mon.c (mon.2) 322 : audit [DBG] from='client.? 172.21.15.50:0/879267744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:48.557164+0000 mon.c (mon.2) 322 : audit [DBG] from='client.? 172.21.15.50:0/879267744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.743355+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.743355+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.747780+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:48.747780+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.048792+0000 mon.a (mon.0) 1593 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.048792+0000 mon.a (mon.0) 1593 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:24:49.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.051243+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.051243+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.051471+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.051471+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.059473+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.059473+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.063722+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.063722+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.065601+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.065601+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.191201+0000 mon.c (mon.2) 323 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:49.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:49 smithi050 bash[21463]: audit 2024-04-26T16:24:49.191201+0000 mon.c (mon.2) 323 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:49.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:49.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:49.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:49.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.382784+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:24:49.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.382784+0000 osd.0 (osd.0) 181 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.386333+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.386333+0000 osd.0 (osd.0) 182 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.447067+0000 osd.5 (osd.5) 169 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.447067+0000 osd.5 (osd.5) 169 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.448230+0000 osd.5 (osd.5) 170 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.448230+0000 osd.5 (osd.5) 170 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.501751+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.501751+0000 osd.6 (osd.6) 115 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.504871+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.504871+0000 osd.6 (osd.6) 116 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:48.557164+0000 mon.c (mon.2) 322 : audit [DBG] from='client.? 172.21.15.50:0/879267744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:48.557164+0000 mon.c (mon.2) 322 : audit [DBG] from='client.? 172.21.15.50:0/879267744' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.743355+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:24:49.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.743355+0000 osd.4 (osd.4) 147 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:24:49.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.747780+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:24:49.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:48.747780+0000 osd.4 (osd.4) 148 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:24:49.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.048792+0000 mon.a (mon.0) 1593 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:24:49.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.048792+0000 mon.a (mon.0) 1593 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:24:49.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.051243+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.051243+0000 mon.a (mon.0) 1594 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.051471+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.051471+0000 mon.a (mon.0) 1595 : audit [INF] from='client.? 172.21.15.50:0/2078527475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.059473+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.059473+0000 mon.a (mon.0) 1596 : cluster [DBG] osdmap e221: 8 total, 8 up, 8 in 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.063722+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.063722+0000 mon.a (mon.0) 1597 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]: dispatch 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.065601+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.065601+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.191201+0000 mon.c (mon.2) 323 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:49.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:49 smithi190 bash[23868]: audit 2024-04-26T16:24:49.191201+0000 mon.c (mon.2) 323 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:49.713 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:49.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:49.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:49.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:49.976 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:49.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:49.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:50.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:50.110 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:50.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:50.240 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:50.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:50.370 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:50.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:50.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.356322+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:24:50.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.356322+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:24:50.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.357906+0000 osd.0 (osd.0) 184 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:24:50.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.357906+0000 osd.0 (osd.0) 184 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:24:50.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.532775+0000 mgr.y (mgr.24401) 301 : cluster [DBG] pgmap v316: 290 pgs: 1 peering, 48 unknown, 241 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 10 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:50.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.532775+0000 mgr.y (mgr.24401) 301 : cluster [DBG] pgmap v316: 290 pgs: 1 peering, 48 unknown, 241 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 10 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:50.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:49.533512+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:49.533512+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:49.535701+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:49.535701+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.771267+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:24:50.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.771267+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:24:50.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.773254+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:24:50.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.773254+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:24:50.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.055438+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:50.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.055438+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:50.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.055658+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:50.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.055658+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:50.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.055851+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-26T16:24:50.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.055851+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-26T16:24:50.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.064651+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-04-26T16:24:50.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.064651+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-04-26T16:24:50.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.073775+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.073775+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.078797+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.078797+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.192259+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:50.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:50 smithi190 bash[23868]: audit 2024-04-26T16:24:50.192259+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:50.500 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:50.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:50.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:50.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:50.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:50.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.356322+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:24:50.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.356322+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:24:50.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.357906+0000 osd.0 (osd.0) 184 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:24:50.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.357906+0000 osd.0 (osd.0) 184 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:24:50.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.532775+0000 mgr.y (mgr.24401) 301 : cluster [DBG] pgmap v316: 290 pgs: 1 peering, 48 unknown, 241 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 10 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:50.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.532775+0000 mgr.y (mgr.24401) 301 : cluster [DBG] pgmap v316: 290 pgs: 1 peering, 48 unknown, 241 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 10 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:50.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:49.533512+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:49.533512+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:49.535701+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:49.535701+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.771267+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.771267+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.773254+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.773254+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:24:50.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.055438+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:50.752 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.055438+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:50.752 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.055658+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:50.752 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.055658+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:50.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.356322+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:24:50.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.356322+0000 osd.0 (osd.0) 183 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:24:50.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.357906+0000 osd.0 (osd.0) 184 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:24:50.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.357906+0000 osd.0 (osd.0) 184 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:24:50.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.532775+0000 mgr.y (mgr.24401) 301 : cluster [DBG] pgmap v316: 290 pgs: 1 peering, 48 unknown, 241 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 10 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:50.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.532775+0000 mgr.y (mgr.24401) 301 : cluster [DBG] pgmap v316: 290 pgs: 1 peering, 48 unknown, 241 active+clean; 592 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.7 KiB/s wr, 10 op/s; 31 B/s, 0 objects/s recovering 2024-04-26T16:24:50.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:49.533512+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:49.533512+0000 mon.c (mon.2) 324 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:49.535701+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:49.535701+0000 mon.a (mon.0) 1599 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-04-26T16:24:50.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.771267+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:24:50.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.771267+0000 osd.4 (osd.4) 149 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:24:50.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.773254+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:24:50.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.773254+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:24:50.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.055438+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:50.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.055438+0000 mon.a (mon.0) 1600 : audit [INF] from='client.? 172.21.15.50:0/2382217812' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73363-24"}]': finished 2024-04-26T16:24:50.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.055658+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:50.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.055658+0000 mon.a (mon.0) 1601 : audit [INF] from='client.? 172.21.15.50:0/3780134797' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi050-73201-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:50.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.055851+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-26T16:24:50.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.055851+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-26T16:24:50.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.064651+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-04-26T16:24:50.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.064651+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-04-26T16:24:50.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.073775+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.073775+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.078797+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.078797+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.192259+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:50.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[29234]: audit 2024-04-26T16:24:50.192259+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:50.757 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.055851+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-26T16:24:50.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.055851+0000 mon.a (mon.0) 1602 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "13"}]': finished 2024-04-26T16:24:50.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.064651+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-04-26T16:24:50.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.064651+0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e222: 8 total, 8 up, 8 in 2024-04-26T16:24:50.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.073775+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.073775+0000 mon.c (mon.2) 325 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.078797+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.078797+0000 mon.a (mon.0) 1604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:50.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.192259+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:50.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:50 smithi050 bash[21463]: audit 2024-04-26T16:24:50.192259+0000 mon.c (mon.2) 326 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:50.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:50.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:50.895 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:50.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:50.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:51.026 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:51.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:51.148 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:51.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:51.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:51.275 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:51.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:51.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:51.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:51.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:51.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:51.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:51.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.974357+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:24:51.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.974357+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:24:51.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.977069+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:24:51.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:49.977069+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:24:51.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.325480+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.325480+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.327996+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.327996+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.812063+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.812063+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.816587+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:50.816587+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.059791+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.059791+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.065502+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.065502+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.072700+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.072700+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.073677+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.50:0/3664412629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.073677+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.50:0/3664412629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.076082+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.076082+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.088108+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.088108+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.193604+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:51.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[29234]: audit 2024-04-26T16:24:51.193604+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.974357+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.974357+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.977069+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:49.977069+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.325480+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.325480+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.327996+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.327996+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.812063+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.812063+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:24:51.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.816587+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:50.816587+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.059791+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.059791+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.065502+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.065502+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.072700+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.072700+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.073677+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.50:0/3664412629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.073677+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.50:0/3664412629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.076082+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.076082+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.088108+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.088108+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.193604+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:51.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:51 smithi050 bash[21463]: audit 2024-04-26T16:24:51.193604+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:51.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:51.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.974357+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.974357+0000 osd.3 (osd.3) 195 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.977069+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:49.977069+0000 osd.3 (osd.3) 196 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.325480+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.325480+0000 osd.0 (osd.0) 185 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.327996+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.327996+0000 osd.0 (osd.0) 186 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.812063+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.812063+0000 osd.4 (osd.4) 151 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:24:51.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.816587+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:50.816587+0000 osd.4 (osd.4) 152 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.059791+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.059791+0000 mon.a (mon.0) 1605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.065502+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.065502+0000 mon.a (mon.0) 1606 : cluster [DBG] osdmap e223: 8 total, 8 up, 8 in 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.072700+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.072700+0000 mon.c (mon.2) 327 : audit [INF] from='client.? 172.21.15.50:0/3268544700' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.073677+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.50:0/3664412629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.073677+0000 mon.b (mon.1) 148 : audit [INF] from='client.? 172.21.15.50:0/3664412629' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.076082+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.076082+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.088108+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.088108+0000 mon.a (mon.0) 1608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]: dispatch 2024-04-26T16:24:51.789 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.193604+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:51.790 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:51 smithi190 bash[23868]: audit 2024-04-26T16:24:51.193604+0000 mon.c (mon.2) 328 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:51.801 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:51.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:51.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:51.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:52.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:52.070 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: Running main() from gmock_main.cc 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [----------] Global test environment set-up. 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2235 ms) 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3005 ms) 2024-04-26T16:24:52.088 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3018 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3028 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2796 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3015 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2999 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2988 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3029 ms) 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-04-26T16:24:52.089 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2995 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3054 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2944 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3017 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2972 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.Flush (3013 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3001 ms) 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-04-26T16:24:52.090 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2992 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3008 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3020 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2969 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3005 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3018 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2978 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3013 ms) 2024-04-26T16:24:52.091 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3024 ms) 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2975 ms) 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [----------] 26 tests from LibRadosAio (77111 ms total) 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-04-26T16:24:52.092 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-04-26T16:24:52.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:52.197 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:52.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:52.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:52.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:52.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:52.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:52.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.293751+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:24:52.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.293751+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.295330+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.295330+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.535556+0000 mgr.y (mgr.24401) 302 : cluster [DBG] pgmap v319: 282 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 2 clean+premerge+peered, 32 creating+peering, 196 active+clean; 599 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.535556+0000 mgr.y (mgr.24401) 302 : cluster [DBG] pgmap v319: 282 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 2 clean+premerge+peered, 32 creating+peering, 196 active+clean; 599 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.591605+0000 osd.6 (osd.6) 117 : cluster [DBG] 12.c scrub starts 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.591605+0000 osd.6 (osd.6) 117 : cluster [DBG] 12.c scrub starts 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.593373+0000 osd.6 (osd.6) 118 : cluster [DBG] 12.c scrub ok 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:51.593373+0000 osd.6 (osd.6) 118 : cluster [DBG] 12.c scrub ok 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.065189+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.065189+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.065455+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.065455+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.074536+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.074536+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.093035+0000 mon.c (mon.2) 329 : audit [INF] from='client.? 172.21.15.50:0/3235215439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.093035+0000 mon.c (mon.2) 329 : audit [INF] from='client.? 172.21.15.50:0/3235215439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.095068+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.095068+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.142372+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.142372+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.143574+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.143574+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.146257+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.146257+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.146739+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.146739+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.148179+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.148179+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.148656+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.148656+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.194671+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.194671+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.253818+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.253818+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.256037+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:52 smithi190 bash[23868]: audit 2024-04-26T16:24:52.256037+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:52.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:24:52.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:24:52.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:52.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.293751+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.293751+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.295330+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.295330+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.535556+0000 mgr.y (mgr.24401) 302 : cluster [DBG] pgmap v319: 282 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 2 clean+premerge+peered, 32 creating+peering, 196 active+clean; 599 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.535556+0000 mgr.y (mgr.24401) 302 : cluster [DBG] pgmap v319: 282 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 2 clean+premerge+peered, 32 creating+peering, 196 active+clean; 599 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.591605+0000 osd.6 (osd.6) 117 : cluster [DBG] 12.c scrub starts 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.591605+0000 osd.6 (osd.6) 117 : cluster [DBG] 12.c scrub starts 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.593373+0000 osd.6 (osd.6) 118 : cluster [DBG] 12.c scrub ok 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:51.593373+0000 osd.6 (osd.6) 118 : cluster [DBG] 12.c scrub ok 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.065189+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.065189+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.065455+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:52.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.065455+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:52.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.074536+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-04-26T16:24:52.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.074536+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-04-26T16:24:52.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.093035+0000 mon.c (mon.2) 329 : audit [INF] from='client.? 172.21.15.50:0/3235215439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.093035+0000 mon.c (mon.2) 329 : audit [INF] from='client.? 172.21.15.50:0/3235215439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.095068+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.095068+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.142372+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.142372+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.143574+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.143574+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.146257+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.146257+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.146739+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.146739+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.148179+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.148179+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.708 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.148656+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.148656+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.194671+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.194671+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.253818+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.253818+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.256037+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[29234]: audit 2024-04-26T16:24:52.256037+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.293751+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.293751+0000 osd.1 (osd.1) 167 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.295330+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.295330+0000 osd.1 (osd.1) 168 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.535556+0000 mgr.y (mgr.24401) 302 : cluster [DBG] pgmap v319: 282 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 2 clean+premerge+peered, 32 creating+peering, 196 active+clean; 599 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.535556+0000 mgr.y (mgr.24401) 302 : cluster [DBG] pgmap v319: 282 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 2 clean+premerge+peered, 32 creating+peering, 196 active+clean; 599 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 2 op/s 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.591605+0000 osd.6 (osd.6) 117 : cluster [DBG] 12.c scrub starts 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.591605+0000 osd.6 (osd.6) 117 : cluster [DBG] 12.c scrub starts 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.593373+0000 osd.6 (osd.6) 118 : cluster [DBG] 12.c scrub ok 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:51.593373+0000 osd.6 (osd.6) 118 : cluster [DBG] 12.c scrub ok 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.065189+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:52.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.065189+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi050-73363-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.065455+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.065455+0000 mon.a (mon.0) 1610 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi050-73193-27"}]': finished 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.074536+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.074536+0000 mon.a (mon.0) 1611 : cluster [DBG] osdmap e224: 8 total, 8 up, 8 in 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.093035+0000 mon.c (mon.2) 329 : audit [INF] from='client.? 172.21.15.50:0/3235215439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.093035+0000 mon.c (mon.2) 329 : audit [INF] from='client.? 172.21.15.50:0/3235215439' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.095068+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.095068+0000 mon.a (mon.0) 1612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.142372+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.142372+0000 mon.c (mon.2) 330 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.143574+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.143574+0000 mon.a (mon.0) 1613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.146257+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.146257+0000 mon.c (mon.2) 331 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.146739+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.146739+0000 mon.a (mon.0) 1614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.148179+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.148179+0000 mon.c (mon.2) 332 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.148656+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.148656+0000 mon.a (mon.0) 1615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.194671+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.194671+0000 mon.c (mon.2) 333 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.253818+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.253818+0000 mon.a (mon.0) 1616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.256037+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.712 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:52 smithi050 bash[21463]: audit 2024-04-26T16:24:52.256037+0000 mon.b (mon.1) 149 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:24:52.728 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:52.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:52.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:52.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:52.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:52.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:53.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:53.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:53.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:53.151 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:53.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:53.255 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:53.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:53.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:53.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:53.502 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.452467+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:53.502 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.452467+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:53.502 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.461778+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.c scrub starts 2024-04-26T16:24:53.502 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.461778+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.c scrub starts 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.466179+0000 osd.5 (osd.5) 172 : cluster [DBG] 3.c scrub ok 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:52.466179+0000 osd.5 (osd.5) 172 : cluster [DBG] 3.c scrub ok 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.068375+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.068375+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.068635+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.068635+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.068917+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.068917+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.078148+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.078148+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.082379+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.082379+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.083998+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.083998+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.086263+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.086263+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.087342+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.087342+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.144854+0000 osd.2 (osd.2) 135 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.144854+0000 osd.2 (osd.2) 135 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.146044+0000 osd.2 (osd.2) 136 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:24:53.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.146044+0000 osd.2 (osd.2) 136 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:24:53.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.195703+0000 mon.c (mon.2) 335 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:53.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:53 smithi190 bash[23868]: audit 2024-04-26T16:24:53.195703+0000 mon.c (mon.2) 335 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:53.517 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:53.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.452467+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.452467+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.461778+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.c scrub starts 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.461778+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.c scrub starts 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.466179+0000 osd.5 (osd.5) 172 : cluster [DBG] 3.c scrub ok 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:52.466179+0000 osd.5 (osd.5) 172 : cluster [DBG] 3.c scrub ok 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.068375+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.068375+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:53.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.068635+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.068635+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.068917+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.068917+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.078148+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.078148+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.082379+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.082379+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.083998+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.452467+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.452467+0000 mon.a (mon.0) 1617 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.461778+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.c scrub starts 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.461778+0000 osd.5 (osd.5) 171 : cluster [DBG] 3.c scrub starts 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.466179+0000 osd.5 (osd.5) 172 : cluster [DBG] 3.c scrub ok 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:52.466179+0000 osd.5 (osd.5) 172 : cluster [DBG] 3.c scrub ok 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.068375+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.068375+0000 mon.a (mon.0) 1618 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi050-73201-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.068635+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.068635+0000 mon.a (mon.0) 1619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi050-73193-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.068917+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.068917+0000 mon.a (mon.0) 1620 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:24:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.078148+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.078148+0000 mon.a (mon.0) 1621 : cluster [DBG] osdmap e225: 8 total, 8 up, 8 in 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.082379+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.082379+0000 mon.c (mon.2) 334 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.083998+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.083998+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.086263+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.086263+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.087342+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.087342+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.144854+0000 osd.2 (osd.2) 135 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.144854+0000 osd.2 (osd.2) 135 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.146044+0000 osd.2 (osd.2) 136 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.146044+0000 osd.2 (osd.2) 136 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.195703+0000 mon.c (mon.2) 335 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[29234]: audit 2024-04-26T16:24:53.195703+0000 mon.c (mon.2) 335 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.083998+0000 mon.a (mon.0) 1622 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.086263+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.086263+0000 mon.a (mon.0) 1623 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.087342+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.087342+0000 mon.b (mon.1) 150 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]: dispatch 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.144854+0000 osd.2 (osd.2) 135 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:24:53.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.144854+0000 osd.2 (osd.2) 135 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:24:53.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.146044+0000 osd.2 (osd.2) 136 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:24:53.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.146044+0000 osd.2 (osd.2) 136 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:24:53.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.195703+0000 mon.c (mon.2) 335 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:53.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:53 smithi050 bash[21463]: audit 2024-04-26T16:24:53.195703+0000 mon.c (mon.2) 335 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:53.649 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:53.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:53.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:53.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:53.912 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:53.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:54.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:54.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:54.117 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:24:54.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:24:54.118 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-26T16:24:54.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-26T16:24:54.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:54.173 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:54.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:54.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:54.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:54.433 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:54.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:54.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:54.678 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:24:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:24:54.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.536436+0000 mgr.y (mgr.24401) 303 : cluster [DBG] pgmap v322: 281 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 clean+premerge+peered, 32 unknown, 196 active+clean; 598 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-04-26T16:24:54.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.536436+0000 mgr.y (mgr.24401) 303 : cluster [DBG] pgmap v322: 281 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 clean+premerge+peered, 32 unknown, 196 active+clean; 598 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-04-26T16:24:54.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:53.537153+0000 mon.c (mon.2) 336 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:53.537153+0000 mon.c (mon.2) 336 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:53.539958+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:53.539958+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.624620+0000 osd.6 (osd.6) 119 : cluster [DBG] 12.4 scrub starts 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.624620+0000 osd.6 (osd.6) 119 : cluster [DBG] 12.4 scrub starts 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.627232+0000 osd.6 (osd.6) 120 : cluster [DBG] 12.4 scrub ok 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:53.627232+0000 osd.6 (osd.6) 120 : cluster [DBG] 12.4 scrub ok 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.069624+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.069624+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.071832+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.071832+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.071938+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.071938+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.092678+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-04-26T16:24:54.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.092678+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.096991+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.50:0/2389873209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.096991+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.50:0/2389873209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.097690+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.097690+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.196596+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[29234]: audit 2024-04-26T16:24:54.196596+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:54.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.536436+0000 mgr.y (mgr.24401) 303 : cluster [DBG] pgmap v322: 281 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 clean+premerge+peered, 32 unknown, 196 active+clean; 598 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.536436+0000 mgr.y (mgr.24401) 303 : cluster [DBG] pgmap v322: 281 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 clean+premerge+peered, 32 unknown, 196 active+clean; 598 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:53.537153+0000 mon.c (mon.2) 336 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:53.537153+0000 mon.c (mon.2) 336 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:53.539958+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:53.539958+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.624620+0000 osd.6 (osd.6) 119 : cluster [DBG] 12.4 scrub starts 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.624620+0000 osd.6 (osd.6) 119 : cluster [DBG] 12.4 scrub starts 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.627232+0000 osd.6 (osd.6) 120 : cluster [DBG] 12.4 scrub ok 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:53.627232+0000 osd.6 (osd.6) 120 : cluster [DBG] 12.4 scrub ok 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.069624+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.069624+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:54.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.071832+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.071832+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.071938+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.071938+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.092678+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.092678+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.096991+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.50:0/2389873209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.096991+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.50:0/2389873209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.097690+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.097690+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.196596+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:54.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:54 smithi050 bash[21463]: audit 2024-04-26T16:24:54.196596+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:54.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:54.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:54.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.536436+0000 mgr.y (mgr.24401) 303 : cluster [DBG] pgmap v322: 281 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 clean+premerge+peered, 32 unknown, 196 active+clean; 598 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.536436+0000 mgr.y (mgr.24401) 303 : cluster [DBG] pgmap v322: 281 pgs: 12 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 1 clean+premerge+peered, 32 unknown, 196 active+clean; 598 KiB data, 576 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1 op/s 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:53.537153+0000 mon.c (mon.2) 336 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:53.537153+0000 mon.c (mon.2) 336 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:53.539958+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:53.539958+0000 mon.a (mon.0) 1624 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.624620+0000 osd.6 (osd.6) 119 : cluster [DBG] 12.4 scrub starts 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.624620+0000 osd.6 (osd.6) 119 : cluster [DBG] 12.4 scrub starts 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.627232+0000 osd.6 (osd.6) 120 : cluster [DBG] 12.4 scrub ok 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:53.627232+0000 osd.6 (osd.6) 120 : cluster [DBG] 12.4 scrub ok 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.069624+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:54.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.069624+0000 mon.a (mon.0) 1625 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.071832+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.071832+0000 mon.a (mon.0) 1626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-13"}]': finished 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.071938+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.071938+0000 mon.a (mon.0) 1627 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "12"}]': finished 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.092678+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.092678+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e226: 8 total, 8 up, 8 in 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.096991+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.50:0/2389873209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.096991+0000 mon.c (mon.2) 337 : audit [INF] from='client.? 172.21.15.50:0/2389873209' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.097690+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.097690+0000 mon.a (mon.0) 1629 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.196596+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:54.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:54 smithi190 bash[23868]: audit 2024-04-26T16:24:54.196596+0000 mon.c (mon.2) 338 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:54.820 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:54.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:54.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:54.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:55.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:55.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:55.229 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:55.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:55.359 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:55.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:55.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:55.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:55.626 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.334937+0000 osd.0 (osd.0) 187 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.334937+0000 osd.0 (osd.0) 187 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.336159+0000 osd.0 (osd.0) 188 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.336159+0000 osd.0 (osd.0) 188 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.877411+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.877411+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.880126+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:54.880126+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.074748+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.074748+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:55.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.074890+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.074890+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.081330+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.081330+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.084092+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.50:0/1056573515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.084092+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.50:0/1056573515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.085035+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.085035+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.197648+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.334937+0000 osd.0 (osd.0) 187 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.334937+0000 osd.0 (osd.0) 187 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.336159+0000 osd.0 (osd.0) 188 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.336159+0000 osd.0 (osd.0) 188 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.877411+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.877411+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.880126+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:54.880126+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.074748+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.074748+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.074890+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.074890+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.081330+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.081330+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.084092+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.50:0/1056573515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.084092+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.50:0/1056573515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.085035+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.085035+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.197648+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: audit 2024-04-26T16:24:55.197648+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.233723+0000 osd.2 (osd.2) 137 : cluster [DBG] 12.9 scrub starts 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.233723+0000 osd.2 (osd.2) 137 : cluster [DBG] 12.9 scrub starts 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.235015+0000 osd.2 (osd.2) 138 : cluster [DBG] 12.9 scrub ok 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.235015+0000 osd.2 (osd.2) 138 : cluster [DBG] 12.9 scrub ok 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: audit 2024-04-26T16:24:55.197648+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.233723+0000 osd.2 (osd.2) 137 : cluster [DBG] 12.9 scrub starts 2024-04-26T16:24:55.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.233723+0000 osd.2 (osd.2) 137 : cluster [DBG] 12.9 scrub starts 2024-04-26T16:24:55.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.235015+0000 osd.2 (osd.2) 138 : cluster [DBG] 12.9 scrub ok 2024-04-26T16:24:55.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:55 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.235015+0000 osd.2 (osd.2) 138 : cluster [DBG] 12.9 scrub ok 2024-04-26T16:24:55.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:55.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.334937+0000 osd.0 (osd.0) 187 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:24:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.334937+0000 osd.0 (osd.0) 187 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:24:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.336159+0000 osd.0 (osd.0) 188 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:24:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.336159+0000 osd.0 (osd.0) 188 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:24:55.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.877411+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.877411+0000 osd.4 (osd.4) 153 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.880126+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:54.880126+0000 osd.4 (osd.4) 154 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.074748+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.074748+0000 mon.a (mon.0) 1630 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi050-73193-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.074890+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.074890+0000 mon.a (mon.0) 1631 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi050-73363-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.081330+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.081330+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e227: 8 total, 8 up, 8 in 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.084092+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.50:0/1056573515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.084092+0000 mon.c (mon.2) 339 : audit [INF] from='client.? 172.21.15.50:0/1056573515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.085035+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.085035+0000 mon.a (mon.0) 1633 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:55.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.197648+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: audit 2024-04-26T16:24:55.197648+0000 mon.c (mon.2) 340 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.233723+0000 osd.2 (osd.2) 137 : cluster [DBG] 12.9 scrub starts 2024-04-26T16:24:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.233723+0000 osd.2 (osd.2) 137 : cluster [DBG] 12.9 scrub starts 2024-04-26T16:24:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.235015+0000 osd.2 (osd.2) 138 : cluster [DBG] 12.9 scrub ok 2024-04-26T16:24:55.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:55 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.235015+0000 osd.2 (osd.2) 138 : cluster [DBG] 12.9 scrub ok 2024-04-26T16:24:55.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:55.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:55.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:56.020 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:56.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:56.143 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:56.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:56.270 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:56.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:56.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:56.402 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:56.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:56.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:56.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.352444+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:24:56.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.352444+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:24:56.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.354308+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.354308+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.537197+0000 mgr.y (mgr.24401) 304 : cluster [DBG] pgmap v325: 289 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 1 clean+premerge+peered, 72 unknown, 185 active+clean; 596 KiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.537197+0000 mgr.y (mgr.24401) 304 : cluster [DBG] pgmap v325: 289 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 1 clean+premerge+peered, 72 unknown, 185 active+clean; 596 KiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.643250+0000 osd.6 (osd.6) 121 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.643250+0000 osd.6 (osd.6) 121 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.644943+0000 osd.6 (osd.6) 122 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:55.644943+0000 osd.6 (osd.6) 122 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.046580+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.046580+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.055990+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.055990+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.061186+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.061186+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.062316+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.062316+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.198702+0000 mon.c (mon.2) 341 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.198702+0000 mon.c (mon.2) 341 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.304178+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.304178+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.306379+0000 mon.c (mon.2) 342 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[23868]: audit 2024-04-26T16:24:56.306379+0000 mon.c (mon.2) 342 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:56.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:56.660 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:56.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:56.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.352444+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:24:56.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.352444+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:24:56.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.354308+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.354308+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.537197+0000 mgr.y (mgr.24401) 304 : cluster [DBG] pgmap v325: 289 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 1 clean+premerge+peered, 72 unknown, 185 active+clean; 596 KiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.537197+0000 mgr.y (mgr.24401) 304 : cluster [DBG] pgmap v325: 289 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 1 clean+premerge+peered, 72 unknown, 185 active+clean; 596 KiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.643250+0000 osd.6 (osd.6) 121 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.643250+0000 osd.6 (osd.6) 121 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.644943+0000 osd.6 (osd.6) 122 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:55.644943+0000 osd.6 (osd.6) 122 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.046580+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.046580+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.055990+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.055990+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-04-26T16:24:56.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.061186+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.061186+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.062316+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.062316+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.198702+0000 mon.c (mon.2) 341 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.198702+0000 mon.c (mon.2) 341 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.304178+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.304178+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.306379+0000 mon.c (mon.2) 342 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[21463]: audit 2024-04-26T16:24:56.306379+0000 mon.c (mon.2) 342 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:56.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.352444+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.352444+0000 osd.0 (osd.0) 189 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.354308+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.354308+0000 osd.0 (osd.0) 190 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.537197+0000 mgr.y (mgr.24401) 304 : cluster [DBG] pgmap v325: 289 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 1 clean+premerge+peered, 72 unknown, 185 active+clean; 596 KiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.537197+0000 mgr.y (mgr.24401) 304 : cluster [DBG] pgmap v325: 289 pgs: 11 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 1 clean+premerge+peered, 72 unknown, 185 active+clean; 596 KiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.643250+0000 osd.6 (osd.6) 121 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.643250+0000 osd.6 (osd.6) 121 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.644943+0000 osd.6 (osd.6) 122 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:55.644943+0000 osd.6 (osd.6) 122 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.046580+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.046580+0000 mon.a (mon.0) 1634 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi050-73201-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.055990+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-04-26T16:24:56.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.055990+0000 mon.a (mon.0) 1635 : cluster [DBG] osdmap e228: 8 total, 8 up, 8 in 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.061186+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.061186+0000 mon.b (mon.1) 151 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.062316+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.062316+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.198702+0000 mon.c (mon.2) 341 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.198702+0000 mon.c (mon.2) 341 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.304178+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.304178+0000 mon.a (mon.0) 1637 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.306379+0000 mon.c (mon.2) 342 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:56.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:56 smithi050 bash[29234]: audit 2024-04-26T16:24:56.306379+0000 mon.c (mon.2) 342 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:24:56.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:56.819 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:24:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:24:56.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:56.921 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:56.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:57.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:57.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:57.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:57.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:57.224 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:57.224 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:57.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:57.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:57.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:57.443 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:57.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:57.574 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.401027+0000 osd.0 (osd.0) 191 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.401027+0000 osd.0 (osd.0) 191 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.402413+0000 osd.0 (osd.0) 192 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.402413+0000 osd.0 (osd.0) 192 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.656726+0000 osd.7 (osd.7) 193 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.656726+0000 osd.7 (osd.7) 193 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.657714+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.657714+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:24:57.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.661164+0000 osd.7 (osd.7) 194 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.661164+0000 osd.7 (osd.7) 194 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.661969+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:56.661969+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.050876+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.050876+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.057331+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.057331+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.071844+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.071844+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.072289+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.50:0/3362762347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.072289+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.50:0/3362762347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.072885+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.401027+0000 osd.0 (osd.0) 191 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.401027+0000 osd.0 (osd.0) 191 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.402413+0000 osd.0 (osd.0) 192 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.402413+0000 osd.0 (osd.0) 192 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.656726+0000 osd.7 (osd.7) 193 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.656726+0000 osd.7 (osd.7) 193 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.657714+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.657714+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.661164+0000 osd.7 (osd.7) 194 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.661164+0000 osd.7 (osd.7) 194 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:24:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.661969+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:56.661969+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.050876+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.050876+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.057331+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.057331+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.071844+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.071844+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.072289+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.50:0/3362762347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.072289+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.50:0/3362762347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.072885+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.072885+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.073306+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.073306+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.199874+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[29234]: audit 2024-04-26T16:24:57.199874+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.072885+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.073306+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.073306+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.199874+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:57 smithi050 bash[21463]: audit 2024-04-26T16:24:57.199874+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:57.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.401027+0000 osd.0 (osd.0) 191 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.401027+0000 osd.0 (osd.0) 191 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.402413+0000 osd.0 (osd.0) 192 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.402413+0000 osd.0 (osd.0) 192 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.656726+0000 osd.7 (osd.7) 193 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.656726+0000 osd.7 (osd.7) 193 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.657714+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.657714+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.661164+0000 osd.7 (osd.7) 194 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:24:57.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.661164+0000 osd.7 (osd.7) 194 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.661969+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:56.661969+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.050876+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.050876+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.057331+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.057331+0000 mon.a (mon.0) 1639 : cluster [DBG] osdmap e229: 8 total, 8 up, 8 in 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.071844+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.071844+0000 mon.c (mon.2) 343 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.072289+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.50:0/3362762347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.072289+0000 mon.c (mon.2) 344 : audit [INF] from='client.? 172.21.15.50:0/3362762347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.072885+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.072885+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.073306+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.073306+0000 mon.a (mon.0) 1641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:57.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.199874+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:57.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:57 smithi190 bash[23868]: audit 2024-04-26T16:24:57.199874+0000 mon.c (mon.2) 345 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:57.704 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:57.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:57.837 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:24:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:24:57.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:57.968 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:57.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:58.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:58.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:58.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:58.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:58.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:58.373 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:58.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:58.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:58.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:58.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:56.691263+0000 mgr.y (mgr.24401) 305 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:58.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:56.691263+0000 mgr.y (mgr.24401) 305 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:58.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.359997+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.3 scrub starts 2024-04-26T16:24:58.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.359997+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.3 scrub starts 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.362260+0000 osd.0 (osd.0) 194 : cluster [DBG] 12.3 scrub ok 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.362260+0000 osd.0 (osd.0) 194 : cluster [DBG] 12.3 scrub ok 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.538247+0000 mgr.y (mgr.24401) 306 : cluster [DBG] pgmap v328: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:57.538247+0000 mgr.y (mgr.24401) 306 : cluster [DBG] pgmap v328: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:57.539011+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:57.539011+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:57.542016+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:57.542016+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.055140+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.055140+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.055397+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:58.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.055397+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.055617+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.055617+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.063999+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.063999+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.067180+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.067180+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.081023+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.081023+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.081908+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.081908+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.116779+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.116779+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.118592+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.118592+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.201131+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: audit 2024-04-26T16:24:58.201131+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.263873+0000 osd.2 (osd.2) 139 : cluster [DBG] 12.a scrub starts 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.263873+0000 osd.2 (osd.2) 139 : cluster [DBG] 12.a scrub starts 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.265761+0000 osd.2 (osd.2) 140 : cluster [DBG] 12.a scrub ok 2024-04-26T16:24:58.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:58 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.265761+0000 osd.2 (osd.2) 140 : cluster [DBG] 12.a scrub ok 2024-04-26T16:24:58.630 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:58.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:56.691263+0000 mgr.y (mgr.24401) 305 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:56.691263+0000 mgr.y (mgr.24401) 305 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.359997+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.3 scrub starts 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.359997+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.3 scrub starts 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.362260+0000 osd.0 (osd.0) 194 : cluster [DBG] 12.3 scrub ok 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.362260+0000 osd.0 (osd.0) 194 : cluster [DBG] 12.3 scrub ok 2024-04-26T16:24:58.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.538247+0000 mgr.y (mgr.24401) 306 : cluster [DBG] pgmap v328: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:57.538247+0000 mgr.y (mgr.24401) 306 : cluster [DBG] pgmap v328: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:57.539011+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:57.539011+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:57.542016+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:57.542016+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.055140+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.055140+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.055397+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.055397+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:58.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:56.691263+0000 mgr.y (mgr.24401) 305 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:56.691263+0000 mgr.y (mgr.24401) 305 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.359997+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.3 scrub starts 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.359997+0000 osd.0 (osd.0) 193 : cluster [DBG] 12.3 scrub starts 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.362260+0000 osd.0 (osd.0) 194 : cluster [DBG] 12.3 scrub ok 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.362260+0000 osd.0 (osd.0) 194 : cluster [DBG] 12.3 scrub ok 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.538247+0000 mgr.y (mgr.24401) 306 : cluster [DBG] pgmap v328: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:57.538247+0000 mgr.y (mgr.24401) 306 : cluster [DBG] pgmap v328: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s wr, 5 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:57.539011+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:57.539011+0000 mon.c (mon.2) 346 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:57.542016+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:57.542016+0000 mon.a (mon.0) 1642 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.055140+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:58.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.055140+0000 mon.a (mon.0) 1643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.055397+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.055397+0000 mon.a (mon.0) 1644 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi050-73363-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.055617+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.055617+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.063999+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.063999+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.067180+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.067180+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.081023+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.081023+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.081908+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.081908+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.116779+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.116779+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.118592+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.118592+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.201131+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.055617+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:24:58.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.055617+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi050-73230-1", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.063999+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.063999+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e230: 8 total, 8 up, 8 in 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.067180+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.067180+0000 mon.c (mon.2) 347 : audit [INF] from='client.? 172.21.15.50:0/3462158912' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.081023+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.081023+0000 mon.a (mon.0) 1647 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.081908+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.081908+0000 mon.a (mon.0) 1648 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.116779+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.116779+0000 mon.a (mon.0) 1649 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.118592+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.118592+0000 mon.b (mon.1) 152 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:24:58.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.201131+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: audit 2024-04-26T16:24:58.201131+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.263873+0000 osd.2 (osd.2) 139 : cluster [DBG] 12.a scrub starts 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.263873+0000 osd.2 (osd.2) 139 : cluster [DBG] 12.a scrub starts 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.265761+0000 osd.2 (osd.2) 140 : cluster [DBG] 12.a scrub ok 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.265761+0000 osd.2 (osd.2) 140 : cluster [DBG] 12.a scrub ok 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: audit 2024-04-26T16:24:58.201131+0000 mon.c (mon.2) 348 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.263873+0000 osd.2 (osd.2) 139 : cluster [DBG] 12.a scrub starts 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.263873+0000 osd.2 (osd.2) 139 : cluster [DBG] 12.a scrub starts 2024-04-26T16:24:58.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.265761+0000 osd.2 (osd.2) 140 : cluster [DBG] 12.a scrub ok 2024-04-26T16:24:58.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:58 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.265761+0000 osd.2 (osd.2) 140 : cluster [DBG] 12.a scrub ok 2024-04-26T16:24:58.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:58.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:58.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:58.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:58.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:59.024 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:24:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:24:59.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:59.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:24:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:24:59.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:59.267 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:24:59.280 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:24:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:24:59.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:59.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:24:59.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:24:59.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:59.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.496191+0000 mon.a (mon.0) 1650 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:59.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.496191+0000 mon.a (mon.0) 1650 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:59.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.680913+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.680913+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.683175+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:58.683175+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.059232+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.059232+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.059384+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.059384+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.059564+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.059564+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.067999+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.067999+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.075893+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.075893+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.076418+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.076418+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.137285+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.137285+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.138300+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.138300+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.138942+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.138942+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.139591+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.139591+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.139899+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.139899+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.140853+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.140853+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.141374+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.141374+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.141877+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.141877+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.142282+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.142282+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.202560+0000 mon.c (mon.2) 352 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:59.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:24:59 smithi190 bash[23868]: audit 2024-04-26T16:24:59.202560+0000 mon.c (mon.2) 352 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:59.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:24:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:24:59.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:59.668 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:24:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:24:59.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.496191+0000 mon.a (mon.0) 1650 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.496191+0000 mon.a (mon.0) 1650 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.680913+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.680913+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.683175+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:58.683175+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.059232+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.059232+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.059384+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:59.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.059384+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.059564+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.059564+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.067999+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.067999+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.075893+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.075893+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.785 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.076418+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.496191+0000 mon.a (mon.0) 1650 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.496191+0000 mon.a (mon.0) 1650 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.680913+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.680913+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.683175+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:58.683175+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.059232+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:59.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.059232+0000 mon.a (mon.0) 1651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi050-73193-28"}]': finished 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.059384+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.059384+0000 mon.a (mon.0) 1652 : audit [INF] from='client.? 172.21.15.50:0/4188921332' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi050-73201-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.059564+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.059564+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.067999+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.067999+0000 mon.a (mon.0) 1654 : cluster [DBG] osdmap e231: 8 total, 8 up, 8 in 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.075893+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.075893+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.076418+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.076418+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.137285+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.137285+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.138300+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.138300+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.138942+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.138942+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.139591+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.139591+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.139899+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.139899+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.140853+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.140853+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.141374+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.141374+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.141877+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.141877+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.142282+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.142282+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.202560+0000 mon.c (mon.2) 352 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[21463]: audit 2024-04-26T16:24:59.202560+0000 mon.c (mon.2) 352 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.076418+0000 mon.b (mon.1) 153 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.137285+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.137285+0000 mon.c (mon.2) 349 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.138300+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.138300+0000 mon.a (mon.0) 1656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.138942+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.138942+0000 mon.a (mon.0) 1657 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.139591+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.139591+0000 mon.c (mon.2) 350 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.139899+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.139899+0000 mon.a (mon.0) 1658 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.140853+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.140853+0000 mon.c (mon.2) 351 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.141374+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.141374+0000 mon.a (mon.0) 1659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.141877+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.141877+0000 mon.a (mon.0) 1660 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:24:59.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.142282+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.142282+0000 mon.a (mon.0) 1661 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:24:59.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.202560+0000 mon.c (mon.2) 352 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:59.791 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:24:59 smithi050 bash[29234]: audit 2024-04-26T16:24:59.202560+0000 mon.c (mon.2) 352 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:24:59.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:24:59.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:59.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:24:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:24:59.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:00.007 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:25:00.007 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(6.2,QREQUEST,e:103,from:4,reservation_nonce:2)", 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:18:47.066435+0000", 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 372.94561203000001, 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019798554999999999, 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.008 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.066435+0000", 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.066435+0000", 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.066438+0000", 2024-04-26T16:25:00.009 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1429999999999998e-06 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.066450+0000", 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2636000000000001e-05 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.085774+0000", 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019323880000000002 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.010 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.085804+0000", 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9569000000000001e-05 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.086066+0000", 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026273300000000002 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.086113+0000", 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6555999999999999e-05 2024-04-26T16:25:00.011 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:18:47.086233+0000", 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012003800000000001 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:248 109.6 109:6035fad1:test-rados-api-smithi050-73372-12::bar:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:01.132755+0000", 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 58.879291139000003, 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.038417163999999997, 2024-04-26T16:25:00.012 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 248 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.132755+0000", 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.013 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.132755+0000", 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.132757+0000", 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5090000000000001e-06 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.014 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.132764+0000", 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.6839999999999999e-06 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.132776+0000", 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2427e-05 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.132786+0000", 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0159e-05 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.015 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.138716+0000", 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0059296330000000001 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.138833+0000", 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000117042 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.138995+0000", 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016189700000000001 2024-04-26T16:25:00.016 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.152489+0000", 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013493745 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.171029+0000", 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018540099000000001 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.171050+0000", 2024-04-26T16:25:00.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.122e-05 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.171125+0000", 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4822999999999998e-05 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.171173+0000", 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7926000000000002e-05 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.018 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:253 109.6 109:6035fad1:test-rados-api-smithi050-73372-12::bar:head [writefull 0~5] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:02.123203+0000", 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 57.888843195, 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021794724000000001, 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:00.019 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 253 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123203+0000", 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123203+0000", 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123205+0000", 2024-04-26T16:25:00.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.68e-06 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123214+0000", 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.448e-06 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123226+0000", 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1187e-05 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123237+0000", 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0923e-05 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123337+0000", 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00010058199999999999 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123442+0000", 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000104578 2024-04-26T16:25:00.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,3", 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.123629+0000", 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000187734 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.130443+0000", 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0068136990000000003 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.136245+0000", 2024-04-26T16:25:00.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058017069999999997 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.144904+0000", 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.008658832 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.144954+0000", 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0194000000000001e-05 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:02.144998+0000", 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4159999999999997e-05 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.3.0:1091 17.9 e176/141 -19:9514177a:::temp_17.9_0_14316_1:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.550851+0000", 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.461195343, 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025184343000000001, 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.025 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.550851+0000", 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.550851+0000", 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.550853+0000", 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0530000000000001e-06 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.561659+0000", 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010805463 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.561690+0000", 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1032999999999999e-05 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.561704+0000", 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3755e-05 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.561792+0000", 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8295999999999996e-05 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.561817+0000", 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5083999999999999e-05 2024-04-26T16:25:00.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.575937+0000", 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014119862 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.576036+0000", 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8796999999999996e-05 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.3.0:1092 17.9 e176/141 -19:9514177a:::temp_17.9_0_14316_1:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.575367+0000", 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.436679734999998, 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017357448000000001, 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.575367+0000", 2024-04-26T16:25:00.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.575367+0000", 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.575368+0000", 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4169999999999999e-06 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.581264+0000", 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058960499999999999 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.581301+0000", 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7082000000000002e-05 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.581319+0000", 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7509000000000001e-05 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.581402+0000", 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3399000000000006e-05 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.581427+0000", 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4939e-05 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.592648+0000", 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011220288 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.592724+0000", 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6763999999999999e-05 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.3.0:1093 17.9 e176/141 -19:9514177a:::temp_17.9_0_14316_1:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.598501+0000", 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.413545755999998, 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023078654000000001, 2024-04-26T16:25:00.034 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.598501+0000", 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.598501+0000", 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.598503+0000", 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3889999999999999e-06 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.605219+0000", 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0067159139999999999 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.605244+0000", 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5060000000000001e-05 2024-04-26T16:25:00.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.605256+0000", 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1654999999999999e-05 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.605359+0000", 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000103082 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.605384+0000", 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5554999999999998e-05 2024-04-26T16:25:00.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.621482+0000", 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016097230000000001 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.621579+0000", 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7769000000000005e-05 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.24770.0:2080 17.9 e176/141 17:9514177a:test-rados-api-smithi050-73363-17::big3.copy:head v 176'1, mlcod=0'0)", 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.615500+0000", 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 55.396546201, 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021422198999999999, 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.615500+0000", 2024-04-26T16:25:00.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.615500+0000", 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.615503+0000", 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3910000000000001e-06 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.617313+0000", 2024-04-26T16:25:00.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0018098669999999999 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.617361+0000", 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8847000000000002e-05 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.617383+0000", 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1353999999999999e-05 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.617751+0000", 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000368358 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.617782+0000", 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0670999999999998e-05 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.636852+0000", 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019070297 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.636923+0000", 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0413999999999994e-05 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:256 157.7 157:e47faca5:test-rados-api-smithi050-73372-12::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e177)", 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:05.076021+0000", 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 54.936025348999998, 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.105873013, 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.043 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 256 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.076021+0000", 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.076021+0000", 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.076023+0000", 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0420000000000001e-06 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.076032+0000", 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4749999999999993e-06 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.076049+0000", 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7238e-05 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.076061+0000", 2024-04-26T16:25:00.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1673e-05 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.116985+0000", 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.040924834 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for blocked object", 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.117126+0000", 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014082299999999999 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.172902+0000", 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055775243000000002 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.172994+0000", 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.2884e-05 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.181470+0000", 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084757500000000006 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.181673+0000", 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020278700000000001 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.181894+0000", 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022126399999999999 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15429.0:63 112.1a 112:5ab71872:test-rados-api-smithi050-73432-14::foo:head [write 0~128] snapc d=[d] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:06.027685+0000", 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.984361686, 2024-04-26T16:25:00.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.145486371, 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15429", 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3301772666", 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 63 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027685+0000", 2024-04-26T16:25:00.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027685+0000", 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027687+0000", 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.708e-06 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027693+0000", 2024-04-26T16:25:00.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9789999999999998e-06 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027707+0000", 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3964e-05 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027718+0000", 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1629e-05 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027803+0000", 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4894000000000004e-05 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.027811+0000", 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8180000000000004e-06 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.113572+0000", 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.085761518999999994 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.113713+0000", 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014073499999999999 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.113875+0000", 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000161433 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.141941+0000", 2024-04-26T16:25:00.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028066524999999998 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.144799+0000", 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028580939999999998 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.173072+0000", 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028272647000000001 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.173128+0000", 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6450000000000003e-05 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.173171+0000", 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2976000000000003e-05 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.40501.0:2 166.1b e178/177 166:d96a01e0:::testobj:head v 178'1, mlcod=0'0)", 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:06.142681+0000", 2024-04-26T16:25:00.056 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 53.869365842000001, 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.039132387999999997, 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.142681+0000", 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.142681+0000", 2024-04-26T16:25:00.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.142681+0000", 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2899999999999999e-07 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.142686+0000", 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5340000000000001e-06 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.058 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.142692+0000", 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.348e-06 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.142698+0000", 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2779999999999999e-06 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.143604+0000", 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090558700000000002 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.143617+0000", 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.269e-05 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.181780+0000", 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.03816311 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.181813+0000", 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3411999999999998e-05 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.060 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.40442.0:2 174.e 174:72c88bc2:::oid_IsCompletePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:11.063269+0000", 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.948777526000001, 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033661009999999998, 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.40442", 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/4068896200", 2024-04-26T16:25:00.061 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.063269+0000", 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.063269+0000", 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.063271+0000", 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.95e-06 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.063278+0000", 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0110000000000003e-06 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.063302+0000", 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3893e-05 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.063316+0000", 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4474999999999999e-05 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.067868+0000", 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045513610000000003 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.067878+0000", 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0076e-05 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.091517+0000", 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023639245999999999 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.091941+0000", 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00042373900000000002 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,3", 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.092099+0000", 2024-04-26T16:25:00.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000158519 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.094869+0000", 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0027698079999999999 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.095493+0000", 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062368500000000002 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.096828+0000", 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013356450000000001 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.096887+0000", 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8255999999999999e-05 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.096930+0000", 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3346000000000001e-05 2024-04-26T16:25:00.067 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.43033.0:2 175.6 175:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:11.079605+0000", 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.932442010000003, 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.043422876999999999, 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.43033", 2024-04-26T16:25:00.068 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/826194399", 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079605+0000", 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079605+0000", 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079607+0000", 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4320000000000002e-06 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079615+0000", 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.5820000000000004e-06 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079626+0000", 2024-04-26T16:25:00.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.114e-05 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079638+0000", 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1977999999999999e-05 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079710+0000", 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2381999999999999e-05 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:00.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.079721+0000", 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1100999999999999e-05 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.112785+0000", 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033063836999999999 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.112922+0000", 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000136515 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.113076+0000", 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000154031 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.115032+0000", 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019564980000000001 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.118566+0000", 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0035338320000000002 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.122962+0000", 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0043957370000000003 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.122996+0000", 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4634000000000001e-05 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:11.123027+0000", 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1177999999999999e-05 2024-04-26T16:25:00.074 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.25249.0:470 109.10 e184/155 109:0807fb86:test-rados-api-smithi050-73372-14::foo0:head v 184'9, mlcod=183'6)", 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:12.071313+0000", 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.940733473999998, 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021377858999999999, 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.071313+0000", 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.071313+0000", 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.071314+0000", 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4899999999999995e-07 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.071332+0000", 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8634000000000002e-05 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.071344+0000", 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2086000000000001e-05 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.071359+0000", 2024-04-26T16:25:00.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4098e-05 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.084793+0000", 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013434543 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.084822+0000", 2024-04-26T16:25:00.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9309e-05 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.091861+0000", 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070388739999999997 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:12.092691+0000", 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082966599999999995 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.080 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(osd.2.0:949 109.8 e220/155 ondisk, result = 0)", 2024-04-26T16:25:00.080 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:49.059995+0000", 2024-04-26T16:25:00.080 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 10.952051204, 2024-04-26T16:25:00.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018826030000000001, 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.059995+0000", 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.059995+0000", 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.059996+0000", 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5900000000000005e-07 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.060002+0000", 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5570000000000003e-06 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.062510+0000", 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002508443 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.062533+0000", 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2546000000000001e-05 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.078783+0000", 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016250257000000001 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.078794+0000", 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0900000000000001e-05 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.078821+0000", 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7368e-05 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.40646.0:8 207.5s0 207:b50e50fa:test-rados-api-smithi050-73432-22::foo:head [delete] snapc 3=[3,2] ondisk+write+known_if_redirected+full_force+supports_pool_eio e223)", 2024-04-26T16:25:00.084 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:51.091209+0000", 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.9208379749999995, 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.020119603, 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.40646", 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1555660762", 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 8 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.091209+0000", 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.091209+0000", 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.091211+0000", 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9740000000000001e-06 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.091219+0000", 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5790000000000004e-06 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.093780+0000", 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025609069999999999 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.093798+0000", 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8201000000000001e-05 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.093927+0000", 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012858499999999999 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.094010+0000", 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3152000000000006e-05 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.094502+0000", 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00049234699999999997 2024-04-26T16:25:00.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095588+0000", 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001085782 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.111249+0000", 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015661185000000001 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.111328+0000", 2024-04-26T16:25:00.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.8891000000000001e-05 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.5.0:641 109.13 e223/155 109:cfa6ab7a:test-rados-api-smithi050-73372-14::foo7:6 v 223'33, mlcod=221'29)", 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:51.095860+0000", 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 8.9161864099999999, 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025803700999999998, 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.090 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095860+0000", 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095860+0000", 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095861+0000", 2024-04-26T16:25:00.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8400000000000004e-07 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095867+0000", 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7749999999999998e-06 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095875+0000", 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4449999999999996e-06 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.095882+0000", 2024-04-26T16:25:00.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7309999999999996e-06 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.108124+0000", 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012242496 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.108144+0000", 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9789e-05 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.121590+0000", 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013445896000000001 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:51.121664+0000", 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3885000000000002e-05 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.40646.0:26 207.5s0 207:ae3ae4e5:test-rados-api-smithi050-73432-24::foo:head [write 0~8192] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-04-26T16:25:00.094 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:56.069699+0000", 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 3.9423472039999998, 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055397287000000003, 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.40646", 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1555660762", 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 26 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.069699+0000", 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.069699+0000", 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.069702+0000", 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8169999999999999e-06 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.069732+0000", 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0113000000000001e-05 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.088227+0000", 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018494666999999999 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.088256+0000", 2024-04-26T16:25:00.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9142e-05 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.097976+0000", 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0097200959999999993 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.098193+0000", 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021717899999999999 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:25:00.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.098818+0000", 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062509600000000001 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.112340+0000", 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013521716 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.125014+0000", 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012673715 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:56.125097+0000", 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2745999999999997e-05 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.40646.0:28 207.5s0 207:ae3ae4e5:test-rados-api-smithi050-73432-24::foo:head [stat,delete] snapc 7=[7,6] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:57.071068+0000", 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.9409781050000001, 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.049317976999999999, 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.100 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.40646", 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1555660762", 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 28 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071068+0000", 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071068+0000", 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071071+0000", 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2800000000000002e-06 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071080+0000", 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6029999999999994e-06 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.078397+0000", 2024-04-26T16:25:00.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0073167579999999996 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.082265+0000", 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003867629 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.082334+0000", 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8830000000000003e-05 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.082432+0000", 2024-04-26T16:25:00.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.8523999999999994e-05 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.088375+0000", 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005943205 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.091558+0000", 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031826089999999999 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.120304+0000", 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028745769000000001 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.120386+0000", 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2769999999999995e-05 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:1405 109.15 109:ae5d747d:test-rados-api-smithi050-73372-16::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:57.071449+0000", 2024-04-26T16:25:00.105 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 2.9405977810000001, 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.049292492, 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1405 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071449+0000", 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071449+0000", 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071450+0000", 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2869999999999999e-06 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.071460+0000", 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6860000000000001e-06 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.078421+0000", 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0069616080000000002 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.078437+0000", 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5749000000000001e-05 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.081332+0000", 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0028944919999999998 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.081491+0000", 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015894699999999999 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,7", 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.081728+0000", 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00023752699999999999 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.091544+0000", 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0098158919999999997 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.093924+0000", 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002379843 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.120146+0000", 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026222134000000001 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.120700+0000", 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000554439 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:57.120741+0000", 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0887999999999999e-05 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:1407 109.15 109:ae5d747d:test-rados-api-smithi050-73372-16::foo:head [writefull 0~5] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-04-26T16:25:00.111 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:58.069838+0000", 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 1.942208376, 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.045342055999999999, 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1407 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:00.112 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.069838+0000", 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.069838+0000", 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.069842+0000", 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8600000000000003e-06 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.113 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.069864+0000", 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1875999999999999e-05 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.069902+0000", 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8538000000000003e-05 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.069931+0000", 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8929e-05 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.114 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.070280+0000", 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034883199999999998 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.070303+0000", 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2861e-05 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.094377+0000", 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024074155 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.115 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.094517+0000", 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000139476 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,7", 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.094776+0000", 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025938099999999999 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.101895+0000", 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071189210000000003 2024-04-26T16:25:00.116 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.108279+0000", 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063835289999999998 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.114951+0000", 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0066724289999999997 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.115121+0000", 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000169705 2024-04-26T16:25:00.117 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:58.115180+0000", 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9564000000000001e-05 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:00.118 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:25:00.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:00.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:00.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:00.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:00.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:00.397 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:00.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:00.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:00.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:00.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:00.659 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:00.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:00.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.226528+0000 osd.1 (osd.1) 169 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.226528+0000 osd.1 (osd.1) 169 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.228329+0000 osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.228329+0000 osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.539365+0000 mgr.y (mgr.24401) 307 : cluster [DBG] pgmap v331: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 7 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.539365+0000 mgr.y (mgr.24401) 307 : cluster [DBG] pgmap v331: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 7 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.572396+0000 osd.5 (osd.5) 173 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.572396+0000 osd.5 (osd.5) 173 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.573726+0000 osd.5 (osd.5) 174 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.573726+0000 osd.5 (osd.5) 174 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.585081+0000 osd.7 (osd.7) 195 : cluster [DBG] 3.f scrub starts 2024-04-26T16:25:00.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.585081+0000 osd.7 (osd.7) 195 : cluster [DBG] 3.f scrub starts 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.587807+0000 osd.7 (osd.7) 196 : cluster [DBG] 3.f scrub ok 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.587807+0000 osd.7 (osd.7) 196 : cluster [DBG] 3.f scrub ok 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.654483+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.5 scrub starts 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.654483+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.5 scrub starts 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.657705+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.5 scrub ok 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:24:59.657705+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.5 scrub ok 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.061395+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.061395+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.061499+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.061499+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.061577+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.061577+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:25:00.066200+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: cluster 2024-04-26T16:25:00.066200+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.066471+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.066471+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.067464+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.067464+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.067634+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.067634+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:00.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.068181+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.068181+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.069343+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.226528+0000 osd.1 (osd.1) 169 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.226528+0000 osd.1 (osd.1) 169 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.228329+0000 osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.228329+0000 osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.539365+0000 mgr.y (mgr.24401) 307 : cluster [DBG] pgmap v331: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 7 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.539365+0000 mgr.y (mgr.24401) 307 : cluster [DBG] pgmap v331: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 7 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.572396+0000 osd.5 (osd.5) 173 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.572396+0000 osd.5 (osd.5) 173 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:25:00.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.573726+0000 osd.5 (osd.5) 174 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.573726+0000 osd.5 (osd.5) 174 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.585081+0000 osd.7 (osd.7) 195 : cluster [DBG] 3.f scrub starts 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.585081+0000 osd.7 (osd.7) 195 : cluster [DBG] 3.f scrub starts 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.587807+0000 osd.7 (osd.7) 196 : cluster [DBG] 3.f scrub ok 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.587807+0000 osd.7 (osd.7) 196 : cluster [DBG] 3.f scrub ok 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.654483+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.5 scrub starts 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.654483+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.5 scrub starts 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.657705+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.5 scrub ok 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:24:59.657705+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.5 scrub ok 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.061395+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.061395+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.061499+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.061499+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.069343+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.071748+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.071748+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.073335+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.073335+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.203773+0000 mon.c (mon.2) 355 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:00.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[29234]: audit 2024-04-26T16:25:00.203773+0000 mon.c (mon.2) 355 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.061577+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.061577+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:25:00.066200+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: cluster 2024-04-26T16:25:00.066200+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.066471+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.066471+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.067464+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.067464+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.067634+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.067634+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.068181+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.068181+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.069343+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.069343+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.071748+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.071748+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.073335+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.073335+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.203773+0000 mon.c (mon.2) 355 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:00.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:00 smithi050 bash[21463]: audit 2024-04-26T16:25:00.203773+0000 mon.c (mon.2) 355 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:00.791 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.226528+0000 osd.1 (osd.1) 169 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.226528+0000 osd.1 (osd.1) 169 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.228329+0000 osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.228329+0000 osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.539365+0000 mgr.y (mgr.24401) 307 : cluster [DBG] pgmap v331: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 7 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.539365+0000 mgr.y (mgr.24401) 307 : cluster [DBG] pgmap v331: 280 pgs: 1 peering, 12 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 32 creating+peering, 32 unknown, 184 active+clean; 628 KiB data, 557 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 12 KiB/s wr, 7 op/s; 63 B/s, 0 objects/s recovering 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.572396+0000 osd.5 (osd.5) 173 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.572396+0000 osd.5 (osd.5) 173 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.573726+0000 osd.5 (osd.5) 174 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:25:00.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.573726+0000 osd.5 (osd.5) 174 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.585081+0000 osd.7 (osd.7) 195 : cluster [DBG] 3.f scrub starts 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.585081+0000 osd.7 (osd.7) 195 : cluster [DBG] 3.f scrub starts 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.587807+0000 osd.7 (osd.7) 196 : cluster [DBG] 3.f scrub ok 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.587807+0000 osd.7 (osd.7) 196 : cluster [DBG] 3.f scrub ok 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.654483+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.5 scrub starts 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.654483+0000 osd.6 (osd.6) 127 : cluster [DBG] 12.5 scrub starts 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.657705+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.5 scrub ok 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:24:59.657705+0000 osd.6 (osd.6) 128 : cluster [DBG] 12.5 scrub ok 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.061395+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:00.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.061395+0000 mon.a (mon.0) 1662 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.061499+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.061499+0000 mon.a (mon.0) 1663 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi050-73193-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.061577+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.061577+0000 mon.a (mon.0) 1664 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:25:00.066200+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: cluster 2024-04-26T16:25:00.066200+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e232: 8 total, 8 up, 8 in 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.066471+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.066471+0000 mon.c (mon.2) 353 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.067464+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.067464+0000 mon.c (mon.2) 354 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.067634+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.067634+0000 mon.a (mon.0) 1666 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:00.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.068181+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.068181+0000 mon.a (mon.0) 1667 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.069343+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.069343+0000 mon.a (mon.0) 1668 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.071748+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.071748+0000 mon.a (mon.0) 1669 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.073335+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.073335+0000 mon.b (mon.1) 154 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.203773+0000 mon.c (mon.2) 355 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:00.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:00 smithi190 bash[23868]: audit 2024-04-26T16:25:00.203773+0000 mon.c (mon.2) 355 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:00.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:00.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:00.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:01.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:01.053 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:01.157 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:01.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:01.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:01.308 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:01.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:01.439 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:01.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:01.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:01.570 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:01.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:01.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:01.701 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:01.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:01.833 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:01.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:01.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:01.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:02.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:00.677713+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1 scrub starts 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:00.677713+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1 scrub starts 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:00.681111+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1 scrub ok 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:00.681111+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1 scrub ok 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.006875+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.006875+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-04-26T16:25:02.082 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.008832+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.008832+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.067860+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.067860+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.070263+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.070263+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.070560+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]': finished 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.070560+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]': finished 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.089111+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.089111+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.129120+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-04-26T16:25:02.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.129120+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.130582+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.50:0/3844474308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.130582+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.50:0/3844474308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.130780+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.130780+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.134746+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.134746+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.205095+0000 mon.c (mon.2) 358 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: audit 2024-04-26T16:25:01.205095+0000 mon.c (mon.2) 358 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.541831+0000 mgr.y (mgr.24401) 308 : cluster [DBG] pgmap v334: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 creating+peering, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:02.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.541831+0000 mgr.y (mgr.24401) 308 : cluster [DBG] pgmap v334: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 creating+peering, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:00.677713+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1 scrub starts 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:00.677713+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1 scrub starts 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:00.681111+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1 scrub ok 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:00.681111+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1 scrub ok 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.006875+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.006875+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.008832+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-04-26T16:25:02.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.008832+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.067860+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.067860+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.070263+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.070263+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.070560+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]': finished 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.070560+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]': finished 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.089111+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.089111+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.129120+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.129120+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-04-26T16:25:02.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.130582+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.50:0/3844474308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.130582+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.50:0/3844474308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.130780+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.130780+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.134746+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.134746+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.205095+0000 mon.c (mon.2) 358 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: audit 2024-04-26T16:25:01.205095+0000 mon.c (mon.2) 358 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.541831+0000 mgr.y (mgr.24401) 308 : cluster [DBG] pgmap v334: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 creating+peering, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:02.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:02 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.541831+0000 mgr.y (mgr.24401) 308 : cluster [DBG] pgmap v334: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 creating+peering, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:02.111 INFO:tasks.workunit.client.0.smithi050.stdout: pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-04-26T16:25:02.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2006 ms) 2024-04-26T16:25:02.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9000 ms total) 2024-04-26T16:25:02.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: 2024-04-26T16:25:02.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-04-26T16:25:02.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4044 ms) 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3970 ms) 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4002 ms) 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12017 ms total) 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (94055 ms total) 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-04-26T16:25:02.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-04-26T16:25:02.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:02.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:02.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:00.677713+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1 scrub starts 2024-04-26T16:25:02.233 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:00.677713+0000 osd.6 (osd.6) 129 : cluster [DBG] 12.1 scrub starts 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:00.681111+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1 scrub ok 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:00.681111+0000 osd.6 (osd.6) 130 : cluster [DBG] 12.1 scrub ok 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.006875+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.006875+0000 osd.3 (osd.3) 197 : cluster [DBG] 6.b scrub starts 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.008832+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.008832+0000 osd.3 (osd.3) 198 : cluster [DBG] 6.b scrub ok 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.067860+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.067860+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.070263+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:02.234 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.070263+0000 mon.a (mon.0) 1671 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.070560+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]': finished 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.070560+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-15", "mode": "writeback"}]': finished 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.089111+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.089111+0000 mon.c (mon.2) 356 : audit [INF] from='client.? 172.21.15.50:0/1555660762' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.129120+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.129120+0000 mon.a (mon.0) 1673 : cluster [DBG] osdmap e233: 8 total, 8 up, 8 in 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.130582+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.50:0/3844474308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.130582+0000 mon.c (mon.2) 357 : audit [INF] from='client.? 172.21.15.50:0/3844474308' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.130780+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.130780+0000 mon.a (mon.0) 1674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.134746+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.134746+0000 mon.a (mon.0) 1675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.205095+0000 mon.c (mon.2) 358 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:02.235 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: audit 2024-04-26T16:25:01.205095+0000 mon.c (mon.2) 358 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:02.236 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.541831+0000 mgr.y (mgr.24401) 308 : cluster [DBG] pgmap v334: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 creating+peering, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:02.236 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:02 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.541831+0000 mgr.y (mgr.24401) 308 : cluster [DBG] pgmap v334: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 creating+peering, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:02.248 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:02.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:02.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:02.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:02.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.639 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:02.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:02.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:02.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:03.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:03.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:03.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:03.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:03.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:03.297 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.712848+0000 osd.6 (osd.6) 131 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.712848+0000 osd.6 (osd.6) 131 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.715457+0000 osd.6 (osd.6) 132 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.715457+0000 osd.6 (osd.6) 132 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.891346+0000 osd.4 (osd.4) 155 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.891346+0000 osd.4 (osd.4) 155 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.893691+0000 osd.4 (osd.4) 156 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:01.893691+0000 osd.4 (osd.4) 156 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.035847+0000 osd.3 (osd.3) 199 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:25:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.035847+0000 osd.3 (osd.3) 199 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.039701+0000 osd.3 (osd.3) 200 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.039701+0000 osd.3 (osd.3) 200 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086027+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086027+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086265+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086265+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086403+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086403+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086529+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.712848+0000 osd.6 (osd.6) 131 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.712848+0000 osd.6 (osd.6) 131 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.715457+0000 osd.6 (osd.6) 132 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.715457+0000 osd.6 (osd.6) 132 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.891346+0000 osd.4 (osd.4) 155 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.891346+0000 osd.4 (osd.4) 155 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.893691+0000 osd.4 (osd.4) 156 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:01.893691+0000 osd.4 (osd.4) 156 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.035847+0000 osd.3 (osd.3) 199 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.035847+0000 osd.3 (osd.3) 199 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.039701+0000 osd.3 (osd.3) 200 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.039701+0000 osd.3 (osd.3) 200 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:25:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086027+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086027+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086265+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086265+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086403+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086403+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086529+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.086529+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.099175+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-04-26T16:25:03.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.099175+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.172898+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.172898+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.175240+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.175240+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.206020+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: audit 2024-04-26T16:25:02.206020+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.269472+0000 osd.2 (osd.2) 141 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.269472+0000 osd.2 (osd.2) 141 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.271622+0000 osd.2 (osd.2) 142 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.271622+0000 osd.2 (osd.2) 142 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.645022+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.645022+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.646799+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:25:03.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.646799+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.086529+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.099175+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.099175+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.172898+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.172898+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.175240+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.175240+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.206020+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: audit 2024-04-26T16:25:02.206020+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.269472+0000 osd.2 (osd.2) 141 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.269472+0000 osd.2 (osd.2) 141 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.271622+0000 osd.2 (osd.2) 142 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.271622+0000 osd.2 (osd.2) 142 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.645022+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.645022+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.646799+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:25:03.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:03 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.646799+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:25:03.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.712848+0000 osd.6 (osd.6) 131 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.712848+0000 osd.6 (osd.6) 131 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.715457+0000 osd.6 (osd.6) 132 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.715457+0000 osd.6 (osd.6) 132 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.891346+0000 osd.4 (osd.4) 155 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.891346+0000 osd.4 (osd.4) 155 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.893691+0000 osd.4 (osd.4) 156 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:01.893691+0000 osd.4 (osd.4) 156 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.035847+0000 osd.3 (osd.3) 199 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.035847+0000 osd.3 (osd.3) 199 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.039701+0000 osd.3 (osd.3) 200 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:25:03.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.039701+0000 osd.3 (osd.3) 200 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086027+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086027+0000 mon.a (mon.0) 1676 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi050-73363-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086265+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086265+0000 mon.a (mon.0) 1677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi050-73193-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086403+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086403+0000 mon.a (mon.0) 1678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi050-73432-21"}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086529+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.086529+0000 mon.a (mon.0) 1679 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi050-73201-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.099175+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.099175+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e234: 8 total, 8 up, 8 in 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.172898+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.172898+0000 mon.a (mon.0) 1681 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.175240+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.175240+0000 mon.b (mon.1) 155 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.206020+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: audit 2024-04-26T16:25:02.206020+0000 mon.c (mon.2) 359 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.269472+0000 osd.2 (osd.2) 141 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.269472+0000 osd.2 (osd.2) 141 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.271622+0000 osd.2 (osd.2) 142 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.271622+0000 osd.2 (osd.2) 142 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.645022+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.645022+0000 osd.7 (osd.7) 197 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.646799+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:25:03.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:03 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.646799+0000 osd.7 (osd.7) 198 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:25:03.426 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:03.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:03.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-26T16:25:03.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:03.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:03.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:03.677 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:03.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:03.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:03.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:03.924 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:03.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:04.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:04.050 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:04.109 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.548948+0000 osd.5 (osd.5) 175 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.548948+0000 osd.5 (osd.5) 175 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.550839+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.550839+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.694601+0000 osd.6 (osd.6) 133 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.694601+0000 osd.6 (osd.6) 133 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.696410+0000 osd.6 (osd.6) 134 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:02.696410+0000 osd.6 (osd.6) 134 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.100371+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.100371+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:04.110 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.105049+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.105049+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.110426+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.110426+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.112450+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.112450+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.207223+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.207223+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.439705+0000 mon.c (mon.2) 361 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: audit 2024-04-26T16:25:03.439705+0000 mon.c (mon.2) 361 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:04.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.542354+0000 mgr.y (mgr.24401) 309 : cluster [DBG] pgmap v337: 255 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 16 unknown, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:04.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:04.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.548948+0000 osd.5 (osd.5) 175 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.548948+0000 osd.5 (osd.5) 175 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.550839+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.550839+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.694601+0000 osd.6 (osd.6) 133 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.694601+0000 osd.6 (osd.6) 133 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:25:04.175 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.696410+0000 osd.6 (osd.6) 134 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:02.696410+0000 osd.6 (osd.6) 134 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.100371+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.100371+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.105049+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.105049+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.110426+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.110426+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.112450+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.112450+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.207223+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:04.176 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.207223+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.439705+0000 mon.c (mon.2) 361 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: audit 2024-04-26T16:25:03.439705+0000 mon.c (mon.2) 361 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.542354+0000 mgr.y (mgr.24401) 309 : cluster [DBG] pgmap v337: 255 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 16 unknown, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.542354+0000 mgr.y (mgr.24401) 309 : cluster [DBG] pgmap v337: 255 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 16 unknown, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.654322+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.654322+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.656835+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:25:04.177 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.656835+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:25:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:04.178 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.542354+0000 mgr.y (mgr.24401) 309 : cluster [DBG] pgmap v337: 255 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 16 unknown, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:04.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.654322+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:25:04.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.654322+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:25:04.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.656835+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:25:04.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.656835+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:25:04.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:04.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:04.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.548948+0000 osd.5 (osd.5) 175 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:25:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.548948+0000 osd.5 (osd.5) 175 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:25:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.550839+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:25:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.550839+0000 osd.5 (osd.5) 176 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:25:04.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.694601+0000 osd.6 (osd.6) 133 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.694601+0000 osd.6 (osd.6) 133 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.696410+0000 osd.6 (osd.6) 134 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:02.696410+0000 osd.6 (osd.6) 134 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.100371+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.100371+0000 mon.a (mon.0) 1682 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.105049+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.105049+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e235: 8 total, 8 up, 8 in 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.110426+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.110426+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.112450+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.112450+0000 mon.b (mon.1) 156 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]: dispatch 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.207223+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.207223+0000 mon.c (mon.2) 360 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:04.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.439705+0000 mon.c (mon.2) 361 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: audit 2024-04-26T16:25:03.439705+0000 mon.c (mon.2) 361 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.542354+0000 mgr.y (mgr.24401) 309 : cluster [DBG] pgmap v337: 255 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 16 unknown, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.542354+0000 mgr.y (mgr.24401) 309 : cluster [DBG] pgmap v337: 255 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 16 unknown, 215 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.654322+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.654322+0000 osd.7 (osd.7) 199 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.656835+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:25:04.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:04 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.656835+0000 osd.7 (osd.7) 200 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:25:04.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:04.413 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:25:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:25:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:25:04.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:04.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:04.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:04.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:04.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:04.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:04.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:04.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:04.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:04.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:05.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:05.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:05.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.567418+0000 osd.5 (osd.5) 177 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.567418+0000 osd.5 (osd.5) 177 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.570119+0000 osd.5 (osd.5) 178 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.570119+0000 osd.5 (osd.5) 178 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.720061+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.720061+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.721232+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:05.191 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:03.721232+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:05.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.066975+0000 osd.3 (osd.3) 201 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:25:05.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.066975+0000 osd.3 (osd.3) 201 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:25:05.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.068164+0000 osd.3 (osd.3) 202 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:25:05.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.068164+0000 osd.3 (osd.3) 202 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:25:05.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.100421+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:05.192 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.100421+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.101900+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.101900+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.109015+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.109015+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.114330+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.114330+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-04-26T16:25:05.193 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.121173+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:05.194 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.121173+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:05.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.567418+0000 osd.5 (osd.5) 177 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:25:05.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.567418+0000 osd.5 (osd.5) 177 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:25:05.194 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.570119+0000 osd.5 (osd.5) 178 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:25:05.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.570119+0000 osd.5 (osd.5) 178 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:25:05.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.720061+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:05.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.720061+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:05.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.721232+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:05.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:03.721232+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:05.195 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.066975+0000 osd.3 (osd.3) 201 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.066975+0000 osd.3 (osd.3) 201 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.068164+0000 osd.3 (osd.3) 202 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.068164+0000 osd.3 (osd.3) 202 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.100421+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.100421+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.101900+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.101900+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.109015+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:05.196 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.109015+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.114330+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.114330+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.121173+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.121173+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.140996+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.140996+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.197 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.141379+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.50:0/3408603554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.141379+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.50:0/3408603554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.141512+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.141512+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.142112+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.142112+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.183573+0000 mon.c (mon.2) 364 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.183573+0000 mon.c (mon.2) 364 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:05.198 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.184347+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.184347+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.189667+0000 mon.a (mon.0) 1692 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.189667+0000 mon.a (mon.0) 1692 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.208243+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: audit 2024-04-26T16:25:04.208243+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.249475+0000 osd.2 (osd.2) 145 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.249475+0000 osd.2 (osd.2) 145 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.250592+0000 osd.2 (osd.2) 146 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.250592+0000 osd.2 (osd.2) 146 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.341453+0000 osd.0 (osd.0) 195 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.341453+0000 osd.0 (osd.0) 195 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.342499+0000 osd.0 (osd.0) 196 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:25:05.199 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.342499+0000 osd.0 (osd.0) 196 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.140996+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.140996+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.141379+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.50:0/3408603554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.141379+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.50:0/3408603554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.141512+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.141512+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.142112+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.142112+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.183573+0000 mon.c (mon.2) 364 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.183573+0000 mon.c (mon.2) 364 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:05.200 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.184347+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.184347+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.189667+0000 mon.a (mon.0) 1692 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.189667+0000 mon.a (mon.0) 1692 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.208243+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: audit 2024-04-26T16:25:04.208243+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.249475+0000 osd.2 (osd.2) 145 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.249475+0000 osd.2 (osd.2) 145 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.250592+0000 osd.2 (osd.2) 146 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.250592+0000 osd.2 (osd.2) 146 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.341453+0000 osd.0 (osd.0) 195 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.341453+0000 osd.0 (osd.0) 195 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.342499+0000 osd.0 (osd.0) 196 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:25:05.201 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:05 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.342499+0000 osd.0 (osd.0) 196 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:25:05.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:05.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:05.361 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:05.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:05.477 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.567418+0000 osd.5 (osd.5) 177 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.567418+0000 osd.5 (osd.5) 177 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.570119+0000 osd.5 (osd.5) 178 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.570119+0000 osd.5 (osd.5) 178 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.720061+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.720061+0000 osd.2 (osd.2) 143 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.721232+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:03.721232+0000 osd.2 (osd.2) 144 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.066975+0000 osd.3 (osd.3) 201 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.066975+0000 osd.3 (osd.3) 201 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.068164+0000 osd.3 (osd.3) 202 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.068164+0000 osd.3 (osd.3) 202 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.100421+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:05.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.100421+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.101900+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.101900+0000 mon.a (mon.0) 1686 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.109015+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.109015+0000 mon.a (mon.0) 1687 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-15"}]': finished 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.114330+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.114330+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e236: 8 total, 8 up, 8 in 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.121173+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:05.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.121173+0000 mon.a (mon.0) 1689 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.140996+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.140996+0000 mon.c (mon.2) 362 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.141379+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.50:0/3408603554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.141379+0000 mon.c (mon.2) 363 : audit [INF] from='client.? 172.21.15.50:0/3408603554' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.141512+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.141512+0000 mon.a (mon.0) 1690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.142112+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.142112+0000 mon.a (mon.0) 1691 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.183573+0000 mon.c (mon.2) 364 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.183573+0000 mon.c (mon.2) 364 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.184347+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.184347+0000 mon.c (mon.2) 365 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:25:05.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.189667+0000 mon.a (mon.0) 1692 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.189667+0000 mon.a (mon.0) 1692 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.208243+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: audit 2024-04-26T16:25:04.208243+0000 mon.c (mon.2) 366 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.249475+0000 osd.2 (osd.2) 145 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.249475+0000 osd.2 (osd.2) 145 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.250592+0000 osd.2 (osd.2) 146 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.250592+0000 osd.2 (osd.2) 146 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.341453+0000 osd.0 (osd.0) 195 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.341453+0000 osd.0 (osd.0) 195 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.342499+0000 osd.0 (osd.0) 196 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:25:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:05 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.342499+0000 osd.0 (osd.0) 196 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:25:05.492 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:05.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:05.622 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:05.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:05.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:05.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:05.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:05.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:05.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:06.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:06.061 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2248 ms) 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (17855 ms) 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6026 ms) 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2993 ms) 2024-04-26T16:25:06.062 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2996 ms) 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2965 ms) 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3029 ms) 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2978 ms) 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3011 ms) 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2997 ms) 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-04-26T16:25:06.063 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2992 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3008 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3022 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2975 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3002 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3016 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2979 ms) 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-04-26T16:25:06.064 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3014 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3019 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2980 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3001 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3011 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3026 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2980 ms) 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-04-26T16:25:06.065 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2996 ms) 2024-04-26T16:25:06.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-04-26T16:25:06.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3036 ms) 2024-04-26T16:25:06.066 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioP (68907 ms total) 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (171 ms) 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (171 ms total) 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (54 ms) 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-04-26T16:25:06.069 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (5 ms) 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (59 ms total) 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (47 ms) 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (3 ms) 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (50 ms total) 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (45 ms) 2024-04-26T16:25:06.070 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (3 ms) 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (48 ms total) 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1044 ms) 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1044 ms total) 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (98016 ms total) 2024-04-26T16:25:06.071 INFO:tasks.workunit.client.0.smithi050.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-04-26T16:25:06.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:06.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:06.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:06.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.784666+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:25:06.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.784666+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:25:06.250 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.786855+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:04.786855+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.187748+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.187748+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.187855+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.187855+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.187953+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.187953+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.194314+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.194314+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.196202+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.196202+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.200259+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.200259+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:06.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.200579+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.200579+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.784666+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.784666+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.786855+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:04.786855+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.187748+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.187748+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.252 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.187855+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.187855+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.187953+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.187953+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.194314+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.194314+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.196202+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.196202+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.200259+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.200259+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.200579+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.200579+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.209281+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:05.209281+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:06.253 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.543317+0000 mgr.y (mgr.24401) 310 : cluster [DBG] pgmap v340: 239 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 183 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.209281+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:05.209281+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.543317+0000 mgr.y (mgr.24401) 310 : cluster [DBG] pgmap v340: 239 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 183 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.543317+0000 mgr.y (mgr.24401) 310 : cluster [DBG] pgmap v340: 239 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 183 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.051953+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.051953+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.052049+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.052049+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:06.057859+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: cluster 2024-04-26T16:25:06.057859+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-04-26T16:25:06.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.074676+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.255 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.074676+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.255 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.083921+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.255 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.083921+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.269 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:06.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:06.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:06.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:06.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.095344+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.095344+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.517 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.102881+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.102881+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.103488+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:06.518 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[29234]: audit 2024-04-26T16:25:06.103488+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:06.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.543317+0000 mgr.y (mgr.24401) 310 : cluster [DBG] pgmap v340: 239 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 183 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:06.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.051953+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.518 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.051953+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.052049+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.052049+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:06.057859+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: cluster 2024-04-26T16:25:06.057859+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.074676+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.074676+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.083921+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.083921+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.095344+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.095344+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.102881+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.102881+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.103488+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:06.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:06 smithi050 bash[21463]: audit 2024-04-26T16:25:06.103488+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:06.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:06.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.784666+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:25:06.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.784666+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:25:06.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.786855+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:25:06.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:04.786855+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:25:06.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.187748+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.187748+0000 mon.a (mon.0) 1693 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.187855+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.187855+0000 mon.a (mon.0) 1694 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.187953+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.187953+0000 mon.a (mon.0) 1695 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi050-73201-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.194314+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.194314+0000 mon.c (mon.2) 367 : audit [INF] from='client.? 172.21.15.50:0/3879900745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.196202+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.196202+0000 mon.a (mon.0) 1696 : cluster [DBG] osdmap e237: 8 total, 8 up, 8 in 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.200259+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.200259+0000 mon.a (mon.0) 1697 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]: dispatch 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.200579+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.200579+0000 mon.a (mon.0) 1698 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]: dispatch 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.209281+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:05.209281+0000 mon.c (mon.2) 368 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.543317+0000 mgr.y (mgr.24401) 310 : cluster [DBG] pgmap v340: 239 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 183 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.543317+0000 mgr.y (mgr.24401) 310 : cluster [DBG] pgmap v340: 239 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 183 active+clean; 588 KiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.051953+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.051953+0000 mon.a (mon.0) 1699 : audit [INF] from='client.? 172.21.15.50:0/940646521' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi050-73363-36"}]': finished 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.052049+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.052049+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi050-73193-29"}]': finished 2024-04-26T16:25:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:06.057859+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: cluster 2024-04-26T16:25:06.057859+0000 mon.a (mon.0) 1701 : cluster [DBG] osdmap e238: 8 total, 8 up, 8 in 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.074676+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.074676+0000 mon.b (mon.1) 157 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.083921+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.083921+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.095344+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.095344+0000 mon.a (mon.0) 1703 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.102881+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.102881+0000 mon.a (mon.0) 1704 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.103488+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[23868]: audit 2024-04-26T16:25:06.103488+0000 mon.a (mon.0) 1705 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:06.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:06.669 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:06.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:06.801 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:06.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:06.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:06.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:07.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:07.047 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:25:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:25:07.073 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:07.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:07.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:07.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:07.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.729203+0000 osd.6 (osd.6) 135 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.729203+0000 osd.6 (osd.6) 135 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.731787+0000 osd.6 (osd.6) 136 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: cluster 2024-04-26T16:25:05.731787+0000 osd.6 (osd.6) 136 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:06.210940+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:06.210940+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:07.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.056183+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.056183+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.056505+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.056505+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: cluster 2024-04-26T16:25:07.065481+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: cluster 2024-04-26T16:25:07.065481+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.068772+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.50:0/890108740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.068772+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.50:0/890108740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.074346+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.074346+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.079010+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.079010+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.117442+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.117442+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.119702+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:07 smithi190 bash[23868]: audit 2024-04-26T16:25:07.119702+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:07.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:07.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.729203+0000 osd.6 (osd.6) 135 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:25:07.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.729203+0000 osd.6 (osd.6) 135 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.731787+0000 osd.6 (osd.6) 136 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: cluster 2024-04-26T16:25:05.731787+0000 osd.6 (osd.6) 136 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:06.210940+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:06.210940+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.056183+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.056183+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.056505+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.056505+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: cluster 2024-04-26T16:25:07.065481+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: cluster 2024-04-26T16:25:07.065481+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.068772+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.50:0/890108740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.068772+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.50:0/890108740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.074346+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.074346+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.729203+0000 osd.6 (osd.6) 135 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.729203+0000 osd.6 (osd.6) 135 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.731787+0000 osd.6 (osd.6) 136 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: cluster 2024-04-26T16:25:05.731787+0000 osd.6 (osd.6) 136 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:06.210940+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:06.210940+0000 mon.c (mon.2) 369 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:07.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.056183+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.056183+0000 mon.a (mon.0) 1706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.056505+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.056505+0000 mon.a (mon.0) 1707 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi050-73193-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: cluster 2024-04-26T16:25:07.065481+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: cluster 2024-04-26T16:25:07.065481+0000 mon.a (mon.0) 1708 : cluster [DBG] osdmap e239: 8 total, 8 up, 8 in 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.068772+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.50:0/890108740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.068772+0000 mon.c (mon.2) 370 : audit [INF] from='client.? 172.21.15.50:0/890108740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.074346+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.074346+0000 mon.a (mon.0) 1709 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:07.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.079010+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.079010+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.117442+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.117442+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.119702+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[29234]: audit 2024-04-26T16:25:07.119702+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.079010+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.079010+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.117442+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.117442+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.119702+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:07 smithi050 bash[21463]: audit 2024-04-26T16:25:07.119702+0000 mon.b (mon.1) 158 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:07.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:07.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:07.588 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:07.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:07.720 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:07.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:07.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:07.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:07.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:07.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:07.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:08.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:08.108 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:08.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:08.238 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:08.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:08.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:06.649030+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:25:08.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:06.649030+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:25:08.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:06.651500+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:25:08.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:06.651500+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:25:08.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:06.702511+0000 mgr.y (mgr.24401) 311 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:06.702511+0000 mgr.y (mgr.24401) 311 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:07.211868+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:07.211868+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:07.544052+0000 mgr.y (mgr.24401) 312 : cluster [DBG] pgmap v343: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:07.544052+0000 mgr.y (mgr.24401) 312 : cluster [DBG] pgmap v343: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.059829+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.059829+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.059931+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.059931+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.073878+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.073878+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.079460+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-04-26T16:25:08.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.079460+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-04-26T16:25:08.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.080488+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:08 smithi190 bash[23868]: audit 2024-04-26T16:25:08.080488+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.369 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:08.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:06.649030+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:06.649030+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:06.651500+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:06.651500+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:06.702511+0000 mgr.y (mgr.24401) 311 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:06.702511+0000 mgr.y (mgr.24401) 311 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:07.211868+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:07.211868+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:07.544052+0000 mgr.y (mgr.24401) 312 : cluster [DBG] pgmap v343: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:08.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:07.544052+0000 mgr.y (mgr.24401) 312 : cluster [DBG] pgmap v343: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.059829+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.059829+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.059931+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.059931+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.073878+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.073878+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.079460+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.079460+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.080488+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[29234]: audit 2024-04-26T16:25:08.080488+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:06.649030+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:06.649030+0000 osd.5 (osd.5) 179 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:06.651500+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:06.651500+0000 osd.5 (osd.5) 180 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:06.702511+0000 mgr.y (mgr.24401) 311 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:06.702511+0000 mgr.y (mgr.24401) 311 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:07.211868+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:07.211868+0000 mon.c (mon.2) 371 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:07.544052+0000 mgr.y (mgr.24401) 312 : cluster [DBG] pgmap v343: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:08.486 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:07.544052+0000 mgr.y (mgr.24401) 312 : cluster [DBG] pgmap v343: 271 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 32 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.059829+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.059829+0000 mon.a (mon.0) 1712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi050-73201-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.059931+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.059931+0000 mon.a (mon.0) 1713 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.073878+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.073878+0000 mon.b (mon.1) 159 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.079460+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.079460+0000 mon.a (mon.0) 1714 : cluster [DBG] osdmap e240: 8 total, 8 up, 8 in 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.080488+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:08 smithi050 bash[21463]: audit 2024-04-26T16:25:08.080488+0000 mon.a (mon.0) 1715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:08.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:08.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:08.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:08.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:08.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:08.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:08.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:08.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:08.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:08.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:09.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:09.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:09.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:09.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:09.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:09.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:08.270650+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:08.270650+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.062345+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.062345+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.062424+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.062424+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.067678+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.067678+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.073737+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.073737+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.074676+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:09 smithi190 bash[23868]: audit 2024-04-26T16:25:09.074676+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.535 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:09.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:08.270650+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:09.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:08.270650+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:09.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.062345+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:09.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.062345+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:09.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.062424+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:09.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.062424+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.067678+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.067678+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.073737+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.073737+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.074676+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[21463]: audit 2024-04-26T16:25:09.074676+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:08.270650+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:08.270650+0000 mon.c (mon.2) 372 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.062345+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.062345+0000 mon.a (mon.0) 1716 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi050-73193-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.062424+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.062424+0000 mon.a (mon.0) 1717 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.067678+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.067678+0000 mon.a (mon.0) 1718 : cluster [DBG] osdmap e241: 8 total, 8 up, 8 in 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.073737+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.073737+0000 mon.a (mon.0) 1719 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.074676+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:09 smithi050 bash[29234]: audit 2024-04-26T16:25:09.074676+0000 mon.b (mon.1) 160 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]: dispatch 2024-04-26T16:25:09.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.665 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:09.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.796 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:09.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:09.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:09.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:10.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:10.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:10.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:10.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 -- 172.21.15.50:0/6089065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 msgr2=0x7f969c109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 --2- 172.21.15.50:0/6089065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c109880 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f9698009990 tx=0x7f9698009ca0 comp rx=0 tx=0).stop 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 -- 172.21.15.50:0/6089065 shutdown_connections 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 --2- 172.21.15.50:0/6089065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f969c06bad0 0x7f969c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 --2- 172.21.15.50:0/6089065 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c109880 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 --2- 172.21.15.50:0/6089065 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06b130 0x7f969c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 -- 172.21.15.50:0/6089065 >> 172.21.15.50:0/6089065 conn(0x7f969c0792d0 msgr2=0x7f969c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 -- 172.21.15.50:0/6089065 shutdown_connections 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 -- 172.21.15.50:0/6089065 wait complete. 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 librados: starting msgr at 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.176+0000 7f96a36a3700 1 librados: starting objecter 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 Processor -- start 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- start start 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 librados: setting wanted keys 2024-04-26T16:25:10.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 librados: calling monclient init 2024-04-26T16:25:10.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f969c06b130 0x7f969c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06bad0 0x7f969c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c19fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f969c114f30 con 0x7f969c1094a0 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f969c1042d0 con 0x7f969c06b130 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f969c1150b0 con 0x7f969c06bad0 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c19fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a0c3e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06bad0 0x7f969c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c19fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53876/0 (socket says 172.21.15.50:53876) 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 -- 172.21.15.50:0/2931612134 learned_addr learned my addr 172.21.15.50:0/2931612134 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a143f700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f969c06b130 0x7f969c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 -- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06bad0 msgr2=0x7f969c19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06bad0 0x7f969c19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:10.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 -- 172.21.15.50:0/2931612134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f969c06b130 msgr2=0x7f969c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f969c06b130 0x7f969c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 -- 172.21.15.50:0/2931612134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f969c1a03a0 con 0x7f969c1094a0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a0c3e700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06bad0 0x7f969c19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a1c40700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c19fc60 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f9694007ee0 tx=0x7f969400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96927fc700 10 client.?.objecter ms_handle_connect 0x7f969c1094a0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96927fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9694004bb0 con 0x7f969c1094a0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9694004d50 con 0x7f969c1094a0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f969c1a0690 con 0x7f969c1094a0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 10 client.30150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f969c1a7f10 con 0x7f969c1094a0 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 librados: init done 2024-04-26T16:25:10.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9694005790 con 0x7f969c1094a0 2024-04-26T16:25:10.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.180+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9680005120 con 0x7f969c1094a0 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f969401a020 con 0x7f969c1094a0 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9688071e50 0x7f9688074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96a143f700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9688071e50 0x7f9688074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96a143f700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9688071e50 0x7f9688074310 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f96980098e0 tx=0x7f9698005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(242..242 src has 87..242) ==== 10794+0+0 (secure 0 0 0) 0x7f9694091900 con 0x7f969c1094a0 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 10 client.30150.objecter ms_dispatch 0x7f969c001550 osd_map(242..242 src has 87..242) 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 3 client.30150.objecter handle_osd_map got epochs [242,242] > 0 2024-04-26T16:25:10.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 3 client.30150.objecter handle_osd_map decoding full epoch 242 2024-04-26T16:25:10.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:10.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 20 client.30150.objecter dump_active .. 0 homeless 2024-04-26T16:25:10.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.184+0000 7f96927fc700 10 client.30150.objecter ms_handle_connect 0x7f9688071e50 2024-04-26T16:25:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:10.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.188+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f9694010070 con 0x7f969c1094a0 2024-04-26T16:25:10.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:10.313 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:10.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.689306+0000 osd.5 (osd.5) 181 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.689306+0000 osd.5 (osd.5) 181 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.691949+0000 osd.5 (osd.5) 182 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.691949+0000 osd.5 (osd.5) 182 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.758026+0000 osd.6 (osd.6) 137 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.758026+0000 osd.6 (osd.6) 137 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.760208+0000 osd.6 (osd.6) 138 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.760208+0000 osd.6 (osd.6) 138 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.965280+0000 osd.3 (osd.3) 203 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.965280+0000 osd.3 (osd.3) 203 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.969049+0000 osd.3 (osd.3) 204 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:08.969049+0000 osd.3 (osd.3) 204 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:25:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:09.272511+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:09.272511+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.544853+0000 mgr.y (mgr.24401) 313 : cluster [DBG] pgmap v346: 247 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 8 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.544853+0000 mgr.y (mgr.24401) 313 : cluster [DBG] pgmap v346: 247 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 8 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.720028+0000 osd.0 (osd.0) 197 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.720028+0000 osd.0 (osd.0) 197 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.720157+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.720157+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.063001+0000 mon.a (mon.0) 1720 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.063001+0000 mon.a (mon.0) 1720 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.065039+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]': finished 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.065039+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]': finished 2024-04-26T16:25:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.078311+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.078311+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.086244+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.086244+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.181506+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.181506+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.183580+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[21463]: audit 2024-04-26T16:25:10.183580+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.689306+0000 osd.5 (osd.5) 181 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.689306+0000 osd.5 (osd.5) 181 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.691949+0000 osd.5 (osd.5) 182 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.691949+0000 osd.5 (osd.5) 182 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.758026+0000 osd.6 (osd.6) 137 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.758026+0000 osd.6 (osd.6) 137 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.760208+0000 osd.6 (osd.6) 138 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.760208+0000 osd.6 (osd.6) 138 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.965280+0000 osd.3 (osd.3) 203 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.965280+0000 osd.3 (osd.3) 203 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.969049+0000 osd.3 (osd.3) 204 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:25:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:08.969049+0000 osd.3 (osd.3) 204 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:09.272511+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:09.272511+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.544853+0000 mgr.y (mgr.24401) 313 : cluster [DBG] pgmap v346: 247 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 8 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.544853+0000 mgr.y (mgr.24401) 313 : cluster [DBG] pgmap v346: 247 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 8 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.720028+0000 osd.0 (osd.0) 197 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.720028+0000 osd.0 (osd.0) 197 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.720157+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.720157+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.063001+0000 mon.a (mon.0) 1720 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.063001+0000 mon.a (mon.0) 1720 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.065039+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]': finished 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.065039+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]': finished 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.078311+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.078311+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.086244+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.086244+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.181506+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.181506+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.183580+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:10 smithi050 bash[29234]: audit 2024-04-26T16:25:10.183580+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:10.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:10.424+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f96800056e0 con 0x7f969c1094a0 2024-04-26T16:25:10.446 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:10.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.689306+0000 osd.5 (osd.5) 181 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.689306+0000 osd.5 (osd.5) 181 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.691949+0000 osd.5 (osd.5) 182 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.691949+0000 osd.5 (osd.5) 182 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.758026+0000 osd.6 (osd.6) 137 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.758026+0000 osd.6 (osd.6) 137 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.760208+0000 osd.6 (osd.6) 138 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.760208+0000 osd.6 (osd.6) 138 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.965280+0000 osd.3 (osd.3) 203 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:25:10.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.965280+0000 osd.3 (osd.3) 203 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.969049+0000 osd.3 (osd.3) 204 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:08.969049+0000 osd.3 (osd.3) 204 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:09.272511+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:09.272511+0000 mon.c (mon.2) 373 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.544853+0000 mgr.y (mgr.24401) 313 : cluster [DBG] pgmap v346: 247 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 8 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.544853+0000 mgr.y (mgr.24401) 313 : cluster [DBG] pgmap v346: 247 pgs: 11 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 8 unknown, 32 creating+peering, 183 active+clean; 587 KiB data, 558 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.720028+0000 osd.0 (osd.0) 197 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.720028+0000 osd.0 (osd.0) 197 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.720157+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:10.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.720157+0000 osd.0 (osd.0) 198 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.063001+0000 mon.a (mon.0) 1720 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.063001+0000 mon.a (mon.0) 1720 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.065039+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]': finished 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.065039+0000 mon.a (mon.0) 1721 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-17", "mode": "writeback"}]': finished 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.078311+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.078311+0000 mon.a (mon.0) 1722 : cluster [DBG] osdmap e242: 8 total, 8 up, 8 in 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.086244+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:10.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.086244+0000 mon.a (mon.0) 1723 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.181506+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.181506+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.183580+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:10 smithi190 bash[23868]: audit 2024-04-26T16:25:10.183580+0000 mon.b (mon.1) 161 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:10.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:10.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:10.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:10.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:10.838 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:10.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:10.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7fb34c195700 1 -- 172.21.15.50:0/1406213296 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344108390 msgr2=0x7fb344108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7fb34c195700 1 --2- 172.21.15.50:0/1406213296 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344108390 0x7fb344108770 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7fb3400098f0 tx=0x7fb340009c00 comp rx=0 tx=0).stop 2024-04-26T16:25:11.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7f96927fc700 1 -- 172.21.15.50:0/2931612134 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v243) ==== 87+0+0 (secure 0 0 0) 0x7f9694005000 con 0x7f969c1094a0 2024-04-26T16:25:11.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7fb34c195700 1 -- 172.21.15.50:0/1406213296 shutdown_connections 2024-04-26T16:25:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7fb34c195700 1 --2- 172.21.15.50:0/1406213296 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb344102ce0 0x7fb34410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7fb34c195700 1 --2- 172.21.15.50:0/1406213296 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344108390 0x7fb344108770 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.064+0000 7fb34c195700 1 --2- 172.21.15.50:0/1406213296 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb344102320 0x7fb3441027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- 172.21.15.50:0/1406213296 >> 172.21.15.50:0/1406213296 conn(0x7fb3440792d0 msgr2=0x7fb3440796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- 172.21.15.50:0/1406213296 shutdown_connections 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- 172.21.15.50:0/1406213296 wait complete. 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 10 librados: watch_flush enter 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 10 librados: watch_flush exit 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 20 client.30150.objecter shutdown clearing up homeless session... 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 10 client.30150.objecter successfully canceled tick 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9688071e50 msgr2=0x7f9688074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9688071e50 0x7f9688074310 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7f96980098e0 tx=0x7f9698005960 comp rx=0 tx=0).stop 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 msgr2=0x7f969c19fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7f96a36a3700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c19fc60 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7f9694007ee0 tx=0x7f969400d3b0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 librados: starting msgr at 2024-04-26T16:25:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 librados: starting objecter 2024-04-26T16:25:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 Processor -- start 2024-04-26T16:25:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- start start 2024-04-26T16:25:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 librados: setting wanted keys 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 librados: calling monclient init 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb344102320 0x7fb344197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 0x7fb344198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb344108390 0x7fb34419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb3441119a0 con 0x7fb344102320 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb344111820 con 0x7fb344108390 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.068+0000 7fb34c195700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb344111b20 con 0x7fb344102ce0 2024-04-26T16:25:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 0x7fb344198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 0x7fb344198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54180/0 (socket says 172.21.15.50:54180) 2024-04-26T16:25:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 -- 172.21.15.50:0/2561540685 learned_addr learned my addr 172.21.15.50:0/2561540685 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb344108390 msgr2=0x7fb34419ce10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T16:25:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb344108390 0x7fb34419ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb344102320 msgr2=0x7fb344197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb344102320 0x7fb344197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34419d350 con 0x7fb344102ce0 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 0x7fb344198300 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fb33400d980 tx=0x7fb33400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb33affd700 10 client.?.objecter ms_handle_connect 0x7fb344102ce0 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7fb33affd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 shutdown_connections 2024-04-26T16:25:11.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7f96a36a3700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9688071e50 0x7f9688074310 unknown :-1 s=CLOSED pgs=3668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7f96a36a3700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f969c06b130 0x7f969c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7f96a36a3700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f969c1094a0 0x7f969c19fc60 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7f96a36a3700 1 --2- 172.21.15.50:0/2931612134 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f969c06bad0 0x7f969c19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.072+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 >> 172.21.15.50:0/2931612134 conn(0x7f969c0792d0 msgr2=0x7f969c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334017070 con 0x7fb344102ce0 2024-04-26T16:25:11.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb334004830 con 0x7fb344102ce0 2024-04-26T16:25:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34419d640 con 0x7fb344102ce0 2024-04-26T16:25:11.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb34c195700 10 client.40763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb34419db00 con 0x7fb344102ce0 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb34c195700 1 librados: init done 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334005280 con 0x7fb344102ce0 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 shutdown_connections 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7f96a36a3700 1 -- 172.21.15.50:0/2931612134 wait complete. 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.080+0000 7f96a36a3700 1 librados: shutdown 2024-04-26T16:25:11.078 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:25:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.088+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb334005420 con 0x7fb344102ce0 2024-04-26T16:25:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb33affd700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb330071e50 0x7fb330074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fb344103a40 con 0x7fb344102ce0 2024-04-26T16:25:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(243..243 src has 87..243) ==== 10398+0+0 (secure 0 0 0) 0x7fb334091ac0 con 0x7fb344102ce0 2024-04-26T16:25:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 osd_map(243..243 src has 87..243) 2024-04-26T16:25:11.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb33affd700 3 client.40763.objecter handle_osd_map got epochs [243,243] > 0 2024-04-26T16:25:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb33affd700 3 client.40763.objecter handle_osd_map decoding full epoch 243 2024-04-26T16:25:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb33affd700 20 client.40763.objecter dump_active .. 0 homeless 2024-04-26T16:25:11.091 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.092+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb330071e50 0x7fb330074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.096+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb330071e50 0x7fb330074310 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7fb340009680 tx=0x7fb340005b40 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.096+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb330071e50 2024-04-26T16:25:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.096+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v243) ==== 54+0+16 (secure 0 0 0) 0x7fb33405d120 con 0x7fb344102ce0 2024-04-26T16:25:11.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.096+0000 7fb34c195700 10 client.40763.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 1 2024-04-26T16:25:11.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb344064280 0x7fb344109010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 1 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 0 1 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 4 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 10 client.40763.objecter _send_command 1 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb34406b700 con 0x7fb344064280 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34a732700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb344064280 0x7fb344109010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb34a732700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb344064280 0x7fb344109010 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.100+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb344064280 2024-04-26T16:25:11.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.104+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb34406b700 con 0x7fb344064280 2024-04-26T16:25:11.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.104+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 1: 0 ) 2024-04-26T16:25:11.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.104+0000 7fb33affd700 10 client.40763.objecter _finish_command 1 = system:0 2024-04-26T16:25:11.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.104+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 4 2024-04-26T16:25:11.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.104+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 0 1 2024-04-26T16:25:11.106 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 10 client.40763.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 2 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 2 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb34406a110 osd=0 3 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 0 2 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 4 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 10 client.40763.objecter _send_command 2 2024-04-26T16:25:11.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.128+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb344109550 con 0x7fb344064280 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb344109550 con 0x7fb344064280 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 10 client.40763.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 4 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 0 2 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 10 client.40763.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 3 2024-04-26T16:25:11.129 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:25:11.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fb344063e70 0x7fb3441a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 3 2024-04-26T16:25:11.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 1 3 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 4 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 10 client.40763.objecter _send_command 3 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fb34402dff0 con 0x7fb344063e70 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fb344063e70 0x7fb3441a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fb344063e70 0x7fb3441a1dd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb344063e70 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb34402dff0 con 0x7fb344063e70 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 3: 0 ) 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 10 client.40763.objecter _finish_command 3 = system:0 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 4 2024-04-26T16:25:11.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.132+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 1 3 2024-04-26T16:25:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 -- 172.21.15.50:0/3266646811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c103e30 msgr2=0x7fc29c1102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 --2- 172.21.15.50:0/3266646811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c103e30 0x7fc29c1102c0 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fc29400b0a0 tx=0x7fc29400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 -- 172.21.15.50:0/3266646811 shutdown_connections 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 --2- 172.21.15.50:0/3266646811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc29c1094f0 0x7fc29c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 --2- 172.21.15.50:0/3266646811 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c103e30 0x7fc29c1102c0 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 --2- 172.21.15.50:0/3266646811 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 0x7fc29c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 -- 172.21.15.50:0/3266646811 >> 172.21.15.50:0/3266646811 conn(0x7fc29c07a5d0 msgr2=0x7fc29c07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 -- 172.21.15.50:0/3266646811 shutdown_connections 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 -- 172.21.15.50:0/3266646811 wait complete. 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 librados: starting msgr at 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.140+0000 7fc2a1db6700 1 librados: starting objecter 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 Processor -- start 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 -- start start 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 librados: setting wanted keys 2024-04-26T16:25:11.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 librados: calling monclient init 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 0x7fc29c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc29c103e30 0x7fc29c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 0x7fc29c19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c112ad0 con 0x7fc29c1094f0 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc29c112950 con 0x7fc29c103e30 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fc29c112c50 con 0x7fc29c103470 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 0x7fc29c19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 0x7fc29c19d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53896/0 (socket says 172.21.15.50:53896) 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29bfff700 1 -- 172.21.15.50:0/2219097201 learned_addr learned my addr 172.21.15.50:0/2219097201 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:11.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 0x7fc29c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29affd700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc29c103e30 0x7fc29c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 -- 172.21.15.50:0/2219097201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc29c103e30 msgr2=0x7fc29c199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc29c103e30 0x7fc29c199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 -- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 msgr2=0x7fc29c19d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 0x7fc29c19d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 -- 172.21.15.50:0/2219097201 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc29c19e120 con 0x7fc29c103470 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29bfff700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 0x7fc29c19d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc29b7fe700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 0x7fc29c199110 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00e9f0 tx=0x7fc28c00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc298ff9700 10 client.?.objecter ms_handle_connect 0x7fc29c103470 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc298ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c00c9c0 con 0x7fc29c103470 2024-04-26T16:25:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fc28c010430 con 0x7fc29c103470 2024-04-26T16:25:11.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc28c018680 con 0x7fc29c103470 2024-04-26T16:25:11.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 -- 172.21.15.50:0/2219097201 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc29c19e410 con 0x7fc29c103470 2024-04-26T16:25:11.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 10 client.40769.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:11.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 -- 172.21.15.50:0/2219097201 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc29c1a5c90 con 0x7fc29c103470 2024-04-26T16:25:11.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2a1db6700 1 librados: init done 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc280005120 con 0x7fc29c103470 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.144+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fc28c00cb60 con 0x7fc29c103470 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc288071f80 0x7fc288074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc29affd700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc288071f80 0x7fc288074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(243..243 src has 87..243) ==== 10398+0+0 (secure 0 0 0) 0x7fc28c091f50 con 0x7fc29c103470 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 10 client.40769.objecter ms_dispatch 0x7fc29c001550 osd_map(243..243 src has 87..243) 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 3 client.40769.objecter handle_osd_map got epochs [243,243] > 0 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 3 client.40769.objecter handle_osd_map decoding full epoch 243 2024-04-26T16:25:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 20 client.40769.objecter dump_active .. 0 homeless 2024-04-26T16:25:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc29affd700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc288071f80 0x7fc288074440 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7fc284009680 tx=0x7fc284009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.148+0000 7fc298ff9700 10 client.40769.objecter ms_handle_connect 0x7fc288071f80 2024-04-26T16:25:11.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fc28c016330 con 0x7fc29c103470 2024-04-26T16:25:11.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 10 client.40763.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 4 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 4 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb344063d00 osd=1 3 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 1 4 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 4 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 10 client.40763.objecter _send_command 4 2024-04-26T16:25:11.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.152+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441a2310 con 0x7fb344063e70 2024-04-26T16:25:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441a2310 con 0x7fb344063e70 2024-04-26T16:25:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 10 client.40763.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 4 2024-04-26T16:25:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 1 4 2024-04-26T16:25:11.154 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:25:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 10 client.40763.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 5 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fb34402d6d0 0x7fb3441a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 5 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 2 5 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 4 2024-04-26T16:25:11.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 10 client.40763.objecter _send_command 5 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fb34410a3e0 con 0x7fb34402d6d0 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fb34402d6d0 0x7fb3441a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb34402d6d0 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fb34402d6d0 0x7fb3441a56b0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb34410a3e0 con 0x7fb34402d6d0 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 5: 0 ) 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 10 client.40763.objecter _finish_command 5 = system:0 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 4 2024-04-26T16:25:11.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.156+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 2 5 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 10 client.40763.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 6 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 6 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb34402d560 osd=2 3 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 2 6 2024-04-26T16:25:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 4 2024-04-26T16:25:11.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 10 client.40763.objecter _send_command 6 2024-04-26T16:25:11.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.176+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441a5bf0 con 0x7fb34402d6d0 2024-04-26T16:25:11.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441a5bf0 con 0x7fb34402d6d0 2024-04-26T16:25:11.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 10 client.40763.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 4 2024-04-26T16:25:11.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 2 6 2024-04-26T16:25:11.176 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 10 client.40763.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 7 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fb344109ac0 0x7fb3441bfc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 7 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 3 7 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 4 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 10 client.40763.objecter _send_command 7 2024-04-26T16:25:11.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fb3441c2aa0 con 0x7fb344109ac0 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34a732700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fb344109ac0 0x7fb3441bfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb344109ac0 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb34a732700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fb344109ac0 0x7fb3441bfc40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3441c2aa0 con 0x7fb344109ac0 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 7: 0 ) 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 10 client.40763.objecter _finish_command 7 = system:0 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 4 2024-04-26T16:25:11.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.180+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 3 7 2024-04-26T16:25:11.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 10 client.40763.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 8 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 8 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb344109950 osd=3 3 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 3 8 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 4 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 10 client.40763.objecter _send_command 8 2024-04-26T16:25:11.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441a6050 con 0x7fb344109ac0 2024-04-26T16:25:11.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441a6050 con 0x7fb344109ac0 2024-04-26T16:25:11.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb33affd700 10 client.40763.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 4 2024-04-26T16:25:11.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 3 8 2024-04-26T16:25:11.199 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:25:11.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 10 client.40763.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 9 2024-04-26T16:25:11.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb3441a31a0 0x7fb3441c3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 9 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 4 9 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 4 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 10 client.40763.objecter _send_command 9 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fb3441c6270 con 0x7fb3441a31a0 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb3441a31a0 0x7fb3441c3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb3441a31a0 2024-04-26T16:25:11.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.200+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb3441a31a0 0x7fb3441c3310 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.204+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3441c6270 con 0x7fb3441a31a0 2024-04-26T16:25:11.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.204+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 9: 0 ) 2024-04-26T16:25:11.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.204+0000 7fb33affd700 10 client.40763.objecter _finish_command 9 = system:0 2024-04-26T16:25:11.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.204+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 4 2024-04-26T16:25:11.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.204+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 4 9 2024-04-26T16:25:11.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:11.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 10 client.40763.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 10 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 10 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441a6050 osd=4 3 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 4 10 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 4 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 10 client.40763.objecter _send_command 10 2024-04-26T16:25:11.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.224+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441c6270 con 0x7fb3441a31a0 2024-04-26T16:25:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441c6270 con 0x7fb3441a31a0 2024-04-26T16:25:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb33affd700 10 client.40763.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 4 2024-04-26T16:25:11.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 4 10 2024-04-26T16:25:11.226 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:25:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 10 client.40763.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 11 2024-04-26T16:25:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fb3441c5600 0x7fb3441c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 11 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 5 11 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 4 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 10 client.40763.objecter _send_command 11 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fb3441c9610 con 0x7fb3441c5600 2024-04-26T16:25:11.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fb3441c5600 0x7fb3441c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb349730700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fb3441c5600 0x7fb3441c5a60 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.228+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb3441c5600 2024-04-26T16:25:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.232+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3441c9610 con 0x7fb3441c5600 2024-04-26T16:25:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.232+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 11: 0 ) 2024-04-26T16:25:11.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.232+0000 7fb33affd700 10 client.40763.objecter _finish_command 11 = system:0 2024-04-26T16:25:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.232+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 4 2024-04-26T16:25:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.232+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 5 11 2024-04-26T16:25:11.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:11.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 10 client.40763.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 12 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 12 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441c6270 osd=5 3 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 5 12 2024-04-26T16:25:11.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 4 2024-04-26T16:25:11.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 10 client.40763.objecter _send_command 12 2024-04-26T16:25:11.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.248+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441c3900 con 0x7fb3441c5600 2024-04-26T16:25:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441c3900 con 0x7fb3441c5600 2024-04-26T16:25:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb33affd700 10 client.40763.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 4 2024-04-26T16:25:11.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 5 12 2024-04-26T16:25:11.251 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:25:11.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 10 client.40763.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 13 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fb3441c3fd0 0x7fb3441c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 13 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 6 13 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 4 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 10 client.40763.objecter _send_command 13 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fb3441c4b40 con 0x7fb3441c3fd0 2024-04-26T16:25:11.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.252+0000 7fb34a732700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fb3441c3fd0 0x7fb3441c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb3441c3fd0 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb34a732700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fb3441c3fd0 0x7fb3441c4410 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3441c4b40 con 0x7fb3441c3fd0 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 13: 0 ) 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb33affd700 10 client.40763.objecter _finish_command 13 = system:0 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 4 2024-04-26T16:25:11.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.256+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 6 13 2024-04-26T16:25:11.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 10 client.40763.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 14 2024-04-26T16:25:11.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 14 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441c3900 osd=6 3 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 6 14 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 4 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 10 client.40763.objecter _send_command 14 2024-04-26T16:25:11.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.272+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441c4b40 con 0x7fb3441c3fd0 2024-04-26T16:25:11.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441c4b40 con 0x7fb3441c3fd0 2024-04-26T16:25:11.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 10 client.40763.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 4 2024-04-26T16:25:11.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 6 14 2024-04-26T16:25:11.276 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 10 client.40763.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 15 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fb3441c4cb0 0x7fb3441cca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 15 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 7 15 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 4 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 10 client.40763.objecter _send_command 15 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fb3441cfb30 con 0x7fb3441c4cb0 2024-04-26T16:25:11.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fb3441c4cb0 0x7fb3441cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:11.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 10 client.40763.objecter ms_handle_connect 0x7fb3441c4cb0 2024-04-26T16:25:11.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb349f31700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fb3441c4cb0 0x7fb3441cca70 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:11.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3441cfb30 con 0x7fb3441c4cb0 2024-04-26T16:25:11.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 15: 0 ) 2024-04-26T16:25:11.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 10 client.40763.objecter _finish_command 15 = system:0 2024-04-26T16:25:11.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 4 2024-04-26T16:25:11.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.280+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 7 15 2024-04-26T16:25:11.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 10 client.40763.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign -1 16 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 20 client.40763.objecter _get_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 15 client.40763.objecter _session_command_op_remove -1 16 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 20 client.40763.objecter get_session s=0x7fb3441c4b40 osd=7 3 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 15 client.40763.objecter _session_command_op_assign 7 16 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 4 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 10 client.40763.objecter _send_command 16 2024-04-26T16:25:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.300+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fb3441ce8c0 con 0x7fb3441c4cb0 2024-04-26T16:25:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb33affd700 1 -- 172.21.15.50:0/2561540685 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fb3441ce8c0 con 0x7fb3441c4cb0 2024-04-26T16:25:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb33affd700 10 client.40763.objecter ms_dispatch 0x7fb344001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb33affd700 10 client.40763.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb33affd700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 4 2024-04-26T16:25:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb33affd700 15 client.40763.objecter _session_command_op_remove 7 16 2024-04-26T16:25:11.300 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:25:11.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 librados: watch_flush enter 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 librados: watch_flush exit 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.0 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb344064280 msgr2=0x7fb344109010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fb344064280 0x7fb344109010 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34406a110 osd=0 1 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.1 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fb344063e70 msgr2=0x7fb3441a1dd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fb344063e70 0x7fb3441a1dd0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344063d00 osd=1 1 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.2 2024-04-26T16:25:11.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fb34402d6d0 msgr2=0x7fb3441a56b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fb34402d6d0 0x7fb3441a56b0 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb34402d560 osd=2 1 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34a732700 1 -- 172.21.15.50:0/2561540685 reap_dead start 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.3 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fb344109ac0 msgr2=0x7fb3441bfc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fb344109ac0 0x7fb3441bfc40 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb344109950 osd=3 1 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.4 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb3441a31a0 msgr2=0x7fb3441c3310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fb3441a31a0 0x7fb3441c3310 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441a6050 osd=4 1 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.5 2024-04-26T16:25:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fb3441c5600 msgr2=0x7fb3441c5a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fb3441c5600 0x7fb3441c5a60 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c6270 osd=5 1 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.6 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fb3441c3fd0 msgr2=0x7fb3441c4410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fb3441c3fd0 0x7fb3441c4410 crc :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c3900 osd=6 1 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter close_session for osd.7 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fb3441c4cb0 msgr2=0x7fb3441cca70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fb3441c4cb0 0x7fb3441cca70 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter put_session s=0x7fb3441c4b40 osd=7 1 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 20 client.40763.objecter shutdown clearing up homeless session... 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 10 client.40763.objecter successfully canceled tick 2024-04-26T16:25:11.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34a732700 1 -- 172.21.15.50:0/2561540685 reap_dead start 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb330071e50 msgr2=0x7fb330074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb330071e50 0x7fb330074310 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7fb340009680 tx=0x7fb340005b40 comp rx=0 tx=0).stop 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 msgr2=0x7fb344198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 0x7fb344198300 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fb33400d980 tx=0x7fb33400dd40 comp rx=0 tx=0).stop 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 shutdown_connections 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb344102ce0 0x7fb344198300 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 --2- 172.21.15.50:0/2561540685 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb330071e50 0x7fb330074310 unknown :-1 s=CLOSED pgs=3669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 >> 172.21.15.50:0/2561540685 conn(0x7fb3440792d0 msgr2=0x7fb344100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 shutdown_connections 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 -- 172.21.15.50:0/2561540685 wait complete. 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.304+0000 7fb34c195700 1 librados: shutdown 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:25:11.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:25:11.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:25:11.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:25:11.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:25:11.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:25:11.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:25:11.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:11.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.710926+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.a scrub starts 2024-04-26T16:25:11.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.710926+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.a scrub starts 2024-04-26T16:25:11.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.712317+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.a scrub ok 2024-04-26T16:25:11.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.712317+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.a scrub ok 2024-04-26T16:25:11.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.774982+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:25:11.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.774982+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.777350+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:09.777350+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.200538+0000 osd.1 (osd.1) 171 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.200538+0000 osd.1 (osd.1) 171 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.202038+0000 osd.1 (osd.1) 172 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.202038+0000 osd.1 (osd.1) 172 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:10.274154+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:10.274154+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:10.429967+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:10.429967+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.051158+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.051158+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:11.346 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.065470+0000 mon.a (mon.0) 1727 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.065470+0000 mon.a (mon.0) 1727 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.067403+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.067403+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.067466+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.067466+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.067529+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.067529+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.082164+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.082164+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.084227+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.084227+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-04-26T16:25:11.347 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.085702+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.085702+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.089132+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:11.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.089132+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:11.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.097628+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.50:0/2561540685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:11.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:11 smithi190 bash[23868]: audit 2024-04-26T16:25:11.097628+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.50:0/2561540685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:11.360 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.392+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7fc280005410 con 0x7fc29c103470 2024-04-26T16:25:11.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.396+0000 7fc298ff9700 1 -- 172.21.15.50:0/2219097201 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v243) ==== 81+0+64032 (secure 0 0 0) 0x7fc28c05d590 con 0x7fc29c103470 2024-04-26T16:25:11.395 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:25:11.395 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:25:11.395 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 243, 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-26T16:07:04.494213+0000", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-26T16:25:11.064943+0000", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-26T16:16:46.862501+0000", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-26T16:16:20.945042+0000", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5801984, 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "noscrub", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-26T16:25:11.396 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 49, 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 225, 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-26T16:25:11.397 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:12:30.809607+0000", 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.398 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-26T16:25:11.399 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.400 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.401 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.402 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-26T16:25:11.403 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 7.8899998664855957, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 7.8899998664855957, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.37999999523162842, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.404 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:27.029484+0000", 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.405 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.406 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "80", 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.407 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.408 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.409 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.410 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.411 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:29.124965+0000", 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.412 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.413 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "82", 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.414 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.415 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.416 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:25:11.417 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.418 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:30.749319+0000", 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.419 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.420 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "89", 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 89, 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.421 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.422 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.423 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 2.6500000953674316, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 2.6500000953674316, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2.3299999237060547, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2.3299999237060547, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.424 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:31.064350+0000", 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.425 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.426 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "84", 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.427 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.428 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.429 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.25, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.25, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.25, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.25, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.430 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:33.192127+0000", 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.431 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.432 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "86", 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.433 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.434 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.435 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.436 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.75, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.75, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.437 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 11, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:23:28.050102+0000", 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.438 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.439 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "142", 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.440 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.441 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.442 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": {}, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.75, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.75, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.443 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 12, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "LibRadosList_smithi050-73230-1", 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:23:28.061478+0000", 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.444 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 11, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 11, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 11, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 11, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 11, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "12.b", 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 232, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 231, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 231, 2024-04-26T16:25:11.445 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "217'14", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "217'13" 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "232", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "232", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "230", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.446 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.447 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.448 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 2.1700000762939453, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 2.1700000762939453, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.9099999666213989, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.9099999666213989, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.449 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 109, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "test-rados-api-smithi050-73372-6", 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:23:42.172881+0000", 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.450 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.451 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "243", 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "243", 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "243", 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "243", 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 7, 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 240, 2024-04-26T16:25:11.452 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [ 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: 222 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.453 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.454 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-26T16:25:11.455 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 222, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "test-rados-api-smithi050-73372-17", 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:25:05.228117+0000", 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8201, 2024-04-26T16:25:11.456 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,incomplete_clones,selfmanaged_snaps", 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.457 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "243", 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "243", 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "243", 2024-04-26T16:25:11.458 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "243", 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 7, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 240, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": 109, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.459 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "writeback", 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.460 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.461 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.5, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.5, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 225, 2024-04-26T16:25:11.462 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "RoundTripCmpExtPP_smithi050-73201-31", 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:25:09.109423+0000", 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 32769, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,creating", 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:25:11.463 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:25:11.464 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "243", 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:25:11.465 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:25:11.466 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:25:11.467 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:25:11.468 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.75, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.75, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 1, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-26T16:25:11.469 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1", 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-26T16:25:11.470 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.471 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.472 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.473 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.474 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/310950989", 2024-04-26T16:25:11.475 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/310950989", 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/310950989", 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/310950989", 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed", 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.476 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.477 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-26T16:25:11.478 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-26T16:25:11.479 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.480 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/3516275849", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/3516275849", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/3516275849", 2024-04-26T16:25:11.481 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/3516275849", 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2", 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.482 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 18, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.483 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.484 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.485 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.486 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/2835380668", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/2835380668", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/2835380668", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/2835380668", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.487 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03", 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.488 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.489 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.490 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.491 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/2541695077", 2024-04-26T16:25:11.492 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/2541695077", 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/2541695077", 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/2541695077", 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "f7220491-e9bf-4188-b203-322e9908558c", 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.493 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6800", 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.494 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6801", 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6802", 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.495 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6803", 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6806", 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.496 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6807", 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6804", 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.497 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6805", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6801/2207965503", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6803/2207965503", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6807/2207965503", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6805/2207965503", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.498 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587", 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 47, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.499 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6808", 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6809", 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.500 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6810", 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6811", 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.501 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6814", 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6815", 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.502 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6812", 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6813", 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.503 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6809/3373187844", 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6811/3373187844", 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6815/3373187844", 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6813/3373187844", 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "94111b7d-7887-4a3a-983b-394294735d0e", 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.504 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6816", 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.505 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6817", 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6818", 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.506 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6819", 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6822", 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.507 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6823", 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6820", 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.508 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6821", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6817/2399532574", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6819/2399532574", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6823/2399532574", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6821/2399532574", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.509 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5", 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 242, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:25:11.510 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6824", 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6825", 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.511 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6826", 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6827", 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.512 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6830", 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6831", 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:25:11.513 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6828", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6829", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6825/2856617796", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6827/2856617796", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6831/2856617796", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6829/2856617796", 2024-04-26T16:25:11.514 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.515 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:25:09.720161+0000", 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:24:09.596090+0000", 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.516 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:25:03.721238+0000", 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.517 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:24:37.707040+0000", 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:24:03.303873+0000", 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.518 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:24:08.260991+0000", 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.519 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:24:27.487359+0000", 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:24:22.208545+0000", 2024-04-26T16:25:11.520 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [], 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.9", 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.521 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2324264713": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/542146552": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/127073210": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/127073210": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/126446279": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/126446279": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/3454723725": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:25:11.522 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2548717623": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/685732569": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2903276422": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/685732569": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2979962403": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/71201882": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/3973398204": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1172169336": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/3444006067": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/577039843": "2024-04-27T16:18:25.317826+0000" 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-26T16:25:11.523 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "testprofile-RoundTrip2_smithi050-73193-30": { 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-device-class": "", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-num-failure-domains": "0", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-osds-per-failure-domain": "0", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-root": "default", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-26T16:25:11.524 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van", 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "w": "8" 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [ 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 16, 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.525 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 20, 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 40, 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.526 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 72, 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.527 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 99, 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 4, 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 7, 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 108, 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.528 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 109, 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 5 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.529 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 128, 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 143, 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.530 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 150, 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 3 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.531 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 157, 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 158, 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.532 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 168, 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 3 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.533 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 173, 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 176, 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:25:11.534 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 179, 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 189, 2024-04-26T16:25:11.535 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 195, 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 3 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 207, 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 6 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 215, 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 7, 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:25:11.537 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-26T16:25:11.538 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:11.539 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:25:11.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:11.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 10 librados: watch_flush enter 2024-04-26T16:25:11.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 10 librados: watch_flush exit 2024-04-26T16:25:11.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 20 client.40769.objecter shutdown clearing up homeless session... 2024-04-26T16:25:11.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 10 client.40769.objecter successfully canceled tick 2024-04-26T16:25:11.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc288071f80 msgr2=0x7fc288074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc288071f80 0x7fc288074440 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7fc284009680 tx=0x7fc284009260 comp rx=0 tx=0).stop 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 msgr2=0x7fc29c199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 0x7fc29c199110 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00e9f0 tx=0x7fc28c00ed00 comp rx=0 tx=0).stop 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 shutdown_connections 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc288071f80 0x7fc288074440 unknown :-1 s=CLOSED pgs=3670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc29c1094f0 0x7fc29c19d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc29c103470 0x7fc29c199110 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 --2- 172.21.15.50:0/2219097201 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc29c103e30 0x7fc29c199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 >> 172.21.15.50:0/2219097201 conn(0x7fc29c07a5d0 msgr2=0x7fc29c101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 shutdown_connections 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 -- 172.21.15.50:0/2219097201 wait complete. 2024-04-26T16:25:11.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:11.400+0000 7fc2927fc700 1 librados: shutdown 2024-04-26T16:25:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:11.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.710926+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.a scrub starts 2024-04-26T16:25:11.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.710926+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.a scrub starts 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.712317+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.a scrub ok 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.712317+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.a scrub ok 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.774982+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.774982+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.777350+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:09.777350+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.200538+0000 osd.1 (osd.1) 171 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.200538+0000 osd.1 (osd.1) 171 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.202038+0000 osd.1 (osd.1) 172 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.202038+0000 osd.1 (osd.1) 172 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:10.274154+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:10.274154+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:11.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:10.429967+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:10.429967+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.051158+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.051158+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.065470+0000 mon.a (mon.0) 1727 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.065470+0000 mon.a (mon.0) 1727 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.067403+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.067403+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.710926+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.a scrub starts 2024-04-26T16:25:11.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.710926+0000 osd.5 (osd.5) 183 : cluster [DBG] 6.a scrub starts 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.712317+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.a scrub ok 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.712317+0000 osd.5 (osd.5) 184 : cluster [DBG] 6.a scrub ok 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.774982+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.774982+0000 osd.4 (osd.4) 159 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.777350+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:09.777350+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.200538+0000 osd.1 (osd.1) 171 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.200538+0000 osd.1 (osd.1) 171 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.202038+0000 osd.1 (osd.1) 172 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.202038+0000 osd.1 (osd.1) 172 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:10.274154+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:10.274154+0000 mon.c (mon.2) 374 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:10.429967+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:10.429967+0000 mon.a (mon.0) 1725 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:25:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.051158+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.051158+0000 mon.a (mon.0) 1726 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.065470+0000 mon.a (mon.0) 1727 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.065470+0000 mon.a (mon.0) 1727 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.067403+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.067403+0000 mon.a (mon.0) 1728 : audit [INF] from='client.? 172.21.15.50:0/2244916449' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi050-73201-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.067466+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.067466+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.067529+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.067529+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:25:11.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.082164+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.082164+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.084227+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.084227+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.085702+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.085702+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.089132+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.089132+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.097628+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.50:0/2561540685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[21463]: audit 2024-04-26T16:25:11.097628+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.50:0/2561540685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.067466+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.067466+0000 mon.a (mon.0) 1729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:11.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.067529+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.067529+0000 mon.a (mon.0) 1730 : audit [INF] from='client.? 172.21.15.50:0/2931612134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.082164+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.082164+0000 mon.b (mon.1) 162 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.084227+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.084227+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e243: 8 total, 8 up, 8 in 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.085702+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.085702+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.089132+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.089132+0000 mon.a (mon.0) 1733 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.097628+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.50:0/2561540685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:11.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:11 smithi050 bash[29234]: audit 2024-04-26T16:25:11.097628+0000 mon.c (mon.2) 375 : audit [DBG] from='client.? 172.21.15.50:0/2561540685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:11.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:11.698 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:11.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:11.830 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:11.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:11.961 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:11.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:12.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:12.092 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:12.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:12.220 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:12.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:12.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:12.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:12.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.805552+0000 osd.4 (osd.4) 161 : cluster [DBG] 2.f scrub starts 2024-04-26T16:25:12.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.805552+0000 osd.4 (osd.4) 161 : cluster [DBG] 2.f scrub starts 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.807656+0000 osd.4 (osd.4) 162 : cluster [DBG] 2.f scrub ok 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:10.807656+0000 osd.4 (osd.4) 162 : cluster [DBG] 2.f scrub ok 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.275449+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.275449+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.334385+0000 mon.a (mon.0) 1734 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.334385+0000 mon.a (mon.0) 1734 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.335581+0000 mon.c (mon.2) 377 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.335581+0000 mon.c (mon.2) 377 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.399073+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? 172.21.15.50:0/2219097201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:11.399073+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? 172.21.15.50:0/2219097201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.547301+0000 mgr.y (mgr.24401) 314 : cluster [DBG] pgmap v349: 271 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 254 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.547301+0000 mgr.y (mgr.24401) 314 : cluster [DBG] pgmap v349: 271 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 254 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:12.068134+0000 mon.a (mon.0) 1735 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:12.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:12.068134+0000 mon.a (mon.0) 1735 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:12.070446+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:12.070446+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:12.070662+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:12.070662+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:12.087866+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: cluster 2024-04-26T16:25:12.087866+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:12.090095+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:12.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:12 smithi190 bash[23868]: audit 2024-04-26T16:25:12.090095+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:12.484 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:12.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:12.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.805552+0000 osd.4 (osd.4) 161 : cluster [DBG] 2.f scrub starts 2024-04-26T16:25:12.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.805552+0000 osd.4 (osd.4) 161 : cluster [DBG] 2.f scrub starts 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.807656+0000 osd.4 (osd.4) 162 : cluster [DBG] 2.f scrub ok 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:10.807656+0000 osd.4 (osd.4) 162 : cluster [DBG] 2.f scrub ok 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.275449+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.275449+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.334385+0000 mon.a (mon.0) 1734 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.334385+0000 mon.a (mon.0) 1734 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.335581+0000 mon.c (mon.2) 377 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.335581+0000 mon.c (mon.2) 377 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.399073+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? 172.21.15.50:0/2219097201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:11.399073+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? 172.21.15.50:0/2219097201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.547301+0000 mgr.y (mgr.24401) 314 : cluster [DBG] pgmap v349: 271 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 254 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.547301+0000 mgr.y (mgr.24401) 314 : cluster [DBG] pgmap v349: 271 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 254 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:12.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:12.068134+0000 mon.a (mon.0) 1735 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:12.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:12.068134+0000 mon.a (mon.0) 1735 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:12.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:12.070446+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:12.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:12.070446+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:12.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:12.070662+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:12.602 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.805552+0000 osd.4 (osd.4) 161 : cluster [DBG] 2.f scrub starts 2024-04-26T16:25:12.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.805552+0000 osd.4 (osd.4) 161 : cluster [DBG] 2.f scrub starts 2024-04-26T16:25:12.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.807656+0000 osd.4 (osd.4) 162 : cluster [DBG] 2.f scrub ok 2024-04-26T16:25:12.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:10.807656+0000 osd.4 (osd.4) 162 : cluster [DBG] 2.f scrub ok 2024-04-26T16:25:12.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.275449+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:12.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.275449+0000 mon.c (mon.2) 376 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:12.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.334385+0000 mon.a (mon.0) 1734 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:12.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.334385+0000 mon.a (mon.0) 1734 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:12.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.335581+0000 mon.c (mon.2) 377 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:12.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.335581+0000 mon.c (mon.2) 377 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:12.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.399073+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? 172.21.15.50:0/2219097201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:25:12.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:11.399073+0000 mon.c (mon.2) 378 : audit [DBG] from='client.? 172.21.15.50:0/2219097201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:25:12.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.547301+0000 mgr.y (mgr.24401) 314 : cluster [DBG] pgmap v349: 271 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 254 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:12.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.547301+0000 mgr.y (mgr.24401) 314 : cluster [DBG] pgmap v349: 271 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 254 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:12.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:12.068134+0000 mon.a (mon.0) 1735 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:12.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:12.068134+0000 mon.a (mon.0) 1735 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:12.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:12.070446+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:12.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:12.070446+0000 mon.a (mon.0) 1736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-17"}]': finished 2024-04-26T16:25:12.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:12.070662+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:12.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:12.070662+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:12.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:12.087866+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-04-26T16:25:12.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: cluster 2024-04-26T16:25:12.087866+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-04-26T16:25:12.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:12.090095+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:12.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[21463]: audit 2024-04-26T16:25:12.090095+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:12.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:12.070662+0000 mon.a (mon.0) 1737 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:12.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:12.087866+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-04-26T16:25:12.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: cluster 2024-04-26T16:25:12.087866+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e244: 8 total, 8 up, 8 in 2024-04-26T16:25:12.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:12.090095+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:12.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:12 smithi050 bash[29234]: audit 2024-04-26T16:25:12.090095+0000 mon.a (mon.0) 1739 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]: dispatch 2024-04-26T16:25:12.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:12.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:12.745 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:12.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:12.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:12.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:12.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:13.007 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:13.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:13.133 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:13.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:13.262 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:13.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.271018+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:13.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.271018+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:13.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.271195+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: cluster 2024-04-26T16:25:11.271195+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:12.276757+0000 mon.c (mon.2) 379 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:12.276757+0000 mon.c (mon.2) 379 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.075070+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.075070+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: cluster 2024-04-26T16:25:13.083956+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: cluster 2024-04-26T16:25:13.083956+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.092001+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.50:0/174295385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.092001+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.50:0/174295385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.116595+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.116595+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.140175+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.140175+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.141654+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.141654+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.142053+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:13 smithi190 bash[23868]: audit 2024-04-26T16:25:13.142053+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:13.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:13.392 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:13.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.271018+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.271018+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.271195+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: cluster 2024-04-26T16:25:11.271195+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:12.276757+0000 mon.c (mon.2) 379 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:12.276757+0000 mon.c (mon.2) 379 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.075070+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.075070+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: cluster 2024-04-26T16:25:13.083956+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-04-26T16:25:13.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: cluster 2024-04-26T16:25:13.083956+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.092001+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.50:0/174295385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.092001+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.50:0/174295385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.116595+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.116595+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.140175+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.140175+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.141654+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.141654+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.142053+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:13.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[29234]: audit 2024-04-26T16:25:13.142053+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.271018+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.271018+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.271195+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: cluster 2024-04-26T16:25:11.271195+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:12.276757+0000 mon.c (mon.2) 379 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:12.276757+0000 mon.c (mon.2) 379 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.075070+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.075070+0000 mon.a (mon.0) 1740 : audit [INF] from='client.? 172.21.15.50:0/607984443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi050-73193-30"}]': finished 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: cluster 2024-04-26T16:25:13.083956+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: cluster 2024-04-26T16:25:13.083956+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e245: 8 total, 8 up, 8 in 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.092001+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.50:0/174295385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.092001+0000 mon.c (mon.2) 380 : audit [INF] from='client.? 172.21.15.50:0/174295385' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.511 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.116595+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.116595+0000 mon.a (mon.0) 1742 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.140175+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.140175+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.141654+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.141654+0000 mon.a (mon.0) 1744 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.142053+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:13.512 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:13 smithi050 bash[21463]: audit 2024-04-26T16:25:13.142053+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:13.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:13.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:13.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:13.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:13.784 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:13.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:13.916 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:13.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:14.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:14.046 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:14.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:14.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:14.271 INFO:tasks.thrashosds.thrasher:j is {'epoch': 243, 'fsid': 'cf170a7a-03e6-11ef-bc93-c7b262605968', 'created': '2024-04-26T16:07:04.494213+0000', 'modified': '2024-04-26T16:25:11.064943+0000', 'last_up_change': '2024-04-26T16:16:46.862501+0000', 'last_in_change': '2024-04-26T16:16:20.945042+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 49, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 225, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T16:12:30.809607+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-26T16:17:27.029484+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-26T16:17:29.124965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-26T16:17:30.749319+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-26T16:17:31.064350+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-26T16:17:33.192127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-26T16:23:28.050102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'LibRadosList_smithi050-73230-1', 'create_time': '2024-04-26T16:23:28.061478+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '12.b', 'ready_epoch': 232, 'last_epoch_started': 231, 'last_epoch_clean': 231, 'source_version': "217'14", 'target_version': "217'13"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '232', 'last_force_op_resend_preluminous': '230', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 109, 'pool_name': 'test-rados-api-smithi050-73372-6', 'create_time': '2024-04-26T16:23:42.172881+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '243', 'last_force_op_resend_prenautilus': '243', 'last_force_op_resend_preluminous': '243', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 240, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [222], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 222, 'pool_name': 'test-rados-api-smithi050-73372-17', 'create_time': '2024-04-26T16:25:05.228117+0000', 'flags': 8201, 'flags_names': 'hashpspool,incomplete_clones,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '243', 'last_force_op_resend_prenautilus': '243', 'last_force_op_resend_preluminous': '243', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 240, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 109, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'writeback', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 225, 'pool_name': 'RoundTripCmpExtPP_smithi050-73201-31', 'create_time': '2024-04-26T16:25:09.109423+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '243', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'b46ea8aa-cd97-425b-90e0-1e70af46b8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 310950989}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 310950989}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 310950989}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 310950989}]}, 'public_addr': '172.21.15.50:6803/310950989', 'cluster_addr': '172.21.15.50:6805/310950989', 'heartbeat_back_addr': '172.21.15.50:6809/310950989', 'heartbeat_front_addr': '172.21.15.50:6807/310950989', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7042f8e0-f5d0-48d2-9cd6-d9445e326eed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3516275849}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 3516275849}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 3516275849}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 3516275849}]}, 'public_addr': '172.21.15.50:6811/3516275849', 'cluster_addr': '172.21.15.50:6813/3516275849', 'heartbeat_back_addr': '172.21.15.50:6817/3516275849', 'heartbeat_front_addr': '172.21.15.50:6815/3516275849', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '33069daa-7fd8-4e9c-aafe-89030f84aad2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 2835380668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 2835380668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 2835380668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 2835380668}]}, 'public_addr': '172.21.15.50:6819/2835380668', 'cluster_addr': '172.21.15.50:6821/2835380668', 'heartbeat_back_addr': '172.21.15.50:6825/2835380668', 'heartbeat_front_addr': '172.21.15.50:6823/2835380668', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '43b6aab5-9b42-4f97-a224-b2872de5cb03', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 2541695077}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 2541695077}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 2541695077}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 2541695077}]}, 'public_addr': '172.21.15.50:6827/2541695077', 'cluster_addr': '172.21.15.50:6829/2541695077', 'heartbeat_back_addr': '172.21.15.50:6833/2541695077', 'heartbeat_front_addr': '172.21.15.50:6831/2541695077', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7220491-e9bf-4188-b203-322e9908558c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 2207965503}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 2207965503}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 2207965503}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 2207965503}]}, 'public_addr': '172.21.15.190:6801/2207965503', 'cluster_addr': '172.21.15.190:6803/2207965503', 'heartbeat_back_addr': '172.21.15.190:6807/2207965503', 'heartbeat_front_addr': '172.21.15.190:6805/2207965503', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ed5c0044-658b-4dfb-8c61-727de6bdd587', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 3373187844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 3373187844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 3373187844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 3373187844}]}, 'public_addr': '172.21.15.190:6809/3373187844', 'cluster_addr': '172.21.15.190:6811/3373187844', 'heartbeat_back_addr': '172.21.15.190:6815/3373187844', 'heartbeat_front_addr': '172.21.15.190:6813/3373187844', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '94111b7d-7887-4a3a-983b-394294735d0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 2399532574}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 2399532574}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 2399532574}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 2399532574}]}, 'public_addr': '172.21.15.190:6817/2399532574', 'cluster_addr': '172.21.15.190:6819/2399532574', 'heartbeat_back_addr': '172.21.15.190:6823/2399532574', 'heartbeat_front_addr': '172.21.15.190:6821/2399532574', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'abbd95e3-07b4-4235-9101-a4dea0f35bb5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 2856617796}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 2856617796}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 2856617796}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 2856617796}]}, 'public_addr': '172.21.15.190:6825/2856617796', 'cluster_addr': '172.21.15.190:6827/2856617796', 'heartbeat_back_addr': '172.21.15.190:6831/2856617796', 'heartbeat_front_addr': '172.21.15.190:6829/2856617796', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:25:09.720161+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:24:09.596090+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:25:03.721238+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:24:37.707040+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:24:03.303873+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:24:08.260991+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:24:27.487359+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:24:22.208545+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/2324264713': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/542146552': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:6801/127073210': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6800/127073210': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6800/126446279': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:6801/126446279': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/3454723725': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:0/2548717623': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6800/685732569': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/2903276422': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6801/685732569': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/2979962403': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/71201882': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/3973398204': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/1172169336': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/3444006067': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/577039843': '2024-04-27T16:18:25.317826+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-RoundTrip2_smithi050-73193-30': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 16, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 20, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 40, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 108, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 109, 'snaps': [{'begin': 3, 'length': 5}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 150, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 158, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 168, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 173, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 207, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 215, 'snaps': [{'begin': 7, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-26T16:25:14.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:25:14.275 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:14.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:13.278361+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:14.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:13.278361+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:14.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: cluster 2024-04-26T16:25:13.548097+0000 mgr.y (mgr.24401) 315 : cluster [DBG] pgmap v352: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:14.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: cluster 2024-04-26T16:25:13.548097+0000 mgr.y (mgr.24401) 315 : cluster [DBG] pgmap v352: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:14.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.079133+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:14.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.079133+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.079433+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.079433+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: cluster 2024-04-26T16:25:14.088799+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: cluster 2024-04-26T16:25:14.088799+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.090199+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.090199+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.094833+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.094833+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.095829+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:14 smithi190 bash[23868]: audit 2024-04-26T16:25:14.095829+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:14.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:13.278361+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:14.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:13.278361+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:14.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: cluster 2024-04-26T16:25:13.548097+0000 mgr.y (mgr.24401) 315 : cluster [DBG] pgmap v352: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:14.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: cluster 2024-04-26T16:25:13.548097+0000 mgr.y (mgr.24401) 315 : cluster [DBG] pgmap v352: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:14.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.079133+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:14.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.079133+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.079433+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.079433+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: cluster 2024-04-26T16:25:14.088799+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: cluster 2024-04-26T16:25:14.088799+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.090199+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.090199+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.094833+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.094833+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.095829+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.407 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[29234]: audit 2024-04-26T16:25:14.095829+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:13.278361+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:13.278361+0000 mon.c (mon.2) 381 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: cluster 2024-04-26T16:25:13.548097+0000 mgr.y (mgr.24401) 315 : cluster [DBG] pgmap v352: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: cluster 2024-04-26T16:25:13.548097+0000 mgr.y (mgr.24401) 315 : cluster [DBG] pgmap v352: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.079133+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.079133+0000 mon.a (mon.0) 1746 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi050-73201-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.079433+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.079433+0000 mon.a (mon.0) 1747 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi050-73193-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: cluster 2024-04-26T16:25:14.088799+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: cluster 2024-04-26T16:25:14.088799+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e246: 8 total, 8 up, 8 in 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.090199+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.090199+0000 mon.a (mon.0) 1749 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.094833+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.094833+0000 mon.a (mon.0) 1750 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.095829+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21463]: audit 2024-04-26T16:25:14.095829+0000 mon.b (mon.1) 163 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:14.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:14.471 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:14.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:14.586 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:25:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:25:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:25:14.601 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:14.632 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:14.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:14.732 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:14.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:14.862 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:14.965 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:14.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:14.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:15.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:15.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:15.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:14.279411+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:15.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:14.279411+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:15.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: cluster 2024-04-26T16:25:14.352389+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:15.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: cluster 2024-04-26T16:25:14.352389+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:15.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: cluster 2024-04-26T16:25:14.352944+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:15.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: cluster 2024-04-26T16:25:14.352944+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:15.083190+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:15.083190+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:15.105786+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:15.105786+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: cluster 2024-04-26T16:25:15.112885+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: cluster 2024-04-26T16:25:15.112885+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:15.115008+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[21463]: audit 2024-04-26T16:25:15.115008+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:14.279411+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:14.279411+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: cluster 2024-04-26T16:25:14.352389+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: cluster 2024-04-26T16:25:14.352389+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: cluster 2024-04-26T16:25:14.352944+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: cluster 2024-04-26T16:25:14.352944+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:15.083190+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:15.083190+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:15.105786+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:15.105786+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: cluster 2024-04-26T16:25:15.112885+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: cluster 2024-04-26T16:25:15.112885+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-04-26T16:25:15.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:15.115008+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:15 smithi050 bash[29234]: audit 2024-04-26T16:25:15.115008+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:15.263 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:15.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:15.393 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:15.496 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:15.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:14.279411+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:14.279411+0000 mon.c (mon.2) 382 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: cluster 2024-04-26T16:25:14.352389+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: cluster 2024-04-26T16:25:14.352389+0000 osd.5 (osd.5) 185 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: cluster 2024-04-26T16:25:14.352944+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: cluster 2024-04-26T16:25:14.352944+0000 osd.5 (osd.5) 186 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:15.083190+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:15.083190+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:15.105786+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:15.105786+0000 mon.b (mon.1) 164 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: cluster 2024-04-26T16:25:15.112885+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: cluster 2024-04-26T16:25:15.112885+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e247: 8 total, 8 up, 8 in 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:15.115008+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:15 smithi190 bash[23868]: audit 2024-04-26T16:25:15.115008+0000 mon.a (mon.0) 1753 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:15.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:15.653 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:15.756 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:15.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:15.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:15.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:15.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:16.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:16.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:16.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:16.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:25:16.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:16.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:25:16.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:16.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:15.280520+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:16.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:15.280520+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: cluster 2024-04-26T16:25:15.548718+0000 mgr.y (mgr.24401) 316 : cluster [DBG] pgmap v355: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: cluster 2024-04-26T16:25:15.548718+0000 mgr.y (mgr.24401) 316 : cluster [DBG] pgmap v355: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.054724+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.054724+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.054862+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.054862+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: cluster 2024-04-26T16:25:16.059377+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: cluster 2024-04-26T16:25:16.059377+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.068908+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.068908+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.071004+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.071004+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:15.280520+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:15.280520+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: cluster 2024-04-26T16:25:15.548718+0000 mgr.y (mgr.24401) 316 : cluster [DBG] pgmap v355: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: cluster 2024-04-26T16:25:15.548718+0000 mgr.y (mgr.24401) 316 : cluster [DBG] pgmap v355: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.054724+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.054724+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.054862+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.054862+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: cluster 2024-04-26T16:25:16.059377+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: cluster 2024-04-26T16:25:16.059377+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.068908+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.068908+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.071004+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.071004+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.075356+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.075356+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.084285+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[29234]: audit 2024-04-26T16:25:16.084285+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.075356+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.075356+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.084285+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:16 smithi050 bash[21463]: audit 2024-04-26T16:25:16.084285+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:16.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:16.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:16.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:16.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:15.280520+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:15.280520+0000 mon.c (mon.2) 383 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: cluster 2024-04-26T16:25:15.548718+0000 mgr.y (mgr.24401) 316 : cluster [DBG] pgmap v355: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: cluster 2024-04-26T16:25:15.548718+0000 mgr.y (mgr.24401) 316 : cluster [DBG] pgmap v355: 239 pgs: 32 unknown, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 190 active+clean; 587 KiB data, 566 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.054724+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.054724+0000 mon.a (mon.0) 1754 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi050-73193-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.054862+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.054862+0000 mon.a (mon.0) 1755 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:16.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: cluster 2024-04-26T16:25:16.059377+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: cluster 2024-04-26T16:25:16.059377+0000 mon.a (mon.0) 1756 : cluster [DBG] osdmap e248: 8 total, 8 up, 8 in 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.068908+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.068908+0000 mon.b (mon.1) 165 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.071004+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.071004+0000 mon.c (mon.2) 384 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.075356+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.075356+0000 mon.a (mon.0) 1757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.084285+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[23868]: audit 2024-04-26T16:25:16.084285+0000 mon.a (mon.0) 1758 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:16.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:16.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:16.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:16.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:16.852 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:16.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:16.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:16.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:17.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:25:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:25:17.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:17.123 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:17.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:17.225 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:17.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:17.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:17.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: cluster 2024-04-26T16:25:16.233172+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: cluster 2024-04-26T16:25:16.233172+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:16.281676+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:16.281676+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.056986+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.056986+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.057126+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.057126+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: cluster 2024-04-26T16:25:17.062267+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: cluster 2024-04-26T16:25:17.062267+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.069753+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.069753+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.070610+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:17 smithi190 bash[23868]: audit 2024-04-26T16:25:17.070610+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.377 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:17.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:17.508 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:17.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:17.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: cluster 2024-04-26T16:25:16.233172+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:17.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: cluster 2024-04-26T16:25:16.233172+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:17.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:16.281676+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:17.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:16.281676+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:17.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.056986+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:17.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.056986+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:17.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.057126+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:17.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.057126+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:17.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: cluster 2024-04-26T16:25:17.062267+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-04-26T16:25:17.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: cluster 2024-04-26T16:25:17.062267+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.069753+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.069753+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.070610+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[21463]: audit 2024-04-26T16:25:17.070610+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: cluster 2024-04-26T16:25:16.233172+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: cluster 2024-04-26T16:25:16.233172+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:16.281676+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:16.281676+0000 mon.c (mon.2) 385 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.056986+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.056986+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi050-73201-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.057126+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:17.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.057126+0000 mon.a (mon.0) 1761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:17.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: cluster 2024-04-26T16:25:17.062267+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-04-26T16:25:17.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: cluster 2024-04-26T16:25:17.062267+0000 mon.a (mon.0) 1762 : cluster [DBG] osdmap e249: 8 total, 8 up, 8 in 2024-04-26T16:25:17.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.069753+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.069753+0000 mon.b (mon.1) 166 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.070610+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:17 smithi050 bash[29234]: audit 2024-04-26T16:25:17.070610+0000 mon.a (mon.0) 1763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]: dispatch 2024-04-26T16:25:17.638 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:17.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:17.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:17.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:17.900 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:17.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:17.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:18.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:18.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:18.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:18.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:18.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:18.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:16.707257+0000 mgr.y (mgr.24401) 317 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:18.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:16.707257+0000 mgr.y (mgr.24401) 317 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:18.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.282846+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:18.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.282846+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:18.271 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: cluster 2024-04-26T16:25:17.549796+0000 mgr.y (mgr.24401) 318 : cluster [DBG] pgmap v358: 279 pgs: 40 creating+peering, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: cluster 2024-04-26T16:25:17.549796+0000 mgr.y (mgr.24401) 318 : cluster [DBG] pgmap v358: 279 pgs: 40 creating+peering, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570162+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570162+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570242+0000 mon.c (mon.2) 388 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570242+0000 mon.c (mon.2) 388 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570295+0000 mon.c (mon.2) 389 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570295+0000 mon.c (mon.2) 389 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570345+0000 mon.c (mon.2) 390 : audit [INF] "var": "eio", 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570345+0000 mon.c (mon.2) 390 : audit [INF] "var": "eio", 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570397+0000 mon.c (mon.2) 391 : audit [INF] "val": "true" 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570397+0000 mon.c (mon.2) 391 : audit [INF] "val": "true" 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570444+0000 mon.c (mon.2) 392 : audit [INF] }]: dispatch 2024-04-26T16:25:18.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.570444+0000 mon.c (mon.2) 392 : audit [INF] }]: dispatch 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571082+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571082+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571167+0000 mon.a (mon.0) 1765 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571167+0000 mon.a (mon.0) 1765 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:16.707257+0000 mgr.y (mgr.24401) 317 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:16.707257+0000 mgr.y (mgr.24401) 317 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.282846+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.282846+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: cluster 2024-04-26T16:25:17.549796+0000 mgr.y (mgr.24401) 318 : cluster [DBG] pgmap v358: 279 pgs: 40 creating+peering, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:18.273 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: cluster 2024-04-26T16:25:17.549796+0000 mgr.y (mgr.24401) 318 : cluster [DBG] pgmap v358: 279 pgs: 40 creating+peering, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570162+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570162+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570242+0000 mon.c (mon.2) 388 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570242+0000 mon.c (mon.2) 388 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570295+0000 mon.c (mon.2) 389 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570295+0000 mon.c (mon.2) 389 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570345+0000 mon.c (mon.2) 390 : audit [INF] "var": "eio", 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570345+0000 mon.c (mon.2) 390 : audit [INF] "var": "eio", 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570397+0000 mon.c (mon.2) 391 : audit [INF] "val": "true" 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570397+0000 mon.c (mon.2) 391 : audit [INF] "val": "true" 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570444+0000 mon.c (mon.2) 392 : audit [INF] }]: dispatch 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.570444+0000 mon.c (mon.2) 392 : audit [INF] }]: dispatch 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571082+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.274 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571082+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571167+0000 mon.a (mon.0) 1765 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571167+0000 mon.a (mon.0) 1765 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571207+0000 mon.a (mon.0) 1766 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571207+0000 mon.a (mon.0) 1766 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571240+0000 mon.a (mon.0) 1767 : audit [INF] "var": "eio", 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571240+0000 mon.a (mon.0) 1767 : audit [INF] "var": "eio", 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571271+0000 mon.a (mon.0) 1768 : audit [INF] "val": "true" 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571271+0000 mon.a (mon.0) 1768 : audit [INF] "val": "true" 2024-04-26T16:25:18.275 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571297+0000 mon.a (mon.0) 1769 : audit [INF] }]: dispatch 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:17.571297+0000 mon.a (mon.0) 1769 : audit [INF] }]: dispatch 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: cluster 2024-04-26T16:25:18.057370+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: cluster 2024-04-26T16:25:18.057370+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059172+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]': finished 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059172+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]': finished 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059286+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059286+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059313+0000 mon.a (mon.0) 1773 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059313+0000 mon.a (mon.0) 1773 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059333+0000 mon.a (mon.0) 1774 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059333+0000 mon.a (mon.0) 1774 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059346+0000 mon.a (mon.0) 1775 : audit [INF] "var": "eio", 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059346+0000 mon.a (mon.0) 1775 : audit [INF] "var": "eio", 2024-04-26T16:25:18.276 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059356+0000 mon.a (mon.0) 1776 : audit [INF] "val": "true" 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059356+0000 mon.a (mon.0) 1776 : audit [INF] "val": "true" 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059365+0000 mon.a (mon.0) 1777 : audit [INF] }]': finished 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.059365+0000 mon.a (mon.0) 1777 : audit [INF] }]': finished 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571207+0000 mon.a (mon.0) 1766 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571207+0000 mon.a (mon.0) 1766 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571240+0000 mon.a (mon.0) 1767 : audit [INF] "var": "eio", 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571240+0000 mon.a (mon.0) 1767 : audit [INF] "var": "eio", 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571271+0000 mon.a (mon.0) 1768 : audit [INF] "val": "true" 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571271+0000 mon.a (mon.0) 1768 : audit [INF] "val": "true" 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571297+0000 mon.a (mon.0) 1769 : audit [INF] }]: dispatch 2024-04-26T16:25:18.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:17.571297+0000 mon.a (mon.0) 1769 : audit [INF] }]: dispatch 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: cluster 2024-04-26T16:25:18.057370+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: cluster 2024-04-26T16:25:18.057370+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059172+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]': finished 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059172+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]': finished 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059286+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059286+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059313+0000 mon.a (mon.0) 1773 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059313+0000 mon.a (mon.0) 1773 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059333+0000 mon.a (mon.0) 1774 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059333+0000 mon.a (mon.0) 1774 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059346+0000 mon.a (mon.0) 1775 : audit [INF] "var": "eio", 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059346+0000 mon.a (mon.0) 1775 : audit [INF] "var": "eio", 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059356+0000 mon.a (mon.0) 1776 : audit [INF] "val": "true" 2024-04-26T16:25:18.278 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059356+0000 mon.a (mon.0) 1776 : audit [INF] "val": "true" 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059365+0000 mon.a (mon.0) 1777 : audit [INF] }]': finished 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.059365+0000 mon.a (mon.0) 1777 : audit [INF] }]': finished 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: cluster 2024-04-26T16:25:18.063429+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: cluster 2024-04-26T16:25:18.063429+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.068282+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.068282+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.132913+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.132913+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.135044+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.279 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[29234]: audit 2024-04-26T16:25:18.135044+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: cluster 2024-04-26T16:25:18.063429+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: cluster 2024-04-26T16:25:18.063429+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.068282+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.068282+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.132913+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.132913+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.135044+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:18 smithi050 bash[21463]: audit 2024-04-26T16:25:18.135044+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:18.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:18.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:18.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:18.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:18.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:16.707257+0000 mgr.y (mgr.24401) 317 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:18.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:16.707257+0000 mgr.y (mgr.24401) 317 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:18.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.282846+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.282846+0000 mon.c (mon.2) 386 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: cluster 2024-04-26T16:25:17.549796+0000 mgr.y (mgr.24401) 318 : cluster [DBG] pgmap v358: 279 pgs: 40 creating+peering, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: cluster 2024-04-26T16:25:17.549796+0000 mgr.y (mgr.24401) 318 : cluster [DBG] pgmap v358: 279 pgs: 40 creating+peering, 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570162+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570162+0000 mon.c (mon.2) 387 : audit [INF] from='client.? 172.21.15.50:0/3201359922' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570242+0000 mon.c (mon.2) 388 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570242+0000 mon.c (mon.2) 388 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570295+0000 mon.c (mon.2) 389 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570295+0000 mon.c (mon.2) 389 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570345+0000 mon.c (mon.2) 390 : audit [INF] "var": "eio", 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570345+0000 mon.c (mon.2) 390 : audit [INF] "var": "eio", 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570397+0000 mon.c (mon.2) 391 : audit [INF] "val": "true" 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570397+0000 mon.c (mon.2) 391 : audit [INF] "val": "true" 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570444+0000 mon.c (mon.2) 392 : audit [INF] }]: dispatch 2024-04-26T16:25:18.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.570444+0000 mon.c (mon.2) 392 : audit [INF] }]: dispatch 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571082+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571082+0000 mon.a (mon.0) 1764 : audit [INF] from='client.? ' entity='client.admin' cmd=[{ 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571167+0000 mon.a (mon.0) 1765 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571167+0000 mon.a (mon.0) 1765 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571207+0000 mon.a (mon.0) 1766 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571207+0000 mon.a (mon.0) 1766 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571240+0000 mon.a (mon.0) 1767 : audit [INF] "var": "eio", 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571240+0000 mon.a (mon.0) 1767 : audit [INF] "var": "eio", 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571271+0000 mon.a (mon.0) 1768 : audit [INF] "val": "true" 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571271+0000 mon.a (mon.0) 1768 : audit [INF] "val": "true" 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571297+0000 mon.a (mon.0) 1769 : audit [INF] }]: dispatch 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:17.571297+0000 mon.a (mon.0) 1769 : audit [INF] }]: dispatch 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: cluster 2024-04-26T16:25:18.057370+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: cluster 2024-04-26T16:25:18.057370+0000 mon.a (mon.0) 1770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059172+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]': finished 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059172+0000 mon.a (mon.0) 1771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-19", "mode": "writeback"}]': finished 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059286+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059286+0000 mon.a (mon.0) 1772 : audit [INF] from='client.? ' entity='client.admin' cmd='[{ 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059313+0000 mon.a (mon.0) 1773 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059313+0000 mon.a (mon.0) 1773 : audit [INF] "prefix": "osd pool set", 2024-04-26T16:25:18.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059333+0000 mon.a (mon.0) 1774 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059333+0000 mon.a (mon.0) 1774 : audit [INF] "pool": "PoolEIOFlag_smithi050-73201-33", 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059346+0000 mon.a (mon.0) 1775 : audit [INF] "var": "eio", 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059346+0000 mon.a (mon.0) 1775 : audit [INF] "var": "eio", 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059356+0000 mon.a (mon.0) 1776 : audit [INF] "val": "true" 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059356+0000 mon.a (mon.0) 1776 : audit [INF] "val": "true" 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059365+0000 mon.a (mon.0) 1777 : audit [INF] }]': finished 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.059365+0000 mon.a (mon.0) 1777 : audit [INF] }]': finished 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: cluster 2024-04-26T16:25:18.063429+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: cluster 2024-04-26T16:25:18.063429+0000 mon.a (mon.0) 1778 : cluster [DBG] osdmap e250: 8 total, 8 up, 8 in 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.068282+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.068282+0000 mon.a (mon.0) 1779 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.132913+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.132913+0000 mon.a (mon.0) 1780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.135044+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:18 smithi190 bash[23868]: audit 2024-04-26T16:25:18.135044+0000 mon.b (mon.1) 167 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:18.547 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:18.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:18.635 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:25:18.635 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:25:18.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:18.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:18.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:18.801 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:18.904 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:18.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:18.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:19.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:19.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:19.215 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:19.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:19.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:19.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:19.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:19.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:18.283806+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:19.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:18.283806+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.061979+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.061979+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.062186+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.062186+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: cluster 2024-04-26T16:25:19.075653+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: cluster 2024-04-26T16:25:19.075653+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.075971+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.075971+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.077012+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.077012+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:19.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.077510+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:19 smithi190 bash[23868]: audit 2024-04-26T16:25:19.077510+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:19.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:18.283806+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:19.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:18.283806+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:19.590 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.061979+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.061979+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.062186+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.062186+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: cluster 2024-04-26T16:25:19.075653+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: cluster 2024-04-26T16:25:19.075653+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.075971+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.075971+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.077012+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.077012+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.077510+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[29234]: audit 2024-04-26T16:25:19.077510+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:18.283806+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:18.283806+0000 mon.c (mon.2) 393 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.061979+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.061979+0000 mon.a (mon.0) 1781 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.062186+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.062186+0000 mon.a (mon.0) 1782 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: cluster 2024-04-26T16:25:19.075653+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: cluster 2024-04-26T16:25:19.075653+0000 mon.a (mon.0) 1783 : cluster [DBG] osdmap e251: 8 total, 8 up, 8 in 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.075971+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.075971+0000 mon.b (mon.1) 168 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.077012+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.077012+0000 mon.a (mon.0) 1784 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.077510+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:19 smithi050 bash[21463]: audit 2024-04-26T16:25:19.077510+0000 mon.a (mon.0) 1785 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]: dispatch 2024-04-26T16:25:19.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:19.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:19.737 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:19.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:19.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-26T16:25:19.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:19.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:19.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:19.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:20.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:20.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:20.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:20.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:20.318 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:20.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:20.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:19.285124+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:20.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:19.285124+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:20.364 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: cluster 2024-04-26T16:25:19.550433+0000 mgr.y (mgr.24401) 319 : cluster [DBG] pgmap v361: 239 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: cluster 2024-04-26T16:25:19.550433+0000 mgr.y (mgr.24401) 319 : cluster [DBG] pgmap v361: 239 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: cluster 2024-04-26T16:25:20.062911+0000 mon.a (mon.0) 1786 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: cluster 2024-04-26T16:25:20.062911+0000 mon.a (mon.0) 1786 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.064833+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.064833+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.064919+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.064919+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: cluster 2024-04-26T16:25:20.069140+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: cluster 2024-04-26T16:25:20.069140+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.081781+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.50:0/2900656996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.081781+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.50:0/2900656996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.082570+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.365 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.082570+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:19.285124+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:19.285124+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: cluster 2024-04-26T16:25:19.550433+0000 mgr.y (mgr.24401) 319 : cluster [DBG] pgmap v361: 239 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: cluster 2024-04-26T16:25:19.550433+0000 mgr.y (mgr.24401) 319 : cluster [DBG] pgmap v361: 239 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: cluster 2024-04-26T16:25:20.062911+0000 mon.a (mon.0) 1786 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: cluster 2024-04-26T16:25:20.062911+0000 mon.a (mon.0) 1786 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.064833+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.064833+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.064919+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.064919+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: cluster 2024-04-26T16:25:20.069140+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-04-26T16:25:20.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: cluster 2024-04-26T16:25:20.069140+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.081781+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.50:0/2900656996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.081781+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.50:0/2900656996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.082570+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.082570+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.101854+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.101854+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.102250+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.102250+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.103272+0000 mon.c (mon.2) 397 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.103272+0000 mon.c (mon.2) 397 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.367 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.103599+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.103599+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.104179+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.104179+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.104534+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[29234]: audit 2024-04-26T16:25:20.104534+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.101854+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.101854+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.102250+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.102250+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.103272+0000 mon.c (mon.2) 397 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.103272+0000 mon.c (mon.2) 397 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.103599+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.103599+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.104179+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.104179+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.104534+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:20 smithi050 bash[21463]: audit 2024-04-26T16:25:20.104534+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:20.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:20.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:19.285124+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:19.285124+0000 mon.c (mon.2) 394 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: cluster 2024-04-26T16:25:19.550433+0000 mgr.y (mgr.24401) 319 : cluster [DBG] pgmap v361: 239 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: cluster 2024-04-26T16:25:19.550433+0000 mgr.y (mgr.24401) 319 : cluster [DBG] pgmap v361: 239 pgs: 7 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 222 active+clean; 587 KiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: cluster 2024-04-26T16:25:20.062911+0000 mon.a (mon.0) 1786 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: cluster 2024-04-26T16:25:20.062911+0000 mon.a (mon.0) 1786 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.064833+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:20.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.064833+0000 mon.a (mon.0) 1787 : audit [INF] from='client.? 172.21.15.50:0/1350639441' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi050-73193-31"}]': finished 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.064919+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.064919+0000 mon.a (mon.0) 1788 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-19"}]': finished 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: cluster 2024-04-26T16:25:20.069140+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: cluster 2024-04-26T16:25:20.069140+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e252: 8 total, 8 up, 8 in 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.081781+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.50:0/2900656996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.081781+0000 mon.c (mon.2) 395 : audit [INF] from='client.? 172.21.15.50:0/2900656996' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.082570+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.082570+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.101854+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.101854+0000 mon.c (mon.2) 396 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.102250+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.102250+0000 mon.a (mon.0) 1791 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.103272+0000 mon.c (mon.2) 397 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.512 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.103272+0000 mon.c (mon.2) 397 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.103599+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.103599+0000 mon.a (mon.0) 1792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.104179+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.104179+0000 mon.c (mon.2) 398 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.104534+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:20 smithi190 bash[23868]: audit 2024-04-26T16:25:20.104534+0000 mon.a (mon.0) 1793 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:20.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:20.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:20.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:20.773 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:20.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:20.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:20.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:21.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:21.033 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:21.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:21.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:21.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:21.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:21.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:20.286677+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:21.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:20.286677+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:21.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.058215+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:21.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.058215+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:21.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.058449+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.058449+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: cluster 2024-04-26T16:25:21.081935+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: cluster 2024-04-26T16:25:21.081935+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.086785+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.086785+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.122510+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:21 smithi190 bash[23868]: audit 2024-04-26T16:25:21.122510+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.343 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:21.343 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:21.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:21.423 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:21.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:21.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:20.286677+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:21.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:20.286677+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:21.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.058215+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:21.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.058215+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:21.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.058449+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:21.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.058449+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: cluster 2024-04-26T16:25:21.081935+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: cluster 2024-04-26T16:25:21.081935+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.086785+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.086785+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.122510+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[29234]: audit 2024-04-26T16:25:21.122510+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:20.286677+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:20.286677+0000 mon.c (mon.2) 399 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.058215+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.058215+0000 mon.a (mon.0) 1794 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi050-73201-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:21.540 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.058449+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.058449+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi050-73193-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: cluster 2024-04-26T16:25:21.081935+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: cluster 2024-04-26T16:25:21.081935+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e253: 8 total, 8 up, 8 in 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.086785+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.086785+0000 mon.c (mon.2) 400 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.122510+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:21 smithi050 bash[21463]: audit 2024-04-26T16:25:21.122510+0000 mon.a (mon.0) 1797 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:21.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:21.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:21.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:21.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:21.813 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:21.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:21.944 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:21.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:22.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:22.090 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:22.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:22.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:22.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: audit 2024-04-26T16:25:21.287897+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: audit 2024-04-26T16:25:21.287897+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:22.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: cluster 2024-04-26T16:25:21.552128+0000 mgr.y (mgr.24401) 320 : cluster [DBG] pgmap v364: 239 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 227 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 511 B/s wr, 26 op/s 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: cluster 2024-04-26T16:25:21.552128+0000 mgr.y (mgr.24401) 320 : cluster [DBG] pgmap v364: 239 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 227 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 511 B/s wr, 26 op/s 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: audit 2024-04-26T16:25:22.082293+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: audit 2024-04-26T16:25:22.082293+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: cluster 2024-04-26T16:25:22.095232+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: cluster 2024-04-26T16:25:22.095232+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: audit 2024-04-26T16:25:22.098671+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:22 smithi190 bash[23868]: audit 2024-04-26T16:25:22.098671+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.348 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:22.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:22.478 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:22.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:22.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: audit 2024-04-26T16:25:21.287897+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:22.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: audit 2024-04-26T16:25:21.287897+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:22.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: cluster 2024-04-26T16:25:21.552128+0000 mgr.y (mgr.24401) 320 : cluster [DBG] pgmap v364: 239 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 227 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 511 B/s wr, 26 op/s 2024-04-26T16:25:22.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: cluster 2024-04-26T16:25:21.552128+0000 mgr.y (mgr.24401) 320 : cluster [DBG] pgmap v364: 239 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 227 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 511 B/s wr, 26 op/s 2024-04-26T16:25:22.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: audit 2024-04-26T16:25:22.082293+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: audit 2024-04-26T16:25:22.082293+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: cluster 2024-04-26T16:25:22.095232+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: cluster 2024-04-26T16:25:22.095232+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: audit 2024-04-26T16:25:22.098671+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[29234]: audit 2024-04-26T16:25:22.098671+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: audit 2024-04-26T16:25:21.287897+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: audit 2024-04-26T16:25:21.287897+0000 mon.c (mon.2) 401 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: cluster 2024-04-26T16:25:21.552128+0000 mgr.y (mgr.24401) 320 : cluster [DBG] pgmap v364: 239 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 227 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 511 B/s wr, 26 op/s 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: cluster 2024-04-26T16:25:21.552128+0000 mgr.y (mgr.24401) 320 : cluster [DBG] pgmap v364: 239 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 227 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 511 B/s wr, 26 op/s 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: audit 2024-04-26T16:25:22.082293+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: audit 2024-04-26T16:25:22.082293+0000 mon.b (mon.1) 169 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: cluster 2024-04-26T16:25:22.095232+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-04-26T16:25:22.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: cluster 2024-04-26T16:25:22.095232+0000 mon.a (mon.0) 1798 : cluster [DBG] osdmap e254: 8 total, 8 up, 8 in 2024-04-26T16:25:22.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: audit 2024-04-26T16:25:22.098671+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:22 smithi050 bash[21463]: audit 2024-04-26T16:25:22.098671+0000 mon.a (mon.0) 1799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:22.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:22.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:22.739 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:22.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:22.869 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:22.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:22.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:23.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:23.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:23.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:23.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:23.282 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:23.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:23.410 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:23.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:22.289065+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:22.289065+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.065861+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.065861+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.065955+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.065955+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: cluster 2024-04-26T16:25:23.070877+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-04-26T16:25:23.525 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: cluster 2024-04-26T16:25:23.070877+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.072304+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.50:0/542221951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.072304+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.50:0/542221951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.072829+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.072829+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.162425+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.162425+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.164165+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:23 smithi190 bash[23868]: audit 2024-04-26T16:25:23.164165+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:22.289065+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:22.289065+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.065861+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.065861+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.065955+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.065955+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: cluster 2024-04-26T16:25:23.070877+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-04-26T16:25:23.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: cluster 2024-04-26T16:25:23.070877+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.072304+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.50:0/542221951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.072304+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.50:0/542221951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.072829+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.072829+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.162425+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.162425+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.164165+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[21463]: audit 2024-04-26T16:25:23.164165+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:22.289065+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:22.289065+0000 mon.c (mon.2) 402 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.065861+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.065861+0000 mon.a (mon.0) 1800 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi050-73193-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.065955+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.065955+0000 mon.a (mon.0) 1801 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: cluster 2024-04-26T16:25:23.070877+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: cluster 2024-04-26T16:25:23.070877+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e255: 8 total, 8 up, 8 in 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.072304+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.50:0/542221951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.072304+0000 mon.c (mon.2) 403 : audit [INF] from='client.? 172.21.15.50:0/542221951' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.072829+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.072829+0000 mon.a (mon.0) 1803 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:23.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.162425+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.162425+0000 mon.a (mon.0) 1804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.164165+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:23 smithi050 bash[29234]: audit 2024-04-26T16:25:23.164165+0000 mon.b (mon.1) 170 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:23.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:23.670 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:23.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:23.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:23.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:23.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:24.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:24.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:24.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:24.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:24.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:24.316 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:23.356092+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:23.356092+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: cluster 2024-04-26T16:25:23.552808+0000 mgr.y (mgr.24401) 321 : cluster [DBG] pgmap v367: 279 pgs: 72 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: cluster 2024-04-26T16:25:23.552808+0000 mgr.y (mgr.24401) 321 : cluster [DBG] pgmap v367: 279 pgs: 72 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.068489+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.068489+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.068589+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.068589+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: cluster 2024-04-26T16:25:24.072957+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-04-26T16:25:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: cluster 2024-04-26T16:25:24.072957+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.082219+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.082219+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.082804+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21463]: audit 2024-04-26T16:25:24.082804+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:23.356092+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:23.356092+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: cluster 2024-04-26T16:25:23.552808+0000 mgr.y (mgr.24401) 321 : cluster [DBG] pgmap v367: 279 pgs: 72 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: cluster 2024-04-26T16:25:23.552808+0000 mgr.y (mgr.24401) 321 : cluster [DBG] pgmap v367: 279 pgs: 72 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.068489+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.068489+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.068589+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.068589+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: cluster 2024-04-26T16:25:24.072957+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: cluster 2024-04-26T16:25:24.072957+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.082219+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.082219+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.082804+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[29234]: audit 2024-04-26T16:25:24.082804+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.415 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:24.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:23.356092+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:23.356092+0000 mon.c (mon.2) 404 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: cluster 2024-04-26T16:25:23.552808+0000 mgr.y (mgr.24401) 321 : cluster [DBG] pgmap v367: 279 pgs: 72 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: cluster 2024-04-26T16:25:23.552808+0000 mgr.y (mgr.24401) 321 : cluster [DBG] pgmap v367: 279 pgs: 72 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.068489+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.068489+0000 mon.a (mon.0) 1805 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi050-73201-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.068589+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.068589+0000 mon.a (mon.0) 1806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: cluster 2024-04-26T16:25:24.072957+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-04-26T16:25:24.431 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: cluster 2024-04-26T16:25:24.072957+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e256: 8 total, 8 up, 8 in 2024-04-26T16:25:24.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.082219+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.082219+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.082804+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:24 smithi190 bash[23868]: audit 2024-04-26T16:25:24.082804+0000 mon.b (mon.1) 171 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:24.445 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:24.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:24.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:24.560 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:25:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:25:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:25:24.576 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:24.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:24.708 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:24.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:24.837 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:24.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:24.968 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:24.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:24.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:25.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:25.098 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:25.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:25.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:25.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:25.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: cluster 2024-04-26T16:25:24.278266+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: cluster 2024-04-26T16:25:24.278266+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:24.357968+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:24.357968+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.071079+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.071079+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: cluster 2024-04-26T16:25:25.078990+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: cluster 2024-04-26T16:25:25.078990+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-04-26T16:25:25.343 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.087382+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.087382+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.087444+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.087444+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.087726+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.087726+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.088281+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:25 smithi190 bash[23868]: audit 2024-04-26T16:25:25.088281+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.357 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:25.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:25.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:25.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: cluster 2024-04-26T16:25:24.278266+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: cluster 2024-04-26T16:25:24.278266+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:24.357968+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:24.357968+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.071079+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.071079+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: cluster 2024-04-26T16:25:25.078990+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: cluster 2024-04-26T16:25:25.078990+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.087382+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.087382+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.087444+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.087444+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.087726+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.087726+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.088281+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[29234]: audit 2024-04-26T16:25:25.088281+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: cluster 2024-04-26T16:25:24.278266+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: cluster 2024-04-26T16:25:24.278266+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:24.357968+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:24.357968+0000 mon.c (mon.2) 405 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.071079+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.071079+0000 mon.a (mon.0) 1810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:25.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: cluster 2024-04-26T16:25:25.078990+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: cluster 2024-04-26T16:25:25.078990+0000 mon.a (mon.0) 1811 : cluster [DBG] osdmap e257: 8 total, 8 up, 8 in 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.087382+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.087382+0000 mon.b (mon.1) 172 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.087444+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.087444+0000 mon.c (mon.2) 406 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.087726+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.087726+0000 mon.a (mon.0) 1812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.088281+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:25 smithi050 bash[21463]: audit 2024-04-26T16:25:25.088281+0000 mon.a (mon.0) 1813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:25.484 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:25.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:25.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:25.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:25.745 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:25.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:25.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:25.874 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:25.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:25.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:25.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:26.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:26.138 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:26.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:26.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:26.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:26.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:25.359701+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:26.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:25.359701+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:26.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: cluster 2024-04-26T16:25:25.553501+0000 mgr.y (mgr.24401) 322 : cluster [DBG] pgmap v370: 239 pgs: 32 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:26.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: cluster 2024-04-26T16:25:25.553501+0000 mgr.y (mgr.24401) 322 : cluster [DBG] pgmap v370: 239 pgs: 32 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:26.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: cluster 2024-04-26T16:25:26.059516+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: cluster 2024-04-26T16:25:26.059516+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.061492+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]': finished 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.061492+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]': finished 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.061573+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.061573+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: cluster 2024-04-26T16:25:26.065567+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: cluster 2024-04-26T16:25:26.065567+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.073096+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.073096+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.074498+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.50:0/1906263429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.074498+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.50:0/1906263429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.081155+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.081155+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.081799+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.081799+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.165776+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.165776+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.168111+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[23868]: audit 2024-04-26T16:25:26.168111+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.401 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:26.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:26.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:26.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:26.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:25.359701+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:26.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:25.359701+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:26.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: cluster 2024-04-26T16:25:25.553501+0000 mgr.y (mgr.24401) 322 : cluster [DBG] pgmap v370: 239 pgs: 32 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:26.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: cluster 2024-04-26T16:25:25.553501+0000 mgr.y (mgr.24401) 322 : cluster [DBG] pgmap v370: 239 pgs: 32 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: cluster 2024-04-26T16:25:26.059516+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: cluster 2024-04-26T16:25:26.059516+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.061492+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]': finished 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.061492+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]': finished 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.061573+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.061573+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: cluster 2024-04-26T16:25:26.065567+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: cluster 2024-04-26T16:25:26.065567+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.073096+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.073096+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.074498+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.50:0/1906263429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.074498+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.50:0/1906263429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:25.359701+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:25.359701+0000 mon.c (mon.2) 407 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: cluster 2024-04-26T16:25:25.553501+0000 mgr.y (mgr.24401) 322 : cluster [DBG] pgmap v370: 239 pgs: 32 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: cluster 2024-04-26T16:25:25.553501+0000 mgr.y (mgr.24401) 322 : cluster [DBG] pgmap v370: 239 pgs: 32 unknown, 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 195 active+clean; 586 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: cluster 2024-04-26T16:25:26.059516+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: cluster 2024-04-26T16:25:26.059516+0000 mon.a (mon.0) 1814 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:26.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.061492+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]': finished 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.061492+0000 mon.a (mon.0) 1815 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-21", "mode": "writeback"}]': finished 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.061573+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.061573+0000 mon.a (mon.0) 1816 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: cluster 2024-04-26T16:25:26.065567+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: cluster 2024-04-26T16:25:26.065567+0000 mon.a (mon.0) 1817 : cluster [DBG] osdmap e258: 8 total, 8 up, 8 in 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.073096+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.073096+0000 mon.c (mon.2) 408 : audit [INF] from='client.? 172.21.15.50:0/3406055995' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.074498+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.50:0/1906263429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.074498+0000 mon.c (mon.2) 409 : audit [INF] from='client.? 172.21.15.50:0/1906263429' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.081155+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.081155+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.547 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.081799+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.081799+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.165776+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.165776+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.168111+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[29234]: audit 2024-04-26T16:25:26.168111+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.081155+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.081155+0000 mon.a (mon.0) 1818 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.081799+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.081799+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.165776+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.165776+0000 mon.a (mon.0) 1820 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.168111+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.549 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:26 smithi050 bash[21463]: audit 2024-04-26T16:25:26.168111+0000 mon.b (mon.1) 173 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:26.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:26.665 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:26.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:26.797 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:26.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:26.911 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:25:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:25:26.929 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:26.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:27.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:27.057 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:27.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:27.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:27.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:27.313 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:27.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:26.349239+0000 mon.c (mon.2) 410 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:26.349239+0000 mon.c (mon.2) 410 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:26.361412+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:26.361412+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.065326+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.065326+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.065540+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:27.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.065540+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.065777+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.065777+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: cluster 2024-04-26T16:25:27.073941+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: cluster 2024-04-26T16:25:27.073941+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.074007+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.074007+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.077114+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.077114+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:26.349239+0000 mon.c (mon.2) 410 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:26.349239+0000 mon.c (mon.2) 410 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:26.361412+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:26.361412+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.065326+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.065326+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.065540+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.065540+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.065777+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.065777+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: cluster 2024-04-26T16:25:27.073941+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: cluster 2024-04-26T16:25:27.073941+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.074007+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.074007+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.077114+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.077114+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.126511+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.126511+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.127242+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.127242+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.131482+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.131482+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.132229+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.132229+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.133085+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.133085+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.133472+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[29234]: audit 2024-04-26T16:25:27.133472+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.126511+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.126511+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.127242+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.127242+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.131482+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.131482+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.132229+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.132229+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.133085+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.133085+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.133472+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:27 smithi050 bash[21463]: audit 2024-04-26T16:25:27.133472+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:27.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:27.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:26.349239+0000 mon.c (mon.2) 410 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:27.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:26.349239+0000 mon.c (mon.2) 410 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:27.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:26.361412+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:26.361412+0000 mon.c (mon.2) 411 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.065326+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.065326+0000 mon.a (mon.0) 1821 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi050-73193-32"}]': finished 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.065540+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.065540+0000 mon.a (mon.0) 1822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.065777+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.065777+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: cluster 2024-04-26T16:25:27.073941+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: cluster 2024-04-26T16:25:27.073941+0000 mon.a (mon.0) 1824 : cluster [DBG] osdmap e259: 8 total, 8 up, 8 in 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.074007+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.074007+0000 mon.b (mon.1) 174 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.077114+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.077114+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.126511+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.126511+0000 mon.c (mon.2) 412 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.127242+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.127242+0000 mon.a (mon.0) 1826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.131482+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.131482+0000 mon.c (mon.2) 413 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.132229+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.132229+0000 mon.a (mon.0) 1827 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.133085+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.133085+0000 mon.c (mon.2) 414 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.133472+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:27 smithi190 bash[23868]: audit 2024-04-26T16:25:27.133472+0000 mon.a (mon.0) 1828 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:27.569 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:27.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:27.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:27.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:27.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:27.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:27.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:28.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:28.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:28.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:28.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:28.233 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:28.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:28.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:28.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:28.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:26.715212+0000 mgr.y (mgr.24401) 323 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:28.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:26.715212+0000 mgr.y (mgr.24401) 323 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:27.362798+0000 mon.c (mon.2) 415 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:27.362798+0000 mon.c (mon.2) 415 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: cluster 2024-04-26T16:25:27.554334+0000 mgr.y (mgr.24401) 324 : cluster [DBG] pgmap v373: 271 pgs: 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: cluster 2024-04-26T16:25:27.554334+0000 mgr.y (mgr.24401) 324 : cluster [DBG] pgmap v373: 271 pgs: 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.065859+0000 mon.a (mon.0) 1829 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.065859+0000 mon.a (mon.0) 1829 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.068111+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.068111+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.068312+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.068312+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.076804+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.076804+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-04-26T16:25:28.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.086546+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:28.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.086546+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:28.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.100540+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.100540+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.101355+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:28 smithi190 bash[23868]: audit 2024-04-26T16:25:28.101355+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.497 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:28.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:28.626 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:26.715212+0000 mgr.y (mgr.24401) 323 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:26.715212+0000 mgr.y (mgr.24401) 323 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:27.362798+0000 mon.c (mon.2) 415 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:27.362798+0000 mon.c (mon.2) 415 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:28.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: cluster 2024-04-26T16:25:27.554334+0000 mgr.y (mgr.24401) 324 : cluster [DBG] pgmap v373: 271 pgs: 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: cluster 2024-04-26T16:25:27.554334+0000 mgr.y (mgr.24401) 324 : cluster [DBG] pgmap v373: 271 pgs: 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.065859+0000 mon.a (mon.0) 1829 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.065859+0000 mon.a (mon.0) 1829 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.068111+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.068111+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.068312+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.068312+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.076804+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.076804+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.086546+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.086546+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.100540+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.100540+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.101355+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[21463]: audit 2024-04-26T16:25:28.101355+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:26.715212+0000 mgr.y (mgr.24401) 323 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:26.715212+0000 mgr.y (mgr.24401) 323 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:27.362798+0000 mon.c (mon.2) 415 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:27.362798+0000 mon.c (mon.2) 415 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: cluster 2024-04-26T16:25:27.554334+0000 mgr.y (mgr.24401) 324 : cluster [DBG] pgmap v373: 271 pgs: 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: cluster 2024-04-26T16:25:27.554334+0000 mgr.y (mgr.24401) 324 : cluster [DBG] pgmap v373: 271 pgs: 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-04-26T16:25:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.065859+0000 mon.a (mon.0) 1829 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.065859+0000 mon.a (mon.0) 1829 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.068111+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.068111+0000 mon.a (mon.0) 1830 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-21"}]': finished 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.068312+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.068312+0000 mon.a (mon.0) 1831 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi050-73193-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.076804+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.076804+0000 mon.a (mon.0) 1832 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.086546+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.086546+0000 mon.a (mon.0) 1833 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.100540+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.100540+0000 mon.c (mon.2) 416 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.101355+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:28 smithi050 bash[29234]: audit 2024-04-26T16:25:28.101355+0000 mon.a (mon.0) 1834 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:28.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:28.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:28.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:28.887 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:28.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:28.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:29.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:29.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:29.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:29.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:29.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:29.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:29.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:29.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:29.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: audit 2024-04-26T16:25:28.363765+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:29.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: audit 2024-04-26T16:25:28.363765+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:29.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.548403+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:29.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.548403+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.548688+0000 osd.1 (osd.1) 174 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: cluster 2024-04-26T16:25:28.548688+0000 osd.1 (osd.1) 174 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: audit 2024-04-26T16:25:29.070749+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: audit 2024-04-26T16:25:29.070749+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: cluster 2024-04-26T16:25:29.093068+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[21463]: cluster 2024-04-26T16:25:29.093068+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: audit 2024-04-26T16:25:28.363765+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: audit 2024-04-26T16:25:28.363765+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.548403+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.548403+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:29.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.548688+0000 osd.1 (osd.1) 174 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:29.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: cluster 2024-04-26T16:25:28.548688+0000 osd.1 (osd.1) 174 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:29.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: audit 2024-04-26T16:25:29.070749+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:29.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: audit 2024-04-26T16:25:29.070749+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:29.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: cluster 2024-04-26T16:25:29.093068+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-04-26T16:25:29.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:29 smithi050 bash[29234]: cluster 2024-04-26T16:25:29.093068+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-04-26T16:25:29.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:29.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: audit 2024-04-26T16:25:28.363765+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:29.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: audit 2024-04-26T16:25:28.363765+0000 mon.c (mon.2) 417 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:29.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.548403+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:29.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.548403+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:29.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.548688+0000 osd.1 (osd.1) 174 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: cluster 2024-04-26T16:25:28.548688+0000 osd.1 (osd.1) 174 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: audit 2024-04-26T16:25:29.070749+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: audit 2024-04-26T16:25:29.070749+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.50:0/2472276736' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: cluster 2024-04-26T16:25:29.093068+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-04-26T16:25:29.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:29 smithi190 bash[23868]: cluster 2024-04-26T16:25:29.093068+0000 mon.a (mon.0) 1836 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2024-04-26T16:25:29.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:29.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:29.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:29.796 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:29.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:29.929 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:29.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:30.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:30.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:30.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:30.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:30.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:30.317 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:30.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:30.448 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:30.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:30.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:29.364921+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:29.364921+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: cluster 2024-04-26T16:25:29.554963+0000 mgr.y (mgr.24401) 325 : cluster [DBG] pgmap v376: 271 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 196 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: cluster 2024-04-26T16:25:29.554963+0000 mgr.y (mgr.24401) 325 : cluster [DBG] pgmap v376: 271 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 196 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.074818+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.074818+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: cluster 2024-04-26T16:25:30.079278+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: cluster 2024-04-26T16:25:30.079278+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.084923+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.50:0/3470601054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.084923+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.50:0/3470601054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.085755+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.085755+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.089969+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.089969+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.091821+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:30 smithi190 bash[23868]: audit 2024-04-26T16:25:30.091821+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.576 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:30.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:29.364921+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:30.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:29.364921+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:30.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: cluster 2024-04-26T16:25:29.554963+0000 mgr.y (mgr.24401) 325 : cluster [DBG] pgmap v376: 271 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 196 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: cluster 2024-04-26T16:25:29.554963+0000 mgr.y (mgr.24401) 325 : cluster [DBG] pgmap v376: 271 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 196 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.074818+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.074818+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: cluster 2024-04-26T16:25:30.079278+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: cluster 2024-04-26T16:25:30.079278+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.084923+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.50:0/3470601054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.084923+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.50:0/3470601054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.085755+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.085755+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.089969+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.089969+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:29.364921+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:29.364921+0000 mon.c (mon.2) 418 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: cluster 2024-04-26T16:25:29.554963+0000 mgr.y (mgr.24401) 325 : cluster [DBG] pgmap v376: 271 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 196 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: cluster 2024-04-26T16:25:29.554963+0000 mgr.y (mgr.24401) 325 : cluster [DBG] pgmap v376: 271 pgs: 32 unknown, 32 creating+peering, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 196 active+clean; 586 KiB data, 559 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.074818+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.074818+0000 mon.a (mon.0) 1837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi050-73193-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: cluster 2024-04-26T16:25:30.079278+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: cluster 2024-04-26T16:25:30.079278+0000 mon.a (mon.0) 1838 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.084923+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.50:0/3470601054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.084923+0000 mon.c (mon.2) 419 : audit [INF] from='client.? 172.21.15.50:0/3470601054' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.085755+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.085755+0000 mon.a (mon.0) 1839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.089969+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.089969+0000 mon.a (mon.0) 1840 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.091821+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[21463]: audit 2024-04-26T16:25:30.091821+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.091821+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:30 smithi050 bash[29234]: audit 2024-04-26T16:25:30.091821+0000 mon.b (mon.1) 175 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:30.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:30.704 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:30.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:30.833 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:30.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:30.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:30.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:31.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:31.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:31.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:31.230 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:31.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:31.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: audit 2024-04-26T16:25:30.366195+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: audit 2024-04-26T16:25:30.366195+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: cluster 2024-04-26T16:25:31.060841+0000 mon.a (mon.0) 1841 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: cluster 2024-04-26T16:25:31.060841+0000 mon.a (mon.0) 1841 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: audit 2024-04-26T16:25:31.077245+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: audit 2024-04-26T16:25:31.077245+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: audit 2024-04-26T16:25:31.077376+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: audit 2024-04-26T16:25:31.077376+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: cluster 2024-04-26T16:25:31.082115+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-04-26T16:25:31.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:31 smithi190 bash[23868]: cluster 2024-04-26T16:25:31.082115+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-04-26T16:25:31.485 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:31.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: audit 2024-04-26T16:25:30.366195+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:31.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: audit 2024-04-26T16:25:30.366195+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:31.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: cluster 2024-04-26T16:25:31.060841+0000 mon.a (mon.0) 1841 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:31.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: cluster 2024-04-26T16:25:31.060841+0000 mon.a (mon.0) 1841 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:31.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: audit 2024-04-26T16:25:31.077245+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.488 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: audit 2024-04-26T16:25:31.077245+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: audit 2024-04-26T16:25:31.077376+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: audit 2024-04-26T16:25:31.077376+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: cluster 2024-04-26T16:25:31.082115+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[29234]: cluster 2024-04-26T16:25:31.082115+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: audit 2024-04-26T16:25:30.366195+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: audit 2024-04-26T16:25:30.366195+0000 mon.c (mon.2) 420 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: cluster 2024-04-26T16:25:31.060841+0000 mon.a (mon.0) 1841 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: cluster 2024-04-26T16:25:31.060841+0000 mon.a (mon.0) 1841 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: audit 2024-04-26T16:25:31.077245+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: audit 2024-04-26T16:25:31.077245+0000 mon.a (mon.0) 1842 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi050-73201-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: audit 2024-04-26T16:25:31.077376+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: audit 2024-04-26T16:25:31.077376+0000 mon.a (mon.0) 1843 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:31.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: cluster 2024-04-26T16:25:31.082115+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-04-26T16:25:31.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:31 smithi050 bash[21463]: cluster 2024-04-26T16:25:31.082115+0000 mon.a (mon.0) 1844 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2024-04-26T16:25:31.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:31.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:31.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:31.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:31.750 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:31.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:31.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:31.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:31.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:32.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:32.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:32.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:32.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:32.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:32.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:32.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:32.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:32.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:31.367763+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:32.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:31.367763+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:32.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: cluster 2024-04-26T16:25:31.556666+0000 mgr.y (mgr.24401) 326 : cluster [DBG] pgmap v379: 343 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 332 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:32.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: cluster 2024-04-26T16:25:31.556666+0000 mgr.y (mgr.24401) 326 : cluster [DBG] pgmap v379: 343 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 332 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:32.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: cluster 2024-04-26T16:25:32.088345+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-04-26T16:25:32.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: cluster 2024-04-26T16:25:32.088345+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.091687+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.091687+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.093471+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.093471+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.164695+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.164695+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.166546+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:32 smithi190 bash[23868]: audit 2024-04-26T16:25:32.166546+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:32.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:32.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:32.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:32.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:32.667 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:31.367763+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:31.367763+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: cluster 2024-04-26T16:25:31.556666+0000 mgr.y (mgr.24401) 326 : cluster [DBG] pgmap v379: 343 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 332 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: cluster 2024-04-26T16:25:31.556666+0000 mgr.y (mgr.24401) 326 : cluster [DBG] pgmap v379: 343 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 332 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: cluster 2024-04-26T16:25:32.088345+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: cluster 2024-04-26T16:25:32.088345+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.091687+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.091687+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.093471+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.093471+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.164695+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.164695+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.166546+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[21463]: audit 2024-04-26T16:25:32.166546+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:31.367763+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:31.367763+0000 mon.c (mon.2) 421 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: cluster 2024-04-26T16:25:31.556666+0000 mgr.y (mgr.24401) 326 : cluster [DBG] pgmap v379: 343 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 332 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: cluster 2024-04-26T16:25:31.556666+0000 mgr.y (mgr.24401) 326 : cluster [DBG] pgmap v379: 343 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 332 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: cluster 2024-04-26T16:25:32.088345+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-04-26T16:25:32.678 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: cluster 2024-04-26T16:25:32.088345+0000 mon.a (mon.0) 1845 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.091687+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.091687+0000 mon.c (mon.2) 422 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.093471+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.093471+0000 mon.a (mon.0) 1846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.164695+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.164695+0000 mon.a (mon.0) 1847 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.166546+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:32 smithi050 bash[29234]: audit 2024-04-26T16:25:32.166546+0000 mon.b (mon.1) 176 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:32.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:32.797 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:32.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:32.924 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:32.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:33.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:33.055 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:33.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:33.183 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:33.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:33.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:33.313 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:33.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:33.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:33.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 -- 172.21.15.50:0/1644066589 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 msgr2=0x7fac7c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 --2- 172.21.15.50:0/1644066589 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 0x7fac7c105ac0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7fac64009990 tx=0x7fac64009ca0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 -- 172.21.15.50:0/1644066589 shutdown_connections 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 --2- 172.21.15.50:0/1644066589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac7c10b640 0x7fac7c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 --2- 172.21.15.50:0/1644066589 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c106000 0x7fac7c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 --2- 172.21.15.50:0/1644066589 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 0x7fac7c105ac0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 -- 172.21.15.50:0/1644066589 >> 172.21.15.50:0/1644066589 conn(0x7fac7c101130 msgr2=0x7fac7c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 -- 172.21.15.50:0/1644066589 shutdown_connections 2024-04-26T16:25:33.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.491+0000 7fac8208f700 1 -- 172.21.15.50:0/1644066589 wait complete. 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 librados: starting msgr at 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 librados: starting objecter 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 Processor -- start 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- start start 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 librados: setting wanted keys 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 librados: calling monclient init 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 0x7fac7c19b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac7c106000 0x7fac7c19b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 0x7fac7c19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fac7c114da0 con 0x7fac7c10b640 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fac7c114c20 con 0x7fac7c106000 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fac7c114f20 con 0x7fac7c105640 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 0x7fac7c19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 0x7fac7c19fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40438/0 (socket says 172.21.15.50:40438) 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 -- 172.21.15.50:0/290728597 learned_addr learned my addr 172.21.15.50:0/290728597 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7b7fe700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 0x7fac7c19b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac72ffd700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac7c106000 0x7fac7c19b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 -- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 msgr2=0x7fac7c19b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 0x7fac7c19b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 -- 172.21.15.50:0/290728597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac7c106000 msgr2=0x7fac7c19b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac7c106000 0x7fac7c19b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 -- 172.21.15.50:0/290728597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac7c1a0320 con 0x7fac7c10b640 2024-04-26T16:25:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac7bfff700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 0x7fac7c19fbe0 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c370 tx=0x7fac7400c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:33.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 10 client.?.objecter ms_handle_connect 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7400ce20 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac7c1a05b0 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fac740090d0 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac74004690 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 10 client.30225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac7c1a7e90 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 librados: init done 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac7c06a110 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fac7401a020 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.495+0000 7fac797fa700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fac68071f80 0x7fac68074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(265..265 src has 87..265) ==== 10069+0+0 (secure 0 0 0) 0x7fac740916b0 con 0x7fac7c10b640 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac797fa700 10 client.30225.objecter ms_dispatch 0x7fac7c001550 osd_map(265..265 src has 87..265) 2024-04-26T16:25:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac797fa700 3 client.30225.objecter handle_osd_map got epochs [265,265] > 0 2024-04-26T16:25:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac797fa700 3 client.30225.objecter handle_osd_map decoding full epoch 265 2024-04-26T16:25:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac7b7fe700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fac68071f80 0x7fac68074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac797fa700 20 client.30225.objecter dump_active .. 0 homeless 2024-04-26T16:25:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac797fa700 10 client.30225.objecter ms_handle_connect 0x7fac68071f80 2024-04-26T16:25:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.499+0000 7fac7b7fe700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fac68071f80 0x7fac68074440 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7fac6c005950 tx=0x7fac6c00a300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.503+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fac74010040 con 0x7fac7c10b640 2024-04-26T16:25:33.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:33.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:32.369055+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:32.369055+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.082678+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.082678+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.082910+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.082910+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: cluster 2024-04-26T16:25:33.090756+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: cluster 2024-04-26T16:25:33.090756+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.091307+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.091307+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.091363+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.091363+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.092320+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.092320+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.094401+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:33 smithi190 bash[23868]: audit 2024-04-26T16:25:33.094401+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.567 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:33.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:32.369055+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:33.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:32.369055+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:33.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.082678+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:33.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.082678+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:33.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.082910+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.082910+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: cluster 2024-04-26T16:25:33.090756+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: cluster 2024-04-26T16:25:33.090756+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.091307+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.091307+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.091363+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.091363+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.092320+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.092320+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.094401+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[21463]: audit 2024-04-26T16:25:33.094401+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:32.369055+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:32.369055+0000 mon.c (mon.2) 423 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.082678+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.082678+0000 mon.a (mon.0) 1848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.082910+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.082910+0000 mon.a (mon.0) 1849 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: cluster 2024-04-26T16:25:33.090756+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: cluster 2024-04-26T16:25:33.090756+0000 mon.a (mon.0) 1850 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.091307+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.091307+0000 mon.c (mon.2) 424 : audit [INF] from='client.? 172.21.15.50:0/810197098' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.091363+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.091363+0000 mon.b (mon.1) 177 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.092320+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.092320+0000 mon.a (mon.0) 1851 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:33.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.094401+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:33 smithi050 bash[29234]: audit 2024-04-26T16:25:33.094401+0000 mon.a (mon.0) 1852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]: dispatch 2024-04-26T16:25:33.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:33.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:33.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.723+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fac7c19c670 con 0x7fac68071f80 2024-04-26T16:25:33.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.751+0000 7fac797fa700 1 -- 172.21.15.50:0/290728597 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+700702 (secure 0 0 0) 0x7fac7c19c670 con 0x7fac68071f80 2024-04-26T16:25:33.756 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 10 librados: watch_flush enter 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 10 librados: watch_flush exit 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 20 client.30225.objecter shutdown clearing up homeless session... 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 10 client.30225.objecter successfully canceled tick 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fac68071f80 msgr2=0x7fac68074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fac68071f80 0x7fac68074440 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7fac6c005950 tx=0x7fac6c00a300 comp rx=0 tx=0).stop 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 msgr2=0x7fac7c19fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:33.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 0x7fac7c19fbe0 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7fac7400c370 tx=0x7fac7400c680 comp rx=0 tx=0).stop 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 shutdown_connections 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fac68071f80 0x7fac68074440 unknown :-1 s=CLOSED pgs=3681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fac7c10b640 0x7fac7c19fbe0 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fac7c105640 0x7fac7c19b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 --2- 172.21.15.50:0/290728597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac7c106000 0x7fac7c19b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 >> 172.21.15.50:0/290728597 conn(0x7fac7c101130 msgr2=0x7fac7c101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 shutdown_connections 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 -- 172.21.15.50:0/290728597 wait complete. 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:33.755+0000 7fac8208f700 1 librados: shutdown 2024-04-26T16:25:33.765 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:25:33.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:33.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:33.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:33.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:33.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:34.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:34.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:34.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:34.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.225 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:34.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.358 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:34.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:34.490 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:34.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:33.370484+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:34.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:33.370484+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:34.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: cluster 2024-04-26T16:25:33.557151+0000 mgr.y (mgr.24401) 327 : cluster [DBG] pgmap v382: 271 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 260 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: cluster 2024-04-26T16:25:33.557151+0000 mgr.y (mgr.24401) 327 : cluster [DBG] pgmap v382: 271 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 260 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.085861+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.085861+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.085976+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.085976+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: cluster 2024-04-26T16:25:34.090605+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-04-26T16:25:34.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: cluster 2024-04-26T16:25:34.090605+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-04-26T16:25:34.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:33.370484+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:33.370484+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: cluster 2024-04-26T16:25:33.557151+0000 mgr.y (mgr.24401) 327 : cluster [DBG] pgmap v382: 271 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 260 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: cluster 2024-04-26T16:25:33.557151+0000 mgr.y (mgr.24401) 327 : cluster [DBG] pgmap v382: 271 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 260 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.085861+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.085861+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.085976+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.085976+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: cluster 2024-04-26T16:25:34.090605+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: cluster 2024-04-26T16:25:34.090605+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.098973+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.098973+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.101135+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.101135+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.130111+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.130111+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.131112+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.131112+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.131674+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:34 smithi190 bash[23868]: audit 2024-04-26T16:25:34.131674+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:34.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:34.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:34.757 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:34.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.878 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:25:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:25:34.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.098973+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.098973+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.101135+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.101135+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.130111+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.130111+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.131112+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.131112+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.131674+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[29234]: audit 2024-04-26T16:25:34.131674+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:34.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:33.370484+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:33.370484+0000 mon.c (mon.2) 425 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: cluster 2024-04-26T16:25:33.557151+0000 mgr.y (mgr.24401) 327 : cluster [DBG] pgmap v382: 271 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 260 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: cluster 2024-04-26T16:25:33.557151+0000 mgr.y (mgr.24401) 327 : cluster [DBG] pgmap v382: 271 pgs: 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 260 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.085861+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.085861+0000 mon.a (mon.0) 1853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.085976+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.085976+0000 mon.a (mon.0) 1854 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi050-73193-33"}]': finished 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: cluster 2024-04-26T16:25:34.090605+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: cluster 2024-04-26T16:25:34.090605+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.098973+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.098973+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.101135+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.101135+0000 mon.b (mon.1) 178 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.130111+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.130111+0000 mon.a (mon.0) 1857 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.131112+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.131112+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.131674+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:34.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:34 smithi050 bash[21463]: audit 2024-04-26T16:25:34.131674+0000 mon.a (mon.0) 1859 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:34.886 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:34.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:34.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:34.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:35.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:35.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:35.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:35.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:35.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:35.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:35.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:35.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:33.731158+0000 mgr.y (mgr.24401) 328 : audit [DBG] from='client.30225 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:33.731158+0000 mgr.y (mgr.24401) 328 : audit [DBG] from='client.30225 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:34.371643+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:34.371643+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: cluster 2024-04-26T16:25:35.087025+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: cluster 2024-04-26T16:25:35.087025+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.089718+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]': finished 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.089718+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]': finished 2024-04-26T16:25:35.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.089930+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.089930+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: cluster 2024-04-26T16:25:35.094893+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: cluster 2024-04-26T16:25:35.094893+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.096617+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.096617+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.102893+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.50:0/2632262014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.102893+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.50:0/2632262014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.103942+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:35 smithi190 bash[23868]: audit 2024-04-26T16:25:35.103942+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:35.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:33.731158+0000 mgr.y (mgr.24401) 328 : audit [DBG] from='client.30225 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:33.731158+0000 mgr.y (mgr.24401) 328 : audit [DBG] from='client.30225 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:34.371643+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:34.371643+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: cluster 2024-04-26T16:25:35.087025+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: cluster 2024-04-26T16:25:35.087025+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.089718+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]': finished 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.089718+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]': finished 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.089930+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.089930+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:35.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: cluster 2024-04-26T16:25:35.094893+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: cluster 2024-04-26T16:25:35.094893+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.096617+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.096617+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.102893+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.50:0/2632262014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.102893+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.50:0/2632262014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.103942+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[29234]: audit 2024-04-26T16:25:35.103942+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:33.731158+0000 mgr.y (mgr.24401) 328 : audit [DBG] from='client.30225 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:33.731158+0000 mgr.y (mgr.24401) 328 : audit [DBG] from='client.30225 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:34.371643+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:35.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:34.371643+0000 mon.c (mon.2) 426 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: cluster 2024-04-26T16:25:35.087025+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: cluster 2024-04-26T16:25:35.087025+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.089718+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]': finished 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.089718+0000 mon.a (mon.0) 1861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-23", "mode": "writeback"}]': finished 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.089930+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.089930+0000 mon.a (mon.0) 1862 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi050-73193-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: cluster 2024-04-26T16:25:35.094893+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: cluster 2024-04-26T16:25:35.094893+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.096617+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.096617+0000 mon.a (mon.0) 1864 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.102893+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.50:0/2632262014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.102893+0000 mon.c (mon.2) 427 : audit [INF] from='client.? 172.21.15.50:0/2632262014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.103942+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:35 smithi050 bash[21463]: audit 2024-04-26T16:25:35.103942+0000 mon.a (mon.0) 1865 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:35.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:35.668 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:35.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:35.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:35.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:35.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:35.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:35.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:36.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:36.061 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:36.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:36.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:36.288 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:36.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:36.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:36.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:36.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:35.372732+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:35.372732+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: cluster 2024-04-26T16:25:35.557986+0000 mgr.y (mgr.24401) 329 : cluster [DBG] pgmap v385: 271 pgs: 32 unknown, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: cluster 2024-04-26T16:25:35.557986+0000 mgr.y (mgr.24401) 329 : cluster [DBG] pgmap v385: 271 pgs: 32 unknown, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:36.067996+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:36.067996+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: cluster 2024-04-26T16:25:36.100462+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: cluster 2024-04-26T16:25:36.100462+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:36.163531+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:36.163531+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:36.165859+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[29234]: audit 2024-04-26T16:25:36.165859+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:35.372732+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:35.372732+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: cluster 2024-04-26T16:25:35.557986+0000 mgr.y (mgr.24401) 329 : cluster [DBG] pgmap v385: 271 pgs: 32 unknown, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: cluster 2024-04-26T16:25:35.557986+0000 mgr.y (mgr.24401) 329 : cluster [DBG] pgmap v385: 271 pgs: 32 unknown, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:36.067996+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:36.067996+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: cluster 2024-04-26T16:25:36.100462+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: cluster 2024-04-26T16:25:36.100462+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:36.163531+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:36.163531+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:36.165859+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:36 smithi050 bash[21463]: audit 2024-04-26T16:25:36.165859+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:36.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:36.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:35.372732+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:35.372732+0000 mon.c (mon.2) 428 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: cluster 2024-04-26T16:25:35.557986+0000 mgr.y (mgr.24401) 329 : cluster [DBG] pgmap v385: 271 pgs: 32 unknown, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: cluster 2024-04-26T16:25:35.557986+0000 mgr.y (mgr.24401) 329 : cluster [DBG] pgmap v385: 271 pgs: 32 unknown, 4 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 228 active+clean; 586 KiB data, 560 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:36.067996+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:36.067996+0000 mon.a (mon.0) 1866 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: cluster 2024-04-26T16:25:36.100462+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: cluster 2024-04-26T16:25:36.100462+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2024-04-26T16:25:36.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:36.163531+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:36.163531+0000 mon.a (mon.0) 1868 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:36.165859+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[23868]: audit 2024-04-26T16:25:36.165859+0000 mon.b (mon.1) 179 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:36.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:36.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:36.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:36.966 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:36.967 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:25:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:25:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:36.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:37.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:37.107 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:37.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:37.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:37.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:37.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:37.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:37.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:37.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:37.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: cluster 2024-04-26T16:25:36.354000+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:37.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: cluster 2024-04-26T16:25:36.354000+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:37.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:36.374095+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:37.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:36.374095+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:37.557 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.071888+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:37.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.071888+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:37.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.072105+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:37.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.072105+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:37.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.081111+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-04-26T16:25:37.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.081111+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-04-26T16:25:37.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.084104+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.084104+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.085253+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:37.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.085253+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:37.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.094931+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[29234]: audit 2024-04-26T16:25:37.094931+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: cluster 2024-04-26T16:25:36.354000+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:37.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: cluster 2024-04-26T16:25:36.354000+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:37.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:36.374095+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:37.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:36.374095+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:37.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.071888+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:37.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.071888+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:37.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.072105+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:37.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.072105+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:37.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.081111+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-04-26T16:25:37.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.081111+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-04-26T16:25:37.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.084104+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.084104+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.085253+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:37.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.085253+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:37.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.094931+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:37 smithi050 bash[21463]: audit 2024-04-26T16:25:37.094931+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:37.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:37.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: cluster 2024-04-26T16:25:36.354000+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: cluster 2024-04-26T16:25:36.354000+0000 mon.a (mon.0) 1869 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:36.374095+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:36.374095+0000 mon.c (mon.2) 429 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.071888+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.071888+0000 mon.a (mon.0) 1870 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi050-73193-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.072105+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.072105+0000 mon.a (mon.0) 1871 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.081111+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-04-26T16:25:37.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.081111+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2024-04-26T16:25:37.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.084104+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.084104+0000 mon.b (mon.1) 180 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.085253+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:37.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.085253+0000 mon.a (mon.0) 1873 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:37.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.094931+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:37 smithi190 bash[23868]: audit 2024-04-26T16:25:37.094931+0000 mon.a (mon.0) 1874 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]: dispatch 2024-04-26T16:25:37.814 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:37.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:37.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:37.944 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:37.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:37.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:38.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:38.077 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 -- 172.21.15.50:0/899782783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc104330 msgr2=0x7fd9bc104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/899782783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc104330 0x7fd9bc104710 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009f90 tx=0x7fd9b400b330 comp rx=0 tx=0).stop 2024-04-26T16:25:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 -- 172.21.15.50:0/899782783 shutdown_connections 2024-04-26T16:25:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/899782783 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc1024b0 0x7fd9bc1121a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/899782783 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc104330 0x7fd9bc104710 secure :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4009f90 tx=0x7fd9b400b330 comp rx=0 tx=0).stop 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/899782783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9bc101a10 0x7fd9bc101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.127+0000 7fd9c32a4700 1 -- 172.21.15.50:0/899782783 >> 172.21.15.50:0/899782783 conn(0x7fd9bc078990 msgr2=0x7fd9bc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- 172.21.15.50:0/899782783 shutdown_connections 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- 172.21.15.50:0/899782783 wait complete. 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 librados: starting msgr at 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 librados: starting objecter 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 Processor -- start 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- start start 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 librados: setting wanted keys 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 librados: calling monclient init 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9bc101a10 0x7fd9bc19b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc1024b0 0x7fd9bc19bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 0x7fd9bc1a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc114e00 con 0x7fd9bc19fec0 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc114c80 con 0x7fd9bc101a10 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd9bc114f80 con 0x7fd9bc1024b0 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c1841700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 0x7fd9bc1a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc1024b0 0x7fd9bc19bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c1841700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 0x7fd9bc1a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58532/0 (socket says 172.21.15.50:58532) 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c1841700 1 -- 172.21.15.50:0/2883211812 learned_addr learned my addr 172.21.15.50:0/2883211812 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c1040700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9bc101a10 0x7fd9bc19b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 -- 172.21.15.50:0/2883211812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9bc101a10 msgr2=0x7fd9bc19b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9bc101a10 0x7fd9bc19b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 -- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 msgr2=0x7fd9bc1a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:38.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 0x7fd9bc1a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 -- 172.21.15.50:0/2883211812 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc1a09e0 con 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c1841700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 0x7fd9bc1a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c083f700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc1024b0 0x7fd9bc19bb30 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c970 tx=0x7fd9ac00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9b27fc700 10 client.?.objecter ms_handle_connect 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9b27fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac004a30 con 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc1a0cd0 con 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd9ac005480 con 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac00e940 con 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 10 client.40859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc1a8500 con 0x7fd9bc1024b0 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.131+0000 7fd9c32a4700 1 librados: init done 2024-04-26T16:25:38.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc102a20 con 0x7fd9bc1024b0 2024-04-26T16:25:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd9ac005000 con 0x7fd9bc1024b0 2024-04-26T16:25:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd9a8071fc0 0x7fd9a8074480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(270..270 src has 87..270) ==== 11203+0+0 (secure 0 0 0) 0x7fd9ac091b40 con 0x7fd9bc1024b0 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 10 client.40859.objecter ms_dispatch 0x7fd9bc001550 osd_map(270..270 src has 87..270) 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 3 client.40859.objecter handle_osd_map got epochs [270,270] > 0 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 3 client.40859.objecter handle_osd_map decoding full epoch 270 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9c1040700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd9a8071fc0 0x7fd9a8074480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 20 client.40859.objecter dump_active .. 0 homeless 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9b27fc700 10 client.40859.objecter ms_handle_connect 0x7fd9a8071fc0 2024-04-26T16:25:38.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.135+0000 7fd9c1040700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd9a8071fc0 0x7fd9a8074480 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400d010 tx=0x7fd9b400b680 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.139+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd9ac014030 con 0x7fd9bc1024b0 2024-04-26T16:25:38.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:38.194 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-04-26T16:25:33.556848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600231,"num_objects":363,"num_object_clones":33,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":363,"num_whiteouts":0,"num_read":1507,"num_read_kb":1265,"num_write":1278,"num_write_kb":3063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":1772384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2406,"ondisk_log_size":2406,"up":813,"acting":813,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":839,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":573144,"kb_used_data":34928,"kb_used_omap":12,"kb_used_meta":538035,"kb_avail":749388072,"statfs":{"total":767960285184,"available":767373385728,"internally_reserved":0,"allocated":35766272,"data_stored":20802044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12748,"internal_metadata":550948404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-60,"num_objects":-15,"num_object_clones":-20,"num_object_copies":-45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.002272"},"pg_stats":[{"pgid":"237.10","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.146142+0000","last_change":"2024-04-26T16:25:31.143805+0000","last_active":"2024-04-26T16:25:31.146142+0000","last_peered":"2024-04-26T16:25:31.146142+0000","last_clean":"2024-04-26T16:25:31.146142+0000","last_became_active":"2024-04-26T16:25:31.143266+0000","last_became_peered":"2024-04-26T16:25:31.143266+0000","last_unstale":"2024-04-26T16:25:31.146142+0000","last_undegraded":"2024-04-26T16:25:31.146142+0000","last_fullsized":"2024-04-26T16:25:31.146142+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:32.236969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.17","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.127990+0000","last_change":"2024-04-26T16:25:27.128785+0000","last_active":"2024-04-26T16:25:31.127990+0000","last_peered":"2024-04-26T16:25:31.127990+0000","last_clean":"2024-04-26T16:25:31.127990+0000","last_became_active":"2024-04-26T16:25:27.128430+0000","last_became_peered":"2024-04-26T16:25:27.128430+0000","last_unstale":"2024-04-26T16:25:31.127990+0000","last_undegraded":"2024-04-26T16:25:31.127990+0000","last_fullsized":"2024-04-26T16:25:31.127990+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.011611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.11","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.151630+0000","last_change":"2024-04-26T16:25:31.139589+0000","last_active":"2024-04-26T16:25:31.151630+0000","last_peered":"2024-04-26T16:25:31.151630+0000","last_clean":"2024-04-26T16:25:31.151630+0000","last_became_active":"2024-04-26T16:25:31.138850+0000","last_became_peered":"2024-04-26T16:25:31.138850+0000","last_unstale":"2024-04-26T16:25:31.151630+0000","last_undegraded":"2024-04-26T16:25:31.151630+0000","last_fullsized":"2024-04-26T16:25:31.151630+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:54.243552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.16","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.106352+0000","last_change":"2024-04-26T16:25:27.128662+0000","last_active":"2024-04-26T16:25:31.106352+0000","last_peered":"2024-04-26T16:25:31.106352+0000","last_clean":"2024-04-26T16:25:31.106352+0000","last_became_active":"2024-04-26T16:25:27.128443+0000","last_became_peered":"2024-04-26T16:25:27.128443+0000","last_unstale":"2024-04-26T16:25:31.106352+0000","last_undegraded":"2024-04-26T16:25:31.106352+0000","last_fullsized":"2024-04-26T16:25:31.106352+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:35.087548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.12","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.123579+0000","last_change":"2024-04-26T16:25:31.122811+0000","last_active":"2024-04-26T16:25:31.123579+0000","last_peered":"2024-04-26T16:25:31.123579+0000","last_clean":"2024-04-26T16:25:31.123579+0000","last_became_active":"2024-04-26T16:25:31.122512+0000","last_became_peered":"2024-04-26T16:25:31.122512+0000","last_unstale":"2024-04-26T16:25:31.123579+0000","last_undegraded":"2024-04-26T16:25:31.123579+0000","last_fullsized":"2024-04-26T16:25:31.123579+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:31.091331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.15","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.127682+0000","last_change":"2024-04-26T16:25:27.117860+0000","last_active":"2024-04-26T16:25:31.127682+0000","last_peered":"2024-04-26T16:25:31.127682+0000","last_clean":"2024-04-26T16:25:31.127682+0000","last_became_active":"2024-04-26T16:25:27.117490+0000","last_became_peered":"2024-04-26T16:25:27.117490+0000","last_unstale":"2024-04-26T16:25:31.127682+0000","last_undegraded":"2024-04-26T16:25:31.127682+0000","last_fullsized":"2024-04-26T16:25:31.127682+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:47.753345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.13","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.156005+0000","last_change":"2024-04-26T16:25:31.151868+0000","last_active":"2024-04-26T16:25:31.156005+0000","last_peered":"2024-04-26T16:25:31.156005+0000","last_clean":"2024-04-26T16:25:31.156005+0000","last_became_active":"2024-04-26T16:25:31.151339+0000","last_became_peered":"2024-04-26T16:25:31.151339+0000","last_unstale":"2024-04-26T16:25:31.156005+0000","last_undegraded":"2024-04-26T16:25:31.156005+0000","last_fullsized":"2024-04-26T16:25:31.156005+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:48.247561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.14","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083676+0000","last_change":"2024-04-26T16:25:27.122672+0000","last_active":"2024-04-26T16:25:31.083676+0000","last_peered":"2024-04-26T16:25:31.083676+0000","last_clean":"2024-04-26T16:25:31.083676+0000","last_became_active":"2024-04-26T16:25:27.121735+0000","last_became_peered":"2024-04-26T16:25:27.121735+0000","last_unstale":"2024-04-26T16:25:31.083676+0000","last_undegraded":"2024-04-26T16:25:31.083676+0000","last_fullsized":"2024-04-26T16:25:31.083676+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:27.129030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.14","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.142660+0000","last_change":"2024-04-26T16:25:31.139910+0000","last_active":"2024-04-26T16:25:31.142660+0000","last_peered":"2024-04-26T16:25:31.142660+0000","last_clean":"2024-04-26T16:25:31.142660+0000","last_became_active":"2024-04-26T16:25:31.139625+0000","last_became_peered":"2024-04-26T16:25:31.139625+0000","last_unstale":"2024-04-26T16:25:31.142660+0000","last_undegraded":"2024-04-26T16:25:31.142660+0000","last_fullsized":"2024-04-26T16:25:31.142660+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:52.156445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.13","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.099915+0000","last_change":"2024-04-26T16:25:27.128321+0000","last_active":"2024-04-26T16:25:31.099915+0000","last_peered":"2024-04-26T16:25:31.099915+0000","last_clean":"2024-04-26T16:25:31.099915+0000","last_became_active":"2024-04-26T16:25:27.123039+0000","last_became_peered":"2024-04-26T16:25:27.123039+0000","last_unstale":"2024-04-26T16:25:31.099915+0000","last_undegraded":"2024-04-26T16:25:31.099915+0000","last_fullsized":"2024-04-26T16:25:31.099915+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:43.705194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.15","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.141101+0000","last_change":"2024-04-26T16:25:31.140700+0000","last_active":"2024-04-26T16:25:31.141101+0000","last_peered":"2024-04-26T16:25:31.141101+0000","last_clean":"2024-04-26T16:25:31.141101+0000","last_became_active":"2024-04-26T16:25:31.140394+0000","last_became_peered":"2024-04-26T16:25:31.140394+0000","last_unstale":"2024-04-26T16:25:31.141101+0000","last_undegraded":"2024-04-26T16:25:31.141101+0000","last_fullsized":"2024-04-26T16:25:31.141101+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:42.185052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.12","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.121927+0000","last_change":"2024-04-26T16:25:27.094058+0000","last_active":"2024-04-26T16:25:31.121927+0000","last_peered":"2024-04-26T16:25:31.121927+0000","last_clean":"2024-04-26T16:25:31.121927+0000","last_became_active":"2024-04-26T16:25:27.093730+0000","last_became_peered":"2024-04-26T16:25:27.093730+0000","last_unstale":"2024-04-26T16:25:31.121927+0000","last_undegraded":"2024-04-26T16:25:31.121927+0000","last_fullsized":"2024-04-26T16:25:31.121927+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:36.783752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.16","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.153450+0000","last_change":"2024-04-26T16:25:31.152231+0000","last_active":"2024-04-26T16:25:31.153450+0000","last_peered":"2024-04-26T16:25:31.153450+0000","last_clean":"2024-04-26T16:25:31.153450+0000","last_became_active":"2024-04-26T16:25:31.151927+0000","last_became_peered":"2024-04-26T16:25:31.151927+0000","last_unstale":"2024-04-26T16:25:31.153450+0000","last_undegraded":"2024-04-26T16:25:31.153450+0000","last_fullsized":"2024-04-26T16:25:31.153450+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:55.704159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.11","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094528+0000","last_change":"2024-04-26T16:25:27.110521+0000","last_active":"2024-04-26T16:25:31.094528+0000","last_peered":"2024-04-26T16:25:31.094528+0000","last_clean":"2024-04-26T16:25:31.094528+0000","last_became_active":"2024-04-26T16:25:27.109863+0000","last_became_peered":"2024-04-26T16:25:27.109863+0000","last_unstale":"2024-04-26T16:25:31.094528+0000","last_undegraded":"2024-04-26T16:25:31.094528+0000","last_fullsized":"2024-04-26T16:25:31.094528+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:43.017076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.17","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.164194+0000","last_change":"2024-04-26T16:25:31.163563+0000","last_active":"2024-04-26T16:25:31.164194+0000","last_peered":"2024-04-26T16:25:31.164194+0000","last_clean":"2024-04-26T16:25:31.164194+0000","last_became_active":"2024-04-26T16:25:31.162295+0000","last_became_peered":"2024-04-26T16:25:31.162295+0000","last_unstale":"2024-04-26T16:25:31.164194+0000","last_undegraded":"2024-04-26T16:25:31.164194+0000","last_fullsized":"2024-04-26T16:25:31.164194+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:50.488385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.10","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083162+0000","last_change":"2024-04-26T16:25:27.114457+0000","last_active":"2024-04-26T16:25:31.083162+0000","last_peered":"2024-04-26T16:25:31.083162+0000","last_clean":"2024-04-26T16:25:31.083162+0000","last_became_active":"2024-04-26T16:25:27.114052+0000","last_became_peered":"2024-04-26T16:25:27.114052+0000","last_unstale":"2024-04-26T16:25:31.083162+0000","last_undegraded":"2024-04-26T16:25:31.083162+0000","last_fullsized":"2024-04-26T16:25:31.083162+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:40.231198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.18","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.162952+0000","last_change":"2024-04-26T16:25:31.162332+0000","last_active":"2024-04-26T16:25:31.162952+0000","last_peered":"2024-04-26T16:25:31.162952+0000","last_clean":"2024-04-26T16:25:31.162952+0000","last_became_active":"2024-04-26T16:25:31.159915+0000","last_became_peered":"2024-04-26T16:25:31.159915+0000","last_unstale":"2024-04-26T16:25:31.162952+0000","last_undegraded":"2024-04-26T16:25:31.162952+0000","last_fullsized":"2024-04-26T16:25:31.162952+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:59.671383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.1f","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.127752+0000","last_change":"2024-04-26T16:25:27.106037+0000","last_active":"2024-04-26T16:25:31.127752+0000","last_peered":"2024-04-26T16:25:31.127752+0000","last_clean":"2024-04-26T16:25:31.127752+0000","last_became_active":"2024-04-26T16:25:27.105714+0000","last_became_peered":"2024-04-26T16:25:27.105714+0000","last_unstale":"2024-04-26T16:25:31.127752+0000","last_undegraded":"2024-04-26T16:25:31.127752+0000","last_fullsized":"2024-04-26T16:25:31.127752+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:38.780168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.19","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.151721+0000","last_change":"2024-04-26T16:25:31.147825+0000","last_active":"2024-04-26T16:25:31.151721+0000","last_peered":"2024-04-26T16:25:31.151721+0000","last_clean":"2024-04-26T16:25:31.151721+0000","last_became_active":"2024-04-26T16:25:31.147022+0000","last_became_peered":"2024-04-26T16:25:31.147022+0000","last_unstale":"2024-04-26T16:25:31.151721+0000","last_undegraded":"2024-04-26T16:25:31.151721+0000","last_fullsized":"2024-04-26T16:25:31.151721+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.970405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.1e","version":"261'2","reported_seq":22,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103012+0000","last_change":"2024-04-26T16:25:27.111838+0000","last_active":"2024-04-26T16:25:31.103012+0000","last_peered":"2024-04-26T16:25:31.103012+0000","last_clean":"2024-04-26T16:25:31.103012+0000","last_became_active":"2024-04-26T16:25:27.110985+0000","last_became_peered":"2024-04-26T16:25:27.110985+0000","last_unstale":"2024-04-26T16:25:31.103012+0000","last_undegraded":"2024-04-26T16:25:31.103012+0000","last_fullsized":"2024-04-26T16:25:31.103012+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:26.472807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.1a","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.154331+0000","last_change":"2024-04-26T16:25:31.153729+0000","last_active":"2024-04-26T16:25:31.154331+0000","last_peered":"2024-04-26T16:25:31.154331+0000","last_clean":"2024-04-26T16:25:31.154331+0000","last_became_active":"2024-04-26T16:25:31.153163+0000","last_became_peered":"2024-04-26T16:25:31.153163+0000","last_unstale":"2024-04-26T16:25:31.154331+0000","last_undegraded":"2024-04-26T16:25:31.154331+0000","last_fullsized":"2024-04-26T16:25:31.154331+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:50.615603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.1d","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.106354+0000","last_change":"2024-04-26T16:25:27.111208+0000","last_active":"2024-04-26T16:25:31.106354+0000","last_peered":"2024-04-26T16:25:31.106354+0000","last_clean":"2024-04-26T16:25:31.106354+0000","last_became_active":"2024-04-26T16:25:27.098245+0000","last_became_peered":"2024-04-26T16:25:27.098245+0000","last_unstale":"2024-04-26T16:25:31.106354+0000","last_undegraded":"2024-04-26T16:25:31.106354+0000","last_fullsized":"2024-04-26T16:25:31.106354+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.955125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.1b","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.119604+0000","last_change":"2024-04-26T16:25:31.119331+0000","last_active":"2024-04-26T16:25:31.119604+0000","last_peered":"2024-04-26T16:25:31.119604+0000","last_clean":"2024-04-26T16:25:31.119604+0000","last_became_active":"2024-04-26T16:25:31.118808+0000","last_became_peered":"2024-04-26T16:25:31.118808+0000","last_unstale":"2024-04-26T16:25:31.119604+0000","last_undegraded":"2024-04-26T16:25:31.119604+0000","last_fullsized":"2024-04-26T16:25:31.119604+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:32.169957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.1c","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084552+0000","last_change":"2024-04-26T16:25:27.122400+0000","last_active":"2024-04-26T16:25:31.084552+0000","last_peered":"2024-04-26T16:25:31.084552+0000","last_clean":"2024-04-26T16:25:31.084552+0000","last_became_active":"2024-04-26T16:25:27.121270+0000","last_became_peered":"2024-04-26T16:25:27.121270+0000","last_unstale":"2024-04-26T16:25:31.084552+0000","last_undegraded":"2024-04-26T16:25:31.084552+0000","last_fullsized":"2024-04-26T16:25:31.084552+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:49.821196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.1c","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.113611+0000","last_change":"2024-04-26T16:25:31.112851+0000","last_active":"2024-04-26T16:25:31.113611+0000","last_peered":"2024-04-26T16:25:31.113611+0000","last_clean":"2024-04-26T16:25:31.113611+0000","last_became_active":"2024-04-26T16:25:31.112289+0000","last_became_peered":"2024-04-26T16:25:31.112289+0000","last_unstale":"2024-04-26T16:25:31.113611+0000","last_undegraded":"2024-04-26T16:25:31.113611+0000","last_fullsized":"2024-04-26T16:25:31.113611+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:54.605351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.1b","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.130590+0000","last_change":"2024-04-26T16:25:27.107715+0000","last_active":"2024-04-26T16:25:31.130590+0000","last_peered":"2024-04-26T16:25:31.130590+0000","last_clean":"2024-04-26T16:25:31.130590+0000","last_became_active":"2024-04-26T16:25:27.107386+0000","last_became_peered":"2024-04-26T16:25:27.107386+0000","last_unstale":"2024-04-26T16:25:31.130590+0000","last_undegraded":"2024-04-26T16:25:31.130590+0000","last_fullsized":"2024-04-26T16:25:31.130590+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:26.262074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.1d","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.151012+0000","last_change":"2024-04-26T16:25:31.150446+0000","last_active":"2024-04-26T16:25:31.151012+0000","last_peered":"2024-04-26T16:25:31.151012+0000","last_clean":"2024-04-26T16:25:31.151012+0000","last_became_active":"2024-04-26T16:25:31.150086+0000","last_became_peered":"2024-04-26T16:25:31.150086+0000","last_unstale":"2024-04-26T16:25:31.151012+0000","last_undegraded":"2024-04-26T16:25:31.151012+0000","last_fullsized":"2024-04-26T16:25:31.151012+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.132086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.1a","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.082053+0000","last_change":"2024-04-26T16:25:27.121647+0000","last_active":"2024-04-26T16:25:31.082053+0000","last_peered":"2024-04-26T16:25:31.082053+0000","last_clean":"2024-04-26T16:25:31.082053+0000","last_became_active":"2024-04-26T16:25:27.120888+0000","last_became_peered":"2024-04-26T16:25:27.120888+0000","last_unstale":"2024-04-26T16:25:31.082053+0000","last_undegraded":"2024-04-26T16:25:31.082053+0000","last_fullsized":"2024-04-26T16:25:31.082053+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:45.566931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.1e","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.155431+0000","last_change":"2024-04-26T16:25:31.154142+0000","last_active":"2024-04-26T16:25:31.155431+0000","last_peered":"2024-04-26T16:25:31.155431+0000","last_clean":"2024-04-26T16:25:31.155431+0000","last_became_active":"2024-04-26T16:25:31.153661+0000","last_became_peered":"2024-04-26T16:25:31.153661+0000","last_unstale":"2024-04-26T16:25:31.155431+0000","last_undegraded":"2024-04-26T16:25:31.155431+0000","last_fullsized":"2024-04-26T16:25:31.155431+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:38.581147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.19","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088656+0000","last_change":"2024-04-26T16:25:27.127399+0000","last_active":"2024-04-26T16:25:31.088656+0000","last_peered":"2024-04-26T16:25:31.088656+0000","last_clean":"2024-04-26T16:25:31.088656+0000","last_became_active":"2024-04-26T16:25:27.126957+0000","last_became_peered":"2024-04-26T16:25:27.126957+0000","last_unstale":"2024-04-26T16:25:31.088656+0000","last_undegraded":"2024-04-26T16:25:31.088656+0000","last_fullsized":"2024-04-26T16:25:31.088656+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.515391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.1f","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.154207+0000","last_change":"2024-04-26T16:25:31.152866+0000","last_active":"2024-04-26T16:25:31.154207+0000","last_peered":"2024-04-26T16:25:31.154207+0000","last_clean":"2024-04-26T16:25:31.154207+0000","last_became_active":"2024-04-26T16:25:31.151574+0000","last_became_peered":"2024-04-26T16:25:31.151574+0000","last_unstale":"2024-04-26T16:25:31.154207+0000","last_undegraded":"2024-04-26T16:25:31.154207+0000","last_fullsized":"2024-04-26T16:25:31.154207+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:39.346454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.18","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100263+0000","last_change":"2024-04-26T16:25:27.128680+0000","last_active":"2024-04-26T16:25:31.100263+0000","last_peered":"2024-04-26T16:25:31.100263+0000","last_clean":"2024-04-26T16:25:31.100263+0000","last_became_active":"2024-04-26T16:25:27.128321+0000","last_became_peered":"2024-04-26T16:25:27.128321+0000","last_unstale":"2024-04-26T16:25:31.100263+0000","last_undegraded":"2024-04-26T16:25:31.100263+0000","last_fullsized":"2024-04-26T16:25:31.100263+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:26.915821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.0","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.142372+0000","last_change":"2024-04-26T16:25:31.139785+0000","last_active":"2024-04-26T16:25:31.142372+0000","last_peered":"2024-04-26T16:25:31.142372+0000","last_clean":"2024-04-26T16:25:31.142372+0000","last_became_active":"2024-04-26T16:25:31.139458+0000","last_became_peered":"2024-04-26T16:25:31.139458+0000","last_unstale":"2024-04-26T16:25:31.142372+0000","last_undegraded":"2024-04-26T16:25:31.142372+0000","last_fullsized":"2024-04-26T16:25:31.142372+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:39.355914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.7","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.148891+0000","last_change":"2024-04-26T16:25:27.093871+0000","last_active":"2024-04-26T16:25:31.148891+0000","last_peered":"2024-04-26T16:25:31.148891+0000","last_clean":"2024-04-26T16:25:31.148891+0000","last_became_active":"2024-04-26T16:25:27.093562+0000","last_became_peered":"2024-04-26T16:25:27.093562+0000","last_unstale":"2024-04-26T16:25:31.148891+0000","last_undegraded":"2024-04-26T16:25:31.148891+0000","last_fullsized":"2024-04-26T16:25:31.148891+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:45.244479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"237.1","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.138389+0000","last_change":"2024-04-26T16:25:31.135700+0000","last_active":"2024-04-26T16:25:31.138389+0000","last_peered":"2024-04-26T16:25:31.138389+0000","last_clean":"2024-04-26T16:25:31.138389+0000","last_became_active":"2024-04-26T16:25:31.135207+0000","last_became_peered":"2024-04-26T16:25:31.135207+0000","last_unstale":"2024-04-26T16:25:31.138389+0000","last_undegraded":"2024-04-26T16:25:31.138389+0000","last_fullsized":"2024-04-26T16:25:31.138389+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:49.408446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.6","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.125362+0000","last_change":"2024-04-26T16:25:27.128796+0000","last_active":"2024-04-26T16:25:31.125362+0000","last_peered":"2024-04-26T16:25:31.125362+0000","last_clean":"2024-04-26T16:25:31.125362+0000","last_became_active":"2024-04-26T16:25:27.128457+0000","last_became_peered":"2024-04-26T16:25:27.128457+0000","last_unstale":"2024-04-26T16:25:31.125362+0000","last_undegraded":"2024-04-26T16:25:31.125362+0000","last_fullsized":"2024-04-26T16:25:31.125362+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:42.010916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.2","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.154815+0000","last_change":"2024-04-26T16:25:31.152518+0000","last_active":"2024-04-26T16:25:31.154815+0000","last_peered":"2024-04-26T16:25:31.154815+0000","last_clean":"2024-04-26T16:25:31.154815+0000","last_became_active":"2024-04-26T16:25:31.152172+0000","last_became_peered":"2024-04-26T16:25:31.152172+0000","last_unstale":"2024-04-26T16:25:31.154815+0000","last_undegraded":"2024-04-26T16:25:31.154815+0000","last_fullsized":"2024-04-26T16:25:31.154815+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:49.842342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.5","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084232+0000","last_change":"2024-04-26T16:25:27.132638+0000","last_active":"2024-04-26T16:25:31.084232+0000","last_peered":"2024-04-26T16:25:31.084232+0000","last_clean":"2024-04-26T16:25:31.084232+0000","last_became_active":"2024-04-26T16:25:27.132315+0000","last_became_peered":"2024-04-26T16:25:27.132315+0000","last_unstale":"2024-04-26T16:25:31.084232+0000","last_undegraded":"2024-04-26T16:25:31.084232+0000","last_fullsized":"2024-04-26T16:25:31.084232+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:26.801209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"237.3","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.108588+0000","last_change":"2024-04-26T16:25:31.108062+0000","last_active":"2024-04-26T16:25:31.108588+0000","last_peered":"2024-04-26T16:25:31.108588+0000","last_clean":"2024-04-26T16:25:31.108588+0000","last_became_active":"2024-04-26T16:25:31.107738+0000","last_became_peered":"2024-04-26T16:25:31.107738+0000","last_unstale":"2024-04-26T16:25:31.108588+0000","last_undegraded":"2024-04-26T16:25:31.108588+0000","last_fullsized":"2024-04-26T16:25:31.108588+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.670905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.4","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.101261+0000","last_change":"2024-04-26T16:25:27.122270+0000","last_active":"2024-04-26T16:25:31.101261+0000","last_peered":"2024-04-26T16:25:31.101261+0000","last_clean":"2024-04-26T16:25:31.101261+0000","last_became_active":"2024-04-26T16:25:27.121902+0000","last_became_peered":"2024-04-26T16:25:27.121902+0000","last_unstale":"2024-04-26T16:25:31.101261+0000","last_undegraded":"2024-04-26T16:25:31.101261+0000","last_fullsized":"2024-04-26T16:25:31.101261+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:49.407526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.4","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.142641+0000","last_change":"2024-04-26T16:25:31.139813+0000","last_active":"2024-04-26T16:25:31.142641+0000","last_peered":"2024-04-26T16:25:31.142641+0000","last_clean":"2024-04-26T16:25:31.142641+0000","last_became_active":"2024-04-26T16:25:31.139514+0000","last_became_peered":"2024-04-26T16:25:31.139514+0000","last_unstale":"2024-04-26T16:25:31.142641+0000","last_undegraded":"2024-04-26T16:25:31.142641+0000","last_fullsized":"2024-04-26T16:25:31.142641+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:37.176479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"234.3","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.130059+0000","last_change":"2024-04-26T16:25:27.105984+0000","last_active":"2024-04-26T16:25:31.130059+0000","last_peered":"2024-04-26T16:25:31.130059+0000","last_clean":"2024-04-26T16:25:31.130059+0000","last_became_active":"2024-04-26T16:25:27.105617+0000","last_became_peered":"2024-04-26T16:25:27.105617+0000","last_unstale":"2024-04-26T16:25:31.130059+0000","last_undegraded":"2024-04-26T16:25:31.130059+0000","last_fullsized":"2024-04-26T16:25:31.130059+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:29.029933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.5","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.163987+0000","last_change":"2024-04-26T16:25:31.163549+0000","last_active":"2024-04-26T16:25:31.163987+0000","last_peered":"2024-04-26T16:25:31.163987+0000","last_clean":"2024-04-26T16:25:31.163987+0000","last_became_active":"2024-04-26T16:25:31.163271+0000","last_became_peered":"2024-04-26T16:25:31.163271+0000","last_unstale":"2024-04-26T16:25:31.163987+0000","last_undegraded":"2024-04-26T16:25:31.163987+0000","last_fullsized":"2024-04-26T16:25:31.163987+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:50.712138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.2","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.125395+0000","last_change":"2024-04-26T16:25:27.127759+0000","last_active":"2024-04-26T16:25:31.125395+0000","last_peered":"2024-04-26T16:25:31.125395+0000","last_clean":"2024-04-26T16:25:31.125395+0000","last_became_active":"2024-04-26T16:25:27.127397+0000","last_became_peered":"2024-04-26T16:25:27.127397+0000","last_unstale":"2024-04-26T16:25:31.125395+0000","last_undegraded":"2024-04-26T16:25:31.125395+0000","last_fullsized":"2024-04-26T16:25:31.125395+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:46.258302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"237.6","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.138713+0000","last_change":"2024-04-26T16:25:31.137721+0000","last_active":"2024-04-26T16:25:31.138713+0000","last_peered":"2024-04-26T16:25:31.138713+0000","last_clean":"2024-04-26T16:25:31.138713+0000","last_became_active":"2024-04-26T16:25:31.136904+0000","last_became_peered":"2024-04-26T16:25:31.136904+0000","last_unstale":"2024-04-26T16:25:31.138713+0000","last_undegraded":"2024-04-26T16:25:31.138713+0000","last_fullsized":"2024-04-26T16:25:31.138713+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:41.532131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.1","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095125+0000","last_change":"2024-04-26T16:25:27.122297+0000","last_active":"2024-04-26T16:25:31.095125+0000","last_peered":"2024-04-26T16:25:31.095125+0000","last_clean":"2024-04-26T16:25:31.095125+0000","last_became_active":"2024-04-26T16:25:27.121964+0000","last_became_peered":"2024-04-26T16:25:27.121964+0000","last_unstale":"2024-04-26T16:25:31.095125+0000","last_undegraded":"2024-04-26T16:25:31.095125+0000","last_fullsized":"2024-04-26T16:25:31.095125+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:52.510785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.7","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.131446+0000","last_change":"2024-04-26T16:25:31.131174+0000","last_active":"2024-04-26T16:25:31.131446+0000","last_peered":"2024-04-26T16:25:31.131446+0000","last_clean":"2024-04-26T16:25:31.131446+0000","last_became_active":"2024-04-26T16:25:31.130841+0000","last_became_peered":"2024-04-26T16:25:31.130841+0000","last_unstale":"2024-04-26T16:25:31.131446+0000","last_undegraded":"2024-04-26T16:25:31.131446+0000","last_fullsized":"2024-04-26T16:25:31.131446+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:41.044362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.0","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.087789+0000","last_change":"2024-04-26T16:25:27.089265+0000","last_active":"2024-04-26T16:25:31.087789+0000","last_peered":"2024-04-26T16:25:31.087789+0000","last_clean":"2024-04-26T16:25:31.087789+0000","last_became_active":"2024-04-26T16:25:27.088933+0000","last_became_peered":"2024-04-26T16:25:27.088933+0000","last_unstale":"2024-04-26T16:25:31.087789+0000","last_undegraded":"2024-04-26T16:25:31.087789+0000","last_fullsized":"2024-04-26T16:25:31.087789+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.828293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"237.8","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103487+0000","last_change":"2024-04-26T16:25:31.100880+0000","last_active":"2024-04-26T16:25:31.103487+0000","last_peered":"2024-04-26T16:25:31.103487+0000","last_clean":"2024-04-26T16:25:31.103487+0000","last_became_active":"2024-04-26T16:25:31.100479+0000","last_became_peered":"2024-04-26T16:25:31.100479+0000","last_unstale":"2024-04-26T16:25:31.103487+0000","last_undegraded":"2024-04-26T16:25:31.103487+0000","last_fullsized":"2024-04-26T16:25:31.103487+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:55.479538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.f","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109549+0000","last_change":"2024-04-26T16:25:27.132188+0000","last_active":"2024-04-26T16:25:31.109549+0000","last_peered":"2024-04-26T16:25:31.109549+0000","last_clean":"2024-04-26T16:25:31.109549+0000","last_became_active":"2024-04-26T16:25:27.131825+0000","last_became_peered":"2024-04-26T16:25:27.131825+0000","last_unstale":"2024-04-26T16:25:31.109549+0000","last_undegraded":"2024-04-26T16:25:31.109549+0000","last_fullsized":"2024-04-26T16:25:31.109549+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.599548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.9","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.162900+0000","last_change":"2024-04-26T16:25:31.162197+0000","last_active":"2024-04-26T16:25:31.162900+0000","last_peered":"2024-04-26T16:25:31.162900+0000","last_clean":"2024-04-26T16:25:31.162900+0000","last_became_active":"2024-04-26T16:25:31.160119+0000","last_became_peered":"2024-04-26T16:25:31.160119+0000","last_unstale":"2024-04-26T16:25:31.162900+0000","last_undegraded":"2024-04-26T16:25:31.162900+0000","last_fullsized":"2024-04-26T16:25:31.162900+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:35.284202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"234.e","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.140356+0000","last_change":"2024-04-26T16:25:27.114078+0000","last_active":"2024-04-26T16:25:31.140356+0000","last_peered":"2024-04-26T16:25:31.140356+0000","last_clean":"2024-04-26T16:25:31.140356+0000","last_became_active":"2024-04-26T16:25:27.113788+0000","last_became_peered":"2024-04-26T16:25:27.113788+0000","last_unstale":"2024-04-26T16:25:31.140356+0000","last_undegraded":"2024-04-26T16:25:31.140356+0000","last_fullsized":"2024-04-26T16:25:31.140356+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:48.198979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"237.a","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.140235+0000","last_change":"2024-04-26T16:25:31.137683+0000","last_active":"2024-04-26T16:25:31.140235+0000","last_peered":"2024-04-26T16:25:31.140235+0000","last_clean":"2024-04-26T16:25:31.140235+0000","last_became_active":"2024-04-26T16:25:31.137086+0000","last_became_peered":"2024-04-26T16:25:31.137086+0000","last_unstale":"2024-04-26T16:25:31.140235+0000","last_undegraded":"2024-04-26T16:25:31.140235+0000","last_fullsized":"2024-04-26T16:25:31.140235+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:35.442389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"234.d","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.106453+0000","last_change":"2024-04-26T16:25:27.132142+0000","last_active":"2024-04-26T16:25:31.106453+0000","last_peered":"2024-04-26T16:25:31.106453+0000","last_clean":"2024-04-26T16:25:31.106453+0000","last_became_active":"2024-04-26T16:25:27.131726+0000","last_became_peered":"2024-04-26T16:25:27.131726+0000","last_unstale":"2024-04-26T16:25:31.106453+0000","last_undegraded":"2024-04-26T16:25:31.106453+0000","last_fullsized":"2024-04-26T16:25:31.106453+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:54.945237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"237.b","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122472+0000","last_change":"2024-04-26T16:25:31.122146+0000","last_active":"2024-04-26T16:25:31.122472+0000","last_peered":"2024-04-26T16:25:31.122472+0000","last_clean":"2024-04-26T16:25:31.122472+0000","last_became_active":"2024-04-26T16:25:31.121841+0000","last_became_peered":"2024-04-26T16:25:31.121841+0000","last_unstale":"2024-04-26T16:25:31.122472+0000","last_undegraded":"2024-04-26T16:25:31.122472+0000","last_fullsized":"2024-04-26T16:25:31.122472+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:57.340267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"234.c","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083758+0000","last_change":"2024-04-26T16:25:27.122102+0000","last_active":"2024-04-26T16:25:31.083758+0000","last_peered":"2024-04-26T16:25:31.083758+0000","last_clean":"2024-04-26T16:25:31.083758+0000","last_became_active":"2024-04-26T16:25:27.120734+0000","last_became_peered":"2024-04-26T16:25:27.120734+0000","last_unstale":"2024-04-26T16:25:31.083758+0000","last_undegraded":"2024-04-26T16:25:31.083758+0000","last_fullsized":"2024-04-26T16:25:31.083758+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:39.049283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.c","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.125177+0000","last_change":"2024-04-26T16:25:31.121044+0000","last_active":"2024-04-26T16:25:31.125177+0000","last_peered":"2024-04-26T16:25:31.125177+0000","last_clean":"2024-04-26T16:25:31.125177+0000","last_became_active":"2024-04-26T16:25:31.120431+0000","last_became_peered":"2024-04-26T16:25:31.120431+0000","last_unstale":"2024-04-26T16:25:31.125177+0000","last_undegraded":"2024-04-26T16:25:31.125177+0000","last_fullsized":"2024-04-26T16:25:31.125177+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:58.646034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"234.b","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.099944+0000","last_change":"2024-04-26T16:25:27.114791+0000","last_active":"2024-04-26T16:25:31.099944+0000","last_peered":"2024-04-26T16:25:31.099944+0000","last_clean":"2024-04-26T16:25:31.099944+0000","last_became_active":"2024-04-26T16:25:27.113834+0000","last_became_peered":"2024-04-26T16:25:27.113834+0000","last_unstale":"2024-04-26T16:25:31.099944+0000","last_undegraded":"2024-04-26T16:25:31.099944+0000","last_fullsized":"2024-04-26T16:25:31.099944+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:52.747824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.d","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.124488+0000","last_change":"2024-04-26T16:25:31.124222+0000","last_active":"2024-04-26T16:25:31.124488+0000","last_peered":"2024-04-26T16:25:31.124488+0000","last_clean":"2024-04-26T16:25:31.124488+0000","last_became_active":"2024-04-26T16:25:31.123908+0000","last_became_peered":"2024-04-26T16:25:31.123908+0000","last_unstale":"2024-04-26T16:25:31.124488+0000","last_undegraded":"2024-04-26T16:25:31.124488+0000","last_fullsized":"2024-04-26T16:25:31.124488+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:33.897488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"234.a","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.101506+0000","last_change":"2024-04-26T16:25:27.109424+0000","last_active":"2024-04-26T16:25:31.101506+0000","last_peered":"2024-04-26T16:25:31.101506+0000","last_clean":"2024-04-26T16:25:31.101506+0000","last_became_active":"2024-04-26T16:25:27.108798+0000","last_became_peered":"2024-04-26T16:25:27.108798+0000","last_unstale":"2024-04-26T16:25:31.101506+0000","last_undegraded":"2024-04-26T16:25:31.101506+0000","last_fullsized":"2024-04-26T16:25:31.101506+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:27.800564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"237.e","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.155551+0000","last_change":"2024-04-26T16:25:31.152108+0000","last_active":"2024-04-26T16:25:31.155551+0000","last_peered":"2024-04-26T16:25:31.155551+0000","last_clean":"2024-04-26T16:25:31.155551+0000","last_became_active":"2024-04-26T16:25:31.151109+0000","last_became_peered":"2024-04-26T16:25:31.151109+0000","last_unstale":"2024-04-26T16:25:31.155551+0000","last_undegraded":"2024-04-26T16:25:31.155551+0000","last_fullsized":"2024-04-26T16:25:31.155551+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:46.789978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"234.9","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081151+0000","last_change":"2024-04-26T16:25:27.120427+0000","last_active":"2024-04-26T16:25:31.081151+0000","last_peered":"2024-04-26T16:25:31.081151+0000","last_clean":"2024-04-26T16:25:31.081151+0000","last_became_active":"2024-04-26T16:25:27.116146+0000","last_became_peered":"2024-04-26T16:25:27.116146+0000","last_unstale":"2024-04-26T16:25:31.081151+0000","last_undegraded":"2024-04-26T16:25:31.081151+0000","last_fullsized":"2024-04-26T16:25:31.081151+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:34.932775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"237.f","version":"0'0","reported_seq":10,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.141966+0000","last_change":"2024-04-26T16:25:31.141532+0000","last_active":"2024-04-26T16:25:31.141966+0000","last_peered":"2024-04-26T16:25:31.141966+0000","last_clean":"2024-04-26T16:25:31.141966+0000","last_became_active":"2024-04-26T16:25:31.141232+0000","last_became_peered":"2024-04-26T16:25:31.141232+0000","last_unstale":"2024-04-26T16:25:31.141966+0000","last_undegraded":"2024-04-26T16:25:31.141966+0000","last_fullsized":"2024-04-26T16:25:31.141966+0000","mapping_epoch":262,"log_start":"0'0","ondisk_log_start":"0'0","created":262,"last_epoch_clean":263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:30.070906+0000","last_clean_scrub_stamp":"2024-04-26T16:25:30.070906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:35.631303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"234.8","version":"0'0","reported_seq":19,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.087852+0000","last_change":"2024-04-26T16:25:27.107486+0000","last_active":"2024-04-26T16:25:31.087852+0000","last_peered":"2024-04-26T16:25:31.087852+0000","last_clean":"2024-04-26T16:25:31.087852+0000","last_became_active":"2024-04-26T16:25:27.107045+0000","last_became_peered":"2024-04-26T16:25:27.107045+0000","last_unstale":"2024-04-26T16:25:31.087852+0000","last_undegraded":"2024-04-26T16:25:31.087852+0000","last_fullsized":"2024-04-26T16:25:31.087852+0000","mapping_epoch":258,"log_start":"0'0","ondisk_log_start":"0'0","created":258,"last_epoch_clean":259,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:25:26.058370+0000","last_clean_scrub_stamp":"2024-04-26T16:25:26.058370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:42.276325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.12","version":"224'85","reported_seq":360,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.101718+0000","last_change":"2024-04-26T16:25:02.105656+0000","last_active":"2024-04-26T16:25:31.101718+0000","last_peered":"2024-04-26T16:25:31.101718+0000","last_clean":"2024-04-26T16:25:31.101718+0000","last_became_active":"2024-04-26T16:23:44.188737+0000","last_became_peered":"2024-04-26T16:23:44.188737+0000","last_unstale":"2024-04-26T16:25:31.101718+0000","last_undegraded":"2024-04-26T16:25:31.101718+0000","last_fullsized":"2024-04-26T16:25:31.101718+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088911000000000003,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.13","version":"239'51","reported_seq":335,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.147981+0000","last_change":"2024-04-26T16:25:09.195165+0000","last_active":"2024-04-26T16:25:31.147981+0000","last_peered":"2024-04-26T16:25:31.147981+0000","last_clean":"2024-04-26T16:25:31.147981+0000","last_became_active":"2024-04-26T16:23:44.188069+0000","last_became_peered":"2024-04-26T16:23:44.188069+0000","last_unstale":"2024-04-26T16:25:31.147981+0000","last_undegraded":"2024-04-26T16:25:31.147981+0000","last_fullsized":"2024-04-26T16:25:31.147981+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.140003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.089573364000000003,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.10","version":"261'105","reported_seq":384,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.261729+0000","last_change":"2024-04-26T16:25:31.261762+0000","last_active":"2024-04-26T16:25:31.261729+0000","last_peered":"2024-04-26T16:25:31.261729+0000","last_clean":"2024-04-26T16:25:31.261729+0000","last_became_active":"2024-04-26T16:23:44.222453+0000","last_became_peered":"2024-04-26T16:23:44.222453+0000","last_unstale":"2024-04-26T16:25:31.261729+0000","last_undegraded":"2024-04-26T16:25:31.261729+0000","last_fullsized":"2024-04-26T16:25:31.261729+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.794082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12875963600000001,"stat_sum":{"num_bytes":154,"num_objects":14,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"109.11","version":"236'34","reported_seq":322,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081304+0000","last_change":"2024-04-26T16:25:07.190475+0000","last_active":"2024-04-26T16:25:31.081304+0000","last_peered":"2024-04-26T16:25:31.081304+0000","last_clean":"2024-04-26T16:25:31.081304+0000","last_became_active":"2024-04-26T16:23:44.102085+0000","last_became_peered":"2024-04-26T16:23:44.102085+0000","last_unstale":"2024-04-26T16:25:31.081304+0000","last_undegraded":"2024-04-26T16:25:31.081304+0000","last_fullsized":"2024-04-26T16:25:31.081304+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.106842729,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.16","version":"263'69","reported_seq":349,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.149082+0000","last_change":"2024-04-26T16:25:11.201769+0000","last_active":"2024-04-26T16:25:31.149082+0000","last_peered":"2024-04-26T16:25:31.149082+0000","last_clean":"2024-04-26T16:25:31.149082+0000","last_became_active":"2024-04-26T16:23:44.103556+0000","last_became_peered":"2024-04-26T16:23:44.103556+0000","last_unstale":"2024-04-26T16:25:31.149082+0000","last_undegraded":"2024-04-26T16:25:31.149082+0000","last_fullsized":"2024-04-26T16:25:31.149082+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.072562129000000003,"stat_sum":{"num_bytes":52,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":21,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.17","version":"263'35","reported_seq":332,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128346+0000","last_change":"2024-04-26T16:25:02.139491+0000","last_active":"2024-04-26T16:25:31.128346+0000","last_peered":"2024-04-26T16:25:31.128346+0000","last_clean":"2024-04-26T16:25:31.128346+0000","last_became_active":"2024-04-26T16:23:44.113797+0000","last_became_peered":"2024-04-26T16:23:44.113797+0000","last_unstale":"2024-04-26T16:25:31.128346+0000","last_undegraded":"2024-04-26T16:25:31.128346+0000","last_fullsized":"2024-04-26T16:25:31.128346+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'34","last_scrub_stamp":"2024-04-26T16:25:01.715462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:25:01.715462+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:15.522080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0096211140000000001,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.14","version":"247'34","reported_seq":317,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.090065+0000","last_change":"2024-04-26T16:25:19.207481+0000","last_active":"2024-04-26T16:25:31.090065+0000","last_peered":"2024-04-26T16:25:31.090065+0000","last_clean":"2024-04-26T16:25:31.090065+0000","last_became_active":"2024-04-26T16:23:44.119110+0000","last_became_peered":"2024-04-26T16:23:44.119110+0000","last_unstale":"2024-04-26T16:25:31.090065+0000","last_undegraded":"2024-04-26T16:25:31.090065+0000","last_fullsized":"2024-04-26T16:25:31.090065+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.676606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1129016309999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.15","version":"261'71","reported_seq":353,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.199634+0000","last_change":"2024-04-26T16:25:31.199670+0000","last_active":"2024-04-26T16:25:31.199634+0000","last_peered":"2024-04-26T16:25:31.199634+0000","last_clean":"2024-04-26T16:25:31.199634+0000","last_became_active":"2024-04-26T16:23:44.233737+0000","last_became_peered":"2024-04-26T16:23:44.233737+0000","last_unstale":"2024-04-26T16:25:31.199634+0000","last_undegraded":"2024-04-26T16:25:31.199634+0000","last_fullsized":"2024-04-26T16:25:31.199634+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.160125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112825798,"stat_sum":{"num_bytes":57,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"109.1a","version":"263'85","reported_seq":362,"reported_epoch":263,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T16:25:31.213339+0000","last_change":"2024-04-26T16:25:27.207625+0000","last_active":"2024-04-26T16:25:31.213339+0000","last_peered":"2024-04-26T16:25:31.213339+0000","last_clean":"2024-04-26T16:25:31.213339+0000","last_became_active":"2024-04-26T16:23:44.233699+0000","last_became_peered":"2024-04-26T16:23:44.233699+0000","last_unstale":"2024-04-26T16:25:31.213339+0000","last_undegraded":"2024-04-26T16:25:31.213339+0000","last_fullsized":"2024-04-26T16:25:31.213339+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.160092+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.092629072000000007,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"1"},{"start":"6","length":"1"}]},{"pgid":"109.1b","version":"253'18","reported_seq":308,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.083045+0000","last_change":"2024-04-26T16:25:23.201454+0000","last_active":"2024-04-26T16:25:31.083045+0000","last_peered":"2024-04-26T16:25:31.083045+0000","last_clean":"2024-04-26T16:25:31.083045+0000","last_became_active":"2024-04-26T16:23:44.204827+0000","last_became_peered":"2024-04-26T16:23:44.204827+0000","last_unstale":"2024-04-26T16:25:31.083045+0000","last_undegraded":"2024-04-26T16:25:31.083045+0000","last_fullsized":"2024-04-26T16:25:31.083045+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.160115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.13174530800000001,"stat_sum":{"num_bytes":14,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"109.18","version":"224'34","reported_seq":323,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100360+0000","last_change":"2024-04-26T16:25:02.117958+0000","last_active":"2024-04-26T16:25:31.100360+0000","last_peered":"2024-04-26T16:25:31.100360+0000","last_clean":"2024-04-26T16:25:31.100360+0000","last_became_active":"2024-04-26T16:23:44.259913+0000","last_became_peered":"2024-04-26T16:23:44.259913+0000","last_unstale":"2024-04-26T16:25:31.100360+0000","last_undegraded":"2024-04-26T16:25:31.100360+0000","last_fullsized":"2024-04-26T16:25:31.100360+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038846449999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.19","version":"224'34","reported_seq":322,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.098652+0000","last_change":"2024-04-26T16:25:02.117816+0000","last_active":"2024-04-26T16:25:31.098652+0000","last_peered":"2024-04-26T16:25:31.098652+0000","last_clean":"2024-04-26T16:25:31.098652+0000","last_became_active":"2024-04-26T16:23:44.267057+0000","last_became_peered":"2024-04-26T16:23:44.267057+0000","last_unstale":"2024-04-26T16:25:31.098652+0000","last_undegraded":"2024-04-26T16:25:31.098652+0000","last_fullsized":"2024-04-26T16:25:31.098652+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003646687,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.1e","version":"263'19","reported_seq":307,"reported_epoch":263,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T16:25:31.199800+0000","last_change":"2024-04-26T16:25:31.199800+0000","last_active":"2024-04-26T16:25:31.199800+0000","last_peered":"2024-04-26T16:25:31.199800+0000","last_clean":"2024-04-26T16:25:31.199800+0000","last_became_active":"2024-04-26T16:23:44.248875+0000","last_became_peered":"2024-04-26T16:23:44.248875+0000","last_unstale":"2024-04-26T16:25:31.199800+0000","last_undegraded":"2024-04-26T16:25:31.199800+0000","last_fullsized":"2024-04-26T16:25:31.199800+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.160135+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.054907317999999997,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"109.1f","version":"224'17","reported_seq":310,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103391+0000","last_change":"2024-04-26T16:25:02.117341+0000","last_active":"2024-04-26T16:25:31.103391+0000","last_peered":"2024-04-26T16:25:31.103391+0000","last_clean":"2024-04-26T16:25:31.103391+0000","last_became_active":"2024-04-26T16:23:44.180651+0000","last_became_peered":"2024-04-26T16:23:44.180651+0000","last_unstale":"2024-04-26T16:25:31.103391+0000","last_undegraded":"2024-04-26T16:25:31.103391+0000","last_fullsized":"2024-04-26T16:25:31.103391+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068057930000000001,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.1c","version":"236'51","reported_seq":340,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122089+0000","last_change":"2024-04-26T16:25:07.176471+0000","last_active":"2024-04-26T16:25:31.122089+0000","last_peered":"2024-04-26T16:25:31.122089+0000","last_clean":"2024-04-26T16:25:31.122089+0000","last_became_active":"2024-04-26T16:23:44.113393+0000","last_became_peered":"2024-04-26T16:23:44.113393+0000","last_unstale":"2024-04-26T16:25:31.122089+0000","last_undegraded":"2024-04-26T16:25:31.122089+0000","last_fullsized":"2024-04-26T16:25:31.122089+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.083792801,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.1d","version":"263'112","reported_seq":365,"reported_epoch":263,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T16:25:31.248336+0000","last_change":"2024-04-26T16:25:29.242924+0000","last_active":"2024-04-26T16:25:31.248336+0000","last_peered":"2024-04-26T16:25:31.248336+0000","last_clean":"2024-04-26T16:25:31.248336+0000","last_became_active":"2024-04-26T16:23:44.112550+0000","last_became_peered":"2024-04-26T16:23:44.112550+0000","last_unstale":"2024-04-26T16:25:31.248336+0000","last_undegraded":"2024-04-26T16:25:31.248336+0000","last_fullsized":"2024-04-26T16:25:31.248336+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.794111+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.14174488599999999,"stat_sum":{"num_bytes":220,"num_objects":20,"num_object_clones":12,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"109.2","version":"253'85","reported_seq":365,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088880+0000","last_change":"2024-04-26T16:25:23.236618+0000","last_active":"2024-04-26T16:25:31.088880+0000","last_peered":"2024-04-26T16:25:31.088880+0000","last_clean":"2024-04-26T16:25:31.088880+0000","last_became_active":"2024-04-26T16:23:44.121137+0000","last_became_peered":"2024-04-26T16:23:44.121137+0000","last_unstale":"2024-04-26T16:25:31.088880+0000","last_undegraded":"2024-04-26T16:25:31.088880+0000","last_fullsized":"2024-04-26T16:25:31.088880+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.676548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.13001673799999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.3","version":"259'51","reported_seq":335,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.131122+0000","last_change":"2024-04-26T16:25:29.242818+0000","last_active":"2024-04-26T16:25:31.131122+0000","last_peered":"2024-04-26T16:25:31.131122+0000","last_clean":"2024-04-26T16:25:31.131122+0000","last_became_active":"2024-04-26T16:23:44.116220+0000","last_became_peered":"2024-04-26T16:23:44.116220+0000","last_unstale":"2024-04-26T16:25:31.131122+0000","last_undegraded":"2024-04-26T16:25:31.131122+0000","last_fullsized":"2024-04-26T16:25:31.131122+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.794187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10910800499999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"109.0","version":"224'73","reported_seq":352,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094534+0000","last_change":"2024-04-26T16:25:02.113780+0000","last_active":"2024-04-26T16:25:31.094534+0000","last_peered":"2024-04-26T16:25:31.094534+0000","last_clean":"2024-04-26T16:25:31.094534+0000","last_became_active":"2024-04-26T16:23:44.226589+0000","last_became_peered":"2024-04-26T16:23:44.226589+0000","last_unstale":"2024-04-26T16:25:31.094534+0000","last_undegraded":"2024-04-26T16:25:31.094534+0000","last_fullsized":"2024-04-26T16:25:31.094534+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078415129999999996,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.1","version":"255'78","reported_seq":352,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.142798+0000","last_change":"2024-04-26T16:25:25.233243+0000","last_active":"2024-04-26T16:25:31.142798+0000","last_peered":"2024-04-26T16:25:31.142798+0000","last_clean":"2024-04-26T16:25:31.142798+0000","last_became_active":"2024-04-26T16:23:44.117780+0000","last_became_peered":"2024-04-26T16:23:44.117780+0000","last_unstale":"2024-04-26T16:25:31.142798+0000","last_undegraded":"2024-04-26T16:25:31.142798+0000","last_fullsized":"2024-04-26T16:25:31.142798+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.794163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.137007745,"stat_sum":{"num_bytes":198,"num_objects":18,"num_object_clones":12,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"1"}]},{"pgid":"109.6","version":"263'57","reported_seq":345,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.082893+0000","last_change":"2024-04-26T16:25:27.206915+0000","last_active":"2024-04-26T16:25:31.082893+0000","last_peered":"2024-04-26T16:25:31.082893+0000","last_clean":"2024-04-26T16:25:31.082893+0000","last_became_active":"2024-04-26T16:23:44.248476+0000","last_became_peered":"2024-04-26T16:23:44.248476+0000","last_unstale":"2024-04-26T16:25:31.082893+0000","last_undegraded":"2024-04-26T16:25:31.082893+0000","last_fullsized":"2024-04-26T16:25:31.082893+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.160104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.103422786,"stat_sum":{"num_bytes":46,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"109.7","version":"263'55","reported_seq":339,"reported_epoch":263,"state":"active+clean+snaptrim","last_fresh":"2024-04-26T16:25:31.263897+0000","last_change":"2024-04-26T16:25:31.261818+0000","last_active":"2024-04-26T16:25:31.263897+0000","last_peered":"2024-04-26T16:25:31.263897+0000","last_clean":"2024-04-26T16:25:31.263897+0000","last_became_active":"2024-04-26T16:23:44.200194+0000","last_became_peered":"2024-04-26T16:23:44.200194+0000","last_unstale":"2024-04-26T16:25:31.263897+0000","last_undegraded":"2024-04-26T16:25:31.263897+0000","last_fullsized":"2024-04-26T16:25:31.263897+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.794138+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.12697580999999999,"stat_sum":{"num_bytes":53,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"4","length":"2"}]},{"pgid":"109.4","version":"249'85","reported_seq":366,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084009+0000","last_change":"2024-04-26T16:25:19.177541+0000","last_active":"2024-04-26T16:25:31.084009+0000","last_peered":"2024-04-26T16:25:31.084009+0000","last_clean":"2024-04-26T16:25:31.084009+0000","last_became_active":"2024-04-26T16:23:44.220715+0000","last_became_peered":"2024-04-26T16:23:44.220715+0000","last_unstale":"2024-04-26T16:25:31.084009+0000","last_undegraded":"2024-04-26T16:25:31.084009+0000","last_fullsized":"2024-04-26T16:25:31.084009+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10795856299999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.5","version":"243'34","reported_seq":321,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.096951+0000","last_change":"2024-04-26T16:25:17.214342+0000","last_active":"2024-04-26T16:25:31.096951+0000","last_peered":"2024-04-26T16:25:31.096951+0000","last_clean":"2024-04-26T16:25:31.096951+0000","last_became_active":"2024-04-26T16:23:44.116978+0000","last_became_peered":"2024-04-26T16:23:44.116978+0000","last_unstale":"2024-04-26T16:25:31.096951+0000","last_undegraded":"2024-04-26T16:25:31.096951+0000","last_fullsized":"2024-04-26T16:25:31.096951+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.676484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.092960265,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.a","version":"263'35","reported_seq":319,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.090081+0000","last_change":"2024-04-26T16:25:17.215177+0000","last_active":"2024-04-26T16:25:31.090081+0000","last_peered":"2024-04-26T16:25:31.090081+0000","last_clean":"2024-04-26T16:25:31.090081+0000","last_became_active":"2024-04-26T16:23:44.121619+0000","last_became_peered":"2024-04-26T16:23:44.121619+0000","last_unstale":"2024-04-26T16:25:31.090081+0000","last_undegraded":"2024-04-26T16:25:31.090081+0000","last_fullsized":"2024-04-26T16:25:31.090081+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.676577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.094583507999999997,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.b","version":"236'34","reported_seq":325,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083178+0000","last_change":"2024-04-26T16:25:07.189928+0000","last_active":"2024-04-26T16:25:31.083178+0000","last_peered":"2024-04-26T16:25:31.083178+0000","last_clean":"2024-04-26T16:25:31.083178+0000","last_became_active":"2024-04-26T16:23:44.081905+0000","last_became_peered":"2024-04-26T16:23:44.081905+0000","last_unstale":"2024-04-26T16:25:31.083178+0000","last_undegraded":"2024-04-26T16:25:31.083178+0000","last_fullsized":"2024-04-26T16:25:31.083178+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10765375200000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.8","version":"257'52","reported_seq":335,"reported_epoch":263,"state":"active+clean+snaptrim_wait","last_fresh":"2024-04-26T16:25:31.088709+0000","last_change":"2024-04-26T16:25:27.194135+0000","last_active":"2024-04-26T16:25:31.088709+0000","last_peered":"2024-04-26T16:25:31.088709+0000","last_clean":"2024-04-26T16:25:31.088709+0000","last_became_active":"2024-04-26T16:23:44.205186+0000","last_became_peered":"2024-04-26T16:23:44.205186+0000","last_unstale":"2024-04-26T16:25:31.088709+0000","last_undegraded":"2024-04-26T16:25:31.088709+0000","last_fullsized":"2024-04-26T16:25:31.088709+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.160078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.102901782,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"4","length":"3"}]},{"pgid":"109.9","version":"224'34","reported_seq":318,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095589+0000","last_change":"2024-04-26T16:25:02.106562+0000","last_active":"2024-04-26T16:25:31.095589+0000","last_peered":"2024-04-26T16:25:31.095589+0000","last_clean":"2024-04-26T16:25:31.095589+0000","last_became_active":"2024-04-26T16:23:44.109291+0000","last_became_peered":"2024-04-26T16:23:44.109291+0000","last_unstale":"2024-04-26T16:25:31.095589+0000","last_undegraded":"2024-04-26T16:25:31.095589+0000","last_fullsized":"2024-04-26T16:25:31.095589+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00103929,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.e","version":"258'24","reported_seq":320,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103045+0000","last_change":"2024-04-26T16:25:02.116998+0000","last_active":"2024-04-26T16:25:31.103045+0000","last_peered":"2024-04-26T16:25:31.103045+0000","last_clean":"2024-04-26T16:25:31.103045+0000","last_became_active":"2024-04-26T16:23:44.101672+0000","last_became_peered":"2024-04-26T16:23:44.101672+0000","last_unstale":"2024-04-26T16:25:31.103045+0000","last_undegraded":"2024-04-26T16:25:31.103045+0000","last_fullsized":"2024-04-26T16:25:31.103045+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065896180000000002,"stat_sum":{"num_bytes":24,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,5],"acting":[3,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.f","version":"258'69","reported_seq":341,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.113359+0000","last_change":"2024-04-26T16:25:19.207835+0000","last_active":"2024-04-26T16:25:31.113359+0000","last_peered":"2024-04-26T16:25:31.113359+0000","last_clean":"2024-04-26T16:25:31.113359+0000","last_became_active":"2024-04-26T16:23:44.115717+0000","last_became_peered":"2024-04-26T16:23:44.115717+0000","last_unstale":"2024-04-26T16:25:31.113359+0000","last_undegraded":"2024-04-26T16:25:31.113359+0000","last_fullsized":"2024-04-26T16:25:31.113359+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.676518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.111597248,"stat_sum":{"num_bytes":44,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.c","version":"235'34","reported_seq":319,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083910+0000","last_change":"2024-04-26T16:25:07.189429+0000","last_active":"2024-04-26T16:25:31.083910+0000","last_peered":"2024-04-26T16:25:31.083910+0000","last_clean":"2024-04-26T16:25:31.083910+0000","last_became_active":"2024-04-26T16:23:44.106600+0000","last_became_peered":"2024-04-26T16:23:44.106600+0000","last_unstale":"2024-04-26T16:25:31.083910+0000","last_undegraded":"2024-04-26T16:25:31.083910+0000","last_fullsized":"2024-04-26T16:25:31.083910+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.091065645000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"109.d","version":"236'17","reported_seq":309,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128219+0000","last_change":"2024-04-26T16:25:09.194193+0000","last_active":"2024-04-26T16:25:31.128219+0000","last_peered":"2024-04-26T16:25:31.128219+0000","last_clean":"2024-04-26T16:25:31.128219+0000","last_became_active":"2024-04-26T16:23:44.104408+0000","last_became_peered":"2024-04-26T16:23:44.104408+0000","last_unstale":"2024-04-26T16:25:31.128219+0000","last_undegraded":"2024-04-26T16:25:31.128219+0000","last_fullsized":"2024-04-26T16:25:31.128219+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.089734861999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"4","length":"4"}]},{"pgid":"12.0","version":"217'7","reported_seq":317,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088138+0000","last_change":"2024-04-26T16:24:41.087486+0000","last_active":"2024-04-26T16:25:31.088138+0000","last_peered":"2024-04-26T16:25:31.088138+0000","last_clean":"2024-04-26T16:25:31.088138+0000","last_became_active":"2024-04-26T16:24:41.087104+0000","last_became_peered":"2024-04-26T16:24:41.087104+0000","last_unstale":"2024-04-26T16:25:31.088138+0000","last_undegraded":"2024-04-26T16:25:31.088138+0000","last_fullsized":"2024-04-26T16:25:31.088138+0000","mapping_epoch":140,"log_start":"141'5","ondisk_log_start":"141'5","created":140,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":5,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":254,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.152352+0000","last_change":"2024-04-26T16:23:29.345935+0000","last_active":"2024-04-26T16:25:31.152352+0000","last_peered":"2024-04-26T16:25:31.152352+0000","last_clean":"2024-04-26T16:25:31.152352+0000","last_became_active":"2024-04-26T16:23:29.345650+0000","last_became_peered":"2024-04-26T16:23:29.345650+0000","last_unstale":"2024-04-26T16:25:31.152352+0000","last_undegraded":"2024-04-26T16:25:31.152352+0000","last_fullsized":"2024-04-26T16:25:31.152352+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"109'12","reported_seq":504,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.142220+0000","last_change":"2024-04-26T16:22:58.402880+0000","last_active":"2024-04-26T16:25:31.142220+0000","last_peered":"2024-04-26T16:25:31.142220+0000","last_clean":"2024-04-26T16:25:31.142220+0000","last_became_active":"2024-04-26T16:17:33.070525+0000","last_became_peered":"2024-04-26T16:17:33.070525+0000","last_unstale":"2024-04-26T16:25:31.142220+0000","last_undegraded":"2024-04-26T16:25:31.142220+0000","last_fullsized":"2024-04-26T16:25:31.142220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-26T16:22:58.402636+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_clean_scrub_stamp":"2024-04-26T16:22:58.402636+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"86'15","reported_seq":433,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.118042+0000","last_change":"2024-04-26T16:24:59.587971+0000","last_active":"2024-04-26T16:25:31.118042+0000","last_peered":"2024-04-26T16:25:31.118042+0000","last_clean":"2024-04-26T16:25:31.118042+0000","last_became_active":"2024-04-26T16:17:31.033340+0000","last_became_peered":"2024-04-26T16:17:31.033340+0000","last_unstale":"2024-04-26T16:25:31.118042+0000","last_undegraded":"2024-04-26T16:25:31.118042+0000","last_fullsized":"2024-04-26T16:25:31.118042+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:24:59.587812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.587812+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:18.046219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083369+0000","last_change":"2024-04-26T16:24:45.121653+0000","last_active":"2024-04-26T16:25:31.083369+0000","last_peered":"2024-04-26T16:25:31.083369+0000","last_clean":"2024-04-26T16:25:31.083369+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:25:31.083369+0000","last_undegraded":"2024-04-26T16:25:31.083369+0000","last_fullsized":"2024-04-26T16:25:31.083369+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:45.121528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.121528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.874222+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":408,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122832+0000","last_change":"2024-04-26T16:25:09.712425+0000","last_active":"2024-04-26T16:25:31.122832+0000","last_peered":"2024-04-26T16:25:31.122832+0000","last_clean":"2024-04-26T16:25:31.122832+0000","last_became_active":"2024-04-26T16:17:35.669627+0000","last_became_peered":"2024-04-26T16:17:35.669627+0000","last_unstale":"2024-04-26T16:25:31.122832+0000","last_undegraded":"2024-04-26T16:25:31.122832+0000","last_fullsized":"2024-04-26T16:25:31.122832+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:09.712321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:25:09.712321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:13.017621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"217'14","reported_seq":335,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.110271+0000","last_change":"2024-04-26T16:25:01.097848+0000","last_active":"2024-04-26T16:25:31.110271+0000","last_peered":"2024-04-26T16:25:31.110271+0000","last_clean":"2024-04-26T16:25:31.110271+0000","last_became_active":"2024-04-26T16:25:01.097348+0000","last_became_peered":"2024-04-26T16:25:01.097348+0000","last_unstale":"2024-04-26T16:25:31.110271+0000","last_undegraded":"2024-04-26T16:25:31.110271+0000","last_fullsized":"2024-04-26T16:25:31.110271+0000","mapping_epoch":140,"log_start":"217'14","ondisk_log_start":"217'14","created":140,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'13","last_scrub_stamp":"2024-04-26T16:24:57.362265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:57.362265+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":34,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:13.897554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.133215+0000","last_change":"2024-04-26T16:23:29.355081+0000","last_active":"2024-04-26T16:25:31.133215+0000","last_peered":"2024-04-26T16:25:31.133215+0000","last_clean":"2024-04-26T16:25:31.133215+0000","last_became_active":"2024-04-26T16:23:29.349198+0000","last_became_peered":"2024-04-26T16:23:29.349198+0000","last_unstale":"2024-04-26T16:25:31.133215+0000","last_undegraded":"2024-04-26T16:25:31.133215+0000","last_fullsized":"2024-04-26T16:25:31.133215+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.760171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"86'10","reported_seq":423,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.102863+0000","last_change":"2024-04-26T16:24:52.466440+0000","last_active":"2024-04-26T16:25:31.102863+0000","last_peered":"2024-04-26T16:25:31.102863+0000","last_clean":"2024-04-26T16:25:31.102863+0000","last_became_active":"2024-04-26T16:17:31.040215+0000","last_became_peered":"2024-04-26T16:17:31.040215+0000","last_unstale":"2024-04-26T16:25:31.102863+0000","last_undegraded":"2024-04-26T16:25:31.102863+0000","last_fullsized":"2024-04-26T16:25:31.102863+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:24:52.466192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:52.466192+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:10.401169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.087159+0000","last_change":"2024-04-26T16:23:27.211678+0000","last_active":"2024-04-26T16:25:31.087159+0000","last_peered":"2024-04-26T16:25:31.087159+0000","last_clean":"2024-04-26T16:25:31.087159+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:25:31.087159+0000","last_undegraded":"2024-04-26T16:25:31.087159+0000","last_fullsized":"2024-04-26T16:25:31.087159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:27.211548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:23:27.211548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084127+0000","last_change":"2024-04-26T16:24:32.940701+0000","last_active":"2024-04-26T16:25:31.084127+0000","last_peered":"2024-04-26T16:25:31.084127+0000","last_clean":"2024-04-26T16:25:31.084127+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:25:31.084127+0000","last_undegraded":"2024-04-26T16:25:31.084127+0000","last_fullsized":"2024-04-26T16:25:31.084127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:32.940508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:32.940508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.262390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":407,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.102230+0000","last_change":"2024-04-26T16:24:55.354455+0000","last_active":"2024-04-26T16:25:31.102230+0000","last_peered":"2024-04-26T16:25:31.102230+0000","last_clean":"2024-04-26T16:25:31.102230+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:25:31.102230+0000","last_undegraded":"2024-04-26T16:25:31.102230+0000","last_fullsized":"2024-04-26T16:25:31.102230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:55.354314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:55.354314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:01.227496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":309,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.090120+0000","last_change":"2024-04-26T16:24:33.095545+0000","last_active":"2024-04-26T16:25:31.090120+0000","last_peered":"2024-04-26T16:25:31.090120+0000","last_clean":"2024-04-26T16:25:31.090120+0000","last_became_active":"2024-04-26T16:24:33.094869+0000","last_became_peered":"2024-04-26T16:24:33.094869+0000","last_unstale":"2024-04-26T16:25:31.090120+0000","last_undegraded":"2024-04-26T16:25:31.090120+0000","last_fullsized":"2024-04-26T16:25:31.090120+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.147927+0000","last_change":"2024-04-26T16:23:29.364650+0000","last_active":"2024-04-26T16:25:31.147927+0000","last_peered":"2024-04-26T16:25:31.147927+0000","last_clean":"2024-04-26T16:25:31.147927+0000","last_became_active":"2024-04-26T16:23:29.363404+0000","last_became_peered":"2024-04-26T16:23:29.363404+0000","last_unstale":"2024-04-26T16:25:31.147927+0000","last_undegraded":"2024-04-26T16:25:31.147927+0000","last_fullsized":"2024-04-26T16:25:31.147927+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"86'17","reported_seq":436,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.145068+0000","last_change":"2024-04-26T16:24:47.710795+0000","last_active":"2024-04-26T16:25:31.145068+0000","last_peered":"2024-04-26T16:25:31.145068+0000","last_clean":"2024-04-26T16:25:31.145068+0000","last_became_active":"2024-04-26T16:17:31.030244+0000","last_became_peered":"2024-04-26T16:17:31.030244+0000","last_unstale":"2024-04-26T16:25:31.145068+0000","last_undegraded":"2024-04-26T16:25:31.145068+0000","last_fullsized":"2024-04-26T16:25:31.145068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-04-26T16:24:47.710551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:47.710551+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:56.192034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088247+0000","last_change":"2024-04-26T16:24:28.928745+0000","last_active":"2024-04-26T16:25:31.088247+0000","last_peered":"2024-04-26T16:25:31.088247+0000","last_clean":"2024-04-26T16:25:31.088247+0000","last_became_active":"2024-04-26T16:17:29.017152+0000","last_became_peered":"2024-04-26T16:17:29.017152+0000","last_unstale":"2024-04-26T16:25:31.088247+0000","last_undegraded":"2024-04-26T16:25:31.088247+0000","last_fullsized":"2024-04-26T16:25:31.088247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:28.928433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:28.928433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:45.713025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083316+0000","last_change":"2024-04-26T16:24:34.974099+0000","last_active":"2024-04-26T16:25:31.083316+0000","last_peered":"2024-04-26T16:25:31.083316+0000","last_clean":"2024-04-26T16:25:31.083316+0000","last_became_active":"2024-04-26T16:17:33.065671+0000","last_became_peered":"2024-04-26T16:17:33.065671+0000","last_unstale":"2024-04-26T16:25:31.083316+0000","last_undegraded":"2024-04-26T16:25:31.083316+0000","last_fullsized":"2024-04-26T16:25:31.083316+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:34.973881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:34.973881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:58.177909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":392,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.133299+0000","last_change":"2024-04-26T16:23:18.281302+0000","last_active":"2024-04-26T16:25:31.133299+0000","last_peered":"2024-04-26T16:25:31.133299+0000","last_clean":"2024-04-26T16:25:31.133299+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:25:31.133299+0000","last_undegraded":"2024-04-26T16:25:31.133299+0000","last_fullsized":"2024-04-26T16:25:31.133299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:18.281178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:23:18.281178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.5","version":"217'11","reported_seq":326,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.121856+0000","last_change":"2024-04-26T16:24:59.657823+0000","last_active":"2024-04-26T16:25:31.121856+0000","last_peered":"2024-04-26T16:25:31.121856+0000","last_clean":"2024-04-26T16:25:31.121856+0000","last_became_active":"2024-04-26T16:24:53.126300+0000","last_became_peered":"2024-04-26T16:24:53.126300+0000","last_unstale":"2024-04-26T16:25:31.121856+0000","last_undegraded":"2024-04-26T16:25:31.121856+0000","last_fullsized":"2024-04-26T16:25:31.121856+0000","mapping_epoch":140,"log_start":"217'11","ondisk_log_start":"217'11","created":140,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-04-26T16:24:59.657710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.657710+0000","objects_scrubbed":2,"log_size":0,"log_dups_size":16,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:11.933964+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":263,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086591+0000","last_change":"2024-04-26T16:24:53.146189+0000","last_active":"2024-04-26T16:25:31.086591+0000","last_peered":"2024-04-26T16:25:31.086591+0000","last_clean":"2024-04-26T16:25:31.086591+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:25:31.086591+0000","last_undegraded":"2024-04-26T16:25:31.086591+0000","last_fullsized":"2024-04-26T16:25:31.086591+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:53.146054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:53.146054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:16.168203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"86'19","reported_seq":446,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.099906+0000","last_change":"2024-04-26T16:24:35.397766+0000","last_active":"2024-04-26T16:25:31.099906+0000","last_peered":"2024-04-26T16:25:31.099906+0000","last_clean":"2024-04-26T16:25:31.099906+0000","last_became_active":"2024-04-26T16:17:31.024436+0000","last_became_peered":"2024-04-26T16:17:31.024436+0000","last_unstale":"2024-04-26T16:25:31.099906+0000","last_undegraded":"2024-04-26T16:25:31.099906+0000","last_fullsized":"2024-04-26T16:25:31.099906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'19","last_scrub_stamp":"2024-04-26T16:24:35.397607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:35.397607+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:49.735915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.131911+0000","last_change":"2024-04-26T16:23:00.392419+0000","last_active":"2024-04-26T16:25:31.131911+0000","last_peered":"2024-04-26T16:25:31.131911+0000","last_clean":"2024-04-26T16:25:31.131911+0000","last_became_active":"2024-04-26T16:17:29.026793+0000","last_became_peered":"2024-04-26T16:17:29.026793+0000","last_unstale":"2024-04-26T16:25:31.131911+0000","last_undegraded":"2024-04-26T16:25:31.131911+0000","last_fullsized":"2024-04-26T16:25:31.131911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:00.392134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:23:00.392134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084634+0000","last_change":"2024-04-26T16:24:40.513071+0000","last_active":"2024-04-26T16:25:31.084634+0000","last_peered":"2024-04-26T16:25:31.084634+0000","last_clean":"2024-04-26T16:25:31.084634+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:25:31.084634+0000","last_undegraded":"2024-04-26T16:25:31.084634+0000","last_fullsized":"2024-04-26T16:25:31.084634+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:40.512788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:01.215041+0000","last_clean_scrub_stamp":"2024-04-26T16:24:40.512788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:59.404333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":408,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086767+0000","last_change":"2024-04-26T16:24:39.988286+0000","last_active":"2024-04-26T16:25:31.086767+0000","last_peered":"2024-04-26T16:25:31.086767+0000","last_clean":"2024-04-26T16:25:31.086767+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:25:31.086767+0000","last_undegraded":"2024-04-26T16:25:31.086767+0000","last_fullsized":"2024-04-26T16:25:31.086767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:39.988072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:24:39.988072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.085019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.4","version":"141'2","reported_seq":326,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122229+0000","last_change":"2024-04-26T16:24:57.108921+0000","last_active":"2024-04-26T16:25:31.122229+0000","last_peered":"2024-04-26T16:25:31.122229+0000","last_clean":"2024-04-26T16:25:31.122229+0000","last_became_active":"2024-04-26T16:24:57.105525+0000","last_became_peered":"2024-04-26T16:24:57.105525+0000","last_unstale":"2024-04-26T16:25:31.122229+0000","last_undegraded":"2024-04-26T16:25:31.122229+0000","last_fullsized":"2024-04-26T16:25:31.122229+0000","mapping_epoch":140,"log_start":"141'2","ondisk_log_start":"141'2","created":140,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'2","last_scrub_stamp":"2024-04-26T16:24:53.627247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:53.627247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":2,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.468796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.116752+0000","last_change":"2024-04-26T16:25:05.732009+0000","last_active":"2024-04-26T16:25:31.116752+0000","last_peered":"2024-04-26T16:25:31.116752+0000","last_clean":"2024-04-26T16:25:31.116752+0000","last_became_active":"2024-04-26T16:23:29.321850+0000","last_became_peered":"2024-04-26T16:23:29.321850+0000","last_unstale":"2024-04-26T16:25:31.116752+0000","last_undegraded":"2024-04-26T16:25:31.116752+0000","last_fullsized":"2024-04-26T16:25:31.116752+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:05.731798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:05.731798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:20.989539+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"122'13","reported_seq":428,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100571+0000","last_change":"2024-04-26T16:24:38.091964+0000","last_active":"2024-04-26T16:25:31.100571+0000","last_peered":"2024-04-26T16:25:31.100571+0000","last_clean":"2024-04-26T16:25:31.100571+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:25:31.100571+0000","last_undegraded":"2024-04-26T16:25:31.100571+0000","last_fullsized":"2024-04-26T16:25:31.100571+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'13","last_scrub_stamp":"2024-04-26T16:24:38.091834+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:24:38.091834+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:42.293706+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.082529+0000","last_change":"2024-04-26T16:23:09.110676+0000","last_active":"2024-04-26T16:25:31.082529+0000","last_peered":"2024-04-26T16:25:31.082529+0000","last_clean":"2024-04-26T16:25:31.082529+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:25:31.082529+0000","last_undegraded":"2024-04-26T16:25:31.082529+0000","last_fullsized":"2024-04-26T16:25:31.082529+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:09.110555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:23:09.110555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"86'11","reported_seq":519,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086846+0000","last_change":"2024-04-26T16:24:44.127188+0000","last_active":"2024-04-26T16:25:31.086846+0000","last_peered":"2024-04-26T16:25:31.086846+0000","last_clean":"2024-04-26T16:25:31.086846+0000","last_became_active":"2024-04-26T16:17:33.066299+0000","last_became_peered":"2024-04-26T16:17:33.066299+0000","last_unstale":"2024-04-26T16:25:31.086846+0000","last_undegraded":"2024-04-26T16:25:31.086846+0000","last_fullsized":"2024-04-26T16:25:31.086846+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:44.127038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:44.127038+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.040882+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":402,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.096349+0000","last_change":"2024-04-26T16:24:33.816283+0000","last_active":"2024-04-26T16:25:31.096349+0000","last_peered":"2024-04-26T16:25:31.096349+0000","last_clean":"2024-04-26T16:25:31.096349+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:25:31.096349+0000","last_undegraded":"2024-04-26T16:25:31.096349+0000","last_fullsized":"2024-04-26T16:25:31.096349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:33.816076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:24:33.816076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:48.327675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"217'9","reported_seq":323,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083174+0000","last_change":"2024-04-26T16:24:45.110504+0000","last_active":"2024-04-26T16:25:31.083174+0000","last_peered":"2024-04-26T16:25:31.083174+0000","last_clean":"2024-04-26T16:25:31.083174+0000","last_became_active":"2024-04-26T16:24:45.109009+0000","last_became_peered":"2024-04-26T16:24:45.109009+0000","last_unstale":"2024-04-26T16:25:31.083174+0000","last_undegraded":"2024-04-26T16:25:31.083174+0000","last_fullsized":"2024-04-26T16:25:31.083174+0000","mapping_epoch":140,"log_start":"141'5","ondisk_log_start":"141'5","created":140,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":10,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":256,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084119+0000","last_change":"2024-04-26T16:23:29.220146+0000","last_active":"2024-04-26T16:25:31.084119+0000","last_peered":"2024-04-26T16:25:31.084119+0000","last_clean":"2024-04-26T16:25:31.084119+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:25:31.084119+0000","last_undegraded":"2024-04-26T16:25:31.084119+0000","last_fullsized":"2024-04-26T16:25:31.084119+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"86'15","reported_seq":433,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094946+0000","last_change":"2024-04-26T16:24:44.951341+0000","last_active":"2024-04-26T16:25:31.094946+0000","last_peered":"2024-04-26T16:25:31.094946+0000","last_clean":"2024-04-26T16:25:31.094946+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:25:31.094946+0000","last_undegraded":"2024-04-26T16:25:31.094946+0000","last_fullsized":"2024-04-26T16:25:31.094946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:24:44.951225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:44.951225+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:01.894612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":428,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081658+0000","last_change":"2024-04-26T16:24:39.549952+0000","last_active":"2024-04-26T16:25:31.081658+0000","last_peered":"2024-04-26T16:25:31.081658+0000","last_clean":"2024-04-26T16:25:31.081658+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:25:31.081658+0000","last_undegraded":"2024-04-26T16:25:31.081658+0000","last_fullsized":"2024-04-26T16:25:31.081658+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:39.549794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:39.549794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:04.252074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"115'12","reported_seq":514,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095432+0000","last_change":"2024-04-26T16:24:32.795646+0000","last_active":"2024-04-26T16:25:31.095432+0000","last_peered":"2024-04-26T16:25:31.095432+0000","last_clean":"2024-04-26T16:25:31.095432+0000","last_became_active":"2024-04-26T16:17:33.070130+0000","last_became_peered":"2024-04-26T16:17:33.070130+0000","last_unstale":"2024-04-26T16:25:31.095432+0000","last_undegraded":"2024-04-26T16:25:31.095432+0000","last_fullsized":"2024-04-26T16:25:31.095432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'12","last_scrub_stamp":"2024-04-26T16:24:32.795341+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:24:32.795341+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:41.456233+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128691+0000","last_change":"2024-04-26T16:24:27.350110+0000","last_active":"2024-04-26T16:25:31.128691+0000","last_peered":"2024-04-26T16:25:31.128691+0000","last_clean":"2024-04-26T16:25:31.128691+0000","last_became_active":"2024-04-26T16:17:35.077832+0000","last_became_peered":"2024-04-26T16:17:35.077832+0000","last_unstale":"2024-04-26T16:25:31.128691+0000","last_undegraded":"2024-04-26T16:25:31.128691+0000","last_fullsized":"2024-04-26T16:25:31.128691+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:27.349541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:24:27.349541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:35.797464+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.6","version":"217'13","reported_seq":332,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086762+0000","last_change":"2024-04-26T16:24:49.093776+0000","last_active":"2024-04-26T16:25:31.086762+0000","last_peered":"2024-04-26T16:25:31.086762+0000","last_clean":"2024-04-26T16:25:31.086762+0000","last_became_active":"2024-04-26T16:24:49.093346+0000","last_became_peered":"2024-04-26T16:24:49.093346+0000","last_unstale":"2024-04-26T16:25:31.086762+0000","last_undegraded":"2024-04-26T16:25:31.086762+0000","last_fullsized":"2024-04-26T16:25:31.086762+0000","mapping_epoch":140,"log_start":"217'13","ondisk_log_start":"217'13","created":140,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":18,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":275,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.113504+0000","last_change":"2024-04-26T16:24:58.683427+0000","last_active":"2024-04-26T16:25:31.113504+0000","last_peered":"2024-04-26T16:25:31.113504+0000","last_clean":"2024-04-26T16:25:31.113504+0000","last_became_active":"2024-04-26T16:23:29.334960+0000","last_became_peered":"2024-04-26T16:23:29.334960+0000","last_unstale":"2024-04-26T16:25:31.113504+0000","last_undegraded":"2024-04-26T16:25:31.113504+0000","last_fullsized":"2024-04-26T16:25:31.113504+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:24:58.683180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:58.683180+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.730706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"86'12","reported_seq":433,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094729+0000","last_change":"2024-04-26T16:24:50.816864+0000","last_active":"2024-04-26T16:25:31.094729+0000","last_peered":"2024-04-26T16:25:31.094729+0000","last_clean":"2024-04-26T16:25:31.094729+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:25:31.094729+0000","last_undegraded":"2024-04-26T16:25:31.094729+0000","last_fullsized":"2024-04-26T16:25:31.094729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:24:50.816601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:50.816601+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:19.960490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.133836+0000","last_change":"2024-04-26T16:24:32.644198+0000","last_active":"2024-04-26T16:25:31.133836+0000","last_peered":"2024-04-26T16:25:31.133836+0000","last_clean":"2024-04-26T16:25:31.133836+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:25:31.133836+0000","last_undegraded":"2024-04-26T16:25:31.133836+0000","last_fullsized":"2024-04-26T16:25:31.133836+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:24:32.643939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:32.643939+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:58.227461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.152982+0000","last_change":"2024-04-26T16:24:44.421633+0000","last_active":"2024-04-26T16:25:31.152982+0000","last_peered":"2024-04-26T16:25:31.152982+0000","last_clean":"2024-04-26T16:25:31.152982+0000","last_became_active":"2024-04-26T16:17:33.068923+0000","last_became_peered":"2024-04-26T16:17:33.068923+0000","last_unstale":"2024-04-26T16:25:31.152982+0000","last_undegraded":"2024-04-26T16:25:31.152982+0000","last_fullsized":"2024-04-26T16:25:31.152982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:44.421411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:24:44.421411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:47.860473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.101795+0000","last_change":"2024-04-26T16:24:25.959890+0000","last_active":"2024-04-26T16:25:31.101795+0000","last_peered":"2024-04-26T16:25:31.101795+0000","last_clean":"2024-04-26T16:25:31.101795+0000","last_became_active":"2024-04-26T16:17:35.673130+0000","last_became_peered":"2024-04-26T16:17:35.673130+0000","last_unstale":"2024-04-26T16:25:31.101795+0000","last_undegraded":"2024-04-26T16:25:31.101795+0000","last_fullsized":"2024-04-26T16:25:31.101795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:25.959789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:25.959789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":317,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.080393+0000","last_change":"2024-04-26T16:24:55.235127+0000","last_active":"2024-04-26T16:25:31.080393+0000","last_peered":"2024-04-26T16:25:31.080393+0000","last_clean":"2024-04-26T16:25:31.080393+0000","last_became_active":"2024-04-26T16:24:05.172609+0000","last_became_peered":"2024-04-26T16:24:05.172609+0000","last_unstale":"2024-04-26T16:25:31.080393+0000","last_undegraded":"2024-04-26T16:25:31.080393+0000","last_fullsized":"2024-04-26T16:25:31.080393+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:55.235020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:55.235020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.488202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095718+0000","last_change":"2024-04-26T16:23:29.318651+0000","last_active":"2024-04-26T16:25:31.095718+0000","last_peered":"2024-04-26T16:25:31.095718+0000","last_clean":"2024-04-26T16:25:31.095718+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:25:31.095718+0000","last_undegraded":"2024-04-26T16:25:31.095718+0000","last_fullsized":"2024-04-26T16:25:31.095718+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":512,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095297+0000","last_change":"2024-04-26T16:24:39.048494+0000","last_active":"2024-04-26T16:25:31.095297+0000","last_peered":"2024-04-26T16:25:31.095297+0000","last_clean":"2024-04-26T16:25:31.095297+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:25:31.095297+0000","last_undegraded":"2024-04-26T16:25:31.095297+0000","last_fullsized":"2024-04-26T16:25:31.095297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:24:39.048148+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:24:39.048148+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:07.664043+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":415,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095299+0000","last_change":"2024-04-26T16:23:13.552842+0000","last_active":"2024-04-26T16:25:31.095299+0000","last_peered":"2024-04-26T16:25:31.095299+0000","last_clean":"2024-04-26T16:25:31.095299+0000","last_became_active":"2024-04-26T16:17:32.027388+0000","last_became_peered":"2024-04-26T16:17:32.027388+0000","last_unstale":"2024-04-26T16:25:31.095299+0000","last_undegraded":"2024-04-26T16:25:31.095299+0000","last_fullsized":"2024-04-26T16:25:31.095299+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:13.552494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:13.552494+0000","last_clean_scrub_stamp":"2024-04-26T16:23:13.552494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"107'18","reported_seq":439,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.106542+0000","last_change":"2024-04-26T16:24:44.374273+0000","last_active":"2024-04-26T16:25:31.106542+0000","last_peered":"2024-04-26T16:25:31.106542+0000","last_clean":"2024-04-26T16:25:31.106542+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:25:31.106542+0000","last_undegraded":"2024-04-26T16:25:31.106542+0000","last_fullsized":"2024-04-26T16:25:31.106542+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'18","last_scrub_stamp":"2024-04-26T16:24:44.374010+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:24:44.374010+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:49.865802+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":419,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128738+0000","last_change":"2024-04-26T16:24:46.494549+0000","last_active":"2024-04-26T16:25:31.128738+0000","last_peered":"2024-04-26T16:25:31.128738+0000","last_clean":"2024-04-26T16:25:31.128738+0000","last_became_active":"2024-04-26T16:17:29.006985+0000","last_became_peered":"2024-04-26T16:17:29.006985+0000","last_unstale":"2024-04-26T16:25:31.128738+0000","last_undegraded":"2024-04-26T16:25:31.128738+0000","last_fullsized":"2024-04-26T16:25:31.128738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:46.494451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:24:46.494451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.432703+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.131544+0000","last_change":"2024-04-26T16:24:31.596973+0000","last_active":"2024-04-26T16:25:31.131544+0000","last_peered":"2024-04-26T16:25:31.131544+0000","last_clean":"2024-04-26T16:25:31.131544+0000","last_became_active":"2024-04-26T16:17:35.670672+0000","last_became_peered":"2024-04-26T16:17:35.670672+0000","last_unstale":"2024-04-26T16:25:31.131544+0000","last_undegraded":"2024-04-26T16:25:31.131544+0000","last_fullsized":"2024-04-26T16:25:31.131544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:31.596755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:31.596755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:48.623860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.8","version":"141'2","reported_seq":311,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095049+0000","last_change":"2024-04-26T16:24:09.140563+0000","last_active":"2024-04-26T16:25:31.095049+0000","last_peered":"2024-04-26T16:25:31.095049+0000","last_clean":"2024-04-26T16:25:31.095049+0000","last_became_active":"2024-04-26T16:24:09.140205+0000","last_became_peered":"2024-04-26T16:24:09.140205+0000","last_unstale":"2024-04-26T16:25:31.095049+0000","last_undegraded":"2024-04-26T16:25:31.095049+0000","last_fullsized":"2024-04-26T16:25:31.095049+0000","mapping_epoch":140,"log_start":"141'2","ondisk_log_start":"141'2","created":140,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":4,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.154485+0000","last_change":"2024-04-26T16:23:29.352011+0000","last_active":"2024-04-26T16:25:31.154485+0000","last_peered":"2024-04-26T16:25:31.154485+0000","last_clean":"2024-04-26T16:25:31.154485+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:25:31.154485+0000","last_undegraded":"2024-04-26T16:25:31.154485+0000","last_fullsized":"2024-04-26T16:25:31.154485+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":406,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094420+0000","last_change":"2024-04-26T16:24:48.748118+0000","last_active":"2024-04-26T16:25:31.094420+0000","last_peered":"2024-04-26T16:25:31.094420+0000","last_clean":"2024-04-26T16:25:31.094420+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:25:31.094420+0000","last_undegraded":"2024-04-26T16:25:31.094420+0000","last_fullsized":"2024-04-26T16:25:31.094420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:48.747802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:24:48.747802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:53.011219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":877,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.095196+0000","last_change":"2024-04-26T16:24:29.992556+0000","last_active":"2024-04-26T16:25:31.095196+0000","last_peered":"2024-04-26T16:25:31.095196+0000","last_clean":"2024-04-26T16:25:31.095196+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:25:31.095196+0000","last_undegraded":"2024-04-26T16:25:31.095196+0000","last_fullsized":"2024-04-26T16:25:31.095196+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:24:29.992326+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:24:29.992326+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:33.835674+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"121'18","reported_seq":440,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103751+0000","last_change":"2024-04-26T16:24:43.903168+0000","last_active":"2024-04-26T16:25:31.103751+0000","last_peered":"2024-04-26T16:25:31.103751+0000","last_clean":"2024-04-26T16:25:31.103751+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:25:31.103751+0000","last_undegraded":"2024-04-26T16:25:31.103751+0000","last_fullsized":"2024-04-26T16:25:31.103751+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'18","last_scrub_stamp":"2024-04-26T16:24:43.903026+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:24:43.903026+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.814807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":416,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086796+0000","last_change":"2024-04-26T16:23:08.126537+0000","last_active":"2024-04-26T16:25:31.086796+0000","last_peered":"2024-04-26T16:25:31.086796+0000","last_clean":"2024-04-26T16:25:31.086796+0000","last_became_active":"2024-04-26T16:17:29.004853+0000","last_became_peered":"2024-04-26T16:17:29.004853+0000","last_unstale":"2024-04-26T16:25:31.086796+0000","last_undegraded":"2024-04-26T16:25:31.086796+0000","last_fullsized":"2024-04-26T16:25:31.086796+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:23:08.126400+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:23:08.126400+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":402,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.089572+0000","last_change":"2024-04-26T16:23:10.458085+0000","last_active":"2024-04-26T16:25:31.089572+0000","last_peered":"2024-04-26T16:25:31.089572+0000","last_clean":"2024-04-26T16:25:31.089572+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:25:31.089572+0000","last_undegraded":"2024-04-26T16:25:31.089572+0000","last_fullsized":"2024-04-26T16:25:31.089572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:10.457864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:23:10.457864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084190+0000","last_change":"2024-04-26T16:23:29.285467+0000","last_active":"2024-04-26T16:25:31.084190+0000","last_peered":"2024-04-26T16:25:31.084190+0000","last_clean":"2024-04-26T16:25:31.084190+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:25:31.084190+0000","last_undegraded":"2024-04-26T16:25:31.084190+0000","last_fullsized":"2024-04-26T16:25:31.084190+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":411,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.125579+0000","last_change":"2024-04-26T16:24:55.645085+0000","last_active":"2024-04-26T16:25:31.125579+0000","last_peered":"2024-04-26T16:25:31.125579+0000","last_clean":"2024-04-26T16:25:31.125579+0000","last_became_active":"2024-04-26T16:17:33.057789+0000","last_became_peered":"2024-04-26T16:17:33.057789+0000","last_unstale":"2024-04-26T16:25:31.125579+0000","last_undegraded":"2024-04-26T16:25:31.125579+0000","last_fullsized":"2024-04-26T16:25:31.125579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:55.644951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:24:55.644951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:17.519963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"107'40","reported_seq":567,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084270+0000","last_change":"2024-04-26T16:23:12.114448+0000","last_active":"2024-04-26T16:25:31.084270+0000","last_peered":"2024-04-26T16:25:31.084270+0000","last_clean":"2024-04-26T16:25:31.084270+0000","last_became_active":"2024-04-26T16:17:31.024426+0000","last_became_peered":"2024-04-26T16:17:31.024426+0000","last_unstale":"2024-04-26T16:25:31.084270+0000","last_undegraded":"2024-04-26T16:25:31.084270+0000","last_fullsized":"2024-04-26T16:25:31.084270+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'40","last_scrub_stamp":"2024-04-26T16:23:12.114300+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:23:12.114300+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.147858+0000","last_change":"2024-04-26T16:25:03.657007+0000","last_active":"2024-04-26T16:25:31.147858+0000","last_peered":"2024-04-26T16:25:31.147858+0000","last_clean":"2024-04-26T16:25:31.147858+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:25:31.147858+0000","last_undegraded":"2024-04-26T16:25:31.147858+0000","last_fullsized":"2024-04-26T16:25:31.147858+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:25:03.656843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:03.656843+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:27.626855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":410,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081779+0000","last_change":"2024-04-26T16:24:36.579624+0000","last_active":"2024-04-26T16:25:31.081779+0000","last_peered":"2024-04-26T16:25:31.081779+0000","last_clean":"2024-04-26T16:25:31.081779+0000","last_became_active":"2024-04-26T16:17:35.673254+0000","last_became_peered":"2024-04-26T16:17:35.673254+0000","last_unstale":"2024-04-26T16:25:31.081779+0000","last_undegraded":"2024-04-26T16:25:31.081779+0000","last_fullsized":"2024-04-26T16:25:31.081779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:36.579507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:36.579507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.495320+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.098723+0000","last_change":"2024-04-26T16:23:29.231375+0000","last_active":"2024-04-26T16:25:31.098723+0000","last_peered":"2024-04-26T16:25:31.098723+0000","last_clean":"2024-04-26T16:25:31.098723+0000","last_became_active":"2024-04-26T16:23:29.230684+0000","last_became_peered":"2024-04-26T16:23:29.230684+0000","last_unstale":"2024-04-26T16:25:31.098723+0000","last_undegraded":"2024-04-26T16:25:31.098723+0000","last_fullsized":"2024-04-26T16:25:31.098723+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.152910+0000","last_change":"2024-04-26T16:24:35.443362+0000","last_active":"2024-04-26T16:25:31.152910+0000","last_peered":"2024-04-26T16:25:31.152910+0000","last_clean":"2024-04-26T16:25:31.152910+0000","last_became_active":"2024-04-26T16:17:33.046107+0000","last_became_peered":"2024-04-26T16:17:33.046107+0000","last_unstale":"2024-04-26T16:25:31.152910+0000","last_undegraded":"2024-04-26T16:25:31.152910+0000","last_fullsized":"2024-04-26T16:25:31.152910+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:35.443125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:35.443125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:51.306100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.133479+0000","last_change":"2024-04-26T16:24:45.666044+0000","last_active":"2024-04-26T16:25:31.133479+0000","last_peered":"2024-04-26T16:25:31.133479+0000","last_clean":"2024-04-26T16:25:31.133479+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:25:31.133479+0000","last_undegraded":"2024-04-26T16:25:31.133479+0000","last_fullsized":"2024-04-26T16:25:31.133479+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:45.665745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.665745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.092587+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"112'21","reported_seq":443,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.098977+0000","last_change":"2024-04-26T16:24:37.356098+0000","last_active":"2024-04-26T16:25:31.098977+0000","last_peered":"2024-04-26T16:25:31.098977+0000","last_clean":"2024-04-26T16:25:31.098977+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:25:31.098977+0000","last_undegraded":"2024-04-26T16:25:31.098977+0000","last_fullsized":"2024-04-26T16:25:31.098977+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'21","last_scrub_stamp":"2024-04-26T16:24:37.355703+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:24:37.355703+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:04.259684+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":402,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084795+0000","last_change":"2024-04-26T16:24:51.295435+0000","last_active":"2024-04-26T16:25:31.084795+0000","last_peered":"2024-04-26T16:25:31.084795+0000","last_clean":"2024-04-26T16:25:31.084795+0000","last_became_active":"2024-04-26T16:17:35.090592+0000","last_became_peered":"2024-04-26T16:17:35.090592+0000","last_unstale":"2024-04-26T16:25:31.084795+0000","last_undegraded":"2024-04-26T16:25:31.084795+0000","last_fullsized":"2024-04-26T16:25:31.084795+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:51.295336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:24:51.295336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:03.987629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.102170+0000","last_change":"2024-04-26T16:23:29.298916+0000","last_active":"2024-04-26T16:25:31.102170+0000","last_peered":"2024-04-26T16:25:31.102170+0000","last_clean":"2024-04-26T16:25:31.102170+0000","last_became_active":"2024-04-26T16:23:29.298328+0000","last_became_peered":"2024-04-26T16:23:29.298328+0000","last_unstale":"2024-04-26T16:25:31.102170+0000","last_undegraded":"2024-04-26T16:25:31.102170+0000","last_fullsized":"2024-04-26T16:25:31.102170+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":396,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.134046+0000","last_change":"2024-04-26T16:22:47.335520+0000","last_active":"2024-04-26T16:25:31.134046+0000","last_peered":"2024-04-26T16:25:31.134046+0000","last_clean":"2024-04-26T16:25:31.134046+0000","last_became_active":"2024-04-26T16:17:33.061816+0000","last_became_peered":"2024-04-26T16:17:33.061816+0000","last_unstale":"2024-04-26T16:25:31.134046+0000","last_undegraded":"2024-04-26T16:25:31.134046+0000","last_fullsized":"2024-04-26T16:25:31.134046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:22:47.335312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:22:47.335312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":692,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.133646+0000","last_change":"2024-04-26T16:23:29.377810+0000","last_active":"2024-04-26T16:25:31.133646+0000","last_peered":"2024-04-26T16:25:31.133646+0000","last_clean":"2024-04-26T16:25:31.133646+0000","last_became_active":"2024-04-26T16:16:50.925983+0000","last_became_peered":"2024-04-26T16:16:50.925983+0000","last_unstale":"2024-04-26T16:25:31.133646+0000","last_undegraded":"2024-04-26T16:25:31.133646+0000","last_fullsized":"2024-04-26T16:25:31.133646+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:23:29.377635+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:23:29.377635+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759856+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.148026+0000","last_change":"2024-04-26T16:25:06.651725+0000","last_active":"2024-04-26T16:25:31.148026+0000","last_peered":"2024-04-26T16:25:31.148026+0000","last_clean":"2024-04-26T16:25:31.148026+0000","last_became_active":"2024-04-26T16:17:29.013835+0000","last_became_peered":"2024-04-26T16:17:29.013835+0000","last_unstale":"2024-04-26T16:25:31.148026+0000","last_undegraded":"2024-04-26T16:25:31.148026+0000","last_fullsized":"2024-04-26T16:25:31.148026+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:06.651513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:06.651513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.342175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"124'15","reported_seq":436,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.102284+0000","last_change":"2024-04-26T16:25:08.969306+0000","last_active":"2024-04-26T16:25:31.102284+0000","last_peered":"2024-04-26T16:25:31.102284+0000","last_clean":"2024-04-26T16:25:31.102284+0000","last_became_active":"2024-04-26T16:17:31.048785+0000","last_became_peered":"2024-04-26T16:17:31.048785+0000","last_unstale":"2024-04-26T16:25:31.102284+0000","last_undegraded":"2024-04-26T16:25:31.102284+0000","last_fullsized":"2024-04-26T16:25:31.102284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'15","last_scrub_stamp":"2024-04-26T16:25:08.969060+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:25:08.969060+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.284297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.153077+0000","last_change":"2024-04-26T16:24:48.448353+0000","last_active":"2024-04-26T16:25:31.153077+0000","last_peered":"2024-04-26T16:25:31.153077+0000","last_clean":"2024-04-26T16:25:31.153077+0000","last_became_active":"2024-04-26T16:17:35.087291+0000","last_became_peered":"2024-04-26T16:17:35.087291+0000","last_unstale":"2024-04-26T16:25:31.153077+0000","last_undegraded":"2024-04-26T16:25:31.153077+0000","last_fullsized":"2024-04-26T16:25:31.153077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:48.448237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:48.448237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:56.229488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":263,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094639+0000","last_change":"2024-04-26T16:25:04.068276+0000","last_active":"2024-04-26T16:25:31.094639+0000","last_peered":"2024-04-26T16:25:31.094639+0000","last_clean":"2024-04-26T16:25:31.094639+0000","last_became_active":"2024-04-26T16:23:29.292434+0000","last_became_peered":"2024-04-26T16:23:29.292434+0000","last_unstale":"2024-04-26T16:25:31.094639+0000","last_undegraded":"2024-04-26T16:25:31.094639+0000","last_fullsized":"2024-04-26T16:25:31.094639+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:04.068168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:04.068168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.539186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083715+0000","last_change":"2024-04-26T16:24:29.964695+0000","last_active":"2024-04-26T16:25:31.083715+0000","last_peered":"2024-04-26T16:25:31.083715+0000","last_clean":"2024-04-26T16:25:31.083715+0000","last_became_active":"2024-04-26T16:17:33.066780+0000","last_became_peered":"2024-04-26T16:17:33.066780+0000","last_unstale":"2024-04-26T16:25:31.083715+0000","last_undegraded":"2024-04-26T16:25:31.083715+0000","last_fullsized":"2024-04-26T16:25:31.083715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:29.964476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:29.964476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.079407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081203+0000","last_change":"2024-04-26T16:24:33.969174+0000","last_active":"2024-04-26T16:25:31.081203+0000","last_peered":"2024-04-26T16:25:31.081203+0000","last_clean":"2024-04-26T16:25:31.081203+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:25:31.081203+0000","last_undegraded":"2024-04-26T16:25:31.081203+0000","last_fullsized":"2024-04-26T16:25:31.081203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:33.969041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:24:33.969041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:50.976751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"112'26","reported_seq":458,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.083626+0000","last_change":"2024-04-26T16:24:34.463573+0000","last_active":"2024-04-26T16:25:31.083626+0000","last_peered":"2024-04-26T16:25:31.083626+0000","last_clean":"2024-04-26T16:25:31.083626+0000","last_became_active":"2024-04-26T16:17:31.023021+0000","last_became_peered":"2024-04-26T16:17:31.023021+0000","last_unstale":"2024-04-26T16:25:31.083626+0000","last_undegraded":"2024-04-26T16:25:31.083626+0000","last_fullsized":"2024-04-26T16:25:31.083626+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-04-26T16:24:34.463446+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_clean_scrub_stamp":"2024-04-26T16:24:34.463446+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:47.905333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.147818+0000","last_change":"2024-04-26T16:24:42.751470+0000","last_active":"2024-04-26T16:25:31.147818+0000","last_peered":"2024-04-26T16:25:31.147818+0000","last_clean":"2024-04-26T16:25:31.147818+0000","last_became_active":"2024-04-26T16:17:35.670585+0000","last_became_peered":"2024-04-26T16:17:35.670585+0000","last_unstale":"2024-04-26T16:25:31.147818+0000","last_undegraded":"2024-04-26T16:25:31.147818+0000","last_fullsized":"2024-04-26T16:25:31.147818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:42.751183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:42.751183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:47.720265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100278+0000","last_change":"2024-04-26T16:23:29.222598+0000","last_active":"2024-04-26T16:25:31.100278+0000","last_peered":"2024-04-26T16:25:31.100278+0000","last_clean":"2024-04-26T16:25:31.100278+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:25:31.100278+0000","last_undegraded":"2024-04-26T16:25:31.100278+0000","last_fullsized":"2024-04-26T16:25:31.100278+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":411,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109491+0000","last_change":"2024-04-26T16:24:56.402531+0000","last_active":"2024-04-26T16:25:31.109491+0000","last_peered":"2024-04-26T16:25:31.109491+0000","last_clean":"2024-04-26T16:25:31.109491+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:25:31.109491+0000","last_undegraded":"2024-04-26T16:25:31.109491+0000","last_fullsized":"2024-04-26T16:25:31.109491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:56.402418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:56.402418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:00.842608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"103'27","reported_seq":454,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.090559+0000","last_change":"2024-04-26T16:24:34.832910+0000","last_active":"2024-04-26T16:25:31.090559+0000","last_peered":"2024-04-26T16:25:31.090559+0000","last_clean":"2024-04-26T16:25:31.090559+0000","last_became_active":"2024-04-26T16:17:31.030904+0000","last_became_peered":"2024-04-26T16:17:31.030904+0000","last_unstale":"2024-04-26T16:25:31.090559+0000","last_undegraded":"2024-04-26T16:25:31.090559+0000","last_fullsized":"2024-04-26T16:25:31.090559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'27","last_scrub_stamp":"2024-04-26T16:24:34.832776+0000","last_deep_scrub":"103'27","last_deep_scrub_stamp":"2024-04-26T16:18:40.947097+0000","last_clean_scrub_stamp":"2024-04-26T16:24:34.832776+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:46.113575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.123634+0000","last_change":"2024-04-26T16:24:59.573837+0000","last_active":"2024-04-26T16:25:31.123634+0000","last_peered":"2024-04-26T16:25:31.123634+0000","last_clean":"2024-04-26T16:25:31.123634+0000","last_became_active":"2024-04-26T16:17:29.022734+0000","last_became_peered":"2024-04-26T16:17:29.022734+0000","last_unstale":"2024-04-26T16:25:31.123634+0000","last_undegraded":"2024-04-26T16:25:31.123634+0000","last_fullsized":"2024-04-26T16:25:31.123634+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:59.573732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.573732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:06.169182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":408,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103275+0000","last_change":"2024-04-26T16:24:46.914560+0000","last_active":"2024-04-26T16:25:31.103275+0000","last_peered":"2024-04-26T16:25:31.103275+0000","last_clean":"2024-04-26T16:25:31.103275+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:25:31.103275+0000","last_undegraded":"2024-04-26T16:25:31.103275+0000","last_fullsized":"2024-04-26T16:25:31.103275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:46.914314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:46.914314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.491836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.a","version":"217'16","reported_seq":333,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086494+0000","last_change":"2024-04-26T16:24:58.265966+0000","last_active":"2024-04-26T16:25:31.086494+0000","last_peered":"2024-04-26T16:25:31.086494+0000","last_clean":"2024-04-26T16:25:31.086494+0000","last_became_active":"2024-04-26T16:24:01.155043+0000","last_became_peered":"2024-04-26T16:24:01.155043+0000","last_unstale":"2024-04-26T16:25:31.086494+0000","last_undegraded":"2024-04-26T16:25:31.086494+0000","last_fullsized":"2024-04-26T16:25:31.086494+0000","mapping_epoch":140,"log_start":"141'10","ondisk_log_start":"141'10","created":140,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'16","last_scrub_stamp":"2024-04-26T16:24:58.265769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:58.265769+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":15,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:20.478241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086988+0000","last_change":"2024-04-26T16:23:29.219193+0000","last_active":"2024-04-26T16:25:31.086988+0000","last_peered":"2024-04-26T16:25:31.086988+0000","last_clean":"2024-04-26T16:25:31.086988+0000","last_became_active":"2024-04-26T16:23:29.218856+0000","last_became_peered":"2024-04-26T16:23:29.218856+0000","last_unstale":"2024-04-26T16:25:31.086988+0000","last_undegraded":"2024-04-26T16:25:31.086988+0000","last_fullsized":"2024-04-26T16:25:31.086988+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:35.160523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"86'11","reported_seq":521,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.112766+0000","last_change":"2024-04-26T16:24:39.342028+0000","last_active":"2024-04-26T16:25:31.112766+0000","last_peered":"2024-04-26T16:25:31.112766+0000","last_clean":"2024-04-26T16:25:31.112766+0000","last_became_active":"2024-04-26T16:17:33.069020+0000","last_became_peered":"2024-04-26T16:17:33.069020+0000","last_unstale":"2024-04-26T16:25:31.112766+0000","last_undegraded":"2024-04-26T16:25:31.112766+0000","last_fullsized":"2024-04-26T16:25:31.112766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:39.341918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:39.341918+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:07.980726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":417,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.082964+0000","last_change":"2024-04-26T16:23:04.126208+0000","last_active":"2024-04-26T16:25:31.082964+0000","last_peered":"2024-04-26T16:25:31.082964+0000","last_clean":"2024-04-26T16:25:31.082964+0000","last_became_active":"2024-04-26T16:17:32.046437+0000","last_became_peered":"2024-04-26T16:17:32.046437+0000","last_unstale":"2024-04-26T16:25:31.082964+0000","last_undegraded":"2024-04-26T16:25:31.082964+0000","last_fullsized":"2024-04-26T16:25:31.082964+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:23:04.126063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:23:04.126063+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"86'16","reported_seq":527,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.126625+0000","last_change":"2024-04-26T16:23:29.354344+0000","last_active":"2024-04-26T16:25:31.126625+0000","last_peered":"2024-04-26T16:25:31.126625+0000","last_clean":"2024-04-26T16:25:31.126625+0000","last_became_active":"2024-04-26T16:17:31.040723+0000","last_became_peered":"2024-04-26T16:17:31.040723+0000","last_unstale":"2024-04-26T16:25:31.126625+0000","last_undegraded":"2024-04-26T16:25:31.126625+0000","last_fullsized":"2024-04-26T16:25:31.126625+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'16","last_scrub_stamp":"2024-04-26T16:23:29.354187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:23:29.354187+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139931+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.084354+0000","last_change":"2024-04-26T16:23:23.108688+0000","last_active":"2024-04-26T16:25:31.084354+0000","last_peered":"2024-04-26T16:25:31.084354+0000","last_clean":"2024-04-26T16:25:31.084354+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:25:31.084354+0000","last_undegraded":"2024-04-26T16:25:31.084354+0000","last_fullsized":"2024-04-26T16:25:31.084354+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:23:23.108346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":399,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100615+0000","last_change":"2024-04-26T16:24:26.307811+0000","last_active":"2024-04-26T16:25:31.100615+0000","last_peered":"2024-04-26T16:25:31.100615+0000","last_clean":"2024-04-26T16:25:31.100615+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:25:31.100615+0000","last_undegraded":"2024-04-26T16:25:31.100615+0000","last_fullsized":"2024-04-26T16:25:31.100615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:26.307698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:26.307698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:37.952408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1","version":"217'11","reported_seq":325,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.112730+0000","last_change":"2024-04-26T16:25:00.681375+0000","last_active":"2024-04-26T16:25:31.112730+0000","last_peered":"2024-04-26T16:25:31.112730+0000","last_clean":"2024-04-26T16:25:31.112730+0000","last_became_active":"2024-04-26T16:24:37.094649+0000","last_became_peered":"2024-04-26T16:24:37.094649+0000","last_unstale":"2024-04-26T16:25:31.112730+0000","last_undegraded":"2024-04-26T16:25:31.112730+0000","last_fullsized":"2024-04-26T16:25:31.112730+0000","mapping_epoch":140,"log_start":"141'7","ondisk_log_start":"141'7","created":140,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'11","last_scrub_stamp":"2024-04-26T16:25:00.681122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:00.681122+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":12,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.839202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.106442+0000","last_change":"2024-04-26T16:25:04.342584+0000","last_active":"2024-04-26T16:25:31.106442+0000","last_peered":"2024-04-26T16:25:31.106442+0000","last_clean":"2024-04-26T16:25:31.106442+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:25:31.106442+0000","last_undegraded":"2024-04-26T16:25:31.106442+0000","last_fullsized":"2024-04-26T16:25:31.106442+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:04.342503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:04.342503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:22.175143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.090322+0000","last_change":"2024-04-26T16:24:36.929034+0000","last_active":"2024-04-26T16:25:31.090322+0000","last_peered":"2024-04-26T16:25:31.090322+0000","last_clean":"2024-04-26T16:25:31.090322+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:25:31.090322+0000","last_undegraded":"2024-04-26T16:25:31.090322+0000","last_fullsized":"2024-04-26T16:25:31.090322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:36.928884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:36.928884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:49.804547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"86'11","reported_seq":427,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.145148+0000","last_change":"2024-04-26T16:23:01.365389+0000","last_active":"2024-04-26T16:25:31.145148+0000","last_peered":"2024-04-26T16:25:31.145148+0000","last_clean":"2024-04-26T16:25:31.145148+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:25:31.145148+0000","last_undegraded":"2024-04-26T16:25:31.145148+0000","last_fullsized":"2024-04-26T16:25:31.145148+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:23:01.365107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:23:01.365107+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":438,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.113967+0000","last_change":"2024-04-26T16:25:10.807777+0000","last_active":"2024-04-26T16:25:31.113967+0000","last_peered":"2024-04-26T16:25:31.113967+0000","last_clean":"2024-04-26T16:25:31.113967+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:25:31.113967+0000","last_undegraded":"2024-04-26T16:25:31.113967+0000","last_fullsized":"2024-04-26T16:25:31.113967+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:25:10.807661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:10.807661+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:19.853289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100797+0000","last_change":"2024-04-26T16:25:01.008927+0000","last_active":"2024-04-26T16:25:31.100797+0000","last_peered":"2024-04-26T16:25:31.100797+0000","last_clean":"2024-04-26T16:25:31.100797+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:25:31.100797+0000","last_undegraded":"2024-04-26T16:25:31.100797+0000","last_fullsized":"2024-04-26T16:25:31.100797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:01.008836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:25:01.008836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.594170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.098774+0000","last_change":"2024-04-26T16:24:54.336258+0000","last_active":"2024-04-26T16:25:31.098774+0000","last_peered":"2024-04-26T16:25:31.098774+0000","last_clean":"2024-04-26T16:25:31.098774+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:25:31.098774+0000","last_undegraded":"2024-04-26T16:25:31.098774+0000","last_fullsized":"2024-04-26T16:25:31.098774+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:54.336163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:54.336163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.640782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"86'11","reported_seq":427,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.141979+0000","last_change":"2024-04-26T16:24:56.661419+0000","last_active":"2024-04-26T16:25:31.141979+0000","last_peered":"2024-04-26T16:25:31.141979+0000","last_clean":"2024-04-26T16:25:31.141979+0000","last_became_active":"2024-04-26T16:17:31.033294+0000","last_became_peered":"2024-04-26T16:17:31.033294+0000","last_unstale":"2024-04-26T16:25:31.141979+0000","last_undegraded":"2024-04-26T16:25:31.141979+0000","last_fullsized":"2024-04-26T16:25:31.141979+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:56.661177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:56.661177+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:11.403066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088242+0000","last_change":"2024-04-26T16:24:38.945746+0000","last_active":"2024-04-26T16:25:31.088242+0000","last_peered":"2024-04-26T16:25:31.088242+0000","last_clean":"2024-04-26T16:25:31.088242+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:25:31.088242+0000","last_undegraded":"2024-04-26T16:25:31.088242+0000","last_fullsized":"2024-04-26T16:25:31.088242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:38.945639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:38.945639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.371290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109083+0000","last_change":"2024-04-26T16:24:30.989478+0000","last_active":"2024-04-26T16:25:31.109083+0000","last_peered":"2024-04-26T16:25:31.109083+0000","last_clean":"2024-04-26T16:25:31.109083+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:25:31.109083+0000","last_undegraded":"2024-04-26T16:25:31.109083+0000","last_fullsized":"2024-04-26T16:25:31.109083+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:30.989287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:40.032566+0000","last_clean_scrub_stamp":"2024-04-26T16:24:30.989287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:37.563096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":408,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088528+0000","last_change":"2024-04-26T16:24:42.021813+0000","last_active":"2024-04-26T16:25:31.088528+0000","last_peered":"2024-04-26T16:25:31.088528+0000","last_clean":"2024-04-26T16:25:31.088528+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:25:31.088528+0000","last_undegraded":"2024-04-26T16:25:31.088528+0000","last_fullsized":"2024-04-26T16:25:31.088528+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:42.021591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:42.021591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:10.232777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":263,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088078+0000","last_change":"2024-04-26T16:25:04.250677+0000","last_active":"2024-04-26T16:25:31.088078+0000","last_peered":"2024-04-26T16:25:31.088078+0000","last_clean":"2024-04-26T16:25:31.088078+0000","last_became_active":"2024-04-26T16:23:29.219590+0000","last_became_peered":"2024-04-26T16:23:29.219590+0000","last_unstale":"2024-04-26T16:25:31.088078+0000","last_undegraded":"2024-04-26T16:25:31.088078+0000","last_fullsized":"2024-04-26T16:25:31.088078+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:04.250597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:04.250597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.234681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"86'4","reported_seq":421,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103572+0000","last_change":"2024-04-26T16:24:43.423216+0000","last_active":"2024-04-26T16:25:31.103572+0000","last_peered":"2024-04-26T16:25:31.103572+0000","last_clean":"2024-04-26T16:25:31.103572+0000","last_became_active":"2024-04-26T16:17:31.026037+0000","last_became_peered":"2024-04-26T16:17:31.026037+0000","last_unstale":"2024-04-26T16:25:31.103572+0000","last_undegraded":"2024-04-26T16:25:31.103572+0000","last_fullsized":"2024-04-26T16:25:31.103572+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'4","last_scrub_stamp":"2024-04-26T16:24:43.423097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:43.423097+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:50.760472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":411,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.116391+0000","last_change":"2024-04-26T16:24:44.447804+0000","last_active":"2024-04-26T16:25:31.116391+0000","last_peered":"2024-04-26T16:25:31.116391+0000","last_clean":"2024-04-26T16:25:31.116391+0000","last_became_active":"2024-04-26T16:17:28.999634+0000","last_became_peered":"2024-04-26T16:17:28.999634+0000","last_unstale":"2024-04-26T16:25:31.116391+0000","last_undegraded":"2024-04-26T16:25:31.116391+0000","last_fullsized":"2024-04-26T16:25:31.116391+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:44.447603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:44.447603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.509056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.151853+0000","last_change":"2024-04-26T16:25:02.550958+0000","last_active":"2024-04-26T16:25:31.151853+0000","last_peered":"2024-04-26T16:25:31.151853+0000","last_clean":"2024-04-26T16:25:31.151853+0000","last_became_active":"2024-04-26T16:17:33.050011+0000","last_became_peered":"2024-04-26T16:17:33.050011+0000","last_unstale":"2024-04-26T16:25:31.151853+0000","last_undegraded":"2024-04-26T16:25:31.151853+0000","last_fullsized":"2024-04-26T16:25:31.151853+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.550844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.550844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:28.020700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":392,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.147749+0000","last_change":"2024-04-26T16:23:17.311832+0000","last_active":"2024-04-26T16:25:31.147749+0000","last_peered":"2024-04-26T16:25:31.147749+0000","last_clean":"2024-04-26T16:25:31.147749+0000","last_became_active":"2024-04-26T16:17:35.671754+0000","last_became_peered":"2024-04-26T16:17:35.671754+0000","last_unstale":"2024-04-26T16:25:31.147749+0000","last_undegraded":"2024-04-26T16:25:31.147749+0000","last_fullsized":"2024-04-26T16:25:31.147749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:17.311734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:23:17.311734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.760080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.089867+0000","last_change":"2024-04-26T16:23:29.295401+0000","last_active":"2024-04-26T16:25:31.089867+0000","last_peered":"2024-04-26T16:25:31.089867+0000","last_clean":"2024-04-26T16:25:31.089867+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:25:31.089867+0000","last_undegraded":"2024-04-26T16:25:31.089867+0000","last_fullsized":"2024-04-26T16:25:31.089867+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"86'11","reported_seq":427,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.116908+0000","last_change":"2024-04-26T16:23:23.251635+0000","last_active":"2024-04-26T16:25:31.116908+0000","last_peered":"2024-04-26T16:25:31.116908+0000","last_clean":"2024-04-26T16:25:31.116908+0000","last_became_active":"2024-04-26T16:17:31.034570+0000","last_became_peered":"2024-04-26T16:17:31.034570+0000","last_unstale":"2024-04-26T16:25:31.116908+0000","last_undegraded":"2024-04-26T16:25:31.116908+0000","last_fullsized":"2024-04-26T16:25:31.116908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:23:23.251301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:23:23.251301+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.760037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.123138+0000","last_change":"2024-04-26T16:23:22.126661+0000","last_active":"2024-04-26T16:25:31.123138+0000","last_peered":"2024-04-26T16:25:31.123138+0000","last_clean":"2024-04-26T16:25:31.123138+0000","last_became_active":"2024-04-26T16:17:29.025290+0000","last_became_peered":"2024-04-26T16:17:29.025290+0000","last_unstale":"2024-04-26T16:25:31.123138+0000","last_undegraded":"2024-04-26T16:25:31.123138+0000","last_fullsized":"2024-04-26T16:25:31.123138+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:22.126437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:23:22.126437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"101'12","reported_seq":520,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.129272+0000","last_change":"2024-04-26T16:24:38.428722+0000","last_active":"2024-04-26T16:25:31.129272+0000","last_peered":"2024-04-26T16:25:31.129272+0000","last_clean":"2024-04-26T16:25:31.129272+0000","last_became_active":"2024-04-26T16:17:33.062000+0000","last_became_peered":"2024-04-26T16:17:33.062000+0000","last_unstale":"2024-04-26T16:25:31.129272+0000","last_undegraded":"2024-04-26T16:25:31.129272+0000","last_fullsized":"2024-04-26T16:25:31.129272+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-26T16:24:38.428577+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:24:38.428577+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:06.832247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":399,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109328+0000","last_change":"2024-04-26T16:24:24.269149+0000","last_active":"2024-04-26T16:25:31.109328+0000","last_peered":"2024-04-26T16:25:31.109328+0000","last_clean":"2024-04-26T16:25:31.109328+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:25:31.109328+0000","last_undegraded":"2024-04-26T16:25:31.109328+0000","last_fullsized":"2024-04-26T16:25:31.109328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:24.269039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:24.269039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:38.083013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.123398+0000","last_change":"2024-04-26T16:25:08.692217+0000","last_active":"2024-04-26T16:25:31.123398+0000","last_peered":"2024-04-26T16:25:31.123398+0000","last_clean":"2024-04-26T16:25:31.123398+0000","last_became_active":"2024-04-26T16:23:29.289035+0000","last_became_peered":"2024-04-26T16:23:29.289035+0000","last_unstale":"2024-04-26T16:25:31.123398+0000","last_undegraded":"2024-04-26T16:25:31.123398+0000","last_fullsized":"2024-04-26T16:25:31.123398+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:08.691959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:08.691959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:21.089566+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"121'13","reported_seq":435,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.105844+0000","last_change":"2024-04-26T16:24:43.402954+0000","last_active":"2024-04-26T16:25:31.105844+0000","last_peered":"2024-04-26T16:25:31.105844+0000","last_clean":"2024-04-26T16:25:31.105844+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:25:31.105844+0000","last_undegraded":"2024-04-26T16:25:31.105844+0000","last_fullsized":"2024-04-26T16:25:31.105844+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'13","last_scrub_stamp":"2024-04-26T16:24:43.402743+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:24:43.402743+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:53.590988+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":419,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.110732+0000","last_change":"2024-04-26T16:24:47.352286+0000","last_active":"2024-04-26T16:25:31.110732+0000","last_peered":"2024-04-26T16:25:31.110732+0000","last_clean":"2024-04-26T16:25:31.110732+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:25:31.110732+0000","last_undegraded":"2024-04-26T16:25:31.110732+0000","last_fullsized":"2024-04-26T16:25:31.110732+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:47.352029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:47.352029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:01.876998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":511,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.101631+0000","last_change":"2024-04-26T16:24:49.977333+0000","last_active":"2024-04-26T16:25:31.101631+0000","last_peered":"2024-04-26T16:25:31.101631+0000","last_clean":"2024-04-26T16:25:31.101631+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:25:31.101631+0000","last_undegraded":"2024-04-26T16:25:31.101631+0000","last_fullsized":"2024-04-26T16:25:31.101631+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:49.977080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:49.977080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:54.324889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":402,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.115118+0000","last_change":"2024-04-26T16:24:49.773405+0000","last_active":"2024-04-26T16:25:31.115118+0000","last_peered":"2024-04-26T16:25:31.115118+0000","last_clean":"2024-04-26T16:25:31.115118+0000","last_became_active":"2024-04-26T16:17:35.070054+0000","last_became_peered":"2024-04-26T16:17:35.070054+0000","last_unstale":"2024-04-26T16:25:31.115118+0000","last_undegraded":"2024-04-26T16:25:31.115118+0000","last_fullsized":"2024-04-26T16:25:31.115118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:49.773260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:49.773260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:19.459322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.148152+0000","last_change":"2024-04-26T16:23:29.248282+0000","last_active":"2024-04-26T16:25:31.148152+0000","last_peered":"2024-04-26T16:25:31.148152+0000","last_clean":"2024-04-26T16:25:31.148152+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:25:31.148152+0000","last_undegraded":"2024-04-26T16:25:31.148152+0000","last_fullsized":"2024-04-26T16:25:31.148152+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.760124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"121'13","reported_seq":428,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.147960+0000","last_change":"2024-04-26T16:24:30.582687+0000","last_active":"2024-04-26T16:25:31.147960+0000","last_peered":"2024-04-26T16:25:31.147960+0000","last_clean":"2024-04-26T16:25:31.147960+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:25:31.147960+0000","last_undegraded":"2024-04-26T16:25:31.147960+0000","last_fullsized":"2024-04-26T16:25:31.147960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'13","last_scrub_stamp":"2024-04-26T16:24:30.582564+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:24:30.582564+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:52.989614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":419,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122646+0000","last_change":"2024-04-26T16:24:29.336470+0000","last_active":"2024-04-26T16:25:31.122646+0000","last_peered":"2024-04-26T16:25:31.122646+0000","last_clean":"2024-04-26T16:25:31.122646+0000","last_became_active":"2024-04-26T16:17:29.013639+0000","last_became_peered":"2024-04-26T16:17:29.013639+0000","last_unstale":"2024-04-26T16:25:31.122646+0000","last_undegraded":"2024-04-26T16:25:31.122646+0000","last_fullsized":"2024-04-26T16:25:31.122646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:29.336150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:24:29.336150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:56.339325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100689+0000","last_change":"2024-04-26T16:23:19.295036+0000","last_active":"2024-04-26T16:25:31.100689+0000","last_peered":"2024-04-26T16:25:31.100689+0000","last_clean":"2024-04-26T16:25:31.100689+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:25:31.100689+0000","last_undegraded":"2024-04-26T16:25:31.100689+0000","last_fullsized":"2024-04-26T16:25:31.100689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:19.294899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:23:19.294899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":407,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100463+0000","last_change":"2024-04-26T16:24:49.358018+0000","last_active":"2024-04-26T16:25:31.100463+0000","last_peered":"2024-04-26T16:25:31.100463+0000","last_clean":"2024-04-26T16:25:31.100463+0000","last_became_active":"2024-04-26T16:17:35.078105+0000","last_became_peered":"2024-04-26T16:17:35.078105+0000","last_unstale":"2024-04-26T16:25:31.100463+0000","last_undegraded":"2024-04-26T16:25:31.100463+0000","last_fullsized":"2024-04-26T16:25:31.100463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:49.357911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:24:49.357911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.981016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122433+0000","last_change":"2024-04-26T16:25:02.696524+0000","last_active":"2024-04-26T16:25:31.122433+0000","last_peered":"2024-04-26T16:25:31.122433+0000","last_clean":"2024-04-26T16:25:31.122433+0000","last_became_active":"2024-04-26T16:23:29.319769+0000","last_became_peered":"2024-04-26T16:23:29.319769+0000","last_unstale":"2024-04-26T16:25:31.122433+0000","last_undegraded":"2024-04-26T16:25:31.122433+0000","last_fullsized":"2024-04-26T16:25:31.122433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.696415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.696415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:20.188519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"86'10","reported_seq":425,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.090799+0000","last_change":"2024-04-26T16:24:35.794410+0000","last_active":"2024-04-26T16:25:31.090799+0000","last_peered":"2024-04-26T16:25:31.090799+0000","last_clean":"2024-04-26T16:25:31.090799+0000","last_became_active":"2024-04-26T16:17:31.021476+0000","last_became_peered":"2024-04-26T16:17:31.021476+0000","last_unstale":"2024-04-26T16:25:31.090799+0000","last_undegraded":"2024-04-26T16:25:31.090799+0000","last_fullsized":"2024-04-26T16:25:31.090799+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:24:35.794173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:35.794173+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:36.365073+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086983+0000","last_change":"2024-04-26T16:24:26.467129+0000","last_active":"2024-04-26T16:25:31.086983+0000","last_peered":"2024-04-26T16:25:31.086983+0000","last_clean":"2024-04-26T16:25:31.086983+0000","last_became_active":"2024-04-26T16:17:29.011346+0000","last_became_peered":"2024-04-26T16:17:29.011346+0000","last_unstale":"2024-04-26T16:25:31.086983+0000","last_undegraded":"2024-04-26T16:25:31.086983+0000","last_fullsized":"2024-04-26T16:25:31.086983+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:26.466980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:26.466980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:44.296564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":406,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.082826+0000","last_change":"2024-04-26T16:24:59.228582+0000","last_active":"2024-04-26T16:25:31.082826+0000","last_peered":"2024-04-26T16:25:31.082826+0000","last_clean":"2024-04-26T16:25:31.082826+0000","last_became_active":"2024-04-26T16:17:33.048202+0000","last_became_peered":"2024-04-26T16:17:33.048202+0000","last_unstale":"2024-04-26T16:25:31.082826+0000","last_undegraded":"2024-04-26T16:25:31.082826+0000","last_fullsized":"2024-04-26T16:25:31.082826+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:59.228337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.228337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:10.370040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.096050+0000","last_change":"2024-04-26T16:23:24.330068+0000","last_active":"2024-04-26T16:25:31.096050+0000","last_peered":"2024-04-26T16:25:31.096050+0000","last_clean":"2024-04-26T16:25:31.096050+0000","last_became_active":"2024-04-26T16:17:35.094615+0000","last_became_peered":"2024-04-26T16:17:35.094615+0000","last_unstale":"2024-04-26T16:25:31.096050+0000","last_undegraded":"2024-04-26T16:25:31.096050+0000","last_fullsized":"2024-04-26T16:25:31.096050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:24.329922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:23:24.329922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":263,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.086905+0000","last_change":"2024-04-26T16:25:02.271846+0000","last_active":"2024-04-26T16:25:31.086905+0000","last_peered":"2024-04-26T16:25:31.086905+0000","last_clean":"2024-04-26T16:25:31.086905+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:25:31.086905+0000","last_undegraded":"2024-04-26T16:25:31.086905+0000","last_fullsized":"2024-04-26T16:25:31.086905+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.271632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.271632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:27.551401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"86'6","reported_seq":424,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109775+0000","last_change":"2024-04-26T16:24:48.386582+0000","last_active":"2024-04-26T16:25:31.109775+0000","last_peered":"2024-04-26T16:25:31.109775+0000","last_clean":"2024-04-26T16:25:31.109775+0000","last_became_active":"2024-04-26T16:17:31.053359+0000","last_became_peered":"2024-04-26T16:17:31.053359+0000","last_unstale":"2024-04-26T16:25:31.109775+0000","last_undegraded":"2024-04-26T16:25:31.109775+0000","last_fullsized":"2024-04-26T16:25:31.109775+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'6","last_scrub_stamp":"2024-04-26T16:24:48.386347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:48.386347+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.134604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.152326+0000","last_change":"2024-04-26T16:24:33.433898+0000","last_active":"2024-04-26T16:25:31.152326+0000","last_peered":"2024-04-26T16:25:31.152326+0000","last_clean":"2024-04-26T16:25:31.152326+0000","last_became_active":"2024-04-26T16:17:29.022454+0000","last_became_peered":"2024-04-26T16:17:29.022454+0000","last_unstale":"2024-04-26T16:25:31.152326+0000","last_undegraded":"2024-04-26T16:25:31.152326+0000","last_fullsized":"2024-04-26T16:25:31.152326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:33.433610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:33.433610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:51.570118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":403,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.116124+0000","last_change":"2024-04-26T16:24:31.376475+0000","last_active":"2024-04-26T16:25:31.116124+0000","last_peered":"2024-04-26T16:25:31.116124+0000","last_clean":"2024-04-26T16:25:31.116124+0000","last_became_active":"2024-04-26T16:17:33.059705+0000","last_became_peered":"2024-04-26T16:17:33.059705+0000","last_unstale":"2024-04-26T16:25:31.116124+0000","last_undegraded":"2024-04-26T16:25:31.116124+0000","last_fullsized":"2024-04-26T16:25:31.116124+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:31.376374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:31.376374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:37.896891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":392,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.124603+0000","last_change":"2024-04-26T16:23:04.332322+0000","last_active":"2024-04-26T16:25:31.124603+0000","last_peered":"2024-04-26T16:25:31.124603+0000","last_clean":"2024-04-26T16:25:31.124603+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:25:31.124603+0000","last_undegraded":"2024-04-26T16:25:31.124603+0000","last_fullsized":"2024-04-26T16:25:31.124603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:04.332212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:23:04.332212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":256,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.148733+0000","last_change":"2024-04-26T16:23:29.348658+0000","last_active":"2024-04-26T16:25:31.148733+0000","last_peered":"2024-04-26T16:25:31.148733+0000","last_clean":"2024-04-26T16:25:31.148733+0000","last_became_active":"2024-04-26T16:23:29.344221+0000","last_became_peered":"2024-04-26T16:23:29.344221+0000","last_unstale":"2024-04-26T16:25:31.148733+0000","last_undegraded":"2024-04-26T16:25:31.148733+0000","last_fullsized":"2024-04-26T16:25:31.148733+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.760255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"86'9","reported_seq":424,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.151769+0000","last_change":"2024-04-26T16:23:28.218569+0000","last_active":"2024-04-26T16:25:31.151769+0000","last_peered":"2024-04-26T16:25:31.151769+0000","last_clean":"2024-04-26T16:25:31.151769+0000","last_became_active":"2024-04-26T16:17:31.024113+0000","last_became_peered":"2024-04-26T16:17:31.024113+0000","last_unstale":"2024-04-26T16:25:31.151769+0000","last_undegraded":"2024-04-26T16:25:31.151769+0000","last_fullsized":"2024-04-26T16:25:31.151769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:23:28.218456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.218456+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139676+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":419,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128480+0000","last_change":"2024-04-26T16:24:48.505023+0000","last_active":"2024-04-26T16:25:31.128480+0000","last_peered":"2024-04-26T16:25:31.128480+0000","last_clean":"2024-04-26T16:25:31.128480+0000","last_became_active":"2024-04-26T16:17:29.018752+0000","last_became_peered":"2024-04-26T16:17:29.018752+0000","last_unstale":"2024-04-26T16:25:31.128480+0000","last_undegraded":"2024-04-26T16:25:31.128480+0000","last_fullsized":"2024-04-26T16:25:31.128480+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:48.504879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:24:48.504879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:14.927501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.143377+0000","last_change":"2024-04-26T16:24:43.710578+0000","last_active":"2024-04-26T16:25:31.143377+0000","last_peered":"2024-04-26T16:25:31.143377+0000","last_clean":"2024-04-26T16:25:31.143377+0000","last_became_active":"2024-04-26T16:17:33.075775+0000","last_became_peered":"2024-04-26T16:17:33.075775+0000","last_unstale":"2024-04-26T16:25:31.143377+0000","last_undegraded":"2024-04-26T16:25:31.143377+0000","last_fullsized":"2024-04-26T16:25:31.143377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:43.710377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:43.710377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:44.544757+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100898+0000","last_change":"2024-04-26T16:24:28.987326+0000","last_active":"2024-04-26T16:25:31.100898+0000","last_peered":"2024-04-26T16:25:31.100898+0000","last_clean":"2024-04-26T16:25:31.100898+0000","last_became_active":"2024-04-26T16:17:35.677366+0000","last_became_peered":"2024-04-26T16:17:35.677366+0000","last_unstale":"2024-04-26T16:25:31.100898+0000","last_undegraded":"2024-04-26T16:25:31.100898+0000","last_fullsized":"2024-04-26T16:25:31.100898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:28.987078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:28.987078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:35.404157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081563+0000","last_change":"2024-04-26T16:23:29.296256+0000","last_active":"2024-04-26T16:25:31.081563+0000","last_peered":"2024-04-26T16:25:31.081563+0000","last_clean":"2024-04-26T16:25:31.081563+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:25:31.081563+0000","last_undegraded":"2024-04-26T16:25:31.081563+0000","last_fullsized":"2024-04-26T16:25:31.081563+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:35.160554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"118'22","reported_seq":442,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081504+0000","last_change":"2024-04-26T16:23:21.058659+0000","last_active":"2024-04-26T16:25:31.081504+0000","last_peered":"2024-04-26T16:25:31.081504+0000","last_clean":"2024-04-26T16:25:31.081504+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:25:31.081504+0000","last_undegraded":"2024-04-26T16:25:31.081504+0000","last_fullsized":"2024-04-26T16:25:31.081504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-04-26T16:23:21.058552+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:23:21.058552+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":420,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.107970+0000","last_change":"2024-04-26T16:24:46.456740+0000","last_active":"2024-04-26T16:25:31.107970+0000","last_peered":"2024-04-26T16:25:31.107970+0000","last_clean":"2024-04-26T16:25:31.107970+0000","last_became_active":"2024-04-26T16:17:29.025319+0000","last_became_peered":"2024-04-26T16:17:29.025319+0000","last_unstale":"2024-04-26T16:25:31.107970+0000","last_undegraded":"2024-04-26T16:25:31.107970+0000","last_fullsized":"2024-04-26T16:25:31.107970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:46.456441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:46.456441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:06.107481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"86'11","reported_seq":521,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128997+0000","last_change":"2024-04-26T16:24:56.662316+0000","last_active":"2024-04-26T16:25:31.128997+0000","last_peered":"2024-04-26T16:25:31.128997+0000","last_clean":"2024-04-26T16:25:31.128997+0000","last_became_active":"2024-04-26T16:17:33.061926+0000","last_became_peered":"2024-04-26T16:17:33.061926+0000","last_unstale":"2024-04-26T16:25:31.128997+0000","last_undegraded":"2024-04-26T16:25:31.128997+0000","last_fullsized":"2024-04-26T16:25:31.128997+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:56.661981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:56.661981+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.332814+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":394,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.138275+0000","last_change":"2024-04-26T16:22:59.363651+0000","last_active":"2024-04-26T16:25:31.138275+0000","last_peered":"2024-04-26T16:25:31.138275+0000","last_clean":"2024-04-26T16:25:31.138275+0000","last_became_active":"2024-04-26T16:17:35.081916+0000","last_became_peered":"2024-04-26T16:17:35.081916+0000","last_unstale":"2024-04-26T16:25:31.138275+0000","last_undegraded":"2024-04-26T16:25:31.138275+0000","last_fullsized":"2024-04-26T16:25:31.138275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:22:59.363265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:22:59.363265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":254,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.101439+0000","last_change":"2024-04-26T16:23:29.356489+0000","last_active":"2024-04-26T16:25:31.101439+0000","last_peered":"2024-04-26T16:25:31.101439+0000","last_clean":"2024-04-26T16:25:31.101439+0000","last_became_active":"2024-04-26T16:23:29.356195+0000","last_became_peered":"2024-04-26T16:23:29.356195+0000","last_unstale":"2024-04-26T16:25:31.101439+0000","last_undegraded":"2024-04-26T16:25:31.101439+0000","last_fullsized":"2024-04-26T16:25:31.101439+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"86'9","reported_seq":424,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.094732+0000","last_change":"2024-04-26T16:23:27.383388+0000","last_active":"2024-04-26T16:25:31.094732+0000","last_peered":"2024-04-26T16:25:31.094732+0000","last_clean":"2024-04-26T16:25:31.094732+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:25:31.094732+0000","last_undegraded":"2024-04-26T16:25:31.094732+0000","last_fullsized":"2024-04-26T16:25:31.094732+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:23:27.383220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:23:27.383220+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:32.005570+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.096389+0000","last_change":"2024-04-26T16:24:45.903529+0000","last_active":"2024-04-26T16:25:31.096389+0000","last_peered":"2024-04-26T16:25:31.096389+0000","last_clean":"2024-04-26T16:25:31.096389+0000","last_became_active":"2024-04-26T16:17:28.999524+0000","last_became_peered":"2024-04-26T16:17:28.999524+0000","last_unstale":"2024-04-26T16:25:31.096389+0000","last_undegraded":"2024-04-26T16:25:31.096389+0000","last_fullsized":"2024-04-26T16:25:31.096389+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:24:45.903320+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.903320+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:08.756415+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":411,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.106652+0000","last_change":"2024-04-26T16:24:31.308473+0000","last_active":"2024-04-26T16:25:31.106652+0000","last_peered":"2024-04-26T16:25:31.106652+0000","last_clean":"2024-04-26T16:25:31.106652+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:25:31.106652+0000","last_undegraded":"2024-04-26T16:25:31.106652+0000","last_fullsized":"2024-04-26T16:25:31.106652+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:31.308253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:24:31.308253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:52.341288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":402,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.087344+0000","last_change":"2024-04-26T16:22:53.910723+0000","last_active":"2024-04-26T16:25:31.087344+0000","last_peered":"2024-04-26T16:25:31.087344+0000","last_clean":"2024-04-26T16:25:31.087344+0000","last_became_active":"2024-04-26T16:17:35.080236+0000","last_became_peered":"2024-04-26T16:17:35.080236+0000","last_unstale":"2024-04-26T16:25:31.087344+0000","last_undegraded":"2024-04-26T16:25:31.087344+0000","last_fullsized":"2024-04-26T16:25:31.087344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:22:53.910581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:22:53.910581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":254,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.148263+0000","last_change":"2024-04-26T16:23:29.344842+0000","last_active":"2024-04-26T16:25:31.148263+0000","last_peered":"2024-04-26T16:25:31.148263+0000","last_clean":"2024-04-26T16:25:31.148263+0000","last_became_active":"2024-04-26T16:23:29.344462+0000","last_became_peered":"2024-04-26T16:23:29.344462+0000","last_unstale":"2024-04-26T16:25:31.148263+0000","last_undegraded":"2024-04-26T16:25:31.148263+0000","last_fullsized":"2024-04-26T16:25:31.148263+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"114'7","reported_seq":427,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.112564+0000","last_change":"2024-04-26T16:24:46.346808+0000","last_active":"2024-04-26T16:25:31.112564+0000","last_peered":"2024-04-26T16:25:31.112564+0000","last_clean":"2024-04-26T16:25:31.112564+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:25:31.112564+0000","last_undegraded":"2024-04-26T16:25:31.112564+0000","last_fullsized":"2024-04-26T16:25:31.112564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-04-26T16:24:46.346568+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:23:25.467392+0000","last_clean_scrub_stamp":"2024-04-26T16:24:46.346568+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.398929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":419,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.125551+0000","last_change":"2024-04-26T16:24:26.368577+0000","last_active":"2024-04-26T16:25:31.125551+0000","last_peered":"2024-04-26T16:25:31.125551+0000","last_clean":"2024-04-26T16:25:31.125551+0000","last_became_active":"2024-04-26T16:17:29.004486+0000","last_became_peered":"2024-04-26T16:17:29.004486+0000","last_unstale":"2024-04-26T16:25:31.125551+0000","last_undegraded":"2024-04-26T16:25:31.125551+0000","last_fullsized":"2024-04-26T16:25:31.125551+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:26.368233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:26.368233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:53.785550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.087214+0000","last_change":"2024-04-26T16:25:10.202126+0000","last_active":"2024-04-26T16:25:31.087214+0000","last_peered":"2024-04-26T16:25:31.087214+0000","last_clean":"2024-04-26T16:25:31.087214+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:25:31.087214+0000","last_undegraded":"2024-04-26T16:25:31.087214+0000","last_fullsized":"2024-04-26T16:25:31.087214+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:10.202042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:25:10.202042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:29.479565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":410,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.096620+0000","last_change":"2024-04-26T16:25:01.893898+0000","last_active":"2024-04-26T16:25:31.096620+0000","last_peered":"2024-04-26T16:25:31.096620+0000","last_clean":"2024-04-26T16:25:31.096620+0000","last_became_active":"2024-04-26T16:17:35.674743+0000","last_became_peered":"2024-04-26T16:17:35.674743+0000","last_unstale":"2024-04-26T16:25:31.096620+0000","last_undegraded":"2024-04-26T16:25:31.096620+0000","last_fullsized":"2024-04-26T16:25:31.096620+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:01.893700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:25:01.893700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:19.397807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.124002+0000","last_change":"2024-04-26T16:23:29.358177+0000","last_active":"2024-04-26T16:25:31.124002+0000","last_peered":"2024-04-26T16:25:31.124002+0000","last_clean":"2024-04-26T16:25:31.124002+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:25:31.124002+0000","last_undegraded":"2024-04-26T16:25:31.124002+0000","last_fullsized":"2024-04-26T16:25:31.124002+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.760212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"86'9","reported_seq":434,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.089590+0000","last_change":"2024-04-26T16:24:54.880270+0000","last_active":"2024-04-26T16:25:31.089590+0000","last_peered":"2024-04-26T16:25:31.089590+0000","last_clean":"2024-04-26T16:25:31.089590+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:25:31.089590+0000","last_undegraded":"2024-04-26T16:25:31.089590+0000","last_fullsized":"2024-04-26T16:25:31.089590+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:24:54.880131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:54.880131+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:16.506723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100455+0000","last_change":"2024-04-26T16:24:40.982753+0000","last_active":"2024-04-26T16:25:31.100455+0000","last_peered":"2024-04-26T16:25:31.100455+0000","last_clean":"2024-04-26T16:25:31.100455+0000","last_became_active":"2024-04-26T16:17:29.005070+0000","last_became_peered":"2024-04-26T16:17:29.005070+0000","last_unstale":"2024-04-26T16:25:31.100455+0000","last_undegraded":"2024-04-26T16:25:31.100455+0000","last_fullsized":"2024-04-26T16:25:31.100455+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:40.982532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:40.982532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:07.937618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":406,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.091170+0000","last_change":"2024-04-26T16:24:45.765799+0000","last_active":"2024-04-26T16:25:31.091170+0000","last_peered":"2024-04-26T16:25:31.091170+0000","last_clean":"2024-04-26T16:25:31.091170+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:25:31.091170+0000","last_undegraded":"2024-04-26T16:25:31.091170+0000","last_fullsized":"2024-04-26T16:25:31.091170+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:45.765668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.765668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:47.632652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.108782+0000","last_change":"2024-04-26T16:24:36.099734+0000","last_active":"2024-04-26T16:25:31.108782+0000","last_peered":"2024-04-26T16:25:31.108782+0000","last_clean":"2024-04-26T16:25:31.108782+0000","last_became_active":"2024-04-26T16:17:35.673178+0000","last_became_peered":"2024-04-26T16:17:35.673178+0000","last_unstale":"2024-04-26T16:25:31.108782+0000","last_undegraded":"2024-04-26T16:25:31.108782+0000","last_fullsized":"2024-04-26T16:25:31.108782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:36.099628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:36.099628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:37.950538+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":263,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.087066+0000","last_change":"2024-04-26T16:24:46.109697+0000","last_active":"2024-04-26T16:25:31.087066+0000","last_peered":"2024-04-26T16:25:31.087066+0000","last_clean":"2024-04-26T16:25:31.087066+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:25:31.087066+0000","last_undegraded":"2024-04-26T16:25:31.087066+0000","last_fullsized":"2024-04-26T16:25:31.087066+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:46.109595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:46.109595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:52.268399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"126'17","reported_seq":436,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.126453+0000","last_change":"2024-04-26T16:24:34.475335+0000","last_active":"2024-04-26T16:25:31.126453+0000","last_peered":"2024-04-26T16:25:31.126453+0000","last_clean":"2024-04-26T16:25:31.126453+0000","last_became_active":"2024-04-26T16:17:31.019223+0000","last_became_peered":"2024-04-26T16:17:31.019223+0000","last_unstale":"2024-04-26T16:25:31.126453+0000","last_undegraded":"2024-04-26T16:25:31.126453+0000","last_fullsized":"2024-04-26T16:25:31.126453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'17","last_scrub_stamp":"2024-04-26T16:24:34.475095+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:24:34.475095+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:58.265902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.133402+0000","last_change":"2024-04-26T16:25:02.646994+0000","last_active":"2024-04-26T16:25:31.133402+0000","last_peered":"2024-04-26T16:25:31.133402+0000","last_clean":"2024-04-26T16:25:31.133402+0000","last_became_active":"2024-04-26T16:17:29.025659+0000","last_became_peered":"2024-04-26T16:17:29.025659+0000","last_unstale":"2024-04-26T16:25:31.133402+0000","last_undegraded":"2024-04-26T16:25:31.133402+0000","last_fullsized":"2024-04-26T16:25:31.133402+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.646805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.646805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:02.719092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":404,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.146523+0000","last_change":"2024-04-26T16:23:19.110115+0000","last_active":"2024-04-26T16:25:31.146523+0000","last_peered":"2024-04-26T16:25:31.146523+0000","last_clean":"2024-04-26T16:25:31.146523+0000","last_became_active":"2024-04-26T16:17:33.071659+0000","last_became_peered":"2024-04-26T16:17:33.071659+0000","last_unstale":"2024-04-26T16:25:31.146523+0000","last_undegraded":"2024-04-26T16:25:31.146523+0000","last_fullsized":"2024-04-26T16:25:31.146523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:19.109914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:23:19.109914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":399,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100781+0000","last_change":"2024-04-26T16:24:29.309510+0000","last_active":"2024-04-26T16:25:31.100781+0000","last_peered":"2024-04-26T16:25:31.100781+0000","last_clean":"2024-04-26T16:25:31.100781+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:25:31.100781+0000","last_undegraded":"2024-04-26T16:25:31.100781+0000","last_fullsized":"2024-04-26T16:25:31.100781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:29.309297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:29.309297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:55.873479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":262,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.122337+0000","last_change":"2024-04-26T16:25:08.760428+0000","last_active":"2024-04-26T16:25:31.122337+0000","last_peered":"2024-04-26T16:25:31.122337+0000","last_clean":"2024-04-26T16:25:31.122337+0000","last_became_active":"2024-04-26T16:23:29.295178+0000","last_became_peered":"2024-04-26T16:23:29.295178+0000","last_unstale":"2024-04-26T16:25:31.122337+0000","last_undegraded":"2024-04-26T16:25:31.122337+0000","last_fullsized":"2024-04-26T16:25:31.122337+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:08.760219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:08.760219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:30.039015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"86'15","reported_seq":433,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.128562+0000","last_change":"2024-04-26T16:24:28.385074+0000","last_active":"2024-04-26T16:25:31.128562+0000","last_peered":"2024-04-26T16:25:31.128562+0000","last_clean":"2024-04-26T16:25:31.128562+0000","last_became_active":"2024-04-26T16:17:31.022500+0000","last_became_peered":"2024-04-26T16:17:31.022500+0000","last_unstale":"2024-04-26T16:25:31.128562+0000","last_undegraded":"2024-04-26T16:25:31.128562+0000","last_fullsized":"2024-04-26T16:25:31.128562+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:24:28.384893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:28.384893+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:41.614351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":412,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.148052+0000","last_change":"2024-04-26T16:23:19.274168+0000","last_active":"2024-04-26T16:25:31.148052+0000","last_peered":"2024-04-26T16:25:31.148052+0000","last_clean":"2024-04-26T16:25:31.148052+0000","last_became_active":"2024-04-26T16:17:29.022163+0000","last_became_peered":"2024-04-26T16:17:29.022163+0000","last_unstale":"2024-04-26T16:25:31.148052+0000","last_undegraded":"2024-04-26T16:25:31.148052+0000","last_fullsized":"2024-04-26T16:25:31.148052+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:19.274034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:23:19.274034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759611+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":396,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.124389+0000","last_change":"2024-04-26T16:22:54.411091+0000","last_active":"2024-04-26T16:25:31.124389+0000","last_peered":"2024-04-26T16:25:31.124389+0000","last_clean":"2024-04-26T16:25:31.124389+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:25:31.124389+0000","last_undegraded":"2024-04-26T16:25:31.124389+0000","last_fullsized":"2024-04-26T16:25:31.124389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:22:54.410964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:22:54.410964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":409,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.127370+0000","last_change":"2024-04-26T16:25:03.570352+0000","last_active":"2024-04-26T16:25:31.127370+0000","last_peered":"2024-04-26T16:25:31.127370+0000","last_clean":"2024-04-26T16:25:31.127370+0000","last_became_active":"2024-04-26T16:17:35.087319+0000","last_became_peered":"2024-04-26T16:17:35.087319+0000","last_unstale":"2024-04-26T16:25:31.127370+0000","last_undegraded":"2024-04-26T16:25:31.127370+0000","last_fullsized":"2024-04-26T16:25:31.127370+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:25:03.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:25:03.570129+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:31.367772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":255,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.152579+0000","last_change":"2024-04-26T16:23:29.358942+0000","last_active":"2024-04-26T16:25:31.152579+0000","last_peered":"2024-04-26T16:25:31.152579+0000","last_clean":"2024-04-26T16:25:31.152579+0000","last_became_active":"2024-04-26T16:23:29.355816+0000","last_became_peered":"2024-04-26T16:23:29.355816+0000","last_unstale":"2024-04-26T16:25:31.152579+0000","last_undegraded":"2024-04-26T16:25:31.152579+0000","last_fullsized":"2024-04-26T16:25:31.152579+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.759942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"86'11","reported_seq":434,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109888+0000","last_change":"2024-04-26T16:24:42.399809+0000","last_active":"2024-04-26T16:25:31.109888+0000","last_peered":"2024-04-26T16:25:31.109888+0000","last_clean":"2024-04-26T16:25:31.109888+0000","last_became_active":"2024-04-26T16:17:31.041819+0000","last_became_peered":"2024-04-26T16:17:31.041819+0000","last_unstale":"2024-04-26T16:25:31.109888+0000","last_undegraded":"2024-04-26T16:25:31.109888+0000","last_fullsized":"2024-04-26T16:25:31.109888+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:42.399614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:42.399614+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:01.881419+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":421,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103280+0000","last_change":"2024-04-26T16:24:37.085176+0000","last_active":"2024-04-26T16:25:31.103280+0000","last_peered":"2024-04-26T16:25:31.103280+0000","last_clean":"2024-04-26T16:25:31.103280+0000","last_became_active":"2024-04-26T16:17:29.016870+0000","last_became_peered":"2024-04-26T16:17:29.016870+0000","last_unstale":"2024-04-26T16:25:31.103280+0000","last_undegraded":"2024-04-26T16:25:31.103280+0000","last_fullsized":"2024-04-26T16:25:31.103280+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:37.085072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:37.085072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:38.687768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":406,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.081516+0000","last_change":"2024-04-26T16:22:56.994422+0000","last_active":"2024-04-26T16:25:31.081516+0000","last_peered":"2024-04-26T16:25:31.081516+0000","last_clean":"2024-04-26T16:25:31.081516+0000","last_became_active":"2024-04-26T16:17:33.058899+0000","last_became_peered":"2024-04-26T16:17:33.058899+0000","last_unstale":"2024-04-26T16:25:31.081516+0000","last_undegraded":"2024-04-26T16:25:31.081516+0000","last_fullsized":"2024-04-26T16:25:31.081516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:22:56.994283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:22:56.994283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":410,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.089233+0000","last_change":"2024-04-26T16:25:09.777592+0000","last_active":"2024-04-26T16:25:31.089233+0000","last_peered":"2024-04-26T16:25:31.089233+0000","last_clean":"2024-04-26T16:25:31.089233+0000","last_became_active":"2024-04-26T16:17:35.065560+0000","last_became_peered":"2024-04-26T16:17:35.065560+0000","last_unstale":"2024-04-26T16:25:31.089233+0000","last_undegraded":"2024-04-26T16:25:31.089233+0000","last_fullsized":"2024-04-26T16:25:31.089233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:09.777360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:25:09.777360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:15.414490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":254,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.151619+0000","last_change":"2024-04-26T16:23:29.336846+0000","last_active":"2024-04-26T16:25:31.151619+0000","last_peered":"2024-04-26T16:25:31.151619+0000","last_clean":"2024-04-26T16:25:31.151619+0000","last_became_active":"2024-04-26T16:23:29.335961+0000","last_became_peered":"2024-04-26T16:23:29.335961+0000","last_unstale":"2024-04-26T16:25:31.151619+0000","last_undegraded":"2024-04-26T16:25:31.151619+0000","last_fullsized":"2024-04-26T16:25:31.151619+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.139615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"86'10","reported_seq":431,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.103399+0000","last_change":"2024-04-26T16:25:02.039962+0000","last_active":"2024-04-26T16:25:31.103399+0000","last_peered":"2024-04-26T16:25:31.103399+0000","last_clean":"2024-04-26T16:25:31.103399+0000","last_became_active":"2024-04-26T16:17:31.049997+0000","last_became_peered":"2024-04-26T16:17:31.049997+0000","last_unstale":"2024-04-26T16:25:31.103399+0000","last_undegraded":"2024-04-26T16:25:31.103399+0000","last_fullsized":"2024-04-26T16:25:31.103399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:25:02.039711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.039711+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:25.374057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":419,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.109922+0000","last_change":"2024-04-26T16:24:50.328260+0000","last_active":"2024-04-26T16:25:31.109922+0000","last_peered":"2024-04-26T16:25:31.109922+0000","last_clean":"2024-04-26T16:25:31.109922+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:25:31.109922+0000","last_undegraded":"2024-04-26T16:25:31.109922+0000","last_fullsized":"2024-04-26T16:25:31.109922+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:50.328009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:15.410743+0000","last_clean_scrub_stamp":"2024-04-26T16:24:50.328009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:09.380170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":414,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.088063+0000","last_change":"2024-04-26T16:25:04.787080+0000","last_active":"2024-04-26T16:25:31.088063+0000","last_peered":"2024-04-26T16:25:31.088063+0000","last_clean":"2024-04-26T16:25:31.088063+0000","last_became_active":"2024-04-26T16:17:33.070922+0000","last_became_peered":"2024-04-26T16:17:33.070922+0000","last_unstale":"2024-04-26T16:25:31.088063+0000","last_undegraded":"2024-04-26T16:25:31.088063+0000","last_fullsized":"2024-04-26T16:25:31.088063+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:04.786867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:25:04.786867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:26:11.269237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":400,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-04-26T16:25:31.100151+0000","last_change":"2024-04-26T16:24:32.014732+0000","last_active":"2024-04-26T16:25:31.100151+0000","last_peered":"2024-04-26T16:25:31.100151+0000","last_clean":"2024-04-26T16:25:31.100151+0000","last_became_active":"2024-04-26T16:17:35.677526+0000","last_became_peered":"2024-04-26T16:17:35.677526+0000","last_unstale":"2024-04-26T16:25:31.100151+0000","last_undegraded":"2024-04-26T16:25:31.100151+0000","last_fullsized":"2024-04-26T16:25:31.100151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:32.014404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:24:32.014404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:25:34.017418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":237,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":234,"num_pg":32,"stat_sum":{"num_bytes":128,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":109,"num_pg":32,"stat_sum":{"num_bytes":1543,"num_objects":147,"num_object_clones":33,"num_object_copies":441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":147,"num_whiteouts":0,"num_read":89,"num_read_kb":1,"num_write":522,"num_write_kb":520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1794048,"data_stored":4629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1671,"ondisk_log_size":1671,"up":96,"acting":96,"num_store_stats":8},{"poolid":12,"num_pg":11,"stat_sum":{"num_bytes":2048,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":6400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":33,"acting":33,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":701,"num_read_kb":458,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":470,"ondisk_log_size":470,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743531,"num_pgs":125,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":76340,"kb_used_data":4884,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93668812,"statfs":{"total":95995035648,"available":95916863488,"internally_reserved":0,"allocated":5001216,"data_stored":2965470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813135989,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49724,"kb_used_data":4652,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93695428,"statfs":{"total":95995035648,"available":95944118272,"internally_reserved":0,"allocated":4763648,"data_stored":2961577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":46135753},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575017,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67224,"kb_used_data":4104,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93677928,"statfs":{"total":95995035648,"available":95926198272,"internally_reserved":0,"allocated":4202496,"data_stored":2385266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71904,"kb_used_data":4608,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673248,"statfs":{"total":95995035648,"available":95921405952,"internally_reserved":0,"allocated":4718592,"data_stored":2963934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411486,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71440,"kb_used_data":4152,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673712,"statfs":{"total":95995035648,"available":95921881088,"internally_reserved":0,"allocated":4251648,"data_stored":2384709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084435,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93676,"kb_used_data":4252,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93651476,"statfs":{"total":95995035648,"available":95899111424,"internally_reserved":0,"allocated":4354048,"data_stored":2385133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":91552200},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855497,"num_pgs":111,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71456,"kb_used_data":4176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673696,"statfs":{"total":95995035648,"available":95921864704,"internally_reserved":0,"allocated":4276224,"data_stored":2372402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463039,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71380,"kb_used_data":4100,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93673772,"statfs":{"total":95995035648,"available":95921942528,"internally_reserved":0,"allocated":4198400,"data_stored":2383553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167936,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":234,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":237,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:25:38.206 INFO:tasks.thrashosds.thrasher:Setting 2.15 to [2, 6, 5] 2024-04-26T16:25:38.206 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.15', '2', '6', '5'] 2024-04-26T16:25:38.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd pg-upmap 2.15 2 6 5 2024-04-26T16:25:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.208 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:38.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:38.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:38.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:38.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.383+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7fd9bc02e490 con 0x7fd9bc1024b0 2024-04-26T16:25:38.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:38.464 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:38.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:38.594 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:38.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:36.716528+0000 mgr.y (mgr.24401) 330 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:36.716528+0000 mgr.y (mgr.24401) 330 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:37.375375+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:37.375375+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.558779+0000 mgr.y (mgr.24401) 331 : cluster [DBG] pgmap v388: 311 pgs: 40 unknown, 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 257 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.558779+0000 mgr.y (mgr.24401) 331 : cluster [DBG] pgmap v388: 311 pgs: 40 unknown, 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 257 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.587743+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.587743+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.588035+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:37.588035+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.072788+0000 mon.a (mon.0) 1875 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.072788+0000 mon.a (mon.0) 1875 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:38.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:38.075281+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:38.075281+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:38.075498+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: audit 2024-04-26T16:25:38.075498+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.085796+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.085796+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:36.716528+0000 mgr.y (mgr.24401) 330 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:36.716528+0000 mgr.y (mgr.24401) 330 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:37.375375+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:37.375375+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.558779+0000 mgr.y (mgr.24401) 331 : cluster [DBG] pgmap v388: 311 pgs: 40 unknown, 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 257 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.558779+0000 mgr.y (mgr.24401) 331 : cluster [DBG] pgmap v388: 311 pgs: 40 unknown, 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 257 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.587743+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.587743+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.588035+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:37.588035+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.072788+0000 mon.a (mon.0) 1875 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.072788+0000 mon.a (mon.0) 1875 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:38.075281+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:38.075281+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:38.075498+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: audit 2024-04-26T16:25:38.075498+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.085796+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-04-26T16:25:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:38 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.085796+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-04-26T16:25:38.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:38.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:38.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:36.716528+0000 mgr.y (mgr.24401) 330 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:38.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:36.716528+0000 mgr.y (mgr.24401) 330 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:38.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:37.375375+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:37.375375+0000 mon.c (mon.2) 430 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.558779+0000 mgr.y (mgr.24401) 331 : cluster [DBG] pgmap v388: 311 pgs: 40 unknown, 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 257 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.558779+0000 mgr.y (mgr.24401) 331 : cluster [DBG] pgmap v388: 311 pgs: 40 unknown, 6 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 257 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.587743+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.587743+0000 osd.6 (osd.6) 139 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.588035+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:37.588035+0000 osd.6 (osd.6) 140 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.072788+0000 mon.a (mon.0) 1875 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.072788+0000 mon.a (mon.0) 1875 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:38.075281+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:38.075281+0000 mon.a (mon.0) 1876 : audit [INF] from='client.? 172.21.15.50:0/1762540472' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:38.075498+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:38.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: audit 2024-04-26T16:25:38.075498+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-23"}]': finished 2024-04-26T16:25:38.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.085796+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-04-26T16:25:38.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:38 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.085796+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2024-04-26T16:25:38.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:38.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/2862329866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c101ae0 msgr2=0x7f706c101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:38.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- 172.21.15.50:0/2862329866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c101ae0 0x7f706c101f60 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f705c009990 tx=0x7f705c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/2862329866 shutdown_connections 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- 172.21.15.50:0/2862329866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f706c1024a0 0x7f706c10e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- 172.21.15.50:0/2862329866 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f706c107a90 0x7f706c107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- 172.21.15.50:0/2862329866 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c101ae0 0x7f706c101f60 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/2862329866 >> 172.21.15.50:0/2862329866 conn(0x7f706c078990 msgr2=0x7f706c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/2862329866 shutdown_connections 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/2862329866 wait complete. 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 librados: starting msgr at 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 librados: starting objecter 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 Processor -- start 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- start start 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 librados: setting wanted keys 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 librados: calling monclient init 2024-04-26T16:25:38.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f706c101ae0 0x7f706c103ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f706c1024a0 0x7f706c104410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 0x7f706c104950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f706c1111f0 con 0x7f706c101ae0 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f706c111070 con 0x7f706c1024a0 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f706c111370 con 0x7f706c107a90 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 0x7f706c104950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f70713df700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f706c101ae0 0x7f706c103ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 0x7f706c104950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40474/0 (socket says 172.21.15.50:40474) 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 -- 172.21.15.50:0/201357410 learned_addr learned my addr 172.21.15.50:0/201357410 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:38.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f706c1024a0 0x7f706c104410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f706c1024a0 msgr2=0x7f706c104410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f706c1024a0 0x7f706c104410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f706c101ae0 msgr2=0x7f706c103ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f706c101ae0 0x7f706c103ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f706c1a7660 con 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f706c1024a0 0x7f706c104410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 0x7f706c104950 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f7064007d70 tx=0x7f706400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f70627fc700 10 client.?.objecter ms_handle_connect 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f70627fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7064016020 con 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f706c1a7950 con 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f706400ad40 con 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7064011380 con 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 10 client.40871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f706c1a7ee0 con 0x7f706c107a90 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.931+0000 7f7073643700 1 librados: init done 2024-04-26T16:25:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f706c1031d0 con 0x7f706c107a90 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7064019020 con 0x7f706c107a90 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7058071f80 0x7f7058074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(270..270 src has 87..270) ==== 11203+0+0 (secure 0 0 0) 0x7f7064092140 con 0x7f706c107a90 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 osd_map(270..270 src has 87..270) 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 3 client.40871.objecter handle_osd_map got epochs [270,270] > 0 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 3 client.40871.objecter handle_osd_map decoding full epoch 270 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7058071f80 0x7f7058074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 20 client.40871.objecter dump_active .. 0 homeless 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v270) ==== 54+0+16 (secure 0 0 0) 0x7f7064020020 con 0x7f706c107a90 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 10 client.40871.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 1 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f706c064030 0x7f706c064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 1 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 0 1 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 4 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 10 client.40871.objecter _send_command 1 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f706c1053b0 con 0x7f706c064030 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f706c064030 0x7f706c064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7058071f80 0x7f7058074440 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f7068009be0 tx=0x7f7068009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f7058071f80 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f706c064030 0x7f706c064410 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c064030 2024-04-26T16:25:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c1053b0 con 0x7f706c064030 2024-04-26T16:25:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 1: 0 ) 2024-04-26T16:25:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 10 client.40871.objecter _finish_command 1 = system:0 2024-04-26T16:25:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 4 2024-04-26T16:25:38.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.935+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 0 1 2024-04-26T16:25:38.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 10 client.40871.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 2 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 2 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c06a110 osd=0 3 2024-04-26T16:25:38.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 0 2 2024-04-26T16:25:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 4 2024-04-26T16:25:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 10 client.40871.objecter _send_command 2 2024-04-26T16:25:38.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c063ab0 con 0x7f706c064030 2024-04-26T16:25:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c063ab0 con 0x7f706c064030 2024-04-26T16:25:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70627fc700 10 client.40871.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 4 2024-04-26T16:25:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 0 2 2024-04-26T16:25:38.955 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 10 client.40871.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 3 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f706c10a5e0 0x7f706c1ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 3 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 1 3 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 4 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 10 client.40871.objecter _send_command 3 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f706c02e600 con 0x7f706c10a5e0 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f706c10a5e0 0x7f706c1ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c10a5e0 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.955+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f706c10a5e0 0x7f706c1ac130 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.959+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c02e600 con 0x7f706c10a5e0 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.959+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 3: 0 ) 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.959+0000 7f70627fc700 10 client.40871.objecter _finish_command 3 = system:0 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.959+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 4 2024-04-26T16:25:38.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.959+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 1 3 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 10 client.40871.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 4 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 4 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c02e490 osd=1 3 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 1 4 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 4 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 10 client.40871.objecter _send_command 4 2024-04-26T16:25:38.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c1ac670 con 0x7f706c10a5e0 2024-04-26T16:25:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c1ac670 con 0x7f706c10a5e0 2024-04-26T16:25:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.975+0000 7f70627fc700 10 client.40871.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 4 2024-04-26T16:25:38.976 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 1 4 2024-04-26T16:25:38.976 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:25:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 10 client.40871.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 5 2024-04-26T16:25:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f706c109cc0 0x7f706c1af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 5 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 2 5 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 4 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 10 client.40871.objecter _send_command 5 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f706c02dff0 con 0x7f706c109cc0 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f706c109cc0 0x7f706c1af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f706c109cc0 0x7f706c1af9b0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c109cc0 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c02dff0 con 0x7f706c109cc0 2024-04-26T16:25:38.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 5: 0 ) 2024-04-26T16:25:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 10 client.40871.objecter _finish_command 5 = system:0 2024-04-26T16:25:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 4 2024-04-26T16:25:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.979+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 2 5 2024-04-26T16:25:38.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:38.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 10 client.40871.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 6 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 6 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c109b50 osd=2 3 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 2 6 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 4 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 10 client.40871.objecter _send_command 6 2024-04-26T16:25:38.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.995+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c02d560 con 0x7f706c109cc0 2024-04-26T16:25:38.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c02d560 con 0x7f706c109cc0 2024-04-26T16:25:38.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:38.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 10 client.40871.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:38.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 4 2024-04-26T16:25:38.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 2 6 2024-04-26T16:25:38.997 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:25:38.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 10 client.40871.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:38.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 7 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f706c1ad500 0x7f706c1b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 7 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 3 7 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 4 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 10 client.40871.objecter _send_command 7 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f706c1b5ee0 con 0x7f706c1ad500 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f706c1ad500 0x7f706c1b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c1ad500 2024-04-26T16:25:38.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f706c1ad500 0x7f706c1b3080 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:38.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c1b5ee0 con 0x7f706c1ad500 2024-04-26T16:25:38.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 7: 0 ) 2024-04-26T16:25:38.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 10 client.40871.objecter _finish_command 7 = system:0 2024-04-26T16:25:38.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 4 2024-04-26T16:25:38.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:38.999+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 3 7 2024-04-26T16:25:39.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 10 client.40871.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 8 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 8 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c02d960 osd=3 3 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 3 8 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 4 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 10 client.40871.objecter _send_command 8 2024-04-26T16:25:39.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c1acb90 con 0x7f706c1ad500 2024-04-26T16:25:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c1acb90 con 0x7f706c1ad500 2024-04-26T16:25:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70627fc700 10 client.40871.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 4 2024-04-26T16:25:39.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 3 8 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 10 client.40871.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 9 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f706c1b0bd0 0x7f706c1b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 9 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 4 9 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 4 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 10 client.40871.objecter _send_command 9 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f706c1acdd0 con 0x7f706c1b0bd0 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f706c1b0bd0 0x7f706c1b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c1b0bd0 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.019+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f706c1b0bd0 0x7f706c1b4740 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.023+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c1acdd0 con 0x7f706c1b0bd0 2024-04-26T16:25:39.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.023+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 9: 0 ) 2024-04-26T16:25:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.023+0000 7f70627fc700 10 client.40871.objecter _finish_command 9 = system:0 2024-04-26T16:25:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.023+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 4 2024-04-26T16:25:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.023+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 4 9 2024-04-26T16:25:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 10 client.40871.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:39.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 10 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 10 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1acb90 osd=4 3 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 4 10 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 4 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 10 client.40871.objecter _send_command 10 2024-04-26T16:25:39.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.039+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c1acdd0 con 0x7f706c1b0bd0 2024-04-26T16:25:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c1acdd0 con 0x7f706c1b0bd0 2024-04-26T16:25:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:39.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 10 client.40871.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 4 2024-04-26T16:25:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 4 10 2024-04-26T16:25:39.042 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:25:39.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 10 client.40871.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 11 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f706c1b5190 0x7f706c1b55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 11 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 5 11 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 4 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 10 client.40871.objecter _send_command 11 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f706c1bcab0 con 0x7f706c1b5190 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f706c1b5190 0x7f706c1b55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:39.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c1b5190 2024-04-26T16:25:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f7070bde700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f706c1b5190 0x7f706c1b55f0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f705c032a50 con 0x7f706c1b5190 2024-04-26T16:25:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 11: 0 ) 2024-04-26T16:25:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 10 client.40871.objecter _finish_command 11 = system:0 2024-04-26T16:25:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 4 2024-04-26T16:25:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.043+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 5 11 2024-04-26T16:25:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 10 client.40871.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 12 2024-04-26T16:25:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 12 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1acdd0 osd=5 3 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 5 12 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 4 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 10 client.40871.objecter _send_command 12 2024-04-26T16:25:39.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c1bb930 con 0x7f706c1b5190 2024-04-26T16:25:39.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c1bb930 con 0x7f706c1b5190 2024-04-26T16:25:39.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:39.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f70627fc700 10 client.40871.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:39.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 4 2024-04-26T16:25:39.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.063+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 5 12 2024-04-26T16:25:39.064 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 10 client.40871.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 13 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f706c1bc050 0x7f706c1bc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 13 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 6 13 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 4 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 10 client.40871.objecter _send_command 13 2024-04-26T16:25:39.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f706c1bfe40 con 0x7f706c1bc050 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f706c1bc050 0x7f706c1bc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c1bc050 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f7071be0700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f706c1bc050 0x7f706c1bc490 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c1bfe40 con 0x7f706c1bc050 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 13: 0 ) 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f70627fc700 10 client.40871.objecter _finish_command 13 = system:0 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 4 2024-04-26T16:25:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.067+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 6 13 2024-04-26T16:25:39.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.075+0000 7fd9b27fc700 1 -- 172.21.15.50:0/2883211812 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v271) ==== 97+0+0 (secure 0 0 0) 0x7fd9ac05ce70 con 0x7fd9bc1024b0 2024-04-26T16:25:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.079+0000 7fd9c32a4700 10 librados: watch_flush enter 2024-04-26T16:25:39.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 10 librados: watch_flush exit 2024-04-26T16:25:39.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 20 client.40859.objecter shutdown clearing up homeless session... 2024-04-26T16:25:39.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 10 client.40859.objecter successfully canceled tick 2024-04-26T16:25:39.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd9a8071fc0 msgr2=0x7fd9a8074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd9a8071fc0 0x7fd9a8074480 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 crypto rx=0x7fd9b400d010 tx=0x7fd9b400b680 comp rx=0 tx=0).stop 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc1024b0 msgr2=0x7fd9bc19bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc1024b0 0x7fd9bc19bb30 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac00c970 tx=0x7fd9ac00cc80 comp rx=0 tx=0).stop 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 shutdown_connections 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd9a8071fc0 0x7fd9a8074480 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9bc101a10 0x7fd9bc19b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9bc19fec0 0x7fd9bc1a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 --2- 172.21.15.50:0/2883211812 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd9bc1024b0 0x7fd9bc19bb30 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.083+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 >> 172.21.15.50:0/2883211812 conn(0x7fd9bc078990 msgr2=0x7fd9bc0ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:39.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.087+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 shutdown_connections 2024-04-26T16:25:39.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.091+0000 7fd9c32a4700 1 -- 172.21.15.50:0/2883211812 wait complete. 2024-04-26T16:25:39.090 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.091+0000 7fd9c32a4700 1 librados: shutdown 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 10 client.40871.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 14 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 14 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1bb930 osd=6 3 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 6 14 2024-04-26T16:25:39.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 4 2024-04-26T16:25:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 10 client.40871.objecter _send_command 14 2024-04-26T16:25:39.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.103+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c1bfe40 con 0x7f706c1bc050 2024-04-26T16:25:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c1bfe40 con 0x7f706c1bc050 2024-04-26T16:25:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 10 client.40871.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 4 2024-04-26T16:25:39.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 6 14 2024-04-26T16:25:39.114 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:25:39.114 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 10 client.40871.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 15 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f706c1bf180 0x7f706c1bf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 15 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 7 15 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 4 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 10 client.40871.objecter _send_command 15 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f706c1c3210 con 0x7f706c1bf180 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f706c1bf180 0x7f706c1bf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 10 client.40871.objecter ms_handle_connect 0x7f706c1bf180 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70713df700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f706c1bf180 0x7f706c1bf5e0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f706c1c3210 con 0x7f706c1bf180 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 15: 0 ) 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 10 client.40871.objecter _finish_command 15 = system:0 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 4 2024-04-26T16:25:39.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.115+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 7 15 2024-04-26T16:25:39.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 10 client.40871.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign -1 16 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 20 client.40871.objecter _get_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 15 client.40871.objecter _session_command_op_remove -1 16 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 20 client.40871.objecter get_session s=0x7f706c1bfe40 osd=7 3 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 15 client.40871.objecter _session_command_op_assign 7 16 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 4 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 10 client.40871.objecter _send_command 16 2024-04-26T16:25:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.135+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f706c1b8130 con 0x7f706c1bf180 2024-04-26T16:25:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f70627fc700 1 -- 172.21.15.50:0/201357410 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f706c1b8130 con 0x7f706c1bf180 2024-04-26T16:25:39.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f70627fc700 10 client.40871.objecter ms_dispatch 0x7f706c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:25:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f70627fc700 10 client.40871.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:25:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f70627fc700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 4 2024-04-26T16:25:39.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f70627fc700 15 client.40871.objecter _session_command_op_remove 7 16 2024-04-26T16:25:39.138 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 librados: watch_flush enter 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 librados: watch_flush exit 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.0 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f706c064030 msgr2=0x7f706c064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f706c064030 0x7f706c064410 crc :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c06a110 osd=0 1 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.1 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f706c10a5e0 msgr2=0x7f706c1ac130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f706c10a5e0 0x7f706c1ac130 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02e490 osd=1 1 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.2 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f706c109cc0 msgr2=0x7f706c1af9b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f706c109cc0 0x7f706c1af9b0 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c109b50 osd=2 1 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.3 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f706c1ad500 msgr2=0x7f706c1b3080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f706c1ad500 0x7f706c1b3080 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7071be0700 1 -- 172.21.15.50:0/201357410 reap_dead start 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c02d960 osd=3 1 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.4 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f706c1b0bd0 msgr2=0x7f706c1b4740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f706c1b0bd0 0x7f706c1b4740 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acb90 osd=4 1 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.5 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f706c1b5190 msgr2=0x7f706c1b55f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f706c1b5190 0x7f706c1b55f0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1acdd0 osd=5 1 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.6 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f706c1bc050 msgr2=0x7f706c1bc490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f706c1bc050 0x7f706c1bc490 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bb930 osd=6 1 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter close_session for osd.7 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f706c1bf180 msgr2=0x7f706c1bf5e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f706c1bf180 0x7f706c1bf5e0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter put_session s=0x7f706c1bfe40 osd=7 1 2024-04-26T16:25:39.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 20 client.40871.objecter shutdown clearing up homeless session... 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 10 client.40871.objecter successfully canceled tick 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7071be0700 1 -- 172.21.15.50:0/201357410 reap_dead start 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7058071f80 msgr2=0x7f7058074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7058071f80 0x7f7058074440 secure :-1 s=READY pgs=3687 cs=0 l=1 rev1=1 crypto rx=0x7f7068009be0 tx=0x7f7068009260 comp rx=0 tx=0).stop 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 msgr2=0x7f706c104950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 0x7f706c104950 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f7064007d70 tx=0x7f706400a340 comp rx=0 tx=0).stop 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 shutdown_connections 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f706c107a90 0x7f706c104950 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 --2- 172.21.15.50:0/201357410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7058071f80 0x7f7058074440 unknown :-1 s=CLOSED pgs=3687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 >> 172.21.15.50:0/201357410 conn(0x7f706c078990 msgr2=0x7f706c0fffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 shutdown_connections 2024-04-26T16:25:39.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 -- 172.21.15.50:0/201357410 wait complete. 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:39.139+0000 7f7073643700 1 librados: shutdown 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:25:39.143 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:25:39.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-26T16:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-26T16:25:39.224 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:25:39.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:39.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:39.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.378493+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:39.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.378493+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.390132+0000 mon.c (mon.2) 432 : audit [INF] from='client.? 172.21.15.50:0/2883211812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.390132+0000 mon.c (mon.2) 432 : audit [INF] from='client.? 172.21.15.50:0/2883211812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.390446+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.390446+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.647683+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.647683+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.647883+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:38.647883+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.940464+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? 172.21.15.50:0/201357410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:38.940464+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? 172.21.15.50:0/201357410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:39.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:39.075652+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:39.376 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:39.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:39.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:39.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.378493+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.378493+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.390132+0000 mon.c (mon.2) 432 : audit [INF] from='client.? 172.21.15.50:0/2883211812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.390132+0000 mon.c (mon.2) 432 : audit [INF] from='client.? 172.21.15.50:0/2883211812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.390446+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.390446+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.647683+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.647683+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.647883+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:38.647883+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.940464+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? 172.21.15.50:0/201357410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:38.940464+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? 172.21.15.50:0/201357410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:39.075652+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:39.075652+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.076899+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.076899+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:39.080482+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: cluster 2024-04-26T16:25:39.080482+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.087172+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.50:0/2537316229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.087172+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.50:0/2537316229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.117764+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.117764+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.118059+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:39.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:39 smithi190 bash[23868]: audit 2024-04-26T16:25:39.118059+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:39.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:39.075652+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:39.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.076899+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:25:39.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.076899+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:25:39.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:39.080482+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-04-26T16:25:39.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: cluster 2024-04-26T16:25:39.080482+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-04-26T16:25:39.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.087172+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.50:0/2537316229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.087172+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.50:0/2537316229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.117764+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.117764+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.118059+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[29234]: audit 2024-04-26T16:25:39.118059+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.378493+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.378493+0000 mon.c (mon.2) 431 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.390132+0000 mon.c (mon.2) 432 : audit [INF] from='client.? 172.21.15.50:0/2883211812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.390132+0000 mon.c (mon.2) 432 : audit [INF] from='client.? 172.21.15.50:0/2883211812' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.390446+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.390446+0000 mon.a (mon.0) 1879 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.647683+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.647683+0000 osd.3 (osd.3) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.647883+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:38.647883+0000 osd.3 (osd.3) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.940464+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? 172.21.15.50:0/201357410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:38.940464+0000 mon.c (mon.2) 433 : audit [DBG] from='client.? 172.21.15.50:0/201357410' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:39.075652+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:39.075652+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.076899+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.076899+0000 mon.a (mon.0) 1881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:39.080482+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-04-26T16:25:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: cluster 2024-04-26T16:25:39.080482+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2024-04-26T16:25:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.087172+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.50:0/2537316229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.087172+0000 mon.b (mon.1) 181 : audit [INF] from='client.? 172.21.15.50:0/2537316229' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.117764+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.117764+0000 mon.a (mon.0) 1883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.118059+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:39 smithi050 bash[21463]: audit 2024-04-26T16:25:39.118059+0000 mon.a (mon.0) 1884 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:39.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:39.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:39.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:39.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:39.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:39.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:39.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:40.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:40.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:40.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:40.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:40.283 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:40.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:40.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:40.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:40.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:39.386271+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:39.386271+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: cluster 2024-04-26T16:25:39.559705+0000 mgr.y (mgr.24401) 332 : cluster [DBG] pgmap v391: 303 pgs: 64 unknown, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 230 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: cluster 2024-04-26T16:25:39.559705+0000 mgr.y (mgr.24401) 332 : cluster [DBG] pgmap v391: 303 pgs: 64 unknown, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 230 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.079992+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.079992+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.080098+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.080098+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: cluster 2024-04-26T16:25:40.085104+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: cluster 2024-04-26T16:25:40.085104+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.089823+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.089823+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.095279+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.095279+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.096936+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:40 smithi190 bash[23868]: audit 2024-04-26T16:25:40.096936+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.544 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:40.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:39.386271+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:39.386271+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: cluster 2024-04-26T16:25:39.559705+0000 mgr.y (mgr.24401) 332 : cluster [DBG] pgmap v391: 303 pgs: 64 unknown, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 230 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: cluster 2024-04-26T16:25:39.559705+0000 mgr.y (mgr.24401) 332 : cluster [DBG] pgmap v391: 303 pgs: 64 unknown, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 230 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.079992+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:40.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.079992+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.080098+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.080098+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: cluster 2024-04-26T16:25:40.085104+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: cluster 2024-04-26T16:25:40.085104+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.089823+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.089823+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.095279+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.095279+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.096936+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[21463]: audit 2024-04-26T16:25:40.096936+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:39.386271+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:39.386271+0000 mon.c (mon.2) 434 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: cluster 2024-04-26T16:25:39.559705+0000 mgr.y (mgr.24401) 332 : cluster [DBG] pgmap v391: 303 pgs: 64 unknown, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 230 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: cluster 2024-04-26T16:25:39.559705+0000 mgr.y (mgr.24401) 332 : cluster [DBG] pgmap v391: 303 pgs: 64 unknown, 6 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 230 active+clean; 586 KiB data, 573 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.079992+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.079992+0000 mon.a (mon.0) 1885 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.080098+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.080098+0000 mon.a (mon.0) 1886 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: cluster 2024-04-26T16:25:40.085104+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: cluster 2024-04-26T16:25:40.085104+0000 mon.a (mon.0) 1887 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.089823+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.089823+0000 mon.a (mon.0) 1888 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]: dispatch 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.095279+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.095279+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.096936+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:40 smithi050 bash[29234]: audit 2024-04-26T16:25:40.096936+0000 mon.b (mon.1) 182 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:40.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:40.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:40.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:40.805 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:40.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:40.937 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:40.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:41.040 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:41.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:41.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:41.193 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:41.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:41.324 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:41.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:41.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:41.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-26T16:25:41.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:41.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: cluster 2024-04-26T16:25:40.175797+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: cluster 2024-04-26T16:25:40.175797+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: cluster 2024-04-26T16:25:40.176237+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: cluster 2024-04-26T16:25:40.176237+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:40.388119+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:40.388119+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.083295+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.083295+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.083399+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.083399+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: cluster 2024-04-26T16:25:41.088524+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: cluster 2024-04-26T16:25:41.088524+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.093642+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.50:0/4161745678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.093642+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.50:0/4161745678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.097687+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.097687+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.128695+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.128695+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.129114+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.129114+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.130128+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.130128+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.130506+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.130506+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.131160+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.131160+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.131597+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:41 smithi190 bash[23868]: audit 2024-04-26T16:25:41.131597+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: cluster 2024-04-26T16:25:40.175797+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: cluster 2024-04-26T16:25:40.175797+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: cluster 2024-04-26T16:25:40.176237+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: cluster 2024-04-26T16:25:40.176237+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:40.388119+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:40.388119+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.083295+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.083295+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.083399+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.083399+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: cluster 2024-04-26T16:25:41.088524+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-04-26T16:25:41.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: cluster 2024-04-26T16:25:41.088524+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.093642+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.50:0/4161745678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.093642+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.50:0/4161745678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.097687+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.097687+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.128695+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: cluster 2024-04-26T16:25:40.175797+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: cluster 2024-04-26T16:25:40.175797+0000 osd.7 (osd.7) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: cluster 2024-04-26T16:25:40.176237+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: cluster 2024-04-26T16:25:40.176237+0000 osd.7 (osd.7) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:25:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:40.388119+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:40.388119+0000 mon.c (mon.2) 435 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.083295+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.083295+0000 mon.a (mon.0) 1890 : audit [INF] from='client.? 172.21.15.50:0/394222664' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi050-73193-34"}]': finished 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.083399+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.083399+0000 mon.a (mon.0) 1891 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: cluster 2024-04-26T16:25:41.088524+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-04-26T16:25:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: cluster 2024-04-26T16:25:41.088524+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2024-04-26T16:25:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.093642+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.50:0/4161745678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.093642+0000 mon.c (mon.2) 436 : audit [INF] from='client.? 172.21.15.50:0/4161745678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.097687+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.097687+0000 mon.a (mon.0) 1893 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.128695+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.128695+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.129114+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.129114+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.130128+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.130128+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.130506+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.130506+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.131160+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.131160+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.131597+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[29234]: audit 2024-04-26T16:25:41.131597+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.128695+0000 mon.c (mon.2) 437 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.129114+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.129114+0000 mon.a (mon.0) 1894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.130128+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.130128+0000 mon.c (mon.2) 438 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.130506+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.130506+0000 mon.a (mon.0) 1895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:41.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.131160+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.131160+0000 mon.c (mon.2) 439 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.131597+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:41 smithi050 bash[21463]: audit 2024-04-26T16:25:41.131597+0000 mon.a (mon.0) 1896 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:41.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:41.736 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:41.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:41.868 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:41.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:41.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:42.001 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:42.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:42.137 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:42.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:42.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:42.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:42.400 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:42.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:42.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:42.530 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:41.379235+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:41.379235+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:41.392658+0000 mon.c (mon.2) 440 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:41.392658+0000 mon.c (mon.2) 440 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:41.393515+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:41.393515+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:42.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: cluster 2024-04-26T16:25:41.561174+0000 mgr.y (mgr.24401) 333 : cluster [DBG] pgmap v394: 367 pgs: 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: cluster 2024-04-26T16:25:41.561174+0000 mgr.y (mgr.24401) 333 : cluster [DBG] pgmap v394: 367 pgs: 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.086953+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.086953+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.087265+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.087265+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: cluster 2024-04-26T16:25:42.097412+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: cluster 2024-04-26T16:25:42.097412+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.098629+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.098629+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.103359+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.103359+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.149541+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.149541+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.151757+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[21463]: audit 2024-04-26T16:25:42.151757+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:41.379235+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:41.379235+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:41.392658+0000 mon.c (mon.2) 440 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:41.392658+0000 mon.c (mon.2) 440 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:41.393515+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:42.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:41.393515+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: cluster 2024-04-26T16:25:41.561174+0000 mgr.y (mgr.24401) 333 : cluster [DBG] pgmap v394: 367 pgs: 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: cluster 2024-04-26T16:25:41.561174+0000 mgr.y (mgr.24401) 333 : cluster [DBG] pgmap v394: 367 pgs: 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.086953+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.086953+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.087265+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.087265+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: cluster 2024-04-26T16:25:42.097412+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: cluster 2024-04-26T16:25:42.097412+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.098629+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.098629+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.103359+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.103359+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.149541+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.149541+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.151757+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:42 smithi050 bash[29234]: audit 2024-04-26T16:25:42.151757+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:42.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:42.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:42.793 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:42.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:41.379235+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:41.379235+0000 mon.a (mon.0) 1897 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:41.392658+0000 mon.c (mon.2) 440 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:41.392658+0000 mon.c (mon.2) 440 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:41.393515+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:41.393515+0000 mon.c (mon.2) 441 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: cluster 2024-04-26T16:25:41.561174+0000 mgr.y (mgr.24401) 333 : cluster [DBG] pgmap v394: 367 pgs: 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: cluster 2024-04-26T16:25:41.561174+0000 mgr.y (mgr.24401) 333 : cluster [DBG] pgmap v394: 367 pgs: 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.086953+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.086953+0000 mon.a (mon.0) 1898 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.087265+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.087265+0000 mon.a (mon.0) 1899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi050-73193-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: cluster 2024-04-26T16:25:42.097412+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-04-26T16:25:42.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: cluster 2024-04-26T16:25:42.097412+0000 mon.a (mon.0) 1900 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.098629+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.098629+0000 mon.c (mon.2) 442 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.103359+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.103359+0000 mon.a (mon.0) 1901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.149541+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.149541+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.151757+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:42 smithi190 bash[23868]: audit 2024-04-26T16:25:42.151757+0000 mon.b (mon.1) 183 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:42.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:42.924 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:42.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:43.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:43.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:43.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:43.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:43.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:43.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:43.437 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:43.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:25:43.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:25:43.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:43.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:42.395068+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:42.395068+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.089985+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.089985+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.103009+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.103009+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: cluster 2024-04-26T16:25:43.112441+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: cluster 2024-04-26T16:25:43.112441+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.113452+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.113452+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.114095+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.50:0/3719047547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.114095+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.50:0/3719047547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.117356+0000 mon.a (mon.0) 1906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[21463]: audit 2024-04-26T16:25:43.117356+0000 mon.a (mon.0) 1906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:42.395068+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:42.395068+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.089985+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.089985+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.103009+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.103009+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: cluster 2024-04-26T16:25:43.112441+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: cluster 2024-04-26T16:25:43.112441+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.113452+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.113452+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.114095+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.50:0/3719047547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.114095+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.50:0/3719047547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.117356+0000 mon.a (mon.0) 1906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:43 smithi050 bash[29234]: audit 2024-04-26T16:25:43.117356+0000 mon.a (mon.0) 1906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:43.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:42.395068+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:43.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:42.395068+0000 mon.c (mon.2) 443 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:43.686 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.089985+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.089985+0000 mon.a (mon.0) 1903 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.103009+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.103009+0000 mon.b (mon.1) 184 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: cluster 2024-04-26T16:25:43.112441+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: cluster 2024-04-26T16:25:43.112441+0000 mon.a (mon.0) 1904 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.113452+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.113452+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.114095+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.50:0/3719047547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.114095+0000 mon.c (mon.2) 444 : audit [INF] from='client.? 172.21.15.50:0/3719047547' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.117356+0000 mon.a (mon.0) 1906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:43 smithi190 bash[23868]: audit 2024-04-26T16:25:43.117356+0000 mon.a (mon.0) 1906 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:43.698 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:43.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:43.829 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:43.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:43.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:43.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:44.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:44.094 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:44.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:44.236 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:44.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:44.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:44.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:44.498 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:44.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:44.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:25:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:25:44.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:43.397116+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:44.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:43.397116+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:44.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: cluster 2024-04-26T16:25:43.561875+0000 mgr.y (mgr.24401) 334 : cluster [DBG] pgmap v397: 399 pgs: 32 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: cluster 2024-04-26T16:25:43.561875+0000 mgr.y (mgr.24401) 334 : cluster [DBG] pgmap v397: 399 pgs: 32 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.092458+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.092458+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.092695+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.092695+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.092819+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.092819+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: cluster 2024-04-26T16:25:44.097924+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: cluster 2024-04-26T16:25:44.097924+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.102439+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.102439+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.103589+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[29234]: audit 2024-04-26T16:25:44.103589+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:43.397116+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:43.397116+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: cluster 2024-04-26T16:25:43.561875+0000 mgr.y (mgr.24401) 334 : cluster [DBG] pgmap v397: 399 pgs: 32 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: cluster 2024-04-26T16:25:43.561875+0000 mgr.y (mgr.24401) 334 : cluster [DBG] pgmap v397: 399 pgs: 32 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.092458+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.092458+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.092695+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.092695+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.092819+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.092819+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: cluster 2024-04-26T16:25:44.097924+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: cluster 2024-04-26T16:25:44.097924+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-04-26T16:25:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.102439+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.102439+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.103589+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:44 smithi050 bash[21463]: audit 2024-04-26T16:25:44.103589+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:44.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:43.397116+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:43.397116+0000 mon.c (mon.2) 445 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: cluster 2024-04-26T16:25:43.561875+0000 mgr.y (mgr.24401) 334 : cluster [DBG] pgmap v397: 399 pgs: 32 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: cluster 2024-04-26T16:25:43.561875+0000 mgr.y (mgr.24401) 334 : cluster [DBG] pgmap v397: 399 pgs: 32 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.092458+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.092458+0000 mon.a (mon.0) 1907 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi050-73193-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.092695+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.092695+0000 mon.a (mon.0) 1908 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.092819+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.092819+0000 mon.a (mon.0) 1909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi050-73201-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:44.662 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: cluster 2024-04-26T16:25:44.097924+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-04-26T16:25:44.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: cluster 2024-04-26T16:25:44.097924+0000 mon.a (mon.0) 1910 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2024-04-26T16:25:44.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.102439+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.102439+0000 mon.b (mon.1) 185 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.103589+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.663 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:44 smithi190 bash[23868]: audit 2024-04-26T16:25:44.103589+0000 mon.a (mon.0) 1911 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]: dispatch 2024-04-26T16:25:44.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:44.759 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:44.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:44.889 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:44.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:44.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:45.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:45.040 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:45.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:45.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:45.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:45.278 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:45.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:45.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:45.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:45.543 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:45.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:44.398679+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:44.398679+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: cluster 2024-04-26T16:25:45.092792+0000 mon.a (mon.0) 1912 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: cluster 2024-04-26T16:25:45.092792+0000 mon.a (mon.0) 1912 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.094659+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]': finished 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.094659+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]': finished 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: cluster 2024-04-26T16:25:45.099321+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: cluster 2024-04-26T16:25:45.099321+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.177554+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.177554+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.179873+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.179873+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.400072+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:45 smithi190 bash[23868]: audit 2024-04-26T16:25:45.400072+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:45.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:44.398679+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:44.398679+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: cluster 2024-04-26T16:25:45.092792+0000 mon.a (mon.0) 1912 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:45.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: cluster 2024-04-26T16:25:45.092792+0000 mon.a (mon.0) 1912 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:45.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.094659+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]': finished 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.094659+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]': finished 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: cluster 2024-04-26T16:25:45.099321+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: cluster 2024-04-26T16:25:45.099321+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.177554+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.177554+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.179873+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.179873+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.400072+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[21463]: audit 2024-04-26T16:25:45.400072+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:44.398679+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:44.398679+0000 mon.c (mon.2) 446 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: cluster 2024-04-26T16:25:45.092792+0000 mon.a (mon.0) 1912 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: cluster 2024-04-26T16:25:45.092792+0000 mon.a (mon.0) 1912 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.094659+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]': finished 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.094659+0000 mon.a (mon.0) 1913 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-25", "mode": "writeback"}]': finished 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: cluster 2024-04-26T16:25:45.099321+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: cluster 2024-04-26T16:25:45.099321+0000 mon.a (mon.0) 1914 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.177554+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.177554+0000 mon.a (mon.0) 1915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.179873+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.179873+0000 mon.b (mon.1) 186 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.400072+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.661 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:45 smithi050 bash[29234]: audit 2024-04-26T16:25:45.400072+0000 mon.c (mon.2) 447 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:45.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:45.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:45.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:45.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:45.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:45.937 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:45.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:46.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.062 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:46.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:46.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:46.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:46.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:46.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:46.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:46.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:46.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:46.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: cluster 2024-04-26T16:25:45.562559+0000 mgr.y (mgr.24401) 335 : cluster [DBG] pgmap v400: 375 pgs: 8 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: cluster 2024-04-26T16:25:45.562559+0000 mgr.y (mgr.24401) 335 : cluster [DBG] pgmap v400: 375 pgs: 8 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.072540+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.072540+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: cluster 2024-04-26T16:25:46.076876+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: cluster 2024-04-26T16:25:46.076876+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.079189+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.079189+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.080997+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.080997+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.089660+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.089660+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.090313+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.090313+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.401987+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:46.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[23868]: audit 2024-04-26T16:25:46.401987+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:46.583 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:46.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.713 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: cluster 2024-04-26T16:25:45.562559+0000 mgr.y (mgr.24401) 335 : cluster [DBG] pgmap v400: 375 pgs: 8 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: cluster 2024-04-26T16:25:45.562559+0000 mgr.y (mgr.24401) 335 : cluster [DBG] pgmap v400: 375 pgs: 8 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.072540+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.072540+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: cluster 2024-04-26T16:25:46.076876+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: cluster 2024-04-26T16:25:46.076876+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-04-26T16:25:46.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.079189+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.079189+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.080997+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.080997+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.089660+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.089660+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.090313+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.090313+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.401987+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:46.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[21463]: audit 2024-04-26T16:25:46.401987+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:46.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: cluster 2024-04-26T16:25:45.562559+0000 mgr.y (mgr.24401) 335 : cluster [DBG] pgmap v400: 375 pgs: 8 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:46.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: cluster 2024-04-26T16:25:45.562559+0000 mgr.y (mgr.24401) 335 : cluster [DBG] pgmap v400: 375 pgs: 8 unknown, 32 creating+peering, 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 330 active+clean; 586 KiB data, 577 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:46.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.072540+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:46.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.072540+0000 mon.a (mon.0) 1916 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:46.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: cluster 2024-04-26T16:25:46.076876+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: cluster 2024-04-26T16:25:46.076876+0000 mon.a (mon.0) 1917 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.079189+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.079189+0000 mon.a (mon.0) 1918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.080997+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.080997+0000 mon.b (mon.1) 187 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.089660+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.089660+0000 mon.c (mon.2) 448 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.090313+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.090313+0000 mon.a (mon.0) 1919 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.401987+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:46.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:46 smithi050 bash[29234]: audit 2024-04-26T16:25:46.401987+0000 mon.c (mon.2) 449 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:46.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:46.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:25:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:25:46.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:46.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:46.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:46.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:47.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:47.110 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:47.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:47.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:47.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:47.375 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:47.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:47.506 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:47.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: cluster 2024-04-26T16:25:47.072796+0000 mon.a (mon.0) 1920 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: cluster 2024-04-26T16:25:47.072796+0000 mon.a (mon.0) 1920 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.074580+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.074580+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.074710+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.074710+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: cluster 2024-04-26T16:25:47.079015+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-04-26T16:25:47.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: cluster 2024-04-26T16:25:47.079015+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.085556+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.085556+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.086981+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.086981+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.403338+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[21463]: audit 2024-04-26T16:25:47.403338+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: cluster 2024-04-26T16:25:47.072796+0000 mon.a (mon.0) 1920 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: cluster 2024-04-26T16:25:47.072796+0000 mon.a (mon.0) 1920 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.074580+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.074580+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.074710+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.074710+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: cluster 2024-04-26T16:25:47.079015+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: cluster 2024-04-26T16:25:47.079015+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.085556+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.085556+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.086981+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.086981+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.403338+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:47 smithi050 bash[29234]: audit 2024-04-26T16:25:47.403338+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:47.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:47.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:47.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: cluster 2024-04-26T16:25:47.072796+0000 mon.a (mon.0) 1920 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:47.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: cluster 2024-04-26T16:25:47.072796+0000 mon.a (mon.0) 1920 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.074580+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.074580+0000 mon.a (mon.0) 1921 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-25"}]': finished 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.074710+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.074710+0000 mon.a (mon.0) 1922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: cluster 2024-04-26T16:25:47.079015+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: cluster 2024-04-26T16:25:47.079015+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.085556+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.085556+0000 mon.c (mon.2) 450 : audit [INF] from='client.? 172.21.15.50:0/3332766413' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.086981+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.086981+0000 mon.a (mon.0) 1924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]: dispatch 2024-04-26T16:25:47.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.403338+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:47.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:47 smithi190 bash[23868]: audit 2024-04-26T16:25:47.403338+0000 mon.c (mon.2) 451 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:47.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:47.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:47.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:47.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:48.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:48.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:48.107 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:48.108 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:25:48.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:48.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:48.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:48.281 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:48.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:48.404 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:48.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:46.727147+0000 mgr.y (mgr.24401) 336 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:48.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:46.727147+0000 mgr.y (mgr.24401) 336 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:48.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: cluster 2024-04-26T16:25:47.563466+0000 mgr.y (mgr.24401) 337 : cluster [DBG] pgmap v403: 303 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 299 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-26T16:25:48.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:48.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:48.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:46.727147+0000 mgr.y (mgr.24401) 336 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:46.727147+0000 mgr.y (mgr.24401) 336 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: cluster 2024-04-26T16:25:47.563466+0000 mgr.y (mgr.24401) 337 : cluster [DBG] pgmap v403: 303 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 299 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: cluster 2024-04-26T16:25:47.563466+0000 mgr.y (mgr.24401) 337 : cluster [DBG] pgmap v403: 303 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 299 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.077390+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.077390+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: cluster 2024-04-26T16:25:48.081917+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: cluster 2024-04-26T16:25:48.081917+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.117641+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.117641+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.118408+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.118408+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.119257+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.119257+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.119618+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.119618+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.120207+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.120207+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.120561+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.120561+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.404340+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:48.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:48 smithi190 bash[23868]: audit 2024-04-26T16:25:48.404340+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:48.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:48.663 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:48.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:48.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:46.727147+0000 mgr.y (mgr.24401) 336 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:46.727147+0000 mgr.y (mgr.24401) 336 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: cluster 2024-04-26T16:25:47.563466+0000 mgr.y (mgr.24401) 337 : cluster [DBG] pgmap v403: 303 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 299 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: cluster 2024-04-26T16:25:47.563466+0000 mgr.y (mgr.24401) 337 : cluster [DBG] pgmap v403: 303 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 299 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.077390+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.077390+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: cluster 2024-04-26T16:25:48.081917+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: cluster 2024-04-26T16:25:48.081917+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.117641+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.117641+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.118408+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.118408+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.119257+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.119257+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.119618+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.119618+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: cluster 2024-04-26T16:25:47.563466+0000 mgr.y (mgr.24401) 337 : cluster [DBG] pgmap v403: 303 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 299 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.077390+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.077390+0000 mon.a (mon.0) 1925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi050-73193-35"}]': finished 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: cluster 2024-04-26T16:25:48.081917+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: cluster 2024-04-26T16:25:48.081917+0000 mon.a (mon.0) 1926 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.117641+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.117641+0000 mon.c (mon.2) 452 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.118408+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.118408+0000 mon.a (mon.0) 1927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.780 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.119257+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.119257+0000 mon.c (mon.2) 453 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.119618+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.119618+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.120207+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.120207+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.120561+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.120561+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.404340+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[21463]: audit 2024-04-26T16:25:48.404340+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.120207+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.120207+0000 mon.c (mon.2) 454 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.120561+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.120561+0000 mon.a (mon.0) 1929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:48.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.404340+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:48.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:48 smithi050 bash[29234]: audit 2024-04-26T16:25:48.404340+0000 mon.c (mon.2) 455 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:48.795 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:48.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:48.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:48.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:49.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:49.057 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:49.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:49.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:49.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:49.316 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:49.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:49.430 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: cluster 2024-04-26T16:25:48.418167+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:49.430 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: cluster 2024-04-26T16:25:48.418167+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.080768+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.080768+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: cluster 2024-04-26T16:25:49.089366+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: cluster 2024-04-26T16:25:49.089366+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.095251+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.095251+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.097539+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.097539+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: cluster 2024-04-26T16:25:48.418167+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: cluster 2024-04-26T16:25:48.418167+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:49.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.080768+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.080768+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: cluster 2024-04-26T16:25:49.089366+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: cluster 2024-04-26T16:25:49.089366+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.095251+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.095251+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.097539+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.097539+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.098969+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.098969+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.448 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:49.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:49.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:49.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: cluster 2024-04-26T16:25:48.418167+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:49.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: cluster 2024-04-26T16:25:48.418167+0000 mon.a (mon.0) 1930 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:49.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.080768+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.080768+0000 mon.a (mon.0) 1931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi050-73193-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: cluster 2024-04-26T16:25:49.089366+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: cluster 2024-04-26T16:25:49.089366+0000 mon.a (mon.0) 1932 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.095251+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.095251+0000 mon.c (mon.2) 456 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.097539+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.097539+0000 mon.a (mon.0) 1933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.098969+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.098969+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.104277+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.104277+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.405552+0000 mon.c (mon.2) 457 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:49.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:49 smithi190 bash[23868]: audit 2024-04-26T16:25:49.405552+0000 mon.c (mon.2) 457 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:49.578 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:49.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:49.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:49.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.104277+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.104277+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.405552+0000 mon.c (mon.2) 457 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:49.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[29234]: audit 2024-04-26T16:25:49.405552+0000 mon.c (mon.2) 457 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:49.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.098969+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.098969+0000 mon.b (mon.1) 188 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.104277+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.104277+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:49.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.405552+0000 mon.c (mon.2) 457 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:49.756 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:49 smithi050 bash[21463]: audit 2024-04-26T16:25:49.405552+0000 mon.c (mon.2) 457 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:49.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:49.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:49.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:49.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:49.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:50.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:50.115 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:50.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:50.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:50.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:50.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:50.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:50.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:50.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:50.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: cluster 2024-04-26T16:25:49.564014+0000 mgr.y (mgr.24401) 338 : cluster [DBG] pgmap v406: 239 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 203 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: cluster 2024-04-26T16:25:49.564014+0000 mgr.y (mgr.24401) 338 : cluster [DBG] pgmap v406: 239 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 203 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.086460+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.086460+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: cluster 2024-04-26T16:25:50.130319+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: cluster 2024-04-26T16:25:50.130319+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.134005+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.50:0/2798715303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.134005+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.50:0/2798715303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.137182+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.137182+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.407187+0000 mon.c (mon.2) 459 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[29234]: audit 2024-04-26T16:25:50.407187+0000 mon.c (mon.2) 459 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:50.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: cluster 2024-04-26T16:25:49.564014+0000 mgr.y (mgr.24401) 338 : cluster [DBG] pgmap v406: 239 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 203 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: cluster 2024-04-26T16:25:49.564014+0000 mgr.y (mgr.24401) 338 : cluster [DBG] pgmap v406: 239 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 203 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.086460+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.086460+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: cluster 2024-04-26T16:25:50.130319+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: cluster 2024-04-26T16:25:50.130319+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.134005+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.50:0/2798715303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.134005+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.50:0/2798715303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.137182+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.137182+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.407187+0000 mon.c (mon.2) 459 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:50 smithi050 bash[21463]: audit 2024-04-26T16:25:50.407187+0000 mon.c (mon.2) 459 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:50.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:50.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:50.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: cluster 2024-04-26T16:25:49.564014+0000 mgr.y (mgr.24401) 338 : cluster [DBG] pgmap v406: 239 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 203 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:50.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: cluster 2024-04-26T16:25:49.564014+0000 mgr.y (mgr.24401) 338 : cluster [DBG] pgmap v406: 239 pgs: 32 unknown, 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 203 active+clean; 586 KiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.086460+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.086460+0000 mon.a (mon.0) 1935 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: cluster 2024-04-26T16:25:50.130319+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: cluster 2024-04-26T16:25:50.130319+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.134005+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.50:0/2798715303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.134005+0000 mon.c (mon.2) 458 : audit [INF] from='client.? 172.21.15.50:0/2798715303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.137182+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.137182+0000 mon.a (mon.0) 1937 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.407187+0000 mon.c (mon.2) 459 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:50 smithi190 bash[23868]: audit 2024-04-26T16:25:50.407187+0000 mon.c (mon.2) 459 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:50.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:50.897 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:50.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:50.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:51.028 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:51.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:51.152 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:51.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:51.283 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:51.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:51.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:51.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:51.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:51.548 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:51.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:51.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:51.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:51.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:51.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:51.940 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:51.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:52.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:52.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:52.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:52.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:52.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:52.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.076781+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:52.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.076781+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:52.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.076991+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:52.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.076991+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: cluster 2024-04-26T16:25:51.085903+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: cluster 2024-04-26T16:25:51.085903+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.131694+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.131694+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.134192+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.134192+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.408469+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: audit 2024-04-26T16:25:51.408469+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: cluster 2024-04-26T16:25:51.566353+0000 mgr.y (mgr.24401) 339 : cluster [DBG] pgmap v409: 279 pgs: 8 creating+peering, 271 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:52.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:52 smithi190 bash[23868]: cluster 2024-04-26T16:25:51.566353+0000 mgr.y (mgr.24401) 339 : cluster [DBG] pgmap v409: 279 pgs: 8 creating+peering, 271 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:52.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.076781+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.076781+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.076991+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.076991+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: cluster 2024-04-26T16:25:51.085903+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: cluster 2024-04-26T16:25:51.085903+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.131694+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.131694+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.134192+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.134192+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.408469+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: audit 2024-04-26T16:25:51.408469+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: cluster 2024-04-26T16:25:51.566353+0000 mgr.y (mgr.24401) 339 : cluster [DBG] pgmap v409: 279 pgs: 8 creating+peering, 271 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[21463]: cluster 2024-04-26T16:25:51.566353+0000 mgr.y (mgr.24401) 339 : cluster [DBG] pgmap v409: 279 pgs: 8 creating+peering, 271 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.076781+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.076781+0000 mon.a (mon.0) 1938 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi050-73193-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.076991+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.076991+0000 mon.a (mon.0) 1939 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi050-73201-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: cluster 2024-04-26T16:25:51.085903+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: cluster 2024-04-26T16:25:51.085903+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.131694+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.131694+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.134192+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.134192+0000 mon.b (mon.1) 189 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:25:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.408469+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: audit 2024-04-26T16:25:51.408469+0000 mon.c (mon.2) 460 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: cluster 2024-04-26T16:25:51.566353+0000 mgr.y (mgr.24401) 339 : cluster [DBG] pgmap v409: 279 pgs: 8 creating+peering, 271 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:52 smithi050 bash[29234]: cluster 2024-04-26T16:25:51.566353+0000 mgr.y (mgr.24401) 339 : cluster [DBG] pgmap v409: 279 pgs: 8 creating+peering, 271 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.5 KiB/s wr, 1 op/s 2024-04-26T16:25:52.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:52.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:52.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:52.589 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:52.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:52.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:52.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:52.852 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:52.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:52.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:52.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:53.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:53.122 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:53.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:53.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:53.252 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:53.277 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.085130+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:53.277 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.085130+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:53.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.085130+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.085130+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: cluster 2024-04-26T16:25:52.090962+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: cluster 2024-04-26T16:25:52.090962+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.098727+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.098727+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.102491+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.102491+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.410156+0000 mon.c (mon.2) 461 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:53.364 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:53 smithi190 bash[23868]: audit 2024-04-26T16:25:52.410156+0000 mon.c (mon.2) 461 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:53.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.085130+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:53.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: cluster 2024-04-26T16:25:52.090962+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-04-26T16:25:53.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: cluster 2024-04-26T16:25:52.090962+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-04-26T16:25:53.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.098727+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.098727+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.102491+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.102491+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.410156+0000 mon.c (mon.2) 461 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[21463]: audit 2024-04-26T16:25:52.410156+0000 mon.c (mon.2) 461 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.085130+0000 mon.a (mon.0) 1942 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: cluster 2024-04-26T16:25:52.090962+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: cluster 2024-04-26T16:25:52.090962+0000 mon.a (mon.0) 1943 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.098727+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.098727+0000 mon.b (mon.1) 190 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.102491+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.102491+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:53.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.410156+0000 mon.c (mon.2) 461 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:53.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:53 smithi050 bash[29234]: audit 2024-04-26T16:25:52.410156+0000 mon.c (mon.2) 461 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.381 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:53.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:53.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:53.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:53.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:53.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:53.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:53.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:53.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:53.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:53.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:54.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.035 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:54.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:54.166 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:54.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.088415+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:54.279 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.088415+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:53.098123+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:53.098123+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.099984+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.099984+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.100152+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.100152+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.100300+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.50:0/2788753622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.100300+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.50:0/2788753622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.103972+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.103972+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.217569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.217569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.280 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.218309+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.088415+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.088415+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:53.098123+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:53.098123+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.099984+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.099984+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.100152+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.100152+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.100300+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.50:0/2788753622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.100300+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.50:0/2788753622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.281 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.103972+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.103972+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.217569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.217569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.218309+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.218309+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.411772+0000 mon.c (mon.2) 464 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:54.282 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:53.411772+0000 mon.c (mon.2) 464 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:53.566989+0000 mgr.y (mgr.24401) 340 : cluster [DBG] pgmap v412: 271 pgs: 32 unknown, 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:53.566989+0000 mgr.y (mgr.24401) 340 : cluster [DBG] pgmap v412: 271 pgs: 32 unknown, 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:54.088902+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:54.088902+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.090853+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]': finished 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.090853+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]': finished 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.090942+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.090942+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.091060+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.091060+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:54.096353+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: cluster 2024-04-26T16:25:54.096353+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.098983+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.098983+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.283 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.100336+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21463]: audit 2024-04-26T16:25:54.100336+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.218309+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.411772+0000 mon.c (mon.2) 464 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:53.411772+0000 mon.c (mon.2) 464 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:53.566989+0000 mgr.y (mgr.24401) 340 : cluster [DBG] pgmap v412: 271 pgs: 32 unknown, 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:53.566989+0000 mgr.y (mgr.24401) 340 : cluster [DBG] pgmap v412: 271 pgs: 32 unknown, 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:54.088902+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:54.088902+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.090853+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]': finished 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.090853+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]': finished 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.090942+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:54.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.090942+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.091060+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.091060+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:54.096353+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: cluster 2024-04-26T16:25:54.096353+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.098983+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.098983+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.100336+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.285 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[29234]: audit 2024-04-26T16:25:54.100336+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:54.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.088415+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.088415+0000 mon.a (mon.0) 1945 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:53.098123+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:53.098123+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.099984+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.099984+0000 mon.c (mon.2) 462 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.100152+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.409 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.100152+0000 mon.b (mon.1) 191 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.100300+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.50:0/2788753622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.100300+0000 mon.c (mon.2) 463 : audit [INF] from='client.? 172.21.15.50:0/2788753622' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.103972+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.103972+0000 mon.a (mon.0) 1947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.217569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.217569+0000 mon.a (mon.0) 1948 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.218309+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.410 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.218309+0000 mon.a (mon.0) 1949 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.411772+0000 mon.c (mon.2) 464 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:53.411772+0000 mon.c (mon.2) 464 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:53.566989+0000 mgr.y (mgr.24401) 340 : cluster [DBG] pgmap v412: 271 pgs: 32 unknown, 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:53.566989+0000 mgr.y (mgr.24401) 340 : cluster [DBG] pgmap v412: 271 pgs: 32 unknown, 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:54.088902+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:54.088902+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.090853+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]': finished 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.090853+0000 mon.a (mon.0) 1951 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-27", "mode": "writeback"}]': finished 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.090942+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.090942+0000 mon.a (mon.0) 1952 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.091060+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.091060+0000 mon.a (mon.0) 1953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi050-73201-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:25:54.411 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:54.096353+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-04-26T16:25:54.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: cluster 2024-04-26T16:25:54.096353+0000 mon.a (mon.0) 1954 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2024-04-26T16:25:54.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.098983+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.098983+0000 mon.c (mon.2) 465 : audit [INF] from='client.? 172.21.15.50:0/3710165660' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.100336+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:54 smithi190 bash[23868]: audit 2024-04-26T16:25:54.100336+0000 mon.a (mon.0) 1955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]: dispatch 2024-04-26T16:25:54.426 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:54.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.541 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:25:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:25:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:25:54.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:54.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:54.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:54.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:54.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:54.954 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:54.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:55.055 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:55.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:55.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:55.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:55.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[29234]: audit 2024-04-26T16:25:54.413427+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[29234]: audit 2024-04-26T16:25:54.413427+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[29234]: audit 2024-04-26T16:25:55.093139+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[29234]: audit 2024-04-26T16:25:55.093139+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[29234]: cluster 2024-04-26T16:25:55.097392+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[29234]: cluster 2024-04-26T16:25:55.097392+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[21463]: audit 2024-04-26T16:25:54.413427+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:55.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[21463]: audit 2024-04-26T16:25:54.413427+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:55.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[21463]: audit 2024-04-26T16:25:55.093139+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:55.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[21463]: audit 2024-04-26T16:25:55.093139+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:55.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[21463]: cluster 2024-04-26T16:25:55.097392+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-04-26T16:25:55.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:55 smithi050 bash[21463]: cluster 2024-04-26T16:25:55.097392+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-04-26T16:25:55.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:55 smithi190 bash[23868]: audit 2024-04-26T16:25:54.413427+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:55.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:55 smithi190 bash[23868]: audit 2024-04-26T16:25:54.413427+0000 mon.c (mon.2) 466 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:55.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:55 smithi190 bash[23868]: audit 2024-04-26T16:25:55.093139+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:55.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:55 smithi190 bash[23868]: audit 2024-04-26T16:25:55.093139+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi050-73193-36"}]': finished 2024-04-26T16:25:55.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:55 smithi190 bash[23868]: cluster 2024-04-26T16:25:55.097392+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-04-26T16:25:55.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:55 smithi190 bash[23868]: cluster 2024-04-26T16:25:55.097392+0000 mon.a (mon.0) 1957 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2024-04-26T16:25:55.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:55.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:55.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:55.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:55.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:55.590 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:55.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:55.719 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:55.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:55.849 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:55.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:55.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:55.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:56.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:56.108 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:56.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:56.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:56.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.135008+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.135008+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.137590+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.137590+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.138411+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.138411+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.139067+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.139067+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.139572+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.139572+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.140797+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.140797+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.141294+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.141294+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.135008+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.135008+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.137590+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.137590+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.138411+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.138411+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.139067+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.139067+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.139572+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.139572+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.140797+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.140797+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.141294+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.141294+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.141847+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.141847+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.142194+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.142194+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.193485+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.193485+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.225 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.195857+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.195857+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.414787+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:55.414787+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: cluster 2024-04-26T16:25:55.567633+0000 mgr.y (mgr.24401) 341 : cluster [DBG] pgmap v415: 239 pgs: 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: cluster 2024-04-26T16:25:55.567633+0000 mgr.y (mgr.24401) 341 : cluster [DBG] pgmap v415: 239 pgs: 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: cluster 2024-04-26T16:25:56.075910+0000 mon.a (mon.0) 1965 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: cluster 2024-04-26T16:25:56.075910+0000 mon.a (mon.0) 1965 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:56.226 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.080011+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.141847+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.227 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.141847+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.142194+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.142194+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.193485+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.193485+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.195857+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.195857+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.414787+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:56.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:55.414787+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: cluster 2024-04-26T16:25:55.567633+0000 mgr.y (mgr.24401) 341 : cluster [DBG] pgmap v415: 239 pgs: 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: cluster 2024-04-26T16:25:55.567633+0000 mgr.y (mgr.24401) 341 : cluster [DBG] pgmap v415: 239 pgs: 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: cluster 2024-04-26T16:25:56.075910+0000 mon.a (mon.0) 1965 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: cluster 2024-04-26T16:25:56.075910+0000 mon.a (mon.0) 1965 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.080011+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.080011+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.080215+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.080215+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.080440+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.080440+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.087402+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.087402+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: cluster 2024-04-26T16:25:56.088119+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: cluster 2024-04-26T16:25:56.088119+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.088785+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.088785+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.230 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.089606+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.089606+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.092100+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.092100+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.093886+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[21463]: audit 2024-04-26T16:25:56.093886+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.080011+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.080215+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.080215+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.080440+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:56.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.080440+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.087402+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.087402+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: cluster 2024-04-26T16:25:56.088119+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: cluster 2024-04-26T16:25:56.088119+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.088785+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.088785+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.089606+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.089606+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.092100+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.092100+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.093886+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.232 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:56 smithi050 bash[29234]: audit 2024-04-26T16:25:56.093886+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:56.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:56.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:56.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:56.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.135008+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.135008+0000 mon.a (mon.0) 1958 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.137590+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.137590+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.138411+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.138411+0000 mon.c (mon.2) 467 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.139067+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.139067+0000 mon.a (mon.0) 1960 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.139572+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.139572+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.140797+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.140797+0000 mon.c (mon.2) 468 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.141294+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.141294+0000 mon.a (mon.0) 1962 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.141847+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.141847+0000 mon.c (mon.2) 469 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.142194+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.142194+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:25:56.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.193485+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.193485+0000 mon.a (mon.0) 1964 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.195857+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.195857+0000 mon.b (mon.1) 192 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.414787+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:55.414787+0000 mon.c (mon.2) 470 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: cluster 2024-04-26T16:25:55.567633+0000 mgr.y (mgr.24401) 341 : cluster [DBG] pgmap v415: 239 pgs: 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: cluster 2024-04-26T16:25:55.567633+0000 mgr.y (mgr.24401) 341 : cluster [DBG] pgmap v415: 239 pgs: 239 active+clean; 586 KiB data, 587 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: cluster 2024-04-26T16:25:56.075910+0000 mon.a (mon.0) 1965 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:56.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: cluster 2024-04-26T16:25:56.075910+0000 mon.a (mon.0) 1965 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.080011+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.080011+0000 mon.a (mon.0) 1966 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.080215+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.080215+0000 mon.a (mon.0) 1967 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.080440+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.080440+0000 mon.a (mon.0) 1968 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.087402+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.087402+0000 mon.c (mon.2) 471 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: cluster 2024-04-26T16:25:56.088119+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: cluster 2024-04-26T16:25:56.088119+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.088785+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.088785+0000 mon.a (mon.0) 1970 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.089606+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.089606+0000 mon.a (mon.0) 1971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:25:56.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.092100+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.092100+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.093886+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[23868]: audit 2024-04-26T16:25:56.093886+0000 mon.b (mon.1) 193 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]: dispatch 2024-04-26T16:25:56.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:56.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:56.630 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:56.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:56.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:25:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:25:56.761 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:56.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:56.892 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:56.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:56.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:56.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:57.024 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:57.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:57.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:57.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:57.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.415920+0000 mon.c (mon.2) 472 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.415920+0000 mon.c (mon.2) 472 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.416652+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.416652+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.416958+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.416958+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.434990+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.434990+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.436528+0000 mon.c (mon.2) 474 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:56.436528+0000 mon.c (mon.2) 474 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: cluster 2024-04-26T16:25:57.081213+0000 mon.a (mon.0) 1975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: cluster 2024-04-26T16:25:57.081213+0000 mon.a (mon.0) 1975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.083546+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.083546+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.083778+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.083778+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:25:57.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.095389+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.095389+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.097256+0000 mon.c (mon.2) 476 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.097256+0000 mon.c (mon.2) 476 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.100503+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.100503+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: cluster 2024-04-26T16:25:57.122395+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: cluster 2024-04-26T16:25:57.122395+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.124220+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:57 smithi190 bash[23868]: audit 2024-04-26T16:25:57.124220+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.283 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.415920+0000 mon.c (mon.2) 472 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.415920+0000 mon.c (mon.2) 472 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.416652+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.416652+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.416958+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.416958+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.434990+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.434990+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.436528+0000 mon.c (mon.2) 474 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:56.436528+0000 mon.c (mon.2) 474 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: cluster 2024-04-26T16:25:57.081213+0000 mon.a (mon.0) 1975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: cluster 2024-04-26T16:25:57.081213+0000 mon.a (mon.0) 1975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.083546+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.083546+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.083778+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.083778+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.095389+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.415920+0000 mon.c (mon.2) 472 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.415920+0000 mon.c (mon.2) 472 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.416652+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.416652+0000 mon.c (mon.2) 473 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.416958+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.416958+0000 mon.a (mon.0) 1973 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.434990+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.434990+0000 mon.a (mon.0) 1974 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.436528+0000 mon.c (mon.2) 474 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:56.436528+0000 mon.c (mon.2) 474 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: cluster 2024-04-26T16:25:57.081213+0000 mon.a (mon.0) 1975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: cluster 2024-04-26T16:25:57.081213+0000 mon.a (mon.0) 1975 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.083546+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.083546+0000 mon.a (mon.0) 1976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-27"}]': finished 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.083778+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.083778+0000 mon.a (mon.0) 1977 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pg_num","val":"11"}]': finished 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.095389+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.095389+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.097256+0000 mon.c (mon.2) 476 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.097256+0000 mon.c (mon.2) 476 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.100503+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.100503+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: cluster 2024-04-26T16:25:57.122395+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: cluster 2024-04-26T16:25:57.122395+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.124220+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[29234]: audit 2024-04-26T16:25:57.124220+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.095389+0000 mon.c (mon.2) 475 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.097256+0000 mon.c (mon.2) 476 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.097256+0000 mon.c (mon.2) 476 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.100503+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.100503+0000 mon.c (mon.2) 477 : audit [INF] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: cluster 2024-04-26T16:25:57.122395+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-04-26T16:25:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: cluster 2024-04-26T16:25:57.122395+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2024-04-26T16:25:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.124220+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:57 smithi050 bash[21463]: audit 2024-04-26T16:25:57.124220+0000 mon.a (mon.0) 1979 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]: dispatch 2024-04-26T16:25:57.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:57.413 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:57.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:57.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:57.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:57.672 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:57.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:57.801 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:25:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:25:57.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:57.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:57.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:58.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:58.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:58.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:58.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:58.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:56.738641+0000 mgr.y (mgr.24401) 342 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:56.738641+0000 mgr.y (mgr.24401) 342 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: cluster 2024-04-26T16:25:57.568128+0000 mgr.y (mgr.24401) 343 : cluster [DBG] pgmap v418: 239 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 233 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: cluster 2024-04-26T16:25:57.568128+0000 mgr.y (mgr.24401) 343 : cluster [DBG] pgmap v418: 239 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 233 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.087441+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.087441+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.087677+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.087677+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.087914+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.087914+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:25:58.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: cluster 2024-04-26T16:25:58.095063+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-04-26T16:25:58.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: cluster 2024-04-26T16:25:58.095063+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-04-26T16:25:58.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.115521+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:58.306 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:25:58 smithi190 bash[23868]: audit 2024-04-26T16:25:58.115521+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:58.322 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:58.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:56.738641+0000 mgr.y (mgr.24401) 342 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:56.738641+0000 mgr.y (mgr.24401) 342 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: cluster 2024-04-26T16:25:57.568128+0000 mgr.y (mgr.24401) 343 : cluster [DBG] pgmap v418: 239 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 233 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: cluster 2024-04-26T16:25:57.568128+0000 mgr.y (mgr.24401) 343 : cluster [DBG] pgmap v418: 239 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 233 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.087441+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.087441+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.087677+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.087677+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.087914+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.087914+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: cluster 2024-04-26T16:25:58.095063+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: cluster 2024-04-26T16:25:58.095063+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-04-26T16:25:58.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.115521+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[21463]: audit 2024-04-26T16:25:58.115521+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:56.738641+0000 mgr.y (mgr.24401) 342 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:56.738641+0000 mgr.y (mgr.24401) 342 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: cluster 2024-04-26T16:25:57.568128+0000 mgr.y (mgr.24401) 343 : cluster [DBG] pgmap v418: 239 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 233 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: cluster 2024-04-26T16:25:57.568128+0000 mgr.y (mgr.24401) 343 : cluster [DBG] pgmap v418: 239 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 233 active+clean; 586 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.087441+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.087441+0000 mon.a (mon.0) 1980 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:25:58.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.087677+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.087677+0000 mon.a (mon.0) 1981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi050-73193-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.087914+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.087914+0000 mon.a (mon.0) 1982 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi050-73230-1","var":"pgp_num","val":"11"}]': finished 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: cluster 2024-04-26T16:25:58.095063+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: cluster 2024-04-26T16:25:58.095063+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.115521+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:58.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:25:58 smithi050 bash[29234]: audit 2024-04-26T16:25:58.115521+0000 mon.c (mon.2) 478 : audit [DBG] from='client.? 172.21.15.50:0/2426299560' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-04-26T16:25:58.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:25:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:25:58.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:58.585 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:58.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:58.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:58.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:25:58.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:25:58.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:58.848 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:58.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:58.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:25:58.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:58.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:25:59.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:59.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:25:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:25:59.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:59.251 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:25:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:25:59.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:59.381 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:59.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:59.511 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:59.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:59.533 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:25:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-26T16:25:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-26T16:25:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-26T16:25:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.533 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14391.0:43 3.2 3:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e81)", 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:17:31.027527+0000", 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 508.508683862, 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024913109999999999, 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14391", 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 43 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.534 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027527+0000", 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027527+0000", 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.535 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027530+0000", 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2430000000000001e-06 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027546+0000", 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6133e-05 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027580+0000", 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3340000000000003e-05 2024-04-26T16:25:59.536 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027607+0000", 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7319000000000001e-05 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027788+0000", 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00018048700000000001 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:25:59.537 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.027804+0000", 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.679e-05 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.039054+0000", 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011249979 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.039079+0000", 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4341999999999998e-05 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.538 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.052141+0000", 2024-04-26T16:25:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013062261 2024-04-26T16:25:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.539 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:17:31.052440+0000", 2024-04-26T16:25:59.540 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00029921599999999999 2024-04-26T16:25:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.541 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.36058.0:2 156.0 156:0384ab8e:::oid:head [zero 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-04-26T16:25:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:23:59.052074+0000", 2024-04-26T16:25:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 120.484136518, 2024-04-26T16:25:59.542 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1515340540000001, 2024-04-26T16:25:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:59.543 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.36058", 2024-04-26T16:25:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2554158474", 2024-04-26T16:25:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.544 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.052074+0000", 2024-04-26T16:25:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.545 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.052074+0000", 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.052077+0000", 2024-04-26T16:25:59.546 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2859999999999998e-06 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.052083+0000", 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0920000000000003e-06 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.547 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.052096+0000", 2024-04-26T16:25:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3078000000000001e-05 2024-04-26T16:25:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.548 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.052108+0000", 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2661e-05 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.094443+0000", 2024-04-26T16:25:59.549 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.042334526999999997 2024-04-26T16:25:59.550 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.550 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.550 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:59.550 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.094452+0000", 2024-04-26T16:25:59.550 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5049999999999996e-06 2024-04-26T16:25:59.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.551 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.551 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.551 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.165590+0000", 2024-04-26T16:25:59.551 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.071137048 2024-04-26T16:25:59.551 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.552 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.552 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.165684+0000", 2024-04-26T16:25:59.552 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4629999999999994e-05 2024-04-26T16:25:59.552 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.552 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.553 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.553 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.203608+0000", 2024-04-26T16:25:59.553 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.037924226999999998 2024-04-26T16:25:59.553 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.554 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.554 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.554 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.554 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.554 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.23265.0:2 152.6 152:602f83fe:::foo:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-04-26T16:25:59.555 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:23:59.115713+0000", 2024-04-26T16:25:59.555 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 120.420497748, 2024-04-26T16:25:59.555 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044615993999999999, 2024-04-26T16:25:59.556 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.556 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.556 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.556 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.556 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.23265", 2024-04-26T16:25:59.556 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1150997960", 2024-04-26T16:25:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.557 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.115713+0000", 2024-04-26T16:25:59.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.558 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.558 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.558 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.558 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.115713+0000", 2024-04-26T16:25:59.558 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.559 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.559 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.559 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.559 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.115715+0000", 2024-04-26T16:25:59.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.578e-06 2024-04-26T16:25:59.560 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.560 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.560 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.560 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.115721+0000", 2024-04-26T16:25:59.560 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5980000000000002e-06 2024-04-26T16:25:59.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.561 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.561 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.561 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.116096+0000", 2024-04-26T16:25:59.561 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00037493900000000002 2024-04-26T16:25:59.561 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.562 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.562 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.116113+0000", 2024-04-26T16:25:59.562 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6608000000000001e-05 2024-04-26T16:25:59.562 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.562 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.563 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.563 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.116425+0000", 2024-04-26T16:25:59.563 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00031243900000000002 2024-04-26T16:25:59.563 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.563 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.564 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:59.564 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.116435+0000", 2024-04-26T16:25:59.564 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.3950000000000007e-06 2024-04-26T16:25:59.564 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.564 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.565 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.565 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.141670+0000", 2024-04-26T16:25:59.565 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025235377999999999 2024-04-26T16:25:59.565 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.565 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.566 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.566 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.141781+0000", 2024-04-26T16:25:59.566 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110917 2024-04-26T16:25:59.566 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.567 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,7", 2024-04-26T16:25:59.567 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.141923+0000", 2024-04-26T16:25:59.567 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014211000000000001 2024-04-26T16:25:59.567 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.567 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.153271+0000", 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011348052000000001 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.155426+0000", 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0021544680000000001 2024-04-26T16:25:59.568 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.160228+0000", 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004802817 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.160283+0000", 2024-04-26T16:25:59.569 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4471000000000001e-05 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.160329+0000", 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6224000000000003e-05 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.570 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.38401.0:13 163.1b e175/174 163:d96a01e0:::testobj:head v 175'4, mlcod=175'2)", 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:03.211134+0000", 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 116.325077299, 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016078167000000001, 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.571 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211134+0000", 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211134+0000", 2024-04-26T16:25:59.572 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211135+0000", 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3289999999999999e-06 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211149+0000", 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4518000000000001e-05 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.573 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211170+0000", 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0508000000000001e-05 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211181+0000", 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1148000000000001e-05 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211264+0000", 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2848000000000004e-05 2024-04-26T16:25:59.574 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.211286+0000", 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2350000000000001e-05 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.227130+0000", 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015843258999999998 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.227212+0000", 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2206999999999994e-05 2024-04-26T16:25:59.575 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.1.0:652 17.1c e176/141 -19:3be0d449:::temp_17.1c_0_24161_1:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.325126+0000", 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.211084828, 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016214664, 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.576 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.325126+0000", 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.325126+0000", 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.325128+0000", 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.829e-06 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.577 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.329367+0000", 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042395710000000001 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.329397+0000", 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0006e-05 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.578 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.329408+0000", 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0886999999999999e-05 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.329493+0000", 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.4308000000000007e-05 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.329527+0000", 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4687000000000002e-05 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.579 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.341272+0000", 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011744601 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.341341+0000", 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8775000000000005e-05 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.580 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.1.0:653 17.1c e176/141 -19:3be0d449:::temp_17.1c_0_24161_1:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.357366+0000", 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.178845228, 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013961938, 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.357366+0000", 2024-04-26T16:25:59.581 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.357366+0000", 2024-04-26T16:25:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.582 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.583 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.583 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.357367+0000", 2024-04-26T16:25:59.583 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.795e-06 2024-04-26T16:25:59.583 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.583 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.363646+0000", 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006279013 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.585 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.363675+0000", 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8997999999999999e-05 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.363690+0000", 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4280999999999999e-05 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.586 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.587 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.363767+0000", 2024-04-26T16:25:59.587 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7057000000000004e-05 2024-04-26T16:25:59.587 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.587 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.587 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.363791+0000", 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4360000000000001e-05 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.371258+0000", 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.007467034 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.588 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.371328+0000", 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9400000000000006e-05 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.1.0:654 17.1c e176/141 -19:3be0d449:::temp_17.1c_0_24161_1:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.382227+0000", 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.15398409300001, 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017054143000000001, 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.589 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.382227+0000", 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.382227+0000", 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.590 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.382229+0000", 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9350000000000001e-06 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.385344+0000", 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031156740000000001 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.385373+0000", 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8476999999999999e-05 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.591 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.385387+0000", 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4429e-05 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.385505+0000", 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011777600000000001 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.385529+0000", 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.421e-05 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.592 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.399212+0000", 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013682829000000001 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.399281+0000", 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8813000000000007e-05 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.593 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:870 17.1b e176/141 -19:ddd8974c:::temp_17.1b_0_24223_2:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.428820+0000", 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.107390373, 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026800180999999999, 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.428820+0000", 2024-04-26T16:25:59.594 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.428820+0000", 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.428822+0000", 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9139999999999998e-06 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.595 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.446580+0000", 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017757829999999999 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.446626+0000", 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5964999999999998e-05 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.446643+0000", 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6671e-05 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.596 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.446779+0000", 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013587199999999999 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.446808+0000", 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8901999999999999e-05 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.455578+0000", 2024-04-26T16:25:59.597 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0087702409999999998 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.455621+0000", 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2786e-05 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.598 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:871 17.1b e176/141 -19:ddd8974c:::temp_17.1b_0_24223_2:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.455322+0000", 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.080888513, 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025138193, 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.455322+0000", 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.599 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.455322+0000", 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.455323+0000", 2024-04-26T16:25:59.600 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6300000000000004e-07 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.468251+0000", 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012927444999999999 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.468332+0000", 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1650999999999995e-05 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.601 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.468365+0000", 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3083999999999997e-05 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.468463+0000", 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7479000000000006e-05 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.468490+0000", 2024-04-26T16:25:59.602 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6988e-05 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.480391+0000", 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011901382 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.480461+0000", 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9301000000000006e-05 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.603 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:872 17.1b e176/141 -19:ddd8974c:::temp_17.1b_0_24223_2:head v 0'0, mlcod=0'0)", 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.475439+0000", 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.060772164, 2024-04-26T16:25:59.604 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.035986285999999999, 2024-04-26T16:25:59.605 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.605 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.605 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.605 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.605 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.605 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.475439+0000", 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.475439+0000", 2024-04-26T16:25:59.606 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.607 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.607 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.607 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.475440+0000", 2024-04-26T16:25:59.607 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6139999999999999e-06 2024-04-26T16:25:59.607 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.608 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.608 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.488387+0000", 2024-04-26T16:25:59.608 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012946841000000001 2024-04-26T16:25:59.608 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.608 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.488484+0000", 2024-04-26T16:25:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7041999999999997e-05 2024-04-26T16:25:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.609 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.488516+0000", 2024-04-26T16:25:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2277999999999998e-05 2024-04-26T16:25:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.610 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.488613+0000", 2024-04-26T16:25:59.611 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6841000000000003e-05 2024-04-26T16:25:59.611 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.611 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.611 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.611 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.488640+0000", 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6985e-05 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.511208+0000", 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.022568002 2024-04-26T16:25:59.612 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.613 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.613 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.613 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.511425+0000", 2024-04-26T16:25:59.613 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021668300000000001 2024-04-26T16:25:59.613 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.613 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.24770.0:2079 17.1b e176/141 17:ddd8974c:test-rados-api-smithi050-73363-17::big2.copy:head v 176'2, mlcod=0'0)", 2024-04-26T16:25:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.492616+0000", 2024-04-26T16:25:59.614 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.043594934, 2024-04-26T16:25:59.615 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.018966274000000002, 2024-04-26T16:25:59.615 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.615 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.615 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.615 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.615 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.492616+0000", 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.616 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.492616+0000", 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.492618+0000", 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5969999999999999e-06 2024-04-26T16:25:59.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.494980+0000", 2024-04-26T16:25:59.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002362931 2024-04-26T16:25:59.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.495062+0000", 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1709e-05 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.495117+0000", 2024-04-26T16:25:59.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4398000000000003e-05 2024-04-26T16:25:59.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.495219+0000", 2024-04-26T16:25:59.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102056 2024-04-26T16:25:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.495253+0000", 2024-04-26T16:25:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4585000000000002e-05 2024-04-26T16:25:59.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.511433+0000", 2024-04-26T16:25:59.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016179408999999999 2024-04-26T16:25:59.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.511582+0000", 2024-04-26T16:25:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014958899999999999 2024-04-26T16:25:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.624 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24770.0:2080 17.9 17:9514177a:test-rados-api-smithi050-73363-17::big3.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-04-26T16:25:59.624 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.533874+0000", 2024-04-26T16:25:59.624 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 115.00233671700001, 2024-04-26T16:25:59.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.103691567, 2024-04-26T16:25:59.625 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.625 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.625 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24770", 2024-04-26T16:25:59.625 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2506934054", 2024-04-26T16:25:59.626 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2080 2024-04-26T16:25:59.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.626 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.533874+0000", 2024-04-26T16:25:59.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.533874+0000", 2024-04-26T16:25:59.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.533877+0000", 2024-04-26T16:25:59.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0000000000000001e-06 2024-04-26T16:25:59.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.533890+0000", 2024-04-26T16:25:59.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2534999999999999e-05 2024-04-26T16:25:59.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.533918+0000", 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8742999999999999e-05 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.630 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.533932+0000", 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3869000000000001e-05 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.534094+0000", 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016171499999999999 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.534239+0000", 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014499399999999999 2024-04-26T16:25:59.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.614481+0000", 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.080242424000000007 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.626020+0000", 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011539018999999999 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.629489+0000", 2024-04-26T16:25:59.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034684630000000002 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.637441+0000", 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0079525289999999998 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.637502+0000", 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0383000000000001e-05 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.637566+0000", 2024-04-26T16:25:59.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3893000000000006e-05 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.38066.0:2 167.0 167:0384ab8e:::oid:head [append 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:05.107351+0000", 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.428859722, 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.023771934, 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.634 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.38066", 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1040550484", 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107351+0000", 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107351+0000", 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107353+0000", 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7570000000000001e-06 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107364+0000", 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0608000000000001e-05 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107377+0000", 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3188e-05 2024-04-26T16:25:59.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107388+0000", 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1221e-05 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107445+0000", 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7346000000000003e-05 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.107455+0000", 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5740000000000002e-06 2024-04-26T16:25:59.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.118607+0000", 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.011152064 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.118728+0000", 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121375 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,6", 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.118876+0000", 2024-04-26T16:25:59.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014751 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.121989+0000", 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031128499999999999 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.122452+0000", 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046381800000000001 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.131002+0000", 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0085494839999999996 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.131061+0000", 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8967e-05 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:06.131123+0000", 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2172e-05 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.640 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.37397.0:2 165.6 165:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:05.161780+0000", 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 114.374430389, 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015958551000000001, 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.37397", 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1824440060", 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.641 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.161780+0000", 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.161780+0000", 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.161782+0000", 2024-04-26T16:25:59.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.57e-06 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.161788+0000", 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2720000000000003e-06 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.161801+0000", 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2228e-05 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.161813+0000", 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2622999999999999e-05 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.162101+0000", 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00028744699999999997 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.162236+0000", 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000135251 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 4,6", 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.162375+0000", 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000138992 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.171240+0000", 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0088654960000000005 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.175710+0000", 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0044695830000000001 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.177634+0000", 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019237189999999999 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.177688+0000", 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4688000000000002e-05 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.177739+0000", 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0682e-05 2024-04-26T16:25:59.646 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:873 173.10 173:0807fb86:test-rados-api-smithi050-73372-14::foo0:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e190)", 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:18.043752+0000", 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 101.492458728, 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.063931968000000006, 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:59.647 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 873 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.043752+0000", 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.043752+0000", 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.043756+0000", 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.194e-06 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.043779+0000", 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2690000000000001e-05 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.043835+0000", 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6066999999999997e-05 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.043864+0000", 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8892999999999999e-05 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.044108+0000", 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024365699999999999 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.044123+0000", 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5726000000000002e-05 2024-04-26T16:25:59.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.053684+0000", 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0095603319999999995 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for blocked object", 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.053996+0000", 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000312434 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.085599+0000", 2024-04-26T16:25:59.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.031603330999999998 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.085714+0000", 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011457500000000001 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.107372+0000", 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021657837999999999 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.107485+0000", 2024-04-26T16:25:59.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000112803 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:18.107684+0000", 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000199428 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.29895.0:2 183.6 183:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:20.223991+0000", 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 99.312220154000002, 2024-04-26T16:25:59.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015955713, 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.29895", 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/626245122", 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.223991+0000", 2024-04-26T16:25:59.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.223991+0000", 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.223993+0000", 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.633e-06 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.224000+0000", 2024-04-26T16:25:59.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0029999999999997e-06 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.224013+0000", 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2354999999999999e-05 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.224022+0000", 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4939999999999992e-06 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.656 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.225973+0000", 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00195043 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.225983+0000", 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0609e-05 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.231664+0000", 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0056811159999999999 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.231857+0000", 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000192277 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,4", 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.232079+0000", 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022195900000000001 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.234089+0000", 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0020108090000000001 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.234640+0000", 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055026999999999995 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.239758+0000", 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00511802 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.239867+0000", 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109147 2024-04-26T16:25:59.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:20.239946+0000", 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9591000000000004e-05 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:1016 109.5 e221/155 109:a71b5931:test-rados-api-smithi050-73372-14::foo90:6 v 221'22, mlcod=187'16)", 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:49.065361+0000", 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 70.470849959999995, 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021999343000000001, 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.660 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.065361+0000", 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.065361+0000", 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.065361+0000", 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.96e-07 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.065366+0000", 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0080000000000002e-06 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.065373+0000", 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8990000000000004e-06 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.662 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.069835+0000", 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.004462237 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.079061+0000", 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0092259260000000006 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.079099+0000", 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8087000000000003e-05 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.087318+0000", 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0082189829999999992 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.087360+0000", 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1807000000000001e-05 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.664 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:1100 173.10 173:08000000:%01:::head [pgnls start_epoch 224] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e224)", 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:52.088124+0000", 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 67.448086648, 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021803897999999999, 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1100 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.088124+0000", 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.088124+0000", 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.088126+0000", 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0219999999999999e-06 2024-04-26T16:25:59.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.088149+0000", 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3110000000000001e-05 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.103741+0000", 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015591838 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.103770+0000", 2024-04-26T16:25:59.667 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8595999999999999e-05 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.109517+0000", 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057468409999999999 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.109585+0000", 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8654999999999999e-05 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.668 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.109928+0000", 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034283600000000001 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop_reply(osd.3.0:1284 173.11 e224/182 ondisk, result = 0)", 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:52.098355+0000", 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 67.437855553000006, 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.023210607000000001, 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:25:59.669 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.098355+0000", 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.098355+0000", 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.098357+0000", 2024-04-26T16:25:59.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4869999999999999e-06 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.098365+0000", 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9130000000000003e-06 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.111230+0000", 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012865288000000001 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.111248+0000", 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8377e-05 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.121540+0000", 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010291272000000001 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.121550+0000", 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0834e-05 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:52.121566+0000", 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5435999999999999e-05 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.30138.0:1 224.6s0 224:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:25:09.071675+0000", 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.464535714999997, 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0437853619999999, 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:25:59.673 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.30138", 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/607984443", 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.071675+0000", 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.071675+0000", 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.071678+0000", 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6690000000000002e-06 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.071688+0000", 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9820000000000003e-06 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.071709+0000", 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0998000000000001e-05 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.071726+0000", 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7266e-05 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.085537+0000", 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.013811123999999999 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:09.085543+0000", 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5979999999999996e-06 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:10.113569+0000", 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.02802602 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:10.113715+0000", 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014577399999999999 2024-04-26T16:25:59.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:10.114110+0000", 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000395898 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:10.114760+0000", 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065001900000000005 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:10.115415+0000", 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065411400000000002 2024-04-26T16:25:59.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:10.115461+0000", 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5899999999999998e-05 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:25:59.679 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:25:59.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:59.708 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:25:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:25:59.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:59.838 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:59.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 -- 172.21.15.50:0/3755737092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 msgr2=0x7f88c41038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 --2- 172.21.15.50:0/3755737092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c41038f0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7f88ac009990 tx=0x7f88ac009ca0 comp rx=0 tx=0).stop 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 -- 172.21.15.50:0/3755737092 shutdown_connections 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 --2- 172.21.15.50:0/3755737092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88c4103e30 0x7f88c4110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 --2- 172.21.15.50:0/3755737092 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 0x7f88c41098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 --2- 172.21.15.50:0/3755737092 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c41038f0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 -- 172.21.15.50:0/3755737092 >> 172.21.15.50:0/3755737092 conn(0x7f88c407a580 msgr2=0x7f88c407a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 -- 172.21.15.50:0/3755737092 shutdown_connections 2024-04-26T16:25:59.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.846+0000 7f88cab69700 1 -- 172.21.15.50:0/3755737092 wait complete. 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 librados: starting msgr at 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 librados: starting objecter 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 Processor -- start 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- start start 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 librados: setting wanted keys 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 librados: calling monclient init 2024-04-26T16:25:59.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c4199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88c4103e30 0x7f88c4199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 0x7f88c419da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f88c4112ab0 con 0x7f88c41094c0 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f88c4112930 con 0x7f88c4103e30 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f88c4112c30 con 0x7f88c4103470 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c8905700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c4199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c8905700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c4199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33612/0 (socket says 172.21.15.50:33612) 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c8905700 1 -- 172.21.15.50:0/314428794 learned_addr learned my addr 172.21.15.50:0/314428794 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 0x7f88c419da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c3fff700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88c4103e30 0x7f88c4199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 -- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 msgr2=0x7f88c4199150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c4199150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 -- 172.21.15.50:0/314428794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88c4103e30 msgr2=0x7f88c4199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88c4103e30 0x7f88c4199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:25:59.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 -- 172.21.15.50:0/314428794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88c419e160 con 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c8905700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c4199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c9106700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 0x7f88c419da20 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f88b400bfa0 tx=0x7f88b400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c1ffb700 10 client.?.objecter ms_handle_connect 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c1ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4012c90 con 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88c419e450 con 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f88b4012e30 con 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88b4005020 con 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 10 client.30312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88c4075650 con 0x7f88c41094c0 2024-04-26T16:25:59.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 librados: init done 2024-04-26T16:25:59.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c406a110 con 0x7f88c41094c0 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.850+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f88b4004750 con 0x7f88c41094c0 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f88b0071f80 0x7f88b0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(291..291 src has 87..291) ==== 10955+0+0 (secure 0 0 0) 0x7f88b40958c0 con 0x7f88c41094c0 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 10 client.30312.objecter ms_dispatch 0x7f88c4001550 osd_map(291..291 src has 87..291) 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 3 client.30312.objecter handle_osd_map got epochs [291,291] > 0 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 3 client.30312.objecter handle_osd_map decoding full epoch 291 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c8905700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f88b0071f80 0x7f88b0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 20 client.30312.objecter dump_active .. 0 homeless 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c1ffb700 10 client.30312.objecter ms_handle_connect 0x7f88b0071f80 2024-04-26T16:25:59.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.854+0000 7f88c8905700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f88b0071f80 0x7f88b0074440 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f88bc009c10 tx=0x7f88bc00b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:25:59.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:25:59.858+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f88b4060cd0 con 0x7f88c41094c0 2024-04-26T16:25:59.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:59.968 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:25:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:25:59.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:00.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:00.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:00.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.106+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]} v 0) -- 0x7f88c4075910 con 0x7f88c41094c0 2024-04-26T16:26:00.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:00.222 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:00.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:00.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: cluster 2024-04-26T16:25:59.101708+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: cluster 2024-04-26T16:25:59.101708+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.109305+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.109305+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.130116+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.130116+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.143552+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.143552+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.144226+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.144226+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.145212+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.145212+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.145694+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.145694+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.146443+0000 mon.c (mon.2) 481 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.146443+0000 mon.c (mon.2) 481 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.146972+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: audit 2024-04-26T16:25:59.146972+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: cluster 2024-04-26T16:25:59.568806+0000 mgr.y (mgr.24401) 344 : cluster [DBG] pgmap v421: 244 pgs: 48 unknown, 4 active+clean+snaptrim, 192 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[29234]: cluster 2024-04-26T16:25:59.568806+0000 mgr.y (mgr.24401) 344 : cluster [DBG] pgmap v421: 244 pgs: 48 unknown, 4 active+clean+snaptrim, 192 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: cluster 2024-04-26T16:25:59.101708+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: cluster 2024-04-26T16:25:59.101708+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-04-26T16:26:00.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.109305+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.109305+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.130116+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.130116+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.143552+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.143552+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.144226+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.144226+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.145212+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.145212+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.145694+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.145694+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.146443+0000 mon.c (mon.2) 481 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.146443+0000 mon.c (mon.2) 481 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.339 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.146972+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: audit 2024-04-26T16:25:59.146972+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: cluster 2024-04-26T16:25:59.568806+0000 mgr.y (mgr.24401) 344 : cluster [DBG] pgmap v421: 244 pgs: 48 unknown, 4 active+clean+snaptrim, 192 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:00.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:00 smithi050 bash[21463]: cluster 2024-04-26T16:25:59.568806+0000 mgr.y (mgr.24401) 344 : cluster [DBG] pgmap v421: 244 pgs: 48 unknown, 4 active+clean+snaptrim, 192 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:00.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:00.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:00.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: cluster 2024-04-26T16:25:59.101708+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-04-26T16:26:00.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: cluster 2024-04-26T16:25:59.101708+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2024-04-26T16:26:00.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.109305+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.109305+0000 mon.b (mon.1) 194 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.130116+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.130116+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.143552+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.143552+0000 mon.c (mon.2) 479 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.144226+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.144226+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.145212+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.145212+0000 mon.c (mon.2) 480 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.145694+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.145694+0000 mon.a (mon.0) 1987 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:00.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.146443+0000 mon.c (mon.2) 481 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.146443+0000 mon.c (mon.2) 481 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.146972+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: audit 2024-04-26T16:25:59.146972+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:00.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: cluster 2024-04-26T16:25:59.568806+0000 mgr.y (mgr.24401) 344 : cluster [DBG] pgmap v421: 244 pgs: 48 unknown, 4 active+clean+snaptrim, 192 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:00.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:00 smithi190 bash[23868]: cluster 2024-04-26T16:25:59.568806+0000 mgr.y (mgr.24401) 344 : cluster [DBG] pgmap v421: 244 pgs: 48 unknown, 4 active+clean+snaptrim, 192 active+clean; 584 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:00.479 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:00.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:00.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:00.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:00.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:00.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:00.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:00.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:00.873 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:00.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/2968792049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c103e30 msgr2=0x7fa30c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- 172.21.15.50:0/2968792049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c103e30 0x7fa30c110290 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fa30400b0a0 tx=0x7fa30400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/2968792049 shutdown_connections 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- 172.21.15.50:0/2968792049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c103e30 0x7fa30c110290 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- 172.21.15.50:0/2968792049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c1094c0 0x7fa30c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- 172.21.15.50:0/2968792049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa30c103470 0x7fa30c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/2968792049 >> 172.21.15.50:0/2968792049 conn(0x7fa30c07a580 msgr2=0x7fa30c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/2968792049 shutdown_connections 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/2968792049 wait complete. 2024-04-26T16:26:00.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 librados: starting msgr at 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 librados: starting objecter 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 Processor -- start 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- start start 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 librados: setting wanted keys 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 librados: calling monclient init 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 0x7fa30c199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:00.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa30c103e30 0x7fa30c199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:00.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c1094c0 0x7fa30c19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:00.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c112ae0 con 0x7fa30c103470 2024-04-26T16:26:00.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c112960 con 0x7fa30c103e30 2024-04-26T16:26:00.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa30c112c60 con 0x7fa30c1094c0 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 0x7fa30c199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c1094c0 0x7fa30c19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 0x7fa30c199110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34530/0 (socket says 172.21.15.50:34530) 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 -- 172.21.15.50:0/1438503301 learned_addr learned my addr 172.21.15.50:0/1438503301 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30a7fc700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa30c103e30 0x7fa30c199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 -- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c1094c0 msgr2=0x7fa30c19d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c1094c0 0x7fa30c19d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 -- 172.21.15.50:0/1438503301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa30c103e30 msgr2=0x7fa30c199650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa30c103e30 0x7fa30c199650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 -- 172.21.15.50:0/1438503301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c19e120 con 0x7fa30c103470 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa30affd700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 0x7fa30c199110 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00eb30 tx=0x7fa2fc00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3108df700 10 client.?.objecter ms_handle_connect 0x7fa30c103470 2024-04-26T16:26:00.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3108df700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc00cc90 con 0x7fa30c103470 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30c19e410 con 0x7fa30c103470 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 10 client.30318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30c1a5c90 con 0x7fa30c103470 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3118e1700 1 librados: init done 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa2fc009500 con 0x7fa30c103470 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.934+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2fc003b00 con 0x7fa30c103470 2024-04-26T16:26:00.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f8005120 con 0x7fa30c103470 2024-04-26T16:26:00.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa2fc003cc0 con 0x7fa30c103470 2024-04-26T16:26:00.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa2f4071f80 0x7fa2f4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(292..292 src has 87..292) ==== 10456+0+0 (secure 0 0 0) 0x7fa2fc092140 con 0x7fa30c103470 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 10 client.30318.objecter ms_dispatch 0x7fa30c001550 osd_map(292..292 src has 87..292) 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 3 client.30318.objecter handle_osd_map got epochs [292,292] > 0 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 3 client.30318.objecter handle_osd_map decoding full epoch 292 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa30a7fc700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa2f4071f80 0x7fa2f4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 20 client.30318.objecter dump_active .. 0 homeless 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa3108df700 10 client.30318.objecter ms_handle_connect 0x7fa2f4071f80 2024-04-26T16:26:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.938+0000 7fa30a7fc700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa2f4071f80 0x7fa2f4074440 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7fa300005950 tx=0x7fa3000058e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:00.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:00.942+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fa2fc05d740 con 0x7fa30c103470 2024-04-26T16:26:00.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:00.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:01.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.094+0000 7f88c1ffb700 1 -- 172.21.15.50:0/314428794 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]=0 set 2.15 pg_upmap mapping to [2,6,5] v293) ==== 129+0+0 (secure 0 0 0) 0x7f88b4064980 con 0x7f88c41094c0 2024-04-26T16:26:01.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.098+0000 7f88cab69700 10 librados: watch_flush enter 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.098+0000 7f88cab69700 10 librados: watch_flush exit 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.098+0000 7f88cab69700 20 client.30312.objecter shutdown clearing up homeless session... 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.098+0000 7f88cab69700 10 client.30312.objecter successfully canceled tick 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f88b0071f80 msgr2=0x7f88b0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f88b0071f80 0x7f88b0074440 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f88bc009c10 tx=0x7f88bc00b040 comp rx=0 tx=0).stop 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 msgr2=0x7f88c419da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 0x7f88c419da20 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f88b400bfa0 tx=0x7f88b400da00 comp rx=0 tx=0).stop 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 shutdown_connections 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f88b0071f80 0x7f88b0074440 unknown :-1 s=CLOSED pgs=3697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f88c41094c0 0x7f88c419da20 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f88c4103470 0x7f88c4199150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 --2- 172.21.15.50:0/314428794 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f88c4103e30 0x7f88c4199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:01.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 >> 172.21.15.50:0/314428794 conn(0x7f88c407a580 msgr2=0x7f88c4101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:01.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.102+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 shutdown_connections 2024-04-26T16:26:01.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.106+0000 7f88cab69700 1 -- 172.21.15.50:0/314428794 wait complete. 2024-04-26T16:26:01.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.106+0000 7f88cab69700 1 librados: shutdown 2024-04-26T16:26:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:set 2.15 pg_upmap mapping to [2,6,5] 2024-04-26T16:26:01.130 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:01.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:01.206+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fa2f80056e0 con 0x7fa30c103470 2024-04-26T16:26:01.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:01.261 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:01.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.093259+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.093259+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.093348+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.093348+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: cluster 2024-04-26T16:26:00.096940+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: cluster 2024-04-26T16:26:00.096940+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.099002+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.099002+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.102556+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.376 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.102556+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113124+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113124+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113154+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113154+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113368+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113368+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113673+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:00.113673+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: cluster 2024-04-26T16:26:01.079038+0000 mon.a (mon.0) 1996 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: cluster 2024-04-26T16:26:01.079038+0000 mon.a (mon.0) 1996 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.098073+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.098073+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.098187+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]': finished 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.098187+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]': finished 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.098257+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.098257+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.103789+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: audit 2024-04-26T16:26:01.103789+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: cluster 2024-04-26T16:26:01.125774+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-04-26T16:26:01.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:01 smithi190 bash[23868]: cluster 2024-04-26T16:26:01.125774+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.093259+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.093259+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.093348+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.093348+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: cluster 2024-04-26T16:26:00.096940+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: cluster 2024-04-26T16:26:00.096940+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.099002+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.099002+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.102556+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.102556+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113124+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113124+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113154+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.093259+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.093259+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.093348+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.093348+0000 mon.a (mon.0) 1990 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi050-73230-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: cluster 2024-04-26T16:26:00.096940+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-04-26T16:26:01.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: cluster 2024-04-26T16:26:00.096940+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.099002+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.099002+0000 mon.c (mon.2) 482 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.102556+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.102556+0000 mon.a (mon.0) 1992 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113124+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113124+0000 mon.a (mon.0) 1993 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113154+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113154+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113368+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113368+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113673+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:00.113673+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: cluster 2024-04-26T16:26:01.079038+0000 mon.a (mon.0) 1996 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: cluster 2024-04-26T16:26:01.079038+0000 mon.a (mon.0) 1996 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.098073+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.098073+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.098187+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]': finished 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.098187+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]': finished 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.098257+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.098257+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.103789+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: audit 2024-04-26T16:26:01.103789+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: cluster 2024-04-26T16:26:01.125774+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[29234]: cluster 2024-04-26T16:26:01.125774+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-04-26T16:26:01.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113154+0000 mon.c (mon.2) 483 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113368+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]: dispatch 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113368+0000 mon.a (mon.0) 1994 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]: dispatch 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113673+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:00.113673+0000 mon.a (mon.0) 1995 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: cluster 2024-04-26T16:26:01.079038+0000 mon.a (mon.0) 1996 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: cluster 2024-04-26T16:26:01.079038+0000 mon.a (mon.0) 1996 : cluster [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.098073+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.098073+0000 mon.a (mon.0) 1997 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.098187+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]': finished 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.098187+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.50:0/314428794' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.15", "id": [2, 6, 5]}]': finished 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.098257+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.098257+0000 mon.a (mon.0) 1999 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.103789+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: audit 2024-04-26T16:26:01.103789+0000 mon.c (mon.2) 484 : audit [INF] from='client.? 172.21.15.50:0/3004640463' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:01.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: cluster 2024-04-26T16:26:01.125774+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-04-26T16:26:01.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:01 smithi050 bash[21463]: cluster 2024-04-26T16:26:01.125774+0000 mon.a (mon.0) 2000 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2024-04-26T16:26:01.392 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:01.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:01.522 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:01.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:01.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:01.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:01.784 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:01.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:01.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:01.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:02.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:02.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:02.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 -- 172.21.15.50:0/3761180143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 msgr2=0x7fee98109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/3761180143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee98109890 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7fee880098f0 tx=0x7fee88009c00 comp rx=0 tx=0).stop 2024-04-26T16:26:02.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 -- 172.21.15.50:0/3761180143 shutdown_connections 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/3761180143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee981023b0 0x7fee981102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/3761180143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee98109890 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/3761180143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee98101a10 0x7fee98101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 -- 172.21.15.50:0/3761180143 >> 172.21.15.50:0/3761180143 conn(0x7fee98078990 msgr2=0x7fee980ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 -- 172.21.15.50:0/3761180143 shutdown_connections 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 -- 172.21.15.50:0/3761180143 wait complete. 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 librados: starting msgr at 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.078+0000 7fee9f3bd700 1 librados: starting objecter 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 Processor -- start 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- start start 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 librados: setting wanted keys 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 librados: calling monclient init 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee98101a10 0x7fee98199190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee981023b0 0x7fee981996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee9819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee98112a90 con 0x7fee981023b0 2024-04-26T16:26:02.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee98112910 con 0x7fee98101a10 2024-04-26T16:26:02.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fee98112c10 con 0x7fee981094b0 2024-04-26T16:26:02.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee981023b0 0x7fee981996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9d95a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee9819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee981023b0 0x7fee981996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34540/0 (socket says 172.21.15.50:34540) 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9d95a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee9819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33640/0 (socket says 172.21.15.50:33640) 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 -- 172.21.15.50:0/4152116001 learned_addr learned my addr 172.21.15.50:0/4152116001 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee98101a10 0x7fee98199190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 msgr2=0x7fee9819da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee9819da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee98101a10 msgr2=0x7fee98199190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee98101a10 0x7fee98199190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9819e1a0 con 0x7fee981023b0 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee981094b0 0x7fee9819da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee98101a10 0x7fee98199190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee981023b0 0x7fee981996d0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fee9400ec10 tx=0x7fee9400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee8e7fc700 10 client.?.objecter ms_handle_connect 0x7fee981023b0 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee8e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:02.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee9400cd40 con 0x7fee981023b0 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9819e490 con 0x7fee981023b0 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fee94004480 con 0x7fee981023b0 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee94010430 con 0x7fee981023b0 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 10 client.30324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee981a5d10 con 0x7fee981023b0 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 librados: init done 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fee98104d30 con 0x7fee981023b0 2024-04-26T16:26:02.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.082+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fee94003ef0 con 0x7fee981023b0 2024-04-26T16:26:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee84071f80 0x7fee84074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(293..293 src has 87..293) ==== 9936+0+0 (secure 0 0 0) 0x7fee94092310 con 0x7fee981023b0 2024-04-26T16:26:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 osd_map(293..293 src has 87..293) 2024-04-26T16:26:02.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 3 client.30324.objecter handle_osd_map got epochs [293,293] > 0 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 3 client.30324.objecter handle_osd_map decoding full epoch 293 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee84071f80 0x7fee84074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 20 client.30324.objecter dump_active .. 0 homeless 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v293) ==== 54+0+16 (secure 0 0 0) 0x7fee94092700 con 0x7fee981023b0 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 1 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee98064030 0x7fee98064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 1 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 0 1 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 4 2024-04-26T16:26:02.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 10 client.30324.objecter _send_command 1 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee9819a4f0 con 0x7fee98064030 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee84071f80 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee84071f80 0x7fee84074440 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7fee88009810 tx=0x7fee88019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee98064030 0x7fee98064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee98064030 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee98064030 0x7fee98064410 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee9819a4f0 con 0x7fee98064030 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 1: 0 ) 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 1 = system:0 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 4 2024-04-26T16:26:02.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.086+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 0 1 2024-04-26T16:26:02.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.098+0000 7fa3108df700 1 -- 172.21.15.50:0/1438503301 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v294) ==== 91+0+0 (secure 0 0 0) 0x7fa2fc0613f0 con 0x7fa30c103470 2024-04-26T16:26:02.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 10 librados: watch_flush enter 2024-04-26T16:26:02.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 10 librados: watch_flush exit 2024-04-26T16:26:02.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 20 client.30318.objecter shutdown clearing up homeless session... 2024-04-26T16:26:02.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 10 client.30318.objecter successfully canceled tick 2024-04-26T16:26:02.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa2f4071f80 msgr2=0x7fa2f4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa2f4071f80 0x7fa2f4074440 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7fa300005950 tx=0x7fa3000058e0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 msgr2=0x7fa30c199110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 0x7fa30c199110 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc00eb30 tx=0x7fa2fc00ee40 comp rx=0 tx=0).stop 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 shutdown_connections 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa2f4071f80 0x7fa2f4074440 unknown :-1 s=CLOSED pgs=3698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa30c1094c0 0x7fa30c19d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa30c103470 0x7fa30c199110 unknown :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 --2- 172.21.15.50:0/1438503301 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa30c103e30 0x7fa30c199650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 >> 172.21.15.50:0/1438503301 conn(0x7fa30c07a580 msgr2=0x7fa30c10a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.102+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 shutdown_connections 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.106+0000 7fa3118e1700 1 -- 172.21.15.50:0/1438503301 wait complete. 2024-04-26T16:26:02.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.106+0000 7fa3118e1700 1 librados: shutdown 2024-04-26T16:26:02.111 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 2 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 2 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9806a110 osd=0 3 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 0 2 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 4 2024-04-26T16:26:02.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 10 client.30324.objecter _send_command 2 2024-04-26T16:26:02.115 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.118+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee98063ab0 con 0x7fee98064030 2024-04-26T16:26:02.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee98063ab0 con 0x7fee98064030 2024-04-26T16:26:02.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 4 2024-04-26T16:26:02.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 0 2 2024-04-26T16:26:02.119 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9810af10 0x7fee981a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 1 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 4 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 10 client.30324.objecter _send_command 3 2024-04-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fee9802e600 con 0x7fee9810af10 2024-04-26T16:26:02.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.122+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9810af10 0x7fee981a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9810af10 0x7fee981a9fc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee9810af10 2024-04-26T16:26:02.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee9802e600 con 0x7fee9810af10 2024-04-26T16:26:02.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 3: 0 ) 2024-04-26T16:26:02.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 3 = system:0 2024-04-26T16:26:02.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 4 2024-04-26T16:26:02.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.126+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 1 3 2024-04-26T16:26:02.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:02.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 4 2024-04-26T16:26:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 4 2024-04-26T16:26:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9802e490 osd=1 3 2024-04-26T16:26:02.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 1 4 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 4 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 10 client.30324.objecter _send_command 4 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee981aa500 con 0x7fee9810af10 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee981aa500 con 0x7fee9810af10 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 4 2024-04-26T16:26:02.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 1 4 2024-04-26T16:26:02.155 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:26:02.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 5 2024-04-26T16:26:02.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee9810a5f0 0x7fee981ad840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 5 2024-04-26T16:26:02.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 2 5 2024-04-26T16:26:02.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 4 2024-04-26T16:26:02.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 10 client.30324.objecter _send_command 5 2024-04-26T16:26:02.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fee9802e080 con 0x7fee9810a5f0 2024-04-26T16:26:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:01.127354+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:01.127354+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:01.127923+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:01.127923+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:01.209856+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:26:02.157 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:01.209856+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: cluster 2024-04-26T16:26:01.571045+0000 mgr.y (mgr.24401) 345 : cluster [DBG] pgmap v424: 228 pgs: 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: cluster 2024-04-26T16:26:01.571045+0000 mgr.y (mgr.24401) 345 : cluster [DBG] pgmap v424: 228 pgs: 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.085735+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.50:0/4152116001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.085735+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.50:0/4152116001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.098738+0000 mon.a (mon.0) 2005 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.098738+0000 mon.a (mon.0) 2005 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100486+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100486+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100598+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100598+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100673+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100673+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100748+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:26:02.158 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: audit 2024-04-26T16:26:02.100748+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.105603+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.105603+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:01.127354+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:01.127354+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:01.127923+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:01.127923+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:01.209856+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:01.209856+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: cluster 2024-04-26T16:26:01.571045+0000 mgr.y (mgr.24401) 345 : cluster [DBG] pgmap v424: 228 pgs: 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: cluster 2024-04-26T16:26:01.571045+0000 mgr.y (mgr.24401) 345 : cluster [DBG] pgmap v424: 228 pgs: 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:26:02.159 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.085735+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.50:0/4152116001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.085735+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.50:0/4152116001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.098738+0000 mon.a (mon.0) 2005 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.098738+0000 mon.a (mon.0) 2005 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100486+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100486+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100598+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100598+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100673+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100673+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:02.160 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100748+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:26:02.161 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: audit 2024-04-26T16:26:02.100748+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:26:02.161 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.105603+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-04-26T16:26:02.161 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:02 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.105603+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-04-26T16:26:02.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.154+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee9810a5f0 0x7fee981ad840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee9810a5f0 2024-04-26T16:26:02.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee9810a5f0 0x7fee981ad840 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee9802e080 con 0x7fee9810a5f0 2024-04-26T16:26:02.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 5: 0 ) 2024-04-26T16:26:02.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 5 = system:0 2024-04-26T16:26:02.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 4 2024-04-26T16:26:02.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.158+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 2 5 2024-04-26T16:26:02.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:02.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 6 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 6 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9810a480 osd=2 3 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 2 6 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 4 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 10 client.30324.objecter _send_command 6 2024-04-26T16:26:02.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee9802e080 con 0x7fee9810a5f0 2024-04-26T16:26:02.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee9802e080 con 0x7fee9810a5f0 2024-04-26T16:26:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 4 2024-04-26T16:26:02.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 2 6 2024-04-26T16:26:02.175 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 7 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee981ab390 0x7fee981b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 7 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 3 7 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 4 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 10 client.30324.objecter _send_command 7 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fee9802d800 con 0x7fee981ab390 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee981ab390 0x7fee981b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee981ab390 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee981ab390 0x7fee981b0f10 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee9802d800 con 0x7fee981ab390 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 7: 0 ) 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 7 = system:0 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 4 2024-04-26T16:26:02.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.178+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 3 7 2024-04-26T16:26:02.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 8 2024-04-26T16:26:02.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 8 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee9802e080 osd=3 3 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 3 8 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 4 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 10 client.30324.objecter _send_command 8 2024-04-26T16:26:02.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee981aa900 con 0x7fee981ab390 2024-04-26T16:26:02.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee981aa900 con 0x7fee981ab390 2024-04-26T16:26:02.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 4 2024-04-26T16:26:02.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 3 8 2024-04-26T16:26:02.197 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:26:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 9 2024-04-26T16:26:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee981aea60 0x7fee981b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 9 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 4 9 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 4 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 10 client.30324.objecter _send_command 9 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fee98104b30 con 0x7fee981aea60 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.198+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee981aea60 0x7fee981b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee981aea60 2024-04-26T16:26:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee981aea60 0x7fee981b69e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee98104b30 con 0x7fee981aea60 2024-04-26T16:26:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 9: 0 ) 2024-04-26T16:26:02.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 9 = system:0 2024-04-26T16:26:02.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 4 2024-04-26T16:26:02.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.202+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 4 9 2024-04-26T16:26:02.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 10 2024-04-26T16:26:02.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 10 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981aa6b0 osd=4 3 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 4 10 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 4 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 10 client.30324.objecter _send_command 10 2024-04-26T16:26:02.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.218+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee98104b30 con 0x7fee981aea60 2024-04-26T16:26:02.220 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee98104b30 con 0x7fee981aea60 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 4 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 4 10 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 11 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee981b7430 0x7fee981b9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 11 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 5 11 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 4 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 10 client.30324.objecter _send_command 11 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fee981bc680 con 0x7fee981b7430 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee981b7430 0x7fee981b9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee9c958700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee981b7430 0x7fee981b9820 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee981b7430 2024-04-26T16:26:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee981bc680 con 0x7fee981b7430 2024-04-26T16:26:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 11: 0 ) 2024-04-26T16:26:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 11 = system:0 2024-04-26T16:26:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 4 2024-04-26T16:26:02.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.226+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 5 11 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 12 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 12 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981aab80 osd=5 3 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 5 12 2024-04-26T16:26:02.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 4 2024-04-26T16:26:02.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 10 client.30324.objecter _send_command 12 2024-04-26T16:26:02.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.246+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee981adfd0 con 0x7fee981b7430 2024-04-26T16:26:02.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee981adfd0 con 0x7fee981b7430 2024-04-26T16:26:02.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 4 2024-04-26T16:26:02.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 5 12 2024-04-26T16:26:02.246 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:26:02.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 13 2024-04-26T16:26:02.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fee981b2130 0x7fee981baee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 13 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 6 13 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 4 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 10 client.30324.objecter _send_command 13 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fee981adef0 con 0x7fee981b2130 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fee981b2130 0x7fee981baee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee9d95a700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fee981b2130 0x7fee981baee0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.250+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee981b2130 2024-04-26T16:26:02.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.254+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee981adef0 con 0x7fee981b2130 2024-04-26T16:26:02.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.254+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 13: 0 ) 2024-04-26T16:26:02.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.254+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 13 = system:0 2024-04-26T16:26:02.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.254+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 4 2024-04-26T16:26:02.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.254+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 6 13 2024-04-26T16:26:02.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:02.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 14 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 14 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981add80 osd=6 3 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 6 14 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 4 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 10 client.30324.objecter _send_command 14 2024-04-26T16:26:02.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.270+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee981adef0 con 0x7fee981b2130 2024-04-26T16:26:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee981adef0 con 0x7fee981b2130 2024-04-26T16:26:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 4 2024-04-26T16:26:02.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 6 14 2024-04-26T16:26:02.271 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 15 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee981bb930 0x7fee981bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 15 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 7 15 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 4 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 10 client.30324.objecter _send_command 15 2024-04-26T16:26:02.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fee981bc470 con 0x7fee981bb930 2024-04-26T16:26:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.274+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee981bb930 0x7fee981bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee8e7fc700 10 client.30324.objecter ms_handle_connect 0x7fee981bb930 2024-04-26T16:26:02.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee9d159700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee981bb930 0x7fee981bbd90 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee981bc470 con 0x7fee981bb930 2024-04-26T16:26:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 15: 0 ) 2024-04-26T16:26:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 15 = system:0 2024-04-26T16:26:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 4 2024-04-26T16:26:02.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.278+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 7 15 2024-04-26T16:26:02.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 10 client.30324.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign -1 16 2024-04-26T16:26:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 20 client.30324.objecter _get_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_remove -1 16 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 20 client.30324.objecter get_session s=0x7fee981adef0 osd=7 3 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 15 client.30324.objecter _session_command_op_assign 7 16 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 4 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 10 client.30324.objecter _send_command 16 2024-04-26T16:26:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.298+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee981bc650 con 0x7fee981bb930 2024-04-26T16:26:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee8e7fc700 1 -- 172.21.15.50:0/4152116001 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee981bc650 con 0x7fee981bb930 2024-04-26T16:26:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee8e7fc700 10 client.30324.objecter ms_dispatch 0x7fee98001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee8e7fc700 10 client.30324.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee8e7fc700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 4 2024-04-26T16:26:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee8e7fc700 15 client.30324.objecter _session_command_op_remove 7 16 2024-04-26T16:26:02.299 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 librados: watch_flush enter 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 librados: watch_flush exit 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.0 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee98064030 msgr2=0x7fee98064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee98064030 0x7fee98064410 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9806a110 osd=0 1 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.1 2024-04-26T16:26:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9810af10 msgr2=0x7fee981a9fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9810af10 0x7fee981a9fc0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e490 osd=1 1 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.2 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee9810a5f0 msgr2=0x7fee981ad840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee9810a5f0 0x7fee981ad840 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9810a480 osd=2 1 2024-04-26T16:26:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.3 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee981ab390 msgr2=0x7fee981b0f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee981ab390 0x7fee981b0f10 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9d95a700 1 -- 172.21.15.50:0/4152116001 reap_dead start 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee9802e080 osd=3 1 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.4 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee981aea60 msgr2=0x7fee981b69e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee981aea60 0x7fee981b69e0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aa6b0 osd=4 1 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.5 2024-04-26T16:26:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee981b7430 msgr2=0x7fee981b9820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee981b7430 0x7fee981b9820 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981aab80 osd=5 1 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.6 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fee981b2130 msgr2=0x7fee981baee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fee981b2130 0x7fee981baee0 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981add80 osd=6 1 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter close_session for osd.7 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee981bb930 msgr2=0x7fee981bbd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee981bb930 0x7fee981bbd90 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter put_session s=0x7fee981adef0 osd=7 1 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 20 client.30324.objecter shutdown clearing up homeless session... 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 10 client.30324.objecter successfully canceled tick 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee84071f80 msgr2=0x7fee84074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee84071f80 0x7fee84074440 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7fee88009810 tx=0x7fee88019040 comp rx=0 tx=0).stop 2024-04-26T16:26:02.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee981023b0 msgr2=0x7fee981996d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 --2- 172.21.15.50:0/4152116001 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee981023b0 0x7fee981996d0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7fee9400ec10 tx=0x7fee9400ef20 comp rx=0 tx=0).stop 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9d95a700 1 -- 172.21.15.50:0/4152116001 reap_dead start 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9d95a700 1 -- 172.21.15.50:0/4152116001 reap_dead start 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 shutdown_connections 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 >> 172.21.15.50:0/4152116001 conn(0x7fee98078990 msgr2=0x7fee98105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 shutdown_connections 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.302+0000 7fee9f3bd700 1 -- 172.21.15.50:0/4152116001 wait complete. 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:02.306+0000 7fee9f3bd700 1 librados: shutdown 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:26:02.305 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:26:02.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:26:02.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:26:02.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:26:02.306 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:26:02.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:02.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:01.127354+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:02.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:01.127354+0000 mon.a (mon.0) 2001 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]: dispatch 2024-04-26T16:26:02.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:01.127923+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:02.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:01.127923+0000 mon.a (mon.0) 2002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]: dispatch 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:01.209856+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:01.209856+0000 mon.a (mon.0) 2003 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: cluster 2024-04-26T16:26:01.571045+0000 mgr.y (mgr.24401) 345 : cluster [DBG] pgmap v424: 228 pgs: 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: cluster 2024-04-26T16:26:01.571045+0000 mgr.y (mgr.24401) 345 : cluster [DBG] pgmap v424: 228 pgs: 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.085735+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.50:0/4152116001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.085735+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.50:0/4152116001' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.098738+0000 mon.a (mon.0) 2005 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.098738+0000 mon.a (mon.0) 2005 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100486+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100486+0000 mon.a (mon.0) 2006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi050-73230-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:02.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100598+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100598+0000 mon.a (mon.0) 2007 : audit [INF] from='client.? 172.21.15.50:0/3027781296' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-46"}]': finished 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100673+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100673+0000 mon.a (mon.0) 2008 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi050-73193-37"}]': finished 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100748+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: audit 2024-04-26T16:26:02.100748+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? 172.21.15.50:0/1438503301' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.105603+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-04-26T16:26:02.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:02 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.105603+0000 mon.a (mon.0) 2010 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2024-04-26T16:26:02.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:02.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-26T16:26:02.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:02.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:02.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:02.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:02.713 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:02.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:02.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:02.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:02.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:03.052 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:03.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:03.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.314 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:03.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:03.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.128834+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:26:03.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.128834+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:26:03.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.142814+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.142814+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.144446+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.144446+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.144870+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.144870+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.148665+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.148665+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.149032+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.149032+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.149657+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.149657+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.128834+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.128834+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.142814+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.142814+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.144446+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.144446+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.144870+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.144870+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.148665+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.148665+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.149032+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.149032+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.149657+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.149657+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.150191+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.150191+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.150568+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:02.150568+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.152342+0000 osd.3 (osd.3) 207 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.152342+0000 osd.3 (osd.3) 207 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.153548+0000 osd.3 (osd.3) 208 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.153548+0000 osd.3 (osd.3) 208 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.464896+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.464896+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.467009+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:02.467009+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.103152+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.103152+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.103341+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:03.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.103341+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.103611+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.103611+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.109682+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.109682+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.111127+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.111127+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.111820+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.111820+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.115430+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.115430+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.116904+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[29234]: audit 2024-04-26T16:26:03.116904+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.150191+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.150191+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.150568+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:02.150568+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.152342+0000 osd.3 (osd.3) 207 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.152342+0000 osd.3 (osd.3) 207 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.153548+0000 osd.3 (osd.3) 208 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.153548+0000 osd.3 (osd.3) 208 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.464896+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.464896+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.467009+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:02.467009+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-04-26T16:26:03.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.103152+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.103152+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.103341+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.103341+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.103611+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.103611+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.109682+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.109682+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.111127+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.111127+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.111820+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.111820+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.115430+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.115430+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.116904+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:03 smithi050 bash[21463]: audit 2024-04-26T16:26:03.116904+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.447 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:03.494 INFO:tasks.workunit.client.0.smithi050.stdout:ntry=7 expected=7 2024-04-26T16:26:03.494 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-04-26T16:26:03.494 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:02547ec2:::1:head expected=12:02547ec2:::1:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=1 expected=1 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:863748b0:::15:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:863748b0:::15:head expected=12:863748b0:::15:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=15 expected=15 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:6cac518f:::0:head expected=12:6cac518f:::0:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=0 expected=0 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:e960b815:::9:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:e960b815:::9:head expected=12:e960b815:::9:head 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=9 expected=9 2024-04-26T16:26:03.495 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:bd63b0f1:::8:head expected=12:bd63b0f1:::8:head 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=8 expected=8 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:f905c69b:::2:head expected=12:f905c69b:::2:head 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=2 expected=2 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:89d3ae78:::11:head expected=12:89d3ae78:::11:head 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=11 expected=11 2024-04-26T16:26:03.496 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : cursor()=12:52ea6a34:::10:head expected=12:52ea6a34:::10:head 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: : entry=10 expected=10 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (246 ms) 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (75310 ms) 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-04-26T16:26:03.497 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: split 0/5 -> MIN 12:33333333::::head 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: split 1/5 -> 12:33333333::::head 12:66666666::::head 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: split 2/5 -> 12:66666666::::head 12:99999999::::head 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: split 3/5 -> 12:99999999::::head 12:cccccccc::::head 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: split 4/5 -> 12:cccccccc::::head MAX 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (73007 ms) 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] 7 tests from LibRadosList (149029 ms total) 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1320 ms) 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset foo1,foo2,foo3 2024-04-26T16:26:03.498 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo1 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo2 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo3 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset foo1,foo4,foo5 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo4 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo5 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo1 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset foo6,foo7 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo7 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: foo6 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns1:foo4 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns1:foo5 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns2:foo7 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns2:foo6 2024-04-26T16:26:03.499 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: ns1:foo1 2024-04-26T16:26:03.500 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: :foo1 2024-04-26T16:26:03.500 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: :foo2 2024-04-26T16:26:03.500 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: :foo3 2024-04-26T16:26:03.500 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (73 ms) 2024-04-26T16:26:03.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:03.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.128834+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:26:03.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.128834+0000 mon.a (mon.0) 2011 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.142814+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.142814+0000 mon.a (mon.0) 2012 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.144446+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.144446+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.144870+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.144870+0000 mon.b (mon.1) 195 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.148665+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.148665+0000 mon.a (mon.0) 2014 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.149032+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.149032+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.149657+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.149657+0000 mon.a (mon.0) 2016 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.150191+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.150191+0000 mon.a (mon.0) 2017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:03.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.150568+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:02.150568+0000 mon.a (mon.0) 2018 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:03.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.152342+0000 osd.3 (osd.3) 207 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.152342+0000 osd.3 (osd.3) 207 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.153548+0000 osd.3 (osd.3) 208 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.153548+0000 osd.3 (osd.3) 208 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.464896+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.464896+0000 osd.1 (osd.1) 175 : cluster [DBG] 2.a scrub starts 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.467009+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:02.467009+0000 osd.1 (osd.1) 176 : cluster [DBG] 2.a scrub ok 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.103152+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.103152+0000 mon.a (mon.0) 2019 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi050-73193-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.103341+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.103341+0000 mon.a (mon.0) 2020 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.103611+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.103611+0000 mon.a (mon.0) 2021 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi050-73201-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.109682+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.109682+0000 mon.a (mon.0) 2022 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.111127+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.111127+0000 mon.a (mon.0) 2023 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.111820+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.111820+0000 mon.a (mon.0) 2024 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.115430+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.115430+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.116904+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:03 smithi190 bash[23868]: audit 2024-04-26T16:26:03.116904+0000 mon.b (mon.1) 196 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:03.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:03.678 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:03.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:03.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:03.812 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:03.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:03.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:04.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:04.106 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:04.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.504657+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.504657+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.507015+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.507015+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.571723+0000 mgr.y (mgr.24401) 346 : cluster [DBG] pgmap v427: 236 pgs: 8 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.571723+0000 mgr.y (mgr.24401) 346 : cluster [DBG] pgmap v427: 236 pgs: 8 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: audit 2024-04-26T16:26:04.106765+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21463]: audit 2024-04-26T16:26:04.106765+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.504657+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:26:04.223 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.504657+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:26:04.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.507015+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:26:04.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.507015+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:26:04.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.571723+0000 mgr.y (mgr.24401) 346 : cluster [DBG] pgmap v427: 236 pgs: 8 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-04-26T16:26:04.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.571723+0000 mgr.y (mgr.24401) 346 : cluster [DBG] pgmap v427: 236 pgs: 8 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-04-26T16:26:04.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: audit 2024-04-26T16:26:04.106765+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:04.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[29234]: audit 2024-04-26T16:26:04.106765+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:04.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:04.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:04.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.504657+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:26:04.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.504657+0000 osd.1 (osd.1) 177 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:26:04.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.507015+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:26:04.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.507015+0000 osd.1 (osd.1) 178 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:26:04.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.571723+0000 mgr.y (mgr.24401) 346 : cluster [DBG] pgmap v427: 236 pgs: 8 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-04-26T16:26:04.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.571723+0000 mgr.y (mgr.24401) 346 : cluster [DBG] pgmap v427: 236 pgs: 8 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 KiB/s wr, 1 op/s 2024-04-26T16:26:04.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: audit 2024-04-26T16:26:04.106765+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:04.363 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:04 smithi190 bash[23868]: audit 2024-04-26T16:26:04.106765+0000 mon.a (mon.0) 2026 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:04.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:04.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:04.490 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:26:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:26:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:26:04.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:04.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:04.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:04.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:04.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:04.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:04.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:04.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:04.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:04.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:05.016 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:05.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:05.148 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:05.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:05.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.406568+0000 osd.6 (osd.6) 141 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:26:05.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.406568+0000 osd.6 (osd.6) 141 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.412201+0000 osd.6 (osd.6) 142 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.412201+0000 osd.6 (osd.6) 142 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.812355+0000 osd.4 (osd.4) 165 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.812355+0000 osd.4 (osd.4) 165 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.816085+0000 osd.4 (osd.4) 166 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:03.816085+0000 osd.4 (osd.4) 166 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:04.063283+0000 osd.5 (osd.5) 187 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:04.063283+0000 osd.5 (osd.5) 187 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:04.067072+0000 osd.5 (osd.5) 188 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:04.067072+0000 osd.5 (osd.5) 188 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.115916+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.115916+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:04.144172+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:04.144172+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-04-26T16:26:05.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.145507+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.145507+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.145731+0000 mon.c (mon.2) 485 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.145731+0000 mon.c (mon.2) 485 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.154395+0000 mon.a (mon.0) 2029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.154395+0000 mon.a (mon.0) 2029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.232505+0000 mon.c (mon.2) 486 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.232505+0000 mon.c (mon.2) 486 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.909875+0000 mon.c (mon.2) 487 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.909875+0000 mon.c (mon.2) 487 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.910875+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.910875+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.916514+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:04.916514+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:05.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.107677+0000 mon.a (mon.0) 2031 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.107677+0000 mon.a (mon.0) 2031 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110107+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110107+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110358+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110358+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110567+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110567+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110729+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.110729+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.120464+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.120464+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.126582+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.126582+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.129698+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:05 smithi190 bash[23868]: audit 2024-04-26T16:26:05.129698+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:05.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:05.402 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:05.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:05.527 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:05.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.406568+0000 osd.6 (osd.6) 141 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:26:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.406568+0000 osd.6 (osd.6) 141 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:26:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.412201+0000 osd.6 (osd.6) 142 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.412201+0000 osd.6 (osd.6) 142 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.812355+0000 osd.4 (osd.4) 165 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.812355+0000 osd.4 (osd.4) 165 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.816085+0000 osd.4 (osd.4) 166 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:03.816085+0000 osd.4 (osd.4) 166 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:04.063283+0000 osd.5 (osd.5) 187 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:04.063283+0000 osd.5 (osd.5) 187 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:04.067072+0000 osd.5 (osd.5) 188 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:04.067072+0000 osd.5 (osd.5) 188 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.115916+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.115916+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:04.144172+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-04-26T16:26:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:04.144172+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-04-26T16:26:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.145507+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.145507+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.145731+0000 mon.c (mon.2) 485 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.145731+0000 mon.c (mon.2) 485 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.154395+0000 mon.a (mon.0) 2029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.154395+0000 mon.a (mon.0) 2029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.406568+0000 osd.6 (osd.6) 141 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.406568+0000 osd.6 (osd.6) 141 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.412201+0000 osd.6 (osd.6) 142 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.412201+0000 osd.6 (osd.6) 142 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.812355+0000 osd.4 (osd.4) 165 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.812355+0000 osd.4 (osd.4) 165 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.816085+0000 osd.4 (osd.4) 166 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:03.816085+0000 osd.4 (osd.4) 166 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:04.063283+0000 osd.5 (osd.5) 187 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:04.063283+0000 osd.5 (osd.5) 187 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:04.067072+0000 osd.5 (osd.5) 188 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:26:05.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:04.067072+0000 osd.5 (osd.5) 188 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.115916+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.115916+0000 mon.b (mon.1) 197 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:04.144172+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:04.144172+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.145507+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.145507+0000 mon.a (mon.0) 2028 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.145731+0000 mon.c (mon.2) 485 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.145731+0000 mon.c (mon.2) 485 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.154395+0000 mon.a (mon.0) 2029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.154395+0000 mon.a (mon.0) 2029 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.232505+0000 mon.c (mon.2) 486 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.232505+0000 mon.c (mon.2) 486 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.909875+0000 mon.c (mon.2) 487 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.909875+0000 mon.c (mon.2) 487 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.910875+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.910875+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.916514+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:04.916514+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:05.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.107677+0000 mon.a (mon.0) 2031 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:05.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.107677+0000 mon.a (mon.0) 2031 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:05.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110107+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110107+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110358+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110358+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110567+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110567+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110729+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.110729+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.120464+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.120464+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.126582+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.126582+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.129698+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.638 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[21463]: audit 2024-04-26T16:26:05.129698+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.232505+0000 mon.c (mon.2) 486 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.232505+0000 mon.c (mon.2) 486 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.909875+0000 mon.c (mon.2) 487 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.909875+0000 mon.c (mon.2) 487 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.910875+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.910875+0000 mon.c (mon.2) 488 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.916514+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:05.639 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:04.916514+0000 mon.a (mon.0) 2030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:05.640 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.107677+0000 mon.a (mon.0) 2031 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:05.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.107677+0000 mon.a (mon.0) 2031 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:05.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110107+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:05.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110107+0000 mon.a (mon.0) 2032 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi050-73193-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:05.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110358+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:05.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110358+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi050-73201-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:05.641 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110567+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]': finished 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110567+0000 mon.a (mon.0) 2034 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-29", "mode": "writeback"}]': finished 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110729+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.110729+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.120464+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.120464+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.126582+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.126582+0000 mon.c (mon.2) 489 : audit [INF] from='client.? 172.21.15.50:0/1694205834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.129698+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.642 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:05 smithi050 bash[29234]: audit 2024-04-26T16:26:05.129698+0000 mon.a (mon.0) 2037 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]: dispatch 2024-04-26T16:26:05.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:05.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:05.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:05.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:05.802 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:05.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:05.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:06.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:06.062 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:06.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:06.187 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:06.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:06.301 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.052573+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:26:06.301 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.052573+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.055771+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.055771+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.408144+0000 osd.0 (osd.0) 199 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.408144+0000 osd.0 (osd.0) 199 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.410090+0000 osd.0 (osd.0) 200 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.410090+0000 osd.0 (osd.0) 200 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.572678+0000 mgr.y (mgr.24401) 347 : cluster [DBG] pgmap v430: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:05.572678+0000 mgr.y (mgr.24401) 347 : cluster [DBG] pgmap v430: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.081566+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.081566+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: audit 2024-04-26T16:26:06.088009+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: audit 2024-04-26T16:26:06.088009+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.096176+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-04-26T16:26:06.302 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.096176+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-04-26T16:26:06.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.052573+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.052573+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.055771+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.055771+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.408144+0000 osd.0 (osd.0) 199 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.408144+0000 osd.0 (osd.0) 199 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.410090+0000 osd.0 (osd.0) 200 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.410090+0000 osd.0 (osd.0) 200 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.572678+0000 mgr.y (mgr.24401) 347 : cluster [DBG] pgmap v430: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:05.572678+0000 mgr.y (mgr.24401) 347 : cluster [DBG] pgmap v430: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.081566+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:06.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.081566+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: audit 2024-04-26T16:26:06.088009+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: audit 2024-04-26T16:26:06.088009+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.096176+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.096176+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.052573+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.052573+0000 osd.7 (osd.7) 203 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.055771+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.055771+0000 osd.7 (osd.7) 204 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.408144+0000 osd.0 (osd.0) 199 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.408144+0000 osd.0 (osd.0) 199 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.410090+0000 osd.0 (osd.0) 200 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:26:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.410090+0000 osd.0 (osd.0) 200 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.572678+0000 mgr.y (mgr.24401) 347 : cluster [DBG] pgmap v430: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:05.572678+0000 mgr.y (mgr.24401) 347 : cluster [DBG] pgmap v430: 244 pgs: 16 unknown, 1 peering, 227 active+clean; 4.6 MiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.081566+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.081566+0000 mon.a (mon.0) 2038 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: audit 2024-04-26T16:26:06.088009+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: audit 2024-04-26T16:26:06.088009+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi050-73230-2"}]': finished 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.096176+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-04-26T16:26:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:06 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.096176+0000 mon.a (mon.0) 2040 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2024-04-26T16:26:06.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:06.442 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:06.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:06.572 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:06.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:06.701 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:06.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:06.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:06.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:06.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:06.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:06.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:07.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:07.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:26:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:26:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:07.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:07.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:07.308 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:26:07.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.370051+0000 osd.0 (osd.0) 201 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.370051+0000 osd.0 (osd.0) 201 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.373311+0000 osd.0 (osd.0) 202 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.373311+0000 osd.0 (osd.0) 202 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.530745+0000 osd.1 (osd.1) 179 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.530745+0000 osd.1 (osd.1) 179 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.536102+0000 osd.1 (osd.1) 180 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.536102+0000 osd.1 (osd.1) 180 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:26:07.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.102307+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.102307+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.108561+0000 mon.c (mon.2) 490 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.108561+0000 mon.c (mon.2) 490 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.109356+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.109356+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.120291+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.120291+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.120575+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[21463]: audit 2024-04-26T16:26:07.120575+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.370051+0000 osd.0 (osd.0) 201 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.370051+0000 osd.0 (osd.0) 201 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:26:07.356 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.373311+0000 osd.0 (osd.0) 202 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.373311+0000 osd.0 (osd.0) 202 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.530745+0000 osd.1 (osd.1) 179 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.530745+0000 osd.1 (osd.1) 179 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.536102+0000 osd.1 (osd.1) 180 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.536102+0000 osd.1 (osd.1) 180 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.102307+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.102307+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.108561+0000 mon.c (mon.2) 490 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.108561+0000 mon.c (mon.2) 490 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.109356+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.109356+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.120291+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:07.357 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.120291+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:07.358 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.120575+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:07.358 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:07 smithi050 bash[29234]: audit 2024-04-26T16:26:07.120575+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:07.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:26:07.369 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:07.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:26:07.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:07.484 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.370051+0000 osd.0 (osd.0) 201 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:26:07.484 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.370051+0000 osd.0 (osd.0) 201 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:26:07.484 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.373311+0000 osd.0 (osd.0) 202 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:26:07.484 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.373311+0000 osd.0 (osd.0) 202 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.530745+0000 osd.1 (osd.1) 179 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.530745+0000 osd.1 (osd.1) 179 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.536102+0000 osd.1 (osd.1) 180 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.536102+0000 osd.1 (osd.1) 180 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.102307+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.102307+0000 mon.a (mon.0) 2041 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.108561+0000 mon.c (mon.2) 490 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.108561+0000 mon.c (mon.2) 490 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.109356+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.109356+0000 mon.a (mon.0) 2042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.120291+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:07.485 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.120291+0000 mon.a (mon.0) 2043 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:07.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.120575+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:07.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:07 smithi190 bash[23868]: audit 2024-04-26T16:26:07.120575+0000 mon.a (mon.0) 2044 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:07.499 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:07.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:07.603 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:07.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:07.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:07.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:07.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:07.883 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:07.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:07.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:07.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:08.013 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:08.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:08.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:08.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:08.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.451744+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:26:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.451744+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:26:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.452728+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:26:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.452728+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:26:08.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:06.747023+0000 mgr.y (mgr.24401) 348 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:06.747023+0000 mgr.y (mgr.24401) 348 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.826114+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.826114+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.828247+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:06.828247+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.152348+0000 osd.7 (osd.7) 205 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.152348+0000 osd.7 (osd.7) 205 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.156205+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.156205+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.573193+0000 mgr.y (mgr.24401) 349 : cluster [DBG] pgmap v433: 260 pgs: 32 unknown, 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.573193+0000 mgr.y (mgr.24401) 349 : cluster [DBG] pgmap v433: 260 pgs: 32 unknown, 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.095532+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.095532+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:08.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.095815+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.095815+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.096002+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.096002+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.106235+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.106235+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.112433+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.112433+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.112804+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.112804+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.162857+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:08.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:08 smithi190 bash[23868]: audit 2024-04-26T16:26:08.162857+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:08.278 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:08.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:08.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:08.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.451744+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:26:08.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.451744+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.452728+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.452728+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:06.747023+0000 mgr.y (mgr.24401) 348 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:06.747023+0000 mgr.y (mgr.24401) 348 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.826114+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.826114+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:26:08.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.828247+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:06.828247+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.152348+0000 osd.7 (osd.7) 205 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.152348+0000 osd.7 (osd.7) 205 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.156205+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.156205+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.573193+0000 mgr.y (mgr.24401) 349 : cluster [DBG] pgmap v433: 260 pgs: 32 unknown, 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.573193+0000 mgr.y (mgr.24401) 349 : cluster [DBG] pgmap v433: 260 pgs: 32 unknown, 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.095532+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.095532+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.095815+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.095815+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.096002+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.096002+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:08.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.106235+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.106235+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.112433+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.112433+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.112804+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.112804+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.162857+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:08.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[21463]: audit 2024-04-26T16:26:08.162857+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.451744+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.451744+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.452728+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.452728+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:06.747023+0000 mgr.y (mgr.24401) 348 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:06.747023+0000 mgr.y (mgr.24401) 348 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.826114+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.826114+0000 osd.4 (osd.4) 167 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:26:08.530 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.828247+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:06.828247+0000 osd.4 (osd.4) 168 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.152348+0000 osd.7 (osd.7) 205 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.152348+0000 osd.7 (osd.7) 205 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.156205+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.156205+0000 osd.7 (osd.7) 206 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.573193+0000 mgr.y (mgr.24401) 349 : cluster [DBG] pgmap v433: 260 pgs: 32 unknown, 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.573193+0000 mgr.y (mgr.24401) 349 : cluster [DBG] pgmap v433: 260 pgs: 32 unknown, 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.095532+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.095532+0000 mon.a (mon.0) 2045 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73230-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.095815+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.095815+0000 mon.a (mon.0) 2046 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.096002+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.096002+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.106235+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-04-26T16:26:08.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.106235+0000 mon.a (mon.0) 2048 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2024-04-26T16:26:08.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.112433+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:08.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.112433+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]: dispatch 2024-04-26T16:26:08.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.112804+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:08.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.112804+0000 mon.a (mon.0) 2050 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]: dispatch 2024-04-26T16:26:08.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.162857+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:08.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:08 smithi050 bash[29234]: audit 2024-04-26T16:26:08.162857+0000 mon.c (mon.2) 491 : audit [INF] from='client.? 172.21.15.50:0/959979605' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:08.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:08.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:08.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:08.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:08.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:08.933 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:08.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:09.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:09.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:09.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-04-26T16:26:09.111 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 1 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 10 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 13 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 7 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 14 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 0 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 15 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 11 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 5 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 8 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 6 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 3 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 4 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 12 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 9 0 2024-04-26T16:26:09.112 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 2 0 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (97 ms) 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] 3 tests from LibRadosListEC (1490 ms total) 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3002 ms) 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] 1 test from LibRadosListNP (3002 ms total) 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [----------] Global test environment tear-down 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [==========] 11 tests from 3 test suites ran. (161126 ms total) 2024-04-26T16:26:09.113 INFO:tasks.workunit.client.0.smithi050.stdout: api_list: [ PASSED ] 11 tests. 2024-04-26T16:26:09.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:09.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.777037+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:26:09.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.777037+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:26:09.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.779292+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:07.779292+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:08.163815+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:08.163815+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.164399+0000 mon.a (mon.0) 2052 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.164399+0000 mon.a (mon.0) 2052 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:08.200656+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:08.200656+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:08.203146+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:08.203146+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.204036+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:26:09.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.204036+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:26:09.175 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.205347+0000 osd.3 (osd.3) 210 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.205347+0000 osd.3 (osd.3) 210 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.577251+0000 osd.1 (osd.1) 181 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.577251+0000 osd.1 (osd.1) 181 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.578826+0000 osd.1 (osd.1) 182 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:08.578826+0000 osd.1 (osd.1) 182 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098028+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098028+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098117+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098117+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098217+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098217+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098352+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.098352+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.103455+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-04-26T16:26:09.176 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.103455+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-04-26T16:26:09.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.109452+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.109452+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.111518+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.111518+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.132146+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.177 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.132146+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.133307+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.133307+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.133325+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.133325+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.133792+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.133792+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.134305+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.134305+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.134531+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.134531+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.134848+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.178 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.134848+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.179 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.135367+0000 mon.c (mon.2) 494 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.179 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.135367+0000 mon.c (mon.2) 494 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.179 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.135656+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.179 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:09 smithi190 bash[23868]: audit 2024-04-26T16:26:09.135656+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.193 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:09.267 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:26:09.268 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:26:09.268 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:26:09.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd out 5 2024-04-26T16:26:09.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:09.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.777037+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:26:09.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.777037+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:26:09.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.779292+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:26:09.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:07.779292+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:26:09.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:08.163815+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:09.322 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:08.163815+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.164399+0000 mon.a (mon.0) 2052 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.164399+0000 mon.a (mon.0) 2052 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:08.200656+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:08.200656+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:08.203146+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:08.203146+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.204036+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.204036+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.205347+0000 osd.3 (osd.3) 210 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.205347+0000 osd.3 (osd.3) 210 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.577251+0000 osd.1 (osd.1) 181 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.577251+0000 osd.1 (osd.1) 181 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.578826+0000 osd.1 (osd.1) 182 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:26:09.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:08.578826+0000 osd.1 (osd.1) 182 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098028+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.777037+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.777037+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.779292+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:07.779292+0000 osd.4 (osd.4) 170 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:08.163815+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:08.163815+0000 mon.a (mon.0) 2051 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.164399+0000 mon.a (mon.0) 2052 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.164399+0000 mon.a (mon.0) 2052 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-04-26T16:26:09.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:08.200656+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:08.200656+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:08.203146+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:08.203146+0000 mon.b (mon.1) 198 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.204036+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.204036+0000 osd.3 (osd.3) 209 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.205347+0000 osd.3 (osd.3) 210 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.205347+0000 osd.3 (osd.3) 210 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.577251+0000 osd.1 (osd.1) 181 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.577251+0000 osd.1 (osd.1) 181 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.578826+0000 osd.1 (osd.1) 182 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:08.578826+0000 osd.1 (osd.1) 182 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:26:09.325 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098028+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098028+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098117+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098117+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098217+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098217+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098352+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.098352+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.103455+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.103455+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.109452+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.109452+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.111518+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.326 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.111518+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.132146+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.132146+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098028+0000 mon.a (mon.0) 2054 : audit [INF] from='client.? 172.21.15.50:0/2510398972' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi050-73193-38"}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098117+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098117+0000 mon.a (mon.0) 2055 : audit [INF] from='client.? 172.21.15.50:0/2601972082' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi050-73201-47"}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098217+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098217+0000 mon.a (mon.0) 2056 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi050-73230-3","pool2":"test-rados-api-smithi050-73230-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098352+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.098352+0000 mon.a (mon.0) 2057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.103455+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-04-26T16:26:09.327 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.103455+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.109452+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.109452+0000 mon.a (mon.0) 2059 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.111518+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.111518+0000 mon.b (mon.1) 199 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.132146+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.132146+0000 mon.a (mon.0) 2060 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.133307+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.133307+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.133325+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.133325+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.328 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.133792+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.133792+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.134305+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.134305+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.134531+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.134531+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.134848+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.134848+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.135367+0000 mon.c (mon.2) 494 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.135367+0000 mon.c (mon.2) 494 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.135656+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[21463]: audit 2024-04-26T16:26:09.135656+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.329 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.133307+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.133307+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.133325+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.133325+0000 mon.c (mon.2) 492 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.133792+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.133792+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.134305+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.134305+0000 mon.a (mon.0) 2063 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.134531+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.134531+0000 mon.c (mon.2) 493 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.134848+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.134848+0000 mon.a (mon.0) 2064 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.135367+0000 mon.c (mon.2) 494 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.135367+0000 mon.c (mon.2) 494 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.135656+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:09 smithi050 bash[29234]: audit 2024-04-26T16:26:09.135656+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:09.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:09.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:09.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:09.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:09.587 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:09.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:09.718 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:09.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:09.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:09.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:09.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:09.981 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:09.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:10.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:10.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:10.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:10.249 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:10.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.143281+0000 osd.5 (osd.5) 189 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:26:10.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.143281+0000 osd.5 (osd.5) 189 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.151319+0000 osd.5 (osd.5) 190 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.151319+0000 osd.5 (osd.5) 190 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.573963+0000 mgr.y (mgr.24401) 350 : cluster [DBG] pgmap v436: 228 pgs: 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.573963+0000 mgr.y (mgr.24401) 350 : cluster [DBG] pgmap v436: 228 pgs: 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.098377+0000 mon.a (mon.0) 2066 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.098377+0000 mon.a (mon.0) 2066 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.100494+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.100494+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.100656+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.100656+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.100785+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.100785+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.109067+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.109067+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.109461+0000 mon.c (mon.2) 495 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.109461+0000 mon.c (mon.2) 495 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.110047+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.110047+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.110860+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:10 smithi190 bash[23868]: audit 2024-04-26T16:26:10.110860+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:10.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.143281+0000 osd.5 (osd.5) 189 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.143281+0000 osd.5 (osd.5) 189 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.151319+0000 osd.5 (osd.5) 190 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.151319+0000 osd.5 (osd.5) 190 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.573963+0000 mgr.y (mgr.24401) 350 : cluster [DBG] pgmap v436: 228 pgs: 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.573963+0000 mgr.y (mgr.24401) 350 : cluster [DBG] pgmap v436: 228 pgs: 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.098377+0000 mon.a (mon.0) 2066 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.098377+0000 mon.a (mon.0) 2066 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.100494+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.100494+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.100656+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.100656+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.100785+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.100785+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.109067+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.109067+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.109461+0000 mon.c (mon.2) 495 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.143281+0000 osd.5 (osd.5) 189 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.143281+0000 osd.5 (osd.5) 189 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.151319+0000 osd.5 (osd.5) 190 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:26:10.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.151319+0000 osd.5 (osd.5) 190 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.573963+0000 mgr.y (mgr.24401) 350 : cluster [DBG] pgmap v436: 228 pgs: 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.573963+0000 mgr.y (mgr.24401) 350 : cluster [DBG] pgmap v436: 228 pgs: 228 active+clean; 8.6 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 255 B/s wr, 24 op/s 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.098377+0000 mon.a (mon.0) 2066 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.098377+0000 mon.a (mon.0) 2066 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.100494+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.100494+0000 mon.a (mon.0) 2067 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-29"}]': finished 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.100656+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.100656+0000 mon.a (mon.0) 2068 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi050-73193-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.100785+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.100785+0000 mon.a (mon.0) 2069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi050-73201-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.109067+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.109067+0000 mon.a (mon.0) 2070 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2024-04-26T16:26:10.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.109461+0000 mon.c (mon.2) 495 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.109461+0000 mon.c (mon.2) 495 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.110047+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.110047+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.110860+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[21463]: audit 2024-04-26T16:26:10.110860+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.109461+0000 mon.c (mon.2) 495 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.110047+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.110047+0000 mon.a (mon.0) 2071 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:10.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.110860+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:10 smithi050 bash[29234]: audit 2024-04-26T16:26:10.110860+0000 mon.a (mon.0) 2072 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:10.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:10.504 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:10.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:10.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:10.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:10.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:10.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:10.797 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:26:10.797 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:26:10.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:10.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:10.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:11.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:11.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:11.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:11.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:11.284 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:11.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:11.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.429268+0000 osd.2 (osd.2) 147 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.429268+0000 osd.2 (osd.2) 147 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.434174+0000 osd.2 (osd.2) 148 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:09.434174+0000 osd.2 (osd.2) 148 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.118288+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.118288+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.120873+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.120873+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.086370+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.086370+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.144718+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-04-26T16:26:11.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.144718+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.429268+0000 osd.2 (osd.2) 147 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.429268+0000 osd.2 (osd.2) 147 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.434174+0000 osd.2 (osd.2) 148 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:09.434174+0000 osd.2 (osd.2) 148 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.118288+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.118288+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.120873+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.120873+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.086370+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.086370+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:11.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.144718+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-04-26T16:26:11.403 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:11 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.144718+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-04-26T16:26:11.414 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:11.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:11.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.429268+0000 osd.2 (osd.2) 147 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.429268+0000 osd.2 (osd.2) 147 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.434174+0000 osd.2 (osd.2) 148 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:09.434174+0000 osd.2 (osd.2) 148 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.118288+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.118288+0000 osd.5 (osd.5) 191 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:26:11.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.120873+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:26:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.120873+0000 osd.5 (osd.5) 192 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:26:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.086370+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.086370+0000 mon.a (mon.0) 2073 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.144718+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-04-26T16:26:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:11 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.144718+0000 mon.a (mon.0) 2074 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2024-04-26T16:26:11.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-26T16:26:11.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:11.674 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:11.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:11.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:11.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:11.936 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:11.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:12.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:12.066 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:12.128 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:12.130 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:12.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:12.197 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:12.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:12.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:12.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.499306+0000 osd.6 (osd.6) 145 : cluster [DBG] 3.a scrub starts 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.499306+0000 osd.6 (osd.6) 145 : cluster [DBG] 3.a scrub starts 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.504529+0000 osd.6 (osd.6) 146 : cluster [DBG] 3.a scrub ok 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:10.504529+0000 osd.6 (osd.6) 146 : cluster [DBG] 3.a scrub ok 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.138027+0000 osd.5 (osd.5) 193 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.138027+0000 osd.5 (osd.5) 193 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.143685+0000 osd.3 (osd.3) 211 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.143685+0000 osd.3 (osd.3) 211 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.145319+0000 osd.5 (osd.5) 194 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.145319+0000 osd.5 (osd.5) 194 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:26:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.148880+0000 osd.3 (osd.3) 212 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.148880+0000 osd.3 (osd.3) 212 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:11.457365+0000 mon.a (mon.0) 2075 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:11.457365+0000 mon.a (mon.0) 2075 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:11.458802+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:11.458802+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.575306+0000 mgr.y (mgr.24401) 351 : cluster [DBG] pgmap v439: 196 pgs: 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.575306+0000 mgr.y (mgr.24401) 351 : cluster [DBG] pgmap v439: 196 pgs: 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.108151+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.108151+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.108420+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.499306+0000 osd.6 (osd.6) 145 : cluster [DBG] 3.a scrub starts 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.499306+0000 osd.6 (osd.6) 145 : cluster [DBG] 3.a scrub starts 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.504529+0000 osd.6 (osd.6) 146 : cluster [DBG] 3.a scrub ok 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:10.504529+0000 osd.6 (osd.6) 146 : cluster [DBG] 3.a scrub ok 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.138027+0000 osd.5 (osd.5) 193 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.138027+0000 osd.5 (osd.5) 193 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.143685+0000 osd.3 (osd.3) 211 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.143685+0000 osd.3 (osd.3) 211 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.145319+0000 osd.5 (osd.5) 194 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:26:12.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.145319+0000 osd.5 (osd.5) 194 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.148880+0000 osd.3 (osd.3) 212 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.148880+0000 osd.3 (osd.3) 212 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:11.457365+0000 mon.a (mon.0) 2075 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:11.457365+0000 mon.a (mon.0) 2075 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:11.458802+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:11.458802+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.575306+0000 mgr.y (mgr.24401) 351 : cluster [DBG] pgmap v439: 196 pgs: 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.575306+0000 mgr.y (mgr.24401) 351 : cluster [DBG] pgmap v439: 196 pgs: 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.108151+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:12.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.108151+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.108420+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.108420+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.119685+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.119685+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.125104+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.125104+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.135700+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[29234]: audit 2024-04-26T16:26:12.135700+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.108420+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.119685+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.119685+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-04-26T16:26:12.445 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.125104+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.125104+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.135700+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:12 smithi050 bash[21463]: audit 2024-04-26T16:26:12.135700+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:12.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:12.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.499306+0000 osd.6 (osd.6) 145 : cluster [DBG] 3.a scrub starts 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.499306+0000 osd.6 (osd.6) 145 : cluster [DBG] 3.a scrub starts 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.504529+0000 osd.6 (osd.6) 146 : cluster [DBG] 3.a scrub ok 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:10.504529+0000 osd.6 (osd.6) 146 : cluster [DBG] 3.a scrub ok 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.138027+0000 osd.5 (osd.5) 193 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.138027+0000 osd.5 (osd.5) 193 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.143685+0000 osd.3 (osd.3) 211 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.143685+0000 osd.3 (osd.3) 211 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.145319+0000 osd.5 (osd.5) 194 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.145319+0000 osd.5 (osd.5) 194 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:26:12.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.148880+0000 osd.3 (osd.3) 212 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.148880+0000 osd.3 (osd.3) 212 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:11.457365+0000 mon.a (mon.0) 2075 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:11.457365+0000 mon.a (mon.0) 2075 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:11.458802+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:11.458802+0000 mon.c (mon.2) 496 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.575306+0000 mgr.y (mgr.24401) 351 : cluster [DBG] pgmap v439: 196 pgs: 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.575306+0000 mgr.y (mgr.24401) 351 : cluster [DBG] pgmap v439: 196 pgs: 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.108151+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.108151+0000 mon.a (mon.0) 2076 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi050-73193-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.108420+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.108420+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi050-73201-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.119685+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-04-26T16:26:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.119685+0000 mon.a (mon.0) 2078 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2024-04-26T16:26:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.125104+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.125104+0000 mon.b (mon.1) 200 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.135700+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:12 smithi190 bash[23868]: audit 2024-04-26T16:26:12.135700+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:12.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:12.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:12.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:12.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:12.852 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:12.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:12.981 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:12.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:12.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:13.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:13.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:13.141 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:13.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:13.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:13.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:13.374 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.385721+0000 osd.2 (osd.2) 149 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.385721+0000 osd.2 (osd.2) 149 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.394923+0000 osd.2 (osd.2) 150 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.394923+0000 osd.2 (osd.2) 150 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.790852+0000 osd.4 (osd.4) 171 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.790852+0000 osd.4 (osd.4) 171 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.793256+0000 osd.4 (osd.4) 172 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.793256+0000 osd.4 (osd.4) 172 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.954045+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.b scrub starts 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.954045+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.b scrub starts 2024-04-26T16:26:13.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.955178+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.b scrub ok 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:11.955178+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.b scrub ok 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.145456+0000 osd.5 (osd.5) 195 : cluster [DBG] 109.d scrub starts 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.145456+0000 osd.5 (osd.5) 195 : cluster [DBG] 109.d scrub starts 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.147627+0000 osd.5 (osd.5) 196 : cluster [DBG] 109.d scrub ok 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.147627+0000 osd.5 (osd.5) 196 : cluster [DBG] 109.d scrub ok 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.427645+0000 osd.0 (osd.0) 203 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.427645+0000 osd.0 (osd.0) 203 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.430061+0000 osd.0 (osd.0) 204 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.430061+0000 osd.0 (osd.0) 204 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: audit 2024-04-26T16:26:13.111218+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: audit 2024-04-26T16:26:13.111218+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.120592+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.120592+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-04-26T16:26:13.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: audit 2024-04-26T16:26:13.137442+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.385721+0000 osd.2 (osd.2) 149 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.385721+0000 osd.2 (osd.2) 149 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.394923+0000 osd.2 (osd.2) 150 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.394923+0000 osd.2 (osd.2) 150 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.790852+0000 osd.4 (osd.4) 171 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.790852+0000 osd.4 (osd.4) 171 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.793256+0000 osd.4 (osd.4) 172 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.793256+0000 osd.4 (osd.4) 172 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.954045+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.b scrub starts 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.954045+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.b scrub starts 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.955178+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.b scrub ok 2024-04-26T16:26:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:11.955178+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.b scrub ok 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.145456+0000 osd.5 (osd.5) 195 : cluster [DBG] 109.d scrub starts 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.145456+0000 osd.5 (osd.5) 195 : cluster [DBG] 109.d scrub starts 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.147627+0000 osd.5 (osd.5) 196 : cluster [DBG] 109.d scrub ok 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.147627+0000 osd.5 (osd.5) 196 : cluster [DBG] 109.d scrub ok 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.427645+0000 osd.0 (osd.0) 203 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.427645+0000 osd.0 (osd.0) 203 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.430061+0000 osd.0 (osd.0) 204 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.430061+0000 osd.0 (osd.0) 204 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: audit 2024-04-26T16:26:13.111218+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: audit 2024-04-26T16:26:13.111218+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.120592+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.120592+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-04-26T16:26:13.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: audit 2024-04-26T16:26:13.137442+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: audit 2024-04-26T16:26:13.137442+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: audit 2024-04-26T16:26:13.139481+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[29234]: audit 2024-04-26T16:26:13.139481+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: audit 2024-04-26T16:26:13.137442+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: audit 2024-04-26T16:26:13.139481+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:13 smithi050 bash[21463]: audit 2024-04-26T16:26:13.139481+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:13.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.385721+0000 osd.2 (osd.2) 149 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.385721+0000 osd.2 (osd.2) 149 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.394923+0000 osd.2 (osd.2) 150 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.394923+0000 osd.2 (osd.2) 150 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.790852+0000 osd.4 (osd.4) 171 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.790852+0000 osd.4 (osd.4) 171 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.793256+0000 osd.4 (osd.4) 172 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:26:13.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.793256+0000 osd.4 (osd.4) 172 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.954045+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.b scrub starts 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.954045+0000 osd.7 (osd.7) 207 : cluster [DBG] 2.b scrub starts 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.955178+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.b scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:11.955178+0000 osd.7 (osd.7) 208 : cluster [DBG] 2.b scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.145456+0000 osd.5 (osd.5) 195 : cluster [DBG] 109.d scrub starts 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.145456+0000 osd.5 (osd.5) 195 : cluster [DBG] 109.d scrub starts 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.147627+0000 osd.5 (osd.5) 196 : cluster [DBG] 109.d scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.147627+0000 osd.5 (osd.5) 196 : cluster [DBG] 109.d scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.427645+0000 osd.0 (osd.0) 203 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.427645+0000 osd.0 (osd.0) 203 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.430061+0000 osd.0 (osd.0) 204 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.430061+0000 osd.0 (osd.0) 204 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:26:13.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: audit 2024-04-26T16:26:13.111218+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: audit 2024-04-26T16:26:13.111218+0000 mon.a (mon.0) 2080 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.120592+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.120592+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: audit 2024-04-26T16:26:13.137442+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: audit 2024-04-26T16:26:13.137442+0000 mon.a (mon.0) 2082 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: audit 2024-04-26T16:26:13.139481+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:13 smithi190 bash[23868]: audit 2024-04-26T16:26:13.139481+0000 mon.b (mon.1) 201 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:13.609 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:13.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:13.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:13.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:13.768 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:13.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:13.900 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:13.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:14.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:14.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:14.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:14.134 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:14.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:14.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:14.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:14.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:14.411 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:14.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.540995+0000 osd.6 (osd.6) 147 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:26:14.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.540995+0000 osd.6 (osd.6) 147 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:26:14.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.544341+0000 osd.6 (osd.6) 148 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:26:14.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:12.544341+0000 osd.6 (osd.6) 148 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:26:14.430 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.002357+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.002357+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.003816+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.003816+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.185163+0000 osd.5 (osd.5) 197 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.185163+0000 osd.5 (osd.5) 197 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.187608+0000 osd.5 (osd.5) 198 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.187608+0000 osd.5 (osd.5) 198 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.576155+0000 mgr.y (mgr.24401) 352 : cluster [DBG] pgmap v442: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.576155+0000 mgr.y (mgr.24401) 352 : cluster [DBG] pgmap v442: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.114303+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.114303+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.123390+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.123390+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-04-26T16:26:14.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.124531+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.124531+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.125357+0000 mon.c (mon.2) 497 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.540995+0000 osd.6 (osd.6) 147 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.540995+0000 osd.6 (osd.6) 147 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.544341+0000 osd.6 (osd.6) 148 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:12.544341+0000 osd.6 (osd.6) 148 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.002357+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.002357+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.003816+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:26:14.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.003816+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.185163+0000 osd.5 (osd.5) 197 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.185163+0000 osd.5 (osd.5) 197 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.187608+0000 osd.5 (osd.5) 198 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.187608+0000 osd.5 (osd.5) 198 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.576155+0000 mgr.y (mgr.24401) 352 : cluster [DBG] pgmap v442: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.576155+0000 mgr.y (mgr.24401) 352 : cluster [DBG] pgmap v442: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.114303+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.114303+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.123390+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.123390+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.124531+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.124531+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.125357+0000 mon.c (mon.2) 497 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.125357+0000 mon.c (mon.2) 497 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.125985+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.125985+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.126108+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.126108+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.126309+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[29234]: audit 2024-04-26T16:26:14.126309+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.125357+0000 mon.c (mon.2) 497 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.125985+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.125985+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.126108+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.126108+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.126309+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:14.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21463]: audit 2024-04-26T16:26:14.126309+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:14.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:14.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.540995+0000 osd.6 (osd.6) 147 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:26:14.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.540995+0000 osd.6 (osd.6) 147 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:26:14.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.544341+0000 osd.6 (osd.6) 148 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:26:14.526 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:12.544341+0000 osd.6 (osd.6) 148 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.002357+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.002357+0000 osd.7 (osd.7) 209 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.003816+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.003816+0000 osd.7 (osd.7) 210 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.185163+0000 osd.5 (osd.5) 197 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.185163+0000 osd.5 (osd.5) 197 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.187608+0000 osd.5 (osd.5) 198 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.187608+0000 osd.5 (osd.5) 198 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.576155+0000 mgr.y (mgr.24401) 352 : cluster [DBG] pgmap v442: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.576155+0000 mgr.y (mgr.24401) 352 : cluster [DBG] pgmap v442: 244 pgs: 48 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.114303+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.114303+0000 mon.a (mon.0) 2083 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.123390+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-04-26T16:26:14.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.123390+0000 mon.a (mon.0) 2084 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.124531+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.124531+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.125357+0000 mon.c (mon.2) 497 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.125357+0000 mon.c (mon.2) 497 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.125985+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.125985+0000 mon.b (mon.1) 202 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.126108+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.126108+0000 mon.a (mon.0) 2086 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.126309+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:14.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:14 smithi190 bash[23868]: audit 2024-04-26T16:26:14.126309+0000 mon.a (mon.0) 2087 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:14.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:14.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:14.672 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:14.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:14.802 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:14.878 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:26:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:26:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:26:14.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:14.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:14.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:15.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:15.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:15.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:15.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:15.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:15.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:15.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:15.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.400257+0000 osd.2 (osd.2) 151 : cluster [DBG] 11.d scrub starts 2024-04-26T16:26:15.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.400257+0000 osd.2 (osd.2) 151 : cluster [DBG] 11.d scrub starts 2024-04-26T16:26:15.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.402925+0000 osd.2 (osd.2) 152 : cluster [DBG] 11.d scrub ok 2024-04-26T16:26:15.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.402925+0000 osd.2 (osd.2) 152 : cluster [DBG] 11.d scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.545907+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.545907+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.548119+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:13.548119+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.108868+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.108868+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.111051+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.111051+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.396037+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.396037+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.398461+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.398461+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.116514+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.116514+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.116631+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.116631+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.116690+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.116690+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.120487+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.120487+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.120857+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.120857+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.121307+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.121307+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.121659+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.121659+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.126261+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.126261+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.128579+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.446 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:15 smithi190 bash[23868]: audit 2024-04-26T16:26:15.128579+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.458 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:15.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:15.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:15.589 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.400257+0000 osd.2 (osd.2) 151 : cluster [DBG] 11.d scrub starts 2024-04-26T16:26:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.400257+0000 osd.2 (osd.2) 151 : cluster [DBG] 11.d scrub starts 2024-04-26T16:26:15.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.402925+0000 osd.2 (osd.2) 152 : cluster [DBG] 11.d scrub ok 2024-04-26T16:26:15.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.402925+0000 osd.2 (osd.2) 152 : cluster [DBG] 11.d scrub ok 2024-04-26T16:26:15.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.545907+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:26:15.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.545907+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.548119+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:13.548119+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.108868+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.108868+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.111051+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.111051+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.396037+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.396037+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.398461+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.398461+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.116514+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.116514+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.116631+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.116631+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.116690+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.116690+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.120487+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.120487+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.120857+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.120857+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.121307+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.121307+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.121659+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.121659+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.126261+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.126261+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.128579+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[29234]: audit 2024-04-26T16:26:15.128579+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.400257+0000 osd.2 (osd.2) 151 : cluster [DBG] 11.d scrub starts 2024-04-26T16:26:15.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.400257+0000 osd.2 (osd.2) 151 : cluster [DBG] 11.d scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.402925+0000 osd.2 (osd.2) 152 : cluster [DBG] 11.d scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.402925+0000 osd.2 (osd.2) 152 : cluster [DBG] 11.d scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.545907+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.545907+0000 osd.6 (osd.6) 149 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.548119+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:13.548119+0000 osd.6 (osd.6) 150 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.108868+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.108868+0000 osd.3 (osd.3) 213 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.111051+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.111051+0000 osd.3 (osd.3) 214 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.396037+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.396037+0000 osd.0 (osd.0) 205 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.398461+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:26:15.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.398461+0000 osd.0 (osd.0) 206 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.116514+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.116514+0000 mon.a (mon.0) 2088 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.116631+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.116631+0000 mon.a (mon.0) 2089 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.116690+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.116690+0000 mon.a (mon.0) 2090 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.120487+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.120487+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.120857+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.120857+0000 mon.a (mon.0) 2092 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]: dispatch 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.121307+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.121307+0000 mon.c (mon.2) 498 : audit [INF] from='client.? 172.21.15.50:0/2745962637' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.121659+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.121659+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]: dispatch 2024-04-26T16:26:15.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.126261+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.126261+0000 mon.a (mon.0) 2094 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.128579+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.636 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:15 smithi050 bash[21463]: audit 2024-04-26T16:26:15.128579+0000 mon.b (mon.1) 203 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]: dispatch 2024-04-26T16:26:15.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:15.719 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:15.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:15.849 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:15.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:15.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:15.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:16.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:16.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:16.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:16.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.432323+0000 osd.2 (osd.2) 153 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:26:16.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.432323+0000 osd.2 (osd.2) 153 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:26:16.235 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.436280+0000 osd.2 (osd.2) 154 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:14.436280+0000 osd.2 (osd.2) 154 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.156465+0000 osd.3 (osd.3) 215 : cluster [DBG] 11.a scrub starts 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.156465+0000 osd.3 (osd.3) 215 : cluster [DBG] 11.a scrub starts 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.158574+0000 osd.3 (osd.3) 216 : cluster [DBG] 11.a scrub ok 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.158574+0000 osd.3 (osd.3) 216 : cluster [DBG] 11.a scrub ok 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.577003+0000 mgr.y (mgr.24401) 353 : cluster [DBG] pgmap v445: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.577003+0000 mgr.y (mgr.24401) 353 : cluster [DBG] pgmap v445: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.087862+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.087862+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:16.236 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.089320+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.089320+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.091499+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.091499+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.091714+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.091714+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.091917+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]': finished 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.091917+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]': finished 2024-04-26T16:26:16.237 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.102423+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.432323+0000 osd.2 (osd.2) 153 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.432323+0000 osd.2 (osd.2) 153 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.436280+0000 osd.2 (osd.2) 154 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:14.436280+0000 osd.2 (osd.2) 154 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.156465+0000 osd.3 (osd.3) 215 : cluster [DBG] 11.a scrub starts 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.156465+0000 osd.3 (osd.3) 215 : cluster [DBG] 11.a scrub starts 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.158574+0000 osd.3 (osd.3) 216 : cluster [DBG] 11.a scrub ok 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.158574+0000 osd.3 (osd.3) 216 : cluster [DBG] 11.a scrub ok 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.577003+0000 mgr.y (mgr.24401) 353 : cluster [DBG] pgmap v445: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:16.238 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.577003+0000 mgr.y (mgr.24401) 353 : cluster [DBG] pgmap v445: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.087862+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.087862+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.089320+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.089320+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.091499+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.091499+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.091714+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.091714+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.091917+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]': finished 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.091917+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]': finished 2024-04-26T16:26:16.239 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.102423+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-04-26T16:26:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.102423+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-04-26T16:26:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.126258+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.126258+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.126674+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.126674+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.127772+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.127772+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128135+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128135+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128230+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128230+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128784+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128784+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128849+0000 mon.c (mon.2) 502 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.128849+0000 mon.c (mon.2) 502 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129141+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129141+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129511+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129511+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129901+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129901+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129937+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.129937+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.130489+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.242 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[21463]: audit 2024-04-26T16:26:16.130489+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.102423+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.126258+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.126258+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.126674+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.126674+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.127772+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.127772+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.244 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128135+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128135+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128230+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128230+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128784+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128784+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128849+0000 mon.c (mon.2) 502 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.128849+0000 mon.c (mon.2) 502 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129141+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129141+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129511+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129511+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129901+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129901+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.245 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129937+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.129937+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.130489+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.246 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:16 smithi050 bash[29234]: audit 2024-04-26T16:26:16.130489+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.339 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:26:16.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:16.368 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:16.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:16.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.432323+0000 osd.2 (osd.2) 153 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.432323+0000 osd.2 (osd.2) 153 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.436280+0000 osd.2 (osd.2) 154 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:14.436280+0000 osd.2 (osd.2) 154 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.156465+0000 osd.3 (osd.3) 215 : cluster [DBG] 11.a scrub starts 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.156465+0000 osd.3 (osd.3) 215 : cluster [DBG] 11.a scrub starts 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.158574+0000 osd.3 (osd.3) 216 : cluster [DBG] 11.a scrub ok 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.158574+0000 osd.3 (osd.3) 216 : cluster [DBG] 11.a scrub ok 2024-04-26T16:26:16.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.577003+0000 mgr.y (mgr.24401) 353 : cluster [DBG] pgmap v445: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.577003+0000 mgr.y (mgr.24401) 353 : cluster [DBG] pgmap v445: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.087862+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.087862+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.089320+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.089320+0000 mon.a (mon.0) 2096 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.091499+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.091499+0000 mon.a (mon.0) 2097 : audit [INF] from='client.? 172.21.15.50:0/1212580486' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi050-73193-39"}]': finished 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.091714+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.091714+0000 mon.a (mon.0) 2098 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi050-73201-48"}]': finished 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.091917+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]': finished 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.091917+0000 mon.a (mon.0) 2099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-31", "mode": "writeback"}]': finished 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.102423+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.102423+0000 mon.a (mon.0) 2100 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2024-04-26T16:26:16.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.126258+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.126258+0000 mon.c (mon.2) 499 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.126674+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.126674+0000 mon.a (mon.0) 2101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.127772+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.127772+0000 mon.c (mon.2) 500 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128135+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128135+0000 mon.a (mon.0) 2102 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128230+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128230+0000 mon.c (mon.2) 501 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128784+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128784+0000 mon.a (mon.0) 2103 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128849+0000 mon.c (mon.2) 502 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.128849+0000 mon.c (mon.2) 502 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129141+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129141+0000 mon.a (mon.0) 2104 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129511+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129511+0000 mon.c (mon.2) 503 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129901+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129901+0000 mon.c (mon.2) 504 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129937+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.129937+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.130489+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[23868]: audit 2024-04-26T16:26:16.130489+0000 mon.a (mon.0) 2106 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:16.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:16.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:16.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:16.752 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:16.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:16.885 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:16.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:16.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:16.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:17.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:17.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:26:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:26:17.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:17.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:17.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.429528+0000 osd.2 (osd.2) 155 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.429528+0000 osd.2 (osd.2) 155 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.436142+0000 osd.2 (osd.2) 156 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.436142+0000 osd.2 (osd.2) 156 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.785060+0000 osd.4 (osd.4) 173 : cluster [DBG] 109.a scrub starts 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.785060+0000 osd.4 (osd.4) 173 : cluster [DBG] 109.a scrub starts 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.788365+0000 osd.4 (osd.4) 174 : cluster [DBG] 109.a scrub ok 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:15.788365+0000 osd.4 (osd.4) 174 : cluster [DBG] 109.a scrub ok 2024-04-26T16:26:17.248 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.156478+0000 osd.3 (osd.3) 217 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.156478+0000 osd.3 (osd.3) 217 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.158778+0000 osd.3 (osd.3) 218 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.158778+0000 osd.3 (osd.3) 218 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:16.258345+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:16.258345+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:16.260730+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:16.260730+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.487392+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.487392+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.490580+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.490580+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.093769+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.093769+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.094034+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.094034+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.094309+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.094309+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.099054+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.099054+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.103512+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.103512+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.105758+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.105758+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.107691+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.107691+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.250 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.108456+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.108456+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.109743+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.109743+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.110303+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[21463]: audit 2024-04-26T16:26:17.110303+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.429528+0000 osd.2 (osd.2) 155 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.429528+0000 osd.2 (osd.2) 155 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.436142+0000 osd.2 (osd.2) 156 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.436142+0000 osd.2 (osd.2) 156 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.785060+0000 osd.4 (osd.4) 173 : cluster [DBG] 109.a scrub starts 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.785060+0000 osd.4 (osd.4) 173 : cluster [DBG] 109.a scrub starts 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.788365+0000 osd.4 (osd.4) 174 : cluster [DBG] 109.a scrub ok 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:15.788365+0000 osd.4 (osd.4) 174 : cluster [DBG] 109.a scrub ok 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.156478+0000 osd.3 (osd.3) 217 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.156478+0000 osd.3 (osd.3) 217 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.158778+0000 osd.3 (osd.3) 218 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.158778+0000 osd.3 (osd.3) 218 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:26:17.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:16.258345+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:16.258345+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:16.260730+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:16.260730+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.487392+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.487392+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.490580+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.490580+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.093769+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.093769+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.094034+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.094034+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.094309+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:17.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.094309+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.099054+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.099054+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.103512+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.103512+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.105758+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.105758+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.107691+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.107691+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.108456+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.108456+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.109743+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.109743+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.110303+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.254 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:17 smithi050 bash[29234]: audit 2024-04-26T16:26:17.110303+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.269 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:17.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:17.399 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:17.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:17.530 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:17.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:17.538 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.429528+0000 osd.2 (osd.2) 155 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:26:17.538 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.429528+0000 osd.2 (osd.2) 155 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:26:17.538 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.436142+0000 osd.2 (osd.2) 156 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:26:17.538 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.436142+0000 osd.2 (osd.2) 156 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.785060+0000 osd.4 (osd.4) 173 : cluster [DBG] 109.a scrub starts 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.785060+0000 osd.4 (osd.4) 173 : cluster [DBG] 109.a scrub starts 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.788365+0000 osd.4 (osd.4) 174 : cluster [DBG] 109.a scrub ok 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:15.788365+0000 osd.4 (osd.4) 174 : cluster [DBG] 109.a scrub ok 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.156478+0000 osd.3 (osd.3) 217 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.156478+0000 osd.3 (osd.3) 217 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.158778+0000 osd.3 (osd.3) 218 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.158778+0000 osd.3 (osd.3) 218 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:16.258345+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:16.258345+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:16.260730+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:16.260730+0000 mon.b (mon.1) 204 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:17.539 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.487392+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.487392+0000 osd.1 (osd.1) 183 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.490580+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.490580+0000 osd.1 (osd.1) 184 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.093769+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.093769+0000 mon.a (mon.0) 2108 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi050-73201-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.094034+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.094034+0000 mon.a (mon.0) 2109 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi050-73193-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.094309+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.094309+0000 mon.a (mon.0) 2110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.099054+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.099054+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.103512+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.103512+0000 mon.a (mon.0) 2112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.105758+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.105758+0000 mon.b (mon.1) 205 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.107691+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.107691+0000 mon.c (mon.2) 505 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.108456+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.108456+0000 mon.a (mon.0) 2113 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.109743+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.109743+0000 mon.c (mon.2) 506 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.110303+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:17 smithi190 bash[23868]: audit 2024-04-26T16:26:17.110303+0000 mon.a (mon.0) 2114 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:17.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:17.661 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:17.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:17.790 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:17.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:17.921 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:17.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:17.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:18.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:18.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:18.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:18.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:18.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:18.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.407916+0000 osd.2 (osd.2) 157 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:26:18.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.407916+0000 osd.2 (osd.2) 157 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:26:18.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.410741+0000 osd.2 (osd.2) 158 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:26:18.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:16.410741+0000 osd.2 (osd.2) 158 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:26:18.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: audit 2024-04-26T16:26:16.754904+0000 mgr.y (mgr.24401) 354 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:18.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: audit 2024-04-26T16:26:16.754904+0000 mgr.y (mgr.24401) 354 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.177567+0000 osd.3 (osd.3) 219 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.177567+0000 osd.3 (osd.3) 219 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.183378+0000 osd.3 (osd.3) 220 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.183378+0000 osd.3 (osd.3) 220 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.469736+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.469736+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.473603+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.473603+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.577801+0000 mgr.y (mgr.24401) 355 : cluster [DBG] pgmap v448: 228 pgs: 228 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:17.577801+0000 mgr.y (mgr.24401) 355 : cluster [DBG] pgmap v448: 228 pgs: 228 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.094332+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.094332+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:18.341 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: audit 2024-04-26T16:26:18.095659+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:18.342 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: audit 2024-04-26T16:26:18.095659+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:18.342 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.116082+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-04-26T16:26:18.342 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:18 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.116082+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-04-26T16:26:18.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.407916+0000 osd.2 (osd.2) 157 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:26:18.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.407916+0000 osd.2 (osd.2) 157 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:26:18.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.410741+0000 osd.2 (osd.2) 158 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:26:18.353 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:16.410741+0000 osd.2 (osd.2) 158 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: audit 2024-04-26T16:26:16.754904+0000 mgr.y (mgr.24401) 354 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: audit 2024-04-26T16:26:16.754904+0000 mgr.y (mgr.24401) 354 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.177567+0000 osd.3 (osd.3) 219 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.177567+0000 osd.3 (osd.3) 219 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.183378+0000 osd.3 (osd.3) 220 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.183378+0000 osd.3 (osd.3) 220 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.469736+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.469736+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.473603+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.473603+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.577801+0000 mgr.y (mgr.24401) 355 : cluster [DBG] pgmap v448: 228 pgs: 228 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:17.577801+0000 mgr.y (mgr.24401) 355 : cluster [DBG] pgmap v448: 228 pgs: 228 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.094332+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:18.354 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.094332+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: audit 2024-04-26T16:26:18.095659+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: audit 2024-04-26T16:26:18.095659+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.116082+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.116082+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.407916+0000 osd.2 (osd.2) 157 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.407916+0000 osd.2 (osd.2) 157 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.410741+0000 osd.2 (osd.2) 158 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:16.410741+0000 osd.2 (osd.2) 158 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: audit 2024-04-26T16:26:16.754904+0000 mgr.y (mgr.24401) 354 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:18.355 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: audit 2024-04-26T16:26:16.754904+0000 mgr.y (mgr.24401) 354 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.177567+0000 osd.3 (osd.3) 219 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.177567+0000 osd.3 (osd.3) 219 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.183378+0000 osd.3 (osd.3) 220 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.183378+0000 osd.3 (osd.3) 220 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.469736+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.469736+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.473603+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.473603+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.577801+0000 mgr.y (mgr.24401) 355 : cluster [DBG] pgmap v448: 228 pgs: 228 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:17.577801+0000 mgr.y (mgr.24401) 355 : cluster [DBG] pgmap v448: 228 pgs: 228 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:26:18.356 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.094332+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:18.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.094332+0000 mon.a (mon.0) 2115 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:18.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: audit 2024-04-26T16:26:18.095659+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:18.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: audit 2024-04-26T16:26:18.095659+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-31"}]': finished 2024-04-26T16:26:18.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.116082+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-04-26T16:26:18.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:18 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.116082+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2024-04-26T16:26:18.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:18.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:18.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:18.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:18.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:18.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:18.764 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:18.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:18.893 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:18.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:18.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:19.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:19.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:19.146 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:19.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:19.260 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.093733+0000 osd.5 (osd.5) 199 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.093733+0000 osd.5 (osd.5) 199 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.096036+0000 osd.5 (osd.5) 200 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.096036+0000 osd.5 (osd.5) 200 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.423116+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.423116+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.424760+0000 osd.0 (osd.0) 208 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.424760+0000 osd.0 (osd.0) 208 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:26:19.261 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.896718+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:26:19.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.896718+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:26:19.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.898223+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:26:19.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:18.898223+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:26:19.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: audit 2024-04-26T16:26:19.097965+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:19.262 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: audit 2024-04-26T16:26:19.097965+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:19.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: audit 2024-04-26T16:26:19.098114+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:19.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: audit 2024-04-26T16:26:19.098114+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:19.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.102292+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-04-26T16:26:19.263 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.102292+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-04-26T16:26:19.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.093733+0000 osd.5 (osd.5) 199 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:26:19.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.093733+0000 osd.5 (osd.5) 199 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:26:19.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.096036+0000 osd.5 (osd.5) 200 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:26:19.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.096036+0000 osd.5 (osd.5) 200 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:26:19.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.423116+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:26:19.264 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.423116+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.424760+0000 osd.0 (osd.0) 208 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.424760+0000 osd.0 (osd.0) 208 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.896718+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.896718+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.898223+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:18.898223+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: audit 2024-04-26T16:26:19.097965+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: audit 2024-04-26T16:26:19.097965+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:19.265 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: audit 2024-04-26T16:26:19.098114+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:19.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: audit 2024-04-26T16:26:19.098114+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:19.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.102292+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-04-26T16:26:19.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:19 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.102292+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-04-26T16:26:19.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:19.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.093733+0000 osd.5 (osd.5) 199 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.093733+0000 osd.5 (osd.5) 199 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.096036+0000 osd.5 (osd.5) 200 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.096036+0000 osd.5 (osd.5) 200 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.423116+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.423116+0000 osd.0 (osd.0) 207 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.424760+0000 osd.0 (osd.0) 208 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.424760+0000 osd.0 (osd.0) 208 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.896718+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.896718+0000 osd.7 (osd.7) 211 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.898223+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:18.898223+0000 osd.7 (osd.7) 212 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: audit 2024-04-26T16:26:19.097965+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:19.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: audit 2024-04-26T16:26:19.097965+0000 mon.a (mon.0) 2118 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi050-73201-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:19.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: audit 2024-04-26T16:26:19.098114+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:19.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: audit 2024-04-26T16:26:19.098114+0000 mon.a (mon.0) 2119 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi050-73193-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:19.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.102292+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-04-26T16:26:19.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:19 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.102292+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2024-04-26T16:26:19.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:19.401 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:19.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:19.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:19.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:19.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:19.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:19.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:19.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:19.921 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:19.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:19.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:20.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:20.046 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:20.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:20.169 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:20.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:20.182 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:26:20.182 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:26:20.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:20.292 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:20.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:20.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.323572+0000 osd.5 (osd.5) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:20.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.323572+0000 osd.5 (osd.5) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:20.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.323963+0000 osd.5 (osd.5) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:20.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.323963+0000 osd.5 (osd.5) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:20.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.391469+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:26:20.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.391469+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.396154+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.396154+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.578646+0000 mgr.y (mgr.24401) 356 : cluster [DBG] pgmap v451: 212 pgs: 16 unknown, 196 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.578646+0000 mgr.y (mgr.24401) 356 : cluster [DBG] pgmap v451: 212 pgs: 16 unknown, 196 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.869113+0000 osd.7 (osd.7) 213 : cluster [DBG] 3.e scrub starts 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.869113+0000 osd.7 (osd.7) 213 : cluster [DBG] 3.e scrub starts 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.872927+0000 osd.7 (osd.7) 214 : cluster [DBG] 3.e scrub ok 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:19.872927+0000 osd.7 (osd.7) 214 : cluster [DBG] 3.e scrub ok 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:20.109074+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: cluster 2024-04-26T16:26:20.109074+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: audit 2024-04-26T16:26:20.113063+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: audit 2024-04-26T16:26:20.113063+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: audit 2024-04-26T16:26:20.114444+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[21463]: audit 2024-04-26T16:26:20.114444+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.323572+0000 osd.5 (osd.5) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.323572+0000 osd.5 (osd.5) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.323963+0000 osd.5 (osd.5) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.323963+0000 osd.5 (osd.5) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.391469+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.391469+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.396154+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.396154+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.578646+0000 mgr.y (mgr.24401) 356 : cluster [DBG] pgmap v451: 212 pgs: 16 unknown, 196 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:26:20.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.578646+0000 mgr.y (mgr.24401) 356 : cluster [DBG] pgmap v451: 212 pgs: 16 unknown, 196 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.869113+0000 osd.7 (osd.7) 213 : cluster [DBG] 3.e scrub starts 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.869113+0000 osd.7 (osd.7) 213 : cluster [DBG] 3.e scrub starts 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.872927+0000 osd.7 (osd.7) 214 : cluster [DBG] 3.e scrub ok 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:19.872927+0000 osd.7 (osd.7) 214 : cluster [DBG] 3.e scrub ok 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:20.109074+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: cluster 2024-04-26T16:26:20.109074+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: audit 2024-04-26T16:26:20.113063+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: audit 2024-04-26T16:26:20.113063+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: audit 2024-04-26T16:26:20.114444+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:20 smithi050 bash[29234]: audit 2024-04-26T16:26:20.114444+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:20.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:20.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.323572+0000 osd.5 (osd.5) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.323572+0000 osd.5 (osd.5) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.323963+0000 osd.5 (osd.5) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.323963+0000 osd.5 (osd.5) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.391469+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.391469+0000 osd.0 (osd.0) 209 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.396154+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.396154+0000 osd.0 (osd.0) 210 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.578646+0000 mgr.y (mgr.24401) 356 : cluster [DBG] pgmap v451: 212 pgs: 16 unknown, 196 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:26:20.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.578646+0000 mgr.y (mgr.24401) 356 : cluster [DBG] pgmap v451: 212 pgs: 16 unknown, 196 active+clean; 4.6 MiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.869113+0000 osd.7 (osd.7) 213 : cluster [DBG] 3.e scrub starts 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.869113+0000 osd.7 (osd.7) 213 : cluster [DBG] 3.e scrub starts 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.872927+0000 osd.7 (osd.7) 214 : cluster [DBG] 3.e scrub ok 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:19.872927+0000 osd.7 (osd.7) 214 : cluster [DBG] 3.e scrub ok 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:20.109074+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: cluster 2024-04-26T16:26:20.109074+0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: audit 2024-04-26T16:26:20.113063+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: audit 2024-04-26T16:26:20.113063+0000 mon.a (mon.0) 2122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: audit 2024-04-26T16:26:20.114444+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:20 smithi190 bash[23868]: audit 2024-04-26T16:26:20.114444+0000 mon.b (mon.1) 206 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:20.549 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:20.652 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:20.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:20.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:20.816 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:20.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/2525237806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 msgr2=0x7f98dc106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- 172.21.15.50:0/2525237806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc106f80 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f98cc009920 tx=0x7f98cc009c30 comp rx=0 tx=0).stop 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/2525237806 shutdown_connections 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- 172.21.15.50:0/2525237806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 0x7f98dc10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- 172.21.15.50:0/2525237806 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 0x7f98dc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- 172.21.15.50:0/2525237806 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc106f80 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/2525237806 >> 172.21.15.50:0/2525237806 conn(0x7f98dc0792d0 msgr2=0x7f98dc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/2525237806 shutdown_connections 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/2525237806 wait complete. 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 librados: starting msgr at 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 librados: starting objecter 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 Processor -- start 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- start start 2024-04-26T16:26:20.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 librados: setting wanted keys 2024-04-26T16:26:20.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 librados: calling monclient init 2024-04-26T16:26:20.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 0x7f98dc104020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:20.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc104560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 0x7f98dc104aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc111a00 con 0x7f98dc103310 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f98dc111880 con 0x7f98dc1076b0 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f98dc111b80 con 0x7f98dc102960 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc104560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc104560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56354/0 (socket says 172.21.15.50:56354) 2024-04-26T16:26:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 -- 172.21.15.50:0/1572970697 learned_addr learned my addr 172.21.15.50:0/1572970697 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e18c5700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 0x7f98dc104020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e20c6700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 0x7f98dc104aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 -- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 msgr2=0x7f98dc104020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 0x7f98dc104020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 -- 172.21.15.50:0/1572970697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 msgr2=0x7f98dc104aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 0x7f98dc104aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 -- 172.21.15.50:0/1572970697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98dc1a7790 con 0x7f98dc103310 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e20c6700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 0x7f98dc104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:26:20.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e18c5700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 0x7f98dc104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e10c4700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc104560 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc00f950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98d2ffd700 10 client.?.objecter ms_handle_connect 0x7f98dc103310 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98d2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc00b660 con 0x7f98dc103310 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f98cc005040 con 0x7f98dc103310 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98dc1a7a20 con 0x7f98dc103310 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0038c0 con 0x7f98dc103310 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 10 client.30372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98dc1a7e20 con 0x7f98dc103310 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.865+0000 7f98e3b29700 1 librados: init done 2024-04-26T16:26:20.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98dc103a60 con 0x7f98dc103310 2024-04-26T16:26:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f98cc00b800 con 0x7f98dc103310 2024-04-26T16:26:20.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f98c8071f80 0x7f98c8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(312..312 src has 87..312) ==== 10948+0+0 (secure 0 0 0) 0x7f98cc093210 con 0x7f98dc103310 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 10 client.30372.objecter ms_dispatch 0x7f98dc001550 osd_map(312..312 src has 87..312) 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 3 client.30372.objecter handle_osd_map got epochs [312,312] > 0 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 3 client.30372.objecter handle_osd_map decoding full epoch 312 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98e18c5700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f98c8071f80 0x7f98c8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 20 client.30372.objecter dump_active .. 0 homeless 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98d2ffd700 10 client.30372.objecter ms_handle_connect 0x7f98c8071f80 2024-04-26T16:26:20.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.869+0000 7f98e18c5700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f98c8071f80 0x7f98c8074440 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009680 tx=0x7f98d8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:20.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:20.873+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f98cc013030 con 0x7f98dc103310 2024-04-26T16:26:20.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:20.944 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:20.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:21.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-26T16:26:21.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:21.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:21.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.149+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd out", "ids": ["5"]} v 0) -- 0x7f98dc02e490 con 0x7f98dc103310 2024-04-26T16:26:21.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:21.195 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 -- 172.21.15.50:0/2905955956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806bad0 msgr2=0x7f9ef81102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 --2- 172.21.15.50:0/2905955956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806bad0 0x7f9ef81102e0 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000b0a0 tx=0x7f9ef000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 -- 172.21.15.50:0/2905955956 shutdown_connections 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 --2- 172.21.15.50:0/2905955956 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806bad0 0x7f9ef81102e0 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 --2- 172.21.15.50:0/2905955956 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ef81094a0 0x7f9ef8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 --2- 172.21.15.50:0/2905955956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef806b130 0x7f9ef806b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 -- 172.21.15.50:0/2905955956 >> 172.21.15.50:0/2905955956 conn(0x7f9ef80792d0 msgr2=0x7f9ef80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 -- 172.21.15.50:0/2905955956 shutdown_connections 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.213+0000 7f9eff740700 1 -- 172.21.15.50:0/2905955956 wait complete. 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 librados: starting msgr at 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 librados: starting objecter 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 Processor -- start 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 -- start start 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 librados: setting wanted keys 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 librados: calling monclient init 2024-04-26T16:26:21.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806b130 0x7f9ef8198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef806bad0 0x7f9ef81994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ef81094a0 0x7f9ef819d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efd4dc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806b130 0x7f9ef8198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efd4dc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806b130 0x7f9ef8198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56372/0 (socket says 172.21.15.50:56372) 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efd4dc700 1 -- 172.21.15.50:0/801254557 learned_addr learned my addr 172.21.15.50:0/801254557 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef806bad0 0x7f9ef81994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ef81094a0 0x7f9ef819d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef8112ad0 con 0x7f9ef806b130 2024-04-26T16:26:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef8112950 con 0x7f9ef806bad0 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef8112c50 con 0x7f9ef81094a0 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ef81094a0 msgr2=0x7f9ef819d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9ef81094a0 0x7f9ef819d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806b130 msgr2=0x7f9ef8198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806b130 0x7f9ef8198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef819df90 con 0x7f9ef806bad0 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9ef806b130 0x7f9ef8198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef806bad0 0x7f9ef81994c0 secure :-1 s=READY pgs=4134 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009fa0 tx=0x7f9ee800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eee7fc700 10 client.?.objecter ms_handle_connect 0x7f9ef806bad0 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eee7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8013be0 con 0x7f9ef806bad0 2024-04-26T16:26:21.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9ee800ee60 con 0x7f9ef806bad0 2024-04-26T16:26:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef819e280 con 0x7f9ef806bad0 2024-04-26T16:26:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 10 client.43408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee800cd90 con 0x7f9ef806bad0 2024-04-26T16:26:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef8102320 con 0x7f9ef806bad0 2024-04-26T16:26:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eff740700 1 librados: init done 2024-04-26T16:26:21.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f9ed8005100 con 0x7f9ef806bad0 2024-04-26T16:26:21.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.217+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9ee8022020 con 0x7f9ef806bad0 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9ee4071e50 0x7f9ee4074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(313..313 src has 87..313) ==== 10141+0+0 (secure 0 0 0) 0x7f9ee8095a10 con 0x7f9ef806bad0 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 osd_map(313..313 src has 87..313) 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 3 client.43408.objecter handle_osd_map got epochs [313,313] > 0 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 3 client.43408.objecter handle_osd_map decoding full epoch 313 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9ee4071e50 0x7f9ee4074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 20 client.43408.objecter dump_active .. 0 homeless 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v313) ==== 54+0+16 (secure 0 0 0) 0x7f9ee809a220 con 0x7f9ef806bad0 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 1 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f9ed8004730 0x7f9ed8004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 1 2024-04-26T16:26:21.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 0 1 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 4 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 10 client.43408.objecter _send_command 1 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ed8008b70 con 0x7f9ed8004730 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f9ed8004730 0x7f9ed8004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9ee4071e50 0x7f9ee4074310 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4011550 tx=0x7f9ef4009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ee4071e50 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f9ed8004730 0x7f9ed8004b70 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed8004730 2024-04-26T16:26:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed8008b70 con 0x7f9ed8004730 2024-04-26T16:26:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 1: 0 ) 2024-04-26T16:26:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 1 = system:0 2024-04-26T16:26:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 4 2024-04-26T16:26:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.221+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 0 1 2024-04-26T16:26:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 2 2024-04-26T16:26:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 2 2024-04-26T16:26:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed80053e0 osd=0 3 2024-04-26T16:26:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 0 2 2024-04-26T16:26:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 4 2024-04-26T16:26:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 10 client.43408.objecter _send_command 2 2024-04-26T16:26:21.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed80042a0 con 0x7f9ed8004730 2024-04-26T16:26:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed80042a0 con 0x7f9ed8004730 2024-04-26T16:26:21.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 4 2024-04-26T16:26:21.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 0 2 2024-04-26T16:26:21.240 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:26:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 3 2024-04-26T16:26:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f9ed8002750 0x7f9ed8002bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 3 2024-04-26T16:26:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 1 3 2024-04-26T16:26:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 4 2024-04-26T16:26:21.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 10 client.43408.objecter _send_command 3 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f9ed8003110 con 0x7f9ed8002750 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.241+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f9ed8002750 0x7f9ed8002bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed8002750 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f9ed8002750 0x7f9ed8002bd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed8003110 con 0x7f9ed8002750 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 3: 0 ) 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 3 = system:0 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 4 2024-04-26T16:26:21.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.245+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 1 3 2024-04-26T16:26:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 4 2024-04-26T16:26:21.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 4 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed8004530 osd=1 3 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 1 4 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 4 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 10 client.43408.objecter _send_command 4 2024-04-26T16:26:21.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed8003af0 con 0x7f9ed8002750 2024-04-26T16:26:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed8003af0 con 0x7f9ed8002750 2024-04-26T16:26:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 4 2024-04-26T16:26:21.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 1 4 2024-04-26T16:26:21.283 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:26:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 5 2024-04-26T16:26:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f9ed800b240 0x7f9ed800d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 5 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 2 5 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 4 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 10 client.43408.objecter _send_command 5 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f9ed8010520 con 0x7f9ed800b240 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f9ed800b240 0x7f9ed800d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed800b240 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f9ed800b240 0x7f9ed800d630 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed8010520 con 0x7f9ed800b240 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 5: 0 ) 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 5 = system:0 2024-04-26T16:26:21.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 4 2024-04-26T16:26:21.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.285+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 2 5 2024-04-26T16:26:21.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 6 2024-04-26T16:26:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 6 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed8008140 osd=2 3 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 2 6 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 4 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 10 client.43408.objecter _send_command 6 2024-04-26T16:26:21.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed8003630 con 0x7f9ed800b240 2024-04-26T16:26:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed8003630 con 0x7f9ed800b240 2024-04-26T16:26:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 4 2024-04-26T16:26:21.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 2 6 2024-04-26T16:26:21.303 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 7 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f9ed800ecf0 0x7f9ed800f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 7 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 3 7 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 4 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 10 client.43408.objecter _send_command 7 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f9ed80022e0 con 0x7f9ed800ecf0 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f9ed800ecf0 0x7f9ed800f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed800ecf0 2024-04-26T16:26:21.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.305+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f9ed800ecf0 0x7f9ed800f1a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.309+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed80022e0 con 0x7f9ed800ecf0 2024-04-26T16:26:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.309+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 7: 0 ) 2024-04-26T16:26:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.309+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 7 = system:0 2024-04-26T16:26:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.309+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 4 2024-04-26T16:26:21.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.309+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 3 7 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: debug 2024-04-26T16:26:21.149+0000 7f60c8fdd700 -1 mon.a@0(leader).osd e313 definitely_dead 0 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:20.839826+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:20.839826+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:20.841384+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:20.841384+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.104572+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.104572+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.110770+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.110770+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.122308+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.122308+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.123745+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.123745+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.132215+0000 mon.c (mon.2) 507 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.132215+0000 mon.c (mon.2) 507 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.133584+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.133584+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:20.839826+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:20.839826+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:26:21.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:20.841384+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:20.841384+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.104572+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.104572+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.110770+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.110770+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.122308+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.122308+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.123745+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.123745+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.132215+0000 mon.c (mon.2) 507 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.132215+0000 mon.c (mon.2) 507 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.133584+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.133584+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.135610+0000 mon.c (mon.2) 508 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.135610+0000 mon.c (mon.2) 508 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.136167+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.136167+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.154138+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.154138+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.154275+0000 mon.a (mon.0) 2129 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.154275+0000 mon.a (mon.0) 2129 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.226216+0000 mon.b (mon.1) 208 : audit [DBG] from='client.? 172.21.15.50:0/801254557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[29234]: audit 2024-04-26T16:26:21.226216+0000 mon.b (mon.1) 208 : audit [DBG] from='client.? 172.21.15.50:0/801254557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.135610+0000 mon.c (mon.2) 508 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.135610+0000 mon.c (mon.2) 508 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.310 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.136167+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.136167+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.154138+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.154138+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.154275+0000 mon.a (mon.0) 2129 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.154275+0000 mon.a (mon.0) 2129 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.226216+0000 mon.b (mon.1) 208 : audit [DBG] from='client.? 172.21.15.50:0/801254557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:21.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:21 smithi050 bash[21463]: audit 2024-04-26T16:26:21.226216+0000 mon.b (mon.1) 208 : audit [DBG] from='client.? 172.21.15.50:0/801254557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:21.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:21.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 8 2024-04-26T16:26:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 8 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed80038c0 osd=3 3 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 3 8 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 4 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 10 client.43408.objecter _send_command 8 2024-04-26T16:26:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.325+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed8001850 con 0x7f9ed800ecf0 2024-04-26T16:26:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed8001850 con 0x7f9ed800ecf0 2024-04-26T16:26:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 4 2024-04-26T16:26:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 3 8 2024-04-26T16:26:21.327 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:26:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 9 2024-04-26T16:26:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f9ed80019c0 0x7f9ed80146f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 9 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 4 9 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 4 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 10 client.43408.objecter _send_command 9 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f9ed8017550 con 0x7f9ed80019c0 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f9ed80019c0 0x7f9ed80146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed80019c0 2024-04-26T16:26:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.329+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f9ed80019c0 0x7f9ed80146f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed8017550 con 0x7f9ed80019c0 2024-04-26T16:26:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 9: 0 ) 2024-04-26T16:26:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 9 = system:0 2024-04-26T16:26:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 4 2024-04-26T16:26:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 4 9 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- 172.21.15.50:0/322593115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a01094a0 msgr2=0x7f26a0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/322593115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a01094a0 0x7f26a0109880 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7f2690009990 tx=0x7f2690009ca0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- 172.21.15.50:0/322593115 shutdown_connections 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/322593115 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26a006bad0 0x7f26a01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/322593115 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a01094a0 0x7f26a0109880 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/322593115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006b130 0x7f26a006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- 172.21.15.50:0/322593115 >> 172.21.15.50:0/322593115 conn(0x7f26a00792d0 msgr2=0x7f26a00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- 172.21.15.50:0/322593115 shutdown_connections 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- 172.21.15.50:0/322593115 wait complete. 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 librados: starting msgr at 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 librados: starting objecter 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 Processor -- start 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- start start 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 librados: setting wanted keys 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 librados: calling monclient init 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 0x7f26a01023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006bad0 0x7f26a0102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26a01094a0 0x7f26a0102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26a0112a80 con 0x7f26a006b130 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f26a0112900 con 0x7f26a006bad0 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f26a5e4d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f26a0112c00 con 0x7f26a01094a0 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 0x7f26a01023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26a01094a0 0x7f26a0102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 0x7f26a01023f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56378/0 (socket says 172.21.15.50:56378) 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 -- 172.21.15.50:0/518758976 learned_addr learned my addr 172.21.15.50:0/518758976 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269effd700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006bad0 0x7f26a0102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 -- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26a01094a0 msgr2=0x7f26a0102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26a01094a0 0x7f26a0102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 -- 172.21.15.50:0/518758976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006bad0 msgr2=0x7f26a0102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006bad0 0x7f26a0102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269f7fe700 1 -- 172.21.15.50:0/518758976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a01a7650 con 0x7f26a006b130 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.333+0000 7f269effd700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006bad0 0x7f26a0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269f7fe700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 0x7f26a01023f0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f2690000c00 tx=0x7f269000b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 10 client.?.objecter ms_handle_connect 0x7f26a006b130 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690027070 con 0x7f26a006b130 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a01a78e0 con 0x7f26a006b130 2024-04-26T16:26:21.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2690022470 con 0x7f26a006b130 2024-04-26T16:26:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2690022e60 con 0x7f26a006b130 2024-04-26T16:26:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f26a5e4d700 10 client.30384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a01a7d40 con 0x7f26a006b130 2024-04-26T16:26:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f26a5e4d700 1 librados: init done 2024-04-26T16:26:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a006a110 con 0x7f26a006b130 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f269002a070 con 0x7f26a006b130 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2688071f80 0x7f2688074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(313..313 src has 87..313) ==== 10141+0+0 (secure 0 0 0) 0x7f2690092b70 con 0x7f26a006b130 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269effd700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2688071f80 0x7f2688074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 10 client.30384.objecter ms_dispatch 0x7f26a0001550 osd_map(313..313 src has 87..313) 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 3 client.30384.objecter handle_osd_map got epochs [313,313] > 0 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 3 client.30384.objecter handle_osd_map decoding full epoch 313 2024-04-26T16:26:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 20 client.30384.objecter dump_active .. 0 homeless 2024-04-26T16:26:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269effd700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2688071f80 0x7f2688074440 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7f2694005fd0 tx=0x7f2694005e30 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.337+0000 7f269cff9700 10 client.30384.objecter ms_handle_connect 0x7f2688071f80 2024-04-26T16:26:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.341+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f2690013070 con 0x7f26a006b130 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 10 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 10 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed8001850 osd=4 3 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 4 10 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 4 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 10 client.43408.objecter _send_command 10 2024-04-26T16:26:21.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.353+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed800f930 con 0x7f9ed80019c0 2024-04-26T16:26:21.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed800f930 con 0x7f9ed80019c0 2024-04-26T16:26:21.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 4 2024-04-26T16:26:21.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 4 10 2024-04-26T16:26:21.353 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 11 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f9ed800e850 0x7f9ed8015db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 11 2024-04-26T16:26:21.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 5 11 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 4 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 10 client.43408.objecter _send_command 11 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f9ed800f850 con 0x7f9ed800e850 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f9ed800e850 0x7f9ed8015db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed800e850 2024-04-26T16:26:21.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9efccdb700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f9ed800e850 0x7f9ed8015db0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed800f850 con 0x7f9ed800e850 2024-04-26T16:26:21.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 11: 0 ) 2024-04-26T16:26:21.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 11 = system:0 2024-04-26T16:26:21.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 4 2024-04-26T16:26:21.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.357+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 5 11 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 12 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 12 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed800f6e0 osd=5 3 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 5 12 2024-04-26T16:26:21.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 4 2024-04-26T16:26:21.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 10 client.43408.objecter _send_command 12 2024-04-26T16:26:21.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.377+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed800f850 con 0x7f9ed800e850 2024-04-26T16:26:21.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed800f850 con 0x7f9ed800e850 2024-04-26T16:26:21.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 4 2024-04-26T16:26:21.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 5 12 2024-04-26T16:26:21.378 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:26:21.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 13 2024-04-26T16:26:21.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f9ed8016800 0x7f9ed8016c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 13 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 6 13 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 4 2024-04-26T16:26:21.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 10 client.43408.objecter _send_command 13 2024-04-26T16:26:21.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f9ed80172c0 con 0x7f9ed8016800 2024-04-26T16:26:21.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f9ed8016800 0x7f9ed8016c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed8016800 2024-04-26T16:26:21.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.381+0000 7f9efdcdd700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f9ed8016800 0x7f9ed8016c80 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.385+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed80172c0 con 0x7f9ed8016800 2024-04-26T16:26:21.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.385+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 13: 0 ) 2024-04-26T16:26:21.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.385+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 13 = system:0 2024-04-26T16:26:21.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.385+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 4 2024-04-26T16:26:21.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.385+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 6 13 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 14 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 14 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed800f850 osd=6 3 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 6 14 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 4 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 10 client.43408.objecter _send_command 14 2024-04-26T16:26:21.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.401+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed80174d0 con 0x7f9ed8016800 2024-04-26T16:26:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed80174d0 con 0x7f9ed8016800 2024-04-26T16:26:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 4 2024-04-26T16:26:21.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 6 14 2024-04-26T16:26:21.407 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:26:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:21.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 15 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9ed801bc90 0x7f9ed801e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 15 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 7 15 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 4 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 10 client.43408.objecter _send_command 15 2024-04-26T16:26:21.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f9ed8020ee0 con 0x7f9ed801bc90 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.405+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9ed801bc90 0x7f9ed801e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9eee7fc700 10 client.43408.objecter ms_handle_connect 0x7f9ed801bc90 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9efd4dc700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9ed801bc90 0x7f9ed801e080 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ed8020ee0 con 0x7f9ed801bc90 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 15: 0 ) 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 15 = system:0 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 4 2024-04-26T16:26:21.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.409+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 7 15 2024-04-26T16:26:21.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 10 client.43408.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign -1 16 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 20 client.43408.objecter _get_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_remove -1 16 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 20 client.43408.objecter get_session s=0x7f9ed80172c0 osd=7 3 2024-04-26T16:26:21.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 15 client.43408.objecter _session_command_op_assign 7 16 2024-04-26T16:26:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 4 2024-04-26T16:26:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 10 client.43408.objecter _send_command 16 2024-04-26T16:26:21.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.425+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f9ed800fb20 con 0x7f9ed801bc90 2024-04-26T16:26:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9eee7fc700 1 -- 172.21.15.50:0/801254557 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f9ed800fb20 con 0x7f9ed801bc90 2024-04-26T16:26:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9eee7fc700 10 client.43408.objecter ms_dispatch 0x7f9ef8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9eee7fc700 10 client.43408.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9eee7fc700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 4 2024-04-26T16:26:21.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9eee7fc700 15 client.43408.objecter _session_command_op_remove 7 16 2024-04-26T16:26:21.428 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:26:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9ee3fff700 10 librados: watch_flush enter 2024-04-26T16:26:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9ee3fff700 10 librados: watch_flush exit 2024-04-26T16:26:21.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.0 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.429+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f9ed8004730 msgr2=0x7f9ed8004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f9ed8004730 0x7f9ed8004b70 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80053e0 osd=0 1 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.1 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f9ed8002750 msgr2=0x7f9ed8002bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f9ed8002750 0x7f9ed8002bd0 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8004530 osd=1 1 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.2 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f9ed800b240 msgr2=0x7f9ed800d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f9ed800b240 0x7f9ed800d630 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8008140 osd=2 1 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.3 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f9ed800ecf0 msgr2=0x7f9ed800f1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f9ed800ecf0 0x7f9ed800f1a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80038c0 osd=3 1 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.4 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f9ed80019c0 msgr2=0x7f9ed80146f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f9ed80019c0 0x7f9ed80146f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9efdcdd700 1 -- 172.21.15.50:0/801254557 reap_dead start 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9efdcdd700 1 -- 172.21.15.50:0/801254557 reap_dead start 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed8001850 osd=4 1 2024-04-26T16:26:21.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.5 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9efdcdd700 1 -- 172.21.15.50:0/801254557 reap_dead start 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f9ed800e850 msgr2=0x7f9ed8015db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f9ed800e850 0x7f9ed8015db0 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f6e0 osd=5 1 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.6 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f9ed8016800 msgr2=0x7f9ed8016c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f9ed8016800 0x7f9ed8016c80 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed800f850 osd=6 1 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter close_session for osd.7 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9ed801bc90 msgr2=0x7f9ed801e080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f9ed801bc90 0x7f9ed801e080 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter put_session s=0x7f9ed80172c0 osd=7 1 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 20 client.43408.objecter shutdown clearing up homeless session... 2024-04-26T16:26:21.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 10 client.43408.objecter successfully canceled tick 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9ee4071e50 msgr2=0x7f9ee4074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9ee4071e50 0x7f9ee4074310 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4011550 tx=0x7f9ef4009260 comp rx=0 tx=0).stop 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef806bad0 msgr2=0x7f9ef81994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 --2- 172.21.15.50:0/801254557 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9ef806bad0 0x7f9ef81994c0 secure :-1 s=READY pgs=4134 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009fa0 tx=0x7f9ee800ecd0 comp rx=0 tx=0).stop 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9efdcdd700 1 -- 172.21.15.50:0/801254557 reap_dead start 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 shutdown_connections 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 >> 172.21.15.50:0/801254557 conn(0x7f9ef80792d0 msgr2=0x7f9ef8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 shutdown_connections 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 -- 172.21.15.50:0/801254557 wait complete. 2024-04-26T16:26:21.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.433+0000 7f9ee3fff700 1 librados: shutdown 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:26:21.434 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:26:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:21.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:20.839826+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:20.839826+0000 osd.7 (osd.7) 215 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:20.841384+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:20.841384+0000 osd.7 (osd.7) 216 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.104572+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.104572+0000 mon.a (mon.0) 2123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:21.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.110770+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.110770+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.122308+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.122308+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.123745+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.123745+0000 mon.b (mon.1) 207 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.132215+0000 mon.c (mon.2) 507 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.132215+0000 mon.c (mon.2) 507 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.133584+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.133584+0000 mon.a (mon.0) 2126 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.135610+0000 mon.c (mon.2) 508 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.135610+0000 mon.c (mon.2) 508 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.136167+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.136167+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:21.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.154138+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-26T16:26:21.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.154138+0000 mon.a (mon.0) 2128 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-04-26T16:26:21.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.154275+0000 mon.a (mon.0) 2129 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-04-26T16:26:21.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.154275+0000 mon.a (mon.0) 2129 : cluster [INF] Client client.admin marked osd.5 out, while it was still marked up 2024-04-26T16:26:21.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.226216+0000 mon.b (mon.1) 208 : audit [DBG] from='client.? 172.21.15.50:0/801254557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:21.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:21 smithi190 bash[23868]: audit 2024-04-26T16:26:21.226216+0000 mon.b (mon.1) 208 : audit [DBG] from='client.? 172.21.15.50:0/801254557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:21.570 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:21.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:21.577+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f26a01a8000 con 0x7f26a006b130 2024-04-26T16:26:21.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:21.700 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:21.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:21.827 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:21.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:21.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:21.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:22.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:22.089 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:22.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.105+0000 7f98d2ffd700 1 -- 172.21.15.50:0/1572970697 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd out", "ids": ["5"]}]=0 marked out osd.5. v314) ==== 87+0+0 (secure 0 0 0) 0x7f98cc05e620 con 0x7f98dc103310 2024-04-26T16:26:22.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.105+0000 7f269cff9700 1 -- 172.21.15.50:0/518758976 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v314) ==== 101+0+0 (secure 0 0 0) 0x7f269007e0e0 con 0x7f26a006b130 2024-04-26T16:26:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.109+0000 7f98e3b29700 10 librados: watch_flush enter 2024-04-26T16:26:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.109+0000 7f98e3b29700 10 librados: watch_flush exit 2024-04-26T16:26:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.109+0000 7f98e3b29700 20 client.30372.objecter shutdown clearing up homeless session... 2024-04-26T16:26:22.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.109+0000 7f98e3b29700 10 client.30372.objecter successfully canceled tick 2024-04-26T16:26:22.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f98c8071f80 msgr2=0x7f98c8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:22.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f98c8071f80 0x7f98c8074440 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f98d8009680 tx=0x7f98d8009260 comp rx=0 tx=0).stop 2024-04-26T16:26:22.116 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 msgr2=0x7f98dc104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc104560 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc00f950 comp rx=0 tx=0).stop 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 shutdown_connections 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f98c8071f80 0x7f98c8074440 unknown :-1 s=CLOSED pgs=3707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98dc1076b0 0x7f98dc104aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f98dc102960 0x7f98dc104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 --2- 172.21.15.50:0/1572970697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f98dc103310 0x7f98dc104560 secure :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f98cc000c00 tx=0x7f98cc00f950 comp rx=0 tx=0).stop 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 >> 172.21.15.50:0/1572970697 conn(0x7f98dc0792d0 msgr2=0x7f98dc06bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:22.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 shutdown_connections 2024-04-26T16:26:22.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.121+0000 7f98e3b29700 1 -- 172.21.15.50:0/1572970697 wait complete. 2024-04-26T16:26:22.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.125+0000 7f98e3b29700 1 librados: shutdown 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.113+0000 7f26a5e4d700 10 librados: watch_flush enter 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.113+0000 7f26a5e4d700 10 librados: watch_flush exit 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.113+0000 7f26a5e4d700 20 client.30384.objecter shutdown clearing up homeless session... 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.113+0000 7f26a5e4d700 10 client.30384.objecter successfully canceled tick 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2688071f80 msgr2=0x7f2688074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2688071f80 0x7f2688074440 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7f2694005fd0 tx=0x7f2694005e30 comp rx=0 tx=0).stop 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 msgr2=0x7f26a01023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.117+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 0x7f26a01023f0 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7f2690000c00 tx=0x7f269000b6c0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 shutdown_connections 2024-04-26T16:26:22.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2688071f80 0x7f2688074440 unknown :-1 s=CLOSED pgs=3709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f26a01094a0 0x7f26a0102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f26a006b130 0x7f26a01023f0 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 --2- 172.21.15.50:0/518758976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a006bad0 0x7f26a0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 >> 172.21.15.50:0/518758976 conn(0x7f26a00792d0 msgr2=0x7f26a0100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 shutdown_connections 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 -- 172.21.15.50:0/518758976 wait complete. 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:22.129+0000 7f26a5e4d700 1 librados: shutdown 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:26:22.133 INFO:teuthology.orchestra.run.smithi050.stderr:marked out osd.5. 2024-04-26T16:26:22.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:22.210 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:22.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:22.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:22.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.391011+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.391011+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.392646+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.392646+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.401429+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.401429+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.403406+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.403406+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:21.580542+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:21.580542+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.580737+0000 mgr.y (mgr.24401) 357 : cluster [DBG] pgmap v454: 228 pgs: 228 active+clean; 4.6 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-26T16:26:22.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:21.580737+0000 mgr.y (mgr.24401) 357 : cluster [DBG] pgmap v454: 228 pgs: 228 active+clean; 4.6 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.106109+0000 mon.a (mon.0) 2131 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.106109+0000 mon.a (mon.0) 2131 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.108397+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.108397+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.108610+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.108610+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.108828+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:22.450 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.108828+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.109069+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.109069+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.109259+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.109259+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.119220+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.119220+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.122996+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.122996+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.123754+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.123754+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.125547+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.125547+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.128331+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.128331+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.129853+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.129853+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.130597+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:22 smithi190 bash[23868]: audit 2024-04-26T16:26:22.130597+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:22.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.391011+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.391011+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.392646+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.392646+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.401429+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.401429+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.403406+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.403406+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:21.580542+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:26:22.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:21.580542+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.580737+0000 mgr.y (mgr.24401) 357 : cluster [DBG] pgmap v454: 228 pgs: 228 active+clean; 4.6 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:21.580737+0000 mgr.y (mgr.24401) 357 : cluster [DBG] pgmap v454: 228 pgs: 228 active+clean; 4.6 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.106109+0000 mon.a (mon.0) 2131 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.106109+0000 mon.a (mon.0) 2131 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.108397+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.108397+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.108610+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.108610+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.108828+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.108828+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:22.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.109069+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.109069+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.109259+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.109259+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.119220+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.119220+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.122996+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.122996+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.123754+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.123754+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.125547+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.125547+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-04-26T16:26:22.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.128331+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.128331+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.129853+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.129853+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.130597+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[29234]: audit 2024-04-26T16:26:22.130597+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.391011+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.391011+0000 osd.0 (osd.0) 211 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.392646+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.392646+0000 osd.0 (osd.0) 212 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.401429+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.401429+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.403406+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.403406+0000 osd.1 (osd.1) 188 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:21.580542+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:21.580542+0000 mon.a (mon.0) 2130 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.580737+0000 mgr.y (mgr.24401) 357 : cluster [DBG] pgmap v454: 228 pgs: 228 active+clean; 4.6 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:21.580737+0000 mgr.y (mgr.24401) 357 : cluster [DBG] pgmap v454: 228 pgs: 228 active+clean; 4.6 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.106109+0000 mon.a (mon.0) 2131 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:26:22.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.106109+0000 mon.a (mon.0) 2131 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.108397+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.108397+0000 mon.a (mon.0) 2132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.108610+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.108610+0000 mon.a (mon.0) 2133 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.108828+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.108828+0000 mon.a (mon.0) 2134 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.109069+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.109069+0000 mon.a (mon.0) 2135 : audit [INF] from='client.? 172.21.15.50:0/1572970697' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.109259+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.109259+0000 mon.a (mon.0) 2136 : audit [INF] from='client.? 172.21.15.50:0/518758976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.119220+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.119220+0000 mon.b (mon.1) 209 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.122996+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.122996+0000 mon.c (mon.2) 509 : audit [INF] from='client.? 172.21.15.50:0/3594344806' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.123754+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.123754+0000 mon.c (mon.2) 510 : audit [INF] from='client.? 172.21.15.50:0/2207569738' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.125547+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.125547+0000 mon.a (mon.0) 2137 : cluster [DBG] osdmap e314: 8 total, 8 up, 7 in 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.128331+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.128331+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.129853+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.129853+0000 mon.a (mon.0) 2139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.130597+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:22 smithi050 bash[21463]: audit 2024-04-26T16:26:22.130597+0000 mon.a (mon.0) 2140 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]: dispatch 2024-04-26T16:26:22.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:22.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.730 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:22.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.862 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:22.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:22.993 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:22.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:23.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:23.137 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:23.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:23.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:23.264 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:23.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:23.392 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:23.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.164128+0000 osd.3 (osd.3) 221 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.164128+0000 osd.3 (osd.3) 221 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.166033+0000 osd.3 (osd.3) 222 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.166033+0000 osd.3 (osd.3) 222 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.248927+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.248927+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.382629+0000 osd.0 (osd.0) 213 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.382629+0000 osd.0 (osd.0) 213 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.384954+0000 osd.0 (osd.0) 214 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.384954+0000 osd.0 (osd.0) 214 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:26:23.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.425344+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.425344+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.428522+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.428522+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.797586+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.797586+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.799653+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.799653+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.111928+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.111928+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.112204+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:23.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.112204+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:23.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.112455+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:23.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.112455+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:23.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.122377+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-04-26T16:26:23.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.122377+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-04-26T16:26:23.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.127102+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.127102+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.130443+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.130443+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.186518+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.186518+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.188831+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.188831+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.190963+0000 mon.a (mon.0) 2148 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.190963+0000 mon.a (mon.0) 2148 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.194195+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.194195+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.195008+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.195008+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.196182+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.196182+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.196403+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.196403+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.197316+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.197316+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.198161+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.511 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:23 smithi190 bash[23868]: audit 2024-04-26T16:26:23.198161+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.521 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:23.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:23.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.164128+0000 osd.3 (osd.3) 221 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:26:23.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.164128+0000 osd.3 (osd.3) 221 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:26:23.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.166033+0000 osd.3 (osd.3) 222 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:26:23.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.166033+0000 osd.3 (osd.3) 222 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.248927+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.248927+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.382629+0000 osd.0 (osd.0) 213 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.382629+0000 osd.0 (osd.0) 213 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.384954+0000 osd.0 (osd.0) 214 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.384954+0000 osd.0 (osd.0) 214 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.425344+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.425344+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.428522+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.428522+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.797586+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.797586+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:26:23.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.799653+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.799653+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.111928+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.111928+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.112204+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.112204+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.112455+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.112455+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.164128+0000 osd.3 (osd.3) 221 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.164128+0000 osd.3 (osd.3) 221 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.166033+0000 osd.3 (osd.3) 222 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.166033+0000 osd.3 (osd.3) 222 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.248927+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.248927+0000 mon.a (mon.0) 2141 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.382629+0000 osd.0 (osd.0) 213 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.382629+0000 osd.0 (osd.0) 213 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.384954+0000 osd.0 (osd.0) 214 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.384954+0000 osd.0 (osd.0) 214 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.425344+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.425344+0000 osd.1 (osd.1) 189 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.428522+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:26:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.428522+0000 osd.1 (osd.1) 190 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.797586+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.797586+0000 osd.7 (osd.7) 217 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.799653+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.799653+0000 osd.7 (osd.7) 218 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.111928+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.111928+0000 mon.a (mon.0) 2142 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.112204+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.112204+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi050-73193-40"}]': finished 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.112455+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:23.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.112455+0000 mon.a (mon.0) 2144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi050-73201-49"}]': finished 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.122377+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.122377+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.127102+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.127102+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.130443+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.130443+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.186518+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.186518+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.188831+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.188831+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.190963+0000 mon.a (mon.0) 2148 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.190963+0000 mon.a (mon.0) 2148 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.194195+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.194195+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.195008+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.195008+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.122377+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.122377+0000 mon.a (mon.0) 2145 : cluster [DBG] osdmap e315: 8 total, 8 up, 7 in 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.127102+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.127102+0000 mon.b (mon.1) 210 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.130443+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.130443+0000 mon.a (mon.0) 2146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.186518+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.186518+0000 mon.c (mon.2) 511 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.188831+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.188831+0000 mon.a (mon.0) 2147 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.190963+0000 mon.a (mon.0) 2148 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.190963+0000 mon.a (mon.0) 2148 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.194195+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.194195+0000 mon.c (mon.2) 512 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.195008+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.195008+0000 mon.a (mon.0) 2149 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.196182+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.196182+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.196403+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.196403+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.197316+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.197316+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.198161+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[21463]: audit 2024-04-26T16:26:23.198161+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.196182+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.637 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.196182+0000 mon.a (mon.0) 2150 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:23.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.196403+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.196403+0000 mon.c (mon.2) 513 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.197316+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.197316+0000 mon.a (mon.0) 2151 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.198161+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.638 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:23 smithi050 bash[29234]: audit 2024-04-26T16:26:23.198161+0000 mon.a (mon.0) 2152 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:23.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:23.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:23.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:23.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:23.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:24.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:24.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:24.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:24.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:24.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:24.286 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.478684+0000 osd.2 (osd.2) 159 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:26:24.286 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.478684+0000 osd.2 (osd.2) 159 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:26:24.286 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.481085+0000 osd.2 (osd.2) 160 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:26:24.286 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.481085+0000 osd.2 (osd.2) 160 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:26:24.286 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.613015+0000 osd.4 (osd.4) 175 : cluster [DBG] 11.e scrub starts 2024-04-26T16:26:24.286 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.613015+0000 osd.4 (osd.4) 175 : cluster [DBG] 11.e scrub starts 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.615513+0000 osd.4 (osd.4) 176 : cluster [DBG] 11.e scrub ok 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:22.615513+0000 osd.4 (osd.4) 176 : cluster [DBG] 11.e scrub ok 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.581660+0000 mgr.y (mgr.24401) 358 : cluster [DBG] pgmap v457: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.581660+0000 mgr.y (mgr.24401) 358 : cluster [DBG] pgmap v457: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.783880+0000 osd.7 (osd.7) 219 : cluster [DBG] 11.f scrub starts 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.783880+0000 osd.7 (osd.7) 219 : cluster [DBG] 11.f scrub starts 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.786775+0000 osd.7 (osd.7) 220 : cluster [DBG] 11.f scrub ok 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:23.786775+0000 osd.7 (osd.7) 220 : cluster [DBG] 11.f scrub ok 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.112270+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.112270+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:24.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.114131+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]': finished 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.114131+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]': finished 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.114221+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.114221+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.114285+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.114285+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.121680+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.121680+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.123143+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.123143+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.123254+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.123254+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.125121+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[29234]: audit 2024-04-26T16:26:24.125121+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.478684+0000 osd.2 (osd.2) 159 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.478684+0000 osd.2 (osd.2) 159 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.481085+0000 osd.2 (osd.2) 160 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.481085+0000 osd.2 (osd.2) 160 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.613015+0000 osd.4 (osd.4) 175 : cluster [DBG] 11.e scrub starts 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.613015+0000 osd.4 (osd.4) 175 : cluster [DBG] 11.e scrub starts 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.615513+0000 osd.4 (osd.4) 176 : cluster [DBG] 11.e scrub ok 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:22.615513+0000 osd.4 (osd.4) 176 : cluster [DBG] 11.e scrub ok 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.581660+0000 mgr.y (mgr.24401) 358 : cluster [DBG] pgmap v457: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.581660+0000 mgr.y (mgr.24401) 358 : cluster [DBG] pgmap v457: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:24.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.783880+0000 osd.7 (osd.7) 219 : cluster [DBG] 11.f scrub starts 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.783880+0000 osd.7 (osd.7) 219 : cluster [DBG] 11.f scrub starts 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.786775+0000 osd.7 (osd.7) 220 : cluster [DBG] 11.f scrub ok 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:23.786775+0000 osd.7 (osd.7) 220 : cluster [DBG] 11.f scrub ok 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.112270+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.112270+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.114131+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]': finished 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.114131+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]': finished 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.114221+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.114221+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.114285+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.114285+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.121680+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.121680+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-04-26T16:26:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.123143+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.123143+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.123254+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:24.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.123254+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:24.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.125121+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21463]: audit 2024-04-26T16:26:24.125121+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:24.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:24.429 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:24.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.478684+0000 osd.2 (osd.2) 159 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.478684+0000 osd.2 (osd.2) 159 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.481085+0000 osd.2 (osd.2) 160 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.481085+0000 osd.2 (osd.2) 160 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.613015+0000 osd.4 (osd.4) 175 : cluster [DBG] 11.e scrub starts 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.613015+0000 osd.4 (osd.4) 175 : cluster [DBG] 11.e scrub starts 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.615513+0000 osd.4 (osd.4) 176 : cluster [DBG] 11.e scrub ok 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:22.615513+0000 osd.4 (osd.4) 176 : cluster [DBG] 11.e scrub ok 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.581660+0000 mgr.y (mgr.24401) 358 : cluster [DBG] pgmap v457: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.581660+0000 mgr.y (mgr.24401) 358 : cluster [DBG] pgmap v457: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.783880+0000 osd.7 (osd.7) 219 : cluster [DBG] 11.f scrub starts 2024-04-26T16:26:24.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.783880+0000 osd.7 (osd.7) 219 : cluster [DBG] 11.f scrub starts 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.786775+0000 osd.7 (osd.7) 220 : cluster [DBG] 11.f scrub ok 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:23.786775+0000 osd.7 (osd.7) 220 : cluster [DBG] 11.f scrub ok 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.112270+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.112270+0000 mon.a (mon.0) 2153 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.114131+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]': finished 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.114131+0000 mon.a (mon.0) 2154 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-33", "mode": "writeback"}]': finished 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.114221+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.114221+0000 mon.a (mon.0) 2155 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi050-73201-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.114285+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.114285+0000 mon.a (mon.0) 2156 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi050-73193-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.121680+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.121680+0000 mon.a (mon.0) 2157 : cluster [DBG] osdmap e316: 8 total, 8 up, 7 in 2024-04-26T16:26:24.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.123143+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.123143+0000 mon.c (mon.2) 514 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.123254+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:24.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.123254+0000 mon.a (mon.0) 2158 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:24.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.125121+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:24 smithi190 bash[23868]: audit 2024-04-26T16:26:24.125121+0000 mon.a (mon.0) 2159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:24.560 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:24.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:26:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:26:24.628 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:26:24 smithi050 bash[53323]: debug 2024-04-26T16:26:24.493+0000 7fbcab37f700 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94191258311296 err (110) Connection timed out 2024-04-26T16:26:24.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:24.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:24.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:26:24.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:24.820 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:24.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:24.948 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:24.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:24.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:25.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:25.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:25.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:25.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:24.326932+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:24.326932+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:24.329373+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:24.329373+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.481965+0000 osd.1 (osd.1) 191 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.481965+0000 osd.1 (osd.1) 191 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.484592+0000 osd.1 (osd.1) 192 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.484592+0000 osd.1 (osd.1) 192 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:26:25.322 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:25.116829+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:25.116829+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:25.134049+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:25.134049+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.142080+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.142080+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:25.142782+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[29234]: audit 2024-04-26T16:26:25.142782+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:24.326932+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:24.326932+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:24.329373+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:24.329373+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.481965+0000 osd.1 (osd.1) 191 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.481965+0000 osd.1 (osd.1) 191 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.484592+0000 osd.1 (osd.1) 192 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.484592+0000 osd.1 (osd.1) 192 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:25.116829+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:25.116829+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:25.134049+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:25.134049+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.142080+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.142080+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:25.142782+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.324 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:25 smithi050 bash[21463]: audit 2024-04-26T16:26:25.142782+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:25.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:25.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:25.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:24.326932+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:24.326932+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:24.329373+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:24.329373+0000 mon.b (mon.1) 211 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.481965+0000 osd.1 (osd.1) 191 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.481965+0000 osd.1 (osd.1) 191 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.484592+0000 osd.1 (osd.1) 192 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.484592+0000 osd.1 (osd.1) 192 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:25.116829+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:25.116829+0000 mon.a (mon.0) 2161 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:25.134049+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:25.134049+0000 mon.b (mon.1) 212 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.142080+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-04-26T16:26:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.142080+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e317: 8 total, 8 up, 7 in 2024-04-26T16:26:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:25.142782+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:25 smithi190 bash[23868]: audit 2024-04-26T16:26:25.142782+0000 mon.a (mon.0) 2163 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]: dispatch 2024-04-26T16:26:25.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:25.601 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:25.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:25.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:25.811 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:26:25.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:25.858 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:25.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:25.985 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:25.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:26.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:26.129 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:26.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:26.260 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:26.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:26:26.363 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.660990+0000 osd.4 (osd.4) 177 : cluster [DBG] 109.f scrub starts 2024-04-26T16:26:26.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.660990+0000 osd.4 (osd.4) 177 : cluster [DBG] 109.f scrub starts 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.666849+0000 osd.4 (osd.4) 178 : cluster [DBG] 109.f scrub ok 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:24.666849+0000 osd.4 (osd.4) 178 : cluster [DBG] 109.f scrub ok 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.333279+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.333279+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.340563+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.340563+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.505790+0000 mon.c (mon.2) 515 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.505790+0000 mon.c (mon.2) 515 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.506711+0000 mon.c (mon.2) 516 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.506711+0000 mon.c (mon.2) 516 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.506726+0000 mon.a (mon.0) 2164 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.506726+0000 mon.a (mon.0) 2164 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.507258+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.507258+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.507416+0000 mon.c (mon.2) 517 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.507416+0000 mon.c (mon.2) 517 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.507992+0000 mon.a (mon.0) 2166 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.507992+0000 mon.a (mon.0) 2166 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.660990+0000 osd.4 (osd.4) 177 : cluster [DBG] 109.f scrub starts 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.660990+0000 osd.4 (osd.4) 177 : cluster [DBG] 109.f scrub starts 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.666849+0000 osd.4 (osd.4) 178 : cluster [DBG] 109.f scrub ok 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:24.666849+0000 osd.4 (osd.4) 178 : cluster [DBG] 109.f scrub ok 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.333279+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:26:26.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.333279+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.340563+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.340563+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.505790+0000 mon.c (mon.2) 515 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.505790+0000 mon.c (mon.2) 515 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.506711+0000 mon.c (mon.2) 516 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.506711+0000 mon.c (mon.2) 516 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.506726+0000 mon.a (mon.0) 2164 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.506726+0000 mon.a (mon.0) 2164 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.507258+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.507258+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.507416+0000 mon.c (mon.2) 517 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.507416+0000 mon.c (mon.2) 517 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.507992+0000 mon.a (mon.0) 2166 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.377 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.507992+0000 mon.a (mon.0) 2166 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.508071+0000 mon.c (mon.2) 518 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.508071+0000 mon.c (mon.2) 518 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.508636+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.508636+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.508734+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.508734+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.509278+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.509278+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.509386+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.509386+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.509923+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.509923+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.510070+0000 mon.c (mon.2) 521 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.510070+0000 mon.c (mon.2) 521 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.510711+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:25.510711+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.508071+0000 mon.c (mon.2) 518 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.508071+0000 mon.c (mon.2) 518 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.508636+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.508636+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.508734+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.508734+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.509278+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.509278+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.509386+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.509386+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.509923+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.509923+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.510070+0000 mon.c (mon.2) 521 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.510070+0000 mon.c (mon.2) 521 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.510711+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:25.510711+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.582498+0000 mgr.y (mgr.24401) 359 : cluster [DBG] pgmap v460: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.582498+0000 mgr.y (mgr.24401) 359 : cluster [DBG] pgmap v460: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.093445+0000 mon.a (mon.0) 2171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.093445+0000 mon.a (mon.0) 2171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.096286+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.096286+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.096536+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.096536+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.096798+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.096798+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.097047+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.097047+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.097300+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.097300+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-04-26T16:26:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.097559+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]': finished 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.582498+0000 mgr.y (mgr.24401) 359 : cluster [DBG] pgmap v460: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.582498+0000 mgr.y (mgr.24401) 359 : cluster [DBG] pgmap v460: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.093445+0000 mon.a (mon.0) 2171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.093445+0000 mon.a (mon.0) 2171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.096286+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.096286+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:26.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.096536+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.096536+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.096798+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.096798+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.097047+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.097047+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.097300+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.097300+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.097559+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.097559+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098011+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098011+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098275+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098275+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]': finished 2024-04-26T16:26:26.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098495+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098495+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098778+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: audit 2024-04-26T16:26:26.098778+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.108330+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.108330+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.097559+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098011+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098011+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098275+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098275+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098495+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]': finished 2024-04-26T16:26:26.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098495+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]': finished 2024-04-26T16:26:26.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098778+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]': finished 2024-04-26T16:26:26.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: audit 2024-04-26T16:26:26.098778+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]': finished 2024-04-26T16:26:26.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.108330+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-04-26T16:26:26.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:26 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.108330+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-04-26T16:26:26.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:26.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.660990+0000 osd.4 (osd.4) 177 : cluster [DBG] 109.f scrub starts 2024-04-26T16:26:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.660990+0000 osd.4 (osd.4) 177 : cluster [DBG] 109.f scrub starts 2024-04-26T16:26:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.666849+0000 osd.4 (osd.4) 178 : cluster [DBG] 109.f scrub ok 2024-04-26T16:26:26.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:24.666849+0000 osd.4 (osd.4) 178 : cluster [DBG] 109.f scrub ok 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.333279+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.333279+0000 osd.0 (osd.0) 215 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.340563+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.340563+0000 osd.0 (osd.0) 216 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.505790+0000 mon.c (mon.2) 515 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.505790+0000 mon.c (mon.2) 515 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.506711+0000 mon.c (mon.2) 516 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.506711+0000 mon.c (mon.2) 516 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.506726+0000 mon.a (mon.0) 2164 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.506726+0000 mon.a (mon.0) 2164 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.507258+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.507258+0000 mon.a (mon.0) 2165 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-04-26T16:26:26.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.507416+0000 mon.c (mon.2) 517 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.507416+0000 mon.c (mon.2) 517 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.507992+0000 mon.a (mon.0) 2166 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.507992+0000 mon.a (mon.0) 2166 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]: dispatch 2024-04-26T16:26:26.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.508071+0000 mon.c (mon.2) 518 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.508071+0000 mon.c (mon.2) 518 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.507 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.508636+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.508636+0000 mon.a (mon.0) 2167 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.508734+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.508734+0000 mon.c (mon.2) 519 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.509278+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.509278+0000 mon.a (mon.0) 2168 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.509386+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.509386+0000 mon.c (mon.2) 520 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.509923+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.509923+0000 mon.a (mon.0) 2169 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.510070+0000 mon.c (mon.2) 521 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.510070+0000 mon.c (mon.2) 521 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.510711+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.508 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:25.510711+0000 mon.a (mon.0) 2170 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]: dispatch 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.582498+0000 mgr.y (mgr.24401) 359 : cluster [DBG] pgmap v460: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.582498+0000 mgr.y (mgr.24401) 359 : cluster [DBG] pgmap v460: 228 pgs: 228 active+clean; 4.6 MiB data, 544 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.093445+0000 mon.a (mon.0) 2171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.093445+0000 mon.a (mon.0) 2171 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.096286+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.096286+0000 mon.a (mon.0) 2172 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi050-73193-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.096536+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.096536+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi050-73201-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.096798+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.096798+0000 mon.a (mon.0) 2174 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-33"}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.097047+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.097047+0000 mon.a (mon.0) 2175 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.f", "id": [4, 2]}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.097300+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.097300+0000 mon.a (mon.0) 2176 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-04-26T16:26:26.509 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.097559+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.097559+0000 mon.a (mon.0) 2177 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "109.a", "id": [3, 0]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098011+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098011+0000 mon.a (mon.0) 2178 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.3", "id": [7, 2]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098275+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098275+0000 mon.a (mon.0) 2179 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.c", "id": [7, 3]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098495+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098495+0000 mon.a (mon.0) 2180 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.19", "id": [7, 2]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098778+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: audit 2024-04-26T16:26:26.098778+0000 mon.a (mon.0) 2181 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "263.1c", "id": [7, 2]}]': finished 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.108330+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-04-26T16:26:26.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.108330+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e318: 8 total, 8 up, 7 in 2024-04-26T16:26:26.520 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:26.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:26.651 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:26.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:26.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:26.778 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:26.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:26.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:26.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:26.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:27.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:27.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:27.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:26:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:26:27.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:27.165 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:27.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:27.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:27.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:27.428 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:27.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.665641+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.665641+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.665989+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:25.665989+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.366487+0000 osd.0 (osd.0) 217 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.366487+0000 osd.0 (osd.0) 217 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.370722+0000 osd.0 (osd.0) 218 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.370722+0000 osd.0 (osd.0) 218 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:26:27.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: audit 2024-04-26T16:26:26.482334+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: audit 2024-04-26T16:26:26.482334+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: audit 2024-04-26T16:26:26.483680+0000 mon.c (mon.2) 522 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: audit 2024-04-26T16:26:26.483680+0000 mon.c (mon.2) 522 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.489798+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.489798+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.490181+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.490181+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.659170+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.659170+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.661033+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.661033+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.679268+0000 osd.4 (osd.4) 181 : cluster [DBG] 109.2 deep-scrub starts 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.679268+0000 osd.4 (osd.4) 181 : cluster [DBG] 109.2 deep-scrub starts 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.686092+0000 osd.4 (osd.4) 182 : cluster [DBG] 109.2 deep-scrub ok 2024-04-26T16:26:27.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.686092+0000 osd.4 (osd.4) 182 : cluster [DBG] 109.2 deep-scrub ok 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.744585+0000 osd.7 (osd.7) 221 : cluster [DBG] 11.17 deep-scrub starts 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.744585+0000 osd.7 (osd.7) 221 : cluster [DBG] 11.17 deep-scrub starts 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.748266+0000 osd.7 (osd.7) 222 : cluster [DBG] 11.17 deep-scrub ok 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:26.748266+0000 osd.7 (osd.7) 222 : cluster [DBG] 11.17 deep-scrub ok 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.102755+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.102755+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.665641+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.665641+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.665989+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:25.665989+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.366487+0000 osd.0 (osd.0) 217 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.366487+0000 osd.0 (osd.0) 217 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.370722+0000 osd.0 (osd.0) 218 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.370722+0000 osd.0 (osd.0) 218 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: audit 2024-04-26T16:26:26.482334+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: audit 2024-04-26T16:26:26.482334+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: audit 2024-04-26T16:26:26.483680+0000 mon.c (mon.2) 522 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: audit 2024-04-26T16:26:26.483680+0000 mon.c (mon.2) 522 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.489798+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.489798+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.490181+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.490181+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.659170+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.659170+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:26:27.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.661033+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:26:27.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.661033+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.679268+0000 osd.4 (osd.4) 181 : cluster [DBG] 109.2 deep-scrub starts 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.679268+0000 osd.4 (osd.4) 181 : cluster [DBG] 109.2 deep-scrub starts 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.686092+0000 osd.4 (osd.4) 182 : cluster [DBG] 109.2 deep-scrub ok 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.686092+0000 osd.4 (osd.4) 182 : cluster [DBG] 109.2 deep-scrub ok 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.744585+0000 osd.7 (osd.7) 221 : cluster [DBG] 11.17 deep-scrub starts 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.744585+0000 osd.7 (osd.7) 221 : cluster [DBG] 11.17 deep-scrub starts 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.748266+0000 osd.7 (osd.7) 222 : cluster [DBG] 11.17 deep-scrub ok 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:26.748266+0000 osd.7 (osd.7) 222 : cluster [DBG] 11.17 deep-scrub ok 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.102755+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-04-26T16:26:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:27 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.102755+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-04-26T16:26:27.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:27.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:27.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.665641+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.665641+0000 osd.2 (osd.2) 161 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.665989+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:25.665989+0000 osd.2 (osd.2) 162 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.366487+0000 osd.0 (osd.0) 217 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.366487+0000 osd.0 (osd.0) 217 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.370722+0000 osd.0 (osd.0) 218 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.370722+0000 osd.0 (osd.0) 218 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: audit 2024-04-26T16:26:26.482334+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: audit 2024-04-26T16:26:26.482334+0000 mon.a (mon.0) 2183 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: audit 2024-04-26T16:26:26.483680+0000 mon.c (mon.2) 522 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: audit 2024-04-26T16:26:26.483680+0000 mon.c (mon.2) 522 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.489798+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.489798+0000 osd.4 (osd.4) 179 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.490181+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.490181+0000 osd.4 (osd.4) 180 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.659170+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.659170+0000 osd.6 (osd.6) 151 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.661033+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.661033+0000 osd.6 (osd.6) 152 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.679268+0000 osd.4 (osd.4) 181 : cluster [DBG] 109.2 deep-scrub starts 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.679268+0000 osd.4 (osd.4) 181 : cluster [DBG] 109.2 deep-scrub starts 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.686092+0000 osd.4 (osd.4) 182 : cluster [DBG] 109.2 deep-scrub ok 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.686092+0000 osd.4 (osd.4) 182 : cluster [DBG] 109.2 deep-scrub ok 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.744585+0000 osd.7 (osd.7) 221 : cluster [DBG] 11.17 deep-scrub starts 2024-04-26T16:26:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.744585+0000 osd.7 (osd.7) 221 : cluster [DBG] 11.17 deep-scrub starts 2024-04-26T16:26:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.748266+0000 osd.7 (osd.7) 222 : cluster [DBG] 11.17 deep-scrub ok 2024-04-26T16:26:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:26.748266+0000 osd.7 (osd.7) 222 : cluster [DBG] 11.17 deep-scrub ok 2024-04-26T16:26:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.102755+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-04-26T16:26:27.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:27 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.102755+0000 mon.a (mon.0) 2184 : cluster [DBG] osdmap e319: 8 total, 8 up, 7 in 2024-04-26T16:26:27.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:27.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:27.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:27.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:27.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:27.940 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:27.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:28.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:28.072 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:28.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:28.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:28.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:28.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:28.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:28.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:26.762952+0000 mgr.y (mgr.24401) 360 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:28.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:26.762952+0000 mgr.y (mgr.24401) 360 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:28.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.416617+0000 osd.0 (osd.0) 219 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.416617+0000 osd.0 (osd.0) 219 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.420263+0000 osd.0 (osd.0) 220 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.420263+0000 osd.0 (osd.0) 220 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.583175+0000 mgr.y (mgr.24401) 361 : cluster [DBG] pgmap v463: 244 pgs: 7 peering, 16 creating+peering, 221 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 511 B/s wr, 3 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.583175+0000 mgr.y (mgr.24401) 361 : cluster [DBG] pgmap v463: 244 pgs: 7 peering, 16 creating+peering, 221 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 511 B/s wr, 3 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.690785+0000 osd.4 (osd.4) 183 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.690785+0000 osd.4 (osd.4) 183 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.695836+0000 osd.4 (osd.4) 184 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:27.695836+0000 osd.4 (osd.4) 184 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.111906+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-04-26T16:26:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.111906+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-04-26T16:26:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:28.114947+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:28.114947+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:28.119186+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:28.119186+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:28.119992+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:28 smithi190 bash[23868]: audit 2024-04-26T16:26:28.119992+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:28.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:28.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:26.762952+0000 mgr.y (mgr.24401) 360 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:26.762952+0000 mgr.y (mgr.24401) 360 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.416617+0000 osd.0 (osd.0) 219 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.416617+0000 osd.0 (osd.0) 219 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.420263+0000 osd.0 (osd.0) 220 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.420263+0000 osd.0 (osd.0) 220 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.583175+0000 mgr.y (mgr.24401) 361 : cluster [DBG] pgmap v463: 244 pgs: 7 peering, 16 creating+peering, 221 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 511 B/s wr, 3 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.583175+0000 mgr.y (mgr.24401) 361 : cluster [DBG] pgmap v463: 244 pgs: 7 peering, 16 creating+peering, 221 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 511 B/s wr, 3 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.690785+0000 osd.4 (osd.4) 183 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.690785+0000 osd.4 (osd.4) 183 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.695836+0000 osd.4 (osd.4) 184 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:27.695836+0000 osd.4 (osd.4) 184 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:26:28.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.111906+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-04-26T16:26:28.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.111906+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-04-26T16:26:28.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:28.114947+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:28.114947+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:28.119186+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:28.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:28.119186+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:28.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:28.119992+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[21463]: audit 2024-04-26T16:26:28.119992+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:26.762952+0000 mgr.y (mgr.24401) 360 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:28.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:26.762952+0000 mgr.y (mgr.24401) 360 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:28.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.416617+0000 osd.0 (osd.0) 219 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:26:28.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.416617+0000 osd.0 (osd.0) 219 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.420263+0000 osd.0 (osd.0) 220 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.420263+0000 osd.0 (osd.0) 220 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.583175+0000 mgr.y (mgr.24401) 361 : cluster [DBG] pgmap v463: 244 pgs: 7 peering, 16 creating+peering, 221 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 511 B/s wr, 3 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.583175+0000 mgr.y (mgr.24401) 361 : cluster [DBG] pgmap v463: 244 pgs: 7 peering, 16 creating+peering, 221 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 511 B/s wr, 3 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.690785+0000 osd.4 (osd.4) 183 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.690785+0000 osd.4 (osd.4) 183 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.695836+0000 osd.4 (osd.4) 184 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:27.695836+0000 osd.4 (osd.4) 184 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.111906+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-04-26T16:26:28.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.111906+0000 mon.a (mon.0) 2185 : cluster [DBG] osdmap e320: 8 total, 8 up, 7 in 2024-04-26T16:26:28.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:28.114947+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:28.114947+0000 mon.c (mon.2) 523 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:28.119186+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:28.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:28.119186+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:28.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:28.119992+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:28 smithi050 bash[29234]: audit 2024-04-26T16:26:28.119992+0000 mon.a (mon.0) 2187 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:28.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:28.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:28.723 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:28.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:28.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:28.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:28.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:28.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:29.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:29.122 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:29.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:29.252 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:29.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:26:29.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:26:29.291 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-04-26T16:26:29.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump -f json-pretty 2024-04-26T16:26:29.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:29.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:29.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:29.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.295165+0000 mon.a (mon.0) 2188 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.295165+0000 mon.a (mon.0) 2188 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.590433+0000 osd.1 (osd.1) 193 : cluster [DBG] 11.c scrub starts 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.590433+0000 osd.1 (osd.1) 193 : cluster [DBG] 11.c scrub starts 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.591704+0000 osd.1 (osd.1) 194 : cluster [DBG] 11.c scrub ok 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.591704+0000 osd.1 (osd.1) 194 : cluster [DBG] 11.c scrub ok 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.690789+0000 osd.4 (osd.4) 185 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.690789+0000 osd.4 (osd.4) 185 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.702728+0000 osd.4 (osd.4) 186 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:28.702728+0000 osd.4 (osd.4) 186 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.105197+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.105197+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:29.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.105427+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.105427+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.115770+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.115770+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.120341+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.120341+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.127076+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.127076+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.128179+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.128179+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.129656+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.129656+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.130837+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:29 smithi190 bash[23868]: audit 2024-04-26T16:26:29.130837+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:29.542 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.295165+0000 mon.a (mon.0) 2188 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.295165+0000 mon.a (mon.0) 2188 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.590433+0000 osd.1 (osd.1) 193 : cluster [DBG] 11.c scrub starts 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.590433+0000 osd.1 (osd.1) 193 : cluster [DBG] 11.c scrub starts 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.591704+0000 osd.1 (osd.1) 194 : cluster [DBG] 11.c scrub ok 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.591704+0000 osd.1 (osd.1) 194 : cluster [DBG] 11.c scrub ok 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.690789+0000 osd.4 (osd.4) 185 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.690789+0000 osd.4 (osd.4) 185 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.702728+0000 osd.4 (osd.4) 186 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:28.702728+0000 osd.4 (osd.4) 186 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:26:29.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.105197+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.105197+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.105427+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.105427+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.115770+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.115770+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.120341+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.120341+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.127076+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.127076+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.128179+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.128179+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.129656+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.129656+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.130837+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[29234]: audit 2024-04-26T16:26:29.130837+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.295165+0000 mon.a (mon.0) 2188 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.295165+0000 mon.a (mon.0) 2188 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.590433+0000 osd.1 (osd.1) 193 : cluster [DBG] 11.c scrub starts 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.590433+0000 osd.1 (osd.1) 193 : cluster [DBG] 11.c scrub starts 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.591704+0000 osd.1 (osd.1) 194 : cluster [DBG] 11.c scrub ok 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.591704+0000 osd.1 (osd.1) 194 : cluster [DBG] 11.c scrub ok 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.690789+0000 osd.4 (osd.4) 185 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.690789+0000 osd.4 (osd.4) 185 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.702728+0000 osd.4 (osd.4) 186 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:28.702728+0000 osd.4 (osd.4) 186 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.105197+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.105197+0000 mon.a (mon.0) 2189 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.105427+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:29.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.105427+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.115770+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.115770+0000 mon.c (mon.2) 524 : audit [INF] from='client.? 172.21.15.50:0/2547870268' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.120341+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.120341+0000 mon.a (mon.0) 2191 : cluster [DBG] osdmap e321: 8 total, 8 up, 7 in 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.127076+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.127076+0000 mon.b (mon.1) 213 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.128179+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.128179+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.129656+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.129656+0000 mon.a (mon.0) 2193 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.130837+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:29 smithi050 bash[21463]: audit 2024-04-26T16:26:29.130837+0000 mon.a (mon.0) 2194 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:29.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:29.624 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.36826.0:3 158.0 158:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:01.063043+0000", 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 148.64311684200001, 2024-04-26T16:26:29.700 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.096283056000000006, 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.36826", 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/3562244122", 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.701 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063043+0000", 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063043+0000", 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063045+0000", 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3e-06 2024-04-26T16:26:29.702 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063054+0000", 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6750000000000008e-06 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063070+0000", 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6636000000000001e-05 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.703 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063085+0000", 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4589999999999999e-05 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063396+0000", 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031165099999999997 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.063410+0000", 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3519e-05 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.704 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.140929+0000", 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.077519008 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.141073+0000", 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014408800000000001 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,1", 2024-04-26T16:26:29.705 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.141172+0000", 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8696999999999994e-05 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.141953+0000", 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00078132600000000005 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.154747+0000", 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012793848999999999 2024-04-26T16:26:29.706 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.159229+0000", 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044819430000000004 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.159296+0000", 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7267000000000005e-05 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.707 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:01.159326+0000", 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9507000000000001e-05 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.35492.0:2 160.d 160:b3610dda:::oid_RoundTripPP2_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:02.065985+0000", 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 147.640174245, 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.056941762999999999, 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.708 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.35492", 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/438790398", 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.065985+0000", 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.709 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.065985+0000", 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.065987+0000", 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8679999999999999e-06 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.065995+0000", 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7239999999999992e-06 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.710 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.066008+0000", 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3203e-05 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.066020+0000", 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2479e-05 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.066435+0000", 2024-04-26T16:26:29.711 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041500100000000002 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.066446+0000", 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0772e-05 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.097360+0000", 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.030914074999999999 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.712 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.097490+0000", 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013005300000000001 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,7", 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.097640+0000", 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014984800000000001 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.713 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.099201+0000", 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001561024 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.100375+0000", 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001173313 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.122810+0000", 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.022435046 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.714 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.122873+0000", 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3683000000000007e-05 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.122927+0000", 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3674000000000001e-05 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.36826.0:8 158.0 158:0384ab8e:::oid:head [write 128~128] snapc 3=[3,2] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:02.069156+0000", 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 147.63700307400001, 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.055388701999999998, 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.36826", 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/3562244122", 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 8 2024-04-26T16:26:29.716 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069156+0000", 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069156+0000", 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.717 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069158+0000", 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.319e-06 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069162+0000", 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4989999999999996e-06 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069173+0000", 2024-04-26T16:26:29.718 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0672999999999999e-05 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069183+0000", 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0025e-05 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069487+0000", 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030456299999999999 2024-04-26T16:26:29.719 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069625+0000", 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000137195 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,1", 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.069834+0000", 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020985000000000001 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.070419+0000", 2024-04-26T16:26:29.720 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00058452299999999999 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.094442+0000", 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.024022735 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.124397+0000", 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.029954907999999999 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.721 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.124498+0000", 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010122 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:02.124545+0000", 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7191999999999998e-05 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.722 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24770.0:2079 17.1b 17:ddd8974c:test-rados-api-smithi050-73363-17::big2.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:04.413690+0000", 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 145.29246940199999, 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.122283135, 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24770", 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2506934054", 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2079 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.723 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413690+0000", 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413690+0000", 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413693+0000", 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.717e-06 2024-04-26T16:26:29.724 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413705+0000", 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1905999999999999e-05 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413726+0000", 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1708999999999998e-05 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413738+0000", 2024-04-26T16:26:29.725 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1807999999999999e-05 2024-04-26T16:26:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413843+0000", 2024-04-26T16:26:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104503 2024-04-26T16:26:29.726 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.413993+0000", 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000150378 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,3", 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.492942+0000", 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.078949302999999998 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.727 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.500209+0000", 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072664339999999996 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.514382+0000", 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014173187 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.535776+0000", 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021393942999999999 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.728 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.535889+0000", 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112738 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:04.535973+0000", 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4509e-05 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25249.0:260 157.0 157:0291a197:test-rados-api-smithi050-73372-12::baz:2 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e177)", 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:05.217378+0000", 2024-04-26T16:26:29.729 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 144.48878145800001, 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013623880999999999, 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25249", 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 260 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.730 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217378+0000", 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217378+0000", 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217380+0000", 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.933e-06 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.731 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217389+0000", 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2390000000000008e-06 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217409+0000", 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9741999999999998e-05 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217418+0000", 2024-04-26T16:26:29.732 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9979999999999992e-06 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217481+0000", 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3157000000000005e-05 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.217656+0000", 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017495499999999999 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.733 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.221373+0000", 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003717396 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.221506+0000", 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013234499999999999 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.226498+0000", 2024-04-26T16:26:29.734 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049924039999999998 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.226607+0000", 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010933300000000001 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.735 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.230723+0000", 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041151989999999999 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.230807+0000", 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4636999999999996e-05 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.736 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:05.231002+0000", 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019454300000000001 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25249.0:265 157.0 157:00000000:%01:::head [pgnls start_epoch 178] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e178)", 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:06.067844+0000", 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 143.63831573300001, 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020209911000000001, 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.737 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25249", 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 265 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.067844+0000", 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.738 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.067844+0000", 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.067846+0000", 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2730000000000001e-06 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.067854+0000", 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8059999999999995e-06 2024-04-26T16:26:29.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.077581+0000", 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0097269820000000003 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.077599+0000", 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7907e-05 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.740 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.087777+0000", 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010178491 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.087823+0000", 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6088000000000003e-05 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:06.088054+0000", 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023036399999999999 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.741 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(osd.1.0:713 179.0 e189/186 179:0384ab8e:::oid:3 v 189'5, mlcod=187'1)", 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:17.052020+0000", 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 132.654139477, 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.031319935, 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.052020+0000", 2024-04-26T16:26:29.742 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.052020+0000", 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.052021+0000", 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.333e-06 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.743 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.057488+0000", 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005467151 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.057525+0000", 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6424999999999998e-05 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.057548+0000", 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2742e-05 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.744 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.057740+0000", 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000192315 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.057829+0000", 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8986999999999994e-05 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.083256+0000", 2024-04-26T16:26:29.745 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025427077999999999 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:17.083340+0000", 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3904000000000002e-05 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.746 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.40493.0:3 184.0 184:0384ab8e:::oid:head [write 0~128] snapc 2=[2] ondisk+write+ordersnap+known_if_redirected+supports_pool_eio e193)", 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:21.108011+0000", 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 128.59814891100001, 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.040193503999999998, 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.40493", 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/973988771", 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.747 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.108011+0000", 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.108011+0000", 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.748 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.108013+0000", 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1160000000000002e-06 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.108021+0000", 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5259999999999993e-06 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.108038+0000", 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6515e-05 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.749 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.108056+0000", 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.838e-05 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.119095+0000", 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011039269000000001 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.119289+0000", 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019339499999999999 2024-04-26T16:26:29.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 5,7", 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.119503+0000", 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021450399999999999 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.126595+0000", 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070918709999999996 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.751 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.146105+0000", 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.019509782 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.148072+0000", 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001967636 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.148144+0000", 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1211999999999995e-05 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.752 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:21.148204+0000", 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0297999999999998e-05 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.40493.0:5 184.0 184:0384ab8e:::oid:head [write 0~128] snapc 3=[3,2] ondisk+write+ordersnap+known_if_redirected+supports_pool_eio e194)", 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:22.057820+0000", 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 127.64833964499999, 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.054428803999999997, 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.753 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.40493", 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/973988771", 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 5 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.057820+0000", 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.754 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.057820+0000", 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.057823+0000", 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.14e-06 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.057920+0000", 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7489000000000001e-05 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.755 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.057983+0000", 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3100000000000002e-05 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.058010+0000", 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6769000000000001e-05 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.059656+0000", 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001646235 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.756 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.059678+0000", 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1145000000000002e-05 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.070398+0000", 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010720067999999999 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.757 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.070648+0000", 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025007600000000001 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 5,7", 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.071089+0000", 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044109800000000001 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.101682+0000", 2024-04-26T16:26:29.758 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.030592705000000001 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.108175+0000", 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064929569999999997 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.112046+0000", 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038718060000000002 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.759 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.112163+0000", 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116766 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:22.112249+0000", 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5450000000000003e-05 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.40694.0:2 212.1f 212:ff3b4bf2:test-rados-api-smithi050-73363-31::foo:head [write 0~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:24:55.104597+0000", 2024-04-26T16:26:29.760 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 94.601562823999998, 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.027264495, 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.40694", 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2389873209", 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.104597+0000", 2024-04-26T16:26:29.761 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.104597+0000", 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.762 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.104599+0000", 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9999999999999999e-06 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.104610+0000", 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1865000000000001e-05 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.104625+0000", 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4761e-05 2024-04-26T16:26:29.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.104638+0000", 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2408999999999999e-05 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.105525+0000", 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00088785399999999999 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.105532+0000", 2024-04-26T16:26:29.764 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4740000000000002e-06 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.128368+0000", 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.022835774999999999 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.128503+0000", 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013484200000000001 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.128656+0000", 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153661 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.129524+0000", 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00086784000000000004 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.766 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.130049+0000", 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00052508600000000002 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.131799+0000", 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001749795 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.131833+0000", 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4187000000000003e-05 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:24:55.131861+0000", 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7946e-05 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.40727.0:2 218.a 218:53c7e431:::oid_AioUnlockPP_:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:25:02.112715+0000", 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 87.593444003000002, 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.048402162999999998, 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.768 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.40727", 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/3844474308", 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.112715+0000", 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.112715+0000", 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.112718+0000", 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5770000000000001e-06 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.112733+0000", 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5102999999999999e-05 2024-04-26T16:26:29.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.112768+0000", 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4527999999999999e-05 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.112800+0000", 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1885999999999999e-05 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.113037+0000", 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023761400000000001 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.113062+0000", 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4932000000000001e-05 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.151283+0000", 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.038220825999999999 2024-04-26T16:26:29.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.151497+0000", 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021378200000000001 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.151851+0000", 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035468099999999997 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.154131+0000", 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00227954 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.155764+0000", 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00163298 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.161000+0000", 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052362729999999996 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.161064+0000", 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3903000000000001e-05 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:02.161118+0000", 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3538000000000001e-05 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.775 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25249.0:2159 109.f 109:f214ce95:test-rados-api-smithi050-73372-28::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:25:50.105050+0000", 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 39.601109362999999, 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.029538759000000001, 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25249", 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2159 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105050+0000", 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105050+0000", 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105054+0000", 2024-04-26T16:26:29.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.613e-06 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105072+0000", 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8447000000000001e-05 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105103+0000", 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1319000000000003e-05 2024-04-26T16:26:29.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105128+0000", 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4448999999999999e-05 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105726+0000", 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00059817400000000004 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.105736+0000", 2024-04-26T16:26:29.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0020999999999999e-05 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.130994+0000", 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.025258309999999999 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.131114+0000", 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119528 2024-04-26T16:26:29.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,5", 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.131265+0000", 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015060299999999999 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.132366+0000", 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011017959999999999 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.133743+0000", 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001376742 2024-04-26T16:26:29.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.134414+0000", 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067136300000000002 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.134512+0000", 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7554999999999997e-05 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:50.134589+0000", 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6839000000000001e-05 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25249.0:2164 109.f 109:f214ce95:test-rados-api-smithi050-73372-28::foo:head [writefull 0~5] snapc a=[a] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:25:51.093064+0000", 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 38.613094959000001, 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.024453249999999999, 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.783 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25249", 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2164 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093064+0000", 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093064+0000", 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093066+0000", 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.626e-06 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093074+0000", 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9310000000000008e-06 2024-04-26T16:26:29.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093096+0000", 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2450999999999999e-05 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093110+0000", 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3555e-05 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093190+0000", 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9834999999999997e-05 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093329+0000", 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013946999999999999 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,5", 2024-04-26T16:26:29.787 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.093552+0000", 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022292199999999999 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.097396+0000", 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00384411 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.110283+0000", 2024-04-26T16:26:29.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012886470000000001 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.117190+0000", 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069072430000000004 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.117297+0000", 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107067 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.117518+0000", 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022057000000000001 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.40895.0:1 249.6s0 249:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e283)", 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:25:51.111509+0000", 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 38.594650213000001, 2024-04-26T16:26:29.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0313390120000001, 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.40895", 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/3710165660", 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111509+0000", 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111509+0000", 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111511+0000", 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5740000000000001e-06 2024-04-26T16:26:29.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111637+0000", 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012588599999999999 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111647+0000", 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0413e-05 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111656+0000", 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8019999999999998e-06 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111770+0000", 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000114033 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:51.111773+0000", 2024-04-26T16:26:29.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4999999999999999e-06 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.118600+0000", 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0068268760000001 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.118748+0000", 2024-04-26T16:26:29.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000147968 2024-04-26T16:26:29.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.119245+0000", 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049671199999999996 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.136805+0000", 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017559894 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.142679+0000", 2024-04-26T16:26:29.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058743420000000003 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.142848+0000", 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016901199999999999 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(249.6s0 284/283 ECSubWriteReply(tid=1098, last_complete=284'1, committed=1, applied=1))", 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:25:52.121001+0000", 2024-04-26T16:26:29.799 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.585158370999999, 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021402665000000001, 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121001+0000", 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121001+0000", 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121001+0000", 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2300000000000002e-07 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121005+0000", 2024-04-26T16:26:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9299999999999996e-06 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121014+0000", 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2279999999999998e-06 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121025+0000", 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0998999999999999e-05 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.142352+0000", 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021327374999999999 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.142404+0000", 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1709999999999998e-05 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWriteReply(249.6s0 284/283 ECSubWriteReply(tid=1098, last_complete=284'1, committed=1, applied=1))", 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:25:52.121208+0000", 2024-04-26T16:26:29.803 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.584951658000001, 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023379519000000001, 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121208+0000", 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121208+0000", 2024-04-26T16:26:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121209+0000", 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.026e-06 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.121218+0000", 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2220000000000008e-06 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.134672+0000", 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013454068 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.134910+0000", 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023788699999999999 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.142477+0000", 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075672120000000002 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.806 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:25:52.144587+0000", 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021101039999999998 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25249.0:2274 109.e e292/155 109:77740e8c:test-rados-api-smithi050-73372-30::foo:head v 292'25, mlcod=255'23)", 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:26:00.117442+0000", 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 29.588717179, 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.033307639, 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.807 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.117442+0000", 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.117442+0000", 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.117443+0000", 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3000000000000001e-07 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.141621+0000", 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.024178029 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.141646+0000", 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5071000000000001e-05 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.141665+0000", 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9125000000000001e-05 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.141789+0000", 2024-04-26T16:26:29.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012378700000000001 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.141840+0000", 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1578e-05 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.150704+0000", 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008863213 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.811 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:00.150750+0000", 2024-04-26T16:26:29.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6306e-05 2024-04-26T16:26:29.812 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.813 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.813 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.813 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25249.0:2277 109.e e294/155 109:77740e8c:test-rados-api-smithi050-73372-30::foo:head v 294'27, mlcod=258'24)", 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:26:02.117819+0000", 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 27.588340109000001, 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015682819000000001, 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.117819+0000", 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.117819+0000", 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.117821+0000", 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3200000000000001e-06 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.117833+0000", 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1883e-05 2024-04-26T16:26:29.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.117862+0000", 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9019000000000001e-05 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.117884+0000", 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2963e-05 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.118978+0000", 2024-04-26T16:26:29.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010931580000000001 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.119043+0000", 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4978e-05 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.133331+0000", 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01428863 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:02.133502+0000", 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017086799999999999 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.30393.0:1 264.6s0 264:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:26:26.139111+0000", 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.5670484020000002, 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.017813965, 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.818 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.30393", 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/1227594975", 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139111+0000", 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139111+0000", 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139113+0000", 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7129999999999999e-06 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139122+0000", 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1260000000000004e-06 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139141+0000", 2024-04-26T16:26:29.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9187000000000001e-05 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139154+0000", 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3215e-05 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139258+0000", 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104047 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.821 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:26.139262+0000", 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7349999999999998e-06 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.134466+0000", 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99520349699999999 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.134750+0000", 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028421799999999998 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.136422+0000", 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001672605 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.137973+0000", 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015504799999999999 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.156738+0000", 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018765520000000001 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.156925+0000", 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000186622 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25249.0:3008 263.1c 263:38000000::::head [pgnls start_epoch 319] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e319)", 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:26:27.131698+0000", 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 2.5744614540000001, 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.024274161999999998, 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:26:29.824 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25249", 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3008 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.131698+0000", 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.131698+0000", 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:26:29.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.131701+0000", 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5790000000000002e-06 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.131727+0000", 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6397000000000002e-05 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.131752+0000", 2024-04-26T16:26:29.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5545999999999999e-05 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.131778+0000", 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6009000000000001e-05 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.134864+0000", 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030852760000000001 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:26:29.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.134922+0000", 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7983999999999999e-05 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.155734+0000", 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.020812466000000002 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.155790+0000", 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6081000000000001e-05 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:26:29.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:26:27.155972+0000", 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000181824 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:26:29.829 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:26:29.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:29.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:29.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:29.988 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:29.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:30.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:30.122 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:30.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:30.252 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:30.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:30.382 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:30.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-26T16:26:30.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:30.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.374719+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.374719+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.375004+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.375004+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.439395+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.439395+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.440595+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.440595+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.583635+0000 mgr.y (mgr.24401) 362 : cluster [DBG] pgmap v466: 228 pgs: 32 unknown, 3 peering, 193 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.583635+0000 mgr.y (mgr.24401) 362 : cluster [DBG] pgmap v466: 228 pgs: 32 unknown, 3 peering, 193 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.668546+0000 osd.7 (osd.7) 223 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:26:30.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.668546+0000 osd.7 (osd.7) 223 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.671142+0000 osd.7 (osd.7) 224 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.671142+0000 osd.7 (osd.7) 224 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.672104+0000 osd.4 (osd.4) 187 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.672104+0000 osd.4 (osd.4) 187 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.674736+0000 osd.4 (osd.4) 188 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.674736+0000 osd.4 (osd.4) 188 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.108725+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.108725+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.108827+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.108827+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.108967+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.108967+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.113443+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-04-26T16:26:30.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.113443+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.125247+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.125247+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.125540+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.125540+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.164036+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.164036+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.164973+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.164973+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.166118+0000 mon.c (mon.2) 526 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.166118+0000 mon.c (mon.2) 526 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.172625+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.172625+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.492 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.173304+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.173304+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.173626+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:30 smithi190 bash[23868]: audit 2024-04-26T16:26:30.173626+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:30.569 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.374719+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.374719+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.375004+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.375004+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.439395+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.439395+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.440595+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:26:30.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.440595+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.583635+0000 mgr.y (mgr.24401) 362 : cluster [DBG] pgmap v466: 228 pgs: 32 unknown, 3 peering, 193 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.583635+0000 mgr.y (mgr.24401) 362 : cluster [DBG] pgmap v466: 228 pgs: 32 unknown, 3 peering, 193 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.668546+0000 osd.7 (osd.7) 223 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.668546+0000 osd.7 (osd.7) 223 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.671142+0000 osd.7 (osd.7) 224 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.671142+0000 osd.7 (osd.7) 224 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.672104+0000 osd.4 (osd.4) 187 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.672104+0000 osd.4 (osd.4) 187 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.674736+0000 osd.4 (osd.4) 188 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.674736+0000 osd.4 (osd.4) 188 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.108725+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.108725+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:30.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.108827+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.108827+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.108967+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.108967+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.113443+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.113443+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.125247+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.125247+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.125540+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.125540+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.164036+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.164036+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.164973+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.164973+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.166118+0000 mon.c (mon.2) 526 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.166118+0000 mon.c (mon.2) 526 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.172625+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.172625+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.173304+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.173304+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.173626+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[21463]: audit 2024-04-26T16:26:30.173626+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.374719+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.374719+0000 osd.0 (osd.0) 221 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.375004+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.375004+0000 osd.0 (osd.0) 222 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.439395+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.439395+0000 osd.0 (osd.0) 223 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.440595+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.440595+0000 osd.0 (osd.0) 224 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.583635+0000 mgr.y (mgr.24401) 362 : cluster [DBG] pgmap v466: 228 pgs: 32 unknown, 3 peering, 193 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.583635+0000 mgr.y (mgr.24401) 362 : cluster [DBG] pgmap v466: 228 pgs: 32 unknown, 3 peering, 193 active+clean; 4.6 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s; 1.0 MiB/s, 70 objects/s recovering 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.668546+0000 osd.7 (osd.7) 223 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.668546+0000 osd.7 (osd.7) 223 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.671142+0000 osd.7 (osd.7) 224 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.671142+0000 osd.7 (osd.7) 224 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.672104+0000 osd.4 (osd.4) 187 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.672104+0000 osd.4 (osd.4) 187 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.674736+0000 osd.4 (osd.4) 188 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:26:30.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.674736+0000 osd.4 (osd.4) 188 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.108725+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.108725+0000 mon.a (mon.0) 2195 : audit [INF] from='client.? 172.21.15.50:0/1227594975' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi050-73193-41"}]': finished 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.108827+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.108827+0000 mon.a (mon.0) 2196 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi050-73201-50"}]': finished 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.108967+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.108967+0000 mon.a (mon.0) 2197 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.113443+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.113443+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e322: 8 total, 8 up, 7 in 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.125247+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.125247+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.125540+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.125540+0000 mon.b (mon.1) 214 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.164036+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.164036+0000 mon.c (mon.2) 525 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.164973+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.164973+0000 mon.a (mon.0) 2200 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.166118+0000 mon.c (mon.2) 526 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.166118+0000 mon.c (mon.2) 526 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.172625+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.172625+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.173304+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.173304+0000 mon.c (mon.2) 527 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.602 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.173626+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.602 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:30 smithi050 bash[29234]: audit 2024-04-26T16:26:30.173626+0000 mon.a (mon.0) 2202 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:30.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:30.643 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:30.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:30.774 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:30.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:30.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:30.905 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:30.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:31.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:31.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:31.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:31.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:31.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:31.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:31.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:31.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:31.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:31.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:31.554 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:31.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.693612+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:31.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.693612+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:31.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.693967+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:31.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:29.693967+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.160700+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.160700+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.163964+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.163964+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.471824+0000 osd.0 (osd.0) 225 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.471824+0000 osd.0 (osd.0) 225 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.474043+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.474043+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.111146+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.111146+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.111227+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.111227+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.116469+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.116469+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.116575+0000 mon.c (mon.2) 528 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.116575+0000 mon.c (mon.2) 528 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.120266+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.120266+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.122112+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.122112+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.124074+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.50:0/1471852831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.124074+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.50:0/1471852831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.126282+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.126282+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.127920+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:31 smithi190 bash[23868]: audit 2024-04-26T16:26:31.127920+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.693612+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.693612+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.693967+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:29.693967+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.160700+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.160700+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.163964+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.163964+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.471824+0000 osd.0 (osd.0) 225 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.471824+0000 osd.0 (osd.0) 225 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.474043+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.474043+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:26:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.111146+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.111146+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.111227+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.111227+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.116469+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.116469+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.116575+0000 mon.c (mon.2) 528 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.116575+0000 mon.c (mon.2) 528 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.693612+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.693612+0000 osd.1 (osd.1) 195 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.693967+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:29.693967+0000 osd.1 (osd.1) 196 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.160700+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.160700+0000 osd.3 (osd.3) 223 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.163964+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.163964+0000 osd.3 (osd.3) 224 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.471824+0000 osd.0 (osd.0) 225 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.471824+0000 osd.0 (osd.0) 225 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.474043+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.474043+0000 osd.0 (osd.0) 226 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.111146+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.111146+0000 mon.a (mon.0) 2203 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.111227+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.111227+0000 mon.a (mon.0) 2204 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi050-73193-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.116469+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.116469+0000 mon.a (mon.0) 2205 : cluster [DBG] osdmap e323: 8 total, 8 up, 7 in 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.116575+0000 mon.c (mon.2) 528 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.116575+0000 mon.c (mon.2) 528 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.120266+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.120266+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.122112+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.122112+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.124074+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.50:0/1471852831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.124074+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.50:0/1471852831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.126282+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.126282+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.127920+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[21463]: audit 2024-04-26T16:26:31.127920+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.120266+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.120266+0000 mon.a (mon.0) 2206 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.122112+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.122112+0000 mon.b (mon.1) 215 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.124074+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.50:0/1471852831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.124074+0000 mon.b (mon.1) 216 : audit [INF] from='client.? 172.21.15.50:0/1471852831' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.126282+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.126282+0000 mon.a (mon.0) 2207 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.127920+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:31 smithi050 bash[29234]: audit 2024-04-26T16:26:31.127920+0000 mon.a (mon.0) 2208 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:31.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:31.685 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:31.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:31.816 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:31.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:31.948 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:31.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:32.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:32.077 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:32.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:32.203 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:32.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:32.334 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:32.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:32.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:32.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.421396+0000 osd.2 (osd.2) 163 : cluster [DBG] 109.1a deep-scrub starts 2024-04-26T16:26:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.421396+0000 osd.2 (osd.2) 163 : cluster [DBG] 109.1a deep-scrub starts 2024-04-26T16:26:32.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.427675+0000 osd.2 (osd.2) 164 : cluster [DBG] 109.1a deep-scrub ok 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:30.427675+0000 osd.2 (osd.2) 164 : cluster [DBG] 109.1a deep-scrub ok 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.500044+0000 osd.0 (osd.0) 227 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.500044+0000 osd.0 (osd.0) 227 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.502654+0000 osd.0 (osd.0) 228 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.502654+0000 osd.0 (osd.0) 228 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.585440+0000 mgr.y (mgr.24401) 363 : cluster [DBG] pgmap v469: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.585440+0000 mgr.y (mgr.24401) 363 : cluster [DBG] pgmap v469: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.664890+0000 osd.1 (osd.1) 197 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.664890+0000 osd.1 (osd.1) 197 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.667599+0000 osd.1 (osd.1) 198 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.667599+0000 osd.1 (osd.1) 198 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.671855+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.671855+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:26:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.673764+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.673764+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.734689+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.734689+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.736388+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:31.736388+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.115447+0000 mon.a (mon.0) 2209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.115447+0000 mon.a (mon.0) 2209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.115607+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.115607+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.124287+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.124287+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.146488+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.146488+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.148633+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:32 smithi190 bash[23868]: audit 2024-04-26T16:26:32.148633+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.601 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.421396+0000 osd.2 (osd.2) 163 : cluster [DBG] 109.1a deep-scrub starts 2024-04-26T16:26:32.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.421396+0000 osd.2 (osd.2) 163 : cluster [DBG] 109.1a deep-scrub starts 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.427675+0000 osd.2 (osd.2) 164 : cluster [DBG] 109.1a deep-scrub ok 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:30.427675+0000 osd.2 (osd.2) 164 : cluster [DBG] 109.1a deep-scrub ok 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.500044+0000 osd.0 (osd.0) 227 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.500044+0000 osd.0 (osd.0) 227 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.502654+0000 osd.0 (osd.0) 228 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.502654+0000 osd.0 (osd.0) 228 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.585440+0000 mgr.y (mgr.24401) 363 : cluster [DBG] pgmap v469: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.585440+0000 mgr.y (mgr.24401) 363 : cluster [DBG] pgmap v469: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.664890+0000 osd.1 (osd.1) 197 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.664890+0000 osd.1 (osd.1) 197 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.667599+0000 osd.1 (osd.1) 198 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.667599+0000 osd.1 (osd.1) 198 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:26:32.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.671855+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.671855+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.673764+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.673764+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.734689+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.734689+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.736388+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:31.736388+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.115447+0000 mon.a (mon.0) 2209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.115447+0000 mon.a (mon.0) 2209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.115607+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.421396+0000 osd.2 (osd.2) 163 : cluster [DBG] 109.1a deep-scrub starts 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.421396+0000 osd.2 (osd.2) 163 : cluster [DBG] 109.1a deep-scrub starts 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.427675+0000 osd.2 (osd.2) 164 : cluster [DBG] 109.1a deep-scrub ok 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:30.427675+0000 osd.2 (osd.2) 164 : cluster [DBG] 109.1a deep-scrub ok 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.500044+0000 osd.0 (osd.0) 227 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.500044+0000 osd.0 (osd.0) 227 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.502654+0000 osd.0 (osd.0) 228 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.502654+0000 osd.0 (osd.0) 228 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.585440+0000 mgr.y (mgr.24401) 363 : cluster [DBG] pgmap v469: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.585440+0000 mgr.y (mgr.24401) 363 : cluster [DBG] pgmap v469: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.664890+0000 osd.1 (osd.1) 197 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.664890+0000 osd.1 (osd.1) 197 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.667599+0000 osd.1 (osd.1) 198 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.667599+0000 osd.1 (osd.1) 198 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:26:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.671855+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.671855+0000 osd.7 (osd.7) 225 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.673764+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.673764+0000 osd.7 (osd.7) 226 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.734689+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.734689+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.736388+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:31.736388+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.115447+0000 mon.a (mon.0) 2209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.115447+0000 mon.a (mon.0) 2209 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.115607+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.115607+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.124287+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.124287+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.146488+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.146488+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.148633+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[29234]: audit 2024-04-26T16:26:32.148633+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.115607+0000 mon.a (mon.0) 2210 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi050-73201-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.124287+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.124287+0000 mon.b (mon.1) 217 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.146488+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-04-26T16:26:32.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.146488+0000 mon.a (mon.0) 2211 : cluster [DBG] osdmap e324: 8 total, 8 up, 7 in 2024-04-26T16:26:32.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.148633+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:32 smithi050 bash[21463]: audit 2024-04-26T16:26:32.148633+0000 mon.a (mon.0) 2212 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]: dispatch 2024-04-26T16:26:32.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:32.724 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:32.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:32.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:32.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:32.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:32.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:33.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:33.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:33.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:33.233 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:33.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:33.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.203311+0000 osd.3 (osd.3) 225 : cluster [DBG] 109.e scrub starts 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.203311+0000 osd.3 (osd.3) 225 : cluster [DBG] 109.e scrub starts 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.205289+0000 osd.3 (osd.3) 226 : cluster [DBG] 109.e scrub ok 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.205289+0000 osd.3 (osd.3) 226 : cluster [DBG] 109.e scrub ok 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.464964+0000 osd.0 (osd.0) 229 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.464964+0000 osd.0 (osd.0) 229 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.466930+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.466930+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.613630+0000 osd.4 (osd.4) 189 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.613630+0000 osd.4 (osd.4) 189 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.616895+0000 osd.4 (osd.4) 190 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:26:33.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.616895+0000 osd.4 (osd.4) 190 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:26:33.358 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:33.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:33.490 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.203311+0000 osd.3 (osd.3) 225 : cluster [DBG] 109.e scrub starts 2024-04-26T16:26:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.203311+0000 osd.3 (osd.3) 225 : cluster [DBG] 109.e scrub starts 2024-04-26T16:26:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.205289+0000 osd.3 (osd.3) 226 : cluster [DBG] 109.e scrub ok 2024-04-26T16:26:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.205289+0000 osd.3 (osd.3) 226 : cluster [DBG] 109.e scrub ok 2024-04-26T16:26:33.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.464964+0000 osd.0 (osd.0) 229 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.464964+0000 osd.0 (osd.0) 229 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.466930+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.466930+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.613630+0000 osd.4 (osd.4) 189 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.613630+0000 osd.4 (osd.4) 189 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.616895+0000 osd.4 (osd.4) 190 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.616895+0000 osd.4 (osd.4) 190 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.115778+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.115778+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.117647+0000 mon.a (mon.0) 2214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.117647+0000 mon.a (mon.0) 2214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.117731+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]': finished 2024-04-26T16:26:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.117731+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]': finished 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.121852+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.121852+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.154348+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.154348+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.154664+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.154664+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.155439+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.155439+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.155710+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.155710+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.156162+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.156162+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.156423+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:33 smithi190 bash[23868]: audit 2024-04-26T16:26:33.156423+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:33.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.115778+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.115778+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.117647+0000 mon.a (mon.0) 2214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.117647+0000 mon.a (mon.0) 2214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.117731+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]': finished 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.117731+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]': finished 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.121852+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.121852+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.154348+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.154348+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.154664+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.154664+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.155439+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.155439+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.155710+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.155710+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.156162+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.156162+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.156423+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[21463]: audit 2024-04-26T16:26:33.156423+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.203311+0000 osd.3 (osd.3) 225 : cluster [DBG] 109.e scrub starts 2024-04-26T16:26:33.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.203311+0000 osd.3 (osd.3) 225 : cluster [DBG] 109.e scrub starts 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.205289+0000 osd.3 (osd.3) 226 : cluster [DBG] 109.e scrub ok 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.205289+0000 osd.3 (osd.3) 226 : cluster [DBG] 109.e scrub ok 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.464964+0000 osd.0 (osd.0) 229 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.464964+0000 osd.0 (osd.0) 229 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.466930+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.466930+0000 osd.0 (osd.0) 230 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.613630+0000 osd.4 (osd.4) 189 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.613630+0000 osd.4 (osd.4) 189 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.616895+0000 osd.4 (osd.4) 190 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.616895+0000 osd.4 (osd.4) 190 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.115778+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.115778+0000 mon.a (mon.0) 2213 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.117647+0000 mon.a (mon.0) 2214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:33.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.117647+0000 mon.a (mon.0) 2214 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi050-73193-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.117731+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]': finished 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.117731+0000 mon.a (mon.0) 2215 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-35", "mode": "writeback"}]': finished 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.121852+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.121852+0000 mon.a (mon.0) 2216 : cluster [DBG] osdmap e325: 8 total, 8 up, 7 in 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.154348+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.154348+0000 mon.c (mon.2) 529 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.154664+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.154664+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.155439+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.155439+0000 mon.c (mon.2) 530 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.155710+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.155710+0000 mon.a (mon.0) 2218 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.156162+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.156162+0000 mon.c (mon.2) 531 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.156423+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:33 smithi050 bash[29234]: audit 2024-04-26T16:26:33.156423+0000 mon.a (mon.0) 2219 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:33.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:33.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:33.751 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:33.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:33.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:33.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:33.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:33.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:34.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:34.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:34.152 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:34.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:34.280 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:34.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:34.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:34.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.422376+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.c scrub starts 2024-04-26T16:26:34.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.422376+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.c scrub starts 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.424624+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.c scrub ok 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.424624+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.c scrub ok 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.707978+0000 osd.1 (osd.1) 199 : cluster [DBG] 109.b scrub starts 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.707978+0000 osd.1 (osd.1) 199 : cluster [DBG] 109.b scrub starts 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.710497+0000 osd.1 (osd.1) 200 : cluster [DBG] 109.b scrub ok 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:32.710497+0000 osd.1 (osd.1) 200 : cluster [DBG] 109.b scrub ok 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.586136+0000 mgr.y (mgr.24401) 364 : cluster [DBG] pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.586136+0000 mgr.y (mgr.24401) 364 : cluster [DBG] pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.751372+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.751372+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.753064+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.753064+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:26:34.396 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: audit 2024-04-26T16:26:34.121191+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: audit 2024-04-26T16:26:34.121191+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.130671+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.130671+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: audit 2024-04-26T16:26:34.131733+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: audit 2024-04-26T16:26:34.131733+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: audit 2024-04-26T16:26:34.135568+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:34 smithi190 bash[23868]: audit 2024-04-26T16:26:34.135568+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.410 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:34.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:34.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:34.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:34.669 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:34.742 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:26:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.422376+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.c scrub starts 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.422376+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.c scrub starts 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.424624+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.c scrub ok 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.424624+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.c scrub ok 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.707978+0000 osd.1 (osd.1) 199 : cluster [DBG] 109.b scrub starts 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.707978+0000 osd.1 (osd.1) 199 : cluster [DBG] 109.b scrub starts 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.710497+0000 osd.1 (osd.1) 200 : cluster [DBG] 109.b scrub ok 2024-04-26T16:26:34.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:32.710497+0000 osd.1 (osd.1) 200 : cluster [DBG] 109.b scrub ok 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.586136+0000 mgr.y (mgr.24401) 364 : cluster [DBG] pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.586136+0000 mgr.y (mgr.24401) 364 : cluster [DBG] pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.751372+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.751372+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.753064+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.753064+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: audit 2024-04-26T16:26:34.121191+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: audit 2024-04-26T16:26:34.121191+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.130671+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.130671+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: audit 2024-04-26T16:26:34.131733+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: audit 2024-04-26T16:26:34.131733+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: audit 2024-04-26T16:26:34.135568+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[21463]: audit 2024-04-26T16:26:34.135568+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.422376+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.c scrub starts 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.422376+0000 osd.2 (osd.2) 165 : cluster [DBG] 2.c scrub starts 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.424624+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.c scrub ok 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.424624+0000 osd.2 (osd.2) 166 : cluster [DBG] 2.c scrub ok 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.707978+0000 osd.1 (osd.1) 199 : cluster [DBG] 109.b scrub starts 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.707978+0000 osd.1 (osd.1) 199 : cluster [DBG] 109.b scrub starts 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.710497+0000 osd.1 (osd.1) 200 : cluster [DBG] 109.b scrub ok 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:32.710497+0000 osd.1 (osd.1) 200 : cluster [DBG] 109.b scrub ok 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.586136+0000 mgr.y (mgr.24401) 364 : cluster [DBG] pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.586136+0000 mgr.y (mgr.24401) 364 : cluster [DBG] pgmap v472: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 60 B/s, 5 objects/s recovering 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.751372+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.751372+0000 osd.6 (osd.6) 155 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:26:34.745 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.753064+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.753064+0000 osd.6 (osd.6) 156 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: audit 2024-04-26T16:26:34.121191+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: audit 2024-04-26T16:26:34.121191+0000 mon.a (mon.0) 2220 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.130671+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.130671+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e326: 8 total, 8 up, 7 in 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: audit 2024-04-26T16:26:34.131733+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: audit 2024-04-26T16:26:34.131733+0000 mon.c (mon.2) 532 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: audit 2024-04-26T16:26:34.135568+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.746 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:34 smithi050 bash[29234]: audit 2024-04-26T16:26:34.135568+0000 mon.a (mon.0) 2222 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:34.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:34.800 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:34.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:34.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:35.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:35.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:35.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:35.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:35.278 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:26:35.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:35.316 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:35.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:35.443 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:35.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.396791+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.a scrub starts 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.396791+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.a scrub starts 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.398380+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.a scrub ok 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:33.398380+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.a scrub ok 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.208237+0000 osd.3 (osd.3) 227 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.208237+0000 osd.3 (osd.3) 227 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.213798+0000 osd.3 (osd.3) 228 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.213798+0000 osd.3 (osd.3) 228 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.148515+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-04-26T16:26:35.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.148515+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.150295+0000 mon.c (mon.2) 533 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.150295+0000 mon.c (mon.2) 533 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.151224+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.151224+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.243750+0000 mon.a (mon.0) 2225 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.243750+0000 mon.a (mon.0) 2225 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.246058+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[21463]: audit 2024-04-26T16:26:35.246058+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.396791+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.a scrub starts 2024-04-26T16:26:35.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.396791+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.a scrub starts 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.398380+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.a scrub ok 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:33.398380+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.a scrub ok 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.208237+0000 osd.3 (osd.3) 227 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.208237+0000 osd.3 (osd.3) 227 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.213798+0000 osd.3 (osd.3) 228 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.213798+0000 osd.3 (osd.3) 228 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.148515+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.148515+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.150295+0000 mon.c (mon.2) 533 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.150295+0000 mon.c (mon.2) 533 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.151224+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.151224+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.243750+0000 mon.a (mon.0) 2225 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.243750+0000 mon.a (mon.0) 2225 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.246058+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:35 smithi050 bash[29234]: audit 2024-04-26T16:26:35.246058+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.396791+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.a scrub starts 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.396791+0000 osd.2 (osd.2) 167 : cluster [DBG] 5.a scrub starts 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.398380+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.a scrub ok 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:33.398380+0000 osd.2 (osd.2) 168 : cluster [DBG] 5.a scrub ok 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.208237+0000 osd.3 (osd.3) 227 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.208237+0000 osd.3 (osd.3) 227 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.213798+0000 osd.3 (osd.3) 228 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.213798+0000 osd.3 (osd.3) 228 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.148515+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.148515+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e327: 8 total, 8 up, 7 in 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.150295+0000 mon.c (mon.2) 533 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.150295+0000 mon.c (mon.2) 533 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.151224+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.151224+0000 mon.a (mon.0) 2224 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.243750+0000 mon.a (mon.0) 2225 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.243750+0000 mon.a (mon.0) 2225 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.246058+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:35 smithi190 bash[23868]: audit 2024-04-26T16:26:35.246058+0000 mon.b (mon.1) 218 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:35.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:35.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:35.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:35.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:35.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:35.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:35.970 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:35.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:36.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:36.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:36.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:36.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:36.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:36.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.424355+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 deep-scrub starts 2024-04-26T16:26:36.374 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:36.467 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:36.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:36.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:36.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:36.614 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.424355+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 deep-scrub starts 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.424355+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 deep-scrub starts 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.426465+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 deep-scrub ok 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.426465+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 deep-scrub ok 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.695065+0000 osd.1 (osd.1) 201 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.695065+0000 osd.1 (osd.1) 201 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.699205+0000 osd.1 (osd.1) 202 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:34.699205+0000 osd.1 (osd.1) 202 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.235765+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.b scrub starts 2024-04-26T16:26:36.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.235765+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.b scrub starts 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.237926+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.b scrub ok 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.237926+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.b scrub ok 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.435583+0000 osd.0 (osd.0) 231 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.435583+0000 osd.0 (osd.0) 231 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.438245+0000 osd.0 (osd.0) 232 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.438245+0000 osd.0 (osd.0) 232 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.586628+0000 mgr.y (mgr.24401) 365 : cluster [DBG] pgmap v475: 228 pgs: 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.586628+0000 mgr.y (mgr.24401) 365 : cluster [DBG] pgmap v475: 228 pgs: 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.684325+0000 osd.7 (osd.7) 227 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.684325+0000 osd.7 (osd.7) 227 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.687517+0000 osd.7 (osd.7) 228 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.687517+0000 osd.7 (osd.7) 228 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:26:36.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.719771+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.719771+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.721715+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.721715+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.755086+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.755086+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.756571+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:26:36.617 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:35.756571+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.096676+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.096676+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.100317+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.100317+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.100503+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.100503+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.100615+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.100615+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.105863+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.105863+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.110963+0000 mon.c (mon.2) 534 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.618 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.110963+0000 mon.c (mon.2) 534 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.111620+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.111620+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.120527+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.120527+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.123047+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.619 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[23868]: audit 2024-04-26T16:26:36.123047+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.629 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:36.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.424355+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 deep-scrub starts 2024-04-26T16:26:36.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.426465+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 deep-scrub ok 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.426465+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 deep-scrub ok 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.695065+0000 osd.1 (osd.1) 201 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.695065+0000 osd.1 (osd.1) 201 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.699205+0000 osd.1 (osd.1) 202 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:34.699205+0000 osd.1 (osd.1) 202 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.235765+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.b scrub starts 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.235765+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.b scrub starts 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.237926+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.b scrub ok 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.237926+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.b scrub ok 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.435583+0000 osd.0 (osd.0) 231 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.435583+0000 osd.0 (osd.0) 231 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:26:36.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.438245+0000 osd.0 (osd.0) 232 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.438245+0000 osd.0 (osd.0) 232 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.586628+0000 mgr.y (mgr.24401) 365 : cluster [DBG] pgmap v475: 228 pgs: 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.586628+0000 mgr.y (mgr.24401) 365 : cluster [DBG] pgmap v475: 228 pgs: 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.684325+0000 osd.7 (osd.7) 227 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.684325+0000 osd.7 (osd.7) 227 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.687517+0000 osd.7 (osd.7) 228 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.687517+0000 osd.7 (osd.7) 228 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.719771+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.719771+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.721715+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.721715+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.755086+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:26:36.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.755086+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.756571+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:35.756571+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.424355+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 deep-scrub starts 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.424355+0000 osd.2 (osd.2) 169 : cluster [DBG] 5.6 deep-scrub starts 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.426465+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 deep-scrub ok 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.426465+0000 osd.2 (osd.2) 170 : cluster [DBG] 5.6 deep-scrub ok 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.695065+0000 osd.1 (osd.1) 201 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.695065+0000 osd.1 (osd.1) 201 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:26:36.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.699205+0000 osd.1 (osd.1) 202 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:34.699205+0000 osd.1 (osd.1) 202 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.235765+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.b scrub starts 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.235765+0000 osd.3 (osd.3) 229 : cluster [DBG] 11.b scrub starts 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.237926+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.b scrub ok 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.237926+0000 osd.3 (osd.3) 230 : cluster [DBG] 11.b scrub ok 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.435583+0000 osd.0 (osd.0) 231 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.435583+0000 osd.0 (osd.0) 231 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.438245+0000 osd.0 (osd.0) 232 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.438245+0000 osd.0 (osd.0) 232 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.586628+0000 mgr.y (mgr.24401) 365 : cluster [DBG] pgmap v475: 228 pgs: 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.586628+0000 mgr.y (mgr.24401) 365 : cluster [DBG] pgmap v475: 228 pgs: 228 active+clean; 4.6 MiB data, 558 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.684325+0000 osd.7 (osd.7) 227 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:26:36.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.684325+0000 osd.7 (osd.7) 227 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.687517+0000 osd.7 (osd.7) 228 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.687517+0000 osd.7 (osd.7) 228 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.719771+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.719771+0000 osd.4 (osd.4) 191 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.721715+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.721715+0000 osd.4 (osd.4) 192 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.755086+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.755086+0000 osd.6 (osd.6) 157 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.756571+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:35.756571+0000 osd.6 (osd.6) 158 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:26:36.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.096676+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.096676+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.100317+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.100317+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.100503+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.100503+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.100615+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.100615+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.105863+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.105863+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.110963+0000 mon.c (mon.2) 534 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.110963+0000 mon.c (mon.2) 534 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.111620+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.111620+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.120527+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.120527+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.123047+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[21463]: audit 2024-04-26T16:26:36.123047+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.096676+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.096676+0000 mon.a (mon.0) 2226 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.100317+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:36.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.100317+0000 mon.a (mon.0) 2227 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi050-73201-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.100503+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.100503+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.100615+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.100615+0000 mon.a (mon.0) 2229 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.105863+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.105863+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e328: 8 total, 8 up, 7 in 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.110963+0000 mon.c (mon.2) 534 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.110963+0000 mon.c (mon.2) 534 : audit [INF] from='client.? 172.21.15.50:0/3130819353' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.111620+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.111620+0000 mon.a (mon.0) 2231 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]: dispatch 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.120527+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.120527+0000 mon.a (mon.0) 2232 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.123047+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.685 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:36 smithi050 bash[29234]: audit 2024-04-26T16:26:36.123047+0000 mon.b (mon.1) 219 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:36.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:36.757 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:36.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:36.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:36.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:36.989 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:26:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:26:36.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:37.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:37.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7036 ms) 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6996 ms) 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6977 ms) 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7051 ms) 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6961 ms) 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-04-26T16:26:37.122 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7001 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7021 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6997 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6992 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7017 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7001 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7002 ms) 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-04-26T16:26:37.123 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6995 ms) 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7028 ms) 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (6990 ms) 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6994 ms) 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112059 ms total) 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [----------] Global test environment tear-down 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (189171 ms total) 2024-04-26T16:26:37.124 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio: [ PASSED ] 42 tests. 2024-04-26T16:26:37.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:37.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:37.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:37.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:37.409 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:37.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:37.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:37.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:37.671 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:37.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:37.800 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:37.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:37.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:37.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:38.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:38.057 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:38.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:38.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:38.169 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.404654+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.404654+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.405801+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.405801+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.735352+0000 osd.4 (osd.4) 193 : cluster [DBG] 6.e scrub starts 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.735352+0000 osd.4 (osd.4) 193 : cluster [DBG] 6.e scrub starts 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.737544+0000 osd.4 (osd.4) 194 : cluster [DBG] 6.e scrub ok 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:36.737544+0000 osd.4 (osd.4) 194 : cluster [DBG] 6.e scrub ok 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: audit 2024-04-26T16:26:36.770861+0000 mgr.y (mgr.24401) 366 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: audit 2024-04-26T16:26:36.770861+0000 mgr.y (mgr.24401) 366 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: audit 2024-04-26T16:26:37.103743+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: audit 2024-04-26T16:26:37.103743+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:38.170 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: audit 2024-04-26T16:26:37.103868+0000 mon.a (mon.0) 2234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: audit 2024-04-26T16:26:37.103868+0000 mon.a (mon.0) 2234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.107877+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.107877+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.172565+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.172565+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.181700+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.181700+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.360309+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.360309+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.361766+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.361766+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.408025+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:26:38.171 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.408025+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.410241+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.410241+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.587222+0000 mgr.y (mgr.24401) 367 : cluster [DBG] pgmap v478: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.587222+0000 mgr.y (mgr.24401) 367 : cluster [DBG] pgmap v478: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.404654+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.404654+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.405801+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.405801+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:26:38.172 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.735352+0000 osd.4 (osd.4) 193 : cluster [DBG] 6.e scrub starts 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.735352+0000 osd.4 (osd.4) 193 : cluster [DBG] 6.e scrub starts 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.737544+0000 osd.4 (osd.4) 194 : cluster [DBG] 6.e scrub ok 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:36.737544+0000 osd.4 (osd.4) 194 : cluster [DBG] 6.e scrub ok 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: audit 2024-04-26T16:26:36.770861+0000 mgr.y (mgr.24401) 366 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: audit 2024-04-26T16:26:36.770861+0000 mgr.y (mgr.24401) 366 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: audit 2024-04-26T16:26:37.103743+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: audit 2024-04-26T16:26:37.103743+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: audit 2024-04-26T16:26:37.103868+0000 mon.a (mon.0) 2234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: audit 2024-04-26T16:26:37.103868+0000 mon.a (mon.0) 2234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.107877+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.107877+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.172565+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:26:38.173 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.172565+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.181700+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.181700+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.360309+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.360309+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.361766+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.361766+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.408025+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.408025+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.410241+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.410241+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.587222+0000 mgr.y (mgr.24401) 367 : cluster [DBG] pgmap v478: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:38.174 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:38 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.587222+0000 mgr.y (mgr.24401) 367 : cluster [DBG] pgmap v478: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:38.182 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:38.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:38.312 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:38.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.404654+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.404654+0000 osd.2 (osd.2) 171 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.405801+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.405801+0000 osd.2 (osd.2) 172 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.735352+0000 osd.4 (osd.4) 193 : cluster [DBG] 6.e scrub starts 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.735352+0000 osd.4 (osd.4) 193 : cluster [DBG] 6.e scrub starts 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.737544+0000 osd.4 (osd.4) 194 : cluster [DBG] 6.e scrub ok 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:36.737544+0000 osd.4 (osd.4) 194 : cluster [DBG] 6.e scrub ok 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: audit 2024-04-26T16:26:36.770861+0000 mgr.y (mgr.24401) 366 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: audit 2024-04-26T16:26:36.770861+0000 mgr.y (mgr.24401) 366 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: audit 2024-04-26T16:26:37.103743+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:38.428 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: audit 2024-04-26T16:26:37.103743+0000 mon.a (mon.0) 2233 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi050-73193-42"}]': finished 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: audit 2024-04-26T16:26:37.103868+0000 mon.a (mon.0) 2234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: audit 2024-04-26T16:26:37.103868+0000 mon.a (mon.0) 2234 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.107877+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.107877+0000 mon.a (mon.0) 2235 : cluster [DBG] osdmap e329: 8 total, 8 up, 7 in 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.172565+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.172565+0000 osd.3 (osd.3) 231 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.181700+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.181700+0000 osd.3 (osd.3) 232 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.360309+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.360309+0000 osd.2 (osd.2) 173 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.361766+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.361766+0000 osd.2 (osd.2) 174 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.408025+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.408025+0000 osd.0 (osd.0) 233 : cluster [DBG] 2.1f deep-scrub starts 2024-04-26T16:26:38.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.410241+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:26:38.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.410241+0000 osd.0 (osd.0) 234 : cluster [DBG] 2.1f deep-scrub ok 2024-04-26T16:26:38.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.587222+0000 mgr.y (mgr.24401) 367 : cluster [DBG] pgmap v478: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:38.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:38 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.587222+0000 mgr.y (mgr.24401) 367 : cluster [DBG] pgmap v478: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:38.444 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:38.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:38.575 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:38.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:38.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:38.733 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:26:38.733 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:26:38.733 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:26:38.733 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:26:38.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:38.831 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/575229307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378108440 msgr2=0x7fe378108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- 172.21.15.50:0/575229307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378108440 0x7fe378108820 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7fe3680099c0 tx=0x7fe368009cd0 comp rx=0 tx=0).stop 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/575229307 shutdown_connections 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- 172.21.15.50:0/575229307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378103720 0x7fe37810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- 172.21.15.50:0/575229307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378108440 0x7fe378108820 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- 172.21.15.50:0/575229307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe378102d60 0x7fe3781031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/575229307 >> 172.21.15.50:0/575229307 conn(0x7fe3780775d0 msgr2=0x7fe3780779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/575229307 shutdown_connections 2024-04-26T16:26:38.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/575229307 wait complete. 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 librados: starting msgr at 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 librados: starting objecter 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 Processor -- start 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- start start 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 librados: setting wanted keys 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 librados: calling monclient init 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378102d60 0x7fe378199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe378103720 0x7fe3781996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378108440 0x7fe37819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe378111a30 con 0x7fe378102d60 2024-04-26T16:26:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe3781118b0 con 0x7fe378103720 2024-04-26T16:26:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe378111bb0 con 0x7fe378108440 2024-04-26T16:26:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378108440 0x7fe37819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378108440 0x7fe37819da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50306/0 (socket says 172.21.15.50:50306) 2024-04-26T16:26:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 learned_addr learned my addr 172.21.15.50:0/2257677018 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378102d60 0x7fe378199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe378103720 0x7fe3781996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe378103720 msgr2=0x7fe3781996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:38.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe378103720 0x7fe3781996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378102d60 msgr2=0x7fe378199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe378102d60 0x7fe378199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe37819e190 con 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378108440 0x7fe37819da50 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe37000c370 tx=0x7fe37000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe375ffb700 10 client.?.objecter ms_handle_connect 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe375ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3700092e0 con 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe370003a00 con 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370004540 con 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe37819e420 con 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 10 client.41027.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.937+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3781a5d00 con 0x7fe378108440 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 1 librados: init done 2024-04-26T16:26:38.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe37806a110 con 0x7fe378108440 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe37001a030 con 0x7fe378108440 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe360071f80 0x7fe360074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(330..330 src has 87..330) ==== 9938+0+0 (secure 0 0 0) 0x7fe370091690 con 0x7fe378108440 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 osd_map(330..330 src has 87..330) 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 3 client.41027.objecter handle_osd_map got epochs [330,330] > 0 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 3 client.41027.objecter handle_osd_map decoding full epoch 330 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe360071f80 0x7fe360074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 20 client.41027.objecter dump_active .. 0 homeless 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v330) ==== 54+0+16 (secure 0 0 0) 0x7fe370021070 con 0x7fe378108440 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 10 client.41027.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 1 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe378064030 0x7fe378064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 1 2024-04-26T16:26:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 0 1 2024-04-26T16:26:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 4 2024-04-26T16:26:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 10 client.41027.objecter _send_command 1 2024-04-26T16:26:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3781a9060 con 0x7fe378064030 2024-04-26T16:26:38.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe360071f80 2024-04-26T16:26:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe360071f80 0x7fe360074440 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7fe368009990 tx=0x7fe368000bc0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe378064030 0x7fe378064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe378064030 2024-04-26T16:26:38.944 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.941+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe378064030 0x7fe378064410 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.945+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3781a9060 con 0x7fe378064030 2024-04-26T16:26:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.945+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 1: 0 ) 2024-04-26T16:26:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.945+0000 7fe375ffb700 10 client.41027.objecter _finish_command 1 = system:0 2024-04-26T16:26:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.945+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 4 2024-04-26T16:26:38.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.945+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 0 1 2024-04-26T16:26:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 10 client.41027.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 2 2024-04-26T16:26:38.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 2 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37819a4e0 osd=0 3 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 0 2 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 4 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 10 client.41027.objecter _send_command 2 2024-04-26T16:26:38.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe378063ab0 con 0x7fe378064030 2024-04-26T16:26:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe378063ab0 con 0x7fe378064030 2024-04-26T16:26:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe375ffb700 10 client.41027.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 4 2024-04-26T16:26:38.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 0 2 2024-04-26T16:26:38.960 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 10 client.41027.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe37802d310 0x7fe37802d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 1 3 2024-04-26T16:26:38.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 4 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 10 client.41027.objecter _send_command 3 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.961+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe378105f00 con 0x7fe37802d310 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe37802d310 0x7fe37802d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe37802d310 0x7fe37802d730 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe37802d310 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe378105f00 con 0x7fe37802d310 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 3: 0 ) 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe375ffb700 10 client.41027.objecter _finish_command 3 = system:0 2024-04-26T16:26:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 4 2024-04-26T16:26:38.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.965+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 1 3 2024-04-26T16:26:38.972 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:38.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 10 client.41027.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 4 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 4 2024-04-26T16:26:38.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37810a980 osd=1 3 2024-04-26T16:26:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 1 4 2024-04-26T16:26:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 4 2024-04-26T16:26:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 10 client.41027.objecter _send_command 4 2024-04-26T16:26:38.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.981+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe37802dc70 con 0x7fe37802d310 2024-04-26T16:26:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe37802dc70 con 0x7fe37802d310 2024-04-26T16:26:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 10 client.41027.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 4 2024-04-26T16:26:38.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 1 4 2024-04-26T16:26:38.982 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:26:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 10 client.41027.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:38.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 5 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3781ad620 0x7fe3781afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 5 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 2 5 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 4 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 10 client.41027.objecter _send_command 5 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe37802e760 con 0x7fe3781ad620 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3781ad620 0x7fe3781afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3781ad620 0x7fe3781afa10 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe3781ad620 2024-04-26T16:26:38.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe37802e760 con 0x7fe3781ad620 2024-04-26T16:26:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 5: 0 ) 2024-04-26T16:26:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 10 client.41027.objecter _finish_command 5 = system:0 2024-04-26T16:26:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 4 2024-04-26T16:26:38.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:38.985+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 2 5 2024-04-26T16:26:39.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 10 client.41027.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:39.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 6 2024-04-26T16:26:39.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:39.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:39.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:39.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 6 2024-04-26T16:26:39.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37802e5f0 osd=2 3 2024-04-26T16:26:39.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 2 6 2024-04-26T16:26:39.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 4 2024-04-26T16:26:39.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 10 client.41027.objecter _send_command 6 2024-04-26T16:26:39.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.001+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe37802e760 con 0x7fe3781ad620 2024-04-26T16:26:39.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe37802e760 con 0x7fe3781ad620 2024-04-26T16:26:39.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:39.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 10 client.41027.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:39.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 4 2024-04-26T16:26:39.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 2 6 2024-04-26T16:26:39.002 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:26:39.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 10 client.41027.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:39.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 7 2024-04-26T16:26:39.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3781a8bc0 0x7fe3781b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 7 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 3 7 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 4 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 10 client.41027.objecter _send_command 7 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe37802e2c0 con 0x7fe3781a8bc0 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3781a8bc0 0x7fe3781b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe3781a8bc0 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3781a8bc0 0x7fe3781b30e0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe37802e2c0 con 0x7fe3781a8bc0 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 7: 0 ) 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 10 client.41027.objecter _finish_command 7 = system:0 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 4 2024-04-26T16:26:39.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.005+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 3 7 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 10 client.41027.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 8 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 8 2024-04-26T16:26:39.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe37802e760 osd=3 3 2024-04-26T16:26:39.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 3 8 2024-04-26T16:26:39.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 4 2024-04-26T16:26:39.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 10 client.41027.objecter _send_command 8 2024-04-26T16:26:39.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe37802e2c0 con 0x7fe3781a8bc0 2024-04-26T16:26:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe37802e2c0 con 0x7fe3781a8bc0 2024-04-26T16:26:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe375ffb700 10 client.41027.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 4 2024-04-26T16:26:39.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 3 8 2024-04-26T16:26:39.023 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:26:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 10 client.41027.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 9 2024-04-26T16:26:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3781b3ca0 0x7fe3781b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 9 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 4 9 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 4 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 10 client.41027.objecter _send_command 9 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe3781b8fa0 con 0x7fe3781b3ca0 2024-04-26T16:26:39.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.025+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3781b3ca0 0x7fe3781b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe3781b3ca0 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3781b3ca0 0x7fe3781b6090 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3781b8fa0 con 0x7fe3781b3ca0 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 9: 0 ) 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe375ffb700 10 client.41027.objecter _finish_command 9 = system:0 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 4 2024-04-26T16:26:39.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.029+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 4 9 2024-04-26T16:26:39.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 10 client.41027.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 10 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 10 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781b3620 osd=4 3 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 4 10 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 4 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 10 client.41027.objecter _send_command 10 2024-04-26T16:26:39.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.049+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe3781b7dc0 con 0x7fe3781b3ca0 2024-04-26T16:26:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3781b7dc0 con 0x7fe3781b3ca0 2024-04-26T16:26:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 10 client.41027.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 4 2024-04-26T16:26:39.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 4 10 2024-04-26T16:26:39.049 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 10 client.41027.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 11 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3781b84e0 0x7fe3781b8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 11 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 5 11 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 4 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 10 client.41027.objecter _send_command 11 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe3781bc370 con 0x7fe3781b84e0 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3781b84e0 0x7fe3781b8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe3781b84e0 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe377fff700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3781b84e0 0x7fe3781b8940 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3781bc370 con 0x7fe3781b84e0 2024-04-26T16:26:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 11: 0 ) 2024-04-26T16:26:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 10 client.41027.objecter _finish_command 11 = system:0 2024-04-26T16:26:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 4 2024-04-26T16:26:39.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.053+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 5 11 2024-04-26T16:26:39.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 10 client.41027.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 12 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 12 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781b7dc0 osd=5 3 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 5 12 2024-04-26T16:26:39.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 4 2024-04-26T16:26:39.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 10 client.41027.objecter _send_command 12 2024-04-26T16:26:39.071 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.073+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe3781bb1f0 con 0x7fe3781b84e0 2024-04-26T16:26:39.074 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:26:39.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3781bb1f0 con 0x7fe3781b84e0 2024-04-26T16:26:39.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:39.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 10 client.41027.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:39.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 4 2024-04-26T16:26:39.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 5 12 2024-04-26T16:26:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 10 client.41027.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 13 2024-04-26T16:26:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3781bb910 0x7fe3781bbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 13 2024-04-26T16:26:39.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 6 13 2024-04-26T16:26:39.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 4 2024-04-26T16:26:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 10 client.41027.objecter _send_command 13 2024-04-26T16:26:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe3781bf710 con 0x7fe3781bb910 2024-04-26T16:26:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3781bb910 0x7fe3781bbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe3781bb910 2024-04-26T16:26:39.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe37d31c700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3781bb910 0x7fe3781bbd50 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.080 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:39.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3781bf710 con 0x7fe3781bb910 2024-04-26T16:26:39.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 13: 0 ) 2024-04-26T16:26:39.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 10 client.41027.objecter _finish_command 13 = system:0 2024-04-26T16:26:39.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 4 2024-04-26T16:26:39.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.077+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 6 13 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 10 client.41027.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 14 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 14 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781bb1f0 osd=6 3 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 6 14 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 4 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 10 client.41027.objecter _send_command 14 2024-04-26T16:26:39.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.097+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe3781b65f0 con 0x7fe3781bb910 2024-04-26T16:26:39.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3781b65f0 con 0x7fe3781bb910 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 10 client.41027.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 4 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 6 14 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 10 client.41027.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:26:39.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 15 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3781b6d60 0x7fe3781b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 15 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 7 15 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 4 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 10 client.41027.objecter _send_command 15 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe3781b7880 con 0x7fe3781b6d60 2024-04-26T16:26:39.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3781b6d60 0x7fe3781b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 10 client.41027.objecter ms_handle_connect 0x7fe3781b6d60 2024-04-26T16:26:39.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe37cb1b700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3781b6d60 0x7fe3781b7180 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3781b7880 con 0x7fe3781b6d60 2024-04-26T16:26:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 15: 0 ) 2024-04-26T16:26:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 10 client.41027.objecter _finish_command 15 = system:0 2024-04-26T16:26:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 4 2024-04-26T16:26:39.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.101+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 7 15 2024-04-26T16:26:39.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:39.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 10 client.41027.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign -1 16 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 20 client.41027.objecter _get_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_remove -1 16 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 20 client.41027.objecter get_session s=0x7fe3781b65f0 osd=7 3 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 15 client.41027.objecter _session_command_op_assign 7 16 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 4 2024-04-26T16:26:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 10 client.41027.objecter _send_command 16 2024-04-26T16:26:39.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.129+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fe3781b7880 con 0x7fe3781b6d60 2024-04-26T16:26:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe375ffb700 1 -- 172.21.15.50:0/2257677018 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3781b7880 con 0x7fe3781b6d60 2024-04-26T16:26:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe375ffb700 10 client.41027.objecter ms_dispatch 0x7fe378001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:26:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe375ffb700 10 client.41027.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:26:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe375ffb700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 4 2024-04-26T16:26:39.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe375ffb700 15 client.41027.objecter _session_command_op_remove 7 16 2024-04-26T16:26:39.131 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 librados: watch_flush enter 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 librados: watch_flush exit 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.0 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe378064030 msgr2=0x7fe378064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe378064030 0x7fe378064410 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37819a4e0 osd=0 1 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.1 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe37802d310 msgr2=0x7fe37802d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe37802d310 0x7fe37802d730 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37810a980 osd=1 1 2024-04-26T16:26:39.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.2 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3781ad620 msgr2=0x7fe3781afa10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3781ad620 0x7fe3781afa10 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e5f0 osd=2 1 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.3 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3781a8bc0 msgr2=0x7fe3781b30e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3781a8bc0 0x7fe3781b30e0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 reap_dead start 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe37802e760 osd=3 1 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.4 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3781b3ca0 msgr2=0x7fe3781b6090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3781b3ca0 0x7fe3781b6090 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b3620 osd=4 1 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.5 2024-04-26T16:26:39.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 reap_dead start 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3781b84e0 msgr2=0x7fe3781b8940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3781b84e0 0x7fe3781b8940 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b7dc0 osd=5 1 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.6 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3781bb910 msgr2=0x7fe3781bbd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3781bb910 0x7fe3781bbd50 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781bb1f0 osd=6 1 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter close_session for osd.7 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3781b6d60 msgr2=0x7fe3781b7180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3781b6d60 0x7fe3781b7180 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter put_session s=0x7fe3781b65f0 osd=7 1 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 20 client.41027.objecter shutdown clearing up homeless session... 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 10 client.41027.objecter successfully canceled tick 2024-04-26T16:26:39.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe360071f80 msgr2=0x7fe360074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe360071f80 0x7fe360074440 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7fe368009990 tx=0x7fe368000bc0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378108440 msgr2=0x7fe37819da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 --2- 172.21.15.50:0/2257677018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe378108440 0x7fe37819da50 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fe37000c370 tx=0x7fe37000c680 comp rx=0 tx=0).stop 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37d31c700 1 -- 172.21.15.50:0/2257677018 reap_dead start 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 shutdown_connections 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 >> 172.21.15.50:0/2257677018 conn(0x7fe3780775d0 msgr2=0x7fe378101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 shutdown_connections 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 -- 172.21.15.50:0/2257677018 wait complete. 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.133+0000 7fe37db1d700 1 librados: shutdown 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:26:39.135 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:26:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:26:39.136 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:26:39.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:39.218 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.834633+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.1f deep-scrub starts 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.834633+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.1f deep-scrub starts 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.837284+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.1f deep-scrub ok 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: cluster 2024-04-26T16:26:37.837284+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.1f deep-scrub ok 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: cluster 2024-04-26T16:26:38.120915+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: cluster 2024-04-26T16:26:38.120915+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.137982+0000 mon.c (mon.2) 535 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.137982+0000 mon.c (mon.2) 535 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.139265+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.139265+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.179820+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.179820+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.182296+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.219 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.182296+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.220 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.944199+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? 172.21.15.50:0/2257677018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:39.220 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:39 smithi190 bash[23868]: audit 2024-04-26T16:26:38.944199+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? 172.21.15.50:0/2257677018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:39.234 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:39.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:39.363 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.834633+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.1f deep-scrub starts 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.834633+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.1f deep-scrub starts 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.837284+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.1f deep-scrub ok 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: cluster 2024-04-26T16:26:37.837284+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.1f deep-scrub ok 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: cluster 2024-04-26T16:26:38.120915+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: cluster 2024-04-26T16:26:38.120915+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-04-26T16:26:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.137982+0000 mon.c (mon.2) 535 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.137982+0000 mon.c (mon.2) 535 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.139265+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.139265+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.179820+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.179820+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.182296+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.182296+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.944199+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? 172.21.15.50:0/2257677018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:39.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[21463]: audit 2024-04-26T16:26:38.944199+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? 172.21.15.50:0/2257677018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:39.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.834633+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.1f deep-scrub starts 2024-04-26T16:26:39.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.834633+0000 osd.6 (osd.6) 159 : cluster [DBG] 5.1f deep-scrub starts 2024-04-26T16:26:39.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.837284+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.1f deep-scrub ok 2024-04-26T16:26:39.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: cluster 2024-04-26T16:26:37.837284+0000 osd.6 (osd.6) 160 : cluster [DBG] 5.1f deep-scrub ok 2024-04-26T16:26:39.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: cluster 2024-04-26T16:26:38.120915+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-04-26T16:26:39.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: cluster 2024-04-26T16:26:38.120915+0000 mon.a (mon.0) 2236 : cluster [DBG] osdmap e330: 8 total, 8 up, 7 in 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.137982+0000 mon.c (mon.2) 535 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.137982+0000 mon.c (mon.2) 535 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.139265+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.139265+0000 mon.a (mon.0) 2237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.179820+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.179820+0000 mon.a (mon.0) 2238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.182296+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.182296+0000 mon.b (mon.1) 220 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.944199+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? 172.21.15.50:0/2257677018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:39.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:39 smithi050 bash[29234]: audit 2024-04-26T16:26:38.944199+0000 mon.c (mon.2) 536 : audit [DBG] from='client.? 172.21.15.50:0/2257677018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:26:39.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:39.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:39.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-26T16:26:39.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:39.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:39.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:39.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/3404962535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06bbd0 msgr2=0x7fa21c06c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- 172.21.15.50:0/3404962535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06bbd0 0x7fa21c06c050 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7fa210009990 tx=0x7fa210009ca0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/3404962535 shutdown_connections 2024-04-26T16:26:39.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- 172.21.15.50:0/3404962535 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21c06c590 0x7fa21c110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- 172.21.15.50:0/3404962535 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c1094d0 0x7fa21c1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- 172.21.15.50:0/3404962535 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06bbd0 0x7fa21c06c050 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/3404962535 >> 172.21.15.50:0/3404962535 conn(0x7fa21c0793c0 msgr2=0x7fa21c0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/3404962535 shutdown_connections 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/3404962535 wait complete. 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 librados: starting msgr at 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 librados: starting objecter 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 Processor -- start 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- start start 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 librados: setting wanted keys 2024-04-26T16:26:39.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 librados: calling monclient init 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 0x7fa21c199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06c590 0x7fa21c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21c1094d0 0x7fa21c19da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c112ad0 con 0x7fa21c06c590 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa21c112950 con 0x7fa21c1094d0 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa21c112c50 con 0x7fa21c06bbd0 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 0x7fa21c199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06c590 0x7fa21c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 0x7fa21c199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50334/0 (socket says 172.21.15.50:50334) 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21bfff700 1 -- 172.21.15.50:0/144489411 learned_addr learned my addr 172.21.15.50:0/144489411 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa22099c700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21c1094d0 0x7fa21c19da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 -- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 msgr2=0x7fa21c199170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 0x7fa21c199170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 -- 172.21.15.50:0/144489411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21c1094d0 msgr2=0x7fa21c19da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21c1094d0 0x7fa21c19da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 -- 172.21.15.50:0/144489411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21c19e180 con 0x7fa21c06c590 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21bfff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 0x7fa21c199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa21b7fe700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06c590 0x7fa21c1996b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fa210006010 tx=0x7fa210005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2197fa700 10 client.?.objecter ms_handle_connect 0x7fa21c06c590 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2197fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa210026070 con 0x7fa21c06c590 2024-04-26T16:26:39.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa210021410 con 0x7fa21c06c590 2024-04-26T16:26:39.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2100173f0 con 0x7fa21c06c590 2024-04-26T16:26:39.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21c19e410 con 0x7fa21c06c590 2024-04-26T16:26:39.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 10 client.30414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:39.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21c1a5cf0 con 0x7fa21c06c590 2024-04-26T16:26:39.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.713+0000 7fa2223ff700 1 librados: init done 2024-04-26T16:26:39.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa21c06a110 con 0x7fa21c06c590 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa210029070 con 0x7fa21c06c590 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa204071f80 0x7fa204074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(331..331 src has 87..331) ==== 9546+0+0 (secure 0 0 0) 0x7fa210092680 con 0x7fa21c06c590 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 10 client.30414.objecter ms_dispatch 0x7fa21c001550 osd_map(331..331 src has 87..331) 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 3 client.30414.objecter handle_osd_map got epochs [331,331] > 0 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 3 client.30414.objecter handle_osd_map decoding full epoch 331 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa21bfff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa204071f80 0x7fa204074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 20 client.30414.objecter dump_active .. 0 homeless 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa2197fa700 10 client.30414.objecter ms_handle_connect 0x7fa204071f80 2024-04-26T16:26:39.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.717+0000 7fa21bfff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa204071f80 0x7fa204074440 secure :-1 s=READY pgs=3716 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009680 tx=0x7fa20c009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:39.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.721+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fa210013070 con 0x7fa21c06c590 2024-04-26T16:26:39.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:39.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:39.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:39.866 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:39.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:39.957+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fa21c1a5ec0 con 0x7fa21c06c590 2024-04-26T16:26:39.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:39.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:39.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:40.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:40.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:40.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.129+0000 7fa2197fa700 1 -- 172.21.15.50:0/144489411 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v332) ==== 87+0+0 (secure 0 0 0) 0x7fa210061cc0 con 0x7fa21c06c590 2024-04-26T16:26:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 10 librados: watch_flush enter 2024-04-26T16:26:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 10 librados: watch_flush exit 2024-04-26T16:26:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 20 client.30414.objecter shutdown clearing up homeless session... 2024-04-26T16:26:40.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 10 client.30414.objecter successfully canceled tick 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa204071f80 msgr2=0x7fa204074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa204071f80 0x7fa204074440 secure :-1 s=READY pgs=3716 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009680 tx=0x7fa20c009260 comp rx=0 tx=0).stop 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06c590 msgr2=0x7fa21c1996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06c590 0x7fa21c1996b0 secure :-1 s=READY pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fa210006010 tx=0x7fa210005ef0 comp rx=0 tx=0).stop 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 shutdown_connections 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa204071f80 0x7fa204074440 unknown :-1 s=CLOSED pgs=3716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa21c1094d0 0x7fa21c19da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa21c06bbd0 0x7fa21c199170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 --2- 172.21.15.50:0/144489411 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa21c06c590 0x7fa21c1996b0 secure :-1 s=CLOSED pgs=2445 cs=0 l=1 rev1=1 crypto rx=0x7fa210006010 tx=0x7fa210005ef0 comp rx=0 tx=0).stop 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 >> 172.21.15.50:0/144489411 conn(0x7fa21c0793c0 msgr2=0x7fa21c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 shutdown_connections 2024-04-26T16:26:40.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 -- 172.21.15.50:0/144489411 wait complete. 2024-04-26T16:26:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:40.137+0000 7fa2223ff700 1 librados: shutdown 2024-04-26T16:26:40.138 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:26:40.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:40.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:40.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:38.809405+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:26:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:38.809405+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:26:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:38.810541+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:26:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:38.810541+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:26:40.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.114132+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.114132+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.114202+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.114202+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.117714+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.117714+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.119442+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.119442+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.125968+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.125968+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.128630+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.128630+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.133381+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.133381+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:38.809405+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:38.809405+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:38.810541+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:38.810541+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.114132+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.114132+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.114202+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:40.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.114202+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.117714+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.117714+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.119442+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.119442+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.125968+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.125968+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.128630+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.128630+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.133381+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.133381+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.587730+0000 mgr.y (mgr.24401) 368 : cluster [DBG] pgmap v481: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.587730+0000 mgr.y (mgr.24401) 368 : cluster [DBG] pgmap v481: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.961963+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:26:40.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[21463]: audit 2024-04-26T16:26:39.961963+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:26:40.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.587730+0000 mgr.y (mgr.24401) 368 : cluster [DBG] pgmap v481: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:40.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.587730+0000 mgr.y (mgr.24401) 368 : cluster [DBG] pgmap v481: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:40.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.961963+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:26:40.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:40 smithi050 bash[29234]: audit 2024-04-26T16:26:39.961963+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:26:40.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:40.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:38.809405+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:26:40.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:38.809405+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:26:40.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:38.810541+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:26:40.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:38.810541+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.114132+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.114132+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.114202+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.114202+0000 mon.a (mon.0) 2240 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.117714+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.117714+0000 mon.a (mon.0) 2241 : cluster [DBG] osdmap e331: 8 total, 8 up, 7 in 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.119442+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.119442+0000 mon.c (mon.2) 537 : audit [INF] from='client.? 172.21.15.50:0/3935345188' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.125968+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.125968+0000 mon.b (mon.1) 221 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.128630+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.128630+0000 mon.a (mon.0) 2242 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.133381+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.388 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.133381+0000 mon.a (mon.0) 2243 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]: dispatch 2024-04-26T16:26:40.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.587730+0000 mgr.y (mgr.24401) 368 : cluster [DBG] pgmap v481: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:40.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.587730+0000 mgr.y (mgr.24401) 368 : cluster [DBG] pgmap v481: 228 pgs: 228 active+clean; 4.6 MiB data, 526 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:26:40.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.961963+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:26:40.389 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:40 smithi190 bash[23868]: audit 2024-04-26T16:26:39.961963+0000 mon.a (mon.0) 2244 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:26:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:40.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:40.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:40.649 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:40.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:40.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:40.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:40.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:40.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:40.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:41.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:41.045 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:41.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:41.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:41.172 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:41.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.805744+0000 osd.6 (osd.6) 163 : cluster [DBG] 6.a scrub starts 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.805744+0000 osd.6 (osd.6) 163 : cluster [DBG] 6.a scrub starts 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.807341+0000 osd.6 (osd.6) 164 : cluster [DBG] 6.a scrub ok 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:39.807341+0000 osd.6 (osd.6) 164 : cluster [DBG] 6.a scrub ok 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:40.124718+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:40.124718+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:40.124774+0000 mon.a (mon.0) 2246 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:40.124774+0000 mon.a (mon.0) 2246 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.134720+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.134720+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.134802+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:41.288 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.134802+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.134854+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.134854+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:40.138689+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:40.138689+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.166409+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.166409+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.167115+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.167115+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.167753+0000 mon.c (mon.2) 539 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.167753+0000 mon.c (mon.2) 539 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.168059+0000 mon.a (mon.0) 2252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.168059+0000 mon.a (mon.0) 2252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.168608+0000 mon.c (mon.2) 540 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.290 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.168608+0000 mon.c (mon.2) 540 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.290 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.168957+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.290 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: audit 2024-04-26T16:26:40.168957+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.290 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:41.099366+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:41.290 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:41 smithi190 bash[23868]: cluster 2024-04-26T16:26:41.099366+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:41.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.805744+0000 osd.6 (osd.6) 163 : cluster [DBG] 6.a scrub starts 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.805744+0000 osd.6 (osd.6) 163 : cluster [DBG] 6.a scrub starts 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.807341+0000 osd.6 (osd.6) 164 : cluster [DBG] 6.a scrub ok 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:39.807341+0000 osd.6 (osd.6) 164 : cluster [DBG] 6.a scrub ok 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:40.124718+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:40.124718+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:40.124774+0000 mon.a (mon.0) 2246 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:40.124774+0000 mon.a (mon.0) 2246 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.134720+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:41.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.134720+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.134802+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.134802+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.134854+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.134854+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:40.138689+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:40.138689+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.166409+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.166409+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.167115+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.805744+0000 osd.6 (osd.6) 163 : cluster [DBG] 6.a scrub starts 2024-04-26T16:26:41.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.805744+0000 osd.6 (osd.6) 163 : cluster [DBG] 6.a scrub starts 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.807341+0000 osd.6 (osd.6) 164 : cluster [DBG] 6.a scrub ok 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:39.807341+0000 osd.6 (osd.6) 164 : cluster [DBG] 6.a scrub ok 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:40.124718+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:40.124718+0000 mon.a (mon.0) 2245 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:40.124774+0000 mon.a (mon.0) 2246 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:40.124774+0000 mon.a (mon.0) 2246 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.134720+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.134720+0000 mon.a (mon.0) 2247 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi050-73201-52"}]': finished 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.134802+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.134802+0000 mon.a (mon.0) 2248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-35"}]': finished 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.134854+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.134854+0000 mon.a (mon.0) 2249 : audit [INF] from='client.? 172.21.15.50:0/144489411' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:40.138689+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-04-26T16:26:41.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:40.138689+0000 mon.a (mon.0) 2250 : cluster [DBG] osdmap e332: 8 total, 8 up, 7 in 2024-04-26T16:26:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.166409+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.166409+0000 mon.c (mon.2) 538 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.167115+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.167115+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.167753+0000 mon.c (mon.2) 539 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.167753+0000 mon.c (mon.2) 539 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.168059+0000 mon.a (mon.0) 2252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.168059+0000 mon.a (mon.0) 2252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.168608+0000 mon.c (mon.2) 540 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.168608+0000 mon.c (mon.2) 540 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.168957+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: audit 2024-04-26T16:26:40.168957+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:41.099366+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[21463]: cluster 2024-04-26T16:26:41.099366+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.167115+0000 mon.a (mon.0) 2251 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.167753+0000 mon.c (mon.2) 539 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.167753+0000 mon.c (mon.2) 539 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.168059+0000 mon.a (mon.0) 2252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.168059+0000 mon.a (mon.0) 2252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.168608+0000 mon.c (mon.2) 540 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.168608+0000 mon.c (mon.2) 540 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.168957+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: audit 2024-04-26T16:26:40.168957+0000 mon.a (mon.0) 2253 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:41.099366+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:41 smithi050 bash[29234]: cluster 2024-04-26T16:26:41.099366+0000 mon.a (mon.0) 2254 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:41.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:41.435 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:41.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:41.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:41.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:41.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:41.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:41.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:41.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:41.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:42.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:42.084 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:42.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:42.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:42.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.149510+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:42.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.149510+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:42.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: cluster 2024-04-26T16:26:41.154105+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-04-26T16:26:42.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: cluster 2024-04-26T16:26:41.154105+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-04-26T16:26:42.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.155549+0000 mon.c (mon.2) 541 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.155549+0000 mon.c (mon.2) 541 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.156205+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.156205+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.513429+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.513429+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.516230+0000 mon.c (mon.2) 542 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: audit 2024-04-26T16:26:41.516230+0000 mon.c (mon.2) 542 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: cluster 2024-04-26T16:26:41.588944+0000 mgr.y (mgr.24401) 369 : cluster [DBG] pgmap v484: 196 pgs: 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:42.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:42 smithi190 bash[23868]: cluster 2024-04-26T16:26:41.588944+0000 mgr.y (mgr.24401) 369 : cluster [DBG] pgmap v484: 196 pgs: 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:42.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.149510+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.149510+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: cluster 2024-04-26T16:26:41.154105+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: cluster 2024-04-26T16:26:41.154105+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.155549+0000 mon.c (mon.2) 541 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.155549+0000 mon.c (mon.2) 541 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.156205+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.156205+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.337 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.513429+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.513429+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.516230+0000 mon.c (mon.2) 542 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: audit 2024-04-26T16:26:41.516230+0000 mon.c (mon.2) 542 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: cluster 2024-04-26T16:26:41.588944+0000 mgr.y (mgr.24401) 369 : cluster [DBG] pgmap v484: 196 pgs: 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[21463]: cluster 2024-04-26T16:26:41.588944+0000 mgr.y (mgr.24401) 369 : cluster [DBG] pgmap v484: 196 pgs: 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.149510+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.149510+0000 mon.a (mon.0) 2255 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: cluster 2024-04-26T16:26:41.154105+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: cluster 2024-04-26T16:26:41.154105+0000 mon.a (mon.0) 2256 : cluster [DBG] osdmap e333: 8 total, 8 up, 7 in 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.155549+0000 mon.c (mon.2) 541 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.155549+0000 mon.c (mon.2) 541 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.156205+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.156205+0000 mon.a (mon.0) 2257 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.513429+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.513429+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.516230+0000 mon.c (mon.2) 542 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: audit 2024-04-26T16:26:41.516230+0000 mon.c (mon.2) 542 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: cluster 2024-04-26T16:26:41.588944+0000 mgr.y (mgr.24401) 369 : cluster [DBG] pgmap v484: 196 pgs: 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:42.339 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:42 smithi050 bash[29234]: cluster 2024-04-26T16:26:41.588944+0000 mgr.y (mgr.24401) 369 : cluster [DBG] pgmap v484: 196 pgs: 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:42.353 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:42.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:42.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:42.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:42.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:42.619 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:42.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:42.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:42.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:42.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:42.880 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:42.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:42.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:43.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:43.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:43.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:43.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:43.268 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:43.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: cluster 2024-04-26T16:26:42.178450+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: cluster 2024-04-26T16:26:42.178450+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:42.183572+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:42.183572+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:42.230815+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:42.230815+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:43.156965+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:43.156965+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:43.157138+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:43.157138+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:43.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.161918+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.161918+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:43.177893+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[21463]: audit 2024-04-26T16:26:43.177893+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: cluster 2024-04-26T16:26:42.178450+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: cluster 2024-04-26T16:26:42.178450+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:42.183572+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:42.183572+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:42.230815+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:42.230815+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:43.156965+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:43.156965+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:43.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:43.157138+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:43.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:43.157138+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:43.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.161918+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-04-26T16:26:43.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.161918+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-04-26T16:26:43.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:43.177893+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:43.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:43 smithi050 bash[29234]: audit 2024-04-26T16:26:43.177893+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:43.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:43.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:43.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: cluster 2024-04-26T16:26:42.178450+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-04-26T16:26:43.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: cluster 2024-04-26T16:26:42.178450+0000 mon.a (mon.0) 2259 : cluster [DBG] osdmap e334: 8 total, 8 up, 7 in 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:42.183572+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:42.183572+0000 mon.b (mon.1) 222 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:42.230815+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:42.230815+0000 mon.a (mon.0) 2260 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:43.156965+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:43.156965+0000 mon.a (mon.0) 2261 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi050-73201-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:43.157138+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:43.157138+0000 mon.a (mon.0) 2262 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.161918+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.161918+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e335: 8 total, 8 up, 7 in 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:43.177893+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:43.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:43 smithi190 bash[23868]: audit 2024-04-26T16:26:43.177893+0000 mon.b (mon.1) 223 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:43.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:43.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:43.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:43.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:43.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:26:43.786 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:43.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:43.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:44.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:44.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:44.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:44.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:44.278 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:26:44.297 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:44.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:43.175553+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:43.175553+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.589776+0000 mgr.y (mgr.24401) 370 : cluster [DBG] pgmap v487: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.589776+0000 mgr.y (mgr.24401) 370 : cluster [DBG] pgmap v487: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:44.159636+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:44.159636+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: cluster 2024-04-26T16:26:44.167051+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: cluster 2024-04-26T16:26:44.167051+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:44.171728+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:44.171728+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:44.173948+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21463]: audit 2024-04-26T16:26:44.173948+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:43.175553+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:43.175553+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.589776+0000 mgr.y (mgr.24401) 370 : cluster [DBG] pgmap v487: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.589776+0000 mgr.y (mgr.24401) 370 : cluster [DBG] pgmap v487: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:44.159636+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:44.159636+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: cluster 2024-04-26T16:26:44.167051+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: cluster 2024-04-26T16:26:44.167051+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:44.171728+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:44.171728+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:44.173948+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.409 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[29234]: audit 2024-04-26T16:26:44.173948+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:44.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:44.547 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:26:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:26:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:26:44.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:44.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:43.175553+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:44.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:43.175553+0000 mon.a (mon.0) 2264 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.589776+0000 mgr.y (mgr.24401) 370 : cluster [DBG] pgmap v487: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.589776+0000 mgr.y (mgr.24401) 370 : cluster [DBG] pgmap v487: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:44.159636+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:44.159636+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: cluster 2024-04-26T16:26:44.167051+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: cluster 2024-04-26T16:26:44.167051+0000 mon.a (mon.0) 2266 : cluster [DBG] osdmap e336: 8 total, 8 up, 7 in 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:44.171728+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:44.171728+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:44.173948+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:44 smithi190 bash[23868]: audit 2024-04-26T16:26:44.173948+0000 mon.b (mon.1) 224 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:44.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:44.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:44.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:44.821 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:44.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:26:44.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:44.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:44.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:44.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/598678826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04108390 msgr2=0x7f7d04108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- 172.21.15.50:0/598678826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04108390 0x7f7d04108770 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009990 tx=0x7f7d00009ca0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/598678826 shutdown_connections 2024-04-26T16:26:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- 172.21.15.50:0/598678826 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102ce0 0x7f7d0410f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- 172.21.15.50:0/598678826 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04108390 0x7f7d04108770 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- 172.21.15.50:0/598678826 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d04102320 0x7f7d041027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/598678826 >> 172.21.15.50:0/598678826 conn(0x7f7d040792d0 msgr2=0x7f7d040796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/598678826 shutdown_connections 2024-04-26T16:26:45.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/598678826 wait complete. 2024-04-26T16:26:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 librados: starting msgr at 2024-04-26T16:26:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 librados: starting objecter 2024-04-26T16:26:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 Processor -- start 2024-04-26T16:26:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- start start 2024-04-26T16:26:45.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 librados: setting wanted keys 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 librados: calling monclient init 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 0x7f7d0419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04102ce0 0x7f7d0419b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d04108390 0x7f7d0419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7d04111990 con 0x7f7d04102320 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7d04111810 con 0x7f7d04108390 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7d04111b10 con 0x7f7d04102ce0 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 0x7f7d0419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 0x7f7d0419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45608/0 (socket says 172.21.15.50:45608) 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d095fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04102ce0 0x7f7d0419b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 -- 172.21.15.50:0/1406646925 learned_addr learned my addr 172.21.15.50:0/1406646925 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0a5fe700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d04108390 0x7f7d0419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:45.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 -- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04102ce0 msgr2=0x7f7d0419b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04102ce0 0x7f7d0419b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 -- 172.21.15.50:0/1406646925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d04108390 msgr2=0x7f7d0419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d04108390 0x7f7d0419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 -- 172.21.15.50:0/1406646925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d041a0380 con 0x7f7d04102320 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d09dfd700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 0x7f7d0419b370 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f7d000057a0 tx=0x7f7d000057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7cfaffd700 10 client.?.objecter ms_handle_connect 0x7f7d04102320 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7cfaffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00026070 con 0x7f7d04102320 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7d00021410 con 0x7f7d04102320 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d041a0610 con 0x7f7d04102320 2024-04-26T16:26:45.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d000173f0 con 0x7f7d04102320 2024-04-26T16:26:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 10 client.30426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:26:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d041a7ef0 con 0x7f7d04102320 2024-04-26T16:26:45.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.013+0000 7f7d0c061700 1 librados: init done 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d04103a40 con 0x7f7d04102320 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7d00029070 con 0x7f7d04102320 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7cf0071f80 0x7f7cf0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(336..336 src has 87..336) ==== 10297+0+0 (secure 0 0 0) 0x7f7d00092a00 con 0x7f7d04102320 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 10 client.30426.objecter ms_dispatch 0x7f7d04001550 osd_map(336..336 src has 87..336) 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 3 client.30426.objecter handle_osd_map got epochs [336,336] > 0 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 3 client.30426.objecter handle_osd_map decoding full epoch 336 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7d095fc700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7cf0071f80 0x7f7cf0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 20 client.30426.objecter dump_active .. 0 homeless 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7cfaffd700 10 client.30426.objecter ms_handle_connect 0x7f7cf0071f80 2024-04-26T16:26:45.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.017+0000 7f7d095fc700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7cf0071f80 0x7f7cf0074440 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7f7d04079b00 tx=0x7f7cf400a380 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:26:45.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.021+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7d00013070 con 0x7f7d04102320 2024-04-26T16:26:45.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:45.081 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:45.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:45.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:45.211 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:45.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.265+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) -- 0x7f7d0419c6d0 con 0x7f7d04102320 2024-04-26T16:26:45.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7cfaffd700 1 -- 172.21.15.50:0/1406646925 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v337) ==== 81+0+59675 (secure 0 0 0) 0x7f7d00061cd0 con 0x7f7d04102320 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "epoch": 337, 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "fsid": "cf170a7a-03e6-11ef-bc93-c7b262605968", 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "created": "2024-04-26T16:07:04.494213+0000", 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "modified": "2024-04-26T16:26:45.159479+0000", 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "last_up_change": "2024-04-26T16:16:46.862501+0000", 2024-04-26T16:26:45.267 INFO:teuthology.orchestra.run.smithi050.stdout: "last_in_change": "2024-04-26T16:26:22.104732+0000", 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_num": 5801984, 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_set": [ 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "noscrub", 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "pglog_hardlimit", 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "purged_snapdirs", 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "recovery_deletes", 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "sortbitwise" 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_version": 79, 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "full_ratio": 0.94999998807907104, 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-26T16:26:45.268 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_snapshot": "", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_max": 271, 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "max_osd": 8, 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "require_min_compat_client": "luminous", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "min_compat_client": "luminous", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "require_osd_release": "squid", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "allow_crimson": false, 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "pools": [ 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 1, 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".mgr", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:12:30.809607+0000", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:26:45.269 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 1, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 1, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 1, 2024-04-26T16:26:45.270 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 1, 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 1, 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "22", 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.271 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.272 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.273 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.274 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_max": 32, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_min": 1 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "mgr": {} 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 7.8899998664855957, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 7.8899998664855957, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.37999999523162842, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 3, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 3, 2024-04-26T16:26:45.275 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 2, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": ".rgw.root", 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:27.029484+0000", 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.276 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.277 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "80", 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:26:45.278 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.279 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.280 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.7400000095367432, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.7400000095367432, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.281 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 3, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.log", 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:29.124965+0000", 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.282 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.283 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.284 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.284 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.284 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.284 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.284 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.284 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "82", 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.285 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.286 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.287 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.288 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 4, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "datapool", 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:30.749319+0000", 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.289 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 3, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 3, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 3, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 3, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 3, 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.290 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "89", 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 2, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 89, 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.291 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.292 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.293 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "rbd": {} 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 2.6700000762939453, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 2.6700000762939453, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.75, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 2, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 2, 2024-04-26T16:26:45.294 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 5, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.control", 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:31.064350+0000", 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.295 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.296 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "84", 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.297 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.298 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.299 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.4900000095367432, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.4900000095367432, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.309999942779541, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.309999942779541, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.300 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 6, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "default.rgw.meta", 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:17:33.192127+0000", 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.301 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.302 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "86", 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.303 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.304 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.305 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "options": { 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_bias": 4 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "rgw": {} 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.306 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 11, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "ceph_test_rados_api_asio", 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:23:28.050102+0000", 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 1, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool", 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.307 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.308 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "142", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "0", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "0", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 0, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 0, 2024-04-26T16:26:45.309 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.310 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.311 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": {}, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.9900000095367432, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.9900000095367432, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.75, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.75, 2024-04-26T16:26:45.312 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 109, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "test-rados-api-smithi050-73372-6", 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:23:42.172881+0000", 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.313 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.314 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "337", 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "337", 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "337", 2024-04-26T16:26:45.315 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "337", 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 14, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 336, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [ 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: 270 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": -1, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": 270, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": 270, 2024-04-26T16:26:45.316 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.317 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.318 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.7400000095367432, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.7400000095367432, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 270, 2024-04-26T16:26:45.319 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_name": "test-rados-api-smithi050-73372-37", 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "create_time": "2024-04-26T16:26:41.167964+0000", 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "flags": 8193, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type": 1, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 3, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "min_size": 2, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_rule": 0, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_count": 0, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_target": 0, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_barrier": 0, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "object_hash": 2, 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_autoscale_mode": "off", 2024-04-26T16:26:45.320 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num": 32, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num": 32, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_placement_num_target": 32, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_target": 32, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_num_pending": 32, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "last_pg_merge_meta": { 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "source_pgid": "0.0", 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "ready_epoch": 0, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_started": 0, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "last_epoch_clean": 0, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "source_version": "0'0", 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "target_version": "0'0" 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "last_change": "337", 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend": "337", 2024-04-26T16:26:45.321 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_prenautilus": "337", 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "last_force_op_resend_preluminous": "337", 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "auid": 0, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_mode": "selfmanaged", 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_seq": 14, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "snap_epoch": 336, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "pool_snaps": [], 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps": "[]", 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_bytes": 0, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "quota_max_objects": 0, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "tiers": [], 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "tier_of": 109, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "read_tier": -1, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "write_tier": -1, 2024-04-26T16:26:45.322 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_mode": "none", 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_bytes": 0, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "target_max_objects": 0, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_flush_age": 0, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "cache_min_evict_age": 0, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profile": "", 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_params": { 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "none" 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_period": 0, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_count": 0, 2024-04-26T16:26:45.323 INFO:teuthology.orchestra.run.smithi050.stdout: "use_gmt_hitset": true, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "min_read_recency_for_promote": 0, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "min_write_recency_for_promote": 0, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_grade_decay_rate": 0, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "hit_set_search_last_n": 0, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "grade_table": [], 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "stripe_width": 0, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "expected_num_objects": 0, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "fast_read": false, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "options": {}, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "application_metadata": { 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "rados": {} 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "read_balance": { 2024-04-26T16:26:45.324 INFO:teuthology.orchestra.run.smithi050.stdout: "score_type": "Fair distribution", 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "score_acting": 1.7400000095367432, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "score_stable": 1.7400000095367432, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "optimal_score": 0.87999999523162842, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_acting": 1.5299999713897705, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "raw_score_stable": 1.5299999713897705, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity_weighted": 1, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity": 1, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "average_primary_affinity_weighted": 1 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-26T16:26:45.325 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "b46ea8aa-cd97-425b-90e0-1e70af46b8f1", 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 8, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 335, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.326 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6802", 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6803", 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.327 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6804", 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6805", 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.328 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6808", 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6809", 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.329 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6806", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6807", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 310950989 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6803/310950989", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6805/310950989", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6809/310950989", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6807/310950989", 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.330 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "7042f8e0-f5d0-48d2-9cd6-d9445e326eed", 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 13, 2024-04-26T16:26:45.331 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 335, 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6810", 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6811", 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.332 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6812", 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6813", 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6816", 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6817", 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.334 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6814", 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6815", 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3516275849 2024-04-26T16:26:45.335 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6811/3516275849", 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6813/3516275849", 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6817/3516275849", 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6815/3516275849", 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "33069daa-7fd8-4e9c-aafe-89030f84aad2", 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 18, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 335, 2024-04-26T16:26:45.337 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6818", 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6819", 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6820", 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6821", 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.340 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6824", 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6825", 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6822", 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6823", 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2835380668 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6819/2835380668", 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6821/2835380668", 2024-04-26T16:26:45.342 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6825/2835380668", 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6823/2835380668", 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "43b6aab5-9b42-4f97-a224-b2872de5cb03", 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.343 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 28, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 335, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6826", 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6827", 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6828", 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.345 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6829", 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6832", 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.346 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6833", 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6830", 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.50:6831", 2024-04-26T16:26:45.347 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2541695077 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.50:6827/2541695077", 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.50:6829/2541695077", 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.50:6833/2541695077", 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.50:6831/2541695077", 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.348 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "f7220491-e9bf-4188-b203-322e9908558c", 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 35, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 334, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.349 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6800", 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6801", 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.350 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6802", 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6803", 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.351 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6806", 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6807", 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.352 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6804", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6805", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2207965503 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6801/2207965503", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6803/2207965503", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6807/2207965503", 2024-04-26T16:26:45.353 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6805/2207965503", 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "ed5c0044-658b-4dfb-8c61-727de6bdd587", 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 0, 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 0, 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.354 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 47, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 312, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6808", 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.355 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6809", 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6810", 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6811", 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6814", 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6815", 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6812", 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6813", 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 3373187844 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6809/3373187844", 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6811/3373187844", 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6815/3373187844", 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6813/3373187844", 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.359 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "94111b7d-7887-4a3a-983b-394294735d0e", 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 57, 2024-04-26T16:26:45.360 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 334, 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6816", 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6817", 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.361 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6818", 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6819", 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.362 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6822", 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6823", 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.363 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6820", 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6821", 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2399532574 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.364 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6817/2399532574", 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6819/2399532574", 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6823/2399532574", 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6821/2399532574", 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "uuid": "abbd95e3-07b4-4235-9101-a4dea0f35bb5", 2024-04-26T16:26:45.365 INFO:teuthology.orchestra.run.smithi050.stdout: "up": 1, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "in": 1, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "weight": 1, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_affinity": 1, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_begin": 0, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "last_clean_end": 0, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "up_from": 69, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "up_thru": 335, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "down_at": 0, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "lost_at": 0, 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addrs": { 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.366 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6824", 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6825", 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addrs": { 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.367 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6826", 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6827", 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addrs": { 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6830", 2024-04-26T16:26:45.368 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6831", 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addrs": { 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "addrvec": [ 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v2", 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6828", 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "type": "v1", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "addr": "172.21.15.190:6829", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "nonce": 2856617796 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "public_addr": "172.21.15.190:6825/2856617796", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "cluster_addr": "172.21.15.190:6827/2856617796", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_back_addr": "172.21.15.190:6831/2856617796", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "heartbeat_front_addr": "172.21.15.190:6829/2856617796", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "state": [ 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "exists", 2024-04-26T16:26:45.370 INFO:teuthology.orchestra.run.smithi050.stdout: "up" 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "osd_xinfo": [ 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 0, 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:26:29.375012+0000", 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 1, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:26:29.693974+0000", 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 2, 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.372 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:26:25.665994+0000", 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 3, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.373 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:25:38.647892+0000", 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 4, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:26:26.490185+0000", 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.374 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 5, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 65536, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:26:19.323971+0000", 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 6, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.375 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:26:43.724821+0000", 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "osd": 7, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "down_stamp": "0.000000", 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_probability": 0, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "laggy_interval": 0, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "features": 4540701547738038271, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "old_weight": 0, 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "last_purged_snaps_scrub": "2024-04-26T16:25:40.176244+0000", 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: "dead_epoch": 0 2024-04-26T16:26:45.376 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap": [ 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.15", 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "osds": [ 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: 2, 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: 6, 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: 5 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_items": [ 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.377 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "2.9", 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 7, 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "3.f", 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 4, 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 2 2024-04-26T16:26:45.378 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "3.13", 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 4, 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 0 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.379 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "pgid": "109.a", 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "mappings": [ 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "from": 3, 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "to": 0 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_upmap_primaries": [], 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "pg_temp": [], 2024-04-26T16:26:45.380 INFO:teuthology.orchestra.run.smithi050.stdout: "primary_temp": [], 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "blocklist": { 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2324264713": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/542146552": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/127073210": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/127073210": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/126446279": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/126446279": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/3454723725": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2548717623": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6800/685732569": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2903276422": "2024-04-27T16:07:37.705234+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:6801/685732569": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/2979962403": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:26:45.381 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/71201882": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/3973398204": "2024-04-27T16:08:14.593126+0000", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/1172169336": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/3444006067": "2024-04-27T16:18:25.317826+0000", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "172.21.15.50:0/577039843": "2024-04-27T16:18:25.317826+0000" 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "range_blocklist": {}, 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "erasure_code_profiles": { 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "default": { 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van" 2024-04-26T16:26:45.382 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "testprofile-RoundTripAppendPP_smithi050-73201-53": { 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-device-class": "", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-failure-domain": "osd", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-num-failure-domains": "0", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-osds-per-failure-domain": "0", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "crush-root": "default", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "k": "2", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "m": "1", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "plugin": "jerasure", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "technique": "reed_sol_van", 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "w": "8" 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.383 INFO:teuthology.orchestra.run.smithi050.stdout: "removed_snaps_queue": [ 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 16, 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 20, 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:26:45.384 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 40, 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 72, 2024-04-26T16:26:45.385 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 99, 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 4, 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:26:45.386 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 7, 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 108, 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.387 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 109, 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 14, 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.389 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 128, 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 1, 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 143, 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.390 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 150, 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 3 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.391 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 157, 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 158, 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.392 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 168, 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 3 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.393 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 173, 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 3, 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 176, 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.394 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 4 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 179, 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.395 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 189, 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 195, 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 3 2024-04-26T16:26:45.396 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 207, 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 2, 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 6 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 215, 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.397 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 7, 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 237, 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 8, 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.398 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 247, 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 10, 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 253, 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 11, 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 2 2024-04-26T16:26:45.399 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "pool": 266, 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "snaps": [ 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "begin": 14, 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "length": 1 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: ], 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "new_removed_snaps": [], 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "new_purged_snaps": [], 2024-04-26T16:26:45.400 INFO:teuthology.orchestra.run.smithi050.stdout: "crush_node_flags": {}, 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "device_class_flags": {}, 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode": { 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_enabled": false, 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_bucket_count": 0, 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "degraded_stretch_mode": 0, 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "recovering_stretch_mode": 0, 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: "stretch_mode_bucket": 0 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:26:45.401 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:26:45.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 10 librados: watch_flush enter 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 10 librados: watch_flush exit 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 20 client.30426.objecter shutdown clearing up homeless session... 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 10 client.30426.objecter successfully canceled tick 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7cf0071f80 msgr2=0x7f7cf0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7cf0071f80 0x7f7cf0074440 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7f7d04079b00 tx=0x7f7cf400a380 comp rx=0 tx=0).stop 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 msgr2=0x7f7d0419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 0x7f7d0419b370 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f7d000057a0 tx=0x7f7d000057d0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 shutdown_connections 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7cf0071f80 0x7f7cf0074440 unknown :-1 s=CLOSED pgs=3718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d04108390 0x7f7d0419fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7d04102320 0x7f7d0419b370 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 --2- 172.21.15.50:0/1406646925 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7d04102ce0 0x7f7d0419b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:26:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.269+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 >> 172.21.15.50:0/1406646925 conn(0x7f7d040792d0 msgr2=0x7f7d0406b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:26:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.273+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 shutdown_connections 2024-04-26T16:26:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.273+0000 7f7d0c061700 1 -- 172.21.15.50:0/1406646925 wait complete. 2024-04-26T16:26:45.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:26:45.273+0000 7f7d0c061700 1 librados: shutdown 2024-04-26T16:26:45.422 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:45.472 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.724570+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.724570+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.724817+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: cluster 2024-04-26T16:26:43.724817+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.161358+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.161358+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: cluster 2024-04-26T16:26:45.165747+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: cluster 2024-04-26T16:26:45.165747+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.168795+0000 mon.c (mon.2) 543 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.168795+0000 mon.c (mon.2) 543 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.169426+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.169426+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.173986+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.173986+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.176516+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:45 smithi190 bash[23868]: audit 2024-04-26T16:26:45.176516+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:45.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:45.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:45.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.724570+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:45.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.724570+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.724817+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: cluster 2024-04-26T16:26:43.724817+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.161358+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.161358+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: cluster 2024-04-26T16:26:45.165747+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: cluster 2024-04-26T16:26:45.165747+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.168795+0000 mon.c (mon.2) 543 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.168795+0000 mon.c (mon.2) 543 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.169426+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.169426+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.173986+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.173986+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.176516+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[21463]: audit 2024-04-26T16:26:45.176516+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.724570+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.724570+0000 osd.6 (osd.6) 165 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.724817+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: cluster 2024-04-26T16:26:43.724817+0000 osd.6 (osd.6) 166 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.161358+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.161358+0000 mon.a (mon.0) 2268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: cluster 2024-04-26T16:26:45.165747+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: cluster 2024-04-26T16:26:45.165747+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e337: 8 total, 8 up, 7 in 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.168795+0000 mon.c (mon.2) 543 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.168795+0000 mon.c (mon.2) 543 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.169426+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.169426+0000 mon.a (mon.0) 2270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:45.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.173986+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.173986+0000 mon.a (mon.0) 2271 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.176516+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:45 smithi050 bash[29234]: audit 2024-04-26T16:26:45.176516+0000 mon.b (mon.1) 225 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]: dispatch 2024-04-26T16:26:45.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:45.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:45.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:45.813 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:45.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:45.945 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:45.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:46.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:46.076 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:46.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:46.200 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:46.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:45.268675+0000 mon.a (mon.0) 2272 : audit [DBG] from='client.? 172.21.15.50:0/1406646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:45.268675+0000 mon.a (mon.0) 2272 : audit [DBG] from='client.? 172.21.15.50:0/1406646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:45.590592+0000 mgr.y (mgr.24401) 371 : cluster [DBG] pgmap v490: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:45.590592+0000 mgr.y (mgr.24401) 371 : cluster [DBG] pgmap v490: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:46.100601+0000 mon.a (mon.0) 2273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:46.100601+0000 mon.a (mon.0) 2273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:46.102114+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:46.102114+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.105075+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.105075+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:46.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.105289+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]': finished 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.105289+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]': finished 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:46.113231+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: cluster 2024-04-26T16:26:46.113231+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.116487+0000 mon.c (mon.2) 544 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.116487+0000 mon.c (mon.2) 544 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.117649+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[23868]: audit 2024-04-26T16:26:46.117649+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:46.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:46.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:45.268675+0000 mon.a (mon.0) 2272 : audit [DBG] from='client.? 172.21.15.50:0/1406646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:45.268675+0000 mon.a (mon.0) 2272 : audit [DBG] from='client.? 172.21.15.50:0/1406646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:45.590592+0000 mgr.y (mgr.24401) 371 : cluster [DBG] pgmap v490: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:45.590592+0000 mgr.y (mgr.24401) 371 : cluster [DBG] pgmap v490: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:46.100601+0000 mon.a (mon.0) 2273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:46.100601+0000 mon.a (mon.0) 2273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:46.102114+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:46.102114+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:46.477 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.105075+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.105075+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.105289+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]': finished 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.105289+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]': finished 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:46.113231+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: cluster 2024-04-26T16:26:46.113231+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.116487+0000 mon.c (mon.2) 544 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.116487+0000 mon.c (mon.2) 544 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.117649+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.478 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[21463]: audit 2024-04-26T16:26:46.117649+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:45.268675+0000 mon.a (mon.0) 2272 : audit [DBG] from='client.? 172.21.15.50:0/1406646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:45.268675+0000 mon.a (mon.0) 2272 : audit [DBG] from='client.? 172.21.15.50:0/1406646925' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:45.590592+0000 mgr.y (mgr.24401) 371 : cluster [DBG] pgmap v490: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:45.590592+0000 mgr.y (mgr.24401) 371 : cluster [DBG] pgmap v490: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:46.100601+0000 mon.a (mon.0) 2273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:46.100601+0000 mon.a (mon.0) 2273 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:46.102114+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:46.102114+0000 mon.a (mon.0) 2274 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.105075+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.105075+0000 mon.a (mon.0) 2275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.105289+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]': finished 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.105289+0000 mon.a (mon.0) 2276 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-37", "mode": "writeback"}]': finished 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:46.113231+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: cluster 2024-04-26T16:26:46.113231+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e338: 8 total, 8 up, 7 in 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.116487+0000 mon.c (mon.2) 544 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.116487+0000 mon.c (mon.2) 544 : audit [INF] from='client.? 172.21.15.50:0/3558671416' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.117649+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.480 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:46 smithi050 bash[29234]: audit 2024-04-26T16:26:46.117649+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]: dispatch 2024-04-26T16:26:46.483 INFO:tasks.thrashosds.thrasher:j is {'epoch': 337, 'fsid': 'cf170a7a-03e6-11ef-bc93-c7b262605968', 'created': '2024-04-26T16:07:04.494213+0000', 'modified': '2024-04-26T16:26:45.159479+0000', 'last_up_change': '2024-04-26T16:16:46.862501+0000', 'last_in_change': '2024-04-26T16:26:22.104732+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 79, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 271, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T16:12:30.809607+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-04-26T16:17:27.029484+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'default.rgw.log', 'create_time': '2024-04-26T16:17:29.124965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'datapool', 'create_time': '2024-04-26T16:17:30.749319+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 89, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-04-26T16:17:31.064350+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-04-26T16:17:33.192127+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'ceph_test_rados_api_asio', 'create_time': '2024-04-26T16:23:28.050102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 109, 'pool_name': 'test-rados-api-smithi050-73372-6', 'create_time': '2024-04-26T16:23:42.172881+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '337', 'last_force_op_resend': '337', 'last_force_op_resend_prenautilus': '337', 'last_force_op_resend_preluminous': '337', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 336, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [270], 'tier_of': -1, 'read_tier': 270, 'write_tier': 270, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 270, 'pool_name': 'test-rados-api-smithi050-73372-37', 'create_time': '2024-04-26T16:26:41.167964+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '337', 'last_force_op_resend': '337', 'last_force_op_resend_prenautilus': '337', 'last_force_op_resend_preluminous': '337', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 14, 'snap_epoch': 336, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': 109, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'b46ea8aa-cd97-425b-90e0-1e70af46b8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6802', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6803', 'nonce': 310950989}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6804', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6805', 'nonce': 310950989}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6808', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6809', 'nonce': 310950989}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6806', 'nonce': 310950989}, {'type': 'v1', 'addr': '172.21.15.50:6807', 'nonce': 310950989}]}, 'public_addr': '172.21.15.50:6803/310950989', 'cluster_addr': '172.21.15.50:6805/310950989', 'heartbeat_back_addr': '172.21.15.50:6809/310950989', 'heartbeat_front_addr': '172.21.15.50:6807/310950989', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '7042f8e0-f5d0-48d2-9cd6-d9445e326eed', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6810', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6811', 'nonce': 3516275849}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6812', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6813', 'nonce': 3516275849}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6816', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6817', 'nonce': 3516275849}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6814', 'nonce': 3516275849}, {'type': 'v1', 'addr': '172.21.15.50:6815', 'nonce': 3516275849}]}, 'public_addr': '172.21.15.50:6811/3516275849', 'cluster_addr': '172.21.15.50:6813/3516275849', 'heartbeat_back_addr': '172.21.15.50:6817/3516275849', 'heartbeat_front_addr': '172.21.15.50:6815/3516275849', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '33069daa-7fd8-4e9c-aafe-89030f84aad2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6818', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6819', 'nonce': 2835380668}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6820', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6821', 'nonce': 2835380668}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6824', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6825', 'nonce': 2835380668}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6822', 'nonce': 2835380668}, {'type': 'v1', 'addr': '172.21.15.50:6823', 'nonce': 2835380668}]}, 'public_addr': '172.21.15.50:6819/2835380668', 'cluster_addr': '172.21.15.50:6821/2835380668', 'heartbeat_back_addr': '172.21.15.50:6825/2835380668', 'heartbeat_front_addr': '172.21.15.50:6823/2835380668', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '43b6aab5-9b42-4f97-a224-b2872de5cb03', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 28, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6826', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6827', 'nonce': 2541695077}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6828', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6829', 'nonce': 2541695077}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6832', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6833', 'nonce': 2541695077}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.50:6830', 'nonce': 2541695077}, {'type': 'v1', 'addr': '172.21.15.50:6831', 'nonce': 2541695077}]}, 'public_addr': '172.21.15.50:6827/2541695077', 'cluster_addr': '172.21.15.50:6829/2541695077', 'heartbeat_back_addr': '172.21.15.50:6833/2541695077', 'heartbeat_front_addr': '172.21.15.50:6831/2541695077', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'f7220491-e9bf-4188-b203-322e9908558c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 35, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6801', 'nonce': 2207965503}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6803', 'nonce': 2207965503}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6807', 'nonce': 2207965503}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 2207965503}, {'type': 'v1', 'addr': '172.21.15.190:6805', 'nonce': 2207965503}]}, 'public_addr': '172.21.15.190:6801/2207965503', 'cluster_addr': '172.21.15.190:6803/2207965503', 'heartbeat_back_addr': '172.21.15.190:6807/2207965503', 'heartbeat_front_addr': '172.21.15.190:6805/2207965503', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'ed5c0044-658b-4dfb-8c61-727de6bdd587', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 312, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6809', 'nonce': 3373187844}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6811', 'nonce': 3373187844}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6815', 'nonce': 3373187844}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 3373187844}, {'type': 'v1', 'addr': '172.21.15.190:6813', 'nonce': 3373187844}]}, 'public_addr': '172.21.15.190:6809/3373187844', 'cluster_addr': '172.21.15.190:6811/3373187844', 'heartbeat_back_addr': '172.21.15.190:6815/3373187844', 'heartbeat_front_addr': '172.21.15.190:6813/3373187844', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '94111b7d-7887-4a3a-983b-394294735d0e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 57, 'up_thru': 334, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6816', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6817', 'nonce': 2399532574}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6818', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6819', 'nonce': 2399532574}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6822', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6823', 'nonce': 2399532574}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6820', 'nonce': 2399532574}, {'type': 'v1', 'addr': '172.21.15.190:6821', 'nonce': 2399532574}]}, 'public_addr': '172.21.15.190:6817/2399532574', 'cluster_addr': '172.21.15.190:6819/2399532574', 'heartbeat_back_addr': '172.21.15.190:6823/2399532574', 'heartbeat_front_addr': '172.21.15.190:6821/2399532574', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'abbd95e3-07b4-4235-9101-a4dea0f35bb5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 69, 'up_thru': 335, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6824', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6825', 'nonce': 2856617796}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6826', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6827', 'nonce': 2856617796}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6830', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6831', 'nonce': 2856617796}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6828', 'nonce': 2856617796}, {'type': 'v1', 'addr': '172.21.15.190:6829', 'nonce': 2856617796}]}, 'public_addr': '172.21.15.190:6825/2856617796', 'cluster_addr': '172.21.15.190:6827/2856617796', 'heartbeat_back_addr': '172.21.15.190:6831/2856617796', 'heartbeat_front_addr': '172.21.15.190:6829/2856617796', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:26:29.375012+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:26:29.693974+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:26:25.665994+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:25:38.647892+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:26:26.490185+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-04-26T16:26:19.323971+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:26:43.724821+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-04-26T16:25:40.176244+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.15', 'osds': [2, 6, 5]}], 'pg_upmap_items': [{'pgid': '2.9', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '3.f', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.13', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '109.a', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.50:0/2324264713': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/542146552': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:6801/127073210': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6800/127073210': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6800/126446279': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:6801/126446279': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/3454723725': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:0/2548717623': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6800/685732569': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/2903276422': '2024-04-27T16:07:37.705234+0000', '172.21.15.50:6801/685732569': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/2979962403': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/71201882': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/3973398204': '2024-04-27T16:08:14.593126+0000', '172.21.15.50:0/1172169336': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/3444006067': '2024-04-27T16:18:25.317826+0000', '172.21.15.50:0/577039843': '2024-04-27T16:18:25.317826+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'testprofile-RoundTripAppendPP_smithi050-73201-53': {'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'jerasure-per-chunk-alignment': 'false', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van', 'w': '8'}}, 'removed_snaps_queue': [{'pool': 16, 'snaps': [{'begin': 1, 'length': 4}]}, {'pool': 20, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 40, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 72, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 99, 'snaps': [{'begin': 4, 'length': 2}, {'begin': 7, 'length': 2}]}, {'pool': 108, 'snaps': [{'begin': 1, 'length': 1}]}, {'pool': 109, 'snaps': [{'begin': 14, 'length': 1}]}, {'pool': 128, 'snaps': [{'begin': 1, 'length': 2}]}, {'pool': 143, 'snaps': [{'begin': 3, 'length': 1}]}, {'pool': 150, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 157, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 158, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 168, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 173, 'snaps': [{'begin': 3, 'length': 4}]}, {'pool': 176, 'snaps': [{'begin': 2, 'length': 4}]}, {'pool': 179, 'snaps': [{'begin': 2, 'length': 2}]}, {'pool': 189, 'snaps': [{'begin': 2, 'length': 1}]}, {'pool': 195, 'snaps': [{'begin': 2, 'length': 3}]}, {'pool': 207, 'snaps': [{'begin': 2, 'length': 6}]}, {'pool': 215, 'snaps': [{'begin': 7, 'length': 1}]}, {'pool': 237, 'snaps': [{'begin': 8, 'length': 1}]}, {'pool': 247, 'snaps': [{'begin': 10, 'length': 1}]}, {'pool': 253, 'snaps': [{'begin': 11, 'length': 2}]}, {'pool': 266, 'snaps': [{'begin': 14, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-04-26T16:26:46.483 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.15 2024-04-26T16:26:46.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd rm-pg-upmap 2.15 2024-04-26T16:26:46.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:46.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:46.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:46.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:46.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:46.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:46.850 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:46.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:46.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:46.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:47.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:26:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:26:47.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:47.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:47.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:47.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:46.215520+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:46.215520+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:46.218139+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:46.218139+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.107161+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.107161+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.107265+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.107265+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.116698+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.116698+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.117008+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.117008+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.117550+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.117550+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.136996+0000 mon.c (mon.2) 545 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.136996+0000 mon.c (mon.2) 545 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.137488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.137488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.138194+0000 mon.c (mon.2) 546 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.138194+0000 mon.c (mon.2) 546 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.138474+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.138474+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.138945+0000 mon.c (mon.2) 547 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.138945+0000 mon.c (mon.2) 547 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.139244+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[29234]: audit 2024-04-26T16:26:47.139244+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:46.215520+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:46.215520+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:46.218139+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:46.218139+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.107161+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.107161+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.107265+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.107265+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:47.231 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.116698+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.116698+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.117008+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.117008+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.117550+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.117550+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.136996+0000 mon.c (mon.2) 545 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.136996+0000 mon.c (mon.2) 545 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.137488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.137488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.138194+0000 mon.c (mon.2) 546 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.138194+0000 mon.c (mon.2) 546 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.138474+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.138474+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.138945+0000 mon.c (mon.2) 547 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.138945+0000 mon.c (mon.2) 547 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.139244+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:47 smithi050 bash[21463]: audit 2024-04-26T16:26:47.139244+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:47.330 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:26:47.330 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:26:47.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:47.365 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:47.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:47.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:46.215520+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:46.215520+0000 mon.a (mon.0) 2279 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:46.218139+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:46.218139+0000 mon.b (mon.1) 226 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.107161+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.107161+0000 mon.a (mon.0) 2280 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi050-73201-53"}]': finished 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.107265+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.107265+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.116698+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.116698+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e339: 8 total, 8 up, 7 in 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.117008+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.117008+0000 mon.b (mon.1) 227 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.117550+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.117550+0000 mon.a (mon.0) 2283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.136996+0000 mon.c (mon.2) 545 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.136996+0000 mon.c (mon.2) 545 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.137488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.137488+0000 mon.a (mon.0) 2284 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.138194+0000 mon.c (mon.2) 546 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.138194+0000 mon.c (mon.2) 546 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.138474+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.138474+0000 mon.a (mon.0) 2285 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.138945+0000 mon.c (mon.2) 547 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.138945+0000 mon.c (mon.2) 547 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.139244+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:47 smithi190 bash[23868]: audit 2024-04-26T16:26:47.139244+0000 mon.a (mon.0) 2286 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:47.490 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:47.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:47.620 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:47.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:47.752 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:47.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:47.880 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:47.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:47.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:48.012 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:48.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:48.143 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:48.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-26T16:26:48.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:48.274 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:48.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:48.396 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:46.778788+0000 mgr.y (mgr.24401) 372 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:46.778788+0000 mgr.y (mgr.24401) 372 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.591464+0000 mgr.y (mgr.24401) 373 : cluster [DBG] pgmap v493: 228 pgs: 228 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.591464+0000 mgr.y (mgr.24401) 373 : cluster [DBG] pgmap v493: 228 pgs: 228 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.777190+0000 osd.3 (osd.3) 233 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.777190+0000 osd.3 (osd.3) 233 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:48.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.777533+0000 osd.3 (osd.3) 234 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:47.777533+0000 osd.3 (osd.3) 234 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:48.107957+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:48.107957+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.110516+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.110516+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.110743+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.110743+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:48.118345+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: cluster 2024-04-26T16:26:48.118345+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.126403+0000 mon.c (mon.2) 548 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.126403+0000 mon.c (mon.2) 548 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.127309+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.399 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:48 smithi190 bash[23868]: audit 2024-04-26T16:26:48.127309+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:48.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:48.526 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:48.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:46.778788+0000 mgr.y (mgr.24401) 372 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:46.778788+0000 mgr.y (mgr.24401) 372 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.591464+0000 mgr.y (mgr.24401) 373 : cluster [DBG] pgmap v493: 228 pgs: 228 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-26T16:26:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.591464+0000 mgr.y (mgr.24401) 373 : cluster [DBG] pgmap v493: 228 pgs: 228 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-26T16:26:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.777190+0000 osd.3 (osd.3) 233 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.777190+0000 osd.3 (osd.3) 233 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.777533+0000 osd.3 (osd.3) 234 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:47.777533+0000 osd.3 (osd.3) 234 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:48.107957+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:48.107957+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.110516+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.110516+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.110743+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.110743+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:48.118345+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: cluster 2024-04-26T16:26:48.118345+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-04-26T16:26:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.126403+0000 mon.c (mon.2) 548 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:46.778788+0000 mgr.y (mgr.24401) 372 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:46.778788+0000 mgr.y (mgr.24401) 372 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.591464+0000 mgr.y (mgr.24401) 373 : cluster [DBG] pgmap v493: 228 pgs: 228 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.591464+0000 mgr.y (mgr.24401) 373 : cluster [DBG] pgmap v493: 228 pgs: 228 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.777190+0000 osd.3 (osd.3) 233 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.777190+0000 osd.3 (osd.3) 233 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.777533+0000 osd.3 (osd.3) 234 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:47.777533+0000 osd.3 (osd.3) 234 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:48.107957+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:48.107957+0000 mon.a (mon.0) 2287 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.110516+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.110516+0000 mon.a (mon.0) 2288 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-37"}]': finished 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.110743+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.110743+0000 mon.a (mon.0) 2289 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi050-73201-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:48.118345+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: cluster 2024-04-26T16:26:48.118345+0000 mon.a (mon.0) 2290 : cluster [DBG] osdmap e340: 8 total, 8 up, 7 in 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.126403+0000 mon.c (mon.2) 548 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.126403+0000 mon.c (mon.2) 548 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.127309+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[29234]: audit 2024-04-26T16:26:48.127309+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.126403+0000 mon.c (mon.2) 548 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.127309+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:48 smithi050 bash[21463]: audit 2024-04-26T16:26:48.127309+0000 mon.a (mon.0) 2291 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:48.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:48.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:48.790 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:48.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:48.920 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:48.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:49.000 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:49.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.048 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:49.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.177 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:49.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.308 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:49.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:49.439 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:49.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.569 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:49.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.700 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:49.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:49.833 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:49.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:49.961 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:49.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:50.019 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:50.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:50.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:50.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.215 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:50.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.345 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[21463]: cluster 2024-04-26T16:26:49.132269+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[21463]: cluster 2024-04-26T16:26:49.132269+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[21463]: cluster 2024-04-26T16:26:49.592286+0000 mgr.y (mgr.24401) 374 : cluster [DBG] pgmap v496: 196 pgs: 196 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[21463]: cluster 2024-04-26T16:26:49.592286+0000 mgr.y (mgr.24401) 374 : cluster [DBG] pgmap v496: 196 pgs: 196 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[29234]: cluster 2024-04-26T16:26:49.132269+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[29234]: cluster 2024-04-26T16:26:49.132269+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[29234]: cluster 2024-04-26T16:26:49.592286+0000 mgr.y (mgr.24401) 374 : cluster [DBG] pgmap v496: 196 pgs: 196 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:26:50.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:50 smithi050 bash[29234]: cluster 2024-04-26T16:26:49.592286+0000 mgr.y (mgr.24401) 374 : cluster [DBG] pgmap v496: 196 pgs: 196 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:26:50.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.459 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:50 smithi190 bash[23868]: cluster 2024-04-26T16:26:49.132269+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-04-26T16:26:50.459 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:50 smithi190 bash[23868]: cluster 2024-04-26T16:26:49.132269+0000 mon.a (mon.0) 2292 : cluster [DBG] osdmap e341: 8 total, 8 up, 7 in 2024-04-26T16:26:50.459 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:50 smithi190 bash[23868]: cluster 2024-04-26T16:26:49.592286+0000 mgr.y (mgr.24401) 374 : cluster [DBG] pgmap v496: 196 pgs: 196 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:26:50.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:50 smithi190 bash[23868]: cluster 2024-04-26T16:26:49.592286+0000 mgr.y (mgr.24401) 374 : cluster [DBG] pgmap v496: 196 pgs: 196 active+clean; 4.6 MiB data, 531 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:26:50.476 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:50.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.608 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:50.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:50.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:50.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:50.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:50.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:50.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:51.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:51.045 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:26:51.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:51.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: audit 2024-04-26T16:26:50.121393+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:51.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: audit 2024-04-26T16:26:50.121393+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: cluster 2024-04-26T16:26:50.126168+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: cluster 2024-04-26T16:26:50.126168+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: audit 2024-04-26T16:26:50.138973+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: audit 2024-04-26T16:26:50.138973+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: audit 2024-04-26T16:26:50.141256+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: audit 2024-04-26T16:26:50.141256+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: cluster 2024-04-26T16:26:51.104175+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:51.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[21463]: cluster 2024-04-26T16:26:51.104175+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:51.148 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:51.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:51.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: audit 2024-04-26T16:26:50.121393+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: audit 2024-04-26T16:26:50.121393+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: cluster 2024-04-26T16:26:50.126168+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: cluster 2024-04-26T16:26:50.126168+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: audit 2024-04-26T16:26:50.138973+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: audit 2024-04-26T16:26:50.138973+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: audit 2024-04-26T16:26:50.141256+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: audit 2024-04-26T16:26:50.141256+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: cluster 2024-04-26T16:26:51.104175+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:51 smithi050 bash[29234]: cluster 2024-04-26T16:26:51.104175+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:51.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:51.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: audit 2024-04-26T16:26:50.121393+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: audit 2024-04-26T16:26:50.121393+0000 mon.a (mon.0) 2293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi050-73201-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: cluster 2024-04-26T16:26:50.126168+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: cluster 2024-04-26T16:26:50.126168+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e342: 8 total, 8 up, 7 in 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: audit 2024-04-26T16:26:50.138973+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: audit 2024-04-26T16:26:50.138973+0000 mon.a (mon.0) 2295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: audit 2024-04-26T16:26:50.141256+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: audit 2024-04-26T16:26:50.141256+0000 mon.b (mon.1) 228 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: cluster 2024-04-26T16:26:51.104175+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:51.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:51 smithi190 bash[23868]: cluster 2024-04-26T16:26:51.104175+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:51.409 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:51.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:51.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:51.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:51.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:51.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:51.805 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:51.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:51.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:51.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:52.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:52.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:52.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:52.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:52.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:52.303 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: audit 2024-04-26T16:26:51.124465+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: audit 2024-04-26T16:26:51.124465+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: cluster 2024-04-26T16:26:51.128988+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: cluster 2024-04-26T16:26:51.128988+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: audit 2024-04-26T16:26:51.144342+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: audit 2024-04-26T16:26:51.144342+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: audit 2024-04-26T16:26:51.146393+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: audit 2024-04-26T16:26:51.146393+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: cluster 2024-04-26T16:26:51.593489+0000 mgr.y (mgr.24401) 375 : cluster [DBG] pgmap v499: 236 pgs: 236 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:26:52.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:52 smithi190 bash[23868]: cluster 2024-04-26T16:26:51.593489+0000 mgr.y (mgr.24401) 375 : cluster [DBG] pgmap v499: 236 pgs: 236 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:26:52.318 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:52.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: audit 2024-04-26T16:26:51.124465+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: audit 2024-04-26T16:26:51.124465+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: cluster 2024-04-26T16:26:51.128988+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: cluster 2024-04-26T16:26:51.128988+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: audit 2024-04-26T16:26:51.144342+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: audit 2024-04-26T16:26:51.144342+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: audit 2024-04-26T16:26:51.146393+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: audit 2024-04-26T16:26:51.146393+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: cluster 2024-04-26T16:26:51.593489+0000 mgr.y (mgr.24401) 375 : cluster [DBG] pgmap v499: 236 pgs: 236 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[21463]: cluster 2024-04-26T16:26:51.593489+0000 mgr.y (mgr.24401) 375 : cluster [DBG] pgmap v499: 236 pgs: 236 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: audit 2024-04-26T16:26:51.124465+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: audit 2024-04-26T16:26:51.124465+0000 mon.a (mon.0) 2297 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: cluster 2024-04-26T16:26:51.128988+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: cluster 2024-04-26T16:26:51.128988+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e343: 8 total, 8 up, 7 in 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: audit 2024-04-26T16:26:51.144342+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: audit 2024-04-26T16:26:51.144342+0000 mon.a (mon.0) 2299 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: audit 2024-04-26T16:26:51.146393+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: audit 2024-04-26T16:26:51.146393+0000 mon.b (mon.1) 229 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: cluster 2024-04-26T16:26:51.593489+0000 mgr.y (mgr.24401) 375 : cluster [DBG] pgmap v499: 236 pgs: 236 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:26:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:52 smithi050 bash[29234]: cluster 2024-04-26T16:26:51.593489+0000 mgr.y (mgr.24401) 375 : cluster [DBG] pgmap v499: 236 pgs: 236 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:26:52.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:52.450 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:52.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:52.580 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:52.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:52.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:52.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:52.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:52.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:52.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:52.955 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:26:52.973 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:52.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:53.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:53.104 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:53.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:53.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:53.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.142553+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:53.331 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.142553+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:53.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: cluster 2024-04-26T16:26:52.147956+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: cluster 2024-04-26T16:26:52.147956+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.151819+0000 mon.c (mon.2) 549 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.151819+0000 mon.c (mon.2) 549 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.152428+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.152428+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.155935+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.155935+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.158359+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:53 smithi190 bash[23868]: audit 2024-04-26T16:26:52.158359+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.361 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:53.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:53.488 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:53.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.142553+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.142553+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: cluster 2024-04-26T16:26:52.147956+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: cluster 2024-04-26T16:26:52.147956+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.151819+0000 mon.c (mon.2) 549 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.151819+0000 mon.c (mon.2) 549 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.152428+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.152428+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.155935+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.155935+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.158359+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[29234]: audit 2024-04-26T16:26:52.158359+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.142553+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.142553+0000 mon.a (mon.0) 2300 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: cluster 2024-04-26T16:26:52.147956+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: cluster 2024-04-26T16:26:52.147956+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e344: 8 total, 8 up, 7 in 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.151819+0000 mon.c (mon.2) 549 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.151819+0000 mon.c (mon.2) 549 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.152428+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.152428+0000 mon.a (mon.0) 2302 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.155935+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.155935+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.158359+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:53 smithi050 bash[21463]: audit 2024-04-26T16:26:52.158359+0000 mon.b (mon.1) 230 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:53.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:53.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:53.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:53.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:53.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:53.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:53.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:53.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.013 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:54.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:54.145 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:54.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:54.272 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:54.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.156616+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.156616+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.156886+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.156886+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.164480+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.164480+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: cluster 2024-04-26T16:26:53.164766+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: cluster 2024-04-26T16:26:53.164766+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.165070+0000 mon.c (mon.2) 550 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.165070+0000 mon.c (mon.2) 550 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.170937+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.170937+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.171470+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: audit 2024-04-26T16:26:53.171470+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: cluster 2024-04-26T16:26:53.594054+0000 mgr.y (mgr.24401) 376 : cluster [DBG] pgmap v502: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:54 smithi190 bash[23868]: cluster 2024-04-26T16:26:53.594054+0000 mgr.y (mgr.24401) 376 : cluster [DBG] pgmap v502: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:54.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.156616+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:54.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.156616+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:54.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.156886+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:54.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.156886+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:54.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.164480+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.164480+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: cluster 2024-04-26T16:26:53.164766+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: cluster 2024-04-26T16:26:53.164766+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.165070+0000 mon.c (mon.2) 550 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.165070+0000 mon.c (mon.2) 550 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.170937+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.170937+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.171470+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: audit 2024-04-26T16:26:53.171470+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: cluster 2024-04-26T16:26:53.594054+0000 mgr.y (mgr.24401) 376 : cluster [DBG] pgmap v502: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21463]: cluster 2024-04-26T16:26:53.594054+0000 mgr.y (mgr.24401) 376 : cluster [DBG] pgmap v502: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.156616+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:54.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.156616+0000 mon.a (mon.0) 2304 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.156886+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.156886+0000 mon.a (mon.0) 2305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.164480+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.164480+0000 mon.b (mon.1) 231 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: cluster 2024-04-26T16:26:53.164766+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: cluster 2024-04-26T16:26:53.164766+0000 mon.a (mon.0) 2306 : cluster [DBG] osdmap e345: 8 total, 8 up, 7 in 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.165070+0000 mon.c (mon.2) 550 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.165070+0000 mon.c (mon.2) 550 : audit [INF] from='client.? 172.21.15.50:0/2476147740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.170937+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.170937+0000 mon.a (mon.0) 2307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.171470+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: audit 2024-04-26T16:26:53.171470+0000 mon.a (mon.0) 2308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]: dispatch 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: cluster 2024-04-26T16:26:53.594054+0000 mgr.y (mgr.24401) 376 : cluster [DBG] pgmap v502: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:54.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[29234]: cluster 2024-04-26T16:26:53.594054+0000 mgr.y (mgr.24401) 376 : cluster [DBG] pgmap v502: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:26:54.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:54.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:54.521 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:26:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:26:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:26:54.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:54.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:54.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.657 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:54.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.789 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:54.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:54.918 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:54.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:55.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:55.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:55.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:55.178 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:55.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:55.307 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:55.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: cluster 2024-04-26T16:26:54.158522+0000 mon.a (mon.0) 2309 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: cluster 2024-04-26T16:26:54.158522+0000 mon.a (mon.0) 2309 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.168967+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]': finished 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.168967+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]': finished 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.169144+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.169144+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: cluster 2024-04-26T16:26:54.180478+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-04-26T16:26:55.414 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: cluster 2024-04-26T16:26:54.180478+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.210513+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.210513+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.212434+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.212434+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.212973+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.212973+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.322105+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.322105+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.324615+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.415 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[29234]: audit 2024-04-26T16:26:54.324615+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: cluster 2024-04-26T16:26:54.158522+0000 mon.a (mon.0) 2309 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: cluster 2024-04-26T16:26:54.158522+0000 mon.a (mon.0) 2309 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.168967+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]': finished 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.168967+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]': finished 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.169144+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.169144+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: cluster 2024-04-26T16:26:54.180478+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: cluster 2024-04-26T16:26:54.180478+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.210513+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.210513+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.212434+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.212434+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.212973+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.212973+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.322105+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.322105+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.324615+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:55 smithi050 bash[21463]: audit 2024-04-26T16:26:54.324615+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:55.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: cluster 2024-04-26T16:26:54.158522+0000 mon.a (mon.0) 2309 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: cluster 2024-04-26T16:26:54.158522+0000 mon.a (mon.0) 2309 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.168967+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]': finished 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.168967+0000 mon.a (mon.0) 2310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-39", "mode": "writeback"}]': finished 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.169144+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.169144+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi050-73201-54"}]': finished 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: cluster 2024-04-26T16:26:54.180478+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-04-26T16:26:55.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: cluster 2024-04-26T16:26:54.180478+0000 mon.a (mon.0) 2312 : cluster [DBG] osdmap e346: 8 total, 8 up, 7 in 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.210513+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.210513+0000 mon.a (mon.0) 2313 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.212434+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.212434+0000 mon.a (mon.0) 2314 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.212973+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.212973+0000 mon.a (mon.0) 2315 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.322105+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.322105+0000 mon.a (mon.0) 2316 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.324615+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:55 smithi190 bash[23868]: audit 2024-04-26T16:26:54.324615+0000 mon.b (mon.1) 232 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:26:55.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:55.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:55.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:55.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:55.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:55.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:55.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:55.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:56.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:56.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:56.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:56.212 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.183848+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.183848+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.184104+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.184104+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.192089+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.192089+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.193027+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.193027+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.213 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.196448+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:56.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:56.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:56.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:56.361 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:26:56.362 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:26:56.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:56.463 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.183848+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:56.463 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.183848+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:56.463 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.184104+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:56.463 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.184104+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:56.463 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.192089+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.192089+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.193027+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.193027+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.196448+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.196448+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.199031+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:55.199031+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.204805+0000 osd.7 (osd.7) 229 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.204805+0000 osd.7 (osd.7) 229 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.205041+0000 osd.7 (osd.7) 230 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.205041+0000 osd.7 (osd.7) 230 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.594844+0000 mgr.y (mgr.24401) 377 : cluster [DBG] pgmap v505: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:56.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:55.594844+0000 mgr.y (mgr.24401) 377 : cluster [DBG] pgmap v505: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:56.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:56.106666+0000 mon.a (mon.0) 2322 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:56.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:56.106666+0000 mon.a (mon.0) 2322 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:56.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:56.108621+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:56.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: audit 2024-04-26T16:26:56.108621+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:56.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:56.122108+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-04-26T16:26:56.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[23868]: cluster 2024-04-26T16:26:56.122108+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-04-26T16:26:56.475 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:56.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.196448+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.199031+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:55.199031+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.204805+0000 osd.7 (osd.7) 229 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.204805+0000 osd.7 (osd.7) 229 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.205041+0000 osd.7 (osd.7) 230 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.205041+0000 osd.7 (osd.7) 230 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.594844+0000 mgr.y (mgr.24401) 377 : cluster [DBG] pgmap v505: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:55.594844+0000 mgr.y (mgr.24401) 377 : cluster [DBG] pgmap v505: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:56.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:56.106666+0000 mon.a (mon.0) 2322 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:56.106666+0000 mon.a (mon.0) 2322 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:56.108621+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: audit 2024-04-26T16:26:56.108621+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:56.122108+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[21463]: cluster 2024-04-26T16:26:56.122108+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.183848+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.183848+0000 mon.a (mon.0) 2317 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi050-73201-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:26:56.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.184104+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.184104+0000 mon.a (mon.0) 2318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.192089+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.192089+0000 mon.a (mon.0) 2319 : cluster [DBG] osdmap e347: 8 total, 8 up, 7 in 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.193027+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.193027+0000 mon.b (mon.1) 233 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.196448+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.196448+0000 mon.a (mon.0) 2320 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.199031+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:55.199031+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]: dispatch 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.204805+0000 osd.7 (osd.7) 229 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.204805+0000 osd.7 (osd.7) 229 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:26:56.584 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.205041+0000 osd.7 (osd.7) 230 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.205041+0000 osd.7 (osd.7) 230 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.594844+0000 mgr.y (mgr.24401) 377 : cluster [DBG] pgmap v505: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:55.594844+0000 mgr.y (mgr.24401) 377 : cluster [DBG] pgmap v505: 228 pgs: 228 active+clean; 4.6 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:56.106666+0000 mon.a (mon.0) 2322 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:56.106666+0000 mon.a (mon.0) 2322 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:56.108621+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: audit 2024-04-26T16:26:56.108621+0000 mon.a (mon.0) 2323 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-39"}]': finished 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:56.122108+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-04-26T16:26:56.585 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:56 smithi050 bash[29234]: cluster 2024-04-26T16:26:56.122108+0000 mon.a (mon.0) 2324 : cluster [DBG] osdmap e348: 8 total, 8 up, 7 in 2024-04-26T16:26:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:56.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:56.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:56.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:56.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:56.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:56.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:56.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:26:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:26:56.995 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:56.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:56.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:57.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:57.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: seed 73372 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (451 ms) 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10414 ms) 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (153 ms) 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11018 ms total) 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 2024-04-26T16:26:57.133 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (6962 ms) 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (8024 ms) 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9980 ms) 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: my_snaps [3] 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: my_snaps [4,3] 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: my_snaps [5,4,3] 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: promoting some heads 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: promoting from clones for snap 6 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: promoting from clones for snap 5 2024-04-26T16:26:57.134 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: promoting from clones for snap 4 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: promoting from clones for snap 3 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: waiting for scrubs... 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: done waiting 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46050 ms) 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10121 ms) 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7877 ms) 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7997 ms) 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8010 ms) 2024-04-26T16:26:57.135 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (9989 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9005 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10010 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13047 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7970 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8998 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13043 ms) 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-04-26T16:26:57.136 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7976 ms) 2024-04-26T16:26:57.137 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-04-26T16:26:57.137 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8006 ms) 2024-04-26T16:26:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:57.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-26T16:26:57.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:57.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:57.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:57 smithi190 bash[23868]: audit 2024-04-26T16:26:56.541270+0000 mon.c (mon.2) 551 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:57.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:57 smithi190 bash[23868]: audit 2024-04-26T16:26:56.541270+0000 mon.c (mon.2) 551 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:57.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:57 smithi190 bash[23868]: audit 2024-04-26T16:26:57.112020+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:26:57.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:57 smithi190 bash[23868]: audit 2024-04-26T16:26:57.112020+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:26:57.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:57 smithi190 bash[23868]: cluster 2024-04-26T16:26:57.127156+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-04-26T16:26:57.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:57 smithi190 bash[23868]: cluster 2024-04-26T16:26:57.127156+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-04-26T16:26:57.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:57.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[21463]: audit 2024-04-26T16:26:56.541270+0000 mon.c (mon.2) 551 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:57.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[21463]: audit 2024-04-26T16:26:56.541270+0000 mon.c (mon.2) 551 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[21463]: audit 2024-04-26T16:26:57.112020+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[21463]: audit 2024-04-26T16:26:57.112020+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[21463]: cluster 2024-04-26T16:26:57.127156+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[21463]: cluster 2024-04-26T16:26:57.127156+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[29234]: audit 2024-04-26T16:26:56.541270+0000 mon.c (mon.2) 551 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[29234]: audit 2024-04-26T16:26:56.541270+0000 mon.c (mon.2) 551 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[29234]: audit 2024-04-26T16:26:57.112020+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[29234]: audit 2024-04-26T16:26:57.112020+0000 mon.a (mon.0) 2325 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi050-73201-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[29234]: cluster 2024-04-26T16:26:57.127156+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-04-26T16:26:57.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:57 smithi050 bash[29234]: cluster 2024-04-26T16:26:57.127156+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e349: 8 total, 8 up, 7 in 2024-04-26T16:26:57.384 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:26:57.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:57.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:57.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:57.635 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:57.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:57.762 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:57.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:57.893 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:57.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:57.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:58.025 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:58.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:58.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:58.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:58.284 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:58.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:58.387 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:58.397 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: audit 2024-04-26T16:26:56.790182+0000 mgr.y (mgr.24401) 378 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:58.397 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: audit 2024-04-26T16:26:56.790182+0000 mgr.y (mgr.24401) 378 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:58.397 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: cluster 2024-04-26T16:26:57.595690+0000 mgr.y (mgr.24401) 379 : cluster [DBG] pgmap v508: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:26:58.397 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: cluster 2024-04-26T16:26:57.595690+0000 mgr.y (mgr.24401) 379 : cluster [DBG] pgmap v508: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: cluster 2024-04-26T16:26:58.125477+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: cluster 2024-04-26T16:26:58.125477+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: audit 2024-04-26T16:26:58.132419+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: audit 2024-04-26T16:26:58.132419+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: audit 2024-04-26T16:26:58.134642+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[29234]: audit 2024-04-26T16:26:58.134642+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: audit 2024-04-26T16:26:56.790182+0000 mgr.y (mgr.24401) 378 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: audit 2024-04-26T16:26:56.790182+0000 mgr.y (mgr.24401) 378 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:58.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: cluster 2024-04-26T16:26:57.595690+0000 mgr.y (mgr.24401) 379 : cluster [DBG] pgmap v508: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: cluster 2024-04-26T16:26:57.595690+0000 mgr.y (mgr.24401) 379 : cluster [DBG] pgmap v508: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: cluster 2024-04-26T16:26:58.125477+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: cluster 2024-04-26T16:26:58.125477+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: audit 2024-04-26T16:26:58.132419+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: audit 2024-04-26T16:26:58.132419+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: audit 2024-04-26T16:26:58.134642+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:58 smithi050 bash[21463]: audit 2024-04-26T16:26:58.134642+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.414 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:58.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:58.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:58.543 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:58.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: audit 2024-04-26T16:26:56.790182+0000 mgr.y (mgr.24401) 378 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: audit 2024-04-26T16:26:56.790182+0000 mgr.y (mgr.24401) 378 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: cluster 2024-04-26T16:26:57.595690+0000 mgr.y (mgr.24401) 379 : cluster [DBG] pgmap v508: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: cluster 2024-04-26T16:26:57.595690+0000 mgr.y (mgr.24401) 379 : cluster [DBG] pgmap v508: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: cluster 2024-04-26T16:26:58.125477+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: cluster 2024-04-26T16:26:58.125477+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e350: 8 total, 8 up, 7 in 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: audit 2024-04-26T16:26:58.132419+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: audit 2024-04-26T16:26:58.132419+0000 mon.a (mon.0) 2328 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: audit 2024-04-26T16:26:58.134642+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:58 smithi190 bash[23868]: audit 2024-04-26T16:26:58.134642+0000 mon.b (mon.1) 234 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:26:58.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:58.674 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:26:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:26:58.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:58.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:26:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:26:58.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:58.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:58.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:59.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:59.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:59.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:59.192 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:26:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:26:59.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.327 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:26:59.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:26:59.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:59.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:26:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: cluster 2024-04-26T16:26:58.193412+0000 mon.a (mon.0) 2329 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: cluster 2024-04-26T16:26:58.193412+0000 mon.a (mon.0) 2329 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.117929+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.117929+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: cluster 2024-04-26T16:26:59.123069+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: cluster 2024-04-26T16:26:59.123069+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.124967+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.124967+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.131237+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.131237+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.460 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.132742+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[29234]: audit 2024-04-26T16:26:59.132742+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: cluster 2024-04-26T16:26:58.193412+0000 mon.a (mon.0) 2329 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: cluster 2024-04-26T16:26:58.193412+0000 mon.a (mon.0) 2329 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.117929+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.117929+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: cluster 2024-04-26T16:26:59.123069+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: cluster 2024-04-26T16:26:59.123069+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.124967+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.124967+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.131237+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.131237+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.132742+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:26:59 smithi050 bash[21463]: audit 2024-04-26T16:26:59.132742+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: cluster 2024-04-26T16:26:58.193412+0000 mon.a (mon.0) 2329 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: cluster 2024-04-26T16:26:58.193412+0000 mon.a (mon.0) 2329 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.117929+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.117929+0000 mon.a (mon.0) 2330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: cluster 2024-04-26T16:26:59.123069+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: cluster 2024-04-26T16:26:59.123069+0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e351: 8 total, 8 up, 7 in 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.124967+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.124967+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:26:59.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.131237+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.131237+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.132742+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:26:59 smithi190 bash[23868]: audit 2024-04-26T16:26:59.132742+0000 mon.b (mon.1) 235 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:26:59.585 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:59.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:59.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:59.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:26:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:26:59.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:59.845 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:26:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:26:59.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:59.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:26:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:26:59.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:00.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:00.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:00.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:00.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:00.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: cluster 2024-04-26T16:26:59.596512+0000 mgr.y (mgr.24401) 380 : cluster [DBG] pgmap v511: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: cluster 2024-04-26T16:26:59.596512+0000 mgr.y (mgr.24401) 380 : cluster [DBG] pgmap v511: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.120675+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.120675+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.120757+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.120757+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: cluster 2024-04-26T16:27:00.125250+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: cluster 2024-04-26T16:27:00.125250+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.126376+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:27:00.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.126376+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.129513+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.129513+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.132021+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[29234]: audit 2024-04-26T16:27:00.132021+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: cluster 2024-04-26T16:26:59.596512+0000 mgr.y (mgr.24401) 380 : cluster [DBG] pgmap v511: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: cluster 2024-04-26T16:26:59.596512+0000 mgr.y (mgr.24401) 380 : cluster [DBG] pgmap v511: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.120675+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.120675+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.120757+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:00.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.120757+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: cluster 2024-04-26T16:27:00.125250+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: cluster 2024-04-26T16:27:00.125250+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.126376+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.126376+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.129513+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.129513+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.132021+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.350 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:00 smithi050 bash[21463]: audit 2024-04-26T16:27:00.132021+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:00.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:00.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: cluster 2024-04-26T16:26:59.596512+0000 mgr.y (mgr.24401) 380 : cluster [DBG] pgmap v511: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:27:00.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: cluster 2024-04-26T16:26:59.596512+0000 mgr.y (mgr.24401) 380 : cluster [DBG] pgmap v511: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:27:00.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.120675+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.120675+0000 mon.a (mon.0) 2334 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.120757+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.120757+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: cluster 2024-04-26T16:27:00.125250+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: cluster 2024-04-26T16:27:00.125250+0000 mon.a (mon.0) 2336 : cluster [DBG] osdmap e352: 8 total, 8 up, 7 in 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.126376+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.126376+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]: dispatch 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.129513+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.129513+0000 mon.a (mon.0) 2338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.132021+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:00 smithi190 bash[23868]: audit 2024-04-26T16:27:00.132021+0000 mon.b (mon.1) 236 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:00.492 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:00.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:00.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:00.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:00.753 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 -- 172.21.15.50:0/1351023526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c106000 msgr2=0x7f534c112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 --2- 172.21.15.50:0/1351023526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c106000 0x7f534c112510 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f5344009f90 tx=0x7f534400b330 comp rx=0 tx=0).stop 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 -- 172.21.15.50:0/1351023526 shutdown_connections 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 --2- 172.21.15.50:0/1351023526 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c106000 0x7f534c112510 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 --2- 172.21.15.50:0/1351023526 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c10b640 0x7f534c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 --2- 172.21.15.50:0/1351023526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f534c105640 0x7f534c105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.772+0000 7f5353f02700 1 -- 172.21.15.50:0/1351023526 >> 172.21.15.50:0/1351023526 conn(0x7f534c101130 msgr2=0x7f534c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- 172.21.15.50:0/1351023526 shutdown_connections 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- 172.21.15.50:0/1351023526 wait complete. 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 librados: starting msgr at 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 librados: starting objecter 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 Processor -- start 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- start start 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 librados: setting wanted keys 2024-04-26T16:27:00.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 librados: calling monclient init 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c105640 0x7f534c19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f534c106000 0x7f534c19b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c10b640 0x7f534c19fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f534c114d40 con 0x7f534c105640 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f534c114bc0 con 0x7f534c106000 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f534c114ec0 con 0x7f534c10b640 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5351c9e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c105640 0x7f534c19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c10b640 0x7f534c19fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5351c9e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c105640 0x7f534c19b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53028/0 (socket says 172.21.15.50:53028) 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c10b640 0x7f534c19fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43362/0 (socket says 172.21.15.50:43362) 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5351c9e700 1 -- 172.21.15.50:0/2541165908 learned_addr learned my addr 172.21.15.50:0/2541165908 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f534c106000 0x7f534c19b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f534c106000 msgr2=0x7f534c19b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f534c106000 0x7f534c19b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c105640 msgr2=0x7f534c19b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c105640 0x7f534c19b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c1a02c0 con 0x7f534c10b640 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f534c106000 0x7f534c19b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f534c105640 0x7f534c19b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c10b640 0x7f534c19fb80 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f534400b6c0 tx=0x7f5344004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5342ffd700 10 client.?.objecter ms_handle_connect 0x7f534c10b640 2024-04-26T16:27:00.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5342ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:00.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344021070 con 0x7f534c10b640 2024-04-26T16:27:00.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5344007910 con 0x7f534c10b640 2024-04-26T16:27:00.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c1a0550 con 0x7f534c10b640 2024-04-26T16:27:00.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5344005040 con 0x7f534c10b640 2024-04-26T16:27:00.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 10 client.41063.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c1a7e30 con 0x7f534c10b640 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 librados: init done 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f534c06a110 con 0x7f534c10b640 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.776+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5344024070 con 0x7f534c10b640 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5338071f80 0x7f5338074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(352..352 src has 87..352) ==== 9505+0+0 (secure 0 0 0) 0x7f534400f020 con 0x7f534c10b640 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 osd_map(352..352 src has 87..352) 2024-04-26T16:27:00.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 3 client.41063.objecter handle_osd_map got epochs [352,352] > 0 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 3 client.41063.objecter handle_osd_map decoding full epoch 352 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5338071f80 0x7f5338074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 20 client.41063.objecter dump_active .. 0 homeless 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v352) ==== 54+0+16 (secure 0 0 0) 0x7f534402b020 con 0x7f534c10b640 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 10 client.41063.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 1 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f534c064280 0x7f534c10c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 1 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 0 1 2024-04-26T16:27:00.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 4 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 10 client.41063.objecter _send_command 1 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f534c10db80 con 0x7f534c064280 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5338071f80 0x7f5338074440 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f5348009750 tx=0x7f5348009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f5338071f80 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f534c064280 0x7f534c10c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f534c064280 0x7f534c10c4c0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c064280 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c10db80 con 0x7f534c064280 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 1: 0 ) 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 10 client.41063.objecter _finish_command 1 = system:0 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 4 2024-04-26T16:27:00.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.780+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 0 1 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 10 client.41063.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 2 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 2 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c19c610 osd=0 3 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 0 2 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 4 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 10 client.41063.objecter _send_command 2 2024-04-26T16:27:00.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c10ca00 con 0x7f534c064280 2024-04-26T16:27:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c10ca00 con 0x7f534c064280 2024-04-26T16:27:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 10 client.41063.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 4 2024-04-26T16:27:00.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 0 2 2024-04-26T16:27:00.799 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 10 client.41063.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f534c063e70 0x7f534c1ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 1 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 4 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 10 client.41063.objecter _send_command 3 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f534c10d8f0 con 0x7f534c063e70 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f534c063e70 0x7f534c1ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f534c063e70 0x7f534c1ac020 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c063e70 2024-04-26T16:27:00.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c10d8f0 con 0x7f534c063e70 2024-04-26T16:27:00.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 3: 0 ) 2024-04-26T16:27:00.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 10 client.41063.objecter _finish_command 3 = system:0 2024-04-26T16:27:00.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 4 2024-04-26T16:27:00.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.800+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 1 3 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 10 client.41063.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 4 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 4 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c063d00 osd=1 3 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 1 4 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 4 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 10 client.41063.objecter _send_command 4 2024-04-26T16:27:00.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c10ce60 con 0x7f534c063e70 2024-04-26T16:27:00.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c10ce60 con 0x7f534c063e70 2024-04-26T16:27:00.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5342ffd700 10 client.41063.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 4 2024-04-26T16:27:00.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 1 4 2024-04-26T16:27:00.820 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 10 client.41063.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 5 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f534c02dff0 0x7f534c1af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 5 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 2 5 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 4 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 10 client.41063.objecter _send_command 5 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f534c1b2550 con 0x7f534c02dff0 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f534c02dff0 0x7f534c1af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c02dff0 2024-04-26T16:27:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.820+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f534c02dff0 0x7f534c1af6f0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.824+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c1b2550 con 0x7f534c02dff0 2024-04-26T16:27:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.824+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 5: 0 ) 2024-04-26T16:27:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.824+0000 7f5342ffd700 10 client.41063.objecter _finish_command 5 = system:0 2024-04-26T16:27:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.824+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 4 2024-04-26T16:27:00.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.824+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 2 5 2024-04-26T16:27:00.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 10 client.41063.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 6 2024-04-26T16:27:00.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 6 2024-04-26T16:27:00.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c10d260 osd=2 3 2024-04-26T16:27:00.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 2 6 2024-04-26T16:27:00.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 4 2024-04-26T16:27:00.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 10 client.41063.objecter _send_command 6 2024-04-26T16:27:00.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.840+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c02d560 con 0x7f534c02dff0 2024-04-26T16:27:00.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c02d560 con 0x7f534c02dff0 2024-04-26T16:27:00.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 10 client.41063.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 4 2024-04-26T16:27:00.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 2 6 2024-04-26T16:27:00.841 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:27:00.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 10 client.41063.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 7 2024-04-26T16:27:00.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f534c1ad240 0x7f534c1b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 7 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 3 7 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 4 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 10 client.41063.objecter _send_command 7 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f534c1b5c20 con 0x7f534c1ad240 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f534c1ad240 0x7f534c1b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c1ad240 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f534c1ad240 0x7f534c1b2dc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c1b5c20 con 0x7f534c1ad240 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 7: 0 ) 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 10 client.41063.objecter _finish_command 7 = system:0 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 4 2024-04-26T16:27:00.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.844+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 3 7 2024-04-26T16:27:00.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:00.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 10 client.41063.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 8 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 8 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c02d7b0 osd=3 3 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 3 8 2024-04-26T16:27:00.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 4 2024-04-26T16:27:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 10 client.41063.objecter _send_command 8 2024-04-26T16:27:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.860+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c1ac810 con 0x7f534c1ad240 2024-04-26T16:27:00.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c1ac810 con 0x7f534c1ad240 2024-04-26T16:27:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5342ffd700 10 client.41063.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 4 2024-04-26T16:27:00.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 3 8 2024-04-26T16:27:00.863 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 10 client.41063.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 9 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f534c1b0910 0x7f534c1b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 9 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 4 9 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 4 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 10 client.41063.objecter _send_command 9 2024-04-26T16:27:00.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f534c1b92f0 con 0x7f534c1b0910 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f534c1b0910 0x7f534c1b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c1b0910 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.864+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f534c1b0910 0x7f534c1b6490 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.868+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c1b92f0 con 0x7f534c1b0910 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.868+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 9: 0 ) 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.868+0000 7f5342ffd700 10 client.41063.objecter _finish_command 9 = system:0 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.868+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 4 2024-04-26T16:27:00.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.868+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 4 9 2024-04-26T16:27:00.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:00.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 10 client.41063.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 10 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 10 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1ac560 osd=4 3 2024-04-26T16:27:00.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 4 10 2024-04-26T16:27:00.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 4 2024-04-26T16:27:00.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 10 client.41063.objecter _send_command 10 2024-04-26T16:27:00.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.888+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c1b69d0 con 0x7f534c1b0910 2024-04-26T16:27:00.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c1b69d0 con 0x7f534c1b0910 2024-04-26T16:27:00.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5342ffd700 10 client.41063.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 4 2024-04-26T16:27:00.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 4 10 2024-04-26T16:27:00.890 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 10 client.41063.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 11 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f534c1b7050 0x7f534c1b9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 11 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 5 11 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 4 2024-04-26T16:27:00.891 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 10 client.41063.objecter _send_command 11 2024-04-26T16:27:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f534c1bc4b0 con 0x7f534c1b7050 2024-04-26T16:27:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f534c1b7050 0x7f534c1b9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c1b7050 2024-04-26T16:27:00.892 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.892+0000 7f535149d700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f534c1b7050 0x7f534c1b9480 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.896+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c1bc4b0 con 0x7f534c1b7050 2024-04-26T16:27:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.896+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 11: 0 ) 2024-04-26T16:27:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.896+0000 7f5342ffd700 10 client.41063.objecter _finish_command 11 = system:0 2024-04-26T16:27:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.896+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 4 2024-04-26T16:27:00.893 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.896+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 5 11 2024-04-26T16:27:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 10 client.41063.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 12 2024-04-26T16:27:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 12 2024-04-26T16:27:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1b69d0 osd=5 3 2024-04-26T16:27:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 5 12 2024-04-26T16:27:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 4 2024-04-26T16:27:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 10 client.41063.objecter _send_command 12 2024-04-26T16:27:00.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.912+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c1bc510 con 0x7f534c1b7050 2024-04-26T16:27:00.915 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:27:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c1bc510 con 0x7f534c1b7050 2024-04-26T16:27:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5342ffd700 10 client.41063.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 4 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 5 12 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 10 client.41063.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 13 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f534c1bb840 0x7f534c1bbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 13 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 6 13 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 4 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 10 client.41063.objecter _send_command 13 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f534c1bf880 con 0x7f534c1bb840 2024-04-26T16:27:00.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f534c1bb840 0x7f534c1bbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f535249f700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f534c1bb840 0x7f534c1bbca0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.916+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c1bb840 2024-04-26T16:27:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.920+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c1bf880 con 0x7f534c1bb840 2024-04-26T16:27:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.920+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 13: 0 ) 2024-04-26T16:27:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.920+0000 7f5342ffd700 10 client.41063.objecter _finish_command 13 = system:0 2024-04-26T16:27:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.920+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 4 2024-04-26T16:27:00.918 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.920+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 6 13 2024-04-26T16:27:00.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 10 client.41063.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 14 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 14 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1bc510 osd=6 3 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 6 14 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 4 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 10 client.41063.objecter _send_command 14 2024-04-26T16:27:00.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.936+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c1be700 con 0x7f534c1bb840 2024-04-26T16:27:00.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c1be700 con 0x7f534c1bb840 2024-04-26T16:27:00.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5342ffd700 10 client.41063.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 4 2024-04-26T16:27:00.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 6 14 2024-04-26T16:27:00.940 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:27:00.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 10 client.41063.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 15 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f534c1bee70 0x7f534c1bf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 15 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 7 15 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 4 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 10 client.41063.objecter _send_command 15 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.940+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f534c1c2bf0 con 0x7f534c1bee70 2024-04-26T16:27:00.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f534c1bee70 0x7f534c1bf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:00.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5342ffd700 10 client.41063.objecter ms_handle_connect 0x7f534c1bee70 2024-04-26T16:27:00.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5351c9e700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f534c1bee70 0x7f534c1bf290 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:00.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f534c1c2bf0 con 0x7f534c1bee70 2024-04-26T16:27:00.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 15: 0 ) 2024-04-26T16:27:00.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5342ffd700 10 client.41063.objecter _finish_command 15 = system:0 2024-04-26T16:27:00.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 4 2024-04-26T16:27:00.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.944+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 7 15 2024-04-26T16:27:00.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 10 client.41063.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:00.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign -1 16 2024-04-26T16:27:00.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 20 client.41063.objecter _get_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 15 client.41063.objecter _session_command_op_remove -1 16 2024-04-26T16:27:00.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 20 client.41063.objecter get_session s=0x7f534c1be700 osd=7 3 2024-04-26T16:27:00.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 15 client.41063.objecter _session_command_op_assign 7 16 2024-04-26T16:27:00.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 4 2024-04-26T16:27:00.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 10 client.41063.objecter _send_command 16 2024-04-26T16:27:00.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.964+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f534c1b9ab0 con 0x7f534c1bee70 2024-04-26T16:27:00.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5342ffd700 1 -- 172.21.15.50:0/2541165908 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f534c1b9ab0 con 0x7f534c1bee70 2024-04-26T16:27:00.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5342ffd700 10 client.41063.objecter ms_dispatch 0x7f534c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:00.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5342ffd700 10 client.41063.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:00.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5342ffd700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 4 2024-04-26T16:27:00.965 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5342ffd700 15 client.41063.objecter _session_command_op_remove 7 16 2024-04-26T16:27:00.966 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 librados: watch_flush enter 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 librados: watch_flush exit 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.0 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f534c064280 msgr2=0x7f534c10c4c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f534c064280 0x7f534c10c4c0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c19c610 osd=0 1 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.1 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f534c063e70 msgr2=0x7f534c1ac020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f534c063e70 0x7f534c1ac020 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c063d00 osd=1 1 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.2 2024-04-26T16:27:00.967 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f534c02dff0 msgr2=0x7f534c1af6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f534c02dff0 0x7f534c1af6f0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c10d260 osd=2 1 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.3 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f534c1ad240 msgr2=0x7f534c1b2dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f534c1ad240 0x7f534c1b2dc0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f535249f700 1 -- 172.21.15.50:0/2541165908 reap_dead start 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c02d7b0 osd=3 1 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.4 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f534c1b0910 msgr2=0x7f534c1b6490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f534c1b0910 0x7f534c1b6490 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1ac560 osd=4 1 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.5 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f535249f700 1 -- 172.21.15.50:0/2541165908 reap_dead start 2024-04-26T16:27:00.968 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f534c1b7050 msgr2=0x7f534c1b9480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f534c1b7050 0x7f534c1b9480 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1b69d0 osd=5 1 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.6 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f534c1bb840 msgr2=0x7f534c1bbca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7f534c1bb840 0x7f534c1bbca0 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1bc510 osd=6 1 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter close_session for osd.7 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f534c1bee70 msgr2=0x7f534c1bf290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f534c1bee70 0x7f534c1bf290 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter put_session s=0x7f534c1be700 osd=7 1 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 20 client.41063.objecter shutdown clearing up homeless session... 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 10 client.41063.objecter successfully canceled tick 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5338071f80 msgr2=0x7f5338074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5338071f80 0x7f5338074440 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f5348009750 tx=0x7f5348009310 comp rx=0 tx=0).stop 2024-04-26T16:27:00.969 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c10b640 msgr2=0x7f534c19fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 --2- 172.21.15.50:0/2541165908 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f534c10b640 0x7f534c19fb80 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f534400b6c0 tx=0x7f5344004c80 comp rx=0 tx=0).stop 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f535249f700 1 -- 172.21.15.50:0/2541165908 reap_dead start 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 shutdown_connections 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 >> 172.21.15.50:0/2541165908 conn(0x7f534c101130 msgr2=0x7f534c1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 shutdown_connections 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 -- 172.21.15.50:0/2541165908 wait complete. 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:00.968+0000 7f5353f02700 1 librados: shutdown 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:27:00.970 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:27:00.971 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:27:00.971 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:27:00.971 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:27:00.971 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:27:00.971 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:27:00.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:00.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:01.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:01.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:01.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:01.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:01.257 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:01.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:00.782455+0000 mon.c (mon.2) 552 : audit [DBG] from='client.? 172.21.15.50:0/2541165908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:00.782455+0000 mon.c (mon.2) 552 : audit [DBG] from='client.? 172.21.15.50:0/2541165908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.123171+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.123171+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.123279+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.123279+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: cluster 2024-04-26T16:27:01.131229+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-04-26T16:27:01.372 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: cluster 2024-04-26T16:27:01.131229+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.132200+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.132200+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.133017+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.133017+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.161622+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.161622+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.162750+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.162750+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.163177+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:01.373 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:01 smithi190 bash[23868]: audit 2024-04-26T16:27:01.163177+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:00.782455+0000 mon.c (mon.2) 552 : audit [DBG] from='client.? 172.21.15.50:0/2541165908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:00.782455+0000 mon.c (mon.2) 552 : audit [DBG] from='client.? 172.21.15.50:0/2541165908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.123171+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.123171+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.123279+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.123279+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: cluster 2024-04-26T16:27:01.131229+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: cluster 2024-04-26T16:27:01.131229+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.132200+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.132200+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.133017+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.133017+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.161622+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.161622+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.162750+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.162750+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:00.782455+0000 mon.c (mon.2) 552 : audit [DBG] from='client.? 172.21.15.50:0/2541165908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:00.782455+0000 mon.c (mon.2) 552 : audit [DBG] from='client.? 172.21.15.50:0/2541165908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.123171+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.123171+0000 mon.a (mon.0) 2339 : audit [INF] from='client.? 172.21.15.50:0/4230518665' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi050-73201-55"}]': finished 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.123279+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.123279+0000 mon.a (mon.0) 2340 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:01.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: cluster 2024-04-26T16:27:01.131229+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: cluster 2024-04-26T16:27:01.131229+0000 mon.a (mon.0) 2341 : cluster [DBG] osdmap e353: 8 total, 8 up, 7 in 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.132200+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.132200+0000 mon.a (mon.0) 2342 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.133017+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.133017+0000 mon.b (mon.1) 237 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.161622+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.161622+0000 mon.a (mon.0) 2343 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.162750+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.162750+0000 mon.a (mon.0) 2344 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.163177+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:01.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[21463]: audit 2024-04-26T16:27:01.163177+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.163177+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:01 smithi050 bash[29234]: audit 2024-04-26T16:27:01.163177+0000 mon.a (mon.0) 2345 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:01.387 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:01.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:01.518 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:01.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:01.649 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:01.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:01.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:01.781 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:01.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:01.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:01.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:01.941 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:27:02.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:02.040 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:02.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:02.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:02.165 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:02.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:02.289 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:02.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: cluster 2024-04-26T16:27:01.598759+0000 mgr.y (mgr.24401) 381 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: cluster 2024-04-26T16:27:01.598759+0000 mgr.y (mgr.24401) 381 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.125808+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.125808+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.125971+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.125971+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: cluster 2024-04-26T16:27:02.130371+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: cluster 2024-04-26T16:27:02.130371+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.133425+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.133425+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.133998+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.133998+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.136251+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:02 smithi190 bash[23868]: audit 2024-04-26T16:27:02.136251+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 -- 172.21.15.50:0/3509423784 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c81094c0 msgr2=0x7f34c81098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/3509423784 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c81094c0 0x7f34c81098a0 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009990 tx=0x7f34b8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:27:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 -- 172.21.15.50:0/3509423784 shutdown_connections 2024-04-26T16:27:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/3509423784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c8103e30 0x7f34c8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/3509423784 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c81094c0 0x7f34c81098a0 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/3509423784 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 0x7f34c81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 -- 172.21.15.50:0/3509423784 >> 172.21.15.50:0/3509423784 conn(0x7f34c807a580 msgr2=0x7f34c807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 -- 172.21.15.50:0/3509423784 shutdown_connections 2024-04-26T16:27:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.364+0000 7f34cd3b5700 1 -- 172.21.15.50:0/3509423784 wait complete. 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 librados: starting msgr at 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 librados: starting objecter 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 Processor -- start 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- start start 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 librados: setting wanted keys 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 librados: calling monclient init 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 0x7f34c81991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 0x7f34c8199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c81094c0 0x7f34c819daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f34c8112b10 con 0x7f34c8103470 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f34c8112990 con 0x7f34c81094c0 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f34c8112c90 con 0x7f34c8103e30 2024-04-26T16:27:02.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 0x7f34c81991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 0x7f34c8199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 0x7f34c8199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43406/0 (socket says 172.21.15.50:43406) 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c6ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 0x7f34c81991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53048/0 (socket says 172.21.15.50:53048) 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 -- 172.21.15.50:0/2250784860 learned_addr learned my addr 172.21.15.50:0/2250784860 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c77fe700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c81094c0 0x7f34c819daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 -- 172.21.15.50:0/2250784860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c81094c0 msgr2=0x7f34c819daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c81094c0 0x7f34c819daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 -- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 msgr2=0x7f34c81991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 0x7f34c81991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 -- 172.21.15.50:0/2250784860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c819e1e0 con 0x7f34c8103e30 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c77fe700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c81094c0 0x7f34c819daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34c67fc700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 0x7f34c8199710 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f34b000ef10 tx=0x7f34b000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 10 client.?.objecter ms_handle_connect 0x7f34c8103e30 2024-04-26T16:27:02.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b000ca70 con 0x7f34c8103e30 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c819e470 con 0x7f34c8103e30 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f34b0019070 con 0x7f34c8103e30 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0014480 con 0x7f34c8103e30 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 10 client.41075.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c81a5d50 con 0x7f34c8103e30 2024-04-26T16:27:02.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 librados: init done 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c806a110 con 0x7f34c8103e30 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f34b0012050 con 0x7f34c8103e30 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.368+0000 7f34bffff700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f34b4071f80 0x7f34b4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(354..354 src has 87..354) ==== 9730+0+0 (secure 0 0 0) 0x7f34b0091bd0 con 0x7f34c8103e30 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34bffff700 10 client.41075.objecter ms_dispatch 0x7f34c8001550 osd_map(354..354 src has 87..354) 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34bffff700 3 client.41075.objecter handle_osd_map got epochs [354,354] > 0 2024-04-26T16:27:02.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34bffff700 3 client.41075.objecter handle_osd_map decoding full epoch 354 2024-04-26T16:27:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34c6ffd700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f34b4071f80 0x7f34b4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34bffff700 20 client.41075.objecter dump_active .. 0 homeless 2024-04-26T16:27:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34bffff700 10 client.41075.objecter ms_handle_connect 0x7f34b4071f80 2024-04-26T16:27:02.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.372+0000 7f34c6ffd700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f34b4071f80 0x7f34b4074440 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7f34b80098e0 tx=0x7f34b8005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:02.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.376+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f34b0010040 con 0x7f34c8103e30 2024-04-26T16:27:02.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: cluster 2024-04-26T16:27:01.598759+0000 mgr.y (mgr.24401) 381 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: cluster 2024-04-26T16:27:01.598759+0000 mgr.y (mgr.24401) 381 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.125808+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.125808+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.125971+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.125971+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: cluster 2024-04-26T16:27:02.130371+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: cluster 2024-04-26T16:27:02.130371+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.133425+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.133425+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.133998+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.133998+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.136251+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[29234]: audit 2024-04-26T16:27:02.136251+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: cluster 2024-04-26T16:27:01.598759+0000 mgr.y (mgr.24401) 381 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: cluster 2024-04-26T16:27:01.598759+0000 mgr.y (mgr.24401) 381 : cluster [DBG] pgmap v514: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.125808+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.125808+0000 mon.a (mon.0) 2346 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.125971+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.125971+0000 mon.a (mon.0) 2347 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi050-73201-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: cluster 2024-04-26T16:27:02.130371+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: cluster 2024-04-26T16:27:02.130371+0000 mon.a (mon.0) 2348 : cluster [DBG] osdmap e354: 8 total, 8 up, 7 in 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.133425+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.133425+0000 mon.a (mon.0) 2349 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.133998+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.133998+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.136251+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:02 smithi050 bash[21463]: audit 2024-04-26T16:27:02.136251+0000 mon.b (mon.1) 238 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:27:02.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:02.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:02.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:02.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:02.612+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f34c81a5f20 con 0x7f34c8103e30 2024-04-26T16:27:02.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:02.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:02.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:02.806 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:02.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:02.937 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:02.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:03.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:03.068 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:03.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.140+0000 7f34bffff700 1 -- 172.21.15.50:0/2250784860 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v355) ==== 97+0+0 (secure 0 0 0) 0x7f34b005d420 con 0x7f34c8103e30 2024-04-26T16:27:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 10 librados: watch_flush enter 2024-04-26T16:27:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 10 librados: watch_flush exit 2024-04-26T16:27:03.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 20 client.41075.objecter shutdown clearing up homeless session... 2024-04-26T16:27:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 10 client.41075.objecter successfully canceled tick 2024-04-26T16:27:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f34b4071f80 msgr2=0x7f34b4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f34b4071f80 0x7f34b4074440 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7f34b80098e0 tx=0x7f34b8005960 comp rx=0 tx=0).stop 2024-04-26T16:27:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 msgr2=0x7f34c8199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:03.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.144+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 0x7f34c8199710 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f34b000ef10 tx=0x7f34b000c3a0 comp rx=0 tx=0).stop 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 shutdown_connections 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f34b4071f80 0x7f34b4074440 unknown :-1 s=CLOSED pgs=3723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34c81094c0 0x7f34c819daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f34c8103470 0x7f34c81991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 --2- 172.21.15.50:0/2250784860 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f34c8103e30 0x7f34c8199710 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 >> 172.21.15.50:0/2250784860 conn(0x7f34c807a580 msgr2=0x7f34c8101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 shutdown_connections 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 -- 172.21.15.50:0/2250784860 wait complete. 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:03.148+0000 7f34cd3b5700 1 librados: shutdown 2024-04-26T16:27:03.146 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:27:03.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:03.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:03.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:02.617405+0000 mon.c (mon.2) 553 : audit [INF] from='client.? 172.21.15.50:0/2250784860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:02.617405+0000 mon.c (mon.2) 553 : audit [INF] from='client.? 172.21.15.50:0/2250784860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:02.617980+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:02.617980+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: cluster 2024-04-26T16:27:03.127082+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: cluster 2024-04-26T16:27:03.127082+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:03.129406+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:03.129406+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:27:03.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:03.129648+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:03.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: audit 2024-04-26T16:27:03.129648+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:03.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: cluster 2024-04-26T16:27:03.158198+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-04-26T16:27:03.305 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:03 smithi190 bash[23868]: cluster 2024-04-26T16:27:03.158198+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-04-26T16:27:03.318 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:03.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:03.448 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:03.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:03.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:03.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:02.617405+0000 mon.c (mon.2) 553 : audit [INF] from='client.? 172.21.15.50:0/2250784860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:02.617405+0000 mon.c (mon.2) 553 : audit [INF] from='client.? 172.21.15.50:0/2250784860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:02.617980+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:02.617980+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: cluster 2024-04-26T16:27:03.127082+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: cluster 2024-04-26T16:27:03.127082+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:03.129406+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:03.129406+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:03.129648+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: audit 2024-04-26T16:27:03.129648+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: cluster 2024-04-26T16:27:03.158198+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-04-26T16:27:03.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[29234]: cluster 2024-04-26T16:27:03.158198+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:02.617405+0000 mon.c (mon.2) 553 : audit [INF] from='client.? 172.21.15.50:0/2250784860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:02.617405+0000 mon.c (mon.2) 553 : audit [INF] from='client.? 172.21.15.50:0/2250784860' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:02.617980+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:02.617980+0000 mon.a (mon.0) 2351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: cluster 2024-04-26T16:27:03.127082+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: cluster 2024-04-26T16:27:03.127082+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:03.129406+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:03.129406+0000 mon.a (mon.0) 2353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:03.129648+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: audit 2024-04-26T16:27:03.129648+0000 mon.a (mon.0) 2354 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: cluster 2024-04-26T16:27:03.158198+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-04-26T16:27:03.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:03 smithi050 bash[21463]: cluster 2024-04-26T16:27:03.158198+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2024-04-26T16:27:03.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:03.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:03.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:03.842 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:03.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:03.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:03.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:04.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:04.101 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/1132784349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 msgr2=0x7f8e8c102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- 172.21.15.50:0/1132784349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c102d40 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009920 tx=0x7f8e7c009c30 comp rx=0 tx=0).stop 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/1132784349 shutdown_connections 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- 172.21.15.50:0/1132784349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e8c1076b0 0x7f8e8c10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- 172.21.15.50:0/1132784349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c102d40 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- 172.21.15.50:0/1132784349 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/1132784349 >> 172.21.15.50:0/1132784349 conn(0x7f8e8c0792d0 msgr2=0x7f8e8c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/1132784349 shutdown_connections 2024-04-26T16:27:04.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/1132784349 wait complete. 2024-04-26T16:27:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 librados: starting msgr at 2024-04-26T16:27:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 librados: starting objecter 2024-04-26T16:27:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 Processor -- start 2024-04-26T16:27:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- start start 2024-04-26T16:27:04.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 librados: setting wanted keys 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 librados: calling monclient init 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c1102e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c110820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e8c1076b0 0x7f8e8c110d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c112bc0 con 0x7f8e8c103310 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8e8c112a40 con 0x7f8e8c1076b0 2024-04-26T16:27:04.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e913ac700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8c112d40 con 0x7f8e8c102960 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e8a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c110820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e8a7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c110820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53066/0 (socket says 172.21.15.50:53066) 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e8a7fc700 1 -- 172.21.15.50:0/436343053 learned_addr learned my addr 172.21.15.50:0/436343053 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e8affd700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c1102e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.136+0000 7f8e8b7fe700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e8c1076b0 0x7f8e8c110d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8a7fc700 1 -- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 msgr2=0x7f8e8c1102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8a7fc700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c1102e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8a7fc700 1 -- 172.21.15.50:0/436343053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e8c1076b0 msgr2=0x7f8e8c110d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8a7fc700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e8c1076b0 0x7f8e8c110d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8a7fc700 1 -- 172.21.15.50:0/436343053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e8c1a77f0 con 0x7f8e8c103310 2024-04-26T16:27:04.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8affd700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e8a7fc700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c110820 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000ebb0 tx=0x7f8e8000eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e73fff700 10 client.?.objecter ms_handle_connect 0x7f8e8c103310 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e73fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e8000ccf0 con 0x7f8e8c103310 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e8c06cea0 con 0x7f8e8c103310 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e913ac700 10 client.30462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e8c06d430 con 0x7f8e8c103310 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e913ac700 1 librados: init done 2024-04-26T16:27:04.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8e80004480 con 0x7f8e8c103310 2024-04-26T16:27:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e80010430 con 0x7f8e8c103310 2024-04-26T16:27:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e78005150 con 0x7f8e8c103310 2024-04-26T16:27:04.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.140+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8e80003680 con 0x7f8e8c103310 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.144+0000 7f8e73fff700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8e74071fc0 0x7f8e74074480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.144+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(356..356 src has 87..356) ==== 10253+0+0 (secure 0 0 0) 0x7f8e80091f70 con 0x7f8e8c103310 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.144+0000 7f8e73fff700 10 client.30462.objecter ms_dispatch 0x7f8e8c001550 osd_map(356..356 src has 87..356) 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.144+0000 7f8e73fff700 3 client.30462.objecter handle_osd_map got epochs [356,356] > 0 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.144+0000 7f8e73fff700 3 client.30462.objecter handle_osd_map decoding full epoch 356 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.144+0000 7f8e73fff700 20 client.30462.objecter dump_active .. 0 homeless 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.148+0000 7f8e8affd700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8e74071fc0 0x7f8e74074480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.148+0000 7f8e8affd700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8e74071fc0 0x7f8e74074480 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0098f0 tx=0x7f8e7c009440 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.148+0000 7f8e73fff700 10 client.30462.objecter ms_handle_connect 0x7f8e74071fc0 2024-04-26T16:27:04.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.152+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f8e8005d640 con 0x7f8e8c103310 2024-04-26T16:27:04.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:04.229 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:04.332 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:04.355 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:04.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:04.400+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd rm-pg-upmap", "pgid": "2.15"} v 0) -- 0x7f8e78005440 con 0x7f8e8c103310 2024-04-26T16:27:04.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:04.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:04.484 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:27:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: cluster 2024-04-26T16:27:03.599371+0000 mgr.y (mgr.24401) 382 : cluster [DBG] pgmap v517: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: cluster 2024-04-26T16:27:03.599371+0000 mgr.y (mgr.24401) 382 : cluster [DBG] pgmap v517: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.132013+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.132013+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: cluster 2024-04-26T16:27:04.136661+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: cluster 2024-04-26T16:27:04.136661+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.166133+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.166133+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.166972+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.166972+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.168872+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.168872+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.169897+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[29234]: audit 2024-04-26T16:27:04.169897+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: cluster 2024-04-26T16:27:03.599371+0000 mgr.y (mgr.24401) 382 : cluster [DBG] pgmap v517: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: cluster 2024-04-26T16:27:03.599371+0000 mgr.y (mgr.24401) 382 : cluster [DBG] pgmap v517: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.132013+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.132013+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: cluster 2024-04-26T16:27:04.136661+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: cluster 2024-04-26T16:27:04.136661+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-04-26T16:27:04.487 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.166133+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.166133+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.166972+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.166972+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.168872+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.168872+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.169897+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:04 smithi050 bash[21463]: audit 2024-04-26T16:27:04.169897+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: cluster 2024-04-26T16:27:03.599371+0000 mgr.y (mgr.24401) 382 : cluster [DBG] pgmap v517: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: cluster 2024-04-26T16:27:03.599371+0000 mgr.y (mgr.24401) 382 : cluster [DBG] pgmap v517: 228 pgs: 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.132013+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.132013+0000 mon.a (mon.0) 2356 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi050-73201-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: cluster 2024-04-26T16:27:04.136661+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: cluster 2024-04-26T16:27:04.136661+0000 mon.a (mon.0) 2357 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.166133+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.166133+0000 mon.a (mon.0) 2358 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.166972+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.166972+0000 mon.a (mon.0) 2359 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.168872+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.168872+0000 mon.b (mon.1) 239 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.169897+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:04 smithi190 bash[23868]: audit 2024-04-26T16:27:04.169897+0000 mon.b (mon.1) 240 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]: dispatch 2024-04-26T16:27:04.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:04.613 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:04.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:04.744 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:04.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:04.874 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:04.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:04.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:05.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:05.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:05.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:05.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:05.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.128+0000 7f8e73fff700 1 -- 172.21.15.50:0/436343053 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]=0 clear 2.15 pg_upmap mapping v357) ==== 106+0+0 (secure 0 0 0) 0x7f8e800612f0 con 0x7f8e8c103310 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 10 librados: watch_flush enter 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 10 librados: watch_flush exit 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 20 client.30462.objecter shutdown clearing up homeless session... 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 10 client.30462.objecter successfully canceled tick 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8e74071fc0 msgr2=0x7f8e74074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8e74071fc0 0x7f8e74074480 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c0098f0 tx=0x7f8e7c009440 comp rx=0 tx=0).stop 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 msgr2=0x7f8e8c110820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c110820 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7f8e8000ebb0 tx=0x7f8e8000eec0 comp rx=0 tx=0).stop 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 shutdown_connections 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8e74071fc0 0x7f8e74074480 unknown :-1 s=CLOSED pgs=3724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8e8c1076b0 0x7f8e8c110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8e8c102960 0x7f8e8c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:05.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 --2- 172.21.15.50:0/436343053 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8e8c103310 0x7f8e8c110820 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:05.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.132+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 >> 172.21.15.50:0/436343053 conn(0x7f8e8c0792d0 msgr2=0x7f8e8c1005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:05.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 shutdown_connections 2024-04-26T16:27:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.136+0000 7f8e913ac700 1 -- 172.21.15.50:0/436343053 wait complete. 2024-04-26T16:27:05.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:05.136+0000 7f8e913ac700 1 librados: shutdown 2024-04-26T16:27:05.144 INFO:teuthology.orchestra.run.smithi050.stderr:clear 2.15 pg_upmap mapping 2024-04-26T16:27:05.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:05.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:05.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:05.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:04.407060+0000 mon.a (mon.0) 2360 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]: dispatch 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:04.407060+0000 mon.a (mon.0) 2360 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]: dispatch 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:04.977069+0000 mon.c (mon.2) 554 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:04.977069+0000 mon.c (mon.2) 554 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:05.133932+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]': finished 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:05.133932+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]': finished 2024-04-26T16:27:05.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:05.134047+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]': finished 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: audit 2024-04-26T16:27:05.134047+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]': finished 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: cluster 2024-04-26T16:27:05.138192+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[21463]: cluster 2024-04-26T16:27:05.138192+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:04.407060+0000 mon.a (mon.0) 2360 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]: dispatch 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:04.407060+0000 mon.a (mon.0) 2360 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]: dispatch 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:04.977069+0000 mon.c (mon.2) 554 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:04.977069+0000 mon.c (mon.2) 554 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:05.133932+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]': finished 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:05.133932+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]': finished 2024-04-26T16:27:05.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:05.134047+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]': finished 2024-04-26T16:27:05.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: audit 2024-04-26T16:27:05.134047+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]': finished 2024-04-26T16:27:05.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: cluster 2024-04-26T16:27:05.138192+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-04-26T16:27:05.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:05 smithi050 bash[29234]: cluster 2024-04-26T16:27:05.138192+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-04-26T16:27:05.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.383 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:05.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:05.408 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:27:05.409 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:27:05.409 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:27:05.409 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:27:05.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:05.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:05.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:04.407060+0000 mon.a (mon.0) 2360 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]: dispatch 2024-04-26T16:27:05.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:04.407060+0000 mon.a (mon.0) 2360 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]: dispatch 2024-04-26T16:27:05.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:04.977069+0000 mon.c (mon.2) 554 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:04.977069+0000 mon.c (mon.2) 554 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:05.133932+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]': finished 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:05.133932+0000 mon.a (mon.0) 2361 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-41"}]': finished 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:05.134047+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]': finished 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: audit 2024-04-26T16:27:05.134047+0000 mon.a (mon.0) 2362 : audit [INF] from='client.? 172.21.15.50:0/436343053' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "2.15"}]': finished 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: cluster 2024-04-26T16:27:05.138192+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-04-26T16:27:05.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:05 smithi190 bash[23868]: cluster 2024-04-26T16:27:05.138192+0000 mon.a (mon.0) 2363 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2024-04-26T16:27:05.607 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:05.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:05.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:05.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:27:05.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:05.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:05.884 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:05.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:05.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:05.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:06.021 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:06.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:06.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:06.149 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:06.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-26T16:27:06.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:06.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:06.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:06.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: cluster 2024-04-26T16:27:05.599871+0000 mgr.y (mgr.24401) 383 : cluster [DBG] pgmap v520: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:06.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: cluster 2024-04-26T16:27:05.599871+0000 mgr.y (mgr.24401) 383 : cluster [DBG] pgmap v520: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:05.662447+0000 mon.c (mon.2) 555 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:05.662447+0000 mon.c (mon.2) 555 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:05.663301+0000 mon.c (mon.2) 556 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:05.663301+0000 mon.c (mon.2) 556 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:05.669780+0000 mon.a (mon.0) 2364 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:05.669780+0000 mon.a (mon.0) 2364 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: cluster 2024-04-26T16:27:06.109414+0000 mon.a (mon.0) 2365 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: cluster 2024-04-26T16:27:06.109414+0000 mon.a (mon.0) 2365 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: cluster 2024-04-26T16:27:06.121101+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: cluster 2024-04-26T16:27:06.121101+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:06.150450+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:06.395 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[23868]: audit 2024-04-26T16:27:06.150450+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:06.405 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:06.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:06.536 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:06.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: cluster 2024-04-26T16:27:05.599871+0000 mgr.y (mgr.24401) 383 : cluster [DBG] pgmap v520: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:06.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: cluster 2024-04-26T16:27:05.599871+0000 mgr.y (mgr.24401) 383 : cluster [DBG] pgmap v520: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:06.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:05.662447+0000 mon.c (mon.2) 555 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:06.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:05.662447+0000 mon.c (mon.2) 555 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:06.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:05.663301+0000 mon.c (mon.2) 556 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:05.663301+0000 mon.c (mon.2) 556 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:05.669780+0000 mon.a (mon.0) 2364 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:05.669780+0000 mon.a (mon.0) 2364 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: cluster 2024-04-26T16:27:06.109414+0000 mon.a (mon.0) 2365 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: cluster 2024-04-26T16:27:06.109414+0000 mon.a (mon.0) 2365 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: cluster 2024-04-26T16:27:06.121101+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: cluster 2024-04-26T16:27:06.121101+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:06.150450+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[29234]: audit 2024-04-26T16:27:06.150450+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: cluster 2024-04-26T16:27:05.599871+0000 mgr.y (mgr.24401) 383 : cluster [DBG] pgmap v520: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: cluster 2024-04-26T16:27:05.599871+0000 mgr.y (mgr.24401) 383 : cluster [DBG] pgmap v520: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:05.662447+0000 mon.c (mon.2) 555 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:05.662447+0000 mon.c (mon.2) 555 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:05.663301+0000 mon.c (mon.2) 556 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:05.663301+0000 mon.c (mon.2) 556 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:05.669780+0000 mon.a (mon.0) 2364 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:05.669780+0000 mon.a (mon.0) 2364 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: cluster 2024-04-26T16:27:06.109414+0000 mon.a (mon.0) 2365 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: cluster 2024-04-26T16:27:06.109414+0000 mon.a (mon.0) 2365 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: cluster 2024-04-26T16:27:06.121101+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: cluster 2024-04-26T16:27:06.121101+0000 mon.a (mon.0) 2366 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:06.150450+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:06 smithi050 bash[21463]: audit 2024-04-26T16:27:06.150450+0000 mon.a (mon.0) 2367 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:06.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:06.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:06.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:27:06.773 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:06.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:06.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:06.929 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:06.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:07.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:07.044 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:27:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:27:07.058 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:07.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:07.189 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:07.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:07.320 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:07.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:07.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:07.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:07.583 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:07.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:07.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:07.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:07.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:07.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:07.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:08.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:08.108 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:08.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:06.795966+0000 mgr.y (mgr.24401) 384 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:06.795966+0000 mgr.y (mgr.24401) 384 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.117712+0000 mon.a (mon.0) 2368 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.117712+0000 mon.a (mon.0) 2368 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: cluster 2024-04-26T16:27:07.126750+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: cluster 2024-04-26T16:27:07.126750+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.130182+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:08.220 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.130182+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.132729+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.132729+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.133603+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: audit 2024-04-26T16:27:07.133603+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: cluster 2024-04-26T16:27:07.600649+0000 mgr.y (mgr.24401) 385 : cluster [DBG] pgmap v523: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[29234]: cluster 2024-04-26T16:27:07.600649+0000 mgr.y (mgr.24401) 385 : cluster [DBG] pgmap v523: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:06.795966+0000 mgr.y (mgr.24401) 384 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:06.795966+0000 mgr.y (mgr.24401) 384 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.117712+0000 mon.a (mon.0) 2368 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.117712+0000 mon.a (mon.0) 2368 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: cluster 2024-04-26T16:27:07.126750+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-04-26T16:27:08.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: cluster 2024-04-26T16:27:07.126750+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.130182+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.130182+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.132729+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.132729+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.133603+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: audit 2024-04-26T16:27:07.133603+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: cluster 2024-04-26T16:27:07.600649+0000 mgr.y (mgr.24401) 385 : cluster [DBG] pgmap v523: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:08.222 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:08 smithi050 bash[21463]: cluster 2024-04-26T16:27:07.600649+0000 mgr.y (mgr.24401) 385 : cluster [DBG] pgmap v523: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:08.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:06.795966+0000 mgr.y (mgr.24401) 384 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:06.795966+0000 mgr.y (mgr.24401) 384 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.117712+0000 mon.a (mon.0) 2368 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.117712+0000 mon.a (mon.0) 2368 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: cluster 2024-04-26T16:27:07.126750+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: cluster 2024-04-26T16:27:07.126750+0000 mon.a (mon.0) 2369 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.130182+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.130182+0000 mon.a (mon.0) 2370 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]: dispatch 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.132729+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.132729+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.133603+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.349 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: audit 2024-04-26T16:27:07.133603+0000 mon.b (mon.1) 241 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:08.349 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: cluster 2024-04-26T16:27:07.600649+0000 mgr.y (mgr.24401) 385 : cluster [DBG] pgmap v523: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:08.349 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:08 smithi190 bash[23868]: cluster 2024-04-26T16:27:07.600649+0000 mgr.y (mgr.24401) 385 : cluster [DBG] pgmap v523: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:08.364 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:08.467 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:08.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:08.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:08.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:08.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:08.752 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:08.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:08.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:08.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:08.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:09.009 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:09.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:09.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:09.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:09.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.128572+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.128572+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.128712+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.128712+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: cluster 2024-04-26T16:27:08.138975+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: cluster 2024-04-26T16:27:08.138975+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-04-26T16:27:09.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.147835+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.147835+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.148079+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-6","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.148079+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-6","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.149100+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.149100+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.182365+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.182365+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.185131+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.185131+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.185522+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.185522+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.186592+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.186592+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.188292+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.188292+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.189358+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:09 smithi190 bash[23868]: audit 2024-04-26T16:27:08.189358+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.128572+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:09.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.128572+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.128712+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.128712+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: cluster 2024-04-26T16:27:08.138975+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: cluster 2024-04-26T16:27:08.138975+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.147835+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.147835+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.148079+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-6","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.148079+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-6","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.149100+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.149100+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.182365+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.182365+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.128572+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.128572+0000 mon.a (mon.0) 2372 : audit [INF] from='client.? 172.21.15.50:0/4229062425' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi050-73201-56"}]': finished 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.128712+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.128712+0000 mon.a (mon.0) 2373 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: cluster 2024-04-26T16:27:08.138975+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: cluster 2024-04-26T16:27:08.138975+0000 mon.a (mon.0) 2374 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.147835+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.147835+0000 mon.a (mon.0) 2375 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.148079+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-6","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.148079+0000 mon.b (mon.1) 242 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-6","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.149100+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.149100+0000 mon.b (mon.1) 243 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:09.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.182365+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.182365+0000 mon.a (mon.0) 2376 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.185131+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.185131+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.185522+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.185522+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.186592+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.186592+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.188292+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.188292+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.189358+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[21463]: audit 2024-04-26T16:27:08.189358+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.185131+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.185131+0000 mon.b (mon.1) 244 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.185522+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.185522+0000 mon.a (mon.0) 2377 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.186592+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.186592+0000 mon.a (mon.0) 2378 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.188292+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.188292+0000 mon.b (mon.1) 245 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.189358+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:09 smithi050 bash[29234]: audit 2024-04-26T16:27:08.189358+0000 mon.b (mon.1) 246 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:09.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:09.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:09.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:09.538 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:09.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:09.669 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:09.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:09.798 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:09.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:09.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:09.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:10.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:10.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:10.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:10.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.143275+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:10.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.143275+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:10.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.143581+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:10.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.143581+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:10.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: cluster 2024-04-26T16:27:09.150698+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-04-26T16:27:10.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: cluster 2024-04-26T16:27:09.150698+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-04-26T16:27:10.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.152025+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.152025+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.152239+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.152239+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.153055+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.153055+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.153693+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.182 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: audit 2024-04-26T16:27:09.153693+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.182 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: cluster 2024-04-26T16:27:09.601621+0000 mgr.y (mgr.24401) 386 : cluster [DBG] pgmap v526: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:10.182 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[21463]: cluster 2024-04-26T16:27:09.601621+0000 mgr.y (mgr.24401) 386 : cluster [DBG] pgmap v526: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:10.183 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.143275+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:10.183 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.143275+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:10.184 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.143581+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.143581+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: cluster 2024-04-26T16:27:09.150698+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: cluster 2024-04-26T16:27:09.150698+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.152025+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.152025+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.152239+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.152239+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.153055+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.153055+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.153693+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: audit 2024-04-26T16:27:09.153693+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.185 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: cluster 2024-04-26T16:27:09.601621+0000 mgr.y (mgr.24401) 386 : cluster [DBG] pgmap v526: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:10.186 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:10 smithi050 bash[29234]: cluster 2024-04-26T16:27:09.601621+0000 mgr.y (mgr.24401) 386 : cluster [DBG] pgmap v526: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:10.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:10.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:10.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:10.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:10.437 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.143275+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:10.437 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.143275+0000 mon.a (mon.0) 2379 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:10.437 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.143581+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.143581+0000 mon.a (mon.0) 2380 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi050-73201-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: cluster 2024-04-26T16:27:09.150698+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: cluster 2024-04-26T16:27:09.150698+0000 mon.a (mon.0) 2381 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.152025+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.152025+0000 mon.b (mon.1) 247 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.152239+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.152239+0000 mon.b (mon.1) 248 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.153055+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.153055+0000 mon.a (mon.0) 2382 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.153693+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: audit 2024-04-26T16:27:09.153693+0000 mon.a (mon.0) 2383 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]: dispatch 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: cluster 2024-04-26T16:27:09.601621+0000 mgr.y (mgr.24401) 386 : cluster [DBG] pgmap v526: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:10.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:10 smithi190 bash[23868]: cluster 2024-04-26T16:27:09.601621+0000 mgr.y (mgr.24401) 386 : cluster [DBG] pgmap v526: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:10.453 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:10.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:10.495 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:10.496 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:10.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:10.584 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:10.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:10.714 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:10.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:10.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:10.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:10.977 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:10.977 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:27:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:10.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:11.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:11.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:11.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:11.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: audit 2024-04-26T16:27:10.156000+0000 mon.a (mon.0) 2384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]': finished 2024-04-26T16:27:11.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: audit 2024-04-26T16:27:10.156000+0000 mon.a (mon.0) 2384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]': finished 2024-04-26T16:27:11.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: audit 2024-04-26T16:27:10.166972+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: audit 2024-04-26T16:27:10.166972+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: cluster 2024-04-26T16:27:10.178585+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-04-26T16:27:11.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: cluster 2024-04-26T16:27:10.178585+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-04-26T16:27:11.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: audit 2024-04-26T16:27:10.179639+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:11 smithi190 bash[23868]: audit 2024-04-26T16:27:10.179639+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.238 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:11.298 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:27:11.298 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:27:11.299 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-04-26T16:27:11.299 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-26T16:27:11.299 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-26T16:27:11.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-04-26T16:27:11.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:11.369 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: audit 2024-04-26T16:27:10.156000+0000 mon.a (mon.0) 2384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]': finished 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: audit 2024-04-26T16:27:10.156000+0000 mon.a (mon.0) 2384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]': finished 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: audit 2024-04-26T16:27:10.166972+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: audit 2024-04-26T16:27:10.166972+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: cluster 2024-04-26T16:27:10.178585+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: cluster 2024-04-26T16:27:10.178585+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: audit 2024-04-26T16:27:10.179639+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[21463]: audit 2024-04-26T16:27:10.179639+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: audit 2024-04-26T16:27:10.156000+0000 mon.a (mon.0) 2384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]': finished 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: audit 2024-04-26T16:27:10.156000+0000 mon.a (mon.0) 2384 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_count","val": "8"}]': finished 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: audit 2024-04-26T16:27:10.166972+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: audit 2024-04-26T16:27:10.166972+0000 mon.b (mon.1) 249 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: cluster 2024-04-26T16:27:10.178585+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: cluster 2024-04-26T16:27:10.178585+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: audit 2024-04-26T16:27:10.179639+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:11 smithi050 bash[29234]: audit 2024-04-26T16:27:10.179639+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:11.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:11.496 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:11.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:11.628 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:11.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:11.756 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:11.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:11.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:11.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:11.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:12.016 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:12.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:12.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:12.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:12.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:12.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.165173+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.165173+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.165269+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.165269+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: cluster 2024-04-26T16:27:11.169444+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: cluster 2024-04-26T16:27:11.169444+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.179376+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.179376+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.181680+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.181680+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.576507+0000 mon.c (mon.2) 557 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:12.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: audit 2024-04-26T16:27:11.576507+0000 mon.c (mon.2) 557 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: cluster 2024-04-26T16:27:11.604675+0000 mgr.y (mgr.24401) 387 : cluster [DBG] pgmap v529: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:12 smithi190 bash[23868]: cluster 2024-04-26T16:27:11.604675+0000 mgr.y (mgr.24401) 387 : cluster [DBG] pgmap v529: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:12.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:12.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:12.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.165173+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.165173+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.165269+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.165269+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: cluster 2024-04-26T16:27:11.169444+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: cluster 2024-04-26T16:27:11.169444+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.179376+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.179376+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.181680+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.181680+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.576507+0000 mon.c (mon.2) 557 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:12.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: audit 2024-04-26T16:27:11.576507+0000 mon.c (mon.2) 557 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: cluster 2024-04-26T16:27:11.604675+0000 mgr.y (mgr.24401) 387 : cluster [DBG] pgmap v529: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[29234]: cluster 2024-04-26T16:27:11.604675+0000 mgr.y (mgr.24401) 387 : cluster [DBG] pgmap v529: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.165173+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.165173+0000 mon.a (mon.0) 2387 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi050-73201-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.165269+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.165269+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: cluster 2024-04-26T16:27:11.169444+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-04-26T16:27:12.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: cluster 2024-04-26T16:27:11.169444+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.179376+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.179376+0000 mon.a (mon.0) 2390 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.181680+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.181680+0000 mon.b (mon.1) 250 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.576507+0000 mon.c (mon.2) 557 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: audit 2024-04-26T16:27:11.576507+0000 mon.c (mon.2) 557 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: cluster 2024-04-26T16:27:11.604675+0000 mgr.y (mgr.24401) 387 : cluster [DBG] pgmap v529: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:12.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:12 smithi050 bash[21463]: cluster 2024-04-26T16:27:11.604675+0000 mgr.y (mgr.24401) 387 : cluster [DBG] pgmap v529: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:12.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:12.534 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: ok, hit_set contains 275:602f83fe:::foo:head 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8989 ms) 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg_num = 32 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 0 ls 1714148833,0 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 1 ls 1714148833,0 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 2 ls 1714148833,0 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 3 ls 1714148833,0 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 4 ls 1714148833,0 2024-04-26T16:27:12.560 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 5 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 6 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 7 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 8 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 9 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 10 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 11 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 12 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 13 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 14 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 15 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 16 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 17 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 18 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 19 ls 1714148833,0 2024-04-26T16:27:12.561 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 20 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 21 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 22 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 23 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 24 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 25 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 26 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 27 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 28 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 29 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 30 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg 31 ls 1714148833,0 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: pg_num = 32 2024-04-26T16:27:12.562 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6cac518f:::0:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:02547ec2:::1:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f905c69b:::2:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:cfc208b3:::3:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d83876eb:::4:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b29083e3:::5:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c4fdafeb:::6:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:5c6b0b28:::7:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:bd63b0f1:::8:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e960b815:::9:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:52ea6a34:::10:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:89d3ae78:::11:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:de5d7c5f:::12:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:566253c9:::13:head 2024-04-26T16:27:12.563 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:62a1935d:::14:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:863748b0:::15:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3958e169:::16:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4d4dabf9:::17:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:8391935d:::18:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:28883081:::19:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:69259c59:::20:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4bdb80b7:::21:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a11c5d71:::22:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:271af37b:::23:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:95b121be:::24:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:58d1031b:::25:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:0a050783:::26:head 2024-04-26T16:27:12.564 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c709704c:::27:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:cbe56eaf:::28:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:86b4b162:::29:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:70d89383:::30:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:dd450c7c:::31:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6d5729b1:::32:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c388f3fb:::33:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:56cfea31:::34:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9dbc1bf7:::35:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:40b74ccd:::36:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4d5aaf42:::37:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:920f362c:::38:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6cc53222:::39:head 2024-04-26T16:27:12.565 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9cad833f:::40:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:1ea84d41:::41:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c4480ef6:::42:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a694361e:::43:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d1bd33e9:::44:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ddc2cd5d:::45:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:2b782207:::46:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7b187fca:::47:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:90ecdf6f:::48:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a5ed95fe:::49:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ea0eaa55:::50:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f33ef17b:::51:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a0d1b2f6:::52:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:60c5229e:::53:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:edcbc575:::54:head 2024-04-26T16:27:12.566 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:102cf253:::55:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:efb7fb0b:::56:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:50d0a326:::57:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d4dc5daf:::58:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3a130462:::59:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ec87ed71:::60:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d5bc9454:::61:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3ddfe313:::62:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7c2816b9:::63:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:47e00e4d:::64:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c6410c18:::65:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b48ed237:::66:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:cd63ad31:::67:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b179e92b:::68:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:0d9f741a:::69:head 2024-04-26T16:27:12.567 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6d3352ae:::70:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c6d5c19e:::71:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:bc4729c3:::72:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:77e930b9:::73:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:0abeecfd:::74:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b7c37e15:::75:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b6378398:::76:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:02bd68de:::77:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:cc795d2d:::78:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:630d4fea:::79:head 2024-04-26T16:27:12.568 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e0d29ef5:::80:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:fd6f13d2:::81:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:606461d5:::82:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:eadbdc43:::83:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:8761d0bb:::84:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9ef0186f:::85:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e0d41294:::86:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:961de695:::87:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:1423148f:::88:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:633a8fa2:::89:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a8653809:::90:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3dac8b33:::91:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:35aad435:::92:head 2024-04-26T16:27:12.569 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f6dcc343:::93:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:dbbdad87:::94:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:1cb48ce0:::95:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:03cd461c:::96:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:17a4ea99:::97:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9993c9a7:::98:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6394211c:::99:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:94c7ae57:::100:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6fdee5bb:::101:head 2024-04-26T16:27:12.570 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9a477fd1:::102:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:eb850916:::103:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:affc56b9:::104:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b42dc814:::105:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f319f8f0:::106:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9a40b9de:::107:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:8b524f28:::108:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e3de589f:::109:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:90f90a5b:::110:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a7b4f1d7:::111:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:af51766e:::112:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b6f90bd1:::113:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e0261208:::114:head 2024-04-26T16:27:12.571 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c9569ef7:::115:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:61bebe50:::116:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:fe93412b:::117:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d3d38bee:::118:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3100ba0c:::119:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d0560ada:::120:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f0ea8b35:::121:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:766f231a:::122:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a07a2582:::123:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:bd7c6b3a:::124:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:fb2ddaff:::125:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4408e1fe:::126:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ee1df7a7:::127:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c3002909:::128:head 2024-04-26T16:27:12.572 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4f48ffa9:::129:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:edf38733:::130:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c08425c0:::131:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:5f902d98:::132:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:41ea2c93:::133:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:813cee13:::134:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:0131818d:::135:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:26ba5a85:::136:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:381b8a5a:::137:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:28797e47:::138:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:bfca7f22:::139:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:36807075:::140:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:80b03975:::141:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:5c15709b:::142:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f39ea15e:::143:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ea992956:::144:head 2024-04-26T16:27:12.573 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:48887b1c:::145:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:9f24a9dd:::146:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:987f100b:::147:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d2dd3581:::148:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7fed1808:::149:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c80b70e9:::150:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:85ed90f9:::151:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:36428b24:::152:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d044c34a:::153:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7c18bf58:::154:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d1c21232:::155:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a7a3c575:::156:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:87da0633:::157:head 2024-04-26T16:27:12.574 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d5ac3822:::158:head 2024-04-26T16:27:12.575 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3f20522d:::159:head 2024-04-26T16:27:12.575 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6ca26563:::160:head 2024-04-26T16:27:12.575 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:532ce135:::161:head 2024-04-26T16:27:12.575 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:c78863e6:::162:head 2024-04-26T16:27:12.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:12.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:12.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:12.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:12.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:12.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:12.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:13.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:13.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:13.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:13.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:13.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:13.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:13.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: cluster 2024-04-26T16:27:12.167041+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: cluster 2024-04-26T16:27:12.167041+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.168224+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.168224+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: cluster 2024-04-26T16:27:12.174026+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: cluster 2024-04-26T16:27:12.174026+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.568047+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-43","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.568047+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-43","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.747529+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.747529+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.749169+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.749169+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.750038+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.750038+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.751740+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:13 smithi190 bash[23868]: audit 2024-04-26T16:27:12.751740+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:13.436 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:13.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: cluster 2024-04-26T16:27:12.167041+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: cluster 2024-04-26T16:27:12.167041+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.168224+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.168224+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: cluster 2024-04-26T16:27:12.174026+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: cluster 2024-04-26T16:27:12.174026+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.568047+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-43","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.568047+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-43","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.747529+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.747529+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.749169+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.749169+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.750038+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.750038+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.751740+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[29234]: audit 2024-04-26T16:27:12.751740+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: cluster 2024-04-26T16:27:12.167041+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: cluster 2024-04-26T16:27:12.167041+0000 mon.a (mon.0) 2391 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.168224+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.168224+0000 mon.a (mon.0) 2392 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: cluster 2024-04-26T16:27:12.174026+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: cluster 2024-04-26T16:27:12.174026+0000 mon.a (mon.0) 2393 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2024-04-26T16:27:13.550 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.568047+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-43","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.568047+0000 mon.b (mon.1) 251 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi050-73372-43","var": "pg_num","format": "json"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.747529+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.747529+0000 mon.a (mon.0) 2394 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.749169+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.749169+0000 mon.a (mon.0) 2395 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.750038+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.750038+0000 mon.b (mon.1) 252 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.751740+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:13 smithi050 bash[21463]: audit 2024-04-26T16:27:12.751740+0000 mon.b (mon.1) 253 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]: dispatch 2024-04-26T16:27:13.560 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:13.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:13.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:13.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:13.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:13.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:13.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:13.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:13.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:14.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:14.089 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:14.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:14.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:14.227 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:14.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:14.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:14.358 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:14.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:14.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:14.462 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:27:14.462 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:27:14.462 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:27:14.462 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:27:14.463 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:27:14.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: audit 2024-04-26T16:27:13.187847+0000 mon.a (mon.0) 2396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]': finished 2024-04-26T16:27:14.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: audit 2024-04-26T16:27:13.187847+0000 mon.a (mon.0) 2396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]': finished 2024-04-26T16:27:14.470 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: cluster 2024-04-26T16:27:13.192390+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: cluster 2024-04-26T16:27:13.192390+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: audit 2024-04-26T16:27:13.196858+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: audit 2024-04-26T16:27:13.196858+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: audit 2024-04-26T16:27:13.200718+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: audit 2024-04-26T16:27:13.200718+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: cluster 2024-04-26T16:27:13.605354+0000 mgr.y (mgr.24401) 388 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:14.471 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:14 smithi190 bash[23868]: cluster 2024-04-26T16:27:13.605354+0000 mgr.y (mgr.24401) 388 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:14.483 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:14.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:14.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:14.597 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:27:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: audit 2024-04-26T16:27:13.187847+0000 mon.a (mon.0) 2396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]': finished 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: audit 2024-04-26T16:27:13.187847+0000 mon.a (mon.0) 2396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]': finished 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: cluster 2024-04-26T16:27:13.192390+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: cluster 2024-04-26T16:27:13.192390+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: audit 2024-04-26T16:27:13.196858+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: audit 2024-04-26T16:27:13.196858+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: audit 2024-04-26T16:27:13.200718+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: audit 2024-04-26T16:27:13.200718+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: cluster 2024-04-26T16:27:13.605354+0000 mgr.y (mgr.24401) 388 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:14.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[29234]: cluster 2024-04-26T16:27:13.605354+0000 mgr.y (mgr.24401) 388 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: audit 2024-04-26T16:27:13.187847+0000 mon.a (mon.0) 2396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]': finished 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: audit 2024-04-26T16:27:13.187847+0000 mon.a (mon.0) 2396 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-43"}]': finished 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: cluster 2024-04-26T16:27:13.192390+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: cluster 2024-04-26T16:27:13.192390+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: audit 2024-04-26T16:27:13.196858+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: audit 2024-04-26T16:27:13.196858+0000 mon.b (mon.1) 254 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: audit 2024-04-26T16:27:13.200718+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: audit 2024-04-26T16:27:13.200718+0000 mon.a (mon.0) 2398 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: cluster 2024-04-26T16:27:13.605354+0000 mgr.y (mgr.24401) 388 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:14.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:14 smithi050 bash[21463]: cluster 2024-04-26T16:27:13.605354+0000 mgr.y (mgr.24401) 388 : cluster [DBG] pgmap v532: 228 pgs: 228 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:14.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:14.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:14.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:14.972 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:14.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:15.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:15.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:15.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:15.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:15.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-26T16:27:15.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: audit 2024-04-26T16:27:14.206515+0000 mon.a (mon.0) 2399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: audit 2024-04-26T16:27:14.206515+0000 mon.a (mon.0) 2399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: cluster 2024-04-26T16:27:14.215144+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: cluster 2024-04-26T16:27:14.215144+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: audit 2024-04-26T16:27:14.220700+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: audit 2024-04-26T16:27:14.220700+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: audit 2024-04-26T16:27:14.224217+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.350 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[29234]: audit 2024-04-26T16:27:14.224217+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: audit 2024-04-26T16:27:14.206515+0000 mon.a (mon.0) 2399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: audit 2024-04-26T16:27:14.206515+0000 mon.a (mon.0) 2399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: cluster 2024-04-26T16:27:14.215144+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: cluster 2024-04-26T16:27:14.215144+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: audit 2024-04-26T16:27:14.220700+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: audit 2024-04-26T16:27:14.220700+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: audit 2024-04-26T16:27:14.224217+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.351 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:15 smithi050 bash[21463]: audit 2024-04-26T16:27:14.224217+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:15.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:15.477 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: audit 2024-04-26T16:27:14.206515+0000 mon.a (mon.0) 2399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:15.477 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: audit 2024-04-26T16:27:14.206515+0000 mon.a (mon.0) 2399 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:15.477 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: cluster 2024-04-26T16:27:14.215144+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-04-26T16:27:15.477 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: cluster 2024-04-26T16:27:14.215144+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2024-04-26T16:27:15.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: audit 2024-04-26T16:27:14.220700+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: audit 2024-04-26T16:27:14.220700+0000 mon.b (mon.1) 255 : audit [INF] from='client.? 172.21.15.50:0/1359064824' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: audit 2024-04-26T16:27:14.224217+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:15 smithi190 bash[23868]: audit 2024-04-26T16:27:14.224217+0000 mon.a (mon.0) 2401 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]: dispatch 2024-04-26T16:27:15.488 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:15.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:15.619 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:15.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:15.750 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:15.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:15.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:15.879 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:15.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:15.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.009 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:16.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:16.155 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:16.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.269 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.209255+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.209255+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: cluster 2024-04-26T16:27:15.215884+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: cluster 2024-04-26T16:27:15.215884+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.235314+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.235314+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.237847+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.237847+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.255146+0000 mon.c (mon.2) 558 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.255146+0000 mon.c (mon.2) 558 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.255500+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.255500+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.256081+0000 mon.c (mon.2) 559 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.256081+0000 mon.c (mon.2) 559 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.256419+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.256419+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.256954+0000 mon.c (mon.2) 560 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.256954+0000 mon.c (mon.2) 560 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.257263+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:15.257263+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: cluster 2024-04-26T16:27:15.606160+0000 mgr.y (mgr.24401) 389 : cluster [DBG] pgmap v535: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: cluster 2024-04-26T16:27:15.606160+0000 mgr.y (mgr.24401) 389 : cluster [DBG] pgmap v535: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.118885+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.118885+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.119121+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.119121+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: cluster 2024-04-26T16:27:16.129950+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: cluster 2024-04-26T16:27:16.129950+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.136474+0000 mon.c (mon.2) 561 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.136474+0000 mon.c (mon.2) 561 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.143304+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.143304+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.153920+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.153920+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.166369+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[23868]: audit 2024-04-26T16:27:16.166369+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.285 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:16.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:16.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.209255+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:16.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.209255+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:16.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: cluster 2024-04-26T16:27:15.215884+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-04-26T16:27:16.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: cluster 2024-04-26T16:27:15.215884+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-04-26T16:27:16.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.235314+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.235314+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.237847+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.237847+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.255146+0000 mon.c (mon.2) 558 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.255146+0000 mon.c (mon.2) 558 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.255500+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.255500+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.256081+0000 mon.c (mon.2) 559 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.256081+0000 mon.c (mon.2) 559 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.256419+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.256419+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.256954+0000 mon.c (mon.2) 560 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.256954+0000 mon.c (mon.2) 560 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.257263+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:15.257263+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: cluster 2024-04-26T16:27:15.606160+0000 mgr.y (mgr.24401) 389 : cluster [DBG] pgmap v535: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: cluster 2024-04-26T16:27:15.606160+0000 mgr.y (mgr.24401) 389 : cluster [DBG] pgmap v535: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.118885+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:16.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.118885+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.119121+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.119121+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: cluster 2024-04-26T16:27:16.129950+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: cluster 2024-04-26T16:27:16.129950+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.136474+0000 mon.c (mon.2) 561 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.136474+0000 mon.c (mon.2) 561 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.143304+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.143304+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.153920+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.153920+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.166369+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[29234]: audit 2024-04-26T16:27:16.166369+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.209255+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.209255+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi050-73201-57"}]': finished 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: cluster 2024-04-26T16:27:15.215884+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: cluster 2024-04-26T16:27:15.215884+0000 mon.a (mon.0) 2403 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.235314+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.235314+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.237847+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.237847+0000 mon.b (mon.1) 256 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:16.404 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.255146+0000 mon.c (mon.2) 558 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.255146+0000 mon.c (mon.2) 558 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.255500+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.255500+0000 mon.a (mon.0) 2405 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.256081+0000 mon.c (mon.2) 559 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.256081+0000 mon.c (mon.2) 559 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.256419+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.256419+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.256954+0000 mon.c (mon.2) 560 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.256954+0000 mon.c (mon.2) 560 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.257263+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:15.257263+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: cluster 2024-04-26T16:27:15.606160+0000 mgr.y (mgr.24401) 389 : cluster [DBG] pgmap v535: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: cluster 2024-04-26T16:27:15.606160+0000 mgr.y (mgr.24401) 389 : cluster [DBG] pgmap v535: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 550 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:16.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.118885+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.118885+0000 mon.a (mon.0) 2408 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.119121+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.119121+0000 mon.a (mon.0) 2409 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi050-73201-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: cluster 2024-04-26T16:27:16.129950+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: cluster 2024-04-26T16:27:16.129950+0000 mon.a (mon.0) 2410 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.136474+0000 mon.c (mon.2) 561 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.136474+0000 mon.c (mon.2) 561 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.143304+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.143304+0000 mon.b (mon.1) 257 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.153920+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.153920+0000 mon.a (mon.0) 2411 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.166369+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:16 smithi050 bash[21463]: audit 2024-04-26T16:27:16.166369+0000 mon.a (mon.0) 2412 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:16.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:16.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.546 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:16.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:16.677 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:16.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:16.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:16.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.938 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:16.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:16.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- 172.21.15.50:0/2430973804 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec102f60 msgr2=0x7f4dec106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:16.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2430973804 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec102f60 0x7f4dec106f60 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f4de00099c0 tx=0x7f4de0009cd0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- 172.21.15.50:0/2430973804 shutdown_connections 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2430973804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dec107640 0x7f4dec10f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2430973804 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec1025b0 0x7f4dec102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2430973804 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec102f60 0x7f4dec106f60 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- 172.21.15.50:0/2430973804 >> 172.21.15.50:0/2430973804 conn(0x7f4dec0792d0 msgr2=0x7f4dec0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- 172.21.15.50:0/2430973804 shutdown_connections 2024-04-26T16:27:16.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- 172.21.15.50:0/2430973804 wait complete. 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 librados: starting msgr at 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 librados: starting objecter 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 Processor -- start 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- start start 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 librados: setting wanted keys 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 librados: calling monclient init 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dec1025b0 0x7f4dec19b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 0x7f4dec19b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec107640 0x7f4dec19fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec1119d0 con 0x7f4dec102f60 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4dec111850 con 0x7f4dec1025b0 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df36a6700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4dec111b50 con 0x7f4dec107640 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df0c41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 0x7f4dec19b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec107640 0x7f4dec19fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df0c41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 0x7f4dec19b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34548/0 (socket says 172.21.15.50:34548) 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df0c41700 1 -- 172.21.15.50:0/2834014423 learned_addr learned my addr 172.21.15.50:0/2834014423 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1442700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dec1025b0 0x7f4dec19b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 -- 172.21.15.50:0/2834014423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dec1025b0 msgr2=0x7f4dec19b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dec1025b0 0x7f4dec19b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 -- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 msgr2=0x7f4dec19b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 0x7f4dec19b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 -- 172.21.15.50:0/2834014423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dec1a0470 con 0x7f4dec107640 2024-04-26T16:27:16.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df0c41700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 0x7f4dec19b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4df1c43700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec107640 0x7f4dec19fd30 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f4de400cc50 tx=0x7f4de400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4dde7fc700 10 client.?.objecter ms_handle_connect 0x7f4dec107640 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.956+0000 7f4dde7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de4016020 con 0x7f4dec107640 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dec1a0760 con 0x7f4dec107640 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4de4012420 con 0x7f4dec107640 2024-04-26T16:27:16.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df36a6700 10 client.41093.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4de40115e0 con 0x7f4dec107640 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dec1a7fe0 con 0x7f4dec107640 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df36a6700 1 librados: init done 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dec103a40 con 0x7f4dec107640 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4de4019020 con 0x7f4dec107640 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4dd8071e50 0x7f4dd8074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(368..368 src has 87..368) ==== 9688+0+0 (secure 0 0 0) 0x7f4de4091720 con 0x7f4dec107640 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 10 client.41093.objecter ms_dispatch 0x7f4dec001550 osd_map(368..368 src has 87..368) 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 3 client.41093.objecter handle_osd_map got epochs [368,368] > 0 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 3 client.41093.objecter handle_osd_map decoding full epoch 368 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df1442700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4dd8071e50 0x7f4dd8074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 20 client.41093.objecter dump_active .. 0 homeless 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4dde7fc700 10 client.41093.objecter ms_handle_connect 0x7f4dd8071e50 2024-04-26T16:27:16.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.960+0000 7f4df1442700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4dd8071e50 0x7f4dd8074310 secure :-1 s=READY pgs=3727 cs=0 l=1 rev1=1 crypto rx=0x7f4de8009680 tx=0x7f4de8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:16.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.968+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4de4020030 con 0x7f4dec107640 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/3088856381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06bad0 msgr2=0x7fa7dc1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- 172.21.15.50:0/3088856381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06bad0 0x7fa7dc1102e0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400b0a0 tx=0x7fa7d400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/3088856381 shutdown_connections 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- 172.21.15.50:0/3088856381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06bad0 0x7fa7dc1102e0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- 172.21.15.50:0/3088856381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 0x7fa7dc109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- 172.21.15.50:0/3088856381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7dc06b130 0x7fa7dc06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/3088856381 >> 172.21.15.50:0/3088856381 conn(0x7fa7dc0792d0 msgr2=0x7fa7dc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/3088856381 shutdown_connections 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/3088856381 wait complete. 2024-04-26T16:27:16.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 librados: starting msgr at 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 librados: starting objecter 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 Processor -- start 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- start start 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 librados: setting wanted keys 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 librados: calling monclient init 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06b130 0x7fa7dc1990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7dc06bad0 0x7fa7dc199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 0x7fa7dc19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc112ad0 con 0x7fa7dc06b130 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa7dc112950 con 0x7fa7dc06bad0 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa7dc112c50 con 0x7fa7dc1094a0 2024-04-26T16:27:16.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7dbfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 0x7fa7dc19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06b130 0x7fa7dc1990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7dbfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 0x7fa7dc19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51204/0 (socket says 172.21.15.50:51204) 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7dbfff700 1 -- 172.21.15.50:0/1482734511 learned_addr learned my addr 172.21.15.50:0/1482734511 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7dc06bad0 0x7fa7dc199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 msgr2=0x7fa7dc19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 0x7fa7dc19d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7dc06bad0 msgr2=0x7fa7dc199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa7dc06bad0 0x7fa7dc199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7dc19e0e0 con 0x7fa7dc06b130 2024-04-26T16:27:16.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa7dc1094a0 0x7fa7dc19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06b130 0x7fa7dc1990d0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e9f0 tx=0x7fa7cc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7d8ff9700 10 client.?.objecter ms_handle_connect 0x7fa7dc06b130 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7d8ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc00cb70 con 0x7fa7dc06b130 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7dc19e3d0 con 0x7fa7dc06b130 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa7cc004480 con 0x7fa7dc06b130 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7cc003bf0 con 0x7fa7dc06b130 2024-04-26T16:27:16.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 10 client.30483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7dc102320 con 0x7fa7dc06b130 2024-04-26T16:27:16.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.992+0000 7fa7e223d700 1 librados: init done 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa7dc06a110 con 0x7fa7dc06b130 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa7cc003dd0 con 0x7fa7dc06b130 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa7c8071f80 0x7fa7c8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(368..368 src has 87..368) ==== 9688+0+0 (secure 0 0 0) 0x7fa7cc091cd0 con 0x7fa7dc06b130 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 osd_map(368..368 src has 87..368) 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 3 client.30483.objecter handle_osd_map got epochs [368,368] > 0 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 3 client.30483.objecter handle_osd_map decoding full epoch 368 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa7c8071f80 0x7fa7c8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 20 client.30483.objecter dump_active .. 0 homeless 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v368) ==== 54+0+16 (secure 0 0 0) 0x7fa7cc097320 con 0x7fa7dc06b130 2024-04-26T16:27:16.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 10 client.30483.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 1 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa7dc064030 0x7fa7dc064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 1 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 0 1 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 4 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 10 client.30483.objecter _send_command 1 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc103760 con 0x7fa7dc064030 2024-04-26T16:27:16.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7c8071f80 2024-04-26T16:27:17.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa7c8071f80 0x7fa7c8074440 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0796f0 tx=0x7fa7c4009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa7dc064030 0x7fa7dc064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa7dc064030 0x7fa7dc064410 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:16.996+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc064030 2024-04-26T16:27:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.000+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc103760 con 0x7fa7dc064030 2024-04-26T16:27:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.000+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 1: 0 ) 2024-04-26T16:27:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.000+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 1 = system:0 2024-04-26T16:27:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.000+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 4 2024-04-26T16:27:17.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.000+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 0 1 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 10 client.30483.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 2 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 2 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc19a430 osd=0 3 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 0 2 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 4 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 10 client.30483.objecter _send_command 2 2024-04-26T16:27:17.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.016+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc104180 con 0x7fa7dc064030 2024-04-26T16:27:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc104180 con 0x7fa7dc064030 2024-04-26T16:27:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 4 2024-04-26T16:27:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 0 2 2024-04-26T16:27:17.019 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:27:17.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 10 client.30483.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa7dc10a5f0 0x7fa7dc10a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 1 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 4 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 10 client.30483.objecter _send_command 3 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc063d00 con 0x7fa7dc10a5f0 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa7dc10a5f0 0x7fa7dc10a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc10a5f0 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa7dc10a5f0 0x7fa7dc10a9f0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc063d00 con 0x7fa7dc10a5f0 2024-04-26T16:27:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 3: 0 ) 2024-04-26T16:27:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 3 = system:0 2024-04-26T16:27:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 4 2024-04-26T16:27:17.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.020+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 1 3 2024-04-26T16:27:17.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 10 client.30483.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 4 2024-04-26T16:27:17.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 4 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc104390 osd=1 3 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 1 4 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 4 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 10 client.30483.objecter _send_command 4 2024-04-26T16:27:17.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc02d310 con 0x7fa7dc10a5f0 2024-04-26T16:27:17.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc02d310 con 0x7fa7dc10a5f0 2024-04-26T16:27:17.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 4 2024-04-26T16:27:17.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 1 4 2024-04-26T16:27:17.039 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 10 client.30483.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 5 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa7dc102db0 0x7fa7dc1031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 5 2024-04-26T16:27:17.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 2 5 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 4 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 10 client.30483.objecter _send_command 5 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc02e810 con 0x7fa7dc102db0 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa7dc102db0 0x7fa7dc1031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc102db0 2024-04-26T16:27:17.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.040+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa7dc102db0 0x7fa7dc1031f0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:17.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.044+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc02e810 con 0x7fa7dc102db0 2024-04-26T16:27:17.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.044+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 5: 0 ) 2024-04-26T16:27:17.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.044+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 5 = system:0 2024-04-26T16:27:17.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.044+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 4 2024-04-26T16:27:17.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.044+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 2 5 2024-04-26T16:27:17.052 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:27:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:27:17.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 10 client.30483.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 6 2024-04-26T16:27:17.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 6 2024-04-26T16:27:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc02e6a0 osd=2 3 2024-04-26T16:27:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 2 6 2024-04-26T16:27:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 4 2024-04-26T16:27:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 10 client.30483.objecter _send_command 6 2024-04-26T16:27:17.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc102830 con 0x7fa7dc102db0 2024-04-26T16:27:17.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc102830 con 0x7fa7dc102db0 2024-04-26T16:27:17.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 4 2024-04-26T16:27:17.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 2 6 2024-04-26T16:27:17.060 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:27:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 10 client.30483.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 7 2024-04-26T16:27:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa7dc1b1210 0x7fa7dc1b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 7 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 3 7 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 4 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 10 client.30483.objecter _send_command 7 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc02e370 con 0x7fa7dc1b1210 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.060+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa7dc1b1210 0x7fa7dc1b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc1b1210 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa7dc1b1210 0x7fa7dc1b3600 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc02e370 con 0x7fa7dc1b1210 2024-04-26T16:27:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 7: 0 ) 2024-04-26T16:27:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 7 = system:0 2024-04-26T16:27:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 4 2024-04-26T16:27:17.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.064+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 3 7 2024-04-26T16:27:17.069 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:17.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 10 client.30483.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 8 2024-04-26T16:27:17.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 8 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc02dcf0 osd=3 3 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 3 8 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 4 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 10 client.30483.objecter _send_command 8 2024-04-26T16:27:17.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.080+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc02d770 con 0x7fa7dc1b1210 2024-04-26T16:27:17.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc02d770 con 0x7fa7dc1b1210 2024-04-26T16:27:17.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 4 2024-04-26T16:27:17.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 3 8 2024-04-26T16:27:17.081 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:27:17.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 10 client.30483.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 9 2024-04-26T16:27:17.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa7dc02d8e0 0x7fa7dc1b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 9 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 4 9 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 4 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 10 client.30483.objecter _send_command 9 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc1b0ea0 con 0x7fa7dc02d8e0 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa7dc02d8e0 0x7fa7dc1b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa7dc02d8e0 0x7fa7dc1b6cd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc02d8e0 2024-04-26T16:27:17.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc1b0ea0 con 0x7fa7dc02d8e0 2024-04-26T16:27:17.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 9: 0 ) 2024-04-26T16:27:17.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 9 = system:0 2024-04-26T16:27:17.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 4 2024-04-26T16:27:17.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.084+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 4 9 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 10 client.30483.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 10 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 10 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc02d770 osd=4 3 2024-04-26T16:27:17.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 4 10 2024-04-26T16:27:17.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 4 2024-04-26T16:27:17.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 10 client.30483.objecter _send_command 10 2024-04-26T16:27:17.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.104+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc1b0ea0 con 0x7fa7dc02d8e0 2024-04-26T16:27:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc1b0ea0 con 0x7fa7dc02d8e0 2024-04-26T16:27:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 4 2024-04-26T16:27:17.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 4 10 2024-04-26T16:27:17.105 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 10 client.30483.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 11 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa7dc1b7720 0x7fa7dc1b9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 11 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 5 11 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 4 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 10 client.30483.objecter _send_command 11 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc1bcba0 con 0x7fa7dc1b7720 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa7dc1b7720 0x7fa7dc1b9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7daffd700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa7dc1b7720 0x7fa7dc1b9b50 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc1b7720 2024-04-26T16:27:17.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc1bcba0 con 0x7fa7dc1b7720 2024-04-26T16:27:17.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 11: 0 ) 2024-04-26T16:27:17.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 11 = system:0 2024-04-26T16:27:17.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 4 2024-04-26T16:27:17.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.108+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 5 11 2024-04-26T16:27:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 10 client.30483.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 12 2024-04-26T16:27:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 12 2024-04-26T16:27:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc1b0ea0 osd=5 3 2024-04-26T16:27:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 5 12 2024-04-26T16:27:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 4 2024-04-26T16:27:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 10 client.30483.objecter _send_command 12 2024-04-26T16:27:17.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.144+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc1bcba0 con 0x7fa7dc1b7720 2024-04-26T16:27:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc1bcba0 con 0x7fa7dc1b7720 2024-04-26T16:27:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 4 2024-04-26T16:27:17.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 5 12 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 10 client.30483.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 13 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fa7dc1bbed0 0x7fa7dc1bc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 13 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 6 13 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 4 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 10 client.30483.objecter _send_command 13 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc1bff90 con 0x7fa7dc1bbed0 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fa7dc1bbed0 0x7fa7dc1bc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7dbfff700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fa7dc1bbed0 0x7fa7dc1bc350 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc1bbed0 2024-04-26T16:27:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc1bff90 con 0x7fa7dc1bbed0 2024-04-26T16:27:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 13: 0 ) 2024-04-26T16:27:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 13 = system:0 2024-04-26T16:27:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 4 2024-04-26T16:27:17.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.148+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 6 13 2024-04-26T16:27:17.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 10 client.30483.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 14 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 14 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc1bcba0 osd=6 3 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 6 14 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 4 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 10 client.30483.objecter _send_command 14 2024-04-26T16:27:17.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.168+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc1bee10 con 0x7fa7dc1bbed0 2024-04-26T16:27:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc1bee10 con 0x7fa7dc1bbed0 2024-04-26T16:27:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 4 2024-04-26T16:27:17.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 6 14 2024-04-26T16:27:17.170 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:27:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 10 client.30483.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 15 2024-04-26T16:27:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa7dc1bf530 0x7fa7dc1bf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 15 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 7 15 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 4 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 10 client.30483.objecter _send_command 15 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa7dc1c3300 con 0x7fa7dc1bf530 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa7dc1bf530 0x7fa7dc1bf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7db7fe700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa7dc1bf530 0x7fa7dc1bf970 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:17.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 10 client.30483.objecter ms_handle_connect 0x7fa7dc1bf530 2024-04-26T16:27:17.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7dc1c3300 con 0x7fa7dc1bf530 2024-04-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 15: 0 ) 2024-04-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 15 = system:0 2024-04-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 4 2024-04-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.172+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 7 15 2024-04-26T16:27:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 10 client.30483.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign -1 16 2024-04-26T16:27:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 20 client.30483.objecter _get_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_remove -1 16 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 20 client.30483.objecter get_session s=0x7fa7dc1bee10 osd=7 3 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 15 client.30483.objecter _session_command_op_assign 7 16 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 4 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 10 client.30483.objecter _send_command 16 2024-04-26T16:27:17.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.192+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa7dc1ba1a0 con 0x7fa7dc1bf530 2024-04-26T16:27:17.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7d8ff9700 1 -- 172.21.15.50:0/1482734511 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa7dc1ba1a0 con 0x7fa7dc1bf530 2024-04-26T16:27:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7d8ff9700 10 client.30483.objecter ms_dispatch 0x7fa7dc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7d8ff9700 10 client.30483.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7d8ff9700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 4 2024-04-26T16:27:17.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7d8ff9700 15 client.30483.objecter _session_command_op_remove 7 16 2024-04-26T16:27:17.195 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:27:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 librados: watch_flush enter 2024-04-26T16:27:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 librados: watch_flush exit 2024-04-26T16:27:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.0 2024-04-26T16:27:17.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa7dc064030 msgr2=0x7fa7dc064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa7dc064030 0x7fa7dc064410 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc19a430 osd=0 1 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.1 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa7dc10a5f0 msgr2=0x7fa7dc10a9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa7dc10a5f0 0x7fa7dc10a9f0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc104390 osd=1 1 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.2 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa7dc102db0 msgr2=0x7fa7dc1031f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa7dc102db0 0x7fa7dc1031f0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02e6a0 osd=2 1 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.3 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa7dc1b1210 msgr2=0x7fa7dc1b3600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa7dc1b1210 0x7fa7dc1b3600 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02dcf0 osd=3 1 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.4 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa7dc02d8e0 msgr2=0x7fa7dc1b6cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa7dc02d8e0 0x7fa7dc1b6cd0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc02d770 osd=4 1 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.5 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa7dc1b7720 msgr2=0x7fa7dc1b9b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa7dc1b7720 0x7fa7dc1b9b50 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1b0ea0 osd=5 1 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.6 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fa7dc1bbed0 msgr2=0x7fa7dc1bc350 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7dbfff700 1 -- 172.21.15.50:0/1482734511 reap_dead start 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fa7dc1bbed0 0x7fa7dc1bc350 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7dbfff700 1 -- 172.21.15.50:0/1482734511 reap_dead start 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bcba0 osd=6 1 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter close_session for osd.7 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa7dc1bf530 msgr2=0x7fa7dc1bf970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa7dc1bf530 0x7fa7dc1bf970 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter put_session s=0x7fa7dc1bee10 osd=7 1 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 20 client.30483.objecter shutdown clearing up homeless session... 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 10 client.30483.objecter successfully canceled tick 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7dbfff700 1 -- 172.21.15.50:0/1482734511 reap_dead start 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa7c8071f80 msgr2=0x7fa7c8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa7c8071f80 0x7fa7c8074440 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7fa7dc0796f0 tx=0x7fa7c4009260 comp rx=0 tx=0).stop 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06b130 msgr2=0x7fa7dc1990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06b130 0x7fa7dc1990d0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc00e9f0 tx=0x7fa7cc00ed00 comp rx=0 tx=0).stop 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 shutdown_connections 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa7dc06b130 0x7fa7dc1990d0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 --2- 172.21.15.50:0/1482734511 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa7c8071f80 0x7fa7c8074440 unknown :-1 s=CLOSED pgs=3728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 >> 172.21.15.50:0/1482734511 conn(0x7fa7dc0792d0 msgr2=0x7fa7dc1001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 shutdown_connections 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 -- 172.21.15.50:0/1482734511 wait complete. 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.196+0000 7fa7e223d700 1 librados: shutdown 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:27:17.200 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:27:17.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:17.236+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f4dec06a110 con 0x7f4dec107640 2024-04-26T16:27:17.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:17.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:17.416 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:17.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:17.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.000209+0000 mon.a (mon.0) 2413 : audit [DBG] from='client.? 172.21.15.50:0/1482734511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:17.442 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.000209+0000 mon.a (mon.0) 2413 : audit [DBG] from='client.? 172.21.15.50:0/1482734511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.120682+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.120682+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.132825+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.132825+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: cluster 2024-04-26T16:27:17.133749+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: cluster 2024-04-26T16:27:17.133749+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.134546+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[21463]: audit 2024-04-26T16:27:17.134546+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.000209+0000 mon.a (mon.0) 2413 : audit [DBG] from='client.? 172.21.15.50:0/1482734511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.000209+0000 mon.a (mon.0) 2413 : audit [DBG] from='client.? 172.21.15.50:0/1482734511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:17.443 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.120682+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.120682+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.132825+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.132825+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: cluster 2024-04-26T16:27:17.133749+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: cluster 2024-04-26T16:27:17.133749+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.134546+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.444 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:17 smithi050 bash[29234]: audit 2024-04-26T16:27:17.134546+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:17.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:17.569 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.000209+0000 mon.a (mon.0) 2413 : audit [DBG] from='client.? 172.21.15.50:0/1482734511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.000209+0000 mon.a (mon.0) 2413 : audit [DBG] from='client.? 172.21.15.50:0/1482734511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:17.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.120682+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.120682+0000 mon.a (mon.0) 2414 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.132825+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.132825+0000 mon.b (mon.1) 258 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: cluster 2024-04-26T16:27:17.133749+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: cluster 2024-04-26T16:27:17.133749+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.134546+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:17 smithi190 bash[23868]: audit 2024-04-26T16:27:17.134546+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:27:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:17.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:17.701 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:17.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:17.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:17.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:17.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:17.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:18.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:18.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.132+0000 7f4dde7fc700 1 -- 172.21.15.50:0/2834014423 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v370) ==== 91+0+0 (secure 0 0 0) 0x7f4de405cfa0 con 0x7f4dec107640 2024-04-26T16:27:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.136+0000 7f4df36a6700 10 librados: watch_flush enter 2024-04-26T16:27:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 10 librados: watch_flush exit 2024-04-26T16:27:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 20 client.41093.objecter shutdown clearing up homeless session... 2024-04-26T16:27:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 10 client.41093.objecter successfully canceled tick 2024-04-26T16:27:18.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4dd8071e50 msgr2=0x7f4dd8074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4dd8071e50 0x7f4dd8074310 secure :-1 s=READY pgs=3727 cs=0 l=1 rev1=1 crypto rx=0x7f4de8009680 tx=0x7f4de8009260 comp rx=0 tx=0).stop 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec107640 msgr2=0x7f4dec19fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec107640 0x7f4dec19fd30 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f4de400cc50 tx=0x7f4de400cf60 comp rx=0 tx=0).stop 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 shutdown_connections 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4dd8071e50 0x7f4dd8074310 unknown :-1 s=CLOSED pgs=3727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4dec1025b0 0x7f4dec19b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4dec107640 0x7f4dec19fd30 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 --2- 172.21.15.50:0/2834014423 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4dec102f60 0x7f4dec19b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 >> 172.21.15.50:0/2834014423 conn(0x7f4dec0792d0 msgr2=0x7f4dec100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 shutdown_connections 2024-04-26T16:27:18.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 -- 172.21.15.50:0/2834014423 wait complete. 2024-04-26T16:27:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:18.140+0000 7f4df36a6700 1 librados: shutdown 2024-04-26T16:27:18.141 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:27:18.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:18.224 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:18.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:18.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:18.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:18.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:16.807210+0000 mgr.y (mgr.24401) 390 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:16.807210+0000 mgr.y (mgr.24401) 390 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:17.240555+0000 mon.c (mon.2) 562 : audit [INF] from='client.? 172.21.15.50:0/2834014423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:17.240555+0000 mon.c (mon.2) 562 : audit [INF] from='client.? 172.21.15.50:0/2834014423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:17.249711+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:17.249711+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: cluster 2024-04-26T16:27:17.606940+0000 mgr.y (mgr.24401) 391 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: cluster 2024-04-26T16:27:17.606940+0000 mgr.y (mgr.24401) 391 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.121761+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.121761+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.124434+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.124434+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.124634+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.124634+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.124762+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.124762+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.132957+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.132957+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.134027+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.134027+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.137482+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:18 smithi190 bash[23868]: audit 2024-04-26T16:27:18.137482+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:18.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:16.807210+0000 mgr.y (mgr.24401) 390 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:18.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:16.807210+0000 mgr.y (mgr.24401) 390 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:18.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:17.240555+0000 mon.c (mon.2) 562 : audit [INF] from='client.? 172.21.15.50:0/2834014423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:17.240555+0000 mon.c (mon.2) 562 : audit [INF] from='client.? 172.21.15.50:0/2834014423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:17.249711+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:17.249711+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: cluster 2024-04-26T16:27:17.606940+0000 mgr.y (mgr.24401) 391 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: cluster 2024-04-26T16:27:17.606940+0000 mgr.y (mgr.24401) 391 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.121761+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.121761+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.124434+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.124434+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.124634+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.124634+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.124762+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.124762+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:27:18.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.132957+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.132957+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.134027+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.134027+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.137482+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[21463]: audit 2024-04-26T16:27:18.137482+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:16.807210+0000 mgr.y (mgr.24401) 390 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:16.807210+0000 mgr.y (mgr.24401) 390 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:17.240555+0000 mon.c (mon.2) 562 : audit [INF] from='client.? 172.21.15.50:0/2834014423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:17.240555+0000 mon.c (mon.2) 562 : audit [INF] from='client.? 172.21.15.50:0/2834014423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:17.249711+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:17.249711+0000 mon.a (mon.0) 2417 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:27:18.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: cluster 2024-04-26T16:27:17.606940+0000 mgr.y (mgr.24401) 391 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: cluster 2024-04-26T16:27:17.606940+0000 mgr.y (mgr.24401) 391 : cluster [DBG] pgmap v538: 228 pgs: 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.121761+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.121761+0000 mon.a (mon.0) 2418 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.124434+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.124434+0000 mon.a (mon.0) 2419 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi050-73201-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.124634+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.124634+0000 mon.a (mon.0) 2420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.124762+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:27:18.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.124762+0000 mon.a (mon.0) 2421 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:27:18.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.132957+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.132957+0000 mon.b (mon.1) 259 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.134027+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-04-26T16:27:18.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.134027+0000 mon.a (mon.0) 2422 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2024-04-26T16:27:18.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.137482+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:18 smithi050 bash[29234]: audit 2024-04-26T16:27:18.137482+0000 mon.a (mon.0) 2423 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:27:18.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:18.610 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:18.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.741 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:18.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.873 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:18.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:18.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:19.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:19.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:19.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:19.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:19.279 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:19.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:19.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:19.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.245919+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.245919+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.905133+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.905133+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.906352+0000 osd.7 (osd.7) 232 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:18.906352+0000 osd.7 (osd.7) 232 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: audit 2024-04-26T16:27:19.127725+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: audit 2024-04-26T16:27:19.127725+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: audit 2024-04-26T16:27:19.139765+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: audit 2024-04-26T16:27:19.139765+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.142306+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.142306+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: audit 2024-04-26T16:27:19.145607+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: audit 2024-04-26T16:27:19.145607+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.157008+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.157008+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.159454+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:27:19.524 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:19 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.159454+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:27:19.536 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:19.608 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.245919+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.245919+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.905133+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.905133+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.906352+0000 osd.7 (osd.7) 232 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:18.906352+0000 osd.7 (osd.7) 232 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: audit 2024-04-26T16:27:19.127725+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: audit 2024-04-26T16:27:19.127725+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:27:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: audit 2024-04-26T16:27:19.139765+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: audit 2024-04-26T16:27:19.139765+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.142306+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.142306+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: audit 2024-04-26T16:27:19.145607+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: audit 2024-04-26T16:27:19.145607+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.157008+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.157008+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.159454+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.159454+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:27:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.245919+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.245919+0000 mon.a (mon.0) 2424 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.905133+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.905133+0000 osd.7 (osd.7) 231 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.906352+0000 osd.7 (osd.7) 232 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:18.906352+0000 osd.7 (osd.7) 232 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: audit 2024-04-26T16:27:19.127725+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: audit 2024-04-26T16:27:19.127725+0000 mon.a (mon.0) 2425 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: audit 2024-04-26T16:27:19.139765+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: audit 2024-04-26T16:27:19.139765+0000 mon.b (mon.1) 260 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.142306+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.142306+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: audit 2024-04-26T16:27:19.145607+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: audit 2024-04-26T16:27:19.145607+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.157008+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:27:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.157008+0000 osd.6 (osd.6) 167 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:27:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.159454+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:27:19.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:19 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.159454+0000 osd.6 (osd.6) 168 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:27:19.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:19.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:19.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:19.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:19.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:19.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:19.924 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:19.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:20.021 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:27:20.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:20.052 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:20.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:20.181 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:20.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.607751+0000 mgr.y (mgr.24401) 392 : cluster [DBG] pgmap v541: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:20.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.607751+0000 mgr.y (mgr.24401) 392 : cluster [DBG] pgmap v541: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:20.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.130356+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:20.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.130356+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:20.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.138410+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-04-26T16:27:20.251 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.138410+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-04-26T16:27:20.285 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:20.310 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.607751+0000 mgr.y (mgr.24401) 392 : cluster [DBG] pgmap v541: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.607751+0000 mgr.y (mgr.24401) 392 : cluster [DBG] pgmap v541: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.130356+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.130356+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.138410+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.138410+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.138502+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.138502+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.143026+0000 mon.c (mon.2) 563 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.143026+0000 mon.c (mon.2) 563 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.144065+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.144065+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.151349+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: audit 2024-04-26T16:27:20.151349+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.191611+0000 osd.6 (osd.6) 169 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.191611+0000 osd.6 (osd.6) 169 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.194815+0000 osd.6 (osd.6) 170 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:27:20.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:20 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.194815+0000 osd.6 (osd.6) 170 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:27:20.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:20.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:20.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:27:20.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:20.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.138502+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.138502+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.143026+0000 mon.c (mon.2) 563 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.143026+0000 mon.c (mon.2) 563 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.144065+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.144065+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.151349+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: audit 2024-04-26T16:27:20.151349+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.191611+0000 osd.6 (osd.6) 169 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.191611+0000 osd.6 (osd.6) 169 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.194815+0000 osd.6 (osd.6) 170 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.194815+0000 osd.6 (osd.6) 170 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:27:20.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.607751+0000 mgr.y (mgr.24401) 392 : cluster [DBG] pgmap v541: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.607751+0000 mgr.y (mgr.24401) 392 : cluster [DBG] pgmap v541: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 542 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.130356+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.130356+0000 mon.a (mon.0) 2428 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.138410+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.138410+0000 mon.a (mon.0) 2429 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.138502+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.138502+0000 mon.b (mon.1) 261 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.143026+0000 mon.c (mon.2) 563 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.143026+0000 mon.c (mon.2) 563 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.144065+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.144065+0000 mon.a (mon.0) 2430 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.151349+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: audit 2024-04-26T16:27:20.151349+0000 mon.a (mon.0) 2431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:27:20.554 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.191611+0000 osd.6 (osd.6) 169 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:27:20.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.191611+0000 osd.6 (osd.6) 169 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:27:20.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.194815+0000 osd.6 (osd.6) 170 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:27:20.555 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:20 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.194815+0000 osd.6 (osd.6) 170 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:27:20.573 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:20.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:20.703 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:20.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:20.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:20.958 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:20.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:21.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:21.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:21.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:21.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:27:21.208 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:21.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:21.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:21.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:21.447 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.963144+0000 osd.1 (osd.1) 203 : cluster [DBG] 2.d scrub starts 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.963144+0000 osd.1 (osd.1) 203 : cluster [DBG] 2.d scrub starts 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.964806+0000 osd.1 (osd.1) 204 : cluster [DBG] 2.d scrub ok 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: cluster 2024-04-26T16:27:19.964806+0000 osd.1 (osd.1) 204 : cluster [DBG] 2.d scrub ok 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.132070+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.132070+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.132150+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.132150+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: cluster 2024-04-26T16:27:21.136397+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: cluster 2024-04-26T16:27:21.136397+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.138240+0000 mon.c (mon.2) 564 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.138240+0000 mon.c (mon.2) 564 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.448 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.142411+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.449 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:21 smithi190 bash[23868]: audit 2024-04-26T16:27:21.142411+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.963144+0000 osd.1 (osd.1) 203 : cluster [DBG] 2.d scrub starts 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.963144+0000 osd.1 (osd.1) 203 : cluster [DBG] 2.d scrub starts 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.964806+0000 osd.1 (osd.1) 204 : cluster [DBG] 2.d scrub ok 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: cluster 2024-04-26T16:27:19.964806+0000 osd.1 (osd.1) 204 : cluster [DBG] 2.d scrub ok 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.132070+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.132070+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.132150+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.132150+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:27:21.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: cluster 2024-04-26T16:27:21.136397+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: cluster 2024-04-26T16:27:21.136397+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.138240+0000 mon.c (mon.2) 564 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.138240+0000 mon.c (mon.2) 564 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.142411+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[21463]: audit 2024-04-26T16:27:21.142411+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.963144+0000 osd.1 (osd.1) 203 : cluster [DBG] 2.d scrub starts 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.963144+0000 osd.1 (osd.1) 203 : cluster [DBG] 2.d scrub starts 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.964806+0000 osd.1 (osd.1) 204 : cluster [DBG] 2.d scrub ok 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: cluster 2024-04-26T16:27:19.964806+0000 osd.1 (osd.1) 204 : cluster [DBG] 2.d scrub ok 2024-04-26T16:27:21.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.132070+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.132070+0000 mon.a (mon.0) 2432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.132150+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.132150+0000 mon.a (mon.0) 2433 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: cluster 2024-04-26T16:27:21.136397+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: cluster 2024-04-26T16:27:21.136397+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.138240+0000 mon.c (mon.2) 564 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.138240+0000 mon.c (mon.2) 564 : audit [INF] from='client.? 172.21.15.50:0/2071764894' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.142411+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:21 smithi050 bash[29234]: audit 2024-04-26T16:27:21.142411+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]: dispatch 2024-04-26T16:27:21.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:21.590 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:21.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:21.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:21.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:21.849 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:21.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:21.981 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:21.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:22.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:22.113 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:22.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:22.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.805633+0000 osd.4 (osd.4) 195 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:27:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.805633+0000 osd.4 (osd.4) 195 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:27:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.808704+0000 osd.4 (osd.4) 196 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:27:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:20.808704+0000 osd.4 (osd.4) 196 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:27:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:21.609987+0000 mgr.y (mgr.24401) 393 : cluster [DBG] pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:22.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:21.609987+0000 mgr.y (mgr.24401) 393 : cluster [DBG] pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.134444+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.134444+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:22.139243+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: cluster 2024-04-26T16:27:22.139243+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.160196+0000 mon.c (mon.2) 565 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.160196+0000 mon.c (mon.2) 565 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.160511+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.160511+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.161374+0000 mon.c (mon.2) 566 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.161374+0000 mon.c (mon.2) 566 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.161738+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.161738+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.162282+0000 mon.c (mon.2) 567 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.162282+0000 mon.c (mon.2) 567 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.162579+0000 mon.a (mon.0) 2440 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:22 smithi190 bash[23868]: audit 2024-04-26T16:27:22.162579+0000 mon.a (mon.0) 2440 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:22.373 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.805633+0000 osd.4 (osd.4) 195 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.805633+0000 osd.4 (osd.4) 195 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.808704+0000 osd.4 (osd.4) 196 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:20.808704+0000 osd.4 (osd.4) 196 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:21.609987+0000 mgr.y (mgr.24401) 393 : cluster [DBG] pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:21.609987+0000 mgr.y (mgr.24401) 393 : cluster [DBG] pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.134444+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.134444+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:22.139243+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: cluster 2024-04-26T16:27:22.139243+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-04-26T16:27:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.160196+0000 mon.c (mon.2) 565 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.160196+0000 mon.c (mon.2) 565 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.160511+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.160511+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.161374+0000 mon.c (mon.2) 566 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.161374+0000 mon.c (mon.2) 566 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.161738+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.161738+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.162282+0000 mon.c (mon.2) 567 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.162282+0000 mon.c (mon.2) 567 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.162579+0000 mon.a (mon.0) 2440 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[21463]: audit 2024-04-26T16:27:22.162579+0000 mon.a (mon.0) 2440 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.805633+0000 osd.4 (osd.4) 195 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.805633+0000 osd.4 (osd.4) 195 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.808704+0000 osd.4 (osd.4) 196 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:20.808704+0000 osd.4 (osd.4) 196 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:21.609987+0000 mgr.y (mgr.24401) 393 : cluster [DBG] pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:21.609987+0000 mgr.y (mgr.24401) 393 : cluster [DBG] pgmap v544: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.134444+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.134444+0000 mon.a (mon.0) 2436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi050-73201-58"}]': finished 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:22.139243+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: cluster 2024-04-26T16:27:22.139243+0000 mon.a (mon.0) 2437 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2024-04-26T16:27:22.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.160196+0000 mon.c (mon.2) 565 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.160196+0000 mon.c (mon.2) 565 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.160511+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.160511+0000 mon.a (mon.0) 2438 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.161374+0000 mon.c (mon.2) 566 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.161374+0000 mon.c (mon.2) 566 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.161738+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.161738+0000 mon.a (mon.0) 2439 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.162282+0000 mon.c (mon.2) 567 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.162282+0000 mon.c (mon.2) 567 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.162579+0000 mon.a (mon.0) 2440 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:22 smithi050 bash[29234]: audit 2024-04-26T16:27:22.162579+0000 mon.a (mon.0) 2440 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:22.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:22.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:22.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:22.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:22.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:22.762 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:22.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:22.887 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:22.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:22.969 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:27:22.969 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:27:22.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:23.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:23.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:23.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:23.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:23.282 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:23.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:23.411 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:23.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:23.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.661 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:23.680 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:27:23.680 INFO:teuthology.orchestra.run.smithi050.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-26T16:27:23.680 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:27:23.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-26T16:27:23.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:23.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:23.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:24.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:24.041 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:24.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:24.175 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:24.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: audit 2024-04-26T16:27:23.136453+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: audit 2024-04-26T16:27:23.136453+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.140208+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.140208+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: audit 2024-04-26T16:27:23.143433+0000 mon.c (mon.2) 568 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: audit 2024-04-26T16:27:23.143433+0000 mon.c (mon.2) 568 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: audit 2024-04-26T16:27:23.144018+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.290 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: audit 2024-04-26T16:27:23.144018+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.610475+0000 mgr.y (mgr.24401) 394 : cluster [DBG] pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.610475+0000 mgr.y (mgr.24401) 394 : cluster [DBG] pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: audit 2024-04-26T16:27:23.136453+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: audit 2024-04-26T16:27:23.136453+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.140208+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.140208+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: audit 2024-04-26T16:27:23.143433+0000 mon.c (mon.2) 568 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: audit 2024-04-26T16:27:23.143433+0000 mon.c (mon.2) 568 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: audit 2024-04-26T16:27:23.144018+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: audit 2024-04-26T16:27:23.144018+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.291 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.610475+0000 mgr.y (mgr.24401) 394 : cluster [DBG] pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:24.292 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.610475+0000 mgr.y (mgr.24401) 394 : cluster [DBG] pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:24.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:24.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:24.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: audit 2024-04-26T16:27:23.136453+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:24.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: audit 2024-04-26T16:27:23.136453+0000 mon.a (mon.0) 2441 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:24.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.140208+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.140208+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: audit 2024-04-26T16:27:23.143433+0000 mon.c (mon.2) 568 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: audit 2024-04-26T16:27:23.143433+0000 mon.c (mon.2) 568 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: audit 2024-04-26T16:27:23.144018+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: audit 2024-04-26T16:27:23.144018+0000 mon.a (mon.0) 2443 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.610475+0000 mgr.y (mgr.24401) 394 : cluster [DBG] pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:24.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:24 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.610475+0000 mgr.y (mgr.24401) 394 : cluster [DBG] pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:27:24.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:24.437 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:24.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:24.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:24.569 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:24.592 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:27:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:27:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:27:24.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:24.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:24.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:24.831 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:24.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:24.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:24.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:25.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:25.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:25.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:25.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.853653+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:27:25.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.853653+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:27:25.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.855096+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:27:25.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[21463]: cluster 2024-04-26T16:27:23.855096+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:27:25.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[21463]: cluster 2024-04-26T16:27:24.179729+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-04-26T16:27:25.207 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[21463]: cluster 2024-04-26T16:27:24.179729+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-04-26T16:27:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.853653+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:27:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.853653+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:27:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.855096+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:27:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[29234]: cluster 2024-04-26T16:27:23.855096+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:27:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[29234]: cluster 2024-04-26T16:27:24.179729+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-04-26T16:27:25.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:25 smithi050 bash[29234]: cluster 2024-04-26T16:27:24.179729+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-04-26T16:27:25.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:25.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:25.336 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:25 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.853653+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:27:25.336 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:25 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.853653+0000 osd.4 (osd.4) 197 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:27:25.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:25 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.855096+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:27:25.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:25 smithi190 bash[23868]: cluster 2024-04-26T16:27:23.855096+0000 osd.4 (osd.4) 198 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:27:25.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:25 smithi190 bash[23868]: cluster 2024-04-26T16:27:24.179729+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-04-26T16:27:25.337 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:25 smithi190 bash[23868]: cluster 2024-04-26T16:27:24.179729+0000 mon.a (mon.0) 2444 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2024-04-26T16:27:25.353 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:25.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:25.484 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:25.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:25.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:25.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:25.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:25.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:25.741 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:25.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:25.873 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:25.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:25.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:25.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:26.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:26.071 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:26.072 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:26.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:26.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:26.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:26.269 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:26.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:26.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:24.386655+0000 osd.2 (osd.2) 175 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:24.386655+0000 osd.2 (osd.2) 175 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:24.389566+0000 osd.2 (osd.2) 176 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:24.389566+0000 osd.2 (osd.2) 176 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.063280+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.063280+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.065512+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.065512+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: audit 2024-04-26T16:27:25.163300+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: audit 2024-04-26T16:27:25.163300+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.171896+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-04-26T16:27:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.171896+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.236017+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.236017+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.238284+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.238284+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.611046+0000 mgr.y (mgr.24401) 395 : cluster [DBG] pgmap v550: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.611046+0000 mgr.y (mgr.24401) 395 : cluster [DBG] pgmap v550: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.907580+0000 osd.7 (osd.7) 233 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.907580+0000 osd.7 (osd.7) 233 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.909457+0000 osd.7 (osd.7) 234 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.909457+0000 osd.7 (osd.7) 234 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.127080+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.127080+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-04-26T16:27:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:24.386655+0000 osd.2 (osd.2) 175 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:24.386655+0000 osd.2 (osd.2) 175 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:24.389566+0000 osd.2 (osd.2) 176 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:24.389566+0000 osd.2 (osd.2) 176 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.063280+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.063280+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.065512+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.065512+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: audit 2024-04-26T16:27:25.163300+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: audit 2024-04-26T16:27:25.163300+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.171896+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.171896+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.236017+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.236017+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:27:26.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.238284+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.238284+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.611046+0000 mgr.y (mgr.24401) 395 : cluster [DBG] pgmap v550: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.611046+0000 mgr.y (mgr.24401) 395 : cluster [DBG] pgmap v550: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.907580+0000 osd.7 (osd.7) 233 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.907580+0000 osd.7 (osd.7) 233 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.909457+0000 osd.7 (osd.7) 234 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.909457+0000 osd.7 (osd.7) 234 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.127080+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-04-26T16:27:26.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:26 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.127080+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-04-26T16:27:26.404 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:26.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:26.535 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:26.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:26.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:24.386655+0000 osd.2 (osd.2) 175 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:24.386655+0000 osd.2 (osd.2) 175 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:24.389566+0000 osd.2 (osd.2) 176 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:24.389566+0000 osd.2 (osd.2) 176 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.063280+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.063280+0000 osd.3 (osd.3) 235 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.065512+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.065512+0000 osd.3 (osd.3) 236 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: audit 2024-04-26T16:27:25.163300+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: audit 2024-04-26T16:27:25.163300+0000 mon.a (mon.0) 2445 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi050-73201-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.171896+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.171896+0000 mon.a (mon.0) 2446 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2024-04-26T16:27:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.236017+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.236017+0000 osd.0 (osd.0) 235 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.238284+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.238284+0000 osd.0 (osd.0) 236 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.611046+0000 mgr.y (mgr.24401) 395 : cluster [DBG] pgmap v550: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.611046+0000 mgr.y (mgr.24401) 395 : cluster [DBG] pgmap v550: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.907580+0000 osd.7 (osd.7) 233 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.907580+0000 osd.7 (osd.7) 233 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.909457+0000 osd.7 (osd.7) 234 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.909457+0000 osd.7 (osd.7) 234 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.127080+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-04-26T16:27:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.127080+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2024-04-26T16:27:26.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:26.665 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:26.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:26.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:26.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:26.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:26.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:27:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:27:26.928 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:26.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:27.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:27.060 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:27.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:27.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:27.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:27.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:27.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:27.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.395688+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.b scrub starts 2024-04-26T16:27:27.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.395688+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.b scrub starts 2024-04-26T16:27:27.441 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.397790+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.b scrub ok 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:25.397790+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.b scrub ok 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.163187+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.163187+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: audit 2024-04-26T16:27:26.610830+0000 mon.c (mon.2) 569 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: audit 2024-04-26T16:27:26.610830+0000 mon.c (mon.2) 569 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.868171+0000 osd.7 (osd.7) 235 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.868171+0000 osd.7 (osd.7) 235 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.872956+0000 osd.7 (osd.7) 236 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.872956+0000 osd.7 (osd.7) 236 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.133570+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.133570+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: audit 2024-04-26T16:27:27.136299+0000 mon.c (mon.2) 570 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.442 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: audit 2024-04-26T16:27:27.136299+0000 mon.c (mon.2) 570 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: audit 2024-04-26T16:27:27.138116+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:27 smithi190 bash[23868]: audit 2024-04-26T16:27:27.138116+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.454 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:27.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:27.585 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:27.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.395688+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.b scrub starts 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.395688+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.b scrub starts 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.397790+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.b scrub ok 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:25.397790+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.b scrub ok 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.163187+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.163187+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: audit 2024-04-26T16:27:26.610830+0000 mon.c (mon.2) 569 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: audit 2024-04-26T16:27:26.610830+0000 mon.c (mon.2) 569 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.868171+0000 osd.7 (osd.7) 235 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.868171+0000 osd.7 (osd.7) 235 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.872956+0000 osd.7 (osd.7) 236 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:27:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.872956+0000 osd.7 (osd.7) 236 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.133570+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.133570+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: audit 2024-04-26T16:27:27.136299+0000 mon.c (mon.2) 570 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: audit 2024-04-26T16:27:27.136299+0000 mon.c (mon.2) 570 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: audit 2024-04-26T16:27:27.138116+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[29234]: audit 2024-04-26T16:27:27.138116+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.395688+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.b scrub starts 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.395688+0000 osd.2 (osd.2) 177 : cluster [DBG] 5.b scrub starts 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.397790+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.b scrub ok 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:25.397790+0000 osd.2 (osd.2) 178 : cluster [DBG] 5.b scrub ok 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.163187+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.163187+0000 mon.a (mon.0) 2448 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: audit 2024-04-26T16:27:26.610830+0000 mon.c (mon.2) 569 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: audit 2024-04-26T16:27:26.610830+0000 mon.c (mon.2) 569 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.868171+0000 osd.7 (osd.7) 235 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.868171+0000 osd.7 (osd.7) 235 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.872956+0000 osd.7 (osd.7) 236 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.872956+0000 osd.7 (osd.7) 236 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.133570+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.133570+0000 mon.a (mon.0) 2449 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: audit 2024-04-26T16:27:27.136299+0000 mon.c (mon.2) 570 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: audit 2024-04-26T16:27:27.136299+0000 mon.c (mon.2) 570 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: audit 2024-04-26T16:27:27.138116+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:27 smithi050 bash[21463]: audit 2024-04-26T16:27:27.138116+0000 mon.a (mon.0) 2450 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:27.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:27.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:27.851 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:27.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:27.982 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:27.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:28.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:28.111 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:28.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:28.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:26.818479+0000 mgr.y (mgr.24401) 396 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:28.227 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:26.818479+0000 mgr.y (mgr.24401) 396 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.868231+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.868231+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.870768+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:26.870768+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.068956+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.068956+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.071318+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.071318+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.214672+0000 osd.6 (osd.6) 171 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.214672+0000 osd.6 (osd.6) 171 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.216973+0000 osd.6 (osd.6) 172 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.216973+0000 osd.6 (osd.6) 172 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:27:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.611844+0000 mgr.y (mgr.24401) 397 : cluster [DBG] pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:27.611844+0000 mgr.y (mgr.24401) 397 : cluster [DBG] pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:28.127706+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:28.127706+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.135269+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.135269+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:28.137796+0000 mon.c (mon.2) 571 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:28.137796+0000 mon.c (mon.2) 571 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:28.139946+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[21463]: audit 2024-04-26T16:27:28.139946+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.229 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:26.818479+0000 mgr.y (mgr.24401) 396 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:26.818479+0000 mgr.y (mgr.24401) 396 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.868231+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.868231+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.870768+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:26.870768+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.068956+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.068956+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.071318+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.071318+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.214672+0000 osd.6 (osd.6) 171 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.214672+0000 osd.6 (osd.6) 171 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.216973+0000 osd.6 (osd.6) 172 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.216973+0000 osd.6 (osd.6) 172 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.611844+0000 mgr.y (mgr.24401) 397 : cluster [DBG] pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:27.611844+0000 mgr.y (mgr.24401) 397 : cluster [DBG] pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:28.127706+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:28.127706+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.135269+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.135269+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:28.137796+0000 mon.c (mon.2) 571 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:28.137796+0000 mon.c (mon.2) 571 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:28.139946+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.231 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:28 smithi050 bash[29234]: audit 2024-04-26T16:27:28.139946+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:26.818479+0000 mgr.y (mgr.24401) 396 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:26.818479+0000 mgr.y (mgr.24401) 396 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.868231+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.868231+0000 osd.4 (osd.4) 199 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.870768+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:26.870768+0000 osd.4 (osd.4) 200 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.068956+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.068956+0000 osd.3 (osd.3) 237 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:27:28.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.071318+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.071318+0000 osd.3 (osd.3) 238 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.214672+0000 osd.6 (osd.6) 171 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.214672+0000 osd.6 (osd.6) 171 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.216973+0000 osd.6 (osd.6) 172 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.216973+0000 osd.6 (osd.6) 172 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.611844+0000 mgr.y (mgr.24401) 397 : cluster [DBG] pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:27.611844+0000 mgr.y (mgr.24401) 397 : cluster [DBG] pgmap v553: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:28.127706+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:28.127706+0000 mon.a (mon.0) 2451 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.135269+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.135269+0000 mon.a (mon.0) 2452 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:28.137796+0000 mon.c (mon.2) 571 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:28.137796+0000 mon.c (mon.2) 571 : audit [INF] from='client.? 172.21.15.50:0/3919809189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:28.139946+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:28 smithi190 bash[23868]: audit 2024-04-26T16:27:28.139946+0000 mon.a (mon.0) 2453 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]: dispatch 2024-04-26T16:27:28.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:28.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:28.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:28.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:28.565 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:27:28.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:28.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:28.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:28.765 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:28.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:28.894 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:28.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:28.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:29.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:29.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:29.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:29.159 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:29.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:29.264 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.090448+0000 osd.1 (osd.1) 205 : cluster [DBG] 109.c scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.090448+0000 osd.1 (osd.1) 205 : cluster [DBG] 109.c scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.093574+0000 osd.1 (osd.1) 206 : cluster [DBG] 109.c scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.093574+0000 osd.1 (osd.1) 206 : cluster [DBG] 109.c scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.116746+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.116746+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.119008+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.119008+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.315707+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.d scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.315707+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.d scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.318756+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.318756+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.908197+0000 osd.7 (osd.7) 237 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.908197+0000 osd.7 (osd.7) 237 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.909855+0000 osd.7 (osd.7) 238 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:27:29.265 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:28.909855+0000 osd.7 (osd.7) 238 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: audit 2024-04-26T16:27:29.129927+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: audit 2024-04-26T16:27:29.129927+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.134395+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.134395+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.090448+0000 osd.1 (osd.1) 205 : cluster [DBG] 109.c scrub starts 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.090448+0000 osd.1 (osd.1) 205 : cluster [DBG] 109.c scrub starts 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.093574+0000 osd.1 (osd.1) 206 : cluster [DBG] 109.c scrub ok 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.093574+0000 osd.1 (osd.1) 206 : cluster [DBG] 109.c scrub ok 2024-04-26T16:27:29.266 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.116746+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.116746+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.119008+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.119008+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.315707+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.d scrub starts 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.315707+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.d scrub starts 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.318756+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d scrub ok 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.318756+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d scrub ok 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.908197+0000 osd.7 (osd.7) 237 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.908197+0000 osd.7 (osd.7) 237 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.909855+0000 osd.7 (osd.7) 238 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:28.909855+0000 osd.7 (osd.7) 238 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: audit 2024-04-26T16:27:29.129927+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: audit 2024-04-26T16:27:29.129927+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:29.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.134395+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-04-26T16:27:29.268 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:29 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.134395+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-04-26T16:27:29.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:29.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.090448+0000 osd.1 (osd.1) 205 : cluster [DBG] 109.c scrub starts 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.090448+0000 osd.1 (osd.1) 205 : cluster [DBG] 109.c scrub starts 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.093574+0000 osd.1 (osd.1) 206 : cluster [DBG] 109.c scrub ok 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.093574+0000 osd.1 (osd.1) 206 : cluster [DBG] 109.c scrub ok 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.116746+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.116746+0000 osd.3 (osd.3) 239 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.119008+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.119008+0000 osd.3 (osd.3) 240 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:27:29.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.315707+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.d scrub starts 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.315707+0000 osd.2 (osd.2) 179 : cluster [DBG] 5.d scrub starts 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.318756+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d scrub ok 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.318756+0000 osd.2 (osd.2) 180 : cluster [DBG] 5.d scrub ok 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.908197+0000 osd.7 (osd.7) 237 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.908197+0000 osd.7 (osd.7) 237 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.909855+0000 osd.7 (osd.7) 238 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:28.909855+0000 osd.7 (osd.7) 238 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: audit 2024-04-26T16:27:29.129927+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: audit 2024-04-26T16:27:29.129927+0000 mon.a (mon.0) 2454 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi050-73201-59"}]': finished 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.134395+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-04-26T16:27:29.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:29 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.134395+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2024-04-26T16:27:29.418 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:29.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:29.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:29.594 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:27:29.594 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:27:29.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:27:29.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:29.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:29.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:29.798 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:29.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:29.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:29.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:30.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:30.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:30.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:30.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.147957+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.147957+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.149478+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.149478+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.216981+0000 osd.6 (osd.6) 173 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.216981+0000 osd.6 (osd.6) 173 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:27:30.314 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.219018+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.219018+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.240323+0000 osd.0 (osd.0) 237 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.240323+0000 osd.0 (osd.0) 237 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.241752+0000 osd.0 (osd.0) 238 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.241752+0000 osd.0 (osd.0) 238 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.357537+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.f scrub starts 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.357537+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.f scrub starts 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.358969+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.f scrub ok 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.358969+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.f scrub ok 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.612848+0000 mgr.y (mgr.24401) 398 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.612848+0000 mgr.y (mgr.24401) 398 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.138707+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.138707+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-04-26T16:27:30.315 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: audit 2024-04-26T16:27:30.144719+0000 mon.c (mon.2) 572 : audit [INF] from='client.? 172.21.15.50:0/1789150730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: audit 2024-04-26T16:27:30.144719+0000 mon.c (mon.2) 572 : audit [INF] from='client.? 172.21.15.50:0/1789150730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: audit 2024-04-26T16:27:30.146439+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:30 smithi190 bash[23868]: audit 2024-04-26T16:27:30.146439+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.327 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:30.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:30.458 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:30.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.147957+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.147957+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.149478+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.149478+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.216981+0000 osd.6 (osd.6) 173 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.216981+0000 osd.6 (osd.6) 173 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.219018+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:27:30.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.219018+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.240323+0000 osd.0 (osd.0) 237 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.240323+0000 osd.0 (osd.0) 237 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.241752+0000 osd.0 (osd.0) 238 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.241752+0000 osd.0 (osd.0) 238 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.357537+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.f scrub starts 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.357537+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.f scrub starts 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.358969+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.f scrub ok 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.358969+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.f scrub ok 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.612848+0000 mgr.y (mgr.24401) 398 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.612848+0000 mgr.y (mgr.24401) 398 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.138707+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.138707+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-04-26T16:27:30.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: audit 2024-04-26T16:27:30.144719+0000 mon.c (mon.2) 572 : audit [INF] from='client.? 172.21.15.50:0/1789150730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: audit 2024-04-26T16:27:30.144719+0000 mon.c (mon.2) 572 : audit [INF] from='client.? 172.21.15.50:0/1789150730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: audit 2024-04-26T16:27:30.146439+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.147957+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.147957+0000 osd.1 (osd.1) 207 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.149478+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.149478+0000 osd.1 (osd.1) 208 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.216981+0000 osd.6 (osd.6) 173 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.216981+0000 osd.6 (osd.6) 173 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:27:30.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.219018+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.219018+0000 osd.6 (osd.6) 174 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.240323+0000 osd.0 (osd.0) 237 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.240323+0000 osd.0 (osd.0) 237 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.241752+0000 osd.0 (osd.0) 238 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.241752+0000 osd.0 (osd.0) 238 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.357537+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.f scrub starts 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.357537+0000 osd.2 (osd.2) 181 : cluster [DBG] 6.f scrub starts 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.358969+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.f scrub ok 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.358969+0000 osd.2 (osd.2) 182 : cluster [DBG] 6.f scrub ok 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.612848+0000 mgr.y (mgr.24401) 398 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.612848+0000 mgr.y (mgr.24401) 398 : cluster [DBG] pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.138707+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.138707+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2024-04-26T16:27:30.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: audit 2024-04-26T16:27:30.144719+0000 mon.c (mon.2) 572 : audit [INF] from='client.? 172.21.15.50:0/1789150730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: audit 2024-04-26T16:27:30.144719+0000 mon.c (mon.2) 572 : audit [INF] from='client.? 172.21.15.50:0/1789150730' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: audit 2024-04-26T16:27:30.146439+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[29234]: audit 2024-04-26T16:27:30.146439+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:30.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:30.719 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:30.823 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:30.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:30 smithi050 bash[21463]: audit 2024-04-26T16:27:30.146439+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:30.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:30.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:30.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:30.979 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:30.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:31.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:31.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:31.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:31.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.424505+0000 osd.4 (osd.4) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.424505+0000 osd.4 (osd.4) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.424752+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.424752+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.959327+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.959327+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:27:31.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.962871+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:29.962871+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.193070+0000 osd.6 (osd.6) 175 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.193070+0000 osd.6 (osd.6) 175 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.194734+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.194734+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.269175+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.269175+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.270786+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.270786+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: audit 2024-04-26T16:27:31.135828+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: audit 2024-04-26T16:27:31.135828+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.141403+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-04-26T16:27:31.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:31 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.141403+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-04-26T16:27:31.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:31.366 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:31.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.424505+0000 osd.4 (osd.4) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:31.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.424505+0000 osd.4 (osd.4) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:31.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.424752+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:31.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.424752+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.959327+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.959327+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.962871+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:29.962871+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.193070+0000 osd.6 (osd.6) 175 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.193070+0000 osd.6 (osd.6) 175 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.194734+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.194734+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.269175+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.269175+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.270786+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.270786+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: audit 2024-04-26T16:27:31.135828+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: audit 2024-04-26T16:27:31.135828+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:31.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.141403+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.141403+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.424505+0000 osd.4 (osd.4) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.424505+0000 osd.4 (osd.4) 201 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.424752+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.424752+0000 osd.4 (osd.4) 202 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.959327+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.959327+0000 osd.7 (osd.7) 239 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.962871+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:29.962871+0000 osd.7 (osd.7) 240 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.193070+0000 osd.6 (osd.6) 175 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.193070+0000 osd.6 (osd.6) 175 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:27:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.194734+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.194734+0000 osd.6 (osd.6) 176 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.269175+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.269175+0000 osd.0 (osd.0) 239 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.270786+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.270786+0000 osd.0 (osd.0) 240 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: audit 2024-04-26T16:27:31.135828+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: audit 2024-04-26T16:27:31.135828+0000 mon.a (mon.0) 2458 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi050-73201-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.141403+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-04-26T16:27:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:31 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.141403+0000 mon.a (mon.0) 2459 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2024-04-26T16:27:31.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:31.491 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:31.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:31.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:31.619 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:31.633 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:27:31.633 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:27:31.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:31.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:31.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:31.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:31.872 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:31.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:31.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:32.002 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:32.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:32.134 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:32.149 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7462ddf6:::.RoundTripAppendPP (2960 ms) 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3003 ms) 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3022 ms) 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3022 ms) 2024-04-26T16:27:32.150 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: setting pool EIO 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: max_success 98, min_failed 99 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3963 ms) 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3019 ms) 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (114145 ms total) 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-04-26T16:27:32.151 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2990 ms) 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9016 ms) 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14992 ms) 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3001 ms) 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (29999 ms total) 2024-04-26T16:27:32.152 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3002 ms) 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3002 ms total) 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14014 ms) 2024-04-26T16:27:32.153 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6991 ms) 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7025 ms) 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (6996 ms) 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-04-26T16:27:32.154 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3002 ms) 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7016 ms) 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6975 ms) 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7070 ms) 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6945 ms) 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7012 ms) 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-04-26T16:27:32.155 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7091 ms) 2024-04-26T16:27:32.156 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-04-26T16:27:32.156 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6906 ms) 2024-04-26T16:27:32.156 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-04-26T16:27:32.156 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6995 ms) 2024-04-26T16:27:32.156 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-04-26T16:27:32.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:32.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:32.394 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.896553+0000 osd.4 (osd.4) 203 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.896553+0000 osd.4 (osd.4) 203 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.898187+0000 osd.4 (osd.4) 204 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:30.898187+0000 osd.4 (osd.4) 204 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.144059+0000 osd.3 (osd.3) 241 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.144059+0000 osd.3 (osd.3) 241 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.162666+0000 osd.3 (osd.3) 242 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:27:32.464 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.162666+0000 osd.3 (osd.3) 242 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.365121+0000 osd.2 (osd.2) 183 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.365121+0000 osd.2 (osd.2) 183 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.367335+0000 osd.2 (osd.2) 184 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.367335+0000 osd.2 (osd.2) 184 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.614307+0000 mgr.y (mgr.24401) 399 : cluster [DBG] pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.614307+0000 mgr.y (mgr.24401) 399 : cluster [DBG] pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.147162+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.147162+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.189897+0000 mon.c (mon.2) 573 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.189897+0000 mon.c (mon.2) 573 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.190311+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.190311+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.191126+0000 mon.c (mon.2) 574 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.191126+0000 mon.c (mon.2) 574 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.191471+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.191471+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.191990+0000 mon.c (mon.2) 575 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.191990+0000 mon.c (mon.2) 575 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.192323+0000 mon.a (mon.0) 2463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:32 smithi190 bash[23868]: audit 2024-04-26T16:27:32.192323+0000 mon.a (mon.0) 2463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-26T16:27:32.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:32.538 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.896553+0000 osd.4 (osd.4) 203 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.896553+0000 osd.4 (osd.4) 203 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.898187+0000 osd.4 (osd.4) 204 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:30.898187+0000 osd.4 (osd.4) 204 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.144059+0000 osd.3 (osd.3) 241 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.144059+0000 osd.3 (osd.3) 241 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.162666+0000 osd.3 (osd.3) 242 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.162666+0000 osd.3 (osd.3) 242 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:27:32.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.365121+0000 osd.2 (osd.2) 183 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.365121+0000 osd.2 (osd.2) 183 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.367335+0000 osd.2 (osd.2) 184 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.367335+0000 osd.2 (osd.2) 184 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.614307+0000 mgr.y (mgr.24401) 399 : cluster [DBG] pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.614307+0000 mgr.y (mgr.24401) 399 : cluster [DBG] pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.147162+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.147162+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.189897+0000 mon.c (mon.2) 573 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.189897+0000 mon.c (mon.2) 573 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.190311+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.190311+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.191126+0000 mon.c (mon.2) 574 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.191126+0000 mon.c (mon.2) 574 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.896553+0000 osd.4 (osd.4) 203 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.896553+0000 osd.4 (osd.4) 203 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.898187+0000 osd.4 (osd.4) 204 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:30.898187+0000 osd.4 (osd.4) 204 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.144059+0000 osd.3 (osd.3) 241 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.144059+0000 osd.3 (osd.3) 241 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.162666+0000 osd.3 (osd.3) 242 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:27:32.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.162666+0000 osd.3 (osd.3) 242 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.365121+0000 osd.2 (osd.2) 183 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.365121+0000 osd.2 (osd.2) 183 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.367335+0000 osd.2 (osd.2) 184 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.367335+0000 osd.2 (osd.2) 184 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.614307+0000 mgr.y (mgr.24401) 399 : cluster [DBG] pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.614307+0000 mgr.y (mgr.24401) 399 : cluster [DBG] pgmap v559: 260 pgs: 260 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.147162+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.147162+0000 mon.a (mon.0) 2460 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.189897+0000 mon.c (mon.2) 573 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.189897+0000 mon.c (mon.2) 573 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.190311+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.190311+0000 mon.a (mon.0) 2461 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.191126+0000 mon.c (mon.2) 574 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.191126+0000 mon.c (mon.2) 574 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.191471+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.191471+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.191990+0000 mon.c (mon.2) 575 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.191990+0000 mon.c (mon.2) 575 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.192323+0000 mon.a (mon.0) 2463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[21463]: audit 2024-04-26T16:27:32.192323+0000 mon.a (mon.0) 2463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.191471+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.191471+0000 mon.a (mon.0) 2462 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:32.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.191990+0000 mon.c (mon.2) 575 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.191990+0000 mon.c (mon.2) 575 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.192323+0000 mon.a (mon.0) 2463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:32 smithi050 bash[29234]: audit 2024-04-26T16:27:32.192323+0000 mon.a (mon.0) 2463 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:32.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:32.660 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:32.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:32.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:32.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:32.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:32.923 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:32.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:33.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:33.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:33.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:33.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:33.181 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:33.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:33.309 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:33.315 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.946805+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.946805+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.949811+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:31.949811+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.275365+0000 osd.0 (osd.0) 241 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.275365+0000 osd.0 (osd.0) 241 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.277165+0000 osd.0 (osd.0) 242 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.277165+0000 osd.0 (osd.0) 242 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: audit 2024-04-26T16:27:33.141245+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: audit 2024-04-26T16:27:33.141245+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.145391+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.145391+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: audit 2024-04-26T16:27:33.149970+0000 mon.c (mon.2) 576 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: audit 2024-04-26T16:27:33.149970+0000 mon.c (mon.2) 576 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: audit 2024-04-26T16:27:33.150606+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[21463]: audit 2024-04-26T16:27:33.150606+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.946805+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.946805+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.949811+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:31.949811+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.275365+0000 osd.0 (osd.0) 241 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.275365+0000 osd.0 (osd.0) 241 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.277165+0000 osd.0 (osd.0) 242 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.277165+0000 osd.0 (osd.0) 242 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: audit 2024-04-26T16:27:33.141245+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: audit 2024-04-26T16:27:33.141245+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:33.317 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.145391+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-04-26T16:27:33.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.145391+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-04-26T16:27:33.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: audit 2024-04-26T16:27:33.149970+0000 mon.c (mon.2) 576 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: audit 2024-04-26T16:27:33.149970+0000 mon.c (mon.2) 576 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: audit 2024-04-26T16:27:33.150606+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:33 smithi050 bash[29234]: audit 2024-04-26T16:27:33.150606+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- 172.21.15.50:0/3580781562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe3181094f0 msgr2=0x7fe3181098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- 172.21.15.50:0/3580781562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe3181094f0 0x7fe3181098d0 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7fe300009990 tx=0x7fe300009ca0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- 172.21.15.50:0/3580781562 shutdown_connections 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- 172.21.15.50:0/3580781562 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe3181102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- 172.21.15.50:0/3580781562 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe3181094f0 0x7fe3181098d0 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- 172.21.15.50:0/3580781562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe318103470 0x7fe3181038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- 172.21.15.50:0/3580781562 >> 172.21.15.50:0/3580781562 conn(0x7fe31807a5d0 msgr2=0x7fe31807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- 172.21.15.50:0/3580781562 shutdown_connections 2024-04-26T16:27:33.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- 172.21.15.50:0/3580781562 wait complete. 2024-04-26T16:27:33.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 librados: starting msgr at 2024-04-26T16:27:33.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 librados: starting objecter 2024-04-26T16:27:33.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 Processor -- start 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- start start 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 librados: setting wanted keys 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 librados: calling monclient init 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe318103470 0x7fe31819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe31819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3181094f0 0x7fe31819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe318112ac0 con 0x7fe318103e30 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe318112940 con 0x7fe3181094f0 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe31dc40700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe318112c40 con 0x7fe318103470 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe31819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe3177fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe318103470 0x7fe31819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe31819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36098/0 (socket says 172.21.15.50:36098) 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 -- 172.21.15.50:0/2371747126 learned_addr learned my addr 172.21.15.50:0/2371747126 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:33.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3181094f0 0x7fe31819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe318103470 msgr2=0x7fe31819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe318103470 0x7fe31819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3181094f0 msgr2=0x7fe31819fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe3181094f0 0x7fe31819fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3181a02d0 con 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe31819b800 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c9b0 tx=0x7fe30800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe314ff9700 10 client.?.objecter ms_handle_connect 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.395+0000 7fe314ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308007b00 con 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe308007ca0 con 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3181a05c0 con 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308018400 con 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 10 client.30507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3181a7e40 con 0x7fe318103e30 2024-04-26T16:27:33.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 1 librados: init done 2024-04-26T16:27:33.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe31806a110 con 0x7fe318103e30 2024-04-26T16:27:33.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe30800f450 con 0x7fe318103e30 2024-04-26T16:27:33.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe304071f80 0x7fe304074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(385..385 src has 87..385) ==== 9722+0+0 (secure 0 0 0) 0x7fe308091b50 con 0x7fe318103e30 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 osd_map(385..385 src has 87..385) 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 3 client.30507.objecter handle_osd_map got epochs [385,385] > 0 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 3 client.30507.objecter handle_osd_map decoding full epoch 385 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe304071f80 0x7fe304074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 20 client.30507.objecter dump_active .. 0 homeless 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v385) ==== 54+0+16 (secure 0 0 0) 0x7fe308014030 con 0x7fe318103e30 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 10 client.30507.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 1 2024-04-26T16:27:33.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe318064030 0x7fe318064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe304071f80 0x7fe304074440 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7fe30000b2e0 tx=0x7fe300005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe304071f80 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 1 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 0 1 2024-04-26T16:27:33.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 4 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 10 client.30507.objecter _send_command 1 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3181ab200 con 0x7fe318064030 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe318064030 0x7fe318064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe318064030 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.399+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe318064030 0x7fe318064410 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.403+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181ab200 con 0x7fe318064030 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.403+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 1: 0 ) 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.403+0000 7fe314ff9700 10 client.30507.objecter _finish_command 1 = system:0 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.403+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 4 2024-04-26T16:27:33.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.403+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 0 1 2024-04-26T16:27:33.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:33.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 10 client.30507.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 2 2024-04-26T16:27:33.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 2 2024-04-26T16:27:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe31819c620 osd=0 3 2024-04-26T16:27:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 0 2 2024-04-26T16:27:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 4 2024-04-26T16:27:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 10 client.30507.objecter _send_command 2 2024-04-26T16:27:33.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.419+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe318063ab0 con 0x7fe318064030 2024-04-26T16:27:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe318063ab0 con 0x7fe318064030 2024-04-26T16:27:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 10 client.30507.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 4 2024-04-26T16:27:33.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 0 2 2024-04-26T16:27:33.422 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 10 client.30507.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 3 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe31802d310 0x7fe31802d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 3 2024-04-26T16:27:33.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 1 3 2024-04-26T16:27:33.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 4 2024-04-26T16:27:33.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 10 client.30507.objecter _send_command 3 2024-04-26T16:27:33.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe318105e30 con 0x7fe31802d310 2024-04-26T16:27:33.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe31802d310 0x7fe31802d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe31802d310 2024-04-26T16:27:33.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe31802d310 0x7fe31802d730 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe318105e30 con 0x7fe31802d310 2024-04-26T16:27:33.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 3: 0 ) 2024-04-26T16:27:33.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 10 client.30507.objecter _finish_command 3 = system:0 2024-04-26T16:27:33.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 4 2024-04-26T16:27:33.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.423+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 1 3 2024-04-26T16:27:33.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:33.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 10 client.30507.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 4 2024-04-26T16:27:33.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 4 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181057b0 osd=1 3 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 1 4 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 4 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 10 client.30507.objecter _send_command 4 2024-04-26T16:27:33.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.447+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe318105230 con 0x7fe31802d310 2024-04-26T16:27:33.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe318105230 con 0x7fe31802d310 2024-04-26T16:27:33.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe314ff9700 10 client.30507.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 4 2024-04-26T16:27:33.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 1 4 2024-04-26T16:27:33.452 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:27:33.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 10 client.30507.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 5 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3181ad3c0 0x7fe3181af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 5 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 2 5 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 4 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 10 client.30507.objecter _send_command 5 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe3181b2610 con 0x7fe3181ad3c0 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3181ad3c0 0x7fe3181af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe3181ad3c0 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.451+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3181ad3c0 0x7fe3181af7b0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.453 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.455+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181b2610 con 0x7fe3181ad3c0 2024-04-26T16:27:33.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.455+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 5: 0 ) 2024-04-26T16:27:33.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.455+0000 7fe314ff9700 10 client.30507.objecter _finish_command 5 = system:0 2024-04-26T16:27:33.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.455+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 4 2024-04-26T16:27:33.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.455+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 2 5 2024-04-26T16:27:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 -- 172.21.15.50:0/4114524477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 msgr2=0x7ff418108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/4114524477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff418108760 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0098f0 tx=0x7ff40c009c00 comp rx=0 tx=0).stop 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 -- 172.21.15.50:0/4114524477 shutdown_connections 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/4114524477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418102cd0 0x7ff41810f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/4114524477 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff418108760 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/4114524477 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff418102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 -- 172.21.15.50:0/4114524477 >> 172.21.15.50:0/4114524477 conn(0x7ff418079290 msgr2=0x7ff4180796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 -- 172.21.15.50:0/4114524477 shutdown_connections 2024-04-26T16:27:33.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.463+0000 7ff41ed1f700 1 -- 172.21.15.50:0/4114524477 wait complete. 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 librados: starting msgr at 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 librados: starting objecter 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 Processor -- start 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- start start 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 librados: setting wanted keys 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 librados: calling monclient init 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff4181990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418102cd0 0x7ff418199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff41819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff418111980 con 0x7ff418108380 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff418111800 con 0x7ff418102cd0 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff418111b00 con 0x7ff418102310 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff41819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41cabb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff4181990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41cabb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff4181990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36102/0 (socket says 172.21.15.50:36102) 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41cabb700 1 -- 172.21.15.50:0/2902892381 learned_addr learned my addr 172.21.15.50:0/2902892381 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff41819d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36124/0 (socket says 172.21.15.50:36124) 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff417fff700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418102cd0 0x7ff418199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 -- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 msgr2=0x7ff4181990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff4181990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 -- 172.21.15.50:0/2902892381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418102cd0 msgr2=0x7ff418199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418102cd0 0x7ff418199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 -- 172.21.15.50:0/2902892381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41819e100 con 0x7ff418108380 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41cabb700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff4181990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41d2bc700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff41819d9c0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7ff41000cc50 tx=0x7ff41000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff415ffb700 10 client.?.objecter ms_handle_connect 0x7ff418108380 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff415ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff410016020 con 0x7ff418108380 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41819e3f0 con 0x7ff418108380 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff410012420 con 0x7ff418108380 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4100115e0 con 0x7ff418108380 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 10 client.30513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:33.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff418103f20 con 0x7ff418108380 2024-04-26T16:27:33.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 librados: init done 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41806a110 con 0x7ff418108380 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.467+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff410019020 con 0x7ff418108380 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff408071f80 0x7ff408074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(385..385 src has 87..385) ==== 9722+0+0 (secure 0 0 0) 0x7ff410091ef0 con 0x7ff418108380 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 10 client.30513.objecter ms_dispatch 0x7ff418001550 osd_map(385..385 src has 87..385) 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 3 client.30513.objecter handle_osd_map got epochs [385,385] > 0 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 3 client.30513.objecter handle_osd_map decoding full epoch 385 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff41cabb700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff408071f80 0x7ff408074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 20 client.30513.objecter dump_active .. 0 homeless 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff41cabb700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff408071f80 0x7ff408074440 secure :-1 s=READY pgs=3733 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009680 tx=0x7ff40c01d040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7ff415ffb700 10 client.30513.objecter ms_handle_connect 0x7ff408071f80 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 10 client.30507.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 6 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 6 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe318105630 osd=2 3 2024-04-26T16:27:33.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 2 6 2024-04-26T16:27:33.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 4 2024-04-26T16:27:33.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 10 client.30507.objecter _send_command 6 2024-04-26T16:27:33.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.471+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe31802dec0 con 0x7fe3181ad3c0 2024-04-26T16:27:33.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff410020020 con 0x7ff418108380 2024-04-26T16:27:33.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe31802dec0 con 0x7fe3181ad3c0 2024-04-26T16:27:33.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 10 client.30507.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 4 2024-04-26T16:27:33.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 2 6 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 10 client.30507.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 7 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3181b0e70 0x7fe3181b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 7 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 3 7 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 4 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 10 client.30507.objecter _send_command 7 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe3181aaf20 con 0x7fe3181b0e70 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3181b0e70 0x7fe3181b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe3181b0e70 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3181b0e70 0x7fe3181b12d0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181aaf20 con 0x7fe3181b0e70 2024-04-26T16:27:33.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 7: 0 ) 2024-04-26T16:27:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 10 client.30507.objecter _finish_command 7 = system:0 2024-04-26T16:27:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 4 2024-04-26T16:27:33.477 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.475+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 3 7 2024-04-26T16:27:33.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 10 client.30507.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 8 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 8 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181aa850 osd=3 3 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 3 8 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 4 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 10 client.30507.objecter _send_command 8 2024-04-26T16:27:33.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.495+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3181aaf20 con 0x7fe3181b0e70 2024-04-26T16:27:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3181aaf20 con 0x7fe3181b0e70 2024-04-26T16:27:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe314ff9700 10 client.30507.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 4 2024-04-26T16:27:33.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 3 8 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 10 client.30507.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 9 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3181b1fe0 0x7fe3181b67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 9 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 4 9 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 4 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 10 client.30507.objecter _send_command 9 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe3181b9640 con 0x7fe3181b1fe0 2024-04-26T16:27:33.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3181b1fe0 0x7fe3181b67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe3181b1fe0 2024-04-26T16:27:33.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.499+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3181b1fe0 0x7fe3181b67e0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.503+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181b9640 con 0x7fe3181b1fe0 2024-04-26T16:27:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.503+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 9: 0 ) 2024-04-26T16:27:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.503+0000 7fe314ff9700 10 client.30507.objecter _finish_command 9 = system:0 2024-04-26T16:27:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.503+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 4 2024-04-26T16:27:33.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.503+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 4 9 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 10 client.30507.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 10 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 10 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181aaf20 osd=4 3 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 4 10 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 4 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 10 client.30507.objecter _send_command 10 2024-04-26T16:27:33.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.519+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3181b6d20 con 0x7fe3181b1fe0 2024-04-26T16:27:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3181b6d20 con 0x7fe3181b1fe0 2024-04-26T16:27:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe314ff9700 10 client.30507.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 4 2024-04-26T16:27:33.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 4 10 2024-04-26T16:27:33.524 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:27:33.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 10 client.30507.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 11 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3181b7440 0x7fe3181b7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 11 2024-04-26T16:27:33.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 5 11 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 4 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 10 client.30507.objecter _send_command 11 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe3181bca10 con 0x7fe3181b7440 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3181b7440 0x7fe3181b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe3181b7440 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.523+0000 7fe316ffd700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3181b7440 0x7fe3181b7880 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.527+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181bca10 con 0x7fe3181b7440 2024-04-26T16:27:33.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.527+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 11: 0 ) 2024-04-26T16:27:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.527+0000 7fe314ff9700 10 client.30507.objecter _finish_command 11 = system:0 2024-04-26T16:27:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.527+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 4 2024-04-26T16:27:33.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.527+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 5 11 2024-04-26T16:27:33.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 10 client.30507.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 12 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 12 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181b6d20 osd=5 3 2024-04-26T16:27:33.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 5 12 2024-04-26T16:27:33.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 4 2024-04-26T16:27:33.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 10 client.30507.objecter _send_command 12 2024-04-26T16:27:33.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.543+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3181b7ed0 con 0x7fe3181b7440 2024-04-26T16:27:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3181b7ed0 con 0x7fe3181b7440 2024-04-26T16:27:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe314ff9700 10 client.30507.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 4 2024-04-26T16:27:33.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 5 12 2024-04-26T16:27:33.549 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:27:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 10 client.30507.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 13 2024-04-26T16:27:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3181b85f0 0x7fe3181b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 13 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 6 13 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 4 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 10 client.30507.objecter _send_command 13 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.547+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe3181b9110 con 0x7fe3181b85f0 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3181b85f0 0x7fe3181b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe3181b85f0 2024-04-26T16:27:33.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe317fff700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3181b85f0 0x7fe3181b8a50 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.6 v2:172.21.15.190:6816/2399532574 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181b9110 con 0x7fe3181b85f0 2024-04-26T16:27:33.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 13: 0 ) 2024-04-26T16:27:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe314ff9700 10 client.30507.objecter _finish_command 13 = system:0 2024-04-26T16:27:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 4 2024-04-26T16:27:33.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.551+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 6 13 2024-04-26T16:27:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.946805+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:27:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.946805+0000 osd.4 (osd.4) 205 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:27:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.949811+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:27:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:31.949811+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:27:33.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.275365+0000 osd.0 (osd.0) 241 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.275365+0000 osd.0 (osd.0) 241 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.277165+0000 osd.0 (osd.0) 242 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.277165+0000 osd.0 (osd.0) 242 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: audit 2024-04-26T16:27:33.141245+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: audit 2024-04-26T16:27:33.141245+0000 mon.a (mon.0) 2464 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi050-73201-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.145391+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.145391+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: audit 2024-04-26T16:27:33.149970+0000 mon.c (mon.2) 576 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: audit 2024-04-26T16:27:33.149970+0000 mon.c (mon.2) 576 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: audit 2024-04-26T16:27:33.150606+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:33 smithi190 bash[23868]: audit 2024-04-26T16:27:33.150606+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:33.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 10 client.30507.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 14 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 14 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181b7ed0 osd=6 3 2024-04-26T16:27:33.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 6 14 2024-04-26T16:27:33.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 4 2024-04-26T16:27:33.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 10 client.30507.objecter _send_command 14 2024-04-26T16:27:33.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3181b9110 con 0x7fe3181b85f0 2024-04-26T16:27:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.6 v2:172.21.15.190:6816/2399532574 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3181b9110 con 0x7fe3181b85f0 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe314ff9700 10 client.30507.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 4 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 6 14 2024-04-26T16:27:33.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 10 client.30507.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 15 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3181b9280 0x7fe3181bfde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 15 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 7 15 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 4 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 10 client.30507.objecter _send_command 15 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe3181c2e60 con 0x7fe3181b9280 2024-04-26T16:27:33.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.571+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3181b9280 0x7fe3181bfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe3177fe700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3181b9280 0x7fe3181bfde0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe314ff9700 10 client.30507.objecter ms_handle_connect 0x7fe3181b9280 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3181c2e60 con 0x7fe3181b9280 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 15: 0 ) 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe314ff9700 10 client.30507.objecter _finish_command 15 = system:0 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 4 2024-04-26T16:27:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.575+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 7 15 2024-04-26T16:27:33.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 10 client.30507.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:27:33.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign -1 16 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 20 client.30507.objecter _get_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_remove -1 16 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 20 client.30507.objecter get_session s=0x7fe3181b9110 osd=7 3 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 15 client.30507.objecter _session_command_op_assign 7 16 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 4 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 10 client.30507.objecter _send_command 16 2024-04-26T16:27:33.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.591+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe3181c2ec0 con 0x7fe3181b9280 2024-04-26T16:27:33.593 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe314ff9700 1 -- 172.21.15.50:0/2371747126 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe3181c2ec0 con 0x7fe3181b9280 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe314ff9700 10 client.30507.objecter ms_dispatch 0x7fe318001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe314ff9700 10 client.30507.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe314ff9700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 4 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe314ff9700 15 client.30507.objecter _session_command_op_remove 7 16 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 librados: watch_flush enter 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 librados: watch_flush exit 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.0 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe318064030 msgr2=0x7fe318064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe318064030 0x7fe318064410 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe31819c620 osd=0 1 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.1 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe31802d310 msgr2=0x7fe31802d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe31802d310 0x7fe31802d730 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181057b0 osd=1 1 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.2 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3181ad3c0 msgr2=0x7fe3181af7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe3181ad3c0 0x7fe3181af7b0 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe318105630 osd=2 1 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.3 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3181b0e70 msgr2=0x7fe3181b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe3181b0e70 0x7fe3181b12d0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aa850 osd=3 1 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.4 2024-04-26T16:27:33.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3181b1fe0 msgr2=0x7fe3181b67e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe3181b1fe0 0x7fe3181b67e0 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181aaf20 osd=4 1 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.5 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3181b7440 msgr2=0x7fe3181b7880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe3181b7440 0x7fe3181b7880 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b6d20 osd=5 1 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.6 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3181b85f0 msgr2=0x7fe3181b8a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6816/2399532574,v1:172.21.15.190:6817/2399532574] conn(0x7fe3181b85f0 0x7fe3181b8a50 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe317fff700 1 -- 172.21.15.50:0/2371747126 reap_dead start 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b7ed0 osd=6 1 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter close_session for osd.7 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3181b9280 msgr2=0x7fe3181bfde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe3181b9280 0x7fe3181bfde0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe317fff700 1 -- 172.21.15.50:0/2371747126 reap_dead start 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter put_session s=0x7fe3181b9110 osd=7 1 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 20 client.30507.objecter shutdown clearing up homeless session... 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe317fff700 1 -- 172.21.15.50:0/2371747126 reap_dead start 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 10 client.30507.objecter successfully canceled tick 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe317fff700 1 -- 172.21.15.50:0/2371747126 reap_dead start 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe304071f80 msgr2=0x7fe304074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe304071f80 0x7fe304074440 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7fe30000b2e0 tx=0x7fe300005960 comp rx=0 tx=0).stop 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 msgr2=0x7fe31819b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe31819b800 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fe30800c9b0 tx=0x7fe30800ccc0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 shutdown_connections 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe318103e30 0x7fe31819b800 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 --2- 172.21.15.50:0/2371747126 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe304071f80 0x7fe304074440 unknown :-1 s=CLOSED pgs=3732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:33.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 >> 172.21.15.50:0/2371747126 conn(0x7fe31807a5d0 msgr2=0x7fe318075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 shutdown_connections 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 -- 172.21.15.50:0/2371747126 wait complete. 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.595+0000 7fe31dc40700 1 librados: shutdown 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:27:33.599 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:27:33.600 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:27:33.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:33.689 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:33.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:33.719+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ff418104240 con 0x7ff418108380 2024-04-26T16:27:33.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:33.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:33.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:33.945 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:33.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:34.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:34.076 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:34.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.139+0000 7ff415ffb700 1 -- 172.21.15.50:0/2902892381 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v386) ==== 101+0+0 (secure 0 0 0) 0x7ff41005d7d0 con 0x7ff418108380 2024-04-26T16:27:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 10 librados: watch_flush enter 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 10 librados: watch_flush exit 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 20 client.30513.objecter shutdown clearing up homeless session... 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 10 client.30513.objecter successfully canceled tick 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff408071f80 msgr2=0x7ff408074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff408071f80 0x7ff408074440 secure :-1 s=READY pgs=3733 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009680 tx=0x7ff40c01d040 comp rx=0 tx=0).stop 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 msgr2=0x7ff41819d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff41819d9c0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7ff41000cc50 tx=0x7ff41000cf60 comp rx=0 tx=0).stop 2024-04-26T16:27:34.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 shutdown_connections 2024-04-26T16:27:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff408071f80 0x7ff408074440 unknown :-1 s=CLOSED pgs=3733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff418108380 0x7ff41819d9c0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff418102310 0x7ff4181990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 --2- 172.21.15.50:0/2902892381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418102cd0 0x7ff418199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:34.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 >> 172.21.15.50:0/2902892381 conn(0x7ff418079290 msgr2=0x7ff4181061d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 shutdown_connections 2024-04-26T16:27:34.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.143+0000 7ff41ed1f700 1 -- 172.21.15.50:0/2902892381 wait complete. 2024-04-26T16:27:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:34.147+0000 7ff41ed1f700 1 librados: shutdown 2024-04-26T16:27:34.147 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:27:34.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:34.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:34.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.879666+0000 osd.1 (osd.1) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.879666+0000 osd.1 (osd.1) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.879933+0000 osd.1 (osd.1) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:32.879933+0000 osd.1 (osd.1) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.257462+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.257462+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.258794+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.258794+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.259950+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.259950+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.261507+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.261507+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.405294+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? 172.21.15.50:0/2371747126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.405294+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? 172.21.15.50:0/2371747126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.614825+0000 mgr.y (mgr.24401) 400 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.614825+0000 mgr.y (mgr.24401) 400 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-26T16:27:34.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.725967+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.879666+0000 osd.1 (osd.1) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.879666+0000 osd.1 (osd.1) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.879933+0000 osd.1 (osd.1) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:32.879933+0000 osd.1 (osd.1) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.257462+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.257462+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.258794+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.258794+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.259950+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.259950+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.261507+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.261507+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.405294+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? 172.21.15.50:0/2371747126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:34.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.405294+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? 172.21.15.50:0/2371747126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.614825+0000 mgr.y (mgr.24401) 400 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.614825+0000 mgr.y (mgr.24401) 400 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.725967+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:33.725967+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.142038+0000 mon.a (mon.0) 2471 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.142038+0000 mon.a (mon.0) 2471 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:34.143946+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]': finished 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:34.143946+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]': finished 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:34.144071+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: audit 2024-04-26T16:27:34.144071+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:34.318 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.164561+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.164561+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:33.725967+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.142038+0000 mon.a (mon.0) 2471 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.142038+0000 mon.a (mon.0) 2471 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:34.143946+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]': finished 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:34.143946+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]': finished 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:34.144071+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: audit 2024-04-26T16:27:34.144071+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.164561+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-04-26T16:27:34.319 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.164561+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-04-26T16:27:34.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:34.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:34.442 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:27:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:27:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:27:34.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:34.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:34.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.879666+0000 osd.1 (osd.1) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.879666+0000 osd.1 (osd.1) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.879933+0000 osd.1 (osd.1) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:34.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:32.879933+0000 osd.1 (osd.1) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.257462+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.257462+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.258794+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.258794+0000 mon.a (mon.0) 2468 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.259950+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.259950+0000 mon.b (mon.1) 262 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.261507+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.261507+0000 mon.b (mon.1) 263 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.405294+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? 172.21.15.50:0/2371747126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.405294+0000 mon.a (mon.0) 2469 : audit [DBG] from='client.? 172.21.15.50:0/2371747126' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.614825+0000 mgr.y (mgr.24401) 400 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.614825+0000 mgr.y (mgr.24401) 400 : cluster [DBG] pgmap v562: 228 pgs: 228 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.725967+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:34.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:33.725967+0000 mon.a (mon.0) 2470 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.142038+0000 mon.a (mon.0) 2471 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.142038+0000 mon.a (mon.0) 2471 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:34.143946+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]': finished 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:34.143946+0000 mon.a (mon.0) 2472 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-45"}]': finished 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:34.144071+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: audit 2024-04-26T16:27:34.144071+0000 mon.a (mon.0) 2473 : audit [INF] from='client.? 172.21.15.50:0/2902892381' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.164561+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-04-26T16:27:34.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:34 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.164561+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2024-04-26T16:27:34.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:34.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:34.722 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:34.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:34.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:34.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:35.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:35.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:35.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:35.239 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:35.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:35.368 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:35.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.954175+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.e scrub starts 2024-04-26T16:27:35.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.954175+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.e scrub starts 2024-04-26T16:27:35.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.956365+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.e scrub ok 2024-04-26T16:27:35.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:33.956365+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.e scrub ok 2024-04-26T16:27:35.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.192280+0000 osd.6 (osd.6) 177 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.192280+0000 osd.6 (osd.6) 177 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.197543+0000 osd.6 (osd.6) 178 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.197543+0000 osd.6 (osd.6) 178 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.209251+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.b scrub starts 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.209251+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.b scrub starts 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.211851+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.b scrub ok 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.211851+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.b scrub ok 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: audit 2024-04-26T16:27:35.146979+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: audit 2024-04-26T16:27:35.146979+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.152164+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-04-26T16:27:35.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.152164+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.954175+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.e scrub starts 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.954175+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.e scrub starts 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.956365+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.e scrub ok 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:33.956365+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.e scrub ok 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.192280+0000 osd.6 (osd.6) 177 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.192280+0000 osd.6 (osd.6) 177 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.197543+0000 osd.6 (osd.6) 178 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.197543+0000 osd.6 (osd.6) 178 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.209251+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.b scrub starts 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.209251+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.b scrub starts 2024-04-26T16:27:35.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.211851+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.b scrub ok 2024-04-26T16:27:35.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.211851+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.b scrub ok 2024-04-26T16:27:35.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: audit 2024-04-26T16:27:35.146979+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:35.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: audit 2024-04-26T16:27:35.146979+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:35.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.152164+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-04-26T16:27:35.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:35 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.152164+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-04-26T16:27:35.471 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:35.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.954175+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.e scrub starts 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.954175+0000 osd.4 (osd.4) 207 : cluster [DBG] 5.e scrub starts 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.956365+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.e scrub ok 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:33.956365+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.e scrub ok 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.192280+0000 osd.6 (osd.6) 177 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.192280+0000 osd.6 (osd.6) 177 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.197543+0000 osd.6 (osd.6) 178 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.197543+0000 osd.6 (osd.6) 178 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:27:35.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.209251+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.b scrub starts 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.209251+0000 osd.3 (osd.3) 243 : cluster [DBG] 3.b scrub starts 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.211851+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.b scrub ok 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.211851+0000 osd.3 (osd.3) 244 : cluster [DBG] 3.b scrub ok 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: audit 2024-04-26T16:27:35.146979+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: audit 2024-04-26T16:27:35.146979+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi050-73201-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.152164+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-04-26T16:27:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:35 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.152164+0000 mon.a (mon.0) 2476 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2024-04-26T16:27:35.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:35.630 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:35.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:35.760 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:35.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:35.890 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:35.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:35.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:36.023 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:36.122 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:36.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:36.161 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:36.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:36.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:36.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:36.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.945133+0000 osd.7 (osd.7) 241 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:27:36.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.945133+0000 osd.7 (osd.7) 241 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.948796+0000 osd.7 (osd.7) 242 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:34.948796+0000 osd.7 (osd.7) 242 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.196515+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.196515+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.198919+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.198919+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.615762+0000 mgr.y (mgr.24401) 401 : cluster [DBG] pgmap v565: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.615762+0000 mgr.y (mgr.24401) 401 : cluster [DBG] pgmap v565: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.123832+0000 mon.a (mon.0) 2477 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.123832+0000 mon.a (mon.0) 2477 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.132372+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-04-26T16:27:36.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.132372+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-04-26T16:27:36.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: audit 2024-04-26T16:27:36.163455+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: audit 2024-04-26T16:27:36.163455+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: audit 2024-04-26T16:27:36.165311+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[23868]: audit 2024-04-26T16:27:36.165311+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.421 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:36.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:36.553 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:36.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.945133+0000 osd.7 (osd.7) 241 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.945133+0000 osd.7 (osd.7) 241 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.948796+0000 osd.7 (osd.7) 242 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:34.948796+0000 osd.7 (osd.7) 242 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.196515+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.196515+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.198919+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.198919+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.615762+0000 mgr.y (mgr.24401) 401 : cluster [DBG] pgmap v565: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.615762+0000 mgr.y (mgr.24401) 401 : cluster [DBG] pgmap v565: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.123832+0000 mon.a (mon.0) 2477 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.123832+0000 mon.a (mon.0) 2477 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:36.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.132372+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.132372+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: audit 2024-04-26T16:27:36.163455+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: audit 2024-04-26T16:27:36.163455+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: audit 2024-04-26T16:27:36.165311+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[29234]: audit 2024-04-26T16:27:36.165311+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.945133+0000 osd.7 (osd.7) 241 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.945133+0000 osd.7 (osd.7) 241 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.948796+0000 osd.7 (osd.7) 242 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:27:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:34.948796+0000 osd.7 (osd.7) 242 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.196515+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.196515+0000 osd.3 (osd.3) 245 : cluster [DBG] 6.c scrub starts 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.198919+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.198919+0000 osd.3 (osd.3) 246 : cluster [DBG] 6.c scrub ok 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.615762+0000 mgr.y (mgr.24401) 401 : cluster [DBG] pgmap v565: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.615762+0000 mgr.y (mgr.24401) 401 : cluster [DBG] pgmap v565: 204 pgs: 8 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.123832+0000 mon.a (mon.0) 2477 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.123832+0000 mon.a (mon.0) 2477 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.132372+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.132372+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2024-04-26T16:27:36.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: audit 2024-04-26T16:27:36.163455+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: audit 2024-04-26T16:27:36.163455+0000 mon.a (mon.0) 2479 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: audit 2024-04-26T16:27:36.165311+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:36 smithi050 bash[21463]: audit 2024-04-26T16:27:36.165311+0000 mon.b (mon.1) 264 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:36.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:36.684 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:36.787 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:36.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:36.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:36.928 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:27:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:27:36.943 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:36.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:37.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:37.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:37.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:27:37.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:37.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:37.234 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:27:37.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.969592+0000 osd.4 (osd.4) 209 : cluster [DBG] 5.f scrub starts 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.969592+0000 osd.4 (osd.4) 209 : cluster [DBG] 5.f scrub starts 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.971694+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.f scrub ok 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:35.971694+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.f scrub ok 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.189946+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.7 deep-scrub starts 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.189946+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.7 deep-scrub starts 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.191209+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.7 deep-scrub ok 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.191209+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.7 deep-scrub ok 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.292651+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.292651+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.294165+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:27:37.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:36.294165+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.129773+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.129773+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.134660+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.134660+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.136772+0000 mon.c (mon.2) 577 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.136772+0000 mon.c (mon.2) 577 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.140683+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.140683+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.200802+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.200802+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.203377+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.310 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[29234]: audit 2024-04-26T16:27:37.203377+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.969592+0000 osd.4 (osd.4) 209 : cluster [DBG] 5.f scrub starts 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.969592+0000 osd.4 (osd.4) 209 : cluster [DBG] 5.f scrub starts 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.971694+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.f scrub ok 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:35.971694+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.f scrub ok 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.189946+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.7 deep-scrub starts 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.189946+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.7 deep-scrub starts 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.191209+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.7 deep-scrub ok 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.191209+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.7 deep-scrub ok 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.292651+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.292651+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:27:37.311 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.294165+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:36.294165+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.129773+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.129773+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.134660+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.134660+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.136772+0000 mon.c (mon.2) 577 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.136772+0000 mon.c (mon.2) 577 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.140683+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.140683+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.200802+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.200802+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.203377+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.312 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:37 smithi050 bash[21463]: audit 2024-04-26T16:27:37.203377+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.322 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:37.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:37.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:37.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.969592+0000 osd.4 (osd.4) 209 : cluster [DBG] 5.f scrub starts 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.969592+0000 osd.4 (osd.4) 209 : cluster [DBG] 5.f scrub starts 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.971694+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.f scrub ok 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:35.971694+0000 osd.4 (osd.4) 210 : cluster [DBG] 5.f scrub ok 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.189946+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.7 deep-scrub starts 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.189946+0000 osd.1 (osd.1) 211 : cluster [DBG] 5.7 deep-scrub starts 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.191209+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.7 deep-scrub ok 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.191209+0000 osd.1 (osd.1) 212 : cluster [DBG] 5.7 deep-scrub ok 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.292651+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:27:37.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.292651+0000 osd.2 (osd.2) 185 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.294165+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:36.294165+0000 osd.2 (osd.2) 186 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.129773+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.129773+0000 mon.a (mon.0) 2480 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.134660+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.134660+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.136772+0000 mon.c (mon.2) 577 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.136772+0000 mon.c (mon.2) 577 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.140683+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.140683+0000 mon.a (mon.0) 2482 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.200802+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.200802+0000 mon.a (mon.0) 2483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.203377+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:37 smithi190 bash[23868]: audit 2024-04-26T16:27:37.203377+0000 mon.b (mon.1) 265 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:37.582 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:37.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:27:37.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:37.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:37.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:37.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:37.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:37.986 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:37.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:38.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:38.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:38.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:38.247 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:38.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:38.378 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:38.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:36.826648+0000 mgr.y (mgr.24401) 402 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:38.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:36.826648+0000 mgr.y (mgr.24401) 402 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:38.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.014029+0000 osd.7 (osd.7) 243 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:27:38.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.014029+0000 osd.7 (osd.7) 243 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:27:38.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.015800+0000 osd.7 (osd.7) 244 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:27:38.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.015800+0000 osd.7 (osd.7) 244 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:27:38.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.017879+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:27:38.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.017879+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:27:38.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.018963+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:27:38.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.018963+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:27:38.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.171075+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:27:38.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.171075+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.172856+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.172856+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.248573+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.e scrub starts 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.248573+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.e scrub starts 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.249742+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.e scrub ok 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.249742+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.e scrub ok 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.616617+0000 mgr.y (mgr.24401) 403 : cluster [DBG] pgmap v568: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:37.616617+0000 mgr.y (mgr.24401) 403 : cluster [DBG] pgmap v568: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:27:38.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.132818+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:38.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.132818+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:38.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.133176+0000 mon.a (mon.0) 2485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:36.826648+0000 mgr.y (mgr.24401) 402 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:36.826648+0000 mgr.y (mgr.24401) 402 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.014029+0000 osd.7 (osd.7) 243 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.014029+0000 osd.7 (osd.7) 243 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.015800+0000 osd.7 (osd.7) 244 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.015800+0000 osd.7 (osd.7) 244 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.017879+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.017879+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.018963+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.018963+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.171075+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:27:38.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.171075+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.172856+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.172856+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.248573+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.e scrub starts 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.248573+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.e scrub starts 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.249742+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.e scrub ok 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.249742+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.e scrub ok 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.616617+0000 mgr.y (mgr.24401) 403 : cluster [DBG] pgmap v568: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:37.616617+0000 mgr.y (mgr.24401) 403 : cluster [DBG] pgmap v568: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:27:38.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.132818+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.132818+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.133176+0000 mon.a (mon.0) 2485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.133176+0000 mon.a (mon.0) 2485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.141932+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.141932+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.144029+0000 mon.c (mon.2) 578 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.144029+0000 mon.c (mon.2) 578 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.144692+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.144692+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.145752+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.145752+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.146095+0000 mon.a (mon.0) 2488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[29234]: audit 2024-04-26T16:27:38.146095+0000 mon.a (mon.0) 2488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.133176+0000 mon.a (mon.0) 2485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.141932+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.141932+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.144029+0000 mon.c (mon.2) 578 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.144029+0000 mon.c (mon.2) 578 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.144692+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.144692+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.145752+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.145752+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.388 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.146095+0000 mon.a (mon.0) 2488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.389 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:38 smithi050 bash[21463]: audit 2024-04-26T16:27:38.146095+0000 mon.a (mon.0) 2488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:36.826648+0000 mgr.y (mgr.24401) 402 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:36.826648+0000 mgr.y (mgr.24401) 402 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.014029+0000 osd.7 (osd.7) 243 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.014029+0000 osd.7 (osd.7) 243 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.015800+0000 osd.7 (osd.7) 244 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.015800+0000 osd.7 (osd.7) 244 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.017879+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:27:38.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.017879+0000 osd.4 (osd.4) 211 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.018963+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.018963+0000 osd.4 (osd.4) 212 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.171075+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.171075+0000 osd.3 (osd.3) 247 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.172856+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.172856+0000 osd.3 (osd.3) 248 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.248573+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.e scrub starts 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.248573+0000 osd.2 (osd.2) 187 : cluster [DBG] 2.e scrub starts 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.249742+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.e scrub ok 2024-04-26T16:27:38.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.249742+0000 osd.2 (osd.2) 188 : cluster [DBG] 2.e scrub ok 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.616617+0000 mgr.y (mgr.24401) 403 : cluster [DBG] pgmap v568: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:37.616617+0000 mgr.y (mgr.24401) 403 : cluster [DBG] pgmap v568: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.132818+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.132818+0000 mon.a (mon.0) 2484 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.133176+0000 mon.a (mon.0) 2485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.133176+0000 mon.a (mon.0) 2485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.141932+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.141932+0000 mon.a (mon.0) 2486 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.144029+0000 mon.c (mon.2) 578 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.144029+0000 mon.c (mon.2) 578 : audit [INF] from='client.? 172.21.15.50:0/4281395759' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.144692+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.144692+0000 mon.a (mon.0) 2487 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.145752+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.145752+0000 mon.b (mon.1) 266 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:38.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.146095+0000 mon.a (mon.0) 2488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:38 smithi190 bash[23868]: audit 2024-04-26T16:27:38.146095+0000 mon.a (mon.0) 2488 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]: dispatch 2024-04-26T16:27:38.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:38.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:38.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:38.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:38.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:38.773 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:38.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:38.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:38.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:38.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:39.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:39.029 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:39.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:39.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:39.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:39.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.124669+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:27:39.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.124669+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:27:39.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.127179+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:27:39.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.127179+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:27:39.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.181947+0000 osd.1 (osd.1) 213 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:27:39.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.181947+0000 osd.1 (osd.1) 213 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.188265+0000 osd.1 (osd.1) 214 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:38.188265+0000 osd.1 (osd.1) 214 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.136431+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.136431+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.136702+0000 mon.a (mon.0) 2490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.136702+0000 mon.a (mon.0) 2490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.144992+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.144992+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.148191+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.148191+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.148617+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.148617+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.181609+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.279 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.181609+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.182616+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.182616+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.183103+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: audit 2024-04-26T16:27:39.183103+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.198712+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.198712+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.202752+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:27:39.280 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:39 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.202752+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:27:39.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:39.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:39.421 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:39.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:39.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:39.548 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.124669+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.124669+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.127179+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.127179+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.181947+0000 osd.1 (osd.1) 213 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.181947+0000 osd.1 (osd.1) 213 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.188265+0000 osd.1 (osd.1) 214 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:38.188265+0000 osd.1 (osd.1) 214 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.136431+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.136431+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.136702+0000 mon.a (mon.0) 2490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.136702+0000 mon.a (mon.0) 2490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.144992+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.144992+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.148191+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.148191+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.148617+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.148617+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.181609+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.124669+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.124669+0000 osd.3 (osd.3) 249 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.127179+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.127179+0000 osd.3 (osd.3) 250 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.181947+0000 osd.1 (osd.1) 213 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.181947+0000 osd.1 (osd.1) 213 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.188265+0000 osd.1 (osd.1) 214 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:38.188265+0000 osd.1 (osd.1) 214 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.136431+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.136431+0000 mon.a (mon.0) 2489 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.136702+0000 mon.a (mon.0) 2490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.136702+0000 mon.a (mon.0) 2490 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi050-73201-61"}]': finished 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.144992+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-04-26T16:27:39.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.144992+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.148191+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.148191+0000 mon.b (mon.1) 267 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.148617+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.148617+0000 mon.a (mon.0) 2492 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.181609+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.181609+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.182616+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.182616+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.183103+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: audit 2024-04-26T16:27:39.183103+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.198712+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.198712+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:27:39.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.202752+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.202752+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.181609+0000 mon.a (mon.0) 2493 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.182616+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.182616+0000 mon.a (mon.0) 2494 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.183103+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: audit 2024-04-26T16:27:39.183103+0000 mon.a (mon.0) 2495 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.198712+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.198712+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.202752+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:27:39.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:39 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.202752+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:27:39.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:39.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:39.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:39.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:39.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:39.938 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:39.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:40.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:40.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:40.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.198 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:40.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:40.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:40.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:40.420 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:27:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:27:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:27:40.421 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:27:40.421 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:27:40.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.114936+0000 osd.4 (osd.4) 213 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.114936+0000 osd.4 (osd.4) 213 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.116322+0000 osd.4 (osd.4) 214 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.116322+0000 osd.4 (osd.4) 214 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.233820+0000 osd.0 (osd.0) 243 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.233820+0000 osd.0 (osd.0) 243 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:40.432 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.234004+0000 osd.0 (osd.0) 244 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.234004+0000 osd.0 (osd.0) 244 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.617428+0000 mgr.y (mgr.24401) 404 : cluster [DBG] pgmap v571: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:39.617428+0000 mgr.y (mgr.24401) 404 : cluster [DBG] pgmap v571: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.136873+0000 mon.a (mon.0) 2496 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.136873+0000 mon.a (mon.0) 2496 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.139387+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]': finished 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.139387+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]': finished 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.139568+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.139568+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.147396+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.147396+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.148236+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.148236+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:40.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.155245+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.155245+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.157430+0000 mon.b (mon.1) 268 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: audit 2024-04-26T16:27:40.157430+0000 mon.b (mon.1) 268 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.212257+0000 osd.6 (osd.6) 181 : cluster [DBG] 3.a scrub starts 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.212257+0000 osd.6 (osd.6) 181 : cluster [DBG] 3.a scrub starts 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.215912+0000 osd.6 (osd.6) 182 : cluster [DBG] 3.a scrub ok 2024-04-26T16:27:40.434 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:40 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.215912+0000 osd.6 (osd.6) 182 : cluster [DBG] 3.a scrub ok 2024-04-26T16:27:40.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/2872047193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41094e0 msgr2=0x7fd7a41098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/2872047193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41094e0 0x7fd7a41098c0 secure :-1 s=READY pgs=2460 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0009990 tx=0x7fd7a0009ca0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/2872047193 shutdown_connections 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/2872047193 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a4103e60 0x7fd7a41102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/2872047193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41094e0 0x7fd7a41098c0 unknown :-1 s=CLOSED pgs=2460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/2872047193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a41034a0 0x7fd7a4103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/2872047193 >> 172.21.15.50:0/2872047193 conn(0x7fd7a407a590 msgr2=0x7fd7a407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/2872047193 shutdown_connections 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/2872047193 wait complete. 2024-04-26T16:27:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 librados: starting msgr at 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 librados: starting objecter 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 Processor -- start 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- start start 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 librados: setting wanted keys 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 librados: calling monclient init 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 0x7fd7a4199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a4103e60 0x7fd7a4199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a41094e0 0x7fd7a419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4112ad0 con 0x7fd7a41034a0 2024-04-26T16:27:40.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a4112950 con 0x7fd7a4103e60 2024-04-26T16:27:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd7a4112c50 con 0x7fd7a41094e0 2024-04-26T16:27:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 0x7fd7a4199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 0x7fd7a4199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33586/0 (socket says 172.21.15.50:33586) 2024-04-26T16:27:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 -- 172.21.15.50:0/627120950 learned_addr learned my addr 172.21.15.50:0/627120950 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a99ca700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a41094e0 0x7fd7a419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:40.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a89c8700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a4103e60 0x7fd7a4199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 -- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a41094e0 msgr2=0x7fd7a419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a41094e0 0x7fd7a419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 -- 172.21.15.50:0/627120950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a4103e60 msgr2=0x7fd7a4199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a4103e60 0x7fd7a4199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 -- 172.21.15.50:0/627120950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a419e140 con 0x7fd7a41034a0 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a99ca700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a41094e0 0x7fd7a419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7a91c9700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 0x7fd7a4199130 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00057f0 tx=0x7fd7a0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd79a7fc700 10 client.?.objecter ms_handle_connect 0x7fd7a41034a0 2024-04-26T16:27:40.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd79a7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0026070 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a419e3d0 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd7a0021410 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a00173f0 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 10 client.30525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a41a5cb0 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.487+0000 7fd7ab42d700 1 librados: init done 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a406a110 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd7a0029070 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd790071f80 0x7fd790074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(392..392 src has 87..392) ==== 9698+0+0 (secure 0 0 0) 0x7fd7a0092590 con 0x7fd7a41034a0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 10 client.30525.objecter ms_dispatch 0x7fd7a4001550 osd_map(392..392 src has 87..392) 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 3 client.30525.objecter handle_osd_map got epochs [392,392] > 0 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 3 client.30525.objecter handle_osd_map decoding full epoch 392 2024-04-26T16:27:40.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd7a89c8700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd790071f80 0x7fd790074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:40.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 20 client.30525.objecter dump_active .. 0 homeless 2024-04-26T16:27:40.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd79a7fc700 10 client.30525.objecter ms_handle_connect 0x7fd790071f80 2024-04-26T16:27:40.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.491+0000 7fd7a89c8700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd790071f80 0x7fd790074440 secure :-1 s=READY pgs=3735 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4108f50 tx=0x7fd794009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:40.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.495+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd7a0013070 con 0x7fd7a41034a0 2024-04-26T16:27:40.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:40.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.114936+0000 osd.4 (osd.4) 213 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:27:40.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.114936+0000 osd.4 (osd.4) 213 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:27:40.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.116322+0000 osd.4 (osd.4) 214 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:27:40.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.116322+0000 osd.4 (osd.4) 214 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.233820+0000 osd.0 (osd.0) 243 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.233820+0000 osd.0 (osd.0) 243 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.234004+0000 osd.0 (osd.0) 244 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.234004+0000 osd.0 (osd.0) 244 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.617428+0000 mgr.y (mgr.24401) 404 : cluster [DBG] pgmap v571: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:39.617428+0000 mgr.y (mgr.24401) 404 : cluster [DBG] pgmap v571: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.136873+0000 mon.a (mon.0) 2496 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.136873+0000 mon.a (mon.0) 2496 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.139387+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]': finished 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.139387+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]': finished 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.139568+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.139568+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.147396+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.147396+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-04-26T16:27:40.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.148236+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.148236+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.155245+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.155245+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.157430+0000 mon.b (mon.1) 268 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: audit 2024-04-26T16:27:40.157430+0000 mon.b (mon.1) 268 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.212257+0000 osd.6 (osd.6) 181 : cluster [DBG] 3.a scrub starts 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.212257+0000 osd.6 (osd.6) 181 : cluster [DBG] 3.a scrub starts 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.215912+0000 osd.6 (osd.6) 182 : cluster [DBG] 3.a scrub ok 2024-04-26T16:27:40.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.215912+0000 osd.6 (osd.6) 182 : cluster [DBG] 3.a scrub ok 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.114936+0000 osd.4 (osd.4) 213 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.114936+0000 osd.4 (osd.4) 213 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.116322+0000 osd.4 (osd.4) 214 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.116322+0000 osd.4 (osd.4) 214 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.233820+0000 osd.0 (osd.0) 243 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.233820+0000 osd.0 (osd.0) 243 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.234004+0000 osd.0 (osd.0) 244 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.234004+0000 osd.0 (osd.0) 244 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.617428+0000 mgr.y (mgr.24401) 404 : cluster [DBG] pgmap v571: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:39.617428+0000 mgr.y (mgr.24401) 404 : cluster [DBG] pgmap v571: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s 2024-04-26T16:27:40.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.136873+0000 mon.a (mon.0) 2496 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.136873+0000 mon.a (mon.0) 2496 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.139387+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]': finished 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.139387+0000 mon.a (mon.0) 2497 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-47", "mode": "writeback"}]': finished 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.139568+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.139568+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.147396+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.147396+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.148236+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.148236+0000 mon.a (mon.0) 2500 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.155245+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.155245+0000 mon.a (mon.0) 2501 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.157430+0000 mon.b (mon.1) 268 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: audit 2024-04-26T16:27:40.157430+0000 mon.b (mon.1) 268 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:27:40.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.212257+0000 osd.6 (osd.6) 181 : cluster [DBG] 3.a scrub starts 2024-04-26T16:27:40.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.212257+0000 osd.6 (osd.6) 181 : cluster [DBG] 3.a scrub starts 2024-04-26T16:27:40.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.215912+0000 osd.6 (osd.6) 182 : cluster [DBG] 3.a scrub ok 2024-04-26T16:27:40.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:40 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.215912+0000 osd.6 (osd.6) 182 : cluster [DBG] 3.a scrub ok 2024-04-26T16:27:40.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:40.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:40.702 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:40.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.719+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7a4102370 con 0x7fd790071f80 2024-04-26T16:27:40.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.771+0000 7fd79a7fc700 1 -- 172.21.15.50:0/627120950 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+583438 (secure 0 0 0) 0x7fd7a4102370 con 0x7fd790071f80 2024-04-26T16:27:40.776 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:27:40.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 10 librados: watch_flush enter 2024-04-26T16:27:40.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 10 librados: watch_flush exit 2024-04-26T16:27:40.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 20 client.30525.objecter shutdown clearing up homeless session... 2024-04-26T16:27:40.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 10 client.30525.objecter successfully canceled tick 2024-04-26T16:27:40.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd790071f80 msgr2=0x7fd790074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd790071f80 0x7fd790074440 secure :-1 s=READY pgs=3735 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4108f50 tx=0x7fd794009260 comp rx=0 tx=0).stop 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 msgr2=0x7fd7a4199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 0x7fd7a4199130 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fd7a00057f0 tx=0x7fd7a0005e80 comp rx=0 tx=0).stop 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 shutdown_connections 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd790071f80 0x7fd790074440 unknown :-1 s=CLOSED pgs=3735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7a41094e0 0x7fd7a419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7a41034a0 0x7fd7a4199130 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 --2- 172.21.15.50:0/627120950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7a4103e60 0x7fd7a4199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 >> 172.21.15.50:0/627120950 conn(0x7fd7a407a590 msgr2=0x7fd7a4101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 shutdown_connections 2024-04-26T16:27:40.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 -- 172.21.15.50:0/627120950 wait complete. 2024-04-26T16:27:40.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:40.783+0000 7fd7ab42d700 1 librados: shutdown 2024-04-26T16:27:40.788 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:27:40.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:40.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:40.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:40.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:40.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:41.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:41.106 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:41.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-26T16:27:41.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:41.232 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:41.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:41.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:41.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:41.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.036626+0000 osd.2 (osd.2) 189 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:41.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.036626+0000 osd.2 (osd.2) 189 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:41.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.036930+0000 osd.2 (osd.2) 190 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:41.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.036930+0000 osd.2 (osd.2) 190 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:41.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.158199+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.7 deep-scrub starts 2024-04-26T16:27:41.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.158199+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.7 deep-scrub starts 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.164992+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.7 deep-scrub ok 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.164992+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.7 deep-scrub ok 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.181329+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.d scrub starts 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.181329+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.d scrub starts 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.184246+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.d scrub ok 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.184246+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.d scrub ok 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.305376+0000 osd.0 (osd.0) 245 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.305376+0000 osd.0 (osd.0) 245 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.307875+0000 osd.0 (osd.0) 246 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:40.307875+0000 osd.0 (osd.0) 246 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:27:41.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: audit 2024-04-26T16:27:41.148136+0000 mon.a (mon.0) 2502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: audit 2024-04-26T16:27:41.148136+0000 mon.a (mon.0) 2502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: audit 2024-04-26T16:27:41.157428+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: audit 2024-04-26T16:27:41.157428+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.175417+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.175417+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: audit 2024-04-26T16:27:41.177783+0000 mon.a (mon.0) 2504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:41 smithi190 bash[23868]: audit 2024-04-26T16:27:41.177783+0000 mon.a (mon.0) 2504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.491 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:41.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:41.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.036626+0000 osd.2 (osd.2) 189 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.036626+0000 osd.2 (osd.2) 189 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.036930+0000 osd.2 (osd.2) 190 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.036930+0000 osd.2 (osd.2) 190 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.158199+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.7 deep-scrub starts 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.158199+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.7 deep-scrub starts 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.164992+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.7 deep-scrub ok 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.164992+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.7 deep-scrub ok 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.181329+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.d scrub starts 2024-04-26T16:27:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.181329+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.d scrub starts 2024-04-26T16:27:41.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.184246+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.d scrub ok 2024-04-26T16:27:41.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.184246+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.d scrub ok 2024-04-26T16:27:41.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.305376+0000 osd.0 (osd.0) 245 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:27:41.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.305376+0000 osd.0 (osd.0) 245 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:27:41.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.307875+0000 osd.0 (osd.0) 246 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:40.307875+0000 osd.0 (osd.0) 246 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: audit 2024-04-26T16:27:41.148136+0000 mon.a (mon.0) 2502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: audit 2024-04-26T16:27:41.148136+0000 mon.a (mon.0) 2502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: audit 2024-04-26T16:27:41.157428+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: audit 2024-04-26T16:27:41.157428+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.175417+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.175417+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: audit 2024-04-26T16:27:41.177783+0000 mon.a (mon.0) 2504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[29234]: audit 2024-04-26T16:27:41.177783+0000 mon.a (mon.0) 2504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.036626+0000 osd.2 (osd.2) 189 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.036626+0000 osd.2 (osd.2) 189 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.036930+0000 osd.2 (osd.2) 190 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.036930+0000 osd.2 (osd.2) 190 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.158199+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.7 deep-scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.158199+0000 osd.3 (osd.3) 251 : cluster [DBG] 6.7 deep-scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.164992+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.7 deep-scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.164992+0000 osd.3 (osd.3) 252 : cluster [DBG] 6.7 deep-scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.181329+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.d scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.181329+0000 osd.1 (osd.1) 215 : cluster [DBG] 6.d scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.184246+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.d scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.184246+0000 osd.1 (osd.1) 216 : cluster [DBG] 6.d scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.305376+0000 osd.0 (osd.0) 245 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.305376+0000 osd.0 (osd.0) 245 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.307875+0000 osd.0 (osd.0) 246 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:40.307875+0000 osd.0 (osd.0) 246 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:27:41.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: audit 2024-04-26T16:27:41.148136+0000 mon.a (mon.0) 2502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: audit 2024-04-26T16:27:41.148136+0000 mon.a (mon.0) 2502 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: audit 2024-04-26T16:27:41.157428+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: audit 2024-04-26T16:27:41.157428+0000 mon.b (mon.1) 269 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.175417+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.175417+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: audit 2024-04-26T16:27:41.177783+0000 mon.a (mon.0) 2504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:41 smithi050 bash[21463]: audit 2024-04-26T16:27:41.177783+0000 mon.a (mon.0) 2504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:27:41.717 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":572,"stamp":"2024-04-26T16:27:40.142212+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792036,"num_objects":323,"num_object_clones":2,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":1685,"num_read_kb":1436,"num_write":1205,"num_write_kb":15110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":314,"num_bytes_recovered":5967748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2579,"ondisk_log_size":2579,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":687,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":553520,"kb_used_data":70032,"kb_used_omap":10,"kb_used_meta":483253,"kb_avail":655662544,"statfs":{"total":671965249536,"available":671398445056,"internally_reserved":0,"allocated":71712768,"data_stored":53143184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11153,"internal_metadata":494851183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012695"},"pg_stats":[{"pgid":"284.1f","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.143419+0000","last_change":"2024-04-26T16:27:36.143419+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.143419+0000","last_undegraded":"2024-04-26T16:27:36.143419+0000","last_fullsized":"2024-04-26T16:27:36.143419+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"284.1e","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.143741+0000","last_change":"2024-04-26T16:27:36.143741+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.143741+0000","last_undegraded":"2024-04-26T16:27:36.143741+0000","last_fullsized":"2024-04-26T16:27:36.143741+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"284.1d","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.141172+0000","last_change":"2024-04-26T16:27:36.141172+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.141172+0000","last_undegraded":"2024-04-26T16:27:36.141172+0000","last_fullsized":"2024-04-26T16:27:36.141172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"284.1c","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.142854+0000","last_change":"2024-04-26T16:27:36.142854+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.142854+0000","last_undegraded":"2024-04-26T16:27:36.142854+0000","last_fullsized":"2024-04-26T16:27:36.142854+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"284.1b","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.133498+0000","last_change":"2024-04-26T16:27:36.133498+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.133498+0000","last_undegraded":"2024-04-26T16:27:36.133498+0000","last_fullsized":"2024-04-26T16:27:36.133498+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.1a","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.146074+0000","last_change":"2024-04-26T16:27:36.146074+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.146074+0000","last_undegraded":"2024-04-26T16:27:36.146074+0000","last_fullsized":"2024-04-26T16:27:36.146074+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"284.19","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.138944+0000","last_change":"2024-04-26T16:27:36.138944+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.138944+0000","last_undegraded":"2024-04-26T16:27:36.138944+0000","last_fullsized":"2024-04-26T16:27:36.138944+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"284.18","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.146677+0000","last_change":"2024-04-26T16:27:36.146677+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.146677+0000","last_undegraded":"2024-04-26T16:27:36.146677+0000","last_fullsized":"2024-04-26T16:27:36.146677+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"284.17","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.142018+0000","last_change":"2024-04-26T16:27:36.142018+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.142018+0000","last_undegraded":"2024-04-26T16:27:36.142018+0000","last_fullsized":"2024-04-26T16:27:36.142018+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"284.16","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.134584+0000","last_change":"2024-04-26T16:27:36.134584+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.134584+0000","last_undegraded":"2024-04-26T16:27:36.134584+0000","last_fullsized":"2024-04-26T16:27:36.134584+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.15","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.144868+0000","last_change":"2024-04-26T16:27:36.144868+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.144868+0000","last_undegraded":"2024-04-26T16:27:36.144868+0000","last_fullsized":"2024-04-26T16:27:36.144868+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"284.14","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.133907+0000","last_change":"2024-04-26T16:27:36.133907+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.133907+0000","last_undegraded":"2024-04-26T16:27:36.133907+0000","last_fullsized":"2024-04-26T16:27:36.133907+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.13","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.145506+0000","last_change":"2024-04-26T16:27:36.145506+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.145506+0000","last_undegraded":"2024-04-26T16:27:36.145506+0000","last_fullsized":"2024-04-26T16:27:36.145506+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"284.12","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.140620+0000","last_change":"2024-04-26T16:27:36.140620+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.140620+0000","last_undegraded":"2024-04-26T16:27:36.140620+0000","last_fullsized":"2024-04-26T16:27:36.140620+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"284.11","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.139518+0000","last_change":"2024-04-26T16:27:36.139518+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.139518+0000","last_undegraded":"2024-04-26T16:27:36.139518+0000","last_fullsized":"2024-04-26T16:27:36.139518+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"284.10","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.139762+0000","last_change":"2024-04-26T16:27:36.139762+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.139762+0000","last_undegraded":"2024-04-26T16:27:36.139762+0000","last_fullsized":"2024-04-26T16:27:36.139762+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"284.f","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.136429+0000","last_change":"2024-04-26T16:27:36.136429+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.136429+0000","last_undegraded":"2024-04-26T16:27:36.136429+0000","last_fullsized":"2024-04-26T16:27:36.136429+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.e","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.145455+0000","last_change":"2024-04-26T16:27:36.145455+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.145455+0000","last_undegraded":"2024-04-26T16:27:36.145455+0000","last_fullsized":"2024-04-26T16:27:36.145455+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"284.d","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.143882+0000","last_change":"2024-04-26T16:27:36.143882+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.143882+0000","last_undegraded":"2024-04-26T16:27:36.143882+0000","last_fullsized":"2024-04-26T16:27:36.143882+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"284.c","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.142380+0000","last_change":"2024-04-26T16:27:36.142380+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.142380+0000","last_undegraded":"2024-04-26T16:27:36.142380+0000","last_fullsized":"2024-04-26T16:27:36.142380+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"284.b","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.132873+0000","last_change":"2024-04-26T16:27:36.132873+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.132873+0000","last_undegraded":"2024-04-26T16:27:36.132873+0000","last_fullsized":"2024-04-26T16:27:36.132873+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.a","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.147052+0000","last_change":"2024-04-26T16:27:36.147052+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.147052+0000","last_undegraded":"2024-04-26T16:27:36.147052+0000","last_fullsized":"2024-04-26T16:27:36.147052+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"284.9","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.142261+0000","last_change":"2024-04-26T16:27:36.142261+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.142261+0000","last_undegraded":"2024-04-26T16:27:36.142261+0000","last_fullsized":"2024-04-26T16:27:36.142261+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"284.8","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.132004+0000","last_change":"2024-04-26T16:27:36.132004+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.132004+0000","last_undegraded":"2024-04-26T16:27:36.132004+0000","last_fullsized":"2024-04-26T16:27:36.132004+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.4","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.133064+0000","last_change":"2024-04-26T16:27:36.133064+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.133064+0000","last_undegraded":"2024-04-26T16:27:36.133064+0000","last_fullsized":"2024-04-26T16:27:36.133064+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"284.5","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.143815+0000","last_change":"2024-04-26T16:27:36.143815+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.143815+0000","last_undegraded":"2024-04-26T16:27:36.143815+0000","last_fullsized":"2024-04-26T16:27:36.143815+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"284.6","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.144443+0000","last_change":"2024-04-26T16:27:36.144443+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.144443+0000","last_undegraded":"2024-04-26T16:27:36.144443+0000","last_fullsized":"2024-04-26T16:27:36.144443+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"284.7","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.148234+0000","last_change":"2024-04-26T16:27:36.148234+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.148234+0000","last_undegraded":"2024-04-26T16:27:36.148234+0000","last_fullsized":"2024-04-26T16:27:36.148234+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"284.0","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.138568+0000","last_change":"2024-04-26T16:27:36.138568+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.138568+0000","last_undegraded":"2024-04-26T16:27:36.138568+0000","last_fullsized":"2024-04-26T16:27:36.138568+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"284.1","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.140859+0000","last_change":"2024-04-26T16:27:36.140859+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.140859+0000","last_undegraded":"2024-04-26T16:27:36.140859+0000","last_fullsized":"2024-04-26T16:27:36.140859+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"284.2","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.140910+0000","last_change":"2024-04-26T16:27:36.140910+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.140910+0000","last_undegraded":"2024-04-26T16:27:36.140910+0000","last_fullsized":"2024-04-26T16:27:36.140910+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"284.3","version":"0'0","reported_seq":1,"reported_epoch":388,"state":"creating+peering","last_fresh":"2024-04-26T16:27:36.143447+0000","last_change":"2024-04-26T16:27:36.143447+0000","last_active":"2024-04-26T16:27:36.124631+0000","last_peered":"2024-04-26T16:27:36.124631+0000","last_clean":"2024-04-26T16:27:36.124631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:27:36.143447+0000","last_undegraded":"2024-04-26T16:27:36.143447+0000","last_fullsized":"2024-04-26T16:27:36.143447+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":388,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.124631+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.124631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.12","version":"224'85","reported_seq":691,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.136062+0000","last_change":"2024-04-26T16:26:38.150277+0000","last_active":"2024-04-26T16:27:36.136062+0000","last_peered":"2024-04-26T16:27:36.136062+0000","last_clean":"2024-04-26T16:27:36.136062+0000","last_became_active":"2024-04-26T16:23:44.188737+0000","last_became_peered":"2024-04-26T16:23:44.188737+0000","last_unstale":"2024-04-26T16:27:36.136062+0000","last_undegraded":"2024-04-26T16:27:36.136062+0000","last_fullsized":"2024-04-26T16:27:36.136062+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'85","last_scrub_stamp":"2024-04-26T16:26:34.213819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:34.213819+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:36.361802+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0085611379999999994,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.13","version":"239'51","reported_seq":484,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138777+0000","last_change":"2024-04-26T16:26:38.125868+0000","last_active":"2024-04-26T16:27:36.138777+0000","last_peered":"2024-04-26T16:27:36.138777+0000","last_clean":"2024-04-26T16:27:36.138777+0000","last_became_active":"2024-04-26T16:26:23.192498+0000","last_became_peered":"2024-04-26T16:26:23.192498+0000","last_unstale":"2024-04-26T16:27:36.138777+0000","last_undegraded":"2024-04-26T16:27:36.138777+0000","last_fullsized":"2024-04-26T16:27:36.138777+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'51","last_scrub_stamp":"2024-04-26T16:26:04.067083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:04.067083+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026529510000000002,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.10","version":"275'119","reported_seq":806,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147733+0000","last_change":"2024-04-26T16:26:38.160490+0000","last_active":"2024-04-26T16:27:36.147733+0000","last_peered":"2024-04-26T16:27:36.147733+0000","last_clean":"2024-04-26T16:27:36.147733+0000","last_became_active":"2024-04-26T16:26:26.135726+0000","last_became_peered":"2024-04-26T16:26:26.135726+0000","last_unstale":"2024-04-26T16:27:36.147733+0000","last_undegraded":"2024-04-26T16:27:36.147733+0000","last_fullsized":"2024-04-26T16:27:36.147733+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024804152999999999,"stat_sum":{"num_bytes":77,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":0,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":77,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.11","version":"236'34","reported_seq":657,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.137805+0000","last_change":"2024-04-26T16:26:38.136831+0000","last_active":"2024-04-26T16:27:36.137805+0000","last_peered":"2024-04-26T16:27:36.137805+0000","last_clean":"2024-04-26T16:27:36.137805+0000","last_became_active":"2024-04-26T16:23:44.102085+0000","last_became_peered":"2024-04-26T16:23:44.102085+0000","last_unstale":"2024-04-26T16:27:36.137805+0000","last_undegraded":"2024-04-26T16:27:36.137805+0000","last_fullsized":"2024-04-26T16:27:36.137805+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:26:31.667610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:31.667610+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:55.874354+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.018897151000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.16","version":"268'73","reported_seq":599,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149477+0000","last_change":"2024-04-26T16:26:38.151992+0000","last_active":"2024-04-26T16:27:36.149477+0000","last_peered":"2024-04-26T16:27:36.149477+0000","last_clean":"2024-04-26T16:27:36.149477+0000","last_became_active":"2024-04-26T16:26:23.214752+0000","last_became_peered":"2024-04-26T16:26:23.214752+0000","last_unstale":"2024-04-26T16:27:36.149477+0000","last_undegraded":"2024-04-26T16:27:36.149477+0000","last_fullsized":"2024-04-26T16:27:36.149477+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'73","last_scrub_stamp":"2024-04-26T16:26:18.096044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:18.096044+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0052584900000000002,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.17","version":"287'47","reported_seq":718,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.153911+0000","last_change":"2024-04-26T16:27:34.197788+0000","last_active":"2024-04-26T16:27:36.153911+0000","last_peered":"2024-04-26T16:27:36.153911+0000","last_clean":"2024-04-26T16:27:36.153911+0000","last_became_active":"2024-04-26T16:26:23.156799+0000","last_became_peered":"2024-04-26T16:26:23.156799+0000","last_unstale":"2024-04-26T16:27:36.153911+0000","last_undegraded":"2024-04-26T16:27:36.153911+0000","last_fullsized":"2024-04-26T16:27:36.153911+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'47","last_scrub_stamp":"2024-04-26T16:27:34.197557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.197557+0000","objects_scrubbed":6,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:51.932201+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.002099615,"stat_sum":{"num_bytes":45,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":45,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.14","version":"319'36","reported_seq":654,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149511+0000","last_change":"2024-04-26T16:26:38.144501+0000","last_active":"2024-04-26T16:27:36.149511+0000","last_peered":"2024-04-26T16:27:36.149511+0000","last_clean":"2024-04-26T16:27:36.149511+0000","last_became_active":"2024-04-26T16:23:44.119110+0000","last_became_peered":"2024-04-26T16:23:44.119110+0000","last_unstale":"2024-04-26T16:27:36.149511+0000","last_undegraded":"2024-04-26T16:27:36.149511+0000","last_fullsized":"2024-04-26T16:27:36.149511+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'36","last_scrub_stamp":"2024-04-26T16:26:27.695843+0000","last_deep_scrub":"319'36","last_deep_scrub_stamp":"2024-04-26T16:26:27.695843+0000","last_clean_scrub_stamp":"2024-04-26T16:26:27.695843+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:56.992825+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.014694039000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.15","version":"264'73","reported_seq":690,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138204+0000","last_change":"2024-04-26T16:26:38.128801+0000","last_active":"2024-04-26T16:27:36.138204+0000","last_peered":"2024-04-26T16:27:36.138204+0000","last_clean":"2024-04-26T16:27:36.138204+0000","last_became_active":"2024-04-26T16:23:44.233737+0000","last_became_peered":"2024-04-26T16:23:44.233737+0000","last_unstale":"2024-04-26T16:27:36.138204+0000","last_undegraded":"2024-04-26T16:27:36.138204+0000","last_fullsized":"2024-04-26T16:27:36.138204+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'73","last_scrub_stamp":"2024-04-26T16:26:16.410746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:16.410746+0000","objects_scrubbed":5,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:40.500878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013513679999999999,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1a","version":"322'90","reported_seq":761,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141385+0000","last_change":"2024-04-26T16:26:38.123842+0000","last_active":"2024-04-26T16:27:36.141385+0000","last_peered":"2024-04-26T16:27:36.141385+0000","last_clean":"2024-04-26T16:27:36.141385+0000","last_became_active":"2024-04-26T16:26:23.171828+0000","last_became_peered":"2024-04-26T16:26:23.171828+0000","last_unstale":"2024-04-26T16:27:36.141385+0000","last_undegraded":"2024-04-26T16:27:36.141385+0000","last_fullsized":"2024-04-26T16:27:36.141385+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'90","last_scrub_stamp":"2024-04-26T16:26:30.427683+0000","last_deep_scrub":"322'90","last_deep_scrub_stamp":"2024-04-26T16:26:30.427683+0000","last_clean_scrub_stamp":"2024-04-26T16:26:30.427683+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:59.034156+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0015720319999999999,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1b","version":"338'23","reported_seq":666,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138893+0000","last_change":"2024-04-26T16:26:38.126326+0000","last_active":"2024-04-26T16:27:36.138893+0000","last_peered":"2024-04-26T16:27:36.138893+0000","last_clean":"2024-04-26T16:27:36.138893+0000","last_became_active":"2024-04-26T16:26:23.171002+0000","last_became_peered":"2024-04-26T16:26:23.171002+0000","last_unstale":"2024-04-26T16:27:36.138893+0000","last_undegraded":"2024-04-26T16:27:36.138893+0000","last_fullsized":"2024-04-26T16:27:36.138893+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"253'18","last_scrub_stamp":"2024-04-26T16:26:14.436301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:14.436301+0000","objects_scrubbed":2,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002858864,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.18","version":"330'42","reported_seq":691,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141817+0000","last_change":"2024-04-26T16:26:38.162503+0000","last_active":"2024-04-26T16:27:36.141817+0000","last_peered":"2024-04-26T16:27:36.141817+0000","last_clean":"2024-04-26T16:27:36.141817+0000","last_became_active":"2024-04-26T16:26:23.188399+0000","last_became_peered":"2024-04-26T16:26:23.188399+0000","last_unstale":"2024-04-26T16:27:36.141817+0000","last_undegraded":"2024-04-26T16:27:36.141817+0000","last_fullsized":"2024-04-26T16:27:36.141817+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:26:05.410097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:05.410097+0000","objects_scrubbed":2,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.033472043,"stat_sum":{"num_bytes":24,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.19","version":"308'35","reported_seq":656,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144168+0000","last_change":"2024-04-26T16:26:38.136334+0000","last_active":"2024-04-26T16:27:36.144168+0000","last_peered":"2024-04-26T16:27:36.144168+0000","last_clean":"2024-04-26T16:27:36.144168+0000","last_became_active":"2024-04-26T16:23:44.267057+0000","last_became_peered":"2024-04-26T16:23:44.267057+0000","last_unstale":"2024-04-26T16:27:36.144168+0000","last_undegraded":"2024-04-26T16:27:36.144168+0000","last_fullsized":"2024-04-26T16:27:36.144168+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:26:06.373321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:06.373321+0000","objects_scrubbed":2,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064511159999999998,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.1e","version":"263'19","reported_seq":645,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144414+0000","last_change":"2024-04-26T16:26:38.136518+0000","last_active":"2024-04-26T16:27:36.144414+0000","last_peered":"2024-04-26T16:27:36.144414+0000","last_clean":"2024-04-26T16:27:36.144414+0000","last_became_active":"2024-04-26T16:23:44.248875+0000","last_became_peered":"2024-04-26T16:23:44.248875+0000","last_unstale":"2024-04-26T16:27:36.144414+0000","last_undegraded":"2024-04-26T16:27:36.144414+0000","last_fullsized":"2024-04-26T16:27:36.144414+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-04-26T16:26:15.436153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:15.436153+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:44.145978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.006603205,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1f","version":"224'17","reported_seq":641,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149459+0000","last_change":"2024-04-26T16:26:38.152444+0000","last_active":"2024-04-26T16:27:36.149459+0000","last_peered":"2024-04-26T16:27:36.149459+0000","last_clean":"2024-04-26T16:27:36.149459+0000","last_became_active":"2024-04-26T16:23:44.180651+0000","last_became_peered":"2024-04-26T16:23:44.180651+0000","last_unstale":"2024-04-26T16:27:36.149459+0000","last_undegraded":"2024-04-26T16:27:36.149459+0000","last_fullsized":"2024-04-26T16:27:36.149459+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'17","last_scrub_stamp":"2024-04-26T16:26:16.158784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:16.158784+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002771796,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1c","version":"287'56","reported_seq":719,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149523+0000","last_change":"2024-04-26T16:26:38.168680+0000","last_active":"2024-04-26T16:27:36.149523+0000","last_peered":"2024-04-26T16:27:36.149523+0000","last_clean":"2024-04-26T16:27:36.149523+0000","last_became_active":"2024-04-26T16:26:23.196116+0000","last_became_peered":"2024-04-26T16:26:23.196116+0000","last_unstale":"2024-04-26T16:27:36.149523+0000","last_undegraded":"2024-04-26T16:27:36.149523+0000","last_fullsized":"2024-04-26T16:27:36.149523+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-04-26T16:26:03.412212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:03.412212+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0038346769999999999,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.1d","version":"279'136","reported_seq":720,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149532+0000","last_change":"2024-04-26T16:26:38.163555+0000","last_active":"2024-04-26T16:27:36.149532+0000","last_peered":"2024-04-26T16:27:36.149532+0000","last_clean":"2024-04-26T16:27:36.149532+0000","last_became_active":"2024-04-26T16:23:44.112550+0000","last_became_peered":"2024-04-26T16:23:44.112550+0000","last_unstale":"2024-04-26T16:27:36.149532+0000","last_undegraded":"2024-04-26T16:27:36.149532+0000","last_fullsized":"2024-04-26T16:27:36.149532+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024828301000000001,"stat_sum":{"num_bytes":88,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":40,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.2","version":"318'90","reported_seq":705,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.156815+0000","last_change":"2024-04-26T16:26:38.131695+0000","last_active":"2024-04-26T16:27:36.156815+0000","last_peered":"2024-04-26T16:27:36.156815+0000","last_clean":"2024-04-26T16:27:36.156815+0000","last_became_active":"2024-04-26T16:23:44.121137+0000","last_became_peered":"2024-04-26T16:23:44.121137+0000","last_unstale":"2024-04-26T16:27:36.156815+0000","last_undegraded":"2024-04-26T16:27:36.156815+0000","last_fullsized":"2024-04-26T16:27:36.156815+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'90","last_scrub_stamp":"2024-04-26T16:26:26.686096+0000","last_deep_scrub":"318'90","last_deep_scrub_stamp":"2024-04-26T16:26:26.686096+0000","last_clean_scrub_stamp":"2024-04-26T16:26:26.686096+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.007561318,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.3","version":"259'51","reported_seq":709,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.158159+0000","last_change":"2024-04-26T16:27:34.949071+0000","last_active":"2024-04-26T16:27:36.158159+0000","last_peered":"2024-04-26T16:27:36.158159+0000","last_clean":"2024-04-26T16:27:36.158159+0000","last_became_active":"2024-04-26T16:26:23.176098+0000","last_became_peered":"2024-04-26T16:26:23.176098+0000","last_unstale":"2024-04-26T16:27:36.158159+0000","last_undegraded":"2024-04-26T16:27:36.158159+0000","last_fullsized":"2024-04-26T16:27:36.158159+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'51","last_scrub_stamp":"2024-04-26T16:27:34.948809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.948809+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:54.253877+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.023234306999999999,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.0","version":"224'73","reported_seq":683,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.133340+0000","last_change":"2024-04-26T16:26:38.149911+0000","last_active":"2024-04-26T16:27:36.133340+0000","last_peered":"2024-04-26T16:27:36.133340+0000","last_clean":"2024-04-26T16:27:36.133340+0000","last_became_active":"2024-04-26T16:23:44.226589+0000","last_became_peered":"2024-04-26T16:23:44.226589+0000","last_unstale":"2024-04-26T16:27:36.133340+0000","last_undegraded":"2024-04-26T16:27:36.133340+0000","last_fullsized":"2024-04-26T16:27:36.133340+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'73","last_scrub_stamp":"2024-04-26T16:26:22.166037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:22.166037+0000","objects_scrubbed":4,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0071824289999999997,"stat_sum":{"num_bytes":44,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":22,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1","version":"281'102","reported_seq":716,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.156391+0000","last_change":"2024-04-26T16:27:26.873218+0000","last_active":"2024-04-26T16:27:36.156391+0000","last_peered":"2024-04-26T16:27:36.156391+0000","last_clean":"2024-04-26T16:27:36.156391+0000","last_became_active":"2024-04-26T16:23:44.117780+0000","last_became_peered":"2024-04-26T16:23:44.117780+0000","last_unstale":"2024-04-26T16:27:36.156391+0000","last_undegraded":"2024-04-26T16:27:36.156391+0000","last_fullsized":"2024-04-26T16:27:36.156391+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'102","last_scrub_stamp":"2024-04-26T16:27:26.872968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:26.872968+0000","objects_scrubbed":6,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:55.980646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.024877083000000001,"stat_sum":{"num_bytes":66,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.6","version":"346'67","reported_seq":691,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144288+0000","last_change":"2024-04-26T16:26:38.136201+0000","last_active":"2024-04-26T16:27:36.144288+0000","last_peered":"2024-04-26T16:27:36.144288+0000","last_clean":"2024-04-26T16:27:36.144288+0000","last_became_active":"2024-04-26T16:23:44.248476+0000","last_became_peered":"2024-04-26T16:23:44.248476+0000","last_unstale":"2024-04-26T16:27:36.144288+0000","last_undegraded":"2024-04-26T16:27:36.144288+0000","last_fullsized":"2024-04-26T16:27:36.144288+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'66","last_scrub_stamp":"2024-04-26T16:26:11.394935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:11.394935+0000","objects_scrubbed":5,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0065184939999999997,"stat_sum":{"num_bytes":4000051,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":22,"num_write_kb":3927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.7","version":"265'57","reported_seq":674,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166423+0000","last_change":"2024-04-26T16:26:38.167070+0000","last_active":"2024-04-26T16:27:36.166423+0000","last_peered":"2024-04-26T16:27:36.166423+0000","last_clean":"2024-04-26T16:27:36.166423+0000","last_became_active":"2024-04-26T16:23:44.200194+0000","last_became_peered":"2024-04-26T16:23:44.200194+0000","last_unstale":"2024-04-26T16:27:36.166423+0000","last_undegraded":"2024-04-26T16:27:36.166423+0000","last_fullsized":"2024-04-26T16:27:36.166423+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:23:43.006442+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018182487000000001,"stat_sum":{"num_bytes":42,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":18,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.4","version":"249'85","reported_seq":701,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140811+0000","last_change":"2024-04-26T16:26:38.144646+0000","last_active":"2024-04-26T16:27:36.140811+0000","last_peered":"2024-04-26T16:27:36.140811+0000","last_clean":"2024-04-26T16:27:36.140811+0000","last_became_active":"2024-04-26T16:23:44.220715+0000","last_became_peered":"2024-04-26T16:23:44.220715+0000","last_unstale":"2024-04-26T16:27:36.140811+0000","last_undegraded":"2024-04-26T16:27:36.140811+0000","last_fullsized":"2024-04-26T16:27:36.140811+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'85","last_scrub_stamp":"2024-04-26T16:26:34.699219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:34.699219+0000","objects_scrubbed":5,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:48.510754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064610860000000004,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.5","version":"243'34","reported_seq":656,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167601+0000","last_change":"2024-04-26T16:26:38.146862+0000","last_active":"2024-04-26T16:27:36.167601+0000","last_peered":"2024-04-26T16:27:36.167601+0000","last_clean":"2024-04-26T16:27:36.167601+0000","last_became_active":"2024-04-26T16:23:44.116978+0000","last_became_peered":"2024-04-26T16:23:44.116978+0000","last_unstale":"2024-04-26T16:27:36.167601+0000","last_undegraded":"2024-04-26T16:27:36.167601+0000","last_fullsized":"2024-04-26T16:27:36.167601+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'34","last_scrub_stamp":"2024-04-26T16:26:11.793261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:11.793261+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015720004999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.a","version":"268'39","reported_seq":688,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.159041+0000","last_change":"2024-04-26T16:26:38.140234+0000","last_active":"2024-04-26T16:27:36.159041+0000","last_peered":"2024-04-26T16:27:36.159041+0000","last_clean":"2024-04-26T16:27:36.159041+0000","last_became_active":"2024-04-26T16:26:27.141171+0000","last_became_peered":"2024-04-26T16:26:27.141171+0000","last_unstale":"2024-04-26T16:27:36.159041+0000","last_undegraded":"2024-04-26T16:27:36.159041+0000","last_fullsized":"2024-04-26T16:27:36.159041+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'39","last_scrub_stamp":"2024-04-26T16:26:15.788377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:15.788377+0000","objects_scrubbed":2,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010486410999999999,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.b","version":"236'34","reported_seq":660,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141749+0000","last_change":"2024-04-26T16:26:38.140488+0000","last_active":"2024-04-26T16:27:36.141749+0000","last_peered":"2024-04-26T16:27:36.141749+0000","last_clean":"2024-04-26T16:27:36.141749+0000","last_became_active":"2024-04-26T16:23:44.081905+0000","last_became_peered":"2024-04-26T16:23:44.081905+0000","last_unstale":"2024-04-26T16:27:36.141749+0000","last_undegraded":"2024-04-26T16:27:36.141749+0000","last_fullsized":"2024-04-26T16:27:36.141749+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:26:32.710503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:32.710503+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:53.106623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0069674610000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.8","version":"316'54","reported_seq":712,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140210+0000","last_change":"2024-04-26T16:26:38.120788+0000","last_active":"2024-04-26T16:27:36.140210+0000","last_peered":"2024-04-26T16:27:36.140210+0000","last_clean":"2024-04-26T16:27:36.140210+0000","last_became_active":"2024-04-26T16:26:23.201975+0000","last_became_peered":"2024-04-26T16:26:23.201975+0000","last_unstale":"2024-04-26T16:27:36.140210+0000","last_undegraded":"2024-04-26T16:27:36.140210+0000","last_fullsized":"2024-04-26T16:27:36.140210+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'52","last_scrub_stamp":"2024-04-26T16:26:09.434181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:09.434181+0000","objects_scrubbed":4,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001511334,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":18,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.9","version":"224'34","reported_seq":649,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.133194+0000","last_change":"2024-04-26T16:26:38.143255+0000","last_active":"2024-04-26T16:27:36.133194+0000","last_peered":"2024-04-26T16:27:36.133194+0000","last_clean":"2024-04-26T16:27:36.133194+0000","last_became_active":"2024-04-26T16:23:44.109291+0000","last_became_peered":"2024-04-26T16:23:44.109291+0000","last_unstale":"2024-04-26T16:27:36.133194+0000","last_undegraded":"2024-04-26T16:27:36.133194+0000","last_fullsized":"2024-04-26T16:27:36.133194+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:26:11.148889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:11.148889+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0019765540000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.e","version":"300'32","reported_seq":736,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149337+0000","last_change":"2024-04-26T16:26:38.151566+0000","last_active":"2024-04-26T16:27:36.149337+0000","last_peered":"2024-04-26T16:27:36.149337+0000","last_clean":"2024-04-26T16:27:36.149337+0000","last_became_active":"2024-04-26T16:26:23.214391+0000","last_became_peered":"2024-04-26T16:26:23.214391+0000","last_unstale":"2024-04-26T16:27:36.149337+0000","last_undegraded":"2024-04-26T16:27:36.149337+0000","last_fullsized":"2024-04-26T16:27:36.149337+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-04-26T16:26:32.205294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:32.205294+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:59.741681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005085521,"stat_sum":{"num_bytes":4194328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":12,"num_write_kb":8202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.f","version":"287'74","reported_seq":735,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.160361+0000","last_change":"2024-04-26T16:26:38.156172+0000","last_active":"2024-04-26T16:27:36.160361+0000","last_peered":"2024-04-26T16:27:36.160361+0000","last_clean":"2024-04-26T16:27:36.160361+0000","last_became_active":"2024-04-26T16:26:23.198449+0000","last_became_peered":"2024-04-26T16:26:23.198449+0000","last_unstale":"2024-04-26T16:27:36.160361+0000","last_undegraded":"2024-04-26T16:27:36.160361+0000","last_fullsized":"2024-04-26T16:27:36.160361+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'74","last_scrub_stamp":"2024-04-26T16:26:24.666860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:24.666860+0000","objects_scrubbed":6,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:50.156868+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.019139410999999999,"stat_sum":{"num_bytes":49,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.c","version":"235'34","reported_seq":654,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139758+0000","last_change":"2024-04-26T16:27:28.093941+0000","last_active":"2024-04-26T16:27:36.139758+0000","last_peered":"2024-04-26T16:27:36.139758+0000","last_clean":"2024-04-26T16:27:36.139758+0000","last_became_active":"2024-04-26T16:23:44.106600+0000","last_became_peered":"2024-04-26T16:23:44.106600+0000","last_unstale":"2024-04-26T16:27:36.139758+0000","last_undegraded":"2024-04-26T16:27:36.139758+0000","last_fullsized":"2024-04-26T16:27:36.139758+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'34","last_scrub_stamp":"2024-04-26T16:27:28.093587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.093587+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:41.120953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.006139818,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.d","version":"236'17","reported_seq":431,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149068+0000","last_change":"2024-04-26T16:26:38.152585+0000","last_active":"2024-04-26T16:27:36.149068+0000","last_peered":"2024-04-26T16:27:36.149068+0000","last_clean":"2024-04-26T16:27:36.149068+0000","last_became_active":"2024-04-26T16:26:23.211900+0000","last_became_peered":"2024-04-26T16:26:23.211900+0000","last_unstale":"2024-04-26T16:27:36.149068+0000","last_undegraded":"2024-04-26T16:27:36.149068+0000","last_fullsized":"2024-04-26T16:27:36.149068+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-04-26T16:26:12.147632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:26:12.147632+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055977359999999999,"stat_sum":{"num_bytes":11,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":158,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.145368+0000","last_change":"2024-04-26T16:26:23.194547+0000","last_active":"2024-04-26T16:27:36.145368+0000","last_peered":"2024-04-26T16:27:36.145368+0000","last_clean":"2024-04-26T16:27:36.145368+0000","last_became_active":"2024-04-26T16:26:23.194196+0000","last_became_peered":"2024-04-26T16:26:23.194196+0000","last_unstale":"2024-04-26T16:27:36.145368+0000","last_undegraded":"2024-04-26T16:27:36.145368+0000","last_fullsized":"2024-04-26T16:27:36.145368+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:11.145336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:11.145336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"109'12","reported_seq":788,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.155752+0000","last_change":"2024-04-26T16:26:13.003930+0000","last_active":"2024-04-26T16:27:36.155752+0000","last_peered":"2024-04-26T16:27:36.155752+0000","last_clean":"2024-04-26T16:27:36.155752+0000","last_became_active":"2024-04-26T16:17:33.070525+0000","last_became_peered":"2024-04-26T16:17:33.070525+0000","last_unstale":"2024-04-26T16:27:36.155752+0000","last_undegraded":"2024-04-26T16:27:36.155752+0000","last_fullsized":"2024-04-26T16:27:36.155752+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'12","last_scrub_stamp":"2024-04-26T16:26:13.003820+0000","last_deep_scrub":"109'12","last_deep_scrub_stamp":"2024-04-26T16:19:17.690568+0000","last_clean_scrub_stamp":"2024-04-26T16:26:13.003820+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"86'15","reported_seq":706,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167167+0000","last_change":"2024-04-26T16:26:27.167313+0000","last_active":"2024-04-26T16:27:36.167167+0000","last_peered":"2024-04-26T16:27:36.167167+0000","last_clean":"2024-04-26T16:27:36.167167+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:27:36.167167+0000","last_undegraded":"2024-04-26T16:27:36.167167+0000","last_fullsized":"2024-04-26T16:27:36.167167+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:24:59.587812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.587812+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":668,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144722+0000","last_change":"2024-04-26T16:24:45.121653+0000","last_active":"2024-04-26T16:27:36.144722+0000","last_peered":"2024-04-26T16:27:36.144722+0000","last_clean":"2024-04-26T16:27:36.144722+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:27:36.144722+0000","last_undegraded":"2024-04-26T16:27:36.144722+0000","last_fullsized":"2024-04-26T16:27:36.144722+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:45.121528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.121528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":167,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148306+0000","last_change":"2024-04-26T16:26:39.807438+0000","last_active":"2024-04-26T16:27:36.148306+0000","last_peered":"2024-04-26T16:27:36.148306+0000","last_clean":"2024-04-26T16:27:36.148306+0000","last_became_active":"2024-04-26T16:26:23.174034+0000","last_became_peered":"2024-04-26T16:26:23.174034+0000","last_unstale":"2024-04-26T16:27:36.148306+0000","last_undegraded":"2024-04-26T16:27:36.148306+0000","last_fullsized":"2024-04-26T16:27:36.148306+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:39.807345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:39.807345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:46.958702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":514,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.160551+0000","last_change":"2024-04-26T16:27:28.909971+0000","last_active":"2024-04-26T16:27:36.160551+0000","last_peered":"2024-04-26T16:27:36.160551+0000","last_clean":"2024-04-26T16:27:36.160551+0000","last_became_active":"2024-04-26T16:23:29.349198+0000","last_became_peered":"2024-04-26T16:23:29.349198+0000","last_unstale":"2024-04-26T16:27:36.160551+0000","last_undegraded":"2024-04-26T16:27:36.160551+0000","last_fullsized":"2024-04-26T16:27:36.160551+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:28.909860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.909860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:40.224518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"86'10","reported_seq":201,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148451+0000","last_change":"2024-04-26T16:26:23.242678+0000","last_active":"2024-04-26T16:27:36.148451+0000","last_peered":"2024-04-26T16:27:36.148451+0000","last_clean":"2024-04-26T16:27:36.148451+0000","last_became_active":"2024-04-26T16:26:23.198669+0000","last_became_peered":"2024-04-26T16:26:23.198669+0000","last_unstale":"2024-04-26T16:27:36.148451+0000","last_undegraded":"2024-04-26T16:27:36.148451+0000","last_fullsized":"2024-04-26T16:27:36.148451+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:24:52.466192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:52.466192+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":673,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140213+0000","last_change":"2024-04-26T16:27:19.964932+0000","last_active":"2024-04-26T16:27:36.140213+0000","last_peered":"2024-04-26T16:27:36.140213+0000","last_clean":"2024-04-26T16:27:36.140213+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:27:36.140213+0000","last_undegraded":"2024-04-26T16:27:36.140213+0000","last_fullsized":"2024-04-26T16:27:36.140213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:19.964811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:27:19.964811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:32.704929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":668,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141504+0000","last_change":"2024-04-26T16:26:33.398507+0000","last_active":"2024-04-26T16:27:36.141504+0000","last_peered":"2024-04-26T16:27:36.141504+0000","last_clean":"2024-04-26T16:27:36.141504+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:27:36.141504+0000","last_undegraded":"2024-04-26T16:27:36.141504+0000","last_fullsized":"2024-04-26T16:27:36.141504+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:33.398385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:33.398385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:52.297087+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":664,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.146976+0000","last_change":"2024-04-26T16:26:31.502895+0000","last_active":"2024-04-26T16:27:36.146976+0000","last_peered":"2024-04-26T16:27:36.146976+0000","last_clean":"2024-04-26T16:27:36.146976+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:27:36.146976+0000","last_undegraded":"2024-04-26T16:27:36.146976+0000","last_fullsized":"2024-04-26T16:27:36.146976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:31.502664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:31.502664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:40.381837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":520,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166515+0000","last_change":"2024-04-26T16:26:35.687821+0000","last_active":"2024-04-26T16:27:36.166515+0000","last_peered":"2024-04-26T16:27:36.166515+0000","last_clean":"2024-04-26T16:27:36.166515+0000","last_became_active":"2024-04-26T16:26:23.154239+0000","last_became_peered":"2024-04-26T16:26:23.154239+0000","last_unstale":"2024-04-26T16:27:36.166515+0000","last_undegraded":"2024-04-26T16:27:36.166515+0000","last_fullsized":"2024-04-26T16:27:36.166515+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:35.687528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.687528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:02.753831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"388'19","reported_seq":714,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.322894+0000","last_change":"2024-04-26T16:26:23.235907+0000","last_active":"2024-04-26T16:27:36.322894+0000","last_peered":"2024-04-26T16:27:36.322894+0000","last_clean":"2024-04-26T16:27:36.322894+0000","last_became_active":"2024-04-26T16:26:23.154819+0000","last_became_peered":"2024-04-26T16:26:23.154819+0000","last_unstale":"2024-04-26T16:27:36.322894+0000","last_undegraded":"2024-04-26T16:27:36.322894+0000","last_fullsized":"2024-04-26T16:27:36.322894+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'17","last_scrub_stamp":"2024-04-26T16:24:47.710551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:47.710551+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":682,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.142867+0000","last_change":"2024-04-26T16:26:32.424891+0000","last_active":"2024-04-26T16:27:36.142867+0000","last_peered":"2024-04-26T16:27:36.142867+0000","last_clean":"2024-04-26T16:27:36.142867+0000","last_became_active":"2024-04-26T16:26:23.201706+0000","last_became_peered":"2024-04-26T16:26:23.201706+0000","last_unstale":"2024-04-26T16:27:36.142867+0000","last_undegraded":"2024-04-26T16:27:36.142867+0000","last_fullsized":"2024-04-26T16:27:36.142867+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:32.424634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:32.424634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:49.760477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":674,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139263+0000","last_change":"2024-04-26T16:27:25.398012+0000","last_active":"2024-04-26T16:27:36.139263+0000","last_peered":"2024-04-26T16:27:36.139263+0000","last_clean":"2024-04-26T16:27:36.139263+0000","last_became_active":"2024-04-26T16:26:23.172602+0000","last_became_peered":"2024-04-26T16:26:23.172602+0000","last_unstale":"2024-04-26T16:27:36.139263+0000","last_undegraded":"2024-04-26T16:27:36.139263+0000","last_fullsized":"2024-04-26T16:27:36.139263+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:25.397802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.397802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:52.246961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":651,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148185+0000","last_change":"2024-04-26T16:26:31.674001+0000","last_active":"2024-04-26T16:27:36.148185+0000","last_peered":"2024-04-26T16:27:36.148185+0000","last_clean":"2024-04-26T16:27:36.148185+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:27:36.148185+0000","last_undegraded":"2024-04-26T16:27:36.148185+0000","last_fullsized":"2024-04-26T16:27:36.148185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:31.673774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:26:31.673774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:59.008114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":511,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141961+0000","last_change":"2024-04-26T16:24:53.146189+0000","last_active":"2024-04-26T16:27:36.141961+0000","last_peered":"2024-04-26T16:27:36.141961+0000","last_clean":"2024-04-26T16:27:36.141961+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:27:36.141961+0000","last_undegraded":"2024-04-26T16:27:36.141961+0000","last_fullsized":"2024-04-26T16:27:36.141961+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:53.146054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:24:53.146054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"388'21","reported_seq":708,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.307015+0000","last_change":"2024-04-26T16:26:10.504732+0000","last_active":"2024-04-26T16:27:36.307015+0000","last_peered":"2024-04-26T16:27:36.307015+0000","last_clean":"2024-04-26T16:27:36.307015+0000","last_became_active":"2024-04-26T16:17:31.024436+0000","last_became_peered":"2024-04-26T16:17:31.024436+0000","last_unstale":"2024-04-26T16:27:36.307015+0000","last_undegraded":"2024-04-26T16:27:36.307015+0000","last_fullsized":"2024-04-26T16:27:36.307015+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'19","last_scrub_stamp":"2024-04-26T16:26:10.504536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:10.504536+0000","objects_scrubbed":9,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":669,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.159009+0000","last_change":"2024-04-26T16:26:23.199525+0000","last_active":"2024-04-26T16:27:36.159009+0000","last_peered":"2024-04-26T16:27:36.159009+0000","last_clean":"2024-04-26T16:27:36.159009+0000","last_became_active":"2024-04-26T16:26:23.199164+0000","last_became_peered":"2024-04-26T16:26:23.199164+0000","last_unstale":"2024-04-26T16:27:36.159009+0000","last_undegraded":"2024-04-26T16:27:36.159009+0000","last_fullsized":"2024-04-26T16:27:36.159009+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:11.955182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:26:11.955182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":665,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141390+0000","last_change":"2024-04-26T16:24:40.513071+0000","last_active":"2024-04-26T16:27:36.141390+0000","last_peered":"2024-04-26T16:27:36.141390+0000","last_clean":"2024-04-26T16:27:36.141390+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:27:36.141390+0000","last_undegraded":"2024-04-26T16:27:36.141390+0000","last_fullsized":"2024-04-26T16:27:36.141390+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:40.512788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:01.215041+0000","last_clean_scrub_stamp":"2024-04-26T16:24:40.512788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":664,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144149+0000","last_change":"2024-04-26T16:27:29.359154+0000","last_active":"2024-04-26T16:27:36.144149+0000","last_peered":"2024-04-26T16:27:36.144149+0000","last_clean":"2024-04-26T16:27:36.144149+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:27:36.144149+0000","last_undegraded":"2024-04-26T16:27:36.144149+0000","last_fullsized":"2024-04-26T16:27:36.144149+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:29.358979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.358979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:48.489538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":521,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.151497+0000","last_change":"2024-04-26T16:27:27.217110+0000","last_active":"2024-04-26T16:27:36.151497+0000","last_peered":"2024-04-26T16:27:36.151497+0000","last_clean":"2024-04-26T16:27:36.151497+0000","last_became_active":"2024-04-26T16:23:29.321850+0000","last_became_peered":"2024-04-26T16:23:29.321850+0000","last_unstale":"2024-04-26T16:27:36.151497+0000","last_undegraded":"2024-04-26T16:27:36.151497+0000","last_fullsized":"2024-04-26T16:27:36.151497+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:27.216979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:27.216979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:42.975125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"122'13","reported_seq":683,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.145058+0000","last_change":"2024-04-26T16:27:34.211956+0000","last_active":"2024-04-26T16:27:36.145058+0000","last_peered":"2024-04-26T16:27:36.145058+0000","last_clean":"2024-04-26T16:27:36.145058+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:27:36.145058+0000","last_undegraded":"2024-04-26T16:27:36.145058+0000","last_fullsized":"2024-04-26T16:27:36.145058+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'13","last_scrub_stamp":"2024-04-26T16:27:34.211856+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.211856+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:55.860723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":673,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139435+0000","last_change":"2024-04-26T16:26:02.467305+0000","last_active":"2024-04-26T16:27:36.139435+0000","last_peered":"2024-04-26T16:27:36.139435+0000","last_clean":"2024-04-26T16:27:36.139435+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:27:36.139435+0000","last_undegraded":"2024-04-26T16:27:36.139435+0000","last_fullsized":"2024-04-26T16:27:36.139435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:02.467019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:02.467019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"316'12","reported_seq":810,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138326+0000","last_change":"2024-04-26T16:27:28.318920+0000","last_active":"2024-04-26T16:27:36.138326+0000","last_peered":"2024-04-26T16:27:36.138326+0000","last_clean":"2024-04-26T16:27:36.138326+0000","last_became_active":"2024-04-26T16:26:23.194072+0000","last_became_peered":"2024-04-26T16:26:23.194072+0000","last_unstale":"2024-04-26T16:27:36.138326+0000","last_undegraded":"2024-04-26T16:27:36.138326+0000","last_fullsized":"2024-04-26T16:27:36.138326+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'12","last_scrub_stamp":"2024-04-26T16:27:28.318766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.318766+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:30.228977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.162304+0000","last_change":"2024-04-26T16:26:36.737737+0000","last_active":"2024-04-26T16:27:36.162304+0000","last_peered":"2024-04-26T16:27:36.162304+0000","last_clean":"2024-04-26T16:27:36.162304+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:27:36.162304+0000","last_undegraded":"2024-04-26T16:27:36.162304+0000","last_fullsized":"2024-04-26T16:27:36.162304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:36.737552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:26:36.737552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:53.289135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":515,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138114+0000","last_change":"2024-04-26T16:26:24.484959+0000","last_active":"2024-04-26T16:27:36.138114+0000","last_peered":"2024-04-26T16:27:36.138114+0000","last_clean":"2024-04-26T16:27:36.138114+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:27:36.138114+0000","last_undegraded":"2024-04-26T16:27:36.138114+0000","last_fullsized":"2024-04-26T16:27:36.138114+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:24.484611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:24.484611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"86'15","reported_seq":680,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.134019+0000","last_change":"2024-04-26T16:24:44.951341+0000","last_active":"2024-04-26T16:27:36.134019+0000","last_peered":"2024-04-26T16:27:36.134019+0000","last_clean":"2024-04-26T16:27:36.134019+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:27:36.134019+0000","last_undegraded":"2024-04-26T16:27:36.134019+0000","last_fullsized":"2024-04-26T16:27:36.134019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:24:44.951225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:44.951225+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138555+0000","last_change":"2024-04-26T16:24:39.549952+0000","last_active":"2024-04-26T16:27:36.138555+0000","last_peered":"2024-04-26T16:27:36.138555+0000","last_clean":"2024-04-26T16:27:36.138555+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:27:36.138555+0000","last_undegraded":"2024-04-26T16:27:36.138555+0000","last_fullsized":"2024-04-26T16:27:36.138555+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:39.549794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:39.549794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"316'13","reported_seq":808,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.161379+0000","last_change":"2024-04-26T16:27:33.956517+0000","last_active":"2024-04-26T16:27:36.161379+0000","last_peered":"2024-04-26T16:27:36.161379+0000","last_clean":"2024-04-26T16:27:36.161379+0000","last_became_active":"2024-04-26T16:26:23.180811+0000","last_became_peered":"2024-04-26T16:26:23.180811+0000","last_unstale":"2024-04-26T16:27:36.161379+0000","last_undegraded":"2024-04-26T16:27:36.161379+0000","last_fullsized":"2024-04-26T16:27:36.161379+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:27:33.956373+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:27:33.956373+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:50.202853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":159,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139881+0000","last_change":"2024-04-26T16:26:23.196457+0000","last_active":"2024-04-26T16:27:36.139881+0000","last_peered":"2024-04-26T16:27:36.139881+0000","last_clean":"2024-04-26T16:27:36.139881+0000","last_became_active":"2024-04-26T16:26:23.195438+0000","last_became_peered":"2024-04-26T16:26:23.195438+0000","last_unstale":"2024-04-26T16:27:36.139881+0000","last_undegraded":"2024-04-26T16:27:36.139881+0000","last_fullsized":"2024-04-26T16:27:36.139881+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:27.349541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:24:27.349541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":534,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147044+0000","last_change":"2024-04-26T16:27:29.219200+0000","last_active":"2024-04-26T16:27:36.147044+0000","last_peered":"2024-04-26T16:27:36.147044+0000","last_clean":"2024-04-26T16:27:36.147044+0000","last_became_active":"2024-04-26T16:23:29.334960+0000","last_became_peered":"2024-04-26T16:23:29.334960+0000","last_unstale":"2024-04-26T16:27:36.147044+0000","last_undegraded":"2024-04-26T16:27:36.147044+0000","last_fullsized":"2024-04-26T16:27:36.147044+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:27:29.219032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.219032+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:49.741947+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"86'12","reported_seq":684,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.155714+0000","last_change":"2024-04-26T16:24:50.816864+0000","last_active":"2024-04-26T16:27:36.155714+0000","last_peered":"2024-04-26T16:27:36.155714+0000","last_clean":"2024-04-26T16:27:36.155714+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:27:36.155714+0000","last_undegraded":"2024-04-26T16:27:36.155714+0000","last_fullsized":"2024-04-26T16:27:36.155714+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'12","last_scrub_stamp":"2024-04-26T16:24:50.816601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:50.816601+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":671,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.152010+0000","last_change":"2024-04-26T16:24:32.644198+0000","last_active":"2024-04-26T16:27:36.152010+0000","last_peered":"2024-04-26T16:27:36.152010+0000","last_clean":"2024-04-26T16:27:36.152010+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:27:36.152010+0000","last_undegraded":"2024-04-26T16:27:36.152010+0000","last_fullsized":"2024-04-26T16:27:36.152010+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:24:32.643939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:32.643939+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":166,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.162883+0000","last_change":"2024-04-26T16:27:35.971843+0000","last_active":"2024-04-26T16:27:36.162883+0000","last_peered":"2024-04-26T16:27:36.162883+0000","last_clean":"2024-04-26T16:27:36.162883+0000","last_became_active":"2024-04-26T16:26:23.175369+0000","last_became_peered":"2024-04-26T16:26:23.175369+0000","last_unstale":"2024-04-26T16:27:36.162883+0000","last_undegraded":"2024-04-26T16:27:36.162883+0000","last_fullsized":"2024-04-26T16:27:36.162883+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:35.971700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:27:35.971700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:29:04.347640+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148758+0000","last_change":"2024-04-26T16:27:35.199064+0000","last_active":"2024-04-26T16:27:36.148758+0000","last_peered":"2024-04-26T16:27:36.148758+0000","last_clean":"2024-04-26T16:27:36.148758+0000","last_became_active":"2024-04-26T16:26:23.194658+0000","last_became_peered":"2024-04-26T16:26:23.194658+0000","last_unstale":"2024-04-26T16:27:36.148758+0000","last_undegraded":"2024-04-26T16:27:36.148758+0000","last_fullsized":"2024-04-26T16:27:36.148758+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:35.198929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:27:35.198929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:43.801868+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":514,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.161679+0000","last_change":"2024-04-26T16:26:22.615808+0000","last_active":"2024-04-26T16:27:36.161679+0000","last_peered":"2024-04-26T16:27:36.161679+0000","last_clean":"2024-04-26T16:27:36.161679+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:27:36.161679+0000","last_undegraded":"2024-04-26T16:27:36.161679+0000","last_fullsized":"2024-04-26T16:27:36.161679+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:22.615523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:22.615523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:43.824100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":785,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.134287+0000","last_change":"2024-04-26T16:24:39.048494+0000","last_active":"2024-04-26T16:27:36.134287+0000","last_peered":"2024-04-26T16:27:36.134287+0000","last_clean":"2024-04-26T16:27:36.134287+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:27:36.134287+0000","last_undegraded":"2024-04-26T16:27:36.134287+0000","last_fullsized":"2024-04-26T16:27:36.134287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:24:39.048148+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:24:39.048148+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":680,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.156015+0000","last_change":"2024-04-26T16:26:23.171769+0000","last_active":"2024-04-26T16:27:36.156015+0000","last_peered":"2024-04-26T16:27:36.156015+0000","last_clean":"2024-04-26T16:27:36.156015+0000","last_became_active":"2024-04-26T16:26:23.171413+0000","last_became_peered":"2024-04-26T16:26:23.171413+0000","last_unstale":"2024-04-26T16:27:36.156015+0000","last_undegraded":"2024-04-26T16:27:36.156015+0000","last_fullsized":"2024-04-26T16:27:36.156015+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:07.779305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:13.552494+0000","last_clean_scrub_stamp":"2024-04-26T16:26:07.779305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'20","reported_seq":699,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.285203+0000","last_change":"2024-04-26T16:26:19.396581+0000","last_active":"2024-04-26T16:27:36.285203+0000","last_peered":"2024-04-26T16:27:36.285203+0000","last_clean":"2024-04-26T16:27:36.285203+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:27:36.285203+0000","last_undegraded":"2024-04-26T16:27:36.285203+0000","last_fullsized":"2024-04-26T16:27:36.285203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'18","last_scrub_stamp":"2024-04-26T16:26:19.396177+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:26:19.396177+0000","objects_scrubbed":6,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:45.869533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":21,"num_read_kb":15,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":678,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154461+0000","last_change":"2024-04-26T16:26:38.810638+0000","last_active":"2024-04-26T16:27:36.154461+0000","last_peered":"2024-04-26T16:27:36.154461+0000","last_clean":"2024-04-26T16:27:36.154461+0000","last_became_active":"2024-04-26T16:17:29.006985+0000","last_became_peered":"2024-04-26T16:17:29.006985+0000","last_unstale":"2024-04-26T16:27:36.154461+0000","last_undegraded":"2024-04-26T16:27:36.154461+0000","last_fullsized":"2024-04-26T16:27:36.154461+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:38.810545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:26:38.810545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:07.961472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":651,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.158603+0000","last_change":"2024-04-26T16:24:31.596973+0000","last_active":"2024-04-26T16:27:36.158603+0000","last_peered":"2024-04-26T16:27:36.158603+0000","last_clean":"2024-04-26T16:27:36.158603+0000","last_became_active":"2024-04-26T16:17:35.670672+0000","last_became_peered":"2024-04-26T16:17:35.670672+0000","last_unstale":"2024-04-26T16:27:36.158603+0000","last_undegraded":"2024-04-26T16:27:36.158603+0000","last_fullsized":"2024-04-26T16:27:36.158603+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:31.596755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:31.596755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":514,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167544+0000","last_change":"2024-04-26T16:26:23.786897+0000","last_active":"2024-04-26T16:27:36.167544+0000","last_peered":"2024-04-26T16:27:36.167544+0000","last_clean":"2024-04-26T16:27:36.167544+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:27:36.167544+0000","last_undegraded":"2024-04-26T16:27:36.167544+0000","last_fullsized":"2024-04-26T16:27:36.167544+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:23.786780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:23.786780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":665,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.151923+0000","last_change":"2024-04-26T16:27:30.898300+0000","last_active":"2024-04-26T16:27:36.151923+0000","last_peered":"2024-04-26T16:27:36.151923+0000","last_clean":"2024-04-26T16:27:36.151923+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:27:36.151923+0000","last_undegraded":"2024-04-26T16:27:36.151923+0000","last_fullsized":"2024-04-26T16:27:36.151923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:30.898191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:27:30.898191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:31.374107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":1258,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.361641+0000","last_change":"2024-04-26T16:27:25.065659+0000","last_active":"2024-04-26T16:27:36.361641+0000","last_peered":"2024-04-26T16:27:36.361641+0000","last_clean":"2024-04-26T16:27:36.361641+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:27:36.361641+0000","last_undegraded":"2024-04-26T16:27:36.361641+0000","last_fullsized":"2024-04-26T16:27:36.361641+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:27:25.065519+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.065519+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:49.572527+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":586,"num_read_kb":581,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'20","reported_seq":690,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.299586+0000","last_change":"2024-04-26T16:24:43.903168+0000","last_active":"2024-04-26T16:27:36.299586+0000","last_peered":"2024-04-26T16:27:36.299586+0000","last_clean":"2024-04-26T16:27:36.299586+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:27:36.299586+0000","last_undegraded":"2024-04-26T16:27:36.299586+0000","last_fullsized":"2024-04-26T16:27:36.299586+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'18","last_scrub_stamp":"2024-04-26T16:24:43.903026+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:24:43.903026+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":22,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":675,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141960+0000","last_change":"2024-04-26T16:26:22.428735+0000","last_active":"2024-04-26T16:27:36.141960+0000","last_peered":"2024-04-26T16:27:36.141960+0000","last_clean":"2024-04-26T16:27:36.141960+0000","last_became_active":"2024-04-26T16:17:29.004853+0000","last_became_peered":"2024-04-26T16:17:29.004853+0000","last_unstale":"2024-04-26T16:27:36.141960+0000","last_undegraded":"2024-04-26T16:27:36.141960+0000","last_fullsized":"2024-04-26T16:27:36.141960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:26:22.428532+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:26:22.428532+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.158790+0000","last_change":"2024-04-26T16:26:06.828440+0000","last_active":"2024-04-26T16:27:36.158790+0000","last_peered":"2024-04-26T16:27:36.158790+0000","last_clean":"2024-04-26T16:27:36.158790+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:27:36.158790+0000","last_undegraded":"2024-04-26T16:27:36.158790+0000","last_fullsized":"2024-04-26T16:27:36.158790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:06.828256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:26:06.828256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":514,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140938+0000","last_change":"2024-04-26T16:26:28.591807+0000","last_active":"2024-04-26T16:27:36.140938+0000","last_peered":"2024-04-26T16:27:36.140938+0000","last_clean":"2024-04-26T16:27:36.140938+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:27:36.140938+0000","last_undegraded":"2024-04-26T16:27:36.140938+0000","last_fullsized":"2024-04-26T16:27:36.140938+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:28.591708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:28.591708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:46.935191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":676,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147874+0000","last_change":"2024-04-26T16:26:35.756680+0000","last_active":"2024-04-26T16:27:36.147874+0000","last_peered":"2024-04-26T16:27:36.147874+0000","last_clean":"2024-04-26T16:27:36.147874+0000","last_became_active":"2024-04-26T16:26:23.174016+0000","last_became_peered":"2024-04-26T16:26:23.174016+0000","last_unstale":"2024-04-26T16:27:36.147874+0000","last_undegraded":"2024-04-26T16:27:36.147874+0000","last_fullsized":"2024-04-26T16:27:36.147874+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:35.756575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.756575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:05.545255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"316'41","reported_seq":879,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141041+0000","last_change":"2024-04-26T16:26:23.271105+0000","last_active":"2024-04-26T16:27:36.141041+0000","last_peered":"2024-04-26T16:27:36.141041+0000","last_clean":"2024-04-26T16:27:36.141041+0000","last_became_active":"2024-04-26T16:26:23.203408+0000","last_became_peered":"2024-04-26T16:26:23.203408+0000","last_unstale":"2024-04-26T16:27:36.141041+0000","last_undegraded":"2024-04-26T16:27:36.141041+0000","last_fullsized":"2024-04-26T16:27:36.141041+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'40","last_scrub_stamp":"2024-04-26T16:26:06.536113+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:26:06.536113+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":51,"num_read_kb":36,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":671,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166410+0000","last_change":"2024-04-26T16:25:03.657007+0000","last_active":"2024-04-26T16:27:36.166410+0000","last_peered":"2024-04-26T16:27:36.166410+0000","last_clean":"2024-04-26T16:27:36.166410+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:27:36.166410+0000","last_undegraded":"2024-04-26T16:27:36.166410+0000","last_fullsized":"2024-04-26T16:27:36.166410+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:25:03.656843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:03.656843+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138697+0000","last_change":"2024-04-26T16:24:36.579624+0000","last_active":"2024-04-26T16:27:36.138697+0000","last_peered":"2024-04-26T16:27:36.138697+0000","last_clean":"2024-04-26T16:27:36.138697+0000","last_became_active":"2024-04-26T16:17:35.673254+0000","last_became_peered":"2024-04-26T16:17:35.673254+0000","last_unstale":"2024-04-26T16:27:36.138697+0000","last_undegraded":"2024-04-26T16:27:36.138697+0000","last_fullsized":"2024-04-26T16:27:36.138697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:36.579507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:36.579507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":510,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.134120+0000","last_change":"2024-04-26T16:26:17.183477+0000","last_active":"2024-04-26T16:27:36.134120+0000","last_peered":"2024-04-26T16:27:36.134120+0000","last_clean":"2024-04-26T16:27:36.134120+0000","last_became_active":"2024-04-26T16:23:29.230684+0000","last_became_peered":"2024-04-26T16:23:29.230684+0000","last_unstale":"2024-04-26T16:27:36.134120+0000","last_undegraded":"2024-04-26T16:27:36.134120+0000","last_fullsized":"2024-04-26T16:27:36.134120+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:17.183383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:17.183383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":166,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.191284+0000","last_change":"2024-04-26T16:27:36.191311+0000","last_active":"2024-04-26T16:27:36.191284+0000","last_peered":"2024-04-26T16:27:36.191284+0000","last_clean":"2024-04-26T16:27:36.191284+0000","last_became_active":"2024-04-26T16:26:23.191179+0000","last_became_peered":"2024-04-26T16:26:23.191179+0000","last_unstale":"2024-04-26T16:27:36.191284+0000","last_undegraded":"2024-04-26T16:27:36.191284+0000","last_fullsized":"2024-04-26T16:27:36.191284+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.191214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:46.442629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":664,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148549+0000","last_change":"2024-04-26T16:24:45.666044+0000","last_active":"2024-04-26T16:27:36.148549+0000","last_peered":"2024-04-26T16:27:36.148549+0000","last_clean":"2024-04-26T16:27:36.148549+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:27:36.148549+0000","last_undegraded":"2024-04-26T16:27:36.148549+0000","last_fullsized":"2024-04-26T16:27:36.148549+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:45.665745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.665745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'25","reported_seq":706,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.311645+0000","last_change":"2024-04-26T16:26:35.438370+0000","last_active":"2024-04-26T16:27:36.311645+0000","last_peered":"2024-04-26T16:27:36.311645+0000","last_clean":"2024-04-26T16:27:36.311645+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:27:36.311645+0000","last_undegraded":"2024-04-26T16:27:36.311645+0000","last_fullsized":"2024-04-26T16:27:36.311645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'21","last_scrub_stamp":"2024-04-26T16:26:35.438250+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.438250+0000","objects_scrubbed":7,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:52.597359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":20,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":659,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141489+0000","last_change":"2024-04-26T16:26:23.215626+0000","last_active":"2024-04-26T16:27:36.141489+0000","last_peered":"2024-04-26T16:27:36.141489+0000","last_clean":"2024-04-26T16:27:36.141489+0000","last_became_active":"2024-04-26T16:26:23.215342+0000","last_became_peered":"2024-04-26T16:26:23.215342+0000","last_unstale":"2024-04-26T16:27:36.141489+0000","last_undegraded":"2024-04-26T16:27:36.141489+0000","last_fullsized":"2024-04-26T16:27:36.141489+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:51.295336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:24:51.295336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":516,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.136822+0000","last_change":"2024-04-26T16:26:23.209379+0000","last_active":"2024-04-26T16:27:36.136822+0000","last_peered":"2024-04-26T16:27:36.136822+0000","last_clean":"2024-04-26T16:27:36.136822+0000","last_became_active":"2024-04-26T16:26:23.208926+0000","last_became_peered":"2024-04-26T16:26:23.208926+0000","last_unstale":"2024-04-26T16:27:36.136822+0000","last_undegraded":"2024-04-26T16:27:36.136822+0000","last_fullsized":"2024-04-26T16:27:36.136822+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:15.158580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:15.158580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.161106+0000","last_change":"2024-04-26T16:26:23.177350+0000","last_active":"2024-04-26T16:27:36.161106+0000","last_peered":"2024-04-26T16:27:36.161106+0000","last_clean":"2024-04-26T16:27:36.161106+0000","last_became_active":"2024-04-26T16:26:23.176892+0000","last_became_peered":"2024-04-26T16:26:23.176892+0000","last_unstale":"2024-04-26T16:27:36.161106+0000","last_undegraded":"2024-04-26T16:27:36.161106+0000","last_fullsized":"2024-04-26T16:27:36.161106+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:07.156216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:26:07.156216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":951,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.151695+0000","last_change":"2024-04-26T16:26:29.671257+0000","last_active":"2024-04-26T16:27:36.151695+0000","last_peered":"2024-04-26T16:27:36.151695+0000","last_clean":"2024-04-26T16:27:36.151695+0000","last_became_active":"2024-04-26T16:16:50.925983+0000","last_became_peered":"2024-04-26T16:16:50.925983+0000","last_unstale":"2024-04-26T16:27:36.151695+0000","last_undegraded":"2024-04-26T16:27:36.151695+0000","last_fullsized":"2024-04-26T16:27:36.151695+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:26:29.671147+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:26:29.671147+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:50.837120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":156,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139156+0000","last_change":"2024-04-26T16:26:23.201539+0000","last_active":"2024-04-26T16:27:36.139156+0000","last_peered":"2024-04-26T16:27:36.139156+0000","last_clean":"2024-04-26T16:27:36.139156+0000","last_became_active":"2024-04-26T16:26:23.201156+0000","last_became_peered":"2024-04-26T16:26:23.201156+0000","last_unstale":"2024-04-26T16:27:36.139156+0000","last_undegraded":"2024-04-26T16:27:36.139156+0000","last_fullsized":"2024-04-26T16:27:36.139156+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:06.651513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:06.651513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"388'17","reported_seq":704,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.274779+0000","last_change":"2024-04-26T16:26:23.241420+0000","last_active":"2024-04-26T16:27:36.274779+0000","last_peered":"2024-04-26T16:27:36.274779+0000","last_clean":"2024-04-26T16:27:36.274779+0000","last_became_active":"2024-04-26T16:26:23.207215+0000","last_became_peered":"2024-04-26T16:26:23.207215+0000","last_unstale":"2024-04-26T16:27:36.274779+0000","last_undegraded":"2024-04-26T16:27:36.274779+0000","last_fullsized":"2024-04-26T16:27:36.274779+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'15","last_scrub_stamp":"2024-04-26T16:25:08.969060+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:25:08.969060+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":156,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149895+0000","last_change":"2024-04-26T16:26:23.213144+0000","last_active":"2024-04-26T16:27:36.149895+0000","last_peered":"2024-04-26T16:27:36.149895+0000","last_clean":"2024-04-26T16:27:36.149895+0000","last_became_active":"2024-04-26T16:26:23.212670+0000","last_became_peered":"2024-04-26T16:26:23.212670+0000","last_unstale":"2024-04-26T16:27:36.149895+0000","last_undegraded":"2024-04-26T16:27:36.149895+0000","last_fullsized":"2024-04-26T16:27:36.149895+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:48.448237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:48.448237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":516,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.133489+0000","last_change":"2024-04-26T16:26:23.193141+0000","last_active":"2024-04-26T16:27:36.133489+0000","last_peered":"2024-04-26T16:27:36.133489+0000","last_clean":"2024-04-26T16:27:36.133489+0000","last_became_active":"2024-04-26T16:26:23.192794+0000","last_became_peered":"2024-04-26T16:26:23.192794+0000","last_unstale":"2024-04-26T16:27:36.133489+0000","last_undegraded":"2024-04-26T16:27:36.133489+0000","last_fullsized":"2024-04-26T16:27:36.133489+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:04.068168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:04.068168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":674,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.145041+0000","last_change":"2024-04-26T16:26:34.426813+0000","last_active":"2024-04-26T16:27:36.145041+0000","last_peered":"2024-04-26T16:27:36.145041+0000","last_clean":"2024-04-26T16:27:36.145041+0000","last_became_active":"2024-04-26T16:26:23.185332+0000","last_became_peered":"2024-04-26T16:26:23.185332+0000","last_unstale":"2024-04-26T16:27:36.145041+0000","last_undegraded":"2024-04-26T16:27:36.145041+0000","last_fullsized":"2024-04-26T16:27:36.145041+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:26:34.426484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:58.662622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":676,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.143121+0000","last_change":"2024-04-26T16:26:37.361877+0000","last_active":"2024-04-26T16:27:36.143121+0000","last_peered":"2024-04-26T16:27:36.143121+0000","last_clean":"2024-04-26T16:27:36.143121+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:27:36.143121+0000","last_undegraded":"2024-04-26T16:27:36.143121+0000","last_fullsized":"2024-04-26T16:27:36.143121+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:37.361771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:26:37.361771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:57.940548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'28","reported_seq":713,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.309370+0000","last_change":"2024-04-26T16:24:34.463573+0000","last_active":"2024-04-26T16:27:36.309370+0000","last_peered":"2024-04-26T16:27:36.309370+0000","last_clean":"2024-04-26T16:27:36.309370+0000","last_became_active":"2024-04-26T16:17:31.023021+0000","last_became_peered":"2024-04-26T16:17:31.023021+0000","last_unstale":"2024-04-26T16:27:36.309370+0000","last_undegraded":"2024-04-26T16:27:36.309370+0000","last_fullsized":"2024-04-26T16:27:36.309370+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'26","last_scrub_stamp":"2024-04-26T16:24:34.463446+0000","last_deep_scrub":"112'26","last_deep_scrub_stamp":"2024-04-26T16:19:43.114691+0000","last_clean_scrub_stamp":"2024-04-26T16:24:34.463446+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":25,"num_write":22,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":651,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166292+0000","last_change":"2024-04-26T16:24:42.751470+0000","last_active":"2024-04-26T16:27:36.166292+0000","last_peered":"2024-04-26T16:27:36.166292+0000","last_clean":"2024-04-26T16:27:36.166292+0000","last_became_active":"2024-04-26T16:17:35.670585+0000","last_became_peered":"2024-04-26T16:17:35.670585+0000","last_unstale":"2024-04-26T16:27:36.166292+0000","last_undegraded":"2024-04-26T16:27:36.166292+0000","last_fullsized":"2024-04-26T16:27:36.166292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:42.751183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:42.751183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":510,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144724+0000","last_change":"2024-04-26T16:26:35.238080+0000","last_active":"2024-04-26T16:27:36.144724+0000","last_peered":"2024-04-26T16:27:36.144724+0000","last_clean":"2024-04-26T16:27:36.144724+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:27:36.144724+0000","last_undegraded":"2024-04-26T16:27:36.144724+0000","last_fullsized":"2024-04-26T16:27:36.144724+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:35.237931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.237931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:41.916785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":668,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.160026+0000","last_change":"2024-04-26T16:26:30.474193+0000","last_active":"2024-04-26T16:27:36.160026+0000","last_peered":"2024-04-26T16:27:36.160026+0000","last_clean":"2024-04-26T16:27:36.160026+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:27:36.160026+0000","last_undegraded":"2024-04-26T16:27:36.160026+0000","last_fullsized":"2024-04-26T16:27:36.160026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:30.474049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:30.474049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:52.639886+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'29","reported_seq":716,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.282938+0000","last_change":"2024-04-26T16:26:32.617065+0000","last_active":"2024-04-26T16:27:36.282938+0000","last_peered":"2024-04-26T16:27:36.282938+0000","last_clean":"2024-04-26T16:27:36.282938+0000","last_became_active":"2024-04-26T16:17:31.030904+0000","last_became_peered":"2024-04-26T16:17:31.030904+0000","last_unstale":"2024-04-26T16:27:36.282938+0000","last_undegraded":"2024-04-26T16:27:36.282938+0000","last_fullsized":"2024-04-26T16:27:36.282938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'27","last_scrub_stamp":"2024-04-26T16:26:32.616901+0000","last_deep_scrub":"103'27","last_deep_scrub_stamp":"2024-04-26T16:18:40.947097+0000","last_clean_scrub_stamp":"2024-04-26T16:26:32.616901+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:43.199260+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":28,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":158,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139410+0000","last_change":"2024-04-26T16:26:23.185788+0000","last_active":"2024-04-26T16:27:36.139410+0000","last_peered":"2024-04-26T16:27:36.139410+0000","last_clean":"2024-04-26T16:27:36.139410+0000","last_became_active":"2024-04-26T16:26:23.185379+0000","last_became_peered":"2024-04-26T16:26:23.185379+0000","last_unstale":"2024-04-26T16:27:36.139410+0000","last_undegraded":"2024-04-26T16:27:36.139410+0000","last_fullsized":"2024-04-26T16:27:36.139410+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:59.573732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.573732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":655,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149889+0000","last_change":"2024-04-26T16:24:46.914560+0000","last_active":"2024-04-26T16:27:36.149889+0000","last_peered":"2024-04-26T16:27:36.149889+0000","last_clean":"2024-04-26T16:27:36.149889+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:27:36.149889+0000","last_undegraded":"2024-04-26T16:27:36.149889+0000","last_fullsized":"2024-04-26T16:27:36.149889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:46.914314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:46.914314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":511,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138517+0000","last_change":"2024-04-26T16:26:13.403182+0000","last_active":"2024-04-26T16:27:36.138517+0000","last_peered":"2024-04-26T16:27:36.138517+0000","last_clean":"2024-04-26T16:27:36.138517+0000","last_became_active":"2024-04-26T16:23:29.218856+0000","last_became_peered":"2024-04-26T16:23:29.218856+0000","last_unstale":"2024-04-26T16:27:36.138517+0000","last_undegraded":"2024-04-26T16:27:36.138517+0000","last_fullsized":"2024-04-26T16:27:36.138517+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:13.402935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:13.402935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:36.756042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"316'14","reported_seq":815,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149584+0000","last_change":"2024-04-26T16:26:25.340752+0000","last_active":"2024-04-26T16:27:36.149584+0000","last_peered":"2024-04-26T16:27:36.149584+0000","last_clean":"2024-04-26T16:27:36.149584+0000","last_became_active":"2024-04-26T16:26:23.193630+0000","last_became_peered":"2024-04-26T16:26:23.193630+0000","last_unstale":"2024-04-26T16:27:36.149584+0000","last_undegraded":"2024-04-26T16:27:36.149584+0000","last_fullsized":"2024-04-26T16:27:36.149584+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'14","last_scrub_stamp":"2024-04-26T16:26:25.340570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:25.340570+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:40.639911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":686,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139732+0000","last_change":"2024-04-26T16:26:23.209396+0000","last_active":"2024-04-26T16:27:36.139732+0000","last_peered":"2024-04-26T16:27:36.139732+0000","last_clean":"2024-04-26T16:27:36.139732+0000","last_became_active":"2024-04-26T16:26:23.187980+0000","last_became_peered":"2024-04-26T16:26:23.187980+0000","last_unstale":"2024-04-26T16:27:36.139732+0000","last_undegraded":"2024-04-26T16:27:36.139732+0000","last_fullsized":"2024-04-26T16:27:36.139732+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:26:21.403411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:26:21.403411+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:45.360869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"388'27","reported_seq":250,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.315617+0000","last_change":"2024-04-26T16:26:37.181820+0000","last_active":"2024-04-26T16:27:36.315617+0000","last_peered":"2024-04-26T16:27:36.315617+0000","last_clean":"2024-04-26T16:27:36.315617+0000","last_became_active":"2024-04-26T16:26:23.208487+0000","last_became_peered":"2024-04-26T16:26:23.208487+0000","last_unstale":"2024-04-26T16:27:36.315617+0000","last_undegraded":"2024-04-26T16:27:36.315617+0000","last_fullsized":"2024-04-26T16:27:36.315617+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'25","last_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:26:37.181705+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:52.140938+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":27,"num_read_kb":17,"num_write":15,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":673,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141159+0000","last_change":"2024-04-26T16:26:16.490723+0000","last_active":"2024-04-26T16:27:36.141159+0000","last_peered":"2024-04-26T16:27:36.141159+0000","last_clean":"2024-04-26T16:27:36.141159+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:27:36.141159+0000","last_undegraded":"2024-04-26T16:27:36.141159+0000","last_fullsized":"2024-04-26T16:27:36.141159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:16.490588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:26:16.490588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":657,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.143672+0000","last_change":"2024-04-26T16:26:14.398692+0000","last_active":"2024-04-26T16:27:36.143672+0000","last_peered":"2024-04-26T16:27:36.143672+0000","last_clean":"2024-04-26T16:27:36.143672+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:27:36.143672+0000","last_undegraded":"2024-04-26T16:27:36.143672+0000","last_fullsized":"2024-04-26T16:27:36.143672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:14.398471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:14.398471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:43.048026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":519,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154488+0000","last_change":"2024-04-26T16:27:32.277297+0000","last_active":"2024-04-26T16:27:36.154488+0000","last_peered":"2024-04-26T16:27:36.154488+0000","last_clean":"2024-04-26T16:27:36.154488+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:27:36.154488+0000","last_undegraded":"2024-04-26T16:27:36.154488+0000","last_fullsized":"2024-04-26T16:27:36.154488+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:32.277174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:32.277174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:36.337562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":668,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.141693+0000","last_change":"2024-04-26T16:26:36.405902+0000","last_active":"2024-04-26T16:27:36.141693+0000","last_peered":"2024-04-26T16:27:36.141693+0000","last_clean":"2024-04-26T16:27:36.141693+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:27:36.141693+0000","last_undegraded":"2024-04-26T16:27:36.141693+0000","last_fullsized":"2024-04-26T16:27:36.141693+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:36.405806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:36.405806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:50.300705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"86'11","reported_seq":686,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166532+0000","last_change":"2024-04-26T16:26:19.873210+0000","last_active":"2024-04-26T16:27:36.166532+0000","last_peered":"2024-04-26T16:27:36.166532+0000","last_clean":"2024-04-26T16:27:36.166532+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:27:36.166532+0000","last_undegraded":"2024-04-26T16:27:36.166532+0000","last_fullsized":"2024-04-26T16:27:36.166532+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:26:19.872940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:19.872940+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":689,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.160750+0000","last_change":"2024-04-26T16:25:10.807777+0000","last_active":"2024-04-26T16:27:36.160750+0000","last_peered":"2024-04-26T16:27:36.160750+0000","last_clean":"2024-04-26T16:27:36.160750+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:27:36.160750+0000","last_undegraded":"2024-04-26T16:27:36.160750+0000","last_fullsized":"2024-04-26T16:27:36.160750+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:25:10.807661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:10.807661+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":647,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147133+0000","last_change":"2024-04-26T16:25:01.008927+0000","last_active":"2024-04-26T16:27:36.147133+0000","last_peered":"2024-04-26T16:27:36.147133+0000","last_clean":"2024-04-26T16:27:36.147133+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:27:36.147133+0000","last_undegraded":"2024-04-26T16:27:36.147133+0000","last_fullsized":"2024-04-26T16:27:36.147133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:01.008836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:25:01.008836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":519,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144427+0000","last_change":"2024-04-26T16:27:29.241930+0000","last_active":"2024-04-26T16:27:36.144427+0000","last_peered":"2024-04-26T16:27:36.144427+0000","last_clean":"2024-04-26T16:27:36.144427+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:27:36.144427+0000","last_undegraded":"2024-04-26T16:27:36.144427+0000","last_fullsized":"2024-04-26T16:27:36.144427+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:29.241760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.241760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:41.243460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"86'11","reported_seq":678,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154996+0000","last_change":"2024-04-26T16:24:56.661419+0000","last_active":"2024-04-26T16:27:36.154996+0000","last_peered":"2024-04-26T16:27:36.154996+0000","last_clean":"2024-04-26T16:27:36.154996+0000","last_became_active":"2024-04-26T16:17:31.033294+0000","last_became_peered":"2024-04-26T16:17:31.033294+0000","last_unstale":"2024-04-26T16:27:36.154996+0000","last_undegraded":"2024-04-26T16:27:36.154996+0000","last_fullsized":"2024-04-26T16:27:36.154996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:56.661177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:56.661177+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":676,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.294252+0000","last_change":"2024-04-26T16:27:36.294276+0000","last_active":"2024-04-26T16:27:36.294252+0000","last_peered":"2024-04-26T16:27:36.294252+0000","last_clean":"2024-04-26T16:27:36.294252+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:27:36.294252+0000","last_undegraded":"2024-04-26T16:27:36.294252+0000","last_fullsized":"2024-04-26T16:27:36.294252+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:36.294169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:27:36.294169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:37.292778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":659,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149967+0000","last_change":"2024-04-26T16:27:31.162841+0000","last_active":"2024-04-26T16:27:36.149967+0000","last_peered":"2024-04-26T16:27:36.149967+0000","last_clean":"2024-04-26T16:27:36.149967+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:27:36.149967+0000","last_undegraded":"2024-04-26T16:27:36.149967+0000","last_fullsized":"2024-04-26T16:27:36.149967+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:31.162674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:40.032566+0000","last_clean_scrub_stamp":"2024-04-26T16:27:31.162674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:52.604751+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":656,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.143128+0000","last_change":"2024-04-26T16:24:42.021813+0000","last_active":"2024-04-26T16:27:36.143128+0000","last_peered":"2024-04-26T16:27:36.143128+0000","last_clean":"2024-04-26T16:27:36.143128+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:27:36.143128+0000","last_undegraded":"2024-04-26T16:27:36.143128+0000","last_fullsized":"2024-04-26T16:27:36.143128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:42.021591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:42.021591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":519,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140326+0000","last_change":"2024-04-26T16:27:31.367737+0000","last_active":"2024-04-26T16:27:36.140326+0000","last_peered":"2024-04-26T16:27:36.140326+0000","last_clean":"2024-04-26T16:27:36.140326+0000","last_became_active":"2024-04-26T16:23:29.219590+0000","last_became_peered":"2024-04-26T16:23:29.219590+0000","last_unstale":"2024-04-26T16:27:36.140326+0000","last_undegraded":"2024-04-26T16:27:36.140326+0000","last_fullsized":"2024-04-26T16:27:36.140326+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:31.367354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:31.367354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:39.656075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"86'4","reported_seq":692,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.146851+0000","last_change":"2024-04-26T16:26:23.254424+0000","last_active":"2024-04-26T16:27:36.146851+0000","last_peered":"2024-04-26T16:27:36.146851+0000","last_clean":"2024-04-26T16:27:36.146851+0000","last_became_active":"2024-04-26T16:26:23.196264+0000","last_became_peered":"2024-04-26T16:26:23.196264+0000","last_unstale":"2024-04-26T16:27:36.146851+0000","last_undegraded":"2024-04-26T16:27:36.146851+0000","last_fullsized":"2024-04-26T16:27:36.146851+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'4","last_scrub_stamp":"2024-04-26T16:26:12.544356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:12.544356+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:39.076067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":670,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147802+0000","last_change":"2024-04-26T16:26:33.753187+0000","last_active":"2024-04-26T16:27:36.147802+0000","last_peered":"2024-04-26T16:27:36.147802+0000","last_clean":"2024-04-26T16:27:36.147802+0000","last_became_active":"2024-04-26T16:17:28.999634+0000","last_became_peered":"2024-04-26T16:17:28.999634+0000","last_unstale":"2024-04-26T16:27:36.147802+0000","last_undegraded":"2024-04-26T16:27:36.147802+0000","last_fullsized":"2024-04-26T16:27:36.147802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:33.753069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:33.753069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:00.067849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":155,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149603+0000","last_change":"2024-04-26T16:26:23.210141+0000","last_active":"2024-04-26T16:27:36.149603+0000","last_peered":"2024-04-26T16:27:36.149603+0000","last_clean":"2024-04-26T16:27:36.149603+0000","last_became_active":"2024-04-26T16:26:23.209837+0000","last_became_peered":"2024-04-26T16:26:23.209837+0000","last_unstale":"2024-04-26T16:27:36.149603+0000","last_undegraded":"2024-04-26T16:27:36.149603+0000","last_fullsized":"2024-04-26T16:27:36.149603+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.550844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.550844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":651,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166169+0000","last_change":"2024-04-26T16:27:18.906489+0000","last_active":"2024-04-26T16:27:36.166169+0000","last_peered":"2024-04-26T16:27:36.166169+0000","last_clean":"2024-04-26T16:27:36.166169+0000","last_became_active":"2024-04-26T16:17:35.671754+0000","last_became_peered":"2024-04-26T16:17:35.671754+0000","last_unstale":"2024-04-26T16:27:36.166169+0000","last_undegraded":"2024-04-26T16:27:36.166169+0000","last_fullsized":"2024-04-26T16:27:36.166169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:18.906360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:27:18.906360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:33.699067+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":522,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.157302+0000","last_change":"2024-04-26T16:27:20.808939+0000","last_active":"2024-04-26T16:27:36.157302+0000","last_peered":"2024-04-26T16:27:36.157302+0000","last_clean":"2024-04-26T16:27:36.157302+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:27:36.157302+0000","last_undegraded":"2024-04-26T16:27:36.157302+0000","last_fullsized":"2024-04-26T16:27:36.157302+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:20.808714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:20.808714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:32.225311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'13","reported_seq":707,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.328777+0000","last_change":"2024-04-26T16:27:29.963129+0000","last_active":"2024-04-26T16:27:36.328777+0000","last_peered":"2024-04-26T16:27:36.328777+0000","last_clean":"2024-04-26T16:27:36.328777+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:27:36.328777+0000","last_undegraded":"2024-04-26T16:27:36.328777+0000","last_fullsized":"2024-04-26T16:27:36.328777+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:29.962882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.962882+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:54.101192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":163,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.136421+0000","last_change":"2024-04-26T16:26:30.164086+0000","last_active":"2024-04-26T16:27:36.136421+0000","last_peered":"2024-04-26T16:27:36.136421+0000","last_clean":"2024-04-26T16:27:36.136421+0000","last_became_active":"2024-04-26T16:26:23.216238+0000","last_became_peered":"2024-04-26T16:26:23.216238+0000","last_unstale":"2024-04-26T16:27:36.136421+0000","last_undegraded":"2024-04-26T16:27:36.136421+0000","last_fullsized":"2024-04-26T16:27:36.136421+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:30.163969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:30.163969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:56.195993+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"316'13","reported_seq":250,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140048+0000","last_change":"2024-04-26T16:26:23.279103+0000","last_active":"2024-04-26T16:27:36.140048+0000","last_peered":"2024-04-26T16:27:36.140048+0000","last_clean":"2024-04-26T16:27:36.140048+0000","last_became_active":"2024-04-26T16:26:23.215841+0000","last_became_peered":"2024-04-26T16:26:23.215841+0000","last_unstale":"2024-04-26T16:27:36.140048+0000","last_undegraded":"2024-04-26T16:27:36.140048+0000","last_fullsized":"2024-04-26T16:27:36.140048+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'12","last_scrub_stamp":"2024-04-26T16:24:38.428577+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:24:38.428577+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":656,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154931+0000","last_change":"2024-04-26T16:26:18.424934+0000","last_active":"2024-04-26T16:27:36.154931+0000","last_peered":"2024-04-26T16:27:36.154931+0000","last_clean":"2024-04-26T16:27:36.154931+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:27:36.154931+0000","last_undegraded":"2024-04-26T16:27:36.154931+0000","last_fullsized":"2024-04-26T16:27:36.154931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:18.424770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:18.424770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":155,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138761+0000","last_change":"2024-04-26T16:26:23.209619+0000","last_active":"2024-04-26T16:27:36.138761+0000","last_peered":"2024-04-26T16:27:36.138761+0000","last_clean":"2024-04-26T16:27:36.138761+0000","last_became_active":"2024-04-26T16:26:23.209163+0000","last_became_peered":"2024-04-26T16:26:23.209163+0000","last_unstale":"2024-04-26T16:27:36.138761+0000","last_undegraded":"2024-04-26T16:27:36.138761+0000","last_fullsized":"2024-04-26T16:27:36.138761+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:08.691959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:08.691959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"121'13","reported_seq":692,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148188+0000","last_change":"2024-04-26T16:26:22.385102+0000","last_active":"2024-04-26T16:27:36.148188+0000","last_peered":"2024-04-26T16:27:36.148188+0000","last_clean":"2024-04-26T16:27:36.148188+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:27:36.148188+0000","last_undegraded":"2024-04-26T16:27:36.148188+0000","last_fullsized":"2024-04-26T16:27:36.148188+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'13","last_scrub_stamp":"2024-04-26T16:26:22.384959+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:26:22.384959+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:50.162802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":676,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149011+0000","last_change":"2024-04-26T16:26:32.467143+0000","last_active":"2024-04-26T16:27:36.149011+0000","last_peered":"2024-04-26T16:27:36.149011+0000","last_clean":"2024-04-26T16:27:36.149011+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:27:36.149011+0000","last_undegraded":"2024-04-26T16:27:36.149011+0000","last_fullsized":"2024-04-26T16:27:36.149011+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:32.466941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:32.466941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:00.123352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":783,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148645+0000","last_change":"2024-04-26T16:24:49.977333+0000","last_active":"2024-04-26T16:27:36.148645+0000","last_peered":"2024-04-26T16:27:36.148645+0000","last_clean":"2024-04-26T16:27:36.148645+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:27:36.148645+0000","last_undegraded":"2024-04-26T16:27:36.148645+0000","last_fullsized":"2024-04-26T16:27:36.148645+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:24:49.977080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:49.977080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":667,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.161225+0000","last_change":"2024-04-26T16:27:26.870991+0000","last_active":"2024-04-26T16:27:36.161225+0000","last_peered":"2024-04-26T16:27:36.161225+0000","last_clean":"2024-04-26T16:27:36.161225+0000","last_became_active":"2024-04-26T16:26:23.188629+0000","last_became_peered":"2024-04-26T16:26:23.188629+0000","last_unstale":"2024-04-26T16:27:36.161225+0000","last_undegraded":"2024-04-26T16:27:36.161225+0000","last_fullsized":"2024-04-26T16:27:36.161225+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:26.870779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:27:26.870779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:39.013860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":514,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167068+0000","last_change":"2024-04-26T16:27:25.909568+0000","last_active":"2024-04-26T16:27:36.167068+0000","last_peered":"2024-04-26T16:27:36.167068+0000","last_clean":"2024-04-26T16:27:36.167068+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:27:36.167068+0000","last_undegraded":"2024-04-26T16:27:36.167068+0000","last_fullsized":"2024-04-26T16:27:36.167068+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:25.909462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.909462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:26.611243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"121'13","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167078+0000","last_change":"2024-04-26T16:24:30.582687+0000","last_active":"2024-04-26T16:27:36.167078+0000","last_peered":"2024-04-26T16:27:36.167078+0000","last_clean":"2024-04-26T16:27:36.167078+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:27:36.167078+0000","last_undegraded":"2024-04-26T16:27:36.167078+0000","last_fullsized":"2024-04-26T16:27:36.167078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'13","last_scrub_stamp":"2024-04-26T16:24:30.582564+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:24:30.582564+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":684,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.151115+0000","last_change":"2024-04-26T16:26:31.736527+0000","last_active":"2024-04-26T16:27:36.151115+0000","last_peered":"2024-04-26T16:27:36.151115+0000","last_clean":"2024-04-26T16:27:36.151115+0000","last_became_active":"2024-04-26T16:26:23.195571+0000","last_became_peered":"2024-04-26T16:26:23.195571+0000","last_unstale":"2024-04-26T16:27:36.151115+0000","last_undegraded":"2024-04-26T16:27:36.151115+0000","last_fullsized":"2024-04-26T16:27:36.151115+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:31.736395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:26:31.736395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:48.239195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":659,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147006+0000","last_change":"2024-04-26T16:26:02.153686+0000","last_active":"2024-04-26T16:27:36.147006+0000","last_peered":"2024-04-26T16:27:36.147006+0000","last_clean":"2024-04-26T16:27:36.147006+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:27:36.147006+0000","last_undegraded":"2024-04-26T16:27:36.147006+0000","last_fullsized":"2024-04-26T16:27:36.147006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:02.153555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:02.153555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":670,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.142072+0000","last_change":"2024-04-26T16:27:30.270941+0000","last_active":"2024-04-26T16:27:36.142072+0000","last_peered":"2024-04-26T16:27:36.142072+0000","last_clean":"2024-04-26T16:27:36.142072+0000","last_became_active":"2024-04-26T16:26:23.197994+0000","last_became_peered":"2024-04-26T16:26:23.197994+0000","last_unstale":"2024-04-26T16:27:36.142072+0000","last_undegraded":"2024-04-26T16:27:36.142072+0000","last_fullsized":"2024-04-26T16:27:36.142072+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:30.270793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:27:30.270793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:54.426715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":521,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.153883+0000","last_change":"2024-04-26T16:27:20.195158+0000","last_active":"2024-04-26T16:27:36.153883+0000","last_peered":"2024-04-26T16:27:36.153883+0000","last_clean":"2024-04-26T16:27:36.153883+0000","last_became_active":"2024-04-26T16:23:29.319769+0000","last_became_peered":"2024-04-26T16:23:29.319769+0000","last_unstale":"2024-04-26T16:27:36.153883+0000","last_undegraded":"2024-04-26T16:27:36.153883+0000","last_fullsized":"2024-04-26T16:27:36.153883+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:20.194832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:20.194832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:35.975928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"388'12","reported_seq":687,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.269264+0000","last_change":"2024-04-26T16:26:28.703152+0000","last_active":"2024-04-26T16:27:36.269264+0000","last_peered":"2024-04-26T16:27:36.269264+0000","last_clean":"2024-04-26T16:27:36.269264+0000","last_became_active":"2024-04-26T16:17:31.021476+0000","last_became_peered":"2024-04-26T16:17:31.021476+0000","last_unstale":"2024-04-26T16:27:36.269264+0000","last_undegraded":"2024-04-26T16:27:36.269264+0000","last_fullsized":"2024-04-26T16:27:36.269264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:26:28.702748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:28.702748+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":165,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139596+0000","last_change":"2024-04-26T16:27:29.149637+0000","last_active":"2024-04-26T16:27:36.139596+0000","last_peered":"2024-04-26T16:27:36.139596+0000","last_clean":"2024-04-26T16:27:36.139596+0000","last_became_active":"2024-04-26T16:26:23.196104+0000","last_became_peered":"2024-04-26T16:26:23.196104+0000","last_unstale":"2024-04-26T16:27:36.139596+0000","last_undegraded":"2024-04-26T16:27:36.139596+0000","last_fullsized":"2024-04-26T16:27:36.139596+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:29.149485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.149485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:31.408623+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":663,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.139632+0000","last_change":"2024-04-26T16:26:23.213791+0000","last_active":"2024-04-26T16:27:36.139632+0000","last_peered":"2024-04-26T16:27:36.139632+0000","last_clean":"2024-04-26T16:27:36.139632+0000","last_became_active":"2024-04-26T16:26:23.213441+0000","last_became_peered":"2024-04-26T16:26:23.213441+0000","last_unstale":"2024-04-26T16:27:36.139632+0000","last_undegraded":"2024-04-26T16:27:36.139632+0000","last_fullsized":"2024-04-26T16:27:36.139632+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:59.228337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:59.228337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.133437+0000","last_change":"2024-04-26T16:26:23.213527+0000","last_active":"2024-04-26T16:27:36.133437+0000","last_peered":"2024-04-26T16:27:36.133437+0000","last_clean":"2024-04-26T16:27:36.133437+0000","last_became_active":"2024-04-26T16:26:23.211857+0000","last_became_peered":"2024-04-26T16:26:23.211857+0000","last_unstale":"2024-04-26T16:27:36.133437+0000","last_undegraded":"2024-04-26T16:27:36.133437+0000","last_fullsized":"2024-04-26T16:27:36.133437+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:08.205352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:26:08.205352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:37.737007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":511,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144270+0000","last_change":"2024-04-26T16:25:02.271846+0000","last_active":"2024-04-26T16:27:36.144270+0000","last_peered":"2024-04-26T16:27:36.144270+0000","last_clean":"2024-04-26T16:27:36.144270+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:27:36.144270+0000","last_undegraded":"2024-04-26T16:27:36.144270+0000","last_fullsized":"2024-04-26T16:27:36.144270+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.271632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.271632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'8","reported_seq":698,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.261080+0000","last_change":"2024-04-26T16:26:27.420521+0000","last_active":"2024-04-26T16:27:36.261080+0000","last_peered":"2024-04-26T16:27:36.261080+0000","last_clean":"2024-04-26T16:27:36.261080+0000","last_became_active":"2024-04-26T16:26:23.193873+0000","last_became_peered":"2024-04-26T16:26:23.193873+0000","last_unstale":"2024-04-26T16:27:36.261080+0000","last_undegraded":"2024-04-26T16:27:36.261080+0000","last_fullsized":"2024-04-26T16:27:36.261080+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'6","last_scrub_stamp":"2024-04-26T16:26:27.420281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:27.420281+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:45.255366+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":166,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154111+0000","last_change":"2024-04-26T16:26:26.661192+0000","last_active":"2024-04-26T16:27:36.154111+0000","last_peered":"2024-04-26T16:27:36.154111+0000","last_clean":"2024-04-26T16:27:36.154111+0000","last_became_active":"2024-04-26T16:26:23.178508+0000","last_became_peered":"2024-04-26T16:26:23.178508+0000","last_unstale":"2024-04-26T16:27:36.154111+0000","last_undegraded":"2024-04-26T16:27:36.154111+0000","last_fullsized":"2024-04-26T16:27:36.154111+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:26.661038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:26.661038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:51.911185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":662,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147536+0000","last_change":"2024-04-26T16:26:06.452835+0000","last_active":"2024-04-26T16:27:36.147536+0000","last_peered":"2024-04-26T16:27:36.147536+0000","last_clean":"2024-04-26T16:27:36.147536+0000","last_became_active":"2024-04-26T16:17:33.059705+0000","last_became_peered":"2024-04-26T16:17:33.059705+0000","last_unstale":"2024-04-26T16:27:36.147536+0000","last_undegraded":"2024-04-26T16:27:36.147536+0000","last_fullsized":"2024-04-26T16:27:36.147536+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:06.452733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:06.452733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":651,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.156930+0000","last_change":"2024-04-26T16:26:22.799872+0000","last_active":"2024-04-26T16:27:36.156930+0000","last_peered":"2024-04-26T16:27:36.156930+0000","last_clean":"2024-04-26T16:27:36.156930+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:27:36.156930+0000","last_undegraded":"2024-04-26T16:27:36.156930+0000","last_fullsized":"2024-04-26T16:27:36.156930+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:22.799663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:22.799663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:38.957368+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":507,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166780+0000","last_change":"2024-04-26T16:23:29.348658+0000","last_active":"2024-04-26T16:27:36.166780+0000","last_peered":"2024-04-26T16:27:36.166780+0000","last_clean":"2024-04-26T16:27:36.166780+0000","last_became_active":"2024-04-26T16:23:29.344221+0000","last_became_peered":"2024-04-26T16:23:29.344221+0000","last_unstale":"2024-04-26T16:27:36.166780+0000","last_undegraded":"2024-04-26T16:27:36.166780+0000","last_fullsized":"2024-04-26T16:27:36.166780+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'13","reported_seq":209,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.297754+0000","last_change":"2024-04-26T16:26:23.278170+0000","last_active":"2024-04-26T16:27:36.297754+0000","last_peered":"2024-04-26T16:27:36.297754+0000","last_clean":"2024-04-26T16:27:36.297754+0000","last_became_active":"2024-04-26T16:26:23.200216+0000","last_became_peered":"2024-04-26T16:26:23.200216+0000","last_unstale":"2024-04-26T16:27:36.297754+0000","last_undegraded":"2024-04-26T16:27:36.297754+0000","last_fullsized":"2024-04-26T16:27:36.297754+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:23:28.218456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.218456+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":684,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154209+0000","last_change":"2024-04-26T16:27:19.159806+0000","last_active":"2024-04-26T16:27:36.154209+0000","last_peered":"2024-04-26T16:27:36.154209+0000","last_clean":"2024-04-26T16:27:36.154209+0000","last_became_active":"2024-04-26T16:26:23.179294+0000","last_became_peered":"2024-04-26T16:26:23.179294+0000","last_unstale":"2024-04-26T16:27:36.154209+0000","last_undegraded":"2024-04-26T16:27:36.154209+0000","last_fullsized":"2024-04-26T16:27:36.154209+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:19.159474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:27:19.159474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:20.743418+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":655,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.152807+0000","last_change":"2024-04-26T16:24:43.710578+0000","last_active":"2024-04-26T16:27:36.152807+0000","last_peered":"2024-04-26T16:27:36.152807+0000","last_clean":"2024-04-26T16:27:36.152807+0000","last_became_active":"2024-04-26T16:17:33.075775+0000","last_became_peered":"2024-04-26T16:17:33.075775+0000","last_unstale":"2024-04-26T16:27:36.152807+0000","last_undegraded":"2024-04-26T16:27:36.152807+0000","last_fullsized":"2024-04-26T16:27:36.152807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:43.710377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:24:43.710377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":655,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147242+0000","last_change":"2024-04-26T16:27:28.119283+0000","last_active":"2024-04-26T16:27:36.147242+0000","last_peered":"2024-04-26T16:27:36.147242+0000","last_clean":"2024-04-26T16:27:36.147242+0000","last_became_active":"2024-04-26T16:17:35.677366+0000","last_became_peered":"2024-04-26T16:17:35.677366+0000","last_unstale":"2024-04-26T16:27:36.147242+0000","last_undegraded":"2024-04-26T16:27:36.147242+0000","last_fullsized":"2024-04-26T16:27:36.147242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:28.119019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.119019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:31.909893+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":511,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.143449+0000","last_change":"2024-04-26T16:26:22.481291+0000","last_active":"2024-04-26T16:27:36.143449+0000","last_peered":"2024-04-26T16:27:36.143449+0000","last_clean":"2024-04-26T16:27:36.143449+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:27:36.143449+0000","last_undegraded":"2024-04-26T16:27:36.143449+0000","last_fullsized":"2024-04-26T16:27:36.143449+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:22.481097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:22.481097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"118'22","reported_seq":701,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138080+0000","last_change":"2024-04-26T16:26:17.473757+0000","last_active":"2024-04-26T16:27:36.138080+0000","last_peered":"2024-04-26T16:27:36.138080+0000","last_clean":"2024-04-26T16:27:36.138080+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:27:36.138080+0000","last_undegraded":"2024-04-26T16:27:36.138080+0000","last_fullsized":"2024-04-26T16:27:36.138080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-04-26T16:26:17.473609+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:26:17.473609+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":166,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148648+0000","last_change":"2024-04-26T16:27:23.855213+0000","last_active":"2024-04-26T16:27:36.148648+0000","last_peered":"2024-04-26T16:27:36.148648+0000","last_clean":"2024-04-26T16:27:36.148648+0000","last_became_active":"2024-04-26T16:26:23.157460+0000","last_became_peered":"2024-04-26T16:26:23.157460+0000","last_unstale":"2024-04-26T16:27:36.148648+0000","last_undegraded":"2024-04-26T16:27:36.148648+0000","last_fullsized":"2024-04-26T16:27:36.148648+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:23.855100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:27:23.855100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:24.047134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"329'12","reported_seq":806,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.157532+0000","last_change":"2024-04-26T16:26:37.837403+0000","last_active":"2024-04-26T16:27:36.157532+0000","last_peered":"2024-04-26T16:27:36.157532+0000","last_clean":"2024-04-26T16:27:36.157532+0000","last_became_active":"2024-04-26T16:17:33.061926+0000","last_became_peered":"2024-04-26T16:17:33.061926+0000","last_unstale":"2024-04-26T16:27:36.157532+0000","last_undegraded":"2024-04-26T16:27:36.157532+0000","last_fullsized":"2024-04-26T16:27:36.157532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'12","last_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:26:37.837294+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:58.295018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":663,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.165815+0000","last_change":"2024-04-26T16:26:23.278656+0000","last_active":"2024-04-26T16:27:36.165815+0000","last_peered":"2024-04-26T16:27:36.165815+0000","last_clean":"2024-04-26T16:27:36.165815+0000","last_became_active":"2024-04-26T16:26:23.201293+0000","last_became_peered":"2024-04-26T16:26:23.201293+0000","last_unstale":"2024-04-26T16:27:36.165815+0000","last_undegraded":"2024-04-26T16:27:36.165815+0000","last_fullsized":"2024-04-26T16:27:36.165815+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:26:20.841390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:20.841390+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":511,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.146383+0000","last_change":"2024-04-26T16:26:12.430289+0000","last_active":"2024-04-26T16:27:36.146383+0000","last_peered":"2024-04-26T16:27:36.146383+0000","last_clean":"2024-04-26T16:27:36.146383+0000","last_became_active":"2024-04-26T16:23:29.356195+0000","last_became_peered":"2024-04-26T16:23:29.356195+0000","last_unstale":"2024-04-26T16:27:36.146383+0000","last_undegraded":"2024-04-26T16:27:36.146383+0000","last_fullsized":"2024-04-26T16:27:36.146383+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:12.430072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:12.430072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"86'9","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.133631+0000","last_change":"2024-04-26T16:26:14.111227+0000","last_active":"2024-04-26T16:27:36.133631+0000","last_peered":"2024-04-26T16:27:36.133631+0000","last_clean":"2024-04-26T16:27:36.133631+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:27:36.133631+0000","last_undegraded":"2024-04-26T16:27:36.133631+0000","last_fullsized":"2024-04-26T16:27:36.133631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:26:14.111056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:14.111056+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":661,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.133680+0000","last_change":"2024-04-26T16:24:45.903529+0000","last_active":"2024-04-26T16:27:36.133680+0000","last_peered":"2024-04-26T16:27:36.133680+0000","last_clean":"2024-04-26T16:27:36.133680+0000","last_became_active":"2024-04-26T16:17:28.999524+0000","last_became_peered":"2024-04-26T16:17:28.999524+0000","last_unstale":"2024-04-26T16:27:36.133680+0000","last_undegraded":"2024-04-26T16:27:36.133680+0000","last_fullsized":"2024-04-26T16:27:36.133680+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:24:45.903320+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:24:45.903320+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":669,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154749+0000","last_change":"2024-04-26T16:26:21.392791+0000","last_active":"2024-04-26T16:27:36.154749+0000","last_peered":"2024-04-26T16:27:36.154749+0000","last_clean":"2024-04-26T16:27:36.154749+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:27:36.154749+0000","last_undegraded":"2024-04-26T16:27:36.154749+0000","last_fullsized":"2024-04-26T16:27:36.154749+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:21.392655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:26:21.392655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":666,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140617+0000","last_change":"2024-04-26T16:26:23.203769+0000","last_active":"2024-04-26T16:27:36.140617+0000","last_peered":"2024-04-26T16:27:36.140617+0000","last_clean":"2024-04-26T16:27:36.140617+0000","last_became_active":"2024-04-26T16:26:23.203386+0000","last_became_peered":"2024-04-26T16:26:23.203386+0000","last_unstale":"2024-04-26T16:27:36.140617+0000","last_undegraded":"2024-04-26T16:27:36.140617+0000","last_fullsized":"2024-04-26T16:27:36.140617+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:03.507021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:03.507021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":158,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.157539+0000","last_change":"2024-04-26T16:26:23.184063+0000","last_active":"2024-04-26T16:27:36.157539+0000","last_peered":"2024-04-26T16:27:36.157539+0000","last_clean":"2024-04-26T16:27:36.157539+0000","last_became_active":"2024-04-26T16:26:23.183709+0000","last_became_peered":"2024-04-26T16:26:23.183709+0000","last_unstale":"2024-04-26T16:27:36.157539+0000","last_undegraded":"2024-04-26T16:27:36.157539+0000","last_fullsized":"2024-04-26T16:27:36.157539+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:09.151339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:09.151339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"114'7","reported_seq":684,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149303+0000","last_change":"2024-04-26T16:27:25.238408+0000","last_active":"2024-04-26T16:27:36.149303+0000","last_peered":"2024-04-26T16:27:36.149303+0000","last_clean":"2024-04-26T16:27:36.149303+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:27:36.149303+0000","last_undegraded":"2024-04-26T16:27:36.149303+0000","last_fullsized":"2024-04-26T16:27:36.149303+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'7","last_scrub_stamp":"2024-04-26T16:27:25.238290+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:23:25.467392+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.238290+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:41.049701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":678,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.147741+0000","last_change":"2024-04-26T16:26:13.548352+0000","last_active":"2024-04-26T16:27:36.147741+0000","last_peered":"2024-04-26T16:27:36.147741+0000","last_clean":"2024-04-26T16:27:36.147741+0000","last_became_active":"2024-04-26T16:17:29.004486+0000","last_became_peered":"2024-04-26T16:17:29.004486+0000","last_unstale":"2024-04-26T16:27:36.147741+0000","last_undegraded":"2024-04-26T16:27:36.147741+0000","last_fullsized":"2024-04-26T16:27:36.147741+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:13.548133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:13.548133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":665,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.140400+0000","last_change":"2024-04-26T16:25:10.202126+0000","last_active":"2024-04-26T16:27:36.140400+0000","last_peered":"2024-04-26T16:27:36.140400+0000","last_clean":"2024-04-26T16:27:36.140400+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:27:36.140400+0000","last_undegraded":"2024-04-26T16:27:36.140400+0000","last_fullsized":"2024-04-26T16:27:36.140400+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:10.202042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:25:10.202042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":667,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.162490+0000","last_change":"2024-04-26T16:26:23.214984+0000","last_active":"2024-04-26T16:27:36.162490+0000","last_peered":"2024-04-26T16:27:36.162490+0000","last_clean":"2024-04-26T16:27:36.162490+0000","last_became_active":"2024-04-26T16:26:23.214604+0000","last_became_peered":"2024-04-26T16:26:23.214604+0000","last_unstale":"2024-04-26T16:27:36.162490+0000","last_undegraded":"2024-04-26T16:27:36.162490+0000","last_fullsized":"2024-04-26T16:27:36.162490+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:01.893700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:25:01.893700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":506,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.155412+0000","last_change":"2024-04-26T16:23:29.358177+0000","last_active":"2024-04-26T16:27:36.155412+0000","last_peered":"2024-04-26T16:27:36.155412+0000","last_clean":"2024-04-26T16:27:36.155412+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:27:36.155412+0000","last_undegraded":"2024-04-26T16:27:36.155412+0000","last_fullsized":"2024-04-26T16:27:36.155412+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:23:28.083853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'11","reported_seq":688,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.333357+0000","last_change":"2024-04-26T16:24:54.880270+0000","last_active":"2024-04-26T16:27:36.333357+0000","last_peered":"2024-04-26T16:27:36.333357+0000","last_clean":"2024-04-26T16:27:36.333357+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:27:36.333357+0000","last_undegraded":"2024-04-26T16:27:36.333357+0000","last_fullsized":"2024-04-26T16:27:36.333357+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-04-26T16:24:54.880131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:24:54.880131+0000","objects_scrubbed":4,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":14,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":659,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144950+0000","last_change":"2024-04-26T16:24:40.982753+0000","last_active":"2024-04-26T16:27:36.144950+0000","last_peered":"2024-04-26T16:27:36.144950+0000","last_clean":"2024-04-26T16:27:36.144950+0000","last_became_active":"2024-04-26T16:17:29.005070+0000","last_became_peered":"2024-04-26T16:17:29.005070+0000","last_unstale":"2024-04-26T16:27:36.144950+0000","last_undegraded":"2024-04-26T16:27:36.144950+0000","last_fullsized":"2024-04-26T16:27:36.144950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:40.982532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:40.982532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":665,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.158970+0000","last_change":"2024-04-26T16:26:35.721826+0000","last_active":"2024-04-26T16:27:36.158970+0000","last_peered":"2024-04-26T16:27:36.158970+0000","last_clean":"2024-04-26T16:27:36.158970+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:27:36.158970+0000","last_undegraded":"2024-04-26T16:27:36.158970+0000","last_fullsized":"2024-04-26T16:27:36.158970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:35.721719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.721719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:58.188567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":653,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149789+0000","last_change":"2024-04-26T16:26:23.202748+0000","last_active":"2024-04-26T16:27:36.149789+0000","last_peered":"2024-04-26T16:27:36.149789+0000","last_clean":"2024-04-26T16:27:36.149789+0000","last_became_active":"2024-04-26T16:26:23.202322+0000","last_became_peered":"2024-04-26T16:26:23.202322+0000","last_unstale":"2024-04-26T16:27:36.149789+0000","last_undegraded":"2024-04-26T16:27:36.149789+0000","last_fullsized":"2024-04-26T16:27:36.149789+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:36.099628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:24:36.099628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":519,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138643+0000","last_change":"2024-04-26T16:27:24.389718+0000","last_active":"2024-04-26T16:27:36.138643+0000","last_peered":"2024-04-26T16:27:36.138643+0000","last_clean":"2024-04-26T16:27:36.138643+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:27:36.138643+0000","last_undegraded":"2024-04-26T16:27:36.138643+0000","last_fullsized":"2024-04-26T16:27:36.138643+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:24.389574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:24.389574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:52.843701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"126'17","reported_seq":321,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167926+0000","last_change":"2024-04-26T16:26:29.674902+0000","last_active":"2024-04-26T16:27:36.167926+0000","last_peered":"2024-04-26T16:27:36.167926+0000","last_clean":"2024-04-26T16:27:36.167926+0000","last_became_active":"2024-04-26T16:26:23.183587+0000","last_became_peered":"2024-04-26T16:26:23.183587+0000","last_unstale":"2024-04-26T16:27:36.167926+0000","last_undegraded":"2024-04-26T16:27:36.167926+0000","last_fullsized":"2024-04-26T16:27:36.167926+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'17","last_scrub_stamp":"2024-04-26T16:26:29.674746+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:26:29.674746+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:36.517136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":669,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.160665+0000","last_change":"2024-04-26T16:26:23.176524+0000","last_active":"2024-04-26T16:27:36.160665+0000","last_peered":"2024-04-26T16:27:36.160665+0000","last_clean":"2024-04-26T16:27:36.160665+0000","last_became_active":"2024-04-26T16:26:23.175882+0000","last_became_peered":"2024-04-26T16:26:23.175882+0000","last_unstale":"2024-04-26T16:27:36.160665+0000","last_undegraded":"2024-04-26T16:27:36.160665+0000","last_fullsized":"2024-04-26T16:27:36.160665+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:02.646805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.646805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":156,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148605+0000","last_change":"2024-04-26T16:26:23.167482+0000","last_active":"2024-04-26T16:27:36.148605+0000","last_peered":"2024-04-26T16:27:36.148605+0000","last_clean":"2024-04-26T16:27:36.148605+0000","last_became_active":"2024-04-26T16:26:23.166910+0000","last_became_peered":"2024-04-26T16:26:23.166910+0000","last_unstale":"2024-04-26T16:27:36.148605+0000","last_undegraded":"2024-04-26T16:27:36.148605+0000","last_fullsized":"2024-04-26T16:27:36.148605+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:10.120887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:10.120887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":656,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144792+0000","last_change":"2024-04-26T16:26:29.440685+0000","last_active":"2024-04-26T16:27:36.144792+0000","last_peered":"2024-04-26T16:27:36.144792+0000","last_clean":"2024-04-26T16:27:36.144792+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:27:36.144792+0000","last_undegraded":"2024-04-26T16:27:36.144792+0000","last_fullsized":"2024-04-26T16:27:36.144792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:29.440599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:26:29.440599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":521,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.150089+0000","last_change":"2024-04-26T16:27:30.194881+0000","last_active":"2024-04-26T16:27:36.150089+0000","last_peered":"2024-04-26T16:27:36.150089+0000","last_clean":"2024-04-26T16:27:36.150089+0000","last_became_active":"2024-04-26T16:23:29.295178+0000","last_became_peered":"2024-04-26T16:23:29.295178+0000","last_unstale":"2024-04-26T16:27:36.150089+0000","last_undegraded":"2024-04-26T16:27:36.150089+0000","last_fullsized":"2024-04-26T16:27:36.150089+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:30.194740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:30.194740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:32.043292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"86'15","reported_seq":226,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.158758+0000","last_change":"2024-04-26T16:27:31.949939+0000","last_active":"2024-04-26T16:27:36.158758+0000","last_peered":"2024-04-26T16:27:36.158758+0000","last_clean":"2024-04-26T16:27:36.158758+0000","last_became_active":"2024-04-26T16:26:23.198479+0000","last_became_peered":"2024-04-26T16:26:23.198479+0000","last_unstale":"2024-04-26T16:27:36.158758+0000","last_undegraded":"2024-04-26T16:27:36.158758+0000","last_fullsized":"2024-04-26T16:27:36.158758+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'15","last_scrub_stamp":"2024-04-26T16:27:31.949816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:31.949816+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:54.292158+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":671,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.167004+0000","last_change":"2024-04-26T16:26:18.898345+0000","last_active":"2024-04-26T16:27:36.167004+0000","last_peered":"2024-04-26T16:27:36.167004+0000","last_clean":"2024-04-26T16:27:36.167004+0000","last_became_active":"2024-04-26T16:17:29.022163+0000","last_became_peered":"2024-04-26T16:17:29.022163+0000","last_unstale":"2024-04-26T16:27:36.167004+0000","last_undegraded":"2024-04-26T16:27:36.167004+0000","last_fullsized":"2024-04-26T16:27:36.167004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:18.898227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:26:18.898227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":655,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.156529+0000","last_change":"2024-04-26T16:26:05.055934+0000","last_active":"2024-04-26T16:27:36.156529+0000","last_peered":"2024-04-26T16:27:36.156529+0000","last_clean":"2024-04-26T16:27:36.156529+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:27:36.156529+0000","last_undegraded":"2024-04-26T16:27:36.156529+0000","last_fullsized":"2024-04-26T16:27:36.156529+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:05.055778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:05.055778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":174,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138846+0000","last_change":"2024-04-26T16:26:23.257796+0000","last_active":"2024-04-26T16:27:36.138846+0000","last_peered":"2024-04-26T16:27:36.138846+0000","last_clean":"2024-04-26T16:27:36.138846+0000","last_became_active":"2024-04-26T16:26:23.212770+0000","last_became_peered":"2024-04-26T16:26:23.212770+0000","last_unstale":"2024-04-26T16:27:36.138846+0000","last_undegraded":"2024-04-26T16:27:36.138846+0000","last_fullsized":"2024-04-26T16:27:36.138846+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:25:03.570129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:25:03.570129+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":520,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166584+0000","last_change":"2024-04-26T16:26:26.748501+0000","last_active":"2024-04-26T16:27:36.166584+0000","last_peered":"2024-04-26T16:27:36.166584+0000","last_clean":"2024-04-26T16:27:36.166584+0000","last_became_active":"2024-04-26T16:26:23.180322+0000","last_became_peered":"2024-04-26T16:26:23.180322+0000","last_unstale":"2024-04-26T16:27:36.166584+0000","last_undegraded":"2024-04-26T16:27:36.166584+0000","last_fullsized":"2024-04-26T16:27:36.166584+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:26:26.748278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:45.902857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"86'11","reported_seq":709,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154679+0000","last_change":"2024-04-26T16:26:26.371032+0000","last_active":"2024-04-26T16:27:36.154679+0000","last_peered":"2024-04-26T16:27:36.154679+0000","last_clean":"2024-04-26T16:27:36.154679+0000","last_became_active":"2024-04-26T16:26:23.175808+0000","last_became_peered":"2024-04-26T16:26:23.175808+0000","last_unstale":"2024-04-26T16:27:36.154679+0000","last_undegraded":"2024-04-26T16:27:36.154679+0000","last_fullsized":"2024-04-26T16:27:36.154679+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:26:26.370733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:26:26.370733+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:53.751413+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":674,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.149781+0000","last_change":"2024-04-26T16:26:23.214875+0000","last_active":"2024-04-26T16:27:36.149781+0000","last_peered":"2024-04-26T16:27:36.149781+0000","last_clean":"2024-04-26T16:27:36.149781+0000","last_became_active":"2024-04-26T16:26:23.214404+0000","last_became_peered":"2024-04-26T16:26:23.214404+0000","last_unstale":"2024-04-26T16:27:36.149781+0000","last_undegraded":"2024-04-26T16:27:36.149781+0000","last_fullsized":"2024-04-26T16:27:36.149781+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:24:37.085072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:24:37.085072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":671,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.138241+0000","last_change":"2024-04-26T16:26:23.179172+0000","last_active":"2024-04-26T16:27:36.138241+0000","last_peered":"2024-04-26T16:27:36.138241+0000","last_clean":"2024-04-26T16:27:36.138241+0000","last_became_active":"2024-04-26T16:26:23.178678+0000","last_became_peered":"2024-04-26T16:26:23.178678+0000","last_unstale":"2024-04-26T16:27:36.138241+0000","last_undegraded":"2024-04-26T16:27:36.138241+0000","last_fullsized":"2024-04-26T16:27:36.138241+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:08.578831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:26:08.578831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":667,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.158714+0000","last_change":"2024-04-26T16:26:23.198835+0000","last_active":"2024-04-26T16:27:36.158714+0000","last_peered":"2024-04-26T16:27:36.158714+0000","last_clean":"2024-04-26T16:27:36.158714+0000","last_became_active":"2024-04-26T16:26:23.198295+0000","last_became_peered":"2024-04-26T16:26:23.198295+0000","last_unstale":"2024-04-26T16:27:36.158714+0000","last_undegraded":"2024-04-26T16:27:36.158714+0000","last_fullsized":"2024-04-26T16:27:36.158714+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:09.777360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:25:09.777360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":158,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.166248+0000","last_change":"2024-04-26T16:26:23.200674+0000","last_active":"2024-04-26T16:27:36.166248+0000","last_peered":"2024-04-26T16:27:36.166248+0000","last_clean":"2024-04-26T16:27:36.166248+0000","last_became_active":"2024-04-26T16:26:23.199414+0000","last_became_peered":"2024-04-26T16:26:23.199414+0000","last_unstale":"2024-04-26T16:27:36.166248+0000","last_undegraded":"2024-04-26T16:27:36.166248+0000","last_fullsized":"2024-04-26T16:27:36.166248+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:13.187620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:13.187620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:38.930972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"86'10","reported_seq":679,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.150208+0000","last_change":"2024-04-26T16:25:02.039962+0000","last_active":"2024-04-26T16:27:36.150208+0000","last_peered":"2024-04-26T16:27:36.150208+0000","last_clean":"2024-04-26T16:27:36.150208+0000","last_became_active":"2024-04-26T16:17:31.049997+0000","last_became_peered":"2024-04-26T16:17:31.049997+0000","last_unstale":"2024-04-26T16:27:36.150208+0000","last_undegraded":"2024-04-26T16:27:36.150208+0000","last_fullsized":"2024-04-26T16:27:36.150208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'10","last_scrub_stamp":"2024-04-26T16:25:02.039711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:25:02.039711+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":676,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.154799+0000","last_change":"2024-04-26T16:26:37.410556+0000","last_active":"2024-04-26T16:27:36.154799+0000","last_peered":"2024-04-26T16:27:36.154799+0000","last_clean":"2024-04-26T16:27:36.154799+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:27:36.154799+0000","last_undegraded":"2024-04-26T16:27:36.154799+0000","last_fullsized":"2024-04-26T16:27:36.154799+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:26:37.410252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:27:38.701675+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":665,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.148373+0000","last_change":"2024-04-26T16:25:04.787080+0000","last_active":"2024-04-26T16:27:36.148373+0000","last_peered":"2024-04-26T16:27:36.148373+0000","last_clean":"2024-04-26T16:27:36.148373+0000","last_became_active":"2024-04-26T16:17:33.070922+0000","last_became_peered":"2024-04-26T16:17:33.070922+0000","last_unstale":"2024-04-26T16:27:36.148373+0000","last_undegraded":"2024-04-26T16:27:36.148373+0000","last_fullsized":"2024-04-26T16:27:36.148373+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:25:04.786867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:25:04.786867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":655,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-04-26T16:27:36.144654+0000","last_change":"2024-04-26T16:27:27.071533+0000","last_active":"2024-04-26T16:27:36.144654+0000","last_peered":"2024-04-26T16:27:36.144654+0000","last_clean":"2024-04-26T16:27:36.144654+0000","last_became_active":"2024-04-26T16:17:35.677526+0000","last_became_peered":"2024-04-26T16:17:35.677526+0000","last_unstale":"2024-04-26T16:27:36.144654+0000","last_undegraded":"2024-04-26T16:27:36.144654+0000","last_fullsized":"2024-04-26T16:27:36.144654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:27.071329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:27:27.071329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:43.609183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":284,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":109,"num_pg":32,"stat_sum":{"num_bytes":8195524,"num_objects":124,"num_object_clones":2,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124,"num_whiteouts":0,"num_read":99,"num_read_kb":4,"num_write":551,"num_write_kb":12642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":227,"num_bytes_recovered":4194817,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26075136,"data_stored":24586572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1813,"ondisk_log_size":1813,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":745,"num_read_kb":502,"num_write":449,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":512,"ondisk_log_size":512,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":586,"num_read_kb":581,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743556,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74240,"kb_used_data":6944,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670912,"statfs":{"total":95995035648,"available":95919013888,"internally_reserved":0,"allocated":7110656,"data_stored":4409347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":57,"seq":244813136014,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64384,"kb_used_data":6840,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93680768,"statfs":{"total":95995035648,"available":95929106432,"internally_reserved":0,"allocated":7004160,"data_stored":4417519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":58915273},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575042,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85804,"kb_used_data":14336,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93659348,"statfs":{"total":95995035648,"available":95907172352,"internally_reserved":0,"allocated":14680064,"data_stored":12022803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70048,"kb_used_data":6900,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93675104,"statfs":{"total":95995035648,"available":95923306496,"internally_reserved":0,"allocated":7065600,"data_stored":4423842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411511,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85836,"kb_used_data":10232,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659316,"statfs":{"total":95995035648,"available":95907139584,"internally_reserved":0,"allocated":10477568,"data_stored":7831654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084460,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99668,"kb_used_data":14376,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93645484,"statfs":{"total":95995035648,"available":95892975616,"internally_reserved":0,"allocated":14721024,"data_stored":12026481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":87292360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855522,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73540,"kb_used_data":10404,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93671612,"statfs":{"total":95995035648,"available":95919730688,"internally_reserved":0,"allocated":10653696,"data_stored":8011538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176128,"data_stored":404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8409088,"data_stored":8194848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4000544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8458240,"data_stored":8194959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4395008,"data_stored":4194806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":258048,"data_stored":637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":284,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:27:41.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:41.753 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:41.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:41.885 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:41.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:41.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:41.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:42.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:42.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:42.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:42.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:42.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:42.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:42.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:42.538 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:42.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:42.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:42.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:40.725755+0000 mgr.y (mgr.24401) 405 : audit [DBG] from='client.30525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:27:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:40.725755+0000 mgr.y (mgr.24401) 405 : audit [DBG] from='client.30525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:27:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.189901+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.189901+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.192024+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.192024+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.619834+0000 mgr.y (mgr.24401) 406 : cluster [DBG] pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:41.619834+0000 mgr.y (mgr.24401) 406 : cluster [DBG] pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:41.634020+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:41.634020+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:41.637097+0000 mon.c (mon.2) 579 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:41.637097+0000 mon.c (mon.2) 579 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.150566+0000 mon.a (mon.0) 2506 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.150566+0000 mon.a (mon.0) 2506 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.150678+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.150678+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.154847+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.154847+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.160757+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.160757+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.163013+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: audit 2024-04-26T16:27:42.163013+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.263900+0000 osd.2 (osd.2) 191 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.263900+0000 osd.2 (osd.2) 191 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.265770+0000 osd.2 (osd.2) 192 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:27:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:42 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.265770+0000 osd.2 (osd.2) 192 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:27:42.799 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:42.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:40.725755+0000 mgr.y (mgr.24401) 405 : audit [DBG] from='client.30525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:27:42.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:40.725755+0000 mgr.y (mgr.24401) 405 : audit [DBG] from='client.30525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.189901+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.189901+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.192024+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.192024+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.619834+0000 mgr.y (mgr.24401) 406 : cluster [DBG] pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:41.619834+0000 mgr.y (mgr.24401) 406 : cluster [DBG] pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:41.634020+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:41.634020+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:41.637097+0000 mon.c (mon.2) 579 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:41.637097+0000 mon.c (mon.2) 579 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.150566+0000 mon.a (mon.0) 2506 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.150566+0000 mon.a (mon.0) 2506 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.150678+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.150678+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.154847+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.154847+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:40.725755+0000 mgr.y (mgr.24401) 405 : audit [DBG] from='client.30525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:27:42.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:40.725755+0000 mgr.y (mgr.24401) 405 : audit [DBG] from='client.30525 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.189901+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.189901+0000 osd.1 (osd.1) 217 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.192024+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.192024+0000 osd.1 (osd.1) 218 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.619834+0000 mgr.y (mgr.24401) 406 : cluster [DBG] pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:41.619834+0000 mgr.y (mgr.24401) 406 : cluster [DBG] pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:41.634020+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:41.634020+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:41.637097+0000 mon.c (mon.2) 579 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:41.637097+0000 mon.c (mon.2) 579 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.150566+0000 mon.a (mon.0) 2506 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.150566+0000 mon.a (mon.0) 2506 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi050-73201-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.150678+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.150678+0000 mon.a (mon.0) 2507 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.154847+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.154847+0000 mon.a (mon.0) 2508 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.160757+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.160757+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.163013+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: audit 2024-04-26T16:27:42.163013+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.263900+0000 osd.2 (osd.2) 191 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.263900+0000 osd.2 (osd.2) 191 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.265770+0000 osd.2 (osd.2) 192 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.265770+0000 osd.2 (osd.2) 192 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.160757+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.160757+0000 mon.a (mon.0) 2509 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.163013+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: audit 2024-04-26T16:27:42.163013+0000 mon.b (mon.1) 270 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:27:42.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.263900+0000 osd.2 (osd.2) 191 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:27:42.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.263900+0000 osd.2 (osd.2) 191 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:27:42.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.265770+0000 osd.2 (osd.2) 192 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:27:42.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:42 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.265770+0000 osd.2 (osd.2) 192 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:27:42.901 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:42.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:42.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:43.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:43.062 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:43.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:43.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:43.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:43.326 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:43.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:43.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:43.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:43.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:43.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:43.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:43.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:43.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:43.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:43.850 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.181811+0000 osd.3 (osd.3) 253 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.181811+0000 osd.3 (osd.3) 253 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.184646+0000 osd.3 (osd.3) 254 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.184646+0000 osd.3 (osd.3) 254 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.294054+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.294054+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.295877+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:42.295877+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.151294+0000 mon.a (mon.0) 2510 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.151294+0000 mon.a (mon.0) 2510 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.152453+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.152453+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:27:43.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: audit 2024-04-26T16:27:43.154019+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: audit 2024-04-26T16:27:43.154019+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.156137+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.156137+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.162861+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.162861+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: audit 2024-04-26T16:27:43.165503+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: audit 2024-04-26T16:27:43.165503+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: audit 2024-04-26T16:27:43.166594+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: audit 2024-04-26T16:27:43.166594+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.181811+0000 osd.3 (osd.3) 253 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.181811+0000 osd.3 (osd.3) 253 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.184646+0000 osd.3 (osd.3) 254 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.184646+0000 osd.3 (osd.3) 254 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.294054+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.294054+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.295877+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:42.295877+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.151294+0000 mon.a (mon.0) 2510 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.151294+0000 mon.a (mon.0) 2510 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.152453+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.152453+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:27:43.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: audit 2024-04-26T16:27:43.154019+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: audit 2024-04-26T16:27:43.154019+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.156137+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.156137+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.162861+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.162861+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: audit 2024-04-26T16:27:43.165503+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: audit 2024-04-26T16:27:43.165503+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: audit 2024-04-26T16:27:43.166594+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: audit 2024-04-26T16:27:43.166594+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.326077+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.326077+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.326664+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.326664+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.326077+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.326077+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.326664+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:43.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:43 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.326664+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:43.953 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:43.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:43.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:44.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.181811+0000 osd.3 (osd.3) 253 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.181811+0000 osd.3 (osd.3) 253 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.184646+0000 osd.3 (osd.3) 254 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.184646+0000 osd.3 (osd.3) 254 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.294054+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.294054+0000 osd.0 (osd.0) 247 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.295877+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:42.295877+0000 osd.0 (osd.0) 248 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.151294+0000 mon.a (mon.0) 2510 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.151294+0000 mon.a (mon.0) 2510 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.152453+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.152453+0000 osd.4 (osd.4) 215 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: audit 2024-04-26T16:27:43.154019+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:44.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: audit 2024-04-26T16:27:43.154019+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.156137+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.156137+0000 osd.4 (osd.4) 216 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.162861+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.162861+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: audit 2024-04-26T16:27:43.165503+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: audit 2024-04-26T16:27:43.165503+0000 mon.a (mon.0) 2513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: audit 2024-04-26T16:27:43.166594+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: audit 2024-04-26T16:27:43.166594+0000 mon.b (mon.1) 271 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.326077+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.326077+0000 osd.5 (osd.5) 203 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.326664+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:44.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:43 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.326664+0000 osd.5 (osd.5) 204 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:27:44.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:44.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:44.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:44.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:44.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:44.374 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:44.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:44.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:44.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:44.619 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:27:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:27:44.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:44.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:44.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:44.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:44.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.103875+0000 osd.7 (osd.7) 245 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:27:44.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.103875+0000 osd.7 (osd.7) 245 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:27:44.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.106904+0000 osd.7 (osd.7) 246 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:27:44.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.106904+0000 osd.7 (osd.7) 246 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.268497+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.268497+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.270888+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.270888+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.620459+0000 mgr.y (mgr.24401) 407 : cluster [DBG] pgmap v577: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:43.620459+0000 mgr.y (mgr.24401) 407 : cluster [DBG] pgmap v577: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.153841+0000 mon.a (mon.0) 2514 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.153841+0000 mon.a (mon.0) 2514 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.162179+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.162179+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.166388+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.166388+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.169228+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:44.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.169228+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.171629+0000 mon.a (mon.0) 2518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.171629+0000 mon.a (mon.0) 2518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.103875+0000 osd.7 (osd.7) 245 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.103875+0000 osd.7 (osd.7) 245 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.106904+0000 osd.7 (osd.7) 246 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.106904+0000 osd.7 (osd.7) 246 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.268497+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.268497+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.270888+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.270888+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:27:44.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.620459+0000 mgr.y (mgr.24401) 407 : cluster [DBG] pgmap v577: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:43.620459+0000 mgr.y (mgr.24401) 407 : cluster [DBG] pgmap v577: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.153841+0000 mon.a (mon.0) 2514 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.153841+0000 mon.a (mon.0) 2514 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.162179+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.162179+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.166388+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.166388+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.169228+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.169228+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.171629+0000 mon.a (mon.0) 2518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.171629+0000 mon.a (mon.0) 2518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.173740+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: audit 2024-04-26T16:27:44.173740+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.196771+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub starts 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.196771+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub starts 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.200224+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub ok 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.200224+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub ok 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.173740+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: audit 2024-04-26T16:27:44.173740+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.196771+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub starts 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.196771+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub starts 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.200224+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub ok 2024-04-26T16:27:44.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:44 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.200224+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub ok 2024-04-26T16:27:44.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:44.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:45.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:45.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:45.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.103875+0000 osd.7 (osd.7) 245 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.103875+0000 osd.7 (osd.7) 245 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.106904+0000 osd.7 (osd.7) 246 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.106904+0000 osd.7 (osd.7) 246 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.268497+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.268497+0000 osd.6 (osd.6) 183 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.270888+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.270888+0000 osd.6 (osd.6) 184 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.620459+0000 mgr.y (mgr.24401) 407 : cluster [DBG] pgmap v577: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:43.620459+0000 mgr.y (mgr.24401) 407 : cluster [DBG] pgmap v577: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail; 96 KiB/s rd, 5.0 KiB/s wr, 164 op/s 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.153841+0000 mon.a (mon.0) 2514 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.153841+0000 mon.a (mon.0) 2514 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.162179+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.162179+0000 mon.a (mon.0) 2515 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:27:45.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.166388+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.166388+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.169228+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.169228+0000 mon.a (mon.0) 2517 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.171629+0000 mon.a (mon.0) 2518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.171629+0000 mon.a (mon.0) 2518 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.173740+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: audit 2024-04-26T16:27:44.173740+0000 mon.b (mon.1) 272 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.196771+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub starts 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.196771+0000 osd.4 (osd.4) 217 : cluster [DBG] 2.f scrub starts 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.200224+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub ok 2024-04-26T16:27:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:44 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.200224+0000 osd.4 (osd.4) 218 : cluster [DBG] 2.f scrub ok 2024-04-26T16:27:45.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:45.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:45.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:45.306 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:45.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:45.439 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:45.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:45.570 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:45.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:45.701 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:45.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:45.816 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.067844+0000 osd.7 (osd.7) 247 : cluster [DBG] 11.12 deep-scrub starts 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.067844+0000 osd.7 (osd.7) 247 : cluster [DBG] 11.12 deep-scrub starts 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.070130+0000 osd.7 (osd.7) 248 : cluster [DBG] 11.12 deep-scrub ok 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.070130+0000 osd.7 (osd.7) 248 : cluster [DBG] 11.12 deep-scrub ok 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.306946+0000 osd.0 (osd.0) 249 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.306946+0000 osd.0 (osd.0) 249 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.308534+0000 osd.0 (osd.0) 250 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:44.308534+0000 osd.0 (osd.0) 250 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.164976+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.164976+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.165087+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:27:45.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.165087+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.169462+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.169462+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.170441+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.170441+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.175045+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.175045+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.177165+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: audit 2024-04-26T16:27:45.177165+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.185415+0000 osd.2 (osd.2) 193 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.185415+0000 osd.2 (osd.2) 193 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.186844+0000 osd.2 (osd.2) 194 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:27:45.818 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:45 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.186844+0000 osd.2 (osd.2) 194 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:27:45.833 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:45.921 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.067844+0000 osd.7 (osd.7) 247 : cluster [DBG] 11.12 deep-scrub starts 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.067844+0000 osd.7 (osd.7) 247 : cluster [DBG] 11.12 deep-scrub starts 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.070130+0000 osd.7 (osd.7) 248 : cluster [DBG] 11.12 deep-scrub ok 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.070130+0000 osd.7 (osd.7) 248 : cluster [DBG] 11.12 deep-scrub ok 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.306946+0000 osd.0 (osd.0) 249 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.306946+0000 osd.0 (osd.0) 249 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.308534+0000 osd.0 (osd.0) 250 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:44.308534+0000 osd.0 (osd.0) 250 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.164976+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.164976+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.165087+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.165087+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.169462+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-04-26T16:27:45.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.169462+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.170441+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.170441+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.175045+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.175045+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.177165+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: audit 2024-04-26T16:27:45.177165+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.185415+0000 osd.2 (osd.2) 193 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.185415+0000 osd.2 (osd.2) 193 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.186844+0000 osd.2 (osd.2) 194 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:27:45.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.186844+0000 osd.2 (osd.2) 194 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.067844+0000 osd.7 (osd.7) 247 : cluster [DBG] 11.12 deep-scrub starts 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.067844+0000 osd.7 (osd.7) 247 : cluster [DBG] 11.12 deep-scrub starts 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.070130+0000 osd.7 (osd.7) 248 : cluster [DBG] 11.12 deep-scrub ok 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.070130+0000 osd.7 (osd.7) 248 : cluster [DBG] 11.12 deep-scrub ok 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.306946+0000 osd.0 (osd.0) 249 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.306946+0000 osd.0 (osd.0) 249 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.308534+0000 osd.0 (osd.0) 250 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:44.308534+0000 osd.0 (osd.0) 250 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:27:45.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.164976+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.164976+0000 mon.a (mon.0) 2519 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.165087+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.165087+0000 mon.a (mon.0) 2520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.169462+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.169462+0000 mon.a (mon.0) 2521 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.170441+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.170441+0000 mon.a (mon.0) 2522 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]: dispatch 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.175045+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.175045+0000 mon.a (mon.0) 2523 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.177165+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: audit 2024-04-26T16:27:45.177165+0000 mon.b (mon.1) 273 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.185415+0000 osd.2 (osd.2) 193 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.185415+0000 osd.2 (osd.2) 193 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.186844+0000 osd.2 (osd.2) 194 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:27:45.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:45 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.186844+0000 osd.2 (osd.2) 194 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:27:45.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:45.949 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:27:45.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:45.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:46.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:46.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:46.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:46.225 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:46.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:46.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:46.355 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:27:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:27:46.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:46.488 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:46.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:46.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:46.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:46.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:46.726 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:27:46.727 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:27:46.727 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-04-26T16:27:46.727 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T16:27:46.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6 2024-04-26T16:27:46.745 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:46.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.165864+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.165864+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.180897+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.180897+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.271406+0000 osd.1 (osd.1) 219 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.271406+0000 osd.1 (osd.1) 219 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:27:46.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.273666+0000 osd.1 (osd.1) 220 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.273666+0000 osd.1 (osd.1) 220 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.621345+0000 mgr.y (mgr.24401) 408 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:45.621345+0000 mgr.y (mgr.24401) 408 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.131715+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.131715+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.131795+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.131795+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.136378+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.136378+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.159424+0000 mon.c (mon.2) 580 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.159424+0000 mon.c (mon.2) 580 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.159806+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.159806+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.160436+0000 mon.c (mon.2) 581 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.160436+0000 mon.c (mon.2) 581 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.160836+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.160836+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.161394+0000 mon.c (mon.2) 582 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.161394+0000 mon.c (mon.2) 582 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.161792+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.161792+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.212945+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.212945+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.215180+0000 mon.b (mon.1) 274 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.863 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[21463]: audit 2024-04-26T16:27:46.215180+0000 mon.b (mon.1) 274 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.165864+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.165864+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.180897+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.180897+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.271406+0000 osd.1 (osd.1) 219 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.271406+0000 osd.1 (osd.1) 219 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.273666+0000 osd.1 (osd.1) 220 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.273666+0000 osd.1 (osd.1) 220 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:27:46.864 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.621345+0000 mgr.y (mgr.24401) 408 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:45.621345+0000 mgr.y (mgr.24401) 408 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.131715+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.131715+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.131795+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.131795+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.136378+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.136378+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.159424+0000 mon.c (mon.2) 580 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.159424+0000 mon.c (mon.2) 580 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.159806+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.159806+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.160436+0000 mon.c (mon.2) 581 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.160436+0000 mon.c (mon.2) 581 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.160836+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.865 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.160836+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.161394+0000 mon.c (mon.2) 582 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.161394+0000 mon.c (mon.2) 582 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.161792+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.161792+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.212945+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.212945+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.215180+0000 mon.b (mon.1) 274 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.866 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:46 smithi050 bash[29234]: audit 2024-04-26T16:27:46.215180+0000 mon.b (mon.1) 274 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:46.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:46.939 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:27:46.939 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:27:46 smithi190 systemd[1]: Stopping Ceph osd.6 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.165864+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.165864+0000 osd.3 (osd.3) 255 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.180897+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.180897+0000 osd.3 (osd.3) 256 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.271406+0000 osd.1 (osd.1) 219 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.271406+0000 osd.1 (osd.1) 219 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.273666+0000 osd.1 (osd.1) 220 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:27:46.940 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.273666+0000 osd.1 (osd.1) 220 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.621345+0000 mgr.y (mgr.24401) 408 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:45.621345+0000 mgr.y (mgr.24401) 408 : cluster [DBG] pgmap v580: 228 pgs: 228 active+clean; 8.4 MiB data, 541 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.131715+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.131715+0000 mon.a (mon.0) 2524 : audit [INF] from='client.? 172.21.15.50:0/2514584246' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi050-73201-62"}]': finished 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.131795+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.131795+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.136378+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.136378+0000 mon.a (mon.0) 2526 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.159424+0000 mon.c (mon.2) 580 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.159424+0000 mon.c (mon.2) 580 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.159806+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.159806+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.941 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.160436+0000 mon.c (mon.2) 581 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.160436+0000 mon.c (mon.2) 581 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.160836+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.160836+0000 mon.a (mon.0) 2528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.161394+0000 mon.c (mon.2) 582 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.161394+0000 mon.c (mon.2) 582 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.161792+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.161792+0000 mon.a (mon.0) 2529 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.212945+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.212945+0000 mon.a (mon.0) 2530 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.215180+0000 mon.b (mon.1) 274 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.942 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[23868]: audit 2024-04-26T16:27:46.215180+0000 mon.b (mon.1) 274 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:46.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:46.991 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[38017]: debug 2024-04-26T16:27:46.941+0000 7fde4e3ef700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:27:46.991 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[38017]: debug 2024-04-26T16:27:46.941+0000 7fde4e3ef700 -1 osd.6 398 *** Got signal Terminated *** 2024-04-26T16:27:46.991 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:27:46 smithi190 bash[38017]: debug 2024-04-26T16:27:46.941+0000 7fde4e3ef700 -1 osd.6 398 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:27:47.004 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:47.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:47.143 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:47.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:47.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:47.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:47.403 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:47.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:47.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:47.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:47.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.272932+0000 osd.6 (osd.6) 185 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:27:47.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.272932+0000 osd.6 (osd.6) 185 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:27:47.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.276046+0000 osd.6 (osd.6) 186 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.276046+0000 osd.6 (osd.6) 186 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.946025+0000 mon.a (mon.0) 2531 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:46.946025+0000 mon.a (mon.0) 2531 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.133469+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.133469+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.136210+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:47.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.136210+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:47.669 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:47.772 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.272932+0000 osd.6 (osd.6) 185 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.272932+0000 osd.6 (osd.6) 185 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.276046+0000 osd.6 (osd.6) 186 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.276046+0000 osd.6 (osd.6) 186 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.946025+0000 mon.a (mon.0) 2531 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:46.946025+0000 mon.a (mon.0) 2531 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:27:47.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.133469+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.133469+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.136210+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.136210+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.136398+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.136398+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.144655+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e399: 8 total, 7 up, 7 in 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.144655+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e399: 8 total, 7 up, 7 in 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.146436+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.146436+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.151165+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.151165+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.272932+0000 osd.6 (osd.6) 185 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.272932+0000 osd.6 (osd.6) 185 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.276046+0000 osd.6 (osd.6) 186 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.276046+0000 osd.6 (osd.6) 186 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.946025+0000 mon.a (mon.0) 2531 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:46.946025+0000 mon.a (mon.0) 2531 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.133469+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.133469+0000 mon.a (mon.0) 2532 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.136210+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.136210+0000 mon.a (mon.0) 2533 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi050-73201-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.136398+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.136398+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:27:47.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.144655+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e399: 8 total, 7 up, 7 in 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.144655+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e399: 8 total, 7 up, 7 in 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.146436+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.146436+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.151165+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.151165+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.152454+0000 mon.c (mon.2) 583 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.152454+0000 mon.c (mon.2) 583 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.161222+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: audit 2024-04-26T16:27:47.161222+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.787 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.202000+0000 osd.4 (osd.4) 219 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.202000+0000 osd.4 (osd.4) 219 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.205368+0000 osd.4 (osd.4) 220 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.205368+0000 osd.4 (osd.4) 220 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.152454+0000 mon.c (mon.2) 583 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.152454+0000 mon.c (mon.2) 583 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.161222+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: audit 2024-04-26T16:27:47.161222+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.202000+0000 osd.4 (osd.4) 219 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.202000+0000 osd.4 (osd.4) 219 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.205368+0000 osd.4 (osd.4) 220 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:27:47.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:47 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.205368+0000 osd.4 (osd.4) 220 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:27:47.800 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:47.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.136398+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.136398+0000 mon.a (mon.0) 2534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.144655+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e399: 8 total, 7 up, 7 in 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.144655+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e399: 8 total, 7 up, 7 in 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.146436+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.146436+0000 mon.b (mon.1) 275 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.151165+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.151165+0000 mon.a (mon.0) 2536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.152454+0000 mon.c (mon.2) 583 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.152454+0000 mon.c (mon.2) 583 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.161222+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: audit 2024-04-26T16:27:47.161222+0000 mon.a (mon.0) 2537 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.202000+0000 osd.4 (osd.4) 219 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.202000+0000 osd.4 (osd.4) 219 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.205368+0000 osd.4 (osd.4) 220 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:27:47.915 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:47 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.205368+0000 osd.4 (osd.4) 220 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:27:47.928 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:47.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:47.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:48.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:48.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:48.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:48.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:48.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:48.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:46.834934+0000 mgr.y (mgr.24401) 409 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:46.834934+0000 mgr.y (mgr.24401) 409 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:48.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.197187+0000 osd.3 (osd.3) 257 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:27:48.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.197187+0000 osd.3 (osd.3) 257 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:27:48.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.200060+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:27:48.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.727 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:48.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:48.820 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[68056]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-6 2024-04-26T16:27:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:46.834934+0000 mgr.y (mgr.24401) 409 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:46.834934+0000 mgr.y (mgr.24401) 409 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.197187+0000 osd.3 (osd.3) 257 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:27:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.197187+0000 osd.3 (osd.3) 257 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:27:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.200060+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:27:48.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.200060+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.622232+0000 mgr.y (mgr.24401) 410 : cluster [DBG] pgmap v583: 228 pgs: 23 stale+active+clean, 205 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:47.622232+0000 mgr.y (mgr.24401) 410 : cluster [DBG] pgmap v583: 228 pgs: 23 stale+active+clean, 205 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.145641+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.145641+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:48.214414+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e400: 8 total, 7 up, 7 in 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: cluster 2024-04-26T16:27:48.214414+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e400: 8 total, 7 up, 7 in 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.226010+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.226010+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.226795+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.226795+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.228552+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.228552+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.229479+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[23868]: audit 2024-04-26T16:27:48.229479+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:48.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:48.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.200060+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:27:48.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.622232+0000 mgr.y (mgr.24401) 410 : cluster [DBG] pgmap v583: 228 pgs: 23 stale+active+clean, 205 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:48.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:47.622232+0000 mgr.y (mgr.24401) 410 : cluster [DBG] pgmap v583: 228 pgs: 23 stale+active+clean, 205 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:48.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.145641+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:48.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.145641+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:48.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:48.214414+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e400: 8 total, 7 up, 7 in 2024-04-26T16:27:48.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: cluster 2024-04-26T16:27:48.214414+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e400: 8 total, 7 up, 7 in 2024-04-26T16:27:48.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.226010+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.226010+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.226795+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.226795+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.228552+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.228552+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.229479+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.895 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[21463]: audit 2024-04-26T16:27:48.229479+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:46.834934+0000 mgr.y (mgr.24401) 409 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:48.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:46.834934+0000 mgr.y (mgr.24401) 409 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:48.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.197187+0000 osd.3 (osd.3) 257 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:27:48.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.197187+0000 osd.3 (osd.3) 257 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:27:48.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.200060+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:27:48.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.200060+0000 osd.3 (osd.3) 258 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:27:48.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.622232+0000 mgr.y (mgr.24401) 410 : cluster [DBG] pgmap v583: 228 pgs: 23 stale+active+clean, 205 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:48.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:47.622232+0000 mgr.y (mgr.24401) 410 : cluster [DBG] pgmap v583: 228 pgs: 23 stale+active+clean, 205 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.145641+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.145641+0000 mon.a (mon.0) 2538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]': finished 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:48.214414+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e400: 8 total, 7 up, 7 in 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: cluster 2024-04-26T16:27:48.214414+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e400: 8 total, 7 up, 7 in 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.226010+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.226010+0000 mon.a (mon.0) 2540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.226795+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.226795+0000 mon.a (mon.0) 2541 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.228552+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.228552+0000 mon.b (mon.1) 276 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:27:48.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.229479+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.898 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:48 smithi050 bash[29234]: audit 2024-04-26T16:27:48.229479+0000 mon.b (mon.1) 277 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-47"}]: dispatch 2024-04-26T16:27:48.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:48.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:48.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:48.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/2206672225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 msgr2=0x7f10a81098d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- 172.21.15.50:0/2206672225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 0x7f10a81098d0 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f10900098f0 tx=0x7f1090009c00 comp rx=0 tx=0).stop 2024-04-26T16:27:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/2206672225 shutdown_connections 2024-04-26T16:27:49.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- 172.21.15.50:0/2206672225 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10a8103e90 0x7f10a81102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- 172.21.15.50:0/2206672225 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 0x7f10a81098d0 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- 172.21.15.50:0/2206672225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10a81034d0 0x7f10a8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/2206672225 >> 172.21.15.50:0/2206672225 conn(0x7f10a807a500 msgr2=0x7f10a807a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/2206672225 shutdown_connections 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/2206672225 wait complete. 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 librados: starting msgr at 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 librados: starting objecter 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 Processor -- start 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- start start 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 librados: setting wanted keys 2024-04-26T16:27:49.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 librados: calling monclient init 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10a81034d0 0x7f10a8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10a8103e90 0x7f10a8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 0x7f10a80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8112ac0 con 0x7f10a81094f0 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8112940 con 0x7f10a8103e90 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f10a8112c40 con 0x7f10a81034d0 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 0x7f10a80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 0x7f10a80761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60726/0 (socket says 172.21.15.50:60726) 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 -- 172.21.15.50:0/83204712 learned_addr learned my addr 172.21.15.50:0/83204712 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10a81034d0 0x7f10a8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10a8103e90 0x7f10a8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 -- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10a81034d0 msgr2=0x7f10a8075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f10a81034d0 0x7f10a8075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 -- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10a8103e90 msgr2=0x7f10a8075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10a8103e90 0x7f10a8075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a81a7770 con 0x7f10a81094f0 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f10a81094f0 0x7f10a80761a0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f10a000e3c0 tx=0x7f10a000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a4ff9700 10 client.?.objecter ms_handle_connect 0x7f10a81094f0 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a4ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a00090d0 con 0x7f10a81094f0 2024-04-26T16:27:49.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f10a0009270 con 0x7f10a81094f0 2024-04-26T16:27:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0007c30 con 0x7f10a81094f0 2024-04-26T16:27:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a81a7a60 con 0x7f10a81094f0 2024-04-26T16:27:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 10 client.30537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a81a7f50 con 0x7f10a81094f0 2024-04-26T16:27:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 librados: init done 2024-04-26T16:27:49.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.019+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f10a806a110 con 0x7f10a81094f0 2024-04-26T16:27:49.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f10a0012050 con 0x7f10a81094f0 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1094071f80 0x7f1094074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(400..400 src has 87..400) ==== 9795+0+0 (secure 0 0 0) 0x7f10a0091b80 con 0x7f10a81094f0 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 osd_map(400..400 src has 87..400) 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 3 client.30537.objecter handle_osd_map got epochs [400,400] > 0 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 3 client.30537.objecter handle_osd_map decoding full epoch 400 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1094071f80 0x7f1094074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 20 client.30537.objecter dump_active .. 0 homeless 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v400) ==== 54+0+16 (secure 0 0 0) 0x7f10a0022060 con 0x7f10a81094f0 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 10 client.30537.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 1 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f10a8064030 0x7f10a8064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f1094071f80 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1094071f80 0x7f1094074440 secure :-1 s=READY pgs=3737 cs=0 l=1 rev1=1 crypto rx=0x7f10900098c0 tx=0x7f1090019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 1 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 0 1 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a8076c50 osd=0 4 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 10 client.30537.objecter _send_command 1 2024-04-26T16:27:49.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10a81ab3d0 con 0x7f10a8064030 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f10a8064030 0x7f10a8064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a8064030 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f10a8064030 0x7f10a8064410 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a81ab3d0 con 0x7f10a8064030 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 1: 0 ) 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 1 = system:0 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a8076c50 osd=0 4 2024-04-26T16:27:49.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.023+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 0 1 2024-04-26T16:27:49.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/3269514607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 msgr2=0x7fdca0110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- 172.21.15.50:0/3269514607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0110290 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009f90 tx=0x7fdc9800b330 comp rx=0 tx=0).stop 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/3269514607 shutdown_connections 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- 172.21.15.50:0/3269514607 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 0x7fdca01098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- 172.21.15.50:0/3269514607 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0110290 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- 172.21.15.50:0/3269514607 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 0x7fdca01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/3269514607 >> 172.21.15.50:0/3269514607 conn(0x7fdca007a580 msgr2=0x7fdca007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/3269514607 shutdown_connections 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/3269514607 wait complete. 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 librados: starting msgr at 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 librados: starting objecter 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 Processor -- start 2024-04-26T16:27:49.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- start start 2024-04-26T16:27:49.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 librados: setting wanted keys 2024-04-26T16:27:49.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 librados: calling monclient init 2024-04-26T16:27:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 0x7fdca0199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 0x7fdca019d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdca0112a60 con 0x7fdca0103470 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdca01128e0 con 0x7fdca01094c0 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fdca0112be0 con 0x7fdca0103e30 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 0x7fdca0199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 0x7fdca0199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:60742/0 (socket says 172.21.15.50:60742) 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9ffff700 1 -- 172.21.15.50:0/1523688200 learned_addr learned my addr 172.21.15.50:0/1523688200 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca4d18700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 0x7fdca019d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 -- 172.21.15.50:0/1523688200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 msgr2=0x7fdca019d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 0x7fdca019d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 -- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 msgr2=0x7fdca0199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 0x7fdca0199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:49.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 -- 172.21.15.50:0/1523688200 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca019e110 con 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca4d18700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 0x7fdca019d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc977fe700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0199640 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800eb90 tx=0x7fdc8800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9dffb700 10 client.?.objecter ms_handle_connect 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9dffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8800ccf0 con 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca019e400 con 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fdc88004480 con 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88003da0 con 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 10 client.41159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca01a5c80 con 0x7fdca0103e30 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.035+0000 7fdca677b700 1 librados: init done 2024-04-26T16:27:49.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca006a110 con 0x7fdca0103e30 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fdc88004020 con 0x7fdca0103e30 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc8c071f80 0x7fdc8c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(400..400 src has 87..400) ==== 9795+0+0 (secure 0 0 0) 0x7fdc88091ef0 con 0x7fdca0103e30 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 10 client.41159.objecter ms_dispatch 0x7fdca0001550 osd_map(400..400 src has 87..400) 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 3 client.41159.objecter handle_osd_map got epochs [400,400] > 0 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 3 client.41159.objecter handle_osd_map decoding full epoch 400 2024-04-26T16:27:49.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.039+0000 7fdc9dffb700 20 client.41159.objecter dump_active .. 0 homeless 2024-04-26T16:27:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.043+0000 7fdc9ffff700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc8c071f80 0x7fdc8c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.043+0000 7fdc9dffb700 10 client.41159.objecter ms_handle_connect 0x7fdc8c071f80 2024-04-26T16:27:49.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.043+0000 7fdc9ffff700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc8c071f80 0x7fdc8c074440 secure :-1 s=READY pgs=3738 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009ca0 tx=0x7fdc90009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.043+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fdc8805d700 con 0x7fdca0103e30 2024-04-26T16:27:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 10 client.30537.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 2 2024-04-26T16:27:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 2 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a8076c50 osd=0 3 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 0 2 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a8076c50 osd=0 4 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 10 client.30537.objecter _send_command 2 2024-04-26T16:27:49.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.051+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a8063ab0 con 0x7f10a8064030 2024-04-26T16:27:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a8063ab0 con 0x7f10a8064030 2024-04-26T16:27:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a8076c50 osd=0 4 2024-04-26T16:27:49.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 0 2 2024-04-26T16:27:49.054 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:27:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 10 client.30537.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 3 2024-04-26T16:27:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f10a802d310 0x7f10a802d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 1 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a9d0 osd=1 4 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 10 client.30537.objecter _send_command 3 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f10a810b050 con 0x7f10a802d310 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f10a802d310 0x7f10a802d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a802d310 2024-04-26T16:27:49.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f10a802d310 0x7f10a802d6f0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a810b050 con 0x7f10a802d310 2024-04-26T16:27:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 3: 0 ) 2024-04-26T16:27:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 3 = system:0 2024-04-26T16:27:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a810a9d0 osd=1 4 2024-04-26T16:27:49.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.055+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 1 3 2024-04-26T16:27:49.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 10 client.30537.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 4 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 4 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a810a9d0 osd=1 3 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 1 4 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a9d0 osd=1 4 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 10 client.30537.objecter _send_command 4 2024-04-26T16:27:49.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a810a450 con 0x7f10a802d310 2024-04-26T16:27:49.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a810a450 con 0x7f10a802d310 2024-04-26T16:27:49.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a810a9d0 osd=1 4 2024-04-26T16:27:49.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 1 4 2024-04-26T16:27:49.075 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 10 client.30537.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 5 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f10a81ad590 0x7f10a81af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 5 2024-04-26T16:27:49.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 2 5 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a850 osd=2 4 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 10 client.30537.objecter _send_command 5 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f10a81b27e0 con 0x7f10a81ad590 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f10a81ad590 0x7f10a81af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a81ad590 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f10a81ad590 0x7f10a81af980 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a81b27e0 con 0x7f10a81ad590 2024-04-26T16:27:49.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 5: 0 ) 2024-04-26T16:27:49.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 5 = system:0 2024-04-26T16:27:49.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a810a850 osd=2 4 2024-04-26T16:27:49.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.075+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 2 5 2024-04-26T16:27:49.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 10 client.30537.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 6 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 6 2024-04-26T16:27:49.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a810a850 osd=2 3 2024-04-26T16:27:49.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 2 6 2024-04-26T16:27:49.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a810a850 osd=2 4 2024-04-26T16:27:49.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 10 client.30537.objecter _send_command 6 2024-04-26T16:27:49.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.095+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a802de80 con 0x7f10a81ad590 2024-04-26T16:27:49.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a802de80 con 0x7f10a81ad590 2024-04-26T16:27:49.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a810a850 osd=2 4 2024-04-26T16:27:49.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 2 6 2024-04-26T16:27:49.098 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 10 client.30537.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 7 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f10a81aaf30 0x7f10a81b1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 7 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 3 7 2024-04-26T16:27:49.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a802e910 osd=3 4 2024-04-26T16:27:49.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 10 client.30537.objecter _send_command 7 2024-04-26T16:27:49.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f10a81b6140 con 0x7f10a81aaf30 2024-04-26T16:27:49.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f10a81aaf30 0x7f10a81b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a81aaf30 2024-04-26T16:27:49.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f10a81aaf30 0x7f10a81b1040 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a81b6140 con 0x7f10a81aaf30 2024-04-26T16:27:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 7: 0 ) 2024-04-26T16:27:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 7 = system:0 2024-04-26T16:27:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a802e910 osd=3 4 2024-04-26T16:27:49.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.099+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 3 7 2024-04-26T16:27:49.111 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 10 client.30537.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 8 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 8 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a802e910 osd=3 3 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 3 8 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a802e910 osd=3 4 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 10 client.30537.objecter _send_command 8 2024-04-26T16:27:49.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a81aa4a0 con 0x7f10a81aaf30 2024-04-26T16:27:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a81aa4a0 con 0x7f10a81aaf30 2024-04-26T16:27:49.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a802e910 osd=3 4 2024-04-26T16:27:49.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 3 8 2024-04-26T16:27:49.120 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 10 client.30537.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 9 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f10a81b0ba0 0x7f10a81b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 9 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 4 9 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa250 osd=4 4 2024-04-26T16:27:49.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 10 client.30537.objecter _send_command 9 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f10a81aa3f0 con 0x7f10a81b0ba0 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.119+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f10a81b0ba0 0x7f10a81b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a81b0ba0 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a77fe700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f10a81b0ba0 0x7f10a81b49a0 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a81aa3f0 con 0x7f10a81b0ba0 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 9: 0 ) 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 9 = system:0 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a81aa250 osd=4 4 2024-04-26T16:27:49.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.123+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 4 9 2024-04-26T16:27:49.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 10 client.30537.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 10 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 10 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a81aa250 osd=4 3 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 4 10 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa250 osd=4 4 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 10 client.30537.objecter _send_command 10 2024-04-26T16:27:49.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.139+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a81aa3f0 con 0x7f10a81b0ba0 2024-04-26T16:27:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a81aa3f0 con 0x7f10a81b0ba0 2024-04-26T16:27:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a81aa250 osd=4 4 2024-04-26T16:27:49.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 4 10 2024-04-26T16:27:49.148 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:27:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 10 client.30537.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 11 2024-04-26T16:27:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f10a81b53f0 0x7f10a81b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 11 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 5 11 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa3c0 osd=5 4 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 10 client.30537.objecter _send_command 11 2024-04-26T16:27:49.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.147+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f10a81bcc30 con 0x7f10a81b53f0 2024-04-26T16:27:49.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.155+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f10a81b53f0 0x7f10a81b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a6ffd700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f10a81b53f0 0x7f10a81b5850 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a81b53f0 2024-04-26T16:27:49.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a81bcc30 con 0x7f10a81b53f0 2024-04-26T16:27:49.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 11: 0 ) 2024-04-26T16:27:49.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 11 = system:0 2024-04-26T16:27:49.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a81aa3c0 osd=5 4 2024-04-26T16:27:49.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.167+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 5 11 2024-04-26T16:27:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 10 client.30537.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 12 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 12 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a81aa3c0 osd=5 3 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 5 12 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81aa3c0 osd=5 4 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 10 client.30537.objecter _send_command 12 2024-04-26T16:27:49.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.187+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a81ba310 con 0x7f10a81b53f0 2024-04-26T16:27:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a81ba310 con 0x7f10a81b53f0 2024-04-26T16:27:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a81aa3c0 osd=5 4 2024-04-26T16:27:49.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 5 12 2024-04-26T16:27:49.191 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 10 client.30537.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 13 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 20 client.30537.objecter _get_session osd=-1 returning homeless 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 10 client.30537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=401}) -- 0x7f10a81bcc30 con 0x7f10a81094f0 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f10a81bce40 con 0x7f10a81094f0 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 7 ==== osd_map(401..401 src has 87..401) ==== 709+0+0 (secure 0 0 0) 0x7f10a008f170 con 0x7f10a81094f0 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 osd_map(401..401 src has 87..401) 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 3 client.30537.objecter handle_osd_map got epochs [401,401] > 400 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 3 client.30537.objecter handle_osd_map decoding incremental epoch 401 2024-04-26T16:27:49.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 10 client.30537.objecter checking command 13 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 10 client.30537.objecter _check_command_map_dne tid 13 current 401 map_dne_bound 0 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 20 client.30537.objecter dump_active .. 1 homeless 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_get_version_reply(handle=1 version=401) ==== 24+0+0 (secure 0 0 0) 0x7f10a005c5f0 con 0x7f10a81094f0 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a5ffb700 10 client.30537.objecter _check_command_map_dne tid 13 current 401 map_dne_bound 401 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a5ffb700 10 client.30537.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a5ffb700 15 client.30537.objecter _session_command_op_remove -1 13 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 10 client.30537.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 14 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f10a81b2260 0x7f10a81bb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 14 2024-04-26T16:27:49.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 7 14 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81b0110 osd=7 4 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 10 client.30537.objecter _send_command 14 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f10a81b03e0 con 0x7f10a81b2260 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f10a81b2260 0x7f10a81bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a4ff9700 10 client.30537.objecter ms_handle_connect 0x7f10a81b2260 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.191+0000 7f10a7fff700 1 --2- 172.21.15.50:0/83204712 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f10a81b2260 0x7f10a81bb490 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:27:49.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.195+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10a81b03e0 con 0x7f10a81b2260 2024-04-26T16:27:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.195+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 14: 0 ) 2024-04-26T16:27:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.195+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 14 = system:0 2024-04-26T16:27:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.195+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a81b0110 osd=7 4 2024-04-26T16:27:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.195+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 7 14 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 10 client.30537.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign -1 15 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 20 client.30537.objecter _get_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 15 client.30537.objecter _session_command_op_remove -1 15 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 20 client.30537.objecter get_session s=0x7f10a81b0110 osd=7 3 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 15 client.30537.objecter _session_command_op_assign 7 15 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 20 client.30537.objecter put_session s=0x7f10a81b0110 osd=7 4 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 10 client.30537.objecter _send_command 15 2024-04-26T16:27:49.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.211+0000 7f10adf42700 1 -- 172.21.15.50:0/83204712 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f10a81b03e0 con 0x7f10a81b2260 2024-04-26T16:27:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.215+0000 7f10a4ff9700 1 -- 172.21.15.50:0/83204712 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f10a81b03e0 con 0x7f10a81b2260 2024-04-26T16:27:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.215+0000 7f10a4ff9700 10 client.30537.objecter ms_dispatch 0x7f10a8001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:27:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.215+0000 7f10a4ff9700 10 client.30537.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:27:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.215+0000 7f10a4ff9700 20 client.30537.objecter put_session s=0x7f10a81b0110 osd=7 4 2024-04-26T16:27:49.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.215+0000 7f10a4ff9700 15 client.30537.objecter _session_command_op_remove 7 15 2024-04-26T16:27:49.213 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:27:49.214 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:27:49.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:49.237 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:49.295 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:27:48 smithi190 bash[68229]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-6 2024-04-26T16:27:49.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:49.307+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fdca01a5e50 con 0x7fdca0103e30 2024-04-26T16:27:49.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:49.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:49.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:49.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:49.601 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:49.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:49.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:49.734 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:49.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:49.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:49.894 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:49.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:48.209090+0000 osd.0 (osd.0) 251 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:48.209090+0000 osd.0 (osd.0) 251 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:48.212990+0000 osd.0 (osd.0) 252 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:48.212990+0000 osd.0 (osd.0) 252 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.027619+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.50:0/83204712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.027619+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.50:0/83204712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.095799+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.095799+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.097307+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.097307+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.147114+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.147114+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.151321+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e401: 8 total, 7 up, 7 in 2024-04-26T16:27:49.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.151321+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e401: 8 total, 7 up, 7 in 2024-04-26T16:27:49.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.314760+0000 mon.c (mon.2) 584 : audit [INF] from='client.? 172.21.15.50:0/1523688200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.314760+0000 mon.c (mon.2) 584 : audit [INF] from='client.? 172.21.15.50:0/1523688200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.315248+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:49 smithi190 bash[23868]: audit 2024-04-26T16:27:49.315248+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:49.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:49.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:48.209090+0000 osd.0 (osd.0) 251 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:27:49.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:48.209090+0000 osd.0 (osd.0) 251 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:48.212990+0000 osd.0 (osd.0) 252 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:48.212990+0000 osd.0 (osd.0) 252 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.027619+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.50:0/83204712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.027619+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.50:0/83204712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.095799+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.095799+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.097307+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.097307+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.147114+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.147114+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.151321+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e401: 8 total, 7 up, 7 in 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.151321+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e401: 8 total, 7 up, 7 in 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.314760+0000 mon.c (mon.2) 584 : audit [INF] from='client.? 172.21.15.50:0/1523688200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.314760+0000 mon.c (mon.2) 584 : audit [INF] from='client.? 172.21.15.50:0/1523688200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.315248+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[29234]: audit 2024-04-26T16:27:49.315248+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:48.209090+0000 osd.0 (osd.0) 251 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:48.209090+0000 osd.0 (osd.0) 251 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:48.212990+0000 osd.0 (osd.0) 252 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:48.212990+0000 osd.0 (osd.0) 252 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.027619+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.50:0/83204712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.027619+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.50:0/83204712' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.095799+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:27:49.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.095799+0000 osd.2 (osd.2) 195 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.097307+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.097307+0000 osd.2 (osd.2) 196 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.147114+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.147114+0000 mon.a (mon.0) 2543 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi050-73201-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.151321+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e401: 8 total, 7 up, 7 in 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.151321+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e401: 8 total, 7 up, 7 in 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.314760+0000 mon.c (mon.2) 584 : audit [INF] from='client.? 172.21.15.50:0/1523688200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.314760+0000 mon.c (mon.2) 584 : audit [INF] from='client.? 172.21.15.50:0/1523688200' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.315248+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:49 smithi050 bash[21463]: audit 2024-04-26T16:27:49.315248+0000 mon.a (mon.0) 2545 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:27:49.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:50.024 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:50.127 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:50.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.151+0000 7fdc9dffb700 1 -- 172.21.15.50:0/1523688200 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v402) ==== 87+0+0 (secure 0 0 0) 0x7fdc880613b0 con 0x7fdca0103e30 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 10 librados: watch_flush enter 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 10 librados: watch_flush exit 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 20 client.41159.objecter shutdown clearing up homeless session... 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 10 client.41159.objecter successfully canceled tick 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc8c071f80 msgr2=0x7fdc8c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc8c071f80 0x7fdc8c074440 secure :-1 s=READY pgs=3738 cs=0 l=1 rev1=1 crypto rx=0x7fdc90009ca0 tx=0x7fdc90009260 comp rx=0 tx=0).stop 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 msgr2=0x7fdca0199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0199640 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7fdc8800eb90 tx=0x7fdc8800eea0 comp rx=0 tx=0).stop 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 shutdown_connections 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc8c071f80 0x7fdc8c074440 unknown :-1 s=CLOSED pgs=3738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:50.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdca01094c0 0x7fdca019d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdca0103470 0x7fdca0199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 --2- 172.21.15.50:0/1523688200 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdca0103e30 0x7fdca0199640 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 >> 172.21.15.50:0/1523688200 conn(0x7fdca007a580 msgr2=0x7fdca0101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 shutdown_connections 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 -- 172.21.15.50:0/1523688200 wait complete. 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:27:50.155+0000 7fdca677b700 1 librados: shutdown 2024-04-26T16:27:50.163 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:27:50.167 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:50.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:50.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:50.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:50.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:50.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:50.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:50.572 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:27:50.572 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:27:50.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:50.684 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:27:50.754 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:27:50.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:50.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.050550+0000 osd.7 (osd.7) 249 : cluster [DBG] 109.10 deep-scrub starts 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.050550+0000 osd.7 (osd.7) 249 : cluster [DBG] 109.10 deep-scrub starts 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.058536+0000 osd.7 (osd.7) 250 : cluster [DBG] 109.10 deep-scrub ok 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.058536+0000 osd.7 (osd.7) 250 : cluster [DBG] 109.10 deep-scrub ok 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.177689+0000 osd.0 (osd.0) 253 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.177689+0000 osd.0 (osd.0) 253 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.182129+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.182129+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.623082+0000 mgr.y (mgr.24401) 411 : cluster [DBG] pgmap v586: 204 pgs: 8 unknown, 18 stale+active+clean, 178 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:49.623082+0000 mgr.y (mgr.24401) 411 : cluster [DBG] pgmap v586: 204 pgs: 8 unknown, 18 stale+active+clean, 178 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:50.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.148625+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.148625+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.149261+0000 mon.a (mon.0) 2547 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.149261+0000 mon.a (mon.0) 2547 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: audit 2024-04-26T16:27:50.150970+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: audit 2024-04-26T16:27:50.150970+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.154901+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e402: 8 total, 7 up, 7 in 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.154901+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e402: 8 total, 7 up, 7 in 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: audit 2024-04-26T16:27:50.169545+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: audit 2024-04-26T16:27:50.169545+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: audit 2024-04-26T16:27:50.182996+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:50 smithi190 bash[23868]: audit 2024-04-26T16:27:50.182996+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.050550+0000 osd.7 (osd.7) 249 : cluster [DBG] 109.10 deep-scrub starts 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.050550+0000 osd.7 (osd.7) 249 : cluster [DBG] 109.10 deep-scrub starts 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.058536+0000 osd.7 (osd.7) 250 : cluster [DBG] 109.10 deep-scrub ok 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.058536+0000 osd.7 (osd.7) 250 : cluster [DBG] 109.10 deep-scrub ok 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.177689+0000 osd.0 (osd.0) 253 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.177689+0000 osd.0 (osd.0) 253 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.182129+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:27:50.930 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.182129+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.623082+0000 mgr.y (mgr.24401) 411 : cluster [DBG] pgmap v586: 204 pgs: 8 unknown, 18 stale+active+clean, 178 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:49.623082+0000 mgr.y (mgr.24401) 411 : cluster [DBG] pgmap v586: 204 pgs: 8 unknown, 18 stale+active+clean, 178 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.148625+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.148625+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.149261+0000 mon.a (mon.0) 2547 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.149261+0000 mon.a (mon.0) 2547 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: audit 2024-04-26T16:27:50.150970+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: audit 2024-04-26T16:27:50.150970+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.154901+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e402: 8 total, 7 up, 7 in 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.154901+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e402: 8 total, 7 up, 7 in 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: audit 2024-04-26T16:27:50.169545+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: audit 2024-04-26T16:27:50.169545+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: audit 2024-04-26T16:27:50.182996+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.931 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[21463]: audit 2024-04-26T16:27:50.182996+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.050550+0000 osd.7 (osd.7) 249 : cluster [DBG] 109.10 deep-scrub starts 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.050550+0000 osd.7 (osd.7) 249 : cluster [DBG] 109.10 deep-scrub starts 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.058536+0000 osd.7 (osd.7) 250 : cluster [DBG] 109.10 deep-scrub ok 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.058536+0000 osd.7 (osd.7) 250 : cluster [DBG] 109.10 deep-scrub ok 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.177689+0000 osd.0 (osd.0) 253 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.177689+0000 osd.0 (osd.0) 253 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.182129+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.182129+0000 osd.0 (osd.0) 254 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.623082+0000 mgr.y (mgr.24401) 411 : cluster [DBG] pgmap v586: 204 pgs: 8 unknown, 18 stale+active+clean, 178 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:49.623082+0000 mgr.y (mgr.24401) 411 : cluster [DBG] pgmap v586: 204 pgs: 8 unknown, 18 stale+active+clean, 178 active+clean; 8.4 MiB data, 546 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.148625+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.148625+0000 mon.a (mon.0) 2546 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:27:50.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.149261+0000 mon.a (mon.0) 2547 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.149261+0000 mon.a (mon.0) 2547 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: audit 2024-04-26T16:27:50.150970+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: audit 2024-04-26T16:27:50.150970+0000 mon.a (mon.0) 2548 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.154901+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e402: 8 total, 7 up, 7 in 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.154901+0000 mon.a (mon.0) 2549 : cluster [DBG] osdmap e402: 8 total, 7 up, 7 in 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: audit 2024-04-26T16:27:50.169545+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: audit 2024-04-26T16:27:50.169545+0000 mon.b (mon.1) 278 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: audit 2024-04-26T16:27:50.182996+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.933 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:50 smithi050 bash[29234]: audit 2024-04-26T16:27:50.182996+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:27:50.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:50.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:51.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:51.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:51.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:51.209 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:51.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:51.339 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:51.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:51.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:51.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:51.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-26T16:27:51.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:51.599 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:51.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:51.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.021039+0000 osd.7 (osd.7) 251 : cluster [DBG] 109.1d deep-scrub starts 2024-04-26T16:27:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.021039+0000 osd.7 (osd.7) 251 : cluster [DBG] 109.1d deep-scrub starts 2024-04-26T16:27:51.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.033121+0000 osd.7 (osd.7) 252 : cluster [DBG] 109.1d deep-scrub ok 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: cluster 2024-04-26T16:27:50.033121+0000 osd.7 (osd.7) 252 : cluster [DBG] 109.1d deep-scrub ok 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.153914+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.153914+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: cluster 2024-04-26T16:27:51.159579+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e403: 8 total, 7 up, 7 in 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: cluster 2024-04-26T16:27:51.159579+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e403: 8 total, 7 up, 7 in 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.167419+0000 mon.c (mon.2) 585 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.167419+0000 mon.c (mon.2) 585 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.168695+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.168695+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.197337+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.197337+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.199576+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:51 smithi190 bash[23868]: audit 2024-04-26T16:27:51.199576+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:51.855 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:51.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T16:27:51.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.021039+0000 osd.7 (osd.7) 251 : cluster [DBG] 109.1d deep-scrub starts 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.021039+0000 osd.7 (osd.7) 251 : cluster [DBG] 109.1d deep-scrub starts 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.033121+0000 osd.7 (osd.7) 252 : cluster [DBG] 109.1d deep-scrub ok 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: cluster 2024-04-26T16:27:50.033121+0000 osd.7 (osd.7) 252 : cluster [DBG] 109.1d deep-scrub ok 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.153914+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.153914+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: cluster 2024-04-26T16:27:51.159579+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e403: 8 total, 7 up, 7 in 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: cluster 2024-04-26T16:27:51.159579+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e403: 8 total, 7 up, 7 in 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.167419+0000 mon.c (mon.2) 585 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.167419+0000 mon.c (mon.2) 585 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.168695+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.168695+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.197337+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.197337+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.199576+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[21463]: audit 2024-04-26T16:27:51.199576+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.021039+0000 osd.7 (osd.7) 251 : cluster [DBG] 109.1d deep-scrub starts 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.021039+0000 osd.7 (osd.7) 251 : cluster [DBG] 109.1d deep-scrub starts 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.033121+0000 osd.7 (osd.7) 252 : cluster [DBG] 109.1d deep-scrub ok 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: cluster 2024-04-26T16:27:50.033121+0000 osd.7 (osd.7) 252 : cluster [DBG] 109.1d deep-scrub ok 2024-04-26T16:27:51.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.153914+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.153914+0000 mon.a (mon.0) 2551 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: cluster 2024-04-26T16:27:51.159579+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e403: 8 total, 7 up, 7 in 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: cluster 2024-04-26T16:27:51.159579+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e403: 8 total, 7 up, 7 in 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.167419+0000 mon.c (mon.2) 585 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.167419+0000 mon.c (mon.2) 585 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.168695+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.168695+0000 mon.a (mon.0) 2553 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.197337+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.197337+0000 mon.a (mon.0) 2554 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.199576+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:51 smithi050 bash[29234]: audit 2024-04-26T16:27:51.199576+0000 mon.b (mon.1) 279 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:27:51.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:27:51.981 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T16:27:51.981 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-04-26T16:27:51.982 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi190.front.sepia.ceph.com 2024-04-26T16:27:51.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-04-26T16:27:51.984 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:51.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:52.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.112 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:52.192 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:27:52.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:52.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:52.367 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:52.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:52.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:52.629 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:52.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:52.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:27:52.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:52.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: cluster 2024-04-26T16:27:51.625633+0000 mgr.y (mgr.24401) 412 : cluster [DBG] pgmap v589: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: cluster 2024-04-26T16:27:51.625633+0000 mgr.y (mgr.24401) 412 : cluster [DBG] pgmap v589: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: cluster 2024-04-26T16:27:52.153972+0000 mon.a (mon.0) 2555 : cluster [WRN] Health check failed: Degraded data redundancy: 141/1032 objects degraded (13.663%), 33 pgs degraded (PG_DEGRADED) 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: cluster 2024-04-26T16:27:52.153972+0000 mon.a (mon.0) 2555 : cluster [WRN] Health check failed: Degraded data redundancy: 141/1032 objects degraded (13.663%), 33 pgs degraded (PG_DEGRADED) 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.163678+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.163678+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.163956+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.163956+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.170571+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.170571+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: cluster 2024-04-26T16:27:52.172014+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e404: 8 total, 7 up, 7 in 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: cluster 2024-04-26T16:27:52.172014+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e404: 8 total, 7 up, 7 in 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.173213+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.173213+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.174960+0000 mon.c (mon.2) 586 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.174960+0000 mon.c (mon.2) 586 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.176885+0000 mon.a (mon.0) 2560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:52 smithi190 bash[23868]: audit 2024-04-26T16:27:52.176885+0000 mon.a (mon.0) 2560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: cluster 2024-04-26T16:27:51.625633+0000 mgr.y (mgr.24401) 412 : cluster [DBG] pgmap v589: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: cluster 2024-04-26T16:27:51.625633+0000 mgr.y (mgr.24401) 412 : cluster [DBG] pgmap v589: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:52.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: cluster 2024-04-26T16:27:52.153972+0000 mon.a (mon.0) 2555 : cluster [WRN] Health check failed: Degraded data redundancy: 141/1032 objects degraded (13.663%), 33 pgs degraded (PG_DEGRADED) 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: cluster 2024-04-26T16:27:52.153972+0000 mon.a (mon.0) 2555 : cluster [WRN] Health check failed: Degraded data redundancy: 141/1032 objects degraded (13.663%), 33 pgs degraded (PG_DEGRADED) 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.163678+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.163678+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.163956+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.163956+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.170571+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.170571+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: cluster 2024-04-26T16:27:52.172014+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e404: 8 total, 7 up, 7 in 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: cluster 2024-04-26T16:27:52.172014+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e404: 8 total, 7 up, 7 in 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.173213+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.173213+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.174960+0000 mon.c (mon.2) 586 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: cluster 2024-04-26T16:27:51.625633+0000 mgr.y (mgr.24401) 412 : cluster [DBG] pgmap v589: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: cluster 2024-04-26T16:27:51.625633+0000 mgr.y (mgr.24401) 412 : cluster [DBG] pgmap v589: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: cluster 2024-04-26T16:27:52.153972+0000 mon.a (mon.0) 2555 : cluster [WRN] Health check failed: Degraded data redundancy: 141/1032 objects degraded (13.663%), 33 pgs degraded (PG_DEGRADED) 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: cluster 2024-04-26T16:27:52.153972+0000 mon.a (mon.0) 2555 : cluster [WRN] Health check failed: Degraded data redundancy: 141/1032 objects degraded (13.663%), 33 pgs degraded (PG_DEGRADED) 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.163678+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.163678+0000 mon.a (mon.0) 2556 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.163956+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.163956+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.170571+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.170571+0000 mon.b (mon.1) 280 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: cluster 2024-04-26T16:27:52.172014+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e404: 8 total, 7 up, 7 in 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: cluster 2024-04-26T16:27:52.172014+0000 mon.a (mon.0) 2558 : cluster [DBG] osdmap e404: 8 total, 7 up, 7 in 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.173213+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.173213+0000 mon.a (mon.0) 2559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.174960+0000 mon.c (mon.2) 586 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.174960+0000 mon.c (mon.2) 586 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.176885+0000 mon.a (mon.0) 2560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[21463]: audit 2024-04-26T16:27:52.176885+0000 mon.a (mon.0) 2560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.174960+0000 mon.c (mon.2) 586 : audit [INF] from='client.? 172.21.15.50:0/1501736677' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.176885+0000 mon.a (mon.0) 2560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:52 smithi050 bash[29234]: audit 2024-04-26T16:27:52.176885+0000 mon.a (mon.0) 2560 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]: dispatch 2024-04-26T16:27:52.890 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:52.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:52.994 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:53.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:53.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:53.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:53.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:53.272 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:53.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:53.366 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:27:53.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:53.397 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:53.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:53.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:53.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:53.657 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:53.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:53.785 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:53.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:53.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:53.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:54.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:54.046 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:54.090 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:27:54.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:54.176 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:54.279 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:54.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.167013+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.167013+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.167319+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.167319+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: cluster 2024-04-26T16:27:53.175599+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e405: 8 total, 7 up, 7 in 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: cluster 2024-04-26T16:27:53.175599+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e405: 8 total, 7 up, 7 in 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.177328+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.177328+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.179022+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.179022+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.231894+0000 mon.c (mon.2) 587 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.231894+0000 mon.c (mon.2) 587 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.232350+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.232350+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.233092+0000 mon.c (mon.2) 588 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.233092+0000 mon.c (mon.2) 588 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.233457+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.233457+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.234099+0000 mon.c (mon.2) 589 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.234099+0000 mon.c (mon.2) 589 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.234469+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: audit 2024-04-26T16:27:53.234469+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: cluster 2024-04-26T16:27:53.626853+0000 mgr.y (mgr.24401) 413 : cluster [DBG] pgmap v592: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:54 smithi190 bash[23868]: cluster 2024-04-26T16:27:53.626853+0000 mgr.y (mgr.24401) 413 : cluster [DBG] pgmap v592: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:54.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:54.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:54.432 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:27:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:27:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.167013+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.167013+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.167319+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.167319+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: cluster 2024-04-26T16:27:53.175599+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e405: 8 total, 7 up, 7 in 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: cluster 2024-04-26T16:27:53.175599+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e405: 8 total, 7 up, 7 in 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.177328+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.177328+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.179022+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.179022+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.231894+0000 mon.c (mon.2) 587 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.231894+0000 mon.c (mon.2) 587 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.232350+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.232350+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.233092+0000 mon.c (mon.2) 588 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.167013+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.167013+0000 mon.a (mon.0) 2561 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.167319+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.167319+0000 mon.a (mon.0) 2562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi050-73201-63"}]': finished 2024-04-26T16:27:54.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: cluster 2024-04-26T16:27:53.175599+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e405: 8 total, 7 up, 7 in 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: cluster 2024-04-26T16:27:53.175599+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e405: 8 total, 7 up, 7 in 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.177328+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.177328+0000 mon.a (mon.0) 2564 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.179022+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.179022+0000 mon.b (mon.1) 281 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.231894+0000 mon.c (mon.2) 587 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.231894+0000 mon.c (mon.2) 587 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.232350+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.232350+0000 mon.a (mon.0) 2565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.233092+0000 mon.c (mon.2) 588 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.233092+0000 mon.c (mon.2) 588 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.233457+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.233457+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.234099+0000 mon.c (mon.2) 589 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.234099+0000 mon.c (mon.2) 589 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.234469+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: audit 2024-04-26T16:27:53.234469+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: cluster 2024-04-26T16:27:53.626853+0000 mgr.y (mgr.24401) 413 : cluster [DBG] pgmap v592: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[21463]: cluster 2024-04-26T16:27:53.626853+0000 mgr.y (mgr.24401) 413 : cluster [DBG] pgmap v592: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.233092+0000 mon.c (mon.2) 588 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.233457+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.233457+0000 mon.a (mon.0) 2566 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.234099+0000 mon.c (mon.2) 589 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.234099+0000 mon.c (mon.2) 589 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.234469+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: audit 2024-04-26T16:27:53.234469+0000 mon.a (mon.0) 2567 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: cluster 2024-04-26T16:27:53.626853+0000 mgr.y (mgr.24401) 413 : cluster [DBG] pgmap v592: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:54.439 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:54 smithi050 bash[29234]: cluster 2024-04-26T16:27:53.626853+0000 mgr.y (mgr.24401) 413 : cluster [DBG] pgmap v592: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 255 B/s wr, 0 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:54.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:54.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:54.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:54.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:54.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:54.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:54.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:54.954 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:54.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:55.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:55.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:55.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:55.210 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:55.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:55.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.c 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.a 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.3 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.1f 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.1e 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.1b 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.5 2024-04-26T16:27:55.399 INFO:teuthology.orchestra.run.smithi190.stdout:6.1 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:6.15 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:6.13 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:5.f 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:5.9 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:5.3 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:5.1f 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:5.18 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:2.1a 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:6.10 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:3.3 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:284.13 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:6.4 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:284.1 2024-04-26T16:27:55.400 INFO:teuthology.orchestra.run.smithi190.stdout:3.1c 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:2.18 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.9 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:2.15 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.1e 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.d 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.5 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.10 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.1 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:2.14 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.18 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:11.3 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:284.14 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:109.1 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:2.17 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:2.1d 2024-04-26T16:27:55.401 INFO:teuthology.orchestra.run.smithi190.stdout:1.0 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:2.19 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:109.1b 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:11.1c 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:284.1b 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:284.e 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:2.16 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:109.4 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:11.17 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:5.10 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:109.17 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:109.8 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:11.a 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:11.4 2024-04-26T16:27:55.402 INFO:teuthology.orchestra.run.smithi190.stdout:109.1e 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:11.1a 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:11.14 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:109.3 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:109.b 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:109.9 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:2.11 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:4.1 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:2.1b 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:2.1c 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:2.2 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:2.7 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:109.1c 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:284.0 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:284.d 2024-04-26T16:27:55.403 INFO:teuthology.orchestra.run.smithi190.stdout:284.12 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:3.0 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.17 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.18 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:11.13 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:11.8 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.1a 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.1c 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.1e 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.3 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:3.c 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.6 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:3.14 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.4 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:284.a 2024-04-26T16:27:55.404 INFO:teuthology.orchestra.run.smithi190.stdout:3.11 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:109.d 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.10 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.17 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:2.6 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.1d 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:5.2 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.1e 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.1f 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.2 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.5 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.a 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:3.d 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:5.11 2024-04-26T16:27:55.405 INFO:teuthology.orchestra.run.smithi190.stdout:5.16 2024-04-26T16:27:55.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:55.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: cluster 2024-04-26T16:27:54.168248+0000 mon.a (mon.0) 2568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: cluster 2024-04-26T16:27:54.168248+0000 mon.a (mon.0) 2568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.178693+0000 mon.a (mon.0) 2569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]': finished 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.178693+0000 mon.a (mon.0) 2569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]': finished 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.178885+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.178885+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:55.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: cluster 2024-04-26T16:27:54.191618+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e406: 8 total, 7 up, 7 in 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: cluster 2024-04-26T16:27:54.191618+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e406: 8 total, 7 up, 7 in 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.194166+0000 mon.c (mon.2) 590 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.194166+0000 mon.c (mon.2) 590 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.195072+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[29234]: audit 2024-04-26T16:27:54.195072+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: cluster 2024-04-26T16:27:54.168248+0000 mon.a (mon.0) 2568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: cluster 2024-04-26T16:27:54.168248+0000 mon.a (mon.0) 2568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.178693+0000 mon.a (mon.0) 2569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]': finished 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.178693+0000 mon.a (mon.0) 2569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]': finished 2024-04-26T16:27:55.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.178885+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.178885+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: cluster 2024-04-26T16:27:54.191618+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e406: 8 total, 7 up, 7 in 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: cluster 2024-04-26T16:27:54.191618+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e406: 8 total, 7 up, 7 in 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.194166+0000 mon.c (mon.2) 590 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.194166+0000 mon.c (mon.2) 590 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.195072+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:55 smithi050 bash[21463]: audit 2024-04-26T16:27:54.195072+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:55.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:55.640 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: cluster 2024-04-26T16:27:54.168248+0000 mon.a (mon.0) 2568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:55.640 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: cluster 2024-04-26T16:27:54.168248+0000 mon.a (mon.0) 2568 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.178693+0000 mon.a (mon.0) 2569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]': finished 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.178693+0000 mon.a (mon.0) 2569 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-49", "mode": "readproxy"}]': finished 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.178885+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.178885+0000 mon.a (mon.0) 2570 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi050-73201-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: cluster 2024-04-26T16:27:54.191618+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e406: 8 total, 7 up, 7 in 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: cluster 2024-04-26T16:27:54.191618+0000 mon.a (mon.0) 2571 : cluster [DBG] osdmap e406: 8 total, 7 up, 7 in 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.194166+0000 mon.c (mon.2) 590 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.194166+0000 mon.c (mon.2) 590 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.195072+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.641 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:55 smithi190 bash[23868]: audit 2024-04-26T16:27:54.195072+0000 mon.a (mon.0) 2572 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:55.668 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:55.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:55.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:55.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:55.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:55.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:56.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:56.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:56.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:56.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:56.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:56.306 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:56.317 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:55.207937+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e407: 8 total, 7 up, 7 in 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:55.207937+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e407: 8 total, 7 up, 7 in 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:55.628022+0000 mgr.y (mgr.24401) 414 : cluster [DBG] pgmap v595: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:55.628022+0000 mgr.y (mgr.24401) 414 : cluster [DBG] pgmap v595: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:56.132539+0000 mon.a (mon.0) 2574 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:56.132539+0000 mon.a (mon.0) 2574 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:56.329 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: audit 2024-04-26T16:27:56.137275+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: audit 2024-04-26T16:27:56.137275+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:56.145091+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e408: 8 total, 7 up, 7 in 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[21463]: cluster 2024-04-26T16:27:56.145091+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e408: 8 total, 7 up, 7 in 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:55.207937+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e407: 8 total, 7 up, 7 in 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:55.207937+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e407: 8 total, 7 up, 7 in 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:55.628022+0000 mgr.y (mgr.24401) 414 : cluster [DBG] pgmap v595: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:55.628022+0000 mgr.y (mgr.24401) 414 : cluster [DBG] pgmap v595: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:56.132539+0000 mon.a (mon.0) 2574 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:56.132539+0000 mon.a (mon.0) 2574 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:56.330 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: audit 2024-04-26T16:27:56.137275+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:27:56.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: audit 2024-04-26T16:27:56.137275+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:27:56.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:56.145091+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e408: 8 total, 7 up, 7 in 2024-04-26T16:27:56.331 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:56 smithi050 bash[29234]: cluster 2024-04-26T16:27:56.145091+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e408: 8 total, 7 up, 7 in 2024-04-26T16:27:56.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:56.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:55.207937+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e407: 8 total, 7 up, 7 in 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:55.207937+0000 mon.a (mon.0) 2573 : cluster [DBG] osdmap e407: 8 total, 7 up, 7 in 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:55.628022+0000 mgr.y (mgr.24401) 414 : cluster [DBG] pgmap v595: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:55.628022+0000 mgr.y (mgr.24401) 414 : cluster [DBG] pgmap v595: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 547 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:56.132539+0000 mon.a (mon.0) 2574 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:56.132539+0000 mon.a (mon.0) 2574 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: audit 2024-04-26T16:27:56.137275+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: audit 2024-04-26T16:27:56.137275+0000 mon.a (mon.0) 2575 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi050-73201-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:56.145091+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e408: 8 total, 7 up, 7 in 2024-04-26T16:27:56.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[23868]: cluster 2024-04-26T16:27:56.145091+0000 mon.a (mon.0) 2576 : cluster [DBG] osdmap e408: 8 total, 7 up, 7 in 2024-04-26T16:27:56.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 2.11 --file /var/log/ceph/exp.2.11.6 2024-04-26T16:27:56.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:56.577 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:56.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:56.708 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:56.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:56.716 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:27:56.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:56.836 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:56.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:56.968 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:56.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:56.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:57.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:57.072 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:27:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:27:57.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:57.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:57.230 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:57.329 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:57.330 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:27:57.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:57.358 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:57.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:57.489 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:57.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:57.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:57.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:57.746 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:57.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:57.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[29234]: audit 2024-04-26T16:27:56.680632+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:57.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[29234]: audit 2024-04-26T16:27:56.680632+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:57.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[29234]: audit 2024-04-26T16:27:56.683785+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[29234]: audit 2024-04-26T16:27:56.683785+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[29234]: cluster 2024-04-26T16:27:57.144137+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e409: 8 total, 7 up, 7 in 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[29234]: cluster 2024-04-26T16:27:57.144137+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e409: 8 total, 7 up, 7 in 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[21463]: audit 2024-04-26T16:27:56.680632+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[21463]: audit 2024-04-26T16:27:56.680632+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[21463]: audit 2024-04-26T16:27:56.683785+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[21463]: audit 2024-04-26T16:27:56.683785+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[21463]: cluster 2024-04-26T16:27:57.144137+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e409: 8 total, 7 up, 7 in 2024-04-26T16:27:57.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:57 smithi050 bash[21463]: cluster 2024-04-26T16:27:57.144137+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e409: 8 total, 7 up, 7 in 2024-04-26T16:27:57.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:57.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:57.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:57.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:57 smithi190 bash[23868]: audit 2024-04-26T16:27:56.680632+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:57.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:57 smithi190 bash[23868]: audit 2024-04-26T16:27:56.680632+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:27:57.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:57 smithi190 bash[23868]: audit 2024-04-26T16:27:56.683785+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:57.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:57 smithi190 bash[23868]: audit 2024-04-26T16:27:56.683785+0000 mon.c (mon.2) 591 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:27:57.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:57 smithi190 bash[23868]: cluster 2024-04-26T16:27:57.144137+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e409: 8 total, 7 up, 7 in 2024-04-26T16:27:57.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:57 smithi190 bash[23868]: cluster 2024-04-26T16:27:57.144137+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e409: 8 total, 7 up, 7 in 2024-04-26T16:27:58.009 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:58.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:58.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:27:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:27:58.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:58.273 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:27:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:27:58.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:58.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:58.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:58.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:58.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:58.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:58.657 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:27:58.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:58.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:58.793 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:58.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: audit 2024-04-26T16:27:56.842996+0000 mgr.y (mgr.24401) 415 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: audit 2024-04-26T16:27:56.842996+0000 mgr.y (mgr.24401) 415 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: cluster 2024-04-26T16:27:57.628824+0000 mgr.y (mgr.24401) 416 : cluster [DBG] pgmap v598: 236 pgs: 8 creating+peering, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: cluster 2024-04-26T16:27:57.628824+0000 mgr.y (mgr.24401) 416 : cluster [DBG] pgmap v598: 236 pgs: 8 creating+peering, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: cluster 2024-04-26T16:27:58.152061+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e410: 8 total, 7 up, 7 in 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: cluster 2024-04-26T16:27:58.152061+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e410: 8 total, 7 up, 7 in 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: audit 2024-04-26T16:27:58.154573+0000 mon.c (mon.2) 592 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: audit 2024-04-26T16:27:58.154573+0000 mon.c (mon.2) 592 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: audit 2024-04-26T16:27:58.155468+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[21463]: audit 2024-04-26T16:27:58.155468+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: audit 2024-04-26T16:27:56.842996+0000 mgr.y (mgr.24401) 415 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: audit 2024-04-26T16:27:56.842996+0000 mgr.y (mgr.24401) 415 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: cluster 2024-04-26T16:27:57.628824+0000 mgr.y (mgr.24401) 416 : cluster [DBG] pgmap v598: 236 pgs: 8 creating+peering, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: cluster 2024-04-26T16:27:57.628824+0000 mgr.y (mgr.24401) 416 : cluster [DBG] pgmap v598: 236 pgs: 8 creating+peering, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: cluster 2024-04-26T16:27:58.152061+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e410: 8 total, 7 up, 7 in 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: cluster 2024-04-26T16:27:58.152061+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e410: 8 total, 7 up, 7 in 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: audit 2024-04-26T16:27:58.154573+0000 mon.c (mon.2) 592 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: audit 2024-04-26T16:27:58.154573+0000 mon.c (mon.2) 592 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: audit 2024-04-26T16:27:58.155468+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:27:58 smithi050 bash[29234]: audit 2024-04-26T16:27:58.155468+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:58.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:58.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:59.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:59.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: audit 2024-04-26T16:27:56.842996+0000 mgr.y (mgr.24401) 415 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: audit 2024-04-26T16:27:56.842996+0000 mgr.y (mgr.24401) 415 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:27:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: cluster 2024-04-26T16:27:57.628824+0000 mgr.y (mgr.24401) 416 : cluster [DBG] pgmap v598: 236 pgs: 8 creating+peering, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: cluster 2024-04-26T16:27:57.628824+0000 mgr.y (mgr.24401) 416 : cluster [DBG] pgmap v598: 236 pgs: 8 creating+peering, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:27:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: cluster 2024-04-26T16:27:58.152061+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e410: 8 total, 7 up, 7 in 2024-04-26T16:27:59.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: cluster 2024-04-26T16:27:58.152061+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e410: 8 total, 7 up, 7 in 2024-04-26T16:27:59.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: audit 2024-04-26T16:27:58.154573+0000 mon.c (mon.2) 592 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:59.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: audit 2024-04-26T16:27:58.154573+0000 mon.c (mon.2) 592 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:59.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: audit 2024-04-26T16:27:58.155468+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:59.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:27:58 smithi190 bash[23868]: audit 2024-04-26T16:27:58.155468+0000 mon.a (mon.0) 2580 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:27:59.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:59.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:27:59.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:59.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:59.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:59.441 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:27:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:27:59.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:59.566 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:27:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:27:59.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:27:59.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:27:59.762 INFO:teuthology.orchestra.run.smithi190.stderr:Exporting 2.11 info 2.11( empty local-lis/les=78/79 n=0 ec=78/78 lis/c=78/78 les/c/f=79/79/0 sis=78) 2024-04-26T16:27:59.762 INFO:teuthology.orchestra.run.smithi190.stderr:Export successful 2024-04-26T16:27:59.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:59.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:27:59.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:27:59.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:27:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:27:59.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:00.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:00.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:00.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:00.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:00.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: audit 2024-04-26T16:27:59.150827+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:00.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: audit 2024-04-26T16:27:59.150827+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:00.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: cluster 2024-04-26T16:27:59.159414+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e411: 8 total, 7 up, 7 in 2024-04-26T16:28:00.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: cluster 2024-04-26T16:27:59.159414+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e411: 8 total, 7 up, 7 in 2024-04-26T16:28:00.218 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: audit 2024-04-26T16:27:59.160111+0000 mon.c (mon.2) 593 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: audit 2024-04-26T16:27:59.160111+0000 mon.c (mon.2) 593 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: audit 2024-04-26T16:27:59.161813+0000 mon.a (mon.0) 2583 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: audit 2024-04-26T16:27:59.161813+0000 mon.a (mon.0) 2583 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: cluster 2024-04-26T16:27:59.629963+0000 mgr.y (mgr.24401) 417 : cluster [DBG] pgmap v601: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[29234]: cluster 2024-04-26T16:27:59.629963+0000 mgr.y (mgr.24401) 417 : cluster [DBG] pgmap v601: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: audit 2024-04-26T16:27:59.150827+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: audit 2024-04-26T16:27:59.150827+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: cluster 2024-04-26T16:27:59.159414+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e411: 8 total, 7 up, 7 in 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: cluster 2024-04-26T16:27:59.159414+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e411: 8 total, 7 up, 7 in 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: audit 2024-04-26T16:27:59.160111+0000 mon.c (mon.2) 593 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.219 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: audit 2024-04-26T16:27:59.160111+0000 mon.c (mon.2) 593 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: audit 2024-04-26T16:27:59.161813+0000 mon.a (mon.0) 2583 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: audit 2024-04-26T16:27:59.161813+0000 mon.a (mon.0) 2583 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: cluster 2024-04-26T16:27:59.629963+0000 mgr.y (mgr.24401) 417 : cluster [DBG] pgmap v601: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:00.220 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:00 smithi050 bash[21463]: cluster 2024-04-26T16:27:59.629963+0000 mgr.y (mgr.24401) 417 : cluster [DBG] pgmap v601: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:00.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: audit 2024-04-26T16:27:59.150827+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:00.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: audit 2024-04-26T16:27:59.150827+0000 mon.a (mon.0) 2581 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:00.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: cluster 2024-04-26T16:27:59.159414+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e411: 8 total, 7 up, 7 in 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: cluster 2024-04-26T16:27:59.159414+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e411: 8 total, 7 up, 7 in 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: audit 2024-04-26T16:27:59.160111+0000 mon.c (mon.2) 593 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: audit 2024-04-26T16:27:59.160111+0000 mon.c (mon.2) 593 : audit [INF] from='client.? 172.21.15.50:0/3617749135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: audit 2024-04-26T16:27:59.161813+0000 mon.a (mon.0) 2583 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: audit 2024-04-26T16:27:59.161813+0000 mon.a (mon.0) 2583 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]: dispatch 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: cluster 2024-04-26T16:27:59.629963+0000 mgr.y (mgr.24401) 417 : cluster [DBG] pgmap v601: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:00.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:00 smithi190 bash[23868]: cluster 2024-04-26T16:27:59.629963+0000 mgr.y (mgr.24401) 417 : cluster [DBG] pgmap v601: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 551 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:00.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:00.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:00.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:00.482 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:00.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:00.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:00.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:00.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:00.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:00.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:00.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 2.11 2024-04-26T16:28:00.868 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:00.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:00.994 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:00.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:00.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:01.023 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:28:01.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:01.123 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:01.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:01.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:01.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.164475+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:01.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.164475+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:01.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:01.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.164475+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:01.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.164475+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:01.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: cluster 2024-04-26T16:28:00.171920+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e412: 8 total, 7 up, 7 in 2024-04-26T16:28:01.321 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: cluster 2024-04-26T16:28:00.171920+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e412: 8 total, 7 up, 7 in 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.227647+0000 mon.c (mon.2) 594 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.227647+0000 mon.c (mon.2) 594 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.228058+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.228058+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.228710+0000 mon.c (mon.2) 595 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.228710+0000 mon.c (mon.2) 595 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.228993+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.228993+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.229536+0000 mon.c (mon.2) 596 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.229536+0000 mon.c (mon.2) 596 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.229848+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:01 smithi190 bash[23868]: audit 2024-04-26T16:28:00.229848+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:01.377 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:01.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:01.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: cluster 2024-04-26T16:28:00.171920+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e412: 8 total, 7 up, 7 in 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: cluster 2024-04-26T16:28:00.171920+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e412: 8 total, 7 up, 7 in 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.227647+0000 mon.c (mon.2) 594 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.227647+0000 mon.c (mon.2) 594 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.228058+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.228058+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.228710+0000 mon.c (mon.2) 595 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.228710+0000 mon.c (mon.2) 595 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.228993+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.228993+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.229536+0000 mon.c (mon.2) 596 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.229536+0000 mon.c (mon.2) 596 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.229848+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[21463]: audit 2024-04-26T16:28:00.229848+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.164475+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.164475+0000 mon.a (mon.0) 2584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi050-73201-64"}]': finished 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: cluster 2024-04-26T16:28:00.171920+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e412: 8 total, 7 up, 7 in 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: cluster 2024-04-26T16:28:00.171920+0000 mon.a (mon.0) 2585 : cluster [DBG] osdmap e412: 8 total, 7 up, 7 in 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.227647+0000 mon.c (mon.2) 594 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.227647+0000 mon.c (mon.2) 594 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.228058+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.228058+0000 mon.a (mon.0) 2586 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.228710+0000 mon.c (mon.2) 595 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.228710+0000 mon.c (mon.2) 595 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.228993+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.228993+0000 mon.a (mon.0) 2587 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.229536+0000 mon.c (mon.2) 596 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.229536+0000 mon.c (mon.2) 596 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.229848+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:01 smithi050 bash[29234]: audit 2024-04-26T16:28:00.229848+0000 mon.a (mon.0) 2588 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:01.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:01.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:01.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:01.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:01.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:01.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:01.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:01.901 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:01.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:02.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:02.026 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:02.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:02.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: audit 2024-04-26T16:28:01.176665+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:02.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: audit 2024-04-26T16:28:01.176665+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:02.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: audit 2024-04-26T16:28:01.182313+0000 mon.c (mon.2) 597 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.232 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: audit 2024-04-26T16:28:01.182313+0000 mon.c (mon.2) 597 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: cluster 2024-04-26T16:28:01.183833+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e413: 8 total, 7 up, 7 in 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: cluster 2024-04-26T16:28:01.183833+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e413: 8 total, 7 up, 7 in 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: audit 2024-04-26T16:28:01.184662+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: audit 2024-04-26T16:28:01.184662+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: audit 2024-04-26T16:28:01.176665+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: audit 2024-04-26T16:28:01.176665+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: audit 2024-04-26T16:28:01.182313+0000 mon.c (mon.2) 597 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: audit 2024-04-26T16:28:01.182313+0000 mon.c (mon.2) 597 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: cluster 2024-04-26T16:28:01.183833+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e413: 8 total, 7 up, 7 in 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: cluster 2024-04-26T16:28:01.183833+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e413: 8 total, 7 up, 7 in 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: audit 2024-04-26T16:28:01.184662+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.233 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: audit 2024-04-26T16:28:01.184662+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: cluster 2024-04-26T16:28:01.631510+0000 mgr.y (mgr.24401) 418 : cluster [DBG] pgmap v604: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:02.234 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[29234]: cluster 2024-04-26T16:28:01.631510+0000 mgr.y (mgr.24401) 418 : cluster [DBG] pgmap v604: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:02.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:02.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: audit 2024-04-26T16:28:01.176665+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:02.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: audit 2024-04-26T16:28:01.176665+0000 mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi050-73201-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: audit 2024-04-26T16:28:01.182313+0000 mon.c (mon.2) 597 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: audit 2024-04-26T16:28:01.182313+0000 mon.c (mon.2) 597 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: cluster 2024-04-26T16:28:01.183833+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e413: 8 total, 7 up, 7 in 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: cluster 2024-04-26T16:28:01.183833+0000 mon.a (mon.0) 2590 : cluster [DBG] osdmap e413: 8 total, 7 up, 7 in 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: audit 2024-04-26T16:28:01.184662+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: audit 2024-04-26T16:28:01.184662+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: cluster 2024-04-26T16:28:01.631510+0000 mgr.y (mgr.24401) 418 : cluster [DBG] pgmap v604: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:02.273 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:02 smithi190 bash[23868]: cluster 2024-04-26T16:28:01.631510+0000 mgr.y (mgr.24401) 418 : cluster [DBG] pgmap v604: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:02.291 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:02.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:02.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: cluster 2024-04-26T16:28:01.631510+0000 mgr.y (mgr.24401) 418 : cluster [DBG] pgmap v604: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:02.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:02 smithi050 bash[21463]: cluster 2024-04-26T16:28:01.631510+0000 mgr.y (mgr.24401) 418 : cluster [DBG] pgmap v604: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:02.424 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:02.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:02.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:02.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:02.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:02.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:02.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:02.870 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:28:02.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:02.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:02.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:03.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:03.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:03.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:03.208 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:03.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:03.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:03.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[29234]: cluster 2024-04-26T16:28:02.193975+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e414: 8 total, 7 up, 7 in 2024-04-26T16:28:03.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[29234]: cluster 2024-04-26T16:28:02.193975+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e414: 8 total, 7 up, 7 in 2024-04-26T16:28:03.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[29234]: audit 2024-04-26T16:28:03.181628+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:03.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[29234]: audit 2024-04-26T16:28:03.181628+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:03.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[21463]: cluster 2024-04-26T16:28:02.193975+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e414: 8 total, 7 up, 7 in 2024-04-26T16:28:03.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[21463]: cluster 2024-04-26T16:28:02.193975+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e414: 8 total, 7 up, 7 in 2024-04-26T16:28:03.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[21463]: audit 2024-04-26T16:28:03.181628+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:03.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:03 smithi050 bash[21463]: audit 2024-04-26T16:28:03.181628+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:03.567 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:03 smithi190 bash[23868]: cluster 2024-04-26T16:28:02.193975+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e414: 8 total, 7 up, 7 in 2024-04-26T16:28:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:03 smithi190 bash[23868]: cluster 2024-04-26T16:28:02.193975+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e414: 8 total, 7 up, 7 in 2024-04-26T16:28:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:03 smithi190 bash[23868]: audit 2024-04-26T16:28:03.181628+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:03.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:03 smithi190 bash[23868]: audit 2024-04-26T16:28:03.181628+0000 mon.a (mon.0) 2593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi050-73201-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:03.591 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:03.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.724 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:03.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:03.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:03.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:03.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:03.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:04.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:04.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:04.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:04.182 INFO:teuthology.orchestra.run.smithi190.stdout: marking collection for removal 2024-04-26T16:28:04.182 INFO:teuthology.orchestra.run.smithi190.stdout:setting '_remove' omap key 2024-04-26T16:28:04.182 INFO:teuthology.orchestra.run.smithi190.stdout:finish_remove_pgs 2.11_head removing 2.11 2024-04-26T16:28:04.183 INFO:teuthology.orchestra.run.smithi190.stdout:Remove successful 2024-04-26T16:28:04.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[29234]: cluster 2024-04-26T16:28:03.189598+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e415: 8 total, 7 up, 7 in 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[29234]: cluster 2024-04-26T16:28:03.189598+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e415: 8 total, 7 up, 7 in 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[29234]: cluster 2024-04-26T16:28:03.632287+0000 mgr.y (mgr.24401) 419 : cluster [DBG] pgmap v607: 236 pgs: 8 unknown, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[29234]: cluster 2024-04-26T16:28:03.632287+0000 mgr.y (mgr.24401) 419 : cluster [DBG] pgmap v607: 236 pgs: 8 unknown, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[29234]: cluster 2024-04-26T16:28:04.188220+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e416: 8 total, 7 up, 7 in 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[29234]: cluster 2024-04-26T16:28:04.188220+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e416: 8 total, 7 up, 7 in 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21463]: cluster 2024-04-26T16:28:03.189598+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e415: 8 total, 7 up, 7 in 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21463]: cluster 2024-04-26T16:28:03.189598+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e415: 8 total, 7 up, 7 in 2024-04-26T16:28:04.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21463]: cluster 2024-04-26T16:28:03.632287+0000 mgr.y (mgr.24401) 419 : cluster [DBG] pgmap v607: 236 pgs: 8 unknown, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:04.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21463]: cluster 2024-04-26T16:28:03.632287+0000 mgr.y (mgr.24401) 419 : cluster [DBG] pgmap v607: 236 pgs: 8 unknown, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:04.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21463]: cluster 2024-04-26T16:28:04.188220+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e416: 8 total, 7 up, 7 in 2024-04-26T16:28:04.229 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21463]: cluster 2024-04-26T16:28:04.188220+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e416: 8 total, 7 up, 7 in 2024-04-26T16:28:04.239 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:04.341 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:04.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:04.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:04.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:04 smithi190 bash[23868]: cluster 2024-04-26T16:28:03.189598+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e415: 8 total, 7 up, 7 in 2024-04-26T16:28:04.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:04 smithi190 bash[23868]: cluster 2024-04-26T16:28:03.189598+0000 mon.a (mon.0) 2594 : cluster [DBG] osdmap e415: 8 total, 7 up, 7 in 2024-04-26T16:28:04.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:04 smithi190 bash[23868]: cluster 2024-04-26T16:28:03.632287+0000 mgr.y (mgr.24401) 419 : cluster [DBG] pgmap v607: 236 pgs: 8 unknown, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:04.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:04 smithi190 bash[23868]: cluster 2024-04-26T16:28:03.632287+0000 mgr.y (mgr.24401) 419 : cluster [DBG] pgmap v607: 236 pgs: 8 unknown, 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:04.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:04 smithi190 bash[23868]: cluster 2024-04-26T16:28:04.188220+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e416: 8 total, 7 up, 7 in 2024-04-26T16:28:04.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:04 smithi190 bash[23868]: cluster 2024-04-26T16:28:04.188220+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e416: 8 total, 7 up, 7 in 2024-04-26T16:28:04.495 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:04.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:04.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:04.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:04.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:04.756 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:04.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:04.870 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:28:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:28:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:28:04.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:04.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:04.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:05.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:05.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:05.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:05.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:05.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:05.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:05.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:05.527 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: cluster 2024-04-26T16:28:04.202281+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: cluster 2024-04-26T16:28:04.202281+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:04.570813+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:04.570813+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:04.572723+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:04.572723+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.186197+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.186197+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: cluster 2024-04-26T16:28:05.194283+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e417: 8 total, 7 up, 7 in 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: cluster 2024-04-26T16:28:05.194283+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e417: 8 total, 7 up, 7 in 2024-04-26T16:28:05.528 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.197114+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.197114+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.198443+0000 mon.c (mon.2) 598 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.198443+0000 mon.c (mon.2) 598 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.201568+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.201568+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.202372+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:05 smithi190 bash[23868]: audit 2024-04-26T16:28:05.202372+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.11.6 2024-04-26T16:28:05.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: cluster 2024-04-26T16:28:04.202281+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:05.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: cluster 2024-04-26T16:28:04.202281+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:04.570813+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:04.570813+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:04.572723+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:04.572723+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.186197+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.186197+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: cluster 2024-04-26T16:28:05.194283+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e417: 8 total, 7 up, 7 in 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: cluster 2024-04-26T16:28:05.194283+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e417: 8 total, 7 up, 7 in 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.197114+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.197114+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.198443+0000 mon.c (mon.2) 598 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.198443+0000 mon.c (mon.2) 598 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.201568+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.201568+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.202372+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[29234]: audit 2024-04-26T16:28:05.202372+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: cluster 2024-04-26T16:28:04.202281+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: cluster 2024-04-26T16:28:04.202281+0000 mon.a (mon.0) 2596 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:04.570813+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:04.570813+0000 mon.a (mon.0) 2597 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:04.572723+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.534 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:04.572723+0000 mon.b (mon.1) 282 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.186197+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.186197+0000 mon.a (mon.0) 2598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: cluster 2024-04-26T16:28:05.194283+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e417: 8 total, 7 up, 7 in 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: cluster 2024-04-26T16:28:05.194283+0000 mon.a (mon.0) 2599 : cluster [DBG] osdmap e417: 8 total, 7 up, 7 in 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.197114+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.197114+0000 mon.b (mon.1) 283 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.198443+0000 mon.c (mon.2) 598 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.198443+0000 mon.c (mon.2) 598 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.201568+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.201568+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.202372+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:05 smithi050 bash[21463]: audit 2024-04-26T16:28:05.202372+0000 mon.a (mon.0) 2601 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:05.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:05.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:05.675 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:05.699 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:28:05.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:05.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:05.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:05.929 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:05.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:06.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:06.055 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:06.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:06.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:06.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:06.308 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:06.411 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:06.437 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:06.470 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: cluster 2024-04-26T16:28:05.633491+0000 mgr.y (mgr.24401) 420 : cluster [DBG] pgmap v610: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:06.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: cluster 2024-04-26T16:28:05.633491+0000 mgr.y (mgr.24401) 420 : cluster [DBG] pgmap v610: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:06.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:05.738693+0000 mon.c (mon.2) 599 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:06.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:05.738693+0000 mon.c (mon.2) 599 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:06.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: cluster 2024-04-26T16:28:06.138632+0000 mon.a (mon.0) 2602 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:06.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: cluster 2024-04-26T16:28:06.138632+0000 mon.a (mon.0) 2602 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:06.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.141117+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:28:06.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.141117+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:28:06.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.141296+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:06.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.141296+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:06.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: cluster 2024-04-26T16:28:06.148949+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e418: 8 total, 7 up, 7 in 2024-04-26T16:28:06.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: cluster 2024-04-26T16:28:06.148949+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e418: 8 total, 7 up, 7 in 2024-04-26T16:28:06.473 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.150015+0000 mon.c (mon.2) 600 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.473 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.150015+0000 mon.c (mon.2) 600 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.473 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.150770+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.473 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[29234]: audit 2024-04-26T16:28:06.150770+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.473 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: cluster 2024-04-26T16:28:05.633491+0000 mgr.y (mgr.24401) 420 : cluster [DBG] pgmap v610: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:06.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: cluster 2024-04-26T16:28:05.633491+0000 mgr.y (mgr.24401) 420 : cluster [DBG] pgmap v610: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:06.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:05.738693+0000 mon.c (mon.2) 599 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:06.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:05.738693+0000 mon.c (mon.2) 599 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:06.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: cluster 2024-04-26T16:28:06.138632+0000 mon.a (mon.0) 2602 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:06.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: cluster 2024-04-26T16:28:06.138632+0000 mon.a (mon.0) 2602 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:06.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.141117+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:28:06.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.141117+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:28:06.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.141296+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:06.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.141296+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:06.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: cluster 2024-04-26T16:28:06.148949+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e418: 8 total, 7 up, 7 in 2024-04-26T16:28:06.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: cluster 2024-04-26T16:28:06.148949+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e418: 8 total, 7 up, 7 in 2024-04-26T16:28:06.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.150015+0000 mon.c (mon.2) 600 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.150015+0000 mon.c (mon.2) 600 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.150770+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:06 smithi050 bash[21463]: audit 2024-04-26T16:28:06.150770+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:06.560 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: cluster 2024-04-26T16:28:05.633491+0000 mgr.y (mgr.24401) 420 : cluster [DBG] pgmap v610: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: cluster 2024-04-26T16:28:05.633491+0000 mgr.y (mgr.24401) 420 : cluster [DBG] pgmap v610: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:05.738693+0000 mon.c (mon.2) 599 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:06.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:05.738693+0000 mon.c (mon.2) 599 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: cluster 2024-04-26T16:28:06.138632+0000 mon.a (mon.0) 2602 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: cluster 2024-04-26T16:28:06.138632+0000 mon.a (mon.0) 2602 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.141117+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.141117+0000 mon.a (mon.0) 2603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]': finished 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.141296+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.141296+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: cluster 2024-04-26T16:28:06.148949+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e418: 8 total, 7 up, 7 in 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: cluster 2024-04-26T16:28:06.148949+0000 mon.a (mon.0) 2605 : cluster [DBG] osdmap e418: 8 total, 7 up, 7 in 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.150015+0000 mon.c (mon.2) 600 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.150015+0000 mon.c (mon.2) 600 : audit [INF] from='client.? 172.21.15.50:0/2815871730' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.150770+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[23868]: audit 2024-04-26T16:28:06.150770+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]: dispatch 2024-04-26T16:28:06.646 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:28:06.646 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:28:06.663 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:06.683 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:06.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:06.806 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:06.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:28:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:28:06.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:06.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:06.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:07.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:07.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: 163:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:5d165639:::164:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f43765fc:::165:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b4c720e9:::166:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e694b040:::167:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:afa38db2:::168:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:77ba9f53:::169:head 2024-04-26T16:28:07.159 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:87495034:::170:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7c96bf0e:::171:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:dbe346cc:::172:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:e943ec24:::173:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f97a9c0c:::174:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:6f26e74d:::175:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4f95e106:::176:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:0e6f2f8f:::177:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:05db05f1:::178:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:38a78d66:::179:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:d095610b:::180:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:a1a9d709:::181:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:1e5d39db:::182:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:f7df4fb9:::183:head 2024-04-26T16:28:07.160 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:03a7f161:::184:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ba70721e:::185:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:28e5662d:::186:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:973d52de:::187:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:4303eb1c:::188:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:b990b48e:::189:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:29b8165b:::190:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:3547f197:::191:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:7e260936:::192:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:1abec7b1:::193:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:10fdda93:::194:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:15817eea:::195:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:770bab57:::196:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:ed9e13e7:::197:head 2024-04-26T16:28:07.161 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:71471a8f:::198:head 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: checking for 277:10fb1d02:::199:head 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8090 ms) 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: first is 1714148842 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,1714148847,1714148848,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,1714148847,1714148848,0 2024-04-26T16:28:07.162 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,1714148847,1714148848,0 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,1714148847,1714148848,1714148850,1714148851,0 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,1714148847,1714148848,1714148850,1714148851,0 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148842,1714148844,1714148845,1714148847,1714148848,1714148850,1714148851,0 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714148845,1714148847,1714148848,1714148850,1714148851,1714148853,1714148854,0 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: first now 1714148845, trimmed 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20940 ms) 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: foo0 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13994 ms) 2024-04-26T16:28:07.163 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-04-26T16:28:07.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:07.189 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:07.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:07.317 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:07.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:07.447 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:07.502 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:28:07.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.256740+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.256740+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.257776+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.257776+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.259006+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.259006+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.260154+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.260154+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.264852+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.264852+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.277255+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.277255+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.924991+0000 mon.c (mon.2) 601 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.924991+0000 mon.c (mon.2) 601 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.926161+0000 mon.c (mon.2) 602 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:07.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.926161+0000 mon.c (mon.2) 602 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.932484+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:06.932484+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.144005+0000 mon.a (mon.0) 2612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.144005+0000 mon.a (mon.0) 2612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.163441+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e419: 8 total, 7 up, 7 in 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.163441+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e419: 8 total, 7 up, 7 in 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.200190+0000 mon.c (mon.2) 603 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.200190+0000 mon.c (mon.2) 603 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.200653+0000 mon.a (mon.0) 2614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.200653+0000 mon.a (mon.0) 2614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.201729+0000 mon.c (mon.2) 604 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.201729+0000 mon.c (mon.2) 604 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.202066+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.202066+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.202764+0000 mon.c (mon.2) 605 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.202764+0000 mon.c (mon.2) 605 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.203145+0000 mon.a (mon.0) 2616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:07 smithi190 bash[23868]: audit 2024-04-26T16:28:07.203145+0000 mon.a (mon.0) 2616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:07.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.256740+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.256740+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.257776+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.257776+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.259006+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.259006+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.260154+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.260154+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.264852+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.264852+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.277255+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.277255+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.924991+0000 mon.c (mon.2) 601 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.924991+0000 mon.c (mon.2) 601 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.926161+0000 mon.c (mon.2) 602 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:07.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.926161+0000 mon.c (mon.2) 602 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.932484+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:06.932484+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.256740+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.256740+0000 mon.a (mon.0) 2607 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.257776+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.257776+0000 mon.a (mon.0) 2608 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.259006+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.259006+0000 mon.b (mon.1) 284 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.260154+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.260154+0000 mon.b (mon.1) 285 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-49"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.264852+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.264852+0000 mon.a (mon.0) 2609 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.277255+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.277255+0000 mon.a (mon.0) 2610 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.924991+0000 mon.c (mon.2) 601 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.924991+0000 mon.c (mon.2) 601 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.926161+0000 mon.c (mon.2) 602 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.926161+0000 mon.c (mon.2) 602 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.932484+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:06.932484+0000 mon.a (mon.0) 2611 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.144005+0000 mon.a (mon.0) 2612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:07.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.144005+0000 mon.a (mon.0) 2612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.163441+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e419: 8 total, 7 up, 7 in 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.163441+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e419: 8 total, 7 up, 7 in 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.200190+0000 mon.c (mon.2) 603 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.200190+0000 mon.c (mon.2) 603 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.200653+0000 mon.a (mon.0) 2614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.200653+0000 mon.a (mon.0) 2614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.201729+0000 mon.c (mon.2) 604 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.201729+0000 mon.c (mon.2) 604 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.202066+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.202066+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.202764+0000 mon.c (mon.2) 605 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.202764+0000 mon.c (mon.2) 605 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.203145+0000 mon.a (mon.0) 2616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[29234]: audit 2024-04-26T16:28:07.203145+0000 mon.a (mon.0) 2616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.144005+0000 mon.a (mon.0) 2612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.144005+0000 mon.a (mon.0) 2612 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi050-73201-65"}]': finished 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.163441+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e419: 8 total, 7 up, 7 in 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.163441+0000 mon.a (mon.0) 2613 : cluster [DBG] osdmap e419: 8 total, 7 up, 7 in 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.200190+0000 mon.c (mon.2) 603 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.200190+0000 mon.c (mon.2) 603 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.200653+0000 mon.a (mon.0) 2614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.200653+0000 mon.a (mon.0) 2614 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.201729+0000 mon.c (mon.2) 604 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.201729+0000 mon.c (mon.2) 604 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.202066+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.202066+0000 mon.a (mon.0) 2615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.202764+0000 mon.c (mon.2) 605 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.202764+0000 mon.c (mon.2) 605 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.203145+0000 mon.a (mon.0) 2616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:07 smithi050 bash[21463]: audit 2024-04-26T16:28:07.203145+0000 mon.a (mon.0) 2616 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:28:07.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:07.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:07.710 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:07.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:07.839 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:07.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:07.971 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:07.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:08.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:08.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:08.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:08.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:06.851026+0000 mgr.y (mgr.24401) 421 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:06.851026+0000 mgr.y (mgr.24401) 421 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.353479+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.353479+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.353782+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.353782+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:08.337 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.634297+0000 mgr.y (mgr.24401) 422 : cluster [DBG] pgmap v613: 196 pgs: 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:08.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:07.634297+0000 mgr.y (mgr.24401) 422 : cluster [DBG] pgmap v613: 196 pgs: 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:08.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.146464+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:08.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.146464+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:08.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:08.150486+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e420: 8 total, 7 up, 7 in 2024-04-26T16:28:08.338 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: cluster 2024-04-26T16:28:08.150486+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e420: 8 total, 7 up, 7 in 2024-04-26T16:28:08.338 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:06.851026+0000 mgr.y (mgr.24401) 421 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:08.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:08.365 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- 172.21.15.50:0/350303565 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188102320 msgr2=0x7ff1881027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- 172.21.15.50:0/350303565 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188102320 0x7ff1881027a0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7ff1780098f0 tx=0x7ff178009c00 comp rx=0 tx=0).stop 2024-04-26T16:28:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- 172.21.15.50:0/350303565 shutdown_connections 2024-04-26T16:28:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- 172.21.15.50:0/350303565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff188102ce0 0x7ff18810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- 172.21.15.50:0/350303565 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188108390 0x7ff188108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- 172.21.15.50:0/350303565 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188102320 0x7ff1881027a0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- 172.21.15.50:0/350303565 >> 172.21.15.50:0/350303565 conn(0x7ff1880792d0 msgr2=0x7ff1880796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- 172.21.15.50:0/350303565 shutdown_connections 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- 172.21.15.50:0/350303565 wait complete. 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 librados: starting msgr at 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 librados: starting objecter 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 Processor -- start 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- start start 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 librados: setting wanted keys 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 librados: calling monclient init 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188102320 0x7ff18819b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff188102ce0 0x7ff18819b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188108390 0x7ff18819fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff188112b00 con 0x7ff188102320 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff188112980 con 0x7ff188102ce0 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18fac4700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff188112c80 con 0x7ff188108390 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18d860700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188102320 0x7ff18819b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188108390 0x7ff18819fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18d860700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188102320 0x7ff18819b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36500/0 (socket says 172.21.15.50:36500) 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188108390 0x7ff18819fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37514/0 (socket says 172.21.15.50:37514) 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18d860700 1 -- 172.21.15.50:0/2738094435 learned_addr learned my addr 172.21.15.50:0/2738094435 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff188102ce0 0x7ff18819b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 -- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff188102ce0 msgr2=0x7ff18819b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff188102ce0 0x7ff18819b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 -- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188102320 msgr2=0x7ff18819b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188102320 0x7ff18819b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18e061700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1881a0390 con 0x7ff188108390 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.462+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff188102320 0x7ff18819b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff188108390 0x7ff18819fc50 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7ff18000cc50 tx=0x7ff18000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 10 client.?.objecter ms_handle_connect 0x7ff188108390 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff180016020 con 0x7ff188108390 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff180012420 con 0x7ff188108390 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1881a0680 con 0x7ff188108390 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1800115e0 con 0x7ff188108390 2024-04-26T16:28:08.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 10 client.41183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1881a7f00 con 0x7ff188108390 2024-04-26T16:28:08.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 1 librados: init done 2024-04-26T16:28:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff18806a110 con 0x7ff188108390 2024-04-26T16:28:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff180019020 con 0x7ff188108390 2024-04-26T16:28:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff174071f80 0x7ff174074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(420..420 src has 87..420) ==== 9675+0+0 (secure 0 0 0) 0x7ff180091a40 con 0x7ff188108390 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 osd_map(420..420 src has 87..420) 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 3 client.41183.objecter handle_osd_map got epochs [420,420] > 0 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 3 client.41183.objecter handle_osd_map decoding full epoch 420 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff174071f80 0x7ff174074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 20 client.41183.objecter dump_active .. 0 homeless 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v420) ==== 54+0+16 (secure 0 0 0) 0x7ff180091e20 con 0x7ff188108390 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 10 client.41183.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 1 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff188064030 0x7ff188064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 1 2024-04-26T16:28:08.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 0 1 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff18819c6e0 osd=0 4 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 10 client.41183.objecter _send_command 1 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1881ab260 con 0x7ff188064030 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff174071f80 0x7ff174074440 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7ff184009750 tx=0x7ff184009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff174071f80 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff188064030 0x7ff188064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff188064030 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.466+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff188064030 0x7ff188064410 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.470+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881ab260 con 0x7ff188064030 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.470+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 1: 0 ) 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.470+0000 7ff17effd700 10 client.41183.objecter _finish_command 1 = system:0 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.470+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff18819c6e0 osd=0 4 2024-04-26T16:28:08.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.470+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 0 1 2024-04-26T16:28:08.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:08.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.151499+0000 mon.c (mon.2) 606 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.471 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.151499+0000 mon.c (mon.2) 606 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.151941+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.151941+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.157877+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.157877+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.159729+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.472 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[29234]: audit 2024-04-26T16:28:08.159729+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:06.851026+0000 mgr.y (mgr.24401) 421 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:06.851026+0000 mgr.y (mgr.24401) 421 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.353479+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.353479+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.353782+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.353782+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.634297+0000 mgr.y (mgr.24401) 422 : cluster [DBG] pgmap v613: 196 pgs: 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:07.634297+0000 mgr.y (mgr.24401) 422 : cluster [DBG] pgmap v613: 196 pgs: 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.146464+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.146464+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:08.150486+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e420: 8 total, 7 up, 7 in 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: cluster 2024-04-26T16:28:08.150486+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e420: 8 total, 7 up, 7 in 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.151499+0000 mon.c (mon.2) 606 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.151499+0000 mon.c (mon.2) 606 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.151941+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.151941+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.157877+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.157877+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.159729+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:08 smithi190 bash[23868]: audit 2024-04-26T16:28:08.159729+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:06.851026+0000 mgr.y (mgr.24401) 421 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.353479+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.353479+0000 osd.7 (osd.7) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:08.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.353782+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.353782+0000 osd.7 (osd.7) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.634297+0000 mgr.y (mgr.24401) 422 : cluster [DBG] pgmap v613: 196 pgs: 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:07.634297+0000 mgr.y (mgr.24401) 422 : cluster [DBG] pgmap v613: 196 pgs: 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.146464+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.146464+0000 mon.a (mon.0) 2617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi050-73201-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:08.150486+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e420: 8 total, 7 up, 7 in 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: cluster 2024-04-26T16:28:08.150486+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e420: 8 total, 7 up, 7 in 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.151499+0000 mon.c (mon.2) 606 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.151499+0000 mon.c (mon.2) 606 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.151941+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.151941+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.157877+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.157877+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.475 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.159729+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.476 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:08 smithi050 bash[21463]: audit 2024-04-26T16:28:08.159729+0000 mon.b (mon.1) 286 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:08.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 10 client.41183.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 2 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 2 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff18819c6e0 osd=0 3 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 0 2 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff18819c6e0 osd=0 4 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 10 client.41183.objecter _send_command 2 2024-04-26T16:28:08.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.486+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff188063ab0 con 0x7ff188064030 2024-04-26T16:28:08.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff188063ab0 con 0x7ff188064030 2024-04-26T16:28:08.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 10 client.41183.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff18819c6e0 osd=0 4 2024-04-26T16:28:08.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 0 2 2024-04-26T16:28:08.488 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:28:08.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 10 client.41183.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff188104c10 0x7ff18802d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 1 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff188063eb0 osd=1 4 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 10 client.41183.objecter _send_command 3 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff1881aebc0 con 0x7ff188104c10 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff188104c10 0x7ff18802d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff188104c10 2024-04-26T16:28:08.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff188104c10 0x7ff18802d310 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.490 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:08.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881aebc0 con 0x7ff188104c10 2024-04-26T16:28:08.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 3: 0 ) 2024-04-26T16:28:08.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 10 client.41183.objecter _finish_command 3 = system:0 2024-04-26T16:28:08.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff188063eb0 osd=1 4 2024-04-26T16:28:08.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.490+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 1 3 2024-04-26T16:28:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 10 client.41183.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 4 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 4 2024-04-26T16:28:08.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff188063eb0 osd=1 3 2024-04-26T16:28:08.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 1 4 2024-04-26T16:28:08.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff188063eb0 osd=1 4 2024-04-26T16:28:08.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 10 client.41183.objecter _send_command 4 2024-04-26T16:28:08.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff188104180 con 0x7ff188104c10 2024-04-26T16:28:08.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff188104180 con 0x7ff188104c10 2024-04-26T16:28:08.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff17effd700 10 client.41183.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff188063eb0 osd=1 4 2024-04-26T16:28:08.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.510+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 1 4 2024-04-26T16:28:08.509 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 10 client.41183.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 5 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1881ad420 0x7ff1881ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 5 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 2 5 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa8b0 osd=2 4 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 10 client.41183.objecter _send_command 5 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff1881aaf30 con 0x7ff1881ad420 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1881ad420 0x7ff1881ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff1881ad420 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1881ad420 0x7ff1881ad860 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881aaf30 con 0x7ff1881ad420 2024-04-26T16:28:08.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 5: 0 ) 2024-04-26T16:28:08.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff17effd700 10 client.41183.objecter _finish_command 5 = system:0 2024-04-26T16:28:08.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881aa8b0 osd=2 4 2024-04-26T16:28:08.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.514+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 2 5 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 10 client.41183.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 6 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 6 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881aa8b0 osd=2 3 2024-04-26T16:28:08.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 2 6 2024-04-26T16:28:08.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa8b0 osd=2 4 2024-04-26T16:28:08.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 10 client.41183.objecter _send_command 6 2024-04-26T16:28:08.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.542+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff1881aa330 con 0x7ff1881ad420 2024-04-26T16:28:08.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1881aa330 con 0x7ff1881ad420 2024-04-26T16:28:08.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 10 client.41183.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881aa8b0 osd=2 4 2024-04-26T16:28:08.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 2 6 2024-04-26T16:28:08.542 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:28:08.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 10 client.41183.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 7 2024-04-26T16:28:08.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1881b0d80 0x7ff1881b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 7 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 3 7 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa730 osd=3 4 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 10 client.41183.objecter _send_command 7 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff1881b5fd0 con 0x7ff1881b0d80 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1881b0d80 0x7ff1881b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff1881b0d80 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1881b0d80 0x7ff1881b3170 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881b5fd0 con 0x7ff1881b0d80 2024-04-26T16:28:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 7: 0 ) 2024-04-26T16:28:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 10 client.41183.objecter _finish_command 7 = system:0 2024-04-26T16:28:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881aa730 osd=3 4 2024-04-26T16:28:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.546+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 3 7 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 10 client.41183.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 8 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 8 2024-04-26T16:28:08.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881aa730 osd=3 3 2024-04-26T16:28:08.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 3 8 2024-04-26T16:28:08.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881aa730 osd=3 4 2024-04-26T16:28:08.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 10 client.41183.objecter _send_command 8 2024-04-26T16:28:08.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff1881ae050 con 0x7ff1881b0d80 2024-04-26T16:28:08.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1881ae050 con 0x7ff1881b0d80 2024-04-26T16:28:08.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff17effd700 10 client.41183.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881aa730 osd=3 4 2024-04-26T16:28:08.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.566+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 3 8 2024-04-26T16:28:08.565 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:28:08.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 10 client.41183.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 9 2024-04-26T16:28:08.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff18802e530 0x7ff1881b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 9 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 4 9 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ade00 osd=4 4 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 10 client.41183.objecter _send_command 9 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff1881b9930 con 0x7ff18802e530 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff18802e530 0x7ff1881b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff18802e530 2024-04-26T16:28:08.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff18d860700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff18802e530 0x7ff1881b4830 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881b9930 con 0x7ff18802e530 2024-04-26T16:28:08.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 9: 0 ) 2024-04-26T16:28:08.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff17effd700 10 client.41183.objecter _finish_command 9 = system:0 2024-04-26T16:28:08.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881ade00 osd=4 4 2024-04-26T16:28:08.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.570+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 4 9 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 10 client.41183.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 10 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 10 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881ade00 osd=4 3 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 4 10 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ade00 osd=4 4 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 10 client.41183.objecter _send_command 10 2024-04-26T16:28:08.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.586+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff1881b4d70 con 0x7ff18802e530 2024-04-26T16:28:08.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1881b4d70 con 0x7ff18802e530 2024-04-26T16:28:08.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff17effd700 10 client.41183.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881ade00 osd=4 4 2024-04-26T16:28:08.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 4 10 2024-04-26T16:28:08.588 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:28:08.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 10 client.41183.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 11 2024-04-26T16:28:08.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1881b5490 0x7ff1881b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 11 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 5 11 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881b4d70 osd=5 4 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 10 client.41183.objecter _send_command 11 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff1881bcb10 con 0x7ff1881b5490 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.590+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1881b5490 0x7ff1881b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff1881b5490 2024-04-26T16:28:08.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff18d05f700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1881b5490 0x7ff1881b58f0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881bcb10 con 0x7ff1881b5490 2024-04-26T16:28:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 11: 0 ) 2024-04-26T16:28:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff17effd700 10 client.41183.objecter _finish_command 11 = system:0 2024-04-26T16:28:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881b4d70 osd=5 4 2024-04-26T16:28:08.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.594+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 5 11 2024-04-26T16:28:08.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:08.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 10 client.41183.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 12 2024-04-26T16:28:08.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 12 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881b4d70 osd=5 3 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 5 12 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881b4d70 osd=5 4 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 10 client.41183.objecter _send_command 12 2024-04-26T16:28:08.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.614+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff1881ba1a0 con 0x7ff1881b5490 2024-04-26T16:28:08.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:08.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1881ba1a0 con 0x7ff1881b5490 2024-04-26T16:28:08.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 10 client.41183.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881b4d70 osd=5 4 2024-04-26T16:28:08.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 5 12 2024-04-26T16:28:08.615 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 10 client.41183.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 13 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 20 client.41183.objecter _get_session osd=-1 returning homeless 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 10 client.41183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=421}) -- 0x7ff1881bbba0 con 0x7ff188108390 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1881bbee0 con 0x7ff188108390 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_get_version_reply(handle=1 version=420) ==== 24+0+0 (secure 0 0 0) 0x7ff18005d2d0 con 0x7ff188108390 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17ffff700 10 client.41183.objecter _check_command_map_dne tid 13 current 420 map_dne_bound 420 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17ffff700 10 client.41183.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17ffff700 15 client.41183.objecter _session_command_op_remove -1 13 2024-04-26T16:28:08.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 10 client.41183.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 14 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1881ba3b0 0x7ff1881ba7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 14 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 7 14 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ba1d0 osd=7 4 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 10 client.41183.objecter _send_command 14 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7ff1881badc0 con 0x7ff1881ba3b0 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1881ba3b0 0x7ff1881ba7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff17effd700 10 client.41183.objecter ms_handle_connect 0x7ff1881ba3b0 2024-04-26T16:28:08.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.618+0000 7ff18e061700 1 --2- 172.21.15.50:0/2738094435 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1881ba3b0 0x7ff1881ba7f0 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.622+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1881badc0 con 0x7ff1881ba3b0 2024-04-26T16:28:08.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.622+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 14: 0 ) 2024-04-26T16:28:08.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.622+0000 7ff17effd700 10 client.41183.objecter _finish_command 14 = system:0 2024-04-26T16:28:08.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.622+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881ba1d0 osd=7 4 2024-04-26T16:28:08.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.622+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 7 14 2024-04-26T16:28:08.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 10 client.41183.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:08.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign -1 15 2024-04-26T16:28:08.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 20 client.41183.objecter _get_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_remove -1 15 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 20 client.41183.objecter get_session s=0x7ff1881ba1d0 osd=7 3 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 15 client.41183.objecter _session_command_op_assign 7 15 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 20 client.41183.objecter put_session s=0x7ff1881ba1d0 osd=7 4 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 10 client.41183.objecter _send_command 15 2024-04-26T16:28:08.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.638+0000 7ff18fac4700 1 -- 172.21.15.50:0/2738094435 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff1881bad80 con 0x7ff1881ba3b0 2024-04-26T16:28:08.639 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.642+0000 7ff17effd700 1 -- 172.21.15.50:0/2738094435 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1881bad80 con 0x7ff1881ba3b0 2024-04-26T16:28:08.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.642+0000 7ff17effd700 10 client.41183.objecter ms_dispatch 0x7ff188001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:08.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.642+0000 7ff17effd700 10 client.41183.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:08.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.642+0000 7ff17effd700 20 client.41183.objecter put_session s=0x7ff1881ba1d0 osd=7 4 2024-04-26T16:28:08.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.642+0000 7ff17effd700 15 client.41183.objecter _session_command_op_remove 7 15 2024-04-26T16:28:08.640 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:28:08.641 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:28:08.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 -- 172.21.15.50:0/4236156011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 msgr2=0x7f2654108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:08.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 --2- 172.21.15.50:0/4236156011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654108760 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f26440098f0 tx=0x7f2644009c00 comp rx=0 tx=0).stop 2024-04-26T16:28:08.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 -- 172.21.15.50:0/4236156011 shutdown_connections 2024-04-26T16:28:08.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 --2- 172.21.15.50:0/4236156011 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102cd0 0x7f265410f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 --2- 172.21.15.50:0/4236156011 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654108760 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 --2- 172.21.15.50:0/4236156011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2654102310 0x7f2654102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 -- 172.21.15.50:0/4236156011 >> 172.21.15.50:0/4236156011 conn(0x7f2654079290 msgr2=0x7f26540796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 -- 172.21.15.50:0/4236156011 shutdown_connections 2024-04-26T16:28:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 -- 172.21.15.50:0/4236156011 wait complete. 2024-04-26T16:28:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 librados: starting msgr at 2024-04-26T16:28:08.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 librados: starting objecter 2024-04-26T16:28:08.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 Processor -- start 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 -- start start 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 librados: setting wanted keys 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.658+0000 7f2659e54700 1 librados: calling monclient init 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102310 0x7f2654103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2654102cd0 0x7f2654104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f26541119d0 con 0x7f2654108380 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2654111850 con 0x7f2654102cd0 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f2654111b50 con 0x7f2654102310 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f26537fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102310 0x7f2654103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654104a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36510/0 (socket says 172.21.15.50:36510) 2024-04-26T16:28:08.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 -- 172.21.15.50:0/3205801944 learned_addr learned my addr 172.21.15.50:0/3205801944 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2652ffd700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2654102cd0 0x7f2654104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 -- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102310 msgr2=0x7f2654103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102310 0x7f2654103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 -- 172.21.15.50:0/3205801944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2654102cd0 msgr2=0x7f2654104530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2654102cd0 0x7f2654104530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 -- 172.21.15.50:0/3205801944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26541a7690 con 0x7f2654108380 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f26537fe700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102310 0x7f2654103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2653fff700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654104a70 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f264c00e3c0 tx=0x7f264c00e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2650ff9700 10 client.?.objecter ms_handle_connect 0x7f2654108380 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2650ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c0092d0 con 0x7f2654108380 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f264c00ee70 con 0x7f2654108380 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26541a7980 con 0x7f2654108380 2024-04-26T16:28:08.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 10 client.30576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:08.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c007c70 con 0x7f2654108380 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26541a7e70 con 0x7f2654108380 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 librados: init done 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2654103a30 con 0x7f2654108380 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.662+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f264c012050 con 0x7f2654108380 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f263c071e50 0x7f263c074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(420..420 src has 87..420) ==== 9675+0+0 (secure 0 0 0) 0x7f264c0912b0 con 0x7f2654108380 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 10 client.30576.objecter ms_dispatch 0x7f2654001550 osd_map(420..420 src has 87..420) 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 3 client.30576.objecter handle_osd_map got epochs [420,420] > 0 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 3 client.30576.objecter handle_osd_map decoding full epoch 420 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f26537fe700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f263c071e50 0x7f263c074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 20 client.30576.objecter dump_active .. 0 homeless 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f2650ff9700 10 client.30576.objecter ms_handle_connect 0x7f263c071e50 2024-04-26T16:28:08.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.666+0000 7f26537fe700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f263c071e50 0x7f263c074310 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7f2644009810 tx=0x7f2644019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:08.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.670+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f264c010040 con 0x7f2654108380 2024-04-26T16:28:08.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:08.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-26T16:28:08.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:08.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:08.843 INFO:teuthology.orchestra.run.smithi190.stdout:get_pg_num_history pg_num_history pg_num_history(e398 pg_nums {1={20=1},2={78=32},3={80=32},4={81=3},5={82=32},6={84=32},7={140=32},8={140=32},9={140=32},10={140=32},11={140=32},12={140=32,152=31,156=30,160=29,164=28,168=27,172=26,176=25,180=24,184=23,188=22,192=21,196=20,200=19,204=18,208=17,212=16,216=15,220=14,224=13,228=12,232=11},13={140=32},14={140=32},15={141=32},16={141=32},17={141=32},18={141=32},19={141=32},20={141=32},21={141=32},22={141=32},23={141=32},24={141=32},25={141=32},26={141=32},27={141=32},28={141=32},29={141=32},30={141=32},31={141=32},32={141=32},33={141=32},34={141=32},35={141=32},36={141=32},37={141=32},38={141=32},39={141=32},40={141=32},41={141=32},42={143=8},43={143=8},44={143=8},45={143=32},46={143=32},47={143=32},48={143=32},49={143=32},50={143=32},51={143=32},52={144=32},53={144=32},54={144=32},55={144=32},56={144=32},57={144=32},58={144=32},59={144=32},60={144=32},61={144=32},62={144=32},63={145=8},64={145=8},65={145=32},66={145=32},67={145=32},68={145=32},69={145=32},70={146=8},71={146=8},72={146=32},73={146=32},74={147=8},75={147=8},76={147=32},77={147=32},78={147=32},79={147=32},80={147=32},81={147=32},82={147=32},83={148=32},84={148=32},85={149=8},86={149=32},87={149=32},88={149=32},89={150=8},90={150=32},91={150=32},92={150=32},93={150=32},94={150=32},95={150=4},96={151=32},97={151=32},98={151=32},99={152=32},100={152=32},101={153=32},102={153=32},103={153=32},104={153=32},105={153=32},106={153=32},107={154=32},108={155=32},109={155=32},110={155=32},111={156=8},112={156=32},113={156=32},114={156=32},115={156=32},116={156=32},117={156=32},118={157=8},119={157=32},120={157=32},121={158=32},122={159=32},123={159=32},124={159=32},125={159=32},126={159=32},127={160=32},128={160=32},129={161=32},130={161=32},131={161=32},132={162=32},133={162=32},134={163=32},135={163=32},136={164=32},137={164=32},138={164=32},139={165=32},140={165=32},141={166=32},142={166=32},143={167=32},144={167=32},145={167=32},146={168=32},147={168=32},148={168=32},149={169=32},150={170=8},151={170=32},152={170=32},153={171=32},154={171=32},155={171=32},156={171=32},157={172=32},158={172=32},159={173=32},160={173=32},161={173=32},162={174=32},163={174=32},164={176=32},165={176=32},166={177=32},167={177=32},168={178=32},169={179=32},170={179=32},171={180=32},172={180=32},173={182=32},174={182=32},175={182=32},176={185=8},177={185=32},178={185=32},179={186=32},180={188=32},181={188=32},182={191=32},183={191=32},184={192=32},185={194=32},186={194=32},187={196=32},188={197=32},189={197=32},190={197=32},191={200=32},192={200=32},193={203=32},194={203=32},195={204=8},196={206=32},197={206=32},198={209=32},199={209=32},200={212=32},201={212=32},202={215=8},203={215=32},204={215=32},205={216=32},206={218=32},207={219=8},208={220=8},209={221=32},210={223=32},211={224=32},212={226=32},213={227=8},214={227=32},215={228=32},216={229=32},217={230=32},218={233=32},219={234=8},220={234=8},221={236=32},222={238=32},223={239=32},224={241=8},225={242=32},226={245=32},227={246=32},228={248=8},229={248=32},230={252=32},231={254=32},232={255=8},233={255=32},234={258=32},235={260=32},236={262=8},237={262=32},238={262=32},239={267=32},240={269=8},241={269=32},242={271=32},243={272=32},244={273=32},245={275=32},246={276=8},247={281=32},248={282=32},249={283=8},250={285=32},251={290=8},252={290=8},253={291=32},254={294=8},255={297=8},256={297=8},257={299=32},258={304=8},259={304=8},260={304=32},261={311=8},262={311=8},263={312=32},264={318=8},265={318=8},266={321=32},267={323=32},268={325=8},269={328=8},270={334=32},271={335=8},272={342=8},273={342=32},274={349=8},275={350=32},276={356=8},277={359=32},278={363=8},279={367=32},280={370=8},281={377=8},282={382=32},283={387=8},284={388=32},285={394=8}} deleted_pools 142,7,142,8,142,9,142,10,142,31,142,35,142,37,143,15,143,19,143,22,143,23,143,24,143,25,143,27,143,28,143,29,143,30,143,32,143,33,143,34,143,36,143,38,143,39,143,41,144,13,144,14,144,26,144,43,144,45,144,46,144,47,145,40,145,42,145,44,145,49,146,54,146,55,146,56,146,57,146,58,146,59,146,60,146,61,146,62,146,69,147,48,147,63,147,64,147,67,147,68,148,52,148,70,148,71,148,73,148,82,149,76,149,77,149,78,149,79,149,80,149,81,150,53,150,72,150,84,150,86,151,16,151,74,151,75,151,85,151,88,152,65,152,89,152,90,152,91,152,92,152,93,152,94,152,96,153,21,153,51,153,98,154,18,154,97,154,100,154,105,155,20,155,101,155,102,155,103,155,104,155,106,156,107,157,110,158,111,158,113,158,114,158,115,158,116,158,117,159,66,159,108,159,120,160,50,160,121,160,122,161,125,161,126,162,127,163,118,163,119,163,129,163,131,164,132,164,133,164,134,165,130,165,135,166,128,166,137,166,138,167,99,167,139,167,140,168,95,168,123,168,124,168,142,169,83,169,144,169,145,169,146,170,141,170,147,170,148,170,149,171,136,171,143,172,151,172,152,173,153,173,155,173,156,175,160,175,161,176,162,176,163,177,158,177,159,178,154,178,164,178,165,179,166,179,167,180,87,180,150,181,157,181,169,181,170,182,171,182,172,184,174,184,175,185,168,187,177,187,178,190,180,190,181,191,179,193,182,193,183,194,176,195,184,196,112,196,185,196,186,199,188,199,190,201,189,202,191,202,192,205,193,205,194,208,196,208,197,211,198,211,199,212,17,214,195,214,200,214,201,215,187,217,203,217,204,219,205,220,202,220,206,222,208,223,209,225,210,226,211,227,173,228,212,229,213,229,214,231,216,232,207,232,217,235,218,236,219,236,220,237,215,238,221,241,223,243,224,244,225,245,222,247,226,250,228,251,229,253,227,254,230,257,232,257,233,261,231,264,236,264,238,265,235,266,234,271,237,271,240,277,245,278,244,278,246,279,242,280,241,280,243,281,239,284,248,285,249,287,250,290,247,291,12,292,251,292,252,296,254,299,255,299,256,301,257,303,253,306,258,306,259,311,260,313,261,313,262,320,263,320,264,320,265,325,267,327,268,330,269,333,266,337,271,341,270,344,272,349,273,351,274,358,275,358,276,365,278,366,277,372,280,379,281,384,282,387,279,389,283,396,285) 2024-04-26T16:28:08.843 INFO:teuthology.orchestra.run.smithi190.stdout:Importing pgid 2.11 2024-04-26T16:28:08.843 INFO:teuthology.orchestra.run.smithi190.stdout:write_pg epoch 366 info 2.11( empty local-lis/les=78/79 n=0 ec=78/78 lis/c=78/78 les/c/f=79/79/0 sis=78) 2024-04-26T16:28:08.843 INFO:teuthology.orchestra.run.smithi190.stdout:Import successful 2024-04-26T16:28:08.859 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:08.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:08.910+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f2654105520 con 0x7f2654108380 2024-04-26T16:28:08.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:08.995 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:08.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:09.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:09.125 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:09.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.146+0000 7f2650ff9700 1 -- 172.21.15.50:0/3205801944 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v421) ==== 97+0+0 (secure 0 0 0) 0x7f264c05cb40 con 0x7f2654108380 2024-04-26T16:28:09.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.154+0000 7f2659e54700 10 librados: watch_flush enter 2024-04-26T16:28:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.154+0000 7f2659e54700 10 librados: watch_flush exit 2024-04-26T16:28:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.154+0000 7f2659e54700 20 client.30576.objecter shutdown clearing up homeless session... 2024-04-26T16:28:09.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.154+0000 7f2659e54700 10 client.30576.objecter successfully canceled tick 2024-04-26T16:28:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.158+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f263c071e50 msgr2=0x7f263c074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.158+0000 7f2659e54700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f263c071e50 0x7f263c074310 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7f2644009810 tx=0x7f2644019040 comp rx=0 tx=0).stop 2024-04-26T16:28:09.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.158+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 msgr2=0x7f2654104a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.158+0000 7f2659e54700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654104a70 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f264c00e3c0 tx=0x7f264c00e6d0 comp rx=0 tx=0).stop 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 shutdown_connections 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f263c071e50 0x7f263c074310 unknown :-1 s=CLOSED pgs=3743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2654108380 0x7f2654104a70 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2654102310 0x7f2654103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 --2- 172.21.15.50:0/3205801944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2654102cd0 0x7f2654104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:09.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 >> 172.21.15.50:0/3205801944 conn(0x7f2654079290 msgr2=0x7f26541062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 shutdown_connections 2024-04-26T16:28:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 -- 172.21.15.50:0/3205801944 wait complete. 2024-04-26T16:28:09.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:09.162+0000 7f2659e54700 1 librados: shutdown 2024-04-26T16:28:09.176 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:28:09.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:09.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:09.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:09.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:08.469305+0000 mon.c (mon.2) 607 : audit [DBG] from='client.? 172.21.15.50:0/2738094435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:08.469305+0000 mon.c (mon.2) 607 : audit [DBG] from='client.? 172.21.15.50:0/2738094435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:08.912955+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:08.912955+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: cluster 2024-04-26T16:28:09.148030+0000 mon.a (mon.0) 2622 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: cluster 2024-04-26T16:28:09.148030+0000 mon.a (mon.0) 2622 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.150386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.150386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:09.397 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.150537+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.150537+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: cluster 2024-04-26T16:28:09.181555+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e421: 8 total, 7 up, 7 in 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: cluster 2024-04-26T16:28:09.181555+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e421: 8 total, 7 up, 7 in 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.200668+0000 mon.a (mon.0) 2626 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.200668+0000 mon.a (mon.0) 2626 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.202759+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.398 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:09 smithi190 bash[23868]: audit 2024-04-26T16:28:09.202759+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:08.469305+0000 mon.c (mon.2) 607 : audit [DBG] from='client.? 172.21.15.50:0/2738094435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:08.469305+0000 mon.c (mon.2) 607 : audit [DBG] from='client.? 172.21.15.50:0/2738094435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:08.912955+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:08.912955+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: cluster 2024-04-26T16:28:09.148030+0000 mon.a (mon.0) 2622 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: cluster 2024-04-26T16:28:09.148030+0000 mon.a (mon.0) 2622 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.150386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.150386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.150537+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:09.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.150537+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: cluster 2024-04-26T16:28:09.181555+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e421: 8 total, 7 up, 7 in 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: cluster 2024-04-26T16:28:09.181555+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e421: 8 total, 7 up, 7 in 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.200668+0000 mon.a (mon.0) 2626 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.200668+0000 mon.a (mon.0) 2626 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.202759+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[29234]: audit 2024-04-26T16:28:09.202759+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:08.469305+0000 mon.c (mon.2) 607 : audit [DBG] from='client.? 172.21.15.50:0/2738094435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:08.469305+0000 mon.c (mon.2) 607 : audit [DBG] from='client.? 172.21.15.50:0/2738094435' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:08.912955+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:08.912955+0000 mon.a (mon.0) 2621 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: cluster 2024-04-26T16:28:09.148030+0000 mon.a (mon.0) 2622 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: cluster 2024-04-26T16:28:09.148030+0000 mon.a (mon.0) 2622 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.150386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.150386+0000 mon.a (mon.0) 2623 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.150537+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.150537+0000 mon.a (mon.0) 2624 : audit [INF] from='client.? 172.21.15.50:0/3205801944' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: cluster 2024-04-26T16:28:09.181555+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e421: 8 total, 7 up, 7 in 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: cluster 2024-04-26T16:28:09.181555+0000 mon.a (mon.0) 2625 : cluster [DBG] osdmap e421: 8 total, 7 up, 7 in 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.200668+0000 mon.a (mon.0) 2626 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.200668+0000 mon.a (mon.0) 2626 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.202759+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:09 smithi050 bash[21463]: audit 2024-04-26T16:28:09.202759+0000 mon.b (mon.1) 287 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:28:09.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:09.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:09.640 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:09.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:09.771 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:09.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:09.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:09.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:10.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:10.032 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:10.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:28:10.118 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:28:10.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:10.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:10.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: cluster 2024-04-26T16:28:09.635513+0000 mgr.y (mgr.24401) 423 : cluster [DBG] pgmap v616: 228 pgs: 32 unknown, 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: cluster 2024-04-26T16:28:09.635513+0000 mgr.y (mgr.24401) 423 : cluster [DBG] pgmap v616: 228 pgs: 32 unknown, 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.153527+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.153527+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.153797+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.153797+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: cluster 2024-04-26T16:28:10.162803+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e422: 8 total, 7 up, 7 in 2024-04-26T16:28:10.277 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: cluster 2024-04-26T16:28:10.162803+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e422: 8 total, 7 up, 7 in 2024-04-26T16:28:10.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.168999+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.168999+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.170582+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.278 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:10 smithi190 bash[23868]: audit 2024-04-26T16:28:10.170582+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/exp.2.11.6 2024-04-26T16:28:10.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:10.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:10.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:10.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:10.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: cluster 2024-04-26T16:28:09.635513+0000 mgr.y (mgr.24401) 423 : cluster [DBG] pgmap v616: 228 pgs: 32 unknown, 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:10.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: cluster 2024-04-26T16:28:09.635513+0000 mgr.y (mgr.24401) 423 : cluster [DBG] pgmap v616: 228 pgs: 32 unknown, 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.153527+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.153527+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.153797+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.153797+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: cluster 2024-04-26T16:28:10.162803+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e422: 8 total, 7 up, 7 in 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: cluster 2024-04-26T16:28:10.162803+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e422: 8 total, 7 up, 7 in 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.168999+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.168999+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.170582+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[29234]: audit 2024-04-26T16:28:10.170582+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: cluster 2024-04-26T16:28:09.635513+0000 mgr.y (mgr.24401) 423 : cluster [DBG] pgmap v616: 228 pgs: 32 unknown, 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: cluster 2024-04-26T16:28:09.635513+0000 mgr.y (mgr.24401) 423 : cluster [DBG] pgmap v616: 228 pgs: 32 unknown, 46 active+undersized, 33 active+undersized+degraded, 117 active+clean; 8.4 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 141/1032 objects degraded (13.663%) 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.153527+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.153527+0000 mon.a (mon.0) 2627 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi050-73201-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.153797+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.153797+0000 mon.a (mon.0) 2628 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: cluster 2024-04-26T16:28:10.162803+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e422: 8 total, 7 up, 7 in 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: cluster 2024-04-26T16:28:10.162803+0000 mon.a (mon.0) 2629 : cluster [DBG] osdmap e422: 8 total, 7 up, 7 in 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.168999+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.168999+0000 mon.a (mon.0) 2630 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.170582+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:10 smithi050 bash[21463]: audit 2024-04-26T16:28:10.170582+0000 mon.b (mon.1) 288 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:10.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:10.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:10.694 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:10.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:10.826 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:10.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:10.958 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:10.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:11.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:11.090 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:11.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:11.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:11.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:11.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:11.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:11.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: cluster 2024-04-26T16:28:11.140532+0000 mon.a (mon.0) 2631 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: cluster 2024-04-26T16:28:11.140532+0000 mon.a (mon.0) 2631 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: audit 2024-04-26T16:28:11.156193+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: audit 2024-04-26T16:28:11.156193+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: audit 2024-04-26T16:28:11.164812+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: audit 2024-04-26T16:28:11.164812+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: cluster 2024-04-26T16:28:11.171001+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e423: 8 total, 7 up, 7 in 2024-04-26T16:28:11.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: cluster 2024-04-26T16:28:11.171001+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e423: 8 total, 7 up, 7 in 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: audit 2024-04-26T16:28:11.173695+0000 mon.a (mon.0) 2634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[29234]: audit 2024-04-26T16:28:11.173695+0000 mon.a (mon.0) 2634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: cluster 2024-04-26T16:28:11.140532+0000 mon.a (mon.0) 2631 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: cluster 2024-04-26T16:28:11.140532+0000 mon.a (mon.0) 2631 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: audit 2024-04-26T16:28:11.156193+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: audit 2024-04-26T16:28:11.156193+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: audit 2024-04-26T16:28:11.164812+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: audit 2024-04-26T16:28:11.164812+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: cluster 2024-04-26T16:28:11.171001+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e423: 8 total, 7 up, 7 in 2024-04-26T16:28:11.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: cluster 2024-04-26T16:28:11.171001+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e423: 8 total, 7 up, 7 in 2024-04-26T16:28:11.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: audit 2024-04-26T16:28:11.173695+0000 mon.a (mon.0) 2634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:11 smithi050 bash[21463]: audit 2024-04-26T16:28:11.173695+0000 mon.a (mon.0) 2634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: cluster 2024-04-26T16:28:11.140532+0000 mon.a (mon.0) 2631 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:11.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: cluster 2024-04-26T16:28:11.140532+0000 mon.a (mon.0) 2631 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: audit 2024-04-26T16:28:11.156193+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: audit 2024-04-26T16:28:11.156193+0000 mon.a (mon.0) 2632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: audit 2024-04-26T16:28:11.164812+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: audit 2024-04-26T16:28:11.164812+0000 mon.b (mon.1) 289 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: cluster 2024-04-26T16:28:11.171001+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e423: 8 total, 7 up, 7 in 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: cluster 2024-04-26T16:28:11.171001+0000 mon.a (mon.0) 2633 : cluster [DBG] osdmap e423: 8 total, 7 up, 7 in 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: audit 2024-04-26T16:28:11.173695+0000 mon.a (mon.0) 2634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:11 smithi190 bash[23868]: audit 2024-04-26T16:28:11.173695+0000 mon.a (mon.0) 2634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]: dispatch 2024-04-26T16:28:11.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:11.610 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:11.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:11.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:11.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:11.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:11.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:11.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:11.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:11.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:12.002 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:12.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:12.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:12.120 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:28:12.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:12.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:12.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:12.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:12.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: cluster 2024-04-26T16:28:11.637974+0000 mgr.y (mgr.24401) 424 : cluster [DBG] pgmap v619: 236 pgs: 61 active+undersized, 34 active+undersized+degraded, 141 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 143/1047 objects degraded (13.658%) 2024-04-26T16:28:12.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: cluster 2024-04-26T16:28:11.637974+0000 mgr.y (mgr.24401) 424 : cluster [DBG] pgmap v619: 236 pgs: 61 active+undersized, 34 active+undersized+degraded, 141 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 143/1047 objects degraded (13.658%) 2024-04-26T16:28:12.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:11.724575+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:12.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:11.724575+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:12.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:11.727877+0000 mon.c (mon.2) 608 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:11.727877+0000 mon.c (mon.2) 608 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.156609+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.156609+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.158617+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]': finished 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.158617+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]': finished 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.163824+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e424: 8 total, 7 up, 7 in 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.163824+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e424: 8 total, 7 up, 7 in 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.167334+0000 mon.c (mon.2) 609 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.167334+0000 mon.c (mon.2) 609 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.168196+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.168196+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.240755+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.240755+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.242764+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[21463]: audit 2024-04-26T16:28:12.242764+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: cluster 2024-04-26T16:28:11.637974+0000 mgr.y (mgr.24401) 424 : cluster [DBG] pgmap v619: 236 pgs: 61 active+undersized, 34 active+undersized+degraded, 141 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 143/1047 objects degraded (13.658%) 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: cluster 2024-04-26T16:28:11.637974+0000 mgr.y (mgr.24401) 424 : cluster [DBG] pgmap v619: 236 pgs: 61 active+undersized, 34 active+undersized+degraded, 141 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 143/1047 objects degraded (13.658%) 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:11.724575+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:11.724575+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:11.727877+0000 mon.c (mon.2) 608 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:11.727877+0000 mon.c (mon.2) 608 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.156609+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.156609+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.158617+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]': finished 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.158617+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]': finished 2024-04-26T16:28:12.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.163824+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e424: 8 total, 7 up, 7 in 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.163824+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e424: 8 total, 7 up, 7 in 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.167334+0000 mon.c (mon.2) 609 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.167334+0000 mon.c (mon.2) 609 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.168196+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.168196+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.240755+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.240755+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.242764+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:12 smithi050 bash[29234]: audit 2024-04-26T16:28:12.242764+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:12.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: cluster 2024-04-26T16:28:11.637974+0000 mgr.y (mgr.24401) 424 : cluster [DBG] pgmap v619: 236 pgs: 61 active+undersized, 34 active+undersized+degraded, 141 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 143/1047 objects degraded (13.658%) 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: cluster 2024-04-26T16:28:11.637974+0000 mgr.y (mgr.24401) 424 : cluster [DBG] pgmap v619: 236 pgs: 61 active+undersized, 34 active+undersized+degraded, 141 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 143/1047 objects degraded (13.658%) 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:11.724575+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:11.724575+0000 mon.a (mon.0) 2635 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:11.727877+0000 mon.c (mon.2) 608 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:11.727877+0000 mon.c (mon.2) 608 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.156609+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.156609+0000 mon.a (mon.0) 2636 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.158617+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]': finished 2024-04-26T16:28:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.158617+0000 mon.a (mon.0) 2637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-51", "mode": "writeback"}]': finished 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.163824+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e424: 8 total, 7 up, 7 in 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.163824+0000 mon.a (mon.0) 2638 : cluster [DBG] osdmap e424: 8 total, 7 up, 7 in 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.167334+0000 mon.c (mon.2) 609 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.167334+0000 mon.c (mon.2) 609 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.168196+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.168196+0000 mon.a (mon.0) 2639 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.240755+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.240755+0000 mon.a (mon.0) 2640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.242764+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:12 smithi190 bash[23868]: audit 2024-04-26T16:28:12.242764+0000 mon.b (mon.1) 290 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:28:12.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:12.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:12.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:12.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:12.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:28:12.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:12.913 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:12.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:12.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:13.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:13.035 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:13.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:13.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:13.272 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:13.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.244638+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: Degraded data redundancy: 143/1047 objects degraded (13.658%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:13.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.244638+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: Degraded data redundancy: 143/1047 objects degraded (13.658%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:13.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.815498+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:13.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.815498+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:13.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.815734+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:12.815734+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.161072+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.161072+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.161361+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.161361+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:13.167511+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: cluster 2024-04-26T16:28:13.167511+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.168703+0000 mon.c (mon.2) 610 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.168703+0000 mon.c (mon.2) 610 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.169550+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.169550+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.170915+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.286 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.170915+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.244638+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: Degraded data redundancy: 143/1047 objects degraded (13.658%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.244638+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: Degraded data redundancy: 143/1047 objects degraded (13.658%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.815498+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.815498+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.815734+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:12.815734+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.161072+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.161072+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.161361+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.161361+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:13.167511+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-04-26T16:28:13.287 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: cluster 2024-04-26T16:28:13.167511+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.168703+0000 mon.c (mon.2) 610 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.168703+0000 mon.c (mon.2) 610 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.169550+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.169550+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.170915+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.170915+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.172731+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[21463]: audit 2024-04-26T16:28:13.172731+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.172731+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:13 smithi050 bash[29234]: audit 2024-04-26T16:28:13.172731+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.300 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:13.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:13.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:13.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.244638+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: Degraded data redundancy: 143/1047 objects degraded (13.658%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.244638+0000 mon.a (mon.0) 2641 : cluster [WRN] Health check update: Degraded data redundancy: 143/1047 objects degraded (13.658%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.815498+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.815498+0000 osd.3 (osd.3) 259 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.815734+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:12.815734+0000 osd.3 (osd.3) 260 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.161072+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.161072+0000 mon.a (mon.0) 2642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.161361+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.161361+0000 mon.a (mon.0) 2643 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:13.167511+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-04-26T16:28:13.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: cluster 2024-04-26T16:28:13.167511+0000 mon.a (mon.0) 2644 : cluster [DBG] osdmap e425: 8 total, 7 up, 7 in 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.168703+0000 mon.c (mon.2) 610 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.168703+0000 mon.c (mon.2) 610 : audit [INF] from='client.? 172.21.15.50:0/3265644725' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.169550+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.169550+0000 mon.a (mon.0) 2645 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.170915+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.170915+0000 mon.a (mon.0) 2646 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.172731+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:13 smithi190 bash[23868]: audit 2024-04-26T16:28:13.172731+0000 mon.b (mon.1) 291 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:28:13.561 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:13.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:13.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:13.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:13.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:13.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:13.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:13.957 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:13.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:14.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:14.087 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:14.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:14.180 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3017 ms) 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6996 ms) 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6991 ms) 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7045 ms) 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6999 ms) 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6980 ms) 2024-04-26T16:28:14.181 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-04-26T16:28:14.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7012 ms) 2024-04-26T16:28:14.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139079 ms total) 2024-04-26T16:28:14.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: 2024-04-26T16:28:14.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-04-26T16:28:14.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (286226 ms total) 2024-04-26T16:28:14.182 INFO:tasks.workunit.client.0.smithi050.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-04-26T16:28:14.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:14.218 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:14.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:14.349 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:14.372 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:14.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:14.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: cluster 2024-04-26T16:28:13.639264+0000 mgr.y (mgr.24401) 425 : cluster [DBG] pgmap v622: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:14.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: cluster 2024-04-26T16:28:13.639264+0000 mgr.y (mgr.24401) 425 : cluster [DBG] pgmap v622: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.164528+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.164528+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.164755+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.164755+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: cluster 2024-04-26T16:28:14.172088+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: cluster 2024-04-26T16:28:14.172088+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.174827+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.174827+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.175025+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:14 smithi190 bash[23868]: audit 2024-04-26T16:28:14.175025+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.480 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:14.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:14.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:14.595 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:28:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: cluster 2024-04-26T16:28:13.639264+0000 mgr.y (mgr.24401) 425 : cluster [DBG] pgmap v622: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: cluster 2024-04-26T16:28:13.639264+0000 mgr.y (mgr.24401) 425 : cluster [DBG] pgmap v622: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.164528+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.164528+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.164755+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.164755+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: cluster 2024-04-26T16:28:14.172088+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: cluster 2024-04-26T16:28:14.172088+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.174827+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.174827+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.175025+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[29234]: audit 2024-04-26T16:28:14.175025+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: cluster 2024-04-26T16:28:13.639264+0000 mgr.y (mgr.24401) 425 : cluster [DBG] pgmap v622: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: cluster 2024-04-26T16:28:13.639264+0000 mgr.y (mgr.24401) 425 : cluster [DBG] pgmap v622: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.164528+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.164528+0000 mon.a (mon.0) 2647 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi050-73201-66"}]': finished 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.164755+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.164755+0000 mon.a (mon.0) 2648 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: cluster 2024-04-26T16:28:14.172088+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: cluster 2024-04-26T16:28:14.172088+0000 mon.a (mon.0) 2649 : cluster [DBG] osdmap e426: 8 total, 7 up, 7 in 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.174827+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.174827+0000 mon.b (mon.1) 292 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.175025+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.597 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:14 smithi050 bash[21463]: audit 2024-04-26T16:28:14.175025+0000 mon.a (mon.0) 2650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:28:14.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:14.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:14.745 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:14.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:14.878 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:14.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:14.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:14.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.007 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:15.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:15.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:15.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.274 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:15.293 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-04-26T16:28:15.293 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-04-26T16:28:15.293 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-04-26T16:28:15.294 DEBUG:teuthology.orchestra.run.smithi190:osd.6> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T16:28:15.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl start ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6 2024-04-26T16:28:15.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-26T16:28:15.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: cluster 2024-04-26T16:28:15.164841+0000 mon.a (mon.0) 2651 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:15.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: cluster 2024-04-26T16:28:15.164841+0000 mon.a (mon.0) 2651 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:15.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: audit 2024-04-26T16:28:15.166660+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:28:15.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: audit 2024-04-26T16:28:15.166660+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:28:15.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: cluster 2024-04-26T16:28:15.171732+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: cluster 2024-04-26T16:28:15.171732+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: audit 2024-04-26T16:28:15.176493+0000 mon.a (mon.0) 2654 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: audit 2024-04-26T16:28:15.176493+0000 mon.a (mon.0) 2654 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: audit 2024-04-26T16:28:15.177882+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[21463]: audit 2024-04-26T16:28:15.177882+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: cluster 2024-04-26T16:28:15.164841+0000 mon.a (mon.0) 2651 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: cluster 2024-04-26T16:28:15.164841+0000 mon.a (mon.0) 2651 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: audit 2024-04-26T16:28:15.166660+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:28:15.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: audit 2024-04-26T16:28:15.166660+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:28:15.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: cluster 2024-04-26T16:28:15.171732+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-04-26T16:28:15.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: cluster 2024-04-26T16:28:15.171732+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-04-26T16:28:15.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: audit 2024-04-26T16:28:15.176493+0000 mon.a (mon.0) 2654 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: audit 2024-04-26T16:28:15.176493+0000 mon.a (mon.0) 2654 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: audit 2024-04-26T16:28:15.177882+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:15 smithi050 bash[29234]: audit 2024-04-26T16:28:15.177882+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:15.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:15.516 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:15 smithi190 systemd[1]: Started Ceph osd.6 for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: cluster 2024-04-26T16:28:15.164841+0000 mon.a (mon.0) 2651 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: cluster 2024-04-26T16:28:15.164841+0000 mon.a (mon.0) 2651 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: audit 2024-04-26T16:28:15.166660+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: audit 2024-04-26T16:28:15.166660+0000 mon.a (mon.0) 2652 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: cluster 2024-04-26T16:28:15.171732+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: cluster 2024-04-26T16:28:15.171732+0000 mon.a (mon.0) 2653 : cluster [DBG] osdmap e427: 8 total, 7 up, 7 in 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: audit 2024-04-26T16:28:15.176493+0000 mon.a (mon.0) 2654 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: audit 2024-04-26T16:28:15.176493+0000 mon.a (mon.0) 2654 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: audit 2024-04-26T16:28:15.177882+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:15 smithi190 bash[23868]: audit 2024-04-26T16:28:15.177882+0000 mon.b (mon.1) 293 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:28:15.532 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:15.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:15.655 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:15.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:15.785 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:15.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:15.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:16.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:16.046 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:16.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:16.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:16.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: cluster 2024-04-26T16:28:15.640441+0000 mgr.y (mgr.24401) 426 : cluster [DBG] pgmap v625: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: cluster 2024-04-26T16:28:15.640441+0000 mgr.y (mgr.24401) 426 : cluster [DBG] pgmap v625: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: audit 2024-04-26T16:28:16.145598+0000 mon.a (mon.0) 2655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: audit 2024-04-26T16:28:16.145598+0000 mon.a (mon.0) 2655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: cluster 2024-04-26T16:28:16.150154+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: cluster 2024-04-26T16:28:16.150154+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-04-26T16:28:16.267 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: audit 2024-04-26T16:28:16.156831+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: cluster 2024-04-26T16:28:15.640441+0000 mgr.y (mgr.24401) 426 : cluster [DBG] pgmap v625: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: cluster 2024-04-26T16:28:15.640441+0000 mgr.y (mgr.24401) 426 : cluster [DBG] pgmap v625: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: audit 2024-04-26T16:28:16.145598+0000 mon.a (mon.0) 2655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: audit 2024-04-26T16:28:16.145598+0000 mon.a (mon.0) 2655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: cluster 2024-04-26T16:28:16.150154+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: cluster 2024-04-26T16:28:16.150154+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: audit 2024-04-26T16:28:16.156831+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: audit 2024-04-26T16:28:16.156831+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.291 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: audit 2024-04-26T16:28:16.161382+0000 mon.a (mon.0) 2657 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.292 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[23868]: audit 2024-04-26T16:28:16.161382+0000 mon.a (mon.0) 2657 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.306 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:16.406 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:16.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:16.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: audit 2024-04-26T16:28:16.156831+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: audit 2024-04-26T16:28:16.161382+0000 mon.a (mon.0) 2657 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[21463]: audit 2024-04-26T16:28:16.161382+0000 mon.a (mon.0) 2657 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: cluster 2024-04-26T16:28:15.640441+0000 mgr.y (mgr.24401) 426 : cluster [DBG] pgmap v625: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:16.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: cluster 2024-04-26T16:28:15.640441+0000 mgr.y (mgr.24401) 426 : cluster [DBG] pgmap v625: 228 pgs: 58 active+undersized, 33 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 142/1044 objects degraded (13.602%) 2024-04-26T16:28:16.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: audit 2024-04-26T16:28:16.145598+0000 mon.a (mon.0) 2655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: audit 2024-04-26T16:28:16.145598+0000 mon.a (mon.0) 2655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: cluster 2024-04-26T16:28:16.150154+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: cluster 2024-04-26T16:28:16.150154+0000 mon.a (mon.0) 2656 : cluster [DBG] osdmap e428: 8 total, 7 up, 7 in 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: audit 2024-04-26T16:28:16.156831+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: audit 2024-04-26T16:28:16.156831+0000 mon.b (mon.1) 294 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: audit 2024-04-26T16:28:16.161382+0000 mon.a (mon.0) 2657 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:16 smithi050 bash[29234]: audit 2024-04-26T16:28:16.161382+0000 mon.a (mon.0) 2657 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:28:16.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:16.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:16.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:16.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:16.694 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:16.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:16.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:16.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:16.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:16.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:17.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:17.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:28:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:28:17.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:17.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:17.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:17.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:17.330 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:17.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:17.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:17.571 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T16:28:17.572 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:17 smithi190 bash[71032]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-26T16:28:17.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:17.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:17.701 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:17.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:17.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:17.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:17.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:17.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:18.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:18.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:18.184 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:18.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: audit 2024-04-26T16:28:16.859180+0000 mgr.y (mgr.24401) 427 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: audit 2024-04-26T16:28:16.859180+0000 mgr.y (mgr.24401) 427 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: audit 2024-04-26T16:28:17.156769+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: audit 2024-04-26T16:28:17.156769+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: cluster 2024-04-26T16:28:17.164143+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: cluster 2024-04-26T16:28:17.164143+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: cluster 2024-04-26T16:28:17.641232+0000 mgr.y (mgr.24401) 428 : cluster [DBG] pgmap v628: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:18.230 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[23868]: cluster 2024-04-26T16:28:17.641232+0000 mgr.y (mgr.24401) 428 : cluster [DBG] pgmap v628: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.247 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:18.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:18.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: audit 2024-04-26T16:28:16.859180+0000 mgr.y (mgr.24401) 427 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: audit 2024-04-26T16:28:16.859180+0000 mgr.y (mgr.24401) 427 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: audit 2024-04-26T16:28:17.156769+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: audit 2024-04-26T16:28:17.156769+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: cluster 2024-04-26T16:28:17.164143+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: cluster 2024-04-26T16:28:17.164143+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: cluster 2024-04-26T16:28:17.641232+0000 mgr.y (mgr.24401) 428 : cluster [DBG] pgmap v628: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[21463]: cluster 2024-04-26T16:28:17.641232+0000 mgr.y (mgr.24401) 428 : cluster [DBG] pgmap v628: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: audit 2024-04-26T16:28:16.859180+0000 mgr.y (mgr.24401) 427 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:18.360 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: audit 2024-04-26T16:28:16.859180+0000 mgr.y (mgr.24401) 427 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:18.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: audit 2024-04-26T16:28:17.156769+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:28:18.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: audit 2024-04-26T16:28:17.156769+0000 mon.a (mon.0) 2658 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-51","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:28:18.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: cluster 2024-04-26T16:28:17.164143+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-04-26T16:28:18.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: cluster 2024-04-26T16:28:17.164143+0000 mon.a (mon.0) 2659 : cluster [DBG] osdmap e429: 8 total, 7 up, 7 in 2024-04-26T16:28:18.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: cluster 2024-04-26T16:28:17.641232+0000 mgr.y (mgr.24401) 428 : cluster [DBG] pgmap v628: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:18.361 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:18 smithi050 bash[29234]: cluster 2024-04-26T16:28:17.641232+0000 mgr.y (mgr.24401) 428 : cluster [DBG] pgmap v628: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:18.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:18.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:18.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:18.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:18.625 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:18.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:18.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:18.821 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:18 smithi190 bash[71351]: debug 2024-04-26T16:28:18.596+0000 7f9e33f11700 1 -- 172.21.15.190:0/1748181577 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f981f9d680 con 0x55f981ffac00 2024-04-26T16:28:18.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:18.887 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:18.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:18.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:19.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:19.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:19.148 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:19.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:19.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:19.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:19.393 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:19 smithi190 bash[71351]: debug 2024-04-26T16:28:19.132+0000 7f9e3d0a97c0 -1 Falling back to public interface 2024-04-26T16:28:19.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:19 smithi190 bash[23868]: cluster 2024-04-26T16:28:18.156763+0000 mon.a (mon.0) 2660 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:28:19.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:19 smithi190 bash[23868]: cluster 2024-04-26T16:28:18.156763+0000 mon.a (mon.0) 2660 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:28:19.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:19 smithi190 bash[23868]: cluster 2024-04-26T16:28:18.156821+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: Degraded data redundancy: 144/1056 objects degraded (13.636%), 35 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:19.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:19 smithi190 bash[23868]: cluster 2024-04-26T16:28:18.156821+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: Degraded data redundancy: 144/1056 objects degraded (13.636%), 35 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:19.410 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:19.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:19.544 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:19.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:19.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[21463]: cluster 2024-04-26T16:28:18.156763+0000 mon.a (mon.0) 2660 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:28:19.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[21463]: cluster 2024-04-26T16:28:18.156763+0000 mon.a (mon.0) 2660 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:28:19.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[21463]: cluster 2024-04-26T16:28:18.156821+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: Degraded data redundancy: 144/1056 objects degraded (13.636%), 35 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:19.659 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[21463]: cluster 2024-04-26T16:28:18.156821+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: Degraded data redundancy: 144/1056 objects degraded (13.636%), 35 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:19.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[29234]: cluster 2024-04-26T16:28:18.156763+0000 mon.a (mon.0) 2660 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:28:19.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[29234]: cluster 2024-04-26T16:28:18.156763+0000 mon.a (mon.0) 2660 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:28:19.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[29234]: cluster 2024-04-26T16:28:18.156821+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: Degraded data redundancy: 144/1056 objects degraded (13.636%), 35 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:19.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:19 smithi050 bash[29234]: cluster 2024-04-26T16:28:18.156821+0000 mon.a (mon.0) 2661 : cluster [WRN] Health check update: Degraded data redundancy: 144/1056 objects degraded (13.636%), 35 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:19.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:19.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:19.804 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:19.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:19.936 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:19.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:19.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:20.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:20.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:20.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:20.198 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:20.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:20.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:20.429 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:20 smithi050 bash[29234]: cluster 2024-04-26T16:28:19.642454+0000 mgr.y (mgr.24401) 429 : cluster [DBG] pgmap v629: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:20.429 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:20 smithi050 bash[29234]: cluster 2024-04-26T16:28:19.642454+0000 mgr.y (mgr.24401) 429 : cluster [DBG] pgmap v629: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:20.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:20 smithi050 bash[21463]: cluster 2024-04-26T16:28:19.642454+0000 mgr.y (mgr.24401) 429 : cluster [DBG] pgmap v629: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:20.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:20 smithi050 bash[21463]: cluster 2024-04-26T16:28:19.642454+0000 mgr.y (mgr.24401) 429 : cluster [DBG] pgmap v629: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:20.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:20.434 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:28:20.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:20 smithi190 bash[23868]: cluster 2024-04-26T16:28:19.642454+0000 mgr.y (mgr.24401) 429 : cluster [DBG] pgmap v629: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:20.454 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:20 smithi190 bash[23868]: cluster 2024-04-26T16:28:19.642454+0000 mgr.y (mgr.24401) 429 : cluster [DBG] pgmap v629: 228 pgs: 56 active+undersized, 35 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s; 144/1056 objects degraded (13.636%) 2024-04-26T16:28:20.455 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:20.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:20.586 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:20.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:20.714 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:20.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:20.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:20.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:20.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:20.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:21.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:21.101 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:21.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:21.226 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:21.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:21.354 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:21.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:21.485 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:21.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:21.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:21.746 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:21.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:21.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:21.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:21.965 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:28:21.965 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:28:21.965 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-26T16:28:21.965 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-26T16:28:21.965 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:28:21.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:22.000 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:22.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:22.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:22.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:22.241 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:22 smithi190 bash[23868]: cluster 2024-04-26T16:28:21.156635+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-04-26T16:28:22.241 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:22 smithi190 bash[23868]: cluster 2024-04-26T16:28:21.156635+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-04-26T16:28:22.241 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:22 smithi190 bash[23868]: cluster 2024-04-26T16:28:21.643912+0000 mgr.y (mgr.24401) 430 : cluster [DBG] pgmap v631: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:22.241 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:22 smithi190 bash[23868]: cluster 2024-04-26T16:28:21.643912+0000 mgr.y (mgr.24401) 430 : cluster [DBG] pgmap v631: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:22.254 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:22.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[29234]: cluster 2024-04-26T16:28:21.156635+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[29234]: cluster 2024-04-26T16:28:21.156635+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[29234]: cluster 2024-04-26T16:28:21.643912+0000 mgr.y (mgr.24401) 430 : cluster [DBG] pgmap v631: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[29234]: cluster 2024-04-26T16:28:21.643912+0000 mgr.y (mgr.24401) 430 : cluster [DBG] pgmap v631: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[21463]: cluster 2024-04-26T16:28:21.156635+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[21463]: cluster 2024-04-26T16:28:21.156635+0000 mon.a (mon.0) 2662 : cluster [DBG] osdmap e430: 8 total, 7 up, 7 in 2024-04-26T16:28:22.369 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[21463]: cluster 2024-04-26T16:28:21.643912+0000 mgr.y (mgr.24401) 430 : cluster [DBG] pgmap v631: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:22.370 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:22 smithi050 bash[21463]: cluster 2024-04-26T16:28:21.643912+0000 mgr.y (mgr.24401) 430 : cluster [DBG] pgmap v631: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:22.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:22.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:22.501 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:22 smithi190 bash[71351]: debug 2024-04-26T16:28:22.259+0000 7f9e3d0a97c0 -1 osd.6 398 log_to_monitors true 2024-04-26T16:28:22.517 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:22.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:22.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:22.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:22.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:22.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:23.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:23.042 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:23.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:23.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:23.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[21463]: audit 2024-04-26T16:28:22.267441+0000 mon.a (mon.0) 2663 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[29234]: audit 2024-04-26T16:28:22.267441+0000 mon.a (mon.0) 2663 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[29234]: audit 2024-04-26T16:28:22.267441+0000 mon.a (mon.0) 2663 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[29234]: audit 2024-04-26T16:28:22.269050+0000 mon.b (mon.1) 295 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.224 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[29234]: audit 2024-04-26T16:28:22.269050+0000 mon.b (mon.1) 295 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:23.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:23 smithi190 bash[23868]: audit 2024-04-26T16:28:22.267441+0000 mon.a (mon.0) 2663 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:23 smithi190 bash[23868]: audit 2024-04-26T16:28:22.267441+0000 mon.a (mon.0) 2663 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:23 smithi190 bash[23868]: audit 2024-04-26T16:28:22.269050+0000 mon.b (mon.1) 295 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:23 smithi190 bash[23868]: audit 2024-04-26T16:28:22.269050+0000 mon.b (mon.1) 295 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.305 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:23.383 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:28:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:28:23.383 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:28:23.383 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:28:23.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:23.425 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:28:23 smithi190 bash[71351]: debug 2024-04-26T16:28:23.343+0000 7f9e2e501700 -1 osd.6 431 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T16:28:23.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:23.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[21463]: audit 2024-04-26T16:28:22.267441+0000 mon.a (mon.0) 2663 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[21463]: audit 2024-04-26T16:28:22.269050+0000 mon.b (mon.1) 295 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:23 smithi050 bash[21463]: audit 2024-04-26T16:28:22.269050+0000 mon.b (mon.1) 295 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:28:23.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:23.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:23.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:23.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:23.831 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:23.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:23.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:23.962 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:23.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:24.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:24.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:24.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:24.210 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: audit 2024-04-26T16:28:23.165969+0000 mon.a (mon.0) 2664 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:28:24.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: audit 2024-04-26T16:28:23.165969+0000 mon.a (mon.0) 2664 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:28:24.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.173322+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-04-26T16:28:24.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.173322+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-04-26T16:28:24.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: audit 2024-04-26T16:28:23.173847+0000 mon.b (mon.1) 296 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: audit 2024-04-26T16:28:23.173847+0000 mon.b (mon.1) 296 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: audit 2024-04-26T16:28:23.174207+0000 mon.a (mon.0) 2666 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.212 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: audit 2024-04-26T16:28:23.174207+0000 mon.a (mon.0) 2666 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.212 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.644958+0000 mgr.y (mgr.24401) 431 : cluster [DBG] pgmap v633: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 789 B/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:24.212 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:24 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.644958+0000 mgr.y (mgr.24401) 431 : cluster [DBG] pgmap v633: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 789 B/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:24.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-26T16:28:24.233 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:24.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:24.359 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: audit 2024-04-26T16:28:23.165969+0000 mon.a (mon.0) 2664 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:28:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: audit 2024-04-26T16:28:23.165969+0000 mon.a (mon.0) 2664 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:28:24.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.173322+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.173322+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: audit 2024-04-26T16:28:23.173847+0000 mon.b (mon.1) 296 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: audit 2024-04-26T16:28:23.173847+0000 mon.b (mon.1) 296 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: audit 2024-04-26T16:28:23.174207+0000 mon.a (mon.0) 2666 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: audit 2024-04-26T16:28:23.174207+0000 mon.a (mon.0) 2666 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.644958+0000 mgr.y (mgr.24401) 431 : cluster [DBG] pgmap v633: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 789 B/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.644958+0000 mgr.y (mgr.24401) 431 : cluster [DBG] pgmap v633: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 789 B/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: audit 2024-04-26T16:28:23.165969+0000 mon.a (mon.0) 2664 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: audit 2024-04-26T16:28:23.165969+0000 mon.a (mon.0) 2664 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.173322+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.173322+0000 mon.a (mon.0) 2665 : cluster [DBG] osdmap e431: 8 total, 7 up, 7 in 2024-04-26T16:28:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: audit 2024-04-26T16:28:23.173847+0000 mon.b (mon.1) 296 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: audit 2024-04-26T16:28:23.173847+0000 mon.b (mon.1) 296 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: audit 2024-04-26T16:28:23.174207+0000 mon.a (mon.0) 2666 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: audit 2024-04-26T16:28:23.174207+0000 mon.a (mon.0) 2666 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:28:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.644958+0000 mgr.y (mgr.24401) 431 : cluster [DBG] pgmap v633: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 789 B/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.644958+0000 mgr.y (mgr.24401) 431 : cluster [DBG] pgmap v633: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 789 B/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:24.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:24.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:24.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:24.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:24.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:24.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:28:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:28:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:28:24.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:24.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:24.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:24.878 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:24.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:24.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:25.010 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:25.113 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:25.141 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:25.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:25.272 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:25.375 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: cluster 2024-04-26T16:28:24.177249+0000 mon.a (mon.0) 2667 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: cluster 2024-04-26T16:28:24.177249+0000 mon.a (mon.0) 2667 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: cluster 2024-04-26T16:28:24.189922+0000 mon.a (mon.0) 2668 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] boot 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: cluster 2024-04-26T16:28:24.189922+0000 mon.a (mon.0) 2668 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] boot 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: cluster 2024-04-26T16:28:24.189976+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: cluster 2024-04-26T16:28:24.189976+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: audit 2024-04-26T16:28:24.198415+0000 mon.c (mon.2) 611 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:28:25.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[29234]: audit 2024-04-26T16:28:24.198415+0000 mon.c (mon.2) 611 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: cluster 2024-04-26T16:28:24.177249+0000 mon.a (mon.0) 2667 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: cluster 2024-04-26T16:28:24.177249+0000 mon.a (mon.0) 2667 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: cluster 2024-04-26T16:28:24.189922+0000 mon.a (mon.0) 2668 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] boot 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: cluster 2024-04-26T16:28:24.189922+0000 mon.a (mon.0) 2668 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] boot 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: cluster 2024-04-26T16:28:24.189976+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: cluster 2024-04-26T16:28:24.189976+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: audit 2024-04-26T16:28:24.198415+0000 mon.c (mon.2) 611 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:28:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:25 smithi050 bash[21463]: audit 2024-04-26T16:28:24.198415+0000 mon.c (mon.2) 611 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:28:25.396 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:25.499 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:25.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:25.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: cluster 2024-04-26T16:28:24.177249+0000 mon.a (mon.0) 2667 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: cluster 2024-04-26T16:28:24.177249+0000 mon.a (mon.0) 2667 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: cluster 2024-04-26T16:28:24.189922+0000 mon.a (mon.0) 2668 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] boot 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: cluster 2024-04-26T16:28:24.189922+0000 mon.a (mon.0) 2668 : cluster [INF] osd.6 [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] boot 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: cluster 2024-04-26T16:28:24.189976+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: cluster 2024-04-26T16:28:24.189976+0000 mon.a (mon.0) 2669 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: audit 2024-04-26T16:28:24.198415+0000 mon.c (mon.2) 611 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:28:25.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:25 smithi190 bash[23868]: audit 2024-04-26T16:28:24.198415+0000 mon.c (mon.2) 611 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:28:25.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:25.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:25.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:25.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:25.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:25.921 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:25.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:26.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:26.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-26T16:28:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:26.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:26.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:26.181 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:26.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:26.309 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.289510+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.289510+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.290025+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:23.290025+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:25.199871+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:25.199871+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:25.646247+0000 mgr.y (mgr.24401) 432 : cluster [DBG] pgmap v636: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:25.646247+0000 mgr.y (mgr.24401) 432 : cluster [DBG] pgmap v636: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:26.148291+0000 mon.a (mon.0) 2671 : cluster [WRN] Health check update: Degraded data redundancy: 143/1050 objects degraded (13.619%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:26.148291+0000 mon.a (mon.0) 2671 : cluster [WRN] Health check update: Degraded data redundancy: 143/1050 objects degraded (13.619%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:26.153619+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[21463]: cluster 2024-04-26T16:28:26.153619+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.289510+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.289510+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.290025+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:23.290025+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:25.199871+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:25.199871+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:25.646247+0000 mgr.y (mgr.24401) 432 : cluster [DBG] pgmap v636: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:25.646247+0000 mgr.y (mgr.24401) 432 : cluster [DBG] pgmap v636: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:26.148291+0000 mon.a (mon.0) 2671 : cluster [WRN] Health check update: Degraded data redundancy: 143/1050 objects degraded (13.619%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:26.148291+0000 mon.a (mon.0) 2671 : cluster [WRN] Health check update: Degraded data redundancy: 143/1050 objects degraded (13.619%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:26.153619+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-04-26T16:28:26.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:26 smithi050 bash[29234]: cluster 2024-04-26T16:28:26.153619+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-04-26T16:28:26.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:26.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:26.544 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:26.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.289510+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.289510+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.290025+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:23.290025+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:25.199871+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:25.199871+0000 mon.a (mon.0) 2670 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:25.646247+0000 mgr.y (mgr.24401) 432 : cluster [DBG] pgmap v636: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:25.646247+0000 mgr.y (mgr.24401) 432 : cluster [DBG] pgmap v636: 228 pgs: 57 active+undersized, 34 active+undersized+degraded, 137 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s; 143/1050 objects degraded (13.619%) 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:26.148291+0000 mon.a (mon.0) 2671 : cluster [WRN] Health check update: Degraded data redundancy: 143/1050 objects degraded (13.619%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:26.148291+0000 mon.a (mon.0) 2671 : cluster [WRN] Health check update: Degraded data redundancy: 143/1050 objects degraded (13.619%), 34 pgs degraded (PG_DEGRADED) 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:26.153619+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-04-26T16:28:26.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[23868]: cluster 2024-04-26T16:28:26.153619+0000 mon.a (mon.0) 2672 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2024-04-26T16:28:26.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:26.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:26.702 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:26.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:26.831 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:26.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:26.934 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:26.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:26.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:27.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:27.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:28:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:28:27.091 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:27.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:27.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:27.221 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:27.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:27.351 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:27.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:27.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:27.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:27.607 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 -- 172.21.15.50:0/976129315 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18102320 msgr2=0x7fcc181027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 --2- 172.21.15.50:0/976129315 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18102320 0x7fcc181027a0 secure :-1 s=READY pgs=2656 cs=0 l=1 rev1=1 crypto rx=0x7fcc08009990 tx=0x7fcc08009ca0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 -- 172.21.15.50:0/976129315 shutdown_connections 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 --2- 172.21.15.50:0/976129315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc18108390 0x7fcc18108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 --2- 172.21.15.50:0/976129315 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 0x7fcc1810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 --2- 172.21.15.50:0/976129315 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18102320 0x7fcc181027a0 unknown :-1 s=CLOSED pgs=2656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 -- 172.21.15.50:0/976129315 >> 172.21.15.50:0/976129315 conn(0x7fcc180792d0 msgr2=0x7fcc180796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 -- 172.21.15.50:0/976129315 shutdown_connections 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 -- 172.21.15.50:0/976129315 wait complete. 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 librados: starting msgr at 2024-04-26T16:28:27.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 librados: starting objecter 2024-04-26T16:28:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 Processor -- start 2024-04-26T16:28:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 -- start start 2024-04-26T16:28:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 librados: setting wanted keys 2024-04-26T16:28:27.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.674+0000 7fcc205e2700 1 librados: calling monclient init 2024-04-26T16:28:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc18102320 0x7fcc18104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 0x7fcc18104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18108390 0x7fcc18104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcc181119a0 con 0x7fcc18102ce0 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcc18111820 con 0x7fcc18102320 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fcc18111b20 con 0x7fcc18108390 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1db7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 0x7fcc18104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18108390 0x7fcc18104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1db7d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 0x7fcc18104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34098/0 (socket says 172.21.15.50:34098) 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1db7d700 1 -- 172.21.15.50:0/591319990 learned_addr learned my addr 172.21.15.50:0/591319990 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc18102320 0x7fcc18104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc18102320 msgr2=0x7fcc18104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc18102320 0x7fcc18104000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 msgr2=0x7fcc18104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 0x7fcc18104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc181a7660 con 0x7fcc18108390 2024-04-26T16:28:27.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcc18102ce0 0x7fcc18104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18108390 0x7fcc18104a80 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400e970 tx=0x7fcc1400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 10 client.?.objecter ms_handle_connect 0x7fcc18108390 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1400cbd0 con 0x7fcc18108390 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fcc1400ee60 con 0x7fcc18108390 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc181a7950 con 0x7fcc18108390 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14004d90 con 0x7fcc18108390 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 10 client.41201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc181a7e90 con 0x7fcc18108390 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 librados: init done 2024-04-26T16:28:27.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fcc1806a110 con 0x7fcc18108390 2024-04-26T16:28:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fcc14005020 con 0x7fcc18108390 2024-04-26T16:28:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc04071f80 0x7fcc04074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(434..434 src has 87..434) ==== 9395+0+0 (secure 0 0 0) 0x7fcc14091b50 con 0x7fcc18108390 2024-04-26T16:28:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 osd_map(434..434 src has 87..434) 2024-04-26T16:28:27.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 3 client.41201.objecter handle_osd_map got epochs [434,434] > 0 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 3 client.41201.objecter handle_osd_map decoding full epoch 434 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc04071f80 0x7fcc04074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 20 client.41201.objecter dump_active .. 0 homeless 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.678+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v434) ==== 54+0+16 (secure 0 0 0) 0x7fcc14092080 con 0x7fcc18108390 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 10 client.41201.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 1 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcc18064030 0x7fcc18064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 1 2024-04-26T16:28:27.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 0 1 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 4 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 10 client.41201.objecter _send_command 1 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc181ab310 con 0x7fcc18064030 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc04071f80 0x7fcc04074440 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7fcc100060b0 tx=0x7fcc100085c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc04071f80 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcc18064030 0x7fcc18064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcc18064030 0x7fcc18064410 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc18064030 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181ab310 con 0x7fcc18064030 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 1: 0 ) 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 1 = system:0 2024-04-26T16:28:27.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 4 2024-04-26T16:28:27.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.682+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 0 1 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 10 client.41201.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 2 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 2 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc18105530 osd=0 3 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 0 2 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 4 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 10 client.41201.objecter _send_command 2 2024-04-26T16:28:27.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.698+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc18063ab0 con 0x7fcc18064030 2024-04-26T16:28:27.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc18063ab0 con 0x7fcc18064030 2024-04-26T16:28:27.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 4 2024-04-26T16:28:27.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 0 2 2024-04-26T16:28:27.699 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 10 client.41201.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcc1802d310 0x7fcc1802d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 1 3 2024-04-26T16:28:27.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 4 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 10 client.41201.objecter _send_command 3 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fcc1810bfe0 con 0x7fcc1802d310 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcc1802d310 0x7fcc1802d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc1802d310 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcc1802d310 0x7fcc1802d6f0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc1810bfe0 con 0x7fcc1802d310 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 3: 0 ) 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 3 = system:0 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 4 2024-04-26T16:28:27.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.702+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 1 3 2024-04-26T16:28:27.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 10 client.41201.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 4 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 4 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc1810b960 osd=1 3 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 1 4 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 4 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 10 client.41201.objecter _send_command 4 2024-04-26T16:28:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc1810b3e0 con 0x7fcc1802d310 2024-04-26T16:28:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc1810b3e0 con 0x7fcc1802d310 2024-04-26T16:28:27.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 4 2024-04-26T16:28:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 1 4 2024-04-26T16:28:27.720 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:28:27.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 10 client.41201.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 5 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcc181ad4d0 0x7fcc181af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 5 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 2 5 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 4 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 10 client.41201.objecter _send_command 5 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fcc181b2720 con 0x7fcc181ad4d0 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcc181ad4d0 0x7fcc181af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc181ad4d0 2024-04-26T16:28:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.722+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcc181ad4d0 0x7fcc181af8c0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.726+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181b2720 con 0x7fcc181ad4d0 2024-04-26T16:28:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.726+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 5: 0 ) 2024-04-26T16:28:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.726+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 5 = system:0 2024-04-26T16:28:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.726+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 4 2024-04-26T16:28:27.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.726+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 2 5 2024-04-26T16:28:27.735 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:27.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 10 client.41201.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 6 2024-04-26T16:28:27.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 6 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc1810b7e0 osd=2 3 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 2 6 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 4 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 10 client.41201.objecter _send_command 6 2024-04-26T16:28:27.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc1802de80 con 0x7fcc181ad4d0 2024-04-26T16:28:27.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc1802de80 con 0x7fcc181ad4d0 2024-04-26T16:28:27.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.740 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 4 2024-04-26T16:28:27.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 2 6 2024-04-26T16:28:27.741 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 10 client.41201.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 7 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcc181aae70 0x7fcc181b0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 7 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 3 7 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 4 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 10 client.41201.objecter _send_command 7 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fcc181b6080 con 0x7fcc181aae70 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.742+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcc181aae70 0x7fcc181b0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc181aae70 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcc181aae70 0x7fcc181b0f80 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181b6080 con 0x7fcc181aae70 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 7: 0 ) 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 7 = system:0 2024-04-26T16:28:27.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 4 2024-04-26T16:28:27.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.746+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 3 7 2024-04-26T16:28:27.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 10 client.41201.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 8 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 8 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc1802e910 osd=3 3 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 3 8 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 4 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 10 client.41201.objecter _send_command 8 2024-04-26T16:28:27.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.762+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc181aa440 con 0x7fcc181aae70 2024-04-26T16:28:27.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc181aa440 con 0x7fcc181aae70 2024-04-26T16:28:27.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 4 2024-04-26T16:28:27.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 3 8 2024-04-26T16:28:27.762 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:28:27.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 10 client.41201.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 9 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcc181b0ae0 0x7fcc181b48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 9 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 4 9 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 4 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 10 client.41201.objecter _send_command 9 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fcc181aa770 con 0x7fcc181b0ae0 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcc181b0ae0 0x7fcc181b48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc181b0ae0 2024-04-26T16:28:27.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcc181b0ae0 0x7fcc181b48e0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181aa770 con 0x7fcc181b0ae0 2024-04-26T16:28:27.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 9: 0 ) 2024-04-26T16:28:27.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 9 = system:0 2024-04-26T16:28:27.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 4 2024-04-26T16:28:27.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.766+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 4 9 2024-04-26T16:28:27.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 10 client.41201.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 10 2024-04-26T16:28:27.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 10 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181aa460 osd=4 3 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 4 10 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 4 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 10 client.41201.objecter _send_command 10 2024-04-26T16:28:27.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.790+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc181b4e20 con 0x7fcc181b0ae0 2024-04-26T16:28:27.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc181b4e20 con 0x7fcc181b0ae0 2024-04-26T16:28:27.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 4 2024-04-26T16:28:27.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 4 10 2024-04-26T16:28:27.790 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 10 client.41201.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 11 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcc181b5540 0x7fcc181b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 11 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 5 11 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 4 2024-04-26T16:28:27.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 10 client.41201.objecter _send_command 11 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fcc181bcb70 con 0x7fcc181b5540 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcc181b5540 0x7fcc181b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc181b5540 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc1db7d700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcc181b5540 0x7fcc181b5980 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181bcb70 con 0x7fcc181b5540 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 11: 0 ) 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 11 = system:0 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 4 2024-04-26T16:28:27.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.794+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 5 11 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 10 client.41201.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 12 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 12 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181b4e20 osd=5 3 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 5 12 2024-04-26T16:28:27.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 4 2024-04-26T16:28:27.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 10 client.41201.objecter _send_command 12 2024-04-26T16:28:27.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.814+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc181ba250 con 0x7fcc181b5540 2024-04-26T16:28:27.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc181ba250 con 0x7fcc181b5540 2024-04-26T16:28:27.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 4 2024-04-26T16:28:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 5 12 2024-04-26T16:28:27.815 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:28:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 10 client.41201.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 13 2024-04-26T16:28:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcc181ba920 0x7fcc181bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 13 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 6 13 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 4 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 10 client.41201.objecter _send_command 13 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fcc181bfe40 con 0x7fcc181ba920 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcc181ba920 0x7fcc181bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc181ba920 2024-04-26T16:28:27.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.818+0000 7fcc1eb7f700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcc181ba920 0x7fcc181bcdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.822+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181bfe40 con 0x7fcc181ba920 2024-04-26T16:28:27.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.822+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 13: 0 ) 2024-04-26T16:28:27.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.822+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 13 = system:0 2024-04-26T16:28:27.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.822+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 4 2024-04-26T16:28:27.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.822+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 6 13 2024-04-26T16:28:27.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 10 client.41201.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 14 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 14 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181ba250 osd=6 3 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 6 14 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 4 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 10 client.41201.objecter _send_command 14 2024-04-26T16:28:27.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.838+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc181bebd0 con 0x7fcc181ba920 2024-04-26T16:28:27.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc181bebd0 con 0x7fcc181ba920 2024-04-26T16:28:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 4 2024-04-26T16:28:27.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 6 14 2024-04-26T16:28:27.841 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:28:27.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 10 client.41201.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:27.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 15 2024-04-26T16:28:27.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcc181bf340 0x7fcc181bf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:27.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 15 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 7 15 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 4 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 10 client.41201.objecter _send_command 15 2024-04-26T16:28:27.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fcc181be720 con 0x7fcc181bf340 2024-04-26T16:28:27.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.842+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcc181bf340 0x7fcc181bf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:27.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc1e37e700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcc181bf340 0x7fcc181bf7a0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:27.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc0f7fe700 10 client.41201.objecter ms_handle_connect 0x7fcc181bf340 2024-04-26T16:28:27.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc181be720 con 0x7fcc181bf340 2024-04-26T16:28:27.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 15: 0 ) 2024-04-26T16:28:27.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 15 = system:0 2024-04-26T16:28:27.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 4 2024-04-26T16:28:27.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.846+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 7 15 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:26.772001+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:26.772001+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:26.775343+0000 mon.c (mon.2) 612 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:26.775343+0000 mon.c (mon.2) 612 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:26.867306+0000 mgr.y (mgr.24401) 433 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:26.867306+0000 mgr.y (mgr.24401) 433 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:27.193279+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:27.193279+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:27.194949+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:27.194949+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: cluster 2024-04-26T16:28:27.647706+0000 mgr.y (mgr.24401) 434 : cluster [DBG] pgmap v638: 228 pgs: 16 active+undersized, 6 active+undersized+degraded, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 24/1050 objects degraded (2.286%); 3 B/s, 0 objects/s recovering 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: cluster 2024-04-26T16:28:27.647706+0000 mgr.y (mgr.24401) 434 : cluster [DBG] pgmap v638: 228 pgs: 16 active+undersized, 6 active+undersized+degraded, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 24/1050 objects degraded (2.286%); 3 B/s, 0 objects/s recovering 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:27.681600+0000 mon.c (mon.2) 613 : audit [DBG] from='client.? 172.21.15.50:0/591319990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[21463]: audit 2024-04-26T16:28:27.681600+0000 mon.c (mon.2) 613 : audit [DBG] from='client.? 172.21.15.50:0/591319990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:26.772001+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:26.772001+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:26.775343+0000 mon.c (mon.2) 612 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:26.775343+0000 mon.c (mon.2) 612 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:26.867306+0000 mgr.y (mgr.24401) 433 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:26.867306+0000 mgr.y (mgr.24401) 433 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:27.193279+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:27.193279+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:27.194949+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:27.194949+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:27.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: cluster 2024-04-26T16:28:27.647706+0000 mgr.y (mgr.24401) 434 : cluster [DBG] pgmap v638: 228 pgs: 16 active+undersized, 6 active+undersized+degraded, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 24/1050 objects degraded (2.286%); 3 B/s, 0 objects/s recovering 2024-04-26T16:28:27.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: cluster 2024-04-26T16:28:27.647706+0000 mgr.y (mgr.24401) 434 : cluster [DBG] pgmap v638: 228 pgs: 16 active+undersized, 6 active+undersized+degraded, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 24/1050 objects degraded (2.286%); 3 B/s, 0 objects/s recovering 2024-04-26T16:28:27.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:27.681600+0000 mon.c (mon.2) 613 : audit [DBG] from='client.? 172.21.15.50:0/591319990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:27.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:27 smithi050 bash[29234]: audit 2024-04-26T16:28:27.681600+0000 mon.c (mon.2) 613 : audit [DBG] from='client.? 172.21.15.50:0/591319990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:27.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:27.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 10 client.41201.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:28:27.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign -1 16 2024-04-26T16:28:27.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 20 client.41201.objecter _get_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_remove -1 16 2024-04-26T16:28:27.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 20 client.41201.objecter get_session s=0x7fcc181bebd0 osd=7 3 2024-04-26T16:28:27.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 15 client.41201.objecter _session_command_op_assign 7 16 2024-04-26T16:28:27.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 4 2024-04-26T16:28:27.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 10 client.41201.objecter _send_command 16 2024-04-26T16:28:27.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.866+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fcc181be6e0 con 0x7fcc181bf340 2024-04-26T16:28:27.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc0f7fe700 1 -- 172.21.15.50:0/591319990 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcc181be6e0 con 0x7fcc181bf340 2024-04-26T16:28:27.866 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc0f7fe700 10 client.41201.objecter ms_dispatch 0x7fcc18001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc0f7fe700 10 client.41201.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc0f7fe700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 4 2024-04-26T16:28:27.867 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc0f7fe700 15 client.41201.objecter _session_command_op_remove 7 16 2024-04-26T16:28:27.867 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 librados: watch_flush enter 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 librados: watch_flush exit 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.0 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcc18064030 msgr2=0x7fcc18064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcc18064030 0x7fcc18064410 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc18105530 osd=0 1 2024-04-26T16:28:27.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.1 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcc1802d310 msgr2=0x7fcc1802d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcc1802d310 0x7fcc1802d6f0 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b960 osd=1 1 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.2 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcc181ad4d0 msgr2=0x7fcc181af8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcc181ad4d0 0x7fcc181af8c0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1810b7e0 osd=2 1 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.3 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcc181aae70 msgr2=0x7fcc181b0f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcc181aae70 0x7fcc181b0f80 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc1eb7f700 1 -- 172.21.15.50:0/591319990 reap_dead start 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc1802e910 osd=3 1 2024-04-26T16:28:27.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.4 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcc181b0ae0 msgr2=0x7fcc181b48e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcc181b0ae0 0x7fcc181b48e0 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc1eb7f700 1 -- 172.21.15.50:0/591319990 reap_dead start 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181aa460 osd=4 1 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.5 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcc181b5540 msgr2=0x7fcc181b5980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcc181b5540 0x7fcc181b5980 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181b4e20 osd=5 1 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.6 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcc181ba920 msgr2=0x7fcc181bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcc181ba920 0x7fcc181bcdb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181ba250 osd=6 1 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter close_session for osd.7 2024-04-26T16:28:27.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcc181bf340 msgr2=0x7fcc181bf7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcc181bf340 0x7fcc181bf7a0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter put_session s=0x7fcc181bebd0 osd=7 1 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 20 client.41201.objecter shutdown clearing up homeless session... 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 10 client.41201.objecter successfully canceled tick 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc04071f80 msgr2=0x7fcc04074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcc04071f80 0x7fcc04074440 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7fcc100060b0 tx=0x7fcc100085c0 comp rx=0 tx=0).stop 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18108390 msgr2=0x7fcc18104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 --2- 172.21.15.50:0/591319990 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcc18108390 0x7fcc18104a80 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400e970 tx=0x7fcc1400ec80 comp rx=0 tx=0).stop 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc1eb7f700 1 -- 172.21.15.50:0/591319990 reap_dead start 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 shutdown_connections 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 >> 172.21.15.50:0/591319990 conn(0x7fcc180792d0 msgr2=0x7fcc18101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 shutdown_connections 2024-04-26T16:28:27.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 -- 172.21.15.50:0/591319990 wait complete. 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:27.870+0000 7fcc205e2700 1 librados: shutdown 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:28:27.872 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:28:27.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:27.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:27.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/384462791 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 msgr2=0x7f56ac10f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- 172.21.15.50:0/384462791 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac10f1c0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f56a400b0a0 tx=0x7f56a400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/384462791 shutdown_connections 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- 172.21.15.50:0/384462791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56ac108380 0x7f56ac108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- 172.21.15.50:0/384462791 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac10f1c0 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- 172.21.15.50:0/384462791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac102310 0x7f56ac102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/384462791 >> 172.21.15.50:0/384462791 conn(0x7f56ac079290 msgr2=0x7f56ac0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/384462791 shutdown_connections 2024-04-26T16:28:28.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/384462791 wait complete. 2024-04-26T16:28:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 librados: starting msgr at 2024-04-26T16:28:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 librados: starting objecter 2024-04-26T16:28:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 Processor -- start 2024-04-26T16:28:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- start start 2024-04-26T16:28:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 librados: setting wanted keys 2024-04-26T16:28:28.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 librados: calling monclient init 2024-04-26T16:28:28.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56ac102310 0x7f56ac103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:28.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:28.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac108380 0x7f56ac104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:28.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac111990 con 0x7f56ac108380 2024-04-26T16:28:28.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac111810 con 0x7f56ac102310 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f56ac111b10 con 0x7f56ac102cd0 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b109c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac108380 0x7f56ac104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b109c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac108380 0x7f56ac104a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43214/0 (socket says 172.21.15.50:43214) 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b109c700 1 -- 172.21.15.50:0/1046001090 learned_addr learned my addr 172.21.15.50:0/1046001090 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b089b700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56ac102310 0x7f56ac103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 -- 172.21.15.50:0/1046001090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56ac102310 msgr2=0x7f56ac103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56ac102310 0x7f56ac103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 -- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac108380 msgr2=0x7f56ac104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac108380 0x7f56ac104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 -- 172.21.15.50:0/1046001090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ac1a7650 con 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56abfff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac104530 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f569800e9f0 tx=0x7f569800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56a9ffb700 10 client.?.objecter ms_handle_connect 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56a9ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569800cb70 con 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56ac1a7940 con 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5698004480 con 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5698003bf0 con 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 10 client.41207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56ac1a7e80 con 0x7f56ac102cd0 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 librados: init done 2024-04-26T16:28:28.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.062+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56ac06a110 con 0x7f56ac102cd0 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5698003dd0 con 0x7f56ac102cd0 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f569c071f80 0x7f569c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(435..435 src has 87..435) ==== 9395+0+0 (secure 0 0 0) 0x7f5698091c40 con 0x7f56ac102cd0 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 10 client.41207.objecter ms_dispatch 0x7f56ac001550 osd_map(435..435 src has 87..435) 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 3 client.41207.objecter handle_osd_map got epochs [435,435] > 0 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 3 client.41207.objecter handle_osd_map decoding full epoch 435 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56b089b700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f569c071f80 0x7f569c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:28.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 20 client.41207.objecter dump_active .. 0 homeless 2024-04-26T16:28:28.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56a9ffb700 10 client.41207.objecter ms_handle_connect 0x7f569c071f80 2024-04-26T16:28:28.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.066+0000 7f56b089b700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f569c071f80 0x7f569c074440 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7f56a000bbb0 tx=0x7f56a000b590 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:28.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.070+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f569805d5e0 con 0x7f56ac102cd0 2024-04-26T16:28:28.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:26.772001+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:26.772001+0000 mon.a (mon.0) 2673 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:26.775343+0000 mon.c (mon.2) 612 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:26.775343+0000 mon.c (mon.2) 612 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:26.867306+0000 mgr.y (mgr.24401) 433 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:26.867306+0000 mgr.y (mgr.24401) 433 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:27.193279+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:27.193279+0000 mon.a (mon.0) 2674 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:27.194949+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:27.194949+0000 mon.b (mon.1) 297 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: cluster 2024-04-26T16:28:27.647706+0000 mgr.y (mgr.24401) 434 : cluster [DBG] pgmap v638: 228 pgs: 16 active+undersized, 6 active+undersized+degraded, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 24/1050 objects degraded (2.286%); 3 B/s, 0 objects/s recovering 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: cluster 2024-04-26T16:28:27.647706+0000 mgr.y (mgr.24401) 434 : cluster [DBG] pgmap v638: 228 pgs: 16 active+undersized, 6 active+undersized+degraded, 206 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 24/1050 objects degraded (2.286%); 3 B/s, 0 objects/s recovering 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:27.681600+0000 mon.c (mon.2) 613 : audit [DBG] from='client.? 172.21.15.50:0/591319990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:28.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:27 smithi190 bash[23868]: audit 2024-04-26T16:28:27.681600+0000 mon.c (mon.2) 613 : audit [DBG] from='client.? 172.21.15.50:0/591319990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:28.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:28.111 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:28.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:28.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:28.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:28.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.314+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f56ac1a80b0 con 0x7f56ac102cd0 2024-04-26T16:28:28.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:28.366 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:28.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:28.493 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:28.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:28.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:28.623 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:28.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:28.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:28.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.778+0000 7f56a9ffb700 1 -- 172.21.15.50:0/1046001090 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v436) ==== 91+0+0 (secure 0 0 0) 0x7f5698061290 con 0x7f56ac102cd0 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 10 librados: watch_flush enter 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 10 librados: watch_flush exit 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 20 client.41207.objecter shutdown clearing up homeless session... 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 10 client.41207.objecter successfully canceled tick 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f569c071f80 msgr2=0x7f569c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f569c071f80 0x7f569c074440 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7f56a000bbb0 tx=0x7f56a000b590 comp rx=0 tx=0).stop 2024-04-26T16:28:28.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 msgr2=0x7f56ac104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:28.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac104530 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7f569800e9f0 tx=0x7f569800ed00 comp rx=0 tx=0).stop 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 shutdown_connections 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f569c071f80 0x7f569c074440 unknown :-1 s=CLOSED pgs=3746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f56ac102310 0x7f56ac103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f56ac108380 0x7f56ac104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 --2- 172.21.15.50:0/1046001090 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f56ac102cd0 0x7f56ac104530 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 >> 172.21.15.50:0/1046001090 conn(0x7f56ac079290 msgr2=0x7f56ac1010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 shutdown_connections 2024-04-26T16:28:28.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 -- 172.21.15.50:0/1046001090 wait complete. 2024-04-26T16:28:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:28.782+0000 7f56b2aff700 1 librados: shutdown 2024-04-26T16:28:28.782 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:28:28.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:28.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:28.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:28.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:28.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:27.772465+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:28.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:27.772465+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: cluster 2024-04-26T16:28:27.778093+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: cluster 2024-04-26T16:28:27.778093+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:27.780396+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:27.780396+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:27.781072+0000 mon.a (mon.0) 2677 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:27.781072+0000 mon.a (mon.0) 2677 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:28.315535+0000 mon.c (mon.2) 614 : audit [INF] from='client.? 172.21.15.50:0/1046001090' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:28.315535+0000 mon.c (mon.2) 614 : audit [INF] from='client.? 172.21.15.50:0/1046001090' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:28.315936+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[21463]: audit 2024-04-26T16:28:28.315936+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.992 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:27.772465+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:27.772465+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: cluster 2024-04-26T16:28:27.778093+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: cluster 2024-04-26T16:28:27.778093+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:27.780396+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:27.780396+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:27.781072+0000 mon.a (mon.0) 2677 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:27.781072+0000 mon.a (mon.0) 2677 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:28.315535+0000 mon.c (mon.2) 614 : audit [INF] from='client.? 172.21.15.50:0/1046001090' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:28.315535+0000 mon.c (mon.2) 614 : audit [INF] from='client.? 172.21.15.50:0/1046001090' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:28.315936+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:28.993 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:28 smithi050 bash[29234]: audit 2024-04-26T16:28:28.315936+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:29.003 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:27.772465+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:27.772465+0000 mon.a (mon.0) 2675 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: cluster 2024-04-26T16:28:27.778093+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: cluster 2024-04-26T16:28:27.778093+0000 mon.a (mon.0) 2676 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:27.780396+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:27.780396+0000 mon.b (mon.1) 298 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:27.781072+0000 mon.a (mon.0) 2677 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:27.781072+0000 mon.a (mon.0) 2677 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:29.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:28.315535+0000 mon.c (mon.2) 614 : audit [INF] from='client.? 172.21.15.50:0/1046001090' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:29.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:28.315535+0000 mon.c (mon.2) 614 : audit [INF] from='client.? 172.21.15.50:0/1046001090' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:29.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:28.315936+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:29.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:28 smithi190 bash[23868]: audit 2024-04-26T16:28:28.315936+0000 mon.a (mon.0) 2678 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:28:29.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:29.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:29.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:29.266 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:29.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:29.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:29.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:29.526 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:29.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:29.657 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:29.760 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:29.795 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:29.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:29.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:29.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:30.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:30.050 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:30.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.772905+0000 mon.a (mon.0) 2679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:30.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.772905+0000 mon.a (mon.0) 2679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:30.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.774724+0000 mon.a (mon.0) 2680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:30.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.774724+0000 mon.a (mon.0) 2680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:30.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.774838+0000 mon.a (mon.0) 2681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.774838+0000 mon.a (mon.0) 2681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.778576+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.778576+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.829291+0000 osd.2 (osd.2) 197 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.829291+0000 osd.2 (osd.2) 197 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.831675+0000 osd.2 (osd.2) 198 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:28.831675+0000 osd.2 (osd.2) 198 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.838572+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.838572+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.839846+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.839846+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.840505+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.840505+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.841886+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: audit 2024-04-26T16:28:28.841886+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.648650+0000 mgr.y (mgr.24401) 435 : cluster [DBG] pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-04-26T16:28:30.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:29 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.648650+0000 mgr.y (mgr.24401) 435 : cluster [DBG] pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-04-26T16:28:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.772905+0000 mon.a (mon.0) 2679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.772905+0000 mon.a (mon.0) 2679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.774724+0000 mon.a (mon.0) 2680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.774724+0000 mon.a (mon.0) 2680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.774838+0000 mon.a (mon.0) 2681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.774838+0000 mon.a (mon.0) 2681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.778576+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.778576+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.829291+0000 osd.2 (osd.2) 197 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.829291+0000 osd.2 (osd.2) 197 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.831675+0000 osd.2 (osd.2) 198 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:28.831675+0000 osd.2 (osd.2) 198 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.838572+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.838572+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.839846+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.839846+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.840505+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.840505+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.841886+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.772905+0000 mon.a (mon.0) 2679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.772905+0000 mon.a (mon.0) 2679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.774724+0000 mon.a (mon.0) 2680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.774724+0000 mon.a (mon.0) 2680 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]': finished 2024-04-26T16:28:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.774838+0000 mon.a (mon.0) 2681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.774838+0000 mon.a (mon.0) 2681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.778576+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.778576+0000 mon.a (mon.0) 2682 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.829291+0000 osd.2 (osd.2) 197 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.829291+0000 osd.2 (osd.2) 197 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.831675+0000 osd.2 (osd.2) 198 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:28.831675+0000 osd.2 (osd.2) 198 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.838572+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.838572+0000 mon.a (mon.0) 2683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.839846+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.839846+0000 mon.a (mon.0) 2684 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.840505+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.840505+0000 mon.b (mon.1) 299 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.841886+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: audit 2024-04-26T16:28:28.841886+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.648650+0000 mgr.y (mgr.24401) 435 : cluster [DBG] pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-04-26T16:28:30.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.648650+0000 mgr.y (mgr.24401) 435 : cluster [DBG] pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-04-26T16:28:30.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: audit 2024-04-26T16:28:28.841886+0000 mon.b (mon.1) 300 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-51"}]: dispatch 2024-04-26T16:28:30.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.648650+0000 mgr.y (mgr.24401) 435 : cluster [DBG] pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-04-26T16:28:30.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:29 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.648650+0000 mgr.y (mgr.24401) 435 : cluster [DBG] pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-04-26T16:28:30.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:30.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:30.313 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:30.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:30.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:30.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:30.480 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:30.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:30.578 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:30.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:30.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:30.706 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:30.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:30.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:30 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.784943+0000 mon.a (mon.0) 2685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:28:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:30 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.784943+0000 mon.a (mon.0) 2685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:28:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:30 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.784972+0000 mon.a (mon.0) 2686 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1050 objects degraded (2.286%), 6 pgs degraded) 2024-04-26T16:28:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:30 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.784972+0000 mon.a (mon.0) 2686 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1050 objects degraded (2.286%), 6 pgs degraded) 2024-04-26T16:28:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:30 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.796516+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-04-26T16:28:30.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:30 smithi190 bash[23868]: cluster 2024-04-26T16:28:29.796516+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-04-26T16:28:30.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:30.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:30.978 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:30.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:31.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:28:31.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:31.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.784943+0000 mon.a (mon.0) 2685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:28:31.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.784943+0000 mon.a (mon.0) 2685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:28:31.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.784972+0000 mon.a (mon.0) 2686 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1050 objects degraded (2.286%), 6 pgs degraded) 2024-04-26T16:28:31.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.784972+0000 mon.a (mon.0) 2686 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1050 objects degraded (2.286%), 6 pgs degraded) 2024-04-26T16:28:31.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.796516+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-04-26T16:28:31.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[29234]: cluster 2024-04-26T16:28:29.796516+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-04-26T16:28:31.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.784943+0000 mon.a (mon.0) 2685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:28:31.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.784943+0000 mon.a (mon.0) 2685 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:28:31.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.784972+0000 mon.a (mon.0) 2686 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1050 objects degraded (2.286%), 6 pgs degraded) 2024-04-26T16:28:31.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.784972+0000 mon.a (mon.0) 2686 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/1050 objects degraded (2.286%), 6 pgs degraded) 2024-04-26T16:28:31.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.796516+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-04-26T16:28:31.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:30 smithi050 bash[21463]: cluster 2024-04-26T16:28:29.796516+0000 mon.a (mon.0) 2687 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2024-04-26T16:28:31.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:31.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:31.233 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:31.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:31.363 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:31.466 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:31.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:31.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:31.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:31.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:31.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:31.889 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:31.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:31.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:28:31.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:30.437138+0000 osd.4 (osd.4) 221 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:30.437138+0000 osd.4 (osd.4) 221 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:30.440335+0000 osd.4 (osd.4) 222 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:30.440335+0000 osd.4 (osd.4) 222 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:30.821608+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:30.821608+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-04-26T16:28:32.002 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: audit 2024-04-26T16:28:30.832614+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: audit 2024-04-26T16:28:30.832614+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: audit 2024-04-26T16:28:30.833724+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: audit 2024-04-26T16:28:30.833724+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.649983+0000 mgr.y (mgr.24401) 436 : cluster [DBG] pgmap v644: 228 pgs: 28 creating+peering, 4 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:32.006 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.649983+0000 mgr.y (mgr.24401) 436 : cluster [DBG] pgmap v644: 228 pgs: 28 creating+peering, 4 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:32.006 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:30.437138+0000 osd.4 (osd.4) 221 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:30.437138+0000 osd.4 (osd.4) 221 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:30.440335+0000 osd.4 (osd.4) 222 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:30.440335+0000 osd.4 (osd.4) 222 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:30.821608+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:30.821608+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: audit 2024-04-26T16:28:30.832614+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: audit 2024-04-26T16:28:30.832614+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: audit 2024-04-26T16:28:30.833724+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: audit 2024-04-26T16:28:30.833724+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.649983+0000 mgr.y (mgr.24401) 436 : cluster [DBG] pgmap v644: 228 pgs: 28 creating+peering, 4 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:32.007 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:31 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.649983+0000 mgr.y (mgr.24401) 436 : cluster [DBG] pgmap v644: 228 pgs: 28 creating+peering, 4 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:32.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:30.437138+0000 osd.4 (osd.4) 221 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:30.437138+0000 osd.4 (osd.4) 221 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:30.440335+0000 osd.4 (osd.4) 222 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:30.440335+0000 osd.4 (osd.4) 222 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:30.821608+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:30.821608+0000 mon.a (mon.0) 2688 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: audit 2024-04-26T16:28:30.832614+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: audit 2024-04-26T16:28:30.832614+0000 mon.a (mon.0) 2689 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: audit 2024-04-26T16:28:30.833724+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: audit 2024-04-26T16:28:30.833724+0000 mon.b (mon.1) 301 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.649983+0000 mgr.y (mgr.24401) 436 : cluster [DBG] pgmap v644: 228 pgs: 28 creating+peering, 4 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:31 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.649983+0000 mgr.y (mgr.24401) 436 : cluster [DBG] pgmap v644: 228 pgs: 28 creating+peering, 4 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:32.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:32.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:32.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:32.277 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:32.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:32.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:32.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:32.538 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:32.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:32.664 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:32.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:32.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:32.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:32.906 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.419476+0000 osd.4 (osd.4) 223 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.419476+0000 osd.4 (osd.4) 223 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.422176+0000 osd.4 (osd.4) 224 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.422176+0000 osd.4 (osd.4) 224 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.825578+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.825578+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.830239+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.830239+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.875398+0000 osd.2 (osd.2) 199 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.875398+0000 osd.2 (osd.2) 199 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.878192+0000 osd.2 (osd.2) 200 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:31.878192+0000 osd.2 (osd.2) 200 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:28:32.907 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.899378+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.899378+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.900196+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.900196+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.901459+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.901459+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.902338+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: audit 2024-04-26T16:28:31.902338+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.151499+0000 mon.a (mon.0) 2694 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:32.908 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.151499+0000 mon.a (mon.0) 2694 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.419476+0000 osd.4 (osd.4) 223 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.419476+0000 osd.4 (osd.4) 223 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.422176+0000 osd.4 (osd.4) 224 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.422176+0000 osd.4 (osd.4) 224 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.825578+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.825578+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.830239+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.830239+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.875398+0000 osd.2 (osd.2) 199 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.875398+0000 osd.2 (osd.2) 199 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.878192+0000 osd.2 (osd.2) 200 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:31.878192+0000 osd.2 (osd.2) 200 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:28:32.909 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.899378+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.899378+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.900196+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.900196+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.901459+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.901459+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.902338+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: audit 2024-04-26T16:28:31.902338+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.151499+0000 mon.a (mon.0) 2694 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:32.910 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:32 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.151499+0000 mon.a (mon.0) 2694 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:32.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:32.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:32.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:33.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:33.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:33.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:33.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.419476+0000 osd.4 (osd.4) 223 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:28:33.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.419476+0000 osd.4 (osd.4) 223 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:28:33.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.422176+0000 osd.4 (osd.4) 224 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:28:33.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.422176+0000 osd.4 (osd.4) 224 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:28:33.167 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.825578+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.825578+0000 mon.a (mon.0) 2690 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.830239+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.830239+0000 mon.a (mon.0) 2691 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.875398+0000 osd.2 (osd.2) 199 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.875398+0000 osd.2 (osd.2) 199 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.878192+0000 osd.2 (osd.2) 200 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:31.878192+0000 osd.2 (osd.2) 200 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.899378+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.899378+0000 mon.a (mon.0) 2692 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.900196+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.900196+0000 mon.a (mon.0) 2693 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.901459+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.901459+0000 mon.b (mon.1) 302 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:33.168 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.902338+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:33.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: audit 2024-04-26T16:28:31.902338+0000 mon.b (mon.1) 303 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-53"}]: dispatch 2024-04-26T16:28:33.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.151499+0000 mon.a (mon.0) 2694 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:33.169 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:32 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.151499+0000 mon.a (mon.0) 2694 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:33.182 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:33.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:33.310 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:33.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:33.413 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:33.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:33.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:33.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:33.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:33.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:33.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:33.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:33.836 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:33.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.407734+0000 osd.4 (osd.4) 225 : cluster [DBG] 109.a scrub starts 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.407734+0000 osd.4 (osd.4) 225 : cluster [DBG] 109.a scrub starts 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.409883+0000 osd.4 (osd.4) 226 : cluster [DBG] 109.a scrub ok 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.409883+0000 osd.4 (osd.4) 226 : cluster [DBG] 109.a scrub ok 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.840041+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.840041+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.978565+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.978565+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:28:33.954 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.981790+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:28:33.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:32.981790+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:28:33.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.650904+0000 mgr.y (mgr.24401) 437 : cluster [DBG] pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:33.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:33 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.650904+0000 mgr.y (mgr.24401) 437 : cluster [DBG] pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:33.969 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:33.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:34.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:34.084 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.407734+0000 osd.4 (osd.4) 225 : cluster [DBG] 109.a scrub starts 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.407734+0000 osd.4 (osd.4) 225 : cluster [DBG] 109.a scrub starts 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.409883+0000 osd.4 (osd.4) 226 : cluster [DBG] 109.a scrub ok 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.409883+0000 osd.4 (osd.4) 226 : cluster [DBG] 109.a scrub ok 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.840041+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.840041+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.978565+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.978565+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.981790+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:32.981790+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.650904+0000 mgr.y (mgr.24401) 437 : cluster [DBG] pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:34.085 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.650904+0000 mgr.y (mgr.24401) 437 : cluster [DBG] pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.407734+0000 osd.4 (osd.4) 225 : cluster [DBG] 109.a scrub starts 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.407734+0000 osd.4 (osd.4) 225 : cluster [DBG] 109.a scrub starts 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.409883+0000 osd.4 (osd.4) 226 : cluster [DBG] 109.a scrub ok 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.409883+0000 osd.4 (osd.4) 226 : cluster [DBG] 109.a scrub ok 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.840041+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.840041+0000 mon.a (mon.0) 2695 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.978565+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.978565+0000 osd.1 (osd.1) 221 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.981790+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:32.981790+0000 osd.1 (osd.1) 222 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.650904+0000 mgr.y (mgr.24401) 437 : cluster [DBG] pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:34.086 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:33 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.650904+0000 mgr.y (mgr.24401) 437 : cluster [DBG] pgmap v647: 196 pgs: 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:28:34.097 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:34.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:34.227 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:34.330 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:34.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:34.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:34.494 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:34.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:34.625 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:34.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:28:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:28:34.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:34.755 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:34.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:34.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:34.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:35.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:35.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.163689+0000 osd.3 (osd.3) 261 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.163689+0000 osd.3 (osd.3) 261 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.172447+0000 osd.3 (osd.3) 262 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.172447+0000 osd.3 (osd.3) 262 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.266491+0000 osd.6 (osd.6) 3 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.266491+0000 osd.6 (osd.6) 3 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.269512+0000 osd.6 (osd.6) 4 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.269512+0000 osd.6 (osd.6) 4 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.865045+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.865045+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: audit 2024-04-26T16:28:33.869636+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: audit 2024-04-26T16:28:33.869636+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: audit 2024-04-26T16:28:33.873016+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: audit 2024-04-26T16:28:33.873016+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.893010+0000 osd.2 (osd.2) 201 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.893010+0000 osd.2 (osd.2) 201 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.896475+0000 osd.2 (osd.2) 202 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[21463]: cluster 2024-04-26T16:28:33.896475+0000 osd.2 (osd.2) 202 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.163689+0000 osd.3 (osd.3) 261 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.163689+0000 osd.3 (osd.3) 261 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.172447+0000 osd.3 (osd.3) 262 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.172447+0000 osd.3 (osd.3) 262 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:28:35.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.266491+0000 osd.6 (osd.6) 3 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.266491+0000 osd.6 (osd.6) 3 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.269512+0000 osd.6 (osd.6) 4 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.269512+0000 osd.6 (osd.6) 4 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.865045+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.865045+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: audit 2024-04-26T16:28:33.869636+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: audit 2024-04-26T16:28:33.869636+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: audit 2024-04-26T16:28:33.873016+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: audit 2024-04-26T16:28:33.873016+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.893010+0000 osd.2 (osd.2) 201 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.893010+0000 osd.2 (osd.2) 201 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.896475+0000 osd.2 (osd.2) 202 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:28:35.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:34 smithi050 bash[29234]: cluster 2024-04-26T16:28:33.896475+0000 osd.2 (osd.2) 202 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:28:35.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:35.144 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.163689+0000 osd.3 (osd.3) 261 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:28:35.144 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.163689+0000 osd.3 (osd.3) 261 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:28:35.144 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.172447+0000 osd.3 (osd.3) 262 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:28:35.144 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.172447+0000 osd.3 (osd.3) 262 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.266491+0000 osd.6 (osd.6) 3 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.266491+0000 osd.6 (osd.6) 3 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.269512+0000 osd.6 (osd.6) 4 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.269512+0000 osd.6 (osd.6) 4 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.865045+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.865045+0000 mon.a (mon.0) 2696 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: audit 2024-04-26T16:28:33.869636+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: audit 2024-04-26T16:28:33.869636+0000 mon.b (mon.1) 304 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: audit 2024-04-26T16:28:33.873016+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: audit 2024-04-26T16:28:33.873016+0000 mon.a (mon.0) 2697 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.893010+0000 osd.2 (osd.2) 201 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.893010+0000 osd.2 (osd.2) 201 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.896475+0000 osd.2 (osd.2) 202 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:28:35.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:34 smithi190 bash[23868]: cluster 2024-04-26T16:28:33.896475+0000 osd.2 (osd.2) 202 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:28:35.159 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:35.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:35.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:35.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:35.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:35.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:35.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:35.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:35.681 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:35.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:35.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:35.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:35.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:35.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:35.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:36.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:34.165410+0000 osd.3 (osd.3) 263 : cluster [DBG] 3.c scrub starts 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:34.165410+0000 osd.3 (osd.3) 263 : cluster [DBG] 3.c scrub starts 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:34.168073+0000 osd.3 (osd.3) 264 : cluster [DBG] 3.c scrub ok 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:34.168073+0000 osd.3 (osd.3) 264 : cluster [DBG] 3.c scrub ok 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.852005+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.852005+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:34.857884+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:34.857884+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.896189+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.896189+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:36.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.946928+0000 mon.a (mon.0) 2700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.946928+0000 mon.a (mon.0) 2700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.948350+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.948350+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.948825+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.948825+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.950339+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: audit 2024-04-26T16:28:34.950339+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.229963+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.229963+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.232451+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.232451+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.651710+0000 mgr.y (mgr.24401) 438 : cluster [DBG] pgmap v650: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:36.056 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.651710+0000 mgr.y (mgr.24401) 438 : cluster [DBG] pgmap v650: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.858780+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.858780+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:34.165410+0000 osd.3 (osd.3) 263 : cluster [DBG] 3.c scrub starts 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:34.165410+0000 osd.3 (osd.3) 263 : cluster [DBG] 3.c scrub starts 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:34.168073+0000 osd.3 (osd.3) 264 : cluster [DBG] 3.c scrub ok 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:34.168073+0000 osd.3 (osd.3) 264 : cluster [DBG] 3.c scrub ok 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.852005+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.852005+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:34.857884+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-04-26T16:28:36.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:34.857884+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.896189+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.896189+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.946928+0000 mon.a (mon.0) 2700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.946928+0000 mon.a (mon.0) 2700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.948350+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.948350+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.948825+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.948825+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.950339+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: audit 2024-04-26T16:28:34.950339+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.229963+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.229963+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.232451+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:28:36.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.232451+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:28:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.651710+0000 mgr.y (mgr.24401) 438 : cluster [DBG] pgmap v650: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.651710+0000 mgr.y (mgr.24401) 438 : cluster [DBG] pgmap v650: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.858780+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-04-26T16:28:36.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:35 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.858780+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-04-26T16:28:36.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:36.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:34.165410+0000 osd.3 (osd.3) 263 : cluster [DBG] 3.c scrub starts 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:34.165410+0000 osd.3 (osd.3) 263 : cluster [DBG] 3.c scrub starts 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:34.168073+0000 osd.3 (osd.3) 264 : cluster [DBG] 3.c scrub ok 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:34.168073+0000 osd.3 (osd.3) 264 : cluster [DBG] 3.c scrub ok 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.852005+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.852005+0000 mon.a (mon.0) 2698 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:34.857884+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:34.857884+0000 mon.a (mon.0) 2699 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.896189+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:36.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.896189+0000 mon.b (mon.1) 305 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.946928+0000 mon.a (mon.0) 2700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.946928+0000 mon.a (mon.0) 2700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.948350+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.948350+0000 mon.a (mon.0) 2701 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.948825+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.948825+0000 mon.b (mon.1) 306 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.950339+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: audit 2024-04-26T16:28:34.950339+0000 mon.b (mon.1) 307 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-55"}]: dispatch 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.229963+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.229963+0000 osd.0 (osd.0) 255 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:28:36.190 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.232451+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:28:36.191 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.232451+0000 osd.0 (osd.0) 256 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:28:36.191 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.651710+0000 mgr.y (mgr.24401) 438 : cluster [DBG] pgmap v650: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:36.191 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.651710+0000 mgr.y (mgr.24401) 438 : cluster [DBG] pgmap v650: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 518 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:36.191 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.858780+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-04-26T16:28:36.191 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:35 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.858780+0000 mon.a (mon.0) 2702 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2024-04-26T16:28:36.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:36.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:36.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:36.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:36.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:36.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:36.596 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:36.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:36.722 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:36.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:36.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/3097944873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c1076b0 msgr2=0x7f376c10f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:36.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- 172.21.15.50:0/3097944873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c1076b0 0x7f376c10f250 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7f3764009fc0 tx=0x7f376400b330 comp rx=0 tx=0).stop 2024-04-26T16:28:36.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/3097944873 shutdown_connections 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- 172.21.15.50:0/3097944873 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c1076b0 0x7f376c10f250 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- 172.21.15.50:0/3097944873 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c102960 0x7f376c102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- 172.21.15.50:0/3097944873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f376c103310 0x7f376c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/3097944873 >> 172.21.15.50:0/3097944873 conn(0x7f376c0792d0 msgr2=0x7f376c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/3097944873 shutdown_connections 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/3097944873 wait complete. 2024-04-26T16:28:36.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:36.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 librados: starting msgr at 2024-04-26T16:28:36.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 librados: starting objecter 2024-04-26T16:28:36.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 Processor -- start 2024-04-26T16:28:36.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- start start 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 librados: setting wanted keys 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 librados: calling monclient init 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f376c102960 0x7f376c19bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 0x7f376c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 0x7f376c1a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3772667700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 0x7f376c1a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 0x7f376c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 0x7f376c19c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:43240/0 (socket says 172.21.15.50:43240) 2024-04-26T16:28:36.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3772667700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 0x7f376c1a6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35060/0 (socket says 172.21.15.50:35060) 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 -- 172.21.15.50:0/2850015279 learned_addr learned my addr 172.21.15.50:0/2850015279 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06ca20 con 0x7f376c103310 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f376c06c8a0 con 0x7f376c102960 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f376c110aa0 con 0x7f376c1076b0 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771e66700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f376c102960 0x7f376c19bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 -- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 msgr2=0x7f376c1a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 0x7f376c1a6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 -- 172.21.15.50:0/2850015279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f376c102960 msgr2=0x7f376c19bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f376c102960 0x7f376c19bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c1a7410 con 0x7f376c103310 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3772667700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 0x7f376c1a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3771665700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 0x7f376c19c4e0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f375c00d950 tx=0x7f375c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3762ffd700 10 client.?.objecter ms_handle_connect 0x7f376c103310 2024-04-26T16:28:36.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.849+0000 7f3762ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:36.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c017070 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f375c009d90 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c1a7700 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c010dc0 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f37740ca700 10 client.30600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c1a7c90 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f37740ca700 1 librados: init done 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c103a60 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f375c01a070 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3758071f80 0x7f3758074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(444..444 src has 87..444) ==== 9348+0+0 (secure 0 0 0) 0x7f375c091250 con 0x7f376c103310 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 10 client.30600.objecter ms_dispatch 0x7f376c001550 osd_map(444..444 src has 87..444) 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 3 client.30600.objecter handle_osd_map got epochs [444,444] > 0 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 3 client.30600.objecter handle_osd_map decoding full epoch 444 2024-04-26T16:28:36.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3771e66700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3758071f80 0x7f3758074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:36.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 20 client.30600.objecter dump_active .. 0 homeless 2024-04-26T16:28:36.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3762ffd700 10 client.30600.objecter ms_handle_connect 0x7f3758071f80 2024-04-26T16:28:36.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.853+0000 7f3771e66700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3758071f80 0x7f3758074440 secure :-1 s=READY pgs=3747 cs=0 l=1 rev1=1 crypto rx=0x7f3768009750 tx=0x7f3768009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:36.861+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f375c05cd20 con 0x7f376c103310 2024-04-26T16:28:36.906 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:28:36.906 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:28:36.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:36.978 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:36.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:37.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:37.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.109+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f376c10ede0 con 0x7f376c103310 2024-04-26T16:28:37.110 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:37.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.153+0000 7f3762ffd700 1 -- 172.21.15.50:0/2850015279 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v445) ==== 113+0+0 (secure 0 0 0) 0x7f375c0609d0 con 0x7f376c103310 2024-04-26T16:28:37.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.157+0000 7f37740ca700 10 librados: watch_flush enter 2024-04-26T16:28:37.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 10 librados: watch_flush exit 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 20 client.30600.objecter shutdown clearing up homeless session... 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 10 client.30600.objecter successfully canceled tick 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3758071f80 msgr2=0x7f3758074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3758071f80 0x7f3758074440 secure :-1 s=READY pgs=3747 cs=0 l=1 rev1=1 crypto rx=0x7f3768009750 tx=0x7f3768009310 comp rx=0 tx=0).stop 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 msgr2=0x7f376c19c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 0x7f376c19c4e0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f375c00d950 tx=0x7f375c00dc60 comp rx=0 tx=0).stop 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 shutdown_connections 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3758071f80 0x7f3758074440 unknown :-1 s=CLOSED pgs=3747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f376c102960 0x7f376c19bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f376c1076b0 0x7f376c1a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 --2- 172.21.15.50:0/2850015279 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f376c103310 0x7f376c19c4e0 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 >> 172.21.15.50:0/2850015279 conn(0x7f376c0792d0 msgr2=0x7f376c103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:37.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 shutdown_connections 2024-04-26T16:28:37.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 -- 172.21.15.50:0/2850015279 wait complete. 2024-04-26T16:28:37.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:37.161+0000 7f37740ca700 1 librados: shutdown 2024-04-26T16:28:37.167 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T16:28:37.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:28:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.921096+0000 osd.1 (osd.1) 223 : cluster [DBG] 5.c scrub starts 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.921096+0000 osd.1 (osd.1) 223 : cluster [DBG] 5.c scrub starts 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.924925+0000 osd.1 (osd.1) 224 : cluster [DBG] 5.c scrub ok 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.924925+0000 osd.1 (osd.1) 224 : cluster [DBG] 5.c scrub ok 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.968994+0000 osd.7 (osd.7) 255 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.968994+0000 osd.7 (osd.7) 255 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.971211+0000 osd.7 (osd.7) 256 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:35.971211+0000 osd.7 (osd.7) 256 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:28:37.207 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.169149+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.169149+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: audit 2024-04-26T16:28:36.182830+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: audit 2024-04-26T16:28:36.182830+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: audit 2024-04-26T16:28:36.183270+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: audit 2024-04-26T16:28:36.183270+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.215464+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.215464+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.216713+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.216713+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: audit 2024-04-26T16:28:37.113899+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:28:37.208 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[29234]: audit 2024-04-26T16:28:37.113899+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.921096+0000 osd.1 (osd.1) 223 : cluster [DBG] 5.c scrub starts 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.921096+0000 osd.1 (osd.1) 223 : cluster [DBG] 5.c scrub starts 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.924925+0000 osd.1 (osd.1) 224 : cluster [DBG] 5.c scrub ok 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.924925+0000 osd.1 (osd.1) 224 : cluster [DBG] 5.c scrub ok 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.968994+0000 osd.7 (osd.7) 255 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.968994+0000 osd.7 (osd.7) 255 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.971211+0000 osd.7 (osd.7) 256 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:35.971211+0000 osd.7 (osd.7) 256 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.169149+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.169149+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-04-26T16:28:37.209 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: audit 2024-04-26T16:28:36.182830+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: audit 2024-04-26T16:28:36.182830+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: audit 2024-04-26T16:28:36.183270+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: audit 2024-04-26T16:28:36.183270+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.215464+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.215464+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.216713+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.216713+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: audit 2024-04-26T16:28:37.113899+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:28:37.210 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:37 smithi050 bash[21463]: audit 2024-04-26T16:28:37.113899+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:28:37.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.921096+0000 osd.1 (osd.1) 223 : cluster [DBG] 5.c scrub starts 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.921096+0000 osd.1 (osd.1) 223 : cluster [DBG] 5.c scrub starts 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.924925+0000 osd.1 (osd.1) 224 : cluster [DBG] 5.c scrub ok 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.924925+0000 osd.1 (osd.1) 224 : cluster [DBG] 5.c scrub ok 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.968994+0000 osd.7 (osd.7) 255 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.968994+0000 osd.7 (osd.7) 255 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.971211+0000 osd.7 (osd.7) 256 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:35.971211+0000 osd.7 (osd.7) 256 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.169149+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.169149+0000 mon.a (mon.0) 2703 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: audit 2024-04-26T16:28:36.182830+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: audit 2024-04-26T16:28:36.182830+0000 mon.a (mon.0) 2704 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.223 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: audit 2024-04-26T16:28:36.183270+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: audit 2024-04-26T16:28:36.183270+0000 mon.b (mon.1) 308 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.215464+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.215464+0000 osd.3 (osd.3) 265 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.216713+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.216713+0000 osd.3 (osd.3) 266 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: audit 2024-04-26T16:28:37.113899+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:28:37.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:37 smithi190 bash[23868]: audit 2024-04-26T16:28:37.113899+0000 mon.a (mon.0) 2705 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:28:37.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:37.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:37.367 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:37.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:37.495 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:37.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:37.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:37.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:37.759 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:37.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:37.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:37.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:37.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:38.023 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:38.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:38.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:38.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:38.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.861562+0000 osd.2 (osd.2) 203 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:28:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.861562+0000 osd.2 (osd.2) 203 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:28:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.863587+0000 osd.2 (osd.2) 204 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:28:38.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:36.863587+0000 osd.2 (osd.2) 204 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:36.875399+0000 mgr.y (mgr.24401) 439 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:36.875399+0000 mgr.y (mgr.24401) 439 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.158505+0000 mon.a (mon.0) 2706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.158505+0000 mon.a (mon.0) 2706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.158742+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.158742+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.166601+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.166601+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.206123+0000 osd.3 (osd.3) 267 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.206123+0000 osd.3 (osd.3) 267 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.207597+0000 osd.3 (osd.3) 268 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.207597+0000 osd.3 (osd.3) 268 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:28:38.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.234996+0000 mon.a (mon.0) 2709 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.234996+0000 mon.a (mon.0) 2709 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.235803+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.235803+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.236979+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.236979+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.237966+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: audit 2024-04-26T16:28:37.237966+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.652347+0000 mgr.y (mgr.24401) 440 : cluster [DBG] pgmap v654: 228 pgs: 25 creating+peering, 7 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:38.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:38 smithi190 bash[23868]: cluster 2024-04-26T16:28:37.652347+0000 mgr.y (mgr.24401) 440 : cluster [DBG] pgmap v654: 228 pgs: 25 creating+peering, 7 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:38.386 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:38.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.861562+0000 osd.2 (osd.2) 203 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:28:38.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.861562+0000 osd.2 (osd.2) 203 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:28:38.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.863587+0000 osd.2 (osd.2) 204 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:28:38.398 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:36.863587+0000 osd.2 (osd.2) 204 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:36.875399+0000 mgr.y (mgr.24401) 439 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:36.875399+0000 mgr.y (mgr.24401) 439 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.158505+0000 mon.a (mon.0) 2706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.158505+0000 mon.a (mon.0) 2706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.158742+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.158742+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.166601+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.166601+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.206123+0000 osd.3 (osd.3) 267 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.206123+0000 osd.3 (osd.3) 267 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.207597+0000 osd.3 (osd.3) 268 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.207597+0000 osd.3 (osd.3) 268 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.234996+0000 mon.a (mon.0) 2709 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.234996+0000 mon.a (mon.0) 2709 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.235803+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.235803+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.861562+0000 osd.2 (osd.2) 203 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.861562+0000 osd.2 (osd.2) 203 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.863587+0000 osd.2 (osd.2) 204 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:36.863587+0000 osd.2 (osd.2) 204 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:36.875399+0000 mgr.y (mgr.24401) 439 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:36.875399+0000 mgr.y (mgr.24401) 439 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:38.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.158505+0000 mon.a (mon.0) 2706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.158505+0000 mon.a (mon.0) 2706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.158742+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.158742+0000 mon.a (mon.0) 2707 : audit [INF] from='client.? 172.21.15.50:0/2850015279' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.166601+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.166601+0000 mon.a (mon.0) 2708 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.206123+0000 osd.3 (osd.3) 267 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.206123+0000 osd.3 (osd.3) 267 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.207597+0000 osd.3 (osd.3) 268 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.207597+0000 osd.3 (osd.3) 268 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:28:38.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.234996+0000 mon.a (mon.0) 2709 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.234996+0000 mon.a (mon.0) 2709 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.235803+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.235803+0000 mon.a (mon.0) 2710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.236979+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.236979+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.237966+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: audit 2024-04-26T16:28:37.237966+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.652347+0000 mgr.y (mgr.24401) 440 : cluster [DBG] pgmap v654: 228 pgs: 25 creating+peering, 7 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:38.402 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[21463]: cluster 2024-04-26T16:28:37.652347+0000 mgr.y (mgr.24401) 440 : cluster [DBG] pgmap v654: 228 pgs: 25 creating+peering, 7 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:38.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.236979+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.236979+0000 mon.b (mon.1) 309 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:38.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.237966+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: audit 2024-04-26T16:28:37.237966+0000 mon.b (mon.1) 310 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-57"}]: dispatch 2024-04-26T16:28:38.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.652347+0000 mgr.y (mgr.24401) 440 : cluster [DBG] pgmap v654: 228 pgs: 25 creating+peering, 7 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:38.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:38 smithi050 bash[29234]: cluster 2024-04-26T16:28:37.652347+0000 mgr.y (mgr.24401) 440 : cluster [DBG] pgmap v654: 228 pgs: 25 creating+peering, 7 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:38.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:38.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:38.543 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:38.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-26T16:28:38.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:38.673 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:38.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:38.803 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:38.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:38.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:38.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:39.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:39.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:39.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:39.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:39.243 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-26T16:28:39.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:39.351 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:39.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[21463]: cluster 2024-04-26T16:28:38.009386+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:28:39.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[21463]: cluster 2024-04-26T16:28:38.009386+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[21463]: cluster 2024-04-26T16:28:38.011956+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[21463]: cluster 2024-04-26T16:28:38.011956+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[21463]: cluster 2024-04-26T16:28:38.187316+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[21463]: cluster 2024-04-26T16:28:38.187316+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[29234]: cluster 2024-04-26T16:28:38.009386+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[29234]: cluster 2024-04-26T16:28:38.009386+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[29234]: cluster 2024-04-26T16:28:38.011956+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[29234]: cluster 2024-04-26T16:28:38.011956+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[29234]: cluster 2024-04-26T16:28:38.187316+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-04-26T16:28:39.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:39 smithi050 bash[29234]: cluster 2024-04-26T16:28:38.187316+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-04-26T16:28:39.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:39.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:39 smithi190 bash[23868]: cluster 2024-04-26T16:28:38.009386+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:28:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:39 smithi190 bash[23868]: cluster 2024-04-26T16:28:38.009386+0000 osd.7 (osd.7) 257 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:28:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:39 smithi190 bash[23868]: cluster 2024-04-26T16:28:38.011956+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:28:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:39 smithi190 bash[23868]: cluster 2024-04-26T16:28:38.011956+0000 osd.7 (osd.7) 258 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:28:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:39 smithi190 bash[23868]: cluster 2024-04-26T16:28:38.187316+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-04-26T16:28:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:39 smithi190 bash[23868]: cluster 2024-04-26T16:28:38.187316+0000 mon.a (mon.0) 2711 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2024-04-26T16:28:39.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:39.606 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:39.653 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:39.654 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:39.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:39.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:39.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:39.865 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:39.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:39.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:39.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:39.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:40.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:40.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:40.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:40.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.160749+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:28:40.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.160749+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.164757+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.164757+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.165214+0000 osd.4 (osd.4) 228 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.165214+0000 osd.4 (osd.4) 228 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.169110+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.169110+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.200722+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.200722+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: audit 2024-04-26T16:28:39.210320+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: audit 2024-04-26T16:28:39.210320+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: audit 2024-04-26T16:28:39.211565+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: audit 2024-04-26T16:28:39.211565+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.240 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.281285+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.281285+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.282546+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.282546+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.653171+0000 mgr.y (mgr.24401) 441 : cluster [DBG] pgmap v657: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.653171+0000 mgr.y (mgr.24401) 441 : cluster [DBG] pgmap v657: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.160749+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.160749+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.164757+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.164757+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.165214+0000 osd.4 (osd.4) 228 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.165214+0000 osd.4 (osd.4) 228 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:40.241 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.169110+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.169110+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.200722+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.200722+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: audit 2024-04-26T16:28:39.210320+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: audit 2024-04-26T16:28:39.210320+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: audit 2024-04-26T16:28:39.211565+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: audit 2024-04-26T16:28:39.211565+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.281285+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.281285+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.282546+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.282546+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.653171+0000 mgr.y (mgr.24401) 441 : cluster [DBG] pgmap v657: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:40.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:40 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.653171+0000 mgr.y (mgr.24401) 441 : cluster [DBG] pgmap v657: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:40.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:40.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:40.382 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:40.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:40.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.160749+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.160749+0000 osd.3 (osd.3) 269 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.164757+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.164757+0000 osd.4 (osd.4) 227 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.165214+0000 osd.4 (osd.4) 228 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.165214+0000 osd.4 (osd.4) 228 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.169110+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.169110+0000 osd.3 (osd.3) 270 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:28:40.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.200722+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.200722+0000 mon.a (mon.0) 2712 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: audit 2024-04-26T16:28:39.210320+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: audit 2024-04-26T16:28:39.210320+0000 mon.a (mon.0) 2713 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: audit 2024-04-26T16:28:39.211565+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: audit 2024-04-26T16:28:39.211565+0000 mon.b (mon.1) 311 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.281285+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.281285+0000 osd.0 (osd.0) 257 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.282546+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.282546+0000 osd.0 (osd.0) 258 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.653171+0000 mgr.y (mgr.24401) 441 : cluster [DBG] pgmap v657: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:40.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:40 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.653171+0000 mgr.y (mgr.24401) 441 : cluster [DBG] pgmap v657: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 519 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-04-26T16:28:40.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:40.641 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:40.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.774 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:40.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:40.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:40.902 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:40.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:40.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:41.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:41.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:41.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:41.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:41.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:41.297 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.421855+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.421855+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.423244+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.423244+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.828885+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.828885+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.831337+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:39.831337+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.149428+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.149428+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.151821+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:28:41.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.151821+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.177827+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.b scrub starts 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.177827+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.b scrub starts 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.179441+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.b scrub ok 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.179441+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.b scrub ok 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.196402+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.196402+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.197815+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.197815+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.202410+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.202410+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.226717+0000 mon.b (mon.1) 312 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.226717+0000 mon.b (mon.1) 312 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.284123+0000 osd.0 (osd.0) 259 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:28:41.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.284123+0000 osd.0 (osd.0) 259 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.421855+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.421855+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.423244+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.423244+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.828885+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.828885+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.831337+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:39.831337+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.149428+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:28:41.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.149428+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.151821+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.151821+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.177827+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.b scrub starts 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.177827+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.b scrub starts 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.179441+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.b scrub ok 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.179441+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.b scrub ok 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.196402+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.196402+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.197815+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.197815+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.202410+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.202410+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.226717+0000 mon.b (mon.1) 312 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.226717+0000 mon.b (mon.1) 312 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:41.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.284123+0000 osd.0 (osd.0) 259 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.284123+0000 osd.0 (osd.0) 259 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.285513+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.285513+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.290765+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.290765+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.291878+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.291878+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.292643+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.292643+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.293852+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[21463]: audit 2024-04-26T16:28:40.293852+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.285513+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.285513+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.290765+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.290765+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.291878+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.291878+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.292643+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.292643+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.293852+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:41 smithi050 bash[29234]: audit 2024-04-26T16:28:40.293852+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:41.428 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:41.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:41.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:41.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.421855+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.421855+0000 osd.4 (osd.4) 229 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.423244+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.423244+0000 osd.4 (osd.4) 230 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.828885+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.828885+0000 osd.1 (osd.1) 225 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.831337+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:39.831337+0000 osd.1 (osd.1) 226 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.149428+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.149428+0000 osd.6 (osd.6) 5 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.151821+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.151821+0000 osd.6 (osd.6) 6 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.177827+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.b scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.177827+0000 osd.3 (osd.3) 271 : cluster [DBG] 6.b scrub starts 2024-04-26T16:28:41.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.179441+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.b scrub ok 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.179441+0000 osd.3 (osd.3) 272 : cluster [DBG] 6.b scrub ok 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.196402+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.196402+0000 mon.a (mon.0) 2714 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.197815+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.197815+0000 mon.a (mon.0) 2715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.202410+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.202410+0000 mon.a (mon.0) 2716 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.226717+0000 mon.b (mon.1) 312 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.226717+0000 mon.b (mon.1) 312 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.284123+0000 osd.0 (osd.0) 259 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.284123+0000 osd.0 (osd.0) 259 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.285513+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.285513+0000 osd.0 (osd.0) 260 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:28:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.290765+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.290765+0000 mon.a (mon.0) 2717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.291878+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.291878+0000 mon.a (mon.0) 2718 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.292643+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.292643+0000 mon.b (mon.1) 313 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.293852+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:41 smithi190 bash[23868]: audit 2024-04-26T16:28:40.293852+0000 mon.b (mon.1) 314 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-59"}]: dispatch 2024-04-26T16:28:41.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:41.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:41.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:41.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:41.810 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:41.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:41.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:41.942 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:41.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:41.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:42.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:42.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:42.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:42.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:42.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.453882+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:28:42.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.453882+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:28:42.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.455417+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:28:42.305 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.455417+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.783591+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.783591+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.784931+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.784931+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.943299+0000 osd.2 (osd.2) 205 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.943299+0000 osd.2 (osd.2) 205 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.945774+0000 osd.2 (osd.2) 206 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:40.945774+0000 osd.2 (osd.2) 206 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.165341+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.165341+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.168271+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.168271+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.244750+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-04-26T16:28:42.306 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.244750+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.654234+0000 mgr.y (mgr.24401) 442 : cluster [DBG] pgmap v660: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.654234+0000 mgr.y (mgr.24401) 442 : cluster [DBG] pgmap v660: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: audit 2024-04-26T16:28:41.815260+0000 mon.a (mon.0) 2720 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: audit 2024-04-26T16:28:41.815260+0000 mon.a (mon.0) 2720 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: audit 2024-04-26T16:28:41.817093+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[21463]: audit 2024-04-26T16:28:41.817093+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.453882+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.453882+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.455417+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:28:42.307 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.455417+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.783591+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.783591+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.784931+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.784931+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.943299+0000 osd.2 (osd.2) 205 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.943299+0000 osd.2 (osd.2) 205 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.945774+0000 osd.2 (osd.2) 206 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:40.945774+0000 osd.2 (osd.2) 206 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.165341+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.165341+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.168271+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.168271+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.244750+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-04-26T16:28:42.308 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.244750+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-04-26T16:28:42.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.654234+0000 mgr.y (mgr.24401) 442 : cluster [DBG] pgmap v660: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:42.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.654234+0000 mgr.y (mgr.24401) 442 : cluster [DBG] pgmap v660: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:42.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: audit 2024-04-26T16:28:41.815260+0000 mon.a (mon.0) 2720 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:42.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: audit 2024-04-26T16:28:41.815260+0000 mon.a (mon.0) 2720 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:42.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: audit 2024-04-26T16:28:41.817093+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:42.309 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:42 smithi050 bash[29234]: audit 2024-04-26T16:28:41.817093+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:42.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:42.340 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:42.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:42.471 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:42.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.453882+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:28:42.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.453882+0000 osd.4 (osd.4) 231 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:28:42.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.455417+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:28:42.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.455417+0000 osd.4 (osd.4) 232 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:28:42.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.783591+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:28:42.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.783591+0000 osd.1 (osd.1) 227 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.784931+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.784931+0000 osd.1 (osd.1) 228 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.943299+0000 osd.2 (osd.2) 205 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.943299+0000 osd.2 (osd.2) 205 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.945774+0000 osd.2 (osd.2) 206 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:40.945774+0000 osd.2 (osd.2) 206 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.165341+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.165341+0000 osd.6 (osd.6) 7 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.168271+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.168271+0000 osd.6 (osd.6) 8 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.244750+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.244750+0000 mon.a (mon.0) 2719 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.654234+0000 mgr.y (mgr.24401) 442 : cluster [DBG] pgmap v660: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:42.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.654234+0000 mgr.y (mgr.24401) 442 : cluster [DBG] pgmap v660: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:42.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: audit 2024-04-26T16:28:41.815260+0000 mon.a (mon.0) 2720 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:42.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: audit 2024-04-26T16:28:41.815260+0000 mon.a (mon.0) 2720 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:42.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: audit 2024-04-26T16:28:41.817093+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:42.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:42 smithi190 bash[23868]: audit 2024-04-26T16:28:41.817093+0000 mon.c (mon.2) 615 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:42.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:42.602 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:42.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:42.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:42.734 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:42.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:42.865 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:42.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:42.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:42.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:42.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:43.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:43.130 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:43.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:43.276 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:43.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:43.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:43.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.745690+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.745690+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.747534+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.747534+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.969582+0000 osd.2 (osd.2) 207 : cluster [DBG] 11.d scrub starts 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.969582+0000 osd.2 (osd.2) 207 : cluster [DBG] 11.d scrub starts 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.972223+0000 osd.2 (osd.2) 208 : cluster [DBG] 11.d scrub ok 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:41.972223+0000 osd.2 (osd.2) 208 : cluster [DBG] 11.d scrub ok 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.252550+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.252550+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: audit 2024-04-26T16:28:42.268686+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: audit 2024-04-26T16:28:42.268686+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: audit 2024-04-26T16:28:42.269876+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: audit 2024-04-26T16:28:42.269876+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.413053+0000 osd.1 (osd.1) 231 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:43.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.413053+0000 osd.1 (osd.1) 231 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:43.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.413428+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:43.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:43 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.413428+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:43.406 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:43.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:43.521 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.745690+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.745690+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.747534+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.747534+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.969582+0000 osd.2 (osd.2) 207 : cluster [DBG] 11.d scrub starts 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.969582+0000 osd.2 (osd.2) 207 : cluster [DBG] 11.d scrub starts 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.972223+0000 osd.2 (osd.2) 208 : cluster [DBG] 11.d scrub ok 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:41.972223+0000 osd.2 (osd.2) 208 : cluster [DBG] 11.d scrub ok 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.252550+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.252550+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: audit 2024-04-26T16:28:42.268686+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: audit 2024-04-26T16:28:42.268686+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: audit 2024-04-26T16:28:42.269876+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: audit 2024-04-26T16:28:42.269876+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.413053+0000 osd.1 (osd.1) 231 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.413053+0000 osd.1 (osd.1) 231 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.413428+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.413428+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.745690+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.745690+0000 osd.1 (osd.1) 229 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.747534+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.747534+0000 osd.1 (osd.1) 230 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.969582+0000 osd.2 (osd.2) 207 : cluster [DBG] 11.d scrub starts 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.969582+0000 osd.2 (osd.2) 207 : cluster [DBG] 11.d scrub starts 2024-04-26T16:28:43.523 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.972223+0000 osd.2 (osd.2) 208 : cluster [DBG] 11.d scrub ok 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:41.972223+0000 osd.2 (osd.2) 208 : cluster [DBG] 11.d scrub ok 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.252550+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.252550+0000 mon.a (mon.0) 2721 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: audit 2024-04-26T16:28:42.268686+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: audit 2024-04-26T16:28:42.268686+0000 mon.a (mon.0) 2722 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: audit 2024-04-26T16:28:42.269876+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: audit 2024-04-26T16:28:42.269876+0000 mon.b (mon.1) 315 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.413053+0000 osd.1 (osd.1) 231 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.413053+0000 osd.1 (osd.1) 231 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.413428+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:43.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:43 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.413428+0000 osd.1 (osd.1) 232 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:43.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:43.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:43.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:43.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:43.802 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:43.806 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:43.807 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:43.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:43.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:43.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:44.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:44.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:44.194 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:44.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.331 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:44.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:44.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.794388+0000 osd.1 (osd.1) 233 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:28:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.794388+0000 osd.1 (osd.1) 233 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:28:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.796765+0000 osd.1 (osd.1) 234 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:28:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:42.796765+0000 osd.1 (osd.1) 234 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:28:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.123376+0000 osd.3 (osd.3) 273 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:28:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.123376+0000 osd.3 (osd.3) 273 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.125891+0000 osd.3 (osd.3) 274 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.125891+0000 osd.3 (osd.3) 274 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.249175+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.249175+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.259844+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.259844+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.274333+0000 mon.b (mon.1) 316 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.274333+0000 mon.b (mon.1) 316 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.300027+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.300027+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.301630+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: audit 2024-04-26T16:28:43.301630+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.353507+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.353507+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.355026+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.355026+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.655068+0000 mgr.y (mgr.24401) 443 : cluster [DBG] pgmap v663: 228 pgs: 6 creating+peering, 26 unknown, 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:44.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:44 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.655068+0000 mgr.y (mgr.24401) 443 : cluster [DBG] pgmap v663: 228 pgs: 6 creating+peering, 26 unknown, 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:44.576 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:28:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:28:44.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.794388+0000 osd.1 (osd.1) 233 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:28:44.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.794388+0000 osd.1 (osd.1) 233 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:28:44.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.796765+0000 osd.1 (osd.1) 234 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:28:44.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:42.796765+0000 osd.1 (osd.1) 234 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.123376+0000 osd.3 (osd.3) 273 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.123376+0000 osd.3 (osd.3) 273 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.125891+0000 osd.3 (osd.3) 274 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.125891+0000 osd.3 (osd.3) 274 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.249175+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.249175+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.259844+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.259844+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-04-26T16:28:44.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.274333+0000 mon.b (mon.1) 316 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.274333+0000 mon.b (mon.1) 316 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.300027+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.300027+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.301630+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: audit 2024-04-26T16:28:43.301630+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.353507+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.353507+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.355026+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:28:44.579 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.355026+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.794388+0000 osd.1 (osd.1) 233 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.794388+0000 osd.1 (osd.1) 233 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.796765+0000 osd.1 (osd.1) 234 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:42.796765+0000 osd.1 (osd.1) 234 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.123376+0000 osd.3 (osd.3) 273 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.123376+0000 osd.3 (osd.3) 273 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.125891+0000 osd.3 (osd.3) 274 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.125891+0000 osd.3 (osd.3) 274 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.249175+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.249175+0000 mon.a (mon.0) 2723 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.259844+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.259844+0000 mon.a (mon.0) 2724 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.274333+0000 mon.b (mon.1) 316 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:44.580 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.274333+0000 mon.b (mon.1) 316 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:44.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.300027+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.300027+0000 mon.a (mon.0) 2725 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.301630+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: audit 2024-04-26T16:28:43.301630+0000 mon.b (mon.1) 317 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:44.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.353507+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:28:44.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.353507+0000 osd.0 (osd.0) 261 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:28:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.355026+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:28:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.355026+0000 osd.0 (osd.0) 262 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:28:44.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.655068+0000 mgr.y (mgr.24401) 443 : cluster [DBG] pgmap v663: 228 pgs: 6 creating+peering, 26 unknown, 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:44.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.655068+0000 mgr.y (mgr.24401) 443 : cluster [DBG] pgmap v663: 228 pgs: 6 creating+peering, 26 unknown, 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.655068+0000 mgr.y (mgr.24401) 443 : cluster [DBG] pgmap v663: 228 pgs: 6 creating+peering, 26 unknown, 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:44.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:44 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.655068+0000 mgr.y (mgr.24401) 443 : cluster [DBG] pgmap v663: 228 pgs: 6 creating+peering, 26 unknown, 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:28:44.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:44.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:44.723 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:44.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:44.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:44.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:44.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:45.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:45.117 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:45.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:45.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:45.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:45.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.822346+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:28:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.822346+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:28:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.823723+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:28:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.823723+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:28:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.974841+0000 osd.7 (osd.7) 259 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.974841+0000 osd.7 (osd.7) 259 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.978715+0000 osd.7 (osd.7) 260 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:43.978715+0000 osd.7 (osd.7) 260 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.301978+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.301978+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.309454+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.309454+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.372378+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.372378+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.373863+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.373863+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.374135+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.374135+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.375744+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[21463]: audit 2024-04-26T16:28:44.375744+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.822346+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.822346+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.823723+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.823723+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.974841+0000 osd.7 (osd.7) 259 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.974841+0000 osd.7 (osd.7) 259 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.978715+0000 osd.7 (osd.7) 260 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:43.978715+0000 osd.7 (osd.7) 260 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:28:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.301978+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.301978+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.309454+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.309454+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.372378+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.372378+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.373863+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.373863+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.374135+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.374135+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.375744+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:45 smithi050 bash[29234]: audit 2024-04-26T16:28:44.375744+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:45.489 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.822346+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.822346+0000 osd.1 (osd.1) 235 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.823723+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.823723+0000 osd.1 (osd.1) 236 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.974841+0000 osd.7 (osd.7) 259 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.974841+0000 osd.7 (osd.7) 259 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.978715+0000 osd.7 (osd.7) 260 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:43.978715+0000 osd.7 (osd.7) 260 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.301978+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.301978+0000 mon.a (mon.0) 2726 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.309454+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.309454+0000 mon.a (mon.0) 2727 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.372378+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.490 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.372378+0000 mon.a (mon.0) 2728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.373863+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.373863+0000 mon.a (mon.0) 2729 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.374135+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.374135+0000 mon.b (mon.1) 318 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:28:45.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.375744+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.491 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:45 smithi190 bash[23868]: audit 2024-04-26T16:28:44.375744+0000 mon.b (mon.1) 319 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-61"}]: dispatch 2024-04-26T16:28:45.508 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:45.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:45.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:45.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:45.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:45.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:45.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:45.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:46.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:46.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:46.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:46.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:46.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:46.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:46.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.500019+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.500019+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.502684+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.502684+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.961635+0000 osd.2 (osd.2) 209 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.961635+0000 osd.2 (osd.2) 209 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.964253+0000 osd.2 (osd.2) 210 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.964253+0000 osd.2 (osd.2) 210 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.981503+0000 osd.7 (osd.7) 261 : cluster [DBG] 3.d scrub starts 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.981503+0000 osd.7 (osd.7) 261 : cluster [DBG] 3.d scrub starts 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.986026+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.d scrub ok 2024-04-26T16:28:46.404 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:44.986026+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.d scrub ok 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.311500+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.311500+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.450163+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.450163+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.456933+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.456933+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.502869+0000 osd.4 (osd.4) 235 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.502869+0000 osd.4 (osd.4) 235 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.509861+0000 osd.4 (osd.4) 236 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.509861+0000 osd.4 (osd.4) 236 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.655856+0000 mgr.y (mgr.24401) 444 : cluster [DBG] pgmap v666: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:45.655856+0000 mgr.y (mgr.24401) 444 : cluster [DBG] pgmap v666: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.156703+0000 mon.a (mon.0) 2731 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.156703+0000 mon.a (mon.0) 2731 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:46.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.164870+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.164870+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: audit 2024-04-26T16:28:46.171743+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: audit 2024-04-26T16:28:46.171743+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: audit 2024-04-26T16:28:46.171744+0000 mon.a (mon.0) 2733 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[29234]: audit 2024-04-26T16:28:46.171744+0000 mon.a (mon.0) 2733 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.500019+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:28:46.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.500019+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.502684+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.502684+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.961635+0000 osd.2 (osd.2) 209 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.961635+0000 osd.2 (osd.2) 209 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.964253+0000 osd.2 (osd.2) 210 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.964253+0000 osd.2 (osd.2) 210 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.981503+0000 osd.7 (osd.7) 261 : cluster [DBG] 3.d scrub starts 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.981503+0000 osd.7 (osd.7) 261 : cluster [DBG] 3.d scrub starts 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.986026+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.d scrub ok 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:44.986026+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.d scrub ok 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.311500+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.311500+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.450163+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:28:46.407 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.450163+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.456933+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.456933+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.502869+0000 osd.4 (osd.4) 235 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.502869+0000 osd.4 (osd.4) 235 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.509861+0000 osd.4 (osd.4) 236 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.509861+0000 osd.4 (osd.4) 236 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.655856+0000 mgr.y (mgr.24401) 444 : cluster [DBG] pgmap v666: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:45.655856+0000 mgr.y (mgr.24401) 444 : cluster [DBG] pgmap v666: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.156703+0000 mon.a (mon.0) 2731 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.156703+0000 mon.a (mon.0) 2731 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.164870+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.164870+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: audit 2024-04-26T16:28:46.171743+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: audit 2024-04-26T16:28:46.171743+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: audit 2024-04-26T16:28:46.171744+0000 mon.a (mon.0) 2733 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:46 smithi050 bash[21463]: audit 2024-04-26T16:28:46.171744+0000 mon.a (mon.0) 2733 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:46.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:46.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:46.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 -- 172.21.15.50:0/3753479575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc06bad0 msgr2=0x7f3bfc1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/3753479575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc1102e0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b0a0 tx=0x7f3bf400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 -- 172.21.15.50:0/3753479575 shutdown_connections 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/3753479575 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc1102e0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/3753479575 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc1094a0 0x7f3bfc109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/3753479575 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06b130 0x7f3bfc06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 -- 172.21.15.50:0/3753479575 >> 172.21.15.50:0/3753479575 conn(0x7f3bfc0792d0 msgr2=0x7f3bfc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:46.561 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.561+0000 7f3c03a9e700 1 -- 172.21.15.50:0/3753479575 shutdown_connections 2024-04-26T16:28:46.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- 172.21.15.50:0/3753479575 wait complete. 2024-04-26T16:28:46.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 librados: starting msgr at 2024-04-26T16:28:46.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 librados: starting objecter 2024-04-26T16:28:46.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 Processor -- start 2024-04-26T16:28:46.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- start start 2024-04-26T16:28:46.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 librados: setting wanted keys 2024-04-26T16:28:46.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 librados: calling monclient init 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 0x7f3bfc19b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc19b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 0x7f3bfc19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc112ad0 con 0x7f3bfc1094a0 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3bfc112950 con 0x7f3bfc06bad0 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3bfc112c50 con 0x7f3bfc06b130 2024-04-26T16:28:46.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 0x7f3bfc19b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0203b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 0x7f3bfc19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 0x7f3bfc19b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:32936/0 (socket says 172.21.15.50:32936) 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0203b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 0x7f3bfc19fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46886/0 (socket says 172.21.15.50:46886) 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 -- 172.21.15.50:0/2762247130 learned_addr learned my addr 172.21.15.50:0/2762247130 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c01039700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc19b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 -- 172.21.15.50:0/2762247130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06bad0 msgr2=0x7f3bfc19b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc19b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 -- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 msgr2=0x7f3bfc19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 0x7f3bfc19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 -- 172.21.15.50:0/2762247130 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bfc1a0300 con 0x7f3bfc06b130 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0203b700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 0x7f3bfc19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c01039700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc19b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c0183a700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 0x7f3bfc19b2f0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800eb30 tx=0x7f3bf800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3bf2ffd700 10 client.?.objecter ms_handle_connect 0x7f3bfc06b130 2024-04-26T16:28:46.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3bf2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:46.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf800cc90 con 0x7f3bfc06b130 2024-04-26T16:28:46.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3bf800ce30 con 0x7f3bfc06b130 2024-04-26T16:28:46.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bfc1a05f0 con 0x7f3bfc06b130 2024-04-26T16:28:46.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8003d80 con 0x7f3bfc06b130 2024-04-26T16:28:46.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 10 client.41219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:46.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bfc1a7e70 con 0x7f3bfc06b130 2024-04-26T16:28:46.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 librados: init done 2024-04-26T16:28:46.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bfc06a110 con 0x7f3bfc06b130 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.565+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3bf8003f20 con 0x7f3bfc06b130 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3be8071f80 0x7f3be8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(454..454 src has 87..454) ==== 9364+0+0 (secure 0 0 0) 0x7f3bf8091c70 con 0x7f3bfc06b130 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 10 client.41219.objecter ms_dispatch 0x7f3bfc001550 osd_map(454..454 src has 87..454) 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 3 client.41219.objecter handle_osd_map got epochs [454,454] > 0 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 3 client.41219.objecter handle_osd_map decoding full epoch 454 2024-04-26T16:28:46.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3c01039700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3be8071f80 0x7f3be8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 20 client.41219.objecter dump_active .. 0 homeless 2024-04-26T16:28:46.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3c01039700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3be8071f80 0x7f3be8074440 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7f3bec005fd0 tx=0x7f3bec005e20 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.569+0000 7f3bf2ffd700 10 client.41219.objecter ms_handle_connect 0x7f3be8071f80 2024-04-26T16:28:46.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.573+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3bf805d5d0 con 0x7f3bfc06b130 2024-04-26T16:28:46.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.500019+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:28:46.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.500019+0000 osd.4 (osd.4) 233 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:28:46.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.502684+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.502684+0000 osd.4 (osd.4) 234 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.961635+0000 osd.2 (osd.2) 209 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.961635+0000 osd.2 (osd.2) 209 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.964253+0000 osd.2 (osd.2) 210 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.964253+0000 osd.2 (osd.2) 210 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.981503+0000 osd.7 (osd.7) 261 : cluster [DBG] 3.d scrub starts 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.981503+0000 osd.7 (osd.7) 261 : cluster [DBG] 3.d scrub starts 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.986026+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.d scrub ok 2024-04-26T16:28:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:44.986026+0000 osd.7 (osd.7) 262 : cluster [DBG] 3.d scrub ok 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.311500+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.311500+0000 mon.a (mon.0) 2730 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.450163+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.450163+0000 osd.0 (osd.0) 263 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.456933+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.456933+0000 osd.0 (osd.0) 264 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.502869+0000 osd.4 (osd.4) 235 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.502869+0000 osd.4 (osd.4) 235 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:28:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.509861+0000 osd.4 (osd.4) 236 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.509861+0000 osd.4 (osd.4) 236 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.655856+0000 mgr.y (mgr.24401) 444 : cluster [DBG] pgmap v666: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:45.655856+0000 mgr.y (mgr.24401) 444 : cluster [DBG] pgmap v666: 196 pgs: 196 active+clean; 8.4 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.156703+0000 mon.a (mon.0) 2731 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.156703+0000 mon.a (mon.0) 2731 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.164870+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.164870+0000 mon.a (mon.0) 2732 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: audit 2024-04-26T16:28:46.171743+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: audit 2024-04-26T16:28:46.171743+0000 mon.b (mon.1) 320 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: audit 2024-04-26T16:28:46.171744+0000 mon.a (mon.0) 2733 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[23868]: audit 2024-04-26T16:28:46.171744+0000 mon.a (mon.0) 2733 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:28:46.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:46.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:46.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 -- 172.21.15.50:0/3464540633 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8109560 msgr2=0x7f96e8109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 --2- 172.21.15.50:0/3464540633 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8109560 0x7f96e8109940 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009990 tx=0x7f96dc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 -- 172.21.15.50:0/3464540633 shutdown_connections 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 --2- 172.21.15.50:0/3464540633 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96e8103e80 0x7f96e8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 --2- 172.21.15.50:0/3464540633 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8109560 0x7f96e8109940 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 --2- 172.21.15.50:0/3464540633 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96e81034c0 0x7f96e8103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 -- 172.21.15.50:0/3464540633 >> 172.21.15.50:0/3464540633 conn(0x7f96e807a5d0 msgr2=0x7f96e807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 -- 172.21.15.50:0/3464540633 shutdown_connections 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 -- 172.21.15.50:0/3464540633 wait complete. 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 librados: starting msgr at 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 librados: starting objecter 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 Processor -- start 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 -- start start 2024-04-26T16:28:46.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 librados: setting wanted keys 2024-04-26T16:28:46.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.725+0000 7f96eed2f700 1 librados: calling monclient init 2024-04-26T16:28:46.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96e81034c0 0x7f96e819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 0x7f96e819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96e8109560 0x7f96e819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f96e8112b20 con 0x7f96e8109560 2024-04-26T16:28:46.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f96e81129a0 con 0x7f96e81034c0 2024-04-26T16:28:46.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f96e8112ca0 con 0x7f96e8103e80 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 0x7f96e819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 0x7f96e819b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:32950/0 (socket says 172.21.15.50:32950) 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 -- 172.21.15.50:0/2552253179 learned_addr learned my addr 172.21.15.50:0/2552253179 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96e8109560 0x7f96e819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96e81034c0 0x7f96e819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96e81034c0 msgr2=0x7f96e819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96e81034c0 0x7f96e819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96e8109560 msgr2=0x7f96e819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96e8109560 0x7f96e819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e81a0330 con 0x7f96e8103e80 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f96e8109560 0x7f96e819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 0x7f96e819b860 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f96d400eb30 tx=0x7f96d400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 10 client.?.objecter ms_handle_connect 0x7f96e8103e80 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:46.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d400cc90 con 0x7f96e8103e80 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e81a0620 con 0x7f96e8103e80 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f96d400ce30 con 0x7f96e8103e80 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4003d80 con 0x7f96e8103e80 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 10 client.41225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e81a7ea0 con 0x7f96e8103e80 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 librados: init done 2024-04-26T16:28:46.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f96e806a110 con 0x7f96e8103e80 2024-04-26T16:28:46.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f96d4003f20 con 0x7f96e8103e80 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f96d8071f80 0x7f96d8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(454..454 src has 87..454) ==== 9364+0+0 (secure 0 0 0) 0x7f96d4091cd0 con 0x7f96e8103e80 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 osd_map(454..454 src has 87..454) 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 3 client.41225.objecter handle_osd_map got epochs [454,454] > 0 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96e5ffb700 3 client.41225.objecter handle_osd_map decoding full epoch 454 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.729+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f96d8071f80 0x7f96d8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 20 client.41225.objecter dump_active .. 0 homeless 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v454) ==== 54+0+16 (secure 0 0 0) 0x7f96d4097320 con 0x7f96e8103e80 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 10 client.41225.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 1 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f96e8064030 0x7f96e8064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f96d8071f80 0x7f96d8074440 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f96dc006010 tx=0x7f96dc01a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 1 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 0 1 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 4 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 10 client.41225.objecter _send_command 1 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96d8071f80 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96e81ab200 con 0x7f96e8064030 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f96e8064030 0x7f96e8064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f96e8064030 0x7f96e8064410 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e8064030 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81ab200 con 0x7f96e8064030 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 1: 0 ) 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 1 = system:0 2024-04-26T16:28:46.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 4 2024-04-26T16:28:46.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.733+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 0 1 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 10 client.41225.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 2 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 2 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e819c680 osd=0 3 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 0 2 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 4 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 10 client.41225.objecter _send_command 2 2024-04-26T16:28:46.749 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.749+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e8063ab0 con 0x7f96e8064030 2024-04-26T16:28:46.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e8063ab0 con 0x7f96e8064030 2024-04-26T16:28:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 4 2024-04-26T16:28:46.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 0 2 2024-04-26T16:28:46.798 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 10 client.41225.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f96e802d310 0x7f96e802d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 1 3 2024-04-26T16:28:46.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 4 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 10 client.41225.objecter _send_command 3 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f96e8105ea0 con 0x7f96e802d310 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f96e802d310 0x7f96e802d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e802d310 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f96e802d310 0x7f96e802d730 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e8105ea0 con 0x7f96e802d310 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 3: 0 ) 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 3 = system:0 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 4 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.753+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 1 3 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 10 client.41225.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 4 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 4 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e8105820 osd=1 3 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 1 4 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 4 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 10 client.41225.objecter _send_command 4 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e81052a0 con 0x7f96e802d310 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e81052a0 con 0x7f96e802d310 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 4 2024-04-26T16:28:46.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 1 4 2024-04-26T16:28:46.802 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:28:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 10 client.41225.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 5 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f96e81ad3c0 0x7f96e81af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 5 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 2 5 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 4 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 10 client.41225.objecter _send_command 5 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f96e81b2610 con 0x7f96e81ad3c0 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f96e81ad3c0 0x7f96e81af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e81ad3c0 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.773+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f96e81ad3c0 0x7f96e81af7b0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.777+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81b2610 con 0x7f96e81ad3c0 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.777+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 5: 0 ) 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.777+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 5 = system:0 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.777+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 4 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.777+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 2 5 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 10 client.41225.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 6 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 6 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81056a0 osd=2 3 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 2 6 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 4 2024-04-26T16:28:46.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 10 client.41225.objecter _send_command 6 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e802dec0 con 0x7f96e81ad3c0 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e802dec0 con 0x7f96e81ad3c0 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 4 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 2 6 2024-04-26T16:28:46.805 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 10 client.41225.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 7 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f96e81b0e70 0x7f96e81b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 7 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 3 7 2024-04-26T16:28:46.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 4 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 10 client.41225.objecter _send_command 7 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f96e81aaf70 con 0x7f96e81b0e70 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.793+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f96e81b0e70 0x7f96e81b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e81b0e70 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f96e81b0e70 0x7f96e81b12d0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81aaf70 con 0x7f96e81b0e70 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 7: 0 ) 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 7 = system:0 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 4 2024-04-26T16:28:46.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.797+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 3 7 2024-04-26T16:28:46.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:46.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 10 client.41225.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 8 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 8 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81aa850 osd=3 3 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 3 8 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 4 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 10 client.41225.objecter _send_command 8 2024-04-26T16:28:46.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.813+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e81aa2d0 con 0x7f96e81b0e70 2024-04-26T16:28:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e81aa2d0 con 0x7f96e81b0e70 2024-04-26T16:28:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 4 2024-04-26T16:28:46.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 3 8 2024-04-26T16:28:46.815 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 10 client.41225.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 9 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f96e81b47d0 0x7f96e81b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 9 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 4 9 2024-04-26T16:28:46.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 4 2024-04-26T16:28:46.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 10 client.41225.objecter _send_command 9 2024-04-26T16:28:46.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f96e81aa280 con 0x7f96e81b47d0 2024-04-26T16:28:46.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f3bfc19c650 con 0x7f3bfc06b130 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f96e81b47d0 0x7f96e81b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e81b47d0 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.817+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f96e81b47d0 0x7f96e81b6bc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.821+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81aa280 con 0x7f96e81b47d0 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.821+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 9: 0 ) 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.821+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 9 = system:0 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.821+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 4 2024-04-26T16:28:46.818 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.821+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 4 9 2024-04-26T16:28:46.835 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 10 client.41225.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 10 2024-04-26T16:28:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 10 2024-04-26T16:28:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81aa080 osd=4 3 2024-04-26T16:28:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 4 10 2024-04-26T16:28:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 4 2024-04-26T16:28:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 10 client.41225.objecter _send_command 10 2024-04-26T16:28:46.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.837+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e81b7100 con 0x7f96e81b47d0 2024-04-26T16:28:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e81b7100 con 0x7f96e81b47d0 2024-04-26T16:28:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 4 2024-04-26T16:28:46.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 4 10 2024-04-26T16:28:46.838 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:28:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 10 client.41225.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 11 2024-04-26T16:28:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f96e81b7820 0x7f96e81b9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 11 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 5 11 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 4 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 10 client.41225.objecter _send_command 11 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f96e81bcbb0 con 0x7f96e81b7820 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f96e81b7820 0x7f96e81b9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e81b7820 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e7fff700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f96e81b7820 0x7f96e81b9c10 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81bcbb0 con 0x7f96e81b7820 2024-04-26T16:28:46.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 11: 0 ) 2024-04-26T16:28:46.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 11 = system:0 2024-04-26T16:28:46.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 4 2024-04-26T16:28:46.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.841+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 5 11 2024-04-26T16:28:46.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 10 client.41225.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 12 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 12 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81aa5f0 osd=5 3 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 5 12 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 4 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 10 client.41225.objecter _send_command 12 2024-04-26T16:28:46.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e81bcb70 con 0x7f96e81b7820 2024-04-26T16:28:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e81bcb70 con 0x7f96e81b7820 2024-04-26T16:28:46.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 4 2024-04-26T16:28:46.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 5 12 2024-04-26T16:28:46.860 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:28:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 10 client.41225.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 13 2024-04-26T16:28:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f96e81ba6b0 0x7f96e81bab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 13 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 6 13 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 4 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 10 client.41225.objecter _send_command 13 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f96e81bff50 con 0x7f96e81ba6b0 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.861+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f96e81ba6b0 0x7f96e81bab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e81ba6b0 2024-04-26T16:28:46.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96ed2cc700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f96e81ba6b0 0x7f96e81bab10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81bff50 con 0x7f96e81ba6b0 2024-04-26T16:28:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 13: 0 ) 2024-04-26T16:28:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 13 = system:0 2024-04-26T16:28:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 4 2024-04-26T16:28:46.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.865+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 6 13 2024-04-26T16:28:46.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 10 client.41225.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 14 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 14 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81bcb70 osd=6 3 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 6 14 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 4 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 10 client.41225.objecter _send_command 14 2024-04-26T16:28:46.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.881+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e81bb160 con 0x7f96e81ba6b0 2024-04-26T16:28:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e81bb160 con 0x7f96e81ba6b0 2024-04-26T16:28:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 4 2024-04-26T16:28:46.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 6 14 2024-04-26T16:28:46.882 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 10 client.41225.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 15 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f96e81bb880 0x7f96e81bbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 15 2024-04-26T16:28:46.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 7 15 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 4 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 10 client.41225.objecter _send_command 15 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f96e81bc3a0 con 0x7f96e81bb880 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f96e81bb880 0x7f96e81bbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96ecacb700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f96e81bb880 0x7f96e81bbce0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 10 client.41225.objecter ms_handle_connect 0x7f96e81bb880 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e81bc3a0 con 0x7f96e81bb880 2024-04-26T16:28:46.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 15: 0 ) 2024-04-26T16:28:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 15 = system:0 2024-04-26T16:28:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 4 2024-04-26T16:28:46.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.885+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 7 15 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 10 client.41225.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign -1 16 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 20 client.41225.objecter _get_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_remove -1 16 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 20 client.41225.objecter get_session s=0x7f96e81bb160 osd=7 3 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 15 client.41225.objecter _session_command_op_assign 7 16 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 4 2024-04-26T16:28:46.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 10 client.41225.objecter _send_command 16 2024-04-26T16:28:46.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.905+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f96e81bc3a0 con 0x7f96e81bb880 2024-04-26T16:28:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96e5ffb700 1 -- 172.21.15.50:0/2552253179 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f96e81bc3a0 con 0x7f96e81bb880 2024-04-26T16:28:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96e5ffb700 10 client.41225.objecter ms_dispatch 0x7f96e8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:28:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96e5ffb700 10 client.41225.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:28:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96e5ffb700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 4 2024-04-26T16:28:46.906 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96e5ffb700 15 client.41225.objecter _session_command_op_remove 7 16 2024-04-26T16:28:46.906 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:28:46.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 librados: watch_flush enter 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 librados: watch_flush exit 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.0 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f96e8064030 msgr2=0x7f96e8064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f96e8064030 0x7f96e8064410 crc :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e819c680 osd=0 1 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.1 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f96e802d310 msgr2=0x7f96e802d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f96e802d310 0x7f96e802d730 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e8105820 osd=1 1 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.2 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f96e81ad3c0 msgr2=0x7f96e81af7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f96e81ad3c0 0x7f96e81af7b0 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81056a0 osd=2 1 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.3 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f96e81b0e70 msgr2=0x7f96e81b12d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96ed2cc700 1 -- 172.21.15.50:0/2552253179 reap_dead start 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f96e81b0e70 0x7f96e81b12d0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa850 osd=3 1 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.4 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f96e81b47d0 msgr2=0x7f96e81b6bc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f96e81b47d0 0x7f96e81b6bc0 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa080 osd=4 1 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.5 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f96e81b7820 msgr2=0x7f96e81b9c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f96e81b7820 0x7f96e81b9c10 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81aa5f0 osd=5 1 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.6 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f96e81ba6b0 msgr2=0x7f96e81bab10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f96e81ba6b0 0x7f96e81bab10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bcb70 osd=6 1 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter close_session for osd.7 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f96e81bb880 msgr2=0x7f96e81bbce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f96e81bb880 0x7f96e81bbce0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter put_session s=0x7f96e81bb160 osd=7 1 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 20 client.41225.objecter shutdown clearing up homeless session... 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 10 client.41225.objecter successfully canceled tick 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96ed2cc700 1 -- 172.21.15.50:0/2552253179 reap_dead start 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f96d8071f80 msgr2=0x7f96d8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f96d8071f80 0x7f96d8074440 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f96dc006010 tx=0x7f96dc01a040 comp rx=0 tx=0).stop 2024-04-26T16:28:46.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 msgr2=0x7f96e819b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:46.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 0x7f96e819b860 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f96d400eb30 tx=0x7f96d400ee40 comp rx=0 tx=0).stop 2024-04-26T16:28:46.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 shutdown_connections 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f96e8103e80 0x7f96e819b860 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 --2- 172.21.15.50:0/2552253179 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f96d8071f80 0x7f96d8074440 unknown :-1 s=CLOSED pgs=3749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 >> 172.21.15.50:0/2552253179 conn(0x7f96e807a5d0 msgr2=0x7f96e810e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 shutdown_connections 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 -- 172.21.15.50:0/2552253179 wait complete. 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:46.909+0000 7f96eed2f700 1 librados: shutdown 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:28:46.914 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:28:46.915 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:28:46.915 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:28:46.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:46.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:46.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:47.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:47.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:28:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:28:47.095 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:47.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.161+0000 7f3bf2ffd700 1 -- 172.21.15.50:0/2762247130 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v455) ==== 101+0+0 (secure 0 0 0) 0x7f3bf8061280 con 0x7f3bfc06b130 2024-04-26T16:28:47.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 10 librados: watch_flush enter 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 10 librados: watch_flush exit 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 20 client.41219.objecter shutdown clearing up homeless session... 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 10 client.41219.objecter successfully canceled tick 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3be8071f80 msgr2=0x7f3be8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3be8071f80 0x7f3be8074440 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7f3bec005fd0 tx=0x7f3bec005e20 comp rx=0 tx=0).stop 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 msgr2=0x7f3bfc19b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:47.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.169+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 0x7f3bfc19b2f0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7f3bf800eb30 tx=0x7f3bf800ee40 comp rx=0 tx=0).stop 2024-04-26T16:28:47.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 shutdown_connections 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3be8071f80 0x7f3be8074440 unknown :-1 s=CLOSED pgs=3748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3bfc1094a0 0x7f3bfc19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3bfc06b130 0x7f3bfc19b2f0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 --2- 172.21.15.50:0/2762247130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bfc06bad0 0x7f3bfc19b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 >> 172.21.15.50:0/2762247130 conn(0x7f3bfc0792d0 msgr2=0x7f3bfc100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 shutdown_connections 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 -- 172.21.15.50:0/2762247130 wait complete. 2024-04-26T16:28:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:47.173+0000 7f3c03a9e700 1 librados: shutdown 2024-04-26T16:28:47.174 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:28:47.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:47.219 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:47.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:47.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:47.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.102496+0000 osd.3 (osd.3) 275 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:28:47.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.102496+0000 osd.3 (osd.3) 275 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:28:47.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.104256+0000 osd.3 (osd.3) 276 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:28:47.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.104256+0000 osd.3 (osd.3) 276 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:28:47.453 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:46.734424+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? 172.21.15.50:0/2552253179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:46.734424+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? 172.21.15.50:0/2552253179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:46.821871+0000 mon.c (mon.2) 617 : audit [INF] from='client.? 172.21.15.50:0/2762247130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:46.821871+0000 mon.c (mon.2) 617 : audit [INF] from='client.? 172.21.15.50:0/2762247130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:46.822276+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:46.822276+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.160251+0000 mon.a (mon.0) 2735 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.160251+0000 mon.a (mon.0) 2735 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.161742+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.161742+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.161823+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.161823+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.169811+0000 mon.b (mon.1) 321 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:47.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.169811+0000 mon.b (mon.1) 321 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:47.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.171744+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-04-26T16:28:47.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.171744+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-04-26T16:28:47.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.180642+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.180642+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.180738+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[29234]: audit 2024-04-26T16:28:47.180738+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.102496+0000 osd.3 (osd.3) 275 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:28:47.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.102496+0000 osd.3 (osd.3) 275 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:28:47.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.104256+0000 osd.3 (osd.3) 276 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:28:47.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.104256+0000 osd.3 (osd.3) 276 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:28:47.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:46.734424+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? 172.21.15.50:0/2552253179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:47.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:46.734424+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? 172.21.15.50:0/2552253179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:46.821871+0000 mon.c (mon.2) 617 : audit [INF] from='client.? 172.21.15.50:0/2762247130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:46.821871+0000 mon.c (mon.2) 617 : audit [INF] from='client.? 172.21.15.50:0/2762247130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:46.822276+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:46.822276+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.160251+0000 mon.a (mon.0) 2735 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.160251+0000 mon.a (mon.0) 2735 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.161742+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:47.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.161742+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.161823+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.161823+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.169811+0000 mon.b (mon.1) 321 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.169811+0000 mon.b (mon.1) 321 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.171744+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.171744+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.180642+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.180642+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.180738+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:47 smithi050 bash[21463]: audit 2024-04-26T16:28:47.180738+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:47.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.102496+0000 osd.3 (osd.3) 275 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:28:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.102496+0000 osd.3 (osd.3) 275 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:28:47.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.104256+0000 osd.3 (osd.3) 276 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.104256+0000 osd.3 (osd.3) 276 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:46.734424+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? 172.21.15.50:0/2552253179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:46.734424+0000 mon.c (mon.2) 616 : audit [DBG] from='client.? 172.21.15.50:0/2552253179' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:46.821871+0000 mon.c (mon.2) 617 : audit [INF] from='client.? 172.21.15.50:0/2762247130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:46.821871+0000 mon.c (mon.2) 617 : audit [INF] from='client.? 172.21.15.50:0/2762247130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:46.822276+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:46.822276+0000 mon.a (mon.0) 2734 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.160251+0000 mon.a (mon.0) 2735 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.160251+0000 mon.a (mon.0) 2735 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.161742+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.161742+0000 mon.a (mon.0) 2736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.161823+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.161823+0000 mon.a (mon.0) 2737 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:28:47.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.169811+0000 mon.b (mon.1) 321 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.169811+0000 mon.b (mon.1) 321 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.171744+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.171744+0000 mon.a (mon.0) 2738 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.180642+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.180642+0000 mon.b (mon.1) 322 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.180738+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:47 smithi190 bash[23868]: audit 2024-04-26T16:28:47.180738+0000 mon.a (mon.0) 2739 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:28:47.599 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:47.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:47.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:47.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:47.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:47.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:47.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:47.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:48.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:48.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:48.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:48.256 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:48.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:48.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:48.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:48.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.865129+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:28:48.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.865129+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:28:48.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.866186+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:28:48.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.866186+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:28:48.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: audit 2024-04-26T16:28:46.883447+0000 mgr.y (mgr.24401) 445 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: audit 2024-04-26T16:28:46.883447+0000 mgr.y (mgr.24401) 445 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.986233+0000 osd.7 (osd.7) 263 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.986233+0000 osd.7 (osd.7) 263 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.987681+0000 osd.7 (osd.7) 264 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:46.987681+0000 osd.7 (osd.7) 264 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.095672+0000 osd.3 (osd.3) 277 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.095672+0000 osd.3 (osd.3) 277 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.096847+0000 osd.3 (osd.3) 278 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.096847+0000 osd.3 (osd.3) 278 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.513552+0000 osd.4 (osd.4) 237 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.513552+0000 osd.4 (osd.4) 237 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.517710+0000 osd.4 (osd.4) 238 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.517710+0000 osd.4 (osd.4) 238 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:28:48.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.656690+0000 mgr.y (mgr.24401) 446 : cluster [DBG] pgmap v669: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:28:48.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.656690+0000 mgr.y (mgr.24401) 446 : cluster [DBG] pgmap v669: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:28:48.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: audit 2024-04-26T16:28:48.163922+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:48.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: audit 2024-04-26T16:28:48.163922+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:48.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.169179+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-04-26T16:28:48.506 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:48 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.169179+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-04-26T16:28:48.517 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:48.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:48.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.865129+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.865129+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.866186+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.866186+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: audit 2024-04-26T16:28:46.883447+0000 mgr.y (mgr.24401) 445 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: audit 2024-04-26T16:28:46.883447+0000 mgr.y (mgr.24401) 445 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.986233+0000 osd.7 (osd.7) 263 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.986233+0000 osd.7 (osd.7) 263 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.987681+0000 osd.7 (osd.7) 264 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:46.987681+0000 osd.7 (osd.7) 264 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.095672+0000 osd.3 (osd.3) 277 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.095672+0000 osd.3 (osd.3) 277 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.096847+0000 osd.3 (osd.3) 278 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:28:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.096847+0000 osd.3 (osd.3) 278 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.513552+0000 osd.4 (osd.4) 237 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.513552+0000 osd.4 (osd.4) 237 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.517710+0000 osd.4 (osd.4) 238 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.517710+0000 osd.4 (osd.4) 238 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.656690+0000 mgr.y (mgr.24401) 446 : cluster [DBG] pgmap v669: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.656690+0000 mgr.y (mgr.24401) 446 : cluster [DBG] pgmap v669: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: audit 2024-04-26T16:28:48.163922+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: audit 2024-04-26T16:28:48.163922+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.169179+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-04-26T16:28:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.169179+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.865129+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.865129+0000 osd.1 (osd.1) 237 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.866186+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.866186+0000 osd.1 (osd.1) 238 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: audit 2024-04-26T16:28:46.883447+0000 mgr.y (mgr.24401) 445 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: audit 2024-04-26T16:28:46.883447+0000 mgr.y (mgr.24401) 445 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.986233+0000 osd.7 (osd.7) 263 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.986233+0000 osd.7 (osd.7) 263 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.987681+0000 osd.7 (osd.7) 264 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:46.987681+0000 osd.7 (osd.7) 264 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.095672+0000 osd.3 (osd.3) 277 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.095672+0000 osd.3 (osd.3) 277 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.096847+0000 osd.3 (osd.3) 278 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:28:48.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.096847+0000 osd.3 (osd.3) 278 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.513552+0000 osd.4 (osd.4) 237 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.513552+0000 osd.4 (osd.4) 237 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.517710+0000 osd.4 (osd.4) 238 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.517710+0000 osd.4 (osd.4) 238 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.656690+0000 mgr.y (mgr.24401) 446 : cluster [DBG] pgmap v669: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.656690+0000 mgr.y (mgr.24401) 446 : cluster [DBG] pgmap v669: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: audit 2024-04-26T16:28:48.163922+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: audit 2024-04-26T16:28:48.163922+0000 mon.a (mon.0) 2740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.169179+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-04-26T16:28:48.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:48 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.169179+0000 mon.a (mon.0) 2741 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2024-04-26T16:28:48.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:48.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:48.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:48.915 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:48.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:49.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:49.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:49.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:49.178 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:49.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:49.310 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:49.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:49.424 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.877377+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.877377+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.878924+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:47.878924+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.137583+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.137583+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.140155+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.140155+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.169855+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-04-26T16:28:49.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:49 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.169855+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-04-26T16:28:49.442 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:49.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:49.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.877377+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:28:49.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.877377+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.878924+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:47.878924+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.137583+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.137583+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.140155+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.140155+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.169855+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.169855+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.877377+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.877377+0000 osd.1 (osd.1) 239 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:28:49.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.878924+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:47.878924+0000 osd.1 (osd.1) 240 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.137583+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.137583+0000 osd.3 (osd.3) 279 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.140155+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.140155+0000 osd.3 (osd.3) 280 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.169855+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-04-26T16:28:49.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:49 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.169855+0000 mon.a (mon.0) 2742 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2024-04-26T16:28:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:49.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:49.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:49.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:49.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:49.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:49.965 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:49.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:50.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:50.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:50.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:50.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:50.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:50.351 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:50.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:50.482 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:50.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:50.597 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.834643+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:28:50.597 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.834643+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:28:50.597 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.836850+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:48.836850+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.445232+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.445232+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.448411+0000 osd.0 (osd.0) 266 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.448411+0000 osd.0 (osd.0) 266 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.470093+0000 osd.4 (osd.4) 239 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.470093+0000 osd.4 (osd.4) 239 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.475037+0000 osd.4 (osd.4) 240 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.475037+0000 osd.4 (osd.4) 240 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.657556+0000 mgr.y (mgr.24401) 447 : cluster [DBG] pgmap v672: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:49.657556+0000 mgr.y (mgr.24401) 447 : cluster [DBG] pgmap v672: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:28:50.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.074943+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:28:50.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.074943+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:28:50.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.076603+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:28:50.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.076603+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:28:50.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.171539+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-04-26T16:28:50.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:50 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.171539+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-04-26T16:28:50.614 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:50.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:50.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.834643+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:28:50.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.834643+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.836850+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:48.836850+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.445232+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.445232+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.448411+0000 osd.0 (osd.0) 266 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.448411+0000 osd.0 (osd.0) 266 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.470093+0000 osd.4 (osd.4) 239 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.470093+0000 osd.4 (osd.4) 239 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.475037+0000 osd.4 (osd.4) 240 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.475037+0000 osd.4 (osd.4) 240 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.657556+0000 mgr.y (mgr.24401) 447 : cluster [DBG] pgmap v672: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:28:50.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:49.657556+0000 mgr.y (mgr.24401) 447 : cluster [DBG] pgmap v672: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.074943+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.074943+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.076603+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.076603+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.171539+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.171539+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.834643+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.834643+0000 osd.1 (osd.1) 241 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.836850+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:48.836850+0000 osd.1 (osd.1) 242 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:28:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.445232+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.445232+0000 osd.0 (osd.0) 265 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.448411+0000 osd.0 (osd.0) 266 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.448411+0000 osd.0 (osd.0) 266 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.470093+0000 osd.4 (osd.4) 239 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.470093+0000 osd.4 (osd.4) 239 : cluster [DBG] 109.14 deep-scrub starts 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.475037+0000 osd.4 (osd.4) 240 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.475037+0000 osd.4 (osd.4) 240 : cluster [DBG] 109.14 deep-scrub ok 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.657556+0000 mgr.y (mgr.24401) 447 : cluster [DBG] pgmap v672: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:49.657556+0000 mgr.y (mgr.24401) 447 : cluster [DBG] pgmap v672: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.074943+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.074943+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.076603+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:28:50.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.076603+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:28:50.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.171539+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-04-26T16:28:50.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:50 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.171539+0000 mon.a (mon.0) 2743 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2024-04-26T16:28:50.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:50.745 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:50.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:50.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:50.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:50.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:50.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:51.007 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:51.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:51.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:51.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:51.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:51.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:51.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.032112+0000 osd.2 (osd.2) 211 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.032112+0000 osd.2 (osd.2) 211 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.034988+0000 osd.2 (osd.2) 212 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.034988+0000 osd.2 (osd.2) 212 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.448654+0000 osd.0 (osd.0) 267 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.448654+0000 osd.0 (osd.0) 267 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.453438+0000 osd.0 (osd.0) 268 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.453438+0000 osd.0 (osd.0) 268 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.508819+0000 osd.4 (osd.4) 241 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.508819+0000 osd.4 (osd.4) 241 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.511925+0000 osd.4 (osd.4) 242 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.511925+0000 osd.4 (osd.4) 242 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:51.174327+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-04-26T16:28:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[21463]: cluster 2024-04-26T16:28:51.174327+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.032112+0000 osd.2 (osd.2) 211 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.032112+0000 osd.2 (osd.2) 211 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.034988+0000 osd.2 (osd.2) 212 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.034988+0000 osd.2 (osd.2) 212 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.448654+0000 osd.0 (osd.0) 267 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.448654+0000 osd.0 (osd.0) 267 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.453438+0000 osd.0 (osd.0) 268 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.453438+0000 osd.0 (osd.0) 268 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.508819+0000 osd.4 (osd.4) 241 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.508819+0000 osd.4 (osd.4) 241 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.511925+0000 osd.4 (osd.4) 242 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:28:51.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.511925+0000 osd.4 (osd.4) 242 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:28:51.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:51.174327+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-04-26T16:28:51.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:51 smithi050 bash[29234]: cluster 2024-04-26T16:28:51.174327+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-04-26T16:28:51.400 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:51.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:51.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:51.531 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:51.634 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:51.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:51.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.032112+0000 osd.2 (osd.2) 211 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.032112+0000 osd.2 (osd.2) 211 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.034988+0000 osd.2 (osd.2) 212 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.034988+0000 osd.2 (osd.2) 212 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.448654+0000 osd.0 (osd.0) 267 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.448654+0000 osd.0 (osd.0) 267 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.453438+0000 osd.0 (osd.0) 268 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.453438+0000 osd.0 (osd.0) 268 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.508819+0000 osd.4 (osd.4) 241 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.508819+0000 osd.4 (osd.4) 241 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.511925+0000 osd.4 (osd.4) 242 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.511925+0000 osd.4 (osd.4) 242 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:28:51.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:51.174327+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-04-26T16:28:51.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:51 smithi190 bash[23868]: cluster 2024-04-26T16:28:51.174327+0000 mon.a (mon.0) 2744 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2024-04-26T16:28:51.795 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:51.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:51.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:51.926 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:51.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:51.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:52.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:52.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:28:52.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:52.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:52.191 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:52.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:52.320 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:52.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.969224+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.969224+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.980864+0000 osd.7 (osd.7) 266 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.980864+0000 osd.7 (osd.7) 266 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.998681+0000 osd.2 (osd.2) 213 : cluster [DBG] 109.1b deep-scrub starts 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:50.998681+0000 osd.2 (osd.2) 213 : cluster [DBG] 109.1b deep-scrub starts 2024-04-26T16:28:52.435 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:51.005529+0000 osd.2 (osd.2) 214 : cluster [DBG] 109.1b deep-scrub ok 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:51.005529+0000 osd.2 (osd.2) 214 : cluster [DBG] 109.1b deep-scrub ok 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:51.659268+0000 mgr.y (mgr.24401) 448 : cluster [DBG] pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 529 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:51.659268+0000 mgr.y (mgr.24401) 448 : cluster [DBG] pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 529 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.134253+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.134253+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.136377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.136377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.180046+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-04-26T16:28:52.436 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:52 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.180046+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-04-26T16:28:52.449 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:52.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:52.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.969224+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:28:52.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.969224+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:28:52.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.980864+0000 osd.7 (osd.7) 266 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:28:52.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.980864+0000 osd.7 (osd.7) 266 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.998681+0000 osd.2 (osd.2) 213 : cluster [DBG] 109.1b deep-scrub starts 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:50.998681+0000 osd.2 (osd.2) 213 : cluster [DBG] 109.1b deep-scrub starts 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:51.005529+0000 osd.2 (osd.2) 214 : cluster [DBG] 109.1b deep-scrub ok 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:51.005529+0000 osd.2 (osd.2) 214 : cluster [DBG] 109.1b deep-scrub ok 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:51.659268+0000 mgr.y (mgr.24401) 448 : cluster [DBG] pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 529 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:51.659268+0000 mgr.y (mgr.24401) 448 : cluster [DBG] pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 529 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.134253+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.134253+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.136377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.136377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.180046+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-04-26T16:28:52.565 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.180046+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.969224+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.969224+0000 osd.7 (osd.7) 265 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.980864+0000 osd.7 (osd.7) 266 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.980864+0000 osd.7 (osd.7) 266 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.998681+0000 osd.2 (osd.2) 213 : cluster [DBG] 109.1b deep-scrub starts 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:50.998681+0000 osd.2 (osd.2) 213 : cluster [DBG] 109.1b deep-scrub starts 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:51.005529+0000 osd.2 (osd.2) 214 : cluster [DBG] 109.1b deep-scrub ok 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:51.005529+0000 osd.2 (osd.2) 214 : cluster [DBG] 109.1b deep-scrub ok 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:51.659268+0000 mgr.y (mgr.24401) 448 : cluster [DBG] pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 529 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:51.659268+0000 mgr.y (mgr.24401) 448 : cluster [DBG] pgmap v675: 228 pgs: 228 active+clean; 8.4 MiB data, 529 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.134253+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.134253+0000 osd.6 (osd.6) 11 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:28:52.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.136377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:28:52.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.136377+0000 osd.6 (osd.6) 12 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:28:52.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.180046+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-04-26T16:28:52.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:52 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.180046+0000 mon.a (mon.0) 2745 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2024-04-26T16:28:52.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:52.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:52.707 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:52.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:52.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:52.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:52.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:52.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:53.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:53.102 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:53.204 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:53.232 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:53.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:53.365 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:53.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:53.496 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:53.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:53.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:53.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.018838+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:28:53.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.018838+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:28:53.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.021534+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.021534+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.095499+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.095499+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.098247+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.098247+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.132461+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.132461+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.134179+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.134179+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.018838+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:28:53.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.018838+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.021534+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.021534+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.095499+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.095499+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.098247+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.098247+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.132461+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.132461+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.134179+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:28:53.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:53 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.134179+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:28:53.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:53.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:53.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.018838+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.018838+0000 osd.7 (osd.7) 267 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.021534+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.021534+0000 osd.7 (osd.7) 268 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.095499+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.095499+0000 osd.3 (osd.3) 281 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.098247+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.098247+0000 osd.3 (osd.3) 282 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.132461+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:28:53.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.132461+0000 osd.6 (osd.6) 13 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:28:53.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.134179+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:28:53.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:53 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.134179+0000 osd.6 (osd.6) 14 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:28:53.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:53.892 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:53.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:53.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:53.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:28:54.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:54.126 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:54.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:54.280 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:54.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3753571541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c105640 msgr2=0x7f5f0c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3753571541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c105640 0x7f5f0c105ac0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009990 tx=0x7f5efc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3753571541 shutdown_connections 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3753571541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c106000 0x7f5f0c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3753571541 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c10b640 0x7f5f0c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3753571541 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c105640 0x7f5f0c105ac0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3753571541 >> 172.21.15.50:0/3753571541 conn(0x7f5f0c101130 msgr2=0x7f5f0c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3753571541 shutdown_connections 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3753571541 wait complete. 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 librados: starting msgr at 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 librados: starting objecter 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 Processor -- start 2024-04-26T16:28:54.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- start start 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 librados: setting wanted keys 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 librados: calling monclient init 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 0x7f5f0c19b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c106000 0x7f5f0c19b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c10b640 0x7f5f0c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c114d40 con 0x7f5f0c106000 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c114bc0 con 0x7f5f0c10b640 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0c114ec0 con 0x7f5f0c105640 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 0x7f5f0c19b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c106000 0x7f5f0c19b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 0x7f5f0c19b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54528/0 (socket says 172.21.15.50:54528) 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 -- 172.21.15.50:0/3333213542 learned_addr learned my addr 172.21.15.50:0/3333213542 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1082a700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c10b640 0x7f5f0c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:54.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 -- 172.21.15.50:0/3333213542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c10b640 msgr2=0x7f5f0c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c10b640 0x7f5f0c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 -- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c106000 msgr2=0x7f5f0c19b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c106000 0x7f5f0c19b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 -- 172.21.15.50:0/3333213542 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c1a02f0 con 0x7f5f0c105640 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0b7fe700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c106000 0x7f5f0c19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1082a700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c10b640 0x7f5f0c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f0bfff700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 0x7f5f0c19b2e0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400eb30 tx=0x7f5ef400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f097fa700 10 client.?.objecter ms_handle_connect 0x7f5f0c105640 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f097fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef400cc90 con 0x7f5f0c105640 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c1a05e0 con 0x7f5f0c105640 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5ef400ce30 con 0x7f5f0c105640 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 10 client.41231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:28:54.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4003d80 con 0x7f5f0c105640 2024-04-26T16:28:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c1a7e60 con 0x7f5f0c105640 2024-04-26T16:28:54.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 librados: init done 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.349+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0c06a110 con 0x7f5f0c105640 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5ef4003f20 con 0x7f5f0c105640 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ef8071f80 0x7f5ef8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f0b7fe700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ef8071f80 0x7f5ef8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(460..460 src has 87..460) ==== 9433+0+0 (secure 0 0 0) 0x7f5ef4091cd0 con 0x7f5f0c105640 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 10 client.41231.objecter ms_dispatch 0x7f5f0c001550 osd_map(460..460 src has 87..460) 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 3 client.41231.objecter handle_osd_map got epochs [460,460] > 0 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 3 client.41231.objecter handle_osd_map decoding full epoch 460 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 20 client.41231.objecter dump_active .. 0 homeless 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f0b7fe700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ef8071f80 0x7f5ef8074440 secure :-1 s=READY pgs=3750 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00b2e0 tx=0x7f5efc01a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:28:54.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.353+0000 7f5f097fa700 10 client.41231.objecter ms_handle_connect 0x7f5ef8071f80 2024-04-26T16:28:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.357+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5ef4016330 con 0x7f5f0c105640 2024-04-26T16:28:54.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:54.411 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:54.467 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:28:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:28:54.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.960956+0000 osd.2 (osd.2) 215 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:28:54.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.960956+0000 osd.2 (osd.2) 215 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:28:54.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.963594+0000 osd.2 (osd.2) 216 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:28:54.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:52.963594+0000 osd.2 (osd.2) 216 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.487707+0000 osd.0 (osd.0) 269 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.487707+0000 osd.0 (osd.0) 269 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.489841+0000 osd.0 (osd.0) 270 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.489841+0000 osd.0 (osd.0) 270 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.660003+0000 mgr.y (mgr.24401) 449 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.660003+0000 mgr.y (mgr.24401) 449 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.960956+0000 osd.2 (osd.2) 215 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.960956+0000 osd.2 (osd.2) 215 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.963594+0000 osd.2 (osd.2) 216 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:52.963594+0000 osd.2 (osd.2) 216 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:28:54.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.487707+0000 osd.0 (osd.0) 269 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:28:54.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.487707+0000 osd.0 (osd.0) 269 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:28:54.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.489841+0000 osd.0 (osd.0) 270 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:28:54.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.489841+0000 osd.0 (osd.0) 270 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:28:54.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.660003+0000 mgr.y (mgr.24401) 449 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-26T16:28:54.469 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:54 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.660003+0000 mgr.y (mgr.24401) 449 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-26T16:28:54.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:54.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.960956+0000 osd.2 (osd.2) 215 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.960956+0000 osd.2 (osd.2) 215 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.963594+0000 osd.2 (osd.2) 216 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:52.963594+0000 osd.2 (osd.2) 216 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.487707+0000 osd.0 (osd.0) 269 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.487707+0000 osd.0 (osd.0) 269 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.489841+0000 osd.0 (osd.0) 270 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.489841+0000 osd.0 (osd.0) 270 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.660003+0000 mgr.y (mgr.24401) 449 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-26T16:28:54.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:54 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.660003+0000 mgr.y (mgr.24401) 449 : cluster [DBG] pgmap v677: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 KiB/s wr, 6 op/s 2024-04-26T16:28:54.613 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:54.613+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f5f0c19c640 con 0x7f5f0c105640 2024-04-26T16:28:54.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:54.650 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:28:54.650 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:28:54.650 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-26T16:28:54.650 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:28:54.666 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:54.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:54.796 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:54.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:54.927 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:54.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:55.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:55.058 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:55.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:55.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:55.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:55.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:55.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:55.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.369+0000 7f5f097fa700 1 -- 172.21.15.50:0/3333213542 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v461) ==== 113+0+0 (secure 0 0 0) 0x7f5ef405d650 con 0x7f5f0c105640 2024-04-26T16:28:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 10 librados: watch_flush enter 2024-04-26T16:28:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 10 librados: watch_flush exit 2024-04-26T16:28:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 20 client.41231.objecter shutdown clearing up homeless session... 2024-04-26T16:28:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 10 client.41231.objecter successfully canceled tick 2024-04-26T16:28:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ef8071f80 msgr2=0x7f5ef8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:55.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ef8071f80 0x7f5ef8074440 secure :-1 s=READY pgs=3750 cs=0 l=1 rev1=1 crypto rx=0x7f5efc00b2e0 tx=0x7f5efc01a040 comp rx=0 tx=0).stop 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 msgr2=0x7f5f0c19b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 0x7f5f0c19b2e0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400eb30 tx=0x7f5ef400ee40 comp rx=0 tx=0).stop 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 shutdown_connections 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ef8071f80 0x7f5ef8074440 unknown :-1 s=CLOSED pgs=3750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5f0c10b640 0x7f5f0c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5f0c105640 0x7f5f0c19b2e0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 --2- 172.21.15.50:0/3333213542 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5f0c106000 0x7f5f0c19b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 >> 172.21.15.50:0/3333213542 conn(0x7f5f0c101130 msgr2=0x7f5f0c101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 shutdown_connections 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 -- 172.21.15.50:0/3333213542 wait complete. 2024-04-26T16:28:55.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:28:55.373+0000 7f5f1228d700 1 librados: shutdown 2024-04-26T16:28:55.374 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T16:28:55.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:55.450 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:55.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:55.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.963606+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.c scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.963606+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.c scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.965325+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.c scrub ok 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:53.965325+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.c scrub ok 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.107120+0000 osd.3 (osd.3) 283 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.107120+0000 osd.3 (osd.3) 283 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.109524+0000 osd.3 (osd.3) 284 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.109524+0000 osd.3 (osd.3) 284 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.494489+0000 osd.0 (osd.0) 271 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.494489+0000 osd.0 (osd.0) 271 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.496767+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.496767+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.560804+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:28:55.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.560804+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.566884+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: cluster 2024-04-26T16:28:54.566884+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: audit 2024-04-26T16:28:54.619289+0000 mon.c (mon.2) 618 : audit [INF] from='client.? 172.21.15.50:0/3333213542' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: audit 2024-04-26T16:28:54.619289+0000 mon.c (mon.2) 618 : audit [INF] from='client.? 172.21.15.50:0/3333213542' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: audit 2024-04-26T16:28:54.619737+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[29234]: audit 2024-04-26T16:28:54.619737+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.963606+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.c scrub starts 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.963606+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.c scrub starts 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.965325+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.c scrub ok 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:53.965325+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.c scrub ok 2024-04-26T16:28:55.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.107120+0000 osd.3 (osd.3) 283 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.107120+0000 osd.3 (osd.3) 283 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.109524+0000 osd.3 (osd.3) 284 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.109524+0000 osd.3 (osd.3) 284 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.494489+0000 osd.0 (osd.0) 271 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.494489+0000 osd.0 (osd.0) 271 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.496767+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.496767+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.560804+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.560804+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.566884+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: cluster 2024-04-26T16:28:54.566884+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: audit 2024-04-26T16:28:54.619289+0000 mon.c (mon.2) 618 : audit [INF] from='client.? 172.21.15.50:0/3333213542' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: audit 2024-04-26T16:28:54.619289+0000 mon.c (mon.2) 618 : audit [INF] from='client.? 172.21.15.50:0/3333213542' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: audit 2024-04-26T16:28:54.619737+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:55 smithi050 bash[21463]: audit 2024-04-26T16:28:54.619737+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:55.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:55.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.963606+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.c scrub starts 2024-04-26T16:28:55.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.963606+0000 osd.2 (osd.2) 217 : cluster [DBG] 2.c scrub starts 2024-04-26T16:28:55.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.965325+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.c scrub ok 2024-04-26T16:28:55.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:53.965325+0000 osd.2 (osd.2) 218 : cluster [DBG] 2.c scrub ok 2024-04-26T16:28:55.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.107120+0000 osd.3 (osd.3) 283 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.107120+0000 osd.3 (osd.3) 283 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.109524+0000 osd.3 (osd.3) 284 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.109524+0000 osd.3 (osd.3) 284 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.494489+0000 osd.0 (osd.0) 271 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.494489+0000 osd.0 (osd.0) 271 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.496767+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.496767+0000 osd.0 (osd.0) 272 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.560804+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.560804+0000 osd.4 (osd.4) 243 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.566884+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: cluster 2024-04-26T16:28:54.566884+0000 osd.4 (osd.4) 244 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: audit 2024-04-26T16:28:54.619289+0000 mon.c (mon.2) 618 : audit [INF] from='client.? 172.21.15.50:0/3333213542' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: audit 2024-04-26T16:28:54.619289+0000 mon.c (mon.2) 618 : audit [INF] from='client.? 172.21.15.50:0/3333213542' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: audit 2024-04-26T16:28:54.619737+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.695 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:55 smithi190 bash[23868]: audit 2024-04-26T16:28:54.619737+0000 mon.a (mon.0) 2746 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:28:55.709 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:55.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:55.839 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:55.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:55.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-26T16:28:55.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:55.964 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:55.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:55.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:56.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:56.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:56.191 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:56.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:56.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:56.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:56.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: audit 2024-04-26T16:28:55.368915+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: audit 2024-04-26T16:28:55.368915+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.373221+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.373221+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.584183+0000 osd.4 (osd.4) 245 : cluster [DBG] 11.e scrub starts 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.584183+0000 osd.4 (osd.4) 245 : cluster [DBG] 11.e scrub starts 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.586443+0000 osd.4 (osd.4) 246 : cluster [DBG] 11.e scrub ok 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.586443+0000 osd.4 (osd.4) 246 : cluster [DBG] 11.e scrub ok 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.660626+0000 mgr.y (mgr.24401) 450 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.660626+0000 mgr.y (mgr.24401) 450 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-26T16:28:56.461 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:56.166346+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[21463]: cluster 2024-04-26T16:28:56.166346+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: audit 2024-04-26T16:28:55.368915+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: audit 2024-04-26T16:28:55.368915+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.373221+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.373221+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.584183+0000 osd.4 (osd.4) 245 : cluster [DBG] 11.e scrub starts 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.584183+0000 osd.4 (osd.4) 245 : cluster [DBG] 11.e scrub starts 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.586443+0000 osd.4 (osd.4) 246 : cluster [DBG] 11.e scrub ok 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.586443+0000 osd.4 (osd.4) 246 : cluster [DBG] 11.e scrub ok 2024-04-26T16:28:56.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.660626+0000 mgr.y (mgr.24401) 450 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-26T16:28:56.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.660626+0000 mgr.y (mgr.24401) 450 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-26T16:28:56.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:56.166346+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-04-26T16:28:56.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:56 smithi050 bash[29234]: cluster 2024-04-26T16:28:56.166346+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-04-26T16:28:56.473 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:56.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-26T16:28:56.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:56.604 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:56.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: audit 2024-04-26T16:28:55.368915+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: audit 2024-04-26T16:28:55.368915+0000 mon.a (mon.0) 2747 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.373221+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.373221+0000 mon.a (mon.0) 2748 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.584183+0000 osd.4 (osd.4) 245 : cluster [DBG] 11.e scrub starts 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.584183+0000 osd.4 (osd.4) 245 : cluster [DBG] 11.e scrub starts 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.586443+0000 osd.4 (osd.4) 246 : cluster [DBG] 11.e scrub ok 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.586443+0000 osd.4 (osd.4) 246 : cluster [DBG] 11.e scrub ok 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.660626+0000 mgr.y (mgr.24401) 450 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.660626+0000 mgr.y (mgr.24401) 450 : cluster [DBG] pgmap v679: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-26T16:28:56.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:56.166346+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-04-26T16:28:56.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[23868]: cluster 2024-04-26T16:28:56.166346+0000 mon.a (mon.0) 2749 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2024-04-26T16:28:56.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:56.853 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:56.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:56.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:56.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:57.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:57.098 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:28:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:28:57.115 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:57.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:57.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:57.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:57.359 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:28:57.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:57.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:57.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:57.506 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:57.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:57.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.735768+0000 osd.1 (osd.1) 243 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:28:57.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.735768+0000 osd.1 (osd.1) 243 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.739144+0000 osd.1 (osd.1) 244 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.739144+0000 osd.1 (osd.1) 244 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.871837+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.871837+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.871996+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:55.871996+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:56.600653+0000 osd.4 (osd.4) 247 : cluster [DBG] 109.f scrub starts 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:56.600653+0000 osd.4 (osd.4) 247 : cluster [DBG] 109.f scrub starts 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:56.606541+0000 osd.4 (osd.4) 248 : cluster [DBG] 109.f scrub ok 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: cluster 2024-04-26T16:28:56.606541+0000 osd.4 (osd.4) 248 : cluster [DBG] 109.f scrub ok 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: audit 2024-04-26T16:28:56.849721+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: audit 2024-04-26T16:28:56.849721+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:57.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: audit 2024-04-26T16:28:56.854275+0000 mon.c (mon.2) 619 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:57.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:57 smithi190 bash[23868]: audit 2024-04-26T16:28:56.854275+0000 mon.c (mon.2) 619 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:57.635 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:28:57.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:28:57.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:57.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:57.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:57.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.735768+0000 osd.1 (osd.1) 243 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:28:57.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.735768+0000 osd.1 (osd.1) 243 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:28:57.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.739144+0000 osd.1 (osd.1) 244 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:28:57.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.739144+0000 osd.1 (osd.1) 244 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:28:57.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.871837+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:57.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.871837+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.871996+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:55.871996+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:56.600653+0000 osd.4 (osd.4) 247 : cluster [DBG] 109.f scrub starts 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:56.600653+0000 osd.4 (osd.4) 247 : cluster [DBG] 109.f scrub starts 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:56.606541+0000 osd.4 (osd.4) 248 : cluster [DBG] 109.f scrub ok 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: cluster 2024-04-26T16:28:56.606541+0000 osd.4 (osd.4) 248 : cluster [DBG] 109.f scrub ok 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: audit 2024-04-26T16:28:56.849721+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: audit 2024-04-26T16:28:56.849721+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: audit 2024-04-26T16:28:56.854275+0000 mon.c (mon.2) 619 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[29234]: audit 2024-04-26T16:28:56.854275+0000 mon.c (mon.2) 619 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:57.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.735768+0000 osd.1 (osd.1) 243 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.735768+0000 osd.1 (osd.1) 243 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.739144+0000 osd.1 (osd.1) 244 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.739144+0000 osd.1 (osd.1) 244 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.871837+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.871837+0000 osd.2 (osd.2) 219 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.871996+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:55.871996+0000 osd.2 (osd.2) 220 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:56.600653+0000 osd.4 (osd.4) 247 : cluster [DBG] 109.f scrub starts 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:56.600653+0000 osd.4 (osd.4) 247 : cluster [DBG] 109.f scrub starts 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:56.606541+0000 osd.4 (osd.4) 248 : cluster [DBG] 109.f scrub ok 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: cluster 2024-04-26T16:28:56.606541+0000 osd.4 (osd.4) 248 : cluster [DBG] 109.f scrub ok 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: audit 2024-04-26T16:28:56.849721+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: audit 2024-04-26T16:28:56.849721+0000 mon.a (mon.0) 2750 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:28:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: audit 2024-04-26T16:28:56.854275+0000 mon.c (mon.2) 619 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:57 smithi050 bash[21463]: audit 2024-04-26T16:28:56.854275+0000 mon.c (mon.2) 619 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:28:57.897 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:57.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:58.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:58.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:58.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:58.158 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:58.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:58.289 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:58.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:58.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:58.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:58.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: audit 2024-04-26T16:28:56.891654+0000 mgr.y (mgr.24401) 451 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:58.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: audit 2024-04-26T16:28:56.891654+0000 mgr.y (mgr.24401) 451 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:58.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.061216+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:28:58.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.061216+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:28:58.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.063584+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.063584+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.655839+0000 osd.0 (osd.0) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.655839+0000 osd.0 (osd.0) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.656019+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.656019+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.662790+0000 mgr.y (mgr.24401) 452 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[29234]: cluster 2024-04-26T16:28:57.662790+0000 mgr.y (mgr.24401) 452 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: audit 2024-04-26T16:28:56.891654+0000 mgr.y (mgr.24401) 451 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: audit 2024-04-26T16:28:56.891654+0000 mgr.y (mgr.24401) 451 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:58.541 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.061216+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.061216+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.063584+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.063584+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.655839+0000 osd.0 (osd.0) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.655839+0000 osd.0 (osd.0) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.656019+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.656019+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.662790+0000 mgr.y (mgr.24401) 452 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:28:58.542 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:58 smithi050 bash[21463]: cluster 2024-04-26T16:28:57.662790+0000 mgr.y (mgr.24401) 452 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:28:58.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:58.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:58.680 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:28:58.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:28:58.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:58.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: audit 2024-04-26T16:28:56.891654+0000 mgr.y (mgr.24401) 451 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: audit 2024-04-26T16:28:56.891654+0000 mgr.y (mgr.24401) 451 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.061216+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.061216+0000 osd.7 (osd.7) 269 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.063584+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.063584+0000 osd.7 (osd.7) 270 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.655839+0000 osd.0 (osd.0) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.655839+0000 osd.0 (osd.0) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.656019+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.656019+0000 osd.0 (osd.0) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.662790+0000 mgr.y (mgr.24401) 452 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:28:58.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:58 smithi190 bash[23868]: cluster 2024-04-26T16:28:57.662790+0000 mgr.y (mgr.24401) 452 : cluster [DBG] pgmap v681: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:28:58.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:58.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:28:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:58.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:28:59.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:59.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:28:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:28:59.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:59.197 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:28:59.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:59.327 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:59.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:59.458 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:28:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:28:59.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:59 smithi190 bash[23868]: cluster 2024-04-26T16:28:58.071177+0000 osd.3 (osd.3) 285 : cluster [DBG] 11.a scrub starts 2024-04-26T16:28:59.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:59 smithi190 bash[23868]: cluster 2024-04-26T16:28:58.071177+0000 osd.3 (osd.3) 285 : cluster [DBG] 11.a scrub starts 2024-04-26T16:28:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:59 smithi190 bash[23868]: cluster 2024-04-26T16:28:58.073182+0000 osd.3 (osd.3) 286 : cluster [DBG] 11.a scrub ok 2024-04-26T16:28:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:59 smithi190 bash[23868]: cluster 2024-04-26T16:28:58.073182+0000 osd.3 (osd.3) 286 : cluster [DBG] 11.a scrub ok 2024-04-26T16:28:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:59 smithi190 bash[23868]: cluster 2024-04-26T16:28:58.407813+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-04-26T16:28:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:28:59 smithi190 bash[23868]: cluster 2024-04-26T16:28:58.407813+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-04-26T16:28:59.589 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[21463]: cluster 2024-04-26T16:28:58.071177+0000 osd.3 (osd.3) 285 : cluster [DBG] 11.a scrub starts 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[21463]: cluster 2024-04-26T16:28:58.071177+0000 osd.3 (osd.3) 285 : cluster [DBG] 11.a scrub starts 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[21463]: cluster 2024-04-26T16:28:58.073182+0000 osd.3 (osd.3) 286 : cluster [DBG] 11.a scrub ok 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[21463]: cluster 2024-04-26T16:28:58.073182+0000 osd.3 (osd.3) 286 : cluster [DBG] 11.a scrub ok 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[21463]: cluster 2024-04-26T16:28:58.407813+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[21463]: cluster 2024-04-26T16:28:58.407813+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-04-26T16:28:59.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[29234]: cluster 2024-04-26T16:28:58.071177+0000 osd.3 (osd.3) 285 : cluster [DBG] 11.a scrub starts 2024-04-26T16:28:59.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[29234]: cluster 2024-04-26T16:28:58.071177+0000 osd.3 (osd.3) 285 : cluster [DBG] 11.a scrub starts 2024-04-26T16:28:59.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[29234]: cluster 2024-04-26T16:28:58.073182+0000 osd.3 (osd.3) 286 : cluster [DBG] 11.a scrub ok 2024-04-26T16:28:59.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[29234]: cluster 2024-04-26T16:28:58.073182+0000 osd.3 (osd.3) 286 : cluster [DBG] 11.a scrub ok 2024-04-26T16:28:59.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[29234]: cluster 2024-04-26T16:28:58.407813+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-04-26T16:28:59.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:28:59 smithi050 bash[29234]: cluster 2024-04-26T16:28:58.407813+0000 mon.a (mon.0) 2751 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2024-04-26T16:28:59.692 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:59.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:28:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:28:59.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:59.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:28:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:28:59.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:59.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:28:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:28:59.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:00.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:00.098 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:00.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:00.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:00.233 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:00.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:00.362 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:00.432 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:00.433 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:00.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:00.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:00.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.036455+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:29:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.036455+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:29:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.039565+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:29:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.039565+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:29:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.043249+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:29:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.043249+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.045404+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.045404+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.597878+0000 osd.4 (osd.4) 249 : cluster [DBG] 6.e scrub starts 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.597878+0000 osd.4 (osd.4) 249 : cluster [DBG] 6.e scrub starts 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.599799+0000 osd.4 (osd.4) 250 : cluster [DBG] 6.e scrub ok 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.599799+0000 osd.4 (osd.4) 250 : cluster [DBG] 6.e scrub ok 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.663458+0000 mgr.y (mgr.24401) 453 : cluster [DBG] pgmap v683: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:28:59.663458+0000 mgr.y (mgr.24401) 453 : cluster [DBG] pgmap v683: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.090144+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.090144+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.093183+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:29:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.093183+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:29:00.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:00.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:00.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:00.747 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:00.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:00.878 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.036455+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.036455+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.039565+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.039565+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.043249+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.043249+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.045404+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.045404+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:29:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.597878+0000 osd.4 (osd.4) 249 : cluster [DBG] 6.e scrub starts 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.597878+0000 osd.4 (osd.4) 249 : cluster [DBG] 6.e scrub starts 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.599799+0000 osd.4 (osd.4) 250 : cluster [DBG] 6.e scrub ok 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.599799+0000 osd.4 (osd.4) 250 : cluster [DBG] 6.e scrub ok 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.663458+0000 mgr.y (mgr.24401) 453 : cluster [DBG] pgmap v683: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:28:59.663458+0000 mgr.y (mgr.24401) 453 : cluster [DBG] pgmap v683: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.090144+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.090144+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.093183+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.093183+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.036455+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.036455+0000 osd.3 (osd.3) 287 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:29:00.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.039565+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.039565+0000 osd.3 (osd.3) 288 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.043249+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.043249+0000 osd.7 (osd.7) 271 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.045404+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.045404+0000 osd.7 (osd.7) 272 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.597878+0000 osd.4 (osd.4) 249 : cluster [DBG] 6.e scrub starts 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.597878+0000 osd.4 (osd.4) 249 : cluster [DBG] 6.e scrub starts 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.599799+0000 osd.4 (osd.4) 250 : cluster [DBG] 6.e scrub ok 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.599799+0000 osd.4 (osd.4) 250 : cluster [DBG] 6.e scrub ok 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.663458+0000 mgr.y (mgr.24401) 453 : cluster [DBG] pgmap v683: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:28:59.663458+0000 mgr.y (mgr.24401) 453 : cluster [DBG] pgmap v683: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.090144+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.090144+0000 osd.6 (osd.6) 15 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:29:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.093183+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:29:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.093183+0000 osd.6 (osd.6) 16 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:29:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:00.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:00.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:01.011 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:01.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:01.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:01.142 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:01.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:01.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:01.273 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:01.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:01.404 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:01.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:01.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.041294+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:29:01.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.041294+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:29:01.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.045438+0000 osd.7 (osd.7) 274 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:29:01.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.045438+0000 osd.7 (osd.7) 274 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:29:01.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.062064+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:29:01.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.062064+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:29:01.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.063643+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:29:01.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:00.063643+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:29:01.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.173731+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-04-26T16:29:01.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.173731+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-04-26T16:29:01.535 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:01.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:01.666 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:01.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.041294+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.041294+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.045438+0000 osd.7 (osd.7) 274 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.045438+0000 osd.7 (osd.7) 274 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.062064+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.062064+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.063643+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:00.063643+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.173731+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-04-26T16:29:01.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.173731+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.041294+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.041294+0000 osd.7 (osd.7) 273 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.045438+0000 osd.7 (osd.7) 274 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.045438+0000 osd.7 (osd.7) 274 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.062064+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.062064+0000 osd.3 (osd.3) 289 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.063643+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:00.063643+0000 osd.3 (osd.3) 290 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.173731+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-04-26T16:29:01.782 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.173731+0000 mon.a (mon.0) 2752 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2024-04-26T16:29:01.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:01.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:01.928 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:01.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:02.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:02.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:02.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:02.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:02.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:02.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/1711876669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 msgr2=0x7f0bcc1102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- 172.21.15.50:0/1711876669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 0x7f0bcc1102f0 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4009f90 tx=0x7f0bc400b330 comp rx=0 tx=0).stop 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/1711876669 shutdown_connections 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- 172.21.15.50:0/1711876669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bcc1094b0 0x7f0bcc109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- 172.21.15.50:0/1711876669 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 0x7f0bcc1102f0 unknown :-1 s=CLOSED pgs=2664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- 172.21.15.50:0/1711876669 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bcc101a10 0x7f0bcc101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/1711876669 >> 172.21.15.50:0/1711876669 conn(0x7f0bcc078990 msgr2=0x7f0bcc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/1711876669 shutdown_connections 2024-04-26T16:29:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/1711876669 wait complete. 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 librados: starting msgr at 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 librados: starting objecter 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 Processor -- start 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- start start 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 librados: setting wanted keys 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 librados: calling monclient init 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bcc101a10 0x7f0bcc19b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 0x7f0bcc19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bcc1094b0 0x7f0bcc19fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc112a90 con 0x7f0bcc1094b0 2024-04-26T16:29:02.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0bcc112910 con 0x7f0bcc101a10 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0bcc112c10 con 0x7f0bcc1023b0 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bca59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 0x7f0bcc19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bca59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 0x7f0bcc19b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38154/0 (socket says 172.21.15.50:38154) 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bcc1094b0 0x7f0bcc19fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bca59c700 1 -- 172.21.15.50:0/3278551714 learned_addr learned my addr 172.21.15.50:0/3278551714 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bcc101a10 0x7f0bcc19b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 msgr2=0x7f0bcc19b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bcc1023b0 0x7f0bcc19b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bcc101a10 msgr2=0x7f0bcc19b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bcc101a10 0x7f0bcc19b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bcc1a02a0 con 0x7f0bcc1094b0 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bcc1094b0 0x7f0bcc19fb60 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d010 tx=0x7f0bc4004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bb3fff700 10 client.?.objecter ms_handle_connect 0x7f0bcc1094b0 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bb3fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4021070 con 0x7f0bcc1094b0 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0bc4007910 con 0x7f0bcc1094b0 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bcc1a0530 con 0x7f0bcc1094b0 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4005040 con 0x7f0bcc1094b0 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 10 client.30624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bcc1a7e10 con 0x7f0bcc1094b0 2024-04-26T16:29:02.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.297+0000 7f0bd1093700 1 librados: init done 2024-04-26T16:29:02.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0bcc06a110 con 0x7f0bcc1094b0 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0bc4024070 con 0x7f0bcc1094b0 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bb4071f80 0x7f0bb4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(464..464 src has 87..464) ==== 9405+0+0 (secure 0 0 0) 0x7f0bc400f020 con 0x7f0bcc1094b0 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 osd_map(464..464 src has 87..464) 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 3 client.30624.objecter handle_osd_map got epochs [464,464] > 0 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 3 client.30624.objecter handle_osd_map decoding full epoch 464 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bb4071f80 0x7f0bb4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 20 client.30624.objecter dump_active .. 0 homeless 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v464) ==== 54+0+16 (secure 0 0 0) 0x7f0bc402b020 con 0x7f0bcc1094b0 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 10 client.30624.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 1 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0bcc064030 0x7f0bcc064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 1 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 0 1 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 4 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 10 client.30624.objecter _send_command 1 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1ab170 con 0x7f0bcc064030 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bb4071f80 0x7f0bb4074440 secure :-1 s=READY pgs=3751 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc005950 tx=0x7f0bbc0058e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bb4071f80 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bca59c700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0bcc064030 0x7f0bcc064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bca59c700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0bcc064030 0x7f0bcc064410 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.301+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc064030 2024-04-26T16:29:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.305+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1ab170 con 0x7f0bcc064030 2024-04-26T16:29:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.305+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 1: 0 ) 2024-04-26T16:29:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.305+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 1 = system:0 2024-04-26T16:29:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.305+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 4 2024-04-26T16:29:02.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.305+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 0 1 2024-04-26T16:29:02.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 10 client.30624.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 2 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 2 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc19c5f0 osd=0 3 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 0 2 2024-04-26T16:29:02.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 4 2024-04-26T16:29:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 10 client.30624.objecter _send_command 2 2024-04-26T16:29:02.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.321+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc063ab0 con 0x7f0bcc064030 2024-04-26T16:29:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc063ab0 con 0x7f0bcc064030 2024-04-26T16:29:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 4 2024-04-26T16:29:02.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 0 2 2024-04-26T16:29:02.322 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:29:02.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 10 client.30624.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0bcc02d310 0x7f0bcc02d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 1 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 4 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 10 client.30624.objecter _send_command 3 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc0feb80 con 0x7f0bcc02d310 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0bcc02d310 0x7f0bcc02d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc02d310 2024-04-26T16:29:02.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0bcc02d310 0x7f0bcc02d6f0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc0feb80 con 0x7f0bcc02d310 2024-04-26T16:29:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 3: 0 ) 2024-04-26T16:29:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 3 = system:0 2024-04-26T16:29:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 4 2024-04-26T16:29:02.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.325+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 1 3 2024-04-26T16:29:02.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 10 client.30624.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 4 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 4 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc0fe500 osd=1 3 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 1 4 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 4 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 10 client.30624.objecter _send_command 4 2024-04-26T16:29:02.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.341+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc0fdf80 con 0x7f0bcc02d310 2024-04-26T16:29:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc0fdf80 con 0x7f0bcc02d310 2024-04-26T16:29:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 4 2024-04-26T16:29:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 1 4 2024-04-26T16:29:02.343 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:29:02.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 10 client.30624.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 5 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bcc1ad330 0x7f0bcc1af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 5 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 2 5 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 4 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 10 client.30624.objecter _send_command 5 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1b2580 con 0x7f0bcc1ad330 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bca59c700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bcc1ad330 0x7f0bcc1af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc1ad330 2024-04-26T16:29:02.344 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bca59c700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bcc1ad330 0x7f0bcc1af720 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1b2580 con 0x7f0bcc1ad330 2024-04-26T16:29:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 5: 0 ) 2024-04-26T16:29:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 5 = system:0 2024-04-26T16:29:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 4 2024-04-26T16:29:02.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.345+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 2 5 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 10 client.30624.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 6 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 6 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc0fe380 osd=2 3 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 2 6 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 4 2024-04-26T16:29:02.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 10 client.30624.objecter _send_command 6 2024-04-26T16:29:02.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc02de80 con 0x7f0bcc1ad330 2024-04-26T16:29:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc02de80 con 0x7f0bcc1ad330 2024-04-26T16:29:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 4 2024-04-26T16:29:02.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 2 6 2024-04-26T16:29:02.363 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 10 client.30624.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 7 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0bcc1aacd0 0x7f0bcc1b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 7 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 3 7 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 4 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 10 client.30624.objecter _send_command 7 2024-04-26T16:29:02.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1b5ee0 con 0x7f0bcc1aacd0 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0bcc1aacd0 0x7f0bcc1b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc1aacd0 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0bcc1aacd0 0x7f0bcc1b0de0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1b5ee0 con 0x7f0bcc1aacd0 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 7: 0 ) 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 7 = system:0 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 4 2024-04-26T16:29:02.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.365+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 3 7 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 10 client.30624.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 8 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 8 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc02de80 osd=3 3 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 3 8 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 4 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 10 client.30624.objecter _send_command 8 2024-04-26T16:29:02.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc1aa300 con 0x7f0bcc1aacd0 2024-04-26T16:29:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc1aa300 con 0x7f0bcc1aacd0 2024-04-26T16:29:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 4 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 3 8 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 10 client.30624.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 9 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0bcc1b0940 0x7f0bcc1b6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 9 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 4 9 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 4 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 10 client.30624.objecter _send_command 9 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1b95b0 con 0x7f0bcc1b0940 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0bcc1b0940 0x7f0bcc1b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc1b0940 2024-04-26T16:29:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.385+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0bcc1b0940 0x7f0bcc1b6750 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.389+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1b95b0 con 0x7f0bcc1b0940 2024-04-26T16:29:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.389+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 9: 0 ) 2024-04-26T16:29:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.389+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 9 = system:0 2024-04-26T16:29:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.389+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 4 2024-04-26T16:29:02.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.389+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 4 9 2024-04-26T16:29:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 10 client.30624.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 10 2024-04-26T16:29:02.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 10 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1aa0b0 osd=4 3 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 4 10 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 4 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 10 client.30624.objecter _send_command 10 2024-04-26T16:29:02.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.409+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc1b6c90 con 0x7f0bcc1b0940 2024-04-26T16:29:02.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc1b6c90 con 0x7f0bcc1b0940 2024-04-26T16:29:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 4 2024-04-26T16:29:02.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 4 10 2024-04-26T16:29:02.410 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 10 client.30624.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 11 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0bcc1b7310 0x7f0bcc1b9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 11 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 5 11 2024-04-26T16:29:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 4 2024-04-26T16:29:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 10 client.30624.objecter _send_command 11 2024-04-26T16:29:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1bc5f0 con 0x7f0bcc1b7310 2024-04-26T16:29:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bca59c700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0bcc1b7310 0x7f0bcc1b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bca59c700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0bcc1b7310 0x7f0bcc1b9700 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.413+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc1b7310 2024-04-26T16:29:02.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:02.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.417+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1bc5f0 con 0x7f0bcc1b7310 2024-04-26T16:29:02.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.417+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 11: 0 ) 2024-04-26T16:29:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.417+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 11 = system:0 2024-04-26T16:29:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.417+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 4 2024-04-26T16:29:02.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.417+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 5 11 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 10 client.30624.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 12 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 12 2024-04-26T16:29:02.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1b6c90 osd=5 3 2024-04-26T16:29:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 5 12 2024-04-26T16:29:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 4 2024-04-26T16:29:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 10 client.30624.objecter _send_command 12 2024-04-26T16:29:02.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.441+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc1b9c40 con 0x7f0bcc1b7310 2024-04-26T16:29:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc1b9c40 con 0x7f0bcc1b7310 2024-04-26T16:29:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 4 2024-04-26T16:29:02.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 5 12 2024-04-26T16:29:02.441 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:29:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 10 client.30624.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 13 2024-04-26T16:29:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f0bcc1ba360 0x7f0bcc1ba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 13 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 6 13 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 4 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 10 client.30624.objecter _send_command 13 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1bfa10 con 0x7f0bcc1ba360 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f0bcc1ba360 0x7f0bcc1ba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc1ba360 2024-04-26T16:29:02.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bcb59e700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f0bcc1ba360 0x7f0bcc1ba7c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1bfa10 con 0x7f0bcc1ba360 2024-04-26T16:29:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 13: 0 ) 2024-04-26T16:29:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 13 = system:0 2024-04-26T16:29:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 4 2024-04-26T16:29:02.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.445+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 6 13 2024-04-26T16:29:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.445 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:02.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:02.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 10 client.30624.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 14 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 14 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1b9c40 osd=6 3 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 6 14 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 4 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 10 client.30624.objecter _send_command 14 2024-04-26T16:29:02.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc1bfa10 con 0x7f0bcc1ba360 2024-04-26T16:29:02.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc1bfa10 con 0x7f0bcc1ba360 2024-04-26T16:29:02.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 4 2024-04-26T16:29:02.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 6 14 2024-04-26T16:29:02.464 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:29:02.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 10 client.30624.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:02.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 15 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0bcc1b18e0 0x7f0bcc1b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 15 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 7 15 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 4 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 10 client.30624.objecter _send_command 15 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc1c2d80 con 0x7f0bcc1b18e0 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0bcc1b18e0 0x7f0bcc1b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bb3fff700 10 client.30624.objecter ms_handle_connect 0x7f0bcc1b18e0 2024-04-26T16:29:02.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.465+0000 7f0bcad9d700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0bcc1b18e0 0x7f0bcc1b1d60 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.469+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bcc1c2d80 con 0x7f0bcc1b18e0 2024-04-26T16:29:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.469+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 15: 0 ) 2024-04-26T16:29:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.469+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 15 = system:0 2024-04-26T16:29:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.469+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 4 2024-04-26T16:29:02.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.469+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 7 15 2024-04-26T16:29:02.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 10 client.30624.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:02.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign -1 16 2024-04-26T16:29:02.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 20 client.30624.objecter _get_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_remove -1 16 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 20 client.30624.objecter get_session s=0x7f0bcc1bfa10 osd=7 3 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 15 client.30624.objecter _session_command_op_assign 7 16 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 4 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 10 client.30624.objecter _send_command 16 2024-04-26T16:29:02.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.485+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0bcc1bad50 con 0x7f0bcc1b18e0 2024-04-26T16:29:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bb3fff700 1 -- 172.21.15.50:0/3278551714 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0bcc1bad50 con 0x7f0bcc1b18e0 2024-04-26T16:29:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bb3fff700 10 client.30624.objecter ms_dispatch 0x7f0bcc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bb3fff700 10 client.30624.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bb3fff700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 4 2024-04-26T16:29:02.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bb3fff700 15 client.30624.objecter _session_command_op_remove 7 16 2024-04-26T16:29:02.487 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:29:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 librados: watch_flush enter 2024-04-26T16:29:02.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 librados: watch_flush exit 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.0 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0bcc064030 msgr2=0x7f0bcc064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0bcc064030 0x7f0bcc064410 crc :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc19c5f0 osd=0 1 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.1 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0bcc02d310 msgr2=0x7f0bcc02d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0bcc02d310 0x7f0bcc02d6f0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe500 osd=1 1 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.2 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bcc1ad330 msgr2=0x7f0bcc1af720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0bcc1ad330 0x7f0bcc1af720 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc0fe380 osd=2 1 2024-04-26T16:29:02.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.3 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0bcc1aacd0 msgr2=0x7f0bcc1b0de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0bcc1aacd0 0x7f0bcc1b0de0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc02de80 osd=3 1 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.4 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bcb59e700 1 -- 172.21.15.50:0/3278551714 reap_dead start 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0bcc1b0940 msgr2=0x7f0bcc1b6750 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0bcc1b0940 0x7f0bcc1b6750 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bcb59e700 1 -- 172.21.15.50:0/3278551714 reap_dead start 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1aa0b0 osd=4 1 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.5 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0bcc1b7310 msgr2=0x7f0bcc1b9700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0bcc1b7310 0x7f0bcc1b9700 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b6c90 osd=5 1 2024-04-26T16:29:02.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.6 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f0bcc1ba360 msgr2=0x7f0bcc1ba7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f0bcc1ba360 0x7f0bcc1ba7c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1b9c40 osd=6 1 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter close_session for osd.7 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0bcc1b18e0 msgr2=0x7f0bcc1b1d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0bcc1b18e0 0x7f0bcc1b1d60 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter put_session s=0x7f0bcc1bfa10 osd=7 1 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 20 client.30624.objecter shutdown clearing up homeless session... 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 10 client.30624.objecter successfully canceled tick 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bb4071f80 msgr2=0x7f0bb4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bb4071f80 0x7f0bb4074440 secure :-1 s=READY pgs=3751 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc005950 tx=0x7f0bbc0058e0 comp rx=0 tx=0).stop 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bcc1094b0 msgr2=0x7f0bcc19fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 --2- 172.21.15.50:0/3278551714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bcc1094b0 0x7f0bcc19fb60 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400d010 tx=0x7f0bc4004d00 comp rx=0 tx=0).stop 2024-04-26T16:29:02.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bcb59e700 1 -- 172.21.15.50:0/3278551714 reap_dead start 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 shutdown_connections 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 >> 172.21.15.50:0/3278551714 conn(0x7f0bcc078990 msgr2=0x7f0bcc106280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 shutdown_connections 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 -- 172.21.15.50:0/3278551714 wait complete. 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:02.489+0000 7f0bd1093700 1 librados: shutdown 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:29:02.492 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:29:02.493 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:29:02.493 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:29:02.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:02.573 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.665481+0000 mgr.y (mgr.24401) 454 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.665481+0000 mgr.y (mgr.24401) 454 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.920767+0000 osd.2 (osd.2) 221 : cluster [DBG] 5.8 deep-scrub starts 2024-04-26T16:29:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.920767+0000 osd.2 (osd.2) 221 : cluster [DBG] 5.8 deep-scrub starts 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.923282+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.8 deep-scrub ok 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.923282+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.8 deep-scrub ok 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: audit 2024-04-26T16:29:02.304403+0000 mon.a (mon.0) 2753 : audit [DBG] from='client.? 172.21.15.50:0/3278551714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[29234]: audit 2024-04-26T16:29:02.304403+0000 mon.a (mon.0) 2753 : audit [DBG] from='client.? 172.21.15.50:0/3278551714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.665481+0000 mgr.y (mgr.24401) 454 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.665481+0000 mgr.y (mgr.24401) 454 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.920767+0000 osd.2 (osd.2) 221 : cluster [DBG] 5.8 deep-scrub starts 2024-04-26T16:29:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.920767+0000 osd.2 (osd.2) 221 : cluster [DBG] 5.8 deep-scrub starts 2024-04-26T16:29:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.923282+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.8 deep-scrub ok 2024-04-26T16:29:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.923282+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.8 deep-scrub ok 2024-04-26T16:29:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: audit 2024-04-26T16:29:02.304403+0000 mon.a (mon.0) 2753 : audit [DBG] from='client.? 172.21.15.50:0/3278551714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:02 smithi050 bash[21463]: audit 2024-04-26T16:29:02.304403+0000 mon.a (mon.0) 2753 : audit [DBG] from='client.? 172.21.15.50:0/3278551714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:02.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:02.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.665481+0000 mgr.y (mgr.24401) 454 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:02.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.665481+0000 mgr.y (mgr.24401) 454 : cluster [DBG] pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:02.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.920767+0000 osd.2 (osd.2) 221 : cluster [DBG] 5.8 deep-scrub starts 2024-04-26T16:29:02.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.920767+0000 osd.2 (osd.2) 221 : cluster [DBG] 5.8 deep-scrub starts 2024-04-26T16:29:02.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.923282+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.8 deep-scrub ok 2024-04-26T16:29:02.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.923282+0000 osd.2 (osd.2) 222 : cluster [DBG] 5.8 deep-scrub ok 2024-04-26T16:29:02.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: audit 2024-04-26T16:29:02.304403+0000 mon.a (mon.0) 2753 : audit [DBG] from='client.? 172.21.15.50:0/3278551714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:02.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:02 smithi190 bash[23868]: audit 2024-04-26T16:29:02.304403+0000 mon.a (mon.0) 2753 : audit [DBG] from='client.? 172.21.15.50:0/3278551714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:02.703 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:02.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:02.832 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:02.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:02.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:02.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:03.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:03.097 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:03.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:03.229 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:03.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:03.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:03.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:03.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.805544+0000 osd.1 (osd.1) 245 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.805544+0000 osd.1 (osd.1) 245 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.807194+0000 osd.1 (osd.1) 246 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.807194+0000 osd.1 (osd.1) 246 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.995964+0000 osd.3 (osd.3) 291 : cluster [DBG] 109.d deep-scrub starts 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:01.995964+0000 osd.3 (osd.3) 291 : cluster [DBG] 109.d deep-scrub starts 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.003893+0000 osd.3 (osd.3) 292 : cluster [DBG] 109.d deep-scrub ok 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.003893+0000 osd.3 (osd.3) 292 : cluster [DBG] 109.d deep-scrub ok 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.111071+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:29:03.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.111071+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.113357+0000 osd.7 (osd.7) 276 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.113357+0000 osd.7 (osd.7) 276 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.271466+0000 osd.5 (osd.5) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.271466+0000 osd.5 (osd.5) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.271905+0000 osd.5 (osd.5) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.271905+0000 osd.5 (osd.5) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.428592+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.428592+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.429826+0000 osd.0 (osd.0) 275 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.429826+0000 osd.0 (osd.0) 275 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.435419+0000 osd.0 (osd.0) 276 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.435419+0000 osd.0 (osd.0) 276 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.960726+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub starts 2024-04-26T16:29:03.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.960726+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub starts 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.962659+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.a scrub ok 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:02.962659+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.a scrub ok 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.160562+0000 osd.6 (osd.6) 17 : cluster [DBG] 6.a scrub starts 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.160562+0000 osd.6 (osd.6) 17 : cluster [DBG] 6.a scrub starts 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.162084+0000 osd.6 (osd.6) 18 : cluster [DBG] 6.a scrub ok 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.162084+0000 osd.6 (osd.6) 18 : cluster [DBG] 6.a scrub ok 2024-04-26T16:29:03.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.805544+0000 osd.1 (osd.1) 245 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.805544+0000 osd.1 (osd.1) 245 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.807194+0000 osd.1 (osd.1) 246 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.807194+0000 osd.1 (osd.1) 246 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.995964+0000 osd.3 (osd.3) 291 : cluster [DBG] 109.d deep-scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:01.995964+0000 osd.3 (osd.3) 291 : cluster [DBG] 109.d deep-scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.003893+0000 osd.3 (osd.3) 292 : cluster [DBG] 109.d deep-scrub ok 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.003893+0000 osd.3 (osd.3) 292 : cluster [DBG] 109.d deep-scrub ok 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.111071+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.111071+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.113357+0000 osd.7 (osd.7) 276 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.113357+0000 osd.7 (osd.7) 276 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.271466+0000 osd.5 (osd.5) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.271466+0000 osd.5 (osd.5) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:03.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.271905+0000 osd.5 (osd.5) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.271905+0000 osd.5 (osd.5) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.428592+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.428592+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.429826+0000 osd.0 (osd.0) 275 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.429826+0000 osd.0 (osd.0) 275 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.435419+0000 osd.0 (osd.0) 276 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.435419+0000 osd.0 (osd.0) 276 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.960726+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub starts 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.960726+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub starts 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.962659+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.a scrub ok 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:02.962659+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.a scrub ok 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.160562+0000 osd.6 (osd.6) 17 : cluster [DBG] 6.a scrub starts 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.160562+0000 osd.6 (osd.6) 17 : cluster [DBG] 6.a scrub starts 2024-04-26T16:29:03.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.162084+0000 osd.6 (osd.6) 18 : cluster [DBG] 6.a scrub ok 2024-04-26T16:29:03.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:03 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.162084+0000 osd.6 (osd.6) 18 : cluster [DBG] 6.a scrub ok 2024-04-26T16:29:03.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:03.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:03.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:03.731 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:03.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:03.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.805544+0000 osd.1 (osd.1) 245 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.805544+0000 osd.1 (osd.1) 245 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.807194+0000 osd.1 (osd.1) 246 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.807194+0000 osd.1 (osd.1) 246 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.995964+0000 osd.3 (osd.3) 291 : cluster [DBG] 109.d deep-scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:01.995964+0000 osd.3 (osd.3) 291 : cluster [DBG] 109.d deep-scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.003893+0000 osd.3 (osd.3) 292 : cluster [DBG] 109.d deep-scrub ok 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.003893+0000 osd.3 (osd.3) 292 : cluster [DBG] 109.d deep-scrub ok 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.111071+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.111071+0000 osd.7 (osd.7) 275 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.113357+0000 osd.7 (osd.7) 276 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.113357+0000 osd.7 (osd.7) 276 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.271466+0000 osd.5 (osd.5) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:03.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.271466+0000 osd.5 (osd.5) 205 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.271905+0000 osd.5 (osd.5) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.271905+0000 osd.5 (osd.5) 206 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.428592+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.428592+0000 mon.a (mon.0) 2754 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.429826+0000 osd.0 (osd.0) 275 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.429826+0000 osd.0 (osd.0) 275 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.435419+0000 osd.0 (osd.0) 276 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.435419+0000 osd.0 (osd.0) 276 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.960726+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.960726+0000 osd.2 (osd.2) 223 : cluster [DBG] 5.a scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.962659+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.a scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:02.962659+0000 osd.2 (osd.2) 224 : cluster [DBG] 5.a scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.160562+0000 osd.6 (osd.6) 17 : cluster [DBG] 6.a scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.160562+0000 osd.6 (osd.6) 17 : cluster [DBG] 6.a scrub starts 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.162084+0000 osd.6 (osd.6) 18 : cluster [DBG] 6.a scrub ok 2024-04-26T16:29:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:03 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.162084+0000 osd.6 (osd.6) 18 : cluster [DBG] 6.a scrub ok 2024-04-26T16:29:03.866 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:03.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:03.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:03.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:04.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:04.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:04.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:04.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:04.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:04.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:04.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:04.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:04.523 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:04.529 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:29:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.039056+0000 osd.3 (osd.3) 293 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.039056+0000 osd.3 (osd.3) 293 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.044067+0000 osd.3 (osd.3) 294 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.044067+0000 osd.3 (osd.3) 294 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.075180+0000 osd.7 (osd.7) 277 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.075180+0000 osd.7 (osd.7) 277 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.080392+0000 osd.7 (osd.7) 278 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:29:04.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.080392+0000 osd.7 (osd.7) 278 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.396493+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.396493+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.398118+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.398118+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.666393+0000 mgr.y (mgr.24401) 455 : cluster [DBG] pgmap v687: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.666393+0000 mgr.y (mgr.24401) 455 : cluster [DBG] pgmap v687: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.996569+0000 osd.2 (osd.2) 225 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.996569+0000 osd.2 (osd.2) 225 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.998951+0000 osd.2 (osd.2) 226 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[21463]: cluster 2024-04-26T16:29:03.998951+0000 osd.2 (osd.2) 226 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.039056+0000 osd.3 (osd.3) 293 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:29:04.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.039056+0000 osd.3 (osd.3) 293 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.044067+0000 osd.3 (osd.3) 294 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.044067+0000 osd.3 (osd.3) 294 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.075180+0000 osd.7 (osd.7) 277 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.075180+0000 osd.7 (osd.7) 277 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.080392+0000 osd.7 (osd.7) 278 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.080392+0000 osd.7 (osd.7) 278 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.396493+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.396493+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.398118+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.398118+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.666393+0000 mgr.y (mgr.24401) 455 : cluster [DBG] pgmap v687: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.666393+0000 mgr.y (mgr.24401) 455 : cluster [DBG] pgmap v687: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.996569+0000 osd.2 (osd.2) 225 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.996569+0000 osd.2 (osd.2) 225 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:29:04.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.998951+0000 osd.2 (osd.2) 226 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:29:04.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:04 smithi050 bash[29234]: cluster 2024-04-26T16:29:03.998951+0000 osd.2 (osd.2) 226 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:29:04.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.039056+0000 osd.3 (osd.3) 293 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:29:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.039056+0000 osd.3 (osd.3) 293 : cluster [DBG] 109.1f scrub starts 2024-04-26T16:29:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.044067+0000 osd.3 (osd.3) 294 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:29:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.044067+0000 osd.3 (osd.3) 294 : cluster [DBG] 109.1f scrub ok 2024-04-26T16:29:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.075180+0000 osd.7 (osd.7) 277 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:29:04.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.075180+0000 osd.7 (osd.7) 277 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.080392+0000 osd.7 (osd.7) 278 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.080392+0000 osd.7 (osd.7) 278 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.396493+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.396493+0000 osd.0 (osd.0) 277 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.398118+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.398118+0000 osd.0 (osd.0) 278 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.666393+0000 mgr.y (mgr.24401) 455 : cluster [DBG] pgmap v687: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.666393+0000 mgr.y (mgr.24401) 455 : cluster [DBG] pgmap v687: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.996569+0000 osd.2 (osd.2) 225 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.996569+0000 osd.2 (osd.2) 225 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.998951+0000 osd.2 (osd.2) 226 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:29:04.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:04 smithi190 bash[23868]: cluster 2024-04-26T16:29:03.998951+0000 osd.2 (osd.2) 226 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:29:04.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:04.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:04.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:04.805 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:04.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:04.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:04.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:05.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:05.061 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:05.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:05.191 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:05.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:05.317 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:05.420 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:05.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:05.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:05.582 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.015173+0000 osd.3 (osd.3) 295 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.015173+0000 osd.3 (osd.3) 295 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.018770+0000 osd.3 (osd.3) 296 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.018770+0000 osd.3 (osd.3) 296 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.484302+0000 osd.4 (osd.4) 251 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.484302+0000 osd.4 (osd.4) 251 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.486369+0000 osd.4 (osd.4) 252 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.486369+0000 osd.4 (osd.4) 252 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:05.007937+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:05.007937+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:05.010039+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:29:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[29234]: cluster 2024-04-26T16:29:05.010039+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.015173+0000 osd.3 (osd.3) 295 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.015173+0000 osd.3 (osd.3) 295 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.018770+0000 osd.3 (osd.3) 296 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.018770+0000 osd.3 (osd.3) 296 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.484302+0000 osd.4 (osd.4) 251 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.484302+0000 osd.4 (osd.4) 251 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.486369+0000 osd.4 (osd.4) 252 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.486369+0000 osd.4 (osd.4) 252 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:05.007937+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:05.007937+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:29:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:05.010039+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:29:05.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:05 smithi050 bash[21463]: cluster 2024-04-26T16:29:05.010039+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:29:05.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.015173+0000 osd.3 (osd.3) 295 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.015173+0000 osd.3 (osd.3) 295 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.018770+0000 osd.3 (osd.3) 296 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.018770+0000 osd.3 (osd.3) 296 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.484302+0000 osd.4 (osd.4) 251 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.484302+0000 osd.4 (osd.4) 251 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:29:05.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.486369+0000 osd.4 (osd.4) 252 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:29:05.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.486369+0000 osd.4 (osd.4) 252 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:29:05.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:05.007937+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:29:05.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:05.007937+0000 osd.2 (osd.2) 227 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:29:05.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:05.010039+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:29:05.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:05 smithi190 bash[23868]: cluster 2024-04-26T16:29:05.010039+0000 osd.2 (osd.2) 228 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:29:05.714 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:05.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:05.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:05.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:05.978 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:05.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:06.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:06.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:06.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:06.239 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:06.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:06.342 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:06.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:06.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:06.499 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.783854+0000 osd.1 (osd.1) 247 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.783854+0000 osd.1 (osd.1) 247 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.788605+0000 osd.1 (osd.1) 248 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:04.788605+0000 osd.1 (osd.1) 248 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:05.667209+0000 mgr.y (mgr.24401) 456 : cluster [DBG] pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 704 B/s rd, 0 op/s 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:05.667209+0000 mgr.y (mgr.24401) 456 : cluster [DBG] pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 704 B/s rd, 0 op/s 2024-04-26T16:29:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:06.032564+0000 osd.2 (osd.2) 229 : cluster [DBG] 5.d scrub starts 2024-04-26T16:29:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:06.032564+0000 osd.2 (osd.2) 229 : cluster [DBG] 5.d scrub starts 2024-04-26T16:29:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:06.035569+0000 osd.2 (osd.2) 230 : cluster [DBG] 5.d scrub ok 2024-04-26T16:29:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:06.035569+0000 osd.2 (osd.2) 230 : cluster [DBG] 5.d scrub ok 2024-04-26T16:29:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:06.172986+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-04-26T16:29:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[23868]: cluster 2024-04-26T16:29:06.172986+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-04-26T16:29:06.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:06.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.783854+0000 osd.1 (osd.1) 247 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:29:06.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.783854+0000 osd.1 (osd.1) 247 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:29:06.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.788605+0000 osd.1 (osd.1) 248 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:29:06.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:04.788605+0000 osd.1 (osd.1) 248 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:05.667209+0000 mgr.y (mgr.24401) 456 : cluster [DBG] pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 704 B/s rd, 0 op/s 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:05.667209+0000 mgr.y (mgr.24401) 456 : cluster [DBG] pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 704 B/s rd, 0 op/s 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:06.032564+0000 osd.2 (osd.2) 229 : cluster [DBG] 5.d scrub starts 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:06.032564+0000 osd.2 (osd.2) 229 : cluster [DBG] 5.d scrub starts 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:06.035569+0000 osd.2 (osd.2) 230 : cluster [DBG] 5.d scrub ok 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:06.035569+0000 osd.2 (osd.2) 230 : cluster [DBG] 5.d scrub ok 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:06.172986+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[21463]: cluster 2024-04-26T16:29:06.172986+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.783854+0000 osd.1 (osd.1) 247 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.783854+0000 osd.1 (osd.1) 247 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.788605+0000 osd.1 (osd.1) 248 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:29:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:04.788605+0000 osd.1 (osd.1) 248 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:05.667209+0000 mgr.y (mgr.24401) 456 : cluster [DBG] pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 704 B/s rd, 0 op/s 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:05.667209+0000 mgr.y (mgr.24401) 456 : cluster [DBG] pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 704 B/s rd, 0 op/s 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:06.032564+0000 osd.2 (osd.2) 229 : cluster [DBG] 5.d scrub starts 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:06.032564+0000 osd.2 (osd.2) 229 : cluster [DBG] 5.d scrub starts 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:06.035569+0000 osd.2 (osd.2) 230 : cluster [DBG] 5.d scrub ok 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:06.035569+0000 osd.2 (osd.2) 230 : cluster [DBG] 5.d scrub ok 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:06.172986+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-04-26T16:29:06.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:06 smithi050 bash[29234]: cluster 2024-04-26T16:29:06.172986+0000 mon.a (mon.0) 2755 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2024-04-26T16:29:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:06.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:06.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:06.762 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:06.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:06.894 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:06.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:06.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:07.024 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:07.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:07.136 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:29:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:29:07.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:07.255 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:07.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:07.381 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:07.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:07.506 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:07.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: audit 2024-04-26T16:29:06.993169+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: audit 2024-04-26T16:29:06.993169+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.073650+0000 osd.2 (osd.2) 231 : cluster [DBG] 2.10 deep-scrub starts 2024-04-26T16:29:07.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.073650+0000 osd.2 (osd.2) 231 : cluster [DBG] 2.10 deep-scrub starts 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.075493+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.10 deep-scrub ok 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.075493+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.10 deep-scrub ok 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.226782+0000 osd.6 (osd.6) 19 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.226782+0000 osd.6 (osd.6) 19 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.228352+0000 osd.6 (osd.6) 20 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.228352+0000 osd.6 (osd.6) 20 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: audit 2024-04-26T16:29:06.993169+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: audit 2024-04-26T16:29:06.993169+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.073650+0000 osd.2 (osd.2) 231 : cluster [DBG] 2.10 deep-scrub starts 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.073650+0000 osd.2 (osd.2) 231 : cluster [DBG] 2.10 deep-scrub starts 2024-04-26T16:29:07.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.075493+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.10 deep-scrub ok 2024-04-26T16:29:07.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.075493+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.10 deep-scrub ok 2024-04-26T16:29:07.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.226782+0000 osd.6 (osd.6) 19 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:29:07.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.226782+0000 osd.6 (osd.6) 19 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:29:07.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.228352+0000 osd.6 (osd.6) 20 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:29:07.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:07 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.228352+0000 osd.6 (osd.6) 20 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:29:07.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:07.661 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:07.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:07.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:07.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: audit 2024-04-26T16:29:06.993169+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: audit 2024-04-26T16:29:06.993169+0000 mon.c (mon.2) 620 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.073650+0000 osd.2 (osd.2) 231 : cluster [DBG] 2.10 deep-scrub starts 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.073650+0000 osd.2 (osd.2) 231 : cluster [DBG] 2.10 deep-scrub starts 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.075493+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.10 deep-scrub ok 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.075493+0000 osd.2 (osd.2) 232 : cluster [DBG] 2.10 deep-scrub ok 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.226782+0000 osd.6 (osd.6) 19 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.226782+0000 osd.6 (osd.6) 19 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.228352+0000 osd.6 (osd.6) 20 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:29:07.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:07 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.228352+0000 osd.6 (osd.6) 20 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:29:07.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:07.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:07.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:07.978 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:29:08.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:08.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:08.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:08.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.304 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:08.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.432 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:06.899721+0000 mgr.y (mgr.24401) 457 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:06.899721+0000 mgr.y (mgr.24401) 457 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.491535+0000 osd.4 (osd.4) 253 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.491535+0000 osd.4 (osd.4) 253 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.493405+0000 osd.4 (osd.4) 254 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.493405+0000 osd.4 (osd.4) 254 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.668238+0000 mgr.y (mgr.24401) 458 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:07.668238+0000 mgr.y (mgr.24401) 458 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.727820+0000 mon.c (mon.2) 621 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.727820+0000 mon.c (mon.2) 621 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.728452+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:06.899721+0000 mgr.y (mgr.24401) 457 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:08.458 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:06.899721+0000 mgr.y (mgr.24401) 457 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:08.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:06.899721+0000 mgr.y (mgr.24401) 457 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:06.899721+0000 mgr.y (mgr.24401) 457 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.491535+0000 osd.4 (osd.4) 253 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.491535+0000 osd.4 (osd.4) 253 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.493405+0000 osd.4 (osd.4) 254 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.493405+0000 osd.4 (osd.4) 254 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.668238+0000 mgr.y (mgr.24401) 458 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:07.668238+0000 mgr.y (mgr.24401) 458 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.727820+0000 mon.c (mon.2) 621 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.727820+0000 mon.c (mon.2) 621 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.728452+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.728452+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.730708+0000 mon.c (mon.2) 622 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.730708+0000 mon.c (mon.2) 622 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.731315+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.731315+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.733318+0000 mon.c (mon.2) 623 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.733318+0000 mon.c (mon.2) 623 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.733882+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.733882+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.736090+0000 mon.c (mon.2) 624 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.736090+0000 mon.c (mon.2) 624 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.736751+0000 mon.c (mon.2) 625 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.736751+0000 mon.c (mon.2) 625 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.736794+0000 mon.a (mon.0) 2759 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.736794+0000 mon.a (mon.0) 2759 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.737835+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.737835+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.740166+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.740166+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.740846+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.740846+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.744901+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.744901+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.752207+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.752207+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.752908+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.752908+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.754988+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.754988+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.755747+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.755747+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.764189+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.764189+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.769600+0000 mon.c (mon.2) 629 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.769600+0000 mon.c (mon.2) 629 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.771573+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:08.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.771573+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:08.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.782874+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: audit 2024-04-26T16:29:07.782874+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.215221+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:29:08.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.215221+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:29:08.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.217146+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:29:08.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:08 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.217146+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:29:08.561 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:08.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:08.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:08.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:08.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.728452+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.730708+0000 mon.c (mon.2) 622 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.730708+0000 mon.c (mon.2) 622 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.731315+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.731315+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.733318+0000 mon.c (mon.2) 623 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.733318+0000 mon.c (mon.2) 623 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.733882+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.733882+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.736090+0000 mon.c (mon.2) 624 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.736090+0000 mon.c (mon.2) 624 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.736751+0000 mon.c (mon.2) 625 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.736751+0000 mon.c (mon.2) 625 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.736794+0000 mon.a (mon.0) 2759 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.736794+0000 mon.a (mon.0) 2759 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.737835+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.737835+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.491535+0000 osd.4 (osd.4) 253 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.491535+0000 osd.4 (osd.4) 253 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.493405+0000 osd.4 (osd.4) 254 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.493405+0000 osd.4 (osd.4) 254 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.668238+0000 mgr.y (mgr.24401) 458 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:08.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:07.668238+0000 mgr.y (mgr.24401) 458 : cluster [DBG] pgmap v690: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.727820+0000 mon.c (mon.2) 621 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.727820+0000 mon.c (mon.2) 621 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.728452+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.728452+0000 mon.a (mon.0) 2756 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.730708+0000 mon.c (mon.2) 622 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.730708+0000 mon.c (mon.2) 622 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.731315+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.731315+0000 mon.a (mon.0) 2757 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.733318+0000 mon.c (mon.2) 623 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.733318+0000 mon.c (mon.2) 623 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.733882+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.733882+0000 mon.a (mon.0) 2758 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.740166+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.740166+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.740846+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.740846+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.744901+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.744901+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.752207+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.752207+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.752908+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.813 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.752908+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.754988+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.754988+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.755747+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.755747+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.764189+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.764189+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.769600+0000 mon.c (mon.2) 629 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.769600+0000 mon.c (mon.2) 629 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.771573+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.771573+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.782874+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: audit 2024-04-26T16:29:07.782874+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.215221+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:29:08.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.215221+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.217146+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.217146+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.736090+0000 mon.c (mon.2) 624 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.736090+0000 mon.c (mon.2) 624 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.736751+0000 mon.c (mon.2) 625 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.736751+0000 mon.c (mon.2) 625 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.736794+0000 mon.a (mon.0) 2759 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.815 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.736794+0000 mon.a (mon.0) 2759 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.737835+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.737835+0000 mon.a (mon.0) 2760 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.740166+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.740166+0000 mon.c (mon.2) 626 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.740846+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.740846+0000 mon.a (mon.0) 2761 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.744901+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.744901+0000 mon.a (mon.0) 2762 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.752207+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.752207+0000 mon.c (mon.2) 627 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.752908+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.752908+0000 mon.a (mon.0) 2763 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.754988+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.754988+0000 mon.c (mon.2) 628 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.755747+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.755747+0000 mon.a (mon.0) 2764 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.764189+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.764189+0000 mon.a (mon.0) 2765 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.769600+0000 mon.c (mon.2) 629 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.769600+0000 mon.c (mon.2) 629 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.771573+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.771573+0000 mon.c (mon.2) 630 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.782874+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: audit 2024-04-26T16:29:07.782874+0000 mon.a (mon.0) 2766 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.215221+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.215221+0000 osd.6 (osd.6) 21 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.217146+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:29:08.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:08 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.217146+0000 osd.6 (osd.6) 22 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:29:08.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:08.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:08.956 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:08.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:09.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:09.087 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:09.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:09.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:09.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:09.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:09.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:09.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:09.611 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:09.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cephadm 2024-04-26T16:29:07.738442+0000 mgr.y (mgr.24401) 459 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3293M 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cephadm 2024-04-26T16:29:07.738442+0000 mgr.y (mgr.24401) 459 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3293M 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cephadm 2024-04-26T16:29:07.757332+0000 mgr.y (mgr.24401) 460 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3292M 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cephadm 2024-04-26T16:29:07.757332+0000 mgr.y (mgr.24401) 460 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3292M 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.442418+0000 osd.0 (osd.0) 279 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.442418+0000 osd.0 (osd.0) 279 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:29:09.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.447456+0000 osd.0 (osd.0) 280 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:29:09.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:09 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.447456+0000 osd.0 (osd.0) 280 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:29:09.743 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:09.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:09.874 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:09.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cephadm 2024-04-26T16:29:07.738442+0000 mgr.y (mgr.24401) 459 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3293M 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cephadm 2024-04-26T16:29:07.738442+0000 mgr.y (mgr.24401) 459 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3293M 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cephadm 2024-04-26T16:29:07.757332+0000 mgr.y (mgr.24401) 460 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3292M 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cephadm 2024-04-26T16:29:07.757332+0000 mgr.y (mgr.24401) 460 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3292M 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.442418+0000 osd.0 (osd.0) 279 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.442418+0000 osd.0 (osd.0) 279 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:29:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.447456+0000 osd.0 (osd.0) 280 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.447456+0000 osd.0 (osd.0) 280 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cephadm 2024-04-26T16:29:07.738442+0000 mgr.y (mgr.24401) 459 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3293M 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cephadm 2024-04-26T16:29:07.738442+0000 mgr.y (mgr.24401) 459 : cephadm [INF] Adjusting osd_memory_target on smithi050 to 3293M 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cephadm 2024-04-26T16:29:07.757332+0000 mgr.y (mgr.24401) 460 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3292M 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cephadm 2024-04-26T16:29:07.757332+0000 mgr.y (mgr.24401) 460 : cephadm [INF] Adjusting osd_memory_target on smithi190 to 3292M 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.442418+0000 osd.0 (osd.0) 279 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.442418+0000 osd.0 (osd.0) 279 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.447456+0000 osd.0 (osd.0) 280 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:29:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:09 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.447456+0000 osd.0 (osd.0) 280 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:29:09.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.006 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:10.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.140 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:10.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:10.271 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:10.374 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:10.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:10.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.534 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:10.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:10.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.662 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:10.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 -- 172.21.15.50:0/2447003945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5181034c0 msgr2=0x7ff518103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:10.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/2447003945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5181034c0 0x7ff518103940 secure :-1 s=READY pgs=2665 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098f0 tx=0x7ff514009c00 comp rx=0 tx=0).stop 2024-04-26T16:29:10.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 -- 172.21.15.50:0/2447003945 shutdown_connections 2024-04-26T16:29:10.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/2447003945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 0x7ff518109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:10.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/2447003945 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff518103e80 0x7ff518110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:10.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/2447003945 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff5181034c0 0x7ff518103940 unknown :-1 s=CLOSED pgs=2665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:10.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 -- 172.21.15.50:0/2447003945 >> 172.21.15.50:0/2447003945 conn(0x7ff51807a5d0 msgr2=0x7ff51807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:10.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 -- 172.21.15.50:0/2447003945 shutdown_connections 2024-04-26T16:29:10.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.725+0000 7ff51f5f7700 1 -- 172.21.15.50:0/2447003945 wait complete. 2024-04-26T16:29:10.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 librados: starting msgr at 2024-04-26T16:29:10.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 librados: starting objecter 2024-04-26T16:29:10.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 Processor -- start 2024-04-26T16:29:10.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- start start 2024-04-26T16:29:10.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 librados: setting wanted keys 2024-04-26T16:29:10.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 librados: calling monclient init 2024-04-26T16:29:10.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 0x7ff51819b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff518103e80 0x7ff51819b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 0x7ff51819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff518112b40 con 0x7ff5181034c0 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff5181129c0 con 0x7ff518109560 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff518112cc0 con 0x7ff518103e80 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 0x7ff51819b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51cb92700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff518103e80 0x7ff51819b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 0x7ff51819b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58056/0 (socket says 172.21.15.50:58056) 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 -- 172.21.15.50:0/4166467317 learned_addr learned my addr 172.21.15.50:0/4166467317 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51db94700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 0x7ff51819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:10.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 -- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff518103e80 msgr2=0x7ff51819b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff518103e80 0x7ff51819b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 -- 172.21.15.50:0/4166467317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 msgr2=0x7ff51819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 0x7ff51819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 -- 172.21.15.50:0/4166467317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5181a03d0 con 0x7ff5181034c0 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51db94700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 0x7ff51819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51cb92700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff518103e80 0x7ff51819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51d393700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 0x7ff51819b3c0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7ff50800d980 tx=0x7ff50800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff50e7fc700 10 client.?.objecter ms_handle_connect 0x7ff5181034c0 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff50e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508017070 con 0x7ff5181034c0 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5181a06c0 con 0x7ff5181034c0 2024-04-26T16:29:10.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff508005020 con 0x7ff5181034c0 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508009990 con 0x7ff5181034c0 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 10 client.30630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5181a7f40 con 0x7ff5181034c0 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 librados: init done 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.729+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff518104bd0 con 0x7ff5181034c0 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff508009b30 con 0x7ff5181034c0 2024-04-26T16:29:10.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff504071f80 0x7ff504074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(467..467 src has 87..467) ==== 9405+0+0 (secure 0 0 0) 0x7ff508091950 con 0x7ff5181034c0 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 10 client.30630.objecter ms_dispatch 0x7ff518001550 osd_map(467..467 src has 87..467) 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 3 client.30630.objecter handle_osd_map got epochs [467,467] > 0 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 3 client.30630.objecter handle_osd_map decoding full epoch 467 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff51cb92700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff504071f80 0x7ff504074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 20 client.30630.objecter dump_active .. 0 homeless 2024-04-26T16:29:10.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff50e7fc700 10 client.30630.objecter ms_handle_connect 0x7ff504071f80 2024-04-26T16:29:10.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.733+0000 7ff51cb92700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff504071f80 0x7ff504074440 secure :-1 s=READY pgs=3752 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098c0 tx=0x7ff514019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:10.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.737+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff508020020 con 0x7ff5181034c0 2024-04-26T16:29:10.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.739860+0000 osd.1 (osd.1) 249 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.739860+0000 osd.1 (osd.1) 249 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.742941+0000 osd.1 (osd.1) 250 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:08.742941+0000 osd.1 (osd.1) 250 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.033546+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.033546+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:29:10.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.035807+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.035807+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.053595+0000 osd.7 (osd.7) 279 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.053595+0000 osd.7 (osd.7) 279 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.053929+0000 osd.7 (osd.7) 280 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.053929+0000 osd.7 (osd.7) 280 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.465524+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.465524+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.482650+0000 osd.4 (osd.4) 255 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.482650+0000 osd.4 (osd.4) 255 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.485651+0000 osd.4 (osd.4) 256 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.485651+0000 osd.4 (osd.4) 256 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.669077+0000 mgr.y (mgr.24401) 461 : cluster [DBG] pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:09.669077+0000 mgr.y (mgr.24401) 461 : cluster [DBG] pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:10.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.205740+0000 osd.6 (osd.6) 23 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.205740+0000 osd.6 (osd.6) 23 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.208214+0000 osd.6 (osd.6) 24 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.208214+0000 osd.6 (osd.6) 24 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.739860+0000 osd.1 (osd.1) 249 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.739860+0000 osd.1 (osd.1) 249 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.742941+0000 osd.1 (osd.1) 250 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:08.742941+0000 osd.1 (osd.1) 250 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:29:10.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.033546+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.033546+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.035807+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.035807+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.053595+0000 osd.7 (osd.7) 279 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.053595+0000 osd.7 (osd.7) 279 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.053929+0000 osd.7 (osd.7) 280 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.053929+0000 osd.7 (osd.7) 280 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.465524+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.465524+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.482650+0000 osd.4 (osd.4) 255 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.482650+0000 osd.4 (osd.4) 255 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.485651+0000 osd.4 (osd.4) 256 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.485651+0000 osd.4 (osd.4) 256 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.669077+0000 mgr.y (mgr.24401) 461 : cluster [DBG] pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:10.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:09.669077+0000 mgr.y (mgr.24401) 461 : cluster [DBG] pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:10.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.205740+0000 osd.6 (osd.6) 23 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:29:10.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.205740+0000 osd.6 (osd.6) 23 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:29:10.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.208214+0000 osd.6 (osd.6) 24 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:29:10.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:10 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.208214+0000 osd.6 (osd.6) 24 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:29:10.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.739860+0000 osd.1 (osd.1) 249 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:29:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.739860+0000 osd.1 (osd.1) 249 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:29:10.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.742941+0000 osd.1 (osd.1) 250 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:08.742941+0000 osd.1 (osd.1) 250 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.033546+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.033546+0000 osd.3 (osd.3) 297 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.035807+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.035807+0000 osd.3 (osd.3) 298 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.053595+0000 osd.7 (osd.7) 279 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.053595+0000 osd.7 (osd.7) 279 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.053929+0000 osd.7 (osd.7) 280 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.053929+0000 osd.7 (osd.7) 280 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.465524+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.465524+0000 mon.a (mon.0) 2767 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.482650+0000 osd.4 (osd.4) 255 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.482650+0000 osd.4 (osd.4) 255 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:29:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.485651+0000 osd.4 (osd.4) 256 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.485651+0000 osd.4 (osd.4) 256 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.669077+0000 mgr.y (mgr.24401) 461 : cluster [DBG] pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:09.669077+0000 mgr.y (mgr.24401) 461 : cluster [DBG] pgmap v692: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.205740+0000 osd.6 (osd.6) 23 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.205740+0000 osd.6 (osd.6) 23 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.208214+0000 osd.6 (osd.6) 24 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:29:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:10 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.208214+0000 osd.6 (osd.6) 24 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:29:10.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.915 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:10.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:10.975 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:10.977+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7ff51819c720 con 0x7ff5181034c0 2024-04-26T16:29:11.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:11.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:11.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:11.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.165+0000 7ff50e7fc700 1 -- 172.21.15.50:0/4166467317 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v468) ==== 87+0+0 (secure 0 0 0) 0x7ff508009e20 con 0x7ff5181034c0 2024-04-26T16:29:11.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 10 librados: watch_flush enter 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 10 librados: watch_flush exit 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 20 client.30630.objecter shutdown clearing up homeless session... 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 10 client.30630.objecter successfully canceled tick 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff504071f80 msgr2=0x7ff504074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff504071f80 0x7ff504074440 secure :-1 s=READY pgs=3752 cs=0 l=1 rev1=1 crypto rx=0x7ff5140098c0 tx=0x7ff514019040 comp rx=0 tx=0).stop 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 msgr2=0x7ff51819b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.169+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 0x7ff51819b3c0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7ff50800d980 tx=0x7ff50800dc90 comp rx=0 tx=0).stop 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 shutdown_connections 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff504071f80 0x7ff504074440 unknown :-1 s=CLOSED pgs=3752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff518109560 0x7ff51819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff5181034c0 0x7ff51819b3c0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 --2- 172.21.15.50:0/4166467317 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff518103e80 0x7ff51819b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 >> 172.21.15.50:0/4166467317 conn(0x7ff51807a5d0 msgr2=0x7ff518075650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:11.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 shutdown_connections 2024-04-26T16:29:11.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 -- 172.21.15.50:0/4166467317 wait complete. 2024-04-26T16:29:11.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:11.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.173+0000 7ff51f5f7700 1 librados: shutdown 2024-04-26T16:29:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:11.176 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:29:11.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:11.300 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:11.364 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:29:11.364 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:29:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 -- 172.21.15.50:0/1348758985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 msgr2=0x7f9748110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1348758985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748110330 secure :-1 s=READY pgs=2666 cs=0 l=1 rev1=1 crypto rx=0x7f9740009f90 tx=0x7f974000b330 comp rx=0 tx=0).stop 2024-04-26T16:29:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 -- 172.21.15.50:0/1348758985 shutdown_connections 2024-04-26T16:29:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1348758985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9748109560 0x7f9748109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1348758985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748110330 unknown :-1 s=CLOSED pgs=2666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1348758985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f97481034c0 0x7f9748103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.381+0000 7f974eb1c700 1 -- 172.21.15.50:0/1348758985 >> 172.21.15.50:0/1348758985 conn(0x7f974807a5d0 msgr2=0x7f974807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- 172.21.15.50:0/1348758985 shutdown_connections 2024-04-26T16:29:11.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- 172.21.15.50:0/1348758985 wait complete. 2024-04-26T16:29:11.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 librados: starting msgr at 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 librados: starting objecter 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 Processor -- start 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- start start 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 librados: setting wanted keys 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 librados: calling monclient init 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97481034c0 0x7f9748075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 0x7f97480761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9748112b40 con 0x7f9748109560 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f97481129c0 con 0x7f97481034c0 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9748112cc0 con 0x7f9748103e80 2024-04-26T16:29:11.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974d0b9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 0x7f97480761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974d0b9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 0x7f97480761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58082/0 (socket says 172.21.15.50:58082) 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974d0b9700 1 -- 172.21.15.50:0/1912847559 learned_addr learned my addr 172.21.15.50:0/1912847559 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974c8b8700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97481034c0 0x7f9748075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 -- 172.21.15.50:0/1912847559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97481034c0 msgr2=0x7f9748075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97481034c0 0x7f9748075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 -- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 msgr2=0x7f97480761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 0x7f97480761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 -- 172.21.15.50:0/1912847559 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97481a7630 con 0x7f9748103e80 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974d0b9700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 0x7f97480761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9747fff700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748075c60 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f973400eb90 tx=0x7f973400eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9745ffb700 10 client.?.objecter ms_handle_connect 0x7f9748103e80 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9745ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:11.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973400ccf0 con 0x7f9748103e80 2024-04-26T16:29:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97481a7920 con 0x7f9748103e80 2024-04-26T16:29:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 10 client.41249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97481a7eb0 con 0x7f9748103e80 2024-04-26T16:29:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 librados: init done 2024-04-26T16:29:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9734004480 con 0x7f9748103e80 2024-04-26T16:29:11.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9734003c10 con 0x7f9748103e80 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.385+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f972c005120 con 0x7f9748103e80 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9734004020 con 0x7f9748103e80 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9738071f80 0x7f9738074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(468..468 src has 87..468) ==== 9405+0+0 (secure 0 0 0) 0x7f9734091cb0 con 0x7f9748103e80 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 10 client.41249.objecter ms_dispatch 0x7f9748001550 osd_map(468..468 src has 87..468) 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 3 client.41249.objecter handle_osd_map got epochs [468,468] > 0 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f974c8b8700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9738071f80 0x7f9738074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 3 client.41249.objecter handle_osd_map decoding full epoch 468 2024-04-26T16:29:11.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 20 client.41249.objecter dump_active .. 0 homeless 2024-04-26T16:29:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f9745ffb700 10 client.41249.objecter ms_handle_connect 0x7f9738071f80 2024-04-26T16:29:11.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.389+0000 7f974c8b8700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9738071f80 0x7f9738074440 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 crypto rx=0x7f973c005fd0 tx=0x7f973c005db0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:11.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.393+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f9734016330 con 0x7f9748103e80 2024-04-26T16:29:11.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:11.431 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:11.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:11.561 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:11.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.002164+0000 osd.3 (osd.3) 299 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:29:11.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.002164+0000 osd.3 (osd.3) 299 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:29:11.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.006573+0000 osd.3 (osd.3) 300 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:29:11.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.006573+0000 osd.3 (osd.3) 300 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:29:11.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.133980+0000 osd.7 (osd.7) 281 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:29:11.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.133980+0000 osd.7 (osd.7) 281 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.136167+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.136167+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.515896+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.515896+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.518347+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.518347+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: audit 2024-04-26T16:29:10.981300+0000 mon.a (mon.0) 2768 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: audit 2024-04-26T16:29:10.981300+0000 mon.a (mon.0) 2768 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:11.168349+0000 mon.a (mon.0) 2769 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:11.168349+0000 mon.a (mon.0) 2769 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: audit 2024-04-26T16:29:11.169984+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: audit 2024-04-26T16:29:11.169984+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:11.174298+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-04-26T16:29:11.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[29234]: cluster 2024-04-26T16:29:11.174298+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.002164+0000 osd.3 (osd.3) 299 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.002164+0000 osd.3 (osd.3) 299 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.006573+0000 osd.3 (osd.3) 300 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.006573+0000 osd.3 (osd.3) 300 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.133980+0000 osd.7 (osd.7) 281 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.133980+0000 osd.7 (osd.7) 281 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.136167+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:29:11.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.136167+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.515896+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.515896+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.518347+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.518347+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: audit 2024-04-26T16:29:10.981300+0000 mon.a (mon.0) 2768 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: audit 2024-04-26T16:29:10.981300+0000 mon.a (mon.0) 2768 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:11.168349+0000 mon.a (mon.0) 2769 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:11.168349+0000 mon.a (mon.0) 2769 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: audit 2024-04-26T16:29:11.169984+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: audit 2024-04-26T16:29:11.169984+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:11.174298+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-04-26T16:29:11.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:11 smithi050 bash[21463]: cluster 2024-04-26T16:29:11.174298+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-04-26T16:29:11.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:11.637+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7f972c005410 con 0x7f9748103e80 2024-04-26T16:29:11.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:11.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:11.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.002164+0000 osd.3 (osd.3) 299 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.002164+0000 osd.3 (osd.3) 299 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.006573+0000 osd.3 (osd.3) 300 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.006573+0000 osd.3 (osd.3) 300 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.133980+0000 osd.7 (osd.7) 281 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.133980+0000 osd.7 (osd.7) 281 : cluster [DBG] 5.9 deep-scrub starts 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.136167+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:29:11.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.136167+0000 osd.7 (osd.7) 282 : cluster [DBG] 5.9 deep-scrub ok 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.515896+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.515896+0000 osd.0 (osd.0) 281 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.518347+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.518347+0000 osd.0 (osd.0) 282 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: audit 2024-04-26T16:29:10.981300+0000 mon.a (mon.0) 2768 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: audit 2024-04-26T16:29:10.981300+0000 mon.a (mon.0) 2768 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:11.168349+0000 mon.a (mon.0) 2769 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:11.168349+0000 mon.a (mon.0) 2769 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:11.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: audit 2024-04-26T16:29:11.169984+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:29:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: audit 2024-04-26T16:29:11.169984+0000 mon.a (mon.0) 2770 : audit [INF] from='client.? 172.21.15.50:0/4166467317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:29:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:11.174298+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-04-26T16:29:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:11 smithi190 bash[23868]: cluster 2024-04-26T16:29:11.174298+0000 mon.a (mon.0) 2771 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2024-04-26T16:29:11.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:11.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:11.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:11.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:12.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:12.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:12.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:12.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:12.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:12.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:12.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:12.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:12.459 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.505+0000 7f9745ffb700 1 -- 172.21.15.50:0/1912847559 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v469) ==== 113+0+0 (secure 0 0 0) 0x7f973405d650 con 0x7f9748103e80 2024-04-26T16:29:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.509+0000 7f974eb1c700 10 librados: watch_flush enter 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 10 librados: watch_flush exit 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 20 client.41249.objecter shutdown clearing up homeless session... 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 10 client.41249.objecter successfully canceled tick 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9738071f80 msgr2=0x7f9738074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9738071f80 0x7f9738074440 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 crypto rx=0x7f973c005fd0 tx=0x7f973c005db0 comp rx=0 tx=0).stop 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 msgr2=0x7f9748075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748075c60 secure :-1 s=READY pgs=2667 cs=0 l=1 rev1=1 crypto rx=0x7f973400eb90 tx=0x7f973400eea0 comp rx=0 tx=0).stop 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 shutdown_connections 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9738071f80 0x7f9738074440 unknown :-1 s=CLOSED pgs=3753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97481034c0 0x7f9748075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9748109560 0x7f97480761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 --2- 172.21.15.50:0/1912847559 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9748103e80 0x7f9748075c60 unknown :-1 s=CLOSED pgs=2667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 >> 172.21.15.50:0/1912847559 conn(0x7f974807a5d0 msgr2=0x7f974810a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 shutdown_connections 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 -- 172.21.15.50:0/1912847559 wait complete. 2024-04-26T16:29:12.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:12.513+0000 7f974eb1c700 1 librados: shutdown 2024-04-26T16:29:12.512 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.748943+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.748943+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.751470+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.751470+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.995265+0000 osd.3 (osd.3) 301 : cluster [DBG] 11.b scrub starts 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.995265+0000 osd.3 (osd.3) 301 : cluster [DBG] 11.b scrub starts 2024-04-26T16:29:12.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.997610+0000 osd.3 (osd.3) 302 : cluster [DBG] 11.b scrub ok 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:10.997610+0000 osd.3 (osd.3) 302 : cluster [DBG] 11.b scrub ok 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.641497+0000 mon.c (mon.2) 631 : audit [INF] from='client.? 172.21.15.50:0/1912847559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.641497+0000 mon.c (mon.2) 631 : audit [INF] from='client.? 172.21.15.50:0/1912847559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.641992+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.641992+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:11.671184+0000 mgr.y (mgr.24401) 462 : cluster [DBG] pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: cluster 2024-04-26T16:29:11.671184+0000 mgr.y (mgr.24401) 462 : cluster [DBG] pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.884820+0000 mon.a (mon.0) 2773 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.884820+0000 mon.a (mon.0) 2773 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.887394+0000 mon.c (mon.2) 632 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[29234]: audit 2024-04-26T16:29:11.887394+0000 mon.c (mon.2) 632 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:12.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.748943+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.748943+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.751470+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.751470+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.995265+0000 osd.3 (osd.3) 301 : cluster [DBG] 11.b scrub starts 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.995265+0000 osd.3 (osd.3) 301 : cluster [DBG] 11.b scrub starts 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.997610+0000 osd.3 (osd.3) 302 : cluster [DBG] 11.b scrub ok 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:10.997610+0000 osd.3 (osd.3) 302 : cluster [DBG] 11.b scrub ok 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.641497+0000 mon.c (mon.2) 631 : audit [INF] from='client.? 172.21.15.50:0/1912847559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.641497+0000 mon.c (mon.2) 631 : audit [INF] from='client.? 172.21.15.50:0/1912847559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.641992+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.641992+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:11.671184+0000 mgr.y (mgr.24401) 462 : cluster [DBG] pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:12.528 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: cluster 2024-04-26T16:29:11.671184+0000 mgr.y (mgr.24401) 462 : cluster [DBG] pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:12.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.884820+0000 mon.a (mon.0) 2773 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:12.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.884820+0000 mon.a (mon.0) 2773 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:12.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.887394+0000 mon.c (mon.2) 632 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:12.529 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:12 smithi050 bash[21463]: audit 2024-04-26T16:29:11.887394+0000 mon.c (mon.2) 632 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:12.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.748943+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:29:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.748943+0000 osd.1 (osd.1) 251 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:29:12.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.751470+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.751470+0000 osd.1 (osd.1) 252 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.995265+0000 osd.3 (osd.3) 301 : cluster [DBG] 11.b scrub starts 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.995265+0000 osd.3 (osd.3) 301 : cluster [DBG] 11.b scrub starts 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.997610+0000 osd.3 (osd.3) 302 : cluster [DBG] 11.b scrub ok 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:10.997610+0000 osd.3 (osd.3) 302 : cluster [DBG] 11.b scrub ok 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.641497+0000 mon.c (mon.2) 631 : audit [INF] from='client.? 172.21.15.50:0/1912847559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.641497+0000 mon.c (mon.2) 631 : audit [INF] from='client.? 172.21.15.50:0/1912847559' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.641992+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.641992+0000 mon.a (mon.0) 2772 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:11.671184+0000 mgr.y (mgr.24401) 462 : cluster [DBG] pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: cluster 2024-04-26T16:29:11.671184+0000 mgr.y (mgr.24401) 462 : cluster [DBG] pgmap v694: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.884820+0000 mon.a (mon.0) 2773 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.884820+0000 mon.a (mon.0) 2773 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:29:12.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.887394+0000 mon.c (mon.2) 632 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:12.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:12 smithi190 bash[23868]: audit 2024-04-26T16:29:11.887394+0000 mon.c (mon.2) 632 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:12.583 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:12.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:12.713 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:12.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:12.845 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:12.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:12.973 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:12.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:13.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:13.105 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:13.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-26T16:29:13.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:13.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:13.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:13.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:13.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:13.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:13.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:13.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[29234]: audit 2024-04-26T16:29:12.499384+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[29234]: audit 2024-04-26T16:29:12.499384+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[29234]: cluster 2024-04-26T16:29:12.507544+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[29234]: cluster 2024-04-26T16:29:12.507544+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[21463]: audit 2024-04-26T16:29:12.499384+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[21463]: audit 2024-04-26T16:29:12.499384+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[21463]: cluster 2024-04-26T16:29:12.507544+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-04-26T16:29:13.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:13 smithi050 bash[21463]: cluster 2024-04-26T16:29:12.507544+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-04-26T16:29:13.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:13.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:13.753 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:13 smithi190 bash[23868]: audit 2024-04-26T16:29:12.499384+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:29:13.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:13 smithi190 bash[23868]: audit 2024-04-26T16:29:12.499384+0000 mon.a (mon.0) 2774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:29:13.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:13 smithi190 bash[23868]: cluster 2024-04-26T16:29:12.507544+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-04-26T16:29:13.754 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:13 smithi190 bash[23868]: cluster 2024-04-26T16:29:12.507544+0000 mon.a (mon.0) 2775 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2024-04-26T16:29:13.770 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:13.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:13.900 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:13.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:13.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-26T16:29:14.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:14.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:14.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:14.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:14.163 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:14.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:14.295 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:14.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:14.425 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:14.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:14.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:14.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:14 smithi050 bash[21463]: cluster 2024-04-26T16:29:13.671856+0000 mgr.y (mgr.24401) 463 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:14.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:14 smithi050 bash[21463]: cluster 2024-04-26T16:29:13.671856+0000 mgr.y (mgr.24401) 463 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:14.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:29:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:29:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:29:14.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:14 smithi050 bash[29234]: cluster 2024-04-26T16:29:13.671856+0000 mgr.y (mgr.24401) 463 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:14.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:14 smithi050 bash[29234]: cluster 2024-04-26T16:29:13.671856+0000 mgr.y (mgr.24401) 463 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:14.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:14.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:14 smithi190 bash[23868]: cluster 2024-04-26T16:29:13.671856+0000 mgr.y (mgr.24401) 463 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:14.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:14 smithi190 bash[23868]: cluster 2024-04-26T16:29:13.671856+0000 mgr.y (mgr.24401) 463 : cluster [DBG] pgmap v696: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:14.687 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:14.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:14.791 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:14.817 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:14.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:14.947 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:14.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:15.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:15.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:15.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:15.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:29:15.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:15.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:15.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:15.490 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:15.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:15.622 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:15.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:15.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:15.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:15.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:15.881 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:15.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:15.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:16.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:16.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:16.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:16.141 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:16.244 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:16.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:16.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:16.398 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:16.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:16.531 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:16.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:16.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:14.924301+0000 osd.3 (osd.3) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:16.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:14.924301+0000 osd.3 (osd.3) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:16.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:14.924661+0000 osd.3 (osd.3) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:16.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:14.924661+0000 osd.3 (osd.3) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:16.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:15.515481+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-04-26T16:29:16.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:15.515481+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-04-26T16:29:16.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:15.672649+0000 mgr.y (mgr.24401) 464 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:16.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[23868]: cluster 2024-04-26T16:29:15.672649+0000 mgr.y (mgr.24401) 464 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:16.662 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:16.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:16.792 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:16.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:14.924301+0000 osd.3 (osd.3) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:16.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:14.924301+0000 osd.3 (osd.3) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:16.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:14.924661+0000 osd.3 (osd.3) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:14.924661+0000 osd.3 (osd.3) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:15.515481+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:15.515481+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:15.672649+0000 mgr.y (mgr.24401) 464 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[29234]: cluster 2024-04-26T16:29:15.672649+0000 mgr.y (mgr.24401) 464 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:14.924301+0000 osd.3 (osd.3) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:14.924301+0000 osd.3 (osd.3) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:14.924661+0000 osd.3 (osd.3) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:14.924661+0000 osd.3 (osd.3) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:15.515481+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-04-26T16:29:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:15.515481+0000 mon.a (mon.0) 2776 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2024-04-26T16:29:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:15.672649+0000 mgr.y (mgr.24401) 464 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:16 smithi050 bash[21463]: cluster 2024-04-26T16:29:15.672649+0000 mgr.y (mgr.24401) 464 : cluster [DBG] pgmap v698: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:29:16.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:16.924 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:16.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:17.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:17.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:17.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:17.169 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:29:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:29:17.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:17.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:17.315 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:17.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:17.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:17.549 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:17.577 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:17.680 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:17.708 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:17.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:17.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:17.838 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:17.839 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:17.839 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:17.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:17.952 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[29234]: audit 2024-04-26T16:29:16.901224+0000 mgr.y (mgr.24401) 465 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:17.952 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[29234]: audit 2024-04-26T16:29:16.901224+0000 mgr.y (mgr.24401) 465 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:17.952 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[29234]: cluster 2024-04-26T16:29:17.674829+0000 mgr.y (mgr.24401) 466 : cluster [DBG] pgmap v699: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:29:17.952 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[29234]: cluster 2024-04-26T16:29:17.674829+0000 mgr.y (mgr.24401) 466 : cluster [DBG] pgmap v699: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:29:17.952 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[21463]: audit 2024-04-26T16:29:16.901224+0000 mgr.y (mgr.24401) 465 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:17.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[21463]: audit 2024-04-26T16:29:16.901224+0000 mgr.y (mgr.24401) 465 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:17.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[21463]: cluster 2024-04-26T16:29:17.674829+0000 mgr.y (mgr.24401) 466 : cluster [DBG] pgmap v699: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:29:17.953 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:17 smithi050 bash[21463]: cluster 2024-04-26T16:29:17.674829+0000 mgr.y (mgr.24401) 466 : cluster [DBG] pgmap v699: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:29:17.966 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:17.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:17.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:18.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:18.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:17 smithi190 bash[23868]: audit 2024-04-26T16:29:16.901224+0000 mgr.y (mgr.24401) 465 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:18.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:17 smithi190 bash[23868]: audit 2024-04-26T16:29:16.901224+0000 mgr.y (mgr.24401) 465 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:18.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:17 smithi190 bash[23868]: cluster 2024-04-26T16:29:17.674829+0000 mgr.y (mgr.24401) 466 : cluster [DBG] pgmap v699: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:29:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:17 smithi190 bash[23868]: cluster 2024-04-26T16:29:17.674829+0000 mgr.y (mgr.24401) 466 : cluster [DBG] pgmap v699: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:29:18.098 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:18.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:18.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:18.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:18.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:18.362 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:18.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:18.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:18.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:18.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:18.625 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:18.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:18.754 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:18.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:18.885 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:18.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:18.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:19.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:19.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:19.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:19.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:19.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:19.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:19.409 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:19.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:19.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:19.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:19.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:19.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:19.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:19.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:19 smithi050 bash[29234]: cluster 2024-04-26T16:29:19.675600+0000 mgr.y (mgr.24401) 467 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:29:19.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:19 smithi050 bash[29234]: cluster 2024-04-26T16:29:19.675600+0000 mgr.y (mgr.24401) 467 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:29:19.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:19 smithi050 bash[21463]: cluster 2024-04-26T16:29:19.675600+0000 mgr.y (mgr.24401) 467 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:29:19.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:19 smithi050 bash[21463]: cluster 2024-04-26T16:29:19.675600+0000 mgr.y (mgr.24401) 467 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:29:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:19.889 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:19.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:19.931 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:19.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:20.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:20.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:19 smithi190 bash[23868]: cluster 2024-04-26T16:29:19.675600+0000 mgr.y (mgr.24401) 467 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:29:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:19 smithi190 bash[23868]: cluster 2024-04-26T16:29:19.675600+0000 mgr.y (mgr.24401) 467 : cluster [DBG] pgmap v700: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:29:20.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:20.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:20.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:20.322 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:20.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:20.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:20.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:20.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- 172.21.15.50:0/3971080901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b48104380 msgr2=0x7f6b4806bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/3971080901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b48104380 0x7f6b4806bb50 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009990 tx=0x7f6b38009ca0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- 172.21.15.50:0/3971080901 shutdown_connections 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/3971080901 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4810fd40 0x7f6b48112130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/3971080901 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b48104380 0x7f6b4806bb50 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/3971080901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b4806c090 0x7f6b4806c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- 172.21.15.50:0/3971080901 >> 172.21.15.50:0/3971080901 conn(0x7f6b480790f0 msgr2=0x7f6b48079500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:20.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- 172.21.15.50:0/3971080901 shutdown_connections 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- 172.21.15.50:0/3971080901 wait complete. 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 librados: starting msgr at 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 librados: starting objecter 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 Processor -- start 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- start start 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 librados: setting wanted keys 2024-04-26T16:29:20.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 librados: calling monclient init 2024-04-26T16:29:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 0x7f6b4819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48104380 0x7f6b4819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b4810fd40 0x7f6b4819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48114d90 con 0x7f6b4806c090 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6b48114c10 con 0x7f6b48104380 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4cee1700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6b48114f10 con 0x7f6b4810fd40 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b46d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b4810fd40 0x7f6b4819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 0x7f6b4819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 0x7f6b4819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58166/0 (socket says 172.21.15.50:58166) 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b4659c700 1 -- 172.21.15.50:0/705594443 learned_addr learned my addr 172.21.15.50:0/705594443 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b46d9d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b4810fd40 0x7f6b4819fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57088/0 (socket says 172.21.15.50:57088) 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.652+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48104380 0x7f6b4819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.655 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4659c700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b4810fd40 msgr2=0x7f6b4819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b4810fd40 0x7f6b4819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4659c700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48104380 msgr2=0x7f6b4819b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6b48104380 0x7f6b4819b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4659c700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b481a0350 con 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6b4810fd40 0x7f6b4819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 0x7f6b4819b340 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f6b38000c00 tx=0x7f6b3800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 10 client.?.objecter ms_handle_connect 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38027070 con 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6b38022470 con 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b481a05e0 con 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38022e60 con 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 10 client.30642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481a7ec0 con 0x7f6b4806c090 2024-04-26T16:29:20.656 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 1 librados: init done 2024-04-26T16:29:20.657 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6b48102970 con 0x7f6b4806c090 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6b3802a070 con 0x7f6b4806c090 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6b30071f80 0x7f6b30074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(470..470 src has 87..470) ==== 9405+0+0 (secure 0 0 0) 0x7f6b38092d40 con 0x7f6b4806c090 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 osd_map(470..470 src has 87..470) 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 3 client.30642.objecter handle_osd_map got epochs [470,470] > 0 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 3 client.30642.objecter handle_osd_map decoding full epoch 470 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6b30071f80 0x7f6b30074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 20 client.30642.objecter dump_active .. 0 homeless 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v470) ==== 54+0+16 (secure 0 0 0) 0x7f6b38013070 con 0x7f6b4806c090 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 1 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6b48064030 0x7f6b48064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 1 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 0 1 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 4 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 10 client.30642.objecter _send_command 1 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b4806a110 con 0x7f6b48064030 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6b30071f80 0x7f6b30074440 secure :-1 s=READY pgs=3754 cs=0 l=1 rev1=1 crypto rx=0x7f6b480799f0 tx=0x7f6b3c005eb0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.656+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b30071f80 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6b48064030 0x7f6b48064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6b48064030 0x7f6b48064410 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b48064030 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b4806a110 con 0x7f6b48064030 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 1: 0 ) 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b377fe700 10 client.30642.objecter _finish_command 1 = system:0 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 4 2024-04-26T16:29:20.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.660+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 0 1 2024-04-26T16:29:20.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 2 2024-04-26T16:29:20.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 2 2024-04-26T16:29:20.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b4819c6a0 osd=0 3 2024-04-26T16:29:20.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 0 2 2024-04-26T16:29:20.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 4 2024-04-26T16:29:20.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 10 client.30642.objecter _send_command 2 2024-04-26T16:29:20.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.676+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b48063ab0 con 0x7f6b48064030 2024-04-26T16:29:20.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b48063ab0 con 0x7f6b48064030 2024-04-26T16:29:20.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 10 client.30642.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 4 2024-04-26T16:29:20.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 0 2 2024-04-26T16:29:20.678 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:29:20.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 3 2024-04-26T16:29:20.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f6b4810a950 0x7f6b481ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 3 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 1 3 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 4 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 10 client.30642.objecter _send_command 3 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6b4802e600 con 0x7f6b4810a950 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f6b4810a950 0x7f6b481ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f6b4810a950 0x7f6b481ac0b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b4810a950 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b4802e600 con 0x7f6b4810a950 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 3: 0 ) 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 10 client.30642.objecter _finish_command 3 = system:0 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 4 2024-04-26T16:29:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.680+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 1 3 2024-04-26T16:29:20.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:20.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 4 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 4 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b4802e490 osd=1 3 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 1 4 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 4 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 10 client.30642.objecter _send_command 4 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481ac5f0 con 0x7f6b4810a950 2024-04-26T16:29:20.699 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:20.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481ac5f0 con 0x7f6b4810a950 2024-04-26T16:29:20.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b377fe700 10 client.30642.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 4 2024-04-26T16:29:20.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 1 4 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 5 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f6b4810a030 0x7f6b481af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 5 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 2 5 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 4 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 10 client.30642.objecter _send_command 5 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f6b4802dff0 con 0x7f6b4810a030 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.700+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f6b4810a030 0x7f6b481af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b4810a030 2024-04-26T16:29:20.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f6b4810a030 0x7f6b481af930 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:20.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b4802dff0 con 0x7f6b4810a030 2024-04-26T16:29:20.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 5: 0 ) 2024-04-26T16:29:20.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b377fe700 10 client.30642.objecter _finish_command 5 = system:0 2024-04-26T16:29:20.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 4 2024-04-26T16:29:20.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.704+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 2 5 2024-04-26T16:29:20.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 6 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 6 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b48109ec0 osd=2 3 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 2 6 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 4 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 10 client.30642.objecter _send_command 6 2024-04-26T16:29:20.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.720+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481afe70 con 0x7f6b4810a030 2024-04-26T16:29:20.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481afe70 con 0x7f6b4810a030 2024-04-26T16:29:20.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 10 client.30642.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 4 2024-04-26T16:29:20.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 2 6 2024-04-26T16:29:20.722 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 7 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f6b481acf70 0x7f6b481ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 7 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 3 7 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 4 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 10 client.30642.objecter _send_command 7 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f6b481b1310 con 0x7f6b481acf70 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f6b481acf70 0x7f6b481ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b481acf70 2024-04-26T16:29:20.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f6b481acf70 0x7f6b481ad3b0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b481b1310 con 0x7f6b481acf70 2024-04-26T16:29:20.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 7: 0 ) 2024-04-26T16:29:20.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 10 client.30642.objecter _finish_command 7 = system:0 2024-04-26T16:29:20.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 4 2024-04-26T16:29:20.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.724+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 3 7 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 8 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 8 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481afe70 osd=3 3 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 3 8 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 4 2024-04-26T16:29:20.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 10 client.30642.objecter _send_command 8 2024-04-26T16:29:20.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481b26d0 con 0x7f6b481acf70 2024-04-26T16:29:20.742 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481b26d0 con 0x7f6b481acf70 2024-04-26T16:29:20.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b377fe700 10 client.30642.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 4 2024-04-26T16:29:20.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 3 8 2024-04-26T16:29:20.743 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:29:20.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 9 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f6b481acab0 0x7f6b481b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 9 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 4 9 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 4 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 10 client.30642.objecter _send_command 9 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f6b481b9360 con 0x7f6b481acab0 2024-04-26T16:29:20.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f6b481acab0 0x7f6b481b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b481acab0 2024-04-26T16:29:20.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.744+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f6b481acab0 0x7f6b481b2230 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.748+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b481b9360 con 0x7f6b481acab0 2024-04-26T16:29:20.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.748+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 9: 0 ) 2024-04-26T16:29:20.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.748+0000 7f6b377fe700 10 client.30642.objecter _finish_command 9 = system:0 2024-04-26T16:29:20.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.748+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 4 2024-04-26T16:29:20.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.748+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 4 9 2024-04-26T16:29:20.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 10 2024-04-26T16:29:20.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 10 2024-04-26T16:29:20.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481b1310 osd=4 3 2024-04-26T16:29:20.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 4 10 2024-04-26T16:29:20.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 4 2024-04-26T16:29:20.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 10 client.30642.objecter _send_command 10 2024-04-26T16:29:20.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.764+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481b6a40 con 0x7f6b481acab0 2024-04-26T16:29:20.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481b6a40 con 0x7f6b481acab0 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b377fe700 10 client.30642.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 4 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 4 10 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 11 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f6b481b89b0 0x7f6b481b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 11 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 5 11 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 4 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 10 client.30642.objecter _send_command 11 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.768+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f6b481bc6d0 con 0x7f6b481b89b0 2024-04-26T16:29:20.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f6b481b89b0 0x7f6b481b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b481b89b0 2024-04-26T16:29:20.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b45d9b700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f6b481b89b0 0x7f6b481b8e10 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b481bc6d0 con 0x7f6b481b89b0 2024-04-26T16:29:20.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 11: 0 ) 2024-04-26T16:29:20.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b377fe700 10 client.30642.objecter _finish_command 11 = system:0 2024-04-26T16:29:20.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 4 2024-04-26T16:29:20.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.772+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 5 11 2024-04-26T16:29:20.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 12 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 12 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481b6a40 osd=5 3 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 5 12 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 4 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 10 client.30642.objecter _send_command 12 2024-04-26T16:29:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.792+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481bb550 con 0x7f6b481b89b0 2024-04-26T16:29:20.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481bb550 con 0x7f6b481b89b0 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b377fe700 10 client.30642.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 4 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 5 12 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.795 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 13 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f6b481bbca0 0x7f6b481bc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 13 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 6 13 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 4 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 10 client.30642.objecter _send_command 13 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f6b481bfa40 con 0x7f6b481bbca0 2024-04-26T16:29:20.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f6b481bbca0 0x7f6b481bc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b481bbca0 2024-04-26T16:29:20.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.796+0000 7f6b46d9d700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f6b481bbca0 0x7f6b481bc100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.800+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b481bfa40 con 0x7f6b481bbca0 2024-04-26T16:29:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.800+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 13: 0 ) 2024-04-26T16:29:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.800+0000 7f6b377fe700 10 client.30642.objecter _finish_command 13 = system:0 2024-04-26T16:29:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.800+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 4 2024-04-26T16:29:20.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.800+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 6 13 2024-04-26T16:29:20.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 14 2024-04-26T16:29:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 14 2024-04-26T16:29:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481bb550 osd=6 3 2024-04-26T16:29:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 6 14 2024-04-26T16:29:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 4 2024-04-26T16:29:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 10 client.30642.objecter _send_command 14 2024-04-26T16:29:20.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.816+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481be8c0 con 0x7f6b481bbca0 2024-04-26T16:29:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481be8c0 con 0x7f6b481bbca0 2024-04-26T16:29:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b377fe700 10 client.30642.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 4 2024-04-26T16:29:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 6 14 2024-04-26T16:29:20.821 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:29:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:20.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 15 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f6b481bf010 0x7f6b481bf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 15 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 7 15 2024-04-26T16:29:20.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 4 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 10 client.30642.objecter _send_command 15 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.820+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f6b481c2db0 con 0x7f6b481bf010 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f6b481bf010 0x7f6b481bf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b377fe700 10 client.30642.objecter ms_handle_connect 0x7f6b481bf010 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b4659c700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f6b481bf010 0x7f6b481bf430 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6b481c2db0 con 0x7f6b481bf010 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 15: 0 ) 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b377fe700 10 client.30642.objecter _finish_command 15 = system:0 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 4 2024-04-26T16:29:20.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.824+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 7 15 2024-04-26T16:29:20.832 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:20.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 10 client.30642.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign -1 16 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 20 client.30642.objecter _get_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_remove -1 16 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 20 client.30642.objecter get_session s=0x7f6b481be8c0 osd=7 3 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 15 client.30642.objecter _session_command_op_assign 7 16 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 4 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 10 client.30642.objecter _send_command 16 2024-04-26T16:29:20.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.840+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6b481b6c20 con 0x7f6b481bf010 2024-04-26T16:29:20.843 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b377fe700 1 -- 172.21.15.50:0/705594443 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6b481b6c20 con 0x7f6b481bf010 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b377fe700 10 client.30642.objecter ms_dispatch 0x7f6b48001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b377fe700 10 client.30642.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b377fe700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 4 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b377fe700 15 client.30642.objecter _session_command_op_remove 7 16 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 librados: watch_flush enter 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 librados: watch_flush exit 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.0 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6b48064030 msgr2=0x7f6b48064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6b48064030 0x7f6b48064410 crc :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4819c6a0 osd=0 1 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.1 2024-04-26T16:29:20.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f6b4810a950 msgr2=0x7f6b481ac0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f6b4810a950 0x7f6b481ac0b0 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b4802e490 osd=1 1 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.2 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f6b4810a030 msgr2=0x7f6b481af930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f6b4810a030 0x7f6b481af930 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b48109ec0 osd=2 1 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.3 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b46d9d700 1 -- 172.21.15.50:0/705594443 reap_dead start 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f6b481acf70 msgr2=0x7f6b481ad3b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f6b481acf70 0x7f6b481ad3b0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481afe70 osd=3 1 2024-04-26T16:29:20.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.4 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f6b481acab0 msgr2=0x7f6b481b2230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f6b481acab0 0x7f6b481b2230 crc :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b1310 osd=4 1 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.5 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f6b481b89b0 msgr2=0x7f6b481b8e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f6b481b89b0 0x7f6b481b8e10 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481b6a40 osd=5 1 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.6 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f6b481bbca0 msgr2=0x7f6b481bc100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f6b481bbca0 0x7f6b481bc100 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481bb550 osd=6 1 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter close_session for osd.7 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f6b481bf010 msgr2=0x7f6b481bf430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f6b481bf010 0x7f6b481bf430 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter put_session s=0x7f6b481be8c0 osd=7 1 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 20 client.30642.objecter shutdown clearing up homeless session... 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 10 client.30642.objecter successfully canceled tick 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b46d9d700 1 -- 172.21.15.50:0/705594443 reap_dead start 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6b30071f80 msgr2=0x7f6b30074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6b30071f80 0x7f6b30074440 secure :-1 s=READY pgs=3754 cs=0 l=1 rev1=1 crypto rx=0x7f6b480799f0 tx=0x7f6b3c005eb0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 msgr2=0x7f6b4819b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 0x7f6b4819b340 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f6b38000c00 tx=0x7f6b3800b6c0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 shutdown_connections 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6b4806c090 0x7f6b4819b340 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 --2- 172.21.15.50:0/705594443 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6b30071f80 0x7f6b30074440 unknown :-1 s=CLOSED pgs=3754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 >> 172.21.15.50:0/705594443 conn(0x7f6b480790f0 msgr2=0x7f6b48101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 shutdown_connections 2024-04-26T16:29:20.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 -- 172.21.15.50:0/705594443 wait complete. 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:20.844+0000 7f6b4cee1700 1 librados: shutdown 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:29:20.849 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:29:20.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:20.962 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:20.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:21.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:21.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:20 smithi190 bash[23868]: audit 2024-04-26T16:29:20.660778+0000 mon.a (mon.0) 2777 : audit [DBG] from='client.? 172.21.15.50:0/705594443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:21.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:20 smithi190 bash[23868]: audit 2024-04-26T16:29:20.660778+0000 mon.a (mon.0) 2777 : audit [DBG] from='client.? 172.21.15.50:0/705594443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:21.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:20 smithi050 bash[29234]: audit 2024-04-26T16:29:20.660778+0000 mon.a (mon.0) 2777 : audit [DBG] from='client.? 172.21.15.50:0/705594443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:21.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:20 smithi050 bash[29234]: audit 2024-04-26T16:29:20.660778+0000 mon.a (mon.0) 2777 : audit [DBG] from='client.? 172.21.15.50:0/705594443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:21.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:20 smithi050 bash[21463]: audit 2024-04-26T16:29:20.660778+0000 mon.a (mon.0) 2777 : audit [DBG] from='client.? 172.21.15.50:0/705594443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:21.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:20 smithi050 bash[21463]: audit 2024-04-26T16:29:20.660778+0000 mon.a (mon.0) 2777 : audit [DBG] from='client.? 172.21.15.50:0/705594443' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:21.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:21.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:21.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:21.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:21.350 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:21.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:21.482 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:21.586 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:21.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:21.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:21.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:21.746 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:21.850 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:21.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:21.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:21.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:22.005 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:22.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.135 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:22.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:22 smithi190 bash[23868]: cluster 2024-04-26T16:29:21.176758+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-04-26T16:29:22.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:22 smithi190 bash[23868]: cluster 2024-04-26T16:29:21.176758+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-04-26T16:29:22.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:22 smithi190 bash[23868]: cluster 2024-04-26T16:29:21.676905+0000 mgr.y (mgr.24401) 468 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:29:22.250 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:22 smithi190 bash[23868]: cluster 2024-04-26T16:29:21.676905+0000 mgr.y (mgr.24401) 468 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:29:22.266 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:22.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:22.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[29234]: cluster 2024-04-26T16:29:21.176758+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-04-26T16:29:22.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[29234]: cluster 2024-04-26T16:29:21.176758+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-04-26T16:29:22.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[29234]: cluster 2024-04-26T16:29:21.676905+0000 mgr.y (mgr.24401) 468 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:29:22.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[29234]: cluster 2024-04-26T16:29:21.676905+0000 mgr.y (mgr.24401) 468 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:29:22.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[21463]: cluster 2024-04-26T16:29:21.176758+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-04-26T16:29:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[21463]: cluster 2024-04-26T16:29:21.176758+0000 mon.a (mon.0) 2778 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2024-04-26T16:29:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[21463]: cluster 2024-04-26T16:29:21.676905+0000 mgr.y (mgr.24401) 468 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:29:22.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:22 smithi050 bash[21463]: cluster 2024-04-26T16:29:21.676905+0000 mgr.y (mgr.24401) 468 : cluster [DBG] pgmap v702: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:29:22.398 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:22.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.529 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:22.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:22.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:22.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:22.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:22.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.915 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:22.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:23.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:23.046 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:23.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:23.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:23.192 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.546949+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:23.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:23.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:23.443 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:23.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.546949+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.548143+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.548143+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.548805+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.548805+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.444 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.550160+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.445 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:23 smithi190 bash[23868]: audit 2024-04-26T16:29:22.550160+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:23.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:23.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:23.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:23.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.546949+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.546949+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.548143+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.548143+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.548805+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.548805+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.550160+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[21463]: audit 2024-04-26T16:29:22.550160+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.546949+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.546949+0000 mon.a (mon.0) 2779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.548143+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.548143+0000 mon.b (mon.1) 323 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.548805+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.588 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.548805+0000 mon.a (mon.0) 2780 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.589 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.550160+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.589 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:23 smithi050 bash[29234]: audit 2024-04-26T16:29:22.550160+0000 mon.b (mon.1) 324 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-63"}]: dispatch 2024-04-26T16:29:23.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:23.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:23.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:23.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:23.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:23.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:23.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:24.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:24.089 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:24.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:24.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:24.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[21463]: cluster 2024-04-26T16:29:23.189579+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-04-26T16:29:24.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[21463]: cluster 2024-04-26T16:29:23.189579+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-04-26T16:29:24.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[21463]: cluster 2024-04-26T16:29:23.677810+0000 mgr.y (mgr.24401) 469 : cluster [DBG] pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:24.323 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[21463]: cluster 2024-04-26T16:29:23.677810+0000 mgr.y (mgr.24401) 469 : cluster [DBG] pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:24.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[29234]: cluster 2024-04-26T16:29:23.189579+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-04-26T16:29:24.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[29234]: cluster 2024-04-26T16:29:23.189579+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-04-26T16:29:24.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[29234]: cluster 2024-04-26T16:29:23.677810+0000 mgr.y (mgr.24401) 469 : cluster [DBG] pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:24.324 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[29234]: cluster 2024-04-26T16:29:23.677810+0000 mgr.y (mgr.24401) 469 : cluster [DBG] pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:24.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:24.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:24 smithi190 bash[23868]: cluster 2024-04-26T16:29:23.189579+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-04-26T16:29:24.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:24 smithi190 bash[23868]: cluster 2024-04-26T16:29:23.189579+0000 mon.a (mon.0) 2781 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2024-04-26T16:29:24.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:24 smithi190 bash[23868]: cluster 2024-04-26T16:29:23.677810+0000 mgr.y (mgr.24401) 469 : cluster [DBG] pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:24.338 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:24 smithi190 bash[23868]: cluster 2024-04-26T16:29:23.677810+0000 mgr.y (mgr.24401) 469 : cluster [DBG] pgmap v704: 196 pgs: 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:24.356 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:24.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:24.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:24.470 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:29:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:29:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:29:24.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:24.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:24.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:24.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:24.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:24.854 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:24.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:24.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:24.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:25.014 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:25.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:25.144 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:25.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:25.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:25 smithi190 bash[23868]: cluster 2024-04-26T16:29:24.217473+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-04-26T16:29:25.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:25 smithi190 bash[23868]: cluster 2024-04-26T16:29:24.217473+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-04-26T16:29:25.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:25 smithi190 bash[23868]: audit 2024-04-26T16:29:24.226878+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:25 smithi190 bash[23868]: audit 2024-04-26T16:29:24.226878+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:25 smithi190 bash[23868]: audit 2024-04-26T16:29:24.227500+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.257 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:25 smithi190 bash[23868]: audit 2024-04-26T16:29:24.227500+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.271 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:25.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:25.401 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:25.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:25.535 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:25.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[29234]: cluster 2024-04-26T16:29:24.217473+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-04-26T16:29:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[29234]: cluster 2024-04-26T16:29:24.217473+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-04-26T16:29:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[29234]: audit 2024-04-26T16:29:24.226878+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[29234]: audit 2024-04-26T16:29:24.226878+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[29234]: audit 2024-04-26T16:29:24.227500+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[29234]: audit 2024-04-26T16:29:24.227500+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[21463]: cluster 2024-04-26T16:29:24.217473+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[21463]: cluster 2024-04-26T16:29:24.217473+0000 mon.a (mon.0) 2782 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[21463]: audit 2024-04-26T16:29:24.226878+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[21463]: audit 2024-04-26T16:29:24.226878+0000 mon.b (mon.1) 325 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[21463]: audit 2024-04-26T16:29:24.227500+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:25 smithi050 bash[21463]: audit 2024-04-26T16:29:24.227500+0000 mon.a (mon.0) 2783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:25.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:25.664 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:25.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:25.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:25.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:25.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:25.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:26.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:26.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:26.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:26.183 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:26.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:26.318 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:26.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.212902+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.212902+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: cluster 2024-04-26T16:29:25.221427+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: cluster 2024-04-26T16:29:25.221427+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.225899+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.225899+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.241955+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.241955+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.259642+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: audit 2024-04-26T16:29:25.259642+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: cluster 2024-04-26T16:29:25.678594+0000 mgr.y (mgr.24401) 470 : cluster [DBG] pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[29234]: cluster 2024-04-26T16:29:25.678594+0000 mgr.y (mgr.24401) 470 : cluster [DBG] pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.212902+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.212902+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: cluster 2024-04-26T16:29:25.221427+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: cluster 2024-04-26T16:29:25.221427+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.225899+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.225899+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:26.342 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.241955+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.343 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.241955+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.343 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.259642+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.343 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: audit 2024-04-26T16:29:25.259642+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.343 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: cluster 2024-04-26T16:29:25.678594+0000 mgr.y (mgr.24401) 470 : cluster [DBG] pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:26.343 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:26 smithi050 bash[21463]: cluster 2024-04-26T16:29:25.678594+0000 mgr.y (mgr.24401) 470 : cluster [DBG] pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:26.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.212902+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.212902+0000 mon.a (mon.0) 2784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: cluster 2024-04-26T16:29:25.221427+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: cluster 2024-04-26T16:29:25.221427+0000 mon.a (mon.0) 2785 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.225899+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.225899+0000 mon.b (mon.1) 326 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.241955+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.241955+0000 mon.b (mon.1) 327 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.259642+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: audit 2024-04-26T16:29:25.259642+0000 mon.a (mon.0) 2786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: cluster 2024-04-26T16:29:25.678594+0000 mgr.y (mgr.24401) 470 : cluster [DBG] pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:26.433 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[23868]: cluster 2024-04-26T16:29:25.678594+0000 mgr.y (mgr.24401) 470 : cluster [DBG] pgmap v707: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:26.447 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:26.550 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:26.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:26.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:26.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:26.699 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:26.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:26.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:26.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:26.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:29:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:29:26.959 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:26.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:27.062 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:27.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:27.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:27.222 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:27.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:27.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:27.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:27.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[21463]: audit 2024-04-26T16:29:26.250359+0000 mon.a (mon.0) 2787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:27.480 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:27 smithi190 bash[23868]: audit 2024-04-26T16:29:26.250359+0000 mon.a (mon.0) 2787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:27 smithi190 bash[23868]: audit 2024-04-26T16:29:26.250359+0000 mon.a (mon.0) 2787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:27 smithi190 bash[23868]: cluster 2024-04-26T16:29:26.285752+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-04-26T16:29:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:27 smithi190 bash[23868]: cluster 2024-04-26T16:29:26.285752+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-04-26T16:29:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:27 smithi190 bash[23868]: audit 2024-04-26T16:29:26.919141+0000 mon.c (mon.2) 633 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:27.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:27 smithi190 bash[23868]: audit 2024-04-26T16:29:26.919141+0000 mon.c (mon.2) 633 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:27.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:27.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:27.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:27.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:27.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:27.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[29234]: audit 2024-04-26T16:29:26.250359+0000 mon.a (mon.0) 2787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[29234]: audit 2024-04-26T16:29:26.250359+0000 mon.a (mon.0) 2787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[29234]: cluster 2024-04-26T16:29:26.285752+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[29234]: cluster 2024-04-26T16:29:26.285752+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[29234]: audit 2024-04-26T16:29:26.919141+0000 mon.c (mon.2) 633 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[29234]: audit 2024-04-26T16:29:26.919141+0000 mon.c (mon.2) 633 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[21463]: audit 2024-04-26T16:29:26.250359+0000 mon.a (mon.0) 2787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[21463]: cluster 2024-04-26T16:29:26.285752+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[21463]: cluster 2024-04-26T16:29:26.285752+0000 mon.a (mon.0) 2788 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2024-04-26T16:29:27.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[21463]: audit 2024-04-26T16:29:26.919141+0000 mon.c (mon.2) 633 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:27.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:27 smithi050 bash[21463]: audit 2024-04-26T16:29:26.919141+0000 mon.c (mon.2) 633 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:27.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:27.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:27.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:28.002 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:28.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:28.132 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:28.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:28.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:28.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:28.394 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:28.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:28.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:28 smithi190 bash[23868]: audit 2024-04-26T16:29:26.912658+0000 mgr.y (mgr.24401) 471 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:28 smithi190 bash[23868]: audit 2024-04-26T16:29:26.912658+0000 mgr.y (mgr.24401) 471 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:28 smithi190 bash[23868]: cluster 2024-04-26T16:29:27.259847+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-04-26T16:29:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:28 smithi190 bash[23868]: cluster 2024-04-26T16:29:27.259847+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-04-26T16:29:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:28 smithi190 bash[23868]: cluster 2024-04-26T16:29:27.679400+0000 mgr.y (mgr.24401) 472 : cluster [DBG] pgmap v710: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:29:28.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:28 smithi190 bash[23868]: cluster 2024-04-26T16:29:27.679400+0000 mgr.y (mgr.24401) 472 : cluster [DBG] pgmap v710: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:29:28.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:28.655 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[29234]: audit 2024-04-26T16:29:26.912658+0000 mgr.y (mgr.24401) 471 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[29234]: audit 2024-04-26T16:29:26.912658+0000 mgr.y (mgr.24401) 471 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[29234]: cluster 2024-04-26T16:29:27.259847+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[29234]: cluster 2024-04-26T16:29:27.259847+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[29234]: cluster 2024-04-26T16:29:27.679400+0000 mgr.y (mgr.24401) 472 : cluster [DBG] pgmap v710: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[29234]: cluster 2024-04-26T16:29:27.679400+0000 mgr.y (mgr.24401) 472 : cluster [DBG] pgmap v710: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[21463]: audit 2024-04-26T16:29:26.912658+0000 mgr.y (mgr.24401) 471 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[21463]: audit 2024-04-26T16:29:26.912658+0000 mgr.y (mgr.24401) 471 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:28.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[21463]: cluster 2024-04-26T16:29:27.259847+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-04-26T16:29:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[21463]: cluster 2024-04-26T16:29:27.259847+0000 mon.a (mon.0) 2789 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2024-04-26T16:29:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[21463]: cluster 2024-04-26T16:29:27.679400+0000 mgr.y (mgr.24401) 472 : cluster [DBG] pgmap v710: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:29:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:28 smithi050 bash[21463]: cluster 2024-04-26T16:29:27.679400+0000 mgr.y (mgr.24401) 472 : cluster [DBG] pgmap v710: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:29:28.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:28.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:28.791 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:28.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:28.921 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:28.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:28.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:29.024 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:29.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:29.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:29.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:29.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[21463]: cluster 2024-04-26T16:29:28.110439+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:29.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[21463]: cluster 2024-04-26T16:29:28.110439+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:29.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[21463]: cluster 2024-04-26T16:29:28.110629+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[21463]: cluster 2024-04-26T16:29:28.110629+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[21463]: cluster 2024-04-26T16:29:28.270899+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[21463]: cluster 2024-04-26T16:29:28.270899+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[29234]: cluster 2024-04-26T16:29:28.110439+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[29234]: cluster 2024-04-26T16:29:28.110439+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[29234]: cluster 2024-04-26T16:29:28.110629+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[29234]: cluster 2024-04-26T16:29:28.110629+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[29234]: cluster 2024-04-26T16:29:28.270899+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-04-26T16:29:29.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:29 smithi050 bash[29234]: cluster 2024-04-26T16:29:28.270899+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-04-26T16:29:29.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:29.443 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:29.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:29.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:29 smithi190 bash[23868]: cluster 2024-04-26T16:29:28.110439+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:29 smithi190 bash[23868]: cluster 2024-04-26T16:29:28.110439+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:29 smithi190 bash[23868]: cluster 2024-04-26T16:29:28.110629+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:29 smithi190 bash[23868]: cluster 2024-04-26T16:29:28.110629+0000 osd.6 (osd.6) 26 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:29 smithi190 bash[23868]: cluster 2024-04-26T16:29:28.270899+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-04-26T16:29:29.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:29 smithi190 bash[23868]: cluster 2024-04-26T16:29:28.270899+0000 mon.a (mon.0) 2790 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2024-04-26T16:29:29.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:29.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:29.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:29.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:29.834 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:29.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:29.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:29.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:30.070 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:30.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:30.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:30.231 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:30.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:30.362 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:30.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:30.476 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:30 smithi190 bash[23868]: cluster 2024-04-26T16:29:29.279916+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-04-26T16:29:30.476 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:30 smithi190 bash[23868]: cluster 2024-04-26T16:29:29.279916+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-04-26T16:29:30.476 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:30 smithi190 bash[23868]: cluster 2024-04-26T16:29:29.680391+0000 mgr.y (mgr.24401) 473 : cluster [DBG] pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:29:30.476 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:30 smithi190 bash[23868]: cluster 2024-04-26T16:29:29.680391+0000 mgr.y (mgr.24401) 473 : cluster [DBG] pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.488+0000 7f487cfb2700 1 -- 172.21.15.50:0/2515769184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48781094c0 msgr2=0x7f48781098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.488+0000 7f487cfb2700 1 --2- 172.21.15.50:0/2515769184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48781094c0 0x7f48781098a0 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7f48680098f0 tx=0x7f4868009c00 comp rx=0 tx=0).stop 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.488+0000 7f487cfb2700 1 -- 172.21.15.50:0/2515769184 shutdown_connections 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.488+0000 7f487cfb2700 1 --2- 172.21.15.50:0/2515769184 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103e30 0x7f4878110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 --2- 172.21.15.50:0/2515769184 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48781094c0 0x7f48781098a0 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 --2- 172.21.15.50:0/2515769184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4878103470 0x7f48781038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- 172.21.15.50:0/2515769184 >> 172.21.15.50:0/2515769184 conn(0x7f487807a580 msgr2=0x7f487807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- 172.21.15.50:0/2515769184 shutdown_connections 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- 172.21.15.50:0/2515769184 wait complete. 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 librados: starting msgr at 2024-04-26T16:29:30.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 librados: starting objecter 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 Processor -- start 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- start start 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 librados: setting wanted keys 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 librados: calling monclient init 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103470 0x7f4878075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 0x7f4878075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48781094c0 0x7f48780761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4878112ab0 con 0x7f4878103e30 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4878112930 con 0x7f48781094c0 2024-04-26T16:29:30.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4878112c30 con 0x7f4878103470 2024-04-26T16:29:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.493 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:30.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:30.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f4875d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 0x7f4878075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103470 0x7f4878075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f4875d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 0x7f4878075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45522/0 (socket says 172.21.15.50:45522) 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f4875d9b700 1 -- 172.21.15.50:0/3388385310 learned_addr learned my addr 172.21.15.50:0/3388385310 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f4876d9d700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48781094c0 0x7f48780761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 -- 172.21.15.50:0/3388385310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48781094c0 msgr2=0x7f48780761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48781094c0 0x7f48780761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 -- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 msgr2=0x7f4878075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 0x7f4878075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 -- 172.21.15.50:0/3388385310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48781a76c0 con 0x7f4878103470 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f4875d9b700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 0x7f4878075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487659c700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103470 0x7f4878075720 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f4868000c00 tx=0x7f486800b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:30.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f48677fe700 10 client.?.objecter ms_handle_connect 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f48677fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486800ba10 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48781a7950 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f486800bbb0 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868003d60 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 10 client.41261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48781a7e30 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 librados: init done 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487806a110 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.492+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4868005040 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4860071f80 0x7f4860074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(478..478 src has 87..478) ==== 9433+0+0 (secure 0 0 0) 0x7f48680924b0 con 0x7f4878103470 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 10 client.41261.objecter ms_dispatch 0x7f4878001550 osd_map(478..478 src has 87..478) 2024-04-26T16:29:30.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 3 client.41261.objecter handle_osd_map got epochs [478,478] > 0 2024-04-26T16:29:30.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 3 client.41261.objecter handle_osd_map decoding full epoch 478 2024-04-26T16:29:30.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f4875d9b700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4860071f80 0x7f4860074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:30.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 20 client.41261.objecter dump_active .. 0 homeless 2024-04-26T16:29:30.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f48677fe700 10 client.41261.objecter ms_handle_connect 0x7f4860071f80 2024-04-26T16:29:30.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.496+0000 7f4875d9b700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4860071f80 0x7f4860074440 secure :-1 s=READY pgs=3755 cs=0 l=1 rev1=1 crypto rx=0x7f4878108f80 tx=0x7f486c00b500 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:30.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.500+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4868013030 con 0x7f4878103470 2024-04-26T16:29:30.594 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:30.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[29234]: cluster 2024-04-26T16:29:29.279916+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-04-26T16:29:30.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[29234]: cluster 2024-04-26T16:29:29.279916+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-04-26T16:29:30.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[29234]: cluster 2024-04-26T16:29:29.680391+0000 mgr.y (mgr.24401) 473 : cluster [DBG] pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:29:30.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[29234]: cluster 2024-04-26T16:29:29.680391+0000 mgr.y (mgr.24401) 473 : cluster [DBG] pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:29:30.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[21463]: cluster 2024-04-26T16:29:29.279916+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-04-26T16:29:30.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[21463]: cluster 2024-04-26T16:29:29.279916+0000 mon.a (mon.0) 2791 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2024-04-26T16:29:30.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[21463]: cluster 2024-04-26T16:29:29.680391+0000 mgr.y (mgr.24401) 473 : cluster [DBG] pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:29:30.605 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:30 smithi050 bash[21463]: cluster 2024-04-26T16:29:29.680391+0000 mgr.y (mgr.24401) 473 : cluster [DBG] pgmap v713: 228 pgs: 228 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:29:30.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:30.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:30.736+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f48781a8000 con 0x7f4878103470 2024-04-26T16:29:30.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:30.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:30.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:30.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:30.995 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:30.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:31.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:31.126 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:31.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:31.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:31.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:31.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:31.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.288+0000 7f48677fe700 1 -- 172.21.15.50:0/3388385310 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v479) ==== 113+0+0 (secure 0 0 0) 0x7f486805de30 con 0x7f4878103470 2024-04-26T16:29:31.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 10 librados: watch_flush enter 2024-04-26T16:29:31.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 10 librados: watch_flush exit 2024-04-26T16:29:31.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 20 client.41261.objecter shutdown clearing up homeless session... 2024-04-26T16:29:31.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 10 client.41261.objecter successfully canceled tick 2024-04-26T16:29:31.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4860071f80 msgr2=0x7f4860074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:31.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4860071f80 0x7f4860074440 secure :-1 s=READY pgs=3755 cs=0 l=1 rev1=1 crypto rx=0x7f4878108f80 tx=0x7f486c00b500 comp rx=0 tx=0).stop 2024-04-26T16:29:31.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103470 msgr2=0x7f4878075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:31.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103470 0x7f4878075720 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f4868000c00 tx=0x7f486800b560 comp rx=0 tx=0).stop 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 shutdown_connections 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4860071f80 0x7f4860074440 unknown :-1 s=CLOSED pgs=3755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48781094c0 0x7f48780761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4878103470 0x7f4878075720 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 --2- 172.21.15.50:0/3388385310 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4878103e30 0x7f4878075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 >> 172.21.15.50:0/3388385310 conn(0x7f487807a580 msgr2=0x7f4878101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 shutdown_connections 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 -- 172.21.15.50:0/3388385310 wait complete. 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:31.292+0000 7f487cfb2700 1 librados: shutdown 2024-04-26T16:29:31.291 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T16:29:31.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:31.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:31.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:31.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:31.511 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:31.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:31.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:31 smithi190 bash[23868]: audit 2024-04-26T16:29:30.742130+0000 mon.c (mon.2) 634 : audit [INF] from='client.? 172.21.15.50:0/3388385310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:31 smithi190 bash[23868]: audit 2024-04-26T16:29:30.742130+0000 mon.c (mon.2) 634 : audit [INF] from='client.? 172.21.15.50:0/3388385310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:31 smithi190 bash[23868]: audit 2024-04-26T16:29:30.742549+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:31 smithi190 bash[23868]: audit 2024-04-26T16:29:30.742549+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[21463]: audit 2024-04-26T16:29:30.742130+0000 mon.c (mon.2) 634 : audit [INF] from='client.? 172.21.15.50:0/3388385310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[21463]: audit 2024-04-26T16:29:30.742130+0000 mon.c (mon.2) 634 : audit [INF] from='client.? 172.21.15.50:0/3388385310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[21463]: audit 2024-04-26T16:29:30.742549+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[21463]: audit 2024-04-26T16:29:30.742549+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[29234]: audit 2024-04-26T16:29:30.742130+0000 mon.c (mon.2) 634 : audit [INF] from='client.? 172.21.15.50:0/3388385310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[29234]: audit 2024-04-26T16:29:30.742130+0000 mon.c (mon.2) 634 : audit [INF] from='client.? 172.21.15.50:0/3388385310' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[29234]: audit 2024-04-26T16:29:30.742549+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:31 smithi050 bash[29234]: audit 2024-04-26T16:29:30.742549+0000 mon.a (mon.0) 2792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:29:31.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.36052.0:2 155.0 155:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:23:59.126267+0000", 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 332.49034418899998, 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0509325519999999, 2024-04-26T16:29:31.616 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.36052", 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1277001254", 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.126267+0000", 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.617 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.126267+0000", 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.126269+0000", 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6920000000000001e-06 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.618 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.126274+0000", 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.3560000000000003e-06 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.126285+0000", 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1683e-05 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.619 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.126296+0000", 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0925e-05 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.130184+0000", 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038880450000000001 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.130193+0000", 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5639999999999997e-06 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.620 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.154882+0000", 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.024688896 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.154995+0000", 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113504 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.155154+0000", 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015921300000000001 2024-04-26T16:29:31.621 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.162677+0000", 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00752288 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.166958+0000", 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042810340000000004 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.622 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.177094+0000", 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010135264999999999 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.177149+0000", 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5821e-05 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.177200+0000", 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.0674000000000003e-05 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.623 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.23268.0:2 151.3 151:c656db43:::oid_RoundTripPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e171)", 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:23:59.133687+0000", 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 332.482924293, 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.029438957000000002, 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.23268", 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2196624441", 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:29:31.624 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.133687+0000", 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.133687+0000", 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.625 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.133689+0000", 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.485e-06 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.133695+0000", 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5050000000000004e-06 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.133707+0000", 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2104e-05 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.626 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.133718+0000", 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0563e-05 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.134469+0000", 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075085199999999999 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.134583+0000", 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114057 2024-04-26T16:29:31.627 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,5", 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.134756+0000", 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000173451 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.139588+0000", 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0048319299999999999 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.152539+0000", 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012950311000000001 2024-04-26T16:29:31.628 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.163011+0000", 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010471852 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.163072+0000", 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.1835999999999995e-05 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:23:59.163126+0000", 2024-04-26T16:29:31.629 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4011000000000002e-05 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15429.0:51 112.4 112:24e065aa:test-rados-api-smithi050-73432-11::foo:head [write 0~1048576] snapc 9=[9] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:00.136336+0000", 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 331.48027581700001, 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032635088, 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.630 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15429", 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3301772666", 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 51 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.136336+0000", 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.631 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.136336+0000", 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.136338+0000", 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0219999999999999e-06 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.136994+0000", 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065572999999999998 2024-04-26T16:29:31.632 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.137011+0000", 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7234000000000001e-05 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.137024+0000", 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3190000000000001e-05 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.633 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.144051+0000", 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0070264899999999998 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.144180+0000", 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012975500000000001 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.144333+0000", 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015299899999999999 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.634 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.168800+0000", 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.02446717 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.168856+0000", 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6039999999999999e-05 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.168915+0000", 2024-04-26T16:29:31.635 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8718000000000003e-05 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.168944+0000", 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8530999999999998e-05 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:00.168971+0000", 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7209e-05 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.636 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.15429.0:53 112.4 112:24e065aa:test-rados-api-smithi050-73432-11::foo:head [stat,delete] snapc a=[a,9] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:01.104800+0000", 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 330.51181197300002, 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.048187143000000002, 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.15429", 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3301772666", 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 53 2024-04-26T16:29:31.637 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.104800+0000", 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.104800+0000", 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.638 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.639 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.104801+0000", 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.406e-06 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.104805+0000", 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0450000000000002e-06 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.104815+0000", 2024-04-26T16:29:31.640 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6059999999999992e-06 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.104825+0000", 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0341e-05 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.106517+0000", 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001692074 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.106618+0000", 2024-04-26T16:29:31.641 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000100531 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,5", 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.106825+0000", 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020745900000000001 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.121717+0000", 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014891503 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.642 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.137473+0000", 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015756497000000001 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.152874+0000", 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015400544 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.152934+0000", 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0272000000000002e-05 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.643 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:01.152987+0000", 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2865000000000003e-05 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:869 158.0 e175/172 158:0384ab8e:::oid:3 v 175'9, mlcod=174'6)", 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:03.088356+0000", 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 328.52825539600002, 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027219285999999999, 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.644 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.645 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.088356+0000", 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.088356+0000", 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.088357+0000", 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.91e-07 2024-04-26T16:29:31.646 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.088362+0000", 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.3680000000000001e-06 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.088368+0000", 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0360000000000003e-06 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.647 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.088377+0000", 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.115e-06 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.095577+0000", 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0071997040000000003 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.095598+0000", 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.092e-05 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.648 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.115530+0000", 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019932406999999999 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:03.115576+0000", 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5445000000000003e-05 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24770.0:2078 17.1c 17:3be0d449:test-rados-api-smithi050-73363-17::big.copy:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.294765+0000", 2024-04-26T16:29:31.649 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.32184633499998, 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.114120738, 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24770", 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2506934054", 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2078 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.650 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294765+0000", 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294765+0000", 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294768+0000", 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8090000000000001e-06 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.651 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294780+0000", 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1843e-05 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294805+0000", 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5366000000000001e-05 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294820+0000", 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4691999999999999e-05 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.652 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.294905+0000", 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.5224999999999998e-05 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.295015+0000", 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000109447 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.653 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,6", 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.393609+0000", 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.098594577000000003 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.401295+0000", 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0076858109999999999 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.404064+0000", 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002768922 2024-04-26T16:29:31.654 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.408794+0000", 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0047304519999999996 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.408849+0000", 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.4775999999999999e-05 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.655 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.408886+0000", 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6817999999999999e-05 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:870 17.1b e176/141 -19:ddd8974c:::temp_17.1b_0_24223_2:head v 0'0, mlcod=0'0)", 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.428607+0000", 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.18800441299999, 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026023820999999999, 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.656 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.428607+0000", 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.428607+0000", 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.657 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.428609+0000", 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1330000000000002e-06 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.442937+0000", 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014327645 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.658 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.442975+0000", 2024-04-26T16:29:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7704999999999999e-05 2024-04-26T16:29:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.659 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.442992+0000", 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7737e-05 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.443080+0000", 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7996000000000002e-05 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.443283+0000", 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020227999999999999 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.660 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.454567+0000", 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011284084999999999 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.454631+0000", 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4239999999999995e-05 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:871 17.1b e176/141 -19:ddd8974c:::temp_17.1b_0_24223_2:head v 0'0, mlcod=0'0)", 2024-04-26T16:29:31.661 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.454563+0000", 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.16204818699998, 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027648061000000002, 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.662 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.454563+0000", 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.454563+0000", 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.454565+0000", 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.35e-06 2024-04-26T16:29:31.663 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.469969+0000", 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.015403929 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.470007+0000", 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8482000000000002e-05 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.664 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.470023+0000", 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5458999999999999e-05 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.470358+0000", 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033537600000000002 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.470387+0000", 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9198000000000001e-05 2024-04-26T16:29:31.665 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.482152+0000", 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011765094 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.482212+0000", 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9172999999999997e-05 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.666 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.669 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:872 17.1b e176/141 -19:ddd8974c:::temp_17.1b_0_24223_2:head v 0'0, mlcod=0'0)", 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:04.477569+0000", 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 327.13904271500002, 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055138394, 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.670 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.477569+0000", 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.477569+0000", 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.477571+0000", 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6300000000000001e-06 2024-04-26T16:29:31.671 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.513882+0000", 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.036311598 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.513915+0000", 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3291999999999999e-05 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.513930+0000", 2024-04-26T16:29:31.672 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4139e-05 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.515153+0000", 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001223642 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.515180+0000", 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7121e-05 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.673 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.532638+0000", 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.017457332999999998 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:04.532707+0000", 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9639000000000002e-05 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.15429.0:59 112.3 e177/156 112:c14002fe:test-rados-api-smithi050-73432-12::foo:head v 177'9, mlcod=161'6)", 2024-04-26T16:29:31.674 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:05.100950+0000", 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 326.51566152800001, 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.027458153999999999, 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.100950+0000", 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.675 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.100950+0000", 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.100950+0000", 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.96e-07 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.100963+0000", 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2964e-05 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.676 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.100971+0000", 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4889999999999998e-06 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.100981+0000", 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0468e-05 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.117111+0000", 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.016129753 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.677 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.117133+0000", 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1688e-05 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.128366+0000", 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.011233538 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:05.128408+0000", 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.1958e-05 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.678 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41863.0:2 170.15 170:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e180)", 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:08.080483+0000", 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 323.53612913799998, 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.034161784000000001, 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.679 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41863", 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3241106947", 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080483+0000", 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.680 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080483+0000", 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080484+0000", 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8169999999999999e-06 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080492+0000", 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3139999999999998e-06 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.681 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080505+0000", 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2835e-05 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080516+0000", 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1413000000000001e-05 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.682 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080587+0000", 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0908999999999995e-05 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.080598+0000", 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0695e-05 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.114528+0000", 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.033930894000000003 2024-04-26T16:29:31.683 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:08.114644+0000", 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115907 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.43060.0:2 181.6 181:602f83fe:::foo:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:17.052745+0000", 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 314.56386666899999, 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.14910495000000001, 2024-04-26T16:29:31.684 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.43060", 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3767464243", 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.052745+0000", 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.685 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.052745+0000", 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.052747+0000", 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.092e-06 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.686 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.052755+0000", 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7100000000000007e-06 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.052769+0000", 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3876999999999999e-05 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.052781+0000", 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2604e-05 2024-04-26T16:29:31.687 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.053115+0000", 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033348500000000001 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.053124+0000", 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.577e-06 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.688 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.193072+0000", 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.13994794899999999 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.193231+0000", 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015893 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 3,7", 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.193394+0000", 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000163012 2024-04-26T16:29:31.689 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.194421+0000", 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001027214 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.200307+0000", 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058858540000000003 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.690 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.201715+0000", 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001407312 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.201794+0000", 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9721999999999993e-05 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:17.201850+0000", 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5612000000000002e-05 2024-04-26T16:29:31.691 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:1013 109.2 e221/155 109:449e828c:test-rados-api-smithi050-73372-14::foo59:6 v 221'47, mlcod=187'43)", 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:49.071550+0000", 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 282.54506121700001, 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028237953999999999, 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.692 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.071550+0000", 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.071550+0000", 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.071552+0000", 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0419999999999999e-06 2024-04-26T16:29:31.693 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.071561+0000", 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.842e-06 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.073981+0000", 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.002419268 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.073992+0000", 2024-04-26T16:29:31.694 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0947999999999999e-05 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074793+0000", 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00080191699999999997 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074815+0000", 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1597999999999999e-05 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.695 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099693+0000", 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.024877999000000001 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099788+0000", 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5340000000000005e-05 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:1014 109.2 e221/155 109:44b51004:test-rados-api-smithi050-73372-14::foo44:6 v 221'49, mlcod=187'43)", 2024-04-26T16:29:31.696 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:49.074012+0000", 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 282.54260003299999, 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025830260000000001, 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074012+0000", 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.697 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074012+0000", 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074012+0000", 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4799999999999997e-07 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074029+0000", 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7090000000000001e-05 2024-04-26T16:29:31.698 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074040+0000", 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0912e-05 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074048+0000", 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.2509999999999997e-06 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.699 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.078127+0000", 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0040798070000000004 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.078152+0000", 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4202999999999998e-05 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099795+0000", 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.021643470000000001 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.700 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099842+0000", 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6779000000000001e-05 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:1015 109.5 e221/155 109:a5dbb5ef:test-rados-api-smithi050-73372-14::foo28:6 v 221'20, mlcod=187'16)", 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:49.074061+0000", 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 282.54255027099998, 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025729426, 2024-04-26T16:29:31.701 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074061+0000", 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074061+0000", 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.702 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074062+0000", 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.0699999999999998e-07 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074068+0000", 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8209999999999998e-06 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074076+0000", 2024-04-26T16:29:31.703 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8529999999999998e-06 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074084+0000", 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.4209999999999998e-06 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074139+0000", 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5213999999999997e-05 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.704 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074167+0000", 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.7820000000000001e-05 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099701+0000", 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025533980000000001 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099791+0000", 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.9909999999999998e-05 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.705 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(osd.4.0:1016 109.5 e221/155 109:a71b5931:test-rados-api-smithi050-73372-14::foo90:6 v 221'22, mlcod=187'16)", 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:49.074213+0000", 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 282.542398757, 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.025620862000000001, 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074213+0000", 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.706 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074213+0000", 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074214+0000", 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9299999999999998e-07 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.707 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074229+0000", 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5505000000000001e-05 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074243+0000", 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3484e-05 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.074251+0000", 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2709999999999992e-06 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.708 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.080015+0000", 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0057646040000000004 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.080038+0000", 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2629e-05 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099799+0000", 2024-04-26T16:29:31.709 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.019760753999999998 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:49.099834+0000", 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5021999999999998e-05 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.30072.0:2 209.d 209:b207418b:::oid_StatRemovePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:24:50.082922+0000", 2024-04-26T16:29:31.710 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 281.53368922700002, 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.044825749999999998, 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.30072", 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3780134797", 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.082922+0000", 2024-04-26T16:29:31.711 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.082922+0000", 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.082925+0000", 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3819999999999998e-06 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.712 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.082934+0000", 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.8629999999999997e-06 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.082948+0000", 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3988000000000001e-05 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.082964+0000", 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5977999999999998e-05 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.713 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.084231+0000", 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0012668619999999999 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.084242+0000", 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1039000000000001e-05 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.112913+0000", 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.028671074000000001 2024-04-26T16:29:31.714 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.113063+0000", 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00015002900000000001 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,2", 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.113225+0000", 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016185200000000001 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.117856+0000", 2024-04-26T16:29:31.715 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0046313379999999996 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.125991+0000", 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0081348289999999997 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.127635+0000", 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0016439040000000001 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.716 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.127695+0000", 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0835000000000003e-05 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:24:50.127748+0000", 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2777e-05 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:2168 247.f 247:f214ce95:test-rados-api-smithi050-73372-28::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e286)", 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:25:54.102745+0000", 2024-04-26T16:29:31.717 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 217.51386618800001, 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058487704000000001, 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2168 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.718 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102745+0000", 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102745+0000", 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102748+0000", 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0700000000000001e-06 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.719 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102756+0000", 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3920000000000003e-06 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102776+0000", 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9873000000000001e-05 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.720 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102789+0000", 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3606e-05 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102876+0000", 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6156999999999995e-05 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for blocked object", 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.103278+0000", 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040282099999999998 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.721 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.158416+0000", 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055137799000000001 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.158546+0000", 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129629 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.161019+0000", 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024728549999999999 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.722 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.161098+0000", 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9550999999999998e-05 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.161233+0000", 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013495099999999999 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.723 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.30300.0:1 251.6s0 251:68f45781:::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:25:58.129330+0000", 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 213.48728140599999, 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0223501859999999, 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.30300", 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3027781296", 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.724 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129330+0000", 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129330+0000", 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129332+0000", 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9360000000000002e-06 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.725 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129340+0000", 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.875e-06 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129355+0000", 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4517e-05 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129367+0000", 2024-04-26T16:29:31.726 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2619000000000001e-05 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129482+0000", 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011439500000000001 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129488+0000", 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0979999999999999e-06 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.727 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.139663+0000", 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0101752289999999 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.139805+0000", 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014170799999999999 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.142262+0000", 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024573720000000002 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:29:31.728 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.143172+0000", 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090992199999999997 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.151585+0000", 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084128109999999992 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.151680+0000", 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5704000000000004e-05 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.729 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.30450.0:1 276.5s0 276:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e356)", 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:27:04.140937+0000", 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 147.475674591, 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0109418809999999, 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.30450", 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/4229062425", 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:29:31.730 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140937+0000", 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140937+0000", 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.731 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140939+0000", 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8759999999999999e-06 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140946+0000", 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8950000000000001e-06 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140958+0000", 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2193e-05 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.732 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140968+0000", 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9820000000000003e-06 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.141021+0000", 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2923999999999997e-05 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.141025+0000", 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6210000000000001e-06 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.733 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:05.151766+0000", 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0107411369999999 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:05.151879+0000", 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011325299999999999 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:29:31.734 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:29:31.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:31.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:31.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:31.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:31.892 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:31.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:31.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:32.022 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:32.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:32.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:32.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:32.286 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:32.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:32.415 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:32.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[21463]: audit 2024-04-26T16:29:31.288857+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:29:32.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[21463]: audit 2024-04-26T16:29:31.288857+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:29:32.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[21463]: cluster 2024-04-26T16:29:31.292487+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-04-26T16:29:32.490 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[21463]: cluster 2024-04-26T16:29:31.292487+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[21463]: cluster 2024-04-26T16:29:31.682574+0000 mgr.y (mgr.24401) 474 : cluster [DBG] pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[21463]: cluster 2024-04-26T16:29:31.682574+0000 mgr.y (mgr.24401) 474 : cluster [DBG] pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[29234]: audit 2024-04-26T16:29:31.288857+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[29234]: audit 2024-04-26T16:29:31.288857+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[29234]: cluster 2024-04-26T16:29:31.292487+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[29234]: cluster 2024-04-26T16:29:31.292487+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[29234]: cluster 2024-04-26T16:29:31.682574+0000 mgr.y (mgr.24401) 474 : cluster [DBG] pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T16:29:32.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:32 smithi050 bash[29234]: cluster 2024-04-26T16:29:31.682574+0000 mgr.y (mgr.24401) 474 : cluster [DBG] pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T16:29:32.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:32.544 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[23868]: audit 2024-04-26T16:29:31.288857+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:29:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[23868]: audit 2024-04-26T16:29:31.288857+0000 mon.a (mon.0) 2793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:29:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[23868]: cluster 2024-04-26T16:29:31.292487+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-04-26T16:29:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[23868]: cluster 2024-04-26T16:29:31.292487+0000 mon.a (mon.0) 2794 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2024-04-26T16:29:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[23868]: cluster 2024-04-26T16:29:31.682574+0000 mgr.y (mgr.24401) 474 : cluster [DBG] pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T16:29:32.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[23868]: cluster 2024-04-26T16:29:31.682574+0000 mgr.y (mgr.24401) 474 : cluster [DBG] pgmap v715: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.7 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-04-26T16:29:32.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:32.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:32.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:32.807 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:32.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:32.941 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:32.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:33.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:33.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:33.071 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:29:32 smithi190 bash[50598]: logger=cleanup t=2024-04-26T16:29:32.812865937Z level=info msg="Completed cleanup jobs" duration=180.695765ms 2024-04-26T16:29:33.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:33.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:33.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:33.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:33.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:33.459 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:33.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:33.591 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:33.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:33.725 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:33.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:33.856 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:33.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:33 smithi050 bash[29234]: cluster 2024-04-26T16:29:33.683413+0000 mgr.y (mgr.24401) 475 : cluster [DBG] pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-26T16:29:33.935 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:33 smithi050 bash[29234]: cluster 2024-04-26T16:29:33.683413+0000 mgr.y (mgr.24401) 475 : cluster [DBG] pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-26T16:29:33.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:33 smithi050 bash[21463]: cluster 2024-04-26T16:29:33.683413+0000 mgr.y (mgr.24401) 475 : cluster [DBG] pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-26T16:29:33.936 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:33 smithi050 bash[21463]: cluster 2024-04-26T16:29:33.683413+0000 mgr.y (mgr.24401) 475 : cluster [DBG] pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-26T16:29:33.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:33.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:33.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:33 smithi190 bash[23868]: cluster 2024-04-26T16:29:33.683413+0000 mgr.y (mgr.24401) 475 : cluster [DBG] pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-26T16:29:34.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:33 smithi190 bash[23868]: cluster 2024-04-26T16:29:33.683413+0000 mgr.y (mgr.24401) 475 : cluster [DBG] pgmap v716: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-04-26T16:29:34.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:34.115 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:34.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:34.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:34.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:34.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:34.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:34.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:34.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-26T16:29:34.507 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:34.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:34.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:34.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:34.770 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:34.797 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:29:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:29:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:29:34.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-26T16:29:34.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:34.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:34.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:34.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:35.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/2701699714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc102ce0 msgr2=0x7f68bc10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- 172.21.15.50:0/2701699714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc102ce0 0x7f68bc10f1d0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009f90 tx=0x7f68b400b330 comp rx=0 tx=0).stop 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/2701699714 shutdown_connections 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- 172.21.15.50:0/2701699714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc108390 0x7f68bc108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- 172.21.15.50:0/2701699714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc102ce0 0x7f68bc10f1d0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- 172.21.15.50:0/2701699714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102320 0x7f68bc1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/2701699714 >> 172.21.15.50:0/2701699714 conn(0x7f68bc0792d0 msgr2=0x7f68bc0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/2701699714 shutdown_connections 2024-04-26T16:29:35.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/2701699714 wait complete. 2024-04-26T16:29:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 librados: starting msgr at 2024-04-26T16:29:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 librados: starting objecter 2024-04-26T16:29:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 Processor -- start 2024-04-26T16:29:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- start start 2024-04-26T16:29:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 librados: setting wanted keys 2024-04-26T16:29:35.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 librados: calling monclient init 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc102320 0x7f68bc104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 0x7f68bc104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 0x7f68bc104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc111970 con 0x7f68bc102ce0 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f68bc1117f0 con 0x7f68bc102320 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f68bc111af0 con 0x7f68bc108390 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68ba7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 0x7f68bc104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 0x7f68bc104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68ba7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 0x7f68bc104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45538/0 (socket says 172.21.15.50:45538) 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68ba7fc700 1 -- 172.21.15.50:0/773069371 learned_addr learned my addr 172.21.15.50:0/773069371 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 0x7f68bc104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54052/0 (socket says 172.21.15.50:54052) 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68baffd700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc102320 0x7f68bc104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:35.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 -- 172.21.15.50:0/773069371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc102320 msgr2=0x7f68bc104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc102320 0x7f68bc104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 -- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 msgr2=0x7f68bc104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 0x7f68bc104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 -- 172.21.15.50:0/773069371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68bc1a75d0 con 0x7f68bc108390 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68ba7fc700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 0x7f68bc104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68baffd700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc102320 0x7f68bc104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68bb7fe700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 0x7f68bc104a80 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f68b400d010 tx=0x7f68b4004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68a3fff700 10 client.?.objecter ms_handle_connect 0x7f68bc108390 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68a3fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4021070 con 0x7f68bc108390 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f68b4007910 con 0x7f68bc108390 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68bc1a7860 con 0x7f68bc108390 2024-04-26T16:29:35.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68b4005040 con 0x7f68bc108390 2024-04-26T16:29:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 10 client.41267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68bc1a7d60 con 0x7f68bc108390 2024-04-26T16:29:35.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 librados: init done 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.036+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68bc06a110 con 0x7f68bc108390 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f68b4024070 con 0x7f68bc108390 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f68a4071f80 0x7f68a4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(479..479 src has 87..479) ==== 9405+0+0 (secure 0 0 0) 0x7f68b400f020 con 0x7f68bc108390 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 10 client.41267.objecter ms_dispatch 0x7f68bc001550 osd_map(479..479 src has 87..479) 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 3 client.41267.objecter handle_osd_map got epochs [479,479] > 0 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 3 client.41267.objecter handle_osd_map decoding full epoch 479 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68baffd700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f68a4071f80 0x7f68a4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 20 client.41267.objecter dump_active .. 0 homeless 2024-04-26T16:29:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68baffd700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f68a4071f80 0x7f68a4074440 secure :-1 s=READY pgs=3756 cs=0 l=1 rev1=1 crypto rx=0x7f68ac005fd0 tx=0x7f68ac00a5e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:35.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.040+0000 7f68a3fff700 10 client.41267.objecter ms_handle_connect 0x7f68a4071f80 2024-04-26T16:29:35.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.044+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f68b402b030 con 0x7f68bc108390 2024-04-26T16:29:35.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.155 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:35.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:35.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:35.281 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:35.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.284+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f68bc1a8080 con 0x7f68bc108390 2024-04-26T16:29:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:35.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.348+0000 7f68a3fff700 1 -- 172.21.15.50:0/773069371 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v480) ==== 97+0+0 (secure 0 0 0) 0x7f68b40253b0 con 0x7f68bc108390 2024-04-26T16:29:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.352+0000 7f68c1332700 10 librados: watch_flush enter 2024-04-26T16:29:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 10 librados: watch_flush exit 2024-04-26T16:29:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 20 client.41267.objecter shutdown clearing up homeless session... 2024-04-26T16:29:35.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 10 client.41267.objecter successfully canceled tick 2024-04-26T16:29:35.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f68a4071f80 msgr2=0x7f68a4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:35.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f68a4071f80 0x7f68a4074440 secure :-1 s=READY pgs=3756 cs=0 l=1 rev1=1 crypto rx=0x7f68ac005fd0 tx=0x7f68ac00a5e0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 msgr2=0x7f68bc104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:35.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.356+0000 7f68c1332700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 0x7f68bc104a80 secure :-1 s=READY pgs=2671 cs=0 l=1 rev1=1 crypto rx=0x7f68b400d010 tx=0x7f68b4004d00 comp rx=0 tx=0).stop 2024-04-26T16:29:35.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 shutdown_connections 2024-04-26T16:29:35.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f68a4071f80 0x7f68a4074440 unknown :-1 s=CLOSED pgs=3756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68bc102320 0x7f68bc104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f68bc108390 0x7f68bc104a80 unknown :-1 s=CLOSED pgs=2671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 --2- 172.21.15.50:0/773069371 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f68bc102ce0 0x7f68bc104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 >> 172.21.15.50:0/773069371 conn(0x7f68bc0792d0 msgr2=0x7f68bc06b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 shutdown_connections 2024-04-26T16:29:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 -- 172.21.15.50:0/773069371 wait complete. 2024-04-26T16:29:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:35.360+0000 7f68c1332700 1 librados: shutdown 2024-04-26T16:29:35.358 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:29:35.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:35.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:35.510 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:35.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:35 smithi190 bash[23868]: audit 2024-04-26T16:29:35.289844+0000 mon.c (mon.2) 635 : audit [INF] from='client.? 172.21.15.50:0/773069371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:35 smithi190 bash[23868]: audit 2024-04-26T16:29:35.289844+0000 mon.c (mon.2) 635 : audit [INF] from='client.? 172.21.15.50:0/773069371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:35 smithi190 bash[23868]: audit 2024-04-26T16:29:35.290458+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:35 smithi190 bash[23868]: audit 2024-04-26T16:29:35.290458+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:35 smithi190 bash[23868]: cluster 2024-04-26T16:29:35.341438+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:35.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:35 smithi190 bash[23868]: cluster 2024-04-26T16:29:35.341438+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[29234]: audit 2024-04-26T16:29:35.289844+0000 mon.c (mon.2) 635 : audit [INF] from='client.? 172.21.15.50:0/773069371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[29234]: audit 2024-04-26T16:29:35.289844+0000 mon.c (mon.2) 635 : audit [INF] from='client.? 172.21.15.50:0/773069371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[29234]: audit 2024-04-26T16:29:35.290458+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[29234]: audit 2024-04-26T16:29:35.290458+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[29234]: cluster 2024-04-26T16:29:35.341438+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:35.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[29234]: cluster 2024-04-26T16:29:35.341438+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:35.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[21463]: audit 2024-04-26T16:29:35.289844+0000 mon.c (mon.2) 635 : audit [INF] from='client.? 172.21.15.50:0/773069371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[21463]: audit 2024-04-26T16:29:35.289844+0000 mon.c (mon.2) 635 : audit [INF] from='client.? 172.21.15.50:0/773069371' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[21463]: audit 2024-04-26T16:29:35.290458+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[21463]: audit 2024-04-26T16:29:35.290458+0000 mon.a (mon.0) 2795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:29:35.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[21463]: cluster 2024-04-26T16:29:35.341438+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:35.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:35 smithi050 bash[21463]: cluster 2024-04-26T16:29:35.341438+0000 mon.a (mon.0) 2796 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:35.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:35.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:35.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:35.803 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:35.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:35.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:35.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:36.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:36.063 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:36.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:36.194 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:36.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:36.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:36.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:36.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: audit 2024-04-26T16:29:35.343871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: audit 2024-04-26T16:29:35.343871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: cluster 2024-04-26T16:29:35.351182+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: cluster 2024-04-26T16:29:35.351182+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: cluster 2024-04-26T16:29:35.683876+0000 mgr.y (mgr.24401) 476 : cluster [DBG] pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: cluster 2024-04-26T16:29:35.683876+0000 mgr.y (mgr.24401) 476 : cluster [DBG] pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: cluster 2024-04-26T16:29:36.186183+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[21463]: cluster 2024-04-26T16:29:36.186183+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-04-26T16:29:36.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: audit 2024-04-26T16:29:35.343871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: audit 2024-04-26T16:29:35.343871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: cluster 2024-04-26T16:29:35.351182+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: cluster 2024-04-26T16:29:35.351182+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: cluster 2024-04-26T16:29:35.683876+0000 mgr.y (mgr.24401) 476 : cluster [DBG] pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: cluster 2024-04-26T16:29:35.683876+0000 mgr.y (mgr.24401) 476 : cluster [DBG] pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: cluster 2024-04-26T16:29:36.186183+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-04-26T16:29:36.537 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:36 smithi050 bash[29234]: cluster 2024-04-26T16:29:36.186183+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-04-26T16:29:36.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:36.583 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:36.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:36.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:36.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: audit 2024-04-26T16:29:35.343871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:29:36.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: audit 2024-04-26T16:29:35.343871+0000 mon.a (mon.0) 2797 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:29:36.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: cluster 2024-04-26T16:29:35.351182+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-04-26T16:29:36.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: cluster 2024-04-26T16:29:35.351182+0000 mon.a (mon.0) 2798 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2024-04-26T16:29:36.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: cluster 2024-04-26T16:29:35.683876+0000 mgr.y (mgr.24401) 476 : cluster [DBG] pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-26T16:29:36.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: cluster 2024-04-26T16:29:35.683876+0000 mgr.y (mgr.24401) 476 : cluster [DBG] pgmap v718: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 639 B/s wr, 4 op/s 2024-04-26T16:29:36.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: cluster 2024-04-26T16:29:36.186183+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-04-26T16:29:36.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[23868]: cluster 2024-04-26T16:29:36.186183+0000 mon.a (mon.0) 2799 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2024-04-26T16:29:36.715 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:36.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:36.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:36.846 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:36.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:36.961 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:29:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:29:36.977 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:36.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:37.009 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:37.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:37.106 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:37.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:37.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:37.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:37.371 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:37.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:37.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:37.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:37.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:37.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:37.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:37.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:37.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:37.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:37.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:37.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:37.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:38.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:37 smithi190 bash[23868]: audit 2024-04-26T16:29:36.924125+0000 mgr.y (mgr.24401) 477 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:38.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:37 smithi190 bash[23868]: audit 2024-04-26T16:29:36.924125+0000 mgr.y (mgr.24401) 477 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:38.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:37 smithi190 bash[23868]: cluster 2024-04-26T16:29:37.686063+0000 mgr.y (mgr.24401) 478 : cluster [DBG] pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 3.1 KiB/s rd, 640 B/s wr, 5 op/s 2024-04-26T16:29:38.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:37 smithi190 bash[23868]: cluster 2024-04-26T16:29:37.686063+0000 mgr.y (mgr.24401) 478 : cluster [DBG] pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 3.1 KiB/s rd, 640 B/s wr, 5 op/s 2024-04-26T16:29:38.025 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:38.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:38.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[29234]: audit 2024-04-26T16:29:36.924125+0000 mgr.y (mgr.24401) 477 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:38.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[29234]: audit 2024-04-26T16:29:36.924125+0000 mgr.y (mgr.24401) 477 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[29234]: cluster 2024-04-26T16:29:37.686063+0000 mgr.y (mgr.24401) 478 : cluster [DBG] pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 3.1 KiB/s rd, 640 B/s wr, 5 op/s 2024-04-26T16:29:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[29234]: cluster 2024-04-26T16:29:37.686063+0000 mgr.y (mgr.24401) 478 : cluster [DBG] pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 3.1 KiB/s rd, 640 B/s wr, 5 op/s 2024-04-26T16:29:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[21463]: audit 2024-04-26T16:29:36.924125+0000 mgr.y (mgr.24401) 477 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[21463]: audit 2024-04-26T16:29:36.924125+0000 mgr.y (mgr.24401) 477 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[21463]: cluster 2024-04-26T16:29:37.686063+0000 mgr.y (mgr.24401) 478 : cluster [DBG] pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 3.1 KiB/s rd, 640 B/s wr, 5 op/s 2024-04-26T16:29:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:37 smithi050 bash[21463]: cluster 2024-04-26T16:29:37.686063+0000 mgr.y (mgr.24401) 478 : cluster [DBG] pgmap v720: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 3.1 KiB/s rd, 640 B/s wr, 5 op/s 2024-04-26T16:29:38.158 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:38.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:38.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:38.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:38.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:38.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:38.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:29:38.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:38.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:38.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:38.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:38.808 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:38.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:38.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:38.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:39.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:39.066 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:39.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:39.197 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:39.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:39.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:39.328 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:39.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:39.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:39.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.326904+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.326904+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.328111+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.328111+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.328406+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.328406+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.329771+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:39 smithi190 bash[23868]: audit 2024-04-26T16:29:39.329771+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.326904+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.326904+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.328111+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.328111+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.328406+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.328406+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.329771+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[29234]: audit 2024-04-26T16:29:39.329771+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.326904+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.326904+0000 mon.a (mon.0) 2800 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.328111+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.328111+0000 mon.b (mon.1) 328 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.328406+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.328406+0000 mon.a (mon.0) 2801 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.329771+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:39 smithi050 bash[21463]: audit 2024-04-26T16:29:39.329771+0000 mon.b (mon.1) 329 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-65"}]: dispatch 2024-04-26T16:29:39.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:39.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:39.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:39.848 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:39.951 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:39.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:39.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:40.066 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:40.066 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:40.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:40.108 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:40.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:40.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:40.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:40.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:40.368 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:40.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:40.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:40.498 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:40.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:40.627 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:40.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:40 smithi050 bash[29234]: cluster 2024-04-26T16:29:39.686803+0000 mgr.y (mgr.24401) 479 : cluster [DBG] pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:40.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:40 smithi050 bash[29234]: cluster 2024-04-26T16:29:39.686803+0000 mgr.y (mgr.24401) 479 : cluster [DBG] pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:40 smithi050 bash[21463]: cluster 2024-04-26T16:29:39.686803+0000 mgr.y (mgr.24401) 479 : cluster [DBG] pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:40.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:40 smithi050 bash[21463]: cluster 2024-04-26T16:29:39.686803+0000 mgr.y (mgr.24401) 479 : cluster [DBG] pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:40.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:40.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:40 smithi190 bash[23868]: cluster 2024-04-26T16:29:39.686803+0000 mgr.y (mgr.24401) 479 : cluster [DBG] pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:40.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:40 smithi190 bash[23868]: cluster 2024-04-26T16:29:39.686803+0000 mgr.y (mgr.24401) 479 : cluster [DBG] pgmap v721: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:29:40.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:40.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:40.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:40.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:41.021 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:41.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:41.150 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:41.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:41.282 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:41.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:41.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:41.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:41.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:41.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:41 smithi050 bash[29234]: cluster 2024-04-26T16:29:40.392135+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-04-26T16:29:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:41 smithi050 bash[29234]: cluster 2024-04-26T16:29:40.392135+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-04-26T16:29:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:41 smithi050 bash[21463]: cluster 2024-04-26T16:29:40.392135+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-04-26T16:29:41.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:41 smithi050 bash[21463]: cluster 2024-04-26T16:29:40.392135+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-04-26T16:29:41.658 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:41.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:41.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:41.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:41 smithi190 bash[23868]: cluster 2024-04-26T16:29:40.392135+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-04-26T16:29:41.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:41 smithi190 bash[23868]: cluster 2024-04-26T16:29:40.392135+0000 mon.a (mon.0) 2802 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2024-04-26T16:29:41.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:41.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:41.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:42.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:42.077 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/4136815271 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 msgr2=0x7f503c110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- 172.21.15.50:0/4136815271 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c110210 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f5034009f90 tx=0x7f503400b330 comp rx=0 tx=0).stop 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/4136815271 shutdown_connections 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- 172.21.15.50:0/4136815271 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c110210 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- 172.21.15.50:0/4136815271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 0x7f503c1088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- 172.21.15.50:0/4136815271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503c101de0 0x7f503c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/4136815271 >> 172.21.15.50:0/4136815271 conn(0x7f503c077640 msgr2=0x7f503c077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/4136815271 shutdown_connections 2024-04-26T16:29:42.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/4136815271 wait complete. 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 librados: starting msgr at 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 librados: starting objecter 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 Processor -- start 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- start start 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 librados: setting wanted keys 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 librados: calling monclient init 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503c101de0 0x7f503c199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 0x7f503c19d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f503c112b00 con 0x7f503c108500 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f503c112980 con 0x7f503c101de0 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f503c112c80 con 0x7f503c1027a0 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5040fb3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 0x7f503c19d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c1995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35298/0 (socket says 172.21.15.50:35298) 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 -- 172.21.15.50:0/941613018 learned_addr learned my addr 172.21.15.50:0/941613018 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5040fb3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 0x7f503c19d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56346/0 (socket says 172.21.15.50:56346) 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503c101de0 0x7f503c199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503c101de0 msgr2=0x7f503c199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503c101de0 0x7f503c199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 msgr2=0x7f503c19d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 0x7f503c19d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503c19e0a0 con 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f503c101de0 0x7f503c199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f503c108500 0x7f503c19d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c1995d0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f503000d980 tx=0x7f503000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f50397fa700 10 client.?.objecter ms_handle_connect 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f50397fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030017070 con 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5030005020 con 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503c19e390 con 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030009990 con 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 10 client.41273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f503c102ef0 con 0x7f503c1027a0 2024-04-26T16:29:42.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.100+0000 7f5042a16700 1 librados: init done 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f503c06a110 con 0x7f503c1027a0 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5030004a30 con 0x7f503c1027a0 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5024071f80 0x7f5024074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(483..483 src has 378..483) ==== 9364+0+0 (secure 0 0 0) 0x7f50300918c0 con 0x7f503c1027a0 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 osd_map(483..483 src has 378..483) 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 3 client.41273.objecter handle_osd_map got epochs [483,483] > 0 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 3 client.41273.objecter handle_osd_map decoding full epoch 483 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5024071f80 0x7f5024074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 20 client.41273.objecter dump_active .. 0 homeless 2024-04-26T16:29:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v483) ==== 54+0+16 (secure 0 0 0) 0x7f5030020020 con 0x7f503c1027a0 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5024071f80 0x7f5024074440 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 crypto rx=0x7f502c009750 tx=0x7f502c009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f5024071f80 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 10 client.41273.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 1 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f503c106d30 0x7f503c02d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 1 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 0 1 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 4 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 10 client.41273.objecter _send_command 1 2024-04-26T16:29:42.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f503c064540 con 0x7f503c106d30 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f503c106d30 0x7f503c02d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f503c106d30 0x7f503c02d310 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c106d30 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c064540 con 0x7f503c106d30 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 1: 0 ) 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 10 client.41273.objecter _finish_command 1 = system:0 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 4 2024-04-26T16:29:42.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.104+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 0 1 2024-04-26T16:29:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 10 client.41273.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 2 2024-04-26T16:29:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.122 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 2 2024-04-26T16:29:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c106bc0 osd=0 3 2024-04-26T16:29:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 0 2 2024-04-26T16:29:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 4 2024-04-26T16:29:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 10 client.41273.objecter _send_command 2 2024-04-26T16:29:42.123 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c02d850 con 0x7f503c106d30 2024-04-26T16:29:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c02d850 con 0x7f503c106d30 2024-04-26T16:29:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f50397fa700 10 client.41273.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 4 2024-04-26T16:29:42.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 0 2 2024-04-26T16:29:42.124 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 10 client.41273.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 3 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f503c063c20 0x7f503c103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 3 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 1 3 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 4 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 10 client.41273.objecter _send_command 3 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f503c104e30 con 0x7f503c063c20 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f503c063c20 0x7f503c103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c063c20 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.124+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f503c063c20 0x7f503c103150 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.128+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c104e30 con 0x7f503c063c20 2024-04-26T16:29:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.128+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 3: 0 ) 2024-04-26T16:29:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.128+0000 7f50397fa700 10 client.41273.objecter _finish_command 3 = system:0 2024-04-26T16:29:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.128+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 4 2024-04-26T16:29:42.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.128+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 1 3 2024-04-26T16:29:42.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 10 client.41273.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 4 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 4 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c063ab0 osd=1 3 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 1 4 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 4 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 10 client.41273.objecter _send_command 4 2024-04-26T16:29:42.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.144+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c103690 con 0x7f503c063c20 2024-04-26T16:29:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c103690 con 0x7f503c063c20 2024-04-26T16:29:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 10 client.41273.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 4 2024-04-26T16:29:42.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 1 4 2024-04-26T16:29:42.145 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 10 client.41273.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 5 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f503c10bae0 0x7f503c10bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 5 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 2 5 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 4 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 10 client.41273.objecter _send_command 5 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f503c104b90 con 0x7f503c10bae0 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f503c10bae0 0x7f503c10bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f503c10bae0 0x7f503c10bf40 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c10bae0 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c104b90 con 0x7f503c10bae0 2024-04-26T16:29:42.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 5: 0 ) 2024-04-26T16:29:42.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 10 client.41273.objecter _finish_command 5 = system:0 2024-04-26T16:29:42.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 4 2024-04-26T16:29:42.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.148+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 2 5 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 10 client.41273.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 6 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 6 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c104a20 osd=2 3 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 2 6 2024-04-26T16:29:42.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 4 2024-04-26T16:29:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 10 client.41273.objecter _send_command 6 2024-04-26T16:29:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.164+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c10b560 con 0x7f503c10bae0 2024-04-26T16:29:42.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c10b560 con 0x7f503c10bae0 2024-04-26T16:29:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 10 client.41273.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 4 2024-04-26T16:29:42.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 2 6 2024-04-26T16:29:42.166 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 10 client.41273.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 7 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f503c104520 0x7f503c1aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 7 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 3 7 2024-04-26T16:29:42.167 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 4 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 10 client.41273.objecter _send_command 7 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f503c1b1c30 con 0x7f503c104520 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f503c104520 0x7f503c1aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c104520 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f503c104520 0x7f503c1aedd0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c1b1c30 con 0x7f503c104520 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 7: 0 ) 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 10 client.41273.objecter _finish_command 7 = system:0 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 4 2024-04-26T16:29:42.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.168+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 3 7 2024-04-26T16:29:42.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 10 client.41273.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 8 2024-04-26T16:29:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 8 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c10b960 osd=3 3 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 3 8 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 4 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 10 client.41273.objecter _send_command 8 2024-04-26T16:29:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c103b50 con 0x7f503c104520 2024-04-26T16:29:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c103b50 con 0x7f503c104520 2024-04-26T16:29:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f50397fa700 10 client.41273.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 4 2024-04-26T16:29:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 3 8 2024-04-26T16:29:42.187 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 10 client.41273.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 9 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f503c1ac920 0x7f503c1b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 9 2024-04-26T16:29:42.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 4 9 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 4 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 10 client.41273.objecter _send_command 9 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f503c103ba0 con 0x7f503c1ac920 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f503c1ac920 0x7f503c1b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f503c1ac920 0x7f503c1b24a0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.188+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c1ac920 2024-04-26T16:29:42.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.192+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f502c0127d0 con 0x7f503c1ac920 2024-04-26T16:29:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.192+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 9: 0 ) 2024-04-26T16:29:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.192+0000 7f50397fa700 10 client.41273.objecter _finish_command 9 = system:0 2024-04-26T16:29:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.192+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 4 2024-04-26T16:29:42.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.192+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 4 9 2024-04-26T16:29:42.208 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:42.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 10 client.41273.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 10 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 10 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1038a0 osd=4 3 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 4 10 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 4 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 10 client.41273.objecter _send_command 10 2024-04-26T16:29:42.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.208+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c1abf50 con 0x7f503c1ac920 2024-04-26T16:29:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c1abf50 con 0x7f503c1ac920 2024-04-26T16:29:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 10 client.41273.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 4 2024-04-26T16:29:42.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 4 10 2024-04-26T16:29:42.212 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:29:42.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 10 client.41273.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 11 2024-04-26T16:29:42.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f503c1afff0 0x7f503c1b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 11 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 5 11 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 4 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 10 client.41273.objecter _send_command 11 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f503c1b8b40 con 0x7f503c1afff0 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f503c1afff0 0x7f503c1b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f503b7fe700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f503c1afff0 0x7f503c1b3b60 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c1afff0 2024-04-26T16:29:42.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c1b8b40 con 0x7f503c1afff0 2024-04-26T16:29:42.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 11: 0 ) 2024-04-26T16:29:42.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 10 client.41273.objecter _finish_command 11 = system:0 2024-04-26T16:29:42.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.212+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 4 2024-04-26T16:29:42.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.216+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 5 11 2024-04-26T16:29:42.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 10 client.41273.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 12 2024-04-26T16:29:42.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 12 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1abd00 osd=5 3 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 5 12 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 4 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 10 client.41273.objecter _send_command 12 2024-04-26T16:29:42.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c1b40a0 con 0x7f503c1afff0 2024-04-26T16:29:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c1b40a0 con 0x7f503c1afff0 2024-04-26T16:29:42.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f50397fa700 10 client.41273.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 4 2024-04-26T16:29:42.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 5 12 2024-04-26T16:29:42.233 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 10 client.41273.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 13 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f503c1b47c0 0x7f503c1b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 13 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 6 13 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 4 2024-04-26T16:29:42.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 10 client.41273.objecter _send_command 13 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.232+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f503c1bbd10 con 0x7f503c1b47c0 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f503c1b47c0 0x7f503c1b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f5040fb3700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f503c1b47c0 0x7f503c1b4c00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c1b47c0 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c1bbd10 con 0x7f503c1b47c0 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 13: 0 ) 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f50397fa700 10 client.41273.objecter _finish_command 13 = system:0 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 4 2024-04-26T16:29:42.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.236+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 6 13 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 10 client.41273.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 14 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 14 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1b40a0 osd=6 3 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 6 14 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 4 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 10 client.41273.objecter _send_command 14 2024-04-26T16:29:42.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.252+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c1b93b0 con 0x7f503c1b47c0 2024-04-26T16:29:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c1b93b0 con 0x7f503c1b47c0 2024-04-26T16:29:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f50397fa700 10 client.41273.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 4 2024-04-26T16:29:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 6 14 2024-04-26T16:29:42.255 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 10 client.41273.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 15 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f503c1b9ad0 0x7f503c1b9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 15 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 7 15 2024-04-26T16:29:42.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 4 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 10 client.41273.objecter _send_command 15 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f503c1ba5f0 con 0x7f503c1b9ad0 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f503c1b9ad0 0x7f503c1b9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f503bfff700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f503c1b9ad0 0x7f503c1b9f10 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.256+0000 7f50397fa700 10 client.41273.objecter ms_handle_connect 0x7f503c1b9ad0 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.260+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f503c1ba5f0 con 0x7f503c1b9ad0 2024-04-26T16:29:42.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.260+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 15: 0 ) 2024-04-26T16:29:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.260+0000 7f50397fa700 10 client.41273.objecter _finish_command 15 = system:0 2024-04-26T16:29:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.260+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 4 2024-04-26T16:29:42.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.260+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 7 15 2024-04-26T16:29:42.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign -1 16 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter _get_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 15 client.41273.objecter _session_command_op_remove -1 16 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter get_session s=0x7f503c1b93b0 osd=7 3 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 15 client.41273.objecter _session_command_op_assign 7 16 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 4 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter _send_command 16 2024-04-26T16:29:42.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f503c1ba5b0 con 0x7f503c1b9ad0 2024-04-26T16:29:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f50397fa700 1 -- 172.21.15.50:0/941613018 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f503c1ba5b0 con 0x7f503c1b9ad0 2024-04-26T16:29:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f50397fa700 10 client.41273.objecter ms_dispatch 0x7f503c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f50397fa700 10 client.41273.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f50397fa700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 4 2024-04-26T16:29:42.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f50397fa700 15 client.41273.objecter _session_command_op_remove 7 16 2024-04-26T16:29:42.280 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:29:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 librados: watch_flush enter 2024-04-26T16:29:42.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 librados: watch_flush exit 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.0 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f503c106d30 msgr2=0x7f503c02d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f503c106d30 0x7f503c02d310 crc :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c106bc0 osd=0 1 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.1 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f503c063c20 msgr2=0x7f503c103150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f503c063c20 0x7f503c103150 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c063ab0 osd=1 1 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.2 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f503c10bae0 msgr2=0x7f503c10bf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f503c10bae0 0x7f503c10bf40 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c104a20 osd=2 1 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.3 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f503c104520 msgr2=0x7f503c1aedd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5040fb3700 1 -- 172.21.15.50:0/941613018 reap_dead start 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f503c104520 0x7f503c1aedd0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c10b960 osd=3 1 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.4 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f503c1ac920 msgr2=0x7f503c1b24a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f503c1ac920 0x7f503c1b24a0 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1038a0 osd=4 1 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.5 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f503c1afff0 msgr2=0x7f503c1b3b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f503c1afff0 0x7f503c1b3b60 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1abd00 osd=5 1 2024-04-26T16:29:42.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.6 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f503c1b47c0 msgr2=0x7f503c1b4c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f503c1b47c0 0x7f503c1b4c00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b40a0 osd=6 1 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter close_session for osd.7 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f503c1b9ad0 msgr2=0x7f503c1b9f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f503c1b9ad0 0x7f503c1b9f10 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter put_session s=0x7f503c1b93b0 osd=7 1 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 20 client.41273.objecter shutdown clearing up homeless session... 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 10 client.41273.objecter successfully canceled tick 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5040fb3700 1 -- 172.21.15.50:0/941613018 reap_dead start 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5024071f80 msgr2=0x7f5024074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5024071f80 0x7f5024074440 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 crypto rx=0x7f502c009750 tx=0x7f502c009310 comp rx=0 tx=0).stop 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 msgr2=0x7f503c1995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c1995d0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f503000d980 tx=0x7f503000dc90 comp rx=0 tx=0).stop 2024-04-26T16:29:42.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 shutdown_connections 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f503c1027a0 0x7f503c1995d0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 --2- 172.21.15.50:0/941613018 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5024071f80 0x7f5024074440 unknown :-1 s=CLOSED pgs=3757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.280+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 >> 172.21.15.50:0/941613018 conn(0x7f503c077640 msgr2=0x7f503c105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.284+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 shutdown_connections 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.284+0000 7f5042a16700 1 -- 172.21.15.50:0/941613018 wait complete. 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:42.284+0000 7f5042a16700 1 librados: shutdown 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:29:42.285 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:29:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:29:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:29:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:29:42.286 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:29:42.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:42.330 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:42.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:42.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:42.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:42.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: cluster 2024-04-26T16:29:41.408258+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: cluster 2024-04-26T16:29:41.408258+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:41.415305+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:41.415305+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:41.416214+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:41.416214+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: cluster 2024-04-26T16:29:41.688232+0000 mgr.y (mgr.24401) 480 : cluster [DBG] pgmap v724: 228 pgs: 27 creating+peering, 5 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: cluster 2024-04-26T16:29:41.688232+0000 mgr.y (mgr.24401) 480 : cluster [DBG] pgmap v724: 228 pgs: 27 creating+peering, 5 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:41.934156+0000 mon.c (mon.2) 636 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:41.934156+0000 mon.c (mon.2) 636 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:42.108051+0000 mon.c (mon.2) 637 : audit [DBG] from='client.? 172.21.15.50:0/941613018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:42.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[21463]: audit 2024-04-26T16:29:42.108051+0000 mon.c (mon.2) 637 : audit [DBG] from='client.? 172.21.15.50:0/941613018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: cluster 2024-04-26T16:29:41.408258+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: cluster 2024-04-26T16:29:41.408258+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:41.415305+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:41.415305+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:41.416214+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:41.416214+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: cluster 2024-04-26T16:29:41.688232+0000 mgr.y (mgr.24401) 480 : cluster [DBG] pgmap v724: 228 pgs: 27 creating+peering, 5 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: cluster 2024-04-26T16:29:41.688232+0000 mgr.y (mgr.24401) 480 : cluster [DBG] pgmap v724: 228 pgs: 27 creating+peering, 5 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:41.934156+0000 mon.c (mon.2) 636 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:41.934156+0000 mon.c (mon.2) 636 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:42.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:42.108051+0000 mon.c (mon.2) 637 : audit [DBG] from='client.? 172.21.15.50:0/941613018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:42.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:42 smithi050 bash[29234]: audit 2024-04-26T16:29:42.108051+0000 mon.c (mon.2) 637 : audit [DBG] from='client.? 172.21.15.50:0/941613018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:42.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:42.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:42.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: cluster 2024-04-26T16:29:41.408258+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-04-26T16:29:42.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: cluster 2024-04-26T16:29:41.408258+0000 mon.a (mon.0) 2803 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2024-04-26T16:29:42.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:41.415305+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:41.415305+0000 mon.a (mon.0) 2804 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:41.416214+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:41.416214+0000 mon.b (mon.1) 330 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: cluster 2024-04-26T16:29:41.688232+0000 mgr.y (mgr.24401) 480 : cluster [DBG] pgmap v724: 228 pgs: 27 creating+peering, 5 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: cluster 2024-04-26T16:29:41.688232+0000 mgr.y (mgr.24401) 480 : cluster [DBG] pgmap v724: 228 pgs: 27 creating+peering, 5 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:41.934156+0000 mon.c (mon.2) 636 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:41.934156+0000 mon.c (mon.2) 636 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:42.108051+0000 mon.c (mon.2) 637 : audit [DBG] from='client.? 172.21.15.50:0/941613018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:42.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:42 smithi190 bash[23868]: audit 2024-04-26T16:29:42.108051+0000 mon.c (mon.2) 637 : audit [DBG] from='client.? 172.21.15.50:0/941613018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:29:42.720 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:42.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:42.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:42.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:42.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:42.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:43.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:43.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:43.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:43.247 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:43.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:43.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:43.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: cluster 2024-04-26T16:29:42.403588+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: cluster 2024-04-26T16:29:42.403588+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: audit 2024-04-26T16:29:42.411698+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: audit 2024-04-26T16:29:42.411698+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: cluster 2024-04-26T16:29:42.417046+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: cluster 2024-04-26T16:29:42.417046+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: audit 2024-04-26T16:29:42.427047+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:43.493 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:43 smithi190 bash[23868]: audit 2024-04-26T16:29:42.427047+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:43.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:43.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:43.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: cluster 2024-04-26T16:29:42.403588+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: cluster 2024-04-26T16:29:42.403588+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: audit 2024-04-26T16:29:42.411698+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: audit 2024-04-26T16:29:42.411698+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: cluster 2024-04-26T16:29:42.417046+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: cluster 2024-04-26T16:29:42.417046+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: audit 2024-04-26T16:29:42.427047+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[21463]: audit 2024-04-26T16:29:42.427047+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: cluster 2024-04-26T16:29:42.403588+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: cluster 2024-04-26T16:29:42.403588+0000 mon.a (mon.0) 2805 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: audit 2024-04-26T16:29:42.411698+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: audit 2024-04-26T16:29:42.411698+0000 mon.a (mon.0) 2806 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: cluster 2024-04-26T16:29:42.417046+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-04-26T16:29:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: cluster 2024-04-26T16:29:42.417046+0000 mon.a (mon.0) 2807 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2024-04-26T16:29:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: audit 2024-04-26T16:29:42.427047+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:43 smithi050 bash[29234]: audit 2024-04-26T16:29:42.427047+0000 mon.b (mon.1) 331 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:43.639 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:43.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:43.767 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:43.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:43.898 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:43.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:44.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.025 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:44.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.155 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:44.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:44.286 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:44.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.415 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:44.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:44.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: cluster 2024-04-26T16:29:43.431140+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-04-26T16:29:44.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: cluster 2024-04-26T16:29:43.431140+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.499501+0000 mon.a (mon.0) 2809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.499501+0000 mon.a (mon.0) 2809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.500317+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.500317+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.501485+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.501485+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.502620+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: audit 2024-04-26T16:29:43.502620+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: cluster 2024-04-26T16:29:43.688994+0000 mgr.y (mgr.24401) 481 : cluster [DBG] pgmap v727: 228 pgs: 27 creating+peering, 201 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:44.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:44 smithi190 bash[23868]: cluster 2024-04-26T16:29:43.688994+0000 mgr.y (mgr.24401) 481 : cluster [DBG] pgmap v727: 228 pgs: 27 creating+peering, 201 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:44.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:29:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:29:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: cluster 2024-04-26T16:29:43.431140+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-04-26T16:29:44.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: cluster 2024-04-26T16:29:43.431140+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.499501+0000 mon.a (mon.0) 2809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.499501+0000 mon.a (mon.0) 2809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.500317+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.500317+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.501485+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.501485+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.502620+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: audit 2024-04-26T16:29:43.502620+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: cluster 2024-04-26T16:29:43.688994+0000 mgr.y (mgr.24401) 481 : cluster [DBG] pgmap v727: 228 pgs: 27 creating+peering, 201 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[29234]: cluster 2024-04-26T16:29:43.688994+0000 mgr.y (mgr.24401) 481 : cluster [DBG] pgmap v727: 228 pgs: 27 creating+peering, 201 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:44.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: cluster 2024-04-26T16:29:43.431140+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: cluster 2024-04-26T16:29:43.431140+0000 mon.a (mon.0) 2808 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.499501+0000 mon.a (mon.0) 2809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.499501+0000 mon.a (mon.0) 2809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.500317+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.500317+0000 mon.b (mon.1) 332 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.501485+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.501485+0000 mon.a (mon.0) 2810 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.502620+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: audit 2024-04-26T16:29:43.502620+0000 mon.b (mon.1) 333 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-67"}]: dispatch 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: cluster 2024-04-26T16:29:43.688994+0000 mgr.y (mgr.24401) 481 : cluster [DBG] pgmap v727: 228 pgs: 27 creating+peering, 201 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:44.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:44 smithi050 bash[21463]: cluster 2024-04-26T16:29:43.688994+0000 mgr.y (mgr.24401) 481 : cluster [DBG] pgmap v727: 228 pgs: 27 creating+peering, 201 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:44.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.670 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:44.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:44.799 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:44.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:44.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:44.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:45.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:45.061 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:45.164 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:45.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:45.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:45.322 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:45.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:45.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:45.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:45.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:45 smithi190 bash[23868]: cluster 2024-04-26T16:29:44.446605+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-04-26T16:29:45.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:45 smithi190 bash[23868]: cluster 2024-04-26T16:29:44.446605+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-04-26T16:29:45.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:45.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:45.730 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:45.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:45.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:45 smithi050 bash[29234]: cluster 2024-04-26T16:29:44.446605+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-04-26T16:29:45.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:45 smithi050 bash[29234]: cluster 2024-04-26T16:29:44.446605+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-04-26T16:29:45.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:45 smithi050 bash[21463]: cluster 2024-04-26T16:29:44.446605+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-04-26T16:29:45.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:45 smithi050 bash[21463]: cluster 2024-04-26T16:29:44.446605+0000 mon.a (mon.0) 2811 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2024-04-26T16:29:45.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:45.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:45.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:45.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:46.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:46.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:46.226 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:46.253 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:46.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:46.367 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:46.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:46.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:46.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:29:46.510 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:46.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:46.642 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:46.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: cluster 2024-04-26T16:29:45.454044+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-04-26T16:29:46.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: cluster 2024-04-26T16:29:45.454044+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-04-26T16:29:46.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:45.477023+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:45.477023+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:45.477665+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:45.477665+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.737 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: cluster 2024-04-26T16:29:45.689832+0000 mgr.y (mgr.24401) 482 : cluster [DBG] pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: cluster 2024-04-26T16:29:45.689832+0000 mgr.y (mgr.24401) 482 : cluster [DBG] pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.195193+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.195193+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: cluster 2024-04-26T16:29:46.201024+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: cluster 2024-04-26T16:29:46.201024+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.211794+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:46.738 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.211794+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.281028+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.281028+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.282181+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: cluster 2024-04-26T16:29:45.454044+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: cluster 2024-04-26T16:29:45.454044+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:45.477023+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:45.477023+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:45.477665+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:45.477665+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: cluster 2024-04-26T16:29:45.689832+0000 mgr.y (mgr.24401) 482 : cluster [DBG] pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: cluster 2024-04-26T16:29:45.689832+0000 mgr.y (mgr.24401) 482 : cluster [DBG] pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.195193+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.195193+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: cluster 2024-04-26T16:29:46.201024+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: cluster 2024-04-26T16:29:46.201024+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.211794+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.211794+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.281028+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.281028+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.282181+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.282181+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.282504+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.282504+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.283854+0000 mon.b (mon.1) 337 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[29234]: audit 2024-04-26T16:29:46.283854+0000 mon.b (mon.1) 337 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.282181+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.282504+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.282504+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.283854+0000 mon.b (mon.1) 337 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:46 smithi050 bash[21463]: audit 2024-04-26T16:29:46.283854+0000 mon.b (mon.1) 337 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:46.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:46.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: cluster 2024-04-26T16:29:45.454044+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-04-26T16:29:46.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: cluster 2024-04-26T16:29:45.454044+0000 mon.a (mon.0) 2812 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2024-04-26T16:29:46.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:45.477023+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:45.477023+0000 mon.a (mon.0) 2813 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:45.477665+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:45.477665+0000 mon.b (mon.1) 334 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: cluster 2024-04-26T16:29:45.689832+0000 mgr.y (mgr.24401) 482 : cluster [DBG] pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: cluster 2024-04-26T16:29:45.689832+0000 mgr.y (mgr.24401) 482 : cluster [DBG] pgmap v730: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:29:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.195193+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.195193+0000 mon.a (mon.0) 2814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: cluster 2024-04-26T16:29:46.201024+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-04-26T16:29:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: cluster 2024-04-26T16:29:46.201024+0000 mon.a (mon.0) 2815 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2024-04-26T16:29:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.211794+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.211794+0000 mon.b (mon.1) 335 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.281028+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.281028+0000 mon.a (mon.0) 2816 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.282181+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.282181+0000 mon.b (mon.1) 336 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:29:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.282504+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.282504+0000 mon.a (mon.0) 2817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.283854+0000 mon.b (mon.1) 337 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[23868]: audit 2024-04-26T16:29:46.283854+0000 mon.b (mon.1) 337 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-69"}]: dispatch 2024-04-26T16:29:46.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:46.898 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:46.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:47.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:47.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:47.133 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:47.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:47.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:47.290 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:47.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:29:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:29:47.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:47.422 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:47.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:47.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:47.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:47.682 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:47.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:47.812 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:47.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:47.916 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:47.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:47.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:47.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:48.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:48.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:48.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:48.206 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:48.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:48.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:48.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:48.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:48.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:48 smithi190 bash[23868]: audit 2024-04-26T16:29:46.932080+0000 mgr.y (mgr.24401) 483 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:48.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:48 smithi190 bash[23868]: audit 2024-04-26T16:29:46.932080+0000 mgr.y (mgr.24401) 483 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:48.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:48 smithi190 bash[23868]: cluster 2024-04-26T16:29:47.202312+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-04-26T16:29:48.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:48 smithi190 bash[23868]: cluster 2024-04-26T16:29:47.202312+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-04-26T16:29:48.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:48 smithi190 bash[23868]: cluster 2024-04-26T16:29:47.690718+0000 mgr.y (mgr.24401) 484 : cluster [DBG] pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-26T16:29:48.510 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:48 smithi190 bash[23868]: cluster 2024-04-26T16:29:47.690718+0000 mgr.y (mgr.24401) 484 : cluster [DBG] pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-26T16:29:48.526 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:48.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[21463]: audit 2024-04-26T16:29:46.932080+0000 mgr.y (mgr.24401) 483 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[21463]: audit 2024-04-26T16:29:46.932080+0000 mgr.y (mgr.24401) 483 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[21463]: cluster 2024-04-26T16:29:47.202312+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-04-26T16:29:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[21463]: cluster 2024-04-26T16:29:47.202312+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-04-26T16:29:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[21463]: cluster 2024-04-26T16:29:47.690718+0000 mgr.y (mgr.24401) 484 : cluster [DBG] pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-26T16:29:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[21463]: cluster 2024-04-26T16:29:47.690718+0000 mgr.y (mgr.24401) 484 : cluster [DBG] pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-26T16:29:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[29234]: audit 2024-04-26T16:29:46.932080+0000 mgr.y (mgr.24401) 483 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[29234]: audit 2024-04-26T16:29:46.932080+0000 mgr.y (mgr.24401) 483 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[29234]: cluster 2024-04-26T16:29:47.202312+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-04-26T16:29:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[29234]: cluster 2024-04-26T16:29:47.202312+0000 mon.a (mon.0) 2818 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2024-04-26T16:29:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[29234]: cluster 2024-04-26T16:29:47.690718+0000 mgr.y (mgr.24401) 484 : cluster [DBG] pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-26T16:29:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:48 smithi050 bash[29234]: cluster 2024-04-26T16:29:47.690718+0000 mgr.y (mgr.24401) 484 : cluster [DBG] pgmap v733: 196 pgs: 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-04-26T16:29:48.658 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:48.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:48.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:48.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:48.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:49.022 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:49.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:49.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:49.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:49.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:49.283 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:29:49.283 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/15364003 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806ba50 msgr2=0x7f4b9810f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/15364003 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806ba50 0x7f4b9810f990 secure :-1 s=READY pgs=2674 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009990 tx=0x7f4b80009ca0 comp rx=0 tx=0).stop 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/15364003 shutdown_connections 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/15364003 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b9806b130 0x7f4b9806b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/15364003 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9810fed0 0x7f4b981122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/15364003 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806ba50 0x7f4b9810f990 unknown :-1 s=CLOSED pgs=2674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/15364003 >> 172.21.15.50:0/15364003 conn(0x7f4b980792d0 msgr2=0x7f4b980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/15364003 shutdown_connections 2024-04-26T16:29:49.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/15364003 wait complete. 2024-04-26T16:29:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 librados: starting msgr at 2024-04-26T16:29:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 librados: starting objecter 2024-04-26T16:29:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 Processor -- start 2024-04-26T16:29:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 -- start start 2024-04-26T16:29:49.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 librados: setting wanted keys 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.284+0000 7f4b9d9f1700 1 librados: calling monclient init 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806b130 0x7f4b9819b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 0x7f4b9819b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b9810fed0 0x7f4b9819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98114dc0 con 0x7f4b9806ba50 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98114c40 con 0x7f4b9810fed0 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4b98114f40 con 0x7f4b9806b130 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b967fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 0x7f4b9819b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:49.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806b130 0x7f4b9819b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b967fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 0x7f4b9819b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38606/0 (socket says 172.21.15.50:38606) 2024-04-26T16:29:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b967fc700 1 -- 172.21.15.50:0/4039346382 learned_addr learned my addr 172.21.15.50:0/4039346382 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:49.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b977fe700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b9810fed0 0x7f4b9819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 -- 172.21.15.50:0/4039346382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b9810fed0 msgr2=0x7f4b9819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b9810fed0 0x7f4b9819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 -- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 msgr2=0x7f4b9819b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:49.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 0x7f4b9819b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:49.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: cluster 2024-04-26T16:29:48.198265+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:49.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: cluster 2024-04-26T16:29:48.198265+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:49.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: cluster 2024-04-26T16:29:48.222250+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-04-26T16:29:49.288 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: cluster 2024-04-26T16:29:48.222250+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-04-26T16:29:49.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: audit 2024-04-26T16:29:48.226849+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: audit 2024-04-26T16:29:48.226849+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: audit 2024-04-26T16:29:48.227688+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.289 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[29234]: audit 2024-04-26T16:29:48.227688+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 -- 172.21.15.50:0/4039346382 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b981a03f0 con 0x7f4b9806b130 2024-04-26T16:29:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b967fc700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 0x7f4b9819b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:29:49.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b96ffd700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806b130 0x7f4b9819b3e0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f4b88007f10 tx=0x7f4b8800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:49.289 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: cluster 2024-04-26T16:29:48.198265+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: cluster 2024-04-26T16:29:48.198265+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: cluster 2024-04-26T16:29:48.222250+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: cluster 2024-04-26T16:29:48.222250+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: audit 2024-04-26T16:29:48.226849+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: audit 2024-04-26T16:29:48.226849+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: audit 2024-04-26T16:29:48.227688+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.290 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:49 smithi050 bash[21463]: audit 2024-04-26T16:29:48.227688+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9c9ef700 10 client.?.objecter ms_handle_connect 0x7f4b9806b130 2024-04-26T16:29:49.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9c9ef700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88004db0 con 0x7f4b9806b130 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4b88005020 con 0x7f4b9806b130 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b88015660 con 0x7f4b9806b130 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b981a06e0 con 0x7f4b9806b130 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 10 client.41279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b981a7f60 con 0x7f4b9806b130 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.288+0000 7f4b9d9f1700 1 librados: init done 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4b8800dac0 con 0x7f4b9806b130 2024-04-26T16:29:49.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4b84072050 0x7f4b84074510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9806a110 con 0x7f4b9806b130 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b967fc700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4b84072050 0x7f4b84074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b967fc700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4b84072050 0x7f4b84074510 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 crypto rx=0x7f4b981021d0 tx=0x7f4b8000b540 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(491..491 src has 378..491) ==== 9377+0+0 (secure 0 0 0) 0x7f4b88091310 con 0x7f4b9806b130 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 10 client.41279.objecter ms_dispatch 0x7f4b98001550 osd_map(491..491 src has 378..491) 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 3 client.41279.objecter handle_osd_map got epochs [491,491] > 0 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 3 client.41279.objecter handle_osd_map decoding full epoch 491 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 20 client.41279.objecter dump_active .. 0 homeless 2024-04-26T16:29:49.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.292+0000 7f4b9c9ef700 10 client.41279.objecter ms_handle_connect 0x7f4b84072050 2024-04-26T16:29:49.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.296+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4b88010070 con 0x7f4b9806b130 2024-04-26T16:29:49.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:49.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:49.431 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:49.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:49.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:49.544+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f4b981a8130 con 0x7f4b9806b130 2024-04-26T16:29:49.545 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: cluster 2024-04-26T16:29:48.198265+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: cluster 2024-04-26T16:29:48.198265+0000 mon.a (mon.0) 2819 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: cluster 2024-04-26T16:29:48.222250+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: cluster 2024-04-26T16:29:48.222250+0000 mon.a (mon.0) 2820 : cluster [DBG] osdmap e490: 8 total, 8 up, 7 in 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: audit 2024-04-26T16:29:48.226849+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: audit 2024-04-26T16:29:48.226849+0000 mon.a (mon.0) 2821 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: audit 2024-04-26T16:29:48.227688+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:49 smithi190 bash[23868]: audit 2024-04-26T16:29:48.227688+0000 mon.b (mon.1) 338 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:29:49.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:49.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:49.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:49.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:49.823 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:49.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:49.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:49.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:50.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:50.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:50.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:50.213 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.224+0000 7f4b9c9ef700 1 -- 172.21.15.50:0/4039346382 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v492) ==== 113+0+0 (secure 0 0 0) 0x7f4b8805cd60 con 0x7f4b9806b130 2024-04-26T16:29:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 10 librados: watch_flush enter 2024-04-26T16:29:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 10 librados: watch_flush exit 2024-04-26T16:29:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 20 client.41279.objecter shutdown clearing up homeless session... 2024-04-26T16:29:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 10 client.41279.objecter successfully canceled tick 2024-04-26T16:29:50.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4b84072050 msgr2=0x7f4b84074510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4b84072050 0x7f4b84074510 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 crypto rx=0x7f4b981021d0 tx=0x7f4b8000b540 comp rx=0 tx=0).stop 2024-04-26T16:29:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806b130 msgr2=0x7f4b9819b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:50.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806b130 0x7f4b9819b3e0 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f4b88007f10 tx=0x7f4b8800d3b0 comp rx=0 tx=0).stop 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 shutdown_connections 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4b84072050 0x7f4b84074510 unknown :-1 s=CLOSED pgs=3758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b9810fed0 0x7f4b9819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4b9806b130 0x7f4b9819b3e0 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 --2- 172.21.15.50:0/4039346382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4b9806ba50 0x7f4b9819b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.228+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 >> 172.21.15.50:0/4039346382 conn(0x7f4b980792d0 msgr2=0x7f4b98111140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.232+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 shutdown_connections 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.232+0000 7f4b9d9f1700 1 -- 172.21.15.50:0/4039346382 wait complete. 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:50.232+0000 7f4b9d9f1700 1 librados: shutdown 2024-04-26T16:29:50.230 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T16:29:50.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.055954+0000 osd.4 (osd.4) 257 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.055954+0000 osd.4 (osd.4) 257 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.056344+0000 osd.4 (osd.4) 258 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:50.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.056344+0000 osd.4 (osd.4) 258 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.215373+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.215373+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.219834+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.219834+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.230157+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.230157+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.251177+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.251177+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.251657+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.251657+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.550456+0000 mon.c (mon.2) 638 : audit [INF] from='client.? 172.21.15.50:0/4039346382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.550456+0000 mon.c (mon.2) 638 : audit [INF] from='client.? 172.21.15.50:0/4039346382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.550826+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: audit 2024-04-26T16:29:49.550826+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.691239+0000 mgr.y (mgr.24401) 485 : cluster [DBG] pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:29:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:50 smithi190 bash[23868]: cluster 2024-04-26T16:29:49.691239+0000 mgr.y (mgr.24401) 485 : cluster [DBG] pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:29:50.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:50.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:50.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.055954+0000 osd.4 (osd.4) 257 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:50.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.055954+0000 osd.4 (osd.4) 257 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:50.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.056344+0000 osd.4 (osd.4) 258 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:50.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.056344+0000 osd.4 (osd.4) 258 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:50.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.215373+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:50.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.215373+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.219834+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.219834+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.230157+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.230157+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.251177+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.251177+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.251657+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.251657+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.550456+0000 mon.c (mon.2) 638 : audit [INF] from='client.? 172.21.15.50:0/4039346382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.550456+0000 mon.c (mon.2) 638 : audit [INF] from='client.? 172.21.15.50:0/4039346382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.550826+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: audit 2024-04-26T16:29:49.550826+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.691239+0000 mgr.y (mgr.24401) 485 : cluster [DBG] pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:29:50.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[29234]: cluster 2024-04-26T16:29:49.691239+0000 mgr.y (mgr.24401) 485 : cluster [DBG] pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.055954+0000 osd.4 (osd.4) 257 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.055954+0000 osd.4 (osd.4) 257 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.056344+0000 osd.4 (osd.4) 258 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.056344+0000 osd.4 (osd.4) 258 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.215373+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.215373+0000 mon.a (mon.0) 2822 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.219834+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.219834+0000 mon.a (mon.0) 2823 : cluster [DBG] osdmap e491: 8 total, 8 up, 7 in 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.230157+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:50.452 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.230157+0000 mon.b (mon.1) 339 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.251177+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.251177+0000 mon.a (mon.0) 2824 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.251657+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.251657+0000 mon.b (mon.1) 340 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.550456+0000 mon.c (mon.2) 638 : audit [INF] from='client.? 172.21.15.50:0/4039346382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.550456+0000 mon.c (mon.2) 638 : audit [INF] from='client.? 172.21.15.50:0/4039346382' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.550826+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: audit 2024-04-26T16:29:49.550826+0000 mon.a (mon.0) 2825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.691239+0000 mgr.y (mgr.24401) 485 : cluster [DBG] pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:29:50.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:50 smithi050 bash[21463]: cluster 2024-04-26T16:29:49.691239+0000 mgr.y (mgr.24401) 485 : cluster [DBG] pgmap v736: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 525 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:29:50.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:50.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:50.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:50.592 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:50.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:50.723 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:50.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:50.855 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:50.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:50.986 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:50.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:51.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:51.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:51.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:51.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:51.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:51.379 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[29234]: audit 2024-04-26T16:29:50.224130+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[29234]: audit 2024-04-26T16:29:50.224130+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[29234]: audit 2024-04-26T16:29:50.224215+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:29:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[29234]: audit 2024-04-26T16:29:50.224215+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:29:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[29234]: cluster 2024-04-26T16:29:50.229914+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e492: 8 total, 8 up, 7 in 2024-04-26T16:29:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[29234]: cluster 2024-04-26T16:29:50.229914+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e492: 8 total, 8 up, 7 in 2024-04-26T16:29:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[21463]: audit 2024-04-26T16:29:50.224130+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[21463]: audit 2024-04-26T16:29:50.224130+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[21463]: audit 2024-04-26T16:29:50.224215+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:29:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[21463]: audit 2024-04-26T16:29:50.224215+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:29:51.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[21463]: cluster 2024-04-26T16:29:50.229914+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e492: 8 total, 8 up, 7 in 2024-04-26T16:29:51.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:51 smithi050 bash[21463]: cluster 2024-04-26T16:29:50.229914+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e492: 8 total, 8 up, 7 in 2024-04-26T16:29:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:51.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:51.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:51 smithi190 bash[23868]: audit 2024-04-26T16:29:50.224130+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:51.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:51 smithi190 bash[23868]: audit 2024-04-26T16:29:50.224130+0000 mon.a (mon.0) 2826 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:29:51.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:51 smithi190 bash[23868]: audit 2024-04-26T16:29:50.224215+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:29:51.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:51 smithi190 bash[23868]: audit 2024-04-26T16:29:50.224215+0000 mon.a (mon.0) 2827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:29:51.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:51 smithi190 bash[23868]: cluster 2024-04-26T16:29:50.229914+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e492: 8 total, 8 up, 7 in 2024-04-26T16:29:51.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:51 smithi190 bash[23868]: cluster 2024-04-26T16:29:50.229914+0000 mon.a (mon.0) 2828 : cluster [DBG] osdmap e492: 8 total, 8 up, 7 in 2024-04-26T16:29:51.510 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:51.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:51.642 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:51.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:51.772 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:51.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:51.903 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:51.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:52.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:52.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:52.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-26T16:29:52.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:52.167 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:52.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:52.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[29234]: cluster 2024-04-26T16:29:51.236019+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e493: 8 total, 8 up, 7 in 2024-04-26T16:29:52.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[29234]: cluster 2024-04-26T16:29:51.236019+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e493: 8 total, 8 up, 7 in 2024-04-26T16:29:52.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[29234]: cluster 2024-04-26T16:29:51.693192+0000 mgr.y (mgr.24401) 486 : cluster [DBG] pgmap v739: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-26T16:29:52.284 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[29234]: cluster 2024-04-26T16:29:51.693192+0000 mgr.y (mgr.24401) 486 : cluster [DBG] pgmap v739: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-26T16:29:52.284 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[21463]: cluster 2024-04-26T16:29:51.236019+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e493: 8 total, 8 up, 7 in 2024-04-26T16:29:52.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[21463]: cluster 2024-04-26T16:29:51.236019+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e493: 8 total, 8 up, 7 in 2024-04-26T16:29:52.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[21463]: cluster 2024-04-26T16:29:51.693192+0000 mgr.y (mgr.24401) 486 : cluster [DBG] pgmap v739: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-26T16:29:52.285 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:52 smithi050 bash[21463]: cluster 2024-04-26T16:29:51.693192+0000 mgr.y (mgr.24401) 486 : cluster [DBG] pgmap v739: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-26T16:29:52.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-26T16:29:52.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:52.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:52.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:52 smithi190 bash[23868]: cluster 2024-04-26T16:29:51.236019+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e493: 8 total, 8 up, 7 in 2024-04-26T16:29:52.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:52 smithi190 bash[23868]: cluster 2024-04-26T16:29:51.236019+0000 mon.a (mon.0) 2829 : cluster [DBG] osdmap e493: 8 total, 8 up, 7 in 2024-04-26T16:29:52.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:52 smithi190 bash[23868]: cluster 2024-04-26T16:29:51.693192+0000 mgr.y (mgr.24401) 486 : cluster [DBG] pgmap v739: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-26T16:29:52.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:52 smithi190 bash[23868]: cluster 2024-04-26T16:29:51.693192+0000 mgr.y (mgr.24401) 486 : cluster [DBG] pgmap v739: 228 pgs: 6 unknown, 222 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-04-26T16:29:52.423 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:52.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:52.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:52.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:52.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:52.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:52.805 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:52.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:52.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:52.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:53.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:53.063 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:53.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:53.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:53.196 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:53.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:53.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:53 smithi190 bash[23868]: cluster 2024-04-26T16:29:52.098437+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:53.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:53 smithi190 bash[23868]: cluster 2024-04-26T16:29:52.098437+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:53.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:53 smithi190 bash[23868]: cluster 2024-04-26T16:29:52.098685+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:53.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:53 smithi190 bash[23868]: cluster 2024-04-26T16:29:52.098685+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:53.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:53 smithi190 bash[23868]: cluster 2024-04-26T16:29:52.253524+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e494: 8 total, 8 up, 7 in 2024-04-26T16:29:53.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:53 smithi190 bash[23868]: cluster 2024-04-26T16:29:52.253524+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e494: 8 total, 8 up, 7 in 2024-04-26T16:29:53.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:53.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:53.455 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:53.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[21463]: cluster 2024-04-26T16:29:52.098437+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:53.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[21463]: cluster 2024-04-26T16:29:52.098437+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:53.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[21463]: cluster 2024-04-26T16:29:52.098685+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:53.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[21463]: cluster 2024-04-26T16:29:52.098685+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:53.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[21463]: cluster 2024-04-26T16:29:52.253524+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e494: 8 total, 8 up, 7 in 2024-04-26T16:29:53.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[21463]: cluster 2024-04-26T16:29:52.253524+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e494: 8 total, 8 up, 7 in 2024-04-26T16:29:53.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[29234]: cluster 2024-04-26T16:29:52.098437+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:53.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[29234]: cluster 2024-04-26T16:29:52.098437+0000 osd.1 (osd.1) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:29:53.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[29234]: cluster 2024-04-26T16:29:52.098685+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:53.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[29234]: cluster 2024-04-26T16:29:52.098685+0000 osd.1 (osd.1) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:29:53.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[29234]: cluster 2024-04-26T16:29:52.253524+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e494: 8 total, 8 up, 7 in 2024-04-26T16:29:53.548 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:53 smithi050 bash[29234]: cluster 2024-04-26T16:29:52.253524+0000 mon.a (mon.0) 2830 : cluster [DBG] osdmap e494: 8 total, 8 up, 7 in 2024-04-26T16:29:53.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:53.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:53.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:53.712 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 -- 172.21.15.50:0/1518307282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8108280 msgr2=0x7fcaf8108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 --2- 172.21.15.50:0/1518307282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8108280 0x7fcaf8108660 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009990 tx=0x7fcaf4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 -- 172.21.15.50:0/1518307282 shutdown_connections 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 --2- 172.21.15.50:0/1518307282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8107660 0x7fcaf810f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 --2- 172.21.15.50:0/1518307282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8108280 0x7fcaf8108660 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 --2- 172.21.15.50:0/1518307282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108c30 0x7fcaf8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 -- 172.21.15.50:0/1518307282 >> 172.21.15.50:0/1518307282 conn(0x7fcaf8078520 msgr2=0x7fcaf80ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.728+0000 7fcb007b3700 1 -- 172.21.15.50:0/1518307282 shutdown_connections 2024-04-26T16:29:53.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- 172.21.15.50:0/1518307282 wait complete. 2024-04-26T16:29:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 librados: starting msgr at 2024-04-26T16:29:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 librados: starting objecter 2024-04-26T16:29:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 Processor -- start 2024-04-26T16:29:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- start start 2024-04-26T16:29:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 librados: setting wanted keys 2024-04-26T16:29:53.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 librados: calling monclient init 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107660 0x7fcaf819b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108280 0x7fcaf819b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 0x7fcaf819fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf8111990 con 0x7fcaf8107660 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf8111810 con 0x7fcaf8108280 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf8111b10 con 0x7fcaf8108c30 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 0x7fcaf819fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafe54f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107660 0x7fcaf819b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafe54f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107660 0x7fcaf819b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38626/0 (socket says 172.21.15.50:38626) 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 0x7fcaf819fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37376/0 (socket says 172.21.15.50:37376) 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafe54f700 1 -- 172.21.15.50:0/2539207876 learned_addr learned my addr 172.21.15.50:0/2539207876 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafdd4e700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108280 0x7fcaf819b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 -- 172.21.15.50:0/2539207876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108280 msgr2=0x7fcaf819b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:53.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108280 0x7fcaf819b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 -- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107660 msgr2=0x7fcaf819b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107660 0x7fcaf819b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 -- 172.21.15.50:0/2539207876 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf81a02f0 con 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafdd4e700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108280 0x7fcaf819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcafed50700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 0x7fcaf819fbb0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fcaf000efb0 tx=0x7fcaf000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcaef7fe700 10 client.?.objecter ms_handle_connect 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcaef7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf000ca50 con 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fcaf000cbf0 con 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf81a05e0 con 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf00046b0 con 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 10 client.41285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf81a7e60 con 0x7fcaf8108c30 2024-04-26T16:29:53.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 librados: init done 2024-04-26T16:29:53.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf8101920 con 0x7fcaf8108c30 2024-04-26T16:29:53.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.732+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fcaf0012050 con 0x7fcaf8108c30 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcae4071f80 0x7fcae4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(494..494 src has 378..494) ==== 9377+0+0 (secure 0 0 0) 0x7fcaf00913e0 con 0x7fcaf8108c30 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 10 client.41285.objecter ms_dispatch 0x7fcaf8001550 osd_map(494..494 src has 378..494) 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 3 client.41285.objecter handle_osd_map got epochs [494,494] > 0 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 3 client.41285.objecter handle_osd_map decoding full epoch 494 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcafe54f700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcae4071f80 0x7fcae4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 20 client.41285.objecter dump_active .. 0 homeless 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcaef7fe700 10 client.41285.objecter ms_handle_connect 0x7fcae4071f80 2024-04-26T16:29:53.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.736+0000 7fcafe54f700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcae4071f80 0x7fcae4074440 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400b2e0 tx=0x7fcaf400b610 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:53.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.740+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fcaf0022060 con 0x7fcaf8108c30 2024-04-26T16:29:53.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:53.837 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:53.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:53.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:53.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:53.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:53.980+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fcaf819c640 con 0x7fcaf8108c30 2024-04-26T16:29:54.067 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:54.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:54.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:54.221 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:54.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:54.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.260+0000 7fcaef7fe700 1 -- 172.21.15.50:0/2539207876 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v495) ==== 91+0+0 (secure 0 0 0) 0x7fcaf0010040 con 0x7fcaf8108c30 2024-04-26T16:29:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.264+0000 7fcb007b3700 10 librados: watch_flush enter 2024-04-26T16:29:54.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.264+0000 7fcb007b3700 10 librados: watch_flush exit 2024-04-26T16:29:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.264+0000 7fcb007b3700 20 client.41285.objecter shutdown clearing up homeless session... 2024-04-26T16:29:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.264+0000 7fcb007b3700 10 client.41285.objecter successfully canceled tick 2024-04-26T16:29:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcae4071f80 msgr2=0x7fcae4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcae4071f80 0x7fcae4074440 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400b2e0 tx=0x7fcaf400b610 comp rx=0 tx=0).stop 2024-04-26T16:29:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 msgr2=0x7fcaf819fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:54.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 0x7fcaf819fbb0 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7fcaf000efb0 tx=0x7fcaf000c3a0 comp rx=0 tx=0).stop 2024-04-26T16:29:54.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 shutdown_connections 2024-04-26T16:29:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcae4071f80 0x7fcae4074440 unknown :-1 s=CLOSED pgs=3759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcaf8108c30 0x7fcaf819fbb0 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcaf8107660 0x7fcaf819b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 --2- 172.21.15.50:0/2539207876 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcaf8108280 0x7fcaf819b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:54.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 >> 172.21.15.50:0/2539207876 conn(0x7fcaf8078520 msgr2=0x7fcaf80ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 shutdown_connections 2024-04-26T16:29:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 -- 172.21.15.50:0/2539207876 wait complete. 2024-04-26T16:29:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:54.268+0000 7fcb007b3700 1 librados: shutdown 2024-04-26T16:29:54.269 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:29:54.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:54.351 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:54.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[29234]: cluster 2024-04-26T16:29:53.694178+0000 mgr.y (mgr.24401) 487 : cluster [DBG] pgmap v741: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-26T16:29:54.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[29234]: cluster 2024-04-26T16:29:53.694178+0000 mgr.y (mgr.24401) 487 : cluster [DBG] pgmap v741: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-26T16:29:54.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[29234]: audit 2024-04-26T16:29:53.984027+0000 mon.c (mon.2) 639 : audit [INF] from='client.? 172.21.15.50:0/2539207876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[29234]: audit 2024-04-26T16:29:53.984027+0000 mon.c (mon.2) 639 : audit [INF] from='client.? 172.21.15.50:0/2539207876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[29234]: audit 2024-04-26T16:29:53.984331+0000 mon.a (mon.0) 2831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[29234]: audit 2024-04-26T16:29:53.984331+0000 mon.a (mon.0) 2831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21463]: cluster 2024-04-26T16:29:53.694178+0000 mgr.y (mgr.24401) 487 : cluster [DBG] pgmap v741: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-26T16:29:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21463]: cluster 2024-04-26T16:29:53.694178+0000 mgr.y (mgr.24401) 487 : cluster [DBG] pgmap v741: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-26T16:29:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21463]: audit 2024-04-26T16:29:53.984027+0000 mon.c (mon.2) 639 : audit [INF] from='client.? 172.21.15.50:0/2539207876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21463]: audit 2024-04-26T16:29:53.984027+0000 mon.c (mon.2) 639 : audit [INF] from='client.? 172.21.15.50:0/2539207876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21463]: audit 2024-04-26T16:29:53.984331+0000 mon.a (mon.0) 2831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21463]: audit 2024-04-26T16:29:53.984331+0000 mon.a (mon.0) 2831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:54.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:54 smithi190 bash[23868]: cluster 2024-04-26T16:29:53.694178+0000 mgr.y (mgr.24401) 487 : cluster [DBG] pgmap v741: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-26T16:29:54.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:54 smithi190 bash[23868]: cluster 2024-04-26T16:29:53.694178+0000 mgr.y (mgr.24401) 487 : cluster [DBG] pgmap v741: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-04-26T16:29:54.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:54 smithi190 bash[23868]: audit 2024-04-26T16:29:53.984027+0000 mon.c (mon.2) 639 : audit [INF] from='client.? 172.21.15.50:0/2539207876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:54 smithi190 bash[23868]: audit 2024-04-26T16:29:53.984027+0000 mon.c (mon.2) 639 : audit [INF] from='client.? 172.21.15.50:0/2539207876' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:54 smithi190 bash[23868]: audit 2024-04-26T16:29:53.984331+0000 mon.a (mon.0) 2831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:54 smithi190 bash[23868]: audit 2024-04-26T16:29:53.984331+0000 mon.a (mon.0) 2831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:29:54.484 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:54.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:54.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:54.613 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:54.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:54.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:29:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:29:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:29:54.715 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:54.743 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:54.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:54.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:54.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:55.006 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:55.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:55.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:55.270 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:55.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[29234]: cluster 2024-04-26T16:29:54.251319+0000 mon.a (mon.0) 2832 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[29234]: cluster 2024-04-26T16:29:54.251319+0000 mon.a (mon.0) 2832 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[29234]: audit 2024-04-26T16:29:54.253607+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[29234]: audit 2024-04-26T16:29:54.253607+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[29234]: cluster 2024-04-26T16:29:54.261355+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e495: 8 total, 8 up, 7 in 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[29234]: cluster 2024-04-26T16:29:54.261355+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e495: 8 total, 8 up, 7 in 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[21463]: cluster 2024-04-26T16:29:54.251319+0000 mon.a (mon.0) 2832 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[21463]: cluster 2024-04-26T16:29:54.251319+0000 mon.a (mon.0) 2832 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[21463]: audit 2024-04-26T16:29:54.253607+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:29:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[21463]: audit 2024-04-26T16:29:54.253607+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:29:55.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[21463]: cluster 2024-04-26T16:29:54.261355+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e495: 8 total, 8 up, 7 in 2024-04-26T16:29:55.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:55 smithi050 bash[21463]: cluster 2024-04-26T16:29:54.261355+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e495: 8 total, 8 up, 7 in 2024-04-26T16:29:55.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:55.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:55.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:55 smithi190 bash[23868]: cluster 2024-04-26T16:29:54.251319+0000 mon.a (mon.0) 2832 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:55.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:55 smithi190 bash[23868]: cluster 2024-04-26T16:29:54.251319+0000 mon.a (mon.0) 2832 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:29:55.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:55 smithi190 bash[23868]: audit 2024-04-26T16:29:54.253607+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:29:55.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:55 smithi190 bash[23868]: audit 2024-04-26T16:29:54.253607+0000 mon.a (mon.0) 2833 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:29:55.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:55 smithi190 bash[23868]: cluster 2024-04-26T16:29:54.261355+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e495: 8 total, 8 up, 7 in 2024-04-26T16:29:55.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:55 smithi190 bash[23868]: cluster 2024-04-26T16:29:54.261355+0000 mon.a (mon.0) 2834 : cluster [DBG] osdmap e495: 8 total, 8 up, 7 in 2024-04-26T16:29:55.533 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:55.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:55.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:55.663 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:55.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:55.794 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:55.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:55.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:55.922 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:55.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:56.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:56.055 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:56.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:56.187 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:56.290 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:55.695005+0000 mgr.y (mgr.24401) 488 : cluster [DBG] pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:55.695005+0000 mgr.y (mgr.24401) 488 : cluster [DBG] pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:55.756658+0000 osd.0 (osd.0) 283 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:55.756658+0000 osd.0 (osd.0) 283 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:55.759059+0000 osd.0 (osd.0) 284 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:55.759059+0000 osd.0 (osd.0) 284 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.207156+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e496: 8 total, 8 up, 7 in 2024-04-26T16:29:56.301 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.207156+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e496: 8 total, 8 up, 7 in 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:55.695005+0000 mgr.y (mgr.24401) 488 : cluster [DBG] pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:55.695005+0000 mgr.y (mgr.24401) 488 : cluster [DBG] pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:55.756658+0000 osd.0 (osd.0) 283 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:55.756658+0000 osd.0 (osd.0) 283 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:55.759059+0000 osd.0 (osd.0) 284 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:55.759059+0000 osd.0 (osd.0) 284 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.207156+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e496: 8 total, 8 up, 7 in 2024-04-26T16:29:56.302 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:56 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.207156+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e496: 8 total, 8 up, 7 in 2024-04-26T16:29:56.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:56.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:56.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:56.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:55.695005+0000 mgr.y (mgr.24401) 488 : cluster [DBG] pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:55.695005+0000 mgr.y (mgr.24401) 488 : cluster [DBG] pgmap v743: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:55.756658+0000 osd.0 (osd.0) 283 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:55.756658+0000 osd.0 (osd.0) 283 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:55.759059+0000 osd.0 (osd.0) 284 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:55.759059+0000 osd.0 (osd.0) 284 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:29:56.557 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.207156+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e496: 8 total, 8 up, 7 in 2024-04-26T16:29:56.558 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.207156+0000 mon.a (mon.0) 2835 : cluster [DBG] osdmap e496: 8 total, 8 up, 7 in 2024-04-26T16:29:56.574 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:56.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:56.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:56.804 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:56.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:29:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:29:56.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:56.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:56.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:57.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:57.095 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:57.198 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:57.227 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:57.273 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:29:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:29:57.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:57.358 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:57.462 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:57.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:57 smithi050 bash[29234]: audit 2024-04-26T16:29:56.957212+0000 mon.c (mon.2) 640 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:57.474 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:57 smithi050 bash[29234]: audit 2024-04-26T16:29:56.957212+0000 mon.c (mon.2) 640 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:57.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:57 smithi050 bash[21463]: audit 2024-04-26T16:29:56.957212+0000 mon.c (mon.2) 640 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:57.474 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:57 smithi050 bash[21463]: audit 2024-04-26T16:29:56.957212+0000 mon.c (mon.2) 640 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:57.491 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:57 smithi190 bash[23868]: audit 2024-04-26T16:29:56.957212+0000 mon.c (mon.2) 640 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:57.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:57 smithi190 bash[23868]: audit 2024-04-26T16:29:56.957212+0000 mon.c (mon.2) 640 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:29:57.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:57.622 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:57.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:57.753 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:57.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:57.883 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:29:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:57.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:29:57.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:58.015 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:58.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:58.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:58.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:58.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:29:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:29:58.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:29:58.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:58.391 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.933097+0000 osd.3 (osd.3) 305 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.933097+0000 osd.3 (osd.3) 305 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.937557+0000 osd.3 (osd.3) 306 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.937557+0000 osd.3 (osd.3) 306 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: audit 2024-04-26T16:29:56.942204+0000 mgr.y (mgr.24401) 489 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: audit 2024-04-26T16:29:56.942204+0000 mgr.y (mgr.24401) 489 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.953165+0000 osd.1 (osd.1) 255 : cluster [DBG] 2.a scrub starts 2024-04-26T16:29:58.392 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.953165+0000 osd.1 (osd.1) 255 : cluster [DBG] 2.a scrub starts 2024-04-26T16:29:58.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.955650+0000 osd.1 (osd.1) 256 : cluster [DBG] 2.a scrub ok 2024-04-26T16:29:58.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:56.955650+0000 osd.1 (osd.1) 256 : cluster [DBG] 2.a scrub ok 2024-04-26T16:29:58.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.140031+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:29:58.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.140031+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:29:58.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.141869+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:29:58.393 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.141869+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:29:58.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.696673+0000 mgr.y (mgr.24401) 490 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s 2024-04-26T16:29:58.394 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:58 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.696673+0000 mgr.y (mgr.24401) 490 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s 2024-04-26T16:29:58.407 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:58.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:58.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:58.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.933097+0000 osd.3 (osd.3) 305 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:29:58.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.933097+0000 osd.3 (osd.3) 305 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:29:58.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.937557+0000 osd.3 (osd.3) 306 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:29:58.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.937557+0000 osd.3 (osd.3) 306 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:29:58.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: audit 2024-04-26T16:29:56.942204+0000 mgr.y (mgr.24401) 489 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:58.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: audit 2024-04-26T16:29:56.942204+0000 mgr.y (mgr.24401) 489 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.953165+0000 osd.1 (osd.1) 255 : cluster [DBG] 2.a scrub starts 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.953165+0000 osd.1 (osd.1) 255 : cluster [DBG] 2.a scrub starts 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.955650+0000 osd.1 (osd.1) 256 : cluster [DBG] 2.a scrub ok 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:56.955650+0000 osd.1 (osd.1) 256 : cluster [DBG] 2.a scrub ok 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.140031+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.140031+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.141869+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.141869+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.696673+0000 mgr.y (mgr.24401) 490 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s 2024-04-26T16:29:58.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.696673+0000 mgr.y (mgr.24401) 490 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s 2024-04-26T16:29:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:58.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.933097+0000 osd.3 (osd.3) 305 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:29:58.535 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.933097+0000 osd.3 (osd.3) 305 : cluster [DBG] 109.0 scrub starts 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.937557+0000 osd.3 (osd.3) 306 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.937557+0000 osd.3 (osd.3) 306 : cluster [DBG] 109.0 scrub ok 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: audit 2024-04-26T16:29:56.942204+0000 mgr.y (mgr.24401) 489 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: audit 2024-04-26T16:29:56.942204+0000 mgr.y (mgr.24401) 489 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.953165+0000 osd.1 (osd.1) 255 : cluster [DBG] 2.a scrub starts 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.953165+0000 osd.1 (osd.1) 255 : cluster [DBG] 2.a scrub starts 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.955650+0000 osd.1 (osd.1) 256 : cluster [DBG] 2.a scrub ok 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:56.955650+0000 osd.1 (osd.1) 256 : cluster [DBG] 2.a scrub ok 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.140031+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.140031+0000 osd.7 (osd.7) 283 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.141869+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.141869+0000 osd.7 (osd.7) 284 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:29:58.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.696673+0000 mgr.y (mgr.24401) 490 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s 2024-04-26T16:29:58.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:58 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.696673+0000 mgr.y (mgr.24401) 490 : cluster [DBG] pgmap v745: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 341 B/s wr, 1 op/s 2024-04-26T16:29:58.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:58.662 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:58.765 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:58.767 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:29:58.795 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:58.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:58.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:29:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:58.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:29:59.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:59.051 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:59.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:59.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:59.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:29:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:29:59.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.925817+0000 osd.1 (osd.1) 257 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:29:59.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.925817+0000 osd.1 (osd.1) 257 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:29:59.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.929063+0000 osd.1 (osd.1) 258 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:29:59.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[21463]: cluster 2024-04-26T16:29:57.929063+0000 osd.1 (osd.1) 258 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:29:59.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.925817+0000 osd.1 (osd.1) 257 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:29:59.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.925817+0000 osd.1 (osd.1) 257 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:29:59.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.929063+0000 osd.1 (osd.1) 258 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:29:59.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:29:59 smithi050 bash[29234]: cluster 2024-04-26T16:29:57.929063+0000 osd.1 (osd.1) 258 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:29:59.439 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:59.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 -- 172.21.15.50:0/4138830827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 msgr2=0x7f717c10f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 --2- 172.21.15.50:0/4138830827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 0x7f717c10f1d0 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7f717400b0a0 tx=0x7f717400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 -- 172.21.15.50:0/4138830827 shutdown_connections 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 --2- 172.21.15.50:0/4138830827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 0x7f717c10f1d0 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 --2- 172.21.15.50:0/4138830827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 0x7f717c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 --2- 172.21.15.50:0/4138830827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f717c102320 0x7f717c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.524+0000 7f7181e21700 1 -- 172.21.15.50:0/4138830827 >> 172.21.15.50:0/4138830827 conn(0x7f717c0792d0 msgr2=0x7f717c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- 172.21.15.50:0/4138830827 shutdown_connections 2024-04-26T16:29:59.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- 172.21.15.50:0/4138830827 wait complete. 2024-04-26T16:29:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 librados: starting msgr at 2024-04-26T16:29:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 librados: starting objecter 2024-04-26T16:29:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 Processor -- start 2024-04-26T16:29:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- start start 2024-04-26T16:29:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 librados: setting wanted keys 2024-04-26T16:29:59.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 librados: calling monclient init 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f717c102320 0x7f717c104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 0x7f717c104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 0x7f717c104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f717c1119d0 con 0x7f717c108390 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f717c111850 con 0x7f717c102320 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f717c111b50 con 0x7f717c102ce0 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717affd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 0x7f717c104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 0x7f717c104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717affd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 0x7f717c104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48092/0 (socket says 172.21.15.50:48092) 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717affd700 1 -- 172.21.15.50:0/599695255 learned_addr learned my addr 172.21.15.50:0/599695255 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f717c102320 0x7f717c104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 msgr2=0x7f717c104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f717c102ce0 0x7f717c104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f717c102320 msgr2=0x7f717c104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f717c102320 0x7f717c104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717c1a75b0 con 0x7f717c108390 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f717c102320 0x7f717c104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 0x7f717c104a80 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f717400b0a0 tx=0x7f7174004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7178ff9700 10 client.?.objecter ms_handle_connect 0x7f717c108390 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7178ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7174021070 con 0x7f717c108390 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f717c1a7840 con 0x7f717c108390 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7174007b80 con 0x7f717c108390 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 10 client.30675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717401ddf0 con 0x7f717c108390 2024-04-26T16:29:59.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f717c1a7d40 con 0x7f717c108390 2024-04-26T16:29:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 librados: init done 2024-04-26T16:29:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.528+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f717c103a40 con 0x7f717c108390 2024-04-26T16:29:59.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7174024070 con 0x7f717c108390 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7168071e50 0x7f7168074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(496..496 src has 378..496) ==== 9377+0+0 (secure 0 0 0) 0x7f717400f020 con 0x7f717c108390 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 osd_map(496..496 src has 378..496) 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 3 client.30675.objecter handle_osd_map got epochs [496,496] > 0 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 3 client.30675.objecter handle_osd_map decoding full epoch 496 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7168071e50 0x7f7168074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 20 client.30675.objecter dump_active .. 0 homeless 2024-04-26T16:29:59.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v496) ==== 54+0+16 (secure 0 0 0) 0x7f717402b020 con 0x7f717c108390 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 10 client.30675.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 1 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f717c064030 0x7f717c064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 1 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 0 1 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 4 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 10 client.30675.objecter _send_command 1 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f717c1054e0 con 0x7f717c064030 2024-04-26T16:29:59.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7168071e50 0x7f7168074310 secure :-1 s=READY pgs=3760 cs=0 l=1 rev1=1 crypto rx=0x7f7164009c90 tx=0x7f7164009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f7168071e50 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f717affd700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f717c064030 0x7f717c064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f717affd700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f717c064030 0x7f717c064410 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c064030 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c1054e0 con 0x7f717c064030 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 1: 0 ) 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 10 client.30675.objecter _finish_command 1 = system:0 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 4 2024-04-26T16:29:59.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.532+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 0 1 2024-04-26T16:29:59.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:59.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 10 client.30675.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 2 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 2 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c06a110 osd=0 3 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 0 2 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 4 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 10 client.30675.objecter _send_command 2 2024-04-26T16:29:59.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c063ab0 con 0x7f717c064030 2024-04-26T16:29:59.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c063ab0 con 0x7f717c064030 2024-04-26T16:29:59.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7178ff9700 10 client.30675.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 4 2024-04-26T16:29:59.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 0 2 2024-04-26T16:29:59.552 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 10 client.30675.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 3 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f717c10bcb0 0x7f717c1abf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 3 2024-04-26T16:29:59.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 1 3 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 4 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 10 client.30675.objecter _send_command 3 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f717c02e600 con 0x7f717c10bcb0 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.552+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f717c10bcb0 0x7f717c1abf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c10bcb0 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f717c10bcb0 0x7f717c1abf30 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c02e600 con 0x7f717c10bcb0 2024-04-26T16:29:59.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 3: 0 ) 2024-04-26T16:29:59.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f7178ff9700 10 client.30675.objecter _finish_command 3 = system:0 2024-04-26T16:29:59.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 4 2024-04-26T16:29:59.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.556+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 1 3 2024-04-26T16:29:59.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:29:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:29:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:59 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.925817+0000 osd.1 (osd.1) 257 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:29:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:59 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.925817+0000 osd.1 (osd.1) 257 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:29:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:59 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.929063+0000 osd.1 (osd.1) 258 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:29:59.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:29:59 smithi190 bash[23868]: cluster 2024-04-26T16:29:57.929063+0000 osd.1 (osd.1) 258 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:29:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 10 client.30675.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 4 2024-04-26T16:29:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 4 2024-04-26T16:29:59.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c02e490 osd=1 3 2024-04-26T16:29:59.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 1 4 2024-04-26T16:29:59.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 4 2024-04-26T16:29:59.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 10 client.30675.objecter _send_command 4 2024-04-26T16:29:59.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.576+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1ac470 con 0x7f717c10bcb0 2024-04-26T16:29:59.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1ac470 con 0x7f717c10bcb0 2024-04-26T16:29:59.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 10 client.30675.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 4 2024-04-26T16:29:59.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 1 4 2024-04-26T16:29:59.579 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:29:59.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 10 client.30675.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 5 2024-04-26T16:29:59.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f717c10b390 0x7f717c1af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 5 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 2 5 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 4 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 10 client.30675.objecter _send_command 5 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f717c02dff0 con 0x7f717c10b390 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f717affd700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f717c10b390 0x7f717c1af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c10b390 2024-04-26T16:29:59.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f717affd700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f717c10b390 0x7f717c1af7b0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c02dff0 con 0x7f717c10b390 2024-04-26T16:29:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 5: 0 ) 2024-04-26T16:29:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 10 client.30675.objecter _finish_command 5 = system:0 2024-04-26T16:29:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 4 2024-04-26T16:29:59.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.580+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 2 5 2024-04-26T16:29:59.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 10 client.30675.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 6 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 6 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c10b220 osd=2 3 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 2 6 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 4 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 10 client.30675.objecter _send_command 6 2024-04-26T16:29:59.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1afcf0 con 0x7f717c10b390 2024-04-26T16:29:59.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1afcf0 con 0x7f717c10b390 2024-04-26T16:29:59.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 10 client.30675.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 4 2024-04-26T16:29:59.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 2 6 2024-04-26T16:29:59.600 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:29:59.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 10 client.30675.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 7 2024-04-26T16:29:59.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f717c02d6d0 0x7f717c1b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 7 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 3 7 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 4 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 10 client.30675.objecter _send_command 7 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f717c1ad490 con 0x7f717c02d6d0 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f717c02d6d0 0x7f717c1b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c02d6d0 2024-04-26T16:29:59.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f717c02d6d0 0x7f717c1b2e80 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c1ad490 con 0x7f717c02d6d0 2024-04-26T16:29:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 7: 0 ) 2024-04-26T16:29:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 10 client.30675.objecter _finish_command 7 = system:0 2024-04-26T16:29:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 4 2024-04-26T16:29:59.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.600+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 3 7 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 10 client.30675.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 8 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 8 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c02d560 osd=3 3 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 3 8 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 4 2024-04-26T16:29:59.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 10 client.30675.objecter _send_command 8 2024-04-26T16:29:59.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1ad4f0 con 0x7f717c02d6d0 2024-04-26T16:29:59.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1ad4f0 con 0x7f717c02d6d0 2024-04-26T16:29:59.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7178ff9700 10 client.30675.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 4 2024-04-26T16:29:59.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.620+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 3 8 2024-04-26T16:29:59.621 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 10 client.30675.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 9 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f717c1b09d0 0x7f717c1b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 9 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 4 9 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 4 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 10 client.30675.objecter _send_command 9 2024-04-26T16:29:59.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f717c1b93b0 con 0x7f717c1b09d0 2024-04-26T16:29:59.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f717c1b09d0 0x7f717c1b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c1b09d0 2024-04-26T16:29:59.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f717c1b09d0 0x7f717c1b6550 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c1b93b0 con 0x7f717c1b09d0 2024-04-26T16:29:59.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 9: 0 ) 2024-04-26T16:29:59.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7178ff9700 10 client.30675.objecter _finish_command 9 = system:0 2024-04-26T16:29:59.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 4 2024-04-26T16:29:59.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.624+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 4 9 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 10 client.30675.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 10 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 10 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1ac950 osd=4 3 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 4 10 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 4 2024-04-26T16:29:59.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 10 client.30675.objecter _send_command 10 2024-04-26T16:29:59.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.644+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1b6a90 con 0x7f717c1b09d0 2024-04-26T16:29:59.646 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1b6a90 con 0x7f717c1b09d0 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7178ff9700 10 client.30675.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 4 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 4 10 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 10 client.30675.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 11 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f717c1b7110 0x7f717c1b9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 11 2024-04-26T16:29:59.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 5 11 2024-04-26T16:29:59.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 4 2024-04-26T16:29:59.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 10 client.30675.objecter _send_command 11 2024-04-26T16:29:59.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f717c1bc5e0 con 0x7f717c1b7110 2024-04-26T16:29:59.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f717affd700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f717c1b7110 0x7f717c1b9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f717affd700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f717c1b7110 0x7f717c1b9520 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.648+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c1b7110 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.652+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c1bc5e0 con 0x7f717c1b7110 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.652+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 11: 0 ) 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.652+0000 7f7178ff9700 10 client.30675.objecter _finish_command 11 = system:0 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.652+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 4 2024-04-26T16:29:59.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.652+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 5 11 2024-04-26T16:29:59.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:59.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 10 client.30675.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 12 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 12 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1b6a90 osd=5 3 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 5 12 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 4 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 10 client.30675.objecter _send_command 12 2024-04-26T16:29:59.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.672+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1bb370 con 0x7f717c1b7110 2024-04-26T16:29:59.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1bb370 con 0x7f717c1b7110 2024-04-26T16:29:59.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 10 client.30675.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 4 2024-04-26T16:29:59.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 5 12 2024-04-26T16:29:59.674 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 10 client.30675.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 13 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f717c1bba90 0x7f717c1bbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 13 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 6 13 2024-04-26T16:29:59.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 4 2024-04-26T16:29:59.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 10 client.30675.objecter _send_command 13 2024-04-26T16:29:59.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f717c1bf9f0 con 0x7f717c1bba90 2024-04-26T16:29:59.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f717c1bba90 0x7f717c1bbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c1bba90 2024-04-26T16:29:59.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f717bfff700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f717c1bba90 0x7f717c1bbed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c1bf9f0 con 0x7f717c1bba90 2024-04-26T16:29:59.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 13: 0 ) 2024-04-26T16:29:59.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 10 client.30675.objecter _finish_command 13 = system:0 2024-04-26T16:29:59.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 4 2024-04-26T16:29:59.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.676+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 6 13 2024-04-26T16:29:59.693 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 10 client.30675.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 14 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 14 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1ad550 osd=6 3 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 6 14 2024-04-26T16:29:59.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 4 2024-04-26T16:29:59.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 10 client.30675.objecter _send_command 14 2024-04-26T16:29:59.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.696+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1be830 con 0x7f717c1bba90 2024-04-26T16:29:59.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:29:59.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1be830 con 0x7f717c1bba90 2024-04-26T16:29:59.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 10 client.30675.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 4 2024-04-26T16:29:59.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 6 14 2024-04-26T16:29:59.698 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:29:59.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 10 client.30675.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:29:59.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 15 2024-04-26T16:29:59.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f717c1bef50 0x7f717c1bf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 15 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 7 15 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 4 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 10 client.30675.objecter _send_command 15 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f717c1c2d90 con 0x7f717c1bef50 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f717c1bef50 0x7f717c1bf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 10 client.30675.objecter ms_handle_connect 0x7f717c1bef50 2024-04-26T16:29:59.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f717b7fe700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f717c1bef50 0x7f717c1bf370 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:29:59.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f717c1c2d90 con 0x7f717c1bef50 2024-04-26T16:29:59.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 15: 0 ) 2024-04-26T16:29:59.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 10 client.30675.objecter _finish_command 15 = system:0 2024-04-26T16:29:59.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 4 2024-04-26T16:29:59.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.700+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 7 15 2024-04-26T16:29:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:29:59.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign -1 16 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter _get_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 15 client.30675.objecter _session_command_op_remove -1 16 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter get_session s=0x7f717c1be830 osd=7 3 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 15 client.30675.objecter _session_command_op_assign 7 16 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 4 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter _send_command 16 2024-04-26T16:29:59.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f717c1b9ba0 con 0x7f717c1bef50 2024-04-26T16:29:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7178ff9700 1 -- 172.21.15.50:0/599695255 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f717c1b9ba0 con 0x7f717c1bef50 2024-04-26T16:29:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7178ff9700 10 client.30675.objecter ms_dispatch 0x7f717c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:29:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7178ff9700 10 client.30675.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:29:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7178ff9700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 4 2024-04-26T16:29:59.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7178ff9700 15 client.30675.objecter _session_command_op_remove 7 16 2024-04-26T16:29:59.720 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 librados: watch_flush enter 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 librados: watch_flush exit 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.0 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f717c064030 msgr2=0x7f717c064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f717c064030 0x7f717c064410 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c06a110 osd=0 1 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.1 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f717c10bcb0 msgr2=0x7f717c1abf30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f717c10bcb0 0x7f717c1abf30 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02e490 osd=1 1 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.2 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f717c10b390 msgr2=0x7f717c1af7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f717c10b390 0x7f717c1af7b0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c10b220 osd=2 1 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.3 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f717c02d6d0 msgr2=0x7f717c1b2e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f717c02d6d0 0x7f717c1b2e80 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c02d560 osd=3 1 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.4 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f717c1b09d0 msgr2=0x7f717c1b6550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f717bfff700 1 -- 172.21.15.50:0/599695255 reap_dead start 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f717c1b09d0 0x7f717c1b6550 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f717bfff700 1 -- 172.21.15.50:0/599695255 reap_dead start 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ac950 osd=4 1 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.5 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f717c1b7110 msgr2=0x7f717c1b9520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f717c1b7110 0x7f717c1b9520 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1b6a90 osd=5 1 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.6 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f717c1bba90 msgr2=0x7f717c1bbed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f717c1bba90 0x7f717c1bbed0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1ad550 osd=6 1 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter close_session for osd.7 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f717c1bef50 msgr2=0x7f717c1bf370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f717c1bef50 0x7f717c1bf370 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter put_session s=0x7f717c1be830 osd=7 1 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 20 client.30675.objecter shutdown clearing up homeless session... 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 10 client.30675.objecter successfully canceled tick 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7168071e50 msgr2=0x7f7168074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7168071e50 0x7f7168074310 secure :-1 s=READY pgs=3760 cs=0 l=1 rev1=1 crypto rx=0x7f7164009c90 tx=0x7f7164009260 comp rx=0 tx=0).stop 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 msgr2=0x7f717c104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f717bfff700 1 -- 172.21.15.50:0/599695255 reap_dead start 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 0x7f717c104a80 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f717400b0a0 tx=0x7f7174004d00 comp rx=0 tx=0).stop 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 shutdown_connections 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 --2- 172.21.15.50:0/599695255 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f717c108390 0x7f717c104a80 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 >> 172.21.15.50:0/599695255 conn(0x7f717c0792d0 msgr2=0x7f717c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 shutdown_connections 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 -- 172.21.15.50:0/599695255 wait complete. 2024-04-26T16:29:59.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:29:59.720+0000 7f7181e21700 1 librados: shutdown 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:29:59.725 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:29:59.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.824 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:29:59.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:29:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:29:59.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:00.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:00.081 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:00.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:00.212 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:00.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:00.342 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:00.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.656866+0000 osd.4 (osd.4) 259 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:30:00.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.656866+0000 osd.4 (osd.4) 259 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:30:00.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.659634+0000 osd.4 (osd.4) 260 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:30:00.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.659634+0000 osd.4 (osd.4) 260 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.931205+0000 osd.3 (osd.3) 307 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.931205+0000 osd.3 (osd.3) 307 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.933156+0000 osd.3 (osd.3) 308 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:58.933156+0000 osd.3 (osd.3) 308 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: audit 2024-04-26T16:29:59.535367+0000 mon.a (mon.0) 2836 : audit [DBG] from='client.? 172.21.15.50:0/599695255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: audit 2024-04-26T16:29:59.535367+0000 mon.a (mon.0) 2836 : audit [DBG] from='client.? 172.21.15.50:0/599695255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:59.697540+0000 mgr.y (mgr.24401) 491 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 687 B/s rd, 275 B/s wr, 1 op/s 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:29:59.697540+0000 mgr.y (mgr.24401) 491 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 687 B/s rd, 275 B/s wr, 1 op/s 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000173+0000 mon.a (mon.0) 2837 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000173+0000 mon.a (mon.0) 2837 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000230+0000 mon.a (mon.0) 2838 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000230+0000 mon.a (mon.0) 2838 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000267+0000 mon.a (mon.0) 2839 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000267+0000 mon.a (mon.0) 2839 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.428 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000276+0000 mon.a (mon.0) 2840 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:30:00.429 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000276+0000 mon.a (mon.0) 2840 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:30:00.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:00.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.656866+0000 osd.4 (osd.4) 259 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.656866+0000 osd.4 (osd.4) 259 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.659634+0000 osd.4 (osd.4) 260 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.659634+0000 osd.4 (osd.4) 260 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.931205+0000 osd.3 (osd.3) 307 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.931205+0000 osd.3 (osd.3) 307 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.933156+0000 osd.3 (osd.3) 308 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:58.933156+0000 osd.3 (osd.3) 308 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: audit 2024-04-26T16:29:59.535367+0000 mon.a (mon.0) 2836 : audit [DBG] from='client.? 172.21.15.50:0/599695255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: audit 2024-04-26T16:29:59.535367+0000 mon.a (mon.0) 2836 : audit [DBG] from='client.? 172.21.15.50:0/599695255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:59.697540+0000 mgr.y (mgr.24401) 491 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 687 B/s rd, 275 B/s wr, 1 op/s 2024-04-26T16:30:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:29:59.697540+0000 mgr.y (mgr.24401) 491 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 687 B/s rd, 275 B/s wr, 1 op/s 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000173+0000 mon.a (mon.0) 2837 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000173+0000 mon.a (mon.0) 2837 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000230+0000 mon.a (mon.0) 2838 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000230+0000 mon.a (mon.0) 2838 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000267+0000 mon.a (mon.0) 2839 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000267+0000 mon.a (mon.0) 2839 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000276+0000 mon.a (mon.0) 2840 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000276+0000 mon.a (mon.0) 2840 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000303+0000 mon.a (mon.0) 2841 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:30:00.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:00 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.000303+0000 mon.a (mon.0) 2841 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:30:00.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:00.603 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:00.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:00.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000303+0000 mon.a (mon.0) 2841 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:30:00.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.000303+0000 mon.a (mon.0) 2841 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.656866+0000 osd.4 (osd.4) 259 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.656866+0000 osd.4 (osd.4) 259 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.659634+0000 osd.4 (osd.4) 260 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.659634+0000 osd.4 (osd.4) 260 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.931205+0000 osd.3 (osd.3) 307 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.931205+0000 osd.3 (osd.3) 307 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.933156+0000 osd.3 (osd.3) 308 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:58.933156+0000 osd.3 (osd.3) 308 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: audit 2024-04-26T16:29:59.535367+0000 mon.a (mon.0) 2836 : audit [DBG] from='client.? 172.21.15.50:0/599695255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: audit 2024-04-26T16:29:59.535367+0000 mon.a (mon.0) 2836 : audit [DBG] from='client.? 172.21.15.50:0/599695255' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:59.697540+0000 mgr.y (mgr.24401) 491 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 687 B/s rd, 275 B/s wr, 1 op/s 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:29:59.697540+0000 mgr.y (mgr.24401) 491 : cluster [DBG] pgmap v746: 228 pgs: 228 active+clean; 8.4 MiB data, 510 MiB used, 625 GiB / 626 GiB avail; 687 B/s rd, 275 B/s wr, 1 op/s 2024-04-26T16:30:00.719 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000173+0000 mon.a (mon.0) 2837 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000173+0000 mon.a (mon.0) 2837 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000230+0000 mon.a (mon.0) 2838 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000230+0000 mon.a (mon.0) 2838 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000267+0000 mon.a (mon.0) 2839 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000267+0000 mon.a (mon.0) 2839 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000276+0000 mon.a (mon.0) 2840 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000276+0000 mon.a (mon.0) 2840 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000303+0000 mon.a (mon.0) 2841 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:30:00.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:00 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.000303+0000 mon.a (mon.0) 2841 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:30:00.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:00.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:00.866 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:00.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:00.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:00.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:01.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:01.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:01.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:01.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:01.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:01.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:01.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:59.404890+0000 osd.2 (osd.2) 233 : cluster [DBG] 6.f scrub starts 2024-04-26T16:30:01.431 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:59.404890+0000 osd.2 (osd.2) 233 : cluster [DBG] 6.f scrub starts 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:59.407439+0000 osd.2 (osd.2) 234 : cluster [DBG] 6.f scrub ok 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:29:59.407439+0000 osd.2 (osd.2) 234 : cluster [DBG] 6.f scrub ok 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.491809+0000 osd.6 (osd.6) 27 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.491809+0000 osd.6 (osd.6) 27 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.494533+0000 osd.6 (osd.6) 28 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.494533+0000 osd.6 (osd.6) 28 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:59.404890+0000 osd.2 (osd.2) 233 : cluster [DBG] 6.f scrub starts 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:59.404890+0000 osd.2 (osd.2) 233 : cluster [DBG] 6.f scrub starts 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:59.407439+0000 osd.2 (osd.2) 234 : cluster [DBG] 6.f scrub ok 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:29:59.407439+0000 osd.2 (osd.2) 234 : cluster [DBG] 6.f scrub ok 2024-04-26T16:30:01.432 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.491809+0000 osd.6 (osd.6) 27 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:30:01.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.491809+0000 osd.6 (osd.6) 27 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:30:01.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.494533+0000 osd.6 (osd.6) 28 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:30:01.433 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.494533+0000 osd.6 (osd.6) 28 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:30:01.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:01.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:59.404890+0000 osd.2 (osd.2) 233 : cluster [DBG] 6.f scrub starts 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:59.404890+0000 osd.2 (osd.2) 233 : cluster [DBG] 6.f scrub starts 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:59.407439+0000 osd.2 (osd.2) 234 : cluster [DBG] 6.f scrub ok 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:29:59.407439+0000 osd.2 (osd.2) 234 : cluster [DBG] 6.f scrub ok 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.491809+0000 osd.6 (osd.6) 27 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.491809+0000 osd.6 (osd.6) 27 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.494533+0000 osd.6 (osd.6) 28 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:30:01.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.494533+0000 osd.6 (osd.6) 28 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:30:01.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:01.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:01.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:01.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:01.791 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:01.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:01.923 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:01.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:02.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:02.052 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:02.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:02.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:02.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:02.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:02.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:02.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:02.446 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:02.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.628728+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.628728+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.630564+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.630564+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.900349+0000 osd.3 (osd.3) 309 : cluster [DBG] 109.e scrub starts 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.900349+0000 osd.3 (osd.3) 309 : cluster [DBG] 109.e scrub starts 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.904187+0000 osd.3 (osd.3) 310 : cluster [DBG] 109.e scrub ok 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.904187+0000 osd.3 (osd.3) 310 : cluster [DBG] 109.e scrub ok 2024-04-26T16:30:02.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.929312+0000 osd.1 (osd.1) 259 : cluster [DBG] 109.b scrub starts 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.929312+0000 osd.1 (osd.1) 259 : cluster [DBG] 109.b scrub starts 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.932353+0000 osd.1 (osd.1) 260 : cluster [DBG] 109.b scrub ok 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:00.932353+0000 osd.1 (osd.1) 260 : cluster [DBG] 109.b scrub ok 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:01.186284+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.b scrub starts 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:01.186284+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.b scrub starts 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:01.188641+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.b scrub ok 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:01.188641+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.b scrub ok 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:01.699795+0000 mgr.y (mgr.24401) 492 : cluster [DBG] pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-26T16:30:02.562 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:02 smithi190 bash[23868]: cluster 2024-04-26T16:30:01.699795+0000 mgr.y (mgr.24401) 492 : cluster [DBG] pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-26T16:30:02.576 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:02.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:02.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.628728+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.628728+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.630564+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.630564+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.900349+0000 osd.3 (osd.3) 309 : cluster [DBG] 109.e scrub starts 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.900349+0000 osd.3 (osd.3) 309 : cluster [DBG] 109.e scrub starts 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.904187+0000 osd.3 (osd.3) 310 : cluster [DBG] 109.e scrub ok 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.904187+0000 osd.3 (osd.3) 310 : cluster [DBG] 109.e scrub ok 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.929312+0000 osd.1 (osd.1) 259 : cluster [DBG] 109.b scrub starts 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.929312+0000 osd.1 (osd.1) 259 : cluster [DBG] 109.b scrub starts 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.932353+0000 osd.1 (osd.1) 260 : cluster [DBG] 109.b scrub ok 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:00.932353+0000 osd.1 (osd.1) 260 : cluster [DBG] 109.b scrub ok 2024-04-26T16:30:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:01.186284+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.b scrub starts 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:01.186284+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.b scrub starts 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:01.188641+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.b scrub ok 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:01.188641+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.b scrub ok 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:01.699795+0000 mgr.y (mgr.24401) 492 : cluster [DBG] pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[21463]: cluster 2024-04-26T16:30:01.699795+0000 mgr.y (mgr.24401) 492 : cluster [DBG] pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.628728+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.628728+0000 osd.4 (osd.4) 261 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.630564+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.630564+0000 osd.4 (osd.4) 262 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.900349+0000 osd.3 (osd.3) 309 : cluster [DBG] 109.e scrub starts 2024-04-26T16:30:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.900349+0000 osd.3 (osd.3) 309 : cluster [DBG] 109.e scrub starts 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.904187+0000 osd.3 (osd.3) 310 : cluster [DBG] 109.e scrub ok 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.904187+0000 osd.3 (osd.3) 310 : cluster [DBG] 109.e scrub ok 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.929312+0000 osd.1 (osd.1) 259 : cluster [DBG] 109.b scrub starts 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.929312+0000 osd.1 (osd.1) 259 : cluster [DBG] 109.b scrub starts 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.932353+0000 osd.1 (osd.1) 260 : cluster [DBG] 109.b scrub ok 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:00.932353+0000 osd.1 (osd.1) 260 : cluster [DBG] 109.b scrub ok 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:01.186284+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.b scrub starts 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:01.186284+0000 osd.7 (osd.7) 285 : cluster [DBG] 2.b scrub starts 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:01.188641+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.b scrub ok 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:01.188641+0000 osd.7 (osd.7) 286 : cluster [DBG] 2.b scrub ok 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:01.699795+0000 mgr.y (mgr.24401) 492 : cluster [DBG] pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-26T16:30:02.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:02 smithi050 bash[29234]: cluster 2024-04-26T16:30:01.699795+0000 mgr.y (mgr.24401) 492 : cluster [DBG] pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 383 B/s wr, 3 op/s 2024-04-26T16:30:02.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:02.707 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:02.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:02.836 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:02.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:02.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:02.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:03.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:03.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:03.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.215 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:03.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:03.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.401195+0000 osd.2 (osd.2) 235 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.401195+0000 osd.2 (osd.2) 235 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.527336+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.527336+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.530363+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.530363+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.540162+0000 osd.6 (osd.6) 29 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.540162+0000 osd.6 (osd.6) 29 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:30:03.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.546126+0000 osd.6 (osd.6) 30 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.546126+0000 osd.6 (osd.6) 30 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.401195+0000 osd.2 (osd.2) 235 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.401195+0000 osd.2 (osd.2) 235 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.527336+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.527336+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.530363+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.530363+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.540162+0000 osd.6 (osd.6) 29 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.540162+0000 osd.6 (osd.6) 29 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.546126+0000 osd.6 (osd.6) 30 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:30:03.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:03 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.546126+0000 osd.6 (osd.6) 30 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:30:03.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:03.461 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.401195+0000 osd.2 (osd.2) 235 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:30:03.461 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.401195+0000 osd.2 (osd.2) 235 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:30:03.461 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.527336+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.527336+0000 osd.0 (osd.0) 285 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.530363+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.530363+0000 osd.0 (osd.0) 286 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.540162+0000 osd.6 (osd.6) 29 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.540162+0000 osd.6 (osd.6) 29 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.546126+0000 osd.6 (osd.6) 30 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:30:03.462 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:03 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.546126+0000 osd.6 (osd.6) 30 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:30:03.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:03.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:03.608 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:03.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.739 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:03.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.867 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:03.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:03.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:03.998 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:04.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:04.128 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:04.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:04.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:04.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:04.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:30:04.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:04.518 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:04.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:04.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:30:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.403721+0000 osd.2 (osd.2) 236 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.403721+0000 osd.2 (osd.2) 236 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.949550+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.949550+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.951623+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:02.951623+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.186611+0000 osd.7 (osd.7) 287 : cluster [DBG] 3.e scrub starts 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.186611+0000 osd.7 (osd.7) 287 : cluster [DBG] 3.e scrub starts 2024-04-26T16:30:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.190964+0000 osd.7 (osd.7) 288 : cluster [DBG] 3.e scrub ok 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.190964+0000 osd.7 (osd.7) 288 : cluster [DBG] 3.e scrub ok 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.517349+0000 osd.0 (osd.0) 287 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.517349+0000 osd.0 (osd.0) 287 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.519603+0000 osd.0 (osd.0) 288 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.519603+0000 osd.0 (osd.0) 288 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.700909+0000 mgr.y (mgr.24401) 493 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 325 B/s wr, 2 op/s 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.700909+0000 mgr.y (mgr.24401) 493 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 325 B/s wr, 2 op/s 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.403721+0000 osd.2 (osd.2) 236 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.403721+0000 osd.2 (osd.2) 236 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.949550+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.949550+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:30:04.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.951623+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:02.951623+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.186611+0000 osd.7 (osd.7) 287 : cluster [DBG] 3.e scrub starts 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.186611+0000 osd.7 (osd.7) 287 : cluster [DBG] 3.e scrub starts 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.190964+0000 osd.7 (osd.7) 288 : cluster [DBG] 3.e scrub ok 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.190964+0000 osd.7 (osd.7) 288 : cluster [DBG] 3.e scrub ok 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.517349+0000 osd.0 (osd.0) 287 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.517349+0000 osd.0 (osd.0) 287 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.519603+0000 osd.0 (osd.0) 288 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.519603+0000 osd.0 (osd.0) 288 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.700909+0000 mgr.y (mgr.24401) 493 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 325 B/s wr, 2 op/s 2024-04-26T16:30:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:04 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.700909+0000 mgr.y (mgr.24401) 493 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 325 B/s wr, 2 op/s 2024-04-26T16:30:04.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:04.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.403721+0000 osd.2 (osd.2) 236 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:30:04.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.403721+0000 osd.2 (osd.2) 236 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.949550+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.949550+0000 osd.1 (osd.1) 261 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.951623+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:02.951623+0000 osd.1 (osd.1) 262 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.186611+0000 osd.7 (osd.7) 287 : cluster [DBG] 3.e scrub starts 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.186611+0000 osd.7 (osd.7) 287 : cluster [DBG] 3.e scrub starts 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.190964+0000 osd.7 (osd.7) 288 : cluster [DBG] 3.e scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.190964+0000 osd.7 (osd.7) 288 : cluster [DBG] 3.e scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.517349+0000 osd.0 (osd.0) 287 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.517349+0000 osd.0 (osd.0) 287 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.519603+0000 osd.0 (osd.0) 288 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.519603+0000 osd.0 (osd.0) 288 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.700909+0000 mgr.y (mgr.24401) 493 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 325 B/s wr, 2 op/s 2024-04-26T16:30:04.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:04 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.700909+0000 mgr.y (mgr.24401) 493 : cluster [DBG] pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 325 B/s wr, 2 op/s 2024-04-26T16:30:04.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:04.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:04.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:05.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:05.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:05.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:05.162 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:05.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:05.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:05.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:05.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.451640+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.b scrub starts 2024-04-26T16:30:05.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.451640+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.b scrub starts 2024-04-26T16:30:05.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.454156+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.b scrub ok 2024-04-26T16:30:05.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:03.454156+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.b scrub ok 2024-04-26T16:30:05.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.579041+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-04-26T16:30:05.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.579041+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-04-26T16:30:05.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.581353+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-04-26T16:30:05.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:05 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.581353+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-04-26T16:30:05.423 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:05.470 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:05.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:05.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.451640+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.b scrub starts 2024-04-26T16:30:05.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.451640+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.b scrub starts 2024-04-26T16:30:05.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.454156+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.b scrub ok 2024-04-26T16:30:05.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:03.454156+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.b scrub ok 2024-04-26T16:30:05.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.579041+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-04-26T16:30:05.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.579041+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.581353+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.581353+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.451640+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.b scrub starts 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.451640+0000 osd.2 (osd.2) 237 : cluster [DBG] 5.b scrub starts 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.454156+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.b scrub ok 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:03.454156+0000 osd.2 (osd.2) 238 : cluster [DBG] 5.b scrub ok 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.579041+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.579041+0000 osd.4 (osd.4) 263 : cluster [DBG] 2.f scrub starts 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.581353+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-04-26T16:30:05.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:05 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.581353+0000 osd.4 (osd.4) 264 : cluster [DBG] 2.f scrub ok 2024-04-26T16:30:05.553 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:05.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:05.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:05.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:05.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:05.939 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:05.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:06.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:06.068 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:06.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:06.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:06.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:06.330 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:06.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:06.463 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:06.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.492581+0000 osd.2 (osd.2) 239 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.492581+0000 osd.2 (osd.2) 239 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.495223+0000 osd.2 (osd.2) 240 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:04.495223+0000 osd.2 (osd.2) 240 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:05.251346+0000 osd.7 (osd.7) 289 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:05.251346+0000 osd.7 (osd.7) 289 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:05.253802+0000 osd.7 (osd.7) 290 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:30:06.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:05.253802+0000 osd.7 (osd.7) 290 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:30:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:05.701488+0000 mgr.y (mgr.24401) 494 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[23868]: cluster 2024-04-26T16:30:05.701488+0000 mgr.y (mgr.24401) 494 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:06.594 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:06.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.492581+0000 osd.2 (osd.2) 239 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.492581+0000 osd.2 (osd.2) 239 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.495223+0000 osd.2 (osd.2) 240 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:04.495223+0000 osd.2 (osd.2) 240 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:05.251346+0000 osd.7 (osd.7) 289 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:05.251346+0000 osd.7 (osd.7) 289 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:05.253802+0000 osd.7 (osd.7) 290 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:05.253802+0000 osd.7 (osd.7) 290 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:05.701488+0000 mgr.y (mgr.24401) 494 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:06.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[29234]: cluster 2024-04-26T16:30:05.701488+0000 mgr.y (mgr.24401) 494 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.492581+0000 osd.2 (osd.2) 239 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.492581+0000 osd.2 (osd.2) 239 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.495223+0000 osd.2 (osd.2) 240 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:04.495223+0000 osd.2 (osd.2) 240 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:05.251346+0000 osd.7 (osd.7) 289 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:05.251346+0000 osd.7 (osd.7) 289 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:05.253802+0000 osd.7 (osd.7) 290 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:05.253802+0000 osd.7 (osd.7) 290 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:05.701488+0000 mgr.y (mgr.24401) 494 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:06 smithi050 bash[21463]: cluster 2024-04-26T16:30:05.701488+0000 mgr.y (mgr.24401) 494 : cluster [DBG] pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:06.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:06.723 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:06.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:06.850 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:06.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:06.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:06.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:06.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:07.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:07.097 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:30:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:30:07.113 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:07.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:07.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:07.244 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:07.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:07.374 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:07.478 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:07.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:07.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:07.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:07 smithi190 bash[23868]: cluster 2024-04-26T16:30:06.481450+0000 osd.2 (osd.2) 241 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:30:07.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:07 smithi190 bash[23868]: cluster 2024-04-26T16:30:06.481450+0000 osd.2 (osd.2) 241 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:30:07.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:07 smithi190 bash[23868]: cluster 2024-04-26T16:30:06.484018+0000 osd.2 (osd.2) 242 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:30:07.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:07 smithi190 bash[23868]: cluster 2024-04-26T16:30:06.484018+0000 osd.2 (osd.2) 242 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:30:07.638 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:07.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:07.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[29234]: cluster 2024-04-26T16:30:06.481450+0000 osd.2 (osd.2) 241 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[29234]: cluster 2024-04-26T16:30:06.481450+0000 osd.2 (osd.2) 241 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[29234]: cluster 2024-04-26T16:30:06.484018+0000 osd.2 (osd.2) 242 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[29234]: cluster 2024-04-26T16:30:06.484018+0000 osd.2 (osd.2) 242 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[21463]: cluster 2024-04-26T16:30:06.481450+0000 osd.2 (osd.2) 241 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[21463]: cluster 2024-04-26T16:30:06.481450+0000 osd.2 (osd.2) 241 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[21463]: cluster 2024-04-26T16:30:06.484018+0000 osd.2 (osd.2) 242 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:30:07.754 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:07 smithi050 bash[21463]: cluster 2024-04-26T16:30:06.484018+0000 osd.2 (osd.2) 242 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:30:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:07.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:07.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:07.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:08.006 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:08.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:08.128 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.149 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:08.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 -- 172.21.15.50:0/4242718286 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 msgr2=0x7ff400109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 --2- 172.21.15.50:0/4242718286 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff400109940 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009990 tx=0x7ff3fc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 -- 172.21.15.50:0/4242718286 shutdown_connections 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 --2- 172.21.15.50:0/4242718286 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 0x7ff400110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 --2- 172.21.15.50:0/4242718286 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff400109940 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 --2- 172.21.15.50:0/4242718286 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 0x7ff400103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 -- 172.21.15.50:0/4242718286 >> 172.21.15.50:0/4242718286 conn(0x7ff40007a5d0 msgr2=0x7ff40007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 -- 172.21.15.50:0/4242718286 shutdown_connections 2024-04-26T16:30:08.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.231+0000 7ff408170700 1 -- 172.21.15.50:0/4242718286 wait complete. 2024-04-26T16:30:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 librados: starting msgr at 2024-04-26T16:30:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 librados: starting objecter 2024-04-26T16:30:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 Processor -- start 2024-04-26T16:30:08.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- start start 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 librados: setting wanted keys 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 librados: calling monclient init 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 0x7ff400075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 0x7ff400075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff4000761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff400112b10 con 0x7ff4001034c0 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff400112990 con 0x7ff400103e80 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff400112c90 con 0x7ff400109560 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff40670d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff4000761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 0x7ff400075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:08.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff40670d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff4000761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49914/0 (socket says 172.21.15.50:49914) 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff40670d700 1 -- 172.21.15.50:0/304318410 learned_addr learned my addr 172.21.15.50:0/304318410 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff40570b700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 0x7ff400075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 -- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 msgr2=0x7ff4000761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff4000761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 -- 172.21.15.50:0/304318410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 msgr2=0x7ff400075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 0x7ff400075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 -- 172.21.15.50:0/304318410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4001a75c0 con 0x7ff4001034c0 2024-04-26T16:30:08.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff40570b700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 0x7ff400075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff405f0c700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 0x7ff400075720 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0059d0 tx=0x7ff3fc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff3f6ffd700 10 client.?.objecter ms_handle_connect 0x7ff4001034c0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff3f6ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc027070 con 0x7ff4001034c0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff3fc022470 con 0x7ff4001034c0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4001a7850 con 0x7ff4001034c0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc017650 con 0x7ff4001034c0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 10 client.30681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4001a7d00 con 0x7ff4001034c0 2024-04-26T16:30:08.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 librados: init done 2024-04-26T16:30:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff400104bd0 con 0x7ff4001034c0 2024-04-26T16:30:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.235+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff3fc02a070 con 0x7ff4001034c0 2024-04-26T16:30:08.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3ec071f80 0x7ff3ec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(496..496 src has 378..496) ==== 9377+0+0 (secure 0 0 0) 0x7ff3fc0929b0 con 0x7ff4001034c0 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 10 client.30681.objecter ms_dispatch 0x7ff400001550 osd_map(496..496 src has 378..496) 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 3 client.30681.objecter handle_osd_map got epochs [496,496] > 0 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 3 client.30681.objecter handle_osd_map decoding full epoch 496 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff40570b700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3ec071f80 0x7ff3ec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 20 client.30681.objecter dump_active .. 0 homeless 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff40570b700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3ec071f80 0x7ff3ec074440 secure :-1 s=READY pgs=3761 cs=0 l=1 rev1=1 crypto rx=0x7ff40007af60 tx=0x7ff3f0009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:08.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.239+0000 7ff3f6ffd700 10 client.30681.objecter ms_handle_connect 0x7ff3ec071f80 2024-04-26T16:30:08.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.243+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff3fc013070 con 0x7ff4001034c0 2024-04-26T16:30:08.252 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:08.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:08.394 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:08.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:08.465 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:30:08.465 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:30:08.465 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-26T16:30:08.465 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-26T16:30:08.466 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:30:08.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:08.503+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7ff400076c50 con 0x7ff4001034c0 2024-04-26T16:30:08.513 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:08.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:08.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:06.946760+0000 mgr.y (mgr.24401) 495 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:06.946760+0000 mgr.y (mgr.24401) 495 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.452670+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:30:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.452670+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.455235+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.455235+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.523728+0000 osd.6 (osd.6) 31 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.523728+0000 osd.6 (osd.6) 31 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.526305+0000 osd.6 (osd.6) 32 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.526305+0000 osd.6 (osd.6) 32 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.703733+0000 mgr.y (mgr.24401) 496 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 267 B/s wr, 2 op/s 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.703733+0000 mgr.y (mgr.24401) 496 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 267 B/s wr, 2 op/s 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:07.850730+0000 mon.c (mon.2) 641 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:07.850730+0000 mon.c (mon.2) 641 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:08.376779+0000 mon.a (mon.0) 2842 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:08.376779+0000 mon.a (mon.0) 2842 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:08.383942+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[21463]: audit 2024-04-26T16:30:08.383942+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:06.946760+0000 mgr.y (mgr.24401) 495 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:06.946760+0000 mgr.y (mgr.24401) 495 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.452670+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.452670+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.455235+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.455235+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.523728+0000 osd.6 (osd.6) 31 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:30:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.523728+0000 osd.6 (osd.6) 31 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.526305+0000 osd.6 (osd.6) 32 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.526305+0000 osd.6 (osd.6) 32 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.703733+0000 mgr.y (mgr.24401) 496 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 267 B/s wr, 2 op/s 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.703733+0000 mgr.y (mgr.24401) 496 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 267 B/s wr, 2 op/s 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:07.850730+0000 mon.c (mon.2) 641 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:07.850730+0000 mon.c (mon.2) 641 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:08.376779+0000 mon.a (mon.0) 2842 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:08.376779+0000 mon.a (mon.0) 2842 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:08.383942+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:08 smithi050 bash[29234]: audit 2024-04-26T16:30:08.383942+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.644 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:08.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:06.946760+0000 mgr.y (mgr.24401) 495 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:08.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:06.946760+0000 mgr.y (mgr.24401) 495 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:08.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.452670+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:30:08.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.452670+0000 osd.2 (osd.2) 243 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:30:08.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.455235+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:30:08.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.455235+0000 osd.2 (osd.2) 244 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.523728+0000 osd.6 (osd.6) 31 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.523728+0000 osd.6 (osd.6) 31 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.526305+0000 osd.6 (osd.6) 32 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.526305+0000 osd.6 (osd.6) 32 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.703733+0000 mgr.y (mgr.24401) 496 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 267 B/s wr, 2 op/s 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.703733+0000 mgr.y (mgr.24401) 496 : cluster [DBG] pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 267 B/s wr, 2 op/s 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:07.850730+0000 mon.c (mon.2) 641 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:07.850730+0000 mon.c (mon.2) 641 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:08.376779+0000 mon.a (mon.0) 2842 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:08.376779+0000 mon.a (mon.0) 2842 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:08.383942+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:08 smithi190 bash[23868]: audit 2024-04-26T16:30:08.383942+0000 mon.a (mon.0) 2843 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:08.749 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:08.880 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:08.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:08.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:09.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:09.041 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:09.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:09.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:09.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:09.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:09.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:09.434 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:09.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.471+0000 7ff3f6ffd700 1 -- 172.21.15.50:0/304318410 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v497) ==== 113+0+0 (secure 0 0 0) 0x7ff3fc05e3e0 con 0x7ff4001034c0 2024-04-26T16:30:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 10 librados: watch_flush enter 2024-04-26T16:30:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 10 librados: watch_flush exit 2024-04-26T16:30:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 20 client.30681.objecter shutdown clearing up homeless session... 2024-04-26T16:30:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 10 client.30681.objecter successfully canceled tick 2024-04-26T16:30:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3ec071f80 msgr2=0x7ff3ec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:09.479 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3ec071f80 0x7ff3ec074440 secure :-1 s=READY pgs=3761 cs=0 l=1 rev1=1 crypto rx=0x7ff40007af60 tx=0x7ff3f0009260 comp rx=0 tx=0).stop 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 msgr2=0x7ff400075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 0x7ff400075720 secure :-1 s=READY pgs=2478 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc0059d0 tx=0x7ff3fc0047b0 comp rx=0 tx=0).stop 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 shutdown_connections 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3ec071f80 0x7ff3ec074440 unknown :-1 s=CLOSED pgs=3761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff400109560 0x7ff4000761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4001034c0 0x7ff400075720 unknown :-1 s=CLOSED pgs=2478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 --2- 172.21.15.50:0/304318410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff400103e80 0x7ff400075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:09.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 >> 172.21.15.50:0/304318410 conn(0x7ff40007a5d0 msgr2=0x7ff400107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:09.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 shutdown_connections 2024-04-26T16:30:09.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 -- 172.21.15.50:0/304318410 wait complete. 2024-04-26T16:30:09.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:09.479+0000 7ff408170700 1 librados: shutdown 2024-04-26T16:30:09.484 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T16:30:09.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:09.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.966794+0000 osd.3 (osd.3) 311 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:30:09.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.966794+0000 osd.3 (osd.3) 311 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:30:09.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.968399+0000 osd.3 (osd.3) 312 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:30:09.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:07.968399+0000 osd.3 (osd.3) 312 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.289766+0000 osd.7 (osd.7) 291 : cluster [DBG] 11.f scrub starts 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.289766+0000 osd.7 (osd.7) 291 : cluster [DBG] 11.f scrub starts 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.292578+0000 osd.7 (osd.7) 292 : cluster [DBG] 11.f scrub ok 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.292578+0000 osd.7 (osd.7) 292 : cluster [DBG] 11.f scrub ok 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.451403+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.e scrub starts 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.451403+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.e scrub starts 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.452367+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.e scrub ok 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.452367+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.e scrub ok 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: audit 2024-04-26T16:30:08.509845+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:30:09.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:09 smithi190 bash[23868]: audit 2024-04-26T16:30:08.509845+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:30:09.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:09.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:09.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:09.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.966794+0000 osd.3 (osd.3) 311 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.966794+0000 osd.3 (osd.3) 311 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.968399+0000 osd.3 (osd.3) 312 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:07.968399+0000 osd.3 (osd.3) 312 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.289766+0000 osd.7 (osd.7) 291 : cluster [DBG] 11.f scrub starts 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.289766+0000 osd.7 (osd.7) 291 : cluster [DBG] 11.f scrub starts 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.292578+0000 osd.7 (osd.7) 292 : cluster [DBG] 11.f scrub ok 2024-04-26T16:30:09.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.292578+0000 osd.7 (osd.7) 292 : cluster [DBG] 11.f scrub ok 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.451403+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.e scrub starts 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.451403+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.e scrub starts 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.452367+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.e scrub ok 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.452367+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.e scrub ok 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: audit 2024-04-26T16:30:08.509845+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[29234]: audit 2024-04-26T16:30:08.509845+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.966794+0000 osd.3 (osd.3) 311 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:30:09.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.966794+0000 osd.3 (osd.3) 311 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.968399+0000 osd.3 (osd.3) 312 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:07.968399+0000 osd.3 (osd.3) 312 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.289766+0000 osd.7 (osd.7) 291 : cluster [DBG] 11.f scrub starts 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.289766+0000 osd.7 (osd.7) 291 : cluster [DBG] 11.f scrub starts 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.292578+0000 osd.7 (osd.7) 292 : cluster [DBG] 11.f scrub ok 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.292578+0000 osd.7 (osd.7) 292 : cluster [DBG] 11.f scrub ok 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.451403+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.e scrub starts 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.451403+0000 osd.2 (osd.2) 245 : cluster [DBG] 2.e scrub starts 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.452367+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.e scrub ok 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.452367+0000 osd.2 (osd.2) 246 : cluster [DBG] 2.e scrub ok 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: audit 2024-04-26T16:30:08.509845+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:30:09.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:09 smithi050 bash[21463]: audit 2024-04-26T16:30:08.509845+0000 mon.a (mon.0) 2844 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:30:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:09.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:09.956 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:09.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:10.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:10.087 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:10.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:10.216 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:10.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:10.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:10.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:10.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:10.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:10.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:10.607 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.944162+0000 osd.3 (osd.3) 313 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.944162+0000 osd.3 (osd.3) 313 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.947460+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:08.947460+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.472252+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.472252+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.475261+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:30:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.475261+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: audit 2024-04-26T16:30:09.478567+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: audit 2024-04-26T16:30:09.478567+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.484004+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e497: 8 total, 8 up, 7 in 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.484004+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e497: 8 total, 8 up, 7 in 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.620581+0000 osd.0 (osd.0) 289 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.620581+0000 osd.0 (osd.0) 289 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.622668+0000 osd.0 (osd.0) 290 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.622668+0000 osd.0 (osd.0) 290 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.704401+0000 mgr.y (mgr.24401) 497 : cluster [DBG] pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.704401+0000 mgr.y (mgr.24401) 497 : cluster [DBG] pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.944162+0000 osd.3 (osd.3) 313 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:30:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.944162+0000 osd.3 (osd.3) 313 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.947460+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:08.947460+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.472252+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.472252+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.475261+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.475261+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: audit 2024-04-26T16:30:09.478567+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: audit 2024-04-26T16:30:09.478567+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.484004+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e497: 8 total, 8 up, 7 in 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.484004+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e497: 8 total, 8 up, 7 in 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.620581+0000 osd.0 (osd.0) 289 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.620581+0000 osd.0 (osd.0) 289 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.622668+0000 osd.0 (osd.0) 290 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:30:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.622668+0000 osd.0 (osd.0) 290 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:30:10.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.704401+0000 mgr.y (mgr.24401) 497 : cluster [DBG] pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:10.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:10 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.704401+0000 mgr.y (mgr.24401) 497 : cluster [DBG] pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:10.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:10.737 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.944162+0000 osd.3 (osd.3) 313 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.944162+0000 osd.3 (osd.3) 313 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.947460+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:08.947460+0000 osd.3 (osd.3) 314 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.472252+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.472252+0000 osd.2 (osd.2) 247 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.475261+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.475261+0000 osd.2 (osd.2) 248 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:30:10.738 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: audit 2024-04-26T16:30:09.478567+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: audit 2024-04-26T16:30:09.478567+0000 mon.a (mon.0) 2845 : audit [INF] from='client.? 172.21.15.50:0/304318410' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.484004+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e497: 8 total, 8 up, 7 in 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.484004+0000 mon.a (mon.0) 2846 : cluster [DBG] osdmap e497: 8 total, 8 up, 7 in 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.620581+0000 osd.0 (osd.0) 289 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.620581+0000 osd.0 (osd.0) 289 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.622668+0000 osd.0 (osd.0) 290 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.622668+0000 osd.0 (osd.0) 290 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.704401+0000 mgr.y (mgr.24401) 497 : cluster [DBG] pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:10.739 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:10 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.704401+0000 mgr.y (mgr.24401) 497 : cluster [DBG] pgmap v752: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 306 B/s wr, 2 op/s 2024-04-26T16:30:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:10.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:10.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:10.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:10.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:11.001 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:11.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:11.132 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:11.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:11.262 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:11.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:11.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:11.387 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:11.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:11.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:11.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.968689+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.968689+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.970500+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: cluster 2024-04-26T16:30:09.970500+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.208300+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e498: 8 total, 8 up, 7 in 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.208300+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e498: 8 total, 8 up, 7 in 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.343231+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.343231+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.343970+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.343970+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.345220+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.345220+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.346302+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:11 smithi190 bash[23868]: audit 2024-04-26T16:30:11.346302+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:11.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:11.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.968689+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.968689+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.970500+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: cluster 2024-04-26T16:30:09.970500+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.208300+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e498: 8 total, 8 up, 7 in 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.208300+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e498: 8 total, 8 up, 7 in 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.343231+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.765 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.343231+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.343970+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.343970+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.345220+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.345220+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.346302+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[21463]: audit 2024-04-26T16:30:11.346302+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.968689+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.968689+0000 osd.3 (osd.3) 315 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:30:11.766 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.970500+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: cluster 2024-04-26T16:30:09.970500+0000 osd.3 (osd.3) 316 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.208300+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e498: 8 total, 8 up, 7 in 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.208300+0000 mon.a (mon.0) 2847 : cluster [DBG] osdmap e498: 8 total, 8 up, 7 in 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.343231+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.343231+0000 mon.a (mon.0) 2848 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.343970+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.343970+0000 mon.b (mon.1) 341 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.345220+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.345220+0000 mon.a (mon.0) 2849 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.346302+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:11 smithi050 bash[29234]: audit 2024-04-26T16:30:11.346302+0000 mon.b (mon.1) 342 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-71"}]: dispatch 2024-04-26T16:30:11.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:11.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:11.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:11.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:12.018 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:12.041 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:12.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:12.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:12.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:12.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:12.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:12.300 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:12.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:12.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:12.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:12.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:12.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:10.927149+0000 osd.1 (osd.1) 263 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:10.927149+0000 osd.1 (osd.1) 263 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:10.929906+0000 osd.1 (osd.1) 264 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:10.929906+0000 osd.1 (osd.1) 264 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.231713+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.231713+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.234627+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:30:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.234627+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.457851+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.a scrub starts 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.457851+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.a scrub starts 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.460641+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.460641+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.502594+0000 osd.2 (osd.2) 249 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.502594+0000 osd.2 (osd.2) 249 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.507554+0000 osd.2 (osd.2) 250 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.507554+0000 osd.2 (osd.2) 250 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.598336+0000 osd.0 (osd.0) 291 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.598336+0000 osd.0 (osd.0) 291 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.600049+0000 osd.0 (osd.0) 292 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.600049+0000 osd.0 (osd.0) 292 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.705550+0000 mgr.y (mgr.24401) 498 : cluster [DBG] pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:30:12.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:11.705550+0000 mgr.y (mgr.24401) 498 : cluster [DBG] pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: audit 2024-04-26T16:30:11.973878+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: audit 2024-04-26T16:30:11.973878+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.221742+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e499: 8 total, 8 up, 7 in 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.221742+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e499: 8 total, 8 up, 7 in 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:10.927149+0000 osd.1 (osd.1) 263 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:10.927149+0000 osd.1 (osd.1) 263 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:10.929906+0000 osd.1 (osd.1) 264 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:10.929906+0000 osd.1 (osd.1) 264 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.231713+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.231713+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:30:12.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.234627+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.234627+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.457851+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.a scrub starts 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.457851+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.a scrub starts 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.460641+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.460641+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.502594+0000 osd.2 (osd.2) 249 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.502594+0000 osd.2 (osd.2) 249 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.507554+0000 osd.2 (osd.2) 250 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.507554+0000 osd.2 (osd.2) 250 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.598336+0000 osd.0 (osd.0) 291 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.598336+0000 osd.0 (osd.0) 291 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.600049+0000 osd.0 (osd.0) 292 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.600049+0000 osd.0 (osd.0) 292 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:30:12.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.705550+0000 mgr.y (mgr.24401) 498 : cluster [DBG] pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:30:12.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:11.705550+0000 mgr.y (mgr.24401) 498 : cluster [DBG] pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:30:12.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: audit 2024-04-26T16:30:11.973878+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:12.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: audit 2024-04-26T16:30:11.973878+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:12.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.221742+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e499: 8 total, 8 up, 7 in 2024-04-26T16:30:12.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:12 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.221742+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e499: 8 total, 8 up, 7 in 2024-04-26T16:30:12.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:12.695 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:12.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:10.927149+0000 osd.1 (osd.1) 263 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:10.927149+0000 osd.1 (osd.1) 263 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:10.929906+0000 osd.1 (osd.1) 264 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:10.929906+0000 osd.1 (osd.1) 264 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.231713+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.231713+0000 osd.7 (osd.7) 293 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.234627+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.234627+0000 osd.7 (osd.7) 294 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.457851+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.a scrub starts 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.457851+0000 osd.6 (osd.6) 33 : cluster [DBG] 3.a scrub starts 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.460641+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub ok 2024-04-26T16:30:12.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.460641+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.a scrub ok 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.502594+0000 osd.2 (osd.2) 249 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.502594+0000 osd.2 (osd.2) 249 : cluster [DBG] 109.8 scrub starts 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.507554+0000 osd.2 (osd.2) 250 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.507554+0000 osd.2 (osd.2) 250 : cluster [DBG] 109.8 scrub ok 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.598336+0000 osd.0 (osd.0) 291 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.598336+0000 osd.0 (osd.0) 291 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.600049+0000 osd.0 (osd.0) 292 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.600049+0000 osd.0 (osd.0) 292 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.705550+0000 mgr.y (mgr.24401) 498 : cluster [DBG] pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:11.705550+0000 mgr.y (mgr.24401) 498 : cluster [DBG] pgmap v754: 228 pgs: 228 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: audit 2024-04-26T16:30:11.973878+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: audit 2024-04-26T16:30:11.973878+0000 mon.c (mon.2) 642 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.221742+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e499: 8 total, 8 up, 7 in 2024-04-26T16:30:12.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:12 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.221742+0000 mon.a (mon.0) 2850 : cluster [DBG] osdmap e499: 8 total, 8 up, 7 in 2024-04-26T16:30:12.824 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:12.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:12.953 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:12.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:13.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:13.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:13.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-26T16:30:13.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:13.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:13.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:13.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:13.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:13.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:13.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:13.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:13.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:13.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:13.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.024782+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:30:13.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.024782+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:30:13.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.027052+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:30:13.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.027052+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:30:13.728 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.500520+0000 osd.2 (osd.2) 251 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.500520+0000 osd.2 (osd.2) 251 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.508633+0000 osd.2 (osd.2) 252 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.508633+0000 osd.2 (osd.2) 252 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.645844+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.645844+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.648261+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.648261+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.974489+0000 osd.2 (osd.2) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.974489+0000 osd.2 (osd.2) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.974737+0000 osd.2 (osd.2) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.974737+0000 osd.2 (osd.2) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.224081+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e500: 8 total, 8 up, 7 in 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.224081+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e500: 8 total, 8 up, 7 in 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: audit 2024-04-26T16:30:13.239640+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.729 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: audit 2024-04-26T16:30:13.239640+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: audit 2024-04-26T16:30:13.240328+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[21463]: audit 2024-04-26T16:30:13.240328+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.024782+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.024782+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.027052+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.027052+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.500520+0000 osd.2 (osd.2) 251 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.500520+0000 osd.2 (osd.2) 251 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.508633+0000 osd.2 (osd.2) 252 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:30:13.730 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.508633+0000 osd.2 (osd.2) 252 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.645844+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.645844+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.648261+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.648261+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.974489+0000 osd.2 (osd.2) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.974489+0000 osd.2 (osd.2) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.974737+0000 osd.2 (osd.2) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:13.731 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.974737+0000 osd.2 (osd.2) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:13.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.224081+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e500: 8 total, 8 up, 7 in 2024-04-26T16:30:13.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.224081+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e500: 8 total, 8 up, 7 in 2024-04-26T16:30:13.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: audit 2024-04-26T16:30:13.239640+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: audit 2024-04-26T16:30:13.239640+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: audit 2024-04-26T16:30:13.240328+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.732 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:13 smithi050 bash[29234]: audit 2024-04-26T16:30:13.240328+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-26T16:30:13.744 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:13.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.024782+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.024782+0000 osd.3 (osd.3) 317 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.027052+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.027052+0000 osd.3 (osd.3) 318 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.500520+0000 osd.2 (osd.2) 251 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.500520+0000 osd.2 (osd.2) 251 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.508633+0000 osd.2 (osd.2) 252 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.508633+0000 osd.2 (osd.2) 252 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.645844+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.645844+0000 osd.0 (osd.0) 293 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:30:13.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.648261+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.648261+0000 osd.0 (osd.0) 294 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.974489+0000 osd.2 (osd.2) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.974489+0000 osd.2 (osd.2) 253 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.974737+0000 osd.2 (osd.2) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.974737+0000 osd.2 (osd.2) 254 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.224081+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e500: 8 total, 8 up, 7 in 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.224081+0000 mon.a (mon.0) 2851 : cluster [DBG] osdmap e500: 8 total, 8 up, 7 in 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: audit 2024-04-26T16:30:13.239640+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: audit 2024-04-26T16:30:13.239640+0000 mon.a (mon.0) 2852 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: audit 2024-04-26T16:30:13.240328+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:13 smithi190 bash[23868]: audit 2024-04-26T16:30:13.240328+0000 mon.b (mon.1) 343 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:13.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:13.869 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:13.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:13.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:13.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:14.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:14.114 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:14.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:14.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:14.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:14.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:14.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:14.507 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:30:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:30:14.515 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:14.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:14.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:14.697 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.951974+0000 osd.1 (osd.1) 265 : cluster [DBG] 11.c scrub starts 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.951974+0000 osd.1 (osd.1) 265 : cluster [DBG] 11.c scrub starts 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.954255+0000 osd.1 (osd.1) 266 : cluster [DBG] 11.c scrub ok 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:12.954255+0000 osd.1 (osd.1) 266 : cluster [DBG] 11.c scrub ok 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.184726+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.184726+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.187187+0000 osd.7 (osd.7) 296 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.187187+0000 osd.7 (osd.7) 296 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.507743+0000 osd.4 (osd.4) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.507743+0000 osd.4 (osd.4) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.509865+0000 osd.4 (osd.4) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.509865+0000 osd.4 (osd.4) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.706228+0000 mgr.y (mgr.24401) 499 : cluster [DBG] pgmap v757: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:14.698 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:13.706228+0000 mgr.y (mgr.24401) 499 : cluster [DBG] pgmap v757: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.223580+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.223580+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.235767+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e501: 8 total, 8 up, 7 in 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.235767+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e501: 8 total, 8 up, 7 in 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.238773+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.238773+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.430209+0000 mon.a (mon.0) 2855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.430209+0000 mon.a (mon.0) 2855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.447122+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[29234]: audit 2024-04-26T16:30:14.447122+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.699 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.951974+0000 osd.1 (osd.1) 265 : cluster [DBG] 11.c scrub starts 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.951974+0000 osd.1 (osd.1) 265 : cluster [DBG] 11.c scrub starts 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.954255+0000 osd.1 (osd.1) 266 : cluster [DBG] 11.c scrub ok 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:12.954255+0000 osd.1 (osd.1) 266 : cluster [DBG] 11.c scrub ok 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.184726+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.184726+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.187187+0000 osd.7 (osd.7) 296 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.187187+0000 osd.7 (osd.7) 296 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.507743+0000 osd.4 (osd.4) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.507743+0000 osd.4 (osd.4) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.509865+0000 osd.4 (osd.4) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.509865+0000 osd.4 (osd.4) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.706228+0000 mgr.y (mgr.24401) 499 : cluster [DBG] pgmap v757: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:14.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:13.706228+0000 mgr.y (mgr.24401) 499 : cluster [DBG] pgmap v757: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.223580+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.223580+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.235767+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e501: 8 total, 8 up, 7 in 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.235767+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e501: 8 total, 8 up, 7 in 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.238773+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.238773+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.430209+0000 mon.a (mon.0) 2855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.430209+0000 mon.a (mon.0) 2855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.447122+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:14 smithi050 bash[21463]: audit 2024-04-26T16:30:14.447122+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:14.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:14.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.951974+0000 osd.1 (osd.1) 265 : cluster [DBG] 11.c scrub starts 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.951974+0000 osd.1 (osd.1) 265 : cluster [DBG] 11.c scrub starts 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.954255+0000 osd.1 (osd.1) 266 : cluster [DBG] 11.c scrub ok 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:12.954255+0000 osd.1 (osd.1) 266 : cluster [DBG] 11.c scrub ok 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.184726+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.184726+0000 osd.7 (osd.7) 295 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.187187+0000 osd.7 (osd.7) 296 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.187187+0000 osd.7 (osd.7) 296 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.507743+0000 osd.4 (osd.4) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.507743+0000 osd.4 (osd.4) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.509865+0000 osd.4 (osd.4) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.509865+0000 osd.4 (osd.4) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:30:14.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.706228+0000 mgr.y (mgr.24401) 499 : cluster [DBG] pgmap v757: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:13.706228+0000 mgr.y (mgr.24401) 499 : cluster [DBG] pgmap v757: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.223580+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.223580+0000 mon.a (mon.0) 2853 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.235767+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e501: 8 total, 8 up, 7 in 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.235767+0000 mon.a (mon.0) 2854 : cluster [DBG] osdmap e501: 8 total, 8 up, 7 in 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.238773+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.238773+0000 mon.b (mon.1) 344 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.430209+0000 mon.a (mon.0) 2855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.430209+0000 mon.a (mon.0) 2855 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.447122+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:14 smithi190 bash[23868]: audit 2024-04-26T16:30:14.447122+0000 mon.a (mon.0) 2856 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:14.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:14.888 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:14.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:14.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:15.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:15.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:15.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:15.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:15.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/2254226504 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e48105640 msgr2=0x7f2e48105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/2254226504 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e48105640 0x7f2e48105ac0 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009990 tx=0x7f2e3c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/2254226504 shutdown_connections 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/2254226504 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e48106000 0x7f2e48112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/2254226504 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e4810b640 0x7f2e4810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/2254226504 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e48105640 0x7f2e48105ac0 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/2254226504 >> 172.21.15.50:0/2254226504 conn(0x7f2e48101130 msgr2=0x7f2e48103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/2254226504 shutdown_connections 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/2254226504 wait complete. 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 librados: starting msgr at 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 librados: starting objecter 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 Processor -- start 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- start start 2024-04-26T16:30:15.267 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 librados: setting wanted keys 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 librados: calling monclient init 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e48105640 0x7f2e4819b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48106000 0x7f2e4819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 0x7f2e4819fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48114d60 con 0x7f2e48106000 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48114be0 con 0x7f2e48105640 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f2e48114ee0 con 0x7f2e4810b640 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 0x7f2e4819fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 0x7f2e4819fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49922/0 (socket says 172.21.15.50:49922) 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 -- 172.21.15.50:0/643064284 learned_addr learned my addr 172.21.15.50:0/643064284 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e477fe700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48106000 0x7f2e4819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:15.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e47fff700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e48105640 0x7f2e4819b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.270 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:15.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 -- 172.21.15.50:0/643064284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e48105640 msgr2=0x7f2e4819b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e48105640 0x7f2e4819b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:15.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 -- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48106000 msgr2=0x7f2e4819b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48106000 0x7f2e4819b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 -- 172.21.15.50:0/643064284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e481a02e0 con 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e477fe700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48106000 0x7f2e4819b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4cae8700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 0x7f2e4819fba0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f2e40007ee0 tx=0x7f2e4000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e457fa700 10 client.?.objecter ms_handle_connect 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e457fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40004bb0 con 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e481a05d0 con 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2e40004d50 con 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40005720 con 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 10 client.41303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e481a7e50 con 0x7f2e4810b640 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.267+0000 7f2e4e54b700 1 librados: init done 2024-04-26T16:30:15.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e4806a110 con 0x7f2e4810b640 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f2e4001a020 con 0x7f2e4810b640 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2e30071f80 0x7f2e30074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(502..502 src has 378..502) ==== 9377+0+0 (secure 0 0 0) 0x7f2e400911c0 con 0x7f2e4810b640 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 10 client.41303.objecter ms_dispatch 0x7f2e48001550 osd_map(502..502 src has 378..502) 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 3 client.41303.objecter handle_osd_map got epochs [502,502] > 0 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 3 client.41303.objecter handle_osd_map decoding full epoch 502 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e47fff700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2e30071f80 0x7f2e30074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 20 client.41303.objecter dump_active .. 0 homeless 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e457fa700 10 client.41303.objecter ms_handle_connect 0x7f2e30071f80 2024-04-26T16:30:15.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.271+0000 7f2e47fff700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2e30071f80 0x7f2e30074440 secure :-1 s=READY pgs=3762 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009680 tx=0x7f2e38009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:15.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.275+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f2e40010070 con 0x7f2e4810b640 2024-04-26T16:30:15.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:15.399 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:15.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:15.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:15.527+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f2e4802e490 con 0x7f2e4810b640 2024-04-26T16:30:15.531 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:15.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:15.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:15.660 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:15.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:15.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.198768+0000 osd.7 (osd.7) 297 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:30:15.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.198768+0000 osd.7 (osd.7) 297 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.201489+0000 osd.7 (osd.7) 298 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.201489+0000 osd.7 (osd.7) 298 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.503291+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.503291+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.529248+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.e scrub starts 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.529248+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.e scrub starts 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.531201+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.e scrub ok 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.531201+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.e scrub ok 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.773428+0000 osd.0 (osd.0) 295 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.773428+0000 osd.0 (osd.0) 295 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.773604+0000 osd.0 (osd.0) 296 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.773604+0000 osd.0 (osd.0) 296 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.228503+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e502: 8 total, 8 up, 7 in 2024-04-26T16:30:15.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:15 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.228503+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e502: 8 total, 8 up, 7 in 2024-04-26T16:30:15.788 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:15.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:15.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.198768+0000 osd.7 (osd.7) 297 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:30:15.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.198768+0000 osd.7 (osd.7) 297 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.201489+0000 osd.7 (osd.7) 298 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.201489+0000 osd.7 (osd.7) 298 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.503291+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.503291+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.529248+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.e scrub starts 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.529248+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.e scrub starts 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.531201+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.e scrub ok 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.531201+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.e scrub ok 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.773428+0000 osd.0 (osd.0) 295 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.773428+0000 osd.0 (osd.0) 295 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.773604+0000 osd.0 (osd.0) 296 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.773604+0000 osd.0 (osd.0) 296 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.228503+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e502: 8 total, 8 up, 7 in 2024-04-26T16:30:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.228503+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e502: 8 total, 8 up, 7 in 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.198768+0000 osd.7 (osd.7) 297 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.198768+0000 osd.7 (osd.7) 297 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.201489+0000 osd.7 (osd.7) 298 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.201489+0000 osd.7 (osd.7) 298 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.503291+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.503291+0000 mon.a (mon.0) 2857 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.529248+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.e scrub starts 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.529248+0000 osd.4 (osd.4) 267 : cluster [DBG] 5.e scrub starts 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.531201+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.e scrub ok 2024-04-26T16:30:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.531201+0000 osd.4 (osd.4) 268 : cluster [DBG] 5.e scrub ok 2024-04-26T16:30:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.773428+0000 osd.0 (osd.0) 295 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.773428+0000 osd.0 (osd.0) 295 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.773604+0000 osd.0 (osd.0) 296 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.773604+0000 osd.0 (osd.0) 296 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.228503+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e502: 8 total, 8 up, 7 in 2024-04-26T16:30:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:15 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.228503+0000 mon.a (mon.0) 2858 : cluster [DBG] osdmap e502: 8 total, 8 up, 7 in 2024-04-26T16:30:15.891 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:15.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:15.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:16.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:16.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:16.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:16.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:16.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.227+0000 7f2e457fa700 1 -- 172.21.15.50:0/643064284 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v503) ==== 101+0+0 (secure 0 0 0) 0x7f2e4005cb70 con 0x7f2e4810b640 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 10 librados: watch_flush enter 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 10 librados: watch_flush exit 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 20 client.41303.objecter shutdown clearing up homeless session... 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 10 client.41303.objecter successfully canceled tick 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2e30071f80 msgr2=0x7f2e30074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2e30071f80 0x7f2e30074440 secure :-1 s=READY pgs=3762 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009680 tx=0x7f2e38009260 comp rx=0 tx=0).stop 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 msgr2=0x7f2e4819fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 0x7f2e4819fba0 secure :-1 s=READY pgs=2680 cs=0 l=1 rev1=1 crypto rx=0x7f2e40007ee0 tx=0x7f2e4000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:30:16.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 shutdown_connections 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2e30071f80 0x7f2e30074440 unknown :-1 s=CLOSED pgs=3762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2e48105640 0x7f2e4819b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2e4810b640 0x7f2e4819fba0 unknown :-1 s=CLOSED pgs=2680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 --2- 172.21.15.50:0/643064284 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2e48106000 0x7f2e4819b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 >> 172.21.15.50:0/643064284 conn(0x7f2e48101130 msgr2=0x7f2e481019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 shutdown_connections 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 -- 172.21.15.50:0/643064284 wait complete. 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:16.231+0000 7f2e4e54b700 1 librados: shutdown 2024-04-26T16:30:16.233 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:30:16.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:16.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:16.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:16.402 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:16.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:16.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:16.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:16.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.929080+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.929080+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.930569+0000 osd.3 (osd.3) 320 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:14.930569+0000 osd.3 (osd.3) 320 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.525637+0000 osd.2 (osd.2) 255 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.525637+0000 osd.2 (osd.2) 255 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.529098+0000 osd.2 (osd.2) 256 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.529098+0000 osd.2 (osd.2) 256 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: audit 2024-04-26T16:30:15.531829+0000 mon.c (mon.2) 643 : audit [INF] from='client.? 172.21.15.50:0/643064284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: audit 2024-04-26T16:30:15.531829+0000 mon.c (mon.2) 643 : audit [INF] from='client.? 172.21.15.50:0/643064284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: audit 2024-04-26T16:30:15.532265+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: audit 2024-04-26T16:30:15.532265+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.707033+0000 mgr.y (mgr.24401) 500 : cluster [DBG] pgmap v760: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.707033+0000 mgr.y (mgr.24401) 500 : cluster [DBG] pgmap v760: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.225452+0000 mon.a (mon.0) 2860 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.225452+0000 mon.a (mon.0) 2860 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: audit 2024-04-26T16:30:16.227134+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: audit 2024-04-26T16:30:16.227134+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.231162+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e503: 8 total, 8 up, 7 in 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.231162+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e503: 8 total, 8 up, 7 in 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.929080+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.929080+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:30:16.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.930569+0000 osd.3 (osd.3) 320 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:14.930569+0000 osd.3 (osd.3) 320 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.525637+0000 osd.2 (osd.2) 255 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.525637+0000 osd.2 (osd.2) 255 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.529098+0000 osd.2 (osd.2) 256 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.529098+0000 osd.2 (osd.2) 256 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: audit 2024-04-26T16:30:15.531829+0000 mon.c (mon.2) 643 : audit [INF] from='client.? 172.21.15.50:0/643064284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: audit 2024-04-26T16:30:15.531829+0000 mon.c (mon.2) 643 : audit [INF] from='client.? 172.21.15.50:0/643064284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: audit 2024-04-26T16:30:15.532265+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: audit 2024-04-26T16:30:15.532265+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.707033+0000 mgr.y (mgr.24401) 500 : cluster [DBG] pgmap v760: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.707033+0000 mgr.y (mgr.24401) 500 : cluster [DBG] pgmap v760: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.225452+0000 mon.a (mon.0) 2860 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:30:16.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.225452+0000 mon.a (mon.0) 2860 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:30:16.675 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: audit 2024-04-26T16:30:16.227134+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:16.675 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: audit 2024-04-26T16:30:16.227134+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:16.675 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.231162+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e503: 8 total, 8 up, 7 in 2024-04-26T16:30:16.675 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:16 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.231162+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e503: 8 total, 8 up, 7 in 2024-04-26T16:30:16.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:16.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:16.801 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.929080+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.929080+0000 osd.3 (osd.3) 319 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.930569+0000 osd.3 (osd.3) 320 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:14.930569+0000 osd.3 (osd.3) 320 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.525637+0000 osd.2 (osd.2) 255 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.525637+0000 osd.2 (osd.2) 255 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.529098+0000 osd.2 (osd.2) 256 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.529098+0000 osd.2 (osd.2) 256 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: audit 2024-04-26T16:30:15.531829+0000 mon.c (mon.2) 643 : audit [INF] from='client.? 172.21.15.50:0/643064284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: audit 2024-04-26T16:30:15.531829+0000 mon.c (mon.2) 643 : audit [INF] from='client.? 172.21.15.50:0/643064284' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: audit 2024-04-26T16:30:15.532265+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: audit 2024-04-26T16:30:15.532265+0000 mon.a (mon.0) 2859 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:16.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.707033+0000 mgr.y (mgr.24401) 500 : cluster [DBG] pgmap v760: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.707033+0000 mgr.y (mgr.24401) 500 : cluster [DBG] pgmap v760: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.225452+0000 mon.a (mon.0) 2860 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.225452+0000 mon.a (mon.0) 2860 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: audit 2024-04-26T16:30:16.227134+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: audit 2024-04-26T16:30:16.227134+0000 mon.a (mon.0) 2861 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.231162+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e503: 8 total, 8 up, 7 in 2024-04-26T16:30:16.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.231162+0000 mon.a (mon.0) 2862 : cluster [DBG] osdmap e503: 8 total, 8 up, 7 in 2024-04-26T16:30:16.818 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:16.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:16.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:16.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:17.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:17.066 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:30:16 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:30:17.083 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:17.186 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:17.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:17.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:17.345 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:17.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:17.473 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:17.577 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:17.604 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:17.707 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:17.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:17.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.946471+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.d scrub starts 2024-04-26T16:30:17.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.946471+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.d scrub starts 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.948181+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub ok 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:15.948181+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub ok 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.189301+0000 osd.7 (osd.7) 299 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.189301+0000 osd.7 (osd.7) 299 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.191685+0000 osd.7 (osd.7) 300 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.191685+0000 osd.7 (osd.7) 300 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.613722+0000 osd.0 (osd.0) 297 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.613722+0000 osd.0 (osd.0) 297 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.615348+0000 osd.0 (osd.0) 298 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:30:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:17 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.615348+0000 osd.0 (osd.0) 298 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:30:17.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:17.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:17.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.946471+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.d scrub starts 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.946471+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.d scrub starts 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.948181+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub ok 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:15.948181+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub ok 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.189301+0000 osd.7 (osd.7) 299 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.189301+0000 osd.7 (osd.7) 299 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.191685+0000 osd.7 (osd.7) 300 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.191685+0000 osd.7 (osd.7) 300 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.613722+0000 osd.0 (osd.0) 297 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:30:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.613722+0000 osd.0 (osd.0) 297 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.615348+0000 osd.0 (osd.0) 298 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.615348+0000 osd.0 (osd.0) 298 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.946471+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.d scrub starts 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.946471+0000 osd.1 (osd.1) 267 : cluster [DBG] 2.d scrub starts 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.948181+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub ok 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:15.948181+0000 osd.1 (osd.1) 268 : cluster [DBG] 2.d scrub ok 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.189301+0000 osd.7 (osd.7) 299 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.189301+0000 osd.7 (osd.7) 299 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.191685+0000 osd.7 (osd.7) 300 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.191685+0000 osd.7 (osd.7) 300 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.613722+0000 osd.0 (osd.0) 297 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:30:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.613722+0000 osd.0 (osd.0) 297 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:30:17.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.615348+0000 osd.0 (osd.0) 298 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:30:17.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:17 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.615348+0000 osd.0 (osd.0) 298 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:30:17.967 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:17.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:17.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:18.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:18.124 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:18.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:18.254 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:18.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:18.384 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:18.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:18.515 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:18.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: audit 2024-04-26T16:30:16.949319+0000 mgr.y (mgr.24401) 501 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: audit 2024-04-26T16:30:16.949319+0000 mgr.y (mgr.24401) 501 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.956369+0000 osd.1 (osd.1) 269 : cluster [DBG] 109.c scrub starts 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.956369+0000 osd.1 (osd.1) 269 : cluster [DBG] 109.c scrub starts 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.958727+0000 osd.1 (osd.1) 270 : cluster [DBG] 109.c scrub ok 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: cluster 2024-04-26T16:30:16.958727+0000 osd.1 (osd.1) 270 : cluster [DBG] 109.c scrub ok 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: cluster 2024-04-26T16:30:17.709269+0000 mgr.y (mgr.24401) 502 : cluster [DBG] pgmap v762: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-26T16:30:18.630 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:18 smithi190 bash[23868]: cluster 2024-04-26T16:30:17.709269+0000 mgr.y (mgr.24401) 502 : cluster [DBG] pgmap v762: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-26T16:30:18.646 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:18.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: audit 2024-04-26T16:30:16.949319+0000 mgr.y (mgr.24401) 501 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:18.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: audit 2024-04-26T16:30:16.949319+0000 mgr.y (mgr.24401) 501 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:18.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.956369+0000 osd.1 (osd.1) 269 : cluster [DBG] 109.c scrub starts 2024-04-26T16:30:18.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.956369+0000 osd.1 (osd.1) 269 : cluster [DBG] 109.c scrub starts 2024-04-26T16:30:18.721 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.958727+0000 osd.1 (osd.1) 270 : cluster [DBG] 109.c scrub ok 2024-04-26T16:30:18.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: cluster 2024-04-26T16:30:16.958727+0000 osd.1 (osd.1) 270 : cluster [DBG] 109.c scrub ok 2024-04-26T16:30:18.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: cluster 2024-04-26T16:30:17.709269+0000 mgr.y (mgr.24401) 502 : cluster [DBG] pgmap v762: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-26T16:30:18.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[21463]: cluster 2024-04-26T16:30:17.709269+0000 mgr.y (mgr.24401) 502 : cluster [DBG] pgmap v762: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-26T16:30:18.722 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: audit 2024-04-26T16:30:16.949319+0000 mgr.y (mgr.24401) 501 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:18.722 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: audit 2024-04-26T16:30:16.949319+0000 mgr.y (mgr.24401) 501 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:18.722 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.956369+0000 osd.1 (osd.1) 269 : cluster [DBG] 109.c scrub starts 2024-04-26T16:30:18.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.956369+0000 osd.1 (osd.1) 269 : cluster [DBG] 109.c scrub starts 2024-04-26T16:30:18.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.958727+0000 osd.1 (osd.1) 270 : cluster [DBG] 109.c scrub ok 2024-04-26T16:30:18.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: cluster 2024-04-26T16:30:16.958727+0000 osd.1 (osd.1) 270 : cluster [DBG] 109.c scrub ok 2024-04-26T16:30:18.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: cluster 2024-04-26T16:30:17.709269+0000 mgr.y (mgr.24401) 502 : cluster [DBG] pgmap v762: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-26T16:30:18.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:18 smithi050 bash[29234]: cluster 2024-04-26T16:30:17.709269+0000 mgr.y (mgr.24401) 502 : cluster [DBG] pgmap v762: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 509 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-26T16:30:18.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:18.774 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:18.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/2103758545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806b130 msgr2=0x7f06d806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:18.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- 172.21.15.50:0/2103758545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806b130 0x7f06d806b590 secure :-1 s=READY pgs=2479 cs=0 l=1 rev1=1 crypto rx=0x7f06c0009990 tx=0x7f06c0009ca0 comp rx=0 tx=0).stop 2024-04-26T16:30:18.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/2103758545 shutdown_connections 2024-04-26T16:30:18.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- 172.21.15.50:0/2103758545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06d81094a0 0x7f06d8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:18.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- 172.21.15.50:0/2103758545 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806bad0 0x7f06d81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:18.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- 172.21.15.50:0/2103758545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806b130 0x7f06d806b590 unknown :-1 s=CLOSED pgs=2479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/2103758545 >> 172.21.15.50:0/2103758545 conn(0x7f06d80792d0 msgr2=0x7f06d80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/2103758545 shutdown_connections 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/2103758545 wait complete. 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 librados: starting msgr at 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 librados: starting objecter 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 Processor -- start 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- start start 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 librados: setting wanted keys 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 librados: calling monclient init 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806b130 0x7f06d819b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806bad0 0x7f06d819b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06d81094a0 0x7f06d819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f06d8112b30 con 0x7f06d806bad0 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f06d81129b0 con 0x7f06d81094a0 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f06d8112cb0 con 0x7f06d806b130 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806b130 0x7f06d819b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806bad0 0x7f06d819b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806b130 0x7f06d819b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52862/0 (socket says 172.21.15.50:52862) 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 -- 172.21.15.50:0/3563160890 learned_addr learned my addr 172.21.15.50:0/3563160890 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06d81094a0 0x7f06d819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06d81094a0 msgr2=0x7f06d819fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f06d81094a0 0x7f06d819fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806bad0 msgr2=0x7f06d819b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806bad0 0x7f06d819b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:18.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d81a0480 con 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f06d806bad0 0x7f06d819b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806b130 0x7f06d819b470 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f06c800d950 tx=0x7f06c800dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d57fa700 10 client.?.objecter ms_handle_connect 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d57fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8016070 con 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f06c8010460 con 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d81a0770 con 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 10 client.41309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c800f620 con 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d81a7ff0 con 0x7f06d806b130 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 librados: init done 2024-04-26T16:30:18.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.855+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f06d806a110 con 0x7f06d806b130 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f06c8019070 con 0x7f06d806b130 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f06c4071f80 0x7f06c4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(503..503 src has 378..503) ==== 9377+0+0 (secure 0 0 0) 0x7f06c80919c0 con 0x7f06d806b130 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 osd_map(503..503 src has 378..503) 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 3 client.41309.objecter handle_osd_map got epochs [503,503] > 0 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 3 client.41309.objecter handle_osd_map decoding full epoch 503 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f06c4071f80 0x7f06c4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 20 client.41309.objecter dump_active .. 0 homeless 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v503) ==== 54+0+16 (secure 0 0 0) 0x7f06c8092080 con 0x7f06d806b130 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 10 client.41309.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 1 2024-04-26T16:30:18.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f06d8064030 0x7f06d8064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 1 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 0 1 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 4 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 10 client.41309.objecter _send_command 1 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06d81ab350 con 0x7f06d8064030 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f06c4071f80 0x7f06c4074440 secure :-1 s=READY pgs=3763 cs=0 l=1 rev1=1 crypto rx=0x7f06c000b730 tx=0x7f06c001a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06c4071f80 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f06d8064030 0x7f06d8064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d8064030 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f06d8064030 0x7f06d8064410 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81ab350 con 0x7f06d8064030 2024-04-26T16:30:18.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 1: 0 ) 2024-04-26T16:30:18.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 10 client.41309.objecter _finish_command 1 = system:0 2024-04-26T16:30:18.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 4 2024-04-26T16:30:18.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.859+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 0 1 2024-04-26T16:30:18.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:18.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 10 client.41309.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:18.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 2 2024-04-26T16:30:18.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 2 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d819c7d0 osd=0 3 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 0 2 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 4 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 10 client.41309.objecter _send_command 2 2024-04-26T16:30:18.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.879+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d8063ab0 con 0x7f06d8064030 2024-04-26T16:30:18.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d8063ab0 con 0x7f06d8064030 2024-04-26T16:30:18.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:18.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 10 client.41309.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:18.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 4 2024-04-26T16:30:18.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 0 2 2024-04-26T16:30:18.882 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:30:18.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 10 client.41309.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:18.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 3 2024-04-26T16:30:18.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f06d802d310 0x7f06d802d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 3 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 1 3 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 4 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 10 client.41309.objecter _send_command 3 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f06d8100fe0 con 0x7f06d802d310 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f06d802d310 0x7f06d802d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d802d310 2024-04-26T16:30:18.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f06d802d310 0x7f06d802d6f0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d8100fe0 con 0x7f06d802d310 2024-04-26T16:30:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 3: 0 ) 2024-04-26T16:30:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 10 client.41309.objecter _finish_command 3 = system:0 2024-04-26T16:30:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 4 2024-04-26T16:30:18.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.883+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 1 3 2024-04-26T16:30:18.898 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:18.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:18.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:18.912 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 10 client.41309.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 4 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 4 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d8100960 osd=1 3 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 1 4 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 4 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 10 client.41309.objecter _send_command 4 2024-04-26T16:30:18.913 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.911+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d81003e0 con 0x7f06d802d310 2024-04-26T16:30:18.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d81003e0 con 0x7f06d802d310 2024-04-26T16:30:18.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:18.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 10 client.41309.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:18.914 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 4 2024-04-26T16:30:18.915 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 1 4 2024-04-26T16:30:18.915 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 10 client.41309.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 5 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f06d81ad510 0x7f06d81af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 5 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 2 5 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 4 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 10 client.41309.objecter _send_command 5 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f06d81b2760 con 0x7f06d81ad510 2024-04-26T16:30:18.916 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f06d81ad510 0x7f06d81af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d81ad510 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f06d81ad510 0x7f06d81af900 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81b2760 con 0x7f06d81ad510 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 5: 0 ) 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 10 client.41309.objecter _finish_command 5 = system:0 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 4 2024-04-26T16:30:18.917 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.915+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 2 5 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 10 client.41309.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 6 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 6 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d8101190 osd=2 3 2024-04-26T16:30:18.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 2 6 2024-04-26T16:30:18.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 4 2024-04-26T16:30:18.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 10 client.41309.objecter _send_command 6 2024-04-26T16:30:18.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d802de80 con 0x7f06d81ad510 2024-04-26T16:30:18.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d802de80 con 0x7f06d81ad510 2024-04-26T16:30:18.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:18.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06d57fa700 10 client.41309.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:18.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 4 2024-04-26T16:30:18.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 2 6 2024-04-26T16:30:18.936 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 10 client.41309.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 7 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f06d81b0fc0 0x7f06d81b1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 7 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 3 7 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 4 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 10 client.41309.objecter _send_command 7 2024-04-26T16:30:18.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f06d81ab020 con 0x7f06d81b0fc0 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f06d81b0fc0 0x7f06d81b1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d81b0fc0 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.935+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f06d81b0fc0 0x7f06d81b1420 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.939+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81ab020 con 0x7f06d81b0fc0 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.939+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 7: 0 ) 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.939+0000 7f06d57fa700 10 client.41309.objecter _finish_command 7 = system:0 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.939+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 4 2024-04-26T16:30:18.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.939+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 3 7 2024-04-26T16:30:18.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 10 client.41309.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:18.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 8 2024-04-26T16:30:18.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 8 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81aa9a0 osd=3 3 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 3 8 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 4 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 10 client.41309.objecter _send_command 8 2024-04-26T16:30:18.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.955+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d81ab020 con 0x7f06d81b0fc0 2024-04-26T16:30:18.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d81ab020 con 0x7f06d81b0fc0 2024-04-26T16:30:18.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:18.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d57fa700 10 client.41309.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:18.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 4 2024-04-26T16:30:18.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 3 8 2024-04-26T16:30:18.958 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 10 client.41309.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 9 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f06d81b2130 0x7f06d81b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 9 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 4 9 2024-04-26T16:30:18.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 4 2024-04-26T16:30:18.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 10 client.41309.objecter _send_command 9 2024-04-26T16:30:18.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f06d81b9790 con 0x7f06d81b2130 2024-04-26T16:30:18.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f06d81b2130 0x7f06d81b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d81b2130 2024-04-26T16:30:18.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.959+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f06d81b2130 0x7f06d81b6930 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.963+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81b9790 con 0x7f06d81b2130 2024-04-26T16:30:18.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.963+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 9: 0 ) 2024-04-26T16:30:18.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.963+0000 7f06d57fa700 10 client.41309.objecter _finish_command 9 = system:0 2024-04-26T16:30:18.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.963+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 4 2024-04-26T16:30:18.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.963+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 4 9 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 10 client.41309.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 10 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 10 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81ab020 osd=4 3 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 4 10 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 4 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 10 client.41309.objecter _send_command 10 2024-04-26T16:30:18.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.979+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d81b6e70 con 0x7f06d81b2130 2024-04-26T16:30:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d81b6e70 con 0x7f06d81b2130 2024-04-26T16:30:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06d57fa700 10 client.41309.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 4 2024-04-26T16:30:18.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 4 10 2024-04-26T16:30:18.984 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 10 client.41309.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 11 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f06d81b75e0 0x7f06d81b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 11 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:18.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 5 11 2024-04-26T16:30:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 4 2024-04-26T16:30:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 10 client.41309.objecter _send_command 11 2024-04-26T16:30:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f06d81bcbb0 con 0x7f06d81b75e0 2024-04-26T16:30:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.983+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f06d81b75e0 0x7f06d81b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d81b75e0 2024-04-26T16:30:18.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d77fe700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f06d81b75e0 0x7f06d81b7a20 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81bcbb0 con 0x7f06d81b75e0 2024-04-26T16:30:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 11: 0 ) 2024-04-26T16:30:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d57fa700 10 client.41309.objecter _finish_command 11 = system:0 2024-04-26T16:30:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 4 2024-04-26T16:30:18.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:18.987+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 5 11 2024-04-26T16:30:19.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 10 client.41309.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 12 2024-04-26T16:30:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:19.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 12 2024-04-26T16:30:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81b6e70 osd=5 3 2024-04-26T16:30:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 5 12 2024-04-26T16:30:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 4 2024-04-26T16:30:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 10 client.41309.objecter _send_command 12 2024-04-26T16:30:19.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.011+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d81b8070 con 0x7f06d81b75e0 2024-04-26T16:30:19.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d81b8070 con 0x7f06d81b75e0 2024-04-26T16:30:19.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:19.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06d57fa700 10 client.41309.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:19.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 4 2024-04-26T16:30:19.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 5 12 2024-04-26T16:30:19.017 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:30:19.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 10 client.41309.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:19.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 13 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f06d81b8790 0x7f06d81b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 13 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 6 13 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 4 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 10 client.41309.objecter _send_command 13 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.015+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f06d81b9240 con 0x7f06d81b8790 2024-04-26T16:30:19.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f06d81b8790 0x7f06d81b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:19.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d81b8790 2024-04-26T16:30:19.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06dcb6d700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f06d81b8790 0x7f06d81b8bf0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81b9240 con 0x7f06d81b8790 2024-04-26T16:30:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 13: 0 ) 2024-04-26T16:30:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06d57fa700 10 client.41309.objecter _finish_command 13 = system:0 2024-04-26T16:30:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 4 2024-04-26T16:30:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.019+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 6 13 2024-04-26T16:30:19.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:19.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 10 client.41309.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 14 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 14 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81b8070 osd=6 3 2024-04-26T16:30:19.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 6 14 2024-04-26T16:30:19.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 4 2024-04-26T16:30:19.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 10 client.41309.objecter _send_command 14 2024-04-26T16:30:19.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.043+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d81b9240 con 0x7f06d81b8790 2024-04-26T16:30:19.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d81b9240 con 0x7f06d81b8790 2024-04-26T16:30:19.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:19.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d57fa700 10 client.41309.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:19.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 4 2024-04-26T16:30:19.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 6 14 2024-04-26T16:30:19.047 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:30:19.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 10 client.41309.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:19.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 15 2024-04-26T16:30:19.047 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f06d81b93b0 0x7f06d81bff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 15 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 7 15 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 4 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 10 client.41309.objecter _send_command 15 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f06d81c2fd0 con 0x7f06d81b93b0 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f06d81b93b0 0x7f06d81bff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d57fa700 10 client.41309.objecter ms_handle_connect 0x7f06d81b93b0 2024-04-26T16:30:19.048 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.047+0000 7f06d7fff700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f06d81b93b0 0x7f06d81bff60 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:19.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.051+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d81c2fd0 con 0x7f06d81b93b0 2024-04-26T16:30:19.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.051+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 15: 0 ) 2024-04-26T16:30:19.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.051+0000 7f06d57fa700 10 client.41309.objecter _finish_command 15 = system:0 2024-04-26T16:30:19.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.051+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 4 2024-04-26T16:30:19.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.051+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 7 15 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 10 client.41309.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign -1 16 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 20 client.41309.objecter _get_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_remove -1 16 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 20 client.41309.objecter get_session s=0x7f06d81b9240 osd=7 3 2024-04-26T16:30:19.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 15 client.41309.objecter _session_command_op_assign 7 16 2024-04-26T16:30:19.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 4 2024-04-26T16:30:19.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 10 client.41309.objecter _send_command 16 2024-04-26T16:30:19.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.067+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f06d81c1d60 con 0x7f06d81b93b0 2024-04-26T16:30:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06d57fa700 1 -- 172.21.15.50:0/3563160890 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f06d81c1d60 con 0x7f06d81b93b0 2024-04-26T16:30:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06d57fa700 10 client.41309.objecter ms_dispatch 0x7f06d8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06d57fa700 10 client.41309.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06d57fa700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 4 2024-04-26T16:30:19.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06d57fa700 15 client.41309.objecter _session_command_op_remove 7 16 2024-04-26T16:30:19.074 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:30:19.074 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 librados: watch_flush enter 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 librados: watch_flush exit 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.0 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f06d8064030 msgr2=0x7f06d8064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f06d8064030 0x7f06d8064410 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d819c7d0 osd=0 1 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.1 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f06d802d310 msgr2=0x7f06d802d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f06d802d310 0x7f06d802d6f0 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8100960 osd=1 1 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.2 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f06d81ad510 msgr2=0x7f06d81af900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f06d81ad510 0x7f06d81af900 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d8101190 osd=2 1 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.3 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f06d81b0fc0 msgr2=0x7f06d81b1420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f06d81b0fc0 0x7f06d81b1420 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81aa9a0 osd=3 1 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.4 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f06d81b2130 msgr2=0x7f06d81b6930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f06d81b2130 0x7f06d81b6930 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81ab020 osd=4 1 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.5 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f06d81b75e0 msgr2=0x7f06d81b7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06dcb6d700 1 -- 172.21.15.50:0/3563160890 reap_dead start 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f06d81b75e0 0x7f06d81b7a20 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06dcb6d700 1 -- 172.21.15.50:0/3563160890 reap_dead start 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b6e70 osd=5 1 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.6 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f06d81b8790 msgr2=0x7f06d81b8bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f06d81b8790 0x7f06d81b8bf0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b8070 osd=6 1 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter close_session for osd.7 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f06d81b93b0 msgr2=0x7f06d81bff60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f06d81b93b0 0x7f06d81bff60 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter put_session s=0x7f06d81b9240 osd=7 1 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 20 client.41309.objecter shutdown clearing up homeless session... 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 10 client.41309.objecter successfully canceled tick 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f06c4071f80 msgr2=0x7f06c4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f06c4071f80 0x7f06c4074440 secure :-1 s=READY pgs=3763 cs=0 l=1 rev1=1 crypto rx=0x7f06c000b730 tx=0x7f06c001a040 comp rx=0 tx=0).stop 2024-04-26T16:30:19.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806b130 msgr2=0x7f06d819b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 --2- 172.21.15.50:0/3563160890 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f06d806b130 0x7f06d819b470 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7f06c800d950 tx=0x7f06c800dd10 comp rx=0 tx=0).stop 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06dcb6d700 1 -- 172.21.15.50:0/3563160890 reap_dead start 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 shutdown_connections 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 >> 172.21.15.50:0/3563160890 conn(0x7f06d80792d0 msgr2=0x7f06d810f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 shutdown_connections 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 -- 172.21.15.50:0/3563160890 wait complete. 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:19.075+0000 7f06de5d0700 1 librados: shutdown 2024-04-26T16:30:19.078 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:30:19.079 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:30:19.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:19.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:19.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:19.280 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:19.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:19.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:19.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:19.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:19.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:19.660 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.307796+0000 osd.6 (osd.6) 35 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.307796+0000 osd.6 (osd.6) 35 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.310733+0000 osd.6 (osd.6) 36 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.310733+0000 osd.6 (osd.6) 36 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.491416+0000 osd.4 (osd.4) 269 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.491416+0000 osd.4 (osd.4) 269 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.494620+0000 osd.4 (osd.4) 270 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:30:19.710 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: cluster 2024-04-26T16:30:18.494620+0000 osd.4 (osd.4) 270 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: audit 2024-04-26T16:30:18.863514+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? 172.21.15.50:0/3563160890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: audit 2024-04-26T16:30:18.863514+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? 172.21.15.50:0/3563160890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: audit 2024-04-26T16:30:19.368560+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: audit 2024-04-26T16:30:19.368560+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: audit 2024-04-26T16:30:19.375292+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[21463]: audit 2024-04-26T16:30:19.375292+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.307796+0000 osd.6 (osd.6) 35 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.307796+0000 osd.6 (osd.6) 35 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.310733+0000 osd.6 (osd.6) 36 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.310733+0000 osd.6 (osd.6) 36 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:30:19.711 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.491416+0000 osd.4 (osd.4) 269 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.491416+0000 osd.4 (osd.4) 269 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.494620+0000 osd.4 (osd.4) 270 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: cluster 2024-04-26T16:30:18.494620+0000 osd.4 (osd.4) 270 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: audit 2024-04-26T16:30:18.863514+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? 172.21.15.50:0/3563160890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: audit 2024-04-26T16:30:18.863514+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? 172.21.15.50:0/3563160890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: audit 2024-04-26T16:30:19.368560+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: audit 2024-04-26T16:30:19.368560+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: audit 2024-04-26T16:30:19.375292+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.712 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:19 smithi050 bash[29234]: audit 2024-04-26T16:30:19.375292+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:19.786 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:19.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.307796+0000 osd.6 (osd.6) 35 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.307796+0000 osd.6 (osd.6) 35 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.310733+0000 osd.6 (osd.6) 36 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.310733+0000 osd.6 (osd.6) 36 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.491416+0000 osd.4 (osd.4) 269 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.491416+0000 osd.4 (osd.4) 269 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.494620+0000 osd.4 (osd.4) 270 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: cluster 2024-04-26T16:30:18.494620+0000 osd.4 (osd.4) 270 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: audit 2024-04-26T16:30:18.863514+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? 172.21.15.50:0/3563160890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: audit 2024-04-26T16:30:18.863514+0000 mon.c (mon.2) 644 : audit [DBG] from='client.? 172.21.15.50:0/3563160890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: audit 2024-04-26T16:30:19.368560+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: audit 2024-04-26T16:30:19.368560+0000 mon.a (mon.0) 2863 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: audit 2024-04-26T16:30:19.375292+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:19 smithi190 bash[23868]: audit 2024-04-26T16:30:19.375292+0000 mon.a (mon.0) 2864 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:19.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:19.916 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:19.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:20.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:20.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:20.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:20.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:20.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:20.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:20.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:20.434 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:20.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:30:20.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:20.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:20.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:20.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.323045+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:30:20.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.323045+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:30:20.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.325084+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:30:20.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.325084+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.530754+0000 osd.2 (osd.2) 257 : cluster [DBG] 11.d scrub starts 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.530754+0000 osd.2 (osd.2) 257 : cluster [DBG] 11.d scrub starts 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.533036+0000 osd.2 (osd.2) 258 : cluster [DBG] 11.d scrub ok 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.533036+0000 osd.2 (osd.2) 258 : cluster [DBG] 11.d scrub ok 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.710261+0000 mgr.y (mgr.24401) 503 : cluster [DBG] pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.710261+0000 mgr.y (mgr.24401) 503 : cluster [DBG] pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: audit 2024-04-26T16:30:19.945265+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: audit 2024-04-26T16:30:19.945265+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: audit 2024-04-26T16:30:19.947121+0000 mon.c (mon.2) 646 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: audit 2024-04-26T16:30:19.947121+0000 mon.c (mon.2) 646 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: audit 2024-04-26T16:30:19.954994+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:20.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:20 smithi190 bash[23868]: audit 2024-04-26T16:30:19.954994+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:20.697 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:20.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:20.828 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.323045+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.323045+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.325084+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.325084+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.530754+0000 osd.2 (osd.2) 257 : cluster [DBG] 11.d scrub starts 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.530754+0000 osd.2 (osd.2) 257 : cluster [DBG] 11.d scrub starts 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.533036+0000 osd.2 (osd.2) 258 : cluster [DBG] 11.d scrub ok 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.533036+0000 osd.2 (osd.2) 258 : cluster [DBG] 11.d scrub ok 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.710261+0000 mgr.y (mgr.24401) 503 : cluster [DBG] pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.710261+0000 mgr.y (mgr.24401) 503 : cluster [DBG] pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: audit 2024-04-26T16:30:19.945265+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: audit 2024-04-26T16:30:19.945265+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: audit 2024-04-26T16:30:19.947121+0000 mon.c (mon.2) 646 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: audit 2024-04-26T16:30:19.947121+0000 mon.c (mon.2) 646 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: audit 2024-04-26T16:30:19.954994+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[29234]: audit 2024-04-26T16:30:19.954994+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.323045+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.323045+0000 osd.6 (osd.6) 37 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.325084+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.325084+0000 osd.6 (osd.6) 38 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.530754+0000 osd.2 (osd.2) 257 : cluster [DBG] 11.d scrub starts 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.530754+0000 osd.2 (osd.2) 257 : cluster [DBG] 11.d scrub starts 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.533036+0000 osd.2 (osd.2) 258 : cluster [DBG] 11.d scrub ok 2024-04-26T16:30:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.533036+0000 osd.2 (osd.2) 258 : cluster [DBG] 11.d scrub ok 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.710261+0000 mgr.y (mgr.24401) 503 : cluster [DBG] pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.710261+0000 mgr.y (mgr.24401) 503 : cluster [DBG] pgmap v763: 228 pgs: 228 active+clean; 8.4 MiB data, 508 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: audit 2024-04-26T16:30:19.945265+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: audit 2024-04-26T16:30:19.945265+0000 mon.c (mon.2) 645 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: audit 2024-04-26T16:30:19.947121+0000 mon.c (mon.2) 646 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: audit 2024-04-26T16:30:19.947121+0000 mon.c (mon.2) 646 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: audit 2024-04-26T16:30:19.954994+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:20 smithi050 bash[21463]: audit 2024-04-26T16:30:19.954994+0000 mon.a (mon.0) 2865 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:20.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:20.959 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:20.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:21.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:21.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:21.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:21.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:21.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:21.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:21.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:21.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:21.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:21.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:21.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:21.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:21.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:21.758 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:21.759 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:21.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.955773+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.955773+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.960111+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:19.960111+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:20.628791+0000 osd.0 (osd.0) 299 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:20.628791+0000 osd.0 (osd.0) 299 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:20.631306+0000 osd.0 (osd.0) 300 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:20.631306+0000 osd.0 (osd.0) 300 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.204263+0000 mon.a (mon.0) 2866 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.204263+0000 mon.a (mon.0) 2866 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.214738+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e504: 8 total, 8 up, 7 in 2024-04-26T16:30:21.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:21 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.214738+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e504: 8 total, 8 up, 7 in 2024-04-26T16:30:21.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:21.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.955773+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:30:21.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.955773+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:30:21.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.960111+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:30:21.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:19.960111+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:20.628791+0000 osd.0 (osd.0) 299 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:20.628791+0000 osd.0 (osd.0) 299 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:20.631306+0000 osd.0 (osd.0) 300 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:20.631306+0000 osd.0 (osd.0) 300 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.204263+0000 mon.a (mon.0) 2866 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.204263+0000 mon.a (mon.0) 2866 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.214738+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e504: 8 total, 8 up, 7 in 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.214738+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e504: 8 total, 8 up, 7 in 2024-04-26T16:30:21.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.955773+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.955773+0000 osd.3 (osd.3) 321 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.960111+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:19.960111+0000 osd.3 (osd.3) 322 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:20.628791+0000 osd.0 (osd.0) 299 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:20.628791+0000 osd.0 (osd.0) 299 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:20.631306+0000 osd.0 (osd.0) 300 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:20.631306+0000 osd.0 (osd.0) 300 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.204263+0000 mon.a (mon.0) 2866 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.204263+0000 mon.a (mon.0) 2866 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.214738+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e504: 8 total, 8 up, 7 in 2024-04-26T16:30:21.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:21 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.214738+0000 mon.a (mon.0) 2867 : cluster [DBG] osdmap e504: 8 total, 8 up, 7 in 2024-04-26T16:30:21.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:21.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:21.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:22.002 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:22.105 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:22.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:22.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:22.265 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:22.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:30:22.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:22.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:22.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:22.520 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:22.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.495238+0000 osd.4 (osd.4) 271 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.495238+0000 osd.4 (osd.4) 271 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.500160+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.500160+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.712488+0000 mgr.y (mgr.24401) 504 : cluster [DBG] pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.712488+0000 mgr.y (mgr.24401) 504 : cluster [DBG] pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.495238+0000 osd.4 (osd.4) 271 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:30:22.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.495238+0000 osd.4 (osd.4) 271 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:30:22.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.500160+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:30:22.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.500160+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:30:22.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.712488+0000 mgr.y (mgr.24401) 504 : cluster [DBG] pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-26T16:30:22.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:22 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.712488+0000 mgr.y (mgr.24401) 504 : cluster [DBG] pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-26T16:30:22.650 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:22.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:22.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:22.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:22.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:22 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.495238+0000 osd.4 (osd.4) 271 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:30:22.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:22 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.495238+0000 osd.4 (osd.4) 271 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:30:22.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:22 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.500160+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:30:22.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:22 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.500160+0000 osd.4 (osd.4) 272 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:30:22.822 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:22 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.712488+0000 mgr.y (mgr.24401) 504 : cluster [DBG] pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-26T16:30:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:22 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.712488+0000 mgr.y (mgr.24401) 504 : cluster [DBG] pgmap v765: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-04-26T16:30:22.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:22.914 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:22.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:22.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:23.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:23.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:23.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:23.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:23.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:23.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:23.310 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:23.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:23.441 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:23.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:23.573 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:23.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:23.688 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.965598+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:30:23.688 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.965598+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:30:23.688 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.967748+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.967748+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.972418+0000 osd.3 (osd.3) 323 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.972418+0000 osd.3 (osd.3) 323 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.978003+0000 osd.3 (osd.3) 324 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:21.978003+0000 osd.3 (osd.3) 324 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.490986+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.490986+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:30:23.689 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.493402+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.493402+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.614310+0000 osd.2 (osd.2) 259 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.614310+0000 osd.2 (osd.2) 259 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.616060+0000 osd.2 (osd.2) 260 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.616060+0000 osd.2 (osd.2) 260 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.622835+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.622835+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.624623+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:30:23.690 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[21463]: cluster 2024-04-26T16:30:22.624623+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.965598+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.965598+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.967748+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.967748+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.972418+0000 osd.3 (osd.3) 323 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.972418+0000 osd.3 (osd.3) 323 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.978003+0000 osd.3 (osd.3) 324 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:21.978003+0000 osd.3 (osd.3) 324 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.490986+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:30:23.691 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.490986+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.493402+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.493402+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.614310+0000 osd.2 (osd.2) 259 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.614310+0000 osd.2 (osd.2) 259 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.616060+0000 osd.2 (osd.2) 260 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.616060+0000 osd.2 (osd.2) 260 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.622835+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.622835+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.624623+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:30:23.692 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:23 smithi050 bash[29234]: cluster 2024-04-26T16:30:22.624623+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:30:23.704 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:23.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.965598+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.965598+0000 osd.1 (osd.1) 271 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.967748+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.967748+0000 osd.1 (osd.1) 272 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.972418+0000 osd.3 (osd.3) 323 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.972418+0000 osd.3 (osd.3) 323 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.978003+0000 osd.3 (osd.3) 324 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:21.978003+0000 osd.3 (osd.3) 324 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.490986+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.490986+0000 osd.4 (osd.4) 273 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.493402+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:30:23.820 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.493402+0000 osd.4 (osd.4) 274 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.614310+0000 osd.2 (osd.2) 259 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.614310+0000 osd.2 (osd.2) 259 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.616060+0000 osd.2 (osd.2) 260 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.616060+0000 osd.2 (osd.2) 260 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.622835+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.622835+0000 osd.0 (osd.0) 301 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.624623+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:30:23.821 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:23 smithi190 bash[23868]: cluster 2024-04-26T16:30:22.624623+0000 osd.0 (osd.0) 302 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:30:23.835 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:23.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:23.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:23.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:24.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:24.095 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:24.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:24.224 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:24.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:24.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:24.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:24.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:24.585 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:24.613 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:24.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:24.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.312462+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.312462+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.313668+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.313668+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.515912+0000 osd.4 (osd.4) 275 : cluster [DBG] 5.f scrub starts 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.515912+0000 osd.4 (osd.4) 275 : cluster [DBG] 5.f scrub starts 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.518316+0000 osd.4 (osd.4) 276 : cluster [DBG] 5.f scrub ok 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.518316+0000 osd.4 (osd.4) 276 : cluster [DBG] 5.f scrub ok 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.713230+0000 mgr.y (mgr.24401) 505 : cluster [DBG] pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-04-26T16:30:24.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:24 smithi190 bash[23868]: cluster 2024-04-26T16:30:23.713230+0000 mgr.y (mgr.24401) 505 : cluster [DBG] pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-04-26T16:30:24.743 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:24.841 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:30:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:30:24.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.312462+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.312462+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.313668+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.313668+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.515912+0000 osd.4 (osd.4) 275 : cluster [DBG] 5.f scrub starts 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.515912+0000 osd.4 (osd.4) 275 : cluster [DBG] 5.f scrub starts 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.518316+0000 osd.4 (osd.4) 276 : cluster [DBG] 5.f scrub ok 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.518316+0000 osd.4 (osd.4) 276 : cluster [DBG] 5.f scrub ok 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.713230+0000 mgr.y (mgr.24401) 505 : cluster [DBG] pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[21463]: cluster 2024-04-26T16:30:23.713230+0000 mgr.y (mgr.24401) 505 : cluster [DBG] pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-04-26T16:30:24.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.312462+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.312462+0000 osd.6 (osd.6) 39 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.313668+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.313668+0000 osd.6 (osd.6) 40 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.515912+0000 osd.4 (osd.4) 275 : cluster [DBG] 5.f scrub starts 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.515912+0000 osd.4 (osd.4) 275 : cluster [DBG] 5.f scrub starts 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.518316+0000 osd.4 (osd.4) 276 : cluster [DBG] 5.f scrub ok 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.518316+0000 osd.4 (osd.4) 276 : cluster [DBG] 5.f scrub ok 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.713230+0000 mgr.y (mgr.24401) 505 : cluster [DBG] pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-04-26T16:30:24.843 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:24 smithi050 bash[29234]: cluster 2024-04-26T16:30:23.713230+0000 mgr.y (mgr.24401) 505 : cluster [DBG] pgmap v766: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-04-26T16:30:24.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:24.874 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:24.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:24.977 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:25.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:25.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:25.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:25.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:25.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:25.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:25.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:25.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:25.531 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:25.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:25.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:25.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:25.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.023772+0000 osd.1 (osd.1) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.023772+0000 osd.1 (osd.1) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.025159+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.025159+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.565017+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.565017+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.567763+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.567763+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.601906+0000 osd.2 (osd.2) 261 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.601906+0000 osd.2 (osd.2) 261 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.606130+0000 osd.2 (osd.2) 262 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:30:25.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:25 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.606130+0000 osd.2 (osd.2) 262 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:30:25.789 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:25.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.023772+0000 osd.1 (osd.1) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:30:25.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.023772+0000 osd.1 (osd.1) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.025159+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.025159+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.565017+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.565017+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.567763+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.567763+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.601906+0000 osd.2 (osd.2) 261 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.601906+0000 osd.2 (osd.2) 261 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.606130+0000 osd.2 (osd.2) 262 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:30:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.606130+0000 osd.2 (osd.2) 262 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.023772+0000 osd.1 (osd.1) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.023772+0000 osd.1 (osd.1) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.025159+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.025159+0000 osd.1 (osd.1) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.565017+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.565017+0000 osd.0 (osd.0) 303 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.567763+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.567763+0000 osd.0 (osd.0) 304 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.601906+0000 osd.2 (osd.2) 261 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:30:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.601906+0000 osd.2 (osd.2) 261 : cluster [DBG] 109.13 scrub starts 2024-04-26T16:30:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.606130+0000 osd.2 (osd.2) 262 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:30:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:25 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.606130+0000 osd.2 (osd.2) 262 : cluster [DBG] 109.13 scrub ok 2024-04-26T16:30:25.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:25.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:25.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:26.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:26.051 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:26.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:26.181 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:26.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:26.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:26.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:26.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:26.431 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:26.450 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:26.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:26.580 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:26.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:26.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:26.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.902819+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:30:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.902819+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:30:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.905837+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:30:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:24.905837+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:30:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.048352+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.d scrub starts 2024-04-26T16:30:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.048352+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.d scrub starts 2024-04-26T16:30:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.049641+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.d scrub ok 2024-04-26T16:30:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.049641+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.d scrub ok 2024-04-26T16:30:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.204984+0000 osd.3 (osd.3) 327 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:26.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.204984+0000 osd.3 (osd.3) 327 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:26.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.205246+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:26.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.205246+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:26.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.326232+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:30:26.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.326232+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:30:26.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.328532+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:30:26.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.328532+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:30:26.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.592272+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2024-04-26T16:30:26.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.592272+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2024-04-26T16:30:26.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.637935+0000 osd.2 (osd.2) 263 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:30:26.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.637935+0000 osd.2 (osd.2) 263 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:30:26.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.642092+0000 osd.2 (osd.2) 264 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:30:26.830 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.642092+0000 osd.2 (osd.2) 264 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:30:26.830 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.714012+0000 mgr.y (mgr.24401) 506 : cluster [DBG] pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:26.830 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:25.714012+0000 mgr.y (mgr.24401) 506 : cluster [DBG] pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:26.830 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.214126+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2024-04-26T16:30:26.830 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.214126+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2024-04-26T16:30:26.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.343085+0000 mon.a (mon.0) 2870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.343085+0000 mon.a (mon.0) 2870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.344037+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.344037+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.831 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.344515+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.832 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.344515+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.832 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.345563+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.832 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[23868]: audit 2024-04-26T16:30:26.345563+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.844 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.902819+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.902819+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.905837+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:24.905837+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.048352+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.d scrub starts 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.048352+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.d scrub starts 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.049641+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.d scrub ok 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.049641+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.d scrub ok 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.204984+0000 osd.3 (osd.3) 327 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.204984+0000 osd.3 (osd.3) 327 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.205246+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.205246+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.326232+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.326232+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.328532+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.328532+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.592272+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.592272+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.637935+0000 osd.2 (osd.2) 263 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.637935+0000 osd.2 (osd.2) 263 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.642092+0000 osd.2 (osd.2) 264 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.642092+0000 osd.2 (osd.2) 264 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.714012+0000 mgr.y (mgr.24401) 506 : cluster [DBG] pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:25.714012+0000 mgr.y (mgr.24401) 506 : cluster [DBG] pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.214126+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.214126+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.343085+0000 mon.a (mon.0) 2870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.902819+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.902819+0000 osd.3 (osd.3) 325 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.905837+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:24.905837+0000 osd.3 (osd.3) 326 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.048352+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.d scrub starts 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.048352+0000 osd.1 (osd.1) 275 : cluster [DBG] 6.d scrub starts 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.049641+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.d scrub ok 2024-04-26T16:30:26.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.049641+0000 osd.1 (osd.1) 276 : cluster [DBG] 6.d scrub ok 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.204984+0000 osd.3 (osd.3) 327 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.204984+0000 osd.3 (osd.3) 327 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.205246+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.205246+0000 osd.3 (osd.3) 328 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.326232+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.326232+0000 osd.6 (osd.6) 41 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.328532+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.328532+0000 osd.6 (osd.6) 42 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.592272+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.592272+0000 mon.a (mon.0) 2868 : cluster [DBG] osdmap e505: 8 total, 8 up, 7 in 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.637935+0000 osd.2 (osd.2) 263 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.637935+0000 osd.2 (osd.2) 263 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:30:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.642092+0000 osd.2 (osd.2) 264 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.642092+0000 osd.2 (osd.2) 264 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.714012+0000 mgr.y (mgr.24401) 506 : cluster [DBG] pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:25.714012+0000 mgr.y (mgr.24401) 506 : cluster [DBG] pgmap v768: 228 pgs: 228 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.214126+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.214126+0000 mon.a (mon.0) 2869 : cluster [DBG] osdmap e506: 8 total, 8 up, 7 in 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.343085+0000 mon.a (mon.0) 2870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.343085+0000 mon.a (mon.0) 2870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.344037+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.344037+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.344515+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.344515+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.345563+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[29234]: audit 2024-04-26T16:30:26.345563+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.343085+0000 mon.a (mon.0) 2870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.344037+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.344037+0000 mon.b (mon.1) 345 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.344515+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.344515+0000 mon.a (mon.0) 2871 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.345563+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:26 smithi050 bash[21463]: audit 2024-04-26T16:30:26.345563+0000 mon.b (mon.1) 346 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-73"}]: dispatch 2024-04-26T16:30:26.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:26.977 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:26.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:27.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:27.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:27.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:27.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:27.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:30:26 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:30:27.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:27.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:27.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:27.502 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:27.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.029775+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.029775+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.031135+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.031135+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.545654+0000 osd.4 (osd.4) 277 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.545654+0000 osd.4 (osd.4) 277 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.547745+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.547745+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.630524+0000 osd.0 (osd.0) 305 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:30:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.630524+0000 osd.0 (osd.0) 305 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:30:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.632634+0000 osd.0 (osd.0) 306 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:30:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.632634+0000 osd.0 (osd.0) 306 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:30:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: audit 2024-04-26T16:30:27.013365+0000 mon.a (mon.0) 2872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: audit 2024-04-26T16:30:27.013365+0000 mon.a (mon.0) 2872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: audit 2024-04-26T16:30:27.016534+0000 mon.c (mon.2) 647 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: audit 2024-04-26T16:30:27.016534+0000 mon.c (mon.2) 647 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.224791+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.224791+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.316543+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.316543+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.316971+0000 osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:27.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.316971+0000 osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.029775+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.029775+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.031135+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.031135+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.545654+0000 osd.4 (osd.4) 277 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.545654+0000 osd.4 (osd.4) 277 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.547745+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.547745+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.630524+0000 osd.0 (osd.0) 305 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.630524+0000 osd.0 (osd.0) 305 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:30:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.632634+0000 osd.0 (osd.0) 306 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.632634+0000 osd.0 (osd.0) 306 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: audit 2024-04-26T16:30:27.013365+0000 mon.a (mon.0) 2872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: audit 2024-04-26T16:30:27.013365+0000 mon.a (mon.0) 2872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: audit 2024-04-26T16:30:27.016534+0000 mon.c (mon.2) 647 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: audit 2024-04-26T16:30:27.016534+0000 mon.c (mon.2) 647 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.224791+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.224791+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.316543+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.316543+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.316971+0000 osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:27.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:27 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.316971+0000 osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.635 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:27.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:27.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:27.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.029775+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:30:27.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.029775+0000 osd.1 (osd.1) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:30:27.750 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.031135+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.031135+0000 osd.1 (osd.1) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.545654+0000 osd.4 (osd.4) 277 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.545654+0000 osd.4 (osd.4) 277 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.547745+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.547745+0000 osd.4 (osd.4) 278 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.630524+0000 osd.0 (osd.0) 305 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.630524+0000 osd.0 (osd.0) 305 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.632634+0000 osd.0 (osd.0) 306 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.632634+0000 osd.0 (osd.0) 306 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: audit 2024-04-26T16:30:27.013365+0000 mon.a (mon.0) 2872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: audit 2024-04-26T16:30:27.013365+0000 mon.a (mon.0) 2872 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: audit 2024-04-26T16:30:27.016534+0000 mon.c (mon.2) 647 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:27.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: audit 2024-04-26T16:30:27.016534+0000 mon.c (mon.2) 647 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:27.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.224791+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2024-04-26T16:30:27.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.224791+0000 mon.a (mon.0) 2873 : cluster [DBG] osdmap e507: 8 total, 8 up, 7 in 2024-04-26T16:30:27.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.316543+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:27.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.316543+0000 osd.5 (osd.5) 207 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:27.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.316971+0000 osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:27.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:27 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.316971+0000 osd.5 (osd.5) 208 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:27.765 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:27.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:27.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:27.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:27.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:28.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:28.026 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:28.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:28.158 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:28.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:28.289 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:28.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:28.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:28.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:28.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:28.619 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:30:28.620 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:30:28.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:28.674 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:28.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.946753+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.946753+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.949347+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:26.949347+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: audit 2024-04-26T16:30:26.960318+0000 mgr.y (mgr.24401) 507 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: audit 2024-04-26T16:30:26.960318+0000 mgr.y (mgr.24401) 507 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:28.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.080291+0000 osd.1 (osd.1) 279 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.080291+0000 osd.1 (osd.1) 279 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.084832+0000 osd.1 (osd.1) 280 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.084832+0000 osd.1 (osd.1) 280 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.714790+0000 mgr.y (mgr.24401) 508 : cluster [DBG] pgmap v771: 196 pgs: 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:27.714790+0000 mgr.y (mgr.24401) 508 : cluster [DBG] pgmap v771: 196 pgs: 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.225011+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.225011+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: audit 2024-04-26T16:30:28.229649+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: audit 2024-04-26T16:30:28.229649+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: audit 2024-04-26T16:30:28.231271+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[29234]: audit 2024-04-26T16:30:28.231271+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.946753+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.946753+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.949347+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:26.949347+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: audit 2024-04-26T16:30:26.960318+0000 mgr.y (mgr.24401) 507 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: audit 2024-04-26T16:30:26.960318+0000 mgr.y (mgr.24401) 507 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.080291+0000 osd.1 (osd.1) 279 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.080291+0000 osd.1 (osd.1) 279 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.084832+0000 osd.1 (osd.1) 280 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.084832+0000 osd.1 (osd.1) 280 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.714790+0000 mgr.y (mgr.24401) 508 : cluster [DBG] pgmap v771: 196 pgs: 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:27.714790+0000 mgr.y (mgr.24401) 508 : cluster [DBG] pgmap v771: 196 pgs: 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.225011+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.225011+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2024-04-26T16:30:28.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: audit 2024-04-26T16:30:28.229649+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: audit 2024-04-26T16:30:28.229649+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: audit 2024-04-26T16:30:28.231271+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:28 smithi050 bash[21463]: audit 2024-04-26T16:30:28.231271+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.803 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:28.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.946753+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.946753+0000 osd.3 (osd.3) 329 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.949347+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:26.949347+0000 osd.3 (osd.3) 330 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: audit 2024-04-26T16:30:26.960318+0000 mgr.y (mgr.24401) 507 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: audit 2024-04-26T16:30:26.960318+0000 mgr.y (mgr.24401) 507 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.080291+0000 osd.1 (osd.1) 279 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:30:28.917 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.080291+0000 osd.1 (osd.1) 279 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.084832+0000 osd.1 (osd.1) 280 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.084832+0000 osd.1 (osd.1) 280 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.714790+0000 mgr.y (mgr.24401) 508 : cluster [DBG] pgmap v771: 196 pgs: 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:27.714790+0000 mgr.y (mgr.24401) 508 : cluster [DBG] pgmap v771: 196 pgs: 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.225011+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.225011+0000 mon.a (mon.0) 2874 : cluster [DBG] osdmap e508: 8 total, 8 up, 7 in 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: audit 2024-04-26T16:30:28.229649+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: audit 2024-04-26T16:30:28.229649+0000 mon.b (mon.1) 347 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: audit 2024-04-26T16:30:28.231271+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.918 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:28 smithi190 bash[23868]: audit 2024-04-26T16:30:28.231271+0000 mon.a (mon.0) 2875 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:28.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:28.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 -- 172.21.15.50:0/718848431 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd09006bb60 msgr2=0x7fd09006bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 --2- 172.21.15.50:0/718848431 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd09006bb60 0x7fd09006bfe0 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00b0a0 tx=0x7fd08c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:30:28.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 -- 172.21.15.50:0/718848431 shutdown_connections 2024-04-26T16:30:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 --2- 172.21.15.50:0/718848431 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09006c520 0x7fd090110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 --2- 172.21.15.50:0/718848431 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd0901095c0 0x7fd0901099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 --2- 172.21.15.50:0/718848431 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd09006bb60 0x7fd09006bfe0 unknown :-1 s=CLOSED pgs=2682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 -- 172.21.15.50:0/718848431 >> 172.21.15.50:0/718848431 conn(0x7fd090079120 msgr2=0x7fd090079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 -- 172.21.15.50:0/718848431 shutdown_connections 2024-04-26T16:30:28.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.935+0000 7fd0980dc700 1 -- 172.21.15.50:0/718848431 wait complete. 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 librados: starting msgr at 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 librados: starting objecter 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 Processor -- start 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- start start 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 librados: setting wanted keys 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 librados: calling monclient init 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09006bb60 0x7fd0901026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 0x7fd090102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0901095c0 0x7fd090103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:28.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd090112b50 con 0x7fd09006c520 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd0901129d0 con 0x7fd09006bb60 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd090112cd0 con 0x7fd0901095c0 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd095677700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 0x7fd090102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd095677700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 0x7fd090102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56310/0 (socket says 172.21.15.50:56310) 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0901095c0 0x7fd090103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd095677700 1 -- 172.21.15.50:0/1630146024 learned_addr learned my addr 172.21.15.50:0/1630146024 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd095e78700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09006bb60 0x7fd0901026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 -- 172.21.15.50:0/1630146024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09006bb60 msgr2=0x7fd0901026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09006bb60 0x7fd0901026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 -- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 msgr2=0x7fd090102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 0x7fd090102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 -- 172.21.15.50:0/1630146024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0901aba90 con 0x7fd0901095c0 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd095677700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 0x7fd090102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:30:28.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd096679700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0901095c0 0x7fd090103170 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fd08800d650 tx=0x7fd08800d960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd086ffd700 10 client.?.objecter ms_handle_connect 0x7fd0901095c0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd086ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088016070 con 0x7fd0901095c0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd088004830 con 0x7fd0901095c0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0901abd20 con 0x7fd0901095c0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088005430 con 0x7fd0901095c0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 10 client.41315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0901ac1d0 con 0x7fd0901095c0 2024-04-26T16:30:28.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 librados: init done 2024-04-26T16:30:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd090104bd0 con 0x7fd0901095c0 2024-04-26T16:30:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.939+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd0880055d0 con 0x7fd0901095c0 2024-04-26T16:30:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd078071f80 0x7fd078074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(508..508 src has 378..508) ==== 9364+0+0 (secure 0 0 0) 0x7fd0880912d0 con 0x7fd0901095c0 2024-04-26T16:30:28.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 10 client.41315.objecter ms_dispatch 0x7fd090001550 osd_map(508..508 src has 378..508) 2024-04-26T16:30:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 3 client.41315.objecter handle_osd_map got epochs [508,508] > 0 2024-04-26T16:30:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 3 client.41315.objecter handle_osd_map decoding full epoch 508 2024-04-26T16:30:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd095e78700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd078071f80 0x7fd078074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 20 client.41315.objecter dump_active .. 0 homeless 2024-04-26T16:30:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd086ffd700 10 client.41315.objecter ms_handle_connect 0x7fd078071f80 2024-04-26T16:30:28.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.943+0000 7fd095e78700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd078071f80 0x7fd078074440 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 crypto rx=0x7fd080009680 tx=0x7fd080009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:28.946 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:28.947+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd08801f020 con 0x7fd0901095c0 2024-04-26T16:30:28.998 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:29.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:29.055 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:29.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:29.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:29.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:29.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.203+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fd0901ac460 con 0x7fd0901095c0 2024-04-26T16:30:29.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.219+0000 7fd086ffd700 1 -- 172.21.15.50:0/1630146024 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v509) ==== 113+0+0 (secure 0 0 0) 0x7fd0880212f0 con 0x7fd0901095c0 2024-04-26T16:30:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 10 librados: watch_flush enter 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 10 librados: watch_flush exit 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 20 client.41315.objecter shutdown clearing up homeless session... 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 10 client.41315.objecter successfully canceled tick 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd078071f80 msgr2=0x7fd078074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd078071f80 0x7fd078074440 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 crypto rx=0x7fd080009680 tx=0x7fd080009260 comp rx=0 tx=0).stop 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0901095c0 msgr2=0x7fd090103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:29.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.223+0000 7fd0980dc700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0901095c0 0x7fd090103170 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7fd08800d650 tx=0x7fd08800d960 comp rx=0 tx=0).stop 2024-04-26T16:30:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.227+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 shutdown_connections 2024-04-26T16:30:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.227+0000 7fd0980dc700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd078071f80 0x7fd078074440 unknown :-1 s=CLOSED pgs=3764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:29.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.227+0000 7fd0980dc700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd09006bb60 0x7fd0901026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.227+0000 7fd0980dc700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd0901095c0 0x7fd090103170 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.227+0000 7fd0980dc700 1 --2- 172.21.15.50:0/1630146024 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd09006c520 0x7fd090102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:29.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.227+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 >> 172.21.15.50:0/1630146024 conn(0x7fd090079120 msgr2=0x7fd090100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.231+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 shutdown_connections 2024-04-26T16:30:29.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.231+0000 7fd0980dc700 1 -- 172.21.15.50:0/1630146024 wait complete. 2024-04-26T16:30:29.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:29.235+0000 7fd0980dc700 1 librados: shutdown 2024-04-26T16:30:29.234 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T16:30:29.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:29.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:29.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:29.429 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:29.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:29.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:29.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:29.676 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.034887+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.034887+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.037105+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.037105+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.353573+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 deep-scrub starts 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.353573+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 deep-scrub starts 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.355918+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 deep-scrub ok 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.355918+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 deep-scrub ok 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.564545+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.564545+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.565977+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.565977+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:30:29.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.208686+0000 mon.c (mon.2) 648 : audit [INF] from='client.? 172.21.15.50:0/1630146024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.208686+0000 mon.c (mon.2) 648 : audit [INF] from='client.? 172.21.15.50:0/1630146024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.208984+0000 mon.a (mon.0) 2876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.208984+0000 mon.a (mon.0) 2876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.218592+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.218592+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.218658+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.218658+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.222311+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.222311+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.249586+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:29.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:29 smithi190 bash[23868]: audit 2024-04-26T16:30:29.249586+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:29.692 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:29.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:29.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.034887+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.034887+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.037105+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.037105+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.353573+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 deep-scrub starts 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.353573+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 deep-scrub starts 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.355918+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 deep-scrub ok 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.355918+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 deep-scrub ok 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.564545+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:30:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.564545+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.565977+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.565977+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.208686+0000 mon.c (mon.2) 648 : audit [INF] from='client.? 172.21.15.50:0/1630146024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.208686+0000 mon.c (mon.2) 648 : audit [INF] from='client.? 172.21.15.50:0/1630146024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.208984+0000 mon.a (mon.0) 2876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.208984+0000 mon.a (mon.0) 2876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.218592+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.218592+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.218658+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.218658+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.222311+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.222311+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2024-04-26T16:30:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.249586+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[21463]: audit 2024-04-26T16:30:29.249586+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.034887+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.034887+0000 osd.1 (osd.1) 281 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.037105+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.037105+0000 osd.1 (osd.1) 282 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.353573+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 deep-scrub starts 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.353573+0000 osd.6 (osd.6) 43 : cluster [DBG] 2.11 deep-scrub starts 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.355918+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 deep-scrub ok 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.355918+0000 osd.6 (osd.6) 44 : cluster [DBG] 2.11 deep-scrub ok 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.564545+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:30:29.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.564545+0000 osd.0 (osd.0) 307 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.565977+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.565977+0000 osd.0 (osd.0) 308 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.208686+0000 mon.c (mon.2) 648 : audit [INF] from='client.? 172.21.15.50:0/1630146024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.208686+0000 mon.c (mon.2) 648 : audit [INF] from='client.? 172.21.15.50:0/1630146024' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.208984+0000 mon.a (mon.0) 2876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.208984+0000 mon.a (mon.0) 2876 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.218592+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.218592+0000 mon.a (mon.0) 2877 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.218658+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.218658+0000 mon.a (mon.0) 2878 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.222311+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.222311+0000 mon.a (mon.0) 2879 : cluster [DBG] osdmap e509: 8 total, 8 up, 7 in 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.249586+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:29 smithi050 bash[29234]: audit 2024-04-26T16:30:29.249586+0000 mon.b (mon.1) 348 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:29.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:29.954 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:29.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:30.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:30.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:30.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:30.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:30.343 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:30.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:30.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:30.606 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:30.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.908134+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:30:30.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.908134+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.910462+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:28.910462+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.049617+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.049617+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.051275+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.051275+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.355153+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.355153+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.357852+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.357852+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.495029+0000 osd.4 (osd.4) 279 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:30:30.723 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.495029+0000 osd.4 (osd.4) 279 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.497407+0000 osd.4 (osd.4) 280 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.497407+0000 osd.4 (osd.4) 280 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.715623+0000 mgr.y (mgr.24401) 509 : cluster [DBG] pgmap v774: 228 pgs: 8 creating+peering, 24 unknown, 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:29.715623+0000 mgr.y (mgr.24401) 509 : cluster [DBG] pgmap v774: 228 pgs: 8 creating+peering, 24 unknown, 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:30.224757+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[21463]: cluster 2024-04-26T16:30:30.224757+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2024-04-26T16:30:30.724 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.908134+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.908134+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.910462+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:28.910462+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.049617+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.049617+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.051275+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.051275+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.355153+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.355153+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.357852+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:30:30.725 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.357852+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.495029+0000 osd.4 (osd.4) 279 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.495029+0000 osd.4 (osd.4) 279 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.497407+0000 osd.4 (osd.4) 280 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.497407+0000 osd.4 (osd.4) 280 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.715623+0000 mgr.y (mgr.24401) 509 : cluster [DBG] pgmap v774: 228 pgs: 8 creating+peering, 24 unknown, 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:29.715623+0000 mgr.y (mgr.24401) 509 : cluster [DBG] pgmap v774: 228 pgs: 8 creating+peering, 24 unknown, 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:30.224757+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2024-04-26T16:30:30.726 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:30 smithi050 bash[29234]: cluster 2024-04-26T16:30:30.224757+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2024-04-26T16:30:30.736 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:30.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:30.851 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.908134+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.908134+0000 osd.3 (osd.3) 331 : cluster [DBG] 5.17 deep-scrub starts 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.910462+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:28.910462+0000 osd.3 (osd.3) 332 : cluster [DBG] 5.17 deep-scrub ok 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.049617+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.049617+0000 osd.1 (osd.1) 283 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.051275+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.051275+0000 osd.1 (osd.1) 284 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.355153+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.355153+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.357852+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.357852+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:30:30.852 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.495029+0000 osd.4 (osd.4) 279 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.495029+0000 osd.4 (osd.4) 279 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.497407+0000 osd.4 (osd.4) 280 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.497407+0000 osd.4 (osd.4) 280 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.715623+0000 mgr.y (mgr.24401) 509 : cluster [DBG] pgmap v774: 228 pgs: 8 creating+peering, 24 unknown, 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:29.715623+0000 mgr.y (mgr.24401) 509 : cluster [DBG] pgmap v774: 228 pgs: 8 creating+peering, 24 unknown, 196 active+clean; 8.4 MiB data, 505 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:30.224757+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2024-04-26T16:30:30.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:30 smithi190 bash[23868]: cluster 2024-04-26T16:30:30.224757+0000 mon.a (mon.0) 2880 : cluster [DBG] osdmap e510: 8 total, 8 up, 7 in 2024-04-26T16:30:30.868 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:30.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:30.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:31.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:31.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:31.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:31.258 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:31.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:31.386 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:31.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:31.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:31.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:31.644 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:31.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:31.775 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:31.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:31.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:31.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:32.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:32.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:32.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-26T16:30:32.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:32.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:32.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.226588+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2024-04-26T16:30:32.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.226588+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.317255+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.317255+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.318179+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.318179+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.318679+0000 mon.a (mon.0) 2883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.318679+0000 mon.a (mon.0) 2883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.319891+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: audit 2024-04-26T16:30:31.319891+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.625323+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.625323+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.627921+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.627921+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.717742+0000 mgr.y (mgr.24401) 510 : cluster [DBG] pgmap v777: 228 pgs: 8 creating+peering, 220 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T16:30:32.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.717742+0000 mgr.y (mgr.24401) 510 : cluster [DBG] pgmap v777: 228 pgs: 8 creating+peering, 220 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T16:30:32.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:32.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:32.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:32.395 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.226588+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2024-04-26T16:30:32.395 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.226588+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2024-04-26T16:30:32.395 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.317255+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.395 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.317255+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.318179+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.318179+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.318679+0000 mon.a (mon.0) 2883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.318679+0000 mon.a (mon.0) 2883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.319891+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: audit 2024-04-26T16:30:31.319891+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.625323+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.625323+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.627921+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.627921+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:30:32.396 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.717742+0000 mgr.y (mgr.24401) 510 : cluster [DBG] pgmap v777: 228 pgs: 8 creating+peering, 220 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T16:30:32.397 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:32 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.717742+0000 mgr.y (mgr.24401) 510 : cluster [DBG] pgmap v777: 228 pgs: 8 creating+peering, 220 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T16:30:32.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:32.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.226588+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.226588+0000 mon.a (mon.0) 2881 : cluster [DBG] osdmap e511: 8 total, 8 up, 7 in 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.317255+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.317255+0000 mon.a (mon.0) 2882 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.318179+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.318179+0000 mon.b (mon.1) 349 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.318679+0000 mon.a (mon.0) 2883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.318679+0000 mon.a (mon.0) 2883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.319891+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: audit 2024-04-26T16:30:31.319891+0000 mon.b (mon.1) 350 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-75"}]: dispatch 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.625323+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.625323+0000 osd.0 (osd.0) 309 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:30:32.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.627921+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:30:32.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.627921+0000 osd.0 (osd.0) 310 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:30:32.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.717742+0000 mgr.y (mgr.24401) 510 : cluster [DBG] pgmap v777: 228 pgs: 8 creating+peering, 220 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T16:30:32.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:32 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.717742+0000 mgr.y (mgr.24401) 510 : cluster [DBG] pgmap v777: 228 pgs: 8 creating+peering, 220 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-04-26T16:30:32.551 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:32.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:32.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:32.683 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:32.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:32.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:32.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:32.945 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:32.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:33.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:33.073 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:33.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:33.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:33.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:33.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:33.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:33.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.538226+0000 osd.4 (osd.4) 281 : cluster [DBG] 109.a scrub starts 2024-04-26T16:30:33.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.538226+0000 osd.4 (osd.4) 281 : cluster [DBG] 109.a scrub starts 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.542101+0000 osd.4 (osd.4) 282 : cluster [DBG] 109.a scrub ok 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.542101+0000 osd.4 (osd.4) 282 : cluster [DBG] 109.a scrub ok 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.613317+0000 osd.2 (osd.2) 265 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.613317+0000 osd.2 (osd.2) 265 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.618707+0000 osd.2 (osd.2) 266 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:31.618707+0000 osd.2 (osd.2) 266 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:32.241174+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[21463]: cluster 2024-04-26T16:30:32.241174+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.538226+0000 osd.4 (osd.4) 281 : cluster [DBG] 109.a scrub starts 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.538226+0000 osd.4 (osd.4) 281 : cluster [DBG] 109.a scrub starts 2024-04-26T16:30:33.450 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.542101+0000 osd.4 (osd.4) 282 : cluster [DBG] 109.a scrub ok 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.542101+0000 osd.4 (osd.4) 282 : cluster [DBG] 109.a scrub ok 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.613317+0000 osd.2 (osd.2) 265 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.613317+0000 osd.2 (osd.2) 265 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.618707+0000 osd.2 (osd.2) 266 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:31.618707+0000 osd.2 (osd.2) 266 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:32.241174+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2024-04-26T16:30:33.451 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:33 smithi050 bash[29234]: cluster 2024-04-26T16:30:32.241174+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2024-04-26T16:30:33.466 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:33.535 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T16:30:33.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:30:33.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.538226+0000 osd.4 (osd.4) 281 : cluster [DBG] 109.a scrub starts 2024-04-26T16:30:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.538226+0000 osd.4 (osd.4) 281 : cluster [DBG] 109.a scrub starts 2024-04-26T16:30:33.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.542101+0000 osd.4 (osd.4) 282 : cluster [DBG] 109.a scrub ok 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.542101+0000 osd.4 (osd.4) 282 : cluster [DBG] 109.a scrub ok 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.613317+0000 osd.2 (osd.2) 265 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.613317+0000 osd.2 (osd.2) 265 : cluster [DBG] 109.6 scrub starts 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.618707+0000 osd.2 (osd.2) 266 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:31.618707+0000 osd.2 (osd.2) 266 : cluster [DBG] 109.6 scrub ok 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:32.241174+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2024-04-26T16:30:33.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:33 smithi190 bash[23868]: cluster 2024-04-26T16:30:32.241174+0000 mon.a (mon.0) 2884 : cluster [DBG] osdmap e512: 8 total, 8 up, 7 in 2024-04-26T16:30:33.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:33.712 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:33.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:33.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:33.862 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:33.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:33.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:33.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:34.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:34.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:34.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:34.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:34.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:32.978606+0000 osd.3 (osd.3) 333 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:32.978606+0000 osd.3 (osd.3) 333 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:32.981341+0000 osd.3 (osd.3) 334 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:32.981341+0000 osd.3 (osd.3) 334 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.248045+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.248045+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.249222+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.249222+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: audit 2024-04-26T16:30:33.264570+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: audit 2024-04-26T16:30:33.264570+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: audit 2024-04-26T16:30:33.265203+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: audit 2024-04-26T16:30:33.265203+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.268013+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.268013+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.687965+0000 osd.0 (osd.0) 311 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.687965+0000 osd.0 (osd.0) 311 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.689560+0000 osd.0 (osd.0) 312 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.689560+0000 osd.0 (osd.0) 312 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.718277+0000 mgr.y (mgr.24401) 511 : cluster [DBG] pgmap v780: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.718277+0000 mgr.y (mgr.24401) 511 : cluster [DBG] pgmap v780: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:32.978606+0000 osd.3 (osd.3) 333 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:30:34.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:32.978606+0000 osd.3 (osd.3) 333 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:32.981341+0000 osd.3 (osd.3) 334 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:32.981341+0000 osd.3 (osd.3) 334 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.248045+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.248045+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.249222+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.249222+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: audit 2024-04-26T16:30:33.264570+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: audit 2024-04-26T16:30:33.264570+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: audit 2024-04-26T16:30:33.265203+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: audit 2024-04-26T16:30:33.265203+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.268013+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.268013+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:30:34.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.687965+0000 osd.0 (osd.0) 311 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:30:34.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.687965+0000 osd.0 (osd.0) 311 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:30:34.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.689560+0000 osd.0 (osd.0) 312 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:30:34.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.689560+0000 osd.0 (osd.0) 312 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:30:34.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.718277+0000 mgr.y (mgr.24401) 511 : cluster [DBG] pgmap v780: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:30:34.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.718277+0000 mgr.y (mgr.24401) 511 : cluster [DBG] pgmap v780: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:30:34.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:34.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:34.529 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:32.978606+0000 osd.3 (osd.3) 333 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:32.978606+0000 osd.3 (osd.3) 333 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:32.981341+0000 osd.3 (osd.3) 334 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:32.981341+0000 osd.3 (osd.3) 334 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.248045+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.248045+0000 osd.6 (osd.6) 47 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.249222+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-04-26T16:30:34.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.249222+0000 mon.a (mon.0) 2885 : cluster [DBG] osdmap e513: 8 total, 8 up, 7 in 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: audit 2024-04-26T16:30:33.264570+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: audit 2024-04-26T16:30:33.264570+0000 mon.a (mon.0) 2886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: audit 2024-04-26T16:30:33.265203+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: audit 2024-04-26T16:30:33.265203+0000 mon.b (mon.1) 351 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.268013+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.268013+0000 osd.6 (osd.6) 48 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.687965+0000 osd.0 (osd.0) 311 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.687965+0000 osd.0 (osd.0) 311 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.689560+0000 osd.0 (osd.0) 312 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.689560+0000 osd.0 (osd.0) 312 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.718277+0000 mgr.y (mgr.24401) 511 : cluster [DBG] pgmap v780: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:30:34.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:34 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.718277+0000 mgr.y (mgr.24401) 511 : cluster [DBG] pgmap v780: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:30:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:34.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:34.663 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:34.667 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:30:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:30:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:30:34.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:34.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:34.897 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:34.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:34.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:35.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:35.051 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:35.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:35.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:35.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:35.291 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:35.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.600856+0000 osd.2 (osd.2) 267 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:30:35.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.600856+0000 osd.2 (osd.2) 267 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.603278+0000 osd.2 (osd.2) 268 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:33.603278+0000 osd.2 (osd.2) 268 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.001788+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.001788+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.002974+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.002974+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.031645+0000 osd.1 (osd.1) 285 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.031645+0000 osd.1 (osd.1) 285 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.033278+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.033278+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.246334+0000 mon.a (mon.0) 2887 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.246334+0000 mon.a (mon.0) 2887 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:35.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: audit 2024-04-26T16:30:34.247862+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: audit 2024-04-26T16:30:34.247862+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.256885+0000 mon.a (mon.0) 2889 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.256885+0000 mon.a (mon.0) 2889 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: audit 2024-04-26T16:30:34.264602+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: audit 2024-04-26T16:30:34.264602+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.485570+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.485570+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.485923+0000 osd.7 (osd.7) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:35 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.485923+0000 osd.7 (osd.7) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:35.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:35.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.600856+0000 osd.2 (osd.2) 267 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:30:35.432 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.600856+0000 osd.2 (osd.2) 267 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.603278+0000 osd.2 (osd.2) 268 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:33.603278+0000 osd.2 (osd.2) 268 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.001788+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.001788+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.002974+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.002974+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.031645+0000 osd.1 (osd.1) 285 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.031645+0000 osd.1 (osd.1) 285 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.033278+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.033278+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.246334+0000 mon.a (mon.0) 2887 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.246334+0000 mon.a (mon.0) 2887 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: audit 2024-04-26T16:30:34.247862+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:35.433 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: audit 2024-04-26T16:30:34.247862+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.256885+0000 mon.a (mon.0) 2889 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.256885+0000 mon.a (mon.0) 2889 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: audit 2024-04-26T16:30:34.264602+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: audit 2024-04-26T16:30:34.264602+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.485570+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.485570+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.485923+0000 osd.7 (osd.7) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.485923+0000 osd.7 (osd.7) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:35.434 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.600856+0000 osd.2 (osd.2) 267 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.600856+0000 osd.2 (osd.2) 267 : cluster [DBG] 109.1e scrub starts 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.603278+0000 osd.2 (osd.2) 268 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:33.603278+0000 osd.2 (osd.2) 268 : cluster [DBG] 109.1e scrub ok 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.001788+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.001788+0000 osd.7 (osd.7) 301 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.002974+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.002974+0000 osd.7 (osd.7) 302 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.031645+0000 osd.1 (osd.1) 285 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.031645+0000 osd.1 (osd.1) 285 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.033278+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.033278+0000 osd.1 (osd.1) 286 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.246334+0000 mon.a (mon.0) 2887 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.246334+0000 mon.a (mon.0) 2887 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:35.435 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: audit 2024-04-26T16:30:34.247862+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: audit 2024-04-26T16:30:34.247862+0000 mon.a (mon.0) 2888 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.256885+0000 mon.a (mon.0) 2889 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.256885+0000 mon.a (mon.0) 2889 : cluster [DBG] osdmap e514: 8 total, 8 up, 7 in 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: audit 2024-04-26T16:30:34.264602+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: audit 2024-04-26T16:30:34.264602+0000 mon.b (mon.1) 352 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.485570+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.485570+0000 osd.7 (osd.7) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.485923+0000 osd.7 (osd.7) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:35.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:35 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.485923+0000 osd.7 (osd.7) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:35.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:35.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:35.579 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:35.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:35.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:35.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:35.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:35.843 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:35.946 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:35.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:35.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:36.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:36.105 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:36.208 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:36.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:36.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:36.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.979799+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.b scrub starts 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.979799+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.b scrub starts 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.982471+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.b scrub ok 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:34.982471+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.b scrub ok 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.049494+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.9 deep-scrub starts 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.049494+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.9 deep-scrub starts 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.051750+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.9 deep-scrub ok 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.051750+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.9 deep-scrub ok 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.278466+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.278466+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-04-26T16:30:36.357 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.351036+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.351036+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.351960+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.351960+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.352728+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.352728+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.353583+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: audit 2024-04-26T16:30:35.353583+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.718745+0000 mgr.y (mgr.24401) 512 : cluster [DBG] pgmap v783: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.718745+0000 mgr.y (mgr.24401) 512 : cluster [DBG] pgmap v783: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:36.220827+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-04-26T16:30:36.358 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[23868]: cluster 2024-04-26T16:30:36.220827+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-04-26T16:30:36.371 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:36.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:36.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:36.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.979799+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.b scrub starts 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.979799+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.b scrub starts 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.982471+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.b scrub ok 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:34.982471+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.b scrub ok 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.049494+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.9 deep-scrub starts 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.049494+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.9 deep-scrub starts 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.051750+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.9 deep-scrub ok 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.051750+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.9 deep-scrub ok 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.278466+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.278466+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-04-26T16:30:36.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.351036+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.351036+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.351960+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.351960+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.352728+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.352728+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.353583+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: audit 2024-04-26T16:30:35.353583+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.718745+0000 mgr.y (mgr.24401) 512 : cluster [DBG] pgmap v783: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.718745+0000 mgr.y (mgr.24401) 512 : cluster [DBG] pgmap v783: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:36.220827+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[21463]: cluster 2024-04-26T16:30:36.220827+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-04-26T16:30:36.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.979799+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.b scrub starts 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.979799+0000 osd.3 (osd.3) 335 : cluster [DBG] 3.b scrub starts 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.982471+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.b scrub ok 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:34.982471+0000 osd.3 (osd.3) 336 : cluster [DBG] 3.b scrub ok 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.049494+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.9 deep-scrub starts 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.049494+0000 osd.1 (osd.1) 287 : cluster [DBG] 2.9 deep-scrub starts 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.051750+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.9 deep-scrub ok 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.051750+0000 osd.1 (osd.1) 288 : cluster [DBG] 2.9 deep-scrub ok 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.278466+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.278466+0000 mon.a (mon.0) 2890 : cluster [DBG] osdmap e515: 8 total, 8 up, 7 in 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.351036+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.351036+0000 mon.a (mon.0) 2891 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.351960+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.351960+0000 mon.b (mon.1) 353 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:30:36.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.352728+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.352728+0000 mon.a (mon.0) 2892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.353583+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: audit 2024-04-26T16:30:35.353583+0000 mon.b (mon.1) 354 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-77"}]: dispatch 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.718745+0000 mgr.y (mgr.24401) 512 : cluster [DBG] pgmap v783: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.718745+0000 mgr.y (mgr.24401) 512 : cluster [DBG] pgmap v783: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:36.220827+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-04-26T16:30:36.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:36 smithi050 bash[29234]: cluster 2024-04-26T16:30:36.220827+0000 mon.a (mon.0) 2893 : cluster [DBG] osdmap e516: 8 total, 8 up, 7 in 2024-04-26T16:30:36.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:36.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:36.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:36.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- 172.21.15.50:0/1065517066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 msgr2=0x7fd6a406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:36.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/1065517066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a406b590 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7fd694009990 tx=0x7fd694009ca0 comp rx=0 tx=0).stop 2024-04-26T16:30:36.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- 172.21.15.50:0/1065517066 shutdown_connections 2024-04-26T16:30:36.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/1065517066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a41094a0 0x7fd6a4109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/1065517066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/1065517066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a406b590 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- 172.21.15.50:0/1065517066 >> 172.21.15.50:0/1065517066 conn(0x7fd6a40792d0 msgr2=0x7fd6a40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- 172.21.15.50:0/1065517066 shutdown_connections 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- 172.21.15.50:0/1065517066 wait complete. 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 librados: starting msgr at 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 librados: starting objecter 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 Processor -- start 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- start start 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 librados: setting wanted keys 2024-04-26T16:30:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 librados: calling monclient init 2024-04-26T16:30:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a4199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a4199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a41094a0 0x7fd6a419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a4112ae0 con 0x7fd6a406bad0 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a4112960 con 0x7fd6a41094a0 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.855+0000 7fd6abbf8700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a4112c60 con 0x7fd6a406b130 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a4199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9193700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a4199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a4199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53582/0 (socket says 172.21.15.50:53582) 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 -- 172.21.15.50:0/340995898 learned_addr learned my addr 172.21.15.50:0/340995898 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9193700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a4199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56336/0 (socket says 172.21.15.50:56336) 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6aa195700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a41094a0 0x7fd6a419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 -- 172.21.15.50:0/340995898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a41094a0 msgr2=0x7fd6a419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a41094a0 0x7fd6a419dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 -- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 msgr2=0x7fd6a4199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a4199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 -- 172.21.15.50:0/340995898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a419e200 con 0x7fd6a406b130 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9193700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a4199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9994700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a4199240 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00060b0 tx=0x7fd6a000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 10 client.?.objecter ms_handle_connect 0x7fd6a406b130 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a000ddd0 con 0x7fd6a406b130 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd6a00129b0 con 0x7fd6a406b130 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a419e4f0 con 0x7fd6a406b130 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a001d990 con 0x7fd6a406b130 2024-04-26T16:30:36.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6abbf8700 10 client.41321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:36.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a4102320 con 0x7fd6a406b130 2024-04-26T16:30:36.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6abbf8700 1 librados: init done 2024-04-26T16:30:36.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a406a110 con 0x7fd6a406b130 2024-04-26T16:30:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd6a0012550 con 0x7fd6a406b130 2024-04-26T16:30:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd690071f80 0x7fd690074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(516..516 src has 378..516) ==== 8992+0+0 (secure 0 0 0) 0x7fd6a00957c0 con 0x7fd6a406b130 2024-04-26T16:30:36.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 10 client.41321.objecter ms_dispatch 0x7fd6a4001550 osd_map(516..516 src has 378..516) 2024-04-26T16:30:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 3 client.41321.objecter handle_osd_map got epochs [516,516] > 0 2024-04-26T16:30:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 3 client.41321.objecter handle_osd_map decoding full epoch 516 2024-04-26T16:30:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9193700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd690071f80 0x7fd690074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 20 client.41321.objecter dump_active .. 0 homeless 2024-04-26T16:30:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd69affd700 10 client.41321.objecter ms_handle_connect 0x7fd690071f80 2024-04-26T16:30:36.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.859+0000 7fd6a9193700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd690071f80 0x7fd690074440 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 crypto rx=0x7fd69400b2e0 tx=0x7fd69401a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:36.865 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:36.867+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd6a0061300 con 0x7fd6a406b130 2024-04-26T16:30:36.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:36.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:36.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:37.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:37.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:37.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.103+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fd6a41025e0 con 0x7fd6a406b130 2024-04-26T16:30:37.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:37.132 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:37.135 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:37.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.215+0000 7fd69affd700 1 -- 172.21.15.50:0/340995898 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v517) ==== 87+0+0 (secure 0 0 0) 0x7fd6a0064fb0 con 0x7fd6a406b130 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 10 librados: watch_flush enter 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 10 librados: watch_flush exit 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 20 client.41321.objecter shutdown clearing up homeless session... 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 10 client.41321.objecter successfully canceled tick 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd690071f80 msgr2=0x7fd690074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd690071f80 0x7fd690074440 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 crypto rx=0x7fd69400b2e0 tx=0x7fd69401a040 comp rx=0 tx=0).stop 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 msgr2=0x7fd6a4199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a4199240 secure :-1 s=READY pgs=2685 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00060b0 tx=0x7fd6a000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 shutdown_connections 2024-04-26T16:30:37.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd690071f80 0x7fd690074440 unknown :-1 s=CLOSED pgs=3765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd6a41094a0 0x7fd6a419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd6a406b130 0x7fd6a4199240 unknown :-1 s=CLOSED pgs=2685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 --2- 172.21.15.50:0/340995898 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd6a406bad0 0x7fd6a4199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 >> 172.21.15.50:0/340995898 conn(0x7fd6a40792d0 msgr2=0x7fd6a4100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 shutdown_connections 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 -- 172.21.15.50:0/340995898 wait complete. 2024-04-26T16:30:37.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:37.223+0000 7fd6abbf8700 1 librados: shutdown 2024-04-26T16:30:37.226 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:30:37.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:37.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:37.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:30:36 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:30:37.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:37.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:37.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.935540+0000 osd.3 (osd.3) 337 : cluster [DBG] 6.c scrub starts 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.935540+0000 osd.3 (osd.3) 337 : cluster [DBG] 6.c scrub starts 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.938116+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub ok 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:35.938116+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub ok 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:36.169496+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:36.169496+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:36.171446+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:36.171446+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.107867+0000 mon.c (mon.2) 649 : audit [INF] from='client.? 172.21.15.50:0/340995898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.107867+0000 mon.c (mon.2) 649 : audit [INF] from='client.? 172.21.15.50:0/340995898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.108346+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.108346+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.499 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:37.215142+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:37.215142+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.217543+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.217543+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:37.225465+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: cluster 2024-04-26T16:30:37.225465+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.233812+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.233812+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.234898+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[29234]: audit 2024-04-26T16:30:37.234898+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.935540+0000 osd.3 (osd.3) 337 : cluster [DBG] 6.c scrub starts 2024-04-26T16:30:37.500 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.935540+0000 osd.3 (osd.3) 337 : cluster [DBG] 6.c scrub starts 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.938116+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub ok 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:35.938116+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub ok 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:36.169496+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:36.169496+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:36.171446+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:36.171446+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.107867+0000 mon.c (mon.2) 649 : audit [INF] from='client.? 172.21.15.50:0/340995898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.107867+0000 mon.c (mon.2) 649 : audit [INF] from='client.? 172.21.15.50:0/340995898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.108346+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.108346+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:37.215142+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:37.215142+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.217543+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:30:37.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.217543+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:30:37.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:37.225465+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-04-26T16:30:37.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: cluster 2024-04-26T16:30:37.225465+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-04-26T16:30:37.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.233812+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.233812+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.234898+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:37 smithi050 bash[21463]: audit 2024-04-26T16:30:37.234898+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.935540+0000 osd.3 (osd.3) 337 : cluster [DBG] 6.c scrub starts 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.935540+0000 osd.3 (osd.3) 337 : cluster [DBG] 6.c scrub starts 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.938116+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub ok 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:35.938116+0000 osd.3 (osd.3) 338 : cluster [DBG] 6.c scrub ok 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:36.169496+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:36.169496+0000 osd.6 (osd.6) 49 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:36.171446+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:36.171446+0000 osd.6 (osd.6) 50 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.107867+0000 mon.c (mon.2) 649 : audit [INF] from='client.? 172.21.15.50:0/340995898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.107867+0000 mon.c (mon.2) 649 : audit [INF] from='client.? 172.21.15.50:0/340995898' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.108346+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.108346+0000 mon.a (mon.0) 2894 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:37.215142+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:37.215142+0000 mon.a (mon.0) 2895 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.217543+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.217543+0000 mon.a (mon.0) 2896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:37.225465+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: cluster 2024-04-26T16:30:37.225465+0000 mon.a (mon.0) 2897 : cluster [DBG] osdmap e517: 8 total, 8 up, 7 in 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.233812+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.233812+0000 mon.a (mon.0) 2898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.234898+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:37 smithi190 bash[23868]: audit 2024-04-26T16:30:37.234898+0000 mon.b (mon.1) 355 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:30:37.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:37.646 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:37.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:37.775 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:37.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:37.906 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:37.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:38.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:38.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:38.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:38.171 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:38.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:38.298 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: audit 2024-04-26T16:30:36.968205+0000 mgr.y (mgr.24401) 513 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: audit 2024-04-26T16:30:36.968205+0000 mgr.y (mgr.24401) 513 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: cluster 2024-04-26T16:30:37.719581+0000 mgr.y (mgr.24401) 514 : cluster [DBG] pgmap v786: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: cluster 2024-04-26T16:30:37.719581+0000 mgr.y (mgr.24401) 514 : cluster [DBG] pgmap v786: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:38.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: audit 2024-04-26T16:30:38.220940+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: audit 2024-04-26T16:30:38.220940+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: cluster 2024-04-26T16:30:38.229190+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: cluster 2024-04-26T16:30:38.229190+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: audit 2024-04-26T16:30:38.237145+0000 mon.b (mon.1) 356 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[21463]: audit 2024-04-26T16:30:38.237145+0000 mon.b (mon.1) 356 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: audit 2024-04-26T16:30:36.968205+0000 mgr.y (mgr.24401) 513 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: audit 2024-04-26T16:30:36.968205+0000 mgr.y (mgr.24401) 513 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: cluster 2024-04-26T16:30:37.719581+0000 mgr.y (mgr.24401) 514 : cluster [DBG] pgmap v786: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: cluster 2024-04-26T16:30:37.719581+0000 mgr.y (mgr.24401) 514 : cluster [DBG] pgmap v786: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: audit 2024-04-26T16:30:38.220940+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:38.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: audit 2024-04-26T16:30:38.220940+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:38.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: cluster 2024-04-26T16:30:38.229190+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-04-26T16:30:38.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: cluster 2024-04-26T16:30:38.229190+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-04-26T16:30:38.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: audit 2024-04-26T16:30:38.237145+0000 mon.b (mon.1) 356 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:38.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:38 smithi050 bash[29234]: audit 2024-04-26T16:30:38.237145+0000 mon.b (mon.1) 356 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:38.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: audit 2024-04-26T16:30:36.968205+0000 mgr.y (mgr.24401) 513 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: audit 2024-04-26T16:30:36.968205+0000 mgr.y (mgr.24401) 513 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: cluster 2024-04-26T16:30:37.719581+0000 mgr.y (mgr.24401) 514 : cluster [DBG] pgmap v786: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: cluster 2024-04-26T16:30:37.719581+0000 mgr.y (mgr.24401) 514 : cluster [DBG] pgmap v786: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: audit 2024-04-26T16:30:38.220940+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:38.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: audit 2024-04-26T16:30:38.220940+0000 mon.a (mon.0) 2899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:30:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: cluster 2024-04-26T16:30:38.229190+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-04-26T16:30:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: cluster 2024-04-26T16:30:38.229190+0000 mon.a (mon.0) 2900 : cluster [DBG] osdmap e518: 8 total, 8 up, 7 in 2024-04-26T16:30:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: audit 2024-04-26T16:30:38.237145+0000 mon.b (mon.1) 356 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:38.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:38 smithi190 bash[23868]: audit 2024-04-26T16:30:38.237145+0000 mon.b (mon.1) 356 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:30:38.427 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:38.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:38.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:38.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:38.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:38.823 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:38.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:38.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:38.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:38.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:39.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:39.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:39.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:39.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:39.216 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:39.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:39.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:39.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:39.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:39.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:39.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:39.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:39.739 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:39.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:39.869 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:39.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:39.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:40.000 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:40.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:40.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:40.131 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:40.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:40.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:40 smithi190 bash[23868]: cluster 2024-04-26T16:30:39.231733+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-04-26T16:30:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:40 smithi190 bash[23868]: cluster 2024-04-26T16:30:39.231733+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-04-26T16:30:40.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:40 smithi190 bash[23868]: cluster 2024-04-26T16:30:39.720397+0000 mgr.y (mgr.24401) 515 : cluster [DBG] pgmap v789: 228 pgs: 2 creating+peering, 30 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:40.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:40 smithi190 bash[23868]: cluster 2024-04-26T16:30:39.720397+0000 mgr.y (mgr.24401) 515 : cluster [DBG] pgmap v789: 228 pgs: 2 creating+peering, 30 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:40.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:40.394 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:40.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:40.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[21463]: cluster 2024-04-26T16:30:39.231733+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[21463]: cluster 2024-04-26T16:30:39.231733+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[21463]: cluster 2024-04-26T16:30:39.720397+0000 mgr.y (mgr.24401) 515 : cluster [DBG] pgmap v789: 228 pgs: 2 creating+peering, 30 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[21463]: cluster 2024-04-26T16:30:39.720397+0000 mgr.y (mgr.24401) 515 : cluster [DBG] pgmap v789: 228 pgs: 2 creating+peering, 30 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[29234]: cluster 2024-04-26T16:30:39.231733+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[29234]: cluster 2024-04-26T16:30:39.231733+0000 mon.a (mon.0) 2901 : cluster [DBG] osdmap e519: 8 total, 8 up, 7 in 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[29234]: cluster 2024-04-26T16:30:39.720397+0000 mgr.y (mgr.24401) 515 : cluster [DBG] pgmap v789: 228 pgs: 2 creating+peering, 30 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:40.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:40 smithi050 bash[29234]: cluster 2024-04-26T16:30:39.720397+0000 mgr.y (mgr.24401) 515 : cluster [DBG] pgmap v789: 228 pgs: 2 creating+peering, 30 unknown, 196 active+clean; 8.4 MiB data, 511 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-04-26T16:30:40.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:40.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:40.654 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:40.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:40.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:40.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- 172.21.15.50:0/2736484814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 msgr2=0x7f6274110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:40.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- 172.21.15.50:0/2736484814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f6274110330 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f626c00b0d0 tx=0x7f626c00b3e0 comp rx=0 tx=0).stop 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- 172.21.15.50:0/2736484814 shutdown_connections 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- 172.21.15.50:0/2736484814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6274109560 0x7f6274109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- 172.21.15.50:0/2736484814 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f6274110330 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- 172.21.15.50:0/2736484814 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62741034c0 0x7f6274103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- 172.21.15.50:0/2736484814 >> 172.21.15.50:0/2736484814 conn(0x7f627407a5d0 msgr2=0x7f627407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- 172.21.15.50:0/2736484814 shutdown_connections 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- 172.21.15.50:0/2736484814 wait complete. 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 librados: starting msgr at 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 librados: starting objecter 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 Processor -- start 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- start start 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 librados: setting wanted keys 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 librados: calling monclient init 2024-04-26T16:30:40.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62741034c0 0x7f6274199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f62741995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 0x7f627419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6274112b40 con 0x7f6274103e80 2024-04-26T16:30:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f62741129c0 con 0x7f62741034c0 2024-04-26T16:30:40.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.811+0000 7f627c76e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6274112cc0 con 0x7f6274109560 2024-04-26T16:30:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f6279d09700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f62741995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 0x7f627419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f6279d09700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f62741995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53168/0 (socket says 172.21.15.50:53168) 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f6279d09700 1 -- 172.21.15.50:0/2165162500 learned_addr learned my addr 172.21.15.50:0/2165162500 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 0x7f627419d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44496/0 (socket says 172.21.15.50:44496) 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62741034c0 0x7f6274199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62741034c0 msgr2=0x7f6274199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62741034c0 0x7f6274199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 msgr2=0x7f62741995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f62741995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627419e090 con 0x7f6274109560 2024-04-26T16:30:40.816 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6274103e80 0x7f62741995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62741034c0 0x7f6274199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 0x7f627419d950 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f626c00b0d0 tx=0x7f626c007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 10 client.?.objecter ms_handle_connect 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c01f030 con 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627419e320 con 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f626c005050 con 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c01b690 con 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 10 client.41327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62741a5c00 con 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 1 librados: init done 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f6274104bd0 con 0x7f6274109560 2024-04-26T16:30:40.817 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f626c022030 con 0x7f6274109560 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6260071f80 0x7f6260074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(520..520 src has 378..520) ==== 9377+0+0 (secure 0 0 0) 0x7f626c00f020 con 0x7f6274109560 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 osd_map(520..520 src has 378..520) 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 3 client.41327.objecter handle_osd_map got epochs [520,520] > 0 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 3 client.41327.objecter handle_osd_map decoding full epoch 520 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6260071f80 0x7f6260074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 20 client.41327.objecter dump_active .. 0 homeless 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v520) ==== 54+0+16 (secure 0 0 0) 0x7f626c029020 con 0x7f6274109560 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f6260071f80 2024-04-26T16:30:40.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 10 client.41327.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6260071f80 0x7f6260074440 secure :-1 s=READY pgs=3766 cs=0 l=1 rev1=1 crypto rx=0x7f6270005950 tx=0x7f6270005f50 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 1 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6274064030 0x7f6274064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 1 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 0 1 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 4 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 10 client.41327.objecter _send_command 1 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f627419a3e0 con 0x7f6274064030 2024-04-26T16:30:40.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.815+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6274064030 0x7f6274064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6274064030 0x7f6274064410 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f6274064030 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f627419a3e0 con 0x7f6274064030 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 1: 0 ) 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f626b7fe700 10 client.41327.objecter _finish_command 1 = system:0 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 4 2024-04-26T16:30:40.821 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.819+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 0 1 2024-04-26T16:30:40.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 10 client.41327.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 2 2024-04-26T16:30:40.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 2 2024-04-26T16:30:40.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627406a110 osd=0 3 2024-04-26T16:30:40.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 0 2 2024-04-26T16:30:40.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 4 2024-04-26T16:30:40.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 10 client.41327.objecter _send_command 2 2024-04-26T16:30:40.837 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.835+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f6274063ab0 con 0x7f6274064030 2024-04-26T16:30:40.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f6274063ab0 con 0x7f6274064030 2024-04-26T16:30:40.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 10 client.41327.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 4 2024-04-26T16:30:40.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 0 2 2024-04-26T16:30:40.838 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 10 client.41327.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 3 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f627410bff0 0x7f62741ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 3 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 1 3 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 4 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 10 client.41327.objecter _send_command 3 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f6274077660 con 0x7f627410bff0 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f627410bff0 0x7f62741ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f627410bff0 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f627410bff0 0x7f62741ac1f0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6274077660 con 0x7f627410bff0 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 3: 0 ) 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 10 client.41327.objecter _finish_command 3 = system:0 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 4 2024-04-26T16:30:40.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.839+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 1 3 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 10 client.41327.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 4 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 4 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627402e490 osd=1 3 2024-04-26T16:30:40.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 1 4 2024-04-26T16:30:40.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 4 2024-04-26T16:30:40.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 10 client.41327.objecter _send_command 4 2024-04-26T16:30:40.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f627402e850 con 0x7f627410bff0 2024-04-26T16:30:40.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f627402e850 con 0x7f627410bff0 2024-04-26T16:30:40.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 10 client.41327.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 4 2024-04-26T16:30:40.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 1 4 2024-04-26T16:30:40.859 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 10 client.41327.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 5 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f627410b6d0 0x7f62741af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 5 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 2 5 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 4 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 10 client.41327.objecter _send_command 5 2024-04-26T16:30:40.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f627402e080 con 0x7f627410b6d0 2024-04-26T16:30:40.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f627410b6d0 0x7f62741af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f627410b6d0 2024-04-26T16:30:40.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f627410b6d0 0x7f62741af8c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f627402e080 con 0x7f627410b6d0 2024-04-26T16:30:40.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 5: 0 ) 2024-04-26T16:30:40.861 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 10 client.41327.objecter _finish_command 5 = system:0 2024-04-26T16:30:40.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 4 2024-04-26T16:30:40.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.859+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 2 5 2024-04-26T16:30:40.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 10 client.41327.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 6 2024-04-26T16:30:40.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 6 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627410b560 osd=2 3 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 2 6 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 4 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 10 client.41327.objecter _send_command 6 2024-04-26T16:30:40.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f627402d560 con 0x7f627410b6d0 2024-04-26T16:30:40.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f627402d560 con 0x7f627410b6d0 2024-04-26T16:30:40.880 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f626b7fe700 10 client.41327.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 4 2024-04-26T16:30:40.881 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 2 6 2024-04-26T16:30:40.881 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:30:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 10 client.41327.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 7 2024-04-26T16:30:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f62741ad410 0x7f62741b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 7 2024-04-26T16:30:40.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 3 7 2024-04-26T16:30:40.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 4 2024-04-26T16:30:40.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 10 client.41327.objecter _send_command 7 2024-04-26T16:30:40.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f62741b5df0 con 0x7f62741ad410 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f62741ad410 0x7f62741b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f62741ad410 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.879+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f62741ad410 0x7f62741b2f90 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.883+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62741b5df0 con 0x7f62741ad410 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.883+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 7: 0 ) 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.883+0000 7f626b7fe700 10 client.41327.objecter _finish_command 7 = system:0 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.883+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 4 2024-04-26T16:30:40.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.883+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 3 7 2024-04-26T16:30:40.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 10 client.41327.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 8 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 8 2024-04-26T16:30:40.900 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f627402d7b0 osd=3 3 2024-04-26T16:30:40.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 3 8 2024-04-26T16:30:40.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 4 2024-04-26T16:30:40.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 10 client.41327.objecter _send_command 8 2024-04-26T16:30:40.901 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.899+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62741aca40 con 0x7f62741ad410 2024-04-26T16:30:40.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62741aca40 con 0x7f62741ad410 2024-04-26T16:30:40.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f626b7fe700 10 client.41327.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.902 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 4 2024-04-26T16:30:40.903 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 3 8 2024-04-26T16:30:40.903 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:30:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 10 client.41327.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 9 2024-04-26T16:30:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f62741b0ae0 0x7f62741b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.904 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 9 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 4 9 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 4 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 10 client.41327.objecter _send_command 9 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f62741b94c0 con 0x7f62741b0ae0 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f62741b0ae0 0x7f62741b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f62741b0ae0 2024-04-26T16:30:40.905 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.903+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f62741b0ae0 0x7f62741b6660 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:40.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.907+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62741b94c0 con 0x7f62741b0ae0 2024-04-26T16:30:40.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.907+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 9: 0 ) 2024-04-26T16:30:40.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.907+0000 7f626b7fe700 10 client.41327.objecter _finish_command 9 = system:0 2024-04-26T16:30:40.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.907+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 4 2024-04-26T16:30:40.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.907+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 4 9 2024-04-26T16:30:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:40.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 10 client.41327.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 10 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 10 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741accd0 osd=4 3 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 4 10 2024-04-26T16:30:40.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 4 2024-04-26T16:30:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 10 client.41327.objecter _send_command 10 2024-04-26T16:30:40.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.923+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62741ac850 con 0x7f62741b0ae0 2024-04-26T16:30:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62741ac850 con 0x7f62741b0ae0 2024-04-26T16:30:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 10 client.41327.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 4 2024-04-26T16:30:40.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 4 10 2024-04-26T16:30:40.928 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:30:40.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 10 client.41327.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 11 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f62741b3ca0 0x7f62741b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 11 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 5 11 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 4 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 10 client.41327.objecter _send_command 11 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f62741bc780 con 0x7f62741b3ca0 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f62741b3ca0 0x7f62741b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f62741b3ca0 2024-04-26T16:30:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f6279d09700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f62741b3ca0 0x7f62741b4100 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62741bc780 con 0x7f62741b3ca0 2024-04-26T16:30:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 11: 0 ) 2024-04-26T16:30:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 10 client.41327.objecter _finish_command 11 = system:0 2024-04-26T16:30:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 4 2024-04-26T16:30:40.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.931+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 5 11 2024-04-26T16:30:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 10 client.41327.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 12 2024-04-26T16:30:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.948 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 12 2024-04-26T16:30:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741ac850 osd=5 3 2024-04-26T16:30:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 5 12 2024-04-26T16:30:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 4 2024-04-26T16:30:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 10 client.41327.objecter _send_command 12 2024-04-26T16:30:40.949 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.947+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62741b9dd0 con 0x7f62741b3ca0 2024-04-26T16:30:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62741b9dd0 con 0x7f62741b3ca0 2024-04-26T16:30:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f626b7fe700 10 client.41327.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 4 2024-04-26T16:30:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 5 12 2024-04-26T16:30:40.953 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:30:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 10 client.41327.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 13 2024-04-26T16:30:40.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f62741ba4f0 0x7f62741ba930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 13 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 6 13 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 4 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 10 client.41327.objecter _send_command 13 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.951+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f62741bfba0 con 0x7f62741ba4f0 2024-04-26T16:30:40.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f62741ba4f0 0x7f62741ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f62741ba4f0 2024-04-26T16:30:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f627ad0b700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f62741ba4f0 0x7f62741ba930 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62741bfba0 con 0x7f62741ba4f0 2024-04-26T16:30:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 13: 0 ) 2024-04-26T16:30:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f626b7fe700 10 client.41327.objecter _finish_command 13 = system:0 2024-04-26T16:30:40.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 4 2024-04-26T16:30:40.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.955+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 6 13 2024-04-26T16:30:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 10 client.41327.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 14 2024-04-26T16:30:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.973 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 14 2024-04-26T16:30:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741b9dd0 osd=6 3 2024-04-26T16:30:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 6 14 2024-04-26T16:30:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 4 2024-04-26T16:30:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 10 client.41327.objecter _send_command 14 2024-04-26T16:30:40.974 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.975+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62741bfba0 con 0x7f62741ba4f0 2024-04-26T16:30:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62741bfba0 con 0x7f62741ba4f0 2024-04-26T16:30:40.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 10 client.41327.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 4 2024-04-26T16:30:40.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 6 14 2024-04-26T16:30:40.978 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 10 client.41327.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 15 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f62741bef40 0x7f62741bf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 15 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 7 15 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 4 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 10 client.41327.objecter _send_command 15 2024-04-26T16:30:40.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f62741c2f40 con 0x7f62741bef40 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f62741bef40 0x7f62741bf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 10 client.41327.objecter ms_handle_connect 0x7f62741bef40 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f627a50a700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f62741bef40 0x7f62741bf3a0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f62741c2f40 con 0x7f62741bef40 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 15: 0 ) 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 10 client.41327.objecter _finish_command 15 = system:0 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 4 2024-04-26T16:30:40.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.979+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 7 15 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 10 client.41327.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign -1 16 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 20 client.41327.objecter _get_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 15 client.41327.objecter _session_command_op_remove -1 16 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 20 client.41327.objecter get_session s=0x7f62741bfba0 osd=7 3 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 15 client.41327.objecter _session_command_op_assign 7 16 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 4 2024-04-26T16:30:40.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 10 client.41327.objecter _send_command 16 2024-04-26T16:30:41.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:40.999+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f62741baec0 con 0x7f62741bef40 2024-04-26T16:30:41.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f626b7fe700 1 -- 172.21.15.50:0/2165162500 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f62741baec0 con 0x7f62741bef40 2024-04-26T16:30:41.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f626b7fe700 10 client.41327.objecter ms_dispatch 0x7f6274001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:30:41.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f626b7fe700 10 client.41327.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:30:41.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f626b7fe700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 4 2024-04-26T16:30:41.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f626b7fe700 15 client.41327.objecter _session_command_op_remove 7 16 2024-04-26T16:30:41.004 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 librados: watch_flush enter 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 librados: watch_flush exit 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.0 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6274064030 msgr2=0x7f6274064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f6274064030 0x7f6274064410 crc :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627406a110 osd=0 1 2024-04-26T16:30:41.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.1 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f627410bff0 msgr2=0x7f62741ac1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f627410bff0 0x7f62741ac1f0 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402e490 osd=1 1 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.2 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f627410b6d0 msgr2=0x7f62741af8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f627410b6d0 0x7f62741af8c0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627410b560 osd=2 1 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.3 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f62741ad410 msgr2=0x7f62741b2f90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627ad0b700 1 -- 172.21.15.50:0/2165162500 reap_dead start 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f62741ad410 0x7f62741b2f90 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f627402d7b0 osd=3 1 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.4 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f62741b0ae0 msgr2=0x7f62741b6660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f62741b0ae0 0x7f62741b6660 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741accd0 osd=4 1 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.5 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f62741b3ca0 msgr2=0x7f62741b4100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f62741b3ca0 0x7f62741b4100 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741ac850 osd=5 1 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.6 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f62741ba4f0 msgr2=0x7f62741ba930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7f62741ba4f0 0x7f62741ba930 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741b9dd0 osd=6 1 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter close_session for osd.7 2024-04-26T16:30:41.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f62741bef40 msgr2=0x7f62741bf3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f62741bef40 0x7f62741bf3a0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter put_session s=0x7f62741bfba0 osd=7 1 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 20 client.41327.objecter shutdown clearing up homeless session... 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 10 client.41327.objecter successfully canceled tick 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627ad0b700 1 -- 172.21.15.50:0/2165162500 reap_dead start 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6260071f80 msgr2=0x7f6260074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6260071f80 0x7f6260074440 secure :-1 s=READY pgs=3766 cs=0 l=1 rev1=1 crypto rx=0x7f6270005950 tx=0x7f6270005f50 comp rx=0 tx=0).stop 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 msgr2=0x7f627419d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:41.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 0x7f627419d950 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f626c00b0d0 tx=0x7f626c007a20 comp rx=0 tx=0).stop 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 shutdown_connections 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6274109560 0x7f627419d950 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 --2- 172.21.15.50:0/2165162500 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6260071f80 0x7f6260074440 unknown :-1 s=CLOSED pgs=3766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 >> 172.21.15.50:0/2165162500 conn(0x7f627407a5d0 msgr2=0x7f627410a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 shutdown_connections 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 -- 172.21.15.50:0/2165162500 wait complete. 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:41.003+0000 7f627c76e700 1 librados: shutdown 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:30:41.009 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:30:41.010 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:30:41.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:41.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:41.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:41.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:41.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.223409+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.223409+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:41.238 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.237883+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.237883+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: audit 2024-04-26T16:30:40.273795+0000 mgr.y (mgr.24401) 516 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: audit 2024-04-26T16:30:40.273795+0000 mgr.y (mgr.24401) 516 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: audit 2024-04-26T16:30:40.274853+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: audit 2024-04-26T16:30:40.274853+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: audit 2024-04-26T16:30:40.821188+0000 mon.c (mon.2) 650 : audit [DBG] from='client.? 172.21.15.50:0/2165162500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: audit 2024-04-26T16:30:40.821188+0000 mon.c (mon.2) 650 : audit [DBG] from='client.? 172.21.15.50:0/2165162500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.878889+0000 osd.7 (osd.7) 305 : cluster [DBG] 305.7 deep-scrub starts 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.878889+0000 osd.7 (osd.7) 305 : cluster [DBG] 305.7 deep-scrub starts 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.884871+0000 osd.7 (osd.7) 306 : cluster [DBG] 305.7 deep-scrub ok 2024-04-26T16:30:41.239 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[21463]: cluster 2024-04-26T16:30:40.884871+0000 osd.7 (osd.7) 306 : cluster [DBG] 305.7 deep-scrub ok 2024-04-26T16:30:41.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:41.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.223409+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:41.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.223409+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:41.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.237883+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-04-26T16:30:41.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.237883+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-04-26T16:30:41.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: audit 2024-04-26T16:30:40.273795+0000 mgr.y (mgr.24401) 516 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.272 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: audit 2024-04-26T16:30:40.273795+0000 mgr.y (mgr.24401) 516 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: audit 2024-04-26T16:30:40.274853+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: audit 2024-04-26T16:30:40.274853+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: audit 2024-04-26T16:30:40.821188+0000 mon.c (mon.2) 650 : audit [DBG] from='client.? 172.21.15.50:0/2165162500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: audit 2024-04-26T16:30:40.821188+0000 mon.c (mon.2) 650 : audit [DBG] from='client.? 172.21.15.50:0/2165162500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.878889+0000 osd.7 (osd.7) 305 : cluster [DBG] 305.7 deep-scrub starts 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.878889+0000 osd.7 (osd.7) 305 : cluster [DBG] 305.7 deep-scrub starts 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.884871+0000 osd.7 (osd.7) 306 : cluster [DBG] 305.7 deep-scrub ok 2024-04-26T16:30:41.273 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:41 smithi050 bash[29234]: cluster 2024-04-26T16:30:40.884871+0000 osd.7 (osd.7) 306 : cluster [DBG] 305.7 deep-scrub ok 2024-04-26T16:30:41.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:41.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:41.416 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:41.519 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:41.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:41.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.223409+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.223409+0000 mon.a (mon.0) 2902 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.237883+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.237883+0000 mon.a (mon.0) 2903 : cluster [DBG] osdmap e520: 8 total, 8 up, 7 in 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: audit 2024-04-26T16:30:40.273795+0000 mgr.y (mgr.24401) 516 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: audit 2024-04-26T16:30:40.273795+0000 mgr.y (mgr.24401) 516 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: audit 2024-04-26T16:30:40.274853+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: audit 2024-04-26T16:30:40.274853+0000 mon.b (mon.1) 357 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "305.7"}]: dispatch 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: audit 2024-04-26T16:30:40.821188+0000 mon.c (mon.2) 650 : audit [DBG] from='client.? 172.21.15.50:0/2165162500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: audit 2024-04-26T16:30:40.821188+0000 mon.c (mon.2) 650 : audit [DBG] from='client.? 172.21.15.50:0/2165162500' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.878889+0000 osd.7 (osd.7) 305 : cluster [DBG] 305.7 deep-scrub starts 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.878889+0000 osd.7 (osd.7) 305 : cluster [DBG] 305.7 deep-scrub starts 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.884871+0000 osd.7 (osd.7) 306 : cluster [DBG] 305.7 deep-scrub ok 2024-04-26T16:30:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:41 smithi190 bash[23868]: cluster 2024-04-26T16:30:40.884871+0000 osd.7 (osd.7) 306 : cluster [DBG] 305.7 deep-scrub ok 2024-04-26T16:30:41.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:41.679 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:41.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:41.811 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:41.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:41.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:41.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:42.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:42.074 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:42.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:30:42.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:42.204 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:42.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:42.335 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:42.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:42.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:42.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.598 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:42.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:42.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:42.727 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:42.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:42.830 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:42.853 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:42.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:42.983 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:42.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:43.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.111 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:43.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[29234]: cluster 2024-04-26T16:30:41.722572+0000 mgr.y (mgr.24401) 517 : cluster [DBG] pgmap v791: 228 pgs: 2 creating+peering, 226 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 909 B/s wr, 2 op/s 2024-04-26T16:30:43.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[29234]: cluster 2024-04-26T16:30:41.722572+0000 mgr.y (mgr.24401) 517 : cluster [DBG] pgmap v791: 228 pgs: 2 creating+peering, 226 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 909 B/s wr, 2 op/s 2024-04-26T16:30:43.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[29234]: audit 2024-04-26T16:30:42.061898+0000 mon.a (mon.0) 2904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:43.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[29234]: audit 2024-04-26T16:30:42.061898+0000 mon.a (mon.0) 2904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:43.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[29234]: audit 2024-04-26T16:30:42.064123+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[29234]: audit 2024-04-26T16:30:42.064123+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[21463]: cluster 2024-04-26T16:30:41.722572+0000 mgr.y (mgr.24401) 517 : cluster [DBG] pgmap v791: 228 pgs: 2 creating+peering, 226 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 909 B/s wr, 2 op/s 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[21463]: cluster 2024-04-26T16:30:41.722572+0000 mgr.y (mgr.24401) 517 : cluster [DBG] pgmap v791: 228 pgs: 2 creating+peering, 226 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 909 B/s wr, 2 op/s 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[21463]: audit 2024-04-26T16:30:42.061898+0000 mon.a (mon.0) 2904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[21463]: audit 2024-04-26T16:30:42.061898+0000 mon.a (mon.0) 2904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[21463]: audit 2024-04-26T16:30:42.064123+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:43 smithi050 bash[21463]: audit 2024-04-26T16:30:42.064123+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:43.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:43.241 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:43.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:43 smithi190 bash[23868]: cluster 2024-04-26T16:30:41.722572+0000 mgr.y (mgr.24401) 517 : cluster [DBG] pgmap v791: 228 pgs: 2 creating+peering, 226 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 909 B/s wr, 2 op/s 2024-04-26T16:30:43.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:43 smithi190 bash[23868]: cluster 2024-04-26T16:30:41.722572+0000 mgr.y (mgr.24401) 517 : cluster [DBG] pgmap v791: 228 pgs: 2 creating+peering, 226 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 909 B/s wr, 2 op/s 2024-04-26T16:30:43.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:43 smithi190 bash[23868]: audit 2024-04-26T16:30:42.061898+0000 mon.a (mon.0) 2904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:43.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:43 smithi190 bash[23868]: audit 2024-04-26T16:30:42.061898+0000 mon.a (mon.0) 2904 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:30:43.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:43 smithi190 bash[23868]: audit 2024-04-26T16:30:42.064123+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:43.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:43 smithi190 bash[23868]: audit 2024-04-26T16:30:42.064123+0000 mon.c (mon.2) 651 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:43.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.370 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:43.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:43.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:43.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.634 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:43.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:43.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:43.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.898 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:43.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:43.911 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:44.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:44.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:44.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:44.159 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:44.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:44.289 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:44.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:44.422 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:44.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:44.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:44.579 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:44.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:44.695 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:30:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:30:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:30:44.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:44.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:44.843 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:44.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:44.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:44.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:45.078 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:45.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:45.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:45.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:45.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:45 smithi050 bash[21463]: cluster 2024-04-26T16:30:43.723724+0000 mgr.y (mgr.24401) 518 : cluster [DBG] pgmap v792: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-04-26T16:30:45.221 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:45 smithi050 bash[21463]: cluster 2024-04-26T16:30:43.723724+0000 mgr.y (mgr.24401) 518 : cluster [DBG] pgmap v792: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-04-26T16:30:45.221 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:45 smithi050 bash[29234]: cluster 2024-04-26T16:30:43.723724+0000 mgr.y (mgr.24401) 518 : cluster [DBG] pgmap v792: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-04-26T16:30:45.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:45 smithi050 bash[29234]: cluster 2024-04-26T16:30:43.723724+0000 mgr.y (mgr.24401) 518 : cluster [DBG] pgmap v792: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-04-26T16:30:45.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:45 smithi190 bash[23868]: cluster 2024-04-26T16:30:43.723724+0000 mgr.y (mgr.24401) 518 : cluster [DBG] pgmap v792: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-04-26T16:30:45.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:45 smithi190 bash[23868]: cluster 2024-04-26T16:30:43.723724+0000 mgr.y (mgr.24401) 518 : cluster [DBG] pgmap v792: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 852 B/s wr, 2 op/s 2024-04-26T16:30:45.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:45.364 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:45.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:30:45.468 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:45.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:45.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:45.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:45.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:45.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:45.773 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:45.833 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:30:45.833 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:30:45.833 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-26T16:30:45.833 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:30:45.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:45.903 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:45.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:46.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:46.034 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:46.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:46.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:46.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:46.285 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:46.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:46.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:46.415 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:46.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:46.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:46.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:46.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:46.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:46.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:46.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:46.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:46.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:46.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:47.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:47.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:47.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:47.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:47.210 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:47.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:47.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:30:46 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:45.724252+0000 mgr.y (mgr.24401) 519 : cluster [DBG] pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 682 B/s wr, 1 op/s 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:45.724252+0000 mgr.y (mgr.24401) 519 : cluster [DBG] pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 682 B/s wr, 1 op/s 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:46.147773+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:46.147773+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:46.148051+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:46.148051+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:47.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:47.072274+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-04-26T16:30:47.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:47 smithi190 bash[23868]: cluster 2024-04-26T16:30:47.072274+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-04-26T16:30:47.341 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:45.724252+0000 mgr.y (mgr.24401) 519 : cluster [DBG] pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 682 B/s wr, 1 op/s 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:45.724252+0000 mgr.y (mgr.24401) 519 : cluster [DBG] pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 682 B/s wr, 1 op/s 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:46.147773+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:46.147773+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:46.148051+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:46.148051+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:47.072274+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[21463]: cluster 2024-04-26T16:30:47.072274+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:45.724252+0000 mgr.y (mgr.24401) 519 : cluster [DBG] pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 682 B/s wr, 1 op/s 2024-04-26T16:30:47.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:45.724252+0000 mgr.y (mgr.24401) 519 : cluster [DBG] pgmap v793: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 682 B/s rd, 682 B/s wr, 1 op/s 2024-04-26T16:30:47.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:46.147773+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:47.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:46.147773+0000 osd.6 (osd.6) 51 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:47.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:46.148051+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:47.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:46.148051+0000 osd.6 (osd.6) 52 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:47.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:47.072274+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-04-26T16:30:47.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:47 smithi050 bash[29234]: cluster 2024-04-26T16:30:47.072274+0000 mon.a (mon.0) 2905 : cluster [DBG] osdmap e521: 8 total, 8 up, 7 in 2024-04-26T16:30:47.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-26T16:30:47.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:47.471 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:47.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:47.602 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:47.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:47.730 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:47.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:47.860 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:47.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:47.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:47.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:47.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:48.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:48.120 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:48.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:48.250 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:48.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:48.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:48 smithi050 bash[21463]: audit 2024-04-26T16:30:46.976496+0000 mgr.y (mgr.24401) 520 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:48.366 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:48 smithi050 bash[21463]: audit 2024-04-26T16:30:46.976496+0000 mgr.y (mgr.24401) 520 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:48.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:48 smithi050 bash[29234]: audit 2024-04-26T16:30:46.976496+0000 mgr.y (mgr.24401) 520 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:48.366 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:48 smithi050 bash[29234]: audit 2024-04-26T16:30:46.976496+0000 mgr.y (mgr.24401) 520 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:48.382 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:48.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:48.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:48 smithi190 bash[23868]: audit 2024-04-26T16:30:46.976496+0000 mgr.y (mgr.24401) 520 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:48.497 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:48 smithi190 bash[23868]: audit 2024-04-26T16:30:46.976496+0000 mgr.y (mgr.24401) 520 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:48.512 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:48.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:48.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:48.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:48.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:48.897 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:48.898 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:48.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:49.001 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:49.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:49.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:49.159 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:49.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:49.287 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:49.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:49 smithi050 bash[21463]: cluster 2024-04-26T16:30:47.725137+0000 mgr.y (mgr.24401) 521 : cluster [DBG] pgmap v795: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-04-26T16:30:49.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:49 smithi050 bash[21463]: cluster 2024-04-26T16:30:47.725137+0000 mgr.y (mgr.24401) 521 : cluster [DBG] pgmap v795: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-04-26T16:30:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:49 smithi050 bash[29234]: cluster 2024-04-26T16:30:47.725137+0000 mgr.y (mgr.24401) 521 : cluster [DBG] pgmap v795: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-04-26T16:30:49.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:49 smithi050 bash[29234]: cluster 2024-04-26T16:30:47.725137+0000 mgr.y (mgr.24401) 521 : cluster [DBG] pgmap v795: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-04-26T16:30:49.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:49.400 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:49 smithi190 bash[23868]: cluster 2024-04-26T16:30:47.725137+0000 mgr.y (mgr.24401) 521 : cluster [DBG] pgmap v795: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-04-26T16:30:49.401 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:49 smithi190 bash[23868]: cluster 2024-04-26T16:30:47.725137+0000 mgr.y (mgr.24401) 521 : cluster [DBG] pgmap v795: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 639 B/s wr, 2 op/s 2024-04-26T16:30:49.415 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:49.518 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:49.546 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:49.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:49.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:49.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:49.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:49.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:49.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:49.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:49.936 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:49.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:50.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:50.067 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:50.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:50.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:50.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:50.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:50.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/631209055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef8102350 msgr2=0x7f5ef81027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:50.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- 172.21.15.50:0/631209055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef8102350 0x7f5ef81027d0 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f5ee00098f0 tx=0x7f5ee0009c00 comp rx=0 tx=0).stop 2024-04-26T16:30:50.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/631209055 shutdown_connections 2024-04-26T16:30:50.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- 172.21.15.50:0/631209055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102d10 0x7f5ef810f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- 172.21.15.50:0/631209055 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5ef81083c0 0x7f5ef81087a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.314 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- 172.21.15.50:0/631209055 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef8102350 0x7f5ef81027d0 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/631209055 >> 172.21.15.50:0/631209055 conn(0x7f5ef8079360 msgr2=0x7f5ef8079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:50.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/631209055 shutdown_connections 2024-04-26T16:30:50.315 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/631209055 wait complete. 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 librados: starting msgr at 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 librados: starting objecter 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 Processor -- start 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- start start 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 librados: setting wanted keys 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 librados: calling monclient init 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102350 0x7f5ef8197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5ef8102d10 0x7f5ef8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:50.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 0x7f5ef819b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef8110960 con 0x7f5ef81083c0 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5ef81107e0 con 0x7f5ef8102350 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5ef8110ae0 con 0x7f5ef8102d10 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 0x7f5ef819b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5ef7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5ef8102d10 0x7f5ef8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 0x7f5ef819b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56994/0 (socket says 172.21.15.50:56994) 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 -- 172.21.15.50:0/4003954935 learned_addr learned my addr 172.21.15.50:0/4003954935 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efcbb0700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102350 0x7f5ef8197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 -- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5ef8102d10 msgr2=0x7f5ef8197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5ef8102d10 0x7f5ef8197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 -- 172.21.15.50:0/4003954935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102350 msgr2=0x7f5ef8197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102350 0x7f5ef8197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 -- 172.21.15.50:0/4003954935 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ef819c030 con 0x7f5ef81083c0 2024-04-26T16:30:50.318 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efcbb0700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102350 0x7f5ef8197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:30:50.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efd3b1700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 0x7f5ef819b8f0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0011400 tx=0x7f5ef0011710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5ef5ffb700 10 client.?.objecter ms_handle_connect 0x7f5ef81083c0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5ef5ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0003c00 con 0x7f5ef81083c0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5ef0003da0 con 0x7f5ef81083c0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0019660 con 0x7f5ef81083c0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ef819c320 con 0x7f5ef81083c0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 10 client.30717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ef81a3ba0 con 0x7f5ef81083c0 2024-04-26T16:30:50.319 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.318+0000 7f5efee14700 1 librados: init done 2024-04-26T16:30:50.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ef8103a70 con 0x7f5ef81083c0 2024-04-26T16:30:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:50.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5ef0004020 con 0x7f5ef81083c0 2024-04-26T16:30:50.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ee4071f80 0x7f5ee4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:50.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(521..521 src has 378..521) ==== 9377+0+0 (secure 0 0 0) 0x7f5ef0095910 con 0x7f5ef81083c0 2024-04-26T16:30:50.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 10 client.30717.objecter ms_dispatch 0x7f5ef8001550 osd_map(521..521 src has 378..521) 2024-04-26T16:30:50.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 3 client.30717.objecter handle_osd_map got epochs [521,521] > 0 2024-04-26T16:30:50.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 3 client.30717.objecter handle_osd_map decoding full epoch 521 2024-04-26T16:30:50.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5efcbb0700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ee4071f80 0x7f5ee4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:50.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 20 client.30717.objecter dump_active .. 0 homeless 2024-04-26T16:30:50.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5ef5ffb700 10 client.30717.objecter ms_handle_connect 0x7f5ee4071f80 2024-04-26T16:30:50.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.322+0000 7f5efcbb0700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ee4071f80 0x7f5ee4074440 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005fd0 tx=0x7f5ee8009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:50.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.326+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5ef0014070 con 0x7f5ef81083c0 2024-04-26T16:30:50.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:50.448 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:50.546 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.550+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5ef8198380 con 0x7f5ee4071f80 2024-04-26T16:30:50.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:50.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.570+0000 7f5ef5ffb700 1 -- 172.21.15.50:0/4003954935 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+589915 (secure 0 0 0) 0x7f5ef8198380 con 0x7f5ee4071f80 2024-04-26T16:30:50.573 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 10 librados: watch_flush enter 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 10 librados: watch_flush exit 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 20 client.30717.objecter shutdown clearing up homeless session... 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 10 client.30717.objecter successfully canceled tick 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ee4071f80 msgr2=0x7f5ee4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ee4071f80 0x7f5ee4074440 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 crypto rx=0x7f5ee8005fd0 tx=0x7f5ee8009310 comp rx=0 tx=0).stop 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 msgr2=0x7f5ef819b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 0x7f5ef819b8f0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0011400 tx=0x7f5ef0011710 comp rx=0 tx=0).stop 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 shutdown_connections 2024-04-26T16:30:50.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5ee4071f80 0x7f5ee4074440 unknown :-1 s=CLOSED pgs=3767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ef8102350 0x7f5ef8197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5ef81083c0 0x7f5ef819b8f0 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 --2- 172.21.15.50:0/4003954935 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5ef8102d10 0x7f5ef8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 >> 172.21.15.50:0/4003954935 conn(0x7f5ef8079360 msgr2=0x7f5ef8100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 shutdown_connections 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 -- 172.21.15.50:0/4003954935 wait complete. 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:50.574+0000 7f5efee14700 1 librados: shutdown 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:30:50.580 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:50.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:50.711 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:50.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:50.839 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:50.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:50.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:50.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:50.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:51.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:51.099 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:51.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:51.234 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:51.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:51.364 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:51.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:51.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:51 smithi190 bash[23868]: cluster 2024-04-26T16:30:49.726195+0000 mgr.y (mgr.24401) 522 : cluster [DBG] pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 539 B/s wr, 2 op/s 2024-04-26T16:30:51.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:51 smithi190 bash[23868]: cluster 2024-04-26T16:30:49.726195+0000 mgr.y (mgr.24401) 522 : cluster [DBG] pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 539 B/s wr, 2 op/s 2024-04-26T16:30:51.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:51 smithi190 bash[23868]: audit 2024-04-26T16:30:50.552497+0000 mgr.y (mgr.24401) 523 : audit [DBG] from='client.30717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:30:51.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:51 smithi190 bash[23868]: audit 2024-04-26T16:30:50.552497+0000 mgr.y (mgr.24401) 523 : audit [DBG] from='client.30717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:30:51.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:51 smithi190 bash[23868]: cluster 2024-04-26T16:30:51.223919+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-04-26T16:30:51.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:51 smithi190 bash[23868]: cluster 2024-04-26T16:30:51.223919+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-04-26T16:30:51.495 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:51.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:51.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:51.624 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:51.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[29234]: cluster 2024-04-26T16:30:49.726195+0000 mgr.y (mgr.24401) 522 : cluster [DBG] pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 539 B/s wr, 2 op/s 2024-04-26T16:30:51.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[29234]: cluster 2024-04-26T16:30:49.726195+0000 mgr.y (mgr.24401) 522 : cluster [DBG] pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 539 B/s wr, 2 op/s 2024-04-26T16:30:51.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[29234]: audit 2024-04-26T16:30:50.552497+0000 mgr.y (mgr.24401) 523 : audit [DBG] from='client.30717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:30:51.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[29234]: audit 2024-04-26T16:30:50.552497+0000 mgr.y (mgr.24401) 523 : audit [DBG] from='client.30717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:30:51.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[29234]: cluster 2024-04-26T16:30:51.223919+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-04-26T16:30:51.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[29234]: cluster 2024-04-26T16:30:51.223919+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-04-26T16:30:51.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[21463]: cluster 2024-04-26T16:30:49.726195+0000 mgr.y (mgr.24401) 522 : cluster [DBG] pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 539 B/s wr, 2 op/s 2024-04-26T16:30:51.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[21463]: cluster 2024-04-26T16:30:49.726195+0000 mgr.y (mgr.24401) 522 : cluster [DBG] pgmap v796: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 539 B/s wr, 2 op/s 2024-04-26T16:30:51.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[21463]: audit 2024-04-26T16:30:50.552497+0000 mgr.y (mgr.24401) 523 : audit [DBG] from='client.30717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:30:51.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[21463]: audit 2024-04-26T16:30:50.552497+0000 mgr.y (mgr.24401) 523 : audit [DBG] from='client.30717 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:30:51.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[21463]: cluster 2024-04-26T16:30:51.223919+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-04-26T16:30:51.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:51 smithi050 bash[21463]: cluster 2024-04-26T16:30:51.223919+0000 mon.a (mon.0) 2906 : cluster [DBG] osdmap e522: 8 total, 8 up, 7 in 2024-04-26T16:30:51.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:51.754 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:51.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:51.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:51.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:51.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:51.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:52.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:52.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:52.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:52.191 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-04-26T16:30:49.725477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792479,"num_objects":378,"num_object_clones":12,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":2271,"num_read_kb":2011,"num_write":1544,"num_write_kb":15187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":5967718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":22,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3034,"ondisk_log_size":3034,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":527652,"kb_used_data":52732,"kb_used_omap":24,"kb_used_meta":474791,"kb_avail":655688412,"statfs":{"total":671965249536,"available":671424933888,"internally_reserved":0,"allocated":53997568,"data_stored":37406858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25087,"internal_metadata":486186497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.492234"},"pg_stats":[{"pgid":"305.1f","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241905+0000","last_change":"2024-04-26T16:30:38.284560+0000","last_active":"2024-04-26T16:30:40.241905+0000","last_peered":"2024-04-26T16:30:40.241905+0000","last_clean":"2024-04-26T16:30:40.241905+0000","last_became_active":"2024-04-26T16:30:38.283969+0000","last_became_peered":"2024-04-26T16:30:38.283969+0000","last_unstale":"2024-04-26T16:30:40.241905+0000","last_undegraded":"2024-04-26T16:30:40.241905+0000","last_fullsized":"2024-04-26T16:30:40.241905+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:05.793631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.1e","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242479+0000","last_change":"2024-04-26T16:30:38.282237+0000","last_active":"2024-04-26T16:30:40.242479+0000","last_peered":"2024-04-26T16:30:40.242479+0000","last_clean":"2024-04-26T16:30:40.242479+0000","last_became_active":"2024-04-26T16:30:38.281611+0000","last_became_peered":"2024-04-26T16:30:38.281611+0000","last_unstale":"2024-04-26T16:30:40.242479+0000","last_undegraded":"2024-04-26T16:30:40.242479+0000","last_fullsized":"2024-04-26T16:30:40.242479+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:06.860269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.1d","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241546+0000","last_change":"2024-04-26T16:30:38.286391+0000","last_active":"2024-04-26T16:30:40.241546+0000","last_peered":"2024-04-26T16:30:40.241546+0000","last_clean":"2024-04-26T16:30:40.241546+0000","last_became_active":"2024-04-26T16:30:38.285394+0000","last_became_peered":"2024-04-26T16:30:38.285394+0000","last_unstale":"2024-04-26T16:30:40.241546+0000","last_undegraded":"2024-04-26T16:30:40.241546+0000","last_fullsized":"2024-04-26T16:30:40.241546+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:02.815015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.1c","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259983+0000","last_change":"2024-04-26T16:30:38.272260+0000","last_active":"2024-04-26T16:30:40.259983+0000","last_peered":"2024-04-26T16:30:40.259983+0000","last_clean":"2024-04-26T16:30:40.259983+0000","last_became_active":"2024-04-26T16:30:38.271595+0000","last_became_peered":"2024-04-26T16:30:38.271595+0000","last_unstale":"2024-04-26T16:30:40.259983+0000","last_undegraded":"2024-04-26T16:30:40.259983+0000","last_fullsized":"2024-04-26T16:30:40.259983+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.454074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.1b","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258836+0000","last_change":"2024-04-26T16:30:38.286429+0000","last_active":"2024-04-26T16:30:40.258836+0000","last_peered":"2024-04-26T16:30:40.258836+0000","last_clean":"2024-04-26T16:30:40.258836+0000","last_became_active":"2024-04-26T16:30:38.285775+0000","last_became_peered":"2024-04-26T16:30:38.285775+0000","last_unstale":"2024-04-26T16:30:40.258836+0000","last_undegraded":"2024-04-26T16:30:40.258836+0000","last_fullsized":"2024-04-26T16:30:40.258836+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:51.938691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.1a","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240181+0000","last_change":"2024-04-26T16:30:38.278044+0000","last_active":"2024-04-26T16:30:40.240181+0000","last_peered":"2024-04-26T16:30:40.240181+0000","last_clean":"2024-04-26T16:30:40.240181+0000","last_became_active":"2024-04-26T16:30:38.277388+0000","last_became_peered":"2024-04-26T16:30:38.277388+0000","last_unstale":"2024-04-26T16:30:40.240181+0000","last_undegraded":"2024-04-26T16:30:40.240181+0000","last_fullsized":"2024-04-26T16:30:40.240181+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:51.209999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.19","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.254006+0000","last_change":"2024-04-26T16:30:38.255035+0000","last_active":"2024-04-26T16:30:40.254006+0000","last_peered":"2024-04-26T16:30:40.254006+0000","last_clean":"2024-04-26T16:30:40.254006+0000","last_became_active":"2024-04-26T16:30:38.254056+0000","last_became_peered":"2024-04-26T16:30:38.254056+0000","last_unstale":"2024-04-26T16:30:40.254006+0000","last_undegraded":"2024-04-26T16:30:40.254006+0000","last_fullsized":"2024-04-26T16:30:40.254006+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:00.824289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.18","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237616+0000","last_change":"2024-04-26T16:30:38.275894+0000","last_active":"2024-04-26T16:30:40.237616+0000","last_peered":"2024-04-26T16:30:40.237616+0000","last_clean":"2024-04-26T16:30:40.237616+0000","last_became_active":"2024-04-26T16:30:38.275620+0000","last_became_peered":"2024-04-26T16:30:38.275620+0000","last_unstale":"2024-04-26T16:30:40.237616+0000","last_undegraded":"2024-04-26T16:30:40.237616+0000","last_fullsized":"2024-04-26T16:30:40.237616+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:03.392164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.17","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.245580+0000","last_change":"2024-04-26T16:30:38.269281+0000","last_active":"2024-04-26T16:30:40.245580+0000","last_peered":"2024-04-26T16:30:40.245580+0000","last_clean":"2024-04-26T16:30:40.245580+0000","last_became_active":"2024-04-26T16:30:38.267625+0000","last_became_peered":"2024-04-26T16:30:38.267625+0000","last_unstale":"2024-04-26T16:30:40.245580+0000","last_undegraded":"2024-04-26T16:30:40.245580+0000","last_fullsized":"2024-04-26T16:30:40.245580+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:00.489780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.16","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260600+0000","last_change":"2024-04-26T16:30:38.287111+0000","last_active":"2024-04-26T16:30:40.260600+0000","last_peered":"2024-04-26T16:30:40.260600+0000","last_clean":"2024-04-26T16:30:40.260600+0000","last_became_active":"2024-04-26T16:30:38.283207+0000","last_became_peered":"2024-04-26T16:30:38.283207+0000","last_unstale":"2024-04-26T16:30:40.260600+0000","last_undegraded":"2024-04-26T16:30:40.260600+0000","last_fullsized":"2024-04-26T16:30:40.260600+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:50.742420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.15","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.256171+0000","last_change":"2024-04-26T16:30:38.260449+0000","last_active":"2024-04-26T16:30:40.256171+0000","last_peered":"2024-04-26T16:30:40.256171+0000","last_clean":"2024-04-26T16:30:40.256171+0000","last_became_active":"2024-04-26T16:30:38.258614+0000","last_became_peered":"2024-04-26T16:30:38.258614+0000","last_unstale":"2024-04-26T16:30:40.256171+0000","last_undegraded":"2024-04-26T16:30:40.256171+0000","last_fullsized":"2024-04-26T16:30:40.256171+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:45.015782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.14","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239928+0000","last_change":"2024-04-26T16:30:38.279813+0000","last_active":"2024-04-26T16:30:40.239928+0000","last_peered":"2024-04-26T16:30:40.239928+0000","last_clean":"2024-04-26T16:30:40.239928+0000","last_became_active":"2024-04-26T16:30:38.279445+0000","last_became_peered":"2024-04-26T16:30:38.279445+0000","last_unstale":"2024-04-26T16:30:40.239928+0000","last_undegraded":"2024-04-26T16:30:40.239928+0000","last_fullsized":"2024-04-26T16:30:40.239928+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:46.953674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.13","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238665+0000","last_change":"2024-04-26T16:30:38.255006+0000","last_active":"2024-04-26T16:30:40.238665+0000","last_peered":"2024-04-26T16:30:40.238665+0000","last_clean":"2024-04-26T16:30:40.238665+0000","last_became_active":"2024-04-26T16:30:38.254020+0000","last_became_peered":"2024-04-26T16:30:38.254020+0000","last_unstale":"2024-04-26T16:30:40.238665+0000","last_undegraded":"2024-04-26T16:30:40.238665+0000","last_fullsized":"2024-04-26T16:30:40.238665+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.454513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.12","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240552+0000","last_change":"2024-04-26T16:30:38.274076+0000","last_active":"2024-04-26T16:30:40.240552+0000","last_peered":"2024-04-26T16:30:40.240552+0000","last_clean":"2024-04-26T16:30:40.240552+0000","last_became_active":"2024-04-26T16:30:38.273352+0000","last_became_peered":"2024-04-26T16:30:38.273352+0000","last_unstale":"2024-04-26T16:30:40.240552+0000","last_undegraded":"2024-04-26T16:30:40.240552+0000","last_fullsized":"2024-04-26T16:30:40.240552+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.467602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.11","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.253935+0000","last_change":"2024-04-26T16:30:38.254867+0000","last_active":"2024-04-26T16:30:40.253935+0000","last_peered":"2024-04-26T16:30:40.253935+0000","last_clean":"2024-04-26T16:30:40.253935+0000","last_became_active":"2024-04-26T16:30:38.252766+0000","last_became_peered":"2024-04-26T16:30:38.252766+0000","last_unstale":"2024-04-26T16:30:40.253935+0000","last_undegraded":"2024-04-26T16:30:40.253935+0000","last_fullsized":"2024-04-26T16:30:40.253935+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.757858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.10","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.236622+0000","last_change":"2024-04-26T16:30:38.281881+0000","last_active":"2024-04-26T16:30:40.236622+0000","last_peered":"2024-04-26T16:30:40.236622+0000","last_clean":"2024-04-26T16:30:40.236622+0000","last_became_active":"2024-04-26T16:30:38.281233+0000","last_became_peered":"2024-04-26T16:30:38.281233+0000","last_unstale":"2024-04-26T16:30:40.236622+0000","last_undegraded":"2024-04-26T16:30:40.236622+0000","last_fullsized":"2024-04-26T16:30:40.236622+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.173484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.f","version":"0'0","reported_seq":18,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.490621+0000","last_change":"2024-04-26T16:30:38.277221+0000","last_active":"2024-04-26T16:30:47.490621+0000","last_peered":"2024-04-26T16:30:47.490621+0000","last_clean":"2024-04-26T16:30:47.490621+0000","last_became_active":"2024-04-26T16:30:38.276682+0000","last_became_peered":"2024-04-26T16:30:38.276682+0000","last_unstale":"2024-04-26T16:30:47.490621+0000","last_undegraded":"2024-04-26T16:30:47.490621+0000","last_fullsized":"2024-04-26T16:30:47.490621+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:43.602963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"305.e","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251933+0000","last_change":"2024-04-26T16:30:38.252070+0000","last_active":"2024-04-26T16:30:40.251933+0000","last_peered":"2024-04-26T16:30:40.251933+0000","last_clean":"2024-04-26T16:30:40.251933+0000","last_became_active":"2024-04-26T16:30:38.251771+0000","last_became_peered":"2024-04-26T16:30:38.251771+0000","last_unstale":"2024-04-26T16:30:40.251933+0000","last_undegraded":"2024-04-26T16:30:40.251933+0000","last_fullsized":"2024-04-26T16:30:40.251933+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.562706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"305.d","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260178+0000","last_change":"2024-04-26T16:30:38.277986+0000","last_active":"2024-04-26T16:30:40.260178+0000","last_peered":"2024-04-26T16:30:40.260178+0000","last_clean":"2024-04-26T16:30:40.260178+0000","last_became_active":"2024-04-26T16:30:38.275995+0000","last_became_peered":"2024-04-26T16:30:38.275995+0000","last_unstale":"2024-04-26T16:30:40.260178+0000","last_undegraded":"2024-04-26T16:30:40.260178+0000","last_fullsized":"2024-04-26T16:30:40.260178+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.113179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.c","version":"0'0","reported_seq":18,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488872+0000","last_change":"2024-04-26T16:30:38.261127+0000","last_active":"2024-04-26T16:30:47.488872+0000","last_peered":"2024-04-26T16:30:47.488872+0000","last_clean":"2024-04-26T16:30:47.488872+0000","last_became_active":"2024-04-26T16:30:38.260771+0000","last_became_peered":"2024-04-26T16:30:38.260771+0000","last_unstale":"2024-04-26T16:30:47.488872+0000","last_undegraded":"2024-04-26T16:30:47.488872+0000","last_fullsized":"2024-04-26T16:30:47.488872+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.451110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"305.b","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.248843+0000","last_change":"2024-04-26T16:30:38.252380+0000","last_active":"2024-04-26T16:30:40.248843+0000","last_peered":"2024-04-26T16:30:40.248843+0000","last_clean":"2024-04-26T16:30:40.248843+0000","last_became_active":"2024-04-26T16:30:38.251357+0000","last_became_peered":"2024-04-26T16:30:38.251357+0000","last_unstale":"2024-04-26T16:30:40.248843+0000","last_undegraded":"2024-04-26T16:30:40.248843+0000","last_fullsized":"2024-04-26T16:30:40.248843+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:58.943730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"305.a","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240077+0000","last_change":"2024-04-26T16:30:38.260121+0000","last_active":"2024-04-26T16:30:40.240077+0000","last_peered":"2024-04-26T16:30:40.240077+0000","last_clean":"2024-04-26T16:30:40.240077+0000","last_became_active":"2024-04-26T16:30:38.257170+0000","last_became_peered":"2024-04-26T16:30:38.257170+0000","last_unstale":"2024-04-26T16:30:40.240077+0000","last_undegraded":"2024-04-26T16:30:40.240077+0000","last_fullsized":"2024-04-26T16:30:40.240077+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:43.910944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.9","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.254894+0000","last_change":"2024-04-26T16:30:38.255591+0000","last_active":"2024-04-26T16:30:40.254894+0000","last_peered":"2024-04-26T16:30:40.254894+0000","last_clean":"2024-04-26T16:30:40.254894+0000","last_became_active":"2024-04-26T16:30:38.252885+0000","last_became_peered":"2024-04-26T16:30:38.252885+0000","last_unstale":"2024-04-26T16:30:40.254894+0000","last_undegraded":"2024-04-26T16:30:40.254894+0000","last_fullsized":"2024-04-26T16:30:40.254894+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:00.409942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.8","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.253610+0000","last_change":"2024-04-26T16:30:38.263700+0000","last_active":"2024-04-26T16:30:40.253610+0000","last_peered":"2024-04-26T16:30:40.253610+0000","last_clean":"2024-04-26T16:30:40.253610+0000","last_became_active":"2024-04-26T16:30:38.262584+0000","last_became_peered":"2024-04-26T16:30:38.262584+0000","last_unstale":"2024-04-26T16:30:40.253610+0000","last_undegraded":"2024-04-26T16:30:40.253610+0000","last_fullsized":"2024-04-26T16:30:40.253610+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.825196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"305.7","version":"520'8","reported_seq":31,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.885119+0000","last_change":"2024-04-26T16:30:40.885119+0000","last_active":"2024-04-26T16:30:40.885119+0000","last_peered":"2024-04-26T16:30:40.885119+0000","last_clean":"2024-04-26T16:30:40.885119+0000","last_became_active":"2024-04-26T16:30:38.254318+0000","last_became_peered":"2024-04-26T16:30:38.254318+0000","last_unstale":"2024-04-26T16:30:40.885119+0000","last_undegraded":"2024-04-26T16:30:40.885119+0000","last_fullsized":"2024-04-26T16:30:40.885119+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'8","last_scrub_stamp":"2024-04-26T16:30:40.884883+0000","last_deep_scrub":"520'8","last_deep_scrub_stamp":"2024-04-26T16:30:40.884883+0000","last_clean_scrub_stamp":"2024-04-26T16:30:40.884883+0000","objects_scrubbed":3,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:51.310124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"305.6","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.245461+0000","last_change":"2024-04-26T16:30:38.254969+0000","last_active":"2024-04-26T16:30:40.245461+0000","last_peered":"2024-04-26T16:30:40.245461+0000","last_clean":"2024-04-26T16:30:40.245461+0000","last_became_active":"2024-04-26T16:30:38.254037+0000","last_became_peered":"2024-04-26T16:30:38.254037+0000","last_unstale":"2024-04-26T16:30:40.245461+0000","last_undegraded":"2024-04-26T16:30:40.245461+0000","last_fullsized":"2024-04-26T16:30:40.245461+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.451746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.5","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241781+0000","last_change":"2024-04-26T16:30:38.285945+0000","last_active":"2024-04-26T16:30:40.241781+0000","last_peered":"2024-04-26T16:30:40.241781+0000","last_clean":"2024-04-26T16:30:40.241781+0000","last_became_active":"2024-04-26T16:30:38.284579+0000","last_became_peered":"2024-04-26T16:30:38.284579+0000","last_unstale":"2024-04-26T16:30:40.241781+0000","last_undegraded":"2024-04-26T16:30:40.241781+0000","last_fullsized":"2024-04-26T16:30:40.241781+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:06.138144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"305.4","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242968+0000","last_change":"2024-04-26T16:30:38.275181+0000","last_active":"2024-04-26T16:30:40.242968+0000","last_peered":"2024-04-26T16:30:40.242968+0000","last_clean":"2024-04-26T16:30:40.242968+0000","last_became_active":"2024-04-26T16:30:38.274451+0000","last_became_peered":"2024-04-26T16:30:38.274451+0000","last_unstale":"2024-04-26T16:30:40.242968+0000","last_undegraded":"2024-04-26T16:30:40.242968+0000","last_fullsized":"2024-04-26T16:30:40.242968+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:03.077892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.3","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241276+0000","last_change":"2024-04-26T16:30:38.275459+0000","last_active":"2024-04-26T16:30:40.241276+0000","last_peered":"2024-04-26T16:30:40.241276+0000","last_clean":"2024-04-26T16:30:40.241276+0000","last_became_active":"2024-04-26T16:30:38.274736+0000","last_became_peered":"2024-04-26T16:30:38.274736+0000","last_unstale":"2024-04-26T16:30:40.241276+0000","last_undegraded":"2024-04-26T16:30:40.241276+0000","last_fullsized":"2024-04-26T16:30:40.241276+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.612781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"305.2","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258649+0000","last_change":"2024-04-26T16:30:38.284640+0000","last_active":"2024-04-26T16:30:40.258649+0000","last_peered":"2024-04-26T16:30:40.258649+0000","last_clean":"2024-04-26T16:30:40.258649+0000","last_became_active":"2024-04-26T16:30:38.283925+0000","last_became_peered":"2024-04-26T16:30:38.283925+0000","last_unstale":"2024-04-26T16:30:40.258649+0000","last_undegraded":"2024-04-26T16:30:40.258649+0000","last_fullsized":"2024-04-26T16:30:40.258649+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:03.841868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"305.1","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237976+0000","last_change":"2024-04-26T16:30:38.280084+0000","last_active":"2024-04-26T16:30:40.237976+0000","last_peered":"2024-04-26T16:30:40.237976+0000","last_clean":"2024-04-26T16:30:40.237976+0000","last_became_active":"2024-04-26T16:30:38.279340+0000","last_became_peered":"2024-04-26T16:30:38.279340+0000","last_unstale":"2024-04-26T16:30:40.237976+0000","last_undegraded":"2024-04-26T16:30:40.237976+0000","last_fullsized":"2024-04-26T16:30:40.237976+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.000051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"305.0","version":"0'0","reported_seq":16,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237612+0000","last_change":"2024-04-26T16:30:38.284564+0000","last_active":"2024-04-26T16:30:40.237612+0000","last_peered":"2024-04-26T16:30:40.237612+0000","last_clean":"2024-04-26T16:30:40.237612+0000","last_became_active":"2024-04-26T16:30:38.283977+0000","last_became_peered":"2024-04-26T16:30:38.283977+0000","last_unstale":"2024-04-26T16:30:40.237612+0000","last_undegraded":"2024-04-26T16:30:40.237612+0000","last_fullsized":"2024-04-26T16:30:40.237612+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":517,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:37.214053+0000","last_clean_scrub_stamp":"2024-04-26T16:30:37.214053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:58.890460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.12","version":"389'86","reported_seq":1009,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258699+0000","last_change":"2024-04-26T16:29:29.291952+0000","last_active":"2024-04-26T16:30:40.258699+0000","last_peered":"2024-04-26T16:30:40.258699+0000","last_clean":"2024-04-26T16:30:40.258699+0000","last_became_active":"2024-04-26T16:23:44.188737+0000","last_became_peered":"2024-04-26T16:23:44.188737+0000","last_unstale":"2024-04-26T16:30:40.258699+0000","last_undegraded":"2024-04-26T16:30:40.258699+0000","last_fullsized":"2024-04-26T16:30:40.258699+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:29:10.006584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.006584+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:51.018888+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011941726,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.13","version":"389'52","reported_seq":810,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239904+0000","last_change":"2024-04-26T16:30:24.606400+0000","last_active":"2024-04-26T16:30:40.239904+0000","last_peered":"2024-04-26T16:30:40.239904+0000","last_clean":"2024-04-26T16:30:40.239904+0000","last_became_active":"2024-04-26T16:26:23.192498+0000","last_became_peered":"2024-04-26T16:26:23.192498+0000","last_unstale":"2024-04-26T16:30:40.239904+0000","last_undegraded":"2024-04-26T16:30:40.239904+0000","last_fullsized":"2024-04-26T16:30:40.239904+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'52","last_scrub_stamp":"2024-04-26T16:30:24.606145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.606145+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:39.989241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064602280000000002,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.10","version":"469'150","reported_seq":1162,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247180+0000","last_change":"2024-04-26T16:29:29.501691+0000","last_active":"2024-04-26T16:30:40.247180+0000","last_peered":"2024-04-26T16:30:40.247180+0000","last_clean":"2024-04-26T16:30:40.247180+0000","last_became_active":"2024-04-26T16:26:26.135726+0000","last_became_peered":"2024-04-26T16:26:26.135726+0000","last_unstale":"2024-04-26T16:30:40.247180+0000","last_undegraded":"2024-04-26T16:30:40.247180+0000","last_fullsized":"2024-04-26T16:30:40.247180+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'126","last_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_deep_scrub":"400'126","last_deep_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_clean_scrub_stamp":"2024-04-26T16:27:49.058542+0000","objects_scrubbed":7,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:50.758357+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001241277,"stat_sum":{"num_bytes":103,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":44,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":77,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.11","version":"236'34","reported_seq":979,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238241+0000","last_change":"2024-04-26T16:29:57.929311+0000","last_active":"2024-04-26T16:30:40.238241+0000","last_peered":"2024-04-26T16:30:40.238241+0000","last_clean":"2024-04-26T16:30:40.238241+0000","last_became_active":"2024-04-26T16:23:44.102085+0000","last_became_peered":"2024-04-26T16:23:44.102085+0000","last_unstale":"2024-04-26T16:30:40.238241+0000","last_undegraded":"2024-04-26T16:30:40.238241+0000","last_fullsized":"2024-04-26T16:30:40.238241+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:29:57.929076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.929076+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:15.200371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066858430000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.16","version":"485'84","reported_seq":927,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260809+0000","last_change":"2024-04-26T16:29:29.294557+0000","last_active":"2024-04-26T16:30:40.260809+0000","last_peered":"2024-04-26T16:30:40.260809+0000","last_clean":"2024-04-26T16:30:40.260809+0000","last_became_active":"2024-04-26T16:26:23.214752+0000","last_became_peered":"2024-04-26T16:26:23.214752+0000","last_unstale":"2024-04-26T16:30:40.260809+0000","last_undegraded":"2024-04-26T16:30:40.260809+0000","last_fullsized":"2024-04-26T16:30:40.260809+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:29:04.018776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.018776+0000","objects_scrubbed":7,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060185220000000001,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.17","version":"520'52","reported_seq":956,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.490070+0000","last_change":"2024-04-26T16:30:02.546379+0000","last_active":"2024-04-26T16:30:47.490070+0000","last_peered":"2024-04-26T16:30:47.490070+0000","last_clean":"2024-04-26T16:30:47.490070+0000","last_became_active":"2024-04-26T16:28:25.260911+0000","last_became_peered":"2024-04-26T16:28:25.260911+0000","last_unstale":"2024-04-26T16:30:47.490070+0000","last_undegraded":"2024-04-26T16:30:47.490070+0000","last_fullsized":"2024-04-26T16:30:47.490070+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'50","last_scrub_stamp":"2024-04-26T16:30:02.546137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.546137+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:08.118757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00057068899999999998,"stat_sum":{"num_bytes":71,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":45,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.14","version":"457'38","reported_seq":978,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240298+0000","last_change":"2024-04-26T16:29:29.288410+0000","last_active":"2024-04-26T16:30:40.240298+0000","last_peered":"2024-04-26T16:30:40.240298+0000","last_clean":"2024-04-26T16:30:40.240298+0000","last_became_active":"2024-04-26T16:23:44.119110+0000","last_became_peered":"2024-04-26T16:23:44.119110+0000","last_unstale":"2024-04-26T16:30:40.240298+0000","last_undegraded":"2024-04-26T16:30:40.240298+0000","last_fullsized":"2024-04-26T16:30:40.240298+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'38","last_scrub_stamp":"2024-04-26T16:28:49.475044+0000","last_deep_scrub":"457'38","last_deep_scrub_stamp":"2024-04-26T16:28:49.475044+0000","last_clean_scrub_stamp":"2024-04-26T16:28:49.475044+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0034343469999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.15","version":"488'78","reported_seq":1021,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241826+0000","last_change":"2024-04-26T16:30:12.508796+0000","last_active":"2024-04-26T16:30:40.241826+0000","last_peered":"2024-04-26T16:30:40.241826+0000","last_clean":"2024-04-26T16:30:40.241826+0000","last_became_active":"2024-04-26T16:23:44.233737+0000","last_became_peered":"2024-04-26T16:23:44.233737+0000","last_unstale":"2024-04-26T16:30:40.241826+0000","last_undegraded":"2024-04-26T16:30:40.241826+0000","last_fullsized":"2024-04-26T16:30:40.241826+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'78","last_scrub_stamp":"2024-04-26T16:30:12.508639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.508639+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.176078+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0046970850000000001,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1a","version":"322'90","reported_seq":1086,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240324+0000","last_change":"2024-04-26T16:30:25.642345+0000","last_active":"2024-04-26T16:30:40.240324+0000","last_peered":"2024-04-26T16:30:40.240324+0000","last_clean":"2024-04-26T16:30:40.240324+0000","last_became_active":"2024-04-26T16:26:23.171828+0000","last_became_peered":"2024-04-26T16:26:23.171828+0000","last_unstale":"2024-04-26T16:30:40.240324+0000","last_undegraded":"2024-04-26T16:30:40.240324+0000","last_fullsized":"2024-04-26T16:30:40.240324+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'90","last_scrub_stamp":"2024-04-26T16:30:25.642102+0000","last_deep_scrub":"322'90","last_deep_scrub_stamp":"2024-04-26T16:26:30.427683+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.642102+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:51.756368+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0046318089999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1b","version":"458'28","reported_seq":995,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239863+0000","last_change":"2024-04-26T16:29:29.513165+0000","last_active":"2024-04-26T16:30:40.239863+0000","last_peered":"2024-04-26T16:30:40.239863+0000","last_clean":"2024-04-26T16:30:40.239863+0000","last_became_active":"2024-04-26T16:28:25.251700+0000","last_became_peered":"2024-04-26T16:28:25.251700+0000","last_unstale":"2024-04-26T16:30:40.239863+0000","last_undegraded":"2024-04-26T16:30:40.239863+0000","last_fullsized":"2024-04-26T16:30:40.239863+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'28","last_scrub_stamp":"2024-04-26T16:28:51.005534+0000","last_deep_scrub":"458'28","last_deep_scrub_stamp":"2024-04-26T16:28:51.005534+0000","last_clean_scrub_stamp":"2024-04-26T16:28:51.005534+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048628739999999997,"stat_sum":{"num_bytes":30,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.18","version":"452'45","reported_seq":1023,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.253561+0000","last_change":"2024-04-26T16:30:20.631404+0000","last_active":"2024-04-26T16:30:40.253561+0000","last_peered":"2024-04-26T16:30:40.253561+0000","last_clean":"2024-04-26T16:30:40.253561+0000","last_became_active":"2024-04-26T16:26:23.188399+0000","last_became_peered":"2024-04-26T16:26:23.188399+0000","last_unstale":"2024-04-26T16:30:40.253561+0000","last_undegraded":"2024-04-26T16:30:40.253561+0000","last_fullsized":"2024-04-26T16:30:40.253561+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'45","last_scrub_stamp":"2024-04-26T16:30:20.631310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:20.631310+0000","objects_scrubbed":6,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:33.855977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0038147540000000001,"stat_sum":{"num_bytes":40,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.19","version":"308'35","reported_seq":983,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.254223+0000","last_change":"2024-04-26T16:30:09.622791+0000","last_active":"2024-04-26T16:30:40.254223+0000","last_peered":"2024-04-26T16:30:40.254223+0000","last_clean":"2024-04-26T16:30:40.254223+0000","last_became_active":"2024-04-26T16:23:44.267057+0000","last_became_peered":"2024-04-26T16:23:44.267057+0000","last_unstale":"2024-04-26T16:30:40.254223+0000","last_undegraded":"2024-04-26T16:30:40.254223+0000","last_fullsized":"2024-04-26T16:30:40.254223+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'35","last_scrub_stamp":"2024-04-26T16:30:09.622674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.622674+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.793091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029077199999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.1e","version":"389'20","reported_seq":979,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243059+0000","last_change":"2024-04-26T16:30:33.603424+0000","last_active":"2024-04-26T16:30:40.243059+0000","last_peered":"2024-04-26T16:30:40.243059+0000","last_clean":"2024-04-26T16:30:40.243059+0000","last_became_active":"2024-04-26T16:28:25.230305+0000","last_became_peered":"2024-04-26T16:28:25.230305+0000","last_unstale":"2024-04-26T16:30:40.243059+0000","last_undegraded":"2024-04-26T16:30:40.243059+0000","last_fullsized":"2024-04-26T16:30:40.243059+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-04-26T16:30:33.603284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.603284+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.405921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005995696,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1f","version":"445'20","reported_seq":961,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260834+0000","last_change":"2024-04-26T16:29:29.292388+0000","last_active":"2024-04-26T16:30:40.260834+0000","last_peered":"2024-04-26T16:30:40.260834+0000","last_clean":"2024-04-26T16:30:40.260834+0000","last_became_active":"2024-04-26T16:23:44.180651+0000","last_became_peered":"2024-04-26T16:23:44.180651+0000","last_unstale":"2024-04-26T16:30:40.260834+0000","last_undegraded":"2024-04-26T16:30:40.260834+0000","last_fullsized":"2024-04-26T16:30:40.260834+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'20","last_scrub_stamp":"2024-04-26T16:29:03.044079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.044079+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0037815560000000002,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1c","version":"287'56","reported_seq":977,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.489881+0000","last_change":"2024-04-26T16:30:18.310854+0000","last_active":"2024-04-26T16:30:47.489881+0000","last_peered":"2024-04-26T16:30:47.489881+0000","last_clean":"2024-04-26T16:30:47.489881+0000","last_became_active":"2024-04-26T16:28:25.253825+0000","last_became_peered":"2024-04-26T16:28:25.253825+0000","last_unstale":"2024-04-26T16:30:47.489881+0000","last_undegraded":"2024-04-26T16:30:47.489881+0000","last_fullsized":"2024-04-26T16:30:47.489881+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-04-26T16:30:18.310740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.310740+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:30.852946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053197300000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.1d","version":"448'147","reported_seq":1051,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249328+0000","last_change":"2024-04-26T16:29:29.505126+0000","last_active":"2024-04-26T16:30:40.249328+0000","last_peered":"2024-04-26T16:30:40.249328+0000","last_clean":"2024-04-26T16:30:40.249328+0000","last_became_active":"2024-04-26T16:23:44.112550+0000","last_became_peered":"2024-04-26T16:23:44.112550+0000","last_unstale":"2024-04-26T16:30:40.249328+0000","last_undegraded":"2024-04-26T16:30:40.249328+0000","last_fullsized":"2024-04-26T16:30:40.249328+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'144","last_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_deep_scrub":"401'144","last_deep_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_clean_scrub_stamp":"2024-04-26T16:27:50.033127+0000","objects_scrubbed":8,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879431+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001087244,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":42,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.2","version":"503'103","reported_seq":1040,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238303+0000","last_change":"2024-04-26T16:29:29.286447+0000","last_active":"2024-04-26T16:30:40.238303+0000","last_peered":"2024-04-26T16:30:40.238303+0000","last_clean":"2024-04-26T16:30:40.238303+0000","last_became_active":"2024-04-26T16:23:44.121137+0000","last_became_peered":"2024-04-26T16:23:44.121137+0000","last_unstale":"2024-04-26T16:30:40.238303+0000","last_undegraded":"2024-04-26T16:30:40.238303+0000","last_fullsized":"2024-04-26T16:30:40.238303+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'91","last_scrub_stamp":"2024-04-26T16:28:50.511931+0000","last_deep_scrub":"318'90","last_deep_scrub_stamp":"2024-04-26T16:26:26.686096+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.511931+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.588621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030250239999999999,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.3","version":"259'51","reported_seq":1029,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249305+0000","last_change":"2024-04-26T16:29:29.502257+0000","last_active":"2024-04-26T16:30:40.249305+0000","last_peered":"2024-04-26T16:30:40.249305+0000","last_clean":"2024-04-26T16:30:40.249305+0000","last_became_active":"2024-04-26T16:28:25.273792+0000","last_became_peered":"2024-04-26T16:28:25.273792+0000","last_unstale":"2024-04-26T16:30:40.249305+0000","last_undegraded":"2024-04-26T16:30:40.249305+0000","last_fullsized":"2024-04-26T16:30:40.249305+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'51","last_scrub_stamp":"2024-04-26T16:27:34.948809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.948809+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879480+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000677325,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.0","version":"389'75","reported_seq":1002,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258284+0000","last_change":"2024-04-26T16:29:56.937825+0000","last_active":"2024-04-26T16:30:40.258284+0000","last_peered":"2024-04-26T16:30:40.258284+0000","last_clean":"2024-04-26T16:30:40.258284+0000","last_became_active":"2024-04-26T16:23:44.226589+0000","last_became_peered":"2024-04-26T16:23:44.226589+0000","last_unstale":"2024-04-26T16:30:40.258284+0000","last_undegraded":"2024-04-26T16:30:40.258284+0000","last_fullsized":"2024-04-26T16:30:40.258284+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'75","last_scrub_stamp":"2024-04-26T16:29:56.937577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.937577+0000","objects_scrubbed":6,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:03.500638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004787085,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1","version":"520'105","reported_seq":1039,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.248317+0000","last_change":"2024-04-26T16:29:29.501949+0000","last_active":"2024-04-26T16:30:40.248317+0000","last_peered":"2024-04-26T16:30:40.248317+0000","last_clean":"2024-04-26T16:30:40.248317+0000","last_became_active":"2024-04-26T16:28:25.268393+0000","last_became_peered":"2024-04-26T16:28:25.268393+0000","last_unstale":"2024-04-26T16:30:40.248317+0000","last_undegraded":"2024-04-26T16:30:40.248317+0000","last_fullsized":"2024-04-26T16:30:40.248317+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'102","last_scrub_stamp":"2024-04-26T16:27:26.872968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:26.872968+0000","objects_scrubbed":6,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086574099999999995,"stat_sum":{"num_bytes":84,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":6,"num_read_kb":0,"num_write":33,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.6","version":"478'83","reported_seq":1031,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242797+0000","last_change":"2024-04-26T16:30:31.618966+0000","last_active":"2024-04-26T16:30:40.242797+0000","last_peered":"2024-04-26T16:30:40.242797+0000","last_clean":"2024-04-26T16:30:40.242797+0000","last_became_active":"2024-04-26T16:23:44.248476+0000","last_became_peered":"2024-04-26T16:23:44.248476+0000","last_unstale":"2024-04-26T16:30:40.242797+0000","last_undegraded":"2024-04-26T16:30:40.242797+0000","last_fullsized":"2024-04-26T16:30:40.242797+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'83","last_scrub_stamp":"2024-04-26T16:30:31.618721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.618721+0000","objects_scrubbed":8,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:41.987667+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.010158063,"stat_sum":{"num_bytes":4000069,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":3932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.7","version":"511'77","reported_seq":1019,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250703+0000","last_change":"2024-04-26T16:29:29.505553+0000","last_active":"2024-04-26T16:30:40.250703+0000","last_peered":"2024-04-26T16:30:40.250703+0000","last_clean":"2024-04-26T16:30:40.250703+0000","last_became_active":"2024-04-26T16:23:44.200194+0000","last_became_peered":"2024-04-26T16:23:44.200194+0000","last_unstale":"2024-04-26T16:30:40.250703+0000","last_undegraded":"2024-04-26T16:30:40.250703+0000","last_fullsized":"2024-04-26T16:30:40.250703+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'57","last_scrub_stamp":"2024-04-26T16:27:37.015805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:37.015805+0000","objects_scrubbed":5,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094604400000000005,"stat_sum":{"num_bytes":96,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.4","version":"389'86","reported_seq":1031,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243962+0000","last_change":"2024-04-26T16:30:10.930053+0000","last_active":"2024-04-26T16:30:40.243962+0000","last_peered":"2024-04-26T16:30:40.243962+0000","last_clean":"2024-04-26T16:30:40.243962+0000","last_became_active":"2024-04-26T16:28:25.254293+0000","last_became_peered":"2024-04-26T16:28:25.254293+0000","last_unstale":"2024-04-26T16:30:40.243962+0000","last_undegraded":"2024-04-26T16:30:40.243962+0000","last_fullsized":"2024-04-26T16:30:40.243962+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:30:10.929911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:10.929911+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.745250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0073012850000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.5","version":"496'52","reported_seq":995,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240060+0000","last_change":"2024-04-26T16:29:29.287619+0000","last_active":"2024-04-26T16:30:40.240060+0000","last_peered":"2024-04-26T16:30:40.240060+0000","last_clean":"2024-04-26T16:30:40.240060+0000","last_became_active":"2024-04-26T16:23:44.116978+0000","last_became_peered":"2024-04-26T16:23:44.116978+0000","last_unstale":"2024-04-26T16:30:40.240060+0000","last_undegraded":"2024-04-26T16:30:40.240060+0000","last_fullsized":"2024-04-26T16:30:40.240060+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:28:31.422182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:31.422182+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.588303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018006140000000001,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.a","version":"515'51","reported_seq":1033,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239382+0000","last_change":"2024-04-26T16:30:31.542354+0000","last_active":"2024-04-26T16:30:40.239382+0000","last_peered":"2024-04-26T16:30:40.239382+0000","last_clean":"2024-04-26T16:30:40.239382+0000","last_became_active":"2024-04-26T16:26:27.141171+0000","last_became_peered":"2024-04-26T16:26:27.141171+0000","last_unstale":"2024-04-26T16:30:40.239382+0000","last_undegraded":"2024-04-26T16:30:40.239382+0000","last_fullsized":"2024-04-26T16:30:40.239382+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'44","last_scrub_stamp":"2024-04-26T16:30:31.542113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.542113+0000","objects_scrubbed":5,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.941519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020446129999999998,"stat_sum":{"num_bytes":53,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.b","version":"236'34","reported_seq":989,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241547+0000","last_change":"2024-04-26T16:30:00.932705+0000","last_active":"2024-04-26T16:30:40.241547+0000","last_peered":"2024-04-26T16:30:40.241547+0000","last_clean":"2024-04-26T16:30:40.241547+0000","last_became_active":"2024-04-26T16:28:25.233607+0000","last_became_peered":"2024-04-26T16:28:25.233607+0000","last_unstale":"2024-04-26T16:30:40.241547+0000","last_undegraded":"2024-04-26T16:30:40.241547+0000","last_fullsized":"2024-04-26T16:30:40.241547+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:30:00.932368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.932368+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:06.317760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049354960000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.8","version":"389'55","reported_seq":1046,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237050+0000","last_change":"2024-04-26T16:30:11.507964+0000","last_active":"2024-04-26T16:30:40.237050+0000","last_peered":"2024-04-26T16:30:40.237050+0000","last_clean":"2024-04-26T16:30:40.237050+0000","last_became_active":"2024-04-26T16:28:25.245886+0000","last_became_peered":"2024-04-26T16:28:25.245886+0000","last_unstale":"2024-04-26T16:30:40.237050+0000","last_undegraded":"2024-04-26T16:30:40.237050+0000","last_fullsized":"2024-04-26T16:30:40.237050+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'55","last_scrub_stamp":"2024-04-26T16:30:11.507584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.507584+0000","objects_scrubbed":5,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.911256+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001167427,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.9","version":"224'34","reported_seq":973,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258342+0000","last_change":"2024-04-26T16:29:29.284837+0000","last_active":"2024-04-26T16:30:40.258342+0000","last_peered":"2024-04-26T16:30:40.258342+0000","last_clean":"2024-04-26T16:30:40.258342+0000","last_became_active":"2024-04-26T16:28:25.245235+0000","last_became_peered":"2024-04-26T16:28:25.245235+0000","last_unstale":"2024-04-26T16:30:40.258342+0000","last_undegraded":"2024-04-26T16:30:40.258342+0000","last_fullsized":"2024-04-26T16:30:40.258342+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:28:54.109529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.109529+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.963024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0053844380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.e","version":"300'32","reported_seq":1053,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260695+0000","last_change":"2024-04-26T16:30:00.904554+0000","last_active":"2024-04-26T16:30:40.260695+0000","last_peered":"2024-04-26T16:30:40.260695+0000","last_clean":"2024-04-26T16:30:40.260695+0000","last_became_active":"2024-04-26T16:26:23.214391+0000","last_became_peered":"2024-04-26T16:26:23.214391+0000","last_unstale":"2024-04-26T16:30:40.260695+0000","last_undegraded":"2024-04-26T16:30:40.260695+0000","last_fullsized":"2024-04-26T16:30:40.260695+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-04-26T16:30:00.904197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.904197+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.604914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00183567,"stat_sum":{"num_bytes":4194328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":12,"num_write_kb":8202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.f","version":"442'78","reported_seq":1062,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242028+0000","last_change":"2024-04-26T16:29:29.290022+0000","last_active":"2024-04-26T16:30:40.242028+0000","last_peered":"2024-04-26T16:30:40.242028+0000","last_clean":"2024-04-26T16:30:40.242028+0000","last_became_active":"2024-04-26T16:26:23.198449+0000","last_became_peered":"2024-04-26T16:26:23.198449+0000","last_unstale":"2024-04-26T16:30:40.242028+0000","last_undegraded":"2024-04-26T16:30:40.242028+0000","last_fullsized":"2024-04-26T16:30:40.242028+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:28:56.606556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:56.606556+0000","objects_scrubbed":7,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.588963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001552419,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.c","version":"389'35","reported_seq":977,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243675+0000","last_change":"2024-04-26T16:30:16.958876+0000","last_active":"2024-04-26T16:30:40.243675+0000","last_peered":"2024-04-26T16:30:40.243675+0000","last_clean":"2024-04-26T16:30:40.243675+0000","last_became_active":"2024-04-26T16:23:44.106600+0000","last_became_peered":"2024-04-26T16:23:44.106600+0000","last_unstale":"2024-04-26T16:30:40.243675+0000","last_undegraded":"2024-04-26T16:30:40.243675+0000","last_fullsized":"2024-04-26T16:30:40.243675+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:16.958732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.958732+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.989385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043915680000000002,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.d","version":"464'22","reported_seq":766,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260552+0000","last_change":"2024-04-26T16:29:29.289068+0000","last_active":"2024-04-26T16:30:40.260552+0000","last_peered":"2024-04-26T16:30:40.260552+0000","last_clean":"2024-04-26T16:30:40.260552+0000","last_became_active":"2024-04-26T16:28:25.242455+0000","last_became_peered":"2024-04-26T16:28:25.242455+0000","last_unstale":"2024-04-26T16:30:40.260552+0000","last_undegraded":"2024-04-26T16:30:40.260552+0000","last_fullsized":"2024-04-26T16:30:40.260552+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'22","last_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_deep_scrub":"464'22","last_deep_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.003905+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.963166+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00091357000000000005,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":432,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.246432+0000","last_change":"2024-04-26T16:28:55.739387+0000","last_active":"2024-04-26T16:30:40.246432+0000","last_peered":"2024-04-26T16:30:40.246432+0000","last_clean":"2024-04-26T16:30:40.246432+0000","last_became_active":"2024-04-26T16:26:23.194196+0000","last_became_peered":"2024-04-26T16:26:23.194196+0000","last_unstale":"2024-04-26T16:30:40.246432+0000","last_undegraded":"2024-04-26T16:30:40.246432+0000","last_fullsized":"2024-04-26T16:30:40.246432+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:55.739153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:55.739153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"433'14","reported_seq":1110,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247981+0000","last_change":"2024-04-26T16:29:10.136335+0000","last_active":"2024-04-26T16:30:40.247981+0000","last_peered":"2024-04-26T16:30:40.247981+0000","last_clean":"2024-04-26T16:30:40.247981+0000","last_became_active":"2024-04-26T16:28:25.267969+0000","last_became_peered":"2024-04-26T16:28:25.267969+0000","last_unstale":"2024-04-26T16:30:40.247981+0000","last_undegraded":"2024-04-26T16:30:40.247981+0000","last_fullsized":"2024-04-26T16:30:40.247981+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.136172+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"458'34","reported_seq":1003,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251442+0000","last_change":"2024-04-26T16:28:50.981113+0000","last_active":"2024-04-26T16:30:40.251442+0000","last_peered":"2024-04-26T16:30:40.251442+0000","last_clean":"2024-04-26T16:30:40.251442+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:30:40.251442+0000","last_undegraded":"2024-04-26T16:30:40.251442+0000","last_fullsized":"2024-04-26T16:30:40.251442+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.980879+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879718+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":945,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243439+0000","last_change":"2024-04-26T16:30:08.452481+0000","last_active":"2024-04-26T16:30:40.243439+0000","last_peered":"2024-04-26T16:30:40.243439+0000","last_clean":"2024-04-26T16:30:40.243439+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:30:40.243439+0000","last_undegraded":"2024-04-26T16:30:40.243439+0000","last_fullsized":"2024-04-26T16:30:40.243439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.452371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.452371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.819274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":329,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488763+0000","last_change":"2024-04-26T16:29:03.162231+0000","last_active":"2024-04-26T16:30:47.488763+0000","last_peered":"2024-04-26T16:30:47.488763+0000","last_clean":"2024-04-26T16:30:47.488763+0000","last_became_active":"2024-04-26T16:28:25.238286+0000","last_became_peered":"2024-04-26T16:28:25.238286+0000","last_unstale":"2024-04-26T16:30:47.488763+0000","last_undegraded":"2024-04-26T16:30:47.488763+0000","last_fullsized":"2024-04-26T16:30:47.488763+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.162088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.162088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:49.130787+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":788,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.248955+0000","last_change":"2024-04-26T16:28:25.277790+0000","last_active":"2024-04-26T16:30:40.248955+0000","last_peered":"2024-04-26T16:30:40.248955+0000","last_clean":"2024-04-26T16:30:40.248955+0000","last_became_active":"2024-04-26T16:28:25.277260+0000","last_became_peered":"2024-04-26T16:28:25.277260+0000","last_unstale":"2024-04-26T16:30:40.248955+0000","last_undegraded":"2024-04-26T16:30:40.248955+0000","last_fullsized":"2024-04-26T16:30:40.248955+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:28.909860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.909860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"388'20","reported_seq":494,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260085+0000","last_change":"2024-04-26T16:28:34.168264+0000","last_active":"2024-04-26T16:30:40.260085+0000","last_peered":"2024-04-26T16:30:40.260085+0000","last_clean":"2024-04-26T16:30:40.260085+0000","last_became_active":"2024-04-26T16:28:25.257143+0000","last_became_peered":"2024-04-26T16:28:25.257143+0000","last_unstale":"2024-04-26T16:30:40.260085+0000","last_undegraded":"2024-04-26T16:30:40.260085+0000","last_fullsized":"2024-04-26T16:30:40.260085+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:34.168079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:34.168079+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":946,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.244919+0000","last_change":"2024-04-26T16:30:15.948290+0000","last_active":"2024-04-26T16:30:40.244919+0000","last_peered":"2024-04-26T16:30:40.244919+0000","last_clean":"2024-04-26T16:30:40.244919+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:30:40.244919+0000","last_undegraded":"2024-04-26T16:30:40.244919+0000","last_fullsized":"2024-04-26T16:30:40.244919+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.948187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.948187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:30.824244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":938,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240518+0000","last_change":"2024-04-26T16:29:02.962820+0000","last_active":"2024-04-26T16:30:40.240518+0000","last_peered":"2024-04-26T16:30:40.240518+0000","last_clean":"2024-04-26T16:30:40.240518+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:30:40.240518+0000","last_undegraded":"2024-04-26T16:30:40.240518+0000","last_fullsized":"2024-04-26T16:30:40.240518+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:02.962666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.962666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":944,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.255173+0000","last_change":"2024-04-26T16:30:26.632966+0000","last_active":"2024-04-26T16:30:40.255173+0000","last_peered":"2024-04-26T16:30:40.255173+0000","last_clean":"2024-04-26T16:30:40.255173+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:30:40.255173+0000","last_undegraded":"2024-04-26T16:30:40.255173+0000","last_fullsized":"2024-04-26T16:30:40.255173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.632648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:33.662364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":794,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250401+0000","last_change":"2024-04-26T16:28:25.270759+0000","last_active":"2024-04-26T16:30:40.250401+0000","last_peered":"2024-04-26T16:30:40.250401+0000","last_clean":"2024-04-26T16:30:40.250401+0000","last_became_active":"2024-04-26T16:28:25.270561+0000","last_became_peered":"2024-04-26T16:28:25.270561+0000","last_unstale":"2024-04-26T16:30:40.250401+0000","last_undegraded":"2024-04-26T16:30:40.250401+0000","last_fullsized":"2024-04-26T16:30:40.250401+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:35.687528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.687528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.878887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"388'31","reported_seq":1013,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250439+0000","last_change":"2024-04-26T16:28:44.986410+0000","last_active":"2024-04-26T16:30:40.250439+0000","last_peered":"2024-04-26T16:30:40.250439+0000","last_clean":"2024-04-26T16:30:40.250439+0000","last_became_active":"2024-04-26T16:28:25.270747+0000","last_became_peered":"2024-04-26T16:28:25.270747+0000","last_unstale":"2024-04-26T16:30:40.250439+0000","last_undegraded":"2024-04-26T16:30:40.250439+0000","last_fullsized":"2024-04-26T16:30:40.250439+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'31","last_scrub_stamp":"2024-04-26T16:28:44.986040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.986040+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":951,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241892+0000","last_change":"2024-04-26T16:28:53.965461+0000","last_active":"2024-04-26T16:30:40.241892+0000","last_peered":"2024-04-26T16:30:40.241892+0000","last_clean":"2024-04-26T16:30:40.241892+0000","last_became_active":"2024-04-26T16:26:23.201706+0000","last_became_peered":"2024-04-26T16:26:23.201706+0000","last_unstale":"2024-04-26T16:30:40.241892+0000","last_undegraded":"2024-04-26T16:30:40.241892+0000","last_fullsized":"2024-04-26T16:30:40.241892+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:53.965330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:53.965330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":943,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241806+0000","last_change":"2024-04-26T16:30:03.454481+0000","last_active":"2024-04-26T16:30:40.241806+0000","last_peered":"2024-04-26T16:30:40.241806+0000","last_clean":"2024-04-26T16:30:40.241806+0000","last_became_active":"2024-04-26T16:26:23.172602+0000","last_became_peered":"2024-04-26T16:26:23.172602+0000","last_unstale":"2024-04-26T16:30:40.241806+0000","last_undegraded":"2024-04-26T16:30:40.241806+0000","last_fullsized":"2024-04-26T16:30:40.241806+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:03.454165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.454165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:11.847181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":916,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247368+0000","last_change":"2024-04-26T16:26:31.674001+0000","last_active":"2024-04-26T16:30:40.247368+0000","last_peered":"2024-04-26T16:30:40.247368+0000","last_clean":"2024-04-26T16:30:40.247368+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:30:40.247368+0000","last_undegraded":"2024-04-26T16:30:40.247368+0000","last_fullsized":"2024-04-26T16:30:40.247368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:31.673774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:26:31.673774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.878994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":789,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242167+0000","last_change":"2024-04-26T16:30:06.484249+0000","last_active":"2024-04-26T16:30:40.242167+0000","last_peered":"2024-04-26T16:30:40.242167+0000","last_clean":"2024-04-26T16:30:40.242167+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:30:40.242167+0000","last_undegraded":"2024-04-26T16:30:40.242167+0000","last_fullsized":"2024-04-26T16:30:40.242167+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:06.484028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:06.484028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.768744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"389'35","reported_seq":937,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488365+0000","last_change":"2024-04-26T16:30:11.460779+0000","last_active":"2024-04-26T16:30:47.488365+0000","last_peered":"2024-04-26T16:30:47.488365+0000","last_clean":"2024-04-26T16:30:47.488365+0000","last_became_active":"2024-04-26T16:28:25.247148+0000","last_became_peered":"2024-04-26T16:28:25.247148+0000","last_unstale":"2024-04-26T16:30:47.488365+0000","last_undegraded":"2024-04-26T16:30:47.488365+0000","last_fullsized":"2024-04-26T16:30:47.488365+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:11.460648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.460648+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.462457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":941,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250483+0000","last_change":"2024-04-26T16:30:01.188873+0000","last_active":"2024-04-26T16:30:40.250483+0000","last_peered":"2024-04-26T16:30:40.250483+0000","last_clean":"2024-04-26T16:30:40.250483+0000","last_became_active":"2024-04-26T16:26:23.199164+0000","last_became_peered":"2024-04-26T16:26:23.199164+0000","last_unstale":"2024-04-26T16:30:40.250483+0000","last_undegraded":"2024-04-26T16:30:40.250483+0000","last_fullsized":"2024-04-26T16:30:40.250483+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:01.188654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:30:01.188654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.499774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":946,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.244692+0000","last_change":"2024-04-26T16:30:29.051438+0000","last_active":"2024-04-26T16:30:40.244692+0000","last_peered":"2024-04-26T16:30:40.244692+0000","last_clean":"2024-04-26T16:30:40.244692+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:30:40.244692+0000","last_undegraded":"2024-04-26T16:30:40.244692+0000","last_fullsized":"2024-04-26T16:30:40.244692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.051284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.626968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":933,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243651+0000","last_change":"2024-04-26T16:29:59.407848+0000","last_active":"2024-04-26T16:30:40.243651+0000","last_peered":"2024-04-26T16:30:40.243651+0000","last_clean":"2024-04-26T16:30:40.243651+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:30:40.243651+0000","last_undegraded":"2024-04-26T16:30:40.243651+0000","last_fullsized":"2024-04-26T16:30:40.243651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:59.407460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:29:59.407460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:20.521427+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":695,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488883+0000","last_change":"2024-04-26T16:30:00.494762+0000","last_active":"2024-04-26T16:30:47.488883+0000","last_peered":"2024-04-26T16:30:47.488883+0000","last_clean":"2024-04-26T16:30:47.488883+0000","last_became_active":"2024-04-26T16:28:25.265097+0000","last_became_peered":"2024-04-26T16:28:25.265097+0000","last_unstale":"2024-04-26T16:30:47.488883+0000","last_undegraded":"2024-04-26T16:30:47.488883+0000","last_fullsized":"2024-04-26T16:30:47.488883+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.494544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.494544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.559293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"388'19","reported_seq":961,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259840+0000","last_change":"2024-04-26T16:30:34.982606+0000","last_active":"2024-04-26T16:30:40.259840+0000","last_peered":"2024-04-26T16:30:40.259840+0000","last_clean":"2024-04-26T16:30:40.259840+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:30:40.259840+0000","last_undegraded":"2024-04-26T16:30:40.259840+0000","last_fullsized":"2024-04-26T16:30:40.259840+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:34.982478+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.982478+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:59.881026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":947,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240707+0000","last_change":"2024-04-26T16:29:56.955956+0000","last_active":"2024-04-26T16:30:40.240707+0000","last_peered":"2024-04-26T16:30:40.240707+0000","last_clean":"2024-04-26T16:30:40.240707+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:30:40.240707+0000","last_undegraded":"2024-04-26T16:30:40.240707+0000","last_fullsized":"2024-04-26T16:30:40.240707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:56.955662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.955662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:09.215855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"316'12","reported_seq":1117,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242131+0000","last_change":"2024-04-26T16:29:06.035849+0000","last_active":"2024-04-26T16:30:40.242131+0000","last_peered":"2024-04-26T16:30:40.242131+0000","last_clean":"2024-04-26T16:30:40.242131+0000","last_became_active":"2024-04-26T16:26:23.194072+0000","last_became_peered":"2024-04-26T16:26:23.194072+0000","last_unstale":"2024-04-26T16:30:40.242131+0000","last_undegraded":"2024-04-26T16:30:40.242131+0000","last_fullsized":"2024-04-26T16:30:40.242131+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'12","last_scrub_stamp":"2024-04-26T16:29:06.035584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:06.035584+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":934,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241470+0000","last_change":"2024-04-26T16:28:59.599961+0000","last_active":"2024-04-26T16:30:40.241470+0000","last_peered":"2024-04-26T16:30:40.241470+0000","last_clean":"2024-04-26T16:30:40.241470+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:30:40.241470+0000","last_undegraded":"2024-04-26T16:30:40.241470+0000","last_fullsized":"2024-04-26T16:30:40.241470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.599808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.599808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":788,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239856+0000","last_change":"2024-04-26T16:29:01.807303+0000","last_active":"2024-04-26T16:30:40.239856+0000","last_peered":"2024-04-26T16:30:40.239856+0000","last_clean":"2024-04-26T16:30:40.239856+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:30:40.239856+0000","last_undegraded":"2024-04-26T16:30:40.239856+0000","last_fullsized":"2024-04-26T16:30:40.239856+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:01.807199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:01.807199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"388'27","reported_seq":976,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258701+0000","last_change":"2024-04-26T16:30:21.978132+0000","last_active":"2024-04-26T16:30:40.258701+0000","last_peered":"2024-04-26T16:30:40.258701+0000","last_clean":"2024-04-26T16:30:40.258701+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:30:40.258701+0000","last_undegraded":"2024-04-26T16:30:40.258701+0000","last_fullsized":"2024-04-26T16:30:40.258701+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:21.978007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.978007+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.780831+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":961,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240454+0000","last_change":"2024-04-26T16:30:35.051954+0000","last_active":"2024-04-26T16:30:40.240454+0000","last_peered":"2024-04-26T16:30:40.240454+0000","last_clean":"2024-04-26T16:30:40.240454+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:30:40.240454+0000","last_undegraded":"2024-04-26T16:30:40.240454+0000","last_fullsized":"2024-04-26T16:30:40.240454+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.051759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:47.746481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"316'13","reported_seq":1119,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241146+0000","last_change":"2024-04-26T16:30:14.531293+0000","last_active":"2024-04-26T16:30:40.241146+0000","last_peered":"2024-04-26T16:30:40.241146+0000","last_clean":"2024-04-26T16:30:40.241146+0000","last_became_active":"2024-04-26T16:26:23.180811+0000","last_became_peered":"2024-04-26T16:26:23.180811+0000","last_unstale":"2024-04-26T16:30:40.241146+0000","last_undegraded":"2024-04-26T16:30:40.241146+0000","last_fullsized":"2024-04-26T16:30:40.241146+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:14.531205+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.531205+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:25.731439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":440,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243256+0000","last_change":"2024-04-26T16:30:25.049754+0000","last_active":"2024-04-26T16:30:40.243256+0000","last_peered":"2024-04-26T16:30:40.243256+0000","last_clean":"2024-04-26T16:30:40.243256+0000","last_became_active":"2024-04-26T16:26:23.195438+0000","last_became_peered":"2024-04-26T16:26:23.195438+0000","last_unstale":"2024-04-26T16:30:40.243256+0000","last_undegraded":"2024-04-26T16:30:40.243256+0000","last_fullsized":"2024-04-26T16:30:40.243256+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.049647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.049647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.587456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":712,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488174+0000","last_change":"2024-04-26T16:30:07.526553+0000","last_active":"2024-04-26T16:30:47.488174+0000","last_peered":"2024-04-26T16:30:47.488174+0000","last_clean":"2024-04-26T16:30:47.488174+0000","last_became_active":"2024-04-26T16:28:25.244320+0000","last_became_peered":"2024-04-26T16:28:25.244320+0000","last_unstale":"2024-04-26T16:30:47.488174+0000","last_undegraded":"2024-04-26T16:30:47.488174+0000","last_fullsized":"2024-04-26T16:30:47.488174+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:30:07.526320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.526320+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:20.026339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"399'20","reported_seq":977,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238385+0000","last_change":"2024-04-26T16:30:21.500286+0000","last_active":"2024-04-26T16:30:40.238385+0000","last_peered":"2024-04-26T16:30:40.238385+0000","last_clean":"2024-04-26T16:30:40.238385+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:30:40.238385+0000","last_undegraded":"2024-04-26T16:30:40.238385+0000","last_fullsized":"2024-04-26T16:30:40.238385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'20","last_scrub_stamp":"2024-04-26T16:30:21.500166+0000","last_deep_scrub":"399'20","last_deep_scrub_stamp":"2024-04-26T16:27:47.205374+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.500166+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.067680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":944,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247728+0000","last_change":"2024-04-26T16:28:46.987810+0000","last_active":"2024-04-26T16:30:40.247728+0000","last_peered":"2024-04-26T16:30:40.247728+0000","last_clean":"2024-04-26T16:30:40.247728+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:30:40.247728+0000","last_undegraded":"2024-04-26T16:30:40.247728+0000","last_fullsized":"2024-04-26T16:30:40.247728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:28:46.987686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.987686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":448,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242952+0000","last_change":"2024-04-26T16:30:23.518613+0000","last_active":"2024-04-26T16:30:40.242952+0000","last_peered":"2024-04-26T16:30:40.242952+0000","last_clean":"2024-04-26T16:30:40.242952+0000","last_became_active":"2024-04-26T16:28:25.248459+0000","last_became_peered":"2024-04-26T16:28:25.248459+0000","last_unstale":"2024-04-26T16:30:40.242952+0000","last_undegraded":"2024-04-26T16:30:40.242952+0000","last_fullsized":"2024-04-26T16:30:40.242952+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.518326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.518326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.755565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":939,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260269+0000","last_change":"2024-04-26T16:30:35.938334+0000","last_active":"2024-04-26T16:30:40.260269+0000","last_peered":"2024-04-26T16:30:40.260269+0000","last_clean":"2024-04-26T16:30:40.260269+0000","last_became_active":"2024-04-26T16:28:25.257629+0000","last_became_peered":"2024-04-26T16:28:25.257629+0000","last_unstale":"2024-04-26T16:30:40.260269+0000","last_undegraded":"2024-04-26T16:30:40.260269+0000","last_fullsized":"2024-04-26T16:30:40.260269+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.938129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.938129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.462202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":788,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241065+0000","last_change":"2024-04-26T16:28:55.586673+0000","last_active":"2024-04-26T16:30:40.241065+0000","last_peered":"2024-04-26T16:30:40.241065+0000","last_clean":"2024-04-26T16:30:40.241065+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:30:40.241065+0000","last_undegraded":"2024-04-26T16:30:40.241065+0000","last_fullsized":"2024-04-26T16:30:40.241065+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:55.586453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:55.586453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589534+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1092,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259010+0000","last_change":"2024-04-26T16:28:37.207675+0000","last_active":"2024-04-26T16:30:40.259010+0000","last_peered":"2024-04-26T16:30:40.259010+0000","last_clean":"2024-04-26T16:30:40.259010+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:30:40.259010+0000","last_undegraded":"2024-04-26T16:30:40.259010+0000","last_fullsized":"2024-04-26T16:30:40.259010+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:28:37.207601+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:28:37.207601+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":963,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238593+0000","last_change":"2024-04-26T16:28:44.503061+0000","last_active":"2024-04-26T16:30:40.238593+0000","last_peered":"2024-04-26T16:30:40.238593+0000","last_clean":"2024-04-26T16:30:40.238593+0000","last_became_active":"2024-04-26T16:28:25.250134+0000","last_became_peered":"2024-04-26T16:28:25.250134+0000","last_unstale":"2024-04-26T16:30:40.238593+0000","last_undegraded":"2024-04-26T16:30:40.238593+0000","last_fullsized":"2024-04-26T16:30:40.238593+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:44.502708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:13.552494+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.502708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'30","reported_seq":986,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.256916+0000","last_change":"2024-04-26T16:28:50.453707+0000","last_active":"2024-04-26T16:30:40.256916+0000","last_peered":"2024-04-26T16:30:40.256916+0000","last_clean":"2024-04-26T16:30:40.256916+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:30:40.256916+0000","last_undegraded":"2024-04-26T16:30:40.256916+0000","last_fullsized":"2024-04-26T16:30:40.256916+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'30","last_scrub_stamp":"2024-04-26T16:28:50.453456+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.453456+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":846,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.489677+0000","last_change":"2024-04-26T16:30:33.268356+0000","last_active":"2024-04-26T16:30:47.489677+0000","last_peered":"2024-04-26T16:30:47.489677+0000","last_clean":"2024-04-26T16:30:47.489677+0000","last_became_active":"2024-04-26T16:28:25.265921+0000","last_became_peered":"2024-04-26T16:28:25.265921+0000","last_unstale":"2024-04-26T16:30:47.489677+0000","last_undegraded":"2024-04-26T16:30:47.489677+0000","last_fullsized":"2024-04-26T16:30:47.489677+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.268035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.268035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.106859+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":933,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249666+0000","last_change":"2024-04-26T16:28:38.012269+0000","last_active":"2024-04-26T16:30:40.249666+0000","last_peered":"2024-04-26T16:30:40.249666+0000","last_clean":"2024-04-26T16:30:40.249666+0000","last_became_active":"2024-04-26T16:28:25.270748+0000","last_became_peered":"2024-04-26T16:28:25.270748+0000","last_unstale":"2024-04-26T16:30:40.249666+0000","last_undegraded":"2024-04-26T16:30:40.249666+0000","last_fullsized":"2024-04-26T16:30:40.249666+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:38.011966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:38.011966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":787,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251535+0000","last_change":"2024-04-26T16:30:08.292894+0000","last_active":"2024-04-26T16:30:40.251535+0000","last_peered":"2024-04-26T16:30:40.251535+0000","last_clean":"2024-04-26T16:30:40.251535+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:30:40.251535+0000","last_undegraded":"2024-04-26T16:30:40.251535+0000","last_fullsized":"2024-04-26T16:30:40.251535+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.292589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.292589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:22.177871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":939,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238380+0000","last_change":"2024-04-26T16:29:09.485874+0000","last_active":"2024-04-26T16:30:40.238380+0000","last_peered":"2024-04-26T16:30:40.238380+0000","last_clean":"2024-04-26T16:30:40.238380+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:30:40.238380+0000","last_undegraded":"2024-04-26T16:30:40.238380+0000","last_fullsized":"2024-04-26T16:30:40.238380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.485662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.485662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589619+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":1716,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:44.772666+0000","last_change":"2024-04-26T16:30:24.906077+0000","last_active":"2024-04-26T16:30:44.772666+0000","last_peered":"2024-04-26T16:30:44.772666+0000","last_clean":"2024-04-26T16:30:44.772666+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:30:44.772666+0000","last_undegraded":"2024-04-26T16:30:44.772666+0000","last_fullsized":"2024-04-26T16:30:44.772666+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:30:24.905850+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.905850+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:29.141938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":775,"num_read_kb":770,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'22","reported_seq":971,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260995+0000","last_change":"2024-04-26T16:30:19.960354+0000","last_active":"2024-04-26T16:30:40.260995+0000","last_peered":"2024-04-26T16:30:40.260995+0000","last_clean":"2024-04-26T16:30:40.260995+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:30:40.260995+0000","last_undegraded":"2024-04-26T16:30:40.260995+0000","last_fullsized":"2024-04-26T16:30:40.260995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:30:19.960123+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.960123+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:22.536698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":958,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.246021+0000","last_change":"2024-04-26T16:29:08.743269+0000","last_active":"2024-04-26T16:30:40.246021+0000","last_peered":"2024-04-26T16:30:40.246021+0000","last_clean":"2024-04-26T16:30:40.246021+0000","last_became_active":"2024-04-26T16:28:25.233774+0000","last_became_peered":"2024-04-26T16:28:25.233774+0000","last_unstale":"2024-04-26T16:30:40.246021+0000","last_undegraded":"2024-04-26T16:30:40.246021+0000","last_fullsized":"2024-04-26T16:30:40.246021+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:29:08.742955+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.742955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":935,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239112+0000","last_change":"2024-04-26T16:28:40.455517+0000","last_active":"2024-04-26T16:30:40.239112+0000","last_peered":"2024-04-26T16:30:40.239112+0000","last_clean":"2024-04-26T16:30:40.239112+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:30:40.239112+0000","last_undegraded":"2024-04-26T16:30:40.239112+0000","last_fullsized":"2024-04-26T16:30:40.239112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.455422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.455422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":788,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.244114+0000","last_change":"2024-04-26T16:30:12.954476+0000","last_active":"2024-04-26T16:30:40.244114+0000","last_peered":"2024-04-26T16:30:40.244114+0000","last_clean":"2024-04-26T16:30:40.244114+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:30:40.244114+0000","last_undegraded":"2024-04-26T16:30:40.244114+0000","last_fullsized":"2024-04-26T16:30:40.244114+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.954265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.954265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:23.703295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":838,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488524+0000","last_change":"2024-04-26T16:30:25.328753+0000","last_active":"2024-04-26T16:30:47.488524+0000","last_peered":"2024-04-26T16:30:47.488524+0000","last_clean":"2024-04-26T16:30:47.488524+0000","last_became_active":"2024-04-26T16:28:25.234888+0000","last_became_peered":"2024-04-26T16:28:25.234888+0000","last_unstale":"2024-04-26T16:30:47.488524+0000","last_undegraded":"2024-04-26T16:30:47.488524+0000","last_fullsized":"2024-04-26T16:30:47.488524+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.328546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.328546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:53.320476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"388'53","reported_seq":1209,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.244319+0000","last_change":"2024-04-26T16:29:10.751587+0000","last_active":"2024-04-26T16:30:40.244319+0000","last_peered":"2024-04-26T16:30:40.244319+0000","last_clean":"2024-04-26T16:30:40.244319+0000","last_became_active":"2024-04-26T16:26:23.203408+0000","last_became_peered":"2024-04-26T16:26:23.203408+0000","last_unstale":"2024-04-26T16:30:40.244319+0000","last_undegraded":"2024-04-26T16:30:40.244319+0000","last_fullsized":"2024-04-26T16:30:40.244319+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'53","last_scrub_stamp":"2024-04-26T16:29:10.751474+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.751474+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":945,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249682+0000","last_change":"2024-04-26T16:29:03.080654+0000","last_active":"2024-04-26T16:30:40.249682+0000","last_peered":"2024-04-26T16:30:40.249682+0000","last_clean":"2024-04-26T16:30:40.249682+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:30:40.249682+0000","last_undegraded":"2024-04-26T16:30:40.249682+0000","last_fullsized":"2024-04-26T16:30:40.249682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.080403+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.880101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":952,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240725+0000","last_change":"2024-04-26T16:30:28.037331+0000","last_active":"2024-04-26T16:30:40.240725+0000","last_peered":"2024-04-26T16:30:40.240725+0000","last_clean":"2024-04-26T16:30:40.240725+0000","last_became_active":"2024-04-26T16:28:25.242325+0000","last_became_peered":"2024-04-26T16:28:25.242325+0000","last_unstale":"2024-04-26T16:30:40.240725+0000","last_undegraded":"2024-04-26T16:30:40.240725+0000","last_fullsized":"2024-04-26T16:30:40.240725+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.037115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.037115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:56.639017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":789,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259356+0000","last_change":"2024-04-26T16:29:58.933284+0000","last_active":"2024-04-26T16:30:40.259356+0000","last_peered":"2024-04-26T16:30:40.259356+0000","last_clean":"2024-04-26T16:30:40.259356+0000","last_became_active":"2024-04-26T16:28:25.245871+0000","last_became_peered":"2024-04-26T16:28:25.245871+0000","last_unstale":"2024-04-26T16:30:40.259356+0000","last_undegraded":"2024-04-26T16:30:40.259356+0000","last_fullsized":"2024-04-26T16:30:40.259356+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.933163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.933163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:22.949904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":440,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.246072+0000","last_change":"2024-04-26T16:30:24.025255+0000","last_active":"2024-04-26T16:30:40.246072+0000","last_peered":"2024-04-26T16:30:40.246072+0000","last_clean":"2024-04-26T16:30:40.246072+0000","last_became_active":"2024-04-26T16:26:23.191179+0000","last_became_peered":"2024-04-26T16:26:23.191179+0000","last_unstale":"2024-04-26T16:30:40.246072+0000","last_undegraded":"2024-04-26T16:30:40.246072+0000","last_fullsized":"2024-04-26T16:30:40.246072+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.025163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.025163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:27.212243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":936,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247373+0000","last_change":"2024-04-26T16:28:59.045623+0000","last_active":"2024-04-26T16:30:40.247373+0000","last_peered":"2024-04-26T16:30:40.247373+0000","last_clean":"2024-04-26T16:30:40.247373+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:30:40.247373+0000","last_undegraded":"2024-04-26T16:30:40.247373+0000","last_fullsized":"2024-04-26T16:30:40.247373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.045415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.045415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'35","reported_seq":993,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.255479+0000","last_change":"2024-04-26T16:29:02.435596+0000","last_active":"2024-04-26T16:30:40.255479+0000","last_peered":"2024-04-26T16:30:40.255479+0000","last_clean":"2024-04-26T16:30:40.255479+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:30:40.255479+0000","last_undegraded":"2024-04-26T16:30:40.255479+0000","last_fullsized":"2024-04-26T16:30:40.255479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:29:02.435425+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.435425+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":942,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.244947+0000","last_change":"2024-04-26T16:28:32.981894+0000","last_active":"2024-04-26T16:30:40.244947+0000","last_peered":"2024-04-26T16:30:40.244947+0000","last_clean":"2024-04-26T16:30:40.244947+0000","last_became_active":"2024-04-26T16:28:25.253094+0000","last_became_peered":"2024-04-26T16:28:25.253094+0000","last_unstale":"2024-04-26T16:30:40.244947+0000","last_undegraded":"2024-04-26T16:30:40.244947+0000","last_fullsized":"2024-04-26T16:30:40.244947+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:32.981795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:28:32.981795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010071+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":795,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259028+0000","last_change":"2024-04-26T16:28:58.073423+0000","last_active":"2024-04-26T16:30:40.259028+0000","last_peered":"2024-04-26T16:30:40.259028+0000","last_clean":"2024-04-26T16:30:40.259028+0000","last_became_active":"2024-04-26T16:28:25.231951+0000","last_became_peered":"2024-04-26T16:28:25.231951+0000","last_unstale":"2024-04-26T16:30:40.259028+0000","last_undegraded":"2024-04-26T16:30:40.259028+0000","last_fullsized":"2024-04-26T16:30:40.259028+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:58.073194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:58.073194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962163+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":934,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250304+0000","last_change":"2024-04-26T16:30:05.254074+0000","last_active":"2024-04-26T16:30:40.250304+0000","last_peered":"2024-04-26T16:30:40.250304+0000","last_clean":"2024-04-26T16:30:40.250304+0000","last_became_active":"2024-04-26T16:26:23.176892+0000","last_became_peered":"2024-04-26T16:26:23.176892+0000","last_unstale":"2024-04-26T16:30:40.250304+0000","last_undegraded":"2024-04-26T16:30:40.250304+0000","last_fullsized":"2024-04-26T16:30:40.250304+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:05.253814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:30:05.253814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:12.490234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1224,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247729+0000","last_change":"2024-04-26T16:28:25.267719+0000","last_active":"2024-04-26T16:30:40.247729+0000","last_peered":"2024-04-26T16:30:40.247729+0000","last_clean":"2024-04-26T16:30:40.247729+0000","last_became_active":"2024-04-26T16:28:25.267305+0000","last_became_peered":"2024-04-26T16:28:25.267305+0000","last_unstale":"2024-04-26T16:30:40.247729+0000","last_undegraded":"2024-04-26T16:30:40.247729+0000","last_fullsized":"2024-04-26T16:30:40.247729+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:26:29.671147+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:26:29.671147+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.878947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":434,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240266+0000","last_change":"2024-04-26T16:30:09.475518+0000","last_active":"2024-04-26T16:30:40.240266+0000","last_peered":"2024-04-26T16:30:40.240266+0000","last_clean":"2024-04-26T16:30:40.240266+0000","last_became_active":"2024-04-26T16:26:23.201156+0000","last_became_peered":"2024-04-26T16:26:23.201156+0000","last_unstale":"2024-04-26T16:30:40.240266+0000","last_undegraded":"2024-04-26T16:30:40.240266+0000","last_fullsized":"2024-04-26T16:30:40.240266+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.475272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.475272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:21.943110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"388'25","reported_seq":995,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264108+0000","last_change":"2024-04-26T16:28:52.098358+0000","last_active":"2024-04-26T16:30:40.264108+0000","last_peered":"2024-04-26T16:30:40.264108+0000","last_clean":"2024-04-26T16:30:40.264108+0000","last_became_active":"2024-04-26T16:28:25.258417+0000","last_became_peered":"2024-04-26T16:28:25.258417+0000","last_unstale":"2024-04-26T16:30:40.264108+0000","last_undegraded":"2024-04-26T16:30:40.264108+0000","last_fullsized":"2024-04-26T16:30:40.264108+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:28:52.098251+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.098251+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":434,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264152+0000","last_change":"2024-04-26T16:30:14.930711+0000","last_active":"2024-04-26T16:30:40.264152+0000","last_peered":"2024-04-26T16:30:40.264152+0000","last_clean":"2024-04-26T16:30:40.264152+0000","last_became_active":"2024-04-26T16:26:23.212670+0000","last_became_peered":"2024-04-26T16:26:23.212670+0000","last_unstale":"2024-04-26T16:30:40.264152+0000","last_undegraded":"2024-04-26T16:30:40.264152+0000","last_fullsized":"2024-04-26T16:30:40.264152+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.930578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.930578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:28.579424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":795,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258330+0000","last_change":"2024-04-26T16:28:43.126227+0000","last_active":"2024-04-26T16:30:40.258330+0000","last_peered":"2024-04-26T16:30:40.258330+0000","last_clean":"2024-04-26T16:30:40.258330+0000","last_became_active":"2024-04-26T16:28:25.245460+0000","last_became_peered":"2024-04-26T16:28:25.245460+0000","last_unstale":"2024-04-26T16:30:40.258330+0000","last_undegraded":"2024-04-26T16:30:40.258330+0000","last_fullsized":"2024-04-26T16:30:40.258330+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:43.125904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.125904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":944,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243906+0000","last_change":"2024-04-26T16:29:05.010298+0000","last_active":"2024-04-26T16:30:40.243906+0000","last_peered":"2024-04-26T16:30:40.243906+0000","last_clean":"2024-04-26T16:30:40.243906+0000","last_became_active":"2024-04-26T16:26:23.185332+0000","last_became_peered":"2024-04-26T16:26:23.185332+0000","last_unstale":"2024-04-26T16:30:40.243906+0000","last_undegraded":"2024-04-26T16:30:40.243906+0000","last_fullsized":"2024-04-26T16:30:40.243906+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:05.010052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:29:05.010052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624287+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":946,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237518+0000","last_change":"2024-04-26T16:29:03.999225+0000","last_active":"2024-04-26T16:30:40.237518+0000","last_peered":"2024-04-26T16:30:40.237518+0000","last_clean":"2024-04-26T16:30:40.237518+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:30:40.237518+0000","last_undegraded":"2024-04-26T16:30:40.237518+0000","last_fullsized":"2024-04-26T16:30:40.237518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.998965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.998965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'38","reported_seq":1018,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239547+0000","last_change":"2024-04-26T16:30:27.085082+0000","last_active":"2024-04-26T16:30:40.239547+0000","last_peered":"2024-04-26T16:30:40.239547+0000","last_clean":"2024-04-26T16:30:40.239547+0000","last_became_active":"2024-04-26T16:28:25.241767+0000","last_became_peered":"2024-04-26T16:28:25.241767+0000","last_unstale":"2024-04-26T16:30:40.239547+0000","last_undegraded":"2024-04-26T16:30:40.239547+0000","last_fullsized":"2024-04-26T16:30:40.239547+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-04-26T16:30:27.084844+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:27:38.188279+0000","last_clean_scrub_stamp":"2024-04-26T16:30:27.084844+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:41.150355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":933,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249672+0000","last_change":"2024-04-26T16:28:52.021759+0000","last_active":"2024-04-26T16:30:40.249672+0000","last_peered":"2024-04-26T16:30:40.249672+0000","last_clean":"2024-04-26T16:30:40.249672+0000","last_became_active":"2024-04-26T16:28:25.266762+0000","last_became_peered":"2024-04-26T16:28:25.266762+0000","last_unstale":"2024-04-26T16:30:40.249672+0000","last_undegraded":"2024-04-26T16:30:40.249672+0000","last_fullsized":"2024-04-26T16:30:40.249672+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.021548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.021548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":780,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259563+0000","last_change":"2024-04-26T16:29:10.997703+0000","last_active":"2024-04-26T16:30:40.259563+0000","last_peered":"2024-04-26T16:30:40.259563+0000","last_clean":"2024-04-26T16:30:40.259563+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:30:40.259563+0000","last_undegraded":"2024-04-26T16:30:40.259563+0000","last_fullsized":"2024-04-26T16:30:40.259563+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.997614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.997614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":940,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.256829+0000","last_change":"2024-04-26T16:29:03.398269+0000","last_active":"2024-04-26T16:30:40.256829+0000","last_peered":"2024-04-26T16:30:40.256829+0000","last_clean":"2024-04-26T16:30:40.256829+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:30:40.256829+0000","last_undegraded":"2024-04-26T16:30:40.256829+0000","last_fullsized":"2024-04-26T16:30:40.256829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.398123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.398123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'37","reported_seq":1011,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239361+0000","last_change":"2024-04-26T16:28:54.567147+0000","last_active":"2024-04-26T16:30:40.239361+0000","last_peered":"2024-04-26T16:30:40.239361+0000","last_clean":"2024-04-26T16:30:40.239361+0000","last_became_active":"2024-04-26T16:28:25.235694+0000","last_became_peered":"2024-04-26T16:28:25.235694+0000","last_unstale":"2024-04-26T16:30:40.239361+0000","last_undegraded":"2024-04-26T16:30:40.239361+0000","last_fullsized":"2024-04-26T16:30:40.239361+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-04-26T16:28:54.566895+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:28:54.566895+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.566895+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.588752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":441,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240597+0000","last_change":"2024-04-26T16:28:42.796996+0000","last_active":"2024-04-26T16:30:40.240597+0000","last_peered":"2024-04-26T16:30:40.240597+0000","last_clean":"2024-04-26T16:30:40.240597+0000","last_became_active":"2024-04-26T16:28:25.233622+0000","last_became_peered":"2024-04-26T16:28:25.233622+0000","last_unstale":"2024-04-26T16:30:40.240597+0000","last_undegraded":"2024-04-26T16:30:40.240597+0000","last_fullsized":"2024-04-26T16:30:40.240597+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:42.796778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:28:42.796778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":925,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264078+0000","last_change":"2024-04-26T16:28:39.169317+0000","last_active":"2024-04-26T16:30:40.264078+0000","last_peered":"2024-04-26T16:30:40.264078+0000","last_clean":"2024-04-26T16:30:40.264078+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:30:40.264078+0000","last_undegraded":"2024-04-26T16:30:40.264078+0000","last_fullsized":"2024-04-26T16:30:40.264078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:39.169120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:39.169120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961864+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":798,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242087+0000","last_change":"2024-04-26T16:30:19.533168+0000","last_active":"2024-04-26T16:30:40.242087+0000","last_peered":"2024-04-26T16:30:40.242087+0000","last_clean":"2024-04-26T16:30:40.242087+0000","last_became_active":"2024-04-26T16:28:25.241015+0000","last_became_peered":"2024-04-26T16:28:25.241015+0000","last_unstale":"2024-04-26T16:30:40.242087+0000","last_undegraded":"2024-04-26T16:30:40.242087+0000","last_fullsized":"2024-04-26T16:30:40.242087+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.533042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.533042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.752688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"433'16","reported_seq":1136,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.256358+0000","last_change":"2024-04-26T16:28:49.448655+0000","last_active":"2024-04-26T16:30:40.256358+0000","last_peered":"2024-04-26T16:30:40.256358+0000","last_clean":"2024-04-26T16:30:40.256358+0000","last_became_active":"2024-04-26T16:28:25.228833+0000","last_became_peered":"2024-04-26T16:28:25.228833+0000","last_unstale":"2024-04-26T16:30:40.256358+0000","last_undegraded":"2024-04-26T16:30:40.256358+0000","last_fullsized":"2024-04-26T16:30:40.256358+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'16","last_scrub_stamp":"2024-04-26T16:28:49.448423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:49.448423+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":960,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240985+0000","last_change":"2024-04-26T16:30:02.951737+0000","last_active":"2024-04-26T16:30:40.240985+0000","last_peered":"2024-04-26T16:30:40.240985+0000","last_clean":"2024-04-26T16:30:40.240985+0000","last_became_active":"2024-04-26T16:26:23.187980+0000","last_became_peered":"2024-04-26T16:26:23.187980+0000","last_unstale":"2024-04-26T16:30:40.240985+0000","last_undegraded":"2024-04-26T16:30:40.240985+0000","last_fullsized":"2024-04-26T16:30:40.240985+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:30:02.951627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.951627+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:12.328056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"433'39","reported_seq":584,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260788+0000","last_change":"2024-04-26T16:30:08.947640+0000","last_active":"2024-04-26T16:30:40.260788+0000","last_peered":"2024-04-26T16:30:40.260788+0000","last_clean":"2024-04-26T16:30:40.260788+0000","last_became_active":"2024-04-26T16:28:25.228181+0000","last_became_peered":"2024-04-26T16:28:25.228181+0000","last_unstale":"2024-04-26T16:30:40.260788+0000","last_undegraded":"2024-04-26T16:30:40.260788+0000","last_fullsized":"2024-04-26T16:30:40.260788+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'39","last_scrub_stamp":"2024-04-26T16:30:08.947465+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.947465+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.991257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":947,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.244398+0000","last_change":"2024-04-26T16:28:47.879030+0000","last_active":"2024-04-26T16:30:40.244398+0000","last_peered":"2024-04-26T16:30:40.244398+0000","last_clean":"2024-04-26T16:30:40.244398+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:30:40.244398+0000","last_undegraded":"2024-04-26T16:30:40.244398+0000","last_fullsized":"2024-04-26T16:30:40.244398+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:47.878929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.878929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":936,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.253867+0000","last_change":"2024-04-26T16:30:31.628236+0000","last_active":"2024-04-26T16:30:40.253867+0000","last_peered":"2024-04-26T16:30:40.253867+0000","last_clean":"2024-04-26T16:30:40.253867+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:30:40.253867+0000","last_undegraded":"2024-04-26T16:30:40.253867+0000","last_fullsized":"2024-04-26T16:30:40.253867+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:31.627931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.627931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.885855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":791,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.256802+0000","last_change":"2024-04-26T16:29:55.759332+0000","last_active":"2024-04-26T16:30:40.256802+0000","last_peered":"2024-04-26T16:30:40.256802+0000","last_clean":"2024-04-26T16:30:40.256802+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:30:40.256802+0000","last_undegraded":"2024-04-26T16:30:40.256802+0000","last_fullsized":"2024-04-26T16:30:40.256802+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:55.759069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:55.759069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:08.476199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":938,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238061+0000","last_change":"2024-04-26T16:29:01.923625+0000","last_active":"2024-04-26T16:30:40.238061+0000","last_peered":"2024-04-26T16:30:40.238061+0000","last_clean":"2024-04-26T16:30:40.238061+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:30:40.238061+0000","last_undegraded":"2024-04-26T16:30:40.238061+0000","last_fullsized":"2024-04-26T16:30:40.238061+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:29:01.923293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"388'19","reported_seq":970,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251383+0000","last_change":"2024-04-26T16:30:03.191223+0000","last_active":"2024-04-26T16:30:40.251383+0000","last_peered":"2024-04-26T16:30:40.251383+0000","last_clean":"2024-04-26T16:30:40.251383+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:30:40.251383+0000","last_undegraded":"2024-04-26T16:30:40.251383+0000","last_fullsized":"2024-04-26T16:30:40.251383+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:03.190975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.190975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.424209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":970,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242544+0000","last_change":"2024-04-26T16:30:04.581495+0000","last_active":"2024-04-26T16:30:40.242544+0000","last_peered":"2024-04-26T16:30:40.242544+0000","last_clean":"2024-04-26T16:30:40.242544+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:30:40.242544+0000","last_undegraded":"2024-04-26T16:30:40.242544+0000","last_fullsized":"2024-04-26T16:30:40.242544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:30:04.581361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.581361+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.126549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":916,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260021+0000","last_change":"2024-04-26T16:28:40.179575+0000","last_active":"2024-04-26T16:30:40.260021+0000","last_peered":"2024-04-26T16:30:40.260021+0000","last_clean":"2024-04-26T16:30:40.260021+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:30:40.260021+0000","last_undegraded":"2024-04-26T16:30:40.260021+0000","last_fullsized":"2024-04-26T16:30:40.260021+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.179446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":791,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.254447+0000","last_change":"2024-04-26T16:30:03.519972+0000","last_active":"2024-04-26T16:30:40.254447+0000","last_peered":"2024-04-26T16:30:40.254447+0000","last_clean":"2024-04-26T16:30:40.254447+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:30:40.254447+0000","last_undegraded":"2024-04-26T16:30:40.254447+0000","last_fullsized":"2024-04-26T16:30:40.254447+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:03.519624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.519624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:17.009661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"388'19","reported_seq":971,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247214+0000","last_change":"2024-04-26T16:29:00.045697+0000","last_active":"2024-04-26T16:30:40.247214+0000","last_peered":"2024-04-26T16:30:40.247214+0000","last_clean":"2024-04-26T16:30:40.247214+0000","last_became_active":"2024-04-26T16:28:25.270038+0000","last_became_peered":"2024-04-26T16:28:25.270038+0000","last_unstale":"2024-04-26T16:30:40.247214+0000","last_undegraded":"2024-04-26T16:30:40.247214+0000","last_fullsized":"2024-04-26T16:30:40.247214+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:00.045450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.045450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879812+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":945,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237797+0000","last_change":"2024-04-26T16:29:07.075600+0000","last_active":"2024-04-26T16:30:40.237797+0000","last_peered":"2024-04-26T16:30:40.237797+0000","last_clean":"2024-04-26T16:30:40.237797+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:30:40.237797+0000","last_undegraded":"2024-04-26T16:30:40.237797+0000","last_fullsized":"2024-04-26T16:30:40.237797+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.075499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.624243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":928,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264117+0000","last_change":"2024-04-26T16:30:28.910693+0000","last_active":"2024-04-26T16:30:40.264117+0000","last_peered":"2024-04-26T16:30:40.264117+0000","last_clean":"2024-04-26T16:30:40.264117+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:30:40.264117+0000","last_undegraded":"2024-04-26T16:30:40.264117+0000","last_fullsized":"2024-04-26T16:30:40.264117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.057825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":933,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243275+0000","last_change":"2024-04-26T16:30:07.455469+0000","last_active":"2024-04-26T16:30:40.243275+0000","last_peered":"2024-04-26T16:30:40.243275+0000","last_clean":"2024-04-26T16:30:40.243275+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:30:40.243275+0000","last_undegraded":"2024-04-26T16:30:40.243275+0000","last_fullsized":"2024-04-26T16:30:40.243275+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.455245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.455245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:20.143258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":798,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237475+0000","last_change":"2024-04-26T16:30:02.403961+0000","last_active":"2024-04-26T16:30:40.237475+0000","last_peered":"2024-04-26T16:30:40.237475+0000","last_clean":"2024-04-26T16:30:40.237475+0000","last_became_active":"2024-04-26T16:28:25.246039+0000","last_became_peered":"2024-04-26T16:28:25.246039+0000","last_unstale":"2024-04-26T16:30:40.237475+0000","last_undegraded":"2024-04-26T16:30:40.237475+0000","last_fullsized":"2024-04-26T16:30:40.237475+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:02.403733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.403733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:32.066798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"388'8","reported_seq":899,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.487995+0000","last_change":"2024-04-26T16:30:29.358181+0000","last_active":"2024-04-26T16:30:47.487995+0000","last_peered":"2024-04-26T16:30:47.487995+0000","last_clean":"2024-04-26T16:30:47.487995+0000","last_became_active":"2024-04-26T16:28:25.260556+0000","last_became_peered":"2024-04-26T16:28:25.260556+0000","last_unstale":"2024-04-26T16:30:47.487995+0000","last_undegraded":"2024-04-26T16:30:47.487995+0000","last_fullsized":"2024-04-26T16:30:47.487995+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'8","last_scrub_stamp":"2024-04-26T16:30:29.357863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.357863+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:36.666658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":828,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488327+0000","last_change":"2024-04-26T16:30:28.356139+0000","last_active":"2024-04-26T16:30:47.488327+0000","last_peered":"2024-04-26T16:30:47.488327+0000","last_clean":"2024-04-26T16:30:47.488327+0000","last_became_active":"2024-04-26T16:28:25.242243+0000","last_became_peered":"2024-04-26T16:28:25.242243+0000","last_unstale":"2024-04-26T16:30:47.488327+0000","last_undegraded":"2024-04-26T16:30:47.488327+0000","last_fullsized":"2024-04-26T16:30:47.488327+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.355929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:58.034883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":433,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260855+0000","last_change":"2024-04-26T16:28:47.096936+0000","last_active":"2024-04-26T16:30:40.260855+0000","last_peered":"2024-04-26T16:30:40.260855+0000","last_clean":"2024-04-26T16:30:40.260855+0000","last_became_active":"2024-04-26T16:28:25.243371+0000","last_became_peered":"2024-04-26T16:28:25.243371+0000","last_unstale":"2024-04-26T16:30:40.260855+0000","last_undegraded":"2024-04-26T16:30:40.260855+0000","last_fullsized":"2024-04-26T16:30:40.260855+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:47.096851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.096851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962257+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":924,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249385+0000","last_change":"2024-04-26T16:28:25.266733+0000","last_active":"2024-04-26T16:30:40.249385+0000","last_peered":"2024-04-26T16:30:40.249385+0000","last_clean":"2024-04-26T16:30:40.249385+0000","last_became_active":"2024-04-26T16:28:25.266217+0000","last_became_peered":"2024-04-26T16:28:25.266217+0000","last_unstale":"2024-04-26T16:30:40.249385+0000","last_undegraded":"2024-04-26T16:30:40.249385+0000","last_fullsized":"2024-04-26T16:30:40.249385+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:18.906360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:27:18.906360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879038+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":795,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238913+0000","last_change":"2024-04-26T16:29:58.659839+0000","last_active":"2024-04-26T16:30:40.238913+0000","last_peered":"2024-04-26T16:30:40.238913+0000","last_clean":"2024-04-26T16:30:40.238913+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:30:40.238913+0000","last_undegraded":"2024-04-26T16:30:40.238913+0000","last_fullsized":"2024-04-26T16:30:40.238913+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.659641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.659641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:17.377583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'19","reported_seq":980,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250524+0000","last_change":"2024-04-26T16:27:29.963129+0000","last_active":"2024-04-26T16:30:40.250524+0000","last_peered":"2024-04-26T16:30:40.250524+0000","last_clean":"2024-04-26T16:30:40.250524+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:30:40.250524+0000","last_undegraded":"2024-04-26T16:30:40.250524+0000","last_fullsized":"2024-04-26T16:30:40.250524+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:29.962882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.962882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879333+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":432,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258778+0000","last_change":"2024-04-26T16:30:07.968532+0000","last_active":"2024-04-26T16:30:40.258778+0000","last_peered":"2024-04-26T16:30:40.258778+0000","last_clean":"2024-04-26T16:30:40.258778+0000","last_became_active":"2024-04-26T16:26:23.216238+0000","last_became_peered":"2024-04-26T16:26:23.216238+0000","last_unstale":"2024-04-26T16:30:40.258778+0000","last_undegraded":"2024-04-26T16:30:40.258778+0000","last_fullsized":"2024-04-26T16:30:40.258778+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.968404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.968404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.834438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"316'13","reported_seq":569,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.245076+0000","last_change":"2024-04-26T16:30:26.031295+0000","last_active":"2024-04-26T16:30:40.245076+0000","last_peered":"2024-04-26T16:30:40.245076+0000","last_clean":"2024-04-26T16:30:40.245076+0000","last_became_active":"2024-04-26T16:26:23.215841+0000","last_became_peered":"2024-04-26T16:26:23.215841+0000","last_unstale":"2024-04-26T16:30:40.245076+0000","last_undegraded":"2024-04-26T16:30:40.245076+0000","last_fullsized":"2024-04-26T16:30:40.245076+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:26.031142+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.031142+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.724917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":935,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.257238+0000","last_change":"2024-04-26T16:30:22.624802+0000","last_active":"2024-04-26T16:30:40.257238+0000","last_peered":"2024-04-26T16:30:40.257238+0000","last_clean":"2024-04-26T16:30:40.257238+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:30:40.257238+0000","last_undegraded":"2024-04-26T16:30:40.257238+0000","last_fullsized":"2024-04-26T16:30:40.257238+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.624646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.624646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.857935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":433,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264117+0000","last_change":"2024-04-26T16:28:46.104373+0000","last_active":"2024-04-26T16:30:40.264117+0000","last_peered":"2024-04-26T16:30:40.264117+0000","last_clean":"2024-04-26T16:30:40.264117+0000","last_became_active":"2024-04-26T16:28:25.241392+0000","last_became_peered":"2024-04-26T16:28:25.241392+0000","last_unstale":"2024-04-26T16:30:40.264117+0000","last_undegraded":"2024-04-26T16:30:40.264117+0000","last_fullsized":"2024-04-26T16:30:40.264117+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.104261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.104261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"388'19","reported_seq":972,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.254825+0000","last_change":"2024-04-26T16:28:54.496909+0000","last_active":"2024-04-26T16:30:40.254825+0000","last_peered":"2024-04-26T16:30:40.254825+0000","last_clean":"2024-04-26T16:30:40.254825+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:30:40.254825+0000","last_undegraded":"2024-04-26T16:30:40.254825+0000","last_fullsized":"2024-04-26T16:30:40.254825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:28:54.496786+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.496786+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":947,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.255950+0000","last_change":"2024-04-26T16:29:10.518620+0000","last_active":"2024-04-26T16:30:40.255950+0000","last_peered":"2024-04-26T16:30:40.255950+0000","last_clean":"2024-04-26T16:30:40.255950+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:30:40.255950+0000","last_undegraded":"2024-04-26T16:30:40.255950+0000","last_fullsized":"2024-04-26T16:30:40.255950+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.518359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1090,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260291+0000","last_change":"2024-04-26T16:27:42.184833+0000","last_active":"2024-04-26T16:30:40.260291+0000","last_peered":"2024-04-26T16:30:40.260291+0000","last_clean":"2024-04-26T16:30:40.260291+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:30:40.260291+0000","last_undegraded":"2024-04-26T16:30:40.260291+0000","last_fullsized":"2024-04-26T16:30:40.260291+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:42.184655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:27:42.184655+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":940,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243026+0000","last_change":"2024-04-26T16:30:00.630668+0000","last_active":"2024-04-26T16:30:40.243026+0000","last_peered":"2024-04-26T16:30:40.243026+0000","last_clean":"2024-04-26T16:30:40.243026+0000","last_became_active":"2024-04-26T16:26:23.188629+0000","last_became_peered":"2024-04-26T16:26:23.188629+0000","last_unstale":"2024-04-26T16:30:40.243026+0000","last_undegraded":"2024-04-26T16:30:40.243026+0000","last_fullsized":"2024-04-26T16:30:40.243026+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.630569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.630569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.233286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":778,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250825+0000","last_change":"2024-04-26T16:27:25.909568+0000","last_active":"2024-04-26T16:30:40.250825+0000","last_peered":"2024-04-26T16:30:40.250825+0000","last_clean":"2024-04-26T16:30:40.250825+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:30:40.250825+0000","last_undegraded":"2024-04-26T16:30:40.250825+0000","last_fullsized":"2024-04-26T16:30:40.250825+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:25.909462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.909462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"388'19","reported_seq":960,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250969+0000","last_change":"2024-04-26T16:28:43.978963+0000","last_active":"2024-04-26T16:30:40.250969+0000","last_peered":"2024-04-26T16:30:40.250969+0000","last_clean":"2024-04-26T16:30:40.250969+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:30:40.250969+0000","last_undegraded":"2024-04-26T16:30:40.250969+0000","last_fullsized":"2024-04-26T16:30:40.250969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:28:43.978726+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.978726+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879666+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":830,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488923+0000","last_change":"2024-04-26T16:28:52.136590+0000","last_active":"2024-04-26T16:30:47.488923+0000","last_peered":"2024-04-26T16:30:47.488923+0000","last_clean":"2024-04-26T16:30:47.488923+0000","last_became_active":"2024-04-26T16:28:25.270839+0000","last_became_peered":"2024-04-26T16:28:25.270839+0000","last_unstale":"2024-04-26T16:30:47.488923+0000","last_undegraded":"2024-04-26T16:30:47.488923+0000","last_fullsized":"2024-04-26T16:30:47.488923+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.136387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.136387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:49.130985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":928,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259982+0000","last_change":"2024-04-26T16:29:00.063758+0000","last_active":"2024-04-26T16:30:40.259982+0000","last_peered":"2024-04-26T16:30:40.259982+0000","last_clean":"2024-04-26T16:30:40.259982+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:30:40.259982+0000","last_undegraded":"2024-04-26T16:30:40.259982+0000","last_fullsized":"2024-04-26T16:30:40.259982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:00.063648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.063648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":950,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.253687+0000","last_change":"2024-04-26T16:30:12.648490+0000","last_active":"2024-04-26T16:30:40.253687+0000","last_peered":"2024-04-26T16:30:40.253687+0000","last_clean":"2024-04-26T16:30:40.253687+0000","last_became_active":"2024-04-26T16:28:25.256886+0000","last_became_peered":"2024-04-26T16:28:25.256886+0000","last_unstale":"2024-04-26T16:30:40.253687+0000","last_undegraded":"2024-04-26T16:30:40.253687+0000","last_fullsized":"2024-04-26T16:30:40.253687+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.648271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.648271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:23.526396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":678,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.489033+0000","last_change":"2024-04-26T16:29:07.228455+0000","last_active":"2024-04-26T16:30:47.489033+0000","last_peered":"2024-04-26T16:30:47.489033+0000","last_clean":"2024-04-26T16:30:47.489033+0000","last_became_active":"2024-04-26T16:28:25.270409+0000","last_became_peered":"2024-04-26T16:28:25.270409+0000","last_unstale":"2024-04-26T16:30:47.489033+0000","last_undegraded":"2024-04-26T16:30:47.489033+0000","last_fullsized":"2024-04-26T16:30:47.489033+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.228356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.228356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:49.130886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"388'20","reported_seq":981,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241094+0000","last_change":"2024-04-26T16:28:47.517956+0000","last_active":"2024-04-26T16:30:40.241094+0000","last_peered":"2024-04-26T16:30:40.241094+0000","last_clean":"2024-04-26T16:30:40.241094+0000","last_became_active":"2024-04-26T16:28:25.258640+0000","last_became_peered":"2024-04-26T16:28:25.258640+0000","last_unstale":"2024-04-26T16:30:40.241094+0000","last_undegraded":"2024-04-26T16:30:40.241094+0000","last_fullsized":"2024-04-26T16:30:40.241094+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:47.517723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.517723+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.588860+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":447,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242934+0000","last_change":"2024-04-26T16:30:21.968003+0000","last_active":"2024-04-26T16:30:40.242934+0000","last_peered":"2024-04-26T16:30:40.242934+0000","last_clean":"2024-04-26T16:30:40.242934+0000","last_became_active":"2024-04-26T16:28:25.226182+0000","last_became_peered":"2024-04-26T16:28:25.226182+0000","last_unstale":"2024-04-26T16:30:40.242934+0000","last_undegraded":"2024-04-26T16:30:40.242934+0000","last_fullsized":"2024-04-26T16:30:40.242934+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:21.967758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.967758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.703747+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":936,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240898+0000","last_change":"2024-04-26T16:28:40.785051+0000","last_active":"2024-04-26T16:30:40.240898+0000","last_peered":"2024-04-26T16:30:40.240898+0000","last_clean":"2024-04-26T16:30:40.240898+0000","last_became_active":"2024-04-26T16:26:23.213441+0000","last_became_peered":"2024-04-26T16:26:23.213441+0000","last_unstale":"2024-04-26T16:30:40.240898+0000","last_undegraded":"2024-04-26T16:30:40.240898+0000","last_fullsized":"2024-04-26T16:30:40.240898+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.784936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.784936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":930,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258506+0000","last_change":"2024-04-26T16:29:09.036096+0000","last_active":"2024-04-26T16:30:40.258506+0000","last_peered":"2024-04-26T16:30:40.258506+0000","last_clean":"2024-04-26T16:30:40.258506+0000","last_became_active":"2024-04-26T16:26:23.211857+0000","last_became_peered":"2024-04-26T16:26:23.211857+0000","last_unstale":"2024-04-26T16:30:40.258506+0000","last_undegraded":"2024-04-26T16:30:40.258506+0000","last_fullsized":"2024-04-26T16:30:40.258506+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.035817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.035817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.963264+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":789,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.243883+0000","last_change":"2024-04-26T16:30:15.529255+0000","last_active":"2024-04-26T16:30:40.243883+0000","last_peered":"2024-04-26T16:30:40.243883+0000","last_clean":"2024-04-26T16:30:40.243883+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:30:40.243883+0000","last_undegraded":"2024-04-26T16:30:40.243883+0000","last_fullsized":"2024-04-26T16:30:40.243883+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.529105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.529105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:19.318671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'12","reported_seq":984,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.257219+0000","last_change":"2024-04-26T16:28:53.490005+0000","last_active":"2024-04-26T16:30:40.257219+0000","last_peered":"2024-04-26T16:30:40.257219+0000","last_clean":"2024-04-26T16:30:40.257219+0000","last_became_active":"2024-04-26T16:28:25.246218+0000","last_became_peered":"2024-04-26T16:28:25.246218+0000","last_unstale":"2024-04-26T16:30:40.257219+0000","last_undegraded":"2024-04-26T16:30:40.257219+0000","last_fullsized":"2024-04-26T16:30:40.257219+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'12","last_scrub_stamp":"2024-04-26T16:28:53.489846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:53.489846+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":310,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.489978+0000","last_change":"2024-04-26T16:30:36.171558+0000","last_active":"2024-04-26T16:30:47.489978+0000","last_peered":"2024-04-26T16:30:47.489978+0000","last_clean":"2024-04-26T16:30:47.489978+0000","last_became_active":"2024-04-26T16:28:25.231296+0000","last_became_peered":"2024-04-26T16:28:25.231296+0000","last_unstale":"2024-04-26T16:30:47.489978+0000","last_undegraded":"2024-04-26T16:30:47.489978+0000","last_fullsized":"2024-04-26T16:30:47.489978+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:36.171451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:36.171451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.052865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":868,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488794+0000","last_change":"2024-04-26T16:30:19.325201+0000","last_active":"2024-04-26T16:30:47.488794+0000","last_peered":"2024-04-26T16:30:47.488794+0000","last_clean":"2024-04-26T16:30:47.488794+0000","last_became_active":"2024-04-26T16:28:25.272341+0000","last_became_peered":"2024-04-26T16:28:25.272341+0000","last_unstale":"2024-04-26T16:30:47.488794+0000","last_undegraded":"2024-04-26T16:30:47.488794+0000","last_fullsized":"2024-04-26T16:30:47.488794+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.325091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.325091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.244723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":923,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.248639+0000","last_change":"2024-04-26T16:30:34.003108+0000","last_active":"2024-04-26T16:30:40.248639+0000","last_peered":"2024-04-26T16:30:40.248639+0000","last_clean":"2024-04-26T16:30:40.248639+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:30:40.248639+0000","last_undegraded":"2024-04-26T16:30:40.248639+0000","last_fullsized":"2024-04-26T16:30:40.248639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:34.002980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.002980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.133005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":788,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251588+0000","last_change":"2024-04-26T16:28:35.971437+0000","last_active":"2024-04-26T16:30:40.251588+0000","last_peered":"2024-04-26T16:30:40.251588+0000","last_clean":"2024-04-26T16:30:40.251588+0000","last_became_active":"2024-04-26T16:28:25.269747+0000","last_became_peered":"2024-04-26T16:28:25.269747+0000","last_unstale":"2024-04-26T16:30:40.251588+0000","last_undegraded":"2024-04-26T16:30:40.251588+0000","last_fullsized":"2024-04-26T16:30:40.251588+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:35.971222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:35.971222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'15","reported_seq":484,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251665+0000","last_change":"2024-04-26T16:27:43.107066+0000","last_active":"2024-04-26T16:30:40.251665+0000","last_peered":"2024-04-26T16:30:40.251665+0000","last_clean":"2024-04-26T16:30:40.251665+0000","last_became_active":"2024-04-26T16:26:23.200216+0000","last_became_peered":"2024-04-26T16:26:23.200216+0000","last_unstale":"2024-04-26T16:30:40.251665+0000","last_undegraded":"2024-04-26T16:30:40.251665+0000","last_fullsized":"2024-04-26T16:30:40.251665+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:27:43.106910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:43.106910+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879238+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":842,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.490359+0000","last_change":"2024-04-26T16:29:08.217390+0000","last_active":"2024-04-26T16:30:47.490359+0000","last_peered":"2024-04-26T16:30:47.490359+0000","last_clean":"2024-04-26T16:30:47.490359+0000","last_became_active":"2024-04-26T16:28:25.244917+0000","last_became_peered":"2024-04-26T16:28:25.244917+0000","last_unstale":"2024-04-26T16:30:47.490359+0000","last_undegraded":"2024-04-26T16:30:47.490359+0000","last_fullsized":"2024-04-26T16:30:47.490359+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:08.217158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.217158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:49.130841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":936,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.248257+0000","last_change":"2024-04-26T16:28:57.063805+0000","last_active":"2024-04-26T16:30:40.248257+0000","last_peered":"2024-04-26T16:30:40.248257+0000","last_clean":"2024-04-26T16:30:40.248257+0000","last_became_active":"2024-04-26T16:28:25.266296+0000","last_became_peered":"2024-04-26T16:28:25.266296+0000","last_unstale":"2024-04-26T16:30:40.248257+0000","last_undegraded":"2024-04-26T16:30:40.248257+0000","last_fullsized":"2024-04-26T16:30:40.248257+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:57.063594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:57.063594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.880007+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":933,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.260112+0000","last_change":"2024-04-26T16:30:09.970620+0000","last_active":"2024-04-26T16:30:40.260112+0000","last_peered":"2024-04-26T16:30:40.260112+0000","last_clean":"2024-04-26T16:30:40.260112+0000","last_became_active":"2024-04-26T16:28:25.228971+0000","last_became_peered":"2024-04-26T16:28:25.228971+0000","last_unstale":"2024-04-26T16:30:40.260112+0000","last_undegraded":"2024-04-26T16:30:40.260112+0000","last_fullsized":"2024-04-26T16:30:40.260112+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.970505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.970505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:39.666703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":789,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238846+0000","last_change":"2024-04-26T16:30:22.616210+0000","last_active":"2024-04-26T16:30:40.238846+0000","last_peered":"2024-04-26T16:30:40.238846+0000","last_clean":"2024-04-26T16:30:40.238846+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:30:40.238846+0000","last_undegraded":"2024-04-26T16:30:40.238846+0000","last_fullsized":"2024-04-26T16:30:40.238846+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.616065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.616065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.002311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"388'34","reported_seq":992,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239450+0000","last_change":"2024-04-26T16:29:04.788890+0000","last_active":"2024-04-26T16:30:40.239450+0000","last_peered":"2024-04-26T16:30:40.239450+0000","last_clean":"2024-04-26T16:30:40.239450+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:30:40.239450+0000","last_undegraded":"2024-04-26T16:30:40.239450+0000","last_fullsized":"2024-04-26T16:30:40.239450+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:29:04.788618+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.788618+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010414+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":448,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.238033+0000","last_change":"2024-04-26T16:29:07.493539+0000","last_active":"2024-04-26T16:30:40.238033+0000","last_peered":"2024-04-26T16:30:40.238033+0000","last_clean":"2024-04-26T16:30:40.238033+0000","last_became_active":"2024-04-26T16:28:25.250387+0000","last_became_peered":"2024-04-26T16:28:25.250387+0000","last_unstale":"2024-04-26T16:30:40.238033+0000","last_undegraded":"2024-04-26T16:30:40.238033+0000","last_fullsized":"2024-04-26T16:30:40.238033+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.493409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.493409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"433'14","reported_seq":989,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.490201+0000","last_change":"2024-04-26T16:29:00.093520+0000","last_active":"2024-04-26T16:30:47.490201+0000","last_peered":"2024-04-26T16:30:47.490201+0000","last_clean":"2024-04-26T16:30:47.490201+0000","last_became_active":"2024-04-26T16:28:25.265640+0000","last_became_peered":"2024-04-26T16:28:25.265640+0000","last_unstale":"2024-04-26T16:30:47.490201+0000","last_undegraded":"2024-04-26T16:30:47.490201+0000","last_fullsized":"2024-04-26T16:30:47.490201+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:00.093200+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.093200+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:49.130941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":935,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250843+0000","last_change":"2024-04-26T16:30:11.234876+0000","last_active":"2024-04-26T16:30:40.250843+0000","last_peered":"2024-04-26T16:30:40.250843+0000","last_clean":"2024-04-26T16:30:40.250843+0000","last_became_active":"2024-04-26T16:26:23.201293+0000","last_became_peered":"2024-04-26T16:26:23.201293+0000","last_unstale":"2024-04-26T16:30:40.250843+0000","last_undegraded":"2024-04-26T16:30:40.250843+0000","last_fullsized":"2024-04-26T16:30:40.250843+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:11.234642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.234642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:11.595192+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":800,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.254767+0000","last_change":"2024-04-26T16:30:16.615509+0000","last_active":"2024-04-26T16:30:40.254767+0000","last_peered":"2024-04-26T16:30:40.254767+0000","last_clean":"2024-04-26T16:30:40.254767+0000","last_became_active":"2024-04-26T16:28:25.247238+0000","last_became_peered":"2024-04-26T16:28:25.247238+0000","last_unstale":"2024-04-26T16:30:40.254767+0000","last_undegraded":"2024-04-26T16:30:40.254767+0000","last_fullsized":"2024-04-26T16:30:40.254767+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.615354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.615354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.463397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"388'15","reported_seq":957,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258413+0000","last_change":"2024-04-26T16:28:59.039798+0000","last_active":"2024-04-26T16:30:40.258413+0000","last_peered":"2024-04-26T16:30:40.258413+0000","last_clean":"2024-04-26T16:30:40.258413+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:30:40.258413+0000","last_undegraded":"2024-04-26T16:30:40.258413+0000","last_fullsized":"2024-04-26T16:30:40.258413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:28:59.039575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.039575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":939,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.258965+0000","last_change":"2024-04-26T16:28:33.172830+0000","last_active":"2024-04-26T16:30:40.258965+0000","last_peered":"2024-04-26T16:30:40.258965+0000","last_clean":"2024-04-26T16:30:40.258965+0000","last_became_active":"2024-04-26T16:28:25.245293+0000","last_became_peered":"2024-04-26T16:28:25.245293+0000","last_unstale":"2024-04-26T16:30:40.258965+0000","last_undegraded":"2024-04-26T16:30:40.258965+0000","last_fullsized":"2024-04-26T16:30:40.258965+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:28:33.172458+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:28:33.172458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961260+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":948,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.257039+0000","last_change":"2024-04-26T16:30:28.566079+0000","last_active":"2024-04-26T16:30:40.257039+0000","last_peered":"2024-04-26T16:30:40.257039+0000","last_clean":"2024-04-26T16:30:40.257039+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:30:40.257039+0000","last_undegraded":"2024-04-26T16:30:40.257039+0000","last_fullsized":"2024-04-26T16:30:40.257039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.565983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.565983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:44.718464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":940,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.245612+0000","last_change":"2024-04-26T16:28:46.866392+0000","last_active":"2024-04-26T16:30:40.245612+0000","last_peered":"2024-04-26T16:30:40.245612+0000","last_clean":"2024-04-26T16:30:40.245612+0000","last_became_active":"2024-04-26T16:26:23.203386+0000","last_became_peered":"2024-04-26T16:26:23.203386+0000","last_unstale":"2024-04-26T16:30:40.245612+0000","last_undegraded":"2024-04-26T16:30:40.245612+0000","last_fullsized":"2024-04-26T16:30:40.245612+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.866222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.866222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":448,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239046+0000","last_change":"2024-04-26T16:30:29.497506+0000","last_active":"2024-04-26T16:30:40.239046+0000","last_peered":"2024-04-26T16:30:40.239046+0000","last_clean":"2024-04-26T16:30:40.239046+0000","last_became_active":"2024-04-26T16:28:25.238120+0000","last_became_peered":"2024-04-26T16:28:25.238120+0000","last_unstale":"2024-04-26T16:30:40.239046+0000","last_undegraded":"2024-04-26T16:30:40.239046+0000","last_fullsized":"2024-04-26T16:30:40.239046+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.497411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.497411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.042909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"388'9","reported_seq":958,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.256024+0000","last_change":"2024-04-26T16:30:02.530560+0000","last_active":"2024-04-26T16:30:40.256024+0000","last_peered":"2024-04-26T16:30:40.256024+0000","last_clean":"2024-04-26T16:30:40.256024+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:30:40.256024+0000","last_undegraded":"2024-04-26T16:30:40.256024+0000","last_fullsized":"2024-04-26T16:30:40.256024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:30:02.530372+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:23:25.467392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.530372+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.198488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":891,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.488631+0000","last_change":"2024-04-26T16:30:23.313786+0000","last_active":"2024-04-26T16:30:47.488631+0000","last_peered":"2024-04-26T16:30:47.488631+0000","last_clean":"2024-04-26T16:30:47.488631+0000","last_became_active":"2024-04-26T16:28:25.264839+0000","last_became_peered":"2024-04-26T16:28:25.264839+0000","last_unstale":"2024-04-26T16:30:47.488631+0000","last_undegraded":"2024-04-26T16:30:47.488631+0000","last_fullsized":"2024-04-26T16:30:47.488631+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.313672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.313672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:29.555420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":938,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.245239+0000","last_change":"2024-04-26T16:28:43.823916+0000","last_active":"2024-04-26T16:30:40.245239+0000","last_peered":"2024-04-26T16:30:40.245239+0000","last_clean":"2024-04-26T16:30:40.245239+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:30:40.245239+0000","last_undegraded":"2024-04-26T16:30:40.245239+0000","last_fullsized":"2024-04-26T16:30:40.245239+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:43.823731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.823731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":958,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241563+0000","last_change":"2024-04-26T16:30:26.547964+0000","last_active":"2024-04-26T16:30:40.241563+0000","last_peered":"2024-04-26T16:30:40.241563+0000","last_clean":"2024-04-26T16:30:40.241563+0000","last_became_active":"2024-04-26T16:28:25.244502+0000","last_became_peered":"2024-04-26T16:28:25.244502+0000","last_unstale":"2024-04-26T16:30:40.241563+0000","last_undegraded":"2024-04-26T16:30:40.241563+0000","last_fullsized":"2024-04-26T16:30:40.241563+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.547756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.547756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.908182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":779,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.247850+0000","last_change":"2024-04-26T16:27:44.070368+0000","last_active":"2024-04-26T16:30:40.247850+0000","last_peered":"2024-04-26T16:30:40.247850+0000","last_clean":"2024-04-26T16:30:40.247850+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:30:40.247850+0000","last_undegraded":"2024-04-26T16:30:40.247850+0000","last_fullsized":"2024-04-26T16:30:40.247850+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:27:44.070141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.879282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'15","reported_seq":975,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239187+0000","last_change":"2024-04-26T16:30:18.494734+0000","last_active":"2024-04-26T16:30:40.239187+0000","last_peered":"2024-04-26T16:30:40.239187+0000","last_clean":"2024-04-26T16:30:40.239187+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:30:40.239187+0000","last_undegraded":"2024-04-26T16:30:40.239187+0000","last_fullsized":"2024-04-26T16:30:40.239187+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:30:18.494624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.494624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:21.675249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":937,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259837+0000","last_change":"2024-04-26T16:28:36.216812+0000","last_active":"2024-04-26T16:30:40.259837+0000","last_peered":"2024-04-26T16:30:40.259837+0000","last_clean":"2024-04-26T16:30:40.259837+0000","last_became_active":"2024-04-26T16:28:25.255455+0000","last_became_peered":"2024-04-26T16:28:25.255455+0000","last_unstale":"2024-04-26T16:30:40.259837+0000","last_undegraded":"2024-04-26T16:30:40.259837+0000","last_fullsized":"2024-04-26T16:30:40.259837+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:36.216718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:36.216718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.961360+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":938,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240656+0000","last_change":"2024-04-26T16:29:04.486590+0000","last_active":"2024-04-26T16:30:40.240656+0000","last_peered":"2024-04-26T16:30:40.240656+0000","last_clean":"2024-04-26T16:30:40.240656+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:30:40.240656+0000","last_undegraded":"2024-04-26T16:30:40.240656+0000","last_fullsized":"2024-04-26T16:30:40.240656+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:04.486382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.486382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":940,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.261006+0000","last_change":"2024-04-26T16:30:26.949591+0000","last_active":"2024-04-26T16:30:40.261006+0000","last_peered":"2024-04-26T16:30:40.261006+0000","last_clean":"2024-04-26T16:30:40.261006+0000","last_became_active":"2024-04-26T16:28:25.245307+0000","last_became_peered":"2024-04-26T16:28:25.245307+0000","last_unstale":"2024-04-26T16:30:40.261006+0000","last_undegraded":"2024-04-26T16:30:40.261006+0000","last_fullsized":"2024-04-26T16:30:40.261006+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.949357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.949357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:30.336295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":789,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.242284+0000","last_change":"2024-04-26T16:30:04.495577+0000","last_active":"2024-04-26T16:30:40.242284+0000","last_peered":"2024-04-26T16:30:40.242284+0000","last_clean":"2024-04-26T16:30:40.242284+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:30:40.242284+0000","last_undegraded":"2024-04-26T16:30:40.242284+0000","last_fullsized":"2024-04-26T16:30:40.242284+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:04.495235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.495235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:27.081892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"388'23","reported_seq":612,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240760+0000","last_change":"2024-04-26T16:28:45.510060+0000","last_active":"2024-04-26T16:30:40.240760+0000","last_peered":"2024-04-26T16:30:40.240760+0000","last_clean":"2024-04-26T16:30:40.240760+0000","last_became_active":"2024-04-26T16:28:25.250379+0000","last_became_peered":"2024-04-26T16:28:25.250379+0000","last_unstale":"2024-04-26T16:30:40.240760+0000","last_undegraded":"2024-04-26T16:30:40.240760+0000","last_fullsized":"2024-04-26T16:30:40.240760+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:28:45.509872+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:28:45.509872+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.589445+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":950,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.249007+0000","last_change":"2024-04-26T16:29:02.113592+0000","last_active":"2024-04-26T16:30:40.249007+0000","last_peered":"2024-04-26T16:30:40.249007+0000","last_clean":"2024-04-26T16:30:40.249007+0000","last_became_active":"2024-04-26T16:28:25.277266+0000","last_became_peered":"2024-04-26T16:28:25.277266+0000","last_unstale":"2024-04-26T16:30:40.249007+0000","last_undegraded":"2024-04-26T16:30:40.249007+0000","last_fullsized":"2024-04-26T16:30:40.249007+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:02.113366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.113366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.880048+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":435,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.255605+0000","last_change":"2024-04-26T16:30:11.600252+0000","last_active":"2024-04-26T16:30:40.255605+0000","last_peered":"2024-04-26T16:30:40.255605+0000","last_clean":"2024-04-26T16:30:40.255605+0000","last_became_active":"2024-04-26T16:26:23.166910+0000","last_became_peered":"2024-04-26T16:26:23.166910+0000","last_unstale":"2024-04-26T16:30:40.255605+0000","last_undegraded":"2024-04-26T16:30:40.255605+0000","last_fullsized":"2024-04-26T16:30:40.255605+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:11.600054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.600054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:12.242206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":935,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.253974+0000","last_change":"2024-04-26T16:30:24.568125+0000","last_active":"2024-04-26T16:30:40.253974+0000","last_peered":"2024-04-26T16:30:40.253974+0000","last_clean":"2024-04-26T16:30:40.253974+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:30:40.253974+0000","last_undegraded":"2024-04-26T16:30:40.253974+0000","last_fullsized":"2024-04-26T16:30:40.253974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.567773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:36.702289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":701,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-04-26T16:30:47.489157+0000","last_change":"2024-04-26T16:29:10.208455+0000","last_active":"2024-04-26T16:30:47.489157+0000","last_peered":"2024-04-26T16:30:47.489157+0000","last_clean":"2024-04-26T16:30:47.489157+0000","last_became_active":"2024-04-26T16:28:25.255571+0000","last_became_peered":"2024-04-26T16:28:25.255571+0000","last_unstale":"2024-04-26T16:30:47.489157+0000","last_undegraded":"2024-04-26T16:30:47.489157+0000","last_fullsized":"2024-04-26T16:30:47.489157+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.208226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.208226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:49.131032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"388'27","reported_seq":526,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.240054+0000","last_change":"2024-04-26T16:30:13.510016+0000","last_active":"2024-04-26T16:30:40.240054+0000","last_peered":"2024-04-26T16:30:40.240054+0000","last_clean":"2024-04-26T16:30:40.240054+0000","last_became_active":"2024-04-26T16:28:25.256896+0000","last_became_peered":"2024-04-26T16:28:25.256896+0000","last_unstale":"2024-04-26T16:30:40.240054+0000","last_undegraded":"2024-04-26T16:30:40.240054+0000","last_fullsized":"2024-04-26T16:30:40.240054+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:13.509869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.509869+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.781981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":952,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.250695+0000","last_change":"2024-04-26T16:30:13.187491+0000","last_active":"2024-04-26T16:30:40.250695+0000","last_peered":"2024-04-26T16:30:40.250695+0000","last_clean":"2024-04-26T16:30:40.250695+0000","last_became_active":"2024-04-26T16:28:25.271550+0000","last_became_peered":"2024-04-26T16:28:25.271550+0000","last_unstale":"2024-04-26T16:30:40.250695+0000","last_undegraded":"2024-04-26T16:30:40.250695+0000","last_fullsized":"2024-04-26T16:30:40.250695+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:13.187197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.187197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.285217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":927,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.248247+0000","last_change":"2024-04-26T16:29:57.141987+0000","last_active":"2024-04-26T16:30:40.248247+0000","last_peered":"2024-04-26T16:30:40.248247+0000","last_clean":"2024-04-26T16:30:40.248247+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:30:40.248247+0000","last_undegraded":"2024-04-26T16:30:40.248247+0000","last_fullsized":"2024-04-26T16:30:40.248247+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:57.141873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.141873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.450280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":455,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.241004+0000","last_change":"2024-04-26T16:30:34.033387+0000","last_active":"2024-04-26T16:30:40.241004+0000","last_peered":"2024-04-26T16:30:40.241004+0000","last_clean":"2024-04-26T16:30:40.241004+0000","last_became_active":"2024-04-26T16:26:23.212770+0000","last_became_peered":"2024-04-26T16:26:23.212770+0000","last_unstale":"2024-04-26T16:30:40.241004+0000","last_undegraded":"2024-04-26T16:30:40.241004+0000","last_fullsized":"2024-04-26T16:30:40.241004+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:34.033282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.033282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:45.039967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":802,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251095+0000","last_change":"2024-04-26T16:30:14.201574+0000","last_active":"2024-04-26T16:30:40.251095+0000","last_peered":"2024-04-26T16:30:40.251095+0000","last_clean":"2024-04-26T16:30:40.251095+0000","last_became_active":"2024-04-26T16:28:25.267368+0000","last_became_peered":"2024-04-26T16:28:25.267368+0000","last_unstale":"2024-04-26T16:30:40.251095+0000","last_undegraded":"2024-04-26T16:30:40.251095+0000","last_fullsized":"2024-04-26T16:30:40.251095+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.201493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.201493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:17.192546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"388'19","reported_seq":1002,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.257300+0000","last_change":"2024-04-26T16:29:08.447892+0000","last_active":"2024-04-26T16:30:40.257300+0000","last_peered":"2024-04-26T16:30:40.257300+0000","last_clean":"2024-04-26T16:30:40.257300+0000","last_became_active":"2024-04-26T16:28:25.244993+0000","last_became_peered":"2024-04-26T16:28:25.244993+0000","last_unstale":"2024-04-26T16:30:40.257300+0000","last_undegraded":"2024-04-26T16:30:40.257300+0000","last_fullsized":"2024-04-26T16:30:40.257300+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:08.447471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.447471+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":951,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264081+0000","last_change":"2024-04-26T16:30:32.981558+0000","last_active":"2024-04-26T16:30:40.264081+0000","last_peered":"2024-04-26T16:30:40.264081+0000","last_clean":"2024-04-26T16:30:40.264081+0000","last_became_active":"2024-04-26T16:26:23.214404+0000","last_became_peered":"2024-04-26T16:26:23.214404+0000","last_unstale":"2024-04-26T16:30:40.264081+0000","last_undegraded":"2024-04-26T16:30:40.264081+0000","last_fullsized":"2024-04-26T16:30:40.264081+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:32.981351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:32.981351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.689430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":944,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239860+0000","last_change":"2024-04-26T16:28:48.837013+0000","last_active":"2024-04-26T16:30:40.239860+0000","last_peered":"2024-04-26T16:30:40.239860+0000","last_clean":"2024-04-26T16:30:40.239860+0000","last_became_active":"2024-04-26T16:26:23.178678+0000","last_became_peered":"2024-04-26T16:26:23.178678+0000","last_unstale":"2024-04-26T16:30:40.239860+0000","last_undegraded":"2024-04-26T16:30:40.239860+0000","last_fullsized":"2024-04-26T16:30:40.239860+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:48.836858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:48.836858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:43.010161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":940,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.239018+0000","last_change":"2024-04-26T16:28:39.423393+0000","last_active":"2024-04-26T16:30:40.239018+0000","last_peered":"2024-04-26T16:30:40.239018+0000","last_clean":"2024-04-26T16:30:40.239018+0000","last_became_active":"2024-04-26T16:26:23.198295+0000","last_became_peered":"2024-04-26T16:26:23.198295+0000","last_unstale":"2024-04-26T16:30:40.239018+0000","last_undegraded":"2024-04-26T16:30:40.239018+0000","last_fullsized":"2024-04-26T16:30:40.239018+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:39.423249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:28:39.423249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:44.588519+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":430,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.251206+0000","last_change":"2024-04-26T16:30:16.192009+0000","last_active":"2024-04-26T16:30:40.251206+0000","last_peered":"2024-04-26T16:30:40.251206+0000","last_clean":"2024-04-26T16:30:40.251206+0000","last_became_active":"2024-04-26T16:26:23.199414+0000","last_became_peered":"2024-04-26T16:26:23.199414+0000","last_unstale":"2024-04-26T16:30:40.251206+0000","last_undegraded":"2024-04-26T16:30:40.251206+0000","last_fullsized":"2024-04-26T16:30:40.251206+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.191696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.191696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.626156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"388'20","reported_seq":972,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.264275+0000","last_change":"2024-04-26T16:28:48.140271+0000","last_active":"2024-04-26T16:30:40.264275+0000","last_peered":"2024-04-26T16:30:40.264275+0000","last_clean":"2024-04-26T16:30:40.264275+0000","last_became_active":"2024-04-26T16:28:25.232220+0000","last_became_peered":"2024-04-26T16:28:25.232220+0000","last_unstale":"2024-04-26T16:30:40.264275+0000","last_undegraded":"2024-04-26T16:30:40.264275+0000","last_fullsized":"2024-04-26T16:30:40.264275+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:48.140161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:48.140161+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:30:42.962372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":956,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.257360+0000","last_change":"2024-04-26T16:30:33.689694+0000","last_active":"2024-04-26T16:30:40.257360+0000","last_peered":"2024-04-26T16:30:40.257360+0000","last_clean":"2024-04-26T16:30:40.257360+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:30:40.257360+0000","last_undegraded":"2024-04-26T16:30:40.257360+0000","last_fullsized":"2024-04-26T16:30:40.257360+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.689564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.689564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:02.010334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":955,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.237829+0000","last_change":"2024-04-26T16:30:22.493662+0000","last_active":"2024-04-26T16:30:40.237829+0000","last_peered":"2024-04-26T16:30:40.237829+0000","last_clean":"2024-04-26T16:30:40.237829+0000","last_became_active":"2024-04-26T16:28:25.250239+0000","last_became_peered":"2024-04-26T16:28:25.250239+0000","last_unstale":"2024-04-26T16:30:40.237829+0000","last_undegraded":"2024-04-26T16:30:40.237829+0000","last_fullsized":"2024-04-26T16:30:40.237829+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.493412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.493412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.496740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":933,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-04-26T16:30:40.259514+0000","last_change":"2024-04-26T16:30:12.027247+0000","last_active":"2024-04-26T16:30:40.259514+0000","last_peered":"2024-04-26T16:30:40.259514+0000","last_clean":"2024-04-26T16:30:40.259514+0000","last_became_active":"2024-04-26T16:28:25.250958+0000","last_became_peered":"2024-04-26T16:28:25.250958+0000","last_unstale":"2024-04-26T16:30:40.259514+0000","last_undegraded":"2024-04-26T16:30:40.259514+0000","last_fullsized":"2024-04-26T16:30:40.259514+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.027062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.027062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.760868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":305,"num_pg":32,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":60,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":7},{"poolid":109,"num_pg":32,"stat_sum":{"num_bytes":8195937,"num_objects":176,"num_object_clones":10,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":0,"num_read":117,"num_read_kb":11,"num_write":632,"num_write_kb":12716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":4194787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26677248,"data_stored":24587787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1988,"ondisk_log_size":1988,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":775,"num_read_kb":770,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743594,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80104,"kb_used_data":4512,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93665048,"statfs":{"total":95995035648,"available":95913009152,"internally_reserved":0,"allocated":4620288,"data_stored":2169465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":432,"seq":1855425871902,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36796,"kb_used_data":4336,"kb_used_omap":15,"kb_used_meta":32432,"kb_avail":93708356,"statfs":{"total":95995035648,"available":95957356544,"internally_reserved":0,"allocated":4440064,"data_stored":2176296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15525,"internal_metadata":33211227},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74996,"kb_used_data":11872,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670156,"statfs":{"total":95995035648,"available":95918239744,"internally_reserved":0,"allocated":12156928,"data_stored":9772072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75876,"kb_used_data":4432,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93669276,"statfs":{"total":95995035648,"available":95917338624,"internally_reserved":0,"allocated":4538368,"data_stored":2169012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411549,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75000,"kb_used_data":7724,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670152,"statfs":{"total":95995035648,"available":95918235648,"internally_reserved":0,"allocated":7909376,"data_stored":5576736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084498,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105492,"kb_used_data":11904,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":93639660,"statfs":{"total":95995035648,"available":95887011840,"internally_reserved":0,"allocated":12189696,"data_stored":9771655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":95812040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855560,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79388,"kb_used_data":7952,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665764,"statfs":{"total":95995035648,"available":95913742336,"internally_reserved":0,"allocated":8142848,"data_stored":5771622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463102,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8195059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4284416,"data_stored":4000664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8556544,"data_stored":8195171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4485120,"data_stored":4194985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":368640,"data_stored":850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":305,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:30:52.201 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:30:52.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:52.282 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:52.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:52.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:52.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:52.538 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:52.641 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:52.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:52.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:52.799 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:52.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:52.930 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:52.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:53.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:53.060 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:53.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:53.190 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:53.195 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:53.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:53.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:53.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:53 smithi190 bash[23868]: cluster 2024-04-26T16:30:51.727759+0000 mgr.y (mgr.24401) 524 : cluster [DBG] pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:53.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:53 smithi190 bash[23868]: cluster 2024-04-26T16:30:51.727759+0000 mgr.y (mgr.24401) 524 : cluster [DBG] pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:53.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:53.451 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:53.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:53.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:53 smithi050 bash[29234]: cluster 2024-04-26T16:30:51.727759+0000 mgr.y (mgr.24401) 524 : cluster [DBG] pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:53.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:53 smithi050 bash[29234]: cluster 2024-04-26T16:30:51.727759+0000 mgr.y (mgr.24401) 524 : cluster [DBG] pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:53.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:53 smithi050 bash[21463]: cluster 2024-04-26T16:30:51.727759+0000 mgr.y (mgr.24401) 524 : cluster [DBG] pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:53.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:53 smithi050 bash[21463]: cluster 2024-04-26T16:30:51.727759+0000 mgr.y (mgr.24401) 524 : cluster [DBG] pgmap v798: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:53.581 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:53.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:53.711 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:53.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:53.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:53.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:53.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:53.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:53.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:54.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:54.103 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:54.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:54.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:54.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:54.363 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:54.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:54.494 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:54.597 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:54.729 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:54.738 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:30:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:30:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:30:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:54.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:54.882 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:54.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 -- 172.21.15.50:0/3374285639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b3810b640 msgr2=0x7f0b3810ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 --2- 172.21.15.50:0/3374285639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b3810b640 0x7f0b3810ba20 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009990 tx=0x7f0b34009ca0 comp rx=0 tx=0).stop 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 -- 172.21.15.50:0/3374285639 shutdown_connections 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 --2- 172.21.15.50:0/3374285639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b38106000 0x7f0b38112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 --2- 172.21.15.50:0/3374285639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b3810b640 0x7f0b3810ba20 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 --2- 172.21.15.50:0/3374285639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b38105640 0x7f0b38105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:54.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 -- 172.21.15.50:0/3374285639 >> 172.21.15.50:0/3374285639 conn(0x7f0b38101130 msgr2=0x7f0b38103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:54.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 -- 172.21.15.50:0/3374285639 shutdown_connections 2024-04-26T16:30:54.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 -- 172.21.15.50:0/3374285639 wait complete. 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 librados: starting msgr at 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 librados: starting objecter 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.958+0000 7f0b3efad700 1 Processor -- start 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- start start 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 librados: setting wanted keys 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 librados: calling monclient init 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b38105640 0x7f0b3819b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 0x7f0b3819b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b3810b640 0x7f0b3819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38114d40 con 0x7f0b3810b640 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38114bc0 con 0x7f0b38105640 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0b38114ec0 con 0x7f0b38106000 2024-04-26T16:30:54.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 0x7f0b3819b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 0x7f0b3819b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50678/0 (socket says 172.21.15.50:50678) 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 -- 172.21.15.50:0/2507944390 learned_addr learned my addr 172.21.15.50:0/2507944390 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3cd49700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b38105640 0x7f0b3819b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3d54a700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b3810b640 0x7f0b3819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 -- 172.21.15.50:0/2507944390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b38105640 msgr2=0x7f0b3819b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b38105640 0x7f0b3819b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 -- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b3810b640 msgr2=0x7f0b3819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b3810b640 0x7f0b3819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 -- 172.21.15.50:0/2507944390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b381a0280 con 0x7f0b38106000 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3d54a700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b3810b640 0x7f0b3819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2ffff700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 0x7f0b3819b7b0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800c9b0 tx=0x7f0b2800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 10 client.?.objecter ms_handle_connect 0x7f0b38106000 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:30:54.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28007b00 con 0x7f0b38106000 2024-04-26T16:30:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0b28007ca0 con 0x7f0b38106000 2024-04-26T16:30:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b381a0570 con 0x7f0b38106000 2024-04-26T16:30:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 10 client.41339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:30:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b28018400 con 0x7f0b38106000 2024-04-26T16:30:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b381a7df0 con 0x7f0b38106000 2024-04-26T16:30:54.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 librados: init done 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b3810dbd0 con 0x7f0b38106000 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0b2800f450 con 0x7f0b38106000 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0b20071f80 0x7f0b20074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(522..522 src has 378..522) ==== 9377+0+0 (secure 0 0 0) 0x7f0b280919b0 con 0x7f0b38106000 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 10 client.41339.objecter ms_dispatch 0x7f0b38001550 osd_map(522..522 src has 378..522) 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 3 client.41339.objecter handle_osd_map got epochs [522,522] > 0 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 3 client.41339.objecter handle_osd_map decoding full epoch 522 2024-04-26T16:30:54.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b3cd49700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0b20071f80 0x7f0b20074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:30:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.962+0000 7f0b2dffb700 20 client.41339.objecter dump_active .. 0 homeless 2024-04-26T16:30:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.966+0000 7f0b2dffb700 10 client.41339.objecter ms_handle_connect 0x7f0b20071f80 2024-04-26T16:30:54.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.966+0000 7f0b3cd49700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0b20071f80 0x7f0b20074440 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 crypto rx=0x7f0b340098e0 tx=0x7f0b34005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:30:54.966 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:54.970+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f0b28014030 con 0x7f0b38106000 2024-04-26T16:30:54.986 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:55.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:55.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:55.136 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.202+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f0b3819c5d0 con 0x7f0b38106000 2024-04-26T16:30:55.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:55.250 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.254+0000 7f0b2dffb700 1 -- 172.21.15.50:0/2507944390 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v523) ==== 97+0+0 (secure 0 0 0) 0x7f0b2805d3e0 con 0x7f0b38106000 2024-04-26T16:30:55.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 10 librados: watch_flush enter 2024-04-26T16:30:55.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 10 librados: watch_flush exit 2024-04-26T16:30:55.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 20 client.41339.objecter shutdown clearing up homeless session... 2024-04-26T16:30:55.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 10 client.41339.objecter successfully canceled tick 2024-04-26T16:30:55.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0b20071f80 msgr2=0x7f0b20074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:55.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0b20071f80 0x7f0b20074440 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 crypto rx=0x7f0b340098e0 tx=0x7f0b34005960 comp rx=0 tx=0).stop 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 msgr2=0x7f0b3819b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 0x7f0b3819b7b0 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800c9b0 tx=0x7f0b2800ccc0 comp rx=0 tx=0).stop 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 shutdown_connections 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0b20071f80 0x7f0b20074440 unknown :-1 s=CLOSED pgs=3768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0b38105640 0x7f0b3819b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0b3810b640 0x7f0b3819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:55.256 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 --2- 172.21.15.50:0/2507944390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0b38106000 0x7f0b3819b7b0 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:30:55.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 >> 172.21.15.50:0/2507944390 conn(0x7f0b38101130 msgr2=0x7f0b38101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:30:55.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 shutdown_connections 2024-04-26T16:30:55.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 -- 172.21.15.50:0/2507944390 wait complete. 2024-04-26T16:30:55.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:30:55.258+0000 7f0b3efad700 1 librados: shutdown 2024-04-26T16:30:55.257 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:30:55.265 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:55.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:55.379 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:55 smithi190 bash[23868]: cluster 2024-04-26T16:30:53.728498+0000 mgr.y (mgr.24401) 525 : cluster [DBG] pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:55.379 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:55 smithi190 bash[23868]: cluster 2024-04-26T16:30:53.728498+0000 mgr.y (mgr.24401) 525 : cluster [DBG] pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:55.379 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:55 smithi190 bash[23868]: audit 2024-04-26T16:30:55.206519+0000 mon.c (mon.2) 652 : audit [INF] from='client.? 172.21.15.50:0/2507944390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.379 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:55 smithi190 bash[23868]: audit 2024-04-26T16:30:55.206519+0000 mon.c (mon.2) 652 : audit [INF] from='client.? 172.21.15.50:0/2507944390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.380 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:55 smithi190 bash[23868]: audit 2024-04-26T16:30:55.206830+0000 mon.a (mon.0) 2907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.380 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:55 smithi190 bash[23868]: audit 2024-04-26T16:30:55.206830+0000 mon.a (mon.0) 2907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:55.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:55.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[29234]: cluster 2024-04-26T16:30:53.728498+0000 mgr.y (mgr.24401) 525 : cluster [DBG] pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:55.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[29234]: cluster 2024-04-26T16:30:53.728498+0000 mgr.y (mgr.24401) 525 : cluster [DBG] pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:55.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[29234]: audit 2024-04-26T16:30:55.206519+0000 mon.c (mon.2) 652 : audit [INF] from='client.? 172.21.15.50:0/2507944390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[29234]: audit 2024-04-26T16:30:55.206519+0000 mon.c (mon.2) 652 : audit [INF] from='client.? 172.21.15.50:0/2507944390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[29234]: audit 2024-04-26T16:30:55.206830+0000 mon.a (mon.0) 2907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[29234]: audit 2024-04-26T16:30:55.206830+0000 mon.a (mon.0) 2907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[21463]: cluster 2024-04-26T16:30:53.728498+0000 mgr.y (mgr.24401) 525 : cluster [DBG] pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:55.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[21463]: cluster 2024-04-26T16:30:53.728498+0000 mgr.y (mgr.24401) 525 : cluster [DBG] pgmap v799: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:30:55.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[21463]: audit 2024-04-26T16:30:55.206519+0000 mon.c (mon.2) 652 : audit [INF] from='client.? 172.21.15.50:0/2507944390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[21463]: audit 2024-04-26T16:30:55.206519+0000 mon.c (mon.2) 652 : audit [INF] from='client.? 172.21.15.50:0/2507944390' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[21463]: audit 2024-04-26T16:30:55.206830+0000 mon.a (mon.0) 2907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.510 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:55 smithi050 bash[21463]: audit 2024-04-26T16:30:55.206830+0000 mon.a (mon.0) 2907 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:30:55.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:55.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:55.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:55.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:55.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:55.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:55.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:55.909 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:55.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:56.013 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:56.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:56.149 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:56.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:56.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:56.305 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: cluster 2024-04-26T16:30:55.244239+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: cluster 2024-04-26T16:30:55.244239+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: audit 2024-04-26T16:30:55.245814+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: audit 2024-04-26T16:30:55.245814+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: cluster 2024-04-26T16:30:55.250221+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-04-26T16:30:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: cluster 2024-04-26T16:30:55.250221+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: cluster 2024-04-26T16:30:56.225286+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[29234]: cluster 2024-04-26T16:30:56.225286+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: cluster 2024-04-26T16:30:55.244239+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: cluster 2024-04-26T16:30:55.244239+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: audit 2024-04-26T16:30:55.245814+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: audit 2024-04-26T16:30:55.245814+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: cluster 2024-04-26T16:30:55.250221+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: cluster 2024-04-26T16:30:55.250221+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: cluster 2024-04-26T16:30:56.225286+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-04-26T16:30:56.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:56 smithi050 bash[21463]: cluster 2024-04-26T16:30:56.225286+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-04-26T16:30:56.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:56.433 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:56.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:56.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: cluster 2024-04-26T16:30:55.244239+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: cluster 2024-04-26T16:30:55.244239+0000 mon.a (mon.0) 2908 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:30:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: audit 2024-04-26T16:30:55.245814+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: audit 2024-04-26T16:30:55.245814+0000 mon.a (mon.0) 2909 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:30:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: cluster 2024-04-26T16:30:55.250221+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-04-26T16:30:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: cluster 2024-04-26T16:30:55.250221+0000 mon.a (mon.0) 2910 : cluster [DBG] osdmap e523: 8 total, 8 up, 7 in 2024-04-26T16:30:56.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: cluster 2024-04-26T16:30:56.225286+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-04-26T16:30:56.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[23868]: cluster 2024-04-26T16:30:56.225286+0000 mon.a (mon.0) 2911 : cluster [DBG] osdmap e524: 8 total, 8 up, 7 in 2024-04-26T16:30:56.670 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:56.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:56.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:56.828 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:56.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:56.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:56.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:57.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:57.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:30:56 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:30:57.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:57.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:57.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:57.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:57.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:57.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:57 smithi190 bash[23868]: cluster 2024-04-26T16:30:55.729271+0000 mgr.y (mgr.24401) 526 : cluster [DBG] pgmap v801: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:57.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:57 smithi190 bash[23868]: cluster 2024-04-26T16:30:55.729271+0000 mgr.y (mgr.24401) 526 : cluster [DBG] pgmap v801: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:57.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:57 smithi190 bash[23868]: audit 2024-04-26T16:30:57.081846+0000 mon.c (mon.2) 653 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:57.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:57 smithi190 bash[23868]: audit 2024-04-26T16:30:57.081846+0000 mon.c (mon.2) 653 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:57.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:57.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:57.479 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:57.504 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[29234]: cluster 2024-04-26T16:30:55.729271+0000 mgr.y (mgr.24401) 526 : cluster [DBG] pgmap v801: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[29234]: cluster 2024-04-26T16:30:55.729271+0000 mgr.y (mgr.24401) 526 : cluster [DBG] pgmap v801: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[29234]: audit 2024-04-26T16:30:57.081846+0000 mon.c (mon.2) 653 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[29234]: audit 2024-04-26T16:30:57.081846+0000 mon.c (mon.2) 653 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[21463]: cluster 2024-04-26T16:30:55.729271+0000 mgr.y (mgr.24401) 526 : cluster [DBG] pgmap v801: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[21463]: cluster 2024-04-26T16:30:55.729271+0000 mgr.y (mgr.24401) 526 : cluster [DBG] pgmap v801: 228 pgs: 228 active+clean; 8.4 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[21463]: audit 2024-04-26T16:30:57.081846+0000 mon.c (mon.2) 653 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:57.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:57 smithi050 bash[21463]: audit 2024-04-26T16:30:57.081846+0000 mon.c (mon.2) 653 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:30:57.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:57.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:57.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:57.743 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:30:57.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:57.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:57.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:57.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:58.003 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:58.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:58.132 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:58.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:58.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:58.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:58.394 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:58.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:58.524 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: cluster 2024-04-26T16:30:56.960752+0000 osd.4 (osd.4) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: cluster 2024-04-26T16:30:56.960752+0000 osd.4 (osd.4) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: cluster 2024-04-26T16:30:56.960996+0000 osd.4 (osd.4) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:58.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: cluster 2024-04-26T16:30:56.960996+0000 osd.4 (osd.4) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: audit 2024-04-26T16:30:56.985195+0000 mgr.y (mgr.24401) 527 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: audit 2024-04-26T16:30:56.985195+0000 mgr.y (mgr.24401) 527 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: cluster 2024-04-26T16:30:57.283137+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-04-26T16:30:58.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:58 smithi190 bash[23868]: cluster 2024-04-26T16:30:57.283137+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-04-26T16:30:58.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: cluster 2024-04-26T16:30:56.960752+0000 osd.4 (osd.4) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: cluster 2024-04-26T16:30:56.960752+0000 osd.4 (osd.4) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: cluster 2024-04-26T16:30:56.960996+0000 osd.4 (osd.4) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: cluster 2024-04-26T16:30:56.960996+0000 osd.4 (osd.4) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: audit 2024-04-26T16:30:56.985195+0000 mgr.y (mgr.24401) 527 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: audit 2024-04-26T16:30:56.985195+0000 mgr.y (mgr.24401) 527 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: cluster 2024-04-26T16:30:57.283137+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-04-26T16:30:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[29234]: cluster 2024-04-26T16:30:57.283137+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: cluster 2024-04-26T16:30:56.960752+0000 osd.4 (osd.4) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: cluster 2024-04-26T16:30:56.960752+0000 osd.4 (osd.4) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: cluster 2024-04-26T16:30:56.960996+0000 osd.4 (osd.4) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: cluster 2024-04-26T16:30:56.960996+0000 osd.4 (osd.4) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: audit 2024-04-26T16:30:56.985195+0000 mgr.y (mgr.24401) 527 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: audit 2024-04-26T16:30:56.985195+0000 mgr.y (mgr.24401) 527 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: cluster 2024-04-26T16:30:57.283137+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-04-26T16:30:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:58 smithi050 bash[21463]: cluster 2024-04-26T16:30:57.283137+0000 mon.a (mon.0) 2912 : cluster [DBG] osdmap e525: 8 total, 8 up, 7 in 2024-04-26T16:30:58.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:30:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:30:58.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:58.783 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:58.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:58.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:58.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:59.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:59.046 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:30:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:30:59.050 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:30:59.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:59.178 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:59.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:59.310 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:30:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:30:59.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:59.439 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:30:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:30:59.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:59.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:59 smithi190 bash[23868]: cluster 2024-04-26T16:30:57.730056+0000 mgr.y (mgr.24401) 528 : cluster [DBG] pgmap v804: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:30:59.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:30:59 smithi190 bash[23868]: cluster 2024-04-26T16:30:57.730056+0000 mgr.y (mgr.24401) 528 : cluster [DBG] pgmap v804: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:30:59.569 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:30:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:30:59.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:59 smithi050 bash[29234]: cluster 2024-04-26T16:30:57.730056+0000 mgr.y (mgr.24401) 528 : cluster [DBG] pgmap v804: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:30:59.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:30:59 smithi050 bash[29234]: cluster 2024-04-26T16:30:57.730056+0000 mgr.y (mgr.24401) 528 : cluster [DBG] pgmap v804: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:30:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:59 smithi050 bash[21463]: cluster 2024-04-26T16:30:57.730056+0000 mgr.y (mgr.24401) 528 : cluster [DBG] pgmap v804: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:30:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:30:59 smithi050 bash[21463]: cluster 2024-04-26T16:30:57.730056+0000 mgr.y (mgr.24401) 528 : cluster [DBG] pgmap v804: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:30:59.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:59.702 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:30:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:30:59.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:59.834 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:30:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:30:59.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:59.965 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:30:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:30:59.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:00.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:00.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:31:00.094 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:00.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:00.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:00.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:00.335 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:31:00 smithi050 bash[32273]: debug 2024-04-26T16:31:00.290+0000 7fec7bdb5700 -1 snap_mapper.add_oid found existing snaps mapped on 305:e47ffa69:test-rados-api-smithi050-73372-80::foo:2, removing 2024-04-26T16:31:00.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:00.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:00.477 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:00.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:00.572 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:31:00 smithi190 bash[27377]: debug 2024-04-26T16:31:00.290+0000 7f5cba9a0700 -1 snap_mapper.add_oid found existing snaps mapped on 305:e47ffa69:test-rados-api-smithi050-73372-80::foo:2, removing 2024-04-26T16:31:00.573 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:31:00 smithi190 bash[43438]: debug 2024-04-26T16:31:00.290+0000 7f9aec8e9700 -1 snap_mapper.add_oid found existing snaps mapped on 305:e47ffa69:test-rados-api-smithi050-73372-80::foo:2, removing 2024-04-26T16:31:00.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:00.606 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:00.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:00.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:00.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:00.869 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:00.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:00.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:01.000 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:01.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:01.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:01.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:01.262 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:01.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:01.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:01.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:59.730848+0000 mgr.y (mgr.24401) 529 : cluster [DBG] pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:31:01.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:59.730848+0000 mgr.y (mgr.24401) 529 : cluster [DBG] pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:59.964962+0000 osd.1 (osd.1) 289 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:59.964962+0000 osd.1 (osd.1) 289 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:59.965188+0000 osd.1 (osd.1) 290 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:30:59.965188+0000 osd.1 (osd.1) 290 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:31:00.298590+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: cluster 2024-04-26T16:31:00.298590+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.304564+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.304564+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.305356+0000 mon.b (mon.1) 358 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.305356+0000 mon.b (mon.1) 358 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.305847+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.305847+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.306801+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[21463]: audit 2024-04-26T16:31:00.306801+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:59.730848+0000 mgr.y (mgr.24401) 529 : cluster [DBG] pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:59.730848+0000 mgr.y (mgr.24401) 529 : cluster [DBG] pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:59.964962+0000 osd.1 (osd.1) 289 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:59.964962+0000 osd.1 (osd.1) 289 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:59.965188+0000 osd.1 (osd.1) 290 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:30:59.965188+0000 osd.1 (osd.1) 290 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:31:00.298590+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-04-26T16:31:01.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: cluster 2024-04-26T16:31:00.298590+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.304564+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.304564+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.305356+0000 mon.b (mon.1) 358 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.305356+0000 mon.b (mon.1) 358 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.305847+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.305847+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.306801+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:01 smithi050 bash[29234]: audit 2024-04-26T16:31:00.306801+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- 172.21.15.50:0/2997902168 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 msgr2=0x7ff1d0110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/2997902168 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0110330 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7ff1c800b0a0 tx=0x7ff1c800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- 172.21.15.50:0/2997902168 shutdown_connections 2024-04-26T16:31:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/2997902168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1d0109560 0x7ff1d0109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/2997902168 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0110330 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/2997902168 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1d01034c0 0x7ff1d0103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- 172.21.15.50:0/2997902168 >> 172.21.15.50:0/2997902168 conn(0x7ff1d007a5d0 msgr2=0x7ff1d007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- 172.21.15.50:0/2997902168 shutdown_connections 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- 172.21.15.50:0/2997902168 wait complete. 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 librados: starting msgr at 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 librados: starting objecter 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 Processor -- start 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- start start 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 librados: setting wanted keys 2024-04-26T16:31:01.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 librados: calling monclient init 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1d01034c0 0x7ff1d0075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1d0109560 0x7ff1d00761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d0112b40 con 0x7ff1d0109560 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d01129c0 con 0x7ff1d01034c0 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.470+0000 7ff1d5a02700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d0112cc0 con 0x7ff1d0103e80 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1cf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1d0109560 0x7ff1d00761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49358/0 (socket says 172.21.15.50:49358) 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1cf7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1d0109560 0x7ff1d00761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54124/0 (socket says 172.21.15.50:54124) 2024-04-26T16:31:01.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 -- 172.21.15.50:0/486939349 learned_addr learned my addr 172.21.15.50:0/486939349 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1d01034c0 0x7ff1d0075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1d01034c0 msgr2=0x7ff1d0075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff1d01034c0 0x7ff1d0075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1d0109560 msgr2=0x7ff1d00761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff1d0109560 0x7ff1d00761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d01a7650 con 0x7ff1d0103e80 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0075c60 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400e9a0 tx=0x7ff1c400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 10 client.?.objecter ms_handle_connect 0x7ff1d0103e80 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400cbd0 con 0x7ff1d0103e80 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff1c400cd70 con 0x7ff1d0103e80 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d01a7940 con 0x7ff1d0103e80 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4018420 con 0x7ff1d0103e80 2024-04-26T16:31:01.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 10 client.41345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:01.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d01a7ed0 con 0x7ff1d0103e80 2024-04-26T16:31:01.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 1 librados: init done 2024-04-26T16:31:01.473 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff1d0104bd0 con 0x7ff1d0103e80 2024-04-26T16:31:01.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff1c4010430 con 0x7ff1d0103e80 2024-04-26T16:31:01.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff1b8071f80 0x7ff1b8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(527..527 src has 378..527) ==== 8992+0+0 (secure 0 0 0) 0x7ff1c4092320 con 0x7ff1d0103e80 2024-04-26T16:31:01.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 osd_map(527..527 src has 378..527) 2024-04-26T16:31:01.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 3 client.41345.objecter handle_osd_map got epochs [527,527] > 0 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 3 client.41345.objecter handle_osd_map decoding full epoch 527 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff1b8071f80 0x7ff1b8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 20 client.41345.objecter dump_active .. 0 homeless 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v527) ==== 54+0+16 (secure 0 0 0) 0x7ff1c40926d0 con 0x7ff1d0103e80 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 1 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff1d0064030 0x7ff1d0064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 1 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 0 1 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 4 2024-04-26T16:31:01.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 10 client.41345.objecter _send_command 1 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1d0076c50 con 0x7ff1d0064030 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff1b8071f80 0x7ff1b8074440 secure :-1 s=READY pgs=3769 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000bbb0 tx=0x7ff1c000b590 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.474+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1b8071f80 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1cf7fe700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff1d0064030 0x7ff1d0064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1cf7fe700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff1d0064030 0x7ff1d0064410 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d0064030 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d0076c50 con 0x7ff1d0064030 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 1: 0 ) 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 1 = system:0 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 4 2024-04-26T16:31:01.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.478+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 0 1 2024-04-26T16:31:01.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:01.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 2 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 2 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d006a110 osd=0 3 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 0 2 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 4 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 10 client.41345.objecter _send_command 2 2024-04-26T16:31:01.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.494+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d0063ab0 con 0x7ff1d0064030 2024-04-26T16:31:01.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d0063ab0 con 0x7ff1d0064030 2024-04-26T16:31:01.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 4 2024-04-26T16:31:01.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 0 2 2024-04-26T16:31:01.495 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:31:01.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff1d010af30 0x7ff1d01ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 1 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 4 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 10 client.41345.objecter _send_command 3 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff1d002e600 con 0x7ff1d010af30 2024-04-26T16:31:01.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff1d010af30 0x7ff1d01ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d010af30 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff1d010af30 0x7ff1d01ac0c0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d002e600 con 0x7ff1d010af30 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 3: 0 ) 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 3 = system:0 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 4 2024-04-26T16:31:01.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.498+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 1 3 2024-04-26T16:31:01.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 4 2024-04-26T16:31:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.518 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 4 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d002e490 osd=1 3 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 1 4 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 4 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 10 client.41345.objecter _send_command 4 2024-04-26T16:31:01.519 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d01ac600 con 0x7ff1d010af30 2024-04-26T16:31:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d01ac600 con 0x7ff1d010af30 2024-04-26T16:31:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 4 2024-04-26T16:31:01.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 1 4 2024-04-26T16:31:01.520 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 5 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1d010a610 0x7ff1d01af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 5 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 2 5 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 4 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 10 client.41345.objecter _send_command 5 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff1d002e080 con 0x7ff1d010a610 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.522+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1d010a610 0x7ff1d01af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d010a610 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1d010a610 0x7ff1d01af940 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d002e080 con 0x7ff1d010a610 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 5: 0 ) 2024-04-26T16:31:01.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 5 = system:0 2024-04-26T16:31:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 4 2024-04-26T16:31:01.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.526+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 2 5 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 6 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 6 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d010a4a0 osd=2 3 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 2 6 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 4 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 10 client.41345.objecter _send_command 6 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.546+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d002d560 con 0x7ff1d010a610 2024-04-26T16:31:01.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.550+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d002d560 con 0x7ff1d010a610 2024-04-26T16:31:01.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.550+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.550+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.550+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 4 2024-04-26T16:31:01.548 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.550+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 2 6 2024-04-26T16:31:01.549 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 7 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1d01ad490 0x7ff1d01b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 7 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 3 7 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 4 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 10 client.41345.objecter _send_command 7 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff1d01b5f10 con 0x7ff1d01ad490 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.554+0000 7ff1cf7fe700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1d01ad490 0x7ff1d01b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1cf7fe700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1d01ad490 0x7ff1d01b3010 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d01ad490 2024-04-26T16:31:01.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d01b5f10 con 0x7ff1d01ad490 2024-04-26T16:31:01.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 7: 0 ) 2024-04-26T16:31:01.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 7 = system:0 2024-04-26T16:31:01.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 4 2024-04-26T16:31:01.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.558+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 3 7 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:59.730848+0000 mgr.y (mgr.24401) 529 : cluster [DBG] pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:59.730848+0000 mgr.y (mgr.24401) 529 : cluster [DBG] pgmap v805: 228 pgs: 228 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:59.964962+0000 osd.1 (osd.1) 289 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:59.964962+0000 osd.1 (osd.1) 289 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:59.965188+0000 osd.1 (osd.1) 290 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:30:59.965188+0000 osd.1 (osd.1) 290 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:31:00.298590+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: cluster 2024-04-26T16:31:00.298590+0000 mon.a (mon.0) 2913 : cluster [DBG] osdmap e526: 8 total, 8 up, 7 in 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.304564+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.304564+0000 mon.a (mon.0) 2914 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.305356+0000 mon.b (mon.1) 358 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.305356+0000 mon.b (mon.1) 358 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.305847+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.305847+0000 mon.a (mon.0) 2915 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.306801+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:01 smithi190 bash[23868]: audit 2024-04-26T16:31:00.306801+0000 mon.b (mon.1) 359 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-79"}]: dispatch 2024-04-26T16:31:01.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 8 2024-04-26T16:31:01.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 8 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d002d960 osd=3 3 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 3 8 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 4 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 10 client.41345.objecter _send_command 8 2024-04-26T16:31:01.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.578+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d01acac0 con 0x7ff1d01ad490 2024-04-26T16:31:01.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d01acac0 con 0x7ff1d01ad490 2024-04-26T16:31:01.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 4 2024-04-26T16:31:01.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 3 8 2024-04-26T16:31:01.578 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 9 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff1d01b0b60 0x7ff1d01b4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 9 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 4 9 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 4 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 10 client.41345.objecter _send_command 9 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff1d01b9900 con 0x7ff1d01b0b60 2024-04-26T16:31:01.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff1d01b0b60 0x7ff1d01b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d01b0b60 2024-04-26T16:31:01.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.582+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff1d01b0b60 0x7ff1d01b4770 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.586+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d01b9900 con 0x7ff1d01b0b60 2024-04-26T16:31:01.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.586+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 9: 0 ) 2024-04-26T16:31:01.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.586+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 9 = system:0 2024-04-26T16:31:01.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.586+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 4 2024-04-26T16:31:01.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.586+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 4 9 2024-04-26T16:31:01.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 10 2024-04-26T16:31:01.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 10 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01ac870 osd=4 3 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 4 10 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 4 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 10 client.41345.objecter _send_command 10 2024-04-26T16:31:01.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.606+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d01b00d0 con 0x7ff1d01b0b60 2024-04-26T16:31:01.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d01b00d0 con 0x7ff1d01b0b60 2024-04-26T16:31:01.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 4 2024-04-26T16:31:01.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 4 10 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 11 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1d01b42d0 0x7ff1d01ba170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 11 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 5 11 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 4 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 10 client.41345.objecter _send_command 11 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff1d01afff0 con 0x7ff1d01b42d0 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.610+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1d01b42d0 0x7ff1d01ba170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d01b42d0 2024-04-26T16:31:01.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1ce7fc700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1d01b42d0 0x7ff1d01ba170 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d01afff0 con 0x7ff1d01b42d0 2024-04-26T16:31:01.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 11: 0 ) 2024-04-26T16:31:01.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 11 = system:0 2024-04-26T16:31:01.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 4 2024-04-26T16:31:01.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.614+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 5 11 2024-04-26T16:31:01.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:01.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 12 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 12 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01afe80 osd=5 3 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 5 12 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 4 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 10 client.41345.objecter _send_command 12 2024-04-26T16:31:01.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.634+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d01afff0 con 0x7ff1d01b42d0 2024-04-26T16:31:01.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:01.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d01afff0 con 0x7ff1d01b42d0 2024-04-26T16:31:01.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 4 2024-04-26T16:31:01.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 5 12 2024-04-26T16:31:01.634 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 13 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff1d01babc0 0x7ff1d01bcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 13 2024-04-26T16:31:01.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 6 13 2024-04-26T16:31:01.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 4 2024-04-26T16:31:01.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 10 client.41345.objecter _send_command 13 2024-04-26T16:31:01.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff1d01bff80 con 0x7ff1d01babc0 2024-04-26T16:31:01.636 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1cf7fe700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff1d01babc0 0x7ff1d01bcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:01.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d01babc0 2024-04-26T16:31:01.637 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.638+0000 7ff1cf7fe700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff1d01babc0 0x7ff1d01bcfb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.642+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d01bff80 con 0x7ff1d01babc0 2024-04-26T16:31:01.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.642+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 13: 0 ) 2024-04-26T16:31:01.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.642+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 13 = system:0 2024-04-26T16:31:01.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.642+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 4 2024-04-26T16:31:01.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.642+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 6 13 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 14 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 14 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01b0240 osd=6 3 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 6 14 2024-04-26T16:31:01.658 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 4 2024-04-26T16:31:01.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 10 client.41345.objecter _send_command 14 2024-04-26T16:31:01.659 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.658+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d01bed10 con 0x7ff1d01babc0 2024-04-26T16:31:01.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.662+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d01bed10 con 0x7ff1d01babc0 2024-04-26T16:31:01.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.662+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.662+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.662+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 4 2024-04-26T16:31:01.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.662+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 6 14 2024-04-26T16:31:01.661 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:31:01.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 15 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1d01bf430 0x7ff1d01bf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 15 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 7 15 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 4 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 10 client.41345.objecter _send_command 15 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff1d01c3350 con 0x7ff1d01bf430 2024-04-26T16:31:01.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1d01bf430 0x7ff1d01bf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:01.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1ceffd700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1d01bf430 0x7ff1d01bf870 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:01.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d4a00700 10 client.41345.objecter ms_handle_connect 0x7ff1d01bf430 2024-04-26T16:31:01.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1d01c3350 con 0x7ff1d01bf430 2024-04-26T16:31:01.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 15: 0 ) 2024-04-26T16:31:01.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 15 = system:0 2024-04-26T16:31:01.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 4 2024-04-26T16:31:01.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.666+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 7 15 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 10 client.41345.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign -1 16 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 20 client.41345.objecter _get_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_remove -1 16 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 20 client.41345.objecter get_session s=0x7ff1d01b00b0 osd=7 3 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 15 client.41345.objecter _session_command_op_assign 7 16 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 4 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 10 client.41345.objecter _send_command 16 2024-04-26T16:31:01.692 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.694+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff1d01bd540 con 0x7ff1d01bf430 2024-04-26T16:31:01.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d4a00700 1 -- 172.21.15.50:0/486939349 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff1d01bd540 con 0x7ff1d01bf430 2024-04-26T16:31:01.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d4a00700 10 client.41345.objecter ms_dispatch 0x7ff1d0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:01.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d4a00700 10 client.41345.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:01.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d4a00700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 4 2024-04-26T16:31:01.696 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d4a00700 15 client.41345.objecter _session_command_op_remove 7 16 2024-04-26T16:31:01.697 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 librados: watch_flush enter 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 librados: watch_flush exit 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.0 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff1d0064030 msgr2=0x7ff1d0064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff1d0064030 0x7ff1d0064410 crc :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d006a110 osd=0 1 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.1 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff1d010af30 msgr2=0x7ff1d01ac0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff1d010af30 0x7ff1d01ac0c0 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002e490 osd=1 1 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.2 2024-04-26T16:31:01.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1d010a610 msgr2=0x7ff1d01af940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff1d010a610 0x7ff1d01af940 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d010a4a0 osd=2 1 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.3 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1d01ad490 msgr2=0x7ff1d01b3010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff1d01ad490 0x7ff1d01b3010 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d002d960 osd=3 1 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.4 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff1d01b0b60 msgr2=0x7ff1d01b4770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff1d01b0b60 0x7ff1d01b4770 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01ac870 osd=4 1 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.5 2024-04-26T16:31:01.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1d01b42d0 msgr2=0x7ff1d01ba170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff1d01b42d0 0x7ff1d01ba170 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01afe80 osd=5 1 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.6 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1cf7fe700 1 -- 172.21.15.50:0/486939349 reap_dead start 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff1d01babc0 msgr2=0x7ff1d01bcfb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff1d01babc0 0x7ff1d01bcfb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1cf7fe700 1 -- 172.21.15.50:0/486939349 reap_dead start 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b0240 osd=6 1 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter close_session for osd.7 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1d01bf430 msgr2=0x7ff1d01bf870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff1d01bf430 0x7ff1d01bf870 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter put_session s=0x7ff1d01b00b0 osd=7 1 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 20 client.41345.objecter shutdown clearing up homeless session... 2024-04-26T16:31:01.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.698+0000 7ff1d5a02700 10 client.41345.objecter successfully canceled tick 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1cf7fe700 1 -- 172.21.15.50:0/486939349 reap_dead start 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1cf7fe700 1 -- 172.21.15.50:0/486939349 reap_dead start 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff1b8071f80 msgr2=0x7ff1b8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff1b8071f80 0x7ff1b8074440 secure :-1 s=READY pgs=3769 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000bbb0 tx=0x7ff1c000b590 comp rx=0 tx=0).stop 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 msgr2=0x7ff1d0075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0075c60 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400e9a0 tx=0x7ff1c400ecb0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 shutdown_connections 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff1d0103e80 0x7ff1d0075c60 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 --2- 172.21.15.50:0/486939349 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff1b8071f80 0x7ff1b8074440 unknown :-1 s=CLOSED pgs=3769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 >> 172.21.15.50:0/486939349 conn(0x7ff1d007a5d0 msgr2=0x7ff1d0101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 shutdown_connections 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 -- 172.21.15.50:0/486939349 wait complete. 2024-04-26T16:31:01.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:01.702+0000 7ff1d5a02700 1 librados: shutdown 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:31:01.702 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:31:01.703 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:31:01.703 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:31:01.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:01.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:01.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:01.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:01.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:01.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:02.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:02.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:02.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:02.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:02.289 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:02.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:02.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[29234]: cluster 2024-04-26T16:31:01.304942+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[29234]: cluster 2024-04-26T16:31:01.304942+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[29234]: audit 2024-04-26T16:31:01.477150+0000 mon.c (mon.2) 654 : audit [DBG] from='client.? 172.21.15.50:0/486939349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[29234]: audit 2024-04-26T16:31:01.477150+0000 mon.c (mon.2) 654 : audit [DBG] from='client.? 172.21.15.50:0/486939349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[21463]: cluster 2024-04-26T16:31:01.304942+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[21463]: cluster 2024-04-26T16:31:01.304942+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[21463]: audit 2024-04-26T16:31:01.477150+0000 mon.c (mon.2) 654 : audit [DBG] from='client.? 172.21.15.50:0/486939349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:02.403 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:02 smithi050 bash[21463]: audit 2024-04-26T16:31:01.477150+0000 mon.c (mon.2) 654 : audit [DBG] from='client.? 172.21.15.50:0/486939349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:02.419 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:02.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:02.549 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:02.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:02.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:02 smithi190 bash[23868]: cluster 2024-04-26T16:31:01.304942+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-04-26T16:31:02.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:02 smithi190 bash[23868]: cluster 2024-04-26T16:31:01.304942+0000 mon.a (mon.0) 2916 : cluster [DBG] osdmap e527: 8 total, 8 up, 7 in 2024-04-26T16:31:02.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:02 smithi190 bash[23868]: audit 2024-04-26T16:31:01.477150+0000 mon.c (mon.2) 654 : audit [DBG] from='client.? 172.21.15.50:0/486939349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:02.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:02 smithi190 bash[23868]: audit 2024-04-26T16:31:01.477150+0000 mon.c (mon.2) 654 : audit [DBG] from='client.? 172.21.15.50:0/486939349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:02.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:02.678 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:02.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:02.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:02.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:02.942 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:02.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:03.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:03.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:03.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:03.206 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:03.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:03.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:03.450 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:03.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: cluster 2024-04-26T16:31:01.731978+0000 mgr.y (mgr.24401) 530 : cluster [DBG] pgmap v808: 196 pgs: 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: cluster 2024-04-26T16:31:01.731978+0000 mgr.y (mgr.24401) 530 : cluster [DBG] pgmap v808: 196 pgs: 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: cluster 2024-04-26T16:31:02.315523+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: cluster 2024-04-26T16:31:02.315523+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: audit 2024-04-26T16:31:02.328809+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: audit 2024-04-26T16:31:02.328809+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: audit 2024-04-26T16:31:02.329393+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[29234]: audit 2024-04-26T16:31:02.329393+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.459 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: cluster 2024-04-26T16:31:01.731978+0000 mgr.y (mgr.24401) 530 : cluster [DBG] pgmap v808: 196 pgs: 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: cluster 2024-04-26T16:31:01.731978+0000 mgr.y (mgr.24401) 530 : cluster [DBG] pgmap v808: 196 pgs: 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: cluster 2024-04-26T16:31:02.315523+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: cluster 2024-04-26T16:31:02.315523+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: audit 2024-04-26T16:31:02.328809+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: audit 2024-04-26T16:31:02.328809+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: audit 2024-04-26T16:31:02.329393+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.460 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:03 smithi050 bash[21463]: audit 2024-04-26T16:31:02.329393+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:03.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: cluster 2024-04-26T16:31:01.731978+0000 mgr.y (mgr.24401) 530 : cluster [DBG] pgmap v808: 196 pgs: 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: cluster 2024-04-26T16:31:01.731978+0000 mgr.y (mgr.24401) 530 : cluster [DBG] pgmap v808: 196 pgs: 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: cluster 2024-04-26T16:31:02.315523+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: cluster 2024-04-26T16:31:02.315523+0000 mon.a (mon.0) 2917 : cluster [DBG] osdmap e528: 8 total, 8 up, 7 in 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: audit 2024-04-26T16:31:02.328809+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: audit 2024-04-26T16:31:02.328809+0000 mon.a (mon.0) 2918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: audit 2024-04-26T16:31:02.329393+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:03 smithi190 bash[23868]: audit 2024-04-26T16:31:02.329393+0000 mon.b (mon.1) 360 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:03.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:03.607 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:03.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:03.736 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:03.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:03.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:03.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:03.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:03.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:04.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:04.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:04.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:04.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:04.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:04.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:04.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.321257+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.321257+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: cluster 2024-04-26T16:31:03.326617+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: cluster 2024-04-26T16:31:03.326617+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.335244+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.335244+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.357732+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.357732+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.358326+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:04 smithi190 bash[23868]: audit 2024-04-26T16:31:03.358326+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.513 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:31:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.321257+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.321257+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: cluster 2024-04-26T16:31:03.326617+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: cluster 2024-04-26T16:31:03.326617+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.335244+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:04.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.335244+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.357732+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.357732+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.358326+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[29234]: audit 2024-04-26T16:31:03.358326+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.321257+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.321257+0000 mon.a (mon.0) 2919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: cluster 2024-04-26T16:31:03.326617+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: cluster 2024-04-26T16:31:03.326617+0000 mon.a (mon.0) 2920 : cluster [DBG] osdmap e529: 8 total, 8 up, 7 in 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.335244+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.335244+0000 mon.b (mon.1) 361 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.357732+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.357732+0000 mon.a (mon.0) 2921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.358326+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:04 smithi050 bash[21463]: audit 2024-04-26T16:31:03.358326+0000 mon.b (mon.1) 362 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:04.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-26T16:31:04.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:04.644 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:04.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:04.774 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:04.877 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:04.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:04.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:05.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:05.035 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:05.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:05.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:05.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:05.297 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:05.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:05.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:05.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: cluster 2024-04-26T16:31:03.732758+0000 mgr.y (mgr.24401) 531 : cluster [DBG] pgmap v811: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: cluster 2024-04-26T16:31:03.732758+0000 mgr.y (mgr.24401) 531 : cluster [DBG] pgmap v811: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: audit 2024-04-26T16:31:04.324110+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: audit 2024-04-26T16:31:04.324110+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: cluster 2024-04-26T16:31:04.328796+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: cluster 2024-04-26T16:31:04.328796+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: audit 2024-04-26T16:31:04.334224+0000 mon.b (mon.1) 363 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: audit 2024-04-26T16:31:04.334224+0000 mon.b (mon.1) 363 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: audit 2024-04-26T16:31:04.341332+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:05 smithi190 bash[23868]: audit 2024-04-26T16:31:04.341332+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.558 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: cluster 2024-04-26T16:31:03.732758+0000 mgr.y (mgr.24401) 531 : cluster [DBG] pgmap v811: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: cluster 2024-04-26T16:31:03.732758+0000 mgr.y (mgr.24401) 531 : cluster [DBG] pgmap v811: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: audit 2024-04-26T16:31:04.324110+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: audit 2024-04-26T16:31:04.324110+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: cluster 2024-04-26T16:31:04.328796+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: cluster 2024-04-26T16:31:04.328796+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: audit 2024-04-26T16:31:04.334224+0000 mon.b (mon.1) 363 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: audit 2024-04-26T16:31:04.334224+0000 mon.b (mon.1) 363 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: audit 2024-04-26T16:31:04.341332+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[29234]: audit 2024-04-26T16:31:04.341332+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: cluster 2024-04-26T16:31:03.732758+0000 mgr.y (mgr.24401) 531 : cluster [DBG] pgmap v811: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: cluster 2024-04-26T16:31:03.732758+0000 mgr.y (mgr.24401) 531 : cluster [DBG] pgmap v811: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: audit 2024-04-26T16:31:04.324110+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: audit 2024-04-26T16:31:04.324110+0000 mon.a (mon.0) 2922 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: cluster 2024-04-26T16:31:04.328796+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: cluster 2024-04-26T16:31:04.328796+0000 mon.a (mon.0) 2923 : cluster [DBG] osdmap e530: 8 total, 8 up, 7 in 2024-04-26T16:31:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: audit 2024-04-26T16:31:04.334224+0000 mon.b (mon.1) 363 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: audit 2024-04-26T16:31:04.334224+0000 mon.b (mon.1) 363 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: audit 2024-04-26T16:31:04.341332+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:05 smithi050 bash[21463]: audit 2024-04-26T16:31:04.341332+0000 mon.a (mon.0) 2924 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:05.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:05.690 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:05.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:05.820 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:05.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:05.951 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:05.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 -- 172.21.15.50:0/3494315085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102ce0 msgr2=0x7f838810f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- 172.21.15.50:0/3494315085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102ce0 0x7f838810f1d0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f838000b0a0 tx=0x7f838000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 -- 172.21.15.50:0/3494315085 shutdown_connections 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- 172.21.15.50:0/3494315085 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102ce0 0x7f838810f1d0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- 172.21.15.50:0/3494315085 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f8388108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- 172.21.15.50:0/3494315085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8388102320 0x7f83881027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 -- 172.21.15.50:0/3494315085 >> 172.21.15.50:0/3494315085 conn(0x7f83880792d0 msgr2=0x7f83880796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 -- 172.21.15.50:0/3494315085 shutdown_connections 2024-04-26T16:31:05.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 -- 172.21.15.50:0/3494315085 wait complete. 2024-04-26T16:31:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 librados: starting msgr at 2024-04-26T16:31:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 librados: starting objecter 2024-04-26T16:31:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 Processor -- start 2024-04-26T16:31:05.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 -- start start 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 librados: setting wanted keys 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 librados: calling monclient init 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102320 0x7f8388199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8388102ce0 0x7f83881995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:05.998+0000 7f838ef8a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f838819d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f83881119a0 con 0x7f8388102320 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8388111820 con 0x7f8388102ce0 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8388111b20 con 0x7f8388108390 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f838819d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838cd26700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102320 0x7f8388199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f838819d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:49378/0 (socket says 172.21.15.50:49378) 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 -- 172.21.15.50:0/828013756 learned_addr learned my addr 172.21.15.50:0/828013756 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:05.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8387fff700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8388102ce0 0x7f83881995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 -- 172.21.15.50:0/828013756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8388102ce0 msgr2=0x7f83881995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8388102ce0 0x7f83881995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 -- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102320 msgr2=0x7f8388199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102320 0x7f8388199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 -- 172.21.15.50:0/828013756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838819e0a0 con 0x7f8388108390 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838cd26700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102320 0x7f8388199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838d527700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f838819d960 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f838000b780 tx=0x7f8380004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 10 client.?.objecter ms_handle_connect 0x7f8388108390 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380021070 con 0x7f8388108390 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8380007b80 con 0x7f8388108390 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838819e330 con 0x7f8388108390 2024-04-26T16:31:06.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 10 client.41351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380005040 con 0x7f8388108390 2024-04-26T16:31:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83881a5c10 con 0x7f8388108390 2024-04-26T16:31:06.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 librados: init done 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388103a40 con 0x7f8388108390 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8380024070 con 0x7f8388108390 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8378071e50 0x7f8378074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(531..531 src has 378..531) ==== 9409+0+0 (secure 0 0 0) 0x7f838000f020 con 0x7f8388108390 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 10 client.41351.objecter ms_dispatch 0x7f8388001550 osd_map(531..531 src has 378..531) 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 3 client.41351.objecter handle_osd_map got epochs [531,531] > 0 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 3 client.41351.objecter handle_osd_map decoding full epoch 531 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f838cd26700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8378071e50 0x7f8378074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.002+0000 7f8385ffb700 20 client.41351.objecter dump_active .. 0 homeless 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.006+0000 7f8385ffb700 10 client.41351.objecter ms_handle_connect 0x7f8378071e50 2024-04-26T16:31:06.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.006+0000 7f838cd26700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8378071e50 0x7f8378074310 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7f837c009680 tx=0x7f837c009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:06.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.010+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f838002b070 con 0x7f8388108390 2024-04-26T16:31:06.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:06.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:06.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:06.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:06.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:06.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.230+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f838806a110 con 0x7f8378071e50 2024-04-26T16:31:06.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.250+0000 7f8385ffb700 1 -- 172.21.15.50:0/828013756 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+583292 (secure 0 0 0) 0x7f838806a110 con 0x7f8378071e50 2024-04-26T16:31:06.253 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:31:06.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 10 librados: watch_flush enter 2024-04-26T16:31:06.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 10 librados: watch_flush exit 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 20 client.41351.objecter shutdown clearing up homeless session... 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 10 client.41351.objecter successfully canceled tick 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8378071e50 msgr2=0x7f8378074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8378071e50 0x7f8378074310 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7f837c009680 tx=0x7f837c009260 comp rx=0 tx=0).stop 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 msgr2=0x7f838819d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f838819d960 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f838000b780 tx=0x7f8380004d00 comp rx=0 tx=0).stop 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 shutdown_connections 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8378071e50 0x7f8378074310 unknown :-1 s=CLOSED pgs=3770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8388108390 0x7f838819d960 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8388102320 0x7f8388199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 --2- 172.21.15.50:0/828013756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8388102ce0 0x7f83881995d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 >> 172.21.15.50:0/828013756 conn(0x7f83880792d0 msgr2=0x7f83881011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 shutdown_connections 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 -- 172.21.15.50:0/828013756 wait complete. 2024-04-26T16:31:06.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:06.254+0000 7f838ef8a700 1 librados: shutdown 2024-04-26T16:31:06.260 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:31:06.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:06.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:06.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:06.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:05.336109+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:06.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:05.336109+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:06.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:05.342988+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:05.342988+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: cluster 2024-04-26T16:31:05.343884+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: cluster 2024-04-26T16:31:05.343884+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:05.345998+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:05.345998+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:06.225243+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:06.225243+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: cluster 2024-04-26T16:31:06.228657+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: cluster 2024-04-26T16:31:06.228657+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:06.236971+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:06.236971+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:06.246077+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[29234]: audit 2024-04-26T16:31:06.246077+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:05.336109+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:05.336109+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:05.342988+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:05.342988+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: cluster 2024-04-26T16:31:05.343884+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: cluster 2024-04-26T16:31:05.343884+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:05.345998+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:05.345998+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:06.225243+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:06.225243+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:06.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: cluster 2024-04-26T16:31:06.228657+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-04-26T16:31:06.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: cluster 2024-04-26T16:31:06.228657+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-04-26T16:31:06.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:06.236971+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:06.236971+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:06.246077+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:06 smithi050 bash[21463]: audit 2024-04-26T16:31:06.246077+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.461 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:06.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:31:06.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:06.591 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:06.694 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:06.719 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:06.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:06.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:05.336109+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:05.336109+0000 mon.a (mon.0) 2925 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:05.342988+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:05.342988+0000 mon.b (mon.1) 364 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: cluster 2024-04-26T16:31:05.343884+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: cluster 2024-04-26T16:31:05.343884+0000 mon.a (mon.0) 2926 : cluster [DBG] osdmap e531: 8 total, 8 up, 7 in 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:05.345998+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:05.345998+0000 mon.a (mon.0) 2927 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:06.225243+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:06.225243+0000 mon.a (mon.0) 2928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: cluster 2024-04-26T16:31:06.228657+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: cluster 2024-04-26T16:31:06.228657+0000 mon.a (mon.0) 2929 : cluster [DBG] osdmap e532: 8 total, 8 up, 7 in 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:06.236971+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:06.236971+0000 mon.b (mon.1) 365 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:06.246077+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[23868]: audit 2024-04-26T16:31:06.246077+0000 mon.a (mon.0) 2930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:06.850 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:06.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:06.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:06.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:07.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:07.094 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:31:06 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:31:07.109 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:07.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:07.242 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:07.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:07.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:07.373 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:07.438 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":815,"stamp":"2024-04-26T16:31:06.225980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8792439,"num_objects":374,"num_object_clones":10,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":2289,"num_read_kb":2029,"num_write":1541,"num_write_kb":15184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":5967718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3027,"ondisk_log_size":3027,"up":609,"acting":609,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":684,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":528316,"kb_used_data":53412,"kb_used_omap":24,"kb_used_meta":474791,"kb_avail":655687748,"statfs":{"total":671965249536,"available":671424253952,"internally_reserved":0,"allocated":54693888,"data_stored":37820026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25087,"internal_metadata":486186497},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.914521"},"pg_stats":[{"pgid":"306.1f","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.335925+0000","last_change":"2024-04-26T16:31:02.335925+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.335925+0000","last_undegraded":"2024-04-26T16:31:02.335925+0000","last_fullsized":"2024-04-26T16:31:02.335925+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.14","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.323596+0000","last_change":"2024-04-26T16:31:02.323596+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.323596+0000","last_undegraded":"2024-04-26T16:31:02.323596+0000","last_fullsized":"2024-04-26T16:31:02.323596+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.10","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.321751+0000","last_change":"2024-04-26T16:31:02.321751+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.321751+0000","last_undegraded":"2024-04-26T16:31:02.321751+0000","last_fullsized":"2024-04-26T16:31:02.321751+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.f","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.325571+0000","last_change":"2024-04-26T16:31:02.325571+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.325571+0000","last_undegraded":"2024-04-26T16:31:02.325571+0000","last_fullsized":"2024-04-26T16:31:02.325571+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.b","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.322440+0000","last_change":"2024-04-26T16:31:02.322440+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.322440+0000","last_undegraded":"2024-04-26T16:31:02.322440+0000","last_fullsized":"2024-04-26T16:31:02.322440+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.6","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.325773+0000","last_change":"2024-04-26T16:31:02.325773+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.325773+0000","last_undegraded":"2024-04-26T16:31:02.325773+0000","last_fullsized":"2024-04-26T16:31:02.325773+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"306.1","version":"0'0","reported_seq":1,"reported_epoch":528,"state":"creating+peering","last_fresh":"2024-04-26T16:31:02.325401+0000","last_change":"2024-04-26T16:31:02.325401+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.325401+0000","last_undegraded":"2024-04-26T16:31:02.325401+0000","last_fullsized":"2024-04-26T16:31:02.325401+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":528,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"306.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:02.302496+0000","last_change":"2024-04-26T16:31:02.302496+0000","last_active":"2024-04-26T16:31:02.302496+0000","last_peered":"2024-04-26T16:31:02.302496+0000","last_clean":"2024-04-26T16:31:02.302496+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:02.302496+0000","last_undegraded":"2024-04-26T16:31:02.302496+0000","last_fullsized":"2024-04-26T16:31:02.302496+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:02.302496+0000","last_clean_scrub_stamp":"2024-04-26T16:31:02.302496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"109.12","version":"389'86","reported_seq":1021,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317549+0000","last_change":"2024-04-26T16:29:29.291952+0000","last_active":"2024-04-26T16:31:00.317549+0000","last_peered":"2024-04-26T16:31:00.317549+0000","last_clean":"2024-04-26T16:31:00.317549+0000","last_became_active":"2024-04-26T16:23:44.188737+0000","last_became_peered":"2024-04-26T16:23:44.188737+0000","last_unstale":"2024-04-26T16:31:00.317549+0000","last_undegraded":"2024-04-26T16:31:00.317549+0000","last_fullsized":"2024-04-26T16:31:00.317549+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:29:10.006584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.006584+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011941726,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.13","version":"389'52","reported_seq":822,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296610+0000","last_change":"2024-04-26T16:30:24.606400+0000","last_active":"2024-04-26T16:31:00.296610+0000","last_peered":"2024-04-26T16:31:00.296610+0000","last_clean":"2024-04-26T16:31:00.296610+0000","last_became_active":"2024-04-26T16:26:23.192498+0000","last_became_peered":"2024-04-26T16:26:23.192498+0000","last_unstale":"2024-04-26T16:31:00.296610+0000","last_undegraded":"2024-04-26T16:31:00.296610+0000","last_fullsized":"2024-04-26T16:31:00.296610+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'52","last_scrub_stamp":"2024-04-26T16:30:24.606145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.606145+0000","objects_scrubbed":4,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:39.989241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064602280000000002,"stat_sum":{"num_bytes":41,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":16,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.10","version":"469'150","reported_seq":1174,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.324766+0000","last_change":"2024-04-26T16:29:29.501691+0000","last_active":"2024-04-26T16:31:00.324766+0000","last_peered":"2024-04-26T16:31:00.324766+0000","last_clean":"2024-04-26T16:31:00.324766+0000","last_became_active":"2024-04-26T16:26:26.135726+0000","last_became_peered":"2024-04-26T16:26:26.135726+0000","last_unstale":"2024-04-26T16:31:00.324766+0000","last_undegraded":"2024-04-26T16:31:00.324766+0000","last_fullsized":"2024-04-26T16:31:00.324766+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'126","last_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_deep_scrub":"400'126","last_deep_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_clean_scrub_stamp":"2024-04-26T16:27:49.058542+0000","objects_scrubbed":7,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001241277,"stat_sum":{"num_bytes":103,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":44,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":77,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.11","version":"236'34","reported_seq":993,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.313338+0000","last_change":"2024-04-26T16:29:57.929311+0000","last_active":"2024-04-26T16:31:01.313338+0000","last_peered":"2024-04-26T16:31:01.313338+0000","last_clean":"2024-04-26T16:31:01.313338+0000","last_became_active":"2024-04-26T16:23:44.102085+0000","last_became_peered":"2024-04-26T16:23:44.102085+0000","last_unstale":"2024-04-26T16:31:01.313338+0000","last_undegraded":"2024-04-26T16:31:01.313338+0000","last_fullsized":"2024-04-26T16:31:01.313338+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:29:57.929076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.929076+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:15.200371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066858430000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.16","version":"485'84","reported_seq":939,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319029+0000","last_change":"2024-04-26T16:29:29.294557+0000","last_active":"2024-04-26T16:31:00.319029+0000","last_peered":"2024-04-26T16:31:00.319029+0000","last_clean":"2024-04-26T16:31:00.319029+0000","last_became_active":"2024-04-26T16:26:23.214752+0000","last_became_peered":"2024-04-26T16:26:23.214752+0000","last_unstale":"2024-04-26T16:31:00.319029+0000","last_undegraded":"2024-04-26T16:31:00.319029+0000","last_fullsized":"2024-04-26T16:31:00.319029+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:29:04.018776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.018776+0000","objects_scrubbed":7,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060185220000000001,"stat_sum":{"num_bytes":78,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":27,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.17","version":"520'52","reported_seq":972,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.338070+0000","last_change":"2024-04-26T16:30:02.546379+0000","last_active":"2024-04-26T16:31:02.338070+0000","last_peered":"2024-04-26T16:31:02.338070+0000","last_clean":"2024-04-26T16:31:02.338070+0000","last_became_active":"2024-04-26T16:28:25.260911+0000","last_became_peered":"2024-04-26T16:28:25.260911+0000","last_unstale":"2024-04-26T16:31:02.338070+0000","last_undegraded":"2024-04-26T16:31:02.338070+0000","last_fullsized":"2024-04-26T16:31:02.338070+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'50","last_scrub_stamp":"2024-04-26T16:30:02.546137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.546137+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:08.118757+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00057068899999999998,"stat_sum":{"num_bytes":71,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":45,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.14","version":"457'38","reported_seq":992,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308822+0000","last_change":"2024-04-26T16:29:29.288410+0000","last_active":"2024-04-26T16:31:01.308822+0000","last_peered":"2024-04-26T16:31:01.308822+0000","last_clean":"2024-04-26T16:31:01.308822+0000","last_became_active":"2024-04-26T16:23:44.119110+0000","last_became_peered":"2024-04-26T16:23:44.119110+0000","last_unstale":"2024-04-26T16:31:01.308822+0000","last_undegraded":"2024-04-26T16:31:01.308822+0000","last_fullsized":"2024-04-26T16:31:01.308822+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'38","last_scrub_stamp":"2024-04-26T16:28:49.475044+0000","last_deep_scrub":"457'38","last_deep_scrub_stamp":"2024-04-26T16:28:49.475044+0000","last_clean_scrub_stamp":"2024-04-26T16:28:49.475044+0000","objects_scrubbed":3,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0034343469999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.15","version":"488'78","reported_seq":1034,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297342+0000","last_change":"2024-04-26T16:30:12.508796+0000","last_active":"2024-04-26T16:31:00.297342+0000","last_peered":"2024-04-26T16:31:00.297342+0000","last_clean":"2024-04-26T16:31:00.297342+0000","last_became_active":"2024-04-26T16:23:44.233737+0000","last_became_peered":"2024-04-26T16:23:44.233737+0000","last_unstale":"2024-04-26T16:31:00.297342+0000","last_undegraded":"2024-04-26T16:31:00.297342+0000","last_fullsized":"2024-04-26T16:31:00.297342+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'78","last_scrub_stamp":"2024-04-26T16:30:12.508639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.508639+0000","objects_scrubbed":6,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.176078+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0046970850000000001,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1a","version":"322'90","reported_seq":1098,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296699+0000","last_change":"2024-04-26T16:30:25.642345+0000","last_active":"2024-04-26T16:31:00.296699+0000","last_peered":"2024-04-26T16:31:00.296699+0000","last_clean":"2024-04-26T16:31:00.296699+0000","last_became_active":"2024-04-26T16:26:23.171828+0000","last_became_peered":"2024-04-26T16:26:23.171828+0000","last_unstale":"2024-04-26T16:31:00.296699+0000","last_undegraded":"2024-04-26T16:31:00.296699+0000","last_fullsized":"2024-04-26T16:31:00.296699+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'90","last_scrub_stamp":"2024-04-26T16:30:25.642102+0000","last_deep_scrub":"322'90","last_deep_scrub_stamp":"2024-04-26T16:26:30.427683+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.642102+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:51.756368+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0046318089999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1b","version":"458'28","reported_seq":1007,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296650+0000","last_change":"2024-04-26T16:29:29.513165+0000","last_active":"2024-04-26T16:31:00.296650+0000","last_peered":"2024-04-26T16:31:00.296650+0000","last_clean":"2024-04-26T16:31:00.296650+0000","last_became_active":"2024-04-26T16:28:25.251700+0000","last_became_peered":"2024-04-26T16:28:25.251700+0000","last_unstale":"2024-04-26T16:31:00.296650+0000","last_undegraded":"2024-04-26T16:31:00.296650+0000","last_fullsized":"2024-04-26T16:31:00.296650+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'28","last_scrub_stamp":"2024-04-26T16:28:51.005534+0000","last_deep_scrub":"458'28","last_deep_scrub_stamp":"2024-04-26T16:28:51.005534+0000","last_clean_scrub_stamp":"2024-04-26T16:28:51.005534+0000","objects_scrubbed":4,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585399+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048628739999999997,"stat_sum":{"num_bytes":30,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":12,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.18","version":"452'45","reported_seq":1038,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.307865+0000","last_change":"2024-04-26T16:30:20.631404+0000","last_active":"2024-04-26T16:31:01.307865+0000","last_peered":"2024-04-26T16:31:01.307865+0000","last_clean":"2024-04-26T16:31:01.307865+0000","last_became_active":"2024-04-26T16:26:23.188399+0000","last_became_peered":"2024-04-26T16:26:23.188399+0000","last_unstale":"2024-04-26T16:31:01.307865+0000","last_undegraded":"2024-04-26T16:31:01.307865+0000","last_fullsized":"2024-04-26T16:31:01.307865+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'45","last_scrub_stamp":"2024-04-26T16:30:20.631310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:20.631310+0000","objects_scrubbed":6,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:33.855977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0038147540000000001,"stat_sum":{"num_bytes":40,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.19","version":"308'35","reported_seq":998,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309327+0000","last_change":"2024-04-26T16:30:09.622791+0000","last_active":"2024-04-26T16:31:01.309327+0000","last_peered":"2024-04-26T16:31:01.309327+0000","last_clean":"2024-04-26T16:31:01.309327+0000","last_became_active":"2024-04-26T16:23:44.267057+0000","last_became_peered":"2024-04-26T16:23:44.267057+0000","last_unstale":"2024-04-26T16:31:01.309327+0000","last_undegraded":"2024-04-26T16:31:01.309327+0000","last_fullsized":"2024-04-26T16:31:01.309327+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'35","last_scrub_stamp":"2024-04-26T16:30:09.622674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.622674+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.793091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029077199999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.1e","version":"389'20","reported_seq":991,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297854+0000","last_change":"2024-04-26T16:30:33.603424+0000","last_active":"2024-04-26T16:31:00.297854+0000","last_peered":"2024-04-26T16:31:00.297854+0000","last_clean":"2024-04-26T16:31:00.297854+0000","last_became_active":"2024-04-26T16:28:25.230305+0000","last_became_peered":"2024-04-26T16:28:25.230305+0000","last_unstale":"2024-04-26T16:31:00.297854+0000","last_undegraded":"2024-04-26T16:31:00.297854+0000","last_fullsized":"2024-04-26T16:31:00.297854+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-04-26T16:30:33.603284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.603284+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.405921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005995696,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1f","version":"445'20","reported_seq":973,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319030+0000","last_change":"2024-04-26T16:29:29.292388+0000","last_active":"2024-04-26T16:31:00.319030+0000","last_peered":"2024-04-26T16:31:00.319030+0000","last_clean":"2024-04-26T16:31:00.319030+0000","last_became_active":"2024-04-26T16:23:44.180651+0000","last_became_peered":"2024-04-26T16:23:44.180651+0000","last_unstale":"2024-04-26T16:31:00.319030+0000","last_undegraded":"2024-04-26T16:31:00.319030+0000","last_fullsized":"2024-04-26T16:31:00.319030+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'20","last_scrub_stamp":"2024-04-26T16:29:03.044079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.044079+0000","objects_scrubbed":3,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059144+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0037815560000000002,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1c","version":"287'56","reported_seq":992,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.333537+0000","last_change":"2024-04-26T16:30:18.310854+0000","last_active":"2024-04-26T16:31:02.333537+0000","last_peered":"2024-04-26T16:31:02.333537+0000","last_clean":"2024-04-26T16:31:02.333537+0000","last_became_active":"2024-04-26T16:28:25.253825+0000","last_became_peered":"2024-04-26T16:28:25.253825+0000","last_unstale":"2024-04-26T16:31:02.333537+0000","last_undegraded":"2024-04-26T16:31:02.333537+0000","last_fullsized":"2024-04-26T16:31:02.333537+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-04-26T16:30:18.310740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.310740+0000","objects_scrubbed":4,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:30.852946+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053197300000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.1d","version":"448'147","reported_seq":1063,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326298+0000","last_change":"2024-04-26T16:29:29.505126+0000","last_active":"2024-04-26T16:31:00.326298+0000","last_peered":"2024-04-26T16:31:00.326298+0000","last_clean":"2024-04-26T16:31:00.326298+0000","last_became_active":"2024-04-26T16:23:44.112550+0000","last_became_peered":"2024-04-26T16:23:44.112550+0000","last_unstale":"2024-04-26T16:31:00.326298+0000","last_undegraded":"2024-04-26T16:31:00.326298+0000","last_fullsized":"2024-04-26T16:31:00.326298+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'144","last_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_deep_scrub":"401'144","last_deep_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_clean_scrub_stamp":"2024-04-26T16:27:50.033127+0000","objects_scrubbed":8,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001087244,"stat_sum":{"num_bytes":98,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":42,"num_write_kb":41,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.2","version":"503'103","reported_seq":1054,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308911+0000","last_change":"2024-04-26T16:29:29.286447+0000","last_active":"2024-04-26T16:31:01.308911+0000","last_peered":"2024-04-26T16:31:01.308911+0000","last_clean":"2024-04-26T16:31:01.308911+0000","last_became_active":"2024-04-26T16:23:44.121137+0000","last_became_peered":"2024-04-26T16:23:44.121137+0000","last_unstale":"2024-04-26T16:31:01.308911+0000","last_undegraded":"2024-04-26T16:31:01.308911+0000","last_fullsized":"2024-04-26T16:31:01.308911+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"403'91","last_scrub_stamp":"2024-04-26T16:28:50.511931+0000","last_deep_scrub":"318'90","last_deep_scrub_stamp":"2024-04-26T16:26:26.686096+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.511931+0000","objects_scrubbed":6,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0030250239999999999,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.3","version":"259'51","reported_seq":1041,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325865+0000","last_change":"2024-04-26T16:29:29.502257+0000","last_active":"2024-04-26T16:31:00.325865+0000","last_peered":"2024-04-26T16:31:00.325865+0000","last_clean":"2024-04-26T16:31:00.325865+0000","last_became_active":"2024-04-26T16:28:25.273792+0000","last_became_peered":"2024-04-26T16:28:25.273792+0000","last_unstale":"2024-04-26T16:31:00.325865+0000","last_undegraded":"2024-04-26T16:31:00.325865+0000","last_fullsized":"2024-04-26T16:31:00.325865+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'51","last_scrub_stamp":"2024-04-26T16:27:34.948809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.948809+0000","objects_scrubbed":3,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000677325,"stat_sum":{"num_bytes":33,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":15,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.0","version":"389'75","reported_seq":1014,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317220+0000","last_change":"2024-04-26T16:29:56.937825+0000","last_active":"2024-04-26T16:31:00.317220+0000","last_peered":"2024-04-26T16:31:00.317220+0000","last_clean":"2024-04-26T16:31:00.317220+0000","last_became_active":"2024-04-26T16:23:44.226589+0000","last_became_peered":"2024-04-26T16:23:44.226589+0000","last_unstale":"2024-04-26T16:31:00.317220+0000","last_undegraded":"2024-04-26T16:31:00.317220+0000","last_fullsized":"2024-04-26T16:31:00.317220+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'75","last_scrub_stamp":"2024-04-26T16:29:56.937577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.937577+0000","objects_scrubbed":6,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:03.500638+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004787085,"stat_sum":{"num_bytes":60,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":24,"num_write_kb":23,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1","version":"526'106","reported_seq":1052,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326193+0000","last_change":"2024-04-26T16:29:29.501949+0000","last_active":"2024-04-26T16:31:00.326193+0000","last_peered":"2024-04-26T16:31:00.326193+0000","last_clean":"2024-04-26T16:31:00.326193+0000","last_became_active":"2024-04-26T16:28:25.268393+0000","last_became_peered":"2024-04-26T16:28:25.268393+0000","last_unstale":"2024-04-26T16:31:00.326193+0000","last_undegraded":"2024-04-26T16:31:00.326193+0000","last_fullsized":"2024-04-26T16:31:00.326193+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'102","last_scrub_stamp":"2024-04-26T16:27:26.872968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:26.872968+0000","objects_scrubbed":6,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086574099999999995,"stat_sum":{"num_bytes":74,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.6","version":"478'83","reported_seq":1043,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297736+0000","last_change":"2024-04-26T16:30:31.618966+0000","last_active":"2024-04-26T16:31:00.297736+0000","last_peered":"2024-04-26T16:31:00.297736+0000","last_clean":"2024-04-26T16:31:00.297736+0000","last_became_active":"2024-04-26T16:23:44.248476+0000","last_became_peered":"2024-04-26T16:23:44.248476+0000","last_unstale":"2024-04-26T16:31:00.297736+0000","last_undegraded":"2024-04-26T16:31:00.297736+0000","last_fullsized":"2024-04-26T16:31:00.297736+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'83","last_scrub_stamp":"2024-04-26T16:30:31.618721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.618721+0000","objects_scrubbed":8,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:41.987667+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.010158063,"stat_sum":{"num_bytes":4000069,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":27,"num_write_kb":3932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.7","version":"511'77","reported_seq":1031,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326868+0000","last_change":"2024-04-26T16:29:29.505553+0000","last_active":"2024-04-26T16:31:00.326868+0000","last_peered":"2024-04-26T16:31:00.326868+0000","last_clean":"2024-04-26T16:31:00.326868+0000","last_became_active":"2024-04-26T16:23:44.200194+0000","last_became_peered":"2024-04-26T16:23:44.200194+0000","last_unstale":"2024-04-26T16:31:00.326868+0000","last_undegraded":"2024-04-26T16:31:00.326868+0000","last_fullsized":"2024-04-26T16:31:00.326868+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'57","last_scrub_stamp":"2024-04-26T16:27:37.015805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:37.015805+0000","objects_scrubbed":5,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094604400000000005,"stat_sum":{"num_bytes":96,"num_objects":12,"num_object_clones":2,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":28,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.4","version":"389'86","reported_seq":1045,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315784+0000","last_change":"2024-04-26T16:30:10.930053+0000","last_active":"2024-04-26T16:31:01.315784+0000","last_peered":"2024-04-26T16:31:01.315784+0000","last_clean":"2024-04-26T16:31:01.315784+0000","last_became_active":"2024-04-26T16:28:25.254293+0000","last_became_peered":"2024-04-26T16:28:25.254293+0000","last_unstale":"2024-04-26T16:31:01.315784+0000","last_undegraded":"2024-04-26T16:31:01.315784+0000","last_fullsized":"2024-04-26T16:31:01.315784+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:30:10.929911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:10.929911+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.745250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0073012850000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.5","version":"496'52","reported_seq":1009,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309081+0000","last_change":"2024-04-26T16:29:29.287619+0000","last_active":"2024-04-26T16:31:01.309081+0000","last_peered":"2024-04-26T16:31:01.309081+0000","last_clean":"2024-04-26T16:31:01.309081+0000","last_became_active":"2024-04-26T16:23:44.116978+0000","last_became_peered":"2024-04-26T16:23:44.116978+0000","last_unstale":"2024-04-26T16:31:01.309081+0000","last_undegraded":"2024-04-26T16:31:01.309081+0000","last_fullsized":"2024-04-26T16:31:01.309081+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:28:31.422182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:31.422182+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0018006140000000001,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":19,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.a","version":"515'51","reported_seq":1047,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310223+0000","last_change":"2024-04-26T16:30:31.542354+0000","last_active":"2024-04-26T16:31:01.310223+0000","last_peered":"2024-04-26T16:31:01.310223+0000","last_clean":"2024-04-26T16:31:01.310223+0000","last_became_active":"2024-04-26T16:26:27.141171+0000","last_became_peered":"2024-04-26T16:26:27.141171+0000","last_unstale":"2024-04-26T16:31:01.310223+0000","last_undegraded":"2024-04-26T16:31:01.310223+0000","last_fullsized":"2024-04-26T16:31:01.310223+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'44","last_scrub_stamp":"2024-04-26T16:30:31.542113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.542113+0000","objects_scrubbed":5,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.941519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020446129999999998,"stat_sum":{"num_bytes":53,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":20,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.b","version":"236'34","reported_seq":1003,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314520+0000","last_change":"2024-04-26T16:30:00.932705+0000","last_active":"2024-04-26T16:31:01.314520+0000","last_peered":"2024-04-26T16:31:01.314520+0000","last_clean":"2024-04-26T16:31:01.314520+0000","last_became_active":"2024-04-26T16:28:25.233607+0000","last_became_peered":"2024-04-26T16:28:25.233607+0000","last_unstale":"2024-04-26T16:31:01.314520+0000","last_undegraded":"2024-04-26T16:31:01.314520+0000","last_fullsized":"2024-04-26T16:31:01.314520+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:30:00.932368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.932368+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:06.317760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049354960000000002,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.8","version":"389'55","reported_seq":1058,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296224+0000","last_change":"2024-04-26T16:30:11.507964+0000","last_active":"2024-04-26T16:31:00.296224+0000","last_peered":"2024-04-26T16:31:00.296224+0000","last_clean":"2024-04-26T16:31:00.296224+0000","last_became_active":"2024-04-26T16:28:25.245886+0000","last_became_peered":"2024-04-26T16:28:25.245886+0000","last_unstale":"2024-04-26T16:31:00.296224+0000","last_undegraded":"2024-04-26T16:31:00.296224+0000","last_fullsized":"2024-04-26T16:31:00.296224+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'55","last_scrub_stamp":"2024-04-26T16:30:11.507584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.507584+0000","objects_scrubbed":5,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.911256+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001167427,"stat_sum":{"num_bytes":49,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":19,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.9","version":"224'34","reported_seq":985,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317234+0000","last_change":"2024-04-26T16:29:29.284837+0000","last_active":"2024-04-26T16:31:00.317234+0000","last_peered":"2024-04-26T16:31:00.317234+0000","last_clean":"2024-04-26T16:31:00.317234+0000","last_became_active":"2024-04-26T16:28:25.245235+0000","last_became_peered":"2024-04-26T16:28:25.245235+0000","last_unstale":"2024-04-26T16:31:00.317234+0000","last_undegraded":"2024-04-26T16:31:00.317234+0000","last_fullsized":"2024-04-26T16:31:00.317234+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:28:54.109529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.109529+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059245+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0053844380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.e","version":"300'32","reported_seq":1065,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318921+0000","last_change":"2024-04-26T16:30:00.904554+0000","last_active":"2024-04-26T16:31:00.318921+0000","last_peered":"2024-04-26T16:31:00.318921+0000","last_clean":"2024-04-26T16:31:00.318921+0000","last_became_active":"2024-04-26T16:26:23.214391+0000","last_became_peered":"2024-04-26T16:26:23.214391+0000","last_unstale":"2024-04-26T16:31:00.318921+0000","last_undegraded":"2024-04-26T16:31:00.318921+0000","last_fullsized":"2024-04-26T16:31:00.318921+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-04-26T16:30:00.904197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.904197+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.604914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00183567,"stat_sum":{"num_bytes":4194328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":12,"num_write_kb":8202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.f","version":"442'78","reported_seq":1076,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312605+0000","last_change":"2024-04-26T16:29:29.290022+0000","last_active":"2024-04-26T16:31:01.312605+0000","last_peered":"2024-04-26T16:31:01.312605+0000","last_clean":"2024-04-26T16:31:01.312605+0000","last_became_active":"2024-04-26T16:26:23.198449+0000","last_became_peered":"2024-04-26T16:26:23.198449+0000","last_unstale":"2024-04-26T16:31:01.312605+0000","last_undegraded":"2024-04-26T16:31:01.312605+0000","last_fullsized":"2024-04-26T16:31:01.312605+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:28:56.606556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:56.606556+0000","objects_scrubbed":7,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694533+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001552419,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":25,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.c","version":"389'35","reported_seq":991,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315772+0000","last_change":"2024-04-26T16:30:16.958876+0000","last_active":"2024-04-26T16:31:01.315772+0000","last_peered":"2024-04-26T16:31:01.315772+0000","last_clean":"2024-04-26T16:31:01.315772+0000","last_became_active":"2024-04-26T16:23:44.106600+0000","last_became_peered":"2024-04-26T16:23:44.106600+0000","last_unstale":"2024-04-26T16:31:01.315772+0000","last_undegraded":"2024-04-26T16:31:01.315772+0000","last_fullsized":"2024-04-26T16:31:01.315772+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:16.958732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.958732+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.989385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043915680000000002,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.d","version":"464'22","reported_seq":779,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318766+0000","last_change":"2024-04-26T16:29:29.289068+0000","last_active":"2024-04-26T16:31:00.318766+0000","last_peered":"2024-04-26T16:31:00.318766+0000","last_clean":"2024-04-26T16:31:00.318766+0000","last_became_active":"2024-04-26T16:28:25.242455+0000","last_became_peered":"2024-04-26T16:28:25.242455+0000","last_unstale":"2024-04-26T16:31:00.318766+0000","last_undegraded":"2024-04-26T16:31:00.318766+0000","last_fullsized":"2024-04-26T16:31:00.318766+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'22","last_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_deep_scrub":"464'22","last_deep_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.003905+0000","objects_scrubbed":3,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059297+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00091357000000000005,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":446,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.318565+0000","last_change":"2024-04-26T16:28:55.739387+0000","last_active":"2024-04-26T16:31:01.318565+0000","last_peered":"2024-04-26T16:31:01.318565+0000","last_clean":"2024-04-26T16:31:01.318565+0000","last_became_active":"2024-04-26T16:26:23.194196+0000","last_became_peered":"2024-04-26T16:26:23.194196+0000","last_unstale":"2024-04-26T16:31:01.318565+0000","last_undegraded":"2024-04-26T16:31:01.318565+0000","last_fullsized":"2024-04-26T16:31:01.318565+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:55.739153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:55.739153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"433'14","reported_seq":1125,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325499+0000","last_change":"2024-04-26T16:29:10.136335+0000","last_active":"2024-04-26T16:31:00.325499+0000","last_peered":"2024-04-26T16:31:00.325499+0000","last_clean":"2024-04-26T16:31:00.325499+0000","last_became_active":"2024-04-26T16:28:25.267969+0000","last_became_peered":"2024-04-26T16:28:25.267969+0000","last_unstale":"2024-04-26T16:31:00.325499+0000","last_undegraded":"2024-04-26T16:31:00.325499+0000","last_fullsized":"2024-04-26T16:31:00.325499+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.136172+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"458'34","reported_seq":1015,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328603+0000","last_change":"2024-04-26T16:28:50.981113+0000","last_active":"2024-04-26T16:31:00.328603+0000","last_peered":"2024-04-26T16:31:00.328603+0000","last_clean":"2024-04-26T16:31:00.328603+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:31:00.328603+0000","last_undegraded":"2024-04-26T16:31:00.328603+0000","last_fullsized":"2024-04-26T16:31:00.328603+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.980879+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":958,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.298239+0000","last_change":"2024-04-26T16:30:08.452481+0000","last_active":"2024-04-26T16:31:00.298239+0000","last_peered":"2024-04-26T16:31:00.298239+0000","last_clean":"2024-04-26T16:31:00.298239+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:31:00.298239+0000","last_undegraded":"2024-04-26T16:31:00.298239+0000","last_fullsized":"2024-04-26T16:31:00.298239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.452371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.452371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.819274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":343,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.326622+0000","last_change":"2024-04-26T16:29:03.162231+0000","last_active":"2024-04-26T16:31:02.326622+0000","last_peered":"2024-04-26T16:31:02.326622+0000","last_clean":"2024-04-26T16:31:02.326622+0000","last_became_active":"2024-04-26T16:28:25.238286+0000","last_became_peered":"2024-04-26T16:28:25.238286+0000","last_unstale":"2024-04-26T16:31:02.326622+0000","last_undegraded":"2024-04-26T16:31:02.326622+0000","last_fullsized":"2024-04-26T16:31:02.326622+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.162088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.162088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":800,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326482+0000","last_change":"2024-04-26T16:28:25.277790+0000","last_active":"2024-04-26T16:31:00.326482+0000","last_peered":"2024-04-26T16:31:00.326482+0000","last_clean":"2024-04-26T16:31:00.326482+0000","last_became_active":"2024-04-26T16:28:25.277260+0000","last_became_peered":"2024-04-26T16:28:25.277260+0000","last_unstale":"2024-04-26T16:31:00.326482+0000","last_undegraded":"2024-04-26T16:31:00.326482+0000","last_fullsized":"2024-04-26T16:31:00.326482+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:28.909860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.909860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"388'20","reported_seq":507,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318611+0000","last_change":"2024-04-26T16:28:34.168264+0000","last_active":"2024-04-26T16:31:00.318611+0000","last_peered":"2024-04-26T16:31:00.318611+0000","last_clean":"2024-04-26T16:31:00.318611+0000","last_became_active":"2024-04-26T16:28:25.257143+0000","last_became_peered":"2024-04-26T16:28:25.257143+0000","last_unstale":"2024-04-26T16:31:00.318611+0000","last_undegraded":"2024-04-26T16:31:00.318611+0000","last_fullsized":"2024-04-26T16:31:00.318611+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:34.168079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:34.168079+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":961,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.316029+0000","last_change":"2024-04-26T16:30:15.948290+0000","last_active":"2024-04-26T16:31:01.316029+0000","last_peered":"2024-04-26T16:31:01.316029+0000","last_clean":"2024-04-26T16:31:01.316029+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:31:01.316029+0000","last_undegraded":"2024-04-26T16:31:01.316029+0000","last_fullsized":"2024-04-26T16:31:01.316029+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.948187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.948187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:30.824244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":950,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296792+0000","last_change":"2024-04-26T16:29:02.962820+0000","last_active":"2024-04-26T16:31:00.296792+0000","last_peered":"2024-04-26T16:31:00.296792+0000","last_clean":"2024-04-26T16:31:00.296792+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:31:00.296792+0000","last_undegraded":"2024-04-26T16:31:00.296792+0000","last_fullsized":"2024-04-26T16:31:00.296792+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:02.962666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.962666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585464+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":958,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.311805+0000","last_change":"2024-04-26T16:30:26.632966+0000","last_active":"2024-04-26T16:31:01.311805+0000","last_peered":"2024-04-26T16:31:01.311805+0000","last_clean":"2024-04-26T16:31:01.311805+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:31:01.311805+0000","last_undegraded":"2024-04-26T16:31:01.311805+0000","last_fullsized":"2024-04-26T16:31:01.311805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.632648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:33.662364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":806,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328544+0000","last_change":"2024-04-26T16:28:25.270759+0000","last_active":"2024-04-26T16:31:00.328544+0000","last_peered":"2024-04-26T16:31:00.328544+0000","last_clean":"2024-04-26T16:31:00.328544+0000","last_became_active":"2024-04-26T16:28:25.270561+0000","last_became_peered":"2024-04-26T16:28:25.270561+0000","last_unstale":"2024-04-26T16:31:00.328544+0000","last_undegraded":"2024-04-26T16:31:00.328544+0000","last_fullsized":"2024-04-26T16:31:00.328544+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:35.687528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:26:35.687528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"388'31","reported_seq":1026,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328769+0000","last_change":"2024-04-26T16:28:44.986410+0000","last_active":"2024-04-26T16:31:00.328769+0000","last_peered":"2024-04-26T16:31:00.328769+0000","last_clean":"2024-04-26T16:31:00.328769+0000","last_became_active":"2024-04-26T16:28:25.270747+0000","last_became_peered":"2024-04-26T16:28:25.270747+0000","last_unstale":"2024-04-26T16:31:00.328769+0000","last_undegraded":"2024-04-26T16:31:00.328769+0000","last_fullsized":"2024-04-26T16:31:00.328769+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'31","last_scrub_stamp":"2024-04-26T16:28:44.986040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.986040+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":964,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296689+0000","last_change":"2024-04-26T16:28:53.965461+0000","last_active":"2024-04-26T16:31:00.296689+0000","last_peered":"2024-04-26T16:31:00.296689+0000","last_clean":"2024-04-26T16:31:00.296689+0000","last_became_active":"2024-04-26T16:26:23.201706+0000","last_became_peered":"2024-04-26T16:26:23.201706+0000","last_unstale":"2024-04-26T16:31:00.296689+0000","last_undegraded":"2024-04-26T16:31:00.296689+0000","last_fullsized":"2024-04-26T16:31:00.296689+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:53.965330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:53.965330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":956,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297099+0000","last_change":"2024-04-26T16:30:03.454481+0000","last_active":"2024-04-26T16:31:00.297099+0000","last_peered":"2024-04-26T16:31:00.297099+0000","last_clean":"2024-04-26T16:31:00.297099+0000","last_became_active":"2024-04-26T16:26:23.172602+0000","last_became_peered":"2024-04-26T16:26:23.172602+0000","last_unstale":"2024-04-26T16:31:00.297099+0000","last_undegraded":"2024-04-26T16:31:00.297099+0000","last_fullsized":"2024-04-26T16:31:00.297099+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:03.454165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.454165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:11.847181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":928,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.324876+0000","last_change":"2024-04-26T16:26:31.674001+0000","last_active":"2024-04-26T16:31:00.324876+0000","last_peered":"2024-04-26T16:31:00.324876+0000","last_clean":"2024-04-26T16:31:00.324876+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:31:00.324876+0000","last_undegraded":"2024-04-26T16:31:00.324876+0000","last_fullsized":"2024-04-26T16:31:00.324876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:26:31.673774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:26:31.673774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":801,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297068+0000","last_change":"2024-04-26T16:30:06.484249+0000","last_active":"2024-04-26T16:31:00.297068+0000","last_peered":"2024-04-26T16:31:00.297068+0000","last_clean":"2024-04-26T16:31:00.297068+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:31:00.297068+0000","last_undegraded":"2024-04-26T16:31:00.297068+0000","last_fullsized":"2024-04-26T16:31:00.297068+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:06.484028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:06.484028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.768744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"389'35","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.329422+0000","last_change":"2024-04-26T16:30:11.460779+0000","last_active":"2024-04-26T16:31:02.329422+0000","last_peered":"2024-04-26T16:31:02.329422+0000","last_clean":"2024-04-26T16:31:02.329422+0000","last_became_active":"2024-04-26T16:28:25.247148+0000","last_became_peered":"2024-04-26T16:28:25.247148+0000","last_unstale":"2024-04-26T16:31:02.329422+0000","last_undegraded":"2024-04-26T16:31:02.329422+0000","last_fullsized":"2024-04-26T16:31:02.329422+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:11.460648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.460648+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.462457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":954,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326347+0000","last_change":"2024-04-26T16:30:01.188873+0000","last_active":"2024-04-26T16:31:00.326347+0000","last_peered":"2024-04-26T16:31:00.326347+0000","last_clean":"2024-04-26T16:31:00.326347+0000","last_became_active":"2024-04-26T16:26:23.199164+0000","last_became_peered":"2024-04-26T16:26:23.199164+0000","last_unstale":"2024-04-26T16:31:00.326347+0000","last_undegraded":"2024-04-26T16:31:00.326347+0000","last_fullsized":"2024-04-26T16:31:00.326347+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:01.188654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:30:01.188654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.499774+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":961,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.316106+0000","last_change":"2024-04-26T16:30:29.051438+0000","last_active":"2024-04-26T16:31:01.316106+0000","last_peered":"2024-04-26T16:31:01.316106+0000","last_clean":"2024-04-26T16:31:01.316106+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:31:01.316106+0000","last_undegraded":"2024-04-26T16:31:01.316106+0000","last_fullsized":"2024-04-26T16:31:01.316106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.051284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.626968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":945,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297878+0000","last_change":"2024-04-26T16:29:59.407848+0000","last_active":"2024-04-26T16:31:00.297878+0000","last_peered":"2024-04-26T16:31:00.297878+0000","last_clean":"2024-04-26T16:31:00.297878+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:31:00.297878+0000","last_undegraded":"2024-04-26T16:31:00.297878+0000","last_fullsized":"2024-04-26T16:31:00.297878+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:59.407460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:29:59.407460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:20.521427+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":709,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.330786+0000","last_change":"2024-04-26T16:30:00.494762+0000","last_active":"2024-04-26T16:31:02.330786+0000","last_peered":"2024-04-26T16:31:02.330786+0000","last_clean":"2024-04-26T16:31:02.330786+0000","last_became_active":"2024-04-26T16:28:25.265097+0000","last_became_peered":"2024-04-26T16:28:25.265097+0000","last_unstale":"2024-04-26T16:31:02.330786+0000","last_undegraded":"2024-04-26T16:31:02.330786+0000","last_fullsized":"2024-04-26T16:31:02.330786+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.494544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.494544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.559293+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"388'19","reported_seq":974,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318262+0000","last_change":"2024-04-26T16:30:34.982606+0000","last_active":"2024-04-26T16:31:00.318262+0000","last_peered":"2024-04-26T16:31:00.318262+0000","last_clean":"2024-04-26T16:31:00.318262+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:31:00.318262+0000","last_undegraded":"2024-04-26T16:31:00.318262+0000","last_fullsized":"2024-04-26T16:31:00.318262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:34.982478+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.982478+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:59.881026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":961,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314738+0000","last_change":"2024-04-26T16:29:56.955956+0000","last_active":"2024-04-26T16:31:01.314738+0000","last_peered":"2024-04-26T16:31:01.314738+0000","last_clean":"2024-04-26T16:31:01.314738+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:31:01.314738+0000","last_undegraded":"2024-04-26T16:31:01.314738+0000","last_fullsized":"2024-04-26T16:31:01.314738+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:56.955662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.955662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:09.215855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"316'12","reported_seq":1132,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297294+0000","last_change":"2024-04-26T16:29:06.035849+0000","last_active":"2024-04-26T16:31:00.297294+0000","last_peered":"2024-04-26T16:31:00.297294+0000","last_clean":"2024-04-26T16:31:00.297294+0000","last_became_active":"2024-04-26T16:26:23.194072+0000","last_became_peered":"2024-04-26T16:26:23.194072+0000","last_unstale":"2024-04-26T16:31:00.297294+0000","last_undegraded":"2024-04-26T16:31:00.297294+0000","last_fullsized":"2024-04-26T16:31:00.297294+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'12","last_scrub_stamp":"2024-04-26T16:29:06.035584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:06.035584+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":948,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312074+0000","last_change":"2024-04-26T16:28:59.599961+0000","last_active":"2024-04-26T16:31:01.312074+0000","last_peered":"2024-04-26T16:31:01.312074+0000","last_clean":"2024-04-26T16:31:01.312074+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:31:01.312074+0000","last_undegraded":"2024-04-26T16:31:01.312074+0000","last_fullsized":"2024-04-26T16:31:01.312074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.599808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.599808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694576+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":802,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310668+0000","last_change":"2024-04-26T16:29:01.807303+0000","last_active":"2024-04-26T16:31:01.310668+0000","last_peered":"2024-04-26T16:31:01.310668+0000","last_clean":"2024-04-26T16:31:01.310668+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:31:01.310668+0000","last_undegraded":"2024-04-26T16:31:01.310668+0000","last_fullsized":"2024-04-26T16:31:01.310668+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:01.807199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:01.807199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"388'27","reported_seq":988,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317550+0000","last_change":"2024-04-26T16:30:21.978132+0000","last_active":"2024-04-26T16:31:00.317550+0000","last_peered":"2024-04-26T16:31:00.317550+0000","last_clean":"2024-04-26T16:31:00.317550+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:31:00.317550+0000","last_undegraded":"2024-04-26T16:31:00.317550+0000","last_fullsized":"2024-04-26T16:31:00.317550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:21.978007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.978007+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.780831+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":975,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.313780+0000","last_change":"2024-04-26T16:30:35.051954+0000","last_active":"2024-04-26T16:31:01.313780+0000","last_peered":"2024-04-26T16:31:01.313780+0000","last_clean":"2024-04-26T16:31:01.313780+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:31:01.313780+0000","last_undegraded":"2024-04-26T16:31:01.313780+0000","last_fullsized":"2024-04-26T16:31:01.313780+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.051759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:47.746481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"316'13","reported_seq":1136,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.311823+0000","last_change":"2024-04-26T16:30:14.531293+0000","last_active":"2024-04-26T16:31:01.311823+0000","last_peered":"2024-04-26T16:31:01.311823+0000","last_clean":"2024-04-26T16:31:01.311823+0000","last_became_active":"2024-04-26T16:26:23.180811+0000","last_became_peered":"2024-04-26T16:26:23.180811+0000","last_unstale":"2024-04-26T16:31:01.311823+0000","last_undegraded":"2024-04-26T16:31:01.311823+0000","last_fullsized":"2024-04-26T16:31:01.311823+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:14.531205+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.531205+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:25.731439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":455,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315805+0000","last_change":"2024-04-26T16:30:25.049754+0000","last_active":"2024-04-26T16:31:01.315805+0000","last_peered":"2024-04-26T16:31:01.315805+0000","last_clean":"2024-04-26T16:31:01.315805+0000","last_became_active":"2024-04-26T16:26:23.195438+0000","last_became_peered":"2024-04-26T16:26:23.195438+0000","last_unstale":"2024-04-26T16:31:01.315805+0000","last_undegraded":"2024-04-26T16:31:01.315805+0000","last_fullsized":"2024-04-26T16:31:01.315805+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.049647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.049647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.587456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":726,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.326213+0000","last_change":"2024-04-26T16:30:07.526553+0000","last_active":"2024-04-26T16:31:02.326213+0000","last_peered":"2024-04-26T16:31:02.326213+0000","last_clean":"2024-04-26T16:31:02.326213+0000","last_became_active":"2024-04-26T16:28:25.244320+0000","last_became_peered":"2024-04-26T16:28:25.244320+0000","last_unstale":"2024-04-26T16:31:02.326213+0000","last_undegraded":"2024-04-26T16:31:02.326213+0000","last_fullsized":"2024-04-26T16:31:02.326213+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:30:07.526320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.526320+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:20.026339+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"399'20","reported_seq":991,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.311104+0000","last_change":"2024-04-26T16:30:21.500286+0000","last_active":"2024-04-26T16:31:01.311104+0000","last_peered":"2024-04-26T16:31:01.311104+0000","last_clean":"2024-04-26T16:31:01.311104+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:31:01.311104+0000","last_undegraded":"2024-04-26T16:31:01.311104+0000","last_fullsized":"2024-04-26T16:31:01.311104+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'20","last_scrub_stamp":"2024-04-26T16:30:21.500166+0000","last_deep_scrub":"399'20","last_deep_scrub_stamp":"2024-04-26T16:27:47.205374+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.500166+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.067680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":956,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325055+0000","last_change":"2024-04-26T16:28:46.987810+0000","last_active":"2024-04-26T16:31:00.325055+0000","last_peered":"2024-04-26T16:31:00.325055+0000","last_clean":"2024-04-26T16:31:00.325055+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:31:00.325055+0000","last_undegraded":"2024-04-26T16:31:00.325055+0000","last_fullsized":"2024-04-26T16:31:00.325055+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:28:46.987686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.987686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":463,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312968+0000","last_change":"2024-04-26T16:30:23.518613+0000","last_active":"2024-04-26T16:31:01.312968+0000","last_peered":"2024-04-26T16:31:01.312968+0000","last_clean":"2024-04-26T16:31:01.312968+0000","last_became_active":"2024-04-26T16:28:25.248459+0000","last_became_peered":"2024-04-26T16:28:25.248459+0000","last_unstale":"2024-04-26T16:31:01.312968+0000","last_undegraded":"2024-04-26T16:31:01.312968+0000","last_fullsized":"2024-04-26T16:31:01.312968+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.518326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.518326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.755565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":952,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318809+0000","last_change":"2024-04-26T16:30:35.938334+0000","last_active":"2024-04-26T16:31:00.318809+0000","last_peered":"2024-04-26T16:31:00.318809+0000","last_clean":"2024-04-26T16:31:00.318809+0000","last_became_active":"2024-04-26T16:28:25.257629+0000","last_became_peered":"2024-04-26T16:28:25.257629+0000","last_unstale":"2024-04-26T16:31:00.318809+0000","last_undegraded":"2024-04-26T16:31:00.318809+0000","last_fullsized":"2024-04-26T16:31:00.318809+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.938129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.938129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.462202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":802,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.311862+0000","last_change":"2024-04-26T16:28:55.586673+0000","last_active":"2024-04-26T16:31:01.311862+0000","last_peered":"2024-04-26T16:31:01.311862+0000","last_clean":"2024-04-26T16:31:01.311862+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:31:01.311862+0000","last_undegraded":"2024-04-26T16:31:01.311862+0000","last_fullsized":"2024-04-26T16:31:01.311862+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:55.586453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:55.586453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1107,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317766+0000","last_change":"2024-04-26T16:28:37.207675+0000","last_active":"2024-04-26T16:31:00.317766+0000","last_peered":"2024-04-26T16:31:00.317766+0000","last_clean":"2024-04-26T16:31:00.317766+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:31:00.317766+0000","last_undegraded":"2024-04-26T16:31:00.317766+0000","last_fullsized":"2024-04-26T16:31:00.317766+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:28:37.207601+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:28:37.207601+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":977,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.311095+0000","last_change":"2024-04-26T16:28:44.503061+0000","last_active":"2024-04-26T16:31:01.311095+0000","last_peered":"2024-04-26T16:31:01.311095+0000","last_clean":"2024-04-26T16:31:01.311095+0000","last_became_active":"2024-04-26T16:28:25.250134+0000","last_became_peered":"2024-04-26T16:28:25.250134+0000","last_unstale":"2024-04-26T16:31:01.311095+0000","last_undegraded":"2024-04-26T16:31:01.311095+0000","last_fullsized":"2024-04-26T16:31:01.311095+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:44.502708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:13.552494+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.502708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'30","reported_seq":1000,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312593+0000","last_change":"2024-04-26T16:28:50.453707+0000","last_active":"2024-04-26T16:31:01.312593+0000","last_peered":"2024-04-26T16:31:01.312593+0000","last_clean":"2024-04-26T16:31:01.312593+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:31:01.312593+0000","last_undegraded":"2024-04-26T16:31:01.312593+0000","last_fullsized":"2024-04-26T16:31:01.312593+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'30","last_scrub_stamp":"2024-04-26T16:28:50.453456+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.453456+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":860,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.336862+0000","last_change":"2024-04-26T16:30:33.268356+0000","last_active":"2024-04-26T16:31:02.336862+0000","last_peered":"2024-04-26T16:31:02.336862+0000","last_clean":"2024-04-26T16:31:02.336862+0000","last_became_active":"2024-04-26T16:28:25.265921+0000","last_became_peered":"2024-04-26T16:28:25.265921+0000","last_unstale":"2024-04-26T16:31:02.336862+0000","last_undegraded":"2024-04-26T16:31:02.336862+0000","last_fullsized":"2024-04-26T16:31:02.336862+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.268035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.268035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.106859+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":945,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326070+0000","last_change":"2024-04-26T16:28:38.012269+0000","last_active":"2024-04-26T16:31:00.326070+0000","last_peered":"2024-04-26T16:31:00.326070+0000","last_clean":"2024-04-26T16:31:00.326070+0000","last_became_active":"2024-04-26T16:28:25.270748+0000","last_became_peered":"2024-04-26T16:28:25.270748+0000","last_unstale":"2024-04-26T16:31:00.326070+0000","last_undegraded":"2024-04-26T16:31:00.326070+0000","last_fullsized":"2024-04-26T16:31:00.326070+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:38.011966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:38.011966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":799,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328887+0000","last_change":"2024-04-26T16:30:08.292894+0000","last_active":"2024-04-26T16:31:00.328887+0000","last_peered":"2024-04-26T16:31:00.328887+0000","last_clean":"2024-04-26T16:31:00.328887+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:31:00.328887+0000","last_undegraded":"2024-04-26T16:31:00.328887+0000","last_fullsized":"2024-04-26T16:31:00.328887+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.292589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.292589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:22.177871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":953,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310962+0000","last_change":"2024-04-26T16:29:09.485874+0000","last_active":"2024-04-26T16:31:01.310962+0000","last_peered":"2024-04-26T16:31:01.310962+0000","last_clean":"2024-04-26T16:31:01.310962+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:31:01.310962+0000","last_undegraded":"2024-04-26T16:31:01.310962+0000","last_fullsized":"2024-04-26T16:31:01.310962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.485662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.485662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694859+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":1743,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317651+0000","last_change":"2024-04-26T16:30:24.906077+0000","last_active":"2024-04-26T16:31:00.317651+0000","last_peered":"2024-04-26T16:31:00.317651+0000","last_clean":"2024-04-26T16:31:00.317651+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:31:00.317651+0000","last_undegraded":"2024-04-26T16:31:00.317651+0000","last_fullsized":"2024-04-26T16:31:00.317651+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:30:24.905850+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.905850+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:29.141938+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":790,"num_read_kb":785,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'22","reported_seq":983,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319206+0000","last_change":"2024-04-26T16:30:19.960354+0000","last_active":"2024-04-26T16:31:00.319206+0000","last_peered":"2024-04-26T16:31:00.319206+0000","last_clean":"2024-04-26T16:31:00.319206+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:31:00.319206+0000","last_undegraded":"2024-04-26T16:31:00.319206+0000","last_fullsized":"2024-04-26T16:31:00.319206+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:30:19.960123+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.960123+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:22.536698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":972,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.317901+0000","last_change":"2024-04-26T16:29:08.743269+0000","last_active":"2024-04-26T16:31:01.317901+0000","last_peered":"2024-04-26T16:31:01.317901+0000","last_clean":"2024-04-26T16:31:01.317901+0000","last_became_active":"2024-04-26T16:28:25.233774+0000","last_became_peered":"2024-04-26T16:28:25.233774+0000","last_unstale":"2024-04-26T16:31:01.317901+0000","last_undegraded":"2024-04-26T16:31:01.317901+0000","last_fullsized":"2024-04-26T16:31:01.317901+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:29:08.742955+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.742955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":949,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309871+0000","last_change":"2024-04-26T16:28:40.455517+0000","last_active":"2024-04-26T16:31:01.309871+0000","last_peered":"2024-04-26T16:31:01.309871+0000","last_clean":"2024-04-26T16:31:01.309871+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:31:01.309871+0000","last_undegraded":"2024-04-26T16:31:01.309871+0000","last_fullsized":"2024-04-26T16:31:01.309871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.455422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.455422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":802,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315865+0000","last_change":"2024-04-26T16:30:12.954476+0000","last_active":"2024-04-26T16:31:01.315865+0000","last_peered":"2024-04-26T16:31:01.315865+0000","last_clean":"2024-04-26T16:31:01.315865+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:31:01.315865+0000","last_undegraded":"2024-04-26T16:31:01.315865+0000","last_fullsized":"2024-04-26T16:31:01.315865+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.954265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.954265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:23.703295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":852,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.326104+0000","last_change":"2024-04-26T16:30:25.328753+0000","last_active":"2024-04-26T16:31:02.326104+0000","last_peered":"2024-04-26T16:31:02.326104+0000","last_clean":"2024-04-26T16:31:02.326104+0000","last_became_active":"2024-04-26T16:28:25.234888+0000","last_became_peered":"2024-04-26T16:28:25.234888+0000","last_unstale":"2024-04-26T16:31:02.326104+0000","last_undegraded":"2024-04-26T16:31:02.326104+0000","last_fullsized":"2024-04-26T16:31:02.326104+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.328546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.328546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:53.320476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"388'53","reported_seq":1226,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315885+0000","last_change":"2024-04-26T16:29:10.751587+0000","last_active":"2024-04-26T16:31:01.315885+0000","last_peered":"2024-04-26T16:31:01.315885+0000","last_clean":"2024-04-26T16:31:01.315885+0000","last_became_active":"2024-04-26T16:26:23.203408+0000","last_became_peered":"2024-04-26T16:26:23.203408+0000","last_unstale":"2024-04-26T16:31:01.315885+0000","last_undegraded":"2024-04-26T16:31:01.315885+0000","last_fullsized":"2024-04-26T16:31:01.315885+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'53","last_scrub_stamp":"2024-04-26T16:29:10.751474+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.751474+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":957,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328561+0000","last_change":"2024-04-26T16:29:03.080654+0000","last_active":"2024-04-26T16:31:00.328561+0000","last_peered":"2024-04-26T16:31:00.328561+0000","last_clean":"2024-04-26T16:31:00.328561+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:31:00.328561+0000","last_undegraded":"2024-04-26T16:31:00.328561+0000","last_fullsized":"2024-04-26T16:31:00.328561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.080403+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":966,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.313882+0000","last_change":"2024-04-26T16:30:28.037331+0000","last_active":"2024-04-26T16:31:01.313882+0000","last_peered":"2024-04-26T16:31:01.313882+0000","last_clean":"2024-04-26T16:31:01.313882+0000","last_became_active":"2024-04-26T16:28:25.242325+0000","last_became_peered":"2024-04-26T16:28:25.242325+0000","last_unstale":"2024-04-26T16:31:01.313882+0000","last_undegraded":"2024-04-26T16:31:01.313882+0000","last_fullsized":"2024-04-26T16:31:01.313882+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.037115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.037115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:56.639017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":801,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317754+0000","last_change":"2024-04-26T16:29:58.933284+0000","last_active":"2024-04-26T16:31:00.317754+0000","last_peered":"2024-04-26T16:31:00.317754+0000","last_clean":"2024-04-26T16:31:00.317754+0000","last_became_active":"2024-04-26T16:28:25.245871+0000","last_became_peered":"2024-04-26T16:28:25.245871+0000","last_unstale":"2024-04-26T16:31:00.317754+0000","last_undegraded":"2024-04-26T16:31:00.317754+0000","last_fullsized":"2024-04-26T16:31:00.317754+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.933163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.933163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:22.949904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":454,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.318410+0000","last_change":"2024-04-26T16:30:24.025255+0000","last_active":"2024-04-26T16:31:01.318410+0000","last_peered":"2024-04-26T16:31:01.318410+0000","last_clean":"2024-04-26T16:31:01.318410+0000","last_became_active":"2024-04-26T16:26:23.191179+0000","last_became_peered":"2024-04-26T16:26:23.191179+0000","last_unstale":"2024-04-26T16:31:01.318410+0000","last_undegraded":"2024-04-26T16:31:01.318410+0000","last_fullsized":"2024-04-26T16:31:01.318410+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.025163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.025163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:27.212243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":948,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.324908+0000","last_change":"2024-04-26T16:28:59.045623+0000","last_active":"2024-04-26T16:31:00.324908+0000","last_peered":"2024-04-26T16:31:00.324908+0000","last_clean":"2024-04-26T16:31:00.324908+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:31:00.324908+0000","last_undegraded":"2024-04-26T16:31:00.324908+0000","last_fullsized":"2024-04-26T16:31:00.324908+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.045415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.045415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'35","reported_seq":1007,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310982+0000","last_change":"2024-04-26T16:29:02.435596+0000","last_active":"2024-04-26T16:31:01.310982+0000","last_peered":"2024-04-26T16:31:01.310982+0000","last_clean":"2024-04-26T16:31:01.310982+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:31:01.310982+0000","last_undegraded":"2024-04-26T16:31:01.310982+0000","last_fullsized":"2024-04-26T16:31:01.310982+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:29:02.435425+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.435425+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":956,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.316094+0000","last_change":"2024-04-26T16:28:32.981894+0000","last_active":"2024-04-26T16:31:01.316094+0000","last_peered":"2024-04-26T16:31:01.316094+0000","last_clean":"2024-04-26T16:31:01.316094+0000","last_became_active":"2024-04-26T16:28:25.253094+0000","last_became_peered":"2024-04-26T16:28:25.253094+0000","last_unstale":"2024-04-26T16:31:01.316094+0000","last_undegraded":"2024-04-26T16:31:01.316094+0000","last_fullsized":"2024-04-26T16:31:01.316094+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:32.981795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:28:32.981795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":807,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317869+0000","last_change":"2024-04-26T16:28:58.073423+0000","last_active":"2024-04-26T16:31:00.317869+0000","last_peered":"2024-04-26T16:31:00.317869+0000","last_clean":"2024-04-26T16:31:00.317869+0000","last_became_active":"2024-04-26T16:28:25.231951+0000","last_became_peered":"2024-04-26T16:28:25.231951+0000","last_unstale":"2024-04-26T16:31:00.317869+0000","last_undegraded":"2024-04-26T16:31:00.317869+0000","last_fullsized":"2024-04-26T16:31:00.317869+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:58.073194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:58.073194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":946,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328618+0000","last_change":"2024-04-26T16:30:05.254074+0000","last_active":"2024-04-26T16:31:00.328618+0000","last_peered":"2024-04-26T16:31:00.328618+0000","last_clean":"2024-04-26T16:31:00.328618+0000","last_became_active":"2024-04-26T16:26:23.176892+0000","last_became_peered":"2024-04-26T16:26:23.176892+0000","last_unstale":"2024-04-26T16:31:00.328618+0000","last_undegraded":"2024-04-26T16:31:00.328618+0000","last_fullsized":"2024-04-26T16:31:00.328618+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:05.253814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:30:05.253814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:12.490234+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1237,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.324958+0000","last_change":"2024-04-26T16:28:25.267719+0000","last_active":"2024-04-26T16:31:00.324958+0000","last_peered":"2024-04-26T16:31:00.324958+0000","last_clean":"2024-04-26T16:31:00.324958+0000","last_became_active":"2024-04-26T16:28:25.267305+0000","last_became_peered":"2024-04-26T16:28:25.267305+0000","last_unstale":"2024-04-26T16:31:00.324958+0000","last_undegraded":"2024-04-26T16:31:00.324958+0000","last_fullsized":"2024-04-26T16:31:00.324958+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:26:29.671147+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:26:29.671147+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":446,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296908+0000","last_change":"2024-04-26T16:30:09.475518+0000","last_active":"2024-04-26T16:31:00.296908+0000","last_peered":"2024-04-26T16:31:00.296908+0000","last_clean":"2024-04-26T16:31:00.296908+0000","last_became_active":"2024-04-26T16:26:23.201156+0000","last_became_peered":"2024-04-26T16:26:23.201156+0000","last_unstale":"2024-04-26T16:31:00.296908+0000","last_undegraded":"2024-04-26T16:31:00.296908+0000","last_fullsized":"2024-04-26T16:31:00.296908+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.475272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.475272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:21.943110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"388'25","reported_seq":1007,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317880+0000","last_change":"2024-04-26T16:28:52.098358+0000","last_active":"2024-04-26T16:31:00.317880+0000","last_peered":"2024-04-26T16:31:00.317880+0000","last_clean":"2024-04-26T16:31:00.317880+0000","last_became_active":"2024-04-26T16:28:25.258417+0000","last_became_peered":"2024-04-26T16:28:25.258417+0000","last_unstale":"2024-04-26T16:31:00.317880+0000","last_undegraded":"2024-04-26T16:31:00.317880+0000","last_fullsized":"2024-04-26T16:31:00.317880+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:28:52.098251+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.098251+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":446,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319335+0000","last_change":"2024-04-26T16:30:14.930711+0000","last_active":"2024-04-26T16:31:00.319335+0000","last_peered":"2024-04-26T16:31:00.319335+0000","last_clean":"2024-04-26T16:31:00.319335+0000","last_became_active":"2024-04-26T16:26:23.212670+0000","last_became_peered":"2024-04-26T16:26:23.212670+0000","last_unstale":"2024-04-26T16:31:00.319335+0000","last_undegraded":"2024-04-26T16:31:00.319335+0000","last_fullsized":"2024-04-26T16:31:00.319335+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.930578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.930578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:28.579424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":807,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317313+0000","last_change":"2024-04-26T16:28:43.126227+0000","last_active":"2024-04-26T16:31:00.317313+0000","last_peered":"2024-04-26T16:31:00.317313+0000","last_clean":"2024-04-26T16:31:00.317313+0000","last_became_active":"2024-04-26T16:28:25.245460+0000","last_became_peered":"2024-04-26T16:28:25.245460+0000","last_unstale":"2024-04-26T16:31:00.317313+0000","last_undegraded":"2024-04-26T16:31:00.317313+0000","last_fullsized":"2024-04-26T16:31:00.317313+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:43.125904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.125904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":956,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.298394+0000","last_change":"2024-04-26T16:29:05.010298+0000","last_active":"2024-04-26T16:31:00.298394+0000","last_peered":"2024-04-26T16:31:00.298394+0000","last_clean":"2024-04-26T16:31:00.298394+0000","last_became_active":"2024-04-26T16:26:23.185332+0000","last_became_peered":"2024-04-26T16:26:23.185332+0000","last_unstale":"2024-04-26T16:31:00.298394+0000","last_undegraded":"2024-04-26T16:31:00.298394+0000","last_fullsized":"2024-04-26T16:31:00.298394+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:05.010052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:29:05.010052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":958,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296445+0000","last_change":"2024-04-26T16:29:03.999225+0000","last_active":"2024-04-26T16:31:00.296445+0000","last_peered":"2024-04-26T16:31:00.296445+0000","last_clean":"2024-04-26T16:31:00.296445+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:31:00.296445+0000","last_undegraded":"2024-04-26T16:31:00.296445+0000","last_fullsized":"2024-04-26T16:31:00.296445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.998965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.998965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'38","reported_seq":1032,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310583+0000","last_change":"2024-04-26T16:30:27.085082+0000","last_active":"2024-04-26T16:31:01.310583+0000","last_peered":"2024-04-26T16:31:01.310583+0000","last_clean":"2024-04-26T16:31:01.310583+0000","last_became_active":"2024-04-26T16:28:25.241767+0000","last_became_peered":"2024-04-26T16:28:25.241767+0000","last_unstale":"2024-04-26T16:31:01.310583+0000","last_undegraded":"2024-04-26T16:31:01.310583+0000","last_fullsized":"2024-04-26T16:31:01.310583+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-04-26T16:30:27.084844+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:27:38.188279+0000","last_clean_scrub_stamp":"2024-04-26T16:30:27.084844+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:41.150355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":945,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326576+0000","last_change":"2024-04-26T16:28:52.021759+0000","last_active":"2024-04-26T16:31:00.326576+0000","last_peered":"2024-04-26T16:31:00.326576+0000","last_clean":"2024-04-26T16:31:00.326576+0000","last_became_active":"2024-04-26T16:28:25.266762+0000","last_became_peered":"2024-04-26T16:28:25.266762+0000","last_unstale":"2024-04-26T16:31:00.326576+0000","last_undegraded":"2024-04-26T16:31:00.326576+0000","last_fullsized":"2024-04-26T16:31:00.326576+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.021548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.021548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":792,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318117+0000","last_change":"2024-04-26T16:29:10.997703+0000","last_active":"2024-04-26T16:31:00.318117+0000","last_peered":"2024-04-26T16:31:00.318117+0000","last_clean":"2024-04-26T16:31:00.318117+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:31:00.318117+0000","last_undegraded":"2024-04-26T16:31:00.318117+0000","last_fullsized":"2024-04-26T16:31:00.318117+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.997614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.997614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":954,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312240+0000","last_change":"2024-04-26T16:29:03.398269+0000","last_active":"2024-04-26T16:31:01.312240+0000","last_peered":"2024-04-26T16:31:01.312240+0000","last_clean":"2024-04-26T16:31:01.312240+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:31:01.312240+0000","last_undegraded":"2024-04-26T16:31:01.312240+0000","last_fullsized":"2024-04-26T16:31:01.312240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.398123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.398123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'37","reported_seq":1025,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309607+0000","last_change":"2024-04-26T16:28:54.567147+0000","last_active":"2024-04-26T16:31:01.309607+0000","last_peered":"2024-04-26T16:31:01.309607+0000","last_clean":"2024-04-26T16:31:01.309607+0000","last_became_active":"2024-04-26T16:28:25.235694+0000","last_became_peered":"2024-04-26T16:28:25.235694+0000","last_unstale":"2024-04-26T16:31:01.309607+0000","last_undegraded":"2024-04-26T16:31:01.309607+0000","last_fullsized":"2024-04-26T16:31:01.309607+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-04-26T16:28:54.566895+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:28:54.566895+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.566895+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694432+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":455,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314684+0000","last_change":"2024-04-26T16:28:42.796996+0000","last_active":"2024-04-26T16:31:01.314684+0000","last_peered":"2024-04-26T16:31:01.314684+0000","last_clean":"2024-04-26T16:31:01.314684+0000","last_became_active":"2024-04-26T16:28:25.233622+0000","last_became_peered":"2024-04-26T16:28:25.233622+0000","last_unstale":"2024-04-26T16:31:01.314684+0000","last_undegraded":"2024-04-26T16:31:01.314684+0000","last_fullsized":"2024-04-26T16:31:01.314684+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:42.796778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:28:42.796778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":937,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319287+0000","last_change":"2024-04-26T16:28:39.169317+0000","last_active":"2024-04-26T16:31:00.319287+0000","last_peered":"2024-04-26T16:31:00.319287+0000","last_clean":"2024-04-26T16:31:00.319287+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:31:00.319287+0000","last_undegraded":"2024-04-26T16:31:00.319287+0000","last_fullsized":"2024-04-26T16:31:00.319287+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:39.169120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:39.169120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058685+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":810,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297437+0000","last_change":"2024-04-26T16:30:19.533168+0000","last_active":"2024-04-26T16:31:00.297437+0000","last_peered":"2024-04-26T16:31:00.297437+0000","last_clean":"2024-04-26T16:31:00.297437+0000","last_became_active":"2024-04-26T16:28:25.241015+0000","last_became_peered":"2024-04-26T16:28:25.241015+0000","last_unstale":"2024-04-26T16:31:00.297437+0000","last_undegraded":"2024-04-26T16:31:00.297437+0000","last_fullsized":"2024-04-26T16:31:00.297437+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.533042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.533042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.752688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"433'16","reported_seq":1153,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309317+0000","last_change":"2024-04-26T16:28:49.448655+0000","last_active":"2024-04-26T16:31:01.309317+0000","last_peered":"2024-04-26T16:31:01.309317+0000","last_clean":"2024-04-26T16:31:01.309317+0000","last_became_active":"2024-04-26T16:28:25.228833+0000","last_became_peered":"2024-04-26T16:28:25.228833+0000","last_unstale":"2024-04-26T16:31:01.309317+0000","last_undegraded":"2024-04-26T16:31:01.309317+0000","last_fullsized":"2024-04-26T16:31:01.309317+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'16","last_scrub_stamp":"2024-04-26T16:28:49.448423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:49.448423+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":974,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314910+0000","last_change":"2024-04-26T16:30:02.951737+0000","last_active":"2024-04-26T16:31:01.314910+0000","last_peered":"2024-04-26T16:31:01.314910+0000","last_clean":"2024-04-26T16:31:01.314910+0000","last_became_active":"2024-04-26T16:26:23.187980+0000","last_became_peered":"2024-04-26T16:26:23.187980+0000","last_unstale":"2024-04-26T16:31:01.314910+0000","last_undegraded":"2024-04-26T16:31:01.314910+0000","last_fullsized":"2024-04-26T16:31:01.314910+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:30:02.951627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.951627+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:12.328056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"433'39","reported_seq":599,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318927+0000","last_change":"2024-04-26T16:30:08.947640+0000","last_active":"2024-04-26T16:31:00.318927+0000","last_peered":"2024-04-26T16:31:00.318927+0000","last_clean":"2024-04-26T16:31:00.318927+0000","last_became_active":"2024-04-26T16:28:25.228181+0000","last_became_peered":"2024-04-26T16:28:25.228181+0000","last_unstale":"2024-04-26T16:31:00.318927+0000","last_undegraded":"2024-04-26T16:31:00.318927+0000","last_fullsized":"2024-04-26T16:31:00.318927+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'39","last_scrub_stamp":"2024-04-26T16:30:08.947465+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.947465+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.991257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":961,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315945+0000","last_change":"2024-04-26T16:28:47.879030+0000","last_active":"2024-04-26T16:31:01.315945+0000","last_peered":"2024-04-26T16:31:01.315945+0000","last_clean":"2024-04-26T16:31:01.315945+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:31:01.315945+0000","last_undegraded":"2024-04-26T16:31:01.315945+0000","last_fullsized":"2024-04-26T16:31:01.315945+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:47.878929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.878929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":950,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308297+0000","last_change":"2024-04-26T16:30:31.628236+0000","last_active":"2024-04-26T16:31:01.308297+0000","last_peered":"2024-04-26T16:31:01.308297+0000","last_clean":"2024-04-26T16:31:01.308297+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:31:01.308297+0000","last_undegraded":"2024-04-26T16:31:01.308297+0000","last_fullsized":"2024-04-26T16:31:01.308297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:31.627931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.627931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.885855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":805,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312477+0000","last_change":"2024-04-26T16:29:55.759332+0000","last_active":"2024-04-26T16:31:01.312477+0000","last_peered":"2024-04-26T16:31:01.312477+0000","last_clean":"2024-04-26T16:31:01.312477+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:31:01.312477+0000","last_undegraded":"2024-04-26T16:31:01.312477+0000","last_fullsized":"2024-04-26T16:31:01.312477+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:55.759069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:55.759069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:08.476199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":950,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296640+0000","last_change":"2024-04-26T16:29:01.923625+0000","last_active":"2024-04-26T16:31:00.296640+0000","last_peered":"2024-04-26T16:31:00.296640+0000","last_clean":"2024-04-26T16:31:00.296640+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:31:00.296640+0000","last_undegraded":"2024-04-26T16:31:00.296640+0000","last_fullsized":"2024-04-26T16:31:00.296640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:29:01.923293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585379+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"388'19","reported_seq":982,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326706+0000","last_change":"2024-04-26T16:30:03.191223+0000","last_active":"2024-04-26T16:31:00.326706+0000","last_peered":"2024-04-26T16:31:00.326706+0000","last_clean":"2024-04-26T16:31:00.326706+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:31:00.326706+0000","last_undegraded":"2024-04-26T16:31:00.326706+0000","last_fullsized":"2024-04-26T16:31:00.326706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:03.190975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.190975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.424209+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":985,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312765+0000","last_change":"2024-04-26T16:30:04.581495+0000","last_active":"2024-04-26T16:31:01.312765+0000","last_peered":"2024-04-26T16:31:01.312765+0000","last_clean":"2024-04-26T16:31:01.312765+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:31:01.312765+0000","last_undegraded":"2024-04-26T16:31:01.312765+0000","last_fullsized":"2024-04-26T16:31:01.312765+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:30:04.581361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.581361+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.126549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":929,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318361+0000","last_change":"2024-04-26T16:28:40.179575+0000","last_active":"2024-04-26T16:31:00.318361+0000","last_peered":"2024-04-26T16:31:00.318361+0000","last_clean":"2024-04-26T16:31:00.318361+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:31:00.318361+0000","last_undegraded":"2024-04-26T16:31:00.318361+0000","last_fullsized":"2024-04-26T16:31:00.318361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.179446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.179446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":805,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309402+0000","last_change":"2024-04-26T16:30:03.519972+0000","last_active":"2024-04-26T16:31:01.309402+0000","last_peered":"2024-04-26T16:31:01.309402+0000","last_clean":"2024-04-26T16:31:01.309402+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:31:01.309402+0000","last_undegraded":"2024-04-26T16:31:01.309402+0000","last_fullsized":"2024-04-26T16:31:01.309402+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:03.519624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.519624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:17.009661+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"388'19","reported_seq":983,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325097+0000","last_change":"2024-04-26T16:29:00.045697+0000","last_active":"2024-04-26T16:31:00.325097+0000","last_peered":"2024-04-26T16:31:00.325097+0000","last_clean":"2024-04-26T16:31:00.325097+0000","last_became_active":"2024-04-26T16:28:25.270038+0000","last_became_peered":"2024-04-26T16:28:25.270038+0000","last_unstale":"2024-04-26T16:31:00.325097+0000","last_undegraded":"2024-04-26T16:31:00.325097+0000","last_fullsized":"2024-04-26T16:31:00.325097+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:00.045450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.045450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":957,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296529+0000","last_change":"2024-04-26T16:29:07.075600+0000","last_active":"2024-04-26T16:31:00.296529+0000","last_peered":"2024-04-26T16:31:00.296529+0000","last_clean":"2024-04-26T16:31:00.296529+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:31:00.296529+0000","last_undegraded":"2024-04-26T16:31:00.296529+0000","last_fullsized":"2024-04-26T16:31:00.296529+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.075499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:00.585431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":940,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319345+0000","last_change":"2024-04-26T16:30:28.910693+0000","last_active":"2024-04-26T16:31:00.319345+0000","last_peered":"2024-04-26T16:31:00.319345+0000","last_clean":"2024-04-26T16:31:00.319345+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:31:00.319345+0000","last_undegraded":"2024-04-26T16:31:00.319345+0000","last_fullsized":"2024-04-26T16:31:00.319345+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.057825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":945,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296995+0000","last_change":"2024-04-26T16:30:07.455469+0000","last_active":"2024-04-26T16:31:00.296995+0000","last_peered":"2024-04-26T16:31:00.296995+0000","last_clean":"2024-04-26T16:31:00.296995+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:31:00.296995+0000","last_undegraded":"2024-04-26T16:31:00.296995+0000","last_fullsized":"2024-04-26T16:31:00.296995+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.455245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.455245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:20.143258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":810,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296343+0000","last_change":"2024-04-26T16:30:02.403961+0000","last_active":"2024-04-26T16:31:00.296343+0000","last_peered":"2024-04-26T16:31:00.296343+0000","last_clean":"2024-04-26T16:31:00.296343+0000","last_became_active":"2024-04-26T16:28:25.246039+0000","last_became_peered":"2024-04-26T16:28:25.246039+0000","last_unstale":"2024-04-26T16:31:00.296343+0000","last_undegraded":"2024-04-26T16:31:00.296343+0000","last_fullsized":"2024-04-26T16:31:00.296343+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:02.403733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.403733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:32.066798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"388'8","reported_seq":913,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.325916+0000","last_change":"2024-04-26T16:30:29.358181+0000","last_active":"2024-04-26T16:31:02.325916+0000","last_peered":"2024-04-26T16:31:02.325916+0000","last_clean":"2024-04-26T16:31:02.325916+0000","last_became_active":"2024-04-26T16:28:25.260556+0000","last_became_peered":"2024-04-26T16:28:25.260556+0000","last_unstale":"2024-04-26T16:31:02.325916+0000","last_undegraded":"2024-04-26T16:31:02.325916+0000","last_fullsized":"2024-04-26T16:31:02.325916+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'8","last_scrub_stamp":"2024-04-26T16:30:29.357863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.357863+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:36.666658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":842,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.326976+0000","last_change":"2024-04-26T16:30:28.356139+0000","last_active":"2024-04-26T16:31:02.326976+0000","last_peered":"2024-04-26T16:31:02.326976+0000","last_clean":"2024-04-26T16:31:02.326976+0000","last_became_active":"2024-04-26T16:28:25.242243+0000","last_became_peered":"2024-04-26T16:28:25.242243+0000","last_unstale":"2024-04-26T16:31:02.326976+0000","last_undegraded":"2024-04-26T16:31:02.326976+0000","last_fullsized":"2024-04-26T16:31:02.326976+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.355929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:58.034883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":445,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319148+0000","last_change":"2024-04-26T16:28:47.096936+0000","last_active":"2024-04-26T16:31:00.319148+0000","last_peered":"2024-04-26T16:31:00.319148+0000","last_clean":"2024-04-26T16:31:00.319148+0000","last_became_active":"2024-04-26T16:28:25.243371+0000","last_became_peered":"2024-04-26T16:28:25.243371+0000","last_unstale":"2024-04-26T16:31:00.319148+0000","last_undegraded":"2024-04-26T16:31:00.319148+0000","last_fullsized":"2024-04-26T16:31:00.319148+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:47.096851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.096851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":936,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326513+0000","last_change":"2024-04-26T16:28:25.266733+0000","last_active":"2024-04-26T16:31:00.326513+0000","last_peered":"2024-04-26T16:31:00.326513+0000","last_clean":"2024-04-26T16:31:00.326513+0000","last_became_active":"2024-04-26T16:28:25.266217+0000","last_became_peered":"2024-04-26T16:28:25.266217+0000","last_unstale":"2024-04-26T16:31:00.326513+0000","last_undegraded":"2024-04-26T16:31:00.326513+0000","last_fullsized":"2024-04-26T16:31:00.326513+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:18.906360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:27:18.906360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":810,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309289+0000","last_change":"2024-04-26T16:29:58.659839+0000","last_active":"2024-04-26T16:31:01.309289+0000","last_peered":"2024-04-26T16:31:01.309289+0000","last_clean":"2024-04-26T16:31:01.309289+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:31:01.309289+0000","last_undegraded":"2024-04-26T16:31:01.309289+0000","last_fullsized":"2024-04-26T16:31:01.309289+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.659641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.659641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:17.377583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'19","reported_seq":992,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326389+0000","last_change":"2024-04-26T16:27:29.963129+0000","last_active":"2024-04-26T16:31:00.326389+0000","last_peered":"2024-04-26T16:31:00.326389+0000","last_clean":"2024-04-26T16:31:00.326389+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:31:00.326389+0000","last_undegraded":"2024-04-26T16:31:00.326389+0000","last_fullsized":"2024-04-26T16:31:00.326389+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:29.962882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.962882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":444,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317626+0000","last_change":"2024-04-26T16:30:07.968532+0000","last_active":"2024-04-26T16:31:00.317626+0000","last_peered":"2024-04-26T16:31:00.317626+0000","last_clean":"2024-04-26T16:31:00.317626+0000","last_became_active":"2024-04-26T16:26:23.216238+0000","last_became_peered":"2024-04-26T16:26:23.216238+0000","last_unstale":"2024-04-26T16:31:00.317626+0000","last_undegraded":"2024-04-26T16:31:00.317626+0000","last_fullsized":"2024-04-26T16:31:00.317626+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.968404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.968404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.834438+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"316'13","reported_seq":586,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315973+0000","last_change":"2024-04-26T16:30:26.031295+0000","last_active":"2024-04-26T16:31:01.315973+0000","last_peered":"2024-04-26T16:31:01.315973+0000","last_clean":"2024-04-26T16:31:01.315973+0000","last_became_active":"2024-04-26T16:26:23.215841+0000","last_became_peered":"2024-04-26T16:26:23.215841+0000","last_unstale":"2024-04-26T16:31:01.315973+0000","last_undegraded":"2024-04-26T16:31:01.315973+0000","last_fullsized":"2024-04-26T16:31:01.315973+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:26.031142+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.031142+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.724917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":949,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312978+0000","last_change":"2024-04-26T16:30:22.624802+0000","last_active":"2024-04-26T16:31:01.312978+0000","last_peered":"2024-04-26T16:31:01.312978+0000","last_clean":"2024-04-26T16:31:01.312978+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:31:01.312978+0000","last_undegraded":"2024-04-26T16:31:01.312978+0000","last_fullsized":"2024-04-26T16:31:01.312978+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.624646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.624646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.857935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":445,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317989+0000","last_change":"2024-04-26T16:28:46.104373+0000","last_active":"2024-04-26T16:31:00.317989+0000","last_peered":"2024-04-26T16:31:00.317989+0000","last_clean":"2024-04-26T16:31:00.317989+0000","last_became_active":"2024-04-26T16:28:25.241392+0000","last_became_peered":"2024-04-26T16:28:25.241392+0000","last_unstale":"2024-04-26T16:31:00.317989+0000","last_undegraded":"2024-04-26T16:31:00.317989+0000","last_fullsized":"2024-04-26T16:31:00.317989+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.104261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.104261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"388'19","reported_seq":986,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308492+0000","last_change":"2024-04-26T16:28:54.496909+0000","last_active":"2024-04-26T16:31:01.308492+0000","last_peered":"2024-04-26T16:31:01.308492+0000","last_clean":"2024-04-26T16:31:01.308492+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:31:01.308492+0000","last_undegraded":"2024-04-26T16:31:01.308492+0000","last_fullsized":"2024-04-26T16:31:01.308492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:28:54.496786+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.496786+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":961,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308906+0000","last_change":"2024-04-26T16:29:10.518620+0000","last_active":"2024-04-26T16:31:01.308906+0000","last_peered":"2024-04-26T16:31:01.308906+0000","last_clean":"2024-04-26T16:31:01.308906+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:31:01.308906+0000","last_undegraded":"2024-04-26T16:31:01.308906+0000","last_fullsized":"2024-04-26T16:31:01.308906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.518359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1105,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318697+0000","last_change":"2024-04-26T16:27:42.184833+0000","last_active":"2024-04-26T16:31:00.318697+0000","last_peered":"2024-04-26T16:31:00.318697+0000","last_clean":"2024-04-26T16:31:00.318697+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:31:00.318697+0000","last_undegraded":"2024-04-26T16:31:00.318697+0000","last_fullsized":"2024-04-26T16:31:00.318697+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:42.184655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:27:42.184655+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":954,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312967+0000","last_change":"2024-04-26T16:30:00.630668+0000","last_active":"2024-04-26T16:31:01.312967+0000","last_peered":"2024-04-26T16:31:01.312967+0000","last_clean":"2024-04-26T16:31:01.312967+0000","last_became_active":"2024-04-26T16:26:23.188629+0000","last_became_peered":"2024-04-26T16:26:23.188629+0000","last_unstale":"2024-04-26T16:31:01.312967+0000","last_undegraded":"2024-04-26T16:31:01.312967+0000","last_fullsized":"2024-04-26T16:31:01.312967+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.630569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.630569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.233286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":791,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328987+0000","last_change":"2024-04-26T16:27:25.909568+0000","last_active":"2024-04-26T16:31:00.328987+0000","last_peered":"2024-04-26T16:31:00.328987+0000","last_clean":"2024-04-26T16:31:00.328987+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:31:00.328987+0000","last_undegraded":"2024-04-26T16:31:00.328987+0000","last_fullsized":"2024-04-26T16:31:00.328987+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:25.909462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.909462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"388'19","reported_seq":972,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.329014+0000","last_change":"2024-04-26T16:28:43.978963+0000","last_active":"2024-04-26T16:31:00.329014+0000","last_peered":"2024-04-26T16:31:00.329014+0000","last_clean":"2024-04-26T16:31:00.329014+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:31:00.329014+0000","last_undegraded":"2024-04-26T16:31:00.329014+0000","last_fullsized":"2024-04-26T16:31:00.329014+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:28:43.978726+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.978726+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":844,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.331141+0000","last_change":"2024-04-26T16:28:52.136590+0000","last_active":"2024-04-26T16:31:02.331141+0000","last_peered":"2024-04-26T16:31:02.331141+0000","last_clean":"2024-04-26T16:31:02.331141+0000","last_became_active":"2024-04-26T16:28:25.270839+0000","last_became_peered":"2024-04-26T16:28:25.270839+0000","last_unstale":"2024-04-26T16:31:02.331141+0000","last_undegraded":"2024-04-26T16:31:02.331141+0000","last_fullsized":"2024-04-26T16:31:02.331141+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.136387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.136387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":940,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318265+0000","last_change":"2024-04-26T16:29:00.063758+0000","last_active":"2024-04-26T16:31:00.318265+0000","last_peered":"2024-04-26T16:31:00.318265+0000","last_clean":"2024-04-26T16:31:00.318265+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:31:00.318265+0000","last_undegraded":"2024-04-26T16:31:00.318265+0000","last_fullsized":"2024-04-26T16:31:00.318265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:00.063648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.063648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":964,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.307953+0000","last_change":"2024-04-26T16:30:12.648490+0000","last_active":"2024-04-26T16:31:01.307953+0000","last_peered":"2024-04-26T16:31:01.307953+0000","last_clean":"2024-04-26T16:31:01.307953+0000","last_became_active":"2024-04-26T16:28:25.256886+0000","last_became_peered":"2024-04-26T16:28:25.256886+0000","last_unstale":"2024-04-26T16:31:01.307953+0000","last_undegraded":"2024-04-26T16:31:01.307953+0000","last_fullsized":"2024-04-26T16:31:01.307953+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.648271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.648271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:23.526396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":693,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.331398+0000","last_change":"2024-04-26T16:29:07.228455+0000","last_active":"2024-04-26T16:31:02.331398+0000","last_peered":"2024-04-26T16:31:02.331398+0000","last_clean":"2024-04-26T16:31:02.331398+0000","last_became_active":"2024-04-26T16:28:25.270409+0000","last_became_peered":"2024-04-26T16:28:25.270409+0000","last_unstale":"2024-04-26T16:31:02.331398+0000","last_undegraded":"2024-04-26T16:31:02.331398+0000","last_fullsized":"2024-04-26T16:31:02.331398+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.228356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.228356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"388'20","reported_seq":995,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309361+0000","last_change":"2024-04-26T16:28:47.517956+0000","last_active":"2024-04-26T16:31:01.309361+0000","last_peered":"2024-04-26T16:31:01.309361+0000","last_clean":"2024-04-26T16:31:01.309361+0000","last_became_active":"2024-04-26T16:28:25.258640+0000","last_became_peered":"2024-04-26T16:28:25.258640+0000","last_unstale":"2024-04-26T16:31:01.309361+0000","last_undegraded":"2024-04-26T16:31:01.309361+0000","last_fullsized":"2024-04-26T16:31:01.309361+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:47.517723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.517723+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694482+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":461,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.315529+0000","last_change":"2024-04-26T16:30:21.968003+0000","last_active":"2024-04-26T16:31:01.315529+0000","last_peered":"2024-04-26T16:31:01.315529+0000","last_clean":"2024-04-26T16:31:01.315529+0000","last_became_active":"2024-04-26T16:28:25.226182+0000","last_became_peered":"2024-04-26T16:28:25.226182+0000","last_unstale":"2024-04-26T16:31:01.315529+0000","last_undegraded":"2024-04-26T16:31:01.315529+0000","last_fullsized":"2024-04-26T16:31:01.315529+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:21.967758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.967758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.703747+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":950,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314853+0000","last_change":"2024-04-26T16:28:40.785051+0000","last_active":"2024-04-26T16:31:01.314853+0000","last_peered":"2024-04-26T16:31:01.314853+0000","last_clean":"2024-04-26T16:31:01.314853+0000","last_became_active":"2024-04-26T16:26:23.213441+0000","last_became_peered":"2024-04-26T16:26:23.213441+0000","last_unstale":"2024-04-26T16:31:01.314853+0000","last_undegraded":"2024-04-26T16:31:01.314853+0000","last_fullsized":"2024-04-26T16:31:01.314853+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.784936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.784936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":942,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317345+0000","last_change":"2024-04-26T16:29:09.036096+0000","last_active":"2024-04-26T16:31:00.317345+0000","last_peered":"2024-04-26T16:31:00.317345+0000","last_clean":"2024-04-26T16:31:00.317345+0000","last_became_active":"2024-04-26T16:26:23.211857+0000","last_became_peered":"2024-04-26T16:26:23.211857+0000","last_unstale":"2024-04-26T16:31:00.317345+0000","last_undegraded":"2024-04-26T16:31:00.317345+0000","last_fullsized":"2024-04-26T16:31:00.317345+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.035817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.035817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":801,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.298128+0000","last_change":"2024-04-26T16:30:15.529255+0000","last_active":"2024-04-26T16:31:00.298128+0000","last_peered":"2024-04-26T16:31:00.298128+0000","last_clean":"2024-04-26T16:31:00.298128+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:31:00.298128+0000","last_undegraded":"2024-04-26T16:31:00.298128+0000","last_fullsized":"2024-04-26T16:31:00.298128+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.529105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.529105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:19.318671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'12","reported_seq":998,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314048+0000","last_change":"2024-04-26T16:28:53.490005+0000","last_active":"2024-04-26T16:31:01.314048+0000","last_peered":"2024-04-26T16:31:01.314048+0000","last_clean":"2024-04-26T16:31:01.314048+0000","last_became_active":"2024-04-26T16:28:25.246218+0000","last_became_peered":"2024-04-26T16:28:25.246218+0000","last_unstale":"2024-04-26T16:31:01.314048+0000","last_undegraded":"2024-04-26T16:31:01.314048+0000","last_fullsized":"2024-04-26T16:31:01.314048+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'12","last_scrub_stamp":"2024-04-26T16:28:53.489846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:53.489846+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":324,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.333377+0000","last_change":"2024-04-26T16:30:36.171558+0000","last_active":"2024-04-26T16:31:02.333377+0000","last_peered":"2024-04-26T16:31:02.333377+0000","last_clean":"2024-04-26T16:31:02.333377+0000","last_became_active":"2024-04-26T16:28:25.231296+0000","last_became_peered":"2024-04-26T16:28:25.231296+0000","last_unstale":"2024-04-26T16:31:02.333377+0000","last_undegraded":"2024-04-26T16:31:02.333377+0000","last_fullsized":"2024-04-26T16:31:02.333377+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:36.171451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:36.171451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.052865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":882,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.327315+0000","last_change":"2024-04-26T16:30:19.325201+0000","last_active":"2024-04-26T16:31:02.327315+0000","last_peered":"2024-04-26T16:31:02.327315+0000","last_clean":"2024-04-26T16:31:02.327315+0000","last_became_active":"2024-04-26T16:28:25.272341+0000","last_became_peered":"2024-04-26T16:28:25.272341+0000","last_unstale":"2024-04-26T16:31:02.327315+0000","last_undegraded":"2024-04-26T16:31:02.327315+0000","last_fullsized":"2024-04-26T16:31:02.327315+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.325091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.325091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.244723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":935,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325892+0000","last_change":"2024-04-26T16:30:34.003108+0000","last_active":"2024-04-26T16:31:00.325892+0000","last_peered":"2024-04-26T16:31:00.325892+0000","last_clean":"2024-04-26T16:31:00.325892+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:31:00.325892+0000","last_undegraded":"2024-04-26T16:31:00.325892+0000","last_fullsized":"2024-04-26T16:31:00.325892+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:34.002980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.002980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.133005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":800,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328780+0000","last_change":"2024-04-26T16:28:35.971437+0000","last_active":"2024-04-26T16:31:00.328780+0000","last_peered":"2024-04-26T16:31:00.328780+0000","last_clean":"2024-04-26T16:31:00.328780+0000","last_became_active":"2024-04-26T16:28:25.269747+0000","last_became_peered":"2024-04-26T16:28:25.269747+0000","last_unstale":"2024-04-26T16:31:00.328780+0000","last_undegraded":"2024-04-26T16:31:00.328780+0000","last_fullsized":"2024-04-26T16:31:00.328780+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:35.971222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:35.971222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'15","reported_seq":496,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328920+0000","last_change":"2024-04-26T16:27:43.107066+0000","last_active":"2024-04-26T16:31:00.328920+0000","last_peered":"2024-04-26T16:31:00.328920+0000","last_clean":"2024-04-26T16:31:00.328920+0000","last_became_active":"2024-04-26T16:26:23.200216+0000","last_became_peered":"2024-04-26T16:26:23.200216+0000","last_unstale":"2024-04-26T16:31:00.328920+0000","last_undegraded":"2024-04-26T16:31:00.328920+0000","last_fullsized":"2024-04-26T16:31:00.328920+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:27:43.106910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:43.106910+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":856,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.338840+0000","last_change":"2024-04-26T16:29:08.217390+0000","last_active":"2024-04-26T16:31:02.338840+0000","last_peered":"2024-04-26T16:31:02.338840+0000","last_clean":"2024-04-26T16:31:02.338840+0000","last_became_active":"2024-04-26T16:28:25.244917+0000","last_became_peered":"2024-04-26T16:28:25.244917+0000","last_unstale":"2024-04-26T16:31:02.338840+0000","last_undegraded":"2024-04-26T16:31:02.338840+0000","last_fullsized":"2024-04-26T16:31:02.338840+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:08.217158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.217158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":948,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325253+0000","last_change":"2024-04-26T16:28:57.063805+0000","last_active":"2024-04-26T16:31:00.325253+0000","last_peered":"2024-04-26T16:31:00.325253+0000","last_clean":"2024-04-26T16:31:00.325253+0000","last_became_active":"2024-04-26T16:28:25.266296+0000","last_became_peered":"2024-04-26T16:28:25.266296+0000","last_unstale":"2024-04-26T16:31:00.325253+0000","last_undegraded":"2024-04-26T16:31:00.325253+0000","last_fullsized":"2024-04-26T16:31:00.325253+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:57.063594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:57.063594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":945,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318351+0000","last_change":"2024-04-26T16:30:09.970620+0000","last_active":"2024-04-26T16:31:00.318351+0000","last_peered":"2024-04-26T16:31:00.318351+0000","last_clean":"2024-04-26T16:31:00.318351+0000","last_became_active":"2024-04-26T16:28:25.228971+0000","last_became_peered":"2024-04-26T16:28:25.228971+0000","last_unstale":"2024-04-26T16:31:00.318351+0000","last_undegraded":"2024-04-26T16:31:00.318351+0000","last_fullsized":"2024-04-26T16:31:00.318351+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.970505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.970505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:39.666703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":801,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.296785+0000","last_change":"2024-04-26T16:30:22.616210+0000","last_active":"2024-04-26T16:31:00.296785+0000","last_peered":"2024-04-26T16:31:00.296785+0000","last_clean":"2024-04-26T16:31:00.296785+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:31:00.296785+0000","last_undegraded":"2024-04-26T16:31:00.296785+0000","last_fullsized":"2024-04-26T16:31:00.296785+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.616065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.616065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.002311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"388'34","reported_seq":1006,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.313551+0000","last_change":"2024-04-26T16:29:04.788890+0000","last_active":"2024-04-26T16:31:01.313551+0000","last_peered":"2024-04-26T16:31:01.313551+0000","last_clean":"2024-04-26T16:31:01.313551+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:31:01.313551+0000","last_undegraded":"2024-04-26T16:31:01.313551+0000","last_fullsized":"2024-04-26T16:31:01.313551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:29:04.788618+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.788618+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":462,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308695+0000","last_change":"2024-04-26T16:29:07.493539+0000","last_active":"2024-04-26T16:31:01.308695+0000","last_peered":"2024-04-26T16:31:01.308695+0000","last_clean":"2024-04-26T16:31:01.308695+0000","last_became_active":"2024-04-26T16:28:25.250387+0000","last_became_peered":"2024-04-26T16:28:25.250387+0000","last_unstale":"2024-04-26T16:31:01.308695+0000","last_undegraded":"2024-04-26T16:31:01.308695+0000","last_fullsized":"2024-04-26T16:31:01.308695+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.493409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.493409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"433'14","reported_seq":1006,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.338538+0000","last_change":"2024-04-26T16:29:00.093520+0000","last_active":"2024-04-26T16:31:02.338538+0000","last_peered":"2024-04-26T16:31:02.338538+0000","last_clean":"2024-04-26T16:31:02.338538+0000","last_became_active":"2024-04-26T16:28:25.265640+0000","last_became_peered":"2024-04-26T16:28:25.265640+0000","last_unstale":"2024-04-26T16:31:02.338538+0000","last_undegraded":"2024-04-26T16:31:02.338538+0000","last_fullsized":"2024-04-26T16:31:02.338538+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:00.093200+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.093200+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":947,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328782+0000","last_change":"2024-04-26T16:30:11.234876+0000","last_active":"2024-04-26T16:31:00.328782+0000","last_peered":"2024-04-26T16:31:00.328782+0000","last_clean":"2024-04-26T16:31:00.328782+0000","last_became_active":"2024-04-26T16:26:23.201293+0000","last_became_peered":"2024-04-26T16:26:23.201293+0000","last_unstale":"2024-04-26T16:31:00.328782+0000","last_undegraded":"2024-04-26T16:31:00.328782+0000","last_fullsized":"2024-04-26T16:31:00.328782+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:11.234642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.234642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:11.595192+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":814,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309456+0000","last_change":"2024-04-26T16:30:16.615509+0000","last_active":"2024-04-26T16:31:01.309456+0000","last_peered":"2024-04-26T16:31:01.309456+0000","last_clean":"2024-04-26T16:31:01.309456+0000","last_became_active":"2024-04-26T16:28:25.247238+0000","last_became_peered":"2024-04-26T16:28:25.247238+0000","last_unstale":"2024-04-26T16:31:01.309456+0000","last_undegraded":"2024-04-26T16:31:01.309456+0000","last_fullsized":"2024-04-26T16:31:01.309456+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.615354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.615354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.463397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"388'15","reported_seq":969,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317342+0000","last_change":"2024-04-26T16:28:59.039798+0000","last_active":"2024-04-26T16:31:00.317342+0000","last_peered":"2024-04-26T16:31:00.317342+0000","last_clean":"2024-04-26T16:31:00.317342+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:31:00.317342+0000","last_undegraded":"2024-04-26T16:31:00.317342+0000","last_fullsized":"2024-04-26T16:31:00.317342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:28:59.039575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.039575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.059011+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":951,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.317479+0000","last_change":"2024-04-26T16:28:33.172830+0000","last_active":"2024-04-26T16:31:00.317479+0000","last_peered":"2024-04-26T16:31:00.317479+0000","last_clean":"2024-04-26T16:31:00.317479+0000","last_became_active":"2024-04-26T16:28:25.245293+0000","last_became_peered":"2024-04-26T16:28:25.245293+0000","last_unstale":"2024-04-26T16:31:00.317479+0000","last_undegraded":"2024-04-26T16:31:00.317479+0000","last_fullsized":"2024-04-26T16:31:00.317479+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:28:33.172458+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:28:33.172458+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058400+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":962,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312710+0000","last_change":"2024-04-26T16:30:28.566079+0000","last_active":"2024-04-26T16:31:01.312710+0000","last_peered":"2024-04-26T16:31:01.312710+0000","last_clean":"2024-04-26T16:31:01.312710+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:31:01.312710+0000","last_undegraded":"2024-04-26T16:31:01.312710+0000","last_fullsized":"2024-04-26T16:31:01.312710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.565983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.565983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:44.718464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":954,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.316719+0000","last_change":"2024-04-26T16:28:46.866392+0000","last_active":"2024-04-26T16:31:01.316719+0000","last_peered":"2024-04-26T16:31:01.316719+0000","last_clean":"2024-04-26T16:31:01.316719+0000","last_became_active":"2024-04-26T16:26:23.203386+0000","last_became_peered":"2024-04-26T16:26:23.203386+0000","last_unstale":"2024-04-26T16:31:01.316719+0000","last_undegraded":"2024-04-26T16:31:01.316719+0000","last_fullsized":"2024-04-26T16:31:01.316719+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.866222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.866222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":462,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309345+0000","last_change":"2024-04-26T16:30:29.497506+0000","last_active":"2024-04-26T16:31:01.309345+0000","last_peered":"2024-04-26T16:31:01.309345+0000","last_clean":"2024-04-26T16:31:01.309345+0000","last_became_active":"2024-04-26T16:28:25.238120+0000","last_became_peered":"2024-04-26T16:28:25.238120+0000","last_unstale":"2024-04-26T16:31:01.309345+0000","last_undegraded":"2024-04-26T16:31:01.309345+0000","last_fullsized":"2024-04-26T16:31:01.309345+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.497411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.497411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.042909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"388'9","reported_seq":972,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309086+0000","last_change":"2024-04-26T16:30:02.530560+0000","last_active":"2024-04-26T16:31:01.309086+0000","last_peered":"2024-04-26T16:31:01.309086+0000","last_clean":"2024-04-26T16:31:01.309086+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:31:01.309086+0000","last_undegraded":"2024-04-26T16:31:01.309086+0000","last_fullsized":"2024-04-26T16:31:01.309086+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:30:02.530372+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:23:25.467392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.530372+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.198488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":906,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.326375+0000","last_change":"2024-04-26T16:30:23.313786+0000","last_active":"2024-04-26T16:31:02.326375+0000","last_peered":"2024-04-26T16:31:02.326375+0000","last_clean":"2024-04-26T16:31:02.326375+0000","last_became_active":"2024-04-26T16:28:25.264839+0000","last_became_peered":"2024-04-26T16:28:25.264839+0000","last_unstale":"2024-04-26T16:31:02.326375+0000","last_undegraded":"2024-04-26T16:31:02.326375+0000","last_fullsized":"2024-04-26T16:31:02.326375+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.313672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.313672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:29.555420+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":952,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.316142+0000","last_change":"2024-04-26T16:28:43.823916+0000","last_active":"2024-04-26T16:31:01.316142+0000","last_peered":"2024-04-26T16:31:01.316142+0000","last_clean":"2024-04-26T16:31:01.316142+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:31:01.316142+0000","last_undegraded":"2024-04-26T16:31:01.316142+0000","last_fullsized":"2024-04-26T16:31:01.316142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:43.823731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.823731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":972,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.312139+0000","last_change":"2024-04-26T16:30:26.547964+0000","last_active":"2024-04-26T16:31:01.312139+0000","last_peered":"2024-04-26T16:31:01.312139+0000","last_clean":"2024-04-26T16:31:01.312139+0000","last_became_active":"2024-04-26T16:28:25.244502+0000","last_became_peered":"2024-04-26T16:28:25.244502+0000","last_unstale":"2024-04-26T16:31:01.312139+0000","last_undegraded":"2024-04-26T16:31:01.312139+0000","last_fullsized":"2024-04-26T16:31:01.312139+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.547756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.547756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.908182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":791,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325464+0000","last_change":"2024-04-26T16:27:44.070368+0000","last_active":"2024-04-26T16:31:00.325464+0000","last_peered":"2024-04-26T16:31:00.325464+0000","last_clean":"2024-04-26T16:31:00.325464+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:31:00.325464+0000","last_undegraded":"2024-04-26T16:31:00.325464+0000","last_fullsized":"2024-04-26T16:31:00.325464+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:27:44.070141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'15","reported_seq":989,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310104+0000","last_change":"2024-04-26T16:30:18.494734+0000","last_active":"2024-04-26T16:31:01.310104+0000","last_peered":"2024-04-26T16:31:01.310104+0000","last_clean":"2024-04-26T16:31:01.310104+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:31:01.310104+0000","last_undegraded":"2024-04-26T16:31:01.310104+0000","last_fullsized":"2024-04-26T16:31:01.310104+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:30:18.494624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.494624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:21.675249+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":949,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318177+0000","last_change":"2024-04-26T16:28:36.216812+0000","last_active":"2024-04-26T16:31:00.318177+0000","last_peered":"2024-04-26T16:31:00.318177+0000","last_clean":"2024-04-26T16:31:00.318177+0000","last_became_active":"2024-04-26T16:28:25.255455+0000","last_became_peered":"2024-04-26T16:28:25.255455+0000","last_unstale":"2024-04-26T16:31:00.318177+0000","last_undegraded":"2024-04-26T16:31:00.318177+0000","last_fullsized":"2024-04-26T16:31:00.318177+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:36.216718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:36.216718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":952,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309585+0000","last_change":"2024-04-26T16:29:04.486590+0000","last_active":"2024-04-26T16:31:01.309585+0000","last_peered":"2024-04-26T16:31:01.309585+0000","last_clean":"2024-04-26T16:31:01.309585+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:31:01.309585+0000","last_undegraded":"2024-04-26T16:31:01.309585+0000","last_fullsized":"2024-04-26T16:31:01.309585+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:04.486382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.486382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":952,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319233+0000","last_change":"2024-04-26T16:30:26.949591+0000","last_active":"2024-04-26T16:31:00.319233+0000","last_peered":"2024-04-26T16:31:00.319233+0000","last_clean":"2024-04-26T16:31:00.319233+0000","last_became_active":"2024-04-26T16:28:25.245307+0000","last_became_peered":"2024-04-26T16:28:25.245307+0000","last_unstale":"2024-04-26T16:31:00.319233+0000","last_undegraded":"2024-04-26T16:31:00.319233+0000","last_fullsized":"2024-04-26T16:31:00.319233+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.949357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.949357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:30.336295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":801,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.297488+0000","last_change":"2024-04-26T16:30:04.495577+0000","last_active":"2024-04-26T16:31:00.297488+0000","last_peered":"2024-04-26T16:31:00.297488+0000","last_clean":"2024-04-26T16:31:00.297488+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:31:00.297488+0000","last_undegraded":"2024-04-26T16:31:00.297488+0000","last_fullsized":"2024-04-26T16:31:00.297488+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:04.495235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.495235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:27.081892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"388'23","reported_seq":626,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.310644+0000","last_change":"2024-04-26T16:28:45.510060+0000","last_active":"2024-04-26T16:31:01.310644+0000","last_peered":"2024-04-26T16:31:01.310644+0000","last_clean":"2024-04-26T16:31:01.310644+0000","last_became_active":"2024-04-26T16:28:25.250379+0000","last_became_peered":"2024-04-26T16:28:25.250379+0000","last_unstale":"2024-04-26T16:31:01.310644+0000","last_undegraded":"2024-04-26T16:31:01.310644+0000","last_fullsized":"2024-04-26T16:31:01.310644+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:28:45.509872+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:28:45.509872+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694762+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":962,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326347+0000","last_change":"2024-04-26T16:29:02.113592+0000","last_active":"2024-04-26T16:31:00.326347+0000","last_peered":"2024-04-26T16:31:00.326347+0000","last_clean":"2024-04-26T16:31:00.326347+0000","last_became_active":"2024-04-26T16:28:25.277266+0000","last_became_peered":"2024-04-26T16:28:25.277266+0000","last_unstale":"2024-04-26T16:31:00.326347+0000","last_undegraded":"2024-04-26T16:31:00.326347+0000","last_fullsized":"2024-04-26T16:31:00.326347+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:02.113366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.113366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":449,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308267+0000","last_change":"2024-04-26T16:30:11.600252+0000","last_active":"2024-04-26T16:31:01.308267+0000","last_peered":"2024-04-26T16:31:01.308267+0000","last_clean":"2024-04-26T16:31:01.308267+0000","last_became_active":"2024-04-26T16:26:23.166910+0000","last_became_peered":"2024-04-26T16:26:23.166910+0000","last_unstale":"2024-04-26T16:31:01.308267+0000","last_undegraded":"2024-04-26T16:31:01.308267+0000","last_fullsized":"2024-04-26T16:31:01.308267+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:11.600054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.600054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:12.242206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":949,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308505+0000","last_change":"2024-04-26T16:30:24.568125+0000","last_active":"2024-04-26T16:31:01.308505+0000","last_peered":"2024-04-26T16:31:01.308505+0000","last_clean":"2024-04-26T16:31:01.308505+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:31:01.308505+0000","last_undegraded":"2024-04-26T16:31:01.308505+0000","last_fullsized":"2024-04-26T16:31:01.308505+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.567773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:36.702289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":715,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-04-26T16:31:02.331951+0000","last_change":"2024-04-26T16:29:10.208455+0000","last_active":"2024-04-26T16:31:02.331951+0000","last_peered":"2024-04-26T16:31:02.331951+0000","last_clean":"2024-04-26T16:31:02.331951+0000","last_became_active":"2024-04-26T16:28:25.255571+0000","last_became_peered":"2024-04-26T16:28:25.255571+0000","last_unstale":"2024-04-26T16:31:02.331951+0000","last_undegraded":"2024-04-26T16:31:02.331951+0000","last_fullsized":"2024-04-26T16:31:02.331951+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.208226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.208226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"388'27","reported_seq":540,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309566+0000","last_change":"2024-04-26T16:30:13.510016+0000","last_active":"2024-04-26T16:31:01.309566+0000","last_peered":"2024-04-26T16:31:01.309566+0000","last_clean":"2024-04-26T16:31:01.309566+0000","last_became_active":"2024-04-26T16:28:25.256896+0000","last_became_peered":"2024-04-26T16:28:25.256896+0000","last_unstale":"2024-04-26T16:31:01.309566+0000","last_undegraded":"2024-04-26T16:31:01.309566+0000","last_fullsized":"2024-04-26T16:31:01.309566+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:13.509869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.509869+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.781981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":964,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.328877+0000","last_change":"2024-04-26T16:30:13.187491+0000","last_active":"2024-04-26T16:31:00.328877+0000","last_peered":"2024-04-26T16:31:00.328877+0000","last_clean":"2024-04-26T16:31:00.328877+0000","last_became_active":"2024-04-26T16:28:25.271550+0000","last_became_peered":"2024-04-26T16:28:25.271550+0000","last_unstale":"2024-04-26T16:31:00.328877+0000","last_undegraded":"2024-04-26T16:31:00.328877+0000","last_fullsized":"2024-04-26T16:31:00.328877+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:13.187197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.187197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.285217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":939,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.325739+0000","last_change":"2024-04-26T16:29:57.141987+0000","last_active":"2024-04-26T16:31:00.325739+0000","last_peered":"2024-04-26T16:31:00.325739+0000","last_clean":"2024-04-26T16:31:00.325739+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:31:00.325739+0000","last_undegraded":"2024-04-26T16:31:00.325739+0000","last_fullsized":"2024-04-26T16:31:00.325739+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:57.141873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.141873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:26.450280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":469,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.313994+0000","last_change":"2024-04-26T16:30:34.033387+0000","last_active":"2024-04-26T16:31:01.313994+0000","last_peered":"2024-04-26T16:31:01.313994+0000","last_clean":"2024-04-26T16:31:01.313994+0000","last_became_active":"2024-04-26T16:26:23.212770+0000","last_became_peered":"2024-04-26T16:26:23.212770+0000","last_unstale":"2024-04-26T16:31:01.313994+0000","last_undegraded":"2024-04-26T16:31:01.313994+0000","last_fullsized":"2024-04-26T16:31:01.313994+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:34.033282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.033282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:45.039967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":814,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326928+0000","last_change":"2024-04-26T16:30:14.201574+0000","last_active":"2024-04-26T16:31:00.326928+0000","last_peered":"2024-04-26T16:31:00.326928+0000","last_clean":"2024-04-26T16:31:00.326928+0000","last_became_active":"2024-04-26T16:28:25.267368+0000","last_became_peered":"2024-04-26T16:28:25.267368+0000","last_unstale":"2024-04-26T16:31:00.326928+0000","last_undegraded":"2024-04-26T16:31:00.326928+0000","last_fullsized":"2024-04-26T16:31:00.326928+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.201493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.201493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:17.192546+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"388'19","reported_seq":1016,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314156+0000","last_change":"2024-04-26T16:29:08.447892+0000","last_active":"2024-04-26T16:31:01.314156+0000","last_peered":"2024-04-26T16:31:01.314156+0000","last_clean":"2024-04-26T16:31:01.314156+0000","last_became_active":"2024-04-26T16:28:25.244993+0000","last_became_peered":"2024-04-26T16:28:25.244993+0000","last_unstale":"2024-04-26T16:31:01.314156+0000","last_undegraded":"2024-04-26T16:31:01.314156+0000","last_fullsized":"2024-04-26T16:31:01.314156+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:08.447471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.447471+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":963,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319261+0000","last_change":"2024-04-26T16:30:32.981558+0000","last_active":"2024-04-26T16:31:00.319261+0000","last_peered":"2024-04-26T16:31:00.319261+0000","last_clean":"2024-04-26T16:31:00.319261+0000","last_became_active":"2024-04-26T16:26:23.214404+0000","last_became_peered":"2024-04-26T16:26:23.214404+0000","last_unstale":"2024-04-26T16:31:00.319261+0000","last_undegraded":"2024-04-26T16:31:00.319261+0000","last_fullsized":"2024-04-26T16:31:00.319261+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:32.981351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:32.981351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.689430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":958,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.313680+0000","last_change":"2024-04-26T16:28:48.837013+0000","last_active":"2024-04-26T16:31:01.313680+0000","last_peered":"2024-04-26T16:31:01.313680+0000","last_clean":"2024-04-26T16:31:01.313680+0000","last_became_active":"2024-04-26T16:26:23.178678+0000","last_became_peered":"2024-04-26T16:26:23.178678+0000","last_unstale":"2024-04-26T16:31:01.313680+0000","last_undegraded":"2024-04-26T16:31:01.313680+0000","last_fullsized":"2024-04-26T16:31:01.313680+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:48.836858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:48.836858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":954,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.309834+0000","last_change":"2024-04-26T16:28:39.423393+0000","last_active":"2024-04-26T16:31:01.309834+0000","last_peered":"2024-04-26T16:31:01.309834+0000","last_clean":"2024-04-26T16:31:01.309834+0000","last_became_active":"2024-04-26T16:26:23.198295+0000","last_became_peered":"2024-04-26T16:26:23.198295+0000","last_unstale":"2024-04-26T16:31:01.309834+0000","last_undegraded":"2024-04-26T16:31:01.309834+0000","last_fullsized":"2024-04-26T16:31:01.309834+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:39.423249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:28:39.423249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:05.694333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":442,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.326531+0000","last_change":"2024-04-26T16:30:16.192009+0000","last_active":"2024-04-26T16:31:00.326531+0000","last_peered":"2024-04-26T16:31:00.326531+0000","last_clean":"2024-04-26T16:31:00.326531+0000","last_became_active":"2024-04-26T16:26:23.199414+0000","last_became_peered":"2024-04-26T16:26:23.199414+0000","last_unstale":"2024-04-26T16:31:00.326531+0000","last_undegraded":"2024-04-26T16:31:00.326531+0000","last_fullsized":"2024-04-26T16:31:00.326531+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.191696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.191696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:24.626156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"388'20","reported_seq":984,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.319423+0000","last_change":"2024-04-26T16:28:48.140271+0000","last_active":"2024-04-26T16:31:00.319423+0000","last_peered":"2024-04-26T16:31:00.319423+0000","last_clean":"2024-04-26T16:31:00.319423+0000","last_became_active":"2024-04-26T16:28:25.232220+0000","last_became_peered":"2024-04-26T16:28:25.232220+0000","last_unstale":"2024-04-26T16:31:00.319423+0000","last_undegraded":"2024-04-26T16:31:00.319423+0000","last_fullsized":"2024-04-26T16:31:00.319423+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:48.140161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:48.140161+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:02.058912+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":970,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.314262+0000","last_change":"2024-04-26T16:30:33.689694+0000","last_active":"2024-04-26T16:31:01.314262+0000","last_peered":"2024-04-26T16:31:01.314262+0000","last_clean":"2024-04-26T16:31:01.314262+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:31:01.314262+0000","last_undegraded":"2024-04-26T16:31:01.314262+0000","last_fullsized":"2024-04-26T16:31:01.314262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.689564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.689564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:02.010334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":969,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-04-26T16:31:01.308559+0000","last_change":"2024-04-26T16:30:22.493662+0000","last_active":"2024-04-26T16:31:01.308559+0000","last_peered":"2024-04-26T16:31:01.308559+0000","last_clean":"2024-04-26T16:31:01.308559+0000","last_became_active":"2024-04-26T16:28:25.250239+0000","last_became_peered":"2024-04-26T16:28:25.250239+0000","last_unstale":"2024-04-26T16:31:01.308559+0000","last_undegraded":"2024-04-26T16:31:01.308559+0000","last_fullsized":"2024-04-26T16:31:01.308559+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.493412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.493412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.496740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":945,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-04-26T16:31:00.318067+0000","last_change":"2024-04-26T16:30:12.027247+0000","last_active":"2024-04-26T16:31:00.318067+0000","last_peered":"2024-04-26T16:31:00.318067+0000","last_clean":"2024-04-26T16:31:00.318067+0000","last_became_active":"2024-04-26T16:28:25.250958+0000","last_became_peered":"2024-04-26T16:28:25.250958+0000","last_unstale":"2024-04-26T16:31:00.318067+0000","last_undegraded":"2024-04-26T16:31:00.318067+0000","last_fullsized":"2024-04-26T16:31:00.318067+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.027062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.027062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:16.760868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":306,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":21,"acting":21,"num_store_stats":1},{"poolid":109,"num_pg":32,"stat_sum":{"num_bytes":8195927,"num_objects":175,"num_object_clones":10,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":120,"num_read_kb":14,"num_write":633,"num_write_kb":12716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":4194787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26664960,"data_stored":24587757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1989,"ondisk_log_size":1989,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":790,"num_read_kb":785,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743597,"num_pgs":108,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80200,"kb_used_data":4604,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93664952,"statfs":{"total":95995035648,"available":95912910848,"internally_reserved":0,"allocated":4714496,"data_stored":2227067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":432,"seq":1855425871905,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36912,"kb_used_data":4448,"kb_used_omap":15,"kb_used_meta":32432,"kb_avail":93708240,"statfs":{"total":95995035648,"available":95957237760,"internally_reserved":0,"allocated":4554752,"data_stored":2243812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15525,"internal_metadata":33211227},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75084,"kb_used_data":11968,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670068,"statfs":{"total":95995035648,"available":95918149632,"internally_reserved":0,"allocated":12255232,"data_stored":9829684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738613,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75964,"kb_used_data":4528,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93669188,"statfs":{"total":95995035648,"available":95917248512,"internally_reserved":0,"allocated":4636672,"data_stored":2226624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411552,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75096,"kb_used_data":7820,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670056,"statfs":{"total":95995035648,"available":95918137344,"internally_reserved":0,"allocated":8007680,"data_stored":5634348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084501,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105576,"kb_used_data":11996,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":93639576,"statfs":{"total":95995035648,"available":95886925824,"internally_reserved":0,"allocated":12283904,"data_stored":9829257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":95812040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855563,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79484,"kb_used_data":8048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93665668,"statfs":{"total":95995035648,"available":95913644032,"internally_reserved":0,"allocated":8241152,"data_stored":5829234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463105,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8511488,"data_stored":8195059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4284416,"data_stored":4000664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8552448,"data_stored":8195161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4485120,"data_stored":4194985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":364544,"data_stored":840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":306,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:31:07.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph status --format=json 2024-04-26T16:31:07.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:07.516 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:07.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: cluster 2024-04-26T16:31:05.733565+0000 mgr.y (mgr.24401) 532 : cluster [DBG] pgmap v814: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:31:07.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: cluster 2024-04-26T16:31:05.733565+0000 mgr.y (mgr.24401) 532 : cluster [DBG] pgmap v814: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:06.234656+0000 mgr.y (mgr.24401) 533 : audit [DBG] from='client.41351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:06.234656+0000 mgr.y (mgr.24401) 533 : audit [DBG] from='client.41351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:07.238757+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:07.238757+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: cluster 2024-04-26T16:31:07.247268+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: cluster 2024-04-26T16:31:07.247268+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:07.289476+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:07.289476+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:07.289656+0000 mon.a (mon.0) 2933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[21463]: audit 2024-04-26T16:31:07.289656+0000 mon.a (mon.0) 2933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: cluster 2024-04-26T16:31:05.733565+0000 mgr.y (mgr.24401) 532 : cluster [DBG] pgmap v814: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: cluster 2024-04-26T16:31:05.733565+0000 mgr.y (mgr.24401) 532 : cluster [DBG] pgmap v814: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:06.234656+0000 mgr.y (mgr.24401) 533 : audit [DBG] from='client.41351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:06.234656+0000 mgr.y (mgr.24401) 533 : audit [DBG] from='client.41351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:07.238757+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:07.238757+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: cluster 2024-04-26T16:31:07.247268+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: cluster 2024-04-26T16:31:07.247268+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:07.289476+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:07.289476+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:07.289656+0000 mon.a (mon.0) 2933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:07 smithi050 bash[29234]: audit 2024-04-26T16:31:07.289656+0000 mon.a (mon.0) 2933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: cluster 2024-04-26T16:31:05.733565+0000 mgr.y (mgr.24401) 532 : cluster [DBG] pgmap v814: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:31:07.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: cluster 2024-04-26T16:31:05.733565+0000 mgr.y (mgr.24401) 532 : cluster [DBG] pgmap v814: 228 pgs: 7 creating+peering, 25 unknown, 196 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-04-26T16:31:07.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:06.234656+0000 mgr.y (mgr.24401) 533 : audit [DBG] from='client.41351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:06.234656+0000 mgr.y (mgr.24401) 533 : audit [DBG] from='client.41351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:07.238757+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:07.238757+0000 mon.a (mon.0) 2931 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: cluster 2024-04-26T16:31:07.247268+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: cluster 2024-04-26T16:31:07.247268+0000 mon.a (mon.0) 2932 : cluster [DBG] osdmap e533: 8 total, 8 up, 7 in 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:07.289476+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:07.289476+0000 mon.b (mon.1) 366 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:07.289656+0000 mon.a (mon.0) 2933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:07 smithi190 bash[23868]: audit 2024-04-26T16:31:07.289656+0000 mon.a (mon.0) 2933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-04-26T16:31:07.645 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:07.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:07.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:07.776 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:07.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:07.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:07.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:07.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:07.912 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:08.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:08.038 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:08.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:08.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.171 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:08.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.303 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:08.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:08.432 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:08.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:06.996786+0000 mgr.y (mgr.24401) 534 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:08.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:06.996786+0000 mgr.y (mgr.24401) 534 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:08.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:08.241613+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:08.241613+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: cluster 2024-04-26T16:31:08.249519+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: cluster 2024-04-26T16:31:08.249519+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:08.294619+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:08.294619+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:08.294685+0000 mon.a (mon.0) 2936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:08 smithi190 bash[23868]: audit 2024-04-26T16:31:08.294685+0000 mon.a (mon.0) 2936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.563 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:06.996786+0000 mgr.y (mgr.24401) 534 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:06.996786+0000 mgr.y (mgr.24401) 534 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:08.241613+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-26T16:31:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:08.241613+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-26T16:31:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: cluster 2024-04-26T16:31:08.249519+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: cluster 2024-04-26T16:31:08.249519+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:08.294619+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:08.294619+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:08.294685+0000 mon.a (mon.0) 2936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[21463]: audit 2024-04-26T16:31:08.294685+0000 mon.a (mon.0) 2936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:06.996786+0000 mgr.y (mgr.24401) 534 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:06.996786+0000 mgr.y (mgr.24401) 534 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:08.241613+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:08.241613+0000 mon.a (mon.0) 2934 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-04-26T16:31:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: cluster 2024-04-26T16:31:08.249519+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2024-04-26T16:31:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: cluster 2024-04-26T16:31:08.249519+0000 mon.a (mon.0) 2935 : cluster [DBG] osdmap e534: 8 total, 8 up, 7 in 2024-04-26T16:31:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:08.294619+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:08.294619+0000 mon.b (mon.1) 367 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:08.294685+0000 mon.a (mon.0) 2936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:08 smithi050 bash[29234]: audit 2024-04-26T16:31:08.294685+0000 mon.a (mon.0) 2936 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-04-26T16:31:08.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:08.696 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:08.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.827 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:08.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:08.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:08.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:09.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:09.091 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:09.194 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:09.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:09.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:09.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:09.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:09.471 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:09.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: cluster 2024-04-26T16:31:07.734391+0000 mgr.y (mgr.24401) 535 : cluster [DBG] pgmap v817: 228 pgs: 7 creating+peering, 221 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:09.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: cluster 2024-04-26T16:31:07.734391+0000 mgr.y (mgr.24401) 535 : cluster [DBG] pgmap v817: 228 pgs: 7 creating+peering, 221 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:09.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: audit 2024-04-26T16:31:09.243705+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: audit 2024-04-26T16:31:09.243705+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: cluster 2024-04-26T16:31:09.247423+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: cluster 2024-04-26T16:31:09.247423+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: audit 2024-04-26T16:31:09.275475+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: audit 2024-04-26T16:31:09.275475+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: audit 2024-04-26T16:31:09.275516+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[21463]: audit 2024-04-26T16:31:09.275516+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: cluster 2024-04-26T16:31:07.734391+0000 mgr.y (mgr.24401) 535 : cluster [DBG] pgmap v817: 228 pgs: 7 creating+peering, 221 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: cluster 2024-04-26T16:31:07.734391+0000 mgr.y (mgr.24401) 535 : cluster [DBG] pgmap v817: 228 pgs: 7 creating+peering, 221 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: audit 2024-04-26T16:31:09.243705+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: audit 2024-04-26T16:31:09.243705+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-26T16:31:09.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: cluster 2024-04-26T16:31:09.247423+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2024-04-26T16:31:09.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: cluster 2024-04-26T16:31:09.247423+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2024-04-26T16:31:09.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: audit 2024-04-26T16:31:09.275475+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: audit 2024-04-26T16:31:09.275475+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: audit 2024-04-26T16:31:09.275516+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.494 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:09 smithi050 bash[29234]: audit 2024-04-26T16:31:09.275516+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:09.601 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:09.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:09.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: cluster 2024-04-26T16:31:07.734391+0000 mgr.y (mgr.24401) 535 : cluster [DBG] pgmap v817: 228 pgs: 7 creating+peering, 221 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:09.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: cluster 2024-04-26T16:31:07.734391+0000 mgr.y (mgr.24401) 535 : cluster [DBG] pgmap v817: 228 pgs: 7 creating+peering, 221 active+clean; 8.4 MiB data, 516 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:09.715 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: audit 2024-04-26T16:31:09.243705+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: audit 2024-04-26T16:31:09.243705+0000 mon.a (mon.0) 2937 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: cluster 2024-04-26T16:31:09.247423+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: cluster 2024-04-26T16:31:09.247423+0000 mon.a (mon.0) 2938 : cluster [DBG] osdmap e535: 8 total, 8 up, 7 in 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: audit 2024-04-26T16:31:09.275475+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: audit 2024-04-26T16:31:09.275475+0000 mon.a (mon.0) 2939 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: audit 2024-04-26T16:31:09.275516+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.716 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:09 smithi190 bash[23868]: audit 2024-04-26T16:31:09.275516+0000 mon.b (mon.1) 368 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:09.731 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:09.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:09.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:09.962 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:09.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:09.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:10.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:10.123 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:10.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:10.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:10.259 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:10.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:10.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:10.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:10.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:10.523 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:10.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:10.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:10.635 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:10.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:10.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:10.784 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:10.887 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:10.917 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:10.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:11.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:11.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:11.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:11.179 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:11.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:11.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:11.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:11.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:11.519 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: cluster 2024-04-26T16:31:09.735242+0000 mgr.y (mgr.24401) 536 : cluster [DBG] pgmap v820: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:11.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: cluster 2024-04-26T16:31:09.735242+0000 mgr.y (mgr.24401) 536 : cluster [DBG] pgmap v820: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:11.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.246186+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:11.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.246186+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:11.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: cluster 2024-04-26T16:31:10.251313+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2024-04-26T16:31:11.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: cluster 2024-04-26T16:31:10.251313+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2024-04-26T16:31:11.520 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.299804+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.299804+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.300406+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.300406+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.300923+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.300923+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.301737+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.521 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[21463]: audit 2024-04-26T16:31:10.301737+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: cluster 2024-04-26T16:31:09.735242+0000 mgr.y (mgr.24401) 536 : cluster [DBG] pgmap v820: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:11.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: cluster 2024-04-26T16:31:09.735242+0000 mgr.y (mgr.24401) 536 : cluster [DBG] pgmap v820: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:11.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.246186+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:11.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.246186+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:11.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: cluster 2024-04-26T16:31:10.251313+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2024-04-26T16:31:11.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: cluster 2024-04-26T16:31:10.251313+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2024-04-26T16:31:11.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.299804+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.299804+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.300406+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.300406+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.300923+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.300923+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.301737+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.524 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:11 smithi050 bash[29234]: audit 2024-04-26T16:31:10.301737+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:11.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: cluster 2024-04-26T16:31:09.735242+0000 mgr.y (mgr.24401) 536 : cluster [DBG] pgmap v820: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:11.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: cluster 2024-04-26T16:31:09.735242+0000 mgr.y (mgr.24401) 536 : cluster [DBG] pgmap v820: 228 pgs: 228 active+clean; 8.4 MiB data, 517 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:11.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.246186+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:11.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.246186+0000 mon.a (mon.0) 2940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:11.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: cluster 2024-04-26T16:31:10.251313+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: cluster 2024-04-26T16:31:10.251313+0000 mon.a (mon.0) 2941 : cluster [DBG] osdmap e536: 8 total, 8 up, 7 in 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.299804+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.299804+0000 mon.a (mon.0) 2942 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.300406+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.300406+0000 mon.b (mon.1) 369 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.300923+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.300923+0000 mon.a (mon.0) 2943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.301737+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.548 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:11 smithi190 bash[23868]: audit 2024-04-26T16:31:10.301737+0000 mon.b (mon.1) 370 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-81"}]: dispatch 2024-04-26T16:31:11.563 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:11.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:11.690 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:11.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:11.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:11.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:11.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:12.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:12.081 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:12.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:12.212 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:12.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:12 smithi190 bash[23868]: cluster 2024-04-26T16:31:11.267542+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2024-04-26T16:31:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:12 smithi190 bash[23868]: cluster 2024-04-26T16:31:11.267542+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2024-04-26T16:31:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:12 smithi190 bash[23868]: audit 2024-04-26T16:31:12.116124+0000 mon.c (mon.2) 655 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:12.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:12 smithi190 bash[23868]: audit 2024-04-26T16:31:12.116124+0000 mon.c (mon.2) 655 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:12.341 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:12.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:12.472 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:12.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[21463]: cluster 2024-04-26T16:31:11.267542+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2024-04-26T16:31:12.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[21463]: cluster 2024-04-26T16:31:11.267542+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2024-04-26T16:31:12.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[21463]: audit 2024-04-26T16:31:12.116124+0000 mon.c (mon.2) 655 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:12.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[21463]: audit 2024-04-26T16:31:12.116124+0000 mon.c (mon.2) 655 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:12.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[29234]: cluster 2024-04-26T16:31:11.267542+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2024-04-26T16:31:12.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[29234]: cluster 2024-04-26T16:31:11.267542+0000 mon.a (mon.0) 2944 : cluster [DBG] osdmap e537: 8 total, 8 up, 7 in 2024-04-26T16:31:12.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[29234]: audit 2024-04-26T16:31:12.116124+0000 mon.c (mon.2) 655 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:12.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:12 smithi050 bash[29234]: audit 2024-04-26T16:31:12.116124+0000 mon.c (mon.2) 655 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:12.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:12.598 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:12.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:12.728 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 -- 172.21.15.50:0/579871545 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 msgr2=0x7fed40108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 --2- 172.21.15.50:0/579871545 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40108770 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00b0a0 tx=0x7fed3c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 -- 172.21.15.50:0/579871545 shutdown_connections 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 --2- 172.21.15.50:0/579871545 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed40102ce0 0x7fed4010f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 --2- 172.21.15.50:0/579871545 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40108770 unknown :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 --2- 172.21.15.50:0/579871545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40102320 0x7fed401027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 -- 172.21.15.50:0/579871545 >> 172.21.15.50:0/579871545 conn(0x7fed400792d0 msgr2=0x7fed400796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 -- 172.21.15.50:0/579871545 shutdown_connections 2024-04-26T16:31:12.732 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 -- 172.21.15.50:0/579871545 wait complete. 2024-04-26T16:31:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 librados: starting msgr at 2024-04-26T16:31:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 librados: starting objecter 2024-04-26T16:31:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 Processor -- start 2024-04-26T16:31:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 -- start start 2024-04-26T16:31:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 librados: setting wanted keys 2024-04-26T16:31:12.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.734+0000 7fed475d0700 1 librados: calling monclient init 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40102320 0x7fed40104000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed40102ce0 0x7fed40104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fed40111990 con 0x7fed40102ce0 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fed40111810 con 0x7fed40102320 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fed40111b10 con 0x7fed40108390 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed44b6b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed40102ce0 0x7fed40104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51528/0 (socket says 172.21.15.50:51528) 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 -- 172.21.15.50:0/2175362708 learned_addr learned my addr 172.21.15.50:0/2175362708 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed4536c700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40102320 0x7fed40104000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 -- 172.21.15.50:0/2175362708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40102320 msgr2=0x7fed40104000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40102320 0x7fed40104000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:12.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 -- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed40102ce0 msgr2=0x7fed40104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed40102ce0 0x7fed40104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 -- 172.21.15.50:0/2175362708 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed401a7670 con 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed45b6d700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40104a80 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e940 tx=0x7fed3800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 10 client.?.objecter ms_handle_connect 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed38004a30 con 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fed38005480 con 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed401a7960 con 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3800caf0 con 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 10 client.41357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed401a7ea0 con 0x7fed40108390 2024-04-26T16:31:12.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 librados: init done 2024-04-26T16:31:12.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4006a110 con 0x7fed40108390 2024-04-26T16:31:12.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fed3800ccf0 con 0x7fed40108390 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(538..538 src has 378..538) ==== 9364+0+0 (secure 0 0 0) 0x7fed38091ba0 con 0x7fed40108390 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 10 client.41357.objecter ms_dispatch 0x7fed40001550 osd_map(538..538 src has 378..538) 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 3 client.41357.objecter handle_osd_map got epochs [538,538] > 0 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 3 client.41357.objecter handle_osd_map decoding full epoch 538 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed4536c700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.738+0000 7fed367fc700 20 client.41357.objecter dump_active .. 0 homeless 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.742+0000 7fed4536c700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 secure :-1 s=READY pgs=3771 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009e60 tx=0x7fed3c00b700 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.742+0000 7fed367fc700 10 client.41357.objecter ms_handle_connect 0x7fed2c071f80 2024-04-26T16:31:12.741 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.746+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fed3805d5e0 con 0x7fed40108390 2024-04-26T16:31:12.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:12.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:12.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:12.982 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:12.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:12.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:12.990+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fed401a8130 con 0x7fed40108390 2024-04-26T16:31:13.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:13.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:13.205 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:13.205 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:13.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:13.237 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.286+0000 7fed367fc700 1 -- 172.21.15.50:0/2175362708 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v539) ==== 91+0+0 (secure 0 0 0) 0x7fed38061290 con 0x7fed40108390 2024-04-26T16:31:13.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 10 librados: watch_flush enter 2024-04-26T16:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 10 librados: watch_flush exit 2024-04-26T16:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 20 client.41357.objecter shutdown clearing up homeless session... 2024-04-26T16:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 10 client.41357.objecter successfully canceled tick 2024-04-26T16:31:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 msgr2=0x7fed2c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 secure :-1 s=READY pgs=3771 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009e60 tx=0x7fed3c00b700 comp rx=0 tx=0).stop 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 msgr2=0x7fed40104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40104a80 secure :-1 s=READY pgs=2693 cs=0 l=1 rev1=1 crypto rx=0x7fed3800e940 tx=0x7fed3800ec50 comp rx=0 tx=0).stop 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 shutdown_connections 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 unknown :-1 s=CLOSED pgs=3771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40102320 0x7fed40104000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed40108390 0x7fed40104a80 unknown :-1 s=CLOSED pgs=2693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 --2- 172.21.15.50:0/2175362708 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed40102ce0 0x7fed40104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 >> 172.21.15.50:0/2175362708 conn(0x7fed400792d0 msgr2=0x7fed4006b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 shutdown_connections 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 -- 172.21.15.50:0/2175362708 wait complete. 2024-04-26T16:31:13.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:13.290+0000 7fed475d0700 1 librados: shutdown 2024-04-26T16:31:13.290 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:31:13.339 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:13.347 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: cluster 2024-04-26T16:31:11.737162+0000 mgr.y (mgr.24401) 537 : cluster [DBG] pgmap v823: 196 pgs: 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 9.5 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: cluster 2024-04-26T16:31:11.737162+0000 mgr.y (mgr.24401) 537 : cluster [DBG] pgmap v823: 196 pgs: 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 9.5 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: cluster 2024-04-26T16:31:12.282133+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: cluster 2024-04-26T16:31:12.282133+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.287075+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.287075+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.290260+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.290260+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.994919+0000 mon.c (mon.2) 656 : audit [INF] from='client.? 172.21.15.50:0/2175362708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.994919+0000 mon.c (mon.2) 656 : audit [INF] from='client.? 172.21.15.50:0/2175362708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.995281+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.348 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[29234]: audit 2024-04-26T16:31:12.995281+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: cluster 2024-04-26T16:31:11.737162+0000 mgr.y (mgr.24401) 537 : cluster [DBG] pgmap v823: 196 pgs: 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 9.5 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: cluster 2024-04-26T16:31:11.737162+0000 mgr.y (mgr.24401) 537 : cluster [DBG] pgmap v823: 196 pgs: 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 9.5 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: cluster 2024-04-26T16:31:12.282133+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: cluster 2024-04-26T16:31:12.282133+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.287075+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.287075+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.290260+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.290260+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.994919+0000 mon.c (mon.2) 656 : audit [INF] from='client.? 172.21.15.50:0/2175362708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.994919+0000 mon.c (mon.2) 656 : audit [INF] from='client.? 172.21.15.50:0/2175362708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.995281+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.349 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:13 smithi050 bash[21463]: audit 2024-04-26T16:31:12.995281+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.358 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:13.461 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:13.484 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: cluster 2024-04-26T16:31:11.737162+0000 mgr.y (mgr.24401) 537 : cluster [DBG] pgmap v823: 196 pgs: 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 9.5 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: cluster 2024-04-26T16:31:11.737162+0000 mgr.y (mgr.24401) 537 : cluster [DBG] pgmap v823: 196 pgs: 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 9.5 KiB/s rd, 12 KiB/s wr, 37 op/s 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: cluster 2024-04-26T16:31:12.282133+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: cluster 2024-04-26T16:31:12.282133+0000 mon.a (mon.0) 2945 : cluster [DBG] osdmap e538: 8 total, 8 up, 7 in 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.287075+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.287075+0000 mon.b (mon.1) 371 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.290260+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.290260+0000 mon.a (mon.0) 2946 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:13.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.994919+0000 mon.c (mon.2) 656 : audit [INF] from='client.? 172.21.15.50:0/2175362708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.994919+0000 mon.c (mon.2) 656 : audit [INF] from='client.? 172.21.15.50:0/2175362708' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.995281+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:13 smithi190 bash[23868]: audit 2024-04-26T16:31:12.995281+0000 mon.a (mon.0) 2947 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:31:13.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:13.617 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:13.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:13.720 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:13.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:13.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:13.876 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:13.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:13.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:14.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:14.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:14.136 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:14.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:14.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:14.380 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.274964+0000 mon.a (mon.0) 2948 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.274964+0000 mon.a (mon.0) 2948 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.285635+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.285635+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.285731+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.285731+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.290044+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.290044+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.296120+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.296120+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.307692+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.307692+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.307751+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:14 smithi190 bash[23868]: audit 2024-04-26T16:31:13.307751+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.399 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:14.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:14.513 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:31:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.274964+0000 mon.a (mon.0) 2948 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.274964+0000 mon.a (mon.0) 2948 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.285635+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.285635+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.285731+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.285731+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.290044+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2024-04-26T16:31:14.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.290044+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.296120+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.296120+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.307692+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.307692+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.307751+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[21463]: audit 2024-04-26T16:31:13.307751+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.274964+0000 mon.a (mon.0) 2948 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.274964+0000 mon.a (mon.0) 2948 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.285635+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.285635+0000 mon.a (mon.0) 2949 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:14.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.285731+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.285731+0000 mon.a (mon.0) 2950 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.290044+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.290044+0000 mon.a (mon.0) 2951 : cluster [DBG] osdmap e539: 8 total, 8 up, 7 in 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.296120+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.296120+0000 mon.b (mon.1) 372 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.307692+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.307692+0000 mon.a (mon.0) 2952 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.307751+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:14 smithi050 bash[29234]: audit 2024-04-26T16:31:13.307751+0000 mon.b (mon.1) 373 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:14.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:14.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:14.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:14.762 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:14.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:14.893 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:14.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:14.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:15.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:15.051 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:15.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:15.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:15.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:15.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.446643+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-04-26T16:31:15.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.446643+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-04-26T16:31:15.300 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.448690+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-04-26T16:31:15.301 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.448690+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-04-26T16:31:15.330 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:15.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:15.460 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:15.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:15.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.737963+0000 mgr.y (mgr.24401) 538 : cluster [DBG] pgmap v826: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-26T16:31:15.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:13.737963+0000 mgr.y (mgr.24401) 538 : cluster [DBG] pgmap v826: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-26T16:31:15.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: audit 2024-04-26T16:31:14.287441+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:15.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: audit 2024-04-26T16:31:14.287441+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:15.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:14.291433+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2024-04-26T16:31:15.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: cluster 2024-04-26T16:31:14.291433+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2024-04-26T16:31:15.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: audit 2024-04-26T16:31:14.296672+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: audit 2024-04-26T16:31:14.296672+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: audit 2024-04-26T16:31:14.301341+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:15 smithi190 bash[23868]: audit 2024-04-26T16:31:14.301341+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.446643+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-04-26T16:31:15.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.446643+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-04-26T16:31:15.578 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.448690+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.448690+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.737963+0000 mgr.y (mgr.24401) 538 : cluster [DBG] pgmap v826: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:13.737963+0000 mgr.y (mgr.24401) 538 : cluster [DBG] pgmap v826: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: audit 2024-04-26T16:31:14.287441+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: audit 2024-04-26T16:31:14.287441+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:14.291433+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: cluster 2024-04-26T16:31:14.291433+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: audit 2024-04-26T16:31:14.296672+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: audit 2024-04-26T16:31:14.296672+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: audit 2024-04-26T16:31:14.301341+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.579 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[21463]: audit 2024-04-26T16:31:14.301341+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.446643+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.446643+0000 osd.2 (osd.2) 269 : cluster [DBG] 5.d scrub starts 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.448690+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.448690+0000 osd.2 (osd.2) 270 : cluster [DBG] 5.d scrub ok 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.737963+0000 mgr.y (mgr.24401) 538 : cluster [DBG] pgmap v826: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:13.737963+0000 mgr.y (mgr.24401) 538 : cluster [DBG] pgmap v826: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 38 op/s 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: audit 2024-04-26T16:31:14.287441+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: audit 2024-04-26T16:31:14.287441+0000 mon.a (mon.0) 2953 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:14.291433+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: cluster 2024-04-26T16:31:14.291433+0000 mon.a (mon.0) 2954 : cluster [DBG] osdmap e540: 8 total, 8 up, 7 in 2024-04-26T16:31:15.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: audit 2024-04-26T16:31:14.296672+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: audit 2024-04-26T16:31:14.296672+0000 mon.b (mon.1) 374 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: audit 2024-04-26T16:31:14.301341+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:15 smithi050 bash[29234]: audit 2024-04-26T16:31:14.301341+0000 mon.a (mon.0) 2955 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:15.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:15.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:15.723 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:15.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:15.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:15.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:15.958 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:15.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:15.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:16.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:16.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:16.222 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:16.256 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:16.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:16.370 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:14.419558+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:14.419558+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:14.421620+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:14.421620+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:15.300392+0000 mon.a (mon.0) 2956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:15.300392+0000 mon.a (mon.0) 2956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.307882+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.307882+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:15.312792+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:15.312792+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:15.313438+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:15.313438+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.744820+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:31:16.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.744820+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.748860+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.748860+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.766924+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.766924+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.768929+0000 osd.7 (osd.7) 308 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.768929+0000 osd.7 (osd.7) 308 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.227555+0000 osd.6 (osd.6) 53 : cluster [DBG] 6.a scrub starts 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.227555+0000 osd.6 (osd.6) 53 : cluster [DBG] 6.a scrub starts 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.230506+0000 osd.6 (osd.6) 54 : cluster [DBG] 6.a scrub ok 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:14.419558+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:31:16.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:14.419558+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:14.421620+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:14.421620+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:15.300392+0000 mon.a (mon.0) 2956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:15.300392+0000 mon.a (mon.0) 2956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.307882+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.307882+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:15.312792+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:15.312792+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:15.313438+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:15.313438+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.744820+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:31:16.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.744820+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.748860+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.748860+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.766924+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.766924+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.768929+0000 osd.7 (osd.7) 308 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.768929+0000 osd.7 (osd.7) 308 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.227555+0000 osd.6 (osd.6) 53 : cluster [DBG] 6.a scrub starts 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.227555+0000 osd.6 (osd.6) 53 : cluster [DBG] 6.a scrub starts 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.230506+0000 osd.6 (osd.6) 54 : cluster [DBG] 6.a scrub ok 2024-04-26T16:31:16.374 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.230506+0000 osd.6 (osd.6) 54 : cluster [DBG] 6.a scrub ok 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:16.231012+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:16.231012+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.239527+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.239527+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:16.242548+0000 mon.b (mon.1) 376 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:16.242548+0000 mon.b (mon.1) 376 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:16.246039+0000 mon.a (mon.0) 2961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[21463]: audit 2024-04-26T16:31:16.246039+0000 mon.a (mon.0) 2961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.230506+0000 osd.6 (osd.6) 54 : cluster [DBG] 6.a scrub ok 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:16.231012+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:16.231012+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:16.375 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.239527+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2024-04-26T16:31:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.239527+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2024-04-26T16:31:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:16.242548+0000 mon.b (mon.1) 376 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:16.242548+0000 mon.b (mon.1) 376 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:16.246039+0000 mon.a (mon.0) 2961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:16 smithi050 bash[29234]: audit 2024-04-26T16:31:16.246039+0000 mon.a (mon.0) 2961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.385 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:16.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:16.516 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:14.419558+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:14.419558+0000 osd.2 (osd.2) 271 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:14.421620+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:14.421620+0000 osd.2 (osd.2) 272 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:15.300392+0000 mon.a (mon.0) 2956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:15.300392+0000 mon.a (mon.0) 2956 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.307882+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.307882+0000 mon.a (mon.0) 2957 : cluster [DBG] osdmap e541: 8 total, 8 up, 7 in 2024-04-26T16:31:16.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:15.312792+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:15.312792+0000 mon.b (mon.1) 375 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:15.313438+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:15.313438+0000 mon.a (mon.0) 2958 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.744820+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.744820+0000 osd.0 (osd.0) 313 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.748860+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:31:16.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.748860+0000 osd.0 (osd.0) 314 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.766924+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.766924+0000 osd.7 (osd.7) 307 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.768929+0000 osd.7 (osd.7) 308 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.768929+0000 osd.7 (osd.7) 308 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.227555+0000 osd.6 (osd.6) 53 : cluster [DBG] 6.a scrub starts 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.227555+0000 osd.6 (osd.6) 53 : cluster [DBG] 6.a scrub starts 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.230506+0000 osd.6 (osd.6) 54 : cluster [DBG] 6.a scrub ok 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.230506+0000 osd.6 (osd.6) 54 : cluster [DBG] 6.a scrub ok 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:16.231012+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:16.231012+0000 mon.a (mon.0) 2959 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.239527+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2024-04-26T16:31:16.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.239527+0000 mon.a (mon.0) 2960 : cluster [DBG] osdmap e542: 8 total, 8 up, 7 in 2024-04-26T16:31:16.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:16.242548+0000 mon.b (mon.1) 376 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:16.242548+0000 mon.b (mon.1) 376 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:16.246039+0000 mon.a (mon.0) 2961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:16 smithi190 bash[23868]: audit 2024-04-26T16:31:16.246039+0000 mon.a (mon.0) 2961 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:16.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:16.648 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:16.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:16.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:16.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:16.910 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:16.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:17.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:17.039 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:17.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:17.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:31:17.170 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:17.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:17.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:17.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:17.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:17.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:17.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:17.557 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.599134+0000 osd.4 (osd.4) 285 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:31:17.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.599134+0000 osd.4 (osd.4) 285 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:31:17.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.602983+0000 osd.4 (osd.4) 286 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:31:17.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.602983+0000 osd.4 (osd.4) 286 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:31:17.558 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.738796+0000 mgr.y (mgr.24401) 539 : cluster [DBG] pgmap v829: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:17.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:15.738796+0000 mgr.y (mgr.24401) 539 : cluster [DBG] pgmap v829: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:17.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.039833+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:31:17.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.039833+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:31:17.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.041963+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:31:17.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.041963+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:31:17.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.709307+0000 osd.0 (osd.0) 315 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:31:17.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.709307+0000 osd.0 (osd.0) 315 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:31:17.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.715733+0000 osd.0 (osd.0) 316 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:31:17.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.715733+0000 osd.0 (osd.0) 316 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:31:17.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.780934+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:31:17.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.780934+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:31:17.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.790138+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:31:17.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.790138+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:31:17.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.222587+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:31:17.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.222587+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:31:17.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.224535+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:31:17.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.224535+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:31:17.561 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: audit 2024-04-26T16:31:17.233425+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:17.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: audit 2024-04-26T16:31:17.233425+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:17.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.241043+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2024-04-26T16:31:17.562 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.241043+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2024-04-26T16:31:17.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.599134+0000 osd.4 (osd.4) 285 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:31:17.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.599134+0000 osd.4 (osd.4) 285 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:31:17.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.602983+0000 osd.4 (osd.4) 286 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:31:17.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.602983+0000 osd.4 (osd.4) 286 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:31:17.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.738796+0000 mgr.y (mgr.24401) 539 : cluster [DBG] pgmap v829: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:17.563 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:15.738796+0000 mgr.y (mgr.24401) 539 : cluster [DBG] pgmap v829: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:17.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.039833+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:31:17.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.039833+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:31:17.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.041963+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:31:17.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.041963+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:31:17.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.709307+0000 osd.0 (osd.0) 315 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:31:17.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.709307+0000 osd.0 (osd.0) 315 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:31:17.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.715733+0000 osd.0 (osd.0) 316 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:31:17.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.715733+0000 osd.0 (osd.0) 316 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:31:17.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.780934+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:31:17.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.780934+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:31:17.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.790138+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:31:17.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.790138+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:31:17.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.222587+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:31:17.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.222587+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:31:17.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.224535+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:31:17.566 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.224535+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:31:17.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: audit 2024-04-26T16:31:17.233425+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:17.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: audit 2024-04-26T16:31:17.233425+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:17.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.241043+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2024-04-26T16:31:17.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:17 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.241043+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2024-04-26T16:31:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.568 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:17.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:17.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.599134+0000 osd.4 (osd.4) 285 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:31:17.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.599134+0000 osd.4 (osd.4) 285 : cluster [DBG] 109.5 scrub starts 2024-04-26T16:31:17.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.602983+0000 osd.4 (osd.4) 286 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:31:17.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.602983+0000 osd.4 (osd.4) 286 : cluster [DBG] 109.5 scrub ok 2024-04-26T16:31:17.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.738796+0000 mgr.y (mgr.24401) 539 : cluster [DBG] pgmap v829: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:15.738796+0000 mgr.y (mgr.24401) 539 : cluster [DBG] pgmap v829: 228 pgs: 5 creating+peering, 27 unknown, 196 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.039833+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.039833+0000 osd.3 (osd.3) 339 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.041963+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.041963+0000 osd.3 (osd.3) 340 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.709307+0000 osd.0 (osd.0) 315 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.709307+0000 osd.0 (osd.0) 315 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.715733+0000 osd.0 (osd.0) 316 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.715733+0000 osd.0 (osd.0) 316 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.780934+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.780934+0000 osd.7 (osd.7) 309 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:31:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.790138+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.790138+0000 osd.7 (osd.7) 310 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.222587+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.222587+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.224535+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.224535+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: audit 2024-04-26T16:31:17.233425+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: audit 2024-04-26T16:31:17.233425+0000 mon.a (mon.0) 2962 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.241043+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2024-04-26T16:31:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:17 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.241043+0000 mon.a (mon.0) 2963 : cluster [DBG] osdmap e543: 8 total, 8 up, 7 in 2024-04-26T16:31:17.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:17.698 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:17.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:17.827 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:17.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:17.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:17.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:17.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:31:18.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:18.088 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:18.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:18.220 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:18.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:18.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:18.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:18.465 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.553204+0000 osd.4 (osd.4) 287 : cluster [DBG] 109.14 scrub starts 2024-04-26T16:31:18.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.553204+0000 osd.4 (osd.4) 287 : cluster [DBG] 109.14 scrub starts 2024-04-26T16:31:18.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.556372+0000 osd.4 (osd.4) 288 : cluster [DBG] 109.14 scrub ok 2024-04-26T16:31:18.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.556372+0000 osd.4 (osd.4) 288 : cluster [DBG] 109.14 scrub ok 2024-04-26T16:31:18.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.899269+0000 osd.2 (osd.2) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:18.466 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.899269+0000 osd.2 (osd.2) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.899584+0000 osd.2 (osd.2) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:16.899584+0000 osd.2 (osd.2) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: audit 2024-04-26T16:31:17.008287+0000 mgr.y (mgr.24401) 540 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: audit 2024-04-26T16:31:17.008287+0000 mgr.y (mgr.24401) 540 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.079335+0000 osd.3 (osd.3) 341 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.079335+0000 osd.3 (osd.3) 341 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:31:18.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.080984+0000 osd.3 (osd.3) 342 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:31:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.080984+0000 osd.3 (osd.3) 342 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:31:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.238660+0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2024-04-26T16:31:18.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:18 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.238660+0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2024-04-26T16:31:18.480 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:18.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.553204+0000 osd.4 (osd.4) 287 : cluster [DBG] 109.14 scrub starts 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.553204+0000 osd.4 (osd.4) 287 : cluster [DBG] 109.14 scrub starts 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.556372+0000 osd.4 (osd.4) 288 : cluster [DBG] 109.14 scrub ok 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.556372+0000 osd.4 (osd.4) 288 : cluster [DBG] 109.14 scrub ok 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.899269+0000 osd.2 (osd.2) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.899269+0000 osd.2 (osd.2) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.899584+0000 osd.2 (osd.2) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:18.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:16.899584+0000 osd.2 (osd.2) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: audit 2024-04-26T16:31:17.008287+0000 mgr.y (mgr.24401) 540 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: audit 2024-04-26T16:31:17.008287+0000 mgr.y (mgr.24401) 540 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.079335+0000 osd.3 (osd.3) 341 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.079335+0000 osd.3 (osd.3) 341 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.080984+0000 osd.3 (osd.3) 342 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.080984+0000 osd.3 (osd.3) 342 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.238660+0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.238660+0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2024-04-26T16:31:18.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.553204+0000 osd.4 (osd.4) 287 : cluster [DBG] 109.14 scrub starts 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.553204+0000 osd.4 (osd.4) 287 : cluster [DBG] 109.14 scrub starts 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.556372+0000 osd.4 (osd.4) 288 : cluster [DBG] 109.14 scrub ok 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.556372+0000 osd.4 (osd.4) 288 : cluster [DBG] 109.14 scrub ok 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.899269+0000 osd.2 (osd.2) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.899269+0000 osd.2 (osd.2) 273 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.899584+0000 osd.2 (osd.2) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:16.899584+0000 osd.2 (osd.2) 274 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: audit 2024-04-26T16:31:17.008287+0000 mgr.y (mgr.24401) 540 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: audit 2024-04-26T16:31:17.008287+0000 mgr.y (mgr.24401) 540 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.079335+0000 osd.3 (osd.3) 341 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.079335+0000 osd.3 (osd.3) 341 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:31:18.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.080984+0000 osd.3 (osd.3) 342 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:31:18.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.080984+0000 osd.3 (osd.3) 342 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:31:18.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.238660+0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2024-04-26T16:31:18.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:18 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.238660+0000 mon.a (mon.0) 2964 : cluster [DBG] osdmap e544: 8 total, 8 up, 7 in 2024-04-26T16:31:18.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:18.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:18.736 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:18.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:18.867 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:18.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:18.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:18.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:19.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:19.118 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:19.149 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:31:19.149 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:31:19.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:19.249 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:19.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:19.379 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:19.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:19.510 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:19.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.739686+0000 mgr.y (mgr.24401) 541 : cluster [DBG] pgmap v832: 228 pgs: 5 creating+peering, 223 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:17.739686+0000 mgr.y (mgr.24401) 541 : cluster [DBG] pgmap v832: 228 pgs: 5 creating+peering, 223 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.048717+0000 osd.3 (osd.3) 343 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.048717+0000 osd.3 (osd.3) 343 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.050858+0000 osd.3 (osd.3) 344 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.050858+0000 osd.3 (osd.3) 344 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.095537+0000 osd.1 (osd.1) 291 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.095537+0000 osd.1 (osd.1) 291 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.097508+0000 osd.1 (osd.1) 292 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:18.097508+0000 osd.1 (osd.1) 292 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:31:19.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.241054+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.241054+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.305144+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.305144+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.305317+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.305317+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.306665+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.306665+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.307120+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:19 smithi190 bash[23868]: audit 2024-04-26T16:31:19.307120+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.739686+0000 mgr.y (mgr.24401) 541 : cluster [DBG] pgmap v832: 228 pgs: 5 creating+peering, 223 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:17.739686+0000 mgr.y (mgr.24401) 541 : cluster [DBG] pgmap v832: 228 pgs: 5 creating+peering, 223 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.048717+0000 osd.3 (osd.3) 343 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.048717+0000 osd.3 (osd.3) 343 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.050858+0000 osd.3 (osd.3) 344 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.050858+0000 osd.3 (osd.3) 344 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.095537+0000 osd.1 (osd.1) 291 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.095537+0000 osd.1 (osd.1) 291 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.097508+0000 osd.1 (osd.1) 292 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:18.097508+0000 osd.1 (osd.1) 292 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.241054+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.241054+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2024-04-26T16:31:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.305144+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.305144+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.305317+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.305317+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.306665+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.306665+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.307120+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[21463]: audit 2024-04-26T16:31:19.307120+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.739686+0000 mgr.y (mgr.24401) 541 : cluster [DBG] pgmap v832: 228 pgs: 5 creating+peering, 223 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:17.739686+0000 mgr.y (mgr.24401) 541 : cluster [DBG] pgmap v832: 228 pgs: 5 creating+peering, 223 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.048717+0000 osd.3 (osd.3) 343 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:31:19.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.048717+0000 osd.3 (osd.3) 343 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.050858+0000 osd.3 (osd.3) 344 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.050858+0000 osd.3 (osd.3) 344 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.095537+0000 osd.1 (osd.1) 291 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.095537+0000 osd.1 (osd.1) 291 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.097508+0000 osd.1 (osd.1) 292 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:18.097508+0000 osd.1 (osd.1) 292 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.241054+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.241054+0000 mon.a (mon.0) 2965 : cluster [DBG] osdmap e545: 8 total, 8 up, 7 in 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.305144+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.305144+0000 mon.a (mon.0) 2966 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.305317+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.305317+0000 mon.b (mon.1) 377 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.306665+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.306665+0000 mon.a (mon.0) 2967 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.307120+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:19 smithi050 bash[29234]: audit 2024-04-26T16:31:19.307120+0000 mon.b (mon.1) 378 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-83"}]: dispatch 2024-04-26T16:31:19.640 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:19.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:19.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:19.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:19.902 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:19.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:20.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:20.032 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:20.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:20.161 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:20.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:20.288 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:20.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:20.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:20.454 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.054525+0000 osd.3 (osd.3) 345 : cluster [DBG] 3.c scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.054525+0000 osd.3 (osd.3) 345 : cluster [DBG] 3.c scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.056983+0000 osd.3 (osd.3) 346 : cluster [DBG] 3.c scrub ok 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.056983+0000 osd.3 (osd.3) 346 : cluster [DBG] 3.c scrub ok 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.088383+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.088383+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.090511+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.090511+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.491369+0000 osd.0 (osd.0) 317 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.491369+0000 osd.0 (osd.0) 317 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.491541+0000 osd.0 (osd.0) 318 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.491541+0000 osd.0 (osd.0) 318 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.855315+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:31:20.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.855315+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.857646+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.857646+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: audit 2024-04-26T16:31:20.043847+0000 mon.c (mon.2) 657 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[21463]: audit 2024-04-26T16:31:20.043847+0000 mon.c (mon.2) 657 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.054525+0000 osd.3 (osd.3) 345 : cluster [DBG] 3.c scrub starts 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.054525+0000 osd.3 (osd.3) 345 : cluster [DBG] 3.c scrub starts 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.056983+0000 osd.3 (osd.3) 346 : cluster [DBG] 3.c scrub ok 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.056983+0000 osd.3 (osd.3) 346 : cluster [DBG] 3.c scrub ok 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.088383+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.088383+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.090511+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.090511+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:31:20.456 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.491369+0000 osd.0 (osd.0) 317 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.491369+0000 osd.0 (osd.0) 317 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.491541+0000 osd.0 (osd.0) 318 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.491541+0000 osd.0 (osd.0) 318 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.855315+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.855315+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.857646+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.857646+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: audit 2024-04-26T16:31:20.043847+0000 mon.c (mon.2) 657 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:20.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:20 smithi050 bash[29234]: audit 2024-04-26T16:31:20.043847+0000 mon.c (mon.2) 657 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:20.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:20.545 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.054525+0000 osd.3 (osd.3) 345 : cluster [DBG] 3.c scrub starts 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.054525+0000 osd.3 (osd.3) 345 : cluster [DBG] 3.c scrub starts 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.056983+0000 osd.3 (osd.3) 346 : cluster [DBG] 3.c scrub ok 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.056983+0000 osd.3 (osd.3) 346 : cluster [DBG] 3.c scrub ok 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.088383+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.088383+0000 osd.1 (osd.1) 293 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.090511+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.090511+0000 osd.1 (osd.1) 294 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.491369+0000 osd.0 (osd.0) 317 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.491369+0000 osd.0 (osd.0) 317 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:20.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.491541+0000 osd.0 (osd.0) 318 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.491541+0000 osd.0 (osd.0) 318 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.855315+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.855315+0000 osd.7 (osd.7) 311 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.857646+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.857646+0000 osd.7 (osd.7) 312 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: audit 2024-04-26T16:31:20.043847+0000 mon.c (mon.2) 657 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:20.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:20 smithi190 bash[23868]: audit 2024-04-26T16:31:20.043847+0000 mon.c (mon.2) 657 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:31:20.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:20.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:20.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:20.803 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:20.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:20.906 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:20.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:20.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:21.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:21.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:21.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:21.193 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:21.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:21.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:21.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:21.437 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.464079+0000 osd.2 (osd.2) 275 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:31:21.437 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.464079+0000 osd.2 (osd.2) 275 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:31:21.437 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.468550+0000 osd.2 (osd.2) 276 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.468550+0000 osd.2 (osd.2) 276 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.740522+0000 mgr.y (mgr.24401) 542 : cluster [DBG] pgmap v835: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:19.740522+0000 mgr.y (mgr.24401) 542 : cluster [DBG] pgmap v835: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.055363+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.055363+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.057505+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.057505+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.322586+0000 osd.6 (osd.6) 57 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.322586+0000 osd.6 (osd.6) 57 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.323818+0000 osd.6 (osd.6) 58 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.323818+0000 osd.6 (osd.6) 58 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:31:21.438 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.331062+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e546: 8 total, 8 up, 7 in 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.331062+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e546: 8 total, 8 up, 7 in 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:20.768357+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:20.768357+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:20.769959+0000 mon.c (mon.2) 659 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:20.769959+0000 mon.c (mon.2) 659 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:20.779757+0000 mon.a (mon.0) 2969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:20.779757+0000 mon.a (mon.0) 2969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.799337+0000 osd.0 (osd.0) 319 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.799337+0000 osd.0 (osd.0) 319 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.801175+0000 osd.0 (osd.0) 320 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:20.801175+0000 osd.0 (osd.0) 320 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.331600+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e547: 8 total, 8 up, 7 in 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.331600+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e547: 8 total, 8 up, 7 in 2024-04-26T16:31:21.439 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:21.340906+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:21.340906+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:21.343384+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.440 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:21 smithi190 bash[23868]: audit 2024-04-26T16:31:21.343384+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.451 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:21.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:21.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.464079+0000 osd.2 (osd.2) 275 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:31:21.566 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.464079+0000 osd.2 (osd.2) 275 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.468550+0000 osd.2 (osd.2) 276 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.468550+0000 osd.2 (osd.2) 276 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.740522+0000 mgr.y (mgr.24401) 542 : cluster [DBG] pgmap v835: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:19.740522+0000 mgr.y (mgr.24401) 542 : cluster [DBG] pgmap v835: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.055363+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.055363+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.057505+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.057505+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.322586+0000 osd.6 (osd.6) 57 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.322586+0000 osd.6 (osd.6) 57 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.323818+0000 osd.6 (osd.6) 58 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:31:21.567 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.323818+0000 osd.6 (osd.6) 58 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.331062+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e546: 8 total, 8 up, 7 in 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.331062+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e546: 8 total, 8 up, 7 in 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:20.768357+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:20.768357+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:20.769959+0000 mon.c (mon.2) 659 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:20.769959+0000 mon.c (mon.2) 659 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:20.779757+0000 mon.a (mon.0) 2969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:20.779757+0000 mon.a (mon.0) 2969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.799337+0000 osd.0 (osd.0) 319 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.799337+0000 osd.0 (osd.0) 319 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.801175+0000 osd.0 (osd.0) 320 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:31:21.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:20.801175+0000 osd.0 (osd.0) 320 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.331600+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e547: 8 total, 8 up, 7 in 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.331600+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e547: 8 total, 8 up, 7 in 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:21.340906+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:21.340906+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:21.343384+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[21463]: audit 2024-04-26T16:31:21.343384+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.464079+0000 osd.2 (osd.2) 275 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.464079+0000 osd.2 (osd.2) 275 : cluster [DBG] 109.1b scrub starts 2024-04-26T16:31:21.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.468550+0000 osd.2 (osd.2) 276 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.468550+0000 osd.2 (osd.2) 276 : cluster [DBG] 109.1b scrub ok 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.740522+0000 mgr.y (mgr.24401) 542 : cluster [DBG] pgmap v835: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:19.740522+0000 mgr.y (mgr.24401) 542 : cluster [DBG] pgmap v835: 228 pgs: 228 active+clean; 8.4 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.055363+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.055363+0000 osd.1 (osd.1) 295 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.057505+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.057505+0000 osd.1 (osd.1) 296 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.322586+0000 osd.6 (osd.6) 57 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:31:21.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.322586+0000 osd.6 (osd.6) 57 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.323818+0000 osd.6 (osd.6) 58 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.323818+0000 osd.6 (osd.6) 58 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.331062+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e546: 8 total, 8 up, 7 in 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.331062+0000 mon.a (mon.0) 2968 : cluster [DBG] osdmap e546: 8 total, 8 up, 7 in 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:20.768357+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:20.768357+0000 mon.c (mon.2) 658 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:20.769959+0000 mon.c (mon.2) 659 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:20.769959+0000 mon.c (mon.2) 659 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:20.779757+0000 mon.a (mon.0) 2969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:20.779757+0000 mon.a (mon.0) 2969 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.799337+0000 osd.0 (osd.0) 319 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.799337+0000 osd.0 (osd.0) 319 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:31:21.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.801175+0000 osd.0 (osd.0) 320 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:20.801175+0000 osd.0 (osd.0) 320 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.331600+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e547: 8 total, 8 up, 7 in 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.331600+0000 mon.a (mon.0) 2970 : cluster [DBG] osdmap e547: 8 total, 8 up, 7 in 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:21.340906+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:21.340906+0000 mon.b (mon.1) 379 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:21.343384+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:21 smithi050 bash[29234]: audit 2024-04-26T16:31:21.343384+0000 mon.a (mon.0) 2971 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:21.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:21.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:21.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:21.701 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:21.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-26T16:31:21.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:21.830 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:21.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:21.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:21.953 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:21.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:22.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:22.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:22.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:22.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:22.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:22.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:22.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:22.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:22.490 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:22.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:22.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.027643+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:31:22.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.027643+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:31:22.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.029552+0000 osd.1 (osd.1) 298 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:31:22.605 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.029552+0000 osd.1 (osd.1) 298 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.316648+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.316648+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.318856+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.318856+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: audit 2024-04-26T16:31:22.330232+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[29234]: audit 2024-04-26T16:31:22.330232+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.027643+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.027643+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.029552+0000 osd.1 (osd.1) 298 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.029552+0000 osd.1 (osd.1) 298 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.316648+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:31:22.606 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.316648+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:31:22.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.318856+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:31:22.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.318856+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:31:22.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: audit 2024-04-26T16:31:22.330232+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:22.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:22 smithi050 bash[21463]: audit 2024-04-26T16:31:22.330232+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:22.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:22.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:22.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.027643+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.027643+0000 osd.1 (osd.1) 297 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.029552+0000 osd.1 (osd.1) 298 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.029552+0000 osd.1 (osd.1) 298 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.316648+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.316648+0000 osd.6 (osd.6) 59 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.318856+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.318856+0000 osd.6 (osd.6) 60 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: audit 2024-04-26T16:31:22.330232+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:22.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:22 smithi190 bash[23868]: audit 2024-04-26T16:31:22.330232+0000 mon.a (mon.0) 2972 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:22.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:22.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:22.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:22.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:23.011 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:23.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:23.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:23.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:23.275 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:23.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.741556+0000 mgr.y (mgr.24401) 543 : cluster [DBG] pgmap v838: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:21.741556+0000 mgr.y (mgr.24401) 543 : cluster [DBG] pgmap v838: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.344531+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e548: 8 total, 8 up, 7 in 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.344531+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e548: 8 total, 8 up, 7 in 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:22.352117+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:22.352117+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.360378+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.360378+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:22.366629+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:22.366629+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:22.374796+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:22.374796+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.829199+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.829199+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.830847+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:22.830847+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:23.332402+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:23.332402+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.337366+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e549: 8 total, 8 up, 7 in 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.337366+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e549: 8 total, 8 up, 7 in 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:23.341894+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:23.341894+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:23.342082+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[21463]: audit 2024-04-26T16:31:23.342082+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:23.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:23.534 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:23.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.741556+0000 mgr.y (mgr.24401) 543 : cluster [DBG] pgmap v838: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:23.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:21.741556+0000 mgr.y (mgr.24401) 543 : cluster [DBG] pgmap v838: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:23.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.344531+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e548: 8 total, 8 up, 7 in 2024-04-26T16:31:23.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.344531+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e548: 8 total, 8 up, 7 in 2024-04-26T16:31:23.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:22.352117+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:23.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:22.352117+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.360378+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.360378+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:22.366629+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:22.366629+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:22.374796+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:22.374796+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.829199+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.829199+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.830847+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:22.830847+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:23.332402+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:23.332402+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.337366+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e549: 8 total, 8 up, 7 in 2024-04-26T16:31:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.337366+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e549: 8 total, 8 up, 7 in 2024-04-26T16:31:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:23.341894+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:23.341894+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:23.342082+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:23 smithi050 bash[29234]: audit 2024-04-26T16:31:23.342082+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:23.662 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:23.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 -- 172.21.15.50:0/805734637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba64109490 msgr2=0x7fba64109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/805734637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba64109490 0x7fba64109870 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fba500098f0 tx=0x7fba50009c00 comp rx=0 tx=0).stop 2024-04-26T16:31:23.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 -- 172.21.15.50:0/805734637 shutdown_connections 2024-04-26T16:31:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/805734637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba6406c360 0x7fba641102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/805734637 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba64109490 0x7fba64109870 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/805734637 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba6406b9c0 0x7fba6406be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 -- 172.21.15.50:0/805734637 >> 172.21.15.50:0/805734637 conn(0x7fba64079290 msgr2=0x7fba640796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 -- 172.21.15.50:0/805734637 shutdown_connections 2024-04-26T16:31:23.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.674+0000 7fba6ab6d700 1 -- 172.21.15.50:0/805734637 wait complete. 2024-04-26T16:31:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 librados: starting msgr at 2024-04-26T16:31:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 librados: starting objecter 2024-04-26T16:31:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 Processor -- start 2024-04-26T16:31:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- start start 2024-04-26T16:31:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 librados: setting wanted keys 2024-04-26T16:31:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 librados: calling monclient init 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba6406b9c0 0x7fba641991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba6406c360 0x7fba64199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 0x7fba6419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fba64112a70 con 0x7fba6406b9c0 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fba641128f0 con 0x7fba6406c360 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fba64112bf0 con 0x7fba64109490 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba68909700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba6406b9c0 0x7fba641991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 0x7fba6419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 0x7fba6419da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56632/0 (socket says 172.21.15.50:56632) 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 learned_addr learned my addr 172.21.15.50:0/463977871 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:23.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba6406c360 0x7fba64199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba6406c360 msgr2=0x7fba64199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba6406c360 0x7fba64199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba6406b9c0 msgr2=0x7fba641991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fba6406b9c0 0x7fba641991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6419e1b0 con 0x7fba64109490 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 0x7fba6419da70 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00ada0 tx=0x7fba5c00c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba61ffb700 10 client.?.objecter ms_handle_connect 0x7fba64109490 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba61ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c017020 con 0x7fba64109490 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fba5c00cde0 con 0x7fba64109490 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c005280 con 0x7fba64109490 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6419e4a0 con 0x7fba64109490 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 10 client.41363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba641a5d20 con 0x7fba64109490 2024-04-26T16:31:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 librados: init done 2024-04-26T16:31:23.679 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.678+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fba6406a110 con 0x7fba64109490 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fba5c01a070 con 0x7fba64109490 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fba54071f80 0x7fba54074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(549..549 src has 496..549) ==== 9393+0+0 (secure 0 0 0) 0x7fba5c090b70 con 0x7fba64109490 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 osd_map(549..549 src has 496..549) 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 3 client.41363.objecter handle_osd_map got epochs [549,549] > 0 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 3 client.41363.objecter handle_osd_map decoding full epoch 549 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fba54071f80 0x7fba54074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 20 client.41363.objecter dump_active .. 0 homeless 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v549) ==== 54+0+16 (secure 0 0 0) 0x7fba5c095220 con 0x7fba64109490 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 1 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fba64064030 0x7fba64064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 1 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 0 1 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 4 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 10 client.41363.objecter _send_command 1 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba641a9080 con 0x7fba64064030 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fba64064030 0x7fba64064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fba54071f80 0x7fba54074440 secure :-1 s=READY pgs=3772 cs=0 l=1 rev1=1 crypto rx=0x7fba500098c0 tx=0x7fba50019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba54071f80 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba64064030 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fba64064030 0x7fba64064410 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba641a9080 con 0x7fba64064030 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 1: 0 ) 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 10 client.41363.objecter _finish_command 1 = system:0 2024-04-26T16:31:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 4 2024-04-26T16:31:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.682+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 0 1 2024-04-26T16:31:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 2 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 2 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6419a500 osd=0 3 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 0 2 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 4 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 10 client.41363.objecter _send_command 2 2024-04-26T16:31:23.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba64063ab0 con 0x7fba64064030 2024-04-26T16:31:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba64063ab0 con 0x7fba64064030 2024-04-26T16:31:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba61ffb700 10 client.41363.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 4 2024-04-26T16:31:23.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 0 2 2024-04-26T16:31:23.700 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fba6402d310 0x7fba6402d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 1 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 4 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 10 client.41363.objecter _send_command 3 2024-04-26T16:31:23.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fba64104320 con 0x7fba6402d310 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fba6402d310 0x7fba6402d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fba6402d310 0x7fba6402d730 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.702+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba6402d310 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.706+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba64104320 con 0x7fba6402d310 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.706+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 3: 0 ) 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.706+0000 7fba61ffb700 10 client.41363.objecter _finish_command 3 = system:0 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.706+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 4 2024-04-26T16:31:23.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.706+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 1 3 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- 172.21.15.50:0/135570924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b806b130 msgr2=0x7f32b806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- 172.21.15.50:0/135570924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b806b130 0x7f32b806b590 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f32ac009990 tx=0x7f32ac009ca0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- 172.21.15.50:0/135570924 shutdown_connections 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- 172.21.15.50:0/135570924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b806bad0 0x7f32b81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- 172.21.15.50:0/135570924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b81094a0 0x7f32b8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- 172.21.15.50:0/135570924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b806b130 0x7f32b806b590 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- 172.21.15.50:0/135570924 >> 172.21.15.50:0/135570924 conn(0x7f32b80792d0 msgr2=0x7f32b80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- 172.21.15.50:0/135570924 shutdown_connections 2024-04-26T16:31:23.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- 172.21.15.50:0/135570924 wait complete. 2024-04-26T16:31:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 librados: starting msgr at 2024-04-26T16:31:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 librados: starting objecter 2024-04-26T16:31:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 Processor -- start 2024-04-26T16:31:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- start start 2024-04-26T16:31:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 librados: setting wanted keys 2024-04-26T16:31:23.713 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 librados: calling monclient init 2024-04-26T16:31:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b806b130 0x7f32b8199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b806bad0 0x7f32b8199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 0x7f32b819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f32b8112ae0 con 0x7f32b81094a0 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f32b8112960 con 0x7f32b806b130 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.714+0000 7f32bdc58700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f32b8112c60 con 0x7f32b806bad0 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 0x7f32b819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b6ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b806bad0 0x7f32b8199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 0x7f32b819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:54596/0 (socket says 172.21.15.50:54596) 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 -- 172.21.15.50:0/2572873790 learned_addr learned my addr 172.21.15.50:0/2572873790 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b77fe700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b806b130 0x7f32b8199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 -- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b806bad0 msgr2=0x7f32b8199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b806bad0 0x7f32b8199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 -- 172.21.15.50:0/2572873790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b806b130 msgr2=0x7f32b8199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b806b130 0x7f32b8199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.715 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 -- 172.21.15.50:0/2572873790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32b819e210 con 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b6ffd700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b806bad0 0x7f32b8199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b7fff700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 0x7f32b819dad0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f32b0007ee0 tx=0x7f32b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 10 client.?.objecter ms_handle_connect 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b000dbd0 con 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f32b000dd70 con 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b0005820 con 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32b819e500 con 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32bdc58700 10 client.30753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32b81a5d80 con 0x7f32b81094a0 2024-04-26T16:31:23.716 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32bdc58700 1 librados: init done 2024-04-26T16:31:23.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32b806a110 con 0x7f32b81094a0 2024-04-26T16:31:23.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f32b001a070 con 0x7f32b81094a0 2024-04-26T16:31:23.717 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.718+0000 7f32b4ff9700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f32a0071f80 0x7f32a0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(549..549 src has 496..549) ==== 9393+0+0 (secure 0 0 0) 0x7f32b00908f0 con 0x7f32b81094a0 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b4ff9700 10 client.30753.objecter ms_dispatch 0x7f32b8001550 osd_map(549..549 src has 496..549) 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b4ff9700 3 client.30753.objecter handle_osd_map got epochs [549,549] > 0 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b4ff9700 3 client.30753.objecter handle_osd_map decoding full epoch 549 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b77fe700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f32a0071f80 0x7f32a0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b4ff9700 20 client.30753.objecter dump_active .. 0 homeless 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b77fe700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f32a0071f80 0x7f32a0074440 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009680 tx=0x7f32a8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.722+0000 7f32b4ff9700 10 client.30753.objecter ms_handle_connect 0x7f32a0071f80 2024-04-26T16:31:23.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 4 2024-04-26T16:31:23.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.721 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 4 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6410a9e0 osd=1 3 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 1 4 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 4 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 10 client.41363.objecter _send_command 4 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba6402dc70 con 0x7fba6402d310 2024-04-26T16:31:23.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f32b0010070 con 0x7f32b81094a0 2024-04-26T16:31:23.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba6402dc70 con 0x7fba6402d310 2024-04-26T16:31:23.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba61ffb700 10 client.41363.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 4 2024-04-26T16:31:23.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 1 4 2024-04-26T16:31:23.724 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:31:23.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 5 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fba641ad640 0x7fba641afa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 5 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 2 5 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 4 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 10 client.41363.objecter _send_command 5 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fba6402e760 con 0x7fba641ad640 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fba641ad640 0x7fba641afa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fba641ad640 0x7fba641afa30 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.726+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba641ad640 2024-04-26T16:31:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.730+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba6402e760 con 0x7fba641ad640 2024-04-26T16:31:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.730+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 5: 0 ) 2024-04-26T16:31:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.730+0000 7fba61ffb700 10 client.41363.objecter _finish_command 5 = system:0 2024-04-26T16:31:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.730+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 4 2024-04-26T16:31:23.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.730+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 2 5 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 6 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 6 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6402e5f0 osd=2 3 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 2 6 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 4 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 10 client.41363.objecter _send_command 6 2024-04-26T16:31:23.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba6402e070 con 0x7fba641ad640 2024-04-26T16:31:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba6402e070 con 0x7fba641ad640 2024-04-26T16:31:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 10 client.41363.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 4 2024-04-26T16:31:23.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 2 6 2024-04-26T16:31:23.755 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 7 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fba641a8be0 0x7fba641b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 7 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 3 7 2024-04-26T16:31:23.756 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 4 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 10 client.41363.objecter _send_command 7 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fba641b6050 con 0x7fba641a8be0 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fba641a8be0 0x7fba641b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba641a8be0 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fba641a8be0 0x7fba641b3100 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba641b6050 con 0x7fba641a8be0 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 7: 0 ) 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 10 client.41363.objecter _finish_command 7 = system:0 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 4 2024-04-26T16:31:23.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.758+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 3 7 2024-04-26T16:31:23.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:23.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 8 2024-04-26T16:31:23.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.774 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 8 2024-04-26T16:31:23.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba6402e470 osd=3 3 2024-04-26T16:31:23.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 3 8 2024-04-26T16:31:23.777 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 4 2024-04-26T16:31:23.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 10 client.41363.objecter _send_command 8 2024-04-26T16:31:23.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba641a8270 con 0x7fba641a8be0 2024-04-26T16:31:23.778 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba641a8270 con 0x7fba641a8be0 2024-04-26T16:31:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba61ffb700 10 client.41363.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 4 2024-04-26T16:31:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 3 8 2024-04-26T16:31:23.779 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.741556+0000 mgr.y (mgr.24401) 543 : cluster [DBG] pgmap v838: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:23.779 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:21.741556+0000 mgr.y (mgr.24401) 543 : cluster [DBG] pgmap v838: 228 pgs: 29 creating+peering, 3 unknown, 196 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:23.779 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.344531+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e548: 8 total, 8 up, 7 in 2024-04-26T16:31:23.779 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.344531+0000 mon.a (mon.0) 2973 : cluster [DBG] osdmap e548: 8 total, 8 up, 7 in 2024-04-26T16:31:23.779 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:22.352117+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:23.779 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:22.352117+0000 mon.b (mon.1) 380 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.360378+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.360378+0000 mon.a (mon.0) 2974 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:22.366629+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:22.366629+0000 mon.b (mon.1) 381 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:22.374796+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:22.374796+0000 mon.a (mon.0) 2975 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.829199+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.829199+0000 osd.0 (osd.0) 321 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.830847+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:22.830847+0000 osd.0 (osd.0) 322 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:23.332402+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:23.332402+0000 mon.a (mon.0) 2976 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.337366+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e549: 8 total, 8 up, 7 in 2024-04-26T16:31:23.780 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.337366+0000 mon.a (mon.0) 2977 : cluster [DBG] osdmap e549: 8 total, 8 up, 7 in 2024-04-26T16:31:23.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:23.341894+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:23.341894+0000 mon.a (mon.0) 2978 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:23.342082+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:23 smithi190 bash[23868]: audit 2024-04-26T16:31:23.342082+0000 mon.b (mon.1) 382 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:23.781 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 9 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fba641b0c50 0x7fba641b4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 9 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 4 9 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 4 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 10 client.41363.objecter _send_command 9 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.778+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fba641b9a10 con 0x7fba641b0c50 2024-04-26T16:31:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fba641b0c50 0x7fba641b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba641b0c50 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fba641b0c50 0x7fba641b4860 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba641b9a10 con 0x7fba641b0c50 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 9: 0 ) 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba61ffb700 10 client.41363.objecter _finish_command 9 = system:0 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 4 2024-04-26T16:31:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.782+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 4 9 2024-04-26T16:31:23.792 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 10 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 10 2024-04-26T16:31:23.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641a8270 osd=4 3 2024-04-26T16:31:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 4 10 2024-04-26T16:31:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 4 2024-04-26T16:31:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 10 client.41363.objecter _send_command 10 2024-04-26T16:31:23.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba641b4da0 con 0x7fba641b0c50 2024-04-26T16:31:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba641b4da0 con 0x7fba641b0c50 2024-04-26T16:31:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba61ffb700 10 client.41363.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 4 2024-04-26T16:31:23.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 4 10 2024-04-26T16:31:23.801 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:31:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.801 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 11 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fba641b54c0 0x7fba641b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.802+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 11 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 5 11 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 4 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 10 client.41363.objecter _send_command 11 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fba641bcbf0 con 0x7fba641b54c0 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fba641b54c0 0x7fba641b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba641b54c0 2024-04-26T16:31:23.802 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba63fff700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fba641b54c0 0x7fba641b5900 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba641bcbf0 con 0x7fba641b54c0 2024-04-26T16:31:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 11: 0 ) 2024-04-26T16:31:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba61ffb700 10 client.41363.objecter _finish_command 11 = system:0 2024-04-26T16:31:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 4 2024-04-26T16:31:23.803 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.806+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 5 11 2024-04-26T16:31:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 12 2024-04-26T16:31:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 12 2024-04-26T16:31:23.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641b4da0 osd=5 3 2024-04-26T16:31:23.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 5 12 2024-04-26T16:31:23.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 4 2024-04-26T16:31:23.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 10 client.41363.objecter _send_command 12 2024-04-26T16:31:23.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.842+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba641bcbf0 con 0x7fba641b54c0 2024-04-26T16:31:23.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba641bcbf0 con 0x7fba641b54c0 2024-04-26T16:31:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba61ffb700 10 client.41363.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 4 2024-04-26T16:31:23.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 5 12 2024-04-26T16:31:23.844 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:31:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 13 2024-04-26T16:31:23.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fba641ba7f0 0x7fba641bac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 13 2024-04-26T16:31:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 6 13 2024-04-26T16:31:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 4 2024-04-26T16:31:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 10 client.41363.objecter _send_command 13 2024-04-26T16:31:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fba641bff90 con 0x7fba641ba7f0 2024-04-26T16:31:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fba641ba7f0 0x7fba641bac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba6910a700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fba641ba7f0 0x7fba641bac50 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.846+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba641ba7f0 2024-04-26T16:31:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.850+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba641bff90 con 0x7fba641ba7f0 2024-04-26T16:31:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.850+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 13: 0 ) 2024-04-26T16:31:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.850+0000 7fba61ffb700 10 client.41363.objecter _finish_command 13 = system:0 2024-04-26T16:31:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.850+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 4 2024-04-26T16:31:23.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.850+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 6 13 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 14 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 14 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641bcbf0 osd=6 3 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 6 14 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 4 2024-04-26T16:31:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 10 client.41363.objecter _send_command 14 2024-04-26T16:31:23.883 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.886+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba641bb250 con 0x7fba641ba7f0 2024-04-26T16:31:23.886 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba641bb250 con 0x7fba641ba7f0 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba61ffb700 10 client.41363.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 4 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 6 14 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:23.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 15 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fba641bb940 0x7fba641bbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 15 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 7 15 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 4 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 10 client.41363.objecter _send_command 15 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fba641bc4a0 con 0x7fba641bb940 2024-04-26T16:31:23.888 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fba641bb940 0x7fba641bbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:23.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba61ffb700 10 client.41363.objecter ms_handle_connect 0x7fba641bb940 2024-04-26T16:31:23.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.890+0000 7fba68909700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fba641bb940 0x7fba641bbdc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:23.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.894+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fba641bc4a0 con 0x7fba641bb940 2024-04-26T16:31:23.889 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.894+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 15: 0 ) 2024-04-26T16:31:23.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.894+0000 7fba61ffb700 10 client.41363.objecter _finish_command 15 = system:0 2024-04-26T16:31:23.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.894+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 4 2024-04-26T16:31:23.890 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.894+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 7 15 2024-04-26T16:31:23.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:23.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:23.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 10 client.41363.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign -1 16 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 20 client.41363.objecter _get_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_remove -1 16 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 20 client.41363.objecter get_session s=0x7fba641bb250 osd=7 3 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 15 client.41363.objecter _session_command_op_assign 7 16 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 4 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 10 client.41363.objecter _send_command 16 2024-04-26T16:31:23.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.930+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fba641bc680 con 0x7fba641bb940 2024-04-26T16:31:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba61ffb700 1 -- 172.21.15.50:0/463977871 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fba641bc680 con 0x7fba641bb940 2024-04-26T16:31:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba61ffb700 10 client.41363.objecter ms_dispatch 0x7fba64001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba61ffb700 10 client.41363.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:23.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba61ffb700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 4 2024-04-26T16:31:23.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba61ffb700 15 client.41363.objecter _session_command_op_remove 7 16 2024-04-26T16:31:23.931 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 librados: watch_flush enter 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 librados: watch_flush exit 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.0 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fba64064030 msgr2=0x7fba64064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fba64064030 0x7fba64064410 crc :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6419a500 osd=0 1 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.1 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fba6402d310 msgr2=0x7fba6402d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fba6402d310 0x7fba6402d730 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6410a9e0 osd=1 1 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.2 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fba641ad640 msgr2=0x7fba641afa30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fba641ad640 0x7fba641afa30 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e5f0 osd=2 1 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.3 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fba641a8be0 msgr2=0x7fba641b3100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fba641a8be0 0x7fba641b3100 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba6402e470 osd=3 1 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.4 2024-04-26T16:31:23.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fba641b0c50 msgr2=0x7fba641b4860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fba641b0c50 0x7fba641b4860 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641a8270 osd=4 1 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.5 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fba641b54c0 msgr2=0x7fba641b5900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fba641b54c0 0x7fba641b5900 crc :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641b4da0 osd=5 1 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.6 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fba641ba7f0 msgr2=0x7fba641bac50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fba641ba7f0 0x7fba641bac50 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bcbf0 osd=6 1 2024-04-26T16:31:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter close_session for osd.7 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fba641bb940 msgr2=0x7fba641bbdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fba641bb940 0x7fba641bbdc0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter put_session s=0x7fba641bb250 osd=7 1 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 20 client.41363.objecter shutdown clearing up homeless session... 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 10 client.41363.objecter successfully canceled tick 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 reap_dead start 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 reap_dead start 2024-04-26T16:31:23.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 reap_dead start 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 reap_dead start 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6910a700 1 -- 172.21.15.50:0/463977871 reap_dead start 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fba54071f80 msgr2=0x7fba54074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fba54071f80 0x7fba54074440 secure :-1 s=READY pgs=3772 cs=0 l=1 rev1=1 crypto rx=0x7fba500098c0 tx=0x7fba50019040 comp rx=0 tx=0).stop 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 msgr2=0x7fba6419da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 0x7fba6419da70 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00ada0 tx=0x7fba5c00c330 comp rx=0 tx=0).stop 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 shutdown_connections 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fba64109490 0x7fba6419da70 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 --2- 172.21.15.50:0/463977871 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fba54071f80 0x7fba54074440 unknown :-1 s=CLOSED pgs=3772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:23.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 >> 172.21.15.50:0/463977871 conn(0x7fba64079290 msgr2=0x7fba64100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 shutdown_connections 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 -- 172.21.15.50:0/463977871 wait complete. 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:23.934+0000 7fba6ab6d700 1 librados: shutdown 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:31:23.938 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:31:24.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:24.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:24.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f32b819a560 con 0x7f32b81094a0 2024-04-26T16:31:24.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32b4ff9700 1 -- 172.21.15.50:0/2572873790 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+2635 (secure 0 0 0) 0x7f32b005c290 con 0x7f32b81094a0 2024-04-26T16:31:24.130 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:31:24.130 INFO:teuthology.orchestra.run.smithi050.stdout:{"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"2 pool(s) do not have an application enabled","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1305,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":549,"num_osds":8,"num_up_osds":8,"osd_up_since":1714148904,"num_in_osds":7,"osd_in_since":1714148782,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":196},{"state_name":"creating+peering","count":29},{"state_name":"unknown","count":3}],"num_pgs":228,"num_pools":9,"num_objects":382,"data_bytes":8801846,"bytes_used":561876992,"bytes_avail":671403372544,"bytes_total":671965249536,"unknown_pgs_ratio":0.01315789483487606,"inactive_pgs_ratio":0.12719298899173737,"read_bytes_sec":2558,"write_bytes_sec":4350,"read_op_per_sec":4,"write_op_per_sec":5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.50:8443/","prometheus":"http://172.21.15.50:9283/"}},"servicemap":{"epoch":24,"modified":"2024-04-26T16:29:23.677518+0000","services":{"rgw":{"daemons":{"summary":"","14391":{"start_epoch":13,"start_stamp":"2024-04-26T16:17:36.262039+0000","gid":14391,"addr":"172.21.15.50:0/2760378525","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-3305-ga430f9ed (a430f9ed84de3008ca5b632b1ed304bac0e753f6) squid (dev)","ceph_version_short":"19.0.0-3305-ga430f9ed","container_hostname":"smithi050","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:4b71a575c225eecd96353b1d64f11c7118b9683e4e8491c513d9f702bd9a1f45","cpu":"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi050","id":"foo.a","kernel_description":"#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024","kernel_version":"5.15.0-105-generic","mem_swap_kb":"2097148","mem_total_kb":"32753052","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"7966ee1e-0b7e-41f4-81f4-af012f440daa","zone_name":"default","zonegroup_id":"1dd73a9b-9e3d-41f4-b462-225b78b09f40","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-04-26T16:31:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 10 librados: watch_flush enter 2024-04-26T16:31:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 10 librados: watch_flush exit 2024-04-26T16:31:24.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 20 client.30753.objecter shutdown clearing up homeless session... 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 10 client.30753.objecter successfully canceled tick 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f32a0071f80 msgr2=0x7f32a0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f32a0071f80 0x7f32a0074440 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f32a8009680 tx=0x7f32a8009260 comp rx=0 tx=0).stop 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 msgr2=0x7f32b819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 0x7f32b819dad0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f32b0007ee0 tx=0x7f32b000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 shutdown_connections 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f32a0071f80 0x7f32a0074440 unknown :-1 s=CLOSED pgs=3773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f32b806b130 0x7f32b8199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f32b81094a0 0x7f32b819dad0 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 --2- 172.21.15.50:0/2572873790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f32b806bad0 0x7f32b8199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 >> 172.21.15.50:0/2572873790 conn(0x7f32b80792d0 msgr2=0x7f32b8100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:24.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 shutdown_connections 2024-04-26T16:31:24.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 -- 172.21.15.50:0/2572873790 wait complete. 2024-04-26T16:31:24.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:24.134+0000 7f32bdc58700 1 librados: shutdown 2024-04-26T16:31:24.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:24.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:24.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:24.300 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:24.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:24.404 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:24.414 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:31:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.093581+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.093581+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.095859+0000 osd.1 (osd.1) 300 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.095859+0000 osd.1 (osd.1) 300 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.467796+0000 osd.4 (osd.4) 289 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.467796+0000 osd.4 (osd.4) 289 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.471175+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:31:24.416 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.471175+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:23.683652+0000 mon.c (mon.2) 660 : audit [DBG] from='client.? 172.21.15.50:0/463977871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:23.683652+0000 mon.c (mon.2) 660 : audit [DBG] from='client.? 172.21.15.50:0/463977871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.135534+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? 172.21.15.50:0/2572873790' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.135534+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? 172.21.15.50:0/2572873790' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.335117+0000 mon.a (mon.0) 2980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.335117+0000 mon.a (mon.0) 2980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:24.341967+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e550: 8 total, 8 up, 7 in 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: cluster 2024-04-26T16:31:24.341967+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e550: 8 total, 8 up, 7 in 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.344379+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.344379+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.345573+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[21463]: audit 2024-04-26T16:31:24.345573+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.093581+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.093581+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.095859+0000 osd.1 (osd.1) 300 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.095859+0000 osd.1 (osd.1) 300 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.467796+0000 osd.4 (osd.4) 289 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.467796+0000 osd.4 (osd.4) 289 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.471175+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.471175+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:23.683652+0000 mon.c (mon.2) 660 : audit [DBG] from='client.? 172.21.15.50:0/463977871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:23.683652+0000 mon.c (mon.2) 660 : audit [DBG] from='client.? 172.21.15.50:0/463977871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.135534+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? 172.21.15.50:0/2572873790' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T16:31:24.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.135534+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? 172.21.15.50:0/2572873790' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.335117+0000 mon.a (mon.0) 2980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.335117+0000 mon.a (mon.0) 2980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:24.341967+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e550: 8 total, 8 up, 7 in 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: cluster 2024-04-26T16:31:24.341967+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e550: 8 total, 8 up, 7 in 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.344379+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.344379+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.345573+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:24 smithi050 bash[29234]: audit 2024-04-26T16:31:24.345573+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:24.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:24.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.093581+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.093581+0000 osd.1 (osd.1) 299 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.095859+0000 osd.1 (osd.1) 300 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.095859+0000 osd.1 (osd.1) 300 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.467796+0000 osd.4 (osd.4) 289 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.467796+0000 osd.4 (osd.4) 289 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.471175+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.471175+0000 osd.4 (osd.4) 290 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:23.683652+0000 mon.c (mon.2) 660 : audit [DBG] from='client.? 172.21.15.50:0/463977871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:23.683652+0000 mon.c (mon.2) 660 : audit [DBG] from='client.? 172.21.15.50:0/463977871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:24.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.135534+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? 172.21.15.50:0/2572873790' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.135534+0000 mon.a (mon.0) 2979 : audit [DBG] from='client.? 172.21.15.50:0/2572873790' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.335117+0000 mon.a (mon.0) 2980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.335117+0000 mon.a (mon.0) 2980 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_tier","val": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:24.341967+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e550: 8 total, 8 up, 7 in 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: cluster 2024-04-26T16:31:24.341967+0000 mon.a (mon.0) 2981 : cluster [DBG] osdmap e550: 8 total, 8 up, 7 in 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.344379+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.344379+0000 mon.b (mon.1) 383 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.345573+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:24 smithi190 bash[23868]: audit 2024-04-26T16:31:24.345573+0000 mon.a (mon.0) 2982 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:24.554 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:24.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:24.684 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:24.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:24.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:24.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:24.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:24.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:24.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:25.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:25.079 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:25.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:25.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:25.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:25.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:25.347 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:25.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:25.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:25.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:25.592 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.453932+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.c scrub starts 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.453932+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.c scrub starts 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.455390+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.c scrub ok 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.455390+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.c scrub ok 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.741960+0000 mgr.y (mgr.24401) 544 : cluster [DBG] pgmap v841: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:23.741960+0000 mgr.y (mgr.24401) 544 : cluster [DBG] pgmap v841: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:24.116092+0000 osd.1 (osd.1) 301 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:24.116092+0000 osd.1 (osd.1) 301 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:24.119182+0000 osd.1 (osd.1) 302 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:24.119182+0000 osd.1 (osd.1) 302 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: audit 2024-04-26T16:31:25.337867+0000 mon.a (mon.0) 2983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: audit 2024-04-26T16:31:25.337867+0000 mon.a (mon.0) 2983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.345991+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e551: 8 total, 8 up, 7 in 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.345991+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e551: 8 total, 8 up, 7 in 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: audit 2024-04-26T16:31:25.348302+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: audit 2024-04-26T16:31:25.348302+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: audit 2024-04-26T16:31:25.348540+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[21463]: audit 2024-04-26T16:31:25.348540+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.453932+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.c scrub starts 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.453932+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.c scrub starts 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.455390+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.c scrub ok 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.455390+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.c scrub ok 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.741960+0000 mgr.y (mgr.24401) 544 : cluster [DBG] pgmap v841: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:25.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:23.741960+0000 mgr.y (mgr.24401) 544 : cluster [DBG] pgmap v841: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:24.116092+0000 osd.1 (osd.1) 301 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:24.116092+0000 osd.1 (osd.1) 301 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:24.119182+0000 osd.1 (osd.1) 302 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:24.119182+0000 osd.1 (osd.1) 302 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: audit 2024-04-26T16:31:25.337867+0000 mon.a (mon.0) 2983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: audit 2024-04-26T16:31:25.337867+0000 mon.a (mon.0) 2983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.345991+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e551: 8 total, 8 up, 7 in 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.345991+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e551: 8 total, 8 up, 7 in 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: audit 2024-04-26T16:31:25.348302+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: audit 2024-04-26T16:31:25.348302+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: audit 2024-04-26T16:31:25.348540+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:25 smithi050 bash[29234]: audit 2024-04-26T16:31:25.348540+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.605 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:25.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:25.738 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:25.806 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-04-26T16:31:25.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.453932+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.c scrub starts 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.453932+0000 osd.2 (osd.2) 277 : cluster [DBG] 2.c scrub starts 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.455390+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.c scrub ok 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.455390+0000 osd.2 (osd.2) 278 : cluster [DBG] 2.c scrub ok 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.741960+0000 mgr.y (mgr.24401) 544 : cluster [DBG] pgmap v841: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:23.741960+0000 mgr.y (mgr.24401) 544 : cluster [DBG] pgmap v841: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:24.116092+0000 osd.1 (osd.1) 301 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:24.116092+0000 osd.1 (osd.1) 301 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:31:25.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:24.119182+0000 osd.1 (osd.1) 302 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:24.119182+0000 osd.1 (osd.1) 302 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: audit 2024-04-26T16:31:25.337867+0000 mon.a (mon.0) 2983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: audit 2024-04-26T16:31:25.337867+0000 mon.a (mon.0) 2983 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.345991+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e551: 8 total, 8 up, 7 in 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.345991+0000 mon.a (mon.0) 2984 : cluster [DBG] osdmap e551: 8 total, 8 up, 7 in 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: audit 2024-04-26T16:31:25.348302+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: audit 2024-04-26T16:31:25.348302+0000 mon.b (mon.1) 384 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: audit 2024-04-26T16:31:25.348540+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:25 smithi190 bash[23868]: audit 2024-04-26T16:31:25.348540+0000 mon.a (mon.0) 2985 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:25.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:25.867 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:25.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:25.992 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:25.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:26.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:26.123 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:26.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:26.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:26.297 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:26.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:26.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:26.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:26.515 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:26.550 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:31:26.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.409256+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.409256+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.411468+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.411468+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.228940+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.228940+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.231299+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.231299+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:31:26.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: audit 2024-04-26T16:31:26.339947+0000 mon.a (mon.0) 2986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: audit 2024-04-26T16:31:26.339947+0000 mon.a (mon.0) 2986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.345990+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e552: 8 total, 8 up, 7 in 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.345990+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e552: 8 total, 8 up, 7 in 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.409256+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.409256+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.411468+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.411468+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.228940+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.228940+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:31:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.231299+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:31:26.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.231299+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:31:26.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: audit 2024-04-26T16:31:26.339947+0000 mon.a (mon.0) 2986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:26.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: audit 2024-04-26T16:31:26.339947+0000 mon.a (mon.0) 2986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:26.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.345990+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e552: 8 total, 8 up, 7 in 2024-04-26T16:31:26.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:26 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.345990+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e552: 8 total, 8 up, 7 in 2024-04-26T16:31:26.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:26.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:26.775 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.409256+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.409256+0000 osd.2 (osd.2) 279 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.411468+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.411468+0000 osd.2 (osd.2) 280 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.228940+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.228940+0000 osd.6 (osd.6) 61 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:31:26.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.231299+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:31:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.231299+0000 osd.6 (osd.6) 62 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:31:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: audit 2024-04-26T16:31:26.339947+0000 mon.a (mon.0) 2986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: audit 2024-04-26T16:31:26.339947+0000 mon.a (mon.0) 2986 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.345990+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e552: 8 total, 8 up, 7 in 2024-04-26T16:31:26.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:26 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.345990+0000 mon.a (mon.0) 2987 : cluster [DBG] osdmap e552: 8 total, 8 up, 7 in 2024-04-26T16:31:26.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:26.905 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:26.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:26.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:27.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:27.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:27.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:27.150 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:31:27.166 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:27.269 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:27.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:27.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.742507+0000 mgr.y (mgr.24401) 545 : cluster [DBG] pgmap v844: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:25.742507+0000 mgr.y (mgr.24401) 545 : cluster [DBG] pgmap v844: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.124886+0000 osd.1 (osd.1) 303 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.124886+0000 osd.1 (osd.1) 303 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.127058+0000 osd.1 (osd.1) 304 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.127058+0000 osd.1 (osd.1) 304 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:31:27.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.376116+0000 osd.2 (osd.2) 281 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.376116+0000 osd.2 (osd.2) 281 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.379349+0000 osd.2 (osd.2) 282 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.379349+0000 osd.2 (osd.2) 282 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.480256+0000 osd.4 (osd.4) 291 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.480256+0000 osd.4 (osd.4) 291 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.485762+0000 osd.4 (osd.4) 292 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.485762+0000 osd.4 (osd.4) 292 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.774805+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.774805+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.779252+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:26.779252+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:31:27.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: audit 2024-04-26T16:31:27.149343+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:27.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: audit 2024-04-26T16:31:27.149343+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:27.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.346647+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e553: 8 total, 8 up, 7 in 2024-04-26T16:31:27.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:27 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.346647+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e553: 8 total, 8 up, 7 in 2024-04-26T16:31:27.428 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:27.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:27.559 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:27.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:27.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.742507+0000 mgr.y (mgr.24401) 545 : cluster [DBG] pgmap v844: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:25.742507+0000 mgr.y (mgr.24401) 545 : cluster [DBG] pgmap v844: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.124886+0000 osd.1 (osd.1) 303 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.124886+0000 osd.1 (osd.1) 303 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.127058+0000 osd.1 (osd.1) 304 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.127058+0000 osd.1 (osd.1) 304 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.376116+0000 osd.2 (osd.2) 281 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.376116+0000 osd.2 (osd.2) 281 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.379349+0000 osd.2 (osd.2) 282 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.379349+0000 osd.2 (osd.2) 282 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.480256+0000 osd.4 (osd.4) 291 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:31:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.480256+0000 osd.4 (osd.4) 291 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.485762+0000 osd.4 (osd.4) 292 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.485762+0000 osd.4 (osd.4) 292 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.774805+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.774805+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.779252+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:26.779252+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: audit 2024-04-26T16:31:27.149343+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: audit 2024-04-26T16:31:27.149343+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.346647+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e553: 8 total, 8 up, 7 in 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.346647+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e553: 8 total, 8 up, 7 in 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.742507+0000 mgr.y (mgr.24401) 545 : cluster [DBG] pgmap v844: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:27.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:25.742507+0000 mgr.y (mgr.24401) 545 : cluster [DBG] pgmap v844: 228 pgs: 29 creating+peering, 199 active+clean; 8.4 MiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.124886+0000 osd.1 (osd.1) 303 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.124886+0000 osd.1 (osd.1) 303 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.127058+0000 osd.1 (osd.1) 304 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.127058+0000 osd.1 (osd.1) 304 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.376116+0000 osd.2 (osd.2) 281 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.376116+0000 osd.2 (osd.2) 281 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.379349+0000 osd.2 (osd.2) 282 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.379349+0000 osd.2 (osd.2) 282 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.480256+0000 osd.4 (osd.4) 291 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.480256+0000 osd.4 (osd.4) 291 : cluster [DBG] 109.2 scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.485762+0000 osd.4 (osd.4) 292 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.485762+0000 osd.4 (osd.4) 292 : cluster [DBG] 109.2 scrub ok 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.774805+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:31:27.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.774805+0000 osd.0 (osd.0) 323 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:31:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.779252+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:31:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:26.779252+0000 osd.0 (osd.0) 324 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:31:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: audit 2024-04-26T16:31:27.149343+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: audit 2024-04-26T16:31:27.149343+0000 mon.c (mon.2) 661 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.346647+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e553: 8 total, 8 up, 7 in 2024-04-26T16:31:27.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:27 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.346647+0000 mon.a (mon.0) 2988 : cluster [DBG] osdmap e553: 8 total, 8 up, 7 in 2024-04-26T16:31:27.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:27.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:27.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:27.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:27.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:31:27.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:27.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:27.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:28.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:28.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:28.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:28.192 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:28.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:28.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:28.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:28.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:28.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:28.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: audit 2024-04-26T16:31:27.017217+0000 mgr.y (mgr.24401) 546 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: audit 2024-04-26T16:31:27.017217+0000 mgr.y (mgr.24401) 546 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.020407+0000 osd.3 (osd.3) 347 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.020407+0000 osd.3 (osd.3) 347 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.023418+0000 osd.3 (osd.3) 348 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.023418+0000 osd.3 (osd.3) 348 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.790010+0000 osd.0 (osd.0) 325 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.790010+0000 osd.0 (osd.0) 325 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:31:28.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.794093+0000 osd.0 (osd.0) 326 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:31:28.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.794093+0000 osd.0 (osd.0) 326 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:31:28.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.347796+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e554: 8 total, 8 up, 7 in 2024-04-26T16:31:28.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:28 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.347796+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e554: 8 total, 8 up, 7 in 2024-04-26T16:31:28.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:28.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: audit 2024-04-26T16:31:27.017217+0000 mgr.y (mgr.24401) 546 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:28.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: audit 2024-04-26T16:31:27.017217+0000 mgr.y (mgr.24401) 546 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:28.659 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.020407+0000 osd.3 (osd.3) 347 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.020407+0000 osd.3 (osd.3) 347 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.023418+0000 osd.3 (osd.3) 348 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.023418+0000 osd.3 (osd.3) 348 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.790010+0000 osd.0 (osd.0) 325 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.790010+0000 osd.0 (osd.0) 325 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.794093+0000 osd.0 (osd.0) 326 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.794093+0000 osd.0 (osd.0) 326 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.347796+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e554: 8 total, 8 up, 7 in 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.347796+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e554: 8 total, 8 up, 7 in 2024-04-26T16:31:28.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: audit 2024-04-26T16:31:27.017217+0000 mgr.y (mgr.24401) 546 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: audit 2024-04-26T16:31:27.017217+0000 mgr.y (mgr.24401) 546 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.020407+0000 osd.3 (osd.3) 347 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.020407+0000 osd.3 (osd.3) 347 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.023418+0000 osd.3 (osd.3) 348 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.023418+0000 osd.3 (osd.3) 348 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.790010+0000 osd.0 (osd.0) 325 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.790010+0000 osd.0 (osd.0) 325 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.794093+0000 osd.0 (osd.0) 326 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.794093+0000 osd.0 (osd.0) 326 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.347796+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e554: 8 total, 8 up, 7 in 2024-04-26T16:31:28.661 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:28 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.347796+0000 mon.a (mon.0) 2989 : cluster [DBG] osdmap e554: 8 total, 8 up, 7 in 2024-04-26T16:31:28.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:28.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:28.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:28.835 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:28.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:28.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:28.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:29.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:29.096 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:29.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:29.228 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:29.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:29.359 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:29.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:29.488 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:29.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:29.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.743276+0000 mgr.y (mgr.24401) 547 : cluster [DBG] pgmap v847: 228 pgs: 228 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:29.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:27.743276+0000 mgr.y (mgr.24401) 547 : cluster [DBG] pgmap v847: 228 pgs: 228 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:29.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.068354+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.b scrub starts 2024-04-26T16:31:29.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.068354+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.b scrub starts 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.069883+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.b scrub ok 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.069883+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.b scrub ok 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.373686+0000 osd.2 (osd.2) 283 : cluster [DBG] 5.a scrub starts 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.373686+0000 osd.2 (osd.2) 283 : cluster [DBG] 5.a scrub starts 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.375366+0000 osd.2 (osd.2) 284 : cluster [DBG] 5.a scrub ok 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:28.375366+0000 osd.2 (osd.2) 284 : cluster [DBG] 5.a scrub ok 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.420078+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.420078+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.420305+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.420305+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.421503+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.421503+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.421838+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: audit 2024-04-26T16:31:28.421838+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.603 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.172592+0000 osd.6 (osd.6) 63 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:31:29.604 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.172592+0000 osd.6 (osd.6) 63 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:31:29.604 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.174762+0000 osd.6 (osd.6) 64 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:31:29.604 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:29 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.174762+0000 osd.6 (osd.6) 64 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:31:29.615 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:29.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:29.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:29.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:29.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:29.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.743276+0000 mgr.y (mgr.24401) 547 : cluster [DBG] pgmap v847: 228 pgs: 228 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:29.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:27.743276+0000 mgr.y (mgr.24401) 547 : cluster [DBG] pgmap v847: 228 pgs: 228 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:29.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.068354+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.b scrub starts 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.068354+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.b scrub starts 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.069883+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.b scrub ok 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.069883+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.b scrub ok 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.373686+0000 osd.2 (osd.2) 283 : cluster [DBG] 5.a scrub starts 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.373686+0000 osd.2 (osd.2) 283 : cluster [DBG] 5.a scrub starts 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.375366+0000 osd.2 (osd.2) 284 : cluster [DBG] 5.a scrub ok 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:28.375366+0000 osd.2 (osd.2) 284 : cluster [DBG] 5.a scrub ok 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.420078+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.420078+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.420305+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.420305+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.421503+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.856 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.421503+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.421838+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: audit 2024-04-26T16:31:28.421838+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.172592+0000 osd.6 (osd.6) 63 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.172592+0000 osd.6 (osd.6) 63 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.174762+0000 osd.6 (osd.6) 64 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.174762+0000 osd.6 (osd.6) 64 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:31:29.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.743276+0000 mgr.y (mgr.24401) 547 : cluster [DBG] pgmap v847: 228 pgs: 228 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:27.743276+0000 mgr.y (mgr.24401) 547 : cluster [DBG] pgmap v847: 228 pgs: 228 active+clean; 8.4 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.068354+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.b scrub starts 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.068354+0000 osd.3 (osd.3) 349 : cluster [DBG] 6.b scrub starts 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.069883+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.b scrub ok 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.069883+0000 osd.3 (osd.3) 350 : cluster [DBG] 6.b scrub ok 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.373686+0000 osd.2 (osd.2) 283 : cluster [DBG] 5.a scrub starts 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.373686+0000 osd.2 (osd.2) 283 : cluster [DBG] 5.a scrub starts 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.375366+0000 osd.2 (osd.2) 284 : cluster [DBG] 5.a scrub ok 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:28.375366+0000 osd.2 (osd.2) 284 : cluster [DBG] 5.a scrub ok 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.420078+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.420078+0000 mon.a (mon.0) 2990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.420305+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.420305+0000 mon.b (mon.1) 385 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.421503+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.421503+0000 mon.a (mon.0) 2991 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.421838+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: audit 2024-04-26T16:31:28.421838+0000 mon.b (mon.1) 386 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-85"}]: dispatch 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.172592+0000 osd.6 (osd.6) 63 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.172592+0000 osd.6 (osd.6) 63 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.174762+0000 osd.6 (osd.6) 64 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:31:29.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:29 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.174762+0000 osd.6 (osd.6) 64 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:31:29.860 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:31:29.860 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:31:29.860 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:31:29.860 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:31:29.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:29.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:29.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:30.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:30.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.134 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:30.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:30.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:30.264 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:30.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:30.376 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:30.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:30.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:30.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:30.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 -- 172.21.15.50:0/2200631353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f810fe50 msgr2=0x7fd7f8112240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:30.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 --2- 172.21.15.50:0/2200631353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f810fe50 0x7fd7f8112240 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009990 tx=0x7fd7f4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 -- 172.21.15.50:0/2200631353 shutdown_connections 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 --2- 172.21.15.50:0/2200631353 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8075650 0x7fd7f8075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 --2- 172.21.15.50:0/2200631353 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f810fe50 0x7fd7f8112240 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 --2- 172.21.15.50:0/2200631353 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f8075f70 0x7fd7f810f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 -- 172.21.15.50:0/2200631353 >> 172.21.15.50:0/2200631353 conn(0x7fd7f807a5d0 msgr2=0x7fd7f807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 -- 172.21.15.50:0/2200631353 shutdown_connections 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 -- 172.21.15.50:0/2200631353 wait complete. 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 librados: starting msgr at 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 librados: starting objecter 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.590+0000 7fd80044e700 1 Processor -- start 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- start start 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 librados: setting wanted keys 2024-04-26T16:31:30.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 librados: calling monclient init 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 0x7fd7f819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8075f70 0x7fd7f819b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f810fe50 0x7fd7f819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8114dc0 con 0x7fd7f8075650 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8114c40 con 0x7fd7f8075f70 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd7f8114f40 con 0x7fd7f810fe50 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 0x7fd7f819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 0x7fd7f819b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58514/0 (socket says 172.21.15.50:58514) 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 -- 172.21.15.50:0/4209075856 learned_addr learned my addr 172.21.15.50:0/4209075856 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe9eb700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f810fe50 0x7fd7f819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:30.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 -- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f810fe50 msgr2=0x7fd7f819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f810fe50 0x7fd7f819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 -- 172.21.15.50:0/4209075856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8075f70 msgr2=0x7fd7f819b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8075f70 0x7fd7f819b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 -- 172.21.15.50:0/4209075856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f81a0310 con 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fe1ea700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 0x7fd7f819b300 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000ada0 tx=0x7fd7f000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 10 client.?.objecter ms_handle_connect 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0017020 con 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f81a0600 con 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd7f000cde0 con 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7f0005300 con 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 10 client.30759.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f81a7e80 con 0x7fd7f8075650 2024-04-26T16:31:30.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 librados: init done 2024-04-26T16:31:30.593 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f806a110 con 0x7fd7f8075650 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd7f0004ad0 con 0x7fd7f8075650 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd7e4071f80 0x7fd7e4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(556..556 src has 496..556) ==== 9364+0+0 (secure 0 0 0) 0x7fd7f00911f0 con 0x7fd7f8075650 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 10 client.30759.objecter ms_dispatch 0x7fd7f8001550 osd_map(556..556 src has 496..556) 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 3 client.30759.objecter handle_osd_map got epochs [556,556] > 0 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 3 client.30759.objecter handle_osd_map decoding full epoch 556 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7fd9e9700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd7e4071f80 0x7fd7e4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.594+0000 7fd7ef7fe700 20 client.30759.objecter dump_active .. 0 homeless 2024-04-26T16:31:30.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.598+0000 7fd7ef7fe700 10 client.30759.objecter ms_handle_connect 0x7fd7e4071f80 2024-04-26T16:31:30.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.598+0000 7fd7fd9e9700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd7e4071f80 0x7fd7e4074440 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8103060 tx=0x7fd7e8009580 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:30.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.602+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd7f0021060 con 0x7fd7f8075650 2024-04-26T16:31:30.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:30.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:30.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:30 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.422460+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e555: 8 total, 8 up, 7 in 2024-04-26T16:31:30.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:30 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.422460+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e555: 8 total, 8 up, 7 in 2024-04-26T16:31:30.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-26T16:31:30.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:30.786 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:30.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:30.838+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fd7f802e490 con 0x7fd7f8075650 2024-04-26T16:31:30.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:30 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.422460+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e555: 8 total, 8 up, 7 in 2024-04-26T16:31:30.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:30 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.422460+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e555: 8 total, 8 up, 7 in 2024-04-26T16:31:30.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:30 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.422460+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e555: 8 total, 8 up, 7 in 2024-04-26T16:31:30.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:30 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.422460+0000 mon.a (mon.0) 2992 : cluster [DBG] osdmap e555: 8 total, 8 up, 7 in 2024-04-26T16:31:30.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:30.913 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:30.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:31.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:31.042 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:31.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:31.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:31.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:31.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:31.304 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:31.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:31.434 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:31.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.438+0000 7fd7ef7fe700 1 -- 172.21.15.50:0/4209075856 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v557) ==== 101+0+0 (secure 0 0 0) 0x7fd7f005cc70 con 0x7fd7f8075650 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 10 librados: watch_flush enter 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 10 librados: watch_flush exit 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 20 client.30759.objecter shutdown clearing up homeless session... 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 10 client.30759.objecter successfully canceled tick 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd7e4071f80 msgr2=0x7fd7e4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd7e4071f80 0x7fd7e4074440 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8103060 tx=0x7fd7e8009580 comp rx=0 tx=0).stop 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 msgr2=0x7fd7f819b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 0x7fd7f819b300 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000ada0 tx=0x7fd7f000c330 comp rx=0 tx=0).stop 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 shutdown_connections 2024-04-26T16:31:31.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd7e4071f80 0x7fd7e4074440 unknown :-1 s=CLOSED pgs=3774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:31.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd7f810fe50 0x7fd7f819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:31.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7f8075650 0x7fd7f819b300 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:31.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 --2- 172.21.15.50:0/4209075856 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7f8075f70 0x7fd7f819b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:31.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 >> 172.21.15.50:0/4209075856 conn(0x7fd7f807a5d0 msgr2=0x7fd7f8100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:31.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.446+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 shutdown_connections 2024-04-26T16:31:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.450+0000 7fd80044e700 1 -- 172.21.15.50:0/4209075856 wait complete. 2024-04-26T16:31:31.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:31.450+0000 7fd80044e700 1 librados: shutdown 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18012 ms) 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22635 ms) 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3046 ms) 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3016 ms) 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2331 ms) 2024-04-26T16:31:31.452 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3057 ms) 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4070 ms) 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (37873 ms) 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17207 ms) 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4048 ms) 2024-04-26T16:31:31.453 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (2759 ms) 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25020 ms) 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15009 ms) 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5006 ms) 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.454 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3984 ms) 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: waiting for scrubs... 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: done waiting 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25083 ms) 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9964 ms) 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9062 ms) 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9088 ms) 2024-04-26T16:31:31.455 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-04-26T16:31:31.457 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:31:31.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:31.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:31.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.744139+0000 mgr.y (mgr.24401) 548 : cluster [DBG] pgmap v850: 196 pgs: 196 active+clean; 8.4 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:31.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:29.744139+0000 mgr.y (mgr.24401) 548 : cluster [DBG] pgmap v850: 196 pgs: 196 active+clean; 8.4 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:31.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.016842+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:31:31.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.016842+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:31:31.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.019662+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:31:31.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.019662+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.431898+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e556: 8 total, 8 up, 7 in 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.431898+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e556: 8 total, 8 up, 7 in 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: audit 2024-04-26T16:31:30.440847+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: audit 2024-04-26T16:31:30.440847+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: audit 2024-04-26T16:31:30.444945+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: audit 2024-04-26T16:31:30.444945+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.773708+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.773708+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.777292+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:30.777292+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: audit 2024-04-26T16:31:30.840216+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: audit 2024-04-26T16:31:30.840216+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:31:31.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.241299+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:31.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:31 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.241299+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:31.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:31.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:31.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:31.818 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.744139+0000 mgr.y (mgr.24401) 548 : cluster [DBG] pgmap v850: 196 pgs: 196 active+clean; 8.4 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:29.744139+0000 mgr.y (mgr.24401) 548 : cluster [DBG] pgmap v850: 196 pgs: 196 active+clean; 8.4 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.016842+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.016842+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.019662+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.019662+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.431898+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e556: 8 total, 8 up, 7 in 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.431898+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e556: 8 total, 8 up, 7 in 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: audit 2024-04-26T16:31:30.440847+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: audit 2024-04-26T16:31:30.440847+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: audit 2024-04-26T16:31:30.444945+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: audit 2024-04-26T16:31:30.444945+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.773708+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.773708+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.777292+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:30.777292+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: audit 2024-04-26T16:31:30.840216+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: audit 2024-04-26T16:31:30.840216+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.241299+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.241299+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.744139+0000 mgr.y (mgr.24401) 548 : cluster [DBG] pgmap v850: 196 pgs: 196 active+clean; 8.4 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:29.744139+0000 mgr.y (mgr.24401) 548 : cluster [DBG] pgmap v850: 196 pgs: 196 active+clean; 8.4 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.016842+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.016842+0000 osd.7 (osd.7) 313 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.019662+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.019662+0000 osd.7 (osd.7) 314 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.431898+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e556: 8 total, 8 up, 7 in 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.431898+0000 mon.a (mon.0) 2993 : cluster [DBG] osdmap e556: 8 total, 8 up, 7 in 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: audit 2024-04-26T16:31:30.440847+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: audit 2024-04-26T16:31:30.440847+0000 mon.b (mon.1) 387 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: audit 2024-04-26T16:31:30.444945+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: audit 2024-04-26T16:31:30.444945+0000 mon.a (mon.0) 2994 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.773708+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.773708+0000 osd.0 (osd.0) 327 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.777292+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:31:31.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:30.777292+0000 osd.0 (osd.0) 328 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:31:31.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: audit 2024-04-26T16:31:30.840216+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:31:31.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: audit 2024-04-26T16:31:30.840216+0000 mon.a (mon.0) 2995 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:31:31.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.241299+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:31.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:31 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.241299+0000 mon.a (mon.0) 2996 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:31.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:31.946 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:31.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:32.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:32.075 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:32.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:32.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:32.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:32.336 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:32.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.430105+0000 mon.a (mon.0) 2997 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.430105+0000 mon.a (mon.0) 2997 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: audit 2024-04-26T16:31:31.441573+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: audit 2024-04-26T16:31:31.441573+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: audit 2024-04-26T16:31:31.441835+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: audit 2024-04-26T16:31:31.441835+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.448264+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e557: 8 total, 8 up, 7 in 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.448264+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e557: 8 total, 8 up, 7 in 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: audit 2024-04-26T16:31:31.453577+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:32.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: audit 2024-04-26T16:31:31.453577+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.762139+0000 osd.0 (osd.0) 329 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.762139+0000 osd.0 (osd.0) 329 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.764551+0000 osd.0 (osd.0) 330 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.764551+0000 osd.0 (osd.0) 330 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.430105+0000 mon.a (mon.0) 2997 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.430105+0000 mon.a (mon.0) 2997 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: audit 2024-04-26T16:31:31.441573+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: audit 2024-04-26T16:31:31.441573+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: audit 2024-04-26T16:31:31.441835+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: audit 2024-04-26T16:31:31.441835+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:31:32.457 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.448264+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e557: 8 total, 8 up, 7 in 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.448264+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e557: 8 total, 8 up, 7 in 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: audit 2024-04-26T16:31:31.453577+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: audit 2024-04-26T16:31:31.453577+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.762139+0000 osd.0 (osd.0) 329 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.762139+0000 osd.0 (osd.0) 329 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.764551+0000 osd.0 (osd.0) 330 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:31:32.458 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:32 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.764551+0000 osd.0 (osd.0) 330 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:31:32.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.430105+0000 mon.a (mon.0) 2997 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:31:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.430105+0000 mon.a (mon.0) 2997 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:31:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: audit 2024-04-26T16:31:31.441573+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: audit 2024-04-26T16:31:31.441573+0000 mon.a (mon.0) 2998 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: audit 2024-04-26T16:31:31.441835+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:31:32.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: audit 2024-04-26T16:31:31.441835+0000 mon.a (mon.0) 2999 : audit [INF] from='client.? 172.21.15.50:0/4209075856' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.448264+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e557: 8 total, 8 up, 7 in 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.448264+0000 mon.a (mon.0) 3000 : cluster [DBG] osdmap e557: 8 total, 8 up, 7 in 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: audit 2024-04-26T16:31:31.453577+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: audit 2024-04-26T16:31:31.453577+0000 mon.b (mon.1) 388 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.762139+0000 osd.0 (osd.0) 329 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.762139+0000 osd.0 (osd.0) 329 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.764551+0000 osd.0 (osd.0) 330 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:31:32.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:32 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.764551+0000 osd.0 (osd.0) 330 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:31:32.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:32.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:32.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:32.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:32.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:32.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:32.998 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:33.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:33.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:33.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:33.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:33.255 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:33.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:33.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.746291+0000 mgr.y (mgr.24401) 549 : cluster [DBG] pgmap v853: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: cluster 2024-04-26T16:31:31.746291+0000 mgr.y (mgr.24401) 549 : cluster [DBG] pgmap v853: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: cluster 2024-04-26T16:31:32.457740+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e558: 8 total, 8 up, 7 in 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: cluster 2024-04-26T16:31:32.457740+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e558: 8 total, 8 up, 7 in 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: audit 2024-04-26T16:31:32.461112+0000 mon.b (mon.1) 389 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: audit 2024-04-26T16:31:32.461112+0000 mon.b (mon.1) 389 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: audit 2024-04-26T16:31:32.461841+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.484 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[29234]: audit 2024-04-26T16:31:32.461841+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.746291+0000 mgr.y (mgr.24401) 549 : cluster [DBG] pgmap v853: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: cluster 2024-04-26T16:31:31.746291+0000 mgr.y (mgr.24401) 549 : cluster [DBG] pgmap v853: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: cluster 2024-04-26T16:31:32.457740+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e558: 8 total, 8 up, 7 in 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: cluster 2024-04-26T16:31:32.457740+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e558: 8 total, 8 up, 7 in 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: audit 2024-04-26T16:31:32.461112+0000 mon.b (mon.1) 389 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: audit 2024-04-26T16:31:32.461112+0000 mon.b (mon.1) 389 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: audit 2024-04-26T16:31:32.461841+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:33 smithi050 bash[21463]: audit 2024-04-26T16:31:32.461841+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:33.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.746291+0000 mgr.y (mgr.24401) 549 : cluster [DBG] pgmap v853: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-26T16:31:33.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: cluster 2024-04-26T16:31:31.746291+0000 mgr.y (mgr.24401) 549 : cluster [DBG] pgmap v853: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-04-26T16:31:33.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: cluster 2024-04-26T16:31:32.457740+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e558: 8 total, 8 up, 7 in 2024-04-26T16:31:33.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: cluster 2024-04-26T16:31:32.457740+0000 mon.a (mon.0) 3001 : cluster [DBG] osdmap e558: 8 total, 8 up, 7 in 2024-04-26T16:31:33.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: audit 2024-04-26T16:31:32.461112+0000 mon.b (mon.1) 389 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.498 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: audit 2024-04-26T16:31:32.461112+0000 mon.b (mon.1) 389 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: audit 2024-04-26T16:31:32.461841+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:33 smithi190 bash[23868]: audit 2024-04-26T16:31:32.461841+0000 mon.a (mon.0) 3002 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:33.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:33.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:33.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:33.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:33.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:33.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:33.904 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:33.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:34.007 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:34.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:34.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:34.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:34.167 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:34.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:34.297 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:34.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:34.428 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:34.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:34.557 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:34.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:31:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:31:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.047139+0000 osd.1 (osd.1) 305 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:31:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.047139+0000 osd.1 (osd.1) 305 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:31:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.048636+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:31:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.048636+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:31:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: audit 2024-04-26T16:31:33.449474+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:34.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: audit 2024-04-26T16:31:33.449474+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.454223+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e559: 8 total, 8 up, 7 in 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.454223+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e559: 8 total, 8 up, 7 in 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: audit 2024-04-26T16:31:33.473473+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: audit 2024-04-26T16:31:33.473473+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: audit 2024-04-26T16:31:33.473844+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: audit 2024-04-26T16:31:33.473844+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.495266+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.495266+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.497391+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.497391+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:31:34.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.047139+0000 osd.1 (osd.1) 305 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.047139+0000 osd.1 (osd.1) 305 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.048636+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.048636+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: audit 2024-04-26T16:31:33.449474+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: audit 2024-04-26T16:31:33.449474+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.454223+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e559: 8 total, 8 up, 7 in 2024-04-26T16:31:34.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.454223+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e559: 8 total, 8 up, 7 in 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: audit 2024-04-26T16:31:33.473473+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: audit 2024-04-26T16:31:33.473473+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: audit 2024-04-26T16:31:33.473844+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: audit 2024-04-26T16:31:33.473844+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.495266+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.495266+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:31:34.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.497391+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:31:34.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:34 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.497391+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:31:34.656 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:34.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:34.685 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:34.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:34.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.047139+0000 osd.1 (osd.1) 305 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.047139+0000 osd.1 (osd.1) 305 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.048636+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.048636+0000 osd.1 (osd.1) 306 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: audit 2024-04-26T16:31:33.449474+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: audit 2024-04-26T16:31:33.449474+0000 mon.a (mon.0) 3003 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:34.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.454223+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e559: 8 total, 8 up, 7 in 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.454223+0000 mon.a (mon.0) 3004 : cluster [DBG] osdmap e559: 8 total, 8 up, 7 in 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: audit 2024-04-26T16:31:33.473473+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: audit 2024-04-26T16:31:33.473473+0000 mon.b (mon.1) 390 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: audit 2024-04-26T16:31:33.473844+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: audit 2024-04-26T16:31:33.473844+0000 mon.a (mon.0) 3005 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]: dispatch 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.495266+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.495266+0000 osd.2 (osd.2) 285 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.497391+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:31:34.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:34 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.497391+0000 osd.2 (osd.2) 286 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:31:34.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:34.947 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:34.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:35.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:35.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:35.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:35.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:35.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:31:35.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:35.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:35.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:35.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:35.495 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:31:35.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.747192+0000 mgr.y (mgr.24401) 550 : cluster [DBG] pgmap v856: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:33.747192+0000 mgr.y (mgr.24401) 550 : cluster [DBG] pgmap v856: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.088756+0000 osd.3 (osd.3) 351 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.088756+0000 osd.3 (osd.3) 351 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.090688+0000 osd.3 (osd.3) 352 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.090688+0000 osd.3 (osd.3) 352 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:31:35.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: audit 2024-04-26T16:31:34.464886+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]': finished 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: audit 2024-04-26T16:31:34.464886+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]': finished 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.469308+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e560: 8 total, 8 up, 7 in 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.469308+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e560: 8 total, 8 up, 7 in 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: audit 2024-04-26T16:31:34.483395+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: audit 2024-04-26T16:31:34.483395+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: audit 2024-04-26T16:31:34.483528+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: audit 2024-04-26T16:31:34.483528+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.506048+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.506048+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.507824+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.507824+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-04-26T16:31:35.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.853809+0000 osd.0 (osd.0) 331 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.853809+0000 osd.0 (osd.0) 331 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.857298+0000 osd.0 (osd.0) 332 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[29234]: cluster 2024-04-26T16:31:34.857298+0000 osd.0 (osd.0) 332 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.747192+0000 mgr.y (mgr.24401) 550 : cluster [DBG] pgmap v856: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:33.747192+0000 mgr.y (mgr.24401) 550 : cluster [DBG] pgmap v856: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.088756+0000 osd.3 (osd.3) 351 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.088756+0000 osd.3 (osd.3) 351 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:31:35.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.090688+0000 osd.3 (osd.3) 352 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.090688+0000 osd.3 (osd.3) 352 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: audit 2024-04-26T16:31:34.464886+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]': finished 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: audit 2024-04-26T16:31:34.464886+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]': finished 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.469308+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e560: 8 total, 8 up, 7 in 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.469308+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e560: 8 total, 8 up, 7 in 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: audit 2024-04-26T16:31:34.483395+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: audit 2024-04-26T16:31:34.483395+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: audit 2024-04-26T16:31:34.483528+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: audit 2024-04-26T16:31:34.483528+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.506048+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-04-26T16:31:35.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.506048+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-04-26T16:31:35.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.507824+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-04-26T16:31:35.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.507824+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-04-26T16:31:35.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.853809+0000 osd.0 (osd.0) 331 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:31:35.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.853809+0000 osd.0 (osd.0) 331 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:31:35.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.857298+0000 osd.0 (osd.0) 332 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:31:35.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:35 smithi050 bash[21463]: cluster 2024-04-26T16:31:34.857298+0000 osd.0 (osd.0) 332 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:31:35.596 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:35.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:35.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:35.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:35.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.747192+0000 mgr.y (mgr.24401) 550 : cluster [DBG] pgmap v856: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T16:31:35.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:33.747192+0000 mgr.y (mgr.24401) 550 : cluster [DBG] pgmap v856: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-04-26T16:31:35.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.088756+0000 osd.3 (osd.3) 351 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.088756+0000 osd.3 (osd.3) 351 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.090688+0000 osd.3 (osd.3) 352 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.090688+0000 osd.3 (osd.3) 352 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: audit 2024-04-26T16:31:34.464886+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]': finished 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: audit 2024-04-26T16:31:34.464886+0000 mon.a (mon.0) 3006 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_tier","val": "test-rados-api-smithi050-73372-89-test-flush"}]': finished 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.469308+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e560: 8 total, 8 up, 7 in 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.469308+0000 mon.a (mon.0) 3007 : cluster [DBG] osdmap e560: 8 total, 8 up, 7 in 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: audit 2024-04-26T16:31:34.483395+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: audit 2024-04-26T16:31:34.483395+0000 mon.a (mon.0) 3008 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: audit 2024-04-26T16:31:34.483528+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: audit 2024-04-26T16:31:34.483528+0000 mon.b (mon.1) 391 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.506048+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-04-26T16:31:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.506048+0000 osd.2 (osd.2) 287 : cluster [DBG] 5.b scrub starts 2024-04-26T16:31:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.507824+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-04-26T16:31:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.507824+0000 osd.2 (osd.2) 288 : cluster [DBG] 5.b scrub ok 2024-04-26T16:31:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.853809+0000 osd.0 (osd.0) 331 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:31:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.853809+0000 osd.0 (osd.0) 331 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:31:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.857298+0000 osd.0 (osd.0) 332 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:31:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:35 smithi190 bash[23868]: cluster 2024-04-26T16:31:34.857298+0000 osd.0 (osd.0) 332 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:31:35.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:35.860 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:35.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:35.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:35.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:36.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:36.122 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:36.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:36.257 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:36.360 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:36.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:36.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:36.509 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/3214170506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806b130 msgr2=0x7fe13806b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:36.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/3214170506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806b130 0x7fe13806b510 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fe128009990 tx=0x7fe128009ca0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/3214170506 shutdown_connections 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/3214170506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe13810fed0 0x7fe1381122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/3214170506 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806b130 0x7fe13806b510 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/3214170506 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe13806ba50 0x7fe13810f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/3214170506 >> 172.21.15.50:0/3214170506 conn(0x7fe1380792d0 msgr2=0x7fe1380796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/3214170506 shutdown_connections 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/3214170506 wait complete. 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 librados: starting msgr at 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 librados: starting objecter 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 Processor -- start 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- start start 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 librados: setting wanted keys 2024-04-26T16:31:36.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 librados: calling monclient init 2024-04-26T16:31:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe13806b130 0x7fe13819b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 0x7fe13819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe13810fed0 0x7fe13819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe138114d30 con 0x7fe13806ba50 2024-04-26T16:31:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe138114bb0 con 0x7fe13806b130 2024-04-26T16:31:36.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe138114eb0 con 0x7fe13810fed0 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 0x7fe13819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13ca88700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe13810fed0 0x7fe13819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 0x7fe13819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58526/0 (socket says 172.21.15.50:58526) 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 -- 172.21.15.50:0/1136058106 learned_addr learned my addr 172.21.15.50:0/1136058106 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe137fff700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe13806b130 0x7fe13819b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 -- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe13810fed0 msgr2=0x7fe13819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe13810fed0 0x7fe13819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 -- 172.21.15.50:0/1136058106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe13806b130 msgr2=0x7fe13819b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe13806b130 0x7fe13819b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 -- 172.21.15.50:0/1136058106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1381a0350 con 0x7fe13806ba50 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1377fe700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 0x7fe13819b8d0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00d950 tx=0x7fe12c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:36.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1357fa700 10 client.?.objecter ms_handle_connect 0x7fe13806ba50 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1357fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c016070 con 0x7fe13806ba50 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe12c00fcf0 con 0x7fe13806ba50 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1381a0640 con 0x7fe13806ba50 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe12c010df0 con 0x7fe13806ba50 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 10 client.30765.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1381a7ec0 con 0x7fe13806ba50 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.569+0000 7fe13e4eb700 1 librados: init done 2024-04-26T16:31:36.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe13806a110 con 0x7fe13806ba50 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe12c019070 con 0x7fe13806ba50 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe120071f80 0x7fe120074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(562..562 src has 496..562) ==== 9827+0+0 (secure 0 0 0) 0x7fe12c091720 con 0x7fe13806ba50 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 10 client.30765.objecter ms_dispatch 0x7fe138001550 osd_map(562..562 src has 496..562) 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 3 client.30765.objecter handle_osd_map got epochs [562,562] > 0 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 3 client.30765.objecter handle_osd_map decoding full epoch 562 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe137fff700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe120071f80 0x7fe120074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 20 client.30765.objecter dump_active .. 0 homeless 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe137fff700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe120071f80 0x7fe120074440 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7fe1280098e0 tx=0x7fe128005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:36.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.573+0000 7fe1357fa700 10 client.30765.objecter ms_handle_connect 0x7fe120071f80 2024-04-26T16:31:36.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.577+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fe12c05cf30 con 0x7fe13806ba50 2024-04-26T16:31:36.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:36.622 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.019554+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.019554+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.019929+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.019929+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:35.476766+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:35.476766+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.480386+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e561: 8 total, 8 up, 7 in 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.480386+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e561: 8 total, 8 up, 7 in 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:35.485791+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:35.485791+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:35.495022+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:35.495022+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.242981+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.242981+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:36.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:36.247010+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: audit 2024-04-26T16:31:36.247010+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.254935+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e562: 8 total, 8 up, 7 in 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.254935+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e562: 8 total, 8 up, 7 in 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.019554+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.019554+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.019929+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.019929+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:35.476766+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:35.476766+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.480386+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e561: 8 total, 8 up, 7 in 2024-04-26T16:31:36.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.480386+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e561: 8 total, 8 up, 7 in 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:35.485791+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:35.485791+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:35.495022+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:35.495022+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.242981+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.242981+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:36.247010+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: audit 2024-04-26T16:31:36.247010+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.254935+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e562: 8 total, 8 up, 7 in 2024-04-26T16:31:36.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:36 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.254935+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e562: 8 total, 8 up, 7 in 2024-04-26T16:31:36.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:36.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:36.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.019554+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:36.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.019554+0000 osd.7 (osd.7) 315 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:36.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.019929+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:36.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.019929+0000 osd.7 (osd.7) 316 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:36.745 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:35.476766+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:35.476766+0000 mon.a (mon.0) 3009 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.480386+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e561: 8 total, 8 up, 7 in 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.480386+0000 mon.a (mon.0) 3010 : cluster [DBG] osdmap e561: 8 total, 8 up, 7 in 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:35.485791+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:35.485791+0000 mon.b (mon.1) 392 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:35.495022+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:35.495022+0000 mon.a (mon.0) 3011 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.242981+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.242981+0000 mon.a (mon.0) 3012 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:36.247010+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: audit 2024-04-26T16:31:36.247010+0000 mon.a (mon.0) 3013 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:31:36.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.254935+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e562: 8 total, 8 up, 7 in 2024-04-26T16:31:36.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:36 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.254935+0000 mon.a (mon.0) 3014 : cluster [DBG] osdmap e562: 8 total, 8 up, 7 in 2024-04-26T16:31:36.758 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:36.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.817+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe13819c6a0 con 0x7fe120071f80 2024-04-26T16:31:36.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.845+0000 7fe1357fa700 1 -- 172.21.15.50:0/1136058106 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+661743 (secure 0 0 0) 0x7fe13819c6a0 con 0x7fe120071f80 2024-04-26T16:31:36.850 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:31:36.857 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 10 librados: watch_flush enter 2024-04-26T16:31:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 10 librados: watch_flush exit 2024-04-26T16:31:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 20 client.30765.objecter shutdown clearing up homeless session... 2024-04-26T16:31:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 10 client.30765.objecter successfully canceled tick 2024-04-26T16:31:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe120071f80 msgr2=0x7fe120074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe120071f80 0x7fe120074440 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7fe1280098e0 tx=0x7fe128005960 comp rx=0 tx=0).stop 2024-04-26T16:31:36.858 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 msgr2=0x7fe13819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 0x7fe13819b8d0 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fe12c00d950 tx=0x7fe12c00dc60 comp rx=0 tx=0).stop 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 shutdown_connections 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe120071f80 0x7fe120074440 unknown :-1 s=CLOSED pgs=3775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe13806b130 0x7fe13819b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe13810fed0 0x7fe13819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 --2- 172.21.15.50:0/1136058106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe13806ba50 0x7fe13819b8d0 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 >> 172.21.15.50:0/1136058106 conn(0x7fe1380792d0 msgr2=0x7fe138111790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 shutdown_connections 2024-04-26T16:31:36.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 -- 172.21.15.50:0/1136058106 wait complete. 2024-04-26T16:31:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:36.849+0000 7fe13e4eb700 1 librados: shutdown 2024-04-26T16:31:36.860 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:31:36.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:36.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:36.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:36.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:37.008 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:37.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:37.139 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:37.141 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:31:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:37.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:37.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:37.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:37.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:37.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:37.542 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.747932+0000 mgr.y (mgr.24401) 551 : cluster [DBG] pgmap v859: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:35.747932+0000 mgr.y (mgr.24401) 551 : cluster [DBG] pgmap v859: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.075023+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.075023+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.079378+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.079378+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.804945+0000 osd.0 (osd.0) 333 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.804945+0000 osd.0 (osd.0) 333 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.807338+0000 osd.0 (osd.0) 334 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:36.807338+0000 osd.0 (osd.0) 334 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.275288+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e563: 8 total, 8 up, 7 in 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.275288+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e563: 8 total, 8 up, 7 in 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.331330+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.331330+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.331410+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.331410+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.332429+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.332429+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.332718+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[29234]: audit 2024-04-26T16:31:37.332718+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.747932+0000 mgr.y (mgr.24401) 551 : cluster [DBG] pgmap v859: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:35.747932+0000 mgr.y (mgr.24401) 551 : cluster [DBG] pgmap v859: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.075023+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.075023+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:31:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.079378+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.079378+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.804945+0000 osd.0 (osd.0) 333 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.804945+0000 osd.0 (osd.0) 333 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.807338+0000 osd.0 (osd.0) 334 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:36.807338+0000 osd.0 (osd.0) 334 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.275288+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e563: 8 total, 8 up, 7 in 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.275288+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e563: 8 total, 8 up, 7 in 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.331330+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.331330+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.331410+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.331410+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.332429+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.332429+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.332718+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:37 smithi050 bash[21463]: audit 2024-04-26T16:31:37.332718+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.546 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:37.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:37.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:37.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:37.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:37.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:37.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:37.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.747932+0000 mgr.y (mgr.24401) 551 : cluster [DBG] pgmap v859: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:37.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:35.747932+0000 mgr.y (mgr.24401) 551 : cluster [DBG] pgmap v859: 260 pgs: 9 creating+peering, 23 unknown, 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:37.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.075023+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:31:37.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.075023+0000 osd.1 (osd.1) 307 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.079378+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.079378+0000 osd.1 (osd.1) 308 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.804945+0000 osd.0 (osd.0) 333 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.804945+0000 osd.0 (osd.0) 333 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.807338+0000 osd.0 (osd.0) 334 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:36.807338+0000 osd.0 (osd.0) 334 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:31:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.275288+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e563: 8 total, 8 up, 7 in 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.275288+0000 mon.a (mon.0) 3015 : cluster [DBG] osdmap e563: 8 total, 8 up, 7 in 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.331330+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.331330+0000 mon.a (mon.0) 3016 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.331410+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.331410+0000 mon.b (mon.1) 393 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.332429+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.332429+0000 mon.a (mon.0) 3017 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.332718+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:37 smithi190 bash[23868]: audit 2024-04-26T16:31:37.332718+0000 mon.b (mon.1) 394 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-87"}]: dispatch 2024-04-26T16:31:37.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:37.937 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:37.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:38.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.066 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:38.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:38.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:38.302 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:38.325 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:38.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.449 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:38.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:38.557 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:31:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:31:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:31:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:31:38.558 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:31:38.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:38.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.686 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":860,"stamp":"2024-04-26T16:31:36.248827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8811253,"num_objects":390,"num_object_clones":10,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":2414,"num_read_kb":2104,"num_write":1679,"num_write_kb":15269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":5967718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3119,"ondisk_log_size":3119,"up":711,"acting":711,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":655,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":546244,"kb_used_data":43832,"kb_used_omap":24,"kb_used_meta":501159,"kb_avail":655669820,"statfs":{"total":671965249536,"available":671405895680,"internally_reserved":0,"allocated":44883968,"data_stored":31548262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25087,"internal_metadata":513187329},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.799842"},"pg_stats":[{"pgid":"310.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.1f","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.463117+0000","last_change":"2024-04-26T16:31:31.462710+0000","last_active":"2024-04-26T16:31:31.463117+0000","last_peered":"2024-04-26T16:31:31.463117+0000","last_clean":"2024-04-26T16:31:31.463117+0000","last_became_active":"2024-04-26T16:31:31.460459+0000","last_became_peered":"2024-04-26T16:31:31.460459+0000","last_unstale":"2024-04-26T16:31:31.463117+0000","last_undegraded":"2024-04-26T16:31:31.463117+0000","last_fullsized":"2024-04-26T16:31:31.463117+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:58.673578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"310.1d","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.453359+0000","last_change":"2024-04-26T16:31:32.453359+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.453359+0000","last_undegraded":"2024-04-26T16:31:32.453359+0000","last_fullsized":"2024-04-26T16:31:32.453359+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.1e","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.464704+0000","last_change":"2024-04-26T16:31:31.458695+0000","last_active":"2024-04-26T16:31:31.464704+0000","last_peered":"2024-04-26T16:31:31.464704+0000","last_clean":"2024-04-26T16:31:31.464704+0000","last_became_active":"2024-04-26T16:31:31.458351+0000","last_became_peered":"2024-04-26T16:31:31.458351+0000","last_unstale":"2024-04-26T16:31:31.464704+0000","last_undegraded":"2024-04-26T16:31:31.464704+0000","last_fullsized":"2024-04-26T16:31:31.464704+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.737507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"310.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.1d","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.477411+0000","last_change":"2024-04-26T16:31:31.476352+0000","last_active":"2024-04-26T16:31:31.477411+0000","last_peered":"2024-04-26T16:31:31.477411+0000","last_clean":"2024-04-26T16:31:31.477411+0000","last_became_active":"2024-04-26T16:31:31.475914+0000","last_became_peered":"2024-04-26T16:31:31.475914+0000","last_unstale":"2024-04-26T16:31:31.477411+0000","last_undegraded":"2024-04-26T16:31:31.477411+0000","last_fullsized":"2024-04-26T16:31:31.477411+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:57.222386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"310.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.1c","version":"0'0","reported_seq":13,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.458852+0000","last_change":"2024-04-26T16:31:31.466058+0000","last_active":"2024-04-26T16:31:32.458852+0000","last_peered":"2024-04-26T16:31:32.458852+0000","last_clean":"2024-04-26T16:31:32.458852+0000","last_became_active":"2024-04-26T16:31:31.465740+0000","last_became_peered":"2024-04-26T16:31:31.465740+0000","last_unstale":"2024-04-26T16:31:32.458852+0000","last_undegraded":"2024-04-26T16:31:32.458852+0000","last_fullsized":"2024-04-26T16:31:32.458852+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.531176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"310.18","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.452004+0000","last_change":"2024-04-26T16:31:32.452004+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.452004+0000","last_undegraded":"2024-04-26T16:31:32.452004+0000","last_fullsized":"2024-04-26T16:31:32.452004+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.1b","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.478429+0000","last_change":"2024-04-26T16:31:31.476735+0000","last_active":"2024-04-26T16:31:31.478429+0000","last_peered":"2024-04-26T16:31:31.478429+0000","last_clean":"2024-04-26T16:31:31.478429+0000","last_became_active":"2024-04-26T16:31:31.476326+0000","last_became_peered":"2024-04-26T16:31:31.476326+0000","last_unstale":"2024-04-26T16:31:31.478429+0000","last_undegraded":"2024-04-26T16:31:31.478429+0000","last_fullsized":"2024-04-26T16:31:31.478429+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:44.512322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"310.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.1a","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.472683+0000","last_change":"2024-04-26T16:31:31.472277+0000","last_active":"2024-04-26T16:31:31.472683+0000","last_peered":"2024-04-26T16:31:31.472683+0000","last_clean":"2024-04-26T16:31:31.472683+0000","last_became_active":"2024-04-26T16:31:31.469698+0000","last_became_peered":"2024-04-26T16:31:31.469698+0000","last_unstale":"2024-04-26T16:31:31.472683+0000","last_undegraded":"2024-04-26T16:31:31.472683+0000","last_fullsized":"2024-04-26T16:31:31.472683+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:54.379802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"310.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.19","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.461392+0000","last_change":"2024-04-26T16:31:31.460554+0000","last_active":"2024-04-26T16:31:31.461392+0000","last_peered":"2024-04-26T16:31:31.461392+0000","last_clean":"2024-04-26T16:31:31.461392+0000","last_became_active":"2024-04-26T16:31:31.460198+0000","last_became_peered":"2024-04-26T16:31:31.460198+0000","last_unstale":"2024-04-26T16:31:31.461392+0000","last_undegraded":"2024-04-26T16:31:31.461392+0000","last_fullsized":"2024-04-26T16:31:31.461392+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:33.115509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"310.1b","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.452069+0000","last_change":"2024-04-26T16:31:32.452069+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.452069+0000","last_undegraded":"2024-04-26T16:31:32.452069+0000","last_fullsized":"2024-04-26T16:31:32.452069+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.18","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.472085+0000","last_change":"2024-04-26T16:31:31.469380+0000","last_active":"2024-04-26T16:31:31.472085+0000","last_peered":"2024-04-26T16:31:31.472085+0000","last_clean":"2024-04-26T16:31:31.472085+0000","last_became_active":"2024-04-26T16:31:31.468984+0000","last_became_peered":"2024-04-26T16:31:31.468984+0000","last_unstale":"2024-04-26T16:31:31.472085+0000","last_undegraded":"2024-04-26T16:31:31.472085+0000","last_fullsized":"2024-04-26T16:31:31.472085+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:59.929567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"310.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.17","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.470287+0000","last_change":"2024-04-26T16:31:31.466208+0000","last_active":"2024-04-26T16:31:31.470287+0000","last_peered":"2024-04-26T16:31:31.470287+0000","last_clean":"2024-04-26T16:31:31.470287+0000","last_became_active":"2024-04-26T16:31:31.465703+0000","last_became_peered":"2024-04-26T16:31:31.465703+0000","last_unstale":"2024-04-26T16:31:31.470287+0000","last_undegraded":"2024-04-26T16:31:31.470287+0000","last_fullsized":"2024-04-26T16:31:31.470287+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:49.721957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"310.15","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.452002+0000","last_change":"2024-04-26T16:31:32.452002+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.452002+0000","last_undegraded":"2024-04-26T16:31:32.452002+0000","last_fullsized":"2024-04-26T16:31:32.452002+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.16","version":"0'0","reported_seq":13,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459042+0000","last_change":"2024-04-26T16:31:31.478578+0000","last_active":"2024-04-26T16:31:32.459042+0000","last_peered":"2024-04-26T16:31:32.459042+0000","last_clean":"2024-04-26T16:31:32.459042+0000","last_became_active":"2024-04-26T16:31:31.478305+0000","last_became_peered":"2024-04-26T16:31:31.478305+0000","last_unstale":"2024-04-26T16:31:32.459042+0000","last_undegraded":"2024-04-26T16:31:32.459042+0000","last_fullsized":"2024-04-26T16:31:32.459042+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.077728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"310.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.15","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.477635+0000","last_change":"2024-04-26T16:31:31.476763+0000","last_active":"2024-04-26T16:31:31.477635+0000","last_peered":"2024-04-26T16:31:31.477635+0000","last_clean":"2024-04-26T16:31:31.477635+0000","last_became_active":"2024-04-26T16:31:31.476441+0000","last_became_peered":"2024-04-26T16:31:31.476441+0000","last_unstale":"2024-04-26T16:31:31.477635+0000","last_undegraded":"2024-04-26T16:31:31.477635+0000","last_fullsized":"2024-04-26T16:31:31.477635+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:37.065727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"310.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.14","version":"0'0","reported_seq":13,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.460979+0000","last_change":"2024-04-26T16:31:31.476082+0000","last_active":"2024-04-26T16:31:32.460979+0000","last_peered":"2024-04-26T16:31:32.460979+0000","last_clean":"2024-04-26T16:31:32.460979+0000","last_became_active":"2024-04-26T16:31:31.475808+0000","last_became_peered":"2024-04-26T16:31:31.475808+0000","last_unstale":"2024-04-26T16:31:32.460979+0000","last_undegraded":"2024-04-26T16:31:32.460979+0000","last_fullsized":"2024-04-26T16:31:32.460979+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:35.637321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"310.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.13","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.460947+0000","last_change":"2024-04-26T16:31:31.460422+0000","last_active":"2024-04-26T16:31:31.460947+0000","last_peered":"2024-04-26T16:31:31.460947+0000","last_clean":"2024-04-26T16:31:31.460947+0000","last_became_active":"2024-04-26T16:31:31.460112+0000","last_became_peered":"2024-04-26T16:31:31.460112+0000","last_unstale":"2024-04-26T16:31:31.460947+0000","last_undegraded":"2024-04-26T16:31:31.460947+0000","last_fullsized":"2024-04-26T16:31:31.460947+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:00.380391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"310.11","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.453007+0000","last_change":"2024-04-26T16:31:32.453007+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.453007+0000","last_undegraded":"2024-04-26T16:31:32.453007+0000","last_fullsized":"2024-04-26T16:31:32.453007+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.12","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.461599+0000","last_change":"2024-04-26T16:31:31.459904+0000","last_active":"2024-04-26T16:31:31.461599+0000","last_peered":"2024-04-26T16:31:31.461599+0000","last_clean":"2024-04-26T16:31:31.461599+0000","last_became_active":"2024-04-26T16:31:31.459415+0000","last_became_peered":"2024-04-26T16:31:31.459415+0000","last_unstale":"2024-04-26T16:31:31.461599+0000","last_undegraded":"2024-04-26T16:31:31.461599+0000","last_fullsized":"2024-04-26T16:31:31.461599+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:54.598068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"310.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.11","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.480694+0000","last_change":"2024-04-26T16:31:31.480294+0000","last_active":"2024-04-26T16:31:31.480694+0000","last_peered":"2024-04-26T16:31:31.480694+0000","last_clean":"2024-04-26T16:31:31.480694+0000","last_became_active":"2024-04-26T16:31:31.479911+0000","last_became_peered":"2024-04-26T16:31:31.479911+0000","last_unstale":"2024-04-26T16:31:31.480694+0000","last_undegraded":"2024-04-26T16:31:31.480694+0000","last_fullsized":"2024-04-26T16:31:31.480694+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.612379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"310.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.10","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.464294+0000","last_change":"2024-04-26T16:31:31.463537+0000","last_active":"2024-04-26T16:31:31.464294+0000","last_peered":"2024-04-26T16:31:31.464294+0000","last_clean":"2024-04-26T16:31:31.464294+0000","last_became_active":"2024-04-26T16:31:31.463223+0000","last_became_peered":"2024-04-26T16:31:31.463223+0000","last_unstale":"2024-04-26T16:31:31.464294+0000","last_undegraded":"2024-04-26T16:31:31.464294+0000","last_fullsized":"2024-04-26T16:31:31.464294+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:46.606986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"310.c","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.453393+0000","last_change":"2024-04-26T16:31:32.453393+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.453393+0000","last_undegraded":"2024-04-26T16:31:32.453393+0000","last_fullsized":"2024-04-26T16:31:32.453393+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.f","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.479553+0000","last_change":"2024-04-26T16:31:31.477498+0000","last_active":"2024-04-26T16:31:31.479553+0000","last_peered":"2024-04-26T16:31:31.479553+0000","last_clean":"2024-04-26T16:31:31.479553+0000","last_became_active":"2024-04-26T16:31:31.477150+0000","last_became_peered":"2024-04-26T16:31:31.477150+0000","last_unstale":"2024-04-26T16:31:31.479553+0000","last_undegraded":"2024-04-26T16:31:31.479553+0000","last_fullsized":"2024-04-26T16:31:31.479553+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:49.769469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"310.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.e","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.479028+0000","last_change":"2024-04-26T16:31:31.477838+0000","last_active":"2024-04-26T16:31:31.479028+0000","last_peered":"2024-04-26T16:31:31.479028+0000","last_clean":"2024-04-26T16:31:31.479028+0000","last_became_active":"2024-04-26T16:31:31.477563+0000","last_became_peered":"2024-04-26T16:31:31.477563+0000","last_unstale":"2024-04-26T16:31:31.479028+0000","last_undegraded":"2024-04-26T16:31:31.479028+0000","last_fullsized":"2024-04-26T16:31:31.479028+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:56.326036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"310.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.d","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.469610+0000","last_change":"2024-04-26T16:31:31.469329+0000","last_active":"2024-04-26T16:31:31.469610+0000","last_peered":"2024-04-26T16:31:31.469610+0000","last_clean":"2024-04-26T16:31:31.469610+0000","last_became_active":"2024-04-26T16:31:31.469031+0000","last_became_peered":"2024-04-26T16:31:31.469031+0000","last_unstale":"2024-04-26T16:31:31.469610+0000","last_undegraded":"2024-04-26T16:31:31.469610+0000","last_fullsized":"2024-04-26T16:31:31.469610+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:58.917758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"310.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.c","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.472780+0000","last_change":"2024-04-26T16:31:31.472491+0000","last_active":"2024-04-26T16:31:31.472780+0000","last_peered":"2024-04-26T16:31:31.472780+0000","last_clean":"2024-04-26T16:31:31.472780+0000","last_became_active":"2024-04-26T16:31:31.469788+0000","last_became_peered":"2024-04-26T16:31:31.469788+0000","last_unstale":"2024-04-26T16:31:31.472780+0000","last_undegraded":"2024-04-26T16:31:31.472780+0000","last_fullsized":"2024-04-26T16:31:31.472780+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:35.816305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"310.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.b","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.483199+0000","last_change":"2024-04-26T16:31:31.482647+0000","last_active":"2024-04-26T16:31:31.483199+0000","last_peered":"2024-04-26T16:31:31.483199+0000","last_clean":"2024-04-26T16:31:31.483199+0000","last_became_active":"2024-04-26T16:31:31.482096+0000","last_became_peered":"2024-04-26T16:31:31.482096+0000","last_unstale":"2024-04-26T16:31:31.483199+0000","last_undegraded":"2024-04-26T16:31:31.483199+0000","last_fullsized":"2024-04-26T16:31:31.483199+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:33.216133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"310.9","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.452157+0000","last_change":"2024-04-26T16:31:32.452157+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.452157+0000","last_undegraded":"2024-04-26T16:31:32.452157+0000","last_fullsized":"2024-04-26T16:31:32.452157+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.a","version":"0'0","reported_seq":13,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.457624+0000","last_change":"2024-04-26T16:31:31.468674+0000","last_active":"2024-04-26T16:31:32.457624+0000","last_peered":"2024-04-26T16:31:32.457624+0000","last_clean":"2024-04-26T16:31:32.457624+0000","last_became_active":"2024-04-26T16:31:31.468414+0000","last_became_peered":"2024-04-26T16:31:31.468414+0000","last_unstale":"2024-04-26T16:31:32.457624+0000","last_undegraded":"2024-04-26T16:31:32.457624+0000","last_fullsized":"2024-04-26T16:31:32.457624+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.458334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"310.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.9","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.466711+0000","last_change":"2024-04-26T16:31:31.466223+0000","last_active":"2024-04-26T16:31:31.466711+0000","last_peered":"2024-04-26T16:31:31.466711+0000","last_clean":"2024-04-26T16:31:31.466711+0000","last_became_active":"2024-04-26T16:31:31.465936+0000","last_became_peered":"2024-04-26T16:31:31.465936+0000","last_unstale":"2024-04-26T16:31:31.466711+0000","last_undegraded":"2024-04-26T16:31:31.466711+0000","last_fullsized":"2024-04-26T16:31:31.466711+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:40.496696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"310.b","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.453340+0000","last_change":"2024-04-26T16:31:32.453340+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.453340+0000","last_undegraded":"2024-04-26T16:31:32.453340+0000","last_fullsized":"2024-04-26T16:31:32.453340+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.8","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.461951+0000","last_change":"2024-04-26T16:31:31.461664+0000","last_active":"2024-04-26T16:31:31.461951+0000","last_peered":"2024-04-26T16:31:31.461951+0000","last_clean":"2024-04-26T16:31:31.461951+0000","last_became_active":"2024-04-26T16:31:31.461247+0000","last_became_peered":"2024-04-26T16:31:31.461247+0000","last_unstale":"2024-04-26T16:31:31.461951+0000","last_undegraded":"2024-04-26T16:31:31.461951+0000","last_fullsized":"2024-04-26T16:31:31.461951+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:45.699731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"310.4","version":"0'0","reported_seq":1,"reported_epoch":558,"state":"creating+peering","last_fresh":"2024-04-26T16:31:32.452290+0000","last_change":"2024-04-26T16:31:32.452290+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.452290+0000","last_undegraded":"2024-04-26T16:31:32.452290+0000","last_fullsized":"2024-04-26T16:31:32.452290+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":558,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"309.7","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.480401+0000","last_change":"2024-04-26T16:31:31.479351+0000","last_active":"2024-04-26T16:31:31.480401+0000","last_peered":"2024-04-26T16:31:31.480401+0000","last_clean":"2024-04-26T16:31:31.480401+0000","last_became_active":"2024-04-26T16:31:31.478975+0000","last_became_peered":"2024-04-26T16:31:31.478975+0000","last_unstale":"2024-04-26T16:31:31.480401+0000","last_undegraded":"2024-04-26T16:31:31.480401+0000","last_fullsized":"2024-04-26T16:31:31.480401+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:54.256348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"310.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.6","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.479561+0000","last_change":"2024-04-26T16:31:31.477907+0000","last_active":"2024-04-26T16:31:31.479561+0000","last_peered":"2024-04-26T16:31:31.479561+0000","last_clean":"2024-04-26T16:31:31.479561+0000","last_became_active":"2024-04-26T16:31:31.477509+0000","last_became_peered":"2024-04-26T16:31:31.477509+0000","last_unstale":"2024-04-26T16:31:31.479561+0000","last_undegraded":"2024-04-26T16:31:31.479561+0000","last_fullsized":"2024-04-26T16:31:31.479561+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:44.712864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"310.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.5","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.482176+0000","last_change":"2024-04-26T16:31:31.481657+0000","last_active":"2024-04-26T16:31:31.482176+0000","last_peered":"2024-04-26T16:31:31.482176+0000","last_clean":"2024-04-26T16:31:31.482176+0000","last_became_active":"2024-04-26T16:31:31.481350+0000","last_became_peered":"2024-04-26T16:31:31.481350+0000","last_unstale":"2024-04-26T16:31:31.482176+0000","last_undegraded":"2024-04-26T16:31:31.482176+0000","last_fullsized":"2024-04-26T16:31:31.482176+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:53.707521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"310.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.4","version":"0'0","reported_seq":13,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.462375+0000","last_change":"2024-04-26T16:31:31.461491+0000","last_active":"2024-04-26T16:31:32.462375+0000","last_peered":"2024-04-26T16:31:32.462375+0000","last_clean":"2024-04-26T16:31:32.462375+0000","last_became_active":"2024-04-26T16:31:31.461027+0000","last_became_peered":"2024-04-26T16:31:31.461027+0000","last_unstale":"2024-04-26T16:31:32.462375+0000","last_undegraded":"2024-04-26T16:31:32.462375+0000","last_fullsized":"2024-04-26T16:31:32.462375+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:34.368227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"310.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.3","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.457867+0000","last_change":"2024-04-26T16:31:31.457364+0000","last_active":"2024-04-26T16:31:31.457867+0000","last_peered":"2024-04-26T16:31:31.457867+0000","last_clean":"2024-04-26T16:31:31.457867+0000","last_became_active":"2024-04-26T16:31:31.456858+0000","last_became_peered":"2024-04-26T16:31:31.456858+0000","last_unstale":"2024-04-26T16:31:31.457867+0000","last_undegraded":"2024-04-26T16:31:31.457867+0000","last_fullsized":"2024-04-26T16:31:31.457867+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:56.904836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"310.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.2","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.465663+0000","last_change":"2024-04-26T16:31:31.465327+0000","last_active":"2024-04-26T16:31:31.465663+0000","last_peered":"2024-04-26T16:31:31.465663+0000","last_clean":"2024-04-26T16:31:31.465663+0000","last_became_active":"2024-04-26T16:31:31.464998+0000","last_became_peered":"2024-04-26T16:31:31.464998+0000","last_unstale":"2024-04-26T16:31:31.465663+0000","last_undegraded":"2024-04-26T16:31:31.465663+0000","last_fullsized":"2024-04-26T16:31:31.465663+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:34.980532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"310.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.1","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.466678+0000","last_change":"2024-04-26T16:31:31.466165+0000","last_active":"2024-04-26T16:31:31.466678+0000","last_peered":"2024-04-26T16:31:31.466678+0000","last_clean":"2024-04-26T16:31:31.466678+0000","last_became_active":"2024-04-26T16:31:31.465738+0000","last_became_peered":"2024-04-26T16:31:31.465738+0000","last_unstale":"2024-04-26T16:31:31.466678+0000","last_undegraded":"2024-04-26T16:31:31.466678+0000","last_fullsized":"2024-04-26T16:31:31.466678+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.807190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"310.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-26T16:31:32.443553+0000","last_change":"2024-04-26T16:31:32.443553+0000","last_active":"2024-04-26T16:31:32.443553+0000","last_peered":"2024-04-26T16:31:32.443553+0000","last_clean":"2024-04-26T16:31:32.443553+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-26T16:31:32.443553+0000","last_undegraded":"2024-04-26T16:31:32.443553+0000","last_fullsized":"2024-04-26T16:31:32.443553+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:32.443553+0000","last_clean_scrub_stamp":"2024-04-26T16:31:32.443553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"309.0","version":"0'0","reported_seq":10,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.477312+0000","last_change":"2024-04-26T16:31:31.476031+0000","last_active":"2024-04-26T16:31:31.477312+0000","last_peered":"2024-04-26T16:31:31.477312+0000","last_clean":"2024-04-26T16:31:31.477312+0000","last_became_active":"2024-04-26T16:31:31.475729+0000","last_became_peered":"2024-04-26T16:31:31.475729+0000","last_unstale":"2024-04-26T16:31:31.477312+0000","last_undegraded":"2024-04-26T16:31:31.477312+0000","last_fullsized":"2024-04-26T16:31:31.477312+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":556,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:30.419311+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.419311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.956876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.12","version":"554'87","reported_seq":1083,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.445711+0000","last_change":"2024-04-26T16:29:29.291952+0000","last_active":"2024-04-26T16:31:31.445711+0000","last_peered":"2024-04-26T16:31:31.445711+0000","last_clean":"2024-04-26T16:31:31.445711+0000","last_became_active":"2024-04-26T16:23:44.188737+0000","last_became_peered":"2024-04-26T16:23:44.188737+0000","last_unstale":"2024-04-26T16:31:31.445711+0000","last_undegraded":"2024-04-26T16:31:31.445711+0000","last_fullsized":"2024-04-26T16:31:31.445711+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:29:10.006584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.006584+0000","objects_scrubbed":6,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011941726,"stat_sum":{"num_bytes":1278,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.13","version":"545'55","reported_seq":887,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.471694+0000","last_change":"2024-04-26T16:30:24.606400+0000","last_active":"2024-04-26T16:31:31.471694+0000","last_peered":"2024-04-26T16:31:31.471694+0000","last_clean":"2024-04-26T16:31:31.471694+0000","last_became_active":"2024-04-26T16:26:23.192498+0000","last_became_peered":"2024-04-26T16:26:23.192498+0000","last_unstale":"2024-04-26T16:31:31.471694+0000","last_undegraded":"2024-04-26T16:31:31.471694+0000","last_fullsized":"2024-04-26T16:31:31.471694+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'52","last_scrub_stamp":"2024-04-26T16:30:24.606145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.606145+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:39.989241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064602280000000002,"stat_sum":{"num_bytes":1378,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.10","version":"469'150","reported_seq":1235,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452457+0000","last_change":"2024-04-26T16:29:29.501691+0000","last_active":"2024-04-26T16:31:31.452457+0000","last_peered":"2024-04-26T16:31:31.452457+0000","last_clean":"2024-04-26T16:31:31.452457+0000","last_became_active":"2024-04-26T16:26:26.135726+0000","last_became_peered":"2024-04-26T16:26:26.135726+0000","last_unstale":"2024-04-26T16:31:31.452457+0000","last_undegraded":"2024-04-26T16:31:31.452457+0000","last_fullsized":"2024-04-26T16:31:31.452457+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'126","last_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_deep_scrub":"400'126","last_deep_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_clean_scrub_stamp":"2024-04-26T16:27:49.058542+0000","objects_scrubbed":7,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001241277,"stat_sum":{"num_bytes":103,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":44,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":77,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.11","version":"535'36","reported_seq":1055,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449235+0000","last_change":"2024-04-26T16:29:57.929311+0000","last_active":"2024-04-26T16:31:31.449235+0000","last_peered":"2024-04-26T16:31:31.449235+0000","last_clean":"2024-04-26T16:31:31.449235+0000","last_became_active":"2024-04-26T16:23:44.102085+0000","last_became_peered":"2024-04-26T16:23:44.102085+0000","last_unstale":"2024-04-26T16:31:31.449235+0000","last_undegraded":"2024-04-26T16:31:31.449235+0000","last_fullsized":"2024-04-26T16:31:31.449235+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:29:57.929076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.929076+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066858430000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.16","version":"545'87","reported_seq":1005,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.456846+0000","last_change":"2024-04-26T16:29:29.294557+0000","last_active":"2024-04-26T16:31:31.456846+0000","last_peered":"2024-04-26T16:31:31.456846+0000","last_clean":"2024-04-26T16:31:31.456846+0000","last_became_active":"2024-04-26T16:26:23.214752+0000","last_became_peered":"2024-04-26T16:26:23.214752+0000","last_unstale":"2024-04-26T16:31:31.456846+0000","last_undegraded":"2024-04-26T16:31:31.456846+0000","last_fullsized":"2024-04-26T16:31:31.456846+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:29:04.018776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.018776+0000","objects_scrubbed":7,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060185220000000001,"stat_sum":{"num_bytes":599,"num_objects":10,"num_object_clones":1,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":32,"num_write_kb":36,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.17","version":"520'52","reported_seq":1032,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459544+0000","last_change":"2024-04-26T16:30:02.546379+0000","last_active":"2024-04-26T16:31:32.459544+0000","last_peered":"2024-04-26T16:31:32.459544+0000","last_clean":"2024-04-26T16:31:32.459544+0000","last_became_active":"2024-04-26T16:28:25.260911+0000","last_became_peered":"2024-04-26T16:28:25.260911+0000","last_unstale":"2024-04-26T16:31:32.459544+0000","last_undegraded":"2024-04-26T16:31:32.459544+0000","last_fullsized":"2024-04-26T16:31:32.459544+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'50","last_scrub_stamp":"2024-04-26T16:30:02.546137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.546137+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00057068899999999998,"stat_sum":{"num_bytes":71,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":45,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.14","version":"536'42","reported_seq":1063,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449813+0000","last_change":"2024-04-26T16:31:16.556513+0000","last_active":"2024-04-26T16:31:31.449813+0000","last_peered":"2024-04-26T16:31:31.449813+0000","last_clean":"2024-04-26T16:31:31.449813+0000","last_became_active":"2024-04-26T16:23:44.119110+0000","last_became_peered":"2024-04-26T16:23:44.119110+0000","last_unstale":"2024-04-26T16:31:31.449813+0000","last_undegraded":"2024-04-26T16:31:31.449813+0000","last_fullsized":"2024-04-26T16:31:31.449813+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'42","last_scrub_stamp":"2024-04-26T16:31:16.556378+0000","last_deep_scrub":"457'38","last_deep_scrub_stamp":"2024-04-26T16:28:49.475044+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.556378+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:36.641364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0034343469999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.15","version":"535'82","reported_seq":1099,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.474677+0000","last_change":"2024-04-26T16:30:12.508796+0000","last_active":"2024-04-26T16:31:31.474677+0000","last_peered":"2024-04-26T16:31:31.474677+0000","last_clean":"2024-04-26T16:31:31.474677+0000","last_became_active":"2024-04-26T16:23:44.233737+0000","last_became_peered":"2024-04-26T16:23:44.233737+0000","last_unstale":"2024-04-26T16:31:31.474677+0000","last_undegraded":"2024-04-26T16:31:31.474677+0000","last_fullsized":"2024-04-26T16:31:31.474677+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'78","last_scrub_stamp":"2024-04-26T16:30:12.508639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.508639+0000","objects_scrubbed":6,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0046970850000000001,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1a","version":"322'90","reported_seq":1160,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.471248+0000","last_change":"2024-04-26T16:30:25.642345+0000","last_active":"2024-04-26T16:31:31.471248+0000","last_peered":"2024-04-26T16:31:31.471248+0000","last_clean":"2024-04-26T16:31:31.471248+0000","last_became_active":"2024-04-26T16:26:23.171828+0000","last_became_peered":"2024-04-26T16:26:23.171828+0000","last_unstale":"2024-04-26T16:31:31.471248+0000","last_undegraded":"2024-04-26T16:31:31.471248+0000","last_fullsized":"2024-04-26T16:31:31.471248+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'90","last_scrub_stamp":"2024-04-26T16:30:25.642102+0000","last_deep_scrub":"322'90","last_deep_scrub_stamp":"2024-04-26T16:26:30.427683+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.642102+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:51.756368+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0046318089999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1b","version":"554'32","reported_seq":1082,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.471698+0000","last_change":"2024-04-26T16:31:19.468806+0000","last_active":"2024-04-26T16:31:31.471698+0000","last_peered":"2024-04-26T16:31:31.471698+0000","last_clean":"2024-04-26T16:31:31.471698+0000","last_became_active":"2024-04-26T16:28:25.251700+0000","last_became_peered":"2024-04-26T16:28:25.251700+0000","last_unstale":"2024-04-26T16:31:31.471698+0000","last_undegraded":"2024-04-26T16:31:31.471698+0000","last_fullsized":"2024-04-26T16:31:31.471698+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'30","last_scrub_stamp":"2024-04-26T16:31:19.468562+0000","last_deep_scrub":"458'28","last_deep_scrub_stamp":"2024-04-26T16:28:51.005534+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.468562+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:23.245859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048628739999999997,"stat_sum":{"num_bytes":1251,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":19,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.18","version":"545'46","reported_seq":1097,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.464162+0000","last_change":"2024-04-26T16:30:20.631404+0000","last_active":"2024-04-26T16:31:31.464162+0000","last_peered":"2024-04-26T16:31:31.464162+0000","last_clean":"2024-04-26T16:31:31.464162+0000","last_became_active":"2024-04-26T16:26:23.188399+0000","last_became_peered":"2024-04-26T16:26:23.188399+0000","last_unstale":"2024-04-26T16:31:31.464162+0000","last_undegraded":"2024-04-26T16:31:31.464162+0000","last_fullsized":"2024-04-26T16:31:31.464162+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'45","last_scrub_stamp":"2024-04-26T16:30:20.631310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:20.631310+0000","objects_scrubbed":6,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:33.855977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0038147540000000001,"stat_sum":{"num_bytes":1255,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.19","version":"554'40","reported_seq":1061,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.465111+0000","last_change":"2024-04-26T16:30:09.622791+0000","last_active":"2024-04-26T16:31:31.465111+0000","last_peered":"2024-04-26T16:31:31.465111+0000","last_clean":"2024-04-26T16:31:31.465111+0000","last_became_active":"2024-04-26T16:23:44.267057+0000","last_became_peered":"2024-04-26T16:23:44.267057+0000","last_unstale":"2024-04-26T16:31:31.465111+0000","last_undegraded":"2024-04-26T16:31:31.465111+0000","last_fullsized":"2024-04-26T16:31:31.465111+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'35","last_scrub_stamp":"2024-04-26T16:30:09.622674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.622674+0000","objects_scrubbed":3,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029077199999999999,"stat_sum":{"num_bytes":597,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":19,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.1e","version":"536'26","reported_seq":1058,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.474975+0000","last_change":"2024-04-26T16:30:33.603424+0000","last_active":"2024-04-26T16:31:31.474975+0000","last_peered":"2024-04-26T16:31:31.474975+0000","last_clean":"2024-04-26T16:31:31.474975+0000","last_became_active":"2024-04-26T16:28:25.230305+0000","last_became_peered":"2024-04-26T16:28:25.230305+0000","last_unstale":"2024-04-26T16:31:31.474975+0000","last_undegraded":"2024-04-26T16:31:31.474975+0000","last_fullsized":"2024-04-26T16:31:31.474975+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-04-26T16:30:33.603284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.603284+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:52.405921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005995696,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":14,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1f","version":"545'28","reported_seq":1044,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468957+0000","last_change":"2024-04-26T16:29:29.292388+0000","last_active":"2024-04-26T16:31:31.468957+0000","last_peered":"2024-04-26T16:31:31.468957+0000","last_clean":"2024-04-26T16:31:31.468957+0000","last_became_active":"2024-04-26T16:23:44.180651+0000","last_became_peered":"2024-04-26T16:23:44.180651+0000","last_unstale":"2024-04-26T16:31:31.468957+0000","last_undegraded":"2024-04-26T16:31:31.468957+0000","last_fullsized":"2024-04-26T16:31:31.468957+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'20","last_scrub_stamp":"2024-04-26T16:29:03.044079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.044079+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0037815560000000002,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1c","version":"536'60","reported_seq":1056,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.461708+0000","last_change":"2024-04-26T16:30:18.310854+0000","last_active":"2024-04-26T16:31:32.461708+0000","last_peered":"2024-04-26T16:31:32.461708+0000","last_clean":"2024-04-26T16:31:32.461708+0000","last_became_active":"2024-04-26T16:28:25.253825+0000","last_became_peered":"2024-04-26T16:28:25.253825+0000","last_unstale":"2024-04-26T16:31:32.461708+0000","last_undegraded":"2024-04-26T16:31:32.461708+0000","last_fullsized":"2024-04-26T16:31:32.461708+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-04-26T16:30:18.310740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.310740+0000","objects_scrubbed":4,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053197300000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":23,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.1d","version":"554'150","reported_seq":1128,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455475+0000","last_change":"2024-04-26T16:29:29.505126+0000","last_active":"2024-04-26T16:31:31.455475+0000","last_peered":"2024-04-26T16:31:31.455475+0000","last_clean":"2024-04-26T16:31:31.455475+0000","last_became_active":"2024-04-26T16:23:44.112550+0000","last_became_peered":"2024-04-26T16:23:44.112550+0000","last_unstale":"2024-04-26T16:31:31.455475+0000","last_undegraded":"2024-04-26T16:31:31.455475+0000","last_fullsized":"2024-04-26T16:31:31.455475+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'144","last_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_deep_scrub":"401'144","last_deep_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_clean_scrub_stamp":"2024-04-26T16:27:50.033127+0000","objects_scrubbed":8,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001087244,"stat_sum":{"num_bytes":2408,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":47,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.2","version":"503'103","reported_seq":1121,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449463+0000","last_change":"2024-04-26T16:31:26.486129+0000","last_active":"2024-04-26T16:31:31.449463+0000","last_peered":"2024-04-26T16:31:31.449463+0000","last_clean":"2024-04-26T16:31:31.449463+0000","last_became_active":"2024-04-26T16:23:44.121137+0000","last_became_peered":"2024-04-26T16:23:44.121137+0000","last_unstale":"2024-04-26T16:31:31.449463+0000","last_undegraded":"2024-04-26T16:31:31.449463+0000","last_fullsized":"2024-04-26T16:31:31.449463+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'103","last_scrub_stamp":"2024-04-26T16:31:26.485773+0000","last_deep_scrub":"318'90","last_deep_scrub_stamp":"2024-04-26T16:26:26.686096+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.485773+0000","objects_scrubbed":9,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:46.138862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0030250239999999999,"stat_sum":{"num_bytes":93,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":29,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.3","version":"545'52","reported_seq":1103,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454519+0000","last_change":"2024-04-26T16:29:29.502257+0000","last_active":"2024-04-26T16:31:31.454519+0000","last_peered":"2024-04-26T16:31:31.454519+0000","last_clean":"2024-04-26T16:31:31.454519+0000","last_became_active":"2024-04-26T16:28:25.273792+0000","last_became_peered":"2024-04-26T16:28:25.273792+0000","last_unstale":"2024-04-26T16:31:31.454519+0000","last_undegraded":"2024-04-26T16:31:31.454519+0000","last_fullsized":"2024-04-26T16:31:31.454519+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'51","last_scrub_stamp":"2024-04-26T16:27:34.948809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.948809+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000677325,"stat_sum":{"num_bytes":600,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.0","version":"545'76","reported_seq":1076,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.445748+0000","last_change":"2024-04-26T16:29:56.937825+0000","last_active":"2024-04-26T16:31:31.445748+0000","last_peered":"2024-04-26T16:31:31.445748+0000","last_clean":"2024-04-26T16:31:31.445748+0000","last_became_active":"2024-04-26T16:23:44.226589+0000","last_became_peered":"2024-04-26T16:23:44.226589+0000","last_unstale":"2024-04-26T16:31:31.445748+0000","last_undegraded":"2024-04-26T16:31:31.445748+0000","last_fullsized":"2024-04-26T16:31:31.445748+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'75","last_scrub_stamp":"2024-04-26T16:29:56.937577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.937577+0000","objects_scrubbed":6,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004787085,"stat_sum":{"num_bytes":2370,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1","version":"545'111","reported_seq":1118,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453975+0000","last_change":"2024-04-26T16:29:29.501949+0000","last_active":"2024-04-26T16:31:31.453975+0000","last_peered":"2024-04-26T16:31:31.453975+0000","last_clean":"2024-04-26T16:31:31.453975+0000","last_became_active":"2024-04-26T16:28:25.268393+0000","last_became_peered":"2024-04-26T16:28:25.268393+0000","last_unstale":"2024-04-26T16:31:31.453975+0000","last_undegraded":"2024-04-26T16:31:31.453975+0000","last_fullsized":"2024-04-26T16:31:31.453975+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'102","last_scrub_stamp":"2024-04-26T16:27:26.872968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:26.872968+0000","objects_scrubbed":6,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086574099999999995,"stat_sum":{"num_bytes":1289,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":42,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.6","version":"554'86","reported_seq":1107,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.474835+0000","last_change":"2024-04-26T16:30:31.618966+0000","last_active":"2024-04-26T16:31:31.474835+0000","last_peered":"2024-04-26T16:31:31.474835+0000","last_clean":"2024-04-26T16:31:31.474835+0000","last_became_active":"2024-04-26T16:23:44.248476+0000","last_became_peered":"2024-04-26T16:23:44.248476+0000","last_unstale":"2024-04-26T16:31:31.474835+0000","last_undegraded":"2024-04-26T16:31:31.474835+0000","last_fullsized":"2024-04-26T16:31:31.474835+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'83","last_scrub_stamp":"2024-04-26T16:30:31.618721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.618721+0000","objects_scrubbed":8,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:41.987667+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.010158063,"stat_sum":{"num_bytes":4000069,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":30,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.7","version":"554'84","reported_seq":1100,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455798+0000","last_change":"2024-04-26T16:29:29.505553+0000","last_active":"2024-04-26T16:31:31.455798+0000","last_peered":"2024-04-26T16:31:31.455798+0000","last_clean":"2024-04-26T16:31:31.455798+0000","last_became_active":"2024-04-26T16:23:44.200194+0000","last_became_peered":"2024-04-26T16:23:44.200194+0000","last_unstale":"2024-04-26T16:31:31.455798+0000","last_undegraded":"2024-04-26T16:31:31.455798+0000","last_fullsized":"2024-04-26T16:31:31.455798+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'57","last_scrub_stamp":"2024-04-26T16:27:37.015805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:37.015805+0000","objects_scrubbed":5,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094604400000000005,"stat_sum":{"num_bytes":1311,"num_objects":13,"num_object_clones":2,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":18,"num_read_kb":7,"num_write":39,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.4","version":"389'86","reported_seq":1105,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452076+0000","last_change":"2024-04-26T16:30:10.930053+0000","last_active":"2024-04-26T16:31:31.452076+0000","last_peered":"2024-04-26T16:31:31.452076+0000","last_clean":"2024-04-26T16:31:31.452076+0000","last_became_active":"2024-04-26T16:28:25.254293+0000","last_became_peered":"2024-04-26T16:28:25.254293+0000","last_unstale":"2024-04-26T16:31:31.452076+0000","last_undegraded":"2024-04-26T16:31:31.452076+0000","last_fullsized":"2024-04-26T16:31:31.452076+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:30:10.929911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:10.929911+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.745250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0073012850000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.5","version":"545'55","reported_seq":1080,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450205+0000","last_change":"2024-04-26T16:31:15.603225+0000","last_active":"2024-04-26T16:31:31.450205+0000","last_peered":"2024-04-26T16:31:31.450205+0000","last_clean":"2024-04-26T16:31:31.450205+0000","last_became_active":"2024-04-26T16:23:44.116978+0000","last_became_peered":"2024-04-26T16:23:44.116978+0000","last_unstale":"2024-04-26T16:31:31.450205+0000","last_undegraded":"2024-04-26T16:31:31.450205+0000","last_fullsized":"2024-04-26T16:31:31.450205+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'54","last_scrub_stamp":"2024-04-26T16:31:15.602994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:31:15.602994+0000","objects_scrubbed":5,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:34.524864+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018006140000000001,"stat_sum":{"num_bytes":750,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.a","version":"545'58","reported_seq":1113,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449944+0000","last_change":"2024-04-26T16:30:31.542354+0000","last_active":"2024-04-26T16:31:31.449944+0000","last_peered":"2024-04-26T16:31:31.449944+0000","last_clean":"2024-04-26T16:31:31.449944+0000","last_became_active":"2024-04-26T16:26:27.141171+0000","last_became_peered":"2024-04-26T16:26:27.141171+0000","last_unstale":"2024-04-26T16:31:31.449944+0000","last_undegraded":"2024-04-26T16:31:31.449944+0000","last_fullsized":"2024-04-26T16:31:31.449944+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'44","last_scrub_stamp":"2024-04-26T16:30:31.542113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.542113+0000","objects_scrubbed":5,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.941519+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020446129999999998,"stat_sum":{"num_bytes":53,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":28,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.b","version":"545'37","reported_seq":1067,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449137+0000","last_change":"2024-04-26T16:30:00.932705+0000","last_active":"2024-04-26T16:31:31.449137+0000","last_peered":"2024-04-26T16:31:31.449137+0000","last_clean":"2024-04-26T16:31:31.449137+0000","last_became_active":"2024-04-26T16:28:25.233607+0000","last_became_peered":"2024-04-26T16:28:25.233607+0000","last_unstale":"2024-04-26T16:31:31.449137+0000","last_undegraded":"2024-04-26T16:31:31.449137+0000","last_fullsized":"2024-04-26T16:31:31.449137+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:30:00.932368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.932368+0000","objects_scrubbed":2,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049354960000000002,"stat_sum":{"num_bytes":1564,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.8","version":"554'56","reported_seq":1120,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.470618+0000","last_change":"2024-04-26T16:30:11.507964+0000","last_active":"2024-04-26T16:31:31.470618+0000","last_peered":"2024-04-26T16:31:31.470618+0000","last_clean":"2024-04-26T16:31:31.470618+0000","last_became_active":"2024-04-26T16:28:25.245886+0000","last_became_peered":"2024-04-26T16:28:25.245886+0000","last_unstale":"2024-04-26T16:31:31.470618+0000","last_undegraded":"2024-04-26T16:31:31.470618+0000","last_fullsized":"2024-04-26T16:31:31.470618+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'55","last_scrub_stamp":"2024-04-26T16:30:11.507584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.507584+0000","objects_scrubbed":5,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.911256+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001167427,"stat_sum":{"num_bytes":1591,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.9","version":"224'34","reported_seq":1046,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.447478+0000","last_change":"2024-04-26T16:29:29.284837+0000","last_active":"2024-04-26T16:31:31.447478+0000","last_peered":"2024-04-26T16:31:31.447478+0000","last_clean":"2024-04-26T16:31:31.447478+0000","last_became_active":"2024-04-26T16:28:25.245235+0000","last_became_peered":"2024-04-26T16:28:25.245235+0000","last_unstale":"2024-04-26T16:31:31.447478+0000","last_undegraded":"2024-04-26T16:31:31.447478+0000","last_fullsized":"2024-04-26T16:31:31.447478+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:28:54.109529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.109529+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0053844380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.e","version":"554'34","reported_seq":1128,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.456705+0000","last_change":"2024-04-26T16:30:00.904554+0000","last_active":"2024-04-26T16:31:31.456705+0000","last_peered":"2024-04-26T16:31:31.456705+0000","last_clean":"2024-04-26T16:31:31.456705+0000","last_became_active":"2024-04-26T16:26:23.214391+0000","last_became_peered":"2024-04-26T16:26:23.214391+0000","last_unstale":"2024-04-26T16:31:31.456705+0000","last_undegraded":"2024-04-26T16:31:31.456705+0000","last_fullsized":"2024-04-26T16:31:31.456705+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-04-26T16:30:00.904197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.904197+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00183567,"stat_sum":{"num_bytes":4194328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":15,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.f","version":"536'84","reported_seq":1141,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451079+0000","last_change":"2024-04-26T16:29:29.290022+0000","last_active":"2024-04-26T16:31:31.451079+0000","last_peered":"2024-04-26T16:31:31.451079+0000","last_clean":"2024-04-26T16:31:31.451079+0000","last_became_active":"2024-04-26T16:26:23.198449+0000","last_became_peered":"2024-04-26T16:26:23.198449+0000","last_unstale":"2024-04-26T16:31:31.451079+0000","last_undegraded":"2024-04-26T16:31:31.451079+0000","last_fullsized":"2024-04-26T16:31:31.451079+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:28:56.606556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:56.606556+0000","objects_scrubbed":7,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001552419,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.c","version":"389'35","reported_seq":1051,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451897+0000","last_change":"2024-04-26T16:30:16.958876+0000","last_active":"2024-04-26T16:31:31.451897+0000","last_peered":"2024-04-26T16:31:31.451897+0000","last_clean":"2024-04-26T16:31:31.451897+0000","last_became_active":"2024-04-26T16:23:44.106600+0000","last_became_peered":"2024-04-26T16:23:44.106600+0000","last_unstale":"2024-04-26T16:31:31.451897+0000","last_undegraded":"2024-04-26T16:31:31.451897+0000","last_fullsized":"2024-04-26T16:31:31.451897+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:16.958732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.958732+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.989385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043915680000000002,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.d","version":"554'27","reported_seq":845,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455214+0000","last_change":"2024-04-26T16:29:29.289068+0000","last_active":"2024-04-26T16:31:31.455214+0000","last_peered":"2024-04-26T16:31:31.455214+0000","last_clean":"2024-04-26T16:31:31.455214+0000","last_became_active":"2024-04-26T16:28:25.242455+0000","last_became_peered":"2024-04-26T16:28:25.242455+0000","last_unstale":"2024-04-26T16:31:31.455214+0000","last_undegraded":"2024-04-26T16:31:31.455214+0000","last_fullsized":"2024-04-26T16:31:31.455214+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'22","last_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_deep_scrub":"464'22","last_deep_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.003905+0000","objects_scrubbed":3,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00091357000000000005,"stat_sum":{"num_bytes":1364,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":514,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455794+0000","last_change":"2024-04-26T16:31:24.119396+0000","last_active":"2024-04-26T16:31:31.455794+0000","last_peered":"2024-04-26T16:31:31.455794+0000","last_clean":"2024-04-26T16:31:31.455794+0000","last_became_active":"2024-04-26T16:26:23.194196+0000","last_became_peered":"2024-04-26T16:26:23.194196+0000","last_unstale":"2024-04-26T16:31:31.455794+0000","last_undegraded":"2024-04-26T16:31:31.455794+0000","last_fullsized":"2024-04-26T16:31:31.455794+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:24.119192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:24.119192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.738696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"433'14","reported_seq":1192,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453738+0000","last_change":"2024-04-26T16:29:10.136335+0000","last_active":"2024-04-26T16:31:31.453738+0000","last_peered":"2024-04-26T16:31:31.453738+0000","last_clean":"2024-04-26T16:31:31.453738+0000","last_became_active":"2024-04-26T16:28:25.267969+0000","last_became_peered":"2024-04-26T16:28:25.267969+0000","last_unstale":"2024-04-26T16:31:31.453738+0000","last_undegraded":"2024-04-26T16:31:31.453738+0000","last_fullsized":"2024-04-26T16:31:31.453738+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.136172+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"458'34","reported_seq":1077,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459310+0000","last_change":"2024-04-26T16:28:50.981113+0000","last_active":"2024-04-26T16:31:31.459310+0000","last_peered":"2024-04-26T16:31:31.459310+0000","last_clean":"2024-04-26T16:31:31.459310+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:31:31.459310+0000","last_undegraded":"2024-04-26T16:31:31.459310+0000","last_fullsized":"2024-04-26T16:31:31.459310+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.980879+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1019,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.475707+0000","last_change":"2024-04-26T16:30:08.452481+0000","last_active":"2024-04-26T16:31:31.475707+0000","last_peered":"2024-04-26T16:31:31.475707+0000","last_clean":"2024-04-26T16:31:31.475707+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:31:31.475707+0000","last_undegraded":"2024-04-26T16:31:31.475707+0000","last_fullsized":"2024-04-26T16:31:31.475707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.452371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.452371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.819274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":411,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.457976+0000","last_change":"2024-04-26T16:31:16.230723+0000","last_active":"2024-04-26T16:31:32.457976+0000","last_peered":"2024-04-26T16:31:32.457976+0000","last_clean":"2024-04-26T16:31:32.457976+0000","last_became_active":"2024-04-26T16:28:25.238286+0000","last_became_peered":"2024-04-26T16:28:25.238286+0000","last_unstale":"2024-04-26T16:31:32.457976+0000","last_undegraded":"2024-04-26T16:31:32.457976+0000","last_fullsized":"2024-04-26T16:31:32.457976+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:16.230516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.230516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.279036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":861,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454978+0000","last_change":"2024-04-26T16:28:25.277790+0000","last_active":"2024-04-26T16:31:31.454978+0000","last_peered":"2024-04-26T16:31:31.454978+0000","last_clean":"2024-04-26T16:31:31.454978+0000","last_became_active":"2024-04-26T16:28:25.277260+0000","last_became_peered":"2024-04-26T16:28:25.277260+0000","last_unstale":"2024-04-26T16:31:31.454978+0000","last_undegraded":"2024-04-26T16:31:31.454978+0000","last_fullsized":"2024-04-26T16:31:31.454978+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:28.909860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.909860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"388'20","reported_seq":576,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454583+0000","last_change":"2024-04-26T16:31:19.057114+0000","last_active":"2024-04-26T16:31:31.454583+0000","last_peered":"2024-04-26T16:31:31.454583+0000","last_clean":"2024-04-26T16:31:31.454583+0000","last_became_active":"2024-04-26T16:28:25.257143+0000","last_became_peered":"2024-04-26T16:28:25.257143+0000","last_unstale":"2024-04-26T16:31:31.454583+0000","last_undegraded":"2024-04-26T16:31:31.454583+0000","last_fullsized":"2024-04-26T16:31:31.454583+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:19.056987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.056987+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.250805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1021,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452327+0000","last_change":"2024-04-26T16:30:15.948290+0000","last_active":"2024-04-26T16:31:31.452327+0000","last_peered":"2024-04-26T16:31:31.452327+0000","last_clean":"2024-04-26T16:31:31.452327+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:31:31.452327+0000","last_undegraded":"2024-04-26T16:31:31.452327+0000","last_fullsized":"2024-04-26T16:31:31.452327+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.948187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.948187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1019,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.471285+0000","last_change":"2024-04-26T16:31:28.375454+0000","last_active":"2024-04-26T16:31:31.471285+0000","last_peered":"2024-04-26T16:31:31.471285+0000","last_clean":"2024-04-26T16:31:31.471285+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:31:31.471285+0000","last_undegraded":"2024-04-26T16:31:31.471285+0000","last_fullsized":"2024-04-26T16:31:31.471285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:28.375370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:28.375370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.246169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1016,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.465797+0000","last_change":"2024-04-26T16:30:26.632966+0000","last_active":"2024-04-26T16:31:31.465797+0000","last_peered":"2024-04-26T16:31:31.465797+0000","last_clean":"2024-04-26T16:31:31.465797+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:31:31.465797+0000","last_undegraded":"2024-04-26T16:31:31.465797+0000","last_fullsized":"2024-04-26T16:31:31.465797+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.632648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:33.662364+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":875,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455903+0000","last_change":"2024-04-26T16:31:15.769031+0000","last_active":"2024-04-26T16:31:31.455903+0000","last_peered":"2024-04-26T16:31:31.455903+0000","last_clean":"2024-04-26T16:31:31.455903+0000","last_became_active":"2024-04-26T16:28:25.270561+0000","last_became_peered":"2024-04-26T16:28:25.270561+0000","last_unstale":"2024-04-26T16:31:31.455903+0000","last_undegraded":"2024-04-26T16:31:31.455903+0000","last_fullsized":"2024-04-26T16:31:31.455903+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:15.768934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:15.768934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:17.469191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"388'31","reported_seq":1087,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.456002+0000","last_change":"2024-04-26T16:28:44.986410+0000","last_active":"2024-04-26T16:31:31.456002+0000","last_peered":"2024-04-26T16:31:31.456002+0000","last_clean":"2024-04-26T16:31:31.456002+0000","last_became_active":"2024-04-26T16:28:25.270747+0000","last_became_peered":"2024-04-26T16:28:25.270747+0000","last_unstale":"2024-04-26T16:31:31.456002+0000","last_undegraded":"2024-04-26T16:31:31.456002+0000","last_fullsized":"2024-04-26T16:31:31.456002+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'31","last_scrub_stamp":"2024-04-26T16:28:44.986040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.986040+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1033,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.473126+0000","last_change":"2024-04-26T16:31:23.456034+0000","last_active":"2024-04-26T16:31:31.473126+0000","last_peered":"2024-04-26T16:31:31.473126+0000","last_clean":"2024-04-26T16:31:31.473126+0000","last_became_active":"2024-04-26T16:26:23.201706+0000","last_became_peered":"2024-04-26T16:26:23.201706+0000","last_unstale":"2024-04-26T16:31:31.473126+0000","last_undegraded":"2024-04-26T16:31:31.473126+0000","last_fullsized":"2024-04-26T16:31:31.473126+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.455395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.455395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:45.158377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1017,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.473340+0000","last_change":"2024-04-26T16:30:03.454481+0000","last_active":"2024-04-26T16:31:31.473340+0000","last_peered":"2024-04-26T16:31:31.473340+0000","last_clean":"2024-04-26T16:31:31.473340+0000","last_became_active":"2024-04-26T16:26:23.172602+0000","last_became_peered":"2024-04-26T16:26:23.172602+0000","last_unstale":"2024-04-26T16:31:31.473340+0000","last_undegraded":"2024-04-26T16:31:31.473340+0000","last_fullsized":"2024-04-26T16:31:31.473340+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:03.454165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.454165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":997,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452663+0000","last_change":"2024-04-26T16:31:19.857894+0000","last_active":"2024-04-26T16:31:31.452663+0000","last_peered":"2024-04-26T16:31:31.452663+0000","last_clean":"2024-04-26T16:31:31.452663+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:31:31.452663+0000","last_undegraded":"2024-04-26T16:31:31.452663+0000","last_fullsized":"2024-04-26T16:31:31.452663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:19.857656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.857656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:43.917067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.471557+0000","last_change":"2024-04-26T16:30:06.484249+0000","last_active":"2024-04-26T16:31:31.471557+0000","last_peered":"2024-04-26T16:31:31.471557+0000","last_clean":"2024-04-26T16:31:31.471557+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:31:31.471557+0000","last_undegraded":"2024-04-26T16:31:31.471557+0000","last_fullsized":"2024-04-26T16:31:31.471557+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:06.484028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:06.484028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"389'35","reported_seq":1011,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.457608+0000","last_change":"2024-04-26T16:30:11.460779+0000","last_active":"2024-04-26T16:31:32.457608+0000","last_peered":"2024-04-26T16:31:32.457608+0000","last_clean":"2024-04-26T16:31:32.457608+0000","last_became_active":"2024-04-26T16:28:25.247148+0000","last_became_peered":"2024-04-26T16:28:25.247148+0000","last_unstale":"2024-04-26T16:31:32.457608+0000","last_undegraded":"2024-04-26T16:31:32.457608+0000","last_fullsized":"2024-04-26T16:31:32.457608+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:11.460648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.460648+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1015,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454940+0000","last_change":"2024-04-26T16:30:01.188873+0000","last_active":"2024-04-26T16:31:31.454940+0000","last_peered":"2024-04-26T16:31:31.454940+0000","last_clean":"2024-04-26T16:31:31.454940+0000","last_became_active":"2024-04-26T16:26:23.199164+0000","last_became_peered":"2024-04-26T16:26:23.199164+0000","last_unstale":"2024-04-26T16:31:31.454940+0000","last_undegraded":"2024-04-26T16:31:31.454940+0000","last_fullsized":"2024-04-26T16:31:31.454940+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:01.188654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:30:01.188654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1021,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454687+0000","last_change":"2024-04-26T16:30:29.051438+0000","last_active":"2024-04-26T16:31:31.454687+0000","last_peered":"2024-04-26T16:31:31.454687+0000","last_clean":"2024-04-26T16:31:31.454687+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:31:31.454687+0000","last_undegraded":"2024-04-26T16:31:31.454687+0000","last_fullsized":"2024-04-26T16:31:31.454687+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.051284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:34.626968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1007,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.475340+0000","last_change":"2024-04-26T16:29:59.407848+0000","last_active":"2024-04-26T16:31:31.475340+0000","last_peered":"2024-04-26T16:31:31.475340+0000","last_clean":"2024-04-26T16:31:31.475340+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:31:31.475340+0000","last_undegraded":"2024-04-26T16:31:31.475340+0000","last_fullsized":"2024-04-26T16:31:31.475340+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:59.407460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:29:59.407460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":769,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.455373+0000","last_change":"2024-04-26T16:30:00.494762+0000","last_active":"2024-04-26T16:31:32.455373+0000","last_peered":"2024-04-26T16:31:32.455373+0000","last_clean":"2024-04-26T16:31:32.455373+0000","last_became_active":"2024-04-26T16:28:25.265097+0000","last_became_peered":"2024-04-26T16:28:25.265097+0000","last_unstale":"2024-04-26T16:31:32.455373+0000","last_undegraded":"2024-04-26T16:31:32.455373+0000","last_fullsized":"2024-04-26T16:31:32.455373+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.494544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.494544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"388'19","reported_seq":1035,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453333+0000","last_change":"2024-04-26T16:30:34.982606+0000","last_active":"2024-04-26T16:31:31.453333+0000","last_peered":"2024-04-26T16:31:31.453333+0000","last_clean":"2024-04-26T16:31:31.453333+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:31:31.453333+0000","last_undegraded":"2024-04-26T16:31:31.453333+0000","last_fullsized":"2024-04-26T16:31:31.453333+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:34.982478+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.982478+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:59.881026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1021,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451780+0000","last_change":"2024-04-26T16:29:56.955956+0000","last_active":"2024-04-26T16:31:31.451780+0000","last_peered":"2024-04-26T16:31:31.451780+0000","last_clean":"2024-04-26T16:31:31.451780+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:31:31.451780+0000","last_undegraded":"2024-04-26T16:31:31.451780+0000","last_fullsized":"2024-04-26T16:31:31.451780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:56.955662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.955662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"316'12","reported_seq":1208,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.474684+0000","last_change":"2024-04-26T16:31:13.448819+0000","last_active":"2024-04-26T16:31:31.474684+0000","last_peered":"2024-04-26T16:31:31.474684+0000","last_clean":"2024-04-26T16:31:31.474684+0000","last_became_active":"2024-04-26T16:26:23.194072+0000","last_became_peered":"2024-04-26T16:26:23.194072+0000","last_unstale":"2024-04-26T16:31:31.474684+0000","last_undegraded":"2024-04-26T16:31:31.474684+0000","last_fullsized":"2024-04-26T16:31:31.474684+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'12","last_scrub_stamp":"2024-04-26T16:31:13.448695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:13.448695+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.006144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1008,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450985+0000","last_change":"2024-04-26T16:28:59.599961+0000","last_active":"2024-04-26T16:31:31.450985+0000","last_peered":"2024-04-26T16:31:31.450985+0000","last_clean":"2024-04-26T16:31:31.450985+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:31:31.450985+0000","last_undegraded":"2024-04-26T16:31:31.450985+0000","last_fullsized":"2024-04-26T16:31:31.450985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.599808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.599808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":863,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449970+0000","last_change":"2024-04-26T16:29:01.807303+0000","last_active":"2024-04-26T16:31:31.449970+0000","last_peered":"2024-04-26T16:31:31.449970+0000","last_clean":"2024-04-26T16:31:31.449970+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:31:31.449970+0000","last_undegraded":"2024-04-26T16:31:31.449970+0000","last_fullsized":"2024-04-26T16:31:31.449970+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:01.807199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:01.807199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"388'27","reported_seq":1049,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446445+0000","last_change":"2024-04-26T16:30:21.978132+0000","last_active":"2024-04-26T16:31:31.446445+0000","last_peered":"2024-04-26T16:31:31.446445+0000","last_clean":"2024-04-26T16:31:31.446445+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:31:31.446445+0000","last_undegraded":"2024-04-26T16:31:31.446445+0000","last_fullsized":"2024-04-26T16:31:31.446445+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:21.978007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.978007+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1035,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449661+0000","last_change":"2024-04-26T16:30:35.051954+0000","last_active":"2024-04-26T16:31:31.449661+0000","last_peered":"2024-04-26T16:31:31.449661+0000","last_clean":"2024-04-26T16:31:31.449661+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:31:31.449661+0000","last_undegraded":"2024-04-26T16:31:31.449661+0000","last_fullsized":"2024-04-26T16:31:31.449661+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.051759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:47.746481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"316'13","reported_seq":1202,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450616+0000","last_change":"2024-04-26T16:30:14.531293+0000","last_active":"2024-04-26T16:31:31.450616+0000","last_peered":"2024-04-26T16:31:31.450616+0000","last_clean":"2024-04-26T16:31:31.450616+0000","last_became_active":"2024-04-26T16:26:23.180811+0000","last_became_peered":"2024-04-26T16:26:23.180811+0000","last_unstale":"2024-04-26T16:31:31.450616+0000","last_undegraded":"2024-04-26T16:31:31.450616+0000","last_fullsized":"2024-04-26T16:31:31.450616+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:14.531205+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.531205+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":515,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452082+0000","last_change":"2024-04-26T16:30:25.049754+0000","last_active":"2024-04-26T16:31:31.452082+0000","last_peered":"2024-04-26T16:31:31.452082+0000","last_clean":"2024-04-26T16:31:31.452082+0000","last_became_active":"2024-04-26T16:26:23.195438+0000","last_became_peered":"2024-04-26T16:26:23.195438+0000","last_unstale":"2024-04-26T16:31:31.452082+0000","last_undegraded":"2024-04-26T16:31:31.452082+0000","last_fullsized":"2024-04-26T16:31:31.452082+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.049647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.049647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.587456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":786,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.454496+0000","last_change":"2024-04-26T16:30:07.526553+0000","last_active":"2024-04-26T16:31:32.454496+0000","last_peered":"2024-04-26T16:31:32.454496+0000","last_clean":"2024-04-26T16:31:32.454496+0000","last_became_active":"2024-04-26T16:28:25.244320+0000","last_became_peered":"2024-04-26T16:28:25.244320+0000","last_unstale":"2024-04-26T16:31:32.454496+0000","last_undegraded":"2024-04-26T16:31:32.454496+0000","last_fullsized":"2024-04-26T16:31:32.454496+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:30:07.526320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.526320+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"399'20","reported_seq":1050,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449092+0000","last_change":"2024-04-26T16:30:21.500286+0000","last_active":"2024-04-26T16:31:31.449092+0000","last_peered":"2024-04-26T16:31:31.449092+0000","last_clean":"2024-04-26T16:31:31.449092+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:31:31.449092+0000","last_undegraded":"2024-04-26T16:31:31.449092+0000","last_fullsized":"2024-04-26T16:31:31.449092+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'20","last_scrub_stamp":"2024-04-26T16:30:21.500166+0000","last_deep_scrub":"399'20","last_deep_scrub_stamp":"2024-04-26T16:27:47.205374+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.500166+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.067680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1017,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453363+0000","last_change":"2024-04-26T16:28:46.987810+0000","last_active":"2024-04-26T16:31:31.453363+0000","last_peered":"2024-04-26T16:31:31.453363+0000","last_clean":"2024-04-26T16:31:31.453363+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:31:31.453363+0000","last_undegraded":"2024-04-26T16:31:31.453363+0000","last_fullsized":"2024-04-26T16:31:31.453363+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:28:46.987686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.987686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":522,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451396+0000","last_change":"2024-04-26T16:30:23.518613+0000","last_active":"2024-04-26T16:31:31.451396+0000","last_peered":"2024-04-26T16:31:31.451396+0000","last_clean":"2024-04-26T16:31:31.451396+0000","last_became_active":"2024-04-26T16:28:25.248459+0000","last_became_peered":"2024-04-26T16:28:25.248459+0000","last_unstale":"2024-04-26T16:31:31.451396+0000","last_undegraded":"2024-04-26T16:31:31.451396+0000","last_fullsized":"2024-04-26T16:31:31.451396+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.518326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.518326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.755565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1013,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454820+0000","last_change":"2024-04-26T16:30:35.938334+0000","last_active":"2024-04-26T16:31:31.454820+0000","last_peered":"2024-04-26T16:31:31.454820+0000","last_clean":"2024-04-26T16:31:31.454820+0000","last_became_active":"2024-04-26T16:28:25.257629+0000","last_became_peered":"2024-04-26T16:28:25.257629+0000","last_unstale":"2024-04-26T16:31:31.454820+0000","last_undegraded":"2024-04-26T16:31:31.454820+0000","last_fullsized":"2024-04-26T16:31:31.454820+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.938129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.938129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.462202+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":861,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450654+0000","last_change":"2024-04-26T16:28:55.586673+0000","last_active":"2024-04-26T16:31:31.450654+0000","last_peered":"2024-04-26T16:31:31.450654+0000","last_clean":"2024-04-26T16:31:31.450654+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:31:31.450654+0000","last_undegraded":"2024-04-26T16:31:31.450654+0000","last_fullsized":"2024-04-26T16:31:31.450654+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:55.586453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:55.586453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1182,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446780+0000","last_change":"2024-04-26T16:31:27.023667+0000","last_active":"2024-04-26T16:31:31.446780+0000","last_peered":"2024-04-26T16:31:31.446780+0000","last_clean":"2024-04-26T16:31:31.446780+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:31:31.446780+0000","last_undegraded":"2024-04-26T16:31:31.446780+0000","last_fullsized":"2024-04-26T16:31:31.446780+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:31:27.023432+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:31:27.023432+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:56.061694+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1036,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449244+0000","last_change":"2024-04-26T16:28:44.503061+0000","last_active":"2024-04-26T16:31:31.449244+0000","last_peered":"2024-04-26T16:31:31.449244+0000","last_clean":"2024-04-26T16:31:31.449244+0000","last_became_active":"2024-04-26T16:28:25.250134+0000","last_became_peered":"2024-04-26T16:28:25.250134+0000","last_unstale":"2024-04-26T16:31:31.449244+0000","last_undegraded":"2024-04-26T16:31:31.449244+0000","last_fullsized":"2024-04-26T16:31:31.449244+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:44.502708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:13.552494+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.502708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'30","reported_seq":1066,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.467879+0000","last_change":"2024-04-26T16:31:15.749111+0000","last_active":"2024-04-26T16:31:31.467879+0000","last_peered":"2024-04-26T16:31:31.467879+0000","last_clean":"2024-04-26T16:31:31.467879+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:31:31.467879+0000","last_undegraded":"2024-04-26T16:31:31.467879+0000","last_fullsized":"2024-04-26T16:31:31.467879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'30","last_scrub_stamp":"2024-04-26T16:31:15.748874+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:31:15.748874+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.294364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":920,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.458584+0000","last_change":"2024-04-26T16:30:33.268356+0000","last_active":"2024-04-26T16:31:32.458584+0000","last_peered":"2024-04-26T16:31:32.458584+0000","last_clean":"2024-04-26T16:31:32.458584+0000","last_became_active":"2024-04-26T16:28:25.265921+0000","last_became_peered":"2024-04-26T16:28:25.265921+0000","last_unstale":"2024-04-26T16:31:32.458584+0000","last_undegraded":"2024-04-26T16:31:32.458584+0000","last_fullsized":"2024-04-26T16:31:32.458584+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.268035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.268035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.106859+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1006,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454764+0000","last_change":"2024-04-26T16:28:38.012269+0000","last_active":"2024-04-26T16:31:31.454764+0000","last_peered":"2024-04-26T16:31:31.454764+0000","last_clean":"2024-04-26T16:31:31.454764+0000","last_became_active":"2024-04-26T16:28:25.270748+0000","last_became_peered":"2024-04-26T16:28:25.270748+0000","last_unstale":"2024-04-26T16:31:31.454764+0000","last_undegraded":"2024-04-26T16:31:31.454764+0000","last_fullsized":"2024-04-26T16:31:31.454764+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:38.011966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:38.011966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":860,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459571+0000","last_change":"2024-04-26T16:30:08.292894+0000","last_active":"2024-04-26T16:31:31.459571+0000","last_peered":"2024-04-26T16:31:31.459571+0000","last_clean":"2024-04-26T16:31:31.459571+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:31:31.459571+0000","last_undegraded":"2024-04-26T16:31:31.459571+0000","last_fullsized":"2024-04-26T16:31:31.459571+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.292589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.292589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1012,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.448943+0000","last_change":"2024-04-26T16:29:09.485874+0000","last_active":"2024-04-26T16:31:31.448943+0000","last_peered":"2024-04-26T16:31:31.448943+0000","last_clean":"2024-04-26T16:31:31.448943+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:31:31.448943+0000","last_undegraded":"2024-04-26T16:31:31.448943+0000","last_fullsized":"2024-04-26T16:31:31.448943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.485662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.485662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":1834,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446615+0000","last_change":"2024-04-26T16:30:24.906077+0000","last_active":"2024-04-26T16:31:31.446615+0000","last_peered":"2024-04-26T16:31:31.446615+0000","last_clean":"2024-04-26T16:31:31.446615+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:31:31.446615+0000","last_undegraded":"2024-04-26T16:31:31.446615+0000","last_fullsized":"2024-04-26T16:31:31.446615+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:30:24.905850+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.905850+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":820,"num_read_kb":815,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'22","reported_seq":1044,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.469205+0000","last_change":"2024-04-26T16:30:19.960354+0000","last_active":"2024-04-26T16:31:31.469205+0000","last_peered":"2024-04-26T16:31:31.469205+0000","last_clean":"2024-04-26T16:31:31.469205+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:31:31.469205+0000","last_undegraded":"2024-04-26T16:31:31.469205+0000","last_fullsized":"2024-04-26T16:31:31.469205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:30:19.960123+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.960123+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1032,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452192+0000","last_change":"2024-04-26T16:29:08.743269+0000","last_active":"2024-04-26T16:31:31.452192+0000","last_peered":"2024-04-26T16:31:31.452192+0000","last_clean":"2024-04-26T16:31:31.452192+0000","last_became_active":"2024-04-26T16:28:25.233774+0000","last_became_peered":"2024-04-26T16:28:25.233774+0000","last_unstale":"2024-04-26T16:31:31.452192+0000","last_undegraded":"2024-04-26T16:31:31.452192+0000","last_fullsized":"2024-04-26T16:31:31.452192+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:29:08.742955+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:29:08.742955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1008,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449810+0000","last_change":"2024-04-26T16:28:40.455517+0000","last_active":"2024-04-26T16:31:31.449810+0000","last_peered":"2024-04-26T16:31:31.449810+0000","last_clean":"2024-04-26T16:31:31.449810+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:31:31.449810+0000","last_undegraded":"2024-04-26T16:31:31.449810+0000","last_fullsized":"2024-04-26T16:31:31.449810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.455422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.455422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452212+0000","last_change":"2024-04-26T16:30:12.954476+0000","last_active":"2024-04-26T16:31:31.452212+0000","last_peered":"2024-04-26T16:31:31.452212+0000","last_clean":"2024-04-26T16:31:31.452212+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:31:31.452212+0000","last_undegraded":"2024-04-26T16:31:31.452212+0000","last_fullsized":"2024-04-26T16:31:31.452212+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.954265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.954265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":912,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.457713+0000","last_change":"2024-04-26T16:30:25.328753+0000","last_active":"2024-04-26T16:31:32.457713+0000","last_peered":"2024-04-26T16:31:32.457713+0000","last_clean":"2024-04-26T16:31:32.457713+0000","last_became_active":"2024-04-26T16:28:25.234888+0000","last_became_peered":"2024-04-26T16:28:25.234888+0000","last_unstale":"2024-04-26T16:31:32.457713+0000","last_undegraded":"2024-04-26T16:31:32.457713+0000","last_fullsized":"2024-04-26T16:31:32.457713+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.328546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.328546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:53.320476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"388'53","reported_seq":1292,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452340+0000","last_change":"2024-04-26T16:29:10.751587+0000","last_active":"2024-04-26T16:31:31.452340+0000","last_peered":"2024-04-26T16:31:31.452340+0000","last_clean":"2024-04-26T16:31:31.452340+0000","last_became_active":"2024-04-26T16:26:23.203408+0000","last_became_peered":"2024-04-26T16:26:23.203408+0000","last_unstale":"2024-04-26T16:31:31.452340+0000","last_undegraded":"2024-04-26T16:31:31.452340+0000","last_fullsized":"2024-04-26T16:31:31.452340+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'53","last_scrub_stamp":"2024-04-26T16:29:10.751474+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.751474+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1018,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455794+0000","last_change":"2024-04-26T16:29:03.080654+0000","last_active":"2024-04-26T16:31:31.455794+0000","last_peered":"2024-04-26T16:31:31.455794+0000","last_clean":"2024-04-26T16:31:31.455794+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:31:31.455794+0000","last_undegraded":"2024-04-26T16:31:31.455794+0000","last_fullsized":"2024-04-26T16:31:31.455794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.080403+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1026,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450320+0000","last_change":"2024-04-26T16:30:28.037331+0000","last_active":"2024-04-26T16:31:31.450320+0000","last_peered":"2024-04-26T16:31:31.450320+0000","last_clean":"2024-04-26T16:31:31.450320+0000","last_became_active":"2024-04-26T16:28:25.242325+0000","last_became_peered":"2024-04-26T16:28:25.242325+0000","last_unstale":"2024-04-26T16:31:31.450320+0000","last_undegraded":"2024-04-26T16:31:31.450320+0000","last_fullsized":"2024-04-26T16:31:31.450320+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.037115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.037115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:56.639017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.448429+0000","last_change":"2024-04-26T16:29:58.933284+0000","last_active":"2024-04-26T16:31:31.448429+0000","last_peered":"2024-04-26T16:31:31.448429+0000","last_clean":"2024-04-26T16:31:31.448429+0000","last_became_active":"2024-04-26T16:28:25.245871+0000","last_became_peered":"2024-04-26T16:28:25.245871+0000","last_unstale":"2024-04-26T16:31:31.448429+0000","last_undegraded":"2024-04-26T16:31:31.448429+0000","last_fullsized":"2024-04-26T16:31:31.448429+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.933163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.933163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":514,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455620+0000","last_change":"2024-04-26T16:30:24.025255+0000","last_active":"2024-04-26T16:31:31.455620+0000","last_peered":"2024-04-26T16:31:31.455620+0000","last_clean":"2024-04-26T16:31:31.455620+0000","last_became_active":"2024-04-26T16:26:23.191179+0000","last_became_peered":"2024-04-26T16:26:23.191179+0000","last_unstale":"2024-04-26T16:31:31.455620+0000","last_undegraded":"2024-04-26T16:31:31.455620+0000","last_fullsized":"2024-04-26T16:31:31.455620+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.025163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.025163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1009,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452769+0000","last_change":"2024-04-26T16:28:59.045623+0000","last_active":"2024-04-26T16:31:31.452769+0000","last_peered":"2024-04-26T16:31:31.452769+0000","last_clean":"2024-04-26T16:31:31.452769+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:31:31.452769+0000","last_undegraded":"2024-04-26T16:31:31.452769+0000","last_fullsized":"2024-04-26T16:31:31.452769+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.045415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.045415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'35","reported_seq":1073,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.465467+0000","last_change":"2024-04-26T16:31:30.777450+0000","last_active":"2024-04-26T16:31:31.465467+0000","last_peered":"2024-04-26T16:31:31.465467+0000","last_clean":"2024-04-26T16:31:31.465467+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:31:31.465467+0000","last_undegraded":"2024-04-26T16:31:31.465467+0000","last_fullsized":"2024-04-26T16:31:31.465467+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:31:30.777298+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.777298+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:43.117201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1024,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455037+0000","last_change":"2024-04-26T16:31:18.097724+0000","last_active":"2024-04-26T16:31:31.455037+0000","last_peered":"2024-04-26T16:31:31.455037+0000","last_clean":"2024-04-26T16:31:31.455037+0000","last_became_active":"2024-04-26T16:28:25.253094+0000","last_became_peered":"2024-04-26T16:28:25.253094+0000","last_unstale":"2024-04-26T16:31:31.455037+0000","last_undegraded":"2024-04-26T16:31:31.455037+0000","last_fullsized":"2024-04-26T16:31:31.455037+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:18.097517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:31:18.097517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.069333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":868,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446065+0000","last_change":"2024-04-26T16:28:58.073423+0000","last_active":"2024-04-26T16:31:31.446065+0000","last_peered":"2024-04-26T16:31:31.446065+0000","last_clean":"2024-04-26T16:31:31.446065+0000","last_became_active":"2024-04-26T16:28:25.231951+0000","last_became_peered":"2024-04-26T16:28:25.231951+0000","last_unstale":"2024-04-26T16:31:31.446065+0000","last_undegraded":"2024-04-26T16:31:31.446065+0000","last_fullsized":"2024-04-26T16:31:31.446065+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:58.073194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:58.073194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1007,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.456094+0000","last_change":"2024-04-26T16:30:05.254074+0000","last_active":"2024-04-26T16:31:31.456094+0000","last_peered":"2024-04-26T16:31:31.456094+0000","last_clean":"2024-04-26T16:31:31.456094+0000","last_became_active":"2024-04-26T16:26:23.176892+0000","last_became_peered":"2024-04-26T16:26:23.176892+0000","last_unstale":"2024-04-26T16:31:31.456094+0000","last_undegraded":"2024-04-26T16:31:31.456094+0000","last_fullsized":"2024-04-26T16:31:31.456094+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:05.253814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:30:05.253814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1306,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453229+0000","last_change":"2024-04-26T16:31:16.790464+0000","last_active":"2024-04-26T16:31:31.453229+0000","last_peered":"2024-04-26T16:31:31.453229+0000","last_clean":"2024-04-26T16:31:31.453229+0000","last_became_active":"2024-04-26T16:28:25.267305+0000","last_became_peered":"2024-04-26T16:28:25.267305+0000","last_unstale":"2024-04-26T16:31:31.453229+0000","last_undegraded":"2024-04-26T16:31:31.453229+0000","last_fullsized":"2024-04-26T16:31:31.453229+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:31:16.790149+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.790149+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:35.256369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":507,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.473123+0000","last_change":"2024-04-26T16:30:09.475518+0000","last_active":"2024-04-26T16:31:31.473123+0000","last_peered":"2024-04-26T16:31:31.473123+0000","last_clean":"2024-04-26T16:31:31.473123+0000","last_became_active":"2024-04-26T16:26:23.201156+0000","last_became_peered":"2024-04-26T16:26:23.201156+0000","last_unstale":"2024-04-26T16:31:31.473123+0000","last_undegraded":"2024-04-26T16:31:31.473123+0000","last_fullsized":"2024-04-26T16:31:31.473123+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.475272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.475272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"388'25","reported_seq":1068,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446167+0000","last_change":"2024-04-26T16:28:52.098358+0000","last_active":"2024-04-26T16:31:31.446167+0000","last_peered":"2024-04-26T16:31:31.446167+0000","last_clean":"2024-04-26T16:31:31.446167+0000","last_became_active":"2024-04-26T16:28:25.258417+0000","last_became_peered":"2024-04-26T16:28:25.258417+0000","last_unstale":"2024-04-26T16:31:31.446167+0000","last_undegraded":"2024-04-26T16:31:31.446167+0000","last_fullsized":"2024-04-26T16:31:31.446167+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:28:52.098251+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.098251+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":507,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.469413+0000","last_change":"2024-04-26T16:30:14.930711+0000","last_active":"2024-04-26T16:31:31.469413+0000","last_peered":"2024-04-26T16:31:31.469413+0000","last_clean":"2024-04-26T16:31:31.469413+0000","last_became_active":"2024-04-26T16:26:23.212670+0000","last_became_peered":"2024-04-26T16:26:23.212670+0000","last_unstale":"2024-04-26T16:31:31.469413+0000","last_undegraded":"2024-04-26T16:31:31.469413+0000","last_fullsized":"2024-04-26T16:31:31.469413+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.930578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.930578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":868,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.445944+0000","last_change":"2024-04-26T16:28:43.126227+0000","last_active":"2024-04-26T16:31:31.445944+0000","last_peered":"2024-04-26T16:31:31.445944+0000","last_clean":"2024-04-26T16:31:31.445944+0000","last_became_active":"2024-04-26T16:28:25.245460+0000","last_became_peered":"2024-04-26T16:28:25.245460+0000","last_unstale":"2024-04-26T16:31:31.445944+0000","last_undegraded":"2024-04-26T16:31:31.445944+0000","last_fullsized":"2024-04-26T16:31:31.445944+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:43.125904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.125904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1025,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.475898+0000","last_change":"2024-04-26T16:31:26.379451+0000","last_active":"2024-04-26T16:31:31.475898+0000","last_peered":"2024-04-26T16:31:31.475898+0000","last_clean":"2024-04-26T16:31:31.475898+0000","last_became_active":"2024-04-26T16:26:23.185332+0000","last_became_peered":"2024-04-26T16:26:23.185332+0000","last_unstale":"2024-04-26T16:31:31.475898+0000","last_undegraded":"2024-04-26T16:31:31.475898+0000","last_fullsized":"2024-04-26T16:31:31.475898+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.379353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.379353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:33.121445+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.471522+0000","last_change":"2024-04-26T16:29:03.999225+0000","last_active":"2024-04-26T16:31:31.471522+0000","last_peered":"2024-04-26T16:31:31.471522+0000","last_clean":"2024-04-26T16:31:31.471522+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:31:31.471522+0000","last_undegraded":"2024-04-26T16:31:31.471522+0000","last_fullsized":"2024-04-26T16:31:31.471522+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:03.998965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.998965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'38","reported_seq":1092,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451234+0000","last_change":"2024-04-26T16:30:27.085082+0000","last_active":"2024-04-26T16:31:31.451234+0000","last_peered":"2024-04-26T16:31:31.451234+0000","last_clean":"2024-04-26T16:31:31.451234+0000","last_became_active":"2024-04-26T16:28:25.241767+0000","last_became_peered":"2024-04-26T16:28:25.241767+0000","last_unstale":"2024-04-26T16:31:31.451234+0000","last_undegraded":"2024-04-26T16:31:31.451234+0000","last_fullsized":"2024-04-26T16:31:31.451234+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-04-26T16:30:27.084844+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:27:38.188279+0000","last_clean_scrub_stamp":"2024-04-26T16:30:27.084844+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:41.150355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1006,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455688+0000","last_change":"2024-04-26T16:28:52.021759+0000","last_active":"2024-04-26T16:31:31.455688+0000","last_peered":"2024-04-26T16:31:31.455688+0000","last_clean":"2024-04-26T16:31:31.455688+0000","last_became_active":"2024-04-26T16:28:25.266762+0000","last_became_peered":"2024-04-26T16:28:25.266762+0000","last_unstale":"2024-04-26T16:31:31.455688+0000","last_undegraded":"2024-04-26T16:31:31.455688+0000","last_fullsized":"2024-04-26T16:31:31.455688+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.021548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.021548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":853,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452802+0000","last_change":"2024-04-26T16:29:10.997703+0000","last_active":"2024-04-26T16:31:31.452802+0000","last_peered":"2024-04-26T16:31:31.452802+0000","last_clean":"2024-04-26T16:31:31.452802+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:31:31.452802+0000","last_undegraded":"2024-04-26T16:31:31.452802+0000","last_fullsized":"2024-04-26T16:31:31.452802+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.997614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.997614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1020,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.467333+0000","last_change":"2024-04-26T16:31:22.830960+0000","last_active":"2024-04-26T16:31:31.467333+0000","last_peered":"2024-04-26T16:31:31.467333+0000","last_clean":"2024-04-26T16:31:31.467333+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:31:31.467333+0000","last_undegraded":"2024-04-26T16:31:31.467333+0000","last_fullsized":"2024-04-26T16:31:31.467333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:22.830853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:22.830853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.646479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'37","reported_seq":1084,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450120+0000","last_change":"2024-04-26T16:28:54.567147+0000","last_active":"2024-04-26T16:31:31.450120+0000","last_peered":"2024-04-26T16:31:31.450120+0000","last_clean":"2024-04-26T16:31:31.450120+0000","last_became_active":"2024-04-26T16:28:25.235694+0000","last_became_peered":"2024-04-26T16:28:25.235694+0000","last_unstale":"2024-04-26T16:31:31.450120+0000","last_undegraded":"2024-04-26T16:31:31.450120+0000","last_fullsized":"2024-04-26T16:31:31.450120+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-04-26T16:28:54.566895+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:28:54.566895+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.566895+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":523,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451758+0000","last_change":"2024-04-26T16:31:23.096196+0000","last_active":"2024-04-26T16:31:31.451758+0000","last_peered":"2024-04-26T16:31:31.451758+0000","last_clean":"2024-04-26T16:31:31.451758+0000","last_became_active":"2024-04-26T16:28:25.233622+0000","last_became_peered":"2024-04-26T16:28:25.233622+0000","last_unstale":"2024-04-26T16:31:31.451758+0000","last_undegraded":"2024-04-26T16:31:31.451758+0000","last_fullsized":"2024-04-26T16:31:31.451758+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.095870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.095870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:40.734359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":998,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.457294+0000","last_change":"2024-04-26T16:28:39.169317+0000","last_active":"2024-04-26T16:31:31.457294+0000","last_peered":"2024-04-26T16:31:31.457294+0000","last_clean":"2024-04-26T16:31:31.457294+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:31:31.457294+0000","last_undegraded":"2024-04-26T16:31:31.457294+0000","last_fullsized":"2024-04-26T16:31:31.457294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:39.169120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:39.169120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":871,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.474829+0000","last_change":"2024-04-26T16:30:19.533168+0000","last_active":"2024-04-26T16:31:31.474829+0000","last_peered":"2024-04-26T16:31:31.474829+0000","last_clean":"2024-04-26T16:31:31.474829+0000","last_became_active":"2024-04-26T16:28:25.241015+0000","last_became_peered":"2024-04-26T16:28:25.241015+0000","last_unstale":"2024-04-26T16:31:31.474829+0000","last_undegraded":"2024-04-26T16:31:31.474829+0000","last_fullsized":"2024-04-26T16:31:31.474829+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.533042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.533042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.752688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"433'16","reported_seq":1225,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468966+0000","last_change":"2024-04-26T16:31:16.716656+0000","last_active":"2024-04-26T16:31:31.468966+0000","last_peered":"2024-04-26T16:31:31.468966+0000","last_clean":"2024-04-26T16:31:31.468966+0000","last_became_active":"2024-04-26T16:28:25.228833+0000","last_became_peered":"2024-04-26T16:28:25.228833+0000","last_unstale":"2024-04-26T16:31:31.468966+0000","last_undegraded":"2024-04-26T16:31:31.468966+0000","last_fullsized":"2024-04-26T16:31:31.468966+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'16","last_scrub_stamp":"2024-04-26T16:31:16.715748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.715748+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:31.298504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1034,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454896+0000","last_change":"2024-04-26T16:30:02.951737+0000","last_active":"2024-04-26T16:31:31.454896+0000","last_peered":"2024-04-26T16:31:31.454896+0000","last_clean":"2024-04-26T16:31:31.454896+0000","last_became_active":"2024-04-26T16:26:23.187980+0000","last_became_peered":"2024-04-26T16:26:23.187980+0000","last_unstale":"2024-04-26T16:31:31.454896+0000","last_undegraded":"2024-04-26T16:31:31.454896+0000","last_fullsized":"2024-04-26T16:31:31.454896+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:30:02.951627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.951627+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"433'39","reported_seq":666,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455681+0000","last_change":"2024-04-26T16:30:08.947640+0000","last_active":"2024-04-26T16:31:31.455681+0000","last_peered":"2024-04-26T16:31:31.455681+0000","last_clean":"2024-04-26T16:31:31.455681+0000","last_became_active":"2024-04-26T16:28:25.228181+0000","last_became_peered":"2024-04-26T16:28:25.228181+0000","last_unstale":"2024-04-26T16:31:31.455681+0000","last_undegraded":"2024-04-26T16:31:31.455681+0000","last_fullsized":"2024-04-26T16:31:31.455681+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'39","last_scrub_stamp":"2024-04-26T16:30:08.947465+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.947465+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1029,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452476+0000","last_change":"2024-04-26T16:31:26.127314+0000","last_active":"2024-04-26T16:31:31.452476+0000","last_peered":"2024-04-26T16:31:31.452476+0000","last_clean":"2024-04-26T16:31:31.452476+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:31:31.452476+0000","last_undegraded":"2024-04-26T16:31:31.452476+0000","last_fullsized":"2024-04-26T16:31:31.452476+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.127067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.127067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.976272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1008,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.464413+0000","last_change":"2024-04-26T16:30:31.628236+0000","last_active":"2024-04-26T16:31:31.464413+0000","last_peered":"2024-04-26T16:31:31.464413+0000","last_clean":"2024-04-26T16:31:31.464413+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:31:31.464413+0000","last_undegraded":"2024-04-26T16:31:31.464413+0000","last_fullsized":"2024-04-26T16:31:31.464413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:31.627931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.627931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:37.885855+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":863,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.467718+0000","last_change":"2024-04-26T16:29:55.759332+0000","last_active":"2024-04-26T16:31:31.467718+0000","last_peered":"2024-04-26T16:31:31.467718+0000","last_clean":"2024-04-26T16:31:31.467718+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:31:31.467718+0000","last_undegraded":"2024-04-26T16:31:31.467718+0000","last_fullsized":"2024-04-26T16:31:31.467718+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:55.759069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:55.759069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1019,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.463287+0000","last_change":"2024-04-26T16:31:14.421824+0000","last_active":"2024-04-26T16:31:31.463287+0000","last_peered":"2024-04-26T16:31:31.463287+0000","last_clean":"2024-04-26T16:31:31.463287+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:31:31.463287+0000","last_undegraded":"2024-04-26T16:31:31.463287+0000","last_fullsized":"2024-04-26T16:31:31.463287+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:14.421630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:31:14.421630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:17.025811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"388'19","reported_seq":1044,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459062+0000","last_change":"2024-04-26T16:30:03.191223+0000","last_active":"2024-04-26T16:31:31.459062+0000","last_peered":"2024-04-26T16:31:31.459062+0000","last_clean":"2024-04-26T16:31:31.459062+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:31:31.459062+0000","last_undegraded":"2024-04-26T16:31:31.459062+0000","last_fullsized":"2024-04-26T16:31:31.459062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:03.190975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.190975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1044,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451242+0000","last_change":"2024-04-26T16:30:04.581495+0000","last_active":"2024-04-26T16:31:31.451242+0000","last_peered":"2024-04-26T16:31:31.451242+0000","last_clean":"2024-04-26T16:31:31.451242+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:31:31.451242+0000","last_undegraded":"2024-04-26T16:31:31.451242+0000","last_fullsized":"2024-04-26T16:31:31.451242+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:30:04.581361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.581361+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":998,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453525+0000","last_change":"2024-04-26T16:31:28.069974+0000","last_active":"2024-04-26T16:31:31.453525+0000","last_peered":"2024-04-26T16:31:31.453525+0000","last_clean":"2024-04-26T16:31:31.453525+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:31:31.453525+0000","last_undegraded":"2024-04-26T16:31:31.453525+0000","last_fullsized":"2024-04-26T16:31:31.453525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:28.069887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:28.069887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:48.574721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":863,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.465246+0000","last_change":"2024-04-26T16:30:03.519972+0000","last_active":"2024-04-26T16:31:31.465246+0000","last_peered":"2024-04-26T16:31:31.465246+0000","last_clean":"2024-04-26T16:31:31.465246+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:31:31.465246+0000","last_undegraded":"2024-04-26T16:31:31.465246+0000","last_fullsized":"2024-04-26T16:31:31.465246+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:03.519624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.519624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"388'19","reported_seq":1045,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453469+0000","last_change":"2024-04-26T16:29:00.045697+0000","last_active":"2024-04-26T16:31:31.453469+0000","last_peered":"2024-04-26T16:31:31.453469+0000","last_clean":"2024-04-26T16:31:31.453469+0000","last_became_active":"2024-04-26T16:28:25.270038+0000","last_became_peered":"2024-04-26T16:28:25.270038+0000","last_unstale":"2024-04-26T16:31:31.453469+0000","last_undegraded":"2024-04-26T16:31:31.453469+0000","last_fullsized":"2024-04-26T16:31:31.453469+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:00.045450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.045450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1027,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.460316+0000","last_change":"2024-04-26T16:31:25.411760+0000","last_active":"2024-04-26T16:31:31.460316+0000","last_peered":"2024-04-26T16:31:31.460316+0000","last_clean":"2024-04-26T16:31:31.460316+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:31:31.460316+0000","last_undegraded":"2024-04-26T16:31:31.460316+0000","last_fullsized":"2024-04-26T16:31:31.460316+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:25.411479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:31:25.411479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:53.171008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1001,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.469514+0000","last_change":"2024-04-26T16:30:28.910693+0000","last_active":"2024-04-26T16:31:31.469514+0000","last_peered":"2024-04-26T16:31:31.469514+0000","last_clean":"2024-04-26T16:31:31.469514+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:31:31.469514+0000","last_undegraded":"2024-04-26T16:31:31.469514+0000","last_fullsized":"2024-04-26T16:31:31.469514+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:42.057825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1006,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.473315+0000","last_change":"2024-04-26T16:30:07.455469+0000","last_active":"2024-04-26T16:31:31.473315+0000","last_peered":"2024-04-26T16:31:31.473315+0000","last_clean":"2024-04-26T16:31:31.473315+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:31:31.473315+0000","last_undegraded":"2024-04-26T16:31:31.473315+0000","last_fullsized":"2024-04-26T16:31:31.473315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.455245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.455245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":871,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.460111+0000","last_change":"2024-04-26T16:30:02.403961+0000","last_active":"2024-04-26T16:31:31.460111+0000","last_peered":"2024-04-26T16:31:31.460111+0000","last_clean":"2024-04-26T16:31:31.460111+0000","last_became_active":"2024-04-26T16:28:25.246039+0000","last_became_peered":"2024-04-26T16:28:25.246039+0000","last_unstale":"2024-04-26T16:31:31.460111+0000","last_undegraded":"2024-04-26T16:31:31.460111+0000","last_fullsized":"2024-04-26T16:31:31.460111+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:02.403733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.403733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:32.066798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"388'8","reported_seq":973,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.454661+0000","last_change":"2024-04-26T16:30:29.358181+0000","last_active":"2024-04-26T16:31:32.454661+0000","last_peered":"2024-04-26T16:31:32.454661+0000","last_clean":"2024-04-26T16:31:32.454661+0000","last_became_active":"2024-04-26T16:28:25.260556+0000","last_became_peered":"2024-04-26T16:28:25.260556+0000","last_unstale":"2024-04-26T16:31:32.454661+0000","last_undegraded":"2024-04-26T16:31:32.454661+0000","last_fullsized":"2024-04-26T16:31:32.454661+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'8","last_scrub_stamp":"2024-04-26T16:30:29.357863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.357863+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:36.666658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":902,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.454897+0000","last_change":"2024-04-26T16:30:28.356139+0000","last_active":"2024-04-26T16:31:32.454897+0000","last_peered":"2024-04-26T16:31:32.454897+0000","last_clean":"2024-04-26T16:31:32.454897+0000","last_became_active":"2024-04-26T16:28:25.242243+0000","last_became_peered":"2024-04-26T16:28:25.242243+0000","last_unstale":"2024-04-26T16:31:32.454897+0000","last_undegraded":"2024-04-26T16:31:32.454897+0000","last_fullsized":"2024-04-26T16:31:32.454897+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.355929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:58.034883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":507,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.456979+0000","last_change":"2024-04-26T16:28:47.096936+0000","last_active":"2024-04-26T16:31:31.456979+0000","last_peered":"2024-04-26T16:31:31.456979+0000","last_clean":"2024-04-26T16:31:31.456979+0000","last_became_active":"2024-04-26T16:28:25.243371+0000","last_became_peered":"2024-04-26T16:28:25.243371+0000","last_unstale":"2024-04-26T16:31:31.456979+0000","last_undegraded":"2024-04-26T16:31:31.456979+0000","last_fullsized":"2024-04-26T16:31:31.456979+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:47.096851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.096851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1005,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455586+0000","last_change":"2024-04-26T16:31:30.020041+0000","last_active":"2024-04-26T16:31:31.455586+0000","last_peered":"2024-04-26T16:31:31.455586+0000","last_clean":"2024-04-26T16:31:31.455586+0000","last_became_active":"2024-04-26T16:28:25.266217+0000","last_became_peered":"2024-04-26T16:28:25.266217+0000","last_unstale":"2024-04-26T16:31:31.455586+0000","last_undegraded":"2024-04-26T16:31:31.455586+0000","last_fullsized":"2024-04-26T16:31:31.455586+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.019680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.019680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.472819+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":869,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449785+0000","last_change":"2024-04-26T16:29:58.659839+0000","last_active":"2024-04-26T16:31:31.449785+0000","last_peered":"2024-04-26T16:31:31.449785+0000","last_clean":"2024-04-26T16:31:31.449785+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:31:31.449785+0000","last_undegraded":"2024-04-26T16:31:31.449785+0000","last_fullsized":"2024-04-26T16:31:31.449785+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.659641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.659641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'19","reported_seq":1053,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455065+0000","last_change":"2024-04-26T16:27:29.963129+0000","last_active":"2024-04-26T16:31:31.455065+0000","last_peered":"2024-04-26T16:31:31.455065+0000","last_clean":"2024-04-26T16:31:31.455065+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:31:31.455065+0000","last_undegraded":"2024-04-26T16:31:31.455065+0000","last_fullsized":"2024-04-26T16:31:31.455065+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:29.962882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.962882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":506,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446574+0000","last_change":"2024-04-26T16:30:07.968532+0000","last_active":"2024-04-26T16:31:31.446574+0000","last_peered":"2024-04-26T16:31:31.446574+0000","last_clean":"2024-04-26T16:31:31.446574+0000","last_became_active":"2024-04-26T16:26:23.216238+0000","last_became_peered":"2024-04-26T16:26:23.216238+0000","last_unstale":"2024-04-26T16:31:31.446574+0000","last_undegraded":"2024-04-26T16:31:31.446574+0000","last_fullsized":"2024-04-26T16:31:31.446574+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.968404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.968404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"316'13","reported_seq":653,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452210+0000","last_change":"2024-04-26T16:30:26.031295+0000","last_active":"2024-04-26T16:31:31.452210+0000","last_peered":"2024-04-26T16:31:31.452210+0000","last_clean":"2024-04-26T16:31:31.452210+0000","last_became_active":"2024-04-26T16:26:23.215841+0000","last_became_peered":"2024-04-26T16:26:23.215841+0000","last_unstale":"2024-04-26T16:31:31.452210+0000","last_undegraded":"2024-04-26T16:31:31.452210+0000","last_fullsized":"2024-04-26T16:31:31.452210+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:26.031142+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.031142+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.724917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1007,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468071+0000","last_change":"2024-04-26T16:30:22.624802+0000","last_active":"2024-04-26T16:31:31.468071+0000","last_peered":"2024-04-26T16:31:31.468071+0000","last_clean":"2024-04-26T16:31:31.468071+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:31:31.468071+0000","last_undegraded":"2024-04-26T16:31:31.468071+0000","last_fullsized":"2024-04-26T16:31:31.468071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.624646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.624646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:31.857935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":507,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446334+0000","last_change":"2024-04-26T16:28:46.104373+0000","last_active":"2024-04-26T16:31:31.446334+0000","last_peered":"2024-04-26T16:31:31.446334+0000","last_clean":"2024-04-26T16:31:31.446334+0000","last_became_active":"2024-04-26T16:28:25.241392+0000","last_became_peered":"2024-04-26T16:28:25.241392+0000","last_unstale":"2024-04-26T16:31:31.446334+0000","last_undegraded":"2024-04-26T16:31:31.446334+0000","last_fullsized":"2024-04-26T16:31:31.446334+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.104261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.104261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"388'19","reported_seq":1053,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.465942+0000","last_change":"2024-04-26T16:31:27.794473+0000","last_active":"2024-04-26T16:31:31.465942+0000","last_peered":"2024-04-26T16:31:31.465942+0000","last_clean":"2024-04-26T16:31:31.465942+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:31:31.465942+0000","last_undegraded":"2024-04-26T16:31:31.465942+0000","last_fullsized":"2024-04-26T16:31:31.465942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:31:27.794108+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:31:27.794108+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.668954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1020,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.472319+0000","last_change":"2024-04-26T16:29:10.518620+0000","last_active":"2024-04-26T16:31:31.472319+0000","last_peered":"2024-04-26T16:31:31.472319+0000","last_clean":"2024-04-26T16:31:31.472319+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:31:31.472319+0000","last_undegraded":"2024-04-26T16:31:31.472319+0000","last_fullsized":"2024-04-26T16:31:31.472319+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.518359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1180,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454702+0000","last_change":"2024-04-26T16:31:16.042144+0000","last_active":"2024-04-26T16:31:31.454702+0000","last_peered":"2024-04-26T16:31:31.454702+0000","last_clean":"2024-04-26T16:31:31.454702+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:31:31.454702+0000","last_undegraded":"2024-04-26T16:31:31.454702+0000","last_fullsized":"2024-04-26T16:31:31.454702+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:31:16.041975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.041975+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:30.595842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1013,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451403+0000","last_change":"2024-04-26T16:30:00.630668+0000","last_active":"2024-04-26T16:31:31.451403+0000","last_peered":"2024-04-26T16:31:31.451403+0000","last_clean":"2024-04-26T16:31:31.451403+0000","last_became_active":"2024-04-26T16:26:23.188629+0000","last_became_peered":"2024-04-26T16:26:23.188629+0000","last_unstale":"2024-04-26T16:31:31.451403+0000","last_undegraded":"2024-04-26T16:31:31.451403+0000","last_fullsized":"2024-04-26T16:31:31.451403+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.630569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.630569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":852,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459014+0000","last_change":"2024-04-26T16:27:25.909568+0000","last_active":"2024-04-26T16:31:31.459014+0000","last_peered":"2024-04-26T16:31:31.459014+0000","last_clean":"2024-04-26T16:31:31.459014+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:31:31.459014+0000","last_undegraded":"2024-04-26T16:31:31.459014+0000","last_fullsized":"2024-04-26T16:31:31.459014+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:25.909462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:25.909462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"388'19","reported_seq":1033,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459008+0000","last_change":"2024-04-26T16:28:43.978963+0000","last_active":"2024-04-26T16:31:31.459008+0000","last_peered":"2024-04-26T16:31:31.459008+0000","last_clean":"2024-04-26T16:31:31.459008+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:31:31.459008+0000","last_undegraded":"2024-04-26T16:31:31.459008+0000","last_fullsized":"2024-04-26T16:31:31.459008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:28:43.978726+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.978726+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":912,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459021+0000","last_change":"2024-04-26T16:31:26.231532+0000","last_active":"2024-04-26T16:31:32.459021+0000","last_peered":"2024-04-26T16:31:32.459021+0000","last_clean":"2024-04-26T16:31:32.459021+0000","last_became_active":"2024-04-26T16:28:25.270839+0000","last_became_peered":"2024-04-26T16:28:25.270839+0000","last_unstale":"2024-04-26T16:31:32.459021+0000","last_undegraded":"2024-04-26T16:31:32.459021+0000","last_fullsized":"2024-04-26T16:31:32.459021+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.231313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.231313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.537096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1001,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453491+0000","last_change":"2024-04-26T16:29:00.063758+0000","last_active":"2024-04-26T16:31:31.453491+0000","last_peered":"2024-04-26T16:31:31.453491+0000","last_clean":"2024-04-26T16:31:31.453491+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:31:31.453491+0000","last_undegraded":"2024-04-26T16:31:31.453491+0000","last_fullsized":"2024-04-26T16:31:31.453491+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:00.063648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.063648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1022,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.464291+0000","last_change":"2024-04-26T16:30:12.648490+0000","last_active":"2024-04-26T16:31:31.464291+0000","last_peered":"2024-04-26T16:31:31.464291+0000","last_clean":"2024-04-26T16:31:31.464291+0000","last_became_active":"2024-04-26T16:28:25.256886+0000","last_became_peered":"2024-04-26T16:28:25.256886+0000","last_unstale":"2024-04-26T16:31:31.464291+0000","last_undegraded":"2024-04-26T16:31:31.464291+0000","last_fullsized":"2024-04-26T16:31:31.464291+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.648271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.648271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":761,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459289+0000","last_change":"2024-04-26T16:31:20.323922+0000","last_active":"2024-04-26T16:31:32.459289+0000","last_peered":"2024-04-26T16:31:32.459289+0000","last_clean":"2024-04-26T16:31:32.459289+0000","last_became_active":"2024-04-26T16:28:25.270409+0000","last_became_peered":"2024-04-26T16:28:25.270409+0000","last_unstale":"2024-04-26T16:31:32.459289+0000","last_undegraded":"2024-04-26T16:31:32.459289+0000","last_fullsized":"2024-04-26T16:31:32.459289+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:20.323822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:20.323822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:42.514922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"388'20","reported_seq":1054,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449994+0000","last_change":"2024-04-26T16:28:47.517956+0000","last_active":"2024-04-26T16:31:31.449994+0000","last_peered":"2024-04-26T16:31:31.449994+0000","last_clean":"2024-04-26T16:31:31.449994+0000","last_became_active":"2024-04-26T16:28:25.258640+0000","last_became_peered":"2024-04-26T16:28:25.258640+0000","last_unstale":"2024-04-26T16:31:31.449994+0000","last_undegraded":"2024-04-26T16:31:31.449994+0000","last_fullsized":"2024-04-26T16:31:31.449994+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:47.517723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.517723+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":522,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451869+0000","last_change":"2024-04-26T16:30:21.968003+0000","last_active":"2024-04-26T16:31:31.451869+0000","last_peered":"2024-04-26T16:31:31.451869+0000","last_clean":"2024-04-26T16:31:31.451869+0000","last_became_active":"2024-04-26T16:28:25.226182+0000","last_became_peered":"2024-04-26T16:28:25.226182+0000","last_unstale":"2024-04-26T16:31:31.451869+0000","last_undegraded":"2024-04-26T16:31:31.451869+0000","last_fullsized":"2024-04-26T16:31:31.451869+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:21.967758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.967758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.703747+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1019,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.454930+0000","last_change":"2024-04-26T16:31:19.090724+0000","last_active":"2024-04-26T16:31:31.454930+0000","last_peered":"2024-04-26T16:31:31.454930+0000","last_clean":"2024-04-26T16:31:31.454930+0000","last_became_active":"2024-04-26T16:26:23.213441+0000","last_became_peered":"2024-04-26T16:26:23.213441+0000","last_unstale":"2024-04-26T16:31:31.454930+0000","last_undegraded":"2024-04-26T16:31:31.454930+0000","last_fullsized":"2024-04-26T16:31:31.454930+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:19.090521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.090521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.095425+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1003,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.447680+0000","last_change":"2024-04-26T16:29:09.036096+0000","last_active":"2024-04-26T16:31:31.447680+0000","last_peered":"2024-04-26T16:31:31.447680+0000","last_clean":"2024-04-26T16:31:31.447680+0000","last_became_active":"2024-04-26T16:26:23.211857+0000","last_became_peered":"2024-04-26T16:26:23.211857+0000","last_unstale":"2024-04-26T16:31:31.447680+0000","last_undegraded":"2024-04-26T16:31:31.447680+0000","last_fullsized":"2024-04-26T16:31:31.447680+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.035817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.035817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.475486+0000","last_change":"2024-04-26T16:30:15.529255+0000","last_active":"2024-04-26T16:31:31.475486+0000","last_peered":"2024-04-26T16:31:31.475486+0000","last_clean":"2024-04-26T16:31:31.475486+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:31:31.475486+0000","last_undegraded":"2024-04-26T16:31:31.475486+0000","last_fullsized":"2024-04-26T16:31:31.475486+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.529105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.529105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'12","reported_seq":1064,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468611+0000","last_change":"2024-04-26T16:31:20.801299+0000","last_active":"2024-04-26T16:31:31.468611+0000","last_peered":"2024-04-26T16:31:31.468611+0000","last_clean":"2024-04-26T16:31:31.468611+0000","last_became_active":"2024-04-26T16:28:25.246218+0000","last_became_peered":"2024-04-26T16:28:25.246218+0000","last_unstale":"2024-04-26T16:31:31.468611+0000","last_undegraded":"2024-04-26T16:31:31.468611+0000","last_fullsized":"2024-04-26T16:31:31.468611+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'12","last_scrub_stamp":"2024-04-26T16:31:20.801180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:20.801180+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:43.238767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":384,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459412+0000","last_change":"2024-04-26T16:30:36.171558+0000","last_active":"2024-04-26T16:31:32.459412+0000","last_peered":"2024-04-26T16:31:32.459412+0000","last_clean":"2024-04-26T16:31:32.459412+0000","last_became_active":"2024-04-26T16:28:25.231296+0000","last_became_peered":"2024-04-26T16:28:25.231296+0000","last_unstale":"2024-04-26T16:31:32.459412+0000","last_undegraded":"2024-04-26T16:31:32.459412+0000","last_fullsized":"2024-04-26T16:31:32.459412+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:36.171451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:36.171451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.052865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":942,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.455216+0000","last_change":"2024-04-26T16:30:19.325201+0000","last_active":"2024-04-26T16:31:32.455216+0000","last_peered":"2024-04-26T16:31:32.455216+0000","last_clean":"2024-04-26T16:31:32.455216+0000","last_became_active":"2024-04-26T16:28:25.272341+0000","last_became_peered":"2024-04-26T16:28:25.272341+0000","last_unstale":"2024-04-26T16:31:32.455216+0000","last_undegraded":"2024-04-26T16:31:32.455216+0000","last_fullsized":"2024-04-26T16:31:32.455216+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.325091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.325091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.244723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":996,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453781+0000","last_change":"2024-04-26T16:30:34.003108+0000","last_active":"2024-04-26T16:31:31.453781+0000","last_peered":"2024-04-26T16:31:31.453781+0000","last_clean":"2024-04-26T16:31:31.453781+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:31:31.453781+0000","last_undegraded":"2024-04-26T16:31:31.453781+0000","last_fullsized":"2024-04-26T16:31:31.453781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:34.002980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.002980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.133005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459306+0000","last_change":"2024-04-26T16:28:35.971437+0000","last_active":"2024-04-26T16:31:31.459306+0000","last_peered":"2024-04-26T16:31:31.459306+0000","last_clean":"2024-04-26T16:31:31.459306+0000","last_became_active":"2024-04-26T16:28:25.269747+0000","last_became_peered":"2024-04-26T16:28:25.269747+0000","last_unstale":"2024-04-26T16:31:31.459306+0000","last_undegraded":"2024-04-26T16:31:31.459306+0000","last_fullsized":"2024-04-26T16:31:31.459306+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:35.971222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:35.971222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'15","reported_seq":558,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.459290+0000","last_change":"2024-04-26T16:27:43.107066+0000","last_active":"2024-04-26T16:31:31.459290+0000","last_peered":"2024-04-26T16:31:31.459290+0000","last_clean":"2024-04-26T16:31:31.459290+0000","last_became_active":"2024-04-26T16:26:23.200216+0000","last_became_peered":"2024-04-26T16:26:23.200216+0000","last_unstale":"2024-04-26T16:31:31.459290+0000","last_undegraded":"2024-04-26T16:31:31.459290+0000","last_fullsized":"2024-04-26T16:31:31.459290+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:27:43.106910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:43.106910+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":923,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459309+0000","last_change":"2024-04-26T16:31:17.224683+0000","last_active":"2024-04-26T16:31:32.459309+0000","last_peered":"2024-04-26T16:31:32.459309+0000","last_clean":"2024-04-26T16:31:32.459309+0000","last_became_active":"2024-04-26T16:28:25.244917+0000","last_became_peered":"2024-04-26T16:28:25.244917+0000","last_unstale":"2024-04-26T16:31:32.459309+0000","last_undegraded":"2024-04-26T16:31:32.459309+0000","last_fullsized":"2024-04-26T16:31:32.459309+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:17.224543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:31:17.224543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:22.633061+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1010,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453868+0000","last_change":"2024-04-26T16:28:57.063805+0000","last_active":"2024-04-26T16:31:31.453868+0000","last_peered":"2024-04-26T16:31:31.453868+0000","last_clean":"2024-04-26T16:31:31.453868+0000","last_became_active":"2024-04-26T16:28:25.266296+0000","last_became_peered":"2024-04-26T16:28:25.266296+0000","last_unstale":"2024-04-26T16:31:31.453868+0000","last_undegraded":"2024-04-26T16:31:31.453868+0000","last_fullsized":"2024-04-26T16:31:31.453868+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:57.063594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:57.063594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1006,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453622+0000","last_change":"2024-04-26T16:30:09.970620+0000","last_active":"2024-04-26T16:31:31.453622+0000","last_peered":"2024-04-26T16:31:31.453622+0000","last_clean":"2024-04-26T16:31:31.453622+0000","last_became_active":"2024-04-26T16:28:25.228971+0000","last_became_peered":"2024-04-26T16:28:25.228971+0000","last_unstale":"2024-04-26T16:31:31.453622+0000","last_undegraded":"2024-04-26T16:31:31.453622+0000","last_fullsized":"2024-04-26T16:31:31.453622+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.970505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.970505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:39.666703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.473281+0000","last_change":"2024-04-26T16:30:22.616210+0000","last_active":"2024-04-26T16:31:31.473281+0000","last_peered":"2024-04-26T16:31:31.473281+0000","last_clean":"2024-04-26T16:31:31.473281+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:31:31.473281+0000","last_undegraded":"2024-04-26T16:31:31.473281+0000","last_fullsized":"2024-04-26T16:31:31.473281+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.616065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.616065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"388'34","reported_seq":1066,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449395+0000","last_change":"2024-04-26T16:29:04.788890+0000","last_active":"2024-04-26T16:31:31.449395+0000","last_peered":"2024-04-26T16:31:31.449395+0000","last_clean":"2024-04-26T16:31:31.449395+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:31:31.449395+0000","last_undegraded":"2024-04-26T16:31:31.449395+0000","last_fullsized":"2024-04-26T16:31:31.449395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:29:04.788618+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.788618+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":522,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449349+0000","last_change":"2024-04-26T16:29:07.493539+0000","last_active":"2024-04-26T16:31:31.449349+0000","last_peered":"2024-04-26T16:31:31.449349+0000","last_clean":"2024-04-26T16:31:31.449349+0000","last_became_active":"2024-04-26T16:28:25.250387+0000","last_became_peered":"2024-04-26T16:28:25.250387+0000","last_unstale":"2024-04-26T16:31:31.449349+0000","last_undegraded":"2024-04-26T16:31:31.449349+0000","last_fullsized":"2024-04-26T16:31:31.449349+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.493409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.493409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"433'14","reported_seq":1079,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459176+0000","last_change":"2024-04-26T16:31:21.319007+0000","last_active":"2024-04-26T16:31:32.459176+0000","last_peered":"2024-04-26T16:31:32.459176+0000","last_clean":"2024-04-26T16:31:32.459176+0000","last_became_active":"2024-04-26T16:28:25.265640+0000","last_became_peered":"2024-04-26T16:28:25.265640+0000","last_unstale":"2024-04-26T16:31:32.459176+0000","last_undegraded":"2024-04-26T16:31:32.459176+0000","last_fullsized":"2024-04-26T16:31:32.459176+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:31:21.318863+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:31:21.318863+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:28.958833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":1008,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.458896+0000","last_change":"2024-04-26T16:30:11.234876+0000","last_active":"2024-04-26T16:31:31.458896+0000","last_peered":"2024-04-26T16:31:31.458896+0000","last_clean":"2024-04-26T16:31:31.458896+0000","last_became_active":"2024-04-26T16:26:23.201293+0000","last_became_peered":"2024-04-26T16:26:23.201293+0000","last_unstale":"2024-04-26T16:31:31.458896+0000","last_undegraded":"2024-04-26T16:31:31.458896+0000","last_fullsized":"2024-04-26T16:31:31.458896+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:11.234642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.234642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":872,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.466476+0000","last_change":"2024-04-26T16:30:16.615509+0000","last_active":"2024-04-26T16:31:31.466476+0000","last_peered":"2024-04-26T16:31:31.466476+0000","last_clean":"2024-04-26T16:31:31.466476+0000","last_became_active":"2024-04-26T16:28:25.247238+0000","last_became_peered":"2024-04-26T16:28:25.247238+0000","last_unstale":"2024-04-26T16:31:31.466476+0000","last_undegraded":"2024-04-26T16:31:31.466476+0000","last_fullsized":"2024-04-26T16:31:31.466476+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.615354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.615354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:40.463397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"388'15","reported_seq":1030,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.446098+0000","last_change":"2024-04-26T16:28:59.039798+0000","last_active":"2024-04-26T16:31:31.446098+0000","last_peered":"2024-04-26T16:31:31.446098+0000","last_clean":"2024-04-26T16:31:31.446098+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:31:31.446098+0000","last_undegraded":"2024-04-26T16:31:31.446098+0000","last_fullsized":"2024-04-26T16:31:31.446098+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:28:59.039575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.039575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1020,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.447908+0000","last_change":"2024-04-26T16:31:17.081118+0000","last_active":"2024-04-26T16:31:31.447908+0000","last_peered":"2024-04-26T16:31:31.447908+0000","last_clean":"2024-04-26T16:31:31.447908+0000","last_became_active":"2024-04-26T16:28:25.245293+0000","last_became_peered":"2024-04-26T16:28:25.245293+0000","last_unstale":"2024-04-26T16:31:31.447908+0000","last_undegraded":"2024-04-26T16:31:31.447908+0000","last_fullsized":"2024-04-26T16:31:31.447908+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:31:17.080989+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:31:17.080989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.286820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1020,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.467938+0000","last_change":"2024-04-26T16:30:28.566079+0000","last_active":"2024-04-26T16:31:31.467938+0000","last_peered":"2024-04-26T16:31:31.467938+0000","last_clean":"2024-04-26T16:31:31.467938+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:31:31.467938+0000","last_undegraded":"2024-04-26T16:31:31.467938+0000","last_fullsized":"2024-04-26T16:31:31.467938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.565983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.565983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:44.718464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1014,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452574+0000","last_change":"2024-04-26T16:28:46.866392+0000","last_active":"2024-04-26T16:31:31.452574+0000","last_peered":"2024-04-26T16:31:31.452574+0000","last_clean":"2024-04-26T16:31:31.452574+0000","last_became_active":"2024-04-26T16:26:23.203386+0000","last_became_peered":"2024-04-26T16:26:23.203386+0000","last_unstale":"2024-04-26T16:31:31.452574+0000","last_undegraded":"2024-04-26T16:31:31.452574+0000","last_fullsized":"2024-04-26T16:31:31.452574+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.866222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.866222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":522,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449904+0000","last_change":"2024-04-26T16:30:29.497506+0000","last_active":"2024-04-26T16:31:31.449904+0000","last_peered":"2024-04-26T16:31:31.449904+0000","last_clean":"2024-04-26T16:31:31.449904+0000","last_became_active":"2024-04-26T16:28:25.238120+0000","last_became_peered":"2024-04-26T16:28:25.238120+0000","last_unstale":"2024-04-26T16:31:31.449904+0000","last_undegraded":"2024-04-26T16:31:31.449904+0000","last_fullsized":"2024-04-26T16:31:31.449904+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.497411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.497411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:49.042909+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"388'9","reported_seq":1030,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468744+0000","last_change":"2024-04-26T16:30:02.530560+0000","last_active":"2024-04-26T16:31:31.468744+0000","last_peered":"2024-04-26T16:31:31.468744+0000","last_clean":"2024-04-26T16:31:31.468744+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:31:31.468744+0000","last_undegraded":"2024-04-26T16:31:31.468744+0000","last_fullsized":"2024-04-26T16:31:31.468744+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:30:02.530372+0000","last_deep_scrub":"114'7","last_deep_scrub_stamp":"2024-04-26T16:23:25.467392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.530372+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":966,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.457820+0000","last_change":"2024-04-26T16:30:23.313786+0000","last_active":"2024-04-26T16:31:32.457820+0000","last_peered":"2024-04-26T16:31:32.457820+0000","last_clean":"2024-04-26T16:31:32.457820+0000","last_became_active":"2024-04-26T16:28:25.264839+0000","last_became_peered":"2024-04-26T16:28:25.264839+0000","last_unstale":"2024-04-26T16:31:32.457820+0000","last_undegraded":"2024-04-26T16:31:32.457820+0000","last_fullsized":"2024-04-26T16:31:32.457820+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.313672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.313672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1020,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.452428+0000","last_change":"2024-04-26T16:31:21.029710+0000","last_active":"2024-04-26T16:31:31.452428+0000","last_peered":"2024-04-26T16:31:31.452428+0000","last_clean":"2024-04-26T16:31:31.452428+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:31:31.452428+0000","last_undegraded":"2024-04-26T16:31:31.452428+0000","last_fullsized":"2024-04-26T16:31:31.452428+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:21.029556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:21.029556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.610748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1031,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450872+0000","last_change":"2024-04-26T16:30:26.547964+0000","last_active":"2024-04-26T16:31:31.450872+0000","last_peered":"2024-04-26T16:31:31.450872+0000","last_clean":"2024-04-26T16:31:31.450872+0000","last_became_active":"2024-04-26T16:28:25.244502+0000","last_became_peered":"2024-04-26T16:28:25.244502+0000","last_unstale":"2024-04-26T16:31:31.450872+0000","last_undegraded":"2024-04-26T16:31:31.450872+0000","last_fullsized":"2024-04-26T16:31:31.450872+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.547756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.547756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.908182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":852,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453293+0000","last_change":"2024-04-26T16:27:44.070368+0000","last_active":"2024-04-26T16:31:31.453293+0000","last_peered":"2024-04-26T16:31:31.453293+0000","last_clean":"2024-04-26T16:31:31.453293+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:31:31.453293+0000","last_undegraded":"2024-04-26T16:31:31.453293+0000","last_fullsized":"2024-04-26T16:31:31.453293+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:27:44.070141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'15","reported_seq":1048,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449936+0000","last_change":"2024-04-26T16:30:18.494734+0000","last_active":"2024-04-26T16:31:31.449936+0000","last_peered":"2024-04-26T16:31:31.449936+0000","last_clean":"2024-04-26T16:31:31.449936+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:31:31.449936+0000","last_undegraded":"2024-04-26T16:31:31.449936+0000","last_fullsized":"2024-04-26T16:31:31.449936+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:30:18.494624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.494624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1018,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453227+0000","last_change":"2024-04-26T16:31:18.051005+0000","last_active":"2024-04-26T16:31:31.453227+0000","last_peered":"2024-04-26T16:31:31.453227+0000","last_clean":"2024-04-26T16:31:31.453227+0000","last_became_active":"2024-04-26T16:28:25.255455+0000","last_became_peered":"2024-04-26T16:28:25.255455+0000","last_unstale":"2024-04-26T16:31:31.453227+0000","last_undegraded":"2024-04-26T16:31:31.453227+0000","last_fullsized":"2024-04-26T16:31:31.453227+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:18.050865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:18.050865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:20.999045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1011,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450070+0000","last_change":"2024-04-26T16:29:04.486590+0000","last_active":"2024-04-26T16:31:31.450070+0000","last_peered":"2024-04-26T16:31:31.450070+0000","last_clean":"2024-04-26T16:31:31.450070+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:31:31.450070+0000","last_undegraded":"2024-04-26T16:31:31.450070+0000","last_fullsized":"2024-04-26T16:31:31.450070+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:04.486382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.486382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1013,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.469311+0000","last_change":"2024-04-26T16:30:26.949591+0000","last_active":"2024-04-26T16:31:31.469311+0000","last_peered":"2024-04-26T16:31:31.469311+0000","last_clean":"2024-04-26T16:31:31.469311+0000","last_became_active":"2024-04-26T16:28:25.245307+0000","last_became_peered":"2024-04-26T16:28:25.245307+0000","last_unstale":"2024-04-26T16:31:31.469311+0000","last_undegraded":"2024-04-26T16:31:31.469311+0000","last_fullsized":"2024-04-26T16:31:31.469311+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.949357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.949357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":862,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.474889+0000","last_change":"2024-04-26T16:30:04.495577+0000","last_active":"2024-04-26T16:31:31.474889+0000","last_peered":"2024-04-26T16:31:31.474889+0000","last_clean":"2024-04-26T16:31:31.474889+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:31:31.474889+0000","last_undegraded":"2024-04-26T16:31:31.474889+0000","last_fullsized":"2024-04-26T16:31:31.474889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:04.495235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.495235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"388'23","reported_seq":686,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450771+0000","last_change":"2024-04-26T16:28:45.510060+0000","last_active":"2024-04-26T16:31:31.450771+0000","last_peered":"2024-04-26T16:31:31.450771+0000","last_clean":"2024-04-26T16:31:31.450771+0000","last_became_active":"2024-04-26T16:28:25.250379+0000","last_became_peered":"2024-04-26T16:28:25.250379+0000","last_unstale":"2024-04-26T16:31:31.450771+0000","last_undegraded":"2024-04-26T16:31:31.450771+0000","last_fullsized":"2024-04-26T16:31:31.450771+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:28:45.509872+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:28:45.509872+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1023,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455035+0000","last_change":"2024-04-26T16:29:02.113592+0000","last_active":"2024-04-26T16:31:31.455035+0000","last_peered":"2024-04-26T16:31:31.455035+0000","last_clean":"2024-04-26T16:31:31.455035+0000","last_became_active":"2024-04-26T16:28:25.277266+0000","last_became_peered":"2024-04-26T16:28:25.277266+0000","last_unstale":"2024-04-26T16:31:31.455035+0000","last_undegraded":"2024-04-26T16:31:31.455035+0000","last_fullsized":"2024-04-26T16:31:31.455035+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:02.113366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.113366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":508,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468319+0000","last_change":"2024-04-26T16:30:11.600252+0000","last_active":"2024-04-26T16:31:31.468319+0000","last_peered":"2024-04-26T16:31:31.468319+0000","last_clean":"2024-04-26T16:31:31.468319+0000","last_became_active":"2024-04-26T16:26:23.166910+0000","last_became_peered":"2024-04-26T16:26:23.166910+0000","last_unstale":"2024-04-26T16:31:31.468319+0000","last_undegraded":"2024-04-26T16:31:31.468319+0000","last_fullsized":"2024-04-26T16:31:31.468319+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:11.600054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.600054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1007,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.464800+0000","last_change":"2024-04-26T16:30:24.568125+0000","last_active":"2024-04-26T16:31:31.464800+0000","last_peered":"2024-04-26T16:31:31.464800+0000","last_clean":"2024-04-26T16:31:31.464800+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:31:31.464800+0000","last_undegraded":"2024-04-26T16:31:31.464800+0000","last_fullsized":"2024-04-26T16:31:31.464800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.567773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:36.702289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":783,"reported_epoch":558,"state":"active+clean","last_fresh":"2024-04-26T16:31:32.459949+0000","last_change":"2024-04-26T16:31:29.175004+0000","last_active":"2024-04-26T16:31:32.459949+0000","last_peered":"2024-04-26T16:31:32.459949+0000","last_clean":"2024-04-26T16:31:32.459949+0000","last_became_active":"2024-04-26T16:28:25.255571+0000","last_became_peered":"2024-04-26T16:28:25.255571+0000","last_unstale":"2024-04-26T16:31:32.459949+0000","last_undegraded":"2024-04-26T16:31:32.459949+0000","last_fullsized":"2024-04-26T16:31:32.459949+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:29.174775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:29.174775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.408020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"388'27","reported_seq":600,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449873+0000","last_change":"2024-04-26T16:30:13.510016+0000","last_active":"2024-04-26T16:31:31.449873+0000","last_peered":"2024-04-26T16:31:31.449873+0000","last_clean":"2024-04-26T16:31:31.449873+0000","last_became_active":"2024-04-26T16:28:25.256896+0000","last_became_peered":"2024-04-26T16:28:25.256896+0000","last_unstale":"2024-04-26T16:31:31.449873+0000","last_undegraded":"2024-04-26T16:31:31.449873+0000","last_fullsized":"2024-04-26T16:31:31.449873+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:13.509869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.509869+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:38.781981+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1025,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.456114+0000","last_change":"2024-04-26T16:30:13.187491+0000","last_active":"2024-04-26T16:31:31.456114+0000","last_peered":"2024-04-26T16:31:31.456114+0000","last_clean":"2024-04-26T16:31:31.456114+0000","last_became_active":"2024-04-26T16:28:25.271550+0000","last_became_peered":"2024-04-26T16:28:25.271550+0000","last_unstale":"2024-04-26T16:31:31.456114+0000","last_undegraded":"2024-04-26T16:31:31.456114+0000","last_fullsized":"2024-04-26T16:31:31.456114+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:13.187197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.187197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1000,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.453538+0000","last_change":"2024-04-26T16:29:57.141987+0000","last_active":"2024-04-26T16:31:31.453538+0000","last_peered":"2024-04-26T16:31:31.453538+0000","last_clean":"2024-04-26T16:31:31.453538+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:31:31.453538+0000","last_undegraded":"2024-04-26T16:31:31.453538+0000","last_fullsized":"2024-04-26T16:31:31.453538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:57.141873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.141873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":530,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.450433+0000","last_change":"2024-04-26T16:30:34.033387+0000","last_active":"2024-04-26T16:31:31.450433+0000","last_peered":"2024-04-26T16:31:31.450433+0000","last_clean":"2024-04-26T16:31:31.450433+0000","last_became_active":"2024-04-26T16:26:23.212770+0000","last_became_peered":"2024-04-26T16:26:23.212770+0000","last_unstale":"2024-04-26T16:31:31.450433+0000","last_undegraded":"2024-04-26T16:31:31.450433+0000","last_fullsized":"2024-04-26T16:31:31.450433+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:34.033282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.033282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:45.039967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":875,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.455907+0000","last_change":"2024-04-26T16:30:14.201574+0000","last_active":"2024-04-26T16:31:31.455907+0000","last_peered":"2024-04-26T16:31:31.455907+0000","last_clean":"2024-04-26T16:31:31.455907+0000","last_became_active":"2024-04-26T16:28:25.267368+0000","last_became_peered":"2024-04-26T16:28:25.267368+0000","last_unstale":"2024-04-26T16:31:31.455907+0000","last_undegraded":"2024-04-26T16:31:31.455907+0000","last_fullsized":"2024-04-26T16:31:31.455907+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.201493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.201493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"388'19","reported_seq":1082,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468751+0000","last_change":"2024-04-26T16:31:26.779377+0000","last_active":"2024-04-26T16:31:31.468751+0000","last_peered":"2024-04-26T16:31:31.468751+0000","last_clean":"2024-04-26T16:31:31.468751+0000","last_became_active":"2024-04-26T16:28:25.244993+0000","last_became_peered":"2024-04-26T16:28:25.244993+0000","last_unstale":"2024-04-26T16:31:31.468751+0000","last_undegraded":"2024-04-26T16:31:31.468751+0000","last_fullsized":"2024-04-26T16:31:31.468751+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:31:26.779258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.779258+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.712424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1025,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.457226+0000","last_change":"2024-04-26T16:30:32.981558+0000","last_active":"2024-04-26T16:31:31.457226+0000","last_peered":"2024-04-26T16:31:31.457226+0000","last_clean":"2024-04-26T16:31:31.457226+0000","last_became_active":"2024-04-26T16:26:23.214404+0000","last_became_peered":"2024-04-26T16:26:23.214404+0000","last_unstale":"2024-04-26T16:31:31.457226+0000","last_undegraded":"2024-04-26T16:31:31.457226+0000","last_fullsized":"2024-04-26T16:31:31.457226+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:32.981351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:32.981351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:35.689430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1026,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449524+0000","last_change":"2024-04-26T16:31:20.057831+0000","last_active":"2024-04-26T16:31:31.449524+0000","last_peered":"2024-04-26T16:31:31.449524+0000","last_clean":"2024-04-26T16:31:31.449524+0000","last_became_active":"2024-04-26T16:26:23.178678+0000","last_became_peered":"2024-04-26T16:26:23.178678+0000","last_unstale":"2024-04-26T16:31:31.449524+0000","last_undegraded":"2024-04-26T16:31:31.449524+0000","last_fullsized":"2024-04-26T16:31:31.449524+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:20.057515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:20.057515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:31.293526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1021,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449751+0000","last_change":"2024-04-26T16:31:23.471502+0000","last_active":"2024-04-26T16:31:31.449751+0000","last_peered":"2024-04-26T16:31:31.449751+0000","last_clean":"2024-04-26T16:31:31.449751+0000","last_became_active":"2024-04-26T16:26:23.198295+0000","last_became_peered":"2024-04-26T16:26:23.198295+0000","last_unstale":"2024-04-26T16:31:31.449751+0000","last_undegraded":"2024-04-26T16:31:31.449751+0000","last_fullsized":"2024-04-26T16:31:31.449751+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.471197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.471197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:50.817089+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":504,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.458921+0000","last_change":"2024-04-26T16:30:16.192009+0000","last_active":"2024-04-26T16:31:31.458921+0000","last_peered":"2024-04-26T16:31:31.458921+0000","last_clean":"2024-04-26T16:31:31.458921+0000","last_became_active":"2024-04-26T16:26:23.199414+0000","last_became_peered":"2024-04-26T16:26:23.199414+0000","last_unstale":"2024-04-26T16:31:31.458921+0000","last_undegraded":"2024-04-26T16:31:31.458921+0000","last_fullsized":"2024-04-26T16:31:31.458921+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.191696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.191696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"388'20","reported_seq":1045,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.457567+0000","last_change":"2024-04-26T16:28:48.140271+0000","last_active":"2024-04-26T16:31:31.457567+0000","last_peered":"2024-04-26T16:31:31.457567+0000","last_clean":"2024-04-26T16:31:31.457567+0000","last_became_active":"2024-04-26T16:28:25.232220+0000","last_became_peered":"2024-04-26T16:28:25.232220+0000","last_unstale":"2024-04-26T16:31:31.457567+0000","last_undegraded":"2024-04-26T16:31:31.457567+0000","last_fullsized":"2024-04-26T16:31:31.457567+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:48.140161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:48.140161+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1028,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.468854+0000","last_change":"2024-04-26T16:30:33.689694+0000","last_active":"2024-04-26T16:31:31.468854+0000","last_peered":"2024-04-26T16:31:31.468854+0000","last_clean":"2024-04-26T16:31:31.468854+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:31:31.468854+0000","last_undegraded":"2024-04-26T16:31:31.468854+0000","last_fullsized":"2024-04-26T16:31:31.468854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.689564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.689564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:02.010334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1028,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.449207+0000","last_change":"2024-04-26T16:30:22.493662+0000","last_active":"2024-04-26T16:31:31.449207+0000","last_peered":"2024-04-26T16:31:31.449207+0000","last_clean":"2024-04-26T16:31:31.449207+0000","last_became_active":"2024-04-26T16:28:25.250239+0000","last_became_peered":"2024-04-26T16:28:25.250239+0000","last_unstale":"2024-04-26T16:31:31.449207+0000","last_undegraded":"2024-04-26T16:31:31.449207+0000","last_fullsized":"2024-04-26T16:31:31.449207+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.493412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.493412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:48.496740+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1006,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-04-26T16:31:31.451569+0000","last_change":"2024-04-26T16:30:12.027247+0000","last_active":"2024-04-26T16:31:31.451569+0000","last_peered":"2024-04-26T16:31:31.451569+0000","last_clean":"2024-04-26T16:31:31.451569+0000","last_became_active":"2024-04-26T16:28:25.250958+0000","last_became_peered":"2024-04-26T16:28:25.250958+0000","last_unstale":"2024-04-26T16:31:31.451569+0000","last_undegraded":"2024-04-26T16:31:31.451569+0000","last_fullsized":"2024-04-26T16:31:31.451569+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.027062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.027062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":310,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":27,"acting":27,"num_store_stats":1},{"poolid":309,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":7},{"poolid":109,"num_pg":32,"stat_sum":{"num_bytes":8214741,"num_objects":191,"num_object_clones":10,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":215,"num_read_kb":59,"num_write":771,"num_write_kb":12801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":4194787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":20,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26861568,"data_stored":24644199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2081,"ondisk_log_size":2081,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":820,"num_read_kb":815,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743603,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97092,"kb_used_data":3236,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93648060,"statfs":{"total":95995035648,"available":95895613440,"internally_reserved":0,"allocated":3313664,"data_stored":1331944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":432,"seq":1855425871911,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39700,"kb_used_data":3080,"kb_used_omap":15,"kb_used_meta":36592,"kb_avail":93705452,"statfs":{"total":95995035648,"available":95954382848,"internally_reserved":0,"allocated":3153920,"data_stored":1348188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15525,"internal_metadata":37471067},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77872,"kb_used_data":10592,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93667280,"statfs":{"total":95995035648,"available":95915294720,"internally_reserved":0,"allocated":10846208,"data_stored":8930574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738619,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74588,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93670564,"statfs":{"total":95995035648,"available":95918657536,"internally_reserved":0,"allocated":3227648,"data_stored":1329586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74192,"kb_used_data":6460,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93670960,"statfs":{"total":95995035648,"available":95919063040,"internally_reserved":0,"allocated":6615040,"data_stored":4737829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084507,"num_pgs":103,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104220,"kb_used_data":10640,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":93640932,"statfs":{"total":95995035648,"available":95888314368,"internally_reserved":0,"allocated":10895360,"data_stored":8937766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":95812040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855569,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78580,"kb_used_data":6672,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93666572,"statfs":{"total":95995035648,"available":95914569728,"internally_reserved":0,"allocated":6832128,"data_stored":4932375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":294912,"data_stored":7757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8527872,"data_stored":8200132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4008328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8589312,"data_stored":8207853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4509696,"data_stored":4202329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":7880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":397312,"data_stored":9920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":309,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":310,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:31:38.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: audit 2024-04-26T16:31:36.819502+0000 mgr.y (mgr.24401) 552 : audit [DBG] from='client.30765 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: audit 2024-04-26T16:31:36.819502+0000 mgr.y (mgr.24401) 552 : audit [DBG] from='client.30765 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: audit 2024-04-26T16:31:37.028312+0000 mgr.y (mgr.24401) 553 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: audit 2024-04-26T16:31:37.028312+0000 mgr.y (mgr.24401) 553 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.784868+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.784868+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.787515+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.787515+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.235920+0000 osd.6 (osd.6) 65 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.235920+0000 osd.6 (osd.6) 65 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.241954+0000 osd.6 (osd.6) 66 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.241954+0000 osd.6 (osd.6) 66 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:31:38.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.274571+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e564: 8 total, 8 up, 7 in 2024-04-26T16:31:38.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:38 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.274571+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e564: 8 total, 8 up, 7 in 2024-04-26T16:31:38.708 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:38.811 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: audit 2024-04-26T16:31:36.819502+0000 mgr.y (mgr.24401) 552 : audit [DBG] from='client.30765 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: audit 2024-04-26T16:31:36.819502+0000 mgr.y (mgr.24401) 552 : audit [DBG] from='client.30765 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: audit 2024-04-26T16:31:37.028312+0000 mgr.y (mgr.24401) 553 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: audit 2024-04-26T16:31:37.028312+0000 mgr.y (mgr.24401) 553 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.784868+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.784868+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:31:38.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.787515+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.787515+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.235920+0000 osd.6 (osd.6) 65 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.235920+0000 osd.6 (osd.6) 65 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.241954+0000 osd.6 (osd.6) 66 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.241954+0000 osd.6 (osd.6) 66 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.274571+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e564: 8 total, 8 up, 7 in 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.274571+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e564: 8 total, 8 up, 7 in 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: audit 2024-04-26T16:31:36.819502+0000 mgr.y (mgr.24401) 552 : audit [DBG] from='client.30765 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: audit 2024-04-26T16:31:36.819502+0000 mgr.y (mgr.24401) 552 : audit [DBG] from='client.30765 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: audit 2024-04-26T16:31:37.028312+0000 mgr.y (mgr.24401) 553 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:38.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: audit 2024-04-26T16:31:37.028312+0000 mgr.y (mgr.24401) 553 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.784868+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.784868+0000 osd.0 (osd.0) 335 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.787515+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.787515+0000 osd.0 (osd.0) 336 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.235920+0000 osd.6 (osd.6) 65 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.235920+0000 osd.6 (osd.6) 65 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.241954+0000 osd.6 (osd.6) 66 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.241954+0000 osd.6 (osd.6) 66 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.274571+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e564: 8 total, 8 up, 7 in 2024-04-26T16:31:38.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:38 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.274571+0000 mon.a (mon.0) 3018 : cluster [DBG] osdmap e564: 8 total, 8 up, 7 in 2024-04-26T16:31:38.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:38.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.968 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:38.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:39.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:39.097 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:39.200 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:39.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:39.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-26T16:31:39.324 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:39.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:39.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:39.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:39.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.748600+0000 mgr.y (mgr.24401) 554 : cluster [DBG] pgmap v862: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:37.748600+0000 mgr.y (mgr.24401) 554 : cluster [DBG] pgmap v862: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.043448+0000 osd.1 (osd.1) 309 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.043448+0000 osd.1 (osd.1) 309 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.044797+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.044797+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.045939+0000 osd.1 (osd.1) 310 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:31:39.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.045939+0000 osd.1 (osd.1) 310 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.047292+0000 osd.3 (osd.3) 354 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.047292+0000 osd.3 (osd.3) 354 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.494635+0000 osd.4 (osd.4) 293 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.494635+0000 osd.4 (osd.4) 293 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.498793+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.498793+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.761661+0000 osd.0 (osd.0) 337 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.761661+0000 osd.0 (osd.0) 337 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.763178+0000 osd.0 (osd.0) 338 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:38.763178+0000 osd.0 (osd.0) 338 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:31:39.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.276579+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e565: 8 total, 8 up, 7 in 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.276579+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e565: 8 total, 8 up, 7 in 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: audit 2024-04-26T16:31:39.287523+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: audit 2024-04-26T16:31:39.287523+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: audit 2024-04-26T16:31:39.287743+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: audit 2024-04-26T16:31:39.287743+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.297795+0000 osd.6 (osd.6) 67 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:31:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:39 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.297795+0000 osd.6 (osd.6) 67 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:31:39.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:39.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:39.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:39.737 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:39.840 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.748600+0000 mgr.y (mgr.24401) 554 : cluster [DBG] pgmap v862: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:37.748600+0000 mgr.y (mgr.24401) 554 : cluster [DBG] pgmap v862: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.043448+0000 osd.1 (osd.1) 309 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.043448+0000 osd.1 (osd.1) 309 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.044797+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.044797+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:31:39.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.045939+0000 osd.1 (osd.1) 310 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.045939+0000 osd.1 (osd.1) 310 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.047292+0000 osd.3 (osd.3) 354 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.047292+0000 osd.3 (osd.3) 354 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.494635+0000 osd.4 (osd.4) 293 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.494635+0000 osd.4 (osd.4) 293 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.498793+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.498793+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.761661+0000 osd.0 (osd.0) 337 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.761661+0000 osd.0 (osd.0) 337 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.763178+0000 osd.0 (osd.0) 338 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:38.763178+0000 osd.0 (osd.0) 338 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.276579+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e565: 8 total, 8 up, 7 in 2024-04-26T16:31:39.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.276579+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e565: 8 total, 8 up, 7 in 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: audit 2024-04-26T16:31:39.287523+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: audit 2024-04-26T16:31:39.287523+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: audit 2024-04-26T16:31:39.287743+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: audit 2024-04-26T16:31:39.287743+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.297795+0000 osd.6 (osd.6) 67 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.297795+0000 osd.6 (osd.6) 67 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:31:39.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.748600+0000 mgr.y (mgr.24401) 554 : cluster [DBG] pgmap v862: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:37.748600+0000 mgr.y (mgr.24401) 554 : cluster [DBG] pgmap v862: 228 pgs: 228 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.043448+0000 osd.1 (osd.1) 309 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.043448+0000 osd.1 (osd.1) 309 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.044797+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.044797+0000 osd.3 (osd.3) 353 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.045939+0000 osd.1 (osd.1) 310 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.045939+0000 osd.1 (osd.1) 310 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.047292+0000 osd.3 (osd.3) 354 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.047292+0000 osd.3 (osd.3) 354 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.494635+0000 osd.4 (osd.4) 293 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.494635+0000 osd.4 (osd.4) 293 : cluster [DBG] 3.3 deep-scrub starts 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.498793+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:31:39.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.498793+0000 osd.4 (osd.4) 294 : cluster [DBG] 3.3 deep-scrub ok 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.761661+0000 osd.0 (osd.0) 337 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.761661+0000 osd.0 (osd.0) 337 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.763178+0000 osd.0 (osd.0) 338 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:38.763178+0000 osd.0 (osd.0) 338 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.276579+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e565: 8 total, 8 up, 7 in 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.276579+0000 mon.a (mon.0) 3019 : cluster [DBG] osdmap e565: 8 total, 8 up, 7 in 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: audit 2024-04-26T16:31:39.287523+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: audit 2024-04-26T16:31:39.287523+0000 mon.b (mon.1) 395 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: audit 2024-04-26T16:31:39.287743+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: audit 2024-04-26T16:31:39.287743+0000 mon.a (mon.0) 3020 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.297795+0000 osd.6 (osd.6) 67 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:31:39.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:39 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.297795+0000 osd.6 (osd.6) 67 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:31:39.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:39.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:39.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:39.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:40.097 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:40.120 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- 172.21.15.50:0/2845023800 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006c510 msgr2=0x7fee901102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- 172.21.15.50:0/2845023800 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006c510 0x7fee901102b0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fee8800b0a0 tx=0x7fee8800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- 172.21.15.50:0/2845023800 shutdown_connections 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- 172.21.15.50:0/2845023800 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006c510 0x7fee901102b0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- 172.21.15.50:0/2845023800 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee901094e0 0x7fee901098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- 172.21.15.50:0/2845023800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee9006bb50 0x7fee9006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- 172.21.15.50:0/2845023800 >> 172.21.15.50:0/2845023800 conn(0x7fee90078190 msgr2=0x7fee900785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- 172.21.15.50:0/2845023800 shutdown_connections 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- 172.21.15.50:0/2845023800 wait complete. 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 librados: starting msgr at 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 librados: starting objecter 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 Processor -- start 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- start start 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 librados: setting wanted keys 2024-04-26T16:31:40.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 librados: calling monclient init 2024-04-26T16:31:40.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 0x7fee90199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9006c510 0x7fee901995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee901094e0 0x7fee9019d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fee90112ad0 con 0x7fee9006c510 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fee90112950 con 0x7fee901094e0 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.169+0000 7fee957c4700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fee90112c50 con 0x7fee9006bb50 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 0x7fee90199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 0x7fee90199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36270/0 (socket says 172.21.15.50:36270) 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 -- 172.21.15.50:0/1112194336 learned_addr learned my addr 172.21.15.50:0/1112194336 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9006c510 0x7fee901995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee901094e0 0x7fee9019d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee901094e0 msgr2=0x7fee9019d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee901094e0 0x7fee9019d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9006c510 msgr2=0x7fee901995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fee9006c510 0x7fee901995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee9019e080 con 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee901094e0 0x7fee9019d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 0x7fee90199070 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fee8400e9a0 tx=0x7fee8400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 10 client.?.objecter ms_handle_connect 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8400cbd0 con 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fee8400cd70 con 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee9019e370 con 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee84018420 con 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 10 client.41387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fee901a5bf0 con 0x7fee9006bb50 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 1 librados: init done 2024-04-26T16:31:40.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fee9006a110 con 0x7fee9006bb50 2024-04-26T16:31:40.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fee84010430 con 0x7fee9006bb50 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee78071f80 0x7fee78074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(565..565 src has 496..565) ==== 9364+0+0 (secure 0 0 0) 0x7fee84092470 con 0x7fee9006bb50 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 osd_map(565..565 src has 496..565) 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 3 client.41387.objecter handle_osd_map got epochs [565,565] > 0 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 3 client.41387.objecter handle_osd_map decoding full epoch 565 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee78071f80 0x7fee78074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 20 client.41387.objecter dump_active .. 0 homeless 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v565) ==== 54+0+16 (secure 0 0 0) 0x7fee840928c0 con 0x7fee9006bb50 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee78071f80 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee78071f80 0x7fee78074440 secure :-1 s=READY pgs=3776 cs=0 l=1 rev1=1 crypto rx=0x7fee90108ef0 tx=0x7fee80009190 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 10 client.41387.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 1 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.173+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee90064030 0x7fee90064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 1 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 0 1 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 4 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 10 client.41387.objecter _send_command 1 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee901a9070 con 0x7fee90064030 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee90064030 0x7fee90064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee90064030 0x7fee90064410 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee90064030 2024-04-26T16:31:40.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee901a9070 con 0x7fee90064030 2024-04-26T16:31:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 1: 0 ) 2024-04-26T16:31:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee77fff700 10 client.41387.objecter _finish_command 1 = system:0 2024-04-26T16:31:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 4 2024-04-26T16:31:40.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.177+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 0 1 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 10 client.41387.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 2 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 2 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9019a3d0 osd=0 3 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 0 2 2024-04-26T16:31:40.192 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 4 2024-04-26T16:31:40.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 10 client.41387.objecter _send_command 2 2024-04-26T16:31:40.193 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.193+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee90063ab0 con 0x7fee90064030 2024-04-26T16:31:40.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee90063ab0 con 0x7fee90064030 2024-04-26T16:31:40.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 10 client.41387.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 4 2024-04-26T16:31:40.194 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 0 2 2024-04-26T16:31:40.194 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 10 client.41387.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9002d310 0x7fee9002d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 1 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 4 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 10 client.41387.objecter _send_command 3 2024-04-26T16:31:40.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fee9010b020 con 0x7fee9002d310 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9002d310 0x7fee9002d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee9002d310 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9002d310 0x7fee9002d710 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee9010b020 con 0x7fee9002d310 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 3: 0 ) 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 10 client.41387.objecter _finish_command 3 = system:0 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 4 2024-04-26T16:31:40.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.197+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 1 3 2024-04-26T16:31:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 10 client.41387.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 4 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 4 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9010a9a0 osd=1 3 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 1 4 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 4 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 10 client.41387.objecter _send_command 4 2024-04-26T16:31:40.214 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee9010a420 con 0x7fee9002d310 2024-04-26T16:31:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee9010a420 con 0x7fee9002d310 2024-04-26T16:31:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee77fff700 10 client.41387.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 4 2024-04-26T16:31:40.215 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 1 4 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 10 client.41387.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 5 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee90102670 0x7fee90102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 5 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 2 5 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 4 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 10 client.41387.objecter _send_command 5 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fee901b27d0 con 0x7fee90102670 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee90102670 0x7fee90102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee90102670 0x7fee90102a90 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.217+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee90102670 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.221+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee901b27d0 con 0x7fee90102670 2024-04-26T16:31:40.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.221+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 5: 0 ) 2024-04-26T16:31:40.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.221+0000 7fee77fff700 10 client.41387.objecter _finish_command 5 = system:0 2024-04-26T16:31:40.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.221+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 4 2024-04-26T16:31:40.218 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.221+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 2 5 2024-04-26T16:31:40.223 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:40.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 10 client.41387.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 6 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 6 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9002e420 osd=2 3 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 2 6 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 4 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 10 client.41387.objecter _send_command 6 2024-04-26T16:31:40.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.237+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee9002dea0 con 0x7fee90102670 2024-04-26T16:31:40.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee9002dea0 con 0x7fee90102670 2024-04-26T16:31:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 10 client.41387.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 4 2024-04-26T16:31:40.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 2 6 2024-04-26T16:31:40.238 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:31:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 10 client.41387.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 7 2024-04-26T16:31:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee901a8bd0 0x7fee90102fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 7 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 3 7 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 4 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 10 client.41387.objecter _send_command 7 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fee9002e260 con 0x7fee901a8bd0 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee901a8bd0 0x7fee90102fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee901a8bd0 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee901a8bd0 0x7fee90102fd0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee8800bbd0 con 0x7fee901a8bd0 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 7: 0 ) 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 10 client.41387.objecter _finish_command 7 = system:0 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 4 2024-04-26T16:31:40.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.241+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 3 7 2024-04-26T16:31:40.244 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:40.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 10 client.41387.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 8 2024-04-26T16:31:40.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 8 2024-04-26T16:31:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee9002e0f0 osd=3 3 2024-04-26T16:31:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 3 8 2024-04-26T16:31:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 4 2024-04-26T16:31:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 10 client.41387.objecter _send_command 8 2024-04-26T16:31:40.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee901a8140 con 0x7fee901a8bd0 2024-04-26T16:31:40.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee901a8140 con 0x7fee901a8bd0 2024-04-26T16:31:40.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee77fff700 10 client.41387.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 4 2024-04-26T16:31:40.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 3 8 2024-04-26T16:31:40.260 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 10 client.41387.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 9 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee901b2330 0x7fee901b69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 9 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 4 9 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 4 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 10 client.41387.objecter _send_command 9 2024-04-26T16:31:40.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.261+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fee901a80f0 con 0x7fee901b2330 2024-04-26T16:31:40.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee901b2330 0x7fee901b69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee901b2330 0x7fee901b69a0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee901b2330 2024-04-26T16:31:40.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee901a80f0 con 0x7fee901b2330 2024-04-26T16:31:40.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 9: 0 ) 2024-04-26T16:31:40.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee77fff700 10 client.41387.objecter _finish_command 9 = system:0 2024-04-26T16:31:40.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 4 2024-04-26T16:31:40.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.265+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 4 9 2024-04-26T16:31:40.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 10 client.41387.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 10 2024-04-26T16:31:40.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 10 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901a7ef0 osd=4 3 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 4 10 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 4 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 10 client.41387.objecter _send_command 10 2024-04-26T16:31:40.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.297+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee901a80f0 con 0x7fee901b2330 2024-04-26T16:31:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.305+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee901a80f0 con 0x7fee901b2330 2024-04-26T16:31:40.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.305+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.305+0000 7fee77fff700 10 client.41387.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.305+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 4 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.305+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 4 10 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 10 client.41387.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 11 2024-04-26T16:31:40.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee901b7440 0x7fee901b7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 11 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 5 11 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 4 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 10 client.41387.objecter _send_command 11 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fee901bca40 con 0x7fee901b7440 2024-04-26T16:31:40.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee901b7440 0x7fee901b7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee901b7440 2024-04-26T16:31:40.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.309+0000 7fee8e7fc700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee901b7440 0x7fee901b7880 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.313+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee901bca40 con 0x7fee901b7440 2024-04-26T16:31:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.313+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 11: 0 ) 2024-04-26T16:31:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.313+0000 7fee77fff700 10 client.41387.objecter _finish_command 11 = system:0 2024-04-26T16:31:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.313+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 4 2024-04-26T16:31:40.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.313+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 5 11 2024-04-26T16:31:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 10 client.41387.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 12 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 12 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901a80f0 osd=5 3 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 5 12 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 4 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 10 client.41387.objecter _send_command 12 2024-04-26T16:31:40.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.329+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee901b7ed0 con 0x7fee901b7440 2024-04-26T16:31:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee901b7ed0 con 0x7fee901b7440 2024-04-26T16:31:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee77fff700 10 client.41387.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 4 2024-04-26T16:31:40.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 5 12 2024-04-26T16:31:40.333 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:31:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 10 client.41387.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 13 2024-04-26T16:31:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fee901b8550 0x7fee901b89b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 13 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 6 13 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 4 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 10 client.41387.objecter _send_command 13 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.333+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fee901b90a0 con 0x7fee901b8550 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fee901b8550 0x7fee901b89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee8f7fe700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fee901b8550 0x7fee901b89b0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee901b8550 2024-04-26T16:31:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee901b90a0 con 0x7fee901b8550 2024-04-26T16:31:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 13: 0 ) 2024-04-26T16:31:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee77fff700 10 client.41387.objecter _finish_command 13 = system:0 2024-04-26T16:31:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 4 2024-04-26T16:31:40.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.337+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 6 13 2024-04-26T16:31:40.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 10 client.41387.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 14 2024-04-26T16:31:40.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 14 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901b7ed0 osd=6 3 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 6 14 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 4 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 10 client.41387.objecter _send_command 14 2024-04-26T16:31:40.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.353+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee901b90a0 con 0x7fee901b8550 2024-04-26T16:31:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.357+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee901b90a0 con 0x7fee901b8550 2024-04-26T16:31:40.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.357+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.357+0000 7fee77fff700 10 client.41387.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.357+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 4 2024-04-26T16:31:40.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.357+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 6 14 2024-04-26T16:31:40.357 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 10 client.41387.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 15 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee901b9260 0x7fee901bfe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 15 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 7 15 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 4 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 10 client.41387.objecter _send_command 15 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fee901c2ef0 con 0x7fee901b9260 2024-04-26T16:31:40.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee901b9260 0x7fee901bfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee8effd700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee901b9260 0x7fee901bfe30 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:40.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee77fff700 10 client.41387.objecter ms_handle_connect 0x7fee901b9260 2024-04-26T16:31:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fee901c2ef0 con 0x7fee901b9260 2024-04-26T16:31:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 15: 0 ) 2024-04-26T16:31:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee77fff700 10 client.41387.objecter _finish_command 15 = system:0 2024-04-26T16:31:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 4 2024-04-26T16:31:40.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.361+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 7 15 2024-04-26T16:31:40.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:40.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign -1 16 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter _get_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 15 client.41387.objecter _session_command_op_remove -1 16 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter get_session s=0x7fee901b90a0 osd=7 3 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 15 client.41387.objecter _session_command_op_assign 7 16 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 4 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter _send_command 16 2024-04-26T16:31:40.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fee901c1c80 con 0x7fee901b9260 2024-04-26T16:31:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee77fff700 1 -- 172.21.15.50:0/1112194336 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fee901c1c80 con 0x7fee901b9260 2024-04-26T16:31:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee77fff700 10 client.41387.objecter ms_dispatch 0x7fee90001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee77fff700 10 client.41387.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee77fff700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 4 2024-04-26T16:31:40.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee77fff700 15 client.41387.objecter _session_command_op_remove 7 16 2024-04-26T16:31:40.379 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 librados: watch_flush enter 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 librados: watch_flush exit 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.0 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee90064030 msgr2=0x7fee90064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fee90064030 0x7fee90064410 crc :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9019a3d0 osd=0 1 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.1 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9002d310 msgr2=0x7fee9002d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fee9002d310 0x7fee9002d710 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.381 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9010a9a0 osd=1 1 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.2 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee90102670 msgr2=0x7fee90102a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fee90102670 0x7fee90102a90 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e420 osd=2 1 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.3 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee901a8bd0 msgr2=0x7fee90102fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fee901a8bd0 0x7fee90102fd0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee9002e0f0 osd=3 1 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.4 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee901b2330 msgr2=0x7fee901b69a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fee901b2330 0x7fee901b69a0 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee8f7fe700 1 -- 172.21.15.50:0/1112194336 reap_dead start 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a7ef0 osd=4 1 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.5 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee901b7440 msgr2=0x7fee901b7880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fee901b7440 0x7fee901b7880 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee8f7fe700 1 -- 172.21.15.50:0/1112194336 reap_dead start 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901a80f0 osd=5 1 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.6 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fee901b8550 msgr2=0x7fee901b89b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fee901b8550 0x7fee901b89b0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b7ed0 osd=6 1 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter close_session for osd.7 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee901b9260 msgr2=0x7fee901bfe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fee901b9260 0x7fee901bfe30 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee8f7fe700 1 -- 172.21.15.50:0/1112194336 reap_dead start 2024-04-26T16:31:40.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter put_session s=0x7fee901b90a0 osd=7 1 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 20 client.41387.objecter shutdown clearing up homeless session... 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 10 client.41387.objecter successfully canceled tick 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee78071f80 msgr2=0x7fee78074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee78071f80 0x7fee78074440 secure :-1 s=READY pgs=3776 cs=0 l=1 rev1=1 crypto rx=0x7fee90108ef0 tx=0x7fee80009190 comp rx=0 tx=0).stop 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 msgr2=0x7fee90199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 0x7fee90199070 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fee8400e9a0 tx=0x7fee8400ecb0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 shutdown_connections 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fee9006bb50 0x7fee90199070 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 --2- 172.21.15.50:0/1112194336 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fee78071f80 0x7fee78074440 unknown :-1 s=CLOSED pgs=3776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 >> 172.21.15.50:0/1112194336 conn(0x7fee90078190 msgr2=0x7fee90100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 shutdown_connections 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 -- 172.21.15.50:0/1112194336 wait complete. 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:40.381+0000 7fee957c4700 1 librados: shutdown 2024-04-26T16:31:40.384 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:31:40.385 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:31:40.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:40.508 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:40.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:40.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:40.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.300539+0000 osd.6 (osd.6) 68 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:31:40.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.300539+0000 osd.6 (osd.6) 68 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:31:40.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.500567+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:31:40.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.500567+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:31:40.679 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.502274+0000 osd.4 (osd.4) 296 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.502274+0000 osd.4 (osd.4) 296 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.787811+0000 osd.0 (osd.0) 339 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.787811+0000 osd.0 (osd.0) 339 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.789602+0000 osd.0 (osd.0) 340 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.789602+0000 osd.0 (osd.0) 340 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.177382+0000 mon.c (mon.2) 662 : audit [DBG] from='client.? 172.21.15.50:0/1112194336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.177382+0000 mon.c (mon.2) 662 : audit [DBG] from='client.? 172.21.15.50:0/1112194336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.274217+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.274217+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.278133+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e566: 8 total, 8 up, 7 in 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.278133+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e566: 8 total, 8 up, 7 in 2024-04-26T16:31:40.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.285087+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.285087+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.299761+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.299761+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.300615+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[29234]: audit 2024-04-26T16:31:40.300615+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.300539+0000 osd.6 (osd.6) 68 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.300539+0000 osd.6 (osd.6) 68 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.500567+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.500567+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.502274+0000 osd.4 (osd.4) 296 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:31:40.681 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.502274+0000 osd.4 (osd.4) 296 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.787811+0000 osd.0 (osd.0) 339 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.787811+0000 osd.0 (osd.0) 339 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.789602+0000 osd.0 (osd.0) 340 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.789602+0000 osd.0 (osd.0) 340 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.177382+0000 mon.c (mon.2) 662 : audit [DBG] from='client.? 172.21.15.50:0/1112194336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.177382+0000 mon.c (mon.2) 662 : audit [DBG] from='client.? 172.21.15.50:0/1112194336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.274217+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.274217+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.278133+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e566: 8 total, 8 up, 7 in 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.278133+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e566: 8 total, 8 up, 7 in 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.285087+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:40.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.285087+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:40.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.299761+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.299761+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.300615+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:40 smithi050 bash[21463]: audit 2024-04-26T16:31:40.300615+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:40.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.300539+0000 osd.6 (osd.6) 68 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:31:40.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.300539+0000 osd.6 (osd.6) 68 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.500567+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.500567+0000 osd.4 (osd.4) 295 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.502274+0000 osd.4 (osd.4) 296 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.502274+0000 osd.4 (osd.4) 296 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.787811+0000 osd.0 (osd.0) 339 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.787811+0000 osd.0 (osd.0) 339 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.789602+0000 osd.0 (osd.0) 340 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.789602+0000 osd.0 (osd.0) 340 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.177382+0000 mon.c (mon.2) 662 : audit [DBG] from='client.? 172.21.15.50:0/1112194336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:40.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.177382+0000 mon.c (mon.2) 662 : audit [DBG] from='client.? 172.21.15.50:0/1112194336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.274217+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.274217+0000 mon.a (mon.0) 3021 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.278133+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e566: 8 total, 8 up, 7 in 2024-04-26T16:31:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.278133+0000 mon.a (mon.0) 3022 : cluster [DBG] osdmap e566: 8 total, 8 up, 7 in 2024-04-26T16:31:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.285087+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.285087+0000 mon.b (mon.1) 396 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.299761+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.299761+0000 mon.b (mon.1) 397 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.300615+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:40 smithi190 bash[23868]: audit 2024-04-26T16:31:40.300615+0000 mon.a (mon.0) 3023 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:31:40.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:40.966 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:40.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:41.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:41.095 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:41.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:41.227 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:41.331 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:41.355 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:41.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:41.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:41.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:41.612 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.749126+0000 mgr.y (mgr.24401) 555 : cluster [DBG] pgmap v865: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:39.749126+0000 mgr.y (mgr.24401) 555 : cluster [DBG] pgmap v865: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.033541+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.033541+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.035877+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.035877+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.488592+0000 osd.4 (osd.4) 297 : cluster [DBG] 109.f scrub starts 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.488592+0000 osd.4 (osd.4) 297 : cluster [DBG] 109.f scrub starts 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.494258+0000 osd.4 (osd.4) 298 : cluster [DBG] 109.f scrub ok 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.494258+0000 osd.4 (osd.4) 298 : cluster [DBG] 109.f scrub ok 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.820994+0000 osd.0 (osd.0) 341 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.820994+0000 osd.0 (osd.0) 341 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:31:41.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.823609+0000 osd.0 (osd.0) 342 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:40.823609+0000 osd.0 (osd.0) 342 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: audit 2024-04-26T16:31:41.249060+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: audit 2024-04-26T16:31:41.249060+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.256871+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e567: 8 total, 8 up, 7 in 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.256871+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e567: 8 total, 8 up, 7 in 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.749126+0000 mgr.y (mgr.24401) 555 : cluster [DBG] pgmap v865: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:39.749126+0000 mgr.y (mgr.24401) 555 : cluster [DBG] pgmap v865: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.033541+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.033541+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:31:41.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.035877+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.035877+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.488592+0000 osd.4 (osd.4) 297 : cluster [DBG] 109.f scrub starts 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.488592+0000 osd.4 (osd.4) 297 : cluster [DBG] 109.f scrub starts 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.494258+0000 osd.4 (osd.4) 298 : cluster [DBG] 109.f scrub ok 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.494258+0000 osd.4 (osd.4) 298 : cluster [DBG] 109.f scrub ok 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.820994+0000 osd.0 (osd.0) 341 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.820994+0000 osd.0 (osd.0) 341 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.823609+0000 osd.0 (osd.0) 342 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:40.823609+0000 osd.0 (osd.0) 342 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: audit 2024-04-26T16:31:41.249060+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: audit 2024-04-26T16:31:41.249060+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.256871+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e567: 8 total, 8 up, 7 in 2024-04-26T16:31:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:41 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.256871+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e567: 8 total, 8 up, 7 in 2024-04-26T16:31:41.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:31:41.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:41.743 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:41.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.749126+0000 mgr.y (mgr.24401) 555 : cluster [DBG] pgmap v865: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:41.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:39.749126+0000 mgr.y (mgr.24401) 555 : cluster [DBG] pgmap v865: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:31:41.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.033541+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:31:41.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.033541+0000 osd.1 (osd.1) 311 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:31:41.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.035877+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.035877+0000 osd.1 (osd.1) 312 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.488592+0000 osd.4 (osd.4) 297 : cluster [DBG] 109.f scrub starts 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.488592+0000 osd.4 (osd.4) 297 : cluster [DBG] 109.f scrub starts 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.494258+0000 osd.4 (osd.4) 298 : cluster [DBG] 109.f scrub ok 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.494258+0000 osd.4 (osd.4) 298 : cluster [DBG] 109.f scrub ok 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.820994+0000 osd.0 (osd.0) 341 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.820994+0000 osd.0 (osd.0) 341 : cluster [DBG] 109.19 scrub starts 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.823609+0000 osd.0 (osd.0) 342 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:40.823609+0000 osd.0 (osd.0) 342 : cluster [DBG] 109.19 scrub ok 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: audit 2024-04-26T16:31:41.249060+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: audit 2024-04-26T16:31:41.249060+0000 mon.a (mon.0) 3024 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.256871+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e567: 8 total, 8 up, 7 in 2024-04-26T16:31:41.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:41 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.256871+0000 mon.a (mon.0) 3025 : cluster [DBG] osdmap e567: 8 total, 8 up, 7 in 2024-04-26T16:31:41.846 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:41.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:41.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:41.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:41.979 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:42.004 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:42.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:42.130 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:42.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:42.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:42.259 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:42.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:42.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:42.391 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:42.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:42.524 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:42.627 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:42.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:42.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:42.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.028993+0000 osd.3 (osd.3) 355 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:31:42.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.028993+0000 osd.3 (osd.3) 355 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:31:42.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.030277+0000 osd.3 (osd.3) 356 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:31:42.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.030277+0000 osd.3 (osd.3) 356 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:31:42.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.053478+0000 osd.1 (osd.1) 313 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:31:42.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.053478+0000 osd.1 (osd.1) 313 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:31:42.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.056160+0000 osd.1 (osd.1) 314 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:31:42.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.056160+0000 osd.1 (osd.1) 314 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.358189+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.358189+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.360847+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.360847+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.526654+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.e scrub starts 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.526654+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.e scrub starts 2024-04-26T16:31:42.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.528855+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.e scrub ok 2024-04-26T16:31:42.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.528855+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.e scrub ok 2024-04-26T16:31:42.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: audit 2024-04-26T16:31:42.191713+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:42.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: audit 2024-04-26T16:31:42.191713+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:42.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: audit 2024-04-26T16:31:42.194712+0000 mon.c (mon.2) 663 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:42.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: audit 2024-04-26T16:31:42.194712+0000 mon.c (mon.2) 663 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:42.777 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.271717+0000 mon.a (mon.0) 3027 : cluster [DBG] osdmap e568: 8 total, 8 up, 7 in 2024-04-26T16:31:42.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.271717+0000 mon.a (mon.0) 3027 : cluster [DBG] osdmap e568: 8 total, 8 up, 7 in 2024-04-26T16:31:42.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.028993+0000 osd.3 (osd.3) 355 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:31:42.778 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.028993+0000 osd.3 (osd.3) 355 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.030277+0000 osd.3 (osd.3) 356 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.030277+0000 osd.3 (osd.3) 356 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.053478+0000 osd.1 (osd.1) 313 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.053478+0000 osd.1 (osd.1) 313 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.056160+0000 osd.1 (osd.1) 314 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.056160+0000 osd.1 (osd.1) 314 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.358189+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:31:42.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.358189+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.360847+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.360847+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.526654+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.e scrub starts 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.526654+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.e scrub starts 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.528855+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.e scrub ok 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.528855+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.e scrub ok 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: audit 2024-04-26T16:31:42.191713+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: audit 2024-04-26T16:31:42.191713+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:42.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: audit 2024-04-26T16:31:42.194712+0000 mon.c (mon.2) 663 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:42.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: audit 2024-04-26T16:31:42.194712+0000 mon.c (mon.2) 663 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:42.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.271717+0000 mon.a (mon.0) 3027 : cluster [DBG] osdmap e568: 8 total, 8 up, 7 in 2024-04-26T16:31:42.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:42 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.271717+0000 mon.a (mon.0) 3027 : cluster [DBG] osdmap e568: 8 total, 8 up, 7 in 2024-04-26T16:31:42.790 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:42.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.028993+0000 osd.3 (osd.3) 355 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.028993+0000 osd.3 (osd.3) 355 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.030277+0000 osd.3 (osd.3) 356 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.030277+0000 osd.3 (osd.3) 356 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.053478+0000 osd.1 (osd.1) 313 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.053478+0000 osd.1 (osd.1) 313 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.056160+0000 osd.1 (osd.1) 314 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.056160+0000 osd.1 (osd.1) 314 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.358189+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.358189+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.360847+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.360847+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.526654+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.e scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.526654+0000 osd.4 (osd.4) 299 : cluster [DBG] 6.e scrub starts 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.528855+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.e scrub ok 2024-04-26T16:31:42.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.528855+0000 osd.4 (osd.4) 300 : cluster [DBG] 6.e scrub ok 2024-04-26T16:31:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: audit 2024-04-26T16:31:42.191713+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: audit 2024-04-26T16:31:42.191713+0000 mon.a (mon.0) 3026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: audit 2024-04-26T16:31:42.194712+0000 mon.c (mon.2) 663 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: audit 2024-04-26T16:31:42.194712+0000 mon.c (mon.2) 663 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.271717+0000 mon.a (mon.0) 3027 : cluster [DBG] osdmap e568: 8 total, 8 up, 7 in 2024-04-26T16:31:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:42 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.271717+0000 mon.a (mon.0) 3027 : cluster [DBG] osdmap e568: 8 total, 8 up, 7 in 2024-04-26T16:31:42.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:42.920 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:42.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:43.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:43.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:43.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:43.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:43.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:43.313 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:43.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:43.444 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:43.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:43.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:31:43.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.751218+0000 mgr.y (mgr.24401) 556 : cluster [DBG] pgmap v868: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T16:31:43.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:41.751218+0000 mgr.y (mgr.24401) 556 : cluster [DBG] pgmap v868: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T16:31:43.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.520512+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:43.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.520512+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:43.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.528701+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:31:43.572 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.528701+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.531677+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:42.531677+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.275658+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e569: 8 total, 8 up, 7 in 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.275658+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e569: 8 total, 8 up, 7 in 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.339605+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.339605+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.339786+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.339786+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.341494+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.341494+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.341496+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[21463]: audit 2024-04-26T16:31:43.341496+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.751218+0000 mgr.y (mgr.24401) 556 : cluster [DBG] pgmap v868: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:41.751218+0000 mgr.y (mgr.24401) 556 : cluster [DBG] pgmap v868: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.520512+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.520512+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.528701+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.528701+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:31:43.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.531677+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:31:43.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:42.531677+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:31:43.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.275658+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e569: 8 total, 8 up, 7 in 2024-04-26T16:31:43.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.275658+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e569: 8 total, 8 up, 7 in 2024-04-26T16:31:43.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.339605+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.339605+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.339786+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.339786+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.341494+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.341494+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.341496+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.576 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:43 smithi050 bash[29234]: audit 2024-04-26T16:31:43.341496+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.577 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:43.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:43.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:43.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.751218+0000 mgr.y (mgr.24401) 556 : cluster [DBG] pgmap v868: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T16:31:43.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:41.751218+0000 mgr.y (mgr.24401) 556 : cluster [DBG] pgmap v868: 228 pgs: 3 unknown, 225 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-04-26T16:31:43.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.520512+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:43.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.520512+0000 mon.a (mon.0) 3028 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:43.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.528701+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:31:43.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.528701+0000 osd.4 (osd.4) 301 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:31:43.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.531677+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:31:43.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:42.531677+0000 osd.4 (osd.4) 302 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:31:43.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.275658+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e569: 8 total, 8 up, 7 in 2024-04-26T16:31:43.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.275658+0000 mon.a (mon.0) 3029 : cluster [DBG] osdmap e569: 8 total, 8 up, 7 in 2024-04-26T16:31:43.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.339605+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.339605+0000 mon.b (mon.1) 398 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.339786+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.339786+0000 mon.a (mon.0) 3030 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:43.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.341494+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.341494+0000 mon.a (mon.0) 3031 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.341496+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:43 smithi190 bash[23868]: audit 2024-04-26T16:31:43.341496+0000 mon.b (mon.1) 399 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-90"}]: dispatch 2024-04-26T16:31:43.707 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:43.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:43.837 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:43.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:43.968 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:43.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:44.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:44.088 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:31:44.099 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:44.202 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:44.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:44.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:44.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:44.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:44.486 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:44.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:44.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:31:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.243266+0000 osd.7 (osd.7) 317 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.243266+0000 osd.7 (osd.7) 317 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.245552+0000 osd.7 (osd.7) 318 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.245552+0000 osd.7 (osd.7) 318 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.399917+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.399917+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.402196+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:31:44.599 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.402196+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.776455+0000 osd.0 (osd.0) 343 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.776455+0000 osd.0 (osd.0) 343 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.779027+0000 osd.0 (osd.0) 344 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.779027+0000 osd.0 (osd.0) 344 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.273415+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e570: 8 total, 8 up, 7 in 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.273415+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e570: 8 total, 8 up, 7 in 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.243266+0000 osd.7 (osd.7) 317 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.243266+0000 osd.7 (osd.7) 317 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.245552+0000 osd.7 (osd.7) 318 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.245552+0000 osd.7 (osd.7) 318 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.399917+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:31:44.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.399917+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.402196+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.402196+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.776455+0000 osd.0 (osd.0) 343 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.776455+0000 osd.0 (osd.0) 343 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.779027+0000 osd.0 (osd.0) 344 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.779027+0000 osd.0 (osd.0) 344 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.273415+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e570: 8 total, 8 up, 7 in 2024-04-26T16:31:44.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:44 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.273415+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e570: 8 total, 8 up, 7 in 2024-04-26T16:31:44.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:44.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.243266+0000 osd.7 (osd.7) 317 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.243266+0000 osd.7 (osd.7) 317 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.245552+0000 osd.7 (osd.7) 318 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.245552+0000 osd.7 (osd.7) 318 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.399917+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.399917+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:31:44.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.402196+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.402196+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.776455+0000 osd.0 (osd.0) 343 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.776455+0000 osd.0 (osd.0) 343 : cluster [DBG] 109.18 scrub starts 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.779027+0000 osd.0 (osd.0) 344 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.779027+0000 osd.0 (osd.0) 344 : cluster [DBG] 109.18 scrub ok 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.273415+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e570: 8 total, 8 up, 7 in 2024-04-26T16:31:44.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:44 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.273415+0000 mon.a (mon.0) 3032 : cluster [DBG] osdmap e570: 8 total, 8 up, 7 in 2024-04-26T16:31:44.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:44.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:44.873 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:44.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:44.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:44.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:45.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:45.127 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:45.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:45.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:45.257 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:45.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:45.387 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:45.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:45.512 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:45.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:45.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:45.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.751699+0000 mgr.y (mgr.24401) 557 : cluster [DBG] pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T16:31:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:43.751699+0000 mgr.y (mgr.24401) 557 : cluster [DBG] pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T16:31:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.419232+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.a scrub starts 2024-04-26T16:31:45.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.419232+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.a scrub starts 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.435904+0000 osd.6 (osd.6) 74 : cluster [DBG] 3.a scrub ok 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.435904+0000 osd.6 (osd.6) 74 : cluster [DBG] 3.a scrub ok 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.822111+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.822111+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.832928+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:44.832928+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.282223+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e571: 8 total, 8 up, 7 in 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.282223+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e571: 8 total, 8 up, 7 in 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: audit 2024-04-26T16:31:45.284948+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: audit 2024-04-26T16:31:45.284948+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: audit 2024-04-26T16:31:45.286105+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:45 smithi190 bash[23868]: audit 2024-04-26T16:31:45.286105+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:45.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:45.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.751699+0000 mgr.y (mgr.24401) 557 : cluster [DBG] pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T16:31:45.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:43.751699+0000 mgr.y (mgr.24401) 557 : cluster [DBG] pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.419232+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.a scrub starts 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.419232+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.a scrub starts 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.435904+0000 osd.6 (osd.6) 74 : cluster [DBG] 3.a scrub ok 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.435904+0000 osd.6 (osd.6) 74 : cluster [DBG] 3.a scrub ok 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.822111+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.822111+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.832928+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:44.832928+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.282223+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e571: 8 total, 8 up, 7 in 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.282223+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e571: 8 total, 8 up, 7 in 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: audit 2024-04-26T16:31:45.284948+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: audit 2024-04-26T16:31:45.284948+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: audit 2024-04-26T16:31:45.286105+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[29234]: audit 2024-04-26T16:31:45.286105+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.751699+0000 mgr.y (mgr.24401) 557 : cluster [DBG] pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:43.751699+0000 mgr.y (mgr.24401) 557 : cluster [DBG] pgmap v871: 228 pgs: 228 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.419232+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.a scrub starts 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.419232+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.a scrub starts 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.435904+0000 osd.6 (osd.6) 74 : cluster [DBG] 3.a scrub ok 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.435904+0000 osd.6 (osd.6) 74 : cluster [DBG] 3.a scrub ok 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.822111+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.822111+0000 osd.0 (osd.0) 345 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.832928+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:44.832928+0000 osd.0 (osd.0) 346 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:31:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.282223+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e571: 8 total, 8 up, 7 in 2024-04-26T16:31:45.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.282223+0000 mon.a (mon.0) 3033 : cluster [DBG] osdmap e571: 8 total, 8 up, 7 in 2024-04-26T16:31:45.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: audit 2024-04-26T16:31:45.284948+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: audit 2024-04-26T16:31:45.284948+0000 mon.b (mon.1) 400 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: audit 2024-04-26T16:31:45.286105+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:45 smithi050 bash[21463]: audit 2024-04-26T16:31:45.286105+0000 mon.a (mon.0) 3034 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:45.904 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:45.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:46.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/3149947397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e981094c0 msgr2=0x7f5e981098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:46.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/3149947397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e981094c0 0x7f5e981098a0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009990 tx=0x7f5e94009ca0 comp rx=0 tx=0).stop 2024-04-26T16:31:46.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/3149947397 shutdown_connections 2024-04-26T16:31:46.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/3149947397 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e9806c490 0x7f5e98110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/3149947397 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e981094c0 0x7f5e981098a0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/3149947397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9806baf0 0x7f5e9806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/3149947397 >> 172.21.15.50:0/3149947397 conn(0x7f5e980792d0 msgr2=0x7f5e980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/3149947397 shutdown_connections 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/3149947397 wait complete. 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 librados: starting msgr at 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 librados: starting objecter 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 Processor -- start 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 -- start start 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 librados: setting wanted keys 2024-04-26T16:31:46.025 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.025+0000 7f5e9f3b5700 1 librados: calling monclient init 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9806baf0 0x7f5e98199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 0x7f5e98199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 0x7f5e9819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5e98112aa0 con 0x7f5e981094c0 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5e98112920 con 0x7f5e9806baf0 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5e98112c20 con 0x7f5e9806c490 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 0x7f5e9819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9c950700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 0x7f5e98199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:46.027 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 0x7f5e9819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58870/0 (socket says 172.21.15.50:58870) 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9c950700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 0x7f5e98199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36288/0 (socket says 172.21.15.50:36288) 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 -- 172.21.15.50:0/1541691580 learned_addr learned my addr 172.21.15.50:0/1541691580 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d151700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9806baf0 0x7f5e98199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 -- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 msgr2=0x7f5e98199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 0x7f5e98199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 -- 172.21.15.50:0/1541691580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9806baf0 msgr2=0x7f5e98199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9806baf0 0x7f5e98199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 -- 172.21.15.50:0/1541691580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e9819e210 con 0x7f5e981094c0 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9c950700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 0x7f5e98199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9d952700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 0x7f5e9819dad0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c370 tx=0x7f5e9000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 10 client.?.objecter ms_handle_connect 0x7f5e981094c0 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e900092e0 con 0x7f5e981094c0 2024-04-26T16:31:46.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5e90003a00 con 0x7f5e981094c0 2024-04-26T16:31:46.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e9819e4a0 con 0x7f5e981094c0 2024-04-26T16:31:46.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e90004540 con 0x7f5e981094c0 2024-04-26T16:31:46.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 10 client.30777.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:46.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e981a5d80 con 0x7f5e981094c0 2024-04-26T16:31:46.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e9f3b5700 1 librados: init done 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5e9001a020 con 0x7f5e981094c0 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.029+0000 7f5e8a7fc700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e84071f80 0x7f5e84074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(571..571 src has 496..571) ==== 9364+0+0 (secure 0 0 0) 0x7f5e90090ea0 con 0x7f5e981094c0 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e8a7fc700 10 client.30777.objecter ms_dispatch 0x7f5e98001550 osd_map(571..571 src has 496..571) 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e8a7fc700 3 client.30777.objecter handle_osd_map got epochs [571,571] > 0 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e8a7fc700 3 client.30777.objecter handle_osd_map decoding full epoch 571 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e9d151700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e84071f80 0x7f5e84074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e8a7fc700 20 client.30777.objecter dump_active .. 0 homeless 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e8a7fc700 10 client.30777.objecter ms_handle_connect 0x7f5e84071f80 2024-04-26T16:31:46.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e9d151700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e84071f80 0x7f5e84074440 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400b2e0 tx=0x7f5e94005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:46.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.033+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e98104b70 con 0x7f5e981094c0 2024-04-26T16:31:46.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:46.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.037+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5e90021060 con 0x7f5e981094c0 2024-04-26T16:31:46.134 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:46.256 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:46.289+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f5e9819a560 con 0x7f5e981094c0 2024-04-26T16:31:46.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:46.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:46.424 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:46.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:46.558 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:46.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.690 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:46.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.167031+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.a scrub starts 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.167031+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.a scrub starts 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.169400+0000 osd.1 (osd.1) 316 : cluster [DBG] 2.a scrub ok 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.169400+0000 osd.1 (osd.1) 316 : cluster [DBG] 2.a scrub ok 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.419186+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.419186+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.421494+0000 osd.6 (osd.6) 76 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:31:46.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.421494+0000 osd.6 (osd.6) 76 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.862013+0000 osd.0 (osd.0) 347 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.862013+0000 osd.0 (osd.0) 347 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.864532+0000 osd.0 (osd.0) 348 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.864532+0000 osd.0 (osd.0) 348 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: audit 2024-04-26T16:31:46.276007+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: audit 2024-04-26T16:31:46.276007+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:46.280941+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e572: 8 total, 8 up, 7 in 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: cluster 2024-04-26T16:31:46.280941+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e572: 8 total, 8 up, 7 in 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: audit 2024-04-26T16:31:46.297129+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: audit 2024-04-26T16:31:46.297129+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: audit 2024-04-26T16:31:46.300684+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:31:46.807 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[29234]: audit 2024-04-26T16:31:46.300684+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.167031+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.a scrub starts 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.167031+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.a scrub starts 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.169400+0000 osd.1 (osd.1) 316 : cluster [DBG] 2.a scrub ok 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.169400+0000 osd.1 (osd.1) 316 : cluster [DBG] 2.a scrub ok 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.419186+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.419186+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.421494+0000 osd.6 (osd.6) 76 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.421494+0000 osd.6 (osd.6) 76 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.862013+0000 osd.0 (osd.0) 347 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.862013+0000 osd.0 (osd.0) 347 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:31:46.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.864532+0000 osd.0 (osd.0) 348 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.864532+0000 osd.0 (osd.0) 348 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: audit 2024-04-26T16:31:46.276007+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: audit 2024-04-26T16:31:46.276007+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:46.280941+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e572: 8 total, 8 up, 7 in 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: cluster 2024-04-26T16:31:46.280941+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e572: 8 total, 8 up, 7 in 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: audit 2024-04-26T16:31:46.297129+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: audit 2024-04-26T16:31:46.297129+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: audit 2024-04-26T16:31:46.300684+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:31:46.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:46 smithi050 bash[21463]: audit 2024-04-26T16:31:46.300684+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:31:46.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.167031+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.a scrub starts 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.167031+0000 osd.1 (osd.1) 315 : cluster [DBG] 2.a scrub starts 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.169400+0000 osd.1 (osd.1) 316 : cluster [DBG] 2.a scrub ok 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.169400+0000 osd.1 (osd.1) 316 : cluster [DBG] 2.a scrub ok 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.419186+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.419186+0000 osd.6 (osd.6) 75 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.421494+0000 osd.6 (osd.6) 76 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.421494+0000 osd.6 (osd.6) 76 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.862013+0000 osd.0 (osd.0) 347 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:31:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.862013+0000 osd.0 (osd.0) 347 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.864532+0000 osd.0 (osd.0) 348 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.864532+0000 osd.0 (osd.0) 348 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: audit 2024-04-26T16:31:46.276007+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: audit 2024-04-26T16:31:46.276007+0000 mon.a (mon.0) 3035 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:46.280941+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e572: 8 total, 8 up, 7 in 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: cluster 2024-04-26T16:31:46.280941+0000 mon.a (mon.0) 3036 : cluster [DBG] osdmap e572: 8 total, 8 up, 7 in 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: audit 2024-04-26T16:31:46.297129+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: audit 2024-04-26T16:31:46.297129+0000 mon.b (mon.1) 401 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: audit 2024-04-26T16:31:46.300684+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:31:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:46 smithi190 bash[23868]: audit 2024-04-26T16:31:46.300684+0000 mon.a (mon.0) 3037 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:31:46.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:46.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:47.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:47.082 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:47.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:47.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:47.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.277+0000 7f5e8a7fc700 1 -- 172.21.15.50:0/1541691580 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v573) ==== 87+0+0 (secure 0 0 0) 0x7f5e90010040 con 0x7f5e981094c0 2024-04-26T16:31:47.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 10 librados: watch_flush enter 2024-04-26T16:31:47.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 10 librados: watch_flush exit 2024-04-26T16:31:47.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 20 client.30777.objecter shutdown clearing up homeless session... 2024-04-26T16:31:47.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 10 client.30777.objecter successfully canceled tick 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e84071f80 msgr2=0x7f5e84074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e84071f80 0x7f5e84074440 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400b2e0 tx=0x7f5e94005960 comp rx=0 tx=0).stop 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 msgr2=0x7f5e9819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 0x7f5e9819dad0 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000c370 tx=0x7f5e9000c680 comp rx=0 tx=0).stop 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 shutdown_connections 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5e84071f80 0x7f5e84074440 unknown :-1 s=CLOSED pgs=3777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5e9806baf0 0x7f5e98199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5e981094c0 0x7f5e9819dad0 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:47.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 --2- 172.21.15.50:0/1541691580 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5e9806c490 0x7f5e98199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:47.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.281+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 >> 172.21.15.50:0/1541691580 conn(0x7f5e980792d0 msgr2=0x7f5e98100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.285+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 shutdown_connections 2024-04-26T16:31:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.285+0000 7f5e9f3b5700 1 -- 172.21.15.50:0/1541691580 wait complete. 2024-04-26T16:31:47.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:47.285+0000 7f5e9f3b5700 1 librados: shutdown 2024-04-26T16:31:47.283 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:31:47.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:47.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:31:47.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:47.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:47.464 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.752554+0000 mgr.y (mgr.24401) 558 : cluster [DBG] pgmap v874: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:45.752554+0000 mgr.y (mgr.24401) 558 : cluster [DBG] pgmap v874: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:46.842238+0000 osd.0 (osd.0) 349 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:46.842238+0000 osd.0 (osd.0) 349 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:46.843815+0000 osd.0 (osd.0) 350 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:46.843815+0000 osd.0 (osd.0) 350 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.276734+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.276734+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:47.551 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: audit 2024-04-26T16:31:47.279082+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:31:47.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: audit 2024-04-26T16:31:47.279082+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:31:47.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.290743+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e573: 8 total, 8 up, 7 in 2024-04-26T16:31:47.552 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.290743+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e573: 8 total, 8 up, 7 in 2024-04-26T16:31:47.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:47.588 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:47.594 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:31:47.594 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:31:47.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.752554+0000 mgr.y (mgr.24401) 558 : cluster [DBG] pgmap v874: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:47.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:45.752554+0000 mgr.y (mgr.24401) 558 : cluster [DBG] pgmap v874: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:47.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:46.842238+0000 osd.0 (osd.0) 349 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:31:47.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:46.842238+0000 osd.0 (osd.0) 349 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:31:47.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:46.843815+0000 osd.0 (osd.0) 350 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:31:47.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:46.843815+0000 osd.0 (osd.0) 350 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:31:47.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.276734+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:47.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.276734+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:47.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: audit 2024-04-26T16:31:47.279082+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:31:47.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: audit 2024-04-26T16:31:47.279082+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:31:47.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.290743+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e573: 8 total, 8 up, 7 in 2024-04-26T16:31:47.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:47 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.290743+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e573: 8 total, 8 up, 7 in 2024-04-26T16:31:47.664 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.752554+0000 mgr.y (mgr.24401) 558 : cluster [DBG] pgmap v874: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:47.664 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:45.752554+0000 mgr.y (mgr.24401) 558 : cluster [DBG] pgmap v874: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail 2024-04-26T16:31:47.664 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:46.842238+0000 osd.0 (osd.0) 349 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:31:47.665 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:46.842238+0000 osd.0 (osd.0) 349 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:31:47.665 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:46.843815+0000 osd.0 (osd.0) 350 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:31:47.665 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:46.843815+0000 osd.0 (osd.0) 350 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:31:47.665 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.276734+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:47.665 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.276734+0000 mon.a (mon.0) 3038 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:31:47.665 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: audit 2024-04-26T16:31:47.279082+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:31:47.666 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: audit 2024-04-26T16:31:47.279082+0000 mon.a (mon.0) 3039 : audit [INF] from='client.? 172.21.15.50:0/1541691580' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:31:47.666 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.290743+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e573: 8 total, 8 up, 7 in 2024-04-26T16:31:47.666 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:47 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.290743+0000 mon.a (mon.0) 3040 : cluster [DBG] osdmap e573: 8 total, 8 up, 7 in 2024-04-26T16:31:47.690 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:47.720 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:47.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:47.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:47.853 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:47.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:47.984 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:47.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:48.087 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:48.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:48.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:48.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:48.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:48.374 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:48.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:48.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:48.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-26T16:31:48.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:48.618 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:47.039891+0000 mgr.y (mgr.24401) 559 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:47.039891+0000 mgr.y (mgr.24401) 559 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.140112+0000 osd.1 (osd.1) 317 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.140112+0000 osd.1 (osd.1) 317 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.142198+0000 osd.1 (osd.1) 318 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.142198+0000 osd.1 (osd.1) 318 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.426393+0000 osd.6 (osd.6) 77 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.426393+0000 osd.6 (osd.6) 77 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:48.285971+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e574: 8 total, 8 up, 7 in 2024-04-26T16:31:48.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: cluster 2024-04-26T16:31:48.285971+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e574: 8 total, 8 up, 7 in 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.343804+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.343804+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.343927+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.343927+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.345152+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.345152+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.345200+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[21463]: audit 2024-04-26T16:31:48.345200+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:47.039891+0000 mgr.y (mgr.24401) 559 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:47.039891+0000 mgr.y (mgr.24401) 559 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.140112+0000 osd.1 (osd.1) 317 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:31:48.620 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.140112+0000 osd.1 (osd.1) 317 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.142198+0000 osd.1 (osd.1) 318 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.142198+0000 osd.1 (osd.1) 318 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.426393+0000 osd.6 (osd.6) 77 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.426393+0000 osd.6 (osd.6) 77 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:48.285971+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e574: 8 total, 8 up, 7 in 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: cluster 2024-04-26T16:31:48.285971+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e574: 8 total, 8 up, 7 in 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.343804+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.343804+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.343927+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.343927+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.345152+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.345152+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.345200+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.621 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:48 smithi050 bash[29234]: audit 2024-04-26T16:31:48.345200+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.631 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:48.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:48.753 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:48.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:47.039891+0000 mgr.y (mgr.24401) 559 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:48.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:47.039891+0000 mgr.y (mgr.24401) 559 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:48.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.140112+0000 osd.1 (osd.1) 317 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.140112+0000 osd.1 (osd.1) 317 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.142198+0000 osd.1 (osd.1) 318 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.142198+0000 osd.1 (osd.1) 318 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.426393+0000 osd.6 (osd.6) 77 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.426393+0000 osd.6 (osd.6) 77 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:48.285971+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e574: 8 total, 8 up, 7 in 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: cluster 2024-04-26T16:31:48.285971+0000 mon.a (mon.0) 3041 : cluster [DBG] osdmap e574: 8 total, 8 up, 7 in 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.343804+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.343804+0000 mon.b (mon.1) 402 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.343927+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.343927+0000 mon.a (mon.0) 3042 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.345152+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.345152+0000 mon.b (mon.1) 403 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.345200+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:48 smithi190 bash[23868]: audit 2024-04-26T16:31:48.345200+0000 mon.a (mon.0) 3043 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-92"}]: dispatch 2024-04-26T16:31:48.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:48.881 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:48.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:48.973 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:48.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:49.009 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:49.033 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:49.112 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:49.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:49.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:49.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:49.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:49.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:49.400 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:49.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:49.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:49.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:49 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.753401+0000 mgr.y (mgr.24401) 560 : cluster [DBG] pgmap v877: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:31:49.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:49 smithi190 bash[23868]: cluster 2024-04-26T16:31:47.753401+0000 mgr.y (mgr.24401) 560 : cluster [DBG] pgmap v877: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:31:49.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:49 smithi190 bash[23868]: cluster 2024-04-26T16:31:48.552266+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:49.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:49 smithi190 bash[23868]: cluster 2024-04-26T16:31:48.552266+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:49.646 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:49 smithi190 bash[23868]: cluster 2024-04-26T16:31:49.288465+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e575: 8 total, 8 up, 7 in 2024-04-26T16:31:49.647 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:49 smithi190 bash[23868]: cluster 2024-04-26T16:31:49.288465+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e575: 8 total, 8 up, 7 in 2024-04-26T16:31:49.662 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:49.765 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:49.789 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.753401+0000 mgr.y (mgr.24401) 560 : cluster [DBG] pgmap v877: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[21463]: cluster 2024-04-26T16:31:47.753401+0000 mgr.y (mgr.24401) 560 : cluster [DBG] pgmap v877: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[21463]: cluster 2024-04-26T16:31:48.552266+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[21463]: cluster 2024-04-26T16:31:48.552266+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[21463]: cluster 2024-04-26T16:31:49.288465+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e575: 8 total, 8 up, 7 in 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[21463]: cluster 2024-04-26T16:31:49.288465+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e575: 8 total, 8 up, 7 in 2024-04-26T16:31:49.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.753401+0000 mgr.y (mgr.24401) 560 : cluster [DBG] pgmap v877: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:31:49.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[29234]: cluster 2024-04-26T16:31:47.753401+0000 mgr.y (mgr.24401) 560 : cluster [DBG] pgmap v877: 228 pgs: 8 unknown, 220 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:31:49.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[29234]: cluster 2024-04-26T16:31:48.552266+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:49.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[29234]: cluster 2024-04-26T16:31:48.552266+0000 mon.a (mon.0) 3044 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:31:49.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[29234]: cluster 2024-04-26T16:31:49.288465+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e575: 8 total, 8 up, 7 in 2024-04-26T16:31:49.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:49 smithi050 bash[29234]: cluster 2024-04-26T16:31:49.288465+0000 mon.a (mon.0) 3045 : cluster [DBG] osdmap e575: 8 total, 8 up, 7 in 2024-04-26T16:31:49.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:49.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:49.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:50.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:50.049 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:50.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:50.180 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:50.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:50.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:50.421 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:50.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:50.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:50.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:31:50.576 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:50.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:50.706 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:50.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:50.836 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:50.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:50.939 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:50.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:50.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:51.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:51.088 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:51.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:51.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:51.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:51.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:51.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:51.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: cluster 2024-04-26T16:31:49.754259+0000 mgr.y (mgr.24401) 561 : cluster [DBG] pgmap v880: 196 pgs: 1 active+clean+scrubbing, 195 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:31:51.461 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: cluster 2024-04-26T16:31:49.754259+0000 mgr.y (mgr.24401) 561 : cluster [DBG] pgmap v880: 196 pgs: 1 active+clean+scrubbing, 195 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: cluster 2024-04-26T16:31:50.295164+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e576: 8 total, 8 up, 7 in 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: cluster 2024-04-26T16:31:50.295164+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e576: 8 total, 8 up, 7 in 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: audit 2024-04-26T16:31:50.304687+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: audit 2024-04-26T16:31:50.304687+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: audit 2024-04-26T16:31:50.306156+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: audit 2024-04-26T16:31:50.306156+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: audit 2024-04-26T16:31:51.253252+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: audit 2024-04-26T16:31:51.253252+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: cluster 2024-04-26T16:31:51.257664+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e577: 8 total, 8 up, 7 in 2024-04-26T16:31:51.462 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[29234]: cluster 2024-04-26T16:31:51.257664+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e577: 8 total, 8 up, 7 in 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: cluster 2024-04-26T16:31:49.754259+0000 mgr.y (mgr.24401) 561 : cluster [DBG] pgmap v880: 196 pgs: 1 active+clean+scrubbing, 195 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: cluster 2024-04-26T16:31:49.754259+0000 mgr.y (mgr.24401) 561 : cluster [DBG] pgmap v880: 196 pgs: 1 active+clean+scrubbing, 195 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: cluster 2024-04-26T16:31:50.295164+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e576: 8 total, 8 up, 7 in 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: cluster 2024-04-26T16:31:50.295164+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e576: 8 total, 8 up, 7 in 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: audit 2024-04-26T16:31:50.304687+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: audit 2024-04-26T16:31:50.304687+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: audit 2024-04-26T16:31:50.306156+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: audit 2024-04-26T16:31:50.306156+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: audit 2024-04-26T16:31:51.253252+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:51.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: audit 2024-04-26T16:31:51.253252+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:51.464 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: cluster 2024-04-26T16:31:51.257664+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e577: 8 total, 8 up, 7 in 2024-04-26T16:31:51.464 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:51 smithi050 bash[21463]: cluster 2024-04-26T16:31:51.257664+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e577: 8 total, 8 up, 7 in 2024-04-26T16:31:51.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:51.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: cluster 2024-04-26T16:31:49.754259+0000 mgr.y (mgr.24401) 561 : cluster [DBG] pgmap v880: 196 pgs: 1 active+clean+scrubbing, 195 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:31:51.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: cluster 2024-04-26T16:31:49.754259+0000 mgr.y (mgr.24401) 561 : cluster [DBG] pgmap v880: 196 pgs: 1 active+clean+scrubbing, 195 active+clean; 8.4 MiB data, 538 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: cluster 2024-04-26T16:31:50.295164+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e576: 8 total, 8 up, 7 in 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: cluster 2024-04-26T16:31:50.295164+0000 mon.a (mon.0) 3046 : cluster [DBG] osdmap e576: 8 total, 8 up, 7 in 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: audit 2024-04-26T16:31:50.304687+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: audit 2024-04-26T16:31:50.304687+0000 mon.b (mon.1) 404 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: audit 2024-04-26T16:31:50.306156+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: audit 2024-04-26T16:31:50.306156+0000 mon.a (mon.0) 3047 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: audit 2024-04-26T16:31:51.253252+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: audit 2024-04-26T16:31:51.253252+0000 mon.a (mon.0) 3048 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: cluster 2024-04-26T16:31:51.257664+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e577: 8 total, 8 up, 7 in 2024-04-26T16:31:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:51 smithi190 bash[23868]: cluster 2024-04-26T16:31:51.257664+0000 mon.a (mon.0) 3049 : cluster [DBG] osdmap e577: 8 total, 8 up, 7 in 2024-04-26T16:31:51.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:51.608 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:51.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:51.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:51.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:51.872 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:51.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:51.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:52.003 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:52.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:52.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:52.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:52.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:52.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: cluster 2024-04-26T16:31:50.620409+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: cluster 2024-04-26T16:31:50.620409+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: cluster 2024-04-26T16:31:50.620775+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: cluster 2024-04-26T16:31:50.620775+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:52.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: audit 2024-04-26T16:31:51.283538+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: audit 2024-04-26T16:31:51.283538+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: cluster 2024-04-26T16:31:52.265321+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e578: 8 total, 8 up, 7 in 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[21463]: cluster 2024-04-26T16:31:52.265321+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e578: 8 total, 8 up, 7 in 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: cluster 2024-04-26T16:31:50.620409+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: cluster 2024-04-26T16:31:50.620409+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: cluster 2024-04-26T16:31:50.620775+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:52.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: cluster 2024-04-26T16:31:50.620775+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: audit 2024-04-26T16:31:51.283538+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: audit 2024-04-26T16:31:51.283538+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: cluster 2024-04-26T16:31:52.265321+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e578: 8 total, 8 up, 7 in 2024-04-26T16:31:52.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:52 smithi050 bash[29234]: cluster 2024-04-26T16:31:52.265321+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e578: 8 total, 8 up, 7 in 2024-04-26T16:31:52.401 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:52.504 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:52.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: cluster 2024-04-26T16:31:50.620409+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: cluster 2024-04-26T16:31:50.620409+0000 osd.5 (osd.5) 209 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: cluster 2024-04-26T16:31:50.620775+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: cluster 2024-04-26T16:31:50.620775+0000 osd.5 (osd.5) 210 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: audit 2024-04-26T16:31:51.283538+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: audit 2024-04-26T16:31:51.283538+0000 mon.b (mon.1) 405 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: cluster 2024-04-26T16:31:52.265321+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e578: 8 total, 8 up, 7 in 2024-04-26T16:31:52.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:52 smithi190 bash[23868]: cluster 2024-04-26T16:31:52.265321+0000 mon.a (mon.0) 3050 : cluster [DBG] osdmap e578: 8 total, 8 up, 7 in 2024-04-26T16:31:52.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:52.663 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:52.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:52.792 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:52.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:52.922 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:52.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:53.026 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:53.055 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:53.158 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:53.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.315 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.317 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:31:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:53.418 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:53.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:53.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:53.559 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:53 smithi190 bash[23868]: cluster 2024-04-26T16:31:51.756562+0000 mgr.y (mgr.24401) 562 : cluster [DBG] pgmap v883: 228 pgs: 4 unknown, 1 active+clean+scrubbing, 223 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:31:53.559 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:53 smithi190 bash[23868]: cluster 2024-04-26T16:31:51.756562+0000 mgr.y (mgr.24401) 562 : cluster [DBG] pgmap v883: 228 pgs: 4 unknown, 1 active+clean+scrubbing, 223 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:31:53.559 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:53 smithi190 bash[23868]: cluster 2024-04-26T16:31:53.263064+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e579: 8 total, 8 up, 7 in 2024-04-26T16:31:53.559 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:53 smithi190 bash[23868]: cluster 2024-04-26T16:31:53.263064+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e579: 8 total, 8 up, 7 in 2024-04-26T16:31:53.575 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[21463]: cluster 2024-04-26T16:31:51.756562+0000 mgr.y (mgr.24401) 562 : cluster [DBG] pgmap v883: 228 pgs: 4 unknown, 1 active+clean+scrubbing, 223 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[21463]: cluster 2024-04-26T16:31:51.756562+0000 mgr.y (mgr.24401) 562 : cluster [DBG] pgmap v883: 228 pgs: 4 unknown, 1 active+clean+scrubbing, 223 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[21463]: cluster 2024-04-26T16:31:53.263064+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e579: 8 total, 8 up, 7 in 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[21463]: cluster 2024-04-26T16:31:53.263064+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e579: 8 total, 8 up, 7 in 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[29234]: cluster 2024-04-26T16:31:51.756562+0000 mgr.y (mgr.24401) 562 : cluster [DBG] pgmap v883: 228 pgs: 4 unknown, 1 active+clean+scrubbing, 223 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[29234]: cluster 2024-04-26T16:31:51.756562+0000 mgr.y (mgr.24401) 562 : cluster [DBG] pgmap v883: 228 pgs: 4 unknown, 1 active+clean+scrubbing, 223 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[29234]: cluster 2024-04-26T16:31:53.263064+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e579: 8 total, 8 up, 7 in 2024-04-26T16:31:53.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:53 smithi050 bash[29234]: cluster 2024-04-26T16:31:53.263064+0000 mon.a (mon.0) 3051 : cluster [DBG] osdmap e579: 8 total, 8 up, 7 in 2024-04-26T16:31:53.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:53.703 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:53.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:53.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:53.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:54.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:54.095 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:54.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:54.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:54.224 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:54.327 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:54.355 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:54.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:54.480 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:54.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:54.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:54.593 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:31:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:31:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:31:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:54.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:54.739 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:54.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:54.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:54.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:54.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:54.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:54.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 -- 172.21.15.50:0/1092744488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b8109490 msgr2=0x7fe0b8109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 --2- 172.21.15.50:0/1092744488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b8109490 0x7fe0b8109870 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009990 tx=0x7fe0ac009ca0 comp rx=0 tx=0).stop 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 -- 172.21.15.50:0/1092744488 shutdown_connections 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 --2- 172.21.15.50:0/1092744488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0b806c360 0x7fe0b81102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 --2- 172.21.15.50:0/1092744488 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b8109490 0x7fe0b8109870 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 --2- 172.21.15.50:0/1092744488 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 0x7fe0b806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 -- 172.21.15.50:0/1092744488 >> 172.21.15.50:0/1092744488 conn(0x7fe0b8079290 msgr2=0x7fe0b80796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 -- 172.21.15.50:0/1092744488 shutdown_connections 2024-04-26T16:31:54.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 -- 172.21.15.50:0/1092744488 wait complete. 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 librados: starting msgr at 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 librados: starting objecter 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 Processor -- start 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 -- start start 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 librados: setting wanted keys 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.993+0000 7fe0beefe700 1 librados: calling monclient init 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 0x7fe0b8199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b806c360 0x7fe0b8199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0b8109490 0x7fe0b819d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:54.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8112a70 con 0x7fe0b806b9c0 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81128f0 con 0x7fe0b8109490 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b8112bf0 con 0x7fe0b806c360 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b806c360 0x7fe0b8199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0bcc9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 0x7fe0b8199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0bcc9a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 0x7fe0b8199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36392/0 (socket says 172.21.15.50:36392) 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0bcc9a700 1 -- 172.21.15.50:0/378990714 learned_addr learned my addr 172.21.15.50:0/378990714 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b806c360 0x7fe0b8199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40074/0 (socket says 172.21.15.50:40074) 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0b8109490 0x7fe0b819d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0b8109490 msgr2=0x7fe0b819d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0b8109490 0x7fe0b819d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 msgr2=0x7fe0b8199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 0x7fe0b8199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b819e110 con 0x7fe0b806c360 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe0b806b9c0 0x7fe0b8199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:31:54.995 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b806c360 0x7fe0b8199640 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009c50 tx=0x7fe0a4009f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:54.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 10 client.?.objecter ms_handle_connect 0x7fe0b806c360 2024-04-26T16:31:54.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:54.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a401e070 con 0x7fe0b806c360 2024-04-26T16:31:54.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe0a4019410 con 0x7fe0b806c360 2024-04-26T16:31:54.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b819e400 con 0x7fe0b806c360 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4018610 con 0x7fe0b806c360 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 10 client.41399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b8102310 con 0x7fe0b806c360 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 librados: init done 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fe0b806a110 con 0x7fe0b806c360 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe0a4021070 con 0x7fe0b806c360 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe0a8071f80 0x7fe0a8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:54.997 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(579..579 src has 496..579) ==== 9377+0+0 (secure 0 0 0) 0x7fe0a4094fd0 con 0x7fe0b806c360 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 osd_map(579..579 src has 496..579) 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 3 client.41399.objecter handle_osd_map got epochs [579,579] > 0 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 3 client.41399.objecter handle_osd_map decoding full epoch 579 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe0a8071f80 0x7fe0a8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 20 client.41399.objecter dump_active .. 0 homeless 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:54.997+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v579) ==== 54+0+16 (secure 0 0 0) 0x7fe0a40975f0 con 0x7fe0b806c360 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 10 client.41399.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 1 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe0b8100c50 0x7fe0b802d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:54.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 1 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 0 1 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 4 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 10 client.41399.objecter _send_command 1 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0b8064540 con 0x7fe0b8100c50 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0a8071f80 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe0a8071f80 0x7fe0a8074440 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac006010 tx=0x7fe0ac00b630 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe0b8100c50 0x7fe0b802d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b8100c50 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe0b8100c50 0x7fe0b802d310 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:54.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b8064540 con 0x7fe0b8100c50 2024-04-26T16:31:55.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 1: 0 ) 2024-04-26T16:31:55.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 1 = system:0 2024-04-26T16:31:55.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 4 2024-04-26T16:31:55.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.001+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 0 1 2024-04-26T16:31:55.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 10 client.41399.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 2 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 2 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8100ae0 osd=0 3 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 0 2 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 4 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 10 client.41399.objecter _send_command 2 2024-04-26T16:31:55.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.037+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b802d850 con 0x7fe0b8100c50 2024-04-26T16:31:55.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b802d850 con 0x7fe0b8100c50 2024-04-26T16:31:55.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 4 2024-04-26T16:31:55.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 0 2 2024-04-26T16:31:55.039 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:31:55.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 10 client.41399.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe0b8063c20 0x7fe0b81025d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 1 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 4 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 10 client.41399.objecter _send_command 3 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fe0b81042b0 con 0x7fe0b8063c20 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe0b8063c20 0x7fe0b81025d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b8063c20 2024-04-26T16:31:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe0b8063c20 0x7fe0b81025d0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b81042b0 con 0x7fe0b8063c20 2024-04-26T16:31:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 3: 0 ) 2024-04-26T16:31:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 3 = system:0 2024-04-26T16:31:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 4 2024-04-26T16:31:55.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.041+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 1 3 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 10 client.41399.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 4 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 4 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8063ab0 osd=1 3 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 1 4 2024-04-26T16:31:55.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 4 2024-04-26T16:31:55.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 10 client.41399.objecter _send_command 4 2024-04-26T16:31:55.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b8102b10 con 0x7fe0b8063c20 2024-04-26T16:31:55.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b8102b10 con 0x7fe0b8063c20 2024-04-26T16:31:55.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 4 2024-04-26T16:31:55.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 1 4 2024-04-26T16:31:55.060 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 10 client.41399.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 5 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe0b810a9e0 0x7fe0b810ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 5 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 2 5 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 4 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 10 client.41399.objecter _send_command 5 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fe0b8104010 con 0x7fe0b810a9e0 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.061+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe0b810a9e0 0x7fe0b810ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b810a9e0 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe0b810a9e0 0x7fe0b810ae00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b8104010 con 0x7fe0b810a9e0 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 5: 0 ) 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 5 = system:0 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 4 2024-04-26T16:31:55.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.065+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 2 5 2024-04-26T16:31:55.090 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:55.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 10 client.41399.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 6 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 6 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8103ea0 osd=2 3 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 2 6 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 4 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 10 client.41399.objecter _send_command 6 2024-04-26T16:31:55.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.101+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b810a460 con 0x7fe0b810a9e0 2024-04-26T16:31:55.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b810a460 con 0x7fe0b810a9e0 2024-04-26T16:31:55.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 4 2024-04-26T16:31:55.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 2 6 2024-04-26T16:31:55.101 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 10 client.41399.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 7 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe0b81039a0 0x7fe0b81aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 7 2024-04-26T16:31:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 3 7 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 4 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 10 client.41399.objecter _send_command 7 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fe0b810a820 con 0x7fe0b81039a0 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe0b81039a0 0x7fe0b81aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b81039a0 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe0b81039a0 0x7fe0b81aee40 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b00166f0 con 0x7fe0b81039a0 2024-04-26T16:31:55.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 7: 0 ) 2024-04-26T16:31:55.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 7 = system:0 2024-04-26T16:31:55.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 4 2024-04-26T16:31:55.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.105+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 3 7 2024-04-26T16:31:55.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:55.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 10 client.41399.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 8 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 8 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b810a6b0 osd=3 3 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 3 8 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 4 2024-04-26T16:31:55.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 10 client.41399.objecter _send_command 8 2024-04-26T16:31:55.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.125+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b8102f70 con 0x7fe0b81039a0 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/1711453380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 msgr2=0x7fe55010f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- 172.21.15.50:0/1711453380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55010f1d0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fe548009f90 tx=0x7fe54800b330 comp rx=0 tx=0).stop 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/1711453380 shutdown_connections 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- 172.21.15.50:0/1711453380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 0x7fe550108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- 172.21.15.50:0/1711453380 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55010f1d0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- 172.21.15.50:0/1711453380 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe5501027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/1711453380 >> 172.21.15.50:0/1711453380 conn(0x7fe5500792d0 msgr2=0x7fe5500796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/1711453380 shutdown_connections 2024-04-26T16:31:55.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/1711453380 wait complete. 2024-04-26T16:31:55.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b8102f70 con 0x7fe0b81039a0 2024-04-26T16:31:55.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 4 2024-04-26T16:31:55.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 3 8 2024-04-26T16:31:55.130 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 10 client.41399.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 9 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe0b81ac990 0x7fe0b81b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 9 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 4 9 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 4 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 10 client.41399.objecter _send_command 9 2024-04-26T16:31:55.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fe0b81b5370 con 0x7fe0b81ac990 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 librados: starting msgr at 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 librados: starting objecter 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 Processor -- start 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- start start 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 librados: setting wanted keys 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 librados: calling monclient init 2024-04-26T16:31:55.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe0b81ac990 0x7fe0b81b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe55019b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55019b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 0x7fe55019fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe55019b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe550111970 con 0x7fe550102ce0 2024-04-26T16:31:55.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe55019b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40096/0 (socket says 172.21.15.50:40096) 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe5501117f0 con 0x7fe550108390 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 -- 172.21.15.50:0/3948475160 learned_addr learned my addr 172.21.15.50:0/3948475160 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe550111af0 con 0x7fe550102320 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54effd700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55019b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54ffff700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 0x7fe55019fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 -- 172.21.15.50:0/3948475160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 msgr2=0x7fe55019fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 0x7fe55019fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 -- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 msgr2=0x7fe55019b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55019b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b81ac990 2024-04-26T16:31:55.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe0b81ac990 0x7fe0b81b2510 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 -- 172.21.15.50:0/3948475160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5501a02a0 con 0x7fe550102320 2024-04-26T16:31:55.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54effd700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55019b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:31:55.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54ffff700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 0x7fe55019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54f7fe700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe55019b290 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fe54000d950 tx=0x7fe54000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54cff9700 10 client.?.objecter ms_handle_connect 0x7fe550102320 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54cff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540016070 con 0x7fe550102320 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe540009da0 con 0x7fe550102320 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5501a0590 con 0x7fe550102320 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.129+0000 7fe5561a0700 10 client.41405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540010e60 con 0x7fe550102320 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5501a7e10 con 0x7fe550102320 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe5561a0700 1 librados: init done 2024-04-26T16:31:55.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b81b5370 con 0x7fe0b81ac990 2024-04-26T16:31:55.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 9: 0 ) 2024-04-26T16:31:55.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 9 = system:0 2024-04-26T16:31:55.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 4 2024-04-26T16:31:55.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 4 9 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe550103a40 con 0x7fe550102320 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe540019070 con 0x7fe550102320 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe538071e50 0x7fe538074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(579..579 src has 496..579) ==== 9377+0+0 (secure 0 0 0) 0x7fe540090ef0 con 0x7fe550102320 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 10 client.41405.objecter ms_dispatch 0x7fe550001550 osd_map(579..579 src has 496..579) 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 3 client.41405.objecter handle_osd_map got epochs [579,579] > 0 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 3 client.41405.objecter handle_osd_map decoding full epoch 579 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54effd700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe538071e50 0x7fe538074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 20 client.41405.objecter dump_active .. 0 homeless 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54cff9700 10 client.41405.objecter ms_handle_connect 0x7fe538071e50 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.133+0000 7fe54effd700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe538071e50 0x7fe538074310 secure :-1 s=READY pgs=3779 cs=0 l=1 rev1=1 crypto rx=0x7fe544009750 tx=0x7fe544009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.137+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fe54005c920 con 0x7fe550102320 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 10 client.41399.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 10 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 10 2024-04-26T16:31:55.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b8102d20 osd=4 3 2024-04-26T16:31:55.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 4 10 2024-04-26T16:31:55.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 4 2024-04-26T16:31:55.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 10 client.41399.objecter _send_command 10 2024-04-26T16:31:55.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.149+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b81b2a50 con 0x7fe0b81ac990 2024-04-26T16:31:55.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b81b2a50 con 0x7fe0b81ac990 2024-04-26T16:31:55.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 4 2024-04-26T16:31:55.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 4 10 2024-04-26T16:31:55.153 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 10 client.41399.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 11 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe0b81b30d0 0x7fe0b81b54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 11 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 5 11 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 4 2024-04-26T16:31:55.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 10 client.41399.objecter _send_command 11 2024-04-26T16:31:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fe0b81b8510 con 0x7fe0b81b30d0 2024-04-26T16:31:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe0b81b30d0 0x7fe0b81b54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b81b30d0 2024-04-26T16:31:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b7fff700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe0b81b30d0 0x7fe0b81b54e0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b81b8510 con 0x7fe0b81b30d0 2024-04-26T16:31:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 11: 0 ) 2024-04-26T16:31:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 11 = system:0 2024-04-26T16:31:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 4 2024-04-26T16:31:55.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.157+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 5 11 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 10 client.41399.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 12 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 12 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b81b2a50 osd=5 3 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 5 12 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 4 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 10 client.41399.objecter _send_command 12 2024-04-26T16:31:55.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.177+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b81b72e0 con 0x7fe0b81b30d0 2024-04-26T16:31:55.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b81b72e0 con 0x7fe0b81b30d0 2024-04-26T16:31:55.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 4 2024-04-26T16:31:55.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 5 12 2024-04-26T16:31:55.178 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 10 client.41399.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 13 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fe0b81b7a00 0x7fe0b81b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 13 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 6 13 2024-04-26T16:31:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 4 2024-04-26T16:31:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 10 client.41399.objecter _send_command 13 2024-04-26T16:31:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fe0b81bb930 con 0x7fe0b81b7a00 2024-04-26T16:31:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fe0b81b7a00 0x7fe0b81b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b81b7a00 2024-04-26T16:31:55.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.181+0000 7fe0bd49b700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fe0b81b7a00 0x7fe0b81b7e60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.185+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b81bb930 con 0x7fe0b81b7a00 2024-04-26T16:31:55.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.185+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 13: 0 ) 2024-04-26T16:31:55.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.185+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 13 = system:0 2024-04-26T16:31:55.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.185+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 4 2024-04-26T16:31:55.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.185+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 6 13 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 10 client.41399.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 14 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 14 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b81b72e0 osd=6 3 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 6 14 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 4 2024-04-26T16:31:55.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 10 client.41399.objecter _send_command 14 2024-04-26T16:31:55.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b81bb930 con 0x7fe0b81b7a00 2024-04-26T16:31:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b81bb930 con 0x7fe0b81b7a00 2024-04-26T16:31:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 4 2024-04-26T16:31:55.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 6 14 2024-04-26T16:31:55.200 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 10 client.41399.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 15 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe0b81bac70 0x7fe0b81bb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.201 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 15 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 7 15 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 4 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 10 client.41399.objecter _send_command 15 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.201+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fe0b81becf0 con 0x7fe0b81bac70 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe0b81bac70 0x7fe0b81bb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0b5ffb700 10 client.41399.objecter ms_handle_connect 0x7fe0b81bac70 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0bcc9a700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe0b81bac70 0x7fe0b81bb0b0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe0b81becf0 con 0x7fe0b81bac70 2024-04-26T16:31:55.202 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 15: 0 ) 2024-04-26T16:31:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 15 = system:0 2024-04-26T16:31:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 4 2024-04-26T16:31:55.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.205+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 7 15 2024-04-26T16:31:55.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 10 client.41399.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign -1 16 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 20 client.41399.objecter _get_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_remove -1 16 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 20 client.41399.objecter get_session s=0x7fe0b81bb930 osd=7 3 2024-04-26T16:31:55.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 15 client.41399.objecter _session_command_op_assign 7 16 2024-04-26T16:31:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 4 2024-04-26T16:31:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 10 client.41399.objecter _send_command 16 2024-04-26T16:31:55.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.221+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fe0b81b5b10 con 0x7fe0b81bac70 2024-04-26T16:31:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0b5ffb700 1 -- 172.21.15.50:0/378990714 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fe0b81b5b10 con 0x7fe0b81bac70 2024-04-26T16:31:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0b5ffb700 10 client.41399.objecter ms_dispatch 0x7fe0b8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:31:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0b5ffb700 10 client.41399.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:31:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0b5ffb700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 4 2024-04-26T16:31:55.223 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0b5ffb700 15 client.41399.objecter _session_command_op_remove 7 16 2024-04-26T16:31:55.224 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 librados: watch_flush enter 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 librados: watch_flush exit 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.0 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe0b8100c50 msgr2=0x7fe0b802d310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fe0b8100c50 0x7fe0b802d310 crc :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8100ae0 osd=0 1 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.1 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe0b8063c20 msgr2=0x7fe0b81025d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fe0b8063c20 0x7fe0b81025d0 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8063ab0 osd=1 1 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.2 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe0b810a9e0 msgr2=0x7fe0b810ae00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fe0b810a9e0 0x7fe0b810ae00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8103ea0 osd=2 1 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.3 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe0b81039a0 msgr2=0x7fe0b81aee40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fe0b81039a0 0x7fe0b81aee40 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b810a6b0 osd=3 1 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.4 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe0b81ac990 msgr2=0x7fe0b81b2510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fe0b81ac990 0x7fe0b81b2510 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b8102d20 osd=4 1 2024-04-26T16:31:55.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.5 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0bd49b700 1 -- 172.21.15.50:0/378990714 reap_dead start 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe0b81b30d0 msgr2=0x7fe0b81b54e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fe0b81b30d0 0x7fe0b81b54e0 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0bd49b700 1 -- 172.21.15.50:0/378990714 reap_dead start 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0bd49b700 1 -- 172.21.15.50:0/378990714 reap_dead start 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b2a50 osd=5 1 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.6 2024-04-26T16:31:55.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fe0b81b7a00 msgr2=0x7fe0b81b7e60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fe0b81b7a00 0x7fe0b81b7e60 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81b72e0 osd=6 1 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter close_session for osd.7 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe0b81bac70 msgr2=0x7fe0b81bb0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fe0b81bac70 0x7fe0b81bb0b0 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter put_session s=0x7fe0b81bb930 osd=7 1 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 20 client.41399.objecter shutdown clearing up homeless session... 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 10 client.41399.objecter successfully canceled tick 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe0a8071f80 msgr2=0x7fe0a8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe0a8071f80 0x7fe0a8074440 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac006010 tx=0x7fe0ac00b630 comp rx=0 tx=0).stop 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b806c360 msgr2=0x7fe0b8199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 --2- 172.21.15.50:0/378990714 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe0b806c360 0x7fe0b8199640 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009c50 tx=0x7fe0a4009f60 comp rx=0 tx=0).stop 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0bd49b700 1 -- 172.21.15.50:0/378990714 reap_dead start 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 shutdown_connections 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 >> 172.21.15.50:0/378990714 conn(0x7fe0b8079290 msgr2=0x7fe0b8100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 shutdown_connections 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 -- 172.21.15.50:0/378990714 wait complete. 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.225+0000 7fe0beefe700 1 librados: shutdown 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:31:55.229 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:31:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:31:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:31:55.230 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:31:55.240 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:55.332 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:31:55.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:55.365 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:55.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.373+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe55019c5f0 con 0x7fe538071e50 2024-04-26T16:31:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: cluster 2024-04-26T16:31:53.757428+0000 mgr.y (mgr.24401) 563 : cluster [DBG] pgmap v886: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:31:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: cluster 2024-04-26T16:31:53.757428+0000 mgr.y (mgr.24401) 563 : cluster [DBG] pgmap v886: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:31:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: cluster 2024-04-26T16:31:54.351365+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: cluster 2024-04-26T16:31:54.351365+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: cluster 2024-04-26T16:31:54.351594+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:55.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: cluster 2024-04-26T16:31:54.351594+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: audit 2024-04-26T16:31:55.002289+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? 172.21.15.50:0/378990714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[29234]: audit 2024-04-26T16:31:55.002289+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? 172.21.15.50:0/378990714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: cluster 2024-04-26T16:31:53.757428+0000 mgr.y (mgr.24401) 563 : cluster [DBG] pgmap v886: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: cluster 2024-04-26T16:31:53.757428+0000 mgr.y (mgr.24401) 563 : cluster [DBG] pgmap v886: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: cluster 2024-04-26T16:31:54.351365+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: cluster 2024-04-26T16:31:54.351365+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: cluster 2024-04-26T16:31:54.351594+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: cluster 2024-04-26T16:31:54.351594+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: audit 2024-04-26T16:31:55.002289+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? 172.21.15.50:0/378990714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:55.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:55 smithi050 bash[21463]: audit 2024-04-26T16:31:55.002289+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? 172.21.15.50:0/378990714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:55.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.389+0000 7fe54cff9700 1 -- 172.21.15.50:0/3948475160 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+589439 (secure 0 0 0) 0x7fe55019c5f0 con 0x7fe538071e50 2024-04-26T16:31:55.394 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:31:55.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 10 librados: watch_flush enter 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 10 librados: watch_flush exit 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 20 client.41405.objecter shutdown clearing up homeless session... 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 10 client.41405.objecter successfully canceled tick 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe538071e50 msgr2=0x7fe538074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe538071e50 0x7fe538074310 secure :-1 s=READY pgs=3779 cs=0 l=1 rev1=1 crypto rx=0x7fe544009750 tx=0x7fe544009310 comp rx=0 tx=0).stop 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 msgr2=0x7fe55019b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe55019b290 secure :-1 s=READY pgs=2700 cs=0 l=1 rev1=1 crypto rx=0x7fe54000d950 tx=0x7fe54000dc60 comp rx=0 tx=0).stop 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 shutdown_connections 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe538071e50 0x7fe538074310 unknown :-1 s=CLOSED pgs=3779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550108390 0x7fe55019fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe550102320 0x7fe55019b290 unknown :-1 s=CLOSED pgs=2700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 --2- 172.21.15.50:0/3948475160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe550102ce0 0x7fe55019b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:31:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 >> 172.21.15.50:0/3948475160 conn(0x7fe5500792d0 msgr2=0x7fe55006b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:31:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 shutdown_connections 2024-04-26T16:31:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 -- 172.21.15.50:0/3948475160 wait complete. 2024-04-26T16:31:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:31:55.393+0000 7fe5561a0700 1 librados: shutdown 2024-04-26T16:31:55.401 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:31:55.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: cluster 2024-04-26T16:31:53.757428+0000 mgr.y (mgr.24401) 563 : cluster [DBG] pgmap v886: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: cluster 2024-04-26T16:31:53.757428+0000 mgr.y (mgr.24401) 563 : cluster [DBG] pgmap v886: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: cluster 2024-04-26T16:31:54.351365+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: cluster 2024-04-26T16:31:54.351365+0000 osd.3 (osd.3) 357 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: cluster 2024-04-26T16:31:54.351594+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: cluster 2024-04-26T16:31:54.351594+0000 osd.3 (osd.3) 358 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: audit 2024-04-26T16:31:55.002289+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? 172.21.15.50:0/378990714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:55.480 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:55 smithi190 bash[23868]: audit 2024-04-26T16:31:55.002289+0000 mon.c (mon.2) 664 : audit [DBG] from='client.? 172.21.15.50:0/378990714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:31:55.496 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:55.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:55.626 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:55.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:55.756 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:55.859 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:55.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:55.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:55.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:56.020 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:31:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:31:56.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:56.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:56.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:56.288 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:56.386 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:31:56.387 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:31:56.387 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:31:56.387 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:31:56.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[29234]: cluster 2024-04-26T16:31:55.322005+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e580: 8 total, 8 up, 7 in 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[29234]: cluster 2024-04-26T16:31:55.322005+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e580: 8 total, 8 up, 7 in 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[29234]: audit 2024-04-26T16:31:55.375581+0000 mgr.y (mgr.24401) 564 : audit [DBG] from='client.41405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[29234]: audit 2024-04-26T16:31:55.375581+0000 mgr.y (mgr.24401) 564 : audit [DBG] from='client.41405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[29234]: cluster 2024-04-26T16:31:56.259173+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e581: 8 total, 8 up, 7 in 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[29234]: cluster 2024-04-26T16:31:56.259173+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e581: 8 total, 8 up, 7 in 2024-04-26T16:31:56.405 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[21463]: cluster 2024-04-26T16:31:55.322005+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e580: 8 total, 8 up, 7 in 2024-04-26T16:31:56.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[21463]: cluster 2024-04-26T16:31:55.322005+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e580: 8 total, 8 up, 7 in 2024-04-26T16:31:56.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[21463]: audit 2024-04-26T16:31:55.375581+0000 mgr.y (mgr.24401) 564 : audit [DBG] from='client.41405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:56.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[21463]: audit 2024-04-26T16:31:55.375581+0000 mgr.y (mgr.24401) 564 : audit [DBG] from='client.41405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:56.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[21463]: cluster 2024-04-26T16:31:56.259173+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e581: 8 total, 8 up, 7 in 2024-04-26T16:31:56.406 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:56 smithi050 bash[21463]: cluster 2024-04-26T16:31:56.259173+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e581: 8 total, 8 up, 7 in 2024-04-26T16:31:56.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:56.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:56.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:56.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:56 smithi190 bash[23868]: cluster 2024-04-26T16:31:55.322005+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e580: 8 total, 8 up, 7 in 2024-04-26T16:31:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:56 smithi190 bash[23868]: cluster 2024-04-26T16:31:55.322005+0000 mon.a (mon.0) 3052 : cluster [DBG] osdmap e580: 8 total, 8 up, 7 in 2024-04-26T16:31:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:56 smithi190 bash[23868]: audit 2024-04-26T16:31:55.375581+0000 mgr.y (mgr.24401) 564 : audit [DBG] from='client.41405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:56 smithi190 bash[23868]: audit 2024-04-26T16:31:55.375581+0000 mgr.y (mgr.24401) 564 : audit [DBG] from='client.41405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:31:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:56 smithi190 bash[23868]: cluster 2024-04-26T16:31:56.259173+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e581: 8 total, 8 up, 7 in 2024-04-26T16:31:56.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:56 smithi190 bash[23868]: cluster 2024-04-26T16:31:56.259173+0000 mon.a (mon.0) 3053 : cluster [DBG] osdmap e581: 8 total, 8 up, 7 in 2024-04-26T16:31:56.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:56.680 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:56.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:56.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:56.809 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:56.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:56.940 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:56.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:57.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:57.069 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:57.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-26T16:31:57.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:57.202 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:31:57.214 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:57.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:57.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:57.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:57.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[23868]: cluster 2024-04-26T16:31:55.758167+0000 mgr.y (mgr.24401) 565 : cluster [DBG] pgmap v888: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-26T16:31:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[23868]: cluster 2024-04-26T16:31:55.758167+0000 mgr.y (mgr.24401) 565 : cluster [DBG] pgmap v888: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-26T16:31:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[23868]: audit 2024-04-26T16:31:57.235653+0000 mon.a (mon.0) 3054 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[23868]: audit 2024-04-26T16:31:57.235653+0000 mon.a (mon.0) 3054 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[23868]: audit 2024-04-26T16:31:57.238047+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:57.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:57 smithi190 bash[23868]: audit 2024-04-26T16:31:57.238047+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:57.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[21463]: cluster 2024-04-26T16:31:55.758167+0000 mgr.y (mgr.24401) 565 : cluster [DBG] pgmap v888: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[21463]: cluster 2024-04-26T16:31:55.758167+0000 mgr.y (mgr.24401) 565 : cluster [DBG] pgmap v888: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[21463]: audit 2024-04-26T16:31:57.235653+0000 mon.a (mon.0) 3054 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[21463]: audit 2024-04-26T16:31:57.235653+0000 mon.a (mon.0) 3054 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[21463]: audit 2024-04-26T16:31:57.238047+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[21463]: audit 2024-04-26T16:31:57.238047+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[29234]: cluster 2024-04-26T16:31:55.758167+0000 mgr.y (mgr.24401) 565 : cluster [DBG] pgmap v888: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[29234]: cluster 2024-04-26T16:31:55.758167+0000 mgr.y (mgr.24401) 565 : cluster [DBG] pgmap v888: 228 pgs: 228 active+clean; 8.4 MiB data, 539 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[29234]: audit 2024-04-26T16:31:57.235653+0000 mon.a (mon.0) 3054 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[29234]: audit 2024-04-26T16:31:57.235653+0000 mon.a (mon.0) 3054 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:31:57.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[29234]: audit 2024-04-26T16:31:57.238047+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:57.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:57 smithi050 bash[29234]: audit 2024-04-26T16:31:57.238047+0000 mon.c (mon.2) 665 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:31:57.600 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.603 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":887,"stamp":"2024-04-26T16:31:55.314965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8811356,"num_objects":402,"num_object_clones":14,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":2438,"num_read_kb":2127,"num_write":1697,"num_write_kb":15285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":5967718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3152,"ondisk_log_size":3152,"up":684,"acting":684,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":666,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":551760,"kb_used_data":46300,"kb_used_omap":24,"kb_used_meta":505319,"kb_avail":655664304,"statfs":{"total":671965249536,"available":671400247296,"internally_reserved":0,"allocated":47411200,"data_stored":32979517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25087,"internal_metadata":517447169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":80,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.024925"},"pg_stats":[{"pgid":"313.1f","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268210+0000","last_change":"2024-04-26T16:31:51.267635+0000","last_active":"2024-04-26T16:31:51.268210+0000","last_peered":"2024-04-26T16:31:51.268210+0000","last_clean":"2024-04-26T16:31:51.268210+0000","last_became_active":"2024-04-26T16:31:51.266654+0000","last_became_peered":"2024-04-26T16:31:51.266654+0000","last_unstale":"2024-04-26T16:31:51.268210+0000","last_undegraded":"2024-04-26T16:31:51.268210+0000","last_fullsized":"2024-04-26T16:31:51.268210+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:59.552994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"313.1e","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281085+0000","last_change":"2024-04-26T16:31:51.278515+0000","last_active":"2024-04-26T16:31:51.281085+0000","last_peered":"2024-04-26T16:31:51.281085+0000","last_clean":"2024-04-26T16:31:51.281085+0000","last_became_active":"2024-04-26T16:31:51.277569+0000","last_became_peered":"2024-04-26T16:31:51.277569+0000","last_unstale":"2024-04-26T16:31:51.281085+0000","last_undegraded":"2024-04-26T16:31:51.281085+0000","last_fullsized":"2024-04-26T16:31:51.281085+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:13.553628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.1d","version":"0'0","reported_seq":13,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826408+0000","last_change":"2024-04-26T16:31:51.283530+0000","last_active":"2024-04-26T16:31:52.826408+0000","last_peered":"2024-04-26T16:31:52.826408+0000","last_clean":"2024-04-26T16:31:52.826408+0000","last_became_active":"2024-04-26T16:31:51.282674+0000","last_became_peered":"2024-04-26T16:31:51.282674+0000","last_unstale":"2024-04-26T16:31:52.826408+0000","last_undegraded":"2024-04-26T16:31:52.826408+0000","last_fullsized":"2024-04-26T16:31:52.826408+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:00.775062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.1c","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.272461+0000","last_change":"2024-04-26T16:31:51.271619+0000","last_active":"2024-04-26T16:31:51.272461+0000","last_peered":"2024-04-26T16:31:51.272461+0000","last_clean":"2024-04-26T16:31:51.272461+0000","last_became_active":"2024-04-26T16:31:51.268132+0000","last_became_peered":"2024-04-26T16:31:51.268132+0000","last_unstale":"2024-04-26T16:31:51.272461+0000","last_undegraded":"2024-04-26T16:31:51.272461+0000","last_fullsized":"2024-04-26T16:31:51.272461+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:11.764670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"313.1b","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.282690+0000","last_change":"2024-04-26T16:31:51.282212+0000","last_active":"2024-04-26T16:31:51.282690+0000","last_peered":"2024-04-26T16:31:51.282690+0000","last_clean":"2024-04-26T16:31:51.282690+0000","last_became_active":"2024-04-26T16:31:51.281800+0000","last_became_peered":"2024-04-26T16:31:51.281800+0000","last_unstale":"2024-04-26T16:31:51.282690+0000","last_undegraded":"2024-04-26T16:31:51.282690+0000","last_fullsized":"2024-04-26T16:31:51.282690+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:09.867280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.1a","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.279888+0000","last_change":"2024-04-26T16:31:51.275644+0000","last_active":"2024-04-26T16:31:51.279888+0000","last_peered":"2024-04-26T16:31:51.279888+0000","last_clean":"2024-04-26T16:31:51.279888+0000","last_became_active":"2024-04-26T16:31:51.275181+0000","last_became_peered":"2024-04-26T16:31:51.275181+0000","last_unstale":"2024-04-26T16:31:51.279888+0000","last_undegraded":"2024-04-26T16:31:51.279888+0000","last_fullsized":"2024-04-26T16:31:51.279888+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.744792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.19","version":"0'0","reported_seq":13,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825538+0000","last_change":"2024-04-26T16:31:51.300119+0000","last_active":"2024-04-26T16:31:52.825538+0000","last_peered":"2024-04-26T16:31:52.825538+0000","last_clean":"2024-04-26T16:31:52.825538+0000","last_became_active":"2024-04-26T16:31:51.299739+0000","last_became_peered":"2024-04-26T16:31:51.299739+0000","last_unstale":"2024-04-26T16:31:52.825538+0000","last_undegraded":"2024-04-26T16:31:52.825538+0000","last_fullsized":"2024-04-26T16:31:52.825538+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:15.417708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.18","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.288237+0000","last_change":"2024-04-26T16:31:51.284671+0000","last_active":"2024-04-26T16:31:51.288237+0000","last_peered":"2024-04-26T16:31:51.288237+0000","last_clean":"2024-04-26T16:31:51.288237+0000","last_became_active":"2024-04-26T16:31:51.280928+0000","last_became_peered":"2024-04-26T16:31:51.280928+0000","last_unstale":"2024-04-26T16:31:51.288237+0000","last_undegraded":"2024-04-26T16:31:51.288237+0000","last_fullsized":"2024-04-26T16:31:51.288237+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:02.879861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.17","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.288297+0000","last_change":"2024-04-26T16:31:51.278930+0000","last_active":"2024-04-26T16:31:51.288297+0000","last_peered":"2024-04-26T16:31:51.288297+0000","last_clean":"2024-04-26T16:31:51.288297+0000","last_became_active":"2024-04-26T16:31:51.274618+0000","last_became_peered":"2024-04-26T16:31:51.274618+0000","last_unstale":"2024-04-26T16:31:51.288297+0000","last_undegraded":"2024-04-26T16:31:51.288297+0000","last_fullsized":"2024-04-26T16:31:51.288297+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:08.273221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.16","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.284282+0000","last_change":"2024-04-26T16:31:51.283656+0000","last_active":"2024-04-26T16:31:51.284282+0000","last_peered":"2024-04-26T16:31:51.284282+0000","last_clean":"2024-04-26T16:31:51.284282+0000","last_became_active":"2024-04-26T16:31:51.282783+0000","last_became_peered":"2024-04-26T16:31:51.282783+0000","last_unstale":"2024-04-26T16:31:51.284282+0000","last_undegraded":"2024-04-26T16:31:51.284282+0000","last_fullsized":"2024-04-26T16:31:51.284282+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:04.410007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.15","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.280569+0000","last_change":"2024-04-26T16:31:51.278056+0000","last_active":"2024-04-26T16:31:51.280569+0000","last_peered":"2024-04-26T16:31:51.280569+0000","last_clean":"2024-04-26T16:31:51.280569+0000","last_became_active":"2024-04-26T16:31:51.277462+0000","last_became_peered":"2024-04-26T16:31:51.277462+0000","last_unstale":"2024-04-26T16:31:51.280569+0000","last_undegraded":"2024-04-26T16:31:51.280569+0000","last_fullsized":"2024-04-26T16:31:51.280569+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:53.344189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.14","version":"577'1","reported_seq":11,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.313751+0000","last_change":"2024-04-26T16:31:51.298868+0000","last_active":"2024-04-26T16:31:51.313751+0000","last_peered":"2024-04-26T16:31:51.313751+0000","last_clean":"2024-04-26T16:31:51.313751+0000","last_became_active":"2024-04-26T16:31:51.297299+0000","last_became_peered":"2024-04-26T16:31:51.297299+0000","last_unstale":"2024-04-26T16:31:51.313751+0000","last_undegraded":"2024-04-26T16:31:51.313751+0000","last_fullsized":"2024-04-26T16:31:51.313751+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:19.968473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.13","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.285042+0000","last_change":"2024-04-26T16:31:51.284597+0000","last_active":"2024-04-26T16:31:51.285042+0000","last_peered":"2024-04-26T16:31:51.285042+0000","last_clean":"2024-04-26T16:31:51.285042+0000","last_became_active":"2024-04-26T16:31:51.284379+0000","last_became_peered":"2024-04-26T16:31:51.284379+0000","last_unstale":"2024-04-26T16:31:51.285042+0000","last_undegraded":"2024-04-26T16:31:51.285042+0000","last_fullsized":"2024-04-26T16:31:51.285042+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.001093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.12","version":"577'1","reported_seq":11,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.312015+0000","last_change":"2024-04-26T16:31:51.282132+0000","last_active":"2024-04-26T16:31:51.312015+0000","last_peered":"2024-04-26T16:31:51.312015+0000","last_clean":"2024-04-26T16:31:51.312015+0000","last_became_active":"2024-04-26T16:31:51.281696+0000","last_became_peered":"2024-04-26T16:31:51.281696+0000","last_unstale":"2024-04-26T16:31:51.312015+0000","last_undegraded":"2024-04-26T16:31:51.312015+0000","last_fullsized":"2024-04-26T16:31:51.312015+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:10.440921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.11","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.293825+0000","last_change":"2024-04-26T16:31:51.292835+0000","last_active":"2024-04-26T16:31:51.293825+0000","last_peered":"2024-04-26T16:31:51.293825+0000","last_clean":"2024-04-26T16:31:51.293825+0000","last_became_active":"2024-04-26T16:31:51.279231+0000","last_became_peered":"2024-04-26T16:31:51.279231+0000","last_unstale":"2024-04-26T16:31:51.293825+0000","last_undegraded":"2024-04-26T16:31:51.293825+0000","last_fullsized":"2024-04-26T16:31:51.293825+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.803505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.10","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.283952+0000","last_change":"2024-04-26T16:31:51.282542+0000","last_active":"2024-04-26T16:31:51.283952+0000","last_peered":"2024-04-26T16:31:51.283952+0000","last_clean":"2024-04-26T16:31:51.283952+0000","last_became_active":"2024-04-26T16:31:51.282013+0000","last_became_peered":"2024-04-26T16:31:51.282013+0000","last_unstale":"2024-04-26T16:31:51.283952+0000","last_undegraded":"2024-04-26T16:31:51.283952+0000","last_fullsized":"2024-04-26T16:31:51.283952+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:58.071239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.f","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.280053+0000","last_change":"2024-04-26T16:31:51.275077+0000","last_active":"2024-04-26T16:31:51.280053+0000","last_peered":"2024-04-26T16:31:51.280053+0000","last_clean":"2024-04-26T16:31:51.280053+0000","last_became_active":"2024-04-26T16:31:51.274748+0000","last_became_peered":"2024-04-26T16:31:51.274748+0000","last_unstale":"2024-04-26T16:31:51.280053+0000","last_undegraded":"2024-04-26T16:31:51.280053+0000","last_fullsized":"2024-04-26T16:31:51.280053+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:57.675449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.e","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.290985+0000","last_change":"2024-04-26T16:31:51.285407+0000","last_active":"2024-04-26T16:31:51.290985+0000","last_peered":"2024-04-26T16:31:51.290985+0000","last_clean":"2024-04-26T16:31:51.290985+0000","last_became_active":"2024-04-26T16:31:51.284511+0000","last_became_peered":"2024-04-26T16:31:51.284511+0000","last_unstale":"2024-04-26T16:31:51.290985+0000","last_undegraded":"2024-04-26T16:31:51.290985+0000","last_fullsized":"2024-04-26T16:31:51.290985+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:58.388112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.d","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.305959+0000","last_change":"2024-04-26T16:31:51.302051+0000","last_active":"2024-04-26T16:31:51.305959+0000","last_peered":"2024-04-26T16:31:51.305959+0000","last_clean":"2024-04-26T16:31:51.305959+0000","last_became_active":"2024-04-26T16:31:51.301796+0000","last_became_peered":"2024-04-26T16:31:51.301796+0000","last_unstale":"2024-04-26T16:31:51.305959+0000","last_undegraded":"2024-04-26T16:31:51.305959+0000","last_fullsized":"2024-04-26T16:31:51.305959+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:18.279001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.c","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.305957+0000","last_change":"2024-04-26T16:31:51.301570+0000","last_active":"2024-04-26T16:31:51.305957+0000","last_peered":"2024-04-26T16:31:51.305957+0000","last_clean":"2024-04-26T16:31:51.305957+0000","last_became_active":"2024-04-26T16:31:51.298771+0000","last_became_peered":"2024-04-26T16:31:51.298771+0000","last_unstale":"2024-04-26T16:31:51.305957+0000","last_undegraded":"2024-04-26T16:31:51.305957+0000","last_fullsized":"2024-04-26T16:31:51.305957+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.276819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.b","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281005+0000","last_change":"2024-04-26T16:31:51.279734+0000","last_active":"2024-04-26T16:31:51.281005+0000","last_peered":"2024-04-26T16:31:51.281005+0000","last_clean":"2024-04-26T16:31:51.281005+0000","last_became_active":"2024-04-26T16:31:51.279361+0000","last_became_peered":"2024-04-26T16:31:51.279361+0000","last_unstale":"2024-04-26T16:31:51.281005+0000","last_undegraded":"2024-04-26T16:31:51.281005+0000","last_fullsized":"2024-04-26T16:31:51.281005+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:00.995978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.a","version":"577'1","reported_seq":11,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.311206+0000","last_change":"2024-04-26T16:31:51.288558+0000","last_active":"2024-04-26T16:31:51.311206+0000","last_peered":"2024-04-26T16:31:51.311206+0000","last_clean":"2024-04-26T16:31:51.311206+0000","last_became_active":"2024-04-26T16:31:51.287521+0000","last_became_peered":"2024-04-26T16:31:51.287521+0000","last_unstale":"2024-04-26T16:31:51.311206+0000","last_undegraded":"2024-04-26T16:31:51.311206+0000","last_fullsized":"2024-04-26T16:31:51.311206+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:06.338973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.9","version":"577'1","reported_seq":11,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.308051+0000","last_change":"2024-04-26T16:31:51.295286+0000","last_active":"2024-04-26T16:31:51.308051+0000","last_peered":"2024-04-26T16:31:51.308051+0000","last_clean":"2024-04-26T16:31:51.308051+0000","last_became_active":"2024-04-26T16:31:51.293108+0000","last_became_peered":"2024-04-26T16:31:51.293108+0000","last_unstale":"2024-04-26T16:31:51.308051+0000","last_undegraded":"2024-04-26T16:31:51.308051+0000","last_fullsized":"2024-04-26T16:31:51.308051+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.949497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.8","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.286460+0000","last_change":"2024-04-26T16:31:51.276957+0000","last_active":"2024-04-26T16:31:51.286460+0000","last_peered":"2024-04-26T16:31:51.286460+0000","last_clean":"2024-04-26T16:31:51.286460+0000","last_became_active":"2024-04-26T16:31:51.276096+0000","last_became_peered":"2024-04-26T16:31:51.276096+0000","last_unstale":"2024-04-26T16:31:51.286460+0000","last_undegraded":"2024-04-26T16:31:51.286460+0000","last_fullsized":"2024-04-26T16:31:51.286460+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:01.283511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"313.7","version":"0'0","reported_seq":13,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826788+0000","last_change":"2024-04-26T16:31:51.289836+0000","last_active":"2024-04-26T16:31:52.826788+0000","last_peered":"2024-04-26T16:31:52.826788+0000","last_clean":"2024-04-26T16:31:52.826788+0000","last_became_active":"2024-04-26T16:31:51.288701+0000","last_became_peered":"2024-04-26T16:31:51.288701+0000","last_unstale":"2024-04-26T16:31:52.826788+0000","last_undegraded":"2024-04-26T16:31:52.826788+0000","last_fullsized":"2024-04-26T16:31:52.826788+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:14.123907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.6","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281415+0000","last_change":"2024-04-26T16:31:51.280680+0000","last_active":"2024-04-26T16:31:51.281415+0000","last_peered":"2024-04-26T16:31:51.281415+0000","last_clean":"2024-04-26T16:31:51.281415+0000","last_became_active":"2024-04-26T16:31:51.280385+0000","last_became_peered":"2024-04-26T16:31:51.280385+0000","last_unstale":"2024-04-26T16:31:51.281415+0000","last_undegraded":"2024-04-26T16:31:51.281415+0000","last_fullsized":"2024-04-26T16:31:51.281415+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:54.799881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.5","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.286073+0000","last_change":"2024-04-26T16:31:51.285735+0000","last_active":"2024-04-26T16:31:51.286073+0000","last_peered":"2024-04-26T16:31:51.286073+0000","last_clean":"2024-04-26T16:31:51.286073+0000","last_became_active":"2024-04-26T16:31:51.285408+0000","last_became_peered":"2024-04-26T16:31:51.285408+0000","last_unstale":"2024-04-26T16:31:51.286073+0000","last_undegraded":"2024-04-26T16:31:51.286073+0000","last_fullsized":"2024-04-26T16:31:51.286073+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.565182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"313.4","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.287494+0000","last_change":"2024-04-26T16:31:51.287230+0000","last_active":"2024-04-26T16:31:51.287494+0000","last_peered":"2024-04-26T16:31:51.287494+0000","last_clean":"2024-04-26T16:31:51.287494+0000","last_became_active":"2024-04-26T16:31:51.286862+0000","last_became_peered":"2024-04-26T16:31:51.286862+0000","last_unstale":"2024-04-26T16:31:51.287494+0000","last_undegraded":"2024-04-26T16:31:51.287494+0000","last_fullsized":"2024-04-26T16:31:51.287494+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:11.676554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"313.3","version":"0'0","reported_seq":13,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825909+0000","last_change":"2024-04-26T16:31:51.283425+0000","last_active":"2024-04-26T16:31:52.825909+0000","last_peered":"2024-04-26T16:31:52.825909+0000","last_clean":"2024-04-26T16:31:52.825909+0000","last_became_active":"2024-04-26T16:31:51.282449+0000","last_became_peered":"2024-04-26T16:31:51.282449+0000","last_unstale":"2024-04-26T16:31:52.825909+0000","last_undegraded":"2024-04-26T16:31:52.825909+0000","last_fullsized":"2024-04-26T16:31:52.825909+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:02.991540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"313.2","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.305955+0000","last_change":"2024-04-26T16:31:51.298941+0000","last_active":"2024-04-26T16:31:51.305955+0000","last_peered":"2024-04-26T16:31:51.305955+0000","last_clean":"2024-04-26T16:31:51.305955+0000","last_became_active":"2024-04-26T16:31:51.298154+0000","last_became_peered":"2024-04-26T16:31:51.298154+0000","last_unstale":"2024-04-26T16:31:51.305955+0000","last_undegraded":"2024-04-26T16:31:51.305955+0000","last_fullsized":"2024-04-26T16:31:51.305955+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:58.842557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"313.1","version":"0'0","reported_seq":10,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.294232+0000","last_change":"2024-04-26T16:31:51.294022+0000","last_active":"2024-04-26T16:31:51.294232+0000","last_peered":"2024-04-26T16:31:51.294232+0000","last_clean":"2024-04-26T16:31:51.294232+0000","last_became_active":"2024-04-26T16:31:51.293703+0000","last_became_peered":"2024-04-26T16:31:51.293703+0000","last_unstale":"2024-04-26T16:31:51.294232+0000","last_undegraded":"2024-04-26T16:31:51.294232+0000","last_fullsized":"2024-04-26T16:31:51.294232+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:04.846825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"313.0","version":"577'1","reported_seq":11,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.309292+0000","last_change":"2024-04-26T16:31:51.279505+0000","last_active":"2024-04-26T16:31:51.309292+0000","last_peered":"2024-04-26T16:31:51.309292+0000","last_clean":"2024-04-26T16:31:51.309292+0000","last_became_active":"2024-04-26T16:31:51.278997+0000","last_became_peered":"2024-04-26T16:31:51.278997+0000","last_unstale":"2024-04-26T16:31:51.309292+0000","last_undegraded":"2024-04-26T16:31:51.309292+0000","last_fullsized":"2024-04-26T16:31:51.309292+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":576,"last_epoch_clean":577,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:50.283678+0000","last_clean_scrub_stamp":"2024-04-26T16:31:50.283678+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:18.883895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.12","version":"554'87","reported_seq":1128,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268410+0000","last_change":"2024-04-26T16:29:29.291952+0000","last_active":"2024-04-26T16:31:51.268410+0000","last_peered":"2024-04-26T16:31:51.268410+0000","last_clean":"2024-04-26T16:31:51.268410+0000","last_became_active":"2024-04-26T16:23:44.188737+0000","last_became_peered":"2024-04-26T16:23:44.188737+0000","last_unstale":"2024-04-26T16:31:51.268410+0000","last_undegraded":"2024-04-26T16:31:51.268410+0000","last_fullsized":"2024-04-26T16:31:51.268410+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:29:10.006584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.006584+0000","objects_scrubbed":6,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011941726,"stat_sum":{"num_bytes":1278,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":28,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.13","version":"545'55","reported_seq":932,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.257701+0000","last_change":"2024-04-26T16:30:24.606400+0000","last_active":"2024-04-26T16:31:51.257701+0000","last_peered":"2024-04-26T16:31:51.257701+0000","last_clean":"2024-04-26T16:31:51.257701+0000","last_became_active":"2024-04-26T16:26:23.192498+0000","last_became_peered":"2024-04-26T16:26:23.192498+0000","last_unstale":"2024-04-26T16:31:51.257701+0000","last_undegraded":"2024-04-26T16:31:51.257701+0000","last_fullsized":"2024-04-26T16:31:51.257701+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'52","last_scrub_stamp":"2024-04-26T16:30:24.606145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.606145+0000","objects_scrubbed":4,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0064602280000000002,"stat_sum":{"num_bytes":1378,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":21,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.10","version":"469'150","reported_seq":1280,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281333+0000","last_change":"2024-04-26T16:29:29.501691+0000","last_active":"2024-04-26T16:31:51.281333+0000","last_peered":"2024-04-26T16:31:51.281333+0000","last_clean":"2024-04-26T16:31:51.281333+0000","last_became_active":"2024-04-26T16:26:26.135726+0000","last_became_peered":"2024-04-26T16:26:26.135726+0000","last_unstale":"2024-04-26T16:31:51.281333+0000","last_undegraded":"2024-04-26T16:31:51.281333+0000","last_fullsized":"2024-04-26T16:31:51.281333+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'126","last_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_deep_scrub":"400'126","last_deep_scrub_stamp":"2024-04-26T16:27:49.058542+0000","last_clean_scrub_stamp":"2024-04-26T16:27:49.058542+0000","objects_scrubbed":7,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286821+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001241277,"stat_sum":{"num_bytes":103,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":0,"num_write":44,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":77,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.11","version":"535'36","reported_seq":1100,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.259872+0000","last_change":"2024-04-26T16:29:57.929311+0000","last_active":"2024-04-26T16:31:51.259872+0000","last_peered":"2024-04-26T16:31:51.259872+0000","last_clean":"2024-04-26T16:31:51.259872+0000","last_became_active":"2024-04-26T16:23:44.102085+0000","last_became_peered":"2024-04-26T16:23:44.102085+0000","last_unstale":"2024-04-26T16:31:51.259872+0000","last_undegraded":"2024-04-26T16:31:51.259872+0000","last_fullsized":"2024-04-26T16:31:51.259872+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:29:57.929076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.929076+0000","objects_scrubbed":2,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0066858430000000003,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":13,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.16","version":"574'97","reported_seq":1060,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.272498+0000","last_change":"2024-04-26T16:29:29.294557+0000","last_active":"2024-04-26T16:31:51.272498+0000","last_peered":"2024-04-26T16:31:51.272498+0000","last_clean":"2024-04-26T16:31:51.272498+0000","last_became_active":"2024-04-26T16:26:23.214752+0000","last_became_peered":"2024-04-26T16:26:23.214752+0000","last_unstale":"2024-04-26T16:31:51.272498+0000","last_undegraded":"2024-04-26T16:31:51.272498+0000","last_fullsized":"2024-04-26T16:31:51.272498+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'78","last_scrub_stamp":"2024-04-26T16:29:04.018776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.018776+0000","objects_scrubbed":7,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0060185220000000001,"stat_sum":{"num_bytes":629,"num_objects":13,"num_object_clones":3,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":12,"num_read_kb":5,"num_write":35,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.17","version":"520'52","reported_seq":1086,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.827047+0000","last_change":"2024-04-26T16:31:38.242221+0000","last_active":"2024-04-26T16:31:52.827047+0000","last_peered":"2024-04-26T16:31:52.827047+0000","last_clean":"2024-04-26T16:31:52.827047+0000","last_became_active":"2024-04-26T16:28:25.260911+0000","last_became_peered":"2024-04-26T16:28:25.260911+0000","last_unstale":"2024-04-26T16:31:52.827047+0000","last_undegraded":"2024-04-26T16:31:52.827047+0000","last_fullsized":"2024-04-26T16:31:52.827047+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'52","last_scrub_stamp":"2024-04-26T16:31:38.241965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.241965+0000","objects_scrubbed":9,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:44.615182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00057068899999999998,"stat_sum":{"num_bytes":71,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":20,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":45,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.14","version":"536'42","reported_seq":1108,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266335+0000","last_change":"2024-04-26T16:31:16.556513+0000","last_active":"2024-04-26T16:31:51.266335+0000","last_peered":"2024-04-26T16:31:51.266335+0000","last_clean":"2024-04-26T16:31:51.266335+0000","last_became_active":"2024-04-26T16:23:44.119110+0000","last_became_peered":"2024-04-26T16:23:44.119110+0000","last_unstale":"2024-04-26T16:31:51.266335+0000","last_undegraded":"2024-04-26T16:31:51.266335+0000","last_fullsized":"2024-04-26T16:31:51.266335+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'42","last_scrub_stamp":"2024-04-26T16:31:16.556378+0000","last_deep_scrub":"457'38","last_deep_scrub_stamp":"2024-04-26T16:28:49.475044+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.556378+0000","objects_scrubbed":3,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:31.199477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0034343469999999999,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":17,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.15","version":"535'82","reported_seq":1144,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278419+0000","last_change":"2024-04-26T16:30:12.508796+0000","last_active":"2024-04-26T16:31:51.278419+0000","last_peered":"2024-04-26T16:31:51.278419+0000","last_clean":"2024-04-26T16:31:51.278419+0000","last_became_active":"2024-04-26T16:23:44.233737+0000","last_became_peered":"2024-04-26T16:23:44.233737+0000","last_unstale":"2024-04-26T16:31:51.278419+0000","last_undegraded":"2024-04-26T16:31:51.278419+0000","last_fullsized":"2024-04-26T16:31:51.278419+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'78","last_scrub_stamp":"2024-04-26T16:30:12.508639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.508639+0000","objects_scrubbed":6,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582163+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0046970850000000001,"stat_sum":{"num_bytes":59,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":3,"num_write":31,"num_write_kb":28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1a","version":"322'90","reported_seq":1205,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.257429+0000","last_change":"2024-04-26T16:30:25.642345+0000","last_active":"2024-04-26T16:31:51.257429+0000","last_peered":"2024-04-26T16:31:51.257429+0000","last_clean":"2024-04-26T16:31:51.257429+0000","last_became_active":"2024-04-26T16:26:23.171828+0000","last_became_peered":"2024-04-26T16:26:23.171828+0000","last_unstale":"2024-04-26T16:31:51.257429+0000","last_undegraded":"2024-04-26T16:31:51.257429+0000","last_fullsized":"2024-04-26T16:31:51.257429+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"322'90","last_scrub_stamp":"2024-04-26T16:30:25.642102+0000","last_deep_scrub":"322'90","last_deep_scrub_stamp":"2024-04-26T16:26:30.427683+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.642102+0000","objects_scrubbed":5,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.583055+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0046318089999999998,"stat_sum":{"num_bytes":55,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":25,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":55,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1b","version":"554'32","reported_seq":1127,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.257868+0000","last_change":"2024-04-26T16:31:19.468806+0000","last_active":"2024-04-26T16:31:51.257868+0000","last_peered":"2024-04-26T16:31:51.257868+0000","last_clean":"2024-04-26T16:31:51.257868+0000","last_became_active":"2024-04-26T16:28:25.251700+0000","last_became_peered":"2024-04-26T16:28:25.251700+0000","last_unstale":"2024-04-26T16:31:51.257868+0000","last_undegraded":"2024-04-26T16:31:51.257868+0000","last_fullsized":"2024-04-26T16:31:51.257868+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'30","last_scrub_stamp":"2024-04-26T16:31:19.468562+0000","last_deep_scrub":"458'28","last_deep_scrub_stamp":"2024-04-26T16:28:51.005534+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.468562+0000","objects_scrubbed":4,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:29.641122+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0048628739999999997,"stat_sum":{"num_bytes":1251,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":19,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.18","version":"545'46","reported_seq":1150,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.261980+0000","last_change":"2024-04-26T16:31:43.779148+0000","last_active":"2024-04-26T16:31:51.261980+0000","last_peered":"2024-04-26T16:31:51.261980+0000","last_clean":"2024-04-26T16:31:51.261980+0000","last_became_active":"2024-04-26T16:26:23.188399+0000","last_became_peered":"2024-04-26T16:26:23.188399+0000","last_unstale":"2024-04-26T16:31:51.261980+0000","last_undegraded":"2024-04-26T16:31:51.261980+0000","last_fullsized":"2024-04-26T16:31:51.261980+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"545'46","last_scrub_stamp":"2024-04-26T16:31:43.779031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.779031+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.025137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038147540000000001,"stat_sum":{"num_bytes":1255,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.19","version":"554'40","reported_seq":1114,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.261529+0000","last_change":"2024-04-26T16:31:40.823736+0000","last_active":"2024-04-26T16:31:51.261529+0000","last_peered":"2024-04-26T16:31:51.261529+0000","last_clean":"2024-04-26T16:31:51.261529+0000","last_became_active":"2024-04-26T16:23:44.267057+0000","last_became_peered":"2024-04-26T16:23:44.267057+0000","last_unstale":"2024-04-26T16:31:51.261529+0000","last_undegraded":"2024-04-26T16:31:51.261529+0000","last_fullsized":"2024-04-26T16:31:51.261529+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'40","last_scrub_stamp":"2024-04-26T16:31:40.823616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:31:40.823616+0000","objects_scrubbed":4,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:42.692342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0029077199999999999,"stat_sum":{"num_bytes":597,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":19,"num_write_kb":15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"109.1e","version":"536'26","reported_seq":1104,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.279069+0000","last_change":"2024-04-26T16:30:33.603424+0000","last_active":"2024-04-26T16:31:51.279069+0000","last_peered":"2024-04-26T16:31:51.279069+0000","last_clean":"2024-04-26T16:31:51.279069+0000","last_became_active":"2024-04-26T16:28:25.230305+0000","last_became_peered":"2024-04-26T16:28:25.230305+0000","last_unstale":"2024-04-26T16:31:51.279069+0000","last_undegraded":"2024-04-26T16:31:51.279069+0000","last_fullsized":"2024-04-26T16:31:51.279069+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'20","last_scrub_stamp":"2024-04-26T16:30:33.603284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.603284+0000","objects_scrubbed":3,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.583030+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005995696,"stat_sum":{"num_bytes":27,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":14,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.1f","version":"545'28","reported_seq":1089,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.273771+0000","last_change":"2024-04-26T16:29:29.292388+0000","last_active":"2024-04-26T16:31:51.273771+0000","last_peered":"2024-04-26T16:31:51.273771+0000","last_clean":"2024-04-26T16:31:51.273771+0000","last_became_active":"2024-04-26T16:23:44.180651+0000","last_became_peered":"2024-04-26T16:23:44.180651+0000","last_unstale":"2024-04-26T16:31:51.273771+0000","last_undegraded":"2024-04-26T16:31:51.273771+0000","last_fullsized":"2024-04-26T16:31:51.273771+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'20","last_scrub_stamp":"2024-04-26T16:29:03.044079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.044079+0000","objects_scrubbed":3,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0037815560000000002,"stat_sum":{"num_bytes":19,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":19,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1c","version":"536'60","reported_seq":1105,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826999+0000","last_change":"2024-04-26T16:31:48.304922+0000","last_active":"2024-04-26T16:31:52.826999+0000","last_peered":"2024-04-26T16:31:52.826999+0000","last_clean":"2024-04-26T16:31:52.826999+0000","last_became_active":"2024-04-26T16:28:25.253825+0000","last_became_peered":"2024-04-26T16:28:25.253825+0000","last_unstale":"2024-04-26T16:31:52.826999+0000","last_undegraded":"2024-04-26T16:31:52.826999+0000","last_fullsized":"2024-04-26T16:31:52.826999+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'56","last_scrub_stamp":"2024-04-26T16:30:18.310740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.310740+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00053197300000000004,"stat_sum":{"num_bytes":38,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":23,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"109.1d","version":"563'153","reported_seq":1176,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297087+0000","last_change":"2024-04-26T16:29:29.505126+0000","last_active":"2024-04-26T16:31:51.297087+0000","last_peered":"2024-04-26T16:31:51.297087+0000","last_clean":"2024-04-26T16:31:51.297087+0000","last_became_active":"2024-04-26T16:23:44.112550+0000","last_became_peered":"2024-04-26T16:23:44.112550+0000","last_unstale":"2024-04-26T16:31:51.297087+0000","last_undegraded":"2024-04-26T16:31:51.297087+0000","last_fullsized":"2024-04-26T16:31:51.297087+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'144","last_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_deep_scrub":"401'144","last_deep_scrub_stamp":"2024-04-26T16:27:50.033127+0000","last_clean_scrub_stamp":"2024-04-26T16:27:50.033127+0000","objects_scrubbed":8,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286768+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001087244,"stat_sum":{"num_bytes":2408,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":8,"num_read_kb":2,"num_write":50,"num_write_kb":46,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.2","version":"569'117","reported_seq":1182,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.264893+0000","last_change":"2024-04-26T16:31:26.486129+0000","last_active":"2024-04-26T16:31:51.264893+0000","last_peered":"2024-04-26T16:31:51.264893+0000","last_clean":"2024-04-26T16:31:51.264893+0000","last_became_active":"2024-04-26T16:23:44.121137+0000","last_became_peered":"2024-04-26T16:23:44.121137+0000","last_unstale":"2024-04-26T16:31:51.264893+0000","last_undegraded":"2024-04-26T16:31:51.264893+0000","last_fullsized":"2024-04-26T16:31:51.264893+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'103","last_scrub_stamp":"2024-04-26T16:31:26.485773+0000","last_deep_scrub":"318'90","last_deep_scrub_stamp":"2024-04-26T16:26:26.686096+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.485773+0000","objects_scrubbed":9,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:36.433933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0030250239999999999,"stat_sum":{"num_bytes":106,"num_objects":12,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":35,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.3","version":"545'52","reported_seq":1148,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.291115+0000","last_change":"2024-04-26T16:29:29.502257+0000","last_active":"2024-04-26T16:31:51.291115+0000","last_peered":"2024-04-26T16:31:51.291115+0000","last_clean":"2024-04-26T16:31:51.291115+0000","last_became_active":"2024-04-26T16:28:25.273792+0000","last_became_peered":"2024-04-26T16:28:25.273792+0000","last_unstale":"2024-04-26T16:31:51.291115+0000","last_undegraded":"2024-04-26T16:31:51.291115+0000","last_fullsized":"2024-04-26T16:31:51.291115+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'51","last_scrub_stamp":"2024-04-26T16:27:34.948809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:34.948809+0000","objects_scrubbed":3,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000677325,"stat_sum":{"num_bytes":600,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":17,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":33,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.0","version":"545'76","reported_seq":1122,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265503+0000","last_change":"2024-04-26T16:29:56.937825+0000","last_active":"2024-04-26T16:31:51.265503+0000","last_peered":"2024-04-26T16:31:51.265503+0000","last_clean":"2024-04-26T16:31:51.265503+0000","last_became_active":"2024-04-26T16:23:44.226589+0000","last_became_peered":"2024-04-26T16:23:44.226589+0000","last_unstale":"2024-04-26T16:31:51.265503+0000","last_undegraded":"2024-04-26T16:31:51.265503+0000","last_fullsized":"2024-04-26T16:31:51.265503+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'75","last_scrub_stamp":"2024-04-26T16:29:56.937577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:29:56.937577+0000","objects_scrubbed":6,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004787085,"stat_sum":{"num_bytes":2370,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.1","version":"545'111","reported_seq":1163,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.290013+0000","last_change":"2024-04-26T16:29:29.501949+0000","last_active":"2024-04-26T16:31:51.290013+0000","last_peered":"2024-04-26T16:31:51.290013+0000","last_clean":"2024-04-26T16:31:51.290013+0000","last_became_active":"2024-04-26T16:28:25.268393+0000","last_became_peered":"2024-04-26T16:28:25.268393+0000","last_unstale":"2024-04-26T16:31:51.290013+0000","last_undegraded":"2024-04-26T16:31:51.290013+0000","last_fullsized":"2024-04-26T16:31:51.290013+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'102","last_scrub_stamp":"2024-04-26T16:27:26.872968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:26.872968+0000","objects_scrubbed":6,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00086574099999999995,"stat_sum":{"num_bytes":1289,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":42,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.6","version":"554'86","reported_seq":1152,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278952+0000","last_change":"2024-04-26T16:30:31.618966+0000","last_active":"2024-04-26T16:31:51.278952+0000","last_peered":"2024-04-26T16:31:51.278952+0000","last_clean":"2024-04-26T16:31:51.278952+0000","last_became_active":"2024-04-26T16:23:44.248476+0000","last_became_peered":"2024-04-26T16:23:44.248476+0000","last_unstale":"2024-04-26T16:31:51.278952+0000","last_undegraded":"2024-04-26T16:31:51.278952+0000","last_fullsized":"2024-04-26T16:31:51.278952+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'83","last_scrub_stamp":"2024-04-26T16:30:31.618721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.618721+0000","objects_scrubbed":8,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582982+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":2.010158063,"stat_sum":{"num_bytes":4000069,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":7,"num_read_kb":1,"num_write":30,"num_write_kb":3933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.7","version":"554'84","reported_seq":1145,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297293+0000","last_change":"2024-04-26T16:29:29.505553+0000","last_active":"2024-04-26T16:31:51.297293+0000","last_peered":"2024-04-26T16:31:51.297293+0000","last_clean":"2024-04-26T16:31:51.297293+0000","last_became_active":"2024-04-26T16:23:44.200194+0000","last_became_peered":"2024-04-26T16:23:44.200194+0000","last_unstale":"2024-04-26T16:31:51.297293+0000","last_undegraded":"2024-04-26T16:31:51.297293+0000","last_fullsized":"2024-04-26T16:31:51.297293+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'57","last_scrub_stamp":"2024-04-26T16:27:37.015805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:27:37.015805+0000","objects_scrubbed":5,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00094604400000000005,"stat_sum":{"num_bytes":1311,"num_objects":13,"num_object_clones":2,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":18,"num_read_kb":7,"num_write":39,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"109.4","version":"389'86","reported_seq":1150,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.270908+0000","last_change":"2024-04-26T16:30:10.930053+0000","last_active":"2024-04-26T16:31:51.270908+0000","last_peered":"2024-04-26T16:31:51.270908+0000","last_clean":"2024-04-26T16:31:51.270908+0000","last_became_active":"2024-04-26T16:28:25.254293+0000","last_became_peered":"2024-04-26T16:28:25.254293+0000","last_unstale":"2024-04-26T16:31:51.270908+0000","last_undegraded":"2024-04-26T16:31:51.270908+0000","last_fullsized":"2024-04-26T16:31:51.270908+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'86","last_scrub_stamp":"2024-04-26T16:30:10.929911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:10.929911+0000","objects_scrubbed":6,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0073012850000000002,"stat_sum":{"num_bytes":63,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":26,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.5","version":"545'55","reported_seq":1125,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266845+0000","last_change":"2024-04-26T16:31:15.603225+0000","last_active":"2024-04-26T16:31:51.266845+0000","last_peered":"2024-04-26T16:31:51.266845+0000","last_clean":"2024-04-26T16:31:51.266845+0000","last_became_active":"2024-04-26T16:23:44.116978+0000","last_became_peered":"2024-04-26T16:23:44.116978+0000","last_unstale":"2024-04-26T16:31:51.266845+0000","last_undegraded":"2024-04-26T16:31:51.266845+0000","last_fullsized":"2024-04-26T16:31:51.266845+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"535'54","last_scrub_stamp":"2024-04-26T16:31:15.602994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:31:15.602994+0000","objects_scrubbed":5,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:32.615263+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018006140000000001,"stat_sum":{"num_bytes":750,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":24,"num_write_kb":21,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.a","version":"577'59","reported_seq":1159,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267064+0000","last_change":"2024-04-26T16:30:31.542354+0000","last_active":"2024-04-26T16:31:51.267064+0000","last_peered":"2024-04-26T16:31:51.267064+0000","last_clean":"2024-04-26T16:31:51.267064+0000","last_became_active":"2024-04-26T16:26:27.141171+0000","last_became_peered":"2024-04-26T16:26:27.141171+0000","last_unstale":"2024-04-26T16:31:51.267064+0000","last_undegraded":"2024-04-26T16:31:51.267064+0000","last_fullsized":"2024-04-26T16:31:51.267064+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'44","last_scrub_stamp":"2024-04-26T16:30:31.542113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.542113+0000","objects_scrubbed":5,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0020446129999999998,"stat_sum":{"num_bytes":63,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":29,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.b","version":"545'37","reported_seq":1112,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267162+0000","last_change":"2024-04-26T16:30:00.932705+0000","last_active":"2024-04-26T16:31:51.267162+0000","last_peered":"2024-04-26T16:31:51.267162+0000","last_clean":"2024-04-26T16:31:51.267162+0000","last_became_active":"2024-04-26T16:28:25.233607+0000","last_became_peered":"2024-04-26T16:28:25.233607+0000","last_unstale":"2024-04-26T16:31:51.267162+0000","last_undegraded":"2024-04-26T16:31:51.267162+0000","last_fullsized":"2024-04-26T16:31:51.267162+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-04-26T16:30:00.932368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.932368+0000","objects_scrubbed":2,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049354960000000002,"stat_sum":{"num_bytes":1564,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.8","version":"554'56","reported_seq":1165,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.255984+0000","last_change":"2024-04-26T16:30:11.507964+0000","last_active":"2024-04-26T16:31:51.255984+0000","last_peered":"2024-04-26T16:31:51.255984+0000","last_clean":"2024-04-26T16:31:51.255984+0000","last_became_active":"2024-04-26T16:28:25.245886+0000","last_became_peered":"2024-04-26T16:28:25.245886+0000","last_unstale":"2024-04-26T16:31:51.255984+0000","last_undegraded":"2024-04-26T16:31:51.255984+0000","last_fullsized":"2024-04-26T16:31:51.255984+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'55","last_scrub_stamp":"2024-04-26T16:30:11.507584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.507584+0000","objects_scrubbed":5,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582781+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.001167427,"stat_sum":{"num_bytes":1591,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":5,"num_read_kb":0,"num_write":21,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":41,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"109.9","version":"224'34","reported_seq":1091,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262897+0000","last_change":"2024-04-26T16:29:29.284837+0000","last_active":"2024-04-26T16:31:51.262897+0000","last_peered":"2024-04-26T16:31:51.262897+0000","last_clean":"2024-04-26T16:31:51.262897+0000","last_became_active":"2024-04-26T16:28:25.245235+0000","last_became_peered":"2024-04-26T16:28:25.245235+0000","last_unstale":"2024-04-26T16:31:51.262897+0000","last_undegraded":"2024-04-26T16:31:51.262897+0000","last_fullsized":"2024-04-26T16:31:51.262897+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"224'34","last_scrub_stamp":"2024-04-26T16:28:54.109529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:28:54.109529+0000","objects_scrubbed":2,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0053844380000000001,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.e","version":"554'34","reported_seq":1173,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.272334+0000","last_change":"2024-04-26T16:30:00.904554+0000","last_active":"2024-04-26T16:31:51.272334+0000","last_peered":"2024-04-26T16:31:51.272334+0000","last_clean":"2024-04-26T16:31:51.272334+0000","last_became_active":"2024-04-26T16:26:23.214391+0000","last_became_peered":"2024-04-26T16:26:23.214391+0000","last_unstale":"2024-04-26T16:31:51.272334+0000","last_undegraded":"2024-04-26T16:31:51.272334+0000","last_fullsized":"2024-04-26T16:31:51.272334+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'32","last_scrub_stamp":"2024-04-26T16:30:00.904197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.904197+0000","objects_scrubbed":4,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00183567,"stat_sum":{"num_bytes":4194328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":1,"num_write":15,"num_write_kb":8204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4194328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"109.f","version":"536'84","reported_seq":1194,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.274734+0000","last_change":"2024-04-26T16:31:40.494552+0000","last_active":"2024-04-26T16:31:51.274734+0000","last_peered":"2024-04-26T16:31:51.274734+0000","last_clean":"2024-04-26T16:31:51.274734+0000","last_became_active":"2024-04-26T16:26:23.198449+0000","last_became_peered":"2024-04-26T16:26:23.198449+0000","last_unstale":"2024-04-26T16:31:51.274734+0000","last_undegraded":"2024-04-26T16:31:51.274734+0000","last_fullsized":"2024-04-26T16:31:51.274734+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"536'84","last_scrub_stamp":"2024-04-26T16:31:40.494272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:31:40.494272+0000","objects_scrubbed":7,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:09.231807+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001552419,"stat_sum":{"num_bytes":54,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":34,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":49,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"109.c","version":"389'35","reported_seq":1096,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.270738+0000","last_change":"2024-04-26T16:30:16.958876+0000","last_active":"2024-04-26T16:31:51.270738+0000","last_peered":"2024-04-26T16:31:51.270738+0000","last_clean":"2024-04-26T16:31:51.270738+0000","last_became_active":"2024-04-26T16:23:44.106600+0000","last_became_peered":"2024-04-26T16:23:44.106600+0000","last_unstale":"2024-04-26T16:31:51.270738+0000","last_undegraded":"2024-04-26T16:31:51.270738+0000","last_fullsized":"2024-04-26T16:31:51.270738+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:30:16.958732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:43.006442+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.958732+0000","objects_scrubbed":3,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0043915680000000002,"stat_sum":{"num_bytes":30,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":11,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"109.d","version":"554'27","reported_seq":890,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.272067+0000","last_change":"2024-04-26T16:29:29.289068+0000","last_active":"2024-04-26T16:31:51.272067+0000","last_peered":"2024-04-26T16:31:51.272067+0000","last_clean":"2024-04-26T16:31:51.272067+0000","last_became_active":"2024-04-26T16:28:25.242455+0000","last_became_peered":"2024-04-26T16:28:25.242455+0000","last_unstale":"2024-04-26T16:31:51.272067+0000","last_undegraded":"2024-04-26T16:31:51.272067+0000","last_fullsized":"2024-04-26T16:31:51.272067+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'22","last_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_deep_scrub":"464'22","last_deep_scrub_stamp":"2024-04-26T16:29:02.003905+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.003905+0000","objects_scrubbed":3,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930375+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00091357000000000005,"stat_sum":{"num_bytes":1364,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":15,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":554,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281476+0000","last_change":"2024-04-26T16:31:24.119396+0000","last_active":"2024-04-26T16:31:51.281476+0000","last_peered":"2024-04-26T16:31:51.281476+0000","last_clean":"2024-04-26T16:31:51.281476+0000","last_became_active":"2024-04-26T16:26:23.194196+0000","last_became_peered":"2024-04-26T16:26:23.194196+0000","last_unstale":"2024-04-26T16:31:51.281476+0000","last_undegraded":"2024-04-26T16:31:51.281476+0000","last_fullsized":"2024-04-26T16:31:51.281476+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:24.119192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:24.119192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.738696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"433'14","reported_seq":1236,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.289571+0000","last_change":"2024-04-26T16:29:10.136335+0000","last_active":"2024-04-26T16:31:51.289571+0000","last_peered":"2024-04-26T16:31:51.289571+0000","last_clean":"2024-04-26T16:31:51.289571+0000","last_became_active":"2024-04-26T16:28:25.267969+0000","last_became_peered":"2024-04-26T16:28:25.267969+0000","last_unstale":"2024-04-26T16:31:51.289571+0000","last_undegraded":"2024-04-26T16:31:51.289571+0000","last_fullsized":"2024-04-26T16:31:51.289571+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.136172+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"458'34","reported_seq":1117,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298206+0000","last_change":"2024-04-26T16:28:50.981113+0000","last_active":"2024-04-26T16:31:51.298206+0000","last_peered":"2024-04-26T16:31:51.298206+0000","last_clean":"2024-04-26T16:31:51.298206+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:31:51.298206+0000","last_undegraded":"2024-04-26T16:31:51.298206+0000","last_fullsized":"2024-04-26T16:31:51.298206+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:28:50.980879+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1059,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.279379+0000","last_change":"2024-04-26T16:30:08.452481+0000","last_active":"2024-04-26T16:31:51.279379+0000","last_peered":"2024-04-26T16:31:51.279379+0000","last_clean":"2024-04-26T16:31:51.279379+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:31:51.279379+0000","last_undegraded":"2024-04-26T16:31:51.279379+0000","last_fullsized":"2024-04-26T16:31:51.279379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.452371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.452371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":451,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825874+0000","last_change":"2024-04-26T16:31:16.230723+0000","last_active":"2024-04-26T16:31:52.825874+0000","last_peered":"2024-04-26T16:31:52.825874+0000","last_clean":"2024-04-26T16:31:52.825874+0000","last_became_active":"2024-04-26T16:28:25.238286+0000","last_became_peered":"2024-04-26T16:28:25.238286+0000","last_unstale":"2024-04-26T16:31:52.825874+0000","last_undegraded":"2024-04-26T16:31:52.825874+0000","last_fullsized":"2024-04-26T16:31:52.825874+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:16.230516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.230516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.279036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":902,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.295833+0000","last_change":"2024-04-26T16:28:25.277790+0000","last_active":"2024-04-26T16:31:51.295833+0000","last_peered":"2024-04-26T16:31:51.295833+0000","last_clean":"2024-04-26T16:31:51.295833+0000","last_became_active":"2024-04-26T16:28:25.277260+0000","last_became_peered":"2024-04-26T16:28:25.277260+0000","last_unstale":"2024-04-26T16:31:51.295833+0000","last_undegraded":"2024-04-26T16:31:51.295833+0000","last_fullsized":"2024-04-26T16:31:51.295833+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:28.909860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:27:28.909860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286417+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"388'20","reported_seq":616,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268501+0000","last_change":"2024-04-26T16:31:19.057114+0000","last_active":"2024-04-26T16:31:51.268501+0000","last_peered":"2024-04-26T16:31:51.268501+0000","last_clean":"2024-04-26T16:31:51.268501+0000","last_became_active":"2024-04-26T16:28:25.257143+0000","last_became_peered":"2024-04-26T16:28:25.257143+0000","last_unstale":"2024-04-26T16:31:51.268501+0000","last_undegraded":"2024-04-26T16:31:51.268501+0000","last_fullsized":"2024-04-26T16:31:51.268501+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:19.056987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.056987+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.250805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1061,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271269+0000","last_change":"2024-04-26T16:30:15.948290+0000","last_active":"2024-04-26T16:31:51.271269+0000","last_peered":"2024-04-26T16:31:51.271269+0000","last_clean":"2024-04-26T16:31:51.271269+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:31:51.271269+0000","last_undegraded":"2024-04-26T16:31:51.271269+0000","last_fullsized":"2024-04-26T16:31:51.271269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.948187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.948187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1059,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.257573+0000","last_change":"2024-04-26T16:31:28.375454+0000","last_active":"2024-04-26T16:31:51.257573+0000","last_peered":"2024-04-26T16:31:51.257573+0000","last_clean":"2024-04-26T16:31:51.257573+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:31:51.257573+0000","last_undegraded":"2024-04-26T16:31:51.257573+0000","last_fullsized":"2024-04-26T16:31:51.257573+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:28.375370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:28.375370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.246169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1064,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262145+0000","last_change":"2024-04-26T16:31:45.864772+0000","last_active":"2024-04-26T16:31:51.262145+0000","last_peered":"2024-04-26T16:31:51.262145+0000","last_clean":"2024-04-26T16:31:51.262145+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:31:51.262145+0000","last_undegraded":"2024-04-26T16:31:51.262145+0000","last_fullsized":"2024-04-26T16:31:51.262145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:45.864542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:31:45.864542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:46.318481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":916,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298470+0000","last_change":"2024-04-26T16:31:15.769031+0000","last_active":"2024-04-26T16:31:51.298470+0000","last_peered":"2024-04-26T16:31:51.298470+0000","last_clean":"2024-04-26T16:31:51.298470+0000","last_became_active":"2024-04-26T16:28:25.270561+0000","last_became_peered":"2024-04-26T16:28:25.270561+0000","last_unstale":"2024-04-26T16:31:51.298470+0000","last_undegraded":"2024-04-26T16:31:51.298470+0000","last_fullsized":"2024-04-26T16:31:51.298470+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:15.768934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:15.768934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:17.469191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"388'31","reported_seq":1127,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298155+0000","last_change":"2024-04-26T16:28:44.986410+0000","last_active":"2024-04-26T16:31:51.298155+0000","last_peered":"2024-04-26T16:31:51.298155+0000","last_clean":"2024-04-26T16:31:51.298155+0000","last_became_active":"2024-04-26T16:28:25.270747+0000","last_became_peered":"2024-04-26T16:28:25.270747+0000","last_unstale":"2024-04-26T16:31:51.298155+0000","last_undegraded":"2024-04-26T16:31:51.298155+0000","last_fullsized":"2024-04-26T16:31:51.298155+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'31","last_scrub_stamp":"2024-04-26T16:28:44.986040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:44.986040+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287171+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1073,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.273426+0000","last_change":"2024-04-26T16:31:23.456034+0000","last_active":"2024-04-26T16:31:51.273426+0000","last_peered":"2024-04-26T16:31:51.273426+0000","last_clean":"2024-04-26T16:31:51.273426+0000","last_became_active":"2024-04-26T16:26:23.201706+0000","last_became_peered":"2024-04-26T16:26:23.201706+0000","last_unstale":"2024-04-26T16:31:51.273426+0000","last_undegraded":"2024-04-26T16:31:51.273426+0000","last_fullsized":"2024-04-26T16:31:51.273426+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.455395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.455395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:45.158377+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1065,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.258472+0000","last_change":"2024-04-26T16:31:34.507941+0000","last_active":"2024-04-26T16:31:51.258472+0000","last_peered":"2024-04-26T16:31:51.258472+0000","last_clean":"2024-04-26T16:31:51.258472+0000","last_became_active":"2024-04-26T16:26:23.172602+0000","last_became_peered":"2024-04-26T16:26:23.172602+0000","last_unstale":"2024-04-26T16:31:51.258472+0000","last_undegraded":"2024-04-26T16:31:51.258472+0000","last_fullsized":"2024-04-26T16:31:51.258472+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:34.507829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:34.507829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:45.759886+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1037,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281828+0000","last_change":"2024-04-26T16:31:19.857894+0000","last_active":"2024-04-26T16:31:51.281828+0000","last_peered":"2024-04-26T16:31:51.281828+0000","last_clean":"2024-04-26T16:31:51.281828+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:31:51.281828+0000","last_undegraded":"2024-04-26T16:31:51.281828+0000","last_fullsized":"2024-04-26T16:31:51.281828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:19.857656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.857656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:43.917067+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":903,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.258539+0000","last_change":"2024-04-26T16:30:06.484249+0000","last_active":"2024-04-26T16:31:51.258539+0000","last_peered":"2024-04-26T16:31:51.258539+0000","last_clean":"2024-04-26T16:31:51.258539+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:31:51.258539+0000","last_undegraded":"2024-04-26T16:31:51.258539+0000","last_fullsized":"2024-04-26T16:31:51.258539+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:06.484028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:06.484028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582230+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"389'35","reported_seq":1059,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825541+0000","last_change":"2024-04-26T16:31:44.436288+0000","last_active":"2024-04-26T16:31:52.825541+0000","last_peered":"2024-04-26T16:31:52.825541+0000","last_clean":"2024-04-26T16:31:52.825541+0000","last_became_active":"2024-04-26T16:28:25.247148+0000","last_became_peered":"2024-04-26T16:28:25.247148+0000","last_unstale":"2024-04-26T16:31:52.825541+0000","last_undegraded":"2024-04-26T16:31:52.825541+0000","last_fullsized":"2024-04-26T16:31:52.825541+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:31:44.435924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:44.435924+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:59.504205+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1055,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.291510+0000","last_change":"2024-04-26T16:30:01.188873+0000","last_active":"2024-04-26T16:31:51.291510+0000","last_peered":"2024-04-26T16:31:51.291510+0000","last_clean":"2024-04-26T16:31:51.291510+0000","last_became_active":"2024-04-26T16:26:23.199164+0000","last_became_peered":"2024-04-26T16:26:23.199164+0000","last_unstale":"2024-04-26T16:31:51.291510+0000","last_undegraded":"2024-04-26T16:31:51.291510+0000","last_fullsized":"2024-04-26T16:31:51.291510+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:01.188654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:30:01.188654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1061,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271640+0000","last_change":"2024-04-26T16:30:29.051438+0000","last_active":"2024-04-26T16:31:51.271640+0000","last_peered":"2024-04-26T16:31:51.271640+0000","last_clean":"2024-04-26T16:31:51.271640+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:31:51.271640+0000","last_undegraded":"2024-04-26T16:31:51.271640+0000","last_fullsized":"2024-04-26T16:31:51.271640+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.051284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1047,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.279876+0000","last_change":"2024-04-26T16:29:59.407848+0000","last_active":"2024-04-26T16:31:51.279876+0000","last_peered":"2024-04-26T16:31:51.279876+0000","last_clean":"2024-04-26T16:31:51.279876+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:31:51.279876+0000","last_undegraded":"2024-04-26T16:31:51.279876+0000","last_fullsized":"2024-04-26T16:31:51.279876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:59.407460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:29:59.407460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582409+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":817,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825978+0000","last_change":"2024-04-26T16:31:39.300720+0000","last_active":"2024-04-26T16:31:52.825978+0000","last_peered":"2024-04-26T16:31:52.825978+0000","last_clean":"2024-04-26T16:31:52.825978+0000","last_became_active":"2024-04-26T16:28:25.265097+0000","last_became_peered":"2024-04-26T16:28:25.265097+0000","last_unstale":"2024-04-26T16:31:52.825978+0000","last_undegraded":"2024-04-26T16:31:52.825978+0000","last_fullsized":"2024-04-26T16:31:52.825978+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:39.300548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.300548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:46.004707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"388'19","reported_seq":1075,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266091+0000","last_change":"2024-04-26T16:30:34.982606+0000","last_active":"2024-04-26T16:31:51.266091+0000","last_peered":"2024-04-26T16:31:51.266091+0000","last_clean":"2024-04-26T16:31:51.266091+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:31:51.266091+0000","last_undegraded":"2024-04-26T16:31:51.266091+0000","last_fullsized":"2024-04-26T16:31:51.266091+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:34.982478+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.982478+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:59.881026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1069,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266953+0000","last_change":"2024-04-26T16:31:45.169649+0000","last_active":"2024-04-26T16:31:51.266953+0000","last_peered":"2024-04-26T16:31:51.266953+0000","last_clean":"2024-04-26T16:31:51.266953+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:31:51.266953+0000","last_undegraded":"2024-04-26T16:31:51.266953+0000","last_fullsized":"2024-04-26T16:31:51.266953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:45.169414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:45.169414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:06.848255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"316'12","reported_seq":1252,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278505+0000","last_change":"2024-04-26T16:31:13.448819+0000","last_active":"2024-04-26T16:31:51.278505+0000","last_peered":"2024-04-26T16:31:51.278505+0000","last_clean":"2024-04-26T16:31:51.278505+0000","last_became_active":"2024-04-26T16:26:23.194072+0000","last_became_peered":"2024-04-26T16:26:23.194072+0000","last_unstale":"2024-04-26T16:31:51.278505+0000","last_undegraded":"2024-04-26T16:31:51.278505+0000","last_fullsized":"2024-04-26T16:31:51.278505+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'12","last_scrub_stamp":"2024-04-26T16:31:13.448695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:13.448695+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.006144+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1056,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268966+0000","last_change":"2024-04-26T16:31:41.529119+0000","last_active":"2024-04-26T16:31:51.268966+0000","last_peered":"2024-04-26T16:31:51.268966+0000","last_clean":"2024-04-26T16:31:51.268966+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:31:51.268966+0000","last_undegraded":"2024-04-26T16:31:51.268966+0000","last_fullsized":"2024-04-26T16:31:51.268966+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:41.528869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:31:41.528869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:10.418761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":911,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.259866+0000","last_change":"2024-04-26T16:31:38.046173+0000","last_active":"2024-04-26T16:31:51.259866+0000","last_peered":"2024-04-26T16:31:51.259866+0000","last_clean":"2024-04-26T16:31:51.259866+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:31:51.259866+0000","last_undegraded":"2024-04-26T16:31:51.259866+0000","last_fullsized":"2024-04-26T16:31:51.259866+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:38.045950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.045950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:53.980362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"388'27","reported_seq":1089,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266161+0000","last_change":"2024-04-26T16:30:21.978132+0000","last_active":"2024-04-26T16:31:51.266161+0000","last_peered":"2024-04-26T16:31:51.266161+0000","last_clean":"2024-04-26T16:31:51.266161+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:31:51.266161+0000","last_undegraded":"2024-04-26T16:31:51.266161+0000","last_fullsized":"2024-04-26T16:31:51.266161+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:21.978007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.978007+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1075,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.260390+0000","last_change":"2024-04-26T16:30:35.051954+0000","last_active":"2024-04-26T16:31:51.260390+0000","last_peered":"2024-04-26T16:31:51.260390+0000","last_clean":"2024-04-26T16:31:51.260390+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:31:51.260390+0000","last_undegraded":"2024-04-26T16:31:51.260390+0000","last_fullsized":"2024-04-26T16:31:51.260390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.051759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"316'13","reported_seq":1246,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268213+0000","last_change":"2024-04-26T16:30:14.531293+0000","last_active":"2024-04-26T16:31:51.268213+0000","last_peered":"2024-04-26T16:31:51.268213+0000","last_clean":"2024-04-26T16:31:51.268213+0000","last_became_active":"2024-04-26T16:26:23.180811+0000","last_became_peered":"2024-04-26T16:26:23.180811+0000","last_unstale":"2024-04-26T16:31:51.268213+0000","last_undegraded":"2024-04-26T16:31:51.268213+0000","last_fullsized":"2024-04-26T16:31:51.268213+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:14.531205+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.531205+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":555,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.270868+0000","last_change":"2024-04-26T16:30:25.049754+0000","last_active":"2024-04-26T16:31:51.270868+0000","last_peered":"2024-04-26T16:31:51.270868+0000","last_clean":"2024-04-26T16:31:51.270868+0000","last_became_active":"2024-04-26T16:26:23.195438+0000","last_became_peered":"2024-04-26T16:26:23.195438+0000","last_unstale":"2024-04-26T16:31:51.270868+0000","last_undegraded":"2024-04-26T16:31:51.270868+0000","last_fullsized":"2024-04-26T16:31:51.270868+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.049647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.049647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":834,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825505+0000","last_change":"2024-04-26T16:31:41.361082+0000","last_active":"2024-04-26T16:31:52.825505+0000","last_peered":"2024-04-26T16:31:52.825505+0000","last_clean":"2024-04-26T16:31:52.825505+0000","last_became_active":"2024-04-26T16:28:25.244320+0000","last_became_peered":"2024-04-26T16:28:25.244320+0000","last_unstale":"2024-04-26T16:31:52.825505+0000","last_undegraded":"2024-04-26T16:31:52.825505+0000","last_fullsized":"2024-04-26T16:31:52.825505+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:31:41.360858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:41.360858+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:44.034018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"399'20","reported_seq":1090,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.264474+0000","last_change":"2024-04-26T16:30:21.500286+0000","last_active":"2024-04-26T16:31:51.264474+0000","last_peered":"2024-04-26T16:31:51.264474+0000","last_clean":"2024-04-26T16:31:51.264474+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:31:51.264474+0000","last_undegraded":"2024-04-26T16:31:51.264474+0000","last_fullsized":"2024-04-26T16:31:51.264474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'20","last_scrub_stamp":"2024-04-26T16:30:21.500166+0000","last_deep_scrub":"399'20","last_deep_scrub_stamp":"2024-04-26T16:27:47.205374+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.500166+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1057,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.282620+0000","last_change":"2024-04-26T16:28:46.987810+0000","last_active":"2024-04-26T16:31:51.282620+0000","last_peered":"2024-04-26T16:31:51.282620+0000","last_clean":"2024-04-26T16:31:51.282620+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:31:51.282620+0000","last_undegraded":"2024-04-26T16:31:51.282620+0000","last_fullsized":"2024-04-26T16:31:51.282620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:28:46.987686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.987686+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287218+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":562,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278088+0000","last_change":"2024-04-26T16:30:23.518613+0000","last_active":"2024-04-26T16:31:51.278088+0000","last_peered":"2024-04-26T16:31:51.278088+0000","last_clean":"2024-04-26T16:31:51.278088+0000","last_became_active":"2024-04-26T16:28:25.248459+0000","last_became_peered":"2024-04-26T16:28:25.248459+0000","last_unstale":"2024-04-26T16:31:51.278088+0000","last_undegraded":"2024-04-26T16:31:51.278088+0000","last_fullsized":"2024-04-26T16:31:51.278088+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.518326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.518326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1053,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268733+0000","last_change":"2024-04-26T16:30:35.938334+0000","last_active":"2024-04-26T16:31:51.268733+0000","last_peered":"2024-04-26T16:31:51.268733+0000","last_clean":"2024-04-26T16:31:51.268733+0000","last_became_active":"2024-04-26T16:28:25.257629+0000","last_became_peered":"2024-04-26T16:28:25.257629+0000","last_unstale":"2024-04-26T16:31:51.268733+0000","last_undegraded":"2024-04-26T16:31:51.268733+0000","last_fullsized":"2024-04-26T16:31:51.268733+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.938129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.938129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.931084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":901,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268509+0000","last_change":"2024-04-26T16:28:55.586673+0000","last_active":"2024-04-26T16:31:51.268509+0000","last_peered":"2024-04-26T16:31:51.268509+0000","last_clean":"2024-04-26T16:31:51.268509+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:31:51.268509+0000","last_undegraded":"2024-04-26T16:31:51.268509+0000","last_fullsized":"2024-04-26T16:31:51.268509+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:55.586453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:55.586453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1227,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266509+0000","last_change":"2024-04-26T16:31:27.023667+0000","last_active":"2024-04-26T16:31:51.266509+0000","last_peered":"2024-04-26T16:31:51.266509+0000","last_clean":"2024-04-26T16:31:51.266509+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:31:51.266509+0000","last_undegraded":"2024-04-26T16:31:51.266509+0000","last_fullsized":"2024-04-26T16:31:51.266509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:31:27.023432+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:21:06.017712+0000","last_clean_scrub_stamp":"2024-04-26T16:31:27.023432+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:56.061694+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1084,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.264675+0000","last_change":"2024-04-26T16:31:42.531951+0000","last_active":"2024-04-26T16:31:51.264675+0000","last_peered":"2024-04-26T16:31:51.264675+0000","last_clean":"2024-04-26T16:31:51.264675+0000","last_became_active":"2024-04-26T16:28:25.250134+0000","last_became_peered":"2024-04-26T16:28:25.250134+0000","last_unstale":"2024-04-26T16:31:51.264675+0000","last_undegraded":"2024-04-26T16:31:51.264675+0000","last_fullsized":"2024-04-26T16:31:51.264675+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:31:42.531687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.916214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'30","reported_seq":1106,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265851+0000","last_change":"2024-04-26T16:31:15.749111+0000","last_active":"2024-04-26T16:31:51.265851+0000","last_peered":"2024-04-26T16:31:51.265851+0000","last_clean":"2024-04-26T16:31:51.265851+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:31:51.265851+0000","last_undegraded":"2024-04-26T16:31:51.265851+0000","last_fullsized":"2024-04-26T16:31:51.265851+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'30","last_scrub_stamp":"2024-04-26T16:31:15.748874+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:31:15.748874+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.294364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":960,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826747+0000","last_change":"2024-04-26T16:30:33.268356+0000","last_active":"2024-04-26T16:31:52.826747+0000","last_peered":"2024-04-26T16:31:52.826747+0000","last_clean":"2024-04-26T16:31:52.826747+0000","last_became_active":"2024-04-26T16:28:25.265921+0000","last_became_peered":"2024-04-26T16:28:25.265921+0000","last_unstale":"2024-04-26T16:31:52.826747+0000","last_undegraded":"2024-04-26T16:31:52.826747+0000","last_fullsized":"2024-04-26T16:31:52.826747+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.268035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.268035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.106859+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1046,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.291292+0000","last_change":"2024-04-26T16:28:38.012269+0000","last_active":"2024-04-26T16:31:51.291292+0000","last_peered":"2024-04-26T16:31:51.291292+0000","last_clean":"2024-04-26T16:31:51.291292+0000","last_became_active":"2024-04-26T16:28:25.270748+0000","last_became_peered":"2024-04-26T16:28:25.270748+0000","last_unstale":"2024-04-26T16:31:51.291292+0000","last_undegraded":"2024-04-26T16:31:51.291292+0000","last_fullsized":"2024-04-26T16:31:51.291292+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:38.011966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:38.011966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":900,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298352+0000","last_change":"2024-04-26T16:30:08.292894+0000","last_active":"2024-04-26T16:31:51.298352+0000","last_peered":"2024-04-26T16:31:51.298352+0000","last_clean":"2024-04-26T16:31:51.298352+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:31:51.298352+0000","last_undegraded":"2024-04-26T16:31:51.298352+0000","last_fullsized":"2024-04-26T16:31:51.298352+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:08.292589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.292589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287663+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1052,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.264384+0000","last_change":"2024-04-26T16:29:09.485874+0000","last_active":"2024-04-26T16:31:51.264384+0000","last_peered":"2024-04-26T16:31:51.264384+0000","last_clean":"2024-04-26T16:31:51.264384+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:31:51.264384+0000","last_undegraded":"2024-04-26T16:31:51.264384+0000","last_fullsized":"2024-04-26T16:31:51.264384+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.485662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.946995+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.485662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":1895,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266321+0000","last_change":"2024-04-26T16:30:24.906077+0000","last_active":"2024-04-26T16:31:51.266321+0000","last_peered":"2024-04-26T16:31:51.266321+0000","last_clean":"2024-04-26T16:31:51.266321+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:31:51.266321+0000","last_undegraded":"2024-04-26T16:31:51.266321+0000","last_fullsized":"2024-04-26T16:31:51.266321+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:30:24.905850+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.905850+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930953+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":840,"num_read_kb":835,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'22","reported_seq":1084,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.274193+0000","last_change":"2024-04-26T16:30:19.960354+0000","last_active":"2024-04-26T16:31:51.274193+0000","last_peered":"2024-04-26T16:31:51.274193+0000","last_clean":"2024-04-26T16:31:51.274193+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:31:51.274193+0000","last_undegraded":"2024-04-26T16:31:51.274193+0000","last_fullsized":"2024-04-26T16:31:51.274193+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:30:19.960123+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.960123+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1080,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.279978+0000","last_change":"2024-04-26T16:31:40.035997+0000","last_active":"2024-04-26T16:31:51.279978+0000","last_peered":"2024-04-26T16:31:51.279978+0000","last_clean":"2024-04-26T16:31:51.279978+0000","last_became_active":"2024-04-26T16:28:25.233774+0000","last_became_peered":"2024-04-26T16:28:25.233774+0000","last_unstale":"2024-04-26T16:31:51.279978+0000","last_undegraded":"2024-04-26T16:31:51.279978+0000","last_fullsized":"2024-04-26T16:31:51.279978+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:31:40.035882+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:31:40.035882+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.386853+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1048,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266259+0000","last_change":"2024-04-26T16:28:40.455517+0000","last_active":"2024-04-26T16:31:51.266259+0000","last_peered":"2024-04-26T16:31:51.266259+0000","last_clean":"2024-04-26T16:31:51.266259+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:31:51.266259+0000","last_undegraded":"2024-04-26T16:31:51.266259+0000","last_fullsized":"2024-04-26T16:31:51.266259+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:40.455422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:28:40.455422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":902,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271141+0000","last_change":"2024-04-26T16:30:12.954476+0000","last_active":"2024-04-26T16:31:51.271141+0000","last_peered":"2024-04-26T16:31:51.271141+0000","last_clean":"2024-04-26T16:31:51.271141+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:31:51.271141+0000","last_undegraded":"2024-04-26T16:31:51.271141+0000","last_fullsized":"2024-04-26T16:31:51.271141+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.954265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.954265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":952,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825733+0000","last_change":"2024-04-26T16:30:25.328753+0000","last_active":"2024-04-26T16:31:52.825733+0000","last_peered":"2024-04-26T16:31:52.825733+0000","last_clean":"2024-04-26T16:31:52.825733+0000","last_became_active":"2024-04-26T16:28:25.234888+0000","last_became_peered":"2024-04-26T16:28:25.234888+0000","last_unstale":"2024-04-26T16:31:52.825733+0000","last_undegraded":"2024-04-26T16:31:52.825733+0000","last_fullsized":"2024-04-26T16:31:52.825733+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:25.328546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:30:25.328546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:53.320476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"388'53","reported_seq":1344,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271240+0000","last_change":"2024-04-26T16:31:41.056282+0000","last_active":"2024-04-26T16:31:51.271240+0000","last_peered":"2024-04-26T16:31:51.271240+0000","last_clean":"2024-04-26T16:31:51.271240+0000","last_became_active":"2024-04-26T16:26:23.203408+0000","last_became_peered":"2024-04-26T16:26:23.203408+0000","last_unstale":"2024-04-26T16:31:51.271240+0000","last_undegraded":"2024-04-26T16:31:51.271240+0000","last_fullsized":"2024-04-26T16:31:51.271240+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'53","last_scrub_stamp":"2024-04-26T16:31:41.056164+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:31:41.056164+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:03.174643+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1058,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297844+0000","last_change":"2024-04-26T16:29:03.080654+0000","last_active":"2024-04-26T16:31:51.297844+0000","last_peered":"2024-04-26T16:31:51.297844+0000","last_clean":"2024-04-26T16:31:51.297844+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:31:51.297844+0000","last_undegraded":"2024-04-26T16:31:51.297844+0000","last_fullsized":"2024-04-26T16:31:51.297844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:29:03.080403+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1066,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.260493+0000","last_change":"2024-04-26T16:30:28.037331+0000","last_active":"2024-04-26T16:31:51.260493+0000","last_peered":"2024-04-26T16:31:51.260493+0000","last_clean":"2024-04-26T16:31:51.260493+0000","last_became_active":"2024-04-26T16:28:25.242325+0000","last_became_peered":"2024-04-26T16:28:25.242325+0000","last_unstale":"2024-04-26T16:31:51.260493+0000","last_undegraded":"2024-04-26T16:31:51.260493+0000","last_fullsized":"2024-04-26T16:31:51.260493+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.037115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.037115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:56.639017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":902,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.263428+0000","last_change":"2024-04-26T16:29:58.933284+0000","last_active":"2024-04-26T16:31:51.263428+0000","last_peered":"2024-04-26T16:31:51.263428+0000","last_clean":"2024-04-26T16:31:51.263428+0000","last_became_active":"2024-04-26T16:28:25.245871+0000","last_became_peered":"2024-04-26T16:28:25.245871+0000","last_unstale":"2024-04-26T16:31:51.263428+0000","last_undegraded":"2024-04-26T16:31:51.263428+0000","last_fullsized":"2024-04-26T16:31:51.263428+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.933163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.933163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930799+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":554,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.281248+0000","last_change":"2024-04-26T16:30:24.025255+0000","last_active":"2024-04-26T16:31:51.281248+0000","last_peered":"2024-04-26T16:31:51.281248+0000","last_clean":"2024-04-26T16:31:51.281248+0000","last_became_active":"2024-04-26T16:26:23.191179+0000","last_became_peered":"2024-04-26T16:26:23.191179+0000","last_unstale":"2024-04-26T16:31:51.281248+0000","last_undegraded":"2024-04-26T16:31:51.281248+0000","last_fullsized":"2024-04-26T16:31:51.281248+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:24.025163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:30:24.025163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1050,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.282084+0000","last_change":"2024-04-26T16:28:59.045623+0000","last_active":"2024-04-26T16:31:51.282084+0000","last_peered":"2024-04-26T16:31:51.282084+0000","last_clean":"2024-04-26T16:31:51.282084+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:31:51.282084+0000","last_undegraded":"2024-04-26T16:31:51.282084+0000","last_fullsized":"2024-04-26T16:31:51.282084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:59.045415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.045415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'35","reported_seq":1113,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.261874+0000","last_change":"2024-04-26T16:31:30.777450+0000","last_active":"2024-04-26T16:31:51.261874+0000","last_peered":"2024-04-26T16:31:51.261874+0000","last_clean":"2024-04-26T16:31:51.261874+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:31:51.261874+0000","last_undegraded":"2024-04-26T16:31:51.261874+0000","last_fullsized":"2024-04-26T16:31:51.261874+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:31:30.777298+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.777298+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:43.117201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1064,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271776+0000","last_change":"2024-04-26T16:31:18.097724+0000","last_active":"2024-04-26T16:31:51.271776+0000","last_peered":"2024-04-26T16:31:51.271776+0000","last_clean":"2024-04-26T16:31:51.271776+0000","last_became_active":"2024-04-26T16:28:25.253094+0000","last_became_peered":"2024-04-26T16:28:25.253094+0000","last_unstale":"2024-04-26T16:31:51.271776+0000","last_undegraded":"2024-04-26T16:31:51.271776+0000","last_fullsized":"2024-04-26T16:31:51.271776+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:18.097517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:31:18.097517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.069333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":908,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.269034+0000","last_change":"2024-04-26T16:28:58.073423+0000","last_active":"2024-04-26T16:31:51.269034+0000","last_peered":"2024-04-26T16:31:51.269034+0000","last_clean":"2024-04-26T16:31:51.269034+0000","last_became_active":"2024-04-26T16:28:25.231951+0000","last_became_peered":"2024-04-26T16:28:25.231951+0000","last_unstale":"2024-04-26T16:31:51.269034+0000","last_undegraded":"2024-04-26T16:31:51.269034+0000","last_fullsized":"2024-04-26T16:31:51.269034+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:58.073194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:58.073194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1047,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.296598+0000","last_change":"2024-04-26T16:30:05.254074+0000","last_active":"2024-04-26T16:31:51.296598+0000","last_peered":"2024-04-26T16:31:51.296598+0000","last_clean":"2024-04-26T16:31:51.296598+0000","last_became_active":"2024-04-26T16:26:23.176892+0000","last_became_peered":"2024-04-26T16:26:23.176892+0000","last_unstale":"2024-04-26T16:31:51.296598+0000","last_undegraded":"2024-04-26T16:31:51.296598+0000","last_fullsized":"2024-04-26T16:31:51.296598+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:05.253814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:30:05.253814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1346,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.282339+0000","last_change":"2024-04-26T16:31:16.790464+0000","last_active":"2024-04-26T16:31:51.282339+0000","last_peered":"2024-04-26T16:31:51.282339+0000","last_clean":"2024-04-26T16:31:51.282339+0000","last_became_active":"2024-04-26T16:28:25.267305+0000","last_became_peered":"2024-04-26T16:28:25.267305+0000","last_unstale":"2024-04-26T16:31:51.282339+0000","last_undegraded":"2024-04-26T16:31:51.282339+0000","last_fullsized":"2024-04-26T16:31:51.282339+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:31:16.790149+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.790149+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:35.256369+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":547,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.258248+0000","last_change":"2024-04-26T16:30:09.475518+0000","last_active":"2024-04-26T16:31:51.258248+0000","last_peered":"2024-04-26T16:31:51.258248+0000","last_clean":"2024-04-26T16:31:51.258248+0000","last_became_active":"2024-04-26T16:26:23.201156+0000","last_became_peered":"2024-04-26T16:26:23.201156+0000","last_unstale":"2024-04-26T16:31:51.258248+0000","last_undegraded":"2024-04-26T16:31:51.258248+0000","last_fullsized":"2024-04-26T16:31:51.258248+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.475272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.475272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"388'25","reported_seq":1108,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.269187+0000","last_change":"2024-04-26T16:28:52.098358+0000","last_active":"2024-04-26T16:31:51.269187+0000","last_peered":"2024-04-26T16:31:51.269187+0000","last_clean":"2024-04-26T16:31:51.269187+0000","last_became_active":"2024-04-26T16:28:25.258417+0000","last_became_peered":"2024-04-26T16:28:25.258417+0000","last_unstale":"2024-04-26T16:31:51.269187+0000","last_undegraded":"2024-04-26T16:31:51.269187+0000","last_fullsized":"2024-04-26T16:31:51.269187+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:28:52.098251+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.098251+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":547,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.274542+0000","last_change":"2024-04-26T16:30:14.930711+0000","last_active":"2024-04-26T16:31:51.274542+0000","last_peered":"2024-04-26T16:31:51.274542+0000","last_clean":"2024-04-26T16:31:51.274542+0000","last_became_active":"2024-04-26T16:26:23.212670+0000","last_became_peered":"2024-04-26T16:26:23.212670+0000","last_unstale":"2024-04-26T16:31:51.274542+0000","last_undegraded":"2024-04-26T16:31:51.274542+0000","last_fullsized":"2024-04-26T16:31:51.274542+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.930578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.930578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":916,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265671+0000","last_change":"2024-04-26T16:31:34.090811+0000","last_active":"2024-04-26T16:31:51.265671+0000","last_peered":"2024-04-26T16:31:51.265671+0000","last_clean":"2024-04-26T16:31:51.265671+0000","last_became_active":"2024-04-26T16:28:25.245460+0000","last_became_peered":"2024-04-26T16:28:25.245460+0000","last_unstale":"2024-04-26T16:31:51.265671+0000","last_undegraded":"2024-04-26T16:31:51.265671+0000","last_fullsized":"2024-04-26T16:31:51.265671+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:34.090693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:34.090693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.112015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1065,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.279688+0000","last_change":"2024-04-26T16:31:26.379451+0000","last_active":"2024-04-26T16:31:51.279688+0000","last_peered":"2024-04-26T16:31:51.279688+0000","last_clean":"2024-04-26T16:31:51.279688+0000","last_became_active":"2024-04-26T16:26:23.185332+0000","last_became_peered":"2024-04-26T16:26:23.185332+0000","last_unstale":"2024-04-26T16:31:51.279688+0000","last_undegraded":"2024-04-26T16:31:51.279688+0000","last_fullsized":"2024-04-26T16:31:51.279688+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.379353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.379353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:33.121445+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1067,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.256597+0000","last_change":"2024-04-26T16:31:33.497555+0000","last_active":"2024-04-26T16:31:51.256597+0000","last_peered":"2024-04-26T16:31:51.256597+0000","last_clean":"2024-04-26T16:31:51.256597+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:31:51.256597+0000","last_undegraded":"2024-04-26T16:31:51.256597+0000","last_fullsized":"2024-04-26T16:31:51.256597+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:33.497400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:31:33.497400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:44.694409+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'38","reported_seq":1133,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.259701+0000","last_change":"2024-04-26T16:30:27.085082+0000","last_active":"2024-04-26T16:31:51.259701+0000","last_peered":"2024-04-26T16:31:51.259701+0000","last_clean":"2024-04-26T16:31:51.259701+0000","last_became_active":"2024-04-26T16:28:25.241767+0000","last_became_peered":"2024-04-26T16:28:25.241767+0000","last_unstale":"2024-04-26T16:31:51.259701+0000","last_undegraded":"2024-04-26T16:31:51.259701+0000","last_fullsized":"2024-04-26T16:31:51.259701+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-04-26T16:30:27.084844+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:27:38.188279+0000","last_clean_scrub_stamp":"2024-04-26T16:30:27.084844+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1046,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297558+0000","last_change":"2024-04-26T16:28:52.021759+0000","last_active":"2024-04-26T16:31:51.297558+0000","last_peered":"2024-04-26T16:31:51.297558+0000","last_clean":"2024-04-26T16:31:51.297558+0000","last_became_active":"2024-04-26T16:28:25.266762+0000","last_became_peered":"2024-04-26T16:28:25.266762+0000","last_unstale":"2024-04-26T16:31:51.297558+0000","last_undegraded":"2024-04-26T16:31:51.297558+0000","last_fullsized":"2024-04-26T16:31:51.297558+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.021548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.021548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":893,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265701+0000","last_change":"2024-04-26T16:29:10.997703+0000","last_active":"2024-04-26T16:31:51.265701+0000","last_peered":"2024-04-26T16:31:51.265701+0000","last_clean":"2024-04-26T16:31:51.265701+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:31:51.265701+0000","last_undegraded":"2024-04-26T16:31:51.265701+0000","last_fullsized":"2024-04-26T16:31:51.265701+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:10.997614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.997614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1060,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265307+0000","last_change":"2024-04-26T16:31:22.830960+0000","last_active":"2024-04-26T16:31:51.265307+0000","last_peered":"2024-04-26T16:31:51.265307+0000","last_clean":"2024-04-26T16:31:51.265307+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:31:51.265307+0000","last_undegraded":"2024-04-26T16:31:51.265307+0000","last_fullsized":"2024-04-26T16:31:51.265307+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:22.830853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:22.830853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.646479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'37","reported_seq":1132,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266449+0000","last_change":"2024-04-26T16:31:38.498977+0000","last_active":"2024-04-26T16:31:51.266449+0000","last_peered":"2024-04-26T16:31:51.266449+0000","last_clean":"2024-04-26T16:31:51.266449+0000","last_became_active":"2024-04-26T16:28:25.235694+0000","last_became_peered":"2024-04-26T16:28:25.235694+0000","last_unstale":"2024-04-26T16:31:51.266449+0000","last_undegraded":"2024-04-26T16:31:51.266449+0000","last_fullsized":"2024-04-26T16:31:51.266449+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.498802+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.853889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":563,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266828+0000","last_change":"2024-04-26T16:31:23.096196+0000","last_active":"2024-04-26T16:31:51.266828+0000","last_peered":"2024-04-26T16:31:51.266828+0000","last_clean":"2024-04-26T16:31:51.266828+0000","last_became_active":"2024-04-26T16:28:25.233622+0000","last_became_peered":"2024-04-26T16:28:25.233622+0000","last_unstale":"2024-04-26T16:31:51.266828+0000","last_undegraded":"2024-04-26T16:31:51.266828+0000","last_fullsized":"2024-04-26T16:31:51.266828+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.095870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.095870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:40.734359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1046,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.273187+0000","last_change":"2024-04-26T16:31:38.047566+0000","last_active":"2024-04-26T16:31:51.273187+0000","last_peered":"2024-04-26T16:31:51.273187+0000","last_clean":"2024-04-26T16:31:51.273187+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:31:51.273187+0000","last_undegraded":"2024-04-26T16:31:51.273187+0000","last_fullsized":"2024-04-26T16:31:51.273187+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:38.047301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.047301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:48.441945+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":911,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278613+0000","last_change":"2024-04-26T16:30:19.533168+0000","last_active":"2024-04-26T16:31:51.278613+0000","last_peered":"2024-04-26T16:31:51.278613+0000","last_clean":"2024-04-26T16:31:51.278613+0000","last_became_active":"2024-04-26T16:28:25.241015+0000","last_became_peered":"2024-04-26T16:28:25.241015+0000","last_unstale":"2024-04-26T16:31:51.278613+0000","last_undegraded":"2024-04-26T16:31:51.278613+0000","last_fullsized":"2024-04-26T16:31:51.278613+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.533042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.533042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.583008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"433'16","reported_seq":1269,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265362+0000","last_change":"2024-04-26T16:31:16.716656+0000","last_active":"2024-04-26T16:31:51.265362+0000","last_peered":"2024-04-26T16:31:51.265362+0000","last_clean":"2024-04-26T16:31:51.265362+0000","last_became_active":"2024-04-26T16:28:25.228833+0000","last_became_peered":"2024-04-26T16:28:25.228833+0000","last_unstale":"2024-04-26T16:31:51.265362+0000","last_undegraded":"2024-04-26T16:31:51.265362+0000","last_fullsized":"2024-04-26T16:31:51.265362+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'16","last_scrub_stamp":"2024-04-26T16:31:16.715748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.715748+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:31.298504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1082,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267262+0000","last_change":"2024-04-26T16:31:47.142353+0000","last_active":"2024-04-26T16:31:51.267262+0000","last_peered":"2024-04-26T16:31:51.267262+0000","last_clean":"2024-04-26T16:31:51.267262+0000","last_became_active":"2024-04-26T16:26:23.187980+0000","last_became_peered":"2024-04-26T16:26:23.187980+0000","last_unstale":"2024-04-26T16:31:51.267262+0000","last_undegraded":"2024-04-26T16:31:51.267262+0000","last_fullsized":"2024-04-26T16:31:51.267262+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:31:47.142204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:31:47.142204+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.227249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"433'39","reported_seq":710,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.272317+0000","last_change":"2024-04-26T16:30:08.947640+0000","last_active":"2024-04-26T16:31:51.272317+0000","last_peered":"2024-04-26T16:31:51.272317+0000","last_clean":"2024-04-26T16:31:51.272317+0000","last_became_active":"2024-04-26T16:28:25.228181+0000","last_became_peered":"2024-04-26T16:28:25.228181+0000","last_unstale":"2024-04-26T16:31:51.272317+0000","last_undegraded":"2024-04-26T16:31:51.272317+0000","last_fullsized":"2024-04-26T16:31:51.272317+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'39","last_scrub_stamp":"2024-04-26T16:30:08.947465+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:30:08.947465+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1069,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271266+0000","last_change":"2024-04-26T16:31:26.127314+0000","last_active":"2024-04-26T16:31:51.271266+0000","last_peered":"2024-04-26T16:31:51.271266+0000","last_clean":"2024-04-26T16:31:51.271266+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:31:51.271266+0000","last_undegraded":"2024-04-26T16:31:51.271266+0000","last_fullsized":"2024-04-26T16:31:51.271266+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.127067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.127067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.976272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1049,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262593+0000","last_change":"2024-04-26T16:30:31.628236+0000","last_active":"2024-04-26T16:31:51.262593+0000","last_peered":"2024-04-26T16:31:51.262593+0000","last_clean":"2024-04-26T16:31:51.262593+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:31:51.262593+0000","last_undegraded":"2024-04-26T16:31:51.262593+0000","last_fullsized":"2024-04-26T16:31:51.262593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:31.627931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:31.627931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.829730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":911,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265651+0000","last_change":"2024-04-26T16:31:36.807486+0000","last_active":"2024-04-26T16:31:51.265651+0000","last_peered":"2024-04-26T16:31:51.265651+0000","last_clean":"2024-04-26T16:31:51.265651+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:31:51.265651+0000","last_undegraded":"2024-04-26T16:31:51.265651+0000","last_fullsized":"2024-04-26T16:31:51.265651+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:36.807347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:36.807347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:01.200386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1059,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.256559+0000","last_change":"2024-04-26T16:31:14.421824+0000","last_active":"2024-04-26T16:31:51.256559+0000","last_peered":"2024-04-26T16:31:51.256559+0000","last_clean":"2024-04-26T16:31:51.256559+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:31:51.256559+0000","last_undegraded":"2024-04-26T16:31:51.256559+0000","last_fullsized":"2024-04-26T16:31:51.256559+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:14.421630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:31:14.421630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:17.025811+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"388'19","reported_seq":1084,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297548+0000","last_change":"2024-04-26T16:30:03.191223+0000","last_active":"2024-04-26T16:31:51.297548+0000","last_peered":"2024-04-26T16:31:51.297548+0000","last_clean":"2024-04-26T16:31:51.297548+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:31:51.297548+0000","last_undegraded":"2024-04-26T16:31:51.297548+0000","last_fullsized":"2024-04-26T16:31:51.297548+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:30:03.190975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:03.190975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287720+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1084,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.275175+0000","last_change":"2024-04-26T16:30:04.581495+0000","last_active":"2024-04-26T16:31:51.275175+0000","last_peered":"2024-04-26T16:31:51.275175+0000","last_clean":"2024-04-26T16:31:51.275175+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:31:51.275175+0000","last_undegraded":"2024-04-26T16:31:51.275175+0000","last_fullsized":"2024-04-26T16:31:51.275175+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:30:04.581361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.581361+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1038,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266296+0000","last_change":"2024-04-26T16:31:28.069974+0000","last_active":"2024-04-26T16:31:51.266296+0000","last_peered":"2024-04-26T16:31:51.266296+0000","last_clean":"2024-04-26T16:31:51.266296+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:31:51.266296+0000","last_undegraded":"2024-04-26T16:31:51.266296+0000","last_fullsized":"2024-04-26T16:31:51.266296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:28.069887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:28.069887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:48.574721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":911,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.261645+0000","last_change":"2024-04-26T16:31:38.763329+0000","last_active":"2024-04-26T16:31:51.261645+0000","last_peered":"2024-04-26T16:31:51.261645+0000","last_clean":"2024-04-26T16:31:51.261645+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:31:51.261645+0000","last_undegraded":"2024-04-26T16:31:51.261645+0000","last_fullsized":"2024-04-26T16:31:51.261645+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:38.763183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.763183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.422583+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"388'19","reported_seq":1085,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.289202+0000","last_change":"2024-04-26T16:29:00.045697+0000","last_active":"2024-04-26T16:31:51.289202+0000","last_peered":"2024-04-26T16:31:51.289202+0000","last_clean":"2024-04-26T16:31:51.289202+0000","last_became_active":"2024-04-26T16:28:25.270038+0000","last_became_peered":"2024-04-26T16:28:25.270038+0000","last_unstale":"2024-04-26T16:31:51.289202+0000","last_undegraded":"2024-04-26T16:31:51.289202+0000","last_fullsized":"2024-04-26T16:31:51.289202+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:00.045450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.045450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287124+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1067,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.256399+0000","last_change":"2024-04-26T16:31:25.411760+0000","last_active":"2024-04-26T16:31:51.256399+0000","last_peered":"2024-04-26T16:31:51.256399+0000","last_clean":"2024-04-26T16:31:51.256399+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:31:51.256399+0000","last_undegraded":"2024-04-26T16:31:51.256399+0000","last_fullsized":"2024-04-26T16:31:51.256399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:25.411479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:31:25.411479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:53.171008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1041,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.274683+0000","last_change":"2024-04-26T16:30:28.910693+0000","last_active":"2024-04-26T16:31:51.274683+0000","last_peered":"2024-04-26T16:31:51.274683+0000","last_clean":"2024-04-26T16:31:51.274683+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:31:51.274683+0000","last_undegraded":"2024-04-26T16:31:51.274683+0000","last_fullsized":"2024-04-26T16:31:51.274683+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.910474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.931186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1047,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.277145+0000","last_change":"2024-04-26T16:30:07.455469+0000","last_active":"2024-04-26T16:31:51.277145+0000","last_peered":"2024-04-26T16:31:51.277145+0000","last_clean":"2024-04-26T16:31:51.277145+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:31:51.277145+0000","last_undegraded":"2024-04-26T16:31:51.277145+0000","last_fullsized":"2024-04-26T16:31:51.277145+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.455245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.455245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582353+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":911,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.255972+0000","last_change":"2024-04-26T16:30:02.403961+0000","last_active":"2024-04-26T16:31:51.255972+0000","last_peered":"2024-04-26T16:31:51.255972+0000","last_clean":"2024-04-26T16:31:51.255972+0000","last_became_active":"2024-04-26T16:28:25.246039+0000","last_became_peered":"2024-04-26T16:28:25.246039+0000","last_unstale":"2024-04-26T16:31:51.255972+0000","last_undegraded":"2024-04-26T16:31:51.255972+0000","last_fullsized":"2024-04-26T16:31:51.255972+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:02.403733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:02.403733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"388'8","reported_seq":1021,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825523+0000","last_change":"2024-04-26T16:31:45.421620+0000","last_active":"2024-04-26T16:31:52.825523+0000","last_peered":"2024-04-26T16:31:52.825523+0000","last_clean":"2024-04-26T16:31:52.825523+0000","last_became_active":"2024-04-26T16:28:25.260556+0000","last_became_peered":"2024-04-26T16:28:25.260556+0000","last_unstale":"2024-04-26T16:31:52.825523+0000","last_undegraded":"2024-04-26T16:31:52.825523+0000","last_fullsized":"2024-04-26T16:31:52.825523+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'8","last_scrub_stamp":"2024-04-26T16:31:45.421499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:45.421499+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.896590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":942,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825830+0000","last_change":"2024-04-26T16:30:28.356139+0000","last_active":"2024-04-26T16:31:52.825830+0000","last_peered":"2024-04-26T16:31:52.825830+0000","last_clean":"2024-04-26T16:31:52.825830+0000","last_became_active":"2024-04-26T16:28:25.242243+0000","last_became_peered":"2024-04-26T16:28:25.242243+0000","last_unstale":"2024-04-26T16:31:52.825830+0000","last_undegraded":"2024-04-26T16:31:52.825830+0000","last_fullsized":"2024-04-26T16:31:52.825830+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.355929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:58.034883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":547,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.272659+0000","last_change":"2024-04-26T16:28:47.096936+0000","last_active":"2024-04-26T16:31:51.272659+0000","last_peered":"2024-04-26T16:31:51.272659+0000","last_clean":"2024-04-26T16:31:51.272659+0000","last_became_active":"2024-04-26T16:28:25.243371+0000","last_became_peered":"2024-04-26T16:28:25.243371+0000","last_unstale":"2024-04-26T16:31:51.272659+0000","last_undegraded":"2024-04-26T16:31:51.272659+0000","last_fullsized":"2024-04-26T16:31:51.272659+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:47.096851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:47.096851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1045,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297349+0000","last_change":"2024-04-26T16:31:30.020041+0000","last_active":"2024-04-26T16:31:51.297349+0000","last_peered":"2024-04-26T16:31:51.297349+0000","last_clean":"2024-04-26T16:31:51.297349+0000","last_became_active":"2024-04-26T16:28:25.266217+0000","last_became_peered":"2024-04-26T16:28:25.266217+0000","last_unstale":"2024-04-26T16:31:51.297349+0000","last_undegraded":"2024-04-26T16:31:51.297349+0000","last_fullsized":"2024-04-26T16:31:51.297349+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.019680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.019680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.472819+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":909,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265845+0000","last_change":"2024-04-26T16:29:58.659839+0000","last_active":"2024-04-26T16:31:51.265845+0000","last_peered":"2024-04-26T16:31:51.265845+0000","last_clean":"2024-04-26T16:31:51.265845+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:31:51.265845+0000","last_undegraded":"2024-04-26T16:31:51.265845+0000","last_fullsized":"2024-04-26T16:31:51.265845+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.659641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.659641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'19","reported_seq":1094,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.291525+0000","last_change":"2024-04-26T16:27:29.963129+0000","last_active":"2024-04-26T16:31:51.291525+0000","last_peered":"2024-04-26T16:31:51.291525+0000","last_clean":"2024-04-26T16:31:51.291525+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:31:51.291525+0000","last_undegraded":"2024-04-26T16:31:51.291525+0000","last_fullsized":"2024-04-26T16:31:51.291525+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:27:29.962882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:29.962882+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286668+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":546,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268702+0000","last_change":"2024-04-26T16:30:07.968532+0000","last_active":"2024-04-26T16:31:51.268702+0000","last_peered":"2024-04-26T16:31:51.268702+0000","last_clean":"2024-04-26T16:31:51.268702+0000","last_became_active":"2024-04-26T16:26:23.216238+0000","last_became_peered":"2024-04-26T16:26:23.216238+0000","last_unstale":"2024-04-26T16:31:51.268702+0000","last_undegraded":"2024-04-26T16:31:51.268702+0000","last_fullsized":"2024-04-26T16:31:51.268702+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:07.968404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:07.968404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"316'13","reported_seq":697,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271050+0000","last_change":"2024-04-26T16:30:26.031295+0000","last_active":"2024-04-26T16:31:51.271050+0000","last_peered":"2024-04-26T16:31:51.271050+0000","last_clean":"2024-04-26T16:31:51.271050+0000","last_became_active":"2024-04-26T16:26:23.215841+0000","last_became_peered":"2024-04-26T16:26:23.215841+0000","last_unstale":"2024-04-26T16:31:51.271050+0000","last_undegraded":"2024-04-26T16:31:51.271050+0000","last_fullsized":"2024-04-26T16:31:51.271050+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'13","last_scrub_stamp":"2024-04-26T16:30:26.031142+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.031142+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1055,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.269564+0000","last_change":"2024-04-26T16:31:44.833272+0000","last_active":"2024-04-26T16:31:51.269564+0000","last_peered":"2024-04-26T16:31:51.269564+0000","last_clean":"2024-04-26T16:31:51.269564+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:31:51.269564+0000","last_undegraded":"2024-04-26T16:31:51.269564+0000","last_fullsized":"2024-04-26T16:31:51.269564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:44.832946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:44.832946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:11.745386+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":547,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.269448+0000","last_change":"2024-04-26T16:28:46.104373+0000","last_active":"2024-04-26T16:31:51.269448+0000","last_peered":"2024-04-26T16:31:51.269448+0000","last_clean":"2024-04-26T16:31:51.269448+0000","last_became_active":"2024-04-26T16:28:25.241392+0000","last_became_peered":"2024-04-26T16:28:25.241392+0000","last_unstale":"2024-04-26T16:31:51.269448+0000","last_undegraded":"2024-04-26T16:31:51.269448+0000","last_fullsized":"2024-04-26T16:31:51.269448+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:46.104261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:46.104261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"388'19","reported_seq":1093,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.263106+0000","last_change":"2024-04-26T16:31:27.794473+0000","last_active":"2024-04-26T16:31:51.263106+0000","last_peered":"2024-04-26T16:31:51.263106+0000","last_clean":"2024-04-26T16:31:51.263106+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:31:51.263106+0000","last_undegraded":"2024-04-26T16:31:51.263106+0000","last_fullsized":"2024-04-26T16:31:51.263106+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:31:27.794108+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:31:27.794108+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.668954+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1068,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265056+0000","last_change":"2024-04-26T16:31:31.764805+0000","last_active":"2024-04-26T16:31:51.265056+0000","last_peered":"2024-04-26T16:31:51.265056+0000","last_clean":"2024-04-26T16:31:51.265056+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:31:51.265056+0000","last_undegraded":"2024-04-26T16:31:51.265056+0000","last_fullsized":"2024-04-26T16:31:51.265056+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:31.764565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_clean_scrub_stamp":"2024-04-26T16:31:31.764565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:52.423394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1225,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268662+0000","last_change":"2024-04-26T16:31:16.042144+0000","last_active":"2024-04-26T16:31:51.268662+0000","last_peered":"2024-04-26T16:31:51.268662+0000","last_clean":"2024-04-26T16:31:51.268662+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:31:51.268662+0000","last_undegraded":"2024-04-26T16:31:51.268662+0000","last_fullsized":"2024-04-26T16:31:51.268662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:31:16.041975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.041975+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:30.595842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1053,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278363+0000","last_change":"2024-04-26T16:30:00.630668+0000","last_active":"2024-04-26T16:31:51.278363+0000","last_peered":"2024-04-26T16:31:51.278363+0000","last_clean":"2024-04-26T16:31:51.278363+0000","last_became_active":"2024-04-26T16:26:23.188629+0000","last_became_peered":"2024-04-26T16:26:23.188629+0000","last_unstale":"2024-04-26T16:31:51.278363+0000","last_undegraded":"2024-04-26T16:31:51.278363+0000","last_fullsized":"2024-04-26T16:31:51.278363+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:00.630569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:00.630569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":900,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.299075+0000","last_change":"2024-04-26T16:31:43.245777+0000","last_active":"2024-04-26T16:31:51.299075+0000","last_peered":"2024-04-26T16:31:51.299075+0000","last_clean":"2024-04-26T16:31:51.299075+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:31:51.299075+0000","last_undegraded":"2024-04-26T16:31:51.299075+0000","last_fullsized":"2024-04-26T16:31:51.299075+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:43.245562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.245562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.403346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"388'19","reported_seq":1073,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298538+0000","last_change":"2024-04-26T16:28:43.978963+0000","last_active":"2024-04-26T16:31:51.298538+0000","last_peered":"2024-04-26T16:31:51.298538+0000","last_clean":"2024-04-26T16:31:51.298538+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:31:51.298538+0000","last_undegraded":"2024-04-26T16:31:51.298538+0000","last_fullsized":"2024-04-26T16:31:51.298538+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:28:43.978726+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:28:43.978726+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286959+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":952,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826068+0000","last_change":"2024-04-26T16:31:26.231532+0000","last_active":"2024-04-26T16:31:52.826068+0000","last_peered":"2024-04-26T16:31:52.826068+0000","last_clean":"2024-04-26T16:31:52.826068+0000","last_became_active":"2024-04-26T16:28:25.270839+0000","last_became_peered":"2024-04-26T16:28:25.270839+0000","last_unstale":"2024-04-26T16:31:52.826068+0000","last_undegraded":"2024-04-26T16:31:52.826068+0000","last_fullsized":"2024-04-26T16:31:52.826068+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.231313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.231313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:27.537096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1050,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266181+0000","last_change":"2024-04-26T16:31:41.030378+0000","last_active":"2024-04-26T16:31:51.266181+0000","last_peered":"2024-04-26T16:31:51.266181+0000","last_clean":"2024-04-26T16:31:51.266181+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:31:51.266181+0000","last_undegraded":"2024-04-26T16:31:51.266181+0000","last_fullsized":"2024-04-26T16:31:51.266181+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:41.030282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:41.030282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:53.056335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1071,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262431+0000","last_change":"2024-04-26T16:31:39.789722+0000","last_active":"2024-04-26T16:31:51.262431+0000","last_peered":"2024-04-26T16:31:51.262431+0000","last_clean":"2024-04-26T16:31:51.262431+0000","last_became_active":"2024-04-26T16:28:25.256886+0000","last_became_peered":"2024-04-26T16:28:25.256886+0000","last_unstale":"2024-04-26T16:31:51.262431+0000","last_undegraded":"2024-04-26T16:31:51.262431+0000","last_fullsized":"2024-04-26T16:31:51.262431+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:39.789606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.789606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:08.842421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":801,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826353+0000","last_change":"2024-04-26T16:31:20.323922+0000","last_active":"2024-04-26T16:31:52.826353+0000","last_peered":"2024-04-26T16:31:52.826353+0000","last_clean":"2024-04-26T16:31:52.826353+0000","last_became_active":"2024-04-26T16:28:25.270409+0000","last_became_peered":"2024-04-26T16:28:25.270409+0000","last_unstale":"2024-04-26T16:31:52.826353+0000","last_undegraded":"2024-04-26T16:31:52.826353+0000","last_fullsized":"2024-04-26T16:31:52.826353+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:20.323822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:20.323822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:42.514922+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"388'20","reported_seq":1103,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267112+0000","last_change":"2024-04-26T16:31:39.502396+0000","last_active":"2024-04-26T16:31:51.267112+0000","last_peered":"2024-04-26T16:31:51.267112+0000","last_clean":"2024-04-26T16:31:51.267112+0000","last_became_active":"2024-04-26T16:28:25.258640+0000","last_became_peered":"2024-04-26T16:28:25.258640+0000","last_unstale":"2024-04-26T16:31:51.267112+0000","last_undegraded":"2024-04-26T16:31:51.267112+0000","last_fullsized":"2024-04-26T16:31:51.267112+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:39.502283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.502283+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.231999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":562,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.268357+0000","last_change":"2024-04-26T16:30:21.968003+0000","last_active":"2024-04-26T16:31:51.268357+0000","last_peered":"2024-04-26T16:31:51.268357+0000","last_clean":"2024-04-26T16:31:51.268357+0000","last_became_active":"2024-04-26T16:28:25.226182+0000","last_became_peered":"2024-04-26T16:28:25.226182+0000","last_unstale":"2024-04-26T16:31:51.268357+0000","last_undegraded":"2024-04-26T16:31:51.268357+0000","last_fullsized":"2024-04-26T16:31:51.268357+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:21.967758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:21.967758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1059,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267115+0000","last_change":"2024-04-26T16:31:19.090724+0000","last_active":"2024-04-26T16:31:51.267115+0000","last_peered":"2024-04-26T16:31:51.267115+0000","last_clean":"2024-04-26T16:31:51.267115+0000","last_became_active":"2024-04-26T16:26:23.213441+0000","last_became_peered":"2024-04-26T16:26:23.213441+0000","last_unstale":"2024-04-26T16:31:51.267115+0000","last_undegraded":"2024-04-26T16:31:51.267115+0000","last_fullsized":"2024-04-26T16:31:51.267115+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:19.090521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.090521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.095425+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1044,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262979+0000","last_change":"2024-04-26T16:29:09.036096+0000","last_active":"2024-04-26T16:31:51.262979+0000","last_peered":"2024-04-26T16:31:51.262979+0000","last_clean":"2024-04-26T16:31:51.262979+0000","last_became_active":"2024-04-26T16:26:23.211857+0000","last_became_peered":"2024-04-26T16:26:23.211857+0000","last_unstale":"2024-04-26T16:31:51.262979+0000","last_undegraded":"2024-04-26T16:31:51.262979+0000","last_fullsized":"2024-04-26T16:31:51.262979+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:09.035817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:29:09.035817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":903,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.280088+0000","last_change":"2024-04-26T16:30:15.529255+0000","last_active":"2024-04-26T16:31:51.280088+0000","last_peered":"2024-04-26T16:31:51.280088+0000","last_clean":"2024-04-26T16:31:51.280088+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:31:51.280088+0000","last_undegraded":"2024-04-26T16:31:51.280088+0000","last_fullsized":"2024-04-26T16:31:51.280088+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:15.529105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:15.529105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'12","reported_seq":1104,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.269851+0000","last_change":"2024-04-26T16:31:20.801299+0000","last_active":"2024-04-26T16:31:51.269851+0000","last_peered":"2024-04-26T16:31:51.269851+0000","last_clean":"2024-04-26T16:31:51.269851+0000","last_became_active":"2024-04-26T16:28:25.246218+0000","last_became_peered":"2024-04-26T16:28:25.246218+0000","last_unstale":"2024-04-26T16:31:51.269851+0000","last_undegraded":"2024-04-26T16:31:51.269851+0000","last_fullsized":"2024-04-26T16:31:51.269851+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'12","last_scrub_stamp":"2024-04-26T16:31:20.801180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:20.801180+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:43.238767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":424,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.827019+0000","last_change":"2024-04-26T16:30:36.171558+0000","last_active":"2024-04-26T16:31:52.827019+0000","last_peered":"2024-04-26T16:31:52.827019+0000","last_clean":"2024-04-26T16:31:52.827019+0000","last_became_active":"2024-04-26T16:28:25.231296+0000","last_became_peered":"2024-04-26T16:28:25.231296+0000","last_unstale":"2024-04-26T16:31:52.827019+0000","last_undegraded":"2024-04-26T16:31:52.827019+0000","last_fullsized":"2024-04-26T16:31:52.827019+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:36.171451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:36.171451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:57.453859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":982,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826166+0000","last_change":"2024-04-26T16:30:19.325201+0000","last_active":"2024-04-26T16:31:52.826166+0000","last_peered":"2024-04-26T16:31:52.826166+0000","last_clean":"2024-04-26T16:31:52.826166+0000","last_became_active":"2024-04-26T16:28:25.272341+0000","last_became_peered":"2024-04-26T16:28:25.272341+0000","last_unstale":"2024-04-26T16:31:52.826166+0000","last_undegraded":"2024-04-26T16:31:52.826166+0000","last_fullsized":"2024-04-26T16:31:52.826166+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.325091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.325091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:57.453835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1037,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.294379+0000","last_change":"2024-04-26T16:30:34.003108+0000","last_active":"2024-04-26T16:31:51.294379+0000","last_peered":"2024-04-26T16:31:51.294379+0000","last_clean":"2024-04-26T16:31:51.294379+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:31:51.294379+0000","last_undegraded":"2024-04-26T16:31:51.294379+0000","last_fullsized":"2024-04-26T16:31:51.294379+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:34.002980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.002980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:01.133005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":902,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297826+0000","last_change":"2024-04-26T16:28:35.971437+0000","last_active":"2024-04-26T16:31:51.297826+0000","last_peered":"2024-04-26T16:31:51.297826+0000","last_clean":"2024-04-26T16:31:51.297826+0000","last_became_active":"2024-04-26T16:28:25.269747+0000","last_became_peered":"2024-04-26T16:28:25.269747+0000","last_unstale":"2024-04-26T16:31:51.297826+0000","last_undegraded":"2024-04-26T16:31:51.297826+0000","last_fullsized":"2024-04-26T16:31:51.297826+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:35.971222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:28:35.971222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'15","reported_seq":598,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297956+0000","last_change":"2024-04-26T16:27:43.107066+0000","last_active":"2024-04-26T16:31:51.297956+0000","last_peered":"2024-04-26T16:31:51.297956+0000","last_clean":"2024-04-26T16:31:51.297956+0000","last_became_active":"2024-04-26T16:26:23.200216+0000","last_became_peered":"2024-04-26T16:26:23.200216+0000","last_unstale":"2024-04-26T16:31:51.297956+0000","last_undegraded":"2024-04-26T16:31:51.297956+0000","last_fullsized":"2024-04-26T16:31:51.297956+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:27:43.106910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:27:43.106910+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286523+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":963,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.827162+0000","last_change":"2024-04-26T16:31:17.224683+0000","last_active":"2024-04-26T16:31:52.827162+0000","last_peered":"2024-04-26T16:31:52.827162+0000","last_clean":"2024-04-26T16:31:52.827162+0000","last_became_active":"2024-04-26T16:28:25.244917+0000","last_became_peered":"2024-04-26T16:28:25.244917+0000","last_unstale":"2024-04-26T16:31:52.827162+0000","last_undegraded":"2024-04-26T16:31:52.827162+0000","last_fullsized":"2024-04-26T16:31:52.827162+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:17.224543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:31:17.224543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:22.633061+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1050,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.287018+0000","last_change":"2024-04-26T16:28:57.063805+0000","last_active":"2024-04-26T16:31:51.287018+0000","last_peered":"2024-04-26T16:31:51.287018+0000","last_clean":"2024-04-26T16:31:51.287018+0000","last_became_active":"2024-04-26T16:28:25.266296+0000","last_became_peered":"2024-04-26T16:28:25.266296+0000","last_unstale":"2024-04-26T16:31:51.287018+0000","last_undegraded":"2024-04-26T16:31:51.287018+0000","last_fullsized":"2024-04-26T16:31:51.287018+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:57.063594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:57.063594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1047,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266463+0000","last_change":"2024-04-26T16:30:09.970620+0000","last_active":"2024-04-26T16:31:51.266463+0000","last_peered":"2024-04-26T16:31:51.266463+0000","last_clean":"2024-04-26T16:31:51.266463+0000","last_became_active":"2024-04-26T16:28:25.228971+0000","last_became_peered":"2024-04-26T16:28:25.228971+0000","last_unstale":"2024-04-26T16:31:51.266463+0000","last_undegraded":"2024-04-26T16:31:51.266463+0000","last_fullsized":"2024-04-26T16:31:51.266463+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.970505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.970505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.931051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":902,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.257041+0000","last_change":"2024-04-26T16:30:22.616210+0000","last_active":"2024-04-26T16:31:51.257041+0000","last_peered":"2024-04-26T16:31:51.257041+0000","last_clean":"2024-04-26T16:31:51.257041+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:31:51.257041+0000","last_undegraded":"2024-04-26T16:31:51.257041+0000","last_fullsized":"2024-04-26T16:31:51.257041+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.616065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.616065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582875+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"388'34","reported_seq":1114,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.260170+0000","last_change":"2024-04-26T16:31:36.079785+0000","last_active":"2024-04-26T16:31:51.260170+0000","last_peered":"2024-04-26T16:31:51.260170+0000","last_clean":"2024-04-26T16:31:51.260170+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:31:51.260170+0000","last_undegraded":"2024-04-26T16:31:51.260170+0000","last_fullsized":"2024-04-26T16:31:51.260170+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:31:36.079388+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:31:36.079388+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:41.990419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":562,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262753+0000","last_change":"2024-04-26T16:29:07.493539+0000","last_active":"2024-04-26T16:31:51.262753+0000","last_peered":"2024-04-26T16:31:51.262753+0000","last_clean":"2024-04-26T16:31:51.262753+0000","last_became_active":"2024-04-26T16:28:25.250387+0000","last_became_peered":"2024-04-26T16:28:25.250387+0000","last_unstale":"2024-04-26T16:31:51.262753+0000","last_undegraded":"2024-04-26T16:31:51.262753+0000","last_fullsized":"2024-04-26T16:31:51.262753+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:07.493409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:07.493409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"433'14","reported_seq":1123,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.827066+0000","last_change":"2024-04-26T16:31:21.319007+0000","last_active":"2024-04-26T16:31:52.827066+0000","last_peered":"2024-04-26T16:31:52.827066+0000","last_clean":"2024-04-26T16:31:52.827066+0000","last_became_active":"2024-04-26T16:28:25.265640+0000","last_became_peered":"2024-04-26T16:28:25.265640+0000","last_unstale":"2024-04-26T16:31:52.827066+0000","last_undegraded":"2024-04-26T16:31:52.827066+0000","last_fullsized":"2024-04-26T16:31:52.827066+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:31:21.318863+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:31:21.318863+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:28.958833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":1048,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298490+0000","last_change":"2024-04-26T16:30:11.234876+0000","last_active":"2024-04-26T16:31:51.298490+0000","last_peered":"2024-04-26T16:31:51.298490+0000","last_clean":"2024-04-26T16:31:51.298490+0000","last_became_active":"2024-04-26T16:26:23.201293+0000","last_became_peered":"2024-04-26T16:26:23.201293+0000","last_unstale":"2024-04-26T16:31:51.298490+0000","last_undegraded":"2024-04-26T16:31:51.298490+0000","last_fullsized":"2024-04-26T16:31:51.298490+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:11.234642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:11.234642+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287489+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":912,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267274+0000","last_change":"2024-04-26T16:30:16.615509+0000","last_active":"2024-04-26T16:31:51.267274+0000","last_peered":"2024-04-26T16:31:51.267274+0000","last_clean":"2024-04-26T16:31:51.267274+0000","last_became_active":"2024-04-26T16:28:25.247238+0000","last_became_peered":"2024-04-26T16:28:25.247238+0000","last_unstale":"2024-04-26T16:31:51.267274+0000","last_undegraded":"2024-04-26T16:31:51.267274+0000","last_fullsized":"2024-04-26T16:31:51.267274+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.615354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.615354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.829781+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"388'15","reported_seq":1070,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265816+0000","last_change":"2024-04-26T16:28:59.039798+0000","last_active":"2024-04-26T16:31:51.265816+0000","last_peered":"2024-04-26T16:31:51.265816+0000","last_clean":"2024-04-26T16:31:51.265816+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:31:51.265816+0000","last_undegraded":"2024-04-26T16:31:51.265816+0000","last_fullsized":"2024-04-26T16:31:51.265816+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:28:59.039575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:59.039575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1060,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.263132+0000","last_change":"2024-04-26T16:31:17.081118+0000","last_active":"2024-04-26T16:31:51.263132+0000","last_peered":"2024-04-26T16:31:51.263132+0000","last_clean":"2024-04-26T16:31:51.263132+0000","last_became_active":"2024-04-26T16:28:25.245293+0000","last_became_peered":"2024-04-26T16:28:25.245293+0000","last_unstale":"2024-04-26T16:31:51.263132+0000","last_undegraded":"2024-04-26T16:31:51.263132+0000","last_fullsized":"2024-04-26T16:31:51.263132+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:31:17.080989+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:31:17.080989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:38.286820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1061,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266064+0000","last_change":"2024-04-26T16:30:28.566079+0000","last_active":"2024-04-26T16:31:51.266064+0000","last_peered":"2024-04-26T16:31:51.266064+0000","last_clean":"2024-04-26T16:31:51.266064+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:31:51.266064+0000","last_undegraded":"2024-04-26T16:31:51.266064+0000","last_fullsized":"2024-04-26T16:31:51.266064+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.565983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.565983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.829832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1062,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271596+0000","last_change":"2024-04-26T16:31:33.048781+0000","last_active":"2024-04-26T16:31:51.271596+0000","last_peered":"2024-04-26T16:31:51.271596+0000","last_clean":"2024-04-26T16:31:51.271596+0000","last_became_active":"2024-04-26T16:26:23.203386+0000","last_became_peered":"2024-04-26T16:26:23.203386+0000","last_unstale":"2024-04-26T16:31:51.271596+0000","last_undegraded":"2024-04-26T16:31:51.271596+0000","last_fullsized":"2024-04-26T16:31:51.271596+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:33.048641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:33.048641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.001557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":562,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266063+0000","last_change":"2024-04-26T16:30:29.497506+0000","last_active":"2024-04-26T16:31:51.266063+0000","last_peered":"2024-04-26T16:31:51.266063+0000","last_clean":"2024-04-26T16:31:51.266063+0000","last_became_active":"2024-04-26T16:28:25.238120+0000","last_became_peered":"2024-04-26T16:28:25.238120+0000","last_unstale":"2024-04-26T16:31:51.266063+0000","last_undegraded":"2024-04-26T16:31:51.266063+0000","last_fullsized":"2024-04-26T16:31:51.266063+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:29.497411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:29.497411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"388'9","reported_seq":1078,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265240+0000","last_change":"2024-04-26T16:31:34.857477+0000","last_active":"2024-04-26T16:31:51.265240+0000","last_peered":"2024-04-26T16:31:51.265240+0000","last_clean":"2024-04-26T16:31:51.265240+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:31:51.265240+0000","last_undegraded":"2024-04-26T16:31:51.265240+0000","last_fullsized":"2024-04-26T16:31:51.265240+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:31:34.857307+0000","last_deep_scrub":"388'9","last_deep_scrub_stamp":"2024-04-26T16:31:34.857307+0000","last_clean_scrub_stamp":"2024-04-26T16:31:34.857307+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:46.181105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1014,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.825685+0000","last_change":"2024-04-26T16:31:43.402411+0000","last_active":"2024-04-26T16:31:52.825685+0000","last_peered":"2024-04-26T16:31:52.825685+0000","last_clean":"2024-04-26T16:31:52.825685+0000","last_became_active":"2024-04-26T16:28:25.264839+0000","last_became_peered":"2024-04-26T16:28:25.264839+0000","last_unstale":"2024-04-26T16:31:52.825685+0000","last_undegraded":"2024-04-26T16:31:52.825685+0000","last_fullsized":"2024-04-26T16:31:52.825685+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:43.402207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.402207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:54.302434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1060,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.271331+0000","last_change":"2024-04-26T16:31:21.029710+0000","last_active":"2024-04-26T16:31:51.271331+0000","last_peered":"2024-04-26T16:31:51.271331+0000","last_clean":"2024-04-26T16:31:51.271331+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:31:51.271331+0000","last_undegraded":"2024-04-26T16:31:51.271331+0000","last_fullsized":"2024-04-26T16:31:51.271331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:21.029556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:21.029556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:39.610748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1072,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.269082+0000","last_change":"2024-04-26T16:30:26.547964+0000","last_active":"2024-04-26T16:31:51.269082+0000","last_peered":"2024-04-26T16:31:51.269082+0000","last_clean":"2024-04-26T16:31:51.269082+0000","last_became_active":"2024-04-26T16:28:25.244502+0000","last_became_peered":"2024-04-26T16:28:25.244502+0000","last_unstale":"2024-04-26T16:31:51.269082+0000","last_undegraded":"2024-04-26T16:31:51.269082+0000","last_fullsized":"2024-04-26T16:31:51.269082+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.547756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.547756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":892,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.291482+0000","last_change":"2024-04-26T16:27:44.070368+0000","last_active":"2024-04-26T16:31:51.291482+0000","last_peered":"2024-04-26T16:31:51.291482+0000","last_clean":"2024-04-26T16:31:51.291482+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:31:51.291482+0000","last_undegraded":"2024-04-26T16:31:51.291482+0000","last_fullsized":"2024-04-26T16:31:51.291482+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:27:44.070141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.286568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'15","reported_seq":1088,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266634+0000","last_change":"2024-04-26T16:30:18.494734+0000","last_active":"2024-04-26T16:31:51.266634+0000","last_peered":"2024-04-26T16:31:51.266634+0000","last_clean":"2024-04-26T16:31:51.266634+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:31:51.266634+0000","last_undegraded":"2024-04-26T16:31:51.266634+0000","last_fullsized":"2024-04-26T16:31:51.266634+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:30:18.494624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:18.494624+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1058,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265944+0000","last_change":"2024-04-26T16:31:18.051005+0000","last_active":"2024-04-26T16:31:51.265944+0000","last_peered":"2024-04-26T16:31:51.265944+0000","last_clean":"2024-04-26T16:31:51.265944+0000","last_became_active":"2024-04-26T16:28:25.255455+0000","last_became_peered":"2024-04-26T16:28:25.255455+0000","last_unstale":"2024-04-26T16:31:51.265944+0000","last_undegraded":"2024-04-26T16:31:51.265944+0000","last_fullsized":"2024-04-26T16:31:51.265944+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:18.050865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:18.050865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:20.999045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1051,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267538+0000","last_change":"2024-04-26T16:29:04.486590+0000","last_active":"2024-04-26T16:31:51.267538+0000","last_peered":"2024-04-26T16:31:51.267538+0000","last_clean":"2024-04-26T16:31:51.267538+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:31:51.267538+0000","last_undegraded":"2024-04-26T16:31:51.267538+0000","last_fullsized":"2024-04-26T16:31:51.267538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:04.486382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:04.486382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1053,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.274384+0000","last_change":"2024-04-26T16:30:26.949591+0000","last_active":"2024-04-26T16:31:51.274384+0000","last_peered":"2024-04-26T16:31:51.274384+0000","last_clean":"2024-04-26T16:31:51.274384+0000","last_became_active":"2024-04-26T16:28:25.245307+0000","last_became_peered":"2024-04-26T16:28:25.245307+0000","last_unstale":"2024-04-26T16:31:51.274384+0000","last_undegraded":"2024-04-26T16:31:51.274384+0000","last_fullsized":"2024-04-26T16:31:51.274384+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.949357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.949357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":902,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.278669+0000","last_change":"2024-04-26T16:30:04.495577+0000","last_active":"2024-04-26T16:31:51.278669+0000","last_peered":"2024-04-26T16:31:51.278669+0000","last_clean":"2024-04-26T16:31:51.278669+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:31:51.278669+0000","last_undegraded":"2024-04-26T16:31:51.278669+0000","last_fullsized":"2024-04-26T16:31:51.278669+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:04.495235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:04.495235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.582521+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"388'23","reported_seq":726,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.267699+0000","last_change":"2024-04-26T16:28:45.510060+0000","last_active":"2024-04-26T16:31:51.267699+0000","last_peered":"2024-04-26T16:31:51.267699+0000","last_clean":"2024-04-26T16:31:51.267699+0000","last_became_active":"2024-04-26T16:28:25.250379+0000","last_became_peered":"2024-04-26T16:28:25.250379+0000","last_unstale":"2024-04-26T16:31:51.267699+0000","last_undegraded":"2024-04-26T16:31:51.267699+0000","last_fullsized":"2024-04-26T16:31:51.267699+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:28:45.509872+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:28:45.509872+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1063,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.295942+0000","last_change":"2024-04-26T16:29:02.113592+0000","last_active":"2024-04-26T16:31:51.295942+0000","last_peered":"2024-04-26T16:31:51.295942+0000","last_clean":"2024-04-26T16:31:51.295942+0000","last_became_active":"2024-04-26T16:28:25.277266+0000","last_became_peered":"2024-04-26T16:28:25.277266+0000","last_unstale":"2024-04-26T16:31:51.295942+0000","last_undegraded":"2024-04-26T16:31:51.295942+0000","last_fullsized":"2024-04-26T16:31:51.295942+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:02.113366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:29:02.113366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":556,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.264823+0000","last_change":"2024-04-26T16:31:37.787782+0000","last_active":"2024-04-26T16:31:51.264823+0000","last_peered":"2024-04-26T16:31:51.264823+0000","last_clean":"2024-04-26T16:31:51.264823+0000","last_became_active":"2024-04-26T16:26:23.166910+0000","last_became_peered":"2024-04-26T16:26:23.166910+0000","last_unstale":"2024-04-26T16:31:51.264823+0000","last_undegraded":"2024-04-26T16:31:51.264823+0000","last_fullsized":"2024-04-26T16:31:51.264823+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:37.787531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:37.787531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:50.634430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1055,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262855+0000","last_change":"2024-04-26T16:31:46.843949+0000","last_active":"2024-04-26T16:31:51.262855+0000","last_peered":"2024-04-26T16:31:51.262855+0000","last_clean":"2024-04-26T16:31:51.262855+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:31:51.262855+0000","last_undegraded":"2024-04-26T16:31:51.262855+0000","last_fullsized":"2024-04-26T16:31:51.262855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:46.843820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_clean_scrub_stamp":"2024-04-26T16:31:46.843820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:50.107235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":823,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-04-26T16:31:52.826463+0000","last_change":"2024-04-26T16:31:29.175004+0000","last_active":"2024-04-26T16:31:52.826463+0000","last_peered":"2024-04-26T16:31:52.826463+0000","last_clean":"2024-04-26T16:31:52.826463+0000","last_became_active":"2024-04-26T16:28:25.255571+0000","last_became_peered":"2024-04-26T16:28:25.255571+0000","last_unstale":"2024-04-26T16:31:52.826463+0000","last_undegraded":"2024-04-26T16:31:52.826463+0000","last_fullsized":"2024-04-26T16:31:52.826463+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:29.174775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:29.174775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:47.408020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"388'27","reported_seq":640,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.266654+0000","last_change":"2024-04-26T16:30:13.510016+0000","last_active":"2024-04-26T16:31:51.266654+0000","last_peered":"2024-04-26T16:31:51.266654+0000","last_clean":"2024-04-26T16:31:51.266654+0000","last_became_active":"2024-04-26T16:28:25.256896+0000","last_became_peered":"2024-04-26T16:28:25.256896+0000","last_unstale":"2024-04-26T16:31:51.266654+0000","last_undegraded":"2024-04-26T16:31:51.266654+0000","last_fullsized":"2024-04-26T16:31:51.266654+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:30:13.509869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.509869+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1065,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.298352+0000","last_change":"2024-04-26T16:30:13.187491+0000","last_active":"2024-04-26T16:31:51.298352+0000","last_peered":"2024-04-26T16:31:51.298352+0000","last_clean":"2024-04-26T16:31:51.298352+0000","last_became_active":"2024-04-26T16:28:25.271550+0000","last_became_peered":"2024-04-26T16:28:25.271550+0000","last_unstale":"2024-04-26T16:31:51.298352+0000","last_undegraded":"2024-04-26T16:31:51.298352+0000","last_fullsized":"2024-04-26T16:31:51.298352+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:13.187197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.187197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1040,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.294587+0000","last_change":"2024-04-26T16:29:57.141987+0000","last_active":"2024-04-26T16:31:51.294587+0000","last_peered":"2024-04-26T16:31:51.294587+0000","last_clean":"2024-04-26T16:31:51.294587+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:31:51.294587+0000","last_undegraded":"2024-04-26T16:31:51.294587+0000","last_fullsized":"2024-04-26T16:31:51.294587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:57.141873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:29:57.141873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":570,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.260612+0000","last_change":"2024-04-26T16:30:34.033387+0000","last_active":"2024-04-26T16:31:51.260612+0000","last_peered":"2024-04-26T16:31:51.260612+0000","last_clean":"2024-04-26T16:31:51.260612+0000","last_became_active":"2024-04-26T16:26:23.212770+0000","last_became_peered":"2024-04-26T16:26:23.212770+0000","last_unstale":"2024-04-26T16:31:51.260612+0000","last_undegraded":"2024-04-26T16:31:51.260612+0000","last_fullsized":"2024-04-26T16:31:51.260612+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:30:34.033282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.033282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":915,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297404+0000","last_change":"2024-04-26T16:30:14.201574+0000","last_active":"2024-04-26T16:31:51.297404+0000","last_peered":"2024-04-26T16:31:51.297404+0000","last_clean":"2024-04-26T16:31:51.297404+0000","last_became_active":"2024-04-26T16:28:25.267368+0000","last_became_peered":"2024-04-26T16:28:25.267368+0000","last_unstale":"2024-04-26T16:31:51.297404+0000","last_undegraded":"2024-04-26T16:31:51.297404+0000","last_fullsized":"2024-04-26T16:31:51.297404+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.201493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.201493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"388'19","reported_seq":1122,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.270013+0000","last_change":"2024-04-26T16:31:26.779377+0000","last_active":"2024-04-26T16:31:51.270013+0000","last_peered":"2024-04-26T16:31:51.270013+0000","last_clean":"2024-04-26T16:31:51.270013+0000","last_became_active":"2024-04-26T16:28:25.244993+0000","last_became_peered":"2024-04-26T16:28:25.244993+0000","last_unstale":"2024-04-26T16:31:51.270013+0000","last_undegraded":"2024-04-26T16:31:51.270013+0000","last_fullsized":"2024-04-26T16:31:51.270013+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:31:26.779258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.779258+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:51.712424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1065,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.273015+0000","last_change":"2024-04-26T16:30:32.981558+0000","last_active":"2024-04-26T16:31:51.273015+0000","last_peered":"2024-04-26T16:31:51.273015+0000","last_clean":"2024-04-26T16:31:51.273015+0000","last_became_active":"2024-04-26T16:26:23.214404+0000","last_became_peered":"2024-04-26T16:26:23.214404+0000","last_unstale":"2024-04-26T16:31:51.273015+0000","last_undegraded":"2024-04-26T16:31:51.273015+0000","last_fullsized":"2024-04-26T16:31:51.273015+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:32.981351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:32.981351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.931019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1066,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.260283+0000","last_change":"2024-04-26T16:31:20.057831+0000","last_active":"2024-04-26T16:31:51.260283+0000","last_peered":"2024-04-26T16:31:51.260283+0000","last_clean":"2024-04-26T16:31:51.260283+0000","last_became_active":"2024-04-26T16:26:23.178678+0000","last_became_peered":"2024-04-26T16:26:23.178678+0000","last_unstale":"2024-04-26T16:31:51.260283+0000","last_undegraded":"2024-04-26T16:31:51.260283+0000","last_fullsized":"2024-04-26T16:31:51.260283+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:20.057515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:20.057515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:31.293526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1061,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265959+0000","last_change":"2024-04-26T16:31:23.471502+0000","last_active":"2024-04-26T16:31:51.265959+0000","last_peered":"2024-04-26T16:31:51.265959+0000","last_clean":"2024-04-26T16:31:51.265959+0000","last_became_active":"2024-04-26T16:26:23.198295+0000","last_became_peered":"2024-04-26T16:26:23.198295+0000","last_unstale":"2024-04-26T16:31:51.265959+0000","last_undegraded":"2024-04-26T16:31:51.265959+0000","last_fullsized":"2024-04-26T16:31:51.265959+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.471197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.471197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:50.817089+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":544,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.297195+0000","last_change":"2024-04-26T16:30:16.192009+0000","last_active":"2024-04-26T16:31:51.297195+0000","last_peered":"2024-04-26T16:31:51.297195+0000","last_clean":"2024-04-26T16:31:51.297195+0000","last_became_active":"2024-04-26T16:26:23.199414+0000","last_became_peered":"2024-04-26T16:26:23.199414+0000","last_unstale":"2024-04-26T16:31:51.297195+0000","last_undegraded":"2024-04-26T16:31:51.297195+0000","last_fullsized":"2024-04-26T16:31:51.297195+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:16.191696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:16.191696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:54.287831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"388'20","reported_seq":1086,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.273614+0000","last_change":"2024-04-26T16:28:48.140271+0000","last_active":"2024-04-26T16:31:51.273614+0000","last_peered":"2024-04-26T16:31:51.273614+0000","last_clean":"2024-04-26T16:31:51.273614+0000","last_became_active":"2024-04-26T16:28:25.232220+0000","last_became_peered":"2024-04-26T16:28:25.232220+0000","last_unstale":"2024-04-26T16:31:51.273614+0000","last_undegraded":"2024-04-26T16:31:51.273614+0000","last_fullsized":"2024-04-26T16:31:51.273614+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:28:48.140161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:28:48.140161+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1068,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.270020+0000","last_change":"2024-04-26T16:30:33.689694+0000","last_active":"2024-04-26T16:31:51.270020+0000","last_peered":"2024-04-26T16:31:51.270020+0000","last_clean":"2024-04-26T16:31:51.270020+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:31:51.270020+0000","last_undegraded":"2024-04-26T16:31:51.270020+0000","last_fullsized":"2024-04-26T16:31:51.270020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:33.689564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:30:33.689564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:02.010334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1068,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.262468+0000","last_change":"2024-04-26T16:30:22.493662+0000","last_active":"2024-04-26T16:31:51.262468+0000","last_peered":"2024-04-26T16:31:51.262468+0000","last_clean":"2024-04-26T16:31:51.262468+0000","last_became_active":"2024-04-26T16:28:25.250239+0000","last_became_peered":"2024-04-26T16:28:25.250239+0000","last_unstale":"2024-04-26T16:31:51.262468+0000","last_undegraded":"2024-04-26T16:31:51.262468+0000","last_fullsized":"2024-04-26T16:31:51.262468+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.493412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.493412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1046,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-04-26T16:31:51.265569+0000","last_change":"2024-04-26T16:30:12.027247+0000","last_active":"2024-04-26T16:31:51.265569+0000","last_peered":"2024-04-26T16:31:51.265569+0000","last_clean":"2024-04-26T16:31:51.265569+0000","last_became_active":"2024-04-26T16:28:25.250958+0000","last_became_peered":"2024-04-26T16:28:25.250958+0000","last_unstale":"2024-04-26T16:31:51.265569+0000","last_undegraded":"2024-04-26T16:31:51.265569+0000","last_fullsized":"2024-04-26T16:31:51.265569+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.027062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.027062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:31:55.930606+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":313,"num_pg":32,"stat_sum":{"num_bytes":50,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":7},{"poolid":109,"num_pg":32,"stat_sum":{"num_bytes":8214794,"num_objects":198,"num_object_clones":14,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":219,"num_read_kb":62,"num_write":784,"num_write_kb":12812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":4194787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":25,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26935296,"data_stored":24644409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2109,"ondisk_log_size":2109,"up":96,"acting":96,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1124,"num_read_kb":881,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":1392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":840,"num_read_kb":835,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743607,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97252,"kb_used_data":3600,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93647900,"statfs":{"total":95995035648,"available":95895449600,"internally_reserved":0,"allocated":3686400,"data_stored":1543654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":432,"seq":1855425871915,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40032,"kb_used_data":3408,"kb_used_omap":15,"kb_used_meta":36592,"kb_avail":93705120,"statfs":{"total":95995035648,"available":95954042880,"internally_reserved":0,"allocated":3489792,"data_stored":1539212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15525,"internal_metadata":37471067},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78224,"kb_used_data":10948,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93666928,"statfs":{"total":95995035648,"available":95914934272,"internally_reserved":0,"allocated":11210752,"data_stored":9132291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74932,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93670220,"statfs":{"total":95995035648,"available":95918305280,"internally_reserved":0,"allocated":3579904,"data_stored":1531273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411562,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78260,"kb_used_data":6816,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93666892,"statfs":{"total":95995035648,"available":95914897408,"internally_reserved":0,"allocated":6979584,"data_stored":4949519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084511,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104568,"kb_used_data":10984,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":93640584,"statfs":{"total":95995035648,"available":95887958016,"internally_reserved":0,"allocated":11247616,"data_stored":9139453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":95812040},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855573,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78492,"kb_used_data":7048,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93666660,"statfs":{"total":95995035648,"available":95914659840,"internally_reserved":0,"allocated":7217152,"data_stored":5144115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":201863463115,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":7797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8548352,"data_stored":8200192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4317184,"data_stored":4008328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8597504,"data_stored":8207883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4526080,"data_stored":4202369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":7880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":109,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":9960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":40,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":20,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":313,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":30,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:31:57.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:57.612 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-26T16:31:57.612 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:31:57.613 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-04-26T16:31:57.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd in 5 2024-04-26T16:31:57.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:57.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:57.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:57.861 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:31:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:31:57.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:57.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:57.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:58.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:58.121 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:58.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:58.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:58.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:58.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:31:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:31:58.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:58.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:58 smithi190 bash[23868]: audit 2024-04-26T16:31:57.047513+0000 mgr.y (mgr.24401) 566 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:58.496 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:58 smithi190 bash[23868]: audit 2024-04-26T16:31:57.047513+0000 mgr.y (mgr.24401) 566 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:58.512 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:58.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:58 smithi050 bash[21463]: audit 2024-04-26T16:31:57.047513+0000 mgr.y (mgr.24401) 566 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:58 smithi050 bash[21463]: audit 2024-04-26T16:31:57.047513+0000 mgr.y (mgr.24401) 566 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:58.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:58 smithi050 bash[29234]: audit 2024-04-26T16:31:57.047513+0000 mgr.y (mgr.24401) 566 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:58.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:58 smithi050 bash[29234]: audit 2024-04-26T16:31:57.047513+0000 mgr.y (mgr.24401) 566 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:31:58.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:58.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:58.775 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:58.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:58.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:58.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:59.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:59.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:31:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:31:59.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:59.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:59.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:59.299 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:31:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:31:59.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:59.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:31:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:31:59.529 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:59.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:59.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:31:59.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:59 smithi190 bash[23868]: cluster 2024-04-26T16:31:57.759422+0000 mgr.y (mgr.24401) 567 : cluster [DBG] pgmap v890: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 372 B/s wr, 4 op/s 2024-04-26T16:31:59.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:31:59 smithi190 bash[23868]: cluster 2024-04-26T16:31:57.759422+0000 mgr.y (mgr.24401) 567 : cluster [DBG] pgmap v890: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 372 B/s wr, 4 op/s 2024-04-26T16:31:59.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:31:59.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:59 smithi050 bash[29234]: cluster 2024-04-26T16:31:57.759422+0000 mgr.y (mgr.24401) 567 : cluster [DBG] pgmap v890: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 372 B/s wr, 4 op/s 2024-04-26T16:31:59.612 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:31:59 smithi050 bash[29234]: cluster 2024-04-26T16:31:57.759422+0000 mgr.y (mgr.24401) 567 : cluster [DBG] pgmap v890: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 372 B/s wr, 4 op/s 2024-04-26T16:31:59.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:59 smithi050 bash[21463]: cluster 2024-04-26T16:31:57.759422+0000 mgr.y (mgr.24401) 567 : cluster [DBG] pgmap v890: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 372 B/s wr, 4 op/s 2024-04-26T16:31:59.613 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:31:59 smithi050 bash[21463]: cluster 2024-04-26T16:31:57.759422+0000 mgr.y (mgr.24401) 567 : cluster [DBG] pgmap v890: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.6 KiB/s rd, 372 B/s wr, 4 op/s 2024-04-26T16:31:59.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:59.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:59.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:59.819 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:31:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:31:59.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:59.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:31:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:31:59.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:00.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:00.080 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:00.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:00.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:00.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:00.339 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:00.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:00.469 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:00.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:00.600 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:00.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:00.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:00.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/4094566425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f67001034e0 msgr2=0x7f6700103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- 172.21.15.50:0/4094566425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f67001034e0 0x7f6700103960 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f66f0009990 tx=0x7f66f0009ca0 comp rx=0 tx=0).stop 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/4094566425 shutdown_connections 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- 172.21.15.50:0/4094566425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6700103ea0 0x7f67001102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- 172.21.15.50:0/4094566425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 0x7f6700109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- 172.21.15.50:0/4094566425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f67001034e0 0x7f6700103960 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/4094566425 >> 172.21.15.50:0/4094566425 conn(0x7f670007a5e0 msgr2=0x7f670007a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/4094566425 shutdown_connections 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/4094566425 wait complete. 2024-04-26T16:32:00.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 librados: starting msgr at 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 librados: starting objecter 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 Processor -- start 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- start start 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 librados: setting wanted keys 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 librados: calling monclient init 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67001034e0 0x7f67001991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 0x7f6700199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 0x7f670019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6700112a80 con 0x7f6700109480 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6700112900 con 0x7f67001034e0 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6700112c00 con 0x7f6700103ea0 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 0x7f6700199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6706c25700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 0x7f670019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 0x7f6700199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53616/0 (socket says 172.21.15.50:53616) 2024-04-26T16:32:00.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 -- 172.21.15.50:0/1645504570 learned_addr learned my addr 172.21.15.50:0/1645504570 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6706424700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67001034e0 0x7f67001991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 -- 172.21.15.50:0/1645504570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67001034e0 msgr2=0x7f67001991e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67001034e0 0x7f67001991e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 -- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 msgr2=0x7f670019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 0x7f670019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 -- 172.21.15.50:0/1645504570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670019e1f0 con 0x7f6700103ea0 2024-04-26T16:32:00.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6706c25700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 0x7f670019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6705c23700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 0x7f6700199720 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f66f00057a0 tx=0x7f66f00057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f66f77fe700 10 client.?.objecter ms_handle_connect 0x7f6700103ea0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f66f77fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f0026070 con 0x7f6700103ea0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f670019e480 con 0x7f6700103ea0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f66f0021410 con 0x7f6700103ea0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00173f0 con 0x7f6700103ea0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 10 client.41411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f67001a5d60 con 0x7f6700103ea0 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 librados: init done 2024-04-26T16:32:00.835 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.833+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f670006a110 con 0x7f6700103ea0 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f66f0029070 con 0x7f6700103ea0 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f66ec071f80 0x7f66ec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(581..581 src has 496..581) ==== 9377+0+0 (secure 0 0 0) 0x7f66f0092720 con 0x7f6700103ea0 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 10 client.41411.objecter ms_dispatch 0x7f6700001550 osd_map(581..581 src has 496..581) 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 3 client.41411.objecter handle_osd_map got epochs [581,581] > 0 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 3 client.41411.objecter handle_osd_map decoding full epoch 581 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f6706424700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f66ec071f80 0x7f66ec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 20 client.41411.objecter dump_active .. 0 homeless 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f6706424700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f66ec071f80 0x7f66ec074440 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009bd0 tx=0x7f66fc009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:00.836 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.837+0000 7f66f77fe700 10 client.41411.objecter ms_handle_connect 0x7f66ec071f80 2024-04-26T16:32:00.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:00.841+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f66f0013070 con 0x7f6700103ea0 2024-04-26T16:32:00.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:00.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:00.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:00.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:01.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:01.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.085+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f67001a5f30 con 0x7f6700103ea0 2024-04-26T16:32:01.103 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:01.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:01.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:01.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.357+0000 7f66f77fe700 1 -- 172.21.15.50:0/1645504570 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v582) ==== 97+0+0 (secure 0 0 0) 0x7f66f0061d80 con 0x7f6700103ea0 2024-04-26T16:32:01.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 10 librados: watch_flush enter 2024-04-26T16:32:01.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 10 librados: watch_flush exit 2024-04-26T16:32:01.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 20 client.41411.objecter shutdown clearing up homeless session... 2024-04-26T16:32:01.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 10 client.41411.objecter successfully canceled tick 2024-04-26T16:32:01.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f66ec071f80 msgr2=0x7f66ec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:01.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f66ec071f80 0x7f66ec074440 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f66fc009bd0 tx=0x7f66fc009260 comp rx=0 tx=0).stop 2024-04-26T16:32:01.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 msgr2=0x7f6700199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:01.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 0x7f6700199720 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f66f00057a0 tx=0x7f66f00057d0 comp rx=0 tx=0).stop 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 shutdown_connections 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f66ec071f80 0x7f66ec074440 unknown :-1 s=CLOSED pgs=3780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67001034e0 0x7f67001991e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6700109480 0x7f670019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 --2- 172.21.15.50:0/1645504570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6700103ea0 0x7f6700199720 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.361+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 >> 172.21.15.50:0/1645504570 conn(0x7f670007a5e0 msgr2=0x7f6700101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.365+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 shutdown_connections 2024-04-26T16:32:01.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.365+0000 7f6708688700 1 -- 172.21.15.50:0/1645504570 wait complete. 2024-04-26T16:32:01.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:01.365+0000 7f6708688700 1 librados: shutdown 2024-04-26T16:32:01.363 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:32:01.363 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:01.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:01.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:01.495 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:01.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:01.598 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:01.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[21463]: cluster 2024-04-26T16:31:59.760084+0000 mgr.y (mgr.24401) 568 : cluster [DBG] pgmap v891: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 315 B/s wr, 3 op/s 2024-04-26T16:32:01.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[21463]: cluster 2024-04-26T16:31:59.760084+0000 mgr.y (mgr.24401) 568 : cluster [DBG] pgmap v891: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 315 B/s wr, 3 op/s 2024-04-26T16:32:01.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[21463]: audit 2024-04-26T16:32:01.088957+0000 mon.c (mon.2) 666 : audit [INF] from='client.? 172.21.15.50:0/1645504570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[21463]: audit 2024-04-26T16:32:01.088957+0000 mon.c (mon.2) 666 : audit [INF] from='client.? 172.21.15.50:0/1645504570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[21463]: audit 2024-04-26T16:32:01.089423+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[21463]: audit 2024-04-26T16:32:01.089423+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[29234]: cluster 2024-04-26T16:31:59.760084+0000 mgr.y (mgr.24401) 568 : cluster [DBG] pgmap v891: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 315 B/s wr, 3 op/s 2024-04-26T16:32:01.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[29234]: cluster 2024-04-26T16:31:59.760084+0000 mgr.y (mgr.24401) 568 : cluster [DBG] pgmap v891: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 315 B/s wr, 3 op/s 2024-04-26T16:32:01.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[29234]: audit 2024-04-26T16:32:01.088957+0000 mon.c (mon.2) 666 : audit [INF] from='client.? 172.21.15.50:0/1645504570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[29234]: audit 2024-04-26T16:32:01.088957+0000 mon.c (mon.2) 666 : audit [INF] from='client.? 172.21.15.50:0/1645504570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[29234]: audit 2024-04-26T16:32:01.089423+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.611 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:01 smithi050 bash[29234]: audit 2024-04-26T16:32:01.089423+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.627 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:01.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:01.758 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:01.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:01 smithi190 bash[23868]: cluster 2024-04-26T16:31:59.760084+0000 mgr.y (mgr.24401) 568 : cluster [DBG] pgmap v891: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 315 B/s wr, 3 op/s 2024-04-26T16:32:01.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:01 smithi190 bash[23868]: cluster 2024-04-26T16:31:59.760084+0000 mgr.y (mgr.24401) 568 : cluster [DBG] pgmap v891: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 315 B/s wr, 3 op/s 2024-04-26T16:32:01.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:01 smithi190 bash[23868]: audit 2024-04-26T16:32:01.088957+0000 mon.c (mon.2) 666 : audit [INF] from='client.? 172.21.15.50:0/1645504570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:01 smithi190 bash[23868]: audit 2024-04-26T16:32:01.088957+0000 mon.c (mon.2) 666 : audit [INF] from='client.? 172.21.15.50:0/1645504570' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:01 smithi190 bash[23868]: audit 2024-04-26T16:32:01.089423+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:01 smithi190 bash[23868]: audit 2024-04-26T16:32:01.089423+0000 mon.a (mon.0) 3055 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:01.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:01.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:01.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:01.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:01.913 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:01.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:02.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:02.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:02.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:02.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:02.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:02.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:02.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:02.421 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:02.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:02.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:02.552 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[29234]: cluster 2024-04-26T16:32:01.347774+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[29234]: cluster 2024-04-26T16:32:01.347774+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[29234]: audit 2024-04-26T16:32:01.354206+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[29234]: audit 2024-04-26T16:32:01.354206+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[29234]: cluster 2024-04-26T16:32:01.361840+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e582: 8 total, 8 up, 7 in 2024-04-26T16:32:02.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[29234]: cluster 2024-04-26T16:32:01.361840+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e582: 8 total, 8 up, 7 in 2024-04-26T16:32:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[21463]: cluster 2024-04-26T16:32:01.347774+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[21463]: cluster 2024-04-26T16:32:01.347774+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[21463]: audit 2024-04-26T16:32:01.354206+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[21463]: audit 2024-04-26T16:32:01.354206+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[21463]: cluster 2024-04-26T16:32:01.361840+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e582: 8 total, 8 up, 7 in 2024-04-26T16:32:02.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:02 smithi050 bash[21463]: cluster 2024-04-26T16:32:01.361840+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e582: 8 total, 8 up, 7 in 2024-04-26T16:32:02.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:02.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:02.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:02.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:02 smithi190 bash[23868]: cluster 2024-04-26T16:32:01.347774+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:02.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:02 smithi190 bash[23868]: cluster 2024-04-26T16:32:01.347774+0000 mon.a (mon.0) 3056 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:02.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:02 smithi190 bash[23868]: audit 2024-04-26T16:32:01.354206+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:02.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:02 smithi190 bash[23868]: audit 2024-04-26T16:32:01.354206+0000 mon.a (mon.0) 3057 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:02.799 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:02 smithi190 bash[23868]: cluster 2024-04-26T16:32:01.361840+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e582: 8 total, 8 up, 7 in 2024-04-26T16:32:02.800 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:02 smithi190 bash[23868]: cluster 2024-04-26T16:32:01.361840+0000 mon.a (mon.0) 3058 : cluster [DBG] osdmap e582: 8 total, 8 up, 7 in 2024-04-26T16:32:02.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:02.828 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:02.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:02.941 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:02.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:03.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:03.064 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:03.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:03.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.331 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:03.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:03.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:03.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:03 smithi190 bash[23868]: cluster 2024-04-26T16:32:01.761785+0000 mgr.y (mgr.24401) 569 : cluster [DBG] pgmap v893: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 317 B/s wr, 4 op/s 2024-04-26T16:32:03.443 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:03 smithi190 bash[23868]: cluster 2024-04-26T16:32:01.761785+0000 mgr.y (mgr.24401) 569 : cluster [DBG] pgmap v893: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 317 B/s wr, 4 op/s 2024-04-26T16:32:03.458 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:03.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:03.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:03 smithi050 bash[21463]: cluster 2024-04-26T16:32:01.761785+0000 mgr.y (mgr.24401) 569 : cluster [DBG] pgmap v893: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 317 B/s wr, 4 op/s 2024-04-26T16:32:03.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:03 smithi050 bash[21463]: cluster 2024-04-26T16:32:01.761785+0000 mgr.y (mgr.24401) 569 : cluster [DBG] pgmap v893: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 317 B/s wr, 4 op/s 2024-04-26T16:32:03.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:03 smithi050 bash[29234]: cluster 2024-04-26T16:32:01.761785+0000 mgr.y (mgr.24401) 569 : cluster [DBG] pgmap v893: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 317 B/s wr, 4 op/s 2024-04-26T16:32:03.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:03 smithi050 bash[29234]: cluster 2024-04-26T16:32:01.761785+0000 mgr.y (mgr.24401) 569 : cluster [DBG] pgmap v893: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 317 B/s wr, 4 op/s 2024-04-26T16:32:03.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:03.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:03.716 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:03.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:03.845 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:03.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:03.978 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:03.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:03.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:04.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:04.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:04.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:04.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:04.234 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:04.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:04.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:32:04.365 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:04.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:04 smithi050 bash[21463]: cluster 2024-04-26T16:32:03.389764+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e583: 8 total, 8 up, 7 in 2024-04-26T16:32:04.410 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:04 smithi050 bash[21463]: cluster 2024-04-26T16:32:03.389764+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e583: 8 total, 8 up, 7 in 2024-04-26T16:32:04.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:04 smithi050 bash[29234]: cluster 2024-04-26T16:32:03.389764+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e583: 8 total, 8 up, 7 in 2024-04-26T16:32:04.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:04 smithi050 bash[29234]: cluster 2024-04-26T16:32:03.389764+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e583: 8 total, 8 up, 7 in 2024-04-26T16:32:04.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:04.476 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:32:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:32:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:32:04.497 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:04.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:04.610 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:04 smithi190 bash[23868]: cluster 2024-04-26T16:32:03.389764+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e583: 8 total, 8 up, 7 in 2024-04-26T16:32:04.610 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:04 smithi190 bash[23868]: cluster 2024-04-26T16:32:03.389764+0000 mon.a (mon.0) 3059 : cluster [DBG] osdmap e583: 8 total, 8 up, 7 in 2024-04-26T16:32:04.624 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:04.727 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:04.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:04.795 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:32:04.795 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:32:04.795 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:32:04.796 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:32:04.796 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:32:04.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:04.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:04.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:04.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:04.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:05.016 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:05.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:05.147 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:05.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:05.271 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:05.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:05.403 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:05.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:05.534 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:05.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-26T16:32:05.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:05 smithi050 bash[21463]: cluster 2024-04-26T16:32:03.762575+0000 mgr.y (mgr.24401) 570 : cluster [DBG] pgmap v895: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 272 B/s wr, 3 op/s 2024-04-26T16:32:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:05 smithi050 bash[21463]: cluster 2024-04-26T16:32:03.762575+0000 mgr.y (mgr.24401) 570 : cluster [DBG] pgmap v895: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 272 B/s wr, 3 op/s 2024-04-26T16:32:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:05 smithi050 bash[29234]: cluster 2024-04-26T16:32:03.762575+0000 mgr.y (mgr.24401) 570 : cluster [DBG] pgmap v895: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 272 B/s wr, 3 op/s 2024-04-26T16:32:05.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:05 smithi050 bash[29234]: cluster 2024-04-26T16:32:03.762575+0000 mgr.y (mgr.24401) 570 : cluster [DBG] pgmap v895: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 272 B/s wr, 3 op/s 2024-04-26T16:32:05.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:05.648 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:05.663 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:05.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:05.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:05.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:05 smithi190 bash[23868]: cluster 2024-04-26T16:32:03.762575+0000 mgr.y (mgr.24401) 570 : cluster [DBG] pgmap v895: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 272 B/s wr, 3 op/s 2024-04-26T16:32:05.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:05 smithi190 bash[23868]: cluster 2024-04-26T16:32:03.762575+0000 mgr.y (mgr.24401) 570 : cluster [DBG] pgmap v895: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 1.9 KiB/s rd, 272 B/s wr, 3 op/s 2024-04-26T16:32:05.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:05.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:05.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:06.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:06.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:06.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:06.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:06.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:06.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:06.424 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:06.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:06.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:06.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:06.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:06.813 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:06.841 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:06.944 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:06.965 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:06.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:07.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:07.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:07.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:07.226 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:07.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:32:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:32:07.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:07.356 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:07.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:07.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:07.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:07.590 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:07.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:07.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/1517721330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 msgr2=0x7f43ac06bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:07.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- 172.21.15.50:0/1517721330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac06bfb0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7f43a400b0a0 tx=0x7f43a400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:32:07.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/1517721330 shutdown_connections 2024-04-26T16:32:07.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- 172.21.15.50:0/1517721330 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac10fe70 0x7f43ac112260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:07.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- 172.21.15.50:0/1517721330 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac06bfb0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- 172.21.15.50:0/1517721330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ac06c4f0 0x7f43ac10f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/1517721330 >> 172.21.15.50:0/1517721330 conn(0x7f43ac0793c0 msgr2=0x7f43ac0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/1517721330 shutdown_connections 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/1517721330 wait complete. 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 librados: starting msgr at 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 librados: starting objecter 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 Processor -- start 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- start start 2024-04-26T16:32:07.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 librados: setting wanted keys 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 librados: calling monclient init 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac19b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 0x7f43ac19b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ac10fe70 0x7f43ac19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac114db0 con 0x7f43ac06bbd0 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43ac114c30 con 0x7f43ac10fe70 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f43ac114f30 con 0x7f43ac06c4f0 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43abfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac19b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 0x7f43ac19b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 0x7f43ac19b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:53634/0 (socket says 172.21.15.50:53634) 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 -- 172.21.15.50:0/2101205041 learned_addr learned my addr 172.21.15.50:0/2101205041 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43abfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac19b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34628/0 (socket says 172.21.15.50:34628) 2024-04-26T16:32:07.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b0e4a700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ac10fe70 0x7f43ac19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 -- 172.21.15.50:0/2101205041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ac10fe70 msgr2=0x7f43ac19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ac10fe70 0x7f43ac19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 -- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 msgr2=0x7f43ac19b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac19b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 -- 172.21.15.50:0/2101205041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43ac1a0350 con 0x7f43ac06c4f0 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43abfff700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac19b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43ab7fe700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 0x7f43ac19b880 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f439400d950 tx=0x7f439400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43a97fa700 10 client.?.objecter ms_handle_connect 0x7f43ac06c4f0 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43a97fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394016070 con 0x7f43ac06c4f0 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43ac1a0640 con 0x7f43ac06c4f0 2024-04-26T16:32:07.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4394009da0 con 0x7f43ac06c4f0 2024-04-26T16:32:07.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4394010e60 con 0x7f43ac06c4f0 2024-04-26T16:32:07.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 10 client.41417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:07.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f43ac1a7ec0 con 0x7f43ac06c4f0 2024-04-26T16:32:07.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 librados: init done 2024-04-26T16:32:07.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.673+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43ac06a110 con 0x7f43ac06c4f0 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4394019070 con 0x7f43ac06c4f0 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(583..583 src has 496..583) ==== 9433+0+0 (secure 0 0 0) 0x7f4394091640 con 0x7f43ac06c4f0 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 10 client.41417.objecter ms_dispatch 0x7f43ac001550 osd_map(583..583 src has 496..583) 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 3 client.41417.objecter handle_osd_map got epochs [583,583] > 0 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 3 client.41417.objecter handle_osd_map decoding full epoch 583 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43abfff700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 20 client.41417.objecter dump_active .. 0 homeless 2024-04-26T16:32:07.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43a97fa700 10 client.41417.objecter ms_handle_connect 0x7f4398071f80 2024-04-26T16:32:07.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.677+0000 7f43abfff700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009fd0 tx=0x7f43a4006210 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:07.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.681+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f439405cfc0 con 0x7f43ac06c4f0 2024-04-26T16:32:07.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:07.715 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:07 smithi050 bash[21463]: cluster 2024-04-26T16:32:05.763421+0000 mgr.y (mgr.24401) 571 : cluster [DBG] pgmap v896: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:32:07.715 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:07 smithi050 bash[21463]: cluster 2024-04-26T16:32:05.763421+0000 mgr.y (mgr.24401) 571 : cluster [DBG] pgmap v896: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:32:07.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:07 smithi050 bash[29234]: cluster 2024-04-26T16:32:05.763421+0000 mgr.y (mgr.24401) 571 : cluster [DBG] pgmap v896: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:32:07.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:07 smithi050 bash[29234]: cluster 2024-04-26T16:32:05.763421+0000 mgr.y (mgr.24401) 571 : cluster [DBG] pgmap v896: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:32:07.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:07 smithi190 bash[23868]: cluster 2024-04-26T16:32:05.763421+0000 mgr.y (mgr.24401) 571 : cluster [DBG] pgmap v896: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:32:07.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:07 smithi190 bash[23868]: cluster 2024-04-26T16:32:05.763421+0000 mgr.y (mgr.24401) 571 : cluster [DBG] pgmap v896: 228 pgs: 228 active+clean; 8.4 MiB data, 543 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:32:07.741 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:07.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:07.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:07.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:07.937+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd in", "ids": ["5"]} v 0) -- 0x7f43ac1a8090 con 0x7f43ac06c4f0 2024-04-26T16:32:07.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:07.976 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:07 smithi050 bash[21463]: debug 2024-04-26T16:32:07.941+0000 7f60c8fdd700 -1 mon.a@0(leader).osd e583 definitely_dead 0 2024-04-26T16:32:07.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:07.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:08.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:08.120 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:08.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:08.251 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:08.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:08.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:08.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.401+0000 7f43a97fa700 1 -- 172.21.15.50:0/2101205041 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd in", "ids": ["5"]}]=0 marked in osd.5. v584) ==== 85+0+0 (secure 0 0 0) 0x7f4394060c70 con 0x7f43ac06c4f0 2024-04-26T16:32:08.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.405+0000 7f43b28ad700 10 librados: watch_flush enter 2024-04-26T16:32:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 10 librados: watch_flush exit 2024-04-26T16:32:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 20 client.41417.objecter shutdown clearing up homeless session... 2024-04-26T16:32:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 10 client.41417.objecter successfully canceled tick 2024-04-26T16:32:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 msgr2=0x7f4398074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:08.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 crypto rx=0x7f43a4009fd0 tx=0x7f43a4006210 comp rx=0 tx=0).stop 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 msgr2=0x7f43ac19b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 0x7f43ac19b880 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f439400d950 tx=0x7f439400dc60 comp rx=0 tx=0).stop 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 shutdown_connections 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 unknown :-1 s=CLOSED pgs=3781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43ac10fe70 0x7f43ac19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43ac06bbd0 0x7f43ac19b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 --2- 172.21.15.50:0/2101205041 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43ac06c4f0 0x7f43ac19b880 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:08.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.409+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 >> 172.21.15.50:0/2101205041 conn(0x7f43ac0793c0 msgr2=0x7f43ac1110e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.413+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 shutdown_connections 2024-04-26T16:32:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.417+0000 7f43b28ad700 1 -- 172.21.15.50:0/2101205041 wait complete. 2024-04-26T16:32:08.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:08.417+0000 7f43b28ad700 1 librados: shutdown 2024-04-26T16:32:08.425 INFO:teuthology.orchestra.run.smithi050.stderr:marked in osd.5. 2024-04-26T16:32:08.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:08.505 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:08.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:08.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[21463]: audit 2024-04-26T16:32:07.054464+0000 mgr.y (mgr.24401) 572 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[21463]: audit 2024-04-26T16:32:07.054464+0000 mgr.y (mgr.24401) 572 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[21463]: audit 2024-04-26T16:32:07.943053+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? 172.21.15.50:0/2101205041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[21463]: audit 2024-04-26T16:32:07.943053+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? 172.21.15.50:0/2101205041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[21463]: audit 2024-04-26T16:32:07.943724+0000 mon.a (mon.0) 3060 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[21463]: audit 2024-04-26T16:32:07.943724+0000 mon.a (mon.0) 3060 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[29234]: audit 2024-04-26T16:32:07.054464+0000 mgr.y (mgr.24401) 572 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[29234]: audit 2024-04-26T16:32:07.054464+0000 mgr.y (mgr.24401) 572 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[29234]: audit 2024-04-26T16:32:07.943053+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? 172.21.15.50:0/2101205041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[29234]: audit 2024-04-26T16:32:07.943053+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? 172.21.15.50:0/2101205041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[29234]: audit 2024-04-26T16:32:07.943724+0000 mon.a (mon.0) 3060 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:08 smithi050 bash[29234]: audit 2024-04-26T16:32:07.943724+0000 mon.a (mon.0) 3060 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:08.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:08.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:08.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:08 smithi190 bash[23868]: audit 2024-04-26T16:32:07.054464+0000 mgr.y (mgr.24401) 572 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:08.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:08 smithi190 bash[23868]: audit 2024-04-26T16:32:07.054464+0000 mgr.y (mgr.24401) 572 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:08.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:08 smithi190 bash[23868]: audit 2024-04-26T16:32:07.943053+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? 172.21.15.50:0/2101205041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:08 smithi190 bash[23868]: audit 2024-04-26T16:32:07.943053+0000 mon.c (mon.2) 667 : audit [DBG] from='client.? 172.21.15.50:0/2101205041' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:08 smithi190 bash[23868]: audit 2024-04-26T16:32:07.943724+0000 mon.a (mon.0) 3060 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:08 smithi190 bash[23868]: audit 2024-04-26T16:32:07.943724+0000 mon.a (mon.0) 3060 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-04-26T16:32:08.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:08.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:08.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:09.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:09.033 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:09.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:09.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:09.264 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:09.291 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:09.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:09.441 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:09.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:09.568 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[21463]: cluster 2024-04-26T16:32:07.764470+0000 mgr.y (mgr.24401) 573 : cluster [DBG] pgmap v897: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[21463]: cluster 2024-04-26T16:32:07.764470+0000 mgr.y (mgr.24401) 573 : cluster [DBG] pgmap v897: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[21463]: audit 2024-04-26T16:32:08.400000+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[21463]: audit 2024-04-26T16:32:08.400000+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[21463]: cluster 2024-04-26T16:32:08.405189+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[21463]: cluster 2024-04-26T16:32:08.405189+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[29234]: cluster 2024-04-26T16:32:07.764470+0000 mgr.y (mgr.24401) 573 : cluster [DBG] pgmap v897: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:09.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[29234]: cluster 2024-04-26T16:32:07.764470+0000 mgr.y (mgr.24401) 573 : cluster [DBG] pgmap v897: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[29234]: audit 2024-04-26T16:32:08.400000+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-26T16:32:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[29234]: audit 2024-04-26T16:32:08.400000+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-26T16:32:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[29234]: cluster 2024-04-26T16:32:08.405189+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-04-26T16:32:09.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:09 smithi050 bash[29234]: cluster 2024-04-26T16:32:08.405189+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-04-26T16:32:09.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:09.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:09 smithi190 bash[23868]: cluster 2024-04-26T16:32:07.764470+0000 mgr.y (mgr.24401) 573 : cluster [DBG] pgmap v897: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:09.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:09 smithi190 bash[23868]: cluster 2024-04-26T16:32:07.764470+0000 mgr.y (mgr.24401) 573 : cluster [DBG] pgmap v897: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:09.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:09 smithi190 bash[23868]: audit 2024-04-26T16:32:08.400000+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-26T16:32:09.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:09 smithi190 bash[23868]: audit 2024-04-26T16:32:08.400000+0000 mon.a (mon.0) 3061 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-04-26T16:32:09.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:09 smithi190 bash[23868]: cluster 2024-04-26T16:32:08.405189+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-04-26T16:32:09.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:09 smithi190 bash[23868]: cluster 2024-04-26T16:32:08.405189+0000 mon.a (mon.0) 3062 : cluster [DBG] osdmap e584: 8 total, 8 up, 8 in 2024-04-26T16:32:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.698 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:09.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:09.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:09.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:09.930 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:09.957 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:09.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:09.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:10.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:10.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:10.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:10.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:10.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:10.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:10.349 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:10.363 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-04-26T16:32:10.376 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:10.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:10.475 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:10.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:10.606 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:10.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:10 smithi050 bash[21463]: cluster 2024-04-26T16:32:09.406632+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-04-26T16:32:10.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:10 smithi050 bash[21463]: cluster 2024-04-26T16:32:09.406632+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-04-26T16:32:10.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:10 smithi050 bash[29234]: cluster 2024-04-26T16:32:09.406632+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-04-26T16:32:10.613 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:10 smithi050 bash[29234]: cluster 2024-04-26T16:32:09.406632+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-04-26T16:32:10.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:10.739 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:10 smithi190 bash[23868]: cluster 2024-04-26T16:32:09.406632+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-04-26T16:32:10.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:10 smithi190 bash[23868]: cluster 2024-04-26T16:32:09.406632+0000 mon.a (mon.0) 3063 : cluster [DBG] osdmap e585: 8 total, 8 up, 8 in 2024-04-26T16:32:10.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:10.871 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:10.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:10.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:11.001 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:11.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:11.132 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:11.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:11.262 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:11.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:11.368 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:11.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:11.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:11.526 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:11.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:11 smithi190 bash[23868]: cluster 2024-04-26T16:32:09.765332+0000 mgr.y (mgr.24401) 574 : cluster [DBG] pgmap v900: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:11.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:11 smithi190 bash[23868]: cluster 2024-04-26T16:32:09.765332+0000 mgr.y (mgr.24401) 574 : cluster [DBG] pgmap v900: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:11.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:11 smithi190 bash[23868]: cluster 2024-04-26T16:32:10.430509+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-04-26T16:32:11.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:11 smithi190 bash[23868]: cluster 2024-04-26T16:32:10.430509+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-04-26T16:32:11.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:11 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.270611+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-04-26T16:32:11.532 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:11 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.270611+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-04-26T16:32:11.629 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:11.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[29234]: cluster 2024-04-26T16:32:09.765332+0000 mgr.y (mgr.24401) 574 : cluster [DBG] pgmap v900: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:11.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[29234]: cluster 2024-04-26T16:32:09.765332+0000 mgr.y (mgr.24401) 574 : cluster [DBG] pgmap v900: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:11.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[29234]: cluster 2024-04-26T16:32:10.430509+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-04-26T16:32:11.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[29234]: cluster 2024-04-26T16:32:10.430509+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-04-26T16:32:11.670 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.270611+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.270611+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[21463]: cluster 2024-04-26T16:32:09.765332+0000 mgr.y (mgr.24401) 574 : cluster [DBG] pgmap v900: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[21463]: cluster 2024-04-26T16:32:09.765332+0000 mgr.y (mgr.24401) 574 : cluster [DBG] pgmap v900: 228 pgs: 228 active+clean; 8.4 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[21463]: cluster 2024-04-26T16:32:10.430509+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[21463]: cluster 2024-04-26T16:32:10.430509+0000 mon.a (mon.0) 3064 : cluster [DBG] osdmap e586: 8 total, 8 up, 8 in 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.270611+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-04-26T16:32:11.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:11 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.270611+0000 mon.a (mon.0) 3065 : cluster [DBG] osdmap e587: 8 total, 8 up, 8 in 2024-04-26T16:32:11.687 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:11.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:11.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:11.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:11.945 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:11.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:12.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:12.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:12.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:12.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- 172.21.15.50:0/620004561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 msgr2=0x7fed401102f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- 172.21.15.50:0/620004561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401102f0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7fed380098f0 tx=0x7fed38009c00 comp rx=0 tx=0).stop 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- 172.21.15.50:0/620004561 shutdown_connections 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- 172.21.15.50:0/620004561 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401102f0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- 172.21.15.50:0/620004561 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed401094b0 0x7fed40109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- 172.21.15.50:0/620004561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40101a10 0x7fed40101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- 172.21.15.50:0/620004561 >> 172.21.15.50:0/620004561 conn(0x7fed40078990 msgr2=0x7fed400ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- 172.21.15.50:0/620004561 shutdown_connections 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- 172.21.15.50:0/620004561 wait complete. 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 librados: starting msgr at 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 librados: starting objecter 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 Processor -- start 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- start start 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 librados: setting wanted keys 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 librados: calling monclient init 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40101a10 0x7fed401990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed401094b0 0x7fed4019d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fed40112ae0 con 0x7fed401094b0 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fed40112960 con 0x7fed40101a10 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed45e84700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fed40112c60 con 0x7fed401023b0 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed3effd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed3ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed401094b0 0x7fed4019d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed3effd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39454/0 (socket says 172.21.15.50:39454) 2024-04-26T16:32:12.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed3effd700 1 -- 172.21.15.50:0/1166434102 learned_addr learned my addr 172.21.15.50:0/1166434102 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.353+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40101a10 0x7fed401990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3effd700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40101a10 msgr2=0x7fed401990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed40101a10 0x7fed401990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3effd700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed401094b0 msgr2=0x7fed4019d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fed401094b0 0x7fed4019d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3effd700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4019e0b0 con 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401995e0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fed2800c9b0 tx=0x7fed2800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 10 client.?.objecter ms_handle_connect 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28007b00 con 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4019e3a0 con 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fed28007ca0 con 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed28018400 con 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 10 client.41423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed401a5c20 con 0x7fed401023b0 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 1 librados: init done 2024-04-26T16:32:12.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fed20005100 con 0x7fed401023b0 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fed28004850 con 0x7fed401023b0 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(588..588 src has 496..588) ==== 9377+0+0 (secure 0 0 0) 0x7fed28091940 con 0x7fed401023b0 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 osd_map(588..588 src has 496..588) 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 3 client.41423.objecter handle_osd_map got epochs [588,588] > 0 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 3 client.41423.objecter handle_osd_map decoding full epoch 588 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 20 client.41423.objecter dump_active .. 0 homeless 2024-04-26T16:32:12.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v588) ==== 54+0+16 (secure 0 0 0) 0x7fed28014030 con 0x7fed401023b0 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 10 client.41423.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 1 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fed20004730 0x7fed20004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 1 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 0 1 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 4 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 10 client.41423.objecter _send_command 1 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed20008b70 con 0x7fed20004730 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed2c071f80 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7fed30009fd0 tx=0x7fed3000b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.357+0000 7fed3ffff700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fed20004730 0x7fed20004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3ffff700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fed20004730 0x7fed20004b70 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed20004730 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed20008b70 con 0x7fed20004730 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 1: 0 ) 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3cff9700 10 client.41423.objecter _finish_command 1 = system:0 2024-04-26T16:32:12.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 4 2024-04-26T16:32:12.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.361+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 0 1 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 10 client.41423.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 2 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 2 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed200053e0 osd=0 3 2024-04-26T16:32:12.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 0 2 2024-04-26T16:32:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 4 2024-04-26T16:32:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 10 client.41423.objecter _send_command 2 2024-04-26T16:32:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.377+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed20004240 con 0x7fed20004730 2024-04-26T16:32:12.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed20004240 con 0x7fed20004730 2024-04-26T16:32:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 10 client.41423.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 4 2024-04-26T16:32:12.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 0 2 2024-04-26T16:32:12.378 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 10 client.41423.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 3 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fed20002750 0x7fed20002bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 3 2024-04-26T16:32:12.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 1 3 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 4 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 10 client.41423.objecter _send_command 3 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fed200030f0 con 0x7fed20002750 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fed20002750 0x7fed20002bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed20002750 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fed20002750 0x7fed20002bb0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed200030f0 con 0x7fed20002750 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 3: 0 ) 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 10 client.41423.objecter _finish_command 3 = system:0 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 4 2024-04-26T16:32:12.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.381+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 1 3 2024-04-26T16:32:12.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 10 client.41423.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 4 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 4 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed200081c0 osd=1 3 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 1 4 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 4 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 10 client.41423.objecter _send_command 4 2024-04-26T16:32:12.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed20003ad0 con 0x7fed20002750 2024-04-26T16:32:12.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed20003ad0 con 0x7fed20002750 2024-04-26T16:32:12.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3cff9700 10 client.41423.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 4 2024-04-26T16:32:12.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 1 4 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 10 client.41423.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 5 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fed2000b240 0x7fed2000b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 5 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 2 5 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 4 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 10 client.41423.objecter _send_command 5 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fed2000d3f0 con 0x7fed2000b240 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fed2000b240 0x7fed2000b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed2000b240 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.401+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fed2000b240 0x7fed2000b680 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.401 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.405+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed2000d3f0 con 0x7fed2000b240 2024-04-26T16:32:12.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.405+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 5: 0 ) 2024-04-26T16:32:12.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.405+0000 7fed3cff9700 10 client.41423.objecter _finish_command 5 = system:0 2024-04-26T16:32:12.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.405+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 4 2024-04-26T16:32:12.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.405+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 2 5 2024-04-26T16:32:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 10 client.41423.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 6 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 6 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed20003d30 osd=2 3 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 2 6 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 4 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 10 client.41423.objecter _send_command 6 2024-04-26T16:32:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.421+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed20003550 con 0x7fed2000b240 2024-04-26T16:32:12.424 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:32:12.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed20003550 con 0x7fed2000b240 2024-04-26T16:32:12.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed3cff9700 10 client.41423.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 4 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 2 6 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 10 client.41423.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 7 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fed20002090 0x7fed20015a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 7 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 3 7 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 4 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 10 client.41423.objecter _send_command 7 2024-04-26T16:32:12.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.425+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fed2000d050 con 0x7fed20002090 2024-04-26T16:32:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3ffff700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fed20002090 0x7fed20015a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3ffff700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fed20002090 0x7fed20015a80 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed20002090 2024-04-26T16:32:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed2000d050 con 0x7fed20002090 2024-04-26T16:32:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 7: 0 ) 2024-04-26T16:32:12.426 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3cff9700 10 client.41423.objecter _finish_command 7 = system:0 2024-04-26T16:32:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 4 2024-04-26T16:32:12.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.429+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 3 7 2024-04-26T16:32:12.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 10 client.41423.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 8 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 8 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2000cd40 osd=3 3 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 3 8 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 4 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 10 client.41423.objecter _send_command 8 2024-04-26T16:32:12.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.449+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed20001600 con 0x7fed20002090 2024-04-26T16:32:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed20001600 con 0x7fed20002090 2024-04-26T16:32:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 10 client.41423.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 4 2024-04-26T16:32:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 3 8 2024-04-26T16:32:12.449 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:32:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 10 client.41423.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 9 2024-04-26T16:32:12.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fed2000c8a0 0x7fed20019150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 9 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 4 9 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 4 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 10 client.41423.objecter _send_command 9 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fed2001c000 con 0x7fed2000c8a0 2024-04-26T16:32:12.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fed2000c8a0 0x7fed20019150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed2000c8a0 2024-04-26T16:32:12.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fed2000c8a0 0x7fed20019150 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed2001c000 con 0x7fed2000c8a0 2024-04-26T16:32:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 9: 0 ) 2024-04-26T16:32:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 10 client.41423.objecter _finish_command 9 = system:0 2024-04-26T16:32:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 4 2024-04-26T16:32:12.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.453+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 4 9 2024-04-26T16:32:12.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.915678+0000 osd.4 (osd.4) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:12.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.915678+0000 osd.4 (osd.4) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:12.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.915949+0000 osd.4 (osd.4) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.915949+0000 osd.4 (osd.4) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: audit 2024-04-26T16:32:12.254473+0000 mon.c (mon.2) 668 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: audit 2024-04-26T16:32:12.254473+0000 mon.c (mon.2) 668 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: cluster 2024-04-26T16:32:12.268560+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: cluster 2024-04-26T16:32:12.268560+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: audit 2024-04-26T16:32:12.361693+0000 mon.c (mon.2) 669 : audit [DBG] from='client.? 172.21.15.50:0/1166434102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[29234]: audit 2024-04-26T16:32:12.361693+0000 mon.c (mon.2) 669 : audit [DBG] from='client.? 172.21.15.50:0/1166434102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.915678+0000 osd.4 (osd.4) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.915678+0000 osd.4 (osd.4) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.915949+0000 osd.4 (osd.4) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.915949+0000 osd.4 (osd.4) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:12.455 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: audit 2024-04-26T16:32:12.254473+0000 mon.c (mon.2) 668 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:12.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: audit 2024-04-26T16:32:12.254473+0000 mon.c (mon.2) 668 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:12.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: cluster 2024-04-26T16:32:12.268560+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-04-26T16:32:12.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: cluster 2024-04-26T16:32:12.268560+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-04-26T16:32:12.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: audit 2024-04-26T16:32:12.361693+0000 mon.c (mon.2) 669 : audit [DBG] from='client.? 172.21.15.50:0/1166434102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:12.456 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:12 smithi050 bash[21463]: audit 2024-04-26T16:32:12.361693+0000 mon.c (mon.2) 669 : audit [DBG] from='client.? 172.21.15.50:0/1166434102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:12.456 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 10 client.41423.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 10 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 10 2024-04-26T16:32:12.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed20001850 osd=4 3 2024-04-26T16:32:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 4 10 2024-04-26T16:32:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 4 2024-04-26T16:32:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 10 client.41423.objecter _send_command 10 2024-04-26T16:32:12.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.473+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed2001c000 con 0x7fed2000c8a0 2024-04-26T16:32:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed2001c000 con 0x7fed2000c8a0 2024-04-26T16:32:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 10 client.41423.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 4 2024-04-26T16:32:12.474 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 4 10 2024-04-26T16:32:12.474 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 10 client.41423.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 11 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fed20019ba0 0x7fed2001a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 11 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 5 11 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 4 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 10 client.41423.objecter _send_command 11 2024-04-26T16:32:12.475 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fed2001f3d0 con 0x7fed20019ba0 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fed20019ba0 0x7fed2001a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed20019ba0 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3effd700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fed20019ba0 0x7fed2001a000 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed2001f3d0 con 0x7fed20019ba0 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 11: 0 ) 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 10 client.41423.objecter _finish_command 11 = system:0 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 4 2024-04-26T16:32:12.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.477+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 5 11 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 10 client.41423.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 12 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 12 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2001c000 osd=5 3 2024-04-26T16:32:12.495 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 5 12 2024-04-26T16:32:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 4 2024-04-26T16:32:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 10 client.41423.objecter _send_command 12 2024-04-26T16:32:12.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.497+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed2001a650 con 0x7fed20019ba0 2024-04-26T16:32:12.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed2001a650 con 0x7fed20019ba0 2024-04-26T16:32:12.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3cff9700 10 client.41423.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 4 2024-04-26T16:32:12.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 5 12 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 10 client.41423.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 13 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fed2001ad20 0x7fed2001b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 13 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 6 13 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 4 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 10 client.41423.objecter _send_command 13 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fed2001b8b0 con 0x7fed2001ad20 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3ffff700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fed2001ad20 0x7fed2001b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed2001ad20 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.501+0000 7fed3ffff700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fed2001ad20 0x7fed2001b180 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.505+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed2001b8b0 con 0x7fed2001ad20 2024-04-26T16:32:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.505+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 13: 0 ) 2024-04-26T16:32:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.505+0000 7fed3cff9700 10 client.41423.objecter _finish_command 13 = system:0 2024-04-26T16:32:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.505+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 4 2024-04-26T16:32:12.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.505+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 6 13 2024-04-26T16:32:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 10 client.41423.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 14 2024-04-26T16:32:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.520 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 14 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2001a650 osd=6 3 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 6 14 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 4 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 10 client.41423.objecter _send_command 14 2024-04-26T16:32:12.521 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.521+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed2001b8b0 con 0x7fed2001ad20 2024-04-26T16:32:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed2001b8b0 con 0x7fed2001ad20 2024-04-26T16:32:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3cff9700 10 client.41423.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 4 2024-04-26T16:32:12.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 6 14 2024-04-26T16:32:12.523 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:32:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 10 client.41423.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 15 2024-04-26T16:32:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fed20020770 0x7fed20022b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:12.523 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 15 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 7 15 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 4 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 10 client.41423.objecter _send_command 15 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fed200259c0 con 0x7fed20020770 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fed20020770 0x7fed20022b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3cff9700 10 client.41423.objecter ms_handle_connect 0x7fed20020770 2024-04-26T16:32:12.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.525+0000 7fed3f7fe700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fed20020770 0x7fed20022b60 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.529+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fed200259c0 con 0x7fed20020770 2024-04-26T16:32:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.529+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 15: 0 ) 2024-04-26T16:32:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.529+0000 7fed3cff9700 10 client.41423.objecter _finish_command 15 = system:0 2024-04-26T16:32:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.529+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 4 2024-04-26T16:32:12.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.529+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 7 15 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 10 client.41423.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign -1 16 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 20 client.41423.objecter _get_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 15 client.41423.objecter _session_command_op_remove -1 16 2024-04-26T16:32:12.543 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 20 client.41423.objecter get_session s=0x7fed2001b8b0 osd=7 3 2024-04-26T16:32:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 15 client.41423.objecter _session_command_op_assign 7 16 2024-04-26T16:32:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 4 2024-04-26T16:32:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 10 client.41423.objecter _send_command 16 2024-04-26T16:32:12.544 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.545+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fed200230a0 con 0x7fed20020770 2024-04-26T16:32:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3cff9700 1 -- 172.21.15.50:0/1166434102 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fed200230a0 con 0x7fed20020770 2024-04-26T16:32:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3cff9700 10 client.41423.objecter ms_dispatch 0x7fed40001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3cff9700 10 client.41423.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3cff9700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 4 2024-04-26T16:32:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3cff9700 15 client.41423.objecter _session_command_op_remove 7 16 2024-04-26T16:32:12.551 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 librados: watch_flush enter 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 librados: watch_flush exit 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.0 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fed20004730 msgr2=0x7fed20004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fed20004730 0x7fed20004b70 crc :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200053e0 osd=0 1 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.1 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fed20002750 msgr2=0x7fed20002bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fed20002750 0x7fed20002bb0 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed200081c0 osd=1 1 2024-04-26T16:32:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.2 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fed2000b240 msgr2=0x7fed2000b680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fed2000b240 0x7fed2000b680 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20003d30 osd=2 1 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.3 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fed20002090 msgr2=0x7fed20015a80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3ffff700 1 -- 172.21.15.50:0/1166434102 reap_dead start 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fed20002090 0x7fed20015a80 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2000cd40 osd=3 1 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.4 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fed2000c8a0 msgr2=0x7fed20019150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fed2000c8a0 0x7fed20019150 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed20001850 osd=4 1 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.5 2024-04-26T16:32:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fed20019ba0 msgr2=0x7fed2001a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fed20019ba0 0x7fed2001a000 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001c000 osd=5 1 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.6 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fed2001ad20 msgr2=0x7fed2001b180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fed2001ad20 0x7fed2001b180 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001a650 osd=6 1 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter close_session for osd.7 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fed20020770 msgr2=0x7fed20022b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fed20020770 0x7fed20022b60 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter put_session s=0x7fed2001b8b0 osd=7 1 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 20 client.41423.objecter shutdown clearing up homeless session... 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 10 client.41423.objecter successfully canceled tick 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed3ffff700 1 -- 172.21.15.50:0/1166434102 reap_dead start 2024-04-26T16:32:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 msgr2=0x7fed2c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7fed30009fd0 tx=0x7fed3000b040 comp rx=0 tx=0).stop 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 msgr2=0x7fed401995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401995e0 secure :-1 s=READY pgs=2705 cs=0 l=1 rev1=1 crypto rx=0x7fed2800c9b0 tx=0x7fed2800ccc0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 shutdown_connections 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fed401023b0 0x7fed401995e0 unknown :-1 s=CLOSED pgs=2705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 --2- 172.21.15.50:0/1166434102 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fed2c071f80 0x7fed2c074440 unknown :-1 s=CLOSED pgs=3782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 >> 172.21.15.50:0/1166434102 conn(0x7fed40078990 msgr2=0x7fed40105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 shutdown_connections 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 -- 172.21.15.50:0/1166434102 wait complete. 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:12.549+0000 7fed45e84700 1 librados: shutdown 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:32:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:32:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:32:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:32:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:32:12.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:12.587 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:12.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.915678+0000 osd.4 (osd.4) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.915678+0000 osd.4 (osd.4) 303 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.915949+0000 osd.4 (osd.4) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.915949+0000 osd.4 (osd.4) 304 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: audit 2024-04-26T16:32:12.254473+0000 mon.c (mon.2) 668 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: audit 2024-04-26T16:32:12.254473+0000 mon.c (mon.2) 668 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: cluster 2024-04-26T16:32:12.268560+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-04-26T16:32:12.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: cluster 2024-04-26T16:32:12.268560+0000 mon.a (mon.0) 3066 : cluster [DBG] osdmap e588: 8 total, 8 up, 8 in 2024-04-26T16:32:12.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: audit 2024-04-26T16:32:12.361693+0000 mon.c (mon.2) 669 : audit [DBG] from='client.? 172.21.15.50:0/1166434102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:12.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:12 smithi190 bash[23868]: audit 2024-04-26T16:32:12.361693+0000 mon.c (mon.2) 669 : audit [DBG] from='client.? 172.21.15.50:0/1166434102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:12.714 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:12.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:12.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:12.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:12.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:13.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:13.113 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:13.216 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:13.245 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:13.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:13.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:13.376 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:13.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:13.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:13 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.767180+0000 mgr.y (mgr.24401) 575 : cluster [DBG] pgmap v903: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s, 72 objects/s recovering 2024-04-26T16:32:13.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:13 smithi050 bash[21463]: cluster 2024-04-26T16:32:11.767180+0000 mgr.y (mgr.24401) 575 : cluster [DBG] pgmap v903: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s, 72 objects/s recovering 2024-04-26T16:32:13.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:13 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.767180+0000 mgr.y (mgr.24401) 575 : cluster [DBG] pgmap v903: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s, 72 objects/s recovering 2024-04-26T16:32:13.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:13 smithi050 bash[29234]: cluster 2024-04-26T16:32:11.767180+0000 mgr.y (mgr.24401) 575 : cluster [DBG] pgmap v903: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s, 72 objects/s recovering 2024-04-26T16:32:13.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:13.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:13.623 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:13 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.767180+0000 mgr.y (mgr.24401) 575 : cluster [DBG] pgmap v903: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s, 72 objects/s recovering 2024-04-26T16:32:13.623 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:13 smithi190 bash[23868]: cluster 2024-04-26T16:32:11.767180+0000 mgr.y (mgr.24401) 575 : cluster [DBG] pgmap v903: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.0 MiB/s, 72 objects/s recovering 2024-04-26T16:32:13.635 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:13.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:13.732 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:32:13.732 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:32:13.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:13.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:13.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:13.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:13.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:13.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:14.027 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:14.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:14.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:14.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:14.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:14.283 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:14.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:14.419 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:14.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-26T16:32:14.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.435525+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.435525+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.445379+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.445379+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.446747+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.446747+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.447353+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:14 smithi190 bash[23868]: audit 2024-04-26T16:32:13.447353+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.563 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:14.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:14.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:14.688 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:14.791 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:14.802 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:32:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.435525+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.435525+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.445379+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.445379+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.446747+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.446747+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.447353+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.806 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[29234]: audit 2024-04-26T16:32:13.447353+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.435525+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.435525+0000 mon.b (mon.1) 406 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.445379+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.445379+0000 mon.a (mon.0) 3067 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.446747+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.446747+0000 mon.b (mon.1) 407 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.447353+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:14 smithi050 bash[21463]: audit 2024-04-26T16:32:13.447353+0000 mon.a (mon.0) 3068 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-94"}]: dispatch 2024-04-26T16:32:14.817 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:14.921 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:14.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:14.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:14.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:15.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.080 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:15.183 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:15.316 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:15.363 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-04-26T16:32:15.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:32:15.364 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-26T16:32:15.364 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-26T16:32:15.364 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-26T16:32:15.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-04-26T16:32:15.447 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:15.483 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:15.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:13.767874+0000 mgr.y (mgr.24401) 576 : cluster [DBG] pgmap v905: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 941 KiB/s, 75 objects/s recovering 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:13.767874+0000 mgr.y (mgr.24401) 576 : cluster [DBG] pgmap v905: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 941 KiB/s, 75 objects/s recovering 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:14.454543+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:14.454543+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.256370+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.256370+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.256645+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.256645+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:13.767874+0000 mgr.y (mgr.24401) 576 : cluster [DBG] pgmap v905: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 941 KiB/s, 75 objects/s recovering 2024-04-26T16:32:15.607 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:13.767874+0000 mgr.y (mgr.24401) 576 : cluster [DBG] pgmap v905: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 941 KiB/s, 75 objects/s recovering 2024-04-26T16:32:15.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:14.454543+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-04-26T16:32:15.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:14.454543+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-04-26T16:32:15.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.256370+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:15.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.256370+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:15.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.256645+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:15.608 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:15 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.256645+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:15.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.609 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:15.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:15.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:15.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:15.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/3178030037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1094e0 msgr2=0x7fdaec1098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:15.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- 172.21.15.50:0/3178030037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1094e0 0x7fdaec1098c0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009f90 tx=0x7fdae800b330 comp rx=0 tx=0).stop 2024-04-26T16:32:15.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/3178030037 shutdown_connections 2024-04-26T16:32:15.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- 172.21.15.50:0/3178030037 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaec103e60 0x7fdaec1102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:15.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- 172.21.15.50:0/3178030037 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1094e0 0x7fdaec1098c0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:15.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- 172.21.15.50:0/3178030037 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1034a0 0x7fdaec103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:15.766 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/3178030037 >> 172.21.15.50:0/3178030037 conn(0x7fdaec07a590 msgr2=0x7fdaec07a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:15.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/3178030037 shutdown_connections 2024-04-26T16:32:15.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/3178030037 wait complete. 2024-04-26T16:32:15.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 librados: starting msgr at 2024-04-26T16:32:15.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 librados: starting objecter 2024-04-26T16:32:15.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 Processor -- start 2024-04-26T16:32:15.767 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- start start 2024-04-26T16:32:15.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 librados: setting wanted keys 2024-04-26T16:32:15.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 librados: calling monclient init 2024-04-26T16:32:15.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 0x7fdaec199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:15.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaec103e60 0x7fdaec199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:15.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 0x7fdaec19da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:15.768 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec112a70 con 0x7fdaec1034a0 2024-04-26T16:32:15.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdaec1128f0 con 0x7fdaec103e60 2024-04-26T16:32:15.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fdaec112bf0 con 0x7fdaec1094e0 2024-04-26T16:32:15.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf26eb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 0x7fdaec19da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:15.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 0x7fdaec199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:15.769 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf26eb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 0x7fdaec19da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39470/0 (socket says 172.21.15.50:39470) 2024-04-26T16:32:15.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf26eb700 1 -- 172.21.15.50:0/1543608106 learned_addr learned my addr 172.21.15.50:0/1543608106 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:15.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 0x7fdaec199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37186/0 (socket says 172.21.15.50:37186) 2024-04-26T16:32:15.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf16e9700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaec103e60 0x7fdaec199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:15.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 -- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 msgr2=0x7fdaec19da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:15.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 0x7fdaec19da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:15.770 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 -- 172.21.15.50:0/1543608106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaec103e60 msgr2=0x7fdaec199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:15.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaec103e60 0x7fdaec199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:15.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 -- 172.21.15.50:0/1543608106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaec19e160 con 0x7fdaec1034a0 2024-04-26T16:32:15.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf26eb700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 0x7fdaec19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:32:15.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf1eea700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 0x7fdaec199150 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fdae8000f80 tx=0x7fdae8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:15.771 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdae2ffd700 10 client.?.objecter ms_handle_connect 0x7fdaec1034a0 2024-04-26T16:32:15.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdae2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:15.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8021070 con 0x7fdaec1034a0 2024-04-26T16:32:15.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fdae8007910 con 0x7fdaec1034a0 2024-04-26T16:32:15.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaec19e3f0 con 0x7fdaec1034a0 2024-04-26T16:32:15.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8005040 con 0x7fdaec1034a0 2024-04-26T16:32:15.772 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 10 client.30813.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:15.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaec1a5cd0 con 0x7fdaec1034a0 2024-04-26T16:32:15.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.765+0000 7fdaf414e700 1 librados: init done 2024-04-26T16:32:15.773 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaec06a110 con 0x7fdaec1034a0 2024-04-26T16:32:15.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fdae8024070 con 0x7fdaec1034a0 2024-04-26T16:32:15.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdad8071f80 0x7fdad8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:15.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(590..590 src has 496..590) ==== 9364+0+0 (secure 0 0 0) 0x7fdae800f020 con 0x7fdaec1034a0 2024-04-26T16:32:15.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 10 client.30813.objecter ms_dispatch 0x7fdaec001550 osd_map(590..590 src has 496..590) 2024-04-26T16:32:15.775 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 3 client.30813.objecter handle_osd_map got epochs [590,590] > 0 2024-04-26T16:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 3 client.30813.objecter handle_osd_map decoding full epoch 590 2024-04-26T16:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdaf16e9700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdad8071f80 0x7fdad8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 20 client.30813.objecter dump_active .. 0 homeless 2024-04-26T16:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdaf16e9700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdad8071f80 0x7fdad8074440 secure :-1 s=READY pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009750 tx=0x7fdadc009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.769+0000 7fdae2ffd700 10 client.30813.objecter ms_handle_connect 0x7fdad8071f80 2024-04-26T16:32:15.776 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:15.773+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fdae802b020 con 0x7fdaec1034a0 2024-04-26T16:32:15.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:13.767874+0000 mgr.y (mgr.24401) 576 : cluster [DBG] pgmap v905: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 941 KiB/s, 75 objects/s recovering 2024-04-26T16:32:15.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:13.767874+0000 mgr.y (mgr.24401) 576 : cluster [DBG] pgmap v905: 228 pgs: 1 peering, 227 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 941 KiB/s, 75 objects/s recovering 2024-04-26T16:32:15.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:14.454543+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-04-26T16:32:15.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:14.454543+0000 mon.a (mon.0) 3069 : cluster [DBG] osdmap e589: 8 total, 8 up, 8 in 2024-04-26T16:32:15.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.256370+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:15.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.256370+0000 osd.6 (osd.6) 78 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:15.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.256645+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:15.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:15 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.256645+0000 osd.6 (osd.6) 79 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:15.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:15.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:15.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.989 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:15.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:16.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.013+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fdaec1a5ea0 con 0x7fdaec1034a0 2024-04-26T16:32:16.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:16.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:32:16.122 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:16.224 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:16.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:16.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.261+0000 7fdae2ffd700 1 -- 172.21.15.50:0/1543608106 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v591) ==== 91+0+0 (secure 0 0 0) 0x7fdae805e920 con 0x7fdaec1034a0 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 10 librados: watch_flush enter 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 10 librados: watch_flush exit 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 20 client.30813.objecter shutdown clearing up homeless session... 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 10 client.30813.objecter successfully canceled tick 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdad8071f80 msgr2=0x7fdad8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdad8071f80 0x7fdad8074440 secure :-1 s=READY pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009750 tx=0x7fdadc009310 comp rx=0 tx=0).stop 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 msgr2=0x7fdaec199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 0x7fdaec199150 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7fdae8000f80 tx=0x7fdae8004c80 comp rx=0 tx=0).stop 2024-04-26T16:32:16.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 shutdown_connections 2024-04-26T16:32:16.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdad8071f80 0x7fdad8074440 secure :-1 s=CLOSED pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009750 tx=0x7fdadc009310 comp rx=0 tx=0).stop 2024-04-26T16:32:16.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdaec1094e0 0x7fdaec19da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:16.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdaec1034a0 0x7fdaec199150 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:16.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 --2- 172.21.15.50:0/1543608106 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdaec103e60 0x7fdaec199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:16.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 >> 172.21.15.50:0/1543608106 conn(0x7fdaec07a590 msgr2=0x7fdaec101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:16.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.265+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 shutdown_connections 2024-04-26T16:32:16.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.269+0000 7fdaf414e700 1 -- 172.21.15.50:0/1543608106 wait complete. 2024-04-26T16:32:16.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:16.273+0000 7fdaf414e700 1 librados: shutdown 2024-04-26T16:32:16.277 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:32:16.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:16.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:16.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:16.499 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:16.500 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.466756+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.466756+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:15.478346+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:15.478346+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:15.479817+0000 mon.a (mon.0) 3071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:15.479817+0000 mon.a (mon.0) 3071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.017702+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.017702+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.262898+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.262898+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.264682+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.264682+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.264822+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:32:16.501 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.264822+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.271373+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.271373+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.302367+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[29234]: audit 2024-04-26T16:32:16.302367+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.466756+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.466756+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:15.478346+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:15.478346+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:15.479817+0000 mon.a (mon.0) 3071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:15.479817+0000 mon.a (mon.0) 3071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.017702+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.017702+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.262898+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.262898+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.264682+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.264682+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.264822+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.264822+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.271373+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.271373+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.302367+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:16.503 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:16 smithi050 bash[21463]: audit 2024-04-26T16:32:16.302367+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:16.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:16.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:16.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:16.763 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:16.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.466756+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-04-26T16:32:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.466756+0000 mon.a (mon.0) 3070 : cluster [DBG] osdmap e590: 8 total, 8 up, 8 in 2024-04-26T16:32:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:15.478346+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:15.478346+0000 mon.b (mon.1) 408 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:15.479817+0000 mon.a (mon.0) 3071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:15.479817+0000 mon.a (mon.0) 3071 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.017702+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.017702+0000 mon.a (mon.0) 3072 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.262898+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.262898+0000 mon.a (mon.0) 3073 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.264682+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.264682+0000 mon.a (mon.0) 3074 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.264822+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.264822+0000 mon.a (mon.0) 3075 : audit [INF] from='client.? 172.21.15.50:0/1543608106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.271373+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.271373+0000 mon.a (mon.0) 3076 : cluster [DBG] osdmap e591: 8 total, 8 up, 8 in 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.302367+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:16.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:16 smithi190 bash[23868]: audit 2024-04-26T16:32:16.302367+0000 mon.b (mon.1) 409 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:16.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:16.894 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:16.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:16.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:17.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:17.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:17.140 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:32:17.157 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:17.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:17.286 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:17.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:17.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:17.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:17.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:17.548 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:17.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.768333+0000 mgr.y (mgr.24401) 577 : cluster [DBG] pgmap v908: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 912 KiB/s, 72 objects/s recovering 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.768333+0000 mgr.y (mgr.24401) 577 : cluster [DBG] pgmap v908: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 912 KiB/s, 72 objects/s recovering 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.820449+0000 osd.1 (osd.1) 319 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.820449+0000 osd.1 (osd.1) 319 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.820582+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:15.820582+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:17.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.338242+0000 osd.7 (osd.7) 319 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.338242+0000 osd.7 (osd.7) 319 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.340011+0000 osd.7 (osd.7) 320 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.340011+0000 osd.7 (osd.7) 320 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.461417+0000 mon.a (mon.0) 3077 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:16.461417+0000 mon.a (mon.0) 3077 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.178978+0000 osd.6 (osd.6) 80 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.178978+0000 osd.6 (osd.6) 80 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.181362+0000 osd.6 (osd.6) 81 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:32:17.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.181362+0000 osd.6 (osd.6) 81 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:32:17.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.221246+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:32:17.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.221246+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:32:17.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.223537+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:32:17.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.223537+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:32:17.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.286200+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-04-26T16:32:17.666 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.286200+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.768333+0000 mgr.y (mgr.24401) 577 : cluster [DBG] pgmap v908: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 912 KiB/s, 72 objects/s recovering 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.768333+0000 mgr.y (mgr.24401) 577 : cluster [DBG] pgmap v908: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 912 KiB/s, 72 objects/s recovering 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.820449+0000 osd.1 (osd.1) 319 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.820449+0000 osd.1 (osd.1) 319 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.820582+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:15.820582+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.338242+0000 osd.7 (osd.7) 319 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.338242+0000 osd.7 (osd.7) 319 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.340011+0000 osd.7 (osd.7) 320 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.340011+0000 osd.7 (osd.7) 320 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.461417+0000 mon.a (mon.0) 3077 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:17.667 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:16.461417+0000 mon.a (mon.0) 3077 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.178978+0000 osd.6 (osd.6) 80 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.178978+0000 osd.6 (osd.6) 80 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.181362+0000 osd.6 (osd.6) 81 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.181362+0000 osd.6 (osd.6) 81 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.221246+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.221246+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.223537+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.223537+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.286200+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-04-26T16:32:17.668 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:17 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.286200+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-04-26T16:32:17.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:17.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:17.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.768333+0000 mgr.y (mgr.24401) 577 : cluster [DBG] pgmap v908: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 912 KiB/s, 72 objects/s recovering 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.768333+0000 mgr.y (mgr.24401) 577 : cluster [DBG] pgmap v908: 228 pgs: 32 unknown, 1 peering, 195 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 912 KiB/s, 72 objects/s recovering 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.820449+0000 osd.1 (osd.1) 319 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.820449+0000 osd.1 (osd.1) 319 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.820582+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:15.820582+0000 osd.1 (osd.1) 320 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.338242+0000 osd.7 (osd.7) 319 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.338242+0000 osd.7 (osd.7) 319 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.340011+0000 osd.7 (osd.7) 320 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.340011+0000 osd.7 (osd.7) 320 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.461417+0000 mon.a (mon.0) 3077 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:17.795 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:16.461417+0000 mon.a (mon.0) 3077 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.178978+0000 osd.6 (osd.6) 80 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.178978+0000 osd.6 (osd.6) 80 : cluster [DBG] 109.1c scrub starts 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.181362+0000 osd.6 (osd.6) 81 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.181362+0000 osd.6 (osd.6) 81 : cluster [DBG] 109.1c scrub ok 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.221246+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.221246+0000 osd.4 (osd.4) 305 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.223537+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.223537+0000 osd.4 (osd.4) 306 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.286200+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-04-26T16:32:17.796 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:17 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.286200+0000 mon.a (mon.0) 3078 : cluster [DBG] osdmap e592: 8 total, 8 up, 8 in 2024-04-26T16:32:17.809 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:17.913 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:17.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:17.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:18.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:18.071 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:18.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:18.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:18.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:18.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:18.332 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:18.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:18.464 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: audit 2024-04-26T16:32:17.065958+0000 mgr.y (mgr.24401) 578 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: audit 2024-04-26T16:32:17.065958+0000 mgr.y (mgr.24401) 578 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.352762+0000 osd.7 (osd.7) 321 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.352762+0000 osd.7 (osd.7) 321 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.355942+0000 osd.7 (osd.7) 322 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.355942+0000 osd.7 (osd.7) 322 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.569994+0000 osd.2 (osd.2) 289 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.569994+0000 osd.2 (osd.2) 289 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.571286+0000 osd.2 (osd.2) 290 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:32:18.484 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.571286+0000 osd.2 (osd.2) 290 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.755457+0000 osd.0 (osd.0) 351 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.755457+0000 osd.0 (osd.0) 351 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.757200+0000 osd.0 (osd.0) 352 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.757200+0000 osd.0 (osd.0) 352 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.823613+0000 osd.3 (osd.3) 359 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.823613+0000 osd.3 (osd.3) 359 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.828707+0000 osd.3 (osd.3) 360 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.828707+0000 osd.3 (osd.3) 360 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: audit 2024-04-26T16:32:17.065958+0000 mgr.y (mgr.24401) 578 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: audit 2024-04-26T16:32:17.065958+0000 mgr.y (mgr.24401) 578 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.352762+0000 osd.7 (osd.7) 321 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:32:18.485 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.352762+0000 osd.7 (osd.7) 321 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.355942+0000 osd.7 (osd.7) 322 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.355942+0000 osd.7 (osd.7) 322 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.569994+0000 osd.2 (osd.2) 289 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.569994+0000 osd.2 (osd.2) 289 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.571286+0000 osd.2 (osd.2) 290 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.571286+0000 osd.2 (osd.2) 290 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.755457+0000 osd.0 (osd.0) 351 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.755457+0000 osd.0 (osd.0) 351 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.757200+0000 osd.0 (osd.0) 352 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:32:18.486 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.757200+0000 osd.0 (osd.0) 352 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:32:18.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.597 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:18.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:18.698 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:18.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:18.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.823613+0000 osd.3 (osd.3) 359 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:32:18.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.823613+0000 osd.3 (osd.3) 359 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:32:18.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.828707+0000 osd.3 (osd.3) 360 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:32:18.729 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:18 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.828707+0000 osd.3 (osd.3) 360 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:32:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: audit 2024-04-26T16:32:17.065958+0000 mgr.y (mgr.24401) 578 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: audit 2024-04-26T16:32:17.065958+0000 mgr.y (mgr.24401) 578 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.352762+0000 osd.7 (osd.7) 321 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.352762+0000 osd.7 (osd.7) 321 : cluster [DBG] 109.3 scrub starts 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.355942+0000 osd.7 (osd.7) 322 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.355942+0000 osd.7 (osd.7) 322 : cluster [DBG] 109.3 scrub ok 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.569994+0000 osd.2 (osd.2) 289 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.569994+0000 osd.2 (osd.2) 289 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.571286+0000 osd.2 (osd.2) 290 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:32:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.571286+0000 osd.2 (osd.2) 290 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.755457+0000 osd.0 (osd.0) 351 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.755457+0000 osd.0 (osd.0) 351 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.757200+0000 osd.0 (osd.0) 352 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.757200+0000 osd.0 (osd.0) 352 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.823613+0000 osd.3 (osd.3) 359 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.823613+0000 osd.3 (osd.3) 359 : cluster [DBG] 109.12 scrub starts 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.828707+0000 osd.3 (osd.3) 360 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:32:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:18 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.828707+0000 osd.3 (osd.3) 360 : cluster [DBG] 109.12 scrub ok 2024-04-26T16:32:18.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:18.858 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:18.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:18.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:18.989 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:18.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:19.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:19.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:19.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:19.252 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:19.272 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:19.273 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:19.355 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:19.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:19.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:19.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:32:19.514 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.769130+0000 mgr.y (mgr.24401) 579 : cluster [DBG] pgmap v911: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:17.769130+0000 mgr.y (mgr.24401) 579 : cluster [DBG] pgmap v911: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.368278+0000 osd.7 (osd.7) 323 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.368278+0000 osd.7 (osd.7) 323 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.373039+0000 osd.7 (osd.7) 324 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.373039+0000 osd.7 (osd.7) 324 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.531728+0000 osd.2 (osd.2) 291 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.531728+0000 osd.2 (osd.2) 291 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.535454+0000 osd.2 (osd.2) 292 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:18.535454+0000 osd.2 (osd.2) 292 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:32:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.223460+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:32:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.223460+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:32:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.225717+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:32:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:19 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.225717+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:32:19.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:19.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.769130+0000 mgr.y (mgr.24401) 579 : cluster [DBG] pgmap v911: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:19.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:17.769130+0000 mgr.y (mgr.24401) 579 : cluster [DBG] pgmap v911: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:19.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.368278+0000 osd.7 (osd.7) 323 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.368278+0000 osd.7 (osd.7) 323 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.373039+0000 osd.7 (osd.7) 324 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.373039+0000 osd.7 (osd.7) 324 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.531728+0000 osd.2 (osd.2) 291 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.531728+0000 osd.2 (osd.2) 291 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.535454+0000 osd.2 (osd.2) 292 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:18.535454+0000 osd.2 (osd.2) 292 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.223460+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.223460+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.225717+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.225717+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:32:19.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.769130+0000 mgr.y (mgr.24401) 579 : cluster [DBG] pgmap v911: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:17.769130+0000 mgr.y (mgr.24401) 579 : cluster [DBG] pgmap v911: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.368278+0000 osd.7 (osd.7) 323 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.368278+0000 osd.7 (osd.7) 323 : cluster [DBG] 109.1 scrub starts 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.373039+0000 osd.7 (osd.7) 324 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.373039+0000 osd.7 (osd.7) 324 : cluster [DBG] 109.1 scrub ok 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.531728+0000 osd.2 (osd.2) 291 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.531728+0000 osd.2 (osd.2) 291 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.535454+0000 osd.2 (osd.2) 292 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:18.535454+0000 osd.2 (osd.2) 292 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.223460+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:32:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.223460+0000 osd.4 (osd.4) 307 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:32:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.225717+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:32:19.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:19 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.225717+0000 osd.4 (osd.4) 308 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:32:19.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:19.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:19.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:19.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:19.896 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:19.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:19.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:20.027 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:20.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:20.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:20.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:20.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:20.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:20.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:20.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.286650+0000 osd.1 (osd.1) 321 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.286650+0000 osd.1 (osd.1) 321 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.289901+0000 osd.1 (osd.1) 322 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.289901+0000 osd.1 (osd.1) 322 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.400404+0000 osd.7 (osd.7) 325 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.400404+0000 osd.7 (osd.7) 325 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.402186+0000 osd.7 (osd.7) 326 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.402186+0000 osd.7 (osd.7) 326 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.835464+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.835464+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:32:20.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.838935+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:32:20.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.838935+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:32:20.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.265458+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:32:20.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.265458+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:32:20.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.267965+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:32:20.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:20 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.267965+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:32:20.552 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:20.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:20.683 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:20.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:20.814 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:20.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:20.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.286650+0000 osd.1 (osd.1) 321 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.286650+0000 osd.1 (osd.1) 321 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.289901+0000 osd.1 (osd.1) 322 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.289901+0000 osd.1 (osd.1) 322 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.400404+0000 osd.7 (osd.7) 325 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.400404+0000 osd.7 (osd.7) 325 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.402186+0000 osd.7 (osd.7) 326 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.402186+0000 osd.7 (osd.7) 326 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.835464+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:32:20.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.835464+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.838935+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.838935+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.265458+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.265458+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.267965+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.267965+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.286650+0000 osd.1 (osd.1) 321 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.286650+0000 osd.1 (osd.1) 321 : cluster [DBG] 109.11 scrub starts 2024-04-26T16:32:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.289901+0000 osd.1 (osd.1) 322 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.289901+0000 osd.1 (osd.1) 322 : cluster [DBG] 109.11 scrub ok 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.400404+0000 osd.7 (osd.7) 325 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.400404+0000 osd.7 (osd.7) 325 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.402186+0000 osd.7 (osd.7) 326 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.402186+0000 osd.7 (osd.7) 326 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.835464+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.835464+0000 osd.3 (osd.3) 361 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:32:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.838935+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:32:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.838935+0000 osd.3 (osd.3) 362 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:32:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.265458+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:32:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.265458+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:32:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.267965+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:32:20.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:20 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.267965+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:32:20.912 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:20.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:20.943 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:20.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:21.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:21.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:21.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:21.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:21.200 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:21.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:21.327 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:21.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:21.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:21.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:21.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:21.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.769860+0000 mgr.y (mgr.24401) 580 : cluster [DBG] pgmap v912: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-26T16:32:21.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:19.769860+0000 mgr.y (mgr.24401) 580 : cluster [DBG] pgmap v912: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-26T16:32:21.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.269263+0000 osd.1 (osd.1) 323 : cluster [DBG] 109.b scrub starts 2024-04-26T16:32:21.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.269263+0000 osd.1 (osd.1) 323 : cluster [DBG] 109.b scrub starts 2024-04-26T16:32:21.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.272927+0000 osd.1 (osd.1) 324 : cluster [DBG] 109.b scrub ok 2024-04-26T16:32:21.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:20.272927+0000 osd.1 (osd.1) 324 : cluster [DBG] 109.b scrub ok 2024-04-26T16:32:21.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: audit 2024-04-26T16:32:20.857349+0000 mon.c (mon.2) 670 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:21.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: audit 2024-04-26T16:32:20.857349+0000 mon.c (mon.2) 670 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:21.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.270510+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-04-26T16:32:21.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:21 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.270510+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-04-26T16:32:21.577 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:21.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.769860+0000 mgr.y (mgr.24401) 580 : cluster [DBG] pgmap v912: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-26T16:32:21.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:19.769860+0000 mgr.y (mgr.24401) 580 : cluster [DBG] pgmap v912: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.269263+0000 osd.1 (osd.1) 323 : cluster [DBG] 109.b scrub starts 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.269263+0000 osd.1 (osd.1) 323 : cluster [DBG] 109.b scrub starts 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.272927+0000 osd.1 (osd.1) 324 : cluster [DBG] 109.b scrub ok 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:20.272927+0000 osd.1 (osd.1) 324 : cluster [DBG] 109.b scrub ok 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: audit 2024-04-26T16:32:20.857349+0000 mon.c (mon.2) 670 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: audit 2024-04-26T16:32:20.857349+0000 mon.c (mon.2) 670 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.270510+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.270510+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.769860+0000 mgr.y (mgr.24401) 580 : cluster [DBG] pgmap v912: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:19.769860+0000 mgr.y (mgr.24401) 580 : cluster [DBG] pgmap v912: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.269263+0000 osd.1 (osd.1) 323 : cluster [DBG] 109.b scrub starts 2024-04-26T16:32:21.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.269263+0000 osd.1 (osd.1) 323 : cluster [DBG] 109.b scrub starts 2024-04-26T16:32:21.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.272927+0000 osd.1 (osd.1) 324 : cluster [DBG] 109.b scrub ok 2024-04-26T16:32:21.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:20.272927+0000 osd.1 (osd.1) 324 : cluster [DBG] 109.b scrub ok 2024-04-26T16:32:21.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: audit 2024-04-26T16:32:20.857349+0000 mon.c (mon.2) 670 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:21.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: audit 2024-04-26T16:32:20.857349+0000 mon.c (mon.2) 670 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:32:21.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.270510+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-04-26T16:32:21.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:21 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.270510+0000 mon.a (mon.0) 3079 : cluster [DBG] osdmap e593: 8 total, 8 up, 8 in 2024-04-26T16:32:21.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:21.699 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:21.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:21.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:21.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:21.959 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:21.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:22.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:22.086 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:22.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:22.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:22.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:22.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:22.348 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:22.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:22.479 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:22.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:22.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:22.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:22.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.312210+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:32:22.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.312210+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:32:22.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.314011+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:32:22.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.314011+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:32:22.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: audit 2024-04-26T16:32:21.593768+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:22.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: audit 2024-04-26T16:32:21.593768+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: audit 2024-04-26T16:32:21.595382+0000 mon.c (mon.2) 672 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: audit 2024-04-26T16:32:21.595382+0000 mon.c (mon.2) 672 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: audit 2024-04-26T16:32:21.607149+0000 mon.a (mon.0) 3080 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: audit 2024-04-26T16:32:21.607149+0000 mon.a (mon.0) 3080 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.608238+0000 osd.2 (osd.2) 293 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.608238+0000 osd.2 (osd.2) 293 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.610023+0000 osd.2 (osd.2) 294 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.610023+0000 osd.2 (osd.2) 294 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.153917+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.153917+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.158725+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:32:22.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:22 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.158725+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:32:22.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.312210+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.312210+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.314011+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.314011+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: audit 2024-04-26T16:32:21.593768+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: audit 2024-04-26T16:32:21.593768+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:22.786 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: audit 2024-04-26T16:32:21.595382+0000 mon.c (mon.2) 672 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: audit 2024-04-26T16:32:21.595382+0000 mon.c (mon.2) 672 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: audit 2024-04-26T16:32:21.607149+0000 mon.a (mon.0) 3080 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: audit 2024-04-26T16:32:21.607149+0000 mon.a (mon.0) 3080 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.608238+0000 osd.2 (osd.2) 293 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.608238+0000 osd.2 (osd.2) 293 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.610023+0000 osd.2 (osd.2) 294 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.610023+0000 osd.2 (osd.2) 294 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.153917+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.153917+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.158725+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:32:22.787 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.158725+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.312210+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.312210+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.314011+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.314011+0000 osd.6 (osd.6) 85 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: audit 2024-04-26T16:32:21.593768+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: audit 2024-04-26T16:32:21.593768+0000 mon.c (mon.2) 671 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: audit 2024-04-26T16:32:21.595382+0000 mon.c (mon.2) 672 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: audit 2024-04-26T16:32:21.595382+0000 mon.c (mon.2) 672 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: audit 2024-04-26T16:32:21.607149+0000 mon.a (mon.0) 3080 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: audit 2024-04-26T16:32:21.607149+0000 mon.a (mon.0) 3080 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.608238+0000 osd.2 (osd.2) 293 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:32:22.788 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.608238+0000 osd.2 (osd.2) 293 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:32:22.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.610023+0000 osd.2 (osd.2) 294 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:32:22.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.610023+0000 osd.2 (osd.2) 294 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:32:22.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.153917+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:32:22.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.153917+0000 osd.5 (osd.5) 211 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:32:22.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.158725+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:32:22.789 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:22 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.158725+0000 osd.5 (osd.5) 212 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:32:22.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:22.865 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:22.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:22.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:22.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:23.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:23.121 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:23.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.249 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:23.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:23.380 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:23.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.506 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:23.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:23.738 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:23.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.771952+0000 mgr.y (mgr.24401) 581 : cluster [DBG] pgmap v914: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:32:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:21.771952+0000 mgr.y (mgr.24401) 581 : cluster [DBG] pgmap v914: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:32:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.353788+0000 osd.4 (osd.4) 309 : cluster [DBG] 11.e scrub starts 2024-04-26T16:32:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.353788+0000 osd.4 (osd.4) 309 : cluster [DBG] 11.e scrub starts 2024-04-26T16:32:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.355773+0000 osd.4 (osd.4) 310 : cluster [DBG] 11.e scrub ok 2024-04-26T16:32:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.355773+0000 osd.4 (osd.4) 310 : cluster [DBG] 11.e scrub ok 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.631622+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.f scrub starts 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.631622+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.f scrub starts 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.633335+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.f scrub ok 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:22.633335+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.f scrub ok 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.281967+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.281967+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.283513+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:32:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:23 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.283513+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:32:23.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.771952+0000 mgr.y (mgr.24401) 581 : cluster [DBG] pgmap v914: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:21.771952+0000 mgr.y (mgr.24401) 581 : cluster [DBG] pgmap v914: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.353788+0000 osd.4 (osd.4) 309 : cluster [DBG] 11.e scrub starts 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.353788+0000 osd.4 (osd.4) 309 : cluster [DBG] 11.e scrub starts 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.355773+0000 osd.4 (osd.4) 310 : cluster [DBG] 11.e scrub ok 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.355773+0000 osd.4 (osd.4) 310 : cluster [DBG] 11.e scrub ok 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.631622+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.f scrub starts 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.631622+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.f scrub starts 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.633335+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.f scrub ok 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:22.633335+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.f scrub ok 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.281967+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:32:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.281967+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.283513+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.283513+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.771952+0000 mgr.y (mgr.24401) 581 : cluster [DBG] pgmap v914: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:21.771952+0000 mgr.y (mgr.24401) 581 : cluster [DBG] pgmap v914: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.353788+0000 osd.4 (osd.4) 309 : cluster [DBG] 11.e scrub starts 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.353788+0000 osd.4 (osd.4) 309 : cluster [DBG] 11.e scrub starts 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.355773+0000 osd.4 (osd.4) 310 : cluster [DBG] 11.e scrub ok 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.355773+0000 osd.4 (osd.4) 310 : cluster [DBG] 11.e scrub ok 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.631622+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.f scrub starts 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.631622+0000 osd.2 (osd.2) 295 : cluster [DBG] 6.f scrub starts 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.633335+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.f scrub ok 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:22.633335+0000 osd.2 (osd.2) 296 : cluster [DBG] 6.f scrub ok 2024-04-26T16:32:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.281967+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:32:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.281967+0000 osd.6 (osd.6) 86 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:32:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.283513+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:32:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:23 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.283513+0000 osd.6 (osd.6) 87 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:32:23.898 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:23.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:24.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:24.028 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:24.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:24.158 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:24.188 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:24.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:24.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:24.306 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:32:24.306 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-26T16:32:24.306 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:32:24.324 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:32:24.324 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:32:24.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:24.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:24.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:24.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:24.653 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:24.681 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:24.782 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:32:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:32:24.782 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.481678+0000 osd.7 (osd.7) 327 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.481678+0000 osd.7 (osd.7) 327 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.483738+0000 osd.7 (osd.7) 328 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.483738+0000 osd.7 (osd.7) 328 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.727388+0000 osd.0 (osd.0) 353 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.727388+0000 osd.0 (osd.0) 353 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.729616+0000 osd.0 (osd.0) 354 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.729616+0000 osd.0 (osd.0) 354 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.481678+0000 osd.7 (osd.7) 327 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.481678+0000 osd.7 (osd.7) 327 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:32:24.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.483738+0000 osd.7 (osd.7) 328 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:32:24.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.483738+0000 osd.7 (osd.7) 328 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:32:24.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.727388+0000 osd.0 (osd.0) 353 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:32:24.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.727388+0000 osd.0 (osd.0) 353 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:32:24.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.729616+0000 osd.0 (osd.0) 354 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:32:24.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:24 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.729616+0000 osd.0 (osd.0) 354 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:32:24.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:24.813 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:24.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.481678+0000 osd.7 (osd.7) 327 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.481678+0000 osd.7 (osd.7) 327 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.483738+0000 osd.7 (osd.7) 328 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.483738+0000 osd.7 (osd.7) 328 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.727388+0000 osd.0 (osd.0) 353 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.727388+0000 osd.0 (osd.0) 353 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.729616+0000 osd.0 (osd.0) 354 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:32:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:24 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.729616+0000 osd.0 (osd.0) 354 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:32:24.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:24.944 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:24.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:25.047 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:25.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:25.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:25.311 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:25.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:25.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:25.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:25.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:25.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:25.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.772658+0000 mgr.y (mgr.24401) 582 : cluster [DBG] pgmap v915: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 818 B/s wr, 2 op/s 2024-04-26T16:32:25.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:23.772658+0000 mgr.y (mgr.24401) 582 : cluster [DBG] pgmap v915: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 818 B/s wr, 2 op/s 2024-04-26T16:32:25.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.697061+0000 osd.0 (osd.0) 355 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.697061+0000 osd.0 (osd.0) 355 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.699473+0000 osd.0 (osd.0) 356 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.699473+0000 osd.0 (osd.0) 356 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.824045+0000 osd.3 (osd.3) 363 : cluster [DBG] 11.b scrub starts 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.824045+0000 osd.3 (osd.3) 363 : cluster [DBG] 11.b scrub starts 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.826164+0000 osd.3 (osd.3) 364 : cluster [DBG] 11.b scrub ok 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:24.826164+0000 osd.3 (osd.3) 364 : cluster [DBG] 11.b scrub ok 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.280500+0000 osd.4 (osd.4) 311 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.280500+0000 osd.4 (osd.4) 311 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.282979+0000 osd.4 (osd.4) 312 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.282979+0000 osd.4 (osd.4) 312 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:32:25.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.772658+0000 mgr.y (mgr.24401) 582 : cluster [DBG] pgmap v915: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 818 B/s wr, 2 op/s 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:23.772658+0000 mgr.y (mgr.24401) 582 : cluster [DBG] pgmap v915: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 818 B/s wr, 2 op/s 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.697061+0000 osd.0 (osd.0) 355 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.697061+0000 osd.0 (osd.0) 355 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.699473+0000 osd.0 (osd.0) 356 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.699473+0000 osd.0 (osd.0) 356 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.824045+0000 osd.3 (osd.3) 363 : cluster [DBG] 11.b scrub starts 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.824045+0000 osd.3 (osd.3) 363 : cluster [DBG] 11.b scrub starts 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.826164+0000 osd.3 (osd.3) 364 : cluster [DBG] 11.b scrub ok 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:24.826164+0000 osd.3 (osd.3) 364 : cluster [DBG] 11.b scrub ok 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.280500+0000 osd.4 (osd.4) 311 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.280500+0000 osd.4 (osd.4) 311 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.282979+0000 osd.4 (osd.4) 312 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:32:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:25 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.282979+0000 osd.4 (osd.4) 312 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:32:25.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:25.735 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:25.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:25.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-26T16:32:25.863 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:25.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.772658+0000 mgr.y (mgr.24401) 582 : cluster [DBG] pgmap v915: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 818 B/s wr, 2 op/s 2024-04-26T16:32:25.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:23.772658+0000 mgr.y (mgr.24401) 582 : cluster [DBG] pgmap v915: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 818 B/s wr, 2 op/s 2024-04-26T16:32:25.865 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.697061+0000 osd.0 (osd.0) 355 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.697061+0000 osd.0 (osd.0) 355 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.699473+0000 osd.0 (osd.0) 356 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.699473+0000 osd.0 (osd.0) 356 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.824045+0000 osd.3 (osd.3) 363 : cluster [DBG] 11.b scrub starts 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.824045+0000 osd.3 (osd.3) 363 : cluster [DBG] 11.b scrub starts 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.826164+0000 osd.3 (osd.3) 364 : cluster [DBG] 11.b scrub ok 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:24.826164+0000 osd.3 (osd.3) 364 : cluster [DBG] 11.b scrub ok 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.280500+0000 osd.4 (osd.4) 311 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.280500+0000 osd.4 (osd.4) 311 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.282979+0000 osd.4 (osd.4) 312 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:32:25.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:25 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.282979+0000 osd.4 (osd.4) 312 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:32:25.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:25.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:25.993 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:25.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:26.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:26.118 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:26.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:26.241 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- 172.21.15.50:0/3873097332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 msgr2=0x7ff83c06be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3873097332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 0x7ff83c06be20 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7ff828009990 tx=0x7ff828009ca0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- 172.21.15.50:0/3873097332 shutdown_connections 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3873097332 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff83c109490 0x7ff83c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3873097332 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c06c360 0x7ff83c1102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3873097332 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 0x7ff83c06be20 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- 172.21.15.50:0/3873097332 >> 172.21.15.50:0/3873097332 conn(0x7ff83c079290 msgr2=0x7ff83c0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- 172.21.15.50:0/3873097332 shutdown_connections 2024-04-26T16:32:26.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- 172.21.15.50:0/3873097332 wait complete. 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 librados: starting msgr at 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 librados: starting objecter 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 Processor -- start 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- start start 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 librados: setting wanted keys 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 librados: calling monclient init 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 0x7ff83c1991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff83c06c360 0x7ff83c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c109490 0x7ff83c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff83c112ad0 con 0x7ff83c06b9c0 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff83c112950 con 0x7ff83c06c360 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff842fe9700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff83c112c50 con 0x7ff83c109490 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff841586700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c109490 0x7ff83c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 0x7ff83c1991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff841586700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c109490 0x7ff83c19dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:38428/0 (socket says 172.21.15.50:38428) 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff841586700 1 -- 172.21.15.50:0/3703075780 learned_addr learned my addr 172.21.15.50:0/3703075780 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff83c06c360 0x7ff83c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c109490 msgr2=0x7ff83c19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c109490 0x7ff83c19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff83c06c360 msgr2=0x7ff83c199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff83c06c360 0x7ff83c199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c19e200 con 0x7ff83c06b9c0 2024-04-26T16:32:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff83c109490 0x7ff83c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:32:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 0x7ff83c1991f0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7ff83000d750 tx=0x7ff83000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff839ffb700 10 client.?.objecter ms_handle_connect 0x7ff83c06b9c0 2024-04-26T16:32:26.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.277+0000 7ff839ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff830016070 con 0x7ff83c06b9c0 2024-04-26T16:32:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83c19e4f0 con 0x7ff83c06b9c0 2024-04-26T16:32:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 10 client.30819.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:26.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff83c1a5d70 con 0x7ff83c06b9c0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 1 librados: init done 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff830004830 con 0x7ff83c06b9c0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff830005450 con 0x7ff83c06b9c0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff83c06a110 con 0x7ff83c06b9c0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff830004220 con 0x7ff83c06b9c0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff82c071f80 0x7ff82c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(593..593 src has 496..593) ==== 9377+0+0 (secure 0 0 0) 0x7ff830091c90 con 0x7ff83c06b9c0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 osd_map(593..593 src has 496..593) 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 3 client.30819.objecter handle_osd_map got epochs [593,593] > 0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 3 client.30819.objecter handle_osd_map decoding full epoch 593 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff82c071f80 0x7ff82c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 20 client.30819.objecter dump_active .. 0 homeless 2024-04-26T16:32:26.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v593) ==== 54+0+16 (secure 0 0 0) 0x7ff83001f020 con 0x7ff83c06b9c0 2024-04-26T16:32:26.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 10 client.30819.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 1 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff83c064030 0x7ff83c064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 1 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 0 1 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 4 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 10 client.30819.objecter _send_command 1 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1a9130 con 0x7ff83c064030 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff82c071f80 0x7ff82c074440 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7ff828000c00 tx=0x7ff828005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.281+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff82c071f80 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff83c064030 0x7ff83c064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c064030 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff83c064030 0x7ff83c064410 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1a9130 con 0x7ff83c064030 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 1: 0 ) 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff839ffb700 10 client.30819.objecter _finish_command 1 = system:0 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 4 2024-04-26T16:32:26.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.285+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 0 1 2024-04-26T16:32:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 10 client.30819.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 2 2024-04-26T16:32:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 2 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c19a550 osd=0 3 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 0 2 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 4 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 10 client.30819.objecter _send_command 2 2024-04-26T16:32:26.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.301+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c063ab0 con 0x7ff83c064030 2024-04-26T16:32:26.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c063ab0 con 0x7ff83c064030 2024-04-26T16:32:26.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 10 client.30819.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 4 2024-04-26T16:32:26.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 0 2 2024-04-26T16:32:26.302 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:32:26.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 10 client.30819.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 3 2024-04-26T16:32:26.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff83c1a92e0 0x7ff83c1a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 1 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 4 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 10 client.30819.objecter _send_command 3 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff83c104320 con 0x7ff83c1a92e0 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff83c1a92e0 0x7ff83c1a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c1a92e0 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff83c1a92e0 0x7ff83c1a9700 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c104320 con 0x7ff83c1a92e0 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 3: 0 ) 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 10 client.30819.objecter _finish_command 3 = system:0 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 4 2024-04-26T16:32:26.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.305+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 1 3 2024-04-26T16:32:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 10 client.30819.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 4 2024-04-26T16:32:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 4 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1a8780 osd=1 3 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 1 4 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 4 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 10 client.30819.objecter _send_command 4 2024-04-26T16:32:26.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c1a9c40 con 0x7ff83c1a92e0 2024-04-26T16:32:26.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c1a9c40 con 0x7ff83c1a92e0 2024-04-26T16:32:26.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff839ffb700 10 client.30819.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 4 2024-04-26T16:32:26.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 1 4 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 10 client.30819.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 5 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff83c02d820 0x7ff83c02dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 5 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 2 5 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 4 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 10 client.30819.objecter _send_command 5 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1aa210 con 0x7ff83c02d820 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff83c02d820 0x7ff83c02dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c02d820 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.325+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff83c02d820 0x7ff83c02dc60 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.329+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1aa210 con 0x7ff83c02d820 2024-04-26T16:32:26.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.329+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 5: 0 ) 2024-04-26T16:32:26.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.329+0000 7ff839ffb700 10 client.30819.objecter _finish_command 5 = system:0 2024-04-26T16:32:26.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.329+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 4 2024-04-26T16:32:26.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.329+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 2 5 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 10 client.30819.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 6 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 6 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1aa0a0 osd=2 3 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 2 6 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 4 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 10 client.30819.objecter _send_command 6 2024-04-26T16:32:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c1a8200 con 0x7ff83c02d820 2024-04-26T16:32:26.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:26.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c1a8200 con 0x7ff83c02d820 2024-04-26T16:32:26.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff839ffb700 10 client.30819.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 4 2024-04-26T16:32:26.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.345+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 2 6 2024-04-26T16:32:26.345 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:32:26.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 10 client.30819.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 7 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff83c1b0d90 0x7ff83c1b3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 7 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 3 7 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 4 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 10 client.30819.objecter _send_command 7 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1b5fe0 con 0x7ff83c1b0d90 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff83c1b0d90 0x7ff83c1b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c1b0d90 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff83c1b0d90 0x7ff83c1b3180 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1b5fe0 con 0x7ff83c1b0d90 2024-04-26T16:32:26.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 7: 0 ) 2024-04-26T16:32:26.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff839ffb700 10 client.30819.objecter _finish_command 7 = system:0 2024-04-26T16:32:26.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 4 2024-04-26T16:32:26.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.349+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 3 7 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 10 client.30819.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 8 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 8 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1a8600 osd=3 3 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 3 8 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 4 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 10 client.30819.objecter _send_command 8 2024-04-26T16:32:26.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c02e4b0 con 0x7ff83c1b0d90 2024-04-26T16:32:26.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c02e4b0 con 0x7ff83c1b0d90 2024-04-26T16:32:26.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff839ffb700 10 client.30819.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 4 2024-04-26T16:32:26.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 3 8 2024-04-26T16:32:26.372 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:32:26.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 10 client.30819.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 9 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff83c10af60 0x7ff83c1b6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 9 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 4 9 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 4 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 10 client.30819.objecter _send_command 9 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1b96b0 con 0x7ff83c10af60 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.373+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff83c10af60 0x7ff83c1b6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c10af60 2024-04-26T16:32:26.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff83c10af60 0x7ff83c1b6850 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1b96b0 con 0x7ff83c10af60 2024-04-26T16:32:26.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 9: 0 ) 2024-04-26T16:32:26.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff839ffb700 10 client.30819.objecter _finish_command 9 = system:0 2024-04-26T16:32:26.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 4 2024-04-26T16:32:26.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.377+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 4 9 2024-04-26T16:32:26.375 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 10 client.30819.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 10 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 10 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c02e200 osd=4 3 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 4 10 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 4 2024-04-26T16:32:26.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 10 client.30819.objecter _send_command 10 2024-04-26T16:32:26.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c10a4d0 con 0x7ff83c10af60 2024-04-26T16:32:26.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c10a4d0 con 0x7ff83c10af60 2024-04-26T16:32:26.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff839ffb700 10 client.30819.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 4 2024-04-26T16:32:26.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 4 10 2024-04-26T16:32:26.396 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:32:26.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 10 client.30819.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 11 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff83c10a640 0x7ff83c1b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 11 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 5 11 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 4 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 10 client.30819.objecter _send_command 11 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1b4600 con 0x7ff83c10a640 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.397+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff83c10a640 0x7ff83c1b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c10a640 2024-04-26T16:32:26.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff83bfff700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff83c10a640 0x7ff83c1b9f20 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1b4600 con 0x7ff83c10a640 2024-04-26T16:32:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 11: 0 ) 2024-04-26T16:32:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff839ffb700 10 client.30819.objecter _finish_command 11 = system:0 2024-04-26T16:32:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 4 2024-04-26T16:32:26.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.401+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 5 11 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 10 client.30819.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 12 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 12 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c10a4d0 osd=5 3 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 5 12 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 4 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 10 client.30819.objecter _send_command 12 2024-04-26T16:32:26.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.417+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c1b4620 con 0x7ff83c10a640 2024-04-26T16:32:26.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff82800bab0 con 0x7ff83c10a640 2024-04-26T16:32:26.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff839ffb700 10 client.30819.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 4 2024-04-26T16:32:26.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 5 12 2024-04-26T16:32:26.420 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:32:26.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 10 client.30819.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 13 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff83c1bab90 0x7ff83c1bafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 13 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 6 13 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 4 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 10 client.30819.objecter _send_command 13 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1c0060 con 0x7ff83c1bab90 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.421+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff83c1bab90 0x7ff83c1bafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c1bab90 2024-04-26T16:32:26.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff841586700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff83c1bab90 0x7ff83c1bafd0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1c0060 con 0x7ff83c1bab90 2024-04-26T16:32:26.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 13: 0 ) 2024-04-26T16:32:26.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff839ffb700 10 client.30819.objecter _finish_command 13 = system:0 2024-04-26T16:32:26.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 4 2024-04-26T16:32:26.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.425+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 6 13 2024-04-26T16:32:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 10 client.30819.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 14 2024-04-26T16:32:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 14 2024-04-26T16:32:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1ba4c0 osd=6 3 2024-04-26T16:32:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 6 14 2024-04-26T16:32:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 4 2024-04-26T16:32:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 10 client.30819.objecter _send_command 14 2024-04-26T16:32:26.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.441+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c1c0060 con 0x7ff83c1bab90 2024-04-26T16:32:26.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c1c0060 con 0x7ff83c1bab90 2024-04-26T16:32:26.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff839ffb700 10 client.30819.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 4 2024-04-26T16:32:26.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 6 14 2024-04-26T16:32:26.442 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 10 client.30819.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 15 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff83c1bf3c0 0x7ff83c1bf840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 15 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 7 15 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 4 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 10 client.30819.objecter _send_command 15 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff83c1c3400 con 0x7ff83c1bf3c0 2024-04-26T16:32:26.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff83c1bf3c0 0x7ff83c1bf840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:26.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff839ffb700 10 client.30819.objecter ms_handle_connect 0x7ff83c1bf3c0 2024-04-26T16:32:26.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.445+0000 7ff840d85700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff83c1bf3c0 0x7ff83c1bf840 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:26.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.449+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff83c1c3400 con 0x7ff83c1bf3c0 2024-04-26T16:32:26.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.449+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 15: 0 ) 2024-04-26T16:32:26.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.449+0000 7ff839ffb700 10 client.30819.objecter _finish_command 15 = system:0 2024-04-26T16:32:26.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.449+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 4 2024-04-26T16:32:26.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.449+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 7 15 2024-04-26T16:32:26.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 10 client.30819.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign -1 16 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 20 client.30819.objecter _get_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_remove -1 16 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 20 client.30819.objecter get_session s=0x7ff83c1c0060 osd=7 3 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 15 client.30819.objecter _session_command_op_assign 7 16 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 4 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 10 client.30819.objecter _send_command 16 2024-04-26T16:32:26.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.465+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff83c1bb620 con 0x7ff83c1bf3c0 2024-04-26T16:32:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff839ffb700 1 -- 172.21.15.50:0/3703075780 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff83c1bb620 con 0x7ff83c1bf3c0 2024-04-26T16:32:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff839ffb700 10 client.30819.objecter ms_dispatch 0x7ff83c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff839ffb700 10 client.30819.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff839ffb700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 4 2024-04-26T16:32:26.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff839ffb700 15 client.30819.objecter _session_command_op_remove 7 16 2024-04-26T16:32:26.467 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 librados: watch_flush enter 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 librados: watch_flush exit 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.0 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff83c064030 msgr2=0x7ff83c064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff83c064030 0x7ff83c064410 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c19a550 osd=0 1 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.1 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff83c1a92e0 msgr2=0x7ff83c1a9700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff83c1a92e0 0x7ff83c1a9700 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8780 osd=1 1 2024-04-26T16:32:26.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.2 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff83c02d820 msgr2=0x7ff83c02dc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff83c02d820 0x7ff83c02dc60 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1aa0a0 osd=2 1 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.3 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff83c1b0d90 msgr2=0x7ff83c1b3180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff83c1b0d90 0x7ff83c1b3180 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1a8600 osd=3 1 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.4 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff83c10af60 msgr2=0x7ff83c1b6850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff83c10af60 0x7ff83c1b6850 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff841586700 1 -- 172.21.15.50:0/3703075780 reap_dead start 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff841586700 1 -- 172.21.15.50:0/3703075780 reap_dead start 2024-04-26T16:32:26.470 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c02e200 osd=4 1 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.5 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff83c10a640 msgr2=0x7ff83c1b9f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff83c10a640 0x7ff83c1b9f20 crc :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c10a4d0 osd=5 1 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.6 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff83c1bab90 msgr2=0x7ff83c1bafd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff83c1bab90 0x7ff83c1bafd0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1ba4c0 osd=6 1 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter close_session for osd.7 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff83c1bf3c0 msgr2=0x7ff83c1bf840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff83c1bf3c0 0x7ff83c1bf840 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter put_session s=0x7ff83c1c0060 osd=7 1 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 20 client.30819.objecter shutdown clearing up homeless session... 2024-04-26T16:32:26.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 10 client.30819.objecter successfully canceled tick 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff82c071f80 msgr2=0x7ff82c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff82c071f80 0x7ff82c074440 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7ff828000c00 tx=0x7ff828005960 comp rx=0 tx=0).stop 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 msgr2=0x7ff83c1991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 --2- 172.21.15.50:0/3703075780 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff83c06b9c0 0x7ff83c1991f0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7ff83000d750 tx=0x7ff83000da60 comp rx=0 tx=0).stop 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff841586700 1 -- 172.21.15.50:0/3703075780 reap_dead start 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 shutdown_connections 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 >> 172.21.15.50:0/3703075780 conn(0x7ff83c079290 msgr2=0x7ff83c100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 shutdown_connections 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 -- 172.21.15.50:0/3703075780 wait complete. 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:26.469+0000 7ff842fe9700 1 librados: shutdown 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:32:26.472 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:32:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:32:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:32:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:32:26.473 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:32:26.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:26.503 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:26.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:26.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:26.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:26.768 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:26.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.444830+0000 osd.7 (osd.7) 329 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:32:26.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.444830+0000 osd.7 (osd.7) 329 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:32:26.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.455188+0000 osd.7 (osd.7) 330 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:32:26.816 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.455188+0000 osd.7 (osd.7) 330 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.588529+0000 osd.2 (osd.2) 297 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.588529+0000 osd.2 (osd.2) 297 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.592893+0000 osd.2 (osd.2) 298 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.592893+0000 osd.2 (osd.2) 298 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.701545+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.701545+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.704643+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.704643+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.733751+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.733751+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.734000+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.734000+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.860184+0000 osd.3 (osd.3) 365 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:32:26.817 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.860184+0000 osd.3 (osd.3) 365 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.865380+0000 osd.3 (osd.3) 366 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.865380+0000 osd.3 (osd.3) 366 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.266617+0000 mon.a (mon.0) 3081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.266617+0000 mon.a (mon.0) 3081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: audit 2024-04-26T16:32:26.284896+0000 mon.a (mon.0) 3082 : audit [DBG] from='client.? 172.21.15.50:0/3703075780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: audit 2024-04-26T16:32:26.284896+0000 mon.a (mon.0) 3082 : audit [DBG] from='client.? 172.21.15.50:0/3703075780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: audit 2024-04-26T16:32:26.357848+0000 mon.c (mon.2) 673 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: audit 2024-04-26T16:32:26.357848+0000 mon.c (mon.2) 673 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: audit 2024-04-26T16:32:26.358438+0000 mon.a (mon.0) 3083 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[21463]: audit 2024-04-26T16:32:26.358438+0000 mon.a (mon.0) 3083 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.818 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.444830+0000 osd.7 (osd.7) 329 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.444830+0000 osd.7 (osd.7) 329 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.455188+0000 osd.7 (osd.7) 330 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.455188+0000 osd.7 (osd.7) 330 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.588529+0000 osd.2 (osd.2) 297 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.588529+0000 osd.2 (osd.2) 297 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.592893+0000 osd.2 (osd.2) 298 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:32:26.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.592893+0000 osd.2 (osd.2) 298 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.701545+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.701545+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.704643+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.704643+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.733751+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.733751+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.734000+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.734000+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.860184+0000 osd.3 (osd.3) 365 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.860184+0000 osd.3 (osd.3) 365 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.865380+0000 osd.3 (osd.3) 366 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.865380+0000 osd.3 (osd.3) 366 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:32:26.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.266617+0000 mon.a (mon.0) 3081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.266617+0000 mon.a (mon.0) 3081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: audit 2024-04-26T16:32:26.284896+0000 mon.a (mon.0) 3082 : audit [DBG] from='client.? 172.21.15.50:0/3703075780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: audit 2024-04-26T16:32:26.284896+0000 mon.a (mon.0) 3082 : audit [DBG] from='client.? 172.21.15.50:0/3703075780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: audit 2024-04-26T16:32:26.357848+0000 mon.c (mon.2) 673 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: audit 2024-04-26T16:32:26.357848+0000 mon.c (mon.2) 673 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: audit 2024-04-26T16:32:26.358438+0000 mon.a (mon.0) 3083 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:26 smithi050 bash[29234]: audit 2024-04-26T16:32:26.358438+0000 mon.a (mon.0) 3083 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:26.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:26.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:26.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:26.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:27.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:27.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.444830+0000 osd.7 (osd.7) 329 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.444830+0000 osd.7 (osd.7) 329 : cluster [DBG] 109.7 scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.455188+0000 osd.7 (osd.7) 330 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.455188+0000 osd.7 (osd.7) 330 : cluster [DBG] 109.7 scrub ok 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.588529+0000 osd.2 (osd.2) 297 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.588529+0000 osd.2 (osd.2) 297 : cluster [DBG] 109.15 scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.592893+0000 osd.2 (osd.2) 298 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.592893+0000 osd.2 (osd.2) 298 : cluster [DBG] 109.15 scrub ok 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.701545+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.701545+0000 osd.0 (osd.0) 357 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.704643+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.704643+0000 osd.0 (osd.0) 358 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.733751+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:27.018 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.733751+0000 osd.2 (osd.2) 299 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.734000+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.734000+0000 osd.2 (osd.2) 300 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.860184+0000 osd.3 (osd.3) 365 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.860184+0000 osd.3 (osd.3) 365 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.865380+0000 osd.3 (osd.3) 366 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.865380+0000 osd.3 (osd.3) 366 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.266617+0000 mon.a (mon.0) 3081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.266617+0000 mon.a (mon.0) 3081 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: audit 2024-04-26T16:32:26.284896+0000 mon.a (mon.0) 3082 : audit [DBG] from='client.? 172.21.15.50:0/3703075780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: audit 2024-04-26T16:32:26.284896+0000 mon.a (mon.0) 3082 : audit [DBG] from='client.? 172.21.15.50:0/3703075780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: audit 2024-04-26T16:32:26.357848+0000 mon.c (mon.2) 673 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: audit 2024-04-26T16:32:26.357848+0000 mon.c (mon.2) 673 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: audit 2024-04-26T16:32:26.358438+0000 mon.a (mon.0) 3083 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:27.019 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:26 smithi190 bash[23868]: audit 2024-04-26T16:32:26.358438+0000 mon.a (mon.0) 3083 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]: dispatch 2024-04-26T16:32:27.033 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:27.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:27.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:27.164 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:27.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:27.281 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:32:27.296 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:27.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:27.427 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:27.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:27.541 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[27377]: debug 2024-04-26T16:32:27.326+0000 7f5cb819b700 -1 snap_mapper.add_oid found existing snaps mapped on 109:387d2352:test-rados-api-smithi050-73372-97::foo:21, removing 2024-04-26T16:32:27.541 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[71351]: debug 2024-04-26T16:32:27.326+0000 7f9e1ece2700 -1 snap_mapper.add_oid found existing snaps mapped on 109:387d2352:test-rados-api-smithi050-73372-97::foo:21, removing 2024-04-26T16:32:27.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[32690]: debug 2024-04-26T16:32:27.326+0000 7f1a30023700 -1 snap_mapper.add_oid found existing snaps mapped on 109:387d2352:test-rados-api-smithi050-73372-97::foo:21, removing 2024-04-26T16:32:27.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:27.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:27.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:27.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:27.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:27.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.773491+0000 mgr.y (mgr.24401) 583 : cluster [DBG] pgmap v916: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 241 B/s wr, 1 op/s 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:25.773491+0000 mgr.y (mgr.24401) 583 : cluster [DBG] pgmap v916: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 241 B/s wr, 1 op/s 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.362519+0000 osd.1 (osd.1) 325 : cluster [DBG] 11.c scrub starts 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.362519+0000 osd.1 (osd.1) 325 : cluster [DBG] 11.c scrub starts 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.368017+0000 osd.1 (osd.1) 326 : cluster [DBG] 11.c scrub ok 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.368017+0000 osd.1 (osd.1) 326 : cluster [DBG] 11.c scrub ok 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: audit 2024-04-26T16:32:26.629660+0000 mon.a (mon.0) 3084 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]': finished 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: audit 2024-04-26T16:32:26.629660+0000 mon.a (mon.0) 3084 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]': finished 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.633616+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-04-26T16:32:27.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.633616+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.889499+0000 osd.3 (osd.3) 367 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.889499+0000 osd.3 (osd.3) 367 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.895497+0000 osd.3 (osd.3) 368 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:26.895497+0000 osd.3 (osd.3) 368 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.223045+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.223045+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.226021+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.226021+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: audit 2024-04-26T16:32:27.273022+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:27.803 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:27 smithi190 bash[23868]: audit 2024-04-26T16:32:27.273022+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:27.816 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.773491+0000 mgr.y (mgr.24401) 583 : cluster [DBG] pgmap v916: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 241 B/s wr, 1 op/s 2024-04-26T16:32:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:25.773491+0000 mgr.y (mgr.24401) 583 : cluster [DBG] pgmap v916: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 241 B/s wr, 1 op/s 2024-04-26T16:32:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.362519+0000 osd.1 (osd.1) 325 : cluster [DBG] 11.c scrub starts 2024-04-26T16:32:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.362519+0000 osd.1 (osd.1) 325 : cluster [DBG] 11.c scrub starts 2024-04-26T16:32:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.368017+0000 osd.1 (osd.1) 326 : cluster [DBG] 11.c scrub ok 2024-04-26T16:32:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.368017+0000 osd.1 (osd.1) 326 : cluster [DBG] 11.c scrub ok 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: audit 2024-04-26T16:32:26.629660+0000 mon.a (mon.0) 3084 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]': finished 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: audit 2024-04-26T16:32:26.629660+0000 mon.a (mon.0) 3084 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]': finished 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.633616+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.633616+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.889499+0000 osd.3 (osd.3) 367 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.889499+0000 osd.3 (osd.3) 367 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.895497+0000 osd.3 (osd.3) 368 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:26.895497+0000 osd.3 (osd.3) 368 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.223045+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.223045+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.226021+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.226021+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: audit 2024-04-26T16:32:27.273022+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[21463]: audit 2024-04-26T16:32:27.273022+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.773491+0000 mgr.y (mgr.24401) 583 : cluster [DBG] pgmap v916: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 241 B/s wr, 1 op/s 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:25.773491+0000 mgr.y (mgr.24401) 583 : cluster [DBG] pgmap v916: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 241 B/s wr, 1 op/s 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.362519+0000 osd.1 (osd.1) 325 : cluster [DBG] 11.c scrub starts 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.362519+0000 osd.1 (osd.1) 325 : cluster [DBG] 11.c scrub starts 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.368017+0000 osd.1 (osd.1) 326 : cluster [DBG] 11.c scrub ok 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.368017+0000 osd.1 (osd.1) 326 : cluster [DBG] 11.c scrub ok 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: audit 2024-04-26T16:32:26.629660+0000 mon.a (mon.0) 3084 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]': finished 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: audit 2024-04-26T16:32:26.629660+0000 mon.a (mon.0) 3084 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "314.8", "id": [6, 0]}]': finished 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.633616+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.633616+0000 mon.a (mon.0) 3085 : cluster [DBG] osdmap e594: 8 total, 8 up, 8 in 2024-04-26T16:32:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.889499+0000 osd.3 (osd.3) 367 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.889499+0000 osd.3 (osd.3) 367 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.895497+0000 osd.3 (osd.3) 368 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:26.895497+0000 osd.3 (osd.3) 368 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.223045+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.223045+0000 osd.5 (osd.5) 213 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.226021+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.226021+0000 osd.5 (osd.5) 214 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: audit 2024-04-26T16:32:27.273022+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:27.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:27 smithi050 bash[29234]: audit 2024-04-26T16:32:27.273022+0000 mon.c (mon.2) 674 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:27.920 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:27.945 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:27.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:28.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:28.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:28.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:28.202 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:28.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:28.331 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:28.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:28.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:28.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:28.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:28.589 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:28.693 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:28.721 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.074287+0000 mgr.y (mgr.24401) 584 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.074287+0000 mgr.y (mgr.24401) 584 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.360763+0000 osd.1 (osd.1) 327 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:32:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.360763+0000 osd.1 (osd.1) 327 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:32:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.365184+0000 osd.1 (osd.1) 328 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:32:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.365184+0000 osd.1 (osd.1) 328 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.642456+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.642456+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.689429+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.689429+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.690396+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.690396+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.691318+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.691318+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.692109+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: audit 2024-04-26T16:32:27.692109+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.178795+0000 osd.5 (osd.5) 215 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.178795+0000 osd.5 (osd.5) 215 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:32:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.181559+0000 osd.5 (osd.5) 216 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:32:28.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:28 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.181559+0000 osd.5 (osd.5) 216 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:32:28.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:28.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:28.954 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.074287+0000 mgr.y (mgr.24401) 584 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.074287+0000 mgr.y (mgr.24401) 584 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.360763+0000 osd.1 (osd.1) 327 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.360763+0000 osd.1 (osd.1) 327 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.365184+0000 osd.1 (osd.1) 328 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.365184+0000 osd.1 (osd.1) 328 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:32:28.966 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.642456+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.642456+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.689429+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.689429+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.690396+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.690396+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.691318+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.691318+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.692109+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: audit 2024-04-26T16:32:27.692109+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.178795+0000 osd.5 (osd.5) 215 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.178795+0000 osd.5 (osd.5) 215 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.181559+0000 osd.5 (osd.5) 216 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:32:28.967 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.181559+0000 osd.5 (osd.5) 216 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.074287+0000 mgr.y (mgr.24401) 584 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.074287+0000 mgr.y (mgr.24401) 584 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.360763+0000 osd.1 (osd.1) 327 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.360763+0000 osd.1 (osd.1) 327 : cluster [DBG] 109.4 scrub starts 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.365184+0000 osd.1 (osd.1) 328 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.365184+0000 osd.1 (osd.1) 328 : cluster [DBG] 109.4 scrub ok 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.642456+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.642456+0000 mon.a (mon.0) 3086 : cluster [DBG] osdmap e595: 8 total, 8 up, 8 in 2024-04-26T16:32:28.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.689429+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.689429+0000 mon.b (mon.1) 410 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.690396+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.690396+0000 mon.a (mon.0) 3087 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.691318+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.691318+0000 mon.b (mon.1) 411 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.692109+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: audit 2024-04-26T16:32:27.692109+0000 mon.a (mon.0) 3088 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-96"}]: dispatch 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.178795+0000 osd.5 (osd.5) 215 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.178795+0000 osd.5 (osd.5) 215 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.181559+0000 osd.5 (osd.5) 216 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:32:28.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:28 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.181559+0000 osd.5 (osd.5) 216 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:32:28.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:28.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:29.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.110 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:29.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:29.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:29.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.374 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3888121933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1908101800 msgr2=0x7f1908101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:29.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- 172.21.15.50:0/3888121933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1908101800 0x7f1908101be0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f19040099c0 tx=0x7f1904009cd0 comp rx=0 tx=0).stop 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3888121933 shutdown_connections 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- 172.21.15.50:0/3888121933 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19081098e0 0x7f1908110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- 172.21.15.50:0/3888121933 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1908101800 0x7f1908101be0 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- 172.21.15.50:0/3888121933 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f19081021b0 0x7f1908109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3888121933 >> 172.21.15.50:0/3888121933 conn(0x7f1908078520 msgr2=0x7f19080ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3888121933 shutdown_connections 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3888121933 wait complete. 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 librados: starting msgr at 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 librados: starting objecter 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 Processor -- start 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- start start 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 librados: setting wanted keys 2024-04-26T16:32:29.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 librados: calling monclient init 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 0x7f190819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19081021b0 0x7f190819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f19081098e0 0x7f190819fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1908112ab0 con 0x7f1908101800 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1908112930 con 0x7f19081021b0 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1908112c30 con 0x7f19081098e0 2024-04-26T16:32:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 0x7f190819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190e532700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f19081098e0 0x7f190819fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 0x7f190819b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39592/0 (socket says 172.21.15.50:39592) 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 -- 172.21.15.50:0/3497828113 learned_addr learned my addr 172.21.15.50:0/3497828113 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190d530700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19081021b0 0x7f190819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 -- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f19081098e0 msgr2=0x7f190819fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f19081098e0 0x7f190819fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 -- 172.21.15.50:0/3497828113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19081021b0 msgr2=0x7f190819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19081021b0 0x7f190819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 -- 172.21.15.50:0/3497828113 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19081a0310 con 0x7f1908101800 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190e532700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f19081098e0 0x7f190819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190dd31700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 0x7f190819b350 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f190400b580 tx=0x7f190400fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f18feffd700 10 client.?.objecter ms_handle_connect 0x7f1908101800 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f18feffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:29.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904027070 con 0x7f1908101800 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19081a05a0 con 0x7f1908101800 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f190400fe00 con 0x7f1908101800 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1904022e00 con 0x7f1908101800 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 10 client.30825.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19081a7e80 con 0x7f1908101800 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 librados: init done 2024-04-26T16:32:29.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.436+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1908102930 con 0x7f1908101800 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f190402a070 con 0x7f1908101800 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f18f4071f80 0x7f18f4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(596..596 src has 496..596) ==== 8992+0+0 (secure 0 0 0) 0x7f19040924b0 con 0x7f1908101800 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f190d530700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f18f4071f80 0x7f18f4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 10 client.30825.objecter ms_dispatch 0x7f1908001550 osd_map(596..596 src has 496..596) 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 3 client.30825.objecter handle_osd_map got epochs [596,596] > 0 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 3 client.30825.objecter handle_osd_map decoding full epoch 596 2024-04-26T16:32:29.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 20 client.30825.objecter dump_active .. 0 homeless 2024-04-26T16:32:29.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f18feffd700 10 client.30825.objecter ms_handle_connect 0x7f18f4071f80 2024-04-26T16:32:29.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.440+0000 7f190d530700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f18f4071f80 0x7f18f4074440 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009680 tx=0x7f18f8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:29.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.444+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1904013070 con 0x7f1908101800 2024-04-26T16:32:29.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:29.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.638 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:29.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:29.696+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f190819c660 con 0x7f1908101800 2024-04-26T16:32:29.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:29.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:29.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:29.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:29.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:29.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:29.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:32:29.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.774299+0000 mgr.y (mgr.24401) 585 : cluster [DBG] pgmap v919: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:29.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:27.774299+0000 mgr.y (mgr.24401) 585 : cluster [DBG] pgmap v919: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:29.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.649515+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-04-26T16:32:29.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.649515+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-04-26T16:32:29.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.748028+0000 osd.0 (osd.0) 359 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:32:29.986 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.748028+0000 osd.0 (osd.0) 359 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.750681+0000 osd.0 (osd.0) 360 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:28.750681+0000 osd.0 (osd.0) 360 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.652844+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.652844+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: audit 2024-04-26T16:32:29.654412+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: audit 2024-04-26T16:32:29.654412+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: audit 2024-04-26T16:32:29.656272+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[29234]: audit 2024-04-26T16:32:29.656272+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.774299+0000 mgr.y (mgr.24401) 585 : cluster [DBG] pgmap v919: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:27.774299+0000 mgr.y (mgr.24401) 585 : cluster [DBG] pgmap v919: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.649515+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-04-26T16:32:29.987 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.649515+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.748028+0000 osd.0 (osd.0) 359 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.748028+0000 osd.0 (osd.0) 359 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.750681+0000 osd.0 (osd.0) 360 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:28.750681+0000 osd.0 (osd.0) 360 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.652844+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.652844+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: audit 2024-04-26T16:32:29.654412+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: audit 2024-04-26T16:32:29.654412+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: audit 2024-04-26T16:32:29.656272+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.988 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:29 smithi050 bash[21463]: audit 2024-04-26T16:32:29.656272+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:29.996 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:30.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.774299+0000 mgr.y (mgr.24401) 585 : cluster [DBG] pgmap v919: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:27.774299+0000 mgr.y (mgr.24401) 585 : cluster [DBG] pgmap v919: 228 pgs: 228 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.649515+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-04-26T16:32:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.649515+0000 mon.a (mon.0) 3089 : cluster [DBG] osdmap e596: 8 total, 8 up, 8 in 2024-04-26T16:32:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.748028+0000 osd.0 (osd.0) 359 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:32:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.748028+0000 osd.0 (osd.0) 359 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.750681+0000 osd.0 (osd.0) 360 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:28.750681+0000 osd.0 (osd.0) 360 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.652844+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.652844+0000 mon.a (mon.0) 3090 : cluster [DBG] osdmap e597: 8 total, 8 up, 8 in 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: audit 2024-04-26T16:32:29.654412+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: audit 2024-04-26T16:32:29.654412+0000 mon.b (mon.1) 412 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: audit 2024-04-26T16:32:29.656272+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:29 smithi190 bash[23868]: audit 2024-04-26T16:32:29.656272+0000 mon.a (mon.0) 3091 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:30.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:30.151 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:30.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:30.279 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:30.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:30.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:30.410 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:30.513 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:30.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:30.633 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:30.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.644+0000 7f18feffd700 1 -- 172.21.15.50:0/3497828113 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v598) ==== 101+0+0 (secure 0 0 0) 0x7f190405e110 con 0x7f1908101800 2024-04-26T16:32:30.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.648+0000 7f190ff95700 10 librados: watch_flush enter 2024-04-26T16:32:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.648+0000 7f190ff95700 10 librados: watch_flush exit 2024-04-26T16:32:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.648+0000 7f190ff95700 20 client.30825.objecter shutdown clearing up homeless session... 2024-04-26T16:32:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.648+0000 7f190ff95700 10 client.30825.objecter successfully canceled tick 2024-04-26T16:32:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f18f4071f80 msgr2=0x7f18f4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f18f4071f80 0x7f18f4074440 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 crypto rx=0x7f18f8009680 tx=0x7f18f8009260 comp rx=0 tx=0).stop 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 msgr2=0x7f190819b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 0x7f190819b350 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f190400b580 tx=0x7f190400fa00 comp rx=0 tx=0).stop 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 shutdown_connections 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f18f4071f80 0x7f18f4074440 unknown :-1 s=CLOSED pgs=3785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f19081098e0 0x7f190819fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1908101800 0x7f190819b350 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:30.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 --2- 172.21.15.50:0/3497828113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19081021b0 0x7f190819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:30.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 >> 172.21.15.50:0/3497828113 conn(0x7f1908078520 msgr2=0x7f1908103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:30.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 shutdown_connections 2024-04-26T16:32:30.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 -- 172.21.15.50:0/3497828113 wait complete. 2024-04-26T16:32:30.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:30.652+0000 7f190ff95700 1 librados: shutdown 2024-04-26T16:32:30.660 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:32:30.680 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:30.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.509022+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.509022+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.512084+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.512084+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: audit 2024-04-26T16:32:29.699982+0000 mon.a (mon.0) 3092 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: audit 2024-04-26T16:32:29.699982+0000 mon.a (mon.0) 3092 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.796082+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.796082+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:32:30.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.797509+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.797509+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.235593+0000 osd.5 (osd.5) 217 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.235593+0000 osd.5 (osd.5) 217 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.237994+0000 osd.5 (osd.5) 218 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.237994+0000 osd.5 (osd.5) 218 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.645645+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.645645+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: audit 2024-04-26T16:32:30.647950+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: audit 2024-04-26T16:32:30.647950+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: audit 2024-04-26T16:32:30.648225+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: audit 2024-04-26T16:32:30.648225+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.656107+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-04-26T16:32:30.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.656107+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.509022+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.509022+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.512084+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.512084+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: audit 2024-04-26T16:32:29.699982+0000 mon.a (mon.0) 3092 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: audit 2024-04-26T16:32:29.699982+0000 mon.a (mon.0) 3092 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.796082+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.796082+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.797509+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.797509+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.235593+0000 osd.5 (osd.5) 217 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.235593+0000 osd.5 (osd.5) 217 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.237994+0000 osd.5 (osd.5) 218 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:32:30.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.237994+0000 osd.5 (osd.5) 218 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.645645+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.645645+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: audit 2024-04-26T16:32:30.647950+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: audit 2024-04-26T16:32:30.647950+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: audit 2024-04-26T16:32:30.648225+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: audit 2024-04-26T16:32:30.648225+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.656107+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-04-26T16:32:30.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:30 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.656107+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-04-26T16:32:30.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:30.832 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-04-26T16:32:30.832 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:32:30.832 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-04-26T16:32:30.832 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-26T16:32:30.832 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-26T16:32:30.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-04-26T16:32:30.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:30.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:30.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:31.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:31.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.509022+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.509022+0000 osd.7 (osd.7) 331 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.512084+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.512084+0000 osd.7 (osd.7) 332 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: audit 2024-04-26T16:32:29.699982+0000 mon.a (mon.0) 3092 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: audit 2024-04-26T16:32:29.699982+0000 mon.a (mon.0) 3092 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.796082+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.796082+0000 osd.0 (osd.0) 361 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.797509+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.797509+0000 osd.0 (osd.0) 362 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.235593+0000 osd.5 (osd.5) 217 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:32:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.235593+0000 osd.5 (osd.5) 217 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.237994+0000 osd.5 (osd.5) 218 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.237994+0000 osd.5 (osd.5) 218 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.645645+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.645645+0000 mon.a (mon.0) 3093 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: audit 2024-04-26T16:32:30.647950+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: audit 2024-04-26T16:32:30.647950+0000 mon.a (mon.0) 3094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: audit 2024-04-26T16:32:30.648225+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: audit 2024-04-26T16:32:30.648225+0000 mon.a (mon.0) 3095 : audit [INF] from='client.? 172.21.15.50:0/3497828113' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.656107+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-04-26T16:32:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:30 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.656107+0000 mon.a (mon.0) 3096 : cluster [DBG] osdmap e598: 8 total, 8 up, 8 in 2024-04-26T16:32:31.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:31.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:31.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:31.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:31.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:31.459 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:31.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:31.589 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:31.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.774764+0000 mgr.y (mgr.24401) 586 : cluster [DBG] pgmap v922: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:29.774764+0000 mgr.y (mgr.24401) 586 : cluster [DBG] pgmap v922: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:30.704498+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:30.704498+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:30.705154+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:30.705154+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.802653+0000 osd.3 (osd.3) 369 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.802653+0000 osd.3 (osd.3) 369 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.805310+0000 osd.3 (osd.3) 370 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:32:31.704 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:30.805310+0000 osd.3 (osd.3) 370 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.266919+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.266919+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.267825+0000 mon.a (mon.0) 3098 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.267825+0000 mon.a (mon.0) 3098 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.269470+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.269470+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:31.271222+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:31.271222+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.278724+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.278724+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:31.280539+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:31.280539+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:31.281735+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.705 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:31 smithi190 bash[23868]: audit 2024-04-26T16:32:31.281735+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.717 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:31.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:31.848 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:31.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.774764+0000 mgr.y (mgr.24401) 586 : cluster [DBG] pgmap v922: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:29.774764+0000 mgr.y (mgr.24401) 586 : cluster [DBG] pgmap v922: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:30.704498+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:30.704498+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:30.705154+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:30.705154+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.802653+0000 osd.3 (osd.3) 369 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.802653+0000 osd.3 (osd.3) 369 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.805310+0000 osd.3 (osd.3) 370 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:30.805310+0000 osd.3 (osd.3) 370 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.266919+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.266919+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.267825+0000 mon.a (mon.0) 3098 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.267825+0000 mon.a (mon.0) 3098 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.269470+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.269470+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:31.271222+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:31.271222+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.278724+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.774764+0000 mgr.y (mgr.24401) 586 : cluster [DBG] pgmap v922: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:29.774764+0000 mgr.y (mgr.24401) 586 : cluster [DBG] pgmap v922: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:30.704498+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:30.704498+0000 mon.b (mon.1) 413 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:30.705154+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:30.705154+0000 mon.a (mon.0) 3097 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.802653+0000 osd.3 (osd.3) 369 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.802653+0000 osd.3 (osd.3) 369 : cluster [DBG] 109.9 scrub starts 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.805310+0000 osd.3 (osd.3) 370 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:30.805310+0000 osd.3 (osd.3) 370 : cluster [DBG] 109.9 scrub ok 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.266919+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.266919+0000 osd.6 (osd.6) 88 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.267825+0000 mon.a (mon.0) 3098 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.267825+0000 mon.a (mon.0) 3098 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.269470+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.269470+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:31.271222+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:31.271222+0000 mon.a (mon.0) 3099 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.278724+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.278724+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:31.280539+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:31.280539+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:31.281735+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[21463]: audit 2024-04-26T16:32:31.281735+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.278724+0000 mon.a (mon.0) 3100 : cluster [DBG] osdmap e599: 8 total, 8 up, 8 in 2024-04-26T16:32:31.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:31.280539+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:31.280539+0000 mon.b (mon.1) 414 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:31.281735+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:31 smithi050 bash[29234]: audit 2024-04-26T16:32:31.281735+0000 mon.a (mon.0) 3101 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]: dispatch 2024-04-26T16:32:31.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:31.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:31.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.115 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:32.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:32.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:32.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.372 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:32.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:32.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:32.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.634 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:32.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:32.759 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:32.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.656273+0000 osd.2 (osd.2) 301 : cluster [DBG] 109.8 deep-scrub starts 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.656273+0000 osd.2 (osd.2) 301 : cluster [DBG] 109.8 deep-scrub starts 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.668527+0000 osd.2 (osd.2) 302 : cluster [DBG] 109.8 deep-scrub ok 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.668527+0000 osd.2 (osd.2) 302 : cluster [DBG] 109.8 deep-scrub ok 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.271373+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.271373+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: audit 2024-04-26T16:32:32.272882+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]': finished 2024-04-26T16:32:32.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: audit 2024-04-26T16:32:32.272882+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]': finished 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.276988+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.276988+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: audit 2024-04-26T16:32:32.277942+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: audit 2024-04-26T16:32:32.277942+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: audit 2024-04-26T16:32:32.278759+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: audit 2024-04-26T16:32:32.278759+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.294870+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.294870+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.298123+0000 osd.6 (osd.6) 91 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:32:32.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:32 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.298123+0000 osd.6 (osd.6) 91 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:32:32.887 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:32.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:32.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:33.018 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:33.121 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:33.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.656273+0000 osd.2 (osd.2) 301 : cluster [DBG] 109.8 deep-scrub starts 2024-04-26T16:32:33.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.656273+0000 osd.2 (osd.2) 301 : cluster [DBG] 109.8 deep-scrub starts 2024-04-26T16:32:33.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.668527+0000 osd.2 (osd.2) 302 : cluster [DBG] 109.8 deep-scrub ok 2024-04-26T16:32:33.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.668527+0000 osd.2 (osd.2) 302 : cluster [DBG] 109.8 deep-scrub ok 2024-04-26T16:32:33.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.271373+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:33.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.271373+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: audit 2024-04-26T16:32:32.272882+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]': finished 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: audit 2024-04-26T16:32:32.272882+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]': finished 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.276988+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.276988+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: audit 2024-04-26T16:32:32.277942+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: audit 2024-04-26T16:32:32.277942+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: audit 2024-04-26T16:32:32.278759+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: audit 2024-04-26T16:32:32.278759+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.294870+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.294870+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.298123+0000 osd.6 (osd.6) 91 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.298123+0000 osd.6 (osd.6) 91 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.656273+0000 osd.2 (osd.2) 301 : cluster [DBG] 109.8 deep-scrub starts 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.656273+0000 osd.2 (osd.2) 301 : cluster [DBG] 109.8 deep-scrub starts 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.668527+0000 osd.2 (osd.2) 302 : cluster [DBG] 109.8 deep-scrub ok 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.668527+0000 osd.2 (osd.2) 302 : cluster [DBG] 109.8 deep-scrub ok 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.271373+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.271373+0000 mon.a (mon.0) 3102 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: audit 2024-04-26T16:32:32.272882+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]': finished 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: audit 2024-04-26T16:32:32.272882+0000 mon.a (mon.0) 3103 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-98", "mode": "writeback"}]': finished 2024-04-26T16:32:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.276988+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.276988+0000 mon.a (mon.0) 3104 : cluster [DBG] osdmap e600: 8 total, 8 up, 8 in 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: audit 2024-04-26T16:32:32.277942+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: audit 2024-04-26T16:32:32.277942+0000 mon.b (mon.1) 415 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: audit 2024-04-26T16:32:32.278759+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: audit 2024-04-26T16:32:32.278759+0000 mon.a (mon.0) 3105 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.294870+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.294870+0000 osd.6 (osd.6) 90 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.298123+0000 osd.6 (osd.6) 91 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:32:33.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:32 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.298123+0000 osd.6 (osd.6) 91 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:32:33.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:33.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:33.253 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:33.286 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:33.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:33.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:33.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:33.547 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:33.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:33.677 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:33.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:33.810 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:33.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:33.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:33.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:34.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.776990+0000 mgr.y (mgr.24401) 587 : cluster [DBG] pgmap v925: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:31.776990+0000 mgr.y (mgr.24401) 587 : cluster [DBG] pgmap v925: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.513863+0000 osd.7 (osd.7) 333 : cluster [DBG] 109.1d scrub starts 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.513863+0000 osd.7 (osd.7) 333 : cluster [DBG] 109.1d scrub starts 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.516794+0000 osd.7 (osd.7) 334 : cluster [DBG] 109.1d scrub ok 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:32.516794+0000 osd.7 (osd.7) 334 : cluster [DBG] 109.1d scrub ok 2024-04-26T16:32:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.221633+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.221633+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.223598+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.223598+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: audit 2024-04-26T16:32:33.276416+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: audit 2024-04-26T16:32:33.276416+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.281873+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.281873+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: audit 2024-04-26T16:32:33.287201+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: audit 2024-04-26T16:32:33.287201+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: audit 2024-04-26T16:32:33.288306+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.030 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[21463]: audit 2024-04-26T16:32:33.288306+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.776990+0000 mgr.y (mgr.24401) 587 : cluster [DBG] pgmap v925: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:31.776990+0000 mgr.y (mgr.24401) 587 : cluster [DBG] pgmap v925: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.513863+0000 osd.7 (osd.7) 333 : cluster [DBG] 109.1d scrub starts 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.513863+0000 osd.7 (osd.7) 333 : cluster [DBG] 109.1d scrub starts 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.516794+0000 osd.7 (osd.7) 334 : cluster [DBG] 109.1d scrub ok 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:32.516794+0000 osd.7 (osd.7) 334 : cluster [DBG] 109.1d scrub ok 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.221633+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.221633+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.223598+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.223598+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:32:34.031 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: audit 2024-04-26T16:32:33.276416+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: audit 2024-04-26T16:32:33.276416+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.281873+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.281873+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: audit 2024-04-26T16:32:33.287201+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: audit 2024-04-26T16:32:33.287201+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: audit 2024-04-26T16:32:33.288306+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.032 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:33 smithi050 bash[29234]: audit 2024-04-26T16:32:33.288306+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:34.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.776990+0000 mgr.y (mgr.24401) 587 : cluster [DBG] pgmap v925: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:31.776990+0000 mgr.y (mgr.24401) 587 : cluster [DBG] pgmap v925: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.513863+0000 osd.7 (osd.7) 333 : cluster [DBG] 109.1d scrub starts 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.513863+0000 osd.7 (osd.7) 333 : cluster [DBG] 109.1d scrub starts 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.516794+0000 osd.7 (osd.7) 334 : cluster [DBG] 109.1d scrub ok 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:32.516794+0000 osd.7 (osd.7) 334 : cluster [DBG] 109.1d scrub ok 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.221633+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:32:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.221633+0000 osd.4 (osd.4) 313 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.223598+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.223598+0000 osd.4 (osd.4) 314 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: audit 2024-04-26T16:32:33.276416+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: audit 2024-04-26T16:32:33.276416+0000 mon.a (mon.0) 3106 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.281873+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.281873+0000 mon.a (mon.0) 3107 : cluster [DBG] osdmap e601: 8 total, 8 up, 8 in 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: audit 2024-04-26T16:32:33.287201+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: audit 2024-04-26T16:32:33.287201+0000 mon.b (mon.1) 416 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: audit 2024-04-26T16:32:33.288306+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:33 smithi190 bash[23868]: audit 2024-04-26T16:32:33.288306+0000 mon.a (mon.0) 3108 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:32:34.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:34.193 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:34.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:34.312 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:32:34.312 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:32:34.312 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:32:34.312 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:32:34.312 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:32:34.317 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:34.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:34.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:34.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:34.580 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:34.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:34.704 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:32:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:32:34.709 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:34.710 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:34.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:34.840 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:34.943 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.678354+0000 osd.2 (osd.2) 303 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.678354+0000 osd.2 (osd.2) 303 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.684522+0000 osd.2 (osd.2) 304 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.684522+0000 osd.2 (osd.2) 304 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.276772+0000 mon.a (mon.0) 3109 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.276772+0000 mon.a (mon.0) 3109 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: audit 2024-04-26T16:32:34.278673+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: audit 2024-04-26T16:32:34.278673+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:32:34.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.286619+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-04-26T16:32:34.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.286619+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-04-26T16:32:34.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: audit 2024-04-26T16:32:34.288111+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: audit 2024-04-26T16:32:34.288111+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: audit 2024-04-26T16:32:34.289441+0000 mon.a (mon.0) 3112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[21463]: audit 2024-04-26T16:32:34.289441+0000 mon.a (mon.0) 3112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.678354+0000 osd.2 (osd.2) 303 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:32:34.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.678354+0000 osd.2 (osd.2) 303 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:32:34.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.684522+0000 osd.2 (osd.2) 304 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:32:34.958 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.684522+0000 osd.2 (osd.2) 304 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:32:34.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.276772+0000 mon.a (mon.0) 3109 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:34.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.276772+0000 mon.a (mon.0) 3109 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:34.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: audit 2024-04-26T16:32:34.278673+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:32:34.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: audit 2024-04-26T16:32:34.278673+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:32:34.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.286619+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-04-26T16:32:34.959 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.286619+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-04-26T16:32:34.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: audit 2024-04-26T16:32:34.288111+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: audit 2024-04-26T16:32:34.288111+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: audit 2024-04-26T16:32:34.289441+0000 mon.a (mon.0) 3112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.960 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:34 smithi050 bash[29234]: audit 2024-04-26T16:32:34.289441+0000 mon.a (mon.0) 3112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:34.972 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:34.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:35.075 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:35.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:35.205 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.233 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.678354+0000 osd.2 (osd.2) 303 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:32:35.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.678354+0000 osd.2 (osd.2) 303 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.684522+0000 osd.2 (osd.2) 304 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.684522+0000 osd.2 (osd.2) 304 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.276772+0000 mon.a (mon.0) 3109 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.276772+0000 mon.a (mon.0) 3109 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: audit 2024-04-26T16:32:34.278673+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: audit 2024-04-26T16:32:34.278673+0000 mon.a (mon.0) 3110 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.286619+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.286619+0000 mon.a (mon.0) 3111 : cluster [DBG] osdmap e602: 8 total, 8 up, 8 in 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: audit 2024-04-26T16:32:34.288111+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: audit 2024-04-26T16:32:34.288111+0000 mon.b (mon.1) 417 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: audit 2024-04-26T16:32:34.289441+0000 mon.a (mon.0) 3112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:35.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:34 smithi190 bash[23868]: audit 2024-04-26T16:32:34.289441+0000 mon.a (mon.0) 3112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]: dispatch 2024-04-26T16:32:35.337 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:35.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:35.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.496 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:35.554 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:35.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.622 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:35.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.777871+0000 mgr.y (mgr.24401) 588 : cluster [DBG] pgmap v928: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:32:35.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:33.777871+0000 mgr.y (mgr.24401) 588 : cluster [DBG] pgmap v928: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:32:35.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.306342+0000 osd.1 (osd.1) 329 : cluster [DBG] 109.c scrub starts 2024-04-26T16:32:35.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.306342+0000 osd.1 (osd.1) 329 : cluster [DBG] 109.c scrub starts 2024-04-26T16:32:35.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.308419+0000 osd.1 (osd.1) 330 : cluster [DBG] 109.c scrub ok 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.308419+0000 osd.1 (osd.1) 330 : cluster [DBG] 109.c scrub ok 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.580381+0000 osd.7 (osd.7) 335 : cluster [DBG] 109.10 scrub starts 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.580381+0000 osd.7 (osd.7) 335 : cluster [DBG] 109.10 scrub starts 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.583405+0000 osd.7 (osd.7) 336 : cluster [DBG] 109.10 scrub ok 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:34.583405+0000 osd.7 (osd.7) 336 : cluster [DBG] 109.10 scrub ok 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: audit 2024-04-26T16:32:35.281196+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]': finished 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: audit 2024-04-26T16:32:35.281196+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]': finished 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.285247+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.285247+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: audit 2024-04-26T16:32:35.289516+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: audit 2024-04-26T16:32:35.289516+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.739 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: audit 2024-04-26T16:32:35.290502+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[21463]: audit 2024-04-26T16:32:35.290502+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.777871+0000 mgr.y (mgr.24401) 588 : cluster [DBG] pgmap v928: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:33.777871+0000 mgr.y (mgr.24401) 588 : cluster [DBG] pgmap v928: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.306342+0000 osd.1 (osd.1) 329 : cluster [DBG] 109.c scrub starts 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.306342+0000 osd.1 (osd.1) 329 : cluster [DBG] 109.c scrub starts 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.308419+0000 osd.1 (osd.1) 330 : cluster [DBG] 109.c scrub ok 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.308419+0000 osd.1 (osd.1) 330 : cluster [DBG] 109.c scrub ok 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.580381+0000 osd.7 (osd.7) 335 : cluster [DBG] 109.10 scrub starts 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.580381+0000 osd.7 (osd.7) 335 : cluster [DBG] 109.10 scrub starts 2024-04-26T16:32:35.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.583405+0000 osd.7 (osd.7) 336 : cluster [DBG] 109.10 scrub ok 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:34.583405+0000 osd.7 (osd.7) 336 : cluster [DBG] 109.10 scrub ok 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: audit 2024-04-26T16:32:35.281196+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]': finished 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: audit 2024-04-26T16:32:35.281196+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]': finished 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.285247+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.285247+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: audit 2024-04-26T16:32:35.289516+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: audit 2024-04-26T16:32:35.289516+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: audit 2024-04-26T16:32:35.290502+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.741 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:35 smithi050 bash[29234]: audit 2024-04-26T16:32:35.290502+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:35.754 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:35.857 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:35.881 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:35.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:35.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-26T16:32:35.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:36.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.777871+0000 mgr.y (mgr.24401) 588 : cluster [DBG] pgmap v928: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:32:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:33.777871+0000 mgr.y (mgr.24401) 588 : cluster [DBG] pgmap v928: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:32:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.306342+0000 osd.1 (osd.1) 329 : cluster [DBG] 109.c scrub starts 2024-04-26T16:32:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.306342+0000 osd.1 (osd.1) 329 : cluster [DBG] 109.c scrub starts 2024-04-26T16:32:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.308419+0000 osd.1 (osd.1) 330 : cluster [DBG] 109.c scrub ok 2024-04-26T16:32:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.308419+0000 osd.1 (osd.1) 330 : cluster [DBG] 109.c scrub ok 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.580381+0000 osd.7 (osd.7) 335 : cluster [DBG] 109.10 scrub starts 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.580381+0000 osd.7 (osd.7) 335 : cluster [DBG] 109.10 scrub starts 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.583405+0000 osd.7 (osd.7) 336 : cluster [DBG] 109.10 scrub ok 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:34.583405+0000 osd.7 (osd.7) 336 : cluster [DBG] 109.10 scrub ok 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: audit 2024-04-26T16:32:35.281196+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]': finished 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: audit 2024-04-26T16:32:35.281196+0000 mon.a (mon.0) 3113 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_count","val": "1"}]': finished 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.285247+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.285247+0000 mon.a (mon.0) 3114 : cluster [DBG] osdmap e603: 8 total, 8 up, 8 in 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: audit 2024-04-26T16:32:35.289516+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: audit 2024-04-26T16:32:35.289516+0000 mon.b (mon.1) 418 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: audit 2024-04-26T16:32:35.290502+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:35 smithi190 bash[23868]: audit 2024-04-26T16:32:35.290502+0000 mon.a (mon.0) 3115 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:36.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:36.136 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:36.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:36.258 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:36.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:36.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:36.388 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:36.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:36.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:36.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:36.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:36.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:36.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:36.780 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.597495+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.597495+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.599760+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.599760+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.136604+0000 osd.4 (osd.4) 315 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.136604+0000 osd.4 (osd.4) 315 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.138886+0000 osd.4 (osd.4) 316 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.138886+0000 osd.4 (osd.4) 316 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:32:36.781 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.269991+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.269991+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: audit 2024-04-26T16:32:36.283673+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: audit 2024-04-26T16:32:36.283673+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: audit 2024-04-26T16:32:36.288232+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: audit 2024-04-26T16:32:36.288232+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.292283+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.292283+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: audit 2024-04-26T16:32:36.299667+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:36.782 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:36 smithi190 bash[23868]: audit 2024-04-26T16:32:36.299667+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:36.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:36.912 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:36.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:37.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.597495+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.597495+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.599760+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.599760+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.136604+0000 osd.4 (osd.4) 315 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.136604+0000 osd.4 (osd.4) 315 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.138886+0000 osd.4 (osd.4) 316 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.138886+0000 osd.4 (osd.4) 316 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.269991+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.269991+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:37.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: audit 2024-04-26T16:32:36.283673+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: audit 2024-04-26T16:32:36.283673+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: audit 2024-04-26T16:32:36.288232+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: audit 2024-04-26T16:32:36.288232+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.292283+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.292283+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: audit 2024-04-26T16:32:36.299667+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[21463]: audit 2024-04-26T16:32:36.299667+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.597495+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.597495+0000 osd.7 (osd.7) 337 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:32:37.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.599760+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.599760+0000 osd.7 (osd.7) 338 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.136604+0000 osd.4 (osd.4) 315 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.136604+0000 osd.4 (osd.4) 315 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.138886+0000 osd.4 (osd.4) 316 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.138886+0000 osd.4 (osd.4) 316 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.269991+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.269991+0000 mon.a (mon.0) 3116 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: audit 2024-04-26T16:32:36.283673+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: audit 2024-04-26T16:32:36.283673+0000 mon.a (mon.0) 3117 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: audit 2024-04-26T16:32:36.288232+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: audit 2024-04-26T16:32:36.288232+0000 mon.b (mon.1) 419 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.292283+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.292283+0000 mon.a (mon.0) 3118 : cluster [DBG] osdmap e604: 8 total, 8 up, 8 in 2024-04-26T16:32:37.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: audit 2024-04-26T16:32:36.299667+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:36 smithi050 bash[29234]: audit 2024-04-26T16:32:36.299667+0000 mon.a (mon.0) 3119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]: dispatch 2024-04-26T16:32:37.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:37.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:37.155 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:32:37.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:37.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:37.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:37.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:37.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:37.437 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:37.541 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:37.569 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:37.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:37.700 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:37.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:37.831 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.778513+0000 mgr.y (mgr.24401) 589 : cluster [DBG] pgmap v931: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.778513+0000 mgr.y (mgr.24401) 589 : cluster [DBG] pgmap v931: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.937434+0000 osd.3 (osd.3) 371 : cluster [DBG] 109.1f deep-scrub starts 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.937434+0000 osd.3 (osd.3) 371 : cluster [DBG] 109.1f deep-scrub starts 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.945504+0000 osd.3 (osd.3) 372 : cluster [DBG] 109.1f deep-scrub ok 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:35.945504+0000 osd.3 (osd.3) 372 : cluster [DBG] 109.1f deep-scrub ok 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.547656+0000 osd.0 (osd.0) 363 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.547656+0000 osd.0 (osd.0) 363 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:37.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.547816+0000 osd.0 (osd.0) 364 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:36.547816+0000 osd.0 (osd.0) 364 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.098327+0000 osd.4 (osd.4) 317 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.098327+0000 osd.4 (osd.4) 317 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.100363+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.100363+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: audit 2024-04-26T16:32:37.286775+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]': finished 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: audit 2024-04-26T16:32:37.286775+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]': finished 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.298073+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.298073+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: audit 2024-04-26T16:32:37.340756+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: audit 2024-04-26T16:32:37.340756+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: audit 2024-04-26T16:32:37.341793+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[21463]: audit 2024-04-26T16:32:37.341793+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.778513+0000 mgr.y (mgr.24401) 589 : cluster [DBG] pgmap v931: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.778513+0000 mgr.y (mgr.24401) 589 : cluster [DBG] pgmap v931: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.937434+0000 osd.3 (osd.3) 371 : cluster [DBG] 109.1f deep-scrub starts 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.937434+0000 osd.3 (osd.3) 371 : cluster [DBG] 109.1f deep-scrub starts 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.945504+0000 osd.3 (osd.3) 372 : cluster [DBG] 109.1f deep-scrub ok 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:35.945504+0000 osd.3 (osd.3) 372 : cluster [DBG] 109.1f deep-scrub ok 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.547656+0000 osd.0 (osd.0) 363 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:37.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.547656+0000 osd.0 (osd.0) 363 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.547816+0000 osd.0 (osd.0) 364 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:36.547816+0000 osd.0 (osd.0) 364 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.098327+0000 osd.4 (osd.4) 317 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.098327+0000 osd.4 (osd.4) 317 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.100363+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.100363+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: audit 2024-04-26T16:32:37.286775+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]': finished 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: audit 2024-04-26T16:32:37.286775+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]': finished 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.298073+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.298073+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: audit 2024-04-26T16:32:37.340756+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: audit 2024-04-26T16:32:37.340756+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: audit 2024-04-26T16:32:37.341793+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:37 smithi050 bash[29234]: audit 2024-04-26T16:32:37.341793+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:37.945 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.778513+0000 mgr.y (mgr.24401) 589 : cluster [DBG] pgmap v931: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.778513+0000 mgr.y (mgr.24401) 589 : cluster [DBG] pgmap v931: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.937434+0000 osd.3 (osd.3) 371 : cluster [DBG] 109.1f deep-scrub starts 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.937434+0000 osd.3 (osd.3) 371 : cluster [DBG] 109.1f deep-scrub starts 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.945504+0000 osd.3 (osd.3) 372 : cluster [DBG] 109.1f deep-scrub ok 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:35.945504+0000 osd.3 (osd.3) 372 : cluster [DBG] 109.1f deep-scrub ok 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.547656+0000 osd.0 (osd.0) 363 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.547656+0000 osd.0 (osd.0) 363 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.547816+0000 osd.0 (osd.0) 364 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:36.547816+0000 osd.0 (osd.0) 364 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.098327+0000 osd.4 (osd.4) 317 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.098327+0000 osd.4 (osd.4) 317 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.100363+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:32:37.946 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.100363+0000 osd.4 (osd.4) 318 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: audit 2024-04-26T16:32:37.286775+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]': finished 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: audit 2024-04-26T16:32:37.286775+0000 mon.a (mon.0) 3120 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-98","var": "target_max_objects","val": "250"}]': finished 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.298073+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.298073+0000 mon.a (mon.0) 3121 : cluster [DBG] osdmap e605: 8 total, 8 up, 8 in 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: audit 2024-04-26T16:32:37.340756+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: audit 2024-04-26T16:32:37.340756+0000 mon.b (mon.1) 420 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: audit 2024-04-26T16:32:37.341793+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.947 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:37 smithi190 bash[23868]: audit 2024-04-26T16:32:37.341793+0000 mon.a (mon.0) 3122 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:37.959 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:37.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:38.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:38.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:38.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:38.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:38.224 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:38.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:38.338 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:32:38.339 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "" 2024-04-26T16:32:38.339 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:32:38.348 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:38.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:38.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:38.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:38.608 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:38.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:38.733 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:38.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:37.082169+0000 mgr.y (mgr.24401) 590 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:38.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:37.082169+0000 mgr.y (mgr.24401) 590 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:38.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.827502+0000 osd.0 (osd.0) 365 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:32:38.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.827502+0000 osd.0 (osd.0) 365 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:32:38.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.831520+0000 osd.0 (osd.0) 366 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.831520+0000 osd.0 (osd.0) 366 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.194326+0000 osd.5 (osd.5) 219 : cluster [DBG] 109.d scrub starts 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.194326+0000 osd.5 (osd.5) 219 : cluster [DBG] 109.d scrub starts 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.196571+0000 osd.5 (osd.5) 220 : cluster [DBG] 109.d scrub ok 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.196571+0000 osd.5 (osd.5) 220 : cluster [DBG] 109.d scrub ok 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.198229+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.198229+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.199621+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.199621+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:38.289363+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:38.289363+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:38.299960+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:38.299960+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.300838+0000 mon.a (mon.0) 3124 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-04-26T16:32:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.300838+0000 mon.a (mon.0) 3124 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-04-26T16:32:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:38.301765+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:38 smithi190 bash[23868]: audit 2024-04-26T16:32:38.301765+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:37.082169+0000 mgr.y (mgr.24401) 590 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:37.082169+0000 mgr.y (mgr.24401) 590 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.827502+0000 osd.0 (osd.0) 365 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.827502+0000 osd.0 (osd.0) 365 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.831520+0000 osd.0 (osd.0) 366 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.831520+0000 osd.0 (osd.0) 366 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.194326+0000 osd.5 (osd.5) 219 : cluster [DBG] 109.d scrub starts 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.194326+0000 osd.5 (osd.5) 219 : cluster [DBG] 109.d scrub starts 2024-04-26T16:32:38.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.196571+0000 osd.5 (osd.5) 220 : cluster [DBG] 109.d scrub ok 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.196571+0000 osd.5 (osd.5) 220 : cluster [DBG] 109.d scrub ok 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.198229+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.198229+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.199621+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.199621+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:38.289363+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:38.289363+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:38.299960+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:38.299960+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.300838+0000 mon.a (mon.0) 3124 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.300838+0000 mon.a (mon.0) 3124 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:38.301765+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[21463]: audit 2024-04-26T16:32:38.301765+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:37.082169+0000 mgr.y (mgr.24401) 590 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:37.082169+0000 mgr.y (mgr.24401) 590 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.827502+0000 osd.0 (osd.0) 365 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.827502+0000 osd.0 (osd.0) 365 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.831520+0000 osd.0 (osd.0) 366 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.831520+0000 osd.0 (osd.0) 366 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.194326+0000 osd.5 (osd.5) 219 : cluster [DBG] 109.d scrub starts 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.194326+0000 osd.5 (osd.5) 219 : cluster [DBG] 109.d scrub starts 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.196571+0000 osd.5 (osd.5) 220 : cluster [DBG] 109.d scrub ok 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.196571+0000 osd.5 (osd.5) 220 : cluster [DBG] 109.d scrub ok 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.198229+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.198229+0000 osd.6 (osd.6) 92 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:32:38.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.199621+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.199621+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:38.289363+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:38.289363+0000 mon.a (mon.0) 3123 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:38.299960+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:38.299960+0000 mon.b (mon.1) 421 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.300838+0000 mon.a (mon.0) 3124 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.300838+0000 mon.a (mon.0) 3124 : cluster [DBG] osdmap e606: 8 total, 8 up, 8 in 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:38.301765+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.852 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:38 smithi050 bash[29234]: audit 2024-04-26T16:32:38.301765+0000 mon.a (mon.0) 3125 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]: dispatch 2024-04-26T16:32:38.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:38.962 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:38.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:38.993 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:38.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:39.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:39.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:39.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:39.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:39.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:39.383 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:39.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:39.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:39.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:39.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:39.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:39.773 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:39.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.779356+0000 mgr.y (mgr.24401) 591 : cluster [DBG] pgmap v934: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:37.779356+0000 mgr.y (mgr.24401) 591 : cluster [DBG] pgmap v934: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.350460+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.d scrub starts 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.350460+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.d scrub starts 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.352592+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub ok 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.352592+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub ok 2024-04-26T16:32:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.583308+0000 osd.2 (osd.2) 305 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.583308+0000 osd.2 (osd.2) 305 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.586234+0000 osd.2 (osd.2) 306 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.586234+0000 osd.2 (osd.2) 306 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: audit 2024-04-26T16:32:39.292162+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: audit 2024-04-26T16:32:39.292162+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.299788+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.299788+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.362610+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.362610+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.365196+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.365196+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:32:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.779356+0000 mgr.y (mgr.24401) 591 : cluster [DBG] pgmap v934: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:37.779356+0000 mgr.y (mgr.24401) 591 : cluster [DBG] pgmap v934: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.350460+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.d scrub starts 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.350460+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.d scrub starts 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.352592+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub ok 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.352592+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub ok 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.583308+0000 osd.2 (osd.2) 305 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.583308+0000 osd.2 (osd.2) 305 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.586234+0000 osd.2 (osd.2) 306 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.586234+0000 osd.2 (osd.2) 306 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: audit 2024-04-26T16:32:39.292162+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: audit 2024-04-26T16:32:39.292162+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:39.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.299788+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-04-26T16:32:39.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.299788+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-04-26T16:32:39.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.362610+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:32:39.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.362610+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:32:39.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.365196+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:32:39.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:39 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.365196+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:32:39.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:39.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:40.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.779356+0000 mgr.y (mgr.24401) 591 : cluster [DBG] pgmap v934: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:37.779356+0000 mgr.y (mgr.24401) 591 : cluster [DBG] pgmap v934: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.350460+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.d scrub starts 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.350460+0000 osd.1 (osd.1) 331 : cluster [DBG] 2.d scrub starts 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.352592+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub ok 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.352592+0000 osd.1 (osd.1) 332 : cluster [DBG] 2.d scrub ok 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.583308+0000 osd.2 (osd.2) 305 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.583308+0000 osd.2 (osd.2) 305 : cluster [DBG] 109.1a scrub starts 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.586234+0000 osd.2 (osd.2) 306 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:32:40.021 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.586234+0000 osd.2 (osd.2) 306 : cluster [DBG] 109.1a scrub ok 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: audit 2024-04-26T16:32:39.292162+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: audit 2024-04-26T16:32:39.292162+0000 mon.a (mon.0) 3126 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-98"}]': finished 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.299788+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.299788+0000 mon.a (mon.0) 3127 : cluster [DBG] osdmap e607: 8 total, 8 up, 8 in 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.362610+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.362610+0000 osd.1 (osd.1) 333 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.365196+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:32:40.022 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:39 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.365196+0000 osd.1 (osd.1) 334 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:32:40.035 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:40.138 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:40.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:40.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.298 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:40.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:40.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:40.559 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:40.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.692 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:40.705 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:40.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.930783+0000 osd.3 (osd.3) 373 : cluster [DBG] 11.a scrub starts 2024-04-26T16:32:40.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.930783+0000 osd.3 (osd.3) 373 : cluster [DBG] 11.a scrub starts 2024-04-26T16:32:40.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.932705+0000 osd.3 (osd.3) 374 : cluster [DBG] 11.a scrub ok 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:38.932705+0000 osd.3 (osd.3) 374 : cluster [DBG] 11.a scrub ok 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.609556+0000 osd.2 (osd.2) 307 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.609556+0000 osd.2 (osd.2) 307 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.612332+0000 osd.2 (osd.2) 308 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.612332+0000 osd.2 (osd.2) 308 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.699660+0000 osd.7 (osd.7) 339 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.699660+0000 osd.7 (osd.7) 339 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.702019+0000 osd.7 (osd.7) 340 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.702019+0000 osd.7 (osd.7) 340 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.780126+0000 mgr.y (mgr.24401) 592 : cluster [DBG] pgmap v937: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.780126+0000 mgr.y (mgr.24401) 592 : cluster [DBG] pgmap v937: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.167732+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.f scrub starts 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.167732+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.f scrub starts 2024-04-26T16:32:40.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.170677+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.f scrub ok 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.170677+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.f scrub ok 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.302224+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.302224+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.406279+0000 osd.1 (osd.1) 335 : cluster [DBG] 5.c scrub starts 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.406279+0000 osd.1 (osd.1) 335 : cluster [DBG] 5.c scrub starts 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.409146+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub ok 2024-04-26T16:32:40.808 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:40 smithi190 bash[23868]: cluster 2024-04-26T16:32:40.409146+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub ok 2024-04-26T16:32:40.821 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:40.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:40.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:40.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:40.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.930783+0000 osd.3 (osd.3) 373 : cluster [DBG] 11.a scrub starts 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.930783+0000 osd.3 (osd.3) 373 : cluster [DBG] 11.a scrub starts 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.932705+0000 osd.3 (osd.3) 374 : cluster [DBG] 11.a scrub ok 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:38.932705+0000 osd.3 (osd.3) 374 : cluster [DBG] 11.a scrub ok 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.609556+0000 osd.2 (osd.2) 307 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.609556+0000 osd.2 (osd.2) 307 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.612332+0000 osd.2 (osd.2) 308 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.612332+0000 osd.2 (osd.2) 308 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.699660+0000 osd.7 (osd.7) 339 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:32:41.040 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.699660+0000 osd.7 (osd.7) 339 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.702019+0000 osd.7 (osd.7) 340 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.702019+0000 osd.7 (osd.7) 340 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.780126+0000 mgr.y (mgr.24401) 592 : cluster [DBG] pgmap v937: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.780126+0000 mgr.y (mgr.24401) 592 : cluster [DBG] pgmap v937: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.167732+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.f scrub starts 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.167732+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.f scrub starts 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.170677+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.f scrub ok 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.170677+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.f scrub ok 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.302224+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.302224+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.406279+0000 osd.1 (osd.1) 335 : cluster [DBG] 5.c scrub starts 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.406279+0000 osd.1 (osd.1) 335 : cluster [DBG] 5.c scrub starts 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.409146+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub ok 2024-04-26T16:32:41.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[21463]: cluster 2024-04-26T16:32:40.409146+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub ok 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.930783+0000 osd.3 (osd.3) 373 : cluster [DBG] 11.a scrub starts 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.930783+0000 osd.3 (osd.3) 373 : cluster [DBG] 11.a scrub starts 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.932705+0000 osd.3 (osd.3) 374 : cluster [DBG] 11.a scrub ok 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:38.932705+0000 osd.3 (osd.3) 374 : cluster [DBG] 11.a scrub ok 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.609556+0000 osd.2 (osd.2) 307 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.609556+0000 osd.2 (osd.2) 307 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.612332+0000 osd.2 (osd.2) 308 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.612332+0000 osd.2 (osd.2) 308 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.699660+0000 osd.7 (osd.7) 339 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.699660+0000 osd.7 (osd.7) 339 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.702019+0000 osd.7 (osd.7) 340 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:32:41.042 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.702019+0000 osd.7 (osd.7) 340 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.780126+0000 mgr.y (mgr.24401) 592 : cluster [DBG] pgmap v937: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.780126+0000 mgr.y (mgr.24401) 592 : cluster [DBG] pgmap v937: 228 pgs: 228 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.167732+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.f scrub starts 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.167732+0000 osd.4 (osd.4) 319 : cluster [DBG] 2.f scrub starts 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.170677+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.f scrub ok 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.170677+0000 osd.4 (osd.4) 320 : cluster [DBG] 2.f scrub ok 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.302224+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.302224+0000 mon.a (mon.0) 3128 : cluster [DBG] osdmap e608: 8 total, 8 up, 8 in 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.406279+0000 osd.1 (osd.1) 335 : cluster [DBG] 5.c scrub starts 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.406279+0000 osd.1 (osd.1) 335 : cluster [DBG] 5.c scrub starts 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.409146+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub ok 2024-04-26T16:32:41.043 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:40 smithi050 bash[29234]: cluster 2024-04-26T16:32:40.409146+0000 osd.1 (osd.1) 336 : cluster [DBG] 5.c scrub ok 2024-04-26T16:32:41.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:41.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:41.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:41.211 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:41.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/3482968950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8108380 msgr2=0x7ff7f8108760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/3482968950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8108380 0x7ff7f8108760 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8009990 tx=0x7ff7e8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/3482968950 shutdown_connections 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/3482968950 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7f8102cd0 0x7ff7f810f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/3482968950 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8108380 0x7ff7f8108760 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/3482968950 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/3482968950 >> 172.21.15.50:0/3482968950 conn(0x7ff7f8079290 msgr2=0x7ff7f80796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/3482968950 shutdown_connections 2024-04-26T16:32:41.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/3482968950 wait complete. 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 librados: starting msgr at 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 librados: starting objecter 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 Processor -- start 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- start start 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 librados: setting wanted keys 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 librados: calling monclient init 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f81990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8102cd0 0x7ff7f8199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7f8108380 0x7ff7f819d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8111980 con 0x7ff7f8102310 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f8111800 con 0x7ff7f8108380 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f8111b00 con 0x7ff7f8102cd0 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f81990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f5d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8102cd0 0x7ff7f8199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f81990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56836/0 (socket says 172.21.15.50:56836) 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 -- 172.21.15.50:0/1224841824 learned_addr learned my addr 172.21.15.50:0/1224841824 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:41.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7f8108380 0x7ff7f819d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8102cd0 msgr2=0x7ff7f8199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8102cd0 0x7ff7f8199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7f8108380 msgr2=0x7ff7f819d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff7f8108380 0x7ff7f819d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f819e0e0 con 0x7ff7f8102310 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff7f8102cd0 0x7ff7f8199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f81990d0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8000c00 tx=0x7ff7e8004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7e77fe700 10 client.?.objecter ms_handle_connect 0x7ff7f8102310 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7e77fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8027070 con 0x7ff7f8102310 2024-04-26T16:32:41.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f819e370 con 0x7ff7f8102310 2024-04-26T16:32:41.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff7e8022470 con 0x7ff7f8102310 2024-04-26T16:32:41.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 10 client.30831.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:41.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8017650 con 0x7ff7f8102310 2024-04-26T16:32:41.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f81a5c50 con 0x7ff7f8102310 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.220+0000 7ff7fcc2f700 1 librados: init done 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff7f806a110 con 0x7ff7f8102310 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff7e802a070 con 0x7ff7f8102310 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff7e0071f80 0x7ff7e0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(608..608 src has 496..608) ==== 8992+0+0 (secure 0 0 0) 0x7ff7e8092eb0 con 0x7ff7f8102310 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 osd_map(608..608 src has 496..608) 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 3 client.30831.objecter handle_osd_map got epochs [608,608] > 0 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 3 client.30831.objecter handle_osd_map decoding full epoch 608 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 20 client.30831.objecter dump_active .. 0 homeless 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v608) ==== 54+0+16 (secure 0 0 0) 0x7ff7e8013070 con 0x7ff7f8102310 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff7e0071f80 0x7ff7e0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 1 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff7f802d310 0x7ff7f8105f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 1 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 0 1 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 4 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 1 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff7f802e870 con 0x7ff7f802d310 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff7f802d310 0x7ff7f8105f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f802d310 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff7f802d310 0x7ff7f8105f30 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7e0071f80 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff7e0071f80 0x7ff7e0074440 secure :-1 s=READY pgs=3786 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80798e0 tx=0x7ff7ec009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f802e870 con 0x7ff7f802d310 2024-04-26T16:32:41.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 1: 0 ) 2024-04-26T16:32:41.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 1 = system:0 2024-04-26T16:32:41.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 4 2024-04-26T16:32:41.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.224+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 0 1 2024-04-26T16:32:41.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 2 2024-04-26T16:32:41.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 2 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f819a430 osd=0 3 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 0 2 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 4 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 2 2024-04-26T16:32:41.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.244+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f8106470 con 0x7ff7f802d310 2024-04-26T16:32:41.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f8106470 con 0x7ff7f802d310 2024-04-26T16:32:41.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 4 2024-04-26T16:32:41.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 0 2 2024-04-26T16:32:41.247 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:32:41.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff7f802dd70 0x7ff7f802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 1 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 4 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 3 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff7f802e6b0 con 0x7ff7f802dd70 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff7f802dd70 0x7ff7f802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff7f802dd70 0x7ff7f802e170 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.248+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f802dd70 2024-04-26T16:32:41.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.252+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7e805e9b0 con 0x7ff7f802dd70 2024-04-26T16:32:41.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.252+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 3: 0 ) 2024-04-26T16:32:41.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.252+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 3 = system:0 2024-04-26T16:32:41.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.252+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 4 2024-04-26T16:32:41.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.252+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 1 3 2024-04-26T16:32:41.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 4 2024-04-26T16:32:41.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 4 2024-04-26T16:32:41.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f802d6f0 osd=1 3 2024-04-26T16:32:41.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 1 4 2024-04-26T16:32:41.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 4 2024-04-26T16:32:41.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 4 2024-04-26T16:32:41.270 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f8063d00 con 0x7ff7f802dd70 2024-04-26T16:32:41.272 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:32:41.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f8063d00 con 0x7ff7f802dd70 2024-04-26T16:32:41.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 4 2024-04-26T16:32:41.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.272+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 1 4 2024-04-26T16:32:41.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 5 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff7f810bca0 0x7ff7f81aafc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 5 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 2 5 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 4 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 5 2024-04-26T16:32:41.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff7f81ade20 con 0x7ff7f810bca0 2024-04-26T16:32:41.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.276+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff7f810bca0 0x7ff7f81aafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.284+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff7f810bca0 0x7ff7f81aafc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.288+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f810bca0 2024-04-26T16:32:41.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.288+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f81ade20 con 0x7ff7f810bca0 2024-04-26T16:32:41.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.288+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 5: 0 ) 2024-04-26T16:32:41.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.288+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 5 = system:0 2024-04-26T16:32:41.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.288+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 4 2024-04-26T16:32:41.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.288+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 2 5 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 6 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 6 2024-04-26T16:32:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f8064100 osd=2 3 2024-04-26T16:32:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 2 6 2024-04-26T16:32:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 4 2024-04-26T16:32:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 6 2024-04-26T16:32:41.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f810b2d0 con 0x7ff7f810bca0 2024-04-26T16:32:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f810b2d0 con 0x7ff7f810bca0 2024-04-26T16:32:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 4 2024-04-26T16:32:41.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 2 6 2024-04-26T16:32:41.308 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 7 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff7f81a8b10 0x7ff7f81ac680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 7 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 3 7 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 4 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 7 2024-04-26T16:32:41.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.308+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff7f81b1810 con 0x7ff7f81a8b10 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff7f81a8b10 0x7ff7f81ac680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f81a8b10 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff7f81a8b10 0x7ff7f81ac680 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f81b1810 con 0x7ff7f81a8b10 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 7: 0 ) 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 7 = system:0 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 4 2024-04-26T16:32:41.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.312+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 3 7 2024-04-26T16:32:41.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:41.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 8 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 8 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f810b2d0 osd=3 3 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 3 8 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 4 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 8 2024-04-26T16:32:41.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f81a8080 con 0x7ff7f81a8b10 2024-04-26T16:32:41.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f81a8080 con 0x7ff7f81a8b10 2024-04-26T16:32:41.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 4 2024-04-26T16:32:41.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.332+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 3 8 2024-04-26T16:32:41.333 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:32:41.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 9 2024-04-26T16:32:41.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff7f81a7e30 0x7ff7f81b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 9 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 4 9 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 4 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 9 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff7f81ac290 con 0x7ff7f81a7e30 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff7f81a7e30 0x7ff7f81b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff7f81a7e30 0x7ff7f81b0070 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f81a7e30 2024-04-26T16:32:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f81ac290 con 0x7ff7f81a7e30 2024-04-26T16:32:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 9: 0 ) 2024-04-26T16:32:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 9 = system:0 2024-04-26T16:32:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 4 2024-04-26T16:32:41.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.336+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 4 9 2024-04-26T16:32:41.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 10 2024-04-26T16:32:41.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.353 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 10 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81b1840 osd=4 3 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 4 10 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 4 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 10 2024-04-26T16:32:41.354 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.356+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f81ac290 con 0x7ff7f81a7e30 2024-04-26T16:32:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f81ac290 con 0x7ff7f81a7e30 2024-04-26T16:32:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 4 2024-04-26T16:32:41.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 4 10 2024-04-26T16:32:41.358 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:32:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 11 2024-04-26T16:32:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff7f81b0ac0 0x7ff7f81b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 11 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 5 11 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 4 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 11 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff7f81b1540 con 0x7ff7f81b0ac0 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff7f81b0ac0 0x7ff7f81b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f81b0ac0 2024-04-26T16:32:41.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.360+0000 7ff7f5d9b700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff7f81b0ac0 0x7ff7f81b0f20 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.364+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f81b1540 con 0x7ff7f81b0ac0 2024-04-26T16:32:41.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.364+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 11: 0 ) 2024-04-26T16:32:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.364+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 11 = system:0 2024-04-26T16:32:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.364+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 4 2024-04-26T16:32:41.361 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.364+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 5 11 2024-04-26T16:32:41.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 12 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 12 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81ac290 osd=5 3 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 5 12 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 4 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 12 2024-04-26T16:32:41.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.380+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f81b1540 con 0x7ff7f81b0ac0 2024-04-26T16:32:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f81b1540 con 0x7ff7f81b0ac0 2024-04-26T16:32:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 4 2024-04-26T16:32:41.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 5 12 2024-04-26T16:32:41.384 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 13 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff7f81b5f40 0x7ff7f81b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 13 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 6 13 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 4 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 13 2024-04-26T16:32:41.385 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.384+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff7f81bb530 con 0x7ff7f81b5f40 2024-04-26T16:32:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff7f81b5f40 0x7ff7f81b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f81b5f40 2024-04-26T16:32:41.386 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7f6d9d700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff7f81b5f40 0x7ff7f81b63a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f81bb530 con 0x7ff7f81b5f40 2024-04-26T16:32:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 13: 0 ) 2024-04-26T16:32:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 13 = system:0 2024-04-26T16:32:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 4 2024-04-26T16:32:41.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.388+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 6 13 2024-04-26T16:32:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 14 2024-04-26T16:32:41.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 14 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81b1540 osd=6 3 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 6 14 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 4 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 14 2024-04-26T16:32:41.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.408+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f81b69f0 con 0x7ff7f81b5f40 2024-04-26T16:32:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f81b69f0 con 0x7ff7f81b5f40 2024-04-26T16:32:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 4 2024-04-26T16:32:41.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 6 14 2024-04-26T16:32:41.410 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 15 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff7f81b7110 0x7ff7f81b7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 15 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 7 15 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 4 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 15 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff7f81b7be0 con 0x7ff7f81b7110 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff7f81b7110 0x7ff7f81b7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7e77fe700 10 client.30831.objecter ms_handle_connect 0x7ff7f81b7110 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.412+0000 7ff7f659c700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff7f81b7110 0x7ff7f81b7550 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:41.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.416+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff7f81b7be0 con 0x7ff7f81b7110 2024-04-26T16:32:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.416+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 15: 0 ) 2024-04-26T16:32:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.416+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 15 = system:0 2024-04-26T16:32:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.416+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 4 2024-04-26T16:32:41.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.416+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 7 15 2024-04-26T16:32:41.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 10 client.30831.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign -1 16 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 20 client.30831.objecter _get_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_remove -1 16 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 20 client.30831.objecter get_session s=0x7ff7f81b69f0 osd=7 3 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 15 client.30831.objecter _session_command_op_assign 7 16 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 4 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 10 client.30831.objecter _send_command 16 2024-04-26T16:32:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.432+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff7f81b7be0 con 0x7ff7f81b7110 2024-04-26T16:32:41.435 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:32:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7e77fe700 1 -- 172.21.15.50:0/1224841824 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff7f81b7be0 con 0x7ff7f81b7110 2024-04-26T16:32:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7e77fe700 10 client.30831.objecter ms_dispatch 0x7ff7f8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7e77fe700 10 client.30831.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7e77fe700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 4 2024-04-26T16:32:41.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7e77fe700 15 client.30831.objecter _session_command_op_remove 7 16 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 librados: watch_flush enter 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 librados: watch_flush exit 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.0 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff7f802d310 msgr2=0x7ff7f8105f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff7f802d310 0x7ff7f8105f30 crc :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f819a430 osd=0 1 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.1 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff7f802dd70 msgr2=0x7ff7f802e170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff7f802dd70 0x7ff7f802e170 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f802d6f0 osd=1 1 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.2 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff7f810bca0 msgr2=0x7ff7f81aafc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff7f810bca0 0x7ff7f81aafc0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f8064100 osd=2 1 2024-04-26T16:32:41.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.3 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff7f81a8b10 msgr2=0x7ff7f81ac680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff7f81a8b10 0x7ff7f81ac680 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f810b2d0 osd=3 1 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.4 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff7f81a7e30 msgr2=0x7ff7f81b0070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff7f81a7e30 0x7ff7f81b0070 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1840 osd=4 1 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.5 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff7f81b0ac0 msgr2=0x7ff7f81b0f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff7f81b0ac0 0x7ff7f81b0f20 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81ac290 osd=5 1 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7f6d9d700 1 -- 172.21.15.50:0/1224841824 reap_dead start 2024-04-26T16:32:41.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.6 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff7f81b5f40 msgr2=0x7ff7f81b63a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7ff7f81b5f40 0x7ff7f81b63a0 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b1540 osd=6 1 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter close_session for osd.7 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff7f81b7110 msgr2=0x7ff7f81b7550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff7f81b7110 0x7ff7f81b7550 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter put_session s=0x7ff7f81b69f0 osd=7 1 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 20 client.30831.objecter shutdown clearing up homeless session... 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 10 client.30831.objecter successfully canceled tick 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7f6d9d700 1 -- 172.21.15.50:0/1224841824 reap_dead start 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7f6d9d700 1 -- 172.21.15.50:0/1224841824 reap_dead start 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7f6d9d700 1 -- 172.21.15.50:0/1224841824 reap_dead start 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff7e0071f80 msgr2=0x7ff7e0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff7e0071f80 0x7ff7e0074440 secure :-1 s=READY pgs=3786 cs=0 l=1 rev1=1 crypto rx=0x7ff7f80798e0 tx=0x7ff7ec009260 comp rx=0 tx=0).stop 2024-04-26T16:32:41.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 msgr2=0x7ff7f81990d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f81990d0 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7ff7e8000c00 tx=0x7ff7e8004790 comp rx=0 tx=0).stop 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 shutdown_connections 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff7f8102310 0x7ff7f81990d0 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 --2- 172.21.15.50:0/1224841824 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff7e0071f80 0x7ff7e0074440 unknown :-1 s=CLOSED pgs=3786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 >> 172.21.15.50:0/1224841824 conn(0x7ff7f8079290 msgr2=0x7ff7f8101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 shutdown_connections 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 -- 172.21.15.50:0/1224841824 wait complete. 2024-04-26T16:32:41.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:41.436+0000 7ff7fcc2f700 1 librados: shutdown 2024-04-26T16:32:41.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:32:41.441 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:32:41.470 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:41.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:41.600 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:41.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:41.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:41.835 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:41.864 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:41.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.954694+0000 osd.3 (osd.3) 375 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:32:41.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.954694+0000 osd.3 (osd.3) 375 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.956568+0000 osd.3 (osd.3) 376 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: cluster 2024-04-26T16:32:39.956568+0000 osd.3 (osd.3) 376 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: audit 2024-04-26T16:32:41.227530+0000 mon.a (mon.0) 3129 : audit [DBG] from='client.? 172.21.15.50:0/1224841824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: audit 2024-04-26T16:32:41.227530+0000 mon.a (mon.0) 3129 : audit [DBG] from='client.? 172.21.15.50:0/1224841824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.277716+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.277716+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: audit 2024-04-26T16:32:41.280666+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: audit 2024-04-26T16:32:41.280666+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: audit 2024-04-26T16:32:41.290853+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.956 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[29234]: audit 2024-04-26T16:32:41.290853+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.954694+0000 osd.3 (osd.3) 375 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.954694+0000 osd.3 (osd.3) 375 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.956568+0000 osd.3 (osd.3) 376 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: cluster 2024-04-26T16:32:39.956568+0000 osd.3 (osd.3) 376 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: audit 2024-04-26T16:32:41.227530+0000 mon.a (mon.0) 3129 : audit [DBG] from='client.? 172.21.15.50:0/1224841824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: audit 2024-04-26T16:32:41.227530+0000 mon.a (mon.0) 3129 : audit [DBG] from='client.? 172.21.15.50:0/1224841824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.277716+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.277716+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: audit 2024-04-26T16:32:41.280666+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: audit 2024-04-26T16:32:41.280666+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: audit 2024-04-26T16:32:41.290853+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.958 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:41 smithi050 bash[21463]: audit 2024-04-26T16:32:41.290853+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:41.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:41.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:41.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:41.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.954694+0000 osd.3 (osd.3) 375 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.954694+0000 osd.3 (osd.3) 375 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.956568+0000 osd.3 (osd.3) 376 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: cluster 2024-04-26T16:32:39.956568+0000 osd.3 (osd.3) 376 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: audit 2024-04-26T16:32:41.227530+0000 mon.a (mon.0) 3129 : audit [DBG] from='client.? 172.21.15.50:0/1224841824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: audit 2024-04-26T16:32:41.227530+0000 mon.a (mon.0) 3129 : audit [DBG] from='client.? 172.21.15.50:0/1224841824' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.277716+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.277716+0000 mon.a (mon.0) 3130 : cluster [DBG] osdmap e609: 8 total, 8 up, 8 in 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: audit 2024-04-26T16:32:41.280666+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: audit 2024-04-26T16:32:41.280666+0000 mon.b (mon.1) 422 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: audit 2024-04-26T16:32:41.290853+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:42.053 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:41 smithi190 bash[23868]: audit 2024-04-26T16:32:41.290853+0000 mon.a (mon.0) 3131 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:42.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:42.125 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:42.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:42.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:42.358 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:42.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:42.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:42.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:42.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:42.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:42.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:42.771 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:42.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:42.895 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:42.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:42.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:43.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.614991+0000 osd.2 (osd.2) 309 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.614991+0000 osd.2 (osd.2) 309 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.617206+0000 osd.2 (osd.2) 310 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.617206+0000 osd.2 (osd.2) 310 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.781506+0000 mgr.y (mgr.24401) 593 : cluster [DBG] pgmap v940: 228 pgs: 26 creating+peering, 6 unknown, 196 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.781506+0000 mgr.y (mgr.24401) 593 : cluster [DBG] pgmap v940: 228 pgs: 26 creating+peering, 6 unknown, 196 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.791663+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.791663+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.794100+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:41.794100+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.163661+0000 osd.5 (osd.5) 221 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.163661+0000 osd.5 (osd.5) 221 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:32:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.165734+0000 osd.5 (osd.5) 222 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.165734+0000 osd.5 (osd.5) 222 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.276702+0000 mon.a (mon.0) 3132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.276702+0000 mon.a (mon.0) 3132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.281366+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.281366+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.288440+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.288440+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.290125+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.290125+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.319806+0000 mon.a (mon.0) 3135 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.319806+0000 mon.a (mon.0) 3135 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.321362+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:42 smithi190 bash[23868]: audit 2024-04-26T16:32:42.321362+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:43.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.614991+0000 osd.2 (osd.2) 309 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:32:43.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.614991+0000 osd.2 (osd.2) 309 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.617206+0000 osd.2 (osd.2) 310 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.617206+0000 osd.2 (osd.2) 310 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.781506+0000 mgr.y (mgr.24401) 593 : cluster [DBG] pgmap v940: 228 pgs: 26 creating+peering, 6 unknown, 196 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.781506+0000 mgr.y (mgr.24401) 593 : cluster [DBG] pgmap v940: 228 pgs: 26 creating+peering, 6 unknown, 196 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.791663+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.791663+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.794100+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:41.794100+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.163661+0000 osd.5 (osd.5) 221 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.163661+0000 osd.5 (osd.5) 221 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.165734+0000 osd.5 (osd.5) 222 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.165734+0000 osd.5 (osd.5) 222 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:32:43.089 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: audit 2024-04-26T16:32:42.276702+0000 mon.a (mon.0) 3132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[29234]: audit 2024-04-26T16:32:42.276702+0000 mon.a (mon.0) 3132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.281366+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.281366+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.288440+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.288440+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.290125+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.614991+0000 osd.2 (osd.2) 309 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.614991+0000 osd.2 (osd.2) 309 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:32:43.090 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.617206+0000 osd.2 (osd.2) 310 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.617206+0000 osd.2 (osd.2) 310 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.781506+0000 mgr.y (mgr.24401) 593 : cluster [DBG] pgmap v940: 228 pgs: 26 creating+peering, 6 unknown, 196 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.781506+0000 mgr.y (mgr.24401) 593 : cluster [DBG] pgmap v940: 228 pgs: 26 creating+peering, 6 unknown, 196 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.791663+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.791663+0000 osd.0 (osd.0) 367 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.794100+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:41.794100+0000 osd.0 (osd.0) 368 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.163661+0000 osd.5 (osd.5) 221 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.163661+0000 osd.5 (osd.5) 221 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.165734+0000 osd.5 (osd.5) 222 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.165734+0000 osd.5 (osd.5) 222 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:32:43.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.276702+0000 mon.a (mon.0) 3132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.276702+0000 mon.a (mon.0) 3132 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.281366+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.281366+0000 mon.a (mon.0) 3133 : cluster [DBG] osdmap e610: 8 total, 8 up, 8 in 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.288440+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.288440+0000 mon.b (mon.1) 423 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.290125+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.290125+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.319806+0000 mon.a (mon.0) 3135 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.319806+0000 mon.a (mon.0) 3135 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.321362+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:43.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:42 smithi050 bash[21463]: audit 2024-04-26T16:32:42.321362+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:43.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:43.160 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:43.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:43.294 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:43.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.290125+0000 mon.a (mon.0) 3134 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:32:43.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.319806+0000 mon.a (mon.0) 3135 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:43.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.319806+0000 mon.a (mon.0) 3135 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:43.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.321362+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:43.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:42.321362+0000 mon.c (mon.2) 675 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:43.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:43.425 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:43.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:43.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:43.553 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:43.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:43.679 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:43.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.759562+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:42.759562+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.172894+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.172894+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.174889+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.174889+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:43.279707+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:43.279707+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.288018+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-04-26T16:32:43.794 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.288018+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:43.292461+0000 mon.b (mon.1) 424 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:43.292461+0000 mon.b (mon.1) 424 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:43.294367+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: audit 2024-04-26T16:32:43.294367+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.420891+0000 osd.1 (osd.1) 337 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.420891+0000 osd.1 (osd.1) 337 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.426508+0000 osd.1 (osd.1) 338 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.426508+0000 osd.1 (osd.1) 338 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:32:43.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.759562+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:42.759562+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.172894+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.172894+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.174889+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.174889+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: audit 2024-04-26T16:32:43.279707+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: audit 2024-04-26T16:32:43.279707+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.288018+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.288018+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: audit 2024-04-26T16:32:43.292461+0000 mon.b (mon.1) 424 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: audit 2024-04-26T16:32:43.292461+0000 mon.b (mon.1) 424 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: audit 2024-04-26T16:32:43.294367+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: audit 2024-04-26T16:32:43.294367+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:43.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.420891+0000 osd.1 (osd.1) 337 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:32:43.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.420891+0000 osd.1 (osd.1) 337 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:32:43.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.426508+0000 osd.1 (osd.1) 338 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:32:43.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:43 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.426508+0000 osd.1 (osd.1) 338 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:32:43.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:43.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:43.939 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:43.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:44.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.759562+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:42.759562+0000 mon.a (mon.0) 3136 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.172894+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.172894+0000 osd.4 (osd.4) 321 : cluster [DBG] 5.e scrub starts 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.174889+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.174889+0000 osd.4 (osd.4) 322 : cluster [DBG] 5.e scrub ok 2024-04-26T16:32:44.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: audit 2024-04-26T16:32:43.279707+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: audit 2024-04-26T16:32:43.279707+0000 mon.a (mon.0) 3137 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.288018+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.288018+0000 mon.a (mon.0) 3138 : cluster [DBG] osdmap e611: 8 total, 8 up, 8 in 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: audit 2024-04-26T16:32:43.292461+0000 mon.b (mon.1) 424 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: audit 2024-04-26T16:32:43.292461+0000 mon.b (mon.1) 424 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: audit 2024-04-26T16:32:43.294367+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: audit 2024-04-26T16:32:43.294367+0000 mon.a (mon.0) 3139 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.420891+0000 osd.1 (osd.1) 337 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.420891+0000 osd.1 (osd.1) 337 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.426508+0000 osd.1 (osd.1) 338 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:32:44.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:43 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.426508+0000 osd.1 (osd.1) 338 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:32:44.066 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:44.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:44.153 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:32:44.153 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:32:44.169 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-04-26T16:32:44.169 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:32:44.169 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-04-26T16:32:44.169 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-04-26T16:32:44.169 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-04-26T16:32:44.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-04-26T16:32:44.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:44.199 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:44.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:44.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.451 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:44.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.581 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:44.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:32:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:32:44.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:44.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:44.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:44.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:44.837 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:44.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.968 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:44.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.659897+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.659897+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.668362+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.668362+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.782282+0000 mgr.y (mgr.24401) 594 : cluster [DBG] pgmap v943: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:43.782282+0000 mgr.y (mgr.24401) 594 : cluster [DBG] pgmap v943: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.073094+0000 osd.6 (osd.6) 94 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:32:45.058 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.073094+0000 osd.6 (osd.6) 94 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.078095+0000 osd.6 (osd.6) 95 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.078095+0000 osd.6 (osd.6) 95 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.186295+0000 osd.5 (osd.5) 223 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.186295+0000 osd.5 (osd.5) 223 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.189868+0000 osd.5 (osd.5) 224 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.189868+0000 osd.5 (osd.5) 224 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: audit 2024-04-26T16:32:44.282292+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: audit 2024-04-26T16:32:44.282292+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.290075+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.290075+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: audit 2024-04-26T16:32:44.290238+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: audit 2024-04-26T16:32:44.290238+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: audit 2024-04-26T16:32:44.292635+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: audit 2024-04-26T16:32:44.292635+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.389830+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.389830+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.393206+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.393206+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.659897+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.659897+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.668362+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.668362+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.782282+0000 mgr.y (mgr.24401) 594 : cluster [DBG] pgmap v943: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:43.782282+0000 mgr.y (mgr.24401) 594 : cluster [DBG] pgmap v943: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.073094+0000 osd.6 (osd.6) 94 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.073094+0000 osd.6 (osd.6) 94 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:32:45.060 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.078095+0000 osd.6 (osd.6) 95 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.078095+0000 osd.6 (osd.6) 95 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.186295+0000 osd.5 (osd.5) 223 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.186295+0000 osd.5 (osd.5) 223 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.189868+0000 osd.5 (osd.5) 224 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.189868+0000 osd.5 (osd.5) 224 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: audit 2024-04-26T16:32:44.282292+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: audit 2024-04-26T16:32:44.282292+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.290075+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.290075+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: audit 2024-04-26T16:32:44.290238+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: audit 2024-04-26T16:32:44.290238+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: audit 2024-04-26T16:32:44.292635+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: audit 2024-04-26T16:32:44.292635+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.389830+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.389830+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:32:45.061 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.393206+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:32:45.062 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:44 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.393206+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:32:45.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.659897+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:32:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.659897+0000 osd.7 (osd.7) 341 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:32:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.668362+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:32:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.668362+0000 osd.7 (osd.7) 342 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:32:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.782282+0000 mgr.y (mgr.24401) 594 : cluster [DBG] pgmap v943: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:43.782282+0000 mgr.y (mgr.24401) 594 : cluster [DBG] pgmap v943: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.073094+0000 osd.6 (osd.6) 94 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.073094+0000 osd.6 (osd.6) 94 : cluster [DBG] 109.17 scrub starts 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.078095+0000 osd.6 (osd.6) 95 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.078095+0000 osd.6 (osd.6) 95 : cluster [DBG] 109.17 scrub ok 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.186295+0000 osd.5 (osd.5) 223 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.186295+0000 osd.5 (osd.5) 223 : cluster [DBG] 109.16 scrub starts 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.189868+0000 osd.5 (osd.5) 224 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.189868+0000 osd.5 (osd.5) 224 : cluster [DBG] 109.16 scrub ok 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: audit 2024-04-26T16:32:44.282292+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: audit 2024-04-26T16:32:44.282292+0000 mon.a (mon.0) 3140 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-6", "overlaypool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.290075+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.290075+0000 mon.a (mon.0) 3141 : cluster [DBG] osdmap e612: 8 total, 8 up, 8 in 2024-04-26T16:32:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: audit 2024-04-26T16:32:44.290238+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: audit 2024-04-26T16:32:44.290238+0000 mon.b (mon.1) 425 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: audit 2024-04-26T16:32:44.292635+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: audit 2024-04-26T16:32:44.292635+0000 mon.a (mon.0) 3142 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]: dispatch 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.389830+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.389830+0000 osd.1 (osd.1) 339 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.393206+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:32:45.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:44 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.393206+0000 osd.1 (osd.1) 340 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:32:45.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-26T16:32:45.096 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:45.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:45.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:45.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:45.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- 172.21.15.50:0/3997090253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 msgr2=0x7fa50010f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:45.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- 172.21.15.50:0/3997090253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50010f1d0 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800b0a0 tx=0x7fa4f800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:32:45.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- 172.21.15.50:0/3997090253 shutdown_connections 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- 172.21.15.50:0/3997090253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa500108390 0x7fa500108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- 172.21.15.50:0/3997090253 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50010f1d0 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- 172.21.15.50:0/3997090253 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500102320 0x7fa5001027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- 172.21.15.50:0/3997090253 >> 172.21.15.50:0/3997090253 conn(0x7fa5000792d0 msgr2=0x7fa5000796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- 172.21.15.50:0/3997090253 shutdown_connections 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- 172.21.15.50:0/3997090253 wait complete. 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 librados: starting msgr at 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 librados: starting objecter 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 Processor -- start 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- start start 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 librados: setting wanted keys 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 librados: calling monclient init 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa500102320 0x7fa50019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50019b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:45.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 0x7fa50019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa5001119a0 con 0x7fa500102ce0 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa500111820 con 0x7fa500102320 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.240+0000 7fa5082d0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa500111b20 con 0x7fa500108390 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50686d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 0x7fa50019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50019b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50686d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 0x7fa50019fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:39914/0 (socket says 172.21.15.50:39914) 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50686d700 1 -- 172.21.15.50:0/2419380953 learned_addr learned my addr 172.21.15.50:0/2419380953 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:45.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50606c700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa500102320 0x7fa50019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:45.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 -- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 msgr2=0x7fa50019fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:45.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 0x7fa50019fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:45.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 -- 172.21.15.50:0/2419380953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa500102320 msgr2=0x7fa50019b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa500102320 0x7fa50019b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 -- 172.21.15.50:0/2419380953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5001a02c0 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50686d700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 0x7fa50019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50586b700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50019b7f0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000e9f0 tx=0x7fa4f000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 10 client.?.objecter ms_handle_connect 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f000cb70 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa4f000ee90 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5001a05b0 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0003b80 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa5082d0700 10 client.30837.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5001a7e30 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa5082d0700 1 librados: init done 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa50006a110 con 0x7fa500102ce0 2024-04-26T16:32:45.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa4f0003d20 con 0x7fa500102ce0 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa4ec071f80 0x7fa4ec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(612..612 src has 496..612) ==== 9386+0+0 (secure 0 0 0) 0x7fa4f0091b10 con 0x7fa500102ce0 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 10 client.30837.objecter ms_dispatch 0x7fa500001550 osd_map(612..612 src has 496..612) 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 3 client.30837.objecter handle_osd_map got epochs [612,612] > 0 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 3 client.30837.objecter handle_osd_map decoding full epoch 612 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50606c700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa4ec071f80 0x7fa4ec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 20 client.30837.objecter dump_active .. 0 homeless 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa4f77fe700 10 client.30837.objecter ms_handle_connect 0x7fa4ec071f80 2024-04-26T16:32:45.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.244+0000 7fa50606c700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa4ec071f80 0x7fa4ec074440 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc005950 tx=0x7fa4fc00b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:45.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.252+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fa4f005d4c0 con 0x7fa500102ce0 2024-04-26T16:32:45.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:45.347 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:45.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:45.474 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:45.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:45.492+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fa5001a8000 con 0x7fa500102ce0 2024-04-26T16:32:45.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:32:45.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:45.605 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:45.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:45.734 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:45.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.645170+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.645170+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.660561+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:44.660561+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:45.282397+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:45.282397+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.283649+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]': finished 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.283649+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]': finished 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.286861+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.286861+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:45.287587+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-04-26T16:32:45.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: cluster 2024-04-26T16:32:45.287587+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.288800+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.288800+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.496348+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[21463]: audit 2024-04-26T16:32:45.496348+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.645170+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.645170+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.660561+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:44.660561+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:45.282397+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:45.282397+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:45.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.283649+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]': finished 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.283649+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]': finished 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.286861+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.286861+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:45.287587+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: cluster 2024-04-26T16:32:45.287587+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.288800+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.288800+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.496348+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:32:45.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:45 smithi050 bash[29234]: audit 2024-04-26T16:32:45.496348+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:32:45.861 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:45.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:45.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.645170+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.645170+0000 osd.7 (osd.7) 343 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.660561+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:44.660561+0000 osd.7 (osd.7) 344 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:45.282397+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:45.282397+0000 mon.a (mon.0) 3143 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.283649+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]': finished 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.283649+0000 mon.a (mon.0) 3144 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-100", "mode": "writeback"}]': finished 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.286861+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.286861+0000 mon.b (mon.1) 426 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:45.287587+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: cluster 2024-04-26T16:32:45.287587+0000 mon.a (mon.0) 3145 : cluster [DBG] osdmap e613: 8 total, 8 up, 8 in 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.288800+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.288800+0000 mon.a (mon.0) 3146 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.496348+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:32:45.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:45 smithi190 bash[23868]: audit 2024-04-26T16:32:45.496348+0000 mon.a (mon.0) 3147 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:32:45.993 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:45.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:46.096 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:46.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:46.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:46.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:46.252 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:46.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:46.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.284+0000 7fa4f77fe700 1 -- 172.21.15.50:0/2419380953 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v614) ==== 87+0+0 (secure 0 0 0) 0x7fa4f0061170 con 0x7fa500102ce0 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 10 librados: watch_flush enter 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 10 librados: watch_flush exit 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 20 client.30837.objecter shutdown clearing up homeless session... 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 10 client.30837.objecter successfully canceled tick 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa4ec071f80 msgr2=0x7fa4ec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa4ec071f80 0x7fa4ec074440 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc005950 tx=0x7fa4fc00b410 comp rx=0 tx=0).stop 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 msgr2=0x7fa50019b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:46.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50019b7f0 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000e9f0 tx=0x7fa4f000ed00 comp rx=0 tx=0).stop 2024-04-26T16:32:46.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 shutdown_connections 2024-04-26T16:32:46.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa4ec071f80 0x7fa4ec074440 unknown :-1 s=CLOSED pgs=3787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:46.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa500102320 0x7fa50019b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:46.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa500108390 0x7fa50019fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:46.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 --2- 172.21.15.50:0/2419380953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa500102ce0 0x7fa50019b7f0 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:46.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.292+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 >> 172.21.15.50:0/2419380953 conn(0x7fa5000792d0 msgr2=0x7fa50006b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:46.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.296+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 shutdown_connections 2024-04-26T16:32:46.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.296+0000 7fa5082d0700 1 -- 172.21.15.50:0/2419380953 wait complete. 2024-04-26T16:32:46.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:46.296+0000 7fa5082d0700 1 librados: shutdown 2024-04-26T16:32:46.295 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:32:46.354 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:46.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:46.479 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:46.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:46.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:46.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:46.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:46.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.766 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:46.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:46.867 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:46.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:46.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:46.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:47.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:47.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:47.156 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:47.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:47.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:47.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:45.782747+0000 mgr.y (mgr.24401) 595 : cluster [DBG] pgmap v946: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:47.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:45.782747+0000 mgr.y (mgr.24401) 595 : cluster [DBG] pgmap v946: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:47.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.118666+0000 osd.6 (osd.6) 96 : cluster [DBG] 11.1c deep-scrub starts 2024-04-26T16:32:47.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.118666+0000 osd.6 (osd.6) 96 : cluster [DBG] 11.1c deep-scrub starts 2024-04-26T16:32:47.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.121100+0000 osd.6 (osd.6) 97 : cluster [DBG] 11.1c deep-scrub ok 2024-04-26T16:32:47.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.121100+0000 osd.6 (osd.6) 97 : cluster [DBG] 11.1c deep-scrub ok 2024-04-26T16:32:47.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.166927+0000 osd.4 (osd.4) 323 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:32:47.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.166927+0000 osd.4 (osd.4) 323 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.169188+0000 osd.4 (osd.4) 324 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.169188+0000 osd.4 (osd.4) 324 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.284440+0000 mon.a (mon.0) 3148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.284440+0000 mon.a (mon.0) 3148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.286773+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.286773+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:32:47.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.287028+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:32:47.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.287028+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:32:47.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.294467+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-04-26T16:32:47.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: cluster 2024-04-26T16:32:46.294467+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-04-26T16:32:47.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.295312+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.295312+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.385 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.297110+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.386 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[21463]: audit 2024-04-26T16:32:46.297110+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:45.782747+0000 mgr.y (mgr.24401) 595 : cluster [DBG] pgmap v946: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:47.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:45.782747+0000 mgr.y (mgr.24401) 595 : cluster [DBG] pgmap v946: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:47.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.118666+0000 osd.6 (osd.6) 96 : cluster [DBG] 11.1c deep-scrub starts 2024-04-26T16:32:47.386 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.118666+0000 osd.6 (osd.6) 96 : cluster [DBG] 11.1c deep-scrub starts 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.121100+0000 osd.6 (osd.6) 97 : cluster [DBG] 11.1c deep-scrub ok 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.121100+0000 osd.6 (osd.6) 97 : cluster [DBG] 11.1c deep-scrub ok 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.166927+0000 osd.4 (osd.4) 323 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.166927+0000 osd.4 (osd.4) 323 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.169188+0000 osd.4 (osd.4) 324 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.169188+0000 osd.4 (osd.4) 324 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:32:47.387 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.284440+0000 mon.a (mon.0) 3148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:47.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.284440+0000 mon.a (mon.0) 3148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:47.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.286773+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:32:47.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.286773+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:32:47.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.287028+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:32:47.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.287028+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:32:47.388 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.294467+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-04-26T16:32:47.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: cluster 2024-04-26T16:32:46.294467+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-04-26T16:32:47.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.295312+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.295312+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.297110+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.389 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:47 smithi050 bash[29234]: audit 2024-04-26T16:32:46.297110+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:47.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:32:47.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:45.782747+0000 mgr.y (mgr.24401) 595 : cluster [DBG] pgmap v946: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:47.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:45.782747+0000 mgr.y (mgr.24401) 595 : cluster [DBG] pgmap v946: 228 pgs: 26 creating+peering, 202 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:47.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.118666+0000 osd.6 (osd.6) 96 : cluster [DBG] 11.1c deep-scrub starts 2024-04-26T16:32:47.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.118666+0000 osd.6 (osd.6) 96 : cluster [DBG] 11.1c deep-scrub starts 2024-04-26T16:32:47.404 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.121100+0000 osd.6 (osd.6) 97 : cluster [DBG] 11.1c deep-scrub ok 2024-04-26T16:32:47.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.121100+0000 osd.6 (osd.6) 97 : cluster [DBG] 11.1c deep-scrub ok 2024-04-26T16:32:47.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.166927+0000 osd.4 (osd.4) 323 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:32:47.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.166927+0000 osd.4 (osd.4) 323 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:32:47.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.169188+0000 osd.4 (osd.4) 324 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:32:47.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.169188+0000 osd.4 (osd.4) 324 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:32:47.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.284440+0000 mon.a (mon.0) 3148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.284440+0000 mon.a (mon.0) 3148 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.286773+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.286773+0000 mon.a (mon.0) 3149 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.287028+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.287028+0000 mon.a (mon.0) 3150 : audit [INF] from='client.? 172.21.15.50:0/2419380953' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.294467+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-04-26T16:32:47.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: cluster 2024-04-26T16:32:46.294467+0000 mon.a (mon.0) 3151 : cluster [DBG] osdmap e614: 8 total, 8 up, 8 in 2024-04-26T16:32:47.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.295312+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.295312+0000 mon.b (mon.1) 427 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.297110+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:47 smithi190 bash[23868]: audit 2024-04-26T16:32:46.297110+0000 mon.a (mon.0) 3152 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:32:47.419 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:47.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:47.548 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:47.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:47.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:47.677 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:47.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:47.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:47.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:47.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:47.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:47.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:48.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:48.064 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:48.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:48.192 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:48.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:48.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:48.323 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:48.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:48.451 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:48.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.090568+0000 mgr.y (mgr.24401) 596 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.090568+0000 mgr.y (mgr.24401) 596 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.292803+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.292803+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: cluster 2024-04-26T16:32:47.297393+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: cluster 2024-04-26T16:32:47.297393+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-04-26T16:32:48.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.300004+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.300004+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.301255+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:48 smithi190 bash[23868]: audit 2024-04-26T16:32:47.301255+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.579 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.090568+0000 mgr.y (mgr.24401) 596 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.090568+0000 mgr.y (mgr.24401) 596 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.292803+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.292803+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: cluster 2024-04-26T16:32:47.297393+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: cluster 2024-04-26T16:32:47.297393+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.300004+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.300004+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.301255+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[21463]: audit 2024-04-26T16:32:47.301255+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.090568+0000 mgr.y (mgr.24401) 596 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.090568+0000 mgr.y (mgr.24401) 596 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.292803+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.292803+0000 mon.a (mon.0) 3153 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: cluster 2024-04-26T16:32:47.297393+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: cluster 2024-04-26T16:32:47.297393+0000 mon.a (mon.0) 3154 : cluster [DBG] osdmap e615: 8 total, 8 up, 8 in 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.300004+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.300004+0000 mon.b (mon.1) 428 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.301255+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:48 smithi050 bash[29234]: audit 2024-04-26T16:32:47.301255+0000 mon.a (mon.0) 3155 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:32:48.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:48.706 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:48.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:48.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:48.931 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:48.931 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:32:48.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:48.966 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:48.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:49.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:49.096 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:49.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:49.225 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:49.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:49.356 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:49.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:49.472 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: cluster 2024-04-26T16:32:47.783538+0000 mgr.y (mgr.24401) 597 : cluster [DBG] pgmap v949: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: cluster 2024-04-26T16:32:47.783538+0000 mgr.y (mgr.24401) 597 : cluster [DBG] pgmap v949: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: cluster 2024-04-26T16:32:48.293217+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: cluster 2024-04-26T16:32:48.293217+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: audit 2024-04-26T16:32:48.295497+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: audit 2024-04-26T16:32:48.295497+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: cluster 2024-04-26T16:32:48.303821+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: cluster 2024-04-26T16:32:48.303821+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: audit 2024-04-26T16:32:48.319350+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: audit 2024-04-26T16:32:48.319350+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: audit 2024-04-26T16:32:48.320801+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:49 smithi190 bash[23868]: audit 2024-04-26T16:32:48.320801+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.488 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:49.578 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:32:49.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: cluster 2024-04-26T16:32:47.783538+0000 mgr.y (mgr.24401) 597 : cluster [DBG] pgmap v949: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: cluster 2024-04-26T16:32:47.783538+0000 mgr.y (mgr.24401) 597 : cluster [DBG] pgmap v949: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: cluster 2024-04-26T16:32:48.293217+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: cluster 2024-04-26T16:32:48.293217+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: audit 2024-04-26T16:32:48.295497+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: audit 2024-04-26T16:32:48.295497+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: cluster 2024-04-26T16:32:48.303821+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: cluster 2024-04-26T16:32:48.303821+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: audit 2024-04-26T16:32:48.319350+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: audit 2024-04-26T16:32:48.319350+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: audit 2024-04-26T16:32:48.320801+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[29234]: audit 2024-04-26T16:32:48.320801+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: cluster 2024-04-26T16:32:47.783538+0000 mgr.y (mgr.24401) 597 : cluster [DBG] pgmap v949: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: cluster 2024-04-26T16:32:47.783538+0000 mgr.y (mgr.24401) 597 : cluster [DBG] pgmap v949: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: cluster 2024-04-26T16:32:48.293217+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: cluster 2024-04-26T16:32:48.293217+0000 mon.a (mon.0) 3156 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: audit 2024-04-26T16:32:48.295497+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: audit 2024-04-26T16:32:48.295497+0000 mon.a (mon.0) 3157 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: cluster 2024-04-26T16:32:48.303821+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: cluster 2024-04-26T16:32:48.303821+0000 mon.a (mon.0) 3158 : cluster [DBG] osdmap e616: 8 total, 8 up, 8 in 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: audit 2024-04-26T16:32:48.319350+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: audit 2024-04-26T16:32:48.319350+0000 mon.b (mon.1) 429 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: audit 2024-04-26T16:32:48.320801+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:49 smithi050 bash[21463]: audit 2024-04-26T16:32:48.320801+0000 mon.a (mon.0) 3159 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-04-26T16:32:49.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:49.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:49.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:49.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:49.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:49.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:49.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:49.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:50.004 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:50.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:50.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:50.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:50.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:50.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: audit 2024-04-26T16:32:49.324179+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-26T16:32:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: audit 2024-04-26T16:32:49.324179+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-26T16:32:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: cluster 2024-04-26T16:32:49.332346+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-04-26T16:32:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: cluster 2024-04-26T16:32:49.332346+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-04-26T16:32:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: audit 2024-04-26T16:32:49.398571+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.375 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: audit 2024-04-26T16:32:49.398571+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: audit 2024-04-26T16:32:49.399680+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[21463]: audit 2024-04-26T16:32:49.399680+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: audit 2024-04-26T16:32:49.324179+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: audit 2024-04-26T16:32:49.324179+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: cluster 2024-04-26T16:32:49.332346+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: cluster 2024-04-26T16:32:49.332346+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: audit 2024-04-26T16:32:49.398571+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: audit 2024-04-26T16:32:49.398571+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: audit 2024-04-26T16:32:49.399680+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.376 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:50 smithi050 bash[29234]: audit 2024-04-26T16:32:49.399680+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:50.491 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:50.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:50.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: audit 2024-04-26T16:32:49.324179+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-26T16:32:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: audit 2024-04-26T16:32:49.324179+0000 mon.a (mon.0) 3160 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-04-26T16:32:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: cluster 2024-04-26T16:32:49.332346+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-04-26T16:32:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: cluster 2024-04-26T16:32:49.332346+0000 mon.a (mon.0) 3161 : cluster [DBG] osdmap e617: 8 total, 8 up, 8 in 2024-04-26T16:32:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: audit 2024-04-26T16:32:49.398571+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: audit 2024-04-26T16:32:49.398571+0000 mon.b (mon.1) 430 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: audit 2024-04-26T16:32:49.399680+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:50 smithi190 bash[23868]: audit 2024-04-26T16:32:49.399680+0000 mon.a (mon.0) 3162 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:50.646 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:50.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:50.776 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:50.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:50.907 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:50.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:51.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:51.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:51.121 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:51.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:51.154 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:51.257 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:51.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:51.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:51.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:49.783978+0000 mgr.y (mgr.24401) 598 : cluster [DBG] pgmap v952: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:51.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:49.783978+0000 mgr.y (mgr.24401) 598 : cluster [DBG] pgmap v952: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:51.408 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:50.333167+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:50.333167+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:50.343321+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:50.343321+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:50.344459+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:50.344459+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:50.345120+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:50.345120+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:51.276138+0000 mon.a (mon.0) 3166 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:51.276138+0000 mon.a (mon.0) 3166 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:51.279714+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: audit 2024-04-26T16:32:51.279714+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:51.284489+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-04-26T16:32:51.409 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[21463]: cluster 2024-04-26T16:32:51.284489+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:49.783978+0000 mgr.y (mgr.24401) 598 : cluster [DBG] pgmap v952: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:49.783978+0000 mgr.y (mgr.24401) 598 : cluster [DBG] pgmap v952: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:50.333167+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:50.333167+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:50.343321+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:50.343321+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:50.344459+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:50.344459+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:50.345120+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:50.345120+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:51.276138+0000 mon.a (mon.0) 3166 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:51.410 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:51.276138+0000 mon.a (mon.0) 3166 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:51.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:51.279714+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:51.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: audit 2024-04-26T16:32:51.279714+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:51.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:51.284489+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-04-26T16:32:51.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:51 smithi050 bash[29234]: cluster 2024-04-26T16:32:51.284489+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-04-26T16:32:51.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:51.524 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:51.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:49.783978+0000 mgr.y (mgr.24401) 598 : cluster [DBG] pgmap v952: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:49.783978+0000 mgr.y (mgr.24401) 598 : cluster [DBG] pgmap v952: 228 pgs: 228 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:50.333167+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:50.333167+0000 mon.a (mon.0) 3163 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]': finished 2024-04-26T16:32:51.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:50.343321+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:50.343321+0000 mon.b (mon.1) 431 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:50.344459+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:50.344459+0000 mon.a (mon.0) 3164 : cluster [DBG] osdmap e618: 8 total, 8 up, 8 in 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:50.345120+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:50.345120+0000 mon.a (mon.0) 3165 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]: dispatch 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:51.276138+0000 mon.a (mon.0) 3166 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:51.276138+0000 mon.a (mon.0) 3166 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:51.279714+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: audit 2024-04-26T16:32:51.279714+0000 mon.a (mon.0) 3167 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-100"}]': finished 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:51.284489+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-04-26T16:32:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:51 smithi190 bash[23868]: cluster 2024-04-26T16:32:51.284489+0000 mon.a (mon.0) 3168 : cluster [DBG] osdmap e619: 8 total, 8 up, 8 in 2024-04-26T16:32:51.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:51.682 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:51.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:51.812 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:51.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:51.942 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:51.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:52.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:52.072 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:52.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:52.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:52.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:52.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:52.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:52.331 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:52.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:52.462 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:52.565 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:52.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:52.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:52.720 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:52.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:52.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:52.847 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:52.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:52.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:52.976 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:52.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:53.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:53.108 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:53.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:53.242 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:53.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:53.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:53.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:53.489 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:53.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[29234]: cluster 2024-04-26T16:32:51.786113+0000 mgr.y (mgr.24401) 599 : cluster [DBG] pgmap v955: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[29234]: cluster 2024-04-26T16:32:51.786113+0000 mgr.y (mgr.24401) 599 : cluster [DBG] pgmap v955: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[29234]: cluster 2024-04-26T16:32:52.345285+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[29234]: cluster 2024-04-26T16:32:52.345285+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[21463]: cluster 2024-04-26T16:32:51.786113+0000 mgr.y (mgr.24401) 599 : cluster [DBG] pgmap v955: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[21463]: cluster 2024-04-26T16:32:51.786113+0000 mgr.y (mgr.24401) 599 : cluster [DBG] pgmap v955: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[21463]: cluster 2024-04-26T16:32:52.345285+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-04-26T16:32:53.587 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:53 smithi050 bash[21463]: cluster 2024-04-26T16:32:52.345285+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-04-26T16:32:53.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:53.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:53 smithi190 bash[23868]: cluster 2024-04-26T16:32:51.786113+0000 mgr.y (mgr.24401) 599 : cluster [DBG] pgmap v955: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:53.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:53 smithi190 bash[23868]: cluster 2024-04-26T16:32:51.786113+0000 mgr.y (mgr.24401) 599 : cluster [DBG] pgmap v955: 228 pgs: 228 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-26T16:32:53.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:53 smithi190 bash[23868]: cluster 2024-04-26T16:32:52.345285+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-04-26T16:32:53.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:53 smithi190 bash[23868]: cluster 2024-04-26T16:32:52.345285+0000 mon.a (mon.0) 3169 : cluster [DBG] osdmap e620: 8 total, 8 up, 8 in 2024-04-26T16:32:53.645 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:53.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:53.775 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:53.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:53.905 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:53.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:53.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:54.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:54.033 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:54.136 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:54.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:54.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:54.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:54.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:54.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:54.319 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:32:54.319 INFO:teuthology.orchestra.run.smithi190.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-04-26T16:32:54.319 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:32:54.325 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:32:54.325 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:32:54.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21463]: cluster 2024-04-26T16:32:53.357000+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-04-26T16:32:54.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21463]: cluster 2024-04-26T16:32:53.357000+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-04-26T16:32:54.357 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21463]: audit 2024-04-26T16:32:53.366271+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21463]: audit 2024-04-26T16:32:53.366271+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21463]: audit 2024-04-26T16:32:53.371845+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.358 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21463]: audit 2024-04-26T16:32:53.371845+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:54 smithi190 bash[23868]: cluster 2024-04-26T16:32:53.357000+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-04-26T16:32:54.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:54 smithi190 bash[23868]: cluster 2024-04-26T16:32:53.357000+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-04-26T16:32:54.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:54 smithi190 bash[23868]: audit 2024-04-26T16:32:53.366271+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.367 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:54 smithi190 bash[23868]: audit 2024-04-26T16:32:53.366271+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:54 smithi190 bash[23868]: audit 2024-04-26T16:32:53.371845+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.368 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:54 smithi190 bash[23868]: audit 2024-04-26T16:32:53.371845+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:54.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[29234]: cluster 2024-04-26T16:32:53.357000+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-04-26T16:32:54.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[29234]: cluster 2024-04-26T16:32:53.357000+0000 mon.a (mon.0) 3170 : cluster [DBG] osdmap e621: 8 total, 8 up, 8 in 2024-04-26T16:32:54.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[29234]: audit 2024-04-26T16:32:53.366271+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[29234]: audit 2024-04-26T16:32:53.366271+0000 mon.b (mon.1) 432 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[29234]: audit 2024-04-26T16:32:53.371845+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.411 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[29234]: audit 2024-04-26T16:32:53.371845+0000 mon.a (mon.0) 3171 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:32:54.415 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:54.447 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:32:54.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:54.542 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:54.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:32:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:32:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:32:54.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:54.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:54.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:54.801 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:54.905 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:54.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:54.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:55.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:55.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:55.169 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:55.196 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:55.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:55.328 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:55.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:55.457 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:55.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: cluster 2024-04-26T16:32:53.786903+0000 mgr.y (mgr.24401) 600 : cluster [DBG] pgmap v958: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: cluster 2024-04-26T16:32:53.786903+0000 mgr.y (mgr.24401) 600 : cluster [DBG] pgmap v958: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.358730+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.358730+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: cluster 2024-04-26T16:32:54.374687+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: cluster 2024-04-26T16:32:54.374687+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-04-26T16:32:55.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.382111+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.382111+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.391179+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.391179+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.394495+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[29234]: audit 2024-04-26T16:32:54.394495+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: cluster 2024-04-26T16:32:53.786903+0000 mgr.y (mgr.24401) 600 : cluster [DBG] pgmap v958: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: cluster 2024-04-26T16:32:53.786903+0000 mgr.y (mgr.24401) 600 : cluster [DBG] pgmap v958: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.358730+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:55.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.358730+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: cluster 2024-04-26T16:32:54.374687+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: cluster 2024-04-26T16:32:54.374687+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.382111+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.382111+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.391179+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.391179+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.394495+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:55 smithi050 bash[21463]: audit 2024-04-26T16:32:54.394495+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: cluster 2024-04-26T16:32:53.786903+0000 mgr.y (mgr.24401) 600 : cluster [DBG] pgmap v958: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: cluster 2024-04-26T16:32:53.786903+0000 mgr.y (mgr.24401) 600 : cluster [DBG] pgmap v958: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.358730+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.358730+0000 mon.a (mon.0) 3172 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: cluster 2024-04-26T16:32:54.374687+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: cluster 2024-04-26T16:32:54.374687+0000 mon.a (mon.0) 3173 : cluster [DBG] osdmap e622: 8 total, 8 up, 8 in 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.382111+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.382111+0000 mon.b (mon.1) 433 : audit [DBG] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.391179+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.391179+0000 mon.b (mon.1) 434 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.394495+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:55 smithi190 bash[23868]: audit 2024-04-26T16:32:54.394495+0000 mon.a (mon.0) 3174 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-04-26T16:32:55.589 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:55.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:55.718 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:55.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-26T16:32:55.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:55.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:55.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:55.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:55.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:56.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:56.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:56.203 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:56.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:56.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:56.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:56.469 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:32:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 -- 172.21.15.50:0/3194108049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd8103e30 msgr2=0x7fcfd8110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/3194108049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd8103e30 0x7fcfd8110290 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0009f90 tx=0x7fcfd000b330 comp rx=0 tx=0).stop 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 -- 172.21.15.50:0/3194108049 shutdown_connections 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/3194108049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd81094c0 0x7fcfd81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/3194108049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd8103e30 0x7fcfd8110290 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/3194108049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 0x7fcfd81038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 -- 172.21.15.50:0/3194108049 >> 172.21.15.50:0/3194108049 conn(0x7fcfd807a580 msgr2=0x7fcfd807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 -- 172.21.15.50:0/3194108049 shutdown_connections 2024-04-26T16:32:56.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.528+0000 7fcfe05a6700 1 -- 172.21.15.50:0/3194108049 wait complete. 2024-04-26T16:32:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 librados: starting msgr at 2024-04-26T16:32:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 librados: starting objecter 2024-04-26T16:32:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 Processor -- start 2024-04-26T16:32:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- start start 2024-04-26T16:32:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 librados: setting wanted keys 2024-04-26T16:32:56.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 librados: calling monclient init 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 0x7fcfd8075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd8103e30 0x7fcfd8075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd81094c0 0x7fcfd80761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd8112a60 con 0x7fcfd81094c0 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd81128e0 con 0x7fcfd8103e30 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd8112be0 con 0x7fcfd8103470 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfdeb43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd81094c0 0x7fcfd80761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 0x7fcfd8075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfdeb43700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd81094c0 0x7fcfd80761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34516/0 (socket says 172.21.15.50:34516) 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfdeb43700 1 -- 172.21.15.50:0/4153300515 learned_addr learned my addr 172.21.15.50:0/4153300515 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd8103e30 0x7fcfd8075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd8103e30 msgr2=0x7fcfd8075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcfd8103e30 0x7fcfd8075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd81094c0 msgr2=0x7fcfd80761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fcfd81094c0 0x7fcfd80761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd81a75b0 con 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 0x7fcfd8075720 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400c970 tx=0x7fcfd400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfcf7fe700 10 client.?.objecter ms_handle_connect 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfcf7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400ce60 con 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd81a78a0 con 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fcfd4004830 con 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4005280 con 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 10 client.41459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd81a7e30 con 0x7fcfd8103470 2024-04-26T16:32:56.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 librados: init done 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fcfd806a110 con 0x7fcfd8103470 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.532+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fcfd4004c50 con 0x7fcfd8103470 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcfc4071f80 0x7fcfc4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(624..624 src has 496..624) ==== 9413+0+0 (secure 0 0 0) 0x7fcfd4091480 con 0x7fcfd8103470 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 osd_map(624..624 src has 496..624) 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcfc4071f80 0x7fcfc4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 3 client.41459.objecter handle_osd_map got epochs [624,624] > 0 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 3 client.41459.objecter handle_osd_map decoding full epoch 624 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 20 client.41459.objecter dump_active .. 0 homeless 2024-04-26T16:32:56.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v624) ==== 54+0+16 (secure 0 0 0) 0x7fcfd4014030 con 0x7fcfd8103470 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 1 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcfd8064030 0x7fcfd8064450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcfc4071f80 0x7fcfc4074440 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009750 tx=0x7fcfc8009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfc4071f80 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 1 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 0 1 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 4 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 10 client.41459.objecter _send_command 1 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81ab1f0 con 0x7fcfd8064030 2024-04-26T16:32:56.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfdeb43700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcfd8064030 0x7fcfd8064450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd8064030 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfdeb43700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcfd8064030 0x7fcfd8064450 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81ab1f0 con 0x7fcfd8064030 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 1: 0 ) 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 1 = system:0 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 4 2024-04-26T16:32:56.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.536+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 0 1 2024-04-26T16:32:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 2 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 2 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd8076c50 osd=0 3 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 0 2 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 4 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 10 client.41459.objecter _send_command 2 2024-04-26T16:32:56.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd8063ab0 con 0x7fcfd8064030 2024-04-26T16:32:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd8063ab0 con 0x7fcfd8064030 2024-04-26T16:32:56.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 4 2024-04-26T16:32:56.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 0 2 2024-04-26T16:32:56.555 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcfd802d310 0x7fcfd802d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 1 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 4 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 10 client.41459.objecter _send_command 3 2024-04-26T16:32:56.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fcfd810b020 con 0x7fcfd802d310 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcfd802d310 0x7fcfd802d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd802d310 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcfd802d310 0x7fcfd802d710 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd810b020 con 0x7fcfd802d310 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 3: 0 ) 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 3 = system:0 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 4 2024-04-26T16:32:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.556+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 1 3 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 4 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 4 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd810a9a0 osd=1 3 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 1 4 2024-04-26T16:32:56.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 4 2024-04-26T16:32:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 10 client.41459.objecter _send_command 4 2024-04-26T16:32:56.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.576+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd810a420 con 0x7fcfd802d310 2024-04-26T16:32:56.580 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:32:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd810a420 con 0x7fcfd802d310 2024-04-26T16:32:56.580 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 4 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 1 4 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 5 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcfd81ad3b0 0x7fcfd81af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 5 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 2 5 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 4 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 10 client.41459.objecter _send_command 5 2024-04-26T16:32:56.581 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81b2600 con 0x7fcfd81ad3b0 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcfd81ad3b0 0x7fcfd81af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd81ad3b0 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcfd81ad3b0 0x7fcfd81af7a0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81b2600 con 0x7fcfd81ad3b0 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 5: 0 ) 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 5 = system:0 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 4 2024-04-26T16:32:56.582 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.580+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 2 5 2024-04-26T16:32:56.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:56.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 6 2024-04-26T16:32:56.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 6 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd810a820 osd=2 3 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 2 6 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 4 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 10 client.41459.objecter _send_command 6 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.600+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd802dea0 con 0x7fcfd81ad3b0 2024-04-26T16:32:56.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd802dea0 con 0x7fcfd81ad3b0 2024-04-26T16:32:56.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 4 2024-04-26T16:32:56.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 2 6 2024-04-26T16:32:56.601 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:32:56.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 7 2024-04-26T16:32:56.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcfd81b0e60 0x7fcfd81b12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 7 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 3 7 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 4 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 10 client.41459.objecter _send_command 7 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81aafa0 con 0x7fcfd81b0e60 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfdeb43700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcfd81b0e60 0x7fcfd81b12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd81b0e60 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfdeb43700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcfd81b0e60 0x7fcfd81b12a0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81aafa0 con 0x7fcfd81b0e60 2024-04-26T16:32:56.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 7: 0 ) 2024-04-26T16:32:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 7 = system:0 2024-04-26T16:32:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 4 2024-04-26T16:32:56.604 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.604+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 3 7 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 8 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 8 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81aa840 osd=3 3 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 3 8 2024-04-26T16:32:56.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 4 2024-04-26T16:32:56.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 10 client.41459.objecter _send_command 8 2024-04-26T16:32:56.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd81aa380 con 0x7fcfd81b0e60 2024-04-26T16:32:56.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd81aa380 con 0x7fcfd81b0e60 2024-04-26T16:32:56.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 4 2024-04-26T16:32:56.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 3 8 2024-04-26T16:32:56.623 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 9 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcfd81b47c0 0x7fcfd81b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 9 2024-04-26T16:32:56.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 4 9 2024-04-26T16:32:56.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 4 2024-04-26T16:32:56.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 10 client.41459.objecter _send_command 9 2024-04-26T16:32:56.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81aa600 con 0x7fcfd81b47c0 2024-04-26T16:32:56.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.624+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcfd81b47c0 0x7fcfd81b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.626 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:56.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd81b47c0 2024-04-26T16:32:56.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcfd81b47c0 0x7fcfd81b6bb0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: audit 2024-04-26T16:32:55.361835+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:32:56.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: audit 2024-04-26T16:32:55.361835+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: cluster 2024-04-26T16:32:55.370096+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: cluster 2024-04-26T16:32:55.370096+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: audit 2024-04-26T16:32:55.372685+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: audit 2024-04-26T16:32:55.372685+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: audit 2024-04-26T16:32:55.385561+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: audit 2024-04-26T16:32:55.385561+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: cluster 2024-04-26T16:32:56.279049+0000 mon.a (mon.0) 3178 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[29234]: cluster 2024-04-26T16:32:56.279049+0000 mon.a (mon.0) 3178 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:56.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: audit 2024-04-26T16:32:55.361835+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: audit 2024-04-26T16:32:55.361835+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: cluster 2024-04-26T16:32:55.370096+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: cluster 2024-04-26T16:32:55.370096+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: audit 2024-04-26T16:32:55.372685+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: audit 2024-04-26T16:32:55.372685+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: audit 2024-04-26T16:32:55.385561+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: audit 2024-04-26T16:32:55.385561+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: cluster 2024-04-26T16:32:56.279049+0000 mon.a (mon.0) 3178 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:56.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:56 smithi050 bash[21463]: cluster 2024-04-26T16:32:56.279049+0000 mon.a (mon.0) 3178 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:56.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81aa600 con 0x7fcfd81b47c0 2024-04-26T16:32:56.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 9: 0 ) 2024-04-26T16:32:56.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 9 = system:0 2024-04-26T16:32:56.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 4 2024-04-26T16:32:56.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.628+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 4 9 2024-04-26T16:32:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:56.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 10 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 10 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81aa320 osd=4 3 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 4 10 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 4 2024-04-26T16:32:56.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 10 client.41459.objecter _send_command 10 2024-04-26T16:32:56.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.644+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd81b70f0 con 0x7fcfd81b47c0 2024-04-26T16:32:56.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd81b70f0 con 0x7fcfd81b47c0 2024-04-26T16:32:56.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 4 2024-04-26T16:32:56.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 4 10 2024-04-26T16:32:56.647 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:32:56.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 11 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcfd81b7810 0x7fcfd81b9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 11 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 5 11 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 4 2024-04-26T16:32:56.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 10 client.41459.objecter _send_command 11 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81bcba0 con 0x7fcfd81b7810 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcfd81b7810 0x7fcfd81b9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd81b7810 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.648+0000 7fcfddb41700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcfd81b7810 0x7fcfd81b9c00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.652+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81bcba0 con 0x7fcfd81b7810 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.652+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 11: 0 ) 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.652+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 11 = system:0 2024-04-26T16:32:56.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.652+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 4 2024-04-26T16:32:56.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.652+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 5 11 2024-04-26T16:32:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 12 2024-04-26T16:32:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 12 2024-04-26T16:32:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81aa190 osd=5 3 2024-04-26T16:32:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 5 12 2024-04-26T16:32:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 4 2024-04-26T16:32:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 10 client.41459.objecter _send_command 12 2024-04-26T16:32:56.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.668+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd81bcb60 con 0x7fcfd81b7810 2024-04-26T16:32:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd81bcb60 con 0x7fcfd81b7810 2024-04-26T16:32:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 4 2024-04-26T16:32:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 5 12 2024-04-26T16:32:56.669 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:32:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 13 2024-04-26T16:32:56.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcfd81bbed0 0x7fcfd81bc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 13 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 6 13 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 4 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 10 client.41459.objecter _send_command 13 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81bff70 con 0x7fcfd81bbed0 2024-04-26T16:32:56.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfdeb43700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcfd81bbed0 0x7fcfd81bc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd81bbed0 2024-04-26T16:32:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfdeb43700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcfd81bbed0 0x7fcfd81bc330 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.6 v2:172.21.15.190:6816/2027349966 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81bff70 con 0x7fcfd81bbed0 2024-04-26T16:32:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 13: 0 ) 2024-04-26T16:32:56.671 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 13 = system:0 2024-04-26T16:32:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 4 2024-04-26T16:32:56.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.672+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 6 13 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 14 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 14 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81bcb60 osd=6 3 2024-04-26T16:32:56.693 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 6 14 2024-04-26T16:32:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 4 2024-04-26T16:32:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 10 client.41459.objecter _send_command 14 2024-04-26T16:32:56.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.696+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd81ba1f0 con 0x7fcfd81bbed0 2024-04-26T16:32:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.6 v2:172.21.15.190:6816/2027349966 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd81ba1f0 con 0x7fcfd81bbed0 2024-04-26T16:32:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.697 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 4 2024-04-26T16:32:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 6 14 2024-04-26T16:32:56.698 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:32:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:32:56.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 15 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcfd81ba870 0x7fcfd81bacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 15 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 7 15 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 4 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 10 client.41459.objecter _send_command 15 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fcfd81bb350 con 0x7fcfd81ba870 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcfd81ba870 0x7fcfd81bacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfcf7fe700 10 client.41459.objecter ms_handle_connect 0x7fcfd81ba870 2024-04-26T16:32:56.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.700+0000 7fcfde342700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcfd81ba870 0x7fcfd81bacb0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.704+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfd81bb350 con 0x7fcfd81ba870 2024-04-26T16:32:56.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.704+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 15: 0 ) 2024-04-26T16:32:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.704+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 15 = system:0 2024-04-26T16:32:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.704+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 4 2024-04-26T16:32:56.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.704+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 7 15 2024-04-26T16:32:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 10 client.41459.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:32:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign -1 16 2024-04-26T16:32:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 20 client.41459.objecter _get_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_remove -1 16 2024-04-26T16:32:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 20 client.41459.objecter get_session s=0x7fcfd81ba1f0 osd=7 3 2024-04-26T16:32:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 15 client.41459.objecter _session_command_op_assign 7 16 2024-04-26T16:32:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 4 2024-04-26T16:32:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 10 client.41459.objecter _send_command 16 2024-04-26T16:32:56.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.720+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fcfd81bb350 con 0x7fcfd81ba870 2024-04-26T16:32:56.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfcf7fe700 1 -- 172.21.15.50:0/4153300515 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fcfd81bb350 con 0x7fcfd81ba870 2024-04-26T16:32:56.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfcf7fe700 10 client.41459.objecter ms_dispatch 0x7fcfd8001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:32:56.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfcf7fe700 10 client.41459.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:32:56.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfcf7fe700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 4 2024-04-26T16:32:56.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfcf7fe700 15 client.41459.objecter _session_command_op_remove 7 16 2024-04-26T16:32:56.723 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:32:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 librados: watch_flush enter 2024-04-26T16:32:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 librados: watch_flush exit 2024-04-26T16:32:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.0 2024-04-26T16:32:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcfd8064030 msgr2=0x7fcfd8064450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fcfd8064030 0x7fcfd8064450 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd8076c50 osd=0 1 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.1 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcfd802d310 msgr2=0x7fcfd802d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fcfd802d310 0x7fcfd802d710 crc :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a9a0 osd=1 1 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.2 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcfd81ad3b0 msgr2=0x7fcfd81af7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fcfd81ad3b0 0x7fcfd81af7a0 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd810a820 osd=2 1 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.3 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcfd81b0e60 msgr2=0x7fcfd81b12a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfdeb43700 1 -- 172.21.15.50:0/4153300515 reap_dead start 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fcfd81b0e60 0x7fcfd81b12a0 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa840 osd=3 1 2024-04-26T16:32:56.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.4 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcfd81b47c0 msgr2=0x7fcfd81b6bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fcfd81b47c0 0x7fcfd81b6bb0 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa320 osd=4 1 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.5 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcfd81b7810 msgr2=0x7fcfd81b9c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fcfd81b7810 0x7fcfd81b9c00 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81aa190 osd=5 1 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.6 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcfd81bbed0 msgr2=0x7fcfd81bc330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6816/2027349966,v1:172.21.15.190:6817/2027349966] conn(0x7fcfd81bbed0 0x7fcfd81bc330 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81bcb60 osd=6 1 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter close_session for osd.7 2024-04-26T16:32:56.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcfd81ba870 msgr2=0x7fcfd81bacb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fcfd81ba870 0x7fcfd81bacb0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter put_session s=0x7fcfd81ba1f0 osd=7 1 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfdeb43700 1 -- 172.21.15.50:0/4153300515 reap_dead start 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 20 client.41459.objecter shutdown clearing up homeless session... 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 10 client.41459.objecter successfully canceled tick 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcfc4071f80 msgr2=0x7fcfc4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcfc4071f80 0x7fcfc4074440 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7fcfc8009750 tx=0x7fcfc8009310 comp rx=0 tx=0).stop 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 msgr2=0x7fcfd8075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 0x7fcfd8075720 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400c970 tx=0x7fcfd400cc80 comp rx=0 tx=0).stop 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 shutdown_connections 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fcfd8103470 0x7fcfd8075720 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 --2- 172.21.15.50:0/4153300515 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fcfc4071f80 0x7fcfc4074440 unknown :-1 s=CLOSED pgs=3788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 >> 172.21.15.50:0/4153300515 conn(0x7fcfd807a580 msgr2=0x7fcfd8101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 shutdown_connections 2024-04-26T16:32:56.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 -- 172.21.15.50:0/4153300515 wait complete. 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:56.724+0000 7fcfe05a6700 1 librados: shutdown 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:32:56.728 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:32:56.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: audit 2024-04-26T16:32:55.361835+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: audit 2024-04-26T16:32:55.361835+0000 mon.a (mon.0) 3175 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: cluster 2024-04-26T16:32:55.370096+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: cluster 2024-04-26T16:32:55.370096+0000 mon.a (mon.0) 3176 : cluster [DBG] osdmap e623: 8 total, 8 up, 8 in 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: audit 2024-04-26T16:32:55.372685+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: audit 2024-04-26T16:32:55.372685+0000 mon.b (mon.1) 435 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: audit 2024-04-26T16:32:55.385561+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.742 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: audit 2024-04-26T16:32:55.385561+0000 mon.a (mon.0) 3177 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-04-26T16:32:56.743 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: cluster 2024-04-26T16:32:56.279049+0000 mon.a (mon.0) 3178 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:56.743 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:56 smithi190 bash[23868]: cluster 2024-04-26T16:32:56.279049+0000 mon.a (mon.0) 3178 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:32:56.757 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:32:56.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:32:56.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:56.885 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:56.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:56.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:57.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:57.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:57.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:57.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:57.273 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:57.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:57.390 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:32:57.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.410 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:57.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:57.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:57.535 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:57.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: cluster 2024-04-26T16:32:55.787747+0000 mgr.y (mgr.24401) 601 : cluster [DBG] pgmap v961: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: cluster 2024-04-26T16:32:55.787747+0000 mgr.y (mgr.24401) 601 : cluster [DBG] pgmap v961: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.382337+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.382337+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: cluster 2024-04-26T16:32:56.387680+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: cluster 2024-04-26T16:32:56.387680+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.391024+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.391024+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.392392+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.392392+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.539023+0000 mon.c (mon.2) 676 : audit [DBG] from='client.? 172.21.15.50:0/4153300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:57.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:56.539023+0000 mon.c (mon.2) 676 : audit [DBG] from='client.? 172.21.15.50:0/4153300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:57.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:57.344372+0000 mon.a (mon.0) 3182 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:57.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:57.344372+0000 mon.a (mon.0) 3182 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:57.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:57.346555+0000 mon.c (mon.2) 677 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[21463]: audit 2024-04-26T16:32:57.346555+0000 mon.c (mon.2) 677 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: cluster 2024-04-26T16:32:55.787747+0000 mgr.y (mgr.24401) 601 : cluster [DBG] pgmap v961: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: cluster 2024-04-26T16:32:55.787747+0000 mgr.y (mgr.24401) 601 : cluster [DBG] pgmap v961: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.382337+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.382337+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: cluster 2024-04-26T16:32:56.387680+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: cluster 2024-04-26T16:32:56.387680+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.391024+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.391024+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.392392+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.703 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.392392+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.539023+0000 mon.c (mon.2) 676 : audit [DBG] from='client.? 172.21.15.50:0/4153300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:57.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:56.539023+0000 mon.c (mon.2) 676 : audit [DBG] from='client.? 172.21.15.50:0/4153300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:57.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:57.344372+0000 mon.a (mon.0) 3182 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:57.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:57.344372+0000 mon.a (mon.0) 3182 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:57.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:57.346555+0000 mon.c (mon.2) 677 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:57.704 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:57 smithi050 bash[29234]: audit 2024-04-26T16:32:57.346555+0000 mon.c (mon.2) 677 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:57.704 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:57.733 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:57.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: cluster 2024-04-26T16:32:55.787747+0000 mgr.y (mgr.24401) 601 : cluster [DBG] pgmap v961: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: cluster 2024-04-26T16:32:55.787747+0000 mgr.y (mgr.24401) 601 : cluster [DBG] pgmap v961: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.382337+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.382337+0000 mon.a (mon.0) 3179 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: cluster 2024-04-26T16:32:56.387680+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: cluster 2024-04-26T16:32:56.387680+0000 mon.a (mon.0) 3180 : cluster [DBG] osdmap e624: 8 total, 8 up, 8 in 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.391024+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.391024+0000 mon.b (mon.1) 436 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.392392+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.392392+0000 mon.a (mon.0) 3181 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-04-26T16:32:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.539023+0000 mon.c (mon.2) 676 : audit [DBG] from='client.? 172.21.15.50:0/4153300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:56.539023+0000 mon.c (mon.2) 676 : audit [DBG] from='client.? 172.21.15.50:0/4153300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:32:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:57.344372+0000 mon.a (mon.0) 3182 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:57.344372+0000 mon.a (mon.0) 3182 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:32:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:57.346555+0000 mon.c (mon.2) 677 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:57 smithi190 bash[23868]: audit 2024-04-26T16:32:57.346555+0000 mon.c (mon.2) 677 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:32:57.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:57.864 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:57.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:57.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:57.997 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:57.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:57.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:58.100 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:58.127 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:58.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:58.256 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:58.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:58.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:58.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:58.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:58.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.098718+0000 mgr.y (mgr.24401) 602 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:58.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.098718+0000 mgr.y (mgr.24401) 602 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:58.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.385391+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:32:58.499 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.385391+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.393116+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.393116+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.463582+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.463582+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.465100+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.465100+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.465585+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.465585+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.466857+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: audit 2024-04-26T16:32:57.466857+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.508107+0000 osd.7 (osd.7) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.508107+0000 osd.7 (osd.7) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:58.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.508531+0000 osd.7 (osd.7) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:58.501 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:58 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.508531+0000 osd.7 (osd.7) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:58.514 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:58.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.098718+0000 mgr.y (mgr.24401) 602 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.098718+0000 mgr.y (mgr.24401) 602 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.385391+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.385391+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.393116+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.393116+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.463582+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.463582+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.465100+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.465100+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.465585+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.465585+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.466857+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: audit 2024-04-26T16:32:57.466857+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.508107+0000 osd.7 (osd.7) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.508107+0000 osd.7 (osd.7) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.508531+0000 osd.7 (osd.7) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.508531+0000 osd.7 (osd.7) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.098718+0000 mgr.y (mgr.24401) 602 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.098718+0000 mgr.y (mgr.24401) 602 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.385391+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.385391+0000 mon.a (mon.0) 3183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.393116+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.393116+0000 mon.a (mon.0) 3184 : cluster [DBG] osdmap e625: 8 total, 8 up, 8 in 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.463582+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.463582+0000 mon.b (mon.1) 437 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.465100+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.465100+0000 mon.a (mon.0) 3185 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-6"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.465585+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.465585+0000 mon.b (mon.1) 438 : audit [INF] from='client.? 172.21.15.50:0/1381877852' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.466857+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: audit 2024-04-26T16:32:57.466857+0000 mon.a (mon.0) 3186 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-6", "tierpool": "test-rados-api-smithi050-73372-102"}]: dispatch 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.508107+0000 osd.7 (osd.7) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.508107+0000 osd.7 (osd.7) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.508531+0000 osd.7 (osd.7) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:58 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.508531+0000 osd.7 (osd.7) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:32:58.645 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:58.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:58.778 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:32:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:32:58.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:58.909 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:58.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:59.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:59.038 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:59.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:59.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:59.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:59.300 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:32:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:32:59.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:59.438 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:59.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:59.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:59.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:59 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.788332+0000 mgr.y (mgr.24401) 603 : cluster [DBG] pgmap v964: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:59.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:59 smithi190 bash[23868]: cluster 2024-04-26T16:32:57.788332+0000 mgr.y (mgr.24401) 603 : cluster [DBG] pgmap v964: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:59.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:59 smithi190 bash[23868]: cluster 2024-04-26T16:32:58.413199+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-04-26T16:32:59.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:32:59 smithi190 bash[23868]: cluster 2024-04-26T16:32:58.413199+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-04-26T16:32:59.567 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:32:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:32:59.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:59.700 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:32:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:32:59.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.788332+0000 mgr.y (mgr.24401) 603 : cluster [DBG] pgmap v964: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:59.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[21463]: cluster 2024-04-26T16:32:57.788332+0000 mgr.y (mgr.24401) 603 : cluster [DBG] pgmap v964: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:59.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[21463]: cluster 2024-04-26T16:32:58.413199+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-04-26T16:32:59.798 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[21463]: cluster 2024-04-26T16:32:58.413199+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-04-26T16:32:59.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.788332+0000 mgr.y (mgr.24401) 603 : cluster [DBG] pgmap v964: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:59.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[29234]: cluster 2024-04-26T16:32:57.788332+0000 mgr.y (mgr.24401) 603 : cluster [DBG] pgmap v964: 228 pgs: 5 unknown, 223 active+clean; 8.4 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:32:59.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[29234]: cluster 2024-04-26T16:32:58.413199+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-04-26T16:32:59.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:32:59 smithi050 bash[29234]: cluster 2024-04-26T16:32:58.413199+0000 mon.a (mon.0) 3187 : cluster [DBG] osdmap e626: 8 total, 8 up, 8 in 2024-04-26T16:32:59.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:59.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:32:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:32:59.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:59.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:32:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:32:59.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:32:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- 172.21.15.50:0/2096893762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f926806bb70 msgr2=0x7f926806bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:59.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- 172.21.15.50:0/2096893762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f926806bb70 0x7f926806bfd0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f9250009920 tx=0x7f9250009c30 comp rx=0 tx=0).stop 2024-04-26T16:32:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- 172.21.15.50:0/2096893762 shutdown_connections 2024-04-26T16:32:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- 172.21.15.50:0/2096893762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926806c510 0x7f92681102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- 172.21.15.50:0/2096893762 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92681094a0 0x7f9268109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- 172.21.15.50:0/2096893762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f926806bb70 0x7f926806bfd0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:59.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- 172.21.15.50:0/2096893762 >> 172.21.15.50:0/2096893762 conn(0x7f92680792c0 msgr2=0x7f92680796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:32:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- 172.21.15.50:0/2096893762 shutdown_connections 2024-04-26T16:32:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- 172.21.15.50:0/2096893762 wait complete. 2024-04-26T16:32:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 librados: starting msgr at 2024-04-26T16:32:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 librados: starting objecter 2024-04-26T16:32:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 Processor -- start 2024-04-26T16:32:59.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- start start 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 librados: setting wanted keys 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 librados: calling monclient init 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 0x7f92681991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926806c510 0x7f9268199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92681094a0 0x7f926819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9268112ae0 con 0x7f92681094a0 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9268112960 con 0x7f926806c510 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f926deb8700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9268112c60 con 0x7f926806bb70 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f92677fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 0x7f92681991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f92677fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 0x7f92681991e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51232/0 (socket says 172.21.15.50:51232) 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f92677fe700 1 -- 172.21.15.50:0/763490718 learned_addr learned my addr 172.21.15.50:0/763490718 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f9267fff700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92681094a0 0x7f926819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.956+0000 7f9266ffd700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926806c510 0x7f9268199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f92677fe700 1 -- 172.21.15.50:0/763490718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926806c510 msgr2=0x7f9268199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:59.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f92677fe700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926806c510 0x7f9268199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:59.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f92677fe700 1 -- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92681094a0 msgr2=0x7f926819dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:32:59.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f92677fe700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92681094a0 0x7f926819dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:32:59.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f92677fe700 1 -- 172.21.15.50:0/763490718 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926819e1f0 con 0x7f926806bb70 2024-04-26T16:32:59.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f92677fe700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 0x7f92681991e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f925800e3c0 tx=0x7f925800e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:59.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 10 client.?.objecter ms_handle_connect 0x7f926806bb70 2024-04-26T16:32:59.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580090d0 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9258009270 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926819e480 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f926deb8700 10 client.41471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580044c0 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92681a5d60 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f926deb8700 1 librados: init done 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f926810b8b0 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f925801b030 con 0x7f926806bb70 2024-04-26T16:32:59.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9254071e50 0x7f9254074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(627..627 src has 496..627) ==== 8563+0+0 (secure 0 0 0) 0x7f9258090ee0 con 0x7f926806bb70 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 10 client.41471.objecter ms_dispatch 0x7f9268001550 osd_map(627..627 src has 496..627) 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 3 client.41471.objecter handle_osd_map got epochs [627,627] > 0 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 3 client.41471.objecter handle_osd_map decoding full epoch 627 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9266ffd700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9254071e50 0x7f9254074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 20 client.41471.objecter dump_active .. 0 homeless 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9264ff9700 10 client.41471.objecter ms_handle_connect 0x7f9254071e50 2024-04-26T16:32:59.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.960+0000 7f9266ffd700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9254071e50 0x7f9254074310 secure :-1 s=READY pgs=3790 cs=0 l=1 rev1=1 crypto rx=0x7f9268079850 tx=0x7f9250019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:32:59.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:32:59.968+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f9258010040 con 0x7f926806bb70 2024-04-26T16:33:00.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:00.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:00.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:00.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:00.203 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:00.209 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.212+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f9268104b50 con 0x7f926806bb70 2024-04-26T16:33:00.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:00.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:00.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.424+0000 7f9264ff9700 1 -- 172.21.15.50:0/763490718 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v628) ==== 97+0+0 (secure 0 0 0) 0x7f925805cbc0 con 0x7f926806bb70 2024-04-26T16:33:00.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.432+0000 7f926deb8700 10 librados: watch_flush enter 2024-04-26T16:33:00.436 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:00.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 10 librados: watch_flush exit 2024-04-26T16:33:00.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 20 client.41471.objecter shutdown clearing up homeless session... 2024-04-26T16:33:00.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 10 client.41471.objecter successfully canceled tick 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9254071e50 msgr2=0x7f9254074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9254071e50 0x7f9254074310 secure :-1 s=READY pgs=3790 cs=0 l=1 rev1=1 crypto rx=0x7f9268079850 tx=0x7f9250019040 comp rx=0 tx=0).stop 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 msgr2=0x7f92681991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 0x7f92681991e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f925800e3c0 tx=0x7f925800e6d0 comp rx=0 tx=0).stop 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 shutdown_connections 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9254071e50 0x7f9254074310 unknown :-1 s=CLOSED pgs=3790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92681094a0 0x7f926819dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f926806bb70 0x7f92681991e0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 --2- 172.21.15.50:0/763490718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f926806c510 0x7f9268199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 >> 172.21.15.50:0/763490718 conn(0x7f92680792c0 msgr2=0x7f9268100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 shutdown_connections 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 -- 172.21.15.50:0/763490718 wait complete. 2024-04-26T16:33:00.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:00.440+0000 7f926deb8700 1 librados: shutdown 2024-04-26T16:33:00.441 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:33:00.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:00.549 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:33:00.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:33:00.562 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: cluster 2024-04-26T16:32:59.426794+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: cluster 2024-04-26T16:32:59.426794+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.458812+0000 mon.c (mon.2) 678 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.458812+0000 mon.c (mon.2) 678 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.459176+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.459176+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.460294+0000 mon.c (mon.2) 679 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.572 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.460294+0000 mon.c (mon.2) 679 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.460577+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.460577+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.461068+0000 mon.c (mon.2) 680 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.461068+0000 mon.c (mon.2) 680 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.461336+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:32:59.461336+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:33:00.216323+0000 mon.c (mon.2) 681 : audit [INF] from='client.? 172.21.15.50:0/763490718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:33:00.216323+0000 mon.c (mon.2) 681 : audit [INF] from='client.? 172.21.15.50:0/763490718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:33:00.216923+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:00 smithi190 bash[23868]: audit 2024-04-26T16:33:00.216923+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.628 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:00.686 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-04-26T16:33:00.686 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:33:00.687 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 7, 6] 2024-04-26T16:33:00.687 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T16:33:00.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6 2024-04-26T16:33:00.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:00.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: cluster 2024-04-26T16:32:59.426794+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-04-26T16:33:00.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: cluster 2024-04-26T16:32:59.426794+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-04-26T16:33:00.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.458812+0000 mon.c (mon.2) 678 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.458812+0000 mon.c (mon.2) 678 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.737 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.459176+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.459176+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.460294+0000 mon.c (mon.2) 679 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.460294+0000 mon.c (mon.2) 679 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.460577+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.460577+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.461068+0000 mon.c (mon.2) 680 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.461068+0000 mon.c (mon.2) 680 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.461336+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:32:59.461336+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:33:00.216323+0000 mon.c (mon.2) 681 : audit [INF] from='client.? 172.21.15.50:0/763490718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:33:00.216323+0000 mon.c (mon.2) 681 : audit [INF] from='client.? 172.21.15.50:0/763490718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:33:00.216923+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.738 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[21463]: audit 2024-04-26T16:33:00.216923+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: cluster 2024-04-26T16:32:59.426794+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: cluster 2024-04-26T16:32:59.426794+0000 mon.a (mon.0) 3188 : cluster [DBG] osdmap e627: 8 total, 8 up, 8 in 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.458812+0000 mon.c (mon.2) 678 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.458812+0000 mon.c (mon.2) 678 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.459176+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.459176+0000 mon.a (mon.0) 3189 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.460294+0000 mon.c (mon.2) 679 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.739 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.460294+0000 mon.c (mon.2) 679 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.460577+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.460577+0000 mon.a (mon.0) 3190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.461068+0000 mon.c (mon.2) 680 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.461068+0000 mon.c (mon.2) 680 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.461336+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:32:59.461336+0000 mon.a (mon.0) 3191 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:33:00.216323+0000 mon.c (mon.2) 681 : audit [INF] from='client.? 172.21.15.50:0/763490718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:33:00.216323+0000 mon.c (mon.2) 681 : audit [INF] from='client.? 172.21.15.50:0/763490718' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:33:00.216923+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.740 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:00 smithi050 bash[29234]: audit 2024-04-26T16:33:00.216923+0000 mon.a (mon.0) 3192 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:00.755 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:00.858 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:00.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:00.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:00.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:01.019 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:01.074 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:00 smithi190 systemd[1]: Stopping Ceph osd.6 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:33:01.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:01.145 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:01.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:01.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:01.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:01.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:01.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:01.430 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[71351]: debug 2024-04-26T16:33:01.229+0000 7f9e33710700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:33:01.430 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[71351]: debug 2024-04-26T16:33:01.229+0000 7f9e33710700 -1 osd.6 627 *** Got signal Terminated *** 2024-04-26T16:33:01.430 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[71351]: debug 2024-04-26T16:33:01.229+0000 7f9e33710700 -1 osd.6 627 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:33:01.522 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:01.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:32:59.789174+0000 mgr.y (mgr.24401) 604 : cluster [DBG] pgmap v967: 164 pgs: 164 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:32:59.789174+0000 mgr.y (mgr.24401) 604 : cluster [DBG] pgmap v967: 164 pgs: 164 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:00.418664+0000 mon.a (mon.0) 3193 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:00.418664+0000 mon.a (mon.0) 3193 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.421102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.421102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.421258+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.421258+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:00.428955+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:00.428955+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.443831+0000 mon.c (mon.2) 682 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.443831+0000 mon.c (mon.2) 682 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.444740+0000 mon.a (mon.0) 3197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.535 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: audit 2024-04-26T16:33:00.444740+0000 mon.a (mon.0) 3197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.237403+0000 mon.a (mon.0) 3198 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:33:01.536 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.237403+0000 mon.a (mon.0) 3198 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:33:01.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:32:59.789174+0000 mgr.y (mgr.24401) 604 : cluster [DBG] pgmap v967: 164 pgs: 164 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:01.536 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:32:59.789174+0000 mgr.y (mgr.24401) 604 : cluster [DBG] pgmap v967: 164 pgs: 164 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:00.418664+0000 mon.a (mon.0) 3193 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:00.418664+0000 mon.a (mon.0) 3193 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.421102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.421102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.421258+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.421258+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:00.428955+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:00.428955+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.443831+0000 mon.c (mon.2) 682 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.443831+0000 mon.c (mon.2) 682 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.444740+0000 mon.a (mon.0) 3197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: audit 2024-04-26T16:33:00.444740+0000 mon.a (mon.0) 3197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.237403+0000 mon.a (mon.0) 3198 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:33:01.537 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.237403+0000 mon.a (mon.0) 3198 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.280161+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.280161+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.281636+0000 mon.a (mon.0) 3200 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.281636+0000 mon.a (mon.0) 3200 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.307043+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.307043+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.280161+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.280161+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:01.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.281636+0000 mon.a (mon.0) 3200 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:33:01.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.281636+0000 mon.a (mon.0) 3200 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:33:01.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.307043+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-04-26T16:33:01.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.307043+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-04-26T16:33:01.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:32:59.789174+0000 mgr.y (mgr.24401) 604 : cluster [DBG] pgmap v967: 164 pgs: 164 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:32:59.789174+0000 mgr.y (mgr.24401) 604 : cluster [DBG] pgmap v967: 164 pgs: 164 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:00.418664+0000 mon.a (mon.0) 3193 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:00.418664+0000 mon.a (mon.0) 3193 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.421102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.421102+0000 mon.a (mon.0) 3194 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.421258+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.421258+0000 mon.a (mon.0) 3195 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:00.428955+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:00.428955+0000 mon.a (mon.0) 3196 : cluster [DBG] osdmap e628: 8 total, 8 up, 8 in 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.443831+0000 mon.c (mon.2) 682 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.443831+0000 mon.c (mon.2) 682 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.444740+0000 mon.a (mon.0) 3197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: audit 2024-04-26T16:33:00.444740+0000 mon.a (mon.0) 3197 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.237403+0000 mon.a (mon.0) 3198 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:33:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.237403+0000 mon.a (mon.0) 3198 : cluster [INF] osd.6 marked itself down and dead 2024-04-26T16:33:01.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.280161+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:01.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.280161+0000 mon.a (mon.0) 3199 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:01.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.281636+0000 mon.a (mon.0) 3200 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:33:01.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.281636+0000 mon.a (mon.0) 3200 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-26T16:33:01.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.307043+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-04-26T16:33:01.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.307043+0000 mon.a (mon.0) 3201 : cluster [DBG] osdmap e629: 8 total, 7 up, 8 in 2024-04-26T16:33:01.651 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:01.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:01.779 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:01.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:01.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:01.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:01.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:02.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:02.058 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:02.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:02.182 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:02.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:02.320 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:02.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:02.448 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:02.552 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:02.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:02.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:02.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:02.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:02.843 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:02.947 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:02.973 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:02.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:03.068 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:02 smithi190 bash[85925]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-6 2024-04-26T16:33:03.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:03.096 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:03.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:03.225 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:03.324 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[86111]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-6 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.791259+0000 mgr.y (mgr.24401) 605 : cluster [DBG] pgmap v970: 164 pgs: 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: cluster 2024-04-26T16:33:01.791259+0000 mgr.y (mgr.24401) 605 : cluster [DBG] pgmap v970: 164 pgs: 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: audit 2024-04-26T16:33:02.286505+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: audit 2024-04-26T16:33:02.286505+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: cluster 2024-04-26T16:33:02.296477+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: cluster 2024-04-26T16:33:02.296477+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: cluster 2024-04-26T16:33:02.435041+0000 mon.a (mon.0) 3204 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-04-26T16:33:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:03 smithi190 bash[23868]: cluster 2024-04-26T16:33:02.435041+0000 mon.a (mon.0) 3204 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-04-26T16:33:03.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:03.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:03.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:03.480 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:03.583 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:03.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.791259+0000 mgr.y (mgr.24401) 605 : cluster [DBG] pgmap v970: 164 pgs: 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:03.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: cluster 2024-04-26T16:33:01.791259+0000 mgr.y (mgr.24401) 605 : cluster [DBG] pgmap v970: 164 pgs: 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:03.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: audit 2024-04-26T16:33:02.286505+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: audit 2024-04-26T16:33:02.286505+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: cluster 2024-04-26T16:33:02.296477+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: cluster 2024-04-26T16:33:02.296477+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: cluster 2024-04-26T16:33:02.435041+0000 mon.a (mon.0) 3204 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[21463]: cluster 2024-04-26T16:33:02.435041+0000 mon.a (mon.0) 3204 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.791259+0000 mgr.y (mgr.24401) 605 : cluster [DBG] pgmap v970: 164 pgs: 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: cluster 2024-04-26T16:33:01.791259+0000 mgr.y (mgr.24401) 605 : cluster [DBG] pgmap v970: 164 pgs: 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: audit 2024-04-26T16:33:02.286505+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: audit 2024-04-26T16:33:02.286505+0000 mon.a (mon.0) 3202 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi050-73372-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: cluster 2024-04-26T16:33:02.296477+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: cluster 2024-04-26T16:33:02.296477+0000 mon.a (mon.0) 3203 : cluster [DBG] osdmap e630: 8 total, 7 up, 8 in 2024-04-26T16:33:03.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: cluster 2024-04-26T16:33:02.435041+0000 mon.a (mon.0) 3204 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-04-26T16:33:03.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:03 smithi050 bash[29234]: cluster 2024-04-26T16:33:02.435041+0000 mon.a (mon.0) 3204 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-04-26T16:33:03.610 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:03.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:33:03.713 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:03.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:03.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:03.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:03.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:03.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:04.000 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:04.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:04.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:04.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:04.262 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:04.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:04 smithi050 bash[21463]: cluster 2024-04-26T16:33:03.293135+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-04-26T16:33:04.378 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:04 smithi050 bash[21463]: cluster 2024-04-26T16:33:03.293135+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-04-26T16:33:04.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:04 smithi050 bash[29234]: cluster 2024-04-26T16:33:03.293135+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-04-26T16:33:04.378 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:04 smithi050 bash[29234]: cluster 2024-04-26T16:33:03.293135+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-04-26T16:33:04.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:04.495 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:04.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:04 smithi190 bash[23868]: cluster 2024-04-26T16:33:03.293135+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-04-26T16:33:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:04 smithi190 bash[23868]: cluster 2024-04-26T16:33:03.293135+0000 mon.a (mon.0) 3205 : cluster [DBG] osdmap e631: 8 total, 7 up, 8 in 2024-04-26T16:33:04.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:04.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:33:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:33:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:33:04.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:04.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:04.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:04.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:04.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:04.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:04.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:05.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:05.038 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:05.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:05.168 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:05.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:05.298 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:05.332 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:05.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:05.424 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:05.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:05.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:05 smithi190 bash[23868]: cluster 2024-04-26T16:33:03.792014+0000 mgr.y (mgr.24401) 606 : cluster [DBG] pgmap v973: 172 pgs: 8 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:33:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:05 smithi190 bash[23868]: cluster 2024-04-26T16:33:03.792014+0000 mgr.y (mgr.24401) 606 : cluster [DBG] pgmap v973: 172 pgs: 8 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:33:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:05 smithi190 bash[23868]: cluster 2024-04-26T16:33:04.314742+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-04-26T16:33:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:05 smithi190 bash[23868]: cluster 2024-04-26T16:33:04.314742+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-04-26T16:33:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:05 smithi190 bash[23868]: audit 2024-04-26T16:33:04.318992+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:05 smithi190 bash[23868]: audit 2024-04-26T16:33:04.318992+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[21463]: cluster 2024-04-26T16:33:03.792014+0000 mgr.y (mgr.24401) 606 : cluster [DBG] pgmap v973: 172 pgs: 8 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[21463]: cluster 2024-04-26T16:33:03.792014+0000 mgr.y (mgr.24401) 606 : cluster [DBG] pgmap v973: 172 pgs: 8 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[21463]: cluster 2024-04-26T16:33:04.314742+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[21463]: cluster 2024-04-26T16:33:04.314742+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[21463]: audit 2024-04-26T16:33:04.318992+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[21463]: audit 2024-04-26T16:33:04.318992+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[29234]: cluster 2024-04-26T16:33:03.792014+0000 mgr.y (mgr.24401) 606 : cluster [DBG] pgmap v973: 172 pgs: 8 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[29234]: cluster 2024-04-26T16:33:03.792014+0000 mgr.y (mgr.24401) 606 : cluster [DBG] pgmap v973: 172 pgs: 8 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[29234]: cluster 2024-04-26T16:33:04.314742+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-04-26T16:33:05.577 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[29234]: cluster 2024-04-26T16:33:04.314742+0000 mon.a (mon.0) 3206 : cluster [DBG] osdmap e632: 8 total, 7 up, 8 in 2024-04-26T16:33:05.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[29234]: audit 2024-04-26T16:33:04.318992+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:05.578 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:05 smithi050 bash[29234]: audit 2024-04-26T16:33:04.318992+0000 mon.a (mon.0) 3207 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:05.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:05.679 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:05.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:05.809 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-26T16:33:05.809 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:05.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:05.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:05.931 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:05.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:06.034 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:06.062 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:06.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:06.193 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:06.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:06.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:06.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[21463]: audit 2024-04-26T16:33:05.321929+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[21463]: audit 2024-04-26T16:33:05.321929+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.330526+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e633: 8 total, 7 up, 8 in 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.330526+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e633: 8 total, 7 up, 8 in 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[21463]: cluster 2024-04-26T16:33:06.283155+0000 mon.a (mon.0) 3210 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[21463]: cluster 2024-04-26T16:33:06.283155+0000 mon.a (mon.0) 3210 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[29234]: audit 2024-04-26T16:33:05.321929+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[29234]: audit 2024-04-26T16:33:05.321929+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.330526+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e633: 8 total, 7 up, 8 in 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.330526+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e633: 8 total, 7 up, 8 in 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[29234]: cluster 2024-04-26T16:33:06.283155+0000 mon.a (mon.0) 3210 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:06.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:06 smithi050 bash[29234]: cluster 2024-04-26T16:33:06.283155+0000 mon.a (mon.0) 3210 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:06.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:06.453 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:06.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:06 smithi190 bash[23868]: audit 2024-04-26T16:33:05.321929+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:06 smithi190 bash[23868]: audit 2024-04-26T16:33:05.321929+0000 mon.a (mon.0) 3208 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:06 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.330526+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e633: 8 total, 7 up, 8 in 2024-04-26T16:33:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:06 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.330526+0000 mon.a (mon.0) 3209 : cluster [DBG] osdmap e633: 8 total, 7 up, 8 in 2024-04-26T16:33:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:06 smithi190 bash[23868]: cluster 2024-04-26T16:33:06.283155+0000 mon.a (mon.0) 3210 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:06 smithi190 bash[23868]: cluster 2024-04-26T16:33:06.283155+0000 mon.a (mon.0) 3210 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:06.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:06.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:06.713 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:06.817 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:06.846 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:06.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:06.975 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:06.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:07.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:07.106 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:07.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T16:33:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T16:33:07.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-26T16:33:07.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:07.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:33:07.215 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:07 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-26T16:33:07.216 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:33:07 smithi190 systemd[1]: Stopped Ceph osd.6 for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:33:07.232 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:33:07.232 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T16:33:07.232 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-04-26T16:33:07.232 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi190.front.sepia.ceph.com 2024-04-26T16:33:07.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-04-26T16:33:07.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:07.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:07.374 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:07.409 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:33:07.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.676591+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:07.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.676591+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:07.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.677014+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.677014+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.793192+0000 mgr.y (mgr.24401) 607 : cluster [DBG] pgmap v976: 204 pgs: 40 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:05.793192+0000 mgr.y (mgr.24401) 607 : cluster [DBG] pgmap v976: 204 pgs: 40 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:06.343915+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e634: 8 total, 7 up, 8 in 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: cluster 2024-04-26T16:33:06.343915+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e634: 8 total, 7 up, 8 in 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: audit 2024-04-26T16:33:06.354651+0000 mon.a (mon.0) 3212 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:07.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:07 smithi190 bash[23868]: audit 2024-04-26T16:33:06.354651+0000 mon.a (mon.0) 3212 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:07.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.676591+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.676591+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.677014+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.677014+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.793192+0000 mgr.y (mgr.24401) 607 : cluster [DBG] pgmap v976: 204 pgs: 40 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:05.793192+0000 mgr.y (mgr.24401) 607 : cluster [DBG] pgmap v976: 204 pgs: 40 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:06.343915+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e634: 8 total, 7 up, 8 in 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: cluster 2024-04-26T16:33:06.343915+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e634: 8 total, 7 up, 8 in 2024-04-26T16:33:07.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: audit 2024-04-26T16:33:06.354651+0000 mon.a (mon.0) 3212 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[21463]: audit 2024-04-26T16:33:06.354651+0000 mon.a (mon.0) 3212 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.676591+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.676591+0000 osd.5 (osd.5) 225 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.677014+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.677014+0000 osd.5 (osd.5) 226 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.793192+0000 mgr.y (mgr.24401) 607 : cluster [DBG] pgmap v976: 204 pgs: 40 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:05.793192+0000 mgr.y (mgr.24401) 607 : cluster [DBG] pgmap v976: 204 pgs: 40 unknown, 56 peering, 108 active+clean; 583 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:06.343915+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e634: 8 total, 7 up, 8 in 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: cluster 2024-04-26T16:33:06.343915+0000 mon.a (mon.0) 3211 : cluster [DBG] osdmap e634: 8 total, 7 up, 8 in 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: audit 2024-04-26T16:33:06.354651+0000 mon.a (mon.0) 3212 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:07.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:07 smithi050 bash[29234]: audit 2024-04-26T16:33:06.354651+0000 mon.a (mon.0) 3212 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:07.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:07.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:07.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:07.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:07.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:07.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:07.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:07.911 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:07.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:07.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:08.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:08.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:08.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:08.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:08.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:08.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:08.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:08.407 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:08.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: audit 2024-04-26T16:33:07.110182+0000 mgr.y (mgr.24401) 608 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:08.417 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: audit 2024-04-26T16:33:07.110182+0000 mgr.y (mgr.24401) 608 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: audit 2024-04-26T16:33:07.346231+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: audit 2024-04-26T16:33:07.346231+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: cluster 2024-04-26T16:33:07.353395+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e635: 8 total, 7 up, 8 in 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: cluster 2024-04-26T16:33:07.353395+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e635: 8 total, 7 up, 8 in 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: audit 2024-04-26T16:33:07.357265+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[21463]: audit 2024-04-26T16:33:07.357265+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:08.418 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: audit 2024-04-26T16:33:07.110182+0000 mgr.y (mgr.24401) 608 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: audit 2024-04-26T16:33:07.110182+0000 mgr.y (mgr.24401) 608 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: audit 2024-04-26T16:33:07.346231+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: audit 2024-04-26T16:33:07.346231+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: cluster 2024-04-26T16:33:07.353395+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e635: 8 total, 7 up, 8 in 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: cluster 2024-04-26T16:33:07.353395+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e635: 8 total, 7 up, 8 in 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: audit 2024-04-26T16:33:07.357265+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:08.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:08 smithi050 bash[29234]: audit 2024-04-26T16:33:07.357265+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:08.429 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:08.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:08.561 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:08.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: audit 2024-04-26T16:33:07.110182+0000 mgr.y (mgr.24401) 608 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:08.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: audit 2024-04-26T16:33:07.110182+0000 mgr.y (mgr.24401) 608 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:08.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: audit 2024-04-26T16:33:07.346231+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:08.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: audit 2024-04-26T16:33:07.346231+0000 mon.a (mon.0) 3213 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:08.585 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: cluster 2024-04-26T16:33:07.353395+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e635: 8 total, 7 up, 8 in 2024-04-26T16:33:08.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: cluster 2024-04-26T16:33:07.353395+0000 mon.a (mon.0) 3214 : cluster [DBG] osdmap e635: 8 total, 7 up, 8 in 2024-04-26T16:33:08.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: audit 2024-04-26T16:33:07.357265+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:08.586 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:08 smithi190 bash[23868]: audit 2024-04-26T16:33:07.357265+0000 mon.a (mon.0) 3215 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:08.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:08.693 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:08.789 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:08.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:08.821 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:08.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:08.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:08.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:08.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:09.052 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:33:09.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:09.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:09.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:09.217 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:09.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:09.321 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:09.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:09.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:09.486 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:07.793955+0000 mgr.y (mgr.24401) 609 : cluster [DBG] pgmap v979: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:07.793955+0000 mgr.y (mgr.24401) 609 : cluster [DBG] pgmap v979: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:08.346037+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:08.346037+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:08.346083+0000 mon.a (mon.0) 3217 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 14 pgs peering) 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:08.346083+0000 mon.a (mon.0) 3217 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 14 pgs peering) 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: audit 2024-04-26T16:33:08.352956+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: audit 2024-04-26T16:33:08.352956+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:08.358413+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e636: 8 total, 7 up, 8 in 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: cluster 2024-04-26T16:33:08.358413+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e636: 8 total, 7 up, 8 in 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: audit 2024-04-26T16:33:08.360564+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:09 smithi190 bash[23868]: audit 2024-04-26T16:33:08.360564+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:09.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:07.793955+0000 mgr.y (mgr.24401) 609 : cluster [DBG] pgmap v979: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:07.793955+0000 mgr.y (mgr.24401) 609 : cluster [DBG] pgmap v979: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:08.346037+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:08.346037+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:08.346083+0000 mon.a (mon.0) 3217 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 14 pgs peering) 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:08.346083+0000 mon.a (mon.0) 3217 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 14 pgs peering) 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: audit 2024-04-26T16:33:08.352956+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: audit 2024-04-26T16:33:08.352956+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:08.358413+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e636: 8 total, 7 up, 8 in 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: cluster 2024-04-26T16:33:08.358413+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e636: 8 total, 7 up, 8 in 2024-04-26T16:33:09.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: audit 2024-04-26T16:33:08.360564+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[29234]: audit 2024-04-26T16:33:08.360564+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:07.793955+0000 mgr.y (mgr.24401) 609 : cluster [DBG] pgmap v979: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:07.793955+0000 mgr.y (mgr.24401) 609 : cluster [DBG] pgmap v979: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:08.346037+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:08.346037+0000 mon.a (mon.0) 3216 : cluster [WRN] Health check failed: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:08.346083+0000 mon.a (mon.0) 3217 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 14 pgs peering) 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:08.346083+0000 mon.a (mon.0) 3217 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 14 pgs peering) 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: audit 2024-04-26T16:33:08.352956+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:09.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: audit 2024-04-26T16:33:08.352956+0000 mon.a (mon.0) 3218 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:09.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:08.358413+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e636: 8 total, 7 up, 8 in 2024-04-26T16:33:09.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: cluster 2024-04-26T16:33:08.358413+0000 mon.a (mon.0) 3219 : cluster [DBG] osdmap e636: 8 total, 7 up, 8 in 2024-04-26T16:33:09.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: audit 2024-04-26T16:33:08.360564+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:09.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:09 smithi050 bash[21463]: audit 2024-04-26T16:33:08.360564+0000 mon.a (mon.0) 3220 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:09.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:09.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:09.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:09.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:09.876 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:09.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:09.979 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:10.007 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.5 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.1f 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.1e 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.1 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.15 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.13 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:5.3 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.a 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:5.2 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:5.1f 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:6.3 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:5.18 2024-04-26T16:33:10.056 INFO:teuthology.orchestra.run.smithi190.stdout:5.11 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:3.d 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.a 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.4 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.9 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:6.c 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.5 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:2.6 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.d 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:3.11 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.1e 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:4.1 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:2.11 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.14 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:1.0 2024-04-26T16:33:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:11.1c 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:11.8 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:11.10 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:2.14 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:11.17 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:5.9 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:3.10 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:3.14 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:11.3 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:11.18 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:5.10 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:11.1 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:2.16 2024-04-26T16:33:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:2.17 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:3.2 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:6.1b 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:3.c 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:2.19 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:5.f 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:2.1b 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:2.1d 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:2.2 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:3.a 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:2.7 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:3.0 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:3.1d 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:3.1e 2024-04-26T16:33:10.059 INFO:teuthology.orchestra.run.smithi190.stdout:2.1a 2024-04-26T16:33:10.060 INFO:teuthology.orchestra.run.smithi190.stdout:3.3 2024-04-26T16:33:10.110 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:10.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:10.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:10.242 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:10.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:10.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:10.372 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:10.399 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:10.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:10.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[21463]: audit 2024-04-26T16:33:09.362278+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:10.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[21463]: audit 2024-04-26T16:33:09.362278+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:10.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[21463]: cluster 2024-04-26T16:33:09.366131+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e637: 8 total, 7 up, 8 in 2024-04-26T16:33:10.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[21463]: cluster 2024-04-26T16:33:09.366131+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e637: 8 total, 7 up, 8 in 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[21463]: audit 2024-04-26T16:33:09.367516+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[21463]: audit 2024-04-26T16:33:09.367516+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[29234]: audit 2024-04-26T16:33:09.362278+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[29234]: audit 2024-04-26T16:33:09.362278+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[29234]: cluster 2024-04-26T16:33:09.366131+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e637: 8 total, 7 up, 8 in 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[29234]: cluster 2024-04-26T16:33:09.366131+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e637: 8 total, 7 up, 8 in 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[29234]: audit 2024-04-26T16:33:09.367516+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:10.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:10 smithi050 bash[29234]: audit 2024-04-26T16:33:09.367516+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:10.530 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:10.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:10.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:10.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:10 smithi190 bash[23868]: audit 2024-04-26T16:33:09.362278+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:10.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:10 smithi190 bash[23868]: audit 2024-04-26T16:33:09.362278+0000 mon.a (mon.0) 3221 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-107", "overlaypool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:10.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:10 smithi190 bash[23868]: cluster 2024-04-26T16:33:09.366131+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e637: 8 total, 7 up, 8 in 2024-04-26T16:33:10.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:10 smithi190 bash[23868]: cluster 2024-04-26T16:33:09.366131+0000 mon.a (mon.0) 3222 : cluster [DBG] osdmap e637: 8 total, 7 up, 8 in 2024-04-26T16:33:10.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:10 smithi190 bash[23868]: audit 2024-04-26T16:33:09.367516+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:10.645 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:10 smithi190 bash[23868]: audit 2024-04-26T16:33:09.367516+0000 mon.a (mon.0) 3223 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:10.661 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:10.764 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:10.793 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:10.896 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:10.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:10.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/4164311307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa910103e60 msgr2=0x7fa9101102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- 172.21.15.50:0/4164311307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa910103e60 0x7fa9101102b0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7fa90800b0a0 tx=0x7fa90800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/4164311307 shutdown_connections 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- 172.21.15.50:0/4164311307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa9101094e0 0x7fa9101098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- 172.21.15.50:0/4164311307 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa910103e60 0x7fa9101102b0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- 172.21.15.50:0/4164311307 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9101034a0 0x7fa910103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/4164311307 >> 172.21.15.50:0/4164311307 conn(0x7fa91007a590 msgr2=0x7fa91007a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/4164311307 shutdown_connections 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/4164311307 wait complete. 2024-04-26T16:33:11.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 librados: starting msgr at 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 librados: starting objecter 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 Processor -- start 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- start start 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 librados: setting wanted keys 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 librados: calling monclient init 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9101034a0 0x7fa91019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa910103e60 0x7fa91019b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9101094e0 0x7fa91019fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa910112ad0 con 0x7fa9101094e0 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa910112950 con 0x7fa910103e60 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa910112c50 con 0x7fa9101034a0 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9101034a0 0x7fa91019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9101094e0 0x7fa91019fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9101034a0 0x7fa91019b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55576/0 (socket says 172.21.15.50:55576) 2024-04-26T16:33:11.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 -- 172.21.15.50:0/335754758 learned_addr learned my addr 172.21.15.50:0/335754758 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa910103e60 0x7fa91019b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 -- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa910103e60 msgr2=0x7fa91019b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa910103e60 0x7fa91019b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 -- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9101094e0 msgr2=0x7fa91019fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9101094e0 0x7fa91019fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:11.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9101a02e0 con 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa9101094e0 0x7fa91019fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa910103e60 0x7fa91019b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa9101034a0 0x7fa91019b2d0 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7fa90000d980 tx=0x7fa90000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90cff9700 10 client.?.objecter ms_handle_connect 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90cff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900017070 con 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa900005020 con 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9101a05d0 con 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa900009990 con 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 10 client.41483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9101a7e50 con 0x7fa9101034a0 2024-04-26T16:33:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.016+0000 7fa9160c8700 1 librados: init done 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa91006a110 con 0x7fa9101034a0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa900004a30 con 0x7fa9101034a0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa8f8071f80 0x7fa8f8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(638..638 src has 496..638) ==== 10216+0+0 (secure 0 0 0) 0x7fa900091ba0 con 0x7fa9101034a0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 osd_map(638..638 src has 496..638) 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 3 client.41483.objecter handle_osd_map got epochs [638,638] > 0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 3 client.41483.objecter handle_osd_map decoding full epoch 638 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa8f8071f80 0x7fa8f8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 20 client.41483.objecter dump_active .. 0 homeless 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v638) ==== 54+0+16 (secure 0 0 0) 0x7fa900020020 con 0x7fa9101034a0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa8f8071f80 0x7fa8f8074440 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7fa910077740 tx=0x7fa904009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa8f8071f80 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 10 client.41483.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 1 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa910064030 0x7fa910064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 1 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 0 1 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa91019c630 osd=0 4 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 10 client.41483.objecter _send_command 1 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9101ab1b0 con 0x7fa910064030 2024-04-26T16:33:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa910064030 0x7fa910064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa910064030 0x7fa910064410 crc :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa910064030 2024-04-26T16:33:11.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9101ab1b0 con 0x7fa910064030 2024-04-26T16:33:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 1: 0 ) 2024-04-26T16:33:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 10 client.41483.objecter _finish_command 1 = system:0 2024-04-26T16:33:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa91019c630 osd=0 4 2024-04-26T16:33:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.020+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 0 1 2024-04-26T16:33:11.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 10 client.41483.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 2 2024-04-26T16:33:11.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 2 2024-04-26T16:33:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa91019c630 osd=0 3 2024-04-26T16:33:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 0 2 2024-04-26T16:33:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa91019c630 osd=0 4 2024-04-26T16:33:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 10 client.41483.objecter _send_command 2 2024-04-26T16:33:11.039 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.040+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa910063ab0 con 0x7fa910064030 2024-04-26T16:33:11.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa910063ab0 con 0x7fa910064030 2024-04-26T16:33:11.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 10 client.41483.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa91019c630 osd=0 4 2024-04-26T16:33:11.041 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:33:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 0 2 2024-04-26T16:33:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 10 client.41483.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 3 2024-04-26T16:33:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa91002d310 0x7fa91002d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 1 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101057a0 osd=1 4 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 10 client.41483.objecter _send_command 3 2024-04-26T16:33:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa910105e20 con 0x7fa91002d310 2024-04-26T16:33:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa91002d310 0x7fa91002d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa91002d310 0x7fa91002d6f0 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa91002d310 2024-04-26T16:33:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa910105e20 con 0x7fa91002d310 2024-04-26T16:33:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 3: 0 ) 2024-04-26T16:33:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 10 client.41483.objecter _finish_command 3 = system:0 2024-04-26T16:33:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101057a0 osd=1 4 2024-04-26T16:33:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.044+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 1 3 2024-04-26T16:33:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 10 client.41483.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 4 2024-04-26T16:33:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 4 2024-04-26T16:33:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101057a0 osd=1 3 2024-04-26T16:33:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 1 4 2024-04-26T16:33:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101057a0 osd=1 4 2024-04-26T16:33:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 10 client.41483.objecter _send_command 4 2024-04-26T16:33:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa910105220 con 0x7fa91002d310 2024-04-26T16:33:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa910105220 con 0x7fa91002d310 2024-04-26T16:33:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa90cff9700 10 client.41483.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101057a0 osd=1 4 2024-04-26T16:33:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.064+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 1 4 2024-04-26T16:33:11.064 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 10 client.41483.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 5 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa9101ad370 0x7fa9101af760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 5 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 2 5 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa910105fd0 osd=2 4 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 10 client.41483.objecter _send_command 5 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa9101b25c0 con 0x7fa9101ad370 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa9101ad370 0x7fa9101af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa9101ad370 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa9101ad370 0x7fa9101af760 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9101b25c0 con 0x7fa9101ad370 2024-04-26T16:33:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 5: 0 ) 2024-04-26T16:33:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90cff9700 10 client.41483.objecter _finish_command 5 = system:0 2024-04-26T16:33:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa910105fd0 osd=2 4 2024-04-26T16:33:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.068+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 2 5 2024-04-26T16:33:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 10 client.41483.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 6 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 6 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa910105fd0 osd=2 3 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 2 6 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa910105fd0 osd=2 4 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 10 client.41483.objecter _send_command 6 2024-04-26T16:33:11.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.084+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa91002de80 con 0x7fa9101ad370 2024-04-26T16:33:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa91002de80 con 0x7fa9101ad370 2024-04-26T16:33:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 10 client.41483.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa910105fd0 osd=2 4 2024-04-26T16:33:11.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 2 6 2024-04-26T16:33:11.086 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 10 client.41483.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 7 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa9101b0e20 0x7fa9101b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 7 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 3 7 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa800 osd=3 4 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 10 client.41483.objecter _send_command 7 2024-04-26T16:33:11.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa9101aafc0 con 0x7fa9101b0e20 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa9101b0e20 0x7fa9101b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa9101b0e20 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa9101b0e20 0x7fa9101b1260 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9101aafc0 con 0x7fa9101b0e20 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 7: 0 ) 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 10 client.41483.objecter _finish_command 7 = system:0 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101aa800 osd=3 4 2024-04-26T16:33:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.088+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 3 7 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 10 client.41483.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 8 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 8 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101aa800 osd=3 3 2024-04-26T16:33:11.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 3 8 2024-04-26T16:33:11.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa800 osd=3 4 2024-04-26T16:33:11.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 10 client.41483.objecter _send_command 8 2024-04-26T16:33:11.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa9101aa280 con 0x7fa9101b0e20 2024-04-26T16:33:11.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa9101aa280 con 0x7fa9101b0e20 2024-04-26T16:33:11.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa90cff9700 10 client.41483.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101aa800 osd=3 4 2024-04-26T16:33:11.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 3 8 2024-04-26T16:33:11.107 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:33:11.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 10 client.41483.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 9 2024-04-26T16:33:11.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa9101aa3f0 0x7fa9101b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 9 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 4 9 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa280 osd=4 4 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 10 client.41483.objecter _send_command 9 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa9101b95f0 con 0x7fa9101aa3f0 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.108+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa9101aa3f0 0x7fa9101b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90f7fe700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa9101aa3f0 0x7fa9101b6790 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa9101aa3f0 2024-04-26T16:33:11.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9101b95f0 con 0x7fa9101aa3f0 2024-04-26T16:33:11.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 9: 0 ) 2024-04-26T16:33:11.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90cff9700 10 client.41483.objecter _finish_command 9 = system:0 2024-04-26T16:33:11.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101aa280 osd=4 4 2024-04-26T16:33:11.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.112+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 4 9 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 10 client.41483.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 10 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 10 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101aa280 osd=4 3 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 4 10 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101aa280 osd=4 4 2024-04-26T16:33:11.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 10 client.41483.objecter _send_command 10 2024-04-26T16:33:11.129 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.128+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa9101aa090 con 0x7fa9101aa3f0 2024-04-26T16:33:11.132 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.132+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa9101aa090 con 0x7fa9101aa3f0 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.132+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.132+0000 7fa90cff9700 10 client.41483.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.132+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101aa280 osd=4 4 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.132+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 4 10 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 10 client.41483.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.133 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 11 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa9101b7350 0x7fa9101b9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 11 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 5 11 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101b6cd0 osd=5 4 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 10 client.41483.objecter _send_command 11 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa9101bc840 con 0x7fa9101b7350 2024-04-26T16:33:11.134 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa9101b7350 0x7fa9101b9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90effd700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa9101b7350 0x7fa9101b9810 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa9101b7350 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9101bc840 con 0x7fa9101b7350 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 11: 0 ) 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90cff9700 10 client.41483.objecter _finish_command 11 = system:0 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101b6cd0 osd=5 4 2024-04-26T16:33:11.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.136+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 5 11 2024-04-26T16:33:11.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 10 client.41483.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 12 2024-04-26T16:33:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 12 2024-04-26T16:33:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101b6cd0 osd=5 3 2024-04-26T16:33:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 5 12 2024-04-26T16:33:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101b6cd0 osd=5 4 2024-04-26T16:33:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 10 client.41483.objecter _send_command 12 2024-04-26T16:33:11.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.156+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa9101bb610 con 0x7fa9101b7350 2024-04-26T16:33:11.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa9101bb610 con 0x7fa9101b7350 2024-04-26T16:33:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 10 client.41483.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101b6cd0 osd=5 4 2024-04-26T16:33:11.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 5 12 2024-04-26T16:33:11.158 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 10 client.41483.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 13 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 20 client.41483.objecter _get_session osd=-1 returning homeless 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 10 client.41483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=639}) -- 0x7fa9101bb820 con 0x7fa9101034a0 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa9101bb610 con 0x7fa9101034a0 2024-04-26T16:33:11.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_get_version_reply(handle=1 version=638) ==== 24+0+0 (secure 0 0 0) 0x7fa900060ec0 con 0x7fa9101034a0 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90dffb700 10 client.41483.objecter _check_command_map_dne tid 13 current 638 map_dne_bound 638 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90dffb700 10 client.41483.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90dffb700 15 client.41483.objecter _session_command_op_remove -1 13 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 10 client.41483.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 14 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa9101b9e40 0x7fa9101ba280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 14 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 7 14 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101bc370 osd=7 4 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 10 client.41483.objecter _send_command 14 2024-04-26T16:33:11.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fa9101ba7c0 con 0x7fa9101b9e40 2024-04-26T16:33:11.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa9101b9e40 0x7fa9101ba280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:11.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90cff9700 10 client.41483.objecter ms_handle_connect 0x7fa9101b9e40 2024-04-26T16:33:11.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.160+0000 7fa90ffff700 1 --2- 172.21.15.50:0/335754758 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa9101b9e40 0x7fa9101ba280 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:11.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.164+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9101ba7c0 con 0x7fa9101b9e40 2024-04-26T16:33:11.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.164+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 14: 0 ) 2024-04-26T16:33:11.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.164+0000 7fa90cff9700 10 client.41483.objecter _finish_command 14 = system:0 2024-04-26T16:33:11.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.164+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101bc370 osd=7 4 2024-04-26T16:33:11.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.164+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 7 14 2024-04-26T16:33:11.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:11.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 10 client.41483.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:11.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign -1 15 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 20 client.41483.objecter _get_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_remove -1 15 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 20 client.41483.objecter get_session s=0x7fa9101bc370 osd=7 3 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 15 client.41483.objecter _session_command_op_assign 7 15 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 20 client.41483.objecter put_session s=0x7fa9101bc370 osd=7 4 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 10 client.41483.objecter _send_command 15 2024-04-26T16:33:11.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.184+0000 7fa9160c8700 1 -- 172.21.15.50:0/335754758 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa9101ba7c0 con 0x7fa9101b9e40 2024-04-26T16:33:11.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.188+0000 7fa90cff9700 1 -- 172.21.15.50:0/335754758 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa9101ba7c0 con 0x7fa9101b9e40 2024-04-26T16:33:11.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.188+0000 7fa90cff9700 10 client.41483.objecter ms_dispatch 0x7fa910001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:11.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.188+0000 7fa90cff9700 10 client.41483.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:11.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.188+0000 7fa90cff9700 20 client.41483.objecter put_session s=0x7fa9101bc370 osd=7 4 2024-04-26T16:33:11.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:11.188+0000 7fa90cff9700 15 client.41483.objecter _session_command_op_remove 7 15 2024-04-26T16:33:11.185 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:33:11.186 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:33:11.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op export --pgid 3.0 --file /var/log/ceph/exp.3.0.6 2024-04-26T16:33:11.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:11.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:11.400 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:11.409 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:33:11.427 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:11.531 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:11.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:09.794810+0000 mgr.y (mgr.24401) 610 : cluster [DBG] pgmap v982: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:11.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:09.794810+0000 mgr.y (mgr.24401) 610 : cluster [DBG] pgmap v982: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:11.543 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:10.362829+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:10.362829+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:10.365223+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:10.365223+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:10.372081+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e638: 8 total, 7 up, 8 in 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:10.372081+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e638: 8 total, 7 up, 8 in 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:10.420010+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]: dispatch 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:10.420010+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]: dispatch 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:11.024712+0000 mon.c (mon.2) 683 : audit [DBG] from='client.? 172.21.15.50:0/335754758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:11.024712+0000 mon.c (mon.2) 683 : audit [DBG] from='client.? 172.21.15.50:0/335754758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:11.288709+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]': finished 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:11.288709+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]': finished 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:11.293754+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e639: 8 total, 7 up, 8 in 2024-04-26T16:33:11.544 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: cluster 2024-04-26T16:33:11.293754+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e639: 8 total, 7 up, 8 in 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:11.298478+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[29234]: audit 2024-04-26T16:33:11.298478+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:09.794810+0000 mgr.y (mgr.24401) 610 : cluster [DBG] pgmap v982: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:09.794810+0000 mgr.y (mgr.24401) 610 : cluster [DBG] pgmap v982: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:10.362829+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:10.362829+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:10.365223+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:10.365223+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:10.372081+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e638: 8 total, 7 up, 8 in 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:10.372081+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e638: 8 total, 7 up, 8 in 2024-04-26T16:33:11.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:10.420010+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]: dispatch 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:10.420010+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]: dispatch 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:11.024712+0000 mon.c (mon.2) 683 : audit [DBG] from='client.? 172.21.15.50:0/335754758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:11.024712+0000 mon.c (mon.2) 683 : audit [DBG] from='client.? 172.21.15.50:0/335754758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:11.288709+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]': finished 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:11.288709+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]': finished 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:11.293754+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e639: 8 total, 7 up, 8 in 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: cluster 2024-04-26T16:33:11.293754+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e639: 8 total, 7 up, 8 in 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:11.298478+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:11.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:11 smithi050 bash[21463]: audit 2024-04-26T16:33:11.298478+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:11.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:11.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:09.794810+0000 mgr.y (mgr.24401) 610 : cluster [DBG] pgmap v982: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:11.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:09.794810+0000 mgr.y (mgr.24401) 610 : cluster [DBG] pgmap v982: 236 pgs: 57 active+undersized, 19 active+undersized+degraded, 32 creating+peering, 128 active+clean; 583 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:11.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:10.362829+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:10.362829+0000 mon.a (mon.0) 3224 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:10.365223+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:10.365223+0000 mon.a (mon.0) 3225 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-107-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:10.372081+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e638: 8 total, 7 up, 8 in 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:10.372081+0000 mon.a (mon.0) 3226 : cluster [DBG] osdmap e638: 8 total, 7 up, 8 in 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:10.420010+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]: dispatch 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:10.420010+0000 mon.a (mon.0) 3227 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]: dispatch 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:11.024712+0000 mon.c (mon.2) 683 : audit [DBG] from='client.? 172.21.15.50:0/335754758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:11.024712+0000 mon.c (mon.2) 683 : audit [DBG] from='client.? 172.21.15.50:0/335754758' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:11.288709+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]': finished 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:11.288709+0000 mon.a (mon.0) 3228 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-107"}]': finished 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:11.293754+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e639: 8 total, 7 up, 8 in 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: cluster 2024-04-26T16:33:11.293754+0000 mon.a (mon.0) 3229 : cluster [DBG] osdmap e639: 8 total, 7 up, 8 in 2024-04-26T16:33:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:11.298478+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:11.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:11 smithi190 bash[23868]: audit 2024-04-26T16:33:11.298478+0000 mon.a (mon.0) 3230 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]: dispatch 2024-04-26T16:33:11.662 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:11.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:11.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:11.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:11.950 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:11.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:12.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:12.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:12.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:12.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:12.451 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:12.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:12.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:12.605 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:12.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: cluster 2024-04-26T16:33:12.288876+0000 mon.a (mon.0) 3231 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:12.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: cluster 2024-04-26T16:33:12.288876+0000 mon.a (mon.0) 3231 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: audit 2024-04-26T16:33:12.291166+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: audit 2024-04-26T16:33:12.291166+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: cluster 2024-04-26T16:33:12.295907+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e640: 8 total, 7 up, 8 in 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: cluster 2024-04-26T16:33:12.295907+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e640: 8 total, 7 up, 8 in 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: audit 2024-04-26T16:33:12.371430+0000 mon.a (mon.0) 3234 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: audit 2024-04-26T16:33:12.371430+0000 mon.a (mon.0) 3234 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: audit 2024-04-26T16:33:12.372868+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[29234]: audit 2024-04-26T16:33:12.372868+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: cluster 2024-04-26T16:33:12.288876+0000 mon.a (mon.0) 3231 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: cluster 2024-04-26T16:33:12.288876+0000 mon.a (mon.0) 3231 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: audit 2024-04-26T16:33:12.291166+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:12.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: audit 2024-04-26T16:33:12.291166+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: cluster 2024-04-26T16:33:12.295907+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e640: 8 total, 7 up, 8 in 2024-04-26T16:33:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: cluster 2024-04-26T16:33:12.295907+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e640: 8 total, 7 up, 8 in 2024-04-26T16:33:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: audit 2024-04-26T16:33:12.371430+0000 mon.a (mon.0) 3234 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: audit 2024-04-26T16:33:12.371430+0000 mon.a (mon.0) 3234 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: audit 2024-04-26T16:33:12.372868+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:12.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:12 smithi050 bash[21463]: audit 2024-04-26T16:33:12.372868+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:12.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: cluster 2024-04-26T16:33:12.288876+0000 mon.a (mon.0) 3231 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:12.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: cluster 2024-04-26T16:33:12.288876+0000 mon.a (mon.0) 3231 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:12.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: audit 2024-04-26T16:33:12.291166+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:12.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: audit 2024-04-26T16:33:12.291166+0000 mon.a (mon.0) 3232 : audit [INF] from='client.? 172.21.15.50:0/962716576' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-107", "tierpool": "test-rados-api-smithi050-73372-107-cache"}]': finished 2024-04-26T16:33:12.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: cluster 2024-04-26T16:33:12.295907+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e640: 8 total, 7 up, 8 in 2024-04-26T16:33:12.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: cluster 2024-04-26T16:33:12.295907+0000 mon.a (mon.0) 3233 : cluster [DBG] osdmap e640: 8 total, 7 up, 8 in 2024-04-26T16:33:12.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: audit 2024-04-26T16:33:12.371430+0000 mon.a (mon.0) 3234 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:12.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: audit 2024-04-26T16:33:12.371430+0000 mon.a (mon.0) 3234 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:12.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: audit 2024-04-26T16:33:12.372868+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:12.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:12 smithi190 bash[23868]: audit 2024-04-26T16:33:12.372868+0000 mon.c (mon.2) 684 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:12.720 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:33:12.738 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:12.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:12.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:12.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:12.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:13.002 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:13.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:13.131 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:13.235 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:13.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:33:13.286 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:33:13.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:13.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:13.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:13.521 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:13.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:13.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[29234]: cluster 2024-04-26T16:33:11.796390+0000 mgr.y (mgr.24401) 611 : cluster [DBG] pgmap v985: 236 pgs: 63 active+undersized, 20 active+undersized+degraded, 153 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 75/606 objects degraded (12.376%) 2024-04-26T16:33:13.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[29234]: cluster 2024-04-26T16:33:11.796390+0000 mgr.y (mgr.24401) 611 : cluster [DBG] pgmap v985: 236 pgs: 63 active+undersized, 20 active+undersized+degraded, 153 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 75/606 objects degraded (12.376%) 2024-04-26T16:33:13.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.296934+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2024-04-26T16:33:13.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.296934+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2024-04-26T16:33:13.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[21463]: cluster 2024-04-26T16:33:11.796390+0000 mgr.y (mgr.24401) 611 : cluster [DBG] pgmap v985: 236 pgs: 63 active+undersized, 20 active+undersized+degraded, 153 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 75/606 objects degraded (12.376%) 2024-04-26T16:33:13.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[21463]: cluster 2024-04-26T16:33:11.796390+0000 mgr.y (mgr.24401) 611 : cluster [DBG] pgmap v985: 236 pgs: 63 active+undersized, 20 active+undersized+degraded, 153 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 75/606 objects degraded (12.376%) 2024-04-26T16:33:13.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.296934+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2024-04-26T16:33:13.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:13 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.296934+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2024-04-26T16:33:13.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:13.703 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:33:13.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:13 smithi190 bash[23868]: cluster 2024-04-26T16:33:11.796390+0000 mgr.y (mgr.24401) 611 : cluster [DBG] pgmap v985: 236 pgs: 63 active+undersized, 20 active+undersized+degraded, 153 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 75/606 objects degraded (12.376%) 2024-04-26T16:33:13.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:13 smithi190 bash[23868]: cluster 2024-04-26T16:33:11.796390+0000 mgr.y (mgr.24401) 611 : cluster [DBG] pgmap v985: 236 pgs: 63 active+undersized, 20 active+undersized+degraded, 153 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s; 75/606 objects degraded (12.376%) 2024-04-26T16:33:13.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:13 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.296934+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2024-04-26T16:33:13.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:13 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.296934+0000 mon.a (mon.0) 3235 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2024-04-26T16:33:13.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:13.783 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:13.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:13.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:13.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:14.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:14.046 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:14.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:14.176 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:14.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:14.305 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:14.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:14.436 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:14.539 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:14.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:14.666 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:33:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:33:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:33:14.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:14.697 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:14.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:14.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 -- 172.21.15.50:0/2601426239 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 msgr2=0x7f400010ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 --2- 172.21.15.50:0/2601426239 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 0x7f400010ba40 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009990 tx=0x7f3ff0009ca0 comp rx=0 tx=0).stop 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 -- 172.21.15.50:0/2601426239 shutdown_connections 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 --2- 172.21.15.50:0/2601426239 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4000106020 0x7f4000112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 --2- 172.21.15.50:0/2601426239 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 0x7f400010ba40 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 --2- 172.21.15.50:0/2601426239 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f4000105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 -- 172.21.15.50:0/2601426239 >> 172.21.15.50:0/2601426239 conn(0x7f4000101150 msgr2=0x7f4000103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 -- 172.21.15.50:0/2601426239 shutdown_connections 2024-04-26T16:33:14.808 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 -- 172.21.15.50:0/2601426239 wait complete. 2024-04-26T16:33:14.809 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 librados: starting msgr at 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.808+0000 7f4006a97700 1 librados: starting objecter 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 Processor -- start 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 -- start start 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 librados: setting wanted keys 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 librados: calling monclient init 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f400019b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4000106020 0x7f400019b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 0x7f400019fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f400019b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f400019b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.50:37654/0 (socket says 172.21.15.50:37654) 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 -- 172.21.15.50:0/899676791 learned_addr learned my addr 172.21.15.50:0/899676791 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4000114dc0 con 0x7f4000106020 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4000114c40 con 0x7f4000105660 2024-04-26T16:33:14.810 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4000114f40 con 0x7f400010b660 2024-04-26T16:33:14.811 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 -- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 msgr2=0x7f400019fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 0x7f400019fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 -- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4000106020 msgr2=0x7f400019b8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4000106020 0x7f400019b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40001a03a0 con 0x7f4000105660 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4004833700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f400019b390 secure :-1 s=READY pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff0005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f3ffdffb700 10 client.?.objecter ms_handle_connect 0x7f4000105660 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f3ffdffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0027070 con 0x7f4000105660 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40001a0630 con 0x7f4000105660 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 10 client.43780.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40001a7f10 con 0x7f4000105660 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f4006a97700 1 librados: init done 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3ff000fad0 con 0x7f4000105660 2024-04-26T16:33:14.812 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.812+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff002b680 con 0x7f4000105660 2024-04-26T16:33:14.813 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe0005120 con 0x7f4000105660 2024-04-26T16:33:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3ff002a070 con 0x7f4000105660 2024-04-26T16:33:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fec071f80 0x7f3fec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:14.814 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(642..642 src has 496..642) ==== 9430+0+0 (secure 0 0 0) 0x7f3ff0092810 con 0x7f4000105660 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 10 client.43780.objecter ms_dispatch 0x7f4000001550 osd_map(642..642 src has 496..642) 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 3 client.43780.objecter handle_osd_map got epochs [642,642] > 0 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 3 client.43780.objecter handle_osd_map decoding full epoch 642 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffffff700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fec071f80 0x7f3fec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 20 client.43780.objecter dump_active .. 0 homeless 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffdffb700 10 client.43780.objecter ms_handle_connect 0x7f3fec071f80 2024-04-26T16:33:14.815 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.815+0000 7f3ffffff700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fec071f80 0x7f3fec074440 secure :-1 s=READY pgs=3794 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009510 tx=0x7f3fe8009300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:14.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:14.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:14.831+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3ff0013070 con 0x7f4000105660 2024-04-26T16:33:14.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:14.949 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:14.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:15.052 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:15.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.063+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f3fe00056e0 con 0x7f4000105660 2024-04-26T16:33:15.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:15.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:15.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:15.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:15.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:33:15.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:15.312 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f3ffdffb700 1 -- 172.21.15.50:0/899676791 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v643) ==== 91+0+0 (secure 0 0 0) 0x7f3ff005e290 con 0x7f4000105660 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 10 librados: watch_flush enter 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 10 librados: watch_flush exit 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 20 client.43780.objecter shutdown clearing up homeless session... 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 10 client.43780.objecter successfully canceled tick 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fec071f80 msgr2=0x7f3fec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fec071f80 0x7f3fec074440 secure :-1 s=READY pgs=3794 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009510 tx=0x7f3fe8009300 comp rx=0 tx=0).stop 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 msgr2=0x7f400019b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f400019b390 secure :-1 s=READY pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff0005ef0 comp rx=0 tx=0).stop 2024-04-26T16:33:15.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 shutdown_connections 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fec071f80 0x7f3fec074440 unknown :-1 s=CLOSED pgs=3794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4000105660 0x7f400019b390 secure :-1 s=CLOSED pgs=4142 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0000c00 tx=0x7f3ff0005ef0 comp rx=0 tx=0).stop 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f400010b660 0x7f400019fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 --2- 172.21.15.50:0/899676791 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4000106020 0x7f400019b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.315+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 >> 172.21.15.50:0/899676791 conn(0x7f4000101150 msgr2=0x7f4000101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.319+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 shutdown_connections 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.319+0000 7f4006a97700 1 -- 172.21.15.50:0/899676791 wait complete. 2024-04-26T16:33:15.317 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:15.319+0000 7f4006a97700 1 librados: shutdown 2024-04-26T16:33:15.318 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:33:15.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.789657+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:15.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.789657+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:15.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.789929+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:15.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.789929+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:15.331 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.797377+0000 mgr.y (mgr.24401) 612 : cluster [DBG] pgmap v988: 204 pgs: 56 active+undersized, 20 active+undersized+degraded, 128 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:13.797377+0000 mgr.y (mgr.24401) 612 : cluster [DBG] pgmap v988: 204 pgs: 56 active+undersized, 20 active+undersized+degraded, 128 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:14.301327+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:14.301327+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.334250+0000 mon.c (mon.2) 685 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.334250+0000 mon.c (mon.2) 685 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.334697+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.334697+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.336174+0000 mon.c (mon.2) 686 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.336174+0000 mon.c (mon.2) 686 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.336551+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.336551+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.332 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.337168+0000 mon.c (mon.2) 687 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.789657+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.789657+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.789929+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.789929+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.797377+0000 mgr.y (mgr.24401) 612 : cluster [DBG] pgmap v988: 204 pgs: 56 active+undersized, 20 active+undersized+degraded, 128 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:13.797377+0000 mgr.y (mgr.24401) 612 : cluster [DBG] pgmap v988: 204 pgs: 56 active+undersized, 20 active+undersized+degraded, 128 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:14.301327+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:14.301327+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.334250+0000 mon.c (mon.2) 685 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.334250+0000 mon.c (mon.2) 685 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.334697+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.333 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.334697+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.336174+0000 mon.c (mon.2) 686 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.336174+0000 mon.c (mon.2) 686 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.336551+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.336551+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.337168+0000 mon.c (mon.2) 687 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.337168+0000 mon.c (mon.2) 687 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.337536+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:14.337536+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:14.387317+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:15.334 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: cluster 2024-04-26T16:33:14.387317+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:15.069423+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.50:0/899676791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:15.069423+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.50:0/899676791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:15.070793+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[21463]: audit 2024-04-26T16:33:15.070793+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.337168+0000 mon.c (mon.2) 687 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.337536+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:14.337536+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:14.387317+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: cluster 2024-04-26T16:33:14.387317+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:15.069423+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.50:0/899676791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:15.069423+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.50:0/899676791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.335 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:15.070793+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.336 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:15 smithi050 bash[29234]: audit 2024-04-26T16:33:15.070793+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:15.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:15.437 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:15.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:15.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:15.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.789657+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:15.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.789657+0000 osd.4 (osd.4) 325 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.789929+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.789929+0000 osd.4 (osd.4) 326 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.797377+0000 mgr.y (mgr.24401) 612 : cluster [DBG] pgmap v988: 204 pgs: 56 active+undersized, 20 active+undersized+degraded, 128 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:13.797377+0000 mgr.y (mgr.24401) 612 : cluster [DBG] pgmap v988: 204 pgs: 56 active+undersized, 20 active+undersized+degraded, 128 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:14.301327+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:14.301327+0000 mon.a (mon.0) 3236 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.334250+0000 mon.c (mon.2) 685 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.334250+0000 mon.c (mon.2) 685 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.334697+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.334697+0000 mon.a (mon.0) 3237 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.336174+0000 mon.c (mon.2) 686 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.336174+0000 mon.c (mon.2) 686 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.570 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.336551+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.336551+0000 mon.a (mon.0) 3238 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.337168+0000 mon.c (mon.2) 687 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.337168+0000 mon.c (mon.2) 687 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.337536+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:14.337536+0000 mon.a (mon.0) 3239 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:14.387317+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: cluster 2024-04-26T16:33:14.387317+0000 mon.a (mon.0) 3240 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:15.069423+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.50:0/899676791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:15.069423+0000 mon.b (mon.1) 439 : audit [INF] from='client.? 172.21.15.50:0/899676791' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:15.070793+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:15 smithi190 bash[23868]: audit 2024-04-26T16:33:15.070793+0000 mon.a (mon.0) 3241 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:33:15.572 INFO:teuthology.orchestra.run.smithi190.stderr:Exporting 3.0 info 3.0( v 388'38 (0'0,388'38] local-lis/les=432/433 n=8 ec=80/80 lis/c=432/432 les/c/f=433/433/0 sis=432) 2024-04-26T16:33:15.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:00569cb6:::obj_delete_at_hint.0000000078:head# 2024-04-26T16:33:15.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:013d6c33:gc::gc.30:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:0216b5b6:::obj_delete_at_hint.0000000070:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:0255b7ba:::obj_delete_at_hint.0000000104:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:02683654:reshard::reshard.0000000010:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:03bbb29a:::obj_delete_at_hint.0000000026:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:0667bb13:gc::gc.13:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Read #3:0705c68b:::obj_delete_at_hint.0000000028:head# 2024-04-26T16:33:15.574 INFO:teuthology.orchestra.run.smithi190.stderr:Export successful 2024-04-26T16:33:15.581 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:15.684 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:15.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:15.815 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:15.842 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:15.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:15.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:15.974 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:15.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:16.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:16.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:16.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:16.238 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:16.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:16.362 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:16.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:16.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.297169+0000 mon.a (mon.0) 3242 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:16.478 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.297169+0000 mon.a (mon.0) 3242 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.309917+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.309917+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.310138+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.310138+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.316956+0000 mon.c (mon.2) 688 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.316956+0000 mon.c (mon.2) 688 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.317595+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.317595+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.319457+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: audit 2024-04-26T16:33:15.319457+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.792503+0000 osd.3 (osd.3) 377 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.792503+0000 osd.3 (osd.3) 377 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.792966+0000 osd.3 (osd.3) 378 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:16.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:16 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.792966+0000 osd.3 (osd.3) 378 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.297169+0000 mon.a (mon.0) 3242 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.297169+0000 mon.a (mon.0) 3242 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.309917+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.309917+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.310138+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.310138+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.316956+0000 mon.c (mon.2) 688 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.316956+0000 mon.c (mon.2) 688 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.317595+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2024-04-26T16:33:16.490 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.317595+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.319457+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: audit 2024-04-26T16:33:15.319457+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.792503+0000 osd.3 (osd.3) 377 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.792503+0000 osd.3 (osd.3) 377 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.792966+0000 osd.3 (osd.3) 378 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.792966+0000 osd.3 (osd.3) 378 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.297169+0000 mon.a (mon.0) 3242 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.297169+0000 mon.a (mon.0) 3242 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:16.491 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.309917+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.309917+0000 mon.a (mon.0) 3243 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.310138+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.310138+0000 mon.a (mon.0) 3244 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.316956+0000 mon.c (mon.2) 688 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.316956+0000 mon.c (mon.2) 688 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.317595+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.317595+0000 mon.a (mon.0) 3245 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.319457+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: audit 2024-04-26T16:33:15.319457+0000 mon.a (mon.0) 3246 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.792503+0000 osd.3 (osd.3) 377 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.792503+0000 osd.3 (osd.3) 377 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.792966+0000 osd.3 (osd.3) 378 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:16.492 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:16 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.792966+0000 osd.3 (osd.3) 378 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:16.493 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:16.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:16.622 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:16.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:16.753 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:16.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:16.885 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:16.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:16.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:16.988 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:17.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:17.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:17.148 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:17.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:17.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:17.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:17.382 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:17.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:17.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.654503+0000 osd.2 (osd.2) 311 : cluster [DBG] 5.b scrub starts 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.654503+0000 osd.2 (osd.2) 311 : cluster [DBG] 5.b scrub starts 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.656782+0000 osd.2 (osd.2) 312 : cluster [DBG] 5.b scrub ok 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.656782+0000 osd.2 (osd.2) 312 : cluster [DBG] 5.b scrub ok 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.798389+0000 mgr.y (mgr.24401) 613 : cluster [DBG] pgmap v991: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:15.798389+0000 mgr.y (mgr.24401) 613 : cluster [DBG] pgmap v991: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:16.334875+0000 mon.a (mon.0) 3247 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2024-04-26T16:33:17.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[29234]: cluster 2024-04-26T16:33:16.334875+0000 mon.a (mon.0) 3247 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.654503+0000 osd.2 (osd.2) 311 : cluster [DBG] 5.b scrub starts 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.654503+0000 osd.2 (osd.2) 311 : cluster [DBG] 5.b scrub starts 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.656782+0000 osd.2 (osd.2) 312 : cluster [DBG] 5.b scrub ok 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.656782+0000 osd.2 (osd.2) 312 : cluster [DBG] 5.b scrub ok 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.798389+0000 mgr.y (mgr.24401) 613 : cluster [DBG] pgmap v991: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:15.798389+0000 mgr.y (mgr.24401) 613 : cluster [DBG] pgmap v991: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:16.334875+0000 mon.a (mon.0) 3247 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2024-04-26T16:33:17.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:17 smithi050 bash[21463]: cluster 2024-04-26T16:33:16.334875+0000 mon.a (mon.0) 3247 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2024-04-26T16:33:17.542 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:17.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:33:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.654503+0000 osd.2 (osd.2) 311 : cluster [DBG] 5.b scrub starts 2024-04-26T16:33:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.654503+0000 osd.2 (osd.2) 311 : cluster [DBG] 5.b scrub starts 2024-04-26T16:33:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.656782+0000 osd.2 (osd.2) 312 : cluster [DBG] 5.b scrub ok 2024-04-26T16:33:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.656782+0000 osd.2 (osd.2) 312 : cluster [DBG] 5.b scrub ok 2024-04-26T16:33:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.798389+0000 mgr.y (mgr.24401) 613 : cluster [DBG] pgmap v991: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:15.798389+0000 mgr.y (mgr.24401) 613 : cluster [DBG] pgmap v991: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:16.334875+0000 mon.a (mon.0) 3247 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2024-04-26T16:33:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:17 smithi190 bash[23868]: cluster 2024-04-26T16:33:16.334875+0000 mon.a (mon.0) 3247 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2024-04-26T16:33:17.645 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:17.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:17.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:17.805 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:17.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:17.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:17.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:18.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:18.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --force --op remove --pgid 3.0 2024-04-26T16:33:18.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:18.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:18.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:18.190 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:18.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:18.229 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:33:18.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:18.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:18.389 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-04-26T16:33:18.417 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:18.445 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:18.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:33:18.502 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:18 smithi190 bash[23868]: audit 2024-04-26T16:33:17.118858+0000 mgr.y (mgr.24401) 614 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:18.502 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:18 smithi190 bash[23868]: audit 2024-04-26T16:33:17.118858+0000 mgr.y (mgr.24401) 614 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:18.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:18 smithi190 bash[23868]: audit 2024-04-26T16:33:17.321727+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:18.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:18.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:18 smithi190 bash[23868]: audit 2024-04-26T16:33:17.321727+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:18.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:18 smithi190 bash[23868]: cluster 2024-04-26T16:33:17.326793+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2024-04-26T16:33:18.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:18 smithi190 bash[23868]: cluster 2024-04-26T16:33:17.326793+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2024-04-26T16:33:18.574 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:18.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[29234]: audit 2024-04-26T16:33:17.118858+0000 mgr.y (mgr.24401) 614 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[29234]: audit 2024-04-26T16:33:17.118858+0000 mgr.y (mgr.24401) 614 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[29234]: audit 2024-04-26T16:33:17.321727+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[29234]: audit 2024-04-26T16:33:17.321727+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[29234]: cluster 2024-04-26T16:33:17.326793+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[29234]: cluster 2024-04-26T16:33:17.326793+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[21463]: audit 2024-04-26T16:33:17.118858+0000 mgr.y (mgr.24401) 614 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[21463]: audit 2024-04-26T16:33:17.118858+0000 mgr.y (mgr.24401) 614 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[21463]: audit 2024-04-26T16:33:17.321727+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[21463]: audit 2024-04-26T16:33:17.321727+0000 mon.a (mon.0) 3248 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:18.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[21463]: cluster 2024-04-26T16:33:17.326793+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2024-04-26T16:33:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:18 smithi050 bash[21463]: cluster 2024-04-26T16:33:17.326793+0000 mon.a (mon.0) 3249 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2024-04-26T16:33:18.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:18.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:18.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:18.821 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:18.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:18.949 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:18.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:18.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:19.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:19.081 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:19.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:19.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:19.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:19.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:19.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:19.485 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: cluster 2024-04-26T16:33:17.799442+0000 mgr.y (mgr.24401) 615 : cluster [DBG] pgmap v994: 180 pgs: 8 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: cluster 2024-04-26T16:33:17.799442+0000 mgr.y (mgr.24401) 615 : cluster [DBG] pgmap v994: 180 pgs: 8 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:19.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: cluster 2024-04-26T16:33:18.323116+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: cluster 2024-04-26T16:33:18.323116+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: cluster 2024-04-26T16:33:18.348720+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: cluster 2024-04-26T16:33:18.348720+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: audit 2024-04-26T16:33:18.351146+0000 mon.c (mon.2) 689 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: audit 2024-04-26T16:33:18.351146+0000 mon.c (mon.2) 689 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: audit 2024-04-26T16:33:18.353139+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:19 smithi190 bash[23868]: audit 2024-04-26T16:33:18.353139+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.588 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:19.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: cluster 2024-04-26T16:33:17.799442+0000 mgr.y (mgr.24401) 615 : cluster [DBG] pgmap v994: 180 pgs: 8 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: cluster 2024-04-26T16:33:17.799442+0000 mgr.y (mgr.24401) 615 : cluster [DBG] pgmap v994: 180 pgs: 8 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: cluster 2024-04-26T16:33:18.323116+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: cluster 2024-04-26T16:33:18.323116+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: cluster 2024-04-26T16:33:18.348720+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: cluster 2024-04-26T16:33:18.348720+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: audit 2024-04-26T16:33:18.351146+0000 mon.c (mon.2) 689 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: audit 2024-04-26T16:33:18.351146+0000 mon.c (mon.2) 689 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: audit 2024-04-26T16:33:18.353139+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[29234]: audit 2024-04-26T16:33:18.353139+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.600 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: cluster 2024-04-26T16:33:17.799442+0000 mgr.y (mgr.24401) 615 : cluster [DBG] pgmap v994: 180 pgs: 8 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: cluster 2024-04-26T16:33:17.799442+0000 mgr.y (mgr.24401) 615 : cluster [DBG] pgmap v994: 180 pgs: 8 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: cluster 2024-04-26T16:33:18.323116+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: cluster 2024-04-26T16:33:18.323116+0000 mon.a (mon.0) 3250 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: cluster 2024-04-26T16:33:18.348720+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: cluster 2024-04-26T16:33:18.348720+0000 mon.a (mon.0) 3251 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: audit 2024-04-26T16:33:18.351146+0000 mon.c (mon.2) 689 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: audit 2024-04-26T16:33:18.351146+0000 mon.c (mon.2) 689 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: audit 2024-04-26T16:33:18.353139+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.601 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:19 smithi050 bash[21463]: audit 2024-04-26T16:33:18.353139+0000 mon.a (mon.0) 3252 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:19.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:19.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:19.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:19.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:19.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:19.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:19.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:20.010 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:20.070 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:20.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:20.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:20.239 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:20.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-26T16:33:20.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-04-26T16:33:20.370 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:20.370 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: audit 2024-04-26T16:33:19.339872+0000 mon.a (mon.0) 3253 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:20.370 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: audit 2024-04-26T16:33:19.339872+0000 mon.a (mon.0) 3253 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.355410+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.355410+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: audit 2024-04-26T16:33:19.364041+0000 mon.c (mon.2) 690 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: audit 2024-04-26T16:33:19.364041+0000 mon.c (mon.2) 690 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: audit 2024-04-26T16:33:19.364865+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: audit 2024-04-26T16:33:19.364865+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.584902+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.f scrub starts 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.584902+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.f scrub starts 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.589361+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.f scrub ok 2024-04-26T16:33:20.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.589361+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.f scrub ok 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: audit 2024-04-26T16:33:19.339872+0000 mon.a (mon.0) 3253 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: audit 2024-04-26T16:33:19.339872+0000 mon.a (mon.0) 3253 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.355410+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.355410+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: audit 2024-04-26T16:33:19.364041+0000 mon.c (mon.2) 690 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: audit 2024-04-26T16:33:19.364041+0000 mon.c (mon.2) 690 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: audit 2024-04-26T16:33:19.364865+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: audit 2024-04-26T16:33:19.364865+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.584902+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.f scrub starts 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.584902+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.f scrub starts 2024-04-26T16:33:20.372 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.589361+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.f scrub ok 2024-04-26T16:33:20.373 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:20 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.589361+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.f scrub ok 2024-04-26T16:33:20.396 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:20.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:20.525 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:20.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:20.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: audit 2024-04-26T16:33:19.339872+0000 mon.a (mon.0) 3253 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: audit 2024-04-26T16:33:19.339872+0000 mon.a (mon.0) 3253 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.355410+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2024-04-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.355410+0000 mon.a (mon.0) 3254 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2024-04-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: audit 2024-04-26T16:33:19.364041+0000 mon.c (mon.2) 690 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: audit 2024-04-26T16:33:19.364041+0000 mon.c (mon.2) 690 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: audit 2024-04-26T16:33:19.364865+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: audit 2024-04-26T16:33:19.364865+0000 mon.a (mon.0) 3255 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.584902+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.f scrub starts 2024-04-26T16:33:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.584902+0000 osd.7 (osd.7) 347 : cluster [DBG] 3.f scrub starts 2024-04-26T16:33:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.589361+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.f scrub ok 2024-04-26T16:33:20.702 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:20 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.589361+0000 osd.7 (osd.7) 348 : cluster [DBG] 3.f scrub ok 2024-04-26T16:33:20.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:20.785 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:20.888 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:20.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:20.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:21.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:21.049 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:21.153 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:21.174 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:21.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:21.301 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:21.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:33:21.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:21.416 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.748196+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.e scrub starts 2024-04-26T16:33:21.416 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.748196+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.e scrub starts 2024-04-26T16:33:21.416 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.749724+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.e scrub ok 2024-04-26T16:33:21.416 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.749724+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.e scrub ok 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.800132+0000 mgr.y (mgr.24401) 616 : cluster [DBG] pgmap v997: 212 pgs: 40 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:19.800132+0000 mgr.y (mgr.24401) 616 : cluster [DBG] pgmap v997: 212 pgs: 40 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:20.341727+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:20.341727+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:20.346018+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:20.346018+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:20.346766+0000 mon.c (mon.2) 691 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:20.346766+0000 mon.c (mon.2) 691 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:20.347294+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:20.347294+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.289900+0000 mon.a (mon.0) 3259 : cluster [WRN] Health check update: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.289900+0000 mon.a (mon.0) 3259 : cluster [WRN] Health check update: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:21.417 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:21.293939+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.748196+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.e scrub starts 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.748196+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.e scrub starts 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.749724+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.e scrub ok 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.749724+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.e scrub ok 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.800132+0000 mgr.y (mgr.24401) 616 : cluster [DBG] pgmap v997: 212 pgs: 40 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:19.800132+0000 mgr.y (mgr.24401) 616 : cluster [DBG] pgmap v997: 212 pgs: 40 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:21.418 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:20.341727+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:20.341727+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:20.346018+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:20.346018+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:20.346766+0000 mon.c (mon.2) 691 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:20.346766+0000 mon.c (mon.2) 691 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:20.347294+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:20.347294+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.289900+0000 mon.a (mon.0) 3259 : cluster [WRN] Health check update: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.289900+0000 mon.a (mon.0) 3259 : cluster [WRN] Health check update: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:21.293939+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.419 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:21.293939+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.302694+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.302694+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:21.307916+0000 mon.c (mon.2) 692 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:21.307916+0000 mon.c (mon.2) 692 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:21.322094+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[29234]: audit 2024-04-26T16:33:21.322094+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:21.293939+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.302694+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.302694+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:21.307916+0000 mon.c (mon.2) 692 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:21.307916+0000 mon.c (mon.2) 692 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:21.322094+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.420 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:21 smithi050 bash[21463]: audit 2024-04-26T16:33:21.322094+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:21.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:21.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.748196+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.e scrub starts 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.748196+0000 osd.2 (osd.2) 313 : cluster [DBG] 2.e scrub starts 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.749724+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.e scrub ok 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.749724+0000 osd.2 (osd.2) 314 : cluster [DBG] 2.e scrub ok 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.800132+0000 mgr.y (mgr.24401) 616 : cluster [DBG] pgmap v997: 212 pgs: 40 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:19.800132+0000 mgr.y (mgr.24401) 616 : cluster [DBG] pgmap v997: 212 pgs: 40 unknown, 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:20.341727+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:20.341727+0000 mon.a (mon.0) 3256 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:20.346018+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:20.346018+0000 mon.a (mon.0) 3257 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:20.346766+0000 mon.c (mon.2) 691 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:20.346766+0000 mon.c (mon.2) 691 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:20.347294+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:20.347294+0000 mon.a (mon.0) 3258 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.289900+0000 mon.a (mon.0) 3259 : cluster [WRN] Health check update: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.289900+0000 mon.a (mon.0) 3259 : cluster [WRN] Health check update: Degraded data redundancy: 74/600 objects degraded (12.333%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:21.293939+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:21.293939+0000 mon.a (mon.0) 3260 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-109", "overlaypool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.302694+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.302694+0000 mon.a (mon.0) 3261 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:21.307916+0000 mon.c (mon.2) 692 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:21.307916+0000 mon.c (mon.2) 692 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:21.322094+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:21 smithi190 bash[23868]: audit 2024-04-26T16:33:21.322094+0000 mon.a (mon.0) 3262 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]: dispatch 2024-04-26T16:33:21.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:21.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:21.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:21.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:21.818 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:21.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:21.947 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:21.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:22.050 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:22.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:22.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:22.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:22.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:22.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:22.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:22.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:22.455 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:22.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.544523+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.544523+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.546991+0000 osd.1 (osd.1) 342 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.546991+0000 osd.1 (osd.1) 342 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.557522+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.557522+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.559456+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.559456+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:21.653116+0000 mon.c (mon.2) 693 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:22.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:21.653116+0000 mon.c (mon.2) 693 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.309628+0000 mon.a (mon.0) 3263 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.309628+0000 mon.a (mon.0) 3263 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.311607+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.311607+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.315327+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.315327+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.316852+0000 mon.c (mon.2) 694 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.316852+0000 mon.c (mon.2) 694 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.318229+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.318229+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.327362+0000 mon.c (mon.2) 695 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:22.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.544523+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.544523+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.546991+0000 osd.1 (osd.1) 342 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.546991+0000 osd.1 (osd.1) 342 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.557522+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.557522+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.559456+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.559456+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:21.653116+0000 mon.c (mon.2) 693 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:21.653116+0000 mon.c (mon.2) 693 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.309628+0000 mon.a (mon.0) 3263 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.309628+0000 mon.a (mon.0) 3263 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.311607+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.311607+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.315327+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2024-04-26T16:33:22.572 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.315327+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.316852+0000 mon.c (mon.2) 694 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.316852+0000 mon.c (mon.2) 694 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.318229+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.318229+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.327362+0000 mon.c (mon.2) 695 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.327362+0000 mon.c (mon.2) 695 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.328325+0000 mon.c (mon.2) 696 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.328325+0000 mon.c (mon.2) 696 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.334274+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:22.573 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[21463]: audit 2024-04-26T16:33:22.334274+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:22.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.327362+0000 mon.c (mon.2) 695 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:22.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.328325+0000 mon.c (mon.2) 696 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:22.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.328325+0000 mon.c (mon.2) 696 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:22.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.334274+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:22.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:22 smithi050 bash[29234]: audit 2024-04-26T16:33:22.334274+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:22.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:22.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:22.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.544523+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.544523+0000 osd.1 (osd.1) 341 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.546991+0000 osd.1 (osd.1) 342 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.546991+0000 osd.1 (osd.1) 342 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.557522+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.557522+0000 osd.7 (osd.7) 349 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.559456+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.559456+0000 osd.7 (osd.7) 350 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:21.653116+0000 mon.c (mon.2) 693 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:21.653116+0000 mon.c (mon.2) 693 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.309628+0000 mon.a (mon.0) 3263 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.309628+0000 mon.a (mon.0) 3263 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.311607+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:22.700 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.311607+0000 mon.a (mon.0) 3264 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-109-cache", "mode": "writeback"}]': finished 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.315327+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.315327+0000 mon.a (mon.0) 3265 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.316852+0000 mon.c (mon.2) 694 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.316852+0000 mon.c (mon.2) 694 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.318229+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.318229+0000 mon.a (mon.0) 3266 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.327362+0000 mon.c (mon.2) 695 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.327362+0000 mon.c (mon.2) 695 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.328325+0000 mon.c (mon.2) 696 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.328325+0000 mon.c (mon.2) 696 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.334274+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:22.701 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:22 smithi190 bash[23868]: audit 2024-04-26T16:33:22.334274+0000 mon.a (mon.0) 3267 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:22.713 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:22.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:22.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:22.868 INFO:teuthology.orchestra.run.smithi190.stdout: marking collection for removal 2024-04-26T16:33:22.868 INFO:teuthology.orchestra.run.smithi190.stdout:setting '_remove' omap key 2024-04-26T16:33:22.868 INFO:teuthology.orchestra.run.smithi190.stdout:finish_remove_pgs 3.0_head removing 3.0 2024-04-26T16:33:22.869 INFO:teuthology.orchestra.run.smithi190.stdout:Remove successful 2024-04-26T16:33:22.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:22.972 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:22.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:23.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:23.100 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:23.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:23.229 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:23.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.755151+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.755151+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.756981+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.756981+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.801227+0000 mgr.y (mgr.24401) 617 : cluster [DBG] pgmap v1000: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:21.801227+0000 mgr.y (mgr.24401) 617 : cluster [DBG] pgmap v1000: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.112685+0000 osd.3 (osd.3) 379 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.112685+0000 osd.3 (osd.3) 379 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.114847+0000 osd.3 (osd.3) 380 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.114847+0000 osd.3 (osd.3) 380 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.351662+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:33:23.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.351662+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.354056+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.354056+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.505189+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.505189+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.507551+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:22.507551+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: audit 2024-04-26T16:33:23.314630+0000 mon.a (mon.0) 3268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: audit 2024-04-26T16:33:23.314630+0000 mon.a (mon.0) 3268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.322000+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.322000+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: audit 2024-04-26T16:33:23.324576+0000 mon.c (mon.2) 697 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.755151+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.755151+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.756981+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.756981+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.801227+0000 mgr.y (mgr.24401) 617 : cluster [DBG] pgmap v1000: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:21.801227+0000 mgr.y (mgr.24401) 617 : cluster [DBG] pgmap v1000: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.112685+0000 osd.3 (osd.3) 379 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.112685+0000 osd.3 (osd.3) 379 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.114847+0000 osd.3 (osd.3) 380 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.114847+0000 osd.3 (osd.3) 380 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.351662+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.351662+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:33:23.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.354056+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.354056+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.505189+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.505189+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.507551+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:22.507551+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: audit 2024-04-26T16:33:23.314630+0000 mon.a (mon.0) 3268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: audit 2024-04-26T16:33:23.314630+0000 mon.a (mon.0) 3268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.322000+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.322000+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: audit 2024-04-26T16:33:23.324576+0000 mon.c (mon.2) 697 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: audit 2024-04-26T16:33:23.324576+0000 mon.c (mon.2) 697 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: audit 2024-04-26T16:33:23.326116+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[21463]: audit 2024-04-26T16:33:23.326116+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: audit 2024-04-26T16:33:23.324576+0000 mon.c (mon.2) 697 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: audit 2024-04-26T16:33:23.326116+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:23 smithi050 bash[29234]: audit 2024-04-26T16:33:23.326116+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:23.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:23.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:23.516 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:23.543 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.755151+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.755151+0000 osd.2 (osd.2) 315 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.756981+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.756981+0000 osd.2 (osd.2) 316 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.801227+0000 mgr.y (mgr.24401) 617 : cluster [DBG] pgmap v1000: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:21.801227+0000 mgr.y (mgr.24401) 617 : cluster [DBG] pgmap v1000: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.112685+0000 osd.3 (osd.3) 379 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:33:23.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.112685+0000 osd.3 (osd.3) 379 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.114847+0000 osd.3 (osd.3) 380 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.114847+0000 osd.3 (osd.3) 380 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.351662+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.351662+0000 osd.5 (osd.5) 227 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.354056+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.354056+0000 osd.5 (osd.5) 228 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.505189+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.505189+0000 osd.1 (osd.1) 343 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.507551+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:22.507551+0000 osd.1 (osd.1) 344 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:33:23.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: audit 2024-04-26T16:33:23.314630+0000 mon.a (mon.0) 3268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: audit 2024-04-26T16:33:23.314630+0000 mon.a (mon.0) 3268 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.322000+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.322000+0000 mon.a (mon.0) 3269 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: audit 2024-04-26T16:33:23.324576+0000 mon.c (mon.2) 697 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: audit 2024-04-26T16:33:23.324576+0000 mon.c (mon.2) 697 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: audit 2024-04-26T16:33:23.326116+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:23 smithi190 bash[23868]: audit 2024-04-26T16:33:23.326116+0000 mon.a (mon.0) 3270 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:33:23.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:23.675 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:23.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:23.805 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:23.908 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:23.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:23.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:24.038 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:24.066 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:24.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:24.197 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:24.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:24.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell -n osd.6 --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.0.6 2024-04-26T16:33:24.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:24.355 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:24.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:24.461 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/osd.6/config 2024-04-26T16:33:24.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.360967+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:33:24.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.360967+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:33:24.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.365185+0000 osd.5 (osd.5) 230 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:33:24.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.365185+0000 osd.5 (osd.5) 230 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:33:24.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: audit 2024-04-26T16:33:24.317474+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:33:24.467 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: audit 2024-04-26T16:33:24.317474+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:33:24.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: cluster 2024-04-26T16:33:24.325177+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2024-04-26T16:33:24.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: cluster 2024-04-26T16:33:24.325177+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2024-04-26T16:33:24.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: audit 2024-04-26T16:33:24.327573+0000 mon.c (mon.2) 698 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: audit 2024-04-26T16:33:24.327573+0000 mon.c (mon.2) 698 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: audit 2024-04-26T16:33:24.330310+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:24 smithi190 bash[23868]: audit 2024-04-26T16:33:24.330310+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.479 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:24.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:24.628 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:33:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:33:24.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.360967+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:33:24.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.360967+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.365185+0000 osd.5 (osd.5) 230 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.365185+0000 osd.5 (osd.5) 230 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: audit 2024-04-26T16:33:24.317474+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: audit 2024-04-26T16:33:24.317474+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: cluster 2024-04-26T16:33:24.325177+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: cluster 2024-04-26T16:33:24.325177+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: audit 2024-04-26T16:33:24.327573+0000 mon.c (mon.2) 698 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: audit 2024-04-26T16:33:24.327573+0000 mon.c (mon.2) 698 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: audit 2024-04-26T16:33:24.330310+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[29234]: audit 2024-04-26T16:33:24.330310+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.360967+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:33:24.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.360967+0000 osd.5 (osd.5) 229 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.365185+0000 osd.5 (osd.5) 230 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.365185+0000 osd.5 (osd.5) 230 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: audit 2024-04-26T16:33:24.317474+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: audit 2024-04-26T16:33:24.317474+0000 mon.a (mon.0) 3271 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: cluster 2024-04-26T16:33:24.325177+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: cluster 2024-04-26T16:33:24.325177+0000 mon.a (mon.0) 3272 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: audit 2024-04-26T16:33:24.327573+0000 mon.c (mon.2) 698 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: audit 2024-04-26T16:33:24.327573+0000 mon.c (mon.2) 698 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: audit 2024-04-26T16:33:24.330310+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:24 smithi050 bash[21463]: audit 2024-04-26T16:33:24.330310+0000 mon.a (mon.0) 3273 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:33:24.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:24.742 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:24.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:24.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:24.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:24.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:25.006 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:25.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:25.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:25.138 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:25.151 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:25.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:25.267 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:25.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:25.397 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:25.501 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.801927+0000 mgr.y (mgr.24401) 618 : cluster [DBG] pgmap v1003: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: cluster 2024-04-26T16:33:23.801927+0000 mgr.y (mgr.24401) 618 : cluster [DBG] pgmap v1003: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.318070+0000 mon.a (mon.0) 3274 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.318070+0000 mon.a (mon.0) 3274 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: audit 2024-04-26T16:33:25.320132+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: audit 2024-04-26T16:33:25.320132+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.327875+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2024-04-26T16:33:25.513 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.327875+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: audit 2024-04-26T16:33:25.332476+0000 mon.c (mon.2) 699 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: audit 2024-04-26T16:33:25.332476+0000 mon.c (mon.2) 699 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: audit 2024-04-26T16:33:25.333142+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[29234]: audit 2024-04-26T16:33:25.333142+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.801927+0000 mgr.y (mgr.24401) 618 : cluster [DBG] pgmap v1003: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: cluster 2024-04-26T16:33:23.801927+0000 mgr.y (mgr.24401) 618 : cluster [DBG] pgmap v1003: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.318070+0000 mon.a (mon.0) 3274 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.318070+0000 mon.a (mon.0) 3274 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:25.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: audit 2024-04-26T16:33:25.320132+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: audit 2024-04-26T16:33:25.320132+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.327875+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.327875+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: audit 2024-04-26T16:33:25.332476+0000 mon.c (mon.2) 699 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: audit 2024-04-26T16:33:25.332476+0000 mon.c (mon.2) 699 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: audit 2024-04-26T16:33:25.333142+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.515 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:25 smithi050 bash[21463]: audit 2024-04-26T16:33:25.333142+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.529 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:25.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.801927+0000 mgr.y (mgr.24401) 618 : cluster [DBG] pgmap v1003: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: cluster 2024-04-26T16:33:23.801927+0000 mgr.y (mgr.24401) 618 : cluster [DBG] pgmap v1003: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.318070+0000 mon.a (mon.0) 3274 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.318070+0000 mon.a (mon.0) 3274 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: audit 2024-04-26T16:33:25.320132+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: audit 2024-04-26T16:33:25.320132+0000 mon.a (mon.0) 3275 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.327875+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.327875+0000 mon.a (mon.0) 3276 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2024-04-26T16:33:25.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: audit 2024-04-26T16:33:25.332476+0000 mon.c (mon.2) 699 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.544 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: audit 2024-04-26T16:33:25.332476+0000 mon.c (mon.2) 699 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.544 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: audit 2024-04-26T16:33:25.333142+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.544 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:25 smithi190 bash[23868]: audit 2024-04-26T16:33:25.333142+0000 mon.a (mon.0) 3277 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-04-26T16:33:25.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:25.656 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:25.759 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:25.784 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:25.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:25.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:25.914 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:25.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:26.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:26.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:26.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:26.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:26.272 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-04-26T16:33:26.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:26.308 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:26.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:26.436 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:26.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:26.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:24.710705+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub starts 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:24.710705+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub starts 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:24.713612+0000 osd.2 (osd.2) 318 : cluster [DBG] 5.d scrub ok 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:24.713612+0000 osd.2 (osd.2) 318 : cluster [DBG] 5.d scrub ok 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.436736+0000 osd.1 (osd.1) 345 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.436736+0000 osd.1 (osd.1) 345 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.441810+0000 osd.1 (osd.1) 346 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.441810+0000 osd.1 (osd.1) 346 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: audit 2024-04-26T16:33:26.325856+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: audit 2024-04-26T16:33:26.325856+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.334604+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2024-04-26T16:33:26.551 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:26 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.334604+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2024-04-26T16:33:26.564 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:24.710705+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub starts 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:24.710705+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub starts 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:24.713612+0000 osd.2 (osd.2) 318 : cluster [DBG] 5.d scrub ok 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:24.713612+0000 osd.2 (osd.2) 318 : cluster [DBG] 5.d scrub ok 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.436736+0000 osd.1 (osd.1) 345 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.436736+0000 osd.1 (osd.1) 345 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.441810+0000 osd.1 (osd.1) 346 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.441810+0000 osd.1 (osd.1) 346 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: audit 2024-04-26T16:33:26.325856+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-26T16:33:26.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: audit 2024-04-26T16:33:26.325856+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.334604+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.334604+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:24.710705+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub starts 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:24.710705+0000 osd.2 (osd.2) 317 : cluster [DBG] 5.d scrub starts 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:24.713612+0000 osd.2 (osd.2) 318 : cluster [DBG] 5.d scrub ok 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:24.713612+0000 osd.2 (osd.2) 318 : cluster [DBG] 5.d scrub ok 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.436736+0000 osd.1 (osd.1) 345 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:33:26.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.436736+0000 osd.1 (osd.1) 345 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:33:26.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.441810+0000 osd.1 (osd.1) 346 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:33:26.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.441810+0000 osd.1 (osd.1) 346 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:33:26.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: audit 2024-04-26T16:33:26.325856+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-26T16:33:26.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: audit 2024-04-26T16:33:26.325856+0000 mon.a (mon.0) 3278 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-04-26T16:33:26.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.334604+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2024-04-26T16:33:26.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:26 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.334604+0000 mon.a (mon.0) 3279 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2024-04-26T16:33:26.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:26.690 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:26.793 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:26.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:26.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:26.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 -- 172.21.15.50:0/2448576558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5781101d0 msgr2=0x7fa5781125c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:27.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 --2- 172.21.15.50:0/2448576558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5781101d0 0x7fa5781125c0 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fa570009f90 tx=0x7fa57000b330 comp rx=0 tx=0).stop 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 -- 172.21.15.50:0/2448576558 shutdown_connections 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 --2- 172.21.15.50:0/2448576558 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa5781101d0 0x7fa5781125c0 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 --2- 172.21.15.50:0/2448576558 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa57806bbd0 0x7fa57806bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 --2- 172.21.15.50:0/2448576558 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57806c4f0 0x7fa57810fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 -- 172.21.15.50:0/2448576558 >> 172.21.15.50:0/2448576558 conn(0x7fa578079120 msgr2=0x7fa578079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 -- 172.21.15.50:0/2448576558 shutdown_connections 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.003+0000 7fa57e619700 1 -- 172.21.15.50:0/2448576558 wait complete. 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 librados: starting msgr at 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 librados: starting objecter 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 Processor -- start 2024-04-26T16:33:27.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- start start 2024-04-26T16:33:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 librados: setting wanted keys 2024-04-26T16:33:27.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 librados: calling monclient init 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57806bbd0 0x7fa57819b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa57806c4f0 0x7fa57819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa5781101d0 0x7fa57819fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa578114e10 con 0x7fa57806c4f0 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa578114c90 con 0x7fa57806bbd0 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa578114f90 con 0x7fa5781101d0 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa5781101d0 0x7fa57819fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa57806c4f0 0x7fa57819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa5781101d0 0x7fa57819fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40582/0 (socket says 172.21.15.50:40582) 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5777fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa57806c4f0 0x7fa57819b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:48674/0 (socket says 172.21.15.50:48674) 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 -- 172.21.15.50:0/3254693631 learned_addr learned my addr 172.21.15.50:0/3254693631 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57806bbd0 0x7fa57819b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 -- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57806bbd0 msgr2=0x7fa57819b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57806bbd0 0x7fa57819b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 -- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa57806c4f0 msgr2=0x7fa57819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa57806c4f0 0x7fa57819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:27.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5781a02d0 con 0x7fa5781101d0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57806bbd0 0x7fa57819b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa5781101d0 0x7fa57819fb90 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fa57000d010 tx=0x7fa570007ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 10 client.?.objecter ms_handle_connect 0x7fa5781101d0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa570021070 con 0x7fa5781101d0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa57001d450 con 0x7fa5781101d0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5781a0560 con 0x7fa5781101d0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5700052a0 con 0x7fa5781101d0 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 10 client.41498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:27.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5781a7e40 con 0x7fa5781101d0 2024-04-26T16:33:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 librados: init done 2024-04-26T16:33:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa57806a110 con 0x7fa5781101d0 2024-04-26T16:33:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa570024070 con 0x7fa5781101d0 2024-04-26T16:33:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.007+0000 7fa5757fa700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa564071f80 0x7fa564074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(654..654 src has 496..654) ==== 10676+0+0 (secure 0 0 0) 0x7fa57000f020 con 0x7fa5781101d0 2024-04-26T16:33:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 osd_map(654..654 src has 496..654) 2024-04-26T16:33:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 3 client.41498.objecter handle_osd_map got epochs [654,654] > 0 2024-04-26T16:33:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 3 client.41498.objecter handle_osd_map decoding full epoch 654 2024-04-26T16:33:27.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa564071f80 0x7fa564074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 20 client.41498.objecter dump_active .. 0 homeless 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v654) ==== 54+0+16 (secure 0 0 0) 0x7fa57002b020 con 0x7fa5781101d0 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa564071f80 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa564071f80 0x7fa564074440 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7fa57807a5b0 tx=0x7fa568009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 10 client.41498.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 1 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa578064280 0x7fa578109c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 1 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 0 1 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57819c620 osd=0 4 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 10 client.41498.objecter _send_command 1 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa57810b310 con 0x7fa578064280 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5777fe700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa578064280 0x7fa578109c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa578064280 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5777fe700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa578064280 0x7fa578109c50 crc :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa57810b310 con 0x7fa578064280 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 1: 0 ) 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 10 client.41498.objecter _finish_command 1 = system:0 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa57819c620 osd=0 4 2024-04-26T16:33:27.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.011+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 0 1 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 10 client.41498.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 2 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 2 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa57819c620 osd=0 3 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 0 2 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57819c620 osd=0 4 2024-04-26T16:33:27.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 10 client.41498.objecter _send_command 2 2024-04-26T16:33:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa57810a190 con 0x7fa578064280 2024-04-26T16:33:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa57810a190 con 0x7fa578064280 2024-04-26T16:33:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 10 client.41498.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa57819c620 osd=0 4 2024-04-26T16:33:27.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 0 2 2024-04-26T16:33:27.031 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 10 client.41498.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa578063e70 0x7fa5781ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 1 3 2024-04-26T16:33:27.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa578063d00 osd=1 4 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 10 client.41498.objecter _send_command 3 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa57810b080 con 0x7fa578063e70 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa578063e70 0x7fa5781ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa578063e70 0x7fa5781ac030 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa578063e70 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa57810b080 con 0x7fa578063e70 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 3: 0 ) 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 10 client.41498.objecter _finish_command 3 = system:0 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa578063d00 osd=1 4 2024-04-26T16:33:27.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.031+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 1 3 2024-04-26T16:33:27.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 10 client.41498.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 4 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 4 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa578063d00 osd=1 3 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 1 4 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa578063d00 osd=1 4 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 10 client.41498.objecter _send_command 4 2024-04-26T16:33:27.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa57810a5f0 con 0x7fa578063e70 2024-04-26T16:33:27.051 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa57810a5f0 con 0x7fa578063e70 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5757fa700 10 client.41498.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa578063d00 osd=1 4 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 1 4 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 10 client.41498.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.052 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 5 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa57802dff0 0x7fa5781af700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 5 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 2 5 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57810a9f0 osd=2 4 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 10 client.41498.objecter _send_command 5 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa5781b2560 con 0x7fa57802dff0 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5777fe700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa57802dff0 0x7fa5781af700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa57802dff0 2024-04-26T16:33:27.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.051+0000 7fa5777fe700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa57802dff0 0x7fa5781af700 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.055+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5781b2560 con 0x7fa57802dff0 2024-04-26T16:33:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.055+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 5: 0 ) 2024-04-26T16:33:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.055+0000 7fa5757fa700 10 client.41498.objecter _finish_command 5 = system:0 2024-04-26T16:33:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.055+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa57810a9f0 osd=2 4 2024-04-26T16:33:27.054 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.055+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 2 5 2024-04-26T16:33:27.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:27.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 10 client.41498.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 6 2024-04-26T16:33:27.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 6 2024-04-26T16:33:27.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa57810a9f0 osd=2 3 2024-04-26T16:33:27.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 2 6 2024-04-26T16:33:27.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57810a9f0 osd=2 4 2024-04-26T16:33:27.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 10 client.41498.objecter _send_command 6 2024-04-26T16:33:27.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.075+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa57802d560 con 0x7fa57802dff0 2024-04-26T16:33:27.077 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:27.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa57802d560 con 0x7fa57802dff0 2024-04-26T16:33:27.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 10 client.41498.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa57810a9f0 osd=2 4 2024-04-26T16:33:27.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 2 6 2024-04-26T16:33:27.079 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:33:27.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 10 client.41498.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 7 2024-04-26T16:33:27.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa5781ad250 0x7fa5781b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 7 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 3 7 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57802d560 osd=3 4 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 10 client.41498.objecter _send_command 7 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa5781b5ec0 con 0x7fa5781ad250 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa5781ad250 0x7fa5781b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa5781ad250 2024-04-26T16:33:27.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa5781ad250 0x7fa5781b0dc0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:27.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5781b5ec0 con 0x7fa5781ad250 2024-04-26T16:33:27.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 7: 0 ) 2024-04-26T16:33:27.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 10 client.41498.objecter _finish_command 7 = system:0 2024-04-26T16:33:27.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa57802d560 osd=3 4 2024-04-26T16:33:27.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.079+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 3 7 2024-04-26T16:33:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 10 client.41498.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 8 2024-04-26T16:33:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.097 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 8 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa57802d560 osd=3 3 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 3 8 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa57802d560 osd=3 4 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 10 client.41498.objecter _send_command 8 2024-04-26T16:33:27.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5781ac820 con 0x7fa5781ad250 2024-04-26T16:33:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5781ac820 con 0x7fa5781ad250 2024-04-26T16:33:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa5757fa700 10 client.41498.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa57802d560 osd=3 4 2024-04-26T16:33:27.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 3 8 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 10 client.41498.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 9 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa5781b0920 0x7fa5781b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 9 2024-04-26T16:33:27.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 4 9 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781ac820 osd=4 4 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 10 client.41498.objecter _send_command 9 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa5781acb10 con 0x7fa5781b0920 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.099+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa5781b0920 0x7fa5781b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa5781b0920 2024-04-26T16:33:27.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa577fff700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa5781b0920 0x7fa5781b6730 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5781acb10 con 0x7fa5781b0920 2024-04-26T16:33:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 9: 0 ) 2024-04-26T16:33:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa5757fa700 10 client.41498.objecter _finish_command 9 = system:0 2024-04-26T16:33:27.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa5781ac820 osd=4 4 2024-04-26T16:33:27.103 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.103+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 4 9 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 10 client.41498.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 10 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 10 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa5781ac820 osd=4 3 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 4 10 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781ac820 osd=4 4 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 10 client.41498.objecter _send_command 10 2024-04-26T16:33:27.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.119+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5781acb10 con 0x7fa5781b0920 2024-04-26T16:33:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.123+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5781acb10 con 0x7fa5781b0920 2024-04-26T16:33:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 10 client.41498.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.124 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa5781ac820 osd=4 4 2024-04-26T16:33:27.125 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 4 10 2024-04-26T16:33:27.125 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 10 client.41498.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 11 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa5781b7180 0x7fa5781b9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 11 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 5 11 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781acb10 osd=5 4 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 10 client.41498.objecter _send_command 11 2024-04-26T16:33:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa5781bc570 con 0x7fa5781b7180 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5777fe700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa5781b7180 0x7fa5781b9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa5781b7180 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5777fe700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa5781b7180 0x7fa5781b9570 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5781bc570 con 0x7fa5781b7180 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 11: 0 ) 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 10 client.41498.objecter _finish_command 11 = system:0 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa5781acb10 osd=5 4 2024-04-26T16:33:27.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.127+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 5 11 2024-04-26T16:33:27.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 10 client.41498.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 12 2024-04-26T16:33:27.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 12 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa5781acb10 osd=5 3 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 5 12 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781acb10 osd=5 4 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 10 client.41498.objecter _send_command 12 2024-04-26T16:33:27.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5781b9ba0 con 0x7fa5781b7180 2024-04-26T16:33:27.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5781b9ba0 con 0x7fa5781b7180 2024-04-26T16:33:27.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5757fa700 10 client.41498.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa5781acb10 osd=5 4 2024-04-26T16:33:27.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 5 12 2024-04-26T16:33:27.148 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:33:27.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 10 client.41498.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 13 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter _get_session osd=-1 returning homeless 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 10 client.41498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=655}) -- 0x7fa5781bb550 con 0x7fa5781101d0 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5781b9ba0 con 0x7fa5781101d0 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_get_version_reply(handle=1 version=654) ==== 24+0+0 (secure 0 0 0) 0x7fa570093060 con 0x7fa5781101d0 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5767fc700 10 client.41498.objecter _check_command_map_dne tid 13 current 654 map_dne_bound 654 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5767fc700 10 client.41498.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa5767fc700 15 client.41498.objecter _session_command_op_remove -1 13 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 10 client.41498.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 14 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa5781b9db0 0x7fa5781ba210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.149 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 14 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 7 14 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781bc0a0 osd=7 4 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 10 client.41498.objecter _send_command 14 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.147+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fa5781bc310 con 0x7fa5781b9db0 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa5781b9db0 0x7fa5781ba210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa5757fa700 10 client.41498.objecter ms_handle_connect 0x7fa5781b9db0 2024-04-26T16:33:27.150 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa57cbb6700 1 --2- 172.21.15.50:0/3254693631 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa5781b9db0 0x7fa5781ba210 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:27.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5781bc310 con 0x7fa5781b9db0 2024-04-26T16:33:27.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 14: 0 ) 2024-04-26T16:33:27.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa5757fa700 10 client.41498.objecter _finish_command 14 = system:0 2024-04-26T16:33:27.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa5781bc0a0 osd=7 4 2024-04-26T16:33:27.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.151+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 7 14 2024-04-26T16:33:27.168 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 10 client.41498.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign -1 15 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 20 client.41498.objecter _get_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 15 client.41498.objecter _session_command_op_remove -1 15 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 20 client.41498.objecter get_session s=0x7fa5781bc0a0 osd=7 3 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 15 client.41498.objecter _session_command_op_assign 7 15 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 20 client.41498.objecter put_session s=0x7fa5781bc0a0 osd=7 4 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 10 client.41498.objecter _send_command 15 2024-04-26T16:33:27.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa57e619700 1 -- 172.21.15.50:0/3254693631 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7fa5781bc310 con 0x7fa5781b9db0 2024-04-26T16:33:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa5757fa700 1 -- 172.21.15.50:0/3254693631 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa5781bc310 con 0x7fa5781b9db0 2024-04-26T16:33:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa5757fa700 10 client.41498.objecter ms_dispatch 0x7fa578001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa5757fa700 10 client.41498.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa5757fa700 20 client.41498.objecter put_session s=0x7fa5781bc0a0 osd=7 4 2024-04-26T16:33:27.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:27.171+0000 7fa5757fa700 15 client.41498.objecter _session_command_op_remove 7 15 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:33:27.172 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:33:27.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:27.201 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:27.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:27.316 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:33:27.332 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:27.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.802667+0000 mgr.y (mgr.24401) 619 : cluster [DBG] pgmap v1006: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:27.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:25.802667+0000 mgr.y (mgr.24401) 619 : cluster [DBG] pgmap v1006: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:27.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.165338+0000 osd.3 (osd.3) 381 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:33:27.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.165338+0000 osd.3 (osd.3) 381 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.168532+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.168532+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.323382+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.323382+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.326250+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.326250+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: audit 2024-04-26T16:33:26.399470+0000 mon.c (mon.2) 700 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: audit 2024-04-26T16:33:26.399470+0000 mon.c (mon.2) 700 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: audit 2024-04-26T16:33:26.399941+0000 mon.a (mon.0) 3280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: audit 2024-04-26T16:33:26.399941+0000 mon.a (mon.0) 3280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.682997+0000 osd.0 (osd.0) 369 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.682997+0000 osd.0 (osd.0) 369 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:33:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.686155+0000 osd.0 (osd.0) 370 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:26.686155+0000 osd.0 (osd.0) 370 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: audit 2024-04-26T16:33:27.013962+0000 mon.c (mon.2) 701 : audit [DBG] from='client.? 172.21.15.50:0/3254693631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: audit 2024-04-26T16:33:27.013962+0000 mon.c (mon.2) 701 : audit [DBG] from='client.? 172.21.15.50:0/3254693631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.280670+0000 osd.4 (osd.4) 327 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.280670+0000 osd.4 (osd.4) 327 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.286146+0000 osd.4 (osd.4) 328 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:33:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:27 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.286146+0000 osd.4 (osd.4) 328 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:33:27.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:27.456 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:27.546 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.802667+0000 mgr.y (mgr.24401) 619 : cluster [DBG] pgmap v1006: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:25.802667+0000 mgr.y (mgr.24401) 619 : cluster [DBG] pgmap v1006: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.165338+0000 osd.3 (osd.3) 381 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.165338+0000 osd.3 (osd.3) 381 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.168532+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.168532+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.323382+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.323382+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.326250+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.326250+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: audit 2024-04-26T16:33:26.399470+0000 mon.c (mon.2) 700 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: audit 2024-04-26T16:33:26.399470+0000 mon.c (mon.2) 700 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: audit 2024-04-26T16:33:26.399941+0000 mon.a (mon.0) 3280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.547 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: audit 2024-04-26T16:33:26.399941+0000 mon.a (mon.0) 3280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.682997+0000 osd.0 (osd.0) 369 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.682997+0000 osd.0 (osd.0) 369 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.686155+0000 osd.0 (osd.0) 370 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:26.686155+0000 osd.0 (osd.0) 370 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: audit 2024-04-26T16:33:27.013962+0000 mon.c (mon.2) 701 : audit [DBG] from='client.? 172.21.15.50:0/3254693631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: audit 2024-04-26T16:33:27.013962+0000 mon.c (mon.2) 701 : audit [DBG] from='client.? 172.21.15.50:0/3254693631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.280670+0000 osd.4 (osd.4) 327 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.280670+0000 osd.4 (osd.4) 327 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.286146+0000 osd.4 (osd.4) 328 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:33:27.548 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.286146+0000 osd.4 (osd.4) 328 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.802667+0000 mgr.y (mgr.24401) 619 : cluster [DBG] pgmap v1006: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:25.802667+0000 mgr.y (mgr.24401) 619 : cluster [DBG] pgmap v1006: 212 pgs: 1 active+clean+scrubbing, 58 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 74/600 objects degraded (12.333%) 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.165338+0000 osd.3 (osd.3) 381 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.165338+0000 osd.3 (osd.3) 381 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.168532+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.168532+0000 osd.3 (osd.3) 382 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.323382+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.323382+0000 osd.5 (osd.5) 231 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.326250+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.326250+0000 osd.5 (osd.5) 232 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: audit 2024-04-26T16:33:26.399470+0000 mon.c (mon.2) 700 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: audit 2024-04-26T16:33:26.399470+0000 mon.c (mon.2) 700 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: audit 2024-04-26T16:33:26.399941+0000 mon.a (mon.0) 3280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.549 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: audit 2024-04-26T16:33:26.399941+0000 mon.a (mon.0) 3280 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.682997+0000 osd.0 (osd.0) 369 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.682997+0000 osd.0 (osd.0) 369 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.686155+0000 osd.0 (osd.0) 370 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:26.686155+0000 osd.0 (osd.0) 370 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: audit 2024-04-26T16:33:27.013962+0000 mon.c (mon.2) 701 : audit [DBG] from='client.? 172.21.15.50:0/3254693631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: audit 2024-04-26T16:33:27.013962+0000 mon.c (mon.2) 701 : audit [DBG] from='client.? 172.21.15.50:0/3254693631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.280670+0000 osd.4 (osd.4) 327 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.280670+0000 osd.4 (osd.4) 327 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.286146+0000 osd.4 (osd.4) 328 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:33:27.550 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:27 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.286146+0000 osd.4 (osd.4) 328 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:33:27.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:27.586 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:27.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:27.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:get_pg_num_history pg_num_history pg_num_history(e627 pg_nums {1={20=1},2={78=32},3={80=32},4={81=3},5={82=32},6={84=32},11={140=32},109={155=32},301={490=32},302={500=32},303={508=32},304={513=32},305={517=32},306={528=32},307={538=32},308={547=32},309={556=32},310={558=32},311={565=32},312={571=32},313={576=32},314={590=32},315={597=32},316={609=32},317={621=32}} deleted_pools 499,301,507,302,512,303,516,304,527,305,537,306,546,307,555,308,563,310,564,309,570,311,575,312,589,313,596,314,608,315,620,316,626,317,627,109) 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Importing pgid 3.0 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:00569cb6:::obj_delete_at_hint.0000000078:head# 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:013d6c33:gc::gc.30:head# 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:0216b5b6:::obj_delete_at_hint.0000000070:head# 2024-04-26T16:33:27.788 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:0255b7ba:::obj_delete_at_hint.0000000104:head# 2024-04-26T16:33:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:02683654:reshard::reshard.0000000010:head# 2024-04-26T16:33:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.789 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:03bbb29a:::obj_delete_at_hint.0000000026:head# 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:0667bb13:gc::gc.13:head# 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:Write #3:0705c68b:::obj_delete_at_hint.0000000028:head# 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:write_pg epoch 611 info 3.0( v 388'38 (0'0,388'38] local-lis/les=432/433 n=8 ec=80/80 lis/c=432/432 les/c/f=433/433/0 sis=432) 2024-04-26T16:33:27.790 INFO:teuthology.orchestra.run.smithi190.stdout:Import successful 2024-04-26T16:33:27.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:27.852 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:27.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:27.981 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:27.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:28.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:28.114 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:28.217 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:28.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:28.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:28.378 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:28.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:28.507 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:28.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.130019+0000 mgr.y (mgr.24401) 620 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.130019+0000 mgr.y (mgr.24401) 620 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.326822+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:33:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.326822+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:33:28.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.328454+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.328454+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.384468+0000 osd.1 (osd.1) 347 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.384468+0000 osd.1 (osd.1) 347 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.385944+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.385944+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.390247+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.390247+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.391230+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.391230+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.394551+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.394551+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.400160+0000 mon.c (mon.2) 703 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.400160+0000 mon.c (mon.2) 703 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.400514+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: audit 2024-04-26T16:33:27.400514+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.282730+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.282730+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.284731+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.284731+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.130019+0000 mgr.y (mgr.24401) 620 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.130019+0000 mgr.y (mgr.24401) 620 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:28.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.326822+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.326822+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.328454+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.328454+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.384468+0000 osd.1 (osd.1) 347 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.384468+0000 osd.1 (osd.1) 347 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.385944+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.385944+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.390247+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.390247+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.391230+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.391230+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.394551+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.394551+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2024-04-26T16:33:28.627 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.400160+0000 mon.c (mon.2) 703 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.400160+0000 mon.c (mon.2) 703 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.400514+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: audit 2024-04-26T16:33:27.400514+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.282730+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.282730+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.284731+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:33:28.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:28 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.284731+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:33:28.638 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:28.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:28.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:28.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:28.897 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:28.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:29.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:29.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.130019+0000 mgr.y (mgr.24401) 620 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.130019+0000 mgr.y (mgr.24401) 620 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.326822+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.326822+0000 osd.5 (osd.5) 233 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.328454+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.328454+0000 osd.5 (osd.5) 234 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.384468+0000 osd.1 (osd.1) 347 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:33:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.384468+0000 osd.1 (osd.1) 347 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.385944+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.385944+0000 osd.1 (osd.1) 348 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.390247+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.390247+0000 mon.c (mon.2) 702 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.391230+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.391230+0000 mon.a (mon.0) 3281 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.394551+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.394551+0000 mon.a (mon.0) 3282 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.400160+0000 mon.c (mon.2) 703 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.400160+0000 mon.c (mon.2) 703 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.400514+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: audit 2024-04-26T16:33:27.400514+0000 mon.a (mon.0) 3283 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]: dispatch 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.282730+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.282730+0000 osd.4 (osd.4) 329 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:33:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.284731+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:33:29.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:28 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.284731+0000 osd.4 (osd.4) 330 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:33:29.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:29.158 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:29.261 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:29.289 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:29.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:29.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:29.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/exp.3.0.6 2024-04-26T16:33:29.530 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.803572+0000 mgr.y (mgr.24401) 621 : cluster [DBG] pgmap v1009: 212 pgs: 57 active+undersized, 20 active+undersized+degraded, 135 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:27.803572+0000 mgr.y (mgr.24401) 621 : cluster [DBG] pgmap v1009: 212 pgs: 57 active+undersized, 20 active+undersized+degraded, 135 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.278968+0000 osd.5 (osd.5) 235 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.278968+0000 osd.5 (osd.5) 235 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.281920+0000 osd.5 (osd.5) 236 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.281920+0000 osd.5 (osd.5) 236 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.391975+0000 mon.a (mon.0) 3284 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.391975+0000 mon.a (mon.0) 3284 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: audit 2024-04-26T16:33:28.393092+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: audit 2024-04-26T16:33:28.393092+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.397531+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.397531+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.766328+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.766328+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:33:29.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.787372+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:28.787372+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.257266+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.e scrub starts 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.257266+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.e scrub starts 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.259578+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.e scrub ok 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.259578+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.e scrub ok 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.399256+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.399256+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.803572+0000 mgr.y (mgr.24401) 621 : cluster [DBG] pgmap v1009: 212 pgs: 57 active+undersized, 20 active+undersized+degraded, 135 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:27.803572+0000 mgr.y (mgr.24401) 621 : cluster [DBG] pgmap v1009: 212 pgs: 57 active+undersized, 20 active+undersized+degraded, 135 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.278968+0000 osd.5 (osd.5) 235 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:33:29.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.278968+0000 osd.5 (osd.5) 235 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.281920+0000 osd.5 (osd.5) 236 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.281920+0000 osd.5 (osd.5) 236 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.391975+0000 mon.a (mon.0) 3284 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.391975+0000 mon.a (mon.0) 3284 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: audit 2024-04-26T16:33:28.393092+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: audit 2024-04-26T16:33:28.393092+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.397531+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.397531+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.766328+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.766328+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.787372+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:28.787372+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.257266+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.e scrub starts 2024-04-26T16:33:29.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.257266+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.e scrub starts 2024-04-26T16:33:29.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.259578+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.e scrub ok 2024-04-26T16:33:29.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.259578+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.e scrub ok 2024-04-26T16:33:29.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.399256+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2024-04-26T16:33:29.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:29 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.399256+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2024-04-26T16:33:29.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:29.557 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:29.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.803572+0000 mgr.y (mgr.24401) 621 : cluster [DBG] pgmap v1009: 212 pgs: 57 active+undersized, 20 active+undersized+degraded, 135 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:27.803572+0000 mgr.y (mgr.24401) 621 : cluster [DBG] pgmap v1009: 212 pgs: 57 active+undersized, 20 active+undersized+degraded, 135 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.278968+0000 osd.5 (osd.5) 235 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.278968+0000 osd.5 (osd.5) 235 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.281920+0000 osd.5 (osd.5) 236 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.281920+0000 osd.5 (osd.5) 236 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.391975+0000 mon.a (mon.0) 3284 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.391975+0000 mon.a (mon.0) 3284 : cluster [WRN] Health check update: Degraded data redundancy: 75/603 objects degraded (12.438%), 20 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: audit 2024-04-26T16:33:28.393092+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:29.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: audit 2024-04-26T16:33:28.393092+0000 mon.a (mon.0) 3285 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-109", "tierpool": "test-rados-api-smithi050-73372-109-cache"}]': finished 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.397531+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.397531+0000 mon.a (mon.0) 3286 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.766328+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.766328+0000 osd.0 (osd.0) 371 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.787372+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:28.787372+0000 osd.0 (osd.0) 372 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.257266+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.e scrub starts 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.257266+0000 osd.4 (osd.4) 331 : cluster [DBG] 6.e scrub starts 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.259578+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.e scrub ok 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.259578+0000 osd.4 (osd.4) 332 : cluster [DBG] 6.e scrub ok 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.399256+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2024-04-26T16:33:29.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:29 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.399256+0000 mon.a (mon.0) 3287 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2024-04-26T16:33:29.687 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:29.790 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:29.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:33:29.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:33:29.924 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:29.953 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:30.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:30.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:30.189 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:30.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:30.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:30.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:30.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:30.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.372731+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.372731+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.374747+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.374747+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.554611+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.554611+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.556395+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.556395+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.372731+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.372731+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:33:30.596 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.374747+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:33:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.374747+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:33:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.554611+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:33:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.554611+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:33:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.556395+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:33:30.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:30 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.556395+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:33:30.612 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:30.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:30.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.372731+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:33:30.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.372731+0000 osd.1 (osd.1) 349 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:33:30.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.374747+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:33:30.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.374747+0000 osd.1 (osd.1) 350 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:33:30.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.554611+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:33:30.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.554611+0000 osd.7 (osd.7) 351 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:33:30.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.556395+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:33:30.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:30 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.556395+0000 osd.7 (osd.7) 352 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:33:30.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:30.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:30.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:30.870 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:30.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:30.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:31.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:31.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:31.132 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:31.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:31.262 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:31.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:31.391 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:31.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:31.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:31.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.804581+0000 mgr.y (mgr.24401) 622 : cluster [DBG] pgmap v1012: 180 pgs: 45 active+undersized, 20 active+undersized+degraded, 115 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:29.804581+0000 mgr.y (mgr.24401) 622 : cluster [DBG] pgmap v1012: 180 pgs: 45 active+undersized, 20 active+undersized+degraded, 115 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:30.415531+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:30.415531+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:30.417223+0000 mon.c (mon.2) 704 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:30.417223+0000 mon.c (mon.2) 704 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:30.424354+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:30.424354+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.254702+0000 osd.4 (osd.4) 333 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.254702+0000 osd.4 (osd.4) 333 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.257038+0000 osd.4 (osd.4) 334 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.257038+0000 osd.4 (osd.4) 334 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:31.298935+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:31.298935+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.307048+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.307048+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:31.309052+0000 mon.c (mon.2) 705 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:31.309052+0000 mon.c (mon.2) 705 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.804581+0000 mgr.y (mgr.24401) 622 : cluster [DBG] pgmap v1012: 180 pgs: 45 active+undersized, 20 active+undersized+degraded, 115 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:29.804581+0000 mgr.y (mgr.24401) 622 : cluster [DBG] pgmap v1012: 180 pgs: 45 active+undersized, 20 active+undersized+degraded, 115 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:30.415531+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:30.415531+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:30.417223+0000 mon.c (mon.2) 704 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:30.417223+0000 mon.c (mon.2) 704 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:30.424354+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:30.424354+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.254702+0000 osd.4 (osd.4) 333 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.254702+0000 osd.4 (osd.4) 333 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.257038+0000 osd.4 (osd.4) 334 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.257038+0000 osd.4 (osd.4) 334 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:31.298935+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:31.298935+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:31.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.307048+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.307048+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:31.309052+0000 mon.c (mon.2) 705 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:31.309052+0000 mon.c (mon.2) 705 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:31.310520+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[21463]: audit 2024-04-26T16:33:31.310520+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:31.310520+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:31 smithi050 bash[29234]: audit 2024-04-26T16:33:31.310520+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:31.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:31.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.804581+0000 mgr.y (mgr.24401) 622 : cluster [DBG] pgmap v1012: 180 pgs: 45 active+undersized, 20 active+undersized+degraded, 115 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:29.804581+0000 mgr.y (mgr.24401) 622 : cluster [DBG] pgmap v1012: 180 pgs: 45 active+undersized, 20 active+undersized+degraded, 115 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 75/603 objects degraded (12.438%) 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:30.415531+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:30.415531+0000 mon.a (mon.0) 3288 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:30.417223+0000 mon.c (mon.2) 704 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:30.417223+0000 mon.c (mon.2) 704 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:30.424354+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:30.424354+0000 mon.a (mon.0) 3289 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.254702+0000 osd.4 (osd.4) 333 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.254702+0000 osd.4 (osd.4) 333 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.257038+0000 osd.4 (osd.4) 334 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.257038+0000 osd.4 (osd.4) 334 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:31.298935+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:31.768 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:31.298935+0000 mon.a (mon.0) 3290 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:31.769 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.307048+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2024-04-26T16:33:31.769 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.307048+0000 mon.a (mon.0) 3291 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2024-04-26T16:33:31.769 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:31.309052+0000 mon.c (mon.2) 705 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.769 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:31.309052+0000 mon.c (mon.2) 705 : audit [INF] from='client.? 172.21.15.50:0/2762891111' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.769 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:31.310520+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.769 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:31 smithi190 bash[23868]: audit 2024-04-26T16:33:31.310520+0000 mon.a (mon.0) 3292 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]: dispatch 2024-04-26T16:33:31.782 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:31.883 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:33:31.885 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:31.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:31.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:32.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:32.042 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:32.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:32.173 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:32.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:32.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:32.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:32.408 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:32.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:32.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:32.565 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:32.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:32.697 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:32.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:32.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:32.958 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:32.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:33.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:33.088 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:33.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:33.215 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- 172.21.15.50:0/507377192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 msgr2=0x7fb8081099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- 172.21.15.50:0/507377192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb8081099a0 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8009990 tx=0x7fb7f8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- 172.21.15.50:0/507377192 shutdown_connections 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- 172.21.15.50:0/507377192 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- 172.21.15.50:0/507377192 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb8081099a0 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- 172.21.15.50:0/507377192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb80806bb60 0x7fb80806bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- 172.21.15.50:0/507377192 >> 172.21.15.50:0/507377192 conn(0x7fb808079120 msgr2=0x7fb808079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:33.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- 172.21.15.50:0/507377192 shutdown_connections 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- 172.21.15.50:0/507377192 wait complete. 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 librados: starting msgr at 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 librados: starting objecter 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 Processor -- start 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- start start 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 librados: setting wanted keys 2024-04-26T16:33:33.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 librados: calling monclient init 2024-04-26T16:33:33.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb80806bb60 0x7fb8081990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:33.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb80819d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.287+0000 7fb80d626700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb808112b10 con 0x7fb8081095c0 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb808112990 con 0x7fb80806bb60 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb808112c90 con 0x7fb80806c520 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb80819d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8077fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb80819d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56376/0 (socket says 172.21.15.50:56376) 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8077fe700 1 -- 172.21.15.50:0/3892748009 learned_addr learned my addr 172.21.15.50:0/3892748009 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50254/0 (socket says 172.21.15.50:50254) 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb806ffd700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb80806bb60 0x7fb8081990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 -- 172.21.15.50:0/3892748009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb80806bb60 msgr2=0x7fb8081990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb80806bb60 0x7fb8081990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:33.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 -- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 msgr2=0x7fb80819d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb80819d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 -- 172.21.15.50:0/3892748009 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb80819e100 con 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8077fe700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb80819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb8067fc700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808199630 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00d950 tx=0x7fb7fc00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 10 client.?.objecter ms_handle_connect 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc016070 con 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb7fc00fcf0 con 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc010d40 con 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb80819e3f0 con 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 10 client.41507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8081a5c70 con 0x7fb80806c520 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 1 librados: init done 2024-04-26T16:33:33.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb80806a110 con 0x7fb80806c520 2024-04-26T16:33:33.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb7fc019070 con 0x7fb80806c520 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb7f0071f80 0x7fb7f0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(660..660 src has 496..660) ==== 9430+0+0 (secure 0 0 0) 0x7fb7fc091940 con 0x7fb80806c520 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 10 client.41507.objecter ms_dispatch 0x7fb808001550 osd_map(660..660 src has 496..660) 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 3 client.41507.objecter handle_osd_map got epochs [660,660] > 0 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 3 client.41507.objecter handle_osd_map decoding full epoch 660 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb806ffd700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb7f0071f80 0x7fb7f0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.291+0000 7fb7effff700 20 client.41507.objecter dump_active .. 0 homeless 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.295+0000 7fb806ffd700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb7f0071f80 0x7fb7f0074440 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098e0 tx=0x7fb7f8005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:33.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.295+0000 7fb7effff700 10 client.41507.objecter ms_handle_connect 0x7fb7f0071f80 2024-04-26T16:33:33.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.299+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb7fc05d3c0 con 0x7fb80806c520 2024-04-26T16:33:33.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:33.343 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:33.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.749512+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.c scrub starts 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.749512+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.c scrub starts 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.754551+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.c scrub ok 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.754551+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.c scrub ok 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.806742+0000 mgr.y (mgr.24401) 623 : cluster [DBG] pgmap v1015: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:31.806742+0000 mgr.y (mgr.24401) 623 : cluster [DBG] pgmap v1015: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.148080+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.148080+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.149891+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.149891+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: audit 2024-04-26T16:33:32.301790+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: audit 2024-04-26T16:33:32.301790+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:33.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.305973+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.305973+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.421932+0000 mon.a (mon.0) 3295 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.421932+0000 mon.a (mon.0) 3295 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.749512+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.c scrub starts 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.749512+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.c scrub starts 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.754551+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.c scrub ok 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.754551+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.c scrub ok 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.806742+0000 mgr.y (mgr.24401) 623 : cluster [DBG] pgmap v1015: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:31.806742+0000 mgr.y (mgr.24401) 623 : cluster [DBG] pgmap v1015: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.148080+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.148080+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:33:33.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.149891+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.149891+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: audit 2024-04-26T16:33:32.301790+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: audit 2024-04-26T16:33:32.301790+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.305973+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.305973+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.421932+0000 mon.a (mon.0) 3295 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:33.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:33 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.421932+0000 mon.a (mon.0) 3295 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:33.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:33.471 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:33.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:33.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:33.543+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fb80819a450 con 0x7fb80806c520 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.749512+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.c scrub starts 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.749512+0000 osd.2 (osd.2) 319 : cluster [DBG] 2.c scrub starts 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.754551+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.c scrub ok 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.754551+0000 osd.2 (osd.2) 320 : cluster [DBG] 2.c scrub ok 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.806742+0000 mgr.y (mgr.24401) 623 : cluster [DBG] pgmap v1015: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:31.806742+0000 mgr.y (mgr.24401) 623 : cluster [DBG] pgmap v1015: 172 pgs: 41 active+undersized, 19 active+undersized+degraded, 112 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 74/600 objects degraded (12.333%) 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.148080+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.148080+0000 osd.3 (osd.3) 383 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.149891+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.149891+0000 osd.3 (osd.3) 384 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: audit 2024-04-26T16:33:32.301790+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:33.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: audit 2024-04-26T16:33:32.301790+0000 mon.a (mon.0) 3293 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-109"}]': finished 2024-04-26T16:33:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.305973+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2024-04-26T16:33:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.305973+0000 mon.a (mon.0) 3294 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2024-04-26T16:33:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.421932+0000 mon.a (mon.0) 3295 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:33.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:33 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.421932+0000 mon.a (mon.0) 3295 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:33.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:33.663 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:33.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:33.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:33.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:33.914 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:33.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:33.978 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:33:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:33:33.978 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-26T16:33:33.978 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:33:34.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:34.036 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:34.139 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:34.162 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:34.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:34.294 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:34.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.331+0000 7fb7effff700 1 -- 172.21.15.50:0/3892748009 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v662) ==== 101+0+0 (secure 0 0 0) 0x7fb80819a450 con 0x7fb80806c520 2024-04-26T16:33:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 10 librados: watch_flush enter 2024-04-26T16:33:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 10 librados: watch_flush exit 2024-04-26T16:33:34.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 20 client.41507.objecter shutdown clearing up homeless session... 2024-04-26T16:33:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 10 client.41507.objecter successfully canceled tick 2024-04-26T16:33:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb7f0071f80 msgr2=0x7fb7f0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:34.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb7f0071f80 0x7fb7f0074440 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098e0 tx=0x7fb7f8005960 comp rx=0 tx=0).stop 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 msgr2=0x7fb808199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808199630 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00d950 tx=0x7fb7fc00dc60 comp rx=0 tx=0).stop 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 shutdown_connections 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb7f0071f80 0x7fb7f0074440 unknown :-1 s=CLOSED pgs=3809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb80806bb60 0x7fb8081990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb8081095c0 0x7fb80819d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 --2- 172.21.15.50:0/3892748009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb80806c520 0x7fb808199630 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 >> 172.21.15.50:0/3892748009 conn(0x7fb808079120 msgr2=0x7fb808100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 shutdown_connections 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 -- 172.21.15.50:0/3892748009 wait complete. 2024-04-26T16:33:34.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:34.335+0000 7fb80d626700 1 librados: shutdown 2024-04-26T16:33:34.337 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:33:34.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:34.417 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:34.492 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-04-26T16:33:34.493 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:33:34.493 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-26T16:33:34.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:33:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.726529+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.726529+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.728929+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:32.728929+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.316810+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2024-04-26T16:33:34.505 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.316810+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.318015+0000 mon.c (mon.2) 706 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.318015+0000 mon.c (mon.2) 706 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.318378+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.318378+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.549205+0000 mon.c (mon.2) 707 : audit [INF] from='client.? 172.21.15.50:0/3892748009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.549205+0000 mon.c (mon.2) 707 : audit [INF] from='client.? 172.21.15.50:0/3892748009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.549687+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: audit 2024-04-26T16:33:33.549687+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.849503+0000 osd.0 (osd.0) 373 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.849503+0000 osd.0 (osd.0) 373 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.851168+0000 osd.0 (osd.0) 374 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:33:34.506 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.851168+0000 osd.0 (osd.0) 374 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.726529+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.726529+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.728929+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:32.728929+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.316810+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.316810+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.318015+0000 mon.c (mon.2) 706 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.318015+0000 mon.c (mon.2) 706 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.318378+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.318378+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.549205+0000 mon.c (mon.2) 707 : audit [INF] from='client.? 172.21.15.50:0/3892748009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.507 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.549205+0000 mon.c (mon.2) 707 : audit [INF] from='client.? 172.21.15.50:0/3892748009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.549687+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: audit 2024-04-26T16:33:33.549687+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.849503+0000 osd.0 (osd.0) 373 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:33:34.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.849503+0000 osd.0 (osd.0) 373 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:33:34.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.851168+0000 osd.0 (osd.0) 374 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:33:34.508 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:34 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.851168+0000 osd.0 (osd.0) 374 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:33:34.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:34.543 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.726529+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-04-26T16:33:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.726529+0000 osd.2 (osd.2) 321 : cluster [DBG] 5.a scrub starts 2024-04-26T16:33:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.728929+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-04-26T16:33:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:32.728929+0000 osd.2 (osd.2) 322 : cluster [DBG] 5.a scrub ok 2024-04-26T16:33:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.316810+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2024-04-26T16:33:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.316810+0000 mon.a (mon.0) 3296 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.318015+0000 mon.c (mon.2) 706 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.318015+0000 mon.c (mon.2) 706 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.318378+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.318378+0000 mon.a (mon.0) 3297 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.549205+0000 mon.c (mon.2) 707 : audit [INF] from='client.? 172.21.15.50:0/3892748009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.549205+0000 mon.c (mon.2) 707 : audit [INF] from='client.? 172.21.15.50:0/3892748009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.549687+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: audit 2024-04-26T16:33:33.549687+0000 mon.a (mon.0) 3298 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.849503+0000 osd.0 (osd.0) 373 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:33:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.849503+0000 osd.0 (osd.0) 373 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:33:34.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.851168+0000 osd.0 (osd.0) 374 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:33:34.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:34 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.851168+0000 osd.0 (osd.0) 374 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:33:34.646 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:34.673 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:34.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:34.805 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:34.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:34.937 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:34.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:35.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:35.068 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:35.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:35.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:35.200 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:35.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:35.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:35.415 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.807364+0000 mgr.y (mgr.24401) 624 : cluster [DBG] pgmap v1018: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:33.807364+0000 mgr.y (mgr.24401) 624 : cluster [DBG] pgmap v1018: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.309644+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check update: Degraded data redundancy: 73/597 objects degraded (12.228%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.309644+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check update: Degraded data redundancy: 73/597 objects degraded (12.228%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.311270+0000 mon.a (mon.0) 3300 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.311270+0000 mon.a (mon.0) 3300 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.322730+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.322730+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.322893+0000 mon.a (mon.0) 3302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.322893+0000 mon.a (mon.0) 3302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:35.441 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.336512+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.336512+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.339685+0000 mon.c (mon.2) 708 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.339685+0000 mon.c (mon.2) 708 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.340460+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: audit 2024-04-26T16:33:34.340460+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.580833+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.580833+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.584611+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.584611+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.835915+0000 osd.0 (osd.0) 375 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.835915+0000 osd.0 (osd.0) 375 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.838672+0000 osd.0 (osd.0) 376 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:33:35.442 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.838672+0000 osd.0 (osd.0) 376 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.807364+0000 mgr.y (mgr.24401) 624 : cluster [DBG] pgmap v1018: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:33.807364+0000 mgr.y (mgr.24401) 624 : cluster [DBG] pgmap v1018: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.309644+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check update: Degraded data redundancy: 73/597 objects degraded (12.228%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.309644+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check update: Degraded data redundancy: 73/597 objects degraded (12.228%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.311270+0000 mon.a (mon.0) 3300 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.311270+0000 mon.a (mon.0) 3300 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.322730+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.322730+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.322893+0000 mon.a (mon.0) 3302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:35.443 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.322893+0000 mon.a (mon.0) 3302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.336512+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.336512+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.339685+0000 mon.c (mon.2) 708 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.339685+0000 mon.c (mon.2) 708 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.340460+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: audit 2024-04-26T16:33:34.340460+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.580833+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.580833+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.584611+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.584611+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.835915+0000 osd.0 (osd.0) 375 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.835915+0000 osd.0 (osd.0) 375 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.838672+0000 osd.0 (osd.0) 376 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:33:35.444 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:35 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.838672+0000 osd.0 (osd.0) 376 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:33:35.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:35.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:35.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:35.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:35.597 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.807364+0000 mgr.y (mgr.24401) 624 : cluster [DBG] pgmap v1018: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:33.807364+0000 mgr.y (mgr.24401) 624 : cluster [DBG] pgmap v1018: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.309644+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check update: Degraded data redundancy: 73/597 objects degraded (12.228%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.309644+0000 mon.a (mon.0) 3299 : cluster [WRN] Health check update: Degraded data redundancy: 73/597 objects degraded (12.228%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.311270+0000 mon.a (mon.0) 3300 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.311270+0000 mon.a (mon.0) 3300 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.322730+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.322730+0000 mon.a (mon.0) 3301 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.322893+0000 mon.a (mon.0) 3302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.322893+0000 mon.a (mon.0) 3302 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.336512+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2024-04-26T16:33:35.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.336512+0000 mon.a (mon.0) 3303 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.339685+0000 mon.c (mon.2) 708 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.339685+0000 mon.c (mon.2) 708 : audit [INF] from='client.? 172.21.15.50:0/3028810125' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.340460+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: audit 2024-04-26T16:33:34.340460+0000 mon.a (mon.0) 3304 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]: dispatch 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.580833+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.580833+0000 osd.7 (osd.7) 353 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.584611+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.584611+0000 osd.7 (osd.7) 354 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.835915+0000 osd.0 (osd.0) 375 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.835915+0000 osd.0 (osd.0) 375 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.838672+0000 osd.0 (osd.0) 376 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:33:35.600 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:35 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.838672+0000 osd.0 (osd.0) 376 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:33:35.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:35.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:35.730 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:35.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:35.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:35.863 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:35.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:35.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:35.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:36.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:36.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:36.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:36.256 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:36.359 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:36.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.730389+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:33:36.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.730389+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:33:36.368 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.732294+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:34.732294+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.326658+0000 osd.5 (osd.5) 237 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.326658+0000 osd.5 (osd.5) 237 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.326673+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.326673+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.334732+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.334732+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.343019+0000 osd.5 (osd.5) 238 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.343019+0000 osd.5 (osd.5) 238 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.365565+0000 mon.c (mon.2) 709 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.365565+0000 mon.c (mon.2) 709 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.365969+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.365969+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.369 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.366828+0000 mon.c (mon.2) 710 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.366828+0000 mon.c (mon.2) 710 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.367223+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.367223+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.367837+0000 mon.c (mon.2) 711 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.367837+0000 mon.c (mon.2) 711 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.368227+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[29234]: audit 2024-04-26T16:33:35.368227+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.730389+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.730389+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.732294+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:34.732294+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.326658+0000 osd.5 (osd.5) 237 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.326658+0000 osd.5 (osd.5) 237 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.326673+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.326673+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.334732+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.334732+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.343019+0000 osd.5 (osd.5) 238 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.343019+0000 osd.5 (osd.5) 238 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:33:36.371 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.365565+0000 mon.c (mon.2) 709 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.365565+0000 mon.c (mon.2) 709 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.365969+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.365969+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.366828+0000 mon.c (mon.2) 710 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.366828+0000 mon.c (mon.2) 710 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.367223+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.367223+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.367837+0000 mon.c (mon.2) 711 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.367837+0000 mon.c (mon.2) 711 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.368227+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:36 smithi050 bash[21463]: audit 2024-04-26T16:33:35.368227+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.381 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:36.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:36.484 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:36.512 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:36.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:36.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.730389+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:33:36.627 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.730389+0000 osd.2 (osd.2) 323 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.732294+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:34.732294+0000 osd.2 (osd.2) 324 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.326658+0000 osd.5 (osd.5) 237 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.326658+0000 osd.5 (osd.5) 237 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.326673+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.326673+0000 mon.a (mon.0) 3305 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi050-73372-104"}]': finished 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.334732+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.334732+0000 mon.a (mon.0) 3306 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.343019+0000 osd.5 (osd.5) 238 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:33:36.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.343019+0000 osd.5 (osd.5) 238 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.365565+0000 mon.c (mon.2) 709 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.365565+0000 mon.c (mon.2) 709 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.365969+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.365969+0000 mon.a (mon.0) 3307 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.366828+0000 mon.c (mon.2) 710 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.366828+0000 mon.c (mon.2) 710 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.367223+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.367223+0000 mon.a (mon.0) 3308 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.367837+0000 mon.c (mon.2) 711 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.367837+0000 mon.c (mon.2) 711 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.368227+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:36 smithi190 bash[23868]: audit 2024-04-26T16:33:35.368227+0000 mon.a (mon.0) 3309 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-04-26T16:33:36.641 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:36.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:36.771 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:36.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:36.902 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:36.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:37.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:37.034 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:37.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:37.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:37.166 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:37.270 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:37.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:37.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:33:37.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:37.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.808327+0000 mgr.y (mgr.24401) 625 : cluster [DBG] pgmap v1021: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 73/597 objects degraded (12.228%) 2024-04-26T16:33:37.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:35.808327+0000 mgr.y (mgr.24401) 625 : cluster [DBG] pgmap v1021: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 73/597 objects degraded (12.228%) 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: audit 2024-04-26T16:33:36.358233+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: audit 2024-04-26T16:33:36.358233+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.364630+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.364630+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: audit 2024-04-26T16:33:36.365879+0000 mon.c (mon.2) 712 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: audit 2024-04-26T16:33:36.365879+0000 mon.c (mon.2) 712 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: audit 2024-04-26T16:33:36.366405+0000 mon.a (mon.0) 3312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: audit 2024-04-26T16:33:36.366405+0000 mon.a (mon.0) 3312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.583075+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.583075+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.586945+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.586945+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:33:37.414 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.693774+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:37.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.693774+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:37.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.694187+0000 osd.1 (osd.1) 352 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:37.415 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:37 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.694187+0000 osd.1 (osd.1) 352 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:37.427 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:37.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:37.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.808327+0000 mgr.y (mgr.24401) 625 : cluster [DBG] pgmap v1021: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 73/597 objects degraded (12.228%) 2024-04-26T16:33:37.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:35.808327+0000 mgr.y (mgr.24401) 625 : cluster [DBG] pgmap v1021: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 73/597 objects degraded (12.228%) 2024-04-26T16:33:37.543 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: audit 2024-04-26T16:33:36.358233+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: audit 2024-04-26T16:33:36.358233+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.364630+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.364630+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: audit 2024-04-26T16:33:36.365879+0000 mon.c (mon.2) 712 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: audit 2024-04-26T16:33:36.365879+0000 mon.c (mon.2) 712 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: audit 2024-04-26T16:33:36.366405+0000 mon.a (mon.0) 3312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: audit 2024-04-26T16:33:36.366405+0000 mon.a (mon.0) 3312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.583075+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.583075+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.586945+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.586945+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.693774+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.693774+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:37.544 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.694187+0000 osd.1 (osd.1) 352 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.694187+0000 osd.1 (osd.1) 352 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.808327+0000 mgr.y (mgr.24401) 625 : cluster [DBG] pgmap v1021: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 73/597 objects degraded (12.228%) 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:35.808327+0000 mgr.y (mgr.24401) 625 : cluster [DBG] pgmap v1021: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 663 MiB used, 715 GiB / 715 GiB avail; 73/597 objects degraded (12.228%) 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: audit 2024-04-26T16:33:36.358233+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: audit 2024-04-26T16:33:36.358233+0000 mon.a (mon.0) 3310 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi050-73372-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.364630+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.364630+0000 mon.a (mon.0) 3311 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: audit 2024-04-26T16:33:36.365879+0000 mon.c (mon.2) 712 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: audit 2024-04-26T16:33:36.365879+0000 mon.c (mon.2) 712 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: audit 2024-04-26T16:33:36.366405+0000 mon.a (mon.0) 3312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.545 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: audit 2024-04-26T16:33:36.366405+0000 mon.a (mon.0) 3312 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.583075+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.583075+0000 osd.7 (osd.7) 355 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.586945+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.586945+0000 osd.7 (osd.7) 356 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.693774+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.693774+0000 osd.1 (osd.1) 351 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.694187+0000 osd.1 (osd.1) 352 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:37.546 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:37 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.694187+0000 osd.1 (osd.1) 352 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:37.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:37.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:37.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:37.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:37.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:37.816 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:37.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:37.942 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:37.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:38.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:38.073 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:38.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:33:38.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:38.222 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:38.325 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:38.350 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:38.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.739058+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.739058+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.741728+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:36.741728+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: audit 2024-04-26T16:33:37.134941+0000 mgr.y (mgr.24401) 626 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: audit 2024-04-26T16:33:37.134941+0000 mgr.y (mgr.24401) 626 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.323465+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:33:38.462 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.323465+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.325587+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.325587+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.378510+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.378510+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.398349+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.398349+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.401055+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.401055+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: audit 2024-04-26T16:33:38.363285+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: audit 2024-04-26T16:33:38.363285+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.366891+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.366891+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2024-04-26T16:33:38.463 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.739058+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.739058+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.741728+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:36.741728+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: audit 2024-04-26T16:33:37.134941+0000 mgr.y (mgr.24401) 626 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: audit 2024-04-26T16:33:37.134941+0000 mgr.y (mgr.24401) 626 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.323465+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.323465+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.325587+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.325587+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.378510+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.378510+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.398349+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.398349+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.401055+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:33:38.464 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.401055+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:33:38.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: audit 2024-04-26T16:33:38.363285+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:38.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: audit 2024-04-26T16:33:38.363285+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:38.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.366891+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2024-04-26T16:33:38.465 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:38 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.366891+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2024-04-26T16:33:38.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:38.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:38.607 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:38.710 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:38.738 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.739058+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.739058+0000 osd.2 (osd.2) 325 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.741728+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:36.741728+0000 osd.2 (osd.2) 326 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: audit 2024-04-26T16:33:37.134941+0000 mgr.y (mgr.24401) 626 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: audit 2024-04-26T16:33:37.134941+0000 mgr.y (mgr.24401) 626 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.323465+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.323465+0000 osd.5 (osd.5) 239 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.325587+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.325587+0000 osd.5 (osd.5) 240 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:33:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.378510+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.378510+0000 mon.a (mon.0) 3313 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.398349+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.398349+0000 osd.1 (osd.1) 353 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.401055+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.401055+0000 osd.1 (osd.1) 354 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: audit 2024-04-26T16:33:38.363285+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: audit 2024-04-26T16:33:38.363285+0000 mon.a (mon.0) 3314 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi050-73372-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.366891+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2024-04-26T16:33:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:38 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.366891+0000 mon.a (mon.0) 3315 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2024-04-26T16:33:38.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:38.871 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:38.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:38.936 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:38.974 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:38.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:39.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:39.129 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:39.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:39.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:39.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:39.403 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:39.507 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:39.521 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.787692+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.787692+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.790325+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.790325+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.809365+0000 mgr.y (mgr.24401) 627 : cluster [DBG] pgmap v1024: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:37.809365+0000 mgr.y (mgr.24401) 627 : cluster [DBG] pgmap v1024: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.276362+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.276362+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.278808+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.278808+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.430386+0000 osd.1 (osd.1) 355 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.430386+0000 osd.1 (osd.1) 355 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:33:39.522 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.431934+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.431934+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.751277+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.751277+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.754235+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:38.754235+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.376052+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.376052+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: audit 2024-04-26T16:33:39.385118+0000 mon.c (mon.2) 713 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: audit 2024-04-26T16:33:39.385118+0000 mon.c (mon.2) 713 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: audit 2024-04-26T16:33:39.386292+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.523 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[29234]: audit 2024-04-26T16:33:39.386292+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.787692+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.787692+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.790325+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.790325+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.809365+0000 mgr.y (mgr.24401) 627 : cluster [DBG] pgmap v1024: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:37.809365+0000 mgr.y (mgr.24401) 627 : cluster [DBG] pgmap v1024: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.276362+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.276362+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.278808+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.278808+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.430386+0000 osd.1 (osd.1) 355 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.430386+0000 osd.1 (osd.1) 355 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:33:39.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.431934+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.431934+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.751277+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.751277+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.754235+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:38.754235+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.376052+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.376052+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: audit 2024-04-26T16:33:39.385118+0000 mon.c (mon.2) 713 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: audit 2024-04-26T16:33:39.385118+0000 mon.c (mon.2) 713 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: audit 2024-04-26T16:33:39.386292+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:39 smithi050 bash[21463]: audit 2024-04-26T16:33:39.386292+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.538 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:39.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:39.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:39.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:39.798 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.787692+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.787692+0000 osd.2 (osd.2) 327 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.790325+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.790325+0000 osd.2 (osd.2) 328 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.809365+0000 mgr.y (mgr.24401) 627 : cluster [DBG] pgmap v1024: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:37.809365+0000 mgr.y (mgr.24401) 627 : cluster [DBG] pgmap v1024: 164 pgs: 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:39.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.276362+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.276362+0000 osd.5 (osd.5) 241 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.278808+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.278808+0000 osd.5 (osd.5) 242 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.430386+0000 osd.1 (osd.1) 355 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.430386+0000 osd.1 (osd.1) 355 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.431934+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.431934+0000 osd.1 (osd.1) 356 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.751277+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.751277+0000 osd.0 (osd.0) 377 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.754235+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:38.754235+0000 osd.0 (osd.0) 378 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.376052+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2024-04-26T16:33:39.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.376052+0000 mon.a (mon.0) 3316 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2024-04-26T16:33:39.812 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: audit 2024-04-26T16:33:39.385118+0000 mon.c (mon.2) 713 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.812 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: audit 2024-04-26T16:33:39.385118+0000 mon.c (mon.2) 713 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.812 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: audit 2024-04-26T16:33:39.386292+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.812 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:39 smithi190 bash[23868]: audit 2024-04-26T16:33:39.386292+0000 mon.a (mon.0) 3317 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:39.902 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:39.933 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:39.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:39.949 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:39.950 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:40.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:40.062 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:40.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:40.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:40.192 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:40.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:40.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:40.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:40.448 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:40.551 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:40.579 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.287689+0000 osd.5 (osd.5) 243 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.287689+0000 osd.5 (osd.5) 243 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.290124+0000 osd.5 (osd.5) 244 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.290124+0000 osd.5 (osd.5) 244 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.361562+0000 osd.4 (osd.4) 335 : cluster [DBG] 11.e scrub starts 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.361562+0000 osd.4 (osd.4) 335 : cluster [DBG] 11.e scrub starts 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.364264+0000 osd.4 (osd.4) 336 : cluster [DBG] 11.e scrub ok 2024-04-26T16:33:40.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.364264+0000 osd.4 (osd.4) 336 : cluster [DBG] 11.e scrub ok 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.414745+0000 osd.1 (osd.1) 357 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.414745+0000 osd.1 (osd.1) 357 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.416832+0000 osd.1 (osd.1) 358 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.416832+0000 osd.1 (osd.1) 358 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.719945+0000 osd.0 (osd.0) 379 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.719945+0000 osd.0 (osd.0) 379 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.721521+0000 osd.0 (osd.0) 380 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.721521+0000 osd.0 (osd.0) 380 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: audit 2024-04-26T16:33:40.369748+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: audit 2024-04-26T16:33:40.369748+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.374965+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2024-04-26T16:33:40.583 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.374965+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.287689+0000 osd.5 (osd.5) 243 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.287689+0000 osd.5 (osd.5) 243 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.290124+0000 osd.5 (osd.5) 244 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.290124+0000 osd.5 (osd.5) 244 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.361562+0000 osd.4 (osd.4) 335 : cluster [DBG] 11.e scrub starts 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.361562+0000 osd.4 (osd.4) 335 : cluster [DBG] 11.e scrub starts 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.364264+0000 osd.4 (osd.4) 336 : cluster [DBG] 11.e scrub ok 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.364264+0000 osd.4 (osd.4) 336 : cluster [DBG] 11.e scrub ok 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.414745+0000 osd.1 (osd.1) 357 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.414745+0000 osd.1 (osd.1) 357 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.416832+0000 osd.1 (osd.1) 358 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.416832+0000 osd.1 (osd.1) 358 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:33:40.584 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.719945+0000 osd.0 (osd.0) 379 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.719945+0000 osd.0 (osd.0) 379 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.721521+0000 osd.0 (osd.0) 380 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.721521+0000 osd.0 (osd.0) 380 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: audit 2024-04-26T16:33:40.369748+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: audit 2024-04-26T16:33:40.369748+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.374965+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2024-04-26T16:33:40.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:40 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.374965+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2024-04-26T16:33:40.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:40.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:40.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.287689+0000 osd.5 (osd.5) 243 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.287689+0000 osd.5 (osd.5) 243 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.290124+0000 osd.5 (osd.5) 244 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.290124+0000 osd.5 (osd.5) 244 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.361562+0000 osd.4 (osd.4) 335 : cluster [DBG] 11.e scrub starts 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.361562+0000 osd.4 (osd.4) 335 : cluster [DBG] 11.e scrub starts 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.364264+0000 osd.4 (osd.4) 336 : cluster [DBG] 11.e scrub ok 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.364264+0000 osd.4 (osd.4) 336 : cluster [DBG] 11.e scrub ok 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.414745+0000 osd.1 (osd.1) 357 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.414745+0000 osd.1 (osd.1) 357 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.416832+0000 osd.1 (osd.1) 358 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.416832+0000 osd.1 (osd.1) 358 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:33:40.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.719945+0000 osd.0 (osd.0) 379 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.719945+0000 osd.0 (osd.0) 379 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.721521+0000 osd.0 (osd.0) 380 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.721521+0000 osd.0 (osd.0) 380 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: audit 2024-04-26T16:33:40.369748+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: audit 2024-04-26T16:33:40.369748+0000 mon.a (mon.0) 3318 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.374965+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2024-04-26T16:33:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:40 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.374965+0000 mon.a (mon.0) 3319 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2024-04-26T16:33:40.839 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:40.942 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:40.971 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:40.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:41.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:41.101 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:41.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:41.232 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:41.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:41.370 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:41.472 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:41.499 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:41.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:41.614 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.810430+0000 mgr.y (mgr.24401) 628 : cluster [DBG] pgmap v1027: 204 pgs: 40 unknown, 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:41.614 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.810430+0000 mgr.y (mgr.24401) 628 : cluster [DBG] pgmap v1027: 204 pgs: 40 unknown, 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:41.614 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.834852+0000 osd.2 (osd.2) 329 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:33:41.614 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.834852+0000 osd.2 (osd.2) 329 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:33:41.614 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.842896+0000 osd.2 (osd.2) 330 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:39.842896+0000 osd.2 (osd.2) 330 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.237773+0000 osd.3 (osd.3) 385 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.237773+0000 osd.3 (osd.3) 385 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.239886+0000 osd.3 (osd.3) 386 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.239886+0000 osd.3 (osd.3) 386 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.402067+0000 mon.a (mon.0) 3320 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.402067+0000 mon.a (mon.0) 3320 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:40.411752+0000 mon.c (mon.2) 714 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:40.411752+0000 mon.c (mon.2) 714 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:40.419494+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:40.419494+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.750925+0000 osd.0 (osd.0) 381 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:33:41.615 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.750925+0000 osd.0 (osd.0) 381 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.752769+0000 osd.0 (osd.0) 382 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.752769+0000 osd.0 (osd.0) 382 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:41.306824+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:41.306824+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.315331+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.315331+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:41.317268+0000 mon.c (mon.2) 715 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:41.317268+0000 mon.c (mon.2) 715 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:41.320931+0000 mon.a (mon.0) 3324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.616 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:41 smithi190 bash[23868]: audit 2024-04-26T16:33:41.320931+0000 mon.a (mon.0) 3324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.630 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:41.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:41.758 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:41.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:41.873 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.810430+0000 mgr.y (mgr.24401) 628 : cluster [DBG] pgmap v1027: 204 pgs: 40 unknown, 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:41.873 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.810430+0000 mgr.y (mgr.24401) 628 : cluster [DBG] pgmap v1027: 204 pgs: 40 unknown, 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:41.873 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.834852+0000 osd.2 (osd.2) 329 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.834852+0000 osd.2 (osd.2) 329 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.842896+0000 osd.2 (osd.2) 330 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:39.842896+0000 osd.2 (osd.2) 330 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.237773+0000 osd.3 (osd.3) 385 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.237773+0000 osd.3 (osd.3) 385 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.239886+0000 osd.3 (osd.3) 386 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.239886+0000 osd.3 (osd.3) 386 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.402067+0000 mon.a (mon.0) 3320 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.402067+0000 mon.a (mon.0) 3320 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:40.411752+0000 mon.c (mon.2) 714 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:40.411752+0000 mon.c (mon.2) 714 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:40.419494+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:40.419494+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.750925+0000 osd.0 (osd.0) 381 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.750925+0000 osd.0 (osd.0) 381 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.752769+0000 osd.0 (osd.0) 382 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.752769+0000 osd.0 (osd.0) 382 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:41.306824+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:41.306824+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.315331+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.315331+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:41.317268+0000 mon.c (mon.2) 715 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:41.317268+0000 mon.c (mon.2) 715 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:41.320931+0000 mon.a (mon.0) 3324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.875 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[29234]: audit 2024-04-26T16:33:41.320931+0000 mon.a (mon.0) 3324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.810430+0000 mgr.y (mgr.24401) 628 : cluster [DBG] pgmap v1027: 204 pgs: 40 unknown, 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.810430+0000 mgr.y (mgr.24401) 628 : cluster [DBG] pgmap v1027: 204 pgs: 40 unknown, 38 active+undersized, 18 active+undersized+degraded, 108 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 73/597 objects degraded (12.228%) 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.834852+0000 osd.2 (osd.2) 329 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.834852+0000 osd.2 (osd.2) 329 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.842896+0000 osd.2 (osd.2) 330 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:39.842896+0000 osd.2 (osd.2) 330 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.237773+0000 osd.3 (osd.3) 385 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.237773+0000 osd.3 (osd.3) 385 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.239886+0000 osd.3 (osd.3) 386 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.239886+0000 osd.3 (osd.3) 386 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:33:41.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.402067+0000 mon.a (mon.0) 3320 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.402067+0000 mon.a (mon.0) 3320 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:40.411752+0000 mon.c (mon.2) 714 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:40.411752+0000 mon.c (mon.2) 714 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:40.419494+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:40.419494+0000 mon.a (mon.0) 3321 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.750925+0000 osd.0 (osd.0) 381 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.750925+0000 osd.0 (osd.0) 381 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.752769+0000 osd.0 (osd.0) 382 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.752769+0000 osd.0 (osd.0) 382 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:41.306824+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:41.306824+0000 mon.a (mon.0) 3322 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.315331+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2024-04-26T16:33:41.877 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.315331+0000 mon.a (mon.0) 3323 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2024-04-26T16:33:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:41.317268+0000 mon.c (mon.2) 715 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:41.317268+0000 mon.c (mon.2) 715 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:41.320931+0000 mon.a (mon.0) 3324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.878 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:41 smithi050 bash[21463]: audit 2024-04-26T16:33:41.320931+0000 mon.a (mon.0) 3324 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:41.889 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:41.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:33:41.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:42.017 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:42.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:42.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:42.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:42.275 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:42.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:42.404 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:42.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:42.508 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:42.536 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:42.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:42.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:42.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.687080+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.687080+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.687468+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:40.687468+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.195454+0000 osd.3 (osd.3) 387 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.195454+0000 osd.3 (osd.3) 387 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.197184+0000 osd.3 (osd.3) 388 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.197184+0000 osd.3 (osd.3) 388 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:33:42.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.793612+0000 osd.0 (osd.0) 383 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.793612+0000 osd.0 (osd.0) 383 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.801731+0000 osd.0 (osd.0) 384 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.801731+0000 osd.0 (osd.0) 384 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: audit 2024-04-26T16:33:42.309628+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: audit 2024-04-26T16:33:42.309628+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.318272+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.318272+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.345523+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.345523+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.347465+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.347465+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: audit 2024-04-26T16:33:42.375949+0000 mon.c (mon.2) 716 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: audit 2024-04-26T16:33:42.375949+0000 mon.c (mon.2) 716 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: audit 2024-04-26T16:33:42.376697+0000 mon.a (mon.0) 3327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[21463]: audit 2024-04-26T16:33:42.376697+0000 mon.a (mon.0) 3327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.687080+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.687080+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.687468+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:40.687468+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.195454+0000 osd.3 (osd.3) 387 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.195454+0000 osd.3 (osd.3) 387 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.197184+0000 osd.3 (osd.3) 388 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.197184+0000 osd.3 (osd.3) 388 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.793612+0000 osd.0 (osd.0) 383 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:33:42.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.793612+0000 osd.0 (osd.0) 383 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.801731+0000 osd.0 (osd.0) 384 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.801731+0000 osd.0 (osd.0) 384 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: audit 2024-04-26T16:33:42.309628+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: audit 2024-04-26T16:33:42.309628+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.318272+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.318272+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.345523+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.345523+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.347465+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.347465+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: audit 2024-04-26T16:33:42.375949+0000 mon.c (mon.2) 716 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: audit 2024-04-26T16:33:42.375949+0000 mon.c (mon.2) 716 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: audit 2024-04-26T16:33:42.376697+0000 mon.a (mon.0) 3327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:42 smithi050 bash[29234]: audit 2024-04-26T16:33:42.376697+0000 mon.a (mon.0) 3327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.687080+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.687080+0000 osd.2 (osd.2) 331 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.687468+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:40.687468+0000 osd.2 (osd.2) 332 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:42.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.195454+0000 osd.3 (osd.3) 387 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.195454+0000 osd.3 (osd.3) 387 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.197184+0000 osd.3 (osd.3) 388 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.197184+0000 osd.3 (osd.3) 388 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.793612+0000 osd.0 (osd.0) 383 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.793612+0000 osd.0 (osd.0) 383 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.801731+0000 osd.0 (osd.0) 384 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.801731+0000 osd.0 (osd.0) 384 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: audit 2024-04-26T16:33:42.309628+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: audit 2024-04-26T16:33:42.309628+0000 mon.a (mon.0) 3325 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.318272+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.318272+0000 mon.a (mon.0) 3326 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.345523+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.345523+0000 osd.4 (osd.4) 337 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:33:42.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.347465+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:33:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.347465+0000 osd.4 (osd.4) 338 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:33:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: audit 2024-04-26T16:33:42.375949+0000 mon.c (mon.2) 716 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: audit 2024-04-26T16:33:42.375949+0000 mon.c (mon.2) 716 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: audit 2024-04-26T16:33:42.376697+0000 mon.a (mon.0) 3327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.785 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:42 smithi190 bash[23868]: audit 2024-04-26T16:33:42.376697+0000 mon.a (mon.0) 3327 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:42.798 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:42.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:42.929 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:42.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:43.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:43.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:43.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:43.192 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:43.295 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:43.329 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:43.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.809855+0000 osd.2 (osd.2) 333 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:33:43.425 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.809855+0000 osd.2 (osd.2) 333 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:33:43.426 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.813133+0000 mgr.y (mgr.24401) 629 : cluster [DBG] pgmap v1030: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:43.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:43.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:43.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:43.582 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.809855+0000 osd.2 (osd.2) 333 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.809855+0000 osd.2 (osd.2) 333 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.813133+0000 mgr.y (mgr.24401) 629 : cluster [DBG] pgmap v1030: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.813133+0000 mgr.y (mgr.24401) 629 : cluster [DBG] pgmap v1030: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.813403+0000 osd.2 (osd.2) 334 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:41.813403+0000 osd.2 (osd.2) 334 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.226558+0000 osd.3 (osd.3) 389 : cluster [DBG] 3.b scrub starts 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.226558+0000 osd.3 (osd.3) 389 : cluster [DBG] 3.b scrub starts 2024-04-26T16:33:43.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.230558+0000 osd.3 (osd.3) 390 : cluster [DBG] 3.b scrub ok 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.230558+0000 osd.3 (osd.3) 390 : cluster [DBG] 3.b scrub ok 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.417383+0000 mon.a (mon.0) 3328 : cluster [WRN] Health check update: Degraded data redundancy: 74/603 objects degraded (12.272%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:42.417383+0000 mon.a (mon.0) 3328 : cluster [WRN] Health check update: Degraded data redundancy: 74/603 objects degraded (12.272%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:42.433158+0000 mon.a (mon.0) 3329 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:42.433158+0000 mon.a (mon.0) 3329 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:42.437513+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:42.437513+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:43.312702+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:43.312702+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.323125+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.323125+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:43.324538+0000 mon.c (mon.2) 718 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:43.324538+0000 mon.c (mon.2) 718 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:43.325769+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[21463]: audit 2024-04-26T16:33:43.325769+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.809855+0000 osd.2 (osd.2) 333 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.809855+0000 osd.2 (osd.2) 333 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.813133+0000 mgr.y (mgr.24401) 629 : cluster [DBG] pgmap v1030: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.813133+0000 mgr.y (mgr.24401) 629 : cluster [DBG] pgmap v1030: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.813403+0000 osd.2 (osd.2) 334 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:41.813403+0000 osd.2 (osd.2) 334 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.226558+0000 osd.3 (osd.3) 389 : cluster [DBG] 3.b scrub starts 2024-04-26T16:33:43.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.226558+0000 osd.3 (osd.3) 389 : cluster [DBG] 3.b scrub starts 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.230558+0000 osd.3 (osd.3) 390 : cluster [DBG] 3.b scrub ok 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.230558+0000 osd.3 (osd.3) 390 : cluster [DBG] 3.b scrub ok 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.417383+0000 mon.a (mon.0) 3328 : cluster [WRN] Health check update: Degraded data redundancy: 74/603 objects degraded (12.272%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:42.417383+0000 mon.a (mon.0) 3328 : cluster [WRN] Health check update: Degraded data redundancy: 74/603 objects degraded (12.272%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:42.433158+0000 mon.a (mon.0) 3329 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:42.433158+0000 mon.a (mon.0) 3329 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:42.437513+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:42.437513+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:43.312702+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:43.312702+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.323125+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.323125+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:43.324538+0000 mon.c (mon.2) 718 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:43.324538+0000 mon.c (mon.2) 718 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:43.325769+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:43 smithi050 bash[29234]: audit 2024-04-26T16:33:43.325769+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.685 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:43.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:43.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.813133+0000 mgr.y (mgr.24401) 629 : cluster [DBG] pgmap v1030: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.813403+0000 osd.2 (osd.2) 334 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:41.813403+0000 osd.2 (osd.2) 334 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.226558+0000 osd.3 (osd.3) 389 : cluster [DBG] 3.b scrub starts 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.226558+0000 osd.3 (osd.3) 389 : cluster [DBG] 3.b scrub starts 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.230558+0000 osd.3 (osd.3) 390 : cluster [DBG] 3.b scrub ok 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.230558+0000 osd.3 (osd.3) 390 : cluster [DBG] 3.b scrub ok 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.417383+0000 mon.a (mon.0) 3328 : cluster [WRN] Health check update: Degraded data redundancy: 74/603 objects degraded (12.272%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:42.417383+0000 mon.a (mon.0) 3328 : cluster [WRN] Health check update: Degraded data redundancy: 74/603 objects degraded (12.272%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:42.433158+0000 mon.a (mon.0) 3329 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:43.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:42.433158+0000 mon.a (mon.0) 3329 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:42.437513+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:42.437513+0000 mon.c (mon.2) 717 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:43.312702+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:43.312702+0000 mon.a (mon.0) 3330 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.323125+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.323125+0000 mon.a (mon.0) 3331 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:43.324538+0000 mon.c (mon.2) 718 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:43.324538+0000 mon.c (mon.2) 718 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:43.325769+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:43 smithi190 bash[23868]: audit 2024-04-26T16:33:43.325769+0000 mon.a (mon.0) 3332 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]: dispatch 2024-04-26T16:33:43.837 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:43.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:43.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:43.968 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:43.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:44.071 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:44.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:44.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:44.226 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:44.329 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:44.359 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:44.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:44.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:44.473 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.738155+0000 osd.0 (osd.0) 385 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.738155+0000 osd.0 (osd.0) 385 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.739642+0000 osd.0 (osd.0) 386 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.739642+0000 osd.0 (osd.0) 386 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: audit 2024-04-26T16:33:44.315118+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: audit 2024-04-26T16:33:44.315118+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: cluster 2024-04-26T16:33:44.323412+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2024-04-26T16:33:44.474 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:44 smithi190 bash[23868]: cluster 2024-04-26T16:33:44.323412+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2024-04-26T16:33:44.488 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:44.591 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:33:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.738155+0000 osd.0 (osd.0) 385 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.738155+0000 osd.0 (osd.0) 385 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.739642+0000 osd.0 (osd.0) 386 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.739642+0000 osd.0 (osd.0) 386 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: audit 2024-04-26T16:33:44.315118+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: audit 2024-04-26T16:33:44.315118+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: cluster 2024-04-26T16:33:44.323412+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2024-04-26T16:33:44.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[29234]: cluster 2024-04-26T16:33:44.323412+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.738155+0000 osd.0 (osd.0) 385 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.738155+0000 osd.0 (osd.0) 385 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.739642+0000 osd.0 (osd.0) 386 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.739642+0000 osd.0 (osd.0) 386 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: audit 2024-04-26T16:33:44.315118+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: audit 2024-04-26T16:33:44.315118+0000 mon.a (mon.0) 3333 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-112"}]': finished 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: cluster 2024-04-26T16:33:44.323412+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2024-04-26T16:33:44.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:44 smithi050 bash[21463]: cluster 2024-04-26T16:33:44.323412+0000 mon.a (mon.0) 3334 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2024-04-26T16:33:44.620 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:44.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:44.750 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:44.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:44.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:44.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:44.989 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:45.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:45.122 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:45.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:45.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:45.280 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:45.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:45.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:45.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:45.511 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:45.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:45.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:45 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.814265+0000 mgr.y (mgr.24401) 630 : cluster [DBG] pgmap v1033: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:45.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:45 smithi190 bash[23868]: cluster 2024-04-26T16:33:43.814265+0000 mgr.y (mgr.24401) 630 : cluster [DBG] pgmap v1033: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:45.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:45 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.326004+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2024-04-26T16:33:45.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:45 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.326004+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2024-04-26T16:33:45.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:45.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:45.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:45.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.814265+0000 mgr.y (mgr.24401) 630 : cluster [DBG] pgmap v1033: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:45.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[29234]: cluster 2024-04-26T16:33:43.814265+0000 mgr.y (mgr.24401) 630 : cluster [DBG] pgmap v1033: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:45.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.326004+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2024-04-26T16:33:45.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.326004+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2024-04-26T16:33:45.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.814265+0000 mgr.y (mgr.24401) 630 : cluster [DBG] pgmap v1033: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:45.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[21463]: cluster 2024-04-26T16:33:43.814265+0000 mgr.y (mgr.24401) 630 : cluster [DBG] pgmap v1033: 204 pgs: 51 active+undersized, 19 active+undersized+degraded, 134 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 74/603 objects degraded (12.272%) 2024-04-26T16:33:45.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.326004+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2024-04-26T16:33:45.778 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:45 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.326004+0000 mon.a (mon.0) 3335 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2024-04-26T16:33:45.799 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:45.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:45.929 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:45.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:46.032 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:46.060 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:46.163 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:46.184 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:46.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:46.320 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:46.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:46.454 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:46.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:46.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:46.567 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.519738+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.b scrub starts 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.519738+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.b scrub starts 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.527399+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.b scrub ok 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.527399+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.b scrub ok 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.306116+0000 mon.a (mon.0) 3336 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.306116+0000 mon.a (mon.0) 3336 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.325259+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.325259+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: audit 2024-04-26T16:33:46.327701+0000 mon.c (mon.2) 719 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: audit 2024-04-26T16:33:46.327701+0000 mon.c (mon.2) 719 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: audit 2024-04-26T16:33:46.329980+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[29234]: audit 2024-04-26T16:33:46.329980+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.519738+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.b scrub starts 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.519738+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.b scrub starts 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.527399+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.b scrub ok 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.527399+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.b scrub ok 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.306116+0000 mon.a (mon.0) 3336 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.306116+0000 mon.a (mon.0) 3336 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.325259+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.325259+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: audit 2024-04-26T16:33:46.327701+0000 mon.c (mon.2) 719 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: audit 2024-04-26T16:33:46.327701+0000 mon.c (mon.2) 719 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: audit 2024-04-26T16:33:46.329980+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:46 smithi050 bash[21463]: audit 2024-04-26T16:33:46.329980+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:46.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:46.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:46.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.519738+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.b scrub starts 2024-04-26T16:33:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.519738+0000 osd.7 (osd.7) 357 : cluster [DBG] 2.b scrub starts 2024-04-26T16:33:46.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.527399+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.b scrub ok 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.527399+0000 osd.7 (osd.7) 358 : cluster [DBG] 2.b scrub ok 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.306116+0000 mon.a (mon.0) 3336 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.306116+0000 mon.a (mon.0) 3336 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.325259+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.325259+0000 mon.a (mon.0) 3337 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: audit 2024-04-26T16:33:46.327701+0000 mon.c (mon.2) 719 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: audit 2024-04-26T16:33:46.327701+0000 mon.c (mon.2) 719 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: audit 2024-04-26T16:33:46.329980+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:46 smithi190 bash[23868]: audit 2024-04-26T16:33:46.329980+0000 mon.a (mon.0) 3338 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:46.842 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:46.945 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:46.975 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:46.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:47.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:47.106 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:47.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:47.235 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:47.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:47.349 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:33:47.364 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:47.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:47.493 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:47.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.815357+0000 mgr.y (mgr.24401) 631 : cluster [DBG] pgmap v1036: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 73/600 objects degraded (12.167%) 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:45.815357+0000 mgr.y (mgr.24401) 631 : cluster [DBG] pgmap v1036: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 73/600 objects degraded (12.167%) 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.204555+0000 osd.3 (osd.3) 391 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.204555+0000 osd.3 (osd.3) 391 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.206492+0000 osd.3 (osd.3) 392 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.206492+0000 osd.3 (osd.3) 392 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.442615+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.442615+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.444920+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:46.444920+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:33:47.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: audit 2024-04-26T16:33:47.324199+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: audit 2024-04-26T16:33:47.324199+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.333311+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.333311+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: audit 2024-04-26T16:33:47.404624+0000 mon.c (mon.2) 720 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: audit 2024-04-26T16:33:47.404624+0000 mon.c (mon.2) 720 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: audit 2024-04-26T16:33:47.407417+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:47 smithi190 bash[23868]: audit 2024-04-26T16:33:47.407417+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.625 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:47.728 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:47.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.815357+0000 mgr.y (mgr.24401) 631 : cluster [DBG] pgmap v1036: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 73/600 objects degraded (12.167%) 2024-04-26T16:33:47.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:45.815357+0000 mgr.y (mgr.24401) 631 : cluster [DBG] pgmap v1036: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 73/600 objects degraded (12.167%) 2024-04-26T16:33:47.740 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.204555+0000 osd.3 (osd.3) 391 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.204555+0000 osd.3 (osd.3) 391 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.206492+0000 osd.3 (osd.3) 392 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.206492+0000 osd.3 (osd.3) 392 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.442615+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.442615+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.444920+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:46.444920+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: audit 2024-04-26T16:33:47.324199+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: audit 2024-04-26T16:33:47.324199+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.333311+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.333311+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: audit 2024-04-26T16:33:47.404624+0000 mon.c (mon.2) 720 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: audit 2024-04-26T16:33:47.404624+0000 mon.c (mon.2) 720 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.741 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: audit 2024-04-26T16:33:47.407417+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[21463]: audit 2024-04-26T16:33:47.407417+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.815357+0000 mgr.y (mgr.24401) 631 : cluster [DBG] pgmap v1036: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 73/600 objects degraded (12.167%) 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:45.815357+0000 mgr.y (mgr.24401) 631 : cluster [DBG] pgmap v1036: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 664 MiB used, 715 GiB / 715 GiB avail; 73/600 objects degraded (12.167%) 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.204555+0000 osd.3 (osd.3) 391 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.204555+0000 osd.3 (osd.3) 391 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.206492+0000 osd.3 (osd.3) 392 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.206492+0000 osd.3 (osd.3) 392 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.442615+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:33:47.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.442615+0000 osd.1 (osd.1) 359 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.444920+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:46.444920+0000 osd.1 (osd.1) 360 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: audit 2024-04-26T16:33:47.324199+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: audit 2024-04-26T16:33:47.324199+0000 mon.a (mon.0) 3339 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.333311+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.333311+0000 mon.a (mon.0) 3340 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: audit 2024-04-26T16:33:47.404624+0000 mon.c (mon.2) 720 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: audit 2024-04-26T16:33:47.404624+0000 mon.c (mon.2) 720 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: audit 2024-04-26T16:33:47.407417+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.743 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:47 smithi050 bash[29234]: audit 2024-04-26T16:33:47.407417+0000 mon.a (mon.0) 3341 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:47.756 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:47.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:47.887 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:47.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:47.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:48.016 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:48.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:48.145 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:48.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:48.278 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:48.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:48.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:48.512 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:48.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:47.142737+0000 mgr.y (mgr.24401) 632 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:48.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:47.142737+0000 mgr.y (mgr.24401) 632 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:48.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.199675+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:33:48.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.199675+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:33:48.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.201159+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:33:48.524 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.201159+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:48.326695+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:48.326695+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:48.332869+0000 mon.c (mon.2) 721 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:48.332869+0000 mon.c (mon.2) 721 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.334865+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.334865+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:48.335837+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[21463]: audit 2024-04-26T16:33:48.335837+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:47.142737+0000 mgr.y (mgr.24401) 632 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:47.142737+0000 mgr.y (mgr.24401) 632 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:48.525 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.199675+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.199675+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.201159+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.201159+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:48.326695+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:48.326695+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:48.332869+0000 mon.c (mon.2) 721 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:48.332869+0000 mon.c (mon.2) 721 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.334865+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.334865+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:48.335837+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.526 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:48 smithi050 bash[29234]: audit 2024-04-26T16:33:48.335837+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.539 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:48.642 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:48.670 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:48.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:48.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:47.142737+0000 mgr.y (mgr.24401) 632 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:47.142737+0000 mgr.y (mgr.24401) 632 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.199675+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.199675+0000 osd.3 (osd.3) 393 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.201159+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.201159+0000 osd.3 (osd.3) 394 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:48.326695+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:48.326695+0000 mon.a (mon.0) 3342 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:48.332869+0000 mon.c (mon.2) 721 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:48.332869+0000 mon.c (mon.2) 721 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.787 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.334865+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2024-04-26T16:33:48.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.334865+0000 mon.a (mon.0) 3343 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2024-04-26T16:33:48.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:48.335837+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.788 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:48 smithi190 bash[23868]: audit 2024-04-26T16:33:48.335837+0000 mon.a (mon.0) 3344 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:48.800 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:48.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:48.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:48.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:49.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:49.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:49.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:49.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:49.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:49.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:49.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:49.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:49.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:49.595 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:49.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.815977+0000 mgr.y (mgr.24401) 633 : cluster [DBG] pgmap v1039: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:47.815977+0000 mgr.y (mgr.24401) 633 : cluster [DBG] pgmap v1039: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.159519+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.159519+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.162219+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.162219+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:33:49.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.188693+0000 osd.5 (osd.5) 245 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.188693+0000 osd.5 (osd.5) 245 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.192971+0000 osd.5 (osd.5) 246 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.192971+0000 osd.5 (osd.5) 246 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.423488+0000 osd.1 (osd.1) 361 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.423488+0000 osd.1 (osd.1) 361 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.425912+0000 osd.1 (osd.1) 362 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.425912+0000 osd.1 (osd.1) 362 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.751543+0000 osd.2 (osd.2) 335 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.751543+0000 osd.2 (osd.2) 335 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.754480+0000 osd.2 (osd.2) 336 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:48.754480+0000 osd.2 (osd.2) 336 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: audit 2024-04-26T16:33:49.328996+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: audit 2024-04-26T16:33:49.328996+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.334184+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2024-04-26T16:33:49.711 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.334184+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2024-04-26T16:33:49.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: audit 2024-04-26T16:33:49.335477+0000 mon.c (mon.2) 722 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: audit 2024-04-26T16:33:49.335477+0000 mon.c (mon.2) 722 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: audit 2024-04-26T16:33:49.336006+0000 mon.a (mon.0) 3347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.712 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:49 smithi190 bash[23868]: audit 2024-04-26T16:33:49.336006+0000 mon.a (mon.0) 3347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.725 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:49.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.815977+0000 mgr.y (mgr.24401) 633 : cluster [DBG] pgmap v1039: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:47.815977+0000 mgr.y (mgr.24401) 633 : cluster [DBG] pgmap v1039: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.159519+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.159519+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.162219+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.162219+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.188693+0000 osd.5 (osd.5) 245 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.188693+0000 osd.5 (osd.5) 245 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.192971+0000 osd.5 (osd.5) 246 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:33:49.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.192971+0000 osd.5 (osd.5) 246 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.423488+0000 osd.1 (osd.1) 361 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.423488+0000 osd.1 (osd.1) 361 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.425912+0000 osd.1 (osd.1) 362 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.425912+0000 osd.1 (osd.1) 362 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.751543+0000 osd.2 (osd.2) 335 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.751543+0000 osd.2 (osd.2) 335 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.754480+0000 osd.2 (osd.2) 336 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:48.754480+0000 osd.2 (osd.2) 336 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: audit 2024-04-26T16:33:49.328996+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: audit 2024-04-26T16:33:49.328996+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.334184+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.334184+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: audit 2024-04-26T16:33:49.335477+0000 mon.c (mon.2) 722 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: audit 2024-04-26T16:33:49.335477+0000 mon.c (mon.2) 722 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: audit 2024-04-26T16:33:49.336006+0000 mon.a (mon.0) 3347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.842 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[29234]: audit 2024-04-26T16:33:49.336006+0000 mon.a (mon.0) 3347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.815977+0000 mgr.y (mgr.24401) 633 : cluster [DBG] pgmap v1039: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:49.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:47.815977+0000 mgr.y (mgr.24401) 633 : cluster [DBG] pgmap v1039: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:49.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.159519+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:33:49.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.159519+0000 osd.3 (osd.3) 395 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.162219+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.162219+0000 osd.3 (osd.3) 396 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.188693+0000 osd.5 (osd.5) 245 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.188693+0000 osd.5 (osd.5) 245 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.192971+0000 osd.5 (osd.5) 246 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.192971+0000 osd.5 (osd.5) 246 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.423488+0000 osd.1 (osd.1) 361 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.423488+0000 osd.1 (osd.1) 361 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.425912+0000 osd.1 (osd.1) 362 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.425912+0000 osd.1 (osd.1) 362 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.751543+0000 osd.2 (osd.2) 335 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.751543+0000 osd.2 (osd.2) 335 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:33:49.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.754480+0000 osd.2 (osd.2) 336 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:48.754480+0000 osd.2 (osd.2) 336 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: audit 2024-04-26T16:33:49.328996+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: audit 2024-04-26T16:33:49.328996+0000 mon.a (mon.0) 3345 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.334184+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.334184+0000 mon.a (mon.0) 3346 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: audit 2024-04-26T16:33:49.335477+0000 mon.c (mon.2) 722 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: audit 2024-04-26T16:33:49.335477+0000 mon.c (mon.2) 722 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: audit 2024-04-26T16:33:49.336006+0000 mon.a (mon.0) 3347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.844 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:49 smithi050 bash[21463]: audit 2024-04-26T16:33:49.336006+0000 mon.a (mon.0) 3347 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]: dispatch 2024-04-26T16:33:49.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:49.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:49.981 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:49.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:50.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:50.112 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:50.215 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:50.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:50.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:50.346 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:50.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:50.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:50.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:50.608 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:50.637 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:50.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:50.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.414499+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.414499+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.416433+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.416433+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.748817+0000 osd.2 (osd.2) 337 : cluster [DBG] 6.f scrub starts 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.748817+0000 osd.2 (osd.2) 337 : cluster [DBG] 6.f scrub starts 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.755851+0000 osd.2 (osd.2) 338 : cluster [DBG] 6.f scrub ok 2024-04-26T16:33:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.755851+0000 osd.2 (osd.2) 338 : cluster [DBG] 6.f scrub ok 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.231613+0000 osd.4 (osd.4) 339 : cluster [DBG] 2.f scrub starts 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.231613+0000 osd.4 (osd.4) 339 : cluster [DBG] 2.f scrub starts 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.235065+0000 osd.4 (osd.4) 340 : cluster [DBG] 2.f scrub ok 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.235065+0000 osd.4 (osd.4) 340 : cluster [DBG] 2.f scrub ok 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.329493+0000 mon.a (mon.0) 3348 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.329493+0000 mon.a (mon.0) 3348 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: audit 2024-04-26T16:33:50.331801+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]': finished 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: audit 2024-04-26T16:33:50.331801+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]': finished 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.339436+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.339436+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: audit 2024-04-26T16:33:50.415263+0000 mon.c (mon.2) 723 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: audit 2024-04-26T16:33:50.415263+0000 mon.c (mon.2) 723 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: audit 2024-04-26T16:33:50.415798+0000 mon.a (mon.0) 3351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:50 smithi190 bash[23868]: audit 2024-04-26T16:33:50.415798+0000 mon.a (mon.0) 3351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.414499+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.414499+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.416433+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.416433+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.748817+0000 osd.2 (osd.2) 337 : cluster [DBG] 6.f scrub starts 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.748817+0000 osd.2 (osd.2) 337 : cluster [DBG] 6.f scrub starts 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.755851+0000 osd.2 (osd.2) 338 : cluster [DBG] 6.f scrub ok 2024-04-26T16:33:50.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.755851+0000 osd.2 (osd.2) 338 : cluster [DBG] 6.f scrub ok 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.231613+0000 osd.4 (osd.4) 339 : cluster [DBG] 2.f scrub starts 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.231613+0000 osd.4 (osd.4) 339 : cluster [DBG] 2.f scrub starts 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.235065+0000 osd.4 (osd.4) 340 : cluster [DBG] 2.f scrub ok 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.235065+0000 osd.4 (osd.4) 340 : cluster [DBG] 2.f scrub ok 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.329493+0000 mon.a (mon.0) 3348 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.329493+0000 mon.a (mon.0) 3348 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: audit 2024-04-26T16:33:50.331801+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]': finished 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: audit 2024-04-26T16:33:50.331801+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]': finished 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.339436+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.339436+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: audit 2024-04-26T16:33:50.415263+0000 mon.c (mon.2) 723 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: audit 2024-04-26T16:33:50.415263+0000 mon.c (mon.2) 723 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: audit 2024-04-26T16:33:50.415798+0000 mon.a (mon.0) 3351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[29234]: audit 2024-04-26T16:33:50.415798+0000 mon.a (mon.0) 3351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.414499+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.414499+0000 osd.1 (osd.1) 363 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.416433+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.416433+0000 osd.1 (osd.1) 364 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.748817+0000 osd.2 (osd.2) 337 : cluster [DBG] 6.f scrub starts 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.748817+0000 osd.2 (osd.2) 337 : cluster [DBG] 6.f scrub starts 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.755851+0000 osd.2 (osd.2) 338 : cluster [DBG] 6.f scrub ok 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.755851+0000 osd.2 (osd.2) 338 : cluster [DBG] 6.f scrub ok 2024-04-26T16:33:50.834 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.231613+0000 osd.4 (osd.4) 339 : cluster [DBG] 2.f scrub starts 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.231613+0000 osd.4 (osd.4) 339 : cluster [DBG] 2.f scrub starts 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.235065+0000 osd.4 (osd.4) 340 : cluster [DBG] 2.f scrub ok 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.235065+0000 osd.4 (osd.4) 340 : cluster [DBG] 2.f scrub ok 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.329493+0000 mon.a (mon.0) 3348 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.329493+0000 mon.a (mon.0) 3348 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: audit 2024-04-26T16:33:50.331801+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]': finished 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: audit 2024-04-26T16:33:50.331801+0000 mon.a (mon.0) 3349 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-114", "mode": "writeback"}]': finished 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.339436+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.339436+0000 mon.a (mon.0) 3350 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: audit 2024-04-26T16:33:50.415263+0000 mon.c (mon.2) 723 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: audit 2024-04-26T16:33:50.415263+0000 mon.c (mon.2) 723 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: audit 2024-04-26T16:33:50.415798+0000 mon.a (mon.0) 3351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:50 smithi050 bash[21463]: audit 2024-04-26T16:33:50.415798+0000 mon.a (mon.0) 3351 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:33:50.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:50.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:50.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:51.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:51.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:51.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:51.163 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:51.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:51.295 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:51.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:51.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:51.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:51.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:51.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:51.679 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:51.782 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:51.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.816686+0000 mgr.y (mgr.24401) 634 : cluster [DBG] pgmap v1042: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:51.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:49.816686+0000 mgr.y (mgr.24401) 634 : cluster [DBG] pgmap v1042: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:51.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.386017+0000 osd.1 (osd.1) 365 : cluster [DBG] 11.c scrub starts 2024-04-26T16:33:51.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.386017+0000 osd.1 (osd.1) 365 : cluster [DBG] 11.c scrub starts 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.387867+0000 osd.1 (osd.1) 366 : cluster [DBG] 11.c scrub ok 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.387867+0000 osd.1 (osd.1) 366 : cluster [DBG] 11.c scrub ok 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.499293+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.499293+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.501571+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:50.501571+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.203352+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.203352+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.205683+0000 osd.4 (osd.4) 342 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.205683+0000 osd.4 (osd.4) 342 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.308016+0000 mon.a (mon.0) 3352 : cluster [WRN] Health check update: Degraded data redundancy: 73/600 objects degraded (12.167%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.308016+0000 mon.a (mon.0) 3352 : cluster [WRN] Health check update: Degraded data redundancy: 73/600 objects degraded (12.167%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:51.796 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: audit 2024-04-26T16:33:51.310429+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: audit 2024-04-26T16:33:51.310429+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.316485+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.316485+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: audit 2024-04-26T16:33:51.319810+0000 mon.c (mon.2) 724 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: audit 2024-04-26T16:33:51.319810+0000 mon.c (mon.2) 724 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: audit 2024-04-26T16:33:51.324835+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[21463]: audit 2024-04-26T16:33:51.324835+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.816686+0000 mgr.y (mgr.24401) 634 : cluster [DBG] pgmap v1042: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:49.816686+0000 mgr.y (mgr.24401) 634 : cluster [DBG] pgmap v1042: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.386017+0000 osd.1 (osd.1) 365 : cluster [DBG] 11.c scrub starts 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.386017+0000 osd.1 (osd.1) 365 : cluster [DBG] 11.c scrub starts 2024-04-26T16:33:51.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.387867+0000 osd.1 (osd.1) 366 : cluster [DBG] 11.c scrub ok 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.387867+0000 osd.1 (osd.1) 366 : cluster [DBG] 11.c scrub ok 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.499293+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.499293+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.501571+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:50.501571+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.203352+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.203352+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.205683+0000 osd.4 (osd.4) 342 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.205683+0000 osd.4 (osd.4) 342 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.308016+0000 mon.a (mon.0) 3352 : cluster [WRN] Health check update: Degraded data redundancy: 73/600 objects degraded (12.167%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.308016+0000 mon.a (mon.0) 3352 : cluster [WRN] Health check update: Degraded data redundancy: 73/600 objects degraded (12.167%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: audit 2024-04-26T16:33:51.310429+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:51.798 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: audit 2024-04-26T16:33:51.310429+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:51.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.316485+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2024-04-26T16:33:51.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.316485+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2024-04-26T16:33:51.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: audit 2024-04-26T16:33:51.319810+0000 mon.c (mon.2) 724 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: audit 2024-04-26T16:33:51.319810+0000 mon.c (mon.2) 724 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: audit 2024-04-26T16:33:51.324835+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:51 smithi050 bash[29234]: audit 2024-04-26T16:33:51.324835+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.816686+0000 mgr.y (mgr.24401) 634 : cluster [DBG] pgmap v1042: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:49.816686+0000 mgr.y (mgr.24401) 634 : cluster [DBG] pgmap v1042: 204 pgs: 32 creating+peering, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 73/600 objects degraded (12.167%) 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.386017+0000 osd.1 (osd.1) 365 : cluster [DBG] 11.c scrub starts 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.386017+0000 osd.1 (osd.1) 365 : cluster [DBG] 11.c scrub starts 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.387867+0000 osd.1 (osd.1) 366 : cluster [DBG] 11.c scrub ok 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.387867+0000 osd.1 (osd.1) 366 : cluster [DBG] 11.c scrub ok 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.499293+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.499293+0000 osd.7 (osd.7) 359 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:33:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.501571+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:50.501571+0000 osd.7 (osd.7) 360 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.203352+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.203352+0000 osd.4 (osd.4) 341 : cluster [DBG] 5.1 deep-scrub starts 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.205683+0000 osd.4 (osd.4) 342 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.205683+0000 osd.4 (osd.4) 342 : cluster [DBG] 5.1 deep-scrub ok 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.308016+0000 mon.a (mon.0) 3352 : cluster [WRN] Health check update: Degraded data redundancy: 73/600 objects degraded (12.167%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.308016+0000 mon.a (mon.0) 3352 : cluster [WRN] Health check update: Degraded data redundancy: 73/600 objects degraded (12.167%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: audit 2024-04-26T16:33:51.310429+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: audit 2024-04-26T16:33:51.310429+0000 mon.a (mon.0) 3353 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.316485+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.316485+0000 mon.a (mon.0) 3354 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: audit 2024-04-26T16:33:51.319810+0000 mon.c (mon.2) 724 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: audit 2024-04-26T16:33:51.319810+0000 mon.c (mon.2) 724 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: audit 2024-04-26T16:33:51.324835+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:51 smithi190 bash[23868]: audit 2024-04-26T16:33:51.324835+0000 mon.a (mon.0) 3355 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]: dispatch 2024-04-26T16:33:51.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:51.941 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:51.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:52.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:52.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:52.176 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:52.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:52.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:52.334 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:52.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:52.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:52.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:52.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:52.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:52 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.320935+0000 mon.a (mon.0) 3356 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:52.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:52 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.320935+0000 mon.a (mon.0) 3356 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:52.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:52 smithi190 bash[23868]: audit 2024-04-26T16:33:52.323302+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:52.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:52 smithi190 bash[23868]: audit 2024-04-26T16:33:52.323302+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:52.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:52 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.331186+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2024-04-26T16:33:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:52 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.331186+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2024-04-26T16:33:52.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:52.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:52.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:52.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.320935+0000 mon.a (mon.0) 3356 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:52.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.320935+0000 mon.a (mon.0) 3356 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:52.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[21463]: audit 2024-04-26T16:33:52.323302+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:52.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[21463]: audit 2024-04-26T16:33:52.323302+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:52.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.331186+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2024-04-26T16:33:52.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.331186+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2024-04-26T16:33:52.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.320935+0000 mon.a (mon.0) 3356 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:52.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.320935+0000 mon.a (mon.0) 3356 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:33:52.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[29234]: audit 2024-04-26T16:33:52.323302+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:52.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[29234]: audit 2024-04-26T16:33:52.323302+0000 mon.a (mon.0) 3357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-114"}]': finished 2024-04-26T16:33:52.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.331186+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2024-04-26T16:33:52.769 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:52 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.331186+0000 mon.a (mon.0) 3358 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2024-04-26T16:33:52.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:52.854 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:52.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:52.984 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:52.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:53.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:53.113 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:53.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:53.245 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:53.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:53.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:53.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- 172.21.15.50:0/2106624123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84105e60 msgr2=0x7f0d841062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- 172.21.15.50:0/2106624123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84105e60 0x7f0d841062f0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c00b0a0 tx=0x7f0d7c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- 172.21.15.50:0/2106624123 shutdown_connections 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- 172.21.15.50:0/2106624123 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84105e60 0x7f0d841062f0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- 172.21.15.50:0/2106624123 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d84108350 0x7f0d8406b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- 172.21.15.50:0/2106624123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d8406b670 0x7f0d8406bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- 172.21.15.50:0/2106624123 >> 172.21.15.50:0/2106624123 conn(0x7f0d840792d0 msgr2=0x7f0d840796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:53.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- 172.21.15.50:0/2106624123 shutdown_connections 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- 172.21.15.50:0/2106624123 wait complete. 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 librados: starting msgr at 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 librados: starting objecter 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 Processor -- start 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- start start 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 librados: setting wanted keys 2024-04-26T16:33:53.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 librados: calling monclient init 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8406b670 0x7f0d8419f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d84105e60 0x7f0d8419fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84108350 0x7f0d841a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84119130 con 0x7f0d8406b670 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0d84118fb0 con 0x7f0d84105e60 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d89616700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0d841192b0 con 0x7f0d84108350 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.438+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8406b670 0x7f0d8419f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8406b670 0x7f0d8419f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45316/0 (socket says 172.21.15.50:45316) 2024-04-26T16:33:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 -- 172.21.15.50:0/3903701051 learned_addr learned my addr 172.21.15.50:0/3903701051 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84108350 0x7f0d841a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d84105e60 0x7f0d8419fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 -- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84108350 msgr2=0x7f0d841a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d84108350 0x7f0d841a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:53.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 -- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d84105e60 msgr2=0x7f0d8419fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:53.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d84105e60 0x7f0d8419fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:53.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d841a4710 con 0x7f0d8406b670 2024-04-26T16:33:53.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d8406b670 0x7f0d8419f700 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f0d7400eb30 tx=0x7f0d7400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 10 client.?.objecter ms_handle_connect 0x7f0d8406b670 2024-04-26T16:33:53.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:53.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7400cbd0 con 0x7f0d8406b670 2024-04-26T16:33:53.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d841a4a00 con 0x7f0d8406b670 2024-04-26T16:33:53.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0d74004480 con 0x7f0d8406b670 2024-04-26T16:33:53.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74003c30 con 0x7f0d8406b670 2024-04-26T16:33:53.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d89616700 10 client.30903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d841ac280 con 0x7f0d8406b670 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d89616700 1 librados: init done 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0d8406a110 con 0x7f0d8406b670 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0d74003dd0 con 0x7f0d8406b670 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d6c071f80 0x7f0d6c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(681..681 src has 577..681) ==== 9422+0+0 (secure 0 0 0) 0x7f0d740920a0 con 0x7f0d8406b670 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 osd_map(681..681 src has 577..681) 2024-04-26T16:33:53.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 3 client.30903.objecter handle_osd_map got epochs [681,681] > 0 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.442+0000 7f0d6bfff700 3 client.30903.objecter handle_osd_map decoding full epoch 681 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d6c071f80 0x7f0d6c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 20 client.30903.objecter dump_active .. 0 homeless 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v681) ==== 54+0+16 (secure 0 0 0) 0x7f0d740924f0 con 0x7f0d8406b670 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d6c071f80 0x7f0d6c074440 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f0d84105db0 tx=0x7f0d7800b3c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d6c071f80 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 10 client.30903.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 1 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d84064280 0x7f0d84100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 1 2024-04-26T16:33:53.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 0 1 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841a0a60 osd=0 4 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 10 client.30903.objecter _send_command 1 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d841020e0 con 0x7f0d84064280 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d84064280 0x7f0d84100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d84064280 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d84064280 0x7f0d84100a20 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d841020e0 con 0x7f0d84064280 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 1: 0 ) 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 1 = system:0 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841a0a60 osd=0 4 2024-04-26T16:33:53.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.446+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 0 1 2024-04-26T16:33:53.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 10 client.30903.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 2 2024-04-26T16:33:53.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 2 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841a0a60 osd=0 3 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 0 2 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841a0a60 osd=0 4 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 10 client.30903.objecter _send_command 2 2024-04-26T16:33:53.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d84100f60 con 0x7f0d84064280 2024-04-26T16:33:53.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d84100f60 con 0x7f0d84064280 2024-04-26T16:33:53.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841a0a60 osd=0 4 2024-04-26T16:33:53.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 0 2 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 10 client.30903.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 3 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d84063e70 0x7f0d841b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 3 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 1 3 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d84063d00 osd=1 4 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 10 client.30903.objecter _send_command 3 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0d84101e50 con 0x7f0d84063e70 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d84063e70 0x7f0d841b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d84063e70 0x7f0d841b0530 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.466+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d84063e70 2024-04-26T16:33:53.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.470+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d84101e50 con 0x7f0d84063e70 2024-04-26T16:33:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.470+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 3: 0 ) 2024-04-26T16:33:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.470+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 3 = system:0 2024-04-26T16:33:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.470+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d84063d00 osd=1 4 2024-04-26T16:33:53.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.470+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 1 3 2024-04-26T16:33:53.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:53.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 10 client.30903.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 4 2024-04-26T16:33:53.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 4 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d84063d00 osd=1 3 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 1 4 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d84063d00 osd=1 4 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 10 client.30903.objecter _send_command 4 2024-04-26T16:33:53.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.490+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d841013c0 con 0x7f0d84063e70 2024-04-26T16:33:53.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d841013c0 con 0x7f0d84063e70 2024-04-26T16:33:53.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d84063d00 osd=1 4 2024-04-26T16:33:53.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 1 4 2024-04-26T16:33:53.491 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:33:53.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 10 client.30903.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 5 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d8402dff0 0x7f0d841b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 5 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 2 5 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841017c0 osd=2 4 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 10 client.30903.objecter _send_command 5 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0d841b6a60 con 0x7f0d8402dff0 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d8402dff0 0x7f0d841b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d8402dff0 0x7f0d841b3c00 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d8402dff0 2024-04-26T16:33:53.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d841b6a60 con 0x7f0d8402dff0 2024-04-26T16:33:53.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 5: 0 ) 2024-04-26T16:33:53.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 5 = system:0 2024-04-26T16:33:53.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841017c0 osd=2 4 2024-04-26T16:33:53.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.494+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 2 5 2024-04-26T16:33:53.495 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:53.501 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.818191+0000 mgr.y (mgr.24401) 635 : cluster [DBG] pgmap v1045: 204 pgs: 52 active+undersized, 18 active+undersized+degraded, 134 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:51.818191+0000 mgr.y (mgr.24401) 635 : cluster [DBG] pgmap v1045: 204 pgs: 52 active+undersized, 18 active+undersized+degraded, 134 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.507465+0000 osd.7 (osd.7) 361 : cluster [DBG] 11.f scrub starts 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.507465+0000 osd.7 (osd.7) 361 : cluster [DBG] 11.f scrub starts 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.513921+0000 osd.7 (osd.7) 362 : cluster [DBG] 11.f scrub ok 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:52.513921+0000 osd.7 (osd.7) 362 : cluster [DBG] 11.f scrub ok 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.329866+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.329866+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: audit 2024-04-26T16:33:53.445473+0000 mon.a (mon.0) 3360 : audit [DBG] from='client.? 172.21.15.50:0/3903701051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[21463]: audit 2024-04-26T16:33:53.445473+0000 mon.a (mon.0) 3360 : audit [DBG] from='client.? 172.21.15.50:0/3903701051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.818191+0000 mgr.y (mgr.24401) 635 : cluster [DBG] pgmap v1045: 204 pgs: 52 active+undersized, 18 active+undersized+degraded, 134 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:53.502 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:51.818191+0000 mgr.y (mgr.24401) 635 : cluster [DBG] pgmap v1045: 204 pgs: 52 active+undersized, 18 active+undersized+degraded, 134 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.507465+0000 osd.7 (osd.7) 361 : cluster [DBG] 11.f scrub starts 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.507465+0000 osd.7 (osd.7) 361 : cluster [DBG] 11.f scrub starts 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.513921+0000 osd.7 (osd.7) 362 : cluster [DBG] 11.f scrub ok 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:52.513921+0000 osd.7 (osd.7) 362 : cluster [DBG] 11.f scrub ok 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.329866+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.329866+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: audit 2024-04-26T16:33:53.445473+0000 mon.a (mon.0) 3360 : audit [DBG] from='client.? 172.21.15.50:0/3903701051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:53.503 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:53 smithi050 bash[29234]: audit 2024-04-26T16:33:53.445473+0000 mon.a (mon.0) 3360 : audit [DBG] from='client.? 172.21.15.50:0/3903701051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:53.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 10 client.30903.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 6 2024-04-26T16:33:53.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 6 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841017c0 osd=2 3 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 2 6 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841017c0 osd=2 4 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 10 client.30903.objecter _send_command 6 2024-04-26T16:33:53.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.514+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d8402d560 con 0x7f0d8402dff0 2024-04-26T16:33:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d8402d560 con 0x7f0d8402dff0 2024-04-26T16:33:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841017c0 osd=2 4 2024-04-26T16:33:53.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 2 6 2024-04-26T16:33:53.514 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:33:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 10 client.30903.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 7 2024-04-26T16:33:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d841b52c0 0x7f0d841b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 7 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 3 7 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1240 osd=3 4 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 10 client.30903.objecter _send_command 7 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0d841b18c0 con 0x7f0d841b52c0 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d841b52c0 0x7f0d841b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d841b52c0 2024-04-26T16:33:53.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d841b52c0 0x7f0d841b56e0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d841b18c0 con 0x7f0d841b52c0 2024-04-26T16:33:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 7: 0 ) 2024-04-26T16:33:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 7 = system:0 2024-04-26T16:33:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841b1240 osd=3 4 2024-04-26T16:33:53.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.518+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 3 7 2024-04-26T16:33:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 10 client.30903.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 8 2024-04-26T16:33:53.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 8 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841b1240 osd=3 3 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 3 8 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1240 osd=3 4 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 10 client.30903.objecter _send_command 8 2024-04-26T16:33:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d841b0cc0 con 0x7f0d841b52c0 2024-04-26T16:33:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d841b0cc0 con 0x7f0d841b52c0 2024-04-26T16:33:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841b1240 osd=3 4 2024-04-26T16:33:53.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 3 8 2024-04-26T16:33:53.535 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 10 client.30903.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 9 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d841b63f0 0x7f0d841bac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 9 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 4 9 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1a70 osd=4 4 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 10 client.30903.objecter _send_command 9 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0d841bda90 con 0x7f0d841b63f0 2024-04-26T16:33:53.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d841b63f0 0x7f0d841bac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d841b63f0 2024-04-26T16:33:53.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.538+0000 7f0d82ffd700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d841b63f0 0x7f0d841bac30 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.542+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d841bda90 con 0x7f0d841b63f0 2024-04-26T16:33:53.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.542+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 9: 0 ) 2024-04-26T16:33:53.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.542+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 9 = system:0 2024-04-26T16:33:53.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.542+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841b1a70 osd=4 4 2024-04-26T16:33:53.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.542+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 4 9 2024-04-26T16:33:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 10 client.30903.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 10 2024-04-26T16:33:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 10 2024-04-26T16:33:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841b1a70 osd=4 3 2024-04-26T16:33:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 4 10 2024-04-26T16:33:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b1a70 osd=4 4 2024-04-26T16:33:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 10 client.30903.objecter _send_command 10 2024-04-26T16:33:53.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.562+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d841b0ad0 con 0x7f0d841b63f0 2024-04-26T16:33:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d841b0ad0 con 0x7f0d841b63f0 2024-04-26T16:33:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841b1a70 osd=4 4 2024-04-26T16:33:53.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 4 10 2024-04-26T16:33:53.565 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:33:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 10 client.30903.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 11 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d841b4910 0x7f0d841b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 11 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 5 11 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b0ad0 osd=5 4 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 10 client.30903.objecter _send_command 11 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0d841c0e90 con 0x7f0d841b4910 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.566+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d841b4910 0x7f0d841b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d841b4910 2024-04-26T16:33:53.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d827fc700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d841b4910 0x7f0d841b4d70 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d841c0e90 con 0x7f0d841b4910 2024-04-26T16:33:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 11: 0 ) 2024-04-26T16:33:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 11 = system:0 2024-04-26T16:33:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841b0ad0 osd=5 4 2024-04-26T16:33:53.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.570+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 5 11 2024-04-26T16:33:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 10 client.30903.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 12 2024-04-26T16:33:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 12 2024-04-26T16:33:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841b0ad0 osd=5 3 2024-04-26T16:33:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 5 12 2024-04-26T16:33:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841b0ad0 osd=5 4 2024-04-26T16:33:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 10 client.30903.objecter _send_command 12 2024-04-26T16:33:53.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.586+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d841c0e90 con 0x7f0d841b4910 2024-04-26T16:33:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d841c0e90 con 0x7f0d841b4910 2024-04-26T16:33:53.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841b0ad0 osd=5 4 2024-04-26T16:33:53.589 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 5 12 2024-04-26T16:33:53.589 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 10 client.30903.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 13 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 20 client.30903.objecter _get_session osd=-1 returning homeless 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 10 client.30903.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=682}) -- 0x7f0d841bc9c0 con 0x7f0d8406b670 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0d841c0e90 con 0x7f0d8406b670 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_get_version_reply(handle=1 version=681) ==== 24+0+0 (secure 0 0 0) 0x7f0d74065f10 con 0x7f0d8406b670 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d817fa700 10 client.30903.objecter _check_command_map_dne tid 13 current 681 map_dne_bound 681 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d817fa700 10 client.30903.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d817fa700 15 client.30903.objecter _session_command_op_remove -1 13 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 10 client.30903.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 14 2024-04-26T16:33:53.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d841bb1d0 0x7f0d841bb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 14 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 7 14 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841bd510 osd=7 4 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 10 client.30903.objecter _send_command 14 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f0d841bd8d0 con 0x7f0d841bb1d0 2024-04-26T16:33:53.591 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d841bb1d0 0x7f0d841bb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 10 client.30903.objecter ms_handle_connect 0x7f0d841bb1d0 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d837fe700 1 --2- 172.21.15.50:0/3903701051 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d841bb1d0 0x7f0d841bb5f0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d7c004cd0 con 0x7f0d841bb1d0 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 14: 0 ) 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 14 = system:0 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841bd510 osd=7 4 2024-04-26T16:33:53.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.594+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 7 14 2024-04-26T16:33:53.599 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 10 client.30903.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:33:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign -1 15 2024-04-26T16:33:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 20 client.30903.objecter _get_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 15 client.30903.objecter _session_command_op_remove -1 15 2024-04-26T16:33:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 20 client.30903.objecter get_session s=0x7f0d841bd510 osd=7 3 2024-04-26T16:33:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 15 client.30903.objecter _session_command_op_assign 7 15 2024-04-26T16:33:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 20 client.30903.objecter put_session s=0x7f0d841bd510 osd=7 4 2024-04-26T16:33:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 10 client.30903.objecter _send_command 15 2024-04-26T16:33:53.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.614+0000 7f0d89616700 1 -- 172.21.15.50:0/3903701051 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f0d841bbb90 con 0x7f0d841bb1d0 2024-04-26T16:33:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.618+0000 7f0d6bfff700 1 -- 172.21.15.50:0/3903701051 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d841bbb90 con 0x7f0d841bb1d0 2024-04-26T16:33:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.618+0000 7f0d6bfff700 10 client.30903.objecter ms_dispatch 0x7f0d84001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:33:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.618+0000 7f0d6bfff700 10 client.30903.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:33:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.618+0000 7f0d6bfff700 20 client.30903.objecter put_session s=0x7f0d841bd510 osd=7 4 2024-04-26T16:33:53.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:53.618+0000 7f0d6bfff700 15 client.30903.objecter _session_command_op_remove 7 15 2024-04-26T16:33:53.615 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:33:53.615 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:33:53.615 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:33:53.615 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:33:53.615 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:33:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:33:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:33:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:33:53.616 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:33:53.618 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:53.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:53.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.818191+0000 mgr.y (mgr.24401) 635 : cluster [DBG] pgmap v1045: 204 pgs: 52 active+undersized, 18 active+undersized+degraded, 134 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:51.818191+0000 mgr.y (mgr.24401) 635 : cluster [DBG] pgmap v1045: 204 pgs: 52 active+undersized, 18 active+undersized+degraded, 134 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.507465+0000 osd.7 (osd.7) 361 : cluster [DBG] 11.f scrub starts 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.507465+0000 osd.7 (osd.7) 361 : cluster [DBG] 11.f scrub starts 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.513921+0000 osd.7 (osd.7) 362 : cluster [DBG] 11.f scrub ok 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:52.513921+0000 osd.7 (osd.7) 362 : cluster [DBG] 11.f scrub ok 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.329866+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.329866+0000 mon.a (mon.0) 3359 : cluster [DBG] osdmap e681: 8 total, 7 up, 8 in 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: audit 2024-04-26T16:33:53.445473+0000 mon.a (mon.0) 3360 : audit [DBG] from='client.? 172.21.15.50:0/3903701051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:53 smithi190 bash[23868]: audit 2024-04-26T16:33:53.445473+0000 mon.a (mon.0) 3360 : audit [DBG] from='client.? 172.21.15.50:0/3903701051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:33:53.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:53.875 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:53.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:53.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:54.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:54.150 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:54.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:54.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:54.413 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:33:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:33:54.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:54.435 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:54.537 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:54.580 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:33:54.580 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:33:54.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:54.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.174104+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:33:54.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.174104+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:33:54.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.175500+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:33:54.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.175500+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:33:54.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.792691+0000 osd.0 (osd.0) 387 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:33:54.674 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.792691+0000 osd.0 (osd.0) 387 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.794371+0000 osd.0 (osd.0) 388 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.794371+0000 osd.0 (osd.0) 388 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.271265+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.271265+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.273508+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.273508+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.332263+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.332263+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: audit 2024-04-26T16:33:54.336785+0000 mon.c (mon.2) 725 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: audit 2024-04-26T16:33:54.336785+0000 mon.c (mon.2) 725 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: audit 2024-04-26T16:33:54.337302+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[29234]: audit 2024-04-26T16:33:54.337302+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.174104+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.174104+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.175500+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.175500+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.792691+0000 osd.0 (osd.0) 387 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.792691+0000 osd.0 (osd.0) 387 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.794371+0000 osd.0 (osd.0) 388 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.794371+0000 osd.0 (osd.0) 388 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.271265+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.271265+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.273508+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:33:54.676 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.273508+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:33:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.332263+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2024-04-26T16:33:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.332263+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2024-04-26T16:33:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: audit 2024-04-26T16:33:54.336785+0000 mon.c (mon.2) 725 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: audit 2024-04-26T16:33:54.336785+0000 mon.c (mon.2) 725 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: audit 2024-04-26T16:33:54.337302+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.677 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:54 smithi050 bash[21463]: audit 2024-04-26T16:33:54.337302+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.685 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:54.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:54.814 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.174104+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.174104+0000 osd.3 (osd.3) 397 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.175500+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.175500+0000 osd.3 (osd.3) 398 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.792691+0000 osd.0 (osd.0) 387 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.792691+0000 osd.0 (osd.0) 387 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.794371+0000 osd.0 (osd.0) 388 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.794371+0000 osd.0 (osd.0) 388 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.271265+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.271265+0000 osd.4 (osd.4) 343 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.273508+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:33:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.273508+0000 osd.4 (osd.4) 344 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:33:54.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.332263+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2024-04-26T16:33:54.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.332263+0000 mon.a (mon.0) 3361 : cluster [DBG] osdmap e682: 8 total, 7 up, 8 in 2024-04-26T16:33:54.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: audit 2024-04-26T16:33:54.336785+0000 mon.c (mon.2) 725 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: audit 2024-04-26T16:33:54.336785+0000 mon.c (mon.2) 725 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: audit 2024-04-26T16:33:54.337302+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:54 smithi190 bash[23868]: audit 2024-04-26T16:33:54.337302+0000 mon.a (mon.0) 3362 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:33:54.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:54.945 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:54.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:55.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:55.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:55.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:55.210 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:55.313 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:55.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:55.446 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:55.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:55.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:55.602 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:55.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:55.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.818860+0000 mgr.y (mgr.24401) 636 : cluster [DBG] pgmap v1048: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:53.818860+0000 mgr.y (mgr.24401) 636 : cluster [DBG] pgmap v1048: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.177099+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.177099+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.179316+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.179316+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.458222+0000 osd.7 (osd.7) 363 : cluster [DBG] 3.e scrub starts 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.458222+0000 osd.7 (osd.7) 363 : cluster [DBG] 3.e scrub starts 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.462275+0000 osd.7 (osd.7) 364 : cluster [DBG] 3.e scrub ok 2024-04-26T16:33:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.462275+0000 osd.7 (osd.7) 364 : cluster [DBG] 3.e scrub ok 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.804713+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.804713+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.807078+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:54.807078+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.048216+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.d scrub starts 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.048216+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.d scrub starts 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.050598+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.d scrub ok 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.050598+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.d scrub ok 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: audit 2024-04-26T16:33:55.331029+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: audit 2024-04-26T16:33:55.331029+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.336425+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2024-04-26T16:33:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:55 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.336425+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2024-04-26T16:33:55.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:55.865 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:55.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:55.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.818860+0000 mgr.y (mgr.24401) 636 : cluster [DBG] pgmap v1048: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:55.981 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:53.818860+0000 mgr.y (mgr.24401) 636 : cluster [DBG] pgmap v1048: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.177099+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.177099+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.179316+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.179316+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.458222+0000 osd.7 (osd.7) 363 : cluster [DBG] 3.e scrub starts 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.458222+0000 osd.7 (osd.7) 363 : cluster [DBG] 3.e scrub starts 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.462275+0000 osd.7 (osd.7) 364 : cluster [DBG] 3.e scrub ok 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.462275+0000 osd.7 (osd.7) 364 : cluster [DBG] 3.e scrub ok 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.804713+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.804713+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.807078+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:33:55.982 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:54.807078+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.048216+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.d scrub starts 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.048216+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.d scrub starts 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.050598+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.d scrub ok 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.050598+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.d scrub ok 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: audit 2024-04-26T16:33:55.331029+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: audit 2024-04-26T16:33:55.331029+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.336425+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2024-04-26T16:33:55.983 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.336425+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.818860+0000 mgr.y (mgr.24401) 636 : cluster [DBG] pgmap v1048: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:53.818860+0000 mgr.y (mgr.24401) 636 : cluster [DBG] pgmap v1048: 172 pgs: 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 73/603 objects degraded (12.106%) 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.177099+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.177099+0000 osd.3 (osd.3) 399 : cluster [DBG] 5.0 deep-scrub starts 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.179316+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.179316+0000 osd.3 (osd.3) 400 : cluster [DBG] 5.0 deep-scrub ok 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.458222+0000 osd.7 (osd.7) 363 : cluster [DBG] 3.e scrub starts 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.458222+0000 osd.7 (osd.7) 363 : cluster [DBG] 3.e scrub starts 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.462275+0000 osd.7 (osd.7) 364 : cluster [DBG] 3.e scrub ok 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.462275+0000 osd.7 (osd.7) 364 : cluster [DBG] 3.e scrub ok 2024-04-26T16:33:55.984 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.804713+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.804713+0000 osd.2 (osd.2) 339 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.807078+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:54.807078+0000 osd.2 (osd.2) 340 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.048216+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.d scrub starts 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.048216+0000 osd.5 (osd.5) 247 : cluster [DBG] 6.d scrub starts 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.050598+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.d scrub ok 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.050598+0000 osd.5 (osd.5) 248 : cluster [DBG] 6.d scrub ok 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: audit 2024-04-26T16:33:55.331029+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: audit 2024-04-26T16:33:55.331029+0000 mon.a (mon.0) 3363 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.336425+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2024-04-26T16:33:55.985 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:55 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.336425+0000 mon.a (mon.0) 3364 : cluster [DBG] osdmap e683: 8 total, 7 up, 8 in 2024-04-26T16:33:55.995 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:55.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:56.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:56.125 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:56.229 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:56.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:56.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:56.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:56.389 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:33:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:33:56.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:56.522 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.482527+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.482527+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.484808+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.484808+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.708021+0000 osd.0 (osd.0) 389 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.708021+0000 osd.0 (osd.0) 389 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:56.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.708308+0000 osd.0 (osd.0) 390 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:56.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.482527+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.482527+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.484808+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.484808+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.708021+0000 osd.0 (osd.0) 389 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.708021+0000 osd.0 (osd.0) 389 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.708308+0000 osd.0 (osd.0) 390 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:56.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.708308+0000 osd.0 (osd.0) 390 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.063904+0000 osd.5 (osd.5) 249 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.063904+0000 osd.5 (osd.5) 249 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.069774+0000 osd.5 (osd.5) 250 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.069774+0000 osd.5 (osd.5) 250 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.309695+0000 mon.a (mon.0) 3365 : cluster [WRN] Health check update: Degraded data redundancy: 73/603 objects degraded (12.106%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.309695+0000 mon.a (mon.0) 3365 : cluster [WRN] Health check update: Degraded data redundancy: 73/603 objects degraded (12.106%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.337026+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.337026+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: audit 2024-04-26T16:33:56.369046+0000 mon.c (mon.2) 726 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: audit 2024-04-26T16:33:56.369046+0000 mon.c (mon.2) 726 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: audit 2024-04-26T16:33:56.369856+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:56 smithi190 bash[23868]: audit 2024-04-26T16:33:56.369856+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.654 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:56.757 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:56.785 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.708308+0000 osd.0 (osd.0) 390 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.063904+0000 osd.5 (osd.5) 249 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.063904+0000 osd.5 (osd.5) 249 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.069774+0000 osd.5 (osd.5) 250 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.069774+0000 osd.5 (osd.5) 250 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.309695+0000 mon.a (mon.0) 3365 : cluster [WRN] Health check update: Degraded data redundancy: 73/603 objects degraded (12.106%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.309695+0000 mon.a (mon.0) 3365 : cluster [WRN] Health check update: Degraded data redundancy: 73/603 objects degraded (12.106%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.337026+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2024-04-26T16:33:56.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.337026+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: audit 2024-04-26T16:33:56.369046+0000 mon.c (mon.2) 726 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: audit 2024-04-26T16:33:56.369046+0000 mon.c (mon.2) 726 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: audit 2024-04-26T16:33:56.369856+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[21463]: audit 2024-04-26T16:33:56.369856+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.482527+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.482527+0000 osd.7 (osd.7) 365 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.484808+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.484808+0000 osd.7 (osd.7) 366 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.708021+0000 osd.0 (osd.0) 389 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.708021+0000 osd.0 (osd.0) 389 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:33:56.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.708308+0000 osd.0 (osd.0) 390 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.708308+0000 osd.0 (osd.0) 390 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.063904+0000 osd.5 (osd.5) 249 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.063904+0000 osd.5 (osd.5) 249 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.069774+0000 osd.5 (osd.5) 250 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.069774+0000 osd.5 (osd.5) 250 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.309695+0000 mon.a (mon.0) 3365 : cluster [WRN] Health check update: Degraded data redundancy: 73/603 objects degraded (12.106%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.309695+0000 mon.a (mon.0) 3365 : cluster [WRN] Health check update: Degraded data redundancy: 73/603 objects degraded (12.106%), 18 pgs degraded (PG_DEGRADED) 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.337026+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.337026+0000 mon.a (mon.0) 3366 : cluster [DBG] osdmap e684: 8 total, 7 up, 8 in 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: audit 2024-04-26T16:33:56.369046+0000 mon.c (mon.2) 726 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: audit 2024-04-26T16:33:56.369046+0000 mon.c (mon.2) 726 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: audit 2024-04-26T16:33:56.369856+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:56 smithi050 bash[29234]: audit 2024-04-26T16:33:56.369856+0000 mon.a (mon.0) 3367 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:33:56.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:56.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:56.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:57.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:57.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:57.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:57.180 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:57.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:57.311 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:57.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:57.414 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:57.442 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:57.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:33:57.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:57.573 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.819636+0000 mgr.y (mgr.24401) 637 : cluster [DBG] pgmap v1051: 204 pgs: 32 unknown, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 73/603 objects degraded (12.106%) 2024-04-26T16:33:57.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:55.819636+0000 mgr.y (mgr.24401) 637 : cluster [DBG] pgmap v1051: 204 pgs: 32 unknown, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 73/603 objects degraded (12.106%) 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.521767+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.521767+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.523807+0000 osd.7 (osd.7) 368 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.523807+0000 osd.7 (osd.7) 368 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.877218+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.877218+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.881723+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:56.881723+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.335719+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.335719+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.339866+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.339866+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2024-04-26T16:33:57.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.341293+0000 mon.c (mon.2) 727 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.341293+0000 mon.c (mon.2) 727 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.342107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.342107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.458003+0000 mon.c (mon.2) 728 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[29234]: audit 2024-04-26T16:33:57.458003+0000 mon.c (mon.2) 728 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.819636+0000 mgr.y (mgr.24401) 637 : cluster [DBG] pgmap v1051: 204 pgs: 32 unknown, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 73/603 objects degraded (12.106%) 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:55.819636+0000 mgr.y (mgr.24401) 637 : cluster [DBG] pgmap v1051: 204 pgs: 32 unknown, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 73/603 objects degraded (12.106%) 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.521767+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.521767+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:33:57.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.523807+0000 osd.7 (osd.7) 368 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.523807+0000 osd.7 (osd.7) 368 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.877218+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.877218+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.881723+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:56.881723+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.335719+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.335719+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.339866+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.339866+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.341293+0000 mon.c (mon.2) 727 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.341293+0000 mon.c (mon.2) 727 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.342107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.342107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.458003+0000 mon.c (mon.2) 728 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:57.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:57 smithi050 bash[21463]: audit 2024-04-26T16:33:57.458003+0000 mon.c (mon.2) 728 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:57.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:57.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:33:57.730 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:33:57.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.819636+0000 mgr.y (mgr.24401) 637 : cluster [DBG] pgmap v1051: 204 pgs: 32 unknown, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 73/603 objects degraded (12.106%) 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:55.819636+0000 mgr.y (mgr.24401) 637 : cluster [DBG] pgmap v1051: 204 pgs: 32 unknown, 41 active+undersized, 18 active+undersized+degraded, 113 active+clean; 583 KiB data, 661 MiB used, 715 GiB / 715 GiB avail; 73/603 objects degraded (12.106%) 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.521767+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.521767+0000 osd.7 (osd.7) 367 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.523807+0000 osd.7 (osd.7) 368 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.523807+0000 osd.7 (osd.7) 368 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.877218+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.877218+0000 osd.0 (osd.0) 391 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.881723+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:56.881723+0000 osd.0 (osd.0) 392 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.335719+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.335719+0000 mon.a (mon.0) 3368 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.339866+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2024-04-26T16:33:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.339866+0000 mon.a (mon.0) 3369 : cluster [DBG] osdmap e685: 8 total, 7 up, 8 in 2024-04-26T16:33:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.341293+0000 mon.c (mon.2) 727 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.341293+0000 mon.c (mon.2) 727 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.342107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.342107+0000 mon.a (mon.0) 3370 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:33:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.458003+0000 mon.c (mon.2) 728 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:57 smithi190 bash[23868]: audit 2024-04-26T16:33:57.458003+0000 mon.c (mon.2) 728 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:33:57.838 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:57.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:57.941 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:57.969 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:57.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:58.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:58.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:58.123 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-04-26T16:33:58.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:58.232 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:58.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:58.335 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:58.365 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:33:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:33:58.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- 172.21.15.50:0/2901679012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406bad0 msgr2=0x7f75941102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- 172.21.15.50:0/2901679012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406bad0 0x7f75941102e0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f758c00b0a0 tx=0x7f758c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- 172.21.15.50:0/2901679012 shutdown_connections 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- 172.21.15.50:0/2901679012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75941094a0 0x7f7594109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- 172.21.15.50:0/2901679012 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406bad0 0x7f75941102e0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- 172.21.15.50:0/2901679012 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406b130 0x7f759406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- 172.21.15.50:0/2901679012 >> 172.21.15.50:0/2901679012 conn(0x7f75940792d0 msgr2=0x7f75940796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:58.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- 172.21.15.50:0/2901679012 shutdown_connections 2024-04-26T16:33:58.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- 172.21.15.50:0/2901679012 wait complete. 2024-04-26T16:33:58.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 librados: starting msgr at 2024-04-26T16:33:58.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 librados: starting objecter 2024-04-26T16:33:58.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 Processor -- start 2024-04-26T16:33:58.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- start start 2024-04-26T16:33:58.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 librados: setting wanted keys 2024-04-26T16:33:58.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 librados: calling monclient init 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 0x7f759419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 0x7f759419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75941094a0 0x7f759419fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7594112ad0 con 0x7f759406b130 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7594112950 con 0x7f75941094a0 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f75989d7700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7594112c50 con 0x7f759406bad0 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f7591d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 0x7f759419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 0x7f759419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 0x7f759419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33276/0 (socket says 172.21.15.50:33276) 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 -- 172.21.15.50:0/2337759430 learned_addr learned my addr 172.21.15.50:0/2337759430 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f7591d9b700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 0x7f759419b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40676/0 (socket says 172.21.15.50:40676) 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f7592d9d700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75941094a0 0x7f759419fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 -- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 msgr2=0x7f759419b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 0x7f759419b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 -- 172.21.15.50:0/2337759430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75941094a0 msgr2=0x7f759419fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75941094a0 0x7f759419fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f759259c700 1 -- 172.21.15.50:0/2337759430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75941a02a0 con 0x7f759406b130 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.490+0000 7f7591d9b700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 0x7f759419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f759259c700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 0x7f759419b290 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f758400d950 tx=0x7f758400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 10 client.?.objecter ms_handle_connect 0x7f759406b130 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584016070 con 0x7f759406b130 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75941a0590 con 0x7f759406b130 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f758400fcf0 con 0x7f759406b130 2024-04-26T16:33:58.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75989d7700 10 client.30909.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:58.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7584010df0 con 0x7f759406b130 2024-04-26T16:33:58.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75941a7e10 con 0x7f759406b130 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75989d7700 1 librados: init done 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7594104b50 con 0x7f759406b130 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7584019070 con 0x7f759406b130 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f757c071e50 0x7f757c074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 577..686) ==== 9803+0+0 (secure 0 0 0) 0x7f75840913a0 con 0x7f759406b130 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 10 client.30909.objecter ms_dispatch 0x7f7594001550 osd_map(686..686 src has 577..686) 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 3 client.30909.objecter handle_osd_map got epochs [686,686] > 0 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 3 client.30909.objecter handle_osd_map decoding full epoch 686 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f7591d9b700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f757c071e50 0x7f757c074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.494+0000 7f75837fe700 20 client.30909.objecter dump_active .. 0 homeless 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.498+0000 7f75837fe700 10 client.30909.objecter ms_handle_connect 0x7f757c071e50 2024-04-26T16:33:58.493 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.498+0000 7f7591d9b700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f757c071e50 0x7f757c074310 secure :-1 s=READY pgs=3812 cs=0 l=1 rev1=1 crypto rx=0x7f7588009680 tx=0x7f7588009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:58.497 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:58.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.502+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f758405cc30 con 0x7f759406b130 2024-04-26T16:33:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:58.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- 172.21.15.50:0/3764395552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50406c550 msgr2=0x7fe504110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- 172.21.15.50:0/3764395552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50406c550 0x7fe504110300 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0098f0 tx=0x7fe4fc009c00 comp rx=0 tx=0).stop 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- 172.21.15.50:0/3764395552 shutdown_connections 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- 172.21.15.50:0/3764395552 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5041094c0 0x7fe5041098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- 172.21.15.50:0/3764395552 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe50406c550 0x7fe504110300 unknown :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- 172.21.15.50:0/3764395552 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406bb90 0x7fe50406c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- 172.21.15.50:0/3764395552 >> 172.21.15.50:0/3764395552 conn(0x7fe504078370 msgr2=0x7fe504078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- 172.21.15.50:0/3764395552 shutdown_connections 2024-04-26T16:33:58.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- 172.21.15.50:0/3764395552 wait complete. 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 librados: starting msgr at 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 librados: starting objecter 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 Processor -- start 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- start start 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 librados: setting wanted keys 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 librados: calling monclient init 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe50406bb90 0x7fe504102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 0x7fe5041029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5041094c0 0x7fe504102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe504112ac0 con 0x7fe5041094c0 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe504112940 con 0x7fe50406bb90 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe50a036700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe504112c40 con 0x7fe50406c550 2024-04-26T16:33:58.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe502ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 0x7fe5041029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe502ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 0x7fe5041029c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40696/0 (socket says 172.21.15.50:40696) 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe503fff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5041094c0 0x7fe504102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.570+0000 7fe502ffd700 1 -- 172.21.15.50:0/3445487460 learned_addr learned my addr 172.21.15.50:0/3445487460 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe5037fe700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe50406bb90 0x7fe504102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe503fff700 1 -- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 msgr2=0x7fe5041029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe503fff700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 0x7fe5041029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe503fff700 1 -- 172.21.15.50:0/3445487460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe50406bb90 msgr2=0x7fe504102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe503fff700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe50406bb90 0x7fe504102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe503fff700 1 -- 172.21.15.50:0/3445487460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5041a75d0 con 0x7fe5041094c0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe502ffd700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 0x7fe5041029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe503fff700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5041094c0 0x7fe504102f00 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009230 tx=0x7fe4fc00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 10 client.?.objecter ms_handle_connect 0x7fe5041094c0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc025070 con 0x7fe5041094c0 2024-04-26T16:33:58.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5041a7860 con 0x7fe5041094c0 2024-04-26T16:33:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe4fc0054a0 con 0x7fe5041094c0 2024-04-26T16:33:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe50a036700 10 client.30915.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:33:58.572 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc003a30 con 0x7fe5041094c0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5041a7d10 con 0x7fe5041094c0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe50a036700 1 librados: init done 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe504104b90 con 0x7fe5041094c0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe4fc028070 con 0x7fe5041094c0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe4f0071e50 0x7fe4f0074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(686..686 src has 577..686) ==== 9803+0+0 (secure 0 0 0) 0x7fe4fc092190 con 0x7fe5041094c0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 10 client.30915.objecter ms_dispatch 0x7fe504001550 osd_map(686..686 src has 577..686) 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 3 client.30915.objecter handle_osd_map got epochs [686,686] > 0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 3 client.30915.objecter handle_osd_map decoding full epoch 686 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe5037fe700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe4f0071e50 0x7fe4f0074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.574+0000 7fe500ff9700 20 client.30915.objecter dump_active .. 0 homeless 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.578+0000 7fe5037fe700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe4f0071e50 0x7fe4f0074310 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400e270 tx=0x7fe4f4009040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:33:58.573 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.578+0000 7fe500ff9700 10 client.30915.objecter ms_handle_connect 0x7fe4f0071e50 2024-04-26T16:33:58.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.582+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fe4fc013030 con 0x7fe5041094c0 2024-04-26T16:33:58.600 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:58.608 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:57.154135+0000 mgr.y (mgr.24401) 638 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:57.154135+0000 mgr.y (mgr.24401) 638 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.481437+0000 osd.1 (osd.1) 367 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.481437+0000 osd.1 (osd.1) 367 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.483206+0000 osd.1 (osd.1) 368 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.483206+0000 osd.1 (osd.1) 368 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.526808+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.526808+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.532045+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.532045+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.842754+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.842754+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.844355+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.844355+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:33:58.609 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:58.338536+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:58.338536+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:58.346496+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: cluster 2024-04-26T16:33:58.346496+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:58.349813+0000 mon.c (mon.2) 729 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:58.349813+0000 mon.c (mon.2) 729 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:58.350820+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[29234]: audit 2024-04-26T16:33:58.350820+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:57.154135+0000 mgr.y (mgr.24401) 638 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:57.154135+0000 mgr.y (mgr.24401) 638 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.481437+0000 osd.1 (osd.1) 367 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:33:58.610 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.481437+0000 osd.1 (osd.1) 367 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.483206+0000 osd.1 (osd.1) 368 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.483206+0000 osd.1 (osd.1) 368 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.526808+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.526808+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.532045+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.532045+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.842754+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.842754+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.844355+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.844355+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:58.338536+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:58.338536+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:58.346496+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2024-04-26T16:33:58.611 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: cluster 2024-04-26T16:33:58.346496+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2024-04-26T16:33:58.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:58.349813+0000 mon.c (mon.2) 729 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:58.349813+0000 mon.c (mon.2) 729 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:58.350820+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.612 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:58 smithi050 bash[21463]: audit 2024-04-26T16:33:58.350820+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.623 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:33:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:33:58.726 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:58.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.734+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f759419c5f0 con 0x7f757c071e50 2024-04-26T16:33:58.748 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:58.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.758+0000 7f75837fe700 1 -- 172.21.15.50:0/2337759430 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+530409 (secure 0 0 0) 0x7f759419c5f0 con 0x7f757c071e50 2024-04-26T16:33:58.759 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:33:58.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 10 librados: watch_flush enter 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 10 librados: watch_flush exit 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 20 client.30909.objecter shutdown clearing up homeless session... 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 10 client.30909.objecter successfully canceled tick 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f757c071e50 msgr2=0x7f757c074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f757c071e50 0x7f757c074310 secure :-1 s=READY pgs=3812 cs=0 l=1 rev1=1 crypto rx=0x7f7588009680 tx=0x7f7588009260 comp rx=0 tx=0).stop 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 msgr2=0x7f759419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 0x7f759419b290 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f758400d950 tx=0x7f758400dc60 comp rx=0 tx=0).stop 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 shutdown_connections 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f757c071e50 0x7f757c074310 unknown :-1 s=CLOSED pgs=3812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f75941094a0 0x7f759419fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f759406b130 0x7f759419b290 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 --2- 172.21.15.50:0/2337759430 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f759406bad0 0x7f759419b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 >> 172.21.15.50:0/2337759430 conn(0x7f75940792d0 msgr2=0x7f759410e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:58.764 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 shutdown_connections 2024-04-26T16:33:58.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 -- 172.21.15.50:0/2337759430 wait complete. 2024-04-26T16:33:58.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.762+0000 7f75989d7700 1 librados: shutdown 2024-04-26T16:33:58.765 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:33:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:57.154135+0000 mgr.y (mgr.24401) 638 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:57.154135+0000 mgr.y (mgr.24401) 638 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:33:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.481437+0000 osd.1 (osd.1) 367 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:33:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.481437+0000 osd.1 (osd.1) 367 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:33:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.483206+0000 osd.1 (osd.1) 368 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:33:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.483206+0000 osd.1 (osd.1) 368 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.526808+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.526808+0000 osd.7 (osd.7) 369 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.532045+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.532045+0000 osd.7 (osd.7) 370 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.842754+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.842754+0000 osd.0 (osd.0) 393 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.844355+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.844355+0000 osd.0 (osd.0) 394 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:58.338536+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:58.338536+0000 mon.a (mon.0) 3371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:58.346496+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: cluster 2024-04-26T16:33:58.346496+0000 mon.a (mon.0) 3372 : cluster [DBG] osdmap e686: 8 total, 7 up, 8 in 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:58.349813+0000 mon.c (mon.2) 729 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:58.349813+0000 mon.c (mon.2) 729 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:58.350820+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:58 smithi190 bash[23868]: audit 2024-04-26T16:33:58.350820+0000 mon.a (mon.0) 3373 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]: dispatch 2024-04-26T16:33:58.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:58.834+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7fe5041039b0 con 0x7fe5041094c0 2024-04-26T16:33:58.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:58.878 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:33:58.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:58.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:33:58.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:59.006 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:59.109 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:59.138 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:59.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:59.270 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:59.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.338+0000 7fe500ff9700 1 -- 172.21.15.50:0/3445487460 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v687) ==== 87+0+0 (secure 0 0 0) 0x7fe4fc05d9a0 con 0x7fe5041094c0 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.342+0000 7fe50a036700 10 librados: watch_flush enter 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.342+0000 7fe50a036700 10 librados: watch_flush exit 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.342+0000 7fe50a036700 20 client.30915.objecter shutdown clearing up homeless session... 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.342+0000 7fe50a036700 10 client.30915.objecter successfully canceled tick 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe4f0071e50 msgr2=0x7fe4f0074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe4f0071e50 0x7fe4f0074310 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7fe4f400e270 tx=0x7fe4f4009040 comp rx=0 tx=0).stop 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5041094c0 msgr2=0x7fe504102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5041094c0 0x7fe504102f00 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc009230 tx=0x7fe4fc00fa00 comp rx=0 tx=0).stop 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 shutdown_connections 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe4f0071e50 0x7fe4f0074310 unknown :-1 s=CLOSED pgs=3813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe50406bb90 0x7fe504102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:59.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5041094c0 0x7fe504102f00 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:59.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 --2- 172.21.15.50:0/3445487460 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe50406c550 0x7fe5041029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:33:59.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 >> 172.21.15.50:0/3445487460 conn(0x7fe504078370 msgr2=0x7fe504104fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:33:59.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 shutdown_connections 2024-04-26T16:33:59.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 -- 172.21.15.50:0/3445487460 wait complete. 2024-04-26T16:33:59.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:33:59.346+0000 7fe50a036700 1 librados: shutdown 2024-04-26T16:33:59.346 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:33:59.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:59.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:59.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:59.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:59.524 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:33:59.628 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.820680+0000 mgr.y (mgr.24401) 639 : cluster [DBG] pgmap v1054: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:33:59.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:57.820680+0000 mgr.y (mgr.24401) 639 : cluster [DBG] pgmap v1054: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:58.736543+0000 mgr.y (mgr.24401) 640 : audit [DBG] from='client.30909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:58.736543+0000 mgr.y (mgr.24401) 640 : audit [DBG] from='client.30909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:58.837706+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:58.837706+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.339197+0000 mon.a (mon.0) 3375 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.339197+0000 mon.a (mon.0) 3375 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.339233+0000 mon.a (mon.0) 3376 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:59.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.339233+0000 mon.a (mon.0) 3376 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:59.341325+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]': finished 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:59.341325+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]': finished 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:59.341488+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:59.341488+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.346812+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.346812+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2024-04-26T16:33:59.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:59.375520+0000 mon.c (mon.2) 730 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[21463]: audit 2024-04-26T16:33:59.375520+0000 mon.c (mon.2) 730 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.820680+0000 mgr.y (mgr.24401) 639 : cluster [DBG] pgmap v1054: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:57.820680+0000 mgr.y (mgr.24401) 639 : cluster [DBG] pgmap v1054: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:58.736543+0000 mgr.y (mgr.24401) 640 : audit [DBG] from='client.30909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:58.736543+0000 mgr.y (mgr.24401) 640 : audit [DBG] from='client.30909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:58.837706+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:33:59.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:58.837706+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.339197+0000 mon.a (mon.0) 3375 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.339197+0000 mon.a (mon.0) 3375 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.339233+0000 mon.a (mon.0) 3376 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.339233+0000 mon.a (mon.0) 3376 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:59.341325+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]': finished 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:59.341325+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]': finished 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:59.341488+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:59.341488+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.346812+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.346812+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:59.375520+0000 mon.c (mon.2) 730 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:33:59.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:33:59 smithi050 bash[29234]: audit 2024-04-26T16:33:59.375520+0000 mon.c (mon.2) 730 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:33:59.658 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:33:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:33:59.730 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:33:59.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:59.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:33:59.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:33:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.820680+0000 mgr.y (mgr.24401) 639 : cluster [DBG] pgmap v1054: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:33:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:57.820680+0000 mgr.y (mgr.24401) 639 : cluster [DBG] pgmap v1054: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:33:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:58.736543+0000 mgr.y (mgr.24401) 640 : audit [DBG] from='client.30909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:33:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:58.736543+0000 mgr.y (mgr.24401) 640 : audit [DBG] from='client.30909 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:33:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:58.837706+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:58.837706+0000 mon.a (mon.0) 3374 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.339197+0000 mon.a (mon.0) 3375 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.339197+0000 mon.a (mon.0) 3375 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.339233+0000 mon.a (mon.0) 3376 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.339233+0000 mon.a (mon.0) 3376 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:59.341325+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]': finished 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:59.341325+0000 mon.a (mon.0) 3377 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-116", "mode": "writeback"}]': finished 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:59.341488+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:59.341488+0000 mon.a (mon.0) 3378 : audit [INF] from='client.? 172.21.15.50:0/3445487460' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.346812+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.346812+0000 mon.a (mon.0) 3379 : cluster [DBG] osdmap e687: 8 total, 7 up, 8 in 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:59.375520+0000 mon.c (mon.2) 730 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:33:59.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:33:59 smithi190 bash[23868]: audit 2024-04-26T16:33:59.375520+0000 mon.c (mon.2) 730 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:33:59.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:59.932 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:33:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:33:59.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:00.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:00.065 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:00.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:00.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:00.301 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:00.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:00.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:00.457 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:00.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:00.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:00.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[21463]: audit 2024-04-26T16:33:59.375791+0000 mgr.y (mgr.24401) 641 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:34:00.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[21463]: audit 2024-04-26T16:33:59.375791+0000 mgr.y (mgr.24401) 641 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:34:00.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.510619+0000 osd.7 (osd.7) 371 : cluster [DBG] 326.2 scrub starts 2024-04-26T16:34:00.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.510619+0000 osd.7 (osd.7) 371 : cluster [DBG] 326.2 scrub starts 2024-04-26T16:34:00.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.519173+0000 osd.7 (osd.7) 372 : cluster [DBG] 326.2 scrub ok 2024-04-26T16:34:00.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.519173+0000 osd.7 (osd.7) 372 : cluster [DBG] 326.2 scrub ok 2024-04-26T16:34:00.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[29234]: audit 2024-04-26T16:33:59.375791+0000 mgr.y (mgr.24401) 641 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:34:00.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[29234]: audit 2024-04-26T16:33:59.375791+0000 mgr.y (mgr.24401) 641 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:34:00.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.510619+0000 osd.7 (osd.7) 371 : cluster [DBG] 326.2 scrub starts 2024-04-26T16:34:00.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.510619+0000 osd.7 (osd.7) 371 : cluster [DBG] 326.2 scrub starts 2024-04-26T16:34:00.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.519173+0000 osd.7 (osd.7) 372 : cluster [DBG] 326.2 scrub ok 2024-04-26T16:34:00.686 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:00 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.519173+0000 osd.7 (osd.7) 372 : cluster [DBG] 326.2 scrub ok 2024-04-26T16:34:00.687 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1055,"stamp":"2024-04-26T16:33:58.340208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596571,"num_objects":209,"num_object_clones":4,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":75,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":1,"num_read":2341,"num_read_kb":2186,"num_write":918,"num_write_kb":2477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1774538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1069,"ondisk_log_size":1069,"up":544,"acting":544,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":612,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":681476,"kb_used_data":26812,"kb_used_omap":26,"kb_used_meta":654501,"kb_avail":749279740,"statfs":{"total":767960285184,"available":767262453760,"internally_reserved":0,"allocated":27455488,"data_stored":11934522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26682,"internal_metadata":670209990},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":47,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":5,"num_read_kb":5,"num_write":8,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.010871"},"pg_stats":[{"pgid":"326.1f","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338625+0000","last_change":"2024-04-26T16:33:55.375154+0000","last_active":"2024-04-26T16:33:56.338625+0000","last_peered":"2024-04-26T16:33:56.338625+0000","last_clean":"2024-04-26T16:33:56.338625+0000","last_became_active":"2024-04-26T16:33:55.374737+0000","last_became_peered":"2024-04-26T16:33:55.374737+0000","last_unstale":"2024-04-26T16:33:56.338625+0000","last_undegraded":"2024-04-26T16:33:56.338625+0000","last_fullsized":"2024-04-26T16:33:56.338625+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:13.744209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"326.1e","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341283+0000","last_change":"2024-04-26T16:33:55.360841+0000","last_active":"2024-04-26T16:33:56.341283+0000","last_peered":"2024-04-26T16:33:56.341283+0000","last_clean":"2024-04-26T16:33:56.341283+0000","last_became_active":"2024-04-26T16:33:55.360484+0000","last_became_peered":"2024-04-26T16:33:55.360484+0000","last_unstale":"2024-04-26T16:33:56.341283+0000","last_undegraded":"2024-04-26T16:33:56.341283+0000","last_fullsized":"2024-04-26T16:33:56.341283+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.018167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.1d","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.355816+0000","last_change":"2024-04-26T16:33:55.355414+0000","last_active":"2024-04-26T16:33:56.355816+0000","last_peered":"2024-04-26T16:33:56.355816+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.355414+0000","last_became_peered":"2024-04-26T16:33:55.355414+0000","last_unstale":"2024-04-26T16:33:56.355816+0000","last_undegraded":"2024-04-26T16:33:56.355816+0000","last_fullsized":"2024-04-26T16:33:55.343457+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:02.542678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"326.1c","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.347257+0000","last_change":"2024-04-26T16:33:55.348643+0000","last_active":"2024-04-26T16:33:56.347257+0000","last_peered":"2024-04-26T16:33:56.347257+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.348643+0000","last_became_peered":"2024-04-26T16:33:55.348643+0000","last_unstale":"2024-04-26T16:33:56.347257+0000","last_undegraded":"2024-04-26T16:33:56.347257+0000","last_fullsized":"2024-04-26T16:33:55.340783+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:58.374452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.1b","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.353260+0000","last_change":"2024-04-26T16:33:55.366358+0000","last_active":"2024-04-26T16:33:56.353260+0000","last_peered":"2024-04-26T16:33:56.353260+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.366358+0000","last_became_peered":"2024-04-26T16:33:55.366358+0000","last_unstale":"2024-04-26T16:33:56.353260+0000","last_undegraded":"2024-04-26T16:33:56.353260+0000","last_fullsized":"2024-04-26T16:33:55.342852+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:56.026689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"326.1a","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363464+0000","last_change":"2024-04-26T16:33:55.359040+0000","last_active":"2024-04-26T16:33:56.363464+0000","last_peered":"2024-04-26T16:33:56.363464+0000","last_clean":"2024-04-26T16:33:56.363464+0000","last_became_active":"2024-04-26T16:33:55.358517+0000","last_became_peered":"2024-04-26T16:33:55.358517+0000","last_unstale":"2024-04-26T16:33:56.363464+0000","last_undegraded":"2024-04-26T16:33:56.363464+0000","last_fullsized":"2024-04-26T16:33:56.363464+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:13.827160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.19","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.368772+0000","last_change":"2024-04-26T16:33:55.360511+0000","last_active":"2024-04-26T16:33:56.368772+0000","last_peered":"2024-04-26T16:33:56.368772+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.360511+0000","last_became_peered":"2024-04-26T16:33:55.360511+0000","last_unstale":"2024-04-26T16:33:56.368772+0000","last_undegraded":"2024-04-26T16:33:56.368772+0000","last_fullsized":"2024-04-26T16:33:55.343783+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:23.600105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.18","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368121+0000","last_change":"2024-04-26T16:33:55.363334+0000","last_active":"2024-04-26T16:33:56.368121+0000","last_peered":"2024-04-26T16:33:56.368121+0000","last_clean":"2024-04-26T16:33:56.368121+0000","last_became_active":"2024-04-26T16:33:55.362429+0000","last_became_peered":"2024-04-26T16:33:55.362429+0000","last_unstale":"2024-04-26T16:33:56.368121+0000","last_undegraded":"2024-04-26T16:33:56.368121+0000","last_fullsized":"2024-04-26T16:33:56.368121+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.779790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.17","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.364881+0000","last_change":"2024-04-26T16:33:55.353387+0000","last_active":"2024-04-26T16:33:56.364881+0000","last_peered":"2024-04-26T16:33:56.364881+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.353387+0000","last_became_peered":"2024-04-26T16:33:55.353387+0000","last_unstale":"2024-04-26T16:33:56.364881+0000","last_undegraded":"2024-04-26T16:33:56.364881+0000","last_fullsized":"2024-04-26T16:33:55.343012+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.037687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.16","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.364897+0000","last_change":"2024-04-26T16:33:55.356216+0000","last_active":"2024-04-26T16:33:56.364897+0000","last_peered":"2024-04-26T16:33:56.364897+0000","last_clean":"2024-04-26T16:33:56.364897+0000","last_became_active":"2024-04-26T16:33:55.355869+0000","last_became_peered":"2024-04-26T16:33:55.355869+0000","last_unstale":"2024-04-26T16:33:56.364897+0000","last_undegraded":"2024-04-26T16:33:56.364897+0000","last_fullsized":"2024-04-26T16:33:56.364897+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.325528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.15","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341220+0000","last_change":"2024-04-26T16:33:55.358007+0000","last_active":"2024-04-26T16:33:56.341220+0000","last_peered":"2024-04-26T16:33:56.341220+0000","last_clean":"2024-04-26T16:33:56.341220+0000","last_became_active":"2024-04-26T16:33:55.357685+0000","last_became_peered":"2024-04-26T16:33:55.357685+0000","last_unstale":"2024-04-26T16:33:56.341220+0000","last_undegraded":"2024-04-26T16:33:56.341220+0000","last_fullsized":"2024-04-26T16:33:56.341220+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:07.831241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.14","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.341274+0000","last_change":"2024-04-26T16:33:55.346097+0000","last_active":"2024-04-26T16:33:56.341274+0000","last_peered":"2024-04-26T16:33:56.341274+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.346097+0000","last_became_peered":"2024-04-26T16:33:55.346097+0000","last_unstale":"2024-04-26T16:33:56.341274+0000","last_undegraded":"2024-04-26T16:33:56.341274+0000","last_fullsized":"2024-04-26T16:33:55.334169+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.725962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.13","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363314+0000","last_change":"2024-04-26T16:33:55.352530+0000","last_active":"2024-04-26T16:33:56.363314+0000","last_peered":"2024-04-26T16:33:56.363314+0000","last_clean":"2024-04-26T16:33:56.363314+0000","last_became_active":"2024-04-26T16:33:55.352135+0000","last_became_peered":"2024-04-26T16:33:55.352135+0000","last_unstale":"2024-04-26T16:33:56.363314+0000","last_undegraded":"2024-04-26T16:33:56.363314+0000","last_fullsized":"2024-04-26T16:33:56.363314+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.515115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.12","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339861+0000","last_change":"2024-04-26T16:33:55.360976+0000","last_active":"2024-04-26T16:33:56.339861+0000","last_peered":"2024-04-26T16:33:56.339861+0000","last_clean":"2024-04-26T16:33:56.339861+0000","last_became_active":"2024-04-26T16:33:55.360002+0000","last_became_peered":"2024-04-26T16:33:55.360002+0000","last_unstale":"2024-04-26T16:33:56.339861+0000","last_undegraded":"2024-04-26T16:33:56.339861+0000","last_fullsized":"2024-04-26T16:33:56.339861+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:09.034397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.11","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.344439+0000","last_change":"2024-04-26T16:33:55.370673+0000","last_active":"2024-04-26T16:33:56.344439+0000","last_peered":"2024-04-26T16:33:56.344439+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.370673+0000","last_became_peered":"2024-04-26T16:33:55.370673+0000","last_unstale":"2024-04-26T16:33:56.344439+0000","last_undegraded":"2024-04-26T16:33:56.344439+0000","last_fullsized":"2024-04-26T16:33:55.342768+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.887247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.10","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.343867+0000","last_change":"2024-04-26T16:33:55.365816+0000","last_active":"2024-04-26T16:33:56.343867+0000","last_peered":"2024-04-26T16:33:56.343867+0000","last_clean":"2024-04-26T16:33:56.343867+0000","last_became_active":"2024-04-26T16:33:55.365338+0000","last_became_peered":"2024-04-26T16:33:55.365338+0000","last_unstale":"2024-04-26T16:33:56.343867+0000","last_undegraded":"2024-04-26T16:33:56.343867+0000","last_fullsized":"2024-04-26T16:33:56.343867+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.079882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.f","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341797+0000","last_change":"2024-04-26T16:33:55.373566+0000","last_active":"2024-04-26T16:33:56.341797+0000","last_peered":"2024-04-26T16:33:56.341797+0000","last_clean":"2024-04-26T16:33:56.341797+0000","last_became_active":"2024-04-26T16:33:55.373283+0000","last_became_peered":"2024-04-26T16:33:55.373283+0000","last_unstale":"2024-04-26T16:33:56.341797+0000","last_undegraded":"2024-04-26T16:33:56.341797+0000","last_fullsized":"2024-04-26T16:33:56.341797+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:05.085961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.e","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.365310+0000","last_change":"2024-04-26T16:33:55.355829+0000","last_active":"2024-04-26T16:33:56.365310+0000","last_peered":"2024-04-26T16:33:56.365310+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.355829+0000","last_became_peered":"2024-04-26T16:33:55.355829+0000","last_unstale":"2024-04-26T16:33:56.365310+0000","last_undegraded":"2024-04-26T16:33:56.365310+0000","last_fullsized":"2024-04-26T16:33:55.341477+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.176023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.d","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.346147+0000","last_change":"2024-04-26T16:33:55.365648+0000","last_active":"2024-04-26T16:33:56.346147+0000","last_peered":"2024-04-26T16:33:56.346147+0000","last_clean":"2024-04-26T16:33:56.346147+0000","last_became_active":"2024-04-26T16:33:55.365288+0000","last_became_peered":"2024-04-26T16:33:55.365288+0000","last_unstale":"2024-04-26T16:33:56.346147+0000","last_undegraded":"2024-04-26T16:33:56.346147+0000","last_fullsized":"2024-04-26T16:33:56.346147+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:16.672822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.c","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340248+0000","last_change":"2024-04-26T16:33:55.362863+0000","last_active":"2024-04-26T16:33:56.340248+0000","last_peered":"2024-04-26T16:33:56.340248+0000","last_clean":"2024-04-26T16:33:56.340248+0000","last_became_active":"2024-04-26T16:33:55.361982+0000","last_became_peered":"2024-04-26T16:33:55.361982+0000","last_unstale":"2024-04-26T16:33:56.340248+0000","last_undegraded":"2024-04-26T16:33:56.340248+0000","last_fullsized":"2024-04-26T16:33:56.340248+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:07.991648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.b","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.342443+0000","last_change":"2024-04-26T16:33:55.353062+0000","last_active":"2024-04-26T16:33:56.342443+0000","last_peered":"2024-04-26T16:33:56.342443+0000","last_clean":"2024-04-26T16:33:56.342443+0000","last_became_active":"2024-04-26T16:33:55.352369+0000","last_became_peered":"2024-04-26T16:33:55.352369+0000","last_unstale":"2024-04-26T16:33:56.342443+0000","last_undegraded":"2024-04-26T16:33:56.342443+0000","last_fullsized":"2024-04-26T16:33:56.342443+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.569745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.a","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363546+0000","last_change":"2024-04-26T16:33:55.359502+0000","last_active":"2024-04-26T16:33:56.363546+0000","last_peered":"2024-04-26T16:33:56.363546+0000","last_clean":"2024-04-26T16:33:56.363546+0000","last_became_active":"2024-04-26T16:33:55.359179+0000","last_became_peered":"2024-04-26T16:33:55.359179+0000","last_unstale":"2024-04-26T16:33:56.363546+0000","last_undegraded":"2024-04-26T16:33:56.363546+0000","last_fullsized":"2024-04-26T16:33:56.363546+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:05.023326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.9","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340436+0000","last_change":"2024-04-26T16:33:55.350795+0000","last_active":"2024-04-26T16:33:56.340436+0000","last_peered":"2024-04-26T16:33:56.340436+0000","last_clean":"2024-04-26T16:33:56.340436+0000","last_became_active":"2024-04-26T16:33:55.349152+0000","last_became_peered":"2024-04-26T16:33:55.349152+0000","last_unstale":"2024-04-26T16:33:56.340436+0000","last_undegraded":"2024-04-26T16:33:56.340436+0000","last_fullsized":"2024-04-26T16:33:56.340436+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.411821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.8","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368273+0000","last_change":"2024-04-26T16:33:55.363377+0000","last_active":"2024-04-26T16:33:56.368273+0000","last_peered":"2024-04-26T16:33:56.368273+0000","last_clean":"2024-04-26T16:33:56.368273+0000","last_became_active":"2024-04-26T16:33:55.362529+0000","last_became_peered":"2024-04-26T16:33:55.362529+0000","last_unstale":"2024-04-26T16:33:56.368273+0000","last_undegraded":"2024-04-26T16:33:56.368273+0000","last_fullsized":"2024-04-26T16:33:56.368273+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:19.240514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.2","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368596+0000","last_change":"2024-04-26T16:33:55.370818+0000","last_active":"2024-04-26T16:33:56.368596+0000","last_peered":"2024-04-26T16:33:56.368596+0000","last_clean":"2024-04-26T16:33:56.368596+0000","last_became_active":"2024-04-26T16:33:55.369432+0000","last_became_peered":"2024-04-26T16:33:55.369432+0000","last_unstale":"2024-04-26T16:33:56.368596+0000","last_undegraded":"2024-04-26T16:33:56.368596+0000","last_fullsized":"2024-04-26T16:33:56.368596+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.484805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.7","version":"684'3","reported_seq":52,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.355988+0000","last_change":"2024-04-26T16:33:56.354617+0000","last_active":"2024-04-26T16:33:56.355988+0000","last_peered":"2024-04-26T16:33:56.355988+0000","last_clean":"2024-04-26T16:33:38.361109+0000","last_became_active":"2024-04-26T16:33:39.395238+0000","last_became_peered":"2024-04-26T16:33:39.395238+0000","last_unstale":"2024-04-26T16:33:56.355988+0000","last_undegraded":"2024-04-26T16:33:55.373093+0000","last_fullsized":"2024-04-26T16:33:39.388686+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:50.248300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,1],"acting":[5,2147483647,1],"avail_no_missing":["5(0)","1(2)"],"object_location_counts":[{"shards":"1(2),5(0)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"326.3","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.353231+0000","last_change":"2024-04-26T16:33:55.368911+0000","last_active":"2024-04-26T16:33:56.353231+0000","last_peered":"2024-04-26T16:33:56.353231+0000","last_clean":"2024-04-26T16:33:56.353231+0000","last_became_active":"2024-04-26T16:33:55.357778+0000","last_became_peered":"2024-04-26T16:33:55.357778+0000","last_unstale":"2024-04-26T16:33:56.353231+0000","last_undegraded":"2024-04-26T16:33:56.353231+0000","last_fullsized":"2024-04-26T16:33:56.353231+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.345235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"323.6","version":"0'0","reported_seq":54,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341188+0000","last_change":"2024-04-26T16:33:39.397262+0000","last_active":"2024-04-26T16:33:56.341188+0000","last_peered":"2024-04-26T16:33:56.341188+0000","last_clean":"2024-04-26T16:33:56.341188+0000","last_became_active":"2024-04-26T16:33:39.394825+0000","last_became_peered":"2024-04-26T16:33:39.394825+0000","last_unstale":"2024-04-26T16:33:56.341188+0000","last_undegraded":"2024-04-26T16:33:56.341188+0000","last_fullsized":"2024-04-26T16:33:56.341188+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.159234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.0","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338652+0000","last_change":"2024-04-26T16:33:55.369042+0000","last_active":"2024-04-26T16:33:56.338652+0000","last_peered":"2024-04-26T16:33:56.338652+0000","last_clean":"2024-04-26T16:33:56.338652+0000","last_became_active":"2024-04-26T16:33:55.367891+0000","last_became_peered":"2024-04-26T16:33:55.367891+0000","last_unstale":"2024-04-26T16:33:56.338652+0000","last_undegraded":"2024-04-26T16:33:56.338652+0000","last_fullsized":"2024-04-26T16:33:56.338652+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.999279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.5","version":"0'0","reported_seq":54,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340750+0000","last_change":"2024-04-26T16:33:39.412685+0000","last_active":"2024-04-26T16:33:56.340750+0000","last_peered":"2024-04-26T16:33:56.340750+0000","last_clean":"2024-04-26T16:33:56.340750+0000","last_became_active":"2024-04-26T16:33:39.411800+0000","last_became_peered":"2024-04-26T16:33:39.411800+0000","last_unstale":"2024-04-26T16:33:56.340750+0000","last_undegraded":"2024-04-26T16:33:56.340750+0000","last_fullsized":"2024-04-26T16:33:56.340750+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.349385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"326.1","version":"0'0","reported_seq":13,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.352505+0000","last_change":"2024-04-26T16:33:55.354312+0000","last_active":"2024-04-26T16:33:56.352505+0000","last_peered":"2024-04-26T16:33:56.352505+0000","last_clean":"2024-04-26T16:33:56.352505+0000","last_became_active":"2024-04-26T16:33:55.353325+0000","last_became_peered":"2024-04-26T16:33:55.353325+0000","last_unstale":"2024-04-26T16:33:56.352505+0000","last_undegraded":"2024-04-26T16:33:56.352505+0000","last_fullsized":"2024-04-26T16:33:56.352505+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:19.722492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"323.4","version":"684'4","reported_seq":58,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363874+0000","last_change":"2024-04-26T16:33:39.407057+0000","last_active":"2024-04-26T16:33:56.363874+0000","last_peered":"2024-04-26T16:33:56.363874+0000","last_clean":"2024-04-26T16:33:56.363874+0000","last_became_active":"2024-04-26T16:33:39.406081+0000","last_became_peered":"2024-04-26T16:33:39.406081+0000","last_unstale":"2024-04-26T16:33:56.363874+0000","last_undegraded":"2024-04-26T16:33:56.363874+0000","last_fullsized":"2024-04-26T16:33:56.363874+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:43.341524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"326.6","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.342105+0000","last_change":"2024-04-26T16:33:55.366004+0000","last_active":"2024-04-26T16:33:56.342105+0000","last_peered":"2024-04-26T16:33:56.342105+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.366004+0000","last_became_peered":"2024-04-26T16:33:55.366004+0000","last_unstale":"2024-04-26T16:33:56.342105+0000","last_undegraded":"2024-04-26T16:33:56.342105+0000","last_fullsized":"2024-04-26T16:33:55.334708+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.374140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"323.3","version":"0'0","reported_seq":51,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.345090+0000","last_change":"2024-04-26T16:33:39.398035+0000","last_active":"2024-04-26T16:33:56.345090+0000","last_peered":"2024-04-26T16:33:56.345090+0000","last_clean":"2024-04-26T16:33:38.361109+0000","last_became_active":"2024-04-26T16:33:39.398035+0000","last_became_peered":"2024-04-26T16:33:39.398035+0000","last_unstale":"2024-04-26T16:33:56.345090+0000","last_undegraded":"2024-04-26T16:33:56.345090+0000","last_fullsized":"2024-04-26T16:33:39.372843+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:47.854737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,4],"acting":[3,2147483647,4],"avail_no_missing":["3(0)","4(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"326.7","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.342278+0000","last_change":"2024-04-26T16:33:55.369966+0000","last_active":"2024-04-26T16:33:56.342278+0000","last_peered":"2024-04-26T16:33:56.342278+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.369966+0000","last_became_peered":"2024-04-26T16:33:55.369966+0000","last_unstale":"2024-04-26T16:33:56.342278+0000","last_undegraded":"2024-04-26T16:33:56.342278+0000","last_fullsized":"2024-04-26T16:33:55.336607+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.921684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.2","version":"684'6","reported_seq":58,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.344112+0000","last_change":"2024-04-26T16:33:39.413740+0000","last_active":"2024-04-26T16:33:56.344112+0000","last_peered":"2024-04-26T16:33:56.344112+0000","last_clean":"2024-04-26T16:33:56.344112+0000","last_became_active":"2024-04-26T16:33:39.413256+0000","last_became_peered":"2024-04-26T16:33:39.413256+0000","last_unstale":"2024-04-26T16:33:56.344112+0000","last_undegraded":"2024-04-26T16:33:56.344112+0000","last_fullsized":"2024-04-26T16:33:56.344112+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.766893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"326.4","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.353394+0000","last_change":"2024-04-26T16:33:55.366391+0000","last_active":"2024-04-26T16:33:56.353394+0000","last_peered":"2024-04-26T16:33:56.353394+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.366391+0000","last_became_peered":"2024-04-26T16:33:55.366391+0000","last_unstale":"2024-04-26T16:33:56.353394+0000","last_undegraded":"2024-04-26T16:33:56.353394+0000","last_fullsized":"2024-04-26T16:33:55.343232+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.852325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"323.1","version":"675'1","reported_seq":55,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368358+0000","last_change":"2024-04-26T16:33:39.395161+0000","last_active":"2024-04-26T16:33:56.368358+0000","last_peered":"2024-04-26T16:33:56.368358+0000","last_clean":"2024-04-26T16:33:56.368358+0000","last_became_active":"2024-04-26T16:33:39.394480+0000","last_became_peered":"2024-04-26T16:33:39.394480+0000","last_unstale":"2024-04-26T16:33:56.368358+0000","last_undegraded":"2024-04-26T16:33:56.368358+0000","last_fullsized":"2024-04-26T16:33:56.368358+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:52.849717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"326.5","version":"0'0","reported_seq":10,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.371866+0000","last_change":"2024-04-26T16:33:55.366402+0000","last_active":"2024-04-26T16:33:56.371866+0000","last_peered":"2024-04-26T16:33:56.371866+0000","last_clean":"2024-04-26T16:33:54.325352+0000","last_became_active":"2024-04-26T16:33:55.366402+0000","last_became_peered":"2024-04-26T16:33:55.366402+0000","last_unstale":"2024-04-26T16:33:56.371866+0000","last_undegraded":"2024-04-26T16:33:56.371866+0000","last_fullsized":"2024-04-26T16:33:55.360066+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":682,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.325352+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.325352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:02.932854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"323.0","version":"0'0","reported_seq":51,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.336766+0000","last_change":"2024-04-26T16:33:39.388092+0000","last_active":"2024-04-26T16:33:56.336766+0000","last_peered":"2024-04-26T16:33:56.336766+0000","last_clean":"2024-04-26T16:33:38.361109+0000","last_became_active":"2024-04-26T16:33:39.388092+0000","last_became_peered":"2024-04-26T16:33:39.388092+0000","last_unstale":"2024-04-26T16:33:56.336766+0000","last_undegraded":"2024-04-26T16:33:56.336766+0000","last_fullsized":"2024-04-26T16:33:39.381136+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.749818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2147483647],"acting":[1,4,2147483647],"avail_no_missing":["1(0)","4(1)"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.356522+0000","last_change":"2024-04-26T16:33:56.069998+0000","last_active":"2024-04-26T16:33:56.356522+0000","last_peered":"2024-04-26T16:33:56.356522+0000","last_clean":"2024-04-26T16:33:56.356522+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:33:56.356522+0000","last_undegraded":"2024-04-26T16:33:56.356522+0000","last_fullsized":"2024-04-26T16:33:56.356522+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:56.069784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:56.069784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:21.210472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"630'15","reported_seq":1475,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.370161+0000","last_change":"2024-04-26T16:33:02.314019+0000","last_active":"2024-04-26T16:33:56.370161+0000","last_peered":"2024-04-26T16:33:56.370161+0000","last_clean":"2024-04-26T16:33:01.315658+0000","last_became_active":"2024-04-26T16:33:02.314019+0000","last_became_peered":"2024-04-26T16:33:02.314019+0000","last_unstale":"2024-04-26T16:33:56.370161+0000","last_undegraded":"2024-04-26T16:33:02.305255+0000","last_fullsized":"2024-04-26T16:33:02.305084+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.136172+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:57.505331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"458'34","reported_seq":1335,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.372413+0000","last_change":"2024-04-26T16:33:19.589656+0000","last_active":"2024-04-26T16:33:56.372413+0000","last_peered":"2024-04-26T16:33:56.372413+0000","last_clean":"2024-04-26T16:33:56.372413+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:33:56.372413+0000","last_undegraded":"2024-04-26T16:33:56.372413+0000","last_fullsized":"2024-04-26T16:33:56.372413+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:33:19.589375+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:33:19.589375+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:46.758750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1281,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338261+0000","last_change":"2024-04-26T16:33:19.749849+0000","last_active":"2024-04-26T16:33:56.338261+0000","last_peered":"2024-04-26T16:33:56.338261+0000","last_clean":"2024-04-26T16:33:56.338261+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:33:56.338261+0000","last_undegraded":"2024-04-26T16:33:56.338261+0000","last_fullsized":"2024-04-26T16:33:56.338261+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:19.749730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:33:19.749730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:30.516761+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":206,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.353718+0000","last_change":"2024-04-26T16:33:02.307160+0000","last_active":"2024-04-26T16:33:56.353718+0000","last_peered":"2024-04-26T16:33:56.353718+0000","last_clean":"2024-04-26T16:33:00.424958+0000","last_became_active":"2024-04-26T16:33:02.307160+0000","last_became_peered":"2024-04-26T16:33:02.307160+0000","last_unstale":"2024-04-26T16:33:56.353718+0000","last_undegraded":"2024-04-26T16:33:56.353718+0000","last_fullsized":"2024-04-26T16:33:02.291975+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:16.230516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.230516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1123,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.371482+0000","last_change":"2024-04-26T16:33:02.320002+0000","last_active":"2024-04-26T16:33:56.371482+0000","last_peered":"2024-04-26T16:33:56.371482+0000","last_clean":"2024-04-26T16:33:01.305664+0000","last_became_active":"2024-04-26T16:33:02.320002+0000","last_became_peered":"2024-04-26T16:33:02.320002+0000","last_unstale":"2024-04-26T16:33:56.371482+0000","last_undegraded":"2024-04-26T16:33:56.371482+0000","last_fullsized":"2024-04-26T16:33:02.303449+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:19.402195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:19.402195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"388'20","reported_seq":218,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.354553+0000","last_change":"2024-04-26T16:33:02.316135+0000","last_active":"2024-04-26T16:33:56.354553+0000","last_peered":"2024-04-26T16:33:56.354553+0000","last_clean":"2024-04-26T16:33:00.426585+0000","last_became_active":"2024-04-26T16:33:02.316135+0000","last_became_peered":"2024-04-26T16:33:02.316135+0000","last_unstale":"2024-04-26T16:33:56.354553+0000","last_undegraded":"2024-04-26T16:33:02.292669+0000","last_fullsized":"2024-04-26T16:33:02.292330+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:19.056987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.056987+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1284,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341243+0000","last_change":"2024-04-26T16:32:38.352954+0000","last_active":"2024-04-26T16:33:56.341243+0000","last_peered":"2024-04-26T16:33:56.341243+0000","last_clean":"2024-04-26T16:33:56.341243+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:33:56.341243+0000","last_undegraded":"2024-04-26T16:33:56.341243+0000","last_fullsized":"2024-04-26T16:33:56.341243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:38.352608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:32:38.352608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:03.984259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1281,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.336903+0000","last_change":"2024-04-26T16:33:32.729265+0000","last_active":"2024-04-26T16:33:56.336903+0000","last_peered":"2024-04-26T16:33:56.336903+0000","last_clean":"2024-04-26T16:33:56.336903+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:33:56.336903+0000","last_undegraded":"2024-04-26T16:33:56.336903+0000","last_fullsized":"2024-04-26T16:33:56.336903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:32.728939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:32.728939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.074056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1283,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363390+0000","last_change":"2024-04-26T16:33:38.754358+0000","last_active":"2024-04-26T16:33:56.363390+0000","last_peered":"2024-04-26T16:33:56.363390+0000","last_clean":"2024-04-26T16:33:56.363390+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:33:56.363390+0000","last_undegraded":"2024-04-26T16:33:56.363390+0000","last_fullsized":"2024-04-26T16:33:56.363390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.754239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.754239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.201822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1143,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.375081+0000","last_change":"2024-04-26T16:33:02.316826+0000","last_active":"2024-04-26T16:33:56.375081+0000","last_peered":"2024-04-26T16:33:56.375081+0000","last_clean":"2024-04-26T16:33:01.300984+0000","last_became_active":"2024-04-26T16:33:02.316826+0000","last_became_peered":"2024-04-26T16:33:02.316826+0000","last_unstale":"2024-04-26T16:33:56.375081+0000","last_undegraded":"2024-04-26T16:33:56.375081+0000","last_fullsized":"2024-04-26T16:33:02.299005+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:16.340018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:16.340018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"612'39","reported_seq":1380,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.374931+0000","last_change":"2024-04-26T16:33:02.322018+0000","last_active":"2024-04-26T16:33:56.374931+0000","last_peered":"2024-04-26T16:33:56.374931+0000","last_clean":"2024-04-26T16:33:01.301948+0000","last_became_active":"2024-04-26T16:33:02.322018+0000","last_became_peered":"2024-04-26T16:33:02.322018+0000","last_unstale":"2024-04-26T16:33:56.374931+0000","last_undegraded":"2024-04-26T16:33:02.297921+0000","last_fullsized":"2024-04-26T16:33:02.297601+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'39","last_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_deep_scrub":"612'39","last_deep_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_clean_scrub_stamp":"2024-04-26T16:32:44.660567+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:03.236961+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1301,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.337868+0000","last_change":"2024-04-26T16:33:31.754803+0000","last_active":"2024-04-26T16:33:56.337868+0000","last_peered":"2024-04-26T16:33:56.337868+0000","last_clean":"2024-04-26T16:33:56.337868+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:33:56.337868+0000","last_undegraded":"2024-04-26T16:33:56.337868+0000","last_fullsized":"2024-04-26T16:33:56.337868+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:31.754561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:31.754561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.140085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1293,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.337522+0000","last_change":"2024-04-26T16:33:15.657104+0000","last_active":"2024-04-26T16:33:56.337522+0000","last_peered":"2024-04-26T16:33:56.337522+0000","last_clean":"2024-04-26T16:33:56.337522+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:33:56.337522+0000","last_undegraded":"2024-04-26T16:33:56.337522+0000","last_fullsized":"2024-04-26T16:33:56.337522+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:15.656794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:15.656794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:25.435492+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1247,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368437+0000","last_change":"2024-04-26T16:31:19.857894+0000","last_active":"2024-04-26T16:33:56.368437+0000","last_peered":"2024-04-26T16:33:56.368437+0000","last_clean":"2024-04-26T16:33:56.368437+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:33:56.368437+0000","last_undegraded":"2024-04-26T16:33:56.368437+0000","last_fullsized":"2024-04-26T16:33:56.368437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:19.857656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.857656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1133,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.337320+0000","last_change":"2024-04-26T16:33:41.813706+0000","last_active":"2024-04-26T16:33:56.337320+0000","last_peered":"2024-04-26T16:33:56.337320+0000","last_clean":"2024-04-26T16:33:56.337320+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:33:56.337320+0000","last_undegraded":"2024-04-26T16:33:56.337320+0000","last_fullsized":"2024-04-26T16:33:56.337320+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:41.813414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:41.813414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.948002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"389'35","reported_seq":916,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.340247+0000","last_change":"2024-04-26T16:33:02.313411+0000","last_active":"2024-04-26T16:33:56.340247+0000","last_peered":"2024-04-26T16:33:56.340247+0000","last_clean":"2024-04-26T16:27:37.176266+0000","last_became_active":"2024-04-26T16:33:02.313411+0000","last_became_peered":"2024-04-26T16:33:02.313411+0000","last_unstale":"2024-04-26T16:33:56.340247+0000","last_undegraded":"2024-04-26T16:33:02.302284+0000","last_fullsized":"2024-04-26T16:33:02.302146+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:31:44.435924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:44.435924+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1279,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.372091+0000","last_change":"2024-04-26T16:33:45.527712+0000","last_active":"2024-04-26T16:33:56.372091+0000","last_peered":"2024-04-26T16:33:56.372091+0000","last_clean":"2024-04-26T16:33:56.372091+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:33:56.372091+0000","last_undegraded":"2024-04-26T16:33:56.372091+0000","last_fullsized":"2024-04-26T16:33:56.372091+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:45.527416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:33:45.527416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.112750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1284,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341155+0000","last_change":"2024-04-26T16:32:40.409359+0000","last_active":"2024-04-26T16:33:56.341155+0000","last_peered":"2024-04-26T16:33:56.341155+0000","last_clean":"2024-04-26T16:33:56.341155+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:33:56.341155+0000","last_undegraded":"2024-04-26T16:33:56.341155+0000","last_fullsized":"2024-04-26T16:33:56.341155+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:40.409156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:32:40.409156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:06.697031+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1277,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338862+0000","last_change":"2024-04-26T16:33:49.756193+0000","last_active":"2024-04-26T16:33:56.338862+0000","last_peered":"2024-04-26T16:33:56.338862+0000","last_clean":"2024-04-26T16:33:56.338862+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:33:56.338862+0000","last_undegraded":"2024-04-26T16:33:56.338862+0000","last_fullsized":"2024-04-26T16:33:56.338862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:49.755868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:33:49.755868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:00.792996+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":621,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.369469+0000","last_change":"2024-04-26T16:33:02.321421+0000","last_active":"2024-04-26T16:33:56.369469+0000","last_peered":"2024-04-26T16:33:56.369469+0000","last_clean":"2024-04-26T16:23:28.083853+0000","last_became_active":"2024-04-26T16:33:02.321421+0000","last_became_peered":"2024-04-26T16:33:02.321421+0000","last_unstale":"2024-04-26T16:33:56.369469+0000","last_undegraded":"2024-04-26T16:33:56.369469+0000","last_fullsized":"2024-04-26T16:33:02.303814+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:39.300548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.300548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"388'19","reported_seq":1294,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.346020+0000","last_change":"2024-04-26T16:33:42.230824+0000","last_active":"2024-04-26T16:33:56.346020+0000","last_peered":"2024-04-26T16:33:56.346020+0000","last_clean":"2024-04-26T16:33:56.346020+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:33:56.346020+0000","last_undegraded":"2024-04-26T16:33:56.346020+0000","last_fullsized":"2024-04-26T16:33:56.346020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:33:42.230574+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:33:42.230574+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.382496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1292,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339934+0000","last_change":"2024-04-26T16:33:49.416576+0000","last_active":"2024-04-26T16:33:56.339934+0000","last_peered":"2024-04-26T16:33:56.339934+0000","last_clean":"2024-04-26T16:33:56.339934+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:33:56.339934+0000","last_undegraded":"2024-04-26T16:33:56.339934+0000","last_fullsized":"2024-04-26T16:33:56.339934+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:49.416438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:49.416438+0000","last_clean_scrub_stamp":"2024-04-26T16:33:49.416438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:52.833664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":1509,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.337933+0000","last_change":"2024-04-26T16:33:24.713742+0000","last_active":"2024-04-26T16:33:56.337933+0000","last_peered":"2024-04-26T16:33:56.337933+0000","last_clean":"2024-04-26T16:33:56.337933+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:33:56.337933+0000","last_undegraded":"2024-04-26T16:33:56.337933+0000","last_fullsized":"2024-04-26T16:33:56.337933+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:33:24.713618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:24.713618+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:36.444990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1279,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.343112+0000","last_change":"2024-04-26T16:33:29.259855+0000","last_active":"2024-04-26T16:33:56.343112+0000","last_peered":"2024-04-26T16:33:56.343112+0000","last_clean":"2024-04-26T16:33:56.343112+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:33:56.343112+0000","last_undegraded":"2024-04-26T16:33:56.343112+0000","last_fullsized":"2024-04-26T16:33:56.343112+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:29.259588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:33:29.259588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:58.179092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1134,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339782+0000","last_change":"2024-04-26T16:33:48.426154+0000","last_active":"2024-04-26T16:33:56.339782+0000","last_peered":"2024-04-26T16:33:56.339782+0000","last_clean":"2024-04-26T16:33:56.339782+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:33:56.339782+0000","last_undegraded":"2024-04-26T16:33:56.339782+0000","last_fullsized":"2024-04-26T16:33:56.339782+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.425925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.425925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:59.626821+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"388'27","reported_seq":1308,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.344174+0000","last_change":"2024-04-26T16:33:26.168674+0000","last_active":"2024-04-26T16:33:56.344174+0000","last_peered":"2024-04-26T16:33:56.344174+0000","last_clean":"2024-04-26T16:33:56.344174+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:33:56.344174+0000","last_undegraded":"2024-04-26T16:33:56.344174+0000","last_fullsized":"2024-04-26T16:33:56.344174+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:33:26.168537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:33:26.168537+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:29.027831+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1298,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339785+0000","last_change":"2024-04-26T16:32:44.393515+0000","last_active":"2024-04-26T16:33:56.339785+0000","last_peered":"2024-04-26T16:33:56.339785+0000","last_clean":"2024-04-26T16:33:56.339785+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:33:56.339785+0000","last_undegraded":"2024-04-26T16:33:56.339785+0000","last_fullsized":"2024-04-26T16:33:56.339785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:44.393215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:32:44.393215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:07.194110+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":1504,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.342627+0000","last_change":"2024-04-26T16:32:43.175004+0000","last_active":"2024-04-26T16:33:56.342627+0000","last_peered":"2024-04-26T16:33:56.342627+0000","last_clean":"2024-04-26T16:33:56.342627+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:33:56.342627+0000","last_undegraded":"2024-04-26T16:33:56.342627+0000","last_fullsized":"2024-04-26T16:33:56.342627+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:32:43.174894+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:32:43.174894+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:01.508985+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.356399+0000","last_change":"2024-04-26T16:33:55.050828+0000","last_active":"2024-04-26T16:33:56.356399+0000","last_peered":"2024-04-26T16:33:56.356399+0000","last_clean":"2024-04-26T16:33:56.356399+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:33:56.356399+0000","last_undegraded":"2024-04-26T16:33:56.356399+0000","last_fullsized":"2024-04-26T16:33:56.356399+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:55.050611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:33:55.050611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:22.527158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":205,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.345986+0000","last_change":"2024-04-26T16:33:02.308041+0000","last_active":"2024-04-26T16:33:56.345986+0000","last_peered":"2024-04-26T16:33:56.345986+0000","last_clean":"2024-04-26T16:23:29.388954+0000","last_became_active":"2024-04-26T16:33:02.308041+0000","last_became_peered":"2024-04-26T16:33:02.308041+0000","last_unstale":"2024-04-26T16:33:56.345986+0000","last_undegraded":"2024-04-26T16:33:02.293884+0000","last_fullsized":"2024-04-26T16:33:02.293557+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:31:41.360858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:41.360858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"399'20","reported_seq":1313,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339597+0000","last_change":"2024-04-26T16:33:27.286559+0000","last_active":"2024-04-26T16:33:56.339597+0000","last_peered":"2024-04-26T16:33:56.339597+0000","last_clean":"2024-04-26T16:33:56.339597+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:33:56.339597+0000","last_undegraded":"2024-04-26T16:33:56.339597+0000","last_fullsized":"2024-04-26T16:33:56.339597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'20","last_scrub_stamp":"2024-04-26T16:33:27.286169+0000","last_deep_scrub":"399'20","last_deep_scrub_stamp":"2024-04-26T16:27:47.205374+0000","last_clean_scrub_stamp":"2024-04-26T16:33:27.286169+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:37.326526+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1275,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368989+0000","last_change":"2024-04-26T16:33:29.556528+0000","last_active":"2024-04-26T16:33:56.368989+0000","last_peered":"2024-04-26T16:33:56.368989+0000","last_clean":"2024-04-26T16:33:56.368989+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:33:56.368989+0000","last_undegraded":"2024-04-26T16:33:56.368989+0000","last_fullsized":"2024-04-26T16:33:56.368989+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:33:29.556400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:29.556400+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:34.184154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":206,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.356818+0000","last_change":"2024-04-26T16:33:02.317029+0000","last_active":"2024-04-26T16:33:56.356818+0000","last_peered":"2024-04-26T16:33:56.356818+0000","last_clean":"2024-04-26T16:33:00.427039+0000","last_became_active":"2024-04-26T16:33:02.317029+0000","last_became_peered":"2024-04-26T16:33:02.317029+0000","last_unstale":"2024-04-26T16:33:56.356818+0000","last_undegraded":"2024-04-26T16:33:56.356818+0000","last_fullsized":"2024-04-26T16:33:02.293791+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.518326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.518326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:00.045687+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1273,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.347328+0000","last_change":"2024-04-26T16:33:02.314384+0000","last_active":"2024-04-26T16:33:56.347328+0000","last_peered":"2024-04-26T16:33:56.347328+0000","last_clean":"2024-04-26T16:33:01.291281+0000","last_became_active":"2024-04-26T16:33:02.314384+0000","last_became_peered":"2024-04-26T16:33:02.314384+0000","last_unstale":"2024-04-26T16:33:56.347328+0000","last_undegraded":"2024-04-26T16:33:56.347328+0000","last_fullsized":"2024-04-26T16:33:02.294741+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.938129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.938129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1132,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.342771+0000","last_change":"2024-04-26T16:33:39.364471+0000","last_active":"2024-04-26T16:33:56.342771+0000","last_peered":"2024-04-26T16:33:56.342771+0000","last_clean":"2024-04-26T16:33:56.342771+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:33:56.342771+0000","last_undegraded":"2024-04-26T16:33:56.342771+0000","last_fullsized":"2024-04-26T16:33:56.342771+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.364274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.364274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:05.724315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1471,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.344334+0000","last_change":"2024-04-26T16:33:54.179489+0000","last_active":"2024-04-26T16:33:56.344334+0000","last_peered":"2024-04-26T16:33:56.344334+0000","last_clean":"2024-04-26T16:33:56.344334+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:33:56.344334+0000","last_undegraded":"2024-04-26T16:33:56.344334+0000","last_fullsized":"2024-04-26T16:33:56.344334+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.179322+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.010524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1308,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.339436+0000","last_change":"2024-04-26T16:33:02.314228+0000","last_active":"2024-04-26T16:33:56.339436+0000","last_peered":"2024-04-26T16:33:56.339436+0000","last_clean":"2024-04-26T16:33:00.882827+0000","last_became_active":"2024-04-26T16:33:02.314228+0000","last_became_peered":"2024-04-26T16:33:02.314228+0000","last_unstale":"2024-04-26T16:33:56.339436+0000","last_undegraded":"2024-04-26T16:33:56.339436+0000","last_fullsized":"2024-04-26T16:33:02.292892+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:31:42.531687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'30","reported_seq":1325,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.365371+0000","last_change":"2024-04-26T16:32:28.750812+0000","last_active":"2024-04-26T16:33:56.365371+0000","last_peered":"2024-04-26T16:33:56.365371+0000","last_clean":"2024-04-26T16:33:56.365371+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:33:56.365371+0000","last_undegraded":"2024-04-26T16:33:56.365371+0000","last_fullsized":"2024-04-26T16:33:56.365371+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'30","last_scrub_stamp":"2024-04-26T16:32:28.750687+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:32:28.750687+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":764,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.372036+0000","last_change":"2024-04-26T16:33:02.323596+0000","last_active":"2024-04-26T16:33:56.372036+0000","last_peered":"2024-04-26T16:33:56.372036+0000","last_clean":"2024-04-26T16:17:27.960068+0000","last_became_active":"2024-04-26T16:33:02.323596+0000","last_became_peered":"2024-04-26T16:33:02.323596+0000","last_unstale":"2024-04-26T16:33:56.372036+0000","last_undegraded":"2024-04-26T16:33:56.372036+0000","last_fullsized":"2024-04-26T16:33:02.313028+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:31.269480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:32:31.269480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1267,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.369919+0000","last_change":"2024-04-26T16:33:02.319882+0000","last_active":"2024-04-26T16:33:56.369919+0000","last_peered":"2024-04-26T16:33:56.369919+0000","last_clean":"2024-04-26T16:33:01.307621+0000","last_became_active":"2024-04-26T16:33:02.319882+0000","last_became_peered":"2024-04-26T16:33:02.319882+0000","last_unstale":"2024-04-26T16:33:56.369919+0000","last_undegraded":"2024-04-26T16:33:56.369919+0000","last_fullsized":"2024-04-26T16:33:02.304094+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:35.599770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:32:35.599770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.967890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1118,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.374461+0000","last_change":"2024-04-26T16:33:52.514069+0000","last_active":"2024-04-26T16:33:56.374461+0000","last_peered":"2024-04-26T16:33:56.374461+0000","last_clean":"2024-04-26T16:33:56.374461+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:33:56.374461+0000","last_undegraded":"2024-04-26T16:33:56.374461+0000","last_fullsized":"2024-04-26T16:33:56.374461+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:52.513926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:52.513926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:19.536574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1283,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338873+0000","last_change":"2024-04-26T16:33:51.205929+0000","last_active":"2024-04-26T16:33:56.338873+0000","last_peered":"2024-04-26T16:33:56.338873+0000","last_clean":"2024-04-26T16:33:56.338873+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:33:56.338873+0000","last_undegraded":"2024-04-26T16:33:56.338873+0000","last_fullsized":"2024-04-26T16:33:56.338873+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:33:51.205695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:59.469040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":2239,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.344210+0000","last_change":"2024-04-26T16:33:32.150005+0000","last_active":"2024-04-26T16:33:56.344210+0000","last_peered":"2024-04-26T16:33:56.344210+0000","last_clean":"2024-04-26T16:33:56.344210+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:33:56.344210+0000","last_undegraded":"2024-04-26T16:33:56.344210+0000","last_fullsized":"2024-04-26T16:33:56.344210+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:33:32.149895+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:33:32.149895+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:50.322385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":965,"num_read_kb":960,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'22","reported_seq":1303,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.347946+0000","last_change":"2024-04-26T16:33:22.114952+0000","last_active":"2024-04-26T16:33:56.347946+0000","last_peered":"2024-04-26T16:33:56.347946+0000","last_clean":"2024-04-26T16:33:56.347946+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:33:56.347946+0000","last_undegraded":"2024-04-26T16:33:56.347946+0000","last_fullsized":"2024-04-26T16:33:56.347946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:33:22.114852+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:33:22.114852+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:39.837888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1298,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.342124+0000","last_change":"2024-04-26T16:33:02.312665+0000","last_active":"2024-04-26T16:33:56.342124+0000","last_peered":"2024-04-26T16:33:56.342124+0000","last_clean":"2024-04-26T16:33:00.436780+0000","last_became_active":"2024-04-26T16:33:02.312665+0000","last_became_peered":"2024-04-26T16:33:02.312665+0000","last_unstale":"2024-04-26T16:33:56.342124+0000","last_undegraded":"2024-04-26T16:33:02.306363+0000","last_fullsized":"2024-04-26T16:33:02.305830+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:31:40.035882+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:31:40.035882+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1279,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339215+0000","last_change":"2024-04-26T16:33:42.347570+0000","last_active":"2024-04-26T16:33:56.339215+0000","last_peered":"2024-04-26T16:33:56.339215+0000","last_clean":"2024-04-26T16:33:56.339215+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:33:56.339215+0000","last_undegraded":"2024-04-26T16:33:56.339215+0000","last_fullsized":"2024-04-26T16:33:56.339215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:42.347471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:33:42.347471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:11.070217+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1133,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340608+0000","last_change":"2024-04-26T16:33:50.388012+0000","last_active":"2024-04-26T16:33:56.340608+0000","last_peered":"2024-04-26T16:33:56.340608+0000","last_clean":"2024-04-26T16:33:56.340608+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:33:56.340608+0000","last_undegraded":"2024-04-26T16:33:56.340608+0000","last_fullsized":"2024-04-26T16:33:56.340608+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:50.387871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:50.387871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:10.382696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1080,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.363673+0000","last_change":"2024-04-26T16:33:02.320523+0000","last_active":"2024-04-26T16:33:56.363673+0000","last_peered":"2024-04-26T16:33:56.363673+0000","last_clean":"2024-04-26T16:32:03.398109+0000","last_became_active":"2024-04-26T16:33:02.320523+0000","last_became_peered":"2024-04-26T16:33:02.320523+0000","last_unstale":"2024-04-26T16:33:56.363673+0000","last_undegraded":"2024-04-26T16:33:56.363673+0000","last_fullsized":"2024-04-26T16:33:02.302153+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:20.267979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:32:20.267979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"585'54","reported_seq":1620,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340716+0000","last_change":"2024-04-26T16:33:37.401179+0000","last_active":"2024-04-26T16:33:56.340716+0000","last_peered":"2024-04-26T16:33:56.340716+0000","last_clean":"2024-04-26T16:33:56.340716+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:33:56.340716+0000","last_undegraded":"2024-04-26T16:33:56.340716+0000","last_fullsized":"2024-04-26T16:33:56.340716+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'54","last_scrub_stamp":"2024-04-26T16:33:37.401059+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:33:37.401059+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.890971+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1276,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.374607+0000","last_change":"2024-04-26T16:33:36.587348+0000","last_active":"2024-04-26T16:33:56.374607+0000","last_peered":"2024-04-26T16:33:56.374607+0000","last_clean":"2024-04-26T16:33:56.374607+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:33:56.374607+0000","last_undegraded":"2024-04-26T16:33:56.374607+0000","last_fullsized":"2024-04-26T16:33:56.374607+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:33:36.586965+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:33:36.586965+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:49.837702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1284,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.339835+0000","last_change":"2024-04-26T16:33:02.319216+0000","last_active":"2024-04-26T16:33:56.339835+0000","last_peered":"2024-04-26T16:33:56.339835+0000","last_clean":"2024-04-26T16:33:00.429539+0000","last_became_active":"2024-04-26T16:33:02.319216+0000","last_became_peered":"2024-04-26T16:33:02.319216+0000","last_unstale":"2024-04-26T16:33:56.339835+0000","last_undegraded":"2024-04-26T16:33:56.339835+0000","last_fullsized":"2024-04-26T16:33:02.310178+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.037115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.037115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.465667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1116,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.345545+0000","last_change":"2024-04-26T16:33:02.307539+0000","last_active":"2024-04-26T16:33:56.345545+0000","last_peered":"2024-04-26T16:33:56.345545+0000","last_clean":"2024-04-26T16:33:01.287119+0000","last_became_active":"2024-04-26T16:33:02.307539+0000","last_became_peered":"2024-04-26T16:33:02.307539+0000","last_unstale":"2024-04-26T16:33:56.345545+0000","last_undegraded":"2024-04-26T16:33:56.345545+0000","last_fullsized":"2024-04-26T16:33:02.292490+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.933163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.933163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.356539+0000","last_change":"2024-04-26T16:33:27.328627+0000","last_active":"2024-04-26T16:33:56.356539+0000","last_peered":"2024-04-26T16:33:56.356539+0000","last_clean":"2024-04-26T16:33:56.356539+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:33:56.356539+0000","last_undegraded":"2024-04-26T16:33:56.356539+0000","last_fullsized":"2024-04-26T16:33:56.356539+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:27.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:33:27.328459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.188842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1268,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368512+0000","last_change":"2024-04-26T16:33:34.584942+0000","last_active":"2024-04-26T16:33:56.368512+0000","last_peered":"2024-04-26T16:33:56.368512+0000","last_clean":"2024-04-26T16:33:56.368512+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:33:56.368512+0000","last_undegraded":"2024-04-26T16:33:56.368512+0000","last_fullsized":"2024-04-26T16:33:56.368512+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:34.584623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:34.584623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:49.315788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'35","reported_seq":1332,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363137+0000","last_change":"2024-04-26T16:33:28.787554+0000","last_active":"2024-04-26T16:33:56.363137+0000","last_peered":"2024-04-26T16:33:56.363137+0000","last_clean":"2024-04-26T16:33:56.363137+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:33:56.363137+0000","last_undegraded":"2024-04-26T16:33:56.363137+0000","last_fullsized":"2024-04-26T16:33:56.363137+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:33:28.787378+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:33:28.787378+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.392393+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1293,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341303+0000","last_change":"2024-04-26T16:33:39.416961+0000","last_active":"2024-04-26T16:33:56.341303+0000","last_peered":"2024-04-26T16:33:56.341303+0000","last_clean":"2024-04-26T16:33:56.341303+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:33:56.341303+0000","last_undegraded":"2024-04-26T16:33:56.341303+0000","last_fullsized":"2024-04-26T16:33:56.341303+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.416837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.416837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.327845+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1136,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.344949+0000","last_change":"2024-04-26T16:33:02.301995+0000","last_active":"2024-04-26T16:33:56.344949+0000","last_peered":"2024-04-26T16:33:56.344949+0000","last_clean":"2024-04-26T16:33:01.286780+0000","last_became_active":"2024-04-26T16:33:02.301995+0000","last_became_peered":"2024-04-26T16:33:02.301995+0000","last_unstale":"2024-04-26T16:33:56.344949+0000","last_undegraded":"2024-04-26T16:33:56.344949+0000","last_fullsized":"2024-04-26T16:33:02.291530+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:38.932710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:38.932710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:06.162969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1264,"reported_epoch":684,"state":"active+clean+scrubbing","last_fresh":"2024-04-26T16:33:56.372140+0000","last_change":"2024-04-26T16:33:56.519736+0000","last_active":"2024-04-26T16:33:56.372140+0000","last_peered":"2024-04-26T16:33:56.372140+0000","last_clean":"2024-04-26T16:33:56.372140+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:33:56.372140+0000","last_undegraded":"2024-04-26T16:33:56.372140+0000","last_fullsized":"2024-04-26T16:33:56.372140+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:05.253814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:30:05.253814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1559,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.368757+0000","last_change":"2024-04-26T16:33:02.322706+0000","last_active":"2024-04-26T16:33:56.368757+0000","last_peered":"2024-04-26T16:33:56.368757+0000","last_clean":"2024-04-26T16:33:01.292631+0000","last_became_active":"2024-04-26T16:33:02.322706+0000","last_became_peered":"2024-04-26T16:33:02.322706+0000","last_unstale":"2024-04-26T16:33:56.368757+0000","last_undegraded":"2024-04-26T16:33:02.299845+0000","last_fullsized":"2024-04-26T16:33:02.299528+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:31:16.790149+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.790149+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.353993+0000","last_change":"2024-04-26T16:33:37.325806+0000","last_active":"2024-04-26T16:33:56.353993+0000","last_peered":"2024-04-26T16:33:56.353993+0000","last_clean":"2024-04-26T16:33:56.353993+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:33:56.353993+0000","last_undegraded":"2024-04-26T16:33:56.353993+0000","last_fullsized":"2024-04-26T16:33:56.353993+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:37.325601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:37.325601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.121266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"388'25","reported_seq":1348,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.344995+0000","last_change":"2024-04-26T16:33:02.307634+0000","last_active":"2024-04-26T16:33:56.344995+0000","last_peered":"2024-04-26T16:33:56.344995+0000","last_clean":"2024-04-26T16:33:01.286876+0000","last_became_active":"2024-04-26T16:33:02.307634+0000","last_became_peered":"2024-04-26T16:33:02.307634+0000","last_unstale":"2024-04-26T16:33:56.344995+0000","last_undegraded":"2024-04-26T16:33:02.293090+0000","last_fullsized":"2024-04-26T16:33:02.292755+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:32:26.895513+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:32:26.895513+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.356712+0000","last_change":"2024-04-26T16:33:23.365570+0000","last_active":"2024-04-26T16:33:56.356712+0000","last_peered":"2024-04-26T16:33:56.356712+0000","last_clean":"2024-04-26T16:33:56.356712+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:33:56.356712+0000","last_undegraded":"2024-04-26T16:33:56.356712+0000","last_fullsized":"2024-04-26T16:33:56.356712+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:23.365201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:33:23.365201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:31.529486+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1136,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.343711+0000","last_change":"2024-04-26T16:33:02.315902+0000","last_active":"2024-04-26T16:33:56.343711+0000","last_peered":"2024-04-26T16:33:56.343711+0000","last_clean":"2024-04-26T16:33:01.286013+0000","last_became_active":"2024-04-26T16:33:02.315902+0000","last_became_peered":"2024-04-26T16:33:02.315902+0000","last_unstale":"2024-04-26T16:33:56.343711+0000","last_undegraded":"2024-04-26T16:33:56.343711+0000","last_fullsized":"2024-04-26T16:33:02.290039+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:34.090693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:34.090693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1293,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338603+0000","last_change":"2024-04-26T16:33:37.790562+0000","last_active":"2024-04-26T16:33:56.338603+0000","last_peered":"2024-04-26T16:33:56.338603+0000","last_clean":"2024-04-26T16:33:56.338603+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:33:56.338603+0000","last_undegraded":"2024-04-26T16:33:56.338603+0000","last_fullsized":"2024-04-26T16:33:56.338603+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:37.790335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:33:37.790335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:52.849928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1289,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.336353+0000","last_change":"2024-04-26T16:33:34.732441+0000","last_active":"2024-04-26T16:33:56.336353+0000","last_peered":"2024-04-26T16:33:56.336353+0000","last_clean":"2024-04-26T16:33:56.336353+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:33:56.336353+0000","last_undegraded":"2024-04-26T16:33:56.336353+0000","last_fullsized":"2024-04-26T16:33:56.336353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:34.732301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:33:34.732301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:49.246935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'38","reported_seq":1359,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.339422+0000","last_change":"2024-04-26T16:33:02.308957+0000","last_active":"2024-04-26T16:33:56.339422+0000","last_peered":"2024-04-26T16:33:56.339422+0000","last_clean":"2024-04-26T16:33:00.426570+0000","last_became_active":"2024-04-26T16:33:02.308957+0000","last_became_peered":"2024-04-26T16:33:02.308957+0000","last_unstale":"2024-04-26T16:33:56.339422+0000","last_undegraded":"2024-04-26T16:33:02.294203+0000","last_fullsized":"2024-04-26T16:33:02.293611+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_clean_scrub_stamp":"2024-04-26T16:32:43.426520+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:03.897947+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1259,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.370414+0000","last_change":"2024-04-26T16:33:02.322714+0000","last_active":"2024-04-26T16:33:56.370414+0000","last_peered":"2024-04-26T16:33:56.370414+0000","last_clean":"2024-04-26T16:33:01.310812+0000","last_became_active":"2024-04-26T16:33:02.322714+0000","last_became_peered":"2024-04-26T16:33:02.322714+0000","last_unstale":"2024-04-26T16:33:56.370414+0000","last_undegraded":"2024-04-26T16:33:56.370414+0000","last_fullsized":"2024-04-26T16:33:02.305830+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.021548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.021548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:57.505320+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1112,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.345630+0000","last_change":"2024-04-26T16:32:24.826314+0000","last_active":"2024-04-26T16:33:56.345630+0000","last_peered":"2024-04-26T16:33:56.345630+0000","last_clean":"2024-04-26T16:33:56.345630+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:33:56.345630+0000","last_undegraded":"2024-04-26T16:33:56.345630+0000","last_fullsized":"2024-04-26T16:33:56.345630+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:24.826176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:24.826176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1279,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.364389+0000","last_change":"2024-04-26T16:32:29.797597+0000","last_active":"2024-04-26T16:33:56.364389+0000","last_peered":"2024-04-26T16:33:56.364389+0000","last_clean":"2024-04-26T16:33:56.364389+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:33:56.364389+0000","last_undegraded":"2024-04-26T16:33:56.364389+0000","last_fullsized":"2024-04-26T16:33:56.364389+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:29.797513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:32:29.797513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'37","reported_seq":1350,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.340554+0000","last_change":"2024-04-26T16:33:02.309201+0000","last_active":"2024-04-26T16:33:56.340554+0000","last_peered":"2024-04-26T16:33:56.340554+0000","last_clean":"2024-04-26T16:33:00.883327+0000","last_became_active":"2024-04-26T16:33:02.309201+0000","last_became_peered":"2024-04-26T16:33:02.309201+0000","last_unstale":"2024-04-26T16:33:56.340554+0000","last_undegraded":"2024-04-26T16:33:02.295666+0000","last_fullsized":"2024-04-26T16:33:02.295319+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.498802+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":206,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.353849+0000","last_change":"2024-04-26T16:33:02.308807+0000","last_active":"2024-04-26T16:33:56.353849+0000","last_peered":"2024-04-26T16:33:56.353849+0000","last_clean":"2024-04-26T16:33:00.425097+0000","last_became_active":"2024-04-26T16:33:02.308807+0000","last_became_peered":"2024-04-26T16:33:02.308807+0000","last_unstale":"2024-04-26T16:33:56.353849+0000","last_undegraded":"2024-04-26T16:33:56.353849+0000","last_fullsized":"2024-04-26T16:33:02.292154+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.095870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.095870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1265,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.347270+0000","last_change":"2024-04-26T16:33:47.201263+0000","last_active":"2024-04-26T16:33:56.347270+0000","last_peered":"2024-04-26T16:33:56.347270+0000","last_clean":"2024-04-26T16:33:56.347270+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:33:56.347270+0000","last_undegraded":"2024-04-26T16:33:56.347270+0000","last_fullsized":"2024-04-26T16:33:56.347270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:47.201163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:47.201163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:10.355966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1128,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.338281+0000","last_change":"2024-04-26T16:33:02.314801+0000","last_active":"2024-04-26T16:33:56.338281+0000","last_peered":"2024-04-26T16:33:56.338281+0000","last_clean":"2024-04-26T16:33:00.432203+0000","last_became_active":"2024-04-26T16:33:02.314801+0000","last_became_peered":"2024-04-26T16:33:02.314801+0000","last_unstale":"2024-04-26T16:33:56.338281+0000","last_undegraded":"2024-04-26T16:33:56.338281+0000","last_fullsized":"2024-04-26T16:33:02.305638+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.533042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.533042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:59.802325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"630'18","reported_seq":1519,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.365365+0000","last_change":"2024-04-26T16:33:02.308373+0000","last_active":"2024-04-26T16:33:56.365365+0000","last_peered":"2024-04-26T16:33:56.365365+0000","last_clean":"2024-04-26T16:33:01.297193+0000","last_became_active":"2024-04-26T16:33:02.308373+0000","last_became_peered":"2024-04-26T16:33:02.308373+0000","last_unstale":"2024-04-26T16:33:56.365365+0000","last_undegraded":"2024-04-26T16:33:02.303866+0000","last_fullsized":"2024-04-26T16:33:02.303632+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'16","last_scrub_stamp":"2024-04-26T16:31:16.715748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.715748+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.791132+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1315,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340185+0000","last_change":"2024-04-26T16:33:46.445082+0000","last_active":"2024-04-26T16:33:56.340185+0000","last_peered":"2024-04-26T16:33:56.340185+0000","last_clean":"2024-04-26T16:33:56.340185+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:33:56.340185+0000","last_undegraded":"2024-04-26T16:33:56.340185+0000","last_fullsized":"2024-04-26T16:33:56.340185+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:33:46.444925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:33:46.444925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:47.401381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"585'40","reported_seq":251,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.356387+0000","last_change":"2024-04-26T16:33:35.343288+0000","last_active":"2024-04-26T16:33:56.356387+0000","last_peered":"2024-04-26T16:33:56.356387+0000","last_clean":"2024-04-26T16:33:56.356387+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:33:56.356387+0000","last_undegraded":"2024-04-26T16:33:56.356387+0000","last_fullsized":"2024-04-26T16:33:56.356387+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'40","last_scrub_stamp":"2024-04-26T16:33:35.343032+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:33:35.343032+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.610222+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1292,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340822+0000","last_change":"2024-04-26T16:33:22.507810+0000","last_active":"2024-04-26T16:33:56.340822+0000","last_peered":"2024-04-26T16:33:56.340822+0000","last_clean":"2024-04-26T16:33:56.340822+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:33:56.340822+0000","last_undegraded":"2024-04-26T16:33:56.340822+0000","last_fullsized":"2024-04-26T16:33:56.340822+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:22.507561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:33:22.507561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:29.250619+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1268,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.362704+0000","last_change":"2024-04-26T16:32:17.757412+0000","last_active":"2024-04-26T16:33:56.362704+0000","last_peered":"2024-04-26T16:33:56.362704+0000","last_clean":"2024-04-26T16:33:56.362704+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:33:56.362704+0000","last_undegraded":"2024-04-26T16:33:56.362704+0000","last_fullsized":"2024-04-26T16:33:56.362704+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:17.757210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:32:17.757210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1130,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.365067+0000","last_change":"2024-04-26T16:33:43.739749+0000","last_active":"2024-04-26T16:33:56.365067+0000","last_peered":"2024-04-26T16:33:56.365067+0000","last_clean":"2024-04-26T16:33:56.365067+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:33:56.365067+0000","last_undegraded":"2024-04-26T16:33:56.365067+0000","last_fullsized":"2024-04-26T16:33:56.365067+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:43.739649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:43.739649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.398957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1281,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.336212+0000","last_change":"2024-04-26T16:33:21.757129+0000","last_active":"2024-04-26T16:33:56.336212+0000","last_peered":"2024-04-26T16:33:56.336212+0000","last_clean":"2024-04-26T16:33:56.336212+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:33:56.336212+0000","last_undegraded":"2024-04-26T16:33:56.336212+0000","last_fullsized":"2024-04-26T16:33:56.336212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:21.756987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:33:21.756987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:47.441073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"388'19","reported_seq":1302,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.371285+0000","last_change":"2024-04-26T16:33:54.462883+0000","last_active":"2024-04-26T16:33:56.371285+0000","last_peered":"2024-04-26T16:33:56.371285+0000","last_clean":"2024-04-26T16:33:56.371285+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:33:56.371285+0000","last_undegraded":"2024-04-26T16:33:56.371285+0000","last_fullsized":"2024-04-26T16:33:56.371285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:33:54.462286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.462286+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.077525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1315,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.342438+0000","last_change":"2024-04-26T16:33:50.235407+0000","last_active":"2024-04-26T16:33:56.342438+0000","last_peered":"2024-04-26T16:33:56.342438+0000","last_clean":"2024-04-26T16:33:56.342438+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:33:56.342438+0000","last_undegraded":"2024-04-26T16:33:56.342438+0000","last_fullsized":"2024-04-26T16:33:56.342438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:33:50.235076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:50.235076+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:17.308186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1257,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.346383+0000","last_change":"2024-04-26T16:33:48.162635+0000","last_active":"2024-04-26T16:33:56.346383+0000","last_peered":"2024-04-26T16:33:56.346383+0000","last_clean":"2024-04-26T16:33:56.346383+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:33:56.346383+0000","last_undegraded":"2024-04-26T16:33:56.346383+0000","last_fullsized":"2024-04-26T16:33:56.346383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.162241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:48.162241+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.162241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.570512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1130,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.362875+0000","last_change":"2024-04-26T16:33:26.686394+0000","last_active":"2024-04-26T16:33:56.362875+0000","last_peered":"2024-04-26T16:33:56.362875+0000","last_clean":"2024-04-26T16:33:56.362875+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:33:56.362875+0000","last_undegraded":"2024-04-26T16:33:56.362875+0000","last_fullsized":"2024-04-26T16:33:56.362875+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:26.686165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:26.686165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.752786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"388'19","reported_seq":1298,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.369079+0000","last_change":"2024-04-26T16:33:02.313583+0000","last_active":"2024-04-26T16:33:56.369079+0000","last_peered":"2024-04-26T16:33:56.369079+0000","last_clean":"2024-04-26T16:33:01.312890+0000","last_became_active":"2024-04-26T16:33:02.313583+0000","last_became_peered":"2024-04-26T16:33:02.313583+0000","last_unstale":"2024-04-26T16:33:56.369079+0000","last_undegraded":"2024-04-26T16:33:02.301033+0000","last_fullsized":"2024-04-26T16:33:02.300902+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:00.045450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.045450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:57.505305+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1289,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.336086+0000","last_change":"2024-04-26T16:33:36.741950+0000","last_active":"2024-04-26T16:33:56.336086+0000","last_peered":"2024-04-26T16:33:56.336086+0000","last_clean":"2024-04-26T16:33:56.336086+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:33:56.336086+0000","last_undegraded":"2024-04-26T16:33:56.336086+0000","last_fullsized":"2024-04-26T16:33:56.336086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:36.741738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:33:36.741738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.880852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1260,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.347743+0000","last_change":"2024-04-26T16:33:40.240142+0000","last_active":"2024-04-26T16:33:56.347743+0000","last_peered":"2024-04-26T16:33:56.347743+0000","last_clean":"2024-04-26T16:33:56.347743+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:33:56.347743+0000","last_undegraded":"2024-04-26T16:33:56.347743+0000","last_fullsized":"2024-04-26T16:33:56.347743+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:40.239897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:33:40.239897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.412941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1277,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338233+0000","last_change":"2024-04-26T16:33:54.807377+0000","last_active":"2024-04-26T16:33:56.338233+0000","last_peered":"2024-04-26T16:33:56.338233+0000","last_clean":"2024-04-26T16:33:56.338233+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:33:56.338233+0000","last_undegraded":"2024-04-26T16:33:56.338233+0000","last_fullsized":"2024-04-26T16:33:56.338233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.807120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:08.419903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1136,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.335676+0000","last_change":"2024-04-26T16:33:02.314995+0000","last_active":"2024-04-26T16:33:56.335676+0000","last_peered":"2024-04-26T16:33:56.335676+0000","last_clean":"2024-04-26T16:33:00.427322+0000","last_became_active":"2024-04-26T16:33:02.314995+0000","last_became_peered":"2024-04-26T16:33:02.314995+0000","last_unstale":"2024-04-26T16:33:56.335676+0000","last_undegraded":"2024-04-26T16:33:56.335676+0000","last_fullsized":"2024-04-26T16:33:02.300713+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:41.617220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:41.617220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:05.422973+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"388'8","reported_seq":883,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.363423+0000","last_change":"2024-04-26T16:33:02.317319+0000","last_active":"2024-04-26T16:33:56.363423+0000","last_peered":"2024-04-26T16:33:56.363423+0000","last_clean":"2024-04-26T16:27:36.832282+0000","last_became_active":"2024-04-26T16:33:02.317319+0000","last_became_peered":"2024-04-26T16:33:02.317319+0000","last_unstale":"2024-04-26T16:33:56.363423+0000","last_undegraded":"2024-04-26T16:33:02.301756+0000","last_fullsized":"2024-04-26T16:33:02.301404+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'8","last_scrub_stamp":"2024-04-26T16:31:45.421499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:45.421499+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.791145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":748,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.340139+0000","last_change":"2024-04-26T16:33:02.312790+0000","last_active":"2024-04-26T16:33:56.340139+0000","last_peered":"2024-04-26T16:33:56.340139+0000","last_clean":"2024-04-26T16:17:27.960068+0000","last_became_active":"2024-04-26T16:33:02.312790+0000","last_became_peered":"2024-04-26T16:33:02.312790+0000","last_unstale":"2024-04-26T16:33:56.340139+0000","last_undegraded":"2024-04-26T16:33:56.340139+0000","last_fullsized":"2024-04-26T16:33:02.294363+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:23.283518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:32:23.283518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.355781+0000","last_change":"2024-04-26T16:32:30.238246+0000","last_active":"2024-04-26T16:33:56.355781+0000","last_peered":"2024-04-26T16:33:56.355781+0000","last_clean":"2024-04-26T16:33:56.355781+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:33:56.355781+0000","last_undegraded":"2024-04-26T16:33:56.355781+0000","last_fullsized":"2024-04-26T16:33:56.355781+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:30.238008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:32:30.238008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1258,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.370345+0000","last_change":"2024-04-26T16:33:02.319994+0000","last_active":"2024-04-26T16:33:56.370345+0000","last_peered":"2024-04-26T16:33:56.370345+0000","last_clean":"2024-04-26T16:33:01.316461+0000","last_became_active":"2024-04-26T16:33:02.319994+0000","last_became_peered":"2024-04-26T16:33:02.319994+0000","last_unstale":"2024-04-26T16:33:56.370345+0000","last_undegraded":"2024-04-26T16:33:56.370345+0000","last_fullsized":"2024-04-26T16:33:02.304876+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.019680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.019680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1132,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339597+0000","last_change":"2024-04-26T16:32:36.138996+0000","last_active":"2024-04-26T16:33:56.339597+0000","last_peered":"2024-04-26T16:33:56.339597+0000","last_clean":"2024-04-26T16:33:56.339597+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:33:56.339597+0000","last_undegraded":"2024-04-26T16:33:56.339597+0000","last_fullsized":"2024-04-26T16:33:56.339597+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:36.138890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:36.138890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'19","reported_seq":1312,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.372316+0000","last_change":"2024-04-26T16:32:29.512249+0000","last_active":"2024-04-26T16:33:56.372316+0000","last_peered":"2024-04-26T16:33:56.372316+0000","last_clean":"2024-04-26T16:33:56.372316+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:33:56.372316+0000","last_undegraded":"2024-04-26T16:33:56.372316+0000","last_fullsized":"2024-04-26T16:33:56.372316+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:32:29.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:29.512093+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.353542+0000","last_change":"2024-04-26T16:33:22.354265+0000","last_active":"2024-04-26T16:33:56.353542+0000","last_peered":"2024-04-26T16:33:56.353542+0000","last_clean":"2024-04-26T16:33:56.353542+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:33:56.353542+0000","last_undegraded":"2024-04-26T16:33:56.353542+0000","last_fullsized":"2024-04-26T16:33:56.353542+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:22.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:22.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:22.614466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":237,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.356087+0000","last_change":"2024-04-26T16:33:39.290376+0000","last_active":"2024-04-26T16:33:56.356087+0000","last_peered":"2024-04-26T16:33:56.356087+0000","last_clean":"2024-04-26T16:33:56.356087+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:33:56.356087+0000","last_undegraded":"2024-04-26T16:33:56.356087+0000","last_fullsized":"2024-04-26T16:33:56.356087+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:33:39.290137+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.290137+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.254553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1274,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.365858+0000","last_change":"2024-04-26T16:33:53.794483+0000","last_active":"2024-04-26T16:33:56.365858+0000","last_peered":"2024-04-26T16:33:56.365858+0000","last_clean":"2024-04-26T16:33:56.365858+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:33:56.365858+0000","last_undegraded":"2024-04-26T16:33:56.365858+0000","last_fullsized":"2024-04-26T16:33:56.365858+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:53.794377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:53.794377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:21.434784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.353205+0000","last_change":"2024-04-26T16:32:28.181782+0000","last_active":"2024-04-26T16:33:56.353205+0000","last_peered":"2024-04-26T16:33:56.353205+0000","last_clean":"2024-04-26T16:33:56.353205+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:33:56.353205+0000","last_undegraded":"2024-04-26T16:33:56.353205+0000","last_fullsized":"2024-04-26T16:33:56.353205+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:28.181572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:28.181572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"388'19","reported_seq":1312,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.363660+0000","last_change":"2024-04-26T16:33:41.802094+0000","last_active":"2024-04-26T16:33:56.363660+0000","last_peered":"2024-04-26T16:33:56.363660+0000","last_clean":"2024-04-26T16:33:56.363660+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:33:56.363660+0000","last_undegraded":"2024-04-26T16:33:56.363660+0000","last_fullsized":"2024-04-26T16:33:56.363660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:33:41.801750+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:33:41.801750+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.839175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1287,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.364787+0000","last_change":"2024-04-26T16:33:40.752964+0000","last_active":"2024-04-26T16:33:56.364787+0000","last_peered":"2024-04-26T16:33:56.364787+0000","last_clean":"2024-04-26T16:33:56.364787+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:33:56.364787+0000","last_undegraded":"2024-04-26T16:33:56.364787+0000","last_fullsized":"2024-04-26T16:33:56.364787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:40.752777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_clean_scrub_stamp":"2024-04-26T16:33:40.752777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.127925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1469,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.347238+0000","last_change":"2024-04-26T16:33:46.206611+0000","last_active":"2024-04-26T16:33:56.347238+0000","last_peered":"2024-04-26T16:33:56.347238+0000","last_clean":"2024-04-26T16:33:56.347238+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:33:56.347238+0000","last_undegraded":"2024-04-26T16:33:56.347238+0000","last_fullsized":"2024-04-26T16:33:56.347238+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:33:46.206497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:46.206497+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:56.209377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1290,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.343330+0000","last_change":"2024-04-26T16:33:54.273720+0000","last_active":"2024-04-26T16:33:56.343330+0000","last_peered":"2024-04-26T16:33:56.343330+0000","last_clean":"2024-04-26T16:33:56.343330+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:33:56.343330+0000","last_undegraded":"2024-04-26T16:33:56.343330+0000","last_fullsized":"2024-04-26T16:33:56.343330+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.273521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.273521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.343130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1110,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.375398+0000","last_change":"2024-04-26T16:31:43.245777+0000","last_active":"2024-04-26T16:33:56.375398+0000","last_peered":"2024-04-26T16:33:56.375398+0000","last_clean":"2024-04-26T16:33:56.375398+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:33:56.375398+0000","last_undegraded":"2024-04-26T16:33:56.375398+0000","last_fullsized":"2024-04-26T16:33:56.375398+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:43.245562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.245562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"388'19","reported_seq":1291,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.375535+0000","last_change":"2024-04-26T16:32:43.668627+0000","last_active":"2024-04-26T16:33:56.375535+0000","last_peered":"2024-04-26T16:33:56.375535+0000","last_clean":"2024-04-26T16:33:56.375535+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:33:56.375535+0000","last_undegraded":"2024-04-26T16:33:56.375535+0000","last_fullsized":"2024-04-26T16:33:56.375535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:32:43.668373+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:32:43.668373+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:12.043545+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1079,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.346868+0000","last_change":"2024-04-26T16:33:02.311544+0000","last_active":"2024-04-26T16:33:56.346868+0000","last_peered":"2024-04-26T16:33:56.346868+0000","last_clean":"2024-04-26T16:32:03.398676+0000","last_became_active":"2024-04-26T16:33:02.311544+0000","last_became_peered":"2024-04-26T16:33:02.311544+0000","last_unstale":"2024-04-26T16:33:56.346868+0000","last_undegraded":"2024-04-26T16:33:56.346868+0000","last_fullsized":"2024-04-26T16:33:02.293789+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.231313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.231313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172656+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1269,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.346271+0000","last_change":"2024-04-26T16:33:53.175628+0000","last_active":"2024-04-26T16:33:56.346271+0000","last_peered":"2024-04-26T16:33:56.346271+0000","last_clean":"2024-04-26T16:33:56.346271+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:33:56.346271+0000","last_undegraded":"2024-04-26T16:33:56.346271+0000","last_fullsized":"2024-04-26T16:33:56.346271+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:53.175513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:53.175513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:13.618639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1296,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.362711+0000","last_change":"2024-04-26T16:33:33.851329+0000","last_active":"2024-04-26T16:33:56.362711+0000","last_peered":"2024-04-26T16:33:56.362711+0000","last_clean":"2024-04-26T16:33:56.362711+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:33:56.362711+0000","last_undegraded":"2024-04-26T16:33:56.362711+0000","last_fullsized":"2024-04-26T16:33:56.362711+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:33.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:33:33.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:56.630344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":605,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.370896+0000","last_change":"2024-04-26T16:33:02.320021+0000","last_active":"2024-04-26T16:33:56.370896+0000","last_peered":"2024-04-26T16:33:56.370896+0000","last_clean":"2024-04-26T16:23:28.083853+0000","last_became_active":"2024-04-26T16:33:02.320021+0000","last_became_peered":"2024-04-26T16:33:02.320021+0000","last_unstale":"2024-04-26T16:33:56.370896+0000","last_undegraded":"2024-04-26T16:33:56.370896+0000","last_fullsized":"2024-04-26T16:33:02.304132+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:32:46.121113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:59.208798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"388'20","reported_seq":1321,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.341902+0000","last_change":"2024-04-26T16:33:02.308922+0000","last_active":"2024-04-26T16:33:56.341902+0000","last_peered":"2024-04-26T16:33:56.341902+0000","last_clean":"2024-04-26T16:33:00.883291+0000","last_became_active":"2024-04-26T16:33:02.308922+0000","last_became_peered":"2024-04-26T16:33:02.308922+0000","last_unstale":"2024-04-26T16:33:56.341902+0000","last_undegraded":"2024-04-26T16:33:02.296135+0000","last_fullsized":"2024-04-26T16:33:02.296006+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:39.502283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.502283+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":791,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340851+0000","last_change":"2024-04-26T16:32:39.365451+0000","last_active":"2024-04-26T16:33:56.340851+0000","last_peered":"2024-04-26T16:33:56.340851+0000","last_clean":"2024-04-26T16:33:56.340851+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:33:56.340851+0000","last_undegraded":"2024-04-26T16:33:56.340851+0000","last_fullsized":"2024-04-26T16:33:56.340851+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.365213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.365213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:57.401780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1288,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.340070+0000","last_change":"2024-04-26T16:33:38.432035+0000","last_active":"2024-04-26T16:33:56.340070+0000","last_peered":"2024-04-26T16:33:56.340070+0000","last_clean":"2024-04-26T16:33:56.340070+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:33:56.340070+0000","last_undegraded":"2024-04-26T16:33:56.340070+0000","last_fullsized":"2024-04-26T16:33:56.340070+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.431938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.431938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.466782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1269,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.344107+0000","last_change":"2024-04-26T16:32:39.956683+0000","last_active":"2024-04-26T16:33:56.344107+0000","last_peered":"2024-04-26T16:33:56.344107+0000","last_clean":"2024-04-26T16:33:56.344107+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:33:56.344107+0000","last_undegraded":"2024-04-26T16:33:56.344107+0000","last_fullsized":"2024-04-26T16:33:56.344107+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.956573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.956573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1133,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338933+0000","last_change":"2024-04-26T16:33:39.843269+0000","last_active":"2024-04-26T16:33:56.338933+0000","last_peered":"2024-04-26T16:33:56.338933+0000","last_clean":"2024-04-26T16:33:56.338933+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:33:56.338933+0000","last_undegraded":"2024-04-26T16:33:56.338933+0000","last_fullsized":"2024-04-26T16:33:56.338933+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.842910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.842910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.244346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'12","reported_seq":1337,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.364852+0000","last_change":"2024-04-26T16:32:41.794259+0000","last_active":"2024-04-26T16:33:56.364852+0000","last_peered":"2024-04-26T16:33:56.364852+0000","last_clean":"2024-04-26T16:33:56.364852+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:33:56.364852+0000","last_undegraded":"2024-04-26T16:33:56.364852+0000","last_fullsized":"2024-04-26T16:33:56.364852+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'12","last_scrub_stamp":"2024-04-26T16:32:41.794109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:41.794109+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":206,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.356089+0000","last_change":"2024-04-26T16:33:02.310872+0000","last_active":"2024-04-26T16:33:56.356089+0000","last_peered":"2024-04-26T16:33:56.356089+0000","last_clean":"2024-04-26T16:33:00.427073+0000","last_became_active":"2024-04-26T16:33:02.310872+0000","last_became_peered":"2024-04-26T16:33:02.310872+0000","last_unstale":"2024-04-26T16:33:56.356089+0000","last_undegraded":"2024-04-26T16:33:56.356089+0000","last_fullsized":"2024-04-26T16:33:02.291591+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:36.171451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:36.171451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:00.045717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":796,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.338765+0000","last_change":"2024-04-26T16:33:02.314347+0000","last_active":"2024-04-26T16:33:56.338765+0000","last_peered":"2024-04-26T16:33:56.338765+0000","last_clean":"2024-04-26T16:17:32.003751+0000","last_became_active":"2024-04-26T16:33:02.314347+0000","last_became_peered":"2024-04-26T16:33:02.314347+0000","last_unstale":"2024-04-26T16:33:56.338765+0000","last_undegraded":"2024-04-26T16:33:56.338765+0000","last_fullsized":"2024-04-26T16:33:02.294198+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:21.314021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:32:21.314021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1247,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.371599+0000","last_change":"2024-04-26T16:30:34.003108+0000","last_active":"2024-04-26T16:33:56.371599+0000","last_peered":"2024-04-26T16:33:56.371599+0000","last_clean":"2024-04-26T16:33:56.371599+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:33:56.371599+0000","last_undegraded":"2024-04-26T16:33:56.371599+0000","last_fullsized":"2024-04-26T16:33:56.371599+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:34.002980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:34.002980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1123,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.371557+0000","last_change":"2024-04-26T16:33:02.323544+0000","last_active":"2024-04-26T16:33:56.371557+0000","last_peered":"2024-04-26T16:33:56.371557+0000","last_clean":"2024-04-26T16:33:01.306833+0000","last_became_active":"2024-04-26T16:33:02.323544+0000","last_became_peered":"2024-04-26T16:33:02.323544+0000","last_unstale":"2024-04-26T16:33:56.371557+0000","last_undegraded":"2024-04-26T16:33:56.371557+0000","last_fullsized":"2024-04-26T16:33:02.308613+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.702030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:04.130614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'15","reported_seq":221,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.355843+0000","last_change":"2024-04-26T16:32:22.159040+0000","last_active":"2024-04-26T16:33:56.355843+0000","last_peered":"2024-04-26T16:33:56.355843+0000","last_clean":"2024-04-26T16:33:56.355843+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:33:56.355843+0000","last_undegraded":"2024-04-26T16:33:56.355843+0000","last_fullsized":"2024-04-26T16:33:56.355843+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:32:22.158736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:22.158736+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1088,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.355751+0000","last_change":"2024-04-26T16:33:02.311865+0000","last_active":"2024-04-26T16:33:56.355751+0000","last_peered":"2024-04-26T16:33:56.355751+0000","last_clean":"2024-04-26T16:32:03.400197+0000","last_became_active":"2024-04-26T16:33:02.311865+0000","last_became_peered":"2024-04-26T16:33:02.311865+0000","last_unstale":"2024-04-26T16:33:56.355751+0000","last_undegraded":"2024-04-26T16:33:56.355751+0000","last_fullsized":"2024-04-26T16:33:02.291881+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:38.199629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:32:38.199629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1263,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.369710+0000","last_change":"2024-04-26T16:33:02.315798+0000","last_active":"2024-04-26T16:33:56.369710+0000","last_peered":"2024-04-26T16:33:56.369710+0000","last_clean":"2024-04-26T16:33:01.295761+0000","last_became_active":"2024-04-26T16:33:02.315798+0000","last_became_peered":"2024-04-26T16:33:02.315798+0000","last_unstale":"2024-04-26T16:33:56.369710+0000","last_undegraded":"2024-04-26T16:33:56.369710+0000","last_fullsized":"2024-04-26T16:33:02.301792+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:57.063594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:57.063594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:57.505340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1261,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.346714+0000","last_change":"2024-04-26T16:33:02.318857+0000","last_active":"2024-04-26T16:33:56.346714+0000","last_peered":"2024-04-26T16:33:56.346714+0000","last_clean":"2024-04-26T16:33:01.289540+0000","last_became_active":"2024-04-26T16:33:02.318857+0000","last_became_peered":"2024-04-26T16:33:02.318857+0000","last_unstale":"2024-04-26T16:33:56.346714+0000","last_undegraded":"2024-04-26T16:33:56.346714+0000","last_fullsized":"2024-04-26T16:33:02.299149+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.970505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.970505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1124,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.336696+0000","last_change":"2024-04-26T16:32:39.612584+0000","last_active":"2024-04-26T16:33:56.336696+0000","last_peered":"2024-04-26T16:33:56.336696+0000","last_clean":"2024-04-26T16:33:56.336696+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:33:56.336696+0000","last_undegraded":"2024-04-26T16:33:56.336696+0000","last_fullsized":"2024-04-26T16:33:56.336696+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.612345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.612345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:09.098521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"388'34","reported_seq":1337,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339527+0000","last_change":"2024-04-26T16:33:25.442084+0000","last_active":"2024-04-26T16:33:56.339527+0000","last_peered":"2024-04-26T16:33:56.339527+0000","last_clean":"2024-04-26T16:33:56.339527+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:33:56.339527+0000","last_undegraded":"2024-04-26T16:33:56.339527+0000","last_fullsized":"2024-04-26T16:33:56.339527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:33:25.441825+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:33:25.441825+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.910885+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.352437+0000","last_change":"2024-04-26T16:32:42.165983+0000","last_active":"2024-04-26T16:33:56.352437+0000","last_peered":"2024-04-26T16:33:56.352437+0000","last_clean":"2024-04-26T16:33:56.352437+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:33:56.352437+0000","last_undegraded":"2024-04-26T16:33:56.352437+0000","last_fullsized":"2024-04-26T16:33:56.352437+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:42.165747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:32:42.165747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"630'15","reported_seq":877,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.343042+0000","last_change":"2024-04-26T16:33:02.315178+0000","last_active":"2024-04-26T16:33:56.343042+0000","last_peered":"2024-04-26T16:33:56.343042+0000","last_clean":"2024-04-26T16:26:37.835971+0000","last_became_active":"2024-04-26T16:33:02.315178+0000","last_became_peered":"2024-04-26T16:33:02.315178+0000","last_unstale":"2024-04-26T16:33:56.343042+0000","last_undegraded":"2024-04-26T16:33:02.300874+0000","last_fullsized":"2024-04-26T16:33:02.300519+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:32:32.298132+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:32:32.298132+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":1276,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.374369+0000","last_change":"2024-04-26T16:33:50.501731+0000","last_active":"2024-04-26T16:33:56.374369+0000","last_peered":"2024-04-26T16:33:56.374369+0000","last_clean":"2024-04-26T16:33:56.374369+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:33:56.374369+0000","last_undegraded":"2024-04-26T16:33:56.374369+0000","last_fullsized":"2024-04-26T16:33:56.374369+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:33:50.501577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:50.501577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:09.016490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1134,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.364287+0000","last_change":"2024-04-26T16:33:02.322776+0000","last_active":"2024-04-26T16:33:56.364287+0000","last_peered":"2024-04-26T16:33:56.364287+0000","last_clean":"2024-04-26T16:33:01.293240+0000","last_became_active":"2024-04-26T16:33:02.322776+0000","last_became_peered":"2024-04-26T16:33:02.322776+0000","last_unstale":"2024-04-26T16:33:56.364287+0000","last_undegraded":"2024-04-26T16:33:56.364287+0000","last_fullsized":"2024-04-26T16:33:02.310024+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:23.729621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:23.729621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"388'15","reported_seq":1289,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.343999+0000","last_change":"2024-04-26T16:32:25.865774+0000","last_active":"2024-04-26T16:33:56.343999+0000","last_peered":"2024-04-26T16:33:56.343999+0000","last_clean":"2024-04-26T16:33:56.343999+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:33:56.343999+0000","last_undegraded":"2024-04-26T16:33:56.343999+0000","last_fullsized":"2024-04-26T16:33:56.343999+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:32:25.865394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:25.865394+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1274,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.344594+0000","last_change":"2024-04-26T16:33:02.319315+0000","last_active":"2024-04-26T16:33:56.344594+0000","last_peered":"2024-04-26T16:33:56.344594+0000","last_clean":"2024-04-26T16:33:01.286597+0000","last_became_active":"2024-04-26T16:33:02.319315+0000","last_became_peered":"2024-04-26T16:33:02.319315+0000","last_unstale":"2024-04-26T16:33:56.344594+0000","last_undegraded":"2024-04-26T16:33:02.291067+0000","last_fullsized":"2024-04-26T16:33:02.290734+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:31:17.080989+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:31:17.080989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1280,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.365576+0000","last_change":"2024-04-26T16:32:24.699798+0000","last_active":"2024-04-26T16:33:56.365576+0000","last_peered":"2024-04-26T16:33:56.365576+0000","last_clean":"2024-04-26T16:33:56.365576+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:33:56.365576+0000","last_undegraded":"2024-04-26T16:33:56.365576+0000","last_fullsized":"2024-04-26T16:33:56.365576+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:24.699486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:32:24.699486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1291,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341447+0000","last_change":"2024-04-26T16:33:27.386095+0000","last_active":"2024-04-26T16:33:56.341447+0000","last_peered":"2024-04-26T16:33:56.341447+0000","last_clean":"2024-04-26T16:33:56.341447+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:33:56.341447+0000","last_undegraded":"2024-04-26T16:33:56.341447+0000","last_fullsized":"2024-04-26T16:33:56.341447+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:27.385948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:27.385948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:27.578267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.353871+0000","last_change":"2024-04-26T16:33:48.193084+0000","last_active":"2024-04-26T16:33:56.353871+0000","last_peered":"2024-04-26T16:33:56.353871+0000","last_clean":"2024-04-26T16:33:56.353871+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:33:56.353871+0000","last_undegraded":"2024-04-26T16:33:56.353871+0000","last_fullsized":"2024-04-26T16:33:56.353871+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.192976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.192976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:16.040593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"388'9","reported_seq":1297,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.365282+0000","last_change":"2024-04-26T16:33:34.838916+0000","last_active":"2024-04-26T16:33:56.365282+0000","last_peered":"2024-04-26T16:33:56.365282+0000","last_clean":"2024-04-26T16:33:56.365282+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:33:56.365282+0000","last_undegraded":"2024-04-26T16:33:56.365282+0000","last_fullsized":"2024-04-26T16:33:56.365282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:33:34.838685+0000","last_deep_scrub":"388'9","last_deep_scrub_stamp":"2024-04-26T16:31:34.857307+0000","last_clean_scrub_stamp":"2024-04-26T16:33:34.838685+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:46.606726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":820,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.340237+0000","last_change":"2024-04-26T16:33:02.322246+0000","last_active":"2024-04-26T16:33:56.340237+0000","last_peered":"2024-04-26T16:33:56.340237+0000","last_clean":"2024-04-26T16:17:27.960068+0000","last_became_active":"2024-04-26T16:33:02.322246+0000","last_became_peered":"2024-04-26T16:33:02.322246+0000","last_unstale":"2024-04-26T16:33:56.340237+0000","last_undegraded":"2024-04-26T16:33:56.340237+0000","last_fullsized":"2024-04-26T16:33:02.302200+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:43.402207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.402207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1283,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.341404+0000","last_change":"2024-04-26T16:33:21.547155+0000","last_active":"2024-04-26T16:33:56.341404+0000","last_peered":"2024-04-26T16:33:56.341404+0000","last_clean":"2024-04-26T16:33:56.341404+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:33:56.341404+0000","last_undegraded":"2024-04-26T16:33:56.341404+0000","last_fullsized":"2024-04-26T16:33:56.341404+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:21.547000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:21.547000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:40.287217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1304,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.343163+0000","last_change":"2024-04-26T16:33:02.314889+0000","last_active":"2024-04-26T16:33:56.343163+0000","last_peered":"2024-04-26T16:33:56.343163+0000","last_clean":"2024-04-26T16:33:00.884617+0000","last_became_active":"2024-04-26T16:33:02.314889+0000","last_became_peered":"2024-04-26T16:33:02.314889+0000","last_unstale":"2024-04-26T16:33:56.343163+0000","last_undegraded":"2024-04-26T16:33:56.343163+0000","last_fullsized":"2024-04-26T16:33:02.299849+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:46.169198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:32:46.169198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:04.220145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1110,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.368858+0000","last_change":"2024-04-26T16:32:23.483970+0000","last_active":"2024-04-26T16:33:56.368858+0000","last_peered":"2024-04-26T16:33:56.368858+0000","last_clean":"2024-04-26T16:33:56.368858+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:33:56.368858+0000","last_undegraded":"2024-04-26T16:33:56.368858+0000","last_fullsized":"2024-04-26T16:33:56.368858+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:23.483751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:32:23.483751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'15","reported_seq":1311,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339514+0000","last_change":"2024-04-26T16:32:37.100484+0000","last_active":"2024-04-26T16:33:56.339514+0000","last_peered":"2024-04-26T16:33:56.339514+0000","last_clean":"2024-04-26T16:33:56.339514+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:33:56.339514+0000","last_undegraded":"2024-04-26T16:33:56.339514+0000","last_fullsized":"2024-04-26T16:33:56.339514+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:32:37.100368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:37.100368+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:07.027299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1272,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.345789+0000","last_change":"2024-04-26T16:33:02.312980+0000","last_active":"2024-04-26T16:33:56.345789+0000","last_peered":"2024-04-26T16:33:56.345789+0000","last_clean":"2024-04-26T16:33:01.288488+0000","last_became_active":"2024-04-26T16:33:02.312980+0000","last_became_peered":"2024-04-26T16:33:02.312980+0000","last_unstale":"2024-04-26T16:33:56.345789+0000","last_undegraded":"2024-04-26T16:33:56.345789+0000","last_fullsized":"2024-04-26T16:33:02.297277+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:18.050865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:18.050865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1282,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.342245+0000","last_change":"2024-04-26T16:33:31.257232+0000","last_active":"2024-04-26T16:33:56.342245+0000","last_peered":"2024-04-26T16:33:56.342245+0000","last_clean":"2024-04-26T16:33:56.342245+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:33:56.342245+0000","last_undegraded":"2024-04-26T16:33:56.342245+0000","last_fullsized":"2024-04-26T16:33:56.342245+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:31.257048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:31.257048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.103036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1273,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.346652+0000","last_change":"2024-04-26T16:33:02.314710+0000","last_active":"2024-04-26T16:33:56.346652+0000","last_peered":"2024-04-26T16:33:56.346652+0000","last_clean":"2024-04-26T16:33:01.292655+0000","last_became_active":"2024-04-26T16:33:02.314710+0000","last_became_peered":"2024-04-26T16:33:02.314710+0000","last_unstale":"2024-04-26T16:33:56.346652+0000","last_undegraded":"2024-04-26T16:33:56.346652+0000","last_fullsized":"2024-04-26T16:33:02.296386+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.949357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.949357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1132,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.338429+0000","last_change":"2024-04-26T16:33:48.754733+0000","last_active":"2024-04-26T16:33:56.338429+0000","last_peered":"2024-04-26T16:33:56.338429+0000","last_clean":"2024-04-26T16:33:56.338429+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:33:56.338429+0000","last_undegraded":"2024-04-26T16:33:56.338429+0000","last_fullsized":"2024-04-26T16:33:56.338429+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.754493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.754493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:14.498272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"388'23","reported_seq":226,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.355737+0000","last_change":"2024-04-26T16:33:02.306522+0000","last_active":"2024-04-26T16:33:56.355737+0000","last_peered":"2024-04-26T16:33:56.355737+0000","last_clean":"2024-04-26T16:33:00.425354+0000","last_became_active":"2024-04-26T16:33:02.306522+0000","last_became_peered":"2024-04-26T16:33:02.306522+0000","last_unstale":"2024-04-26T16:33:56.355737+0000","last_undegraded":"2024-04-26T16:33:02.292286+0000","last_fullsized":"2024-04-26T16:33:02.292114+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:32:27.226032+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:32:27.226032+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1287,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.370132+0000","last_change":"2024-04-26T16:33:21.559562+0000","last_active":"2024-04-26T16:33:56.370132+0000","last_peered":"2024-04-26T16:33:56.370132+0000","last_clean":"2024-04-26T16:33:56.370132+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:33:56.370132+0000","last_undegraded":"2024-04-26T16:33:56.370132+0000","last_fullsized":"2024-04-26T16:33:56.370132+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:21.559461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:21.559461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:39.361798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":203,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.353705+0000","last_change":"2024-04-26T16:32:09.479571+0000","last_active":"2024-04-26T16:33:56.353705+0000","last_peered":"2024-04-26T16:33:56.353705+0000","last_clean":"2024-04-26T16:33:56.353705+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:33:56.353705+0000","last_undegraded":"2024-04-26T16:33:56.353705+0000","last_fullsized":"2024-04-26T16:33:56.353705+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:37.787531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:37.787531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1274,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.362815+0000","last_change":"2024-04-26T16:33:39.721652+0000","last_active":"2024-04-26T16:33:56.362815+0000","last_peered":"2024-04-26T16:33:56.362815+0000","last_clean":"2024-04-26T16:33:56.362815+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:33:56.362815+0000","last_undegraded":"2024-04-26T16:33:56.362815+0000","last_fullsized":"2024-04-26T16:33:56.362815+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.721526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.721526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.845743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":629,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.341044+0000","last_change":"2024-04-26T16:33:02.311079+0000","last_active":"2024-04-26T16:33:56.341044+0000","last_peered":"2024-04-26T16:33:56.341044+0000","last_clean":"2024-04-26T16:23:28.083853+0000","last_became_active":"2024-04-26T16:33:02.311079+0000","last_became_peered":"2024-04-26T16:33:02.311079+0000","last_unstale":"2024-04-26T16:33:56.341044+0000","last_undegraded":"2024-04-26T16:33:56.341044+0000","last_fullsized":"2024-04-26T16:33:02.295710+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:29.174775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:29.174775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"388'27","reported_seq":227,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.354237+0000","last_change":"2024-04-26T16:33:26.326370+0000","last_active":"2024-04-26T16:33:56.354237+0000","last_peered":"2024-04-26T16:33:56.354237+0000","last_clean":"2024-04-26T16:33:56.354237+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:33:56.354237+0000","last_undegraded":"2024-04-26T16:33:56.354237+0000","last_fullsized":"2024-04-26T16:33:56.354237+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:33:26.326257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:33:26.326257+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:28.257424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1278,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.375298+0000","last_change":"2024-04-26T16:33:02.322670+0000","last_active":"2024-04-26T16:33:56.375298+0000","last_peered":"2024-04-26T16:33:56.375298+0000","last_clean":"2024-04-26T16:33:01.309323+0000","last_became_active":"2024-04-26T16:33:02.322670+0000","last_became_peered":"2024-04-26T16:33:02.322670+0000","last_unstale":"2024-04-26T16:33:56.375298+0000","last_undegraded":"2024-04-26T16:33:56.375298+0000","last_fullsized":"2024-04-26T16:33:02.306498+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:13.187197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.187197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1258,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.369917+0000","last_change":"2024-04-26T16:33:55.485027+0000","last_active":"2024-04-26T16:33:56.369917+0000","last_peered":"2024-04-26T16:33:56.369917+0000","last_clean":"2024-04-26T16:33:56.369917+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:33:56.369917+0000","last_undegraded":"2024-04-26T16:33:56.369917+0000","last_fullsized":"2024-04-26T16:33:56.369917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:55.484821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:55.484821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.714695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":215,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.352707+0000","last_change":"2024-04-26T16:33:38.279048+0000","last_active":"2024-04-26T16:33:56.352707+0000","last_peered":"2024-04-26T16:33:56.352707+0000","last_clean":"2024-04-26T16:33:56.352707+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:33:56.352707+0000","last_undegraded":"2024-04-26T16:33:56.352707+0000","last_fullsized":"2024-04-26T16:33:56.352707+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:33:38.278822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.278822+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.457275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1134,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.370572+0000","last_change":"2024-04-26T16:33:02.307065+0000","last_active":"2024-04-26T16:33:56.370572+0000","last_peered":"2024-04-26T16:33:56.370572+0000","last_clean":"2024-04-26T16:33:01.300871+0000","last_became_active":"2024-04-26T16:33:02.307065+0000","last_became_peered":"2024-04-26T16:33:02.307065+0000","last_unstale":"2024-04-26T16:33:56.370572+0000","last_undegraded":"2024-04-26T16:33:56.370572+0000","last_fullsized":"2024-04-26T16:33:02.297497+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.201493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.201493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:57.505349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"388'19","reported_seq":1359,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.364947+0000","last_change":"2024-04-26T16:32:37.831812+0000","last_active":"2024-04-26T16:33:56.364947+0000","last_peered":"2024-04-26T16:33:56.364947+0000","last_clean":"2024-04-26T16:33:56.364947+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:33:56.364947+0000","last_undegraded":"2024-04-26T16:33:56.364947+0000","last_fullsized":"2024-04-26T16:33:56.364947+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:32:37.831532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:37.831532+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1290,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.346952+0000","last_change":"2024-04-26T16:33:41.197285+0000","last_active":"2024-04-26T16:33:56.346952+0000","last_peered":"2024-04-26T16:33:56.346952+0000","last_clean":"2024-04-26T16:33:56.346952+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:33:56.346952+0000","last_undegraded":"2024-04-26T16:33:56.346952+0000","last_fullsized":"2024-04-26T16:33:56.346952+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:41.197189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:41.197189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:45.248727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1295,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339664+0000","last_change":"2024-04-26T16:33:29.374847+0000","last_active":"2024-04-26T16:33:56.339664+0000","last_peered":"2024-04-26T16:33:56.339664+0000","last_clean":"2024-04-26T16:33:56.339664+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:33:56.339664+0000","last_undegraded":"2024-04-26T16:33:56.339664+0000","last_fullsized":"2024-04-26T16:33:56.339664+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:32.268089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1290,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.339031+0000","last_change":"2024-04-26T16:33:28.284959+0000","last_active":"2024-04-26T16:33:56.339031+0000","last_peered":"2024-04-26T16:33:56.339031+0000","last_clean":"2024-04-26T16:33:56.339031+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:33:56.339031+0000","last_undegraded":"2024-04-26T16:33:56.339031+0000","last_fullsized":"2024-04-26T16:33:56.339031+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:28.284744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:33:28.284744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.225161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":211,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.354804+0000","last_change":"2024-04-26T16:33:28.282157+0000","last_active":"2024-04-26T16:33:56.354804+0000","last_peered":"2024-04-26T16:33:56.354804+0000","last_clean":"2024-04-26T16:33:56.354804+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:33:56.354804+0000","last_undegraded":"2024-04-26T16:33:56.354804+0000","last_fullsized":"2024-04-26T16:33:56.354804+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:28.281930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:28.281930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.595110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"388'20","reported_seq":1308,"reported_epoch":684,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:33:56.347631+0000","last_change":"2024-04-26T16:33:02.312610+0000","last_active":"2024-04-26T16:33:56.347631+0000","last_peered":"2024-04-26T16:33:56.347631+0000","last_clean":"2024-04-26T16:33:01.292320+0000","last_became_active":"2024-04-26T16:33:02.312610+0000","last_became_peered":"2024-04-26T16:33:02.312610+0000","last_unstale":"2024-04-26T16:33:56.347631+0000","last_undegraded":"2024-04-26T16:33:02.300907+0000","last_fullsized":"2024-04-26T16:33:02.300567+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:32:19.838946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:19.838946+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1287,"reported_epoch":684,"state":"active+clean","last_fresh":"2024-04-26T16:33:56.364996+0000","last_change":"2024-04-26T16:32:25.705015+0000","last_active":"2024-04-26T16:33:56.364996+0000","last_peered":"2024-04-26T16:33:56.364996+0000","last_clean":"2024-04-26T16:33:56.364996+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:33:56.364996+0000","last_undegraded":"2024-04-26T16:33:56.364996+0000","last_fullsized":"2024-04-26T16:33:56.364996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:25.704662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:32:25.704662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1286,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.338741+0000","last_change":"2024-04-26T16:33:02.310960+0000","last_active":"2024-04-26T16:33:56.338741+0000","last_peered":"2024-04-26T16:33:56.338741+0000","last_clean":"2024-04-26T16:33:00.882233+0000","last_became_active":"2024-04-26T16:33:02.310960+0000","last_became_peered":"2024-04-26T16:33:02.310960+0000","last_unstale":"2024-04-26T16:33:56.338741+0000","last_undegraded":"2024-04-26T16:33:56.338741+0000","last_fullsized":"2024-04-26T16:33:02.295568+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.493412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.493412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1260,"reported_epoch":684,"state":"active+undersized","last_fresh":"2024-04-26T16:33:56.345103+0000","last_change":"2024-04-26T16:33:02.307893+0000","last_active":"2024-04-26T16:33:56.345103+0000","last_peered":"2024-04-26T16:33:56.345103+0000","last_clean":"2024-04-26T16:33:01.286500+0000","last_became_active":"2024-04-26T16:33:02.307893+0000","last_became_peered":"2024-04-26T16:33:02.307893+0000","last_unstale":"2024-04-26T16:33:56.345103+0000","last_undegraded":"2024-04-26T16:33:56.345103+0000","last_fullsized":"2024-04-26T16:33:02.292847+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.027062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.027062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:58.172499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":326,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":84,"num_store_stats":7},{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":59,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":10,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":110592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14,"ondisk_log_size":14,"up":24,"acting":24,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":82,"acting":82,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":85,"acting":85,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":965,"num_read_kb":960,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":88,"acting":88,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":86,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743632,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102896,"kb_used_data":3684,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93642256,"statfs":{"total":95995035648,"available":95889670144,"internally_reserved":0,"allocated":3772416,"data_stored":1855245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":432,"seq":1855425871928,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40780,"kb_used_data":4160,"kb_used_omap":15,"kb_used_meta":36592,"kb_avail":93704372,"statfs":{"total":95995035648,"available":95953276928,"internally_reserved":0,"allocated":4259840,"data_stored":2015739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15525,"internal_metadata":37471067},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575118,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92568,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93652584,"statfs":{"total":95995035648,"available":95900246016,"internally_reserved":0,"allocated":3149824,"data_stored":1252672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738648,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83420,"kb_used_data":3656,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93661732,"statfs":{"total":95995035648,"available":95909613568,"internally_reserved":0,"allocated":3743744,"data_stored":1835710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411587,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96708,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93648444,"statfs":{"total":95995035648,"available":95896006656,"internally_reserved":0,"allocated":3137536,"data_stored":1256034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084536,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102208,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":93642944,"statfs":{"total":95995035648,"available":95890374656,"internally_reserved":0,"allocated":3125248,"data_stored":1243548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":101513672},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855598,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96724,"kb_used_data":3072,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93648428,"statfs":{"total":95995035648,"available":95895990272,"internally_reserved":0,"allocated":3145728,"data_stored":1236112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463140,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66172,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93678980,"statfs":{"total":95995035648,"available":95927275520,"internally_reserved":0,"allocated":3121152,"data_stored":1239462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":16384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":326,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:34:00.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-04-26T16:34:00.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:00 smithi190 bash[23868]: audit 2024-04-26T16:33:59.375791+0000 mgr.y (mgr.24401) 641 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:34:00.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:00 smithi190 bash[23868]: audit 2024-04-26T16:33:59.375791+0000 mgr.y (mgr.24401) 641 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "326.2"}]: dispatch 2024-04-26T16:34:00.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:00 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.510619+0000 osd.7 (osd.7) 371 : cluster [DBG] 326.2 scrub starts 2024-04-26T16:34:00.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:00 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.510619+0000 osd.7 (osd.7) 371 : cluster [DBG] 326.2 scrub starts 2024-04-26T16:34:00.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:00 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.519173+0000 osd.7 (osd.7) 372 : cluster [DBG] 326.2 scrub ok 2024-04-26T16:34:00.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:00 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.519173+0000 osd.7 (osd.7) 372 : cluster [DBG] 326.2 scrub ok 2024-04-26T16:34:00.725 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:00.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:00.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:00.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:00.980 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:00.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:01.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.110 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:01.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.242 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:01.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.368 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:01.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.497 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:01.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.821306+0000 mgr.y (mgr.24401) 642 : cluster [DBG] pgmap v1057: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:34:01.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:01 smithi190 bash[23868]: cluster 2024-04-26T16:33:59.821306+0000 mgr.y (mgr.24401) 642 : cluster [DBG] pgmap v1057: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:34:01.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:01.310907+0000 mon.a (mon.0) 3380 : cluster [WRN] Health check update: Degraded data redundancy: 75/627 objects degraded (11.962%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:34:01.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:01.310907+0000 mon.a (mon.0) 3380 : cluster [WRN] Health check update: Degraded data redundancy: 75/627 objects degraded (11.962%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:34:01.628 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.821306+0000 mgr.y (mgr.24401) 642 : cluster [DBG] pgmap v1057: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:34:01.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[21463]: cluster 2024-04-26T16:33:59.821306+0000 mgr.y (mgr.24401) 642 : cluster [DBG] pgmap v1057: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:34:01.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:01.310907+0000 mon.a (mon.0) 3380 : cluster [WRN] Health check update: Degraded data redundancy: 75/627 objects degraded (11.962%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:34:01.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:01.310907+0000 mon.a (mon.0) 3380 : cluster [WRN] Health check update: Degraded data redundancy: 75/627 objects degraded (11.962%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:34:01.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.821306+0000 mgr.y (mgr.24401) 642 : cluster [DBG] pgmap v1057: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:34:01.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[29234]: cluster 2024-04-26T16:33:59.821306+0000 mgr.y (mgr.24401) 642 : cluster [DBG] pgmap v1057: 204 pgs: 1 active+clean+scrubbing, 52 active+undersized, 19 active+undersized+degraded, 132 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 75/627 objects degraded (11.962%) 2024-04-26T16:34:01.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:01.310907+0000 mon.a (mon.0) 3380 : cluster [WRN] Health check update: Degraded data redundancy: 75/627 objects degraded (11.962%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:34:01.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:01.310907+0000 mon.a (mon.0) 3380 : cluster [WRN] Health check update: Degraded data redundancy: 75/627 objects degraded (11.962%), 19 pgs degraded (PG_DEGRADED) 2024-04-26T16:34:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:01.732 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:01.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:01.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.890 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:01.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:01.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:02.020 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:02.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:02.148 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:02.251 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:02.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:02.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:02.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:02.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:02.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:02.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:02.672 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:02.775 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:02.801 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:02.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:02.930 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:02.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:03.033 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:03.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:03.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:03.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:03.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:03.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:34:03.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:03.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:03.452 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:03.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:03.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:03 smithi190 bash[23868]: cluster 2024-04-26T16:34:01.822770+0000 mgr.y (mgr.24401) 643 : cluster [DBG] pgmap v1058: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:03 smithi190 bash[23868]: cluster 2024-04-26T16:34:01.822770+0000 mgr.y (mgr.24401) 643 : cluster [DBG] pgmap v1058: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:03.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:03 smithi050 bash[21463]: cluster 2024-04-26T16:34:01.822770+0000 mgr.y (mgr.24401) 643 : cluster [DBG] pgmap v1058: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:03.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:03 smithi050 bash[21463]: cluster 2024-04-26T16:34:01.822770+0000 mgr.y (mgr.24401) 643 : cluster [DBG] pgmap v1058: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:03.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:03 smithi050 bash[29234]: cluster 2024-04-26T16:34:01.822770+0000 mgr.y (mgr.24401) 643 : cluster [DBG] pgmap v1058: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:03.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:03 smithi050 bash[29234]: cluster 2024-04-26T16:34:01.822770+0000 mgr.y (mgr.24401) 643 : cluster [DBG] pgmap v1058: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.3 KiB/s wr, 4 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:03.677 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:03.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:03.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:03.838 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:03.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:03.969 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:03.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:04.072 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:04.098 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:04.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:04.229 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:04.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:04.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:04.361 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:04.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:04.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:04.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:04.621 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:04.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:04.737 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:34:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:34:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:34:04.752 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:04.848 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:04.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:04.882 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:04.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:04.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:05.013 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:05.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:05.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:05.145 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:05.249 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:05.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:05.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:05.408 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:05.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:05.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:03.824332+0000 mgr.y (mgr.24401) 644 : cluster [DBG] pgmap v1059: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:03.824332+0000 mgr.y (mgr.24401) 644 : cluster [DBG] pgmap v1059: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:04.331916+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:04.331916+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:05.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:04.332239+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:04.332239+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:04.538926+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[21463]: cluster 2024-04-26T16:34:04.538926+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:03.824332+0000 mgr.y (mgr.24401) 644 : cluster [DBG] pgmap v1059: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:03.824332+0000 mgr.y (mgr.24401) 644 : cluster [DBG] pgmap v1059: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:04.331916+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:04.331916+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:04.332239+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:04.332239+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:04.538926+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2024-04-26T16:34:05.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:05 smithi050 bash[29234]: cluster 2024-04-26T16:34:04.538926+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2024-04-26T16:34:05.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:03.824332+0000 mgr.y (mgr.24401) 644 : cluster [DBG] pgmap v1059: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:03.824332+0000 mgr.y (mgr.24401) 644 : cluster [DBG] pgmap v1059: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:04.331916+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:04.331916+0000 osd.7 (osd.7) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:04.332239+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:04.332239+0000 osd.7 (osd.7) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:04.538926+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2024-04-26T16:34:05.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:05 smithi190 bash[23868]: cluster 2024-04-26T16:34:04.538926+0000 mon.a (mon.0) 3381 : cluster [DBG] osdmap e688: 8 total, 7 up, 8 in 2024-04-26T16:34:05.668 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:05.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:05.796 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:05.863 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:05.863 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:05.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:05.920 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:05.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:06.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:06.052 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:06.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:06.181 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:06.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:06.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:06.422 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:06.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:06.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:06.580 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:06.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:06.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[21463]: cluster 2024-04-26T16:34:06.312517+0000 mon.a (mon.0) 3382 : cluster [WRN] Health check update: Degraded data redundancy: 76/633 objects degraded (12.006%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:06.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[21463]: cluster 2024-04-26T16:34:06.312517+0000 mon.a (mon.0) 3382 : cluster [WRN] Health check update: Degraded data redundancy: 76/633 objects degraded (12.006%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:06.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[21463]: cluster 2024-04-26T16:34:06.319590+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2024-04-26T16:34:06.695 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[21463]: cluster 2024-04-26T16:34:06.319590+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2024-04-26T16:34:06.695 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[29234]: cluster 2024-04-26T16:34:06.312517+0000 mon.a (mon.0) 3382 : cluster [WRN] Health check update: Degraded data redundancy: 76/633 objects degraded (12.006%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:06.695 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[29234]: cluster 2024-04-26T16:34:06.312517+0000 mon.a (mon.0) 3382 : cluster [WRN] Health check update: Degraded data redundancy: 76/633 objects degraded (12.006%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:06.696 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[29234]: cluster 2024-04-26T16:34:06.319590+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2024-04-26T16:34:06.696 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:06 smithi050 bash[29234]: cluster 2024-04-26T16:34:06.319590+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2024-04-26T16:34:06.710 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:06.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:06 smithi190 bash[23868]: cluster 2024-04-26T16:34:06.312517+0000 mon.a (mon.0) 3382 : cluster [WRN] Health check update: Degraded data redundancy: 76/633 objects degraded (12.006%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:06 smithi190 bash[23868]: cluster 2024-04-26T16:34:06.312517+0000 mon.a (mon.0) 3382 : cluster [WRN] Health check update: Degraded data redundancy: 76/633 objects degraded (12.006%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:06.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:06 smithi190 bash[23868]: cluster 2024-04-26T16:34:06.319590+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2024-04-26T16:34:06.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:06 smithi190 bash[23868]: cluster 2024-04-26T16:34:06.319590+0000 mon.a (mon.0) 3383 : cluster [DBG] osdmap e689: 8 total, 7 up, 8 in 2024-04-26T16:34:06.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:06.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:06.973 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:06.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:06.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:07.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:07.104 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:07.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:07.207 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:07.235 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:07.338 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:07.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:07.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:07.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:07.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:34:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:34:07.605 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:07.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:07 smithi050 bash[21463]: cluster 2024-04-26T16:34:05.826048+0000 mgr.y (mgr.24401) 645 : cluster [DBG] pgmap v1061: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:07.617 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:07 smithi050 bash[21463]: cluster 2024-04-26T16:34:05.826048+0000 mgr.y (mgr.24401) 645 : cluster [DBG] pgmap v1061: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:07.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:07 smithi050 bash[29234]: cluster 2024-04-26T16:34:05.826048+0000 mgr.y (mgr.24401) 645 : cluster [DBG] pgmap v1061: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:07.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:07 smithi050 bash[29234]: cluster 2024-04-26T16:34:05.826048+0000 mgr.y (mgr.24401) 645 : cluster [DBG] pgmap v1061: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:07.634 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:07.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:07.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:07.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:07 smithi190 bash[23868]: cluster 2024-04-26T16:34:05.826048+0000 mgr.y (mgr.24401) 645 : cluster [DBG] pgmap v1061: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:07 smithi190 bash[23868]: cluster 2024-04-26T16:34:05.826048+0000 mgr.y (mgr.24401) 645 : cluster [DBG] pgmap v1061: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:07.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:07.895 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:07.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:07.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:08.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:08.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:08.156 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:08.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:08.288 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:08.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:08.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:08.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:08.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:08.656 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:08.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:08 smithi050 bash[21463]: audit 2024-04-26T16:34:07.162669+0000 mgr.y (mgr.24401) 646 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:08.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:08 smithi050 bash[21463]: audit 2024-04-26T16:34:07.162669+0000 mgr.y (mgr.24401) 646 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:08.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:08 smithi050 bash[29234]: audit 2024-04-26T16:34:07.162669+0000 mgr.y (mgr.24401) 646 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:08.665 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:08 smithi050 bash[29234]: audit 2024-04-26T16:34:07.162669+0000 mgr.y (mgr.24401) 646 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:08.677 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:08.780 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:08.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:08 smithi190 bash[23868]: audit 2024-04-26T16:34:07.162669+0000 mgr.y (mgr.24401) 646 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:08 smithi190 bash[23868]: audit 2024-04-26T16:34:07.162669+0000 mgr.y (mgr.24401) 646 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:08.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:08.936 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:08.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:09.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:09.064 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:09.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:09.194 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:09.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:09.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:09.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:09.457 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:09.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:09.560 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:09.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:09.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:09.724 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:09.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:09 smithi190 bash[23868]: cluster 2024-04-26T16:34:07.827629+0000 mgr.y (mgr.24401) 647 : cluster [DBG] pgmap v1063: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:09 smithi190 bash[23868]: cluster 2024-04-26T16:34:07.827629+0000 mgr.y (mgr.24401) 647 : cluster [DBG] pgmap v1063: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:09 smithi190 bash[23868]: cluster 2024-04-26T16:34:08.735240+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:09 smithi190 bash[23868]: cluster 2024-04-26T16:34:08.735240+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:09 smithi190 bash[23868]: cluster 2024-04-26T16:34:08.735521+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:09 smithi190 bash[23868]: cluster 2024-04-26T16:34:08.735521+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:09.827 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[21463]: cluster 2024-04-26T16:34:07.827629+0000 mgr.y (mgr.24401) 647 : cluster [DBG] pgmap v1063: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[21463]: cluster 2024-04-26T16:34:07.827629+0000 mgr.y (mgr.24401) 647 : cluster [DBG] pgmap v1063: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[21463]: cluster 2024-04-26T16:34:08.735240+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[21463]: cluster 2024-04-26T16:34:08.735240+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[21463]: cluster 2024-04-26T16:34:08.735521+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[21463]: cluster 2024-04-26T16:34:08.735521+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:09.839 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[29234]: cluster 2024-04-26T16:34:07.827629+0000 mgr.y (mgr.24401) 647 : cluster [DBG] pgmap v1063: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:09.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[29234]: cluster 2024-04-26T16:34:07.827629+0000 mgr.y (mgr.24401) 647 : cluster [DBG] pgmap v1063: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:09.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[29234]: cluster 2024-04-26T16:34:08.735240+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:09.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[29234]: cluster 2024-04-26T16:34:08.735240+0000 osd.5 (osd.5) 251 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:09.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[29234]: cluster 2024-04-26T16:34:08.735521+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:09.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:09 smithi050 bash[29234]: cluster 2024-04-26T16:34:08.735521+0000 osd.5 (osd.5) 252 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:09.855 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:09.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:09.986 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:09.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:10.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:10.116 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:10.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:10.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:10.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:10.377 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:10.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:10.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:10.508 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:10.611 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:10.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:10.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:10.754 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:10 smithi050 bash[21463]: cluster 2024-04-26T16:34:09.566445+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2024-04-26T16:34:10.755 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:10 smithi050 bash[21463]: cluster 2024-04-26T16:34:09.566445+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2024-04-26T16:34:10.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:10 smithi050 bash[29234]: cluster 2024-04-26T16:34:09.566445+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2024-04-26T16:34:10.755 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:10 smithi050 bash[29234]: cluster 2024-04-26T16:34:09.566445+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2024-04-26T16:34:10.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:10 smithi190 bash[23868]: cluster 2024-04-26T16:34:09.566445+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2024-04-26T16:34:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:10 smithi190 bash[23868]: cluster 2024-04-26T16:34:09.566445+0000 mon.a (mon.0) 3384 : cluster [DBG] osdmap e690: 8 total, 7 up, 8 in 2024-04-26T16:34:10.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:10.902 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:10.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:11.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:11.033 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:11.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:11.167 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:11.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:11.298 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:11.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:11.429 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:11.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:11.560 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:11.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:11.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:11.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:11.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:11 smithi190 bash[23868]: cluster 2024-04-26T16:34:09.829323+0000 mgr.y (mgr.24401) 648 : cluster [DBG] pgmap v1065: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:11.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:11 smithi190 bash[23868]: cluster 2024-04-26T16:34:09.829323+0000 mgr.y (mgr.24401) 648 : cluster [DBG] pgmap v1065: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:11.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:11 smithi190 bash[23868]: cluster 2024-04-26T16:34:11.326109+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2024-04-26T16:34:11.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:11 smithi190 bash[23868]: cluster 2024-04-26T16:34:11.326109+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[29234]: cluster 2024-04-26T16:34:09.829323+0000 mgr.y (mgr.24401) 648 : cluster [DBG] pgmap v1065: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[29234]: cluster 2024-04-26T16:34:09.829323+0000 mgr.y (mgr.24401) 648 : cluster [DBG] pgmap v1065: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[29234]: cluster 2024-04-26T16:34:11.326109+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[29234]: cluster 2024-04-26T16:34:11.326109+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[21463]: cluster 2024-04-26T16:34:09.829323+0000 mgr.y (mgr.24401) 648 : cluster [DBG] pgmap v1065: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[21463]: cluster 2024-04-26T16:34:09.829323+0000 mgr.y (mgr.24401) 648 : cluster [DBG] pgmap v1065: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:11.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[21463]: cluster 2024-04-26T16:34:11.326109+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2024-04-26T16:34:11.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:11 smithi050 bash[21463]: cluster 2024-04-26T16:34:11.326109+0000 mon.a (mon.0) 3385 : cluster [DBG] osdmap e691: 8 total, 7 up, 8 in 2024-04-26T16:34:11.821 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:11.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:11.952 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:11.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:12.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:12.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:12.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:12.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:12.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:12.342 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:12.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:12.470 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:12.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:12.602 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:12.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:12.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:12.733 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:12.836 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:12.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:12.961 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:12.991 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:12.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:13.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:13.125 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:13.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:13.251 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:13.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:13.382 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:13.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[29234]: cluster 2024-04-26T16:34:11.832276+0000 mgr.y (mgr.24401) 649 : cluster [DBG] pgmap v1067: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[29234]: cluster 2024-04-26T16:34:11.832276+0000 mgr.y (mgr.24401) 649 : cluster [DBG] pgmap v1067: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[29234]: audit 2024-04-26T16:34:12.483637+0000 mon.a (mon.0) 3386 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[29234]: audit 2024-04-26T16:34:12.483637+0000 mon.a (mon.0) 3386 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[29234]: audit 2024-04-26T16:34:12.486428+0000 mon.c (mon.2) 731 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[29234]: audit 2024-04-26T16:34:12.486428+0000 mon.c (mon.2) 731 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:13.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[21463]: cluster 2024-04-26T16:34:11.832276+0000 mgr.y (mgr.24401) 649 : cluster [DBG] pgmap v1067: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:13.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[21463]: cluster 2024-04-26T16:34:11.832276+0000 mgr.y (mgr.24401) 649 : cluster [DBG] pgmap v1067: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:13.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[21463]: audit 2024-04-26T16:34:12.483637+0000 mon.a (mon.0) 3386 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:13.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[21463]: audit 2024-04-26T16:34:12.483637+0000 mon.a (mon.0) 3386 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:13.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[21463]: audit 2024-04-26T16:34:12.486428+0000 mon.c (mon.2) 731 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:13.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:13 smithi050 bash[21463]: audit 2024-04-26T16:34:12.486428+0000 mon.c (mon.2) 731 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:13.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:13.614 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:13.642 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:13.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:13.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:13.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:13 smithi190 bash[23868]: cluster 2024-04-26T16:34:11.832276+0000 mgr.y (mgr.24401) 649 : cluster [DBG] pgmap v1067: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:13 smithi190 bash[23868]: cluster 2024-04-26T16:34:11.832276+0000 mgr.y (mgr.24401) 649 : cluster [DBG] pgmap v1067: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:13 smithi190 bash[23868]: audit 2024-04-26T16:34:12.483637+0000 mon.a (mon.0) 3386 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:13 smithi190 bash[23868]: audit 2024-04-26T16:34:12.483637+0000 mon.a (mon.0) 3386 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:13 smithi190 bash[23868]: audit 2024-04-26T16:34:12.486428+0000 mon.c (mon.2) 731 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:13 smithi190 bash[23868]: audit 2024-04-26T16:34:12.486428+0000 mon.c (mon.2) 731 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:13.876 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:13.903 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:13.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:14.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:14.034 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:14.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:14.165 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:14.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:14.295 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:14.398 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:14.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:14.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:14.540 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:34:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:34:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:34:14.554 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:14.657 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:14.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:14.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:14.818 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:14.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:14.949 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:14.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:15.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:15.079 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:15.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:15.210 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:15.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:15.342 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:15.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:15.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:15.575 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: cluster 2024-04-26T16:34:13.833998+0000 mgr.y (mgr.24401) 650 : cluster [DBG] pgmap v1068: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: cluster 2024-04-26T16:34:13.833998+0000 mgr.y (mgr.24401) 650 : cluster [DBG] pgmap v1068: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: cluster 2024-04-26T16:34:14.494732+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: cluster 2024-04-26T16:34:14.494732+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: audit 2024-04-26T16:34:14.548914+0000 mon.c (mon.2) 732 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: audit 2024-04-26T16:34:14.548914+0000 mon.c (mon.2) 732 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.585 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: audit 2024-04-26T16:34:14.549389+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[21463]: audit 2024-04-26T16:34:14.549389+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: cluster 2024-04-26T16:34:13.833998+0000 mgr.y (mgr.24401) 650 : cluster [DBG] pgmap v1068: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: cluster 2024-04-26T16:34:13.833998+0000 mgr.y (mgr.24401) 650 : cluster [DBG] pgmap v1068: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: cluster 2024-04-26T16:34:14.494732+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: cluster 2024-04-26T16:34:14.494732+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: audit 2024-04-26T16:34:14.548914+0000 mon.c (mon.2) 732 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: audit 2024-04-26T16:34:14.548914+0000 mon.c (mon.2) 732 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: audit 2024-04-26T16:34:14.549389+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.586 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:15 smithi050 bash[29234]: audit 2024-04-26T16:34:14.549389+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.600 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:15.703 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:15.732 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: cluster 2024-04-26T16:34:13.833998+0000 mgr.y (mgr.24401) 650 : cluster [DBG] pgmap v1068: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: cluster 2024-04-26T16:34:13.833998+0000 mgr.y (mgr.24401) 650 : cluster [DBG] pgmap v1068: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: cluster 2024-04-26T16:34:14.494732+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: cluster 2024-04-26T16:34:14.494732+0000 mon.a (mon.0) 3387 : cluster [DBG] osdmap e692: 8 total, 7 up, 8 in 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: audit 2024-04-26T16:34:14.548914+0000 mon.c (mon.2) 732 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: audit 2024-04-26T16:34:14.548914+0000 mon.c (mon.2) 732 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: audit 2024-04-26T16:34:14.549389+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:15 smithi190 bash[23868]: audit 2024-04-26T16:34:14.549389+0000 mon.a (mon.0) 3388 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:15.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:15.862 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:15.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:15.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:15.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:16.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:16.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:16.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:16.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:16.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:16.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:16.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:16.514 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:16.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: audit 2024-04-26T16:34:15.491856+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: audit 2024-04-26T16:34:15.491856+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: cluster 2024-04-26T16:34:15.495736+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: cluster 2024-04-26T16:34:15.495736+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: audit 2024-04-26T16:34:15.497096+0000 mon.c (mon.2) 733 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: audit 2024-04-26T16:34:15.497096+0000 mon.c (mon.2) 733 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: audit 2024-04-26T16:34:15.499937+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: audit 2024-04-26T16:34:15.499937+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: cluster 2024-04-26T16:34:16.492414+0000 mon.a (mon.0) 3392 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:16.619 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[29234]: cluster 2024-04-26T16:34:16.492414+0000 mon.a (mon.0) 3392 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: audit 2024-04-26T16:34:15.491856+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: audit 2024-04-26T16:34:15.491856+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: cluster 2024-04-26T16:34:15.495736+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: cluster 2024-04-26T16:34:15.495736+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: audit 2024-04-26T16:34:15.497096+0000 mon.c (mon.2) 733 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: audit 2024-04-26T16:34:15.497096+0000 mon.c (mon.2) 733 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: audit 2024-04-26T16:34:15.499937+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: audit 2024-04-26T16:34:15.499937+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: cluster 2024-04-26T16:34:16.492414+0000 mon.a (mon.0) 3392 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:16.620 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:16 smithi050 bash[21463]: cluster 2024-04-26T16:34:16.492414+0000 mon.a (mon.0) 3392 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:16.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:16.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:16.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: audit 2024-04-26T16:34:15.491856+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: audit 2024-04-26T16:34:15.491856+0000 mon.a (mon.0) 3389 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: cluster 2024-04-26T16:34:15.495736+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: cluster 2024-04-26T16:34:15.495736+0000 mon.a (mon.0) 3390 : cluster [DBG] osdmap e693: 8 total, 7 up, 8 in 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: audit 2024-04-26T16:34:15.497096+0000 mon.c (mon.2) 733 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: audit 2024-04-26T16:34:15.497096+0000 mon.c (mon.2) 733 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: audit 2024-04-26T16:34:15.499937+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: audit 2024-04-26T16:34:15.499937+0000 mon.a (mon.0) 3391 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]: dispatch 2024-04-26T16:34:16.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: cluster 2024-04-26T16:34:16.492414+0000 mon.a (mon.0) 3392 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:16.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:16 smithi190 bash[23868]: cluster 2024-04-26T16:34:16.492414+0000 mon.a (mon.0) 3392 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:16.770 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:16.795 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:34:16.795 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:34:16.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:16.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:16.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:17.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:17.029 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:17.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:17.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:17.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:17.281 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:17.385 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:17.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:17.506 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:34:17.515 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:17.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[23868]: cluster 2024-04-26T16:34:15.835743+0000 mgr.y (mgr.24401) 651 : cluster [DBG] pgmap v1071: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:17.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[23868]: cluster 2024-04-26T16:34:15.835743+0000 mgr.y (mgr.24401) 651 : cluster [DBG] pgmap v1071: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[23868]: audit 2024-04-26T16:34:16.494593+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:34:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[23868]: audit 2024-04-26T16:34:16.494593+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:34:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[23868]: cluster 2024-04-26T16:34:16.498773+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2024-04-26T16:34:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:17 smithi190 bash[23868]: cluster 2024-04-26T16:34:16.498773+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2024-04-26T16:34:17.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:17.669 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:17.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:17.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:17.797 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:17.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[29234]: cluster 2024-04-26T16:34:15.835743+0000 mgr.y (mgr.24401) 651 : cluster [DBG] pgmap v1071: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[29234]: cluster 2024-04-26T16:34:15.835743+0000 mgr.y (mgr.24401) 651 : cluster [DBG] pgmap v1071: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[29234]: audit 2024-04-26T16:34:16.494593+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[29234]: audit 2024-04-26T16:34:16.494593+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[29234]: cluster 2024-04-26T16:34:16.498773+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[29234]: cluster 2024-04-26T16:34:16.498773+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2024-04-26T16:34:17.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[21463]: cluster 2024-04-26T16:34:15.835743+0000 mgr.y (mgr.24401) 651 : cluster [DBG] pgmap v1071: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[21463]: cluster 2024-04-26T16:34:15.835743+0000 mgr.y (mgr.24401) 651 : cluster [DBG] pgmap v1071: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 76/633 objects degraded (12.006%) 2024-04-26T16:34:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[21463]: audit 2024-04-26T16:34:16.494593+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:34:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[21463]: audit 2024-04-26T16:34:16.494593+0000 mon.a (mon.0) 3393 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-116"}]': finished 2024-04-26T16:34:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[21463]: cluster 2024-04-26T16:34:16.498773+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2024-04-26T16:34:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:17 smithi050 bash[21463]: cluster 2024-04-26T16:34:16.498773+0000 mon.a (mon.0) 3394 : cluster [DBG] osdmap e694: 8 total, 7 up, 8 in 2024-04-26T16:34:17.900 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:17.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:17.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:17.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:18.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:18.059 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:18.162 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:18.191 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:18.294 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:18.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:18.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:18.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:18 smithi050 bash[21463]: audit 2024-04-26T16:34:17.169810+0000 mgr.y (mgr.24401) 652 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:18.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:18 smithi050 bash[21463]: audit 2024-04-26T16:34:17.169810+0000 mgr.y (mgr.24401) 652 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:18.558 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:18.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:18.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:18 smithi050 bash[29234]: audit 2024-04-26T16:34:17.169810+0000 mgr.y (mgr.24401) 652 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:18.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:18 smithi050 bash[29234]: audit 2024-04-26T16:34:17.169810+0000 mgr.y (mgr.24401) 652 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:18.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:18 smithi190 bash[23868]: audit 2024-04-26T16:34:17.169810+0000 mgr.y (mgr.24401) 652 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:18 smithi190 bash[23868]: audit 2024-04-26T16:34:17.169810+0000 mgr.y (mgr.24401) 652 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:18.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:18.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:18.978 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:18.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:19.081 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:19.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:19.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:19.242 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:19.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:19.376 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:19.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:19.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:19.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[29234]: cluster 2024-04-26T16:34:17.838539+0000 mgr.y (mgr.24401) 653 : cluster [DBG] pgmap v1073: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[29234]: cluster 2024-04-26T16:34:17.838539+0000 mgr.y (mgr.24401) 653 : cluster [DBG] pgmap v1073: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[29234]: cluster 2024-04-26T16:34:18.510543+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[29234]: cluster 2024-04-26T16:34:18.510543+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:19.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[29234]: cluster 2024-04-26T16:34:18.510609+0000 mon.a (mon.0) 3396 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[29234]: cluster 2024-04-26T16:34:18.510609+0000 mon.a (mon.0) 3396 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[21463]: cluster 2024-04-26T16:34:17.838539+0000 mgr.y (mgr.24401) 653 : cluster [DBG] pgmap v1073: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[21463]: cluster 2024-04-26T16:34:17.838539+0000 mgr.y (mgr.24401) 653 : cluster [DBG] pgmap v1073: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[21463]: cluster 2024-04-26T16:34:18.510543+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[21463]: cluster 2024-04-26T16:34:18.510543+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[21463]: cluster 2024-04-26T16:34:18.510609+0000 mon.a (mon.0) 3396 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:19.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:19 smithi050 bash[21463]: cluster 2024-04-26T16:34:18.510609+0000 mon.a (mon.0) 3396 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:19.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:19.659 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8853 ms) 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6002 ms) 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5017 ms) 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25161 ms) 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14194 ms) 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-04-26T16:34:19.660 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (11662 ms) 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 00000010 69 73 |is| 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 00000012 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12035 ms) 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: require_osd_release = squid 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6074 ms) 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (554348 ms total) 2024-04-26T16:34:19.661 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1023 ms) 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (10978 ms) 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18004 ms) 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (3 ms) 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30008 ms total) 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 2024-04-26T16:34:19.662 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6959 ms) 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8001 ms) 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: waiting for scrub... 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: done waiting 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 19 more times... 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 18 more times... 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 17 more times... 2024-04-26T16:34:19.663 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:34:19.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:19.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:19 smithi190 bash[23868]: cluster 2024-04-26T16:34:17.838539+0000 mgr.y (mgr.24401) 653 : cluster [DBG] pgmap v1073: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:19.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:19 smithi190 bash[23868]: cluster 2024-04-26T16:34:17.838539+0000 mgr.y (mgr.24401) 653 : cluster [DBG] pgmap v1073: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:19.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:19 smithi190 bash[23868]: cluster 2024-04-26T16:34:18.510543+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:19.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:19 smithi190 bash[23868]: cluster 2024-04-26T16:34:18.510543+0000 mon.a (mon.0) 3395 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:19.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:19 smithi190 bash[23868]: cluster 2024-04-26T16:34:18.510609+0000 mon.a (mon.0) 3396 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:19.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:19 smithi190 bash[23868]: cluster 2024-04-26T16:34:18.510609+0000 mon.a (mon.0) 3396 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:19.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:19.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:19.897 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:19.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:20.000 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:20.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:20.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.159 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:20.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.289 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:20.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-04-26T16:34:20.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.419 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:20.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:20.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:20.680 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/269238694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e006bb60 msgr2=0x7f48e006bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/269238694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e006bb60 0x7f48e006bfe0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7f48c80098f0 tx=0x7f48c8009c00 comp rx=0 tx=0).stop 2024-04-26T16:34:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/269238694 shutdown_connections 2024-04-26T16:34:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/269238694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48e006c520 0x7f48e0110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:20.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/269238694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e01095c0 0x7f48e01099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/269238694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e006bb60 0x7f48e006bfe0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/269238694 >> 172.21.15.50:0/269238694 conn(0x7f48e0079120 msgr2=0x7f48e0079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/269238694 shutdown_connections 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/269238694 wait complete. 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 librados: starting msgr at 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 librados: starting objecter 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 Processor -- start 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- start start 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 librados: setting wanted keys 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 librados: calling monclient init 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 0x7f48e01991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48e006c520 0x7f48e01996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e01095c0 0x7f48e019da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:20.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f48e0112bf0 con 0x7f48e006bb60 2024-04-26T16:34:20.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f48e0112a70 con 0x7f48e006c520 2024-04-26T16:34:20.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f48e0112d70 con 0x7f48e01095c0 2024-04-26T16:34:20.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 0x7f48e01991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:20.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e5245700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e01095c0 0x7f48e019da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:20.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 0x7f48e01991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38472/0 (socket says 172.21.15.50:38472) 2024-04-26T16:34:20.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 -- 172.21.15.50:0/2750639425 learned_addr learned my addr 172.21.15.50:0/2750639425 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:34:20.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:20.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48dffff700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48e006c520 0x7f48e01996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:20.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 -- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e01095c0 msgr2=0x7f48e019da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e01095c0 0x7f48e019da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 -- 172.21.15.50:0/2750639425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48e006c520 msgr2=0x7f48e01996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48e006c520 0x7f48e01996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 -- 172.21.15.50:0/2750639425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48e019e1c0 con 0x7f48e006bb60 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e4a44700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 0x7f48e01991b0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f48d000e3c0 tx=0x7f48d000e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48ddffb700 10 client.?.objecter ms_handle_connect 0x7f48e006bb60 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48ddffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d00090d0 con 0x7f48e006bb60 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f48d0009270 con 0x7f48e006bb60 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48e019e450 con 0x7f48e006bb60 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d00044c0 con 0x7f48e006bb60 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 10 client.30921.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:20.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48e01a5d30 con 0x7f48e006bb60 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 librados: init done 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.785+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48e006a110 con 0x7f48e006bb60 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f48d001b030 con 0x7f48e006bb60 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f48cc071f80 0x7f48cc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(694..694 src has 577..694) ==== 9851+0+0 (secure 0 0 0) 0x7f48d0091660 con 0x7f48e006bb60 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48dffff700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f48cc071f80 0x7f48cc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 10 client.30921.objecter ms_dispatch 0x7f48e0001550 osd_map(694..694 src has 577..694) 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 3 client.30921.objecter handle_osd_map got epochs [694,694] > 0 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 3 client.30921.objecter handle_osd_map decoding full epoch 694 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 20 client.30921.objecter dump_active .. 0 homeless 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48ddffb700 10 client.30921.objecter ms_handle_connect 0x7f48cc071f80 2024-04-26T16:34:20.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.789+0000 7f48dffff700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f48cc071f80 0x7f48cc074440 secure :-1 s=READY pgs=3814 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009810 tx=0x7f48c8019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:20.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:20.793+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f48d0022060 con 0x7f48e006bb60 2024-04-26T16:34:20.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:20.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:20.928 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:20.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:21.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.021+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"} v 0) -- 0x7f48e01a5f00 con 0x7f48e006bb60 2024-04-26T16:34:21.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:21.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:21.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:21.177 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/2664980382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 msgr2=0x7f149410f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- 172.21.15.50:0/2664980382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 0x7f149410f1d0 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f148c00b0a0 tx=0x7f148c00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/2664980382 shutdown_connections 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- 172.21.15.50:0/2664980382 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 0x7f149410f1d0 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- 172.21.15.50:0/2664980382 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494108390 0x7f1494108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- 172.21.15.50:0/2664980382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1494102320 0x7f14941027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/2664980382 >> 172.21.15.50:0/2664980382 conn(0x7f14940792d0 msgr2=0x7f14940796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/2664980382 shutdown_connections 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/2664980382 wait complete. 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 librados: starting msgr at 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 librados: starting objecter 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 Processor -- start 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- start start 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 librados: setting wanted keys 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 librados: calling monclient init 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1494102320 0x7f1494199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 0x7f1494199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494108390 0x7f149419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f14941119d0 con 0x7f1494108390 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1494111850 con 0x7f1494102320 2024-04-26T16:34:21.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1494111b50 con 0x7f1494102ce0 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494108390 0x7f149419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494108390 0x7f149419d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38496/0 (socket says 172.21.15.50:38496) 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 -- 172.21.15.50:0/1826376641 learned_addr learned my addr 172.21.15.50:0/1826376641 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 0x7f1494199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1494102320 0x7f1494199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 -- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 msgr2=0x7f1494199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 0x7f1494199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 -- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1494102320 msgr2=0x7f1494199120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1494102320 0x7f1494199120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149419e0e0 con 0x7f1494108390 2024-04-26T16:34:21.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1494102ce0 0x7f1494199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1494108390 0x7f149419d9a0 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7f148c00b780 tx=0x7f148c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f148b7fe700 10 client.?.objecter ms_handle_connect 0x7f1494108390 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f148b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c021070 con 0x7f1494108390 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149419e370 con 0x7f1494108390 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f148c007b80 con 0x7f1494108390 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 10 client.30927.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c005040 con 0x7f1494108390 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14941a5c50 con 0x7f1494108390 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 librados: init done 2024-04-26T16:34:21.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.249+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f149406a110 con 0x7f1494108390 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f148c024070 con 0x7f1494108390 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1480071f80 0x7f1480074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(694..694 src has 577..694) ==== 9851+0+0 (secure 0 0 0) 0x7f148c00f020 con 0x7f1494108390 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 osd_map(694..694 src has 577..694) 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 3 client.30927.objecter handle_osd_map got epochs [694,694] > 0 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 3 client.30927.objecter handle_osd_map decoding full epoch 694 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1480071f80 0x7f1480074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 20 client.30927.objecter dump_active .. 0 homeless 2024-04-26T16:34:21.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v694) ==== 54+0+16 (secure 0 0 0) 0x7f148c02b020 con 0x7f1494108390 2024-04-26T16:34:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 10 client.30927.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 1 2024-04-26T16:34:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1494064030 0x7f1494064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 1 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 0 1 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149419a430 osd=0 4 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 10 client.30927.objecter _send_command 1 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14941a8fb0 con 0x7f1494064030 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1480071f80 0x7f1480074440 secure :-1 s=READY pgs=3815 cs=0 l=1 rev1=1 crypto rx=0x7f1490009680 tx=0x7f1490009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f1480071f80 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1494064030 0x7f1494064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f1494064030 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1494064030 0x7f1494064410 crc :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14941a8fb0 con 0x7f1494064030 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 1: 0 ) 2024-04-26T16:34:21.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 10 client.30927.objecter _finish_command 1 = system:0 2024-04-26T16:34:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f149419a430 osd=0 4 2024-04-26T16:34:21.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.253+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 0 1 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 10 client.30927.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 2 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 2 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f149419a430 osd=0 3 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 0 2 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149419a430 osd=0 4 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 10 client.30927.objecter _send_command 2 2024-04-26T16:34:21.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f1494063ab0 con 0x7f1494064030 2024-04-26T16:34:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f1494063ab0 con 0x7f1494064030 2024-04-26T16:34:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.272 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f148b7fe700 10 client.30927.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f149419a430 osd=0 4 2024-04-26T16:34:21.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 0 2 2024-04-26T16:34:21.273 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 10 client.30927.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.273+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f149402d310 0x7f149402d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 1 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149410b7a0 osd=1 4 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 10 client.30927.objecter _send_command 3 2024-04-26T16:34:21.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f149410be20 con 0x7f149402d310 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f149402d310 0x7f149402d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f149402d310 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f149402d310 0x7f149402d710 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f149410be20 con 0x7f149402d310 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 3: 0 ) 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f148b7fe700 10 client.30927.objecter _finish_command 3 = system:0 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f149410b7a0 osd=1 4 2024-04-26T16:34:21.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.277+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 1 3 2024-04-26T16:34:21.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 10 client.30927.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 4 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 4 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f149410b7a0 osd=1 3 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 1 4 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149410b7a0 osd=1 4 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 10 client.30927.objecter _send_command 4 2024-04-26T16:34:21.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f149410b220 con 0x7f149402d310 2024-04-26T16:34:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f149410b220 con 0x7f149402d310 2024-04-26T16:34:21.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f148b7fe700 10 client.30927.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f149410b7a0 osd=1 4 2024-04-26T16:34:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.297+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 1 4 2024-04-26T16:34:21.297 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:34:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 10 client.30927.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 5 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f14941ab170 0x7f14941ad560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 5 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 2 5 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149402e420 osd=2 4 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 10 client.30927.objecter _send_command 5 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f14941b03c0 con 0x7f14941ab170 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f14941ab170 0x7f14941ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f14941ab170 2024-04-26T16:34:21.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f14941ab170 0x7f14941ad560 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14941b03c0 con 0x7f14941ab170 2024-04-26T16:34:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 5: 0 ) 2024-04-26T16:34:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f148b7fe700 10 client.30927.objecter _finish_command 5 = system:0 2024-04-26T16:34:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f149402e420 osd=2 4 2024-04-26T16:34:21.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.301+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 2 5 2024-04-26T16:34:21.301 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:21.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.317+0000 7f48ddffb700 1 -- 172.21.15.50:0/2750639425 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]=0 set pool 1 pgp_num to 1 v695) ==== 129+0+0 (secure 0 0 0) 0x7f48d0010040 con 0x7f48e006bb60 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 10 client.30927.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 6 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 6 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f149402e420 osd=2 3 2024-04-26T16:34:21.321 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 2 6 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f149402e420 osd=2 4 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f149c433700 10 client.30927.objecter _send_command 6 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f149402dea0 con 0x7f14941ab170 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 10 librados: watch_flush enter 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 10 librados: watch_flush exit 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 20 client.30921.objecter shutdown clearing up homeless session... 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 10 client.30921.objecter successfully canceled tick 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f48cc071f80 msgr2=0x7f48cc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f48cc071f80 0x7f48cc074440 secure :-1 s=READY pgs=3814 cs=0 l=1 rev1=1 crypto rx=0x7f48c8009810 tx=0x7f48c8019040 comp rx=0 tx=0).stop 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 msgr2=0x7f48e01991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.321+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 0x7f48e01991b0 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f48d000e3c0 tx=0x7f48d000e6d0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.322 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 shutdown_connections 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f48cc071f80 0x7f48cc074440 unknown :-1 s=CLOSED pgs=3814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f48e01095c0 0x7f48e019da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f48e006bb60 0x7f48e01991b0 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 --2- 172.21.15.50:0/2750639425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48e006c520 0x7f48e01996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 >> 172.21.15.50:0/2750639425 conn(0x7f48e0079120 msgr2=0x7f48e0100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 shutdown_connections 2024-04-26T16:34:21.323 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 -- 172.21.15.50:0/2750639425 wait complete. 2024-04-26T16:34:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.325+0000 7f48e6ca8700 1 librados: shutdown 2024-04-26T16:34:21.324 INFO:teuthology.orchestra.run.smithi050.stderr:set pool 1 pgp_num to 1 2024-04-26T16:34:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f149402dea0 con 0x7f14941ab170 2024-04-26T16:34:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 10 client.30927.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f149402e420 osd=2 4 2024-04-26T16:34:21.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 2 6 2024-04-26T16:34:21.325 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 10 client.30927.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 7 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f14941aec20 0x7f14941af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 7 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 3 7 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a8600 osd=3 4 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 10 client.30927.objecter _send_command 7 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f14941a8d10 con 0x7f14941aec20 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f14941aec20 0x7f14941af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f14941aec20 2024-04-26T16:34:21.327 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f14941aec20 0x7f14941af080 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 -- 172.21.15.50:0/2809789585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 msgr2=0x7f62fc101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- 172.21.15.50:0/2809789585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc101e70 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00faf0 tx=0x7f62ec00fe00 comp rx=0 tx=0).stop 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 -- 172.21.15.50:0/2809789585 shutdown_connections 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- 172.21.15.50:0/2809789585 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 0x7f62fc1102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- 172.21.15.50:0/2809789585 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- 172.21.15.50:0/2809789585 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc101e70 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 -- 172.21.15.50:0/2809789585 >> 172.21.15.50:0/2809789585 conn(0x7f62fc078990 msgr2=0x7f62fc0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 -- 172.21.15.50:0/2809789585 shutdown_connections 2024-04-26T16:34:21.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 -- 172.21.15.50:0/2809789585 wait complete. 2024-04-26T16:34:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14941a8d10 con 0x7f14941aec20 2024-04-26T16:34:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 7: 0 ) 2024-04-26T16:34:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 10 client.30927.objecter _finish_command 7 = system:0 2024-04-26T16:34:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f14941a8600 osd=3 4 2024-04-26T16:34:21.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 3 7 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 librados: starting msgr at 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 librados: starting objecter 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 Processor -- start 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 -- start start 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 librados: setting wanted keys 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 librados: calling monclient init 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 0x7f62fc197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f63044de700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc19b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f6301a79700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 0x7f62fc197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f6301a79700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 0x7f62fc197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.50:53606/0 (socket says 172.21.15.50:53606) 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f6302a7b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc19b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.329+0000 7f630227a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.333+0000 7f6302a7b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc19b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:38522/0 (socket says 172.21.15.50:38522) 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.333+0000 7f6301a79700 1 -- 172.21.15.50:0/874794074 learned_addr learned my addr 172.21.15.50:0/874794074 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.333+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc1047c0 con 0x7f62fc1094b0 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.333+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc104640 con 0x7f62fc1023b0 2024-04-26T16:34:21.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.333+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc104940 con 0x7f62fc101a10 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f6302a7b700 1 -- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 msgr2=0x7f62fc196d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f6302a7b700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc196d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f6302a7b700 1 -- 172.21.15.50:0/874794074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 msgr2=0x7f62fc197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f6302a7b700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 0x7f62fc197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f6302a7b700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc19bd50 con 0x7f62fc1094b0 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f630227a700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:34:21.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.337+0000 7f6302a7b700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc19b610 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f62f400ab40 tx=0x7f62f400ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 10 client.?.objecter ms_handle_connect 0x7f62fc1094b0 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4017020 con 0x7f62fc1094b0 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f62f4004bb0 con 0x7f62fc1094b0 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62fc19c040 con 0x7f62fc1094b0 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f63044de700 10 client.30933.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:21.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f4005020 con 0x7f62fc1094b0 2024-04-26T16:34:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62fc1a38c0 con 0x7f62fc1094b0 2024-04-26T16:34:21.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f63044de700 1 librados: init done 2024-04-26T16:34:21.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f62f400c580 con 0x7f62fc1094b0 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.341+0000 7f62f37fe700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f62e8071fd0 0x7f62e8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(695..695 src has 577..695) ==== 9851+0+0 (secure 0 0 0) 0x7f62f4090fe0 con 0x7f62fc1094b0 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f62f37fe700 10 client.30933.objecter ms_dispatch 0x7f62fc001550 osd_map(695..695 src has 577..695) 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f62f37fe700 3 client.30933.objecter handle_osd_map got epochs [695,695] > 0 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f62f37fe700 3 client.30933.objecter handle_osd_map decoding full epoch 695 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f630227a700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f62e8071fd0 0x7f62e8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f62f37fe700 20 client.30933.objecter dump_active .. 0 homeless 2024-04-26T16:34:21.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f62f37fe700 10 client.30933.objecter ms_handle_connect 0x7f62e8071fd0 2024-04-26T16:34:21.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f630227a700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f62e8071fd0 0x7f62e8074490 secure :-1 s=READY pgs=3816 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009fd0 tx=0x7f62f8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.345+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62e0005120 con 0x7f62fc1094b0 2024-04-26T16:34:21.348 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.349+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f62f4021020 con 0x7f62fc1094b0 2024-04-26T16:34:21.368 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:34:21.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.365+0000 7f149c433700 10 client.30927.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 8 2024-04-26T16:34:21.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 8 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f14941a8600 osd=3 3 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 3 8 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a8600 osd=3 4 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 10 client.30927.objecter _send_command 8 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f14941a8080 con 0x7f14941aec20 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14941a8080 con 0x7f14941aec20 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f148b7fe700 10 client.30927.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f14941a8600 osd=3 4 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 3 8 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 10 client.30927.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 9 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1494103f30 0x7f1494104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 9 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 4 9 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a7e30 osd=4 4 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 10 client.30927.objecter _send_command 9 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.369+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f14941a7fa0 con 0x7f1494103f30 2024-04-26T16:34:21.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1494103f30 0x7f1494104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f149a1cf700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f1494103f30 0x7f1494104390 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f1494103f30 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14941a7fa0 con 0x7f1494103f30 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 9: 0 ) 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f148b7fe700 10 client.30927.objecter _finish_command 9 = system:0 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f14941a7e30 osd=4 4 2024-04-26T16:34:21.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.373+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 4 9 2024-04-26T16:34:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 10 client.30927.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 10 2024-04-26T16:34:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 10 2024-04-26T16:34:21.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f14941a7e30 osd=4 3 2024-04-26T16:34:21.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 4 10 2024-04-26T16:34:21.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14941a7e30 osd=4 4 2024-04-26T16:34:21.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 10 client.30927.objecter _send_command 10 2024-04-26T16:34:21.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.393+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f14940008d0 con 0x7f1494103f30 2024-04-26T16:34:21.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14940008d0 con 0x7f1494103f30 2024-04-26T16:34:21.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 10 client.30927.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f14941a7e30 osd=4 4 2024-04-26T16:34:21.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 4 10 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 10 client.30927.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 11 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1494104de0 0x7f1494105220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.394 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 11 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 5 11 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14940008d0 osd=5 4 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 10 client.30927.objecter _send_command 11 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f14941b6630 con 0x7f1494104de0 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1494104de0 0x7f1494105220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f1494104de0 2024-04-26T16:34:21.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f14999ce700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f1494104de0 0x7f1494105220 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14941b6630 con 0x7f1494104de0 2024-04-26T16:34:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 11: 0 ) 2024-04-26T16:34:21.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 10 client.30927.objecter _finish_command 11 = system:0 2024-04-26T16:34:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f14940008d0 osd=5 4 2024-04-26T16:34:21.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.397+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 5 11 2024-04-26T16:34:21.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 10 client.30927.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 12 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 12 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f14940008d0 osd=5 3 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 5 12 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f14940008d0 osd=5 4 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 10 client.30927.objecter _send_command 12 2024-04-26T16:34:21.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.417+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f14941a81f0 con 0x7f1494104de0 2024-04-26T16:34:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14941a81f0 con 0x7f1494104de0 2024-04-26T16:34:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 10 client.30927.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f14940008d0 osd=5 4 2024-04-26T16:34:21.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 5 12 2024-04-26T16:34:21.419 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:34:21.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 10 client.30927.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 13 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 20 client.30927.objecter _get_session osd=-1 returning homeless 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 10 client.30927.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=695}) -- 0x7f1494105760 con 0x7f1494108390 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14941a81f0 con 0x7f1494108390 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 7 ==== osd_map(695..695 src has 577..695) ==== 658+0+0 (secure 0 0 0) 0x7f148c090740 con 0x7f1494108390 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 osd_map(695..695 src has 577..695) 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 3 client.30927.objecter handle_osd_map got epochs [695,695] > 694 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 3 client.30927.objecter handle_osd_map decoding incremental epoch 695 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 10 client.30927.objecter checking command 13 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 10 client.30927.objecter _check_command_map_dne tid 13 current 695 map_dne_bound 0 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 20 client.30927.objecter dump_active .. 1 homeless 2024-04-26T16:34:21.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== mon.0 v2:172.21.15.50:3300/0 8 ==== mon_get_version_reply(handle=1 version=695) ==== 24+0+0 (secure 0 0 0) 0x7f148c05e960 con 0x7f1494108390 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f14989cc700 10 client.30927.objecter _check_command_map_dne tid 13 current 695 map_dne_bound 695 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f14989cc700 10 client.30927.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f14989cc700 15 client.30927.objecter _session_command_op_remove -1 13 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 10 client.30927.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 14 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f14941bf6b0 0x7f14941c1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 14 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 7 14 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f1494105e30 osd=7 4 2024-04-26T16:34:21.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 10 client.30927.objecter _send_command 14 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f14941c4940 con 0x7f14941bf6b0 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f14941bf6b0 0x7f14941c1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f148b7fe700 10 client.30927.objecter ms_handle_connect 0x7f14941bf6b0 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.421+0000 7f149a9d0700 1 --2- 172.21.15.50:0/1826376641 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f14941bf6b0 0x7f14941c1aa0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.425+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14941c4940 con 0x7f14941bf6b0 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.425+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 14: 0 ) 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.425+0000 7f148b7fe700 10 client.30927.objecter _finish_command 14 = system:0 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.425+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f1494105e30 osd=7 4 2024-04-26T16:34:21.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.425+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 7 14 2024-04-26T16:34:21.433 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:21.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 10 client.30927.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:34:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign -1 15 2024-04-26T16:34:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 20 client.30927.objecter _get_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 15 client.30927.objecter _session_command_op_remove -1 15 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 20 client.30927.objecter get_session s=0x7f1494105e30 osd=7 3 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 15 client.30927.objecter _session_command_op_assign 7 15 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 20 client.30927.objecter put_session s=0x7f1494105e30 osd=7 4 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 10 client.30927.objecter _send_command 15 2024-04-26T16:34:21.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.441+0000 7f149c433700 1 -- 172.21.15.50:0/1826376641 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f14941c1fe0 con 0x7f14941bf6b0 2024-04-26T16:34:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.445+0000 7f148b7fe700 1 -- 172.21.15.50:0/1826376641 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f14941c1fe0 con 0x7f14941bf6b0 2024-04-26T16:34:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.445+0000 7f148b7fe700 10 client.30927.objecter ms_dispatch 0x7f1494001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.445+0000 7f148b7fe700 10 client.30927.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.445+0000 7f148b7fe700 20 client.30927.objecter put_session s=0x7f1494105e30 osd=7 4 2024-04-26T16:34:21.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.445+0000 7f148b7fe700 15 client.30927.objecter _session_command_op_remove 7 15 2024-04-26T16:34:21.442 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:34:21.443 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:34:21.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:21.564 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:21.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:21.601+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f62e00056e0 con 0x7f62fc1094b0 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: cluster 2024-04-26T16:34:19.840236+0000 mgr.y (mgr.24401) 654 : cluster [DBG] pgmap v1074: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: cluster 2024-04-26T16:34:19.840236+0000 mgr.y (mgr.24401) 654 : cluster [DBG] pgmap v1074: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: audit 2024-04-26T16:34:21.026430+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: audit 2024-04-26T16:34:21.026430+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: audit 2024-04-26T16:34:21.256629+0000 mon.a (mon.0) 3398 : audit [DBG] from='client.? 172.21.15.50:0/1826376641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: audit 2024-04-26T16:34:21.256629+0000 mon.a (mon.0) 3398 : audit [DBG] from='client.? 172.21.15.50:0/1826376641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: audit 2024-04-26T16:34:21.322018+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: audit 2024-04-26T16:34:21.322018+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-26T16:34:21.625 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: cluster 2024-04-26T16:34:21.325968+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[29234]: cluster 2024-04-26T16:34:21.325968+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: cluster 2024-04-26T16:34:19.840236+0000 mgr.y (mgr.24401) 654 : cluster [DBG] pgmap v1074: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: cluster 2024-04-26T16:34:19.840236+0000 mgr.y (mgr.24401) 654 : cluster [DBG] pgmap v1074: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: audit 2024-04-26T16:34:21.026430+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: audit 2024-04-26T16:34:21.026430+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: audit 2024-04-26T16:34:21.256629+0000 mon.a (mon.0) 3398 : audit [DBG] from='client.? 172.21.15.50:0/1826376641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: audit 2024-04-26T16:34:21.256629+0000 mon.a (mon.0) 3398 : audit [DBG] from='client.? 172.21.15.50:0/1826376641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: audit 2024-04-26T16:34:21.322018+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: audit 2024-04-26T16:34:21.322018+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: cluster 2024-04-26T16:34:21.325968+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2024-04-26T16:34:21.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:21 smithi050 bash[21463]: cluster 2024-04-26T16:34:21.325968+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2024-04-26T16:34:21.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:21.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: cluster 2024-04-26T16:34:19.840236+0000 mgr.y (mgr.24401) 654 : cluster [DBG] pgmap v1074: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:21.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: cluster 2024-04-26T16:34:19.840236+0000 mgr.y (mgr.24401) 654 : cluster [DBG] pgmap v1074: 204 pgs: 1 active+clean+snaptrim, 51 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: audit 2024-04-26T16:34:21.026430+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: audit 2024-04-26T16:34:21.026430+0000 mon.a (mon.0) 3397 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: audit 2024-04-26T16:34:21.256629+0000 mon.a (mon.0) 3398 : audit [DBG] from='client.? 172.21.15.50:0/1826376641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: audit 2024-04-26T16:34:21.256629+0000 mon.a (mon.0) 3398 : audit [DBG] from='client.? 172.21.15.50:0/1826376641' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: audit 2024-04-26T16:34:21.322018+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: audit 2024-04-26T16:34:21.322018+0000 mon.a (mon.0) 3399 : audit [INF] from='client.? 172.21.15.50:0/2750639425' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: cluster 2024-04-26T16:34:21.325968+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2024-04-26T16:34:21.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:21 smithi190 bash[23868]: cluster 2024-04-26T16:34:21.325968+0000 mon.a (mon.0) 3400 : cluster [DBG] osdmap e695: 8 total, 7 up, 8 in 2024-04-26T16:34:21.693 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:21.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:21.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:21.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:21.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:21.953 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:21.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:22.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:22.082 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:22.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:22.215 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:22.318 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:22.345 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:22.448 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:22.471 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:22.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.529+0000 7f62f37fe700 1 -- 172.21.15.50:0/874794074 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v696) ==== 97+0+0 (secure 0 0 0) 0x7f62f405c8c0 con 0x7f62fc1094b0 2024-04-26T16:34:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 10 librados: watch_flush enter 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 10 librados: watch_flush exit 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 20 client.30933.objecter shutdown clearing up homeless session... 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 10 client.30933.objecter successfully canceled tick 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f62e8071fd0 msgr2=0x7f62e8074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f62e8071fd0 0x7f62e8074490 secure :-1 s=READY pgs=3816 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009fd0 tx=0x7f62f8009260 comp rx=0 tx=0).stop 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 msgr2=0x7f62fc19b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc19b610 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f62f400ab40 tx=0x7f62f400ae50 comp rx=0 tx=0).stop 2024-04-26T16:34:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 shutdown_connections 2024-04-26T16:34:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f62e8071fd0 0x7f62e8074490 unknown :-1 s=CLOSED pgs=3816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62fc1094b0 0x7f62fc19b610 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62fc101a10 0x7f62fc196d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 --2- 172.21.15.50:0/874794074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62fc1023b0 0x7f62fc197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 >> 172.21.15.50:0/874794074 conn(0x7f62fc078990 msgr2=0x7f62fc106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 shutdown_connections 2024-04-26T16:34:22.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 -- 172.21.15.50:0/874794074 wait complete. 2024-04-26T16:34:22.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:22.533+0000 7f63044de700 1 librados: shutdown 2024-04-26T16:34:22.534 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:34:22.574 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:22.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:22.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:22.719 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:22.822 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:22 smithi190 bash[23868]: audit 2024-04-26T16:34:21.605395+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:34:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:22 smithi190 bash[23868]: audit 2024-04-26T16:34:21.605395+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:34:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:22 smithi190 bash[23868]: audit 2024-04-26T16:34:22.394914+0000 mon.c (mon.2) 734 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:22 smithi190 bash[23868]: audit 2024-04-26T16:34:22.394914+0000 mon.c (mon.2) 734 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:22.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[29234]: audit 2024-04-26T16:34:21.605395+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:34:22.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[29234]: audit 2024-04-26T16:34:21.605395+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:34:22.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[29234]: audit 2024-04-26T16:34:22.394914+0000 mon.c (mon.2) 734 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:22.833 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[29234]: audit 2024-04-26T16:34:22.394914+0000 mon.c (mon.2) 734 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:22.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[21463]: audit 2024-04-26T16:34:21.605395+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:34:22.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[21463]: audit 2024-04-26T16:34:21.605395+0000 mon.a (mon.0) 3401 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:34:22.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[21463]: audit 2024-04-26T16:34:22.394914+0000 mon.c (mon.2) 734 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:22.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:22 smithi050 bash[21463]: audit 2024-04-26T16:34:22.394914+0000 mon.c (mon.2) 734 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:34:22.844 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:22.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:22.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:22.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:22.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:23.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:23.098 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:23.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:23.228 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:23.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:23.355 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:23.458 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:23.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:23.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:23.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:23.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:23.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: cluster 2024-04-26T16:34:21.841646+0000 mgr.y (mgr.24401) 655 : cluster [DBG] pgmap v1076: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:23.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: cluster 2024-04-26T16:34:21.841646+0000 mgr.y (mgr.24401) 655 : cluster [DBG] pgmap v1076: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: cluster 2024-04-26T16:34:22.524809+0000 mon.a (mon.0) 3402 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: cluster 2024-04-26T16:34:22.524809+0000 mon.a (mon.0) 3402 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:22.532709+0000 mon.a (mon.0) 3403 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:22.532709+0000 mon.a (mon.0) 3403 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: cluster 2024-04-26T16:34:22.537588+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: cluster 2024-04-26T16:34:22.537588+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:23.079481+0000 mon.c (mon.2) 735 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:23.079481+0000 mon.c (mon.2) 735 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:23.081271+0000 mon.c (mon.2) 736 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:23.081271+0000 mon.c (mon.2) 736 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:23.093809+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:23.734 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[21463]: audit 2024-04-26T16:34:23.093809+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: cluster 2024-04-26T16:34:21.841646+0000 mgr.y (mgr.24401) 655 : cluster [DBG] pgmap v1076: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: cluster 2024-04-26T16:34:21.841646+0000 mgr.y (mgr.24401) 655 : cluster [DBG] pgmap v1076: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: cluster 2024-04-26T16:34:22.524809+0000 mon.a (mon.0) 3402 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: cluster 2024-04-26T16:34:22.524809+0000 mon.a (mon.0) 3402 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:22.532709+0000 mon.a (mon.0) 3403 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:22.532709+0000 mon.a (mon.0) 3403 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: cluster 2024-04-26T16:34:22.537588+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: cluster 2024-04-26T16:34:22.537588+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:23.079481+0000 mon.c (mon.2) 735 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:23.079481+0000 mon.c (mon.2) 735 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:23.081271+0000 mon.c (mon.2) 736 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:23.081271+0000 mon.c (mon.2) 736 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:23.093809+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:23.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:23 smithi050 bash[29234]: audit 2024-04-26T16:34:23.093809+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:23.750 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: cluster 2024-04-26T16:34:21.841646+0000 mgr.y (mgr.24401) 655 : cluster [DBG] pgmap v1076: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: cluster 2024-04-26T16:34:21.841646+0000 mgr.y (mgr.24401) 655 : cluster [DBG] pgmap v1076: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: cluster 2024-04-26T16:34:22.524809+0000 mon.a (mon.0) 3402 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: cluster 2024-04-26T16:34:22.524809+0000 mon.a (mon.0) 3402 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:22.532709+0000 mon.a (mon.0) 3403 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:22.532709+0000 mon.a (mon.0) 3403 : audit [INF] from='client.? 172.21.15.50:0/874794074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: cluster 2024-04-26T16:34:22.537588+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: cluster 2024-04-26T16:34:22.537588+0000 mon.a (mon.0) 3404 : cluster [DBG] osdmap e696: 8 total, 7 up, 8 in 2024-04-26T16:34:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:23.079481+0000 mon.c (mon.2) 735 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:23.079481+0000 mon.c (mon.2) 735 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:34:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:23.081271+0000 mon.c (mon.2) 736 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:23.081271+0000 mon.c (mon.2) 736 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:34:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:23.093809+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:23 smithi190 bash[23868]: audit 2024-04-26T16:34:23.093809+0000 mon.a (mon.0) 3405 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:23.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:23.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:23.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:23.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:24.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:24.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:24.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:24.249 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.276 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:24.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.406 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:34:24.502 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:34:24.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:24.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.667 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:24.715 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:34:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:34:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:34:24.770 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:24.799 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:24.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:24.930 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:24.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:25.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:25.062 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:25.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:25.192 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:25.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:25.324 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:25.427 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:25.456 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:25.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[21463]: cluster 2024-04-26T16:34:23.843379+0000 mgr.y (mgr.24401) 656 : cluster [DBG] pgmap v1078: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[21463]: cluster 2024-04-26T16:34:23.843379+0000 mgr.y (mgr.24401) 656 : cluster [DBG] pgmap v1078: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.034724+0000 osd.4 (osd.4) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.034724+0000 osd.4 (osd.4) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.035174+0000 osd.4 (osd.4) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.035174+0000 osd.4 (osd.4) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:25.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[29234]: cluster 2024-04-26T16:34:23.843379+0000 mgr.y (mgr.24401) 656 : cluster [DBG] pgmap v1078: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:25.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[29234]: cluster 2024-04-26T16:34:23.843379+0000 mgr.y (mgr.24401) 656 : cluster [DBG] pgmap v1078: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:25.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.034724+0000 osd.4 (osd.4) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:25.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.034724+0000 osd.4 (osd.4) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:25.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.035174+0000 osd.4 (osd.4) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:25.571 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:25 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.035174+0000 osd.4 (osd.4) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:25.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:25.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:25.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:25.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:25.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:25 smithi190 bash[23868]: cluster 2024-04-26T16:34:23.843379+0000 mgr.y (mgr.24401) 656 : cluster [DBG] pgmap v1078: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:25 smithi190 bash[23868]: cluster 2024-04-26T16:34:23.843379+0000 mgr.y (mgr.24401) 656 : cluster [DBG] pgmap v1078: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:25 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.034724+0000 osd.4 (osd.4) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:25 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.034724+0000 osd.4 (osd.4) 345 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:25 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.035174+0000 osd.4 (osd.4) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:25 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.035174+0000 osd.4 (osd.4) 346 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:25.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:25.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:25.976 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:25.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:26.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:26.106 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:26.209 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:26.231 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:26.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:26.362 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:26.465 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:26.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:26.503 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:34:26.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:26.607 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:26 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.550767+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2024-04-26T16:34:26.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:26 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.550767+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2024-04-26T16:34:26.623 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:26.717 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:34:26.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:26.754 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:26.856 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:26.868 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:26 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.550767+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2024-04-26T16:34:26.869 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:26 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.550767+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2024-04-26T16:34:26.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:26 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.550767+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2024-04-26T16:34:26.869 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:26 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.550767+0000 mon.a (mon.0) 3406 : cluster [DBG] osdmap e697: 8 total, 7 up, 8 in 2024-04-26T16:34:26.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:26.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:26.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:27.011 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:27.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:27.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:27.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:27.273 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:27.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:27.402 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:27.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:27.517 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:34:27.533 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:27.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:27.663 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:27.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:27.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:27.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.844464+0000 mgr.y (mgr.24401) 657 : cluster [DBG] pgmap v1080: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[23868]: cluster 2024-04-26T16:34:25.844464+0000 mgr.y (mgr.24401) 657 : cluster [DBG] pgmap v1080: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[23868]: audit 2024-04-26T16:34:27.512523+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[23868]: audit 2024-04-26T16:34:27.512523+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[23868]: audit 2024-04-26T16:34:27.515806+0000 mon.c (mon.2) 737 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:27 smithi190 bash[23868]: audit 2024-04-26T16:34:27.515806+0000 mon.c (mon.2) 737 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.844464+0000 mgr.y (mgr.24401) 657 : cluster [DBG] pgmap v1080: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[29234]: cluster 2024-04-26T16:34:25.844464+0000 mgr.y (mgr.24401) 657 : cluster [DBG] pgmap v1080: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[29234]: audit 2024-04-26T16:34:27.512523+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[29234]: audit 2024-04-26T16:34:27.512523+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[29234]: audit 2024-04-26T16:34:27.515806+0000 mon.c (mon.2) 737 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[29234]: audit 2024-04-26T16:34:27.515806+0000 mon.c (mon.2) 737 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.844464+0000 mgr.y (mgr.24401) 657 : cluster [DBG] pgmap v1080: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[21463]: cluster 2024-04-26T16:34:25.844464+0000 mgr.y (mgr.24401) 657 : cluster [DBG] pgmap v1080: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[21463]: audit 2024-04-26T16:34:27.512523+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[21463]: audit 2024-04-26T16:34:27.512523+0000 mon.a (mon.0) 3407 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[21463]: audit 2024-04-26T16:34:27.515806+0000 mon.c (mon.2) 737 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:27 smithi050 bash[21463]: audit 2024-04-26T16:34:27.515806+0000 mon.c (mon.2) 737 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:27.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:27.924 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:27.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:28.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:28.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:28.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:28.184 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:28.224 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:28.287 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:28.314 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:28.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:28.443 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:28.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:28.572 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:28.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:28.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:28.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:28 smithi190 bash[23868]: audit 2024-04-26T16:34:27.180420+0000 mgr.y (mgr.24401) 658 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:28.687 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:28 smithi190 bash[23868]: audit 2024-04-26T16:34:27.180420+0000 mgr.y (mgr.24401) 658 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:28.700 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:28.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:28.829 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:28.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:28.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:28 smithi050 bash[29234]: audit 2024-04-26T16:34:27.180420+0000 mgr.y (mgr.24401) 658 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:28.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:28 smithi050 bash[29234]: audit 2024-04-26T16:34:27.180420+0000 mgr.y (mgr.24401) 658 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:28.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:28 smithi050 bash[21463]: audit 2024-04-26T16:34:27.180420+0000 mgr.y (mgr.24401) 658 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:28.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:28 smithi050 bash[21463]: audit 2024-04-26T16:34:27.180420+0000 mgr.y (mgr.24401) 658 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:28.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:28.960 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:28.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:28.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:29.063 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:29.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:29.197 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:29.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:29.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:29.353 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:29.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:29.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-04-26T16:34:29.470 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:34:29.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:34:29.484 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:29.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:29.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:29 smithi190 bash[23868]: cluster 2024-04-26T16:34:27.847513+0000 mgr.y (mgr.24401) 659 : cluster [DBG] pgmap v1081: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:29.598 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:29 smithi190 bash[23868]: cluster 2024-04-26T16:34:27.847513+0000 mgr.y (mgr.24401) 659 : cluster [DBG] pgmap v1081: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:29.613 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:29.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:29.740 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:29 smithi050 bash[29234]: cluster 2024-04-26T16:34:27.847513+0000 mgr.y (mgr.24401) 659 : cluster [DBG] pgmap v1081: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:29.742 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:29 smithi050 bash[29234]: cluster 2024-04-26T16:34:27.847513+0000 mgr.y (mgr.24401) 659 : cluster [DBG] pgmap v1081: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:29.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:29 smithi050 bash[21463]: cluster 2024-04-26T16:34:27.847513+0000 mgr.y (mgr.24401) 659 : cluster [DBG] pgmap v1081: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:29.743 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:29 smithi050 bash[21463]: cluster 2024-04-26T16:34:27.847513+0000 mgr.y (mgr.24401) 659 : cluster [DBG] pgmap v1081: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:29.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:29.959 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:29.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:30.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:30.090 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:30.193 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:30.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:30.351 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:30.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:30.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:30.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:30.614 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:30.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:30.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:30.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:30.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:30.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:30.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:31.104 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:31.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:31.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:31.262 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:31.324 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:34:31 smithi190 bash[50598]: logger=sqlstore.transactions t=2024-04-26T16:34:31.062446153Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-04-26T16:34:31.342 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:31.343 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:31.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:31.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:31.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:31.513 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:31.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:31.628 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:31 smithi050 bash[21463]: cluster 2024-04-26T16:34:29.848530+0000 mgr.y (mgr.24401) 660 : cluster [DBG] pgmap v1082: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:31.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:31 smithi050 bash[21463]: cluster 2024-04-26T16:34:29.848530+0000 mgr.y (mgr.24401) 660 : cluster [DBG] pgmap v1082: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:31 smithi050 bash[29234]: cluster 2024-04-26T16:34:29.848530+0000 mgr.y (mgr.24401) 660 : cluster [DBG] pgmap v1082: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:31.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:31 smithi050 bash[29234]: cluster 2024-04-26T16:34:29.848530+0000 mgr.y (mgr.24401) 660 : cluster [DBG] pgmap v1082: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:31.646 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:31.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:31.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:31 smithi190 bash[23868]: cluster 2024-04-26T16:34:29.848530+0000 mgr.y (mgr.24401) 660 : cluster [DBG] pgmap v1082: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:31.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:31 smithi190 bash[23868]: cluster 2024-04-26T16:34:29.848530+0000 mgr.y (mgr.24401) 660 : cluster [DBG] pgmap v1082: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:31.777 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:31.881 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:31.910 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:31.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:32.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:32.040 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:32.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:32.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:32.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:32.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:32.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:32.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:32.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:32.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:32.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:32.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:32.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:32.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:32.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:32.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:32.957 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:32.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:33.060 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:33.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:33.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:33.218 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:33.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:33.326 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:33.349 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:33.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:33.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:33.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:33.596 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:33 smithi190 bash[23868]: cluster 2024-04-26T16:34:31.851570+0000 mgr.y (mgr.24401) 661 : cluster [DBG] pgmap v1083: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:33.596 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:33 smithi190 bash[23868]: cluster 2024-04-26T16:34:31.851570+0000 mgr.y (mgr.24401) 661 : cluster [DBG] pgmap v1083: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:33.613 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:33.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:33 smithi050 bash[21463]: cluster 2024-04-26T16:34:31.851570+0000 mgr.y (mgr.24401) 661 : cluster [DBG] pgmap v1083: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:33.660 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:33 smithi050 bash[21463]: cluster 2024-04-26T16:34:31.851570+0000 mgr.y (mgr.24401) 661 : cluster [DBG] pgmap v1083: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:33.716 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:33.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:33 smithi050 bash[29234]: cluster 2024-04-26T16:34:31.851570+0000 mgr.y (mgr.24401) 661 : cluster [DBG] pgmap v1083: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:33.728 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:33 smithi050 bash[29234]: cluster 2024-04-26T16:34:31.851570+0000 mgr.y (mgr.24401) 661 : cluster [DBG] pgmap v1083: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:33.746 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:33.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:33.875 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:33.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:33.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:34.006 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:34.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:34.137 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:34.241 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:34.269 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:34.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:34.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:34.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:34.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:34.532 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:34.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:34:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:34:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:34:34.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:34.661 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:34.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:34.792 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:34.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:34.895 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:34.916 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:34.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:35.018 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:34:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:34:35.019 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-26T16:34:35.019 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:34:35.019 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:35.040 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:35.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:35.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.299 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:35.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:35.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:35.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:35.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:35 smithi050 bash[21463]: cluster 2024-04-26T16:34:33.853432+0000 mgr.y (mgr.24401) 662 : cluster [DBG] pgmap v1084: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:35.671 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:35 smithi050 bash[21463]: cluster 2024-04-26T16:34:33.853432+0000 mgr.y (mgr.24401) 662 : cluster [DBG] pgmap v1084: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:35.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:35 smithi050 bash[29234]: cluster 2024-04-26T16:34:33.853432+0000 mgr.y (mgr.24401) 662 : cluster [DBG] pgmap v1084: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:35.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:35 smithi050 bash[29234]: cluster 2024-04-26T16:34:33.853432+0000 mgr.y (mgr.24401) 662 : cluster [DBG] pgmap v1084: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:35.683 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:35.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:35.798 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:35 smithi190 bash[23868]: cluster 2024-04-26T16:34:33.853432+0000 mgr.y (mgr.24401) 662 : cluster [DBG] pgmap v1084: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:35.798 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:35 smithi190 bash[23868]: cluster 2024-04-26T16:34:33.853432+0000 mgr.y (mgr.24401) 662 : cluster [DBG] pgmap v1084: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:35.812 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:35.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.942 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:35.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:36.045 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:36.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:36.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:36.207 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:36.310 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:36.338 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:36.442 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:36.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:36.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:36.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:36.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:36.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:36.739 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:36.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:36.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:36.975 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:37.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:37.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:37.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:37.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:37.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:37.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-04-26T16:34:37.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:37.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:37.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:37.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:34:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:34:37.525 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:37.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[21463]: cluster 2024-04-26T16:34:35.855227+0000 mgr.y (mgr.24401) 663 : cluster [DBG] pgmap v1085: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[21463]: cluster 2024-04-26T16:34:35.855227+0000 mgr.y (mgr.24401) 663 : cluster [DBG] pgmap v1085: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[21463]: cluster 2024-04-26T16:34:36.597706+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2024-04-26T16:34:37.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[21463]: cluster 2024-04-26T16:34:36.597706+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2024-04-26T16:34:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[29234]: cluster 2024-04-26T16:34:35.855227+0000 mgr.y (mgr.24401) 663 : cluster [DBG] pgmap v1085: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[29234]: cluster 2024-04-26T16:34:35.855227+0000 mgr.y (mgr.24401) 663 : cluster [DBG] pgmap v1085: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[29234]: cluster 2024-04-26T16:34:36.597706+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2024-04-26T16:34:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:37 smithi050 bash[29234]: cluster 2024-04-26T16:34:36.597706+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2024-04-26T16:34:37.655 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:37.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:37.781 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:37 smithi190 bash[23868]: cluster 2024-04-26T16:34:35.855227+0000 mgr.y (mgr.24401) 663 : cluster [DBG] pgmap v1085: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:37 smithi190 bash[23868]: cluster 2024-04-26T16:34:35.855227+0000 mgr.y (mgr.24401) 663 : cluster [DBG] pgmap v1085: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:37 smithi190 bash[23868]: cluster 2024-04-26T16:34:36.597706+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2024-04-26T16:34:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:37 smithi190 bash[23868]: cluster 2024-04-26T16:34:36.597706+0000 mon.a (mon.0) 3408 : cluster [DBG] osdmap e698: 8 total, 7 up, 8 in 2024-04-26T16:34:37.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:37.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:37.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:38.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:38.045 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:38.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:38.177 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:38.281 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:38.309 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:38.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:38.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:38.441 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:38.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:38.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:38.570 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:38.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:38.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:38.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: audit 2024-04-26T16:34:37.191579+0000 mgr.y (mgr.24401) 664 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: audit 2024-04-26T16:34:37.191579+0000 mgr.y (mgr.24401) 664 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: cluster 2024-04-26T16:34:37.612674+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: cluster 2024-04-26T16:34:37.612674+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: audit 2024-04-26T16:34:37.614163+0000 mon.c (mon.2) 738 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: audit 2024-04-26T16:34:37.614163+0000 mon.c (mon.2) 738 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: audit 2024-04-26T16:34:37.615000+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:38 smithi190 bash[23868]: audit 2024-04-26T16:34:37.615000+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:38.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:38.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:38.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:38.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:39.034 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: audit 2024-04-26T16:34:37.191579+0000 mgr.y (mgr.24401) 664 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:39.034 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: audit 2024-04-26T16:34:37.191579+0000 mgr.y (mgr.24401) 664 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:39.034 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: cluster 2024-04-26T16:34:37.612674+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: cluster 2024-04-26T16:34:37.612674+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: audit 2024-04-26T16:34:37.614163+0000 mon.c (mon.2) 738 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: audit 2024-04-26T16:34:37.614163+0000 mon.c (mon.2) 738 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: audit 2024-04-26T16:34:37.615000+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[29234]: audit 2024-04-26T16:34:37.615000+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: audit 2024-04-26T16:34:37.191579+0000 mgr.y (mgr.24401) 664 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: audit 2024-04-26T16:34:37.191579+0000 mgr.y (mgr.24401) 664 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: cluster 2024-04-26T16:34:37.612674+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: cluster 2024-04-26T16:34:37.612674+0000 mon.a (mon.0) 3409 : cluster [DBG] osdmap e699: 8 total, 7 up, 8 in 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: audit 2024-04-26T16:34:37.614163+0000 mon.c (mon.2) 738 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: audit 2024-04-26T16:34:37.614163+0000 mon.c (mon.2) 738 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: audit 2024-04-26T16:34:37.615000+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:38 smithi050 bash[21463]: audit 2024-04-26T16:34:37.615000+0000 mon.a (mon.0) 3410 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:39.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:39.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:39.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:39.214 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:39.317 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:39.346 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:39.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:39.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:39.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:39.603 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:39.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:39.706 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:39.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:39.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:39.866 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:39.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: cluster 2024-04-26T16:34:37.857070+0000 mgr.y (mgr.24401) 665 : cluster [DBG] pgmap v1088: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/618 objects degraded (12.136%) 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: cluster 2024-04-26T16:34:37.857070+0000 mgr.y (mgr.24401) 665 : cluster [DBG] pgmap v1088: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/618 objects degraded (12.136%) 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: cluster 2024-04-26T16:34:38.602265+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: Degraded data redundancy: 75/618 objects degraded (12.136%), 19 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: cluster 2024-04-26T16:34:38.602265+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: Degraded data redundancy: 75/618 objects degraded (12.136%), 19 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: audit 2024-04-26T16:34:38.612822+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: audit 2024-04-26T16:34:38.612822+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:39.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: cluster 2024-04-26T16:34:38.623472+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[29234]: cluster 2024-04-26T16:34:38.623472+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: cluster 2024-04-26T16:34:37.857070+0000 mgr.y (mgr.24401) 665 : cluster [DBG] pgmap v1088: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/618 objects degraded (12.136%) 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: cluster 2024-04-26T16:34:37.857070+0000 mgr.y (mgr.24401) 665 : cluster [DBG] pgmap v1088: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/618 objects degraded (12.136%) 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: cluster 2024-04-26T16:34:38.602265+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: Degraded data redundancy: 75/618 objects degraded (12.136%), 19 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: cluster 2024-04-26T16:34:38.602265+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: Degraded data redundancy: 75/618 objects degraded (12.136%), 19 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: audit 2024-04-26T16:34:38.612822+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: audit 2024-04-26T16:34:38.612822+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: cluster 2024-04-26T16:34:38.623472+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2024-04-26T16:34:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:39 smithi050 bash[21463]: cluster 2024-04-26T16:34:38.623472+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2024-04-26T16:34:39.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:39.980 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: cluster 2024-04-26T16:34:37.857070+0000 mgr.y (mgr.24401) 665 : cluster [DBG] pgmap v1088: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/618 objects degraded (12.136%) 2024-04-26T16:34:39.980 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: cluster 2024-04-26T16:34:37.857070+0000 mgr.y (mgr.24401) 665 : cluster [DBG] pgmap v1088: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 75/618 objects degraded (12.136%) 2024-04-26T16:34:39.980 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: cluster 2024-04-26T16:34:38.602265+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: Degraded data redundancy: 75/618 objects degraded (12.136%), 19 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:39.980 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: cluster 2024-04-26T16:34:38.602265+0000 mon.a (mon.0) 3411 : cluster [WRN] Health check update: Degraded data redundancy: 75/618 objects degraded (12.136%), 19 pgs degraded, 56 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:39.980 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: audit 2024-04-26T16:34:38.612822+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:39.981 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: audit 2024-04-26T16:34:38.612822+0000 mon.a (mon.0) 3412 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:39.981 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: cluster 2024-04-26T16:34:38.623472+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2024-04-26T16:34:39.981 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:39 smithi190 bash[23868]: cluster 2024-04-26T16:34:38.623472+0000 mon.a (mon.0) 3413 : cluster [DBG] osdmap e700: 8 total, 7 up, 8 in 2024-04-26T16:34:39.997 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:40.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:40.128 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:40.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:40.259 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:40.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:40.389 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:40.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:40.520 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:40.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:40.652 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:40.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.619623+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.619623+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: audit 2024-04-26T16:34:39.632694+0000 mon.c (mon.2) 739 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: audit 2024-04-26T16:34:39.632694+0000 mon.c (mon.2) 739 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: audit 2024-04-26T16:34:39.633144+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: audit 2024-04-26T16:34:39.633144+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.735479+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.735479+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.735785+0000 osd.1 (osd.1) 370 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:40.767 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.735785+0000 osd.1 (osd.1) 370 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.619623+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.619623+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: audit 2024-04-26T16:34:39.632694+0000 mon.c (mon.2) 739 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: audit 2024-04-26T16:34:39.632694+0000 mon.c (mon.2) 739 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: audit 2024-04-26T16:34:39.633144+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: audit 2024-04-26T16:34:39.633144+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.735479+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.735479+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.735785+0000 osd.1 (osd.1) 370 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:40.768 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:40 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.735785+0000 osd.1 (osd.1) 370 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:40.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:40.886 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:40.915 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:40.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:41.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:41.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.619623+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2024-04-26T16:34:41.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.619623+0000 mon.a (mon.0) 3414 : cluster [DBG] osdmap e701: 8 total, 7 up, 8 in 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: audit 2024-04-26T16:34:39.632694+0000 mon.c (mon.2) 739 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: audit 2024-04-26T16:34:39.632694+0000 mon.c (mon.2) 739 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: audit 2024-04-26T16:34:39.633144+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: audit 2024-04-26T16:34:39.633144+0000 mon.a (mon.0) 3415 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.735479+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.735479+0000 osd.1 (osd.1) 369 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.735785+0000 osd.1 (osd.1) 370 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:41.031 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:40 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.735785+0000 osd.1 (osd.1) 370 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:41.047 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:41.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:41.177 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:41.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:41.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:41.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:41.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:41.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:41.431 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:41.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:41.561 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:41.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:41.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:41.795 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:41.820 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.858817+0000 mgr.y (mgr.24401) 666 : cluster [DBG] pgmap v1091: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 75/618 objects degraded (12.136%) 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:39.858817+0000 mgr.y (mgr.24401) 666 : cluster [DBG] pgmap v1091: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 75/618 objects degraded (12.136%) 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:40.624936+0000 mon.a (mon.0) 3416 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:40.624936+0000 mon.a (mon.0) 3416 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:40.626559+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:40.626559+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:40.636280+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:40.636280+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:40.637458+0000 mon.c (mon.2) 740 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:40.637458+0000 mon.c (mon.2) 740 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:40.638511+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:40.638511+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:41.328547+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:41.328547+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:41.334187+0000 mon.c (mon.2) 741 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:41.334187+0000 mon.c (mon.2) 741 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:41.335433+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: cluster 2024-04-26T16:34:41.335433+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:41.336546+0000 mon.a (mon.0) 3422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:41 smithi190 bash[23868]: audit 2024-04-26T16:34:41.336546+0000 mon.a (mon.0) 3422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.858817+0000 mgr.y (mgr.24401) 666 : cluster [DBG] pgmap v1091: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 75/618 objects degraded (12.136%) 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:39.858817+0000 mgr.y (mgr.24401) 666 : cluster [DBG] pgmap v1091: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 75/618 objects degraded (12.136%) 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:40.624936+0000 mon.a (mon.0) 3416 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:40.624936+0000 mon.a (mon.0) 3416 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:40.626559+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:40.626559+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:41.899 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:40.636280+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:40.636280+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:40.637458+0000 mon.c (mon.2) 740 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:40.637458+0000 mon.c (mon.2) 740 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:40.638511+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:40.638511+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:41.328547+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:41.900 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:41.328547+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.858817+0000 mgr.y (mgr.24401) 666 : cluster [DBG] pgmap v1091: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 75/618 objects degraded (12.136%) 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:39.858817+0000 mgr.y (mgr.24401) 666 : cluster [DBG] pgmap v1091: 204 pgs: 32 unknown, 40 active+undersized, 19 active+undersized+degraded, 113 active+clean; 583 KiB data, 671 MiB used, 715 GiB / 715 GiB avail; 75/618 objects degraded (12.136%) 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:40.624936+0000 mon.a (mon.0) 3416 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:40.624936+0000 mon.a (mon.0) 3416 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:40.626559+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:40.626559+0000 mon.a (mon.0) 3417 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:40.636280+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:40.636280+0000 mon.a (mon.0) 3418 : cluster [DBG] osdmap e702: 8 total, 7 up, 8 in 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:40.637458+0000 mon.c (mon.2) 740 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:40.637458+0000 mon.c (mon.2) 740 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:40.638511+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:40.638511+0000 mon.a (mon.0) 3419 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:41.328547+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:41.901 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:41.328547+0000 mon.a (mon.0) 3420 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:41.334187+0000 mon.c (mon.2) 741 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:41.334187+0000 mon.c (mon.2) 741 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:41.335433+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: cluster 2024-04-26T16:34:41.335433+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:41.336546+0000 mon.a (mon.0) 3422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[21463]: audit 2024-04-26T16:34:41.336546+0000 mon.a (mon.0) 3422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:41.334187+0000 mon.c (mon.2) 741 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:41.334187+0000 mon.c (mon.2) 741 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:41.335433+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2024-04-26T16:34:41.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: cluster 2024-04-26T16:34:41.335433+0000 mon.a (mon.0) 3421 : cluster [DBG] osdmap e703: 8 total, 7 up, 8 in 2024-04-26T16:34:41.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:41.336546+0000 mon.a (mon.0) 3422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.903 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:41 smithi050 bash[29234]: audit 2024-04-26T16:34:41.336546+0000 mon.a (mon.0) 3422 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]: dispatch 2024-04-26T16:34:41.923 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:41.944 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:41.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- 172.21.15.50:0/3157831953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08102320 msgr2=0x7fdb081027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- 172.21.15.50:0/3157831953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08102320 0x7fdb081027a0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7fdaf8009990 tx=0x7fdaf8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- 172.21.15.50:0/3157831953 shutdown_connections 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- 172.21.15.50:0/3157831953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb08108390 0x7fdb08108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- 172.21.15.50:0/3157831953 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdb08102ce0 0x7fdb0810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- 172.21.15.50:0/3157831953 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08102320 0x7fdb081027a0 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- 172.21.15.50:0/3157831953 >> 172.21.15.50:0/3157831953 conn(0x7fdb080792d0 msgr2=0x7fdb080796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:41.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- 172.21.15.50:0/3157831953 shutdown_connections 2024-04-26T16:34:41.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- 172.21.15.50:0/3157831953 wait complete. 2024-04-26T16:34:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 librados: starting msgr at 2024-04-26T16:34:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 librados: starting objecter 2024-04-26T16:34:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 Processor -- start 2024-04-26T16:34:41.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- start start 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 librados: setting wanted keys 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 librados: calling monclient init 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb08102320 0x7fdb08199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdb08102ce0 0x7fdb08199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08108390 0x7fdb0819d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdb081119a0 con 0x7fdb08108390 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdb08111820 con 0x7fdb08102320 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0f27f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fdb08111b20 con 0x7fdb08102ce0 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0d81c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08108390 0x7fdb0819d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:41.986 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdb08102ce0 0x7fdb08199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdb08102ce0 0x7fdb08199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41304/0 (socket says 172.21.15.50:41304) 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 -- 172.21.15.50:0/2565493205 learned_addr learned my addr 172.21.15.50:0/2565493205 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb08102320 0x7fdb08199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 -- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb08102320 msgr2=0x7fdb08199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb08102320 0x7fdb08199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 -- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08108390 msgr2=0x7fdb0819d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:41.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08108390 0x7fdb0819d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0c81a700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0819e120 con 0x7fdb08102ce0 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.985+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdb08108390 0x7fdb0819d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdb08102ce0 0x7fdb08199650 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80057a0 tx=0x7fdaf80057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 10 client.?.objecter ms_handle_connect 0x7fdb08102ce0 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8026070 con 0x7fdb08102ce0 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fdaf8021410 con 0x7fdb08102ce0 2024-04-26T16:34:41.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80173f0 con 0x7fdb08102ce0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0f27f700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0819e3b0 con 0x7fdb08102ce0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0f27f700 10 client.41555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0f27f700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0819e810 con 0x7fdb08102ce0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0f27f700 1 librados: init done 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0f27f700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fdb0806a110 con 0x7fdb08102ce0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fdaf8009d10 con 0x7fdb08102ce0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdaf4071f80 0x7fdaf4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(703..703 src has 577..703) ==== 9859+0+0 (secure 0 0 0) 0x7fdaf8092520 con 0x7fdb08102ce0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 osd_map(703..703 src has 577..703) 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 3 client.41555.objecter handle_osd_map got epochs [703,703] > 0 2024-04-26T16:34:41.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 3 client.41555.objecter handle_osd_map decoding full epoch 703 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdaf4071f80 0x7fdaf4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 20 client.41555.objecter dump_active .. 0 homeless 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.989+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v703) ==== 54+0+16 (secure 0 0 0) 0x7fdb0819e810 con 0x7fdb08102ce0 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 1 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fdae8002ff0 0x7fdae80054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdaf4071f80 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:41.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdaf4071f80 0x7fdaf4074440 secure :-1 s=READY pgs=3817 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005950 tx=0x7fdb0400b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 1 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 0 1 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae80028e0 osd=0 4 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 10 client.41555.objecter _send_command 1 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdae8008530 con 0x7fdae8002ff0 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fdae8002ff0 0x7fdae80054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fdae8002ff0 0x7fdae80054b0 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:41.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae8002ff0 2024-04-26T16:34:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae8008530 con 0x7fdae8002ff0 2024-04-26T16:34:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 1: 0 ) 2024-04-26T16:34:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 1 = system:0 2024-04-26T16:34:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae80028e0 osd=0 4 2024-04-26T16:34:41.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:41.993+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 0 1 2024-04-26T16:34:42.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 2 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 2 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae80028e0 osd=0 3 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 0 2 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae80028e0 osd=0 4 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 10 client.41555.objecter _send_command 2 2024-04-26T16:34:42.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae8001240 con 0x7fdae8002ff0 2024-04-26T16:34:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae8001240 con 0x7fdae8002ff0 2024-04-26T16:34:42.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae80028e0 osd=0 4 2024-04-26T16:34:42.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 0 2 2024-04-26T16:34:42.012 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fdae8006cb0 0x7fdae8007160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 1 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001f80 osd=1 4 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 10 client.41555.objecter _send_command 3 2024-04-26T16:34:42.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fdae80076a0 con 0x7fdae8006cb0 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fdae8006cb0 0x7fdae8007160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae8006cb0 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.013+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fdae8006cb0 0x7fdae8007160 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.017+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae80076a0 con 0x7fdae8006cb0 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.017+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 3: 0 ) 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.017+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 3 = system:0 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.017+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8001f80 osd=1 4 2024-04-26T16:34:42.014 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.017+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 1 3 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 4 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 4 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8001f80 osd=1 3 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 1 4 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001f80 osd=1 4 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 10 client.41555.objecter _send_command 4 2024-04-26T16:34:42.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.033+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae80080f0 con 0x7fdae8006cb0 2024-04-26T16:34:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae80080f0 con 0x7fdae8006cb0 2024-04-26T16:34:42.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8001f80 osd=1 4 2024-04-26T16:34:42.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 1 4 2024-04-26T16:34:42.034 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 5 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fdae800acf0 0x7fdae800d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 5 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 2 5 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001cd0 osd=2 4 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 10 client.41555.objecter _send_command 5 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fdae8010080 con 0x7fdae800acf0 2024-04-26T16:34:42.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fdae800acf0 0x7fdae800d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae800acf0 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fdae800acf0 0x7fdae800d0e0 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae8010080 con 0x7fdae800acf0 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 5: 0 ) 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 5 = system:0 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8001cd0 osd=2 4 2024-04-26T16:34:42.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.037+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 2 5 2024-04-26T16:34:42.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/4227333445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc103470 msgr2=0x7ff3bc1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/4227333445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc103470 0x7ff3bc1038f0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00098f0 tx=0x7ff3b0009c00 comp rx=0 tx=0).stop 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/4227333445 shutdown_connections 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/4227333445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3bc1094f0 0x7ff3bc1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/4227333445 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/4227333445 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc103470 0x7ff3bc1038f0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/4227333445 >> 172.21.15.50:0/4227333445 conn(0x7ff3bc07a5d0 msgr2=0x7ff3bc07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/4227333445 shutdown_connections 2024-04-26T16:34:42.053 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.053+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/4227333445 wait complete. 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 librados: starting msgr at 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 librados: starting objecter 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 Processor -- start 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 -- start start 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 librados: setting wanted keys 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 librados: calling monclient init 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3bc103470 0x7ff3bc199200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.055 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc1094f0 0x7ff3bc19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc112b20 con 0x7ff3bc1094f0 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff3bc1129a0 con 0x7ff3bc103470 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff3bc112ca0 con 0x7ff3bc103e30 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3bb7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc1094f0 0x7ff3bc19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:41326/0 (socket says 172.21.15.50:41326) 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 -- 172.21.15.50:0/909257961 learned_addr learned my addr 172.21.15.50:0/909257961 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3baffd700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3bc103470 0x7ff3bc199200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 -- 172.21.15.50:0/909257961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3bc103470 msgr2=0x7ff3bc199200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3bc103470 0x7ff3bc199200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 -- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc1094f0 msgr2=0x7ff3bc19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc1094f0 0x7ff3bc19dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.056 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 -- 172.21.15.50:0/909257961 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3bc19e210 con 0x7ff3bc103e30 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3ba7fc700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc199740 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009230 tx=0x7ff3b000b560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c0a5d700 10 client.?.objecter ms_handle_connect 0x7ff3bc103e30 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c0a5d700 10 client.?.objecter resend_mon_ops 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 6 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 6 2024-04-26T16:34:42.057 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8001cd0 osd=2 3 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 2 6 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8001cd0 osd=2 4 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 10 client.41555.objecter _send_command 6 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae8007bc0 con 0x7fdae800acf0 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0025070 con 0x7ff3bc103e30 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff3b000ba40 con 0x7ff3bc103e30 2024-04-26T16:34:42.058 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0003ba0 con 0x7ff3bc103e30 2024-04-26T16:34:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae8007bc0 con 0x7fdae800acf0 2024-04-26T16:34:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3bc19e4a0 con 0x7ff3bc103e30 2024-04-26T16:34:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8001cd0 osd=2 4 2024-04-26T16:34:42.059 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 2 6 2024-04-26T16:34:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 10 client.41561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.057+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3bc1a5d80 con 0x7ff3bc103e30 2024-04-26T16:34:42.060 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:34:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.060 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 7 2024-04-26T16:34:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fdae800e7a0 0x7fdae800ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c1a5f700 1 librados: init done 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 7 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 3 7 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8006300 osd=3 4 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 10 client.41555.objecter _send_command 7 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fdae800f160 con 0x7fdae800e7a0 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fdae800e7a0 0x7fdae800ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae800e7a0 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fdae800e7a0 0x7fdae800ec20 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae800f160 con 0x7fdae800e7a0 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 7: 0 ) 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 7 = system:0 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8006300 osd=3 4 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 3 7 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff3b0028070 con 0x7ff3bc103e30 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3bc06a110 con 0x7ff3bc103e30 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3a4071fd0 0x7ff3a4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(703..703 src has 577..703) ==== 9859+0+0 (secure 0 0 0) 0x7ff3b0092420 con 0x7ff3bc103e30 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 10 client.41561.objecter ms_dispatch 0x7ff3bc001550 osd_map(703..703 src has 577..703) 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 3 client.41561.objecter handle_osd_map got epochs [703,703] > 0 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 3 client.41561.objecter handle_osd_map decoding full epoch 703 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3baffd700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3a4071fd0 0x7ff3a4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.061+0000 7ff3c0a5d700 20 client.41561.objecter dump_active .. 0 homeless 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.065+0000 7ff3c0a5d700 10 client.41561.objecter ms_handle_connect 0x7ff3a4071fd0 2024-04-26T16:34:42.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.065+0000 7ff3baffd700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3a4071fd0 0x7ff3a4074490 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac005950 tx=0x7ff3ac0058e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.069+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff3b0013030 con 0x7ff3bc103e30 2024-04-26T16:34:42.072 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:42.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 8 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 8 2024-04-26T16:34:42.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8006300 osd=3 3 2024-04-26T16:34:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 3 8 2024-04-26T16:34:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8006300 osd=3 4 2024-04-26T16:34:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 10 client.41555.objecter _send_command 8 2024-04-26T16:34:42.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae800fb70 con 0x7fdae800e7a0 2024-04-26T16:34:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae800fb70 con 0x7fdae800e7a0 2024-04-26T16:34:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8006300 osd=3 4 2024-04-26T16:34:42.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 3 8 2024-04-26T16:34:42.080 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 9 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fdae8012240 0x7fdae8014630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 9 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 4 9 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800fb70 osd=4 4 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 10 client.41555.objecter _send_command 9 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fdae8017490 con 0x7fdae8012240 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.081+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fdae8012240 0x7fdae8014630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae8012240 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdb0d01b700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fdae8012240 0x7fdae8014630 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae8017490 con 0x7fdae8012240 2024-04-26T16:34:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 9: 0 ) 2024-04-26T16:34:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 9 = system:0 2024-04-26T16:34:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae800fb70 osd=4 4 2024-04-26T16:34:42.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.085+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 4 9 2024-04-26T16:34:42.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 -- 172.21.15.50:0/2452392390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 msgr2=0x7fdc6c1140b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 --2- 172.21.15.50:0/2452392390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1140b0 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009990 tx=0x7fdc60009ca0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 -- 172.21.15.50:0/2452392390 shutdown_connections 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 --2- 172.21.15.50:0/2452392390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c075650 0x7fdc6c075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 --2- 172.21.15.50:0/2452392390 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c1145f0 0x7fdc6c1169e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 --2- 172.21.15.50:0/2452392390 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1140b0 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 -- 172.21.15.50:0/2452392390 >> 172.21.15.50:0/2452392390 conn(0x7fdc6c07a580 msgr2=0x7fdc6c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 -- 172.21.15.50:0/2452392390 shutdown_connections 2024-04-26T16:34:42.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 -- 172.21.15.50:0/2452392390 wait complete. 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 librados: starting msgr at 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.089+0000 7fdc71103700 1 librados: starting objecter 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 Processor -- start 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- start start 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 librados: setting wanted keys 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 librados: calling monclient init 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 0x7fdc6c1a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c1145f0 0x7fdc6c19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c119160 con 0x7fdc6c075650 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdc6c118fe0 con 0x7fdc6c1145f0 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fdc6c1192e0 con 0x7fdc6c075f70 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 0x7fdc6c1a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 0x7fdc6c1a5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40948/0 (socket says 172.21.15.50:40948) 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 -- 172.21.15.50:0/4169460694 learned_addr learned my addr 172.21.15.50:0/4169460694 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6a59c700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6b59e700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c1145f0 0x7fdc6c19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 -- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 msgr2=0x7fdc6c1a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 -- 172.21.15.50:0/4169460694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c1145f0 msgr2=0x7fdc6c19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c1145f0 0x7fdc6c19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 -- 172.21.15.50:0/4169460694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6c19fd90 con 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6a59c700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc6ad9d700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 0x7fdc6c1a5420 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00e9a0 tx=0x7fdc5c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc53fff700 10 client.?.objecter ms_handle_connect 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc53fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c00cbd0 con 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fdc5c00cd70 con 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5c018680 con 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6c1a0080 con 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 10 client.30951.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6c1ac2f0 con 0x7fdc6c075650 2024-04-26T16:34:42.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 librados: init done 2024-04-26T16:34:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.093+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc6c06a110 con 0x7fdc6c075650 2024-04-26T16:34:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fdc5c010920 con 0x7fdc6c075650 2024-04-26T16:34:42.095 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc54071f80 0x7fdc54074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc6a59c700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc54071f80 0x7fdc54074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(703..703 src has 577..703) ==== 9859+0+0 (secure 0 0 0) 0x7fdc5c091ef0 con 0x7fdc6c075650 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 10 client.30951.objecter ms_dispatch 0x7fdc6c001550 osd_map(703..703 src has 577..703) 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 3 client.30951.objecter handle_osd_map got epochs [703,703] > 0 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 3 client.30951.objecter handle_osd_map decoding full epoch 703 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 20 client.30951.objecter dump_active .. 0 homeless 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc53fff700 10 client.30951.objecter ms_handle_connect 0x7fdc54071f80 2024-04-26T16:34:42.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.097+0000 7fdc6a59c700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc54071f80 0x7fdc54074440 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000b730 tx=0x7fdc6001a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.099 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:42.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.105+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fdc5c05d740 con 0x7fdc6c075650 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 10 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 10 2024-04-26T16:34:42.109 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae800fb70 osd=4 3 2024-04-26T16:34:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 4 10 2024-04-26T16:34:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800fb70 osd=4 4 2024-04-26T16:34:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 10 client.41555.objecter _send_command 10 2024-04-26T16:34:42.110 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae8014b70 con 0x7fdae8012240 2024-04-26T16:34:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae8014b70 con 0x7fdae8012240 2024-04-26T16:34:42.111 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae800fb70 osd=4 4 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 4 10 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 11 2024-04-26T16:34:42.112 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fdae80152c0 0x7fdae80156e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 11 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 5 11 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8014b70 osd=5 4 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 10 client.41555.objecter _send_command 11 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fdae801a890 con 0x7fdae80152c0 2024-04-26T16:34:42.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fdae80152c0 0x7fdae80156e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae80152c0 2024-04-26T16:34:42.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.113+0000 7fdb0c81a700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fdae80152c0 0x7fdae80156e0 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.121+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae801a890 con 0x7fdae80152c0 2024-04-26T16:34:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.121+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 11: 0 ) 2024-04-26T16:34:42.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.121+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 11 = system:0 2024-04-26T16:34:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.121+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8014b70 osd=5 4 2024-04-26T16:34:42.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.121+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 5 11 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 12 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 12 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae8014b70 osd=5 3 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 5 12 2024-04-26T16:34:42.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae8014b70 osd=5 4 2024-04-26T16:34:42.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 10 client.41555.objecter _send_command 12 2024-04-26T16:34:42.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.157+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae8015d30 con 0x7fdae80152c0 2024-04-26T16:34:42.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae8015d30 con 0x7fdae80152c0 2024-04-26T16:34:42.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae8014b70 osd=5 4 2024-04-26T16:34:42.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 5 12 2024-04-26T16:34:42.160 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:34:42.160 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 13 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 20 client.41555.objecter _get_session osd=-1 returning homeless 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 10 client.41555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=704}) -- 0x7fdae801a890 con 0x7fdb08102ce0 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdae801ab80 con 0x7fdb08102ce0 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (secure 0 0 0) 0x7fdaf8059040 con 0x7fdb08102ce0 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaff7fe700 10 client.41555.objecter _check_command_map_dne tid 13 current 703 map_dne_bound 703 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaff7fe700 10 client.41555.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaff7fe700 15 client.41555.objecter _session_command_op_remove -1 13 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 14 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fdae800e300 0x7fdae8015d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 14 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 7 14 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800f5f0 osd=7 4 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 10 client.41555.objecter _send_command 14 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7fdae800f8c0 con 0x7fdae800e300 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.161+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fdae800e300 0x7fdae8015d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:34:42.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdafe7fc700 10 client.41555.objecter ms_handle_connect 0x7fdae800e300 2024-04-26T16:34:42.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdb0d81c700 1 --2- 172.21.15.50:0/2565493205 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fdae800e300 0x7fdae8015d30 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:34:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdae800f8c0 con 0x7fdae800e300 2024-04-26T16:34:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 14: 0 ) 2024-04-26T16:34:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 14 = system:0 2024-04-26T16:34:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae800f5f0 osd=7 4 2024-04-26T16:34:42.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.165+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 7 14 2024-04-26T16:34:42.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 10 client.41555.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign -1 15 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 20 client.41555.objecter _get_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_remove -1 15 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 20 client.41555.objecter get_session s=0x7fdae800f5f0 osd=7 3 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 15 client.41555.objecter _session_command_op_assign 7 15 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 20 client.41555.objecter put_session s=0x7fdae800f5f0 osd=7 4 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 10 client.41555.objecter _send_command 15 2024-04-26T16:34:42.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.181+0000 7fdaf3fff700 1 -- 172.21.15.50:0/2565493205 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fdae8016270 con 0x7fdae800e300 2024-04-26T16:34:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.189+0000 7fdafe7fc700 1 -- 172.21.15.50:0/2565493205 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fdae8016270 con 0x7fdae800e300 2024-04-26T16:34:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.189+0000 7fdafe7fc700 10 client.41555.objecter ms_dispatch 0x7fdb08001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:34:42.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.189+0000 7fdafe7fc700 10 client.41555.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:34:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.189+0000 7fdafe7fc700 20 client.41555.objecter put_session s=0x7fdae800f5f0 osd=7 4 2024-04-26T16:34:42.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.189+0000 7fdafe7fc700 15 client.41555.objecter _session_command_op_remove 7 15 2024-04-26T16:34:42.186 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:34:42.187 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:34:42.200 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:42.304 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:42.316 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.317+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff3bc1022f0 con 0x7ff3a4071fd0 2024-04-26T16:34:42.334 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:42.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.345+0000 7ff3c0a5d700 1 -- 172.21.15.50:0/909257961 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+532074 (secure 0 0 0) 0x7ff3bc1022f0 con 0x7ff3a4071fd0 2024-04-26T16:34:42.348 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 10 librados: watch_flush enter 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 10 librados: watch_flush exit 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 20 client.41561.objecter shutdown clearing up homeless session... 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 10 client.41561.objecter successfully canceled tick 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3a4071fd0 msgr2=0x7ff3a4074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3a4071fd0 0x7ff3a4074490 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac005950 tx=0x7ff3ac0058e0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 msgr2=0x7ff3bc199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc199740 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0009230 tx=0x7ff3b000b560 comp rx=0 tx=0).stop 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 shutdown_connections 2024-04-26T16:34:42.355 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff3a4071fd0 0x7ff3a4074490 unknown :-1 s=CLOSED pgs=3818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3bc103470 0x7ff3bc199200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff3bc1094f0 0x7ff3bc19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 --2- 172.21.15.50:0/909257961 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff3bc103e30 0x7ff3bc199740 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 >> 172.21.15.50:0/909257961 conn(0x7ff3bc07a5d0 msgr2=0x7ff3bc101be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 shutdown_connections 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 -- 172.21.15.50:0/909257961 wait complete. 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.349+0000 7ff3c1a5f700 1 librados: shutdown 2024-04-26T16:34:42.356 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:34:42.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:42.369+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fdc6c1ac520 con 0x7fdc6c075650 2024-04-26T16:34:42.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:42.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:42.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:42.568 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:42.594 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:42.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:42.722 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:42.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:41.994219+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? 172.21.15.50:0/2565493205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:41.994219+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? 172.21.15.50:0/2565493205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: cluster 2024-04-26T16:34:42.328743+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: cluster 2024-04-26T16:34:42.328743+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.330424+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]': finished 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.330424+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]': finished 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: cluster 2024-04-26T16:34:42.333966+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: cluster 2024-04-26T16:34:42.333966+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.373082+0000 mon.a (mon.0) 3426 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.373082+0000 mon.a (mon.0) 3426 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.561614+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.561614+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.566905+0000 mon.c (mon.2) 743 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:42 smithi190 bash[23868]: audit 2024-04-26T16:34:42.566905+0000 mon.c (mon.2) 743 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:42.854 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:42.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:41.994219+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? 172.21.15.50:0/2565493205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:42.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:41.994219+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? 172.21.15.50:0/2565493205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: cluster 2024-04-26T16:34:42.328743+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: cluster 2024-04-26T16:34:42.328743+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.330424+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]': finished 2024-04-26T16:34:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.330424+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]': finished 2024-04-26T16:34:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: cluster 2024-04-26T16:34:42.333966+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2024-04-26T16:34:42.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: cluster 2024-04-26T16:34:42.333966+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2024-04-26T16:34:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.373082+0000 mon.a (mon.0) 3426 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:34:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.373082+0000 mon.a (mon.0) 3426 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:34:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.561614+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.561614+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.566905+0000 mon.c (mon.2) 743 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:42.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[29234]: audit 2024-04-26T16:34:42.566905+0000 mon.c (mon.2) 743 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:41.994219+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? 172.21.15.50:0/2565493205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:41.994219+0000 mon.c (mon.2) 742 : audit [DBG] from='client.? 172.21.15.50:0/2565493205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:34:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: cluster 2024-04-26T16:34:42.328743+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:42.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: cluster 2024-04-26T16:34:42.328743+0000 mon.a (mon.0) 3423 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.330424+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]': finished 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.330424+0000 mon.a (mon.0) 3424 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-118", "mode": "writeback"}]': finished 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: cluster 2024-04-26T16:34:42.333966+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: cluster 2024-04-26T16:34:42.333966+0000 mon.a (mon.0) 3425 : cluster [DBG] osdmap e704: 8 total, 7 up, 8 in 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.373082+0000 mon.a (mon.0) 3426 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.373082+0000 mon.a (mon.0) 3426 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:34:42.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.561614+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:42.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.561614+0000 mon.a (mon.0) 3427 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:42.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.566905+0000 mon.c (mon.2) 743 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:42.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:42 smithi050 bash[21463]: audit 2024-04-26T16:34:42.566905+0000 mon.c (mon.2) 743 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:42.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:42.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:42.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:43.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:43.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:43.220 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:43.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:43.328 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.328+0000 7fdc53fff700 1 -- 172.21.15.50:0/4169460694 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v705) ==== 91+0+0 (secure 0 0 0) 0x7fdc5c0613f0 con 0x7fdc6c075650 2024-04-26T16:34:43.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 10 librados: watch_flush enter 2024-04-26T16:34:43.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 10 librados: watch_flush exit 2024-04-26T16:34:43.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 20 client.30951.objecter shutdown clearing up homeless session... 2024-04-26T16:34:43.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 10 client.30951.objecter successfully canceled tick 2024-04-26T16:34:43.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc54071f80 msgr2=0x7fdc54074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc54071f80 0x7fdc54074440 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000b730 tx=0x7fdc6001a040 comp rx=0 tx=0).stop 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 msgr2=0x7fdc6c1a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.332+0000 7fdc71103700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 0x7fdc6c1a5420 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00e9a0 tx=0x7fdc5c00ed60 comp rx=0 tx=0).stop 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 shutdown_connections 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdc54071f80 0x7fdc54074440 unknown :-1 s=CLOSED pgs=3819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc6c1145f0 0x7fdc6c19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdc6c075650 0x7fdc6c1a5420 secure :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c00e9a0 tx=0x7fdc5c00ed60 comp rx=0 tx=0).stop 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 --2- 172.21.15.50:0/4169460694 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdc6c075f70 0x7fdc6c1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:34:43.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 >> 172.21.15.50:0/4169460694 conn(0x7fdc6c07a580 msgr2=0x7fdc6c113200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:34:43.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 shutdown_connections 2024-04-26T16:34:43.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 -- 172.21.15.50:0/4169460694 wait complete. 2024-04-26T16:34:43.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:34:43.336+0000 7fdc71103700 1 librados: shutdown 2024-04-26T16:34:43.340 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:34:43.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:43.379 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:43.482 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:43.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:43.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:43.643 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:43.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:43.746 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:43.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: cluster 2024-04-26T16:34:41.860783+0000 mgr.y (mgr.24401) 667 : cluster [DBG] pgmap v1094: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:43.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: cluster 2024-04-26T16:34:41.860783+0000 mgr.y (mgr.24401) 667 : cluster [DBG] pgmap v1094: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:42.322250+0000 mgr.y (mgr.24401) 668 : audit [DBG] from='client.41561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:42.322250+0000 mgr.y (mgr.24401) 668 : audit [DBG] from='client.41561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: cluster 2024-04-26T16:34:43.331278+0000 mon.a (mon.0) 3428 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: cluster 2024-04-26T16:34:43.331278+0000 mon.a (mon.0) 3428 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:43.333871+0000 mon.a (mon.0) 3429 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:43.333871+0000 mon.a (mon.0) 3429 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: cluster 2024-04-26T16:34:43.343822+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e705: 8 total, 7 up, 8 in 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: cluster 2024-04-26T16:34:43.343822+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e705: 8 total, 7 up, 8 in 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:43.388970+0000 mon.c (mon.2) 744 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:43.388970+0000 mon.c (mon.2) 744 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:43.389369+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[21463]: audit 2024-04-26T16:34:43.389369+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: cluster 2024-04-26T16:34:41.860783+0000 mgr.y (mgr.24401) 667 : cluster [DBG] pgmap v1094: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: cluster 2024-04-26T16:34:41.860783+0000 mgr.y (mgr.24401) 667 : cluster [DBG] pgmap v1094: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:42.322250+0000 mgr.y (mgr.24401) 668 : audit [DBG] from='client.41561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:42.322250+0000 mgr.y (mgr.24401) 668 : audit [DBG] from='client.41561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: cluster 2024-04-26T16:34:43.331278+0000 mon.a (mon.0) 3428 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: cluster 2024-04-26T16:34:43.331278+0000 mon.a (mon.0) 3428 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:43.333871+0000 mon.a (mon.0) 3429 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:43.333871+0000 mon.a (mon.0) 3429 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:34:43.760 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: cluster 2024-04-26T16:34:43.343822+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e705: 8 total, 7 up, 8 in 2024-04-26T16:34:43.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: cluster 2024-04-26T16:34:43.343822+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e705: 8 total, 7 up, 8 in 2024-04-26T16:34:43.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:43.388970+0000 mon.c (mon.2) 744 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:43.388970+0000 mon.c (mon.2) 744 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:43.389369+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.761 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:43 smithi050 bash[29234]: audit 2024-04-26T16:34:43.389369+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:43.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:43.905 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: cluster 2024-04-26T16:34:41.860783+0000 mgr.y (mgr.24401) 667 : cluster [DBG] pgmap v1094: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: cluster 2024-04-26T16:34:41.860783+0000 mgr.y (mgr.24401) 667 : cluster [DBG] pgmap v1094: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:42.322250+0000 mgr.y (mgr.24401) 668 : audit [DBG] from='client.41561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:42.322250+0000 mgr.y (mgr.24401) 668 : audit [DBG] from='client.41561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: cluster 2024-04-26T16:34:43.331278+0000 mon.a (mon.0) 3428 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: cluster 2024-04-26T16:34:43.331278+0000 mon.a (mon.0) 3428 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:43.333871+0000 mon.a (mon.0) 3429 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:43.333871+0000 mon.a (mon.0) 3429 : audit [INF] from='client.? 172.21.15.50:0/4169460694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: cluster 2024-04-26T16:34:43.343822+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e705: 8 total, 7 up, 8 in 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: cluster 2024-04-26T16:34:43.343822+0000 mon.a (mon.0) 3430 : cluster [DBG] osdmap e705: 8 total, 7 up, 8 in 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:43.388970+0000 mon.c (mon.2) 744 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:43.388970+0000 mon.c (mon.2) 744 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.908 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:43.389369+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.908 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:43 smithi190 bash[23868]: audit 2024-04-26T16:34:43.389369+0000 mon.a (mon.0) 3431 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:43.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:44.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:44.038 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:44.141 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:44.170 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:44.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:44.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:44.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:44.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:44.432 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:44.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:44.565 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:44.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:44.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:44.679 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:34:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:34:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:34:44.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:44.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:44.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:44.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:44.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:44.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:45.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:45.085 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:45.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:45.217 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:45.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:45.349 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:45.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:45.480 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:45.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:45.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:43.862615+0000 mgr.y (mgr.24401) 669 : cluster [DBG] pgmap v1097: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:45.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:43.862615+0000 mgr.y (mgr.24401) 669 : cluster [DBG] pgmap v1097: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:45.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: audit 2024-04-26T16:34:44.336511+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:45.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: audit 2024-04-26T16:34:44.336511+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:45.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.344043+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.344043+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: audit 2024-04-26T16:34:44.346475+0000 mon.c (mon.2) 745 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: audit 2024-04-26T16:34:44.346475+0000 mon.c (mon.2) 745 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: audit 2024-04-26T16:34:44.348495+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: audit 2024-04-26T16:34:44.348495+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.750017+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.d scrub starts 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.750017+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.d scrub starts 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.752727+0000 osd.1 (osd.1) 372 : cluster [DBG] 2.d scrub ok 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.752727+0000 osd.1 (osd.1) 372 : cluster [DBG] 2.d scrub ok 2024-04-26T16:34:45.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:43.862615+0000 mgr.y (mgr.24401) 669 : cluster [DBG] pgmap v1097: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:43.862615+0000 mgr.y (mgr.24401) 669 : cluster [DBG] pgmap v1097: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: audit 2024-04-26T16:34:44.336511+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: audit 2024-04-26T16:34:44.336511+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.344043+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.344043+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: audit 2024-04-26T16:34:44.346475+0000 mon.c (mon.2) 745 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: audit 2024-04-26T16:34:44.346475+0000 mon.c (mon.2) 745 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: audit 2024-04-26T16:34:44.348495+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: audit 2024-04-26T16:34:44.348495+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.750017+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.d scrub starts 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.750017+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.d scrub starts 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.752727+0000 osd.1 (osd.1) 372 : cluster [DBG] 2.d scrub ok 2024-04-26T16:34:45.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:45 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.752727+0000 osd.1 (osd.1) 372 : cluster [DBG] 2.d scrub ok 2024-04-26T16:34:45.609 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:45.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:45.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:43.862615+0000 mgr.y (mgr.24401) 669 : cluster [DBG] pgmap v1097: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:45.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:43.862615+0000 mgr.y (mgr.24401) 669 : cluster [DBG] pgmap v1097: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:45.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: audit 2024-04-26T16:34:44.336511+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:45.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: audit 2024-04-26T16:34:44.336511+0000 mon.a (mon.0) 3432 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:45.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.344043+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2024-04-26T16:34:45.724 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.344043+0000 mon.a (mon.0) 3433 : cluster [DBG] osdmap e706: 8 total, 7 up, 8 in 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: audit 2024-04-26T16:34:44.346475+0000 mon.c (mon.2) 745 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: audit 2024-04-26T16:34:44.346475+0000 mon.c (mon.2) 745 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: audit 2024-04-26T16:34:44.348495+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: audit 2024-04-26T16:34:44.348495+0000 mon.a (mon.0) 3434 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]: dispatch 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.750017+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.d scrub starts 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.750017+0000 osd.1 (osd.1) 371 : cluster [DBG] 2.d scrub starts 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.752727+0000 osd.1 (osd.1) 372 : cluster [DBG] 2.d scrub ok 2024-04-26T16:34:45.725 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:45 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.752727+0000 osd.1 (osd.1) 372 : cluster [DBG] 2.d scrub ok 2024-04-26T16:34:45.741 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:45.845 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:45.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:45.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:45.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:46.003 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:46.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:46.134 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:46.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:46.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:46.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:46.397 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:46.500 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:46.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.889503+0000 osd.2 (osd.2) 341 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:34:46.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.889503+0000 osd.2 (osd.2) 341 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:34:46.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.891487+0000 osd.2 (osd.2) 342 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:34:46.513 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:44.891487+0000 osd.2 (osd.2) 342 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.122551+0000 osd.3 (osd.3) 401 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.122551+0000 osd.3 (osd.3) 401 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.123015+0000 osd.3 (osd.3) 402 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.123015+0000 osd.3 (osd.3) 402 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.337004+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.337004+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: audit 2024-04-26T16:34:45.339478+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: audit 2024-04-26T16:34:45.339478+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.351296+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.351296+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.909758+0000 osd.0 (osd.0) 395 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.909758+0000 osd.0 (osd.0) 395 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.912225+0000 osd.0 (osd.0) 396 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.912225+0000 osd.0 (osd.0) 396 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:46.327927+0000 mon.a (mon.0) 3438 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:46.514 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[21463]: cluster 2024-04-26T16:34:46.327927+0000 mon.a (mon.0) 3438 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.889503+0000 osd.2 (osd.2) 341 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.889503+0000 osd.2 (osd.2) 341 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.891487+0000 osd.2 (osd.2) 342 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:44.891487+0000 osd.2 (osd.2) 342 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.122551+0000 osd.3 (osd.3) 401 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.122551+0000 osd.3 (osd.3) 401 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.123015+0000 osd.3 (osd.3) 402 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.123015+0000 osd.3 (osd.3) 402 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.337004+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.337004+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: audit 2024-04-26T16:34:45.339478+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: audit 2024-04-26T16:34:45.339478+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:46.515 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.351296+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.351296+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.909758+0000 osd.0 (osd.0) 395 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.909758+0000 osd.0 (osd.0) 395 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.912225+0000 osd.0 (osd.0) 396 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.912225+0000 osd.0 (osd.0) 396 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:46.327927+0000 mon.a (mon.0) 3438 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:46.516 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:46 smithi050 bash[29234]: cluster 2024-04-26T16:34:46.327927+0000 mon.a (mon.0) 3438 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:34:46.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:34:46.547 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1094,"stamp":"2024-04-26T16:34:41.859196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596560,"num_objects":208,"num_object_clones":2,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":1,"num_read":2390,"num_read_kb":2231,"num_write":920,"num_write_kb":2479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1774538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1078,"ondisk_log_size":1078,"up":543,"acting":543,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":613,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":691888,"kb_used_data":28916,"kb_used_omap":26,"kb_used_meta":662821,"kb_avail":749269328,"statfs":{"total":767960285184,"available":767251791872,"internally_reserved":0,"allocated":29609984,"data_stored":13290672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26682,"internal_metadata":678729670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.003645"},"pg_stats":[{"pgid":"327.1f","version":"0'0","reported_seq":14,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.360985+0000","last_change":"2024-04-26T16:34:38.657419+0000","last_active":"2024-04-26T16:34:41.360985+0000","last_peered":"2024-04-26T16:34:41.360985+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.657419+0000","last_became_peered":"2024-04-26T16:34:38.657419+0000","last_unstale":"2024-04-26T16:34:41.360985+0000","last_undegraded":"2024-04-26T16:34:41.360985+0000","last_fullsized":"2024-04-26T16:34:38.626616+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:38.963391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"327.1e","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334362+0000","last_change":"2024-04-26T16:34:38.650603+0000","last_active":"2024-04-26T16:34:41.334362+0000","last_peered":"2024-04-26T16:34:41.334362+0000","last_clean":"2024-04-26T16:34:41.334362+0000","last_became_active":"2024-04-26T16:34:38.650224+0000","last_became_peered":"2024-04-26T16:34:38.650224+0000","last_unstale":"2024-04-26T16:34:41.334362+0000","last_undegraded":"2024-04-26T16:34:41.334362+0000","last_fullsized":"2024-04-26T16:34:41.334362+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:38.702040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"327.1d","version":"0'0","reported_seq":16,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.334606+0000","last_change":"2024-04-26T16:34:38.635749+0000","last_active":"2024-04-26T16:34:41.334606+0000","last_peered":"2024-04-26T16:34:41.334606+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.635749+0000","last_became_peered":"2024-04-26T16:34:38.635749+0000","last_unstale":"2024-04-26T16:34:41.334606+0000","last_undegraded":"2024-04-26T16:34:41.334606+0000","last_fullsized":"2024-04-26T16:34:38.629353+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:59.159452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"327.1c","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334788+0000","last_change":"2024-04-26T16:34:38.670531+0000","last_active":"2024-04-26T16:34:41.334788+0000","last_peered":"2024-04-26T16:34:41.334788+0000","last_clean":"2024-04-26T16:34:41.334788+0000","last_became_active":"2024-04-26T16:34:38.670199+0000","last_became_peered":"2024-04-26T16:34:38.670199+0000","last_unstale":"2024-04-26T16:34:41.334788+0000","last_undegraded":"2024-04-26T16:34:41.334788+0000","last_fullsized":"2024-04-26T16:34:41.334788+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:53.772931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.1b","version":"0'0","reported_seq":17,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.462623+0000","last_change":"2024-04-26T16:34:38.633997+0000","last_active":"2024-04-26T16:34:41.462623+0000","last_peered":"2024-04-26T16:34:41.462623+0000","last_clean":"2024-04-26T16:34:41.462623+0000","last_became_active":"2024-04-26T16:34:38.633554+0000","last_became_peered":"2024-04-26T16:34:38.633554+0000","last_unstale":"2024-04-26T16:34:41.462623+0000","last_undegraded":"2024-04-26T16:34:41.462623+0000","last_fullsized":"2024-04-26T16:34:41.462623+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:00.547144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"327.1a","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331791+0000","last_change":"2024-04-26T16:34:38.667310+0000","last_active":"2024-04-26T16:34:41.331791+0000","last_peered":"2024-04-26T16:34:41.331791+0000","last_clean":"2024-04-26T16:34:41.331791+0000","last_became_active":"2024-04-26T16:34:38.660720+0000","last_became_peered":"2024-04-26T16:34:38.660720+0000","last_unstale":"2024-04-26T16:34:41.331791+0000","last_undegraded":"2024-04-26T16:34:41.331791+0000","last_fullsized":"2024-04-26T16:34:41.331791+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:43.982223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.19","version":"0'0","reported_seq":17,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.461992+0000","last_change":"2024-04-26T16:34:38.660755+0000","last_active":"2024-04-26T16:34:41.461992+0000","last_peered":"2024-04-26T16:34:41.461992+0000","last_clean":"2024-04-26T16:34:41.461992+0000","last_became_active":"2024-04-26T16:34:38.660209+0000","last_became_peered":"2024-04-26T16:34:38.660209+0000","last_unstale":"2024-04-26T16:34:41.461992+0000","last_undegraded":"2024-04-26T16:34:41.461992+0000","last_fullsized":"2024-04-26T16:34:41.461992+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:55.839694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"327.18","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331361+0000","last_change":"2024-04-26T16:34:38.661450+0000","last_active":"2024-04-26T16:34:41.331361+0000","last_peered":"2024-04-26T16:34:41.331361+0000","last_clean":"2024-04-26T16:34:41.331361+0000","last_became_active":"2024-04-26T16:34:38.661104+0000","last_became_peered":"2024-04-26T16:34:38.661104+0000","last_unstale":"2024-04-26T16:34:41.331361+0000","last_undegraded":"2024-04-26T16:34:41.331361+0000","last_fullsized":"2024-04-26T16:34:41.331361+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:04.953611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.17","version":"0'0","reported_seq":13,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737747+0000","last_change":"2024-04-26T16:34:38.654318+0000","last_active":"2024-04-26T16:34:39.737747+0000","last_peered":"2024-04-26T16:34:39.737747+0000","last_clean":"2024-04-26T16:34:39.737747+0000","last_became_active":"2024-04-26T16:34:38.653997+0000","last_became_peered":"2024-04-26T16:34:38.653997+0000","last_unstale":"2024-04-26T16:34:39.737747+0000","last_undegraded":"2024-04-26T16:34:39.737747+0000","last_fullsized":"2024-04-26T16:34:39.737747+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:02.267035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"327.16","version":"0'0","reported_seq":16,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.334160+0000","last_change":"2024-04-26T16:34:38.648012+0000","last_active":"2024-04-26T16:34:41.334160+0000","last_peered":"2024-04-26T16:34:41.334160+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.648012+0000","last_became_peered":"2024-04-26T16:34:38.648012+0000","last_unstale":"2024-04-26T16:34:41.334160+0000","last_undegraded":"2024-04-26T16:34:41.334160+0000","last_fullsized":"2024-04-26T16:34:38.630785+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:07.111379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"327.15","version":"0'0","reported_seq":16,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.335039+0000","last_change":"2024-04-26T16:34:38.656960+0000","last_active":"2024-04-26T16:34:41.335039+0000","last_peered":"2024-04-26T16:34:41.335039+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.656960+0000","last_became_peered":"2024-04-26T16:34:38.656960+0000","last_unstale":"2024-04-26T16:34:41.335039+0000","last_undegraded":"2024-04-26T16:34:41.335039+0000","last_fullsized":"2024-04-26T16:34:38.643056+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:43.106644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.14","version":"0'0","reported_seq":13,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.736680+0000","last_change":"2024-04-26T16:34:38.659513+0000","last_active":"2024-04-26T16:34:39.736680+0000","last_peered":"2024-04-26T16:34:39.736680+0000","last_clean":"2024-04-26T16:34:39.736680+0000","last_became_active":"2024-04-26T16:34:38.658821+0000","last_became_peered":"2024-04-26T16:34:38.658821+0000","last_unstale":"2024-04-26T16:34:39.736680+0000","last_undegraded":"2024-04-26T16:34:39.736680+0000","last_fullsized":"2024-04-26T16:34:39.736680+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:47.566666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"327.13","version":"0'0","reported_seq":16,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.336018+0000","last_change":"2024-04-26T16:34:38.635864+0000","last_active":"2024-04-26T16:34:41.336018+0000","last_peered":"2024-04-26T16:34:41.336018+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.635864+0000","last_became_peered":"2024-04-26T16:34:38.635864+0000","last_unstale":"2024-04-26T16:34:41.336018+0000","last_undegraded":"2024-04-26T16:34:41.336018+0000","last_fullsized":"2024-04-26T16:34:38.626068+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:59.068453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"327.12","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334706+0000","last_change":"2024-04-26T16:34:38.666554+0000","last_active":"2024-04-26T16:34:41.334706+0000","last_peered":"2024-04-26T16:34:41.334706+0000","last_clean":"2024-04-26T16:34:41.334706+0000","last_became_active":"2024-04-26T16:34:38.660671+0000","last_became_peered":"2024-04-26T16:34:38.660671+0000","last_unstale":"2024-04-26T16:34:41.334706+0000","last_undegraded":"2024-04-26T16:34:41.334706+0000","last_fullsized":"2024-04-26T16:34:41.334706+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:05.381275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.11","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.337544+0000","last_change":"2024-04-26T16:34:38.644890+0000","last_active":"2024-04-26T16:34:41.337544+0000","last_peered":"2024-04-26T16:34:41.337544+0000","last_clean":"2024-04-26T16:34:41.337544+0000","last_became_active":"2024-04-26T16:34:38.644440+0000","last_became_peered":"2024-04-26T16:34:38.644440+0000","last_unstale":"2024-04-26T16:34:41.337544+0000","last_undegraded":"2024-04-26T16:34:41.337544+0000","last_fullsized":"2024-04-26T16:34:41.337544+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.047911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"327.10","version":"0'0","reported_seq":14,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.461997+0000","last_change":"2024-04-26T16:34:38.654431+0000","last_active":"2024-04-26T16:34:41.461997+0000","last_peered":"2024-04-26T16:34:41.461997+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.654431+0000","last_became_peered":"2024-04-26T16:34:38.654431+0000","last_unstale":"2024-04-26T16:34:41.461997+0000","last_undegraded":"2024-04-26T16:34:41.461997+0000","last_fullsized":"2024-04-26T16:34:38.615309+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:06.306172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"327.f","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.335654+0000","last_change":"2024-04-26T16:34:38.666637+0000","last_active":"2024-04-26T16:34:41.335654+0000","last_peered":"2024-04-26T16:34:41.335654+0000","last_clean":"2024-04-26T16:34:41.335654+0000","last_became_active":"2024-04-26T16:34:38.660069+0000","last_became_peered":"2024-04-26T16:34:38.660069+0000","last_unstale":"2024-04-26T16:34:41.335654+0000","last_undegraded":"2024-04-26T16:34:41.335654+0000","last_fullsized":"2024-04-26T16:34:41.335654+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:41.888892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.e","version":"0'0","reported_seq":13,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.738298+0000","last_change":"2024-04-26T16:34:38.666468+0000","last_active":"2024-04-26T16:34:39.738298+0000","last_peered":"2024-04-26T16:34:39.738298+0000","last_clean":"2024-04-26T16:34:39.738298+0000","last_became_active":"2024-04-26T16:34:38.665793+0000","last_became_peered":"2024-04-26T16:34:38.665793+0000","last_unstale":"2024-04-26T16:34:39.738298+0000","last_undegraded":"2024-04-26T16:34:39.738298+0000","last_fullsized":"2024-04-26T16:34:39.738298+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:37.723878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"327.d","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333048+0000","last_change":"2024-04-26T16:34:38.665464+0000","last_active":"2024-04-26T16:34:41.333048+0000","last_peered":"2024-04-26T16:34:41.333048+0000","last_clean":"2024-04-26T16:34:41.333048+0000","last_became_active":"2024-04-26T16:34:38.662035+0000","last_became_peered":"2024-04-26T16:34:38.662035+0000","last_unstale":"2024-04-26T16:34:41.333048+0000","last_undegraded":"2024-04-26T16:34:41.333048+0000","last_fullsized":"2024-04-26T16:34:41.333048+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:57.742380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"327.c","version":"0'0","reported_seq":10,"reported_epoch":701,"state":"active+undersized","last_fresh":"2024-04-26T16:34:39.736502+0000","last_change":"2024-04-26T16:34:38.667956+0000","last_active":"2024-04-26T16:34:39.736502+0000","last_peered":"2024-04-26T16:34:39.736502+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.667956+0000","last_became_peered":"2024-04-26T16:34:38.667956+0000","last_unstale":"2024-04-26T16:34:39.736502+0000","last_undegraded":"2024-04-26T16:34:39.736502+0000","last_fullsized":"2024-04-26T16:34:38.632036+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:48.651996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"327.b","version":"0'0","reported_seq":13,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.736727+0000","last_change":"2024-04-26T16:34:38.661795+0000","last_active":"2024-04-26T16:34:39.736727+0000","last_peered":"2024-04-26T16:34:39.736727+0000","last_clean":"2024-04-26T16:34:39.736727+0000","last_became_active":"2024-04-26T16:34:38.661450+0000","last_became_peered":"2024-04-26T16:34:38.661450+0000","last_unstale":"2024-04-26T16:34:39.736727+0000","last_undegraded":"2024-04-26T16:34:39.736727+0000","last_fullsized":"2024-04-26T16:34:39.736727+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:43.504380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"327.a","version":"0'0","reported_seq":16,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.332946+0000","last_change":"2024-04-26T16:34:38.661547+0000","last_active":"2024-04-26T16:34:41.332946+0000","last_peered":"2024-04-26T16:34:41.332946+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.661547+0000","last_became_peered":"2024-04-26T16:34:38.661547+0000","last_unstale":"2024-04-26T16:34:41.332946+0000","last_undegraded":"2024-04-26T16:34:41.332946+0000","last_fullsized":"2024-04-26T16:34:38.624506+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:58.531665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"327.9","version":"0'0","reported_seq":10,"reported_epoch":701,"state":"active+undersized","last_fresh":"2024-04-26T16:34:39.736407+0000","last_change":"2024-04-26T16:34:38.663318+0000","last_active":"2024-04-26T16:34:39.736407+0000","last_peered":"2024-04-26T16:34:39.736407+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.663318+0000","last_became_peered":"2024-04-26T16:34:38.663318+0000","last_unstale":"2024-04-26T16:34:39.736407+0000","last_undegraded":"2024-04-26T16:34:39.736407+0000","last_fullsized":"2024-04-26T16:34:38.631827+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:52.012928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"327.8","version":"0'0","reported_seq":14,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.357342+0000","last_change":"2024-04-26T16:34:38.640440+0000","last_active":"2024-04-26T16:34:41.357342+0000","last_peered":"2024-04-26T16:34:41.357342+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.640440+0000","last_became_peered":"2024-04-26T16:34:38.640440+0000","last_unstale":"2024-04-26T16:34:41.357342+0000","last_undegraded":"2024-04-26T16:34:41.357342+0000","last_fullsized":"2024-04-26T16:34:38.616873+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:52.871546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"327.3","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334573+0000","last_change":"2024-04-26T16:34:38.667218+0000","last_active":"2024-04-26T16:34:41.334573+0000","last_peered":"2024-04-26T16:34:41.334573+0000","last_clean":"2024-04-26T16:34:41.334573+0000","last_became_active":"2024-04-26T16:34:38.666857+0000","last_became_peered":"2024-04-26T16:34:38.666857+0000","last_unstale":"2024-04-26T16:34:41.334573+0000","last_undegraded":"2024-04-26T16:34:41.334573+0000","last_fullsized":"2024-04-26T16:34:41.334573+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.398592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.7","version":"684'3","reported_seq":95,"reported_epoch":702,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:40.633563+0000","last_change":"2024-04-26T16:33:56.354617+0000","last_active":"2024-04-26T16:34:40.633563+0000","last_peered":"2024-04-26T16:34:40.633563+0000","last_clean":"2024-04-26T16:33:38.361109+0000","last_became_active":"2024-04-26T16:33:39.395238+0000","last_became_peered":"2024-04-26T16:33:39.395238+0000","last_unstale":"2024-04-26T16:34:40.633563+0000","last_undegraded":"2024-04-26T16:33:55.373093+0000","last_fullsized":"2024-04-26T16:33:39.388686+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.165827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2147483647,1],"acting":[5,2147483647,1],"avail_no_missing":["5(0)","1(2)"],"object_location_counts":[{"shards":"1(2),5(0)","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"327.2","version":"0'0","reported_seq":17,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.360691+0000","last_change":"2024-04-26T16:34:38.658835+0000","last_active":"2024-04-26T16:34:41.360691+0000","last_peered":"2024-04-26T16:34:41.360691+0000","last_clean":"2024-04-26T16:34:41.360691+0000","last_became_active":"2024-04-26T16:34:38.658137+0000","last_became_peered":"2024-04-26T16:34:38.658137+0000","last_unstale":"2024-04-26T16:34:41.360691+0000","last_undegraded":"2024-04-26T16:34:41.360691+0000","last_fullsized":"2024-04-26T16:34:41.360691+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:37.623418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.6","version":"0'0","reported_seq":104,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334923+0000","last_change":"2024-04-26T16:34:14.499714+0000","last_active":"2024-04-26T16:34:41.334923+0000","last_peered":"2024-04-26T16:34:41.334923+0000","last_clean":"2024-04-26T16:34:41.334923+0000","last_became_active":"2024-04-26T16:33:39.394825+0000","last_became_peered":"2024-04-26T16:33:39.394825+0000","last_unstale":"2024-04-26T16:34:41.334923+0000","last_undegraded":"2024-04-26T16:34:41.334923+0000","last_fullsized":"2024-04-26T16:34:41.334923+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0028447279999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"327.1","version":"0'0","reported_seq":16,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.630542+0000","last_change":"2024-04-26T16:34:38.667255+0000","last_active":"2024-04-26T16:34:40.630542+0000","last_peered":"2024-04-26T16:34:40.630542+0000","last_clean":"2024-04-26T16:34:40.630542+0000","last_became_active":"2024-04-26T16:34:38.666770+0000","last_became_peered":"2024-04-26T16:34:38.666770+0000","last_unstale":"2024-04-26T16:34:40.630542+0000","last_undegraded":"2024-04-26T16:34:40.630542+0000","last_fullsized":"2024-04-26T16:34:40.630542+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:43.387783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"323.5","version":"0'0","reported_seq":102,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.464701+0000","last_change":"2024-04-26T16:34:14.506193+0000","last_active":"2024-04-26T16:34:41.464701+0000","last_peered":"2024-04-26T16:34:41.464701+0000","last_clean":"2024-04-26T16:34:41.464701+0000","last_became_active":"2024-04-26T16:33:39.411800+0000","last_became_peered":"2024-04-26T16:33:39.411800+0000","last_unstale":"2024-04-26T16:34:41.464701+0000","last_undegraded":"2024-04-26T16:34:41.464701+0000","last_fullsized":"2024-04-26T16:34:41.464701+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:43.332856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035017490000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"327.0","version":"0'0","reported_seq":13,"reported_epoch":702,"state":"active+undersized","last_fresh":"2024-04-26T16:34:40.630552+0000","last_change":"2024-04-26T16:34:38.641972+0000","last_active":"2024-04-26T16:34:40.630552+0000","last_peered":"2024-04-26T16:34:40.630552+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.641972+0000","last_became_peered":"2024-04-26T16:34:38.641972+0000","last_unstale":"2024-04-26T16:34:40.630552+0000","last_undegraded":"2024-04-26T16:34:40.630552+0000","last_fullsized":"2024-04-26T16:34:38.630682+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:00.948812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"323.4","version":"692'6","reported_seq":104,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.736738+0000","last_change":"2024-04-26T16:34:14.523840+0000","last_active":"2024-04-26T16:34:39.736738+0000","last_peered":"2024-04-26T16:34:39.736738+0000","last_clean":"2024-04-26T16:34:39.736738+0000","last_became_active":"2024-04-26T16:33:39.406081+0000","last_became_peered":"2024-04-26T16:33:39.406081+0000","last_unstale":"2024-04-26T16:34:39.736738+0000","last_undegraded":"2024-04-26T16:34:39.736738+0000","last_fullsized":"2024-04-26T16:34:39.736738+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.780427+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0065697539999999997,"stat_sum":{"num_bytes":9,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"327.7","version":"0'0","reported_seq":14,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.361953+0000","last_change":"2024-04-26T16:34:38.656125+0000","last_active":"2024-04-26T16:34:41.361953+0000","last_peered":"2024-04-26T16:34:41.361953+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.656125+0000","last_became_peered":"2024-04-26T16:34:38.656125+0000","last_unstale":"2024-04-26T16:34:41.361953+0000","last_undegraded":"2024-04-26T16:34:41.361953+0000","last_fullsized":"2024-04-26T16:34:38.628971+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:46.372870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"323.3","version":"0'0","reported_seq":95,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.358050+0000","last_change":"2024-04-26T16:33:39.398035+0000","last_active":"2024-04-26T16:34:41.358050+0000","last_peered":"2024-04-26T16:34:41.358050+0000","last_clean":"2024-04-26T16:33:38.361109+0000","last_became_active":"2024-04-26T16:33:39.398035+0000","last_became_peered":"2024-04-26T16:33:39.398035+0000","last_unstale":"2024-04-26T16:34:41.358050+0000","last_undegraded":"2024-04-26T16:34:41.358050+0000","last_fullsized":"2024-04-26T16:33:39.372843+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.333328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2147483647,4],"acting":[3,2147483647,4],"avail_no_missing":["3(0)","4(2)"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"327.6","version":"0'0","reported_seq":16,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.333142+0000","last_change":"2024-04-26T16:34:38.659019+0000","last_active":"2024-04-26T16:34:41.333142+0000","last_peered":"2024-04-26T16:34:41.333142+0000","last_clean":"2024-04-26T16:34:37.593464+0000","last_became_active":"2024-04-26T16:34:38.659019+0000","last_became_peered":"2024-04-26T16:34:38.659019+0000","last_unstale":"2024-04-26T16:34:41.333142+0000","last_undegraded":"2024-04-26T16:34:41.333142+0000","last_fullsized":"2024-04-26T16:34:38.631345+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:02.728308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"323.2","version":"692'10","reported_seq":117,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333180+0000","last_change":"2024-04-26T16:34:16.506942+0000","last_active":"2024-04-26T16:34:41.333180+0000","last_peered":"2024-04-26T16:34:41.333180+0000","last_clean":"2024-04-26T16:34:41.333180+0000","last_became_active":"2024-04-26T16:33:39.413256+0000","last_became_peered":"2024-04-26T16:33:39.413256+0000","last_unstale":"2024-04-26T16:34:41.333180+0000","last_undegraded":"2024-04-26T16:34:41.333180+0000","last_fullsized":"2024-04-26T16:34:41.333180+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:58.510665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054754640000000002,"stat_sum":{"num_bytes":10,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"327.5","version":"0'0","reported_seq":19,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334122+0000","last_change":"2024-04-26T16:34:38.669815+0000","last_active":"2024-04-26T16:34:41.334122+0000","last_peered":"2024-04-26T16:34:41.334122+0000","last_clean":"2024-04-26T16:34:41.334122+0000","last_became_active":"2024-04-26T16:34:38.669349+0000","last_became_peered":"2024-04-26T16:34:38.669349+0000","last_unstale":"2024-04-26T16:34:41.334122+0000","last_undegraded":"2024-04-26T16:34:41.334122+0000","last_fullsized":"2024-04-26T16:34:41.334122+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:01.497505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"323.1","version":"675'1","reported_seq":105,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332463+0000","last_change":"2024-04-26T16:34:14.504856+0000","last_active":"2024-04-26T16:34:41.332463+0000","last_peered":"2024-04-26T16:34:41.332463+0000","last_clean":"2024-04-26T16:34:41.332463+0000","last_became_active":"2024-04-26T16:33:39.394480+0000","last_became_peered":"2024-04-26T16:33:39.394480+0000","last_unstale":"2024-04-26T16:34:41.332463+0000","last_undegraded":"2024-04-26T16:34:41.332463+0000","last_fullsized":"2024-04-26T16:34:41.332463+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":667,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.114477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010958931,"stat_sum":{"num_bytes":8,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"327.4","version":"0'0","reported_seq":16,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.632535+0000","last_change":"2024-04-26T16:34:38.666971+0000","last_active":"2024-04-26T16:34:40.632535+0000","last_peered":"2024-04-26T16:34:40.632535+0000","last_clean":"2024-04-26T16:34:40.632535+0000","last_became_active":"2024-04-26T16:34:38.666558+0000","last_became_peered":"2024-04-26T16:34:38.666558+0000","last_unstale":"2024-04-26T16:34:40.632535+0000","last_undegraded":"2024-04-26T16:34:40.632535+0000","last_fullsized":"2024-04-26T16:34:40.632535+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":699,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:34:37.593464+0000","last_clean_scrub_stamp":"2024-04-26T16:34:37.593464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:50.861305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"323.0","version":"701'3","reported_seq":99,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.331851+0000","last_change":"2024-04-26T16:34:39.619856+0000","last_active":"2024-04-26T16:34:41.331851+0000","last_peered":"2024-04-26T16:34:41.331851+0000","last_clean":"2024-04-26T16:33:38.361109+0000","last_became_active":"2024-04-26T16:33:39.388092+0000","last_became_peered":"2024-04-26T16:33:39.388092+0000","last_unstale":"2024-04-26T16:34:41.331851+0000","last_undegraded":"2024-04-26T16:34:38.660469+0000","last_fullsized":"2024-04-26T16:33:39.381136+0000","mapping_epoch":666,"log_start":"0'0","ondisk_log_start":"0'0","created":666,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:38.361109+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.361109+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2147483647],"acting":[1,4,2147483647],"avail_no_missing":["1(0)","4(1)"],"object_location_counts":[{"shards":"1(0),4(1)","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.634430+0000","last_change":"2024-04-26T16:33:56.069998+0000","last_active":"2024-04-26T16:34:40.634430+0000","last_peered":"2024-04-26T16:34:40.634430+0000","last_clean":"2024-04-26T16:34:40.634430+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:34:40.634430+0000","last_undegraded":"2024-04-26T16:34:40.634430+0000","last_fullsized":"2024-04-26T16:34:40.634430+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:56.069784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:56.069784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:21.210472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"630'15","reported_seq":1522,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.335867+0000","last_change":"2024-04-26T16:33:02.314019+0000","last_active":"2024-04-26T16:34:41.335867+0000","last_peered":"2024-04-26T16:34:41.335867+0000","last_clean":"2024-04-26T16:33:01.315658+0000","last_became_active":"2024-04-26T16:33:02.314019+0000","last_became_peered":"2024-04-26T16:33:02.314019+0000","last_unstale":"2024-04-26T16:34:41.335867+0000","last_undegraded":"2024-04-26T16:33:02.305255+0000","last_fullsized":"2024-04-26T16:33:02.305084+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:29:10.136172+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"458'34","reported_seq":1373,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.337206+0000","last_change":"2024-04-26T16:33:19.589656+0000","last_active":"2024-04-26T16:34:41.337206+0000","last_peered":"2024-04-26T16:34:41.337206+0000","last_clean":"2024-04-26T16:34:41.337206+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:34:41.337206+0000","last_undegraded":"2024-04-26T16:34:41.337206+0000","last_fullsized":"2024-04-26T16:34:41.337206+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:33:19.589375+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:33:19.589375+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:46.758750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1319,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334763+0000","last_change":"2024-04-26T16:33:19.749849+0000","last_active":"2024-04-26T16:34:41.334763+0000","last_peered":"2024-04-26T16:34:41.334763+0000","last_clean":"2024-04-26T16:34:41.334763+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:34:41.334763+0000","last_undegraded":"2024-04-26T16:34:41.334763+0000","last_fullsized":"2024-04-26T16:34:41.334763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:19.749730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:33:19.749730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.886852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":241,"reported_epoch":702,"state":"active+undersized","last_fresh":"2024-04-26T16:34:40.631718+0000","last_change":"2024-04-26T16:33:02.307160+0000","last_active":"2024-04-26T16:34:40.631718+0000","last_peered":"2024-04-26T16:34:40.631718+0000","last_clean":"2024-04-26T16:33:00.424958+0000","last_became_active":"2024-04-26T16:33:02.307160+0000","last_became_peered":"2024-04-26T16:33:02.307160+0000","last_unstale":"2024-04-26T16:34:40.631718+0000","last_undegraded":"2024-04-26T16:34:40.631718+0000","last_fullsized":"2024-04-26T16:33:02.291975+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:16.230516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.230516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1161,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.333352+0000","last_change":"2024-04-26T16:33:02.320002+0000","last_active":"2024-04-26T16:34:41.333352+0000","last_peered":"2024-04-26T16:34:41.333352+0000","last_clean":"2024-04-26T16:33:01.305664+0000","last_became_active":"2024-04-26T16:33:02.320002+0000","last_became_peered":"2024-04-26T16:33:02.320002+0000","last_unstale":"2024-04-26T16:34:41.333352+0000","last_undegraded":"2024-04-26T16:34:41.333352+0000","last_fullsized":"2024-04-26T16:33:02.303449+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:19.402195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:19.402195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"388'20","reported_seq":253,"reported_epoch":702,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:40.633214+0000","last_change":"2024-04-26T16:33:02.316135+0000","last_active":"2024-04-26T16:34:40.633214+0000","last_peered":"2024-04-26T16:34:40.633214+0000","last_clean":"2024-04-26T16:33:00.426585+0000","last_became_active":"2024-04-26T16:33:02.316135+0000","last_became_peered":"2024-04-26T16:33:02.316135+0000","last_unstale":"2024-04-26T16:34:40.633214+0000","last_undegraded":"2024-04-26T16:33:02.292669+0000","last_fullsized":"2024-04-26T16:33:02.292330+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:19.056987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.056987+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1322,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334417+0000","last_change":"2024-04-26T16:32:38.352954+0000","last_active":"2024-04-26T16:34:41.334417+0000","last_peered":"2024-04-26T16:34:41.334417+0000","last_clean":"2024-04-26T16:34:41.334417+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:34:41.334417+0000","last_undegraded":"2024-04-26T16:34:41.334417+0000","last_fullsized":"2024-04-26T16:34:41.334417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:38.352608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:43.031634+0000","last_clean_scrub_stamp":"2024-04-26T16:32:38.352608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551532+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1319,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331842+0000","last_change":"2024-04-26T16:33:32.729265+0000","last_active":"2024-04-26T16:34:41.331842+0000","last_peered":"2024-04-26T16:34:41.331842+0000","last_clean":"2024-04-26T16:34:41.331842+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:34:41.331842+0000","last_undegraded":"2024-04-26T16:34:41.331842+0000","last_fullsized":"2024-04-26T16:34:41.331842+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:32.728939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:32.728939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.074056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1315,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.736754+0000","last_change":"2024-04-26T16:33:38.754358+0000","last_active":"2024-04-26T16:34:39.736754+0000","last_peered":"2024-04-26T16:34:39.736754+0000","last_clean":"2024-04-26T16:34:39.736754+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:34:39.736754+0000","last_undegraded":"2024-04-26T16:34:39.736754+0000","last_fullsized":"2024-04-26T16:34:39.736754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.754239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.754239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.201822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1181,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.337784+0000","last_change":"2024-04-26T16:33:02.316826+0000","last_active":"2024-04-26T16:34:41.337784+0000","last_peered":"2024-04-26T16:34:41.337784+0000","last_clean":"2024-04-26T16:33:01.300984+0000","last_became_active":"2024-04-26T16:33:02.316826+0000","last_became_peered":"2024-04-26T16:33:02.316826+0000","last_unstale":"2024-04-26T16:34:41.337784+0000","last_undegraded":"2024-04-26T16:34:41.337784+0000","last_fullsized":"2024-04-26T16:33:02.299005+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:16.340018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:16.340018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"612'39","reported_seq":1418,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.337848+0000","last_change":"2024-04-26T16:33:02.322018+0000","last_active":"2024-04-26T16:34:41.337848+0000","last_peered":"2024-04-26T16:34:41.337848+0000","last_clean":"2024-04-26T16:33:01.301948+0000","last_became_active":"2024-04-26T16:33:02.322018+0000","last_became_peered":"2024-04-26T16:33:02.322018+0000","last_unstale":"2024-04-26T16:34:41.337848+0000","last_undegraded":"2024-04-26T16:33:02.297921+0000","last_fullsized":"2024-04-26T16:33:02.297601+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'39","last_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_deep_scrub":"612'39","last_deep_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_clean_scrub_stamp":"2024-04-26T16:32:44.660567+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526850+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1339,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333048+0000","last_change":"2024-04-26T16:33:31.754803+0000","last_active":"2024-04-26T16:34:41.333048+0000","last_peered":"2024-04-26T16:34:41.333048+0000","last_clean":"2024-04-26T16:34:41.333048+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:34:41.333048+0000","last_undegraded":"2024-04-26T16:34:41.333048+0000","last_fullsized":"2024-04-26T16:34:41.333048+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:31.754561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:31.754561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.140085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1331,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332705+0000","last_change":"2024-04-26T16:33:15.657104+0000","last_active":"2024-04-26T16:34:41.332705+0000","last_peered":"2024-04-26T16:34:41.332705+0000","last_clean":"2024-04-26T16:34:41.332705+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:34:41.332705+0000","last_undegraded":"2024-04-26T16:34:41.332705+0000","last_fullsized":"2024-04-26T16:34:41.332705+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:15.656794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:15.656794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.886804+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1285,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332158+0000","last_change":"2024-04-26T16:31:19.857894+0000","last_active":"2024-04-26T16:34:41.332158+0000","last_peered":"2024-04-26T16:34:41.332158+0000","last_clean":"2024-04-26T16:34:41.332158+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:34:41.332158+0000","last_undegraded":"2024-04-26T16:34:41.332158+0000","last_fullsized":"2024-04-26T16:34:41.332158+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:19.857656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:31:19.857656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1171,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332130+0000","last_change":"2024-04-26T16:33:41.813706+0000","last_active":"2024-04-26T16:34:41.332130+0000","last_peered":"2024-04-26T16:34:41.332130+0000","last_clean":"2024-04-26T16:34:41.332130+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:34:41.332130+0000","last_undegraded":"2024-04-26T16:34:41.332130+0000","last_fullsized":"2024-04-26T16:34:41.332130+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:41.813414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:41.813414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.948002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"389'35","reported_seq":953,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.466480+0000","last_change":"2024-04-26T16:33:02.313411+0000","last_active":"2024-04-26T16:34:41.466480+0000","last_peered":"2024-04-26T16:34:41.466480+0000","last_clean":"2024-04-26T16:27:37.176266+0000","last_became_active":"2024-04-26T16:33:02.313411+0000","last_became_peered":"2024-04-26T16:33:02.313411+0000","last_unstale":"2024-04-26T16:34:41.466480+0000","last_undegraded":"2024-04-26T16:33:02.302284+0000","last_fullsized":"2024-04-26T16:33:02.302146+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'35","last_scrub_stamp":"2024-04-26T16:31:44.435924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:44.435924+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.426998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1317,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334847+0000","last_change":"2024-04-26T16:33:45.527712+0000","last_active":"2024-04-26T16:34:41.334847+0000","last_peered":"2024-04-26T16:34:41.334847+0000","last_clean":"2024-04-26T16:34:41.334847+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:34:41.334847+0000","last_undegraded":"2024-04-26T16:34:41.334847+0000","last_fullsized":"2024-04-26T16:34:41.334847+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:45.527416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:33:45.527416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.112750+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1322,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334914+0000","last_change":"2024-04-26T16:32:40.409359+0000","last_active":"2024-04-26T16:34:41.334914+0000","last_peered":"2024-04-26T16:34:41.334914+0000","last_clean":"2024-04-26T16:34:41.334914+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:34:41.334914+0000","last_undegraded":"2024-04-26T16:34:41.334914+0000","last_fullsized":"2024-04-26T16:34:41.334914+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:40.409156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:32:40.409156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551611+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1315,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.335862+0000","last_change":"2024-04-26T16:33:49.756193+0000","last_active":"2024-04-26T16:34:41.335862+0000","last_peered":"2024-04-26T16:34:41.335862+0000","last_clean":"2024-04-26T16:34:41.335862+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:34:41.335862+0000","last_undegraded":"2024-04-26T16:34:41.335862+0000","last_fullsized":"2024-04-26T16:34:41.335862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:49.755868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:21.525230+0000","last_clean_scrub_stamp":"2024-04-26T16:33:49.755868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:00.792996+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":659,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.333326+0000","last_change":"2024-04-26T16:33:02.321421+0000","last_active":"2024-04-26T16:34:41.333326+0000","last_peered":"2024-04-26T16:34:41.333326+0000","last_clean":"2024-04-26T16:23:28.083853+0000","last_became_active":"2024-04-26T16:33:02.321421+0000","last_became_peered":"2024-04-26T16:33:02.321421+0000","last_unstale":"2024-04-26T16:34:41.333326+0000","last_undegraded":"2024-04-26T16:34:41.333326+0000","last_fullsized":"2024-04-26T16:33:02.303814+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:39.300548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.300548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"388'19","reported_seq":1331,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.359930+0000","last_change":"2024-04-26T16:33:42.230824+0000","last_active":"2024-04-26T16:34:41.359930+0000","last_peered":"2024-04-26T16:34:41.359930+0000","last_clean":"2024-04-26T16:34:41.359930+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:34:41.359930+0000","last_undegraded":"2024-04-26T16:34:41.359930+0000","last_fullsized":"2024-04-26T16:34:41.359930+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:33:42.230574+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:33:42.230574+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.382496+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1330,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333666+0000","last_change":"2024-04-26T16:33:49.416576+0000","last_active":"2024-04-26T16:34:41.333666+0000","last_peered":"2024-04-26T16:34:41.333666+0000","last_clean":"2024-04-26T16:34:41.333666+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:34:41.333666+0000","last_undegraded":"2024-04-26T16:34:41.333666+0000","last_fullsized":"2024-04-26T16:34:41.333666+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:49.416438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:49.416438+0000","last_clean_scrub_stamp":"2024-04-26T16:33:49.416438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:52.833664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":1556,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333147+0000","last_change":"2024-04-26T16:33:24.713742+0000","last_active":"2024-04-26T16:34:41.333147+0000","last_peered":"2024-04-26T16:34:41.333147+0000","last_clean":"2024-04-26T16:34:41.333147+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:34:41.333147+0000","last_undegraded":"2024-04-26T16:34:41.333147+0000","last_fullsized":"2024-04-26T16:34:41.333147+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:33:24.713618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:24.713618+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.886907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1316,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.469336+0000","last_change":"2024-04-26T16:33:29.259855+0000","last_active":"2024-04-26T16:34:41.469336+0000","last_peered":"2024-04-26T16:34:41.469336+0000","last_clean":"2024-04-26T16:34:41.469336+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:34:41.469336+0000","last_undegraded":"2024-04-26T16:34:41.469336+0000","last_fullsized":"2024-04-26T16:34:41.469336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:29.259588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:43.602252+0000","last_clean_scrub_stamp":"2024-04-26T16:33:29.259588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:58.179092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1172,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332780+0000","last_change":"2024-04-26T16:33:48.426154+0000","last_active":"2024-04-26T16:34:41.332780+0000","last_peered":"2024-04-26T16:34:41.332780+0000","last_clean":"2024-04-26T16:34:41.332780+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:34:41.332780+0000","last_undegraded":"2024-04-26T16:34:41.332780+0000","last_fullsized":"2024-04-26T16:34:41.332780+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.425925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.425925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:59.626821+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"388'27","reported_seq":1345,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.357808+0000","last_change":"2024-04-26T16:33:26.168674+0000","last_active":"2024-04-26T16:34:41.357808+0000","last_peered":"2024-04-26T16:34:41.357808+0000","last_clean":"2024-04-26T16:34:41.357808+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:34:41.357808+0000","last_undegraded":"2024-04-26T16:34:41.357808+0000","last_fullsized":"2024-04-26T16:34:41.357808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:33:26.168537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:33:26.168537+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1336,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331949+0000","last_change":"2024-04-26T16:32:44.393515+0000","last_active":"2024-04-26T16:34:41.331949+0000","last_peered":"2024-04-26T16:34:41.331949+0000","last_clean":"2024-04-26T16:34:41.331949+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:34:41.331949+0000","last_undegraded":"2024-04-26T16:34:41.331949+0000","last_fullsized":"2024-04-26T16:34:41.331949+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:44.393215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:32:44.393215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551692+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":1550,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.466550+0000","last_change":"2024-04-26T16:32:43.175004+0000","last_active":"2024-04-26T16:34:41.466550+0000","last_peered":"2024-04-26T16:34:41.466550+0000","last_clean":"2024-04-26T16:34:41.466550+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:34:41.466550+0000","last_undegraded":"2024-04-26T16:34:41.466550+0000","last_fullsized":"2024-04-26T16:34:41.466550+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:32:43.174894+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:32:43.174894+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.633331+0000","last_change":"2024-04-26T16:33:55.050828+0000","last_active":"2024-04-26T16:34:40.633331+0000","last_peered":"2024-04-26T16:34:40.633331+0000","last_clean":"2024-04-26T16:34:40.633331+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:34:40.633331+0000","last_undegraded":"2024-04-26T16:34:40.633331+0000","last_fullsized":"2024-04-26T16:34:40.633331+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:55.050611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:22:03.330964+0000","last_clean_scrub_stamp":"2024-04-26T16:33:55.050611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:22.527158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":242,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.360745+0000","last_change":"2024-04-26T16:33:02.308041+0000","last_active":"2024-04-26T16:34:41.360745+0000","last_peered":"2024-04-26T16:34:41.360745+0000","last_clean":"2024-04-26T16:23:29.388954+0000","last_became_active":"2024-04-26T16:33:02.308041+0000","last_became_peered":"2024-04-26T16:33:02.308041+0000","last_unstale":"2024-04-26T16:34:41.360745+0000","last_undegraded":"2024-04-26T16:33:02.293884+0000","last_fullsized":"2024-04-26T16:33:02.293557+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:31:41.360858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:41.360858+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"399'20","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.464740+0000","last_change":"2024-04-26T16:33:27.286559+0000","last_active":"2024-04-26T16:34:41.464740+0000","last_peered":"2024-04-26T16:34:41.464740+0000","last_clean":"2024-04-26T16:34:41.464740+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:34:41.464740+0000","last_undegraded":"2024-04-26T16:34:41.464740+0000","last_fullsized":"2024-04-26T16:34:41.464740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"399'20","last_scrub_stamp":"2024-04-26T16:33:27.286169+0000","last_deep_scrub":"399'20","last_deep_scrub_stamp":"2024-04-26T16:27:47.205374+0000","last_clean_scrub_stamp":"2024-04-26T16:33:27.286169+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427407+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1313,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332688+0000","last_change":"2024-04-26T16:33:29.556528+0000","last_active":"2024-04-26T16:34:41.332688+0000","last_peered":"2024-04-26T16:34:41.332688+0000","last_clean":"2024-04-26T16:34:41.332688+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:34:41.332688+0000","last_undegraded":"2024-04-26T16:34:41.332688+0000","last_fullsized":"2024-04-26T16:34:41.332688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:33:29.556400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:29.556400+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":241,"reported_epoch":702,"state":"active+undersized","last_fresh":"2024-04-26T16:34:40.634606+0000","last_change":"2024-04-26T16:33:02.317029+0000","last_active":"2024-04-26T16:34:40.634606+0000","last_peered":"2024-04-26T16:34:40.634606+0000","last_clean":"2024-04-26T16:33:00.427039+0000","last_became_active":"2024-04-26T16:33:02.317029+0000","last_became_peered":"2024-04-26T16:33:02.317029+0000","last_unstale":"2024-04-26T16:34:40.634606+0000","last_undegraded":"2024-04-26T16:34:40.634606+0000","last_fullsized":"2024-04-26T16:33:02.293791+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:23.518326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:30:23.518326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1310,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.358597+0000","last_change":"2024-04-26T16:33:02.314384+0000","last_active":"2024-04-26T16:34:41.358597+0000","last_peered":"2024-04-26T16:34:41.358597+0000","last_clean":"2024-04-26T16:33:01.291281+0000","last_became_active":"2024-04-26T16:33:02.314384+0000","last_became_peered":"2024-04-26T16:33:02.314384+0000","last_unstale":"2024-04-26T16:34:41.358597+0000","last_undegraded":"2024-04-26T16:34:41.358597+0000","last_fullsized":"2024-04-26T16:33:02.294741+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:35.938129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:35.938129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390578+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1169,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.461173+0000","last_change":"2024-04-26T16:33:39.364471+0000","last_active":"2024-04-26T16:34:41.461173+0000","last_peered":"2024-04-26T16:34:41.461173+0000","last_clean":"2024-04-26T16:34:41.461173+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:34:41.461173+0000","last_undegraded":"2024-04-26T16:34:41.461173+0000","last_fullsized":"2024-04-26T16:34:41.461173+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.364274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.364274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:05.724315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1517,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.358201+0000","last_change":"2024-04-26T16:33:54.179489+0000","last_active":"2024-04-26T16:34:41.358201+0000","last_peered":"2024-04-26T16:34:41.358201+0000","last_clean":"2024-04-26T16:34:41.358201+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:34:41.358201+0000","last_undegraded":"2024-04-26T16:34:41.358201+0000","last_fullsized":"2024-04-26T16:34:41.358201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.179322+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.010524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1346,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.464461+0000","last_change":"2024-04-26T16:33:02.314228+0000","last_active":"2024-04-26T16:34:41.464461+0000","last_peered":"2024-04-26T16:34:41.464461+0000","last_clean":"2024-04-26T16:33:00.882827+0000","last_became_active":"2024-04-26T16:33:02.314228+0000","last_became_peered":"2024-04-26T16:33:02.314228+0000","last_unstale":"2024-04-26T16:34:41.464461+0000","last_undegraded":"2024-04-26T16:34:41.464461+0000","last_fullsized":"2024-04-26T16:33:02.292892+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:31:42.531687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"388'30","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.738112+0000","last_change":"2024-04-26T16:33:56.882142+0000","last_active":"2024-04-26T16:34:39.738112+0000","last_peered":"2024-04-26T16:34:39.738112+0000","last_clean":"2024-04-26T16:34:39.738112+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:34:39.738112+0000","last_undegraded":"2024-04-26T16:34:39.738112+0000","last_fullsized":"2024-04-26T16:34:39.738112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'30","last_scrub_stamp":"2024-04-26T16:33:56.881734+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:33:56.881734+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:13.203043+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":802,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.336925+0000","last_change":"2024-04-26T16:33:02.323596+0000","last_active":"2024-04-26T16:34:41.336925+0000","last_peered":"2024-04-26T16:34:41.336925+0000","last_clean":"2024-04-26T16:17:27.960068+0000","last_became_active":"2024-04-26T16:33:02.323596+0000","last_became_peered":"2024-04-26T16:33:02.323596+0000","last_unstale":"2024-04-26T16:34:41.336925+0000","last_undegraded":"2024-04-26T16:34:41.336925+0000","last_fullsized":"2024-04-26T16:33:02.313028+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:31.269480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:32:31.269480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1306,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.334257+0000","last_change":"2024-04-26T16:33:02.319882+0000","last_active":"2024-04-26T16:34:41.334257+0000","last_peered":"2024-04-26T16:34:41.334257+0000","last_clean":"2024-04-26T16:33:01.307621+0000","last_became_active":"2024-04-26T16:33:02.319882+0000","last_became_peered":"2024-04-26T16:33:02.319882+0000","last_unstale":"2024-04-26T16:34:41.334257+0000","last_undegraded":"2024-04-26T16:34:41.334257+0000","last_fullsized":"2024-04-26T16:33:02.304094+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:35.599770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:32:35.599770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1156,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.337718+0000","last_change":"2024-04-26T16:33:52.514069+0000","last_active":"2024-04-26T16:34:41.337718+0000","last_peered":"2024-04-26T16:34:41.337718+0000","last_clean":"2024-04-26T16:34:41.337718+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:34:41.337718+0000","last_undegraded":"2024-04-26T16:34:41.337718+0000","last_fullsized":"2024-04-26T16:34:41.337718+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:52.513926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:52.513926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:19.536574+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1321,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.463407+0000","last_change":"2024-04-26T16:33:51.205929+0000","last_active":"2024-04-26T16:34:41.463407+0000","last_peered":"2024-04-26T16:34:41.463407+0000","last_clean":"2024-04-26T16:34:41.463407+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:34:41.463407+0000","last_undegraded":"2024-04-26T16:34:41.463407+0000","last_fullsized":"2024-04-26T16:34:41.463407+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:33:51.205695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:59.469040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":2321,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.357992+0000","last_change":"2024-04-26T16:33:32.150005+0000","last_active":"2024-04-26T16:34:41.357992+0000","last_peered":"2024-04-26T16:34:41.357992+0000","last_clean":"2024-04-26T16:34:41.357992+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:34:41.357992+0000","last_undegraded":"2024-04-26T16:34:41.357992+0000","last_fullsized":"2024-04-26T16:34:41.357992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:33:32.149895+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:33:32.149895+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:50.322385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1010,"num_read_kb":1005,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"388'22","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.361208+0000","last_change":"2024-04-26T16:33:22.114952+0000","last_active":"2024-04-26T16:34:41.361208+0000","last_peered":"2024-04-26T16:34:41.361208+0000","last_clean":"2024-04-26T16:34:41.361208+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:34:41.361208+0000","last_undegraded":"2024-04-26T16:34:41.361208+0000","last_fullsized":"2024-04-26T16:34:41.361208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:33:22.114852+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:33:22.114852+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1336,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.335141+0000","last_change":"2024-04-26T16:33:02.312665+0000","last_active":"2024-04-26T16:34:41.335141+0000","last_peered":"2024-04-26T16:34:41.335141+0000","last_clean":"2024-04-26T16:33:00.436780+0000","last_became_active":"2024-04-26T16:33:02.312665+0000","last_became_peered":"2024-04-26T16:33:02.312665+0000","last_unstale":"2024-04-26T16:34:41.335141+0000","last_undegraded":"2024-04-26T16:33:02.306363+0000","last_fullsized":"2024-04-26T16:33:02.305830+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:31:40.035882+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:31:40.035882+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1317,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.463350+0000","last_change":"2024-04-26T16:33:42.347570+0000","last_active":"2024-04-26T16:34:41.463350+0000","last_peered":"2024-04-26T16:34:41.463350+0000","last_clean":"2024-04-26T16:34:41.463350+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:34:41.463350+0000","last_undegraded":"2024-04-26T16:34:41.463350+0000","last_fullsized":"2024-04-26T16:34:41.463350+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:42.347471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:33:42.347471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:11.070217+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1171,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334135+0000","last_change":"2024-04-26T16:33:50.388012+0000","last_active":"2024-04-26T16:34:41.334135+0000","last_peered":"2024-04-26T16:34:41.334135+0000","last_clean":"2024-04-26T16:34:41.334135+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:34:41.334135+0000","last_undegraded":"2024-04-26T16:34:41.334135+0000","last_fullsized":"2024-04-26T16:34:41.334135+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:50.387871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:50.387871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:10.382696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1112,"reported_epoch":701,"state":"active+undersized","last_fresh":"2024-04-26T16:34:39.735824+0000","last_change":"2024-04-26T16:33:02.320523+0000","last_active":"2024-04-26T16:34:39.735824+0000","last_peered":"2024-04-26T16:34:39.735824+0000","last_clean":"2024-04-26T16:32:03.398109+0000","last_became_active":"2024-04-26T16:33:02.320523+0000","last_became_peered":"2024-04-26T16:33:02.320523+0000","last_unstale":"2024-04-26T16:34:39.735824+0000","last_undegraded":"2024-04-26T16:34:39.735824+0000","last_fullsized":"2024-04-26T16:33:02.302153+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:20.267979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:32:20.267979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"585'54","reported_seq":1668,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334296+0000","last_change":"2024-04-26T16:33:37.401179+0000","last_active":"2024-04-26T16:34:41.334296+0000","last_peered":"2024-04-26T16:34:41.334296+0000","last_clean":"2024-04-26T16:34:41.334296+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:34:41.334296+0000","last_undegraded":"2024-04-26T16:34:41.334296+0000","last_fullsized":"2024-04-26T16:34:41.334296+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'54","last_scrub_stamp":"2024-04-26T16:33:37.401059+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:33:37.401059+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.890971+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":54,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1315,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.337657+0000","last_change":"2024-04-26T16:33:36.587348+0000","last_active":"2024-04-26T16:34:41.337657+0000","last_peered":"2024-04-26T16:34:41.337657+0000","last_clean":"2024-04-26T16:34:41.337657+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:34:41.337657+0000","last_undegraded":"2024-04-26T16:34:41.337657+0000","last_fullsized":"2024-04-26T16:34:41.337657+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:33:36.586965+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:33:36.586965+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:49.837702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1323,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.331985+0000","last_change":"2024-04-26T16:33:02.319216+0000","last_active":"2024-04-26T16:34:41.331985+0000","last_peered":"2024-04-26T16:34:41.331985+0000","last_clean":"2024-04-26T16:33:00.429539+0000","last_became_active":"2024-04-26T16:33:02.319216+0000","last_became_peered":"2024-04-26T16:33:02.319216+0000","last_unstale":"2024-04-26T16:34:41.331985+0000","last_undegraded":"2024-04-26T16:34:41.331985+0000","last_fullsized":"2024-04-26T16:33:02.310178+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:28.037115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:28.037115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551335+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1154,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.360981+0000","last_change":"2024-04-26T16:33:02.307539+0000","last_active":"2024-04-26T16:34:41.360981+0000","last_peered":"2024-04-26T16:34:41.360981+0000","last_clean":"2024-04-26T16:33:01.287119+0000","last_became_active":"2024-04-26T16:33:02.307539+0000","last_became_peered":"2024-04-26T16:33:02.307539+0000","last_unstale":"2024-04-26T16:34:41.360981+0000","last_undegraded":"2024-04-26T16:34:41.360981+0000","last_fullsized":"2024-04-26T16:33:02.292490+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:29:58.933163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:29:58.933163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.634433+0000","last_change":"2024-04-26T16:33:27.328627+0000","last_active":"2024-04-26T16:34:40.634433+0000","last_peered":"2024-04-26T16:34:40.634433+0000","last_clean":"2024-04-26T16:34:40.634433+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:34:40.634433+0000","last_undegraded":"2024-04-26T16:34:40.634433+0000","last_fullsized":"2024-04-26T16:34:40.634433+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:27.328459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:33:27.328459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.188842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1306,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332278+0000","last_change":"2024-04-26T16:33:34.584942+0000","last_active":"2024-04-26T16:34:41.332278+0000","last_peered":"2024-04-26T16:34:41.332278+0000","last_clean":"2024-04-26T16:34:41.332278+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:34:41.332278+0000","last_undegraded":"2024-04-26T16:34:41.332278+0000","last_fullsized":"2024-04-26T16:34:41.332278+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:34.584623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:34.584623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:49.315788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"388'35","reported_seq":1365,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.736561+0000","last_change":"2024-04-26T16:33:28.787554+0000","last_active":"2024-04-26T16:34:39.736561+0000","last_peered":"2024-04-26T16:34:39.736561+0000","last_clean":"2024-04-26T16:34:39.736561+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:34:39.736561+0000","last_undegraded":"2024-04-26T16:34:39.736561+0000","last_fullsized":"2024-04-26T16:34:39.736561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:33:28.787378+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:33:28.787378+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.392393+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.335080+0000","last_change":"2024-04-26T16:33:39.416961+0000","last_active":"2024-04-26T16:34:41.335080+0000","last_peered":"2024-04-26T16:34:41.335080+0000","last_clean":"2024-04-26T16:34:41.335080+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:34:41.335080+0000","last_undegraded":"2024-04-26T16:34:41.335080+0000","last_fullsized":"2024-04-26T16:34:41.335080+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.416837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.416837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.327845+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1174,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.358731+0000","last_change":"2024-04-26T16:33:02.301995+0000","last_active":"2024-04-26T16:34:41.358731+0000","last_peered":"2024-04-26T16:34:41.358731+0000","last_clean":"2024-04-26T16:33:01.286780+0000","last_became_active":"2024-04-26T16:33:02.301995+0000","last_became_peered":"2024-04-26T16:33:02.301995+0000","last_unstale":"2024-04-26T16:34:41.358731+0000","last_undegraded":"2024-04-26T16:34:41.358731+0000","last_fullsized":"2024-04-26T16:33:02.291530+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:38.932710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:38.932710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1310,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334412+0000","last_change":"2024-04-26T16:33:56.524032+0000","last_active":"2024-04-26T16:34:41.334412+0000","last_peered":"2024-04-26T16:34:41.334412+0000","last_clean":"2024-04-26T16:34:41.334412+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:34:41.334412+0000","last_undegraded":"2024-04-26T16:34:41.334412+0000","last_fullsized":"2024-04-26T16:34:41.334412+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:56.523819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:33:56.523819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:59.833935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1598,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.332462+0000","last_change":"2024-04-26T16:33:02.322706+0000","last_active":"2024-04-26T16:34:41.332462+0000","last_peered":"2024-04-26T16:34:41.332462+0000","last_clean":"2024-04-26T16:33:01.292631+0000","last_became_active":"2024-04-26T16:33:02.322706+0000","last_became_peered":"2024-04-26T16:33:02.322706+0000","last_unstale":"2024-04-26T16:34:41.332462+0000","last_undegraded":"2024-04-26T16:33:02.299845+0000","last_fullsized":"2024-04-26T16:33:02.299528+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:31:16.790149+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.790149+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.631750+0000","last_change":"2024-04-26T16:33:37.325806+0000","last_active":"2024-04-26T16:34:40.631750+0000","last_peered":"2024-04-26T16:34:40.631750+0000","last_clean":"2024-04-26T16:34:40.631750+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:34:40.631750+0000","last_undegraded":"2024-04-26T16:34:40.631750+0000","last_fullsized":"2024-04-26T16:34:40.631750+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:37.325601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:37.325601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.121266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"388'25","reported_seq":1386,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.358965+0000","last_change":"2024-04-26T16:33:02.307634+0000","last_active":"2024-04-26T16:34:41.358965+0000","last_peered":"2024-04-26T16:34:41.358965+0000","last_clean":"2024-04-26T16:33:01.286876+0000","last_became_active":"2024-04-26T16:33:02.307634+0000","last_became_peered":"2024-04-26T16:33:02.307634+0000","last_unstale":"2024-04-26T16:34:41.358965+0000","last_undegraded":"2024-04-26T16:33:02.293090+0000","last_fullsized":"2024-04-26T16:33:02.292755+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:32:26.895513+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:32:26.895513+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390808+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.634594+0000","last_change":"2024-04-26T16:33:23.365570+0000","last_active":"2024-04-26T16:34:40.634594+0000","last_peered":"2024-04-26T16:34:40.634594+0000","last_clean":"2024-04-26T16:34:40.634594+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:34:40.634594+0000","last_undegraded":"2024-04-26T16:34:40.634594+0000","last_fullsized":"2024-04-26T16:34:40.634594+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:23.365201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:33:23.365201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1174,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.357045+0000","last_change":"2024-04-26T16:33:02.315902+0000","last_active":"2024-04-26T16:34:41.357045+0000","last_peered":"2024-04-26T16:34:41.357045+0000","last_clean":"2024-04-26T16:33:01.286013+0000","last_became_active":"2024-04-26T16:33:02.315902+0000","last_became_peered":"2024-04-26T16:33:02.315902+0000","last_unstale":"2024-04-26T16:34:41.357045+0000","last_undegraded":"2024-04-26T16:34:41.357045+0000","last_fullsized":"2024-04-26T16:33:02.290039+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:34.090693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:34.090693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1331,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.335075+0000","last_change":"2024-04-26T16:33:37.790562+0000","last_active":"2024-04-26T16:34:41.335075+0000","last_peered":"2024-04-26T16:34:41.335075+0000","last_clean":"2024-04-26T16:34:41.335075+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:34:41.335075+0000","last_undegraded":"2024-04-26T16:34:41.335075+0000","last_fullsized":"2024-04-26T16:34:41.335075+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:37.790335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:33:37.790335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:52.849928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1328,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331325+0000","last_change":"2024-04-26T16:33:34.732441+0000","last_active":"2024-04-26T16:34:41.331325+0000","last_peered":"2024-04-26T16:34:41.331325+0000","last_clean":"2024-04-26T16:34:41.331325+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:34:41.331325+0000","last_undegraded":"2024-04-26T16:34:41.331325+0000","last_fullsized":"2024-04-26T16:34:41.331325+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:34.732301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:33:34.732301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:49.246935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"388'38","reported_seq":1397,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.332480+0000","last_change":"2024-04-26T16:33:02.308957+0000","last_active":"2024-04-26T16:34:41.332480+0000","last_peered":"2024-04-26T16:34:41.332480+0000","last_clean":"2024-04-26T16:33:00.426570+0000","last_became_active":"2024-04-26T16:33:02.308957+0000","last_became_peered":"2024-04-26T16:33:02.308957+0000","last_unstale":"2024-04-26T16:34:41.332480+0000","last_undegraded":"2024-04-26T16:33:02.294203+0000","last_fullsized":"2024-04-26T16:33:02.293611+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'38","last_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_clean_scrub_stamp":"2024-04-26T16:32:43.426520+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551428+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1298,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.337045+0000","last_change":"2024-04-26T16:33:02.322714+0000","last_active":"2024-04-26T16:34:41.337045+0000","last_peered":"2024-04-26T16:34:41.337045+0000","last_clean":"2024-04-26T16:33:01.310812+0000","last_became_active":"2024-04-26T16:33:02.322714+0000","last_became_peered":"2024-04-26T16:33:02.322714+0000","last_unstale":"2024-04-26T16:34:41.337045+0000","last_undegraded":"2024-04-26T16:34:41.337045+0000","last_fullsized":"2024-04-26T16:33:02.305830+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:52.021548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:28:52.021548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1149,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.356508+0000","last_change":"2024-04-26T16:32:24.826314+0000","last_active":"2024-04-26T16:34:41.356508+0000","last_peered":"2024-04-26T16:34:41.356508+0000","last_clean":"2024-04-26T16:34:41.356508+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:34:41.356508+0000","last_undegraded":"2024-04-26T16:34:41.356508+0000","last_fullsized":"2024-04-26T16:34:41.356508+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:24.826176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:24.826176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1312,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737346+0000","last_change":"2024-04-26T16:32:29.797597+0000","last_active":"2024-04-26T16:34:39.737346+0000","last_peered":"2024-04-26T16:34:39.737346+0000","last_clean":"2024-04-26T16:34:39.737346+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:34:39.737346+0000","last_undegraded":"2024-04-26T16:34:39.737346+0000","last_fullsized":"2024-04-26T16:34:39.737346+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:29.797513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:32:29.797513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"388'37","reported_seq":1388,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.463487+0000","last_change":"2024-04-26T16:33:02.309201+0000","last_active":"2024-04-26T16:34:41.463487+0000","last_peered":"2024-04-26T16:34:41.463487+0000","last_clean":"2024-04-26T16:33:00.883327+0000","last_became_active":"2024-04-26T16:33:02.309201+0000","last_became_peered":"2024-04-26T16:33:02.309201+0000","last_unstale":"2024-04-26T16:34:41.463487+0000","last_undegraded":"2024-04-26T16:33:02.295666+0000","last_fullsized":"2024-04-26T16:33:02.295319+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'37","last_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:31:38.498802+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":241,"reported_epoch":702,"state":"active+undersized","last_fresh":"2024-04-26T16:34:40.632288+0000","last_change":"2024-04-26T16:33:02.308807+0000","last_active":"2024-04-26T16:34:40.632288+0000","last_peered":"2024-04-26T16:34:40.632288+0000","last_clean":"2024-04-26T16:33:00.425097+0000","last_became_active":"2024-04-26T16:33:02.308807+0000","last_became_peered":"2024-04-26T16:33:02.308807+0000","last_unstale":"2024-04-26T16:34:40.632288+0000","last_undegraded":"2024-04-26T16:34:40.632288+0000","last_fullsized":"2024-04-26T16:33:02.292154+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:23.095870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:31:23.095870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1302,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.360135+0000","last_change":"2024-04-26T16:33:47.201263+0000","last_active":"2024-04-26T16:34:41.360135+0000","last_peered":"2024-04-26T16:34:41.360135+0000","last_clean":"2024-04-26T16:34:41.360135+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:34:41.360135+0000","last_undegraded":"2024-04-26T16:34:41.360135+0000","last_fullsized":"2024-04-26T16:34:41.360135+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:47.201163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:47.201163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:10.355966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1166,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.333437+0000","last_change":"2024-04-26T16:33:02.314801+0000","last_active":"2024-04-26T16:34:41.333437+0000","last_peered":"2024-04-26T16:34:41.333437+0000","last_clean":"2024-04-26T16:33:00.432203+0000","last_became_active":"2024-04-26T16:33:02.314801+0000","last_became_peered":"2024-04-26T16:33:02.314801+0000","last_unstale":"2024-04-26T16:34:41.333437+0000","last_undegraded":"2024-04-26T16:34:41.333437+0000","last_fullsized":"2024-04-26T16:33:02.305638+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:19.533042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:30:19.533042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.886662+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"630'18","reported_seq":1561,"reported_epoch":701,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:39.737880+0000","last_change":"2024-04-26T16:33:02.308373+0000","last_active":"2024-04-26T16:34:39.737880+0000","last_peered":"2024-04-26T16:34:39.737880+0000","last_clean":"2024-04-26T16:33:01.297193+0000","last_became_active":"2024-04-26T16:33:02.308373+0000","last_became_peered":"2024-04-26T16:33:02.308373+0000","last_unstale":"2024-04-26T16:34:39.737880+0000","last_undegraded":"2024-04-26T16:33:02.303866+0000","last_fullsized":"2024-04-26T16:33:02.303632+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'16","last_scrub_stamp":"2024-04-26T16:31:16.715748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:16.715748+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1354,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334218+0000","last_change":"2024-04-26T16:33:46.445082+0000","last_active":"2024-04-26T16:34:41.334218+0000","last_peered":"2024-04-26T16:34:41.334218+0000","last_clean":"2024-04-26T16:34:41.334218+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:34:41.334218+0000","last_undegraded":"2024-04-26T16:34:41.334218+0000","last_fullsized":"2024-04-26T16:34:41.334218+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:33:46.444925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:33:46.444925+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:47.401381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"585'40","reported_seq":295,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.633118+0000","last_change":"2024-04-26T16:33:35.343288+0000","last_active":"2024-04-26T16:34:40.633118+0000","last_peered":"2024-04-26T16:34:40.633118+0000","last_clean":"2024-04-26T16:34:40.633118+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:34:40.633118+0000","last_undegraded":"2024-04-26T16:34:40.633118+0000","last_fullsized":"2024-04-26T16:34:40.633118+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'40","last_scrub_stamp":"2024-04-26T16:33:35.343032+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:33:35.343032+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.610222+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1331,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334453+0000","last_change":"2024-04-26T16:33:22.507810+0000","last_active":"2024-04-26T16:34:41.334453+0000","last_peered":"2024-04-26T16:34:41.334453+0000","last_clean":"2024-04-26T16:34:41.334453+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:34:41.334453+0000","last_undegraded":"2024-04-26T16:34:41.334453+0000","last_fullsized":"2024-04-26T16:34:41.334453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:22.507561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:33:22.507561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551803+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1300,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.735731+0000","last_change":"2024-04-26T16:32:17.757412+0000","last_active":"2024-04-26T16:34:39.735731+0000","last_peered":"2024-04-26T16:34:39.735731+0000","last_clean":"2024-04-26T16:34:39.735731+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:34:39.735731+0000","last_undegraded":"2024-04-26T16:34:39.735731+0000","last_fullsized":"2024-04-26T16:34:39.735731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:17.757210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:32:17.757210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1163,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737959+0000","last_change":"2024-04-26T16:33:43.739749+0000","last_active":"2024-04-26T16:34:39.737959+0000","last_peered":"2024-04-26T16:34:39.737959+0000","last_clean":"2024-04-26T16:34:39.737959+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:34:39.737959+0000","last_undegraded":"2024-04-26T16:34:39.737959+0000","last_fullsized":"2024-04-26T16:34:39.737959+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:43.739649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:43.739649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.398957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1319,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.330974+0000","last_change":"2024-04-26T16:33:21.757129+0000","last_active":"2024-04-26T16:34:41.330974+0000","last_peered":"2024-04-26T16:34:41.330974+0000","last_clean":"2024-04-26T16:34:41.330974+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:34:41.330974+0000","last_undegraded":"2024-04-26T16:34:41.330974+0000","last_fullsized":"2024-04-26T16:34:41.330974+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:21.756987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:33:21.756987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:47.441073+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"388'19","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.335265+0000","last_change":"2024-04-26T16:33:54.462883+0000","last_active":"2024-04-26T16:34:41.335265+0000","last_peered":"2024-04-26T16:34:41.335265+0000","last_clean":"2024-04-26T16:34:41.335265+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:34:41.335265+0000","last_undegraded":"2024-04-26T16:34:41.335265+0000","last_fullsized":"2024-04-26T16:34:41.335265+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:33:54.462286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.462286+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:20.077525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.468605+0000","last_change":"2024-04-26T16:33:50.235407+0000","last_active":"2024-04-26T16:34:41.468605+0000","last_peered":"2024-04-26T16:34:41.468605+0000","last_clean":"2024-04-26T16:34:41.468605+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:34:41.468605+0000","last_undegraded":"2024-04-26T16:34:41.468605+0000","last_fullsized":"2024-04-26T16:34:41.468605+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:33:50.235076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:50.235076+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:17.308186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1294,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.360374+0000","last_change":"2024-04-26T16:33:48.162635+0000","last_active":"2024-04-26T16:34:41.360374+0000","last_peered":"2024-04-26T16:34:41.360374+0000","last_clean":"2024-04-26T16:34:41.360374+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:34:41.360374+0000","last_undegraded":"2024-04-26T16:34:41.360374+0000","last_fullsized":"2024-04-26T16:34:41.360374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.162241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:48.162241+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.162241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.570512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1162,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.735751+0000","last_change":"2024-04-26T16:33:26.686394+0000","last_active":"2024-04-26T16:34:39.735751+0000","last_peered":"2024-04-26T16:34:39.735751+0000","last_clean":"2024-04-26T16:34:39.735751+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:34:39.735751+0000","last_undegraded":"2024-04-26T16:34:39.735751+0000","last_fullsized":"2024-04-26T16:34:39.735751+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:26.686165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:26.686165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.752786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"388'19","reported_seq":1336,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.332840+0000","last_change":"2024-04-26T16:33:02.313583+0000","last_active":"2024-04-26T16:34:41.332840+0000","last_peered":"2024-04-26T16:34:41.332840+0000","last_clean":"2024-04-26T16:33:01.312890+0000","last_became_active":"2024-04-26T16:33:02.313583+0000","last_became_peered":"2024-04-26T16:33:02.313583+0000","last_unstale":"2024-04-26T16:34:41.332840+0000","last_undegraded":"2024-04-26T16:33:02.301033+0000","last_fullsized":"2024-04-26T16:33:02.300902+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:29:00.045450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:29:00.045450+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.330856+0000","last_change":"2024-04-26T16:33:36.741950+0000","last_active":"2024-04-26T16:34:41.330856+0000","last_peered":"2024-04-26T16:34:41.330856+0000","last_clean":"2024-04-26T16:34:41.330856+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:34:41.330856+0000","last_undegraded":"2024-04-26T16:34:41.330856+0000","last_fullsized":"2024-04-26T16:34:41.330856+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:36.741738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:33:36.741738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.880852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1298,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.362283+0000","last_change":"2024-04-26T16:33:40.240142+0000","last_active":"2024-04-26T16:34:41.362283+0000","last_peered":"2024-04-26T16:34:41.362283+0000","last_clean":"2024-04-26T16:34:41.362283+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:34:41.362283+0000","last_undegraded":"2024-04-26T16:34:41.362283+0000","last_fullsized":"2024-04-26T16:34:41.362283+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:40.239897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:33:40.239897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.412941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1315,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333387+0000","last_change":"2024-04-26T16:33:54.807377+0000","last_active":"2024-04-26T16:34:41.333387+0000","last_peered":"2024-04-26T16:34:41.333387+0000","last_clean":"2024-04-26T16:34:41.333387+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:34:41.333387+0000","last_undegraded":"2024-04-26T16:34:41.333387+0000","last_fullsized":"2024-04-26T16:34:41.333387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.807120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:08.419903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1174,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.330759+0000","last_change":"2024-04-26T16:33:02.314995+0000","last_active":"2024-04-26T16:34:41.330759+0000","last_peered":"2024-04-26T16:34:41.330759+0000","last_clean":"2024-04-26T16:33:00.427322+0000","last_became_active":"2024-04-26T16:33:02.314995+0000","last_became_peered":"2024-04-26T16:33:02.314995+0000","last_unstale":"2024-04-26T16:34:41.330759+0000","last_undegraded":"2024-04-26T16:34:41.330759+0000","last_fullsized":"2024-04-26T16:33:02.300713+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:41.617220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:41.617220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.886693+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"388'8","reported_seq":915,"reported_epoch":701,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:39.736241+0000","last_change":"2024-04-26T16:33:02.317319+0000","last_active":"2024-04-26T16:34:39.736241+0000","last_peered":"2024-04-26T16:34:39.736241+0000","last_clean":"2024-04-26T16:27:36.832282+0000","last_became_active":"2024-04-26T16:33:02.317319+0000","last_became_peered":"2024-04-26T16:33:02.317319+0000","last_unstale":"2024-04-26T16:34:39.736241+0000","last_undegraded":"2024-04-26T16:33:02.301756+0000","last_fullsized":"2024-04-26T16:33:02.301404+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'8","last_scrub_stamp":"2024-04-26T16:31:45.421499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:45.421499+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":785,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.465372+0000","last_change":"2024-04-26T16:33:02.312790+0000","last_active":"2024-04-26T16:34:41.465372+0000","last_peered":"2024-04-26T16:34:41.465372+0000","last_clean":"2024-04-26T16:17:27.960068+0000","last_became_active":"2024-04-26T16:33:02.312790+0000","last_became_peered":"2024-04-26T16:33:02.312790+0000","last_unstale":"2024-04-26T16:34:41.465372+0000","last_undegraded":"2024-04-26T16:34:41.465372+0000","last_fullsized":"2024-04-26T16:33:02.294363+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:23.283518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:32:23.283518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.633366+0000","last_change":"2024-04-26T16:32:30.238246+0000","last_active":"2024-04-26T16:34:40.633366+0000","last_peered":"2024-04-26T16:34:40.633366+0000","last_clean":"2024-04-26T16:34:40.633366+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:34:40.633366+0000","last_undegraded":"2024-04-26T16:34:40.633366+0000","last_fullsized":"2024-04-26T16:34:40.633366+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:30.238008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:32:30.238008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1297,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.335090+0000","last_change":"2024-04-26T16:33:02.319994+0000","last_active":"2024-04-26T16:34:41.335090+0000","last_peered":"2024-04-26T16:34:41.335090+0000","last_clean":"2024-04-26T16:33:01.316461+0000","last_became_active":"2024-04-26T16:33:02.319994+0000","last_became_peered":"2024-04-26T16:33:02.319994+0000","last_unstale":"2024-04-26T16:34:41.335090+0000","last_undegraded":"2024-04-26T16:34:41.335090+0000","last_fullsized":"2024-04-26T16:33:02.304876+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:30.019680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:31:30.019680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1169,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.461103+0000","last_change":"2024-04-26T16:32:36.138996+0000","last_active":"2024-04-26T16:34:41.461103+0000","last_peered":"2024-04-26T16:34:41.461103+0000","last_clean":"2024-04-26T16:34:41.461103+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:34:41.461103+0000","last_undegraded":"2024-04-26T16:34:41.461103+0000","last_fullsized":"2024-04-26T16:34:41.461103+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:36.138890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:36.138890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427227+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"388'19","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334923+0000","last_change":"2024-04-26T16:32:29.512249+0000","last_active":"2024-04-26T16:34:41.334923+0000","last_peered":"2024-04-26T16:34:41.334923+0000","last_clean":"2024-04-26T16:34:41.334923+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:34:41.334923+0000","last_undegraded":"2024-04-26T16:34:41.334923+0000","last_fullsized":"2024-04-26T16:34:41.334923+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:32:29.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:29.512093+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.631442+0000","last_change":"2024-04-26T16:33:22.354265+0000","last_active":"2024-04-26T16:34:40.631442+0000","last_peered":"2024-04-26T16:34:40.631442+0000","last_clean":"2024-04-26T16:34:40.631442+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:34:40.631442+0000","last_undegraded":"2024-04-26T16:34:40.631442+0000","last_fullsized":"2024-04-26T16:34:40.631442+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:22.354067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:22.354067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":281,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.632770+0000","last_change":"2024-04-26T16:33:39.290376+0000","last_active":"2024-04-26T16:34:40.632770+0000","last_peered":"2024-04-26T16:34:40.632770+0000","last_clean":"2024-04-26T16:34:40.632770+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:34:40.632770+0000","last_undegraded":"2024-04-26T16:34:40.632770+0000","last_fullsized":"2024-04-26T16:34:40.632770+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:33:39.290137+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.290137+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.254553+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1307,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.738421+0000","last_change":"2024-04-26T16:33:53.794483+0000","last_active":"2024-04-26T16:34:39.738421+0000","last_peered":"2024-04-26T16:34:39.738421+0000","last_clean":"2024-04-26T16:34:39.738421+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:34:39.738421+0000","last_undegraded":"2024-04-26T16:34:39.738421+0000","last_fullsized":"2024-04-26T16:34:39.738421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:53.794377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:53.794377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:21.434784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.631528+0000","last_change":"2024-04-26T16:32:28.181782+0000","last_active":"2024-04-26T16:34:40.631528+0000","last_peered":"2024-04-26T16:34:40.631528+0000","last_clean":"2024-04-26T16:34:40.631528+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:34:40.631528+0000","last_undegraded":"2024-04-26T16:34:40.631528+0000","last_fullsized":"2024-04-26T16:34:40.631528+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:28.181572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:28.181572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"388'19","reported_seq":1344,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.735902+0000","last_change":"2024-04-26T16:33:41.802094+0000","last_active":"2024-04-26T16:34:39.735902+0000","last_peered":"2024-04-26T16:34:39.735902+0000","last_clean":"2024-04-26T16:34:39.735902+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:34:39.735902+0000","last_undegraded":"2024-04-26T16:34:39.735902+0000","last_fullsized":"2024-04-26T16:34:39.735902+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:33:41.801750+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:33:41.801750+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:03.839175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1319,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737140+0000","last_change":"2024-04-26T16:33:40.752964+0000","last_active":"2024-04-26T16:34:39.737140+0000","last_peered":"2024-04-26T16:34:39.737140+0000","last_clean":"2024-04-26T16:34:39.737140+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:34:39.737140+0000","last_undegraded":"2024-04-26T16:34:39.737140+0000","last_fullsized":"2024-04-26T16:34:39.737140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:40.752777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:10.518359+0000","last_clean_scrub_stamp":"2024-04-26T16:33:40.752777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:55.127925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1515,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.358354+0000","last_change":"2024-04-26T16:33:46.206611+0000","last_active":"2024-04-26T16:34:41.358354+0000","last_peered":"2024-04-26T16:34:41.358354+0000","last_clean":"2024-04-26T16:34:41.358354+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:34:41.358354+0000","last_undegraded":"2024-04-26T16:34:41.358354+0000","last_fullsized":"2024-04-26T16:34:41.358354+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:33:46.206497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:46.206497+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:56.209377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1328,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.469678+0000","last_change":"2024-04-26T16:33:54.273720+0000","last_active":"2024-04-26T16:34:41.469678+0000","last_peered":"2024-04-26T16:34:41.469678+0000","last_clean":"2024-04-26T16:34:41.469678+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:34:41.469678+0000","last_undegraded":"2024-04-26T16:34:41.469678+0000","last_fullsized":"2024-04-26T16:34:41.469678+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:54.273521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:54.273521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.343130+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1148,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.338335+0000","last_change":"2024-04-26T16:31:43.245777+0000","last_active":"2024-04-26T16:34:41.338335+0000","last_peered":"2024-04-26T16:34:41.338335+0000","last_clean":"2024-04-26T16:34:41.338335+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:34:41.338335+0000","last_undegraded":"2024-04-26T16:34:41.338335+0000","last_fullsized":"2024-04-26T16:34:41.338335+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:43.245562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.245562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"388'19","reported_seq":1330,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.338438+0000","last_change":"2024-04-26T16:32:43.668627+0000","last_active":"2024-04-26T16:34:41.338438+0000","last_peered":"2024-04-26T16:34:41.338438+0000","last_clean":"2024-04-26T16:34:41.338438+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:34:41.338438+0000","last_undegraded":"2024-04-26T16:34:41.338438+0000","last_fullsized":"2024-04-26T16:34:41.338438+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:32:43.668373+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:32:43.668373+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526891+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1116,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.358061+0000","last_change":"2024-04-26T16:33:02.311544+0000","last_active":"2024-04-26T16:34:41.358061+0000","last_peered":"2024-04-26T16:34:41.358061+0000","last_clean":"2024-04-26T16:32:03.398676+0000","last_became_active":"2024-04-26T16:33:02.311544+0000","last_became_peered":"2024-04-26T16:33:02.311544+0000","last_unstale":"2024-04-26T16:34:41.358061+0000","last_undegraded":"2024-04-26T16:34:41.358061+0000","last_fullsized":"2024-04-26T16:33:02.293789+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:26.231313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:31:26.231313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1306,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.360330+0000","last_change":"2024-04-26T16:33:53.175628+0000","last_active":"2024-04-26T16:34:41.360330+0000","last_peered":"2024-04-26T16:34:41.360330+0000","last_clean":"2024-04-26T16:34:41.360330+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:34:41.360330+0000","last_undegraded":"2024-04-26T16:34:41.360330+0000","last_fullsized":"2024-04-26T16:34:41.360330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:53.175513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:53.175513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:13.618639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1328,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.735605+0000","last_change":"2024-04-26T16:33:33.851329+0000","last_active":"2024-04-26T16:34:39.735605+0000","last_peered":"2024-04-26T16:34:39.735605+0000","last_clean":"2024-04-26T16:34:39.735605+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:34:39.735605+0000","last_undegraded":"2024-04-26T16:34:39.735605+0000","last_fullsized":"2024-04-26T16:34:39.735605+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:33.851177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:39.061987+0000","last_clean_scrub_stamp":"2024-04-26T16:33:33.851177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:56.630344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":643,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.334078+0000","last_change":"2024-04-26T16:33:02.320021+0000","last_active":"2024-04-26T16:34:41.334078+0000","last_peered":"2024-04-26T16:34:41.334078+0000","last_clean":"2024-04-26T16:23:28.083853+0000","last_became_active":"2024-04-26T16:33:02.320021+0000","last_became_peered":"2024-04-26T16:33:02.320021+0000","last_unstale":"2024-04-26T16:34:41.334078+0000","last_undegraded":"2024-04-26T16:34:41.334078+0000","last_fullsized":"2024-04-26T16:33:02.304132+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:32:46.121113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"388'20","reported_seq":1358,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.462849+0000","last_change":"2024-04-26T16:33:02.308922+0000","last_active":"2024-04-26T16:34:41.462849+0000","last_peered":"2024-04-26T16:34:41.462849+0000","last_clean":"2024-04-26T16:33:00.883291+0000","last_became_active":"2024-04-26T16:33:02.308922+0000","last_became_peered":"2024-04-26T16:33:02.308922+0000","last_unstale":"2024-04-26T16:34:41.462849+0000","last_undegraded":"2024-04-26T16:33:02.296135+0000","last_fullsized":"2024-04-26T16:33:02.296006+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:31:39.502283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:31:39.502283+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":837,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334207+0000","last_change":"2024-04-26T16:33:57.483335+0000","last_active":"2024-04-26T16:34:41.334207+0000","last_peered":"2024-04-26T16:34:41.334207+0000","last_clean":"2024-04-26T16:34:41.334207+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:34:41.334207+0000","last_undegraded":"2024-04-26T16:34:41.334207+0000","last_fullsized":"2024-04-26T16:34:41.334207+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:57.483213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:57.483213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:06.062101+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1326,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334081+0000","last_change":"2024-04-26T16:33:38.432035+0000","last_active":"2024-04-26T16:34:41.334081+0000","last_peered":"2024-04-26T16:34:41.334081+0000","last_clean":"2024-04-26T16:34:41.334081+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:34:41.334081+0000","last_undegraded":"2024-04-26T16:34:41.334081+0000","last_fullsized":"2024-04-26T16:34:41.334081+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:38.431938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.431938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.466782+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1306,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.360358+0000","last_change":"2024-04-26T16:32:39.956683+0000","last_active":"2024-04-26T16:34:41.360358+0000","last_peered":"2024-04-26T16:34:41.360358+0000","last_clean":"2024-04-26T16:34:41.360358+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:34:41.360358+0000","last_undegraded":"2024-04-26T16:34:41.360358+0000","last_fullsized":"2024-04-26T16:34:41.360358+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.956573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.956573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1171,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.335995+0000","last_change":"2024-04-26T16:33:39.843269+0000","last_active":"2024-04-26T16:34:41.335995+0000","last_peered":"2024-04-26T16:34:41.335995+0000","last_clean":"2024-04-26T16:34:41.335995+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:34:41.335995+0000","last_undegraded":"2024-04-26T16:34:41.335995+0000","last_fullsized":"2024-04-26T16:34:41.335995+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.842910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.842910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.244346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"388'12","reported_seq":1370,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737822+0000","last_change":"2024-04-26T16:32:41.794259+0000","last_active":"2024-04-26T16:34:39.737822+0000","last_peered":"2024-04-26T16:34:39.737822+0000","last_clean":"2024-04-26T16:34:39.737822+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:34:39.737822+0000","last_undegraded":"2024-04-26T16:34:39.737822+0000","last_fullsized":"2024-04-26T16:34:39.737822+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'12","last_scrub_stamp":"2024-04-26T16:32:41.794109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:41.794109+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":241,"reported_epoch":702,"state":"active+undersized","last_fresh":"2024-04-26T16:34:40.633872+0000","last_change":"2024-04-26T16:33:02.310872+0000","last_active":"2024-04-26T16:34:40.633872+0000","last_peered":"2024-04-26T16:34:40.633872+0000","last_clean":"2024-04-26T16:33:00.427073+0000","last_became_active":"2024-04-26T16:33:02.310872+0000","last_became_peered":"2024-04-26T16:33:02.310872+0000","last_unstale":"2024-04-26T16:34:40.633872+0000","last_undegraded":"2024-04-26T16:34:40.633872+0000","last_fullsized":"2024-04-26T16:33:02.291591+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:36.171451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:36.171451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":833,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.462133+0000","last_change":"2024-04-26T16:33:02.314347+0000","last_active":"2024-04-26T16:34:41.462133+0000","last_peered":"2024-04-26T16:34:41.462133+0000","last_clean":"2024-04-26T16:17:32.003751+0000","last_became_active":"2024-04-26T16:33:02.314347+0000","last_became_peered":"2024-04-26T16:33:02.314347+0000","last_unstale":"2024-04-26T16:34:41.462133+0000","last_undegraded":"2024-04-26T16:34:41.462133+0000","last_fullsized":"2024-04-26T16:33:02.294198+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:21.314021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:32:21.314021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1293,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333901+0000","last_change":"2024-04-26T16:33:57.532437+0000","last_active":"2024-04-26T16:34:41.333901+0000","last_peered":"2024-04-26T16:34:41.333901+0000","last_clean":"2024-04-26T16:34:41.333901+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:34:41.333901+0000","last_undegraded":"2024-04-26T16:34:41.333901+0000","last_fullsized":"2024-04-26T16:34:41.333901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:57.532056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:57.532056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:13.643606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1161,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.335580+0000","last_change":"2024-04-26T16:33:02.323544+0000","last_active":"2024-04-26T16:34:41.335580+0000","last_peered":"2024-04-26T16:34:41.335580+0000","last_clean":"2024-04-26T16:33:01.306833+0000","last_became_active":"2024-04-26T16:33:02.323544+0000","last_became_peered":"2024-04-26T16:33:02.323544+0000","last_unstale":"2024-04-26T16:34:41.335580+0000","last_undegraded":"2024-04-26T16:34:41.335580+0000","last_fullsized":"2024-04-26T16:33:02.308613+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.702030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"388'15","reported_seq":256,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.633640+0000","last_change":"2024-04-26T16:32:22.159040+0000","last_active":"2024-04-26T16:34:40.633640+0000","last_peered":"2024-04-26T16:34:40.633640+0000","last_clean":"2024-04-26T16:34:40.633640+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:34:40.633640+0000","last_undegraded":"2024-04-26T16:34:40.633640+0000","last_fullsized":"2024-04-26T16:34:40.633640+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:32:22.158736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:22.158736+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000301+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1122,"reported_epoch":702,"state":"active+undersized","last_fresh":"2024-04-26T16:34:40.633257+0000","last_change":"2024-04-26T16:33:02.311865+0000","last_active":"2024-04-26T16:34:40.633257+0000","last_peered":"2024-04-26T16:34:40.633257+0000","last_clean":"2024-04-26T16:32:03.400197+0000","last_became_active":"2024-04-26T16:33:02.311865+0000","last_became_peered":"2024-04-26T16:33:02.311865+0000","last_unstale":"2024-04-26T16:34:40.633257+0000","last_undegraded":"2024-04-26T16:34:40.633257+0000","last_fullsized":"2024-04-26T16:33:02.291881+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:38.199629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:32:38.199629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000366+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1301,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.333248+0000","last_change":"2024-04-26T16:33:02.315798+0000","last_active":"2024-04-26T16:34:41.333248+0000","last_peered":"2024-04-26T16:34:41.333248+0000","last_clean":"2024-04-26T16:33:01.295761+0000","last_became_active":"2024-04-26T16:33:02.315798+0000","last_became_peered":"2024-04-26T16:33:02.315798+0000","last_unstale":"2024-04-26T16:34:41.333248+0000","last_undegraded":"2024-04-26T16:34:41.333248+0000","last_fullsized":"2024-04-26T16:33:02.301792+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:28:57.063594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:28:57.063594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1298,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.361165+0000","last_change":"2024-04-26T16:33:02.318857+0000","last_active":"2024-04-26T16:34:41.361165+0000","last_peered":"2024-04-26T16:34:41.361165+0000","last_clean":"2024-04-26T16:33:01.289540+0000","last_became_active":"2024-04-26T16:33:02.318857+0000","last_became_peered":"2024-04-26T16:33:02.318857+0000","last_unstale":"2024-04-26T16:34:41.361165+0000","last_undegraded":"2024-04-26T16:34:41.361165+0000","last_fullsized":"2024-04-26T16:33:02.299149+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:09.970505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:09.970505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1162,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331841+0000","last_change":"2024-04-26T16:32:39.612584+0000","last_active":"2024-04-26T16:34:41.331841+0000","last_peered":"2024-04-26T16:34:41.331841+0000","last_clean":"2024-04-26T16:34:41.331841+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:34:41.331841+0000","last_undegraded":"2024-04-26T16:34:41.331841+0000","last_fullsized":"2024-04-26T16:34:41.331841+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:39.612345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:39.612345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.886751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"388'34","reported_seq":1376,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331809+0000","last_change":"2024-04-26T16:33:25.442084+0000","last_active":"2024-04-26T16:34:41.331809+0000","last_peered":"2024-04-26T16:34:41.331809+0000","last_clean":"2024-04-26T16:34:41.331809+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:34:41.331809+0000","last_undegraded":"2024-04-26T16:34:41.331809+0000","last_fullsized":"2024-04-26T16:34:41.331809+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:33:25.441825+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:33:25.441825+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:48.910885+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.630727+0000","last_change":"2024-04-26T16:32:42.165983+0000","last_active":"2024-04-26T16:34:40.630727+0000","last_peered":"2024-04-26T16:34:40.630727+0000","last_clean":"2024-04-26T16:34:40.630727+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:34:40.630727+0000","last_undegraded":"2024-04-26T16:34:40.630727+0000","last_fullsized":"2024-04-26T16:34:40.630727+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:42.165747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:32:42.165747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"630'15","reported_seq":923,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.469616+0000","last_change":"2024-04-26T16:33:02.315178+0000","last_active":"2024-04-26T16:34:41.469616+0000","last_peered":"2024-04-26T16:34:41.469616+0000","last_clean":"2024-04-26T16:26:37.835971+0000","last_became_active":"2024-04-26T16:33:02.315178+0000","last_became_peered":"2024-04-26T16:33:02.315178+0000","last_unstale":"2024-04-26T16:34:41.469616+0000","last_undegraded":"2024-04-26T16:33:02.300874+0000","last_fullsized":"2024-04-26T16:33:02.300519+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'14","last_scrub_stamp":"2024-04-26T16:32:32.298132+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:32:32.298132+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"86'1","reported_seq":1314,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334664+0000","last_change":"2024-04-26T16:33:50.501731+0000","last_active":"2024-04-26T16:34:41.334664+0000","last_peered":"2024-04-26T16:34:41.334664+0000","last_clean":"2024-04-26T16:34:41.334664+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:34:41.334664+0000","last_undegraded":"2024-04-26T16:34:41.334664+0000","last_fullsized":"2024-04-26T16:34:41.334664+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:33:50.501577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:50.501577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:09.016490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1166,"reported_epoch":701,"state":"active+undersized","last_fresh":"2024-04-26T16:34:39.736950+0000","last_change":"2024-04-26T16:33:02.322776+0000","last_active":"2024-04-26T16:34:39.736950+0000","last_peered":"2024-04-26T16:34:39.736950+0000","last_clean":"2024-04-26T16:33:01.293240+0000","last_became_active":"2024-04-26T16:33:02.322776+0000","last_became_peered":"2024-04-26T16:33:02.322776+0000","last_unstale":"2024-04-26T16:34:39.736950+0000","last_undegraded":"2024-04-26T16:34:39.736950+0000","last_fullsized":"2024-04-26T16:33:02.310024+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:23.729621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:32:23.729621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"388'15","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.357440+0000","last_change":"2024-04-26T16:32:25.865774+0000","last_active":"2024-04-26T16:34:41.357440+0000","last_peered":"2024-04-26T16:34:41.357440+0000","last_clean":"2024-04-26T16:34:41.357440+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:34:41.357440+0000","last_undegraded":"2024-04-26T16:34:41.357440+0000","last_fullsized":"2024-04-26T16:34:41.357440+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:32:25.865394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:25.865394+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1312,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.357542+0000","last_change":"2024-04-26T16:33:02.319315+0000","last_active":"2024-04-26T16:34:41.357542+0000","last_peered":"2024-04-26T16:34:41.357542+0000","last_clean":"2024-04-26T16:33:01.286597+0000","last_became_active":"2024-04-26T16:33:02.319315+0000","last_became_peered":"2024-04-26T16:33:02.319315+0000","last_unstale":"2024-04-26T16:34:41.357542+0000","last_undegraded":"2024-04-26T16:33:02.291067+0000","last_fullsized":"2024-04-26T16:33:02.290734+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:31:17.080989+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:31:17.080989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390613+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1320,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.738293+0000","last_change":"2024-04-26T16:33:57.844462+0000","last_active":"2024-04-26T16:34:39.738293+0000","last_peered":"2024-04-26T16:34:39.738293+0000","last_clean":"2024-04-26T16:34:39.738293+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:34:39.738293+0000","last_undegraded":"2024-04-26T16:34:39.738293+0000","last_fullsized":"2024-04-26T16:34:39.738293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:57.844361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:33:57.844361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:01.646789+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1329,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334754+0000","last_change":"2024-04-26T16:33:27.386095+0000","last_active":"2024-04-26T16:34:41.334754+0000","last_peered":"2024-04-26T16:34:41.334754+0000","last_clean":"2024-04-26T16:34:41.334754+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:34:41.334754+0000","last_undegraded":"2024-04-26T16:34:41.334754+0000","last_fullsized":"2024-04-26T16:34:41.334754+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:27.385948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:27.385948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.631499+0000","last_change":"2024-04-26T16:33:48.193084+0000","last_active":"2024-04-26T16:34:40.631499+0000","last_peered":"2024-04-26T16:34:40.631499+0000","last_clean":"2024-04-26T16:34:40.631499+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:34:40.631499+0000","last_undegraded":"2024-04-26T16:34:40.631499+0000","last_fullsized":"2024-04-26T16:34:40.631499+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.192976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.192976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:16.040593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"388'9","reported_seq":1329,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737437+0000","last_change":"2024-04-26T16:33:34.838916+0000","last_active":"2024-04-26T16:34:39.737437+0000","last_peered":"2024-04-26T16:34:39.737437+0000","last_clean":"2024-04-26T16:34:39.737437+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:34:39.737437+0000","last_undegraded":"2024-04-26T16:34:39.737437+0000","last_fullsized":"2024-04-26T16:34:39.737437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:33:34.838685+0000","last_deep_scrub":"388'9","last_deep_scrub_stamp":"2024-04-26T16:31:34.857307+0000","last_clean_scrub_stamp":"2024-04-26T16:33:34.838685+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:46.606726+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":857,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.465075+0000","last_change":"2024-04-26T16:33:02.322246+0000","last_active":"2024-04-26T16:34:41.465075+0000","last_peered":"2024-04-26T16:34:41.465075+0000","last_clean":"2024-04-26T16:17:27.960068+0000","last_became_active":"2024-04-26T16:33:02.322246+0000","last_became_peered":"2024-04-26T16:33:02.322246+0000","last_unstale":"2024-04-26T16:34:41.465075+0000","last_undegraded":"2024-04-26T16:34:41.465075+0000","last_fullsized":"2024-04-26T16:33:02.302200+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:43.402207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:43.402207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1321,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.334478+0000","last_change":"2024-04-26T16:33:21.547155+0000","last_active":"2024-04-26T16:34:41.334478+0000","last_peered":"2024-04-26T16:34:41.334478+0000","last_clean":"2024-04-26T16:34:41.334478+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:34:41.334478+0000","last_undegraded":"2024-04-26T16:34:41.334478+0000","last_fullsized":"2024-04-26T16:34:41.334478+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:21.547000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:21.547000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:45.597222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.469486+0000","last_change":"2024-04-26T16:33:02.314889+0000","last_active":"2024-04-26T16:34:41.469486+0000","last_peered":"2024-04-26T16:34:41.469486+0000","last_clean":"2024-04-26T16:33:00.884617+0000","last_became_active":"2024-04-26T16:33:02.314889+0000","last_became_peered":"2024-04-26T16:33:02.314889+0000","last_unstale":"2024-04-26T16:34:41.469486+0000","last_undegraded":"2024-04-26T16:34:41.469486+0000","last_fullsized":"2024-04-26T16:33:02.299849+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:46.169198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:32:46.169198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1148,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.332813+0000","last_change":"2024-04-26T16:32:23.483970+0000","last_active":"2024-04-26T16:34:41.332813+0000","last_peered":"2024-04-26T16:34:41.332813+0000","last_clean":"2024-04-26T16:34:41.332813+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:34:41.332813+0000","last_undegraded":"2024-04-26T16:34:41.332813+0000","last_fullsized":"2024-04-26T16:34:41.332813+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:23.483751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:32:23.483751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"388'15","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.466178+0000","last_change":"2024-04-26T16:32:37.100484+0000","last_active":"2024-04-26T16:34:41.466178+0000","last_peered":"2024-04-26T16:34:41.466178+0000","last_clean":"2024-04-26T16:34:41.466178+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:34:41.466178+0000","last_undegraded":"2024-04-26T16:34:41.466178+0000","last_fullsized":"2024-04-26T16:34:41.466178+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:32:37.100368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:37.100368+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1309,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.359888+0000","last_change":"2024-04-26T16:33:02.312980+0000","last_active":"2024-04-26T16:34:41.359888+0000","last_peered":"2024-04-26T16:34:41.359888+0000","last_clean":"2024-04-26T16:33:01.288488+0000","last_became_active":"2024-04-26T16:33:02.312980+0000","last_became_peered":"2024-04-26T16:33:02.312980+0000","last_unstale":"2024-04-26T16:34:41.359888+0000","last_undegraded":"2024-04-26T16:34:41.359888+0000","last_fullsized":"2024-04-26T16:33:02.297277+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:18.050865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:31:18.050865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1319,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.464073+0000","last_change":"2024-04-26T16:33:31.257232+0000","last_active":"2024-04-26T16:34:41.464073+0000","last_peered":"2024-04-26T16:34:41.464073+0000","last_clean":"2024-04-26T16:34:41.464073+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:34:41.464073+0000","last_undegraded":"2024-04-26T16:34:41.464073+0000","last_fullsized":"2024-04-26T16:34:41.464073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:31.257048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:31.257048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.103036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1311,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.362190+0000","last_change":"2024-04-26T16:33:02.314710+0000","last_active":"2024-04-26T16:34:41.362190+0000","last_peered":"2024-04-26T16:34:41.362190+0000","last_clean":"2024-04-26T16:33:01.292655+0000","last_became_active":"2024-04-26T16:33:02.314710+0000","last_became_peered":"2024-04-26T16:33:02.314710+0000","last_unstale":"2024-04-26T16:34:41.362190+0000","last_undegraded":"2024-04-26T16:34:41.362190+0000","last_fullsized":"2024-04-26T16:33:02.296386+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:26.949357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:30:26.949357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1170,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333644+0000","last_change":"2024-04-26T16:33:48.754733+0000","last_active":"2024-04-26T16:34:41.333644+0000","last_peered":"2024-04-26T16:34:41.333644+0000","last_clean":"2024-04-26T16:34:41.333644+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:34:41.333644+0000","last_undegraded":"2024-04-26T16:34:41.333644+0000","last_fullsized":"2024-04-26T16:34:41.333644+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:48.754493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:48.754493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:14.498272+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"388'23","reported_seq":261,"reported_epoch":702,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:40.632750+0000","last_change":"2024-04-26T16:33:02.306522+0000","last_active":"2024-04-26T16:34:40.632750+0000","last_peered":"2024-04-26T16:34:40.632750+0000","last_clean":"2024-04-26T16:33:00.425354+0000","last_became_active":"2024-04-26T16:33:02.306522+0000","last_became_peered":"2024-04-26T16:33:02.306522+0000","last_unstale":"2024-04-26T16:34:40.632750+0000","last_undegraded":"2024-04-26T16:33:02.292286+0000","last_fullsized":"2024-04-26T16:33:02.292114+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:32:27.226032+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:32:27.226032+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1325,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333794+0000","last_change":"2024-04-26T16:33:21.559562+0000","last_active":"2024-04-26T16:34:41.333794+0000","last_peered":"2024-04-26T16:34:41.333794+0000","last_clean":"2024-04-26T16:34:41.333794+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:34:41.333794+0000","last_undegraded":"2024-04-26T16:34:41.333794+0000","last_fullsized":"2024-04-26T16:34:41.333794+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:21.559461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:21.559461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.557090+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":238,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.631591+0000","last_change":"2024-04-26T16:32:09.479571+0000","last_active":"2024-04-26T16:34:40.631591+0000","last_peered":"2024-04-26T16:34:40.631591+0000","last_clean":"2024-04-26T16:34:40.631591+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:34:40.631591+0000","last_undegraded":"2024-04-26T16:34:40.631591+0000","last_fullsized":"2024-04-26T16:34:40.631591+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:37.787531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:31:37.787531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.000251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1307,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.735784+0000","last_change":"2024-04-26T16:33:39.721652+0000","last_active":"2024-04-26T16:34:39.735784+0000","last_peered":"2024-04-26T16:34:39.735784+0000","last_clean":"2024-04-26T16:34:39.735784+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:34:39.735784+0000","last_undegraded":"2024-04-26T16:34:39.735784+0000","last_fullsized":"2024-04-26T16:34:39.735784+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:39.721526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:24.567773+0000","last_clean_scrub_stamp":"2024-04-26T16:33:39.721526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:51.845743+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":666,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.462695+0000","last_change":"2024-04-26T16:33:02.311079+0000","last_active":"2024-04-26T16:34:41.462695+0000","last_peered":"2024-04-26T16:34:41.462695+0000","last_clean":"2024-04-26T16:23:28.083853+0000","last_became_active":"2024-04-26T16:33:02.311079+0000","last_became_peered":"2024-04-26T16:33:02.311079+0000","last_unstale":"2024-04-26T16:34:41.462695+0000","last_undegraded":"2024-04-26T16:34:41.462695+0000","last_fullsized":"2024-04-26T16:33:02.295710+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:31:29.174775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:31:29.174775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.427020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"388'27","reported_seq":262,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.633163+0000","last_change":"2024-04-26T16:33:26.326370+0000","last_active":"2024-04-26T16:34:40.633163+0000","last_peered":"2024-04-26T16:34:40.633163+0000","last_clean":"2024-04-26T16:34:40.633163+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:34:40.633163+0000","last_undegraded":"2024-04-26T16:34:40.633163+0000","last_fullsized":"2024-04-26T16:34:40.633163+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'27","last_scrub_stamp":"2024-04-26T16:33:26.326257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:33:26.326257+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1316,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.338167+0000","last_change":"2024-04-26T16:33:02.322670+0000","last_active":"2024-04-26T16:34:41.338167+0000","last_peered":"2024-04-26T16:34:41.338167+0000","last_clean":"2024-04-26T16:33:01.309323+0000","last_became_active":"2024-04-26T16:33:02.322670+0000","last_became_peered":"2024-04-26T16:33:02.322670+0000","last_unstale":"2024-04-26T16:34:41.338167+0000","last_undegraded":"2024-04-26T16:34:41.338167+0000","last_fullsized":"2024-04-26T16:33:02.306498+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:13.187197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:30:13.187197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1296,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.333467+0000","last_change":"2024-04-26T16:33:55.485027+0000","last_active":"2024-04-26T16:34:41.333467+0000","last_peered":"2024-04-26T16:34:41.333467+0000","last_clean":"2024-04-26T16:34:41.333467+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:34:41.333467+0000","last_undegraded":"2024-04-26T16:34:41.333467+0000","last_fullsized":"2024-04-26T16:34:41.333467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:55.484821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:55.484821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:35:04.714695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":250,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.630705+0000","last_change":"2024-04-26T16:33:38.279048+0000","last_active":"2024-04-26T16:34:40.630705+0000","last_peered":"2024-04-26T16:34:40.630705+0000","last_clean":"2024-04-26T16:34:40.630705+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:34:40.630705+0000","last_undegraded":"2024-04-26T16:34:40.630705+0000","last_fullsized":"2024-04-26T16:34:40.630705+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:33:38.278822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:33:38.278822+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:53.457275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1172,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.335336+0000","last_change":"2024-04-26T16:33:02.307065+0000","last_active":"2024-04-26T16:34:41.335336+0000","last_peered":"2024-04-26T16:34:41.335336+0000","last_clean":"2024-04-26T16:33:01.300871+0000","last_became_active":"2024-04-26T16:33:02.307065+0000","last_became_peered":"2024-04-26T16:33:02.307065+0000","last_unstale":"2024-04-26T16:34:41.335336+0000","last_undegraded":"2024-04-26T16:34:41.335336+0000","last_fullsized":"2024-04-26T16:33:02.297497+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:14.201493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:30:14.201493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.526616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"388'19","reported_seq":1392,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.737975+0000","last_change":"2024-04-26T16:32:37.831812+0000","last_active":"2024-04-26T16:34:39.737975+0000","last_peered":"2024-04-26T16:34:39.737975+0000","last_clean":"2024-04-26T16:34:39.737975+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:34:39.737975+0000","last_undegraded":"2024-04-26T16:34:39.737975+0000","last_fullsized":"2024-04-26T16:34:39.737975+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:32:37.831532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:37.831532+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.360116+0000","last_change":"2024-04-26T16:33:41.197285+0000","last_active":"2024-04-26T16:34:41.360116+0000","last_peered":"2024-04-26T16:34:41.360116+0000","last_clean":"2024-04-26T16:34:41.360116+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:34:41.360116+0000","last_undegraded":"2024-04-26T16:34:41.360116+0000","last_fullsized":"2024-04-26T16:34:41.360116+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:41.197189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:33:41.197189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:45.248727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.331861+0000","last_change":"2024-04-26T16:33:29.374847+0000","last_active":"2024-04-26T16:34:41.331861+0000","last_peered":"2024-04-26T16:34:41.331861+0000","last_clean":"2024-04-26T16:34:41.331861+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:34:41.331861+0000","last_undegraded":"2024-04-26T16:34:41.331861+0000","last_fullsized":"2024-04-26T16:34:41.331861+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:29.374752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:33:29.374752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.551862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-04-26T16:34:41.462695+0000","last_change":"2024-04-26T16:33:28.284959+0000","last_active":"2024-04-26T16:34:41.462695+0000","last_peered":"2024-04-26T16:34:41.462695+0000","last_clean":"2024-04-26T16:34:41.462695+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:34:41.462695+0000","last_undegraded":"2024-04-26T16:34:41.462695+0000","last_fullsized":"2024-04-26T16:34:41.462695+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:28.284744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:33:28.284744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.225161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":246,"reported_epoch":702,"state":"active+clean","last_fresh":"2024-04-26T16:34:40.633077+0000","last_change":"2024-04-26T16:33:28.282157+0000","last_active":"2024-04-26T16:34:40.633077+0000","last_peered":"2024-04-26T16:34:40.633077+0000","last_clean":"2024-04-26T16:34:40.633077+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:34:40.633077+0000","last_undegraded":"2024-04-26T16:34:40.633077+0000","last_fullsized":"2024-04-26T16:34:40.633077+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:33:28.281930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:33:28.281930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:57.595110+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"388'20","reported_seq":1345,"reported_epoch":703,"state":"active+undersized+degraded","last_fresh":"2024-04-26T16:34:41.360433+0000","last_change":"2024-04-26T16:33:02.312610+0000","last_active":"2024-04-26T16:34:41.360433+0000","last_peered":"2024-04-26T16:34:41.360433+0000","last_clean":"2024-04-26T16:33:01.292320+0000","last_became_active":"2024-04-26T16:33:02.312610+0000","last_became_peered":"2024-04-26T16:33:02.312610+0000","last_unstale":"2024-04-26T16:34:41.360433+0000","last_undegraded":"2024-04-26T16:33:02.300907+0000","last_fullsized":"2024-04-26T16:33:02.300567+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:32:19.838946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:32:19.838946+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390696+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1320,"reported_epoch":701,"state":"active+clean","last_fresh":"2024-04-26T16:34:39.738003+0000","last_change":"2024-04-26T16:32:25.705015+0000","last_active":"2024-04-26T16:34:39.738003+0000","last_peered":"2024-04-26T16:34:39.738003+0000","last_clean":"2024-04-26T16:34:39.738003+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:34:39.738003+0000","last_undegraded":"2024-04-26T16:34:39.738003+0000","last_fullsized":"2024-04-26T16:34:39.738003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:32:25.704662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:32:25.704662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1324,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.462243+0000","last_change":"2024-04-26T16:33:02.310960+0000","last_active":"2024-04-26T16:34:41.462243+0000","last_peered":"2024-04-26T16:34:41.462243+0000","last_clean":"2024-04-26T16:33:00.882233+0000","last_became_active":"2024-04-26T16:33:02.310960+0000","last_became_peered":"2024-04-26T16:33:02.310960+0000","last_unstale":"2024-04-26T16:34:41.462243+0000","last_undegraded":"2024-04-26T16:34:41.462243+0000","last_fullsized":"2024-04-26T16:33:02.295568+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:22.493412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:30:22.493412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:44.426964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1297,"reported_epoch":703,"state":"active+undersized","last_fresh":"2024-04-26T16:34:41.358487+0000","last_change":"2024-04-26T16:33:02.307893+0000","last_active":"2024-04-26T16:34:41.358487+0000","last_peered":"2024-04-26T16:34:41.358487+0000","last_clean":"2024-04-26T16:33:01.286500+0000","last_became_active":"2024-04-26T16:33:02.307893+0000","last_became_peered":"2024-04-26T16:33:02.307893+0000","last_unstale":"2024-04-26T16:34:41.358487+0000","last_undegraded":"2024-04-26T16:34:41.358487+0000","last_fullsized":"2024-04-26T16:33:02.292847+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:30:12.027062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:30:12.027062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:34:42.390542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":323,"num_pg":8,"stat_sum":{"num_bytes":48,"num_objects":9,"num_object_clones":2,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":6,"num_read_kb":1,"num_write":12,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":86016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23,"ondisk_log_size":23,"up":24,"acting":24,"num_store_stats":7},{"poolid":327,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":83,"acting":83,"num_store_stats":7},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":82,"acting":82,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":85,"acting":85,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":2,"acting":2,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":85,"acting":85,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1010,"num_read_kb":1005,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":88,"acting":88,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":86,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743641,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107364,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93637788,"statfs":{"total":95995035648,"available":95885094912,"internally_reserved":0,"allocated":4083712,"data_stored":2049307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":432,"seq":1855425871928,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40780,"kb_used_data":4160,"kb_used_omap":15,"kb_used_meta":36592,"kb_avail":93704372,"statfs":{"total":95995035648,"available":95953276928,"internally_reserved":0,"allocated":4259840,"data_stored":2015739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15525,"internal_metadata":37471067},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575127,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92864,"kb_used_data":3376,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93652288,"statfs":{"total":95995035648,"available":95899942912,"internally_reserved":0,"allocated":3457024,"data_stored":1448209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738657,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83708,"kb_used_data":3948,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93661444,"statfs":{"total":95995035648,"available":95909318656,"internally_reserved":0,"allocated":4042752,"data_stored":2023055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411596,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97016,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93648136,"statfs":{"total":95995035648,"available":95895691264,"internally_reserved":0,"allocated":3452928,"data_stored":1454196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084545,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102496,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":93642656,"statfs":{"total":95995035648,"available":95890079744,"internally_reserved":0,"allocated":3424256,"data_stored":1430893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":101513672},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855607,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101180,"kb_used_data":3372,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93643972,"statfs":{"total":95995035648,"available":95891427328,"internally_reserved":0,"allocated":3452928,"data_stored":1431649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463149,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66480,"kb_used_data":3356,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93678672,"statfs":{"total":95995035648,"available":95926960128,"internally_reserved":0,"allocated":3436544,"data_stored":1437624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":323,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":327,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:34:46.630 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:46.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:46.754 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:46.773 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.889503+0000 osd.2 (osd.2) 341 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.889503+0000 osd.2 (osd.2) 341 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.891487+0000 osd.2 (osd.2) 342 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:44.891487+0000 osd.2 (osd.2) 342 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.122551+0000 osd.3 (osd.3) 401 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.122551+0000 osd.3 (osd.3) 401 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:34:46.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.123015+0000 osd.3 (osd.3) 402 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.123015+0000 osd.3 (osd.3) 402 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.337004+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.337004+0000 mon.a (mon.0) 3435 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: audit 2024-04-26T16:34:45.339478+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: audit 2024-04-26T16:34:45.339478+0000 mon.a (mon.0) 3436 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-118"}]': finished 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.351296+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.351296+0000 mon.a (mon.0) 3437 : cluster [DBG] osdmap e707: 8 total, 7 up, 8 in 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.909758+0000 osd.0 (osd.0) 395 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.909758+0000 osd.0 (osd.0) 395 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.912225+0000 osd.0 (osd.0) 396 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.912225+0000 osd.0 (osd.0) 396 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:46.327927+0000 mon.a (mon.0) 3438 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:46.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:46 smithi190 bash[23868]: cluster 2024-04-26T16:34:46.327927+0000 mon.a (mon.0) 3438 : cluster [WRN] Health check update: Degraded data redundancy: 77/624 objects degraded (12.340%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:46.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:46.904 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:46.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:47.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:47.034 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:47.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:47.165 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:47.268 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:47.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:47.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:47.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:47.425 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:47.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:47.540 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:34:47.540 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.863753+0000 mgr.y (mgr.24401) 670 : cluster [DBG] pgmap v1100: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 77/624 objects degraded (12.340%) 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.863753+0000 mgr.y (mgr.24401) 670 : cluster [DBG] pgmap v1100: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 77/624 objects degraded (12.340%) 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.879674+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.b scrub starts 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.879674+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.b scrub starts 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.881361+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.b scrub ok 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.881361+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.b scrub ok 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.907415+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.907415+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.909582+0000 osd.5 (osd.5) 254 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:45.909582+0000 osd.5 (osd.5) 254 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:46.359944+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:46.359944+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:46.375421+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2024-04-26T16:34:47.541 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:47 smithi190 bash[23868]: cluster 2024-04-26T16:34:46.375421+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2024-04-26T16:34:47.556 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.863753+0000 mgr.y (mgr.24401) 670 : cluster [DBG] pgmap v1100: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 77/624 objects degraded (12.340%) 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.863753+0000 mgr.y (mgr.24401) 670 : cluster [DBG] pgmap v1100: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 77/624 objects degraded (12.340%) 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.879674+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.b scrub starts 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.879674+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.b scrub starts 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.881361+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.b scrub ok 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.881361+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.b scrub ok 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.907415+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.907415+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:34:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.909582+0000 osd.5 (osd.5) 254 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:45.909582+0000 osd.5 (osd.5) 254 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:46.359944+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:46.359944+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:46.375421+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[21463]: cluster 2024-04-26T16:34:46.375421+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.863753+0000 mgr.y (mgr.24401) 670 : cluster [DBG] pgmap v1100: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 77/624 objects degraded (12.340%) 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.863753+0000 mgr.y (mgr.24401) 670 : cluster [DBG] pgmap v1100: 204 pgs: 52 active+undersized, 20 active+undersized+degraded, 132 active+clean; 583 KiB data, 676 MiB used, 715 GiB / 715 GiB avail; 77/624 objects degraded (12.340%) 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.879674+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.b scrub starts 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.879674+0000 osd.2 (osd.2) 343 : cluster [DBG] 5.b scrub starts 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.881361+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.b scrub ok 2024-04-26T16:34:47.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.881361+0000 osd.2 (osd.2) 344 : cluster [DBG] 5.b scrub ok 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.907415+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.907415+0000 osd.5 (osd.5) 253 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.909582+0000 osd.5 (osd.5) 254 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:45.909582+0000 osd.5 (osd.5) 254 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:46.359944+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:46.359944+0000 mon.a (mon.0) 3439 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:46.375421+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2024-04-26T16:34:47.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:47 smithi050 bash[29234]: cluster 2024-04-26T16:34:46.375421+0000 mon.a (mon.0) 3440 : cluster [DBG] osdmap e708: 8 total, 7 up, 8 in 2024-04-26T16:34:47.659 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:47.688 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:47.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:47.817 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:47.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:47.948 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:47.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:48.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:48.077 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:48.181 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:48.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:48.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:48.339 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:48.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:48.463 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:48.545 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: audit 2024-04-26T16:34:47.198649+0000 mgr.y (mgr.24401) 671 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: audit 2024-04-26T16:34:47.198649+0000 mgr.y (mgr.24401) 671 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.386881+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.386881+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.392250+0000 osd.4 (osd.4) 347 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.392250+0000 osd.4 (osd.4) 347 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: audit 2024-04-26T16:34:47.394105+0000 mon.c (mon.2) 746 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: audit 2024-04-26T16:34:47.394105+0000 mon.c (mon.2) 746 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.558 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: audit 2024-04-26T16:34:47.396360+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: audit 2024-04-26T16:34:47.396360+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.420859+0000 osd.4 (osd.4) 348 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.420859+0000 osd.4 (osd.4) 348 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.431794+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.431794+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.433214+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.433214+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: audit 2024-04-26T16:34:47.198649+0000 mgr.y (mgr.24401) 671 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: audit 2024-04-26T16:34:47.198649+0000 mgr.y (mgr.24401) 671 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:48.559 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.386881+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.386881+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.392250+0000 osd.4 (osd.4) 347 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.392250+0000 osd.4 (osd.4) 347 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: audit 2024-04-26T16:34:47.394105+0000 mon.c (mon.2) 746 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: audit 2024-04-26T16:34:47.394105+0000 mon.c (mon.2) 746 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: audit 2024-04-26T16:34:47.396360+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: audit 2024-04-26T16:34:47.396360+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.420859+0000 osd.4 (osd.4) 348 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.420859+0000 osd.4 (osd.4) 348 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.431794+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.431794+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.433214+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:34:48.560 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:48 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.433214+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:34:48.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: audit 2024-04-26T16:34:47.198649+0000 mgr.y (mgr.24401) 671 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: audit 2024-04-26T16:34:47.198649+0000 mgr.y (mgr.24401) 671 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.386881+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.386881+0000 mon.a (mon.0) 3441 : cluster [DBG] osdmap e709: 8 total, 7 up, 8 in 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.392250+0000 osd.4 (osd.4) 347 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.392250+0000 osd.4 (osd.4) 347 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: audit 2024-04-26T16:34:47.394105+0000 mon.c (mon.2) 746 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: audit 2024-04-26T16:34:47.394105+0000 mon.c (mon.2) 746 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: audit 2024-04-26T16:34:47.396360+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: audit 2024-04-26T16:34:47.396360+0000 mon.a (mon.0) 3442 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.420859+0000 osd.4 (osd.4) 348 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:34:48.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.420859+0000 osd.4 (osd.4) 348 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:34:48.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.431794+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:34:48.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.431794+0000 osd.7 (osd.7) 375 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:34:48.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.433214+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:34:48.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:48 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.433214+0000 osd.7 (osd.7) 376 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:34:48.594 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:48.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:34:48.697 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:48.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:48.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:48.845 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:48.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:48.974 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:48.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:48.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:49.077 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:49.107 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:49.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:49.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:49.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:49.369 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:49.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:49.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:49.635 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:49.711 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:34:49.711 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:34:49.722 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.787123+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.c scrub starts 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.787123+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.c scrub starts 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.789636+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub ok 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.789636+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub ok 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.864895+0000 mgr.y (mgr.24401) 672 : cluster [DBG] pgmap v1103: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:47.864895+0000 mgr.y (mgr.24401) 672 : cluster [DBG] pgmap v1103: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: audit 2024-04-26T16:34:48.393930+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: audit 2024-04-26T16:34:48.393930+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:48.404910+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: cluster 2024-04-26T16:34:48.404910+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: audit 2024-04-26T16:34:48.436620+0000 mon.c (mon.2) 747 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: audit 2024-04-26T16:34:48.436620+0000 mon.c (mon.2) 747 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: audit 2024-04-26T16:34:48.440902+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.723 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[29234]: audit 2024-04-26T16:34:48.440902+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.787123+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.c scrub starts 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.787123+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.c scrub starts 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.789636+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub ok 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.789636+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub ok 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.864895+0000 mgr.y (mgr.24401) 672 : cluster [DBG] pgmap v1103: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:47.864895+0000 mgr.y (mgr.24401) 672 : cluster [DBG] pgmap v1103: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: audit 2024-04-26T16:34:48.393930+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: audit 2024-04-26T16:34:48.393930+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:48.404910+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: cluster 2024-04-26T16:34:48.404910+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2024-04-26T16:34:49.724 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: audit 2024-04-26T16:34:48.436620+0000 mon.c (mon.2) 747 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: audit 2024-04-26T16:34:48.436620+0000 mon.c (mon.2) 747 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: audit 2024-04-26T16:34:48.440902+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.725 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:49 smithi050 bash[21463]: audit 2024-04-26T16:34:48.440902+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.739 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:49.760 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.787123+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.c scrub starts 2024-04-26T16:34:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.787123+0000 osd.1 (osd.1) 373 : cluster [DBG] 5.c scrub starts 2024-04-26T16:34:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.789636+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub ok 2024-04-26T16:34:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.789636+0000 osd.1 (osd.1) 374 : cluster [DBG] 5.c scrub ok 2024-04-26T16:34:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.864895+0000 mgr.y (mgr.24401) 672 : cluster [DBG] pgmap v1103: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:47.864895+0000 mgr.y (mgr.24401) 672 : cluster [DBG] pgmap v1103: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: audit 2024-04-26T16:34:48.393930+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: audit 2024-04-26T16:34:48.393930+0000 mon.a (mon.0) 3443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:48.404910+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2024-04-26T16:34:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: cluster 2024-04-26T16:34:48.404910+0000 mon.a (mon.0) 3444 : cluster [DBG] osdmap e710: 8 total, 7 up, 8 in 2024-04-26T16:34:49.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: audit 2024-04-26T16:34:48.436620+0000 mon.c (mon.2) 747 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: audit 2024-04-26T16:34:48.436620+0000 mon.c (mon.2) 747 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: audit 2024-04-26T16:34:48.440902+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:49 smithi190 bash[23868]: audit 2024-04-26T16:34:48.440902+0000 mon.a (mon.0) 3445 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:49.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:49.889 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:49.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:49.992 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:50.023 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:50.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:50.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:50.154 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:50.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:50.286 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:50.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:50.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:50.423 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:50.526 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.308304+0000 osd.3 (osd.3) 403 : cluster [DBG] 11.b scrub starts 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.308304+0000 osd.3 (osd.3) 403 : cluster [DBG] 11.b scrub starts 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.313309+0000 osd.3 (osd.3) 404 : cluster [DBG] 11.b scrub ok 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.313309+0000 osd.3 (osd.3) 404 : cluster [DBG] 11.b scrub ok 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: audit 2024-04-26T16:34:49.404992+0000 mon.a (mon.0) 3446 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: audit 2024-04-26T16:34:49.404992+0000 mon.a (mon.0) 3446 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.408950+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2024-04-26T16:34:50.538 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.408950+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: audit 2024-04-26T16:34:49.414073+0000 mon.c (mon.2) 748 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: audit 2024-04-26T16:34:49.414073+0000 mon.c (mon.2) 748 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: audit 2024-04-26T16:34:49.415150+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: audit 2024-04-26T16:34:49.415150+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.435814+0000 osd.7 (osd.7) 377 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.435814+0000 osd.7 (osd.7) 377 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.439481+0000 osd.7 (osd.7) 378 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.439481+0000 osd.7 (osd.7) 378 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:34:50.539 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.308304+0000 osd.3 (osd.3) 403 : cluster [DBG] 11.b scrub starts 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.308304+0000 osd.3 (osd.3) 403 : cluster [DBG] 11.b scrub starts 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.313309+0000 osd.3 (osd.3) 404 : cluster [DBG] 11.b scrub ok 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.313309+0000 osd.3 (osd.3) 404 : cluster [DBG] 11.b scrub ok 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: audit 2024-04-26T16:34:49.404992+0000 mon.a (mon.0) 3446 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: audit 2024-04-26T16:34:49.404992+0000 mon.a (mon.0) 3446 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.408950+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.408950+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: audit 2024-04-26T16:34:49.414073+0000 mon.c (mon.2) 748 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: audit 2024-04-26T16:34:49.414073+0000 mon.c (mon.2) 748 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: audit 2024-04-26T16:34:49.415150+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: audit 2024-04-26T16:34:49.415150+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.435814+0000 osd.7 (osd.7) 377 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.435814+0000 osd.7 (osd.7) 377 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:34:50.540 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.439481+0000 osd.7 (osd.7) 378 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:34:50.541 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:50 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.439481+0000 osd.7 (osd.7) 378 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:34:50.555 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:50.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:50.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.308304+0000 osd.3 (osd.3) 403 : cluster [DBG] 11.b scrub starts 2024-04-26T16:34:50.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.308304+0000 osd.3 (osd.3) 403 : cluster [DBG] 11.b scrub starts 2024-04-26T16:34:50.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.313309+0000 osd.3 (osd.3) 404 : cluster [DBG] 11.b scrub ok 2024-04-26T16:34:50.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.313309+0000 osd.3 (osd.3) 404 : cluster [DBG] 11.b scrub ok 2024-04-26T16:34:50.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: audit 2024-04-26T16:34:49.404992+0000 mon.a (mon.0) 3446 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:50.670 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: audit 2024-04-26T16:34:49.404992+0000 mon.a (mon.0) 3446 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.408950+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.408950+0000 mon.a (mon.0) 3447 : cluster [DBG] osdmap e711: 8 total, 7 up, 8 in 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: audit 2024-04-26T16:34:49.414073+0000 mon.c (mon.2) 748 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: audit 2024-04-26T16:34:49.414073+0000 mon.c (mon.2) 748 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: audit 2024-04-26T16:34:49.415150+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: audit 2024-04-26T16:34:49.415150+0000 mon.a (mon.0) 3448 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.435814+0000 osd.7 (osd.7) 377 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.435814+0000 osd.7 (osd.7) 377 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.439481+0000 osd.7 (osd.7) 378 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:34:50.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:50 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.439481+0000 osd.7 (osd.7) 378 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:34:50.684 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:50.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:50.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:50.851 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-04-26T16:34:50.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:50.944 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:50.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:51.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:51.074 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:51.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:51.205 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:51.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:51.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:51.338 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:51.441 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:51.464 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:51.556 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-04-26T16:34:51.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:34:51.556 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-04-26T16:34:51.556 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 1.000000 2024-04-26T16:34:51.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-26T16:34:51.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.866581+0000 mgr.y (mgr.24401) 673 : cluster [DBG] pgmap v1106: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.866581+0000 mgr.y (mgr.24401) 673 : cluster [DBG] pgmap v1106: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.965391+0000 osd.0 (osd.0) 397 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:34:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.965391+0000 osd.0 (osd.0) 397 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:34:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.967801+0000 osd.0 (osd.0) 398 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:34:51.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:49.967801+0000 osd.0 (osd.0) 398 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:50.400169+0000 osd.7 (osd.7) 379 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:50.400169+0000 osd.7 (osd.7) 379 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:50.402661+0000 osd.7 (osd.7) 380 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:50.402661+0000 osd.7 (osd.7) 380 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:50.417545+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:50.417545+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:50.422954+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:50.422954+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:50.429164+0000 mon.c (mon.2) 749 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:50.429164+0000 mon.c (mon.2) 749 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:50.429625+0000 mon.a (mon.0) 3451 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:50.429625+0000 mon.a (mon.0) 3451 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.330495+0000 mon.a (mon.0) 3452 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:51.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.330495+0000 mon.a (mon.0) 3452 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:51.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:51.332564+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]': finished 2024-04-26T16:34:51.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: audit 2024-04-26T16:34:51.332564+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]': finished 2024-04-26T16:34:51.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.336741+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2024-04-26T16:34:51.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:51 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.336741+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2024-04-26T16:34:51.595 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:51.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.866581+0000 mgr.y (mgr.24401) 673 : cluster [DBG] pgmap v1106: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:51.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.866581+0000 mgr.y (mgr.24401) 673 : cluster [DBG] pgmap v1106: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.965391+0000 osd.0 (osd.0) 397 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.965391+0000 osd.0 (osd.0) 397 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.967801+0000 osd.0 (osd.0) 398 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:49.967801+0000 osd.0 (osd.0) 398 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:50.400169+0000 osd.7 (osd.7) 379 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:50.400169+0000 osd.7 (osd.7) 379 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:50.402661+0000 osd.7 (osd.7) 380 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:50.402661+0000 osd.7 (osd.7) 380 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:50.417545+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:50.417545+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:50.422954+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:50.422954+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:50.429164+0000 mon.c (mon.2) 749 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:50.429164+0000 mon.c (mon.2) 749 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:50.429625+0000 mon.a (mon.0) 3451 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:50.429625+0000 mon.a (mon.0) 3451 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.330495+0000 mon.a (mon.0) 3452 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.330495+0000 mon.a (mon.0) 3452 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:51.332564+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]': finished 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: audit 2024-04-26T16:34:51.332564+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]': finished 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.336741+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.336741+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.866581+0000 mgr.y (mgr.24401) 673 : cluster [DBG] pgmap v1106: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.866581+0000 mgr.y (mgr.24401) 673 : cluster [DBG] pgmap v1106: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 77/624 objects degraded (12.340%) 2024-04-26T16:34:51.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.965391+0000 osd.0 (osd.0) 397 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.965391+0000 osd.0 (osd.0) 397 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.967801+0000 osd.0 (osd.0) 398 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:49.967801+0000 osd.0 (osd.0) 398 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:50.400169+0000 osd.7 (osd.7) 379 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:50.400169+0000 osd.7 (osd.7) 379 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:50.402661+0000 osd.7 (osd.7) 380 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:50.402661+0000 osd.7 (osd.7) 380 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:50.417545+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:50.417545+0000 mon.a (mon.0) 3449 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:50.422954+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:50.422954+0000 mon.a (mon.0) 3450 : cluster [DBG] osdmap e712: 8 total, 7 up, 8 in 2024-04-26T16:34:51.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:50.429164+0000 mon.c (mon.2) 749 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:50.429164+0000 mon.c (mon.2) 749 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:50.429625+0000 mon.a (mon.0) 3451 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:50.429625+0000 mon.a (mon.0) 3451 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]: dispatch 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.330495+0000 mon.a (mon.0) 3452 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.330495+0000 mon.a (mon.0) 3452 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:51.332564+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]': finished 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: audit 2024-04-26T16:34:51.332564+0000 mon.a (mon.0) 3453 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-120", "mode": "writeback"}]': finished 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.336741+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2024-04-26T16:34:51.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:51 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.336741+0000 mon.a (mon.0) 3454 : cluster [DBG] osdmap e713: 8 total, 7 up, 8 in 2024-04-26T16:34:51.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:51.723 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:51.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:51.854 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:51.957 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:51.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:51.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:52.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:52.116 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:52.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:52.246 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:52.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:52.377 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:52.480 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:52.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.373250+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.e scrub starts 2024-04-26T16:34:52.492 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.373250+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.e scrub starts 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.375965+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.e scrub ok 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.375965+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.e scrub ok 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: audit 2024-04-26T16:34:51.444542+0000 mon.c (mon.2) 750 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: audit 2024-04-26T16:34:51.444542+0000 mon.c (mon.2) 750 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: audit 2024-04-26T16:34:51.444991+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: audit 2024-04-26T16:34:51.444991+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.449676+0000 osd.7 (osd.7) 381 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.449676+0000 osd.7 (osd.7) 381 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.453051+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.453051+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.330369+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.330369+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.334188+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:34:52.493 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.334188+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.373250+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.e scrub starts 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.373250+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.e scrub starts 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.375965+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.e scrub ok 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.375965+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.e scrub ok 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: audit 2024-04-26T16:34:51.444542+0000 mon.c (mon.2) 750 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: audit 2024-04-26T16:34:51.444542+0000 mon.c (mon.2) 750 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: audit 2024-04-26T16:34:51.444991+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.494 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: audit 2024-04-26T16:34:51.444991+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.449676+0000 osd.7 (osd.7) 381 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.449676+0000 osd.7 (osd.7) 381 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.453051+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.453051+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.330369+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.330369+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.334188+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:34:52.495 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:52 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.334188+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:34:52.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:52.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:52.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.373250+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.e scrub starts 2024-04-26T16:34:52.621 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.373250+0000 osd.4 (osd.4) 349 : cluster [DBG] 5.e scrub starts 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.375965+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.e scrub ok 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.375965+0000 osd.4 (osd.4) 350 : cluster [DBG] 5.e scrub ok 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: audit 2024-04-26T16:34:51.444542+0000 mon.c (mon.2) 750 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: audit 2024-04-26T16:34:51.444542+0000 mon.c (mon.2) 750 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: audit 2024-04-26T16:34:51.444991+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: audit 2024-04-26T16:34:51.444991+0000 mon.a (mon.0) 3455 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.449676+0000 osd.7 (osd.7) 381 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.449676+0000 osd.7 (osd.7) 381 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.453051+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.453051+0000 osd.7 (osd.7) 382 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.330369+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:34:52.622 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.330369+0000 osd.4 (osd.4) 351 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:34:52.623 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.334188+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:34:52.623 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:52 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.334188+0000 osd.4 (osd.4) 352 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:34:52.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:52.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:52.768 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:52.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:52.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:52.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:52.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:52.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:53.027 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:53.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:53.157 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:53.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:53.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:53.412 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:53.514 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:53.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.764446+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:34:53.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.764446+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:34:53.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.765420+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:34:53.525 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.765420+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.868302+0000 mgr.y (mgr.24401) 674 : cluster [DBG] pgmap v1109: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:51.868302+0000 mgr.y (mgr.24401) 674 : cluster [DBG] pgmap v1109: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.271658+0000 osd.3 (osd.3) 405 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.271658+0000 osd.3 (osd.3) 405 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.273525+0000 osd.3 (osd.3) 406 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.273525+0000 osd.3 (osd.3) 406 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.420355+0000 mon.a (mon.0) 3456 : cluster [WRN] Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.420355+0000 mon.a (mon.0) 3456 : cluster [WRN] Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: audit 2024-04-26T16:34:52.432474+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: audit 2024-04-26T16:34:52.432474+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.436683+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:52.436683+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2024-04-26T16:34:53.526 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: audit 2024-04-26T16:34:52.441494+0000 mon.c (mon.2) 751 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: audit 2024-04-26T16:34:52.441494+0000 mon.c (mon.2) 751 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: audit 2024-04-26T16:34:52.441965+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: audit 2024-04-26T16:34:52.441965+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.320530+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.320530+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.324746+0000 osd.4 (osd.4) 354 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.324746+0000 osd.4 (osd.4) 354 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.764446+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.764446+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:34:53.527 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.765420+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.765420+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.868302+0000 mgr.y (mgr.24401) 674 : cluster [DBG] pgmap v1109: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:51.868302+0000 mgr.y (mgr.24401) 674 : cluster [DBG] pgmap v1109: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.271658+0000 osd.3 (osd.3) 405 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.271658+0000 osd.3 (osd.3) 405 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.273525+0000 osd.3 (osd.3) 406 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.273525+0000 osd.3 (osd.3) 406 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.420355+0000 mon.a (mon.0) 3456 : cluster [WRN] Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.420355+0000 mon.a (mon.0) 3456 : cluster [WRN] Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: audit 2024-04-26T16:34:52.432474+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: audit 2024-04-26T16:34:52.432474+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.436683+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2024-04-26T16:34:53.528 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:52.436683+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: audit 2024-04-26T16:34:52.441494+0000 mon.c (mon.2) 751 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: audit 2024-04-26T16:34:52.441494+0000 mon.c (mon.2) 751 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: audit 2024-04-26T16:34:52.441965+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: audit 2024-04-26T16:34:52.441965+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.320530+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.320530+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.324746+0000 osd.4 (osd.4) 354 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:34:53.529 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:53 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.324746+0000 osd.4 (osd.4) 354 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:34:53.540 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:53.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:53.674 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:53.777 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:53.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.764446+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.764446+0000 osd.1 (osd.1) 375 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.765420+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.765420+0000 osd.1 (osd.1) 376 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.868302+0000 mgr.y (mgr.24401) 674 : cluster [DBG] pgmap v1109: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:51.868302+0000 mgr.y (mgr.24401) 674 : cluster [DBG] pgmap v1109: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.271658+0000 osd.3 (osd.3) 405 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.271658+0000 osd.3 (osd.3) 405 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.273525+0000 osd.3 (osd.3) 406 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:34:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.273525+0000 osd.3 (osd.3) 406 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.420355+0000 mon.a (mon.0) 3456 : cluster [WRN] Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.420355+0000 mon.a (mon.0) 3456 : cluster [WRN] Health check update: Degraded data redundancy: 78/627 objects degraded (12.440%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: audit 2024-04-26T16:34:52.432474+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: audit 2024-04-26T16:34:52.432474+0000 mon.a (mon.0) 3457 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.436683+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:52.436683+0000 mon.a (mon.0) 3458 : cluster [DBG] osdmap e714: 8 total, 7 up, 8 in 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: audit 2024-04-26T16:34:52.441494+0000 mon.c (mon.2) 751 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: audit 2024-04-26T16:34:52.441494+0000 mon.c (mon.2) 751 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: audit 2024-04-26T16:34:52.441965+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: audit 2024-04-26T16:34:52.441965+0000 mon.a (mon.0) 3459 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]: dispatch 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.320530+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.320530+0000 osd.4 (osd.4) 353 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.324746+0000 osd.4 (osd.4) 354 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:34:53.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:53 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.324746+0000 osd.4 (osd.4) 354 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:34:53.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:53.932 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:53.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:53.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:54.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:54.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:54.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:54.195 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:54.298 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:54.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:54.388 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:54.389 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:54.432 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:54.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:54.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:54.573 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:34:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.432730+0000 mon.a (mon.0) 3460 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.432730+0000 mon.a (mon.0) 3460 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: audit 2024-04-26T16:34:53.434137+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: audit 2024-04-26T16:34:53.434137+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.437704+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.437704+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.346570+0000 osd.4 (osd.4) 355 : cluster [DBG] 323.5s0 scrub starts 2024-04-26T16:34:54.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.346570+0000 osd.4 (osd.4) 355 : cluster [DBG] 323.5s0 scrub starts 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.349370+0000 osd.4 (osd.4) 356 : cluster [DBG] 323.5 scrub ok 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.349370+0000 osd.4 (osd.4) 356 : cluster [DBG] 323.5 scrub ok 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.432730+0000 mon.a (mon.0) 3460 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.432730+0000 mon.a (mon.0) 3460 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: audit 2024-04-26T16:34:53.434137+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: audit 2024-04-26T16:34:53.434137+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.437704+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.437704+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.346570+0000 osd.4 (osd.4) 355 : cluster [DBG] 323.5s0 scrub starts 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.346570+0000 osd.4 (osd.4) 355 : cluster [DBG] 323.5s0 scrub starts 2024-04-26T16:34:54.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.349370+0000 osd.4 (osd.4) 356 : cluster [DBG] 323.5 scrub ok 2024-04-26T16:34:54.576 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:54 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.349370+0000 osd.4 (osd.4) 356 : cluster [DBG] 323.5 scrub ok 2024-04-26T16:34:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:54.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:54.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:54.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.432730+0000 mon.a (mon.0) 3460 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.432730+0000 mon.a (mon.0) 3460 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: audit 2024-04-26T16:34:53.434137+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: audit 2024-04-26T16:34:53.434137+0000 mon.a (mon.0) 3461 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-120"}]': finished 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.437704+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.437704+0000 mon.a (mon.0) 3462 : cluster [DBG] osdmap e715: 8 total, 7 up, 8 in 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.346570+0000 osd.4 (osd.4) 355 : cluster [DBG] 323.5s0 scrub starts 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.346570+0000 osd.4 (osd.4) 355 : cluster [DBG] 323.5s0 scrub starts 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.349370+0000 osd.4 (osd.4) 356 : cluster [DBG] 323.5 scrub ok 2024-04-26T16:34:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:54 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.349370+0000 osd.4 (osd.4) 356 : cluster [DBG] 323.5 scrub ok 2024-04-26T16:34:54.854 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:54.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:54.983 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:54.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:55.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:55.116 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:55.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.248 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:55.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:55.402 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:34:55.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.503 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:55.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:55.634 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:55.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:55.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.763008+0000 osd.5 (osd.5) 255 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:34:55.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.763008+0000 osd.5 (osd.5) 255 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:34:55.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.766744+0000 osd.5 (osd.5) 256 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:34:55.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.766744+0000 osd.5 (osd.5) 256 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:34:55.663 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.870132+0000 mgr.y (mgr.24401) 675 : cluster [DBG] pgmap v1112: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:53.870132+0000 mgr.y (mgr.24401) 675 : cluster [DBG] pgmap v1112: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.434187+0000 mon.a (mon.0) 3463 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.434187+0000 mon.a (mon.0) 3463 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.445249+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.445249+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.360447+0000 osd.4 (osd.4) 357 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.360447+0000 osd.4 (osd.4) 357 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.362836+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.362836+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:34:55.664 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.763008+0000 osd.5 (osd.5) 255 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.763008+0000 osd.5 (osd.5) 255 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.766744+0000 osd.5 (osd.5) 256 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.766744+0000 osd.5 (osd.5) 256 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.870132+0000 mgr.y (mgr.24401) 675 : cluster [DBG] pgmap v1112: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:53.870132+0000 mgr.y (mgr.24401) 675 : cluster [DBG] pgmap v1112: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.434187+0000 mon.a (mon.0) 3463 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.434187+0000 mon.a (mon.0) 3463 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.445249+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.445249+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.360447+0000 osd.4 (osd.4) 357 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.360447+0000 osd.4 (osd.4) 357 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.362836+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:34:55.665 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:55 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.362836+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:34:55.737 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.763008+0000 osd.5 (osd.5) 255 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.763008+0000 osd.5 (osd.5) 255 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.766744+0000 osd.5 (osd.5) 256 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.766744+0000 osd.5 (osd.5) 256 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.870132+0000 mgr.y (mgr.24401) 675 : cluster [DBG] pgmap v1112: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:53.870132+0000 mgr.y (mgr.24401) 675 : cluster [DBG] pgmap v1112: 204 pgs: 48 active+undersized, 20 active+undersized+degraded, 136 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s; 78/627 objects degraded (12.440%) 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.434187+0000 mon.a (mon.0) 3463 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.434187+0000 mon.a (mon.0) 3463 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:34:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.445249+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2024-04-26T16:34:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.445249+0000 mon.a (mon.0) 3464 : cluster [DBG] osdmap e716: 8 total, 7 up, 8 in 2024-04-26T16:34:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.360447+0000 osd.4 (osd.4) 357 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:34:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.360447+0000 osd.4 (osd.4) 357 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:34:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.362836+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:34:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:55 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.362836+0000 osd.4 (osd.4) 358 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:34:55.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:55.896 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:55.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:55.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:56.028 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:56.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:56.160 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:56.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:56.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:56.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:56.421 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:56.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:56.550 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:56.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.723882+0000 osd.5 (osd.5) 257 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.723882+0000 osd.5 (osd.5) 257 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.726461+0000 osd.5 (osd.5) 258 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:54.726461+0000 osd.5 (osd.5) 258 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.239491+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.239491+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:34:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.241244+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.241244+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.447459+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.447459+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: audit 2024-04-26T16:34:55.449068+0000 mon.c (mon.2) 752 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: audit 2024-04-26T16:34:55.449068+0000 mon.c (mon.2) 752 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: audit 2024-04-26T16:34:55.449485+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: audit 2024-04-26T16:34:55.449485+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.734718+0000 osd.1 (osd.1) 377 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.734718+0000 osd.1 (osd.1) 377 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.737443+0000 osd.1 (osd.1) 378 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:34:56.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:56 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.737443+0000 osd.1 (osd.1) 378 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:34:56.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:56.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:56.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.723882+0000 osd.5 (osd.5) 257 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.723882+0000 osd.5 (osd.5) 257 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.726461+0000 osd.5 (osd.5) 258 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:54.726461+0000 osd.5 (osd.5) 258 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.239491+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.239491+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.241244+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.241244+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.447459+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2024-04-26T16:34:56.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.447459+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: audit 2024-04-26T16:34:55.449068+0000 mon.c (mon.2) 752 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: audit 2024-04-26T16:34:55.449068+0000 mon.c (mon.2) 752 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: audit 2024-04-26T16:34:55.449485+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: audit 2024-04-26T16:34:55.449485+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.734718+0000 osd.1 (osd.1) 377 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.734718+0000 osd.1 (osd.1) 377 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.737443+0000 osd.1 (osd.1) 378 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.737443+0000 osd.1 (osd.1) 378 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.723882+0000 osd.5 (osd.5) 257 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.723882+0000 osd.5 (osd.5) 257 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.726461+0000 osd.5 (osd.5) 258 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:34:56.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:54.726461+0000 osd.5 (osd.5) 258 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.239491+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.239491+0000 osd.3 (osd.3) 407 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.241244+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.241244+0000 osd.3 (osd.3) 408 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.447459+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.447459+0000 mon.a (mon.0) 3465 : cluster [DBG] osdmap e717: 8 total, 7 up, 8 in 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: audit 2024-04-26T16:34:55.449068+0000 mon.c (mon.2) 752 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: audit 2024-04-26T16:34:55.449068+0000 mon.c (mon.2) 752 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: audit 2024-04-26T16:34:55.449485+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: audit 2024-04-26T16:34:55.449485+0000 mon.a (mon.0) 3466 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.734718+0000 osd.1 (osd.1) 377 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.734718+0000 osd.1 (osd.1) 377 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.737443+0000 osd.1 (osd.1) 378 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:34:56.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:56 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.737443+0000 osd.1 (osd.1) 378 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:34:56.806 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:56.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:56.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:56.938 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:56.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:57.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:57.070 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:57.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:57.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:57.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:57.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:57.335 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:57.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:57.480 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:34:57.573 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.760753+0000 osd.5 (osd.5) 259 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.760753+0000 osd.5 (osd.5) 259 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.762893+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.762893+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.872173+0000 mgr.y (mgr.24401) 676 : cluster [DBG] pgmap v1115: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:55.872173+0000 mgr.y (mgr.24401) 676 : cluster [DBG] pgmap v1115: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.262725+0000 osd.3 (osd.3) 409 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.262725+0000 osd.3 (osd.3) 409 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:34:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.267702+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.267702+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:56.446399+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:56.446399+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.457692+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.457692+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:56.483408+0000 mon.c (mon.2) 753 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:56.483408+0000 mon.c (mon.2) 753 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:56.498464+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:56.498464+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.516269+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:34:57.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.516269+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:34:57.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.518626+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:34:57.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.518626+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:34:57.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:57.450039+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:57.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:57.450039+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:57.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:57.462875+0000 mon.c (mon.2) 754 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:57.462875+0000 mon.c (mon.2) 754 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.465013+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2024-04-26T16:34:57.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.465013+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2024-04-26T16:34:57.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:57.467281+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.578 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:57 smithi190 bash[23868]: audit 2024-04-26T16:34:57.467281+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:57.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:57.730 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:57.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:57.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.760753+0000 osd.5 (osd.5) 259 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:34:57.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.760753+0000 osd.5 (osd.5) 259 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:34:57.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.762893+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:34:57.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.762893+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:34:57.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.872173+0000 mgr.y (mgr.24401) 676 : cluster [DBG] pgmap v1115: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:55.872173+0000 mgr.y (mgr.24401) 676 : cluster [DBG] pgmap v1115: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.262725+0000 osd.3 (osd.3) 409 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.262725+0000 osd.3 (osd.3) 409 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.267702+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.267702+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:56.446399+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:56.446399+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.457692+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.457692+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:56.483408+0000 mon.c (mon.2) 753 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:56.483408+0000 mon.c (mon.2) 753 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:56.498464+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:56.498464+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.516269+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.516269+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.518626+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.518626+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:57.450039+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:57.450039+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:57.462875+0000 mon.c (mon.2) 754 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:57.462875+0000 mon.c (mon.2) 754 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.465013+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.465013+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:57.467281+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[21463]: audit 2024-04-26T16:34:57.467281+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.760753+0000 osd.5 (osd.5) 259 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.760753+0000 osd.5 (osd.5) 259 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.762893+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.762893+0000 osd.5 (osd.5) 260 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.872173+0000 mgr.y (mgr.24401) 676 : cluster [DBG] pgmap v1115: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:55.872173+0000 mgr.y (mgr.24401) 676 : cluster [DBG] pgmap v1115: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 666 MiB used, 715 GiB / 715 GiB avail; 78/627 objects degraded (12.440%) 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.262725+0000 osd.3 (osd.3) 409 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.262725+0000 osd.3 (osd.3) 409 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.267702+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:34:57.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.267702+0000 osd.3 (osd.3) 410 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:56.446399+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:56.446399+0000 mon.a (mon.0) 3467 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.457692+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.457692+0000 mon.a (mon.0) 3468 : cluster [DBG] osdmap e718: 8 total, 7 up, 8 in 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:56.483408+0000 mon.c (mon.2) 753 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:56.483408+0000 mon.c (mon.2) 753 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:56.498464+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:56.498464+0000 mon.a (mon.0) 3469 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.516269+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.516269+0000 osd.7 (osd.7) 383 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.518626+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.518626+0000 osd.7 (osd.7) 384 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:57.450039+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:57.450039+0000 mon.a (mon.0) 3470 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:34:57.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:57.462875+0000 mon.c (mon.2) 754 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:57.462875+0000 mon.c (mon.2) 754 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.465013+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2024-04-26T16:34:57.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.465013+0000 mon.a (mon.0) 3471 : cluster [DBG] osdmap e719: 8 total, 7 up, 8 in 2024-04-26T16:34:57.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:57.467281+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:57 smithi050 bash[29234]: audit 2024-04-26T16:34:57.467281+0000 mon.a (mon.0) 3472 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:34:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.862 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:57.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:57.963 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:57.987 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:57.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:58.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:58.116 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:58.219 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:58.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:58.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:58.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:58.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:58.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:58.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:58.612 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:58.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.712055+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.712055+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.714516+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:56.714516+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:57.206937+0000 mgr.y (mgr.24401) 677 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:57.206937+0000 mgr.y (mgr.24401) 677 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.379799+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.379799+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.381316+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.381316+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:34:58.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:57.606146+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:57.606146+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:57.607699+0000 mon.c (mon.2) 755 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:57.607699+0000 mon.c (mon.2) 755 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.900082+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.900082+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.903347+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.903347+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:58.452387+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:58.452387+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.459894+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.459894+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:58.467618+0000 mon.c (mon.2) 756 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:58.467618+0000 mon.c (mon.2) 756 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:58.468496+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.627 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[21463]: audit 2024-04-26T16:34:58.468496+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.712055+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.712055+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.714516+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:56.714516+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:57.206937+0000 mgr.y (mgr.24401) 677 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:57.206937+0000 mgr.y (mgr.24401) 677 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.379799+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.379799+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.381316+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:34:58.628 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.381316+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:57.606146+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:57.606146+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:57.607699+0000 mon.c (mon.2) 755 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:57.607699+0000 mon.c (mon.2) 755 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.900082+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.900082+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.903347+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.903347+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:58.452387+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:58.452387+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.459894+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.459894+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:58.467618+0000 mon.c (mon.2) 756 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:58.467618+0000 mon.c (mon.2) 756 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:58.468496+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:58 smithi050 bash[29234]: audit 2024-04-26T16:34:58.468496+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:58.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:58.771 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:34:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:34:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.712055+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:34:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.712055+0000 osd.5 (osd.5) 261 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.714516+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:56.714516+0000 osd.5 (osd.5) 262 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:57.206937+0000 mgr.y (mgr.24401) 677 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:57.206937+0000 mgr.y (mgr.24401) 677 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.379799+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.379799+0000 osd.4 (osd.4) 359 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.381316+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.381316+0000 osd.4 (osd.4) 360 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:57.606146+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:57.606146+0000 mon.a (mon.0) 3473 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:57.607699+0000 mon.c (mon.2) 755 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:57.607699+0000 mon.c (mon.2) 755 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.900082+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:34:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.900082+0000 osd.0 (osd.0) 399 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.903347+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.903347+0000 osd.0 (osd.0) 400 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:58.452387+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:58.452387+0000 mon.a (mon.0) 3474 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.459894+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.459894+0000 mon.a (mon.0) 3475 : cluster [DBG] osdmap e720: 8 total, 7 up, 8 in 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:58.467618+0000 mon.c (mon.2) 756 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:58.467618+0000 mon.c (mon.2) 756 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:58.468496+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:58 smithi190 bash[23868]: audit 2024-04-26T16:34:58.468496+0000 mon.a (mon.0) 3476 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]: dispatch 2024-04-26T16:34:58.874 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:58.905 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:34:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:58.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:34:59.008 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:59.038 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:34:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:34:59.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:59.169 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:59.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:59.301 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:34:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:34:59.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:59.430 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:34:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.748251+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:34:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.748251+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:34:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.751742+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:34:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.751742+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:34:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.873276+0000 mgr.y (mgr.24401) 678 : cluster [DBG] pgmap v1118: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:34:59.507 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:57.873276+0000 mgr.y (mgr.24401) 678 : cluster [DBG] pgmap v1118: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.401797+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.e scrub starts 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.401797+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.e scrub starts 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.406126+0000 osd.4 (osd.4) 362 : cluster [DBG] 6.e scrub ok 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.406126+0000 osd.4 (osd.4) 362 : cluster [DBG] 6.e scrub ok 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.554231+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.554231+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.556720+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.556720+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.603170+0000 mon.a (mon.0) 3477 : cluster [WRN] Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.603170+0000 mon.a (mon.0) 3477 : cluster [WRN] Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.916579+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.916579+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.918086+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:58.918086+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-04-26T16:34:59.508 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.452887+0000 mon.a (mon.0) 3478 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.452887+0000 mon.a (mon.0) 3478 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: audit 2024-04-26T16:34:59.455012+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]': finished 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: audit 2024-04-26T16:34:59.455012+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]': finished 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.462960+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.462960+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.748251+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.748251+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.751742+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.751742+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.873276+0000 mgr.y (mgr.24401) 678 : cluster [DBG] pgmap v1118: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:57.873276+0000 mgr.y (mgr.24401) 678 : cluster [DBG] pgmap v1118: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:34:59.509 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.401797+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.e scrub starts 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.401797+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.e scrub starts 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.406126+0000 osd.4 (osd.4) 362 : cluster [DBG] 6.e scrub ok 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.406126+0000 osd.4 (osd.4) 362 : cluster [DBG] 6.e scrub ok 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.554231+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.554231+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.556720+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.556720+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.603170+0000 mon.a (mon.0) 3477 : cluster [WRN] Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.603170+0000 mon.a (mon.0) 3477 : cluster [WRN] Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.916579+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.916579+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.918086+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:58.918086+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.452887+0000 mon.a (mon.0) 3478 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:59.510 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.452887+0000 mon.a (mon.0) 3478 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:59.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: audit 2024-04-26T16:34:59.455012+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]': finished 2024-04-26T16:34:59.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: audit 2024-04-26T16:34:59.455012+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]': finished 2024-04-26T16:34:59.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.462960+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2024-04-26T16:34:59.511 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:34:59 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.462960+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2024-04-26T16:34:59.533 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:59.556 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:59.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:59.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:59.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.748251+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:34:59.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.748251+0000 osd.5 (osd.5) 263 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:34:59.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.751742+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:34:59.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.751742+0000 osd.5 (osd.5) 264 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.873276+0000 mgr.y (mgr.24401) 678 : cluster [DBG] pgmap v1118: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:57.873276+0000 mgr.y (mgr.24401) 678 : cluster [DBG] pgmap v1118: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.401797+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.e scrub starts 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.401797+0000 osd.4 (osd.4) 361 : cluster [DBG] 6.e scrub starts 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.406126+0000 osd.4 (osd.4) 362 : cluster [DBG] 6.e scrub ok 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.406126+0000 osd.4 (osd.4) 362 : cluster [DBG] 6.e scrub ok 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.554231+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.554231+0000 osd.7 (osd.7) 385 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.556720+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.556720+0000 osd.7 (osd.7) 386 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.603170+0000 mon.a (mon.0) 3477 : cluster [WRN] Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.603170+0000 mon.a (mon.0) 3477 : cluster [WRN] Health check update: Degraded data redundancy: 79/630 objects degraded (12.540%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.916579+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.916579+0000 osd.2 (osd.2) 345 : cluster [DBG] 2.e scrub starts 2024-04-26T16:34:59.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.918086+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:58.918086+0000 osd.2 (osd.2) 346 : cluster [DBG] 2.e scrub ok 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.452887+0000 mon.a (mon.0) 3478 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.452887+0000 mon.a (mon.0) 3478 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: audit 2024-04-26T16:34:59.455012+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]': finished 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: audit 2024-04-26T16:34:59.455012+0000 mon.a (mon.0) 3479 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-122", "mode": "writeback"}]': finished 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.462960+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2024-04-26T16:34:59.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:34:59 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.462960+0000 mon.a (mon.0) 3480 : cluster [DBG] osdmap e721: 8 total, 7 up, 8 in 2024-04-26T16:34:59.685 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:34:59.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:34:59.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:59.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:34:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:34:59.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:59.943 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:34:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:34:59.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:00.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:00.073 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:00.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:00.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:00.332 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:00.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.463 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:00.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.593 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:00.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.310777+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:35:00.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.310777+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:35:00.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.313011+0000 osd.3 (osd.3) 412 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:35:00.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.313011+0000 osd.3 (osd.3) 412 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:35:00.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: audit 2024-04-26T16:34:59.564037+0000 mon.c (mon.2) 757 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: audit 2024-04-26T16:34:59.564037+0000 mon.c (mon.2) 757 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: audit 2024-04-26T16:34:59.564385+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: audit 2024-04-26T16:34:59.564385+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.576201+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.576201+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.578535+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.578535+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.725314+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.725314+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.728399+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.728399+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.941823+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.d scrub starts 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.941823+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.d scrub starts 2024-04-26T16:35:00.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.943299+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.d scrub ok 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.943299+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.d scrub ok 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.310777+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.310777+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.313011+0000 osd.3 (osd.3) 412 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.313011+0000 osd.3 (osd.3) 412 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: audit 2024-04-26T16:34:59.564037+0000 mon.c (mon.2) 757 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: audit 2024-04-26T16:34:59.564037+0000 mon.c (mon.2) 757 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: audit 2024-04-26T16:34:59.564385+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: audit 2024-04-26T16:34:59.564385+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.576201+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.576201+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.578535+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:35:00.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.578535+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.725314+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.725314+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.728399+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.728399+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.941823+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.d scrub starts 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.941823+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.d scrub starts 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.943299+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.d scrub ok 2024-04-26T16:35:00.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:00 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.943299+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.d scrub ok 2024-04-26T16:35:00.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.310777+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.310777+0000 osd.3 (osd.3) 411 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.313011+0000 osd.3 (osd.3) 412 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.313011+0000 osd.3 (osd.3) 412 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: audit 2024-04-26T16:34:59.564037+0000 mon.c (mon.2) 757 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: audit 2024-04-26T16:34:59.564037+0000 mon.c (mon.2) 757 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: audit 2024-04-26T16:34:59.564385+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: audit 2024-04-26T16:34:59.564385+0000 mon.a (mon.0) 3481 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.576201+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.576201+0000 osd.7 (osd.7) 387 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.578535+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:35:00.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.578535+0000 osd.7 (osd.7) 388 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.725314+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.725314+0000 osd.1 (osd.1) 379 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.728399+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.728399+0000 osd.1 (osd.1) 380 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.941823+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.d scrub starts 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.941823+0000 osd.2 (osd.2) 347 : cluster [DBG] 5.d scrub starts 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.943299+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.d scrub ok 2024-04-26T16:35:00.708 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:00 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.943299+0000 osd.2 (osd.2) 348 : cluster [DBG] 5.d scrub ok 2024-04-26T16:35:00.722 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:00.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:00.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:00.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:00.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:00.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:00.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:01.089 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:01.119 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:01.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:01.248 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:01.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:01.378 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:01.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:01.481 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:01.510 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.874287+0000 mgr.y (mgr.24401) 679 : cluster [DBG] pgmap v1121: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.874287+0000 mgr.y (mgr.24401) 679 : cluster [DBG] pgmap v1121: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.977629+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.977629+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.979664+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:34:59.979664+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:00.503898+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:00.503898+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:00.512619+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:00.512619+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:00.514545+0000 mon.c (mon.2) 758 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:00.514545+0000 mon.c (mon.2) 758 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:00.518261+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.563 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:00.518261+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:00.561659+0000 osd.7 (osd.7) 389 : cluster [DBG] 323.1s0 scrub starts 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:00.561659+0000 osd.7 (osd.7) 389 : cluster [DBG] 323.1s0 scrub starts 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:00.565092+0000 osd.7 (osd.7) 390 : cluster [DBG] 323.1 scrub ok 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:00.565092+0000 osd.7 (osd.7) 390 : cluster [DBG] 323.1 scrub ok 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.334134+0000 mon.a (mon.0) 3485 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.334134+0000 mon.a (mon.0) 3485 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.335369+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.335369+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:01.338104+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: audit 2024-04-26T16:35:01.338104+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.348597+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.348597+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2024-04-26T16:35:01.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.874287+0000 mgr.y (mgr.24401) 679 : cluster [DBG] pgmap v1121: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.874287+0000 mgr.y (mgr.24401) 679 : cluster [DBG] pgmap v1121: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.977629+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.977629+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.979664+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:34:59.979664+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:00.503898+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:00.503898+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:00.512619+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:00.512619+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:00.514545+0000 mon.c (mon.2) 758 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:00.514545+0000 mon.c (mon.2) 758 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:00.518261+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.565 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:00.518261+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.613 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:01.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:01.744 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:01.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:00.561659+0000 osd.7 (osd.7) 389 : cluster [DBG] 323.1s0 scrub starts 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:00.561659+0000 osd.7 (osd.7) 389 : cluster [DBG] 323.1s0 scrub starts 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:00.565092+0000 osd.7 (osd.7) 390 : cluster [DBG] 323.1 scrub ok 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:00.565092+0000 osd.7 (osd.7) 390 : cluster [DBG] 323.1 scrub ok 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.334134+0000 mon.a (mon.0) 3485 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.334134+0000 mon.a (mon.0) 3485 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.335369+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.335369+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:01.338104+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: audit 2024-04-26T16:35:01.338104+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.348597+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2024-04-26T16:35:01.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.348597+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2024-04-26T16:35:01.770 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.874287+0000 mgr.y (mgr.24401) 679 : cluster [DBG] pgmap v1121: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:35:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.874287+0000 mgr.y (mgr.24401) 679 : cluster [DBG] pgmap v1121: 204 pgs: 5 creating+activating, 5 creating+peering, 4 creating+activating+undersized, 45 active+undersized, 20 active+undersized+degraded, 125 active+clean; 583 KiB data, 688 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 79/630 objects degraded (12.540%) 2024-04-26T16:35:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.977629+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.977629+0000 osd.0 (osd.0) 401 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.979664+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:34:59.979664+0000 osd.0 (osd.0) 402 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:00.503898+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:00.503898+0000 mon.a (mon.0) 3482 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:00.512619+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:00.512619+0000 mon.a (mon.0) 3483 : cluster [DBG] osdmap e722: 8 total, 7 up, 8 in 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:00.514545+0000 mon.c (mon.2) 758 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:00.514545+0000 mon.c (mon.2) 758 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:00.518261+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:00.518261+0000 mon.a (mon.0) 3484 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]: dispatch 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:00.561659+0000 osd.7 (osd.7) 389 : cluster [DBG] 323.1s0 scrub starts 2024-04-26T16:35:01.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:00.561659+0000 osd.7 (osd.7) 389 : cluster [DBG] 323.1s0 scrub starts 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:00.565092+0000 osd.7 (osd.7) 390 : cluster [DBG] 323.1 scrub ok 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:00.565092+0000 osd.7 (osd.7) 390 : cluster [DBG] 323.1 scrub ok 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.334134+0000 mon.a (mon.0) 3485 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.334134+0000 mon.a (mon.0) 3485 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.335369+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.335369+0000 mon.a (mon.0) 3486 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:01.338104+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: audit 2024-04-26T16:35:01.338104+0000 mon.a (mon.0) 3487 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-122"}]': finished 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.348597+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2024-04-26T16:35:01.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.348597+0000 mon.a (mon.0) 3488 : cluster [DBG] osdmap e723: 8 total, 7 up, 8 in 2024-04-26T16:35:01.873 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:01.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:01.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:02.003 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:02.032 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:02.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:02.135 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:02.163 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:02.266 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:02.295 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:02.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:02.421 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:02.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:02.548 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:02.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.600675+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.f scrub starts 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.600675+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.f scrub starts 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.604636+0000 osd.7 (osd.7) 392 : cluster [DBG] 3.f scrub ok 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.604636+0000 osd.7 (osd.7) 392 : cluster [DBG] 3.f scrub ok 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.745387+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.745387+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.749657+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.749657+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.363550+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2024-04-26T16:35:02.582 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.363550+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.600675+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.f scrub starts 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.600675+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.f scrub starts 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.604636+0000 osd.7 (osd.7) 392 : cluster [DBG] 3.f scrub ok 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.604636+0000 osd.7 (osd.7) 392 : cluster [DBG] 3.f scrub ok 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.745387+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.745387+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.749657+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.749657+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.363550+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2024-04-26T16:35:02.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:02 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.363550+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2024-04-26T16:35:02.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:02.681 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:02.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:02.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:02.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.600675+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.f scrub starts 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.600675+0000 osd.7 (osd.7) 391 : cluster [DBG] 3.f scrub starts 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.604636+0000 osd.7 (osd.7) 392 : cluster [DBG] 3.f scrub ok 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.604636+0000 osd.7 (osd.7) 392 : cluster [DBG] 3.f scrub ok 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.745387+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.745387+0000 osd.5 (osd.5) 265 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.749657+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.749657+0000 osd.5 (osd.5) 266 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.363550+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2024-04-26T16:35:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:02 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.363550+0000 mon.a (mon.0) 3489 : cluster [DBG] osdmap e724: 8 total, 7 up, 8 in 2024-04-26T16:35:02.915 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:02.943 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:02.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:03.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:03.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:03.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.205 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:03.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:03.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:03.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:03.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:03.597 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:03.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:03.729 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:03.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.877304+0000 mgr.y (mgr.24401) 680 : cluster [DBG] pgmap v1124: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:03.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:01.877304+0000 mgr.y (mgr.24401) 680 : cluster [DBG] pgmap v1124: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:03.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.399811+0000 osd.3 (osd.3) 413 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:35:03.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.399811+0000 osd.3 (osd.3) 413 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:35:03.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.401363+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.401363+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.680881+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.680881+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.683171+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.683171+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.775659+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.775659+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.777669+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.777669+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.882872+0000 osd.2 (osd.2) 349 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.882872+0000 osd.2 (osd.2) 349 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.883276+0000 osd.2 (osd.2) 350 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:03.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:02.883276+0000 osd.2 (osd.2) 350 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:03.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.363217+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2024-04-26T16:35:03.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.363217+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2024-04-26T16:35:03.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: audit 2024-04-26T16:35:03.365973+0000 mon.c (mon.2) 759 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: audit 2024-04-26T16:35:03.365973+0000 mon.c (mon.2) 759 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: audit 2024-04-26T16:35:03.369455+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:03 smithi190 bash[23868]: audit 2024-04-26T16:35:03.369455+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.877304+0000 mgr.y (mgr.24401) 680 : cluster [DBG] pgmap v1124: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:01.877304+0000 mgr.y (mgr.24401) 680 : cluster [DBG] pgmap v1124: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.399811+0000 osd.3 (osd.3) 413 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.399811+0000 osd.3 (osd.3) 413 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.401363+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.401363+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:35:03.844 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.680881+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.680881+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.683171+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.683171+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.775659+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.775659+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.777669+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.777669+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.882872+0000 osd.2 (osd.2) 349 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.882872+0000 osd.2 (osd.2) 349 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.883276+0000 osd.2 (osd.2) 350 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:02.883276+0000 osd.2 (osd.2) 350 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.363217+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.363217+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2024-04-26T16:35:03.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: audit 2024-04-26T16:35:03.365973+0000 mon.c (mon.2) 759 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: audit 2024-04-26T16:35:03.365973+0000 mon.c (mon.2) 759 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: audit 2024-04-26T16:35:03.369455+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[29234]: audit 2024-04-26T16:35:03.369455+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.877304+0000 mgr.y (mgr.24401) 680 : cluster [DBG] pgmap v1124: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:01.877304+0000 mgr.y (mgr.24401) 680 : cluster [DBG] pgmap v1124: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 3 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.399811+0000 osd.3 (osd.3) 413 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:35:03.846 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.399811+0000 osd.3 (osd.3) 413 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.401363+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.401363+0000 osd.3 (osd.3) 414 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.680881+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.680881+0000 osd.1 (osd.1) 381 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.683171+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.683171+0000 osd.1 (osd.1) 382 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.775659+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.775659+0000 osd.5 (osd.5) 267 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.777669+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.777669+0000 osd.5 (osd.5) 268 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.882872+0000 osd.2 (osd.2) 349 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.882872+0000 osd.2 (osd.2) 349 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:03.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.883276+0000 osd.2 (osd.2) 350 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:02.883276+0000 osd.2 (osd.2) 350 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.363217+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.363217+0000 mon.a (mon.0) 3490 : cluster [DBG] osdmap e725: 8 total, 7 up, 8 in 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: audit 2024-04-26T16:35:03.365973+0000 mon.c (mon.2) 759 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: audit 2024-04-26T16:35:03.365973+0000 mon.c (mon.2) 759 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: audit 2024-04-26T16:35:03.369455+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:03 smithi050 bash[21463]: audit 2024-04-26T16:35:03.369455+0000 mon.a (mon.0) 3491 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:03.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:03.959 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.990 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:03.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:04.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:04.121 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:04.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:04.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:04.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:04.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:04.525 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:04.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:04.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:35:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:35:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.016282+0000 osd.0 (osd.0) 403 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:35:04.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.016282+0000 osd.0 (osd.0) 403 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.018927+0000 osd.0 (osd.0) 404 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.018927+0000 osd.0 (osd.0) 404 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.731393+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.731393+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.731548+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.731548+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.865380+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.865380+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.866822+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.866822+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: audit 2024-04-26T16:35:04.362708+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:04.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: audit 2024-04-26T16:35:04.362708+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.369683+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.369683+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.016282+0000 osd.0 (osd.0) 403 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.016282+0000 osd.0 (osd.0) 403 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.018927+0000 osd.0 (osd.0) 404 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.018927+0000 osd.0 (osd.0) 404 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.731393+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.731393+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.731548+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.731548+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.865380+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-04-26T16:35:04.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.865380+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-04-26T16:35:04.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.866822+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-04-26T16:35:04.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.866822+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-04-26T16:35:04.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: audit 2024-04-26T16:35:04.362708+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:04.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: audit 2024-04-26T16:35:04.362708+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:04.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.369683+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2024-04-26T16:35:04.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:04 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.369683+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2024-04-26T16:35:04.659 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:04.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:04.790 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.016282+0000 osd.0 (osd.0) 403 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.016282+0000 osd.0 (osd.0) 403 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.018927+0000 osd.0 (osd.0) 404 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.018927+0000 osd.0 (osd.0) 404 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.731393+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.731393+0000 osd.0 (osd.0) 405 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.731548+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.731548+0000 osd.0 (osd.0) 406 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.865380+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.865380+0000 osd.2 (osd.2) 351 : cluster [DBG] 2.c scrub starts 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.866822+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.866822+0000 osd.2 (osd.2) 352 : cluster [DBG] 2.c scrub ok 2024-04-26T16:35:04.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: audit 2024-04-26T16:35:04.362708+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:04.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: audit 2024-04-26T16:35:04.362708+0000 mon.a (mon.0) 3492 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:04.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.369683+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2024-04-26T16:35:04.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:04 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.369683+0000 mon.a (mon.0) 3493 : cluster [DBG] osdmap e726: 8 total, 7 up, 8 in 2024-04-26T16:35:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:04.894 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:04.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:04.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:05.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:05.054 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:05.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:05.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:05.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:05.315 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:05.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:05.444 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:05.548 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.878402+0000 mgr.y (mgr.24401) 681 : cluster [DBG] pgmap v1127: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:03.878402+0000 mgr.y (mgr.24401) 681 : cluster [DBG] pgmap v1127: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.418761+0000 osd.4 (osd.4) 363 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.418761+0000 osd.4 (osd.4) 363 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.421108+0000 osd.4 (osd.4) 364 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.421108+0000 osd.4 (osd.4) 364 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:05.373414+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: cluster 2024-04-26T16:35:05.373414+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: audit 2024-04-26T16:35:05.399944+0000 mon.c (mon.2) 760 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: audit 2024-04-26T16:35:05.399944+0000 mon.c (mon.2) 760 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: audit 2024-04-26T16:35:05.401075+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:05 smithi190 bash[23868]: audit 2024-04-26T16:35:05.401075+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.576 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:05.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:05.706 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:05.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.878402+0000 mgr.y (mgr.24401) 681 : cluster [DBG] pgmap v1127: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:05.799 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:03.878402+0000 mgr.y (mgr.24401) 681 : cluster [DBG] pgmap v1127: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.418761+0000 osd.4 (osd.4) 363 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.418761+0000 osd.4 (osd.4) 363 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.421108+0000 osd.4 (osd.4) 364 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.421108+0000 osd.4 (osd.4) 364 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:05.373414+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: cluster 2024-04-26T16:35:05.373414+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: audit 2024-04-26T16:35:05.399944+0000 mon.c (mon.2) 760 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: audit 2024-04-26T16:35:05.399944+0000 mon.c (mon.2) 760 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: audit 2024-04-26T16:35:05.401075+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[29234]: audit 2024-04-26T16:35:05.401075+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.878402+0000 mgr.y (mgr.24401) 681 : cluster [DBG] pgmap v1127: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:05.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:03.878402+0000 mgr.y (mgr.24401) 681 : cluster [DBG] pgmap v1127: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 80/633 objects degraded (12.638%) 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.418761+0000 osd.4 (osd.4) 363 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.418761+0000 osd.4 (osd.4) 363 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.421108+0000 osd.4 (osd.4) 364 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.421108+0000 osd.4 (osd.4) 364 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:05.373414+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: cluster 2024-04-26T16:35:05.373414+0000 mon.a (mon.0) 3494 : cluster [DBG] osdmap e727: 8 total, 7 up, 8 in 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: audit 2024-04-26T16:35:05.399944+0000 mon.c (mon.2) 760 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: audit 2024-04-26T16:35:05.399944+0000 mon.c (mon.2) 760 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: audit 2024-04-26T16:35:05.401075+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.801 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:05 smithi050 bash[21463]: audit 2024-04-26T16:35:05.401075+0000 mon.a (mon.0) 3495 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:05.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:05.837 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:05.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/190980378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec1083d0 msgr2=0x7f8fec1087b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:05.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/190980378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec1083d0 0x7f8fec1087b0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0099c0 tx=0x7f8fdc009cd0 comp rx=0 tx=0).stop 2024-04-26T16:35:05.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/190980378 shutdown_connections 2024-04-26T16:35:05.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/190980378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fec102d10 0x7f8fec10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:05.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/190980378 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec1083d0 0x7f8fec1087b0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:05.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/190980378 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec102350 0x7f8fec1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:05.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/190980378 >> 172.21.15.50:0/190980378 conn(0x7f8fec078390 msgr2=0x7f8fec0787a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:05.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/190980378 shutdown_connections 2024-04-26T16:35:05.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/190980378 wait complete. 2024-04-26T16:35:05.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 librados: starting msgr at 2024-04-26T16:35:05.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 librados: starting objecter 2024-04-26T16:35:05.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 Processor -- start 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- start start 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 librados: setting wanted keys 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 librados: calling monclient init 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fec102350 0x7f8fec1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec102d10 0x7f8fec199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 0x7f8fec19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec111980 con 0x7f8fec1083d0 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec111800 con 0x7f8fec102350 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8fec111b00 con 0x7f8fec102d10 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec102d10 0x7f8fec199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff087d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 0x7f8fec19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff087d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 0x7f8fec19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40062/0 (socket says 172.21.15.50:40062) 2024-04-26T16:35:05.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff087d700 1 -- 172.21.15.50:0/1016070570 learned_addr learned my addr 172.21.15.50:0/1016070570 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8febfff700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fec102350 0x7f8fec1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 -- 172.21.15.50:0/1016070570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fec102350 msgr2=0x7f8fec1990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fec102350 0x7f8fec1990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 -- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 msgr2=0x7f8fec19d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 0x7f8fec19d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 -- 172.21.15.50:0/1016070570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fec19e100 con 0x7f8fec102d10 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff087d700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 0x7f8fec19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8feb7fe700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec102d10 0x7f8fec199630 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000e9f0 tx=0x7f8fe000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8fe97fa700 10 client.?.objecter ms_handle_connect 0x7f8fec102d10 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8fe97fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe000c9c0 con 0x7f8fec102d10 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8fe0010430 con 0x7f8fec102d10 2024-04-26T16:35:05.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fec19e3f0 con 0x7f8fec102d10 2024-04-26T16:35:05.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0018680 con 0x7f8fec102d10 2024-04-26T16:35:05.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 10 client.41573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:05.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fec1a5c70 con 0x7f8fec102d10 2024-04-26T16:35:05.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.932+0000 7f8ff22e0700 1 librados: init done 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fec103a40 con 0x7f8fec102d10 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8fe000cb60 con 0x7f8fec102d10 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8fd4071f80 0x7f8fd4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(727..727 src has 577..727) ==== 9892+0+0 (secure 0 0 0) 0x7f8fe0091ce0 con 0x7f8fec102d10 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 10 client.41573.objecter ms_dispatch 0x7f8fec001550 osd_map(727..727 src has 577..727) 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 3 client.41573.objecter handle_osd_map got epochs [727,727] > 0 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 3 client.41573.objecter handle_osd_map decoding full epoch 727 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8febfff700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8fd4071f80 0x7f8fd4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 20 client.41573.objecter dump_active .. 0 homeless 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8febfff700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8fd4071f80 0x7f8fd4074440 secure :-1 s=READY pgs=3820 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009990 tx=0x7f8fdc009e60 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:05.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.936+0000 7f8fe97fa700 10 client.41573.objecter ms_handle_connect 0x7f8fd4071f80 2024-04-26T16:35:05.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:05.940+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f8fe005d510 con 0x7f8fec102d10 2024-04-26T16:35:05.940 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:05.959 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:05.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:06.061 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:06.083 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:06.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:06.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:06.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.200+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f8fec19a450 con 0x7f8fec102d10 2024-04-26T16:35:06.214 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 -- 172.21.15.50:0/455357554 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006b130 msgr2=0x7f3f4006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/455357554 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006b130 0x7f3f4006b590 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009990 tx=0x7f3f34009ca0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 -- 172.21.15.50:0/455357554 shutdown_connections 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/455357554 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4006bad0 0x7f3f401102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/455357554 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 0x7f3f40109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/455357554 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006b130 0x7f3f4006b590 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 -- 172.21.15.50:0/455357554 >> 172.21.15.50:0/455357554 conn(0x7f3f400792d0 msgr2=0x7f3f400796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:06.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 -- 172.21.15.50:0/455357554 shutdown_connections 2024-04-26T16:35:06.222 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.220+0000 7f3f45d6a700 1 -- 172.21.15.50:0/455357554 wait complete. 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 librados: starting msgr at 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 librados: starting objecter 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 Processor -- start 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- start start 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 librados: setting wanted keys 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 librados: calling monclient init 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4006b130 0x7f3f401023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.224 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006bad0 0x7f3f40102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 0x7f3f40102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3f40112ae0 con 0x7f3f401094a0 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3f40112960 con 0x7f3f4006b130 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3f40112c60 con 0x7f3f4006bad0 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 0x7f3f40102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006bad0 0x7f3f40102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006bad0 0x7f3f40102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45932/0 (socket says 172.21.15.50:45932) 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3f7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4006b130 0x7f3f401023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 -- 172.21.15.50:0/1033461597 learned_addr learned my addr 172.21.15.50:0/1033461597 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3ffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 0x7f3f40102e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40090/0 (socket says 172.21.15.50:40090) 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 -- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4006b130 msgr2=0x7f3f401023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3f4006b130 0x7f3f401023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 -- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 msgr2=0x7f3f40102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 0x7f3f40102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f401a76f0 con 0x7f3f4006bad0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3f401094a0 0x7f3f40102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3f4006bad0 0x7f3f40102930 secure :-1 s=READY pgs=2731 cs=0 l=1 rev1=1 crypto rx=0x7f3f34000c00 tx=0x7f3f34004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3cff9700 10 client.?.objecter ms_handle_connect 0x7f3f4006bad0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3cff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34027070 con 0x7f3f4006bad0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3f34022470 con 0x7f3f4006bad0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f401a7980 con 0x7f3f4006bad0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 10 client.41579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34017650 con 0x7f3f4006bad0 2024-04-26T16:35:06.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f401a7e30 con 0x7f3f4006bad0 2024-04-26T16:35:06.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 librados: init done 2024-04-26T16:35:06.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.224+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f3f4006a110 con 0x7f3f4006bad0 2024-04-26T16:35:06.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3f3402a070 con 0x7f3f4006bad0 2024-04-26T16:35:06.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3f28071f80 0x7f3f28074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(727..727 src has 577..727) ==== 9892+0+0 (secure 0 0 0) 0x7f3f34092b80 con 0x7f3f4006bad0 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 osd_map(727..727 src has 577..727) 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 3 client.41579.objecter handle_osd_map got epochs [727,727] > 0 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 3 client.41579.objecter handle_osd_map decoding full epoch 727 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3f7fe700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3f28071f80 0x7f3f28074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 20 client.41579.objecter dump_active .. 0 homeless 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v727) ==== 54+0+16 (secure 0 0 0) 0x7f3f34013070 con 0x7f3f4006bad0 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3f7fe700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3f28071f80 0x7f3f28074440 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009680 tx=0x7f3f30009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f28071f80 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 1 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f3f40100c60 0x7f3f4002d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 1 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 0 1 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40100af0 osd=0 4 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 10 client.41579.objecter _send_command 1 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f40064540 con 0x7f3f40100c60 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f3f40100c60 0x7f3f4002d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f3f40100c60 0x7f3f4002d310 crc :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.230 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f40100c60 2024-04-26T16:35:06.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f40064540 con 0x7f3f40100c60 2024-04-26T16:35:06.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 1: 0 ) 2024-04-26T16:35:06.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 1 = system:0 2024-04-26T16:35:06.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f40100af0 osd=0 4 2024-04-26T16:35:06.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.228+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 0 1 2024-04-26T16:35:06.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 -- 172.21.15.50:0/3715760825 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964101ae0 msgr2=0x7fd964101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 --2- 172.21.15.50:0/3715760825 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964101ae0 0x7fd964101f60 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7fd95400ba80 tx=0x7fd95400bd90 comp rx=0 tx=0).stop 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 -- 172.21.15.50:0/3715760825 shutdown_connections 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 --2- 172.21.15.50:0/3715760825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd964107a90 0x7fd964107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 --2- 172.21.15.50:0/3715760825 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd9641024a0 0x7fd96410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 --2- 172.21.15.50:0/3715760825 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964101ae0 0x7fd964101f60 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 -- 172.21.15.50:0/3715760825 >> 172.21.15.50:0/3715760825 conn(0x7fd964078990 msgr2=0x7fd9640ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 -- 172.21.15.50:0/3715760825 shutdown_connections 2024-04-26T16:35:06.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 -- 172.21.15.50:0/3715760825 wait complete. 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 librados: starting msgr at 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.232+0000 7fd96c57d700 1 librados: starting objecter 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 Processor -- start 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- start start 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 librados: setting wanted keys 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 librados: calling monclient init 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd964101ae0 0x7fd964196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 0x7fd9641974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 0x7fd96419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641109c0 con 0x7fd964101ae0 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd964110840 con 0x7fd9641024a0 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd964110b40 con 0x7fd964107a90 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96a319700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd964101ae0 0x7fd964196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96ab1a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 0x7fd96419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 0x7fd9641974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 0x7fd9641974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.50:39056/0 (socket says 172.21.15.50:39056) 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 -- 172.21.15.50:0/229762528 learned_addr learned my addr 172.21.15.50:0/229762528 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96ab1a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 0x7fd96419b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45964/0 (socket says 172.21.15.50:45964) 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 -- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 msgr2=0x7fd96419b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 0x7fd96419b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 -- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd964101ae0 msgr2=0x7fd964196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd964101ae0 0x7fd964196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 -- 172.21.15.50:0/229762528 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96419bf80 con 0x7fd9641024a0 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96ab1a700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 0x7fd96419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96a319700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd964101ae0 0x7fd964196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd969b18700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 0x7fd9641974b0 secure :-1 s=READY pgs=4144 cs=0 l=1 rev1=1 crypto rx=0x7fd95400bfd0 tx=0x7fd95400b6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd95b7fe700 10 client.?.objecter ms_handle_connect 0x7fd9641024a0 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd95b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd954023070 con 0x7fd9641024a0 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96419c210 con 0x7fd9641024a0 2024-04-26T16:35:06.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd9540054a0 con 0x7fd9641024a0 2024-04-26T16:35:06.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9540049a0 con 0x7fd9641024a0 2024-04-26T16:35:06.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 10 client.43858.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:06.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9641a3af0 con 0x7fd9641024a0 2024-04-26T16:35:06.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 librados: init done 2024-04-26T16:35:06.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.236+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9641031d0 con 0x7fd9641024a0 2024-04-26T16:35:06.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd954026070 con 0x7fd9641024a0 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd950071f80 0x7fd950074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(727..727 src has 577..727) ==== 9892+0+0 (secure 0 0 0) 0x7fd954011020 con 0x7fd9641024a0 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 10 client.43858.objecter ms_dispatch 0x7fd964001550 osd_map(727..727 src has 577..727) 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 3 client.43858.objecter handle_osd_map got epochs [727,727] > 0 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 3 client.43858.objecter handle_osd_map decoding full epoch 727 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 20 client.43858.objecter dump_active .. 0 homeless 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd96a319700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd950071f80 0x7fd950074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd95b7fe700 10 client.43858.objecter ms_handle_connect 0x7fd950071f80 2024-04-26T16:35:06.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.240+0000 7fd96a319700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd950071f80 0x7fd950074440 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7fd960009fd0 tx=0x7fd960009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.244+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd95402d020 con 0x7fd9641024a0 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 2 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 2 2024-04-26T16:35:06.248 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f40100af0 osd=0 3 2024-04-26T16:35:06.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 0 2 2024-04-26T16:35:06.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40100af0 osd=0 4 2024-04-26T16:35:06.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 10 client.41579.objecter _send_command 2 2024-04-26T16:35:06.249 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.248+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f4002d850 con 0x7f3f40100c60 2024-04-26T16:35:06.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f4002d850 con 0x7f3f40100c60 2024-04-26T16:35:06.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f40100af0 osd=0 4 2024-04-26T16:35:06.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 0 2 2024-04-26T16:35:06.251 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:35:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 3 2024-04-26T16:35:06.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f3f40063c20 0x7f3f401ac080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 3 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 1 3 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40063ab0 osd=1 4 2024-04-26T16:35:06.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 10 client.41579.objecter _send_command 3 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f3f4002e740 con 0x7f3f40063c20 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3f7fe700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f3f40063c20 0x7f3f401ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f40063c20 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3f7fe700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f3f40063c20 0x7f3f401ac080 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.252+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f4002e740 con 0x7f3f40063c20 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.256+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 3: 0 ) 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.256+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 3 = system:0 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.256+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f40063ab0 osd=1 4 2024-04-26T16:35:06.255 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.256+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 1 3 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 4 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 4 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f40063ab0 osd=1 3 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 1 4 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f40063ab0 osd=1 4 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 10 client.41579.objecter _send_command 4 2024-04-26T16:35:06.271 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.272+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f4002dcb0 con 0x7f3f40063c20 2024-04-26T16:35:06.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f4002dcb0 con 0x7f3f40063c20 2024-04-26T16:35:06.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f40063ab0 osd=1 4 2024-04-26T16:35:06.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 1 4 2024-04-26T16:35:06.274 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 5 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f3f4010af70 0x7f3f401af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 5 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 2 5 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f4002e0b0 osd=2 4 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 10 client.41579.objecter _send_command 5 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f3f401b25b0 con 0x7f3f4010af70 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f3f4010af70 0x7f3f401af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.275 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f4010af70 2024-04-26T16:35:06.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f3f4010af70 0x7f3f401af750 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f401b25b0 con 0x7f3f4010af70 2024-04-26T16:35:06.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 5: 0 ) 2024-04-26T16:35:06.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 5 = system:0 2024-04-26T16:35:06.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f4002e0b0 osd=2 4 2024-04-26T16:35:06.276 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.276+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 2 5 2024-04-26T16:35:06.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 6 2024-04-26T16:35:06.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 6 2024-04-26T16:35:06.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f4002e0b0 osd=2 3 2024-04-26T16:35:06.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 2 6 2024-04-26T16:35:06.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f4002e0b0 osd=2 4 2024-04-26T16:35:06.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 10 client.41579.objecter _send_command 6 2024-04-26T16:35:06.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.292+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f4010a4e0 con 0x7f3f4010af70 2024-04-26T16:35:06.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f4010a4e0 con 0x7f3f4010af70 2024-04-26T16:35:06.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f4002e0b0 osd=2 4 2024-04-26T16:35:06.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 2 6 2024-04-26T16:35:06.295 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:35:06.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 7 2024-04-26T16:35:06.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f3f401b0e10 0x7f3f401b1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 7 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 3 7 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401acd90 osd=3 4 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 10 client.41579.objecter _send_command 7 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f3f401ad4b0 con 0x7f3f401b0e10 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f3f401b0e10 0x7f3f401b1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f401b0e10 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f3f401b0e10 0x7f3f401b1250 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f401ad4b0 con 0x7f3f401b0e10 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 7: 0 ) 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 7 = system:0 2024-04-26T16:35:06.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401acd90 osd=3 4 2024-04-26T16:35:06.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.296+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 3 7 2024-04-26T16:35:06.302 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:35:06.302 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-26T16:35:06.302 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-26T16:35:06.302 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-26T16:35:06.302 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.302 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:2168 247.f 247:f214ce95:test-rados-api-smithi050-73372-28::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e286)", 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:25:54.102745+0000", 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 552.20250903199997, 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.058487704000000001, 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2168 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.303 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102745+0000", 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102745+0000", 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102748+0000", 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0700000000000001e-06 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.304 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102756+0000", 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.3920000000000003e-06 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102776+0000", 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9873000000000001e-05 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102789+0000", 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3606e-05 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.305 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.102876+0000", 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6156999999999995e-05 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for blocked object", 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.103278+0000", 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040282099999999998 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.306 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.158416+0000", 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.055137799000000001 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.158546+0000", 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000129629 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.161019+0000", 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024728549999999999 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.307 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.161098+0000", 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.9550999999999998e-05 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:54.161233+0000", 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013495099999999999 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.308 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.30300.0:1 251.6s0 251:68f45781:::oid_SimpleWritePP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e290)", 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:25:58.129330+0000", 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 548.17592424999998, 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0223501859999999, 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.30300", 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3027781296", 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.309 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129330+0000", 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129330+0000", 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129332+0000", 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9360000000000002e-06 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.310 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129340+0000", 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.875e-06 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129355+0000", 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4517e-05 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129367+0000", 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2619000000000001e-05 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.311 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129482+0000", 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011439500000000001 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:58.129488+0000", 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0979999999999999e-06 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.139663+0000", 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0101752289999999 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.312 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.139805+0000", 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014170799999999999 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.142262+0000", 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0024573720000000002 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.143172+0000", 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00090992199999999997 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.313 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.151585+0000", 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0084128109999999992 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:25:59.151680+0000", 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.5704000000000004e-05 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.314 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.30450.0:1 276.5s0 276:ab97ba35:::nonexistent:head [read 0~128] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e356)", 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:27:04.140937+0000", 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 482.16431743499999, 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0109418809999999, 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.30450", 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/4229062425", 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.315 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140937+0000", 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140937+0000", 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140939+0000", 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8759999999999999e-06 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.316 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140946+0000", 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8950000000000001e-06 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140958+0000", 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2193e-05 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.317 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.140968+0000", 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.9820000000000003e-06 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.141021+0000", 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.2923999999999997e-05 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:04.141025+0000", 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6210000000000001e-06 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.318 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:05.151766+0000", 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0107411369999999 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:05.151879+0000", 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011325299999999999 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.319 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14391.0:3462 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94191276271872 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e695)", 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:24.867470+0000", 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 41.437784499000003, 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0010891, 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14391", 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3462 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.320 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867470+0000", 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867470+0000", 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867473+0000", 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.4659999999999999e-06 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.321 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867509+0000", 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.5308999999999997e-05 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867565+0000", 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6441999999999999e-05 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867596+0000", 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.0922e-05 2024-04-26T16:35:06.322 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.867934+0000", 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033772499999999998 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.868200+0000", 2024-04-26T16:35:06.323 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00026651799999999998 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:24.868559+0000", 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000358718 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.24422.0:35 4.2 4:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e653)", 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:25.572406+0000", 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 40.732848070999999, 2024-04-26T16:35:06.324 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00072636, 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.24422", 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/39578498", 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 35 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572406+0000", 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.325 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572406+0000", 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572409+0000", 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9919999999999999e-06 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572426+0000", 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6952999999999998e-05 2024-04-26T16:35:06.326 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572458+0000", 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2001000000000001e-05 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572477+0000", 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8669999999999999e-05 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.572926+0000", 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00044877399999999998 2024-04-26T16:35:06.327 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:25.573133+0000", 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00020697000000000001 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:747 326.1e 326:78000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:28.918738+0000", 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.386516774, 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00059570599999999999, 2024-04-26T16:35:06.328 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 747 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.329 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918738+0000", 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918738+0000", 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918740+0000", 2024-04-26T16:35:06.330 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.046e-06 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918752+0000", 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2415999999999999e-05 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918782+0000", 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9382000000000001e-05 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.331 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918799+0000", 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7073999999999999e-05 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.918972+0000", 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000172967 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.919117+0000", 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000145468 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.332 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:28.919333+0000", 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021635299999999999 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:852 326.15 326:a8000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:32.013238+0000", 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 34.29201655, 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00056206699999999997, 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.333 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 852 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013238+0000", 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.334 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013238+0000", 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013240+0000", 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2129999999999998e-06 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013253+0000", 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3069e-05 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.335 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013284+0000", 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1192999999999998e-05 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013301+0000", 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6849000000000001e-05 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013470+0000", 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016894100000000001 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.336 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013581+0000", 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000110717 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:32.013800+0000", 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00021908499999999999 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.337 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:894 326.f 326:f0000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:33.054024+0000", 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 33.251230519000003, 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000582988, 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 894 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.338 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054024+0000", 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054024+0000", 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.339 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054026+0000", 2024-04-26T16:35:06.341 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.86e-06 2024-04-26T16:35:06.341 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.341 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.341 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054038+0000", 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.203e-05 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054065+0000", 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6891999999999999e-05 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054080+0000", 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5019e-05 2024-04-26T16:35:06.342 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054241+0000", 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000161452 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054356+0000", 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000114449 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:33.054607+0000", 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00025128599999999998 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.343 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:925 326.7 326:e0000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:34.084007+0000", 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.221247691000002, 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00062753899999999998, 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.344 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 925 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084007+0000", 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084007+0000", 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.345 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084010+0000", 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8370000000000001e-06 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084025+0000", 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5586e-05 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084070+0000", 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4694999999999998e-05 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.346 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084103+0000", 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2985999999999999e-05 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084273+0000", 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000170457 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084394+0000", 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000120763 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.347 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.084634+0000", 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000240215 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:927 326.f 326:f0000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:34.086137+0000", 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.219117163999996, 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00055814099999999995, 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.348 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 927 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086137+0000", 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086137+0000", 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.349 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086139+0000", 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0930000000000001e-06 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086152+0000", 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2043e-05 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.350 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086183+0000", 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1142000000000001e-05 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086200+0000", 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6946e-05 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086349+0000", 2024-04-26T16:35:06.351 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000149259 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086452+0000", 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102933 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:34.086696+0000", 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000243725 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.352 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:965 323.0s0 323:11eac614:test-rados-api-smithi050-73372-119::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e700)", 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:38.652257+0000", 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.652997219, 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0083457259999999995, 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 965 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.353 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652257+0000", 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652257+0000", 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652260+0000", 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.266e-06 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.354 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652271+0000", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1324e-05 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652290+0000", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9239e-05 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652305+0000", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4956999999999999e-05 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652484+0000", 2024-04-26T16:35:06.355 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00017848300000000001 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.652754+0000", 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000269924 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.653826+0000", 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001072915 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.655037+0000", 2024-04-26T16:35:06.356 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00121065 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660466+0000", 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0054292790000000004 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660603+0000", 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013668900000000001 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(323.0s0 700/666 ECSubWriteReply(tid=1095, last_complete=700'1, committed=1, applied=1))", 2024-04-26T16:35:06.357 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:38.660104+0000", 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 27.645150678, 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00071019300000000002, 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660104+0000", 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660104+0000", 2024-04-26T16:35:06.358 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660104+0000", 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6399999999999998e-07 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660108+0000", 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.794e-06 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660115+0000", 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8800000000000002e-06 2024-04-26T16:35:06.359 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660126+0000", 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1044e-05 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660275+0000", 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000148913 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:38.660814+0000", 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00053929800000000001 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.360 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:967 323.0s0 323:11eac614:test-rados-api-smithi050-73372-119::foo:head [writefull 0~5] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e701)", 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:39.621458+0000", 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.683796303000001, 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.010835121, 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 967 2024-04-26T16:35:06.361 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621458+0000", 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621458+0000", 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621461+0000", 2024-04-26T16:35:06.362 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.3020000000000002e-06 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621475+0000", 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3984e-05 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621551+0000", 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6327000000000003e-05 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621566+0000", 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5102e-05 2024-04-26T16:35:06.363 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.621938+0000", 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000371877 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.622060+0000", 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012177599999999999 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.622561+0000", 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00050124900000000005 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.364 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.623219+0000", 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00065857700000000003 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.632147+0000", 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0089278229999999997 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.632293+0000", 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014610400000000001 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.365 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDECSubOpWriteReply(323.0s0 701/666 ECSubWriteReply(tid=1097, last_complete=701'3, committed=1, applied=1))", 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:39.631943+0000", 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.673311955999999, 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00060448299999999995, 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.631943+0000", 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.366 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.631943+0000", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.631945+0000", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9269999999999999e-06 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.631956+0000", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1484000000000001e-05 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.367 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.631979+0000", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.298e-05 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.631999+0000", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0114000000000001e-05 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.632080+0000", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1236999999999995e-05 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.632547+0000", 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046674100000000001 2024-04-26T16:35:06.368 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14391.0:3492 3.4 3:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94191276271872 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e698)", 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:34:39.879629+0000", 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 26.425625324999999, 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082641300000000002, 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14391", 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3492 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.369 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879629+0000", 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879629+0000", 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879632+0000", 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2339999999999999e-06 2024-04-26T16:35:06.370 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879647+0000", 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4919000000000001e-05 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879690+0000", 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.2166000000000003e-05 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879709+0000", 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9751000000000001e-05 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.371 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.879954+0000", 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00024447999999999999 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.880123+0000", 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016900500000000001 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:34:39.880456+0000", 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000332858 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.372 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.373 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:06.373 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:06.373 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:35:06.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 8 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 8 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401acd90 osd=3 3 2024-04-26T16:35:06.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 3 8 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401acd90 osd=3 4 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 10 client.41579.objecter _send_command 8 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.324+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f401ac810 con 0x7f3f401b0e10 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f401ac810 con 0x7f3f401b0e10 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401acd90 osd=3 4 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 3 8 2024-04-26T16:35:06.375 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 9 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f3f401ac980 0x7f3f401b6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 9 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 4 9 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac810 osd=4 4 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 10 client.41579.objecter _send_command 9 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f3f401b95e0 con 0x7f3f401ac980 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3f7fe700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f3f401ac980 0x7f3f401b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f401ac980 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.328+0000 7f3f3f7fe700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f3f401ac980 0x7f3f401b6780 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.332+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f401b95e0 con 0x7f3f401ac980 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.332+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 9: 0 ) 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.332+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 9 = system:0 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.332+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401ac810 osd=4 4 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.332+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 4 9 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 10 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 10 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401ac810 osd=4 3 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 4 10 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac810 osd=4 4 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 10 client.41579.objecter _send_command 10 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.352+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f401ac620 con 0x7f3f401ac980 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f401ac620 con 0x7f3f401ac980 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401ac810 osd=4 4 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 4 10 2024-04-26T16:35:06.378 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:35:06.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 11 2024-04-26T16:35:06.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f3f401b0460 0x7f3f401b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.379 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 11 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 5 11 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac620 osd=5 4 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 10 client.41579.objecter _send_command 11 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f3f401bc980 con 0x7f3f401b0460 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.356+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f3f401b0460 0x7f3f401b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f401b0460 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3effd700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f3f401b0460 0x7f3f401b08c0 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f401bc980 con 0x7f3f401b0460 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 11: 0 ) 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 11 = system:0 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401ac620 osd=5 4 2024-04-26T16:35:06.380 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.360+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 5 11 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8fe97fa700 1 -- 172.21.15.50:0/1016070570 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v728) ==== 101+0+0 (secure 0 0 0) 0x7f8fe00611c0 con 0x7f8fec102d10 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 10 librados: watch_flush enter 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 10 librados: watch_flush exit 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 20 client.41573.objecter shutdown clearing up homeless session... 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 10 client.41573.objecter successfully canceled tick 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8fd4071f80 msgr2=0x7f8fd4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8fd4071f80 0x7f8fd4074440 secure :-1 s=READY pgs=3820 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc009990 tx=0x7f8fdc009e60 comp rx=0 tx=0).stop 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec102d10 msgr2=0x7f8fec199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec102d10 0x7f8fec199630 secure :-1 s=READY pgs=2729 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000e9f0 tx=0x7f8fe000ed00 comp rx=0 tx=0).stop 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 shutdown_connections 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8fd4071f80 0x7f8fd4074440 unknown :-1 s=CLOSED pgs=3820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8fec102350 0x7f8fec1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8fec1083d0 0x7f8fec19d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 --2- 172.21.15.50:0/1016070570 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8fec102d10 0x7f8fec199630 unknown :-1 s=CLOSED pgs=2729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 >> 172.21.15.50:0/1016070570 conn(0x7f8fec078390 msgr2=0x7f8fec1011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 shutdown_connections 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 -- 172.21.15.50:0/1016070570 wait complete. 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.372+0000 7f8ff22e0700 1 librados: shutdown 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 12 2024-04-26T16:35:06.383 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 12 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401ac620 osd=5 3 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 5 12 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401ac620 osd=5 4 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 10 client.41579.objecter _send_command 12 2024-04-26T16:35:06.384 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.384+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f401bb800 con 0x7f3f401b0460 2024-04-26T16:35:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f401bb800 con 0x7f3f401b0460 2024-04-26T16:35:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401ac620 osd=5 4 2024-04-26T16:35:06.387 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 5 12 2024-04-26T16:35:06.388 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:35:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.388 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 13 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 20 client.41579.objecter _get_session osd=-1 returning homeless 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 10 client.41579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=728}) -- 0x7f3f401bba60 con 0x7f3f4006bad0 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f401bb800 con 0x7f3f4006bad0 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 7 ==== osd_map(728..728 src has 577..728) ==== 996+0+0 (secure 0 0 0) 0x7f3f34099070 con 0x7f3f4006bad0 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 osd_map(728..728 src has 577..728) 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 3 client.41579.objecter handle_osd_map got epochs [728,728] > 727 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 3 client.41579.objecter handle_osd_map decoding incremental epoch 728 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 10 client.41579.objecter checking command 13 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 10 client.41579.objecter _check_command_map_dne tid 13 current 728 map_dne_bound 0 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 20 client.41579.objecter dump_active .. 1 homeless 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== mon.2 v2:172.21.15.50:3301/0 8 ==== mon_get_version_reply(handle=1 version=728) ==== 24+0+0 (secure 0 0 0) 0x7f3f34022620 con 0x7f3f4006bad0 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3dffb700 10 client.41579.objecter _check_command_map_dne tid 13 current 728 map_dne_bound 728 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3dffb700 10 client.41579.objecter _finish_command 13 = osd:6 osd down 2024-04-26T16:35:06.389 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3dffb700 15 client.41579.objecter _session_command_op_remove -1 13 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 14 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 14 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f3f401b0050 0x7f3f401b7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 14 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 7 14 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401afee0 osd=7 4 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 10 client.41579.objecter _send_command 14 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 14: {"prefix": "get_command_descriptions"}) -- 0x7f3f401b7b60 con 0x7f3f401b0050 2024-04-26T16:35:06.390 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.388+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f3f401b0050 0x7f3f401b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3cff9700 10 client.41579.objecter ms_handle_connect 0x7f3f401b0050 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3ffff700 1 --2- 172.21.15.50:0/1033461597 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f3f401b0050 0x7f3f401b7490 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 14: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f401b7b60 con 0x7f3f401b0050 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 14: 0 ) 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 14 = system:0 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401afee0 osd=7 4 2024-04-26T16:35:06.391 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.392+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 7 14 2024-04-26T16:35:06.402 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 10 client.41579.objecter _submit_command 15 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign -1 15 2024-04-26T16:35:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 20 client.41579.objecter _get_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_remove -1 15 2024-04-26T16:35:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 20 client.41579.objecter get_session s=0x7f3f401afee0 osd=7 3 2024-04-26T16:35:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 15 client.41579.objecter _session_command_op_assign 7 15 2024-04-26T16:35:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 20 client.41579.objecter put_session s=0x7f3f401afee0 osd=7 4 2024-04-26T16:35:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 10 client.41579.objecter _send_command 15 2024-04-26T16:35:06.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.416+0000 7f3f45d6a700 1 -- 172.21.15.50:0/1033461597 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f3f401b7b20 con 0x7f3f401b0050 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.420+0000 7f3f3cff9700 1 -- 172.21.15.50:0/1033461597 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 15: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f3f401b7b20 con 0x7f3f401b0050 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.420+0000 7f3f3cff9700 10 client.41579.objecter ms_dispatch 0x7f3f40001550 command_reply(tid 15: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.420+0000 7f3f3cff9700 10 client.41579.objecter _finish_command 15 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.420+0000 7f3f3cff9700 20 client.41579.objecter put_session s=0x7f3f401afee0 osd=7 4 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.420+0000 7f3f3cff9700 15 client.41579.objecter _session_command_op_remove 7 15 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:35:06.419 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:35:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:35:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:35:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:35:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-04-26T16:35:06.420 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:35:06.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:06.516 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:06.516+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7fd9641982d0 con 0x7fd9641024a0 2024-04-26T16:35:06.534 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:06.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.954019+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:35:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.954019+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:35:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.956857+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:35:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:04.956857+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:35:06.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.206053+0000 mon.c (mon.2) 761 : audit [INF] from='client.? 172.21.15.50:0/1016070570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.206053+0000 mon.c (mon.2) 761 : audit [INF] from='client.? 172.21.15.50:0/1016070570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.206473+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.206473+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.231555+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? 172.21.15.50:0/1033461597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.231555+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? 172.21.15.50:0/1033461597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.337093+0000 mon.a (mon.0) 3497 : cluster [WRN] Health check update: Degraded data redundancy: 80/633 objects degraded (12.638%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.337093+0000 mon.a (mon.0) 3497 : cluster [WRN] Health check update: Degraded data redundancy: 80/633 objects degraded (12.638%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.337126+0000 mon.a (mon.0) 3498 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.337126+0000 mon.a (mon.0) 3498 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.365946+0000 mon.a (mon.0) 3499 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.365946+0000 mon.a (mon.0) 3499 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.367430+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.367430+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.367573+0000 mon.a (mon.0) 3501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.367573+0000 mon.a (mon.0) 3501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:06.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.373382+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.373382+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.374535+0000 mon.c (mon.2) 763 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.374535+0000 mon.c (mon.2) 763 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.375478+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.375478+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.522382+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.50:0/229762528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.522382+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.50:0/229762528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.523520+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[21463]: audit 2024-04-26T16:35:06.523520+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.954019+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.954019+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:35:06.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.956857+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:04.956857+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.206053+0000 mon.c (mon.2) 761 : audit [INF] from='client.? 172.21.15.50:0/1016070570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.206053+0000 mon.c (mon.2) 761 : audit [INF] from='client.? 172.21.15.50:0/1016070570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.206473+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.206473+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.231555+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? 172.21.15.50:0/1033461597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.231555+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? 172.21.15.50:0/1033461597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.337093+0000 mon.a (mon.0) 3497 : cluster [WRN] Health check update: Degraded data redundancy: 80/633 objects degraded (12.638%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.337093+0000 mon.a (mon.0) 3497 : cluster [WRN] Health check update: Degraded data redundancy: 80/633 objects degraded (12.638%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.337126+0000 mon.a (mon.0) 3498 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.337126+0000 mon.a (mon.0) 3498 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.365946+0000 mon.a (mon.0) 3499 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.365946+0000 mon.a (mon.0) 3499 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.367430+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.367430+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:06.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.367573+0000 mon.a (mon.0) 3501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.367573+0000 mon.a (mon.0) 3501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.373382+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.373382+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.374535+0000 mon.c (mon.2) 763 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.374535+0000 mon.c (mon.2) 763 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.375478+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.375478+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.522382+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.50:0/229762528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.522382+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.50:0/229762528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.523520+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:06 smithi050 bash[29234]: audit 2024-04-26T16:35:06.523520+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.638 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:06.663 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:06.766 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:06.794 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:06.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.954019+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:35:06.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.954019+0000 osd.0 (osd.0) 407 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:35:06.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.956857+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:04.956857+0000 osd.0 (osd.0) 408 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.206053+0000 mon.c (mon.2) 761 : audit [INF] from='client.? 172.21.15.50:0/1016070570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.206053+0000 mon.c (mon.2) 761 : audit [INF] from='client.? 172.21.15.50:0/1016070570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.206473+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.206473+0000 mon.a (mon.0) 3496 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.231555+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? 172.21.15.50:0/1033461597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.231555+0000 mon.c (mon.2) 762 : audit [DBG] from='client.? 172.21.15.50:0/1033461597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.337093+0000 mon.a (mon.0) 3497 : cluster [WRN] Health check update: Degraded data redundancy: 80/633 objects degraded (12.638%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.337093+0000 mon.a (mon.0) 3497 : cluster [WRN] Health check update: Degraded data redundancy: 80/633 objects degraded (12.638%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.337126+0000 mon.a (mon.0) 3498 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.337126+0000 mon.a (mon.0) 3498 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:06.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.365946+0000 mon.a (mon.0) 3499 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.365946+0000 mon.a (mon.0) 3499 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.367430+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.367430+0000 mon.a (mon.0) 3500 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.367573+0000 mon.a (mon.0) 3501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.367573+0000 mon.a (mon.0) 3501 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.373382+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.373382+0000 mon.a (mon.0) 3502 : cluster [DBG] osdmap e728: 8 total, 7 up, 8 in 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.374535+0000 mon.c (mon.2) 763 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.374535+0000 mon.c (mon.2) 763 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.375478+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.375478+0000 mon.a (mon.0) 3503 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.522382+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.50:0/229762528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.522382+0000 mon.b (mon.1) 440 : audit [INF] from='client.? 172.21.15.50:0/229762528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.523520+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:06 smithi190 bash[23868]: audit 2024-04-26T16:35:06.523520+0000 mon.a (mon.0) 3504 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:35:06.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:06.925 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:06.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:07.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:07.056 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:07.160 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:07.186 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:07.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:07.317 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:07.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:07.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.372+0000 7fd95b7fe700 1 -- 172.21.15.50:0/229762528 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v729) ==== 133+0+0 (secure 0 0 0) 0x7fd95405e1e0 con 0x7fd9641024a0 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 10 librados: watch_flush enter 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 10 librados: watch_flush exit 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 20 client.43858.objecter shutdown clearing up homeless session... 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 10 client.43858.objecter successfully canceled tick 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd950071f80 msgr2=0x7fd950074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd950071f80 0x7fd950074440 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7fd960009fd0 tx=0x7fd960009260 comp rx=0 tx=0).stop 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 msgr2=0x7fd9641974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:07.375 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 0x7fd9641974b0 secure :-1 s=READY pgs=4144 cs=0 l=1 rev1=1 crypto rx=0x7fd95400bfd0 tx=0x7fd95400b6a0 comp rx=0 tx=0).stop 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 shutdown_connections 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd950071f80 0x7fd950074440 unknown :-1 s=CLOSED pgs=3822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd964107a90 0x7fd96419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd964101ae0 0x7fd964196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 --2- 172.21.15.50:0/229762528 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd9641024a0 0x7fd9641974b0 unknown :-1 s=CLOSED pgs=4144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 >> 172.21.15.50:0/229762528 conn(0x7fd964078990 msgr2=0x7fd964104ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 shutdown_connections 2024-04-26T16:35:07.376 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 -- 172.21.15.50:0/229762528 wait complete. 2024-04-26T16:35:07.377 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:07.376+0000 7fd96c57d700 1 librados: shutdown 2024-04-26T16:35:07.377 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-26T16:35:07.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:07.448 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:07.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:35:07.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:05.879461+0000 mgr.y (mgr.24401) 682 : cluster [DBG] pgmap v1130: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 80/633 objects degraded (12.638%) 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:05.879461+0000 mgr.y (mgr.24401) 682 : cluster [DBG] pgmap v1130: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 80/633 objects degraded (12.638%) 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.413576+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.413576+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.417598+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.417598+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.477332+0000 osd.4 (osd.4) 365 : cluster [DBG] 11.e scrub starts 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.477332+0000 osd.4 (osd.4) 365 : cluster [DBG] 11.e scrub starts 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.480050+0000 osd.4 (osd.4) 366 : cluster [DBG] 11.e scrub ok 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.480050+0000 osd.4 (osd.4) 366 : cluster [DBG] 11.e scrub ok 2024-04-26T16:35:07.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.531666+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.531666+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.533829+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.533829+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.620826+0000 osd.5 (osd.5) 269 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.620826+0000 osd.5 (osd.5) 269 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.624524+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.624524+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.807743+0000 osd.2 (osd.2) 353 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.807743+0000 osd.2 (osd.2) 353 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.808925+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:06.808925+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.370304+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.370304+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.370497+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:35:07.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.370497+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:35:07.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.378363+0000 mon.a (mon.0) 3507 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2024-04-26T16:35:07.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.378363+0000 mon.a (mon.0) 3507 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2024-04-26T16:35:07.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.379113+0000 mon.c (mon.2) 764 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.379113+0000 mon.c (mon.2) 764 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.380042+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:07 smithi190 bash[23868]: audit 2024-04-26T16:35:07.380042+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.580 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:07.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:07.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:07.707 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:07.744 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:05.879461+0000 mgr.y (mgr.24401) 682 : cluster [DBG] pgmap v1130: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 80/633 objects degraded (12.638%) 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:05.879461+0000 mgr.y (mgr.24401) 682 : cluster [DBG] pgmap v1130: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 80/633 objects degraded (12.638%) 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.413576+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.413576+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.417598+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.417598+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.477332+0000 osd.4 (osd.4) 365 : cluster [DBG] 11.e scrub starts 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.477332+0000 osd.4 (osd.4) 365 : cluster [DBG] 11.e scrub starts 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.480050+0000 osd.4 (osd.4) 366 : cluster [DBG] 11.e scrub ok 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.480050+0000 osd.4 (osd.4) 366 : cluster [DBG] 11.e scrub ok 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.531666+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.531666+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:35:07.745 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.533829+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.533829+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.620826+0000 osd.5 (osd.5) 269 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.620826+0000 osd.5 (osd.5) 269 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.624524+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.624524+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.807743+0000 osd.2 (osd.2) 353 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.807743+0000 osd.2 (osd.2) 353 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.808925+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:06.808925+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.370304+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.370304+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.370497+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.370497+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:35:07.746 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.378363+0000 mon.a (mon.0) 3507 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.378363+0000 mon.a (mon.0) 3507 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.379113+0000 mon.c (mon.2) 764 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.379113+0000 mon.c (mon.2) 764 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.380042+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[21463]: audit 2024-04-26T16:35:07.380042+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:05.879461+0000 mgr.y (mgr.24401) 682 : cluster [DBG] pgmap v1130: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 80/633 objects degraded (12.638%) 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:05.879461+0000 mgr.y (mgr.24401) 682 : cluster [DBG] pgmap v1130: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 707 MiB used, 715 GiB / 715 GiB avail; 80/633 objects degraded (12.638%) 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.413576+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.413576+0000 osd.3 (osd.3) 415 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:35:07.747 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.417598+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.417598+0000 osd.3 (osd.3) 416 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.477332+0000 osd.4 (osd.4) 365 : cluster [DBG] 11.e scrub starts 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.477332+0000 osd.4 (osd.4) 365 : cluster [DBG] 11.e scrub starts 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.480050+0000 osd.4 (osd.4) 366 : cluster [DBG] 11.e scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.480050+0000 osd.4 (osd.4) 366 : cluster [DBG] 11.e scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.531666+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.531666+0000 osd.7 (osd.7) 393 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.533829+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.533829+0000 osd.7 (osd.7) 394 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.620826+0000 osd.5 (osd.5) 269 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.620826+0000 osd.5 (osd.5) 269 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.624524+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:35:07.748 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.624524+0000 osd.5 (osd.5) 270 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.807743+0000 osd.2 (osd.2) 353 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.807743+0000 osd.2 (osd.2) 353 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.808925+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:06.808925+0000 osd.2 (osd.2) 354 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.370304+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.370304+0000 mon.a (mon.0) 3505 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.370497+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.370497+0000 mon.a (mon.0) 3506 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.378363+0000 mon.a (mon.0) 3507 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.378363+0000 mon.a (mon.0) 3507 : cluster [DBG] osdmap e729: 8 total, 7 up, 8 in 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.379113+0000 mon.c (mon.2) 764 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.379113+0000 mon.c (mon.2) 764 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.380042+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.749 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:07 smithi050 bash[29234]: audit 2024-04-26T16:35:07.380042+0000 mon.a (mon.0) 3508 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]: dispatch 2024-04-26T16:35:07.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:07.838 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:07.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:07.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:07.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:07.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:08.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:08.101 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:08.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:08.230 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:08.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:08.334 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:08.364 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:08.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:08.494 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:08.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:08.625 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: audit 2024-04-26T16:35:07.214518+0000 mgr.y (mgr.24401) 683 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: audit 2024-04-26T16:35:07.214518+0000 mgr.y (mgr.24401) 683 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.539850+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.539850+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.542132+0000 osd.4 (osd.4) 368 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.542132+0000 osd.4 (osd.4) 368 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.370553+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.370553+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: audit 2024-04-26T16:35:08.372354+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]': finished 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: audit 2024-04-26T16:35:08.372354+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]': finished 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.376112+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.376112+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: audit 2024-04-26T16:35:07.214518+0000 mgr.y (mgr.24401) 683 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: audit 2024-04-26T16:35:07.214518+0000 mgr.y (mgr.24401) 683 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.539850+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.539850+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.542132+0000 osd.4 (osd.4) 368 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.542132+0000 osd.4 (osd.4) 368 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.370553+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.370553+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: audit 2024-04-26T16:35:08.372354+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]': finished 2024-04-26T16:35:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: audit 2024-04-26T16:35:08.372354+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]': finished 2024-04-26T16:35:08.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.376112+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2024-04-26T16:35:08.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:08 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.376112+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2024-04-26T16:35:08.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: audit 2024-04-26T16:35:07.214518+0000 mgr.y (mgr.24401) 683 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: audit 2024-04-26T16:35:07.214518+0000 mgr.y (mgr.24401) 683 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.539850+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.539850+0000 osd.4 (osd.4) 367 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.542132+0000 osd.4 (osd.4) 368 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.542132+0000 osd.4 (osd.4) 368 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.370553+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.370553+0000 mon.a (mon.0) 3509 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: audit 2024-04-26T16:35:08.372354+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]': finished 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: audit 2024-04-26T16:35:08.372354+0000 mon.a (mon.0) 3510 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-124", "mode": "writeback"}]': finished 2024-04-26T16:35:08.740 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.376112+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2024-04-26T16:35:08.741 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:08 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.376112+0000 mon.a (mon.0) 3511 : cluster [DBG] osdmap e730: 8 total, 7 up, 8 in 2024-04-26T16:35:08.753 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:08.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:08.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:08.879 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:08.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:08.982 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:09.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:09.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:09.142 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:09.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:09.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-04-26T16:35:09.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:09.272 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:09.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:09.406 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:09.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:09.534 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:09.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:09.666 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:09.769 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:09.798 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.881140+0000 mgr.y (mgr.24401) 684 : cluster [DBG] pgmap v1133: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:07.881140+0000 mgr.y (mgr.24401) 684 : cluster [DBG] pgmap v1133: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.447260+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.447260+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.450004+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.450004+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:35:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.618207+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.618207+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.620710+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.620710+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.379110+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.379110+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: audit 2024-04-26T16:35:09.416019+0000 mon.c (mon.2) 765 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: audit 2024-04-26T16:35:09.416019+0000 mon.c (mon.2) 765 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: audit 2024-04-26T16:35:09.416418+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:09 smithi190 bash[23868]: audit 2024-04-26T16:35:09.416418+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.881140+0000 mgr.y (mgr.24401) 684 : cluster [DBG] pgmap v1133: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:07.881140+0000 mgr.y (mgr.24401) 684 : cluster [DBG] pgmap v1133: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.447260+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.447260+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.450004+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.450004+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.618207+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.618207+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.620710+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.620710+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.379110+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.379110+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: audit 2024-04-26T16:35:09.416019+0000 mon.c (mon.2) 765 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: audit 2024-04-26T16:35:09.416019+0000 mon.c (mon.2) 765 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: audit 2024-04-26T16:35:09.416418+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[21463]: audit 2024-04-26T16:35:09.416418+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.881140+0000 mgr.y (mgr.24401) 684 : cluster [DBG] pgmap v1133: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:07.881140+0000 mgr.y (mgr.24401) 684 : cluster [DBG] pgmap v1133: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.447260+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.447260+0000 osd.7 (osd.7) 395 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.450004+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.450004+0000 osd.7 (osd.7) 396 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.618207+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.618207+0000 osd.5 (osd.5) 271 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.620710+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.620710+0000 osd.5 (osd.5) 272 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:35:09.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.379110+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2024-04-26T16:35:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.379110+0000 mon.a (mon.0) 3512 : cluster [DBG] osdmap e731: 8 total, 7 up, 8 in 2024-04-26T16:35:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: audit 2024-04-26T16:35:09.416019+0000 mon.c (mon.2) 765 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: audit 2024-04-26T16:35:09.416019+0000 mon.c (mon.2) 765 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: audit 2024-04-26T16:35:09.416418+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:09 smithi050 bash[29234]: audit 2024-04-26T16:35:09.416418+0000 mon.a (mon.0) 3513 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:09.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:09.928 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:09.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-04-26T16:35:10.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-04-26T16:35:10.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:10.058 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:10.161 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:10.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:10.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:10.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:10.322 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:10.425 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:10.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:10.585 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:10.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:10.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.836077+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:35:10.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.836077+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:35:10.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.838324+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:35:10.700 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:08.838324+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.792373+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.a scrub starts 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.792373+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.a scrub starts 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.794138+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.a scrub ok 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.794138+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.a scrub ok 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.922582+0000 osd.0 (osd.0) 409 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.922582+0000 osd.0 (osd.0) 409 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.924652+0000 osd.0 (osd.0) 410 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.924652+0000 osd.0 (osd.0) 410 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: audit 2024-04-26T16:35:10.377263+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: audit 2024-04-26T16:35:10.377263+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.385260+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.385260+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: audit 2024-04-26T16:35:10.387345+0000 mon.c (mon.2) 766 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: audit 2024-04-26T16:35:10.387345+0000 mon.c (mon.2) 766 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: audit 2024-04-26T16:35:10.391545+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[29234]: audit 2024-04-26T16:35:10.391545+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.836077+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.836077+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.838324+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:08.838324+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.792373+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.a scrub starts 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.792373+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.a scrub starts 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.794138+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.a scrub ok 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.794138+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.a scrub ok 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.922582+0000 osd.0 (osd.0) 409 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:35:10.702 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.922582+0000 osd.0 (osd.0) 409 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.924652+0000 osd.0 (osd.0) 410 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.924652+0000 osd.0 (osd.0) 410 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: audit 2024-04-26T16:35:10.377263+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: audit 2024-04-26T16:35:10.377263+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.385260+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.385260+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: audit 2024-04-26T16:35:10.387345+0000 mon.c (mon.2) 766 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: audit 2024-04-26T16:35:10.387345+0000 mon.c (mon.2) 766 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: audit 2024-04-26T16:35:10.391545+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.703 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:10 smithi050 bash[21463]: audit 2024-04-26T16:35:10.391545+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:10.819 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.836077+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.836077+0000 osd.1 (osd.1) 383 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.838324+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:08.838324+0000 osd.1 (osd.1) 384 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.792373+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.a scrub starts 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.792373+0000 osd.2 (osd.2) 355 : cluster [DBG] 5.a scrub starts 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.794138+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.a scrub ok 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.794138+0000 osd.2 (osd.2) 356 : cluster [DBG] 5.a scrub ok 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.922582+0000 osd.0 (osd.0) 409 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:35:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.922582+0000 osd.0 (osd.0) 409 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.924652+0000 osd.0 (osd.0) 410 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.924652+0000 osd.0 (osd.0) 410 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: audit 2024-04-26T16:35:10.377263+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: audit 2024-04-26T16:35:10.377263+0000 mon.a (mon.0) 3514 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.385260+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.385260+0000 mon.a (mon.0) 3515 : cluster [DBG] osdmap e732: 8 total, 7 up, 8 in 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: audit 2024-04-26T16:35:10.387345+0000 mon.c (mon.2) 766 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: audit 2024-04-26T16:35:10.387345+0000 mon.c (mon.2) 766 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: audit 2024-04-26T16:35:10.391545+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:10 smithi190 bash[23868]: audit 2024-04-26T16:35:10.391545+0000 mon.a (mon.0) 3516 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]: dispatch 2024-04-26T16:35:10.850 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:10.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:10.981 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:10.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:11.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:11.110 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:11.213 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:11.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:11.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:11.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:11.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:11.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:11.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:11.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:11.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.874052+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:35:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.874052+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.876148+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.876148+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.882001+0000 mgr.y (mgr.24401) 685 : cluster [DBG] pgmap v1136: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:09.882001+0000 mgr.y (mgr.24401) 685 : cluster [DBG] pgmap v1136: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.395495+0000 osd.3 (osd.3) 417 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.395495+0000 osd.3 (osd.3) 417 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.400160+0000 osd.3 (osd.3) 418 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.400160+0000 osd.3 (osd.3) 418 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.975097+0000 osd.7 (osd.7) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.975097+0000 osd.7 (osd.7) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.975676+0000 osd.7 (osd.7) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.975676+0000 osd.7 (osd.7) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:11.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.338948+0000 mon.a (mon.0) 3517 : cluster [WRN] Health check update: Degraded data redundancy: 82/657 objects degraded (12.481%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.338948+0000 mon.a (mon.0) 3517 : cluster [WRN] Health check update: Degraded data redundancy: 82/657 objects degraded (12.481%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.339016+0000 mon.a (mon.0) 3518 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.339016+0000 mon.a (mon.0) 3518 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.340250+0000 mon.a (mon.0) 3519 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.340250+0000 mon.a (mon.0) 3519 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: audit 2024-04-26T16:35:11.343027+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: audit 2024-04-26T16:35:11.343027+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.352721+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e733: 8 total, 7 up, 8 in 2024-04-26T16:35:11.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:11 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.352721+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e733: 8 total, 7 up, 8 in 2024-04-26T16:35:11.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:11.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.874052+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:35:11.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.874052+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.876148+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.876148+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.882001+0000 mgr.y (mgr.24401) 685 : cluster [DBG] pgmap v1136: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:09.882001+0000 mgr.y (mgr.24401) 685 : cluster [DBG] pgmap v1136: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.395495+0000 osd.3 (osd.3) 417 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.395495+0000 osd.3 (osd.3) 417 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.400160+0000 osd.3 (osd.3) 418 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.400160+0000 osd.3 (osd.3) 418 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.975097+0000 osd.7 (osd.7) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.975097+0000 osd.7 (osd.7) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.975676+0000 osd.7 (osd.7) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.975676+0000 osd.7 (osd.7) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.338948+0000 mon.a (mon.0) 3517 : cluster [WRN] Health check update: Degraded data redundancy: 82/657 objects degraded (12.481%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.338948+0000 mon.a (mon.0) 3517 : cluster [WRN] Health check update: Degraded data redundancy: 82/657 objects degraded (12.481%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.339016+0000 mon.a (mon.0) 3518 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.339016+0000 mon.a (mon.0) 3518 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.340250+0000 mon.a (mon.0) 3519 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.340250+0000 mon.a (mon.0) 3519 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: audit 2024-04-26T16:35:11.343027+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: audit 2024-04-26T16:35:11.343027+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.352721+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e733: 8 total, 7 up, 8 in 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.352721+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e733: 8 total, 7 up, 8 in 2024-04-26T16:35:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.874052+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.874052+0000 osd.1 (osd.1) 385 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.876148+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.876148+0000 osd.1 (osd.1) 386 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.882001+0000 mgr.y (mgr.24401) 685 : cluster [DBG] pgmap v1136: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:09.882001+0000 mgr.y (mgr.24401) 685 : cluster [DBG] pgmap v1136: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s; 82/657 objects degraded (12.481%) 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.395495+0000 osd.3 (osd.3) 417 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.395495+0000 osd.3 (osd.3) 417 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.400160+0000 osd.3 (osd.3) 418 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.400160+0000 osd.3 (osd.3) 418 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.975097+0000 osd.7 (osd.7) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.975097+0000 osd.7 (osd.7) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.975676+0000 osd.7 (osd.7) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.975676+0000 osd.7 (osd.7) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.338948+0000 mon.a (mon.0) 3517 : cluster [WRN] Health check update: Degraded data redundancy: 82/657 objects degraded (12.481%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.338948+0000 mon.a (mon.0) 3517 : cluster [WRN] Health check update: Degraded data redundancy: 82/657 objects degraded (12.481%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.339016+0000 mon.a (mon.0) 3518 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.339016+0000 mon.a (mon.0) 3518 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.340250+0000 mon.a (mon.0) 3519 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.340250+0000 mon.a (mon.0) 3519 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: audit 2024-04-26T16:35:11.343027+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: audit 2024-04-26T16:35:11.343027+0000 mon.a (mon.0) 3520 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-124"}]': finished 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.352721+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e733: 8 total, 7 up, 8 in 2024-04-26T16:35:11.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:11 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.352721+0000 mon.a (mon.0) 3521 : cluster [DBG] osdmap e733: 8 total, 7 up, 8 in 2024-04-26T16:35:11.901 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:11.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:12.004 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:12.026 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:12.030 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:35:12.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:35:12.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:12.158 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:12.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:12.288 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:12.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:12.411 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:12.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:12.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:12.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:12.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:12.673 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:12.776 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:12.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.927704+0000 osd.0 (osd.0) 411 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:35:12.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.927704+0000 osd.0 (osd.0) 411 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:35:12.788 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.934313+0000 osd.0 (osd.0) 412 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:10.934313+0000 osd.0 (osd.0) 412 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.442064+0000 osd.3 (osd.3) 419 : cluster [DBG] 3.b scrub starts 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.442064+0000 osd.3 (osd.3) 419 : cluster [DBG] 3.b scrub starts 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.444410+0000 osd.3 (osd.3) 420 : cluster [DBG] 3.b scrub ok 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.444410+0000 osd.3 (osd.3) 420 : cluster [DBG] 3.b scrub ok 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.870809+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.870809+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.872829+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.872829+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.348699+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e734: 8 total, 7 up, 8 in 2024-04-26T16:35:12.789 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.348699+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e734: 8 total, 7 up, 8 in 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.927704+0000 osd.0 (osd.0) 411 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.927704+0000 osd.0 (osd.0) 411 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.934313+0000 osd.0 (osd.0) 412 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:10.934313+0000 osd.0 (osd.0) 412 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.442064+0000 osd.3 (osd.3) 419 : cluster [DBG] 3.b scrub starts 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.442064+0000 osd.3 (osd.3) 419 : cluster [DBG] 3.b scrub starts 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.444410+0000 osd.3 (osd.3) 420 : cluster [DBG] 3.b scrub ok 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.444410+0000 osd.3 (osd.3) 420 : cluster [DBG] 3.b scrub ok 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.870809+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.870809+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:35:12.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.872829+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:35:12.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.872829+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:35:12.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.348699+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e734: 8 total, 7 up, 8 in 2024-04-26T16:35:12.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:12 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.348699+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e734: 8 total, 7 up, 8 in 2024-04-26T16:35:12.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:12.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.927704+0000 osd.0 (osd.0) 411 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.927704+0000 osd.0 (osd.0) 411 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.934313+0000 osd.0 (osd.0) 412 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:10.934313+0000 osd.0 (osd.0) 412 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.442064+0000 osd.3 (osd.3) 419 : cluster [DBG] 3.b scrub starts 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.442064+0000 osd.3 (osd.3) 419 : cluster [DBG] 3.b scrub starts 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.444410+0000 osd.3 (osd.3) 420 : cluster [DBG] 3.b scrub ok 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.444410+0000 osd.3 (osd.3) 420 : cluster [DBG] 3.b scrub ok 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.870809+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.870809+0000 osd.2 (osd.2) 357 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.872829+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.872829+0000 osd.2 (osd.2) 358 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:35:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.348699+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e734: 8 total, 7 up, 8 in 2024-04-26T16:35:12.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:12 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.348699+0000 mon.a (mon.0) 3522 : cluster [DBG] osdmap e734: 8 total, 7 up, 8 in 2024-04-26T16:35:12.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:12.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:12.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:13.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:13.062 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:13.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:13.193 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:13.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:13.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:13.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:13.454 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:13.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:13.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:13.584 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:13.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:13.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:13.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.884969+0000 mgr.y (mgr.24401) 686 : cluster [DBG] pgmap v1139: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:11.884969+0000 mgr.y (mgr.24401) 686 : cluster [DBG] pgmap v1139: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:12.631169+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:12.631169+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:12.634174+0000 mon.c (mon.2) 767 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:12.634174+0000 mon.c (mon.2) 767 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.821813+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.821813+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.825748+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.825748+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.356830+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e735: 8 total, 7 up, 8 in 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.356830+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e735: 8 total, 7 up, 8 in 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:13.361458+0000 mon.c (mon.2) 768 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:13.361458+0000 mon.c (mon.2) 768 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:13.363405+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:13 smithi190 bash[23868]: audit 2024-04-26T16:35:13.363405+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.845 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:13.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.884969+0000 mgr.y (mgr.24401) 686 : cluster [DBG] pgmap v1139: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:11.884969+0000 mgr.y (mgr.24401) 686 : cluster [DBG] pgmap v1139: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:12.631169+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:12.631169+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:12.634174+0000 mon.c (mon.2) 767 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:12.634174+0000 mon.c (mon.2) 767 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.821813+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.821813+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.825748+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.825748+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.356830+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e735: 8 total, 7 up, 8 in 2024-04-26T16:35:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.356830+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e735: 8 total, 7 up, 8 in 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:13.361458+0000 mon.c (mon.2) 768 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:13.361458+0000 mon.c (mon.2) 768 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:13.363405+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[21463]: audit 2024-04-26T16:35:13.363405+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.884969+0000 mgr.y (mgr.24401) 686 : cluster [DBG] pgmap v1139: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:11.884969+0000 mgr.y (mgr.24401) 686 : cluster [DBG] pgmap v1139: 204 pgs: 53 active+undersized, 20 active+undersized+degraded, 131 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 4 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:12.631169+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:12.631169+0000 mon.a (mon.0) 3523 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:12.634174+0000 mon.c (mon.2) 767 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:12.634174+0000 mon.c (mon.2) 767 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.821813+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:35:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.821813+0000 osd.2 (osd.2) 359 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.825748+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.825748+0000 osd.2 (osd.2) 360 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.356830+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e735: 8 total, 7 up, 8 in 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.356830+0000 mon.a (mon.0) 3524 : cluster [DBG] osdmap e735: 8 total, 7 up, 8 in 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:13.361458+0000 mon.c (mon.2) 768 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:13.361458+0000 mon.c (mon.2) 768 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:13.363405+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:13 smithi050 bash[29234]: audit 2024-04-26T16:35:13.363405+0000 mon.a (mon.0) 3525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:13.949 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:13.956 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:13.985 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:13.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:13.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:14.088 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:14.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:14.218 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:14.247 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:14.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:14.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:14.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:14.499 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:35:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:35:14.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:14.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:14.647 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:14.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:14.777 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:14.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.930226+0000 osd.0 (osd.0) 413 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.930226+0000 osd.0 (osd.0) 413 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.932339+0000 osd.0 (osd.0) 414 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:12.932339+0000 osd.0 (osd.0) 414 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.585685+0000 osd.5 (osd.5) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.585685+0000 osd.5 (osd.5) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.587486+0000 osd.5 (osd.5) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.587486+0000 osd.5 (osd.5) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: audit 2024-04-26T16:35:14.352148+0000 mon.a (mon.0) 3526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: audit 2024-04-26T16:35:14.352148+0000 mon.a (mon.0) 3526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.361121+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e736: 8 total, 7 up, 8 in 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.361121+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e736: 8 total, 7 up, 8 in 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: audit 2024-04-26T16:35:14.372351+0000 mon.c (mon.2) 769 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: audit 2024-04-26T16:35:14.372351+0000 mon.c (mon.2) 769 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: audit 2024-04-26T16:35:14.374153+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[21463]: audit 2024-04-26T16:35:14.374153+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.930226+0000 osd.0 (osd.0) 413 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.930226+0000 osd.0 (osd.0) 413 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.932339+0000 osd.0 (osd.0) 414 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:12.932339+0000 osd.0 (osd.0) 414 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.585685+0000 osd.5 (osd.5) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.585685+0000 osd.5 (osd.5) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.587486+0000 osd.5 (osd.5) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.587486+0000 osd.5 (osd.5) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:35:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: audit 2024-04-26T16:35:14.352148+0000 mon.a (mon.0) 3526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: audit 2024-04-26T16:35:14.352148+0000 mon.a (mon.0) 3526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.361121+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e736: 8 total, 7 up, 8 in 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.361121+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e736: 8 total, 7 up, 8 in 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: audit 2024-04-26T16:35:14.372351+0000 mon.c (mon.2) 769 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: audit 2024-04-26T16:35:14.372351+0000 mon.c (mon.2) 769 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: audit 2024-04-26T16:35:14.374153+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:14 smithi050 bash[29234]: audit 2024-04-26T16:35:14.374153+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:14.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:14.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:15.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.930226+0000 osd.0 (osd.0) 413 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.930226+0000 osd.0 (osd.0) 413 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.932339+0000 osd.0 (osd.0) 414 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:12.932339+0000 osd.0 (osd.0) 414 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.585685+0000 osd.5 (osd.5) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.585685+0000 osd.5 (osd.5) 273 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.587486+0000 osd.5 (osd.5) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:35:15.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.587486+0000 osd.5 (osd.5) 274 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: audit 2024-04-26T16:35:14.352148+0000 mon.a (mon.0) 3526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: audit 2024-04-26T16:35:14.352148+0000 mon.a (mon.0) 3526 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.361121+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e736: 8 total, 7 up, 8 in 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.361121+0000 mon.a (mon.0) 3527 : cluster [DBG] osdmap e736: 8 total, 7 up, 8 in 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: audit 2024-04-26T16:35:14.372351+0000 mon.c (mon.2) 769 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: audit 2024-04-26T16:35:14.372351+0000 mon.c (mon.2) 769 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: audit 2024-04-26T16:35:14.374153+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:15.030 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:14 smithi190 bash[23868]: audit 2024-04-26T16:35:14.374153+0000 mon.a (mon.0) 3528 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:15.045 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:15.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:15.105 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-04-26T16:35:15.105 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-04-26T16:35:15.105 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-04-26T16:35:15.105 DEBUG:teuthology.orchestra.run.smithi190:osd.6> sudo journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service 2024-04-26T16:35:15.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl start ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6 2024-04-26T16:35:15.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:15.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-26T16:35:15.179 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:15.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:15.290 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:15 smithi190 systemd[1]: Started Ceph osd.6 for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:35:15.302 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:15.405 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:15.433 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:15.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:15.563 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:15.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:15.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.886893+0000 mgr.y (mgr.24401) 687 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:15.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:13.886893+0000 mgr.y (mgr.24401) 687 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:15.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.418486+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:35:15.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.418486+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:35:15.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.420933+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:35:15.678 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.420933+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.524643+0000 osd.7 (osd.7) 399 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.524643+0000 osd.7 (osd.7) 399 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.526720+0000 osd.7 (osd.7) 400 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.526720+0000 osd.7 (osd.7) 400 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.800321+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.800321+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.801518+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.801518+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.852609+0000 osd.0 (osd.0) 415 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.852609+0000 osd.0 (osd.0) 415 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.855034+0000 osd.0 (osd.0) 416 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:14.855034+0000 osd.0 (osd.0) 416 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: audit 2024-04-26T16:35:15.355075+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: audit 2024-04-26T16:35:15.355075+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:15.679 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.363477+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e737: 8 total, 7 up, 8 in 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.363477+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e737: 8 total, 7 up, 8 in 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: audit 2024-04-26T16:35:15.365526+0000 mon.c (mon.2) 770 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: audit 2024-04-26T16:35:15.365526+0000 mon.c (mon.2) 770 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: audit 2024-04-26T16:35:15.366368+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[21463]: audit 2024-04-26T16:35:15.366368+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.886893+0000 mgr.y (mgr.24401) 687 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:13.886893+0000 mgr.y (mgr.24401) 687 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.418486+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.418486+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.420933+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.420933+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:35:15.680 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.524643+0000 osd.7 (osd.7) 399 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.524643+0000 osd.7 (osd.7) 399 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.526720+0000 osd.7 (osd.7) 400 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.526720+0000 osd.7 (osd.7) 400 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.800321+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.800321+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.801518+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.801518+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.852609+0000 osd.0 (osd.0) 415 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.852609+0000 osd.0 (osd.0) 415 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.855034+0000 osd.0 (osd.0) 416 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:14.855034+0000 osd.0 (osd.0) 416 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: audit 2024-04-26T16:35:15.355075+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: audit 2024-04-26T16:35:15.355075+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.363477+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e737: 8 total, 7 up, 8 in 2024-04-26T16:35:15.681 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.363477+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e737: 8 total, 7 up, 8 in 2024-04-26T16:35:15.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: audit 2024-04-26T16:35:15.365526+0000 mon.c (mon.2) 770 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: audit 2024-04-26T16:35:15.365526+0000 mon.c (mon.2) 770 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: audit 2024-04-26T16:35:15.366368+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.682 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:15 smithi050 bash[29234]: audit 2024-04-26T16:35:15.366368+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.691 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:15.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:15.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:15.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.886893+0000 mgr.y (mgr.24401) 687 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:15.805 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:13.886893+0000 mgr.y (mgr.24401) 687 : cluster [DBG] pgmap v1142: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.418486+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.418486+0000 osd.3 (osd.3) 421 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.420933+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.420933+0000 osd.3 (osd.3) 422 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.524643+0000 osd.7 (osd.7) 399 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.524643+0000 osd.7 (osd.7) 399 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.526720+0000 osd.7 (osd.7) 400 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.526720+0000 osd.7 (osd.7) 400 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.800321+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.800321+0000 osd.2 (osd.2) 361 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.801518+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.801518+0000 osd.2 (osd.2) 362 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:35:15.806 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.852609+0000 osd.0 (osd.0) 415 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.852609+0000 osd.0 (osd.0) 415 : cluster [DBG] 6.18 deep-scrub starts 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.855034+0000 osd.0 (osd.0) 416 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:14.855034+0000 osd.0 (osd.0) 416 : cluster [DBG] 6.18 deep-scrub ok 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: audit 2024-04-26T16:35:15.355075+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: audit 2024-04-26T16:35:15.355075+0000 mon.a (mon.0) 3529 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.363477+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e737: 8 total, 7 up, 8 in 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.363477+0000 mon.a (mon.0) 3530 : cluster [DBG] osdmap e737: 8 total, 7 up, 8 in 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: audit 2024-04-26T16:35:15.365526+0000 mon.c (mon.2) 770 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: audit 2024-04-26T16:35:15.365526+0000 mon.c (mon.2) 770 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: audit 2024-04-26T16:35:15.366368+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.807 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:15 smithi190 bash[23868]: audit 2024-04-26T16:35:15.366368+0000 mon.a (mon.0) 3531 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:15.821 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:15.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:15.950 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:15.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:16.001 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:16.002 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:16.053 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:16.177 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:16.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:16.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:16.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:16.463 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:16.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:16.592 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:16.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:16.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:16.722 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:16.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.852 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:16.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.830726+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:35:16.967 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.830726+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.832022+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.832022+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.341893+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check update: Degraded data redundancy: 82/645 objects degraded (12.713%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.341893+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check update: Degraded data redundancy: 82/645 objects degraded (12.713%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.341938+0000 mon.a (mon.0) 3533 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.341938+0000 mon.a (mon.0) 3533 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: audit 2024-04-26T16:35:16.356952+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: audit 2024-04-26T16:35:16.356952+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.361575+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e738: 8 total, 7 up, 8 in 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.361575+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e738: 8 total, 7 up, 8 in 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: audit 2024-04-26T16:35:16.365159+0000 mon.c (mon.2) 771 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: audit 2024-04-26T16:35:16.365159+0000 mon.c (mon.2) 771 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: audit 2024-04-26T16:35:16.365768+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[29234]: audit 2024-04-26T16:35:16.365768+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.830726+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.830726+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.832022+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.832022+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.341893+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check update: Degraded data redundancy: 82/645 objects degraded (12.713%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.341893+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check update: Degraded data redundancy: 82/645 objects degraded (12.713%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.341938+0000 mon.a (mon.0) 3533 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.341938+0000 mon.a (mon.0) 3533 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: audit 2024-04-26T16:35:16.356952+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:16.969 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: audit 2024-04-26T16:35:16.356952+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:16.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.361575+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e738: 8 total, 7 up, 8 in 2024-04-26T16:35:16.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.361575+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e738: 8 total, 7 up, 8 in 2024-04-26T16:35:16.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: audit 2024-04-26T16:35:16.365159+0000 mon.c (mon.2) 771 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: audit 2024-04-26T16:35:16.365159+0000 mon.c (mon.2) 771 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: audit 2024-04-26T16:35:16.365768+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:16 smithi050 bash[21463]: audit 2024-04-26T16:35:16.365768+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:16.983 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:16.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.830726+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.830726+0000 osd.0 (osd.0) 417 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.832022+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.832022+0000 osd.0 (osd.0) 418 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.341893+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check update: Degraded data redundancy: 82/645 objects degraded (12.713%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.341893+0000 mon.a (mon.0) 3532 : cluster [WRN] Health check update: Degraded data redundancy: 82/645 objects degraded (12.713%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.341938+0000 mon.a (mon.0) 3533 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.341938+0000 mon.a (mon.0) 3533 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: audit 2024-04-26T16:35:16.356952+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: audit 2024-04-26T16:35:16.356952+0000 mon.a (mon.0) 3534 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.361575+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e738: 8 total, 7 up, 8 in 2024-04-26T16:35:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.361575+0000 mon.a (mon.0) 3535 : cluster [DBG] osdmap e738: 8 total, 7 up, 8 in 2024-04-26T16:35:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: audit 2024-04-26T16:35:16.365159+0000 mon.c (mon.2) 771 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: audit 2024-04-26T16:35:16.365159+0000 mon.c (mon.2) 771 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: audit 2024-04-26T16:35:16.365768+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:16 smithi190 bash[23868]: audit 2024-04-26T16:35:16.365768+0000 mon.a (mon.0) 3536 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]: dispatch 2024-04-26T16:35:17.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:17.115 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:17.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:17.238 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:17.340 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:17.374 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:17.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:17.500 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:17.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:35:17.565 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-26T16:35:17.565 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-26T16:35:17.565 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T16:35:17.565 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-26T16:35:17.565 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-26T16:35:17.565 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-26T16:35:17.566 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-26T16:35:17.566 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-26T16:35:17.566 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[92554]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-26T16:35:17.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:17.629 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:17.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:17.759 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:17.861 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.836655+0000 osd.1 (osd.1) 387 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:35:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.836655+0000 osd.1 (osd.1) 387 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:35:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.838489+0000 osd.1 (osd.1) 388 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:35:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.838489+0000 osd.1 (osd.1) 388 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:35:17.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.887777+0000 mgr.y (mgr.24401) 688 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 82/645 objects degraded (12.713%) 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:15.887777+0000 mgr.y (mgr.24401) 688 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 82/645 objects degraded (12.713%) 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.407982+0000 osd.4 (osd.4) 369 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.407982+0000 osd.4 (osd.4) 369 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.410064+0000 osd.4 (osd.4) 370 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.410064+0000 osd.4 (osd.4) 370 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.563752+0000 osd.5 (osd.5) 275 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.563752+0000 osd.5 (osd.5) 275 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.569702+0000 osd.5 (osd.5) 276 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.569702+0000 osd.5 (osd.5) 276 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.824448+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.824448+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.826609+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:35:17.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.826609+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.357633+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.357633+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: audit 2024-04-26T16:35:17.360024+0000 mon.a (mon.0) 3538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]': finished 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: audit 2024-04-26T16:35:17.360024+0000 mon.a (mon.0) 3538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]': finished 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.365858+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 7 up, 8 in 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.365858+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 7 up, 8 in 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: audit 2024-04-26T16:35:17.440261+0000 mon.c (mon.2) 772 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: audit 2024-04-26T16:35:17.440261+0000 mon.c (mon.2) 772 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: audit 2024-04-26T16:35:17.440660+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[21463]: audit 2024-04-26T16:35:17.440660+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.876 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.836655+0000 osd.1 (osd.1) 387 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.836655+0000 osd.1 (osd.1) 387 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.838489+0000 osd.1 (osd.1) 388 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.838489+0000 osd.1 (osd.1) 388 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.887777+0000 mgr.y (mgr.24401) 688 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 82/645 objects degraded (12.713%) 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:15.887777+0000 mgr.y (mgr.24401) 688 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 82/645 objects degraded (12.713%) 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.407982+0000 osd.4 (osd.4) 369 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.407982+0000 osd.4 (osd.4) 369 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.410064+0000 osd.4 (osd.4) 370 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.410064+0000 osd.4 (osd.4) 370 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.563752+0000 osd.5 (osd.5) 275 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.563752+0000 osd.5 (osd.5) 275 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.569702+0000 osd.5 (osd.5) 276 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:35:17.877 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.569702+0000 osd.5 (osd.5) 276 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.824448+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.824448+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.826609+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.826609+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.357633+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.357633+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: audit 2024-04-26T16:35:17.360024+0000 mon.a (mon.0) 3538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]': finished 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: audit 2024-04-26T16:35:17.360024+0000 mon.a (mon.0) 3538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]': finished 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.365858+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 7 up, 8 in 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.365858+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 7 up, 8 in 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: audit 2024-04-26T16:35:17.440261+0000 mon.c (mon.2) 772 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: audit 2024-04-26T16:35:17.440261+0000 mon.c (mon.2) 772 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: audit 2024-04-26T16:35:17.440660+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:17 smithi050 bash[29234]: audit 2024-04-26T16:35:17.440660+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:17.890 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:17.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:17.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:18.022 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:18.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:18.150 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:18.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:18.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.836655+0000 osd.1 (osd.1) 387 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:35:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.836655+0000 osd.1 (osd.1) 387 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:35:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.838489+0000 osd.1 (osd.1) 388 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:35:18.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.838489+0000 osd.1 (osd.1) 388 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.887777+0000 mgr.y (mgr.24401) 688 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 82/645 objects degraded (12.713%) 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:15.887777+0000 mgr.y (mgr.24401) 688 : cluster [DBG] pgmap v1145: 204 pgs: 32 unknown, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 725 MiB used, 715 GiB / 715 GiB avail; 82/645 objects degraded (12.713%) 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.407982+0000 osd.4 (osd.4) 369 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.407982+0000 osd.4 (osd.4) 369 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.410064+0000 osd.4 (osd.4) 370 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.410064+0000 osd.4 (osd.4) 370 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.563752+0000 osd.5 (osd.5) 275 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.563752+0000 osd.5 (osd.5) 275 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.569702+0000 osd.5 (osd.5) 276 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.569702+0000 osd.5 (osd.5) 276 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.824448+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.824448+0000 osd.2 (osd.2) 363 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.826609+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:35:18.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.826609+0000 osd.2 (osd.2) 364 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.357633+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.357633+0000 mon.a (mon.0) 3537 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: audit 2024-04-26T16:35:17.360024+0000 mon.a (mon.0) 3538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]': finished 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: audit 2024-04-26T16:35:17.360024+0000 mon.a (mon.0) 3538 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-126", "mode": "writeback"}]': finished 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.365858+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 7 up, 8 in 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.365858+0000 mon.a (mon.0) 3539 : cluster [DBG] osdmap e739: 8 total, 7 up, 8 in 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: audit 2024-04-26T16:35:17.440261+0000 mon.c (mon.2) 772 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: audit 2024-04-26T16:35:17.440261+0000 mon.c (mon.2) 772 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: audit 2024-04-26T16:35:17.440660+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:18.266 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:17 smithi190 bash[23868]: audit 2024-04-26T16:35:17.440660+0000 mon.a (mon.0) 3540 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:18.277 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:18.380 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:18.406 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:18.509 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:18.537 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:18.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:18.667 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:18.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:18.794 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:18.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:18.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.810733+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.810733+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.812990+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:16.812990+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:17.222332+0000 mgr.y (mgr.24401) 689 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:17.222332+0000 mgr.y (mgr.24401) 689 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.433488+0000 osd.3 (osd.3) 423 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.433488+0000 osd.3 (osd.3) 423 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.434946+0000 osd.3 (osd.3) 424 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.434946+0000 osd.3 (osd.3) 424 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.598004+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:35:18.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.598004+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.600689+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.600689+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.756645+0000 osd.0 (osd.0) 419 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.756645+0000 osd.0 (osd.0) 419 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.758965+0000 osd.0 (osd.0) 420 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.758965+0000 osd.0 (osd.0) 420 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:18.362761+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:18.362761+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.370060+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 7 up, 8 in 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.370060+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 7 up, 8 in 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:18.375951+0000 mon.c (mon.2) 773 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:18.375951+0000 mon.c (mon.2) 773 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:18.376672+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:18.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:18 smithi190 bash[23868]: audit 2024-04-26T16:35:18.376672+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:18.924 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:18.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:19.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:19.056 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:19.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.810733+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:35:19.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.810733+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.812990+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:16.812990+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:17.222332+0000 mgr.y (mgr.24401) 689 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:17.222332+0000 mgr.y (mgr.24401) 689 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.433488+0000 osd.3 (osd.3) 423 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.433488+0000 osd.3 (osd.3) 423 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.434946+0000 osd.3 (osd.3) 424 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.434946+0000 osd.3 (osd.3) 424 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.598004+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.598004+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.600689+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.600689+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:35:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.756645+0000 osd.0 (osd.0) 419 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.756645+0000 osd.0 (osd.0) 419 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.758965+0000 osd.0 (osd.0) 420 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.758965+0000 osd.0 (osd.0) 420 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:18.362761+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:18.362761+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.370060+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 7 up, 8 in 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.370060+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 7 up, 8 in 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:18.375951+0000 mon.c (mon.2) 773 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:18.375951+0000 mon.c (mon.2) 773 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:18.376672+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[29234]: audit 2024-04-26T16:35:18.376672+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.810733+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.810733+0000 osd.1 (osd.1) 389 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.812990+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:16.812990+0000 osd.1 (osd.1) 390 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:17.222332+0000 mgr.y (mgr.24401) 689 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:17.222332+0000 mgr.y (mgr.24401) 689 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.433488+0000 osd.3 (osd.3) 423 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.433488+0000 osd.3 (osd.3) 423 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.434946+0000 osd.3 (osd.3) 424 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.434946+0000 osd.3 (osd.3) 424 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.598004+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:35:19.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.598004+0000 osd.5 (osd.5) 277 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.600689+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.600689+0000 osd.5 (osd.5) 278 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.756645+0000 osd.0 (osd.0) 419 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.756645+0000 osd.0 (osd.0) 419 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.758965+0000 osd.0 (osd.0) 420 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.758965+0000 osd.0 (osd.0) 420 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:18.362761+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:18.362761+0000 mon.a (mon.0) 3541 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.370060+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 7 up, 8 in 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.370060+0000 mon.a (mon.0) 3542 : cluster [DBG] osdmap e740: 8 total, 7 up, 8 in 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:18.375951+0000 mon.c (mon.2) 773 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:18.375951+0000 mon.c (mon.2) 773 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:18.376672+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:18 smithi050 bash[21463]: audit 2024-04-26T16:35:18.376672+0000 mon.a (mon.0) 3543 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]: dispatch 2024-04-26T16:35:19.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:19.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:19.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:19.317 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:19.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:19.443 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:19.546 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:19.575 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[92888]: debug 2024-04-26T16:35:19.369+0000 7f207223f700 1 -- 172.21.15.190:0/3456240827 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fc9fa13ba0 con 0x55fc9fa77000 2024-04-26T16:35:19.575 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[92888]: debug 2024-04-26T16:35:19.369+0000 7f207223f700 1 -- 172.21.15.190:0/3456240827 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fc9fa19680 con 0x55fc9fa77000 2024-04-26T16:35:19.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:19.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:19.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.802721+0000 osd.1 (osd.1) 391 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:35:19.688 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.802721+0000 osd.1 (osd.1) 391 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.804920+0000 osd.1 (osd.1) 392 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.804920+0000 osd.1 (osd.1) 392 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.888733+0000 mgr.y (mgr.24401) 690 : cluster [DBG] pgmap v1148: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:17.888733+0000 mgr.y (mgr.24401) 690 : cluster [DBG] pgmap v1148: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.448268+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub starts 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.448268+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub starts 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.451741+0000 osd.4 (osd.4) 372 : cluster [DBG] 2.f scrub ok 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.451741+0000 osd.4 (osd.4) 372 : cluster [DBG] 2.f scrub ok 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.582993+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.582993+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.589298+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.589298+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.784363+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:35:19.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.784363+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.787225+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.787225+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.362930+0000 mon.a (mon.0) 3544 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.362930+0000 mon.a (mon.0) 3544 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: audit 2024-04-26T16:35:19.364390+0000 mon.a (mon.0) 3545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: audit 2024-04-26T16:35:19.364390+0000 mon.a (mon.0) 3545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.368694+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e741: 8 total, 7 up, 8 in 2024-04-26T16:35:19.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.368694+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e741: 8 total, 7 up, 8 in 2024-04-26T16:35:19.701 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:19.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:19.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.802721+0000 osd.1 (osd.1) 391 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:35:19.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.802721+0000 osd.1 (osd.1) 391 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:35:19.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.804920+0000 osd.1 (osd.1) 392 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:35:19.818 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.804920+0000 osd.1 (osd.1) 392 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.888733+0000 mgr.y (mgr.24401) 690 : cluster [DBG] pgmap v1148: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:17.888733+0000 mgr.y (mgr.24401) 690 : cluster [DBG] pgmap v1148: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.448268+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub starts 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.448268+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub starts 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.451741+0000 osd.4 (osd.4) 372 : cluster [DBG] 2.f scrub ok 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.451741+0000 osd.4 (osd.4) 372 : cluster [DBG] 2.f scrub ok 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.582993+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:35:19.819 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.582993+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.589298+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.589298+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.784363+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.784363+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.787225+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.787225+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.362930+0000 mon.a (mon.0) 3544 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:19.820 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.362930+0000 mon.a (mon.0) 3544 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:19.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: audit 2024-04-26T16:35:19.364390+0000 mon.a (mon.0) 3545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:19.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: audit 2024-04-26T16:35:19.364390+0000 mon.a (mon.0) 3545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:19.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.368694+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e741: 8 total, 7 up, 8 in 2024-04-26T16:35:19.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.368694+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e741: 8 total, 7 up, 8 in 2024-04-26T16:35:19.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.802721+0000 osd.1 (osd.1) 391 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:35:19.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.802721+0000 osd.1 (osd.1) 391 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.804920+0000 osd.1 (osd.1) 392 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.804920+0000 osd.1 (osd.1) 392 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.888733+0000 mgr.y (mgr.24401) 690 : cluster [DBG] pgmap v1148: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:17.888733+0000 mgr.y (mgr.24401) 690 : cluster [DBG] pgmap v1148: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.448268+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub starts 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.448268+0000 osd.4 (osd.4) 371 : cluster [DBG] 2.f scrub starts 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.451741+0000 osd.4 (osd.4) 372 : cluster [DBG] 2.f scrub ok 2024-04-26T16:35:19.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.451741+0000 osd.4 (osd.4) 372 : cluster [DBG] 2.f scrub ok 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.582993+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.582993+0000 osd.5 (osd.5) 279 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.589298+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.589298+0000 osd.5 (osd.5) 280 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.784363+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.784363+0000 osd.0 (osd.0) 421 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.787225+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.787225+0000 osd.0 (osd.0) 422 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:35:19.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.362930+0000 mon.a (mon.0) 3544 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:19.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.362930+0000 mon.a (mon.0) 3544 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:19.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: audit 2024-04-26T16:35:19.364390+0000 mon.a (mon.0) 3545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:19.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: audit 2024-04-26T16:35:19.364390+0000 mon.a (mon.0) 3545 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-126"}]': finished 2024-04-26T16:35:19.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.368694+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e741: 8 total, 7 up, 8 in 2024-04-26T16:35:19.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:19 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.368694+0000 mon.a (mon.0) 3546 : cluster [DBG] osdmap e741: 8 total, 7 up, 8 in 2024-04-26T16:35:19.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:19.935 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:19.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:19.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:20.066 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:20.074 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:19 smithi190 bash[92888]: debug 2024-04-26T16:35:19.925+0000 7f207b3d77c0 -1 Falling back to public interface 2024-04-26T16:35:20.094 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:20.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:20.225 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:20.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:20.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:20.369 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 16 more times... 2024-04-26T16:35:20.369 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.369 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 15 more times... 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 14 more times... 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 13 more times... 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 12 more times... 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 11 more times... 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 10 more times... 2024-04-26T16:35:20.370 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 9 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 8 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 7 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 6 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 5 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 4 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 3 more times... 2024-04-26T16:35:20.371 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 2 more times... 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi050-73372-117 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 1 more times... 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: failed to clean up; probably need to scrub purged_snaps. 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (43277 ms) 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9777 ms) 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8061 ms) 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7919 ms) 2024-04-26T16:35:20.372 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-04-26T16:35:20.373 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9987 ms) 2024-04-26T16:35:20.373 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-04-26T16:35:20.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:20.486 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:20.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:20.619 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:20.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.823885+0000 osd.1 (osd.1) 393 : cluster [DBG] 323.2s0 deep-scrub starts 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.823885+0000 osd.1 (osd.1) 393 : cluster [DBG] 323.2s0 deep-scrub starts 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.833452+0000 osd.1 (osd.1) 394 : cluster [DBG] 323.2 deep-scrub ok 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:18.833452+0000 osd.1 (osd.1) 394 : cluster [DBG] 323.2 deep-scrub ok 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.758613+0000 osd.0 (osd.0) 423 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.758613+0000 osd.0 (osd.0) 423 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.762709+0000 osd.0 (osd.0) 424 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.762709+0000 osd.0 (osd.0) 424 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:35:20.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.371746+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e742: 8 total, 7 up, 8 in 2024-04-26T16:35:20.734 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:20 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.371746+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e742: 8 total, 7 up, 8 in 2024-04-26T16:35:20.749 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:20.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:20.878 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:20.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:20.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:21.010 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:21.114 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.823885+0000 osd.1 (osd.1) 393 : cluster [DBG] 323.2s0 deep-scrub starts 2024-04-26T16:35:21.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.823885+0000 osd.1 (osd.1) 393 : cluster [DBG] 323.2s0 deep-scrub starts 2024-04-26T16:35:21.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.833452+0000 osd.1 (osd.1) 394 : cluster [DBG] 323.2 deep-scrub ok 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:18.833452+0000 osd.1 (osd.1) 394 : cluster [DBG] 323.2 deep-scrub ok 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.758613+0000 osd.0 (osd.0) 423 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.758613+0000 osd.0 (osd.0) 423 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.762709+0000 osd.0 (osd.0) 424 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.762709+0000 osd.0 (osd.0) 424 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.371746+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e742: 8 total, 7 up, 8 in 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.371746+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e742: 8 total, 7 up, 8 in 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.823885+0000 osd.1 (osd.1) 393 : cluster [DBG] 323.2s0 deep-scrub starts 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.823885+0000 osd.1 (osd.1) 393 : cluster [DBG] 323.2s0 deep-scrub starts 2024-04-26T16:35:21.127 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.833452+0000 osd.1 (osd.1) 394 : cluster [DBG] 323.2 deep-scrub ok 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:18.833452+0000 osd.1 (osd.1) 394 : cluster [DBG] 323.2 deep-scrub ok 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.758613+0000 osd.0 (osd.0) 423 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.758613+0000 osd.0 (osd.0) 423 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.762709+0000 osd.0 (osd.0) 424 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.762709+0000 osd.0 (osd.0) 424 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.371746+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e742: 8 total, 7 up, 8 in 2024-04-26T16:35:21.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:20 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.371746+0000 mon.a (mon.0) 3547 : cluster [DBG] osdmap e742: 8 total, 7 up, 8 in 2024-04-26T16:35:21.142 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:21.245 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:21.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:21.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:21.393 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:21.497 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.517 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:21.586 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:35:21.586 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:35:21.586 INFO:teuthology.orchestra.run.smithi050.stdout: "num_ops": 0 2024-04-26T16:35:21.586 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:35:21.621 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.640 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:21.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.756 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.889863+0000 mgr.y (mgr.24401) 691 : cluster [DBG] pgmap v1151: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:19.889863+0000 mgr.y (mgr.24401) 691 : cluster [DBG] pgmap v1151: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.375664+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.375664+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.379833+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.379833+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.613556+0000 osd.5 (osd.5) 281 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.613556+0000 osd.5 (osd.5) 281 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.622229+0000 osd.5 (osd.5) 282 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.622229+0000 osd.5 (osd.5) 282 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.664442+0000 osd.5 (osd.5) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.664442+0000 osd.5 (osd.5) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.664852+0000 osd.5 (osd.5) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:21.757 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:20.664852+0000 osd.5 (osd.5) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.349559+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e743: 8 total, 7 up, 8 in 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.349559+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e743: 8 total, 7 up, 8 in 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: audit 2024-04-26T16:35:21.352678+0000 mon.c (mon.2) 774 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: audit 2024-04-26T16:35:21.352678+0000 mon.c (mon.2) 774 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: audit 2024-04-26T16:35:21.376289+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[29234]: audit 2024-04-26T16:35:21.376289+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.758 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.889863+0000 mgr.y (mgr.24401) 691 : cluster [DBG] pgmap v1151: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:19.889863+0000 mgr.y (mgr.24401) 691 : cluster [DBG] pgmap v1151: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.375664+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.375664+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.379833+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.379833+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.613556+0000 osd.5 (osd.5) 281 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.613556+0000 osd.5 (osd.5) 281 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.622229+0000 osd.5 (osd.5) 282 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.622229+0000 osd.5 (osd.5) 282 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.664442+0000 osd.5 (osd.5) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.664442+0000 osd.5 (osd.5) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.664852+0000 osd.5 (osd.5) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:20.664852+0000 osd.5 (osd.5) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.349559+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e743: 8 total, 7 up, 8 in 2024-04-26T16:35:21.759 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.349559+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e743: 8 total, 7 up, 8 in 2024-04-26T16:35:21.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: audit 2024-04-26T16:35:21.352678+0000 mon.c (mon.2) 774 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: audit 2024-04-26T16:35:21.352678+0000 mon.c (mon.2) 774 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: audit 2024-04-26T16:35:21.376289+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:21 smithi050 bash[21463]: audit 2024-04-26T16:35:21.376289+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:21.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:21.799 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:35:21.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:21.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:21.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:22.002 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:22.031 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.889863+0000 mgr.y (mgr.24401) 691 : cluster [DBG] pgmap v1151: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:19.889863+0000 mgr.y (mgr.24401) 691 : cluster [DBG] pgmap v1151: 204 pgs: 51 active+undersized, 20 active+undersized+degraded, 133 active+clean; 583 KiB data, 726 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 82/645 objects degraded (12.713%) 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.375664+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.375664+0000 osd.3 (osd.3) 425 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.379833+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.379833+0000 osd.3 (osd.3) 426 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.613556+0000 osd.5 (osd.5) 281 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.613556+0000 osd.5 (osd.5) 281 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.622229+0000 osd.5 (osd.5) 282 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:35:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.622229+0000 osd.5 (osd.5) 282 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.664442+0000 osd.5 (osd.5) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.664442+0000 osd.5 (osd.5) 283 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.664852+0000 osd.5 (osd.5) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:20.664852+0000 osd.5 (osd.5) 284 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.349559+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e743: 8 total, 7 up, 8 in 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.349559+0000 mon.a (mon.0) 3548 : cluster [DBG] osdmap e743: 8 total, 7 up, 8 in 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: audit 2024-04-26T16:35:21.352678+0000 mon.c (mon.2) 774 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: audit 2024-04-26T16:35:21.352678+0000 mon.c (mon.2) 774 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: audit 2024-04-26T16:35:21.376289+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:21 smithi190 bash[23868]: audit 2024-04-26T16:35:21.376289+0000 mon.a (mon.0) 3549 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:22.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:22.160 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:22.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:22.290 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:22.393 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:22.418 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:22.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:22.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:22.548 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:22.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:22.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:22.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:22.807 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:22.910 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:22.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:22.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:23.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:23.062 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:23.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:23.194 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:23.297 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:23.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:23.324 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[92888]: debug 2024-04-26T16:35:23.040+0000 7f207b3d77c0 -1 osd.6 627 log_to_monitors true 2024-04-26T16:35:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:23.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.821683+0000 osd.2 (osd.2) 365 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.821683+0000 osd.2 (osd.2) 365 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.825663+0000 osd.2 (osd.2) 366 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.825663+0000 osd.2 (osd.2) 366 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.892791+0000 mgr.y (mgr.24401) 692 : cluster [DBG] pgmap v1154: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:21.892791+0000 mgr.y (mgr.24401) 692 : cluster [DBG] pgmap v1154: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:22.348730+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:22.348730+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.358942+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e744: 8 total, 7 up, 8 in 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.358942+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e744: 8 total, 7 up, 8 in 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:22.367409+0000 mon.c (mon.2) 775 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.437 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:22.367409+0000 mon.c (mon.2) 775 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:22.380104+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:22.380104+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.678598+0000 mon.a (mon.0) 3553 : cluster [WRN] Health check update: Degraded data redundancy: 83/648 objects degraded (12.809%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.678598+0000 mon.a (mon.0) 3553 : cluster [WRN] Health check update: Degraded data redundancy: 83/648 objects degraded (12.809%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.752999+0000 osd.0 (osd.0) 425 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.821683+0000 osd.2 (osd.2) 365 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.821683+0000 osd.2 (osd.2) 365 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.825663+0000 osd.2 (osd.2) 366 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:35:23.438 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.825663+0000 osd.2 (osd.2) 366 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.892791+0000 mgr.y (mgr.24401) 692 : cluster [DBG] pgmap v1154: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:21.892791+0000 mgr.y (mgr.24401) 692 : cluster [DBG] pgmap v1154: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:22.348730+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:22.348730+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.358942+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e744: 8 total, 7 up, 8 in 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.358942+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e744: 8 total, 7 up, 8 in 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:22.367409+0000 mon.c (mon.2) 775 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:22.367409+0000 mon.c (mon.2) 775 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:22.380104+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:22.380104+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.678598+0000 mon.a (mon.0) 3553 : cluster [WRN] Health check update: Degraded data redundancy: 83/648 objects degraded (12.809%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.678598+0000 mon.a (mon.0) 3553 : cluster [WRN] Health check update: Degraded data redundancy: 83/648 objects degraded (12.809%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:23.439 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.752999+0000 osd.0 (osd.0) 425 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.752999+0000 osd.0 (osd.0) 425 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.755388+0000 osd.0 (osd.0) 426 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: cluster 2024-04-26T16:35:22.755388+0000 osd.0 (osd.0) 426 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:23.047155+0000 mon.b (mon.1) 441 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:23.047155+0000 mon.b (mon.1) 441 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:23.048218+0000 mon.a (mon.0) 3554 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:23.048218+0000 mon.a (mon.0) 3554 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:23.182633+0000 mon.c (mon.2) 776 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[29234]: audit 2024-04-26T16:35:23.182633+0000 mon.c (mon.2) 776 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:23.440 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.752999+0000 osd.0 (osd.0) 425 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.755388+0000 osd.0 (osd.0) 426 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: cluster 2024-04-26T16:35:22.755388+0000 osd.0 (osd.0) 426 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:23.047155+0000 mon.b (mon.1) 441 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:23.047155+0000 mon.b (mon.1) 441 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:23.048218+0000 mon.a (mon.0) 3554 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:23.048218+0000 mon.a (mon.0) 3554 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:23.182633+0000 mon.c (mon.2) 776 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:23.441 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:23 smithi050 bash[21463]: audit 2024-04-26T16:35:23.182633+0000 mon.c (mon.2) 776 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:23.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:23.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:23.573 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:23.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:23.697 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:23.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:23.808 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[92888]: debug 2024-04-26T16:35:23.540+0000 7f206c82f700 -1 osd.6 745 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.821683+0000 osd.2 (osd.2) 365 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.821683+0000 osd.2 (osd.2) 365 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.825663+0000 osd.2 (osd.2) 366 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.825663+0000 osd.2 (osd.2) 366 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.892791+0000 mgr.y (mgr.24401) 692 : cluster [DBG] pgmap v1154: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:21.892791+0000 mgr.y (mgr.24401) 692 : cluster [DBG] pgmap v1154: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 730 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:22.348730+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:22.348730+0000 mon.a (mon.0) 3550 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.358942+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e744: 8 total, 7 up, 8 in 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.358942+0000 mon.a (mon.0) 3551 : cluster [DBG] osdmap e744: 8 total, 7 up, 8 in 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:22.367409+0000 mon.c (mon.2) 775 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.809 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:22.367409+0000 mon.c (mon.2) 775 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:22.380104+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:22.380104+0000 mon.a (mon.0) 3552 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.678598+0000 mon.a (mon.0) 3553 : cluster [WRN] Health check update: Degraded data redundancy: 83/648 objects degraded (12.809%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.678598+0000 mon.a (mon.0) 3553 : cluster [WRN] Health check update: Degraded data redundancy: 83/648 objects degraded (12.809%), 20 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.752999+0000 osd.0 (osd.0) 425 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.752999+0000 osd.0 (osd.0) 425 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.755388+0000 osd.0 (osd.0) 426 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: cluster 2024-04-26T16:35:22.755388+0000 osd.0 (osd.0) 426 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:23.047155+0000 mon.b (mon.1) 441 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:23.047155+0000 mon.b (mon.1) 441 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:23.048218+0000 mon.a (mon.0) 3554 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:23.048218+0000 mon.a (mon.0) 3554 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:23.182633+0000 mon.c (mon.2) 776 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:23.810 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:23 smithi190 bash[23868]: audit 2024-04-26T16:35:23.182633+0000 mon.c (mon.2) 776 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:35:23.819 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:23.824 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-04-26T16:35:23.922 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:23.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:23.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:24.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:24.068 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:24.171 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:24.198 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:24.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:24.330 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:24.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:24.444 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:35:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.318121+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.318121+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.319471+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.319471+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.350553+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.350553+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.350658+0000 mon.a (mon.0) 3556 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.350658+0000 mon.a (mon.0) 3556 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.355342+0000 mon.b (mon.1) 442 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.355342+0000 mon.b (mon.1) 442 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.445 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.356530+0000 mon.c (mon.2) 777 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.356530+0000 mon.c (mon.2) 777 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.356769+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e745: 8 total, 7 up, 8 in 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.356769+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e745: 8 total, 7 up, 8 in 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.358358+0000 mon.a (mon.0) 3558 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.358358+0000 mon.a (mon.0) 3558 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.358878+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.318121+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.318121+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:35:24.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.319471+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.319471+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.350553+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.350553+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.350658+0000 mon.a (mon.0) 3556 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.350658+0000 mon.a (mon.0) 3556 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.355342+0000 mon.b (mon.1) 442 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.355342+0000 mon.b (mon.1) 442 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.356530+0000 mon.c (mon.2) 777 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.356530+0000 mon.c (mon.2) 777 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.356769+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e745: 8 total, 7 up, 8 in 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.356769+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e745: 8 total, 7 up, 8 in 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.358358+0000 mon.a (mon.0) 3558 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.447 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.358358+0000 mon.a (mon.0) 3558 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.358878+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.358878+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.861598+0000 mon.c (mon.2) 778 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.861598+0000 mon.c (mon.2) 778 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.862893+0000 mon.c (mon.2) 779 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.862893+0000 mon.c (mon.2) 779 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.869090+0000 mon.a (mon.0) 3560 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:23.869090+0000 mon.a (mon.0) 3560 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.352007+0000 mon.a (mon.0) 3561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:35:24.448 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.352007+0000 mon.a (mon.0) 3561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:24.354423+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: audit 2024-04-26T16:35:24.354423+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.362310+0000 mon.a (mon.0) 3563 : cluster [INF] osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] boot 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.362310+0000 mon.a (mon.0) 3563 : cluster [INF] osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] boot 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.362455+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.362455+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.358878+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.861598+0000 mon.c (mon.2) 778 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.861598+0000 mon.c (mon.2) 778 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.862893+0000 mon.c (mon.2) 779 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.862893+0000 mon.c (mon.2) 779 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:24.449 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.869090+0000 mon.a (mon.0) 3560 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:23.869090+0000 mon.a (mon.0) 3560 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.352007+0000 mon.a (mon.0) 3561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.352007+0000 mon.a (mon.0) 3561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:24.354423+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: audit 2024-04-26T16:35:24.354423+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.362310+0000 mon.a (mon.0) 3563 : cluster [INF] osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] boot 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.362310+0000 mon.a (mon.0) 3563 : cluster [INF] osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] boot 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.362455+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-04-26T16:35:24.450 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:24 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.362455+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-04-26T16:35:24.459 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:24.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:24.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:24.588 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:24.590 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.318121+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.318121+0000 osd.3 (osd.3) 427 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.319471+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.319471+0000 osd.3 (osd.3) 428 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.350553+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.350553+0000 mon.a (mon.0) 3555 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.350658+0000 mon.a (mon.0) 3556 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.350658+0000 mon.a (mon.0) 3556 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.355342+0000 mon.b (mon.1) 442 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.355342+0000 mon.b (mon.1) 442 : audit [INF] from='osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.356530+0000 mon.c (mon.2) 777 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.356530+0000 mon.c (mon.2) 777 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.356769+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e745: 8 total, 7 up, 8 in 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.356769+0000 mon.a (mon.0) 3557 : cluster [DBG] osdmap e745: 8 total, 7 up, 8 in 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.358358+0000 mon.a (mon.0) 3558 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.358358+0000 mon.a (mon.0) 3558 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.358878+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.358878+0000 mon.a (mon.0) 3559 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.861598+0000 mon.c (mon.2) 778 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.861598+0000 mon.c (mon.2) 778 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.862893+0000 mon.c (mon.2) 779 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.862893+0000 mon.c (mon.2) 779 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.869090+0000 mon.a (mon.0) 3560 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:23.869090+0000 mon.a (mon.0) 3560 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.352007+0000 mon.a (mon.0) 3561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.352007+0000 mon.a (mon.0) 3561 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-26T16:35:24.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:24.354423+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:24.593 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: audit 2024-04-26T16:35:24.354423+0000 mon.a (mon.0) 3562 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:24.593 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.362310+0000 mon.a (mon.0) 3563 : cluster [INF] osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] boot 2024-04-26T16:35:24.593 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.362310+0000 mon.a (mon.0) 3563 : cluster [INF] osd.6 [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] boot 2024-04-26T16:35:24.593 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.362455+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-04-26T16:35:24.593 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:24 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.362455+0000 mon.a (mon.0) 3564 : cluster [DBG] osdmap e746: 8 total, 8 up, 8 in 2024-04-26T16:35:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:24.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:24.720 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:24.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:24.849 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:24.952 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:24.980 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:24.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:24.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:25.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:25.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:25.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:25.244 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:25.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:25.347 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:25.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:25.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:25.503 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.585929+0000 osd.5 (osd.5) 285 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.585929+0000 osd.5 (osd.5) 285 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.593268+0000 osd.5 (osd.5) 286 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.593268+0000 osd.5 (osd.5) 286 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.770855+0000 osd.1 (osd.1) 395 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.770855+0000 osd.1 (osd.1) 395 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.773109+0000 osd.1 (osd.1) 396 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.773109+0000 osd.1 (osd.1) 396 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.893594+0000 mgr.y (mgr.24401) 693 : cluster [DBG] pgmap v1157: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.893594+0000 mgr.y (mgr.24401) 693 : cluster [DBG] pgmap v1157: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: audit 2024-04-26T16:35:24.372671+0000 mon.c (mon.2) 780 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: audit 2024-04-26T16:35:24.372671+0000 mon.c (mon.2) 780 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: audit 2024-04-26T16:35:24.373763+0000 mon.c (mon.2) 781 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.504 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: audit 2024-04-26T16:35:24.373763+0000 mon.c (mon.2) 781 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: audit 2024-04-26T16:35:24.408191+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: audit 2024-04-26T16:35:24.408191+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.867157+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.867157+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.354998+0000 mon.a (mon.0) 3567 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:25.505 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:25 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.354998+0000 mon.a (mon.0) 3567 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:25.515 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:25.532 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:35:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:35:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:35:25.532 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:35:25.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.585929+0000 osd.5 (osd.5) 285 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.585929+0000 osd.5 (osd.5) 285 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.593268+0000 osd.5 (osd.5) 286 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.593268+0000 osd.5 (osd.5) 286 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.770855+0000 osd.1 (osd.1) 395 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.770855+0000 osd.1 (osd.1) 395 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.773109+0000 osd.1 (osd.1) 396 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.773109+0000 osd.1 (osd.1) 396 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:35:25.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.893594+0000 mgr.y (mgr.24401) 693 : cluster [DBG] pgmap v1157: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.893594+0000 mgr.y (mgr.24401) 693 : cluster [DBG] pgmap v1157: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: audit 2024-04-26T16:35:24.372671+0000 mon.c (mon.2) 780 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: audit 2024-04-26T16:35:24.372671+0000 mon.c (mon.2) 780 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: audit 2024-04-26T16:35:24.373763+0000 mon.c (mon.2) 781 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: audit 2024-04-26T16:35:24.373763+0000 mon.c (mon.2) 781 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: audit 2024-04-26T16:35:24.408191+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: audit 2024-04-26T16:35:24.408191+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.867157+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.867157+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.354998+0000 mon.a (mon.0) 3567 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.354998+0000 mon.a (mon.0) 3567 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:25.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.585929+0000 osd.5 (osd.5) 285 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.585929+0000 osd.5 (osd.5) 285 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.593268+0000 osd.5 (osd.5) 286 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.593268+0000 osd.5 (osd.5) 286 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.770855+0000 osd.1 (osd.1) 395 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.770855+0000 osd.1 (osd.1) 395 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.773109+0000 osd.1 (osd.1) 396 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.773109+0000 osd.1 (osd.1) 396 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.893594+0000 mgr.y (mgr.24401) 693 : cluster [DBG] pgmap v1157: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.893594+0000 mgr.y (mgr.24401) 693 : cluster [DBG] pgmap v1157: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s; 83/648 objects degraded (12.809%) 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: audit 2024-04-26T16:35:24.372671+0000 mon.c (mon.2) 780 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: audit 2024-04-26T16:35:24.372671+0000 mon.c (mon.2) 780 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: audit 2024-04-26T16:35:24.373763+0000 mon.c (mon.2) 781 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: audit 2024-04-26T16:35:24.373763+0000 mon.c (mon.2) 781 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: audit 2024-04-26T16:35:24.408191+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: audit 2024-04-26T16:35:24.408191+0000 mon.a (mon.0) 3565 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]: dispatch 2024-04-26T16:35:25.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.867157+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:25.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.867157+0000 mon.a (mon.0) 3566 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:25.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.354998+0000 mon.a (mon.0) 3567 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:25.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:25 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.354998+0000 mon.a (mon.0) 3567 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:25.639 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/2528794391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 msgr2=0x7f694c109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:25.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- 172.21.15.50:0/2528794391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c109890 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f6948009990 tx=0x7f6948009ca0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/2528794391 shutdown_connections 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- 172.21.15.50:0/2528794391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f694c1023b0 0x7f694c1102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- 172.21.15.50:0/2528794391 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c109890 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- 172.21.15.50:0/2528794391 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c101a10 0x7f694c101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/2528794391 >> 172.21.15.50:0/2528794391 conn(0x7f694c078990 msgr2=0x7f694c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/2528794391 shutdown_connections 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/2528794391 wait complete. 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 librados: starting msgr at 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 librados: starting objecter 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 Processor -- start 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- start start 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 librados: setting wanted keys 2024-04-26T16:35:25.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 librados: calling monclient init 2024-04-26T16:35:25.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f694c101a10 0x7f694c1990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 0x7f694c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c19d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c19d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6950ef5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 0x7f694c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c19d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35656/0 (socket says 172.21.15.50:35656) 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6950ef5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 0x7f694c199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50442/0 (socket says 172.21.15.50:50442) 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 -- 172.21.15.50:0/3899871697 learned_addr learned my addr 172.21.15.50:0/3899871697 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f694c112a90 con 0x7f694c1023b0 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f694c112910 con 0x7f694c101a10 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f694c112c10 con 0x7f694c1094b0 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f69516f6700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f694c101a10 0x7f694c1990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 -- 172.21.15.50:0/3899871697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f694c101a10 msgr2=0x7f694c1990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f694c101a10 0x7f694c1990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 -- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 msgr2=0x7f694c199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 0x7f694c199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694c19e100 con 0x7f694c1094b0 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6950ef5700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 0x7f694c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f6951ef7700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c19d9c0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f6944007d70 tx=0x7f694400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f69427fc700 10 client.?.objecter ms_handle_connect 0x7f694c1094b0 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f69427fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6944016020 con 0x7f694c1094b0 2024-04-26T16:35:25.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.683+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f694c19e3f0 con 0x7f694c1094b0 2024-04-26T16:35:25.688 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f695395a700 10 client.41594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f694c102b20 con 0x7f694c1094b0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f695395a700 1 librados: init done 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f694400ac90 con 0x7f694c1094b0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6944011380 con 0x7f694c1094b0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f694c06a110 con 0x7f694c1094b0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6944019020 con 0x7f694c1094b0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6938071e50 0x7f6938074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69516f6700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6938071e50 0x7f6938074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(747..747 src has 577..747) ==== 9947+0+0 (secure 0 0 0) 0x7f6944091a00 con 0x7f694c1094b0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 10 client.41594.objecter ms_dispatch 0x7f694c001550 osd_map(747..747 src has 577..747) 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 3 client.41594.objecter handle_osd_map got epochs [747,747] > 0 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 3 client.41594.objecter handle_osd_map decoding full epoch 747 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 20 client.41594.objecter dump_active .. 0 homeless 2024-04-26T16:35:25.689 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69427fc700 10 client.41594.objecter ms_handle_connect 0x7f6938071e50 2024-04-26T16:35:25.690 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.687+0000 7f69516f6700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6938071e50 0x7f6938074310 secure :-1 s=READY pgs=3824 cs=0 l=1 rev1=1 crypto rx=0x7f694800b2e0 tx=0x7f6948005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.695+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6944020020 con 0x7f694c1094b0 2024-04-26T16:35:25.741 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:25.761 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:25.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/2781703491 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b406b130 msgr2=0x7ff2b406b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:25.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/2781703491 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b406b130 0x7ff2b406b590 secure :-1 s=READY pgs=2735 cs=0 l=1 rev1=1 crypto rx=0x7ff2a80098f0 tx=0x7ff2a8009c00 comp rx=0 tx=0).stop 2024-04-26T16:35:25.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/2781703491 shutdown_connections 2024-04-26T16:35:25.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/2781703491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b41094a0 0x7ff2b4109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/2781703491 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b41102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/2781703491 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b406b130 0x7ff2b406b590 unknown :-1 s=CLOSED pgs=2735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/2781703491 >> 172.21.15.50:0/2781703491 conn(0x7ff2b40792d0 msgr2=0x7ff2b40796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/2781703491 shutdown_connections 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/2781703491 wait complete. 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 librados: starting msgr at 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.823+0000 7ff2b8ea5700 1 librados: starting objecter 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 Processor -- start 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- start start 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 librados: setting wanted keys 2024-04-26T16:35:25.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 librados: calling monclient init 2024-04-26T16:35:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b406b130 0x7ff2b4199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b4199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b41094a0 0x7ff2b419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b4112ae0 con 0x7ff2b406bad0 2024-04-26T16:35:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff2b4112960 con 0x7ff2b406b130 2024-04-26T16:35:25.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff2b4112c60 con 0x7ff2b41094a0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b4199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b4199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50456/0 (socket says 172.21.15.50:50456) 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 -- 172.21.15.50:0/728893143 learned_addr learned my addr 172.21.15.50:0/728893143 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b41094a0 0x7ff2b419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b406b130 0x7ff2b4199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b41094a0 msgr2=0x7ff2b419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff2b41094a0 0x7ff2b419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b406b130 msgr2=0x7ff2b4199220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b406b130 0x7ff2b4199220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2b419e230 con 0x7ff2b406bad0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b4199760 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009230 tx=0x7ff2a800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2a37fe700 10 client.?.objecter ms_handle_connect 0x7ff2b406bad0 2024-04-26T16:35:25.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2a37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8025070 con 0x7ff2b406bad0 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2b419e4c0 con 0x7ff2b406bad0 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff2a8005050 con 0x7ff2b406bad0 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 10 client.30981.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2b41a5da0 con 0x7ff2b406bad0 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 librados: init done 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8004ab0 con 0x7ff2b406bad0 2024-04-26T16:35:25.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff2b406a110 con 0x7ff2b406bad0 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.827+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff2a8028070 con 0x7ff2b406bad0 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff29c071f80 0x7ff29c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(747..747 src has 577..747) ==== 9947+0+0 (secure 0 0 0) 0x7ff2a8092bc0 con 0x7ff2b406bad0 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 osd_map(747..747 src has 577..747) 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 3 client.30981.objecter handle_osd_map got epochs [747,747] > 0 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 3 client.30981.objecter handle_osd_map decoding full epoch 747 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff29c071f80 0x7ff29c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 20 client.30981.objecter dump_active .. 0 homeless 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v747) ==== 54+0+16 (secure 0 0 0) 0x7ff2a802f070 con 0x7ff2b406bad0 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 1 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff2b4064030 0x7ff2b4064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff29c071f80 0x7ff29c074440 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4005950 tx=0x7ff2a400a300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 1 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 0 1 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 4 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 1 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff2b41a9100 con 0x7ff2b4064030 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff29c071f80 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff2b4064030 0x7ff2b4064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff2b4064030 0x7ff2b4064410 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b4064030 2024-04-26T16:35:25.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b41a9100 con 0x7ff2b4064030 2024-04-26T16:35:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 1: 0 ) 2024-04-26T16:35:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 1 = system:0 2024-04-26T16:35:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 4 2024-04-26T16:35:25.834 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.831+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 0 1 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 2 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 2 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b419a580 osd=0 3 2024-04-26T16:35:25.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 0 2 2024-04-26T16:35:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 4 2024-04-26T16:35:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 2 2024-04-26T16:35:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b4063ab0 con 0x7ff2b4064030 2024-04-26T16:35:25.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b4063ab0 con 0x7ff2b4064030 2024-04-26T16:35:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 4 2024-04-26T16:35:25.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 0 2 2024-04-26T16:35:25.852 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff2b402d310 0x7ff2b402d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 1 3 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 4 2024-04-26T16:35:25.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 3 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff2b410b0e0 con 0x7ff2b402d310 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff2b402d310 0x7ff2b402d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b402d310 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff2b402d310 0x7ff2b402d6f0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b410b0e0 con 0x7ff2b402d310 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 3: 0 ) 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 3 = system:0 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 4 2024-04-26T16:35:25.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.851+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 1 3 2024-04-26T16:35:25.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 4 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 4 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b410aa60 osd=1 3 2024-04-26T16:35:25.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 1 4 2024-04-26T16:35:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 4 2024-04-26T16:35:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 4 2024-04-26T16:35:25.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.871+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b410a4e0 con 0x7ff2b402d310 2024-04-26T16:35:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b410a4e0 con 0x7ff2b402d310 2024-04-26T16:35:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 4 2024-04-26T16:35:25.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 1 4 2024-04-26T16:35:25.874 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 5 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff2b41ab2c0 0x7ff2b41ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 5 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 2 5 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 4 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 5 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff2b41b0510 con 0x7ff2b41ab2c0 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff2b41ab2c0 0x7ff2b41ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff2b41ab2c0 0x7ff2b41ad6b0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b41ab2c0 2024-04-26T16:35:25.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b41b0510 con 0x7ff2b41ab2c0 2024-04-26T16:35:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 5: 0 ) 2024-04-26T16:35:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 5 = system:0 2024-04-26T16:35:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 4 2024-04-26T16:35:25.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.875+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 2 5 2024-04-26T16:35:25.884 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:25.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 6 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 6 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b410a8e0 osd=2 3 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 2 6 2024-04-26T16:35:25.907 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 4 2024-04-26T16:35:25.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 6 2024-04-26T16:35:25.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b402de80 con 0x7ff2b41ab2c0 2024-04-26T16:35:25.908 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b402de80 con 0x7ff2b41ab2c0 2024-04-26T16:35:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 4 2024-04-26T16:35:25.909 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 2 6 2024-04-26T16:35:25.909 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 7 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff2b41a8c60 0x7ff2b41aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 7 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 3 7 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 4 2024-04-26T16:35:25.910 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 7 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff2b41b3e70 con 0x7ff2b41a8c60 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.907+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff2b41a8c60 0x7ff2b41aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b41a8c60 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff2b41a8c60 0x7ff2b41aed70 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b41b3e70 con 0x7ff2b41a8c60 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 7: 0 ) 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 7 = system:0 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 4 2024-04-26T16:35:25.911 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.911+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 3 7 2024-04-26T16:35:25.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:25.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 8 2024-04-26T16:35:25.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 8 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b402e910 osd=3 3 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 3 8 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 4 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 8 2024-04-26T16:35:25.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.927+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b41a8230 con 0x7ff2b41a8c60 2024-04-26T16:35:25.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b41a8230 con 0x7ff2b41a8c60 2024-04-26T16:35:25.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:25.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:25.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 4 2024-04-26T16:35:25.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 3 8 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 9 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff2b41ae8d0 0x7ff2b41b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 9 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 4 9 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 4 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 9 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff2b41a84c0 con 0x7ff2b41ae8d0 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff2b41ae8d0 0x7ff2b41b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b41ae8d0 2024-04-26T16:35:25.932 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.931+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff2b41ae8d0 0x7ff2b41b26d0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.935+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b41a84c0 con 0x7ff2b41ae8d0 2024-04-26T16:35:25.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.935+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 9: 0 ) 2024-04-26T16:35:25.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.935+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 9 = system:0 2024-04-26T16:35:25.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.935+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 4 2024-04-26T16:35:25.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.935+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 4 9 2024-04-26T16:35:25.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:25.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 10 2024-04-26T16:35:25.951 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 10 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b41a8230 osd=4 3 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 4 10 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 4 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 10 2024-04-26T16:35:25.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.951+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b4000910 con 0x7ff2b41ae8d0 2024-04-26T16:35:25.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b4000910 con 0x7ff2b41ae8d0 2024-04-26T16:35:25.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:25.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:25.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 4 2024-04-26T16:35:25.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 4 10 2024-04-26T16:35:25.956 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 11 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff2b41b3120 0x7ff2b41b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 11 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 5 11 2024-04-26T16:35:25.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 4 2024-04-26T16:35:25.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 11 2024-04-26T16:35:25.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff2b41ba960 con 0x7ff2b41b3120 2024-04-26T16:35:25.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff2b41b3120 0x7ff2b41b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b41b3120 2024-04-26T16:35:25.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.955+0000 7ff2b1d9b700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff2b41b3120 0x7ff2b41b3580 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.959+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f694c19a450 con 0x7f694c1094b0 2024-04-26T16:35:25.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.959+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b41ba960 con 0x7ff2b41b3120 2024-04-26T16:35:25.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.959+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 11: 0 ) 2024-04-26T16:35:25.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.959+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 11 = system:0 2024-04-26T16:35:25.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.959+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 4 2024-04-26T16:35:25.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.959+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 5 11 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 12 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 12 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b40008d0 osd=5 3 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 5 12 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 4 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 12 2024-04-26T16:35:25.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.975+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b4102320 con 0x7ff2b41b3120 2024-04-26T16:35:25.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b4102320 con 0x7ff2b41b3120 2024-04-26T16:35:25.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:25.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:25.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 4 2024-04-26T16:35:25.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 5 12 2024-04-26T16:35:25.981 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 13 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff2b4102a40 0x7ff2b4102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 13 2024-04-26T16:35:25.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 6 13 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 4 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 13 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.979+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff2b4103560 con 0x7ff2b4102a40 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff2b4102a40 0x7ff2b4102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b4102a40 2024-04-26T16:35:25.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2b2d9d700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff2b4102a40 0x7ff2b4102ec0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:25.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b4103560 con 0x7ff2b4102a40 2024-04-26T16:35:25.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 13: 0 ) 2024-04-26T16:35:25.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 13 = system:0 2024-04-26T16:35:25.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 4 2024-04-26T16:35:25.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:25.983+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 6 13 2024-04-26T16:35:25.987 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 14 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 14 2024-04-26T16:35:26.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b4102320 osd=6 3 2024-04-26T16:35:26.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 6 14 2024-04-26T16:35:26.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 4 2024-04-26T16:35:26.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 14 2024-04-26T16:35:26.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.003+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b4103560 con 0x7ff2b4102a40 2024-04-26T16:35:26.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b4103560 con 0x7ff2b4102a40 2024-04-26T16:35:26.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:26.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:26.006 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 4 2024-04-26T16:35:26.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 6 14 2024-04-26T16:35:26.007 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:35:26.007 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 15 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff2b4103c70 0x7ff2b41c0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 15 2024-04-26T16:35:26.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 7 15 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 4 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 15 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff2b4104220 con 0x7ff2b4103c70 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff2b4103c70 0x7ff2b41c0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2a37fe700 10 client.30981.objecter ms_handle_connect 0x7ff2b4103c70 2024-04-26T16:35:26.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.007+0000 7ff2b259c700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff2b4103c70 0x7ff2b41c0470 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:26.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.011+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff2b4104220 con 0x7ff2b4103c70 2024-04-26T16:35:26.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.011+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 15: 0 ) 2024-04-26T16:35:26.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.011+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 15 = system:0 2024-04-26T16:35:26.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.011+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 4 2024-04-26T16:35:26.010 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.011+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 7 15 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 10 client.30981.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign -1 16 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 20 client.30981.objecter _get_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_remove -1 16 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 20 client.30981.objecter get_session s=0x7ff2b4103560 osd=7 3 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 15 client.30981.objecter _session_command_op_assign 7 16 2024-04-26T16:35:26.028 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 4 2024-04-26T16:35:26.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 10 client.30981.objecter _send_command 16 2024-04-26T16:35:26.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.027+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7ff2b41c09b0 con 0x7ff2b4103c70 2024-04-26T16:35:26.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2a37fe700 1 -- 172.21.15.50:0/728893143 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff2b41c09b0 con 0x7ff2b4103c70 2024-04-26T16:35:26.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2a37fe700 10 client.30981.objecter ms_dispatch 0x7ff2b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:26.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2a37fe700 10 client.30981.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:26.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2a37fe700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 4 2024-04-26T16:35:26.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2a37fe700 15 client.30981.objecter _session_command_op_remove 7 16 2024-04-26T16:35:26.033 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:35:26.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 librados: watch_flush enter 2024-04-26T16:35:26.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 librados: watch_flush exit 2024-04-26T16:35:26.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.0 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff2b4064030 msgr2=0x7ff2b4064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff2b4064030 0x7ff2b4064410 crc :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b419a580 osd=0 1 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.1 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff2b402d310 msgr2=0x7ff2b402d6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff2b402d310 0x7ff2b402d6f0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410aa60 osd=1 1 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.2 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff2b41ab2c0 msgr2=0x7ff2b41ad6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff2b41ab2c0 0x7ff2b41ad6b0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b410a8e0 osd=2 1 2024-04-26T16:35:26.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.3 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff2b41a8c60 msgr2=0x7ff2b41aed70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff2b41a8c60 0x7ff2b41aed70 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b402e910 osd=3 1 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.4 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff2b41ae8d0 msgr2=0x7ff2b41b26d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff2b41ae8d0 0x7ff2b41b26d0 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b41a8230 osd=4 1 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.5 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff2b41b3120 msgr2=0x7ff2b41b3580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff2b41b3120 0x7ff2b41b3580 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b40008d0 osd=5 1 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.6 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff2b4102a40 msgr2=0x7ff2b4102ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff2b4102a40 0x7ff2b4102ec0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4102320 osd=6 1 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter close_session for osd.7 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff2b4103c70 msgr2=0x7ff2b41c0470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff2b4103c70 0x7ff2b41c0470 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter put_session s=0x7ff2b4103560 osd=7 1 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 20 client.30981.objecter shutdown clearing up homeless session... 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 10 client.30981.objecter successfully canceled tick 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b2d9d700 1 -- 172.21.15.50:0/728893143 reap_dead start 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b2d9d700 1 -- 172.21.15.50:0/728893143 reap_dead start 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b2d9d700 1 -- 172.21.15.50:0/728893143 reap_dead start 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b2d9d700 1 -- 172.21.15.50:0/728893143 reap_dead start 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b2d9d700 1 -- 172.21.15.50:0/728893143 reap_dead start 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff29c071f80 msgr2=0x7ff29c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff29c071f80 0x7ff29c074440 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4005950 tx=0x7ff2a400a300 comp rx=0 tx=0).stop 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 msgr2=0x7ff2b4199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b4199760 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7ff2a8009230 tx=0x7ff2a800fa00 comp rx=0 tx=0).stop 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 shutdown_connections 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff2b406bad0 0x7ff2b4199760 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 --2- 172.21.15.50:0/728893143 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff29c071f80 0x7ff29c074440 unknown :-1 s=CLOSED pgs=3825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.031+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 >> 172.21.15.50:0/728893143 conn(0x7ff2b40792d0 msgr2=0x7ff2b4100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.035+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 shutdown_connections 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.035+0000 7ff2b8ea5700 1 -- 172.21.15.50:0/728893143 wait complete. 2024-04-26T16:35:26.038 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.035+0000 7ff2b8ea5700 1 librados: shutdown 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:35:26.039 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:35:26.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:26.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:26.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:26.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:26.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:26.386 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:26.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.395+0000 7f69427fc700 1 -- 172.21.15.50:0/3899871697 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v748) ==== 87+0+0 (secure 0 0 0) 0x7f694405d200 con 0x7f694c1094b0 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 10 librados: watch_flush enter 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 10 librados: watch_flush exit 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 20 client.41594.objecter shutdown clearing up homeless session... 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 10 client.41594.objecter successfully canceled tick 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6938071e50 msgr2=0x7f6938074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6938071e50 0x7f6938074310 secure :-1 s=READY pgs=3824 cs=0 l=1 rev1=1 crypto rx=0x7f694800b2e0 tx=0x7f6948005960 comp rx=0 tx=0).stop 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 msgr2=0x7f694c19d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:26.399 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c19d9c0 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f6944007d70 tx=0x7f694400a340 comp rx=0 tx=0).stop 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 shutdown_connections 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6938071e50 0x7f6938074310 unknown :-1 s=CLOSED pgs=3824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f694c101a10 0x7f694c1990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f694c1094b0 0x7f694c19d9c0 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 --2- 172.21.15.50:0/3899871697 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f694c1023b0 0x7f694c199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 >> 172.21.15.50:0/3899871697 conn(0x7f694c078990 msgr2=0x7f694c105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 shutdown_connections 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 -- 172.21.15.50:0/3899871697 wait complete. 2024-04-26T16:35:26.400 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:26.399+0000 7f695395a700 1 librados: shutdown 2024-04-26T16:35:26.401 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:35:26.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-26T16:35:26.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:26.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.998556+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:26.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.998556+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:26.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.999366+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:26.495 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:23.999366+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.554295+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.554295+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.556368+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.556368+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.768129+0000 osd.2 (osd.2) 367 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.768129+0000 osd.2 (osd.2) 367 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.770525+0000 osd.2 (osd.2) 368 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:24.770525+0000 osd.2 (osd.2) 368 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.327172+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.327172+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.329419+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.329419+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:35:26.496 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.358018+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]': finished 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.358018+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]': finished 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.368325+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.368325+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.833906+0000 mon.a (mon.0) 3570 : audit [DBG] from='client.? 172.21.15.50:0/728893143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.833906+0000 mon.a (mon.0) 3570 : audit [DBG] from='client.? 172.21.15.50:0/728893143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.964565+0000 mon.c (mon.2) 782 : audit [INF] from='client.? 172.21.15.50:0/3899871697' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.964565+0000 mon.c (mon.2) 782 : audit [INF] from='client.? 172.21.15.50:0/3899871697' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.964961+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[29234]: audit 2024-04-26T16:35:25.964961+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.497 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.998556+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.998556+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.999366+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:23.999366+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.554295+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.554295+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.556368+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.556368+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.768129+0000 osd.2 (osd.2) 367 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.768129+0000 osd.2 (osd.2) 367 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.770525+0000 osd.2 (osd.2) 368 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:24.770525+0000 osd.2 (osd.2) 368 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.327172+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.327172+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.329419+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:35:26.498 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.329419+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.358018+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]': finished 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.358018+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]': finished 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.368325+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.368325+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.833906+0000 mon.a (mon.0) 3570 : audit [DBG] from='client.? 172.21.15.50:0/728893143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.833906+0000 mon.a (mon.0) 3570 : audit [DBG] from='client.? 172.21.15.50:0/728893143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.964565+0000 mon.c (mon.2) 782 : audit [INF] from='client.? 172.21.15.50:0/3899871697' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.964565+0000 mon.c (mon.2) 782 : audit [INF] from='client.? 172.21.15.50:0/3899871697' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.964961+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.499 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:26 smithi050 bash[21463]: audit 2024-04-26T16:35:25.964961+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.554 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:26.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:26.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.998556+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:26.671 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.998556+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.999366+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:23.999366+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.554295+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.554295+0000 osd.5 (osd.5) 287 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.556368+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.556368+0000 osd.5 (osd.5) 288 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.768129+0000 osd.2 (osd.2) 367 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.768129+0000 osd.2 (osd.2) 367 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.770525+0000 osd.2 (osd.2) 368 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:24.770525+0000 osd.2 (osd.2) 368 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.327172+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.327172+0000 osd.3 (osd.3) 429 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.329419+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:35:26.672 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.329419+0000 osd.3 (osd.3) 430 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.358018+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]': finished 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.358018+0000 mon.a (mon.0) 3568 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-128", "mode": "writeback"}]': finished 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.368325+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.368325+0000 mon.a (mon.0) 3569 : cluster [DBG] osdmap e747: 8 total, 8 up, 8 in 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.833906+0000 mon.a (mon.0) 3570 : audit [DBG] from='client.? 172.21.15.50:0/728893143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.833906+0000 mon.a (mon.0) 3570 : audit [DBG] from='client.? 172.21.15.50:0/728893143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.964565+0000 mon.c (mon.2) 782 : audit [INF] from='client.? 172.21.15.50:0/3899871697' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.964565+0000 mon.c (mon.2) 782 : audit [INF] from='client.? 172.21.15.50:0/3899871697' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.964961+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:26 smithi190 bash[23868]: audit 2024-04-26T16:35:25.964961+0000 mon.a (mon.0) 3571 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:35:26.685 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:26.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:26.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:26.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:26.919 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:26.946 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:26.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:27.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:27.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:27.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:27.206 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:27.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:27.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:27.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:27.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:27.451 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:35:27.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.798740+0000 osd.2 (osd.2) 369 : cluster [DBG] 11.d scrub starts 2024-04-26T16:35:27.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.798740+0000 osd.2 (osd.2) 369 : cluster [DBG] 11.d scrub starts 2024-04-26T16:35:27.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.801851+0000 osd.2 (osd.2) 370 : cluster [DBG] 11.d scrub ok 2024-04-26T16:35:27.451 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.801851+0000 osd.2 (osd.2) 370 : cluster [DBG] 11.d scrub ok 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.894352+0000 mgr.y (mgr.24401) 694 : cluster [DBG] pgmap v1160: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 83/648 objects degraded (12.809%) 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:25.894352+0000 mgr.y (mgr.24401) 694 : cluster [DBG] pgmap v1160: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 83/648 objects degraded (12.809%) 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.386679+0000 mon.a (mon.0) 3572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.386679+0000 mon.a (mon.0) 3572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: audit 2024-04-26T16:35:26.394033+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: audit 2024-04-26T16:35:26.394033+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.400527+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-04-26T16:35:27.452 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:27 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.400527+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-04-26T16:35:27.466 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:27.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:27.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.798740+0000 osd.2 (osd.2) 369 : cluster [DBG] 11.d scrub starts 2024-04-26T16:35:27.580 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.798740+0000 osd.2 (osd.2) 369 : cluster [DBG] 11.d scrub starts 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.801851+0000 osd.2 (osd.2) 370 : cluster [DBG] 11.d scrub ok 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.801851+0000 osd.2 (osd.2) 370 : cluster [DBG] 11.d scrub ok 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.894352+0000 mgr.y (mgr.24401) 694 : cluster [DBG] pgmap v1160: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 83/648 objects degraded (12.809%) 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:25.894352+0000 mgr.y (mgr.24401) 694 : cluster [DBG] pgmap v1160: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 83/648 objects degraded (12.809%) 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.386679+0000 mon.a (mon.0) 3572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.386679+0000 mon.a (mon.0) 3572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: audit 2024-04-26T16:35:26.394033+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: audit 2024-04-26T16:35:26.394033+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.400527+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.400527+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-04-26T16:35:27.581 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.798740+0000 osd.2 (osd.2) 369 : cluster [DBG] 11.d scrub starts 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.798740+0000 osd.2 (osd.2) 369 : cluster [DBG] 11.d scrub starts 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.801851+0000 osd.2 (osd.2) 370 : cluster [DBG] 11.d scrub ok 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.801851+0000 osd.2 (osd.2) 370 : cluster [DBG] 11.d scrub ok 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.894352+0000 mgr.y (mgr.24401) 694 : cluster [DBG] pgmap v1160: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 83/648 objects degraded (12.809%) 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:25.894352+0000 mgr.y (mgr.24401) 694 : cluster [DBG] pgmap v1160: 204 pgs: 32 creating+peering, 39 active+undersized, 20 active+undersized+degraded, 113 active+clean; 583 KiB data, 723 MiB used, 715 GiB / 715 GiB avail; 83/648 objects degraded (12.809%) 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.386679+0000 mon.a (mon.0) 3572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.386679+0000 mon.a (mon.0) 3572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: audit 2024-04-26T16:35:26.394033+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: audit 2024-04-26T16:35:26.394033+0000 mon.a (mon.0) 3573 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:35:27.582 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.400527+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-04-26T16:35:27.583 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:27 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.400527+0000 mon.a (mon.0) 3574 : cluster [DBG] osdmap e748: 8 total, 8 up, 8 in 2024-04-26T16:35:27.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:27.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:27.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:27.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:27.859 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:27.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:27.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:27.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:27.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:28.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:28.124 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:28.228 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:28.257 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:28.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:28.389 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:28.492 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:28.521 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:28.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:28.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:28.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.524446+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:35:28.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.524446+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:35:28.656 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.526378+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: cluster 2024-04-26T16:35:26.526378+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: audit 2024-04-26T16:35:27.230428+0000 mgr.y (mgr.24401) 695 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: audit 2024-04-26T16:35:27.230428+0000 mgr.y (mgr.24401) 695 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: audit 2024-04-26T16:35:27.675793+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: audit 2024-04-26T16:35:27.675793+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: audit 2024-04-26T16:35:27.678837+0000 mon.c (mon.2) 783 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[21463]: audit 2024-04-26T16:35:27.678837+0000 mon.c (mon.2) 783 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.524446+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.524446+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.526378+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:35:28.657 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: cluster 2024-04-26T16:35:26.526378+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:35:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: audit 2024-04-26T16:35:27.230428+0000 mgr.y (mgr.24401) 695 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: audit 2024-04-26T16:35:27.230428+0000 mgr.y (mgr.24401) 695 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: audit 2024-04-26T16:35:27.675793+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: audit 2024-04-26T16:35:27.675793+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: audit 2024-04-26T16:35:27.678837+0000 mon.c (mon.2) 783 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:28.658 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:28 smithi050 bash[29234]: audit 2024-04-26T16:35:27.678837+0000 mon.c (mon.2) 783 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:28.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:28.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.524446+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:35:28.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.524446+0000 osd.5 (osd.5) 289 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:35:28.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.526378+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:35:28.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: cluster 2024-04-26T16:35:26.526378+0000 osd.5 (osd.5) 290 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:35:28.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: audit 2024-04-26T16:35:27.230428+0000 mgr.y (mgr.24401) 695 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:28.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: audit 2024-04-26T16:35:27.230428+0000 mgr.y (mgr.24401) 695 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:28.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: audit 2024-04-26T16:35:27.675793+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:28.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: audit 2024-04-26T16:35:27.675793+0000 mon.a (mon.0) 3575 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:28.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: audit 2024-04-26T16:35:27.678837+0000 mon.c (mon.2) 783 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:28.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:28 smithi190 bash[23868]: audit 2024-04-26T16:35:27.678837+0000 mon.c (mon.2) 783 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:28.787 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:28.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:28.919 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:28.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:29.023 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:29.051 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:29.155 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:29.183 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:29.286 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:29.307 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:29.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:29.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:27.895787+0000 mgr.y (mgr.24401) 696 : cluster [DBG] pgmap v1162: 204 pgs: 2 active+clean+snaptrim, 5 creating+peering, 10 active+undersized, 4 active+undersized+degraded, 183 active+clean; 583 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 1 op/s; 13/642 objects degraded (2.025%); 12 B/s, 2 objects/s recovering 2024-04-26T16:35:29.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:27.895787+0000 mgr.y (mgr.24401) 696 : cluster [DBG] pgmap v1162: 204 pgs: 2 active+clean+snaptrim, 5 creating+peering, 10 active+undersized, 4 active+undersized+degraded, 183 active+clean; 583 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 1 op/s; 13/642 objects degraded (2.025%); 12 B/s, 2 objects/s recovering 2024-04-26T16:35:29.421 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:28.670513+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:28.670513+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.171915+0000 osd.4 (osd.4) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.171915+0000 osd.4 (osd.4) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.172376+0000 osd.4 (osd.4) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.172376+0000 osd.4 (osd.4) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:27.895787+0000 mgr.y (mgr.24401) 696 : cluster [DBG] pgmap v1162: 204 pgs: 2 active+clean+snaptrim, 5 creating+peering, 10 active+undersized, 4 active+undersized+degraded, 183 active+clean; 583 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 1 op/s; 13/642 objects degraded (2.025%); 12 B/s, 2 objects/s recovering 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:27.895787+0000 mgr.y (mgr.24401) 696 : cluster [DBG] pgmap v1162: 204 pgs: 2 active+clean+snaptrim, 5 creating+peering, 10 active+undersized, 4 active+undersized+degraded, 183 active+clean; 583 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 1 op/s; 13/642 objects degraded (2.025%); 12 B/s, 2 objects/s recovering 2024-04-26T16:35:29.422 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:28.670513+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:29.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:28.670513+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:29.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.171915+0000 osd.4 (osd.4) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:29.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.171915+0000 osd.4 (osd.4) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:29.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.172376+0000 osd.4 (osd.4) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:29.423 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:29 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.172376+0000 osd.4 (osd.4) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:29.436 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:29.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:29.566 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:29.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:27.895787+0000 mgr.y (mgr.24401) 696 : cluster [DBG] pgmap v1162: 204 pgs: 2 active+clean+snaptrim, 5 creating+peering, 10 active+undersized, 4 active+undersized+degraded, 183 active+clean; 583 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 1 op/s; 13/642 objects degraded (2.025%); 12 B/s, 2 objects/s recovering 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:27.895787+0000 mgr.y (mgr.24401) 696 : cluster [DBG] pgmap v1162: 204 pgs: 2 active+clean+snaptrim, 5 creating+peering, 10 active+undersized, 4 active+undersized+degraded, 183 active+clean; 583 KiB data, 741 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 450 B/s wr, 1 op/s; 13/642 objects degraded (2.025%); 12 B/s, 2 objects/s recovering 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:28.670513+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:28.670513+0000 mon.a (mon.0) 3576 : cluster [WRN] Health check update: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.171915+0000 osd.4 (osd.4) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.171915+0000 osd.4 (osd.4) 373 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:29.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.172376+0000 osd.4 (osd.4) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:29.683 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:29 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.172376+0000 osd.4 (osd.4) 374 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:29.697 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:29.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:29.829 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:29.933 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:29.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:29.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:29.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:30.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:30.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:30.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:30.221 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:30.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:30.350 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:30.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:30.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:30.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:30.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:30.612 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:30.661 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:35:30.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:30.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:30.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:30.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:30 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.682929+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-04-26T16:35:30.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:30 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.682929+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-04-26T16:35:30.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:30 smithi190 bash[23868]: audit 2024-04-26T16:35:30.514102+0000 mon.c (mon.2) 784 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:30 smithi190 bash[23868]: audit 2024-04-26T16:35:30.514102+0000 mon.c (mon.2) 784 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:30 smithi190 bash[23868]: audit 2024-04-26T16:35:30.514595+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.856 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:30 smithi190 bash[23868]: audit 2024-04-26T16:35:30.514595+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.862 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.682929+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.682929+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[21463]: audit 2024-04-26T16:35:30.514102+0000 mon.c (mon.2) 784 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[21463]: audit 2024-04-26T16:35:30.514102+0000 mon.c (mon.2) 784 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[21463]: audit 2024-04-26T16:35:30.514595+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[21463]: audit 2024-04-26T16:35:30.514595+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.682929+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.682929+0000 mon.a (mon.0) 3577 : cluster [DBG] osdmap e749: 8 total, 8 up, 8 in 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[29234]: audit 2024-04-26T16:35:30.514102+0000 mon.c (mon.2) 784 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[29234]: audit 2024-04-26T16:35:30.514102+0000 mon.c (mon.2) 784 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.870 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[29234]: audit 2024-04-26T16:35:30.514595+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.871 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:30 smithi050 bash[29234]: audit 2024-04-26T16:35:30.514595+0000 mon.a (mon.0) 3578 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:30.873 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:30.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:30.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:31.001 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:31.103 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:31.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:31.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:31.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:31.356 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:31.394 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:31.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:31.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:31.546 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:31.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:31.677 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:31.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:31.792 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.896607+0000 mgr.y (mgr.24401) 697 : cluster [DBG] pgmap v1164: 204 pgs: 2 active+clean+snaptrim, 202 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 369 B/s wr, 1 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:29.896607+0000 mgr.y (mgr.24401) 697 : cluster [DBG] pgmap v1164: 204 pgs: 2 active+clean+snaptrim, 202 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 369 B/s wr, 1 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:30.676438+0000 mon.a (mon.0) 3579 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized) 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:30.676438+0000 mon.a (mon.0) 3579 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized) 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:30.678465+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:30.678465+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:30.686668+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:30.686668+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:30.694936+0000 mon.c (mon.2) 785 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:30.694936+0000 mon.c (mon.2) 785 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:30.695639+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:30.695639+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:31.793 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:31.347943+0000 mon.a (mon.0) 3583 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:31.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:31.347943+0000 mon.a (mon.0) 3583 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:31.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:31.350356+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:31.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: audit 2024-04-26T16:35:31.350356+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:31.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:31.355069+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-04-26T16:35:31.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:31 smithi190 bash[23868]: cluster 2024-04-26T16:35:31.355069+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-04-26T16:35:31.809 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:31.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:31.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:31.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:32.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:32.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.896607+0000 mgr.y (mgr.24401) 697 : cluster [DBG] pgmap v1164: 204 pgs: 2 active+clean+snaptrim, 202 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 369 B/s wr, 1 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T16:35:32.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:29.896607+0000 mgr.y (mgr.24401) 697 : cluster [DBG] pgmap v1164: 204 pgs: 2 active+clean+snaptrim, 202 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 369 B/s wr, 1 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T16:35:32.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:30.676438+0000 mon.a (mon.0) 3579 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized) 2024-04-26T16:35:32.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:30.676438+0000 mon.a (mon.0) 3579 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized) 2024-04-26T16:35:32.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:30.678465+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:30.678465+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:30.686668+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:30.686668+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:30.694936+0000 mon.c (mon.2) 785 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:30.694936+0000 mon.c (mon.2) 785 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:30.695639+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:30.695639+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:31.347943+0000 mon.a (mon.0) 3583 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:31.347943+0000 mon.a (mon.0) 3583 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:31.350356+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:32.057 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: audit 2024-04-26T16:35:31.350356+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.896607+0000 mgr.y (mgr.24401) 697 : cluster [DBG] pgmap v1164: 204 pgs: 2 active+clean+snaptrim, 202 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 369 B/s wr, 1 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:29.896607+0000 mgr.y (mgr.24401) 697 : cluster [DBG] pgmap v1164: 204 pgs: 2 active+clean+snaptrim, 202 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 369 B/s wr, 1 op/s; 10 B/s, 1 objects/s recovering 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:30.676438+0000 mon.a (mon.0) 3579 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized) 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:30.676438+0000 mon.a (mon.0) 3579 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 13/642 objects degraded (2.025%), 4 pgs degraded, 14 pgs undersized) 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:30.678465+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:30.678465+0000 mon.a (mon.0) 3580 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:30.686668+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:30.686668+0000 mon.a (mon.0) 3581 : cluster [DBG] osdmap e750: 8 total, 8 up, 8 in 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:30.694936+0000 mon.c (mon.2) 785 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:30.694936+0000 mon.c (mon.2) 785 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:30.695639+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.058 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:30.695639+0000 mon.a (mon.0) 3582 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]: dispatch 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:31.347943+0000 mon.a (mon.0) 3583 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:31.347943+0000 mon.a (mon.0) 3583 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:31.350356+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: audit 2024-04-26T16:35:31.350356+0000 mon.a (mon.0) 3584 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-128"}]': finished 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:31.355069+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[29234]: cluster 2024-04-26T16:35:31.355069+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:31.355069+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-04-26T16:35:32.059 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:31 smithi050 bash[21463]: cluster 2024-04-26T16:35:31.355069+0000 mon.a (mon.0) 3585 : cluster [DBG] osdmap e751: 8 total, 8 up, 8 in 2024-04-26T16:35:32.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:32.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:32.203 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:32.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:32.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:32.378 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:32.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:32.460 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:32.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:32.588 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:32.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:32.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:32.719 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:32.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:32.849 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:32.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:32.982 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:32.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:33.084 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:33.113 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:33.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:33.241 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:33.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:33.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:33.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:33.507 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:33.610 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[21463]: cluster 2024-04-26T16:35:31.898283+0000 mgr.y (mgr.24401) 698 : cluster [DBG] pgmap v1167: 204 pgs: 204 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 372 B/s wr, 2 op/s; 5 B/s, 0 objects/s recovering 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[21463]: cluster 2024-04-26T16:35:31.898283+0000 mgr.y (mgr.24401) 698 : cluster [DBG] pgmap v1167: 204 pgs: 204 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 372 B/s wr, 2 op/s; 5 B/s, 0 objects/s recovering 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[21463]: cluster 2024-04-26T16:35:32.369431+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[21463]: cluster 2024-04-26T16:35:32.369431+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[21463]: cluster 2024-04-26T16:35:32.698856+0000 mon.a (mon.0) 3587 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[21463]: cluster 2024-04-26T16:35:32.698856+0000 mon.a (mon.0) 3587 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:33.622 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[29234]: cluster 2024-04-26T16:35:31.898283+0000 mgr.y (mgr.24401) 698 : cluster [DBG] pgmap v1167: 204 pgs: 204 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 372 B/s wr, 2 op/s; 5 B/s, 0 objects/s recovering 2024-04-26T16:35:33.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[29234]: cluster 2024-04-26T16:35:31.898283+0000 mgr.y (mgr.24401) 698 : cluster [DBG] pgmap v1167: 204 pgs: 204 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 372 B/s wr, 2 op/s; 5 B/s, 0 objects/s recovering 2024-04-26T16:35:33.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[29234]: cluster 2024-04-26T16:35:32.369431+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-04-26T16:35:33.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[29234]: cluster 2024-04-26T16:35:32.369431+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-04-26T16:35:33.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[29234]: cluster 2024-04-26T16:35:32.698856+0000 mon.a (mon.0) 3587 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:33.623 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:33 smithi050 bash[29234]: cluster 2024-04-26T16:35:32.698856+0000 mon.a (mon.0) 3587 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:33.637 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:33.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:33.768 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:33.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:33 smithi190 bash[23868]: cluster 2024-04-26T16:35:31.898283+0000 mgr.y (mgr.24401) 698 : cluster [DBG] pgmap v1167: 204 pgs: 204 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 372 B/s wr, 2 op/s; 5 B/s, 0 objects/s recovering 2024-04-26T16:35:33.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:33 smithi190 bash[23868]: cluster 2024-04-26T16:35:31.898283+0000 mgr.y (mgr.24401) 698 : cluster [DBG] pgmap v1167: 204 pgs: 204 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 372 B/s wr, 2 op/s; 5 B/s, 0 objects/s recovering 2024-04-26T16:35:33.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:33 smithi190 bash[23868]: cluster 2024-04-26T16:35:32.369431+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-04-26T16:35:33.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:33 smithi190 bash[23868]: cluster 2024-04-26T16:35:32.369431+0000 mon.a (mon.0) 3586 : cluster [DBG] osdmap e752: 8 total, 8 up, 8 in 2024-04-26T16:35:33.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:33 smithi190 bash[23868]: cluster 2024-04-26T16:35:32.698856+0000 mon.a (mon.0) 3587 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:33.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:33 smithi190 bash[23868]: cluster 2024-04-26T16:35:32.698856+0000 mon.a (mon.0) 3587 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:33.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:33.899 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:33.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:33.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:34.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:34.029 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:34.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:34.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:34.154 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:34.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:34.285 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:34.389 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:34.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: cluster 2024-04-26T16:35:33.370211+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-04-26T16:35:34.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: cluster 2024-04-26T16:35:33.370211+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-04-26T16:35:34.399 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:33.379455+0000 mon.c (mon.2) 786 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:33.379455+0000 mon.c (mon.2) 786 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:33.393014+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:33.393014+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:34.367738+0000 mon.a (mon.0) 3590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:34.367738+0000 mon.a (mon.0) 3590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: cluster 2024-04-26T16:35:34.372655+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: cluster 2024-04-26T16:35:34.372655+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:34.380060+0000 mon.c (mon.2) 787 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21463]: audit 2024-04-26T16:35:34.380060+0000 mon.c (mon.2) 787 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: cluster 2024-04-26T16:35:33.370211+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-04-26T16:35:34.400 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: cluster 2024-04-26T16:35:33.370211+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:33.379455+0000 mon.c (mon.2) 786 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:33.379455+0000 mon.c (mon.2) 786 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:33.393014+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:33.393014+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:34.367738+0000 mon.a (mon.0) 3590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:34.367738+0000 mon.a (mon.0) 3590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: cluster 2024-04-26T16:35:34.372655+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: cluster 2024-04-26T16:35:34.372655+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:34.380060+0000 mon.c (mon.2) 787 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:34.401 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[29234]: audit 2024-04-26T16:35:34.380060+0000 mon.c (mon.2) 787 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:34.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:34.516 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:34.541 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: cluster 2024-04-26T16:35:33.370211+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: cluster 2024-04-26T16:35:33.370211+0000 mon.a (mon.0) 3588 : cluster [DBG] osdmap e753: 8 total, 8 up, 8 in 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:33.379455+0000 mon.c (mon.2) 786 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:33.379455+0000 mon.c (mon.2) 786 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:33.393014+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:33.393014+0000 mon.a (mon.0) 3589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:34.367738+0000 mon.a (mon.0) 3590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:34.367738+0000 mon.a (mon.0) 3590 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: cluster 2024-04-26T16:35:34.372655+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-04-26T16:35:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: cluster 2024-04-26T16:35:34.372655+0000 mon.a (mon.0) 3591 : cluster [DBG] osdmap e754: 8 total, 8 up, 8 in 2024-04-26T16:35:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:34.380060+0000 mon.c (mon.2) 787 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:34.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:34 smithi190 bash[23868]: audit 2024-04-26T16:35:34.380060+0000 mon.c (mon.2) 787 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:34.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:34.655 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:35:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:35:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:35:34.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:34.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:34.800 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:34.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:34.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:34.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:35.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:35.061 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:35.165 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:35.193 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:35.296 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:35.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:35.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:35.453 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:35.556 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:35.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: cluster 2024-04-26T16:35:33.899581+0000 mgr.y (mgr.24401) 699 : cluster [DBG] pgmap v1170: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:35:35.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: cluster 2024-04-26T16:35:33.899581+0000 mgr.y (mgr.24401) 699 : cluster [DBG] pgmap v1170: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:34.380748+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:34.380748+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:35.370304+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:35.370304+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: cluster 2024-04-26T16:35:35.374197+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: cluster 2024-04-26T16:35:35.374197+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:35.379452+0000 mon.c (mon.2) 788 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:35.379452+0000 mon.c (mon.2) 788 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:35.379850+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[21463]: audit 2024-04-26T16:35:35.379850+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: cluster 2024-04-26T16:35:33.899581+0000 mgr.y (mgr.24401) 699 : cluster [DBG] pgmap v1170: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: cluster 2024-04-26T16:35:33.899581+0000 mgr.y (mgr.24401) 699 : cluster [DBG] pgmap v1170: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:34.380748+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:34.380748+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:35.370304+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:35.370304+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: cluster 2024-04-26T16:35:35.374197+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: cluster 2024-04-26T16:35:35.374197+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:35.379452+0000 mon.c (mon.2) 788 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:35.379452+0000 mon.c (mon.2) 788 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:35.379850+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.570 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:35 smithi050 bash[29234]: audit 2024-04-26T16:35:35.379850+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.583 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:35.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:35.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: cluster 2024-04-26T16:35:33.899581+0000 mgr.y (mgr.24401) 699 : cluster [DBG] pgmap v1170: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:35:35.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: cluster 2024-04-26T16:35:33.899581+0000 mgr.y (mgr.24401) 699 : cluster [DBG] pgmap v1170: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:35:35.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:34.380748+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:35.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:34.380748+0000 mon.a (mon.0) 3592 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:35.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:35.370304+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:35.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:35.370304+0000 mon.a (mon.0) 3593 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:35.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: cluster 2024-04-26T16:35:35.374197+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-04-26T16:35:35.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: cluster 2024-04-26T16:35:35.374197+0000 mon.a (mon.0) 3594 : cluster [DBG] osdmap e755: 8 total, 8 up, 8 in 2024-04-26T16:35:35.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:35.379452+0000 mon.c (mon.2) 788 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:35.379452+0000 mon.c (mon.2) 788 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:35.379850+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.699 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:35 smithi190 bash[23868]: audit 2024-04-26T16:35:35.379850+0000 mon.a (mon.0) 3595 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:35.714 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:35.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:35.845 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:35.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:35.975 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:35.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:36.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:36.108 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:36.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:36.238 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:36.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:36.370 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:36.473 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:36.501 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:36.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:36.632 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:36.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:36.761 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:36.762 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:36.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:36.894 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:36.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:36.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:37.026 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:37.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:37.157 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:37.260 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:37.288 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:37.324 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:35:37.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:37.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:37.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:37.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: cluster 2024-04-26T16:35:35.900465+0000 mgr.y (mgr.24401) 700 : cluster [DBG] pgmap v1173: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:37.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: cluster 2024-04-26T16:35:35.900465+0000 mgr.y (mgr.24401) 700 : cluster [DBG] pgmap v1173: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:37.531 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: audit 2024-04-26T16:35:36.351980+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: audit 2024-04-26T16:35:36.351980+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: cluster 2024-04-26T16:35:36.355607+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: cluster 2024-04-26T16:35:36.355607+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: audit 2024-04-26T16:35:36.357924+0000 mon.c (mon.2) 789 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: audit 2024-04-26T16:35:36.357924+0000 mon.c (mon.2) 789 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: audit 2024-04-26T16:35:36.358395+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[29234]: audit 2024-04-26T16:35:36.358395+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: cluster 2024-04-26T16:35:35.900465+0000 mgr.y (mgr.24401) 700 : cluster [DBG] pgmap v1173: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: cluster 2024-04-26T16:35:35.900465+0000 mgr.y (mgr.24401) 700 : cluster [DBG] pgmap v1173: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: audit 2024-04-26T16:35:36.351980+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:37.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: audit 2024-04-26T16:35:36.351980+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:37.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: cluster 2024-04-26T16:35:36.355607+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-04-26T16:35:37.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: cluster 2024-04-26T16:35:36.355607+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-04-26T16:35:37.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: audit 2024-04-26T16:35:36.357924+0000 mon.c (mon.2) 789 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: audit 2024-04-26T16:35:36.357924+0000 mon.c (mon.2) 789 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: audit 2024-04-26T16:35:36.358395+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.533 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:37 smithi050 bash[21463]: audit 2024-04-26T16:35:36.358395+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.547 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: cluster 2024-04-26T16:35:35.900465+0000 mgr.y (mgr.24401) 700 : cluster [DBG] pgmap v1173: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: cluster 2024-04-26T16:35:35.900465+0000 mgr.y (mgr.24401) 700 : cluster [DBG] pgmap v1173: 204 pgs: 2 creating+peering, 30 unknown, 172 active+clean; 583 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: audit 2024-04-26T16:35:36.351980+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: audit 2024-04-26T16:35:36.351980+0000 mon.a (mon.0) 3596 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: cluster 2024-04-26T16:35:36.355607+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: cluster 2024-04-26T16:35:36.355607+0000 mon.a (mon.0) 3597 : cluster [DBG] osdmap e756: 8 total, 8 up, 8 in 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: audit 2024-04-26T16:35:36.357924+0000 mon.c (mon.2) 789 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: audit 2024-04-26T16:35:36.357924+0000 mon.c (mon.2) 789 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: audit 2024-04-26T16:35:36.358395+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:37 smithi190 bash[23868]: audit 2024-04-26T16:35:36.358395+0000 mon.a (mon.0) 3598 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]: dispatch 2024-04-26T16:35:37.650 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:37.678 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:37.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:37.809 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:37.912 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:37.941 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:37.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:38.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:38.071 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:38.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:38.202 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:38.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:38.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:38.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:38.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:38.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.240297+0000 mgr.y (mgr.24401) 701 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.240297+0000 mgr.y (mgr.24401) 701 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: cluster 2024-04-26T16:35:37.352458+0000 mon.a (mon.0) 3599 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: cluster 2024-04-26T16:35:37.352458+0000 mon.a (mon.0) 3599 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.354596+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]': finished 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.354596+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]': finished 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: cluster 2024-04-26T16:35:37.363843+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: cluster 2024-04-26T16:35:37.363843+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.507863+0000 mon.c (mon.2) 790 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.507863+0000 mon.c (mon.2) 790 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.508716+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:38 smithi190 bash[23868]: audit 2024-04-26T16:35:37.508716+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.240297+0000 mgr.y (mgr.24401) 701 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:38.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.240297+0000 mgr.y (mgr.24401) 701 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: cluster 2024-04-26T16:35:37.352458+0000 mon.a (mon.0) 3599 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: cluster 2024-04-26T16:35:37.352458+0000 mon.a (mon.0) 3599 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.354596+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]': finished 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.354596+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]': finished 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: cluster 2024-04-26T16:35:37.363843+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: cluster 2024-04-26T16:35:37.363843+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.507863+0000 mon.c (mon.2) 790 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.507863+0000 mon.c (mon.2) 790 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.508716+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[21463]: audit 2024-04-26T16:35:37.508716+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.240297+0000 mgr.y (mgr.24401) 701 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.240297+0000 mgr.y (mgr.24401) 701 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: cluster 2024-04-26T16:35:37.352458+0000 mon.a (mon.0) 3599 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: cluster 2024-04-26T16:35:37.352458+0000 mon.a (mon.0) 3599 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.354596+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]': finished 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.354596+0000 mon.a (mon.0) 3600 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-130", "mode": "writeback"}]': finished 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: cluster 2024-04-26T16:35:37.363843+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: cluster 2024-04-26T16:35:37.363843+0000 mon.a (mon.0) 3601 : cluster [DBG] osdmap e757: 8 total, 8 up, 8 in 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.507863+0000 mon.c (mon.2) 790 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.507863+0000 mon.c (mon.2) 790 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.508716+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:38 smithi050 bash[29234]: audit 2024-04-26T16:35:37.508716+0000 mon.a (mon.0) 3602 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:38.696 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:38.720 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:38.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:38.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:38.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:38.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:39.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:39.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:39.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:39.242 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:39.345 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:39.470 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:39.492 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:39.535 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:35:39.535 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: cluster 2024-04-26T16:35:37.901311+0000 mgr.y (mgr.24401) 702 : cluster [DBG] pgmap v1176: 204 pgs: 2 creating+peering, 202 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: cluster 2024-04-26T16:35:37.901311+0000 mgr.y (mgr.24401) 702 : cluster [DBG] pgmap v1176: 204 pgs: 2 creating+peering, 202 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: audit 2024-04-26T16:35:38.375394+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: audit 2024-04-26T16:35:38.375394+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: cluster 2024-04-26T16:35:38.386587+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: cluster 2024-04-26T16:35:38.386587+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: audit 2024-04-26T16:35:38.388464+0000 mon.c (mon.2) 791 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: audit 2024-04-26T16:35:38.388464+0000 mon.c (mon.2) 791 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: audit 2024-04-26T16:35:38.389852+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:39 smithi190 bash[23868]: audit 2024-04-26T16:35:38.389852+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:39.621 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: cluster 2024-04-26T16:35:37.901311+0000 mgr.y (mgr.24401) 702 : cluster [DBG] pgmap v1176: 204 pgs: 2 creating+peering, 202 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:39.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: cluster 2024-04-26T16:35:37.901311+0000 mgr.y (mgr.24401) 702 : cluster [DBG] pgmap v1176: 204 pgs: 2 creating+peering, 202 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: audit 2024-04-26T16:35:38.375394+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: audit 2024-04-26T16:35:38.375394+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: cluster 2024-04-26T16:35:38.386587+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: cluster 2024-04-26T16:35:38.386587+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: audit 2024-04-26T16:35:38.388464+0000 mon.c (mon.2) 791 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: audit 2024-04-26T16:35:38.388464+0000 mon.c (mon.2) 791 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: audit 2024-04-26T16:35:38.389852+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[21463]: audit 2024-04-26T16:35:38.389852+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: cluster 2024-04-26T16:35:37.901311+0000 mgr.y (mgr.24401) 702 : cluster [DBG] pgmap v1176: 204 pgs: 2 creating+peering, 202 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:39.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: cluster 2024-04-26T16:35:37.901311+0000 mgr.y (mgr.24401) 702 : cluster [DBG] pgmap v1176: 204 pgs: 2 creating+peering, 202 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: audit 2024-04-26T16:35:38.375394+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: audit 2024-04-26T16:35:38.375394+0000 mon.a (mon.0) 3603 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: cluster 2024-04-26T16:35:38.386587+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: cluster 2024-04-26T16:35:38.386587+0000 mon.a (mon.0) 3604 : cluster [DBG] osdmap e758: 8 total, 8 up, 8 in 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: audit 2024-04-26T16:35:38.388464+0000 mon.c (mon.2) 791 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: audit 2024-04-26T16:35:38.388464+0000 mon.c (mon.2) 791 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: audit 2024-04-26T16:35:38.389852+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:39 smithi050 bash[29234]: audit 2024-04-26T16:35:38.389852+0000 mon.a (mon.0) 3605 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]: dispatch 2024-04-26T16:35:39.725 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:39.753 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:39.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:39.855 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:39.882 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:39.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:39.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:39.985 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:40.014 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:40.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.145 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:40.248 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:40.277 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:40.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:40.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.412 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:40.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[29234]: cluster 2024-04-26T16:35:39.375634+0000 mon.a (mon.0) 3606 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[29234]: cluster 2024-04-26T16:35:39.375634+0000 mon.a (mon.0) 3606 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[29234]: audit 2024-04-26T16:35:39.377399+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[29234]: audit 2024-04-26T16:35:39.377399+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[29234]: cluster 2024-04-26T16:35:39.381281+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-04-26T16:35:40.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[29234]: cluster 2024-04-26T16:35:39.381281+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-04-26T16:35:40.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[21463]: cluster 2024-04-26T16:35:39.375634+0000 mon.a (mon.0) 3606 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:40.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[21463]: cluster 2024-04-26T16:35:39.375634+0000 mon.a (mon.0) 3606 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:40.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[21463]: audit 2024-04-26T16:35:39.377399+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:40.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[21463]: audit 2024-04-26T16:35:39.377399+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:40.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[21463]: cluster 2024-04-26T16:35:39.381281+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-04-26T16:35:40.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:40 smithi050 bash[21463]: cluster 2024-04-26T16:35:39.381281+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-04-26T16:35:40.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:40.669 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:40.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:40 smithi190 bash[23868]: cluster 2024-04-26T16:35:39.375634+0000 mon.a (mon.0) 3606 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:40.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:40 smithi190 bash[23868]: cluster 2024-04-26T16:35:39.375634+0000 mon.a (mon.0) 3606 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:40.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:40 smithi190 bash[23868]: audit 2024-04-26T16:35:39.377399+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:40.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:40 smithi190 bash[23868]: audit 2024-04-26T16:35:39.377399+0000 mon.a (mon.0) 3607 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-130"}]': finished 2024-04-26T16:35:40.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:40 smithi190 bash[23868]: cluster 2024-04-26T16:35:39.381281+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-04-26T16:35:40.786 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:40 smithi190 bash[23868]: cluster 2024-04-26T16:35:39.381281+0000 mon.a (mon.0) 3608 : cluster [DBG] osdmap e759: 8 total, 8 up, 8 in 2024-04-26T16:35:40.800 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:40.903 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:40.931 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:40.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:41.035 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:41.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:41.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:41.193 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:41.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:41.322 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:41.426 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:41.449 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:41.553 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:41.575 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:39.901889+0000 mgr.y (mgr.24401) 703 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:39.901889+0000 mgr.y (mgr.24401) 703 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:40.389049+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:40.389049+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:40.403613+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:40.403613+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-04-26T16:35:41.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:41.369590+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-04-26T16:35:41.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:41 smithi190 bash[23868]: cluster 2024-04-26T16:35:41.369590+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-04-26T16:35:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:41.647 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:39.901889+0000 mgr.y (mgr.24401) 703 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:41.647 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:39.901889+0000 mgr.y (mgr.24401) 703 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:41.647 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:40.389049+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:41.647 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:40.389049+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:41.647 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:40.403613+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-04-26T16:35:41.648 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:40.403613+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-04-26T16:35:41.648 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:41.369590+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-04-26T16:35:41.648 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[29234]: cluster 2024-04-26T16:35:41.369590+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-04-26T16:35:41.648 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:39.901889+0000 mgr.y (mgr.24401) 703 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:41.648 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:39.901889+0000 mgr.y (mgr.24401) 703 : cluster [DBG] pgmap v1179: 204 pgs: 204 active+clean; 583 KiB data, 743 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:41.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:40.389049+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:41.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:40.389049+0000 mon.a (mon.0) 3609 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:41.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:40.403613+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-04-26T16:35:41.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:40.403613+0000 mon.a (mon.0) 3610 : cluster [DBG] osdmap e760: 8 total, 8 up, 8 in 2024-04-26T16:35:41.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:41.369590+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-04-26T16:35:41.649 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:41 smithi050 bash[21463]: cluster 2024-04-26T16:35:41.369590+0000 mon.a (mon.0) 3611 : cluster [DBG] osdmap e761: 8 total, 8 up, 8 in 2024-04-26T16:35:41.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:41.706 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- 172.21.15.50:0/3548209741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff288105640 msgr2=0x7ff288105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- 172.21.15.50:0/3548209741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff288105640 0x7ff288105ac0 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7ff27c0098f0 tx=0x7ff27c009c00 comp rx=0 tx=0).stop 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- 172.21.15.50:0/3548209741 shutdown_connections 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- 172.21.15.50:0/3548209741 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff288106000 0x7ff288112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- 172.21.15.50:0/3548209741 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff28810b640 0x7ff28810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- 172.21.15.50:0/3548209741 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff288105640 0x7ff288105ac0 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:41.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- 172.21.15.50:0/3548209741 >> 172.21.15.50:0/3548209741 conn(0x7ff288101130 msgr2=0x7ff288103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- 172.21.15.50:0/3548209741 shutdown_connections 2024-04-26T16:35:41.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- 172.21.15.50:0/3548209741 wait complete. 2024-04-26T16:35:41.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 librados: starting msgr at 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 librados: starting objecter 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 Processor -- start 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- start start 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 librados: setting wanted keys 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 librados: calling monclient init 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 0x7ff28819b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff288106000 0x7ff28819b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:41.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff28810b640 0x7ff28819fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff288114d60 con 0x7ff288105640 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff288114be0 con 0x7ff288106000 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff288114ee0 con 0x7ff28810b640 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff286ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 0x7ff28819b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff286ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 0x7ff28819b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:39720/0 (socket says 172.21.15.50:39720) 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff286ffd700 1 -- 172.21.15.50:0/3310215297 learned_addr learned my addr 172.21.15.50:0/3310215297 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff28810b640 0x7ff28819fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2867fc700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff288106000 0x7ff28819b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 -- 172.21.15.50:0/3310215297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff288106000 msgr2=0x7ff28819b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:41.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff288106000 0x7ff28819b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 -- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 msgr2=0x7ff28819b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 0x7ff28819b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 -- 172.21.15.50:0/3310215297 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2881a0360 con 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff286ffd700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 0x7ff28819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff2877fe700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff28810b640 0x7ff28819fc20 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7ff280007f10 tx=0x7ff28000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28c935700 10 client.?.objecter ms_handle_connect 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28c935700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff280004db0 con 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff280009e20 con 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2881a0650 con 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.787+0000 7ff28d937700 10 client.41606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2800059a0 con 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2881a7ed0 con 0x7ff28810b640 2024-04-26T16:35:41.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28d937700 1 librados: init done 2024-04-26T16:35:41.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff288106d80 con 0x7ff28810b640 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff28000db20 con 0x7ff28810b640 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff270071e50 0x7ff270074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(761..761 src has 577..761) ==== 9911+0+0 (secure 0 0 0) 0x7ff280091540 con 0x7ff28810b640 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 10 client.41606.objecter ms_dispatch 0x7ff288001550 osd_map(761..761 src has 577..761) 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 3 client.41606.objecter handle_osd_map got epochs [761,761] > 0 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 3 client.41606.objecter handle_osd_map decoding full epoch 761 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff286ffd700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff270071e50 0x7ff270074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 20 client.41606.objecter dump_active .. 0 homeless 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff28c935700 10 client.41606.objecter ms_handle_connect 0x7ff270071e50 2024-04-26T16:35:41.794 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.791+0000 7ff286ffd700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff270071e50 0x7ff270074310 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7ff278009750 tx=0x7ff278009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:41.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:41.795+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff280010070 con 0x7ff28810b640 2024-04-26T16:35:41.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:41.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:41.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:41.950 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:41.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:42.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.039+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7ff28806a110 con 0x7ff28810b640 2024-04-26T16:35:42.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:42.080 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:42.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:42.184 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:42.212 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:42.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:42.348 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:42.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.359+0000 7ff28c935700 1 -- 172.21.15.50:0/3310215297 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v762) ==== 113+0+0 (secure 0 0 0) 0x7ff28005cd60 con 0x7ff28810b640 2024-04-26T16:35:42.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 10 librados: watch_flush enter 2024-04-26T16:35:42.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 10 librados: watch_flush exit 2024-04-26T16:35:42.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 20 client.41606.objecter shutdown clearing up homeless session... 2024-04-26T16:35:42.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 10 client.41606.objecter successfully canceled tick 2024-04-26T16:35:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff270071e50 msgr2=0x7ff270074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff270071e50 0x7ff270074310 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7ff278009750 tx=0x7ff278009310 comp rx=0 tx=0).stop 2024-04-26T16:35:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff28810b640 msgr2=0x7ff28819fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:42.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff28810b640 0x7ff28819fc20 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7ff280007f10 tx=0x7ff28000d3b0 comp rx=0 tx=0).stop 2024-04-26T16:35:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 shutdown_connections 2024-04-26T16:35:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff270071e50 0x7ff270074310 unknown :-1 s=CLOSED pgs=3826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff28810b640 0x7ff28819fc20 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff288105640 0x7ff28819b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 --2- 172.21.15.50:0/3310215297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff288106000 0x7ff28819b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:42.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 >> 172.21.15.50:0/3310215297 conn(0x7ff288101130 msgr2=0x7ff288101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 shutdown_connections 2024-04-26T16:35:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 -- 172.21.15.50:0/3310215297 wait complete. 2024-04-26T16:35:42.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:42.371+0000 7ff28d937700 1 librados: shutdown 2024-04-26T16:35:42.377 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T16:35:42.397 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:41.377493+0000 mon.c (mon.2) 792 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:41.377493+0000 mon.c (mon.2) 792 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:41.397476+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:41.397476+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.045826+0000 mon.c (mon.2) 793 : audit [INF] from='client.? 172.21.15.50:0/3310215297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.045826+0000 mon.c (mon.2) 793 : audit [INF] from='client.? 172.21.15.50:0/3310215297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.046367+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.447 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.046367+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:41.377493+0000 mon.c (mon.2) 792 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:41.377493+0000 mon.c (mon.2) 792 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:41.397476+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:41.397476+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.045826+0000 mon.c (mon.2) 793 : audit [INF] from='client.? 172.21.15.50:0/3310215297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.045826+0000 mon.c (mon.2) 793 : audit [INF] from='client.? 172.21.15.50:0/3310215297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.046367+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.046367+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.359553+0000 mon.a (mon.0) 3614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.359553+0000 mon.a (mon.0) 3614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:42.448 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.359683+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:35:42.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:42.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:42.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:42.605 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:42.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:42.720 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.359553+0000 mon.a (mon.0) 3614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.359553+0000 mon.a (mon.0) 3614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.359683+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.359683+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: cluster 2024-04-26T16:35:42.365881+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: cluster 2024-04-26T16:35:42.365881+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.372124+0000 mon.c (mon.2) 794 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.372124+0000 mon.c (mon.2) 794 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.376587+0000 mon.a (mon.0) 3617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.721 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[29234]: audit 2024-04-26T16:35:42.376587+0000 mon.a (mon.0) 3617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.359683+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: cluster 2024-04-26T16:35:42.365881+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: cluster 2024-04-26T16:35:42.365881+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.372124+0000 mon.c (mon.2) 794 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.372124+0000 mon.c (mon.2) 794 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.376587+0000 mon.a (mon.0) 3617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.722 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:42 smithi050 bash[21463]: audit 2024-04-26T16:35:42.376587+0000 mon.a (mon.0) 3617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.734 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:41.377493+0000 mon.c (mon.2) 792 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:41.377493+0000 mon.c (mon.2) 792 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:41.397476+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:41.397476+0000 mon.a (mon.0) 3612 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.045826+0000 mon.c (mon.2) 793 : audit [INF] from='client.? 172.21.15.50:0/3310215297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.045826+0000 mon.c (mon.2) 793 : audit [INF] from='client.? 172.21.15.50:0/3310215297' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.046367+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.046367+0000 mon.a (mon.0) 3613 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.359553+0000 mon.a (mon.0) 3614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.359553+0000 mon.a (mon.0) 3614 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.359683+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.359683+0000 mon.a (mon.0) 3615 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: cluster 2024-04-26T16:35:42.365881+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: cluster 2024-04-26T16:35:42.365881+0000 mon.a (mon.0) 3616 : cluster [DBG] osdmap e762: 8 total, 8 up, 8 in 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.372124+0000 mon.c (mon.2) 794 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.372124+0000 mon.c (mon.2) 794 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.376587+0000 mon.a (mon.0) 3617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:42 smithi190 bash[23868]: audit 2024-04-26T16:35:42.376587+0000 mon.a (mon.0) 3617 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:42.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:42.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:42.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:42.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:42.998 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:43.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:43.102 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:43.131 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:43.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:43.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:43.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:43.389 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:43.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:43.520 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:43.623 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:43.652 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:43.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:43.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:43.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:43.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:43.914 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:43.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:43.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: cluster 2024-04-26T16:35:41.902872+0000 mgr.y (mgr.24401) 704 : cluster [DBG] pgmap v1182: 204 pgs: 28 creating+peering, 4 unknown, 172 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: cluster 2024-04-26T16:35:41.902872+0000 mgr.y (mgr.24401) 704 : cluster [DBG] pgmap v1182: 204 pgs: 28 creating+peering, 4 unknown, 172 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:42.719864+0000 mon.a (mon.0) 3618 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:42.719864+0000 mon.a (mon.0) 3618 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:42.723268+0000 mon.c (mon.2) 795 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:42.723268+0000 mon.c (mon.2) 795 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:43.362592+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:43.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:43.362592+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: cluster 2024-04-26T16:35:43.370405+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: cluster 2024-04-26T16:35:43.370405+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:43.375012+0000 mon.c (mon.2) 796 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:43.375012+0000 mon.c (mon.2) 796 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:43.375918+0000 mon.a (mon.0) 3621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[21463]: audit 2024-04-26T16:35:43.375918+0000 mon.a (mon.0) 3621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: cluster 2024-04-26T16:35:41.902872+0000 mgr.y (mgr.24401) 704 : cluster [DBG] pgmap v1182: 204 pgs: 28 creating+peering, 4 unknown, 172 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: cluster 2024-04-26T16:35:41.902872+0000 mgr.y (mgr.24401) 704 : cluster [DBG] pgmap v1182: 204 pgs: 28 creating+peering, 4 unknown, 172 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:42.719864+0000 mon.a (mon.0) 3618 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:42.719864+0000 mon.a (mon.0) 3618 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:42.723268+0000 mon.c (mon.2) 795 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:43.971 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:42.723268+0000 mon.c (mon.2) 795 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:43.362592+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:43.362592+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: cluster 2024-04-26T16:35:43.370405+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: cluster 2024-04-26T16:35:43.370405+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:43.375012+0000 mon.c (mon.2) 796 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:43.375012+0000 mon.c (mon.2) 796 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:43.375918+0000 mon.a (mon.0) 3621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:43.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:43 smithi050 bash[29234]: audit 2024-04-26T16:35:43.375918+0000 mon.a (mon.0) 3621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:44.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:44.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: cluster 2024-04-26T16:35:41.902872+0000 mgr.y (mgr.24401) 704 : cluster [DBG] pgmap v1182: 204 pgs: 28 creating+peering, 4 unknown, 172 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: cluster 2024-04-26T16:35:41.902872+0000 mgr.y (mgr.24401) 704 : cluster [DBG] pgmap v1182: 204 pgs: 28 creating+peering, 4 unknown, 172 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:42.719864+0000 mon.a (mon.0) 3618 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:42.719864+0000 mon.a (mon.0) 3618 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:35:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:42.723268+0000 mon.c (mon.2) 795 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:42.723268+0000 mon.c (mon.2) 795 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:43.362592+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:43.362592+0000 mon.a (mon.0) 3619 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: cluster 2024-04-26T16:35:43.370405+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: cluster 2024-04-26T16:35:43.370405+0000 mon.a (mon.0) 3620 : cluster [DBG] osdmap e763: 8 total, 8 up, 8 in 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:43.375012+0000 mon.c (mon.2) 796 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:43.375012+0000 mon.c (mon.2) 796 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:43.375918+0000 mon.a (mon.0) 3621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:43 smithi190 bash[23868]: audit 2024-04-26T16:35:43.375918+0000 mon.a (mon.0) 3621 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:44.147 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.176 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:44.280 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.306 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:44.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:44.437 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:44.540 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.567 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:44.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:35:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:35:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:35:44.671 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:44.695 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:44.798 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.825 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:44.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:44.957 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:44.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:45.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:45.087 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:45.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:45.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:45.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:45.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:45.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:45.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: cluster 2024-04-26T16:35:43.903860+0000 mgr.y (mgr.24401) 705 : cluster [DBG] pgmap v1185: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: cluster 2024-04-26T16:35:43.903860+0000 mgr.y (mgr.24401) 705 : cluster [DBG] pgmap v1185: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: audit 2024-04-26T16:35:44.365288+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: audit 2024-04-26T16:35:44.365288+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: cluster 2024-04-26T16:35:44.373825+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: cluster 2024-04-26T16:35:44.373825+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: audit 2024-04-26T16:35:44.374821+0000 mon.c (mon.2) 797 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: audit 2024-04-26T16:35:44.374821+0000 mon.c (mon.2) 797 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: audit 2024-04-26T16:35:44.375599+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[21463]: audit 2024-04-26T16:35:44.375599+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: cluster 2024-04-26T16:35:43.903860+0000 mgr.y (mgr.24401) 705 : cluster [DBG] pgmap v1185: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:45.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: cluster 2024-04-26T16:35:43.903860+0000 mgr.y (mgr.24401) 705 : cluster [DBG] pgmap v1185: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: audit 2024-04-26T16:35:44.365288+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: audit 2024-04-26T16:35:44.365288+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: cluster 2024-04-26T16:35:44.373825+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: cluster 2024-04-26T16:35:44.373825+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: audit 2024-04-26T16:35:44.374821+0000 mon.c (mon.2) 797 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: audit 2024-04-26T16:35:44.374821+0000 mon.c (mon.2) 797 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: audit 2024-04-26T16:35:44.375599+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.468 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:45 smithi050 bash[29234]: audit 2024-04-26T16:35:44.375599+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.482 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:45.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:45.614 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:45.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:45.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: cluster 2024-04-26T16:35:43.903860+0000 mgr.y (mgr.24401) 705 : cluster [DBG] pgmap v1185: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: cluster 2024-04-26T16:35:43.903860+0000 mgr.y (mgr.24401) 705 : cluster [DBG] pgmap v1185: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: audit 2024-04-26T16:35:44.365288+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: audit 2024-04-26T16:35:44.365288+0000 mon.a (mon.0) 3622 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: cluster 2024-04-26T16:35:44.373825+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: cluster 2024-04-26T16:35:44.373825+0000 mon.a (mon.0) 3623 : cluster [DBG] osdmap e764: 8 total, 8 up, 8 in 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: audit 2024-04-26T16:35:44.374821+0000 mon.c (mon.2) 797 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: audit 2024-04-26T16:35:44.374821+0000 mon.c (mon.2) 797 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: audit 2024-04-26T16:35:44.375599+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:45 smithi190 bash[23868]: audit 2024-04-26T16:35:44.375599+0000 mon.a (mon.0) 3624 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]: dispatch 2024-04-26T16:35:45.744 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:45.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:45.872 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:45.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:45.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:46.002 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:46.106 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:46.130 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:46.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:46.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:46.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:46.398 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:46.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:46.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:46.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:46.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:46.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-26T16:35:46.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: cluster 2024-04-26T16:35:45.365525+0000 mon.a (mon.0) 3625 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:46.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: cluster 2024-04-26T16:35:45.365525+0000 mon.a (mon.0) 3625 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:46.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: audit 2024-04-26T16:35:45.367450+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]': finished 2024-04-26T16:35:46.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: audit 2024-04-26T16:35:45.367450+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]': finished 2024-04-26T16:35:46.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: cluster 2024-04-26T16:35:45.372171+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-04-26T16:35:46.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: cluster 2024-04-26T16:35:45.372171+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-04-26T16:35:46.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: cluster 2024-04-26T16:35:46.355980+0000 mon.a (mon.0) 3628 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:46.643 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:46 smithi190 bash[23868]: cluster 2024-04-26T16:35:46.355980+0000 mon.a (mon.0) 3628 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:46.655 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:46.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:46.770 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: cluster 2024-04-26T16:35:45.365525+0000 mon.a (mon.0) 3625 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:46.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: cluster 2024-04-26T16:35:45.365525+0000 mon.a (mon.0) 3625 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:46.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: audit 2024-04-26T16:35:45.367450+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]': finished 2024-04-26T16:35:46.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: audit 2024-04-26T16:35:45.367450+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]': finished 2024-04-26T16:35:46.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: cluster 2024-04-26T16:35:45.372171+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-04-26T16:35:46.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: cluster 2024-04-26T16:35:45.372171+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-04-26T16:35:46.771 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: cluster 2024-04-26T16:35:46.355980+0000 mon.a (mon.0) 3628 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[29234]: cluster 2024-04-26T16:35:46.355980+0000 mon.a (mon.0) 3628 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: cluster 2024-04-26T16:35:45.365525+0000 mon.a (mon.0) 3625 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: cluster 2024-04-26T16:35:45.365525+0000 mon.a (mon.0) 3625 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: audit 2024-04-26T16:35:45.367450+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]': finished 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: audit 2024-04-26T16:35:45.367450+0000 mon.a (mon.0) 3626 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-132", "mode": "writeback"}]': finished 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: cluster 2024-04-26T16:35:45.372171+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: cluster 2024-04-26T16:35:45.372171+0000 mon.a (mon.0) 3627 : cluster [DBG] osdmap e765: 8 total, 8 up, 8 in 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: cluster 2024-04-26T16:35:46.355980+0000 mon.a (mon.0) 3628 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:46.772 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:46 smithi050 bash[21463]: cluster 2024-04-26T16:35:46.355980+0000 mon.a (mon.0) 3628 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:35:46.783 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:46.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:46.914 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:46.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:47.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:47.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:47.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:47.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:47.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:47.304 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:47.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:47.437 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:47.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:47.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:47.552 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:35:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:35:47.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:47 smithi190 bash[23868]: cluster 2024-04-26T16:35:45.904671+0000 mgr.y (mgr.24401) 706 : cluster [DBG] pgmap v1188: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:47.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:47 smithi190 bash[23868]: cluster 2024-04-26T16:35:45.904671+0000 mgr.y (mgr.24401) 706 : cluster [DBG] pgmap v1188: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:47.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:47 smithi190 bash[23868]: cluster 2024-04-26T16:35:46.395099+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-04-26T16:35:47.553 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:47 smithi190 bash[23868]: cluster 2024-04-26T16:35:46.395099+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-04-26T16:35:47.569 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[21463]: cluster 2024-04-26T16:35:45.904671+0000 mgr.y (mgr.24401) 706 : cluster [DBG] pgmap v1188: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[21463]: cluster 2024-04-26T16:35:45.904671+0000 mgr.y (mgr.24401) 706 : cluster [DBG] pgmap v1188: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[21463]: cluster 2024-04-26T16:35:46.395099+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[21463]: cluster 2024-04-26T16:35:46.395099+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[29234]: cluster 2024-04-26T16:35:45.904671+0000 mgr.y (mgr.24401) 706 : cluster [DBG] pgmap v1188: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[29234]: cluster 2024-04-26T16:35:45.904671+0000 mgr.y (mgr.24401) 706 : cluster [DBG] pgmap v1188: 204 pgs: 28 creating+peering, 176 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[29234]: cluster 2024-04-26T16:35:46.395099+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-04-26T16:35:47.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:47 smithi050 bash[29234]: cluster 2024-04-26T16:35:46.395099+0000 mon.a (mon.0) 3629 : cluster [DBG] osdmap e766: 8 total, 8 up, 8 in 2024-04-26T16:35:47.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:47.700 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:47.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:47.829 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:47.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:47.960 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:47.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:48.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:48.089 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:48.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:48.221 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:48.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:48.350 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:48.453 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:48.460 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- 172.21.15.50:0/2153766924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f810fcb0 msgr2=0x7f20f81120e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2153766924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f810fcb0 0x7f20f81120e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7f20f000b0a0 tx=0x7f20f000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- 172.21.15.50:0/2153766924 shutdown_connections 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2153766924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20f8104340 0x7f20f806bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2153766924 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f810fcb0 0x7f20f81120e0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2153766924 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 0x7f20f806c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- 172.21.15.50:0/2153766924 >> 172.21.15.50:0/2153766924 conn(0x7f20f8079080 msgr2=0x7f20f8079490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- 172.21.15.50:0/2153766924 shutdown_connections 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- 172.21.15.50:0/2153766924 wait complete. 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 librados: starting msgr at 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 librados: starting objecter 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 Processor -- start 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- start start 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 librados: setting wanted keys 2024-04-26T16:35:48.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 librados: calling monclient init 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 0x7f20f819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f8104340 0x7f20f819b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20f810fcb0 0x7f20f819fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8114d90 con 0x7f20f806c070 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f20f8114c10 con 0x7f20f810fcb0 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20fdb99700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f20f8114f10 con 0x7f20f8104340 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20f77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 0x7f20f819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20f77fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 0x7f20f819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57550/0 (socket says 172.21.15.50:57550) 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20f77fe700 1 -- 172.21.15.50:0/2594043879 learned_addr learned my addr 172.21.15.50:0/2594043879 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.462+0000 7f20f6ffd700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f8104340 0x7f20f819b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f7fff700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20f810fcb0 0x7f20f819fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f6ffd700 1 -- 172.21.15.50:0/2594043879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20f810fcb0 msgr2=0x7f20f819fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f6ffd700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20f810fcb0 0x7f20f819fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f6ffd700 1 -- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 msgr2=0x7f20f819b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f6ffd700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 0x7f20f819b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f6ffd700 1 -- 172.21.15.50:0/2594043879 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f81a0330 con 0x7f20f8104340 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f6ffd700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f8104340 0x7f20f819b860 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 10 client.?.objecter ms_handle_connect 0x7f20f8104340 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec00cb70 con 0x7f20f8104340 2024-04-26T16:35:48.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f81a0620 con 0x7f20f8104340 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20fdb99700 10 client.41612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f81a7ea0 con 0x7f20f8104340 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20fdb99700 1 librados: init done 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f20ec004480 con 0x7f20f8104340 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ec003c10 con 0x7f20f8104340 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e4005120 con 0x7f20f8104340 2024-04-26T16:35:48.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f20ec003db0 con 0x7f20f8104340 2024-04-26T16:35:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f20e0071f80 0x7f20e0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(768..768 src has 577..768) ==== 9919+0+0 (secure 0 0 0) 0x7f20ec091c10 con 0x7f20f8104340 2024-04-26T16:35:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 10 client.41612.objecter ms_dispatch 0x7f20f8001550 osd_map(768..768 src has 577..768) 2024-04-26T16:35:48.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 3 client.41612.objecter handle_osd_map got epochs [768,768] > 0 2024-04-26T16:35:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f4ff9700 3 client.41612.objecter handle_osd_map decoding full epoch 768 2024-04-26T16:35:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.466+0000 7f20f77fe700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f20e0071f80 0x7f20e0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.470+0000 7f20f4ff9700 20 client.41612.objecter dump_active .. 0 homeless 2024-04-26T16:35:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.470+0000 7f20f77fe700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f20e0071f80 0x7f20e0074440 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f20e800bcb0 tx=0x7f20e800b480 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.470+0000 7f20f4ff9700 10 client.41612.objecter ms_handle_connect 0x7f20e0071f80 2024-04-26T16:35:48.471 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.474+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f20ec05d3b0 con 0x7f20f8104340 2024-04-26T16:35:48.477 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:48.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:48.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: audit 2024-04-26T16:35:47.246276+0000 mgr.y (mgr.24401) 707 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:48.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: audit 2024-04-26T16:35:47.246276+0000 mgr.y (mgr.24401) 707 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:48.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: cluster 2024-04-26T16:35:47.408609+0000 mon.a (mon.0) 3630 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-04-26T16:35:48.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: cluster 2024-04-26T16:35:47.408609+0000 mon.a (mon.0) 3630 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-04-26T16:35:48.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: audit 2024-04-26T16:35:47.614143+0000 mon.c (mon.2) 798 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: audit 2024-04-26T16:35:47.614143+0000 mon.c (mon.2) 798 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: audit 2024-04-26T16:35:47.614870+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.592 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:48 smithi190 bash[23868]: audit 2024-04-26T16:35:47.614870+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/3904072271 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 msgr2=0x7f0d1c06b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- 172.21.15.50:0/3904072271 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 0x7f0d1c06b590 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x7f0d10009990 tx=0x7f0d10009ca0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/3904072271 shutdown_connections 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- 172.21.15.50:0/3904072271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d1c1094a0 0x7f0d1c109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- 172.21.15.50:0/3904072271 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 0x7f0d1c1102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- 172.21.15.50:0/3904072271 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 0x7f0d1c06b590 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/3904072271 >> 172.21.15.50:0/3904072271 conn(0x7f0d1c0792d0 msgr2=0x7f0d1c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/3904072271 shutdown_connections 2024-04-26T16:35:48.594 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/3904072271 wait complete. 2024-04-26T16:35:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 librados: starting msgr at 2024-04-26T16:35:48.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 librados: starting objecter 2024-04-26T16:35:48.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 Processor -- start 2024-04-26T16:35:48.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- start start 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 librados: setting wanted keys 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 librados: calling monclient init 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 0x7f0d1c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 0x7f0d1c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d1c1094a0 0x7f0d1c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1c112ae0 con 0x7f0d1c06bad0 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1c112960 con 0x7f0d1c1094a0 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1c112c60 con 0x7f0d1c06b130 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 0x7f0d1c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 0x7f0d1c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1affd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 0x7f0d1c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57570/0 (socket says 172.21.15.50:57570) 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1affd700 1 -- 172.21.15.50:0/2007540977 learned_addr learned my addr 172.21.15.50:0/2007540977 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:35:48.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d1c1094a0 0x7f0d1c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d1c1094a0 msgr2=0x7f0d1c19d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d1c1094a0 0x7f0d1c19d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 msgr2=0x7f0d1c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 0x7f0d1c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d1c19e0b0 con 0x7f0d1c06b130 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d1c1094a0 0x7f0d1c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0d1c06bad0 0x7f0d1c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 0x7f0d1c1990a0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c970 tx=0x7f0d0c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d18ff9700 10 client.?.objecter ms_handle_connect 0x7f0d1c06b130 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d18ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:35:48.598 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c014070 con 0x7f0d1c06b130 2024-04-26T16:35:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0d0c004440 con 0x7f0d1c06b130 2024-04-26T16:35:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c003920 con 0x7f0d1c06b130 2024-04-26T16:35:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1c19e3a0 con 0x7f0d1c06b130 2024-04-26T16:35:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 10 client.41618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:35:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d1c1a5c20 con 0x7f0d1c06b130 2024-04-26T16:35:48.599 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 librados: init done 2024-04-26T16:35:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.598+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f0d1c06a110 con 0x7f0d1c06b130 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0d0c017070 con 0x7f0d1c06b130 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d04071f80 0x7f0d04074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(768..768 src has 577..768) ==== 9919+0+0 (secure 0 0 0) 0x7f0d0c091450 con 0x7f0d1c06b130 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 osd_map(768..768 src has 577..768) 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 3 client.41618.objecter handle_osd_map got epochs [768,768] > 0 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 3 client.41618.objecter handle_osd_map decoding full epoch 768 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d04071f80 0x7f0d04074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 20 client.41618.objecter dump_active .. 0 homeless 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v768) ==== 54+0+16 (secure 0 0 0) 0x7f0d0c01e030 con 0x7f0d1c06b130 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 10 client.41618.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 1 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d1c064030 0x7f0d1c064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.601 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 1 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 0 1 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 4 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 10 client.41618.objecter _send_command 1 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c1a8f80 con 0x7f0d1c064030 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d04071f80 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d04071f80 0x7f0d04074440 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c10b3e0 tx=0x7f0d1000b630 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d1c064030 0x7f0d1c064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d1c064030 0x7f0d1c064410 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c064030 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c1a8f80 con 0x7f0d1c064030 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 1: 0 ) 2024-04-26T16:35:48.602 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 1 = system:0 2024-04-26T16:35:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 4 2024-04-26T16:35:48.603 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.602+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 0 1 2024-04-26T16:35:48.608 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 10 client.41618.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 2 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 2 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c19a400 osd=0 3 2024-04-26T16:35:48.618 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 0 2 2024-04-26T16:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 4 2024-04-26T16:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 10 client.41618.objecter _send_command 2 2024-04-26T16:35:48.619 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c063ab0 con 0x7f0d1c064030 2024-04-26T16:35:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c063ab0 con 0x7f0d1c064030 2024-04-26T16:35:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 4 2024-04-26T16:35:48.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 0 2 2024-04-26T16:35:48.622 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 10 client.41618.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 3 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d1c02d310 0x7f0d1c02d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 3 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 1 3 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 4 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 10 client.41618.objecter _send_command 3 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c104330 con 0x7f0d1c02d310 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.622+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d1c02d310 0x7f0d1c02d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c02d310 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d1c02d310 0x7f0d1c02d710 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c104330 con 0x7f0d1c02d310 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 3: 0 ) 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 3 = system:0 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 4 2024-04-26T16:35:48.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.626+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 1 3 2024-04-26T16:35:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: audit 2024-04-26T16:35:47.246276+0000 mgr.y (mgr.24401) 707 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: audit 2024-04-26T16:35:47.246276+0000 mgr.y (mgr.24401) 707 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: cluster 2024-04-26T16:35:47.408609+0000 mon.a (mon.0) 3630 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-04-26T16:35:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: cluster 2024-04-26T16:35:47.408609+0000 mon.a (mon.0) 3630 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-04-26T16:35:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: audit 2024-04-26T16:35:47.614143+0000 mon.c (mon.2) 798 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.629 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: audit 2024-04-26T16:35:47.614143+0000 mon.c (mon.2) 798 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: audit 2024-04-26T16:35:47.614870+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[21463]: audit 2024-04-26T16:35:47.614870+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: audit 2024-04-26T16:35:47.246276+0000 mgr.y (mgr.24401) 707 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: audit 2024-04-26T16:35:47.246276+0000 mgr.y (mgr.24401) 707 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: cluster 2024-04-26T16:35:47.408609+0000 mon.a (mon.0) 3630 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: cluster 2024-04-26T16:35:47.408609+0000 mon.a (mon.0) 3630 : cluster [DBG] osdmap e767: 8 total, 8 up, 8 in 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: audit 2024-04-26T16:35:47.614143+0000 mon.c (mon.2) 798 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: audit 2024-04-26T16:35:47.614143+0000 mon.c (mon.2) 798 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: audit 2024-04-26T16:35:47.614870+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:48 smithi050 bash[29234]: audit 2024-04-26T16:35:47.614870+0000 mon.a (mon.0) 3631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 10 client.41618.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 4 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 4 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c10bc00 osd=1 3 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 1 4 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 4 2024-04-26T16:35:48.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 10 client.41618.objecter _send_command 4 2024-04-26T16:35:48.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.642+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c02dc50 con 0x7f0d1c02d310 2024-04-26T16:35:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c02dc50 con 0x7f0d1c02d310 2024-04-26T16:35:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 4 2024-04-26T16:35:48.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 1 4 2024-04-26T16:35:48.642 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:35:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 10 client.41618.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 5 2024-04-26T16:35:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d1c1ad540 0x7f0d1c1af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 5 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 2 5 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 4 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 10 client.41618.objecter _send_command 5 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c02e740 con 0x7f0d1c1ad540 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d1c1ad540 0x7f0d1c1af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c1ad540 2024-04-26T16:35:48.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d1c1ad540 0x7f0d1c1af930 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c02e740 con 0x7f0d1c1ad540 2024-04-26T16:35:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 5: 0 ) 2024-04-26T16:35:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 5 = system:0 2024-04-26T16:35:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 4 2024-04-26T16:35:48.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.646+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 2 5 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 10 client.41618.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 6 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 6 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c02e5d0 osd=2 3 2024-04-26T16:35:48.661 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 2 6 2024-04-26T16:35:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 4 2024-04-26T16:35:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 10 client.41618.objecter _send_command 6 2024-04-26T16:35:48.662 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.662+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c02e050 con 0x7f0d1c1ad540 2024-04-26T16:35:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c02e050 con 0x7f0d1c1ad540 2024-04-26T16:35:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 4 2024-04-26T16:35:48.663 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 2 6 2024-04-26T16:35:48.663 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 10 client.41618.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 7 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d1c1a8ae0 0x7f0d1c1b3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.664 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 7 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 3 7 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 4 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 10 client.41618.objecter _send_command 7 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c1b6040 con 0x7f0d1c1a8ae0 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d1c1a8ae0 0x7f0d1c1b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c1a8ae0 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d1c1a8ae0 0x7f0d1c1b3000 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.665 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c1b6040 con 0x7f0d1c1a8ae0 2024-04-26T16:35:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 7: 0 ) 2024-04-26T16:35:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 7 = system:0 2024-04-26T16:35:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 4 2024-04-26T16:35:48.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.666+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 3 7 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 10 client.41618.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 8 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 8 2024-04-26T16:35:48.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c02e450 osd=3 3 2024-04-26T16:35:48.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 3 8 2024-04-26T16:35:48.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 4 2024-04-26T16:35:48.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 10 client.41618.objecter _send_command 8 2024-04-26T16:35:48.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c1a8170 con 0x7f0d1c1a8ae0 2024-04-26T16:35:48.683 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c1a8170 con 0x7f0d1c1a8ae0 2024-04-26T16:35:48.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 4 2024-04-26T16:35:48.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 3 8 2024-04-26T16:35:48.684 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:35:48.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 10 client.41618.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 9 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d1c1b0b50 0x7f0d1c1b3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 9 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 4 9 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 4 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 10 client.41618.objecter _send_command 9 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c1b9a30 con 0x7f0d1c1b0b50 2024-04-26T16:35:48.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d1c1b0b50 0x7f0d1c1b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d1c1b0b50 0x7f0d1c1b3660 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.686+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c1b0b50 2024-04-26T16:35:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.690+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c1b9a30 con 0x7f0d1c1b0b50 2024-04-26T16:35:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.690+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 9: 0 ) 2024-04-26T16:35:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.690+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 9 = system:0 2024-04-26T16:35:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.690+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 4 2024-04-26T16:35:48.687 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.690+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 4 9 2024-04-26T16:35:48.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 10 client.41618.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 10 2024-04-26T16:35:48.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 10 2024-04-26T16:35:48.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1a8170 osd=4 3 2024-04-26T16:35:48.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 4 10 2024-04-26T16:35:48.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 4 2024-04-26T16:35:48.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 10 client.41618.objecter _send_command 10 2024-04-26T16:35:48.705 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.706+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c1b3ba0 con 0x7f0d1c1b0b50 2024-04-26T16:35:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c1b3ba0 con 0x7f0d1c1b0b50 2024-04-26T16:35:48.708 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 4 2024-04-26T16:35:48.709 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 4 10 2024-04-26T16:35:48.709 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 10 client.41618.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 11 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d1c1b42c0 0x7f0d1c1b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 11 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 5 11 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 4 2024-04-26T16:35:48.710 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 10 client.41618.objecter _send_command 11 2024-04-26T16:35:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c1bcc10 con 0x7f0d1c1b42c0 2024-04-26T16:35:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.710+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d1c1b42c0 0x7f0d1c1b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d1affd700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d1c1b42c0 0x7f0d1c1b4720 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.711 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c1b42c0 2024-04-26T16:35:48.711 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c1bcc10 con 0x7f0d1c1b42c0 2024-04-26T16:35:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 11: 0 ) 2024-04-26T16:35:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 11 = system:0 2024-04-26T16:35:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 4 2024-04-26T16:35:48.712 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.714+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 5 11 2024-04-26T16:35:48.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.722+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f20e4005410 con 0x7f20f8104340 2024-04-26T16:35:48.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 10 client.41618.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 12 2024-04-26T16:35:48.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 12 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1b3ba0 osd=5 3 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 5 12 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 4 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 10 client.41618.objecter _send_command 12 2024-04-26T16:35:48.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.730+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c1bcc10 con 0x7f0d1c1b42c0 2024-04-26T16:35:48.731 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:48.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c1bcc10 con 0x7f0d1c1b42c0 2024-04-26T16:35:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 4 2024-04-26T16:35:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 5 12 2024-04-26T16:35:48.734 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:35:48.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 10 client.41618.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 13 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f0d1c1ba7b0 0x7f0d1c1babf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 13 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 6 13 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 4 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 10 client.41618.objecter _send_command 13 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c1bff80 con 0x7f0d1c1ba7b0 2024-04-26T16:35:48.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f0d1c1ba7b0 0x7f0d1c1babf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d1bfff700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f0d1c1ba7b0 0x7f0d1c1babf0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c1ba7b0 2024-04-26T16:35:48.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c1bff80 con 0x7f0d1c1ba7b0 2024-04-26T16:35:48.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 13: 0 ) 2024-04-26T16:35:48.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 13 = system:0 2024-04-26T16:35:48.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 4 2024-04-26T16:35:48.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.738+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 6 13 2024-04-26T16:35:48.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 10 client.41618.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 14 2024-04-26T16:35:48.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 14 2024-04-26T16:35:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1bcc10 osd=6 3 2024-04-26T16:35:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 6 14 2024-04-26T16:35:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 4 2024-04-26T16:35:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 10 client.41618.objecter _send_command 14 2024-04-26T16:35:48.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.758+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c1bb240 con 0x7f0d1c1ba7b0 2024-04-26T16:35:48.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c1bb240 con 0x7f0d1c1ba7b0 2024-04-26T16:35:48.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 4 2024-04-26T16:35:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 6 14 2024-04-26T16:35:48.759 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:35:48.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 10 client.41618.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 15 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d1c1bb960 0x7f0d1c1bbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 15 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 7 15 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 4 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 10 client.41618.objecter _send_command 15 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f0d1c1bc430 con 0x7f0d1c1bb960 2024-04-26T16:35:48.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d1c1bb960 0x7f0d1c1bbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:35:48.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d18ff9700 10 client.41618.objecter ms_handle_connect 0x7f0d1c1bb960 2024-04-26T16:35:48.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.762+0000 7f0d1b7fe700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d1c1bb960 0x7f0d1c1bbdc0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:35:48.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.766+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0d1c1bc430 con 0x7f0d1c1bb960 2024-04-26T16:35:48.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.766+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 15: 0 ) 2024-04-26T16:35:48.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.766+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 15 = system:0 2024-04-26T16:35:48.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.766+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 4 2024-04-26T16:35:48.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.766+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 7 15 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 10 client.41618.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign -1 16 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 20 client.41618.objecter _get_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_remove -1 16 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 20 client.41618.objecter get_session s=0x7f0d1c1bb240 osd=7 3 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 15 client.41618.objecter _session_command_op_assign 7 16 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 4 2024-04-26T16:35:48.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 10 client.41618.objecter _send_command 16 2024-04-26T16:35:48.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.782+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f0d1c1bc430 con 0x7f0d1c1bb960 2024-04-26T16:35:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d18ff9700 1 -- 172.21.15.50:0/2007540977 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f0d1c1bc430 con 0x7f0d1c1bb960 2024-04-26T16:35:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d18ff9700 10 client.41618.objecter ms_dispatch 0x7f0d1c001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:35:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d18ff9700 10 client.41618.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:35:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d18ff9700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 4 2024-04-26T16:35:48.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d18ff9700 15 client.41618.objecter _session_command_op_remove 7 16 2024-04-26T16:35:48.786 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:35:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 librados: watch_flush enter 2024-04-26T16:35:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 librados: watch_flush exit 2024-04-26T16:35:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.0 2024-04-26T16:35:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d1c064030 msgr2=0x7f0d1c064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f0d1c064030 0x7f0d1c064410 crc :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c19a400 osd=0 1 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.1 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d1c02d310 msgr2=0x7f0d1c02d710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f0d1c02d310 0x7f0d1c02d710 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c10bc00 osd=1 1 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.2 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d1c1ad540 msgr2=0x7f0d1c1af930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f0d1c1ad540 0x7f0d1c1af930 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e5d0 osd=2 1 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.3 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d1c1a8ae0 msgr2=0x7f0d1c1b3000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f0d1c1a8ae0 0x7f0d1c1b3000 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c02e450 osd=3 1 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.4 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d1c1b0b50 msgr2=0x7f0d1c1b3660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d1bfff700 1 -- 172.21.15.50:0/2007540977 reap_dead start 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d1bfff700 1 -- 172.21.15.50:0/2007540977 reap_dead start 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f0d1c1b0b50 0x7f0d1c1b3660 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1a8170 osd=4 1 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.5 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d1c1b42c0 msgr2=0x7f0d1c1b4720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f0d1c1b42c0 0x7f0d1c1b4720 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1b3ba0 osd=5 1 2024-04-26T16:35:48.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.6 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f0d1c1ba7b0 msgr2=0x7f0d1c1babf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f0d1c1ba7b0 0x7f0d1c1babf0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bcc10 osd=6 1 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter close_session for osd.7 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d1c1bb960 msgr2=0x7f0d1c1bbdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f0d1c1bb960 0x7f0d1c1bbdc0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter put_session s=0x7f0d1c1bb240 osd=7 1 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 20 client.41618.objecter shutdown clearing up homeless session... 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 10 client.41618.objecter successfully canceled tick 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d04071f80 msgr2=0x7f0d04074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0d04071f80 0x7f0d04074440 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c10b3e0 tx=0x7f0d1000b630 comp rx=0 tx=0).stop 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 msgr2=0x7f0d1c1990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d221d4700 1 --2- 172.21.15.50:0/2007540977 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0d1c06b130 0x7f0d1c1990a0 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c00c970 tx=0x7f0d0c00cc80 comp rx=0 tx=0).stop 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.786+0000 7f0d1bfff700 1 -- 172.21.15.50:0/2007540977 reap_dead start 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.790+0000 7f0d1bfff700 1 -- 172.21.15.50:0/2007540977 reap_dead start 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.790+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 shutdown_connections 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.790+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 >> 172.21.15.50:0/2007540977 conn(0x7f0d1c0792d0 msgr2=0x7f0d1c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.790+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 shutdown_connections 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.790+0000 7f0d221d4700 1 -- 172.21.15.50:0/2007540977 wait complete. 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:48.790+0000 7f0d221d4700 1 librados: shutdown 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:35:48.791 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:35:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:35:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:35:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:35:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:35:48.792 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:35:48.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:48.852 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:48.956 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:48.982 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:48.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:49.085 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:49.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:49.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:49.241 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:49.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:49.372 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.426+0000 7f20f4ff9700 1 -- 172.21.15.50:0/2594043879 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v769) ==== 97+0+0 (secure 0 0 0) 0x7f20ec061060 con 0x7f20f8104340 2024-04-26T16:35:49.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 10 librados: watch_flush enter 2024-04-26T16:35:49.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 10 librados: watch_flush exit 2024-04-26T16:35:49.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 20 client.41612.objecter shutdown clearing up homeless session... 2024-04-26T16:35:49.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 10 client.41612.objecter successfully canceled tick 2024-04-26T16:35:49.427 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f20e0071f80 msgr2=0x7f20e0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:49.428 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f20e0071f80 0x7f20e0074440 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f20e800bcb0 tx=0x7f20e800b480 comp rx=0 tx=0).stop 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f8104340 msgr2=0x7f20f819b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f8104340 0x7f20f819b860 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00ed00 comp rx=0 tx=0).stop 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 shutdown_connections 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f20e0071f80 0x7f20e0074440 secure :-1 s=CLOSED pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7f20e800bcb0 tx=0x7f20e800b480 comp rx=0 tx=0).stop 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f20f810fcb0 0x7f20f819fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f20f806c070 0x7f20f819b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:35:49.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 --2- 172.21.15.50:0/2594043879 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f20f8104340 0x7f20f819b860 secure :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7f20ec00e9f0 tx=0x7f20ec00ed00 comp rx=0 tx=0).stop 2024-04-26T16:35:49.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.430+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 >> 172.21.15.50:0/2594043879 conn(0x7f20f8079080 msgr2=0x7f20f80ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:35:49.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.434+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 shutdown_connections 2024-04-26T16:35:49.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.434+0000 7f20fdb99700 1 -- 172.21.15.50:0/2594043879 wait complete. 2024-04-26T16:35:49.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:35:49.434+0000 7f20fdb99700 1 librados: shutdown 2024-04-26T16:35:49.431 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:35:49.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: cluster 2024-04-26T16:35:47.905618+0000 mgr.y (mgr.24401) 708 : cluster [DBG] pgmap v1191: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: cluster 2024-04-26T16:35:47.905618+0000 mgr.y (mgr.24401) 708 : cluster [DBG] pgmap v1191: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.408857+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.408857+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: cluster 2024-04-26T16:35:48.431057+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: cluster 2024-04-26T16:35:48.431057+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.437501+0000 mon.c (mon.2) 799 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.437501+0000 mon.c (mon.2) 799 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.438113+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.438113+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.603642+0000 mon.c (mon.2) 800 : audit [DBG] from='client.? 172.21.15.50:0/2007540977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.603642+0000 mon.c (mon.2) 800 : audit [DBG] from='client.? 172.21.15.50:0/2007540977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.724113+0000 mon.c (mon.2) 801 : audit [INF] from='client.? 172.21.15.50:0/2594043879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.724113+0000 mon.c (mon.2) 801 : audit [INF] from='client.? 172.21.15.50:0/2594043879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.724466+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:49 smithi190 bash[23868]: audit 2024-04-26T16:35:48.724466+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:49.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:49.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:49.735 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: cluster 2024-04-26T16:35:47.905618+0000 mgr.y (mgr.24401) 708 : cluster [DBG] pgmap v1191: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: cluster 2024-04-26T16:35:47.905618+0000 mgr.y (mgr.24401) 708 : cluster [DBG] pgmap v1191: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.408857+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.408857+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: cluster 2024-04-26T16:35:48.431057+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-04-26T16:35:49.748 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: cluster 2024-04-26T16:35:48.431057+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.437501+0000 mon.c (mon.2) 799 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.437501+0000 mon.c (mon.2) 799 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.438113+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.438113+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.603642+0000 mon.c (mon.2) 800 : audit [DBG] from='client.? 172.21.15.50:0/2007540977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.603642+0000 mon.c (mon.2) 800 : audit [DBG] from='client.? 172.21.15.50:0/2007540977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.724113+0000 mon.c (mon.2) 801 : audit [INF] from='client.? 172.21.15.50:0/2594043879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.749 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.724113+0000 mon.c (mon.2) 801 : audit [INF] from='client.? 172.21.15.50:0/2594043879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.724466+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[21463]: audit 2024-04-26T16:35:48.724466+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.750 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: cluster 2024-04-26T16:35:47.905618+0000 mgr.y (mgr.24401) 708 : cluster [DBG] pgmap v1191: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:49.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: cluster 2024-04-26T16:35:47.905618+0000 mgr.y (mgr.24401) 708 : cluster [DBG] pgmap v1191: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:49.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.408857+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:49.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.408857+0000 mon.a (mon.0) 3632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:49.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: cluster 2024-04-26T16:35:48.431057+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-04-26T16:35:49.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: cluster 2024-04-26T16:35:48.431057+0000 mon.a (mon.0) 3633 : cluster [DBG] osdmap e768: 8 total, 8 up, 8 in 2024-04-26T16:35:49.751 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.437501+0000 mon.c (mon.2) 799 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.437501+0000 mon.c (mon.2) 799 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.438113+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.438113+0000 mon.a (mon.0) 3634 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.603642+0000 mon.c (mon.2) 800 : audit [DBG] from='client.? 172.21.15.50:0/2007540977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.603642+0000 mon.c (mon.2) 800 : audit [DBG] from='client.? 172.21.15.50:0/2007540977' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.724113+0000 mon.c (mon.2) 801 : audit [INF] from='client.? 172.21.15.50:0/2594043879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.752 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.724113+0000 mon.c (mon.2) 801 : audit [INF] from='client.? 172.21.15.50:0/2594043879' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.724466+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.753 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:49 smithi050 bash[29234]: audit 2024-04-26T16:35:48.724466+0000 mon.a (mon.0) 3635 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:35:49.764 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:49.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:49.894 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:49.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:49.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:49.997 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.030 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:50.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:50.131 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:50.156 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:50.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:50.258 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:50.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:50.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:50.388 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:50.416 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:50.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:50.546 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:50.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: cluster 2024-04-26T16:35:49.409527+0000 mon.a (mon.0) 3636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:50.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: cluster 2024-04-26T16:35:49.409527+0000 mon.a (mon.0) 3636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:50.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: audit 2024-04-26T16:35:49.412086+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:50.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: audit 2024-04-26T16:35:49.412086+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:50.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: audit 2024-04-26T16:35:49.412349+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: audit 2024-04-26T16:35:49.412349+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: cluster 2024-04-26T16:35:49.420735+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[29234]: cluster 2024-04-26T16:35:49.420735+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: cluster 2024-04-26T16:35:49.409527+0000 mon.a (mon.0) 3636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: cluster 2024-04-26T16:35:49.409527+0000 mon.a (mon.0) 3636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: audit 2024-04-26T16:35:49.412086+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: audit 2024-04-26T16:35:49.412086+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: audit 2024-04-26T16:35:49.412349+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: audit 2024-04-26T16:35:49.412349+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: cluster 2024-04-26T16:35:49.420735+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-04-26T16:35:50.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:50 smithi050 bash[21463]: cluster 2024-04-26T16:35:49.420735+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-04-26T16:35:50.649 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:50.676 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:50.778 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:50.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:50.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: cluster 2024-04-26T16:35:49.409527+0000 mon.a (mon.0) 3636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: cluster 2024-04-26T16:35:49.409527+0000 mon.a (mon.0) 3636 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: audit 2024-04-26T16:35:49.412086+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: audit 2024-04-26T16:35:49.412086+0000 mon.a (mon.0) 3637 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: audit 2024-04-26T16:35:49.412349+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: audit 2024-04-26T16:35:49.412349+0000 mon.a (mon.0) 3638 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: cluster 2024-04-26T16:35:49.420735+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-04-26T16:35:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:50 smithi190 bash[23868]: cluster 2024-04-26T16:35:49.420735+0000 mon.a (mon.0) 3639 : cluster [DBG] osdmap e769: 8 total, 8 up, 8 in 2024-04-26T16:35:50.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:50.939 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:50.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:51.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:51.072 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:51.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:51.202 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:51.293 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:51.305 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:51.328 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:51.431 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:51.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:51.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:51.588 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:51.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: cluster 2024-04-26T16:35:49.906501+0000 mgr.y (mgr.24401) 709 : cluster [DBG] pgmap v1194: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:35:51.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: cluster 2024-04-26T16:35:49.906501+0000 mgr.y (mgr.24401) 709 : cluster [DBG] pgmap v1194: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:35:51.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: cluster 2024-04-26T16:35:50.443360+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-04-26T16:35:51.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: cluster 2024-04-26T16:35:50.443360+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:50.499210+0000 mon.c (mon.2) 802 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:50.499210+0000 mon.c (mon.2) 802 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:50.499710+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:50.499710+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:51.360616+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:51.360616+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: cluster 2024-04-26T16:35:51.364543+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: cluster 2024-04-26T16:35:51.364543+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:51.365720+0000 mon.c (mon.2) 803 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:51.365720+0000 mon.c (mon.2) 803 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:51.366458+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[29234]: audit 2024-04-26T16:35:51.366458+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: cluster 2024-04-26T16:35:49.906501+0000 mgr.y (mgr.24401) 709 : cluster [DBG] pgmap v1194: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: cluster 2024-04-26T16:35:49.906501+0000 mgr.y (mgr.24401) 709 : cluster [DBG] pgmap v1194: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: cluster 2024-04-26T16:35:50.443360+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: cluster 2024-04-26T16:35:50.443360+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:50.499210+0000 mon.c (mon.2) 802 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:50.499210+0000 mon.c (mon.2) 802 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:50.499710+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:50.499710+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:51.360616+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:51.360616+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: cluster 2024-04-26T16:35:51.364543+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: cluster 2024-04-26T16:35:51.364543+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-04-26T16:35:51.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:51.365720+0000 mon.c (mon.2) 803 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:51.365720+0000 mon.c (mon.2) 803 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:51.366458+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:51 smithi050 bash[21463]: audit 2024-04-26T16:35:51.366458+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.691 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:51.717 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:51.821 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:51.848 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:51.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: cluster 2024-04-26T16:35:49.906501+0000 mgr.y (mgr.24401) 709 : cluster [DBG] pgmap v1194: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:35:51.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: cluster 2024-04-26T16:35:49.906501+0000 mgr.y (mgr.24401) 709 : cluster [DBG] pgmap v1194: 204 pgs: 204 active+clean; 583 KiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 2 op/s 2024-04-26T16:35:51.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: cluster 2024-04-26T16:35:50.443360+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-04-26T16:35:51.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: cluster 2024-04-26T16:35:50.443360+0000 mon.a (mon.0) 3640 : cluster [DBG] osdmap e770: 8 total, 8 up, 8 in 2024-04-26T16:35:51.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:50.499210+0000 mon.c (mon.2) 802 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:50.499210+0000 mon.c (mon.2) 802 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:50.499710+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:50.499710+0000 mon.a (mon.0) 3641 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:51.360616+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:51.360616+0000 mon.a (mon.0) 3642 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: cluster 2024-04-26T16:35:51.364543+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: cluster 2024-04-26T16:35:51.364543+0000 mon.a (mon.0) 3643 : cluster [DBG] osdmap e771: 8 total, 8 up, 8 in 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:51.365720+0000 mon.c (mon.2) 803 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:51.365720+0000 mon.c (mon.2) 803 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:51.366458+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:51 smithi190 bash[23868]: audit 2024-04-26T16:35:51.366458+0000 mon.a (mon.0) 3644 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]: dispatch 2024-04-26T16:35:51.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:51.979 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:51.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:52.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:52.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:52.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:52.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:35:52.241 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:52.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:52.370 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:52.474 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:52.494 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:52 smithi190 bash[23868]: cluster 2024-04-26T16:35:52.361123+0000 mon.a (mon.0) 3645 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:52 smithi190 bash[23868]: cluster 2024-04-26T16:35:52.361123+0000 mon.a (mon.0) 3645 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:52 smithi190 bash[23868]: audit 2024-04-26T16:35:52.362974+0000 mon.a (mon.0) 3646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:52 smithi190 bash[23868]: audit 2024-04-26T16:35:52.362974+0000 mon.a (mon.0) 3646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:52 smithi190 bash[23868]: cluster 2024-04-26T16:35:52.366602+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-04-26T16:35:52.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:52 smithi190 bash[23868]: cluster 2024-04-26T16:35:52.366602+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-04-26T16:35:52.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:52.625 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:52.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:52.757 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:52.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:52.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:35:52.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[29234]: cluster 2024-04-26T16:35:52.361123+0000 mon.a (mon.0) 3645 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:52.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[29234]: cluster 2024-04-26T16:35:52.361123+0000 mon.a (mon.0) 3645 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:52.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[29234]: audit 2024-04-26T16:35:52.362974+0000 mon.a (mon.0) 3646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:52.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[29234]: audit 2024-04-26T16:35:52.362974+0000 mon.a (mon.0) 3646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:52.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[29234]: cluster 2024-04-26T16:35:52.366602+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-04-26T16:35:52.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[29234]: cluster 2024-04-26T16:35:52.366602+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-04-26T16:35:52.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[21463]: cluster 2024-04-26T16:35:52.361123+0000 mon.a (mon.0) 3645 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:52.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[21463]: cluster 2024-04-26T16:35:52.361123+0000 mon.a (mon.0) 3645 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:35:52.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[21463]: audit 2024-04-26T16:35:52.362974+0000 mon.a (mon.0) 3646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:52.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[21463]: audit 2024-04-26T16:35:52.362974+0000 mon.a (mon.0) 3646 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-132"}]': finished 2024-04-26T16:35:52.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[21463]: cluster 2024-04-26T16:35:52.366602+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-04-26T16:35:52.873 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:52 smithi050 bash[21463]: cluster 2024-04-26T16:35:52.366602+0000 mon.a (mon.0) 3647 : cluster [DBG] osdmap e772: 8 total, 8 up, 8 in 2024-04-26T16:35:52.886 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:52.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:52.990 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:53.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:53.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:53.141 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:53.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:53.271 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:53.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:53.401 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:53.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:53.530 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[21463]: cluster 2024-04-26T16:35:51.907843+0000 mgr.y (mgr.24401) 710 : cluster [DBG] pgmap v1197: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[21463]: cluster 2024-04-26T16:35:51.907843+0000 mgr.y (mgr.24401) 710 : cluster [DBG] pgmap v1197: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[21463]: cluster 2024-04-26T16:35:53.381143+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[21463]: cluster 2024-04-26T16:35:53.381143+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[29234]: cluster 2024-04-26T16:35:51.907843+0000 mgr.y (mgr.24401) 710 : cluster [DBG] pgmap v1197: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[29234]: cluster 2024-04-26T16:35:51.907843+0000 mgr.y (mgr.24401) 710 : cluster [DBG] pgmap v1197: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[29234]: cluster 2024-04-26T16:35:53.381143+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-04-26T16:35:53.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:53 smithi050 bash[29234]: cluster 2024-04-26T16:35:53.381143+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-04-26T16:35:53.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:53.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:53 smithi190 bash[23868]: cluster 2024-04-26T16:35:51.907843+0000 mgr.y (mgr.24401) 710 : cluster [DBG] pgmap v1197: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:35:53.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:53 smithi190 bash[23868]: cluster 2024-04-26T16:35:51.907843+0000 mgr.y (mgr.24401) 710 : cluster [DBG] pgmap v1197: 204 pgs: 1 active+clean+snaptrim, 203 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-04-26T16:35:53.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:53 smithi190 bash[23868]: cluster 2024-04-26T16:35:53.381143+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-04-26T16:35:53.642 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:53 smithi190 bash[23868]: cluster 2024-04-26T16:35:53.381143+0000 mon.a (mon.0) 3648 : cluster [DBG] osdmap e773: 8 total, 8 up, 8 in 2024-04-26T16:35:53.657 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:53.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:53.789 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:53.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:53.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:53.918 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:53.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:54.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:54.048 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:54.151 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.173 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:54.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:54.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:54.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:54.432 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:54.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.547 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:35:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:35:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:35:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:54.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:54.693 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:54.797 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.824 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:54.926 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.947 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:54.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:55.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:55.077 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:55.180 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:55.208 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:55.312 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:55.340 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:55.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:55.443 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:53.908973+0000 mgr.y (mgr.24401) 711 : cluster [DBG] pgmap v1200: 172 pgs: 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:53.908973+0000 mgr.y (mgr.24401) 711 : cluster [DBG] pgmap v1200: 172 pgs: 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:54.375229+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:54.375229+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: audit 2024-04-26T16:35:54.384029+0000 mon.c (mon.2) 804 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: audit 2024-04-26T16:35:54.384029+0000 mon.c (mon.2) 804 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: audit 2024-04-26T16:35:54.385443+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: audit 2024-04-26T16:35:54.385443+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:54.441560+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:54.441560+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:55.453 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:54.441848+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[21463]: cluster 2024-04-26T16:35:54.441848+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:53.908973+0000 mgr.y (mgr.24401) 711 : cluster [DBG] pgmap v1200: 172 pgs: 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:53.908973+0000 mgr.y (mgr.24401) 711 : cluster [DBG] pgmap v1200: 172 pgs: 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:54.375229+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:54.375229+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: audit 2024-04-26T16:35:54.384029+0000 mon.c (mon.2) 804 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: audit 2024-04-26T16:35:54.384029+0000 mon.c (mon.2) 804 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: audit 2024-04-26T16:35:54.385443+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: audit 2024-04-26T16:35:54.385443+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:54.441560+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:54.441560+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:55.454 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:54.441848+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:55.455 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:55 smithi050 bash[29234]: cluster 2024-04-26T16:35:54.441848+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:55.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:55.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:55.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:55.595 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:55.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:55.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:55.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:53.908973+0000 mgr.y (mgr.24401) 711 : cluster [DBG] pgmap v1200: 172 pgs: 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:53.908973+0000 mgr.y (mgr.24401) 711 : cluster [DBG] pgmap v1200: 172 pgs: 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:54.375229+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:54.375229+0000 mon.a (mon.0) 3649 : cluster [DBG] osdmap e774: 8 total, 8 up, 8 in 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: audit 2024-04-26T16:35:54.384029+0000 mon.c (mon.2) 804 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: audit 2024-04-26T16:35:54.384029+0000 mon.c (mon.2) 804 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: audit 2024-04-26T16:35:54.385443+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: audit 2024-04-26T16:35:54.385443+0000 mon.a (mon.0) 3650 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:54.441560+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:54.441560+0000 osd.3 (osd.3) 431 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:54.441848+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:55.710 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:55 smithi190 bash[23868]: cluster 2024-04-26T16:35:54.441848+0000 osd.3 (osd.3) 432 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:35:55.725 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:55.828 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:55.856 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:55.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:55.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:55.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:55.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:56.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:56.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:56.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:56.253 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:56.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:56.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:56.494 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:56.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: audit 2024-04-26T16:35:55.371966+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: audit 2024-04-26T16:35:55.371966+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: cluster 2024-04-26T16:35:55.376806+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: cluster 2024-04-26T16:35:55.376806+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: audit 2024-04-26T16:35:55.390001+0000 mon.c (mon.2) 805 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: audit 2024-04-26T16:35:55.390001+0000 mon.c (mon.2) 805 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: audit 2024-04-26T16:35:55.390855+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:56 smithi190 bash[23868]: audit 2024-04-26T16:35:55.390855+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: audit 2024-04-26T16:35:55.371966+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: audit 2024-04-26T16:35:55.371966+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: cluster 2024-04-26T16:35:55.376806+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: cluster 2024-04-26T16:35:55.376806+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: audit 2024-04-26T16:35:55.390001+0000 mon.c (mon.2) 805 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: audit 2024-04-26T16:35:55.390001+0000 mon.c (mon.2) 805 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: audit 2024-04-26T16:35:55.390855+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[29234]: audit 2024-04-26T16:35:55.390855+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: audit 2024-04-26T16:35:55.371966+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: audit 2024-04-26T16:35:55.371966+0000 mon.a (mon.0) 3651 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: cluster 2024-04-26T16:35:55.376806+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: cluster 2024-04-26T16:35:55.376806+0000 mon.a (mon.0) 3652 : cluster [DBG] osdmap e775: 8 total, 8 up, 8 in 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: audit 2024-04-26T16:35:55.390001+0000 mon.c (mon.2) 805 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: audit 2024-04-26T16:35:55.390001+0000 mon.c (mon.2) 805 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: audit 2024-04-26T16:35:55.390855+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:56 smithi050 bash[21463]: audit 2024-04-26T16:35:55.390855+0000 mon.a (mon.0) 3653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:35:56.649 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:56.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:56.778 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:56.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:56.932 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:35:56.945 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:56.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:56.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:57.016 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:35:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "size": 20, 2024-04-26T16:35:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 600, 2024-04-26T16:35:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [ 2024-04-26T16:35:57.016 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41021.0:1 269.0s0 269:1e0071ee:::oid_RoundTripSparseReadPP_:head [write 0~128] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:26:36.119656+0000", 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 560.899562221, 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0202599029999999, 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41021", 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3935345188", 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.017 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119656+0000", 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119656+0000", 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119658+0000", 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1059999999999998e-06 2024-04-26T16:35:57.018 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119667+0000", 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.7339999999999997e-06 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119678+0000", 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1446e-05 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.019 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119689+0000", 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1219e-05 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119752+0000", 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2469000000000001e-05 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:36.119757+0000", 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8029999999999996e-06 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.020 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:37.131903+0000", 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0121465380000001 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:37.132238+0000", 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00033472299999999999 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_started", 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:37.133150+0000", 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00091184800000000004 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.021 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_committed", 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:37.134482+0000", 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001332361 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:37.139846+0000", 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005364265 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:26:37.139916+0000", 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9391e-05 2024-04-26T16:35:57.022 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:3835 277.16 277:6cac518f:::0:head [write 0~1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:27:12.183118+0000", 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 524.83610047299999, 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.032878852, 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:35:57.023 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3835 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183118+0000", 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183118+0000", 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.024 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183120+0000", 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.4150000000000002e-06 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183133+0000", 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3108999999999999e-05 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183165+0000", 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.1695000000000001e-05 2024-04-26T16:35:57.025 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183182+0000", 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7674000000000001e-05 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183421+0000", 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000238736 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.183438+0000", 2024-04-26T16:35:57.026 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7082999999999999e-05 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.213890+0000", 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.030451886000000001 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.214055+0000", 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016494400000000001 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 0,7", 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.214239+0000", 2024-04-26T16:35:57.027 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000184441 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.215049+0000", 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000809975 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.215272+0000", 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022252200000000001 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.215743+0000", 2024-04-26T16:35:57.028 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00047073900000000002 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.215883+0000", 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014019600000000001 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:27:12.215996+0000", 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011343700000000001 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.029 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGPushReply(109.1b 585/584 [PushReplyOp(109:d97890cd:test-rados-api-smithi050-73372-48::foo4:head)])", 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:32:09.582253+0000", 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 227.43696473, 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19160011199999999, 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582253+0000", 2024-04-26T16:35:57.030 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582253+0000", 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582254+0000", 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8500000000000001e-07 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582260+0000", 2024-04-26T16:35:57.031 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4060000000000002e-06 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582270+0000", 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4679999999999998e-06 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582276+0000", 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0920000000000003e-06 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.032 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582333+0000", 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.7070000000000001e-05 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.773853+0000", 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.19152039100000001 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDPGRecoveryDeleteReply(109.8 e585,584 109:141d9b39:test-rados-api-smithi050-73372-14::foo79:3,231'35)", 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:32:09.582535+0000", 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 227.436682596, 2024-04-26T16:35:57.033 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.191253496, 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "reached pg", 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582535+0000", 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582535+0000", 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.034 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582536+0000", 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5199999999999996e-07 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.582541+0000", 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5650000000000003e-06 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.772568+0000", 2024-04-26T16:35:57.035 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.190027738 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.772607+0000", 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.824e-05 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.772960+0000", 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00035327600000000002 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.036 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:09.773789+0000", 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00082902500000000005 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.25249.0:8373 109.1a 109:5cd2c9bc:test-rados-api-smithi050-73372-99::propagate-base-tier-error:head [omap-set-vals] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:32:30.656584+0000", 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 206.362633625, 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.047821355000000003, 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:57.037 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.25249", 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/1381877852", 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 8373 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.656584+0000", 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.656584+0000", 2024-04-26T16:35:57.038 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.656586+0000", 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.8679999999999999e-06 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.656598+0000", 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1572e-05 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.656618+0000", 2024-04-26T16:35:57.039 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.0616e-05 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.656635+0000", 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6750000000000001e-05 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.659319+0000", 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0026834509999999999 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.040 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.659483+0000", 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00016396999999999999 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 5,7", 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.665263+0000", 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.005780543 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.677462+0000", 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.012198973 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:35:57.041 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.678063+0000", 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000600513 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.704322+0000", 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.026259161 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.704368+0000", 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6198000000000001e-05 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.042 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:32:30.704406+0000", 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7740000000000001e-05 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41465.0:1 318.7s0 318:e81ddc3a:test-rados-api-smithi050-73372-105::foo:head [undirty] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e630)", 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:33:02.307461+0000", 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 174.71175735700001, 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.012863114, 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.043 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41465", 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/3028810125", 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307461+0000", 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307461+0000", 2024-04-26T16:35:57.044 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307462+0000", 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.792e-06 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307469+0000", 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.9349999999999997e-06 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307479+0000", 2024-04-26T16:35:57.045 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7340000000000003e-06 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307490+0000", 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1255e-05 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307563+0000", 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.3122999999999998e-05 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for peered", 2024-04-26T16:35:57.046 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:02.307573+0000", 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.7790000000000007e-06 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:03.318004+0000", 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0104306059999999 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:03.318411+0000", 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00040726299999999999 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.047 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:33:03.320324+0000", 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0019126270000000001 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(2.3,RELEASE,e:728,from:5,reservation_nonce:0)", 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:06.626829+0000", 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 50.392389186999999, 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0032272529999999998, 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.048 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626829+0000", 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626829+0000", 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.049 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626829+0000", 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5600000000000005e-07 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626834+0000", 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.5600000000000004e-06 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626845+0000", 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0977e-05 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.050 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626856+0000", 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0759e-05 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626988+0000", 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013195899999999999 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.626998+0000", 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0407e-05 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.051 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.629964+0000", 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029655979999999998 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.630008+0000", 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.4376000000000003e-05 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:06.630056+0000", 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.7960999999999997e-05 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.052 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(6.17,QREQUEST,e:729,from:4,reservation_nonce:6)", 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:07.526481+0000", 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 49.492737382999998, 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0063436179999999997, 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.053 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526481+0000", 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526481+0000", 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526483+0000", 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.057e-06 2024-04-26T16:35:57.054 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526494+0000", 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1440000000000001e-05 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526523+0000", 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.8965000000000001e-05 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526540+0000", 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.7206000000000001e-05 2024-04-26T16:35:57.055 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526642+0000", 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102174 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.526649+0000", 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.8249999999999999e-06 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.532773+0000", 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0061240840000000001 2024-04-26T16:35:57.056 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.532788+0000", 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4267000000000001e-05 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:07.532824+0000", 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6600000000000002e-05 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.057 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:1397 330.5 330:a548bc64:test-rados-api-smithi050-73372-125::foo:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e730)", 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:08.377600+0000", 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.641617973999999, 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.014672519, 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1397 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.058 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377600+0000", 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377600+0000", 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377602+0000", 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.9920000000000002e-06 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.059 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377614+0000", 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1826e-05 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377630+0000", 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5718000000000001e-05 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377640+0000", 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.0570999999999999e-05 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.060 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377722+0000", 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2199000000000004e-05 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.377728+0000", 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6439999999999997e-06 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.380902+0000", 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0031737390000000001 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.061 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for blocked object", 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.381241+0000", 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000339427 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.389604+0000", 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0083630110000000001 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.389727+0000", 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012300000000000001 2024-04-26T16:35:57.062 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.392078+0000", 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0023511140000000001 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.392143+0000", 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4393999999999994e-05 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.392273+0000", 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00012988400000000001 2024-04-26T16:35:57.063 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:1401 330.5 330:a548bc64:test-rados-api-smithi050-73372-125::foo:4 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e730)", 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:08.409860+0000", 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.609358454000002, 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0045612430000000004, 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:57.064 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1401 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409860+0000", 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409860+0000", 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409862+0000", 2024-04-26T16:35:57.065 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2180000000000002e-06 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409870+0000", 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.6840000000000002e-06 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409886+0000", 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5787999999999999e-05 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.066 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409896+0000", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.6169999999999996e-06 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.409952+0000", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.6453999999999999e-05 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for blocked object", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.410102+0000", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00014935099999999999 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.413066+0000", 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0029647739999999999 2024-04-26T16:35:57.067 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for rw locks", 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.413131+0000", 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.4938999999999996e-05 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414241+0000", 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0011098519999999999 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414306+0000", 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.5189000000000002e-05 2024-04-26T16:35:57.068 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414421+0000", 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00011437700000000001 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:1402 330.5 330:a548bc64:test-rados-api-smithi050-73372-125::foo:4 [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e730)", 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:08.414635+0000", 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 48.604582655000002, 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001759926, 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.069 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1402 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414635+0000", 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.070 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414635+0000", 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414636+0000", 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.1030000000000001e-06 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414643+0000", 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2979999999999997e-06 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.071 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414657+0000", 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3885e-05 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414666+0000", 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0340000000000002e-06 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414714+0000", 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8594000000000003e-05 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.072 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414786+0000", 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.1307e-05 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "waiting for subops from 1,5", 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.414844+0000", 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.8368000000000001e-05 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op_commit", 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.415236+0000", 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00039242900000000002 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.073 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.415560+0000", 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00032330400000000001 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.416252+0000", 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00069232899999999999 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.416332+0000", 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.0162999999999998e-05 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.074 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:08.416395+0000", 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.3112000000000002e-05 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14391.0:3554 5.d 5:b4812045:::notify.4:head [watch ping cookie 94191258310144 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e726)", 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:09.903442+0000", 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 47.115776036, 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0013372220000000001, 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.075 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14391", 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3554 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.903442+0000", 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.076 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.903442+0000", 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.903446+0000", 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.7400000000000002e-06 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.903478+0000", 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2778999999999999e-05 2024-04-26T16:35:57.077 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.903534+0000", 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.5241999999999997e-05 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.903559+0000", 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5559e-05 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.904542+0000", 2024-04-26T16:35:57.078 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00098226900000000002 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.904657+0000", 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000115779 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:09.904779+0000", 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000121854 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.079 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:1446 330.0 330:00000000::::head [pgnls start_epoch 733] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e733)", 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:11.358498+0000", 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 45.660720308999998, 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0039470759999999999, 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1446 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.080 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358498+0000", 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358498+0000", 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358500+0000", 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5339999999999998e-06 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.081 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358515+0000", 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4691e-05 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358541+0000", 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.6412000000000001e-05 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358555+0000", 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.329e-05 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.082 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358649+0000", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.4622000000000003e-05 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.358657+0000", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.2060000000000007e-06 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.362114+0000", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003456705 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.083 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.362174+0000", 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 5.9806999999999998e-05 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:11.362445+0000", 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000270809 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14391.0:3564 5.d 5:b4812045:::notify.4:head [watch ping cookie 94191258310144 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:14.907418+0000", 2024-04-26T16:35:57.084 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 42.111800293999998, 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001146043, 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14391", 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3564 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.907418+0000", 2024-04-26T16:35:57.085 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.907418+0000", 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.907421+0000", 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.354e-06 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.086 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.907434+0000", 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3225e-05 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.907473+0000", 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8859000000000002e-05 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.907496+0000", 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2813999999999999e-05 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.087 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.908248+0000", 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00075161100000000001 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.908423+0000", 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000175697 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:14.908564+0000", 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000140483 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.088 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_repop(client.41513.0:1512 331.17 e739/735 331:ee25a38f:test-rados-api-smithi050-73372-127::foo:head v 739'2, mlcod=0'0)", 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:17.383796+0000", 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 39.635422306999999, 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.003675106, 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383796+0000", 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.089 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383796+0000", 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383796+0000", 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.7700000000000004e-07 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383805+0000", 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 8.1240000000000005e-06 2024-04-26T16:35:57.090 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383821+0000", 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6413999999999999e-05 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383830+0000", 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 9.0510000000000002e-06 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383932+0000", 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000101512 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.091 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.383957+0000", 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.5408000000000001e-05 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "commit_sent", 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.387432+0000", 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0034750890000000002 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:17.387471+0000", 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8730999999999999e-05 2024-04-26T16:35:57.092 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "MOSDScrubReserve(323.2s2,QREQUEST,e:740,from:1(0),reservation_nonce:2)", 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:18.811475+0000", 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 38.207743331000003, 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.006303749, 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.093 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811475+0000", 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811475+0000", 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811477+0000", 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.39e-06 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.094 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811490+0000", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2933e-05 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811526+0000", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.6136000000000001e-05 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811555+0000", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9198999999999999e-05 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.095 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811694+0000", 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013869300000000001 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.811708+0000", 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.3912e-05 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.817602+0000", 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0058941710000000001 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.096 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.817665+0000", 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.2640999999999999e-05 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:18.817778+0000", 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000113674 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:1634 331.8 331:10000000::::head [pgnls start_epoch 741] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e741)", 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:19.372135+0000", 2024-04-26T16:35:57.097 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 37.647082761999997, 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0025778210000000001, 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 1634 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372135+0000", 2024-04-26T16:35:57.098 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372135+0000", 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372138+0000", 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.9569999999999998e-06 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372151+0000", 2024-04-26T16:35:57.099 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.2679e-05 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372184+0000", 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.3222000000000001e-05 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372199+0000", 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4674999999999999e-05 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372301+0000", 2024-04-26T16:35:57.100 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.000102679 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.372309+0000", 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.6839999999999996e-06 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.374512+0000", 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0022025059999999999 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.374582+0000", 2024-04-26T16:35:57.101 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 7.0829999999999998e-05 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:19.374713+0000", 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00013058899999999999 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.14391.0:3585 5.d 5:b4812045:::notify.4:head [watch ping cookie 94191258310144 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:24.915504+0000", 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 32.103713839999998, 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.001356639, 2024-04-26T16:35:57.102 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.14391", 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 3585 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.915504+0000", 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.103 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.915504+0000", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.915508+0000", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.8569999999999997e-06 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.915524+0000", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.5956000000000001e-05 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.104 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.915585+0000", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.0631999999999998e-05 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.915617+0000", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 3.2098999999999999e-05 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.916054+0000", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00043747300000000002 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.916398+0000", 2024-04-26T16:35:57.105 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00034351999999999999 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:24.916861+0000", 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00046310200000000001 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "description": "osd_op(client.41513.0:2212 334.0 334:00000000::::head [pgnls start_epoch 772] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e772)", 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "initiated_at": "2024-04-26T16:35:52.371733+0000", 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "age": 4.6474847520000004, 2024-04-26T16:35:57.106 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0042792999999999998, 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "continuous": false, 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "type_data": { 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "flag_point": "started", 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "client_info": { 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "client": "client.41513", 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "tid": 2212 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "events": [ 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "initiated", 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371733+0000", 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.107 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "header_read", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371733+0000", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "throttled", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371735+0000", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.2120000000000002e-06 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "all_read", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371752+0000", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.6388999999999998e-05 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "dispatched", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371773+0000", 2024-04-26T16:35:57.108 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 2.1452999999999999e-05 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "queued_for_pg", 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371788+0000", 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 1.4752e-05 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371855+0000", 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 6.7321000000000001e-05 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "op must wait for map", 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.371860+0000", 2024-04-26T16:35:57.109 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.8289999999999998e-06 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "reached_pg", 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.375738+0000", 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.0038779940000000001 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "started", 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.375784+0000", 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 4.6140000000000002e-05 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: }, 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: { 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "event": "done", 2024-04-26T16:35:57.110 INFO:teuthology.orchestra.run.smithi050.stdout: "time": "2024-04-26T16:35:52.376013+0000", 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout: "duration": 0.00022821000000000001 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout: } 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout: ] 2024-04-26T16:35:57.111 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:35:57.111 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:57.135 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:57.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:57.265 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:35:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:35:57.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:57.402 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:57.505 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:57.532 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:57.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: cluster 2024-04-26T16:35:55.909663+0000 mgr.y (mgr.24401) 712 : cluster [DBG] pgmap v1203: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: cluster 2024-04-26T16:35:55.909663+0000 mgr.y (mgr.24401) 712 : cluster [DBG] pgmap v1203: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: audit 2024-04-26T16:35:56.390738+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: audit 2024-04-26T16:35:56.390738+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: audit 2024-04-26T16:35:56.397492+0000 mon.c (mon.2) 806 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: audit 2024-04-26T16:35:56.397492+0000 mon.c (mon.2) 806 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: cluster 2024-04-26T16:35:56.398103+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: cluster 2024-04-26T16:35:56.398103+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: audit 2024-04-26T16:35:56.398814+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:57 smithi190 bash[23868]: audit 2024-04-26T16:35:56.398814+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:57.663 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:57.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: cluster 2024-04-26T16:35:55.909663+0000 mgr.y (mgr.24401) 712 : cluster [DBG] pgmap v1203: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: cluster 2024-04-26T16:35:55.909663+0000 mgr.y (mgr.24401) 712 : cluster [DBG] pgmap v1203: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: audit 2024-04-26T16:35:56.390738+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: audit 2024-04-26T16:35:56.390738+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: audit 2024-04-26T16:35:56.397492+0000 mon.c (mon.2) 806 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: audit 2024-04-26T16:35:56.397492+0000 mon.c (mon.2) 806 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: cluster 2024-04-26T16:35:56.398103+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: cluster 2024-04-26T16:35:56.398103+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-04-26T16:35:57.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: audit 2024-04-26T16:35:56.398814+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[29234]: audit 2024-04-26T16:35:56.398814+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: cluster 2024-04-26T16:35:55.909663+0000 mgr.y (mgr.24401) 712 : cluster [DBG] pgmap v1203: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: cluster 2024-04-26T16:35:55.909663+0000 mgr.y (mgr.24401) 712 : cluster [DBG] pgmap v1203: 204 pgs: 32 unknown, 1 active+clean+snaptrim, 171 active+clean; 583 KiB data, 745 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: audit 2024-04-26T16:35:56.390738+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: audit 2024-04-26T16:35:56.390738+0000 mon.a (mon.0) 3654 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: audit 2024-04-26T16:35:56.397492+0000 mon.c (mon.2) 806 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: audit 2024-04-26T16:35:56.397492+0000 mon.c (mon.2) 806 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: cluster 2024-04-26T16:35:56.398103+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-04-26T16:35:57.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: cluster 2024-04-26T16:35:56.398103+0000 mon.a (mon.0) 3655 : cluster [DBG] osdmap e776: 8 total, 8 up, 8 in 2024-04-26T16:35:57.782 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: audit 2024-04-26T16:35:56.398814+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.782 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:57 smithi050 bash[21463]: audit 2024-04-26T16:35:56.398814+0000 mon.a (mon.0) 3656 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:35:57.795 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:57.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:57.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:57.922 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:57.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:58.025 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:58.053 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:58.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:58.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:58.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:58.319 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:58.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:58.423 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:58.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:58.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.248531+0000 mgr.y (mgr.24401) 713 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.248531+0000 mgr.y (mgr.24401) 713 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.392831+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.392831+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: cluster 2024-04-26T16:35:57.397422+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: cluster 2024-04-26T16:35:57.397422+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-04-26T16:35:58.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.400413+0000 mon.c (mon.2) 807 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.400413+0000 mon.c (mon.2) 807 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.407111+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.407111+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.745190+0000 mon.c (mon.2) 808 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:58.566 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:58 smithi190 bash[23868]: audit 2024-04-26T16:35:57.745190+0000 mon.c (mon.2) 808 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:58.582 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.248531+0000 mgr.y (mgr.24401) 713 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.248531+0000 mgr.y (mgr.24401) 713 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.392831+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:35:58.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.392831+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: cluster 2024-04-26T16:35:57.397422+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: cluster 2024-04-26T16:35:57.397422+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.400413+0000 mon.c (mon.2) 807 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.400413+0000 mon.c (mon.2) 807 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.407111+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.407111+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.745190+0000 mon.c (mon.2) 808 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[21463]: audit 2024-04-26T16:35:57.745190+0000 mon.c (mon.2) 808 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.248531+0000 mgr.y (mgr.24401) 713 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.248531+0000 mgr.y (mgr.24401) 713 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:35:58.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.392831+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.392831+0000 mon.a (mon.0) 3657 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: cluster 2024-04-26T16:35:57.397422+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: cluster 2024-04-26T16:35:57.397422+0000 mon.a (mon.0) 3658 : cluster [DBG] osdmap e777: 8 total, 8 up, 8 in 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.400413+0000 mon.c (mon.2) 807 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.400413+0000 mon.c (mon.2) 807 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.407111+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.407111+0000 mon.a (mon.0) 3659 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]: dispatch 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.745190+0000 mon.c (mon.2) 808 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:58.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:58 smithi050 bash[29234]: audit 2024-04-26T16:35:57.745190+0000 mon.c (mon.2) 808 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:35:58.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:58.715 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:58.818 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:58.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:58.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:58.950 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:58.979 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:58.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:59.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:59.112 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:35:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:35:59.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:59.242 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:35:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:35:59.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:59.373 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:35:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:35:59.476 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:59.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:35:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:35:59.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: cluster 2024-04-26T16:35:57.910427+0000 mgr.y (mgr.24401) 714 : cluster [DBG] pgmap v1206: 204 pgs: 6 unknown, 198 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:59.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: cluster 2024-04-26T16:35:57.910427+0000 mgr.y (mgr.24401) 714 : cluster [DBG] pgmap v1206: 204 pgs: 6 unknown, 198 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:59.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: cluster 2024-04-26T16:35:58.402536+0000 mon.a (mon.0) 3660 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:59.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: cluster 2024-04-26T16:35:58.402536+0000 mon.a (mon.0) 3660 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: audit 2024-04-26T16:35:58.409932+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]': finished 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: audit 2024-04-26T16:35:58.409932+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]': finished 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: cluster 2024-04-26T16:35:58.415733+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: cluster 2024-04-26T16:35:58.415733+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: audit 2024-04-26T16:35:58.499786+0000 mon.c (mon.2) 809 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: audit 2024-04-26T16:35:58.499786+0000 mon.c (mon.2) 809 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: audit 2024-04-26T16:35:58.500311+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:35:59 smithi190 bash[23868]: audit 2024-04-26T16:35:58.500311+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:59.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: cluster 2024-04-26T16:35:57.910427+0000 mgr.y (mgr.24401) 714 : cluster [DBG] pgmap v1206: 204 pgs: 6 unknown, 198 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:59.617 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: cluster 2024-04-26T16:35:57.910427+0000 mgr.y (mgr.24401) 714 : cluster [DBG] pgmap v1206: 204 pgs: 6 unknown, 198 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: cluster 2024-04-26T16:35:58.402536+0000 mon.a (mon.0) 3660 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: cluster 2024-04-26T16:35:58.402536+0000 mon.a (mon.0) 3660 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: audit 2024-04-26T16:35:58.409932+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]': finished 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: audit 2024-04-26T16:35:58.409932+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]': finished 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: cluster 2024-04-26T16:35:58.415733+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: cluster 2024-04-26T16:35:58.415733+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: audit 2024-04-26T16:35:58.499786+0000 mon.c (mon.2) 809 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: audit 2024-04-26T16:35:58.499786+0000 mon.c (mon.2) 809 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: audit 2024-04-26T16:35:58.500311+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.618 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[29234]: audit 2024-04-26T16:35:58.500311+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: cluster 2024-04-26T16:35:57.910427+0000 mgr.y (mgr.24401) 714 : cluster [DBG] pgmap v1206: 204 pgs: 6 unknown, 198 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: cluster 2024-04-26T16:35:57.910427+0000 mgr.y (mgr.24401) 714 : cluster [DBG] pgmap v1206: 204 pgs: 6 unknown, 198 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: cluster 2024-04-26T16:35:58.402536+0000 mon.a (mon.0) 3660 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: cluster 2024-04-26T16:35:58.402536+0000 mon.a (mon.0) 3660 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: audit 2024-04-26T16:35:58.409932+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]': finished 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: audit 2024-04-26T16:35:58.409932+0000 mon.a (mon.0) 3661 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-134", "mode": "writeback"}]': finished 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: cluster 2024-04-26T16:35:58.415733+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: cluster 2024-04-26T16:35:58.415733+0000 mon.a (mon.0) 3662 : cluster [DBG] osdmap e778: 8 total, 8 up, 8 in 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: audit 2024-04-26T16:35:58.499786+0000 mon.c (mon.2) 809 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: audit 2024-04-26T16:35:58.499786+0000 mon.c (mon.2) 809 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: audit 2024-04-26T16:35:58.500311+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.619 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:35:59 smithi050 bash[21463]: audit 2024-04-26T16:35:58.500311+0000 mon.a (mon.0) 3663 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:35:59.633 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:35:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:35:59.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:59.762 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:35:59.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:35:59.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:59.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:35:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:35:59.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:35:59.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:00.027 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:00.130 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:00.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:00.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:00.289 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:00.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:00.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:00.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:00.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:00.542 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:35:59.413003+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:35:59.413003+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: cluster 2024-04-26T16:35:59.420640+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: cluster 2024-04-26T16:35:59.420640+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:35:59.423277+0000 mon.c (mon.2) 810 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:35:59.423277+0000 mon.c (mon.2) 810 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:35:59.425595+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:35:59.425595+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:00.413458+0000 mon.a (mon.0) 3667 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:00.413458+0000 mon.a (mon.0) 3667 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:36:00.415989+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: audit 2024-04-26T16:36:00.415989+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:00.421349+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-04-26T16:36:00.543 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:00.421349+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-04-26T16:36:00.555 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:00.633 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-04-26T16:36:00.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:00.686 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:00.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:00.815 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:35:59.413003+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:35:59.413003+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: cluster 2024-04-26T16:35:59.420640+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: cluster 2024-04-26T16:35:59.420640+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:35:59.423277+0000 mon.c (mon.2) 810 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:35:59.423277+0000 mon.c (mon.2) 810 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:35:59.425595+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:35:59.425595+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:00.413458+0000 mon.a (mon.0) 3667 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:00.413458+0000 mon.a (mon.0) 3667 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:36:00.415989+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: audit 2024-04-26T16:36:00.415989+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:00.421349+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:00.421349+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:35:59.413003+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:35:59.413003+0000 mon.a (mon.0) 3664 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: cluster 2024-04-26T16:35:59.420640+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: cluster 2024-04-26T16:35:59.420640+0000 mon.a (mon.0) 3665 : cluster [DBG] osdmap e779: 8 total, 8 up, 8 in 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:35:59.423277+0000 mon.c (mon.2) 810 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:35:59.423277+0000 mon.c (mon.2) 810 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:35:59.425595+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:35:59.425595+0000 mon.a (mon.0) 3666 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]: dispatch 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:00.413458+0000 mon.a (mon.0) 3667 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:00.413458+0000 mon.a (mon.0) 3667 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:36:00.415989+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: audit 2024-04-26T16:36:00.415989+0000 mon.a (mon.0) 3668 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-134"}]': finished 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:00.421349+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-04-26T16:36:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:00.421349+0000 mon.a (mon.0) 3669 : cluster [DBG] osdmap e780: 8 total, 8 up, 8 in 2024-04-26T16:36:00.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:00.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:00.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:01.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:01.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:01.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:01.174 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:01.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:01.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:01.333 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:01.435 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:59.911135+0000 mgr.y (mgr.24401) 715 : cluster [DBG] pgmap v1209: 204 pgs: 204 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[21463]: cluster 2024-04-26T16:35:59.911135+0000 mgr.y (mgr.24401) 715 : cluster [DBG] pgmap v1209: 204 pgs: 204 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.373557+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.373557+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:59.911135+0000 mgr.y (mgr.24401) 715 : cluster [DBG] pgmap v1209: 204 pgs: 204 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[29234]: cluster 2024-04-26T16:35:59.911135+0000 mgr.y (mgr.24401) 715 : cluster [DBG] pgmap v1209: 204 pgs: 204 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.373557+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-04-26T16:36:01.446 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:01 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.373557+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-04-26T16:36:01.458 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:01.561 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:01.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:59.911135+0000 mgr.y (mgr.24401) 715 : cluster [DBG] pgmap v1209: 204 pgs: 204 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:01 smithi190 bash[23868]: cluster 2024-04-26T16:35:59.911135+0000 mgr.y (mgr.24401) 715 : cluster [DBG] pgmap v1209: 204 pgs: 204 active+clean; 583 KiB data, 746 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:01 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.373557+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-04-26T16:36:01.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:01 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.373557+0000 mon.a (mon.0) 3670 : cluster [DBG] osdmap e781: 8 total, 8 up, 8 in 2024-04-26T16:36:01.586 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:01.689 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:01.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:01.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:01.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:01.845 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:01.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:01.976 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:01.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:02.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:02.108 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:02.211 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:02.240 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:02.343 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.379 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:02.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:02.511 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:02.615 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.641 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:02.745 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.769 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:02.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:02.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:02.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:03.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:03.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:03.147 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:03.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:03.278 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:03.367 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.791901+0000 osd.1 (osd.1) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.791901+0000 osd.1 (osd.1) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.792340+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.792340+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.912973+0000 mgr.y (mgr.24401) 716 : cluster [DBG] pgmap v1212: 172 pgs: 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:01.912973+0000 mgr.y (mgr.24401) 716 : cluster [DBG] pgmap v1212: 172 pgs: 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:02.379445+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: cluster 2024-04-26T16:36:02.379445+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: audit 2024-04-26T16:36:02.389640+0000 mon.c (mon.2) 811 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: audit 2024-04-26T16:36:02.389640+0000 mon.c (mon.2) 811 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: audit 2024-04-26T16:36:02.390082+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.368 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[21463]: audit 2024-04-26T16:36:02.390082+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:03.422 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:03.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.791901+0000 osd.1 (osd.1) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.791901+0000 osd.1 (osd.1) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.792340+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.792340+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.912973+0000 mgr.y (mgr.24401) 716 : cluster [DBG] pgmap v1212: 172 pgs: 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:01.912973+0000 mgr.y (mgr.24401) 716 : cluster [DBG] pgmap v1212: 172 pgs: 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:02.379445+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: cluster 2024-04-26T16:36:02.379445+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-04-26T16:36:03.536 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: audit 2024-04-26T16:36:02.389640+0000 mon.c (mon.2) 811 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: audit 2024-04-26T16:36:02.389640+0000 mon.c (mon.2) 811 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: audit 2024-04-26T16:36:02.390082+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.537 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:03 smithi190 bash[23868]: audit 2024-04-26T16:36:02.390082+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.553 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:03.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.791901+0000 osd.1 (osd.1) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:03.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.791901+0000 osd.1 (osd.1) 397 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:03.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.792340+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:03.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.792340+0000 osd.1 (osd.1) 398 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:03.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.912973+0000 mgr.y (mgr.24401) 716 : cluster [DBG] pgmap v1212: 172 pgs: 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:03.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:01.912973+0000 mgr.y (mgr.24401) 716 : cluster [DBG] pgmap v1212: 172 pgs: 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:03.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:02.379445+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-04-26T16:36:03.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: cluster 2024-04-26T16:36:02.379445+0000 mon.a (mon.0) 3671 : cluster [DBG] osdmap e782: 8 total, 8 up, 8 in 2024-04-26T16:36:03.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: audit 2024-04-26T16:36:02.389640+0000 mon.c (mon.2) 811 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: audit 2024-04-26T16:36:02.389640+0000 mon.c (mon.2) 811 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: audit 2024-04-26T16:36:02.390082+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:03 smithi050 bash[29234]: audit 2024-04-26T16:36:02.390082+0000 mon.a (mon.0) 3672 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:03.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:03.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:03.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:03.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:03.814 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:03.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/59488967 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d68105640 msgr2=0x7f6d68105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/59488967 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d68105640 0x7f6d68105ac0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009990 tx=0x7f6d58009ca0 comp rx=0 tx=0).stop 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/59488967 shutdown_connections 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/59488967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d6810b640 0x7f6d6810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/59488967 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68106000 0x7f6d68112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/59488967 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d68105640 0x7f6d68105ac0 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/59488967 >> 172.21.15.50:0/59488967 conn(0x7f6d68101130 msgr2=0x7f6d68103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/59488967 shutdown_connections 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/59488967 wait complete. 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 librados: starting msgr at 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 librados: starting objecter 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 Processor -- start 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- start start 2024-04-26T16:36:03.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 librados: setting wanted keys 2024-04-26T16:36:03.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 librados: calling monclient init 2024-04-26T16:36:03.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 0x7f6d6819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:03.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d68106000 0x7f6d6819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:03.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 0x7f6d6819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:03.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6d68114df0 con 0x7f6d68105640 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6d68114c70 con 0x7f6d68106000 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6d68114f70 con 0x7f6d6810b640 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 0x7f6d6819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6d6d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 0x7f6d6819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6d6d6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 0x7f6d6819b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46898/0 (socket says 172.21.15.50:46898) 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6d6d6700 1 -- 172.21.15.50:0/3218240884 learned_addr learned my addr 172.21.15.50:0/3218240884 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 0x7f6d6819fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50968/0 (socket says 172.21.15.50:50968) 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ced5700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d68106000 0x7f6d6819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 -- 172.21.15.50:0/3218240884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d68106000 msgr2=0x7f6d6819b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d68106000 0x7f6d6819b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 -- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 msgr2=0x7f6d6819b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:03.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 0x7f6d6819b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 -- 172.21.15.50:0/3218240884 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d681a03b0 con 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6d6d6700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 0x7f6d6819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6ded7700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 0x7f6d6819fc70 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000efb0 tx=0x7f6d6000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d5e7fc700 10 client.?.objecter ms_handle_connect 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d5e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6000ca50 con 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6d6000cbf0 con 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d681a06a0 con 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 10 client.41624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d681a7f20 con 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d6f93a700 1 librados: init done 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.846+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d600046b0 con 0x7f6d6810b640 2024-04-26T16:36:03.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d4c005120 con 0x7f6d6810b640 2024-04-26T16:36:03.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6d60012050 con 0x7f6d6810b640 2024-04-26T16:36:03.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d54071f80 0x7f6d54074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:03.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(783..783 src has 577..783) ==== 9952+0+0 (secure 0 0 0) 0x7f6d60091500 con 0x7f6d6810b640 2024-04-26T16:36:03.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 10 client.41624.objecter ms_dispatch 0x7f6d68001550 osd_map(783..783 src has 577..783) 2024-04-26T16:36:03.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 3 client.41624.objecter handle_osd_map got epochs [783,783] > 0 2024-04-26T16:36:03.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 3 client.41624.objecter handle_osd_map decoding full epoch 783 2024-04-26T16:36:03.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d6d6d6700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d54071f80 0x7f6d54074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:03.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 20 client.41624.objecter dump_active .. 0 homeless 2024-04-26T16:36:03.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d5e7fc700 10 client.41624.objecter ms_handle_connect 0x7f6d54071f80 2024-04-26T16:36:03.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.850+0000 7f6d6d6d6700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d54071f80 0x7f6d54074440 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 crypto rx=0x7f6d64005950 tx=0x7f6d6400a3b0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:03.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:03.858+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6d60022060 con 0x7f6d6810b640 2024-04-26T16:36:03.917 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:03.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:03.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:04.042 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:04.065 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:04.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.094+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7f6d4c005410 con 0x7f6d6810b640 2024-04-26T16:36:04.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:04.197 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:04.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:04.326 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:04.393 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.394+0000 7f6d5e7fc700 1 -- 172.21.15.50:0/3218240884 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v784) ==== 113+0+0 (secure 0 0 0) 0x7f6d60010040 con 0x7f6d6810b640 2024-04-26T16:36:04.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 10 librados: watch_flush enter 2024-04-26T16:36:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 10 librados: watch_flush exit 2024-04-26T16:36:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 20 client.41624.objecter shutdown clearing up homeless session... 2024-04-26T16:36:04.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 10 client.41624.objecter successfully canceled tick 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d54071f80 msgr2=0x7f6d54074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d54071f80 0x7f6d54074440 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 crypto rx=0x7f6d64005950 tx=0x7f6d6400a3b0 comp rx=0 tx=0).stop 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 msgr2=0x7f6d6819fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 0x7f6d6819fc70 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000efb0 tx=0x7f6d6000c3a0 comp rx=0 tx=0).stop 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 shutdown_connections 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d54071f80 0x7f6d54074440 unknown :-1 s=CLOSED pgs=3829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d6810b640 0x7f6d6819fc70 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d68105640 0x7f6d6819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 --2- 172.21.15.50:0/3218240884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d68106000 0x7f6d6819b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:04.397 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 >> 172.21.15.50:0/3218240884 conn(0x7f6d68101130 msgr2=0x7f6d68101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 shutdown_connections 2024-04-26T16:36:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 -- 172.21.15.50:0/3218240884 wait complete. 2024-04-26T16:36:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:04.398+0000 7f6d6f93a700 1 librados: shutdown 2024-04-26T16:36:04.398 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T16:36:04.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:04.450 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:04.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:04.564 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:03.378798+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:03.378798+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: cluster 2024-04-26T16:36:03.383535+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: cluster 2024-04-26T16:36:03.383535+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:03.386168+0000 mon.c (mon.2) 812 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:03.386168+0000 mon.c (mon.2) 812 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:03.388683+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:03.388683+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:04.098166+0000 mon.c (mon.2) 813 : audit [INF] from='client.? 172.21.15.50:0/3218240884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:04.098166+0000 mon.c (mon.2) 813 : audit [INF] from='client.? 172.21.15.50:0/3218240884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:04.098484+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.565 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:04 smithi190 bash[23868]: audit 2024-04-26T16:36:04.098484+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.581 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:04.683 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:04.712 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:04.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:04.842 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:04.850 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:36:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:36:04.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:03.378798+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:04.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:03.378798+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:04.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: cluster 2024-04-26T16:36:03.383535+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-04-26T16:36:04.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: cluster 2024-04-26T16:36:03.383535+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:03.386168+0000 mon.c (mon.2) 812 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:03.386168+0000 mon.c (mon.2) 812 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:03.388683+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:03.388683+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:04.098166+0000 mon.c (mon.2) 813 : audit [INF] from='client.? 172.21.15.50:0/3218240884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:04.098166+0000 mon.c (mon.2) 813 : audit [INF] from='client.? 172.21.15.50:0/3218240884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:04.098484+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[29234]: audit 2024-04-26T16:36:04.098484+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:03.378798+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:03.378798+0000 mon.a (mon.0) 3673 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: cluster 2024-04-26T16:36:03.383535+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-04-26T16:36:04.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: cluster 2024-04-26T16:36:03.383535+0000 mon.a (mon.0) 3674 : cluster [DBG] osdmap e783: 8 total, 8 up, 8 in 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:03.386168+0000 mon.c (mon.2) 812 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:03.386168+0000 mon.c (mon.2) 812 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:03.388683+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:03.388683+0000 mon.a (mon.0) 3675 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:04.098166+0000 mon.c (mon.2) 813 : audit [INF] from='client.? 172.21.15.50:0/3218240884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:04.098166+0000 mon.c (mon.2) 813 : audit [INF] from='client.? 172.21.15.50:0/3218240884' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:04.098484+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:04 smithi050 bash[21463]: audit 2024-04-26T16:36:04.098484+0000 mon.a (mon.0) 3676 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:36:04.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:04.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:04.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:05.074 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:05.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:05.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:05.233 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:05.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:05.366 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:05.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: cluster 2024-04-26T16:36:03.913782+0000 mgr.y (mgr.24401) 717 : cluster [DBG] pgmap v1215: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: cluster 2024-04-26T16:36:03.913782+0000 mgr.y (mgr.24401) 717 : cluster [DBG] pgmap v1215: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.387270+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.387270+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.387384+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:36:05.481 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.387384+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:36:05.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: cluster 2024-04-26T16:36:04.392877+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-04-26T16:36:05.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: cluster 2024-04-26T16:36:04.392877+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-04-26T16:36:05.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.395339+0000 mon.c (mon.2) 814 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.395339+0000 mon.c (mon.2) 814 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.395705+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.482 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:05 smithi190 bash[23868]: audit 2024-04-26T16:36:04.395705+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.496 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:05.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:05.627 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: cluster 2024-04-26T16:36:03.913782+0000 mgr.y (mgr.24401) 717 : cluster [DBG] pgmap v1215: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: cluster 2024-04-26T16:36:03.913782+0000 mgr.y (mgr.24401) 717 : cluster [DBG] pgmap v1215: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.387270+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.387270+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.387384+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.387384+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:36:05.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: cluster 2024-04-26T16:36:04.392877+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: cluster 2024-04-26T16:36:04.392877+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.395339+0000 mon.c (mon.2) 814 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.395339+0000 mon.c (mon.2) 814 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.395705+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[21463]: audit 2024-04-26T16:36:04.395705+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: cluster 2024-04-26T16:36:03.913782+0000 mgr.y (mgr.24401) 717 : cluster [DBG] pgmap v1215: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: cluster 2024-04-26T16:36:03.913782+0000 mgr.y (mgr.24401) 717 : cluster [DBG] pgmap v1215: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.387270+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.387270+0000 mon.a (mon.0) 3677 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.387384+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.387384+0000 mon.a (mon.0) 3678 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: cluster 2024-04-26T16:36:04.392877+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: cluster 2024-04-26T16:36:04.392877+0000 mon.a (mon.0) 3679 : cluster [DBG] osdmap e784: 8 total, 8 up, 8 in 2024-04-26T16:36:05.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.395339+0000 mon.c (mon.2) 814 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.395339+0000 mon.c (mon.2) 814 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.395705+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:05 smithi050 bash[29234]: audit 2024-04-26T16:36:04.395705+0000 mon.a (mon.0) 3680 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:05.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:05.756 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:05.860 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:05.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:05.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:05.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:06.016 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:06.118 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:06.140 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:06.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:06.270 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:06.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:06.373 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:06.407 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:06.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:06.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:05.389620+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:05.389620+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: cluster 2024-04-26T16:36:05.394996+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: cluster 2024-04-26T16:36:05.394996+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:05.395136+0000 mon.c (mon.2) 815 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:05.395136+0000 mon.c (mon.2) 815 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:05.398097+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:05.398097+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.522 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: cluster 2024-04-26T16:36:06.390182+0000 mon.a (mon.0) 3684 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:06.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: cluster 2024-04-26T16:36:06.390182+0000 mon.a (mon.0) 3684 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:06.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:06.392843+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]': finished 2024-04-26T16:36:06.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: audit 2024-04-26T16:36:06.392843+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]': finished 2024-04-26T16:36:06.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: cluster 2024-04-26T16:36:06.401081+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-04-26T16:36:06.523 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:06 smithi190 bash[23868]: cluster 2024-04-26T16:36:06.401081+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-04-26T16:36:06.537 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:06.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:06.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:06.771 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:06.783 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:05.389620+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:05.389620+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: cluster 2024-04-26T16:36:05.394996+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: cluster 2024-04-26T16:36:05.394996+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:05.395136+0000 mon.c (mon.2) 815 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:05.395136+0000 mon.c (mon.2) 815 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:05.398097+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:05.398097+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: cluster 2024-04-26T16:36:06.390182+0000 mon.a (mon.0) 3684 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: cluster 2024-04-26T16:36:06.390182+0000 mon.a (mon.0) 3684 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:06.392843+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]': finished 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: audit 2024-04-26T16:36:06.392843+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]': finished 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: cluster 2024-04-26T16:36:06.401081+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-04-26T16:36:06.784 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[29234]: cluster 2024-04-26T16:36:06.401081+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:05.389620+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:05.389620+0000 mon.a (mon.0) 3681 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: cluster 2024-04-26T16:36:05.394996+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: cluster 2024-04-26T16:36:05.394996+0000 mon.a (mon.0) 3682 : cluster [DBG] osdmap e785: 8 total, 8 up, 8 in 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:05.395136+0000 mon.c (mon.2) 815 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:05.395136+0000 mon.c (mon.2) 815 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:05.398097+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:05.398097+0000 mon.a (mon.0) 3683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]: dispatch 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: cluster 2024-04-26T16:36:06.390182+0000 mon.a (mon.0) 3684 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: cluster 2024-04-26T16:36:06.390182+0000 mon.a (mon.0) 3684 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:06.392843+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]': finished 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: audit 2024-04-26T16:36:06.392843+0000 mon.a (mon.0) 3685 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-136", "mode": "writeback"}]': finished 2024-04-26T16:36:06.785 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: cluster 2024-04-26T16:36:06.401081+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-04-26T16:36:06.786 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:06 smithi050 bash[21463]: cluster 2024-04-26T16:36:06.401081+0000 mon.a (mon.0) 3686 : cluster [DBG] osdmap e786: 8 total, 8 up, 8 in 2024-04-26T16:36:06.802 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:06.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:06.934 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:06.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:07.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:07.063 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:07.166 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:07.194 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:07.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:07.325 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:07.428 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:07.454 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:07.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:07.568 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:36:07.568 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[23868]: cluster 2024-04-26T16:36:05.914352+0000 mgr.y (mgr.24401) 718 : cluster [DBG] pgmap v1218: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:07.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[23868]: cluster 2024-04-26T16:36:05.914352+0000 mgr.y (mgr.24401) 718 : cluster [DBG] pgmap v1218: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:07.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[23868]: audit 2024-04-26T16:36:06.513717+0000 mon.c (mon.2) 816 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[23868]: audit 2024-04-26T16:36:06.513717+0000 mon.c (mon.2) 816 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[23868]: audit 2024-04-26T16:36:06.514442+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.569 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:07 smithi190 bash[23868]: audit 2024-04-26T16:36:06.514442+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.585 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:07.688 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:07.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[21463]: cluster 2024-04-26T16:36:05.914352+0000 mgr.y (mgr.24401) 718 : cluster [DBG] pgmap v1218: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:07.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[21463]: cluster 2024-04-26T16:36:05.914352+0000 mgr.y (mgr.24401) 718 : cluster [DBG] pgmap v1218: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:07.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[21463]: audit 2024-04-26T16:36:06.513717+0000 mon.c (mon.2) 816 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[21463]: audit 2024-04-26T16:36:06.513717+0000 mon.c (mon.2) 816 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.700 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[21463]: audit 2024-04-26T16:36:06.514442+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[21463]: audit 2024-04-26T16:36:06.514442+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[29234]: cluster 2024-04-26T16:36:05.914352+0000 mgr.y (mgr.24401) 718 : cluster [DBG] pgmap v1218: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[29234]: cluster 2024-04-26T16:36:05.914352+0000 mgr.y (mgr.24401) 718 : cluster [DBG] pgmap v1218: 204 pgs: 3 creating+activating, 29 unknown, 172 active+clean; 583 KiB data, 750 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[29234]: audit 2024-04-26T16:36:06.513717+0000 mon.c (mon.2) 816 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[29234]: audit 2024-04-26T16:36:06.513717+0000 mon.c (mon.2) 816 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[29234]: audit 2024-04-26T16:36:06.514442+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.701 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:07 smithi050 bash[29234]: audit 2024-04-26T16:36:06.514442+0000 mon.a (mon.0) 3687 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:07.716 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:07.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:07.847 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:07.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:07.977 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:07.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:07.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:08.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:08.109 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:08.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:08.239 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:08.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:08.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:08.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:08.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-26T16:36:08.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.259320+0000 mgr.y (mgr.24401) 719 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.259320+0000 mgr.y (mgr.24401) 719 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.424301+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.424301+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: cluster 2024-04-26T16:36:07.431348+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: cluster 2024-04-26T16:36:07.431348+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.436192+0000 mon.c (mon.2) 817 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.436192+0000 mon.c (mon.2) 817 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.437070+0000 mon.a (mon.0) 3690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:08 smithi190 bash[23868]: audit 2024-04-26T16:36:07.437070+0000 mon.a (mon.0) 3690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.501 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:08.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.259320+0000 mgr.y (mgr.24401) 719 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.259320+0000 mgr.y (mgr.24401) 719 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.424301+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.424301+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: cluster 2024-04-26T16:36:07.431348+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: cluster 2024-04-26T16:36:07.431348+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.436192+0000 mon.c (mon.2) 817 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.436192+0000 mon.c (mon.2) 817 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.437070+0000 mon.a (mon.0) 3690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[29234]: audit 2024-04-26T16:36:07.437070+0000 mon.a (mon.0) 3690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.259320+0000 mgr.y (mgr.24401) 719 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.259320+0000 mgr.y (mgr.24401) 719 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.424301+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.424301+0000 mon.a (mon.0) 3688 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: cluster 2024-04-26T16:36:07.431348+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: cluster 2024-04-26T16:36:07.431348+0000 mon.a (mon.0) 3689 : cluster [DBG] osdmap e787: 8 total, 8 up, 8 in 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.436192+0000 mon.c (mon.2) 817 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.436192+0000 mon.c (mon.2) 817 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.437070+0000 mon.a (mon.0) 3690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:08 smithi050 bash[21463]: audit 2024-04-26T16:36:07.437070+0000 mon.a (mon.0) 3690 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]: dispatch 2024-04-26T16:36:08.632 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:08.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:08.762 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:08.857 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:08.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:08.888 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:08.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:08.991 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:09.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:09.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:09.149 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:09.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:09.281 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:09.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.412 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:09.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.541 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:09.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: cluster 2024-04-26T16:36:07.915134+0000 mgr.y (mgr.24401) 720 : cluster [DBG] pgmap v1221: 204 pgs: 3 creating+activating, 201 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: cluster 2024-04-26T16:36:07.915134+0000 mgr.y (mgr.24401) 720 : cluster [DBG] pgmap v1221: 204 pgs: 3 creating+activating, 201 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: cluster 2024-04-26T16:36:08.424531+0000 mon.a (mon.0) 3691 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: cluster 2024-04-26T16:36:08.424531+0000 mon.a (mon.0) 3691 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: audit 2024-04-26T16:36:08.426536+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: audit 2024-04-26T16:36:08.426536+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:09.655 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: cluster 2024-04-26T16:36:08.432088+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-04-26T16:36:09.656 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:09 smithi190 bash[23868]: cluster 2024-04-26T16:36:08.432088+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-04-26T16:36:09.670 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:09.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.801 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: cluster 2024-04-26T16:36:07.915134+0000 mgr.y (mgr.24401) 720 : cluster [DBG] pgmap v1221: 204 pgs: 3 creating+activating, 201 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: cluster 2024-04-26T16:36:07.915134+0000 mgr.y (mgr.24401) 720 : cluster [DBG] pgmap v1221: 204 pgs: 3 creating+activating, 201 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: cluster 2024-04-26T16:36:08.424531+0000 mon.a (mon.0) 3691 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: cluster 2024-04-26T16:36:08.424531+0000 mon.a (mon.0) 3691 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: audit 2024-04-26T16:36:08.426536+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: audit 2024-04-26T16:36:08.426536+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: cluster 2024-04-26T16:36:08.432088+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-04-26T16:36:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[29234]: cluster 2024-04-26T16:36:08.432088+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: cluster 2024-04-26T16:36:07.915134+0000 mgr.y (mgr.24401) 720 : cluster [DBG] pgmap v1221: 204 pgs: 3 creating+activating, 201 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: cluster 2024-04-26T16:36:07.915134+0000 mgr.y (mgr.24401) 720 : cluster [DBG] pgmap v1221: 204 pgs: 3 creating+activating, 201 active+clean; 4.4 MiB data, 757 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: cluster 2024-04-26T16:36:08.424531+0000 mon.a (mon.0) 3691 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: cluster 2024-04-26T16:36:08.424531+0000 mon.a (mon.0) 3691 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: audit 2024-04-26T16:36:08.426536+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: audit 2024-04-26T16:36:08.426536+0000 mon.a (mon.0) 3692 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-136"}]': finished 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: cluster 2024-04-26T16:36:08.432088+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-04-26T16:36:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:09 smithi050 bash[21463]: cluster 2024-04-26T16:36:08.432088+0000 mon.a (mon.0) 3693 : cluster [DBG] osdmap e788: 8 total, 8 up, 8 in 2024-04-26T16:36:09.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:09.933 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:09.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:10.036 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:10.064 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:10.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:10.195 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:10.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:10.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:10.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 -- 172.21.15.50:0/1508896396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103e30 msgr2=0x7f4408110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 --2- 172.21.15.50:0/1508896396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103e30 0x7f4408110290 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f4400009990 tx=0x7f4400009ca0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 -- 172.21.15.50:0/1508896396 shutdown_connections 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 --2- 172.21.15.50:0/1508896396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 0x7f44081098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 --2- 172.21.15.50:0/1508896396 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103e30 0x7f4408110290 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 --2- 172.21.15.50:0/1508896396 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103470 0x7f44081038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 -- 172.21.15.50:0/1508896396 >> 172.21.15.50:0/1508896396 conn(0x7f440807a580 msgr2=0x7f440807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:10.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 -- 172.21.15.50:0/1508896396 shutdown_connections 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.450+0000 7f440da55700 1 -- 172.21.15.50:0/1508896396 wait complete. 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 librados: starting msgr at 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 librados: starting objecter 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 Processor -- start 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- start start 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 librados: setting wanted keys 2024-04-26T16:36:10.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 librados: calling monclient init 2024-04-26T16:36:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.453 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:10.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:10.454 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103470 0x7f4408199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 0x7f4408199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 0x7f440819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4408112af0 con 0x7f4408103e30 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4408112970 con 0x7f44081094c0 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4408112c70 con 0x7f4408103470 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f4406ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103470 0x7f4408199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 0x7f4408199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f4406ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103470 0x7f4408199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36760/0 (socket says 172.21.15.50:36760) 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f4406ffd700 1 -- 172.21.15.50:0/2414683474 learned_addr learned my addr 172.21.15.50:0/2414683474 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 0x7f4408199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:55046/0 (socket says 172.21.15.50:55046) 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44077fe700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 0x7f440819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 -- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103470 msgr2=0x7f4408199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:10.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103470 0x7f4408199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 -- 172.21.15.50:0/2414683474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 msgr2=0x7f440819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 0x7f440819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 -- 172.21.15.50:0/2414683474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440819e140 con 0x7f4408103e30 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44077fe700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 0x7f440819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f44067fc700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 0x7f4408199670 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00e9f0 tx=0x7f43fc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440ca53700 10 client.?.objecter ms_handle_connect 0x7f4408103e30 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440ca53700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc00cb70 con 0x7f4408103e30 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440819e430 con 0x7f4408103e30 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f43fc004480 con 0x7f4408103e30 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc003bf0 con 0x7f4408103e30 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 10 client.31011.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:10.456 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4408075650 con 0x7f4408103e30 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 librados: init done 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440806a110 con 0x7f4408103e30 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.454+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f43fc003e50 con 0x7f4408103e30 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f43f0071f80 0x7f43f0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(789..789 src has 577..789) ==== 9566+0+0 (secure 0 0 0) 0x7f43fc091e50 con 0x7f4408103e30 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 10 client.31011.objecter ms_dispatch 0x7f4408001550 osd_map(789..789 src has 577..789) 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 3 client.31011.objecter handle_osd_map got epochs [789,789] > 0 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 3 client.31011.objecter handle_osd_map decoding full epoch 789 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f4406ffd700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f43f0071f80 0x7f43f0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 20 client.31011.objecter dump_active .. 0 homeless 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f440ca53700 10 client.31011.objecter ms_handle_connect 0x7f43f0071f80 2024-04-26T16:36:10.457 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.458+0000 7f4406ffd700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f43f0071f80 0x7f43f0074440 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7f43f80060b0 tx=0x7f43f8009040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:10.459 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.462+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f43fc05d7d0 con 0x7f4408103e30 2024-04-26T16:36:10.554 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:10.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:10 smithi050 bash[21463]: cluster 2024-04-26T16:36:09.477347+0000 mon.a (mon.0) 3694 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-04-26T16:36:10.564 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:10 smithi050 bash[21463]: cluster 2024-04-26T16:36:09.477347+0000 mon.a (mon.0) 3694 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-04-26T16:36:10.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:10 smithi050 bash[29234]: cluster 2024-04-26T16:36:09.477347+0000 mon.a (mon.0) 3694 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-04-26T16:36:10.564 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:10 smithi050 bash[29234]: cluster 2024-04-26T16:36:09.477347+0000 mon.a (mon.0) 3694 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-04-26T16:36:10.578 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:10.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:10.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:10.693 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:10 smithi190 bash[23868]: cluster 2024-04-26T16:36:09.477347+0000 mon.a (mon.0) 3694 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-04-26T16:36:10.694 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:10 smithi190 bash[23868]: cluster 2024-04-26T16:36:09.477347+0000 mon.a (mon.0) 3694 : cluster [DBG] osdmap e789: 8 total, 8 up, 8 in 2024-04-26T16:36:10.706 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.710+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f4408075910 con 0x7f4408103e30 2024-04-26T16:36:10.710 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:10.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:10.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:10.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/4063872416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 msgr2=0x7fa858107e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/4063872416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 0x7fa858107e70 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7fa854009990 tx=0x7fa854009ca0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/4063872416 shutdown_connections 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/4063872416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8581024a0 0x7fa85810e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/4063872416 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 0x7fa858107e70 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/4063872416 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa858101ae0 0x7fa858101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/4063872416 >> 172.21.15.50:0/4063872416 conn(0x7fa858078990 msgr2=0x7fa8580ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/4063872416 shutdown_connections 2024-04-26T16:36:10.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/4063872416 wait complete. 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 librados: starting msgr at 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 librados: starting objecter 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 Processor -- start 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- start start 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 librados: setting wanted keys 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 librados: calling monclient init 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa858101ae0 0x7fa85819b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 0x7fa85819b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 0x7fa85819fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa858111a30 con 0x7fa858107a90 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa8581118b0 con 0x7fa858101ae0 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa858111bb0 con 0x7fa8581024a0 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 0x7fa85819b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85dd68700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 0x7fa85819fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 0x7fa85819b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:36778/0 (socket says 172.21.15.50:36778) 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 -- 172.21.15.50:0/2638756657 learned_addr learned my addr 172.21.15.50:0/2638756657 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa858101ae0 0x7fa85819b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa858101ae0 msgr2=0x7fa85819b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa858101ae0 0x7fa85819b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.958 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 msgr2=0x7fa85819fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 0x7fa85819fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8581a02c0 con 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa858107a90 0x7fa85819fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 0x7fa85819b7f0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fa84800eb30 tx=0x7fa84800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa84e7fc700 10 client.?.objecter ms_handle_connect 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa84e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84800cc90 con 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa84800ce30 con 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8581a05b0 con 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa848003d80 con 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 10 client.41636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8581a7e30 con 0x7fa8581024a0 2024-04-26T16:36:10.959 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 librados: init done 2024-04-26T16:36:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.958+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fa8581031d0 con 0x7fa8581024a0 2024-04-26T16:36:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa848003f20 con 0x7fa8581024a0 2024-04-26T16:36:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa844071f80 0x7fa844074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(790..790 src has 577..790) ==== 9939+0+0 (secure 0 0 0) 0x7fa848092210 con 0x7fa8581024a0 2024-04-26T16:36:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 osd_map(790..790 src has 577..790) 2024-04-26T16:36:10.961 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 3 client.41636.objecter handle_osd_map got epochs [790,790] > 0 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 3 client.41636.objecter handle_osd_map decoding full epoch 790 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa844071f80 0x7fa844074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 20 client.41636.objecter dump_active .. 0 homeless 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v790) ==== 54+0+16 (secure 0 0 0) 0x7fa848092660 con 0x7fa8581024a0 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 1 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa858064280 0x7fa8580fdd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa844071f80 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa844071f80 0x7fa844074440 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7fa85400b730 tx=0x7fa85401a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:10.962 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 1 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 0 1 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 4 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 10 client.41636.objecter _send_command 1 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa85806a110 con 0x7fa858064280 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa858064280 0x7fa8580fdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa858064280 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa858064280 0x7fa8580fdd30 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa85806a110 con 0x7fa858064280 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 1: 0 ) 2024-04-26T16:36:10.963 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 1 = system:0 2024-04-26T16:36:10.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 4 2024-04-26T16:36:10.964 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.962+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 0 1 2024-04-26T16:36:10.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:10.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:10.978 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 2 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 2 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa85819c610 osd=0 3 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 0 2 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 4 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 10 client.41636.objecter _send_command 2 2024-04-26T16:36:10.979 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8580fe270 con 0x7fa858064280 2024-04-26T16:36:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8580fe270 con 0x7fa858064280 2024-04-26T16:36:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 4 2024-04-26T16:36:10.980 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 0 2 2024-04-26T16:36:10.981 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:36:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 3 2024-04-26T16:36:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa858063e70 0x7fa8581ac0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:10.981 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 1 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 4 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 10 client.41636.objecter _send_command 3 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fa85802dff0 con 0x7fa858063e70 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa858063e70 0x7fa8581ac0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa858063e70 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa858063e70 0x7fa8581ac0e0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa85802dff0 con 0x7fa858063e70 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 3: 0 ) 2024-04-26T16:36:10.982 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 3 = system:0 2024-04-26T16:36:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 4 2024-04-26T16:36:10.983 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:10.982+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 1 3 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 4 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 4 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa858063d00 osd=1 3 2024-04-26T16:36:10.999 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 1 4 2024-04-26T16:36:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 4 2024-04-26T16:36:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 10 client.41636.objecter _send_command 4 2024-04-26T16:36:11.000 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8581ac620 con 0x7fa858063e70 2024-04-26T16:36:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8581ac620 con 0x7fa858063e70 2024-04-26T16:36:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 4 2024-04-26T16:36:11.001 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 1 4 2024-04-26T16:36:11.001 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 5 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa85802d6d0 0x7fa8581af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 5 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 2 5 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 4 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 10 client.41636.objecter _send_command 5 2024-04-26T16:36:11.002 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fa8580ff1e0 con 0x7fa85802d6d0 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.002+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa85802d6d0 0x7fa8581af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa85802d6d0 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa85802d6d0 0x7fa8581af960 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8580ff1e0 con 0x7fa85802d6d0 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 5: 0 ) 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 5 = system:0 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 4 2024-04-26T16:36:11.003 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.006+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 2 5 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 6 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 6 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa85802d560 osd=2 3 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 2 6 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 4 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 10 client.41636.objecter _send_command 6 2024-04-26T16:36:11.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8580fe670 con 0x7fa85802d6d0 2024-04-26T16:36:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8580fe670 con 0x7fa85802d6d0 2024-04-26T16:36:11.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 4 2024-04-26T16:36:11.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.022+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 2 6 2024-04-26T16:36:11.022 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 7 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa8581ad4b0 0x7fa8581b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 7 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 3 7 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 4 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 10 client.41636.objecter _send_command 7 2024-04-26T16:36:11.023 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fa8581b5f60 con 0x7fa8581ad4b0 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa8581ad4b0 0x7fa8581b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa8581ad4b0 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa8581ad4b0 0x7fa8581b3030 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8581b5f60 con 0x7fa8581ad4b0 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 7: 0 ) 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 7 = system:0 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 4 2024-04-26T16:36:11.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.026+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 3 7 2024-04-26T16:36:11.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:11.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 8 2024-04-26T16:36:11.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 8 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8580fea70 osd=3 3 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 3 8 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 4 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 10 client.41636.objecter _send_command 8 2024-04-26T16:36:11.041 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.042+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8581aca20 con 0x7fa8581ad4b0 2024-04-26T16:36:11.042 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8581aca20 con 0x7fa8581ad4b0 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 4 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 3 8 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 9 2024-04-26T16:36:11.043 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa8581b0b80 0x7fa8581b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 9 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 4 9 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 4 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 10 client.41636.objecter _send_command 9 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fa8581b9950 con 0x7fa8581b0b80 2024-04-26T16:36:11.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa8581b0b80 0x7fa8581b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa8581b0b80 2024-04-26T16:36:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa8581b0b80 0x7fa8581b35a0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8581b9950 con 0x7fa8581b0b80 2024-04-26T16:36:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 9: 0 ) 2024-04-26T16:36:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 9 = system:0 2024-04-26T16:36:11.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 4 2024-04-26T16:36:11.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.046+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 4 9 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 10 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 10 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581aca20 osd=4 3 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 4 10 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 4 2024-04-26T16:36:11.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 10 client.41636.objecter _send_command 10 2024-04-26T16:36:11.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.066+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8581b3ae0 con 0x7fa8581b0b80 2024-04-26T16:36:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8581b3ae0 con 0x7fa8581b0b80 2024-04-26T16:36:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 4 2024-04-26T16:36:11.067 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 4 10 2024-04-26T16:36:11.068 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:36:11.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 11 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa8581b41b0 0x7fa8581b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 11 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 5 11 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 4 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 10 client.41636.objecter _send_command 11 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fa8581bcbe0 con 0x7fa8581b41b0 2024-04-26T16:36:11.069 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa8581b41b0 0x7fa8581b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa85cd66700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa8581b41b0 0x7fa8581b4630 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa8581b41b0 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8581bcbe0 con 0x7fa8581b41b0 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 11: 0 ) 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 11 = system:0 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 4 2024-04-26T16:36:11.070 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.070+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 5 11 2024-04-26T16:36:11.073 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 12 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 12 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581b3ae0 osd=5 3 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 5 12 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 4 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 10 client.41636.objecter _send_command 12 2024-04-26T16:36:11.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.090+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8581ba210 con 0x7fa8581b41b0 2024-04-26T16:36:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8581ba210 con 0x7fa8581b41b0 2024-04-26T16:36:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 4 2024-04-26T16:36:11.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 5 12 2024-04-26T16:36:11.093 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:36:11.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:11.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 13 2024-04-26T16:36:11.093 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fa8581ba930 0x7fa8581bad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 13 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 6 13 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 4 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 10 client.41636.objecter _send_command 13 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fa8581bff80 con 0x7fa8581ba930 2024-04-26T16:36:11.094 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.094+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fa8581ba930 0x7fa8581bad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa8581ba930 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa85dd68700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fa8581ba930 0x7fa8581bad90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8581bff80 con 0x7fa8581ba930 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 13: 0 ) 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 13 = system:0 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 4 2024-04-26T16:36:11.096 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.098+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 6 13 2024-04-26T16:36:11.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 14 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 14 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581bcc40 osd=6 3 2024-04-26T16:36:11.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 6 14 2024-04-26T16:36:11.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 4 2024-04-26T16:36:11.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 10 client.41636.objecter _send_command 14 2024-04-26T16:36:11.117 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.118+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8581bb2d0 con 0x7fa8581ba930 2024-04-26T16:36:11.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8581bb2d0 con 0x7fa8581ba930 2024-04-26T16:36:11.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 4 2024-04-26T16:36:11.118 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 6 14 2024-04-26T16:36:11.119 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:36:11.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:11.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 15 2024-04-26T16:36:11.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa8581bb9f0 0x7fa8581bbe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:11.119 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 15 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 7 15 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 4 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 10 client.41636.objecter _send_command 15 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fa8581bc5a0 con 0x7fa8581bb9f0 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa8581bb9f0 0x7fa8581bbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa85d567700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa8581bb9f0 0x7fa8581bbe70 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:11.120 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 10 client.41636.objecter ms_handle_connect 0x7fa8581bb9f0 2024-04-26T16:36:11.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa8581bc5a0 con 0x7fa8581bb9f0 2024-04-26T16:36:11.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 15: 0 ) 2024-04-26T16:36:11.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 15 = system:0 2024-04-26T16:36:11.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 4 2024-04-26T16:36:11.121 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.122+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 7 15 2024-04-26T16:36:11.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 10 client.41636.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:11.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign -1 16 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 20 client.41636.objecter _get_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_remove -1 16 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 20 client.41636.objecter get_session s=0x7fa8581bb2d0 osd=7 3 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 15 client.41636.objecter _session_command_op_assign 7 16 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 4 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 10 client.41636.objecter _send_command 16 2024-04-26T16:36:11.139 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.142+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fa8581bc5a0 con 0x7fa8581bb9f0 2024-04-26T16:36:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa84e7fc700 1 -- 172.21.15.50:0/2638756657 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fa8581bc5a0 con 0x7fa8581bb9f0 2024-04-26T16:36:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa84e7fc700 10 client.41636.objecter ms_dispatch 0x7fa858001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa84e7fc700 10 client.41636.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa84e7fc700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 4 2024-04-26T16:36:11.143 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa84e7fc700 15 client.41636.objecter _session_command_op_remove 7 16 2024-04-26T16:36:11.143 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 librados: watch_flush enter 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 librados: watch_flush exit 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.0 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa858064280 msgr2=0x7fa8580fdd30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fa858064280 0x7fa8580fdd30 crc :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85819c610 osd=0 1 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.1 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa858063e70 msgr2=0x7fa8581ac0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fa858063e70 0x7fa8581ac0e0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa858063d00 osd=1 1 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.2 2024-04-26T16:36:11.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa85802d6d0 msgr2=0x7fa8581af960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fa85802d6d0 0x7fa8581af960 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa85802d560 osd=2 1 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.3 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa8581ad4b0 msgr2=0x7fa8581b3030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85dd68700 1 -- 172.21.15.50:0/2638756657 reap_dead start 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fa8581ad4b0 0x7fa8581b3030 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8580fea70 osd=3 1 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.4 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa8581b0b80 msgr2=0x7fa8581b35a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fa8581b0b80 0x7fa8581b35a0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581aca20 osd=4 1 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.5 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa8581b41b0 msgr2=0x7fa8581b4630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fa8581b41b0 0x7fa8581b4630 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581b3ae0 osd=5 1 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.6 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fa8581ba930 msgr2=0x7fa8581bad90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fa8581ba930 0x7fa8581bad90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bcc40 osd=6 1 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter close_session for osd.7 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa8581bb9f0 msgr2=0x7fa8581bbe70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fa8581bb9f0 0x7fa8581bbe70 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter put_session s=0x7fa8581bb2d0 osd=7 1 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 20 client.41636.objecter shutdown clearing up homeless session... 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 10 client.41636.objecter successfully canceled tick 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85dd68700 1 -- 172.21.15.50:0/2638756657 reap_dead start 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa844071f80 msgr2=0x7fa844074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa844071f80 0x7fa844074440 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7fa85400b730 tx=0x7fa85401a040 comp rx=0 tx=0).stop 2024-04-26T16:36:11.147 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 msgr2=0x7fa85819b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 0x7fa85819b7f0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7fa84800eb30 tx=0x7fa84800ee40 comp rx=0 tx=0).stop 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 shutdown_connections 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa8581024a0 0x7fa85819b7f0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 --2- 172.21.15.50:0/2638756657 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa844071f80 0x7fa844074440 unknown :-1 s=CLOSED pgs=3831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 >> 172.21.15.50:0/2638756657 conn(0x7fa858078990 msgr2=0x7fa8581051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 shutdown_connections 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 -- 172.21.15.50:0/2638756657 wait complete. 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.146+0000 7fa85f7cb700 1 librados: shutdown 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:36:11.148 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:36:11.149 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:36:11.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:11.222 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:11.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:11.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:11.354 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:11.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.366+0000 7f440ca53700 1 -- 172.21.15.50:0/2414683474 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v791) ==== 91+0+0 (secure 0 0 0) 0x7f43fc061480 con 0x7f4408103e30 2024-04-26T16:36:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.370+0000 7f440da55700 10 librados: watch_flush enter 2024-04-26T16:36:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.370+0000 7f440da55700 10 librados: watch_flush exit 2024-04-26T16:36:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.370+0000 7f440da55700 20 client.31011.objecter shutdown clearing up homeless session... 2024-04-26T16:36:11.369 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.370+0000 7f440da55700 10 client.31011.objecter successfully canceled tick 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f43f0071f80 msgr2=0x7f43f0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f43f0071f80 0x7f43f0074440 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7f43f80060b0 tx=0x7f43f8009040 comp rx=0 tx=0).stop 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 msgr2=0x7f4408199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 0x7f4408199670 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00e9f0 tx=0x7f43fc00ed00 comp rx=0 tx=0).stop 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 shutdown_connections 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f43f0071f80 0x7f43f0074440 unknown :-1 s=CLOSED pgs=3830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44081094c0 0x7f440819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4408103470 0x7f4408199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 --2- 172.21.15.50:0/2414683474 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4408103e30 0x7f4408199670 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:11.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 >> 172.21.15.50:0/2414683474 conn(0x7f440807a580 msgr2=0x7f4408101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 shutdown_connections 2024-04-26T16:36:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 -- 172.21.15.50:0/2414683474 wait complete. 2024-04-26T16:36:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:11.374+0000 7f440da55700 1 librados: shutdown 2024-04-26T16:36:11.372 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:36:11.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:11.481 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:11.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:11.584 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:11.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:11.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:09.915764+0000 mgr.y (mgr.24401) 721 : cluster [DBG] pgmap v1224: 172 pgs: 172 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:09.915764+0000 mgr.y (mgr.24401) 721 : cluster [DBG] pgmap v1224: 172 pgs: 172 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:10.468081+0000 osd.0 (osd.0) 427 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:10.468081+0000 osd.0 (osd.0) 427 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:10.468242+0000 osd.0 (osd.0) 428 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:10.468242+0000 osd.0 (osd.0) 428 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:10.485607+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-04-26T16:36:11.726 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:10.485607+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.488701+0000 mon.c (mon.2) 818 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.488701+0000 mon.c (mon.2) 818 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.489775+0000 mon.a (mon.0) 3696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.489775+0000 mon.a (mon.0) 3696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.711685+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.711685+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.963550+0000 mon.c (mon.2) 819 : audit [DBG] from='client.? 172.21.15.50:0/2638756657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:10.963550+0000 mon.c (mon.2) 819 : audit [DBG] from='client.? 172.21.15.50:0/2638756657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:11.367444+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:11.367444+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.368855+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:11.727 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.368855+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.368954+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.368954+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:11.377659+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: cluster 2024-04-26T16:36:11.377659+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.379516+0000 mon.c (mon.2) 820 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.379516+0000 mon.c (mon.2) 820 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.381086+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:11 smithi190 bash[23868]: audit 2024-04-26T16:36:11.381086+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.739 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:11.842 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:11.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:09.915764+0000 mgr.y (mgr.24401) 721 : cluster [DBG] pgmap v1224: 172 pgs: 172 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:09.915764+0000 mgr.y (mgr.24401) 721 : cluster [DBG] pgmap v1224: 172 pgs: 172 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:10.468081+0000 osd.0 (osd.0) 427 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:10.468081+0000 osd.0 (osd.0) 427 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:10.468242+0000 osd.0 (osd.0) 428 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:10.468242+0000 osd.0 (osd.0) 428 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:10.485607+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:10.485607+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.488701+0000 mon.c (mon.2) 818 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.488701+0000 mon.c (mon.2) 818 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.489775+0000 mon.a (mon.0) 3696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.489775+0000 mon.a (mon.0) 3696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.711685+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.711685+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:36:11.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.963550+0000 mon.c (mon.2) 819 : audit [DBG] from='client.? 172.21.15.50:0/2638756657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:10.963550+0000 mon.c (mon.2) 819 : audit [DBG] from='client.? 172.21.15.50:0/2638756657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:11.367444+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:11.367444+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.368855+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.368855+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.368954+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.368954+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:11.377659+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-04-26T16:36:11.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: cluster 2024-04-26T16:36:11.377659+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.379516+0000 mon.c (mon.2) 820 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.379516+0000 mon.c (mon.2) 820 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.381086+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[29234]: audit 2024-04-26T16:36:11.381086+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:09.915764+0000 mgr.y (mgr.24401) 721 : cluster [DBG] pgmap v1224: 172 pgs: 172 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:09.915764+0000 mgr.y (mgr.24401) 721 : cluster [DBG] pgmap v1224: 172 pgs: 172 active+clean; 4.4 MiB data, 759 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:10.468081+0000 osd.0 (osd.0) 427 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:10.468081+0000 osd.0 (osd.0) 427 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:11.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:10.468242+0000 osd.0 (osd.0) 428 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:10.468242+0000 osd.0 (osd.0) 428 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:10.485607+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:10.485607+0000 mon.a (mon.0) 3695 : cluster [DBG] osdmap e790: 8 total, 8 up, 8 in 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.488701+0000 mon.c (mon.2) 818 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.488701+0000 mon.c (mon.2) 818 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.489775+0000 mon.a (mon.0) 3696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.489775+0000 mon.a (mon.0) 3696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.711685+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.711685+0000 mon.a (mon.0) 3697 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.963550+0000 mon.c (mon.2) 819 : audit [DBG] from='client.? 172.21.15.50:0/2638756657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:10.963550+0000 mon.c (mon.2) 819 : audit [DBG] from='client.? 172.21.15.50:0/2638756657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:11.367444+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:36:11.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:11.367444+0000 mon.a (mon.0) 3698 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.368855+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.368855+0000 mon.a (mon.0) 3699 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.368954+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.368954+0000 mon.a (mon.0) 3700 : audit [INF] from='client.? 172.21.15.50:0/2414683474' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:11.377659+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: cluster 2024-04-26T16:36:11.377659+0000 mon.a (mon.0) 3701 : cluster [DBG] osdmap e791: 8 total, 8 up, 8 in 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.379516+0000 mon.c (mon.2) 820 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.379516+0000 mon.c (mon.2) 820 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.381086+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:11 smithi050 bash[21463]: audit 2024-04-26T16:36:11.381086+0000 mon.a (mon.0) 3702 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:11.869 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:11.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:11.972 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:12.000 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:12.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:12.131 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:12.234 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:12.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:12.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:12.393 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:12.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:12.522 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:12.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:12.625 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:12.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:12.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:12.781 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:12.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:12.912 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:12.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:13.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:13.042 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:13.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:13.173 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:13.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:13.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:13.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:13.433 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:13.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:13.562 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:13.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: cluster 2024-04-26T16:36:11.917052+0000 mgr.y (mgr.24401) 722 : cluster [DBG] pgmap v1227: 204 pgs: 2 unknown, 202 active+clean; 4.4 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: cluster 2024-04-26T16:36:11.917052+0000 mgr.y (mgr.24401) 722 : cluster [DBG] pgmap v1227: 204 pgs: 2 unknown, 202 active+clean; 4.4 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.372004+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.372004+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: cluster 2024-04-26T16:36:12.380455+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: cluster 2024-04-26T16:36:12.380455+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.384299+0000 mon.c (mon.2) 821 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.384299+0000 mon.c (mon.2) 821 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.385331+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.385331+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.778671+0000 mon.c (mon.2) 822 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:13.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[29234]: audit 2024-04-26T16:36:12.778671+0000 mon.c (mon.2) 822 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: cluster 2024-04-26T16:36:11.917052+0000 mgr.y (mgr.24401) 722 : cluster [DBG] pgmap v1227: 204 pgs: 2 unknown, 202 active+clean; 4.4 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: cluster 2024-04-26T16:36:11.917052+0000 mgr.y (mgr.24401) 722 : cluster [DBG] pgmap v1227: 204 pgs: 2 unknown, 202 active+clean; 4.4 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.372004+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.372004+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: cluster 2024-04-26T16:36:12.380455+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: cluster 2024-04-26T16:36:12.380455+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.384299+0000 mon.c (mon.2) 821 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.384299+0000 mon.c (mon.2) 821 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.385331+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.385331+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.778671+0000 mon.c (mon.2) 822 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:13.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:13 smithi050 bash[21463]: audit 2024-04-26T16:36:12.778671+0000 mon.c (mon.2) 822 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:13.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: cluster 2024-04-26T16:36:11.917052+0000 mgr.y (mgr.24401) 722 : cluster [DBG] pgmap v1227: 204 pgs: 2 unknown, 202 active+clean; 4.4 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: cluster 2024-04-26T16:36:11.917052+0000 mgr.y (mgr.24401) 722 : cluster [DBG] pgmap v1227: 204 pgs: 2 unknown, 202 active+clean; 4.4 MiB data, 765 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.372004+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.372004+0000 mon.a (mon.0) 3703 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: cluster 2024-04-26T16:36:12.380455+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: cluster 2024-04-26T16:36:12.380455+0000 mon.a (mon.0) 3704 : cluster [DBG] osdmap e792: 8 total, 8 up, 8 in 2024-04-26T16:36:13.677 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.384299+0000 mon.c (mon.2) 821 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.384299+0000 mon.c (mon.2) 821 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.385331+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.385331+0000 mon.a (mon.0) 3705 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:13.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.778671+0000 mon.c (mon.2) 822 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:13.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:13 smithi190 bash[23868]: audit 2024-04-26T16:36:12.778671+0000 mon.c (mon.2) 822 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:13.692 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:13.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:13.822 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:13.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:13.952 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:13.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:14.056 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:14.084 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:14.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:14.187 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:14.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:14.320 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:14.352 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:14.455 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:14.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.594 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:36:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.114653+0000 osd.1 (osd.1) 399 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.114653+0000 osd.1 (osd.1) 399 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.117013+0000 osd.1 (osd.1) 400 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.117013+0000 osd.1 (osd.1) 400 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.192266+0000 osd.7 (osd.7) 401 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.192266+0000 osd.7 (osd.7) 401 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.193606+0000 osd.7 (osd.7) 402 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.193606+0000 osd.7 (osd.7) 402 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.337984+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.f scrub starts 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.337984+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.f scrub starts 2024-04-26T16:36:14.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.340220+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.f scrub ok 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.340220+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.f scrub ok 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: audit 2024-04-26T16:36:13.374037+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: audit 2024-04-26T16:36:13.374037+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.377855+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.377855+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: audit 2024-04-26T16:36:13.381903+0000 mon.c (mon.2) 823 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: audit 2024-04-26T16:36:13.381903+0000 mon.c (mon.2) 823 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: audit 2024-04-26T16:36:13.393071+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: audit 2024-04-26T16:36:13.393071+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.514977+0000 osd.3 (osd.3) 433 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.514977+0000 osd.3 (osd.3) 433 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.521651+0000 osd.3 (osd.3) 434 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:36:14.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.521651+0000 osd.3 (osd.3) 434 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.114653+0000 osd.1 (osd.1) 399 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.114653+0000 osd.1 (osd.1) 399 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.117013+0000 osd.1 (osd.1) 400 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.117013+0000 osd.1 (osd.1) 400 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.192266+0000 osd.7 (osd.7) 401 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.192266+0000 osd.7 (osd.7) 401 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.193606+0000 osd.7 (osd.7) 402 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.193606+0000 osd.7 (osd.7) 402 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.337984+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.f scrub starts 2024-04-26T16:36:14.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.337984+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.f scrub starts 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.340220+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.f scrub ok 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.340220+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.f scrub ok 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: audit 2024-04-26T16:36:13.374037+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: audit 2024-04-26T16:36:13.374037+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.377855+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.377855+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: audit 2024-04-26T16:36:13.381903+0000 mon.c (mon.2) 823 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: audit 2024-04-26T16:36:13.381903+0000 mon.c (mon.2) 823 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: audit 2024-04-26T16:36:13.393071+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: audit 2024-04-26T16:36:13.393071+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.514977+0000 osd.3 (osd.3) 433 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.514977+0000 osd.3 (osd.3) 433 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:36:14.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.521651+0000 osd.3 (osd.3) 434 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:36:14.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:14 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.521651+0000 osd.3 (osd.3) 434 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:36:14.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:14.714 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:14.741 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.114653+0000 osd.1 (osd.1) 399 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:36:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.114653+0000 osd.1 (osd.1) 399 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:36:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.117013+0000 osd.1 (osd.1) 400 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:36:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.117013+0000 osd.1 (osd.1) 400 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:36:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.192266+0000 osd.7 (osd.7) 401 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:36:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.192266+0000 osd.7 (osd.7) 401 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.193606+0000 osd.7 (osd.7) 402 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.193606+0000 osd.7 (osd.7) 402 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.337984+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.f scrub starts 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.337984+0000 osd.5 (osd.5) 291 : cluster [DBG] 5.f scrub starts 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.340220+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.f scrub ok 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.340220+0000 osd.5 (osd.5) 292 : cluster [DBG] 5.f scrub ok 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: audit 2024-04-26T16:36:13.374037+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: audit 2024-04-26T16:36:13.374037+0000 mon.a (mon.0) 3706 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.377855+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.377855+0000 mon.a (mon.0) 3707 : cluster [DBG] osdmap e793: 8 total, 8 up, 8 in 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: audit 2024-04-26T16:36:13.381903+0000 mon.c (mon.2) 823 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: audit 2024-04-26T16:36:13.381903+0000 mon.c (mon.2) 823 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: audit 2024-04-26T16:36:13.393071+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: audit 2024-04-26T16:36:13.393071+0000 mon.a (mon.0) 3708 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:14.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.514977+0000 osd.3 (osd.3) 433 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:36:14.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.514977+0000 osd.3 (osd.3) 433 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:36:14.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.521651+0000 osd.3 (osd.3) 434 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:36:14.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:14 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.521651+0000 osd.3 (osd.3) 434 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:36:14.844 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.868 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:14.971 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.995 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:14.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:15.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:15.126 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:15.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:15.254 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:15.282 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:36:15.356 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:15.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:15.486 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:15.506 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:15.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:15.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.918014+0000 mgr.y (mgr.24401) 723 : cluster [DBG] pgmap v1230: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.918014+0000 mgr.y (mgr.24401) 723 : cluster [DBG] pgmap v1230: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.962310+0000 osd.0 (osd.0) 429 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.962310+0000 osd.0 (osd.0) 429 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.964914+0000 osd.0 (osd.0) 430 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:13.964914+0000 osd.0 (osd.0) 430 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.268559+0000 osd.4 (osd.4) 375 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.268559+0000 osd.4 (osd.4) 375 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.273545+0000 osd.4 (osd.4) 376 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.273545+0000 osd.4 (osd.4) 376 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: audit 2024-04-26T16:36:14.387315+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: audit 2024-04-26T16:36:14.387315+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.390891+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.390891+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-04-26T16:36:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: audit 2024-04-26T16:36:14.394511+0000 mon.c (mon.2) 824 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: audit 2024-04-26T16:36:14.394511+0000 mon.c (mon.2) 824 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: audit 2024-04-26T16:36:14.401166+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: audit 2024-04-26T16:36:14.401166+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.554072+0000 osd.3 (osd.3) 435 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.554072+0000 osd.3 (osd.3) 435 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.556701+0000 osd.3 (osd.3) 436 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.556701+0000 osd.3 (osd.3) 436 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.918014+0000 mgr.y (mgr.24401) 723 : cluster [DBG] pgmap v1230: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.918014+0000 mgr.y (mgr.24401) 723 : cluster [DBG] pgmap v1230: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.962310+0000 osd.0 (osd.0) 429 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:36:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.962310+0000 osd.0 (osd.0) 429 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.964914+0000 osd.0 (osd.0) 430 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:13.964914+0000 osd.0 (osd.0) 430 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.268559+0000 osd.4 (osd.4) 375 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.268559+0000 osd.4 (osd.4) 375 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.273545+0000 osd.4 (osd.4) 376 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.273545+0000 osd.4 (osd.4) 376 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: audit 2024-04-26T16:36:14.387315+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: audit 2024-04-26T16:36:14.387315+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.390891+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.390891+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: audit 2024-04-26T16:36:14.394511+0000 mon.c (mon.2) 824 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: audit 2024-04-26T16:36:14.394511+0000 mon.c (mon.2) 824 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: audit 2024-04-26T16:36:14.401166+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: audit 2024-04-26T16:36:14.401166+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.554072+0000 osd.3 (osd.3) 435 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:36:15.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.554072+0000 osd.3 (osd.3) 435 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:36:15.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.556701+0000 osd.3 (osd.3) 436 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:36:15.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:15 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.556701+0000 osd.3 (osd.3) 436 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:36:15.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:15.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.918014+0000 mgr.y (mgr.24401) 723 : cluster [DBG] pgmap v1230: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.918014+0000 mgr.y (mgr.24401) 723 : cluster [DBG] pgmap v1230: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.962310+0000 osd.0 (osd.0) 429 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.962310+0000 osd.0 (osd.0) 429 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.964914+0000 osd.0 (osd.0) 430 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:13.964914+0000 osd.0 (osd.0) 430 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.268559+0000 osd.4 (osd.4) 375 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:36:15.751 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.268559+0000 osd.4 (osd.4) 375 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.273545+0000 osd.4 (osd.4) 376 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.273545+0000 osd.4 (osd.4) 376 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: audit 2024-04-26T16:36:14.387315+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: audit 2024-04-26T16:36:14.387315+0000 mon.a (mon.0) 3709 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.390891+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.390891+0000 mon.a (mon.0) 3710 : cluster [DBG] osdmap e794: 8 total, 8 up, 8 in 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: audit 2024-04-26T16:36:14.394511+0000 mon.c (mon.2) 824 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: audit 2024-04-26T16:36:14.394511+0000 mon.c (mon.2) 824 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: audit 2024-04-26T16:36:14.401166+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: audit 2024-04-26T16:36:14.401166+0000 mon.a (mon.0) 3711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.554072+0000 osd.3 (osd.3) 435 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.554072+0000 osd.3 (osd.3) 435 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.556701+0000 osd.3 (osd.3) 436 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:36:15.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:15 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.556701+0000 osd.3 (osd.3) 436 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:36:15.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:15.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:15.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:15.877 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:15.896 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:15.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:15.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:15.989 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:36:15.989 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:36:15.999 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:16.018 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:16.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:16.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:16.250 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:16.279 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:16.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:36:16.383 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:16.413 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:16.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.937986+0000 osd.0 (osd.0) 431 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.937986+0000 osd.0 (osd.0) 431 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.940480+0000 osd.0 (osd.0) 432 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:14.940480+0000 osd.0 (osd.0) 432 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.183358+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.183358+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.186364+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.186364+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.208319+0000 osd.7 (osd.7) 403 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.208319+0000 osd.7 (osd.7) 403 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:36:16.531 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.209870+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.209870+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: audit 2024-04-26T16:36:15.397656+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: audit 2024-04-26T16:36:15.397656+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.401420+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.401420+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.601757+0000 osd.3 (osd.3) 437 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.601757+0000 osd.3 (osd.3) 437 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.604637+0000 osd.3 (osd.3) 438 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:36:16.532 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.604637+0000 osd.3 (osd.3) 438 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.937986+0000 osd.0 (osd.0) 431 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.937986+0000 osd.0 (osd.0) 431 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.940480+0000 osd.0 (osd.0) 432 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:14.940480+0000 osd.0 (osd.0) 432 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.183358+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.183358+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.186364+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.186364+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.208319+0000 osd.7 (osd.7) 403 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.208319+0000 osd.7 (osd.7) 403 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.209870+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.209870+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:36:16.533 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: audit 2024-04-26T16:36:15.397656+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: audit 2024-04-26T16:36:15.397656+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.401420+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.401420+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.601757+0000 osd.3 (osd.3) 437 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.601757+0000 osd.3 (osd.3) 437 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.604637+0000 osd.3 (osd.3) 438 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:36:16.534 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:16 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.604637+0000 osd.3 (osd.3) 438 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:36:16.543 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:16.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:16.676 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.937986+0000 osd.0 (osd.0) 431 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.937986+0000 osd.0 (osd.0) 431 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.940480+0000 osd.0 (osd.0) 432 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:14.940480+0000 osd.0 (osd.0) 432 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.183358+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.183358+0000 osd.1 (osd.1) 401 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.186364+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.186364+0000 osd.1 (osd.1) 402 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.208319+0000 osd.7 (osd.7) 403 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.208319+0000 osd.7 (osd.7) 403 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:36:16.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.209870+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.209870+0000 osd.7 (osd.7) 404 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: audit 2024-04-26T16:36:15.397656+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: audit 2024-04-26T16:36:15.397656+0000 mon.a (mon.0) 3712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.401420+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.401420+0000 mon.a (mon.0) 3713 : cluster [DBG] osdmap e795: 8 total, 8 up, 8 in 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.601757+0000 osd.3 (osd.3) 437 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.601757+0000 osd.3 (osd.3) 437 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.604637+0000 osd.3 (osd.3) 438 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:36:16.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:16 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.604637+0000 osd.3 (osd.3) 438 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:36:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:16.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:16.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:16.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:16.940 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:16.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:17.043 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:17.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:17.178 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:17.204 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:17.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:17.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:17.439 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:17.465 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:17.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:17.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.918527+0000 mgr.y (mgr.24401) 724 : cluster [DBG] pgmap v1233: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:15.918527+0000 mgr.y (mgr.24401) 724 : cluster [DBG] pgmap v1233: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.200432+0000 osd.1 (osd.1) 403 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.200432+0000 osd.1 (osd.1) 403 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.205448+0000 osd.1 (osd.1) 404 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.205448+0000 osd.1 (osd.1) 404 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:36:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.437443+0000 mon.c (mon.2) 825 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.437443+0000 mon.c (mon.2) 825 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.437863+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.437863+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.438520+0000 mon.c (mon.2) 826 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.438520+0000 mon.c (mon.2) 826 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.438894+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: audit 2024-04-26T16:36:16.438894+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.242528+0000 osd.4 (osd.4) 377 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.242528+0000 osd.4 (osd.4) 377 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.244737+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:36:17.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:17 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.244737+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:36:17.597 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.918527+0000 mgr.y (mgr.24401) 724 : cluster [DBG] pgmap v1233: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:15.918527+0000 mgr.y (mgr.24401) 724 : cluster [DBG] pgmap v1233: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.200432+0000 osd.1 (osd.1) 403 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.200432+0000 osd.1 (osd.1) 403 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.205448+0000 osd.1 (osd.1) 404 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.205448+0000 osd.1 (osd.1) 404 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:36:17.682 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.437443+0000 mon.c (mon.2) 825 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.437443+0000 mon.c (mon.2) 825 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.437863+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.437863+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.438520+0000 mon.c (mon.2) 826 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.438520+0000 mon.c (mon.2) 826 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.438894+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: audit 2024-04-26T16:36:16.438894+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.242528+0000 osd.4 (osd.4) 377 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.242528+0000 osd.4 (osd.4) 377 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.244737+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.244737+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:36:17.683 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.918527+0000 mgr.y (mgr.24401) 724 : cluster [DBG] pgmap v1233: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:15.918527+0000 mgr.y (mgr.24401) 724 : cluster [DBG] pgmap v1233: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.200432+0000 osd.1 (osd.1) 403 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.200432+0000 osd.1 (osd.1) 403 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.205448+0000 osd.1 (osd.1) 404 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.205448+0000 osd.1 (osd.1) 404 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.437443+0000 mon.c (mon.2) 825 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.437443+0000 mon.c (mon.2) 825 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.437863+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.437863+0000 mon.a (mon.0) 3714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.438520+0000 mon.c (mon.2) 826 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.438520+0000 mon.c (mon.2) 826 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.438894+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: audit 2024-04-26T16:36:16.438894+0000 mon.a (mon.0) 3715 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]: dispatch 2024-04-26T16:36:17.684 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.242528+0000 osd.4 (osd.4) 377 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:36:17.685 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.242528+0000 osd.4 (osd.4) 377 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:36:17.685 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.244737+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:36:17.685 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:17 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.244737+0000 osd.4 (osd.4) 378 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:36:17.700 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:17.726 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:17.829 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:17.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:17.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:17.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:17.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:18.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:18.122 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:18.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:18.255 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:18.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:18.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:18.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:18.490 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:18.513 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.660351+0000 osd.2 (osd.2) 371 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.660351+0000 osd.2 (osd.2) 371 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.662907+0000 osd.2 (osd.2) 372 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: cluster 2024-04-26T16:36:16.662907+0000 osd.2 (osd.2) 372 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: audit 2024-04-26T16:36:17.270208+0000 mgr.y (mgr.24401) 725 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: audit 2024-04-26T16:36:17.270208+0000 mgr.y (mgr.24401) 725 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: audit 2024-04-26T16:36:17.420095+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]': finished 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: audit 2024-04-26T16:36:17.420095+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]': finished 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.428478+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-04-26T16:36:18.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:18 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.428478+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-04-26T16:36:18.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:18.637 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-04-26T16:36:18.640 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:18.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:18.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:18.770 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:18.831 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.660351+0000 osd.2 (osd.2) 371 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.660351+0000 osd.2 (osd.2) 371 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.662907+0000 osd.2 (osd.2) 372 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: cluster 2024-04-26T16:36:16.662907+0000 osd.2 (osd.2) 372 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: audit 2024-04-26T16:36:17.270208+0000 mgr.y (mgr.24401) 725 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: audit 2024-04-26T16:36:17.270208+0000 mgr.y (mgr.24401) 725 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: audit 2024-04-26T16:36:17.420095+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]': finished 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: audit 2024-04-26T16:36:17.420095+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]': finished 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.428478+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.428478+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.660351+0000 osd.2 (osd.2) 371 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.660351+0000 osd.2 (osd.2) 371 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:36:18.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.662907+0000 osd.2 (osd.2) 372 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: cluster 2024-04-26T16:36:16.662907+0000 osd.2 (osd.2) 372 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: audit 2024-04-26T16:36:17.270208+0000 mgr.y (mgr.24401) 725 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: audit 2024-04-26T16:36:17.270208+0000 mgr.y (mgr.24401) 725 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: audit 2024-04-26T16:36:17.420095+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]': finished 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: audit 2024-04-26T16:36:17.420095+0000 mon.a (mon.0) 3716 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-138"}]': finished 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.428478+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-04-26T16:36:18.833 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:18 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.428478+0000 mon.a (mon.0) 3717 : cluster [DBG] osdmap e796: 8 total, 8 up, 8 in 2024-04-26T16:36:18.872 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:18.891 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:18.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:18.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:19.021 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:19.124 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:19.151 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:19.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:19.254 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:19.280 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:19.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:19.409 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:19.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:19.536 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:19.639 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:19.668 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:19.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.919316+0000 mgr.y (mgr.24401) 726 : cluster [DBG] pgmap v1235: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:17.919316+0000 mgr.y (mgr.24401) 726 : cluster [DBG] pgmap v1235: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.154976+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.a scrub starts 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.154976+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.a scrub starts 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.156813+0000 osd.1 (osd.1) 406 : cluster [DBG] 2.a scrub ok 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.156813+0000 osd.1 (osd.1) 406 : cluster [DBG] 2.a scrub ok 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.434381+0000 mon.a (mon.0) 3718 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.434381+0000 mon.a (mon.0) 3718 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.448465+0000 mon.a (mon.0) 3719 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-04-26T16:36:19.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:19 smithi190 bash[23868]: cluster 2024-04-26T16:36:18.448465+0000 mon.a (mon.0) 3719 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-04-26T16:36:19.797 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:19.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:19.925 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:19.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:20.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:20.040 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.919316+0000 mgr.y (mgr.24401) 726 : cluster [DBG] pgmap v1235: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:20.040 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:17.919316+0000 mgr.y (mgr.24401) 726 : cluster [DBG] pgmap v1235: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:20.040 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.154976+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.a scrub starts 2024-04-26T16:36:20.040 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.154976+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.a scrub starts 2024-04-26T16:36:20.040 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.156813+0000 osd.1 (osd.1) 406 : cluster [DBG] 2.a scrub ok 2024-04-26T16:36:20.040 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.156813+0000 osd.1 (osd.1) 406 : cluster [DBG] 2.a scrub ok 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.434381+0000 mon.a (mon.0) 3718 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.434381+0000 mon.a (mon.0) 3718 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.448465+0000 mon.a (mon.0) 3719 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[29234]: cluster 2024-04-26T16:36:18.448465+0000 mon.a (mon.0) 3719 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.919316+0000 mgr.y (mgr.24401) 726 : cluster [DBG] pgmap v1235: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:17.919316+0000 mgr.y (mgr.24401) 726 : cluster [DBG] pgmap v1235: 204 pgs: 204 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.154976+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.a scrub starts 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.154976+0000 osd.1 (osd.1) 405 : cluster [DBG] 2.a scrub starts 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.156813+0000 osd.1 (osd.1) 406 : cluster [DBG] 2.a scrub ok 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.156813+0000 osd.1 (osd.1) 406 : cluster [DBG] 2.a scrub ok 2024-04-26T16:36:20.041 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.434381+0000 mon.a (mon.0) 3718 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:20.042 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.434381+0000 mon.a (mon.0) 3718 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:20.042 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.448465+0000 mon.a (mon.0) 3719 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-04-26T16:36:20.042 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:19 smithi050 bash[21463]: cluster 2024-04-26T16:36:18.448465+0000 mon.a (mon.0) 3719 : cluster [DBG] osdmap e797: 8 total, 8 up, 8 in 2024-04-26T16:36:20.054 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:20.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:20.185 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:20.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:20.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:20.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:20.444 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:20.547 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:20.571 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:20.674 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:20.698 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:20.801 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:20.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.163104+0000 osd.7 (osd.7) 405 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:36:20.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.163104+0000 osd.7 (osd.7) 405 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:36:20.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.167119+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:36:20.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.167119+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.201928+0000 osd.1 (osd.1) 407 : cluster [DBG] 323.0s0 scrub starts 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.201928+0000 osd.1 (osd.1) 407 : cluster [DBG] 323.0s0 scrub starts 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.204978+0000 osd.1 (osd.1) 408 : cluster [DBG] 323.0 scrub ok 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.204978+0000 osd.1 (osd.1) 408 : cluster [DBG] 323.0 scrub ok 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.463339+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.463339+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: audit 2024-04-26T16:36:19.467552+0000 mon.c (mon.2) 827 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: audit 2024-04-26T16:36:19.467552+0000 mon.c (mon.2) 827 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: audit 2024-04-26T16:36:19.469141+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[21463]: audit 2024-04-26T16:36:19.469141+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.814 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.163104+0000 osd.7 (osd.7) 405 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.163104+0000 osd.7 (osd.7) 405 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.167119+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.167119+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.201928+0000 osd.1 (osd.1) 407 : cluster [DBG] 323.0s0 scrub starts 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.201928+0000 osd.1 (osd.1) 407 : cluster [DBG] 323.0s0 scrub starts 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.204978+0000 osd.1 (osd.1) 408 : cluster [DBG] 323.0 scrub ok 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.204978+0000 osd.1 (osd.1) 408 : cluster [DBG] 323.0 scrub ok 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.463339+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.463339+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: audit 2024-04-26T16:36:19.467552+0000 mon.c (mon.2) 827 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: audit 2024-04-26T16:36:19.467552+0000 mon.c (mon.2) 827 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: audit 2024-04-26T16:36:19.469141+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.815 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:20 smithi050 bash[29234]: audit 2024-04-26T16:36:19.469141+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.163104+0000 osd.7 (osd.7) 405 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.163104+0000 osd.7 (osd.7) 405 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.167119+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.167119+0000 osd.7 (osd.7) 406 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.201928+0000 osd.1 (osd.1) 407 : cluster [DBG] 323.0s0 scrub starts 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.201928+0000 osd.1 (osd.1) 407 : cluster [DBG] 323.0s0 scrub starts 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.204978+0000 osd.1 (osd.1) 408 : cluster [DBG] 323.0 scrub ok 2024-04-26T16:36:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.204978+0000 osd.1 (osd.1) 408 : cluster [DBG] 323.0 scrub ok 2024-04-26T16:36:20.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.463339+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-04-26T16:36:20.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.463339+0000 mon.a (mon.0) 3720 : cluster [DBG] osdmap e798: 8 total, 8 up, 8 in 2024-04-26T16:36:20.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: audit 2024-04-26T16:36:19.467552+0000 mon.c (mon.2) 827 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: audit 2024-04-26T16:36:19.467552+0000 mon.c (mon.2) 827 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: audit 2024-04-26T16:36:19.469141+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:20 smithi190 bash[23868]: audit 2024-04-26T16:36:19.469141+0000 mon.a (mon.0) 3721 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:20.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:20.960 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:20.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:21.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:21.092 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:21.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:21.216 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:21.319 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:21.347 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:21.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:21.478 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:21.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:21.581 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:21.602 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:21.705 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:21.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.920089+0000 mgr.y (mgr.24401) 727 : cluster [DBG] pgmap v1238: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:21.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:19.920089+0000 mgr.y (mgr.24401) 727 : cluster [DBG] pgmap v1238: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:21.715 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.156255+0000 osd.7 (osd.7) 407 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.156255+0000 osd.7 (osd.7) 407 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.159402+0000 osd.7 (osd.7) 408 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.159402+0000 osd.7 (osd.7) 408 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.224693+0000 osd.1 (osd.1) 409 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.224693+0000 osd.1 (osd.1) 409 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.227429+0000 osd.1 (osd.1) 410 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.227429+0000 osd.1 (osd.1) 410 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: audit 2024-04-26T16:36:20.466142+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: audit 2024-04-26T16:36:20.466142+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.477969+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.477969+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: audit 2024-04-26T16:36:20.483844+0000 mon.c (mon.2) 828 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: audit 2024-04-26T16:36:20.483844+0000 mon.c (mon.2) 828 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: audit 2024-04-26T16:36:20.485991+0000 mon.a (mon.0) 3724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.716 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[29234]: audit 2024-04-26T16:36:20.485991+0000 mon.a (mon.0) 3724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.920089+0000 mgr.y (mgr.24401) 727 : cluster [DBG] pgmap v1238: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:19.920089+0000 mgr.y (mgr.24401) 727 : cluster [DBG] pgmap v1238: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.156255+0000 osd.7 (osd.7) 407 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.156255+0000 osd.7 (osd.7) 407 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.159402+0000 osd.7 (osd.7) 408 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.159402+0000 osd.7 (osd.7) 408 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.224693+0000 osd.1 (osd.1) 409 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.224693+0000 osd.1 (osd.1) 409 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:36:21.717 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.227429+0000 osd.1 (osd.1) 410 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.227429+0000 osd.1 (osd.1) 410 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: audit 2024-04-26T16:36:20.466142+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: audit 2024-04-26T16:36:20.466142+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.477969+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.477969+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: audit 2024-04-26T16:36:20.483844+0000 mon.c (mon.2) 828 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: audit 2024-04-26T16:36:20.483844+0000 mon.c (mon.2) 828 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: audit 2024-04-26T16:36:20.485991+0000 mon.a (mon.0) 3724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.718 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:21 smithi050 bash[21463]: audit 2024-04-26T16:36:20.485991+0000 mon.a (mon.0) 3724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- 172.21.15.50:0/3918429150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe538103e60 msgr2=0x7fe5381102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- 172.21.15.50:0/3918429150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe538103e60 0x7fe5381102b0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b0a0 tx=0x7fe53000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- 172.21.15.50:0/3918429150 shutdown_connections 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- 172.21.15.50:0/3918429150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe5381094e0 0x7fe5381098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- 172.21.15.50:0/3918429150 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe538103e60 0x7fe5381102b0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- 172.21.15.50:0/3918429150 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 0x7fe538103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:21.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- 172.21.15.50:0/3918429150 >> 172.21.15.50:0/3918429150 conn(0x7fe53807a590 msgr2=0x7fe53807a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- 172.21.15.50:0/3918429150 shutdown_connections 2024-04-26T16:36:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- 172.21.15.50:0/3918429150 wait complete. 2024-04-26T16:36:21.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:21.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 librados: starting msgr at 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 librados: starting objecter 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 Processor -- start 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- start start 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 librados: setting wanted keys 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 librados: calling monclient init 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 0x7fe53819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe538103e60 0x7fe53819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe5381094e0 0x7fe53819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fe538112ad0 con 0x7fe5381034a0 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fe538112950 con 0x7fe538103e60 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fe538112c50 con 0x7fe5381094e0 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53da80700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 0x7fe53819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:21.730 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53da80700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 0x7fe53819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:42826/0 (socket says 172.21.15.50:42826) 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53da80700 1 -- 172.21.15.50:0/2351329084 learned_addr learned my addr 172.21.15.50:0/2351329084 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe5381094e0 0x7fe53819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53d27f700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe538103e60 0x7fe53819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 -- 172.21.15.50:0/2351329084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe538103e60 msgr2=0x7fe53819b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe538103e60 0x7fe53819b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 -- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 msgr2=0x7fe53819b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 0x7fe53819b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 -- 172.21.15.50:0/2351329084 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5381a0320 con 0x7fe5381094e0 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53e281700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe5381094e0 0x7fe53819fbe0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b0a0 tx=0x7fe530004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe52effd700 10 client.?.objecter ms_handle_connect 0x7fe5381094e0 2024-04-26T16:36:21.731 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe52effd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:21.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe530021070 con 0x7fe5381094e0 2024-04-26T16:36:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.729+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5381a05b0 con 0x7fe5381094e0 2024-04-26T16:36:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe530007b80 con 0x7fe5381094e0 2024-04-26T16:36:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe53001dd50 con 0x7fe5381094e0 2024-04-26T16:36:21.733 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe53fce4700 10 client.41642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5381a7e90 con 0x7fe5381094e0 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe53fce4700 1 librados: init done 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe53806a110 con 0x7fe5381094e0 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe530024070 con 0x7fe5381094e0 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe524071f80 0x7fe524074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(800..800 src has 577..800) ==== 9947+0+0 (secure 0 0 0) 0x7fe530092c00 con 0x7fe5381094e0 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 10 client.41642.objecter ms_dispatch 0x7fe538001550 osd_map(800..800 src has 577..800) 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 3 client.41642.objecter handle_osd_map got epochs [800,800] > 0 2024-04-26T16:36:21.734 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 3 client.41642.objecter handle_osd_map decoding full epoch 800 2024-04-26T16:36:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe53da80700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe524071f80 0x7fe524074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 20 client.41642.objecter dump_active .. 0 homeless 2024-04-26T16:36:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe52effd700 10 client.41642.objecter ms_handle_connect 0x7fe524071f80 2024-04-26T16:36:21.735 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.733+0000 7fe53da80700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe524071f80 0x7fe524074440 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7fe53400bcb0 tx=0x7fe53400b480 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:21.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.737+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fe530099080 con 0x7fe5381094e0 2024-04-26T16:36:21.833 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:21.835 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.920089+0000 mgr.y (mgr.24401) 727 : cluster [DBG] pgmap v1238: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:19.920089+0000 mgr.y (mgr.24401) 727 : cluster [DBG] pgmap v1238: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.156255+0000 osd.7 (osd.7) 407 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.156255+0000 osd.7 (osd.7) 407 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.159402+0000 osd.7 (osd.7) 408 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.159402+0000 osd.7 (osd.7) 408 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.224693+0000 osd.1 (osd.1) 409 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.224693+0000 osd.1 (osd.1) 409 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.227429+0000 osd.1 (osd.1) 410 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.227429+0000 osd.1 (osd.1) 410 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: audit 2024-04-26T16:36:20.466142+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: audit 2024-04-26T16:36:20.466142+0000 mon.a (mon.0) 3722 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.477969+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.477969+0000 mon.a (mon.0) 3723 : cluster [DBG] osdmap e799: 8 total, 8 up, 8 in 2024-04-26T16:36:21.836 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: audit 2024-04-26T16:36:20.483844+0000 mon.c (mon.2) 828 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: audit 2024-04-26T16:36:20.483844+0000 mon.c (mon.2) 828 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: audit 2024-04-26T16:36:20.485991+0000 mon.a (mon.0) 3724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.837 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:21 smithi190 bash[23868]: audit 2024-04-26T16:36:20.485991+0000 mon.a (mon.0) 3724 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:21.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:21.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:21.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:21.977 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:21.977+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7fe5381a8060 con 0x7fe5381094e0 2024-04-26T16:36:21.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:21.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:22.094 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:22.120 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:22.150 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:22.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.248 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:22.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.378 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:22.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.700884+0000 osd.2 (osd.2) 373 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:36:22.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.700884+0000 osd.2 (osd.2) 373 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:36:22.486 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.702773+0000 osd.2 (osd.2) 374 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:20.702773+0000 osd.2 (osd.2) 374 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.377642+0000 osd.5 (osd.5) 293 : cluster [DBG] 3.c scrub starts 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.377642+0000 osd.5 (osd.5) 293 : cluster [DBG] 3.c scrub starts 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.382325+0000 osd.5 (osd.5) 294 : cluster [DBG] 3.c scrub ok 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.382325+0000 osd.5 (osd.5) 294 : cluster [DBG] 3.c scrub ok 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.474333+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.474333+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.478428+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.478428+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.480019+0000 mon.c (mon.2) 829 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.480019+0000 mon.c (mon.2) 829 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.481405+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.481405+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.487 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.582098+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.582098+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.583968+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.583968+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.700884+0000 osd.2 (osd.2) 373 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.700884+0000 osd.2 (osd.2) 373 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.702773+0000 osd.2 (osd.2) 374 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:20.702773+0000 osd.2 (osd.2) 374 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.377642+0000 osd.5 (osd.5) 293 : cluster [DBG] 3.c scrub starts 2024-04-26T16:36:22.488 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.377642+0000 osd.5 (osd.5) 293 : cluster [DBG] 3.c scrub starts 2024-04-26T16:36:22.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.382325+0000 osd.5 (osd.5) 294 : cluster [DBG] 3.c scrub ok 2024-04-26T16:36:22.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.382325+0000 osd.5 (osd.5) 294 : cluster [DBG] 3.c scrub ok 2024-04-26T16:36:22.489 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.474333+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:22.494 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.493+0000 7fe52effd700 1 -- 172.21.15.50:0/2351329084 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v801) ==== 113+0+0 (secure 0 0 0) 0x7fe53005e380 con 0x7fe5381094e0 2024-04-26T16:36:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 10 librados: watch_flush enter 2024-04-26T16:36:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 10 librados: watch_flush exit 2024-04-26T16:36:22.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 20 client.41642.objecter shutdown clearing up homeless session... 2024-04-26T16:36:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 10 client.41642.objecter successfully canceled tick 2024-04-26T16:36:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe524071f80 msgr2=0x7fe524074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe524071f80 0x7fe524074440 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7fe53400bcb0 tx=0x7fe53400b480 comp rx=0 tx=0).stop 2024-04-26T16:36:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe5381094e0 msgr2=0x7fe53819fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:22.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe5381094e0 0x7fe53819fbe0 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b0a0 tx=0x7fe530004d00 comp rx=0 tx=0).stop 2024-04-26T16:36:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 shutdown_connections 2024-04-26T16:36:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe524071f80 0x7fe524074440 unknown :-1 s=CLOSED pgs=3832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fe5381094e0 0x7fe53819fbe0 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fe5381034a0 0x7fe53819b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 --2- 172.21.15.50:0/2351329084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe538103e60 0x7fe53819b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:22.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 >> 172.21.15.50:0/2351329084 conn(0x7fe53807a590 msgr2=0x7fe53810e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 shutdown_connections 2024-04-26T16:36:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 -- 172.21.15.50:0/2351329084 wait complete. 2024-04-26T16:36:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:22.501+0000 7fe53fce4700 1 librados: shutdown 2024-04-26T16:36:22.502 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-26T16:36:22.507 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:22.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.637 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:22.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.867716+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.867716+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.871554+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.871554+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.983262+0000 mon.c (mon.2) 830 : audit [INF] from='client.? 172.21.15.50:0/2351329084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.983262+0000 mon.c (mon.2) 830 : audit [INF] from='client.? 172.21.15.50:0/2351329084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.983733+0000 mon.a (mon.0) 3728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.752 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: audit 2024-04-26T16:36:21.983733+0000 mon.a (mon.0) 3728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.753 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.150104+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:36:22.753 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.150104+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:36:22.753 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.154364+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:36:22.753 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:22 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.154364+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:36:22.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:22.870 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.700884+0000 osd.2 (osd.2) 373 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:36:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.700884+0000 osd.2 (osd.2) 373 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:36:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.702773+0000 osd.2 (osd.2) 374 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:36:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:20.702773+0000 osd.2 (osd.2) 374 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.377642+0000 osd.5 (osd.5) 293 : cluster [DBG] 3.c scrub starts 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.377642+0000 osd.5 (osd.5) 293 : cluster [DBG] 3.c scrub starts 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.382325+0000 osd.5 (osd.5) 294 : cluster [DBG] 3.c scrub ok 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.382325+0000 osd.5 (osd.5) 294 : cluster [DBG] 3.c scrub ok 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.474333+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.474333+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.478428+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.478428+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.480019+0000 mon.c (mon.2) 829 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.480019+0000 mon.c (mon.2) 829 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.481405+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.481405+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.582098+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.582098+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.583968+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.583968+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.867716+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.867716+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.871554+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.474333+0000 mon.a (mon.0) 3725 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.478428+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.478428+0000 mon.a (mon.0) 3726 : cluster [DBG] osdmap e800: 8 total, 8 up, 8 in 2024-04-26T16:36:22.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.480019+0000 mon.c (mon.2) 829 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.480019+0000 mon.c (mon.2) 829 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.481405+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.481405+0000 mon.a (mon.0) 3727 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]: dispatch 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.582098+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.582098+0000 osd.3 (osd.3) 439 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.583968+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.583968+0000 osd.3 (osd.3) 440 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.867716+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.867716+0000 osd.0 (osd.0) 433 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.871554+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.871554+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.983262+0000 mon.c (mon.2) 830 : audit [INF] from='client.? 172.21.15.50:0/2351329084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.983262+0000 mon.c (mon.2) 830 : audit [INF] from='client.? 172.21.15.50:0/2351329084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.983733+0000 mon.a (mon.0) 3728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: audit 2024-04-26T16:36:21.983733+0000 mon.a (mon.0) 3728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.150104+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.150104+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.154364+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.154364+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.871554+0000 osd.0 (osd.0) 434 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.983262+0000 mon.c (mon.2) 830 : audit [INF] from='client.? 172.21.15.50:0/2351329084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.983262+0000 mon.c (mon.2) 830 : audit [INF] from='client.? 172.21.15.50:0/2351329084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.983733+0000 mon.a (mon.0) 3728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: audit 2024-04-26T16:36:21.983733+0000 mon.a (mon.0) 3728 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.150104+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:36:22.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.150104+0000 osd.4 (osd.4) 379 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:36:22.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.154364+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:36:22.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:22 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.154364+0000 osd.4 (osd.4) 380 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:36:22.900 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:22.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:22.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:23.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:23.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:23.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:23.161 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:23.265 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:23.292 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:23.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:23.422 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:23.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:23.551 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:23.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.921621+0000 mgr.y (mgr.24401) 728 : cluster [DBG] pgmap v1241: 204 pgs: 5 unknown, 199 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:21.921621+0000 mgr.y (mgr.24401) 728 : cluster [DBG] pgmap v1241: 204 pgs: 5 unknown, 199 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.132550+0000 osd.7 (osd.7) 409 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.132550+0000 osd.7 (osd.7) 409 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.138786+0000 osd.7 (osd.7) 410 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.138786+0000 osd.7 (osd.7) 410 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.163047+0000 osd.6 (osd.6) 3 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.163047+0000 osd.6 (osd.6) 3 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.165143+0000 osd.6 (osd.6) 4 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.165143+0000 osd.6 (osd.6) 4 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.449082+0000 osd.2 (osd.2) 375 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.449082+0000 osd.2 (osd.2) 375 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.449456+0000 osd.2 (osd.2) 376 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.449456+0000 osd.2 (osd.2) 376 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:23.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.484639+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.484639+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.484807+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.484807+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.497243+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.497243+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.499038+0000 mon.c (mon.2) 831 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.499038+0000 mon.c (mon.2) 831 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.921621+0000 mgr.y (mgr.24401) 728 : cluster [DBG] pgmap v1241: 204 pgs: 5 unknown, 199 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:23.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:21.921621+0000 mgr.y (mgr.24401) 728 : cluster [DBG] pgmap v1241: 204 pgs: 5 unknown, 199 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.132550+0000 osd.7 (osd.7) 409 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.132550+0000 osd.7 (osd.7) 409 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.138786+0000 osd.7 (osd.7) 410 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.138786+0000 osd.7 (osd.7) 410 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.163047+0000 osd.6 (osd.6) 3 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.163047+0000 osd.6 (osd.6) 3 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.165143+0000 osd.6 (osd.6) 4 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.165143+0000 osd.6 (osd.6) 4 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.449082+0000 osd.2 (osd.2) 375 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.449082+0000 osd.2 (osd.2) 375 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.449456+0000 osd.2 (osd.2) 376 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.449456+0000 osd.2 (osd.2) 376 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.484639+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:36:23.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.484639+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.484807+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.484807+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.497243+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.497243+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.499038+0000 mon.c (mon.2) 831 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.499038+0000 mon.c (mon.2) 831 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.500691+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: audit 2024-04-26T16:36:22.500691+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.593656+0000 osd.3 (osd.3) 441 : cluster [DBG] 6.c scrub starts 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.593656+0000 osd.3 (osd.3) 441 : cluster [DBG] 6.c scrub starts 2024-04-26T16:36:23.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.595447+0000 osd.3 (osd.3) 442 : cluster [DBG] 6.c scrub ok 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.595447+0000 osd.3 (osd.3) 442 : cluster [DBG] 6.c scrub ok 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.194741+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.194741+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.195948+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.195948+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.500691+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: audit 2024-04-26T16:36:22.500691+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.593656+0000 osd.3 (osd.3) 441 : cluster [DBG] 6.c scrub starts 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.593656+0000 osd.3 (osd.3) 441 : cluster [DBG] 6.c scrub starts 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.595447+0000 osd.3 (osd.3) 442 : cluster [DBG] 6.c scrub ok 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.595447+0000 osd.3 (osd.3) 442 : cluster [DBG] 6.c scrub ok 2024-04-26T16:36:23.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.194741+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:36:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.194741+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:36:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.195948+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:36:23.636 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:23 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.195948+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:36:23.654 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:23.785 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:23.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.921621+0000 mgr.y (mgr.24401) 728 : cluster [DBG] pgmap v1241: 204 pgs: 5 unknown, 199 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:21.921621+0000 mgr.y (mgr.24401) 728 : cluster [DBG] pgmap v1241: 204 pgs: 5 unknown, 199 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.132550+0000 osd.7 (osd.7) 409 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:36:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.132550+0000 osd.7 (osd.7) 409 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:36:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.138786+0000 osd.7 (osd.7) 410 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:36:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.138786+0000 osd.7 (osd.7) 410 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.163047+0000 osd.6 (osd.6) 3 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.163047+0000 osd.6 (osd.6) 3 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.165143+0000 osd.6 (osd.6) 4 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.165143+0000 osd.6 (osd.6) 4 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.449082+0000 osd.2 (osd.2) 375 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.449082+0000 osd.2 (osd.2) 375 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.449456+0000 osd.2 (osd.2) 376 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.449456+0000 osd.2 (osd.2) 376 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.484639+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.484639+0000 mon.a (mon.0) 3729 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_count","val": "3"}]': finished 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.484807+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.484807+0000 mon.a (mon.0) 3730 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.497243+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-04-26T16:36:23.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.497243+0000 mon.a (mon.0) 3731 : cluster [DBG] osdmap e801: 8 total, 8 up, 8 in 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.499038+0000 mon.c (mon.2) 831 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.499038+0000 mon.c (mon.2) 831 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.500691+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: audit 2024-04-26T16:36:22.500691+0000 mon.a (mon.0) 3732 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]: dispatch 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.593656+0000 osd.3 (osd.3) 441 : cluster [DBG] 6.c scrub starts 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.593656+0000 osd.3 (osd.3) 441 : cluster [DBG] 6.c scrub starts 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.595447+0000 osd.3 (osd.3) 442 : cluster [DBG] 6.c scrub ok 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.595447+0000 osd.3 (osd.3) 442 : cluster [DBG] 6.c scrub ok 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.194741+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.194741+0000 osd.4 (osd.4) 381 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.195948+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:36:23.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:23 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.195948+0000 osd.4 (osd.4) 382 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:36:23.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:23.941 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:23.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:24.044 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:24.069 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:24.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:24.299 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:24.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:24.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:24.449 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:24.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:24.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:24.575 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:24.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:36:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.747648+0000 osd.2 (osd.2) 377 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.747648+0000 osd.2 (osd.2) 377 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.750074+0000 osd.2 (osd.2) 378 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:22.750074+0000 osd.2 (osd.2) 378 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.140018+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.140018+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.142431+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:36:24.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.142431+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.487427+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.487427+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.495778+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.495778+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.497074+0000 mon.c (mon.2) 832 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.497074+0000 mon.c (mon.2) 832 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.498642+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.498642+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.914603+0000 mon.c (mon.2) 833 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:23.914603+0000 mon.c (mon.2) 833 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:24.489451+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:24.489451+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.493338+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.493338+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-04-26T16:36:24.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:24.496026+0000 mon.c (mon.2) 834 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:24.496026+0000 mon.c (mon.2) 834 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:24.496570+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[29234]: audit 2024-04-26T16:36:24.496570+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.747648+0000 osd.2 (osd.2) 377 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.747648+0000 osd.2 (osd.2) 377 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.750074+0000 osd.2 (osd.2) 378 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:22.750074+0000 osd.2 (osd.2) 378 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:36:24.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.140018+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.140018+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.142431+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.142431+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.487427+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.487427+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.495778+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.495778+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.497074+0000 mon.c (mon.2) 832 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.497074+0000 mon.c (mon.2) 832 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.498642+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.498642+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.914603+0000 mon.c (mon.2) 833 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:23.914603+0000 mon.c (mon.2) 833 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:24.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:24.489451+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:24.489451+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.493338+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.493338+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:24.496026+0000 mon.c (mon.2) 834 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:24.496026+0000 mon.c (mon.2) 834 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:24.496570+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:24 smithi050 bash[21463]: audit 2024-04-26T16:36:24.496570+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.644 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-26T16:36:24.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.747648+0000 osd.2 (osd.2) 377 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.747648+0000 osd.2 (osd.2) 377 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.750074+0000 osd.2 (osd.2) 378 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:22.750074+0000 osd.2 (osd.2) 378 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.140018+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.140018+0000 osd.7 (osd.7) 411 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.142431+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.142431+0000 osd.7 (osd.7) 412 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:36:24.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.487427+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.487427+0000 mon.a (mon.0) 3733 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_period","val": "3"}]': finished 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.495778+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.495778+0000 mon.a (mon.0) 3734 : cluster [DBG] osdmap e802: 8 total, 8 up, 8 in 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.497074+0000 mon.c (mon.2) 832 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.497074+0000 mon.c (mon.2) 832 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.498642+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.498642+0000 mon.a (mon.0) 3735 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.914603+0000 mon.c (mon.2) 833 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:23.914603+0000 mon.c (mon.2) 833 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:24.489451+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:24.489451+0000 mon.a (mon.0) 3736 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.493338+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.493338+0000 mon.a (mon.0) 3737 : cluster [DBG] osdmap e803: 8 total, 8 up, 8 in 2024-04-26T16:36:24.691 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:24.496026+0000 mon.c (mon.2) 834 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:24.496026+0000 mon.c (mon.2) 834 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:24.496570+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.692 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:24 smithi190 bash[23868]: audit 2024-04-26T16:36:24.496570+0000 mon.a (mon.0) 3738 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-04-26T16:36:24.703 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:24.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:24.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:24.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:24.951 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:24.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:25.054 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:25.075 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:25.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:25.126 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:25.127 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:25.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:25.206 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:25.309 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:25.337 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:25.440 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:25.470 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:25.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.922689+0000 mgr.y (mgr.24401) 729 : cluster [DBG] pgmap v1244: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:23.922689+0000 mgr.y (mgr.24401) 729 : cluster [DBG] pgmap v1244: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:25.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.177814+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.177814+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.178046+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.178046+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.179961+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.179961+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.183002+0000 osd.1 (osd.1) 412 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.183002+0000 osd.1 (osd.1) 412 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.614536+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:36:25.594 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.614536+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.616725+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:24.616725+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:24.652081+0000 mon.c (mon.2) 835 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:24.652081+0000 mon.c (mon.2) 835 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:24.653301+0000 mon.c (mon.2) 836 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:24.653301+0000 mon.c (mon.2) 836 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:24.661252+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:24.661252+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:25.595 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.170425+0000 osd.4 (osd.4) 383 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.170425+0000 osd.4 (osd.4) 383 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.174913+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.174913+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:25.491349+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: audit 2024-04-26T16:36:25.491349+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.495439+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-04-26T16:36:25.596 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.495439+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.922689+0000 mgr.y (mgr.24401) 729 : cluster [DBG] pgmap v1244: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:23.922689+0000 mgr.y (mgr.24401) 729 : cluster [DBG] pgmap v1244: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.177814+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.177814+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.178046+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.178046+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.179961+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:36:25.597 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.179961+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.183002+0000 osd.1 (osd.1) 412 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.183002+0000 osd.1 (osd.1) 412 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.614536+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.614536+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.616725+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:24.616725+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:24.652081+0000 mon.c (mon.2) 835 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:25.598 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:24.652081+0000 mon.c (mon.2) 835 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:24.653301+0000 mon.c (mon.2) 836 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:24.653301+0000 mon.c (mon.2) 836 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:24.661252+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:24.661252+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.170425+0000 osd.4 (osd.4) 383 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.170425+0000 osd.4 (osd.4) 383 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.174913+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:36:25.599 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.174913+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:36:25.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:25.491349+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:36:25.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: audit 2024-04-26T16:36:25.491349+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:36:25.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.495439+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-04-26T16:36:25.600 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:25 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.495439+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-04-26T16:36:25.600 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:25.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:25.729 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:25.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.922689+0000 mgr.y (mgr.24401) 729 : cluster [DBG] pgmap v1244: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:23.922689+0000 mgr.y (mgr.24401) 729 : cluster [DBG] pgmap v1244: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.177814+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.177814+0000 osd.6 (osd.6) 5 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.178046+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.178046+0000 osd.1 (osd.1) 411 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:36:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.179961+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.179961+0000 osd.6 (osd.6) 6 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.183002+0000 osd.1 (osd.1) 412 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.183002+0000 osd.1 (osd.1) 412 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.614536+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.614536+0000 osd.3 (osd.3) 443 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.616725+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:24.616725+0000 osd.3 (osd.3) 444 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:24.652081+0000 mon.c (mon.2) 835 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:24.652081+0000 mon.c (mon.2) 835 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:24.653301+0000 mon.c (mon.2) 836 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:24.653301+0000 mon.c (mon.2) 836 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:36:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:24.661252+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:24.661252+0000 mon.a (mon.0) 3739 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.170425+0000 osd.4 (osd.4) 383 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.170425+0000 osd.4 (osd.4) 383 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.174913+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.174913+0000 osd.4 (osd.4) 384 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:25.491349+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: audit 2024-04-26T16:36:25.491349+0000 mon.a (mon.0) 3740 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.495439+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-04-26T16:36:25.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:25 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.495439+0000 mon.a (mon.0) 3741 : cluster [DBG] osdmap e804: 8 total, 8 up, 8 in 2024-04-26T16:36:25.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:25.861 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:25.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:25.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:25.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:26.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:26.119 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:26.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:26.249 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:26.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:26.382 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:26.485 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:26.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:26.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:26.644 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:26.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:26.774 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:26.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.183611+0000 osd.7 (osd.7) 413 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:36:26.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.183611+0000 osd.7 (osd.7) 413 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:36:26.775 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.187941+0000 osd.7 (osd.7) 414 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.187941+0000 osd.7 (osd.7) 414 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.160127+0000 osd.4 (osd.4) 385 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.160127+0000 osd.4 (osd.4) 385 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.161384+0000 osd.4 (osd.4) 386 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.161384+0000 osd.4 (osd.4) 386 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.371819+0000 mon.a (mon.0) 3742 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.371819+0000 mon.a (mon.0) 3742 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.383433+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-04-26T16:36:26.776 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:26 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.383433+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-04-26T16:36:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:26.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.183611+0000 osd.7 (osd.7) 413 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:36:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.183611+0000 osd.7 (osd.7) 413 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:36:26.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.187941+0000 osd.7 (osd.7) 414 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.187941+0000 osd.7 (osd.7) 414 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.160127+0000 osd.4 (osd.4) 385 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.160127+0000 osd.4 (osd.4) 385 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.161384+0000 osd.4 (osd.4) 386 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.161384+0000 osd.4 (osd.4) 386 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.371819+0000 mon.a (mon.0) 3742 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.371819+0000 mon.a (mon.0) 3742 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.383433+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.383433+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.183611+0000 osd.7 (osd.7) 413 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:36:26.886 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.183611+0000 osd.7 (osd.7) 413 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.187941+0000 osd.7 (osd.7) 414 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.187941+0000 osd.7 (osd.7) 414 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.160127+0000 osd.4 (osd.4) 385 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.160127+0000 osd.4 (osd.4) 385 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.161384+0000 osd.4 (osd.4) 386 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.161384+0000 osd.4 (osd.4) 386 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.371819+0000 mon.a (mon.0) 3742 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.371819+0000 mon.a (mon.0) 3742 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.383433+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-04-26T16:36:26.887 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:26 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.383433+0000 mon.a (mon.0) 3743 : cluster [DBG] osdmap e805: 8 total, 8 up, 8 in 2024-04-26T16:36:26.906 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:26.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:27.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:27.032 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:27.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:27.162 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:27.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:27.293 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:27.396 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:27.426 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:27.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:27.551 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:27.611 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:36:27.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.733267+0000 osd.2 (osd.2) 379 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.733267+0000 osd.2 (osd.2) 379 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.736514+0000 osd.2 (osd.2) 380 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.736514+0000 osd.2 (osd.2) 380 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.923572+0000 mgr.y (mgr.24401) 730 : cluster [DBG] pgmap v1247: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:25.923572+0000 mgr.y (mgr.24401) 730 : cluster [DBG] pgmap v1247: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.601827+0000 osd.3 (osd.3) 445 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.601827+0000 osd.3 (osd.3) 445 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.604348+0000 osd.3 (osd.3) 446 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:36:27.612 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:27 smithi190 bash[23868]: cluster 2024-04-26T16:36:26.604348+0000 osd.3 (osd.3) 446 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:36:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:27.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:27.741 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:27.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:27.870 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.733267+0000 osd.2 (osd.2) 379 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.733267+0000 osd.2 (osd.2) 379 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.736514+0000 osd.2 (osd.2) 380 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.736514+0000 osd.2 (osd.2) 380 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.923572+0000 mgr.y (mgr.24401) 730 : cluster [DBG] pgmap v1247: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:25.923572+0000 mgr.y (mgr.24401) 730 : cluster [DBG] pgmap v1247: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.601827+0000 osd.3 (osd.3) 445 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.601827+0000 osd.3 (osd.3) 445 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:36:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.604348+0000 osd.3 (osd.3) 446 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[21463]: cluster 2024-04-26T16:36:26.604348+0000 osd.3 (osd.3) 446 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.733267+0000 osd.2 (osd.2) 379 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.733267+0000 osd.2 (osd.2) 379 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.736514+0000 osd.2 (osd.2) 380 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.736514+0000 osd.2 (osd.2) 380 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.923572+0000 mgr.y (mgr.24401) 730 : cluster [DBG] pgmap v1247: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:25.923572+0000 mgr.y (mgr.24401) 730 : cluster [DBG] pgmap v1247: 204 pgs: 204 active+clean; 4.4 MiB data, 758 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.601827+0000 osd.3 (osd.3) 445 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.601827+0000 osd.3 (osd.3) 445 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.604348+0000 osd.3 (osd.3) 446 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:36:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:27 smithi050 bash[29234]: cluster 2024-04-26T16:36:26.604348+0000 osd.3 (osd.3) 446 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:36:27.973 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:27.996 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:27.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:28.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:28.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:28.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:28.259 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:28.362 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:28.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:28.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:28.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: cluster 2024-04-26T16:36:27.159105+0000 osd.6 (osd.6) 7 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: cluster 2024-04-26T16:36:27.159105+0000 osd.6 (osd.6) 7 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: cluster 2024-04-26T16:36:27.163207+0000 osd.6 (osd.6) 8 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: cluster 2024-04-26T16:36:27.163207+0000 osd.6 (osd.6) 8 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: audit 2024-04-26T16:36:27.278009+0000 mgr.y (mgr.24401) 731 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: audit 2024-04-26T16:36:27.278009+0000 mgr.y (mgr.24401) 731 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:28.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: audit 2024-04-26T16:36:27.811870+0000 mon.c (mon.2) 837 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[21463]: audit 2024-04-26T16:36:27.811870+0000 mon.c (mon.2) 837 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: cluster 2024-04-26T16:36:27.159105+0000 osd.6 (osd.6) 7 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: cluster 2024-04-26T16:36:27.159105+0000 osd.6 (osd.6) 7 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: cluster 2024-04-26T16:36:27.163207+0000 osd.6 (osd.6) 8 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: cluster 2024-04-26T16:36:27.163207+0000 osd.6 (osd.6) 8 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: audit 2024-04-26T16:36:27.278009+0000 mgr.y (mgr.24401) 731 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: audit 2024-04-26T16:36:27.278009+0000 mgr.y (mgr.24401) 731 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: audit 2024-04-26T16:36:27.811870+0000 mon.c (mon.2) 837 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:28.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:28 smithi050 bash[29234]: audit 2024-04-26T16:36:27.811870+0000 mon.c (mon.2) 837 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:28.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:28.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:28.761 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:28.789 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: cluster 2024-04-26T16:36:27.159105+0000 osd.6 (osd.6) 7 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: cluster 2024-04-26T16:36:27.159105+0000 osd.6 (osd.6) 7 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: cluster 2024-04-26T16:36:27.163207+0000 osd.6 (osd.6) 8 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: cluster 2024-04-26T16:36:27.163207+0000 osd.6 (osd.6) 8 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: audit 2024-04-26T16:36:27.278009+0000 mgr.y (mgr.24401) 731 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: audit 2024-04-26T16:36:27.278009+0000 mgr.y (mgr.24401) 731 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: audit 2024-04-26T16:36:27.811870+0000 mon.c (mon.2) 837 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:28 smithi190 bash[23868]: audit 2024-04-26T16:36:27.811870+0000 mon.c (mon.2) 837 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:28.892 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:28.920 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:28.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:29.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:29.050 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:29.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:29.180 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:29.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:29.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:29.309 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:29.412 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:29.440 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:29.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:29.554 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:27.925741+0000 mgr.y (mgr.24401) 732 : cluster [DBG] pgmap v1249: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:27.925741+0000 mgr.y (mgr.24401) 732 : cluster [DBG] pgmap v1249: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.203963+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.203963+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.206622+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.206622+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.924110+0000 osd.0 (osd.0) 435 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.924110+0000 osd.0 (osd.0) 435 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.926145+0000 osd.0 (osd.0) 436 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:28.926145+0000 osd.0 (osd.0) 436 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.461944+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.a scrub starts 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.461944+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.a scrub starts 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.464227+0000 osd.5 (osd.5) 296 : cluster [DBG] 6.a scrub ok 2024-04-26T16:36:29.555 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:29 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.464227+0000 osd.5 (osd.5) 296 : cluster [DBG] 6.a scrub ok 2024-04-26T16:36:29.570 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:29.673 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:29.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:27.925741+0000 mgr.y (mgr.24401) 732 : cluster [DBG] pgmap v1249: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-04-26T16:36:29.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:27.925741+0000 mgr.y (mgr.24401) 732 : cluster [DBG] pgmap v1249: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-04-26T16:36:29.685 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.203963+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.203963+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.206622+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.206622+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.924110+0000 osd.0 (osd.0) 435 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.924110+0000 osd.0 (osd.0) 435 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.926145+0000 osd.0 (osd.0) 436 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:28.926145+0000 osd.0 (osd.0) 436 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.461944+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.a scrub starts 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.461944+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.a scrub starts 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.464227+0000 osd.5 (osd.5) 296 : cluster [DBG] 6.a scrub ok 2024-04-26T16:36:29.686 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.464227+0000 osd.5 (osd.5) 296 : cluster [DBG] 6.a scrub ok 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:27.925741+0000 mgr.y (mgr.24401) 732 : cluster [DBG] pgmap v1249: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:27.925741+0000 mgr.y (mgr.24401) 732 : cluster [DBG] pgmap v1249: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.6 KiB/s wr, 1 op/s 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.203963+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.203963+0000 osd.6 (osd.6) 9 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.206622+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.206622+0000 osd.6 (osd.6) 10 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.924110+0000 osd.0 (osd.0) 435 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.924110+0000 osd.0 (osd.0) 435 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.926145+0000 osd.0 (osd.0) 436 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:28.926145+0000 osd.0 (osd.0) 436 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.461944+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.a scrub starts 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.461944+0000 osd.5 (osd.5) 295 : cluster [DBG] 6.a scrub starts 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.464227+0000 osd.5 (osd.5) 296 : cluster [DBG] 6.a scrub ok 2024-04-26T16:36:29.687 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:29 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.464227+0000 osd.5 (osd.5) 296 : cluster [DBG] 6.a scrub ok 2024-04-26T16:36:29.703 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:29.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:29.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:29.965 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:29.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:30.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:30.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/668807052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806c360 msgr2=0x7f04081102d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- 172.21.15.50:0/668807052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806c360 0x7f04081102d0 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009990 tx=0x7f03f8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/668807052 shutdown_connections 2024-04-26T16:36:30.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- 172.21.15.50:0/668807052 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806c360 0x7f04081102d0 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- 172.21.15.50:0/668807052 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0408109490 0x7f0408109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- 172.21.15.50:0/668807052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f040806b9c0 0x7f040806be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/668807052 >> 172.21.15.50:0/668807052 conn(0x7f0408079290 msgr2=0x7f04080796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/668807052 shutdown_connections 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/668807052 wait complete. 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 librados: starting msgr at 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 librados: starting objecter 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 Processor -- start 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- start start 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 librados: setting wanted keys 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 librados: calling monclient init 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806b9c0 0x7f0408199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 0x7f0408199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0408109490 0x7f040819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0408112ac0 con 0x7f040806c360 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0408112940 con 0x7f0408109490 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0408112c40 con 0x7f040806b9c0 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 0x7f0408199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040659c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806b9c0 0x7f0408199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 0x7f0408199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49340/0 (socket says 172.21.15.50:49340) 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040659c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806b9c0 0x7f0408199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56108/0 (socket says 172.21.15.50:56108) 2024-04-26T16:36:30.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 -- 172.21.15.50:0/1457476779 learned_addr learned my addr 172.21.15.50:0/1457476779 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0408109490 0x7f040819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806b9c0 msgr2=0x7f0408199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806b9c0 0x7f0408199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0408109490 msgr2=0x7f040819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0408109490 0x7f040819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f040819e030 con 0x7f040806c360 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f040806b9c0 0x7f0408199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0408109490 0x7f040819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 0x7f0408199560 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f03fc007c70 tx=0x7f03fc007f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f03f77fe700 10 client.?.objecter ms_handle_connect 0x7f040806c360 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f03f77fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc00e2e0 con 0x7f040806c360 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f040819e320 con 0x7f040806c360 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f03fc00e480 con 0x7f040806c360 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03fc00cb60 con 0x7f040806c360 2024-04-26T16:36:30.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 10 client.31029.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:30.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0408102310 con 0x7f040806c360 2024-04-26T16:36:30.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 librados: init done 2024-04-26T16:36:30.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.173+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f03e8005100 con 0x7f040806c360 2024-04-26T16:36:30.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f03fc00cd00 con 0x7f040806c360 2024-04-26T16:36:30.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03f0071f80 0x7f03f0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(805..805 src has 577..805) ==== 9973+0+0 (secure 0 0 0) 0x7f03fc095180 con 0x7f040806c360 2024-04-26T16:36:30.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 osd_map(805..805 src has 577..805) 2024-04-26T16:36:30.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 3 client.31029.objecter handle_osd_map got epochs [805,805] > 0 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 3 client.31029.objecter handle_osd_map decoding full epoch 805 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 20 client.31029.objecter dump_active .. 0 homeless 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v805) ==== 54+0+16 (secure 0 0 0) 0x7f03fc025060 con 0x7f040806c360 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 10 client.31029.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 1 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f03e8004730 0x7f03e8004b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 1 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 0 1 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 4 2024-04-26T16:36:30.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 10 client.31029.objecter _send_command 1 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03e8008b70 con 0x7f03e8004730 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f03e8004730 0x7f03e8004b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03f0071f80 0x7f03f0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f03e8004730 0x7f03e8004b70 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e8004730 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03f0071f80 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03f0071f80 0x7f03f0074440 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7f0400009480 tx=0x7f040000b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e8008b70 con 0x7f03e8004730 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 1: 0 ) 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 10 client.31029.objecter _finish_command 1 = system:0 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 4 2024-04-26T16:36:30.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.177+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 0 1 2024-04-26T16:36:30.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:30.195 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 10 client.31029.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 2 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 2 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e80053e0 osd=0 3 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 0 2 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 4 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 10 client.31029.objecter _send_command 2 2024-04-26T16:36:30.196 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e80042a0 con 0x7f03e8004730 2024-04-26T16:36:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e80042a0 con 0x7f03e8004730 2024-04-26T16:36:30.197 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f03f77fe700 10 client.31029.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 4 2024-04-26T16:36:30.198 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.197+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 0 2 2024-04-26T16:36:30.198 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 10 client.31029.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 3 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f03e8002750 0x7f03e8002bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 3 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.199 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 1 3 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 4 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 10 client.31029.objecter _send_command 3 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f03e8003110 con 0x7f03e8002750 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f03e8002750 0x7f03e8002bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e8002750 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f03e8002750 0x7f03e8002bd0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e8003110 con 0x7f03e8002750 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 3: 0 ) 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f03f77fe700 10 client.31029.objecter _finish_command 3 = system:0 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 4 2024-04-26T16:36:30.200 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.201+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 1 3 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 10 client.31029.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 4 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 4 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8004530 osd=1 3 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 1 4 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 4 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 10 client.31029.objecter _send_command 4 2024-04-26T16:36:30.217 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.217+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e8003b60 con 0x7f03e8002750 2024-04-26T16:36:30.218 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e8003b60 con 0x7f03e8002750 2024-04-26T16:36:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 10 client.31029.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 4 2024-04-26T16:36:30.219 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 1 4 2024-04-26T16:36:30.219 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 10 client.31029.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 5 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f03e800b240 0x7f03e800d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 5 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.220 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 2 5 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 4 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 10 client.31029.objecter _send_command 5 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f03e8010570 con 0x7f03e800b240 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f03e800b240 0x7f03e800d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f03e800b240 0x7f03e800d630 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e800b240 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e8010570 con 0x7f03e800b240 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 5: 0 ) 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 10 client.31029.objecter _finish_command 5 = system:0 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 4 2024-04-26T16:36:30.221 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.221+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 2 5 2024-04-26T16:36:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 10 client.31029.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 6 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 6 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e80080e0 osd=2 3 2024-04-26T16:36:30.238 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 2 6 2024-04-26T16:36:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 4 2024-04-26T16:36:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 10 client.31029.objecter _send_command 6 2024-04-26T16:36:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e8003600 con 0x7f03e800b240 2024-04-26T16:36:30.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e8003600 con 0x7f03e800b240 2024-04-26T16:36:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 10 client.31029.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 4 2024-04-26T16:36:30.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 2 6 2024-04-26T16:36:30.240 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 10 client.31029.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 7 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f03e800ecf0 0x7f03e800f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 7 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 3 7 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 4 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 10 client.31029.objecter _send_command 7 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f03e800f6b0 con 0x7f03e800ecf0 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f03e800ecf0 0x7f03e800f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e800ecf0 2024-04-26T16:36:30.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f03e800ecf0 0x7f03e800f170 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e800f6b0 con 0x7f03e800ecf0 2024-04-26T16:36:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 7: 0 ) 2024-04-26T16:36:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 10 client.31029.objecter _finish_command 7 = system:0 2024-04-26T16:36:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 4 2024-04-26T16:36:30.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.241+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 3 7 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 10 client.31029.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 8 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 8 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8001b80 osd=3 3 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 3 8 2024-04-26T16:36:30.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 4 2024-04-26T16:36:30.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 10 client.31029.objecter _send_command 8 2024-04-26T16:36:30.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e800f710 con 0x7f03e800ecf0 2024-04-26T16:36:30.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e800f710 con 0x7f03e800ecf0 2024-04-26T16:36:30.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f03f77fe700 10 client.31029.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 4 2024-04-26T16:36:30.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 3 8 2024-04-26T16:36:30.261 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 10 client.31029.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 9 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f03e8001770 0x7f03e8014740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 9 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 4 9 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 4 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 10 client.31029.objecter _send_command 9 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f03e80177e0 con 0x7f03e8001770 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.261+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f03e8001770 0x7f03e8014740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e8001770 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f03e8001770 0x7f03e8014740 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e80177e0 con 0x7f03e8001770 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 9: 0 ) 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f03f77fe700 10 client.31029.objecter _finish_command 9 = system:0 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 4 2024-04-26T16:36:30.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.265+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 4 9 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 10 client.31029.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 10 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 10 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8001600 osd=4 3 2024-04-26T16:36:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 4 10 2024-04-26T16:36:30.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 4 2024-04-26T16:36:30.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 10 client.31029.objecter _send_command 10 2024-04-26T16:36:30.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.281+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e8016570 con 0x7f03e8001770 2024-04-26T16:36:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e8016570 con 0x7f03e8001770 2024-04-26T16:36:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f03f77fe700 10 client.31029.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 4 2024-04-26T16:36:30.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 4 10 2024-04-26T16:36:30.284 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 10 client.31029.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 11 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f03e8016c90 0x7f03e80170f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 11 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 5 11 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 4 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 10 client.31029.objecter _send_command 11 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f03e801ac00 con 0x7f03e8016c90 2024-04-26T16:36:30.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f03e8016c90 0x7f03e80170f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e8016c90 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.285+0000 7f0405d9b700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f03e8016c90 0x7f03e80170f0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.289+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e801ac00 con 0x7f03e8016c90 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.289+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 11: 0 ) 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.289+0000 7f03f77fe700 10 client.31029.objecter _finish_command 11 = system:0 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.289+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 4 2024-04-26T16:36:30.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.289+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 5 11 2024-04-26T16:36:30.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 10 client.31029.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 12 2024-04-26T16:36:30.303 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 12 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8016570 osd=5 3 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 5 12 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 4 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 10 client.31029.objecter _send_command 12 2024-04-26T16:36:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.305+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e801ac00 con 0x7f03e8016c90 2024-04-26T16:36:30.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e801ac00 con 0x7f03e8016c90 2024-04-26T16:36:30.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f03f77fe700 10 client.31029.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 4 2024-04-26T16:36:30.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 5 12 2024-04-26T16:36:30.309 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:36:30.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 10 client.31029.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 13 2024-04-26T16:36:30.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f03e8019fa0 0x7f03e801a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 13 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 6 13 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 4 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 10 client.31029.objecter _send_command 13 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f03e801dfa0 con 0x7f03e8019fa0 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f03e8019fa0 0x7f03e801a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e8019fa0 2024-04-26T16:36:30.310 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.309+0000 7f0406d9d700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f03e8019fa0 0x7f03e801a400 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.313+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e801dfa0 con 0x7f03e8019fa0 2024-04-26T16:36:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.313+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 13: 0 ) 2024-04-26T16:36:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.313+0000 7f03f77fe700 10 client.31029.objecter _finish_command 13 = system:0 2024-04-26T16:36:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.313+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 4 2024-04-26T16:36:30.311 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.313+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 6 13 2024-04-26T16:36:30.323 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:30.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 10 client.31029.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 14 2024-04-26T16:36:30.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.329 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 14 2024-04-26T16:36:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e801ac00 osd=6 3 2024-04-26T16:36:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 6 14 2024-04-26T16:36:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 4 2024-04-26T16:36:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 10 client.31029.objecter _send_command 14 2024-04-26T16:36:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.329+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e8014da0 con 0x7f03e8019fa0 2024-04-26T16:36:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.333+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e8014da0 con 0x7f03e8019fa0 2024-04-26T16:36:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.333+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.333+0000 7f03f77fe700 10 client.31029.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.333+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 4 2024-04-26T16:36:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.333+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 6 14 2024-04-26T16:36:30.334 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 10 client.31029.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 15 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f03e80154c0 0x7f03e8015900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 15 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 7 15 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 4 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 10 client.31029.objecter _send_command 15 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f03e8015fe0 con 0x7f03e80154c0 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f03e80154c0 0x7f03e8015900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f03f77fe700 10 client.31029.objecter ms_handle_connect 0x7f03e80154c0 2024-04-26T16:36:30.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f040659c700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f03e80154c0 0x7f03e8015900 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e8015fe0 con 0x7f03e80154c0 2024-04-26T16:36:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 15: 0 ) 2024-04-26T16:36:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f03f77fe700 10 client.31029.objecter _finish_command 15 = system:0 2024-04-26T16:36:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 4 2024-04-26T16:36:30.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.337+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 7 15 2024-04-26T16:36:30.344 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 10 client.31029.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign -1 16 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 20 client.31029.objecter _get_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 15 client.31029.objecter _session_command_op_remove -1 16 2024-04-26T16:36:30.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 20 client.31029.objecter get_session s=0x7f03e8014da0 osd=7 3 2024-04-26T16:36:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 15 client.31029.objecter _session_command_op_assign 7 16 2024-04-26T16:36:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 4 2024-04-26T16:36:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 10 client.31029.objecter _send_command 16 2024-04-26T16:36:30.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.361+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f03e8015fe0 con 0x7f03e80154c0 2024-04-26T16:36:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f03f77fe700 1 -- 172.21.15.50:0/1457476779 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f03e8015fe0 con 0x7f03e80154c0 2024-04-26T16:36:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f03f77fe700 10 client.31029.objecter ms_dispatch 0x7f0408001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f03f77fe700 10 client.31029.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:30.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f03f77fe700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 4 2024-04-26T16:36:30.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f03f77fe700 15 client.31029.objecter _session_command_op_remove 7 16 2024-04-26T16:36:30.363 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 librados: watch_flush enter 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 librados: watch_flush exit 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.0 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f03e8004730 msgr2=0x7f03e8004b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f03e8004730 0x7f03e8004b70 crc :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80053e0 osd=0 1 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.1 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f03e8002750 msgr2=0x7f03e8002bd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.364 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f03e8002750 0x7f03e8002bd0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8004530 osd=1 1 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.2 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f03e800b240 msgr2=0x7f03e800d630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f03e800b240 0x7f03e800d630 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e80080e0 osd=2 1 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.3 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f03e800ecf0 msgr2=0x7f03e800f170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f03e800ecf0 0x7f03e800f170 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001b80 osd=3 1 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.4 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f03e8001770 msgr2=0x7f03e8014740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f0406d9d700 1 -- 172.21.15.50:0/1457476779 reap_dead start 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f03e8001770 0x7f03e8014740 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f0406d9d700 1 -- 172.21.15.50:0/1457476779 reap_dead start 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8001600 osd=4 1 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.5 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f03e8016c90 msgr2=0x7f03e80170f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f03e8016c90 0x7f03e80170f0 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8016570 osd=5 1 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.6 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f03e8019fa0 msgr2=0x7f03e801a400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f03e8019fa0 0x7f03e801a400 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e801ac00 osd=6 1 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter close_session for osd.7 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f03e80154c0 msgr2=0x7f03e8015900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f03e80154c0 0x7f03e8015900 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter put_session s=0x7f03e8014da0 osd=7 1 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 20 client.31029.objecter shutdown clearing up homeless session... 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 10 client.31029.objecter successfully canceled tick 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03f0071f80 msgr2=0x7f03f0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f03f0071f80 0x7f03f0074440 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7f0400009480 tx=0x7f040000b040 comp rx=0 tx=0).stop 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 msgr2=0x7f0408199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 0x7f0408199560 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f03fc007c70 tx=0x7f03fc007f80 comp rx=0 tx=0).stop 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f0406d9d700 1 -- 172.21.15.50:0/1457476779 reap_dead start 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 shutdown_connections 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 --2- 172.21.15.50:0/1457476779 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f040806c360 0x7f0408199560 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 >> 172.21.15.50:0/1457476779 conn(0x7f0408079290 msgr2=0x7f04081001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 shutdown_connections 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 -- 172.21.15.50:0/1457476779 wait complete. 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:30.365+0000 7f040c92a700 1 librados: shutdown 2024-04-26T16:36:30.367 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:36:30.369 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:36:30.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:30.476 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:30.486 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:30.579 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:30.590 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.231356+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.231356+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.234693+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.234693+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.236244+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.236244+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.238546+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.238546+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.083666+0000 osd.4 (osd.4) 387 : cluster [DBG] 5.e scrub starts 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.083666+0000 osd.4 (osd.4) 387 : cluster [DBG] 5.e scrub starts 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.086622+0000 osd.4 (osd.4) 388 : cluster [DBG] 5.e scrub ok 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.086622+0000 osd.4 (osd.4) 388 : cluster [DBG] 5.e scrub ok 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: audit 2024-04-26T16:36:30.180062+0000 mon.a (mon.0) 3744 : audit [DBG] from='client.? 172.21.15.50:0/1457476779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:30.591 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[21463]: audit 2024-04-26T16:36:30.180062+0000 mon.a (mon.0) 3744 : audit [DBG] from='client.? 172.21.15.50:0/1457476779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.231356+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.231356+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.234693+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.234693+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.236244+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.236244+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.238546+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.238546+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.083666+0000 osd.4 (osd.4) 387 : cluster [DBG] 5.e scrub starts 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.083666+0000 osd.4 (osd.4) 387 : cluster [DBG] 5.e scrub starts 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.086622+0000 osd.4 (osd.4) 388 : cluster [DBG] 5.e scrub ok 2024-04-26T16:36:30.592 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.086622+0000 osd.4 (osd.4) 388 : cluster [DBG] 5.e scrub ok 2024-04-26T16:36:30.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: audit 2024-04-26T16:36:30.180062+0000 mon.a (mon.0) 3744 : audit [DBG] from='client.? 172.21.15.50:0/1457476779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:30.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:30 smithi050 bash[29234]: audit 2024-04-26T16:36:30.180062+0000 mon.a (mon.0) 3744 : audit [DBG] from='client.? 172.21.15.50:0/1457476779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:30.602 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:30.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.231356+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.231356+0000 osd.6 (osd.6) 11 : cluster [DBG] 3.a scrub starts 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.234693+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.234693+0000 osd.6 (osd.6) 12 : cluster [DBG] 3.a scrub ok 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.236244+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.236244+0000 osd.1 (osd.1) 413 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.238546+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.238546+0000 osd.1 (osd.1) 414 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.083666+0000 osd.4 (osd.4) 387 : cluster [DBG] 5.e scrub starts 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.083666+0000 osd.4 (osd.4) 387 : cluster [DBG] 5.e scrub starts 2024-04-26T16:36:30.717 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.086622+0000 osd.4 (osd.4) 388 : cluster [DBG] 5.e scrub ok 2024-04-26T16:36:30.718 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.086622+0000 osd.4 (osd.4) 388 : cluster [DBG] 5.e scrub ok 2024-04-26T16:36:30.718 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: audit 2024-04-26T16:36:30.180062+0000 mon.a (mon.0) 3744 : audit [DBG] from='client.? 172.21.15.50:0/1457476779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:30.718 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:30 smithi190 bash[23868]: audit 2024-04-26T16:36:30.180062+0000 mon.a (mon.0) 3744 : audit [DBG] from='client.? 172.21.15.50:0/1457476779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:30.734 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:30.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:30.863 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:30.966 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:30.994 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:30.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:31.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:31.125 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:31.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:31.254 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:31.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:31.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:31.385 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:31.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:31.516 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:31.620 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:31.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:31.752 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:31.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.926381+0000 mgr.y (mgr.24401) 733 : cluster [DBG] pgmap v1250: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-04-26T16:36:31.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:29.926381+0000 mgr.y (mgr.24401) 733 : cluster [DBG] pgmap v1250: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-04-26T16:36:31.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.274155+0000 osd.6 (osd.6) 13 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:36:31.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.274155+0000 osd.6 (osd.6) 13 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:36:31.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.276010+0000 osd.6 (osd.6) 14 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.276010+0000 osd.6 (osd.6) 14 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.650492+0000 osd.3 (osd.3) 447 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.650492+0000 osd.3 (osd.3) 447 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.652448+0000 osd.3 (osd.3) 448 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.652448+0000 osd.3 (osd.3) 448 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.926381+0000 mgr.y (mgr.24401) 733 : cluster [DBG] pgmap v1250: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:29.926381+0000 mgr.y (mgr.24401) 733 : cluster [DBG] pgmap v1250: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.274155+0000 osd.6 (osd.6) 13 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.274155+0000 osd.6 (osd.6) 13 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.276010+0000 osd.6 (osd.6) 14 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.276010+0000 osd.6 (osd.6) 14 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.650492+0000 osd.3 (osd.3) 447 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:36:31.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.650492+0000 osd.3 (osd.3) 447 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:36:31.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.652448+0000 osd.3 (osd.3) 448 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:36:31.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:31 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.652448+0000 osd.3 (osd.3) 448 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:36:31.780 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.926381+0000 mgr.y (mgr.24401) 733 : cluster [DBG] pgmap v1250: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:29.926381+0000 mgr.y (mgr.24401) 733 : cluster [DBG] pgmap v1250: 204 pgs: 1 active+clean+scrubbing, 203 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.274155+0000 osd.6 (osd.6) 13 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.274155+0000 osd.6 (osd.6) 13 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.276010+0000 osd.6 (osd.6) 14 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.276010+0000 osd.6 (osd.6) 14 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.650492+0000 osd.3 (osd.3) 447 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.650492+0000 osd.3 (osd.3) 447 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:36:31.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.652448+0000 osd.3 (osd.3) 448 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:36:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:31 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.652448+0000 osd.3 (osd.3) 448 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:36:31.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:31.910 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:31.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:32.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:32.041 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:32.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:32.144 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:32.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:32.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:32.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:32.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:32.435 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:32.538 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:32.574 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:32.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.742998+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.b scrub starts 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.742998+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.b scrub starts 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.745339+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.b scrub ok 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:30.745339+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.b scrub ok 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.603415+0000 osd.7 (osd.7) 415 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.603415+0000 osd.7 (osd.7) 415 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.603742+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:32.689 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.603742+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.650798+0000 osd.3 (osd.3) 449 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.650798+0000 osd.3 (osd.3) 449 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.653566+0000 osd.3 (osd.3) 450 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.653566+0000 osd.3 (osd.3) 450 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.017708+0000 osd.4 (osd.4) 389 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.017708+0000 osd.4 (osd.4) 389 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.020087+0000 osd.4 (osd.4) 390 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.020087+0000 osd.4 (osd.4) 390 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.511636+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:36:32.690 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:32 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.511636+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:36:32.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:32.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.742998+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.b scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.742998+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.b scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.745339+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.b scrub ok 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:30.745339+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.b scrub ok 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.603415+0000 osd.7 (osd.7) 415 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.603415+0000 osd.7 (osd.7) 415 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.603742+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.603742+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.650798+0000 osd.3 (osd.3) 449 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.650798+0000 osd.3 (osd.3) 449 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.653566+0000 osd.3 (osd.3) 450 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.653566+0000 osd.3 (osd.3) 450 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.017708+0000 osd.4 (osd.4) 389 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:36:32.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.017708+0000 osd.4 (osd.4) 389 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.020087+0000 osd.4 (osd.4) 390 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.020087+0000 osd.4 (osd.4) 390 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.511636+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.511636+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.742998+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.b scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.742998+0000 osd.2 (osd.2) 381 : cluster [DBG] 5.b scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.745339+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.b scrub ok 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:30.745339+0000 osd.2 (osd.2) 382 : cluster [DBG] 5.b scrub ok 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.603415+0000 osd.7 (osd.7) 415 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.603415+0000 osd.7 (osd.7) 415 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.603742+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:32.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.603742+0000 osd.7 (osd.7) 416 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.650798+0000 osd.3 (osd.3) 449 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.650798+0000 osd.3 (osd.3) 449 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.653566+0000 osd.3 (osd.3) 450 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.653566+0000 osd.3 (osd.3) 450 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.017708+0000 osd.4 (osd.4) 389 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.017708+0000 osd.4 (osd.4) 389 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.020087+0000 osd.4 (osd.4) 390 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.020087+0000 osd.4 (osd.4) 390 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.511636+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:36:32.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:32 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.511636+0000 osd.5 (osd.5) 297 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:36:32.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:32.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:32.965 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:32.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:33.068 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:33.096 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:33.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:33.227 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:33.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:33.357 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:33.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:33.490 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:33.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:33.621 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:33.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:33.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:33 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.928431+0000 mgr.y (mgr.24401) 734 : cluster [DBG] pgmap v1251: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.8 KiB/s wr, 2 op/s 2024-04-26T16:36:33.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:33 smithi190 bash[23868]: cluster 2024-04-26T16:36:31.928431+0000 mgr.y (mgr.24401) 734 : cluster [DBG] pgmap v1251: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.8 KiB/s wr, 2 op/s 2024-04-26T16:36:33.735 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:33 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.514110+0000 osd.5 (osd.5) 298 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:36:33.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:33 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.514110+0000 osd.5 (osd.5) 298 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:36:33.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:33 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.562348+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-04-26T16:36:33.736 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:33 smithi190 bash[23868]: cluster 2024-04-26T16:36:32.562348+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-04-26T16:36:33.750 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:33.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.928431+0000 mgr.y (mgr.24401) 734 : cluster [DBG] pgmap v1251: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.8 KiB/s wr, 2 op/s 2024-04-26T16:36:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[29234]: cluster 2024-04-26T16:36:31.928431+0000 mgr.y (mgr.24401) 734 : cluster [DBG] pgmap v1251: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.8 KiB/s wr, 2 op/s 2024-04-26T16:36:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.514110+0000 osd.5 (osd.5) 298 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:36:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.514110+0000 osd.5 (osd.5) 298 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:36:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.562348+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-04-26T16:36:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[29234]: cluster 2024-04-26T16:36:32.562348+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-04-26T16:36:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.928431+0000 mgr.y (mgr.24401) 734 : cluster [DBG] pgmap v1251: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.8 KiB/s wr, 2 op/s 2024-04-26T16:36:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[21463]: cluster 2024-04-26T16:36:31.928431+0000 mgr.y (mgr.24401) 734 : cluster [DBG] pgmap v1251: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.8 KiB/s wr, 2 op/s 2024-04-26T16:36:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.514110+0000 osd.5 (osd.5) 298 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:36:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.514110+0000 osd.5 (osd.5) 298 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:36:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.562348+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-04-26T16:36:33.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:33 smithi050 bash[21463]: cluster 2024-04-26T16:36:32.562348+0000 mon.a (mon.0) 3745 : cluster [DBG] osdmap e806: 8 total, 8 up, 8 in 2024-04-26T16:36:33.881 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:33.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:33.984 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:34.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:34.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:34.143 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:34.246 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:34.274 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:34.378 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:34.401 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:34.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:34.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:34.533 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:34.553 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:36:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:36:34.560 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.163944+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.163944+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.166168+0000 osd.7 (osd.7) 418 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.166168+0000 osd.7 (osd.7) 418 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.242964+0000 osd.1 (osd.1) 415 : cluster [DBG] 11.c scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.242964+0000 osd.1 (osd.1) 415 : cluster [DBG] 11.c scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.244845+0000 osd.1 (osd.1) 416 : cluster [DBG] 11.c scrub ok 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.244845+0000 osd.1 (osd.1) 416 : cluster [DBG] 11.c scrub ok 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.539526+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.539526+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.541432+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.541432+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.750264+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.c scrub starts 2024-04-26T16:36:34.561 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.750264+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.c scrub starts 2024-04-26T16:36:34.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.751632+0000 osd.2 (osd.2) 384 : cluster [DBG] 2.c scrub ok 2024-04-26T16:36:34.562 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.751632+0000 osd.2 (osd.2) 384 : cluster [DBG] 2.c scrub ok 2024-04-26T16:36:34.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.163944+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:36:34.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.163944+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:36:34.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.166168+0000 osd.7 (osd.7) 418 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:36:34.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.166168+0000 osd.7 (osd.7) 418 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:36:34.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.242964+0000 osd.1 (osd.1) 415 : cluster [DBG] 11.c scrub starts 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.242964+0000 osd.1 (osd.1) 415 : cluster [DBG] 11.c scrub starts 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.244845+0000 osd.1 (osd.1) 416 : cluster [DBG] 11.c scrub ok 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.244845+0000 osd.1 (osd.1) 416 : cluster [DBG] 11.c scrub ok 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.539526+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.539526+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.541432+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.541432+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.750264+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.c scrub starts 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.750264+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.c scrub starts 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.751632+0000 osd.2 (osd.2) 384 : cluster [DBG] 2.c scrub ok 2024-04-26T16:36:34.604 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:34 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.751632+0000 osd.2 (osd.2) 384 : cluster [DBG] 2.c scrub ok 2024-04-26T16:36:34.636 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:34.664 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:34.767 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:34.796 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.163944+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.163944+0000 osd.7 (osd.7) 417 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.166168+0000 osd.7 (osd.7) 418 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.166168+0000 osd.7 (osd.7) 418 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.242964+0000 osd.1 (osd.1) 415 : cluster [DBG] 11.c scrub starts 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.242964+0000 osd.1 (osd.1) 415 : cluster [DBG] 11.c scrub starts 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.244845+0000 osd.1 (osd.1) 416 : cluster [DBG] 11.c scrub ok 2024-04-26T16:36:34.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.244845+0000 osd.1 (osd.1) 416 : cluster [DBG] 11.c scrub ok 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.539526+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.539526+0000 osd.5 (osd.5) 299 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.541432+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.541432+0000 osd.5 (osd.5) 300 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.750264+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.c scrub starts 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.750264+0000 osd.2 (osd.2) 383 : cluster [DBG] 2.c scrub starts 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.751632+0000 osd.2 (osd.2) 384 : cluster [DBG] 2.c scrub ok 2024-04-26T16:36:34.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:34 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.751632+0000 osd.2 (osd.2) 384 : cluster [DBG] 2.c scrub ok 2024-04-26T16:36:34.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:34.925 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:34.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:35.029 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:35.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:35.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:35.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:35.186 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:35.289 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:35.318 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:35.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:35.449 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:35.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:35.578 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:35.681 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:35.711 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:35.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:35.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:35 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.929493+0000 mgr.y (mgr.24401) 735 : cluster [DBG] pgmap v1253: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-26T16:36:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:35 smithi190 bash[23868]: cluster 2024-04-26T16:36:33.929493+0000 mgr.y (mgr.24401) 735 : cluster [DBG] pgmap v1253: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-26T16:36:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:35 smithi190 bash[23868]: cluster 2024-04-26T16:36:34.772142+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:36:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:35 smithi190 bash[23868]: cluster 2024-04-26T16:36:34.772142+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:36:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:35 smithi190 bash[23868]: cluster 2024-04-26T16:36:34.775123+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:36:35.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:35 smithi190 bash[23868]: cluster 2024-04-26T16:36:34.775123+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:36:35.842 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:35.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:35.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.929493+0000 mgr.y (mgr.24401) 735 : cluster [DBG] pgmap v1253: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-26T16:36:35.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[29234]: cluster 2024-04-26T16:36:33.929493+0000 mgr.y (mgr.24401) 735 : cluster [DBG] pgmap v1253: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-26T16:36:35.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[29234]: cluster 2024-04-26T16:36:34.772142+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:36:35.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[29234]: cluster 2024-04-26T16:36:34.772142+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:36:35.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[29234]: cluster 2024-04-26T16:36:34.775123+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:36:35.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[29234]: cluster 2024-04-26T16:36:34.775123+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:36:35.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.929493+0000 mgr.y (mgr.24401) 735 : cluster [DBG] pgmap v1253: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-26T16:36:35.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[21463]: cluster 2024-04-26T16:36:33.929493+0000 mgr.y (mgr.24401) 735 : cluster [DBG] pgmap v1253: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 7.2 KiB/s wr, 2 op/s 2024-04-26T16:36:35.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[21463]: cluster 2024-04-26T16:36:34.772142+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:36:35.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[21463]: cluster 2024-04-26T16:36:34.772142+0000 osd.2 (osd.2) 385 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:36:35.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[21463]: cluster 2024-04-26T16:36:34.775123+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:36:35.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:35 smithi050 bash[21463]: cluster 2024-04-26T16:36:34.775123+0000 osd.2 (osd.2) 386 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:36:35.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:35.973 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:35.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:36.076 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:36.102 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:36.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:36.232 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:36.336 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:36.357 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:36.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:36.489 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:36.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:36.592 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:36.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.210091+0000 osd.7 (osd.7) 419 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:36:36.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.210091+0000 osd.7 (osd.7) 419 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:36:36.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.211438+0000 osd.7 (osd.7) 420 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:36:36.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.211438+0000 osd.7 (osd.7) 420 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:36:36.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.652149+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:36:36.602 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.652149+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.656368+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.656368+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.382854+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.382854+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.210091+0000 osd.7 (osd.7) 419 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.210091+0000 osd.7 (osd.7) 419 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.211438+0000 osd.7 (osd.7) 420 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.211438+0000 osd.7 (osd.7) 420 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.652149+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.652149+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.656368+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:36:36.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.656368+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:36:36.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.382854+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-04-26T16:36:36.604 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:36 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.382854+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-04-26T16:36:36.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:36:36.617 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:36.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:36.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.210091+0000 osd.7 (osd.7) 419 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:36:36.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.210091+0000 osd.7 (osd.7) 419 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:36:36.732 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.211438+0000 osd.7 (osd.7) 420 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.211438+0000 osd.7 (osd.7) 420 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.652149+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.652149+0000 osd.3 (osd.3) 451 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.656368+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.656368+0000 osd.3 (osd.3) 452 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.382854+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-04-26T16:36:36.733 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:36 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.382854+0000 mon.a (mon.0) 3746 : cluster [DBG] osdmap e807: 8 total, 8 up, 8 in 2024-04-26T16:36:36.749 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:36.852 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:36.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:36.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:36.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:37.003 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:37.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:37.134 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:37.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:37.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:37.264 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:37.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:37.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:37.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:37.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:37.522 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:37.561 INFO:teuthology.orchestra.run.smithi050.stdout:{ 2024-04-26T16:36:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "ops": [], 2024-04-26T16:36:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "complaint_time": 30, 2024-04-26T16:36:37.561 INFO:teuthology.orchestra.run.smithi050.stdout: "num_blocked_ops": 0 2024-04-26T16:36:37.561 INFO:teuthology.orchestra.run.smithi050.stdout:} 2024-04-26T16:36:37.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:36:37.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:37.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.930331+0000 mgr.y (mgr.24401) 736 : cluster [DBG] pgmap v1254: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.1 KiB/s wr, 1 op/s 2024-04-26T16:36:37.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.930331+0000 mgr.y (mgr.24401) 736 : cluster [DBG] pgmap v1254: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.1 KiB/s wr, 1 op/s 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.967561+0000 osd.0 (osd.0) 437 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.967561+0000 osd.0 (osd.0) 437 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.971252+0000 osd.0 (osd.0) 438 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:35.971252+0000 osd.0 (osd.0) 438 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.244905+0000 osd.1 (osd.1) 417 : cluster [DBG] 5.c scrub starts 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.244905+0000 osd.1 (osd.1) 417 : cluster [DBG] 5.c scrub starts 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.246968+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub ok 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.246968+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub ok 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.630622+0000 osd.3 (osd.3) 453 : cluster [DBG] 11.a scrub starts 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.630622+0000 osd.3 (osd.3) 453 : cluster [DBG] 11.a scrub starts 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.632365+0000 osd.3 (osd.3) 454 : cluster [DBG] 11.a scrub ok 2024-04-26T16:36:37.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.632365+0000 osd.3 (osd.3) 454 : cluster [DBG] 11.a scrub ok 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.163027+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.163027+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.163306+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.163306+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.930331+0000 mgr.y (mgr.24401) 736 : cluster [DBG] pgmap v1254: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.1 KiB/s wr, 1 op/s 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.930331+0000 mgr.y (mgr.24401) 736 : cluster [DBG] pgmap v1254: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.1 KiB/s wr, 1 op/s 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.967561+0000 osd.0 (osd.0) 437 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.967561+0000 osd.0 (osd.0) 437 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.971252+0000 osd.0 (osd.0) 438 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:36:37.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:35.971252+0000 osd.0 (osd.0) 438 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.244905+0000 osd.1 (osd.1) 417 : cluster [DBG] 5.c scrub starts 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.244905+0000 osd.1 (osd.1) 417 : cluster [DBG] 5.c scrub starts 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.246968+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub ok 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.246968+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub ok 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.630622+0000 osd.3 (osd.3) 453 : cluster [DBG] 11.a scrub starts 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.630622+0000 osd.3 (osd.3) 453 : cluster [DBG] 11.a scrub starts 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.632365+0000 osd.3 (osd.3) 454 : cluster [DBG] 11.a scrub ok 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.632365+0000 osd.3 (osd.3) 454 : cluster [DBG] 11.a scrub ok 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.163027+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.163027+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.163306+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:37.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:37 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.163306+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:37.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:37.751 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:37.779 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.930331+0000 mgr.y (mgr.24401) 736 : cluster [DBG] pgmap v1254: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.1 KiB/s wr, 1 op/s 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.930331+0000 mgr.y (mgr.24401) 736 : cluster [DBG] pgmap v1254: 204 pgs: 204 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 6.1 KiB/s wr, 1 op/s 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.967561+0000 osd.0 (osd.0) 437 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.967561+0000 osd.0 (osd.0) 437 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.971252+0000 osd.0 (osd.0) 438 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:35.971252+0000 osd.0 (osd.0) 438 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.244905+0000 osd.1 (osd.1) 417 : cluster [DBG] 5.c scrub starts 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.244905+0000 osd.1 (osd.1) 417 : cluster [DBG] 5.c scrub starts 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.246968+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub ok 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.246968+0000 osd.1 (osd.1) 418 : cluster [DBG] 5.c scrub ok 2024-04-26T16:36:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.630622+0000 osd.3 (osd.3) 453 : cluster [DBG] 11.a scrub starts 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.630622+0000 osd.3 (osd.3) 453 : cluster [DBG] 11.a scrub starts 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.632365+0000 osd.3 (osd.3) 454 : cluster [DBG] 11.a scrub ok 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.632365+0000 osd.3 (osd.3) 454 : cluster [DBG] 11.a scrub ok 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.163027+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.163027+0000 osd.5 (osd.5) 301 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.163306+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:37.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:37 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.163306+0000 osd.5 (osd.5) 302 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:37.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:37.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:37.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:38.009 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:38.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:38.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:38.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:38.169 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:38.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:38.301 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:38.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:38.431 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:38.534 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:38.563 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:38.666 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:38.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:38.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:38.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.998798+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:36:38.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:36.998798+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:36:38.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.003110+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:36:38.806 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.003110+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.184089+0000 osd.7 (osd.7) 421 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.184089+0000 osd.7 (osd.7) 421 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.185569+0000 osd.7 (osd.7) 422 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.185569+0000 osd.7 (osd.7) 422 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.289239+0000 mgr.y (mgr.24401) 737 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.289239+0000 mgr.y (mgr.24401) 737 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.591154+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.591154+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.591912+0000 mon.c (mon.2) 838 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.591912+0000 mon.c (mon.2) 838 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.592351+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.592351+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.593269+0000 mon.c (mon.2) 839 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.807 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.593269+0000 mon.c (mon.2) 839 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.593579+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: audit 2024-04-26T16:36:37.593579+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.998798+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:36.998798+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.003110+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.003110+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.184089+0000 osd.7 (osd.7) 421 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.184089+0000 osd.7 (osd.7) 421 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.185569+0000 osd.7 (osd.7) 422 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:36:38.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.185569+0000 osd.7 (osd.7) 422 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.289239+0000 mgr.y (mgr.24401) 737 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.289239+0000 mgr.y (mgr.24401) 737 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.591154+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.591154+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.591912+0000 mon.c (mon.2) 838 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.591912+0000 mon.c (mon.2) 838 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.592351+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.592351+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.593269+0000 mon.c (mon.2) 839 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.593269+0000 mon.c (mon.2) 839 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.593579+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: audit 2024-04-26T16:36:37.593579+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.658552+0000 osd.5 (osd.5) 303 : cluster [DBG] 323.7s0 scrub starts 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.658552+0000 osd.5 (osd.5) 303 : cluster [DBG] 323.7s0 scrub starts 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.662369+0000 osd.5 (osd.5) 304 : cluster [DBG] 323.7 scrub ok 2024-04-26T16:36:38.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.662369+0000 osd.5 (osd.5) 304 : cluster [DBG] 323.7 scrub ok 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.672075+0000 osd.3 (osd.3) 455 : cluster [DBG] 323.3s0 scrub starts 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.672075+0000 osd.3 (osd.3) 455 : cluster [DBG] 323.3s0 scrub starts 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.675006+0000 osd.3 (osd.3) 456 : cluster [DBG] 323.3 scrub ok 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.675006+0000 osd.3 (osd.3) 456 : cluster [DBG] 323.3 scrub ok 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.658552+0000 osd.5 (osd.5) 303 : cluster [DBG] 323.7s0 scrub starts 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.658552+0000 osd.5 (osd.5) 303 : cluster [DBG] 323.7s0 scrub starts 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.662369+0000 osd.5 (osd.5) 304 : cluster [DBG] 323.7 scrub ok 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.662369+0000 osd.5 (osd.5) 304 : cluster [DBG] 323.7 scrub ok 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.672075+0000 osd.3 (osd.3) 455 : cluster [DBG] 323.3s0 scrub starts 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.672075+0000 osd.3 (osd.3) 455 : cluster [DBG] 323.3s0 scrub starts 2024-04-26T16:36:38.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.675006+0000 osd.3 (osd.3) 456 : cluster [DBG] 323.3 scrub ok 2024-04-26T16:36:38.811 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:38 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.675006+0000 osd.3 (osd.3) 456 : cluster [DBG] 323.3 scrub ok 2024-04-26T16:36:38.821 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.998798+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:36:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:36.998798+0000 osd.0 (osd.0) 439 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:36:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.003110+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:36:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.003110+0000 osd.0 (osd.0) 440 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:36:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.184089+0000 osd.7 (osd.7) 421 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:36:38.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.184089+0000 osd.7 (osd.7) 421 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.185569+0000 osd.7 (osd.7) 422 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.185569+0000 osd.7 (osd.7) 422 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.289239+0000 mgr.y (mgr.24401) 737 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.289239+0000 mgr.y (mgr.24401) 737 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.591154+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.591154+0000 mon.a (mon.0) 3747 : cluster [DBG] osdmap e808: 8 total, 8 up, 8 in 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.591912+0000 mon.c (mon.2) 838 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.591912+0000 mon.c (mon.2) 838 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.592351+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.592351+0000 mon.a (mon.0) 3748 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.593269+0000 mon.c (mon.2) 839 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.593269+0000 mon.c (mon.2) 839 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.593579+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: audit 2024-04-26T16:36:37.593579+0000 mon.a (mon.0) 3749 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]: dispatch 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.658552+0000 osd.5 (osd.5) 303 : cluster [DBG] 323.7s0 scrub starts 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.658552+0000 osd.5 (osd.5) 303 : cluster [DBG] 323.7s0 scrub starts 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.662369+0000 osd.5 (osd.5) 304 : cluster [DBG] 323.7 scrub ok 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.662369+0000 osd.5 (osd.5) 304 : cluster [DBG] 323.7 scrub ok 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.672075+0000 osd.3 (osd.3) 455 : cluster [DBG] 323.3s0 scrub starts 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.672075+0000 osd.3 (osd.3) 455 : cluster [DBG] 323.3s0 scrub starts 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.675006+0000 osd.3 (osd.3) 456 : cluster [DBG] 323.3 scrub ok 2024-04-26T16:36:38.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:38 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.675006+0000 osd.3 (osd.3) 456 : cluster [DBG] 323.3 scrub ok 2024-04-26T16:36:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- 172.21.15.50:0/4102397608 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 msgr2=0x7fbecc110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/4102397608 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc110310 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fbec400b0a0 tx=0x7fbec400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:36:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- 172.21.15.50:0/4102397608 shutdown_connections 2024-04-26T16:36:38.920 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/4102397608 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc1094d0 0x7fbecc1098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/4102397608 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc110310 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/4102397608 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc06bbd0 0x7fbecc06c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- 172.21.15.50:0/4102397608 >> 172.21.15.50:0/4102397608 conn(0x7fbecc0793c0 msgr2=0x7fbecc0797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- 172.21.15.50:0/4102397608 shutdown_connections 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- 172.21.15.50:0/4102397608 wait complete. 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 librados: starting msgr at 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 librados: starting objecter 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 Processor -- start 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- start start 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 librados: setting wanted keys 2024-04-26T16:36:38.921 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 librados: calling monclient init 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc06bbd0 0x7fbecc199110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc199650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc1094d0 0x7fbecc19d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc112b30 con 0x7fbecc1094d0 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbecc1129b0 con 0x7fbecc06bbd0 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed3ff8700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fbecc112cb0 con 0x7fbecc06c590 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc199650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc199650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56102/0 (socket says 172.21.15.50:56102) 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 -- 172.21.15.50:0/2883109185 learned_addr learned my addr 172.21.15.50:0/2883109185 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed2595700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc1094d0 0x7fbecc19d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1d94700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc06bbd0 0x7fbecc199110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 -- 172.21.15.50:0/2883109185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc06bbd0 msgr2=0x7fbecc199110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:38.923 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc06bbd0 0x7fbecc199110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 -- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc1094d0 msgr2=0x7fbecc19d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc1094d0 0x7fbecc19d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 -- 172.21.15.50:0/2883109185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbecc19e120 con 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1d94700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc06bbd0 0x7fbecc199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbed1593700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc199650 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00eb30 tx=0x7fbebc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbec2ffd700 10 client.?.objecter ms_handle_connect 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.921+0000 7fbec2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc00cc90 con 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fbebc00ce30 con 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbecc19e410 con 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbebc003d80 con 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed3ff8700 10 client.41654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbecc1a5c90 con 0x7fbecc06c590 2024-04-26T16:36:38.924 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed3ff8700 1 librados: init done 2024-04-26T16:36:38.925 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbecc06a110 con 0x7fbecc06c590 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fbebc003f20 con 0x7fbecc06c590 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbeb8071f80 0x7fbeb8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(809..809 src has 577..809) ==== 9939+0+0 (secure 0 0 0) 0x7fbebc091ee0 con 0x7fbecc06c590 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 10 client.41654.objecter ms_dispatch 0x7fbecc001550 osd_map(809..809 src has 577..809) 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 3 client.41654.objecter handle_osd_map got epochs [809,809] > 0 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 3 client.41654.objecter handle_osd_map decoding full epoch 809 2024-04-26T16:36:38.926 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed1d94700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbeb8071f80 0x7fbeb8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 20 client.41654.objecter dump_active .. 0 homeless 2024-04-26T16:36:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbec2ffd700 10 client.41654.objecter ms_handle_connect 0x7fbeb8071f80 2024-04-26T16:36:38.927 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.925+0000 7fbed1d94700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbeb8071f80 0x7fbeb8074440 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7fbec8007900 tx=0x7fbec8008040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:38.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:38.929+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fbebc05d680 con 0x7fbecc06c590 2024-04-26T16:36:38.945 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:38.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:39.048 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:39.074 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:39.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.173+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7fbecc1a5f20 con 0x7fbecc06c590 2024-04-26T16:36:39.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:39.205 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:39.308 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:39.335 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:39.438 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:39.467 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:39.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:39.597 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:39.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.605+0000 7fbec2ffd700 1 -- 172.21.15.50:0/2883109185 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v810) ==== 101+0+0 (secure 0 0 0) 0x7fbebc061330 con 0x7fbecc06c590 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.613+0000 7fbed3ff8700 10 librados: watch_flush enter 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.613+0000 7fbed3ff8700 10 librados: watch_flush exit 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.613+0000 7fbed3ff8700 20 client.41654.objecter shutdown clearing up homeless session... 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.613+0000 7fbed3ff8700 10 client.41654.objecter successfully canceled tick 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbeb8071f80 msgr2=0x7fbeb8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbeb8071f80 0x7fbeb8074440 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7fbec8007900 tx=0x7fbec8008040 comp rx=0 tx=0).stop 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 msgr2=0x7fbecc199650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:39.614 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc199650 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7fbebc00eb30 tx=0x7fbebc00ee40 comp rx=0 tx=0).stop 2024-04-26T16:36:39.615 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 shutdown_connections 2024-04-26T16:36:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbeb8071f80 0x7fbeb8074440 unknown :-1 s=CLOSED pgs=3834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:39.616 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbecc06bbd0 0x7fbecc199110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbecc1094d0 0x7fbecc19d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 --2- 172.21.15.50:0/2883109185 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbecc06c590 0x7fbecc199650 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 >> 172.21.15.50:0/2883109185 conn(0x7fbecc0793c0 msgr2=0x7fbecc104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 shutdown_connections 2024-04-26T16:36:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 -- 172.21.15.50:0/2883109185 wait complete. 2024-04-26T16:36:39.617 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:39.617+0000 7fbed3ff8700 1 librados: shutdown 2024-04-26T16:36:39.618 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:36:39.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:39.729 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:39.832 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:39.860 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:39.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.931041+0000 mgr.y (mgr.24401) 738 : cluster [DBG] pgmap v1257: 204 pgs: 204 active+clean; 4.5 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-04-26T16:36:39.861 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.931041+0000 mgr.y (mgr.24401) 738 : cluster [DBG] pgmap v1257: 204 pgs: 204 active+clean; 4.5 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.996584+0000 osd.0 (osd.0) 441 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.996584+0000 osd.0 (osd.0) 441 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.998575+0000 osd.0 (osd.0) 442 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:37.998575+0000 osd.0 (osd.0) 442 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.154953+0000 osd.7 (osd.7) 423 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.154953+0000 osd.7 (osd.7) 423 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.157408+0000 osd.7 (osd.7) 424 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.157408+0000 osd.7 (osd.7) 424 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: audit 2024-04-26T16:36:38.595045+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]': finished 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: audit 2024-04-26T16:36:38.595045+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]': finished 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.601714+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.601714+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-04-26T16:36:39.862 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.639604+0000 osd.3 (osd.3) 457 : cluster [DBG] 11.b scrub starts 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.639604+0000 osd.3 (osd.3) 457 : cluster [DBG] 11.b scrub starts 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.645043+0000 osd.3 (osd.3) 458 : cluster [DBG] 11.b scrub ok 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.645043+0000 osd.3 (osd.3) 458 : cluster [DBG] 11.b scrub ok 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.707810+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.d scrub starts 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.707810+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.d scrub starts 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.710192+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.d scrub ok 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.710192+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.d scrub ok 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: audit 2024-04-26T16:36:39.176304+0000 mon.c (mon.2) 840 : audit [INF] from='client.? 172.21.15.50:0/2883109185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: audit 2024-04-26T16:36:39.176304+0000 mon.c (mon.2) 840 : audit [INF] from='client.? 172.21.15.50:0/2883109185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: audit 2024-04-26T16:36:39.176710+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.863 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[29234]: audit 2024-04-26T16:36:39.176710+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.931041+0000 mgr.y (mgr.24401) 738 : cluster [DBG] pgmap v1257: 204 pgs: 204 active+clean; 4.5 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.931041+0000 mgr.y (mgr.24401) 738 : cluster [DBG] pgmap v1257: 204 pgs: 204 active+clean; 4.5 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.996584+0000 osd.0 (osd.0) 441 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.996584+0000 osd.0 (osd.0) 441 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.998575+0000 osd.0 (osd.0) 442 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:37.998575+0000 osd.0 (osd.0) 442 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.154953+0000 osd.7 (osd.7) 423 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.154953+0000 osd.7 (osd.7) 423 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.157408+0000 osd.7 (osd.7) 424 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.157408+0000 osd.7 (osd.7) 424 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: audit 2024-04-26T16:36:38.595045+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]': finished 2024-04-26T16:36:39.865 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: audit 2024-04-26T16:36:38.595045+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]': finished 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.601714+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.601714+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.639604+0000 osd.3 (osd.3) 457 : cluster [DBG] 11.b scrub starts 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.639604+0000 osd.3 (osd.3) 457 : cluster [DBG] 11.b scrub starts 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.645043+0000 osd.3 (osd.3) 458 : cluster [DBG] 11.b scrub ok 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.645043+0000 osd.3 (osd.3) 458 : cluster [DBG] 11.b scrub ok 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.707810+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.d scrub starts 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.707810+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.d scrub starts 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.710192+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.d scrub ok 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.710192+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.d scrub ok 2024-04-26T16:36:39.866 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: audit 2024-04-26T16:36:39.176304+0000 mon.c (mon.2) 840 : audit [INF] from='client.? 172.21.15.50:0/2883109185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: audit 2024-04-26T16:36:39.176304+0000 mon.c (mon.2) 840 : audit [INF] from='client.? 172.21.15.50:0/2883109185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.867 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: audit 2024-04-26T16:36:39.176710+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.868 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:39 smithi050 bash[21463]: audit 2024-04-26T16:36:39.176710+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:39.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.931041+0000 mgr.y (mgr.24401) 738 : cluster [DBG] pgmap v1257: 204 pgs: 204 active+clean; 4.5 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.931041+0000 mgr.y (mgr.24401) 738 : cluster [DBG] pgmap v1257: 204 pgs: 204 active+clean; 4.5 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 5.5 KiB/s wr, 1 op/s 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.996584+0000 osd.0 (osd.0) 441 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.996584+0000 osd.0 (osd.0) 441 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.998575+0000 osd.0 (osd.0) 442 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:37.998575+0000 osd.0 (osd.0) 442 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.154953+0000 osd.7 (osd.7) 423 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.154953+0000 osd.7 (osd.7) 423 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.157408+0000 osd.7 (osd.7) 424 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.157408+0000 osd.7 (osd.7) 424 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: audit 2024-04-26T16:36:38.595045+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]': finished 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: audit 2024-04-26T16:36:38.595045+0000 mon.a (mon.0) 3750 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-140"}]': finished 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.601714+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-04-26T16:36:39.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.601714+0000 mon.a (mon.0) 3751 : cluster [DBG] osdmap e809: 8 total, 8 up, 8 in 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.639604+0000 osd.3 (osd.3) 457 : cluster [DBG] 11.b scrub starts 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.639604+0000 osd.3 (osd.3) 457 : cluster [DBG] 11.b scrub starts 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.645043+0000 osd.3 (osd.3) 458 : cluster [DBG] 11.b scrub ok 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.645043+0000 osd.3 (osd.3) 458 : cluster [DBG] 11.b scrub ok 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.707810+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.d scrub starts 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.707810+0000 osd.2 (osd.2) 387 : cluster [DBG] 5.d scrub starts 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.710192+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.d scrub ok 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.710192+0000 osd.2 (osd.2) 388 : cluster [DBG] 5.d scrub ok 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: audit 2024-04-26T16:36:39.176304+0000 mon.c (mon.2) 840 : audit [INF] from='client.? 172.21.15.50:0/2883109185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: audit 2024-04-26T16:36:39.176304+0000 mon.c (mon.2) 840 : audit [INF] from='client.? 172.21.15.50:0/2883109185' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: audit 2024-04-26T16:36:39.176710+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.979 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:39 smithi190 bash[23868]: audit 2024-04-26T16:36:39.176710+0000 mon.a (mon.0) 3752 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:36:39.991 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:39.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:40.095 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:40.123 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:40.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:40.254 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:40.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:40.387 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:40.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:40.518 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:40.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:40.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.968334+0000 osd.0 (osd.0) 443 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:36:40.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.968334+0000 osd.0 (osd.0) 443 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:36:40.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.969863+0000 osd.0 (osd.0) 444 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:36:40.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:38.969863+0000 osd.0 (osd.0) 444 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:36:40.624 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.151112+0000 osd.1 (osd.1) 419 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.151112+0000 osd.1 (osd.1) 419 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.152734+0000 osd.1 (osd.1) 420 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.152734+0000 osd.1 (osd.1) 420 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.154440+0000 osd.7 (osd.7) 425 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.154440+0000 osd.7 (osd.7) 425 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.156929+0000 osd.7 (osd.7) 426 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.156929+0000 osd.7 (osd.7) 426 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.595570+0000 mon.a (mon.0) 3753 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.595570+0000 mon.a (mon.0) 3753 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: audit 2024-04-26T16:36:39.597654+0000 mon.a (mon.0) 3754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: audit 2024-04-26T16:36:39.597654+0000 mon.a (mon.0) 3754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.604386+0000 mon.a (mon.0) 3755 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.604386+0000 mon.a (mon.0) 3755 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.718697+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:36:40.625 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.718697+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:36:40.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.720499+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:36:40.626 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.720499+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:36:40.655 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:40.758 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:40.785 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:40.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.968334+0000 osd.0 (osd.0) 443 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.968334+0000 osd.0 (osd.0) 443 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.969863+0000 osd.0 (osd.0) 444 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:38.969863+0000 osd.0 (osd.0) 444 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.151112+0000 osd.1 (osd.1) 419 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.151112+0000 osd.1 (osd.1) 419 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.152734+0000 osd.1 (osd.1) 420 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:36:40.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.152734+0000 osd.1 (osd.1) 420 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.154440+0000 osd.7 (osd.7) 425 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.154440+0000 osd.7 (osd.7) 425 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.156929+0000 osd.7 (osd.7) 426 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.156929+0000 osd.7 (osd.7) 426 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.595570+0000 mon.a (mon.0) 3753 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.595570+0000 mon.a (mon.0) 3753 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: audit 2024-04-26T16:36:39.597654+0000 mon.a (mon.0) 3754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:36:40.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: audit 2024-04-26T16:36:39.597654+0000 mon.a (mon.0) 3754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:36:40.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.604386+0000 mon.a (mon.0) 3755 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-04-26T16:36:40.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.604386+0000 mon.a (mon.0) 3755 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-04-26T16:36:40.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.718697+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:36:40.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.718697+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:36:40.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.720499+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:36:40.829 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:40 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.720499+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:36:40.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-04-26T16:36:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.968334+0000 osd.0 (osd.0) 443 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:36:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.968334+0000 osd.0 (osd.0) 443 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:36:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.969863+0000 osd.0 (osd.0) 444 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:36:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:38.969863+0000 osd.0 (osd.0) 444 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.151112+0000 osd.1 (osd.1) 419 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.151112+0000 osd.1 (osd.1) 419 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.152734+0000 osd.1 (osd.1) 420 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.152734+0000 osd.1 (osd.1) 420 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.154440+0000 osd.7 (osd.7) 425 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.154440+0000 osd.7 (osd.7) 425 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.156929+0000 osd.7 (osd.7) 426 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.156929+0000 osd.7 (osd.7) 426 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:36:40.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.595570+0000 mon.a (mon.0) 3753 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.595570+0000 mon.a (mon.0) 3753 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: audit 2024-04-26T16:36:39.597654+0000 mon.a (mon.0) 3754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: audit 2024-04-26T16:36:39.597654+0000 mon.a (mon.0) 3754 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.604386+0000 mon.a (mon.0) 3755 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.604386+0000 mon.a (mon.0) 3755 : cluster [DBG] osdmap e810: 8 total, 8 up, 8 in 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.718697+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:36:40.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.718697+0000 osd.2 (osd.2) 389 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:36:40.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.720499+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:36:40.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:40 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.720499+0000 osd.2 (osd.2) 390 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:36:40.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:40.917 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:40.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:41.021 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:41.044 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:41.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:41.174 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:41.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:41.305 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:41.409 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:41.432 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:41.536 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:41.560 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:41.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:41.691 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:41.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:41.795 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:41.822 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.931562+0000 mgr.y (mgr.24401) 739 : cluster [DBG] pgmap v1260: 172 pgs: 172 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:39.931562+0000 mgr.y (mgr.24401) 739 : cluster [DBG] pgmap v1260: 172 pgs: 172 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.631832+0000 mon.a (mon.0) 3756 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-04-26T16:36:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.631832+0000 mon.a (mon.0) 3756 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-04-26T16:36:41.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: audit 2024-04-26T16:36:40.638370+0000 mon.c (mon.2) 841 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: audit 2024-04-26T16:36:40.638370+0000 mon.c (mon.2) 841 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: audit 2024-04-26T16:36:40.641237+0000 mon.a (mon.0) 3757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: audit 2024-04-26T16:36:40.641237+0000 mon.a (mon.0) 3757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.655229+0000 osd.5 (osd.5) 305 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.655229+0000 osd.5 (osd.5) 305 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.657531+0000 osd.5 (osd.5) 306 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.657531+0000 osd.5 (osd.5) 306 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.021002+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.021002+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.021435+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.021435+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:41.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.931562+0000 mgr.y (mgr.24401) 739 : cluster [DBG] pgmap v1260: 172 pgs: 172 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:39.931562+0000 mgr.y (mgr.24401) 739 : cluster [DBG] pgmap v1260: 172 pgs: 172 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.631832+0000 mon.a (mon.0) 3756 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.631832+0000 mon.a (mon.0) 3756 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: audit 2024-04-26T16:36:40.638370+0000 mon.c (mon.2) 841 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: audit 2024-04-26T16:36:40.638370+0000 mon.c (mon.2) 841 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: audit 2024-04-26T16:36:40.641237+0000 mon.a (mon.0) 3757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: audit 2024-04-26T16:36:40.641237+0000 mon.a (mon.0) 3757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.655229+0000 osd.5 (osd.5) 305 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.655229+0000 osd.5 (osd.5) 305 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.657531+0000 osd.5 (osd.5) 306 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.657531+0000 osd.5 (osd.5) 306 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.021002+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:41.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.021002+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:41.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.021435+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:41.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:41 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.021435+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:41.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.931562+0000 mgr.y (mgr.24401) 739 : cluster [DBG] pgmap v1260: 172 pgs: 172 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:39.931562+0000 mgr.y (mgr.24401) 739 : cluster [DBG] pgmap v1260: 172 pgs: 172 active+clean; 4.4 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.631832+0000 mon.a (mon.0) 3756 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.631832+0000 mon.a (mon.0) 3756 : cluster [DBG] osdmap e811: 8 total, 8 up, 8 in 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: audit 2024-04-26T16:36:40.638370+0000 mon.c (mon.2) 841 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: audit 2024-04-26T16:36:40.638370+0000 mon.c (mon.2) 841 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: audit 2024-04-26T16:36:40.641237+0000 mon.a (mon.0) 3757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: audit 2024-04-26T16:36:40.641237+0000 mon.a (mon.0) 3757 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.655229+0000 osd.5 (osd.5) 305 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.655229+0000 osd.5 (osd.5) 305 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:36:41.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.657531+0000 osd.5 (osd.5) 306 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:36:41.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.657531+0000 osd.5 (osd.5) 306 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:36:41.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.021002+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:41.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.021002+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:41.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.021435+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:41.938 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:41 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.021435+0000 osd.6 (osd.6) 16 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:41.954 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:41.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:41.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:42.057 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:42.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:42.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:42.188 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:42.217 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:42.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:42.346 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:42.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:42.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:42.477 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:42.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:42.607 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:42.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:42.738 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:42.841 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:42.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.955711+0000 osd.0 (osd.0) 445 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:36:42.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.955711+0000 osd.0 (osd.0) 445 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:36:42.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.957044+0000 osd.0 (osd.0) 446 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:36:42.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:40.957044+0000 osd.0 (osd.0) 446 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:41.623980+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:41.623980+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.629903+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.629903+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.685501+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.685501+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.686914+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.686914+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:41.710871+0000 mon.c (mon.2) 842 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:41.710871+0000 mon.c (mon.2) 842 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:41.711357+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:41.711357+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.729587+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.e scrub starts 2024-04-26T16:36:42.858 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.729587+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.e scrub starts 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.730951+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.e scrub ok 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.730951+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.e scrub ok 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.909592+0000 osd.0 (osd.0) 447 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.909592+0000 osd.0 (osd.0) 447 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.912871+0000 osd.0 (osd.0) 448 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.912871+0000 osd.0 (osd.0) 448 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:42.625941+0000 mon.a (mon.0) 3761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:42.625941+0000 mon.a (mon.0) 3761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:42.631162+0000 mon.c (mon.2) 843 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:42.631162+0000 mon.c (mon.2) 843 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.632413+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.632413+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:42.633138+0000 mon.a (mon.0) 3763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.859 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[29234]: audit 2024-04-26T16:36:42.633138+0000 mon.a (mon.0) 3763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.955711+0000 osd.0 (osd.0) 445 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.955711+0000 osd.0 (osd.0) 445 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.957044+0000 osd.0 (osd.0) 446 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:40.957044+0000 osd.0 (osd.0) 446 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:41.623980+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:41.623980+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.629903+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-04-26T16:36:42.860 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.629903+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.685501+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.685501+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.686914+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.686914+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:41.710871+0000 mon.c (mon.2) 842 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:41.710871+0000 mon.c (mon.2) 842 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:41.711357+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:41.711357+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.729587+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.e scrub starts 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.729587+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.e scrub starts 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.730951+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.e scrub ok 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.730951+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.e scrub ok 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.909592+0000 osd.0 (osd.0) 447 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:36:42.861 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.909592+0000 osd.0 (osd.0) 447 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.912871+0000 osd.0 (osd.0) 448 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.912871+0000 osd.0 (osd.0) 448 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:42.625941+0000 mon.a (mon.0) 3761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:42.625941+0000 mon.a (mon.0) 3761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:42.631162+0000 mon.c (mon.2) 843 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:42.631162+0000 mon.c (mon.2) 843 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.632413+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.632413+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:42.633138+0000 mon.a (mon.0) 3763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.862 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:42 smithi050 bash[21463]: audit 2024-04-26T16:36:42.633138+0000 mon.a (mon.0) 3763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.870 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:42.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:42.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:42.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.955711+0000 osd.0 (osd.0) 445 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:36:42.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.955711+0000 osd.0 (osd.0) 445 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:36:42.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.957044+0000 osd.0 (osd.0) 446 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:36:42.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:40.957044+0000 osd.0 (osd.0) 446 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:36:42.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:41.623980+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:42.986 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:41.623980+0000 mon.a (mon.0) 3758 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.629903+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.629903+0000 mon.a (mon.0) 3759 : cluster [DBG] osdmap e812: 8 total, 8 up, 8 in 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.685501+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.685501+0000 osd.5 (osd.5) 307 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.686914+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.686914+0000 osd.5 (osd.5) 308 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:41.710871+0000 mon.c (mon.2) 842 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:41.710871+0000 mon.c (mon.2) 842 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:41.711357+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:41.711357+0000 mon.a (mon.0) 3760 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.729587+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.e scrub starts 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.729587+0000 osd.2 (osd.2) 391 : cluster [DBG] 2.e scrub starts 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.730951+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.e scrub ok 2024-04-26T16:36:42.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.730951+0000 osd.2 (osd.2) 392 : cluster [DBG] 2.e scrub ok 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.909592+0000 osd.0 (osd.0) 447 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.909592+0000 osd.0 (osd.0) 447 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.912871+0000 osd.0 (osd.0) 448 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.912871+0000 osd.0 (osd.0) 448 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:42.625941+0000 mon.a (mon.0) 3761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:42.625941+0000 mon.a (mon.0) 3761 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:42.631162+0000 mon.c (mon.2) 843 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:42.631162+0000 mon.c (mon.2) 843 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.632413+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.632413+0000 mon.a (mon.0) 3762 : cluster [DBG] osdmap e813: 8 total, 8 up, 8 in 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:42.633138+0000 mon.a (mon.0) 3763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:42.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:42 smithi190 bash[23868]: audit 2024-04-26T16:36:42.633138+0000 mon.a (mon.0) 3763 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:43.000 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:43.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:43.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:43.130 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:43.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/2688216333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 msgr2=0x7f63d006bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:43.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- 172.21.15.50:0/2688216333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d006bf80 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009990 tx=0x7f63cc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/2688216333 shutdown_connections 2024-04-26T16:36:43.178 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- 172.21.15.50:0/2688216333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d006c4c0 0x7f63d0110310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- 172.21.15.50:0/2688216333 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63d01094d0 0x7f63d01098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- 172.21.15.50:0/2688216333 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d006bf80 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/2688216333 >> 172.21.15.50:0/2688216333 conn(0x7f63d0079360 msgr2=0x7f63d0079770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/2688216333 shutdown_connections 2024-04-26T16:36:43.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/2688216333 wait complete. 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 librados: starting msgr at 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 librados: starting objecter 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 Processor -- start 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- start start 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 librados: setting wanted keys 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 librados: calling monclient init 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d0102420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63d006c4c0 0x7f63d0102960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:43.181 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d01094d0 0x7f63d0102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f63d0112b10 con 0x7f63d006bb20 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f63d0112990 con 0x7f63d01094d0 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f63d0112c90 con 0x7f63d006c4c0 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d0102420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d0102420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40368/0 (socket says 172.21.15.50:40368) 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 -- 172.21.15.50:0/4016572914 learned_addr learned my addr 172.21.15.50:0/4016572914 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d54b3700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63d006c4c0 0x7f63d0102960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d64b5700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d01094d0 0x7f63d0102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 -- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63d006c4c0 msgr2=0x7f63d0102960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63d006c4c0 0x7f63d0102960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 -- 172.21.15.50:0/4016572914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d01094d0 msgr2=0x7f63d0102ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d01094d0 0x7f63d0102ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 -- 172.21.15.50:0/4016572914 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63d01a7680 con 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d5cb4700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d0102420 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f63c000e9a0 tx=0x7f63c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63c6ffd700 10 client.?.objecter ms_handle_connect 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63c6ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c000ca10 con 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f63c000cbb0 con 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63d01a7970 con 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0018420 con 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 10 client.31041.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63d01a7e80 con 0x7f63d006bb20 2024-04-26T16:36:43.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 librados: init done 2024-04-26T16:36:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.181+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63d0104b80 con 0x7f63d006bb20 2024-04-26T16:36:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f63c0010430 con 0x7f63d006bb20 2024-04-26T16:36:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f63bc071f80 0x7f63bc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63d54b3700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f63bc071f80 0x7f63bc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:43.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(813..813 src has 577..813) ==== 9947+0+0 (secure 0 0 0) 0x7f63c0091de0 con 0x7f63d006bb20 2024-04-26T16:36:43.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 10 client.31041.objecter ms_dispatch 0x7f63d0001550 osd_map(813..813 src has 577..813) 2024-04-26T16:36:43.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 3 client.31041.objecter handle_osd_map got epochs [813,813] > 0 2024-04-26T16:36:43.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 3 client.31041.objecter handle_osd_map decoding full epoch 813 2024-04-26T16:36:43.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 20 client.31041.objecter dump_active .. 0 homeless 2024-04-26T16:36:43.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63d54b3700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f63bc071f80 0x7f63bc074440 secure :-1 s=READY pgs=3835 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0098e0 tx=0x7f63cc005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:43.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.185+0000 7f63c6ffd700 10 client.31041.objecter ms_handle_connect 0x7f63bc071f80 2024-04-26T16:36:43.188 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.189+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f63c005d500 con 0x7f63d006bb20 2024-04-26T16:36:43.232 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:43.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:43.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:43.383 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:43.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:43.425 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.425+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f63d01a8050 con 0x7f63d006bb20 2024-04-26T16:36:43.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:43.514 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:43.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:43.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.629+0000 7f63c6ffd700 1 -- 172.21.15.50:0/4016572914 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v814) ==== 113+0+0 (secure 0 0 0) 0x7f63c00611b0 con 0x7f63d006bb20 2024-04-26T16:36:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 10 librados: watch_flush enter 2024-04-26T16:36:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 10 librados: watch_flush exit 2024-04-26T16:36:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 20 client.31041.objecter shutdown clearing up homeless session... 2024-04-26T16:36:43.640 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 10 client.31041.objecter successfully canceled tick 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f63bc071f80 msgr2=0x7f63bc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f63bc071f80 0x7f63bc074440 secure :-1 s=READY pgs=3835 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0098e0 tx=0x7f63cc005960 comp rx=0 tx=0).stop 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 msgr2=0x7f63d0102420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d0102420 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f63c000e9a0 tx=0x7f63c000ecb0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 shutdown_connections 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f63bc071f80 0x7f63bc074440 unknown :-1 s=CLOSED pgs=3835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63d01094d0 0x7f63d0102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f63d006bb20 0x7f63d0102420 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 --2- 172.21.15.50:0/4016572914 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f63d006c4c0 0x7f63d0102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 >> 172.21.15.50:0/4016572914 conn(0x7f63d0079360 msgr2=0x7f63d01001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 shutdown_connections 2024-04-26T16:36:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 -- 172.21.15.50:0/4016572914 wait complete. 2024-04-26T16:36:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:43.641+0000 7f63d7f18700 1 librados: shutdown 2024-04-26T16:36:43.642 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T16:36:43.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:43.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.932544+0000 mgr.y (mgr.24401) 740 : cluster [DBG] pgmap v1263: 204 pgs: 22 creating+peering, 4 unknown, 178 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:41.932544+0000 mgr.y (mgr.24401) 740 : cluster [DBG] pgmap v1263: 204 pgs: 22 creating+peering, 4 unknown, 178 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.148639+0000 osd.1 (osd.1) 421 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.148639+0000 osd.1 (osd.1) 421 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.150460+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.150460+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.178890+0000 osd.7 (osd.7) 427 : cluster [DBG] 3.d scrub starts 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.178890+0000 osd.7 (osd.7) 427 : cluster [DBG] 3.d scrub starts 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.185396+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.d scrub ok 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:42.185396+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.d scrub ok 2024-04-26T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:42.853345+0000 mon.a (mon.0) 3764 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:42.853345+0000 mon.a (mon.0) 3764 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:42.856197+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:42.856197+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.430862+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.430862+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.630410+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.630410+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.630583+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.630583+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.641807+0000 mon.c (mon.2) 845 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.641807+0000 mon.c (mon.2) 845 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.641982+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.641982+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.644050+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:43 smithi190 bash[23868]: audit 2024-04-26T16:36:43.644050+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.777 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:43.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:43.892 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.932544+0000 mgr.y (mgr.24401) 740 : cluster [DBG] pgmap v1263: 204 pgs: 22 creating+peering, 4 unknown, 178 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:41.932544+0000 mgr.y (mgr.24401) 740 : cluster [DBG] pgmap v1263: 204 pgs: 22 creating+peering, 4 unknown, 178 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.148639+0000 osd.1 (osd.1) 421 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.148639+0000 osd.1 (osd.1) 421 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.150460+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.150460+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.178890+0000 osd.7 (osd.7) 427 : cluster [DBG] 3.d scrub starts 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.178890+0000 osd.7 (osd.7) 427 : cluster [DBG] 3.d scrub starts 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.185396+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.d scrub ok 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:42.185396+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.d scrub ok 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:42.853345+0000 mon.a (mon.0) 3764 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:42.853345+0000 mon.a (mon.0) 3764 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:43.893 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:42.856197+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:42.856197+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.430862+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.430862+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.630410+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.630410+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.630583+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.630583+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.641807+0000 mon.c (mon.2) 845 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.641807+0000 mon.c (mon.2) 845 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.641982+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.641982+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.644050+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.894 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[21463]: audit 2024-04-26T16:36:43.644050+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.932544+0000 mgr.y (mgr.24401) 740 : cluster [DBG] pgmap v1263: 204 pgs: 22 creating+peering, 4 unknown, 178 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:41.932544+0000 mgr.y (mgr.24401) 740 : cluster [DBG] pgmap v1263: 204 pgs: 22 creating+peering, 4 unknown, 178 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.148639+0000 osd.1 (osd.1) 421 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.148639+0000 osd.1 (osd.1) 421 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.150460+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.150460+0000 osd.1 (osd.1) 422 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:36:43.895 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.178890+0000 osd.7 (osd.7) 427 : cluster [DBG] 3.d scrub starts 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.178890+0000 osd.7 (osd.7) 427 : cluster [DBG] 3.d scrub starts 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.185396+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.d scrub ok 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:42.185396+0000 osd.7 (osd.7) 428 : cluster [DBG] 3.d scrub ok 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:42.853345+0000 mon.a (mon.0) 3764 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:42.853345+0000 mon.a (mon.0) 3764 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:42.856197+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:42.856197+0000 mon.c (mon.2) 844 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.430862+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.430862+0000 mon.a (mon.0) 3765 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.630410+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.630410+0000 mon.a (mon.0) 3766 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.630583+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.630583+0000 mon.a (mon.0) 3767 : audit [INF] from='client.? 172.21.15.50:0/4016572914' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:36:43.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.641807+0000 mon.c (mon.2) 845 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.641807+0000 mon.c (mon.2) 845 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.641982+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-04-26T16:36:43.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.641982+0000 mon.a (mon.0) 3768 : cluster [DBG] osdmap e814: 8 total, 8 up, 8 in 2024-04-26T16:36:43.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.644050+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.897 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:43 smithi050 bash[29234]: audit 2024-04-26T16:36:43.644050+0000 mon.a (mon.0) 3769 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]: dispatch 2024-04-26T16:36:43.907 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:43.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:44.010 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.039 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:44.142 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.171 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:44.274 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:44.303 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:44.406 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.430 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:44.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:44.561 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:44.570 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set noscrub 2024-04-26T16:36:44.662 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:36:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:36:44.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.889576+0000 osd.0 (osd.0) 449 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:36:44.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.889576+0000 osd.0 (osd.0) 449 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:36:44.662 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.891840+0000 osd.0 (osd.0) 450 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.891840+0000 osd.0 (osd.0) 450 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.630860+0000 mon.a (mon.0) 3770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.630860+0000 mon.a (mon.0) 3770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: audit 2024-04-26T16:36:44.633065+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]': finished 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: audit 2024-04-26T16:36:44.633065+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]': finished 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: audit 2024-04-26T16:36:44.639656+0000 mon.c (mon.2) 846 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: audit 2024-04-26T16:36:44.639656+0000 mon.c (mon.2) 846 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.640771+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.640771+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: audit 2024-04-26T16:36:44.641987+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.663 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[21463]: audit 2024-04-26T16:36:44.641987+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.665 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.889576+0000 osd.0 (osd.0) 449 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.889576+0000 osd.0 (osd.0) 449 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.891840+0000 osd.0 (osd.0) 450 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.891840+0000 osd.0 (osd.0) 450 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.630860+0000 mon.a (mon.0) 3770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.630860+0000 mon.a (mon.0) 3770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: audit 2024-04-26T16:36:44.633065+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]': finished 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: audit 2024-04-26T16:36:44.633065+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]': finished 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: audit 2024-04-26T16:36:44.639656+0000 mon.c (mon.2) 846 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: audit 2024-04-26T16:36:44.639656+0000 mon.c (mon.2) 846 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.640771+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-04-26T16:36:44.675 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.640771+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-04-26T16:36:44.676 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: audit 2024-04-26T16:36:44.641987+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.676 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:44 smithi050 bash[29234]: audit 2024-04-26T16:36:44.641987+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.686 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:44.789 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:44.808 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:44.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.889576+0000 osd.0 (osd.0) 449 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.889576+0000 osd.0 (osd.0) 449 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.891840+0000 osd.0 (osd.0) 450 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.891840+0000 osd.0 (osd.0) 450 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.630860+0000 mon.a (mon.0) 3770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.630860+0000 mon.a (mon.0) 3770 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: audit 2024-04-26T16:36:44.633065+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]': finished 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: audit 2024-04-26T16:36:44.633065+0000 mon.a (mon.0) 3771 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-142", "mode": "writeback"}]': finished 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: audit 2024-04-26T16:36:44.639656+0000 mon.c (mon.2) 846 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: audit 2024-04-26T16:36:44.639656+0000 mon.c (mon.2) 846 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.640771+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.640771+0000 mon.a (mon.0) 3772 : cluster [DBG] osdmap e815: 8 total, 8 up, 8 in 2024-04-26T16:36:44.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: audit 2024-04-26T16:36:44.641987+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:44 smithi190 bash[23868]: audit 2024-04-26T16:36:44.641987+0000 mon.a (mon.0) 3773 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:36:44.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-26T16:36:44.934 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:44.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:45.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.063 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:45.167 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:45.188 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:45.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:45.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.317 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:45.413 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:45.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:45.445 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:45.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.576 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:45.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.705 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:45.807 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:45.819 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.933209+0000 mgr.y (mgr.24401) 741 : cluster [DBG] pgmap v1266: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:43.933209+0000 mgr.y (mgr.24401) 741 : cluster [DBG] pgmap v1266: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.581101+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.581101+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.583669+0000 osd.3 (osd.3) 460 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.583669+0000 osd.3 (osd.3) 460 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.711363+0000 osd.5 (osd.5) 309 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.711363+0000 osd.5 (osd.5) 309 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.718523+0000 osd.5 (osd.5) 310 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.718523+0000 osd.5 (osd.5) 310 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.930407+0000 osd.4 (osd.4) 391 : cluster [DBG] 323.5s0 deep-scrub starts 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.930407+0000 osd.4 (osd.4) 391 : cluster [DBG] 323.5s0 deep-scrub starts 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.936563+0000 osd.4 (osd.4) 392 : cluster [DBG] 323.5 deep-scrub ok 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:44.936563+0000 osd.4 (osd.4) 392 : cluster [DBG] 323.5 deep-scrub ok 2024-04-26T16:36:45.820 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: audit 2024-04-26T16:36:45.637852+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: audit 2024-04-26T16:36:45.637852+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.648508+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.648508+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: audit 2024-04-26T16:36:45.655792+0000 mon.c (mon.2) 847 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: audit 2024-04-26T16:36:45.655792+0000 mon.c (mon.2) 847 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: audit 2024-04-26T16:36:45.656788+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[29234]: audit 2024-04-26T16:36:45.656788+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.933209+0000 mgr.y (mgr.24401) 741 : cluster [DBG] pgmap v1266: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:43.933209+0000 mgr.y (mgr.24401) 741 : cluster [DBG] pgmap v1266: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.581101+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.581101+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:36:45.821 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.583669+0000 osd.3 (osd.3) 460 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.583669+0000 osd.3 (osd.3) 460 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.711363+0000 osd.5 (osd.5) 309 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.711363+0000 osd.5 (osd.5) 309 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.718523+0000 osd.5 (osd.5) 310 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.718523+0000 osd.5 (osd.5) 310 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.930407+0000 osd.4 (osd.4) 391 : cluster [DBG] 323.5s0 deep-scrub starts 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.930407+0000 osd.4 (osd.4) 391 : cluster [DBG] 323.5s0 deep-scrub starts 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.936563+0000 osd.4 (osd.4) 392 : cluster [DBG] 323.5 deep-scrub ok 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:44.936563+0000 osd.4 (osd.4) 392 : cluster [DBG] 323.5 deep-scrub ok 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: audit 2024-04-26T16:36:45.637852+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: audit 2024-04-26T16:36:45.637852+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.648508+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.648508+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-04-26T16:36:45.822 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: audit 2024-04-26T16:36:45.655792+0000 mon.c (mon.2) 847 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: audit 2024-04-26T16:36:45.655792+0000 mon.c (mon.2) 847 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: audit 2024-04-26T16:36:45.656788+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:45 smithi050 bash[21463]: audit 2024-04-26T16:36:45.656788+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.933209+0000 mgr.y (mgr.24401) 741 : cluster [DBG] pgmap v1266: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:43.933209+0000 mgr.y (mgr.24401) 741 : cluster [DBG] pgmap v1266: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.581101+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.581101+0000 osd.3 (osd.3) 459 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.583669+0000 osd.3 (osd.3) 460 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.583669+0000 osd.3 (osd.3) 460 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.711363+0000 osd.5 (osd.5) 309 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.711363+0000 osd.5 (osd.5) 309 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:36:45.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.718523+0000 osd.5 (osd.5) 310 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.718523+0000 osd.5 (osd.5) 310 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.930407+0000 osd.4 (osd.4) 391 : cluster [DBG] 323.5s0 deep-scrub starts 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.930407+0000 osd.4 (osd.4) 391 : cluster [DBG] 323.5s0 deep-scrub starts 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.936563+0000 osd.4 (osd.4) 392 : cluster [DBG] 323.5 deep-scrub ok 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:44.936563+0000 osd.4 (osd.4) 392 : cluster [DBG] 323.5 deep-scrub ok 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: audit 2024-04-26T16:36:45.637852+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: audit 2024-04-26T16:36:45.637852+0000 mon.a (mon.0) 3774 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.648508+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.648508+0000 mon.a (mon.0) 3775 : cluster [DBG] osdmap e816: 8 total, 8 up, 8 in 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: audit 2024-04-26T16:36:45.655792+0000 mon.c (mon.2) 847 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: audit 2024-04-26T16:36:45.655792+0000 mon.c (mon.2) 847 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: audit 2024-04-26T16:36:45.656788+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:45 smithi190 bash[23868]: audit 2024-04-26T16:36:45.656788+0000 mon.a (mon.0) 3776 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:36:45.832 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:45.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:45.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:45.961 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:45.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:46.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:46.090 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:46.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:46.221 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:46.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:46.352 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:46.456 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:46.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:46.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:46.610 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:46.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:46.740 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:46.843 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.631321+0000 osd.3 (osd.3) 461 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.631321+0000 osd.3 (osd.3) 461 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.647509+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.647509+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.737209+0000 osd.2 (osd.2) 393 : cluster [DBG] 5.a scrub starts 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.737209+0000 osd.2 (osd.2) 393 : cluster [DBG] 5.a scrub starts 2024-04-26T16:36:46.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.741580+0000 osd.2 (osd.2) 394 : cluster [DBG] 5.a scrub ok 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.741580+0000 osd.2 (osd.2) 394 : cluster [DBG] 5.a scrub ok 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.877246+0000 osd.0 (osd.0) 451 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.877246+0000 osd.0 (osd.0) 451 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.880881+0000 osd.0 (osd.0) 452 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.880881+0000 osd.0 (osd.0) 452 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.928882+0000 osd.4 (osd.4) 393 : cluster [DBG] 11.e scrub starts 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.928882+0000 osd.4 (osd.4) 393 : cluster [DBG] 11.e scrub starts 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.931377+0000 osd.4 (osd.4) 394 : cluster [DBG] 11.e scrub ok 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.931377+0000 osd.4 (osd.4) 394 : cluster [DBG] 11.e scrub ok 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: audit 2024-04-26T16:36:46.383856+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: audit 2024-04-26T16:36:46.383856+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.391467+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.391467+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: audit 2024-04-26T16:36:46.403788+0000 mon.c (mon.2) 848 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: audit 2024-04-26T16:36:46.403788+0000 mon.c (mon.2) 848 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: audit 2024-04-26T16:36:46.408136+0000 mon.a (mon.0) 3779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[29234]: audit 2024-04-26T16:36:46.408136+0000 mon.a (mon.0) 3779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.631321+0000 osd.3 (osd.3) 461 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.631321+0000 osd.3 (osd.3) 461 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.647509+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.647509+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.737209+0000 osd.2 (osd.2) 393 : cluster [DBG] 5.a scrub starts 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.737209+0000 osd.2 (osd.2) 393 : cluster [DBG] 5.a scrub starts 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.741580+0000 osd.2 (osd.2) 394 : cluster [DBG] 5.a scrub ok 2024-04-26T16:36:46.857 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.741580+0000 osd.2 (osd.2) 394 : cluster [DBG] 5.a scrub ok 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.877246+0000 osd.0 (osd.0) 451 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.877246+0000 osd.0 (osd.0) 451 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.880881+0000 osd.0 (osd.0) 452 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.880881+0000 osd.0 (osd.0) 452 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.928882+0000 osd.4 (osd.4) 393 : cluster [DBG] 11.e scrub starts 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.928882+0000 osd.4 (osd.4) 393 : cluster [DBG] 11.e scrub starts 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.931377+0000 osd.4 (osd.4) 394 : cluster [DBG] 11.e scrub ok 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.931377+0000 osd.4 (osd.4) 394 : cluster [DBG] 11.e scrub ok 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: audit 2024-04-26T16:36:46.383856+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: audit 2024-04-26T16:36:46.383856+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.391467+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.391467+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-04-26T16:36:46.858 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: audit 2024-04-26T16:36:46.403788+0000 mon.c (mon.2) 848 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: audit 2024-04-26T16:36:46.403788+0000 mon.c (mon.2) 848 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: audit 2024-04-26T16:36:46.408136+0000 mon.a (mon.0) 3779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.859 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:46 smithi050 bash[21463]: audit 2024-04-26T16:36:46.408136+0000 mon.a (mon.0) 3779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.872 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:46.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:46.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.631321+0000 osd.3 (osd.3) 461 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.631321+0000 osd.3 (osd.3) 461 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.647509+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.647509+0000 osd.3 (osd.3) 462 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.737209+0000 osd.2 (osd.2) 393 : cluster [DBG] 5.a scrub starts 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.737209+0000 osd.2 (osd.2) 393 : cluster [DBG] 5.a scrub starts 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.741580+0000 osd.2 (osd.2) 394 : cluster [DBG] 5.a scrub ok 2024-04-26T16:36:46.987 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.741580+0000 osd.2 (osd.2) 394 : cluster [DBG] 5.a scrub ok 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.877246+0000 osd.0 (osd.0) 451 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.877246+0000 osd.0 (osd.0) 451 : cluster [DBG] 2.13 deep-scrub starts 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.880881+0000 osd.0 (osd.0) 452 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.880881+0000 osd.0 (osd.0) 452 : cluster [DBG] 2.13 deep-scrub ok 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.928882+0000 osd.4 (osd.4) 393 : cluster [DBG] 11.e scrub starts 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.928882+0000 osd.4 (osd.4) 393 : cluster [DBG] 11.e scrub starts 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.931377+0000 osd.4 (osd.4) 394 : cluster [DBG] 11.e scrub ok 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.931377+0000 osd.4 (osd.4) 394 : cluster [DBG] 11.e scrub ok 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: audit 2024-04-26T16:36:46.383856+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: audit 2024-04-26T16:36:46.383856+0000 mon.a (mon.0) 3777 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.391467+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.391467+0000 mon.a (mon.0) 3778 : cluster [DBG] osdmap e817: 8 total, 8 up, 8 in 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: audit 2024-04-26T16:36:46.403788+0000 mon.c (mon.2) 848 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: audit 2024-04-26T16:36:46.403788+0000 mon.c (mon.2) 848 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: audit 2024-04-26T16:36:46.408136+0000 mon.a (mon.0) 3779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:46.988 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:46 smithi190 bash[23868]: audit 2024-04-26T16:36:46.408136+0000 mon.a (mon.0) 3779 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:36:47.003 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:47.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:47.134 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:47.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:47.263 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:47.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:47.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:47.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:47.440 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:47.498 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:47.519 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:47.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:36:47.622 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:47.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:47.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:47.782 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.933882+0000 mgr.y (mgr.24401) 742 : cluster [DBG] pgmap v1269: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:45.933882+0000 mgr.y (mgr.24401) 742 : cluster [DBG] pgmap v1269: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.230155+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.230155+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.231622+0000 osd.7 (osd.7) 430 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.231622+0000 osd.7 (osd.7) 430 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.666943+0000 osd.3 (osd.3) 463 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.666943+0000 osd.3 (osd.3) 463 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.673164+0000 osd.3 (osd.3) 464 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.673164+0000 osd.3 (osd.3) 464 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.690865+0000 osd.2 (osd.2) 395 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.690865+0000 osd.2 (osd.2) 395 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:36:47.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.698065+0000 osd.2 (osd.2) 396 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:46.698065+0000 osd.2 (osd.2) 396 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.395011+0000 mon.a (mon.0) 3780 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.395011+0000 mon.a (mon.0) 3780 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: audit 2024-04-26T16:36:47.397230+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: audit 2024-04-26T16:36:47.397230+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.405511+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.405511+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: audit 2024-04-26T16:36:47.408709+0000 mon.c (mon.2) 849 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: audit 2024-04-26T16:36:47.408709+0000 mon.c (mon.2) 849 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: audit 2024-04-26T16:36:47.410217+0000 mon.a (mon.0) 3783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[29234]: audit 2024-04-26T16:36:47.410217+0000 mon.a (mon.0) 3783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.933882+0000 mgr.y (mgr.24401) 742 : cluster [DBG] pgmap v1269: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:45.933882+0000 mgr.y (mgr.24401) 742 : cluster [DBG] pgmap v1269: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.230155+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.230155+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.231622+0000 osd.7 (osd.7) 430 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.231622+0000 osd.7 (osd.7) 430 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.666943+0000 osd.3 (osd.3) 463 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.666943+0000 osd.3 (osd.3) 463 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.673164+0000 osd.3 (osd.3) 464 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.673164+0000 osd.3 (osd.3) 464 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.690865+0000 osd.2 (osd.2) 395 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.690865+0000 osd.2 (osd.2) 395 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.698065+0000 osd.2 (osd.2) 396 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:36:47.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:46.698065+0000 osd.2 (osd.2) 396 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.395011+0000 mon.a (mon.0) 3780 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.395011+0000 mon.a (mon.0) 3780 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: audit 2024-04-26T16:36:47.397230+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: audit 2024-04-26T16:36:47.397230+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.405511+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.405511+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: audit 2024-04-26T16:36:47.408709+0000 mon.c (mon.2) 849 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: audit 2024-04-26T16:36:47.408709+0000 mon.c (mon.2) 849 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: audit 2024-04-26T16:36:47.410217+0000 mon.a (mon.0) 3783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:47 smithi050 bash[21463]: audit 2024-04-26T16:36:47.410217+0000 mon.a (mon.0) 3783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:47.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:47.913 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:47.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:48.016 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:48.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.933882+0000 mgr.y (mgr.24401) 742 : cluster [DBG] pgmap v1269: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:45.933882+0000 mgr.y (mgr.24401) 742 : cluster [DBG] pgmap v1269: 204 pgs: 22 creating+peering, 182 active+clean; 4.4 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.230155+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.230155+0000 osd.7 (osd.7) 429 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.231622+0000 osd.7 (osd.7) 430 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.231622+0000 osd.7 (osd.7) 430 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:36:48.096 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.666943+0000 osd.3 (osd.3) 463 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.666943+0000 osd.3 (osd.3) 463 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.673164+0000 osd.3 (osd.3) 464 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.673164+0000 osd.3 (osd.3) 464 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.690865+0000 osd.2 (osd.2) 395 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.690865+0000 osd.2 (osd.2) 395 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.698065+0000 osd.2 (osd.2) 396 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:46.698065+0000 osd.2 (osd.2) 396 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.395011+0000 mon.a (mon.0) 3780 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.395011+0000 mon.a (mon.0) 3780 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: audit 2024-04-26T16:36:47.397230+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: audit 2024-04-26T16:36:47.397230+0000 mon.a (mon.0) 3781 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.405511+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-04-26T16:36:48.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.405511+0000 mon.a (mon.0) 3782 : cluster [DBG] osdmap e818: 8 total, 8 up, 8 in 2024-04-26T16:36:48.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: audit 2024-04-26T16:36:47.408709+0000 mon.c (mon.2) 849 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:48.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: audit 2024-04-26T16:36:47.408709+0000 mon.c (mon.2) 849 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:48.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: audit 2024-04-26T16:36:47.410217+0000 mon.a (mon.0) 3783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:48.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:47 smithi190 bash[23868]: audit 2024-04-26T16:36:47.410217+0000 mon.a (mon.0) 3783 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:36:48.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:48.199 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:48.228 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:48.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:48.358 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:48.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:48.489 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:48.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:48.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:48.696 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.134759+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:36:48.696 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.134759+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:36:48.696 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.136172+0000 osd.1 (osd.1) 424 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.136172+0000 osd.1 (osd.1) 424 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:47.297892+0000 mgr.y (mgr.24401) 743 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:47.297892+0000 mgr.y (mgr.24401) 743 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.445022+0000 osd.6 (osd.6) 17 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.445022+0000 osd.6 (osd.6) 17 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.447289+0000 osd.6 (osd.6) 18 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.447289+0000 osd.6 (osd.6) 18 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.690247+0000 osd.3 (osd.3) 465 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.690247+0000 osd.3 (osd.3) 465 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.693886+0000 osd.3 (osd.3) 466 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.693886+0000 osd.3 (osd.3) 466 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.891845+0000 osd.4 (osd.4) 395 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.891845+0000 osd.4 (osd.4) 395 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.893717+0000 osd.4 (osd.4) 396 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:36:48.697 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.893717+0000 osd.4 (osd.4) 396 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:48.399056+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:48.399056+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.402693+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.402693+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:48.411347+0000 mon.c (mon.2) 850 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:48.411347+0000 mon.c (mon.2) 850 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:48.411917+0000 mon.a (mon.0) 3786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.698 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:48 smithi190 bash[23868]: audit 2024-04-26T16:36:48.411917+0000 mon.a (mon.0) 3786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:48.749 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:48.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:48.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:48.878 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:48.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:48.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.134759+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:36:48.991 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.134759+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.136172+0000 osd.1 (osd.1) 424 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.136172+0000 osd.1 (osd.1) 424 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:47.297892+0000 mgr.y (mgr.24401) 743 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:47.297892+0000 mgr.y (mgr.24401) 743 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.445022+0000 osd.6 (osd.6) 17 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.445022+0000 osd.6 (osd.6) 17 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.447289+0000 osd.6 (osd.6) 18 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.447289+0000 osd.6 (osd.6) 18 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.690247+0000 osd.3 (osd.3) 465 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.690247+0000 osd.3 (osd.3) 465 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.693886+0000 osd.3 (osd.3) 466 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.693886+0000 osd.3 (osd.3) 466 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:36:48.992 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.891845+0000 osd.4 (osd.4) 395 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.891845+0000 osd.4 (osd.4) 395 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.893717+0000 osd.4 (osd.4) 396 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.893717+0000 osd.4 (osd.4) 396 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:48.399056+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:48.399056+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.402693+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.402693+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:48.411347+0000 mon.c (mon.2) 850 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:48.411347+0000 mon.c (mon.2) 850 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:48.411917+0000 mon.a (mon.0) 3786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.993 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[21463]: audit 2024-04-26T16:36:48.411917+0000 mon.a (mon.0) 3786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.134759+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.134759+0000 osd.1 (osd.1) 423 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.136172+0000 osd.1 (osd.1) 424 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.136172+0000 osd.1 (osd.1) 424 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:47.297892+0000 mgr.y (mgr.24401) 743 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:47.297892+0000 mgr.y (mgr.24401) 743 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.445022+0000 osd.6 (osd.6) 17 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.445022+0000 osd.6 (osd.6) 17 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.447289+0000 osd.6 (osd.6) 18 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.447289+0000 osd.6 (osd.6) 18 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.690247+0000 osd.3 (osd.3) 465 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.690247+0000 osd.3 (osd.3) 465 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:36:48.994 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.693886+0000 osd.3 (osd.3) 466 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.693886+0000 osd.3 (osd.3) 466 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.891845+0000 osd.4 (osd.4) 395 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.891845+0000 osd.4 (osd.4) 395 : cluster [DBG] 6.e deep-scrub starts 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.893717+0000 osd.4 (osd.4) 396 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.893717+0000 osd.4 (osd.4) 396 : cluster [DBG] 6.e deep-scrub ok 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:48.399056+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:48.399056+0000 mon.a (mon.0) 3784 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.402693+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.402693+0000 mon.a (mon.0) 3785 : cluster [DBG] osdmap e819: 8 total, 8 up, 8 in 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:48.411347+0000 mon.c (mon.2) 850 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:48.411347+0000 mon.c (mon.2) 850 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:48.411917+0000 mon.a (mon.0) 3786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:48.995 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:48 smithi050 bash[29234]: audit 2024-04-26T16:36:48.411917+0000 mon.a (mon.0) 3786 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-04-26T16:36:49.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:49.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:49.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:49.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:49.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:49.264 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:49.367 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:49.395 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:36:49.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:49.528 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:49.632 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:49.660 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:49.763 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:49.791 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:49.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.934679+0000 mgr.y (mgr.24401) 744 : cluster [DBG] pgmap v1272: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:49.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:47.934679+0000 mgr.y (mgr.24401) 744 : cluster [DBG] pgmap v1272: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:49.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.688544+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-04-26T16:36:49.793 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.688544+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.691409+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.691409+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.919593+0000 osd.4 (osd.4) 397 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.919593+0000 osd.4 (osd.4) 397 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.921087+0000 osd.4 (osd.4) 398 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.921087+0000 osd.4 (osd.4) 398 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.921897+0000 osd.0 (osd.0) 453 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.921897+0000 osd.0 (osd.0) 453 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.923522+0000 osd.0 (osd.0) 454 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:48.923522+0000 osd.0 (osd.0) 454 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: audit 2024-04-26T16:36:49.401221+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:36:49.794 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: audit 2024-04-26T16:36:49.401221+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: audit 2024-04-26T16:36:49.407223+0000 mon.c (mon.2) 851 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: audit 2024-04-26T16:36:49.407223+0000 mon.c (mon.2) 851 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.408413+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.408413+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: audit 2024-04-26T16:36:49.409675+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[21463]: audit 2024-04-26T16:36:49.409675+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.934679+0000 mgr.y (mgr.24401) 744 : cluster [DBG] pgmap v1272: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:47.934679+0000 mgr.y (mgr.24401) 744 : cluster [DBG] pgmap v1272: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.688544+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-04-26T16:36:49.795 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.688544+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.691409+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.691409+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.919593+0000 osd.4 (osd.4) 397 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.919593+0000 osd.4 (osd.4) 397 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.921087+0000 osd.4 (osd.4) 398 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.921087+0000 osd.4 (osd.4) 398 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.921897+0000 osd.0 (osd.0) 453 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.921897+0000 osd.0 (osd.0) 453 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.923522+0000 osd.0 (osd.0) 454 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:48.923522+0000 osd.0 (osd.0) 454 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: audit 2024-04-26T16:36:49.401221+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: audit 2024-04-26T16:36:49.401221+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: audit 2024-04-26T16:36:49.407223+0000 mon.c (mon.2) 851 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.796 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: audit 2024-04-26T16:36:49.407223+0000 mon.c (mon.2) 851 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.408413+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-04-26T16:36:49.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.408413+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-04-26T16:36:49.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: audit 2024-04-26T16:36:49.409675+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.797 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:49 smithi050 bash[29234]: audit 2024-04-26T16:36:49.409675+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.934679+0000 mgr.y (mgr.24401) 744 : cluster [DBG] pgmap v1272: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:47.934679+0000 mgr.y (mgr.24401) 744 : cluster [DBG] pgmap v1272: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.688544+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.688544+0000 osd.2 (osd.2) 397 : cluster [DBG] 6.f scrub starts 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.691409+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.691409+0000 osd.2 (osd.2) 398 : cluster [DBG] 6.f scrub ok 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.919593+0000 osd.4 (osd.4) 397 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.919593+0000 osd.4 (osd.4) 397 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:36:49.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.921087+0000 osd.4 (osd.4) 398 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.921087+0000 osd.4 (osd.4) 398 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.921897+0000 osd.0 (osd.0) 453 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.921897+0000 osd.0 (osd.0) 453 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.923522+0000 osd.0 (osd.0) 454 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:48.923522+0000 osd.0 (osd.0) 454 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: audit 2024-04-26T16:36:49.401221+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: audit 2024-04-26T16:36:49.401221+0000 mon.a (mon.0) 3787 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: audit 2024-04-26T16:36:49.407223+0000 mon.c (mon.2) 851 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: audit 2024-04-26T16:36:49.407223+0000 mon.c (mon.2) 851 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.408413+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.408413+0000 mon.a (mon.0) 3788 : cluster [DBG] osdmap e820: 8 total, 8 up, 8 in 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: audit 2024-04-26T16:36:49.409675+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:49 smithi190 bash[23868]: audit 2024-04-26T16:36:49.409675+0000 mon.a (mon.0) 3789 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-04-26T16:36:49.899 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:49.927 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:49.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:49.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:50.031 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:50.058 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:50.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:50.189 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:50.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:50.292 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:50.321 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:50.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:50.452 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:50.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:50.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:50.582 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:50.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:50.711 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:50.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:50.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:50.841 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:50.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:50.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.188376+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:36:50.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.188376+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:36:50.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.190076+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:36:50.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.190076+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:36:50.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.732002+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:36:50.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.732002+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.734266+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.734266+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: audit 2024-04-26T16:36:50.404180+0000 mon.a (mon.0) 3790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: audit 2024-04-26T16:36:50.404180+0000 mon.a (mon.0) 3790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.412949+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.412949+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: audit 2024-04-26T16:36:50.482756+0000 mon.c (mon.2) 852 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: audit 2024-04-26T16:36:50.482756+0000 mon.c (mon.2) 852 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: audit 2024-04-26T16:36:50.483661+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.956 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:50 smithi190 bash[23868]: audit 2024-04-26T16:36:50.483661+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.970 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:50.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:50.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.188376+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.188376+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.190076+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.190076+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.732002+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.732002+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.734266+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.734266+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: audit 2024-04-26T16:36:50.404180+0000 mon.a (mon.0) 3790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: audit 2024-04-26T16:36:50.404180+0000 mon.a (mon.0) 3790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.412949+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-04-26T16:36:50.976 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.412949+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: audit 2024-04-26T16:36:50.482756+0000 mon.c (mon.2) 852 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: audit 2024-04-26T16:36:50.482756+0000 mon.c (mon.2) 852 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: audit 2024-04-26T16:36:50.483661+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[29234]: audit 2024-04-26T16:36:50.483661+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.188376+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.188376+0000 osd.1 (osd.1) 425 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.190076+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.190076+0000 osd.1 (osd.1) 426 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:36:50.977 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.732002+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.732002+0000 osd.5 (osd.5) 311 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.734266+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.734266+0000 osd.5 (osd.5) 312 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: audit 2024-04-26T16:36:50.404180+0000 mon.a (mon.0) 3790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: audit 2024-04-26T16:36:50.404180+0000 mon.a (mon.0) 3790 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.412949+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.412949+0000 mon.a (mon.0) 3791 : cluster [DBG] osdmap e821: 8 total, 8 up, 8 in 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: audit 2024-04-26T16:36:50.482756+0000 mon.c (mon.2) 852 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: audit 2024-04-26T16:36:50.482756+0000 mon.c (mon.2) 852 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: audit 2024-04-26T16:36:50.483661+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:50.978 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:50 smithi050 bash[21463]: audit 2024-04-26T16:36:50.483661+0000 mon.a (mon.0) 3792 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:51.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:51.101 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:51.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:51.232 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:51.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:51.362 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:51.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:51.490 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:51.595 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:51.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:51.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.935314+0000 mgr.y (mgr.24401) 745 : cluster [DBG] pgmap v1275: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:49.935314+0000 mgr.y (mgr.24401) 745 : cluster [DBG] pgmap v1275: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.249050+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.b scrub starts 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.249050+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.b scrub starts 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.250248+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.b scrub ok 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.250248+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.b scrub ok 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.940869+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:36:51.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.940869+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: audit 2024-04-26T16:36:51.406012+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: audit 2024-04-26T16:36:51.406012+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: audit 2024-04-26T16:36:51.411670+0000 mon.c (mon.2) 853 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: audit 2024-04-26T16:36:51.411670+0000 mon.c (mon.2) 853 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.414225+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.414225+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: audit 2024-04-26T16:36:51.416179+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.731 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:51 smithi190 bash[23868]: audit 2024-04-26T16:36:51.416179+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.746 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:51.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:51.849 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:51.877 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:51.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.935314+0000 mgr.y (mgr.24401) 745 : cluster [DBG] pgmap v1275: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:49.935314+0000 mgr.y (mgr.24401) 745 : cluster [DBG] pgmap v1275: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.249050+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.b scrub starts 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.249050+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.b scrub starts 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.250248+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.b scrub ok 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.250248+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.b scrub ok 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.940869+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.940869+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: audit 2024-04-26T16:36:51.406012+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: audit 2024-04-26T16:36:51.406012+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:51.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: audit 2024-04-26T16:36:51.411670+0000 mon.c (mon.2) 853 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: audit 2024-04-26T16:36:51.411670+0000 mon.c (mon.2) 853 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.414225+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.414225+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: audit 2024-04-26T16:36:51.416179+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[29234]: audit 2024-04-26T16:36:51.416179+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.935314+0000 mgr.y (mgr.24401) 745 : cluster [DBG] pgmap v1275: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:49.935314+0000 mgr.y (mgr.24401) 745 : cluster [DBG] pgmap v1275: 204 pgs: 204 active+clean; 4.4 MiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.249050+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.b scrub starts 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.249050+0000 osd.7 (osd.7) 431 : cluster [DBG] 2.b scrub starts 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.250248+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.b scrub ok 2024-04-26T16:36:51.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.250248+0000 osd.7 (osd.7) 432 : cluster [DBG] 2.b scrub ok 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.940869+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.940869+0000 osd.0 (osd.0) 455 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: audit 2024-04-26T16:36:51.406012+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: audit 2024-04-26T16:36:51.406012+0000 mon.a (mon.0) 3793 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: audit 2024-04-26T16:36:51.411670+0000 mon.c (mon.2) 853 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: audit 2024-04-26T16:36:51.411670+0000 mon.c (mon.2) 853 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.414225+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.414225+0000 mon.a (mon.0) 3794 : cluster [DBG] osdmap e822: 8 total, 8 up, 8 in 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: audit 2024-04-26T16:36:51.416179+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:51 smithi050 bash[21463]: audit 2024-04-26T16:36:51.416179+0000 mon.a (mon.0) 3795 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:51.980 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:52.005 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:52.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:52.133 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:52.236 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:52.263 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:52.365 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:52.392 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:52.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:52.521 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:52.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:52.650 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:52.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:52.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.944506+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:36:52.764 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:50.944506+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.890577+0000 osd.4 (osd.4) 399 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.890577+0000 osd.4 (osd.4) 399 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.894399+0000 osd.4 (osd.4) 400 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.894399+0000 osd.4 (osd.4) 400 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.408042+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.408042+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.412145+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.412145+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.462057+0000 mon.c (mon.2) 854 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.462057+0000 mon.c (mon.2) 854 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.462546+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.462546+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.463423+0000 mon.c (mon.2) 855 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.463423+0000 mon.c (mon.2) 855 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:52.765 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.463861+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:52.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:52 smithi190 bash[23868]: audit 2024-04-26T16:36:52.463861+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:52.781 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:52.884 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:52.913 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:52.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:53.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:53.042 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.944506+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:36:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:50.944506+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:36:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.890577+0000 osd.4 (osd.4) 399 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.890577+0000 osd.4 (osd.4) 399 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.894399+0000 osd.4 (osd.4) 400 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.894399+0000 osd.4 (osd.4) 400 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.408042+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.408042+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.412145+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.412145+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.462057+0000 mon.c (mon.2) 854 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.462057+0000 mon.c (mon.2) 854 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.462546+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.462546+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.463423+0000 mon.c (mon.2) 855 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.463423+0000 mon.c (mon.2) 855 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.463861+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[21463]: audit 2024-04-26T16:36:52.463861+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.944506+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:50.944506+0000 osd.0 (osd.0) 456 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.890577+0000 osd.4 (osd.4) 399 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.890577+0000 osd.4 (osd.4) 399 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.894399+0000 osd.4 (osd.4) 400 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.894399+0000 osd.4 (osd.4) 400 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.408042+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.408042+0000 mon.a (mon.0) 3796 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]': finished 2024-04-26T16:36:53.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.412145+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.412145+0000 mon.a (mon.0) 3797 : cluster [DBG] osdmap e823: 8 total, 8 up, 8 in 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.462057+0000 mon.c (mon.2) 854 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.462057+0000 mon.c (mon.2) 854 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.462546+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.462546+0000 mon.a (mon.0) 3798 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.463423+0000 mon.c (mon.2) 855 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.463423+0000 mon.c (mon.2) 855 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.463861+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:52 smithi050 bash[29234]: audit 2024-04-26T16:36:52.463861+0000 mon.a (mon.0) 3799 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-142"}]: dispatch 2024-04-26T16:36:53.146 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:53.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:53.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:53.308 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:53.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:53.440 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:53.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:53.569 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:53.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:53.672 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:53.703 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:53.806 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:53.836 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:53.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:53.949 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.937175+0000 mgr.y (mgr.24401) 746 : cluster [DBG] pgmap v1278: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:51.937175+0000 mgr.y (mgr.24401) 746 : cluster [DBG] pgmap v1278: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.174987+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.174987+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.176509+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.176509+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.584404+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.584404+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.586684+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:52.586684+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.422780+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.422780+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.544334+0000 osd.2 (osd.2) 401 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:36:53.950 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.544334+0000 osd.2 (osd.2) 401 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.546489+0000 osd.2 (osd.2) 402 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.546489+0000 osd.2 (osd.2) 402 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.937175+0000 mgr.y (mgr.24401) 746 : cluster [DBG] pgmap v1278: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:51.937175+0000 mgr.y (mgr.24401) 746 : cluster [DBG] pgmap v1278: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.174987+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.174987+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.176509+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.176509+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.584404+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.584404+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.586684+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:52.586684+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:36:53.951 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.422780+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-04-26T16:36:53.952 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.422780+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-04-26T16:36:53.952 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.544334+0000 osd.2 (osd.2) 401 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:36:53.952 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.544334+0000 osd.2 (osd.2) 401 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:36:53.952 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.546489+0000 osd.2 (osd.2) 402 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:36:53.952 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:53 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.546489+0000 osd.2 (osd.2) 402 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:36:53.963 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:53.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:54.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.937175+0000 mgr.y (mgr.24401) 746 : cluster [DBG] pgmap v1278: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:51.937175+0000 mgr.y (mgr.24401) 746 : cluster [DBG] pgmap v1278: 204 pgs: 204 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.174987+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.174987+0000 osd.1 (osd.1) 427 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.176509+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.176509+0000 osd.1 (osd.1) 428 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.584404+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.584404+0000 osd.2 (osd.2) 399 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.586684+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:52.586684+0000 osd.2 (osd.2) 400 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:36:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.422780+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-04-26T16:36:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.422780+0000 mon.a (mon.0) 3800 : cluster [DBG] osdmap e824: 8 total, 8 up, 8 in 2024-04-26T16:36:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.544334+0000 osd.2 (osd.2) 401 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:36:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.544334+0000 osd.2 (osd.2) 401 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:36:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.546489+0000 osd.2 (osd.2) 402 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:36:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:53 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.546489+0000 osd.2 (osd.2) 402 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:36:54.093 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:54.196 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:54.223 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:54.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:54.354 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:54.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:54.477 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:54.534 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:36:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:36:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/2884543466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4101ae0 msgr2=0x7f66b4101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/2884543466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4101ae0 0x7f66b4101f60 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009990 tx=0x7f66a8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/2884543466 shutdown_connections 2024-04-26T16:36:54.559 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/2884543466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66b4107a90 0x7f66b4107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/2884543466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b41024a0 0x7f66b410e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/2884543466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4101ae0 0x7f66b4101f60 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/2884543466 >> 172.21.15.50:0/2884543466 conn(0x7f66b4078990 msgr2=0x7f66b40ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/2884543466 shutdown_connections 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/2884543466 wait complete. 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 librados: starting msgr at 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 librados: starting objecter 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 Processor -- start 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- start start 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 librados: setting wanted keys 2024-04-26T16:36:54.560 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 librados: calling monclient init 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b4101ae0 0x7f66b40fde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66b41024a0 0x7f66b40fe340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4107a90 0x7f66b40fe880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f66b4110970 con 0x7f66b4101ae0 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f66b41107f0 con 0x7f66b41024a0 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f66b4110af0 con 0x7f66b4107a90 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b3fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4107a90 0x7f66b40fe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b4101ae0 0x7f66b40fde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b3fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4107a90 0x7f66b40fe880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43152/0 (socket says 172.21.15.50:43152) 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b3fff700 1 -- 172.21.15.50:0/1553211827 learned_addr learned my addr 172.21.15.50:0/1553211827 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66b41024a0 0x7f66b40fe340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.562 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4107a90 msgr2=0x7f66b40fe880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4107a90 0x7f66b40fe880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66b41024a0 msgr2=0x7f66b40fe340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66b41024a0 0x7f66b40fe340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b41a7670 con 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f66b4107a90 0x7f66b40fe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66b41024a0 0x7f66b40fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b4101ae0 0x7f66b40fde00 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f66a400d750 tx=0x7f66a400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b0ff9700 10 client.?.objecter ms_handle_connect 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b0ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4017070 con 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f66a4005020 con 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b41a7960 con 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a400f8f0 con 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 10 client.31047.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b41a7e50 con 0x7f66b4101ae0 2024-04-26T16:36:54.563 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.561+0000 7f66b9f0a700 1 librados: init done 2024-04-26T16:36:54.564 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f66b41031d0 con 0x7f66b4101ae0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f66a4004a30 con 0x7f66b4101ae0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f669c071f80 0x7f669c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(825..825 src has 577..825) ==== 9939+0+0 (secure 0 0 0) 0x7f66a40916f0 con 0x7f66b4101ae0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 osd_map(825..825 src has 577..825) 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 3 client.31047.objecter handle_osd_map got epochs [825,825] > 0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 3 client.31047.objecter handle_osd_map decoding full epoch 825 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f669c071f80 0x7f669c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 20 client.31047.objecter dump_active .. 0 homeless 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v825) ==== 54+0+16 (secure 0 0 0) 0x7f66a4091b40 con 0x7f66b4101ae0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f669c071f80 0x7f669c074440 secure :-1 s=READY pgs=3836 cs=0 l=1 rev1=1 crypto rx=0x7f66a800b2e0 tx=0x7f66a801a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 1 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f669c071f80 2024-04-26T16:36:54.565 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f66b4064280 0x7f66b410a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 1 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 0 1 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 4 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 10 client.31047.objecter _send_command 1 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66b406a110 con 0x7f66b4064280 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f66b4064280 0x7f66b410a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f66b4064280 0x7f66b410a900 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b4064280 2024-04-26T16:36:54.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b406a110 con 0x7f66b4064280 2024-04-26T16:36:54.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 1: 0 ) 2024-04-26T16:36:54.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 1 = system:0 2024-04-26T16:36:54.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 4 2024-04-26T16:36:54.567 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.565+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 0 1 2024-04-26T16:36:54.580 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 2 2024-04-26T16:36:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.583 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 2 2024-04-26T16:36:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b40ff330 osd=0 3 2024-04-26T16:36:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 0 2 2024-04-26T16:36:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 4 2024-04-26T16:36:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 10 client.31047.objecter _send_command 2 2024-04-26T16:36:54.584 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b410ae40 con 0x7f66b4064280 2024-04-26T16:36:54.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b410ae40 con 0x7f66b4064280 2024-04-26T16:36:54.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 4 2024-04-26T16:36:54.585 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 0 2 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 3 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f66b4063e70 0x7f66b41ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.586 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 3 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 1 3 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 4 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 10 client.31047.objecter _send_command 3 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f66b402dff0 con 0x7f66b4063e70 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.585+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f66b4063e70 0x7f66b41ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b4063e70 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f66b4063e70 0x7f66b41ac100 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.587 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b402dff0 con 0x7f66b4063e70 2024-04-26T16:36:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 3: 0 ) 2024-04-26T16:36:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 3 = system:0 2024-04-26T16:36:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 4 2024-04-26T16:36:54.588 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.589+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 1 3 2024-04-26T16:36:54.599 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 4 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 4 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b4063d00 osd=1 3 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 1 4 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 4 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 10 client.31047.objecter _send_command 4 2024-04-26T16:36:54.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.605+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b41ac640 con 0x7f66b4063e70 2024-04-26T16:36:54.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b41ac640 con 0x7f66b4063e70 2024-04-26T16:36:54.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 4 2024-04-26T16:36:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 1 4 2024-04-26T16:36:54.607 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:36:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.607 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 5 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f66b402d6d0 0x7f66b41af980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 5 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 2 5 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 4 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 10 client.31047.objecter _send_command 5 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f66b410bd60 con 0x7f66b402d6d0 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f66b402d6d0 0x7f66b41af980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b402d6d0 2024-04-26T16:36:54.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f66b402d6d0 0x7f66b41af980 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b410bd60 con 0x7f66b402d6d0 2024-04-26T16:36:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 5: 0 ) 2024-04-26T16:36:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 5 = system:0 2024-04-26T16:36:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 4 2024-04-26T16:36:54.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.609+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 2 5 2024-04-26T16:36:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.625+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 6 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 6 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b402d560 osd=2 3 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 2 6 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 4 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 10 client.31047.objecter _send_command 6 2024-04-26T16:36:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b410b240 con 0x7f66b402d6d0 2024-04-26T16:36:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b410b240 con 0x7f66b402d6d0 2024-04-26T16:36:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 4 2024-04-26T16:36:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 2 6 2024-04-26T16:36:54.628 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:36:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 7 2024-04-26T16:36:54.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f66b41ad4d0 0x7f66b41b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 7 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 3 7 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 4 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 10 client.31047.objecter _send_command 7 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f66b41b5fc0 con 0x7f66b41ad4d0 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f66b41ad4d0 0x7f66b41b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b41ad4d0 2024-04-26T16:36:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f66b41ad4d0 0x7f66b41b3050 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b41b5fc0 con 0x7f66b41ad4d0 2024-04-26T16:36:54.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 7: 0 ) 2024-04-26T16:36:54.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 7 = system:0 2024-04-26T16:36:54.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 4 2024-04-26T16:36:54.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.629+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 3 7 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 8 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 8 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b410b640 osd=3 3 2024-04-26T16:36:54.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 3 8 2024-04-26T16:36:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 4 2024-04-26T16:36:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 10 client.31047.objecter _send_command 8 2024-04-26T16:36:54.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b41aca40 con 0x7f66b41ad4d0 2024-04-26T16:36:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b41aca40 con 0x7f66b41ad4d0 2024-04-26T16:36:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 4 2024-04-26T16:36:54.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 3 8 2024-04-26T16:36:54.650 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:36:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 9 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f66b41b0ba0 0x7f66b41b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 9 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 4 9 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 4 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 10 client.31047.objecter _send_command 9 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f66b41b99d0 con 0x7f66b41b0ba0 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.649+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f66b41b0ba0 0x7f66b41b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b41b0ba0 2024-04-26T16:36:54.651 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f66b41b0ba0 0x7f66b41b35e0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b41b99d0 con 0x7f66b41b0ba0 2024-04-26T16:36:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 9: 0 ) 2024-04-26T16:36:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 9 = system:0 2024-04-26T16:36:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 4 2024-04-26T16:36:54.652 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.653+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 4 9 2024-04-26T16:36:54.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 10 2024-04-26T16:36:54.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 10 2024-04-26T16:36:54.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41aca40 osd=4 3 2024-04-26T16:36:54.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 4 10 2024-04-26T16:36:54.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 4 2024-04-26T16:36:54.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 10 client.31047.objecter _send_command 10 2024-04-26T16:36:54.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.669+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b41b3b20 con 0x7f66b41b0ba0 2024-04-26T16:36:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b41b3b20 con 0x7f66b41b0ba0 2024-04-26T16:36:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 4 2024-04-26T16:36:54.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 4 10 2024-04-26T16:36:54.674 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:36:54.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 11 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.673+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f66b41b4290 0x7f66b41b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 11 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 5 11 2024-04-26T16:36:54.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 4 2024-04-26T16:36:54.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 10 client.31047.objecter _send_command 11 2024-04-26T16:36:54.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f66b41bcc70 con 0x7f66b41b4290 2024-04-26T16:36:54.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f66b41b4290 0x7f66b41b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b2ffd700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f66b41b4290 0x7f66b41b46d0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b41b4290 2024-04-26T16:36:54.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b41bcc70 con 0x7f66b41b4290 2024-04-26T16:36:54.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 11: 0 ) 2024-04-26T16:36:54.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 11 = system:0 2024-04-26T16:36:54.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 4 2024-04-26T16:36:54.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.677+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 5 11 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 12 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 12 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41b3b20 osd=5 3 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 5 12 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 4 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 10 client.31047.objecter _send_command 12 2024-04-26T16:36:54.695 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.697+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b41bba60 con 0x7f66b41b4290 2024-04-26T16:36:54.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b41bba60 con 0x7f66b41b4290 2024-04-26T16:36:54.698 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 4 2024-04-26T16:36:54.699 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 5 12 2024-04-26T16:36:54.699 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 13 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f66b41bc180 0x7f66b41bc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.700 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 13 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 6 13 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 4 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 10 client.31047.objecter _send_command 13 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f66b41bffe0 con 0x7f66b41bc180 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f66b41bc180 0x7f66b41bc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b41bc180 2024-04-26T16:36:54.701 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.701+0000 7f66b3fff700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f66b41bc180 0x7f66b41bc5c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:54.702 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.705+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b41bffe0 con 0x7f66b41bc180 2024-04-26T16:36:54.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.705+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 13: 0 ) 2024-04-26T16:36:54.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.705+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 13 = system:0 2024-04-26T16:36:54.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.705+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 4 2024-04-26T16:36:54.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.705+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 6 13 2024-04-26T16:36:54.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 14 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 14 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41bba60 osd=6 3 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 6 14 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 4 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 10 client.31047.objecter _send_command 14 2024-04-26T16:36:54.720 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.721+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b41ba290 con 0x7f66b41bc180 2024-04-26T16:36:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b41ba290 con 0x7f66b41bc180 2024-04-26T16:36:54.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 4 2024-04-26T16:36:54.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 6 14 2024-04-26T16:36:54.723 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 15 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f66b41ba7a0 0x7f66b41babe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 15 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 7 15 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 4 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 10 client.31047.objecter _send_command 15 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f66b41bb120 con 0x7f66b41ba7a0 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f66b41ba7a0 0x7f66b41babe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b37fe700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f66b41ba7a0 0x7f66b41babe0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 10 client.31047.objecter ms_handle_connect 0x7f66b41ba7a0 2024-04-26T16:36:54.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f66b41bb120 con 0x7f66b41ba7a0 2024-04-26T16:36:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 15: 0 ) 2024-04-26T16:36:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 15 = system:0 2024-04-26T16:36:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 4 2024-04-26T16:36:54.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.725+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 7 15 2024-04-26T16:36:54.730 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 10 client.31047.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign -1 16 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 20 client.31047.objecter _get_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_remove -1 16 2024-04-26T16:36:54.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 20 client.31047.objecter get_session s=0x7f66b41bb610 osd=7 3 2024-04-26T16:36:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 15 client.31047.objecter _session_command_op_assign 7 16 2024-04-26T16:36:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 4 2024-04-26T16:36:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 10 client.31047.objecter _send_command 16 2024-04-26T16:36:54.746 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.745+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7f66b41bb120 con 0x7f66b41ba7a0 2024-04-26T16:36:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b0ff9700 1 -- 172.21.15.50:0/1553211827 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f66b41bb120 con 0x7f66b41ba7a0 2024-04-26T16:36:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b0ff9700 10 client.31047.objecter ms_dispatch 0x7f66b4001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:36:54.747 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b0ff9700 10 client.31047.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:36:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b0ff9700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 4 2024-04-26T16:36:54.748 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b0ff9700 15 client.31047.objecter _session_command_op_remove 7 16 2024-04-26T16:36:54.748 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 librados: watch_flush enter 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 librados: watch_flush exit 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.0 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f66b4064280 msgr2=0x7f66b410a900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f66b4064280 0x7f66b410a900 crc :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b40ff330 osd=0 1 2024-04-26T16:36:54.750 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.1 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f66b4063e70 msgr2=0x7f66b41ac100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f66b4063e70 0x7f66b41ac100 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b4063d00 osd=1 1 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.2 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f66b402d6d0 msgr2=0x7f66b41af980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f66b402d6d0 0x7f66b41af980 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b402d560 osd=2 1 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.3 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f66b41ad4d0 msgr2=0x7f66b41b3050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f66b41ad4d0 0x7f66b41b3050 crc :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.751 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b410b640 osd=3 1 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.4 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f66b41b0ba0 msgr2=0x7f66b41b35e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f66b41b0ba0 0x7f66b41b35e0 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41aca40 osd=4 1 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.5 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f66b41b4290 msgr2=0x7f66b41b46d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f66b41b4290 0x7f66b41b46d0 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41b3b20 osd=5 1 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.6 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f66b41bc180 msgr2=0x7f66b41bc5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.752 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f66b41bc180 0x7f66b41bc5c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bba60 osd=6 1 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter close_session for osd.7 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f66b41ba7a0 msgr2=0x7f66b41babe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f66b41ba7a0 0x7f66b41babe0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter put_session s=0x7f66b41bb610 osd=7 1 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 20 client.31047.objecter shutdown clearing up homeless session... 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 10 client.31047.objecter successfully canceled tick 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b3fff700 1 -- 172.21.15.50:0/1553211827 reap_dead start 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b3fff700 1 -- 172.21.15.50:0/1553211827 reap_dead start 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b3fff700 1 -- 172.21.15.50:0/1553211827 reap_dead start 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b3fff700 1 -- 172.21.15.50:0/1553211827 reap_dead start 2024-04-26T16:36:54.753 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b3fff700 1 -- 172.21.15.50:0/1553211827 reap_dead start 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f669c071f80 msgr2=0x7f669c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f669c071f80 0x7f669c074440 secure :-1 s=READY pgs=3836 cs=0 l=1 rev1=1 crypto rx=0x7f66a800b2e0 tx=0x7f66a801a040 comp rx=0 tx=0).stop 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b4101ae0 msgr2=0x7f66b40fde00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b4101ae0 0x7f66b40fde00 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f66a400d750 tx=0x7f66a400da60 comp rx=0 tx=0).stop 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 shutdown_connections 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f66b4101ae0 0x7f66b40fde00 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 --2- 172.21.15.50:0/1553211827 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f669c071f80 0x7f669c074440 unknown :-1 s=CLOSED pgs=3836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 >> 172.21.15.50:0/1553211827 conn(0x7f66b4078990 msgr2=0x7f66b4105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 shutdown_connections 2024-04-26T16:36:54.754 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 -- 172.21.15.50:0/1553211827 wait complete. 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:36:54.749+0000 7f66b9f0a700 1 librados: shutdown 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:36:54.755 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:36:54.834 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.701809+0000 osd.3 (osd.3) 467 : cluster [DBG] 3.b scrub starts 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.701809+0000 osd.3 (osd.3) 467 : cluster [DBG] 3.b scrub starts 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.706300+0000 osd.3 (osd.3) 468 : cluster [DBG] 3.b scrub ok 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.706300+0000 osd.3 (osd.3) 468 : cluster [DBG] 3.b scrub ok 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.939560+0000 osd.4 (osd.4) 401 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.939560+0000 osd.4 (osd.4) 401 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:36:54.845 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.947622+0000 osd.4 (osd.4) 402 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.947622+0000 osd.4 (osd.4) 402 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.418680+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.418680+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: audit 2024-04-26T16:36:54.422920+0000 mon.c (mon.2) 856 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: audit 2024-04-26T16:36:54.422920+0000 mon.c (mon.2) 856 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: audit 2024-04-26T16:36:54.425785+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: audit 2024-04-26T16:36:54.425785+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: audit 2024-04-26T16:36:54.567745+0000 mon.a (mon.0) 3803 : audit [DBG] from='client.? 172.21.15.50:0/1553211827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:54.846 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[29234]: audit 2024-04-26T16:36:54.567745+0000 mon.a (mon.0) 3803 : audit [DBG] from='client.? 172.21.15.50:0/1553211827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.701809+0000 osd.3 (osd.3) 467 : cluster [DBG] 3.b scrub starts 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.701809+0000 osd.3 (osd.3) 467 : cluster [DBG] 3.b scrub starts 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.706300+0000 osd.3 (osd.3) 468 : cluster [DBG] 3.b scrub ok 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.706300+0000 osd.3 (osd.3) 468 : cluster [DBG] 3.b scrub ok 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.939560+0000 osd.4 (osd.4) 401 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.939560+0000 osd.4 (osd.4) 401 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.947622+0000 osd.4 (osd.4) 402 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.947622+0000 osd.4 (osd.4) 402 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.418680+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.418680+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-04-26T16:36:54.847 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: audit 2024-04-26T16:36:54.422920+0000 mon.c (mon.2) 856 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: audit 2024-04-26T16:36:54.422920+0000 mon.c (mon.2) 856 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: audit 2024-04-26T16:36:54.425785+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: audit 2024-04-26T16:36:54.425785+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: audit 2024-04-26T16:36:54.567745+0000 mon.a (mon.0) 3803 : audit [DBG] from='client.? 172.21.15.50:0/1553211827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:54.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:54 smithi050 bash[21463]: audit 2024-04-26T16:36:54.567745+0000 mon.a (mon.0) 3803 : audit [DBG] from='client.? 172.21.15.50:0/1553211827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:54.858 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:54.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:54.972 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.701809+0000 osd.3 (osd.3) 467 : cluster [DBG] 3.b scrub starts 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.701809+0000 osd.3 (osd.3) 467 : cluster [DBG] 3.b scrub starts 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.706300+0000 osd.3 (osd.3) 468 : cluster [DBG] 3.b scrub ok 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.706300+0000 osd.3 (osd.3) 468 : cluster [DBG] 3.b scrub ok 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.939560+0000 osd.4 (osd.4) 401 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.939560+0000 osd.4 (osd.4) 401 : cluster [DBG] 3.9 deep-scrub starts 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.947622+0000 osd.4 (osd.4) 402 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.947622+0000 osd.4 (osd.4) 402 : cluster [DBG] 3.9 deep-scrub ok 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.418680+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.418680+0000 mon.a (mon.0) 3801 : cluster [DBG] osdmap e825: 8 total, 8 up, 8 in 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: audit 2024-04-26T16:36:54.422920+0000 mon.c (mon.2) 856 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.973 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: audit 2024-04-26T16:36:54.422920+0000 mon.c (mon.2) 856 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: audit 2024-04-26T16:36:54.425785+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: audit 2024-04-26T16:36:54.425785+0000 mon.a (mon.0) 3802 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:36:54.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: audit 2024-04-26T16:36:54.567745+0000 mon.a (mon.0) 3803 : audit [DBG] from='client.? 172.21.15.50:0/1553211827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:54.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:54 smithi190 bash[23868]: audit 2024-04-26T16:36:54.567745+0000 mon.a (mon.0) 3803 : audit [DBG] from='client.? 172.21.15.50:0/1553211827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:36:54.989 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:54.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:55.092 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:55.120 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:55.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:55.250 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:55.352 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:55.380 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:55.483 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:55.518 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:55.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:55.649 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:36:55.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:55.779 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:55.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:55.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.937855+0000 mgr.y (mgr.24401) 747 : cluster [DBG] pgmap v1281: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:55.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:53.937855+0000 mgr.y (mgr.24401) 747 : cluster [DBG] pgmap v1281: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:55.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.760440+0000 osd.5 (osd.5) 313 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.760440+0000 osd.5 (osd.5) 313 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.762114+0000 osd.5 (osd.5) 314 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.762114+0000 osd.5 (osd.5) 314 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.944228+0000 osd.4 (osd.4) 403 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.944228+0000 osd.4 (osd.4) 403 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.946605+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.946605+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: audit 2024-04-26T16:36:55.416187+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: audit 2024-04-26T16:36:55.416187+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.420277+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.420277+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-04-26T16:36:55.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: audit 2024-04-26T16:36:55.526044+0000 mon.c (mon.2) 857 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:55.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: audit 2024-04-26T16:36:55.526044+0000 mon.c (mon.2) 857 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:55.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: audit 2024-04-26T16:36:55.526977+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:55.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:55 smithi190 bash[23868]: audit 2024-04-26T16:36:55.526977+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:55.909 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:55.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:55.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:56.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:56.039 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.937855+0000 mgr.y (mgr.24401) 747 : cluster [DBG] pgmap v1281: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:53.937855+0000 mgr.y (mgr.24401) 747 : cluster [DBG] pgmap v1281: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.760440+0000 osd.5 (osd.5) 313 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.760440+0000 osd.5 (osd.5) 313 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.762114+0000 osd.5 (osd.5) 314 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.762114+0000 osd.5 (osd.5) 314 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.944228+0000 osd.4 (osd.4) 403 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.944228+0000 osd.4 (osd.4) 403 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.946605+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.946605+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: audit 2024-04-26T16:36:55.416187+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: audit 2024-04-26T16:36:55.416187+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.420277+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.420277+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: audit 2024-04-26T16:36:55.526044+0000 mon.c (mon.2) 857 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: audit 2024-04-26T16:36:55.526044+0000 mon.c (mon.2) 857 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: audit 2024-04-26T16:36:55.526977+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[21463]: audit 2024-04-26T16:36:55.526977+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.937855+0000 mgr.y (mgr.24401) 747 : cluster [DBG] pgmap v1281: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:53.937855+0000 mgr.y (mgr.24401) 747 : cluster [DBG] pgmap v1281: 172 pgs: 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.760440+0000 osd.5 (osd.5) 313 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.760440+0000 osd.5 (osd.5) 313 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.762114+0000 osd.5 (osd.5) 314 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.762114+0000 osd.5 (osd.5) 314 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:36:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.944228+0000 osd.4 (osd.4) 403 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.944228+0000 osd.4 (osd.4) 403 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.946605+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.946605+0000 osd.4 (osd.4) 404 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: audit 2024-04-26T16:36:55.416187+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: audit 2024-04-26T16:36:55.416187+0000 mon.a (mon.0) 3804 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.420277+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.420277+0000 mon.a (mon.0) 3805 : cluster [DBG] osdmap e826: 8 total, 8 up, 8 in 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: audit 2024-04-26T16:36:55.526044+0000 mon.c (mon.2) 857 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: audit 2024-04-26T16:36:55.526044+0000 mon.c (mon.2) 857 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: audit 2024-04-26T16:36:55.526977+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:55 smithi050 bash[29234]: audit 2024-04-26T16:36:55.526977+0000 mon.a (mon.0) 3806 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:36:56.143 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:56.169 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:56.273 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:56.293 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:56.395 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:56.425 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:56.528 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:56.557 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:56.660 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:56.681 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:56.784 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:56.811 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:56.914 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:56.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.961148+0000 osd.0 (osd.0) 457 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:36:56.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.961148+0000 osd.0 (osd.0) 457 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.962592+0000 osd.0 (osd.0) 458 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:54.962592+0000 osd.0 (osd.0) 458 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.362115+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.362115+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.364794+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.364794+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: audit 2024-04-26T16:36:56.387268+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: audit 2024-04-26T16:36:56.387268+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.391662+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.391662+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: audit 2024-04-26T16:36:56.394274+0000 mon.c (mon.2) 858 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: audit 2024-04-26T16:36:56.394274+0000 mon.c (mon.2) 858 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: audit 2024-04-26T16:36:56.406329+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.924 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[29234]: audit 2024-04-26T16:36:56.406329+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.961148+0000 osd.0 (osd.0) 457 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.961148+0000 osd.0 (osd.0) 457 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.962592+0000 osd.0 (osd.0) 458 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:54.962592+0000 osd.0 (osd.0) 458 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.362115+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.362115+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.364794+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.364794+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: audit 2024-04-26T16:36:56.387268+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: audit 2024-04-26T16:36:56.387268+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.391662+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-04-26T16:36:56.925 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.391662+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-04-26T16:36:56.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: audit 2024-04-26T16:36:56.394274+0000 mon.c (mon.2) 858 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: audit 2024-04-26T16:36:56.394274+0000 mon.c (mon.2) 858 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: audit 2024-04-26T16:36:56.406329+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.926 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:56 smithi050 bash[21463]: audit 2024-04-26T16:36:56.406329+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:56.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:36:56.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:56.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:36:57.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.961148+0000 osd.0 (osd.0) 457 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.961148+0000 osd.0 (osd.0) 457 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.962592+0000 osd.0 (osd.0) 458 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:54.962592+0000 osd.0 (osd.0) 458 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.362115+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.362115+0000 osd.6 (osd.6) 19 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:36:57.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.364794+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.364794+0000 osd.6 (osd.6) 20 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: audit 2024-04-26T16:36:56.387268+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: audit 2024-04-26T16:36:56.387268+0000 mon.a (mon.0) 3807 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.391662+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.391662+0000 mon.a (mon.0) 3808 : cluster [DBG] osdmap e827: 8 total, 8 up, 8 in 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: audit 2024-04-26T16:36:56.394274+0000 mon.c (mon.2) 858 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: audit 2024-04-26T16:36:56.394274+0000 mon.c (mon.2) 858 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: audit 2024-04-26T16:36:56.406329+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:57.052 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:56 smithi190 bash[23868]: audit 2024-04-26T16:36:56.406329+0000 mon.a (mon.0) 3809 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:36:57.066 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:57.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:57.197 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:57.300 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:57.323 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:57.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:57.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:57.453 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:57.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:57.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:36:57.584 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:57.687 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:57.714 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:57.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:57.847 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:57.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:57.979 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:57.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.938649+0000 mgr.y (mgr.24401) 748 : cluster [DBG] pgmap v1284: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:55.938649+0000 mgr.y (mgr.24401) 748 : cluster [DBG] pgmap v1284: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.936155+0000 osd.0 (osd.0) 459 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:36:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.936155+0000 osd.0 (osd.0) 459 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:36:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.939128+0000 osd.0 (osd.0) 460 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.939128+0000 osd.0 (osd.0) 460 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.983399+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c deep-scrub starts 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.983399+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c deep-scrub starts 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.985392+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c deep-scrub ok 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:56.985392+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c deep-scrub ok 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.200448+0000 osd.4 (osd.4) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.200448+0000 osd.4 (osd.4) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.200731+0000 osd.4 (osd.4) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.200731+0000 osd.4 (osd.4) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: audit 2024-04-26T16:36:57.401002+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: audit 2024-04-26T16:36:57.401002+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.404947+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.404947+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: audit 2024-04-26T16:36:57.406211+0000 mon.c (mon.2) 859 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: audit 2024-04-26T16:36:57.406211+0000 mon.c (mon.2) 859 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: audit 2024-04-26T16:36:57.406609+0000 mon.a (mon.0) 3812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:57 smithi190 bash[23868]: audit 2024-04-26T16:36:57.406609+0000 mon.a (mon.0) 3812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.082 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:58.109 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:58.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:58.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.938649+0000 mgr.y (mgr.24401) 748 : cluster [DBG] pgmap v1284: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:58.126 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:55.938649+0000 mgr.y (mgr.24401) 748 : cluster [DBG] pgmap v1284: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.936155+0000 osd.0 (osd.0) 459 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.936155+0000 osd.0 (osd.0) 459 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.939128+0000 osd.0 (osd.0) 460 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.939128+0000 osd.0 (osd.0) 460 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.983399+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c deep-scrub starts 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.983399+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c deep-scrub starts 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.985392+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c deep-scrub ok 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:56.985392+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c deep-scrub ok 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.200448+0000 osd.4 (osd.4) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.200448+0000 osd.4 (osd.4) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.200731+0000 osd.4 (osd.4) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.200731+0000 osd.4 (osd.4) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: audit 2024-04-26T16:36:57.401002+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:36:58.127 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: audit 2024-04-26T16:36:57.401002+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.404947+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.404947+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: audit 2024-04-26T16:36:57.406211+0000 mon.c (mon.2) 859 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: audit 2024-04-26T16:36:57.406211+0000 mon.c (mon.2) 859 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: audit 2024-04-26T16:36:57.406609+0000 mon.a (mon.0) 3812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[21463]: audit 2024-04-26T16:36:57.406609+0000 mon.a (mon.0) 3812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.938649+0000 mgr.y (mgr.24401) 748 : cluster [DBG] pgmap v1284: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:55.938649+0000 mgr.y (mgr.24401) 748 : cluster [DBG] pgmap v1284: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.936155+0000 osd.0 (osd.0) 459 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.936155+0000 osd.0 (osd.0) 459 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.939128+0000 osd.0 (osd.0) 460 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:36:58.128 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.939128+0000 osd.0 (osd.0) 460 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.983399+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c deep-scrub starts 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.983399+0000 osd.4 (osd.4) 405 : cluster [DBG] 5.1c deep-scrub starts 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.985392+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c deep-scrub ok 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:56.985392+0000 osd.4 (osd.4) 406 : cluster [DBG] 5.1c deep-scrub ok 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.200448+0000 osd.4 (osd.4) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.200448+0000 osd.4 (osd.4) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.200731+0000 osd.4 (osd.4) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.200731+0000 osd.4 (osd.4) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: audit 2024-04-26T16:36:57.401002+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: audit 2024-04-26T16:36:57.401002+0000 mon.a (mon.0) 3810 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.404947+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.404947+0000 mon.a (mon.0) 3811 : cluster [DBG] osdmap e828: 8 total, 8 up, 8 in 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: audit 2024-04-26T16:36:57.406211+0000 mon.c (mon.2) 859 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: audit 2024-04-26T16:36:57.406211+0000 mon.c (mon.2) 859 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: audit 2024-04-26T16:36:57.406609+0000 mon.a (mon.0) 3812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:57 smithi050 bash[29234]: audit 2024-04-26T16:36:57.406609+0000 mon.a (mon.0) 3812 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]: dispatch 2024-04-26T16:36:58.212 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:58.241 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:58.344 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:58.372 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:58.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:58.503 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:58.606 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:58.636 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:36:58.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:36:58.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:58.766 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:36:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:36:58.869 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.196748+0000 osd.1 (osd.1) 429 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.196748+0000 osd.1 (osd.1) 429 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.198775+0000 osd.1 (osd.1) 430 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.198775+0000 osd.1 (osd.1) 430 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: audit 2024-04-26T16:36:57.309059+0000 mgr.y (mgr.24401) 749 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: audit 2024-04-26T16:36:57.309059+0000 mgr.y (mgr.24401) 749 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.365995+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.365995+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.368732+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.368732+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:36:58.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: audit 2024-04-26T16:36:57.886174+0000 mon.c (mon.2) 860 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: audit 2024-04-26T16:36:57.886174+0000 mon.c (mon.2) 860 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.932217+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.932217+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.942182+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.942182+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.975072+0000 osd.4 (osd.4) 409 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:36:58.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.975072+0000 osd.4 (osd.4) 409 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:36:58.895 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:58.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:58.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.196748+0000 osd.1 (osd.1) 429 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.196748+0000 osd.1 (osd.1) 429 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.198775+0000 osd.1 (osd.1) 430 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.198775+0000 osd.1 (osd.1) 430 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: audit 2024-04-26T16:36:57.309059+0000 mgr.y (mgr.24401) 749 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: audit 2024-04-26T16:36:57.309059+0000 mgr.y (mgr.24401) 749 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.365995+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:36:59.010 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.365995+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.368732+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.368732+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: audit 2024-04-26T16:36:57.886174+0000 mon.c (mon.2) 860 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: audit 2024-04-26T16:36:57.886174+0000 mon.c (mon.2) 860 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.932217+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.932217+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.942182+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.942182+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.975072+0000 osd.4 (osd.4) 409 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.975072+0000 osd.4 (osd.4) 409 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.977265+0000 osd.4 (osd.4) 410 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.977265+0000 osd.4 (osd.4) 410 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.401593+0000 mon.a (mon.0) 3813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:59.011 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.401593+0000 mon.a (mon.0) 3813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: audit 2024-04-26T16:36:58.403451+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]': finished 2024-04-26T16:36:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: audit 2024-04-26T16:36:58.403451+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]': finished 2024-04-26T16:36:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.407344+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-04-26T16:36:59.012 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:58 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.407344+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-04-26T16:36:59.025 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:59.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:59.155 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:36:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:36:59.178 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.977265+0000 osd.4 (osd.4) 410 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.977265+0000 osd.4 (osd.4) 410 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.401593+0000 mon.a (mon.0) 3813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.401593+0000 mon.a (mon.0) 3813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: audit 2024-04-26T16:36:58.403451+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]': finished 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: audit 2024-04-26T16:36:58.403451+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]': finished 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.407344+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.407344+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-04-26T16:36:59.179 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.196748+0000 osd.1 (osd.1) 429 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.196748+0000 osd.1 (osd.1) 429 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.198775+0000 osd.1 (osd.1) 430 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.198775+0000 osd.1 (osd.1) 430 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: audit 2024-04-26T16:36:57.309059+0000 mgr.y (mgr.24401) 749 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: audit 2024-04-26T16:36:57.309059+0000 mgr.y (mgr.24401) 749 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.365995+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.365995+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.368732+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.368732+0000 osd.6 (osd.6) 22 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: audit 2024-04-26T16:36:57.886174+0000 mon.c (mon.2) 860 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: audit 2024-04-26T16:36:57.886174+0000 mon.c (mon.2) 860 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.932217+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.932217+0000 osd.0 (osd.0) 461 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:36:59.180 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.942182+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.942182+0000 osd.0 (osd.0) 462 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.975072+0000 osd.4 (osd.4) 409 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.975072+0000 osd.4 (osd.4) 409 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.977265+0000 osd.4 (osd.4) 410 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.977265+0000 osd.4 (osd.4) 410 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.401593+0000 mon.a (mon.0) 3813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.401593+0000 mon.a (mon.0) 3813 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: audit 2024-04-26T16:36:58.403451+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]': finished 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: audit 2024-04-26T16:36:58.403451+0000 mon.a (mon.0) 3814 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-144", "mode": "readproxy"}]': finished 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.407344+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-04-26T16:36:59.181 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:58 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.407344+0000 mon.a (mon.0) 3815 : cluster [DBG] osdmap e829: 8 total, 8 up, 8 in 2024-04-26T16:36:59.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:59.287 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:59.391 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:59.420 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:59.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:59.550 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:36:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:36:59.653 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:59.683 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:36:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:36:59.786 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:59.812 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:36:59.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:36:59.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.939626+0000 mgr.y (mgr.24401) 750 : cluster [DBG] pgmap v1287: 204 pgs: 3 unknown, 201 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:57.939626+0000 mgr.y (mgr.24401) 750 : cluster [DBG] pgmap v1287: 204 pgs: 3 unknown, 201 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.917556+0000 osd.0 (osd.0) 463 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.917556+0000 osd.0 (osd.0) 463 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.920290+0000 osd.0 (osd.0) 464 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:58.920290+0000 osd.0 (osd.0) 464 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.473420+0000 osd.2 (osd.2) 403 : cluster [DBG] 11.d scrub starts 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.473420+0000 osd.2 (osd.2) 403 : cluster [DBG] 11.d scrub starts 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.476827+0000 osd.2 (osd.2) 404 : cluster [DBG] 11.d scrub ok 2024-04-26T16:36:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:36:59 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.476827+0000 osd.2 (osd.2) 404 : cluster [DBG] 11.d scrub ok 2024-04-26T16:36:59.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.939626+0000 mgr.y (mgr.24401) 750 : cluster [DBG] pgmap v1287: 204 pgs: 3 unknown, 201 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:57.939626+0000 mgr.y (mgr.24401) 750 : cluster [DBG] pgmap v1287: 204 pgs: 3 unknown, 201 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.917556+0000 osd.0 (osd.0) 463 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.917556+0000 osd.0 (osd.0) 463 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.920290+0000 osd.0 (osd.0) 464 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:58.920290+0000 osd.0 (osd.0) 464 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.473420+0000 osd.2 (osd.2) 403 : cluster [DBG] 11.d scrub starts 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.473420+0000 osd.2 (osd.2) 403 : cluster [DBG] 11.d scrub starts 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.476827+0000 osd.2 (osd.2) 404 : cluster [DBG] 11.d scrub ok 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.476827+0000 osd.2 (osd.2) 404 : cluster [DBG] 11.d scrub ok 2024-04-26T16:36:59.932 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.939626+0000 mgr.y (mgr.24401) 750 : cluster [DBG] pgmap v1287: 204 pgs: 3 unknown, 201 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:57.939626+0000 mgr.y (mgr.24401) 750 : cluster [DBG] pgmap v1287: 204 pgs: 3 unknown, 201 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.917556+0000 osd.0 (osd.0) 463 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.917556+0000 osd.0 (osd.0) 463 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.920290+0000 osd.0 (osd.0) 464 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:58.920290+0000 osd.0 (osd.0) 464 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.473420+0000 osd.2 (osd.2) 403 : cluster [DBG] 11.d scrub starts 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.473420+0000 osd.2 (osd.2) 403 : cluster [DBG] 11.d scrub starts 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.476827+0000 osd.2 (osd.2) 404 : cluster [DBG] 11.d scrub ok 2024-04-26T16:36:59.933 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:36:59 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.476827+0000 osd.2 (osd.2) 404 : cluster [DBG] 11.d scrub ok 2024-04-26T16:36:59.942 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:36:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:36:59.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:00.046 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:00.071 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:00.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:00.201 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:00.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:00.329 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:00.433 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:00.460 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:00.563 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:00.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:00.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:00.720 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:00.825 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:00.849 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:00.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:00.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.132243+0000 osd.1 (osd.1) 431 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:37:00.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.132243+0000 osd.1 (osd.1) 431 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:37:00.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.136397+0000 osd.1 (osd.1) 432 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:37:00.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.136397+0000 osd.1 (osd.1) 432 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:37:00.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.151131+0000 osd.7 (osd.7) 433 : cluster [DBG] 6.1c deep-scrub starts 2024-04-26T16:37:00.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.151131+0000 osd.7 (osd.7) 433 : cluster [DBG] 6.1c deep-scrub starts 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.155038+0000 osd.7 (osd.7) 434 : cluster [DBG] 6.1c deep-scrub ok 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.155038+0000 osd.7 (osd.7) 434 : cluster [DBG] 6.1c deep-scrub ok 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.984584+0000 osd.4 (osd.4) 411 : cluster [DBG] 2.f scrub starts 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.984584+0000 osd.4 (osd.4) 411 : cluster [DBG] 2.f scrub starts 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.986739+0000 osd.4 (osd.4) 412 : cluster [DBG] 2.f scrub ok 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.986739+0000 osd.4 (osd.4) 412 : cluster [DBG] 2.f scrub ok 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.515206+0000 osd.2 (osd.2) 405 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.515206+0000 osd.2 (osd.2) 405 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.520667+0000 osd.2 (osd.2) 406 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:37:00.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.520667+0000 osd.2 (osd.2) 406 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:37:00.980 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:00.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:01.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.132243+0000 osd.1 (osd.1) 431 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.132243+0000 osd.1 (osd.1) 431 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.136397+0000 osd.1 (osd.1) 432 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.136397+0000 osd.1 (osd.1) 432 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.151131+0000 osd.7 (osd.7) 433 : cluster [DBG] 6.1c deep-scrub starts 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.151131+0000 osd.7 (osd.7) 433 : cluster [DBG] 6.1c deep-scrub starts 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.155038+0000 osd.7 (osd.7) 434 : cluster [DBG] 6.1c deep-scrub ok 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.155038+0000 osd.7 (osd.7) 434 : cluster [DBG] 6.1c deep-scrub ok 2024-04-26T16:37:01.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.984584+0000 osd.4 (osd.4) 411 : cluster [DBG] 2.f scrub starts 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.984584+0000 osd.4 (osd.4) 411 : cluster [DBG] 2.f scrub starts 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.986739+0000 osd.4 (osd.4) 412 : cluster [DBG] 2.f scrub ok 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.986739+0000 osd.4 (osd.4) 412 : cluster [DBG] 2.f scrub ok 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.515206+0000 osd.2 (osd.2) 405 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.515206+0000 osd.2 (osd.2) 405 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.520667+0000 osd.2 (osd.2) 406 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.520667+0000 osd.2 (osd.2) 406 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:37:01.098 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.132243+0000 osd.1 (osd.1) 431 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.132243+0000 osd.1 (osd.1) 431 : cluster [DBG] 323.6s0 scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.136397+0000 osd.1 (osd.1) 432 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.136397+0000 osd.1 (osd.1) 432 : cluster [DBG] 323.6 scrub ok 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.151131+0000 osd.7 (osd.7) 433 : cluster [DBG] 6.1c deep-scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.151131+0000 osd.7 (osd.7) 433 : cluster [DBG] 6.1c deep-scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.155038+0000 osd.7 (osd.7) 434 : cluster [DBG] 6.1c deep-scrub ok 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.155038+0000 osd.7 (osd.7) 434 : cluster [DBG] 6.1c deep-scrub ok 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.984584+0000 osd.4 (osd.4) 411 : cluster [DBG] 2.f scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.984584+0000 osd.4 (osd.4) 411 : cluster [DBG] 2.f scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.986739+0000 osd.4 (osd.4) 412 : cluster [DBG] 2.f scrub ok 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.986739+0000 osd.4 (osd.4) 412 : cluster [DBG] 2.f scrub ok 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.515206+0000 osd.2 (osd.2) 405 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.515206+0000 osd.2 (osd.2) 405 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:37:01.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.520667+0000 osd.2 (osd.2) 406 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:37:01.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.520667+0000 osd.2 (osd.2) 406 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:37:01.111 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:01.214 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.243 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:01.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:01.374 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:01.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:01.505 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:01.580 INFO:teuthology.orchestra.run.smithi050.stdout:ERROR: (22) Invalid argument 2024-04-26T16:37:01.580 INFO:teuthology.orchestra.run.smithi050.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:37:01.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:01.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:01.740 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.767 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:01.871 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.899 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:01.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:02.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.940581+0000 mgr.y (mgr.24401) 751 : cluster [DBG] pgmap v1289: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 226 B/s wr, 1 op/s 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:36:59.940581+0000 mgr.y (mgr.24401) 751 : cluster [DBG] pgmap v1289: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 226 B/s wr, 1 op/s 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.146429+0000 osd.7 (osd.7) 435 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.146429+0000 osd.7 (osd.7) 435 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.148727+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.148727+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.943545+0000 osd.0 (osd.0) 465 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:37:02.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.943545+0000 osd.0 (osd.0) 465 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.940581+0000 mgr.y (mgr.24401) 751 : cluster [DBG] pgmap v1289: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 226 B/s wr, 1 op/s 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:36:59.940581+0000 mgr.y (mgr.24401) 751 : cluster [DBG] pgmap v1289: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 226 B/s wr, 1 op/s 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.146429+0000 osd.7 (osd.7) 435 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.146429+0000 osd.7 (osd.7) 435 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.148727+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.148727+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.943545+0000 osd.0 (osd.0) 465 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:37:02.018 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.943545+0000 osd.0 (osd.0) 465 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.940581+0000 mgr.y (mgr.24401) 751 : cluster [DBG] pgmap v1289: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 226 B/s wr, 1 op/s 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:36:59.940581+0000 mgr.y (mgr.24401) 751 : cluster [DBG] pgmap v1289: 204 pgs: 204 active+clean; 4.4 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 905 B/s rd, 226 B/s wr, 1 op/s 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.146429+0000 osd.7 (osd.7) 435 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.146429+0000 osd.7 (osd.7) 435 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.148727+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.148727+0000 osd.7 (osd.7) 436 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.943545+0000 osd.0 (osd.0) 465 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:37:02.019 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.943545+0000 osd.0 (osd.0) 465 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:37:02.029 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:02.086 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:37:02.132 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:02.159 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:02.262 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:02.285 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:02.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:02.414 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:02.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:02.545 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:02.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:02.677 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:02.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:02.806 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:02.909 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:02.937 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:02.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.945230+0000 osd.0 (osd.0) 466 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:00.945230+0000 osd.0 (osd.0) 466 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.054737+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.054737+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.057080+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.057080+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.194078+0000 osd.7 (osd.7) 437 : cluster [DBG] 11.f scrub starts 2024-04-26T16:37:02.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.194078+0000 osd.7 (osd.7) 437 : cluster [DBG] 11.f scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.200911+0000 osd.7 (osd.7) 438 : cluster [DBG] 11.f scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.200911+0000 osd.7 (osd.7) 438 : cluster [DBG] 11.f scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.414923+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.414923+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.416142+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.416142+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.742217+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.742217+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.745257+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.745257+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.921876+0000 osd.0 (osd.0) 467 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.921876+0000 osd.0 (osd.0) 467 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.923643+0000 osd.0 (osd.0) 468 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:37:02.973 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.923643+0000 osd.0 (osd.0) 468 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.945230+0000 osd.0 (osd.0) 466 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:00.945230+0000 osd.0 (osd.0) 466 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.054737+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.054737+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.057080+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.057080+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.194078+0000 osd.7 (osd.7) 437 : cluster [DBG] 11.f scrub starts 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.194078+0000 osd.7 (osd.7) 437 : cluster [DBG] 11.f scrub starts 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.200911+0000 osd.7 (osd.7) 438 : cluster [DBG] 11.f scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.200911+0000 osd.7 (osd.7) 438 : cluster [DBG] 11.f scrub ok 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.414923+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:37:02.974 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.414923+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.416142+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.416142+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.742217+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.742217+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.745257+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.745257+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.921876+0000 osd.0 (osd.0) 467 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.921876+0000 osd.0 (osd.0) 467 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.923643+0000 osd.0 (osd.0) 468 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:37:02.975 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:02 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.923643+0000 osd.0 (osd.0) 468 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:37:03.041 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:03.068 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.945230+0000 osd.0 (osd.0) 466 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:00.945230+0000 osd.0 (osd.0) 466 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.054737+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.054737+0000 osd.1 (osd.1) 433 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.057080+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.057080+0000 osd.1 (osd.1) 434 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.194078+0000 osd.7 (osd.7) 437 : cluster [DBG] 11.f scrub starts 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.194078+0000 osd.7 (osd.7) 437 : cluster [DBG] 11.f scrub starts 2024-04-26T16:37:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.200911+0000 osd.7 (osd.7) 438 : cluster [DBG] 11.f scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.200911+0000 osd.7 (osd.7) 438 : cluster [DBG] 11.f scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.414923+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.414923+0000 osd.6 (osd.6) 23 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.416142+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.416142+0000 osd.6 (osd.6) 24 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.742217+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.742217+0000 osd.3 (osd.3) 469 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.745257+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.745257+0000 osd.3 (osd.3) 470 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.921876+0000 osd.0 (osd.0) 467 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.921876+0000 osd.0 (osd.0) 467 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.923643+0000 osd.0 (osd.0) 468 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:37:03.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:02 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.923643+0000 osd.0 (osd.0) 468 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:37:03.172 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:03.200 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:03.303 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:03.330 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:03.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:03.461 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:03.564 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:03.590 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:03.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:03.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:03.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:03.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:03.852 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:03.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:03.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.941794+0000 mgr.y (mgr.24401) 752 : cluster [DBG] pgmap v1290: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-26T16:37:03.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[29234]: cluster 2024-04-26T16:37:01.941794+0000 mgr.y (mgr.24401) 752 : cluster [DBG] pgmap v1290: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-26T16:37:03.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[29234]: cluster 2024-04-26T16:37:02.915995+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:37:03.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[29234]: cluster 2024-04-26T16:37:02.915995+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:37:03.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[29234]: cluster 2024-04-26T16:37:02.918662+0000 osd.5 (osd.5) 316 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:37:03.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[29234]: cluster 2024-04-26T16:37:02.918662+0000 osd.5 (osd.5) 316 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:37:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.941794+0000 mgr.y (mgr.24401) 752 : cluster [DBG] pgmap v1290: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-26T16:37:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[21463]: cluster 2024-04-26T16:37:01.941794+0000 mgr.y (mgr.24401) 752 : cluster [DBG] pgmap v1290: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-26T16:37:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[21463]: cluster 2024-04-26T16:37:02.915995+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:37:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[21463]: cluster 2024-04-26T16:37:02.915995+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:37:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[21463]: cluster 2024-04-26T16:37:02.918662+0000 osd.5 (osd.5) 316 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:37:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:03 smithi050 bash[21463]: cluster 2024-04-26T16:37:02.918662+0000 osd.5 (osd.5) 316 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:37:03.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:03.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:03 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.941794+0000 mgr.y (mgr.24401) 752 : cluster [DBG] pgmap v1290: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-26T16:37:03.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:03 smithi190 bash[23868]: cluster 2024-04-26T16:37:01.941794+0000 mgr.y (mgr.24401) 752 : cluster [DBG] pgmap v1290: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 170 B/s wr, 1 op/s 2024-04-26T16:37:03.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:03 smithi190 bash[23868]: cluster 2024-04-26T16:37:02.915995+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:37:03.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:03 smithi190 bash[23868]: cluster 2024-04-26T16:37:02.915995+0000 osd.5 (osd.5) 315 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:37:03.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:03 smithi190 bash[23868]: cluster 2024-04-26T16:37:02.918662+0000 osd.5 (osd.5) 316 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:37:03.967 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:03 smithi190 bash[23868]: cluster 2024-04-26T16:37:02.918662+0000 osd.5 (osd.5) 316 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:37:03.981 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:03.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:04.084 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:04.110 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:04.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:04.241 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:04.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:04.370 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:04.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:04.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-04-26T16:37:04.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:04.512 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:04.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:04.636 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:04.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:04.763 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:04.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:04.777 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:37:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:37:04.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:04.893 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:04.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:04.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:05.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:04 smithi190 bash[23868]: cluster 2024-04-26T16:37:03.720313+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:37:05.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:04 smithi190 bash[23868]: cluster 2024-04-26T16:37:03.720313+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:37:05.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:04 smithi190 bash[23868]: cluster 2024-04-26T16:37:03.724424+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:37:05.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:04 smithi190 bash[23868]: cluster 2024-04-26T16:37:03.724424+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:37:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/3961865984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5528107c20 msgr2=0x7f5528108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- 172.21.15.50:0/3961865984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5528107c20 0x7f5528108000 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f5524009990 tx=0x7f5524009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/3961865984 shutdown_connections 2024-04-26T16:37:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- 172.21.15.50:0/3961865984 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f55281027a0 0x7f552810e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- 172.21.15.50:0/3961865984 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5528107c20 0x7f5528108000 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.015 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- 172.21.15.50:0/3961865984 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 0x7f5528102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/3961865984 >> 172.21.15.50:0/3961865984 conn(0x7f5528077640 msgr2=0x7f5528077a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/3961865984 shutdown_connections 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/3961865984 wait complete. 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 librados: starting msgr at 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 librados: starting objecter 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 Processor -- start 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- start start 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 librados: setting wanted keys 2024-04-26T16:37:05.016 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 librados: calling monclient init 2024-04-26T16:37:05.017 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 0x7f5528103740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f55281027a0 0x7f5528103c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5528107c20 0x7f55281041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5528111a70 con 0x7f5528101de0 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f55281118f0 con 0x7f5528107c20 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5528111bf0 con 0x7f55281027a0 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552d685700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f55281027a0 0x7f5528103c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 0x7f5528103740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552d685700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f55281027a0 0x7f5528103c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43824/0 (socket says 172.21.15.50:43824) 2024-04-26T16:37:05.018 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552d685700 1 -- 172.21.15.50:0/668814606 learned_addr learned my addr 172.21.15.50:0/668814606 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552e687700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5528107c20 0x7f55281041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 -- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f55281027a0 msgr2=0x7f5528103c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f55281027a0 0x7f5528103c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 -- 172.21.15.50:0/668814606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5528107c20 msgr2=0x7f55281041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5528107c20 0x7f55281041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 -- 172.21.15.50:0/668814606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55281a7660 con 0x7f5528101de0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552e687700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5528107c20 0x7f55281041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f552de86700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 0x7f5528103740 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f55240059d0 tx=0x7f55240047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f551effd700 10 client.?.objecter ms_handle_connect 0x7f5528101de0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f551effd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524027070 con 0x7f5528101de0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5524022470 con 0x7f5528101de0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55281a78f0 con 0x7f5528101de0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5524017650 con 0x7f5528101de0 2024-04-26T16:37:05.019 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 10 client.31053.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55281a7d40 con 0x7f5528101de0 2024-04-26T16:37:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 librados: init done 2024-04-26T16:37:05.020 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.016+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f552806a110 con 0x7f5528101de0 2024-04-26T16:37:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f552402a070 con 0x7f5528101de0 2024-04-26T16:37:05.021 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5514071f80 0x7f5514074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(829..829 src has 577..829) ==== 9947+0+0 (secure 0 0 0) 0x7f5524093440 con 0x7f5528101de0 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 10 client.31053.objecter ms_dispatch 0x7f5528001550 osd_map(829..829 src has 577..829) 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 3 client.31053.objecter handle_osd_map got epochs [829,829] > 0 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 3 client.31053.objecter handle_osd_map decoding full epoch 829 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f552d685700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5514071f80 0x7f5514074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 20 client.31053.objecter dump_active .. 0 homeless 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f551effd700 10 client.31053.objecter ms_handle_connect 0x7f5514071f80 2024-04-26T16:37:05.022 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.020+0000 7f552d685700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5514071f80 0x7f5514074440 secure :-1 s=READY pgs=3837 cs=0 l=1 rev1=1 crypto rx=0x7f5518009680 tx=0x7f5518009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:05.024 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.024+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f5524031070 con 0x7f5528101de0 2024-04-26T16:37:05.024 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[29234]: cluster 2024-04-26T16:37:03.720313+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:37:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[29234]: cluster 2024-04-26T16:37:03.720313+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:37:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[29234]: cluster 2024-04-26T16:37:03.724424+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:37:05.055 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[29234]: cluster 2024-04-26T16:37:03.724424+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:37:05.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[21463]: cluster 2024-04-26T16:37:03.720313+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:37:05.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[21463]: cluster 2024-04-26T16:37:03.720313+0000 osd.3 (osd.3) 471 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:37:05.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[21463]: cluster 2024-04-26T16:37:03.724424+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:37:05.056 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:04 smithi050 bash[21463]: cluster 2024-04-26T16:37:03.724424+0000 osd.3 (osd.3) 472 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:37:05.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:05.159 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:05.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/2091604282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102310 msgr2=0x7f6fc0102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- 172.21.15.50:0/2091604282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102310 0x7f6fc0102790 secure :-1 s=READY pgs=2527 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009990 tx=0x7f6fbc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/2091604282 shutdown_connections 2024-04-26T16:37:05.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- 172.21.15.50:0/2091604282 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fc0102cd0 0x7f6fc010f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- 172.21.15.50:0/2091604282 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0108380 0x7f6fc0108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- 172.21.15.50:0/2091604282 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102310 0x7f6fc0102790 unknown :-1 s=CLOSED pgs=2527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/2091604282 >> 172.21.15.50:0/2091604282 conn(0x7f6fc0079290 msgr2=0x7f6fc00796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/2091604282 shutdown_connections 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/2091604282 wait complete. 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 librados: starting msgr at 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 librados: starting objecter 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 Processor -- start 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- start start 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 librados: setting wanted keys 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 librados: calling monclient init 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0102310 0x7f6fc0199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 0x7f6fc0199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fc0108380 0x7f6fc019d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0111990 con 0x7f6fc0102cd0 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc0111810 con 0x7f6fc0108380 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc0111b10 con 0x7f6fc0102310 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc557f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 0x7f6fc0199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0102310 0x7f6fc0199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc557f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 0x7f6fc0199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:36832/0 (socket says 172.21.15.50:36832) 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc557f700 1 -- 172.21.15.50:0/3922693762 learned_addr learned my addr 172.21.15.50:0/3922693762 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:05.210 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc6581700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fc0108380 0x7f6fc019d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 -- 172.21.15.50:0/3922693762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fc0108380 msgr2=0x7f6fc019d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fc0108380 0x7f6fc019d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 -- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 msgr2=0x7f6fc0199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 0x7f6fc0199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 -- 172.21.15.50:0/3922693762 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc019e130 con 0x7f6fc0102310 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc557f700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 0x7f6fc0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc5d80700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0102310 0x7f6fc0199120 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400e9f0 tx=0x7f6fb400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fb2ffd700 10 client.?.objecter ms_handle_connect 0x7f6fc0102310 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fb2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb400cbd0 con 0x7f6fc0102310 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/3922693762 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc019e420 con 0x7f6fc0102310 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6fb400cd70 con 0x7f6fc0102310 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fb4018420 con 0x7f6fc0102310 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 10 client.41678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:05.211 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 -- 172.21.15.50:0/3922693762 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc01a5ca0 con 0x7f6fc0102310 2024-04-26T16:37:05.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.208+0000 7f6fc7fe4700 1 librados: init done 2024-04-26T16:37:05.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa8005120 con 0x7f6fc0102310 2024-04-26T16:37:05.212 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6fb4010430 con 0x7f6fc0102310 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6fac071f80 0x7f6fac074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(829..829 src has 577..829) ==== 9947+0+0 (secure 0 0 0) 0x7f6fb4091dc0 con 0x7f6fc0102310 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 10 client.41678.objecter ms_dispatch 0x7f6fc0001550 osd_map(829..829 src has 577..829) 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 3 client.41678.objecter handle_osd_map got epochs [829,829] > 0 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 3 client.41678.objecter handle_osd_map decoding full epoch 829 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fc557f700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6fac071f80 0x7f6fac074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 20 client.41678.objecter dump_active .. 0 homeless 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fb2ffd700 10 client.41678.objecter ms_handle_connect 0x7f6fac071f80 2024-04-26T16:37:05.213 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.212+0000 7f6fc557f700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6fac071f80 0x7f6fac074440 secure :-1 s=READY pgs=3838 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0079b90 tx=0x7f6fbc00b630 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:05.216 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.216+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6fb405d540 con 0x7f6fc0102310 2024-04-26T16:37:05.262 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:05.269 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.268+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd set", "key": "noscrub"} v 0) -- 0x7f552802e490 con 0x7f5528101de0 2024-04-26T16:37:05.289 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:05.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:05.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:05.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.464+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f6fa8005410 con 0x7f6fc0102310 2024-04-26T16:37:05.521 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:05.549 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:05.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:05.680 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:05.783 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:05.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.788+0000 7f551effd700 1 -- 172.21.15.50:0/668814606 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "noscrub"}]=0 noscrub is set v830) ==== 87+0+0 (secure 0 0 0) 0x7f5524013070 con 0x7f5528101de0 2024-04-26T16:37:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 10 librados: watch_flush enter 2024-04-26T16:37:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 10 librados: watch_flush exit 2024-04-26T16:37:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 20 client.31053.objecter shutdown clearing up homeless session... 2024-04-26T16:37:05.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 10 client.31053.objecter successfully canceled tick 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5514071f80 msgr2=0x7f5514074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5514071f80 0x7f5514074440 secure :-1 s=READY pgs=3837 cs=0 l=1 rev1=1 crypto rx=0x7f5518009680 tx=0x7f5518009260 comp rx=0 tx=0).stop 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 msgr2=0x7f5528103740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.792+0000 7f55300ea700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 0x7f5528103740 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f55240059d0 tx=0x7f55240047b0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.796+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 shutdown_connections 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.796+0000 7f55300ea700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5514071f80 0x7f5514074440 unknown :-1 s=CLOSED pgs=3837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.796+0000 7f55300ea700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5528107c20 0x7f55281041c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.796+0000 7f55300ea700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5528101de0 0x7f5528103740 secure :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f55240059d0 tx=0x7f55240047b0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.796+0000 7f55300ea700 1 --2- 172.21.15.50:0/668814606 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f55281027a0 0x7f5528103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.796 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.796+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 >> 172.21.15.50:0/668814606 conn(0x7f5528077640 msgr2=0x7f552810f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:05.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.800+0000 7f6fb2ffd700 1 -- 172.21.15.50:0/3922693762 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v830) ==== 113+0+0 (secure 0 0 0) 0x7f6fb40611f0 con 0x7f6fc0102310 2024-04-26T16:37:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.800+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 shutdown_connections 2024-04-26T16:37:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.800+0000 7f55300ea700 1 -- 172.21.15.50:0/668814606 wait complete. 2024-04-26T16:37:05.800 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.800+0000 7f55300ea700 1 librados: shutdown 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 10 librados: watch_flush enter 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 10 librados: watch_flush exit 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 20 client.41678.objecter shutdown clearing up homeless session... 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 10 client.41678.objecter successfully canceled tick 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6fac071f80 msgr2=0x7f6fac074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6fac071f80 0x7f6fac074440 secure :-1 s=READY pgs=3838 cs=0 l=1 rev1=1 crypto rx=0x7f6fc0079b90 tx=0x7f6fbc00b630 comp rx=0 tx=0).stop 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0102310 msgr2=0x7f6fc0199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0102310 0x7f6fc0199120 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400e9f0 tx=0x7f6fb400ed00 comp rx=0 tx=0).stop 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 shutdown_connections 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6fac071f80 0x7f6fac074440 unknown :-1 s=CLOSED pgs=3838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.804 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6fc0108380 0x7f6fc019d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6fc0102310 0x7f6fc0199120 secure :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f6fb400e9f0 tx=0x7f6fb400ed00 comp rx=0 tx=0).stop 2024-04-26T16:37:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 --2- 172.21.15.50:0/3922693762 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6fc0102cd0 0x7f6fc0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:05.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 >> 172.21.15.50:0/3922693762 conn(0x7f6fc0079290 msgr2=0x7f6fc0100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.804+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 shutdown_connections 2024-04-26T16:37:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.808+0000 7f6fb0ff9700 1 -- 172.21.15.50:0/3922693762 wait complete. 2024-04-26T16:37:05.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:05.808+0000 7f6fb0ff9700 1 librados: shutdown 2024-04-26T16:37:05.807 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T16:37:05.808 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is set 2024-04-26T16:37:05.813 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:05.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:05.946 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:05.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:06.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:06.060 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:03.942752+0000 mgr.y (mgr.24401) 753 : cluster [DBG] pgmap v1291: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 135 B/s wr, 1 op/s 2024-04-26T16:37:06.060 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:03.942752+0000 mgr.y (mgr.24401) 753 : cluster [DBG] pgmap v1291: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 135 B/s wr, 1 op/s 2024-04-26T16:37:06.060 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.030234+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.030234+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.034556+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.034556+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.361936+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.361936+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.364453+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:04.364453+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.006229+0000 osd.1 (osd.1) 437 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.006229+0000 osd.1 (osd.1) 437 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.007574+0000 osd.1 (osd.1) 438 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.007574+0000 osd.1 (osd.1) 438 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: audit 2024-04-26T16:37:05.274554+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: audit 2024-04-26T16:37:05.274554+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: audit 2024-04-26T16:37:05.470319+0000 mon.c (mon.2) 861 : audit [INF] from='client.? 172.21.15.50:0/3922693762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: audit 2024-04-26T16:37:05.470319+0000 mon.c (mon.2) 861 : audit [INF] from='client.? 172.21.15.50:0/3922693762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: audit 2024-04-26T16:37:05.470874+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:05 smithi190 bash[23868]: audit 2024-04-26T16:37:05.470874+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.076 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:03.942752+0000 mgr.y (mgr.24401) 753 : cluster [DBG] pgmap v1291: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 135 B/s wr, 1 op/s 2024-04-26T16:37:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:03.942752+0000 mgr.y (mgr.24401) 753 : cluster [DBG] pgmap v1291: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 135 B/s wr, 1 op/s 2024-04-26T16:37:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.030234+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:37:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.030234+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:37:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.034556+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:37:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.034556+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:37:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.361936+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:37:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.361936+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:37:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.364453+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:37:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:04.364453+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:37:06.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.006229+0000 osd.1 (osd.1) 437 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:37:06.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.006229+0000 osd.1 (osd.1) 437 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:37:06.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.007574+0000 osd.1 (osd.1) 438 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:37:06.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.007574+0000 osd.1 (osd.1) 438 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:37:06.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: audit 2024-04-26T16:37:05.274554+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:37:06.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: audit 2024-04-26T16:37:05.274554+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:37:06.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: audit 2024-04-26T16:37:05.470319+0000 mon.c (mon.2) 861 : audit [INF] from='client.? 172.21.15.50:0/3922693762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: audit 2024-04-26T16:37:05.470319+0000 mon.c (mon.2) 861 : audit [INF] from='client.? 172.21.15.50:0/3922693762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: audit 2024-04-26T16:37:05.470874+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[29234]: audit 2024-04-26T16:37:05.470874+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:03.942752+0000 mgr.y (mgr.24401) 753 : cluster [DBG] pgmap v1291: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 135 B/s wr, 1 op/s 2024-04-26T16:37:06.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:03.942752+0000 mgr.y (mgr.24401) 753 : cluster [DBG] pgmap v1291: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 135 B/s wr, 1 op/s 2024-04-26T16:37:06.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.030234+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:37:06.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.030234+0000 osd.1 (osd.1) 435 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:37:06.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.034556+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:37:06.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.034556+0000 osd.1 (osd.1) 436 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:37:06.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.361936+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:37:06.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.361936+0000 osd.6 (osd.6) 25 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:37:06.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.364453+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:37:06.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:04.364453+0000 osd.6 (osd.6) 26 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:37:06.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.006229+0000 osd.1 (osd.1) 437 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:37:06.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.006229+0000 osd.1 (osd.1) 437 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:37:06.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.007574+0000 osd.1 (osd.1) 438 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:37:06.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.007574+0000 osd.1 (osd.1) 438 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:37:06.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: audit 2024-04-26T16:37:05.274554+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:37:06.137 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: audit 2024-04-26T16:37:05.274554+0000 mon.a (mon.0) 3816 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-04-26T16:37:06.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: audit 2024-04-26T16:37:05.470319+0000 mon.c (mon.2) 861 : audit [INF] from='client.? 172.21.15.50:0/3922693762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: audit 2024-04-26T16:37:05.470319+0000 mon.c (mon.2) 861 : audit [INF] from='client.? 172.21.15.50:0/3922693762' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: audit 2024-04-26T16:37:05.470874+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.138 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:05 smithi050 bash[21463]: audit 2024-04-26T16:37:05.470874+0000 mon.a (mon.0) 3817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:37:06.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:06.204 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:06.307 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:06.336 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:06.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:06.466 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:06.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:06.596 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:06.699 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:06.728 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:06.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:06.860 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:06.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:06.990 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:06.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.790273+0000 mon.a (mon.0) 3818 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.790273+0000 mon.a (mon.0) 3818 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: audit 2024-04-26T16:37:05.792755+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: audit 2024-04-26T16:37:05.792755+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: audit 2024-04-26T16:37:05.792930+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: audit 2024-04-26T16:37:05.792930+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.799057+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.799057+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.836505+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.836505+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:07.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.943276+0000 mgr.y (mgr.24401) 754 : cluster [DBG] pgmap v1293: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:37:07.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:05.943276+0000 mgr.y (mgr.24401) 754 : cluster [DBG] pgmap v1293: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:37:07.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.395363+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-04-26T16:37:07.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:06 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.395363+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-04-26T16:37:07.093 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:07.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.790273+0000 mon.a (mon.0) 3818 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:07.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.790273+0000 mon.a (mon.0) 3818 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:07.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: audit 2024-04-26T16:37:05.792755+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:37:07.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: audit 2024-04-26T16:37:05.792755+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:37:07.105 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: audit 2024-04-26T16:37:05.792930+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: audit 2024-04-26T16:37:05.792930+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.799057+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.799057+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.836505+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.836505+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.943276+0000 mgr.y (mgr.24401) 754 : cluster [DBG] pgmap v1293: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:05.943276+0000 mgr.y (mgr.24401) 754 : cluster [DBG] pgmap v1293: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.395363+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.395363+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.790273+0000 mon.a (mon.0) 3818 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:07.106 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.790273+0000 mon.a (mon.0) 3818 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: audit 2024-04-26T16:37:05.792755+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: audit 2024-04-26T16:37:05.792755+0000 mon.a (mon.0) 3819 : audit [INF] from='client.? 172.21.15.50:0/668814606' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: audit 2024-04-26T16:37:05.792930+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: audit 2024-04-26T16:37:05.792930+0000 mon.a (mon.0) 3820 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.799057+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.799057+0000 mon.a (mon.0) 3821 : cluster [DBG] osdmap e830: 8 total, 8 up, 8 in 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.836505+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.836505+0000 osd.5 (osd.5) 317 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.943276+0000 mgr.y (mgr.24401) 754 : cluster [DBG] pgmap v1293: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:05.943276+0000 mgr.y (mgr.24401) 754 : cluster [DBG] pgmap v1293: 204 pgs: 204 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.395363+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-04-26T16:37:07.107 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:06 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.395363+0000 mon.a (mon.0) 3822 : cluster [DBG] osdmap e831: 8 total, 8 up, 8 in 2024-04-26T16:37:07.121 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:07.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:07.246 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:07.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:07.377 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:07.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:07.492 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:37:07.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:07.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:07.639 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:07.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:07.777 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:07.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:07.802 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-26T16:37:07.882 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:07.925 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:07.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:08.028 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:08.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.212927+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:08.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.212927+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:08.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.353921+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:37:08.038 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.353921+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.357211+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: cluster 2024-04-26T16:37:06.357211+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: audit 2024-04-26T16:37:07.317819+0000 mgr.y (mgr.24401) 755 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[21463]: audit 2024-04-26T16:37:07.317819+0000 mgr.y (mgr.24401) 755 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.212927+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.212927+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.353921+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.353921+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.357211+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: cluster 2024-04-26T16:37:06.357211+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: audit 2024-04-26T16:37:07.317819+0000 mgr.y (mgr.24401) 755 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:08.039 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:07 smithi050 bash[29234]: audit 2024-04-26T16:37:07.317819+0000 mgr.y (mgr.24401) 755 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:08.050 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.212927+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.212927+0000 osd.7 (osd.7) 439 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.353921+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.353921+0000 osd.6 (osd.6) 27 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.357211+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: cluster 2024-04-26T16:37:06.357211+0000 osd.6 (osd.6) 28 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: audit 2024-04-26T16:37:07.317819+0000 mgr.y (mgr.24401) 755 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:07 smithi190 bash[23868]: audit 2024-04-26T16:37:07.317819+0000 mgr.y (mgr.24401) 755 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:08.154 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:08.179 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:08.180 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:08.284 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:08.309 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:08.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:08.441 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:08.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:08.572 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:08.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:08.702 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:08.805 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:08.834 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:08.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:08.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:08 smithi190 bash[23868]: cluster 2024-04-26T16:37:07.944224+0000 mgr.y (mgr.24401) 756 : cluster [DBG] pgmap v1295: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:37:08.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:08 smithi190 bash[23868]: cluster 2024-04-26T16:37:07.944224+0000 mgr.y (mgr.24401) 756 : cluster [DBG] pgmap v1295: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:37:08.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:08 smithi190 bash[23868]: audit 2024-04-26T16:37:08.713186+0000 mon.c (mon.2) 862 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:08.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:08 smithi190 bash[23868]: audit 2024-04-26T16:37:08.713186+0000 mon.c (mon.2) 862 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:08.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:08 smithi190 bash[23868]: audit 2024-04-26T16:37:08.714070+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:08.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:08 smithi190 bash[23868]: audit 2024-04-26T16:37:08.714070+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:08.964 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:08.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:09.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:09.094 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[21463]: cluster 2024-04-26T16:37:07.944224+0000 mgr.y (mgr.24401) 756 : cluster [DBG] pgmap v1295: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:37:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[21463]: cluster 2024-04-26T16:37:07.944224+0000 mgr.y (mgr.24401) 756 : cluster [DBG] pgmap v1295: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:37:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[21463]: audit 2024-04-26T16:37:08.713186+0000 mon.c (mon.2) 862 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[21463]: audit 2024-04-26T16:37:08.713186+0000 mon.c (mon.2) 862 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[21463]: audit 2024-04-26T16:37:08.714070+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[21463]: audit 2024-04-26T16:37:08.714070+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[29234]: cluster 2024-04-26T16:37:07.944224+0000 mgr.y (mgr.24401) 756 : cluster [DBG] pgmap v1295: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:37:09.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[29234]: cluster 2024-04-26T16:37:07.944224+0000 mgr.y (mgr.24401) 756 : cluster [DBG] pgmap v1295: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-26T16:37:09.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[29234]: audit 2024-04-26T16:37:08.713186+0000 mon.c (mon.2) 862 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[29234]: audit 2024-04-26T16:37:08.713186+0000 mon.c (mon.2) 862 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[29234]: audit 2024-04-26T16:37:08.714070+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:08 smithi050 bash[29234]: audit 2024-04-26T16:37:08.714070+0000 mon.a (mon.0) 3823 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:09.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:09.219 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:37:09.224 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:09.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:09.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:09.460 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:09.489 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:09.593 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:09.615 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:09.718 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:09.743 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:09.810 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd set nodeep-scrub 2024-04-26T16:37:09.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:09.877 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:09.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: audit 2024-04-26T16:37:08.830768+0000 mon.a (mon.0) 3824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: audit 2024-04-26T16:37:08.830768+0000 mon.a (mon.0) 3824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: cluster 2024-04-26T16:37:08.836784+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: cluster 2024-04-26T16:37:08.836784+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: audit 2024-04-26T16:37:08.841045+0000 mon.c (mon.2) 863 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: audit 2024-04-26T16:37:08.841045+0000 mon.c (mon.2) 863 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: audit 2024-04-26T16:37:08.841430+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[29234]: audit 2024-04-26T16:37:08.841430+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: audit 2024-04-26T16:37:08.830768+0000 mon.a (mon.0) 3824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: audit 2024-04-26T16:37:08.830768+0000 mon.a (mon.0) 3824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: cluster 2024-04-26T16:37:08.836784+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: cluster 2024-04-26T16:37:08.836784+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: audit 2024-04-26T16:37:08.841045+0000 mon.c (mon.2) 863 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: audit 2024-04-26T16:37:08.841045+0000 mon.c (mon.2) 863 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: audit 2024-04-26T16:37:08.841430+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:09 smithi050 bash[21463]: audit 2024-04-26T16:37:08.841430+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:09.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:10.009 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:10.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:10.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: audit 2024-04-26T16:37:08.830768+0000 mon.a (mon.0) 3824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:10.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: audit 2024-04-26T16:37:08.830768+0000 mon.a (mon.0) 3824 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:10.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: cluster 2024-04-26T16:37:08.836784+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-04-26T16:37:10.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: cluster 2024-04-26T16:37:08.836784+0000 mon.a (mon.0) 3825 : cluster [DBG] osdmap e832: 8 total, 8 up, 8 in 2024-04-26T16:37:10.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: audit 2024-04-26T16:37:08.841045+0000 mon.c (mon.2) 863 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.122 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: audit 2024-04-26T16:37:08.841045+0000 mon.c (mon.2) 863 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: audit 2024-04-26T16:37:08.841430+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:09 smithi190 bash[23868]: audit 2024-04-26T16:37:08.841430+0000 mon.a (mon.0) 3826 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.137 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:10.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:10.267 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:10.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:10.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:10.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:10.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:10.515 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:10.618 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:10.647 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:10.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:10.778 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:10.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:10.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:10.908 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:10.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: cluster 2024-04-26T16:37:09.831469+0000 mon.a (mon.0) 3827 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:10.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: cluster 2024-04-26T16:37:09.831469+0000 mon.a (mon.0) 3827 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.833375+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.833375+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: cluster 2024-04-26T16:37:09.837894+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: cluster 2024-04-26T16:37:09.837894+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.881135+0000 mon.c (mon.2) 864 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.881135+0000 mon.c (mon.2) 864 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.881510+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.881510+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.882061+0000 mon.c (mon.2) 865 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.882061+0000 mon.c (mon.2) 865 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.882322+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: audit 2024-04-26T16:37:09.882322+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:10.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: cluster 2024-04-26T16:37:09.944754+0000 mgr.y (mgr.24401) 757 : cluster [DBG] pgmap v1298: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:10.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:10 smithi190 bash[23868]: cluster 2024-04-26T16:37:09.944754+0000 mgr.y (mgr.24401) 757 : cluster [DBG] pgmap v1298: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:10.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:11.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:11.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: cluster 2024-04-26T16:37:09.831469+0000 mon.a (mon.0) 3827 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:11.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: cluster 2024-04-26T16:37:09.831469+0000 mon.a (mon.0) 3827 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:11.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.833375+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:37:11.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.833375+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:37:11.027 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: cluster 2024-04-26T16:37:09.837894+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: cluster 2024-04-26T16:37:09.837894+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.881135+0000 mon.c (mon.2) 864 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.881135+0000 mon.c (mon.2) 864 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.881510+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.881510+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.882061+0000 mon.c (mon.2) 865 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.882061+0000 mon.c (mon.2) 865 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.882322+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: audit 2024-04-26T16:37:09.882322+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: cluster 2024-04-26T16:37:09.944754+0000 mgr.y (mgr.24401) 757 : cluster [DBG] pgmap v1298: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:11.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[21463]: cluster 2024-04-26T16:37:09.944754+0000 mgr.y (mgr.24401) 757 : cluster [DBG] pgmap v1298: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: cluster 2024-04-26T16:37:09.831469+0000 mon.a (mon.0) 3827 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: cluster 2024-04-26T16:37:09.831469+0000 mon.a (mon.0) 3827 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.833375+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.833375+0000 mon.a (mon.0) 3828 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]': finished 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: cluster 2024-04-26T16:37:09.837894+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: cluster 2024-04-26T16:37:09.837894+0000 mon.a (mon.0) 3829 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.881135+0000 mon.c (mon.2) 864 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.881135+0000 mon.c (mon.2) 864 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.881510+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.881510+0000 mon.a (mon.0) 3830 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:11.029 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.882061+0000 mon.c (mon.2) 865 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.882061+0000 mon.c (mon.2) 865 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.882322+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: audit 2024-04-26T16:37:09.882322+0000 mon.a (mon.0) 3831 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-144"}]: dispatch 2024-04-26T16:37:11.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: cluster 2024-04-26T16:37:09.944754+0000 mgr.y (mgr.24401) 757 : cluster [DBG] pgmap v1298: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:11.030 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:10 smithi050 bash[29234]: cluster 2024-04-26T16:37:09.944754+0000 mgr.y (mgr.24401) 757 : cluster [DBG] pgmap v1298: 204 pgs: 2 active+clean+scrubbing, 202 active+clean; 4.4 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:11.042 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:11.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:11.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:11.172 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:11.276 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:11.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:11.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:11.432 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:11.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:11.564 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:11.667 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:11.694 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:11.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:11.824 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:11.928 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:11.956 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:11.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:12.059 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:12.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: cluster 2024-04-26T16:37:10.844665+0000 mon.a (mon.0) 3832 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:12.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: cluster 2024-04-26T16:37:10.844665+0000 mon.a (mon.0) 3832 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:12.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: cluster 2024-04-26T16:37:10.857838+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: cluster 2024-04-26T16:37:10.857838+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: audit 2024-04-26T16:37:11.409417+0000 mon.c (mon.2) 866 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: audit 2024-04-26T16:37:11.409417+0000 mon.c (mon.2) 866 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: cluster 2024-04-26T16:37:11.414832+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: cluster 2024-04-26T16:37:11.414832+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: audit 2024-04-26T16:37:11.415612+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:11 smithi190 bash[23868]: audit 2024-04-26T16:37:11.415612+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.089 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: cluster 2024-04-26T16:37:10.844665+0000 mon.a (mon.0) 3832 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: cluster 2024-04-26T16:37:10.844665+0000 mon.a (mon.0) 3832 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: cluster 2024-04-26T16:37:10.857838+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: cluster 2024-04-26T16:37:10.857838+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: audit 2024-04-26T16:37:11.409417+0000 mon.c (mon.2) 866 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: audit 2024-04-26T16:37:11.409417+0000 mon.c (mon.2) 866 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: cluster 2024-04-26T16:37:11.414832+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: cluster 2024-04-26T16:37:11.414832+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: audit 2024-04-26T16:37:11.415612+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[21463]: audit 2024-04-26T16:37:11.415612+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: cluster 2024-04-26T16:37:10.844665+0000 mon.a (mon.0) 3832 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: cluster 2024-04-26T16:37:10.844665+0000 mon.a (mon.0) 3832 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: cluster 2024-04-26T16:37:10.857838+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: cluster 2024-04-26T16:37:10.857838+0000 mon.a (mon.0) 3833 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: audit 2024-04-26T16:37:11.409417+0000 mon.c (mon.2) 866 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: audit 2024-04-26T16:37:11.409417+0000 mon.c (mon.2) 866 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: cluster 2024-04-26T16:37:11.414832+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: cluster 2024-04-26T16:37:11.414832+0000 mon.a (mon.0) 3834 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: audit 2024-04-26T16:37:11.415612+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:11 smithi050 bash[29234]: audit 2024-04-26T16:37:11.415612+0000 mon.a (mon.0) 3835 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:12.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:12.218 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:12.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:12.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/376982966 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c01094a0 msgr2=0x7ff4c0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- 172.21.15.50:0/376982966 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c01094a0 0x7ff4c0109880 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009990 tx=0x7ff4b4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.342 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/376982966 shutdown_connections 2024-04-26T16:37:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- 172.21.15.50:0/376982966 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4c006bad0 0x7ff4c01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- 172.21.15.50:0/376982966 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c01094a0 0x7ff4c0109880 unknown :-1 s=CLOSED pgs=2754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- 172.21.15.50:0/376982966 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006b130 0x7ff4c006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/376982966 >> 172.21.15.50:0/376982966 conn(0x7ff4c00792d0 msgr2=0x7ff4c00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/376982966 shutdown_connections 2024-04-26T16:37:12.343 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/376982966 wait complete. 2024-04-26T16:37:12.344 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 librados: starting msgr at 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 librados: starting objecter 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 Processor -- start 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- start start 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 librados: setting wanted keys 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 librados: calling monclient init 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c006b130 0x7ff4c01991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006bad0 0x7ff4c0199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4c01094a0 0x7ff4c019da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c0112ab0 con 0x7ff4c006bad0 2024-04-26T16:37:12.345 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c0112930 con 0x7ff4c01094a0 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c0112c30 con 0x7ff4c006b130 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c4f11700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c006b130 0x7ff4c01991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006bad0 0x7ff4c0199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c4f11700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c006b130 0x7ff4c01991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:45578/0 (socket says 172.21.15.50:45578) 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c4f11700 1 -- 172.21.15.50:0/375403157 learned_addr learned my addr 172.21.15.50:0/375403157 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4c01094a0 0x7ff4c019da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c006b130 msgr2=0x7ff4c01991f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c006b130 0x7ff4c01991f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4c01094a0 msgr2=0x7ff4c019da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff4c01094a0 0x7ff4c019da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c019e1b0 con 0x7ff4c006bad0 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ff4c006b130 0x7ff4c01991f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:37:12.346 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006bad0 0x7ff4c0199730 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac010d00 tx=0x7ff4ac00b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bdffb700 10 client.?.objecter ms_handle_connect 0x7ff4c006bad0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bdffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac010e50 con 0x7ff4c006bad0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c019e4a0 con 0x7ff4c006bad0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff4ac004480 con 0x7ff4c006bad0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac003bf0 con 0x7ff4c006bad0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 10 client.31065.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c0102320 con 0x7ff4c006bad0 2024-04-26T16:37:12.347 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.344+0000 7ff4c7175700 1 librados: init done 2024-04-26T16:37:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:12.349 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff4ac003d90 con 0x7ff4c006bad0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff4b0072120 0x7ff4b00745e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7ff4c0104b50 con 0x7ff4c006bad0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(835..835 src has 577..835) ==== 9939+0+0 (secure 0 0 0) 0x7ff4ac0952d0 con 0x7ff4c006bad0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 osd_map(835..835 src has 577..835) 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 3 client.31065.objecter handle_osd_map got epochs [835,835] > 0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 3 client.31065.objecter handle_osd_map decoding full epoch 835 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff4b0072120 0x7ff4b00745e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 20 client.31065.objecter dump_active .. 0 homeless 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v835) ==== 54+0+16 (secure 0 0 0) 0x7ff4ac060a50 con 0x7ff4c006bad0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4b0072120 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff4b0072120 0x7ff4b00745e0 secure :-1 s=READY pgs=3839 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4006010 tx=0x7ff4b400b630 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.350 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 10 client.31065.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 1 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff4c0064030 0x7ff4c0064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 1 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 0 1 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 4 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 10 client.31065.objecter _send_command 1 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4c006a110 con 0x7ff4c0064030 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff4c0064030 0x7ff4c0064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c0064030 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff4c0064030 0x7ff4c0064410 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.351 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4c006a110 con 0x7ff4c0064030 2024-04-26T16:37:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 1: 0 ) 2024-04-26T16:37:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 1 = system:0 2024-04-26T16:37:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 4 2024-04-26T16:37:12.352 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.348+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 0 1 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 10 client.31065.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 2 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 2 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c019e670 osd=0 3 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 0 2 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 4 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 10 client.31065.objecter _send_command 2 2024-04-26T16:37:12.370 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.368+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c019a500 con 0x7ff4c0064030 2024-04-26T16:37:12.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c019a500 con 0x7ff4c0064030 2024-04-26T16:37:12.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 4 2024-04-26T16:37:12.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 0 2 2024-04-26T16:37:12.372 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 10 client.31065.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff4c010a9d0 0x7ff4c010add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 1 3 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 4 2024-04-26T16:37:12.373 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 10 client.31065.objecter _send_command 3 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7ff4c0063e70 con 0x7ff4c010a9d0 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff4c010a9d0 0x7ff4c010add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c010a9d0 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff4c010a9d0 0x7ff4c010add0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4b40050e0 con 0x7ff4c010a9d0 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 3: 0 ) 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 3 = system:0 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 4 2024-04-26T16:37:12.374 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.372+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 1 3 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 10 client.31065.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 4 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 4 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c0063d00 osd=1 3 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 1 4 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 4 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 10 client.31065.objecter _send_command 4 2024-04-26T16:37:12.395 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.396+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c010a450 con 0x7ff4c010a9d0 2024-04-26T16:37:12.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c010a450 con 0x7ff4c010a9d0 2024-04-26T16:37:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 4 2024-04-26T16:37:12.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 1 4 2024-04-26T16:37:12.416 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 10 client.31065.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 5 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff4c01037c0 0x7ff4c0103be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 5 2024-04-26T16:37:12.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 2 5 2024-04-26T16:37:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 4 2024-04-26T16:37:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 10 client.31065.objecter _send_command 5 2024-04-26T16:37:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7ff4c0104120 con 0x7ff4c01037c0 2024-04-26T16:37:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.416+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff4c01037c0 0x7ff4c0103be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c01037c0 2024-04-26T16:37:12.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff4c01037c0 0x7ff4c0103be0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4ac070a50 con 0x7ff4c01037c0 2024-04-26T16:37:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 5: 0 ) 2024-04-26T16:37:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 5 = system:0 2024-04-26T16:37:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 4 2024-04-26T16:37:12.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.420+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 2 5 2024-04-26T16:37:12.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 10 client.31065.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 6 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 6 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c002dae0 osd=2 3 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 2 6 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 4 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 10 client.31065.objecter _send_command 6 2024-04-26T16:37:12.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c0104360 con 0x7ff4c01037c0 2024-04-26T16:37:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c0104360 con 0x7ff4c01037c0 2024-04-26T16:37:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 4 2024-04-26T16:37:12.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 2 6 2024-04-26T16:37:12.442 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 10 client.31065.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 7 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff4c0103320 0x7ff4c01aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 7 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 3 7 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 4 2024-04-26T16:37:12.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 10 client.31065.objecter _send_command 7 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.440+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7ff4c01b1d40 con 0x7ff4c0103320 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff4c0103320 0x7ff4c01aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c0103320 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff4c0103320 0x7ff4c01aeee0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4c01b1d40 con 0x7ff4c0103320 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 7: 0 ) 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 7 = system:0 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 4 2024-04-26T16:37:12.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.444+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 3 7 2024-04-26T16:37:12.449 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 10 client.31065.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 8 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 8 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c002d7b0 osd=3 3 2024-04-26T16:37:12.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 3 8 2024-04-26T16:37:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 4 2024-04-26T16:37:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 10 client.31065.objecter _send_command 8 2024-04-26T16:37:12.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.460+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c01028f0 con 0x7ff4c0103320 2024-04-26T16:37:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c01028f0 con 0x7ff4c0103320 2024-04-26T16:37:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 4 2024-04-26T16:37:12.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 3 8 2024-04-26T16:37:12.463 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 10 client.31065.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 9 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff4c01aca30 0x7ff4c01b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 9 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 4 9 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 4 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 10 client.31065.objecter _send_command 9 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7ff4c0102940 con 0x7ff4c01aca30 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff4c01aca30 0x7ff4c01b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c01aca30 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff4c01aca30 0x7ff4c01b25b0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4b4030390 con 0x7ff4c01aca30 2024-04-26T16:37:12.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 9: 0 ) 2024-04-26T16:37:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 9 = system:0 2024-04-26T16:37:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 4 2024-04-26T16:37:12.466 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.464+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 4 9 2024-04-26T16:37:12.468 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 10 client.31065.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 10 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 10 2024-04-26T16:37:12.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c0102640 osd=4 3 2024-04-26T16:37:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 4 10 2024-04-26T16:37:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 4 2024-04-26T16:37:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 10 client.31065.objecter _send_command 10 2024-04-26T16:37:12.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.484+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c01abfa0 con 0x7ff4c01aca30 2024-04-26T16:37:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c01abfa0 con 0x7ff4c01aca30 2024-04-26T16:37:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 4 2024-04-26T16:37:12.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 4 10 2024-04-26T16:37:12.487 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:37:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 10 client.31065.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 11 2024-04-26T16:37:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff4c01b0100 0x7ff4c01b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 11 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 5 11 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 4 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 10 client.31065.objecter _send_command 11 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7ff4c01abec0 con 0x7ff4c01b0100 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff4c01b0100 0x7ff4c01b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c01b0100 2024-04-26T16:37:12.488 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bffff700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff4c01b0100 0x7ff4c01b3c70 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4c01abec0 con 0x7ff4c01b0100 2024-04-26T16:37:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 11: 0 ) 2024-04-26T16:37:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 11 = system:0 2024-04-26T16:37:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 4 2024-04-26T16:37:12.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.488+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 5 11 2024-04-26T16:37:12.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 10 client.31065.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 12 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 12 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c01abd50 osd=5 3 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 5 12 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 4 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 10 client.31065.objecter _send_command 12 2024-04-26T16:37:12.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.504+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c01abec0 con 0x7ff4c01b0100 2024-04-26T16:37:12.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c01abec0 con 0x7ff4c01b0100 2024-04-26T16:37:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 4 2024-04-26T16:37:12.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 5 12 2024-04-26T16:37:12.508 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 10 client.31065.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 13 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff4c01b46c0 0x7ff4c01b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 13 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 6 13 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 4 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 10 client.31065.objecter _send_command 13 2024-04-26T16:37:12.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7ff4c01b5170 con 0x7ff4c01b46c0 2024-04-26T16:37:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff4c01b46c0 0x7ff4c01b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4c5712700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff4c01b46c0 0x7ff4c01b4b20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c01b46c0 2024-04-26T16:37:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4c01b5170 con 0x7ff4c01b46c0 2024-04-26T16:37:12.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 13: 0 ) 2024-04-26T16:37:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 13 = system:0 2024-04-26T16:37:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 4 2024-04-26T16:37:12.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.508+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 6 13 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 10 client.31065.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 14 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 14 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c01abec0 osd=6 3 2024-04-26T16:37:12.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 6 14 2024-04-26T16:37:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 4 2024-04-26T16:37:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 10 client.31065.objecter _send_command 14 2024-04-26T16:37:12.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c01b5170 con 0x7ff4c01b46c0 2024-04-26T16:37:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c01b5170 con 0x7ff4c01b46c0 2024-04-26T16:37:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 4 2024-04-26T16:37:12.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 6 14 2024-04-26T16:37:12.530 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:37:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 10 client.31065.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 15 2024-04-26T16:37:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff4c01b99a0 0x7ff4c01bbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:12.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 15 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 7 15 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 4 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 10 client.31065.objecter _send_command 15 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.528+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7ff4c01becf0 con 0x7ff4c01b99a0 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff4c01b99a0 0x7ff4c01bbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4bdffb700 10 client.31065.objecter ms_handle_connect 0x7ff4c01b99a0 2024-04-26T16:37:12.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4c4f11700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff4c01b99a0 0x7ff4c01bbd90 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4c01becf0 con 0x7ff4c01b99a0 2024-04-26T16:37:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 15: 0 ) 2024-04-26T16:37:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 15 = system:0 2024-04-26T16:37:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 4 2024-04-26T16:37:12.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.532+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 7 15 2024-04-26T16:37:12.534 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:37:12.534 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:37:12.534 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:37:12.534 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 10 client.31065.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign -1 16 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 20 client.31065.objecter _get_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_remove -1 16 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 20 client.31065.objecter get_session s=0x7ff4c01ac150 osd=7 3 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 15 client.31065.objecter _session_command_op_assign 7 16 2024-04-26T16:37:12.549 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 4 2024-04-26T16:37:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 10 client.31065.objecter _send_command 16 2024-04-26T16:37:12.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.548+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7ff4c01bc320 con 0x7ff4c01b99a0 2024-04-26T16:37:12.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4bdffb700 1 -- 172.21.15.50:0/375403157 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7ff4c01bc320 con 0x7ff4c01b99a0 2024-04-26T16:37:12.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4bdffb700 10 client.31065.objecter ms_dispatch 0x7ff4c0001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:12.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4bdffb700 10 client.31065.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:12.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4bdffb700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 4 2024-04-26T16:37:12.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4bdffb700 15 client.31065.objecter _session_command_op_remove 7 16 2024-04-26T16:37:12.551 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:37:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 librados: watch_flush enter 2024-04-26T16:37:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 librados: watch_flush exit 2024-04-26T16:37:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.0 2024-04-26T16:37:12.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff4c0064030 msgr2=0x7ff4c0064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7ff4c0064030 0x7ff4c0064410 crc :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c019e670 osd=0 1 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.1 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff4c010a9d0 msgr2=0x7ff4c010add0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7ff4c010a9d0 0x7ff4c010add0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0063d00 osd=1 1 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.2 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff4c01037c0 msgr2=0x7ff4c0103be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7ff4c01037c0 0x7ff4c0103be0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002dae0 osd=2 1 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.3 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff4c0103320 msgr2=0x7ff4c01aeee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7ff4c0103320 0x7ff4c01aeee0 crc :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c002d7b0 osd=3 1 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.4 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff4c01aca30 msgr2=0x7ff4c01b25b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7ff4c01aca30 0x7ff4c01b25b0 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c0102640 osd=4 1 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.5 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff4c01b0100 msgr2=0x7ff4c01b3c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c5712700 1 -- 172.21.15.50:0/375403157 reap_dead start 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7ff4c01b0100 0x7ff4c01b3c70 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c5712700 1 -- 172.21.15.50:0/375403157 reap_dead start 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abd50 osd=5 1 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.6 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c5712700 1 -- 172.21.15.50:0/375403157 reap_dead start 2024-04-26T16:37:12.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff4c01b46c0 msgr2=0x7ff4c01b4b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff4c01b46c0 0x7ff4c01b4b20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01abec0 osd=6 1 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter close_session for osd.7 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff4c01b99a0 msgr2=0x7ff4c01bbd90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff4c01b99a0 0x7ff4c01bbd90 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter put_session s=0x7ff4c01ac150 osd=7 1 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 20 client.31065.objecter shutdown clearing up homeless session... 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 10 client.31065.objecter successfully canceled tick 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff4b0072120 msgr2=0x7ff4b00745e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff4b0072120 0x7ff4b00745e0 secure :-1 s=READY pgs=3839 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4006010 tx=0x7ff4b400b630 comp rx=0 tx=0).stop 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006bad0 msgr2=0x7ff4c0199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006bad0 0x7ff4c0199730 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac010d00 tx=0x7ff4ac00b5a0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 shutdown_connections 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7ff4c01b99a0 0x7ff4c01bbd90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ff4c006bad0 0x7ff4c0199730 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff4b0072120 0x7ff4b00745e0 unknown :-1 s=CLOSED pgs=3839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 --2- 172.21.15.50:0/375403157 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7ff4c01b46c0 0x7ff4c01b4b20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 >> 172.21.15.50:0/375403157 conn(0x7ff4c00792d0 msgr2=0x7ff4c0100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 shutdown_connections 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 -- 172.21.15.50:0/375403157 wait complete. 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:12.552+0000 7ff4c7175700 1 librados: shutdown 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:37:12.556 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:37:12.557 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:37:12.557 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:37:12.557 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:37:12.557 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:37:12.557 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:37:12.572 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:12.593 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:12.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:12.723 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:12.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:12.857 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:12.960 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:12.986 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:12.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: cluster 2024-04-26T16:37:11.946175+0000 mgr.y (mgr.24401) 758 : cluster [DBG] pgmap v1301: 204 pgs: 31 creating+peering, 1 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: cluster 2024-04-26T16:37:11.946175+0000 mgr.y (mgr.24401) 758 : cluster [DBG] pgmap v1301: 204 pgs: 31 creating+peering, 1 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.352411+0000 mon.a (mon.0) 3836 : audit [DBG] from='client.? 172.21.15.50:0/375403157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.352411+0000 mon.a (mon.0) 3836 : audit [DBG] from='client.? 172.21.15.50:0/375403157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.395150+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.395150+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: cluster 2024-04-26T16:37:12.399374+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: cluster 2024-04-26T16:37:12.399374+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.429958+0000 mon.c (mon.2) 867 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.429958+0000 mon.c (mon.2) 867 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.430376+0000 mon.a (mon.0) 3839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:12 smithi190 bash[23868]: audit 2024-04-26T16:37:12.430376+0000 mon.a (mon.0) 3839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:13.117 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: cluster 2024-04-26T16:37:11.946175+0000 mgr.y (mgr.24401) 758 : cluster [DBG] pgmap v1301: 204 pgs: 31 creating+peering, 1 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: cluster 2024-04-26T16:37:11.946175+0000 mgr.y (mgr.24401) 758 : cluster [DBG] pgmap v1301: 204 pgs: 31 creating+peering, 1 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.352411+0000 mon.a (mon.0) 3836 : audit [DBG] from='client.? 172.21.15.50:0/375403157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.352411+0000 mon.a (mon.0) 3836 : audit [DBG] from='client.? 172.21.15.50:0/375403157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.395150+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.395150+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: cluster 2024-04-26T16:37:12.399374+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: cluster 2024-04-26T16:37:12.399374+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.429958+0000 mon.c (mon.2) 867 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.429958+0000 mon.c (mon.2) 867 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.430376+0000 mon.a (mon.0) 3839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[21463]: audit 2024-04-26T16:37:12.430376+0000 mon.a (mon.0) 3839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: cluster 2024-04-26T16:37:11.946175+0000 mgr.y (mgr.24401) 758 : cluster [DBG] pgmap v1301: 204 pgs: 31 creating+peering, 1 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:13.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: cluster 2024-04-26T16:37:11.946175+0000 mgr.y (mgr.24401) 758 : cluster [DBG] pgmap v1301: 204 pgs: 31 creating+peering, 1 unknown, 172 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.352411+0000 mon.a (mon.0) 3836 : audit [DBG] from='client.? 172.21.15.50:0/375403157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.352411+0000 mon.a (mon.0) 3836 : audit [DBG] from='client.? 172.21.15.50:0/375403157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.395150+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.395150+0000 mon.a (mon.0) 3837 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: cluster 2024-04-26T16:37:12.399374+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: cluster 2024-04-26T16:37:12.399374+0000 mon.a (mon.0) 3838 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.429958+0000 mon.c (mon.2) 867 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.429958+0000 mon.c (mon.2) 867 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.430376+0000 mon.a (mon.0) 3839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:12 smithi050 bash[29234]: audit 2024-04-26T16:37:12.430376+0000 mon.a (mon.0) 3839 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:13.221 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:13.250 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:13.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:13.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-04-26T16:37:13.381 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:13.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:13.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:13.509 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:13.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:13.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:13.803 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:13.827 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:13.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:13.961 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:13.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:14.064 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:14.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:12.940747+0000 mon.a (mon.0) 3840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:14.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:12.940747+0000 mon.a (mon.0) 3840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:14.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:12.943163+0000 mon.c (mon.2) 868 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:14.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:12.943163+0000 mon.c (mon.2) 868 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:13.398171+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:13.398171+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.406487+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.406487+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:13.409021+0000 mon.c (mon.2) 869 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:13.409021+0000 mon.c (mon.2) 869 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:13.412018+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: audit 2024-04-26T16:37:13.412018+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.746755+0000 osd.1 (osd.1) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.746755+0000 osd.1 (osd.1) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.747224+0000 osd.1 (osd.1) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:14.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.747224+0000 osd.1 (osd.1) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:12.940747+0000 mon.a (mon.0) 3840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:12.940747+0000 mon.a (mon.0) 3840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:12.943163+0000 mon.c (mon.2) 868 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:12.943163+0000 mon.c (mon.2) 868 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:13.398171+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:13.398171+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.406487+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.406487+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:13.409021+0000 mon.c (mon.2) 869 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:13.409021+0000 mon.c (mon.2) 869 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:13.412018+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: audit 2024-04-26T16:37:13.412018+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.746755+0000 osd.1 (osd.1) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:14.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.746755+0000 osd.1 (osd.1) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:14.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.747224+0000 osd.1 (osd.1) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:14.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:13 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.747224+0000 osd.1 (osd.1) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:14.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:14.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:14.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:12.940747+0000 mon.a (mon.0) 3840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:14.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:12.940747+0000 mon.a (mon.0) 3840 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:14.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:12.943163+0000 mon.c (mon.2) 868 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:14.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:12.943163+0000 mon.c (mon.2) 868 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:14.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:13.398171+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:14.208 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:13.398171+0000 mon.a (mon.0) 3841 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.406487+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.406487+0000 mon.a (mon.0) 3842 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:13.409021+0000 mon.c (mon.2) 869 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:13.409021+0000 mon.c (mon.2) 869 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:13.412018+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: audit 2024-04-26T16:37:13.412018+0000 mon.a (mon.0) 3843 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.746755+0000 osd.1 (osd.1) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.746755+0000 osd.1 (osd.1) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.747224+0000 osd.1 (osd.1) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:14.209 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:13 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.747224+0000 osd.1 (osd.1) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:14.225 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:14.292 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:14.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:14.353 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:14.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:14.486 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:14.589 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:14.600 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:37:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:37:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:37:14.614 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:14.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:14.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:14.745 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:14.848 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:14.873 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:14.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:14.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:15.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:15.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:15.107 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:15.134 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:15.237 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:15.266 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:15.310 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:15.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:15.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:15.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:15.524 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:15.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:15.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.947256+0000 mgr.y (mgr.24401) 759 : cluster [DBG] pgmap v1304: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: cluster 2024-04-26T16:37:13.947256+0000 mgr.y (mgr.24401) 759 : cluster [DBG] pgmap v1304: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: audit 2024-04-26T16:37:14.401165+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: audit 2024-04-26T16:37:14.401165+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: audit 2024-04-26T16:37:14.411719+0000 mon.c (mon.2) 870 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: audit 2024-04-26T16:37:14.411719+0000 mon.c (mon.2) 870 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: cluster 2024-04-26T16:37:14.413780+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: cluster 2024-04-26T16:37:14.413780+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: audit 2024-04-26T16:37:14.416380+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[21463]: audit 2024-04-26T16:37:14.416380+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.947256+0000 mgr.y (mgr.24401) 759 : cluster [DBG] pgmap v1304: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: cluster 2024-04-26T16:37:13.947256+0000 mgr.y (mgr.24401) 759 : cluster [DBG] pgmap v1304: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:15.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: audit 2024-04-26T16:37:14.401165+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: audit 2024-04-26T16:37:14.401165+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: audit 2024-04-26T16:37:14.411719+0000 mon.c (mon.2) 870 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: audit 2024-04-26T16:37:14.411719+0000 mon.c (mon.2) 870 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: cluster 2024-04-26T16:37:14.413780+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: cluster 2024-04-26T16:37:14.413780+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: audit 2024-04-26T16:37:14.416380+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:15 smithi050 bash[29234]: audit 2024-04-26T16:37:14.416380+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.947256+0000 mgr.y (mgr.24401) 759 : cluster [DBG] pgmap v1304: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: cluster 2024-04-26T16:37:13.947256+0000 mgr.y (mgr.24401) 759 : cluster [DBG] pgmap v1304: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: audit 2024-04-26T16:37:14.401165+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: audit 2024-04-26T16:37:14.401165+0000 mon.a (mon.0) 3844 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi050-73372-111", "overlaypool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: audit 2024-04-26T16:37:14.411719+0000 mon.c (mon.2) 870 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: audit 2024-04-26T16:37:14.411719+0000 mon.c (mon.2) 870 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: cluster 2024-04-26T16:37:14.413780+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: cluster 2024-04-26T16:37:14.413780+0000 mon.a (mon.0) 3845 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: audit 2024-04-26T16:37:14.416380+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:15 smithi190 bash[23868]: audit 2024-04-26T16:37:14.416380+0000 mon.a (mon.0) 3846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]: dispatch 2024-04-26T16:37:15.653 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:15.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:15.784 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:15.808 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-04-26T16:37:15.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:15.914 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:15.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:16.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:16.037 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:16.140 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:16.168 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:16.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:16.271 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:16.297 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:16.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:16.434 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: cluster 2024-04-26T16:37:15.401746+0000 mon.a (mon.0) 3847 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: cluster 2024-04-26T16:37:15.401746+0000 mon.a (mon.0) 3847 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: audit 2024-04-26T16:37:15.404425+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]': finished 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: audit 2024-04-26T16:37:15.404425+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]': finished 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: cluster 2024-04-26T16:37:15.411437+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: cluster 2024-04-26T16:37:15.411437+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: audit 2024-04-26T16:37:15.497686+0000 mon.c (mon.2) 871 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.435 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: audit 2024-04-26T16:37:15.497686+0000 mon.c (mon.2) 871 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: audit 2024-04-26T16:37:15.498430+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.436 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: audit 2024-04-26T16:37:15.498430+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: cluster 2024-04-26T16:37:15.401746+0000 mon.a (mon.0) 3847 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:37:16.436 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: cluster 2024-04-26T16:37:15.401746+0000 mon.a (mon.0) 3847 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: audit 2024-04-26T16:37:15.404425+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]': finished 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: audit 2024-04-26T16:37:15.404425+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]': finished 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: cluster 2024-04-26T16:37:15.411437+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: cluster 2024-04-26T16:37:15.411437+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: audit 2024-04-26T16:37:15.497686+0000 mon.c (mon.2) 871 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: audit 2024-04-26T16:37:15.497686+0000 mon.c (mon.2) 871 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: audit 2024-04-26T16:37:15.498430+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: audit 2024-04-26T16:37:15.498430+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: cluster 2024-04-26T16:37:16.392573+0000 mon.a (mon.0) 3851 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:16.437 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[21463]: cluster 2024-04-26T16:37:16.392573+0000 mon.a (mon.0) 3851 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:16.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:16.566 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:16.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:16.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: cluster 2024-04-26T16:37:15.401746+0000 mon.a (mon.0) 3847 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:37:16.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: cluster 2024-04-26T16:37:15.401746+0000 mon.a (mon.0) 3847 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-04-26T16:37:16.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: audit 2024-04-26T16:37:15.404425+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]': finished 2024-04-26T16:37:16.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: audit 2024-04-26T16:37:15.404425+0000 mon.a (mon.0) 3848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi050-73372-146", "mode": "writeback"}]': finished 2024-04-26T16:37:16.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: cluster 2024-04-26T16:37:15.411437+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-04-26T16:37:16.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: cluster 2024-04-26T16:37:15.411437+0000 mon.a (mon.0) 3849 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2024-04-26T16:37:16.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: audit 2024-04-26T16:37:15.497686+0000 mon.c (mon.2) 871 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: audit 2024-04-26T16:37:15.497686+0000 mon.c (mon.2) 871 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: audit 2024-04-26T16:37:15.498430+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: audit 2024-04-26T16:37:15.498430+0000 mon.a (mon.0) 3850 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]: dispatch 2024-04-26T16:37:16.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: cluster 2024-04-26T16:37:16.392573+0000 mon.a (mon.0) 3851 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:16.682 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:16 smithi190 bash[23868]: cluster 2024-04-26T16:37:16.392573+0000 mon.a (mon.0) 3851 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:16.697 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:16.800 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:16.826 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:16.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: cluster 2024-04-26T16:37:16.392573+0000 mon.a (mon.0) 3851 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:16.896 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:16 smithi050 bash[29234]: cluster 2024-04-26T16:37:16.392573+0000 mon.a (mon.0) 3851 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:16.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:16.957 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:16.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:16.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:17.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:17.092 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:17.195 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:17.221 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:17.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:17.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:17.350 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:17.454 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:17.465 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: cluster 2024-04-26T16:37:15.948205+0000 mgr.y (mgr.24401) 760 : cluster [DBG] pgmap v1307: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:17.465 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: cluster 2024-04-26T16:37:15.948205+0000 mgr.y (mgr.24401) 760 : cluster [DBG] pgmap v1307: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:17.465 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: audit 2024-04-26T16:37:16.428437+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: audit 2024-04-26T16:37:16.428437+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: cluster 2024-04-26T16:37:16.439658+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: cluster 2024-04-26T16:37:16.439658+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: audit 2024-04-26T16:37:16.443499+0000 mon.c (mon.2) 872 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: audit 2024-04-26T16:37:16.443499+0000 mon.c (mon.2) 872 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: audit 2024-04-26T16:37:16.444487+0000 mon.a (mon.0) 3854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[21463]: audit 2024-04-26T16:37:16.444487+0000 mon.a (mon.0) 3854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: cluster 2024-04-26T16:37:15.948205+0000 mgr.y (mgr.24401) 760 : cluster [DBG] pgmap v1307: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: cluster 2024-04-26T16:37:15.948205+0000 mgr.y (mgr.24401) 760 : cluster [DBG] pgmap v1307: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:17.466 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: audit 2024-04-26T16:37:16.428437+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: audit 2024-04-26T16:37:16.428437+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: cluster 2024-04-26T16:37:16.439658+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: cluster 2024-04-26T16:37:16.439658+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: audit 2024-04-26T16:37:16.443499+0000 mon.c (mon.2) 872 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: audit 2024-04-26T16:37:16.443499+0000 mon.c (mon.2) 872 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: audit 2024-04-26T16:37:16.444487+0000 mon.a (mon.0) 3854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.467 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:17 smithi050 bash[29234]: audit 2024-04-26T16:37:16.444487+0000 mon.a (mon.0) 3854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.478 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:17.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: cluster 2024-04-26T16:37:15.948205+0000 mgr.y (mgr.24401) 760 : cluster [DBG] pgmap v1307: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: cluster 2024-04-26T16:37:15.948205+0000 mgr.y (mgr.24401) 760 : cluster [DBG] pgmap v1307: 204 pgs: 31 creating+peering, 173 active+clean; 4.4 MiB data, 789 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: audit 2024-04-26T16:37:16.428437+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: audit 2024-04-26T16:37:16.428437+0000 mon.a (mon.0) 3852 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_count","val": "2"}]': finished 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: cluster 2024-04-26T16:37:16.439658+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: cluster 2024-04-26T16:37:16.439658+0000 mon.a (mon.0) 3853 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: audit 2024-04-26T16:37:16.443499+0000 mon.c (mon.2) 872 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: audit 2024-04-26T16:37:16.443499+0000 mon.c (mon.2) 872 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: audit 2024-04-26T16:37:16.444487+0000 mon.a (mon.0) 3854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:17 smithi190 bash[23868]: audit 2024-04-26T16:37:16.444487+0000 mon.a (mon.0) 3854 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]: dispatch 2024-04-26T16:37:17.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:17.611 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:17.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:17.742 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:17.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:17.873 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:17.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:17.976 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:18.004 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:18.107 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:18.134 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:18.144 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:37:18.238 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:18.260 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:18.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:18.390 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:18.493 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:18.523 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:18.626 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.325727+0000 mgr.y (mgr.24401) 761 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.325727+0000 mgr.y (mgr.24401) 761 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.440372+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.440372+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: cluster 2024-04-26T16:37:17.448371+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: cluster 2024-04-26T16:37:17.448371+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.449572+0000 mon.c (mon.2) 873 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.449572+0000 mon.c (mon.2) 873 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.452618+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[21463]: audit 2024-04-26T16:37:17.452618+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.325727+0000 mgr.y (mgr.24401) 761 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.325727+0000 mgr.y (mgr.24401) 761 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.440372+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.440372+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: cluster 2024-04-26T16:37:17.448371+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: cluster 2024-04-26T16:37:17.448371+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.449572+0000 mon.c (mon.2) 873 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.449572+0000 mon.c (mon.2) 873 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.452618+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:18 smithi050 bash[29234]: audit 2024-04-26T16:37:17.452618+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.654 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:18.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:18.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.325727+0000 mgr.y (mgr.24401) 761 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:18.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.325727+0000 mgr.y (mgr.24401) 761 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:18.771 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.440372+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.440372+0000 mon.a (mon.0) 3855 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_period","val": "600"}]': finished 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: cluster 2024-04-26T16:37:17.448371+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: cluster 2024-04-26T16:37:17.448371+0000 mon.a (mon.0) 3856 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.449572+0000 mon.c (mon.2) 873 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.449572+0000 mon.c (mon.2) 873 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.452618+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.772 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:18 smithi190 bash[23868]: audit 2024-04-26T16:37:17.452618+0000 mon.a (mon.0) 3857 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-04-26T16:37:18.787 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:18.890 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:18.921 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:18.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:19.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:19.053 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:19.156 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:19.185 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:19.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:19.314 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:19.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:19.437 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:19.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: cluster 2024-04-26T16:37:17.949085+0000 mgr.y (mgr.24401) 762 : cluster [DBG] pgmap v1310: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: cluster 2024-04-26T16:37:17.949085+0000 mgr.y (mgr.24401) 762 : cluster [DBG] pgmap v1310: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: cluster 2024-04-26T16:37:18.441536+0000 mon.a (mon.0) 3858 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: cluster 2024-04-26T16:37:18.441536+0000 mon.a (mon.0) 3858 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: audit 2024-04-26T16:37:18.449527+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: audit 2024-04-26T16:37:18.449527+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:37:19.549 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: cluster 2024-04-26T16:37:18.461784+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-04-26T16:37:19.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: cluster 2024-04-26T16:37:18.461784+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-04-26T16:37:19.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: audit 2024-04-26T16:37:18.463339+0000 mon.c (mon.2) 874 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: audit 2024-04-26T16:37:18.463339+0000 mon.c (mon.2) 874 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: audit 2024-04-26T16:37:18.464455+0000 mon.a (mon.0) 3861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.550 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:19 smithi190 bash[23868]: audit 2024-04-26T16:37:18.464455+0000 mon.a (mon.0) 3861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.561 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:19.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:19.664 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:19.692 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:19.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: cluster 2024-04-26T16:37:17.949085+0000 mgr.y (mgr.24401) 762 : cluster [DBG] pgmap v1310: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: cluster 2024-04-26T16:37:17.949085+0000 mgr.y (mgr.24401) 762 : cluster [DBG] pgmap v1310: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: cluster 2024-04-26T16:37:18.441536+0000 mon.a (mon.0) 3858 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: cluster 2024-04-26T16:37:18.441536+0000 mon.a (mon.0) 3858 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: audit 2024-04-26T16:37:18.449527+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: audit 2024-04-26T16:37:18.449527+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: cluster 2024-04-26T16:37:18.461784+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: cluster 2024-04-26T16:37:18.461784+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: audit 2024-04-26T16:37:18.463339+0000 mon.c (mon.2) 874 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: audit 2024-04-26T16:37:18.463339+0000 mon.c (mon.2) 874 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.808 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: audit 2024-04-26T16:37:18.464455+0000 mon.a (mon.0) 3861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[29234]: audit 2024-04-26T16:37:18.464455+0000 mon.a (mon.0) 3861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: cluster 2024-04-26T16:37:17.949085+0000 mgr.y (mgr.24401) 762 : cluster [DBG] pgmap v1310: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: cluster 2024-04-26T16:37:17.949085+0000 mgr.y (mgr.24401) 762 : cluster [DBG] pgmap v1310: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: cluster 2024-04-26T16:37:18.441536+0000 mon.a (mon.0) 3858 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: cluster 2024-04-26T16:37:18.441536+0000 mon.a (mon.0) 3858 : cluster [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: audit 2024-04-26T16:37:18.449527+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: audit 2024-04-26T16:37:18.449527+0000 mon.a (mon.0) 3859 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "hit_set_type","val": "bloom"}]': finished 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: cluster 2024-04-26T16:37:18.461784+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: cluster 2024-04-26T16:37:18.461784+0000 mon.a (mon.0) 3860 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: audit 2024-04-26T16:37:18.463339+0000 mon.c (mon.2) 874 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: audit 2024-04-26T16:37:18.463339+0000 mon.c (mon.2) 874 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.809 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: audit 2024-04-26T16:37:18.464455+0000 mon.a (mon.0) 3861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.810 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:19 smithi050 bash[21463]: audit 2024-04-26T16:37:18.464455+0000 mon.a (mon.0) 3861 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-04-26T16:37:19.823 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:19.927 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:19.954 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:19.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:20.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.087 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:20.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:20.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:20.212 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:20.315 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:20.341 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:20.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:20.479 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: audit 2024-04-26T16:37:19.452094+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: audit 2024-04-26T16:37:19.452094+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: cluster 2024-04-26T16:37:19.455787+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: cluster 2024-04-26T16:37:19.455787+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: audit 2024-04-26T16:37:19.462835+0000 mon.c (mon.2) 875 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: audit 2024-04-26T16:37:19.462835+0000 mon.c (mon.2) 875 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: audit 2024-04-26T16:37:19.469098+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:20 smithi190 bash[23868]: audit 2024-04-26T16:37:19.469098+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.582 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: audit 2024-04-26T16:37:19.452094+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: audit 2024-04-26T16:37:19.452094+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: cluster 2024-04-26T16:37:19.455787+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: cluster 2024-04-26T16:37:19.455787+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: audit 2024-04-26T16:37:19.462835+0000 mon.c (mon.2) 875 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: audit 2024-04-26T16:37:19.462835+0000 mon.c (mon.2) 875 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: audit 2024-04-26T16:37:19.469098+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[21463]: audit 2024-04-26T16:37:19.469098+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.593 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: audit 2024-04-26T16:37:19.452094+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: audit 2024-04-26T16:37:19.452094+0000 mon.a (mon.0) 3862 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: cluster 2024-04-26T16:37:19.455787+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: cluster 2024-04-26T16:37:19.455787+0000 mon.a (mon.0) 3863 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: audit 2024-04-26T16:37:19.462835+0000 mon.c (mon.2) 875 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: audit 2024-04-26T16:37:19.462835+0000 mon.c (mon.2) 875 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: audit 2024-04-26T16:37:19.469098+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.594 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:20 smithi050 bash[29234]: audit 2024-04-26T16:37:19.469098+0000 mon.a (mon.0) 3864 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]: dispatch 2024-04-26T16:37:20.607 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:20.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:20.736 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:20.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:20.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:20.969 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:20.997 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:20.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:21.099 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:21.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:21.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:21.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:21.251 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:21.326 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:37:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:37:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:37:21.326 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:37:21.326 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:37:21.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:21.372 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:21.475 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:21.502 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:21.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: cluster 2024-04-26T16:37:19.949685+0000 mgr.y (mgr.24401) 763 : cluster [DBG] pgmap v1313: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:21.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: cluster 2024-04-26T16:37:19.949685+0000 mgr.y (mgr.24401) 763 : cluster [DBG] pgmap v1313: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:21.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: audit 2024-04-26T16:37:20.472874+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:37:21.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: audit 2024-04-26T16:37:20.472874+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:37:21.529 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: cluster 2024-04-26T16:37:20.477299+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-04-26T16:37:21.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: cluster 2024-04-26T16:37:20.477299+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-04-26T16:37:21.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: cluster 2024-04-26T16:37:21.404844+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-04-26T16:37:21.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:21 smithi190 bash[23868]: cluster 2024-04-26T16:37:21.404844+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-04-26T16:37:21.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:21.633 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:21.736 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:21.765 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:21.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: cluster 2024-04-26T16:37:19.949685+0000 mgr.y (mgr.24401) 763 : cluster [DBG] pgmap v1313: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: cluster 2024-04-26T16:37:19.949685+0000 mgr.y (mgr.24401) 763 : cluster [DBG] pgmap v1313: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: audit 2024-04-26T16:37:20.472874+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: audit 2024-04-26T16:37:20.472874+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: cluster 2024-04-26T16:37:20.477299+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: cluster 2024-04-26T16:37:20.477299+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: cluster 2024-04-26T16:37:21.404844+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[21463]: cluster 2024-04-26T16:37:21.404844+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: cluster 2024-04-26T16:37:19.949685+0000 mgr.y (mgr.24401) 763 : cluster [DBG] pgmap v1313: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: cluster 2024-04-26T16:37:19.949685+0000 mgr.y (mgr.24401) 763 : cluster [DBG] pgmap v1313: 204 pgs: 204 active+clean; 4.4 MiB data, 807 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-04-26T16:37:21.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: audit 2024-04-26T16:37:20.472874+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:37:21.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: audit 2024-04-26T16:37:20.472874+0000 mon.a (mon.0) 3865 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-146","var": "target_max_objects","val": "1"}]': finished 2024-04-26T16:37:21.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: cluster 2024-04-26T16:37:20.477299+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-04-26T16:37:21.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: cluster 2024-04-26T16:37:20.477299+0000 mon.a (mon.0) 3866 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2024-04-26T16:37:21.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: cluster 2024-04-26T16:37:21.404844+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-04-26T16:37:21.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:21 smithi050 bash[29234]: cluster 2024-04-26T16:37:21.404844+0000 mon.a (mon.0) 3867 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2024-04-26T16:37:21.868 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:21.894 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:21.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:21.998 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:22.025 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:22.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-04-26T16:37:22.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:22.189 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:22.293 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:22.316 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:22.419 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- 172.21.15.50:0/3037382098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa0341034d0 msgr2=0x7fa034103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- 172.21.15.50:0/3037382098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa0341034d0 0x7fa034103950 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7fa028009990 tx=0x7fa028009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.439 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- 172.21.15.50:0/3037382098 shutdown_connections 2024-04-26T16:37:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- 172.21.15.50:0/3037382098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa034103e90 0x7fa0341102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- 172.21.15.50:0/3037382098 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0341094f0 0x7fa0341098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- 172.21.15.50:0/3037382098 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa0341034d0 0x7fa034103950 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- 172.21.15.50:0/3037382098 >> 172.21.15.50:0/3037382098 conn(0x7fa03407a500 msgr2=0x7fa03407a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- 172.21.15.50:0/3037382098 shutdown_connections 2024-04-26T16:37:22.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- 172.21.15.50:0/3037382098 wait complete. 2024-04-26T16:37:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 librados: starting msgr at 2024-04-26T16:37:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 librados: starting objecter 2024-04-26T16:37:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 Processor -- start 2024-04-26T16:37:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- start start 2024-04-26T16:37:22.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 librados: setting wanted keys 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 librados: calling monclient init 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0341034d0 0x7fa034199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 0x7fa034199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0341094f0 0x7fa03419da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fa034112ad0 con 0x7fa0341094f0 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fa034112950 con 0x7fa0341034d0 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fa034112c50 con 0x7fa034103e90 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 0x7fa034199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 0x7fa034199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60806/0 (socket says 172.21.15.50:60806) 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 -- 172.21.15.50:0/3220106066 learned_addr learned my addr 172.21.15.50:0/3220106066 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa033fff700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0341094f0 0x7fa03419da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa0337fe700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0341034d0 0x7fa034199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 -- 172.21.15.50:0/3220106066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0341034d0 msgr2=0x7fa034199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0341034d0 0x7fa034199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 -- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0341094f0 msgr2=0x7fa03419da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0341094f0 0x7fa03419da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 -- 172.21.15.50:0/3220106066 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03419e140 con 0x7fa034103e90 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa032ffd700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 0x7fa034199670 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa028004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa030ff9700 10 client.?.objecter ms_handle_connect 0x7fa034103e90 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa030ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:22.443 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028027070 con 0x7fa034103e90 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fa028022470 con 0x7fa034103e90 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03419e3d0 con 0x7fa034103e90 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.440+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028017650 con 0x7fa034103e90 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa03a020700 10 client.41690.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0341a5cb0 con 0x7fa034103e90 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa03a020700 1 librados: init done 2024-04-26T16:37:22.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa03406a110 con 0x7fa034103e90 2024-04-26T16:37:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fa02802a070 con 0x7fa034103e90 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa01c071f80 0x7fa01c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(845..845 src has 577..845) ==== 9973+0+0 (secure 0 0 0) 0x7fa028092f60 con 0x7fa034103e90 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 10 client.41690.objecter ms_dispatch 0x7fa034001550 osd_map(845..845 src has 577..845) 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 3 client.41690.objecter handle_osd_map got epochs [845,845] > 0 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 3 client.41690.objecter handle_osd_map decoding full epoch 845 2024-04-26T16:37:22.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa0337fe700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa01c071f80 0x7fa01c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 20 client.41690.objecter dump_active .. 0 homeless 2024-04-26T16:37:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa0337fe700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa01c071f80 0x7fa01c074440 secure :-1 s=READY pgs=3840 cs=0 l=1 rev1=1 crypto rx=0x7fa024009da0 tx=0x7fa02400b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:22.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.444+0000 7fa030ff9700 10 client.41690.objecter ms_handle_connect 0x7fa01c071f80 2024-04-26T16:37:22.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.448+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fa028031080 con 0x7fa034103e90 2024-04-26T16:37:22.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:22.573 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:22.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:22 smithi050 bash[21463]: cluster 2024-04-26T16:37:22.474637+0000 mon.a (mon.0) 3868 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:37:22.603 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:22 smithi050 bash[21463]: cluster 2024-04-26T16:37:22.474637+0000 mon.a (mon.0) 3868 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:37:22.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:22 smithi050 bash[29234]: cluster 2024-04-26T16:37:22.474637+0000 mon.a (mon.0) 3868 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:37:22.603 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:22 smithi050 bash[29234]: cluster 2024-04-26T16:37:22.474637+0000 mon.a (mon.0) 3868 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:37:22.676 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:22.696 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:22.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.704+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7fa0341a5e80 con 0x7fa034103e90 2024-04-26T16:37:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- 172.21.15.50:0/204866160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3784103e60 msgr2=0x7f37841102b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- 172.21.15.50:0/204866160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3784103e60 0x7f37841102b0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f377c009f90 tx=0x7f377c00b330 comp rx=0 tx=0).stop 2024-04-26T16:37:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- 172.21.15.50:0/204866160 shutdown_connections 2024-04-26T16:37:22.757 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- 172.21.15.50:0/204866160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37841094e0 0x7f37841098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- 172.21.15.50:0/204866160 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3784103e60 0x7f37841102b0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- 172.21.15.50:0/204866160 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37841034a0 0x7f3784103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- 172.21.15.50:0/204866160 >> 172.21.15.50:0/204866160 conn(0x7f378407a590 msgr2=0x7f378407a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- 172.21.15.50:0/204866160 shutdown_connections 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- 172.21.15.50:0/204866160 wait complete. 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 librados: starting msgr at 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 librados: starting objecter 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 Processor -- start 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- start start 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 librados: setting wanted keys 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 librados: calling monclient init 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 0x7f3784199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.758 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3784103e60 0x7f3784199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37841094e0 0x7f378419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3784112a80 con 0x7f37841094e0 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3784112900 con 0x7f3784103e60 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378ab7f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3784112c00 con 0x7f37841034a0 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 0x7f3784199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378911c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37841094e0 0x7f378419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 0x7f3784199120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60828/0 (socket says 172.21.15.50:60828) 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 -- 172.21.15.50:0/2170655095 learned_addr learned my addr 172.21.15.50:0/2170655095 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f3783fff700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3784103e60 0x7f3784199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 -- 172.21.15.50:0/2170655095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3784103e60 msgr2=0x7f3784199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3784103e60 0x7f3784199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 -- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37841094e0 msgr2=0x7f378419d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:22.759 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37841094e0 0x7f378419d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:22.760 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 -- 172.21.15.50:0/2170655095 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378419e130 con 0x7f37841034a0 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f378891b700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 0x7f3784199120 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f377800d950 tx=0x7f377800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f3781ffb700 10 client.?.objecter ms_handle_connect 0x7f37841034a0 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.756+0000 7f3781ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778016070 con 0x7f37841034a0 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3778009da0 con 0x7f37841034a0 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3778010e60 con 0x7f37841034a0 2024-04-26T16:37:22.761 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378419e420 con 0x7f37841034a0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f378ab7f700 10 client.41696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f37841a5ca0 con 0x7f37841034a0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f378ab7f700 1 librados: init done 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f378406a110 con 0x7f37841034a0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3778019070 con 0x7f37841034a0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3774071f80 0x7f3774074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(845..845 src has 577..845) ==== 9973+0+0 (secure 0 0 0) 0x7f37780916c0 con 0x7f37841034a0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 10 client.41696.objecter ms_dispatch 0x7f3784001550 osd_map(845..845 src has 577..845) 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 3 client.41696.objecter handle_osd_map got epochs [845,845] > 0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 3 client.41696.objecter handle_osd_map decoding full epoch 845 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3783fff700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3774071f80 0x7f3774074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 20 client.41696.objecter dump_active .. 0 homeless 2024-04-26T16:37:22.762 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3781ffb700 10 client.41696.objecter ms_handle_connect 0x7f3774071f80 2024-04-26T16:37:22.763 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.760+0000 7f3783fff700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3774071f80 0x7f3774074440 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 crypto rx=0x7f3770009750 tx=0x7f3770009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:22.765 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.764+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f377805cf40 con 0x7f37841034a0 2024-04-26T16:37:22.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:22.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:22 smithi190 bash[23868]: cluster 2024-04-26T16:37:22.474637+0000 mon.a (mon.0) 3868 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:37:22.811 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:22 smithi190 bash[23868]: cluster 2024-04-26T16:37:22.474637+0000 mon.a (mon.0) 3868 : cluster [WRN] Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-04-26T16:37:22.826 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:22.929 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:22.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:22.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:22.998 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:22.996+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd set", "key": "nodeep-scrub"} v 0) -- 0x7f37841a5e70 con 0x7f37841034a0 2024-04-26T16:37:23.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:23.085 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:23.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:23.212 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:23.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:23.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:23.343 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:23.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:23.473 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:23.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:23.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.544+0000 7f3781ffb700 1 -- 172.21.15.50:0/2170655095 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd set", "key": "nodeep-scrub"}]=0 nodeep-scrub is set v846) ==== 97+0+0 (secure 0 0 0) 0x7f3778060bf0 con 0x7f37841034a0 2024-04-26T16:37:23.547 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.548+0000 7fa030ff9700 1 -- 172.21.15.50:0/3220106066 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v846) ==== 113+0+0 (secure 0 0 0) 0x7fa028013070 con 0x7fa034103e90 2024-04-26T16:37:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.548+0000 7f378ab7f700 10 librados: watch_flush enter 2024-04-26T16:37:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.548+0000 7f378ab7f700 10 librados: watch_flush exit 2024-04-26T16:37:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.548+0000 7f378ab7f700 20 client.41696.objecter shutdown clearing up homeless session... 2024-04-26T16:37:23.550 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.548+0000 7f378ab7f700 10 client.41696.objecter successfully canceled tick 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3774071f80 msgr2=0x7f3774074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3774071f80 0x7f3774074440 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 crypto rx=0x7f3770009750 tx=0x7f3770009310 comp rx=0 tx=0).stop 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 msgr2=0x7f3784199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 0x7f3784199120 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f377800d950 tx=0x7f377800dc60 comp rx=0 tx=0).stop 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 shutdown_connections 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3774071f80 0x7f3774074440 unknown :-1 s=CLOSED pgs=3841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f37841094e0 0x7f378419d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.551 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f37841034a0 0x7f3784199120 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 --2- 172.21.15.50:0/2170655095 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3784103e60 0x7f3784199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 >> 172.21.15.50:0/2170655095 conn(0x7f378407a590 msgr2=0x7f3784101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:23.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 shutdown_connections 2024-04-26T16:37:23.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 -- 172.21.15.50:0/2170655095 wait complete. 2024-04-26T16:37:23.552 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7f378ab7f700 1 librados: shutdown 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 10 librados: watch_flush enter 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 10 librados: watch_flush exit 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 20 client.41690.objecter shutdown clearing up homeless session... 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 10 client.41690.objecter successfully canceled tick 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa01c071f80 msgr2=0x7fa01c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa01c071f80 0x7fa01c074440 secure :-1 s=READY pgs=3840 cs=0 l=1 rev1=1 crypto rx=0x7fa024009da0 tx=0x7fa02400b040 comp rx=0 tx=0).stop 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 msgr2=0x7fa034199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 0x7fa034199670 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa028004790 comp rx=0 tx=0).stop 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 shutdown_connections 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fa01c071f80 0x7fa01c074440 unknown :-1 s=CLOSED pgs=3840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0341034d0 0x7fa034199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.553 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fa0341094f0 0x7fa03419da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 --2- 172.21.15.50:0/3220106066 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fa034103e90 0x7fa034199670 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 >> 172.21.15.50:0/3220106066 conn(0x7fa03407a500 msgr2=0x7fa034101c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 shutdown_connections 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 -- 172.21.15.50:0/3220106066 wait complete. 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:23.552+0000 7fa03a020700 1 librados: shutdown 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is set 2024-04-26T16:37:23.554 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T16:37:23.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:23.604 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:23.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: cluster 2024-04-26T16:37:21.951950+0000 mgr.y (mgr.24401) 764 : cluster [DBG] pgmap v1316: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: cluster 2024-04-26T16:37:21.951950+0000 mgr.y (mgr.24401) 764 : cluster [DBG] pgmap v1316: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:22.709738+0000 mon.c (mon.2) 876 : audit [INF] from='client.? 172.21.15.50:0/3220106066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:22.709738+0000 mon.c (mon.2) 876 : audit [INF] from='client.? 172.21.15.50:0/3220106066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:22.710062+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:22.710062+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:23.003114+0000 mon.c (mon.2) 877 : audit [INF] from='client.? 172.21.15.50:0/2170655095' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.720 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:23.003114+0000 mon.c (mon.2) 877 : audit [INF] from='client.? 172.21.15.50:0/2170655095' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:23.003376+0000 mon.a (mon.0) 3870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:23 smithi190 bash[23868]: audit 2024-04-26T16:37:23.003376+0000 mon.a (mon.0) 3870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.734 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:23.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:23.849 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: cluster 2024-04-26T16:37:21.951950+0000 mgr.y (mgr.24401) 764 : cluster [DBG] pgmap v1316: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: cluster 2024-04-26T16:37:21.951950+0000 mgr.y (mgr.24401) 764 : cluster [DBG] pgmap v1316: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:22.709738+0000 mon.c (mon.2) 876 : audit [INF] from='client.? 172.21.15.50:0/3220106066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:22.709738+0000 mon.c (mon.2) 876 : audit [INF] from='client.? 172.21.15.50:0/3220106066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:22.710062+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:22.710062+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:23.003114+0000 mon.c (mon.2) 877 : audit [INF] from='client.? 172.21.15.50:0/2170655095' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:23.003114+0000 mon.c (mon.2) 877 : audit [INF] from='client.? 172.21.15.50:0/2170655095' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:23.003376+0000 mon.a (mon.0) 3870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[29234]: audit 2024-04-26T16:37:23.003376+0000 mon.a (mon.0) 3870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: cluster 2024-04-26T16:37:21.951950+0000 mgr.y (mgr.24401) 764 : cluster [DBG] pgmap v1316: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: cluster 2024-04-26T16:37:21.951950+0000 mgr.y (mgr.24401) 764 : cluster [DBG] pgmap v1316: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:22.709738+0000 mon.c (mon.2) 876 : audit [INF] from='client.? 172.21.15.50:0/3220106066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:22.709738+0000 mon.c (mon.2) 876 : audit [INF] from='client.? 172.21.15.50:0/3220106066' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:22.710062+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:22.710062+0000 mon.a (mon.0) 3869 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:23.003114+0000 mon.c (mon.2) 877 : audit [INF] from='client.? 172.21.15.50:0/2170655095' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:23.003114+0000 mon.c (mon.2) 877 : audit [INF] from='client.? 172.21.15.50:0/2170655095' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:23.003376+0000 mon.a (mon.0) 3870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:23 smithi050 bash[21463]: audit 2024-04-26T16:37:23.003376+0000 mon.a (mon.0) 3870 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:23.867 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:23.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:23.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:23.997 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:24.101 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:24.128 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:24.231 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:24.255 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:24.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:24.382 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:24.487 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:24.515 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:24.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:24.619 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:24.648 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:24.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:24.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: cluster 2024-04-26T16:37:23.528422+0000 mon.a (mon.0) 3871 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:24.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: cluster 2024-04-26T16:37:23.528422+0000 mon.a (mon.0) 3871 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:24.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: audit 2024-04-26T16:37:23.539573+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:37:24.761 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: audit 2024-04-26T16:37:23.539573+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:37:24.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: audit 2024-04-26T16:37:23.539721+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:24.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: audit 2024-04-26T16:37:23.539721+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:24.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: cluster 2024-04-26T16:37:23.547280+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-04-26T16:37:24.762 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:24 smithi190 bash[23868]: cluster 2024-04-26T16:37:23.547280+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-04-26T16:37:24.774 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:24.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:24.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:37:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: cluster 2024-04-26T16:37:23.528422+0000 mon.a (mon.0) 3871 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: cluster 2024-04-26T16:37:23.528422+0000 mon.a (mon.0) 3871 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: audit 2024-04-26T16:37:23.539573+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: audit 2024-04-26T16:37:23.539573+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: audit 2024-04-26T16:37:23.539721+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: audit 2024-04-26T16:37:23.539721+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: cluster 2024-04-26T16:37:23.547280+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-04-26T16:37:24.901 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[21463]: cluster 2024-04-26T16:37:23.547280+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: cluster 2024-04-26T16:37:23.528422+0000 mon.a (mon.0) 3871 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: cluster 2024-04-26T16:37:23.528422+0000 mon.a (mon.0) 3871 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: audit 2024-04-26T16:37:23.539573+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: audit 2024-04-26T16:37:23.539573+0000 mon.a (mon.0) 3872 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: audit 2024-04-26T16:37:23.539721+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: audit 2024-04-26T16:37:23.539721+0000 mon.a (mon.0) 3873 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: cluster 2024-04-26T16:37:23.547280+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-04-26T16:37:24.902 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:24 smithi050 bash[29234]: cluster 2024-04-26T16:37:23.547280+0000 mon.a (mon.0) 3874 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2024-04-26T16:37:24.904 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:24.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:25.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:25.032 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:25.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:25.158 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:25.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:25.281 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:25.384 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:25.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:25.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:25.536 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:25.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:25.664 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:25.705 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:23.952955+0000 mgr.y (mgr.24401) 765 : cluster [DBG] pgmap v1318: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:23.952955+0000 mgr.y (mgr.24401) 765 : cluster [DBG] pgmap v1318: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:24.380333+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:24.380333+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:24.380631+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:24.380631+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:24.553515+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: cluster 2024-04-26T16:37:24.553515+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:24.728131+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:24.728131+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:25.706 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:25.463467+0000 mon.c (mon.2) 879 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:25.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:25.463467+0000 mon.c (mon.2) 879 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:25.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:25.464393+0000 mon.c (mon.2) 880 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:25.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:25.464393+0000 mon.c (mon.2) 880 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:25.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:25.470143+0000 mon.a (mon.0) 3876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:25.707 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[29234]: audit 2024-04-26T16:37:25.470143+0000 mon.a (mon.0) 3876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:25.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:23.952955+0000 mgr.y (mgr.24401) 765 : cluster [DBG] pgmap v1318: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:25.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:23.952955+0000 mgr.y (mgr.24401) 765 : cluster [DBG] pgmap v1318: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:25.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:24.380333+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:25.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:24.380333+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:25.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:24.380631+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:25.708 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:24.380631+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:24.553515+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: cluster 2024-04-26T16:37:24.553515+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:24.728131+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:24.728131+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:25.463467+0000 mon.c (mon.2) 879 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:25.463467+0000 mon.c (mon.2) 879 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:25.464393+0000 mon.c (mon.2) 880 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:25.464393+0000 mon.c (mon.2) 880 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:25.470143+0000 mon.a (mon.0) 3876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:25.709 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:25 smithi050 bash[21463]: audit 2024-04-26T16:37:25.470143+0000 mon.a (mon.0) 3876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:25.747 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-26T16:37:25.768 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:25.819 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:23.952955+0000 mgr.y (mgr.24401) 765 : cluster [DBG] pgmap v1318: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:23.952955+0000 mgr.y (mgr.24401) 765 : cluster [DBG] pgmap v1318: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:24.380333+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:24.380333+0000 osd.3 (osd.3) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:24.380631+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:24.380631+0000 osd.3 (osd.3) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:24.553515+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: cluster 2024-04-26T16:37:24.553515+0000 mon.a (mon.0) 3875 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:24.728131+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:24.728131+0000 mon.c (mon.2) 878 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:37:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:25.463467+0000 mon.c (mon.2) 879 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:25.463467+0000 mon.c (mon.2) 879 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:37:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:25.464393+0000 mon.c (mon.2) 880 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:25.464393+0000 mon.c (mon.2) 880 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:37:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:25.470143+0000 mon.a (mon.0) 3876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:25 smithi190 bash[23868]: audit 2024-04-26T16:37:25.470143+0000 mon.a (mon.0) 3876 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:25.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:25.951 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:25.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:26.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:26.076 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:26.081 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:26.185 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:26.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:26.314 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:26.343 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:26.445 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:26.475 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:26.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:26.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:26.606 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:26.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:26.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:26.736 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:26.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:26.865 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:26.895 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:37:26.968 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:26.995 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:26.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:27.098 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:27.126 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:27.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:27.258 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:27.361 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:27.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:27.488 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:27.511 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:27.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:27.625 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:37:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:37:27.625 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:27 smithi190 bash[23868]: cluster 2024-04-26T16:37:25.953478+0000 mgr.y (mgr.24401) 766 : cluster [DBG] pgmap v1320: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:27.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:27 smithi190 bash[23868]: cluster 2024-04-26T16:37:25.953478+0000 mgr.y (mgr.24401) 766 : cluster [DBG] pgmap v1320: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:27 smithi050 bash[29234]: cluster 2024-04-26T16:37:25.953478+0000 mgr.y (mgr.24401) 766 : cluster [DBG] pgmap v1320: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:27.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:27 smithi050 bash[29234]: cluster 2024-04-26T16:37:25.953478+0000 mgr.y (mgr.24401) 766 : cluster [DBG] pgmap v1320: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:27 smithi050 bash[21463]: cluster 2024-04-26T16:37:25.953478+0000 mgr.y (mgr.24401) 766 : cluster [DBG] pgmap v1320: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:27.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:27 smithi050 bash[21463]: cluster 2024-04-26T16:37:25.953478+0000 mgr.y (mgr.24401) 766 : cluster [DBG] pgmap v1320: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 934 B/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:27.640 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:27.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:27.709 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:37:27.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:27.777 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:27.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:27.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:27.902 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:27.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:27.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:28.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:28.027 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:28.129 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:28.157 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:28.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:28.286 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:28.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:28.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:28.417 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:28.520 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.552 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:28.655 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.684 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:28.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.799 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.115502+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.115502+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.115922+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.115922+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: audit 2024-04-26T16:37:27.335473+0000 mgr.y (mgr.24401) 767 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: audit 2024-04-26T16:37:27.335473+0000 mgr.y (mgr.24401) 767 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.560937+0000 mon.a (mon.0) 3877 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.560937+0000 mon.a (mon.0) 3877 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: audit 2024-04-26T16:37:27.982337+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: audit 2024-04-26T16:37:27.982337+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: audit 2024-04-26T16:37:27.984820+0000 mon.c (mon.2) 881 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[21463]: audit 2024-04-26T16:37:27.984820+0000 mon.c (mon.2) 881 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:28.800 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.115502+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.115502+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.115922+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.115922+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: audit 2024-04-26T16:37:27.335473+0000 mgr.y (mgr.24401) 767 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: audit 2024-04-26T16:37:27.335473+0000 mgr.y (mgr.24401) 767 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.560937+0000 mon.a (mon.0) 3877 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.560937+0000 mon.a (mon.0) 3877 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: audit 2024-04-26T16:37:27.982337+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: audit 2024-04-26T16:37:27.982337+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: audit 2024-04-26T16:37:27.984820+0000 mon.c (mon.2) 881 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:28.801 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:28 smithi050 bash[29234]: audit 2024-04-26T16:37:27.984820+0000 mon.c (mon.2) 881 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:28.815 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.115502+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.115502+0000 osd.2 (osd.2) 407 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.115922+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.115922+0000 osd.2 (osd.2) 408 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: audit 2024-04-26T16:37:27.335473+0000 mgr.y (mgr.24401) 767 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:28.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: audit 2024-04-26T16:37:27.335473+0000 mgr.y (mgr.24401) 767 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.560937+0000 mon.a (mon.0) 3877 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-04-26T16:37:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.560937+0000 mon.a (mon.0) 3877 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2024-04-26T16:37:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: audit 2024-04-26T16:37:27.982337+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: audit 2024-04-26T16:37:27.982337+0000 mon.a (mon.0) 3878 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: audit 2024-04-26T16:37:27.984820+0000 mon.c (mon.2) 881 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:28.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:28 smithi190 bash[23868]: audit 2024-04-26T16:37:27.984820+0000 mon.c (mon.2) 881 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:28.918 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.948 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:28.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:29.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:29.079 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:29.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:29.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:29.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:29.341 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:29.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:29.472 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:29.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:29.605 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:29.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:29.735 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:29.834 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:29 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.954067+0000 mgr.y (mgr.24401) 768 : cluster [DBG] pgmap v1322: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:29.835 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:29 smithi190 bash[23868]: cluster 2024-04-26T16:37:27.954067+0000 mgr.y (mgr.24401) 768 : cluster [DBG] pgmap v1322: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:29.838 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:29.849 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:29 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.954067+0000 mgr.y (mgr.24401) 768 : cluster [DBG] pgmap v1322: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:29.850 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:29 smithi050 bash[21463]: cluster 2024-04-26T16:37:27.954067+0000 mgr.y (mgr.24401) 768 : cluster [DBG] pgmap v1322: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:29.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:29 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.954067+0000 mgr.y (mgr.24401) 768 : cluster [DBG] pgmap v1322: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:29.850 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:29 smithi050 bash[29234]: cluster 2024-04-26T16:37:27.954067+0000 mgr.y (mgr.24401) 768 : cluster [DBG] pgmap v1322: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:29.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:29.970 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:29.999 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:2642 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94191258305536 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e399)", 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:27:47.717421+0000", 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 582.29399602800004, 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.45756073200000003, 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.012 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.013 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.013 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:37:30.013 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:37:30.013 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2642 2024-04-26T16:37:30.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.013 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717421+0000", 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717421+0000", 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717423+0000", 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2589999999999999e-06 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717433+0000", 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0015e-05 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717456+0000", 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2164e-05 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717471+0000", 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5417000000000002e-05 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717677+0000", 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000205966 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:47.717689+0000", 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2189e-05 2024-04-26T16:37:30.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:48.166168+0000", 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44847933899999998 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:48.166434+0000", 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026571 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:27:48.174982+0000", 2024-04-26T16:37:30.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085476730000000004 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3296 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94191258305536 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:33:01.880373+0000", 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 268.13104462799998, 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44292036299999998, 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.018 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3296 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880373+0000", 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880373+0000", 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880376+0000", 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0690000000000002e-06 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880392+0000", 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6422999999999998e-05 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880422+0000", 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0324999999999998e-05 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880439+0000", 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6327000000000001e-05 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880518+0000", 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9277999999999995e-05 2024-04-26T16:37:30.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880537+0000", 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8719000000000002e-05 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:02.315822+0000", 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43528493000000001 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:02.315990+0000", 2024-04-26T16:37:30.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000168114 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:02.323293+0000", 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0073031779999999996 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.2.18:60 323.5s0 323:a548bc64:test-rados-api-smithi050-73372-125::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e730)", 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:08.381958+0000", 2024-04-26T16:37:30.023 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 141.62945959499999, 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058861649999999996, 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.381958+0000", 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.381958+0000", 2024-04-26T16:37:30.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.381960+0000", 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0059999999999999e-06 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.381974+0000", 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4113e-05 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.382000+0000", 2024-04-26T16:37:30.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6013e-05 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.382018+0000", 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8379000000000001e-05 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.383599+0000", 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015811519999999999 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.383760+0000", 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000160865 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.387844+0000", 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040836370000000002 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.027 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:1414 330.8 330:10000000:%01:::head [pgnls start_epoch 731] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e731)", 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:09.381904+0000", 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 140.62951339200001, 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065766330000000001, 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1414 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.381904+0000", 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.381904+0000", 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.381907+0000", 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7400000000000002e-06 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.381926+0000", 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8929000000000001e-05 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.381974+0000", 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7352000000000003e-05 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.382003+0000", 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8906000000000001e-05 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.382194+0000", 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019168200000000001 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.382212+0000", 2024-04-26T16:37:30.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7549000000000001e-05 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.388100+0000", 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058884829999999999 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.388202+0000", 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010170000000000001 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.388480+0000", 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027829200000000003 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:1448 330.8 330:10000000::::head [pgnls start_epoch 733] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e733)", 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:11.363483+0000", 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 138.647933924, 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046047559999999998, 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.033 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1448 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363483+0000", 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363483+0000", 2024-04-26T16:37:30.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363487+0000", 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6229999999999999e-06 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363505+0000", 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8219e-05 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363546+0000", 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1118999999999997e-05 2024-04-26T16:37:30.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363575+0000", 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9252e-05 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363734+0000", 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015892 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.363745+0000", 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0995999999999999e-05 2024-04-26T16:37:30.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.367720+0000", 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039742370000000003 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.367828+0000", 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010823399999999999 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.368088+0000", 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026015599999999999 2024-04-26T16:37:30.037 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:1698 332.10 332:0b536d6c:test-rados-api-smithi050-73372-129::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:25.396251+0000", 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 124.615165675, 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.021568873999999998, 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.038 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1698 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396251+0000", 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396251+0000", 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.039 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396253+0000", 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5969999999999999e-06 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396260+0000", 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3429999999999998e-06 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396272+0000", 2024-04-26T16:37:30.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1506e-05 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396284+0000", 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2573e-05 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396339+0000", 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4749e-05 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396350+0000", 2024-04-26T16:37:30.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1086e-05 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.397847+0000", 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014961900000000001 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.398168+0000", 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032133599999999997 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.413475+0000", 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015307383000000001 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.413589+0000", 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113742 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,6", 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.413738+0000", 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014917199999999999 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.043 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414313+0000", 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057516000000000002 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414471+0000", 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015739399999999999 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.417754+0000", 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032830419999999999 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.044 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.417792+0000", 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8268e-05 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.417820+0000", 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8333000000000001e-05 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2179 334.10 334:08000000:%01:::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-04-26T16:37:30.045 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:50.450420+0000", 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 99.560996966000005, 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048020390000000001, 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2179 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450420+0000", 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450420+0000", 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450424+0000", 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6959999999999998e-06 2024-04-26T16:37:30.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450442+0000", 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8323000000000001e-05 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450494+0000", 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2246000000000001e-05 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450526+0000", 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1931999999999999e-05 2024-04-26T16:37:30.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450675+0000", 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014909299999999999 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.450692+0000", 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6441999999999999e-05 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.454886+0000", 2024-04-26T16:37:30.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041937229999999999 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.454990+0000", 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104477 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455222+0000", 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023210700000000001 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2391 336.3 336:c4f8e971:test-rados-api-smithi050-73372-137::foo:head [writefull 0~4000008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e786)", 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.406201+0000", 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.605216638000002, 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026419728, 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2391 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.406201+0000", 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.406201+0000", 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.406203+0000", 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8660000000000002e-06 2024-04-26T16:37:30.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.412125+0000", 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059215800000000001 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.412195+0000", 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9777999999999995e-05 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.412232+0000", 2024-04-26T16:37:30.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7598999999999997e-05 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.412458+0000", 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000225718 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.412973+0000", 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000515107 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.414782+0000", 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018084360000000001 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415096+0000", 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031463200000000002 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,3", 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415515+0000", 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041912200000000002 2024-04-26T16:37:30.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.424381+0000", 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0088658799999999996 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432114+0000", 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077330929999999999 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432248+0000", 2024-04-26T16:37:30.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013333299999999999 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432438+0000", 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000190065 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432620+0000", 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018251899999999999 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(osd.4.0:1496 336.3 e786/782 ondisk, result = 0)", 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.415132+0000", 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.596284679999997, 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084656100000000001, 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415132+0000", 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415132+0000", 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415134+0000", 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7740000000000001e-06 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415145+0000", 2024-04-26T16:37:30.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0399e-05 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415165+0000", 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0136000000000002e-05 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415180+0000", 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4983e-05 2024-04-26T16:37:30.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.423461+0000", 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.008281666 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.423517+0000", 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5201000000000002e-05 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.423598+0000", 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1451000000000003e-05 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.061 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(osd.4.0:1496 336.3 e786/782 ondisk, result = 0)", 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.415235+0000", 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.596182416000005, 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085881840000000004, 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415235+0000", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415235+0000", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415236+0000", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6869999999999999e-06 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415248+0000", 2024-04-26T16:37:30.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.117e-05 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415279+0000", 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1708999999999997e-05 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.415300+0000", 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0908999999999999e-05 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.423663+0000", 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083628230000000001 2024-04-26T16:37:30.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.423676+0000", 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3263e-05 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.423823+0000", 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000146623 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2392 336.3 336:c4f8e971:test-rados-api-smithi050-73372-137::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e786)", 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.432845+0000", 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.578571862000004, 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.033188694999999997, 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2392 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432845+0000", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432845+0000", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432848+0000", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5390000000000002e-06 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432860+0000", 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1919999999999999e-05 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432892+0000", 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2005999999999998e-05 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.432909+0000", 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7295999999999998e-05 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433009+0000", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9864000000000005e-05 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433200+0000", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019071499999999999 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.465672+0000", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.032472543999999999 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.465888+0000", 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021624099999999999 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.466034+0000", 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014557 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.3.28:107 336.3 336:c4f8e971:test-rados-api-smithi050-73372-137::foo:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+flush+map_snap_clone+known_if_redirected+supports_pool_eio e786)", 2024-04-26T16:37:30.070 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.433942+0000", 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.577475023999995, 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069036560000000002, 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433942+0000", 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433942+0000", 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433945+0000", 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6299999999999998e-06 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433957+0000", 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2489e-05 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.433989+0000", 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1646000000000002e-05 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.434009+0000", 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0052999999999999e-05 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.434155+0000", 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000145933 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.434495+0000", 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034022700000000002 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.440846+0000", 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063506780000000002 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(323.3s2 786/746 ECSubWrite(tid=1552, reqid=osd.4.35:148, at_version=786'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T16:37:30.074 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.451160+0000", 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 83.560256988000006, 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074040010000000003, 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.451160+0000", 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.451160+0000", 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.451163+0000", 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3530000000000002e-06 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.453708+0000", 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025456139999999999 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.453746+0000", 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8142999999999997e-05 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.453768+0000", 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2005e-05 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.453969+0000", 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020037200000000001 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.453994+0000", 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5766e-05 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458441+0000", 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044461179999999998 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458564+0000", 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012363000000000001 2024-04-26T16:37:30.078 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2607 338.6 338:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e805)", 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:31.532361+0000", 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 58.479056067999998, 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038158350000000001, 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.079 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2607 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532361+0000", 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532361+0000", 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532365+0000", 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.653e-06 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532405+0000", 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0454999999999998e-05 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532446+0000", 2024-04-26T16:37:30.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0586000000000001e-05 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532475+0000", 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9696000000000002e-05 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.532661+0000", 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018597899999999999 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.082 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.534095+0000", 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001433204 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.534426+0000", 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033104500000000003 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.535183+0000", 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075691500000000004 2024-04-26T16:37:30.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.535449+0000", 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000265927 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.535983+0000", 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00053404600000000004 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.536089+0000", 2024-04-26T16:37:30.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105988 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.536177+0000", 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8341e-05 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2619 338.6 338:602f83fe:::foo:head [append 0~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e807)", 2024-04-26T16:37:30.085 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:37.561676+0000", 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 52.449741316999997, 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040589850000000002, 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2619 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561676+0000", 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561676+0000", 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561679+0000", 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5899999999999999e-06 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561714+0000", 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4848999999999998e-05 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561762+0000", 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7324999999999998e-05 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561785+0000", 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3283000000000001e-05 2024-04-26T16:37:30.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.561997+0000", 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021180799999999999 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.564146+0000", 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021490060000000002 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.564476+0000", 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033049199999999999 2024-04-26T16:37:30.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.565516+0000", 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010394919999999999 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.565613+0000", 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7579999999999997e-05 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.565662+0000", 2024-04-26T16:37:30.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8427000000000002e-05 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.565704+0000", 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2741999999999998e-05 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.565735+0000", 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0391000000000001e-05 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2722 323.5s0 323:b441eaea:test-rados-api-smithi050-73372-143::foo0:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e812)", 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:41.650521+0000", 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 48.360896093000001, 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.026068877000000001, 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2722 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650521+0000", 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650521+0000", 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650524+0000", 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2200000000000001e-06 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650540+0000", 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5843999999999999e-05 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650572+0000", 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2325e-05 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650599+0000", 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6610000000000001e-05 2024-04-26T16:37:30.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650758+0000", 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015873300000000001 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.650778+0000", 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0075999999999999e-05 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.665268+0000", 2024-04-26T16:37:30.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014490349 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.665626+0000", 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035743400000000001 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.666796+0000", 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001169918 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:37:30.096 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.668041+0000", 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001245849 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.676490+0000", 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084490450000000005 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:41.676590+0000", 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9474000000000003e-05 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.3.28:108 323.5s0 323:b441eaea:test-rados-api-smithi050-73372-143::foo0:head [read 0~1] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e821)", 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:50.415540+0000", 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 39.595877551000001, 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0091548329999999994, 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415540+0000", 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415540+0000", 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415543+0000", 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5130000000000001e-06 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415562+0000", 2024-04-26T16:37:30.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8516e-05 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415606+0000", 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4832e-05 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415637+0000", 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0306000000000001e-05 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415854+0000", 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021718799999999999 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.415873+0000", 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9533000000000001e-05 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.422976+0000", 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071021110000000004 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.423201+0000", 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022534399999999999 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.424694+0000", 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014934900000000001 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2817 339.12 339:48000000:%01:::head [pgnls start_epoch 823] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e823)", 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:52.429800+0000", 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 37.581617518000002, 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059277990000000001, 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2817 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.103 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.429800+0000", 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.429800+0000", 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.429803+0000", 2024-04-26T16:37:30.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3469999999999999e-06 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.429820+0000", 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6668999999999999e-05 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.429861+0000", 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1425999999999999e-05 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.429881+0000", 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0086e-05 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.430015+0000", 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013421700000000001 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.430026+0000", 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1043000000000001e-05 2024-04-26T16:37:30.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.433991+0000", 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039646899999999999 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.435484+0000", 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014929710000000001 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.435727+0000", 2024-04-26T16:37:30.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024335000000000001 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(3.b,QREQUEST,e:824,from:3,reservation_nonce:19)", 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:53.695652+0000", 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 36.315764917000003, 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056196600000000003, 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695652+0000", 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695652+0000", 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695655+0000", 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.683e-06 2024-04-26T16:37:30.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695671+0000", 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6399999999999999e-05 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695710+0000", 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9038999999999997e-05 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695744+0000", 2024-04-26T16:37:30.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4131999999999998e-05 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695927+0000", 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018205900000000001 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.695943+0000", 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6912999999999999e-05 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.701117+0000", 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051732080000000003 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.701165+0000", 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8018e-05 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:53.701272+0000", 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010720799999999999 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:3335 341.14 341:2fd3dafd:test-rados-api-smithi050-73372-147::baz:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e839)", 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:37:15.447591+0000", 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 14.563825806000001, 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074866309999999997, 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:30.114 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3335 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447591+0000", 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447591+0000", 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447593+0000", 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0040000000000002e-06 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447602+0000", 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9069999999999996e-06 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447621+0000", 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8644e-05 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447630+0000", 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9400000000000008e-06 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.447711+0000", 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0740000000000003e-05 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.448006+0000", 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029532099999999997 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.450522+0000", 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025161160000000001 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.450589+0000", 2024-04-26T16:37:30.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6749999999999996e-05 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.451873+0000", 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012837759999999999 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.451985+0000", 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112762 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:30.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455078+0000", 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003092671 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:30.120 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:37:30.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:30.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 -- 172.21.15.50:0/1129955949 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06bb60 msgr2=0x7fc3dc06bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.137 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 --2- 172.21.15.50:0/1129955949 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06bb60 0x7fc3dc06bfe0 secure :-1 s=READY pgs=2759 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc009990 tx=0x7fc3cc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 -- 172.21.15.50:0/1129955949 shutdown_connections 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 --2- 172.21.15.50:0/1129955949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3dc1095c0 0x7fc3dc1099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 --2- 172.21.15.50:0/1129955949 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06c520 0x7fc3dc110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 --2- 172.21.15.50:0/1129955949 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06bb60 0x7fc3dc06bfe0 unknown :-1 s=CLOSED pgs=2759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 -- 172.21.15.50:0/1129955949 >> 172.21.15.50:0/1129955949 conn(0x7fc3dc079120 msgr2=0x7fc3dc079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 -- 172.21.15.50:0/1129955949 shutdown_connections 2024-04-26T16:37:30.138 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.136+0000 7fc3e4587700 1 -- 172.21.15.50:0/1129955949 wait complete. 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 librados: starting msgr at 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 librados: starting objecter 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 Processor -- start 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- start start 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 librados: setting wanted keys 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 librados: calling monclient init 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06bb60 0x7fc3dc1990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06c520 0x7fc3dc199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3dc1095c0 0x7fc3dc19d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc112b50 con 0x7fc3dc06bb60 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc1129d0 con 0x7fc3dc1095c0 2024-04-26T16:37:30.140 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fc3dc112cd0 con 0x7fc3dc06c520 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e2323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06bb60 0x7fc3dc1990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06c520 0x7fc3dc199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e2323700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06bb60 0x7fc3dc1990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49668/0 (socket says 172.21.15.50:49668) 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e2323700 1 -- 172.21.15.50:0/2956457069 learned_addr learned my addr 172.21.15.50:0/2956457069 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3dc1095c0 0x7fc3dc19d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3dc1095c0 msgr2=0x7fc3dc19d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3dc1095c0 0x7fc3dc19d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06bb60 msgr2=0x7fc3dc1990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06bb60 0x7fc3dc1990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc19e0f0 con 0x7fc3dc06c520 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fc3dc06bb60 0x7fc3dc1990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06c520 0x7fc3dc199620 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0059d0 tx=0x7fc3cc0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 10 client.?.objecter ms_handle_connect 0x7fc3dc06c520 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:30.141 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc027070 con 0x7fc3dc06c520 2024-04-26T16:37:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fc3cc022470 con 0x7fc3dc06c520 2024-04-26T16:37:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc19e380 con 0x7fc3dc06c520 2024-04-26T16:37:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc017650 con 0x7fc3dc06c520 2024-04-26T16:37:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 10 client.41702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc1a5c60 con 0x7fc3dc06c520 2024-04-26T16:37:30.142 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 librados: init done 2024-04-26T16:37:30.143 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7fc3dc06a110 con 0x7fc3dc06c520 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fc3cc02a070 con 0x7fc3dc06c520 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.140+0000 7fc3d37fe700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc3c8071f80 0x7fc3c8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(849..849 src has 577..849) ==== 9973+0+0 (secure 0 0 0) 0x7fc3cc092bd0 con 0x7fc3dc06c520 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 osd_map(849..849 src has 577..849) 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 3 client.41702.objecter handle_osd_map got epochs [849,849] > 0 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 3 client.41702.objecter handle_osd_map decoding full epoch 849 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc3c8071f80 0x7fc3c8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 20 client.41702.objecter dump_active .. 0 homeless 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v849) ==== 54+0+16 (secure 0 0 0) 0x7fc3cc013070 con 0x7fc3dc06c520 2024-04-26T16:37:30.144 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 10 client.41702.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 1 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc3dc064030 0x7fc3dc064430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 1 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 0 1 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 4 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 10 client.41702.objecter _send_command 1 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc1a8fc0 con 0x7fc3dc064030 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc3c8071f80 0x7fc3c8074440 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8005950 tx=0x7fc3d800b500 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3c8071f80 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc3dc064030 0x7fc3dc064430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.145 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc3dc064030 0x7fc3dc064430 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc064030 2024-04-26T16:37:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc1a8fc0 con 0x7fc3dc064030 2024-04-26T16:37:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 1: 0 ) 2024-04-26T16:37:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 1 = system:0 2024-04-26T16:37:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 4 2024-04-26T16:37:30.146 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.144+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 0 1 2024-04-26T16:37:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:30.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 10 client.41702.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 2 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 2 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc19a440 osd=0 3 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 0 2 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 4 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 10 client.41702.objecter _send_command 2 2024-04-26T16:37:30.162 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.160+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc063ab0 con 0x7fc3dc064030 2024-04-26T16:37:30.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc063ab0 con 0x7fc3dc064030 2024-04-26T16:37:30.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.163 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 4 2024-04-26T16:37:30.164 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 0 2 2024-04-26T16:37:30.164 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 10 client.41702.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc3dc02d310 0x7fc3dc02d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 1 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 4 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 10 client.41702.objecter _send_command 3 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc10b150 con 0x7fc3dc02d310 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc3dc02d310 0x7fc3dc02d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.165 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc02d310 2024-04-26T16:37:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc3dc02d310 0x7fc3dc02d730 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc10b150 con 0x7fc3dc02d310 2024-04-26T16:37:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 3: 0 ) 2024-04-26T16:37:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 3 = system:0 2024-04-26T16:37:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 4 2024-04-26T16:37:30.166 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.164+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 1 3 2024-04-26T16:37:30.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 10 client.41702.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 4 2024-04-26T16:37:30.182 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 4 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc10aad0 osd=1 3 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 1 4 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 4 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 10 client.41702.objecter _send_command 4 2024-04-26T16:37:30.183 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc10a550 con 0x7fc3dc02d310 2024-04-26T16:37:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc10a550 con 0x7fc3dc02d310 2024-04-26T16:37:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 4 2024-04-26T16:37:30.184 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 1 4 2024-04-26T16:37:30.185 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:37:30.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 10 client.41702.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 5 2024-04-26T16:37:30.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc3dc02e5b0 0x7fc3dc102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 5 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 2 5 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 4 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 10 client.41702.objecter _send_command 5 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc1b2720 con 0x7fc3dc02e5b0 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc3dc02e5b0 0x7fc3dc102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc3dc02e5b0 0x7fc3dc102b30 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.184+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc02e5b0 2024-04-26T16:37:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.188+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc1b2720 con 0x7fc3dc02e5b0 2024-04-26T16:37:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.188+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 5: 0 ) 2024-04-26T16:37:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.188+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 5 = system:0 2024-04-26T16:37:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.188+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 4 2024-04-26T16:37:30.187 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.188+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 2 5 2024-04-26T16:37:30.203 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 10 client.41702.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 6 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 6 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc02e440 osd=2 3 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 2 6 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 4 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 10 client.41702.objecter _send_command 6 2024-04-26T16:37:30.204 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc02dec0 con 0x7fc3dc02e5b0 2024-04-26T16:37:30.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc02dec0 con 0x7fc3dc02e5b0 2024-04-26T16:37:30.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.205 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 4 2024-04-26T16:37:30.206 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 2 6 2024-04-26T16:37:30.206 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 10 client.41702.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 7 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc3dc103070 0x7fc3dc1034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 7 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 3 7 2024-04-26T16:37:30.207 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 4 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 10 client.41702.objecter _send_command 7 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.204+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc1a8c90 con 0x7fc3dc103070 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc3dc103070 0x7fc3dc1034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc3dc103070 0x7fc3dc1034f0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc103070 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc1a8c90 con 0x7fc3dc103070 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 7: 0 ) 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 7 = system:0 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 4 2024-04-26T16:37:30.208 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.208+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 3 7 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 10 client.41702.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 8 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 8 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1a8610 osd=3 3 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 3 8 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 4 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 10 client.41702.objecter _send_command 8 2024-04-26T16:37:30.225 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.224+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc1a8c90 con 0x7fc3dc103070 2024-04-26T16:37:30.226 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:37:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc1a8c90 con 0x7fc3dc103070 2024-04-26T16:37:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 4 2024-04-26T16:37:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 3 8 2024-04-26T16:37:30.227 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 10 client.41702.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 9 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc3dc104200 0x7fc3dc1b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 9 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 4 9 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 4 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 10 client.41702.objecter _send_command 9 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc104660 con 0x7fc3dc104200 2024-04-26T16:37:30.228 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc3dc104200 0x7fc3dc1b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc104200 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc3dc104200 0x7fc3dc1b68f0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc104660 con 0x7fc3dc104200 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 9: 0 ) 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 9 = system:0 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 4 2024-04-26T16:37:30.229 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.228+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 4 9 2024-04-26T16:37:30.246 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 10 client.41702.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 10 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 10 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1a8c90 osd=4 3 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 4 10 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 4 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 10 client.41702.objecter _send_command 10 2024-04-26T16:37:30.247 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.248+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc1b6e30 con 0x7fc3dc104200 2024-04-26T16:37:30.247 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc1b6e30 con 0x7fc3dc104200 2024-04-26T16:37:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 4 2024-04-26T16:37:30.251 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 4 10 2024-04-26T16:37:30.251 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 10 client.41702.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 11 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc3dc1b7550 0x7fc3dc1b9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 11 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 5 11 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 4 2024-04-26T16:37:30.252 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 10 client.41702.objecter _send_command 11 2024-04-26T16:37:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc1bc8f0 con 0x7fc3dc1b7550 2024-04-26T16:37:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc3dc1b7550 0x7fc3dc1b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc1b7550 2024-04-26T16:37:30.253 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.252+0000 7fc3e1b22700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc3dc1b7550 0x7fc3dc1b9940 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.256+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc1bc8f0 con 0x7fc3dc1b7550 2024-04-26T16:37:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.256+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 11: 0 ) 2024-04-26T16:37:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.256+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 11 = system:0 2024-04-26T16:37:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.256+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 4 2024-04-26T16:37:30.254 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.256+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 5 11 2024-04-26T16:37:30.268 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 10 client.41702.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 12 2024-04-26T16:37:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.273 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 12 2024-04-26T16:37:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1b6e30 osd=5 3 2024-04-26T16:37:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 5 12 2024-04-26T16:37:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 4 2024-04-26T16:37:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 10 client.41702.objecter _send_command 12 2024-04-26T16:37:30.274 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.272+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc1bc8f0 con 0x7fc3dc1b7550 2024-04-26T16:37:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.276+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc1bc8f0 con 0x7fc3dc1b7550 2024-04-26T16:37:30.277 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.276+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.276+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.276+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 4 2024-04-26T16:37:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.276+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 5 12 2024-04-26T16:37:30.278 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:37:30.278 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 10 client.41702.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 13 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fc3dc1bbc80 0x7fc3dc1bc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 13 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 6 13 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 4 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 10 client.41702.objecter _send_command 13 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc1bfc60 con 0x7fc3dc1bbc80 2024-04-26T16:37:30.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fc3dc1bbc80 0x7fc3dc1bc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3e2b24700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fc3dc1bbc80 0x7fc3dc1bc0e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc1bbc80 2024-04-26T16:37:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc1bfc60 con 0x7fc3dc1bbc80 2024-04-26T16:37:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 13: 0 ) 2024-04-26T16:37:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 13 = system:0 2024-04-26T16:37:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 4 2024-04-26T16:37:30.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.280+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 6 13 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 10 client.41702.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 14 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 14 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1bc8f0 osd=6 3 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 6 14 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 4 2024-04-26T16:37:30.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 10 client.41702.objecter _send_command 14 2024-04-26T16:37:30.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.300+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc1b9ef0 con 0x7fc3dc1bbc80 2024-04-26T16:37:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc1b9ef0 con 0x7fc3dc1bbc80 2024-04-26T16:37:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 4 2024-04-26T16:37:30.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 6 14 2024-04-26T16:37:30.305 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:37:30.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 10 client.41702.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:30.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 15 2024-04-26T16:37:30.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc3dc1bb2d0 0x7fc3dc1ba670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:30.305 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 15 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 7 15 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 4 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 10 client.41702.objecter _send_command 15 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7fc3dc1bada0 con 0x7fc3dc1bb2d0 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc3dc1bb2d0 0x7fc3dc1ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3d37fe700 10 client.41702.objecter ms_handle_connect 0x7fc3dc1bb2d0 2024-04-26T16:37:30.306 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.304+0000 7fc3e2323700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc3dc1bb2d0 0x7fc3dc1ba670 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.308+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3dc1bada0 con 0x7fc3dc1bb2d0 2024-04-26T16:37:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.308+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 15: 0 ) 2024-04-26T16:37:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.308+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 15 = system:0 2024-04-26T16:37:30.307 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.308+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 4 2024-04-26T16:37:30.308 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.308+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 7 15 2024-04-26T16:37:30.325 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 10 client.41702.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}] 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign -1 16 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 20 client.41702.objecter _get_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_remove -1 16 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 20 client.41702.objecter get_session s=0x7fc3dc1b9ef0 osd=7 3 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 15 client.41702.objecter _session_command_op_assign 7 16 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 4 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 10 client.41702.objecter _send_command 16 2024-04-26T16:37:30.326 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.324+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=false"]}) -- 0x7fc3dc1bada0 con 0x7fc3dc1bb2d0 2024-04-26T16:37:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3d37fe700 1 -- 172.21.15.50:0/2956457069 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7fc3dc1bada0 con 0x7fc3dc1bb2d0 2024-04-26T16:37:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3d37fe700 10 client.41702.objecter ms_dispatch 0x7fc3dc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3d37fe700 10 client.41702.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3d37fe700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 4 2024-04-26T16:37:30.330 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3d37fe700 15 client.41702.objecter _session_command_op_remove 7 16 2024-04-26T16:37:30.330 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:37:30.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 librados: watch_flush enter 2024-04-26T16:37:30.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 librados: watch_flush exit 2024-04-26T16:37:30.331 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.0 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc3dc064030 msgr2=0x7fc3dc064430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7fc3dc064030 0x7fc3dc064430 crc :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc19a440 osd=0 1 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.1 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc3dc02d310 msgr2=0x7fc3dc02d730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7fc3dc02d310 0x7fc3dc02d730 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc10aad0 osd=1 1 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.2 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc3dc02e5b0 msgr2=0x7fc3dc102b30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fc3dc02e5b0 0x7fc3dc102b30 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e2b24700 1 -- 172.21.15.50:0/2956457069 reap_dead start 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc02e440 osd=2 1 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.3 2024-04-26T16:37:30.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc3dc103070 msgr2=0x7fc3dc1034f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7fc3dc103070 0x7fc3dc1034f0 crc :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8610 osd=3 1 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.4 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc3dc104200 msgr2=0x7fc3dc1b68f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7fc3dc104200 0x7fc3dc1b68f0 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1a8c90 osd=4 1 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.5 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc3dc1b7550 msgr2=0x7fc3dc1b9940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7fc3dc1b7550 0x7fc3dc1b9940 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b6e30 osd=5 1 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.6 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fc3dc1bbc80 msgr2=0x7fc3dc1bc0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fc3dc1bbc80 0x7fc3dc1bc0e0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1bc8f0 osd=6 1 2024-04-26T16:37:30.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter close_session for osd.7 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc3dc1bb2d0 msgr2=0x7fc3dc1ba670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7fc3dc1bb2d0 0x7fc3dc1ba670 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter put_session s=0x7fc3dc1b9ef0 osd=7 1 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 20 client.41702.objecter shutdown clearing up homeless session... 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 10 client.41702.objecter successfully canceled tick 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc3c8071f80 msgr2=0x7fc3c8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fc3c8071f80 0x7fc3c8074440 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8005950 tx=0x7fc3d800b500 comp rx=0 tx=0).stop 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e2b24700 1 -- 172.21.15.50:0/2956457069 reap_dead start 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06c520 msgr2=0x7fc3dc199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06c520 0x7fc3dc199620 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0059d0 tx=0x7fc3cc0047b0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e2b24700 1 -- 172.21.15.50:0/2956457069 reap_dead start 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 shutdown_connections 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 --2- 172.21.15.50:0/2956457069 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fc3dc06c520 0x7fc3dc199620 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:30.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 >> 172.21.15.50:0/2956457069 conn(0x7fc3dc079120 msgr2=0x7fc3dc100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 shutdown_connections 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 -- 172.21.15.50:0/2956457069 wait complete. 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:30.332+0000 7fc3e4587700 1 librados: shutdown 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:37:30.335 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:37:30.371 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:30.390 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:30.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:30.521 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:30.624 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.500591+0000 osd.0 (osd.0) 469 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.500591+0000 osd.0 (osd.0) 469 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.500767+0000 osd.0 (osd.0) 470 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.500767+0000 osd.0 (osd.0) 470 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.585264+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.585264+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-04-26T16:37:30.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: audit 2024-04-26T16:37:30.146833+0000 mon.c (mon.2) 882 : audit [DBG] from='client.? 172.21.15.50:0/2956457069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: audit 2024-04-26T16:37:30.146833+0000 mon.c (mon.2) 882 : audit [DBG] from='client.? 172.21.15.50:0/2956457069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: audit 2024-04-26T16:37:30.506266+0000 mon.c (mon.2) 883 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: audit 2024-04-26T16:37:30.506266+0000 mon.c (mon.2) 883 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: audit 2024-04-26T16:37:30.507041+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[21463]: audit 2024-04-26T16:37:30.507041+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.500591+0000 osd.0 (osd.0) 469 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.500591+0000 osd.0 (osd.0) 469 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.500767+0000 osd.0 (osd.0) 470 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.500767+0000 osd.0 (osd.0) 470 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.585264+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.585264+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: audit 2024-04-26T16:37:30.146833+0000 mon.c (mon.2) 882 : audit [DBG] from='client.? 172.21.15.50:0/2956457069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:30.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: audit 2024-04-26T16:37:30.146833+0000 mon.c (mon.2) 882 : audit [DBG] from='client.? 172.21.15.50:0/2956457069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:30.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: audit 2024-04-26T16:37:30.506266+0000 mon.c (mon.2) 883 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: audit 2024-04-26T16:37:30.506266+0000 mon.c (mon.2) 883 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: audit 2024-04-26T16:37:30.507041+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:30 smithi050 bash[29234]: audit 2024-04-26T16:37:30.507041+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.651 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:30.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.500591+0000 osd.0 (osd.0) 469 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.500591+0000 osd.0 (osd.0) 469 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.500767+0000 osd.0 (osd.0) 470 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.500767+0000 osd.0 (osd.0) 470 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.585264+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.585264+0000 mon.a (mon.0) 3879 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: audit 2024-04-26T16:37:30.146833+0000 mon.c (mon.2) 882 : audit [DBG] from='client.? 172.21.15.50:0/2956457069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: audit 2024-04-26T16:37:30.146833+0000 mon.c (mon.2) 882 : audit [DBG] from='client.? 172.21.15.50:0/2956457069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: audit 2024-04-26T16:37:30.506266+0000 mon.c (mon.2) 883 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: audit 2024-04-26T16:37:30.506266+0000 mon.c (mon.2) 883 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.766 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: audit 2024-04-26T16:37:30.507041+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.767 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:30 smithi190 bash[23868]: audit 2024-04-26T16:37:30.507041+0000 mon.a (mon.0) 3880 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:30.780 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:30.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-04-26T16:37:30.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:30.925 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:30.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:31.027 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:31.056 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:31.159 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:31.187 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:31.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:31.318 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:31.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:31.449 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:31.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:31.577 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:31.679 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:31.705 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:31.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:31.809 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.955007+0000 mgr.y (mgr.24401) 769 : cluster [DBG] pgmap v1324: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: cluster 2024-04-26T16:37:29.955007+0000 mgr.y (mgr.24401) 769 : cluster [DBG] pgmap v1324: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:30.602472+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:30.602472+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: cluster 2024-04-26T16:37:30.609848+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: cluster 2024-04-26T16:37:30.609848+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:30.612902+0000 mon.c (mon.2) 884 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:30.612902+0000 mon.c (mon.2) 884 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:30.617042+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:30.617042+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.822 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.401535+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.401535+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: cluster 2024-04-26T16:37:31.412546+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: cluster 2024-04-26T16:37:31.412546+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.478891+0000 mon.c (mon.2) 885 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.478891+0000 mon.c (mon.2) 885 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.955007+0000 mgr.y (mgr.24401) 769 : cluster [DBG] pgmap v1324: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: cluster 2024-04-26T16:37:29.955007+0000 mgr.y (mgr.24401) 769 : cluster [DBG] pgmap v1324: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:30.602472+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:30.602472+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: cluster 2024-04-26T16:37:30.609848+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-04-26T16:37:31.823 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: cluster 2024-04-26T16:37:30.609848+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:30.612902+0000 mon.c (mon.2) 884 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:30.612902+0000 mon.c (mon.2) 884 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:30.617042+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:30.617042+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.401535+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.401535+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: cluster 2024-04-26T16:37:31.412546+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: cluster 2024-04-26T16:37:31.412546+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.478891+0000 mon.c (mon.2) 885 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.478891+0000 mon.c (mon.2) 885 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.479673+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.479673+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.481018+0000 mon.c (mon.2) 886 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.481018+0000 mon.c (mon.2) 886 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.481819+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[29234]: audit 2024-04-26T16:37:31.481819+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.479673+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.479673+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.481018+0000 mon.c (mon.2) 886 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.481018+0000 mon.c (mon.2) 886 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.481819+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.825 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:31 smithi050 bash[21463]: audit 2024-04-26T16:37:31.481819+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.955007+0000 mgr.y (mgr.24401) 769 : cluster [DBG] pgmap v1324: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: cluster 2024-04-26T16:37:29.955007+0000 mgr.y (mgr.24401) 769 : cluster [DBG] pgmap v1324: 204 pgs: 204 active+clean; 4.4 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:30.602472+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:30.602472+0000 mon.a (mon.0) 3881 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: cluster 2024-04-26T16:37:30.609848+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: cluster 2024-04-26T16:37:30.609848+0000 mon.a (mon.0) 3882 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2024-04-26T16:37:31.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:30.612902+0000 mon.c (mon.2) 884 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:30.612902+0000 mon.c (mon.2) 884 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:30.617042+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:30.617042+0000 mon.a (mon.0) 3883 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.401535+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.401535+0000 mon.a (mon.0) 3884 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]': finished 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: cluster 2024-04-26T16:37:31.412546+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: cluster 2024-04-26T16:37:31.412546+0000 mon.a (mon.0) 3885 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.478891+0000 mon.c (mon.2) 885 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.478891+0000 mon.c (mon.2) 885 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.479673+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.479673+0000 mon.a (mon.0) 3886 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.481018+0000 mon.c (mon.2) 886 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.481018+0000 mon.c (mon.2) 886 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.481819+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.828 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:31 smithi190 bash[23868]: audit 2024-04-26T16:37:31.481819+0000 mon.a (mon.0) 3887 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-146"}]: dispatch 2024-04-26T16:37:31.835 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:31.938 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:31.967 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:31.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:32.019 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:32.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:32.096 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:32.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:32.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:32.225 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:32.328 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:32.356 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:32.459 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:32.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:32.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:32.618 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:32.721 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:32.747 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:32.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:32.877 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:32.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:32.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:33.009 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:33.033 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:33.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:33.136 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:33.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:33.264 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:33.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:33.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:33.395 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:33.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:33.527 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:33.543 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-04-26T16:37:33.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:33.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:31.956537+0000 mgr.y (mgr.24401) 770 : cluster [DBG] pgmap v1327: 204 pgs: 204 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:33.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:31.956537+0000 mgr.y (mgr.24401) 770 : cluster [DBG] pgmap v1327: 204 pgs: 204 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:33.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:32.424516+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-04-26T16:37:33.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:32.424516+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-04-26T16:37:33.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:32.604702+0000 mon.a (mon.0) 3889 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:33.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:32.604702+0000 mon.a (mon.0) 3889 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:33.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:32.604812+0000 mon.a (mon.0) 3890 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:37:33.639 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:33 smithi190 bash[23868]: cluster 2024-04-26T16:37:32.604812+0000 mon.a (mon.0) 3890 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:37:33.650 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:33.753 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:33.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:31.956537+0000 mgr.y (mgr.24401) 770 : cluster [DBG] pgmap v1327: 204 pgs: 204 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:33.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:31.956537+0000 mgr.y (mgr.24401) 770 : cluster [DBG] pgmap v1327: 204 pgs: 204 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:33.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:32.424516+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-04-26T16:37:33.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:32.424516+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:32.604702+0000 mon.a (mon.0) 3889 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:32.604702+0000 mon.a (mon.0) 3889 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:32.604812+0000 mon.a (mon.0) 3890 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[21463]: cluster 2024-04-26T16:37:32.604812+0000 mon.a (mon.0) 3890 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:31.956537+0000 mgr.y (mgr.24401) 770 : cluster [DBG] pgmap v1327: 204 pgs: 204 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:31.956537+0000 mgr.y (mgr.24401) 770 : cluster [DBG] pgmap v1327: 204 pgs: 204 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:32.424516+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:32.424516+0000 mon.a (mon.0) 3888 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:32.604702+0000 mon.a (mon.0) 3889 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:32.604702+0000 mon.a (mon.0) 3889 : cluster [WRN] Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:33.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:32.604812+0000 mon.a (mon.0) 3890 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:37:33.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:33 smithi050 bash[29234]: cluster 2024-04-26T16:37:32.604812+0000 mon.a (mon.0) 3890 : cluster [INF] Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-04-26T16:37:33.772 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:33.875 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:33.902 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:33.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:34.005 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:34.033 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:34.137 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:34.164 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:34.267 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:34.296 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:34.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:34.427 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:34.477 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:37:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:37:34.530 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:34.558 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:34.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:34.671 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.286924+0000 osd.7 (osd.7) 440 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.286924+0000 osd.7 (osd.7) 440 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.287128+0000 osd.7 (osd.7) 441 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.287128+0000 osd.7 (osd.7) 441 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.434644+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.434644+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: audit 2024-04-26T16:37:33.461839+0000 mon.c (mon.2) 887 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: audit 2024-04-26T16:37:33.461839+0000 mon.c (mon.2) 887 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: audit 2024-04-26T16:37:33.462787+0000 mon.a (mon.0) 3892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: audit 2024-04-26T16:37:33.462787+0000 mon.a (mon.0) 3892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: audit 2024-04-26T16:37:34.430185+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: audit 2024-04-26T16:37:34.430185+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:34.439652+0000 mon.a (mon.0) 3894 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-04-26T16:37:34.672 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[29234]: cluster 2024-04-26T16:37:34.439652+0000 mon.a (mon.0) 3894 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.286924+0000 osd.7 (osd.7) 440 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.286924+0000 osd.7 (osd.7) 440 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.287128+0000 osd.7 (osd.7) 441 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.287128+0000 osd.7 (osd.7) 441 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.434644+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.434644+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: audit 2024-04-26T16:37:33.461839+0000 mon.c (mon.2) 887 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: audit 2024-04-26T16:37:33.461839+0000 mon.c (mon.2) 887 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: audit 2024-04-26T16:37:33.462787+0000 mon.a (mon.0) 3892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: audit 2024-04-26T16:37:33.462787+0000 mon.a (mon.0) 3892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: audit 2024-04-26T16:37:34.430185+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:34.673 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: audit 2024-04-26T16:37:34.430185+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:34.439652+0000 mon.a (mon.0) 3894 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:34 smithi050 bash[21463]: cluster 2024-04-26T16:37:34.439652+0000 mon.a (mon.0) 3894 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.286924+0000 osd.7 (osd.7) 440 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.286924+0000 osd.7 (osd.7) 440 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.287128+0000 osd.7 (osd.7) 441 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.287128+0000 osd.7 (osd.7) 441 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.434644+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.434644+0000 mon.a (mon.0) 3891 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2024-04-26T16:37:34.674 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: audit 2024-04-26T16:37:33.461839+0000 mon.c (mon.2) 887 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: audit 2024-04-26T16:37:33.461839+0000 mon.c (mon.2) 887 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: audit 2024-04-26T16:37:33.462787+0000 mon.a (mon.0) 3892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: audit 2024-04-26T16:37:33.462787+0000 mon.a (mon.0) 3892 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: audit 2024-04-26T16:37:34.430185+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: audit 2024-04-26T16:37:34.430185+0000 mon.a (mon.0) 3893 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:34.439652+0000 mon.a (mon.0) 3894 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-04-26T16:37:34.675 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:34 smithi190 bash[23868]: cluster 2024-04-26T16:37:34.439652+0000 mon.a (mon.0) 3894 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2024-04-26T16:37:34.689 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:34.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:34.816 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:34.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:34.946 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:34.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:35.049 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:35.078 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:35.182 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:35.209 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:35.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:35.338 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:35.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:35.468 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.957155+0000 mgr.y (mgr.24401) 771 : cluster [DBG] pgmap v1330: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: cluster 2024-04-26T16:37:33.957155+0000 mgr.y (mgr.24401) 771 : cluster [DBG] pgmap v1330: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:34.490496+0000 mon.c (mon.2) 888 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:34.490496+0000 mon.c (mon.2) 888 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:34.490972+0000 mon.a (mon.0) 3895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:34.490972+0000 mon.a (mon.0) 3895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.552 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:35.432916+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:35.432916+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: cluster 2024-04-26T16:37:35.443036+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: cluster 2024-04-26T16:37:35.443036+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:35.453007+0000 mon.c (mon.2) 889 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:35.453007+0000 mon.c (mon.2) 889 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:35.453690+0000 mon.a (mon.0) 3898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[29234]: audit 2024-04-26T16:37:35.453690+0000 mon.a (mon.0) 3898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.957155+0000 mgr.y (mgr.24401) 771 : cluster [DBG] pgmap v1330: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: cluster 2024-04-26T16:37:33.957155+0000 mgr.y (mgr.24401) 771 : cluster [DBG] pgmap v1330: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:34.490496+0000 mon.c (mon.2) 888 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:34.490496+0000 mon.c (mon.2) 888 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:34.490972+0000 mon.a (mon.0) 3895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.553 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:34.490972+0000 mon.a (mon.0) 3895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:35.432916+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:35.432916+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: cluster 2024-04-26T16:37:35.443036+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: cluster 2024-04-26T16:37:35.443036+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:35.453007+0000 mon.c (mon.2) 889 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:35.453007+0000 mon.c (mon.2) 889 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:35.453690+0000 mon.a (mon.0) 3898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.554 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:35 smithi050 bash[21463]: audit 2024-04-26T16:37:35.453690+0000 mon.a (mon.0) 3898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.571 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:35.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.957155+0000 mgr.y (mgr.24401) 771 : cluster [DBG] pgmap v1330: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: cluster 2024-04-26T16:37:33.957155+0000 mgr.y (mgr.24401) 771 : cluster [DBG] pgmap v1330: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:34.490496+0000 mon.c (mon.2) 888 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:34.490496+0000 mon.c (mon.2) 888 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:34.490972+0000 mon.a (mon.0) 3895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:34.490972+0000 mon.a (mon.0) 3895 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:35.432916+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:35.432916+0000 mon.a (mon.0) 3896 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148", "force_nonempty": "--force-nonempty" }]': finished 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: cluster 2024-04-26T16:37:35.443036+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: cluster 2024-04-26T16:37:35.443036+0000 mon.a (mon.0) 3897 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:35.453007+0000 mon.c (mon.2) 889 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:35.453007+0000 mon.c (mon.2) 889 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:35.453690+0000 mon.a (mon.0) 3898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:35 smithi190 bash[23868]: audit 2024-04-26T16:37:35.453690+0000 mon.a (mon.0) 3898 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:35.592 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:35.642 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:37:35.695 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:35.718 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:35.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:35.849 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:35.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:35.977 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:35.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:36.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:36.107 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:36.210 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:36.239 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:36.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:36.369 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:36.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:36.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:36.499 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:36.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:36.628 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:36.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:36.758 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:36.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:36.890 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:36.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:36.993 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:37.021 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:37.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:37.146 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:37.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:37.273 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:37.376 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:37.405 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:37.443 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:37:37.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:37.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: cluster 2024-04-26T16:37:35.957913+0000 mgr.y (mgr.24401) 772 : cluster [DBG] pgmap v1333: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:37.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: cluster 2024-04-26T16:37:35.957913+0000 mgr.y (mgr.24401) 772 : cluster [DBG] pgmap v1333: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:37.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.445291+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]': finished 2024-04-26T16:37:37.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.445291+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]': finished 2024-04-26T16:37:37.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: cluster 2024-04-26T16:37:36.453312+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-04-26T16:37:37.520 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: cluster 2024-04-26T16:37:36.453312+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.494438+0000 mon.c (mon.2) 890 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.494438+0000 mon.c (mon.2) 890 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.494754+0000 mon.a (mon.0) 3901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.494754+0000 mon.a (mon.0) 3901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.495299+0000 mon.c (mon.2) 891 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.495299+0000 mon.c (mon.2) 891 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.495564+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.521 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:37 smithi190 bash[23868]: audit 2024-04-26T16:37:36.495564+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.535 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:37.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:37.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:37.660 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:37.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:37.788 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:37.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: cluster 2024-04-26T16:37:35.957913+0000 mgr.y (mgr.24401) 772 : cluster [DBG] pgmap v1333: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:37.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: cluster 2024-04-26T16:37:35.957913+0000 mgr.y (mgr.24401) 772 : cluster [DBG] pgmap v1333: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:37.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.445291+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]': finished 2024-04-26T16:37:37.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.445291+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]': finished 2024-04-26T16:37:37.834 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: cluster 2024-04-26T16:37:36.453312+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: cluster 2024-04-26T16:37:36.453312+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.494438+0000 mon.c (mon.2) 890 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.494438+0000 mon.c (mon.2) 890 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.494754+0000 mon.a (mon.0) 3901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.494754+0000 mon.a (mon.0) 3901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.495299+0000 mon.c (mon.2) 891 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.495299+0000 mon.c (mon.2) 891 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.495564+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[29234]: audit 2024-04-26T16:37:36.495564+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: cluster 2024-04-26T16:37:35.957913+0000 mgr.y (mgr.24401) 772 : cluster [DBG] pgmap v1333: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:37.835 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: cluster 2024-04-26T16:37:35.957913+0000 mgr.y (mgr.24401) 772 : cluster [DBG] pgmap v1333: 204 pgs: 6 creating+peering, 26 unknown, 172 active+clean; 4.4 MiB data, 809 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.445291+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]': finished 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.445291+0000 mon.a (mon.0) 3899 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]': finished 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: cluster 2024-04-26T16:37:36.453312+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: cluster 2024-04-26T16:37:36.453312+0000 mon.a (mon.0) 3900 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.494438+0000 mon.c (mon.2) 890 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.494438+0000 mon.c (mon.2) 890 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.494754+0000 mon.a (mon.0) 3901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.494754+0000 mon.a (mon.0) 3901 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.495299+0000 mon.c (mon.2) 891 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.495299+0000 mon.c (mon.2) 891 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.495564+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.836 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:37 smithi050 bash[21463]: audit 2024-04-26T16:37:36.495564+0000 mon.a (mon.0) 3902 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-148"}]: dispatch 2024-04-26T16:37:37.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:37.912 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:37.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:38.015 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:38.043 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:38.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:38.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:38.174 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:38.277 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:38.298 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:38.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:38.429 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:38.532 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:38.559 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[29234]: audit 2024-04-26T16:37:37.345495+0000 mgr.y (mgr.24401) 773 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[29234]: audit 2024-04-26T16:37:37.345495+0000 mgr.y (mgr.24401) 773 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[29234]: cluster 2024-04-26T16:37:37.489535+0000 mon.a (mon.0) 3903 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-04-26T16:37:38.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[29234]: cluster 2024-04-26T16:37:37.489535+0000 mon.a (mon.0) 3903 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-04-26T16:37:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[21463]: audit 2024-04-26T16:37:37.345495+0000 mgr.y (mgr.24401) 773 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:38.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[21463]: audit 2024-04-26T16:37:37.345495+0000 mgr.y (mgr.24401) 773 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[21463]: cluster 2024-04-26T16:37:37.489535+0000 mon.a (mon.0) 3903 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-04-26T16:37:38.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:38 smithi050 bash[21463]: cluster 2024-04-26T16:37:37.489535+0000 mon.a (mon.0) 3903 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-04-26T16:37:38.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:38.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:38 smithi190 bash[23868]: audit 2024-04-26T16:37:37.345495+0000 mgr.y (mgr.24401) 773 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:38.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:38 smithi190 bash[23868]: audit 2024-04-26T16:37:37.345495+0000 mgr.y (mgr.24401) 773 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:38.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:38 smithi190 bash[23868]: cluster 2024-04-26T16:37:37.489535+0000 mon.a (mon.0) 3903 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-04-26T16:37:38.673 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:38 smithi190 bash[23868]: cluster 2024-04-26T16:37:37.489535+0000 mon.a (mon.0) 3903 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2024-04-26T16:37:38.687 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:38.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:38.816 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:38.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:38.946 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:38.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:39.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:39.069 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:39.173 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:39.182 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:37:39.183 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:37:39.202 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:39.306 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:39.333 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:39.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:39.465 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:39.568 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:39.591 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:39.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: cluster 2024-04-26T16:37:37.958731+0000 mgr.y (mgr.24401) 774 : cluster [DBG] pgmap v1336: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: cluster 2024-04-26T16:37:37.958731+0000 mgr.y (mgr.24401) 774 : cluster [DBG] pgmap v1336: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: cluster 2024-04-26T16:37:38.508111+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: cluster 2024-04-26T16:37:38.508111+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: audit 2024-04-26T16:37:38.513721+0000 mon.c (mon.2) 892 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: audit 2024-04-26T16:37:38.513721+0000 mon.c (mon.2) 892 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.706 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: audit 2024-04-26T16:37:38.514864+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.707 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:39 smithi190 bash[23868]: audit 2024-04-26T16:37:38.514864+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.721 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:39.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:39.851 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: cluster 2024-04-26T16:37:37.958731+0000 mgr.y (mgr.24401) 774 : cluster [DBG] pgmap v1336: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: cluster 2024-04-26T16:37:37.958731+0000 mgr.y (mgr.24401) 774 : cluster [DBG] pgmap v1336: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: cluster 2024-04-26T16:37:38.508111+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: cluster 2024-04-26T16:37:38.508111+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: audit 2024-04-26T16:37:38.513721+0000 mon.c (mon.2) 892 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: audit 2024-04-26T16:37:38.513721+0000 mon.c (mon.2) 892 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: audit 2024-04-26T16:37:38.514864+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[21463]: audit 2024-04-26T16:37:38.514864+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: cluster 2024-04-26T16:37:37.958731+0000 mgr.y (mgr.24401) 774 : cluster [DBG] pgmap v1336: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: cluster 2024-04-26T16:37:37.958731+0000 mgr.y (mgr.24401) 774 : cluster [DBG] pgmap v1336: 172 pgs: 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: cluster 2024-04-26T16:37:38.508111+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: cluster 2024-04-26T16:37:38.508111+0000 mon.a (mon.0) 3904 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: audit 2024-04-26T16:37:38.513721+0000 mon.c (mon.2) 892 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: audit 2024-04-26T16:37:38.513721+0000 mon.c (mon.2) 892 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: audit 2024-04-26T16:37:38.514864+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:39 smithi050 bash[29234]: audit 2024-04-26T16:37:38.514864+0000 mon.a (mon.0) 3905 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:39.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-04-26T16:37:39.955 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:39.977 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:39.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:40.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 -- 172.21.15.50:0/1312942645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102320 msgr2=0x7f38981027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 --2- 172.21.15.50:0/1312942645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102320 0x7f38981027a0 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f3888009990 tx=0x7f3888009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 -- 172.21.15.50:0/1312942645 shutdown_connections 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 --2- 172.21.15.50:0/1312942645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898108390 0x7f3898108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 --2- 172.21.15.50:0/1312942645 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102ce0 0x7f389810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 --2- 172.21.15.50:0/1312942645 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102320 0x7f38981027a0 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.075+0000 7f389f917700 1 -- 172.21.15.50:0/1312942645 >> 172.21.15.50:0/1312942645 conn(0x7f38980792d0 msgr2=0x7f38980796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- 172.21.15.50:0/1312942645 shutdown_connections 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- 172.21.15.50:0/1312942645 wait complete. 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 librados: starting msgr at 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 librados: starting objecter 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 Processor -- start 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- start start 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 librados: setting wanted keys 2024-04-26T16:37:40.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 librados: calling monclient init 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102320 0x7f38981991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 0x7f3898199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898108390 0x7f389819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f38981119a0 con 0x7f3898102320 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3898111820 con 0x7f3898108390 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3898111b20 con 0x7f3898102ce0 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 0x7f3898199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389d6b3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102320 0x7f38981991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 0x7f3898199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60364/0 (socket says 172.21.15.50:60364) 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 -- 172.21.15.50:0/1022105459 learned_addr learned my addr 172.21.15.50:0/1022105459 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389deb4700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898108390 0x7f389819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 -- 172.21.15.50:0/1022105459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898108390 msgr2=0x7f389819da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898108390 0x7f389819da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 -- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102320 msgr2=0x7f38981991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102320 0x7f38981991d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 -- 172.21.15.50:0/1022105459 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f389819e190 con 0x7f3898102ce0 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389d6b3700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102320 0x7f38981991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389ceb2700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 0x7f3898199710 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f38880047b0 tx=0x7f388800bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f388e7fc700 10 client.?.objecter ms_handle_connect 0x7f3898102ce0 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f388e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888026070 con 0x7f3898102ce0 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f38880047e0 con 0x7f3898102ce0 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f389819e420 con 0x7f3898102ce0 2024-04-26T16:37:40.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3888005270 con 0x7f3898102ce0 2024-04-26T16:37:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 10 client.41708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38981a5d00 con 0x7f3898102ce0 2024-04-26T16:37:40.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 librados: init done 2024-04-26T16:37:40.083 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.079+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f389806a110 con 0x7f3898102ce0 2024-04-26T16:37:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3888029070 con 0x7f3898102ce0 2024-04-26T16:37:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(859..859 src has 577..859) ==== 9952+0+0 (secure 0 0 0) 0x7f3888092660 con 0x7f3898102ce0 2024-04-26T16:37:40.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 10 client.41708.objecter ms_dispatch 0x7f3898001550 osd_map(859..859 src has 577..859) 2024-04-26T16:37:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 3 client.41708.objecter handle_osd_map got epochs [859,859] > 0 2024-04-26T16:37:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 3 client.41708.objecter handle_osd_map decoding full epoch 859 2024-04-26T16:37:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f389d6b3700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 20 client.41708.objecter dump_active .. 0 homeless 2024-04-26T16:37:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f389d6b3700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 crypto rx=0x7f3894005950 tx=0x7f389400b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:40.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.083+0000 7f388e7fc700 10 client.41708.objecter ms_handle_connect 0x7f3884071f80 2024-04-26T16:37:40.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.087+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3888013070 con 0x7f3898102ce0 2024-04-26T16:37:40.103 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:40.206 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:40.230 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:40.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:40.334 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.335+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7f38981a5ed0 con 0x7f3898102ce0 2024-04-26T16:37:40.361 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:40.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- 172.21.15.50:0/3375589534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58102880 msgr2=0x7f3b58102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/3375589534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58102880 0x7f3b58102c60 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800b0d0 tx=0x7f3b4800b3e0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- 172.21.15.50:0/3375589534 shutdown_connections 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/3375589534 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5810f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/3375589534 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58102880 0x7f3b58102c60 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/3375589534 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b58103230 0x7f3b58107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- 172.21.15.50:0/3375589534 >> 172.21.15.50:0/3375589534 conn(0x7f3b58079120 msgr2=0x7f3b58079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- 172.21.15.50:0/3375589534 shutdown_connections 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- 172.21.15.50:0/3375589534 wait complete. 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 librados: starting msgr at 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 librados: starting objecter 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 Processor -- start 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- start start 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 librados: setting wanted keys 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 librados: calling monclient init 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b58102880 0x7f3b58199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.403 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 0x7f3b581997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3b58111a90 con 0x7f3b58103230 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3b58111910 con 0x7f3b58102880 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b5d67c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3b58111c10 con 0x7f3b581076e0 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b4e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 0x7f3b581997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b577fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b577fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5819db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:60392/0 (socket says 172.21.15.50:60392) 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b4e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 0x7f3b581997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:37468/0 (socket says 172.21.15.50:37468) 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b577fe700 1 -- 172.21.15.50:0/1986511346 learned_addr learned my addr 172.21.15.50:0/1986511346 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.399+0000 7f3b56ffd700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b58102880 0x7f3b58199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b4e7fc700 1 -- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 msgr2=0x7f3b5819db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b4e7fc700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5819db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.404 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b4e7fc700 1 -- 172.21.15.50:0/1986511346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b58102880 msgr2=0x7f3b58199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b4e7fc700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b58102880 0x7f3b58199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b4e7fc700 1 -- 172.21.15.50:0/1986511346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b5819e240 con 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b577fe700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5819db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b4e7fc700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 0x7f3b581997c0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f3b5806bba0 tx=0x7f3b4000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 10 client.?.objecter ms_handle_connect 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b40016070 con 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3b4000fcf0 con 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b40010d40 con 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b5819e530 con 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b5d67c700 10 client.31095.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b581a5db0 con 0x7f3b58103230 2024-04-26T16:37:40.405 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b5d67c700 1 librados: init done 2024-04-26T16:37:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b5806a110 con 0x7f3b58103230 2024-04-26T16:37:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3b40019070 con 0x7f3b58103230 2024-04-26T16:37:40.406 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b44071f80 0x7f3b44074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(859..859 src has 577..859) ==== 9952+0+0 (secure 0 0 0) 0x7f3b40091690 con 0x7f3b58103230 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 10 client.31095.objecter ms_dispatch 0x7f3b58001550 osd_map(859..859 src has 577..859) 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 3 client.31095.objecter handle_osd_map got epochs [859,859] > 0 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 3 client.31095.objecter handle_osd_map decoding full epoch 859 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b56ffd700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b44071f80 0x7f3b44074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 20 client.31095.objecter dump_active .. 0 homeless 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b56ffd700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b44071f80 0x7f3b44074440 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009fd0 tx=0x7f3b4800b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:40.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.403+0000 7f3b54ff9700 10 client.31095.objecter ms_handle_connect 0x7f3b44071f80 2024-04-26T16:37:40.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.411+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3b4005ce10 con 0x7f3b58103230 2024-04-26T16:37:40.464 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:40.487 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:40.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.535+0000 7f388e7fc700 1 -- 172.21.15.50:0/1022105459 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v860) ==== 113+0+0 (secure 0 0 0) 0x7f388805de60 con 0x7f3898102ce0 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 10 librados: watch_flush enter 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 10 librados: watch_flush exit 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 20 client.41708.objecter shutdown clearing up homeless session... 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 10 client.41708.objecter successfully canceled tick 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 msgr2=0x7f3884074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 crypto rx=0x7f3894005950 tx=0x7f389400b410 comp rx=0 tx=0).stop 2024-04-26T16:37:40.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 msgr2=0x7f3898199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:40.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 0x7f3898199710 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f38880047b0 tx=0x7f388800bfb0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 shutdown_connections 2024-04-26T16:37:40.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3884071f80 0x7f3884074440 unknown :-1 s=CLOSED pgs=3843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3898108390 0x7f389819da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3898102320 0x7f38981991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 --2- 172.21.15.50:0/1022105459 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3898102ce0 0x7f3898199710 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:40.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.539+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 >> 172.21.15.50:0/1022105459 conn(0x7f38980792d0 msgr2=0x7f3898101510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:40.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.543+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 shutdown_connections 2024-04-26T16:37:40.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.543+0000 7f389f917700 1 -- 172.21.15.50:0/1022105459 wait complete. 2024-04-26T16:37:40.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.543+0000 7f389f917700 1 librados: shutdown 2024-04-26T16:37:40.544 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T16:37:40.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:40.607 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.501890+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:40.607 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.501890+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: cluster 2024-04-26T16:37:39.511038+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: cluster 2024-04-26T16:37:39.511038+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.582003+0000 mon.c (mon.2) 893 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.582003+0000 mon.c (mon.2) 893 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.582539+0000 mon.a (mon.0) 3908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.582539+0000 mon.a (mon.0) 3908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.583495+0000 mon.c (mon.2) 894 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.608 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.583495+0000 mon.c (mon.2) 894 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.583963+0000 mon.a (mon.0) 3909 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:39.583963+0000 mon.a (mon.0) 3909 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:40.339640+0000 mon.c (mon.2) 895 : audit [INF] from='client.? 172.21.15.50:0/1022105459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:40.339640+0000 mon.c (mon.2) 895 : audit [INF] from='client.? 172.21.15.50:0/1022105459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:40.340012+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.609 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:40 smithi190 bash[23868]: audit 2024-04-26T16:37:40.340012+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.619 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:40.660 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:40.659+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7f3b581a5fe0 con 0x7f3b58103230 2024-04-26T16:37:40.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:40.750 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:40.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.501890+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:40.824 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.501890+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: cluster 2024-04-26T16:37:39.511038+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: cluster 2024-04-26T16:37:39.511038+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.582003+0000 mon.c (mon.2) 893 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.582003+0000 mon.c (mon.2) 893 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.582539+0000 mon.a (mon.0) 3908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.582539+0000 mon.a (mon.0) 3908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.583495+0000 mon.c (mon.2) 894 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.583495+0000 mon.c (mon.2) 894 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.583963+0000 mon.a (mon.0) 3909 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.825 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:39.583963+0000 mon.a (mon.0) 3909 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:40.339640+0000 mon.c (mon.2) 895 : audit [INF] from='client.? 172.21.15.50:0/1022105459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:40.339640+0000 mon.c (mon.2) 895 : audit [INF] from='client.? 172.21.15.50:0/1022105459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:40.340012+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[29234]: audit 2024-04-26T16:37:40.340012+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.501890+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.501890+0000 mon.a (mon.0) 3906 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:40.826 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: cluster 2024-04-26T16:37:39.511038+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: cluster 2024-04-26T16:37:39.511038+0000 mon.a (mon.0) 3907 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.582003+0000 mon.c (mon.2) 893 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.582003+0000 mon.c (mon.2) 893 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.582539+0000 mon.a (mon.0) 3908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.582539+0000 mon.a (mon.0) 3908 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.583495+0000 mon.c (mon.2) 894 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.583495+0000 mon.c (mon.2) 894 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.583963+0000 mon.a (mon.0) 3909 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:39.583963+0000 mon.a (mon.0) 3909 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-150"}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:40.339640+0000 mon.c (mon.2) 895 : audit [INF] from='client.? 172.21.15.50:0/1022105459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:40.339640+0000 mon.c (mon.2) 895 : audit [INF] from='client.? 172.21.15.50:0/1022105459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:40.340012+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.827 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:40 smithi050 bash[21463]: audit 2024-04-26T16:37:40.340012+0000 mon.a (mon.0) 3910 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:37:40.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:40.880 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:40.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:40.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:40.983 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:41.012 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:41.115 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:41.143 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:41.246 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:41.274 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:41.377 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:41.402 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.403+0000 7f3b54ff9700 1 -- 172.21.15.50:0/1986511346 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v861) ==== 91+0+0 (secure 0 0 0) 0x7f3b40060ac0 con 0x7f3b58103230 2024-04-26T16:37:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 10 librados: watch_flush enter 2024-04-26T16:37:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 10 librados: watch_flush exit 2024-04-26T16:37:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 20 client.31095.objecter shutdown clearing up homeless session... 2024-04-26T16:37:41.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 10 client.31095.objecter successfully canceled tick 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b44071f80 msgr2=0x7f3b44074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b44071f80 0x7f3b44074440 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 crypto rx=0x7f3b48009fd0 tx=0x7f3b4800b040 comp rx=0 tx=0).stop 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 msgr2=0x7f3b581997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 0x7f3b581997c0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f3b5806bba0 tx=0x7f3b4000dc30 comp rx=0 tx=0).stop 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 shutdown_connections 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3b44071f80 0x7f3b44074440 unknown :-1 s=CLOSED pgs=3844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3b58102880 0x7f3b58199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3b581076e0 0x7f3b5819db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 --2- 172.21.15.50:0/1986511346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3b58103230 0x7f3b581997c0 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 >> 172.21.15.50:0/1986511346 conn(0x7f3b58079120 msgr2=0x7f3b5806bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 shutdown_connections 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 -- 172.21.15.50:0/1986511346 wait complete. 2024-04-26T16:37:41.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:41.407+0000 7f3b5d67c700 1 librados: shutdown 2024-04-26T16:37:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:41.420 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:37:41.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:41.540 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:39.959396+0000 mgr.y (mgr.24401) 775 : cluster [DBG] pgmap v1339: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 511 B/s wr, 32 op/s 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:39.959396+0000 mgr.y (mgr.24401) 775 : cluster [DBG] pgmap v1339: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 511 B/s wr, 32 op/s 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:40.353292+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:40.353292+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:40.353595+0000 osd.5 (osd.5) 319 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:40.353595+0000 osd.5 (osd.5) 319 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:40.533383+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:40.533383+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:40.538373+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:40.538373+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:40.665600+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:37:41.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:40.665600+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.402717+0000 mon.a (mon.0) 3914 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.402717+0000 mon.a (mon.0) 3914 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.404302+0000 mon.a (mon.0) 3915 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.404302+0000 mon.a (mon.0) 3915 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:41.406884+0000 mon.a (mon.0) 3916 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:41.406884+0000 mon.a (mon.0) 3916 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.415160+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.415160+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:41.422576+0000 mon.c (mon.2) 896 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:39.959396+0000 mgr.y (mgr.24401) 775 : cluster [DBG] pgmap v1339: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 511 B/s wr, 32 op/s 2024-04-26T16:37:41.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:39.959396+0000 mgr.y (mgr.24401) 775 : cluster [DBG] pgmap v1339: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 511 B/s wr, 32 op/s 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:40.353292+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:40.353292+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:40.353595+0000 osd.5 (osd.5) 319 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:40.353595+0000 osd.5 (osd.5) 319 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:40.533383+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:40.533383+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:40.538373+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:40.538373+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:40.665600+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:40.665600+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.402717+0000 mon.a (mon.0) 3914 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.402717+0000 mon.a (mon.0) 3914 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.404302+0000 mon.a (mon.0) 3915 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.404302+0000 mon.a (mon.0) 3915 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:41.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:41.406884+0000 mon.a (mon.0) 3916 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:41.406884+0000 mon.a (mon.0) 3916 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.415160+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.415160+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:41.422576+0000 mon.c (mon.2) 896 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:41.422576+0000 mon.c (mon.2) 896 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:41.428817+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[21463]: audit 2024-04-26T16:37:41.428817+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:41.422576+0000 mon.c (mon.2) 896 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:41.428817+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.635 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:41 smithi050 bash[29234]: audit 2024-04-26T16:37:41.428817+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.643 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:41.671 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:41.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:41.804 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:39.959396+0000 mgr.y (mgr.24401) 775 : cluster [DBG] pgmap v1339: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 511 B/s wr, 32 op/s 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:39.959396+0000 mgr.y (mgr.24401) 775 : cluster [DBG] pgmap v1339: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 813 MiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 511 B/s wr, 32 op/s 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:40.353292+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:40.353292+0000 osd.5 (osd.5) 318 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:40.353595+0000 osd.5 (osd.5) 319 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:40.353595+0000 osd.5 (osd.5) 319 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:40.533383+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:40.533383+0000 mon.a (mon.0) 3911 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:37:41.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:40.538373+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:40.538373+0000 mon.a (mon.0) 3912 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:40.665600+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:40.665600+0000 mon.a (mon.0) 3913 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.402717+0000 mon.a (mon.0) 3914 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.402717+0000 mon.a (mon.0) 3914 : cluster [WRN] Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.404302+0000 mon.a (mon.0) 3915 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.404302+0000 mon.a (mon.0) 3915 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:41.406884+0000 mon.a (mon.0) 3916 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:41.406884+0000 mon.a (mon.0) 3916 : audit [INF] from='client.? 172.21.15.50:0/1986511346' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.415160+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.415160+0000 mon.a (mon.0) 3917 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:41.422576+0000 mon.c (mon.2) 896 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:41.422576+0000 mon.c (mon.2) 896 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:41.428817+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:41 smithi190 bash[23868]: audit 2024-04-26T16:37:41.428817+0000 mon.a (mon.0) 3918 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:41.907 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:41.935 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:41.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:42.017 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-26T16:37:42.037 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:42.086 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:42.190 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:42.211 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:42.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:42.341 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:42.444 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:42.476 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:42.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:42.605 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:42.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:42.708 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:42.735 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:42.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:42.865 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:42.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:42.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:42.997 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:43.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:43.128 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:43.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:43.145 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:43.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:43.257 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:43.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:43.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:43.389 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:43.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:43.517 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:43.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:43.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.960696+0000 mgr.y (mgr.24401) 776 : cluster [DBG] pgmap v1342: 204 pgs: 27 creating+peering, 5 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 255 B/s wr, 162 op/s 2024-04-26T16:37:43.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:41.960696+0000 mgr.y (mgr.24401) 776 : cluster [DBG] pgmap v1342: 204 pgs: 27 creating+peering, 5 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 255 B/s wr, 162 op/s 2024-04-26T16:37:43.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.409836+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:43.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.409836+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:43.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:42.422698+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:42.422698+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.503397+0000 mon.c (mon.2) 897 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.503397+0000 mon.c (mon.2) 897 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.503773+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.503773+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.504362+0000 mon.c (mon.2) 898 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.504362+0000 mon.c (mon.2) 898 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.504657+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:42.504657+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:42.745279+0000 osd.3 (osd.3) 475 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:37:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:42.745279+0000 osd.3 (osd.3) 475 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:37:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:42.746783+0000 osd.3 (osd.3) 476 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:37:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: cluster 2024-04-26T16:37:42.746783+0000 osd.3 (osd.3) 476 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:37:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:43.019717+0000 mon.a (mon.0) 3923 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:43.019717+0000 mon.a (mon.0) 3923 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:43.022648+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:43 smithi190 bash[23868]: audit 2024-04-26T16:37:43.022648+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:43.645 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:43.748 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:43.760 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.960696+0000 mgr.y (mgr.24401) 776 : cluster [DBG] pgmap v1342: 204 pgs: 27 creating+peering, 5 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 255 B/s wr, 162 op/s 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:41.960696+0000 mgr.y (mgr.24401) 776 : cluster [DBG] pgmap v1342: 204 pgs: 27 creating+peering, 5 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 255 B/s wr, 162 op/s 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.409836+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.409836+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:42.422698+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:42.422698+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.503397+0000 mon.c (mon.2) 897 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.503397+0000 mon.c (mon.2) 897 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.503773+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.503773+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.504362+0000 mon.c (mon.2) 898 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.504362+0000 mon.c (mon.2) 898 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.504657+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.761 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:42.504657+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:42.745279+0000 osd.3 (osd.3) 475 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:42.745279+0000 osd.3 (osd.3) 475 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:42.746783+0000 osd.3 (osd.3) 476 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.960696+0000 mgr.y (mgr.24401) 776 : cluster [DBG] pgmap v1342: 204 pgs: 27 creating+peering, 5 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 255 B/s wr, 162 op/s 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:41.960696+0000 mgr.y (mgr.24401) 776 : cluster [DBG] pgmap v1342: 204 pgs: 27 creating+peering, 5 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 98 KiB/s rd, 255 B/s wr, 162 op/s 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.409836+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.409836+0000 mon.a (mon.0) 3919 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:42.422698+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:42.422698+0000 mon.a (mon.0) 3920 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.503397+0000 mon.c (mon.2) 897 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.503397+0000 mon.c (mon.2) 897 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.762 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.503773+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.503773+0000 mon.a (mon.0) 3921 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.504362+0000 mon.c (mon.2) 898 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.504362+0000 mon.c (mon.2) 898 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.504657+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:42.504657+0000 mon.a (mon.0) 3922 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-152"}]: dispatch 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:42.745279+0000 osd.3 (osd.3) 475 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:42.745279+0000 osd.3 (osd.3) 475 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:42.746783+0000 osd.3 (osd.3) 476 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: cluster 2024-04-26T16:37:42.746783+0000 osd.3 (osd.3) 476 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:43.019717+0000 mon.a (mon.0) 3923 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:43.019717+0000 mon.a (mon.0) 3923 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:43.763 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:43.022648+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:43.764 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[21463]: audit 2024-04-26T16:37:43.022648+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:43.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: cluster 2024-04-26T16:37:42.746783+0000 osd.3 (osd.3) 476 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:37:43.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:43.019717+0000 mon.a (mon.0) 3923 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:43.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:43.019717+0000 mon.a (mon.0) 3923 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:43.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:43.022648+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:43.764 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:43 smithi050 bash[29234]: audit 2024-04-26T16:37:43.022648+0000 mon.c (mon.2) 899 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:43.776 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:43.879 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:43.906 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:43.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:44.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:44.035 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:44.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:44.166 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:44.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:44.295 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:44.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:44.446 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:44.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.254567+0000 osd.6 (osd.6) 29 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.254567+0000 osd.6 (osd.6) 29 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.255956+0000 osd.6 (osd.6) 30 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.255956+0000 osd.6 (osd.6) 30 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.420413+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.420413+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.947677+0000 osd.4 (osd.4) 413 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.947677+0000 osd.4 (osd.4) 413 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.955674+0000 osd.4 (osd.4) 414 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:37:44.563 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:44 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.955674+0000 osd.4 (osd.4) 414 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:37:44.579 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:44.682 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:44.692 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:37:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:37:44.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.254567+0000 osd.6 (osd.6) 29 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:37:44.692 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.254567+0000 osd.6 (osd.6) 29 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.255956+0000 osd.6 (osd.6) 30 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.255956+0000 osd.6 (osd.6) 30 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.420413+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.420413+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.947677+0000 osd.4 (osd.4) 413 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.947677+0000 osd.4 (osd.4) 413 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.955674+0000 osd.4 (osd.4) 414 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.955674+0000 osd.4 (osd.4) 414 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.254567+0000 osd.6 (osd.6) 29 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.254567+0000 osd.6 (osd.6) 29 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.255956+0000 osd.6 (osd.6) 30 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.255956+0000 osd.6 (osd.6) 30 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:37:44.693 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.420413+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-04-26T16:37:44.694 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.420413+0000 mon.a (mon.0) 3924 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2024-04-26T16:37:44.694 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.947677+0000 osd.4 (osd.4) 413 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:37:44.694 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.947677+0000 osd.4 (osd.4) 413 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:37:44.694 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.955674+0000 osd.4 (osd.4) 414 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:37:44.694 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:44 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.955674+0000 osd.4 (osd.4) 414 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:37:44.706 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:44.709 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:37:44.742 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:37:44.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:44.835 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:44.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:44.966 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:44.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:45.069 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:45.093 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:45.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:45.223 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:45.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:45.354 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:45.457 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:45.498 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:45.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:45.628 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.961251+0000 mgr.y (mgr.24401) 777 : cluster [DBG] pgmap v1345: 172 pgs: 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 150 op/s 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:43.961251+0000 mgr.y (mgr.24401) 777 : cluster [DBG] pgmap v1345: 172 pgs: 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 150 op/s 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.216716+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.216716+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.221814+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.221814+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.449112+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-04-26T16:37:45.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.449112+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: audit 2024-04-26T16:37:44.452927+0000 mon.c (mon.2) 900 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: audit 2024-04-26T16:37:44.452927+0000 mon.c (mon.2) 900 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: audit 2024-04-26T16:37:44.455336+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: audit 2024-04-26T16:37:44.455336+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.753784+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.753784+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.755310+0000 osd.3 (osd.3) 478 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.755310+0000 osd.3 (osd.3) 478 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.961251+0000 mgr.y (mgr.24401) 777 : cluster [DBG] pgmap v1345: 172 pgs: 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 150 op/s 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:43.961251+0000 mgr.y (mgr.24401) 777 : cluster [DBG] pgmap v1345: 172 pgs: 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 150 op/s 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.216716+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.216716+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:37:45.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.221814+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.221814+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.449112+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.449112+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: audit 2024-04-26T16:37:44.452927+0000 mon.c (mon.2) 900 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: audit 2024-04-26T16:37:44.452927+0000 mon.c (mon.2) 900 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: audit 2024-04-26T16:37:44.455336+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: audit 2024-04-26T16:37:44.455336+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.753784+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.753784+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.755310+0000 osd.3 (osd.3) 478 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:37:45.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:45 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.755310+0000 osd.3 (osd.3) 478 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:37:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:45.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:45.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.961251+0000 mgr.y (mgr.24401) 777 : cluster [DBG] pgmap v1345: 172 pgs: 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 150 op/s 2024-04-26T16:37:45.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:43.961251+0000 mgr.y (mgr.24401) 777 : cluster [DBG] pgmap v1345: 172 pgs: 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 90 KiB/s rd, 255 B/s wr, 150 op/s 2024-04-26T16:37:45.746 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.216716+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.216716+0000 osd.6 (osd.6) 31 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.221814+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.221814+0000 osd.6 (osd.6) 32 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.449112+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.449112+0000 mon.a (mon.0) 3925 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: audit 2024-04-26T16:37:44.452927+0000 mon.c (mon.2) 900 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: audit 2024-04-26T16:37:44.452927+0000 mon.c (mon.2) 900 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: audit 2024-04-26T16:37:44.455336+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: audit 2024-04-26T16:37:44.455336+0000 mon.a (mon.0) 3926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-04-26T16:37:45.747 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.753784+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:37:45.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.753784+0000 osd.3 (osd.3) 477 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:37:45.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.755310+0000 osd.3 (osd.3) 478 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:37:45.748 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:45 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.755310+0000 osd.3 (osd.3) 478 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:37:45.762 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:45.865 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:45.893 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:45.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:45.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:46.025 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:46.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:46.150 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:46.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:46.274 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:46.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:46.399 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:46.502 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:46.527 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:46.558 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.978136+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.978136+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.980061+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:44.980061+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.118521+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.118521+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.122912+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:37:46.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.122912+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.264194+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.264194+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.265855+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.265855+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.441081+0000 mon.a (mon.0) 3927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.441081+0000 mon.a (mon.0) 3927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.450060+0000 mon.a (mon.0) 3928 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.450060+0000 mon.a (mon.0) 3928 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.457029+0000 mon.c (mon.2) 901 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.457029+0000 mon.c (mon.2) 901 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.459847+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.459847+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.523122+0000 mon.c (mon.2) 902 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.523122+0000 mon.c (mon.2) 902 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.523469+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.523469+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.524063+0000 mon.c (mon.2) 903 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.524063+0000 mon.c (mon.2) 903 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.524348+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: audit 2024-04-26T16:37:45.524348+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.890562+0000 osd.4 (osd.4) 415 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.890562+0000 osd.4 (osd.4) 415 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.892776+0000 osd.4 (osd.4) 416 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:37:46.577 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:46 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.892776+0000 osd.4 (osd.4) 416 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:37:46.631 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:46.655 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1 2024-04-26T16:37:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.1' 2024-04-26T16:37:46.730 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.978136+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.978136+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.980061+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:44.980061+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.118521+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.118521+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.122912+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.122912+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.264194+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.264194+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.265855+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.265855+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.441081+0000 mon.a (mon.0) 3927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:46.731 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.441081+0000 mon.a (mon.0) 3927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.450060+0000 mon.a (mon.0) 3928 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.450060+0000 mon.a (mon.0) 3928 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.457029+0000 mon.c (mon.2) 901 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.457029+0000 mon.c (mon.2) 901 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.459847+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.459847+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.523122+0000 mon.c (mon.2) 902 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.523122+0000 mon.c (mon.2) 902 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.523469+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.523469+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.524063+0000 mon.c (mon.2) 903 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.524063+0000 mon.c (mon.2) 903 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.524348+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.732 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: audit 2024-04-26T16:37:45.524348+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.890562+0000 osd.4 (osd.4) 415 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.890562+0000 osd.4 (osd.4) 415 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.892776+0000 osd.4 (osd.4) 416 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.892776+0000 osd.4 (osd.4) 416 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.978136+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.978136+0000 osd.0 (osd.0) 471 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:37:46.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.980061+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:44.980061+0000 osd.0 (osd.0) 472 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.118521+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.118521+0000 osd.5 (osd.5) 320 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.122912+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.122912+0000 osd.5 (osd.5) 321 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.264194+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.264194+0000 osd.6 (osd.6) 33 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.265855+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.265855+0000 osd.6 (osd.6) 34 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.441081+0000 mon.a (mon.0) 3927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.441081+0000 mon.a (mon.0) 3927 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi050-73372-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.450060+0000 mon.a (mon.0) 3928 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-04-26T16:37:46.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.450060+0000 mon.a (mon.0) 3928 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.457029+0000 mon.c (mon.2) 901 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.457029+0000 mon.c (mon.2) 901 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.459847+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.459847+0000 mon.a (mon.0) 3929 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi050-73372-111","var": "dedup_tier","val": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.523122+0000 mon.c (mon.2) 902 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.523122+0000 mon.c (mon.2) 902 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.523469+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.523469+0000 mon.a (mon.0) 3930 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.524063+0000 mon.c (mon.2) 903 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.524063+0000 mon.c (mon.2) 903 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.524348+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: audit 2024-04-26T16:37:45.524348+0000 mon.a (mon.0) 3931 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi050-73372-111", "tierpool": "test-rados-api-smithi050-73372-154"}]: dispatch 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.890562+0000 osd.4 (osd.4) 415 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:37:46.735 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.890562+0000 osd.4 (osd.4) 415 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:37:46.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.892776+0000 osd.4 (osd.4) 416 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:37:46.736 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:46 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.892776+0000 osd.4 (osd.4) 416 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:37:46.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:46.784 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:46.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:46.913 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:46.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:47.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:47.045 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:47.148 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.175 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:47.278 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:47.307 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:47.410 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.439 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:47.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:37:47.542 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.566 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:47.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:47.696 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:47.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:47.799 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:47.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.962025+0000 mgr.y (mgr.24401) 778 : cluster [DBG] pgmap v1348: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:47.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:45.962025+0000 mgr.y (mgr.24401) 778 : cluster [DBG] pgmap v1348: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:47.811 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.490613+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.490613+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.740559+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.740559+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.742007+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.742007+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.962025+0000 mgr.y (mgr.24401) 778 : cluster [DBG] pgmap v1348: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:45.962025+0000 mgr.y (mgr.24401) 778 : cluster [DBG] pgmap v1348: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.490613+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.490613+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.740559+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:37:47.812 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.740559+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:37:47.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.742007+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:37:47.813 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:47 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.742007+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:37:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.962025+0000 mgr.y (mgr.24401) 778 : cluster [DBG] pgmap v1348: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:45.962025+0000 mgr.y (mgr.24401) 778 : cluster [DBG] pgmap v1348: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.490613+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-04-26T16:37:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.490613+0000 mon.a (mon.0) 3932 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2024-04-26T16:37:47.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.740559+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:37:47.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.740559+0000 osd.3 (osd.3) 479 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:37:47.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.742007+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:37:47.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:47 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.742007+0000 osd.3 (osd.3) 480 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:37:47.828 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:47.931 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.955 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:47.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:48.058 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:48.088 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:48.192 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:48.219 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:48.264 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-04-26T16:37:48.265 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-04-26T16:37:48.322 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:48.351 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:48.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:48.482 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:48.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:48.600 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:37:48.613 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:48.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:48.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.975456+0000 osd.0 (osd.0) 473 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:37:48.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.975456+0000 osd.0 (osd.0) 473 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:37:48.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.980087+0000 osd.0 (osd.0) 474 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:37:48.728 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:46.980087+0000 osd.0 (osd.0) 474 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: audit 2024-04-26T16:37:47.353767+0000 mgr.y (mgr.24401) 779 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: audit 2024-04-26T16:37:47.353767+0000 mgr.y (mgr.24401) 779 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.356034+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.356034+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.358700+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.358700+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.394677+0000 osd.2 (osd.2) 409 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.394677+0000 osd.2 (osd.2) 409 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.396485+0000 osd.2 (osd.2) 410 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.396485+0000 osd.2 (osd.2) 410 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.491657+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.491657+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: audit 2024-04-26T16:37:47.502497+0000 mon.c (mon.2) 904 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:48.729 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: audit 2024-04-26T16:37:47.502497+0000 mon.c (mon.2) 904 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: audit 2024-04-26T16:37:47.502882+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: audit 2024-04-26T16:37:47.502882+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.746818+0000 osd.3 (osd.3) 481 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.746818+0000 osd.3 (osd.3) 481 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.750258+0000 osd.3 (osd.3) 482 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.750258+0000 osd.3 (osd.3) 482 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.844649+0000 osd.4 (osd.4) 417 : cluster [DBG] 5.e scrub starts 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.844649+0000 osd.4 (osd.4) 417 : cluster [DBG] 5.e scrub starts 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.846481+0000 osd.4 (osd.4) 418 : cluster [DBG] 5.e scrub ok 2024-04-26T16:37:48.730 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:48 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.846481+0000 osd.4 (osd.4) 418 : cluster [DBG] 5.e scrub ok 2024-04-26T16:37:48.744 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:48.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.975456+0000 osd.0 (osd.0) 473 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:37:48.750 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.975456+0000 osd.0 (osd.0) 473 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:37:48.847 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:48.876 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:48.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:48.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:49.005 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:49.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:49.108 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.975456+0000 osd.0 (osd.0) 473 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.975456+0000 osd.0 (osd.0) 473 : cluster [DBG] 323.4s0 scrub starts 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.980087+0000 osd.0 (osd.0) 474 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:46.980087+0000 osd.0 (osd.0) 474 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: audit 2024-04-26T16:37:47.353767+0000 mgr.y (mgr.24401) 779 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: audit 2024-04-26T16:37:47.353767+0000 mgr.y (mgr.24401) 779 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.356034+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.356034+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:37:49.121 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.358700+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.358700+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.394677+0000 osd.2 (osd.2) 409 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.394677+0000 osd.2 (osd.2) 409 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.396485+0000 osd.2 (osd.2) 410 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.396485+0000 osd.2 (osd.2) 410 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.491657+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.491657+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: audit 2024-04-26T16:37:47.502497+0000 mon.c (mon.2) 904 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: audit 2024-04-26T16:37:47.502497+0000 mon.c (mon.2) 904 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: audit 2024-04-26T16:37:47.502882+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: audit 2024-04-26T16:37:47.502882+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.746818+0000 osd.3 (osd.3) 481 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.746818+0000 osd.3 (osd.3) 481 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:37:49.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.750258+0000 osd.3 (osd.3) 482 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.750258+0000 osd.3 (osd.3) 482 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.844649+0000 osd.4 (osd.4) 417 : cluster [DBG] 5.e scrub starts 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.844649+0000 osd.4 (osd.4) 417 : cluster [DBG] 5.e scrub starts 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.846481+0000 osd.4 (osd.4) 418 : cluster [DBG] 5.e scrub ok 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.846481+0000 osd.4 (osd.4) 418 : cluster [DBG] 5.e scrub ok 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.980087+0000 osd.0 (osd.0) 474 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:46.980087+0000 osd.0 (osd.0) 474 : cluster [DBG] 323.4 scrub ok 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: audit 2024-04-26T16:37:47.353767+0000 mgr.y (mgr.24401) 779 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: audit 2024-04-26T16:37:47.353767+0000 mgr.y (mgr.24401) 779 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.356034+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.356034+0000 osd.6 (osd.6) 35 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:37:49.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.358700+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.358700+0000 osd.6 (osd.6) 36 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.394677+0000 osd.2 (osd.2) 409 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.394677+0000 osd.2 (osd.2) 409 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.396485+0000 osd.2 (osd.2) 410 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.396485+0000 osd.2 (osd.2) 410 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.491657+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.491657+0000 mon.a (mon.0) 3933 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: audit 2024-04-26T16:37:47.502497+0000 mon.c (mon.2) 904 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: audit 2024-04-26T16:37:47.502497+0000 mon.c (mon.2) 904 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: audit 2024-04-26T16:37:47.502882+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: audit 2024-04-26T16:37:47.502882+0000 mon.a (mon.0) 3934 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.746818+0000 osd.3 (osd.3) 481 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.746818+0000 osd.3 (osd.3) 481 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:37:49.124 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.750258+0000 osd.3 (osd.3) 482 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:37:49.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.750258+0000 osd.3 (osd.3) 482 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:37:49.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.844649+0000 osd.4 (osd.4) 417 : cluster [DBG] 5.e scrub starts 2024-04-26T16:37:49.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.844649+0000 osd.4 (osd.4) 417 : cluster [DBG] 5.e scrub starts 2024-04-26T16:37:49.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.846481+0000 osd.4 (osd.4) 418 : cluster [DBG] 5.e scrub ok 2024-04-26T16:37:49.125 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:48 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.846481+0000 osd.4 (osd.4) 418 : cluster [DBG] 5.e scrub ok 2024-04-26T16:37:49.136 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2 2024-04-26T16:37:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.2' 2024-04-26T16:37:49.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-04-26T16:37:49.240 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:49.261 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0 2024-04-26T16:37:49.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.0' 2024-04-26T16:37:49.364 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:49.384 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2336862976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 msgr2=0x7f8abc103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2336862976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc103940 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009990 tx=0x7f8aac009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2336862976 shutdown_connections 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2336862976 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8abc109560 0x7f8abc109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2336862976 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 0x7f8abc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2336862976 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc103940 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2336862976 >> 172.21.15.50:0/2336862976 conn(0x7f8abc07a5d0 msgr2=0x7f8abc07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2336862976 shutdown_connections 2024-04-26T16:37:49.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2336862976 wait complete. 2024-04-26T16:37:49.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 librados: starting msgr at 2024-04-26T16:37:49.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 librados: starting objecter 2024-04-26T16:37:49.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 Processor -- start 2024-04-26T16:37:49.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- start start 2024-04-26T16:37:49.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 librados: setting wanted keys 2024-04-26T16:37:49.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 librados: calling monclient init 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 0x7f8abc075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8abc109560 0x7f8abc0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc112b90 con 0x7f8abc103e80 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc112a10 con 0x7f8abc109560 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc112d10 con 0x7f8abc1034c0 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1d71700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1d71700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46138/0 (socket says 172.21.15.50:46138) 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1d71700 1 -- 172.21.15.50:0/2947404713 learned_addr learned my addr 172.21.15.50:0/2947404713 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 0x7f8abc075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8abc109560 0x7f8abc0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 msgr2=0x7f8abc075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8abc109560 msgr2=0x7f8abc0761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8abc109560 0x7f8abc0761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc1a76e0 con 0x7f8abc103e80 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8abc1034c0 0x7f8abc075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 0x7f8abc075c60 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f8aac000c00 tx=0x7f8aac004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ab2ffd700 10 client.?.objecter ms_handle_connect 0x7f8abc103e80 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ab2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac026070 con 0x7f8abc103e80 2024-04-26T16:37:49.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8abc1a7970 con 0x7f8abc103e80 2024-04-26T16:37:49.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8aac021410 con 0x7f8abc103e80 2024-04-26T16:37:49.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac0173f0 con 0x7f8abc103e80 2024-04-26T16:37:49.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 10 client.31101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:49.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8abc1a7dc0 con 0x7f8abc103e80 2024-04-26T16:37:49.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.415+0000 7f8ac3fd5700 1 librados: init done 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd ls"} v 0) -- 0x7f8abc104bd0 con 0x7f8abc103e80 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8aac029070 con 0x7f8abc103e80 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8aa8071f80 0x7f8aa8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(868..868 src has 577..868) ==== 8817+0+0 (secure 0 0 0) 0x7f8aac092c60 con 0x7f8abc103e80 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 osd_map(868..868 src has 577..868) 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 3 client.31101.objecter handle_osd_map got epochs [868,868] > 0 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 3 client.31101.objecter handle_osd_map decoding full epoch 868 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8aa8071f80 0x7f8aa8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 20 client.31101.objecter dump_active .. 0 homeless 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "osd ls"}]=0 v868) ==== 54+0+16 (secure 0 0 0) 0x7f8aac013070 con 0x7f8abc103e80 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8aa8071f80 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8aa8071f80 0x7f8aa8074440 secure :-1 s=READY pgs=3845 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8011550 tx=0x7f8ab8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.422 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 1 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f8abc064030 0x7f8abc064410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 1 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 0 1 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 4 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 1 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8abc076c50 con 0x7f8abc064030 2024-04-26T16:37:49.423 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f8abc064030 0x7f8abc064410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc064030 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f8abc064030 0x7f8abc064410 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc076c50 con 0x7f8abc064030 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 1: 0 ) 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 1 = system:0 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 4 2024-04-26T16:37:49.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.419+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 0 1 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 2 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 2 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 2 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc06a110 osd=0 3 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 0 2 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 4 2024-04-26T16:37:49.440 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 2 2024-04-26T16:37:49.441 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc063ab0 con 0x7f8abc064030 2024-04-26T16:37:49.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.439+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc063ab0 con 0x7f8abc064030 2024-04-26T16:37:49.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 2: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 2 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 4 2024-04-26T16:37:49.442 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 0 2 2024-04-26T16:37:49.443 INFO:teuthology.orchestra.run.smithi050.stdout:osd.0: {} 2024-04-26T16:37:49.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 3 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 3 2024-04-26T16:37:49.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f8abc10c080 0x7f8abc1ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.443 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 3 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 1 3 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 4 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 3 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 3: {"prefix": "get_command_descriptions"}) -- 0x7f8abc02e600 con 0x7f8abc10c080 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f8abc10c080 0x7f8abc1ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f8abc10c080 0x7f8abc1ac0a0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc10c080 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 3: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc02e600 con 0x7f8abc10c080 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 3: 0 ) 2024-04-26T16:37:49.444 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 3 = system:0 2024-04-26T16:37:49.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 4 2024-04-26T16:37:49.445 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.443+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 1 3 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 4 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 4 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 4 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc02e490 osd=1 3 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 1 4 2024-04-26T16:37:49.461 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 4 2024-04-26T16:37:49.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 4 2024-04-26T16:37:49.462 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.459+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 4: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1ac5e0 con 0x7f8abc10c080 2024-04-26T16:37:49.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 4: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1ac5e0 con 0x7f8abc10c080 2024-04-26T16:37:49.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 4: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 4 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 4 2024-04-26T16:37:49.463 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 1 4 2024-04-26T16:37:49.463 INFO:teuthology.orchestra.run.smithi050.stdout:osd.1: {} 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 5 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 5 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f8abc10b760 0x7f8abc1af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.464 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 5 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 2 5 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 4 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 5 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 5: {"prefix": "get_command_descriptions"}) -- 0x7f8abc02dff0 con 0x7f8abc10b760 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f8abc10b760 0x7f8abc1af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc10b760 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f8abc10b760 0x7f8abc1af920 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 5: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc02dff0 con 0x7f8abc10b760 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 5: 0 ) 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 5 = system:0 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 4 2024-04-26T16:37:49.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.463+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 2 5 2024-04-26T16:37:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 6 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 6 2024-04-26T16:37:49.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 6 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc10b5f0 osd=2 3 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 2 6 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 4 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 6 2024-04-26T16:37:49.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.479+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 6: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1afe60 con 0x7f8abc10b760 2024-04-26T16:37:49.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 6: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1afe60 con 0x7f8abc10b760 2024-04-26T16:37:49.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 6: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 6 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 4 2024-04-26T16:37:49.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 2 6 2024-04-26T16:37:49.484 INFO:teuthology.orchestra.run.smithi050.stdout:osd.2: {} 2024-04-26T16:37:49.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 7 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 7 2024-04-26T16:37:49.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8abc1acf60 0x7f8abc1ad380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.484 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 7 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 3 7 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 4 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 7 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 7: {"prefix": "get_command_descriptions"}) -- 0x7f8abc1b1320 con 0x7f8abc1acf60 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8abc1acf60 0x7f8abc1ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8abc1acf60 0x7f8abc1ad380 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc1acf60 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 7: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc1b1320 con 0x7f8abc1acf60 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 7: 0 ) 2024-04-26T16:37:49.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 7 = system:0 2024-04-26T16:37:49.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 4 2024-04-26T16:37:49.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.483+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 3 7 2024-04-26T16:37:49.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 8 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 8 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 8 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1afe60 osd=3 3 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 3 8 2024-04-26T16:37:49.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 4 2024-04-26T16:37:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 8 2024-04-26T16:37:49.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.503+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 8: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1b26b0 con 0x7f8abc1acf60 2024-04-26T16:37:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 8: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1b26b0 con 0x7f8abc1acf60 2024-04-26T16:37:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 8: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 8 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 4 2024-04-26T16:37:49.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 3 8 2024-04-26T16:37:49.506 INFO:teuthology.orchestra.run.smithi050.stdout:osd.3: {} 2024-04-26T16:37:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 9 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 9 2024-04-26T16:37:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f8abc1b0e10 0x7f8abc1b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 9 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 4 9 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 4 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 9 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 9: {"prefix": "get_command_descriptions"}) -- 0x7f8abc1acc30 con 0x7f8abc1b0e10 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f8abc1b0e10 0x7f8abc1b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc1b0e10 2024-04-26T16:37:49.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f8abc1b0e10 0x7f8abc1b6a30 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 9: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc1acc30 con 0x7f8abc1b0e10 2024-04-26T16:37:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 9: 0 ) 2024-04-26T16:37:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 9 = system:0 2024-04-26T16:37:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 4 2024-04-26T16:37:49.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.507+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 4 9 2024-04-26T16:37:49.512 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6 2024-04-26T16:37:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.6' 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 10 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 10 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 10 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1b26b0 osd=4 3 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 4 10 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 4 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 10 2024-04-26T16:37:49.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 10: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1acc30 con 0x7f8abc1b0e10 2024-04-26T16:37:49.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 10: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1acc30 con 0x7f8abc1b0e10 2024-04-26T16:37:49.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 10: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 10 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 4 2024-04-26T16:37:49.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 4 10 2024-04-26T16:37:49.529 INFO:teuthology.orchestra.run.smithi050.stdout:osd.4: {} 2024-04-26T16:37:49.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 11 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 11 2024-04-26T16:37:49.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f8abc1b7480 0x7f8abc1b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 11 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 5 11 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 4 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 11 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 11: {"prefix": "get_command_descriptions"}) -- 0x7f8abc1bc780 con 0x7f8abc1b7480 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.527+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f8abc1b7480 0x7f8abc1b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ac1570700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f8abc1b7480 0x7f8abc1b9870 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc1b7480 2024-04-26T16:37:49.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 11: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc1bc780 con 0x7f8abc1b7480 2024-04-26T16:37:49.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 11: 0 ) 2024-04-26T16:37:49.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 11 = system:0 2024-04-26T16:37:49.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 4 2024-04-26T16:37:49.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.531+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 5 11 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8023 ms) 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (11997 ms) 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8035 ms) 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12975 ms) 2024-04-26T16:37:49.546 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7995 ms) 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8106 ms) 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: hmm, no HitSet yet 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: ok, hit_set contains 337:602f83fe:::foo:head 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8967 ms) 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: first is 1714149386 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,1714149391,1714149392,0 2024-04-26T16:37:49.547 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,1714149391,1714149392,0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,1714149391,1714149392,0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,1714149391,1714149392,1714149394,1714149395,0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,1714149391,1714149392,1714149394,1714149395,0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149386,1714149388,1714149389,1714149391,1714149392,1714149394,1714149395,0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: got ls 1714149389,1714149391,1714149392,1714149394,1714149395,1714149397,1714149398,0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: first now 1714149389, trimmed 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21159 ms) 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: foo0 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (13817 ms) 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-04-26T16:37:49.548 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17433 ms) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (21568 ms) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5062 ms) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3052 ms) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (2882 ms) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3070 ms) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (248124 ms total) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (861502 ms total) 2024-04-26T16:37:49.549 INFO:tasks.workunit.client.0.smithi050.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73220 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73220 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73418 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73418 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73781 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73781 2024-04-26T16:37:49.559 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73606 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73606 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73479 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73479 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73308 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73308 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73568 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73568 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73980 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73980 2024-04-26T16:37:49.560 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73993 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73993 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73340 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73340 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73824 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73824 2024-04-26T16:37:49.561 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73694 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73694 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73204 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73204 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73236 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73236 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73740 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73740 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73196 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73196 2024-04-26T16:37:49.562 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73212 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73212 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74086 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74086 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73426 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73426 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74143 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74143 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73527 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73527 2024-04-26T16:37:49.563 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73893 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73893 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74210 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74210 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73441 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73441 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73758 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73758 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74078 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74078 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73830 2024-04-26T16:37:49.564 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73830 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74114 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74114 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73683 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73683 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73400 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73400 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74192 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74192 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73618 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73618 2024-04-26T16:37:49.565 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73188 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73188 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73228 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73228 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73580 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73580 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73244 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73244 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73644 2024-04-26T16:37:49.566 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73644 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=73661 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 73661 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74033 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74033 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ for t in "${!pids[@]}" 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ pid=74225 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ wait 74225 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ exit 0 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ cleanup 2024-04-26T16:37:49.567 INFO:tasks.workunit.client.0.smithi050.stderr:+ pkill -P 73182 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 12 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 12 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 12 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1acc30 osd=5 3 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 5 12 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 4 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 12 2024-04-26T16:37:49.568 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.551+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 12: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1b9db0 con 0x7f8abc1b7480 2024-04-26T16:37:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 12: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1b9db0 con 0x7f8abc1b7480 2024-04-26T16:37:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 12: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 12 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 4 2024-04-26T16:37:49.569 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 5 12 2024-04-26T16:37:49.569 INFO:teuthology.orchestra.run.smithi050.stdout:osd.5: {} 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 13 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 13 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f8abc1ba4d0 0x7f8abc1ba930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 13 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 6 13 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 4 2024-04-26T16:37:49.570 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 13 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 13: {"prefix": "get_command_descriptions"}) -- 0x7f8abc1bfba0 con 0x7f8abc1ba4d0 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f8abc1ba4d0 0x7f8abc1ba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc1ba4d0 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ac2572700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f8abc1ba4d0 0x7f8abc1ba930 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 13: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc1bfba0 con 0x7f8abc1ba4d0 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 13: 0 ) 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 13 = system:0 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 4 2024-04-26T16:37:49.571 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.555+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 6 13 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 14 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 14 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 14 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1b9db0 osd=6 3 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 6 14 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 4 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 14 2024-04-26T16:37:49.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.571+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 14: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1bfba0 con 0x7f8abc1ba4d0 2024-04-26T16:37:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 14: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1bfba0 con 0x7f8abc1ba4d0 2024-04-26T16:37:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 14: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 14 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 4 2024-04-26T16:37:49.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 6 14 2024-04-26T16:37:49.576 INFO:teuthology.orchestra.run.smithi050.stdout:osd.6: {} 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 15 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 15 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f8abc1beee0 0x7f8abc1bf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 15 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.577 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 7 15 2024-04-26T16:37:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 4 2024-04-26T16:37:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 15 2024-04-26T16:37:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 15: {"prefix": "get_command_descriptions"}) -- 0x7f8abc1c2f90 con 0x7f8abc1beee0 2024-04-26T16:37:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f8abc1beee0 0x7f8abc1bf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ac1d71700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f8abc1beee0 0x7f8abc1bf340 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:49.578 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.575+0000 7f8ab2ffd700 10 client.31101.objecter ms_handle_connect 0x7f8abc1beee0 2024-04-26T16:37:49.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.579+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 15: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8abc1c2f90 con 0x7f8abc1beee0 2024-04-26T16:37:49.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.579+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 15: 0 ) 2024-04-26T16:37:49.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.579+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 15 = system:0 2024-04-26T16:37:49.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.579+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 4 2024-04-26T16:37:49.579 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.579+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 7 15 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 10 client.31101.objecter _submit_command 16 [{"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}] 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign -1 16 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 20 client.31101.objecter _get_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_remove -1 16 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 20 client.31101.objecter get_session s=0x7f8abc1bfba0 osd=7 3 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 15 client.31101.objecter _session_command_op_assign 7 16 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 4 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 10 client.31101.objecter _send_command 16 2024-04-26T16:37:49.600 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.599+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 16: {"prefix": "injectargs", "injected_args": ["--osd_enable_op_tracker=true"]}) -- 0x7f8abc1baec0 con 0x7f8abc1beee0 2024-04-26T16:37:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ab2ffd700 1 -- 172.21.15.50:0/2947404713 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 16: 0 osd_enable_op_tracker = '' ) ==== 35+0+3 (crc 0 0 0) 0x7f8abc1baec0 con 0x7f8abc1beee0 2024-04-26T16:37:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ab2ffd700 10 client.31101.objecter ms_dispatch 0x7f8abc001550 command_reply(tid 16: 0 osd_enable_op_tracker = '' ) 2024-04-26T16:37:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ab2ffd700 10 client.31101.objecter _finish_command 16 = system:0 osd_enable_op_tracker = '' 2024-04-26T16:37:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ab2ffd700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 4 2024-04-26T16:37:49.606 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ab2ffd700 15 client.31101.objecter _session_command_op_remove 7 16 2024-04-26T16:37:49.606 INFO:teuthology.orchestra.run.smithi050.stdout:osd.7: {} 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 librados: watch_flush enter 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 librados: watch_flush exit 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.0 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f8abc064030 msgr2=0x7f8abc064410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f8abc064030 0x7f8abc064410 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc06a110 osd=0 1 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.1 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f8abc10c080 msgr2=0x7f8abc1ac0a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f8abc10c080 0x7f8abc1ac0a0 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc02e490 osd=1 1 2024-04-26T16:37:49.608 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.2 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f8abc10b760 msgr2=0x7f8abc1af920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7f8abc10b760 0x7f8abc1af920 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc10b5f0 osd=2 1 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.3 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8abc1acf60 msgr2=0x7f8abc1ad380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac2572700 1 -- 172.21.15.50:0/2947404713 reap_dead start 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8abc1acf60 0x7f8abc1ad380 crc :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1afe60 osd=3 1 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.4 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f8abc1b0e10 msgr2=0x7f8abc1b6a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f8abc1b0e10 0x7f8abc1b6a30 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b26b0 osd=4 1 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.5 2024-04-26T16:37:49.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f8abc1b7480 msgr2=0x7f8abc1b9870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f8abc1b7480 0x7f8abc1b9870 crc :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1acc30 osd=5 1 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.6 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f8abc1ba4d0 msgr2=0x7f8abc1ba930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7f8abc1ba4d0 0x7f8abc1ba930 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1b9db0 osd=6 1 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter close_session for osd.7 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f8abc1beee0 msgr2=0x7f8abc1bf340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f8abc1beee0 0x7f8abc1bf340 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter put_session s=0x7f8abc1bfba0 osd=7 1 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 20 client.31101.objecter shutdown clearing up homeless session... 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 10 client.31101.objecter successfully canceled tick 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac2572700 1 -- 172.21.15.50:0/2947404713 reap_dead start 2024-04-26T16:37:49.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8aa8071f80 msgr2=0x7f8aa8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8aa8071f80 0x7f8aa8074440 secure :-1 s=READY pgs=3845 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8011550 tx=0x7f8ab8009260 comp rx=0 tx=0).stop 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 msgr2=0x7f8abc075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 0x7f8abc075c60 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f8aac000c00 tx=0x7f8aac004790 comp rx=0 tx=0).stop 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 shutdown_connections 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8abc103e80 0x7f8abc075c60 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 --2- 172.21.15.50:0/2947404713 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8aa8071f80 0x7f8aa8074440 unknown :-1 s=CLOSED pgs=3845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 >> 172.21.15.50:0/2947404713 conn(0x7f8abc07a5d0 msgr2=0x7f8abc101ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 shutdown_connections 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 -- 172.21.15.50:0/2947404713 wait complete. 2024-04-26T16:37:49.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:49.607+0000 7f8ac3fd5700 1 librados: shutdown 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.0: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.1: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.2: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.3: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.4: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.5: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.6: osd_enable_op_tracker = '' 2024-04-26T16:37:49.612 INFO:teuthology.orchestra.run.smithi050.stderr:osd.7: osd_enable_op_tracker = '' 2024-04-26T16:37:49.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.962619+0000 mgr.y (mgr.24401) 780 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 583 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:47.962619+0000 mgr.y (mgr.24401) 780 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 583 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.093404+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.093404+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.094878+0000 osd.5 (osd.5) 323 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.094878+0000 osd.5 (osd.5) 323 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.273683+0000 osd.1 (osd.1) 441 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.273683+0000 osd.1 (osd.1) 441 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.275776+0000 osd.1 (osd.1) 442 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.275776+0000 osd.1 (osd.1) 442 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:37:49.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.337536+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.337536+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.340215+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.340215+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.488356+0000 mon.a (mon.0) 3935 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.488356+0000 mon.a (mon.0) 3935 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:48.497961+0000 mon.a (mon.0) 3936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:48.497961+0000 mon.a (mon.0) 3936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.510649+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.510649+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:48.512885+0000 mon.c (mon.2) 905 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:48.512885+0000 mon.c (mon.2) 905 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:48.513511+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.962619+0000 mgr.y (mgr.24401) 780 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 583 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:47.962619+0000 mgr.y (mgr.24401) 780 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 583 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.093404+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.093404+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.094878+0000 osd.5 (osd.5) 323 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.094878+0000 osd.5 (osd.5) 323 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.273683+0000 osd.1 (osd.1) 441 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.273683+0000 osd.1 (osd.1) 441 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.275776+0000 osd.1 (osd.1) 442 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.275776+0000 osd.1 (osd.1) 442 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.337536+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.337536+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.340215+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:37:49.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.340215+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.488356+0000 mon.a (mon.0) 3935 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.488356+0000 mon.a (mon.0) 3935 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:48.497961+0000 mon.a (mon.0) 3936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:48.497961+0000 mon.a (mon.0) 3936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.510649+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.510649+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:48.512885+0000 mon.c (mon.2) 905 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:48.512885+0000 mon.c (mon.2) 905 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:48.513511+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:48.513511+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.761150+0000 osd.3 (osd.3) 483 : cluster [DBG] 6.b scrub starts 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.761150+0000 osd.3 (osd.3) 483 : cluster [DBG] 6.b scrub starts 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.765854+0000 osd.3 (osd.3) 484 : cluster [DBG] 6.b scrub ok 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: cluster 2024-04-26T16:37:48.765854+0000 osd.3 (osd.3) 484 : cluster [DBG] 6.b scrub ok 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:49.423651+0000 mon.a (mon.0) 3939 : audit [DBG] from='client.? 172.21.15.50:0/2947404713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[21463]: audit 2024-04-26T16:37:49.423651+0000 mon.a (mon.0) 3939 : audit [DBG] from='client.? 172.21.15.50:0/2947404713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:48.513511+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.761150+0000 osd.3 (osd.3) 483 : cluster [DBG] 6.b scrub starts 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.761150+0000 osd.3 (osd.3) 483 : cluster [DBG] 6.b scrub starts 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.765854+0000 osd.3 (osd.3) 484 : cluster [DBG] 6.b scrub ok 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: cluster 2024-04-26T16:37:48.765854+0000 osd.3 (osd.3) 484 : cluster [DBG] 6.b scrub ok 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:49.423651+0000 mon.a (mon.0) 3939 : audit [DBG] from='client.? 172.21.15.50:0/2947404713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:49.634 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:49 smithi050 bash[29234]: audit 2024-04-26T16:37:49.423651+0000 mon.a (mon.0) 3939 : audit [DBG] from='client.? 172.21.15.50:0/2947404713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:49.641 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5 2024-04-26T16:37:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.5' 2024-04-26T16:37:49.731 INFO:tasks.workunit.client.0.smithi050.stderr:+ true 2024-04-26T16:37:49.732 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-26T16:37:49.733 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-26T16:37:49.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.962619+0000 mgr.y (mgr.24401) 780 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 583 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:49.755 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:47.962619+0000 mgr.y (mgr.24401) 780 : cluster [DBG] pgmap v1351: 164 pgs: 164 active+clean; 583 KiB data, 798 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.093404+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.093404+0000 osd.5 (osd.5) 322 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.094878+0000 osd.5 (osd.5) 323 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.094878+0000 osd.5 (osd.5) 323 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.273683+0000 osd.1 (osd.1) 441 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.273683+0000 osd.1 (osd.1) 441 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.275776+0000 osd.1 (osd.1) 442 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.275776+0000 osd.1 (osd.1) 442 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.337536+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.337536+0000 osd.6 (osd.6) 37 : cluster [DBG] 3.a scrub starts 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.340215+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.340215+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2024-04-26T16:37:49.756 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.488356+0000 mon.a (mon.0) 3935 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.488356+0000 mon.a (mon.0) 3935 : cluster [WRN] Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:48.497961+0000 mon.a (mon.0) 3936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:48.497961+0000 mon.a (mon.0) 3936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.510649+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.510649+0000 mon.a (mon.0) 3937 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:48.512885+0000 mon.c (mon.2) 905 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:48.512885+0000 mon.c (mon.2) 905 : audit [INF] from='client.? 172.21.15.50:0/2980882740' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:48.513511+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:48.513511+0000 mon.a (mon.0) 3938 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]: dispatch 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.761150+0000 osd.3 (osd.3) 483 : cluster [DBG] 6.b scrub starts 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.761150+0000 osd.3 (osd.3) 483 : cluster [DBG] 6.b scrub starts 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.765854+0000 osd.3 (osd.3) 484 : cluster [DBG] 6.b scrub ok 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: cluster 2024-04-26T16:37:48.765854+0000 osd.3 (osd.3) 484 : cluster [DBG] 6.b scrub ok 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:49.423651+0000 mon.a (mon.0) 3939 : audit [DBG] from='client.? 172.21.15.50:0/2947404713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:49.757 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:49 smithi190 bash[23868]: audit 2024-04-26T16:37:49.423651+0000 mon.a (mon.0) 3939 : audit [DBG] from='client.? 172.21.15.50:0/2947404713' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-04-26T16:37:49.769 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7 2024-04-26T16:37:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.7' 2024-04-26T16:37:49.795 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-04-26T16:37:49.796 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-26T16:37:49.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:49.900 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:49.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:50.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:50.030 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:50.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:50.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:50.145 DEBUG:teuthology.parallel:result is None 2024-04-26T16:37:50.145 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T16:37:50.157 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T16:37:50.157 DEBUG:teuthology.orchestra.run.smithi050:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-26T16:37:50.160 INFO:teuthology.orchestra.run.smithi190.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4 2024-04-26T16:37:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:50.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.4' 2024-04-26T16:37:50.263 DEBUG:teuthology.orchestra.run.smithi050:> sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:50.267 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-26T16:37:50.267 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-04-26T16:37:50.276 INFO:tasks.thrashosds:joining thrashosds 2024-04-26T16:37:50.289 INFO:teuthology.orchestra.run.smithi050.stderr:Error response from daemon: Cannot kill container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3 2024-04-26T16:37:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:37:50.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi050 with status 1: 'sudo docker kill -s 1 ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd.3' 2024-04-26T16:37:50.509 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: audit 2024-04-26T16:37:49.499867+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: audit 2024-04-26T16:37:49.499867+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: audit 2024-04-26T16:37:49.499867+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.503167+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.503167+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.745976+0000 osd.3 (osd.3) 485 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.745976+0000 osd.3 (osd.3) 485 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.748196+0000 osd.3 (osd.3) 486 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.748196+0000 osd.3 (osd.3) 486 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:37:50.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.780972+0000 osd.4 (osd.4) 419 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:37:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.780972+0000 osd.4 (osd.4) 419 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:37:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.783100+0000 osd.4 (osd.4) 420 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:37:50.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:50 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.783100+0000 osd.4 (osd.4) 420 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:37:50.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: audit 2024-04-26T16:37:49.499867+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.503167+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.503167+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.745976+0000 osd.3 (osd.3) 485 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.745976+0000 osd.3 (osd.3) 485 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.748196+0000 osd.3 (osd.3) 486 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.748196+0000 osd.3 (osd.3) 486 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.780972+0000 osd.4 (osd.4) 419 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.780972+0000 osd.4 (osd.4) 419 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.783100+0000 osd.4 (osd.4) 420 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:37:50.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.783100+0000 osd.4 (osd.4) 420 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: audit 2024-04-26T16:37:49.499867+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: audit 2024-04-26T16:37:49.499867+0000 mon.a (mon.0) 3940 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi050-73372-111"}]': finished 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.503167+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.503167+0000 mon.a (mon.0) 3941 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.745976+0000 osd.3 (osd.3) 485 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.745976+0000 osd.3 (osd.3) 485 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.748196+0000 osd.3 (osd.3) 486 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.748196+0000 osd.3 (osd.3) 486 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.780972+0000 osd.4 (osd.4) 419 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:37:50.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.780972+0000 osd.4 (osd.4) 419 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:37:50.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.783100+0000 osd.4 (osd.4) 420 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:37:50.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:50 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.783100+0000 osd.4 (osd.4) 420 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:37:51.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:51 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.963330+0000 mgr.y (mgr.24401) 781 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 583 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:51.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:51 smithi190 bash[23868]: cluster 2024-04-26T16:37:49.963330+0000 mgr.y (mgr.24401) 781 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 583 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:51 smithi190 bash[23868]: cluster 2024-04-26T16:37:50.329563+0000 osd.6 (osd.6) 39 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:37:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:51 smithi190 bash[23868]: cluster 2024-04-26T16:37:50.329563+0000 osd.6 (osd.6) 39 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:37:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:51 smithi190 bash[23868]: cluster 2024-04-26T16:37:50.333983+0000 osd.6 (osd.6) 40 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:37:51.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:51 smithi190 bash[23868]: cluster 2024-04-26T16:37:50.333983+0000 osd.6 (osd.6) 40 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.963330+0000 mgr.y (mgr.24401) 781 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 583 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[21463]: cluster 2024-04-26T16:37:49.963330+0000 mgr.y (mgr.24401) 781 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 583 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[21463]: cluster 2024-04-26T16:37:50.329563+0000 osd.6 (osd.6) 39 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[21463]: cluster 2024-04-26T16:37:50.329563+0000 osd.6 (osd.6) 39 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[21463]: cluster 2024-04-26T16:37:50.333983+0000 osd.6 (osd.6) 40 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[21463]: cluster 2024-04-26T16:37:50.333983+0000 osd.6 (osd.6) 40 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.963330+0000 mgr.y (mgr.24401) 781 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 583 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:51.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[29234]: cluster 2024-04-26T16:37:49.963330+0000 mgr.y (mgr.24401) 781 : cluster [DBG] pgmap v1354: 164 pgs: 164 active+clean; 583 KiB data, 796 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:51.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[29234]: cluster 2024-04-26T16:37:50.329563+0000 osd.6 (osd.6) 39 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:37:51.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[29234]: cluster 2024-04-26T16:37:50.329563+0000 osd.6 (osd.6) 39 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:37:51.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[29234]: cluster 2024-04-26T16:37:50.333983+0000 osd.6 (osd.6) 40 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:37:51.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:51 smithi050 bash[29234]: cluster 2024-04-26T16:37:50.333983+0000 osd.6 (osd.6) 40 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:37:52.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.282513+0000 osd.6 (osd.6) 41 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:37:52.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.282513+0000 osd.6 (osd.6) 41 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.284838+0000 osd.6 (osd.6) 42 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.284838+0000 osd.6 (osd.6) 42 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.419945+0000 osd.2 (osd.2) 411 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.419945+0000 osd.2 (osd.2) 411 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.423224+0000 osd.2 (osd.2) 412 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.423224+0000 osd.2 (osd.2) 412 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.746898+0000 osd.3 (osd.3) 487 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.746898+0000 osd.3 (osd.3) 487 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.748490+0000 osd.3 (osd.3) 488 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.748490+0000 osd.3 (osd.3) 488 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.815998+0000 osd.4 (osd.4) 421 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.815998+0000 osd.4 (osd.4) 421 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:37:52.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.818309+0000 osd.4 (osd.4) 422 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.818309+0000 osd.4 (osd.4) 422 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.025370+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.025370+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.025750+0000 osd.6 (osd.6) 44 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.025750+0000 osd.6 (osd.6) 44 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.282513+0000 osd.6 (osd.6) 41 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.282513+0000 osd.6 (osd.6) 41 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.284838+0000 osd.6 (osd.6) 42 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.284838+0000 osd.6 (osd.6) 42 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.419945+0000 osd.2 (osd.2) 411 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.419945+0000 osd.2 (osd.2) 411 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:37:52.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.423224+0000 osd.2 (osd.2) 412 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.423224+0000 osd.2 (osd.2) 412 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.746898+0000 osd.3 (osd.3) 487 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.746898+0000 osd.3 (osd.3) 487 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.748490+0000 osd.3 (osd.3) 488 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.748490+0000 osd.3 (osd.3) 488 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.815998+0000 osd.4 (osd.4) 421 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.815998+0000 osd.4 (osd.4) 421 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.818309+0000 osd.4 (osd.4) 422 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.818309+0000 osd.4 (osd.4) 422 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.025370+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.025370+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.025750+0000 osd.6 (osd.6) 44 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:52.633 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:52 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.025750+0000 osd.6 (osd.6) 44 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.282513+0000 osd.6 (osd.6) 41 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.282513+0000 osd.6 (osd.6) 41 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.284838+0000 osd.6 (osd.6) 42 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.284838+0000 osd.6 (osd.6) 42 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.419945+0000 osd.2 (osd.2) 411 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.419945+0000 osd.2 (osd.2) 411 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.423224+0000 osd.2 (osd.2) 412 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.423224+0000 osd.2 (osd.2) 412 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:37:52.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.746898+0000 osd.3 (osd.3) 487 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.746898+0000 osd.3 (osd.3) 487 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.748490+0000 osd.3 (osd.3) 488 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.748490+0000 osd.3 (osd.3) 488 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.815998+0000 osd.4 (osd.4) 421 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.815998+0000 osd.4 (osd.4) 421 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.818309+0000 osd.4 (osd.4) 422 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.818309+0000 osd.4 (osd.4) 422 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.025370+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.025370+0000 osd.6 (osd.6) 43 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.025750+0000 osd.6 (osd.6) 44 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:52.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:52 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.025750+0000 osd.6 (osd.6) 44 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:37:53.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.964466+0000 mgr.y (mgr.24401) 782 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:53.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:51.964466+0000 mgr.y (mgr.24401) 782 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:53.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.060833+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:37:53.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.060833+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:37:53.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.063399+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:37:53.573 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.063399+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.544193+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.544193+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.809552+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.809552+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.815022+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.815022+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.881207+0000 osd.0 (osd.0) 475 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.881207+0000 osd.0 (osd.0) 475 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.886304+0000 osd.0 (osd.0) 476 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[29234]: cluster 2024-04-26T16:37:52.886304+0000 osd.0 (osd.0) 476 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:37:53.574 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.964466+0000 mgr.y (mgr.24401) 782 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:51.964466+0000 mgr.y (mgr.24401) 782 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.060833+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.060833+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.063399+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.063399+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.544193+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.544193+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.809552+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.809552+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.815022+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.815022+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.881207+0000 osd.0 (osd.0) 475 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.881207+0000 osd.0 (osd.0) 475 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.886304+0000 osd.0 (osd.0) 476 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:37:53.575 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:53 smithi050 bash[21463]: cluster 2024-04-26T16:37:52.886304+0000 osd.0 (osd.0) 476 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.964466+0000 mgr.y (mgr.24401) 782 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:51.964466+0000 mgr.y (mgr.24401) 782 : cluster [DBG] pgmap v1355: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.060833+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.060833+0000 osd.5 (osd.5) 324 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.063399+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.063399+0000 osd.5 (osd.5) 325 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.544193+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.544193+0000 mon.a (mon.0) 3942 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.809552+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:37:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.809552+0000 osd.4 (osd.4) 423 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:37:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.815022+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:37:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.815022+0000 osd.4 (osd.4) 424 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:37:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.881207+0000 osd.0 (osd.0) 475 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:37:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.881207+0000 osd.0 (osd.0) 475 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:37:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.886304+0000 osd.0 (osd.0) 476 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:37:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:53 smithi190 bash[23868]: cluster 2024-04-26T16:37:52.886304+0000 osd.0 (osd.0) 476 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:37:54.015 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:37:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:54 smithi190 bash[23868]: cluster 2024-04-26T16:37:53.918015+0000 osd.0 (osd.0) 477 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:37:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:54 smithi190 bash[23868]: cluster 2024-04-26T16:37:53.918015+0000 osd.0 (osd.0) 477 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:37:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:54 smithi190 bash[23868]: cluster 2024-04-26T16:37:53.920523+0000 osd.0 (osd.0) 478 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:37:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:54 smithi190 bash[23868]: cluster 2024-04-26T16:37:53.920523+0000 osd.0 (osd.0) 478 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:37:54.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:37:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:37:54.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[21463]: cluster 2024-04-26T16:37:53.918015+0000 osd.0 (osd.0) 477 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:37:54.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[21463]: cluster 2024-04-26T16:37:53.918015+0000 osd.0 (osd.0) 477 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:37:54.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[21463]: cluster 2024-04-26T16:37:53.920523+0000 osd.0 (osd.0) 478 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:37:54.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[21463]: cluster 2024-04-26T16:37:53.920523+0000 osd.0 (osd.0) 478 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:37:54.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[29234]: cluster 2024-04-26T16:37:53.918015+0000 osd.0 (osd.0) 477 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:37:54.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[29234]: cluster 2024-04-26T16:37:53.918015+0000 osd.0 (osd.0) 477 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:37:54.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[29234]: cluster 2024-04-26T16:37:53.920523+0000 osd.0 (osd.0) 478 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:37:54.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:54 smithi050 bash[29234]: cluster 2024-04-26T16:37:53.920523+0000 osd.0 (osd.0) 478 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:37:55.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:53.965316+0000 mgr.y (mgr.24401) 783 : cluster [DBG] pgmap v1357: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:53.965316+0000 mgr.y (mgr.24401) 783 : cluster [DBG] pgmap v1357: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.192831+0000 osd.1 (osd.1) 443 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.192831+0000 osd.1 (osd.1) 443 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.196206+0000 osd.1 (osd.1) 444 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.196206+0000 osd.1 (osd.1) 444 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.769799+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.c scrub starts 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.769799+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.c scrub starts 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.772039+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.c scrub ok 2024-04-26T16:37:55.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:55 smithi190 bash[23868]: cluster 2024-04-26T16:37:54.772039+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.c scrub ok 2024-04-26T16:37:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- 172.21.15.50:0/860823663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4106020 msgr2=0x7f1db4112530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:55.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- 172.21.15.50:0/860823663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4106020 0x7f1db4112530 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00b0a0 tx=0x7f1dac00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- 172.21.15.50:0/860823663 shutdown_connections 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- 172.21.15.50:0/860823663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4106020 0x7f1db4112530 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- 172.21.15.50:0/860823663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1db410b660 0x7f1db410ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- 172.21.15.50:0/860823663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db4105660 0x7f1db4105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- 172.21.15.50:0/860823663 >> 172.21.15.50:0/860823663 conn(0x7f1db4101150 msgr2=0x7f1db4103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- 172.21.15.50:0/860823663 shutdown_connections 2024-04-26T16:37:55.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- 172.21.15.50:0/860823663 wait complete. 2024-04-26T16:37:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 librados: starting msgr at 2024-04-26T16:37:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 librados: starting objecter 2024-04-26T16:37:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 Processor -- start 2024-04-26T16:37:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- start start 2024-04-26T16:37:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 librados: setting wanted keys 2024-04-26T16:37:55.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 librados: calling monclient init 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 0x7f1db419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1db4106020 0x7f1db419b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db410b660 0x7f1db419fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1db4114d80 con 0x7f1db4106020 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1db4114c00 con 0x7f1db410b660 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1db4114f00 con 0x7f1db4105660 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 0x7f1db419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 0x7f1db419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46156/0 (socket says 172.21.15.50:46156) 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 -- 172.21.15.50:0/4080527895 learned_addr learned my addr 172.21.15.50:0/4080527895 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db1d9b700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1db4106020 0x7f1db419b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db2d9d700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db410b660 0x7f1db419fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 -- 172.21.15.50:0/4080527895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db410b660 msgr2=0x7f1db419fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:55.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db410b660 0x7f1db419fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 -- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1db4106020 msgr2=0x7f1db419b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1db4106020 0x7f1db419b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 -- 172.21.15.50:0/4080527895 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db41a02e0 con 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db2d9d700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db410b660 0x7f1db419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db259c700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 0x7f1db419b2d0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f1da400eb30 tx=0x7f1da400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1da37fe700 10 client.?.objecter ms_handle_connect 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1da37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da400cc90 con 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1da400ce30 con 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db41a05d0 con 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.783+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da4003d80 con 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1db8d46700 10 client.41726.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db41a7e50 con 0x7f1db4105660 2024-04-26T16:37:55.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1db8d46700 1 librados: init done 2024-04-26T16:37:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db406a110 con 0x7f1db4105660 2024-04-26T16:37:55.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f1da4003f20 con 0x7f1db4105660 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1d9c071f80 0x7f1d9c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(870..870 src has 781..870) ==== 8707+0+0 (secure 0 0 0) 0x7f1da4091a00 con 0x7f1db4105660 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 10 client.41726.objecter ms_dispatch 0x7f1db4001550 osd_map(870..870 src has 781..870) 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 3 client.41726.objecter handle_osd_map got epochs [870,870] > 0 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 3 client.41726.objecter handle_osd_map decoding full epoch 870 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1db1d9b700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1d9c071f80 0x7f1d9c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 20 client.41726.objecter dump_active .. 0 homeless 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1da37fe700 10 client.41726.objecter ms_handle_connect 0x7f1d9c071f80 2024-04-26T16:37:55.790 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.787+0000 7f1db1d9b700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1d9c071f80 0x7f1d9c074440 secure :-1 s=READY pgs=3846 cs=0 l=1 rev1=1 crypto rx=0x7f1db4103800 tx=0x7f1da8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:55.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:55.791+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1da405d650 con 0x7f1db4105660 2024-04-26T16:37:55.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:53.965316+0000 mgr.y (mgr.24401) 783 : cluster [DBG] pgmap v1357: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:55.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:53.965316+0000 mgr.y (mgr.24401) 783 : cluster [DBG] pgmap v1357: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:55.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.192831+0000 osd.1 (osd.1) 443 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:37:55.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.192831+0000 osd.1 (osd.1) 443 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:37:55.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.196206+0000 osd.1 (osd.1) 444 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.196206+0000 osd.1 (osd.1) 444 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.769799+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.c scrub starts 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.769799+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.c scrub starts 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.772039+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.c scrub ok 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[21463]: cluster 2024-04-26T16:37:54.772039+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.c scrub ok 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:53.965316+0000 mgr.y (mgr.24401) 783 : cluster [DBG] pgmap v1357: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:53.965316+0000 mgr.y (mgr.24401) 783 : cluster [DBG] pgmap v1357: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.192831+0000 osd.1 (osd.1) 443 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.192831+0000 osd.1 (osd.1) 443 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.196206+0000 osd.1 (osd.1) 444 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.196206+0000 osd.1 (osd.1) 444 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:37:55.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.769799+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.c scrub starts 2024-04-26T16:37:55.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.769799+0000 osd.3 (osd.3) 489 : cluster [DBG] 6.c scrub starts 2024-04-26T16:37:55.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.772039+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.c scrub ok 2024-04-26T16:37:55.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:55 smithi050 bash[29234]: cluster 2024-04-26T16:37:54.772039+0000 osd.3 (osd.3) 490 : cluster [DBG] 6.c scrub ok 2024-04-26T16:37:56.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.035+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f1db419c630 con 0x7f1db4105660 2024-04-26T16:37:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/3176015116 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806b130 msgr2=0x7f0a1806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.083 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/3176015116 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806b130 0x7f0a1806b590 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009990 tx=0x7f0a0c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/3176015116 shutdown_connections 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/3176015116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a181094a0 0x7f0a18109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/3176015116 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806bad0 0x7f0a181102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/3176015116 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806b130 0x7f0a1806b590 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/3176015116 >> 172.21.15.50:0/3176015116 conn(0x7f0a180792d0 msgr2=0x7f0a180796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/3176015116 shutdown_connections 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/3176015116 wait complete. 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 librados: starting msgr at 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 librados: starting objecter 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 Processor -- start 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- start start 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 librados: setting wanted keys 2024-04-26T16:37:56.084 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 librados: calling monclient init 2024-04-26T16:37:56.085 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806b130 0x7f0a18199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 0x7f0a18199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a181094a0 0x7f0a1819dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18112ae0 con 0x7f0a1806b130 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18112960 con 0x7f0a181094a0 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0a18112c60 con 0x7f0a1806bad0 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806b130 0x7f0a18199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 0x7f0a18199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806b130 0x7f0a18199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57794/0 (socket says 172.21.15.50:57794) 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 0x7f0a18199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46180/0 (socket says 172.21.15.50:46180) 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1659c700 1 -- 172.21.15.50:0/999487441 learned_addr learned my addr 172.21.15.50:0/999487441 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a16d9d700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a181094a0 0x7f0a1819dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 -- 172.21.15.50:0/999487441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a181094a0 msgr2=0x7f0a1819dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.086 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a181094a0 0x7f0a1819dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 -- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806b130 msgr2=0x7f0a18199240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806b130 0x7f0a18199240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 -- 172.21.15.50:0/999487441 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1819e200 con 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a15d9b700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 0x7f0a18199780 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0059d0 tx=0x7f0a0c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a077fe700 10 client.?.objecter ms_handle_connect 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a077fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c027070 con 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0a0c022470 con 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1819e490 con 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c017650 con 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 10 client.41732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a181a5d70 con 0x7f0a1806bad0 2024-04-26T16:37:56.087 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.083+0000 7f0a1ce58700 1 librados: init done 2024-04-26T16:37:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a18104b50 con 0x7f0a1806bad0 2024-04-26T16:37:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0a0c02a070 con 0x7f0a1806bad0 2024-04-26T16:37:56.088 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0a00071f80 0x7f0a00074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(870..870 src has 781..870) ==== 8707+0+0 (secure 0 0 0) 0x7f0a0c092700 con 0x7f0a1806bad0 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 10 client.41732.objecter ms_dispatch 0x7f0a18001550 osd_map(870..870 src has 781..870) 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 3 client.41732.objecter handle_osd_map got epochs [870,870] > 0 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 3 client.41732.objecter handle_osd_map decoding full epoch 870 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a1659c700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0a00071f80 0x7f0a00074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 20 client.41732.objecter dump_active .. 0 homeless 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a077fe700 10 client.41732.objecter ms_handle_connect 0x7f0a00071f80 2024-04-26T16:37:56.089 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.087+0000 7f0a1659c700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0a00071f80 0x7f0a00074440 secure :-1 s=READY pgs=3847 cs=0 l=1 rev1=1 crypto rx=0x7f0a08005950 tx=0x7f0a0800b500 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:37:56.092 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.091+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f0a0c013070 con 0x7f0a1806bad0 2024-04-26T16:37:56.320 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.319+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7f0a1819a550 con 0x7f0a1806bad0 2024-04-26T16:37:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.427+0000 7f0a077fe700 1 -- 172.21.15.50:0/999487441 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v871) ==== 101+0+0 (secure 0 0 0) 0x7f0a0c022620 con 0x7f0a1806bad0 2024-04-26T16:37:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.427+0000 7f1da37fe700 1 -- 172.21.15.50:0/4080527895 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v871) ==== 113+0+0 (secure 0 0 0) 0x7f1da4061300 con 0x7f1db4105660 2024-04-26T16:37:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 10 librados: watch_flush enter 2024-04-26T16:37:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 10 librados: watch_flush exit 2024-04-26T16:37:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 20 client.41726.objecter shutdown clearing up homeless session... 2024-04-26T16:37:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 10 client.41726.objecter successfully canceled tick 2024-04-26T16:37:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 10 librados: watch_flush enter 2024-04-26T16:37:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 10 librados: watch_flush exit 2024-04-26T16:37:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 20 client.41732.objecter shutdown clearing up homeless session... 2024-04-26T16:37:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 10 client.41732.objecter successfully canceled tick 2024-04-26T16:37:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1d9c071f80 msgr2=0x7f1d9c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1d9c071f80 0x7f1d9c074440 secure :-1 s=READY pgs=3846 cs=0 l=1 rev1=1 crypto rx=0x7f1db4103800 tx=0x7f1da8009260 comp rx=0 tx=0).stop 2024-04-26T16:37:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 msgr2=0x7f1db419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 0x7f1db419b2d0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f1da400eb30 tx=0x7f1da400ee40 comp rx=0 tx=0).stop 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0a00071f80 msgr2=0x7f0a00074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0a00071f80 0x7f0a00074440 secure :-1 s=READY pgs=3847 cs=0 l=1 rev1=1 crypto rx=0x7f0a08005950 tx=0x7f0a0800b500 comp rx=0 tx=0).stop 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 msgr2=0x7f0a18199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 0x7f0a18199780 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c0059d0 tx=0x7f0a0c0047b0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 shutdown_connections 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1d9c071f80 0x7f1d9c074440 unknown :-1 s=CLOSED pgs=3846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1db410b660 0x7f1db419fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1db4105660 0x7f1db419b2d0 unknown :-1 s=CLOSED pgs=2765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 --2- 172.21.15.50:0/4080527895 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1db4106020 0x7f1db419b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 >> 172.21.15.50:0/4080527895 conn(0x7f1db4101150 msgr2=0x7f1db41019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 shutdown_connections 2024-04-26T16:37:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0a00071f80 0x7f0a00074440 unknown :-1 s=CLOSED pgs=3847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0a181094a0 0x7f0a1819dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0a1806b130 0x7f0a18199240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 --2- 172.21.15.50:0/999487441 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0a1806bad0 0x7f0a18199780 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:37:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 >> 172.21.15.50:0/999487441 conn(0x7f0a180792d0 msgr2=0x7f0a18100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:37:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 shutdown_connections 2024-04-26T16:37:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 -- 172.21.15.50:0/4080527895 wait complete. 2024-04-26T16:37:56.437 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f1db8d46700 1 librados: shutdown 2024-04-26T16:37:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 shutdown_connections 2024-04-26T16:37:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 -- 172.21.15.50:0/999487441 wait complete. 2024-04-26T16:37:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:37:56.431+0000 7f0a1ce58700 1 librados: shutdown 2024-04-26T16:37:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:37:56.438 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T16:37:56.766 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:37:56.766 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-04-26T16:37:56.766 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-04-26T16:37:56.766 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.25249.0:8373 109.1a e598/584 109:5cd2c9bc:test-rados-api-smithi050-73372-99::propagate-base-tier-error:head v 598'91, mlcod=0'0)", 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:32:30.665610+0000", 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 326.10330889900001, 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.038252891999999997, 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.665610+0000", 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.665610+0000", 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.665611+0000", 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0900000000000002e-07 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.665618+0000", 2024-04-26T16:37:56.768 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.418e-06 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.665626+0000", 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5750000000000003e-06 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.665636+0000", 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0735e-05 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.769 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.702859+0000", 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.037222808000000003 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.702884+0000", 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5352000000000001e-05 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.703819+0000", 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00093486999999999997 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.770 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:32:30.703863+0000", 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3625000000000003e-05 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:538 326.4 326:20000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 696] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e696)", 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:34:22.725801+0000", 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 214.043117438, 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002847975, 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.771 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 538 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725801+0000", 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.772 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725801+0000", 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725804+0000", 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4200000000000001e-06 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725814+0000", 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6260000000000007e-06 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.773 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725833+0000", 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9091999999999999e-05 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725843+0000", 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0026e-05 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725900+0000", 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7785999999999999e-05 2024-04-26T16:37:56.774 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.725905+0000", 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8060000000000002e-06 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.728395+0000", 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024901609999999999 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.728435+0000", 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9607000000000002e-05 2024-04-26T16:37:56.775 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:22.728649+0000", 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000214451 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:637 326.4 326:20000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:34:25.822627+0000", 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.94629196599999, 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058958049999999996, 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.776 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 637 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822627+0000", 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.777 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822627+0000", 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822631+0000", 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.963e-06 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822651+0000", 2024-04-26T16:37:56.778 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9973e-05 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822693+0000", 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2268000000000003e-05 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822721+0000", 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7766000000000001e-05 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.779 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822878+0000", 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000157119 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.822889+0000", 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0556e-05 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.828309+0000", 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054204159999999999 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.780 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.828365+0000", 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6468000000000002e-05 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.828523+0000", 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000157276 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.781 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(6.17,QREQUEST,e:729,from:4,reservation_nonce:7)", 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:07.532828+0000", 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 169.23609098700001, 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006521336, 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.532828+0000", 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.782 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.532828+0000", 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.532830+0000", 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1059999999999998e-06 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.532842+0000", 2024-04-26T16:37:56.783 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1545999999999999e-05 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.532870+0000", 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8843000000000002e-05 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.532886+0000", 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5985999999999999e-05 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.533030+0000", 2024-04-26T16:37:56.784 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014338299999999999 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.533041+0000", 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1256e-05 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.539196+0000", 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0061547140000000004 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.785 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.539240+0000", 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3795000000000003e-05 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:07.539349+0000", 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109707 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.786 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:1423 330.1a 330:58000000:%01:::head [pgnls start_epoch 731] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e731)", 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:09.400752+0000", 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 167.36816681299999, 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036479640000000001, 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1423 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.787 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400752+0000", 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400752+0000", 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400755+0000", 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8710000000000001e-06 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.788 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400767+0000", 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1579e-05 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400798+0000", 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1823999999999999e-05 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400812+0000", 2024-04-26T16:37:56.789 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3719e-05 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400883+0000", 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1101999999999998e-05 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.400888+0000", 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2769999999999996e-06 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.790 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.404226+0000", 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033375240000000001 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.404275+0000", 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8541999999999997e-05 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:09.404400+0000", 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012552599999999999 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.791 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:1457 330.1a 330:58000000::::head [pgnls start_epoch 733] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e733)", 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:11.378580+0000", 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 165.390338515, 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049767489999999999, 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.792 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1457 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378580+0000", 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378580+0000", 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.793 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378584+0000", 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1300000000000001e-06 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378600+0000", 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6880000000000001e-05 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378640+0000", 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9455000000000001e-05 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.794 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378658+0000", 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8372999999999998e-05 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378829+0000", 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000170519 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.378840+0000", 2024-04-26T16:37:56.795 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1032999999999999e-05 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.383338+0000", 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044979420000000004 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.383395+0000", 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7658999999999998e-05 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.796 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:11.383557+0000", 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016175799999999999 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.3.28:104 323.7s0 323:ee25a38f:test-rados-api-smithi050-73372-127::foo:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e739)", 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:17.378832+0000", 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 159.390086416, 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034198079999999999, 2024-04-26T16:37:56.797 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378832+0000", 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378832+0000", 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.798 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378835+0000", 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1610000000000001e-06 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378843+0000", 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3259999999999996e-06 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378862+0000", 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8822999999999999e-05 2024-04-26T16:37:56.799 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378872+0000", 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8980000000000008e-06 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378940+0000", 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8566999999999996e-05 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378946+0000", 2024-04-26T16:37:56.800 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3299999999999998e-06 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.382165+0000", 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032191400000000001 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.382252+0000", 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7563000000000002e-05 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.801 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.3.28:105 323.7s0 323:ee25a38f:test-rados-api-smithi050-73372-127::foo:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e739)", 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:17.378892+0000", 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 159.39002735899999, 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034390010000000001, 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378892+0000", 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.802 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378892+0000", 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378892+0000", 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0900000000000002e-07 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378897+0000", 2024-04-26T16:37:56.803 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8949999999999997e-06 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378904+0000", 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1840000000000002e-06 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.378910+0000", 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8420000000000001e-06 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.379017+0000", 2024-04-26T16:37:56.804 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106745 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.379026+0000", 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1840000000000005e-06 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.382270+0000", 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032437339999999999 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.382331+0000", 2024-04-26T16:37:56.805 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0908e-05 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.3.28:106 323.7s0 323:ee25a38f:test-rados-api-smithi050-73372-127::foo:head [copy-from ver 2] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e739)", 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:17.408962+0000", 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 159.35995659, 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002036164, 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.806 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.408962+0000", 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.408962+0000", 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.408963+0000", 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1379999999999999e-06 2024-04-26T16:37:56.807 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.408970+0000", 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7240000000000003e-06 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.408983+0000", 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2779000000000001e-05 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.408989+0000", 2024-04-26T16:37:56.808 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4150000000000004e-06 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.409036+0000", 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6202999999999998e-05 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.409140+0000", 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104484 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:37:56.809 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.410101+0000", 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00096100299999999999 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.410458+0000", 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035667199999999999 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.410902+0000", 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044468699999999998 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.810 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:17.410998+0000", 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6058999999999996e-05 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(6.10,QREQUEST,e:740,from:0,reservation_nonce:5)", 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:18.781273+0000", 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 157.98764636800001, 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00206609, 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.811 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781273+0000", 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781273+0000", 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781289+0000", 2024-04-26T16:37:56.812 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6866000000000001e-05 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781305+0000", 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5241e-05 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781336+0000", 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1616999999999999e-05 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781352+0000", 2024-04-26T16:37:56.813 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5291e-05 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781455+0000", 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010343 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.781465+0000", 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6420000000000002e-06 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.814 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.783266+0000", 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001801815 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.783290+0000", 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3442999999999999e-05 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.783339+0000", 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8745000000000002e-05 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.815 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:1646 331.e 331:70000000::::head [pgnls start_epoch 741] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e741)", 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:19.382004+0000", 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 157.38691477500001, 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0034674580000000001, 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.816 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1646 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382004+0000", 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382004+0000", 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.817 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382006+0000", 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.99e-06 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382016+0000", 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7100000000000002e-06 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382036+0000", 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9803999999999999e-05 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.818 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382045+0000", 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1609999999999992e-06 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382111+0000", 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5794000000000001e-05 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.382115+0000", 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.729e-06 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.819 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.385318+0000", 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032029939999999998 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.385353+0000", 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4749999999999998e-05 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:19.385472+0000", 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000118526 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.820 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2296 335.14 335:28000000:%01:::head [pgnls start_epoch 778] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.472406+0000", 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 118.296512831, 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068843489999999997, 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2296 2024-04-26T16:37:56.821 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472406+0000", 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472406+0000", 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.822 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472409+0000", 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.146e-06 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472425+0000", 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5923e-05 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472464+0000", 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8985e-05 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.823 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472483+0000", 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9057000000000001e-05 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472617+0000", 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013364900000000001 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.472628+0000", 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0869999999999999e-05 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.824 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.478980+0000", 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063526399999999997 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.479072+0000", 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1444000000000001e-05 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.479290+0000", 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021863500000000001 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.825 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2330 335.14 335:28000000::::head [pgnls start_epoch 780] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e780)", 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:00.452571+0000", 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 116.316348199, 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058764059999999998, 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.826 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2330 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452571+0000", 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.827 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452571+0000", 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452574+0000", 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4630000000000001e-06 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452592+0000", 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7603000000000001e-05 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.828 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452632+0000", 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0119e-05 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452653+0000", 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1543e-05 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452812+0000", 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000158803 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.829 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.452824+0000", 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1452e-05 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.457963+0000", 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051394789999999997 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.830 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.458204+0000", 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024049400000000001 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.458447+0000", 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024345000000000001 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.831 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2397 336.4 336:20000000:%01:::head [pgnls start_epoch 786] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e786)", 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.471701+0000", 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 110.297217639, 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0056203379999999999, 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.832 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2397 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471701+0000", 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471701+0000", 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.833 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471705+0000", 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3730000000000001e-06 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471721+0000", 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6670000000000001e-05 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471761+0000", 2024-04-26T16:37:56.834 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9202999999999997e-05 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471779+0000", 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8935000000000001e-05 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471973+0000", 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019387899999999999 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.835 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.471985+0000", 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.129e-05 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.477011+0000", 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050267790000000003 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.477105+0000", 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3257999999999994e-05 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.836 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.477322+0000", 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000216951 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2431 336.4 336:20000000::::head [pgnls start_epoch 788] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e788)", 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:08.448920+0000", 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 108.319998582, 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069870189999999997, 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.837 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2431 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.448920+0000", 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.838 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.448920+0000", 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.448924+0000", 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7799999999999998e-06 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.448946+0000", 2024-04-26T16:37:56.839 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1869e-05 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.449006+0000", 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0272000000000002e-05 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.449035+0000", 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8938999999999999e-05 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.840 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.449314+0000", 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027919300000000001 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.449332+0000", 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8006e-05 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.455598+0000", 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0062655979999999998 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.841 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.455688+0000", 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0121e-05 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.455907+0000", 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021924100000000001 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2661 338.c 338:30000000::::head [pgnls start_epoch 809] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e809)", 2024-04-26T16:37:56.842 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:38.624838+0000", 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 78.144080750000001, 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006667302, 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2661 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.843 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.624838+0000", 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.624838+0000", 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.624842+0000", 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8600000000000003e-06 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.844 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.624859+0000", 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7487e-05 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.624899+0000", 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9412000000000001e-05 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.624926+0000", 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7368999999999998e-05 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.845 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.625077+0000", 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015059000000000001 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.625088+0000", 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1603000000000001e-05 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.630648+0000", 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055598319999999998 2024-04-26T16:37:56.846 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.631346+0000", 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00069753299999999996 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.631505+0000", 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015961600000000001 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.847 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "rep_scrubmap(3.16 e584 from shard 7)", 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:44.714360+0000", 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 72.054559362000006, 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020783329999999999, 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.714360+0000", 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.848 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.714360+0000", 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.714362+0000", 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0600000000000002e-06 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.714376+0000", 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4856e-05 2024-04-26T16:37:56.849 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.714395+0000", 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8807000000000001e-05 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.714410+0000", 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5056e-05 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.716296+0000", 2024-04-26T16:37:56.850 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018856470000000001 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.716327+0000", 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0732999999999999e-05 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.716438+0000", 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000111174 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.851 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2755 339.6 339:60000000:test-rados-api-smithi050-73372-143:::head [pgnls start_epoch 821] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e821)", 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:50.437841+0000", 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 66.331078035999994, 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070233969999999998, 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2755 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.852 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.437841+0000", 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.437841+0000", 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.437844+0000", 2024-04-26T16:37:56.853 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5870000000000001e-06 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.437868+0000", 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3088999999999999e-05 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.437907+0000", 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8965999999999999e-05 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.854 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.437929+0000", 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.266e-05 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.438051+0000", 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012211000000000001 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.438063+0000", 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1416e-05 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.855 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.444664+0000", 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066013699999999996 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.444716+0000", 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1745e-05 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:50.444864+0000", 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014845400000000001 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.856 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2820 339.6 339:60000000:%01:::head [pgnls start_epoch 823] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e823)", 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:52.437961+0000", 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 64.330958090999999, 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0062949670000000003, 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:37:56.857 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2820 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.437961+0000", 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.437961+0000", 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.858 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.437966+0000", 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8169999999999998e-06 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.437987+0000", 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0922000000000001e-05 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.438035+0000", 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8013000000000003e-05 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.859 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.438062+0000", 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7106000000000001e-05 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.438234+0000", 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000172559 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.438244+0000", 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0276e-05 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.860 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.444052+0000", 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0058078030000000003 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.444108+0000", 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5251999999999999e-05 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.444256+0000", 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000148219 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.861 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(4.1,RELEASE,e:870,from:4,reservation_nonce:0)", 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:37:52.817790+0000", 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 3.9511288479999997, 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024826340000000001, 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:37:56.862 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.817790+0000", 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.817790+0000", 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.817791+0000", 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1230000000000001e-06 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.863 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.817799+0000", 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2600000000000005e-06 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.817822+0000", 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2189e-05 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.817839+0000", 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7824000000000001e-05 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.864 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.820217+0000", 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002377713 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.820247+0000", 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0213e-05 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:52.820273+0000", 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5312000000000001e-05 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.865 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:37:56.866 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:37:56.866 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:37:56.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.745564+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:37:56.866 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.745564+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.747997+0000 osd.3 (osd.3) 492 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.747997+0000 osd.3 (osd.3) 492 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.771785+0000 osd.4 (osd.4) 425 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.771785+0000 osd.4 (osd.4) 425 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.775580+0000 osd.4 (osd.4) 426 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.775580+0000 osd.4 (osd.4) 426 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.042485+0000 mon.c (mon.2) 906 : audit [INF] from='client.? 172.21.15.50:0/4080527895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.042485+0000 mon.c (mon.2) 906 : audit [INF] from='client.? 172.21.15.50:0/4080527895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.042914+0000 mon.a (mon.0) 3943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.042914+0000 mon.a (mon.0) 3943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.086136+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.086136+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.088648+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.088648+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:37:56.867 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.325079+0000 mon.c (mon.2) 907 : audit [INF] from='client.? 172.21.15.50:0/999487441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.325079+0000 mon.c (mon.2) 907 : audit [INF] from='client.? 172.21.15.50:0/999487441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.325483+0000 mon.a (mon.0) 3944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.325483+0000 mon.a (mon.0) 3944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.421151+0000 mon.a (mon.0) 3945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.421151+0000 mon.a (mon.0) 3945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.423641+0000 mon.a (mon.0) 3946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.423641+0000 mon.a (mon.0) 3946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.423872+0000 mon.a (mon.0) 3947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: audit 2024-04-26T16:37:56.423872+0000 mon.a (mon.0) 3947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.431319+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-04-26T16:37:56.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:56 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.431319+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-04-26T16:37:56.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.745564+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.745564+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.747997+0000 osd.3 (osd.3) 492 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.747997+0000 osd.3 (osd.3) 492 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.771785+0000 osd.4 (osd.4) 425 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.771785+0000 osd.4 (osd.4) 425 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.775580+0000 osd.4 (osd.4) 426 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.775580+0000 osd.4 (osd.4) 426 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.042485+0000 mon.c (mon.2) 906 : audit [INF] from='client.? 172.21.15.50:0/4080527895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.042485+0000 mon.c (mon.2) 906 : audit [INF] from='client.? 172.21.15.50:0/4080527895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.042914+0000 mon.a (mon.0) 3943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.042914+0000 mon.a (mon.0) 3943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.086136+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.086136+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:37:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.088648+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.088648+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.325079+0000 mon.c (mon.2) 907 : audit [INF] from='client.? 172.21.15.50:0/999487441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.325079+0000 mon.c (mon.2) 907 : audit [INF] from='client.? 172.21.15.50:0/999487441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.325483+0000 mon.a (mon.0) 3944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.325483+0000 mon.a (mon.0) 3944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.421151+0000 mon.a (mon.0) 3945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.421151+0000 mon.a (mon.0) 3945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.423641+0000 mon.a (mon.0) 3946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:37:56.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.745564+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.745564+0000 osd.3 (osd.3) 491 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.747997+0000 osd.3 (osd.3) 492 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.747997+0000 osd.3 (osd.3) 492 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.771785+0000 osd.4 (osd.4) 425 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.771785+0000 osd.4 (osd.4) 425 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.775580+0000 osd.4 (osd.4) 426 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.775580+0000 osd.4 (osd.4) 426 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.042485+0000 mon.c (mon.2) 906 : audit [INF] from='client.? 172.21.15.50:0/4080527895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.042485+0000 mon.c (mon.2) 906 : audit [INF] from='client.? 172.21.15.50:0/4080527895' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.042914+0000 mon.a (mon.0) 3943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.042914+0000 mon.a (mon.0) 3943 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.086136+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.086136+0000 osd.5 (osd.5) 326 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:37:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.088648+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.088648+0000 osd.5 (osd.5) 327 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.325079+0000 mon.c (mon.2) 907 : audit [INF] from='client.? 172.21.15.50:0/999487441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.325079+0000 mon.c (mon.2) 907 : audit [INF] from='client.? 172.21.15.50:0/999487441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.325483+0000 mon.a (mon.0) 3944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.325483+0000 mon.a (mon.0) 3944 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.421151+0000 mon.a (mon.0) 3945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.421151+0000 mon.a (mon.0) 3945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.423641+0000 mon.a (mon.0) 3946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.423641+0000 mon.a (mon.0) 3946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.423872+0000 mon.a (mon.0) 3947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: audit 2024-04-26T16:37:56.423872+0000 mon.a (mon.0) 3947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:56.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.431319+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-04-26T16:37:56.885 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.431319+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-04-26T16:37:56.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.423641+0000 mon.a (mon.0) 3946 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:37:56.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.423872+0000 mon.a (mon.0) 3947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:56.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: audit 2024-04-26T16:37:56.423872+0000 mon.a (mon.0) 3947 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:37:56.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.431319+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-04-26T16:37:56.885 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:56 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.431319+0000 mon.a (mon.0) 3948 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2024-04-26T16:37:57.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-04-26T16:37:57.780 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T16:37:57.781 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:37:57.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.965759+0000 mgr.y (mgr.24401) 784 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:55.965759+0000 mgr.y (mgr.24401) 784 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.104966+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.104966+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.108277+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.e scrub ok 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.108277+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.e scrub ok 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.156960+0000 osd.1 (osd.1) 445 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.156960+0000 osd.1 (osd.1) 445 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.159150+0000 osd.1 (osd.1) 446 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.159150+0000 osd.1 (osd.1) 446 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:37:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.305535+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.305535+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.309106+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.309106+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.380010+0000 osd.2 (osd.2) 413 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.380010+0000 osd.2 (osd.2) 413 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.382173+0000 osd.2 (osd.2) 414 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.382173+0000 osd.2 (osd.2) 414 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.782939+0000 osd.4 (osd.4) 427 : cluster [DBG] 6.e scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.782939+0000 osd.4 (osd.4) 427 : cluster [DBG] 6.e scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.784680+0000 osd.4 (osd.4) 428 : cluster [DBG] 6.e scrub ok 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:56.784680+0000 osd.4 (osd.4) 428 : cluster [DBG] 6.e scrub ok 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.116105+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:37:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.116105+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:37:57.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.118288+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:37:57.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:57 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.118288+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:37:57.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.965759+0000 mgr.y (mgr.24401) 784 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-04-26T16:37:57.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:55.965759+0000 mgr.y (mgr.24401) 784 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.104966+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.104966+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.108277+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.e scrub ok 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.108277+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.e scrub ok 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.156960+0000 osd.1 (osd.1) 445 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.156960+0000 osd.1 (osd.1) 445 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.159150+0000 osd.1 (osd.1) 446 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.159150+0000 osd.1 (osd.1) 446 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.305535+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.305535+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.309106+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.309106+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.380010+0000 osd.2 (osd.2) 413 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:37:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.380010+0000 osd.2 (osd.2) 413 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.382173+0000 osd.2 (osd.2) 414 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.382173+0000 osd.2 (osd.2) 414 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.782939+0000 osd.4 (osd.4) 427 : cluster [DBG] 6.e scrub starts 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.782939+0000 osd.4 (osd.4) 427 : cluster [DBG] 6.e scrub starts 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.784680+0000 osd.4 (osd.4) 428 : cluster [DBG] 6.e scrub ok 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:56.784680+0000 osd.4 (osd.4) 428 : cluster [DBG] 6.e scrub ok 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.116105+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.116105+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.118288+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.118288+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:37:57.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.965759+0000 mgr.y (mgr.24401) 784 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:55.965759+0000 mgr.y (mgr.24401) 784 : cluster [DBG] pgmap v1358: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.104966+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.104966+0000 osd.7 (osd.7) 442 : cluster [DBG] 3.e scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.108277+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.e scrub ok 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.108277+0000 osd.7 (osd.7) 443 : cluster [DBG] 3.e scrub ok 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.156960+0000 osd.1 (osd.1) 445 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.156960+0000 osd.1 (osd.1) 445 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.159150+0000 osd.1 (osd.1) 446 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.159150+0000 osd.1 (osd.1) 446 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.305535+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.305535+0000 osd.6 (osd.6) 45 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.309106+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.309106+0000 osd.6 (osd.6) 46 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.380010+0000 osd.2 (osd.2) 413 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:37:57.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.380010+0000 osd.2 (osd.2) 413 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.382173+0000 osd.2 (osd.2) 414 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.382173+0000 osd.2 (osd.2) 414 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.782939+0000 osd.4 (osd.4) 427 : cluster [DBG] 6.e scrub starts 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.782939+0000 osd.4 (osd.4) 427 : cluster [DBG] 6.e scrub starts 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.784680+0000 osd.4 (osd.4) 428 : cluster [DBG] 6.e scrub ok 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:56.784680+0000 osd.4 (osd.4) 428 : cluster [DBG] 6.e scrub ok 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.116105+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.116105+0000 osd.5 (osd.5) 328 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.118288+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:37:57.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:57 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.118288+0000 osd.5 (osd.5) 329 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: audit 2024-04-26T16:37:57.365117+0000 mgr.y (mgr.24401) 785 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: audit 2024-04-26T16:37:57.365117+0000 mgr.y (mgr.24401) 785 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.738643+0000 osd.3 (osd.3) 493 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.738643+0000 osd.3 (osd.3) 493 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.743459+0000 osd.3 (osd.3) 494 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.743459+0000 osd.3 (osd.3) 494 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.833565+0000 osd.0 (osd.0) 479 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.833565+0000 osd.0 (osd.0) 479 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.836063+0000 osd.0 (osd.0) 480 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.836063+0000 osd.0 (osd.0) 480 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:37:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: audit 2024-04-26T16:37:58.065986+0000 mon.a (mon.0) 3949 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: audit 2024-04-26T16:37:58.065986+0000 mon.a (mon.0) 3949 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: audit 2024-04-26T16:37:58.069204+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: audit 2024-04-26T16:37:58.069204+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.130362+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.130362+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.132976+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:37:58.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:58 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.132976+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:37:58.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: audit 2024-04-26T16:37:57.365117+0000 mgr.y (mgr.24401) 785 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: audit 2024-04-26T16:37:57.365117+0000 mgr.y (mgr.24401) 785 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.738643+0000 osd.3 (osd.3) 493 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.738643+0000 osd.3 (osd.3) 493 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.743459+0000 osd.3 (osd.3) 494 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.743459+0000 osd.3 (osd.3) 494 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.833565+0000 osd.0 (osd.0) 479 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.833565+0000 osd.0 (osd.0) 479 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.836063+0000 osd.0 (osd.0) 480 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.836063+0000 osd.0 (osd.0) 480 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: audit 2024-04-26T16:37:58.065986+0000 mon.a (mon.0) 3949 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: audit 2024-04-26T16:37:58.065986+0000 mon.a (mon.0) 3949 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: audit 2024-04-26T16:37:58.069204+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: audit 2024-04-26T16:37:58.069204+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.130362+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.130362+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.132976+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.132976+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: audit 2024-04-26T16:37:57.365117+0000 mgr.y (mgr.24401) 785 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: audit 2024-04-26T16:37:57.365117+0000 mgr.y (mgr.24401) 785 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.738643+0000 osd.3 (osd.3) 493 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.738643+0000 osd.3 (osd.3) 493 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.743459+0000 osd.3 (osd.3) 494 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.743459+0000 osd.3 (osd.3) 494 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.833565+0000 osd.0 (osd.0) 479 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:37:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.833565+0000 osd.0 (osd.0) 479 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.836063+0000 osd.0 (osd.0) 480 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.836063+0000 osd.0 (osd.0) 480 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: audit 2024-04-26T16:37:58.065986+0000 mon.a (mon.0) 3949 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: audit 2024-04-26T16:37:58.065986+0000 mon.a (mon.0) 3949 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: audit 2024-04-26T16:37:58.069204+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: audit 2024-04-26T16:37:58.069204+0000 mon.c (mon.2) 908 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.130362+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.130362+0000 osd.5 (osd.5) 330 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.132976+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:37:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:58 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.132976+0000 osd.5 (osd.5) 331 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:37:59.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset noscrub 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.967831+0000 mgr.y (mgr.24401) 786 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:57.967831+0000 mgr.y (mgr.24401) 786 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.721865+0000 osd.4 (osd.4) 429 : cluster [DBG] 11.e scrub starts 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.721865+0000 osd.4 (osd.4) 429 : cluster [DBG] 11.e scrub starts 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.725069+0000 osd.4 (osd.4) 430 : cluster [DBG] 11.e scrub ok 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.725069+0000 osd.4 (osd.4) 430 : cluster [DBG] 11.e scrub ok 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.735818+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.735818+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.738694+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:37:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:37:59 smithi190 bash[23868]: cluster 2024-04-26T16:37:58.738694+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.967831+0000 mgr.y (mgr.24401) 786 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:57.967831+0000 mgr.y (mgr.24401) 786 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.721865+0000 osd.4 (osd.4) 429 : cluster [DBG] 11.e scrub starts 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.721865+0000 osd.4 (osd.4) 429 : cluster [DBG] 11.e scrub starts 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.725069+0000 osd.4 (osd.4) 430 : cluster [DBG] 11.e scrub ok 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.725069+0000 osd.4 (osd.4) 430 : cluster [DBG] 11.e scrub ok 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.735818+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:37:59.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.735818+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.738694+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[29234]: cluster 2024-04-26T16:37:58.738694+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.967831+0000 mgr.y (mgr.24401) 786 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:57.967831+0000 mgr.y (mgr.24401) 786 : cluster [DBG] pgmap v1360: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.721865+0000 osd.4 (osd.4) 429 : cluster [DBG] 11.e scrub starts 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.721865+0000 osd.4 (osd.4) 429 : cluster [DBG] 11.e scrub starts 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.725069+0000 osd.4 (osd.4) 430 : cluster [DBG] 11.e scrub ok 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.725069+0000 osd.4 (osd.4) 430 : cluster [DBG] 11.e scrub ok 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.735818+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.735818+0000 osd.3 (osd.3) 495 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.738694+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:37:59.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:37:59 smithi050 bash[21463]: cluster 2024-04-26T16:37:58.738694+0000 osd.3 (osd.3) 496 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:38:00.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.106354+0000 osd.7 (osd.7) 444 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.106354+0000 osd.7 (osd.7) 444 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.108808+0000 osd.7 (osd.7) 445 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.108808+0000 osd.7 (osd.7) 445 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.382893+0000 osd.2 (osd.2) 415 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.382893+0000 osd.2 (osd.2) 415 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.386140+0000 osd.2 (osd.2) 416 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.386140+0000 osd.2 (osd.2) 416 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.720476+0000 osd.3 (osd.3) 497 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.720476+0000 osd.3 (osd.3) 497 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.723427+0000 osd.3 (osd.3) 498 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.723427+0000 osd.3 (osd.3) 498 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.433364+0000 osd.2 (osd.2) 417 : cluster [DBG] 5.d scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.433364+0000 osd.2 (osd.2) 417 : cluster [DBG] 5.d scrub starts 2024-04-26T16:38:00.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.436434+0000 osd.2 (osd.2) 418 : cluster [DBG] 5.d scrub ok 2024-04-26T16:38:00.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:00 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.436434+0000 osd.2 (osd.2) 418 : cluster [DBG] 5.d scrub ok 2024-04-26T16:38:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.106354+0000 osd.7 (osd.7) 444 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:38:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.106354+0000 osd.7 (osd.7) 444 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:38:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.108808+0000 osd.7 (osd.7) 445 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.108808+0000 osd.7 (osd.7) 445 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.382893+0000 osd.2 (osd.2) 415 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.382893+0000 osd.2 (osd.2) 415 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.386140+0000 osd.2 (osd.2) 416 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.386140+0000 osd.2 (osd.2) 416 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.720476+0000 osd.3 (osd.3) 497 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.720476+0000 osd.3 (osd.3) 497 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.723427+0000 osd.3 (osd.3) 498 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.723427+0000 osd.3 (osd.3) 498 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.433364+0000 osd.2 (osd.2) 417 : cluster [DBG] 5.d scrub starts 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.433364+0000 osd.2 (osd.2) 417 : cluster [DBG] 5.d scrub starts 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.436434+0000 osd.2 (osd.2) 418 : cluster [DBG] 5.d scrub ok 2024-04-26T16:38:00.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.436434+0000 osd.2 (osd.2) 418 : cluster [DBG] 5.d scrub ok 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.106354+0000 osd.7 (osd.7) 444 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.106354+0000 osd.7 (osd.7) 444 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.108808+0000 osd.7 (osd.7) 445 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.108808+0000 osd.7 (osd.7) 445 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.382893+0000 osd.2 (osd.2) 415 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.382893+0000 osd.2 (osd.2) 415 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.386140+0000 osd.2 (osd.2) 416 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.386140+0000 osd.2 (osd.2) 416 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.720476+0000 osd.3 (osd.3) 497 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.720476+0000 osd.3 (osd.3) 497 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.723427+0000 osd.3 (osd.3) 498 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:38:00.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.723427+0000 osd.3 (osd.3) 498 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:38:00.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.433364+0000 osd.2 (osd.2) 417 : cluster [DBG] 5.d scrub starts 2024-04-26T16:38:00.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.433364+0000 osd.2 (osd.2) 417 : cluster [DBG] 5.d scrub starts 2024-04-26T16:38:00.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.436434+0000 osd.2 (osd.2) 418 : cluster [DBG] 5.d scrub ok 2024-04-26T16:38:00.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:00 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.436434+0000 osd.2 (osd.2) 418 : cluster [DBG] 5.d scrub ok 2024-04-26T16:38:01.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.968665+0000 mgr.y (mgr.24401) 787 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:01.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:37:59.968665+0000 mgr.y (mgr.24401) 787 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:01.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.066657+0000 osd.7 (osd.7) 446 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.066657+0000 osd.7 (osd.7) 446 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.068748+0000 osd.7 (osd.7) 447 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.068748+0000 osd.7 (osd.7) 447 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.192279+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.192279+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.194499+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:00.194499+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.109384+0000 osd.5 (osd.5) 332 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.109384+0000 osd.5 (osd.5) 332 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.112374+0000 osd.5 (osd.5) 333 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.112374+0000 osd.5 (osd.5) 333 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:38:01.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.968665+0000 mgr.y (mgr.24401) 787 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:37:59.968665+0000 mgr.y (mgr.24401) 787 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.066657+0000 osd.7 (osd.7) 446 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.066657+0000 osd.7 (osd.7) 446 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.068748+0000 osd.7 (osd.7) 447 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.068748+0000 osd.7 (osd.7) 447 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.192279+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.192279+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.194499+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:00.194499+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.109384+0000 osd.5 (osd.5) 332 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.109384+0000 osd.5 (osd.5) 332 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.112374+0000 osd.5 (osd.5) 333 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:38:01.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.112374+0000 osd.5 (osd.5) 333 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.968665+0000 mgr.y (mgr.24401) 787 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:37:59.968665+0000 mgr.y (mgr.24401) 787 : cluster [DBG] pgmap v1361: 164 pgs: 164 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.066657+0000 osd.7 (osd.7) 446 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.066657+0000 osd.7 (osd.7) 446 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.068748+0000 osd.7 (osd.7) 447 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.068748+0000 osd.7 (osd.7) 447 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.192279+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.192279+0000 osd.1 (osd.1) 447 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.194499+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:00.194499+0000 osd.1 (osd.1) 448 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.109384+0000 osd.5 (osd.5) 332 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.109384+0000 osd.5 (osd.5) 332 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:38:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.112374+0000 osd.5 (osd.5) 333 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:38:02.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.112374+0000 osd.5 (osd.5) 333 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:38:02.266 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:38:02.562 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.821713+0000 osd.0 (osd.0) 481 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.821713+0000 osd.0 (osd.0) 481 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.824236+0000 osd.0 (osd.0) 482 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.824236+0000 osd.0 (osd.0) 482 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.821713+0000 osd.0 (osd.0) 481 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.821713+0000 osd.0 (osd.0) 481 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:38:02.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.824236+0000 osd.0 (osd.0) 482 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:38:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:02 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.824236+0000 osd.0 (osd.0) 482 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:38:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:02 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.821713+0000 osd.0 (osd.0) 481 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:38:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:02 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.821713+0000 osd.0 (osd.0) 481 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:38:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:02 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.824236+0000 osd.0 (osd.0) 482 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:38:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:02 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.824236+0000 osd.0 (osd.0) 482 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:38:03.576 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:03.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.970422+0000 mgr.y (mgr.24401) 788 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:01.970422+0000 mgr.y (mgr.24401) 788 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.123005+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.123005+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.125406+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.125406+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.226773+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.226773+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.229218+0000 osd.6 (osd.6) 48 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.229218+0000 osd.6 (osd.6) 48 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.655386+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.655386+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:38:03.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.657902+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.657902+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.819864+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.819864+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.823771+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[21463]: cluster 2024-04-26T16:38:02.823771+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.970422+0000 mgr.y (mgr.24401) 788 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:01.970422+0000 mgr.y (mgr.24401) 788 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.123005+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.123005+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:38:03.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.125406+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.125406+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.226773+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.226773+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.229218+0000 osd.6 (osd.6) 48 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.229218+0000 osd.6 (osd.6) 48 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.655386+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.655386+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.657902+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.657902+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.819864+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.819864+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:38:03.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.823771+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:38:03.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:03 smithi050 bash[29234]: cluster 2024-04-26T16:38:02.823771+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.970422+0000 mgr.y (mgr.24401) 788 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:01.970422+0000 mgr.y (mgr.24401) 788 : cluster [DBG] pgmap v1362: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.123005+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.123005+0000 osd.1 (osd.1) 449 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.125406+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.125406+0000 osd.1 (osd.1) 450 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.226773+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.226773+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.229218+0000 osd.6 (osd.6) 48 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.229218+0000 osd.6 (osd.6) 48 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:38:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.655386+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.655386+0000 osd.3 (osd.3) 499 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.657902+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.657902+0000 osd.3 (osd.3) 500 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.819864+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.819864+0000 osd.0 (osd.0) 483 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.823771+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:38:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:03 smithi190 bash[23868]: cluster 2024-04-26T16:38:02.823771+0000 osd.0 (osd.0) 484 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:38:04.597 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:38:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.137520+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.137520+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.140916+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.140916+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.154558+0000 osd.1 (osd.1) 451 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.154558+0000 osd.1 (osd.1) 451 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.160050+0000 osd.1 (osd.1) 452 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.160050+0000 osd.1 (osd.1) 452 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.696365+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:38:04.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.696365+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.698754+0000 osd.3 (osd.3) 502 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.698754+0000 osd.3 (osd.3) 502 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.116811+0000 osd.5 (osd.5) 334 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.116811+0000 osd.5 (osd.5) 334 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.121767+0000 osd.5 (osd.5) 335 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.121767+0000 osd.5 (osd.5) 335 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.137520+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:38:04.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.137520+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.140916+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.140916+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.154558+0000 osd.1 (osd.1) 451 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.154558+0000 osd.1 (osd.1) 451 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.160050+0000 osd.1 (osd.1) 452 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.160050+0000 osd.1 (osd.1) 452 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.696365+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.696365+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.698754+0000 osd.3 (osd.3) 502 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.698754+0000 osd.3 (osd.3) 502 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.116811+0000 osd.5 (osd.5) 334 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.116811+0000 osd.5 (osd.5) 334 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.121767+0000 osd.5 (osd.5) 335 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:38:04.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:04 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.121767+0000 osd.5 (osd.5) 335 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:38:04.965 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:38:04.965 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:38:04.965 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:38:04.965 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.137520+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.137520+0000 osd.7 (osd.7) 448 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.140916+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.140916+0000 osd.7 (osd.7) 449 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.154558+0000 osd.1 (osd.1) 451 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.154558+0000 osd.1 (osd.1) 451 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.160050+0000 osd.1 (osd.1) 452 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.160050+0000 osd.1 (osd.1) 452 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:38:04.977 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.696365+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.696365+0000 osd.3 (osd.3) 501 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.698754+0000 osd.3 (osd.3) 502 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.698754+0000 osd.3 (osd.3) 502 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.116811+0000 osd.5 (osd.5) 334 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.116811+0000 osd.5 (osd.5) 334 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.121767+0000 osd.5 (osd.5) 335 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:38:04.978 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:04 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.121767+0000 osd.5 (osd.5) 335 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:38:05.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.971415+0000 mgr.y (mgr.24401) 789 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:03.971415+0000 mgr.y (mgr.24401) 789 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.130585+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.130585+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.132188+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.132188+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.163154+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.163154+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.165348+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:38:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[21463]: cluster 2024-04-26T16:38:04.165348+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.971415+0000 mgr.y (mgr.24401) 789 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:03.971415+0000 mgr.y (mgr.24401) 789 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.130585+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.130585+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.132188+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.132188+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.163154+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.163154+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.165348+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:38:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:05 smithi050 bash[29234]: cluster 2024-04-26T16:38:04.165348+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:38:06.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.971415+0000 mgr.y (mgr.24401) 789 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:03.971415+0000 mgr.y (mgr.24401) 789 : cluster [DBG] pgmap v1363: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.130585+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.130585+0000 osd.7 (osd.7) 450 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.132188+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.132188+0000 osd.7 (osd.7) 451 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.163154+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.163154+0000 osd.1 (osd.1) 453 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.165348+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:38:06.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:05 smithi190 bash[23868]: cluster 2024-04-26T16:38:04.165348+0000 osd.1 (osd.1) 454 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:38:06.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.194383+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:38:06.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.194383+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.197952+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.197952+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.710076+0000 osd.3 (osd.3) 503 : cluster [DBG] 11.a scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.710076+0000 osd.3 (osd.3) 503 : cluster [DBG] 11.a scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.712510+0000 osd.3 (osd.3) 504 : cluster [DBG] 11.a scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.712510+0000 osd.3 (osd.3) 504 : cluster [DBG] 11.a scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.798321+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.798321+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.800609+0000 osd.4 (osd.4) 432 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.800609+0000 osd.4 (osd.4) 432 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.802714+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.802714+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.806174+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:38:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.806174+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.194383+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.194383+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.197952+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.197952+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.710076+0000 osd.3 (osd.3) 503 : cluster [DBG] 11.a scrub starts 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.710076+0000 osd.3 (osd.3) 503 : cluster [DBG] 11.a scrub starts 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.712510+0000 osd.3 (osd.3) 504 : cluster [DBG] 11.a scrub ok 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.712510+0000 osd.3 (osd.3) 504 : cluster [DBG] 11.a scrub ok 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.798321+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:38:06.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.798321+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:38:06.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.800609+0000 osd.4 (osd.4) 432 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:38:06.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.800609+0000 osd.4 (osd.4) 432 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:38:06.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.802714+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:38:06.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.802714+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:38:06.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.806174+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:38:06.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:06 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.806174+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.194383+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.194383+0000 osd.6 (osd.6) 49 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.197952+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.197952+0000 osd.6 (osd.6) 50 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.710076+0000 osd.3 (osd.3) 503 : cluster [DBG] 11.a scrub starts 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.710076+0000 osd.3 (osd.3) 503 : cluster [DBG] 11.a scrub starts 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.712510+0000 osd.3 (osd.3) 504 : cluster [DBG] 11.a scrub ok 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.712510+0000 osd.3 (osd.3) 504 : cluster [DBG] 11.a scrub ok 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.798321+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.798321+0000 osd.4 (osd.4) 431 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.800609+0000 osd.4 (osd.4) 432 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:38:07.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.800609+0000 osd.4 (osd.4) 432 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:38:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.802714+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:38:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.802714+0000 osd.0 (osd.0) 485 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:38:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.806174+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:38:07.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:06 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.806174+0000 osd.0 (osd.0) 486 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:38:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.971899+0000 mgr.y (mgr.24401) 790 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:07.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[23868]: cluster 2024-04-26T16:38:05.971899+0000 mgr.y (mgr.24401) 790 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:07.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[23868]: cluster 2024-04-26T16:38:06.769472+0000 osd.4 (osd.4) 433 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:38:07.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[23868]: cluster 2024-04-26T16:38:06.769472+0000 osd.4 (osd.4) 433 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:38:07.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[23868]: cluster 2024-04-26T16:38:06.771743+0000 osd.4 (osd.4) 434 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:38:07.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[23868]: cluster 2024-04-26T16:38:06.771743+0000 osd.4 (osd.4) 434 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:38:07.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:38:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:38:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.971899+0000 mgr.y (mgr.24401) 790 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[21463]: cluster 2024-04-26T16:38:05.971899+0000 mgr.y (mgr.24401) 790 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[21463]: cluster 2024-04-26T16:38:06.769472+0000 osd.4 (osd.4) 433 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:38:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[21463]: cluster 2024-04-26T16:38:06.769472+0000 osd.4 (osd.4) 433 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:38:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[21463]: cluster 2024-04-26T16:38:06.771743+0000 osd.4 (osd.4) 434 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:38:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[21463]: cluster 2024-04-26T16:38:06.771743+0000 osd.4 (osd.4) 434 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:38:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.971899+0000 mgr.y (mgr.24401) 790 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[29234]: cluster 2024-04-26T16:38:05.971899+0000 mgr.y (mgr.24401) 790 : cluster [DBG] pgmap v1364: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[29234]: cluster 2024-04-26T16:38:06.769472+0000 osd.4 (osd.4) 433 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:38:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[29234]: cluster 2024-04-26T16:38:06.769472+0000 osd.4 (osd.4) 433 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:38:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[29234]: cluster 2024-04-26T16:38:06.771743+0000 osd.4 (osd.4) 434 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:38:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:07 smithi050 bash[29234]: cluster 2024-04-26T16:38:06.771743+0000 osd.4 (osd.4) 434 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- 172.21.15.50:0/1519571319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 msgr2=0x7f1ed0105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- 172.21.15.50:0/1519571319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed0105ac0 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40098f0 tx=0x7f1ec4009c00 comp rx=0 tx=0).stop 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- 172.21.15.50:0/1519571319 shutdown_connections 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- 172.21.15.50:0/1519571319 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed0106000 0x7f1ed0112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- 172.21.15.50:0/1519571319 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ed010b640 0x7f1ed010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- 172.21.15.50:0/1519571319 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed0105ac0 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- 172.21.15.50:0/1519571319 >> 172.21.15.50:0/1519571319 conn(0x7f1ed0101130 msgr2=0x7f1ed0103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- 172.21.15.50:0/1519571319 shutdown_connections 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- 172.21.15.50:0/1519571319 wait complete. 2024-04-26T16:38:08.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 librados: starting msgr at 2024-04-26T16:38:08.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 librados: starting objecter 2024-04-26T16:38:08.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 Processor -- start 2024-04-26T16:38:08.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- start start 2024-04-26T16:38:08.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 librados: setting wanted keys 2024-04-26T16:38:08.294 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 librados: calling monclient init 2024-04-26T16:38:08.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed0106000 0x7f1ed019b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ed010b640 0x7f1ed019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed0114da0 con 0x7f1ed010b640 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed0114c20 con 0x7f1ed0106000 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed667e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed0114f20 con 0x7f1ed0105640 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ecffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ecffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed019b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57188/0 (socket says 172.21.15.50:57188) 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ecffff700 1 -- 172.21.15.50:0/720955995 learned_addr learned my addr 172.21.15.50:0/720955995 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.291+0000 7f1ed4c1b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ed010b640 0x7f1ed019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecf7fe700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed0106000 0x7f1ed019b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecffff700 1 -- 172.21.15.50:0/720955995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed0106000 msgr2=0x7f1ed019b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecffff700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed0106000 0x7f1ed019b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecffff700 1 -- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ed010b640 msgr2=0x7f1ed019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecffff700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ed010b640 0x7f1ed019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecffff700 1 -- 172.21.15.50:0/720955995 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed01a02f0 con 0x7f1ed0105640 2024-04-26T16:38:08.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecffff700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed019b2e0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000ec10 tx=0x7f1ec000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 10 client.?.objecter ms_handle_connect 0x7f1ed0105640 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000cd40 con 0x7f1ed0105640 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed01a05e0 con 0x7f1ed0105640 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1ec0004480 con 0x7f1ed0105640 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec0010430 con 0x7f1ed0105640 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ed667e700 10 client.41738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed01a7e60 con 0x7f1ed0105640 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ed667e700 1 librados: init done 2024-04-26T16:38:08.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ed006a110 con 0x7f1ed0105640 2024-04-26T16:38:08.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f1ec0003f10 con 0x7f1ed0105640 2024-04-26T16:38:08.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1eb8071f80 0x7f1eb8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(871..871 src has 781..871) ==== 8707+0+0 (secure 0 0 0) 0x7f1ec0091a30 con 0x7f1ed0105640 2024-04-26T16:38:08.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 10 client.41738.objecter ms_dispatch 0x7f1ed0001550 osd_map(871..871 src has 781..871) 2024-04-26T16:38:08.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 3 client.41738.objecter handle_osd_map got epochs [871,871] > 0 2024-04-26T16:38:08.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 3 client.41738.objecter handle_osd_map decoding full epoch 871 2024-04-26T16:38:08.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecf7fe700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1eb8071f80 0x7f1eb8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 20 client.41738.objecter dump_active .. 0 homeless 2024-04-26T16:38:08.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecd7fa700 10 client.41738.objecter ms_handle_connect 0x7f1eb8071f80 2024-04-26T16:38:08.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.295+0000 7f1ecf7fe700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1eb8071f80 0x7f1eb8074440 secure :-1 s=READY pgs=3848 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009810 tx=0x7f1ec4019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:08.302 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.303+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f1ec0016330 con 0x7f1ed0105640 2024-04-26T16:38:08.522 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.519+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1ed019c640 con 0x7f1eb8071f80 2024-04-26T16:38:08.545 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.543+0000 7f1ecd7fa700 1 -- 172.21.15.50:0/720955995 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+426582 (secure 0 0 0) 0x7f1ed019c640 con 0x7f1eb8071f80 2024-04-26T16:38:08.551 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 10 librados: watch_flush enter 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 10 librados: watch_flush exit 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 20 client.41738.objecter shutdown clearing up homeless session... 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 10 client.41738.objecter successfully canceled tick 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1eb8071f80 msgr2=0x7f1eb8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1eb8071f80 0x7f1eb8074440 secure :-1 s=READY pgs=3848 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009810 tx=0x7f1ec4019040 comp rx=0 tx=0).stop 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 msgr2=0x7f1ed019b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed019b2e0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f1ec000ec10 tx=0x7f1ec000ef20 comp rx=0 tx=0).stop 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 shutdown_connections 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1eb8071f80 0x7f1eb8074440 unknown :-1 s=CLOSED pgs=3848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1ed010b640 0x7f1ed019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.554 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1ed0105640 0x7f1ed019b2e0 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 --2- 172.21.15.50:0/720955995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ed0106000 0x7f1ed019b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.547+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 >> 172.21.15.50:0/720955995 conn(0x7f1ed0101130 msgr2=0x7f1ed0101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.551+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 shutdown_connections 2024-04-26T16:38:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.551+0000 7f1ed667e700 1 -- 172.21.15.50:0/720955995 wait complete. 2024-04-26T16:38:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.551+0000 7f1ed667e700 1 librados: shutdown 2024-04-26T16:38:08.555 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:38:08.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:08 smithi050 bash[21463]: audit 2024-04-26T16:38:07.369588+0000 mgr.y (mgr.24401) 791 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:08.848 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:08 smithi050 bash[21463]: audit 2024-04-26T16:38:07.369588+0000 mgr.y (mgr.24401) 791 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:08.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:08 smithi050 bash[29234]: audit 2024-04-26T16:38:07.369588+0000 mgr.y (mgr.24401) 791 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:08.848 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:08 smithi050 bash[29234]: audit 2024-04-26T16:38:07.369588+0000 mgr.y (mgr.24401) 791 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:08.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/1817869639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd71006c590 msgr2=0x7fd710110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- 172.21.15.50:0/1817869639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd71006c590 0x7fd710110310 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7fd70800b0a0 tx=0x7fd70800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/1817869639 shutdown_connections 2024-04-26T16:38:08.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- 172.21.15.50:0/1817869639 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7101094d0 0x7fd7101098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- 172.21.15.50:0/1817869639 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd71006c590 0x7fd710110310 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- 172.21.15.50:0/1817869639 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006bbd0 0x7fd71006c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/1817869639 >> 172.21.15.50:0/1817869639 conn(0x7fd7100793c0 msgr2=0x7fd7100797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/1817869639 shutdown_connections 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/1817869639 wait complete. 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 librados: starting msgr at 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 librados: starting objecter 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 Processor -- start 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- start start 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 librados: setting wanted keys 2024-04-26T16:38:08.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 librados: calling monclient init 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd71006bbd0 0x7fd710102390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006c590 0x7fd7101028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 0x7fd710102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd710112ad0 con 0x7fd7101094d0 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd710112950 con 0x7fd71006bbd0 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd710112c50 con 0x7fd71006c590 2024-04-26T16:38:08.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 0x7fd710102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70dd9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006c590 0x7fd7101028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 0x7fd710102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40398/0 (socket says 172.21.15.50:40398) 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70dd9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006c590 0x7fd7101028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:57204/0 (socket says 172.21.15.50:57204) 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 -- 172.21.15.50:0/4238070144 learned_addr learned my addr 172.21.15.50:0/4238070144 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70e59c700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd71006bbd0 0x7fd710102390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 -- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006c590 msgr2=0x7fd7101028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006c590 0x7fd7101028d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 -- 172.21.15.50:0/4238070144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd71006bbd0 msgr2=0x7fd710102390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd71006bbd0 0x7fd710102390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 -- 172.21.15.50:0/4238070144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7101a7620 con 0x7fd7101094d0 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd70ed9d700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 0x7fd710102e10 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fd70800b780 tx=0x7fd708004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd6ff7fe700 10 client.?.objecter ms_handle_connect 0x7fd7101094d0 2024-04-26T16:38:08.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd6ff7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708021070 con 0x7fd7101094d0 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/4238070144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7101a78b0 con 0x7fd7101094d0 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd708007b80 con 0x7fd7101094d0 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708005040 con 0x7fd7101094d0 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 10 client.31125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 -- 172.21.15.50:0/4238070144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7101a7d10 con 0x7fd7101094d0 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd714c32700 1 librados: init done 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.987+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f4005120 con 0x7fd7101094d0 2024-04-26T16:38:08.992 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd708024070 con 0x7fd7101094d0 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd6f8071f80 0x7fd6f8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(871..871 src has 781..871) ==== 8707+0+0 (secure 0 0 0) 0x7fd7080928c0 con 0x7fd7101094d0 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 10 client.31125.objecter ms_dispatch 0x7fd710001550 osd_map(871..871 src has 781..871) 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd70e59c700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd6f8071f80 0x7fd6f8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 3 client.31125.objecter handle_osd_map got epochs [871,871] > 0 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 3 client.31125.objecter handle_osd_map decoding full epoch 871 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 20 client.31125.objecter dump_active .. 0 homeless 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd6ff7fe700 10 client.31125.objecter ms_handle_connect 0x7fd6f8071f80 2024-04-26T16:38:08.993 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.991+0000 7fd70e59c700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd6f8071f80 0x7fd6f8074440 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7fd700005fd0 tx=0x7fd70000af20 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:08.996 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:08.995+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd70805e510 con 0x7fd7101094d0 2024-04-26T16:38:09.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:08 smithi190 bash[23868]: audit 2024-04-26T16:38:07.369588+0000 mgr.y (mgr.24401) 791 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:09.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:08 smithi190 bash[23868]: audit 2024-04-26T16:38:07.369588+0000 mgr.y (mgr.24401) 791 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:09.226 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.227+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd unset", "key": "noscrub"} v 0) -- 0x7fd6f4005410 con 0x7fd7101094d0 2024-04-26T16:38:09.622 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.619+0000 7fd6ff7fe700 1 -- 172.21.15.50:0/4238070144 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "noscrub"}]=0 noscrub is unset v872) ==== 91+0+0 (secure 0 0 0) 0x7fd7080621c0 con 0x7fd7101094d0 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 10 librados: watch_flush enter 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 10 librados: watch_flush exit 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 20 client.31125.objecter shutdown clearing up homeless session... 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 10 client.31125.objecter successfully canceled tick 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd6f8071f80 msgr2=0x7fd6f8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd6f8071f80 0x7fd6f8074440 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7fd700005fd0 tx=0x7fd70000af20 comp rx=0 tx=0).stop 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 msgr2=0x7fd710102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 0x7fd710102e10 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fd70800b780 tx=0x7fd708004d00 comp rx=0 tx=0).stop 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 shutdown_connections 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd6f8071f80 0x7fd6f8074440 unknown :-1 s=CLOSED pgs=3849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd71006bbd0 0x7fd710102390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:09.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd7101094d0 0x7fd710102e10 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:09.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 --2- 172.21.15.50:0/4238070144 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd71006c590 0x7fd7101028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:09.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 >> 172.21.15.50:0/4238070144 conn(0x7fd7100793c0 msgr2=0x7fd710104fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:09.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.627+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 shutdown_connections 2024-04-26T16:38:09.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.631+0000 7fd6fd7fa700 1 -- 172.21.15.50:0/4238070144 wait complete. 2024-04-26T16:38:09.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:09.631+0000 7fd6fd7fa700 1 librados: shutdown 2024-04-26T16:38:09.634 INFO:teuthology.orchestra.run.smithi050.stderr:noscrub is unset 2024-04-26T16:38:09.823 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1365,"stamp":"2024-04-26T16:38:07.972116+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596512,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3017,"num_read_kb":2863,"num_write":1194,"num_write_kb":2468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1774538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1355,"ondisk_log_size":1355,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":808168,"kb_used_data":47080,"kb_used_omap":34,"kb_used_meta":760989,"kb_avail":749153048,"statfs":{"total":767960285184,"available":767132721152,"internally_reserved":0,"allocated":48209920,"data_stored":25143592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34917,"internal_metadata":779253659},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.545078"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":587,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.896014+0000","last_change":"2024-04-26T16:35:23.593402+0000","last_active":"2024-04-26T16:37:56.896014+0000","last_peered":"2024-04-26T16:37:56.896014+0000","last_clean":"2024-04-26T16:37:56.896014+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:37:56.896014+0000","last_undegraded":"2024-04-26T16:37:56.896014+0000","last_fullsized":"2024-04-26T16:37:56.896014+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:23.593284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:35:23.593284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"747'16","reported_seq":1913,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.036988+0000","last_change":"2024-04-26T16:36:20.159718+0000","last_active":"2024-04-26T16:38:00.036988+0000","last_peered":"2024-04-26T16:38:00.036988+0000","last_clean":"2024-04-26T16:38:00.036988+0000","last_became_active":"2024-04-26T16:35:25.395438+0000","last_became_peered":"2024-04-26T16:35:25.395438+0000","last_unstale":"2024-04-26T16:38:00.036988+0000","last_undegraded":"2024-04-26T16:38:00.036988+0000","last_fullsized":"2024-04-26T16:38:00.036988+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:36:20.159412+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:36:20.159412+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"857'46","reported_seq":1734,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895091+0000","last_change":"2024-04-26T16:35:01.604909+0000","last_active":"2024-04-26T16:37:56.895091+0000","last_peered":"2024-04-26T16:37:56.895091+0000","last_clean":"2024-04-26T16:37:56.895091+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:37:56.895091+0000","last_undegraded":"2024-04-26T16:37:56.895091+0000","last_fullsized":"2024-04-26T16:37:56.895091+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'34","last_scrub_stamp":"2024-04-26T16:35:01.604647+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:35:01.604647+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1672,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.781200+0000","last_change":"2024-04-26T16:36:41.731069+0000","last_active":"2024-04-26T16:37:56.781200+0000","last_peered":"2024-04-26T16:37:56.781200+0000","last_clean":"2024-04-26T16:37:56.781200+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:37:56.781200+0000","last_undegraded":"2024-04-26T16:37:56.781200+0000","last_fullsized":"2024-04-26T16:37:56.781200+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:41.730956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:31.361012+0000","last_clean_scrub_stamp":"2024-04-26T16:36:41.730956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:09.255900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":588,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893984+0000","last_change":"2024-04-26T16:36:29.464449+0000","last_active":"2024-04-26T16:37:56.893984+0000","last_peered":"2024-04-26T16:37:56.893984+0000","last_clean":"2024-04-26T16:37:56.893984+0000","last_became_active":"2024-04-26T16:35:25.382820+0000","last_became_peered":"2024-04-26T16:35:25.382820+0000","last_unstale":"2024-04-26T16:37:56.893984+0000","last_undegraded":"2024-04-26T16:37:56.893984+0000","last_fullsized":"2024-04-26T16:37:56.893984+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:29.464241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:29.464241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1510,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891538+0000","last_change":"2024-04-26T16:36:37.185673+0000","last_active":"2024-04-26T16:37:56.891538+0000","last_peered":"2024-04-26T16:37:56.891538+0000","last_clean":"2024-04-26T16:37:56.891538+0000","last_became_active":"2024-04-26T16:35:25.399977+0000","last_became_peered":"2024-04-26T16:35:25.399977+0000","last_unstale":"2024-04-26T16:37:56.891538+0000","last_undegraded":"2024-04-26T16:37:56.891538+0000","last_fullsized":"2024-04-26T16:37:56.891538+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:37.185574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:37.185574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"857'30","reported_seq":615,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894654+0000","last_change":"2024-04-26T16:36:21.382720+0000","last_active":"2024-04-26T16:37:56.894654+0000","last_peered":"2024-04-26T16:37:56.894654+0000","last_clean":"2024-04-26T16:37:56.894654+0000","last_became_active":"2024-04-26T16:35:25.393291+0000","last_became_peered":"2024-04-26T16:35:25.393291+0000","last_unstale":"2024-04-26T16:37:56.894654+0000","last_undegraded":"2024-04-26T16:37:56.894654+0000","last_fullsized":"2024-04-26T16:37:56.894654+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:36:21.382342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:21.382342+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1676,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432747+0000","last_change":"2024-04-26T16:36:49.190188+0000","last_active":"2024-04-26T16:37:56.432747+0000","last_peered":"2024-04-26T16:37:56.432747+0000","last_clean":"2024-04-26T16:37:56.432747+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:37:56.432747+0000","last_undegraded":"2024-04-26T16:37:56.432747+0000","last_fullsized":"2024-04-26T16:37:56.432747+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:49.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:49.190081+0000","last_clean_scrub_stamp":"2024-04-26T16:36:49.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:08.576305+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1672,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.780299+0000","last_change":"2024-04-26T16:36:45.742872+0000","last_active":"2024-04-26T16:37:56.780299+0000","last_peered":"2024-04-26T16:37:56.780299+0000","last_clean":"2024-04-26T16:37:56.780299+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:37:56.780299+0000","last_undegraded":"2024-04-26T16:37:56.780299+0000","last_fullsized":"2024-04-26T16:37:56.780299+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:45.741598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:45.741598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:13.348472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1668,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892435+0000","last_change":"2024-04-26T16:36:54.962722+0000","last_active":"2024-04-26T16:37:56.892435+0000","last_peered":"2024-04-26T16:37:56.892435+0000","last_clean":"2024-04-26T16:37:56.892435+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:37:56.892435+0000","last_undegraded":"2024-04-26T16:37:56.892435+0000","last_fullsized":"2024-04-26T16:37:56.892435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:54.962598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:36:54.962598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:09.700942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1530,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892168+0000","last_change":"2024-04-26T16:36:35.211569+0000","last_active":"2024-04-26T16:37:56.892168+0000","last_peered":"2024-04-26T16:37:56.892168+0000","last_clean":"2024-04-26T16:37:56.892168+0000","last_became_active":"2024-04-26T16:35:25.385146+0000","last_became_peered":"2024-04-26T16:35:25.385146+0000","last_unstale":"2024-04-26T16:37:56.892168+0000","last_undegraded":"2024-04-26T16:37:56.892168+0000","last_fullsized":"2024-04-26T16:37:56.892168+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:35.211444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:35.211444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"857'53","reported_seq":1788,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893973+0000","last_change":"2024-04-26T16:36:42.185671+0000","last_active":"2024-04-26T16:37:56.893973+0000","last_peered":"2024-04-26T16:37:56.893973+0000","last_clean":"2024-04-26T16:37:56.893973+0000","last_became_active":"2024-04-26T16:35:25.387497+0000","last_became_peered":"2024-04-26T16:35:25.387497+0000","last_unstale":"2024-04-26T16:37:56.893973+0000","last_undegraded":"2024-04-26T16:37:56.893973+0000","last_fullsized":"2024-04-26T16:37:56.893973+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'39","last_scrub_stamp":"2024-04-26T16:36:42.185411+0000","last_deep_scrub":"612'39","last_deep_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_clean_scrub_stamp":"2024-04-26T16:36:42.185411+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:01.984483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1692,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.781505+0000","last_change":"2024-04-26T16:36:33.751840+0000","last_active":"2024-04-26T16:37:56.781505+0000","last_peered":"2024-04-26T16:37:56.781505+0000","last_clean":"2024-04-26T16:37:56.781505+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:37:56.781505+0000","last_undegraded":"2024-04-26T16:37:56.781505+0000","last_fullsized":"2024-04-26T16:37:56.781505+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:33.751640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.751640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1684,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.780874+0000","last_change":"2024-04-26T16:36:30.745608+0000","last_active":"2024-04-26T16:37:56.780874+0000","last_peered":"2024-04-26T16:37:56.780874+0000","last_clean":"2024-04-26T16:37:56.780874+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:37:56.780874+0000","last_undegraded":"2024-04-26T16:37:56.780874+0000","last_fullsized":"2024-04-26T16:37:56.780874+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:30.745352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:30.745352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:57.916045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1636,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:59.109012+0000","last_change":"2024-04-26T16:37:59.109107+0000","last_active":"2024-04-26T16:37:59.109012+0000","last_peered":"2024-04-26T16:37:59.109012+0000","last_clean":"2024-04-26T16:37:59.109012+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:37:59.109012+0000","last_undegraded":"2024-04-26T16:37:59.109012+0000","last_fullsized":"2024-04-26T16:37:59.109012+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:59.108820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.108820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.334820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1524,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.781432+0000","last_change":"2024-04-26T16:36:53.546706+0000","last_active":"2024-04-26T16:37:56.781432+0000","last_peered":"2024-04-26T16:37:56.781432+0000","last_clean":"2024-04-26T16:37:56.781432+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:37:56.781432+0000","last_undegraded":"2024-04-26T16:37:56.781432+0000","last_fullsized":"2024-04-26T16:37:56.781432+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:53.546496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:24.770536+0000","last_clean_scrub_stamp":"2024-04-26T16:36:53.546496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:16.703396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"857'51","reported_seq":1423,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893216+0000","last_change":"2024-04-26T16:37:48.340319+0000","last_active":"2024-04-26T16:37:56.893216+0000","last_peered":"2024-04-26T16:37:56.893216+0000","last_clean":"2024-04-26T16:37:56.893216+0000","last_became_active":"2024-04-26T16:35:25.414489+0000","last_became_peered":"2024-04-26T16:35:25.414489+0000","last_unstale":"2024-04-26T16:37:56.893216+0000","last_undegraded":"2024-04-26T16:37:56.893216+0000","last_fullsized":"2024-04-26T16:37:56.893216+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'51","last_scrub_stamp":"2024-04-26T16:37:48.340220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.340220+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:54.505895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1660,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895601+0000","last_change":"2024-04-26T16:36:50.250362+0000","last_active":"2024-04-26T16:37:56.895601+0000","last_peered":"2024-04-26T16:37:56.895601+0000","last_clean":"2024-04-26T16:37:56.895601+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:37:56.895601+0000","last_undegraded":"2024-04-26T16:37:56.895601+0000","last_fullsized":"2024-04-26T16:37:56.895601+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:50.250252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:36:50.250252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:00.232345+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1676,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.429437+0000","last_change":"2024-04-26T16:36:36.247286+0000","last_active":"2024-04-26T16:37:56.429437+0000","last_peered":"2024-04-26T16:37:56.429437+0000","last_clean":"2024-04-26T16:37:56.429437+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:37:56.429437+0000","last_undegraded":"2024-04-26T16:37:56.429437+0000","last_fullsized":"2024-04-26T16:37:56.429437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:36.246978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:36:36.246978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:00.680828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1668,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.782250+0000","last_change":"2024-04-26T16:36:48.691806+0000","last_active":"2024-04-26T16:37:56.782250+0000","last_peered":"2024-04-26T16:37:56.782250+0000","last_clean":"2024-04-26T16:37:56.782250+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:37:56.782250+0000","last_undegraded":"2024-04-26T16:37:56.782250+0000","last_fullsized":"2024-04-26T16:37:56.782250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.691439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:16.826623+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.691439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:13.790967+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1147,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894052+0000","last_change":"2024-04-26T16:37:50.334215+0000","last_active":"2024-04-26T16:37:56.894052+0000","last_peered":"2024-04-26T16:37:56.894052+0000","last_clean":"2024-04-26T16:37:56.894052+0000","last_became_active":"2024-04-26T16:35:25.388195+0000","last_became_peered":"2024-04-26T16:35:25.388195+0000","last_unstale":"2024-04-26T16:37:56.894052+0000","last_undegraded":"2024-04-26T16:37:56.894052+0000","last_fullsized":"2024-04-26T16:37:56.894052+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:50.333993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:50.333993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.406018+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"857'25","reported_seq":1690,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431660+0000","last_change":"2024-04-26T16:36:53.706447+0000","last_active":"2024-04-26T16:37:56.431660+0000","last_peered":"2024-04-26T16:37:56.431660+0000","last_clean":"2024-04-26T16:37:56.431660+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:37:56.431660+0000","last_undegraded":"2024-04-26T16:37:56.431660+0000","last_fullsized":"2024-04-26T16:37:56.431660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:36:53.706307+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:36:53.706307+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:04.494613+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1676,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432532+0000","last_change":"2024-04-26T16:36:18.156927+0000","last_active":"2024-04-26T16:37:56.432532+0000","last_peered":"2024-04-26T16:37:56.432532+0000","last_clean":"2024-04-26T16:37:56.432532+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:37:56.432532+0000","last_undegraded":"2024-04-26T16:37:56.432532+0000","last_fullsized":"2024-04-26T16:37:56.432532+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:18.156818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:49.416438+0000","last_clean_scrub_stamp":"2024-04-26T16:36:18.156818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":1958,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.436674+0000","last_change":"2024-04-26T16:38:00.436674+0000","last_active":"2024-04-26T16:38:00.436674+0000","last_peered":"2024-04-26T16:38:00.436674+0000","last_clean":"2024-04-26T16:38:00.436674+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:38:00.436674+0000","last_undegraded":"2024-04-26T16:38:00.436674+0000","last_fullsized":"2024-04-26T16:38:00.436674+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:38:00.436448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.436448+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:28.478392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1676,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.784779+0000","last_change":"2024-04-26T16:37:56.784808+0000","last_active":"2024-04-26T16:37:56.784779+0000","last_peered":"2024-04-26T16:37:56.784779+0000","last_clean":"2024-04-26T16:37:56.784779+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:37:56.784779+0000","last_undegraded":"2024-04-26T16:37:56.784779+0000","last_fullsized":"2024-04-26T16:37:56.784779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.784687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.893722+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.784687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.270962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1526,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.430123+0000","last_change":"2024-04-26T16:37:56.159369+0000","last_active":"2024-04-26T16:37:56.430123+0000","last_peered":"2024-04-26T16:37:56.430123+0000","last_clean":"2024-04-26T16:37:56.430123+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:37:56.430123+0000","last_undegraded":"2024-04-26T16:37:56.430123+0000","last_fullsized":"2024-04-26T16:37:56.430123+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.159158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.823331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"857'46","reported_seq":1720,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.428150+0000","last_change":"2024-04-26T16:36:45.647969+0000","last_active":"2024-04-26T16:37:56.428150+0000","last_peered":"2024-04-26T16:37:56.428150+0000","last_clean":"2024-04-26T16:37:56.428150+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:37:56.428150+0000","last_undegraded":"2024-04-26T16:37:56.428150+0000","last_fullsized":"2024-04-26T16:37:56.428150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"815'34","last_scrub_stamp":"2024-04-26T16:36:45.647529+0000","last_deep_scrub":"815'34","last_deep_scrub_stamp":"2024-04-26T16:36:45.647529+0000","last_clean_scrub_stamp":"2024-04-26T16:36:45.647529+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1690,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.429052+0000","last_change":"2024-04-26T16:36:39.152888+0000","last_active":"2024-04-26T16:37:56.429052+0000","last_peered":"2024-04-26T16:37:56.429052+0000","last_clean":"2024-04-26T16:37:56.429052+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:37:56.429052+0000","last_undegraded":"2024-04-26T16:37:56.429052+0000","last_fullsized":"2024-04-26T16:37:56.429052+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:39.152743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:36:39.152743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:02.162935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":1951,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.036886+0000","last_change":"2024-04-26T16:37:47.846639+0000","last_active":"2024-04-26T16:38:00.036886+0000","last_peered":"2024-04-26T16:38:00.036886+0000","last_clean":"2024-04-26T16:38:00.036886+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:38:00.036886+0000","last_undegraded":"2024-04-26T16:38:00.036886+0000","last_fullsized":"2024-04-26T16:38:00.036886+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:37:47.846487+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.846487+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:51.693727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":587,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894317+0000","last_change":"2024-04-26T16:35:24.556605+0000","last_active":"2024-04-26T16:37:56.894317+0000","last_peered":"2024-04-26T16:37:56.894317+0000","last_clean":"2024-04-26T16:37:56.894317+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:37:56.894317+0000","last_undegraded":"2024-04-26T16:37:56.894317+0000","last_fullsized":"2024-04-26T16:37:56.894317+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:24.556379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:24.556379+0000","last_clean_scrub_stamp":"2024-04-26T16:35:24.556379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":1160,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893068+0000","last_change":"2024-04-26T16:36:47.447443+0000","last_active":"2024-04-26T16:37:56.893068+0000","last_peered":"2024-04-26T16:37:56.893068+0000","last_clean":"2024-04-26T16:37:56.893068+0000","last_became_active":"2024-04-26T16:35:25.407637+0000","last_became_peered":"2024-04-26T16:35:25.407637+0000","last_unstale":"2024-04-26T16:37:56.893068+0000","last_undegraded":"2024-04-26T16:37:56.893068+0000","last_fullsized":"2024-04-26T16:37:56.893068+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:36:47.447295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:47.447295+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:11.024769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"856'29","reported_seq":1714,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.430779+0000","last_change":"2024-04-26T16:36:53.947882+0000","last_active":"2024-04-26T16:37:56.430779+0000","last_peered":"2024-04-26T16:37:56.430779+0000","last_clean":"2024-04-26T16:37:56.430779+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:37:56.430779+0000","last_undegraded":"2024-04-26T16:37:56.430779+0000","last_fullsized":"2024-04-26T16:37:56.430779+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"824'23","last_scrub_stamp":"2024-04-26T16:36:53.947636+0000","last_deep_scrub":"824'23","last_deep_scrub_stamp":"2024-04-26T16:36:53.947636+0000","last_clean_scrub_stamp":"2024-04-26T16:36:53.947636+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:01.510567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1664,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:03.141138+0000","last_change":"2024-04-26T16:38:03.141298+0000","last_active":"2024-04-26T16:38:03.141138+0000","last_peered":"2024-04-26T16:38:03.141138+0000","last_clean":"2024-04-26T16:38:03.141138+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:38:03.141138+0000","last_undegraded":"2024-04-26T16:38:03.141138+0000","last_fullsized":"2024-04-26T16:38:03.141138+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:38:03.140926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.140926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.283756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":588,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.896481+0000","last_change":"2024-04-26T16:36:13.340444+0000","last_active":"2024-04-26T16:37:56.896481+0000","last_peered":"2024-04-26T16:37:56.896481+0000","last_clean":"2024-04-26T16:37:56.896481+0000","last_became_active":"2024-04-26T16:35:25.382698+0000","last_became_peered":"2024-04-26T16:35:25.382698+0000","last_unstale":"2024-04-26T16:37:56.896481+0000","last_undegraded":"2024-04-26T16:37:56.896481+0000","last_fullsized":"2024-04-26T16:37:56.896481+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:13.340230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:36:13.340230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1666,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.433883+0000","last_change":"2024-04-26T16:37:54.772260+0000","last_active":"2024-04-26T16:37:56.433883+0000","last_peered":"2024-04-26T16:37:56.433883+0000","last_clean":"2024-04-26T16:37:56.433883+0000","last_became_active":"2024-04-26T16:35:25.416938+0000","last_became_peered":"2024-04-26T16:35:25.416938+0000","last_unstale":"2024-04-26T16:37:56.433883+0000","last_undegraded":"2024-04-26T16:37:56.433883+0000","last_fullsized":"2024-04-26T16:37:56.433883+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:54.772049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:54.772049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.353649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1529,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:58.725228+0000","last_change":"2024-04-26T16:37:58.725326+0000","last_active":"2024-04-26T16:37:58.725228+0000","last_peered":"2024-04-26T16:37:58.725228+0000","last_clean":"2024-04-26T16:37:58.725228+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:37:58.725228+0000","last_undegraded":"2024-04-26T16:37:58.725228+0000","last_fullsized":"2024-04-26T16:37:58.725228+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.725080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.725080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.633294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1908,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.035735+0000","last_change":"2024-04-26T16:37:47.750495+0000","last_active":"2024-04-26T16:38:00.035735+0000","last_peered":"2024-04-26T16:38:00.035735+0000","last_clean":"2024-04-26T16:38:00.035735+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:38:00.035735+0000","last_undegraded":"2024-04-26T16:38:00.035735+0000","last_fullsized":"2024-04-26T16:38:00.035735+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:37:47.750268+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.750268+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.703331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1703,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431082+0000","last_change":"2024-04-26T16:37:52.815375+0000","last_active":"2024-04-26T16:37:56.431082+0000","last_peered":"2024-04-26T16:37:56.431082+0000","last_clean":"2024-04-26T16:37:56.431082+0000","last_became_active":"2024-04-26T16:35:25.411443+0000","last_became_peered":"2024-04-26T16:35:25.411443+0000","last_unstale":"2024-04-26T16:37:56.431082+0000","last_undegraded":"2024-04-26T16:37:56.431082+0000","last_fullsized":"2024-04-26T16:37:56.431082+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.815039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.815039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:12.392298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"857'42","reported_seq":1736,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:05.806340+0000","last_change":"2024-04-26T16:38:05.806435+0000","last_active":"2024-04-26T16:38:05.806340+0000","last_peered":"2024-04-26T16:38:05.806340+0000","last_clean":"2024-04-26T16:38:05.806340+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:38:05.806340+0000","last_undegraded":"2024-04-26T16:38:05.806340+0000","last_fullsized":"2024-04-26T16:38:05.806340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'42","last_scrub_stamp":"2024-04-26T16:38:05.806187+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.806187+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:34.319055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1271,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894605+0000","last_change":"2024-04-26T16:37:06.357605+0000","last_active":"2024-04-26T16:37:56.894605+0000","last_peered":"2024-04-26T16:37:56.894605+0000","last_clean":"2024-04-26T16:37:56.894605+0000","last_became_active":"2024-04-26T16:35:25.406996+0000","last_became_peered":"2024-04-26T16:35:25.406996+0000","last_unstale":"2024-04-26T16:37:56.894605+0000","last_undegraded":"2024-04-26T16:37:56.894605+0000","last_fullsized":"2024-04-26T16:37:56.894605+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:06.357237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:37:06.357237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:29.140874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1654,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892206+0000","last_change":"2024-04-26T16:36:39.157157+0000","last_active":"2024-04-26T16:37:56.892206+0000","last_peered":"2024-04-26T16:37:56.892206+0000","last_clean":"2024-04-26T16:37:56.892206+0000","last_became_active":"2024-04-26T16:35:25.391371+0000","last_became_peered":"2024-04-26T16:35:25.391371+0000","last_unstale":"2024-04-26T16:37:56.892206+0000","last_undegraded":"2024-04-26T16:37:56.892206+0000","last_fullsized":"2024-04-26T16:37:56.892206+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:39.156939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:39.156939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:05.668779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1499,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895395+0000","last_change":"2024-04-26T16:37:01.201260+0000","last_active":"2024-04-26T16:37:56.895395+0000","last_peered":"2024-04-26T16:37:56.895395+0000","last_clean":"2024-04-26T16:37:56.895395+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:37:56.895395+0000","last_undegraded":"2024-04-26T16:37:56.895395+0000","last_fullsized":"2024-04-26T16:37:56.895395+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.200920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.200920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.711340+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1672,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.430759+0000","last_change":"2024-04-26T16:36:32.020307+0000","last_active":"2024-04-26T16:37:56.430759+0000","last_peered":"2024-04-26T16:37:56.430759+0000","last_clean":"2024-04-26T16:37:56.430759+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:37:56.430759+0000","last_undegraded":"2024-04-26T16:37:56.430759+0000","last_fullsized":"2024-04-26T16:37:56.430759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:32.020098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:36:32.020098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:57.305628+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":2875,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:04.764450+0000","last_change":"2024-04-26T16:37:01.745491+0000","last_active":"2024-04-26T16:38:04.764450+0000","last_peered":"2024-04-26T16:38:04.764450+0000","last_clean":"2024-04-26T16:38:04.764450+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:38:04.764450+0000","last_undegraded":"2024-04-26T16:38:04.764450+0000","last_fullsized":"2024-04-26T16:38:04.764450+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:37:01.745268+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.745268+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.918491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1214,"num_read_kb":1209,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"857'26","reported_seq":1696,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432147+0000","last_change":"2024-04-26T16:37:03.724692+0000","last_active":"2024-04-26T16:37:56.432147+0000","last_peered":"2024-04-26T16:37:56.432147+0000","last_clean":"2024-04-26T16:37:56.432147+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:37:56.432147+0000","last_undegraded":"2024-04-26T16:37:56.432147+0000","last_fullsized":"2024-04-26T16:37:56.432147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:37:03.724436+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:37:03.724436+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:28.814780+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1696,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:04.165503+0000","last_change":"2024-04-26T16:38:04.165569+0000","last_active":"2024-04-26T16:38:04.165503+0000","last_peered":"2024-04-26T16:38:04.165503+0000","last_clean":"2024-04-26T16:38:04.165503+0000","last_became_active":"2024-04-26T16:35:25.398461+0000","last_became_peered":"2024-04-26T16:35:25.398461+0000","last_unstale":"2024-04-26T16:38:04.165503+0000","last_undegraded":"2024-04-26T16:38:04.165503+0000","last_fullsized":"2024-04-26T16:38:04.165503+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:38:04.165359+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.165359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.874714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1668,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431002+0000","last_change":"2024-04-26T16:36:57.977447+0000","last_active":"2024-04-26T16:37:56.431002+0000","last_peered":"2024-04-26T16:37:56.431002+0000","last_clean":"2024-04-26T16:37:56.431002+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:37:56.431002+0000","last_undegraded":"2024-04-26T16:37:56.431002+0000","last_fullsized":"2024-04-26T16:37:56.431002+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:57.977272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.977272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:25.432898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1517,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.428637+0000","last_change":"2024-04-26T16:36:33.244989+0000","last_active":"2024-04-26T16:37:56.428637+0000","last_peered":"2024-04-26T16:37:56.428637+0000","last_clean":"2024-04-26T16:37:56.428637+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:37:56.428637+0000","last_undegraded":"2024-04-26T16:37:56.428637+0000","last_fullsized":"2024-04-26T16:37:56.428637+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:33.244849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.244849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1265,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:02.229413+0000","last_change":"2024-04-26T16:38:02.229476+0000","last_active":"2024-04-26T16:38:02.229413+0000","last_peered":"2024-04-26T16:38:02.229413+0000","last_clean":"2024-04-26T16:38:02.229413+0000","last_became_active":"2024-04-26T16:35:25.404395+0000","last_became_peered":"2024-04-26T16:35:25.404395+0000","last_unstale":"2024-04-26T16:38:02.229413+0000","last_undegraded":"2024-04-26T16:38:02.229413+0000","last_fullsized":"2024-04-26T16:38:02.229413+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.229233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.229233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.152859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"857'66","reported_seq":2080,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:03.160319+0000","last_change":"2024-04-26T16:38:03.160319+0000","last_active":"2024-04-26T16:38:03.160319+0000","last_peered":"2024-04-26T16:38:03.160319+0000","last_clean":"2024-04-26T16:38:03.160319+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:38:03.160319+0000","last_undegraded":"2024-04-26T16:38:03.160319+0000","last_fullsized":"2024-04-26T16:38:03.160319+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'66","last_scrub_stamp":"2024-04-26T16:38:03.160063+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.160063+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.536394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1657,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891810+0000","last_change":"2024-04-26T16:35:08.450261+0000","last_active":"2024-04-26T16:37:56.891810+0000","last_peered":"2024-04-26T16:37:56.891810+0000","last_clean":"2024-04-26T16:37:56.891810+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:37:56.891810+0000","last_undegraded":"2024-04-26T16:37:56.891810+0000","last_fullsized":"2024-04-26T16:37:56.891810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:35:08.450018+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:35:08.450018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1682,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:02.125616+0000","last_change":"2024-04-26T16:38:02.125681+0000","last_active":"2024-04-26T16:38:02.125616+0000","last_peered":"2024-04-26T16:38:02.125616+0000","last_clean":"2024-04-26T16:38:02.125616+0000","last_became_active":"2024-04-26T16:35:25.411888+0000","last_became_peered":"2024-04-26T16:35:25.411888+0000","last_unstale":"2024-04-26T16:38:02.125616+0000","last_undegraded":"2024-04-26T16:38:02.125616+0000","last_fullsized":"2024-04-26T16:38:02.125616+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.125415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.125415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.366125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1509,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:57.743671+0000","last_change":"2024-04-26T16:37:57.743728+0000","last_active":"2024-04-26T16:37:57.743671+0000","last_peered":"2024-04-26T16:37:57.743671+0000","last_clean":"2024-04-26T16:37:57.743671+0000","last_became_active":"2024-04-26T16:35:25.417017+0000","last_became_peered":"2024-04-26T16:35:25.417017+0000","last_unstale":"2024-04-26T16:37:57.743671+0000","last_undegraded":"2024-04-26T16:37:57.743671+0000","last_fullsized":"2024-04-26T16:37:57.743671+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:57.743476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.743476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.056336+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:58.133138+0000","last_change":"2024-04-26T16:37:58.133197+0000","last_active":"2024-04-26T16:37:58.133138+0000","last_peered":"2024-04-26T16:37:58.133138+0000","last_clean":"2024-04-26T16:37:58.133138+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:37:58.133138+0000","last_undegraded":"2024-04-26T16:37:58.133138+0000","last_fullsized":"2024-04-26T16:37:58.133138+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.132987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.132987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:00.348823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1649,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891060+0000","last_change":"2024-04-26T16:35:06.534153+0000","last_active":"2024-04-26T16:37:56.891060+0000","last_peered":"2024-04-26T16:37:56.891060+0000","last_clean":"2024-04-26T16:37:56.891060+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:37:56.891060+0000","last_undegraded":"2024-04-26T16:37:56.891060+0000","last_fullsized":"2024-04-26T16:37:56.891060+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:06.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:35:06.533840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"857'49","reported_seq":1738,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891772+0000","last_change":"2024-04-26T16:36:58.920449+0000","last_active":"2024-04-26T16:37:56.891772+0000","last_peered":"2024-04-26T16:37:56.891772+0000","last_clean":"2024-04-26T16:37:56.891772+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:37:56.891772+0000","last_undegraded":"2024-04-26T16:37:56.891772+0000","last_fullsized":"2024-04-26T16:37:56.891772+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:36:58.920294+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:36:58.920294+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:18.593619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1685,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.429437+0000","last_change":"2024-04-26T16:37:05.007705+0000","last_active":"2024-04-26T16:37:56.429437+0000","last_peered":"2024-04-26T16:37:56.429437+0000","last_clean":"2024-04-26T16:37:56.429437+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:37:56.429437+0000","last_undegraded":"2024-04-26T16:37:56.429437+0000","last_fullsized":"2024-04-26T16:37:56.429437+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:05.007579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:37:05.007579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.021319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1529,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:05.712690+0000","last_change":"2024-04-26T16:38:05.712753+0000","last_active":"2024-04-26T16:38:05.712690+0000","last_peered":"2024-04-26T16:38:05.712690+0000","last_clean":"2024-04-26T16:38:05.712690+0000","last_became_active":"2024-04-26T16:35:25.417111+0000","last_became_peered":"2024-04-26T16:35:25.417111+0000","last_unstale":"2024-04-26T16:38:05.712690+0000","last_undegraded":"2024-04-26T16:38:05.712690+0000","last_fullsized":"2024-04-26T16:38:05.712690+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:05.712520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.712520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.895535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1652,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891959+0000","last_change":"2024-04-26T16:35:14.526928+0000","last_active":"2024-04-26T16:37:56.891959+0000","last_peered":"2024-04-26T16:37:56.891959+0000","last_clean":"2024-04-26T16:37:56.891959+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:37:56.891959+0000","last_undegraded":"2024-04-26T16:37:56.891959+0000","last_fullsized":"2024-04-26T16:37:56.891959+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:14.526731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:35:14.526731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1947,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891096+0000","last_change":"2024-04-26T16:36:25.188281+0000","last_active":"2024-04-26T16:37:56.891096+0000","last_peered":"2024-04-26T16:37:56.891096+0000","last_clean":"2024-04-26T16:37:56.891096+0000","last_became_active":"2024-04-26T16:35:25.394870+0000","last_became_peered":"2024-04-26T16:35:25.394870+0000","last_unstale":"2024-04-26T16:37:56.891096+0000","last_undegraded":"2024-04-26T16:37:56.891096+0000","last_fullsized":"2024-04-26T16:37:56.891096+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:36:25.187951+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:36:25.187951+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:57.118465+0000","last_change":"2024-04-26T16:37:57.118532+0000","last_active":"2024-04-26T16:37:57.118465+0000","last_peered":"2024-04-26T16:37:57.118465+0000","last_clean":"2024-04-26T16:37:57.118465+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:37:57.118465+0000","last_undegraded":"2024-04-26T16:37:57.118465+0000","last_fullsized":"2024-04-26T16:37:57.118465+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:57.118297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.118297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.113285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"857'35","reported_seq":1748,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431340+0000","last_change":"2024-04-26T16:36:35.656622+0000","last_active":"2024-04-26T16:37:56.431340+0000","last_peered":"2024-04-26T16:37:56.431340+0000","last_clean":"2024-04-26T16:37:56.431340+0000","last_became_active":"2024-04-26T16:35:25.397667+0000","last_became_peered":"2024-04-26T16:35:25.397667+0000","last_unstale":"2024-04-26T16:37:56.431340+0000","last_undegraded":"2024-04-26T16:37:56.431340+0000","last_fullsized":"2024-04-26T16:37:56.431340+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'25","last_scrub_stamp":"2024-04-26T16:36:35.656379+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:36:35.656379+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.896339+0000","last_change":"2024-04-26T16:37:02.918875+0000","last_active":"2024-04-26T16:37:56.896339+0000","last_peered":"2024-04-26T16:37:56.896339+0000","last_clean":"2024-04-26T16:37:56.896339+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:37:56.896339+0000","last_undegraded":"2024-04-26T16:37:56.896339+0000","last_fullsized":"2024-04-26T16:37:56.896339+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:02.918675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:37:02.918675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:21.709848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1529,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.427305+0000","last_change":"2024-04-26T16:37:42.746944+0000","last_active":"2024-04-26T16:37:56.427305+0000","last_peered":"2024-04-26T16:37:56.427305+0000","last_clean":"2024-04-26T16:37:56.427305+0000","last_became_active":"2024-04-26T16:35:25.392163+0000","last_became_peered":"2024-04-26T16:35:25.392163+0000","last_unstale":"2024-04-26T16:37:56.427305+0000","last_undegraded":"2024-04-26T16:37:56.427305+0000","last_fullsized":"2024-04-26T16:37:56.427305+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:42.746793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:42.746793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.906091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1684,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.781973+0000","last_change":"2024-04-26T16:36:52.586910+0000","last_active":"2024-04-26T16:37:56.781973+0000","last_peered":"2024-04-26T16:37:56.781973+0000","last_clean":"2024-04-26T16:37:56.781973+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:37:56.781973+0000","last_undegraded":"2024-04-26T16:37:56.781973+0000","last_fullsized":"2024-04-26T16:37:56.781973+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:52.586701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:36:52.586701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:09.278861+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1680,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.780095+0000","last_change":"2024-04-26T16:36:39.720640+0000","last_active":"2024-04-26T16:37:56.780095+0000","last_peered":"2024-04-26T16:37:56.780095+0000","last_clean":"2024-04-26T16:37:56.780095+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:37:56.780095+0000","last_undegraded":"2024-04-26T16:37:56.780095+0000","last_fullsized":"2024-04-26T16:37:56.780095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:39.720507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:36:39.720507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"857'50","reported_seq":1775,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.429957+0000","last_change":"2024-04-26T16:37:54.196331+0000","last_active":"2024-04-26T16:37:56.429957+0000","last_peered":"2024-04-26T16:37:56.429957+0000","last_clean":"2024-04-26T16:37:56.429957+0000","last_became_active":"2024-04-26T16:35:25.412117+0000","last_became_peered":"2024-04-26T16:35:25.412117+0000","last_unstale":"2024-04-26T16:37:56.429957+0000","last_undegraded":"2024-04-26T16:37:56.429957+0000","last_fullsized":"2024-04-26T16:37:56.429957+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'50","last_scrub_stamp":"2024-04-26T16:37:54.196211+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_clean_scrub_stamp":"2024-04-26T16:37:54.196211+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.629419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1646,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891668+0000","last_change":"2024-04-26T16:36:13.193711+0000","last_active":"2024-04-26T16:37:56.891668+0000","last_peered":"2024-04-26T16:37:56.891668+0000","last_clean":"2024-04-26T16:37:56.891668+0000","last_became_active":"2024-04-26T16:35:25.398515+0000","last_became_peered":"2024-04-26T16:35:25.398515+0000","last_unstale":"2024-04-26T16:37:56.891668+0000","last_undegraded":"2024-04-26T16:37:56.891668+0000","last_fullsized":"2024-04-26T16:37:56.891668+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:13.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:13.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1499,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431185+0000","last_change":"2024-04-26T16:36:38.645309+0000","last_active":"2024-04-26T16:37:56.431185+0000","last_peered":"2024-04-26T16:37:56.431185+0000","last_clean":"2024-04-26T16:37:56.431185+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:37:56.431185+0000","last_undegraded":"2024-04-26T16:37:56.431185+0000","last_fullsized":"2024-04-26T16:37:56.431185+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:38.645057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:38.645057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1664,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894485+0000","last_change":"2024-04-26T16:36:48.923637+0000","last_active":"2024-04-26T16:37:56.894485+0000","last_peered":"2024-04-26T16:37:56.894485+0000","last_clean":"2024-04-26T16:37:56.894485+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:37:56.894485+0000","last_undegraded":"2024-04-26T16:37:56.894485+0000","last_fullsized":"2024-04-26T16:37:56.894485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.923527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.923527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:14.720801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"857'47","reported_seq":1760,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432740+0000","last_change":"2024-04-26T16:37:55.775734+0000","last_active":"2024-04-26T16:37:56.432740+0000","last_peered":"2024-04-26T16:37:56.432740+0000","last_clean":"2024-04-26T16:37:56.432740+0000","last_became_active":"2024-04-26T16:35:25.395746+0000","last_became_peered":"2024-04-26T16:35:25.395746+0000","last_unstale":"2024-04-26T16:37:56.432740+0000","last_undegraded":"2024-04-26T16:37:56.432740+0000","last_fullsized":"2024-04-26T16:37:56.432740+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'47","last_scrub_stamp":"2024-04-26T16:37:55.775586+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:37:55.775586+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.155493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":588,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894255+0000","last_change":"2024-04-26T16:36:32.514466+0000","last_active":"2024-04-26T16:37:56.894255+0000","last_peered":"2024-04-26T16:37:56.894255+0000","last_clean":"2024-04-26T16:37:56.894255+0000","last_became_active":"2024-04-26T16:35:25.397940+0000","last_became_peered":"2024-04-26T16:35:25.397940+0000","last_unstale":"2024-04-26T16:37:56.894255+0000","last_undegraded":"2024-04-26T16:37:56.894255+0000","last_fullsized":"2024-04-26T16:37:56.894255+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:32.514129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:36:32.514129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1652,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432751+0000","last_change":"2024-04-26T16:37:46.742109+0000","last_active":"2024-04-26T16:37:56.432751+0000","last_peered":"2024-04-26T16:37:56.432751+0000","last_clean":"2024-04-26T16:37:56.432751+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:37:56.432751+0000","last_undegraded":"2024-04-26T16:37:56.432751+0000","last_fullsized":"2024-04-26T16:37:56.432751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:46.742012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:46.742012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.964801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1525,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.781230+0000","last_change":"2024-04-26T16:36:59.476993+0000","last_active":"2024-04-26T16:37:56.781230+0000","last_peered":"2024-04-26T16:37:56.781230+0000","last_clean":"2024-04-26T16:37:56.781230+0000","last_became_active":"2024-04-26T16:35:25.390196+0000","last_became_peered":"2024-04-26T16:35:25.390196+0000","last_unstale":"2024-04-26T16:37:56.781230+0000","last_undegraded":"2024-04-26T16:37:56.781230+0000","last_fullsized":"2024-04-26T16:37:56.781230+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:59.476833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.476833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:15.110881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"747'19","reported_seq":1961,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.035752+0000","last_change":"2024-04-26T16:37:57.836348+0000","last_active":"2024-04-26T16:38:00.035752+0000","last_peered":"2024-04-26T16:38:00.035752+0000","last_clean":"2024-04-26T16:38:00.035752+0000","last_became_active":"2024-04-26T16:35:25.404413+0000","last_became_peered":"2024-04-26T16:35:25.404413+0000","last_unstale":"2024-04-26T16:38:00.035752+0000","last_undegraded":"2024-04-26T16:38:00.035752+0000","last_fullsized":"2024-04-26T16:38:00.035752+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'19","last_scrub_stamp":"2024-04-26T16:37:57.836074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.836074+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:59.025050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1707,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.433151+0000","last_change":"2024-04-26T16:37:48.275884+0000","last_active":"2024-04-26T16:37:56.433151+0000","last_peered":"2024-04-26T16:37:56.433151+0000","last_clean":"2024-04-26T16:37:56.433151+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:37:56.433151+0000","last_undegraded":"2024-04-26T16:37:56.433151+0000","last_fullsized":"2024-04-26T16:37:56.433151+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:37:48.275781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.275781+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:52.249427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"857'52","reported_seq":695,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.037898+0000","last_change":"2024-04-26T16:35:18.589728+0000","last_active":"2024-04-26T16:38:00.037898+0000","last_peered":"2024-04-26T16:38:00.037898+0000","last_clean":"2024-04-26T16:38:00.037898+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:38:00.037898+0000","last_undegraded":"2024-04-26T16:38:00.037898+0000","last_fullsized":"2024-04-26T16:38:00.037898+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'40","last_scrub_stamp":"2024-04-26T16:35:18.589318+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:35:18.589318+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1684,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.429129+0000","last_change":"2024-04-26T16:36:52.176623+0000","last_active":"2024-04-26T16:37:56.429129+0000","last_peered":"2024-04-26T16:37:56.429129+0000","last_clean":"2024-04-26T16:37:56.429129+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:37:56.429129+0000","last_undegraded":"2024-04-26T16:37:56.429129+0000","last_fullsized":"2024-04-26T16:37:56.429129+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:52.176514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:36:52.176514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:10.538041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1661,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:01.824412+0000","last_change":"2024-04-26T16:38:01.824470+0000","last_active":"2024-04-26T16:38:01.824412+0000","last_peered":"2024-04-26T16:38:01.824412+0000","last_clean":"2024-04-26T16:38:01.824412+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:38:01.824412+0000","last_undegraded":"2024-04-26T16:38:01.824412+0000","last_fullsized":"2024-04-26T16:38:01.824412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:01.824258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:01.824258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.185258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1515,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894258+0000","last_change":"2024-04-26T16:37:01.923812+0000","last_active":"2024-04-26T16:37:56.894258+0000","last_peered":"2024-04-26T16:37:56.894258+0000","last_clean":"2024-04-26T16:37:56.894258+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:37:56.894258+0000","last_undegraded":"2024-04-26T16:37:56.894258+0000","last_fullsized":"2024-04-26T16:37:56.894258+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.923649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.923649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:07.981437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1672,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.780162+0000","last_change":"2024-04-26T16:36:46.698276+0000","last_active":"2024-04-26T16:37:56.780162+0000","last_peered":"2024-04-26T16:37:56.780162+0000","last_clean":"2024-04-26T16:37:56.780162+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:37:56.780162+0000","last_undegraded":"2024-04-26T16:37:56.780162+0000","last_fullsized":"2024-04-26T16:37:56.780162+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.698075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.698075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:16.625557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"857'27","reported_seq":1698,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894393+0000","last_change":"2024-04-26T16:37:56.108530+0000","last_active":"2024-04-26T16:37:56.894393+0000","last_peered":"2024-04-26T16:37:56.894393+0000","last_clean":"2024-04-26T16:37:56.894393+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:37:56.894393+0000","last_undegraded":"2024-04-26T16:37:56.894393+0000","last_fullsized":"2024-04-26T16:37:56.894393+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:37:56.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.108290+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.284265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1704,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432210+0000","last_change":"2024-04-26T16:36:59.986857+0000","last_active":"2024-04-26T16:37:56.432210+0000","last_peered":"2024-04-26T16:37:56.432210+0000","last_clean":"2024-04-26T16:37:56.432210+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:37:56.432210+0000","last_undegraded":"2024-04-26T16:37:56.432210+0000","last_fullsized":"2024-04-26T16:37:56.432210+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:36:59.986744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.986744+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.282717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1644,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431972+0000","last_change":"2024-04-26T16:37:48.766181+0000","last_active":"2024-04-26T16:37:56.431972+0000","last_peered":"2024-04-26T16:37:56.431972+0000","last_clean":"2024-04-26T16:37:56.431972+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:37:56.431972+0000","last_undegraded":"2024-04-26T16:37:56.431972+0000","last_fullsized":"2024-04-26T16:37:56.431972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:48.765876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.420943+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.765876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.048973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1515,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891247+0000","last_change":"2024-04-26T16:36:41.913175+0000","last_active":"2024-04-26T16:37:56.891247+0000","last_peered":"2024-04-26T16:37:56.891247+0000","last_clean":"2024-04-26T16:37:56.891247+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:37:56.891247+0000","last_undegraded":"2024-04-26T16:37:56.891247+0000","last_fullsized":"2024-04-26T16:37:56.891247+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:41.912886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:41.912886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:02.918914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"857'27","reported_seq":1696,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891220+0000","last_change":"2024-04-26T16:36:19.167372+0000","last_active":"2024-04-26T16:37:56.891220+0000","last_peered":"2024-04-26T16:37:56.891220+0000","last_clean":"2024-04-26T16:37:56.891220+0000","last_became_active":"2024-04-26T16:35:25.395993+0000","last_became_peered":"2024-04-26T16:35:25.395993+0000","last_unstale":"2024-04-26T16:37:56.891220+0000","last_undegraded":"2024-04-26T16:37:56.891220+0000","last_fullsized":"2024-04-26T16:37:56.891220+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:36:19.167132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:19.167132+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1679,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.780022+0000","last_change":"2024-04-26T16:36:34.775511+0000","last_active":"2024-04-26T16:37:56.780022+0000","last_peered":"2024-04-26T16:37:56.780022+0000","last_clean":"2024-04-26T16:37:56.780022+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:37:56.780022+0000","last_undegraded":"2024-04-26T16:37:56.780022+0000","last_fullsized":"2024-04-26T16:37:56.780022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:34.775144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:36:34.775144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1647,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.434484+0000","last_change":"2024-04-26T16:37:51.748594+0000","last_active":"2024-04-26T16:37:56.434484+0000","last_peered":"2024-04-26T16:37:56.434484+0000","last_clean":"2024-04-26T16:37:56.434484+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:37:56.434484+0000","last_undegraded":"2024-04-26T16:37:56.434484+0000","last_fullsized":"2024-04-26T16:37:56.434484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.748496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.748496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.803195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1667,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.782011+0000","last_change":"2024-04-26T16:37:51.423701+0000","last_active":"2024-04-26T16:37:56.782011+0000","last_peered":"2024-04-26T16:37:56.782011+0000","last_clean":"2024-04-26T16:37:56.782011+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:37:56.782011+0000","last_undegraded":"2024-04-26T16:37:56.782011+0000","last_fullsized":"2024-04-26T16:37:56.782011+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.423243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.423243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:54.335512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1533,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.779996+0000","last_change":"2024-04-26T16:37:56.382398+0000","last_active":"2024-04-26T16:37:56.779996+0000","last_peered":"2024-04-26T16:37:56.779996+0000","last_clean":"2024-04-26T16:37:56.779996+0000","last_became_active":"2024-04-26T16:35:25.399182+0000","last_became_peered":"2024-04-26T16:35:25.399182+0000","last_unstale":"2024-04-26T16:37:56.779996+0000","last_undegraded":"2024-04-26T16:37:56.779996+0000","last_fullsized":"2024-04-26T16:37:56.779996+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.382182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.382182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.313965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"856'12","reported_seq":1406,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893415+0000","last_change":"2024-04-26T16:37:56.309533+0000","last_active":"2024-04-26T16:37:56.893415+0000","last_peered":"2024-04-26T16:37:56.893415+0000","last_clean":"2024-04-26T16:37:56.893415+0000","last_became_active":"2024-04-26T16:35:25.404028+0000","last_became_peered":"2024-04-26T16:35:25.404028+0000","last_unstale":"2024-04-26T16:37:56.893415+0000","last_undegraded":"2024-04-26T16:37:56.893415+0000","last_fullsized":"2024-04-26T16:37:56.893415+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'12","last_scrub_stamp":"2024-04-26T16:37:56.309129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.309129+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.607105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1243,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893726+0000","last_change":"2024-04-26T16:37:01.416249+0000","last_active":"2024-04-26T16:37:56.893726+0000","last_peered":"2024-04-26T16:37:56.893726+0000","last_clean":"2024-04-26T16:37:56.893726+0000","last_became_active":"2024-04-26T16:35:25.403847+0000","last_became_peered":"2024-04-26T16:35:25.403847+0000","last_unstale":"2024-04-26T16:37:56.893726+0000","last_undegraded":"2024-04-26T16:37:56.893726+0000","last_fullsized":"2024-04-26T16:37:56.893726+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.416147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.416147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:13.035252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893387+0000","last_change":"2024-04-26T16:36:41.687047+0000","last_active":"2024-04-26T16:37:56.893387+0000","last_peered":"2024-04-26T16:37:56.893387+0000","last_clean":"2024-04-26T16:37:56.893387+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:37:56.893387+0000","last_undegraded":"2024-04-26T16:37:56.893387+0000","last_fullsized":"2024-04-26T16:37:56.893387+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:41.686919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:41.686919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1645,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891541+0000","last_change":"2024-04-26T16:36:33.166398+0000","last_active":"2024-04-26T16:37:56.891541+0000","last_peered":"2024-04-26T16:37:56.891541+0000","last_clean":"2024-04-26T16:37:56.891541+0000","last_became_active":"2024-04-26T16:35:25.379432+0000","last_became_peered":"2024-04-26T16:35:25.379432+0000","last_unstale":"2024-04-26T16:37:56.891541+0000","last_undegraded":"2024-04-26T16:37:56.891541+0000","last_fullsized":"2024-04-26T16:37:56.891541+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:33.166179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.166179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1528,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431304+0000","last_change":"2024-04-26T16:37:43.955898+0000","last_active":"2024-04-26T16:37:56.431304+0000","last_peered":"2024-04-26T16:37:56.431304+0000","last_clean":"2024-04-26T16:37:56.431304+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:37:56.431304+0000","last_undegraded":"2024-04-26T16:37:56.431304+0000","last_fullsized":"2024-04-26T16:37:56.431304+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:43.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:43.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:12.284132+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"857'27","reported_seq":1704,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895708+0000","last_change":"2024-04-26T16:34:51.453336+0000","last_active":"2024-04-26T16:37:56.895708+0000","last_peered":"2024-04-26T16:37:56.895708+0000","last_clean":"2024-04-26T16:37:56.895708+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:37:56.895708+0000","last_undegraded":"2024-04-26T16:37:56.895708+0000","last_fullsized":"2024-04-26T16:37:56.895708+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:34:51.453064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:34:51.453064+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893726+0000","last_change":"2024-04-26T16:36:49.734509+0000","last_active":"2024-04-26T16:37:56.893726+0000","last_peered":"2024-04-26T16:37:56.893726+0000","last_clean":"2024-04-26T16:37:56.893726+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:37:56.893726+0000","last_undegraded":"2024-04-26T16:37:56.893726+0000","last_fullsized":"2024-04-26T16:37:56.893726+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:49.734276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:49.734276+0000","last_clean_scrub_stamp":"2024-04-26T16:36:49.734276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:18.791205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":671,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.036791+0000","last_change":"2024-04-26T16:37:56.088969+0000","last_active":"2024-04-26T16:38:00.036791+0000","last_peered":"2024-04-26T16:38:00.036791+0000","last_clean":"2024-04-26T16:38:00.036791+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:38:00.036791+0000","last_undegraded":"2024-04-26T16:38:00.036791+0000","last_fullsized":"2024-04-26T16:38:00.036791+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:37:56.088660+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.088660+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.991987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1659,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895014+0000","last_change":"2024-04-26T16:37:53.920778+0000","last_active":"2024-04-26T16:37:56.895014+0000","last_peered":"2024-04-26T16:37:56.895014+0000","last_clean":"2024-04-26T16:37:56.895014+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:37:56.895014+0000","last_undegraded":"2024-04-26T16:37:56.895014+0000","last_fullsized":"2024-04-26T16:37:56.895014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:53.920538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:53.920538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.535913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893453+0000","last_change":"2024-04-26T16:36:54.762255+0000","last_active":"2024-04-26T16:37:56.893453+0000","last_peered":"2024-04-26T16:37:56.893453+0000","last_clean":"2024-04-26T16:37:56.893453+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:37:56.893453+0000","last_undegraded":"2024-04-26T16:37:56.893453+0000","last_fullsized":"2024-04-26T16:37:56.893453+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:54.762119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:54.762119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:08.017921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"857'25","reported_seq":1705,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:02.823944+0000","last_change":"2024-04-26T16:38:02.824020+0000","last_active":"2024-04-26T16:38:02.823944+0000","last_peered":"2024-04-26T16:38:02.823944+0000","last_clean":"2024-04-26T16:38:02.823944+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:38:02.823944+0000","last_undegraded":"2024-04-26T16:38:02.823944+0000","last_fullsized":"2024-04-26T16:38:02.823944+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:38:02.823785+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.823785+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.431260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1671,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894635+0000","last_change":"2024-04-26T16:36:45.881045+0000","last_active":"2024-04-26T16:37:56.894635+0000","last_peered":"2024-04-26T16:37:56.894635+0000","last_clean":"2024-04-26T16:37:56.894635+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:37:56.894635+0000","last_undegraded":"2024-04-26T16:37:56.894635+0000","last_fullsized":"2024-04-26T16:37:56.894635+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_clean_scrub_stamp":"2024-04-26T16:36:45.880889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:10.042742+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1905,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.035722+0000","last_change":"2024-04-26T16:37:44.755462+0000","last_active":"2024-04-26T16:38:00.035722+0000","last_peered":"2024-04-26T16:38:00.035722+0000","last_clean":"2024-04-26T16:38:00.035722+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:38:00.035722+0000","last_undegraded":"2024-04-26T16:38:00.035722+0000","last_fullsized":"2024-04-26T16:38:00.035722+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:37:44.755315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.755315+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:55.270439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1687,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:05.800769+0000","last_change":"2024-04-26T16:38:05.800827+0000","last_active":"2024-04-26T16:38:05.800769+0000","last_peered":"2024-04-26T16:38:05.800769+0000","last_clean":"2024-04-26T16:38:05.800769+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:38:05.800769+0000","last_undegraded":"2024-04-26T16:38:05.800769+0000","last_fullsized":"2024-04-26T16:38:05.800769+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:05.800618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.800618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.334366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1490,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894172+0000","last_change":"2024-04-26T16:34:49.439720+0000","last_active":"2024-04-26T16:37:56.894172+0000","last_peered":"2024-04-26T16:37:56.894172+0000","last_clean":"2024-04-26T16:37:56.894172+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:37:56.894172+0000","last_undegraded":"2024-04-26T16:37:56.894172+0000","last_fullsized":"2024-04-26T16:37:56.894172+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:34:49.439494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:34:49.439494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"857'25","reported_seq":1681,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894267+0000","last_change":"2024-04-26T16:34:56.518735+0000","last_active":"2024-04-26T16:37:56.894267+0000","last_peered":"2024-04-26T16:37:56.894267+0000","last_clean":"2024-04-26T16:37:56.894267+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:37:56.894267+0000","last_undegraded":"2024-04-26T16:37:56.894267+0000","last_fullsized":"2024-04-26T16:37:56.894267+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:34:56.518631+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:34:56.518631+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1325,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895173+0000","last_change":"2024-04-26T16:37:45.265963+0000","last_active":"2024-04-26T16:37:56.895173+0000","last_peered":"2024-04-26T16:37:56.895173+0000","last_clean":"2024-04-26T16:37:56.895173+0000","last_became_active":"2024-04-26T16:35:25.401229+0000","last_became_peered":"2024-04-26T16:35:25.401229+0000","last_unstale":"2024-04-26T16:37:56.895173+0000","last_undegraded":"2024-04-26T16:37:56.895173+0000","last_fullsized":"2024-04-26T16:37:56.895173+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:45.265860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.265860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.529405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1655,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431809+0000","last_change":"2024-04-26T16:37:49.748432+0000","last_active":"2024-04-26T16:37:56.431809+0000","last_peered":"2024-04-26T16:37:56.431809+0000","last_clean":"2024-04-26T16:37:56.431809+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:37:56.431809+0000","last_undegraded":"2024-04-26T16:37:56.431809+0000","last_fullsized":"2024-04-26T16:37:56.431809+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:49.748209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:49.748209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:49.909641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1680,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.890657+0000","last_change":"2024-04-26T16:36:43.892069+0000","last_active":"2024-04-26T16:37:56.890657+0000","last_peered":"2024-04-26T16:37:56.890657+0000","last_clean":"2024-04-26T16:37:56.890657+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:37:56.890657+0000","last_undegraded":"2024-04-26T16:37:56.890657+0000","last_fullsized":"2024-04-26T16:37:56.890657+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:43.891851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:18.787243+0000","last_clean_scrub_stamp":"2024-04-26T16:36:43.891851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:03.864752+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1142,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.895182+0000","last_change":"2024-04-26T16:37:43.256055+0000","last_active":"2024-04-26T16:37:56.895182+0000","last_peered":"2024-04-26T16:37:56.895182+0000","last_clean":"2024-04-26T16:37:56.895182+0000","last_became_active":"2024-04-26T16:35:25.409902+0000","last_became_peered":"2024-04-26T16:35:25.409902+0000","last_unstale":"2024-04-26T16:37:56.895182+0000","last_undegraded":"2024-04-26T16:37:56.895182+0000","last_fullsized":"2024-04-26T16:37:56.895182+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:43.255960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:37:43.255960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:11.195362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"857'30","reported_seq":1730,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431655+0000","last_change":"2024-04-26T16:37:49.783263+0000","last_active":"2024-04-26T16:37:56.431655+0000","last_peered":"2024-04-26T16:37:56.431655+0000","last_clean":"2024-04-26T16:37:56.431655+0000","last_became_active":"2024-04-26T16:35:25.383698+0000","last_became_peered":"2024-04-26T16:35:25.383698+0000","last_unstale":"2024-04-26T16:37:56.431655+0000","last_undegraded":"2024-04-26T16:37:56.431655+0000","last_fullsized":"2024-04-26T16:37:56.431655+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:37:49.783106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:49.783106+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:50.844745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1190,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.194698+0000","last_change":"2024-04-26T16:38:00.194757+0000","last_active":"2024-04-26T16:38:00.194698+0000","last_peered":"2024-04-26T16:38:00.194698+0000","last_clean":"2024-04-26T16:38:00.194698+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:38:00.194698+0000","last_undegraded":"2024-04-26T16:38:00.194698+0000","last_fullsized":"2024-04-26T16:38:00.194698+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:00.194512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.194512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.614121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1679,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432838+0000","last_change":"2024-04-26T16:37:01.057309+0000","last_active":"2024-04-26T16:37:56.432838+0000","last_peered":"2024-04-26T16:37:56.432838+0000","last_clean":"2024-04-26T16:37:56.432838+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:37:56.432838+0000","last_undegraded":"2024-04-26T16:37:56.432838+0000","last_fullsized":"2024-04-26T16:37:56.432838+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.057091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:37:01.057091+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.057091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:18.963967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1655,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.427449+0000","last_change":"2024-04-26T16:36:44.583891+0000","last_active":"2024-04-26T16:37:56.427449+0000","last_peered":"2024-04-26T16:37:56.427449+0000","last_clean":"2024-04-26T16:37:56.427449+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:37:56.427449+0000","last_undegraded":"2024-04-26T16:37:56.427449+0000","last_fullsized":"2024-04-26T16:37:56.427449+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:44.583680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:36:44.583680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:10.136046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1524,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.782535+0000","last_change":"2024-04-26T16:37:00.520805+0000","last_active":"2024-04-26T16:37:56.782535+0000","last_peered":"2024-04-26T16:37:56.782535+0000","last_clean":"2024-04-26T16:37:56.782535+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:37:56.782535+0000","last_undegraded":"2024-04-26T16:37:56.782535+0000","last_fullsized":"2024-04-26T16:37:56.782535+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.520675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.520675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:22.348302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"857'18","reported_seq":1739,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893516+0000","last_change":"2024-04-26T16:37:44.980167+0000","last_active":"2024-04-26T16:37:56.893516+0000","last_peered":"2024-04-26T16:37:56.893516+0000","last_clean":"2024-04-26T16:37:56.893516+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:37:56.893516+0000","last_undegraded":"2024-04-26T16:37:56.893516+0000","last_fullsized":"2024-04-26T16:37:56.893516+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'18","last_scrub_stamp":"2024-04-26T16:37:44.980066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.980066+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.790501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":588,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893625+0000","last_change":"2024-04-26T16:35:26.526741+0000","last_active":"2024-04-26T16:37:56.893625+0000","last_peered":"2024-04-26T16:37:56.893625+0000","last_clean":"2024-04-26T16:37:56.893625+0000","last_became_active":"2024-04-26T16:35:25.389184+0000","last_became_peered":"2024-04-26T16:35:25.389184+0000","last_unstale":"2024-04-26T16:37:56.893625+0000","last_undegraded":"2024-04-26T16:37:56.893625+0000","last_fullsized":"2024-04-26T16:37:56.893625+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:26.526395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:35:26.526395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1283,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893395+0000","last_change":"2024-04-26T16:36:55.365068+0000","last_active":"2024-04-26T16:37:56.893395+0000","last_peered":"2024-04-26T16:37:56.893395+0000","last_clean":"2024-04-26T16:37:56.893395+0000","last_became_active":"2024-04-26T16:35:25.405230+0000","last_became_peered":"2024-04-26T16:35:25.405230+0000","last_unstale":"2024-04-26T16:37:56.893395+0000","last_undegraded":"2024-04-26T16:37:56.893395+0000","last_fullsized":"2024-04-26T16:37:56.893395+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:55.364804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:55.364804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:09.183332+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1635,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893848+0000","last_change":"2024-04-26T16:37:00.148921+0000","last_active":"2024-04-26T16:37:56.893848+0000","last_peered":"2024-04-26T16:37:56.893848+0000","last_clean":"2024-04-26T16:37:56.893848+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:37:56.893848+0000","last_undegraded":"2024-04-26T16:37:56.893848+0000","last_fullsized":"2024-04-26T16:37:56.893848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.148735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.148735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:19.076498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1510,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894967+0000","last_change":"2024-04-26T16:36:38.157649+0000","last_active":"2024-04-26T16:37:56.894967+0000","last_peered":"2024-04-26T16:37:56.894967+0000","last_clean":"2024-04-26T16:37:56.894967+0000","last_became_active":"2024-04-26T16:35:25.394016+0000","last_became_peered":"2024-04-26T16:35:25.394016+0000","last_unstale":"2024-04-26T16:37:56.894967+0000","last_undegraded":"2024-04-26T16:37:56.894967+0000","last_fullsized":"2024-04-26T16:37:56.894967+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:38.157418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_clean_scrub_stamp":"2024-04-26T16:36:38.157418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"856'21","reported_seq":614,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893502+0000","last_change":"2024-04-26T16:36:44.718941+0000","last_active":"2024-04-26T16:37:56.893502+0000","last_peered":"2024-04-26T16:37:56.893502+0000","last_clean":"2024-04-26T16:37:56.893502+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:37:56.893502+0000","last_undegraded":"2024-04-26T16:37:56.893502+0000","last_fullsized":"2024-04-26T16:37:56.893502+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:44.718541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:44.718541+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:00.041329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1294,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894789+0000","last_change":"2024-04-26T16:37:44.221946+0000","last_active":"2024-04-26T16:37:56.894789+0000","last_peered":"2024-04-26T16:37:56.894789+0000","last_clean":"2024-04-26T16:37:56.894789+0000","last_became_active":"2024-04-26T16:35:25.406825+0000","last_became_peered":"2024-04-26T16:35:25.406825+0000","last_unstale":"2024-04-26T16:37:56.894789+0000","last_undegraded":"2024-04-26T16:37:56.894789+0000","last_fullsized":"2024-04-26T16:37:56.894789+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:44.221823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.221823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:13.401315+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1649,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891586+0000","last_change":"2024-04-26T16:36:15.209969+0000","last_active":"2024-04-26T16:37:56.891586+0000","last_peered":"2024-04-26T16:37:56.891586+0000","last_clean":"2024-04-26T16:37:56.891586+0000","last_became_active":"2024-04-26T16:35:25.378783+0000","last_became_peered":"2024-04-26T16:35:25.378783+0000","last_unstale":"2024-04-26T16:37:56.891586+0000","last_undegraded":"2024-04-26T16:37:56.891586+0000","last_fullsized":"2024-04-26T16:37:56.891586+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:15.209875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:15.209875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1653,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432033+0000","last_change":"2024-04-26T16:37:55.748337+0000","last_active":"2024-04-26T16:37:56.432033+0000","last_peered":"2024-04-26T16:37:56.432033+0000","last_clean":"2024-04-26T16:37:56.432033+0000","last_became_active":"2024-04-26T16:35:25.410836+0000","last_became_peered":"2024-04-26T16:35:25.410836+0000","last_unstale":"2024-04-26T16:37:56.432033+0000","last_undegraded":"2024-04-26T16:37:56.432033+0000","last_fullsized":"2024-04-26T16:37:56.432033+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:55.748010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:55.748010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.141796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1523,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:59.386304+0000","last_change":"2024-04-26T16:37:59.386367+0000","last_active":"2024-04-26T16:37:59.386304+0000","last_peered":"2024-04-26T16:37:59.386304+0000","last_clean":"2024-04-26T16:37:59.386304+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:37:59.386304+0000","last_undegraded":"2024-04-26T16:37:59.386304+0000","last_fullsized":"2024-04-26T16:37:59.386304+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:59.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.277947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"857'46","reported_seq":1747,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.428535+0000","last_change":"2024-04-26T16:37:04.034823+0000","last_active":"2024-04-26T16:37:56.428535+0000","last_peered":"2024-04-26T16:37:56.428535+0000","last_clean":"2024-04-26T16:37:56.428535+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:37:56.428535+0000","last_undegraded":"2024-04-26T16:37:56.428535+0000","last_fullsized":"2024-04-26T16:37:56.428535+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:37:04.034569+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:37:04.034569+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:19.723405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891641+0000","last_change":"2024-04-26T16:37:52.063624+0000","last_active":"2024-04-26T16:37:56.891641+0000","last_peered":"2024-04-26T16:37:56.891641+0000","last_clean":"2024-04-26T16:37:56.891641+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:37:56.891641+0000","last_undegraded":"2024-04-26T16:37:56.891641+0000","last_fullsized":"2024-04-26T16:37:56.891641+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.562434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"747'16","reported_seq":1478,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894532+0000","last_change":"2024-04-26T16:37:04.364780+0000","last_active":"2024-04-26T16:37:56.894532+0000","last_peered":"2024-04-26T16:37:56.894532+0000","last_clean":"2024-04-26T16:37:56.894532+0000","last_became_active":"2024-04-26T16:35:25.403416+0000","last_became_peered":"2024-04-26T16:35:25.403416+0000","last_unstale":"2024-04-26T16:37:56.894532+0000","last_undegraded":"2024-04-26T16:37:56.894532+0000","last_fullsized":"2024-04-26T16:37:56.894532+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:37:04.364467+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:37:04.364467+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:04.901532+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"829'2","reported_seq":1658,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892182+0000","last_change":"2024-04-26T16:36:59.155190+0000","last_active":"2024-04-26T16:37:56.892182+0000","last_peered":"2024-04-26T16:37:56.892182+0000","last_clean":"2024-04-26T16:37:56.892182+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:37:56.892182+0000","last_undegraded":"2024-04-26T16:37:56.892182+0000","last_fullsized":"2024-04-26T16:37:56.892182+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'2","last_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_deep_scrub":"829'2","last_deep_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.155048+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.337428+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1525,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892989+0000","last_change":"2024-04-26T16:37:52.886432+0000","last_active":"2024-04-26T16:37:56.892989+0000","last_peered":"2024-04-26T16:37:56.892989+0000","last_clean":"2024-04-26T16:37:56.892989+0000","last_became_active":"2024-04-26T16:35:25.409763+0000","last_became_peered":"2024-04-26T16:35:25.409763+0000","last_unstale":"2024-04-26T16:37:56.892989+0000","last_undegraded":"2024-04-26T16:37:56.892989+0000","last_fullsized":"2024-04-26T16:37:56.892989+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.886311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.886311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.121107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"856'21","reported_seq":1685,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.427567+0000","last_change":"2024-04-26T16:36:47.694088+0000","last_active":"2024-04-26T16:37:56.427567+0000","last_peered":"2024-04-26T16:37:56.427567+0000","last_clean":"2024-04-26T16:37:56.427567+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:37:56.427567+0000","last_undegraded":"2024-04-26T16:37:56.427567+0000","last_fullsized":"2024-04-26T16:37:56.427567+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:47.693892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:47.693892+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:04.653414+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1667,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:59.723599+0000","last_change":"2024-04-26T16:37:59.723665+0000","last_active":"2024-04-26T16:37:59.723599+0000","last_peered":"2024-04-26T16:37:59.723599+0000","last_clean":"2024-04-26T16:37:59.723599+0000","last_became_active":"2024-04-26T16:35:25.417030+0000","last_became_peered":"2024-04-26T16:35:25.417030+0000","last_unstale":"2024-04-26T16:37:59.723599+0000","last_undegraded":"2024-04-26T16:37:59.723599+0000","last_fullsized":"2024-04-26T16:37:59.723599+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:37:59.723438+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.723438+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.920638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1673,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894706+0000","last_change":"2024-04-26T16:37:00.945337+0000","last_active":"2024-04-26T16:37:56.894706+0000","last_peered":"2024-04-26T16:37:56.894706+0000","last_clean":"2024-04-26T16:37:56.894706+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:37:56.894706+0000","last_undegraded":"2024-04-26T16:37:56.894706+0000","last_fullsized":"2024-04-26T16:37:56.894706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.945236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.945236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:06.557754+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1683,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.433104+0000","last_change":"2024-04-26T16:36:57.199063+0000","last_active":"2024-04-26T16:37:56.433104+0000","last_peered":"2024-04-26T16:37:56.433104+0000","last_clean":"2024-04-26T16:37:56.433104+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:37:56.433104+0000","last_undegraded":"2024-04-26T16:37:56.433104+0000","last_fullsized":"2024-04-26T16:37:56.433104+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:57.198785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.198785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:14.740615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:04.121882+0000","last_change":"2024-04-26T16:38:04.121917+0000","last_active":"2024-04-26T16:38:04.121882+0000","last_peered":"2024-04-26T16:38:04.121882+0000","last_clean":"2024-04-26T16:38:04.121882+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:38:04.121882+0000","last_undegraded":"2024-04-26T16:38:04.121882+0000","last_fullsized":"2024-04-26T16:38:04.121882+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:04.121774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.121774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.071499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"857'11","reported_seq":1685,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894801+0000","last_change":"2024-04-26T16:36:57.942537+0000","last_active":"2024-04-26T16:37:56.894801+0000","last_peered":"2024-04-26T16:37:56.894801+0000","last_clean":"2024-04-26T16:37:56.894801+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:37:56.894801+0000","last_undegraded":"2024-04-26T16:37:56.894801+0000","last_fullsized":"2024-04-26T16:37:56.894801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:36:57.942209+0000","last_deep_scrub":"388'9","last_deep_scrub_stamp":"2024-04-26T16:36:57.942209+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.942209+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:21.478333+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1351,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892847+0000","last_change":"2024-04-26T16:37:47.358920+0000","last_active":"2024-04-26T16:37:56.892847+0000","last_peered":"2024-04-26T16:37:56.892847+0000","last_clean":"2024-04-26T16:37:56.892847+0000","last_became_active":"2024-04-26T16:35:25.404997+0000","last_became_peered":"2024-04-26T16:35:25.404997+0000","last_unstale":"2024-04-26T16:37:56.892847+0000","last_undegraded":"2024-04-26T16:37:56.892847+0000","last_fullsized":"2024-04-26T16:37:56.892847+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:47.358713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.358713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:57.165878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1675,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432874+0000","last_change":"2024-04-26T16:36:42.150591+0000","last_active":"2024-04-26T16:37:56.432874+0000","last_peered":"2024-04-26T16:37:56.432874+0000","last_clean":"2024-04-26T16:37:56.432874+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:37:56.432874+0000","last_undegraded":"2024-04-26T16:37:56.432874+0000","last_fullsized":"2024-04-26T16:37:56.432874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:42.150464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:42.150464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1699,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.433120+0000","last_change":"2024-04-26T16:37:51.818589+0000","last_active":"2024-04-26T16:37:56.433120+0000","last_peered":"2024-04-26T16:37:56.433120+0000","last_clean":"2024-04-26T16:37:56.433120+0000","last_became_active":"2024-04-26T16:35:25.383286+0000","last_became_peered":"2024-04-26T16:35:25.383286+0000","last_unstale":"2024-04-26T16:37:56.433120+0000","last_undegraded":"2024-04-26T16:37:56.433120+0000","last_fullsized":"2024-04-26T16:37:56.433120+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.818319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.818319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.894110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1499,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:00.068949+0000","last_change":"2024-04-26T16:38:00.069058+0000","last_active":"2024-04-26T16:38:00.068949+0000","last_peered":"2024-04-26T16:38:00.068949+0000","last_clean":"2024-04-26T16:38:00.068949+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:38:00.068949+0000","last_undegraded":"2024-04-26T16:38:00.068949+0000","last_fullsized":"2024-04-26T16:38:00.068949+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:00.068758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.068758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.541601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"857'21","reported_seq":1709,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431456+0000","last_change":"2024-04-26T16:36:51.894635+0000","last_active":"2024-04-26T16:37:56.431456+0000","last_peered":"2024-04-26T16:37:56.431456+0000","last_clean":"2024-04-26T16:37:56.431456+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:37:56.431456+0000","last_undegraded":"2024-04-26T16:37:56.431456+0000","last_fullsized":"2024-04-26T16:37:56.431456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:51.894411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:51.894411+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:17.890802+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1665,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:03.698923+0000","last_change":"2024-04-26T16:38:03.698985+0000","last_active":"2024-04-26T16:38:03.698923+0000","last_peered":"2024-04-26T16:38:03.698923+0000","last_clean":"2024-04-26T16:38:03.698923+0000","last_became_active":"2024-04-26T16:35:25.392415+0000","last_became_peered":"2024-04-26T16:35:25.392415+0000","last_unstale":"2024-04-26T16:38:03.698923+0000","last_undegraded":"2024-04-26T16:38:03.698923+0000","last_fullsized":"2024-04-26T16:38:03.698923+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:03.698766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.698766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.722484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1671,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431876+0000","last_change":"2024-04-26T16:36:56.985639+0000","last_active":"2024-04-26T16:37:56.431876+0000","last_peered":"2024-04-26T16:37:56.431876+0000","last_clean":"2024-04-26T16:37:56.431876+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:37:56.431876+0000","last_undegraded":"2024-04-26T16:37:56.431876+0000","last_fullsized":"2024-04-26T16:37:56.431876+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_clean_scrub_stamp":"2024-04-26T16:36:56.985402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:19.810617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1666,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:58.738861+0000","last_change":"2024-04-26T16:37:58.738918+0000","last_active":"2024-04-26T16:37:58.738861+0000","last_peered":"2024-04-26T16:37:58.738861+0000","last_clean":"2024-04-26T16:37:58.738861+0000","last_became_active":"2024-04-26T16:35:25.410430+0000","last_became_peered":"2024-04-26T16:35:25.410430+0000","last_unstale":"2024-04-26T16:37:58.738861+0000","last_undegraded":"2024-04-26T16:37:58.738861+0000","last_fullsized":"2024-04-26T16:37:58.738861+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.738705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.738705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.841581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1523,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.781473+0000","last_change":"2024-04-26T16:37:47.396592+0000","last_active":"2024-04-26T16:37:56.781473+0000","last_peered":"2024-04-26T16:37:56.781473+0000","last_clean":"2024-04-26T16:37:56.781473+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:37:56.781473+0000","last_undegraded":"2024-04-26T16:37:56.781473+0000","last_fullsized":"2024-04-26T16:37:56.781473+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:47.396490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.396490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.967350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"857'29","reported_seq":617,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893960+0000","last_change":"2024-04-26T16:36:33.541569+0000","last_active":"2024-04-26T16:37:56.893960+0000","last_peered":"2024-04-26T16:37:56.893960+0000","last_clean":"2024-04-26T16:37:56.893960+0000","last_became_active":"2024-04-26T16:35:25.382813+0000","last_became_peered":"2024-04-26T16:35:25.382813+0000","last_unstale":"2024-04-26T16:37:56.893960+0000","last_undegraded":"2024-04-26T16:37:56.893960+0000","last_fullsized":"2024-04-26T16:37:56.893960+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:36:33.541439+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.541439+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1676,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:04.132282+0000","last_change":"2024-04-26T16:38:04.132311+0000","last_active":"2024-04-26T16:38:04.132282+0000","last_peered":"2024-04-26T16:38:04.132282+0000","last_clean":"2024-04-26T16:38:04.132282+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:38:04.132282+0000","last_undegraded":"2024-04-26T16:38:04.132282+0000","last_fullsized":"2024-04-26T16:38:04.132282+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:04.132193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.132193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:05.515649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":587,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892091+0000","last_change":"2024-04-26T16:36:40.657677+0000","last_active":"2024-04-26T16:37:56.892091+0000","last_peered":"2024-04-26T16:37:56.892091+0000","last_clean":"2024-04-26T16:37:56.892091+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:37:56.892091+0000","last_undegraded":"2024-04-26T16:37:56.892091+0000","last_fullsized":"2024-04-26T16:37:56.892091+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:40.657538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:40.657538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:07.035723+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1659,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.890994+0000","last_change":"2024-04-26T16:36:56.939266+0000","last_active":"2024-04-26T16:37:56.890994+0000","last_peered":"2024-04-26T16:37:56.890994+0000","last_clean":"2024-04-26T16:37:56.890994+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:37:56.890994+0000","last_undegraded":"2024-04-26T16:37:56.890994+0000","last_fullsized":"2024-04-26T16:37:56.890994+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:56.939135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.855045+0000","last_clean_scrub_stamp":"2024-04-26T16:36:56.939135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:59.071622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1130,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894997+0000","last_change":"2024-04-26T16:37:51.285062+0000","last_active":"2024-04-26T16:37:56.894997+0000","last_peered":"2024-04-26T16:37:56.894997+0000","last_clean":"2024-04-26T16:37:56.894997+0000","last_became_active":"2024-04-26T16:35:25.401906+0000","last_became_peered":"2024-04-26T16:35:25.401906+0000","last_unstale":"2024-04-26T16:37:56.894997+0000","last_undegraded":"2024-04-26T16:37:56.894997+0000","last_fullsized":"2024-04-26T16:37:56.894997+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.284847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.284847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.463961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"857'39","reported_seq":632,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894347+0000","last_change":"2024-04-26T16:37:45.123276+0000","last_active":"2024-04-26T16:37:56.894347+0000","last_peered":"2024-04-26T16:37:56.894347+0000","last_clean":"2024-04-26T16:37:56.894347+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:37:56.894347+0000","last_undegraded":"2024-04-26T16:37:56.894347+0000","last_fullsized":"2024-04-26T16:37:56.894347+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'39","last_scrub_stamp":"2024-04-26T16:37:45.122923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.122923+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:02.778170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1665,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.894081+0000","last_change":"2024-04-26T16:36:23.142745+0000","last_active":"2024-04-26T16:37:56.894081+0000","last_peered":"2024-04-26T16:37:56.894081+0000","last_clean":"2024-04-26T16:37:56.894081+0000","last_became_active":"2024-04-26T16:35:25.389963+0000","last_became_peered":"2024-04-26T16:35:25.389963+0000","last_unstale":"2024-04-26T16:37:56.894081+0000","last_undegraded":"2024-04-26T16:37:56.894081+0000","last_fullsized":"2024-04-26T16:37:56.894081+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:23.142443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:23.142443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1639,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892156+0000","last_change":"2024-04-26T16:36:46.231728+0000","last_active":"2024-04-26T16:37:56.892156+0000","last_peered":"2024-04-26T16:37:56.892156+0000","last_clean":"2024-04-26T16:37:56.892156+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:37:56.892156+0000","last_undegraded":"2024-04-26T16:37:56.892156+0000","last_fullsized":"2024-04-26T16:37:56.892156+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.231628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.231628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:14.131471+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":599,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.892011+0000","last_change":"2024-04-26T16:37:48.095046+0000","last_active":"2024-04-26T16:37:56.892011+0000","last_peered":"2024-04-26T16:37:56.892011+0000","last_clean":"2024-04-26T16:37:56.892011+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:37:56.892011+0000","last_undegraded":"2024-04-26T16:37:56.892011+0000","last_fullsized":"2024-04-26T16:37:56.892011+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:37:48.094883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.094883+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:52.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1521,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.891980+0000","last_change":"2024-04-26T16:36:22.139010+0000","last_active":"2024-04-26T16:37:56.891980+0000","last_peered":"2024-04-26T16:37:56.891980+0000","last_clean":"2024-04-26T16:37:56.891980+0000","last_became_active":"2024-04-26T16:35:25.388177+0000","last_became_peered":"2024-04-26T16:35:25.388177+0000","last_unstale":"2024-04-26T16:37:56.891980+0000","last_undegraded":"2024-04-26T16:37:56.891980+0000","last_fullsized":"2024-04-26T16:37:56.891980+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:22.138796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:36:22.138796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"857'27","reported_seq":1756,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893709+0000","last_change":"2024-04-26T16:36:50.944738+0000","last_active":"2024-04-26T16:37:56.893709+0000","last_peered":"2024-04-26T16:37:56.893709+0000","last_clean":"2024-04-26T16:37:56.893709+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:37:56.893709+0000","last_undegraded":"2024-04-26T16:37:56.893709+0000","last_fullsized":"2024-04-26T16:37:56.893709+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:36:50.944517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:50.944517+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1677,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.432530+0000","last_change":"2024-04-26T16:36:46.673477+0000","last_active":"2024-04-26T16:37:56.432530+0000","last_peered":"2024-04-26T16:37:56.432530+0000","last_clean":"2024-04-26T16:37:56.432530+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:37:56.432530+0000","last_undegraded":"2024-04-26T16:37:56.432530+0000","last_fullsized":"2024-04-26T16:37:56.432530+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.673182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.673182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1687,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.428827+0000","last_change":"2024-04-26T16:36:47.136328+0000","last_active":"2024-04-26T16:37:56.428827+0000","last_peered":"2024-04-26T16:37:56.428827+0000","last_clean":"2024-04-26T16:37:56.428827+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:37:56.428827+0000","last_undegraded":"2024-04-26T16:37:56.428827+0000","last_fullsized":"2024-04-26T16:37:56.428827+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:47.136177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.136177+0000","last_clean_scrub_stamp":"2024-04-26T16:36:47.136177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:58.568862+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1679,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.431084+0000","last_change":"2024-04-26T16:36:48.921187+0000","last_active":"2024-04-26T16:37:56.431084+0000","last_peered":"2024-04-26T16:37:56.431084+0000","last_clean":"2024-04-26T16:37:56.431084+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:37:56.431084+0000","last_undegraded":"2024-04-26T16:37:56.431084+0000","last_fullsized":"2024-04-26T16:37:56.431084+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.921092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.921092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:13.986030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":595,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:01.112540+0000","last_change":"2024-04-26T16:38:01.112600+0000","last_active":"2024-04-26T16:38:01.112540+0000","last_peered":"2024-04-26T16:38:01.112540+0000","last_clean":"2024-04-26T16:38:01.112540+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:38:01.112540+0000","last_undegraded":"2024-04-26T16:38:01.112540+0000","last_fullsized":"2024-04-26T16:38:01.112540+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:01.112389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:16.569721+0000","last_clean_scrub_stamp":"2024-04-26T16:38:01.112389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.253547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"857'30","reported_seq":1708,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.433234+0000","last_change":"2024-04-26T16:36:31.653735+0000","last_active":"2024-04-26T16:37:56.433234+0000","last_peered":"2024-04-26T16:37:56.433234+0000","last_clean":"2024-04-26T16:37:56.433234+0000","last_became_active":"2024-04-26T16:35:25.399680+0000","last_became_peered":"2024-04-26T16:35:25.399680+0000","last_unstale":"2024-04-26T16:37:56.433234+0000","last_undegraded":"2024-04-26T16:37:56.433234+0000","last_fullsized":"2024-04-26T16:37:56.433234+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:36:31.653572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:31.653572+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1672,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.893834+0000","last_change":"2024-04-26T16:36:38.969958+0000","last_active":"2024-04-26T16:37:56.893834+0000","last_peered":"2024-04-26T16:37:56.893834+0000","last_clean":"2024-04-26T16:37:56.893834+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:37:56.893834+0000","last_undegraded":"2024-04-26T16:37:56.893834+0000","last_fullsized":"2024-04-26T16:37:56.893834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:38.969869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:36:38.969869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:57.278161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1681,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:37:56.430566+0000","last_change":"2024-04-26T16:37:45.893079+0000","last_active":"2024-04-26T16:37:56.430566+0000","last_peered":"2024-04-26T16:37:56.430566+0000","last_clean":"2024-04-26T16:37:56.430566+0000","last_became_active":"2024-04-26T16:35:25.396616+0000","last_became_peered":"2024-04-26T16:35:25.396616+0000","last_unstale":"2024-04-26T16:37:56.430566+0000","last_undegraded":"2024-04-26T16:37:56.430566+0000","last_fullsized":"2024-04-26T16:37:56.430566+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:45.892786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.892786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:49.656756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1653,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-04-26T16:38:02.658073+0000","last_change":"2024-04-26T16:38:02.658129+0000","last_active":"2024-04-26T16:38:02.658073+0000","last_peered":"2024-04-26T16:38:02.658073+0000","last_clean":"2024-04-26T16:38:02.658073+0000","last_became_active":"2024-04-26T16:35:25.392061+0000","last_became_peered":"2024-04-26T16:35:25.392061+0000","last_unstale":"2024-04-26T16:38:02.658073+0000","last_undegraded":"2024-04-26T16:38:02.658073+0000","last_fullsized":"2024-04-26T16:38:02.658073+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.657915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.657915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.431423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1214,"num_read_kb":1209,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743682,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113792,"kb_used_data":6264,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93631360,"statfs":{"total":95995035648,"available":95878512640,"internally_reserved":0,"allocated":6414336,"data_stored":3513588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":746,"seq":3204045602849,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39648,"kb_used_data":6232,"kb_used_omap":23,"kb_used_meta":33384,"kb_avail":93705504,"statfs":{"total":95995035648,"available":95954436096,"internally_reserved":0,"allocated":6381568,"data_stored":3510822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23760,"internal_metadata":34186032},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575168,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104880,"kb_used_data":5672,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93640272,"statfs":{"total":95995035648,"available":95887638528,"internally_reserved":0,"allocated":5808128,"data_stored":2921786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738698,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109628,"kb_used_data":6260,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93635524,"statfs":{"total":95995035648,"available":95882776576,"internally_reserved":0,"allocated":6410240,"data_stored":3513016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411637,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104864,"kb_used_data":5656,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93640288,"statfs":{"total":95995035648,"available":95887654912,"internally_reserved":0,"allocated":5791744,"data_stored":2921052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084586,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131168,"kb_used_data":5656,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93613984,"statfs":{"total":95995035648,"available":95860719616,"internally_reserved":0,"allocated":5791744,"data_stored":2920854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128514504},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855648,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109052,"kb_used_data":5684,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93636100,"statfs":{"total":95995035648,"available":95883366400,"internally_reserved":0,"allocated":5820416,"data_stored":2921610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463190,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95136,"kb_used_data":5656,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650016,"statfs":{"total":95995035648,"available":95897616384,"internally_reserved":0,"allocated":5791744,"data_stored":2920864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:38:09.831 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:38:09.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:07.973566+0000 mgr.y (mgr.24401) 792 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:09.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:07.973566+0000 mgr.y (mgr.24401) 792 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:09.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.093121+0000 osd.7 (osd.7) 452 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:38:09.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.093121+0000 osd.7 (osd.7) 452 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:38:09.953 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.098179+0000 osd.7 (osd.7) 453 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.098179+0000 osd.7 (osd.7) 453 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.140407+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.140407+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.142713+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.142713+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: audit 2024-04-26T16:38:08.527334+0000 mgr.y (mgr.24401) 793 : audit [DBG] from='client.41738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: audit 2024-04-26T16:38:08.527334+0000 mgr.y (mgr.24401) 793 : audit [DBG] from='client.41738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.783560+0000 osd.0 (osd.0) 487 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.783560+0000 osd.0 (osd.0) 487 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.788146+0000 osd.0 (osd.0) 488 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:08.788146+0000 osd.0 (osd.0) 488 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: audit 2024-04-26T16:38:09.232051+0000 mon.a (mon.0) 3950 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:38:09.954 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: audit 2024-04-26T16:38:09.232051+0000 mon.a (mon.0) 3950 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.289656+0000 osd.5 (osd.5) 336 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.289656+0000 osd.5 (osd.5) 336 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.296127+0000 osd.5 (osd.5) 337 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.296127+0000 osd.5 (osd.5) 337 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.328492+0000 osd.2 (osd.2) 419 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.328492+0000 osd.2 (osd.2) 419 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.330891+0000 osd.2 (osd.2) 420 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.330891+0000 osd.2 (osd.2) 420 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:07.973566+0000 mgr.y (mgr.24401) 792 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:09.955 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:07.973566+0000 mgr.y (mgr.24401) 792 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.093121+0000 osd.7 (osd.7) 452 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.093121+0000 osd.7 (osd.7) 452 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.098179+0000 osd.7 (osd.7) 453 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.098179+0000 osd.7 (osd.7) 453 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.140407+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.140407+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.142713+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.142713+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: audit 2024-04-26T16:38:08.527334+0000 mgr.y (mgr.24401) 793 : audit [DBG] from='client.41738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: audit 2024-04-26T16:38:08.527334+0000 mgr.y (mgr.24401) 793 : audit [DBG] from='client.41738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.783560+0000 osd.0 (osd.0) 487 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.783560+0000 osd.0 (osd.0) 487 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.788146+0000 osd.0 (osd.0) 488 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:38:09.956 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:08.788146+0000 osd.0 (osd.0) 488 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: audit 2024-04-26T16:38:09.232051+0000 mon.a (mon.0) 3950 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: audit 2024-04-26T16:38:09.232051+0000 mon.a (mon.0) 3950 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.289656+0000 osd.5 (osd.5) 336 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.289656+0000 osd.5 (osd.5) 336 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.296127+0000 osd.5 (osd.5) 337 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.296127+0000 osd.5 (osd.5) 337 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.328492+0000 osd.2 (osd.2) 419 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.328492+0000 osd.2 (osd.2) 419 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.330891+0000 osd.2 (osd.2) 420 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:38:09.957 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:09 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.330891+0000 osd.2 (osd.2) 420 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:07.973566+0000 mgr.y (mgr.24401) 792 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:07.973566+0000 mgr.y (mgr.24401) 792 : cluster [DBG] pgmap v1365: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.093121+0000 osd.7 (osd.7) 452 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.093121+0000 osd.7 (osd.7) 452 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.098179+0000 osd.7 (osd.7) 453 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.098179+0000 osd.7 (osd.7) 453 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.140407+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.140407+0000 osd.1 (osd.1) 455 : cluster [DBG] 2.a deep-scrub starts 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.142713+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:38:10.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.142713+0000 osd.1 (osd.1) 456 : cluster [DBG] 2.a deep-scrub ok 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: audit 2024-04-26T16:38:08.527334+0000 mgr.y (mgr.24401) 793 : audit [DBG] from='client.41738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: audit 2024-04-26T16:38:08.527334+0000 mgr.y (mgr.24401) 793 : audit [DBG] from='client.41738 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.783560+0000 osd.0 (osd.0) 487 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.783560+0000 osd.0 (osd.0) 487 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.788146+0000 osd.0 (osd.0) 488 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:08.788146+0000 osd.0 (osd.0) 488 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: audit 2024-04-26T16:38:09.232051+0000 mon.a (mon.0) 3950 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: audit 2024-04-26T16:38:09.232051+0000 mon.a (mon.0) 3950 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.289656+0000 osd.5 (osd.5) 336 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.289656+0000 osd.5 (osd.5) 336 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.296127+0000 osd.5 (osd.5) 337 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.296127+0000 osd.5 (osd.5) 337 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.328492+0000 osd.2 (osd.2) 419 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.328492+0000 osd.2 (osd.2) 419 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:38:10.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.330891+0000 osd.2 (osd.2) 420 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:38:10.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:09 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.330891+0000 osd.2 (osd.2) 420 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:38:10.504 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:38:10.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd unset nodeep-scrub 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.122817+0000 osd.1 (osd.1) 457 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.122817+0000 osd.1 (osd.1) 457 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.125365+0000 osd.1 (osd.1) 458 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.125365+0000 osd.1 (osd.1) 458 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: audit 2024-04-26T16:38:09.626274+0000 mon.a (mon.0) 3951 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: audit 2024-04-26T16:38:09.626274+0000 mon.a (mon.0) 3951 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.635010+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.635010+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.701752+0000 osd.4 (osd.4) 435 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.701752+0000 osd.4 (osd.4) 435 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:38:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.705947+0000 osd.4 (osd.4) 436 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.705947+0000 osd.4 (osd.4) 436 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.825166+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.825166+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.827891+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.827891+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.313784+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.c scrub starts 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.313784+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.c scrub starts 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.315926+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.c scrub ok 2024-04-26T16:38:10.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:10 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.315926+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.c scrub ok 2024-04-26T16:38:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.122817+0000 osd.1 (osd.1) 457 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.122817+0000 osd.1 (osd.1) 457 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.125365+0000 osd.1 (osd.1) 458 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.125365+0000 osd.1 (osd.1) 458 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: audit 2024-04-26T16:38:09.626274+0000 mon.a (mon.0) 3951 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: audit 2024-04-26T16:38:09.626274+0000 mon.a (mon.0) 3951 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.635010+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.635010+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.701752+0000 osd.4 (osd.4) 435 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.701752+0000 osd.4 (osd.4) 435 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.705947+0000 osd.4 (osd.4) 436 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.705947+0000 osd.4 (osd.4) 436 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.825166+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:38:11.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.825166+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.827891+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.827891+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.313784+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.c scrub starts 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.313784+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.c scrub starts 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.315926+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.c scrub ok 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.315926+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.c scrub ok 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.122817+0000 osd.1 (osd.1) 457 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.122817+0000 osd.1 (osd.1) 457 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.125365+0000 osd.1 (osd.1) 458 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.125365+0000 osd.1 (osd.1) 458 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: audit 2024-04-26T16:38:09.626274+0000 mon.a (mon.0) 3951 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: audit 2024-04-26T16:38:09.626274+0000 mon.a (mon.0) 3951 : audit [INF] from='client.? 172.21.15.50:0/4238070144' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-04-26T16:38:11.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.635010+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.635010+0000 mon.a (mon.0) 3952 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.701752+0000 osd.4 (osd.4) 435 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.701752+0000 osd.4 (osd.4) 435 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.705947+0000 osd.4 (osd.4) 436 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.705947+0000 osd.4 (osd.4) 436 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.825166+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.825166+0000 osd.0 (osd.0) 489 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.827891+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.827891+0000 osd.0 (osd.0) 490 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.313784+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.c scrub starts 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.313784+0000 osd.2 (osd.2) 421 : cluster [DBG] 2.c scrub starts 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.315926+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.c scrub ok 2024-04-26T16:38:11.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:10 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.315926+0000 osd.2 (osd.2) 422 : cluster [DBG] 2.c scrub ok 2024-04-26T16:38:11.863 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.974417+0000 mgr.y (mgr.24401) 794 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:11.863 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:09.974417+0000 mgr.y (mgr.24401) 794 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:11.863 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.133300+0000 osd.1 (osd.1) 459 : cluster [DBG] 11.c scrub starts 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.133300+0000 osd.1 (osd.1) 459 : cluster [DBG] 11.c scrub starts 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.136338+0000 osd.1 (osd.1) 460 : cluster [DBG] 11.c scrub ok 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.136338+0000 osd.1 (osd.1) 460 : cluster [DBG] 11.c scrub ok 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.270255+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.11 deep-scrub starts 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.270255+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.11 deep-scrub starts 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.272798+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.11 deep-scrub ok 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:10.272798+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.11 deep-scrub ok 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.350033+0000 osd.2 (osd.2) 423 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.350033+0000 osd.2 (osd.2) 423 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.352334+0000 osd.2 (osd.2) 424 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.352334+0000 osd.2 (osd.2) 424 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.437522+0000 mon.a (mon.0) 3953 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-04-26T16:38:11.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:11 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.437522+0000 mon.a (mon.0) 3953 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-04-26T16:38:12.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.974417+0000 mgr.y (mgr.24401) 794 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:09.974417+0000 mgr.y (mgr.24401) 794 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.133300+0000 osd.1 (osd.1) 459 : cluster [DBG] 11.c scrub starts 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.133300+0000 osd.1 (osd.1) 459 : cluster [DBG] 11.c scrub starts 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.136338+0000 osd.1 (osd.1) 460 : cluster [DBG] 11.c scrub ok 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.136338+0000 osd.1 (osd.1) 460 : cluster [DBG] 11.c scrub ok 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.270255+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.11 deep-scrub starts 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.270255+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.11 deep-scrub starts 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.272798+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.11 deep-scrub ok 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:10.272798+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.11 deep-scrub ok 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.350033+0000 osd.2 (osd.2) 423 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.350033+0000 osd.2 (osd.2) 423 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.352334+0000 osd.2 (osd.2) 424 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.352334+0000 osd.2 (osd.2) 424 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:38:12.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.437522+0000 mon.a (mon.0) 3953 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.437522+0000 mon.a (mon.0) 3953 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.974417+0000 mgr.y (mgr.24401) 794 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:09.974417+0000 mgr.y (mgr.24401) 794 : cluster [DBG] pgmap v1367: 164 pgs: 164 active+clean; 583 KiB data, 789 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.133300+0000 osd.1 (osd.1) 459 : cluster [DBG] 11.c scrub starts 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.133300+0000 osd.1 (osd.1) 459 : cluster [DBG] 11.c scrub starts 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.136338+0000 osd.1 (osd.1) 460 : cluster [DBG] 11.c scrub ok 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.136338+0000 osd.1 (osd.1) 460 : cluster [DBG] 11.c scrub ok 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.270255+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.11 deep-scrub starts 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.270255+0000 osd.6 (osd.6) 51 : cluster [DBG] 5.11 deep-scrub starts 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.272798+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.11 deep-scrub ok 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:10.272798+0000 osd.6 (osd.6) 52 : cluster [DBG] 5.11 deep-scrub ok 2024-04-26T16:38:12.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.350033+0000 osd.2 (osd.2) 423 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:38:12.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.350033+0000 osd.2 (osd.2) 423 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:38:12.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.352334+0000 osd.2 (osd.2) 424 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:38:12.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.352334+0000 osd.2 (osd.2) 424 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:38:12.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.437522+0000 mon.a (mon.0) 3953 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-04-26T16:38:12.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:11 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.437522+0000 mon.a (mon.0) 3953 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2024-04-26T16:38:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:12 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.649427+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:38:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:12 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.649427+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:38:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:12 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.656570+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:38:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:12 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.656570+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:38:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.649427+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:38:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.649427+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:38:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.656570+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:38:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.656570+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:38:13.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.649427+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:38:13.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.649427+0000 osd.3 (osd.3) 505 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:38:13.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.656570+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:38:13.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:12 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.656570+0000 osd.3 (osd.3) 506 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:38:13.281 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:38:13.281 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:38:13.281 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:38:13.281 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:38:13.281 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:38:13.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.976137+0000 mgr.y (mgr.24401) 795 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:11.976137+0000 mgr.y (mgr.24401) 795 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.111854+0000 osd.7 (osd.7) 454 : cluster [DBG] 3.f scrub starts 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.111854+0000 osd.7 (osd.7) 454 : cluster [DBG] 3.f scrub starts 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.118305+0000 osd.7 (osd.7) 455 : cluster [DBG] 3.f scrub ok 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.118305+0000 osd.7 (osd.7) 455 : cluster [DBG] 3.f scrub ok 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.641250+0000 osd.3 (osd.3) 507 : cluster [DBG] 11.b scrub starts 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.641250+0000 osd.3 (osd.3) 507 : cluster [DBG] 11.b scrub starts 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.645012+0000 osd.3 (osd.3) 508 : cluster [DBG] 11.b scrub ok 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.645012+0000 osd.3 (osd.3) 508 : cluster [DBG] 11.b scrub ok 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.851340+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.851340+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:38:13.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.853658+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:12.853658+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: audit 2024-04-26T16:38:13.102447+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: audit 2024-04-26T16:38:13.102447+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.378779+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.b scrub starts 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.378779+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.b scrub starts 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.381031+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.b scrub ok 2024-04-26T16:38:13.993 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:13 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.381031+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.b scrub ok 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.976137+0000 mgr.y (mgr.24401) 795 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:11.976137+0000 mgr.y (mgr.24401) 795 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.111854+0000 osd.7 (osd.7) 454 : cluster [DBG] 3.f scrub starts 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.111854+0000 osd.7 (osd.7) 454 : cluster [DBG] 3.f scrub starts 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.118305+0000 osd.7 (osd.7) 455 : cluster [DBG] 3.f scrub ok 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.118305+0000 osd.7 (osd.7) 455 : cluster [DBG] 3.f scrub ok 2024-04-26T16:38:14.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.641250+0000 osd.3 (osd.3) 507 : cluster [DBG] 11.b scrub starts 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.641250+0000 osd.3 (osd.3) 507 : cluster [DBG] 11.b scrub starts 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.645012+0000 osd.3 (osd.3) 508 : cluster [DBG] 11.b scrub ok 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.645012+0000 osd.3 (osd.3) 508 : cluster [DBG] 11.b scrub ok 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.851340+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.851340+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.853658+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:12.853658+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: audit 2024-04-26T16:38:13.102447+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: audit 2024-04-26T16:38:13.102447+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.378779+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.b scrub starts 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.378779+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.b scrub starts 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.381031+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.b scrub ok 2024-04-26T16:38:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.381031+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.b scrub ok 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.976137+0000 mgr.y (mgr.24401) 795 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:11.976137+0000 mgr.y (mgr.24401) 795 : cluster [DBG] pgmap v1369: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.111854+0000 osd.7 (osd.7) 454 : cluster [DBG] 3.f scrub starts 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.111854+0000 osd.7 (osd.7) 454 : cluster [DBG] 3.f scrub starts 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.118305+0000 osd.7 (osd.7) 455 : cluster [DBG] 3.f scrub ok 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.118305+0000 osd.7 (osd.7) 455 : cluster [DBG] 3.f scrub ok 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.641250+0000 osd.3 (osd.3) 507 : cluster [DBG] 11.b scrub starts 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.641250+0000 osd.3 (osd.3) 507 : cluster [DBG] 11.b scrub starts 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.645012+0000 osd.3 (osd.3) 508 : cluster [DBG] 11.b scrub ok 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.645012+0000 osd.3 (osd.3) 508 : cluster [DBG] 11.b scrub ok 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.851340+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.851340+0000 osd.0 (osd.0) 491 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:38:14.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.853658+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:12.853658+0000 osd.0 (osd.0) 492 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: audit 2024-04-26T16:38:13.102447+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: audit 2024-04-26T16:38:13.102447+0000 mon.c (mon.2) 909 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.378779+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.b scrub starts 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.378779+0000 osd.2 (osd.2) 425 : cluster [DBG] 5.b scrub starts 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.381031+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.b scrub ok 2024-04-26T16:38:14.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:13 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.381031+0000 osd.2 (osd.2) 426 : cluster [DBG] 5.b scrub ok 2024-04-26T16:38:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:14.618 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:14.662 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:38:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:38:15.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.079936+0000 osd.7 (osd.7) 456 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.079936+0000 osd.7 (osd.7) 456 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.084333+0000 osd.7 (osd.7) 457 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.084333+0000 osd.7 (osd.7) 457 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.300601+0000 osd.6 (osd.6) 53 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.300601+0000 osd.6 (osd.6) 53 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.304177+0000 osd.6 (osd.6) 54 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:38:15.104 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:14 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.304177+0000 osd.6 (osd.6) 54 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:38:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.079936+0000 osd.7 (osd.7) 456 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:38:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.079936+0000 osd.7 (osd.7) 456 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:38:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.084333+0000 osd.7 (osd.7) 457 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.084333+0000 osd.7 (osd.7) 457 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.300601+0000 osd.6 (osd.6) 53 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.300601+0000 osd.6 (osd.6) 53 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.304177+0000 osd.6 (osd.6) 54 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.304177+0000 osd.6 (osd.6) 54 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.079936+0000 osd.7 (osd.7) 456 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.079936+0000 osd.7 (osd.7) 456 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.084333+0000 osd.7 (osd.7) 457 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.084333+0000 osd.7 (osd.7) 457 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.300601+0000 osd.6 (osd.6) 53 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.300601+0000 osd.6 (osd.6) 53 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:38:15.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.304177+0000 osd.6 (osd.6) 54 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:38:15.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:14 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.304177+0000 osd.6 (osd.6) 54 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:38:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:15 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.977055+0000 mgr.y (mgr.24401) 796 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:16.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:15 smithi190 bash[23868]: cluster 2024-04-26T16:38:13.977055+0000 mgr.y (mgr.24401) 796 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:16.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:15 smithi190 bash[23868]: cluster 2024-04-26T16:38:14.728450+0000 osd.4 (osd.4) 437 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:38:16.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:15 smithi190 bash[23868]: cluster 2024-04-26T16:38:14.728450+0000 osd.4 (osd.4) 437 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:38:16.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:15 smithi190 bash[23868]: cluster 2024-04-26T16:38:14.730800+0000 osd.4 (osd.4) 438 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:38:16.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:15 smithi190 bash[23868]: cluster 2024-04-26T16:38:14.730800+0000 osd.4 (osd.4) 438 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:38:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.977055+0000 mgr.y (mgr.24401) 796 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[29234]: cluster 2024-04-26T16:38:13.977055+0000 mgr.y (mgr.24401) 796 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[29234]: cluster 2024-04-26T16:38:14.728450+0000 osd.4 (osd.4) 437 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:38:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[29234]: cluster 2024-04-26T16:38:14.728450+0000 osd.4 (osd.4) 437 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:38:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[29234]: cluster 2024-04-26T16:38:14.730800+0000 osd.4 (osd.4) 438 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:38:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[29234]: cluster 2024-04-26T16:38:14.730800+0000 osd.4 (osd.4) 438 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:38:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.977055+0000 mgr.y (mgr.24401) 796 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[21463]: cluster 2024-04-26T16:38:13.977055+0000 mgr.y (mgr.24401) 796 : cluster [DBG] pgmap v1370: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[21463]: cluster 2024-04-26T16:38:14.728450+0000 osd.4 (osd.4) 437 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:38:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[21463]: cluster 2024-04-26T16:38:14.728450+0000 osd.4 (osd.4) 437 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:38:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[21463]: cluster 2024-04-26T16:38:14.730800+0000 osd.4 (osd.4) 438 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:38:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:15 smithi050 bash[21463]: cluster 2024-04-26T16:38:14.730800+0000 osd.4 (osd.4) 438 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.627270+0000 osd.3 (osd.3) 509 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.627270+0000 osd.3 (osd.3) 509 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.632160+0000 osd.1 (osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.632160+0000 osd.1 (osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.632695+0000 osd.1 (osd.1) 462 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.632695+0000 osd.1 (osd.1) 462 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.633133+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.633133+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.676447+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.676447+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.435804+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:38:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.435804+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:38:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.438642+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:38:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.438642+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:38:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.439969+0000 mon.a (mon.0) 3955 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-04-26T16:38:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:16 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.439969+0000 mon.a (mon.0) 3955 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-04-26T16:38:17.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.627270+0000 osd.3 (osd.3) 509 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:38:17.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.627270+0000 osd.3 (osd.3) 509 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:38:17.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.632160+0000 osd.1 (osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:17.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.632160+0000 osd.1 (osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.632695+0000 osd.1 (osd.1) 462 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.632695+0000 osd.1 (osd.1) 462 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.633133+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.633133+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.676447+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.676447+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.435804+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.435804+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.438642+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.438642+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.439969+0000 mon.a (mon.0) 3955 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-04-26T16:38:17.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.439969+0000 mon.a (mon.0) 3955 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.627270+0000 osd.3 (osd.3) 509 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.627270+0000 osd.3 (osd.3) 509 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.632160+0000 osd.1 (osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.632160+0000 osd.1 (osd.1) 461 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.632695+0000 osd.1 (osd.1) 462 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.632695+0000 osd.1 (osd.1) 462 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.633133+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.633133+0000 osd.3 (osd.3) 510 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.676447+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-04-26T16:38:17.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.676447+0000 mon.a (mon.0) 3954 : cluster [DBG] osdmap e874: 8 total, 8 up, 8 in 2024-04-26T16:38:17.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.435804+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:38:17.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.435804+0000 osd.2 (osd.2) 427 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:38:17.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.438642+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:38:17.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.438642+0000 osd.2 (osd.2) 428 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:38:17.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.439969+0000 mon.a (mon.0) 3955 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-04-26T16:38:17.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:16 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.439969+0000 mon.a (mon.0) 3955 : cluster [DBG] osdmap e875: 8 total, 8 up, 8 in 2024-04-26T16:38:17.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:38:18.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.978074+0000 mgr.y (mgr.24401) 797 : cluster [DBG] pgmap v1372: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:15.978074+0000 mgr.y (mgr.24401) 797 : cluster [DBG] pgmap v1372: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.159732+0000 osd.1 (osd.1) 463 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.159732+0000 osd.1 (osd.1) 463 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.162162+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:16.162162+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.227466+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.227466+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.232214+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.232214+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.485838+0000 osd.2 (osd.2) 429 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:38:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.485838+0000 osd.2 (osd.2) 429 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:38:18.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.488676+0000 osd.2 (osd.2) 430 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:38:18.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:17 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.488676+0000 osd.2 (osd.2) 430 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:38:18.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.978074+0000 mgr.y (mgr.24401) 797 : cluster [DBG] pgmap v1372: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-04-26T16:38:18.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:15.978074+0000 mgr.y (mgr.24401) 797 : cluster [DBG] pgmap v1372: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-04-26T16:38:18.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.159732+0000 osd.1 (osd.1) 463 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:38:18.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.159732+0000 osd.1 (osd.1) 463 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.162162+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:16.162162+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.227466+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.227466+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.232214+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.232214+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.485838+0000 osd.2 (osd.2) 429 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.485838+0000 osd.2 (osd.2) 429 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.488676+0000 osd.2 (osd.2) 430 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.488676+0000 osd.2 (osd.2) 430 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:38:18.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.978074+0000 mgr.y (mgr.24401) 797 : cluster [DBG] pgmap v1372: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:15.978074+0000 mgr.y (mgr.24401) 797 : cluster [DBG] pgmap v1372: 164 pgs: 164 active+clean; 583 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 806 B/s rd, 0 op/s 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.159732+0000 osd.1 (osd.1) 463 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.159732+0000 osd.1 (osd.1) 463 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.162162+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:16.162162+0000 osd.1 (osd.1) 464 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.227466+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.227466+0000 osd.5 (osd.5) 338 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.232214+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.232214+0000 osd.5 (osd.5) 339 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.485838+0000 osd.2 (osd.2) 429 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.485838+0000 osd.2 (osd.2) 429 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:38:18.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.488676+0000 osd.2 (osd.2) 430 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:38:18.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:17 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.488676+0000 osd.2 (osd.2) 430 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:38:18.788 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:38:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:18 smithi190 bash[23868]: audit 2024-04-26T16:38:17.378942+0000 mgr.y (mgr.24401) 798 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:18 smithi190 bash[23868]: audit 2024-04-26T16:38:17.378942+0000 mgr.y (mgr.24401) 798 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:18 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.220452+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.d scrub starts 2024-04-26T16:38:19.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:18 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.220452+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.d scrub starts 2024-04-26T16:38:19.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:18 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.223216+0000 osd.5 (osd.5) 341 : cluster [DBG] 6.d scrub ok 2024-04-26T16:38:19.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:18 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.223216+0000 osd.5 (osd.5) 341 : cluster [DBG] 6.d scrub ok 2024-04-26T16:38:19.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[21463]: audit 2024-04-26T16:38:17.378942+0000 mgr.y (mgr.24401) 798 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:19.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[21463]: audit 2024-04-26T16:38:17.378942+0000 mgr.y (mgr.24401) 798 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:19.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.220452+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.d scrub starts 2024-04-26T16:38:19.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.220452+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.d scrub starts 2024-04-26T16:38:19.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.223216+0000 osd.5 (osd.5) 341 : cluster [DBG] 6.d scrub ok 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.223216+0000 osd.5 (osd.5) 341 : cluster [DBG] 6.d scrub ok 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[29234]: audit 2024-04-26T16:38:17.378942+0000 mgr.y (mgr.24401) 798 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[29234]: audit 2024-04-26T16:38:17.378942+0000 mgr.y (mgr.24401) 798 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.220452+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.d scrub starts 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.220452+0000 osd.5 (osd.5) 340 : cluster [DBG] 6.d scrub starts 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.223216+0000 osd.5 (osd.5) 341 : cluster [DBG] 6.d scrub ok 2024-04-26T16:38:19.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:18 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.223216+0000 osd.5 (osd.5) 341 : cluster [DBG] 6.d scrub ok 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/4132841089 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 msgr2=0x7ffa0410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/4132841089 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa0410f1c0 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc009f90 tx=0x7ff9fc00b330 comp rx=0 tx=0).stop 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/4132841089 shutdown_connections 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/4132841089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa04108380 0x7ffa04108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/4132841089 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa0410f1c0 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/4132841089 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa04102310 0x7ffa04102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/4132841089 >> 172.21.15.50:0/4132841089 conn(0x7ffa04079290 msgr2=0x7ffa040796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:19.407 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/4132841089 shutdown_connections 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/4132841089 wait complete. 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 librados: starting msgr at 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 librados: starting objecter 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 Processor -- start 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- start start 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 librados: setting wanted keys 2024-04-26T16:38:19.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 librados: calling monclient init 2024-04-26T16:38:19.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa04102310 0x7ffa04103ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa04104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa04108380 0x7ffa04104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7ffa04111960 con 0x7ffa04108380 2024-04-26T16:38:19.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7ffa041117e0 con 0x7ffa04102310 2024-04-26T16:38:19.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7ffa04111ae0 con 0x7ffa04102cd0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa04104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa09a80700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa04108380 0x7ffa04104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa04104530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44006/0 (socket says 172.21.15.50:44006) 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 -- 172.21.15.50:0/2793097038 learned_addr learned my addr 172.21.15.50:0/2793097038 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0927f700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa04102310 0x7ffa04103ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 -- 172.21.15.50:0/2793097038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa04102310 msgr2=0x7ffa04103ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa04102310 0x7ffa04103ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 -- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa04108380 msgr2=0x7ffa04104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa04108380 0x7ffa04104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 -- 172.21.15.50:0/2793097038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa041a75e0 con 0x7ffa04102cd0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa08a7e700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa04104530 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800ec10 tx=0x7ff9f800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ff9f67fc700 10 client.?.objecter ms_handle_connect 0x7ffa04102cd0 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ff9f67fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:19.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f800cd40 con 0x7ffa04102cd0 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7ff9f8004480 con 0x7ffa04102cd0 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa041a78d0 con 0x7ffa04102cd0 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8010430 con 0x7ffa04102cd0 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 10 client.41750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa041a7e10 con 0x7ffa04102cd0 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 librados: init done 2024-04-26T16:38:19.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.410+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa0406a110 con 0x7ffa04102cd0 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7ff9f80105d0 con 0x7ffa04102cd0 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff9f0071f80 0x7ff9f0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(875..875 src has 781..875) ==== 8707+0+0 (secure 0 0 0) 0x7ff9f8091b60 con 0x7ffa04102cd0 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 10 client.41750.objecter ms_dispatch 0x7ffa04001550 osd_map(875..875 src has 781..875) 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 3 client.41750.objecter handle_osd_map got epochs [875,875] > 0 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 3 client.41750.objecter handle_osd_map decoding full epoch 875 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ffa0927f700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff9f0071f80 0x7ff9f0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 20 client.41750.objecter dump_active .. 0 homeless 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ffa0927f700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff9f0071f80 0x7ff9f0074440 secure :-1 s=READY pgs=3850 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000bbb0 tx=0x7ffa0000b590 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:19.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.414+0000 7ff9f67fc700 10 client.41750.objecter ms_handle_connect 0x7ff9f0071f80 2024-04-26T16:38:19.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.418+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7ff9f805d7b0 con 0x7ffa04102cd0 2024-04-26T16:38:19.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/3737471295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0101a10 msgr2=0x7f7ba0101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/3737471295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0101a10 0x7f7ba0101e70 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f7b880098f0 tx=0x7f7b88009c00 comp rx=0 tx=0).stop 2024-04-26T16:38:19.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/3737471295 shutdown_connections 2024-04-26T16:38:19.446 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/3737471295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01094b0 0x7f7ba0109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/3737471295 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba01023b0 0x7f7ba01102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/3737471295 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba0101a10 0x7f7ba0101e70 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/3737471295 >> 172.21.15.50:0/3737471295 conn(0x7f7ba0078990 msgr2=0x7f7ba00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/3737471295 shutdown_connections 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/3737471295 wait complete. 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 librados: starting msgr at 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 librados: starting objecter 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 Processor -- start 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- start start 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 librados: setting wanted keys 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 librados: calling monclient init 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 0x7f7ba0102bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.447 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01023b0 0x7f7ba0103130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba01094b0 0x7f7ba0103670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba0112b20 con 0x7f7ba01094b0 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba01129a0 con 0x7f7ba01023b0 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.446+0000 7f7ba4fdb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7ba0112ca0 con 0x7f7ba0101a10 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 0x7f7ba0102bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 0x7f7ba0102bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:44026/0 (socket says 172.21.15.50:44026) 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 -- 172.21.15.50:0/2461297229 learned_addr learned my addr 172.21.15.50:0/2461297229 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9ed9d700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba01094b0 0x7f7ba0103670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9dd9b700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01023b0 0x7f7ba0103130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 -- 172.21.15.50:0/2461297229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01023b0 msgr2=0x7f7ba0103130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01023b0 0x7f7ba0103130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 -- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba01094b0 msgr2=0x7f7ba0103670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba01094b0 0x7f7ba0103670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.448 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 -- 172.21.15.50:0/2461297229 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba01a7690 con 0x7f7ba0101a10 2024-04-26T16:38:19.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9dd9b700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01023b0 0x7f7ba0103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:38:19.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9e59c700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 0x7f7ba0102bf0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000d950 tx=0x7f7b9000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:19.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 10 client.?.objecter ms_handle_connect 0x7f7ba0101a10 2024-04-26T16:38:19.449 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:19.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90016070 con 0x7f7ba0101a10 2024-04-26T16:38:19.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7b90009da0 con 0x7f7ba0101a10 2024-04-26T16:38:19.450 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90010e60 con 0x7f7ba0101a10 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba01a7980 con 0x7f7ba0101a10 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7ba4fdb700 10 client.41756.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba01a7e40 con 0x7f7ba0101a10 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7ba4fdb700 1 librados: init done 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ba0104d30 con 0x7f7ba0101a10 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7b90019070 con 0x7f7ba0101a10 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b977fe700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7b8c071f80 0x7f7b8c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.450+0000 7f7b9dd9b700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7b8c071f80 0x7f7b8c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(875..875 src has 781..875) ==== 8707+0+0 (secure 0 0 0) 0x7f7b900911b0 con 0x7f7ba0101a10 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b977fe700 10 client.41756.objecter ms_dispatch 0x7f7ba0001550 osd_map(875..875 src has 781..875) 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b977fe700 3 client.41756.objecter handle_osd_map got epochs [875,875] > 0 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b977fe700 3 client.41756.objecter handle_osd_map decoding full epoch 875 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b977fe700 20 client.41756.objecter dump_active .. 0 homeless 2024-04-26T16:38:19.451 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b977fe700 10 client.41756.objecter ms_handle_connect 0x7f7b8c071f80 2024-04-26T16:38:19.452 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.454+0000 7f7b9dd9b700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7b8c071f80 0x7f7b8c074440 secure :-1 s=READY pgs=3851 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009810 tx=0x7f7b88019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:19.455 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.458+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7b90020070 con 0x7f7ba0101a10 2024-04-26T16:38:19.653 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.654+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) -- 0x7ffa041a8040 con 0x7ffa04102cd0 2024-04-26T16:38:19.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.674+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7ba0063860 con 0x7f7b8c071f80 2024-04-26T16:38:19.694 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.694+0000 7f7b977fe700 1 -- 172.21.15.50:0/2461297229 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+426079 (secure 0 0 0) 0x7f7b8808e070 con 0x7f7b8c071f80 2024-04-26T16:38:19.699 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.698+0000 7f7ba4fdb700 10 librados: watch_flush enter 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.698+0000 7f7ba4fdb700 10 librados: watch_flush exit 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.698+0000 7f7ba4fdb700 20 client.41756.objecter shutdown clearing up homeless session... 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.698+0000 7f7ba4fdb700 10 client.41756.objecter successfully canceled tick 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7b8c071f80 msgr2=0x7f7b8c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7b8c071f80 0x7f7b8c074440 secure :-1 s=READY pgs=3851 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009810 tx=0x7f7b88019040 comp rx=0 tx=0).stop 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 msgr2=0x7f7ba0102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 0x7f7ba0102bf0 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000d950 tx=0x7f7b9000dc60 comp rx=0 tx=0).stop 2024-04-26T16:38:19.703 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 shutdown_connections 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7b8c071f80 0x7f7b8c074440 unknown :-1 s=CLOSED pgs=3851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7ba01094b0 0x7f7ba0103670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7ba0101a10 0x7f7ba0102bf0 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 --2- 172.21.15.50:0/2461297229 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ba01023b0 0x7f7ba0103130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 >> 172.21.15.50:0/2461297229 conn(0x7f7ba0078990 msgr2=0x7f7ba00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 shutdown_connections 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 -- 172.21.15.50:0/2461297229 wait complete. 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.702+0000 7f7ba4fdb700 1 librados: shutdown 2024-04-26T16:38:19.704 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:38:19.714 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.718+0000 7ff9f67fc700 1 -- 172.21.15.50:0/2793097038 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd unset", "key": "nodeep-scrub"}]=0 nodeep-scrub is unset v876) ==== 101+0+0 (secure 0 0 0) 0x7ff9f8061460 con 0x7ffa04102cd0 2024-04-26T16:38:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.718+0000 7ffa0b4e3700 10 librados: watch_flush enter 2024-04-26T16:38:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.718+0000 7ffa0b4e3700 10 librados: watch_flush exit 2024-04-26T16:38:19.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.718+0000 7ffa0b4e3700 20 client.41750.objecter shutdown clearing up homeless session... 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 10 client.41750.objecter successfully canceled tick 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff9f0071f80 msgr2=0x7ff9f0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff9f0071f80 0x7ff9f0074440 secure :-1 s=READY pgs=3850 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000bbb0 tx=0x7ffa0000b590 comp rx=0 tx=0).stop 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 msgr2=0x7ffa04104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa04104530 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800ec10 tx=0x7ff9f800ef20 comp rx=0 tx=0).stop 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 shutdown_connections 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7ff9f0071f80 0x7ff9f0074440 unknown :-1 s=CLOSED pgs=3850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffa04102310 0x7ffa04103ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7ffa04108380 0x7ffa04104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 --2- 172.21.15.50:0/2793097038 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7ffa04102cd0 0x7ffa04104530 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 >> 172.21.15.50:0/2793097038 conn(0x7ffa04079290 msgr2=0x7ffa0406b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 shutdown_connections 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 -- 172.21.15.50:0/2793097038 wait complete. 2024-04-26T16:38:19.719 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:19.722+0000 7ffa0b4e3700 1 librados: shutdown 2024-04-26T16:38:19.720 INFO:teuthology.orchestra.run.smithi050.stderr:nodeep-scrub is unset 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.979277+0000 mgr.y (mgr.24401) 799 : cluster [DBG] pgmap v1374: 164 pgs: 164 active+clean; 583 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:17.979277+0000 mgr.y (mgr.24401) 799 : cluster [DBG] pgmap v1374: 164 pgs: 164 active+clean; 583 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.349257+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.349257+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.351865+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.351865+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.562471+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:38:20.046 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.562471+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.564877+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:18.564877+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.243054+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.16 deep-scrub starts 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.243054+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.16 deep-scrub starts 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.245600+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.16 deep-scrub ok 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.245600+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.16 deep-scrub ok 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: audit 2024-04-26T16:38:19.659003+0000 mon.c (mon.2) 910 : audit [INF] from='client.? 172.21.15.50:0/2793097038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: audit 2024-04-26T16:38:19.659003+0000 mon.c (mon.2) 910 : audit [INF] from='client.? 172.21.15.50:0/2793097038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: audit 2024-04-26T16:38:19.659645+0000 mon.a (mon.0) 3956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[21463]: audit 2024-04-26T16:38:19.659645+0000 mon.a (mon.0) 3956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.979277+0000 mgr.y (mgr.24401) 799 : cluster [DBG] pgmap v1374: 164 pgs: 164 active+clean; 583 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:38:20.047 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:17.979277+0000 mgr.y (mgr.24401) 799 : cluster [DBG] pgmap v1374: 164 pgs: 164 active+clean; 583 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.349257+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.349257+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.351865+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.351865+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.562471+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.562471+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.564877+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:18.564877+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.243054+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.16 deep-scrub starts 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.243054+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.16 deep-scrub starts 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.245600+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.16 deep-scrub ok 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.245600+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.16 deep-scrub ok 2024-04-26T16:38:20.048 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: audit 2024-04-26T16:38:19.659003+0000 mon.c (mon.2) 910 : audit [INF] from='client.? 172.21.15.50:0/2793097038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: audit 2024-04-26T16:38:19.659003+0000 mon.c (mon.2) 910 : audit [INF] from='client.? 172.21.15.50:0/2793097038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: audit 2024-04-26T16:38:19.659645+0000 mon.a (mon.0) 3956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.049 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:19 smithi050 bash[29234]: audit 2024-04-26T16:38:19.659645+0000 mon.a (mon.0) 3956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.979277+0000 mgr.y (mgr.24401) 799 : cluster [DBG] pgmap v1374: 164 pgs: 164 active+clean; 583 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:38:20.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:17.979277+0000 mgr.y (mgr.24401) 799 : cluster [DBG] pgmap v1374: 164 pgs: 164 active+clean; 583 KiB data, 791 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:38:20.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.349257+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:38:20.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.349257+0000 osd.6 (osd.6) 55 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.351865+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.351865+0000 osd.6 (osd.6) 56 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.562471+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.562471+0000 osd.3 (osd.3) 511 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.564877+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:18.564877+0000 osd.3 (osd.3) 512 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.243054+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.16 deep-scrub starts 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.243054+0000 osd.5 (osd.5) 342 : cluster [DBG] 2.16 deep-scrub starts 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.245600+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.16 deep-scrub ok 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.245600+0000 osd.5 (osd.5) 343 : cluster [DBG] 2.16 deep-scrub ok 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: audit 2024-04-26T16:38:19.659003+0000 mon.c (mon.2) 910 : audit [INF] from='client.? 172.21.15.50:0/2793097038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: audit 2024-04-26T16:38:19.659003+0000 mon.c (mon.2) 910 : audit [INF] from='client.? 172.21.15.50:0/2793097038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: audit 2024-04-26T16:38:19.659645+0000 mon.a (mon.0) 3956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:19 smithi190 bash[23868]: audit 2024-04-26T16:38:19.659645+0000 mon.a (mon.0) 3956 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-04-26T16:38:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.564488+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:38:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.564488+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:38:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.568177+0000 osd.3 (osd.3) 514 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:38:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.568177+0000 osd.3 (osd.3) 514 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: audit 2024-04-26T16:38:19.677594+0000 mgr.y (mgr.24401) 800 : audit [DBG] from='client.41756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: audit 2024-04-26T16:38:19.677594+0000 mgr.y (mgr.24401) 800 : audit [DBG] from='client.41756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: audit 2024-04-26T16:38:19.712123+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: audit 2024-04-26T16:38:19.712123+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.715693+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.715693+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.896360+0000 osd.0 (osd.0) 493 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.896360+0000 osd.0 (osd.0) 493 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.900150+0000 osd.0 (osd.0) 494 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.900150+0000 osd.0 (osd.0) 494 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.564488+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:38:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.564488+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.568177+0000 osd.3 (osd.3) 514 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.568177+0000 osd.3 (osd.3) 514 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: audit 2024-04-26T16:38:19.677594+0000 mgr.y (mgr.24401) 800 : audit [DBG] from='client.41756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: audit 2024-04-26T16:38:19.677594+0000 mgr.y (mgr.24401) 800 : audit [DBG] from='client.41756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: audit 2024-04-26T16:38:19.712123+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: audit 2024-04-26T16:38:19.712123+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.715693+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.715693+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.896360+0000 osd.0 (osd.0) 493 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.896360+0000 osd.0 (osd.0) 493 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.900150+0000 osd.0 (osd.0) 494 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:38:20.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:20 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.900150+0000 osd.0 (osd.0) 494 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.564488+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.564488+0000 osd.3 (osd.3) 513 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.568177+0000 osd.3 (osd.3) 514 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.568177+0000 osd.3 (osd.3) 514 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: audit 2024-04-26T16:38:19.677594+0000 mgr.y (mgr.24401) 800 : audit [DBG] from='client.41756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: audit 2024-04-26T16:38:19.677594+0000 mgr.y (mgr.24401) 800 : audit [DBG] from='client.41756 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: audit 2024-04-26T16:38:19.712123+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: audit 2024-04-26T16:38:19.712123+0000 mon.a (mon.0) 3957 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.715693+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.715693+0000 mon.a (mon.0) 3958 : cluster [DBG] osdmap e876: 8 total, 8 up, 8 in 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.896360+0000 osd.0 (osd.0) 493 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.896360+0000 osd.0 (osd.0) 493 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:38:21.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.900150+0000 osd.0 (osd.0) 494 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:38:21.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:20 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.900150+0000 osd.0 (osd.0) 494 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:38:21.161 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1374,"stamp":"2024-04-26T16:38:17.978424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596512,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3027,"num_read_kb":2873,"num_write":1194,"num_write_kb":2468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1774538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1355,"ondisk_log_size":1355,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":809576,"kb_used_data":42376,"kb_used_omap":34,"kb_used_meta":765149,"kb_avail":749151640,"statfs":{"total":767960285184,"available":767131279360,"internally_reserved":0,"allocated":43393024,"data_stored":22096685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34917,"internal_metadata":783513499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.546389"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439100+0000","last_change":"2024-04-26T16:38:09.296402+0000","last_active":"2024-04-26T16:38:16.439100+0000","last_peered":"2024-04-26T16:38:16.439100+0000","last_clean":"2024-04-26T16:38:16.439100+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:38:16.439100+0000","last_undegraded":"2024-04-26T16:38:16.439100+0000","last_fullsized":"2024-04-26T16:38:16.439100+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.296138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.296138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.902182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"747'16","reported_seq":1923,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442558+0000","last_change":"2024-04-26T16:36:20.159718+0000","last_active":"2024-04-26T16:38:16.442558+0000","last_peered":"2024-04-26T16:38:16.442558+0000","last_clean":"2024-04-26T16:38:16.442558+0000","last_became_active":"2024-04-26T16:35:25.395438+0000","last_became_peered":"2024-04-26T16:35:25.395438+0000","last_unstale":"2024-04-26T16:38:16.442558+0000","last_undegraded":"2024-04-26T16:38:16.442558+0000","last_fullsized":"2024-04-26T16:38:16.442558+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:36:20.159412+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:36:20.159412+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"857'46","reported_seq":1750,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.444414+0000","last_change":"2024-04-26T16:38:12.118560+0000","last_active":"2024-04-26T16:38:16.444414+0000","last_peered":"2024-04-26T16:38:16.444414+0000","last_clean":"2024-04-26T16:38:16.444414+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:38:16.444414+0000","last_undegraded":"2024-04-26T16:38:16.444414+0000","last_fullsized":"2024-04-26T16:38:16.444414+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:38:12.118315+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:38:12.118315+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.737219+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1688,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.438872+0000","last_change":"2024-04-26T16:38:16.438920+0000","last_active":"2024-04-26T16:38:16.438872+0000","last_peered":"2024-04-26T16:38:16.438872+0000","last_clean":"2024-04-26T16:38:16.438872+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:38:16.438872+0000","last_undegraded":"2024-04-26T16:38:16.438872+0000","last_fullsized":"2024-04-26T16:38:16.438872+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:16.438650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:16.438650+0000","last_clean_scrub_stamp":"2024-04-26T16:38:16.438650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.590311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":596,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439669+0000","last_change":"2024-04-26T16:36:29.464449+0000","last_active":"2024-04-26T16:38:16.439669+0000","last_peered":"2024-04-26T16:38:16.439669+0000","last_clean":"2024-04-26T16:38:16.439669+0000","last_became_active":"2024-04-26T16:35:25.382820+0000","last_became_peered":"2024-04-26T16:35:25.382820+0000","last_unstale":"2024-04-26T16:38:16.439669+0000","last_undegraded":"2024-04-26T16:38:16.439669+0000","last_fullsized":"2024-04-26T16:38:16.439669+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:29.464241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:29.464241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1518,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441690+0000","last_change":"2024-04-26T16:36:37.185673+0000","last_active":"2024-04-26T16:38:16.441690+0000","last_peered":"2024-04-26T16:38:16.441690+0000","last_clean":"2024-04-26T16:38:16.441690+0000","last_became_active":"2024-04-26T16:35:25.399977+0000","last_became_peered":"2024-04-26T16:35:25.399977+0000","last_unstale":"2024-04-26T16:38:16.441690+0000","last_undegraded":"2024-04-26T16:38:16.441690+0000","last_fullsized":"2024-04-26T16:38:16.441690+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:37.185574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:37.185574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"857'30","reported_seq":623,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439424+0000","last_change":"2024-04-26T16:36:21.382720+0000","last_active":"2024-04-26T16:38:16.439424+0000","last_peered":"2024-04-26T16:38:16.439424+0000","last_clean":"2024-04-26T16:38:16.439424+0000","last_became_active":"2024-04-26T16:35:25.393291+0000","last_became_peered":"2024-04-26T16:35:25.393291+0000","last_unstale":"2024-04-26T16:38:16.439424+0000","last_undegraded":"2024-04-26T16:38:16.439424+0000","last_fullsized":"2024-04-26T16:38:16.439424+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'20","last_scrub_stamp":"2024-04-26T16:36:21.382342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:21.382342+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1682,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.676337+0000","last_change":"2024-04-26T16:36:49.190188+0000","last_active":"2024-04-26T16:38:15.676337+0000","last_peered":"2024-04-26T16:38:15.676337+0000","last_clean":"2024-04-26T16:38:15.676337+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:38:15.676337+0000","last_undegraded":"2024-04-26T16:38:15.676337+0000","last_fullsized":"2024-04-26T16:38:15.676337+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:49.190081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:49.190081+0000","last_clean_scrub_stamp":"2024-04-26T16:36:49.190081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1680,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435333+0000","last_change":"2024-04-26T16:36:45.742872+0000","last_active":"2024-04-26T16:38:16.435333+0000","last_peered":"2024-04-26T16:38:16.435333+0000","last_clean":"2024-04-26T16:38:16.435333+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:38:16.435333+0000","last_undegraded":"2024-04-26T16:38:16.435333+0000","last_fullsized":"2024-04-26T16:38:16.435333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:45.741598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:45.741598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1676,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.606787+0000","last_change":"2024-04-26T16:36:54.962722+0000","last_active":"2024-04-26T16:38:16.606787+0000","last_peered":"2024-04-26T16:38:16.606787+0000","last_clean":"2024-04-26T16:38:16.606787+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:38:16.606787+0000","last_undegraded":"2024-04-26T16:38:16.606787+0000","last_fullsized":"2024-04-26T16:38:16.606787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:54.962598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:36:54.962598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1538,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443464+0000","last_change":"2024-04-26T16:36:35.211569+0000","last_active":"2024-04-26T16:38:16.443464+0000","last_peered":"2024-04-26T16:38:16.443464+0000","last_clean":"2024-04-26T16:38:16.443464+0000","last_became_active":"2024-04-26T16:35:25.385146+0000","last_became_peered":"2024-04-26T16:35:25.385146+0000","last_unstale":"2024-04-26T16:38:16.443464+0000","last_undegraded":"2024-04-26T16:38:16.443464+0000","last_fullsized":"2024-04-26T16:38:16.443464+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:35.211444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:35.211444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"857'53","reported_seq":1796,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443668+0000","last_change":"2024-04-26T16:36:42.185671+0000","last_active":"2024-04-26T16:38:16.443668+0000","last_peered":"2024-04-26T16:38:16.443668+0000","last_clean":"2024-04-26T16:38:16.443668+0000","last_became_active":"2024-04-26T16:35:25.387497+0000","last_became_peered":"2024-04-26T16:35:25.387497+0000","last_unstale":"2024-04-26T16:38:16.443668+0000","last_undegraded":"2024-04-26T16:38:16.443668+0000","last_fullsized":"2024-04-26T16:38:16.443668+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'39","last_scrub_stamp":"2024-04-26T16:36:42.185411+0000","last_deep_scrub":"612'39","last_deep_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_clean_scrub_stamp":"2024-04-26T16:36:42.185411+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1708,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437489+0000","last_change":"2024-04-26T16:38:10.316140+0000","last_active":"2024-04-26T16:38:16.437489+0000","last_peered":"2024-04-26T16:38:16.437489+0000","last_clean":"2024-04-26T16:38:16.437489+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:38:16.437489+0000","last_undegraded":"2024-04-26T16:38:16.437489+0000","last_fullsized":"2024-04-26T16:38:16.437489+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:10.315939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:10.315939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:30.665145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1700,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435712+0000","last_change":"2024-04-26T16:38:13.381270+0000","last_active":"2024-04-26T16:38:16.435712+0000","last_peered":"2024-04-26T16:38:16.435712+0000","last_clean":"2024-04-26T16:38:16.435712+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:38:16.435712+0000","last_undegraded":"2024-04-26T16:38:16.435712+0000","last_fullsized":"2024-04-26T16:38:16.435712+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:13.381046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:13.381046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:32.258914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1644,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440983+0000","last_change":"2024-04-26T16:37:59.109107+0000","last_active":"2024-04-26T16:38:16.440983+0000","last_peered":"2024-04-26T16:38:16.440983+0000","last_clean":"2024-04-26T16:38:16.440983+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:38:16.440983+0000","last_undegraded":"2024-04-26T16:38:16.440983+0000","last_fullsized":"2024-04-26T16:38:16.440983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:59.108820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.108820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.334820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1532,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435930+0000","last_change":"2024-04-26T16:36:53.546706+0000","last_active":"2024-04-26T16:38:16.435930+0000","last_peered":"2024-04-26T16:38:16.435930+0000","last_clean":"2024-04-26T16:38:16.435930+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:38:16.435930+0000","last_undegraded":"2024-04-26T16:38:16.435930+0000","last_fullsized":"2024-04-26T16:38:16.435930+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:53.546496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:24.770536+0000","last_clean_scrub_stamp":"2024-04-26T16:36:53.546496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:16.703396+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"857'51","reported_seq":1427,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656186+0000","last_change":"2024-04-26T16:37:48.340319+0000","last_active":"2024-04-26T16:38:11.656186+0000","last_peered":"2024-04-26T16:38:11.656186+0000","last_clean":"2024-04-26T16:38:11.656186+0000","last_became_active":"2024-04-26T16:35:25.414489+0000","last_became_peered":"2024-04-26T16:35:25.414489+0000","last_unstale":"2024-04-26T16:38:11.656186+0000","last_undegraded":"2024-04-26T16:38:11.656186+0000","last_fullsized":"2024-04-26T16:38:11.656186+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'51","last_scrub_stamp":"2024-04-26T16:37:48.340220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.340220+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:54.505895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1668,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442481+0000","last_change":"2024-04-26T16:36:50.250362+0000","last_active":"2024-04-26T16:38:16.442481+0000","last_peered":"2024-04-26T16:38:16.442481+0000","last_clean":"2024-04-26T16:38:16.442481+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:38:16.442481+0000","last_undegraded":"2024-04-26T16:38:16.442481+0000","last_fullsized":"2024-04-26T16:38:16.442481+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:50.250252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:36:50.250252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1682,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.676931+0000","last_change":"2024-04-26T16:36:36.247286+0000","last_active":"2024-04-26T16:38:15.676931+0000","last_peered":"2024-04-26T16:38:15.676931+0000","last_clean":"2024-04-26T16:38:15.676931+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:38:15.676931+0000","last_undegraded":"2024-04-26T16:38:15.676931+0000","last_fullsized":"2024-04-26T16:38:15.676931+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:36.246978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:29.051284+0000","last_clean_scrub_stamp":"2024-04-26T16:36:36.246978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1676,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437173+0000","last_change":"2024-04-26T16:36:48.691806+0000","last_active":"2024-04-26T16:38:16.437173+0000","last_peered":"2024-04-26T16:38:16.437173+0000","last_clean":"2024-04-26T16:38:16.437173+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:38:16.437173+0000","last_undegraded":"2024-04-26T16:38:16.437173+0000","last_fullsized":"2024-04-26T16:38:16.437173+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.691439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:16.826623+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.691439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1151,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656445+0000","last_change":"2024-04-26T16:37:50.334215+0000","last_active":"2024-04-26T16:38:11.656445+0000","last_peered":"2024-04-26T16:38:11.656445+0000","last_clean":"2024-04-26T16:38:11.656445+0000","last_became_active":"2024-04-26T16:35:25.388195+0000","last_became_peered":"2024-04-26T16:35:25.388195+0000","last_unstale":"2024-04-26T16:38:11.656445+0000","last_undegraded":"2024-04-26T16:38:11.656445+0000","last_fullsized":"2024-04-26T16:38:11.656445+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:50.333993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:50.333993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.406018+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"857'25","reported_seq":1698,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440884+0000","last_change":"2024-04-26T16:36:53.706447+0000","last_active":"2024-04-26T16:38:16.440884+0000","last_peered":"2024-04-26T16:38:16.440884+0000","last_clean":"2024-04-26T16:38:16.440884+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:38:16.440884+0000","last_undegraded":"2024-04-26T16:38:16.440884+0000","last_fullsized":"2024-04-26T16:38:16.440884+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:36:53.706307+0000","last_deep_scrub":"122'13","last_deep_scrub_stamp":"2024-04-26T16:21:23.022966+0000","last_clean_scrub_stamp":"2024-04-26T16:36:53.706307+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1690,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.678759+0000","last_change":"2024-04-26T16:38:08.143051+0000","last_active":"2024-04-26T16:38:15.678759+0000","last_peered":"2024-04-26T16:38:15.678759+0000","last_clean":"2024-04-26T16:38:15.678759+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:38:15.678759+0000","last_undegraded":"2024-04-26T16:38:15.678759+0000","last_fullsized":"2024-04-26T16:38:15.678759+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:08.142728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:08.142728+0000","last_clean_scrub_stamp":"2024-04-26T16:38:08.142728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.038547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":1968,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.436623+0000","last_change":"2024-04-26T16:38:00.436674+0000","last_active":"2024-04-26T16:38:16.436623+0000","last_peered":"2024-04-26T16:38:16.436623+0000","last_clean":"2024-04-26T16:38:16.436623+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:38:16.436623+0000","last_undegraded":"2024-04-26T16:38:16.436623+0000","last_fullsized":"2024-04-26T16:38:16.436623+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:38:00.436448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.436448+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:28.478392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1682,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.329027+0000","last_change":"2024-04-26T16:37:56.784808+0000","last_active":"2024-04-26T16:38:16.329027+0000","last_peered":"2024-04-26T16:38:16.329027+0000","last_clean":"2024-04-26T16:38:16.329027+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:38:16.329027+0000","last_undegraded":"2024-04-26T16:38:16.329027+0000","last_fullsized":"2024-04-26T16:38:16.329027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.784687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.893722+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.784687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.270962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1532,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.678221+0000","last_change":"2024-04-26T16:37:56.159369+0000","last_active":"2024-04-26T16:38:15.678221+0000","last_peered":"2024-04-26T16:38:15.678221+0000","last_clean":"2024-04-26T16:38:15.678221+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:38:15.678221+0000","last_undegraded":"2024-04-26T16:38:15.678221+0000","last_fullsized":"2024-04-26T16:38:15.678221+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.159158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.823331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"857'46","reported_seq":1728,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441625+0000","last_change":"2024-04-26T16:36:45.647969+0000","last_active":"2024-04-26T16:38:16.441625+0000","last_peered":"2024-04-26T16:38:16.441625+0000","last_clean":"2024-04-26T16:38:16.441625+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:38:16.441625+0000","last_undegraded":"2024-04-26T16:38:16.441625+0000","last_fullsized":"2024-04-26T16:38:16.441625+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"815'34","last_scrub_stamp":"2024-04-26T16:36:45.647529+0000","last_deep_scrub":"815'34","last_deep_scrub_stamp":"2024-04-26T16:36:45.647529+0000","last_clean_scrub_stamp":"2024-04-26T16:36:45.647529+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1696,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.674812+0000","last_change":"2024-04-26T16:36:39.152888+0000","last_active":"2024-04-26T16:38:15.674812+0000","last_peered":"2024-04-26T16:38:15.674812+0000","last_clean":"2024-04-26T16:38:15.674812+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:38:15.674812+0000","last_undegraded":"2024-04-26T16:38:15.674812+0000","last_fullsized":"2024-04-26T16:38:15.674812+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:39.152743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:36:39.152743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":1959,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.328419+0000","last_change":"2024-04-26T16:37:47.846639+0000","last_active":"2024-04-26T16:38:16.328419+0000","last_peered":"2024-04-26T16:38:16.328419+0000","last_clean":"2024-04-26T16:38:16.328419+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:38:16.328419+0000","last_undegraded":"2024-04-26T16:38:16.328419+0000","last_fullsized":"2024-04-26T16:38:16.328419+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:37:47.846487+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.846487+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:51.693727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":595,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439502+0000","last_change":"2024-04-26T16:35:24.556605+0000","last_active":"2024-04-26T16:38:16.439502+0000","last_peered":"2024-04-26T16:38:16.439502+0000","last_clean":"2024-04-26T16:38:16.439502+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:38:16.439502+0000","last_undegraded":"2024-04-26T16:38:16.439502+0000","last_fullsized":"2024-04-26T16:38:16.439502+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:24.556379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:24.556379+0000","last_clean_scrub_stamp":"2024-04-26T16:35:24.556379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":1172,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:13.304277+0000","last_change":"2024-04-26T16:38:13.304318+0000","last_active":"2024-04-26T16:38:13.304277+0000","last_peered":"2024-04-26T16:38:13.304277+0000","last_clean":"2024-04-26T16:38:13.304277+0000","last_became_active":"2024-04-26T16:35:25.407637+0000","last_became_peered":"2024-04-26T16:35:25.407637+0000","last_unstale":"2024-04-26T16:38:13.304277+0000","last_undegraded":"2024-04-26T16:38:13.304277+0000","last_fullsized":"2024-04-26T16:38:13.304277+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:38:13.304185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:13.304185+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:15.663444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"856'29","reported_seq":1728,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327197+0000","last_change":"2024-04-26T16:38:09.706190+0000","last_active":"2024-04-26T16:38:16.327197+0000","last_peered":"2024-04-26T16:38:16.327197+0000","last_clean":"2024-04-26T16:38:16.327197+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:38:16.327197+0000","last_undegraded":"2024-04-26T16:38:16.327197+0000","last_fullsized":"2024-04-26T16:38:16.327197+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'29","last_scrub_stamp":"2024-04-26T16:38:09.705960+0000","last_deep_scrub":"824'23","last_deep_scrub_stamp":"2024-04-26T16:36:53.947636+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.705960+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:37.529754+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1672,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441733+0000","last_change":"2024-04-26T16:38:03.141298+0000","last_active":"2024-04-26T16:38:16.441733+0000","last_peered":"2024-04-26T16:38:16.441733+0000","last_clean":"2024-04-26T16:38:16.441733+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:38:16.441733+0000","last_undegraded":"2024-04-26T16:38:16.441733+0000","last_fullsized":"2024-04-26T16:38:16.441733+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:38:03.140926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.140926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.283756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":596,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439910+0000","last_change":"2024-04-26T16:36:13.340444+0000","last_active":"2024-04-26T16:38:16.439910+0000","last_peered":"2024-04-26T16:38:16.439910+0000","last_clean":"2024-04-26T16:38:16.439910+0000","last_became_active":"2024-04-26T16:35:25.382698+0000","last_became_peered":"2024-04-26T16:35:25.382698+0000","last_unstale":"2024-04-26T16:38:16.439910+0000","last_undegraded":"2024-04-26T16:38:16.439910+0000","last_fullsized":"2024-04-26T16:38:16.439910+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:13.340230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:36:13.340230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1674,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440668+0000","last_change":"2024-04-26T16:37:54.772260+0000","last_active":"2024-04-26T16:38:16.440668+0000","last_peered":"2024-04-26T16:38:16.440668+0000","last_clean":"2024-04-26T16:38:16.440668+0000","last_became_active":"2024-04-26T16:35:25.416938+0000","last_became_peered":"2024-04-26T16:35:25.416938+0000","last_unstale":"2024-04-26T16:38:16.440668+0000","last_undegraded":"2024-04-26T16:38:16.440668+0000","last_fullsized":"2024-04-26T16:38:16.440668+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:54.772049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:54.772049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.353649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1535,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.328753+0000","last_change":"2024-04-26T16:37:58.725326+0000","last_active":"2024-04-26T16:38:16.328753+0000","last_peered":"2024-04-26T16:38:16.328753+0000","last_clean":"2024-04-26T16:38:16.328753+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:38:16.328753+0000","last_undegraded":"2024-04-26T16:38:16.328753+0000","last_fullsized":"2024-04-26T16:38:16.328753+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.725080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.725080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.633294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1918,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441811+0000","last_change":"2024-04-26T16:37:47.750495+0000","last_active":"2024-04-26T16:38:16.441811+0000","last_peered":"2024-04-26T16:38:16.441811+0000","last_clean":"2024-04-26T16:38:16.441811+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:38:16.441811+0000","last_undegraded":"2024-04-26T16:38:16.441811+0000","last_fullsized":"2024-04-26T16:38:16.441811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:37:47.750268+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.750268+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.703331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1709,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327411+0000","last_change":"2024-04-26T16:37:52.815375+0000","last_active":"2024-04-26T16:38:16.327411+0000","last_peered":"2024-04-26T16:38:16.327411+0000","last_clean":"2024-04-26T16:38:16.327411+0000","last_became_active":"2024-04-26T16:35:25.411443+0000","last_became_peered":"2024-04-26T16:35:25.411443+0000","last_unstale":"2024-04-26T16:38:16.327411+0000","last_undegraded":"2024-04-26T16:38:16.327411+0000","last_fullsized":"2024-04-26T16:38:16.327411+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.815039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.815039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:12.392298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"857'42","reported_seq":1744,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607796+0000","last_change":"2024-04-26T16:38:05.806435+0000","last_active":"2024-04-26T16:38:16.607796+0000","last_peered":"2024-04-26T16:38:16.607796+0000","last_clean":"2024-04-26T16:38:16.607796+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:38:16.607796+0000","last_undegraded":"2024-04-26T16:38:16.607796+0000","last_fullsized":"2024-04-26T16:38:16.607796+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'42","last_scrub_stamp":"2024-04-26T16:38:05.806187+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.806187+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:34.319055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1275,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656714+0000","last_change":"2024-04-26T16:37:06.357605+0000","last_active":"2024-04-26T16:38:11.656714+0000","last_peered":"2024-04-26T16:38:11.656714+0000","last_clean":"2024-04-26T16:38:11.656714+0000","last_became_active":"2024-04-26T16:35:25.406996+0000","last_became_peered":"2024-04-26T16:35:25.406996+0000","last_unstale":"2024-04-26T16:38:11.656714+0000","last_undegraded":"2024-04-26T16:38:11.656714+0000","last_fullsized":"2024-04-26T16:38:11.656714+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:06.357237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:37:06.357237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:29.140874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1662,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442032+0000","last_change":"2024-04-26T16:36:39.157157+0000","last_active":"2024-04-26T16:38:16.442032+0000","last_peered":"2024-04-26T16:38:16.442032+0000","last_clean":"2024-04-26T16:38:16.442032+0000","last_became_active":"2024-04-26T16:35:25.391371+0000","last_became_peered":"2024-04-26T16:35:25.391371+0000","last_unstale":"2024-04-26T16:38:16.442032+0000","last_undegraded":"2024-04-26T16:38:16.442032+0000","last_fullsized":"2024-04-26T16:38:16.442032+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:39.156939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:39.156939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1507,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.444566+0000","last_change":"2024-04-26T16:37:01.201260+0000","last_active":"2024-04-26T16:38:16.444566+0000","last_peered":"2024-04-26T16:38:16.444566+0000","last_clean":"2024-04-26T16:38:16.444566+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:38:16.444566+0000","last_undegraded":"2024-04-26T16:38:16.444566+0000","last_fullsized":"2024-04-26T16:38:16.444566+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.200920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.200920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.711340+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1686,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327173+0000","last_change":"2024-04-26T16:38:06.771968+0000","last_active":"2024-04-26T16:38:16.327173+0000","last_peered":"2024-04-26T16:38:16.327173+0000","last_clean":"2024-04-26T16:38:16.327173+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:38:16.327173+0000","last_undegraded":"2024-04-26T16:38:16.327173+0000","last_fullsized":"2024-04-26T16:38:16.327173+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:06.771754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:38:06.771754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.556312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":2893,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441699+0000","last_change":"2024-04-26T16:37:01.745491+0000","last_active":"2024-04-26T16:38:16.441699+0000","last_peered":"2024-04-26T16:38:16.441699+0000","last_clean":"2024-04-26T16:38:16.441699+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:38:16.441699+0000","last_undegraded":"2024-04-26T16:38:16.441699+0000","last_fullsized":"2024-04-26T16:38:16.441699+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:37:01.745268+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.745268+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.918491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1224,"num_read_kb":1219,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"857'26","reported_seq":1704,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441049+0000","last_change":"2024-04-26T16:37:03.724692+0000","last_active":"2024-04-26T16:38:16.441049+0000","last_peered":"2024-04-26T16:38:16.441049+0000","last_clean":"2024-04-26T16:38:16.441049+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:38:16.441049+0000","last_undegraded":"2024-04-26T16:38:16.441049+0000","last_fullsized":"2024-04-26T16:38:16.441049+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:37:03.724436+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:37:03.724436+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:28.814780+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1702,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.677334+0000","last_change":"2024-04-26T16:38:04.165569+0000","last_active":"2024-04-26T16:38:15.677334+0000","last_peered":"2024-04-26T16:38:15.677334+0000","last_clean":"2024-04-26T16:38:15.677334+0000","last_became_active":"2024-04-26T16:35:25.398461+0000","last_became_peered":"2024-04-26T16:35:25.398461+0000","last_unstale":"2024-04-26T16:38:15.677334+0000","last_undegraded":"2024-04-26T16:38:15.677334+0000","last_fullsized":"2024-04-26T16:38:15.677334+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:38:04.165359+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.165359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.874714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1674,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327308+0000","last_change":"2024-04-26T16:36:57.977447+0000","last_active":"2024-04-26T16:38:16.327308+0000","last_peered":"2024-04-26T16:38:16.327308+0000","last_clean":"2024-04-26T16:38:16.327308+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:38:16.327308+0000","last_undegraded":"2024-04-26T16:38:16.327308+0000","last_fullsized":"2024-04-26T16:38:16.327308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:57.977272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.977272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:25.432898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1531,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.675700+0000","last_change":"2024-04-26T16:38:10.136556+0000","last_active":"2024-04-26T16:38:15.675700+0000","last_peered":"2024-04-26T16:38:15.675700+0000","last_clean":"2024-04-26T16:38:15.675700+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:38:15.675700+0000","last_undegraded":"2024-04-26T16:38:15.675700+0000","last_fullsized":"2024-04-26T16:38:15.675700+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:10.136352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:10.136352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.456445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1269,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.655827+0000","last_change":"2024-04-26T16:38:02.229476+0000","last_active":"2024-04-26T16:38:11.655827+0000","last_peered":"2024-04-26T16:38:11.655827+0000","last_clean":"2024-04-26T16:38:11.655827+0000","last_became_active":"2024-04-26T16:35:25.404395+0000","last_became_peered":"2024-04-26T16:35:25.404395+0000","last_unstale":"2024-04-26T16:38:11.655827+0000","last_undegraded":"2024-04-26T16:38:11.655827+0000","last_fullsized":"2024-04-26T16:38:11.655827+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.229233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.229233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.152859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"857'66","reported_seq":2088,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.676009+0000","last_change":"2024-04-26T16:38:03.160319+0000","last_active":"2024-04-26T16:38:15.676009+0000","last_peered":"2024-04-26T16:38:15.676009+0000","last_clean":"2024-04-26T16:38:15.676009+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:38:15.676009+0000","last_undegraded":"2024-04-26T16:38:15.676009+0000","last_fullsized":"2024-04-26T16:38:15.676009+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'66","last_scrub_stamp":"2024-04-26T16:38:03.160063+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.160063+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.536394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1665,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443342+0000","last_change":"2024-04-26T16:35:08.450261+0000","last_active":"2024-04-26T16:38:16.443342+0000","last_peered":"2024-04-26T16:38:16.443342+0000","last_clean":"2024-04-26T16:38:16.443342+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:38:16.443342+0000","last_undegraded":"2024-04-26T16:38:16.443342+0000","last_fullsized":"2024-04-26T16:38:16.443342+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:35:08.450018+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:35:08.450018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1688,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.675189+0000","last_change":"2024-04-26T16:38:02.125681+0000","last_active":"2024-04-26T16:38:15.675189+0000","last_peered":"2024-04-26T16:38:15.675189+0000","last_clean":"2024-04-26T16:38:15.675189+0000","last_became_active":"2024-04-26T16:35:25.411888+0000","last_became_peered":"2024-04-26T16:35:25.411888+0000","last_unstale":"2024-04-26T16:38:15.675189+0000","last_undegraded":"2024-04-26T16:38:15.675189+0000","last_fullsized":"2024-04-26T16:38:15.675189+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.125415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.125415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.366125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1517,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441670+0000","last_change":"2024-04-26T16:37:57.743728+0000","last_active":"2024-04-26T16:38:16.441670+0000","last_peered":"2024-04-26T16:38:16.441670+0000","last_clean":"2024-04-26T16:38:16.441670+0000","last_became_active":"2024-04-26T16:35:25.417017+0000","last_became_peered":"2024-04-26T16:35:25.417017+0000","last_unstale":"2024-04-26T16:38:16.441670+0000","last_undegraded":"2024-04-26T16:38:16.441670+0000","last_fullsized":"2024-04-26T16:38:16.441670+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:57.743476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.743476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.056336+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439373+0000","last_change":"2024-04-26T16:37:58.133197+0000","last_active":"2024-04-26T16:38:16.439373+0000","last_peered":"2024-04-26T16:38:16.439373+0000","last_clean":"2024-04-26T16:38:16.439373+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:38:16.439373+0000","last_undegraded":"2024-04-26T16:38:16.439373+0000","last_fullsized":"2024-04-26T16:38:16.439373+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.132987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.132987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:00.348823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1657,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441130+0000","last_change":"2024-04-26T16:35:06.534153+0000","last_active":"2024-04-26T16:38:16.441130+0000","last_peered":"2024-04-26T16:38:16.441130+0000","last_clean":"2024-04-26T16:38:16.441130+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:38:16.441130+0000","last_undegraded":"2024-04-26T16:38:16.441130+0000","last_fullsized":"2024-04-26T16:38:16.441130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:06.533840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:35:06.533840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"857'49","reported_seq":1746,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.606437+0000","last_change":"2024-04-26T16:36:58.920449+0000","last_active":"2024-04-26T16:38:16.606437+0000","last_peered":"2024-04-26T16:38:16.606437+0000","last_clean":"2024-04-26T16:38:16.606437+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:38:16.606437+0000","last_undegraded":"2024-04-26T16:38:16.606437+0000","last_fullsized":"2024-04-26T16:38:16.606437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:36:58.920294+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:36:58.920294+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:18.593619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1691,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.677293+0000","last_change":"2024-04-26T16:37:05.007705+0000","last_active":"2024-04-26T16:38:15.677293+0000","last_peered":"2024-04-26T16:38:15.677293+0000","last_clean":"2024-04-26T16:38:15.677293+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:38:15.677293+0000","last_undegraded":"2024-04-26T16:38:15.677293+0000","last_fullsized":"2024-04-26T16:38:15.677293+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:05.007579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:37:05.007579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.021319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1537,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440310+0000","last_change":"2024-04-26T16:38:05.712753+0000","last_active":"2024-04-26T16:38:16.440310+0000","last_peered":"2024-04-26T16:38:16.440310+0000","last_clean":"2024-04-26T16:38:16.440310+0000","last_became_active":"2024-04-26T16:35:25.417111+0000","last_became_peered":"2024-04-26T16:35:25.417111+0000","last_unstale":"2024-04-26T16:38:16.440310+0000","last_undegraded":"2024-04-26T16:38:16.440310+0000","last_fullsized":"2024-04-26T16:38:16.440310+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:05.712520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.712520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.895535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1660,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442516+0000","last_change":"2024-04-26T16:35:14.526928+0000","last_active":"2024-04-26T16:38:16.442516+0000","last_peered":"2024-04-26T16:38:16.442516+0000","last_clean":"2024-04-26T16:38:16.442516+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:38:16.442516+0000","last_undegraded":"2024-04-26T16:38:16.442516+0000","last_fullsized":"2024-04-26T16:38:16.442516+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:14.526731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:35:14.526731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1955,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441442+0000","last_change":"2024-04-26T16:36:25.188281+0000","last_active":"2024-04-26T16:38:16.441442+0000","last_peered":"2024-04-26T16:38:16.441442+0000","last_clean":"2024-04-26T16:38:16.441442+0000","last_became_active":"2024-04-26T16:35:25.394870+0000","last_became_peered":"2024-04-26T16:35:25.394870+0000","last_unstale":"2024-04-26T16:38:16.441442+0000","last_undegraded":"2024-04-26T16:38:16.441442+0000","last_fullsized":"2024-04-26T16:38:16.441442+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:36:25.187951+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:36:25.187951+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437874+0000","last_change":"2024-04-26T16:37:57.118532+0000","last_active":"2024-04-26T16:38:16.437874+0000","last_peered":"2024-04-26T16:38:16.437874+0000","last_clean":"2024-04-26T16:38:16.437874+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:38:16.437874+0000","last_undegraded":"2024-04-26T16:38:16.437874+0000","last_fullsized":"2024-04-26T16:38:16.437874+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:57.118297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.118297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.113285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"857'35","reported_seq":1764,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440336+0000","last_change":"2024-04-26T16:38:11.656964+0000","last_active":"2024-04-26T16:38:16.440336+0000","last_peered":"2024-04-26T16:38:16.440336+0000","last_clean":"2024-04-26T16:38:16.440336+0000","last_became_active":"2024-04-26T16:35:25.397667+0000","last_became_peered":"2024-04-26T16:35:25.397667+0000","last_unstale":"2024-04-26T16:38:16.440336+0000","last_undegraded":"2024-04-26T16:38:16.440336+0000","last_fullsized":"2024-04-26T16:38:16.440336+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'35","last_scrub_stamp":"2024-04-26T16:38:11.656588+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:38:11.656588+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.518037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439569+0000","last_change":"2024-04-26T16:37:02.918875+0000","last_active":"2024-04-26T16:38:16.439569+0000","last_peered":"2024-04-26T16:38:16.439569+0000","last_clean":"2024-04-26T16:38:16.439569+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:38:16.439569+0000","last_undegraded":"2024-04-26T16:38:16.439569+0000","last_fullsized":"2024-04-26T16:38:16.439569+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:02.918675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:37:02.918675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:21.709848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1537,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439952+0000","last_change":"2024-04-26T16:37:42.746944+0000","last_active":"2024-04-26T16:38:16.439952+0000","last_peered":"2024-04-26T16:38:16.439952+0000","last_clean":"2024-04-26T16:38:16.439952+0000","last_became_active":"2024-04-26T16:35:25.392163+0000","last_became_peered":"2024-04-26T16:35:25.392163+0000","last_unstale":"2024-04-26T16:38:16.439952+0000","last_undegraded":"2024-04-26T16:38:16.439952+0000","last_fullsized":"2024-04-26T16:38:16.439952+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:42.746793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:42.746793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.906091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1692,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437519+0000","last_change":"2024-04-26T16:36:52.586910+0000","last_active":"2024-04-26T16:38:16.437519+0000","last_peered":"2024-04-26T16:38:16.437519+0000","last_clean":"2024-04-26T16:38:16.437519+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:38:16.437519+0000","last_undegraded":"2024-04-26T16:38:16.437519+0000","last_fullsized":"2024-04-26T16:38:16.437519+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:52.586701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:36:52.586701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1697,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435294+0000","last_change":"2024-04-26T16:38:11.352575+0000","last_active":"2024-04-26T16:38:16.435294+0000","last_peered":"2024-04-26T16:38:16.435294+0000","last_clean":"2024-04-26T16:38:16.435294+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:38:16.435294+0000","last_undegraded":"2024-04-26T16:38:16.435294+0000","last_fullsized":"2024-04-26T16:38:16.435294+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:11.352344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:38:11.352344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:31.815779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"857'50","reported_seq":1781,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.677896+0000","last_change":"2024-04-26T16:37:54.196331+0000","last_active":"2024-04-26T16:38:15.677896+0000","last_peered":"2024-04-26T16:38:15.677896+0000","last_clean":"2024-04-26T16:38:15.677896+0000","last_became_active":"2024-04-26T16:35:25.412117+0000","last_became_peered":"2024-04-26T16:35:25.412117+0000","last_unstale":"2024-04-26T16:38:15.677896+0000","last_undegraded":"2024-04-26T16:38:15.677896+0000","last_fullsized":"2024-04-26T16:38:15.677896+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'50","last_scrub_stamp":"2024-04-26T16:37:54.196211+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_clean_scrub_stamp":"2024-04-26T16:37:54.196211+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.629419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1654,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443156+0000","last_change":"2024-04-26T16:36:13.193711+0000","last_active":"2024-04-26T16:38:16.443156+0000","last_peered":"2024-04-26T16:38:16.443156+0000","last_clean":"2024-04-26T16:38:16.443156+0000","last_became_active":"2024-04-26T16:35:25.398515+0000","last_became_peered":"2024-04-26T16:35:25.398515+0000","last_unstale":"2024-04-26T16:38:16.443156+0000","last_undegraded":"2024-04-26T16:38:16.443156+0000","last_fullsized":"2024-04-26T16:38:16.443156+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:13.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:13.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1515,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440577+0000","last_change":"2024-04-26T16:38:12.645178+0000","last_active":"2024-04-26T16:38:16.440577+0000","last_peered":"2024-04-26T16:38:16.440577+0000","last_clean":"2024-04-26T16:38:16.440577+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:38:16.440577+0000","last_undegraded":"2024-04-26T16:38:16.440577+0000","last_fullsized":"2024-04-26T16:38:16.440577+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:12.645018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:12.645018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:42.576598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1672,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.608433+0000","last_change":"2024-04-26T16:36:48.923637+0000","last_active":"2024-04-26T16:38:16.608433+0000","last_peered":"2024-04-26T16:38:16.608433+0000","last_clean":"2024-04-26T16:38:16.608433+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:38:16.608433+0000","last_undegraded":"2024-04-26T16:38:16.608433+0000","last_fullsized":"2024-04-26T16:38:16.608433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.923527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.923527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"857'47","reported_seq":1766,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327949+0000","last_change":"2024-04-26T16:37:55.775734+0000","last_active":"2024-04-26T16:38:16.327949+0000","last_peered":"2024-04-26T16:38:16.327949+0000","last_clean":"2024-04-26T16:38:16.327949+0000","last_became_active":"2024-04-26T16:35:25.395746+0000","last_became_peered":"2024-04-26T16:35:25.395746+0000","last_unstale":"2024-04-26T16:38:16.327949+0000","last_undegraded":"2024-04-26T16:38:16.327949+0000","last_fullsized":"2024-04-26T16:38:16.327949+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'47","last_scrub_stamp":"2024-04-26T16:37:55.775586+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:37:55.775586+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.155493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":596,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439945+0000","last_change":"2024-04-26T16:36:32.514466+0000","last_active":"2024-04-26T16:38:16.439945+0000","last_peered":"2024-04-26T16:38:16.439945+0000","last_clean":"2024-04-26T16:38:16.439945+0000","last_became_active":"2024-04-26T16:35:25.397940+0000","last_became_peered":"2024-04-26T16:35:25.397940+0000","last_unstale":"2024-04-26T16:38:16.439945+0000","last_undegraded":"2024-04-26T16:38:16.439945+0000","last_fullsized":"2024-04-26T16:38:16.439945+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:32.514129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:36:32.514129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1660,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441035+0000","last_change":"2024-04-26T16:37:46.742109+0000","last_active":"2024-04-26T16:38:16.441035+0000","last_peered":"2024-04-26T16:38:16.441035+0000","last_clean":"2024-04-26T16:38:16.441035+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:38:16.441035+0000","last_undegraded":"2024-04-26T16:38:16.441035+0000","last_fullsized":"2024-04-26T16:38:16.441035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:46.742012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:46.742012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.964801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1533,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.436814+0000","last_change":"2024-04-26T16:36:59.476993+0000","last_active":"2024-04-26T16:38:16.436814+0000","last_peered":"2024-04-26T16:38:16.436814+0000","last_clean":"2024-04-26T16:38:16.436814+0000","last_became_active":"2024-04-26T16:35:25.390196+0000","last_became_peered":"2024-04-26T16:35:25.390196+0000","last_unstale":"2024-04-26T16:38:16.436814+0000","last_undegraded":"2024-04-26T16:38:16.436814+0000","last_fullsized":"2024-04-26T16:38:16.436814+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:59.476833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.476833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"747'19","reported_seq":1971,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607830+0000","last_change":"2024-04-26T16:37:57.836348+0000","last_active":"2024-04-26T16:38:16.607830+0000","last_peered":"2024-04-26T16:38:16.607830+0000","last_clean":"2024-04-26T16:38:16.607830+0000","last_became_active":"2024-04-26T16:35:25.404413+0000","last_became_peered":"2024-04-26T16:35:25.404413+0000","last_unstale":"2024-04-26T16:38:16.607830+0000","last_undegraded":"2024-04-26T16:38:16.607830+0000","last_fullsized":"2024-04-26T16:38:16.607830+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'19","last_scrub_stamp":"2024-04-26T16:37:57.836074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.836074+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:59.025050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1713,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.679509+0000","last_change":"2024-04-26T16:37:48.275884+0000","last_active":"2024-04-26T16:38:15.679509+0000","last_peered":"2024-04-26T16:38:15.679509+0000","last_clean":"2024-04-26T16:38:15.679509+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:38:15.679509+0000","last_undegraded":"2024-04-26T16:38:15.679509+0000","last_fullsized":"2024-04-26T16:38:15.679509+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:37:48.275781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.275781+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:52.249427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"857'52","reported_seq":705,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439480+0000","last_change":"2024-04-26T16:35:18.589728+0000","last_active":"2024-04-26T16:38:16.439480+0000","last_peered":"2024-04-26T16:38:16.439480+0000","last_clean":"2024-04-26T16:38:16.439480+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:38:16.439480+0000","last_undegraded":"2024-04-26T16:38:16.439480+0000","last_fullsized":"2024-04-26T16:38:16.439480+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'40","last_scrub_stamp":"2024-04-26T16:35:18.589318+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:35:18.589318+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1690,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.676318+0000","last_change":"2024-04-26T16:36:52.176623+0000","last_active":"2024-04-26T16:38:15.676318+0000","last_peered":"2024-04-26T16:38:15.676318+0000","last_clean":"2024-04-26T16:38:15.676318+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:38:15.676318+0000","last_undegraded":"2024-04-26T16:38:15.676318+0000","last_fullsized":"2024-04-26T16:38:15.676318+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:52.176514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:36:52.176514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1669,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.605899+0000","last_change":"2024-04-26T16:38:01.824470+0000","last_active":"2024-04-26T16:38:16.605899+0000","last_peered":"2024-04-26T16:38:16.605899+0000","last_clean":"2024-04-26T16:38:16.605899+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:38:16.605899+0000","last_undegraded":"2024-04-26T16:38:16.605899+0000","last_fullsized":"2024-04-26T16:38:16.605899+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:01.824258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:01.824258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.185258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1523,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607689+0000","last_change":"2024-04-26T16:37:01.923812+0000","last_active":"2024-04-26T16:38:16.607689+0000","last_peered":"2024-04-26T16:38:16.607689+0000","last_clean":"2024-04-26T16:38:16.607689+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:38:16.607689+0000","last_undegraded":"2024-04-26T16:38:16.607689+0000","last_fullsized":"2024-04-26T16:38:16.607689+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.923649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.923649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1680,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435341+0000","last_change":"2024-04-26T16:36:46.698276+0000","last_active":"2024-04-26T16:38:16.435341+0000","last_peered":"2024-04-26T16:38:16.435341+0000","last_clean":"2024-04-26T16:38:16.435341+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:38:16.435341+0000","last_undegraded":"2024-04-26T16:38:16.435341+0000","last_fullsized":"2024-04-26T16:38:16.435341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.698075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.698075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:16.625557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"857'27","reported_seq":1706,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443699+0000","last_change":"2024-04-26T16:37:56.108530+0000","last_active":"2024-04-26T16:38:16.443699+0000","last_peered":"2024-04-26T16:38:16.443699+0000","last_clean":"2024-04-26T16:38:16.443699+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:38:16.443699+0000","last_undegraded":"2024-04-26T16:38:16.443699+0000","last_fullsized":"2024-04-26T16:38:16.443699+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:37:56.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.108290+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.284265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1710,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.328485+0000","last_change":"2024-04-26T16:36:59.986857+0000","last_active":"2024-04-26T16:38:16.328485+0000","last_peered":"2024-04-26T16:38:16.328485+0000","last_clean":"2024-04-26T16:38:16.328485+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:38:16.328485+0000","last_undegraded":"2024-04-26T16:38:16.328485+0000","last_fullsized":"2024-04-26T16:38:16.328485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:36:59.986744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.986744+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.282717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1652,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441150+0000","last_change":"2024-04-26T16:37:48.766181+0000","last_active":"2024-04-26T16:38:16.441150+0000","last_peered":"2024-04-26T16:38:16.441150+0000","last_clean":"2024-04-26T16:38:16.441150+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:38:16.441150+0000","last_undegraded":"2024-04-26T16:38:16.441150+0000","last_fullsized":"2024-04-26T16:38:16.441150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:48.765876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.420943+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.765876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.048973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1523,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.606141+0000","last_change":"2024-04-26T16:36:41.913175+0000","last_active":"2024-04-26T16:38:16.606141+0000","last_peered":"2024-04-26T16:38:16.606141+0000","last_clean":"2024-04-26T16:38:16.606141+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:38:16.606141+0000","last_undegraded":"2024-04-26T16:38:16.606141+0000","last_fullsized":"2024-04-26T16:38:16.606141+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:41.912886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:41.912886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"857'27","reported_seq":1704,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441590+0000","last_change":"2024-04-26T16:36:19.167372+0000","last_active":"2024-04-26T16:38:16.441590+0000","last_peered":"2024-04-26T16:38:16.441590+0000","last_clean":"2024-04-26T16:38:16.441590+0000","last_became_active":"2024-04-26T16:35:25.395993+0000","last_became_peered":"2024-04-26T16:35:25.395993+0000","last_unstale":"2024-04-26T16:38:16.441590+0000","last_undegraded":"2024-04-26T16:38:16.441590+0000","last_fullsized":"2024-04-26T16:38:16.441590+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:36:19.167132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:19.167132+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1695,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435009+0000","last_change":"2024-04-26T16:38:09.331153+0000","last_active":"2024-04-26T16:38:16.435009+0000","last_peered":"2024-04-26T16:38:16.435009+0000","last_clean":"2024-04-26T16:38:16.435009+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:38:16.435009+0000","last_undegraded":"2024-04-26T16:38:16.435009+0000","last_fullsized":"2024-04-26T16:38:16.435009+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.330900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.330900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:25.207505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1655,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441489+0000","last_change":"2024-04-26T16:37:51.748594+0000","last_active":"2024-04-26T16:38:16.441489+0000","last_peered":"2024-04-26T16:38:16.441489+0000","last_clean":"2024-04-26T16:38:16.441489+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:38:16.441489+0000","last_undegraded":"2024-04-26T16:38:16.441489+0000","last_fullsized":"2024-04-26T16:38:16.441489+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.748496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.748496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.803195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1675,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437654+0000","last_change":"2024-04-26T16:37:51.423701+0000","last_active":"2024-04-26T16:38:16.437654+0000","last_peered":"2024-04-26T16:38:16.437654+0000","last_clean":"2024-04-26T16:38:16.437654+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:38:16.437654+0000","last_undegraded":"2024-04-26T16:38:16.437654+0000","last_fullsized":"2024-04-26T16:38:16.437654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.423243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.423243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:54.335512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1541,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.434503+0000","last_change":"2024-04-26T16:37:56.382398+0000","last_active":"2024-04-26T16:38:16.434503+0000","last_peered":"2024-04-26T16:38:16.434503+0000","last_clean":"2024-04-26T16:38:16.434503+0000","last_became_active":"2024-04-26T16:35:25.399182+0000","last_became_peered":"2024-04-26T16:35:25.399182+0000","last_unstale":"2024-04-26T16:38:16.434503+0000","last_undegraded":"2024-04-26T16:38:16.434503+0000","last_fullsized":"2024-04-26T16:38:16.434503+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.382182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.382182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.313965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"856'12","reported_seq":1410,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.655213+0000","last_change":"2024-04-26T16:37:56.309533+0000","last_active":"2024-04-26T16:38:11.655213+0000","last_peered":"2024-04-26T16:38:11.655213+0000","last_clean":"2024-04-26T16:38:11.655213+0000","last_became_active":"2024-04-26T16:35:25.404028+0000","last_became_peered":"2024-04-26T16:35:25.404028+0000","last_unstale":"2024-04-26T16:38:11.655213+0000","last_undegraded":"2024-04-26T16:38:11.655213+0000","last_fullsized":"2024-04-26T16:38:11.655213+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'12","last_scrub_stamp":"2024-04-26T16:37:56.309129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.309129+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.607105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1247,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.655451+0000","last_change":"2024-04-26T16:37:01.416249+0000","last_active":"2024-04-26T16:38:11.655451+0000","last_peered":"2024-04-26T16:38:11.655451+0000","last_clean":"2024-04-26T16:38:11.655451+0000","last_became_active":"2024-04-26T16:35:25.403847+0000","last_became_peered":"2024-04-26T16:35:25.403847+0000","last_unstale":"2024-04-26T16:38:11.655451+0000","last_undegraded":"2024-04-26T16:38:11.655451+0000","last_fullsized":"2024-04-26T16:38:11.655451+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.416147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.416147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:13.035252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.438947+0000","last_change":"2024-04-26T16:36:41.687047+0000","last_active":"2024-04-26T16:38:16.438947+0000","last_peered":"2024-04-26T16:38:16.438947+0000","last_clean":"2024-04-26T16:38:16.438947+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:38:16.438947+0000","last_undegraded":"2024-04-26T16:38:16.438947+0000","last_fullsized":"2024-04-26T16:38:16.438947+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:41.686919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:41.686919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1653,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443003+0000","last_change":"2024-04-26T16:36:33.166398+0000","last_active":"2024-04-26T16:38:16.443003+0000","last_peered":"2024-04-26T16:38:16.443003+0000","last_clean":"2024-04-26T16:38:16.443003+0000","last_became_active":"2024-04-26T16:35:25.379432+0000","last_became_peered":"2024-04-26T16:35:25.379432+0000","last_unstale":"2024-04-26T16:38:16.443003+0000","last_undegraded":"2024-04-26T16:38:16.443003+0000","last_fullsized":"2024-04-26T16:38:16.443003+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:33.166179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.166179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1534,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327498+0000","last_change":"2024-04-26T16:37:43.955898+0000","last_active":"2024-04-26T16:38:16.327498+0000","last_peered":"2024-04-26T16:38:16.327498+0000","last_clean":"2024-04-26T16:38:16.327498+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:38:16.327498+0000","last_undegraded":"2024-04-26T16:38:16.327498+0000","last_fullsized":"2024-04-26T16:38:16.327498+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:43.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:43.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:12.284132+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"857'27","reported_seq":1712,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443044+0000","last_change":"2024-04-26T16:34:51.453336+0000","last_active":"2024-04-26T16:38:16.443044+0000","last_peered":"2024-04-26T16:38:16.443044+0000","last_clean":"2024-04-26T16:38:16.443044+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:38:16.443044+0000","last_undegraded":"2024-04-26T16:38:16.443044+0000","last_fullsized":"2024-04-26T16:38:16.443044+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:34:51.453064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:34:51.453064+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439503+0000","last_change":"2024-04-26T16:36:49.734509+0000","last_active":"2024-04-26T16:38:16.439503+0000","last_peered":"2024-04-26T16:38:16.439503+0000","last_clean":"2024-04-26T16:38:16.439503+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:38:16.439503+0000","last_undegraded":"2024-04-26T16:38:16.439503+0000","last_fullsized":"2024-04-26T16:38:16.439503+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:49.734276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:49.734276+0000","last_clean_scrub_stamp":"2024-04-26T16:36:49.734276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:18.791205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":681,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439147+0000","last_change":"2024-04-26T16:37:56.088969+0000","last_active":"2024-04-26T16:38:16.439147+0000","last_peered":"2024-04-26T16:38:16.439147+0000","last_clean":"2024-04-26T16:38:16.439147+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:38:16.439147+0000","last_undegraded":"2024-04-26T16:38:16.439147+0000","last_fullsized":"2024-04-26T16:38:16.439147+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:37:56.088660+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.088660+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.991987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1667,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.608335+0000","last_change":"2024-04-26T16:37:53.920778+0000","last_active":"2024-04-26T16:38:16.608335+0000","last_peered":"2024-04-26T16:38:16.608335+0000","last_clean":"2024-04-26T16:38:16.608335+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:38:16.608335+0000","last_undegraded":"2024-04-26T16:38:16.608335+0000","last_fullsized":"2024-04-26T16:38:16.608335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:53.920538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:53.920538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.535913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437490+0000","last_change":"2024-04-26T16:36:54.762255+0000","last_active":"2024-04-26T16:38:16.437490+0000","last_peered":"2024-04-26T16:38:16.437490+0000","last_clean":"2024-04-26T16:38:16.437490+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:38:16.437490+0000","last_undegraded":"2024-04-26T16:38:16.437490+0000","last_fullsized":"2024-04-26T16:38:16.437490+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:54.762119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:54.762119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"857'25","reported_seq":1713,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.606412+0000","last_change":"2024-04-26T16:38:02.824020+0000","last_active":"2024-04-26T16:38:16.606412+0000","last_peered":"2024-04-26T16:38:16.606412+0000","last_clean":"2024-04-26T16:38:16.606412+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:38:16.606412+0000","last_undegraded":"2024-04-26T16:38:16.606412+0000","last_fullsized":"2024-04-26T16:38:16.606412+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:38:02.823785+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.823785+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.431260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1679,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607374+0000","last_change":"2024-04-26T16:36:45.881045+0000","last_active":"2024-04-26T16:38:16.607374+0000","last_peered":"2024-04-26T16:38:16.607374+0000","last_clean":"2024-04-26T16:38:16.607374+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:38:16.607374+0000","last_undegraded":"2024-04-26T16:38:16.607374+0000","last_fullsized":"2024-04-26T16:38:16.607374+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_clean_scrub_stamp":"2024-04-26T16:36:45.880889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1915,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440368+0000","last_change":"2024-04-26T16:37:44.755462+0000","last_active":"2024-04-26T16:38:16.440368+0000","last_peered":"2024-04-26T16:38:16.440368+0000","last_clean":"2024-04-26T16:38:16.440368+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:38:16.440368+0000","last_undegraded":"2024-04-26T16:38:16.440368+0000","last_fullsized":"2024-04-26T16:38:16.440368+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:37:44.755315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.755315+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:55.270439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1693,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.329064+0000","last_change":"2024-04-26T16:38:05.800827+0000","last_active":"2024-04-26T16:38:16.329064+0000","last_peered":"2024-04-26T16:38:16.329064+0000","last_clean":"2024-04-26T16:38:16.329064+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:38:16.329064+0000","last_undegraded":"2024-04-26T16:38:16.329064+0000","last_fullsized":"2024-04-26T16:38:16.329064+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:05.800618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.800618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.334366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1506,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.444033+0000","last_change":"2024-04-26T16:38:13.084550+0000","last_active":"2024-04-26T16:38:16.444033+0000","last_peered":"2024-04-26T16:38:16.444033+0000","last_clean":"2024-04-26T16:38:16.444033+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:38:16.444033+0000","last_undegraded":"2024-04-26T16:38:16.444033+0000","last_fullsized":"2024-04-26T16:38:16.444033+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:13.084343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:13.084343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.917769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"857'25","reported_seq":1697,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.444062+0000","last_change":"2024-04-26T16:38:08.098586+0000","last_active":"2024-04-26T16:38:16.444062+0000","last_peered":"2024-04-26T16:38:16.444062+0000","last_clean":"2024-04-26T16:38:16.444062+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:38:16.444062+0000","last_undegraded":"2024-04-26T16:38:16.444062+0000","last_fullsized":"2024-04-26T16:38:16.444062+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:38:08.098198+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:38:08.098198+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:13.219884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1329,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656616+0000","last_change":"2024-04-26T16:37:45.265963+0000","last_active":"2024-04-26T16:38:11.656616+0000","last_peered":"2024-04-26T16:38:11.656616+0000","last_clean":"2024-04-26T16:38:11.656616+0000","last_became_active":"2024-04-26T16:35:25.401229+0000","last_became_peered":"2024-04-26T16:35:25.401229+0000","last_unstale":"2024-04-26T16:38:11.656616+0000","last_undegraded":"2024-04-26T16:38:11.656616+0000","last_fullsized":"2024-04-26T16:38:11.656616+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:45.265860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.265860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.529405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1663,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441041+0000","last_change":"2024-04-26T16:37:49.748432+0000","last_active":"2024-04-26T16:38:16.441041+0000","last_peered":"2024-04-26T16:38:16.441041+0000","last_clean":"2024-04-26T16:38:16.441041+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:38:16.441041+0000","last_undegraded":"2024-04-26T16:38:16.441041+0000","last_fullsized":"2024-04-26T16:38:16.441041+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:49.748209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:49.748209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:49.909641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1688,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.605892+0000","last_change":"2024-04-26T16:36:43.892069+0000","last_active":"2024-04-26T16:38:16.605892+0000","last_peered":"2024-04-26T16:38:16.605892+0000","last_clean":"2024-04-26T16:38:16.605892+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:38:16.605892+0000","last_undegraded":"2024-04-26T16:38:16.605892+0000","last_fullsized":"2024-04-26T16:38:16.605892+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:43.891851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:18.787243+0000","last_clean_scrub_stamp":"2024-04-26T16:36:43.891851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1146,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656638+0000","last_change":"2024-04-26T16:37:43.256055+0000","last_active":"2024-04-26T16:38:11.656638+0000","last_peered":"2024-04-26T16:38:11.656638+0000","last_clean":"2024-04-26T16:38:11.656638+0000","last_became_active":"2024-04-26T16:35:25.409902+0000","last_became_peered":"2024-04-26T16:35:25.409902+0000","last_unstale":"2024-04-26T16:38:11.656638+0000","last_undegraded":"2024-04-26T16:38:11.656638+0000","last_fullsized":"2024-04-26T16:38:11.656638+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:43.255960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:37:43.255960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:11.195362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"857'30","reported_seq":1736,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.328339+0000","last_change":"2024-04-26T16:37:49.783263+0000","last_active":"2024-04-26T16:38:16.328339+0000","last_peered":"2024-04-26T16:38:16.328339+0000","last_clean":"2024-04-26T16:38:16.328339+0000","last_became_active":"2024-04-26T16:35:25.383698+0000","last_became_peered":"2024-04-26T16:35:25.383698+0000","last_unstale":"2024-04-26T16:38:16.328339+0000","last_undegraded":"2024-04-26T16:38:16.328339+0000","last_fullsized":"2024-04-26T16:38:16.328339+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:37:49.783106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:49.783106+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:50.844745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1196,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.675884+0000","last_change":"2024-04-26T16:38:00.194757+0000","last_active":"2024-04-26T16:38:15.675884+0000","last_peered":"2024-04-26T16:38:15.675884+0000","last_clean":"2024-04-26T16:38:15.675884+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:38:15.675884+0000","last_undegraded":"2024-04-26T16:38:15.675884+0000","last_fullsized":"2024-04-26T16:38:15.675884+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:00.194512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.194512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.614121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1685,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.679339+0000","last_change":"2024-04-26T16:37:01.057309+0000","last_active":"2024-04-26T16:38:15.679339+0000","last_peered":"2024-04-26T16:38:15.679339+0000","last_clean":"2024-04-26T16:38:15.679339+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:38:15.679339+0000","last_undegraded":"2024-04-26T16:38:15.679339+0000","last_fullsized":"2024-04-26T16:38:15.679339+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.057091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:37:01.057091+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.057091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:18.963967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1663,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441048+0000","last_change":"2024-04-26T16:36:44.583891+0000","last_active":"2024-04-26T16:38:16.441048+0000","last_peered":"2024-04-26T16:38:16.441048+0000","last_clean":"2024-04-26T16:38:16.441048+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:38:16.441048+0000","last_undegraded":"2024-04-26T16:38:16.441048+0000","last_fullsized":"2024-04-26T16:38:16.441048+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:44.583680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:36:44.583680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1532,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437272+0000","last_change":"2024-04-26T16:37:00.520805+0000","last_active":"2024-04-26T16:38:16.437272+0000","last_peered":"2024-04-26T16:38:16.437272+0000","last_clean":"2024-04-26T16:38:16.437272+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:38:16.437272+0000","last_undegraded":"2024-04-26T16:38:16.437272+0000","last_fullsized":"2024-04-26T16:38:16.437272+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.520675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.520675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:22.348302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"857'18","reported_seq":1747,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607937+0000","last_change":"2024-04-26T16:37:44.980167+0000","last_active":"2024-04-26T16:38:16.607937+0000","last_peered":"2024-04-26T16:38:16.607937+0000","last_clean":"2024-04-26T16:38:16.607937+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:38:16.607937+0000","last_undegraded":"2024-04-26T16:38:16.607937+0000","last_fullsized":"2024-04-26T16:38:16.607937+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'18","last_scrub_stamp":"2024-04-26T16:37:44.980066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.980066+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.790501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":596,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439196+0000","last_change":"2024-04-26T16:35:26.526741+0000","last_active":"2024-04-26T16:38:16.439196+0000","last_peered":"2024-04-26T16:38:16.439196+0000","last_clean":"2024-04-26T16:38:16.439196+0000","last_became_active":"2024-04-26T16:35:25.389184+0000","last_became_peered":"2024-04-26T16:35:25.389184+0000","last_unstale":"2024-04-26T16:38:16.439196+0000","last_undegraded":"2024-04-26T16:38:16.439196+0000","last_fullsized":"2024-04-26T16:38:16.439196+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:35:26.526395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:35:26.526395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1295,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.655100+0000","last_change":"2024-04-26T16:38:10.273089+0000","last_active":"2024-04-26T16:38:11.655100+0000","last_peered":"2024-04-26T16:38:11.655100+0000","last_clean":"2024-04-26T16:38:11.655100+0000","last_became_active":"2024-04-26T16:35:25.405230+0000","last_became_peered":"2024-04-26T16:35:25.405230+0000","last_unstale":"2024-04-26T16:38:11.655100+0000","last_undegraded":"2024-04-26T16:38:11.655100+0000","last_fullsized":"2024-04-26T16:38:11.655100+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:10.272810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:10.272810+0000","last_clean_scrub_stamp":"2024-04-26T16:38:10.272810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.052304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1643,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443324+0000","last_change":"2024-04-26T16:37:00.148921+0000","last_active":"2024-04-26T16:38:16.443324+0000","last_peered":"2024-04-26T16:38:16.443324+0000","last_clean":"2024-04-26T16:38:16.443324+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:38:16.443324+0000","last_undegraded":"2024-04-26T16:38:16.443324+0000","last_fullsized":"2024-04-26T16:38:16.443324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.148735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.148735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:19.076498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1518,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.444019+0000","last_change":"2024-04-26T16:36:38.157649+0000","last_active":"2024-04-26T16:38:16.444019+0000","last_peered":"2024-04-26T16:38:16.444019+0000","last_clean":"2024-04-26T16:38:16.444019+0000","last_became_active":"2024-04-26T16:35:25.394016+0000","last_became_peered":"2024-04-26T16:35:25.394016+0000","last_unstale":"2024-04-26T16:38:16.444019+0000","last_undegraded":"2024-04-26T16:38:16.444019+0000","last_fullsized":"2024-04-26T16:38:16.444019+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:38.157418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_clean_scrub_stamp":"2024-04-26T16:36:38.157418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"856'21","reported_seq":622,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439114+0000","last_change":"2024-04-26T16:36:44.718941+0000","last_active":"2024-04-26T16:38:16.439114+0000","last_peered":"2024-04-26T16:38:16.439114+0000","last_clean":"2024-04-26T16:38:16.439114+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:38:16.439114+0000","last_undegraded":"2024-04-26T16:38:16.439114+0000","last_fullsized":"2024-04-26T16:38:16.439114+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:44.718541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:44.718541+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1298,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656899+0000","last_change":"2024-04-26T16:37:44.221946+0000","last_active":"2024-04-26T16:38:11.656899+0000","last_peered":"2024-04-26T16:38:11.656899+0000","last_clean":"2024-04-26T16:38:11.656899+0000","last_became_active":"2024-04-26T16:35:25.406825+0000","last_became_peered":"2024-04-26T16:35:25.406825+0000","last_unstale":"2024-04-26T16:38:11.656899+0000","last_undegraded":"2024-04-26T16:38:11.656899+0000","last_fullsized":"2024-04-26T16:38:11.656899+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:44.221823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.221823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:13.401315+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1657,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442538+0000","last_change":"2024-04-26T16:36:15.209969+0000","last_active":"2024-04-26T16:38:16.442538+0000","last_peered":"2024-04-26T16:38:16.442538+0000","last_clean":"2024-04-26T16:38:16.442538+0000","last_became_active":"2024-04-26T16:35:25.378783+0000","last_became_peered":"2024-04-26T16:35:25.378783+0000","last_unstale":"2024-04-26T16:38:16.442538+0000","last_undegraded":"2024-04-26T16:38:16.442538+0000","last_fullsized":"2024-04-26T16:38:16.442538+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:15.209875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:15.209875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1661,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441204+0000","last_change":"2024-04-26T16:37:55.748337+0000","last_active":"2024-04-26T16:38:16.441204+0000","last_peered":"2024-04-26T16:38:16.441204+0000","last_clean":"2024-04-26T16:38:16.441204+0000","last_became_active":"2024-04-26T16:35:25.410836+0000","last_became_peered":"2024-04-26T16:35:25.410836+0000","last_unstale":"2024-04-26T16:38:16.441204+0000","last_undegraded":"2024-04-26T16:38:16.441204+0000","last_fullsized":"2024-04-26T16:38:16.441204+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:55.748010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:55.748010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.141796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1531,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.435466+0000","last_change":"2024-04-26T16:37:59.386367+0000","last_active":"2024-04-26T16:38:16.435466+0000","last_peered":"2024-04-26T16:38:16.435466+0000","last_clean":"2024-04-26T16:38:16.435466+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:38:16.435466+0000","last_undegraded":"2024-04-26T16:38:16.435466+0000","last_fullsized":"2024-04-26T16:38:16.435466+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:59.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.277947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"857'46","reported_seq":1753,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.674310+0000","last_change":"2024-04-26T16:37:04.034823+0000","last_active":"2024-04-26T16:38:15.674310+0000","last_peered":"2024-04-26T16:38:15.674310+0000","last_clean":"2024-04-26T16:38:15.674310+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:38:15.674310+0000","last_undegraded":"2024-04-26T16:38:15.674310+0000","last_fullsized":"2024-04-26T16:38:15.674310+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:37:04.034569+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:37:04.034569+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:19.723405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.436693+0000","last_change":"2024-04-26T16:37:52.063624+0000","last_active":"2024-04-26T16:38:16.436693+0000","last_peered":"2024-04-26T16:38:16.436693+0000","last_clean":"2024-04-26T16:38:16.436693+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:38:16.436693+0000","last_undegraded":"2024-04-26T16:38:16.436693+0000","last_fullsized":"2024-04-26T16:38:16.436693+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.562434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"747'16","reported_seq":1492,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656762+0000","last_change":"2024-04-26T16:38:05.198183+0000","last_active":"2024-04-26T16:38:11.656762+0000","last_peered":"2024-04-26T16:38:11.656762+0000","last_clean":"2024-04-26T16:38:11.656762+0000","last_became_active":"2024-04-26T16:35:25.403416+0000","last_became_peered":"2024-04-26T16:35:25.403416+0000","last_unstale":"2024-04-26T16:38:11.656762+0000","last_undegraded":"2024-04-26T16:38:11.656762+0000","last_fullsized":"2024-04-26T16:38:11.656762+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:38:05.197963+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.197963+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.854698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"829'2","reported_seq":1666,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442988+0000","last_change":"2024-04-26T16:36:59.155190+0000","last_active":"2024-04-26T16:38:16.442988+0000","last_peered":"2024-04-26T16:38:16.442988+0000","last_clean":"2024-04-26T16:38:16.442988+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:38:16.442988+0000","last_undegraded":"2024-04-26T16:38:16.442988+0000","last_fullsized":"2024-04-26T16:38:16.442988+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'2","last_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_deep_scrub":"829'2","last_deep_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.155048+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.337428+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1533,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607274+0000","last_change":"2024-04-26T16:37:52.886432+0000","last_active":"2024-04-26T16:38:16.607274+0000","last_peered":"2024-04-26T16:38:16.607274+0000","last_clean":"2024-04-26T16:38:16.607274+0000","last_became_active":"2024-04-26T16:35:25.409763+0000","last_became_peered":"2024-04-26T16:35:25.409763+0000","last_unstale":"2024-04-26T16:38:16.607274+0000","last_undegraded":"2024-04-26T16:38:16.607274+0000","last_fullsized":"2024-04-26T16:38:16.607274+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.886311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.886311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.121107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"856'21","reported_seq":1693,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440268+0000","last_change":"2024-04-26T16:36:47.694088+0000","last_active":"2024-04-26T16:38:16.440268+0000","last_peered":"2024-04-26T16:38:16.440268+0000","last_clean":"2024-04-26T16:38:16.440268+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:38:16.440268+0000","last_undegraded":"2024-04-26T16:38:16.440268+0000","last_fullsized":"2024-04-26T16:38:16.440268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:47.693892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:47.693892+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1675,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441286+0000","last_change":"2024-04-26T16:37:59.723665+0000","last_active":"2024-04-26T16:38:16.441286+0000","last_peered":"2024-04-26T16:38:16.441286+0000","last_clean":"2024-04-26T16:38:16.441286+0000","last_became_active":"2024-04-26T16:35:25.417030+0000","last_became_peered":"2024-04-26T16:35:25.417030+0000","last_unstale":"2024-04-26T16:38:16.441286+0000","last_undegraded":"2024-04-26T16:38:16.441286+0000","last_fullsized":"2024-04-26T16:38:16.441286+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:37:59.723438+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.723438+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.920638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1681,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.608079+0000","last_change":"2024-04-26T16:37:00.945337+0000","last_active":"2024-04-26T16:38:16.608079+0000","last_peered":"2024-04-26T16:38:16.608079+0000","last_clean":"2024-04-26T16:38:16.608079+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:38:16.608079+0000","last_undegraded":"2024-04-26T16:38:16.608079+0000","last_fullsized":"2024-04-26T16:38:16.608079+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.945236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:45.901920+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.945236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1689,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.676909+0000","last_change":"2024-04-26T16:36:57.199063+0000","last_active":"2024-04-26T16:38:15.676909+0000","last_peered":"2024-04-26T16:38:15.676909+0000","last_clean":"2024-04-26T16:38:15.676909+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:38:15.676909+0000","last_undegraded":"2024-04-26T16:38:15.676909+0000","last_fullsized":"2024-04-26T16:38:15.676909+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:57.198785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.198785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437761+0000","last_change":"2024-04-26T16:38:04.121917+0000","last_active":"2024-04-26T16:38:16.437761+0000","last_peered":"2024-04-26T16:38:16.437761+0000","last_clean":"2024-04-26T16:38:16.437761+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:38:16.437761+0000","last_undegraded":"2024-04-26T16:38:16.437761+0000","last_fullsized":"2024-04-26T16:38:16.437761+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:04.121774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.121774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.071499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"857'11","reported_seq":1693,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.607524+0000","last_change":"2024-04-26T16:36:57.942537+0000","last_active":"2024-04-26T16:38:16.607524+0000","last_peered":"2024-04-26T16:38:16.607524+0000","last_clean":"2024-04-26T16:38:16.607524+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:38:16.607524+0000","last_undegraded":"2024-04-26T16:38:16.607524+0000","last_fullsized":"2024-04-26T16:38:16.607524+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:36:57.942209+0000","last_deep_scrub":"388'9","last_deep_scrub_stamp":"2024-04-26T16:36:57.942209+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.942209+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:21.478333+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1355,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.654821+0000","last_change":"2024-04-26T16:37:47.358920+0000","last_active":"2024-04-26T16:38:11.654821+0000","last_peered":"2024-04-26T16:38:11.654821+0000","last_clean":"2024-04-26T16:38:11.654821+0000","last_became_active":"2024-04-26T16:35:25.404997+0000","last_became_peered":"2024-04-26T16:35:25.404997+0000","last_unstale":"2024-04-26T16:38:11.654821+0000","last_undegraded":"2024-04-26T16:38:11.654821+0000","last_fullsized":"2024-04-26T16:38:11.654821+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:47.358713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.358713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:57.165878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1689,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:15.676587+0000","last_change":"2024-04-26T16:38:09.125742+0000","last_active":"2024-04-26T16:38:15.676587+0000","last_peered":"2024-04-26T16:38:15.676587+0000","last_clean":"2024-04-26T16:38:15.676587+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:38:15.676587+0000","last_undegraded":"2024-04-26T16:38:15.676587+0000","last_fullsized":"2024-04-26T16:38:15.676587+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.125384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.125384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:15.580879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1705,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.329100+0000","last_change":"2024-04-26T16:37:51.818589+0000","last_active":"2024-04-26T16:38:16.329100+0000","last_peered":"2024-04-26T16:38:16.329100+0000","last_clean":"2024-04-26T16:38:16.329100+0000","last_became_active":"2024-04-26T16:35:25.383286+0000","last_became_peered":"2024-04-26T16:35:25.383286+0000","last_unstale":"2024-04-26T16:38:16.329100+0000","last_undegraded":"2024-04-26T16:38:16.329100+0000","last_fullsized":"2024-04-26T16:38:16.329100+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.818319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.818319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.894110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1507,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441570+0000","last_change":"2024-04-26T16:38:00.069058+0000","last_active":"2024-04-26T16:38:16.441570+0000","last_peered":"2024-04-26T16:38:16.441570+0000","last_clean":"2024-04-26T16:38:16.441570+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:38:16.441570+0000","last_undegraded":"2024-04-26T16:38:16.441570+0000","last_fullsized":"2024-04-26T16:38:16.441570+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:00.068758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.068758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.541601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"857'21","reported_seq":1715,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327653+0000","last_change":"2024-04-26T16:36:51.894635+0000","last_active":"2024-04-26T16:38:16.327653+0000","last_peered":"2024-04-26T16:38:16.327653+0000","last_clean":"2024-04-26T16:38:16.327653+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:38:16.327653+0000","last_undegraded":"2024-04-26T16:38:16.327653+0000","last_fullsized":"2024-04-26T16:38:16.327653+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:51.894411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:51.894411+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:17.890802+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1673,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440762+0000","last_change":"2024-04-26T16:38:03.698985+0000","last_active":"2024-04-26T16:38:16.440762+0000","last_peered":"2024-04-26T16:38:16.440762+0000","last_clean":"2024-04-26T16:38:16.440762+0000","last_became_active":"2024-04-26T16:35:25.392415+0000","last_became_peered":"2024-04-26T16:35:25.392415+0000","last_unstale":"2024-04-26T16:38:16.440762+0000","last_undegraded":"2024-04-26T16:38:16.440762+0000","last_fullsized":"2024-04-26T16:38:16.440762+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:03.698766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.698766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.722484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1677,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.328637+0000","last_change":"2024-04-26T16:36:56.985639+0000","last_active":"2024-04-26T16:38:16.328637+0000","last_peered":"2024-04-26T16:38:16.328637+0000","last_clean":"2024-04-26T16:38:16.328637+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:38:16.328637+0000","last_undegraded":"2024-04-26T16:38:16.328637+0000","last_fullsized":"2024-04-26T16:38:16.328637+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_clean_scrub_stamp":"2024-04-26T16:36:56.985402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:19.810617+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1674,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441345+0000","last_change":"2024-04-26T16:37:58.738918+0000","last_active":"2024-04-26T16:38:16.441345+0000","last_peered":"2024-04-26T16:38:16.441345+0000","last_clean":"2024-04-26T16:38:16.441345+0000","last_became_active":"2024-04-26T16:35:25.410430+0000","last_became_peered":"2024-04-26T16:35:25.410430+0000","last_unstale":"2024-04-26T16:38:16.441345+0000","last_undegraded":"2024-04-26T16:38:16.441345+0000","last_fullsized":"2024-04-26T16:38:16.441345+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.738705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.738705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.841581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1531,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.436912+0000","last_change":"2024-04-26T16:37:47.396592+0000","last_active":"2024-04-26T16:38:16.436912+0000","last_peered":"2024-04-26T16:38:16.436912+0000","last_clean":"2024-04-26T16:38:16.436912+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:38:16.436912+0000","last_undegraded":"2024-04-26T16:38:16.436912+0000","last_fullsized":"2024-04-26T16:38:16.436912+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:47.396490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.396490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.967350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"857'29","reported_seq":625,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.438862+0000","last_change":"2024-04-26T16:36:33.541569+0000","last_active":"2024-04-26T16:38:16.438862+0000","last_peered":"2024-04-26T16:38:16.438862+0000","last_clean":"2024-04-26T16:38:16.438862+0000","last_became_active":"2024-04-26T16:35:25.382813+0000","last_became_peered":"2024-04-26T16:35:25.382813+0000","last_unstale":"2024-04-26T16:38:16.438862+0000","last_undegraded":"2024-04-26T16:38:16.438862+0000","last_fullsized":"2024-04-26T16:38:16.438862+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:36:33.541439+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.541439+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1684,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442017+0000","last_change":"2024-04-26T16:38:04.132311+0000","last_active":"2024-04-26T16:38:16.442017+0000","last_peered":"2024-04-26T16:38:16.442017+0000","last_clean":"2024-04-26T16:38:16.442017+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:38:16.442017+0000","last_undegraded":"2024-04-26T16:38:16.442017+0000","last_fullsized":"2024-04-26T16:38:16.442017+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:04.132193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.132193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:05.515649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":595,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.437614+0000","last_change":"2024-04-26T16:36:40.657677+0000","last_active":"2024-04-26T16:38:16.437614+0000","last_peered":"2024-04-26T16:38:16.437614+0000","last_clean":"2024-04-26T16:38:16.437614+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:38:16.437614+0000","last_undegraded":"2024-04-26T16:38:16.437614+0000","last_fullsized":"2024-04-26T16:38:16.437614+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:40.657538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:40.657538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1675,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.606096+0000","last_change":"2024-04-26T16:38:12.853887+0000","last_active":"2024-04-26T16:38:16.606096+0000","last_peered":"2024-04-26T16:38:16.606096+0000","last_clean":"2024-04-26T16:38:16.606096+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:38:16.606096+0000","last_undegraded":"2024-04-26T16:38:16.606096+0000","last_fullsized":"2024-04-26T16:38:16.606096+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:12.853670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.855045+0000","last_clean_scrub_stamp":"2024-04-26T16:38:12.853670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:40.929680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1134,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-04-26T16:38:11.656432+0000","last_change":"2024-04-26T16:37:51.285062+0000","last_active":"2024-04-26T16:38:11.656432+0000","last_peered":"2024-04-26T16:38:11.656432+0000","last_clean":"2024-04-26T16:38:11.656432+0000","last_became_active":"2024-04-26T16:35:25.401906+0000","last_became_peered":"2024-04-26T16:35:25.401906+0000","last_unstale":"2024-04-26T16:38:11.656432+0000","last_undegraded":"2024-04-26T16:38:11.656432+0000","last_fullsized":"2024-04-26T16:38:11.656432+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.284847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.284847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.463961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"857'39","reported_seq":640,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.439200+0000","last_change":"2024-04-26T16:37:45.123276+0000","last_active":"2024-04-26T16:38:16.439200+0000","last_peered":"2024-04-26T16:38:16.439200+0000","last_clean":"2024-04-26T16:38:16.439200+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:38:16.439200+0000","last_undegraded":"2024-04-26T16:38:16.439200+0000","last_fullsized":"2024-04-26T16:38:16.439200+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'39","last_scrub_stamp":"2024-04-26T16:37:45.122923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.122923+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:02.778170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1673,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443723+0000","last_change":"2024-04-26T16:36:23.142745+0000","last_active":"2024-04-26T16:38:16.443723+0000","last_peered":"2024-04-26T16:38:16.443723+0000","last_clean":"2024-04-26T16:38:16.443723+0000","last_became_active":"2024-04-26T16:35:25.389963+0000","last_became_peered":"2024-04-26T16:35:25.389963+0000","last_unstale":"2024-04-26T16:38:16.443723+0000","last_undegraded":"2024-04-26T16:38:16.443723+0000","last_fullsized":"2024-04-26T16:38:16.443723+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:23.142443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:23.142443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1647,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.442229+0000","last_change":"2024-04-26T16:36:46.231728+0000","last_active":"2024-04-26T16:38:16.442229+0000","last_peered":"2024-04-26T16:38:16.442229+0000","last_clean":"2024-04-26T16:38:16.442229+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:38:16.442229+0000","last_undegraded":"2024-04-26T16:38:16.442229+0000","last_fullsized":"2024-04-26T16:38:16.442229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.231628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.231628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":607,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.436884+0000","last_change":"2024-04-26T16:37:48.095046+0000","last_active":"2024-04-26T16:38:16.436884+0000","last_peered":"2024-04-26T16:38:16.436884+0000","last_clean":"2024-04-26T16:38:16.436884+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:38:16.436884+0000","last_undegraded":"2024-04-26T16:38:16.436884+0000","last_fullsized":"2024-04-26T16:38:16.436884+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:37:48.094883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.094883+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:52.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1529,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.443913+0000","last_change":"2024-04-26T16:36:22.139010+0000","last_active":"2024-04-26T16:38:16.443913+0000","last_peered":"2024-04-26T16:38:16.443913+0000","last_clean":"2024-04-26T16:38:16.443913+0000","last_became_active":"2024-04-26T16:35:25.388177+0000","last_became_peered":"2024-04-26T16:35:25.388177+0000","last_unstale":"2024-04-26T16:38:16.443913+0000","last_undegraded":"2024-04-26T16:38:16.443913+0000","last_fullsized":"2024-04-26T16:38:16.443913+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:22.138796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:36:22.138796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"857'27","reported_seq":1772,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.608020+0000","last_change":"2024-04-26T16:38:08.788411+0000","last_active":"2024-04-26T16:38:16.608020+0000","last_peered":"2024-04-26T16:38:16.608020+0000","last_clean":"2024-04-26T16:38:16.608020+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:38:16.608020+0000","last_undegraded":"2024-04-26T16:38:16.608020+0000","last_fullsized":"2024-04-26T16:38:16.608020+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:38:08.788157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:08.788157+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.504129+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1685,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440828+0000","last_change":"2024-04-26T16:36:46.673477+0000","last_active":"2024-04-26T16:38:16.440828+0000","last_peered":"2024-04-26T16:38:16.440828+0000","last_clean":"2024-04-26T16:38:16.440828+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:38:16.440828+0000","last_undegraded":"2024-04-26T16:38:16.440828+0000","last_fullsized":"2024-04-26T16:38:16.440828+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.673182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.673182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1701,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.162329+0000","last_change":"2024-04-26T16:38:16.162394+0000","last_active":"2024-04-26T16:38:16.162329+0000","last_peered":"2024-04-26T16:38:16.162329+0000","last_clean":"2024-04-26T16:38:16.162329+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:38:16.162329+0000","last_undegraded":"2024-04-26T16:38:16.162329+0000","last_fullsized":"2024-04-26T16:38:16.162329+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:16.162173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.136177+0000","last_clean_scrub_stamp":"2024-04-26T16:38:16.162173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:39.174742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1693,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.327264+0000","last_change":"2024-04-26T16:38:14.731024+0000","last_active":"2024-04-26T16:38:16.327264+0000","last_peered":"2024-04-26T16:38:16.327264+0000","last_clean":"2024-04-26T16:38:16.327264+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:38:16.327264+0000","last_undegraded":"2024-04-26T16:38:16.327264+0000","last_fullsized":"2024-04-26T16:38:16.327264+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:14.730813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:38:14.730813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.405891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":603,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.438691+0000","last_change":"2024-04-26T16:38:01.112600+0000","last_active":"2024-04-26T16:38:16.438691+0000","last_peered":"2024-04-26T16:38:16.438691+0000","last_clean":"2024-04-26T16:38:16.438691+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:38:16.438691+0000","last_undegraded":"2024-04-26T16:38:16.438691+0000","last_fullsized":"2024-04-26T16:38:16.438691+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:01.112389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:16.569721+0000","last_clean_scrub_stamp":"2024-04-26T16:38:01.112389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.253547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"857'30","reported_seq":1724,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.441331+0000","last_change":"2024-04-26T16:38:15.633443+0000","last_active":"2024-04-26T16:38:16.441331+0000","last_peered":"2024-04-26T16:38:16.441331+0000","last_clean":"2024-04-26T16:38:16.441331+0000","last_became_active":"2024-04-26T16:35:25.399680+0000","last_became_peered":"2024-04-26T16:35:25.399680+0000","last_unstale":"2024-04-26T16:38:16.441331+0000","last_undegraded":"2024-04-26T16:38:16.441331+0000","last_fullsized":"2024-04-26T16:38:16.441331+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:38:15.633152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:15.633152+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.147835+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1688,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.608166+0000","last_change":"2024-04-26T16:38:09.828067+0000","last_active":"2024-04-26T16:38:16.608166+0000","last_peered":"2024-04-26T16:38:16.608166+0000","last_clean":"2024-04-26T16:38:16.608166+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:38:16.608166+0000","last_undegraded":"2024-04-26T16:38:16.608166+0000","last_fullsized":"2024-04-26T16:38:16.608166+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.827896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.827896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:37.537447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1687,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.326846+0000","last_change":"2024-04-26T16:37:45.893079+0000","last_active":"2024-04-26T16:38:16.326846+0000","last_peered":"2024-04-26T16:38:16.326846+0000","last_clean":"2024-04-26T16:38:16.326846+0000","last_became_active":"2024-04-26T16:35:25.396616+0000","last_became_peered":"2024-04-26T16:35:25.396616+0000","last_unstale":"2024-04-26T16:38:16.326846+0000","last_undegraded":"2024-04-26T16:38:16.326846+0000","last_fullsized":"2024-04-26T16:38:16.326846+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:45.892786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.892786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:49.656756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1661,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-04-26T16:38:16.440287+0000","last_change":"2024-04-26T16:38:02.658129+0000","last_active":"2024-04-26T16:38:16.440287+0000","last_peered":"2024-04-26T16:38:16.440287+0000","last_clean":"2024-04-26T16:38:16.440287+0000","last_became_active":"2024-04-26T16:35:25.392061+0000","last_became_peered":"2024-04-26T16:35:25.392061+0000","last_unstale":"2024-04-26T16:38:16.440287+0000","last_undegraded":"2024-04-26T16:38:16.440287+0000","last_fullsized":"2024-04-26T16:38:16.440287+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.657915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.657915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.431423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1224,"num_read_kb":1219,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743684,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113588,"kb_used_data":5816,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93631564,"statfs":{"total":95995035648,"available":95878721536,"internally_reserved":0,"allocated":5955584,"data_stored":3229019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":746,"seq":3204045602851,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39444,"kb_used_data":5784,"kb_used_omap":23,"kb_used_meta":33384,"kb_avail":93705708,"statfs":{"total":95995035648,"available":95954644992,"internally_reserved":0,"allocated":5922816,"data_stored":3226253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23760,"internal_metadata":34186032},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575170,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104452,"kb_used_data":5000,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93640700,"statfs":{"total":95995035648,"available":95888076800,"internally_reserved":0,"allocated":5120000,"data_stored":2481309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738700,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109424,"kb_used_data":5812,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93635728,"statfs":{"total":95995035648,"available":95882985472,"internally_reserved":0,"allocated":5951488,"data_stored":3228447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411639,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108596,"kb_used_data":4984,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93636556,"statfs":{"total":95995035648,"available":95883833344,"internally_reserved":0,"allocated":5103616,"data_stored":2480575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084588,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130760,"kb_used_data":4760,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93614392,"statfs":{"total":95995035648,"available":95861137408,"internally_reserved":0,"allocated":4874240,"data_stored":2333654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128514504},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855650,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108604,"kb_used_data":5236,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93636548,"statfs":{"total":95995035648,"available":95883825152,"internally_reserved":0,"allocated":5361664,"data_stored":2637041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463192,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94708,"kb_used_data":4984,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93650444,"statfs":{"total":95995035648,"available":95898054656,"internally_reserved":0,"allocated":5103616,"data_stored":2480387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:38:21.168 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-26T16:38:21.168 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-04-26T16:38:21.168 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-04-26T16:38:21.168 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:2724 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94191276270720 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e429)", 2024-04-26T16:38:21.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:28:24.590890+0000", 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 596.98778639, 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.67865985299999998, 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2724 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.590890+0000", 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.590890+0000", 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.590891+0000", 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.316e-06 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.590899+0000", 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1790000000000008e-06 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.590913+0000", 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.402e-05 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.590930+0000", 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6949e-05 2024-04-26T16:38:21.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.591198+0000", 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026779700000000001 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.591243+0000", 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4925000000000001e-05 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:25.269285+0000", 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.678042272 2024-04-26T16:38:21.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:25.269460+0000", 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000174618 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:25.269550+0000", 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9777000000000005e-05 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:2728 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94191276270720 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:28:24.592931+0000", 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 596.98574583200002, 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.68432920500000005, 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2728 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.592931+0000", 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.592931+0000", 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.592934+0000", 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0869999999999998e-06 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.592949+0000", 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5322e-05 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.593003+0000", 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4268000000000003e-05 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.593032+0000", 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9033000000000001e-05 2024-04-26T16:38:21.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.593206+0000", 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017420599999999999 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:24.593240+0000", 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3451999999999998e-05 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:25.269572+0000", 2024-04-26T16:38:21.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.67633192499999994 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:25.269619+0000", 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7258000000000001e-05 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:28:25.277260+0000", 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0076406540000000002 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3297 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94191276270720 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:33:01.880318+0000", 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 319.69835852599999, 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.44159035299999999, 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3297 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.588 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880318+0000", 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880318+0000", 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880321+0000", 2024-04-26T16:38:21.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5590000000000001e-06 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880339+0000", 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7693000000000002e-05 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880370+0000", 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1195999999999997e-05 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880390+0000", 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0165000000000001e-05 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880519+0000", 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012904400000000001 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:01.880536+0000", 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6504e-05 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:02.314822+0000", 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.43428618099999999 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:02.315016+0000", 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019402300000000001 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:02.321908+0000", 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0068919879999999999 2024-04-26T16:38:21.592 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:108 323.1s0 323:87691819:test-rados-api-smithi050-73372-115::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:33:47.342855+0000", 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 274.23582087099999, 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.059913427999999998, 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.593 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 108 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.342855+0000", 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.342855+0000", 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.342857+0000", 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9350000000000001e-06 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.342867+0000", 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9580000000000002e-06 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.342881+0000", 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3465000000000001e-05 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.342891+0000", 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0482999999999999e-05 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.349580+0000", 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006688523 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.353404+0000", 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038239400000000001 2024-04-26T16:38:21.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.354756+0000", 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013526720000000001 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.357682+0000", 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002925689 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.402671+0000", 2024-04-26T16:38:21.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.044989306999999999 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:33:47.402769+0000", 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7455999999999997e-05 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:635 326.8 326:10000000:test-rados-api-smithi050-73372-117:::head [pgnls start_epoch 697] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e697)", 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:34:25.812971+0000", 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 235.765704879, 2024-04-26T16:38:21.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081594739999999999, 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 635 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.812971+0000", 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.812971+0000", 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.812975+0000", 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.54e-06 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.812992+0000", 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6495999999999999e-05 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.813026+0000", 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4175999999999999e-05 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.813044+0000", 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8281999999999999e-05 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.813183+0000", 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013866599999999999 2024-04-26T16:38:21.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.813193+0000", 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0319999999999999e-05 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.820786+0000", 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075925680000000001 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.820884+0000", 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7986999999999995e-05 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:34:25.821131+0000", 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024743900000000001 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(osd.0.8:65 323.1s0 323:86af2be4:test-rados-api-smithi050-73372-125::bam:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e730)", 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:08.395330+0000", 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 193.18334618700001, 2024-04-26T16:38:21.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0094989159999999996, 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395330+0000", 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395330+0000", 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.604 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395332+0000", 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.897e-06 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395344+0000", 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1823e-05 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395371+0000", 2024-04-26T16:38:21.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6838999999999998e-05 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395389+0000", 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8600000000000001e-05 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395725+0000", 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000335977 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting_for_map not empty", 2024-04-26T16:38:21.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.395736+0000", 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0693000000000001e-05 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.400542+0000", 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048061700000000002 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.400712+0000", 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000169915 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:08.404829+0000", 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041170019999999998 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(2.f,QREQUEST,e:740,from:4,reservation_nonce:21)", 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:18.437728+0000", 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 183.140948278, 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010017559000000001, 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.608 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437728+0000", 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437728+0000", 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437730+0000", 2024-04-26T16:38:21.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0559999999999999e-06 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437741+0000", 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0366e-05 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437770+0000", 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.974e-05 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437787+0000", 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7176e-05 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437921+0000", 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013335199999999999 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.437933+0000", 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.199e-05 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.447651+0000", 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0097179620000000001 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.447682+0000", 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1458999999999998e-05 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:18.447746+0000", 2024-04-26T16:38:21.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3458000000000001e-05 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3592 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94191276270720 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:24.915236+0000", 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 176.66343991700001, 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48179307999999998, 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.613 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3592 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915236+0000", 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915236+0000", 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915239+0000", 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1169999999999998e-06 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915244+0000", 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2220000000000001e-06 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915259+0000", 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5495e-05 2024-04-26T16:38:21.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915272+0000", 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2972e-05 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915478+0000", 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020588900000000001 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.915507+0000", 2024-04-26T16:38:21.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8881999999999999e-05 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396735+0000", 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48122810300000002 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.396946+0000", 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021059900000000001 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.397030+0000", 2024-04-26T16:38:21.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3801000000000006e-05 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3595 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94191276270720 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:24.917504+0000", 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 176.661172781, 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50504878900000005, 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.618 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3595 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917504+0000", 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917504+0000", 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917505+0000", 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4600000000000003e-07 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917509+0000", 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5879999999999999e-06 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917522+0000", 2024-04-26T16:38:21.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2914999999999999e-05 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917531+0000", 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8300000000000002e-06 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917670+0000", 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013945 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.917692+0000", 2024-04-26T16:38:21.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1330000000000001e-05 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.398946+0000", 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48125463800000001 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.399039+0000", 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2645999999999994e-05 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.422552+0000", 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.023513446 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2184 334.c 334:30000000:%01:::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:50.468802+0000", 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 151.10987460800001, 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086657469999999997, 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.623 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2184 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.468802+0000", 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.468802+0000", 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.468807+0000", 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8099999999999997e-06 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.468823+0000", 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6192999999999999e-05 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.468861+0000", 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8371999999999998e-05 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.468887+0000", 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5426e-05 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.469056+0000", 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000169173 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.469072+0000", 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6629e-05 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.477113+0000", 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0080405100000000007 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.477203+0000", 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0403999999999997e-05 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.627 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.477468+0000", 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026423000000000001 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2279 335.8 335:14bec674:test-rados-api-smithi050-73372-135::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e778)", 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.418421+0000", 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 143.16025544799999, 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010950708999999999, 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.628 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2279 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418421+0000", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418421+0000", 2024-04-26T16:38:21.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418425+0000", 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9530000000000001e-06 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418448+0000", 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3065999999999999e-05 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418496+0000", 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7712e-05 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418525+0000", 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9425999999999999e-05 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418704+0000", 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017920800000000001 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.418731+0000", 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7027000000000001e-05 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.423655+0000", 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049236100000000001 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.424196+0000", 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000540619 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.426148+0000", 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001952709 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.426314+0000", 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016563 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,3", 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.426547+0000", 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023278700000000001 2024-04-26T16:38:21.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.427444+0000", 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000897633 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.427554+0000", 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109793 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429200+0000", 2024-04-26T16:38:21.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016463199999999999 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429311+0000", 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110948 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429372+0000", 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0268e-05 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2280 335.8 335:14bec674:test-rados-api-smithi050-73372-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.429721+0000", 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 143.148955688, 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0089667979999999998, 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2280 2024-04-26T16:38:21.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429721+0000", 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429721+0000", 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429723+0000", 2024-04-26T16:38:21.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1059999999999998e-06 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429733+0000", 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0587e-05 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429754+0000", 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0689000000000001e-05 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429768+0000", 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.42e-05 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429890+0000", 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121783 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.430042+0000", 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000151889 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438149+0000", 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081070210000000007 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438425+0000", 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027565599999999999 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438687+0000", 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026286700000000002 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.640 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2281 335.8 335:14bec674:test-rados-api-smithi050-73372-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.429797+0000", 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 143.14887934999999, 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0096069910000000005, 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2281 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.641 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429797+0000", 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429797+0000", 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429798+0000", 2024-04-26T16:38:21.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3400000000000001e-06 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429805+0000", 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5130000000000002e-06 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429823+0000", 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8085999999999999e-05 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.429843+0000", 2024-04-26T16:38:21.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0239999999999999e-05 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.430313+0000", 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00046944700000000001 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.430406+0000", 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2971000000000001e-05 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438856+0000", 2024-04-26T16:38:21.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084500010000000004 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.439160+0000", 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030484199999999999 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.439404+0000", 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000243551 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2284 335.8 335:14bec674:test-rados-api-smithi050-73372-135::foo:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.442784+0000", 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 143.135891982, 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0065730909999999997, 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2284 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.442784+0000", 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.442784+0000", 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.442786+0000", 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7439999999999999e-06 2024-04-26T16:38:21.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.442796+0000", 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5789999999999993e-06 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.442819+0000", 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3482999999999999e-05 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.442838+0000", 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8822999999999999e-05 2024-04-26T16:38:21.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.443338+0000", 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00049980700000000001 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.443476+0000", 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013782900000000001 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.449045+0000", 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055691539999999998 2024-04-26T16:38:21.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.449199+0000", 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015379599999999999 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.449357+0000", 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015887599999999999 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2287 335.8 335:14bec674:test-rados-api-smithi050-73372-135::foo:head [cache-flush] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e778)", 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.451702+0000", 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 143.126974859, 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070506249999999996, 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2287 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.451702+0000", 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.451702+0000", 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.451703+0000", 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0899999999999999e-06 2024-04-26T16:38:21.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.451707+0000", 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.279e-06 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.451723+0000", 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5979999999999999e-05 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.451732+0000", 2024-04-26T16:38:21.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3000000000000007e-06 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.452168+0000", 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000435492 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.452331+0000", 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000163296 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.458253+0000", 2024-04-26T16:38:21.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059221279999999996 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.458509+0000", 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000255536 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.458752+0000", 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024352400000000001 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2327 335.8 335:10000000::::head [pgnls start_epoch 780] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e780)", 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:00.434447+0000", 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 141.14422904899999, 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01391097, 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2327 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434447+0000", 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434447+0000", 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434451+0000", 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.557e-06 2024-04-26T16:38:21.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434466+0000", 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5223e-05 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434502+0000", 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6118999999999998e-05 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434522+0000", 2024-04-26T16:38:21.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9706000000000001e-05 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434705+0000", 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000183332 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.434725+0000", 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0060000000000001e-05 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.448053+0000", 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013327617 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.448153+0000", 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9794e-05 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.448358+0000", 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020556200000000001 2024-04-26T16:38:21.660 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(11.e,QREQUEST,e:816,from:4,reservation_nonce:13)", 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:45.921423+0000", 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 95.657252889000006, 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069618759999999997, 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921423+0000", 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921423+0000", 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921425+0000", 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9199999999999998e-06 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921437+0000", 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1542999999999999e-05 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921468+0000", 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0746e-05 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921490+0000", 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1965999999999999e-05 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921615+0000", 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012544699999999999 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.921626+0000", 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1353e-05 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.928305+0000", 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066781920000000003 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.928332+0000", 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7016e-05 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:45.928385+0000", 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3693000000000002e-05 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.665 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2814 339.c 339:30000000:%01:::head [pgnls start_epoch 823] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e823)", 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:52.421329+0000", 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 89.157347232000006, 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0070307360000000001, 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2814 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421329+0000", 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421329+0000", 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421332+0000", 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1370000000000002e-06 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421351+0000", 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9145999999999999e-05 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421391+0000", 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9218999999999998e-05 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421411+0000", 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0656e-05 2024-04-26T16:38:21.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421576+0000", 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016516600000000001 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.421588+0000", 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.137e-05 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.428014+0000", 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064256799999999996 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.428112+0000", 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8078000000000005e-05 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:52.428360+0000", 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024828399999999998 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.670 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:3336 341.5 341:a186d50f:test-rados-api-smithi050-73372-147::bam:head [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e839)", 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:37:15.455719+0000", 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 66.122957307999997, 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064168480000000002, 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3336 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.671 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455719+0000", 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455719+0000", 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455722+0000", 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.129e-06 2024-04-26T16:38:21.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455734+0000", 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2089999999999999e-05 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455766+0000", 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1359000000000002e-05 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.673 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455786+0000", 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.022e-05 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.455937+0000", 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015072999999999999 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for blocked object", 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.456190+0000", 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000253043 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.458237+0000", 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002047512 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.458323+0000", 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6051000000000007e-05 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.459316+0000", 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00099303200000000003 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.459385+0000", 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8596000000000005e-05 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.462136+0000", 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027510859999999998 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.676 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(5.19,QREQUEST,e:874,from:1,reservation_nonce:9)", 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:38:16.154724+0000", 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 5.4239527110000001, 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046201189999999998, 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154724+0000", 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154724+0000", 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154726+0000", 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1399999999999998e-06 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154736+0000", 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8789999999999998e-06 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154764+0000", 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.809e-05 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154779+0000", 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5106000000000001e-05 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154923+0000", 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014403699999999999 2024-04-26T16:38:21.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.154934+0000", 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1e-05 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.159197+0000", 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042634400000000003 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.159242+0000", 2024-04-26T16:38:21.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4499000000000002e-05 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:16.159344+0000", 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000101928 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:21.681 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.980060+0000 mgr.y (mgr.24401) 801 : cluster [DBG] pgmap v1376: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:19.980060+0000 mgr.y (mgr.24401) 801 : cluster [DBG] pgmap v1376: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.154570+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.154570+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.156950+0000 osd.1 (osd.1) 466 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.156950+0000 osd.1 (osd.1) 466 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.569626+0000 osd.3 (osd.3) 515 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.569626+0000 osd.3 (osd.3) 515 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.573621+0000 osd.3 (osd.3) 516 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.573621+0000 osd.3 (osd.3) 516 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:38:22.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.739948+0000 osd.4 (osd.4) 439 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.739948+0000 osd.4 (osd.4) 439 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.744129+0000 osd.4 (osd.4) 440 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.744129+0000 osd.4 (osd.4) 440 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.885609+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.885609+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.888112+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:20.888112+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:21.435938+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-04-26T16:38:22.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:21 smithi190 bash[23868]: cluster 2024-04-26T16:38:21.435938+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-04-26T16:38:22.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.980060+0000 mgr.y (mgr.24401) 801 : cluster [DBG] pgmap v1376: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:19.980060+0000 mgr.y (mgr.24401) 801 : cluster [DBG] pgmap v1376: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.154570+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.154570+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.156950+0000 osd.1 (osd.1) 466 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.156950+0000 osd.1 (osd.1) 466 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.569626+0000 osd.3 (osd.3) 515 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.569626+0000 osd.3 (osd.3) 515 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.573621+0000 osd.3 (osd.3) 516 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.573621+0000 osd.3 (osd.3) 516 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.739948+0000 osd.4 (osd.4) 439 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.739948+0000 osd.4 (osd.4) 439 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.744129+0000 osd.4 (osd.4) 440 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.744129+0000 osd.4 (osd.4) 440 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.885609+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:38:22.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.885609+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.888112+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:20.888112+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:21.435938+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[21463]: cluster 2024-04-26T16:38:21.435938+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.980060+0000 mgr.y (mgr.24401) 801 : cluster [DBG] pgmap v1376: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:19.980060+0000 mgr.y (mgr.24401) 801 : cluster [DBG] pgmap v1376: 164 pgs: 164 active+clean; 583 KiB data, 790 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.154570+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.154570+0000 osd.1 (osd.1) 465 : cluster [DBG] 5.c deep-scrub starts 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.156950+0000 osd.1 (osd.1) 466 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.156950+0000 osd.1 (osd.1) 466 : cluster [DBG] 5.c deep-scrub ok 2024-04-26T16:38:22.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.569626+0000 osd.3 (osd.3) 515 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.569626+0000 osd.3 (osd.3) 515 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.573621+0000 osd.3 (osd.3) 516 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.573621+0000 osd.3 (osd.3) 516 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.739948+0000 osd.4 (osd.4) 439 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.739948+0000 osd.4 (osd.4) 439 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.744129+0000 osd.4 (osd.4) 440 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.744129+0000 osd.4 (osd.4) 440 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.885609+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.885609+0000 osd.0 (osd.0) 495 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.888112+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:20.888112+0000 osd.0 (osd.0) 496 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:21.435938+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-04-26T16:38:22.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:21 smithi050 bash[29234]: cluster 2024-04-26T16:38:21.435938+0000 mon.a (mon.0) 3959 : cluster [DBG] osdmap e877: 8 total, 8 up, 8 in 2024-04-26T16:38:22.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-04-26T16:38:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.008253+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:38:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.008253+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:38:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.014108+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:38:23.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.014108+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:38:23.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.237662+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.f scrub starts 2024-04-26T16:38:23.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.237662+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.f scrub starts 2024-04-26T16:38:23.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.240277+0000 osd.5 (osd.5) 345 : cluster [DBG] 5.f scrub ok 2024-04-26T16:38:23.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:22 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.240277+0000 osd.5 (osd.5) 345 : cluster [DBG] 5.f scrub ok 2024-04-26T16:38:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.008253+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:38:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.008253+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:38:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.014108+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:38:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.014108+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:38:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.237662+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.f scrub starts 2024-04-26T16:38:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.237662+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.f scrub starts 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.240277+0000 osd.5 (osd.5) 345 : cluster [DBG] 5.f scrub ok 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.240277+0000 osd.5 (osd.5) 345 : cluster [DBG] 5.f scrub ok 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.008253+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.008253+0000 osd.7 (osd.7) 458 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.014108+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.014108+0000 osd.7 (osd.7) 459 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.237662+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.f scrub starts 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.237662+0000 osd.5 (osd.5) 344 : cluster [DBG] 5.f scrub starts 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.240277+0000 osd.5 (osd.5) 345 : cluster [DBG] 5.f scrub ok 2024-04-26T16:38:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:22 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.240277+0000 osd.5 (osd.5) 345 : cluster [DBG] 5.f scrub ok 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:21.981746+0000 mgr.y (mgr.24401) 802 : cluster [DBG] pgmap v1378: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:21.981746+0000 mgr.y (mgr.24401) 802 : cluster [DBG] pgmap v1378: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.127569+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.127569+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.129981+0000 osd.1 (osd.1) 468 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.129981+0000 osd.1 (osd.1) 468 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.519351+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.519351+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.523515+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.523515+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.764037+0000 osd.4 (osd.4) 441 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:38:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.764037+0000 osd.4 (osd.4) 441 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:38:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.766777+0000 osd.4 (osd.4) 442 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:38:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.766777+0000 osd.4 (osd.4) 442 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:38:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.870162+0000 osd.0 (osd.0) 497 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:38:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.870162+0000 osd.0 (osd.0) 497 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:38:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.872564+0000 osd.0 (osd.0) 498 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:38:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:23 smithi190 bash[23868]: cluster 2024-04-26T16:38:22.872564+0000 osd.0 (osd.0) 498 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:38:24.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:21.981746+0000 mgr.y (mgr.24401) 802 : cluster [DBG] pgmap v1378: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:38:24.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:21.981746+0000 mgr.y (mgr.24401) 802 : cluster [DBG] pgmap v1378: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.127569+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.127569+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.129981+0000 osd.1 (osd.1) 468 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.129981+0000 osd.1 (osd.1) 468 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.519351+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.519351+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.523515+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.523515+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.764037+0000 osd.4 (osd.4) 441 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.764037+0000 osd.4 (osd.4) 441 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.766777+0000 osd.4 (osd.4) 442 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.766777+0000 osd.4 (osd.4) 442 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.870162+0000 osd.0 (osd.0) 497 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:38:24.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.870162+0000 osd.0 (osd.0) 497 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.872564+0000 osd.0 (osd.0) 498 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[21463]: cluster 2024-04-26T16:38:22.872564+0000 osd.0 (osd.0) 498 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:21.981746+0000 mgr.y (mgr.24401) 802 : cluster [DBG] pgmap v1378: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:21.981746+0000 mgr.y (mgr.24401) 802 : cluster [DBG] pgmap v1378: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.127569+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.127569+0000 osd.1 (osd.1) 467 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.129981+0000 osd.1 (osd.1) 468 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.129981+0000 osd.1 (osd.1) 468 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.519351+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.519351+0000 osd.3 (osd.3) 517 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.523515+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:38:24.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.523515+0000 osd.3 (osd.3) 518 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.764037+0000 osd.4 (osd.4) 441 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.764037+0000 osd.4 (osd.4) 441 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.766777+0000 osd.4 (osd.4) 442 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.766777+0000 osd.4 (osd.4) 442 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.870162+0000 osd.0 (osd.0) 497 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.870162+0000 osd.0 (osd.0) 497 : cluster [DBG] 5.1e deep-scrub starts 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.872564+0000 osd.0 (osd.0) 498 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:38:24.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:23 smithi050 bash[29234]: cluster 2024-04-26T16:38:22.872564+0000 osd.0 (osd.0) 498 : cluster [DBG] 5.1e deep-scrub ok 2024-04-26T16:38:24.743 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:38:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:38:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:23.830054+0000 osd.0 (osd.0) 499 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:38:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:23.830054+0000 osd.0 (osd.0) 499 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:38:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:23.833235+0000 osd.0 (osd.0) 500 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:38:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:23.833235+0000 osd.0 (osd.0) 500 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:38:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.177299+0000 osd.5 (osd.5) 346 : cluster [DBG] 3.c scrub starts 2024-04-26T16:38:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.177299+0000 osd.5 (osd.5) 346 : cluster [DBG] 3.c scrub starts 2024-04-26T16:38:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.181646+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub ok 2024-04-26T16:38:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:24 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.181646+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub ok 2024-04-26T16:38:25.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:23.830054+0000 osd.0 (osd.0) 499 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:38:25.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:23.830054+0000 osd.0 (osd.0) 499 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:38:25.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:23.833235+0000 osd.0 (osd.0) 500 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:38:25.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:23.833235+0000 osd.0 (osd.0) 500 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:38:25.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.177299+0000 osd.5 (osd.5) 346 : cluster [DBG] 3.c scrub starts 2024-04-26T16:38:25.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.177299+0000 osd.5 (osd.5) 346 : cluster [DBG] 3.c scrub starts 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.181646+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub ok 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.181646+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub ok 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:23.830054+0000 osd.0 (osd.0) 499 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:23.830054+0000 osd.0 (osd.0) 499 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:23.833235+0000 osd.0 (osd.0) 500 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:23.833235+0000 osd.0 (osd.0) 500 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.177299+0000 osd.5 (osd.5) 346 : cluster [DBG] 3.c scrub starts 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.177299+0000 osd.5 (osd.5) 346 : cluster [DBG] 3.c scrub starts 2024-04-26T16:38:25.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.181646+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub ok 2024-04-26T16:38:25.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:24 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.181646+0000 osd.5 (osd.5) 347 : cluster [DBG] 3.c scrub ok 2024-04-26T16:38:26.006 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:26.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:23.982376+0000 mgr.y (mgr.24401) 803 : cluster [DBG] pgmap v1379: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:26.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:23.982376+0000 mgr.y (mgr.24401) 803 : cluster [DBG] pgmap v1379: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:26.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.210948+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.d scrub starts 2024-04-26T16:38:26.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.210948+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.d scrub starts 2024-04-26T16:38:26.023 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.213495+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.d scrub ok 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.213495+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.d scrub ok 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.829678+0000 osd.0 (osd.0) 501 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.829678+0000 osd.0 (osd.0) 501 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.832170+0000 osd.0 (osd.0) 502 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:24.832170+0000 osd.0 (osd.0) 502 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.001597+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.001597+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.003886+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.003886+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: audit 2024-04-26T16:38:25.529880+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[29234]: audit 2024-04-26T16:38:25.529880+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:26.024 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:23.982376+0000 mgr.y (mgr.24401) 803 : cluster [DBG] pgmap v1379: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:23.982376+0000 mgr.y (mgr.24401) 803 : cluster [DBG] pgmap v1379: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.210948+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.d scrub starts 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.210948+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.d scrub starts 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.213495+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.d scrub ok 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.213495+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.d scrub ok 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.829678+0000 osd.0 (osd.0) 501 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.829678+0000 osd.0 (osd.0) 501 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.832170+0000 osd.0 (osd.0) 502 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:24.832170+0000 osd.0 (osd.0) 502 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.001597+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.001597+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.003886+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.003886+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:38:26.025 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: audit 2024-04-26T16:38:25.529880+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:26.026 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:25 smithi050 bash[21463]: audit 2024-04-26T16:38:25.529880+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:23.982376+0000 mgr.y (mgr.24401) 803 : cluster [DBG] pgmap v1379: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:23.982376+0000 mgr.y (mgr.24401) 803 : cluster [DBG] pgmap v1379: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.210948+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.d scrub starts 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.210948+0000 osd.1 (osd.1) 469 : cluster [DBG] 2.d scrub starts 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.213495+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.d scrub ok 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.213495+0000 osd.1 (osd.1) 470 : cluster [DBG] 2.d scrub ok 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.829678+0000 osd.0 (osd.0) 501 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.829678+0000 osd.0 (osd.0) 501 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.832170+0000 osd.0 (osd.0) 502 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:24.832170+0000 osd.0 (osd.0) 502 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.001597+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.001597+0000 osd.7 (osd.7) 460 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.003886+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:38:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.003886+0000 osd.7 (osd.7) 461 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:38:26.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: audit 2024-04-26T16:38:25.529880+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:26.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:25 smithi190 bash[23868]: audit 2024-04-26T16:38:25.529880+0000 mon.c (mon.2) 911 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.220490+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.220490+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.222889+0000 osd.1 (osd.1) 472 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.222889+0000 osd.1 (osd.1) 472 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.819727+0000 osd.4 (osd.4) 443 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.819727+0000 osd.4 (osd.4) 443 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.821457+0000 osd.4 (osd.4) 444 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:38:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.821457+0000 osd.4 (osd.4) 444 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.837406+0000 osd.0 (osd.0) 503 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.837406+0000 osd.0 (osd.0) 503 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.839564+0000 osd.0 (osd.0) 504 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.839564+0000 osd.0 (osd.0) 504 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.043889+0000 osd.7 (osd.7) 462 : cluster [DBG] 2.0 deep-scrub starts 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.043889+0000 osd.7 (osd.7) 462 : cluster [DBG] 2.0 deep-scrub starts 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.045879+0000 osd.7 (osd.7) 463 : cluster [DBG] 2.0 deep-scrub ok 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.045879+0000 osd.7 (osd.7) 463 : cluster [DBG] 2.0 deep-scrub ok 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: audit 2024-04-26T16:38:26.108928+0000 mon.a (mon.0) 3960 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: audit 2024-04-26T16:38:26.108928+0000 mon.a (mon.0) 3960 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: audit 2024-04-26T16:38:26.123465+0000 mon.a (mon.0) 3961 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: audit 2024-04-26T16:38:26.123465+0000 mon.a (mon.0) 3961 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.271208+0000 osd.4 (osd.4) 445 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:27.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.271208+0000 osd.4 (osd.4) 445 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:27.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.271652+0000 osd.4 (osd.4) 446 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:27.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.271652+0000 osd.4 (osd.4) 446 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:27.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.482456+0000 osd.2 (osd.2) 431 : cluster [DBG] 5.a scrub starts 2024-04-26T16:38:27.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.482456+0000 osd.2 (osd.2) 431 : cluster [DBG] 5.a scrub starts 2024-04-26T16:38:27.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.483783+0000 osd.2 (osd.2) 432 : cluster [DBG] 5.a scrub ok 2024-04-26T16:38:27.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:26 smithi190 bash[23868]: cluster 2024-04-26T16:38:26.483783+0000 osd.2 (osd.2) 432 : cluster [DBG] 5.a scrub ok 2024-04-26T16:38:27.118 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.220490+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.220490+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.222889+0000 osd.1 (osd.1) 472 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.222889+0000 osd.1 (osd.1) 472 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.819727+0000 osd.4 (osd.4) 443 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.819727+0000 osd.4 (osd.4) 443 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.821457+0000 osd.4 (osd.4) 444 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.821457+0000 osd.4 (osd.4) 444 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.837406+0000 osd.0 (osd.0) 503 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.837406+0000 osd.0 (osd.0) 503 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:38:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.839564+0000 osd.0 (osd.0) 504 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.839564+0000 osd.0 (osd.0) 504 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.043889+0000 osd.7 (osd.7) 462 : cluster [DBG] 2.0 deep-scrub starts 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.043889+0000 osd.7 (osd.7) 462 : cluster [DBG] 2.0 deep-scrub starts 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.045879+0000 osd.7 (osd.7) 463 : cluster [DBG] 2.0 deep-scrub ok 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.045879+0000 osd.7 (osd.7) 463 : cluster [DBG] 2.0 deep-scrub ok 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: audit 2024-04-26T16:38:26.108928+0000 mon.a (mon.0) 3960 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: audit 2024-04-26T16:38:26.108928+0000 mon.a (mon.0) 3960 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: audit 2024-04-26T16:38:26.123465+0000 mon.a (mon.0) 3961 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: audit 2024-04-26T16:38:26.123465+0000 mon.a (mon.0) 3961 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.271208+0000 osd.4 (osd.4) 445 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.271208+0000 osd.4 (osd.4) 445 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.271652+0000 osd.4 (osd.4) 446 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.271652+0000 osd.4 (osd.4) 446 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.482456+0000 osd.2 (osd.2) 431 : cluster [DBG] 5.a scrub starts 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.482456+0000 osd.2 (osd.2) 431 : cluster [DBG] 5.a scrub starts 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.483783+0000 osd.2 (osd.2) 432 : cluster [DBG] 5.a scrub ok 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[21463]: cluster 2024-04-26T16:38:26.483783+0000 osd.2 (osd.2) 432 : cluster [DBG] 5.a scrub ok 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.220490+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.220490+0000 osd.1 (osd.1) 471 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.222889+0000 osd.1 (osd.1) 472 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:38:27.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.222889+0000 osd.1 (osd.1) 472 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.819727+0000 osd.4 (osd.4) 443 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.819727+0000 osd.4 (osd.4) 443 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.821457+0000 osd.4 (osd.4) 444 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.821457+0000 osd.4 (osd.4) 444 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.837406+0000 osd.0 (osd.0) 503 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.837406+0000 osd.0 (osd.0) 503 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.839564+0000 osd.0 (osd.0) 504 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.839564+0000 osd.0 (osd.0) 504 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.043889+0000 osd.7 (osd.7) 462 : cluster [DBG] 2.0 deep-scrub starts 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.043889+0000 osd.7 (osd.7) 462 : cluster [DBG] 2.0 deep-scrub starts 2024-04-26T16:38:27.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.045879+0000 osd.7 (osd.7) 463 : cluster [DBG] 2.0 deep-scrub ok 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.045879+0000 osd.7 (osd.7) 463 : cluster [DBG] 2.0 deep-scrub ok 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: audit 2024-04-26T16:38:26.108928+0000 mon.a (mon.0) 3960 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: audit 2024-04-26T16:38:26.108928+0000 mon.a (mon.0) 3960 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: audit 2024-04-26T16:38:26.123465+0000 mon.a (mon.0) 3961 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: audit 2024-04-26T16:38:26.123465+0000 mon.a (mon.0) 3961 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.271208+0000 osd.4 (osd.4) 445 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.271208+0000 osd.4 (osd.4) 445 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.271652+0000 osd.4 (osd.4) 446 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.271652+0000 osd.4 (osd.4) 446 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.482456+0000 osd.2 (osd.2) 431 : cluster [DBG] 5.a scrub starts 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.482456+0000 osd.2 (osd.2) 431 : cluster [DBG] 5.a scrub starts 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.483783+0000 osd.2 (osd.2) 432 : cluster [DBG] 5.a scrub ok 2024-04-26T16:38:27.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:26 smithi050 bash[29234]: cluster 2024-04-26T16:38:26.483783+0000 osd.2 (osd.2) 432 : cluster [DBG] 5.a scrub ok 2024-04-26T16:38:27.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:38:28.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.982844+0000 mgr.y (mgr.24401) 804 : cluster [DBG] pgmap v1380: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:28.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: cluster 2024-04-26T16:38:25.982844+0000 mgr.y (mgr.24401) 804 : cluster [DBG] pgmap v1380: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: audit 2024-04-26T16:38:26.757439+0000 mon.c (mon.2) 912 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: audit 2024-04-26T16:38:26.757439+0000 mon.c (mon.2) 912 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: audit 2024-04-26T16:38:26.764847+0000 mon.c (mon.2) 913 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: audit 2024-04-26T16:38:26.764847+0000 mon.c (mon.2) 913 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: audit 2024-04-26T16:38:26.770415+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: audit 2024-04-26T16:38:26.770415+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: cluster 2024-04-26T16:38:27.129845+0000 mon.a (mon.0) 3963 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-04-26T16:38:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:27 smithi190 bash[23868]: cluster 2024-04-26T16:38:27.129845+0000 mon.a (mon.0) 3963 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-04-26T16:38:28.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.982844+0000 mgr.y (mgr.24401) 804 : cluster [DBG] pgmap v1380: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: cluster 2024-04-26T16:38:25.982844+0000 mgr.y (mgr.24401) 804 : cluster [DBG] pgmap v1380: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: audit 2024-04-26T16:38:26.757439+0000 mon.c (mon.2) 912 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: audit 2024-04-26T16:38:26.757439+0000 mon.c (mon.2) 912 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: audit 2024-04-26T16:38:26.764847+0000 mon.c (mon.2) 913 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: audit 2024-04-26T16:38:26.764847+0000 mon.c (mon.2) 913 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: audit 2024-04-26T16:38:26.770415+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: audit 2024-04-26T16:38:26.770415+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: cluster 2024-04-26T16:38:27.129845+0000 mon.a (mon.0) 3963 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[21463]: cluster 2024-04-26T16:38:27.129845+0000 mon.a (mon.0) 3963 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-04-26T16:38:28.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.982844+0000 mgr.y (mgr.24401) 804 : cluster [DBG] pgmap v1380: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: cluster 2024-04-26T16:38:25.982844+0000 mgr.y (mgr.24401) 804 : cluster [DBG] pgmap v1380: 164 pgs: 164 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: audit 2024-04-26T16:38:26.757439+0000 mon.c (mon.2) 912 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: audit 2024-04-26T16:38:26.757439+0000 mon.c (mon.2) 912 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: audit 2024-04-26T16:38:26.764847+0000 mon.c (mon.2) 913 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: audit 2024-04-26T16:38:26.764847+0000 mon.c (mon.2) 913 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: audit 2024-04-26T16:38:26.770415+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: audit 2024-04-26T16:38:26.770415+0000 mon.a (mon.0) 3962 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: cluster 2024-04-26T16:38:27.129845+0000 mon.a (mon.0) 3963 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-04-26T16:38:28.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:27 smithi050 bash[29234]: cluster 2024-04-26T16:38:27.129845+0000 mon.a (mon.0) 3963 : cluster [DBG] osdmap e878: 8 total, 8 up, 8 in 2024-04-26T16:38:29.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: audit 2024-04-26T16:38:27.389488+0000 mgr.y (mgr.24401) 805 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:29.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: audit 2024-04-26T16:38:27.389488+0000 mgr.y (mgr.24401) 805 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:29.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.051120+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:38:29.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.051120+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:38:29.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.054618+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:38:29.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.054618+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:38:29.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: audit 2024-04-26T16:38:28.135320+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:29.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:28 smithi190 bash[23868]: audit 2024-04-26T16:38:28.135320+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: audit 2024-04-26T16:38:27.389488+0000 mgr.y (mgr.24401) 805 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: audit 2024-04-26T16:38:27.389488+0000 mgr.y (mgr.24401) 805 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.051120+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.051120+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.054618+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.054618+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: audit 2024-04-26T16:38:28.135320+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[21463]: audit 2024-04-26T16:38:28.135320+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: audit 2024-04-26T16:38:27.389488+0000 mgr.y (mgr.24401) 805 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: audit 2024-04-26T16:38:27.389488+0000 mgr.y (mgr.24401) 805 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.051120+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.051120+0000 osd.7 (osd.7) 464 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.054618+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.054618+0000 osd.7 (osd.7) 465 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: audit 2024-04-26T16:38:28.135320+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:28 smithi050 bash[29234]: audit 2024-04-26T16:38:28.135320+0000 mon.c (mon.2) 914 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:29.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- 172.21.15.50:0/1650504663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 msgr2=0x7f9dec112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:29.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- 172.21.15.50:0/1650504663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec112510 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009f90 tx=0x7f9de400b330 comp rx=0 tx=0).stop 2024-04-26T16:38:29.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- 172.21.15.50:0/1650504663 shutdown_connections 2024-04-26T16:38:29.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- 172.21.15.50:0/1650504663 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec112510 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- 172.21.15.50:0/1650504663 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dec10b640 0x7f9dec10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- 172.21.15.50:0/1650504663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec105640 0x7f9dec105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- 172.21.15.50:0/1650504663 >> 172.21.15.50:0/1650504663 conn(0x7f9dec101130 msgr2=0x7f9dec103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- 172.21.15.50:0/1650504663 shutdown_connections 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- 172.21.15.50:0/1650504663 wait complete. 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 librados: starting msgr at 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 librados: starting objecter 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 Processor -- start 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- start start 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 librados: setting wanted keys 2024-04-26T16:38:29.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 librados: calling monclient init 2024-04-26T16:38:29.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dec105640 0x7f9dec19b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:29.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec19b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:29.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec10b640 0x7f9dec19fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:29.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec114d60 con 0x7f9dec105640 2024-04-26T16:38:29.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9dec114be0 con 0x7f9dec10b640 2024-04-26T16:38:29.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df4279700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9dec114ee0 con 0x7f9dec106000 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df1814700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec19b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df1814700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec19b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51476/0 (socket says 172.21.15.50:51476) 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.630+0000 7f9df1814700 1 -- 172.21.15.50:0/293964334 learned_addr learned my addr 172.21.15.50:0/293964334 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df2015700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dec105640 0x7f9dec19b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df2816700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec10b640 0x7f9dec19fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df1814700 1 -- 172.21.15.50:0/293964334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec10b640 msgr2=0x7f9dec19fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df1814700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec10b640 0x7f9dec19fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df1814700 1 -- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dec105640 msgr2=0x7f9dec19b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df1814700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dec105640 0x7f9dec19b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df1814700 1 -- 172.21.15.50:0/293964334 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec1a0310 con 0x7f9dec106000 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df1814700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec19b840 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00d980 tx=0x7f9ddc00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 10 client.?.objecter ms_handle_connect 0x7f9dec106000 2024-04-26T16:38:29.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc017070 con 0x7f9dec106000 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dec1a0600 con 0x7f9dec106000 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9ddc005020 con 0x7f9dec106000 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc009990 con 0x7f9dec106000 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df4279700 10 client.41762.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dec1a7e80 con 0x7f9dec106000 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df4279700 1 librados: init done 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dec06a110 con 0x7f9dec106000 2024-04-26T16:38:29.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9ddc004a30 con 0x7f9dec106000 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9dd8071f80 0x7f9dd8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(878..878 src has 827..878) ==== 8707+0+0 (secure 0 0 0) 0x7f9ddc091630 con 0x7f9dec106000 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 10 client.41762.objecter ms_dispatch 0x7f9dec001550 osd_map(878..878 src has 827..878) 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 3 client.41762.objecter handle_osd_map got epochs [878,878] > 0 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 3 client.41762.objecter handle_osd_map decoding full epoch 878 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df2015700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9dd8071f80 0x7f9dd8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 20 client.41762.objecter dump_active .. 0 homeless 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9df2015700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9dd8071f80 0x7f9dd8074440 secure :-1 s=READY pgs=3852 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009750 tx=0x7f9de8009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:29.635 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.634+0000 7f9de37fe700 10 client.41762.objecter ms_handle_connect 0x7f9dd8071f80 2024-04-26T16:38:29.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.642+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f9ddc020020 con 0x7f9dec106000 2024-04-26T16:38:29.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.862+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9dec19c660 con 0x7f9dd8071f80 2024-04-26T16:38:29.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.878+0000 7f9de37fe700 1 -- 172.21.15.50:0/293964334 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+426707 (secure 0 0 0) 0x7f9dec19c660 con 0x7f9dd8071f80 2024-04-26T16:38:29.880 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:38:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 10 librados: watch_flush enter 2024-04-26T16:38:29.884 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 10 librados: watch_flush exit 2024-04-26T16:38:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 20 client.41762.objecter shutdown clearing up homeless session... 2024-04-26T16:38:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 10 client.41762.objecter successfully canceled tick 2024-04-26T16:38:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9dd8071f80 msgr2=0x7f9dd8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9dd8071f80 0x7f9dd8074440 secure :-1 s=READY pgs=3852 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009750 tx=0x7f9de8009310 comp rx=0 tx=0).stop 2024-04-26T16:38:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 msgr2=0x7f9dec19b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:29.885 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec19b840 secure :-1 s=READY pgs=2774 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00d980 tx=0x7f9ddc00dc90 comp rx=0 tx=0).stop 2024-04-26T16:38:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 shutdown_connections 2024-04-26T16:38:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9dd8071f80 0x7f9dd8074440 unknown :-1 s=CLOSED pgs=3852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9dec10b640 0x7f9dec19fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9dec105640 0x7f9dec19b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 --2- 172.21.15.50:0/293964334 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9dec106000 0x7f9dec19b840 unknown :-1 s=CLOSED pgs=2774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:29.886 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 >> 172.21.15.50:0/293964334 conn(0x7f9dec101130 msgr2=0x7f9dec101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 shutdown_connections 2024-04-26T16:38:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 -- 172.21.15.50:0/293964334 wait complete. 2024-04-26T16:38:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:29.882+0000 7f9df4279700 1 librados: shutdown 2024-04-26T16:38:29.887 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:38:29.993 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:38:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:38:29.993 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-04-26T16:38:29.993 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:38:30.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:27.983557+0000 mgr.y (mgr.24401) 806 : cluster [DBG] pgmap v1382: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:27.983557+0000 mgr.y (mgr.24401) 806 : cluster [DBG] pgmap v1382: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.408907+0000 osd.3 (osd.3) 519 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.408907+0000 osd.3 (osd.3) 519 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.411249+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.411249+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.776680+0000 osd.0 (osd.0) 505 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.776680+0000 osd.0 (osd.0) 505 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.779217+0000 osd.0 (osd.0) 506 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:28.779217+0000 osd.0 (osd.0) 506 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.082466+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.082466+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.084295+0000 osd.5 (osd.5) 349 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:38:30.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.084295+0000 osd.5 (osd.5) 349 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:38:30.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.436567+0000 osd.2 (osd.2) 433 : cluster [DBG] 6.f scrub starts 2024-04-26T16:38:30.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.436567+0000 osd.2 (osd.2) 433 : cluster [DBG] 6.f scrub starts 2024-04-26T16:38:30.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.438910+0000 osd.2 (osd.2) 434 : cluster [DBG] 6.f scrub ok 2024-04-26T16:38:30.039 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:29 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.438910+0000 osd.2 (osd.2) 434 : cluster [DBG] 6.f scrub ok 2024-04-26T16:38:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:27.983557+0000 mgr.y (mgr.24401) 806 : cluster [DBG] pgmap v1382: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:27.983557+0000 mgr.y (mgr.24401) 806 : cluster [DBG] pgmap v1382: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.408907+0000 osd.3 (osd.3) 519 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.408907+0000 osd.3 (osd.3) 519 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.411249+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.411249+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.776680+0000 osd.0 (osd.0) 505 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.776680+0000 osd.0 (osd.0) 505 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.779217+0000 osd.0 (osd.0) 506 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:28.779217+0000 osd.0 (osd.0) 506 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.082466+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.082466+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.084295+0000 osd.5 (osd.5) 349 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.084295+0000 osd.5 (osd.5) 349 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:38:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.436567+0000 osd.2 (osd.2) 433 : cluster [DBG] 6.f scrub starts 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.436567+0000 osd.2 (osd.2) 433 : cluster [DBG] 6.f scrub starts 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.438910+0000 osd.2 (osd.2) 434 : cluster [DBG] 6.f scrub ok 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.438910+0000 osd.2 (osd.2) 434 : cluster [DBG] 6.f scrub ok 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:27.983557+0000 mgr.y (mgr.24401) 806 : cluster [DBG] pgmap v1382: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:27.983557+0000 mgr.y (mgr.24401) 806 : cluster [DBG] pgmap v1382: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.408907+0000 osd.3 (osd.3) 519 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.408907+0000 osd.3 (osd.3) 519 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.411249+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.411249+0000 osd.3 (osd.3) 520 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.776680+0000 osd.0 (osd.0) 505 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.776680+0000 osd.0 (osd.0) 505 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:38:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.779217+0000 osd.0 (osd.0) 506 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:28.779217+0000 osd.0 (osd.0) 506 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.082466+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.082466+0000 osd.5 (osd.5) 348 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.084295+0000 osd.5 (osd.5) 349 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.084295+0000 osd.5 (osd.5) 349 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.436567+0000 osd.2 (osd.2) 433 : cluster [DBG] 6.f scrub starts 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.436567+0000 osd.2 (osd.2) 433 : cluster [DBG] 6.f scrub starts 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.438910+0000 osd.2 (osd.2) 434 : cluster [DBG] 6.f scrub ok 2024-04-26T16:38:30.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:29 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.438910+0000 osd.2 (osd.2) 434 : cluster [DBG] 6.f scrub ok 2024-04-26T16:38:30.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-04-26T16:38:30.814 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1382,"stamp":"2024-04-26T16:38:27.983045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596512,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3037,"num_read_kb":2883,"num_write":1194,"num_write_kb":2468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":1774538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1355,"ondisk_log_size":1355,"up":492,"acting":492,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":802064,"kb_used_data":36328,"kb_used_omap":34,"kb_used_meta":765149,"kb_avail":749159152,"statfs":{"total":767960285184,"available":767138971648,"internally_reserved":0,"allocated":37199872,"data_stored":18079696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34917,"internal_metadata":783513499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003534"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996431+0000","last_change":"2024-04-26T16:38:09.296402+0000","last_active":"2024-04-26T16:38:21.996431+0000","last_peered":"2024-04-26T16:38:21.996431+0000","last_clean":"2024-04-26T16:38:21.996431+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:38:21.996431+0000","last_undegraded":"2024-04-26T16:38:21.996431+0000","last_fullsized":"2024-04-26T16:38:21.996431+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.296138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.296138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.902182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"747'16","reported_seq":1929,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009298+0000","last_change":"2024-04-26T16:36:20.159718+0000","last_active":"2024-04-26T16:38:22.009298+0000","last_peered":"2024-04-26T16:38:22.009298+0000","last_clean":"2024-04-26T16:38:22.009298+0000","last_became_active":"2024-04-26T16:35:25.395438+0000","last_became_peered":"2024-04-26T16:35:25.395438+0000","last_unstale":"2024-04-26T16:38:22.009298+0000","last_undegraded":"2024-04-26T16:38:22.009298+0000","last_fullsized":"2024-04-26T16:38:22.009298+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:36:20.159412+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:36:20.159412+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"857'46","reported_seq":1754,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.011685+0000","last_change":"2024-04-26T16:38:12.118560+0000","last_active":"2024-04-26T16:38:22.011685+0000","last_peered":"2024-04-26T16:38:22.011685+0000","last_clean":"2024-04-26T16:38:22.011685+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:38:22.011685+0000","last_undegraded":"2024-04-26T16:38:22.011685+0000","last_fullsized":"2024-04-26T16:38:22.011685+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:38:12.118315+0000","last_deep_scrub":"458'34","last_deep_scrub_stamp":"2024-04-26T16:28:50.980879+0000","last_clean_scrub_stamp":"2024-04-26T16:38:12.118315+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.737219+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1692,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.016184+0000","last_change":"2024-04-26T16:38:16.438920+0000","last_active":"2024-04-26T16:38:22.016184+0000","last_peered":"2024-04-26T16:38:22.016184+0000","last_clean":"2024-04-26T16:38:22.016184+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:38:22.016184+0000","last_undegraded":"2024-04-26T16:38:22.016184+0000","last_fullsized":"2024-04-26T16:38:22.016184+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:16.438650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:16.438650+0000","last_clean_scrub_stamp":"2024-04-26T16:38:16.438650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.590311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":600,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995144+0000","last_change":"2024-04-26T16:36:29.464449+0000","last_active":"2024-04-26T16:38:21.995144+0000","last_peered":"2024-04-26T16:38:21.995144+0000","last_clean":"2024-04-26T16:38:21.995144+0000","last_became_active":"2024-04-26T16:35:25.382820+0000","last_became_peered":"2024-04-26T16:35:25.382820+0000","last_unstale":"2024-04-26T16:38:21.995144+0000","last_undegraded":"2024-04-26T16:38:21.995144+0000","last_fullsized":"2024-04-26T16:38:21.995144+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:29.464241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:29.464241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1522,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009301+0000","last_change":"2024-04-26T16:36:37.185673+0000","last_active":"2024-04-26T16:38:22.009301+0000","last_peered":"2024-04-26T16:38:22.009301+0000","last_clean":"2024-04-26T16:38:22.009301+0000","last_became_active":"2024-04-26T16:35:25.399977+0000","last_became_peered":"2024-04-26T16:35:25.399977+0000","last_unstale":"2024-04-26T16:38:22.009301+0000","last_undegraded":"2024-04-26T16:38:22.009301+0000","last_fullsized":"2024-04-26T16:38:22.009301+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:37.185574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:37.185574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"857'30","reported_seq":635,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:24.181817+0000","last_change":"2024-04-26T16:38:24.181898+0000","last_active":"2024-04-26T16:38:24.181817+0000","last_peered":"2024-04-26T16:38:24.181817+0000","last_clean":"2024-04-26T16:38:24.181817+0000","last_became_active":"2024-04-26T16:35:25.393291+0000","last_became_peered":"2024-04-26T16:35:25.393291+0000","last_unstale":"2024-04-26T16:38:24.181817+0000","last_undegraded":"2024-04-26T16:38:24.181817+0000","last_fullsized":"2024-04-26T16:38:24.181817+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:38:24.181658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:24.181658+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:40.363196+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1696,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:24.213688+0000","last_change":"2024-04-26T16:38:24.213749+0000","last_active":"2024-04-26T16:38:24.213688+0000","last_peered":"2024-04-26T16:38:24.213688+0000","last_clean":"2024-04-26T16:38:24.213688+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:38:24.213688+0000","last_undegraded":"2024-04-26T16:38:24.213688+0000","last_fullsized":"2024-04-26T16:38:24.213688+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:24.213509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:49.190081+0000","last_clean_scrub_stamp":"2024-04-26T16:38:24.213509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:49.129266+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1692,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:26.483849+0000","last_change":"2024-04-26T16:38:26.483871+0000","last_active":"2024-04-26T16:38:26.483849+0000","last_peered":"2024-04-26T16:38:26.483849+0000","last_clean":"2024-04-26T16:38:26.483849+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:38:26.483849+0000","last_undegraded":"2024-04-26T16:38:26.483849+0000","last_fullsized":"2024-04-26T16:38:26.483849+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:26.483786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:26.483786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:42.937895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1688,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:24.832345+0000","last_change":"2024-04-26T16:38:24.832406+0000","last_active":"2024-04-26T16:38:24.832345+0000","last_peered":"2024-04-26T16:38:24.832345+0000","last_clean":"2024-04-26T16:38:24.832345+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:38:24.832345+0000","last_undegraded":"2024-04-26T16:38:24.832345+0000","last_fullsized":"2024-04-26T16:38:24.832345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:24.832183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:26.632648+0000","last_clean_scrub_stamp":"2024-04-26T16:38:24.832183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:48.892299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1542,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010426+0000","last_change":"2024-04-26T16:36:35.211569+0000","last_active":"2024-04-26T16:38:22.010426+0000","last_peered":"2024-04-26T16:38:22.010426+0000","last_clean":"2024-04-26T16:38:22.010426+0000","last_became_active":"2024-04-26T16:35:25.385146+0000","last_became_peered":"2024-04-26T16:35:25.385146+0000","last_unstale":"2024-04-26T16:38:22.010426+0000","last_undegraded":"2024-04-26T16:38:22.010426+0000","last_fullsized":"2024-04-26T16:38:22.010426+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:35.211444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:35.211444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"857'53","reported_seq":1800,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010488+0000","last_change":"2024-04-26T16:36:42.185671+0000","last_active":"2024-04-26T16:38:22.010488+0000","last_peered":"2024-04-26T16:38:22.010488+0000","last_clean":"2024-04-26T16:38:22.010488+0000","last_became_active":"2024-04-26T16:35:25.387497+0000","last_became_peered":"2024-04-26T16:35:25.387497+0000","last_unstale":"2024-04-26T16:38:22.010488+0000","last_undegraded":"2024-04-26T16:38:22.010488+0000","last_fullsized":"2024-04-26T16:38:22.010488+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"612'39","last_scrub_stamp":"2024-04-26T16:36:42.185411+0000","last_deep_scrub":"612'39","last_deep_scrub_stamp":"2024-04-26T16:32:44.660567+0000","last_clean_scrub_stamp":"2024-04-26T16:36:42.185411+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1712,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014801+0000","last_change":"2024-04-26T16:38:10.316140+0000","last_active":"2024-04-26T16:38:22.014801+0000","last_peered":"2024-04-26T16:38:22.014801+0000","last_clean":"2024-04-26T16:38:22.014801+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:38:22.014801+0000","last_undegraded":"2024-04-26T16:38:22.014801+0000","last_fullsized":"2024-04-26T16:38:22.014801+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:10.315939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:10.315939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:30.665145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1704,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014987+0000","last_change":"2024-04-26T16:38:13.381270+0000","last_active":"2024-04-26T16:38:22.014987+0000","last_peered":"2024-04-26T16:38:22.014987+0000","last_clean":"2024-04-26T16:38:22.014987+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:38:22.014987+0000","last_undegraded":"2024-04-26T16:38:22.014987+0000","last_fullsized":"2024-04-26T16:38:22.014987+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:13.381046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:13.381046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:32.258914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1648,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.008121+0000","last_change":"2024-04-26T16:37:59.109107+0000","last_active":"2024-04-26T16:38:22.008121+0000","last_peered":"2024-04-26T16:38:22.008121+0000","last_clean":"2024-04-26T16:38:22.008121+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:38:22.008121+0000","last_undegraded":"2024-04-26T16:38:22.008121+0000","last_fullsized":"2024-04-26T16:38:22.008121+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:59.108820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.108820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.334820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1536,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014369+0000","last_change":"2024-04-26T16:36:53.546706+0000","last_active":"2024-04-26T16:38:22.014369+0000","last_peered":"2024-04-26T16:38:22.014369+0000","last_clean":"2024-04-26T16:38:22.014369+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:38:22.014369+0000","last_undegraded":"2024-04-26T16:38:22.014369+0000","last_fullsized":"2024-04-26T16:38:22.014369+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:53.546496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:24.770536+0000","last_clean_scrub_stamp":"2024-04-26T16:36:53.546496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"857'51","reported_seq":1435,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.128093+0000","last_change":"2024-04-26T16:37:48.340319+0000","last_active":"2024-04-26T16:38:22.128093+0000","last_peered":"2024-04-26T16:38:22.128093+0000","last_clean":"2024-04-26T16:38:22.128093+0000","last_became_active":"2024-04-26T16:35:25.414489+0000","last_became_peered":"2024-04-26T16:35:25.414489+0000","last_unstale":"2024-04-26T16:38:22.128093+0000","last_undegraded":"2024-04-26T16:38:22.128093+0000","last_fullsized":"2024-04-26T16:38:22.128093+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'51","last_scrub_stamp":"2024-04-26T16:37:48.340220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.340220+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:54.505895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1672,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009439+0000","last_change":"2024-04-26T16:36:50.250362+0000","last_active":"2024-04-26T16:38:22.009439+0000","last_peered":"2024-04-26T16:38:22.009439+0000","last_clean":"2024-04-26T16:38:22.009439+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:38:22.009439+0000","last_undegraded":"2024-04-26T16:38:22.009439+0000","last_fullsized":"2024-04-26T16:38:22.009439+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:50.250252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:36.297212+0000","last_clean_scrub_stamp":"2024-04-26T16:36:50.250252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1696,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436177+0000","last_change":"2024-04-26T16:38:20.157200+0000","last_active":"2024-04-26T16:38:21.436177+0000","last_peered":"2024-04-26T16:38:21.436177+0000","last_clean":"2024-04-26T16:38:21.436177+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:38:21.436177+0000","last_undegraded":"2024-04-26T16:38:21.436177+0000","last_fullsized":"2024-04-26T16:38:21.436177+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:20.156962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:20.156962+0000","last_clean_scrub_stamp":"2024-04-26T16:38:20.156962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:42.940449+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1680,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.016348+0000","last_change":"2024-04-26T16:36:48.691806+0000","last_active":"2024-04-26T16:38:22.016348+0000","last_peered":"2024-04-26T16:38:22.016348+0000","last_clean":"2024-04-26T16:38:22.016348+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:38:22.016348+0000","last_undegraded":"2024-04-26T16:38:22.016348+0000","last_fullsized":"2024-04-26T16:38:22.016348+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.691439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:16.826623+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.691439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1159,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129382+0000","last_change":"2024-04-26T16:37:50.334215+0000","last_active":"2024-04-26T16:38:22.129382+0000","last_peered":"2024-04-26T16:38:22.129382+0000","last_clean":"2024-04-26T16:38:22.129382+0000","last_became_active":"2024-04-26T16:35:25.388195+0000","last_became_peered":"2024-04-26T16:35:25.388195+0000","last_unstale":"2024-04-26T16:38:22.129382+0000","last_undegraded":"2024-04-26T16:38:22.129382+0000","last_fullsized":"2024-04-26T16:38:22.129382+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:50.333993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:50.333993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.406018+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"857'25","reported_seq":1710,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436448+0000","last_change":"2024-04-26T16:38:20.573911+0000","last_active":"2024-04-26T16:38:21.436448+0000","last_peered":"2024-04-26T16:38:21.436448+0000","last_clean":"2024-04-26T16:38:21.436448+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:38:21.436448+0000","last_undegraded":"2024-04-26T16:38:21.436448+0000","last_fullsized":"2024-04-26T16:38:21.436448+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:38:20.573631+0000","last_deep_scrub":"857'25","last_deep_scrub_stamp":"2024-04-26T16:38:20.573631+0000","last_clean_scrub_stamp":"2024-04-26T16:38:20.573631+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:32.838616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1696,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436589+0000","last_change":"2024-04-26T16:38:08.143051+0000","last_active":"2024-04-26T16:38:21.436589+0000","last_peered":"2024-04-26T16:38:21.436589+0000","last_clean":"2024-04-26T16:38:21.436589+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:38:21.436589+0000","last_undegraded":"2024-04-26T16:38:21.436589+0000","last_fullsized":"2024-04-26T16:38:21.436589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:08.142728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:08.142728+0000","last_clean_scrub_stamp":"2024-04-26T16:38:08.142728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.038547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":1974,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014827+0000","last_change":"2024-04-26T16:38:00.436674+0000","last_active":"2024-04-26T16:38:22.014827+0000","last_peered":"2024-04-26T16:38:22.014827+0000","last_clean":"2024-04-26T16:38:22.014827+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:38:22.014827+0000","last_undegraded":"2024-04-26T16:38:22.014827+0000","last_fullsized":"2024-04-26T16:38:22.014827+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:38:00.436448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.436448+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:28.478392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1688,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437065+0000","last_change":"2024-04-26T16:37:56.784808+0000","last_active":"2024-04-26T16:38:21.437065+0000","last_peered":"2024-04-26T16:38:21.437065+0000","last_clean":"2024-04-26T16:38:21.437065+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:38:21.437065+0000","last_undegraded":"2024-04-26T16:38:21.437065+0000","last_fullsized":"2024-04-26T16:38:21.437065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.784687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.893722+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.784687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.270962+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1538,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435048+0000","last_change":"2024-04-26T16:37:56.159369+0000","last_active":"2024-04-26T16:38:21.435048+0000","last_peered":"2024-04-26T16:38:21.435048+0000","last_clean":"2024-04-26T16:38:21.435048+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:38:21.435048+0000","last_undegraded":"2024-04-26T16:38:21.435048+0000","last_fullsized":"2024-04-26T16:38:21.435048+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.159158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.159158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.823331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"857'46","reported_seq":1740,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.434758+0000","last_change":"2024-04-26T16:38:19.568361+0000","last_active":"2024-04-26T16:38:21.434758+0000","last_peered":"2024-04-26T16:38:21.434758+0000","last_clean":"2024-04-26T16:38:21.434758+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:38:21.434758+0000","last_undegraded":"2024-04-26T16:38:21.434758+0000","last_fullsized":"2024-04-26T16:38:21.434758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:38:19.568185+0000","last_deep_scrub":"857'46","last_deep_scrub_stamp":"2024-04-26T16:38:19.568185+0000","last_clean_scrub_stamp":"2024-04-26T16:38:19.568185+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:41.130996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1710,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.130146+0000","last_change":"2024-04-26T16:38:22.130209+0000","last_active":"2024-04-26T16:38:22.130146+0000","last_peered":"2024-04-26T16:38:22.130146+0000","last_clean":"2024-04-26T16:38:22.130146+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:38:22.130146+0000","last_undegraded":"2024-04-26T16:38:22.130146+0000","last_fullsized":"2024-04-26T16:38:22.130146+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:22.129990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:38:22.129990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:35.702822+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":1967,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436890+0000","last_change":"2024-04-26T16:37:47.846639+0000","last_active":"2024-04-26T16:38:21.436890+0000","last_peered":"2024-04-26T16:38:21.436890+0000","last_clean":"2024-04-26T16:38:21.436890+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:38:21.436890+0000","last_undegraded":"2024-04-26T16:38:21.436890+0000","last_fullsized":"2024-04-26T16:38:21.436890+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:37:47.846487+0000","last_deep_scrub":"115'12","last_deep_scrub_stamp":"2024-04-26T16:21:46.578476+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.846487+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:51.693727+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996683+0000","last_change":"2024-04-26T16:38:18.223540+0000","last_active":"2024-04-26T16:38:21.996683+0000","last_peered":"2024-04-26T16:38:21.996683+0000","last_clean":"2024-04-26T16:38:21.996683+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:38:21.996683+0000","last_undegraded":"2024-04-26T16:38:21.996683+0000","last_fullsized":"2024-04-26T16:38:21.996683+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:18.223226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:24.556379+0000","last_clean_scrub_stamp":"2024-04-26T16:38:18.223226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:42.733270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":1180,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.128439+0000","last_change":"2024-04-26T16:38:13.304318+0000","last_active":"2024-04-26T16:38:22.128439+0000","last_peered":"2024-04-26T16:38:22.128439+0000","last_clean":"2024-04-26T16:38:22.128439+0000","last_became_active":"2024-04-26T16:35:25.407637+0000","last_became_peered":"2024-04-26T16:35:25.407637+0000","last_unstale":"2024-04-26T16:38:22.128439+0000","last_undegraded":"2024-04-26T16:38:22.128439+0000","last_fullsized":"2024-04-26T16:38:22.128439+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:38:13.304185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:13.304185+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:15.663444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"856'29","reported_seq":1734,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435670+0000","last_change":"2024-04-26T16:38:09.706190+0000","last_active":"2024-04-26T16:38:21.435670+0000","last_peered":"2024-04-26T16:38:21.435670+0000","last_clean":"2024-04-26T16:38:21.435670+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:38:21.435670+0000","last_undegraded":"2024-04-26T16:38:21.435670+0000","last_fullsized":"2024-04-26T16:38:21.435670+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'29","last_scrub_stamp":"2024-04-26T16:38:09.705960+0000","last_deep_scrub":"824'23","last_deep_scrub_stamp":"2024-04-26T16:36:53.947636+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.705960+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:37.529754+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1676,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.008544+0000","last_change":"2024-04-26T16:38:03.141298+0000","last_active":"2024-04-26T16:38:22.008544+0000","last_peered":"2024-04-26T16:38:22.008544+0000","last_clean":"2024-04-26T16:38:22.008544+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:38:22.008544+0000","last_undegraded":"2024-04-26T16:38:22.008544+0000","last_fullsized":"2024-04-26T16:38:22.008544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:38:03.140926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.140926+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.283756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":608,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.240445+0000","last_change":"2024-04-26T16:38:22.240503+0000","last_active":"2024-04-26T16:38:22.240445+0000","last_peered":"2024-04-26T16:38:22.240445+0000","last_clean":"2024-04-26T16:38:22.240445+0000","last_became_active":"2024-04-26T16:35:25.382698+0000","last_became_peered":"2024-04-26T16:35:25.382698+0000","last_unstale":"2024-04-26T16:38:22.240445+0000","last_undegraded":"2024-04-26T16:38:22.240445+0000","last_fullsized":"2024-04-26T16:38:22.240445+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:22.240291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:28.439524+0000","last_clean_scrub_stamp":"2024-04-26T16:38:22.240291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:46.058115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1678,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436705+0000","last_change":"2024-04-26T16:37:54.772260+0000","last_active":"2024-04-26T16:38:21.436705+0000","last_peered":"2024-04-26T16:38:21.436705+0000","last_clean":"2024-04-26T16:38:21.436705+0000","last_became_active":"2024-04-26T16:35:25.416938+0000","last_became_peered":"2024-04-26T16:35:25.416938+0000","last_unstale":"2024-04-26T16:38:21.436705+0000","last_undegraded":"2024-04-26T16:38:21.436705+0000","last_fullsized":"2024-04-26T16:38:21.436705+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:54.772049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:54.772049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.353649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1541,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436947+0000","last_change":"2024-04-26T16:37:58.725326+0000","last_active":"2024-04-26T16:38:21.436947+0000","last_peered":"2024-04-26T16:38:21.436947+0000","last_clean":"2024-04-26T16:38:21.436947+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:38:21.436947+0000","last_undegraded":"2024-04-26T16:38:21.436947+0000","last_fullsized":"2024-04-26T16:38:21.436947+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.725080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.725080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.633294+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":1924,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.434929+0000","last_change":"2024-04-26T16:37:47.750495+0000","last_active":"2024-04-26T16:38:21.434929+0000","last_peered":"2024-04-26T16:38:21.434929+0000","last_clean":"2024-04-26T16:38:21.434929+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:38:21.434929+0000","last_undegraded":"2024-04-26T16:38:21.434929+0000","last_fullsized":"2024-04-26T16:38:21.434929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:37:47.750268+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.750268+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.703331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1716,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435754+0000","last_change":"2024-04-26T16:37:52.815375+0000","last_active":"2024-04-26T16:38:21.435754+0000","last_peered":"2024-04-26T16:38:21.435754+0000","last_clean":"2024-04-26T16:38:21.435754+0000","last_became_active":"2024-04-26T16:35:25.411443+0000","last_became_peered":"2024-04-26T16:35:25.411443+0000","last_unstale":"2024-04-26T16:38:21.435754+0000","last_undegraded":"2024-04-26T16:38:21.435754+0000","last_fullsized":"2024-04-26T16:38:21.435754+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.815039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.815039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:12.392298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"857'42","reported_seq":1748,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.915376+0000","last_change":"2024-04-26T16:38:05.806435+0000","last_active":"2024-04-26T16:38:21.915376+0000","last_peered":"2024-04-26T16:38:21.915376+0000","last_clean":"2024-04-26T16:38:21.915376+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:38:21.915376+0000","last_undegraded":"2024-04-26T16:38:21.915376+0000","last_fullsized":"2024-04-26T16:38:21.915376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'42","last_scrub_stamp":"2024-04-26T16:38:05.806187+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.806187+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:34.319055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1283,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.130494+0000","last_change":"2024-04-26T16:37:06.357605+0000","last_active":"2024-04-26T16:38:22.130494+0000","last_peered":"2024-04-26T16:38:22.130494+0000","last_clean":"2024-04-26T16:38:22.130494+0000","last_became_active":"2024-04-26T16:35:25.406996+0000","last_became_peered":"2024-04-26T16:35:25.406996+0000","last_unstale":"2024-04-26T16:38:22.130494+0000","last_undegraded":"2024-04-26T16:38:22.130494+0000","last_fullsized":"2024-04-26T16:38:22.130494+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:06.357237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:58.917484+0000","last_clean_scrub_stamp":"2024-04-26T16:37:06.357237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:29.140874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1667,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009237+0000","last_change":"2024-04-26T16:36:39.157157+0000","last_active":"2024-04-26T16:38:22.009237+0000","last_peered":"2024-04-26T16:38:22.009237+0000","last_clean":"2024-04-26T16:38:22.009237+0000","last_became_active":"2024-04-26T16:35:25.391371+0000","last_became_peered":"2024-04-26T16:35:25.391371+0000","last_unstale":"2024-04-26T16:38:22.009237+0000","last_undegraded":"2024-04-26T16:38:22.009237+0000","last_fullsized":"2024-04-26T16:38:22.009237+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:39.156939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:39.156939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1511,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.011988+0000","last_change":"2024-04-26T16:37:01.201260+0000","last_active":"2024-04-26T16:38:22.011988+0000","last_peered":"2024-04-26T16:38:22.011988+0000","last_clean":"2024-04-26T16:38:22.011988+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:38:22.011988+0000","last_undegraded":"2024-04-26T16:38:22.011988+0000","last_fullsized":"2024-04-26T16:38:22.011988+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.200920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.200920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.711340+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1693,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435627+0000","last_change":"2024-04-26T16:38:06.771968+0000","last_active":"2024-04-26T16:38:21.435627+0000","last_peered":"2024-04-26T16:38:21.435627+0000","last_clean":"2024-04-26T16:38:21.435627+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:38:21.435627+0000","last_undegraded":"2024-04-26T16:38:21.435627+0000","last_fullsized":"2024-04-26T16:38:21.435627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:06.771754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:38:06.771754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.556312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":2907,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:24.809172+0000","last_change":"2024-04-26T16:37:01.745491+0000","last_active":"2024-04-26T16:38:24.809172+0000","last_peered":"2024-04-26T16:38:24.809172+0000","last_clean":"2024-04-26T16:38:24.809172+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:38:24.809172+0000","last_undegraded":"2024-04-26T16:38:24.809172+0000","last_fullsized":"2024-04-26T16:38:24.809172+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:37:01.745268+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.745268+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.918491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1234,"num_read_kb":1229,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"857'26","reported_seq":1708,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437368+0000","last_change":"2024-04-26T16:37:03.724692+0000","last_active":"2024-04-26T16:38:21.437368+0000","last_peered":"2024-04-26T16:38:21.437368+0000","last_clean":"2024-04-26T16:38:21.437368+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:38:21.437368+0000","last_undegraded":"2024-04-26T16:38:21.437368+0000","last_fullsized":"2024-04-26T16:38:21.437368+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'22","last_scrub_stamp":"2024-04-26T16:37:03.724436+0000","last_deep_scrub":"121'18","last_deep_scrub_stamp":"2024-04-26T16:21:15.047780+0000","last_clean_scrub_stamp":"2024-04-26T16:37:03.724436+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:28.814780+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1708,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435501+0000","last_change":"2024-04-26T16:38:04.165569+0000","last_active":"2024-04-26T16:38:21.435501+0000","last_peered":"2024-04-26T16:38:21.435501+0000","last_clean":"2024-04-26T16:38:21.435501+0000","last_became_active":"2024-04-26T16:35:25.398461+0000","last_became_peered":"2024-04-26T16:35:25.398461+0000","last_unstale":"2024-04-26T16:38:21.435501+0000","last_undegraded":"2024-04-26T16:38:21.435501+0000","last_fullsized":"2024-04-26T16:38:21.435501+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:38:04.165359+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:20:18.047049+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.165359+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.874714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1689,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.821640+0000","last_change":"2024-04-26T16:38:25.821640+0000","last_active":"2024-04-26T16:38:25.821640+0000","last_peered":"2024-04-26T16:38:25.821640+0000","last_clean":"2024-04-26T16:38:25.821640+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:38:25.821640+0000","last_undegraded":"2024-04-26T16:38:25.821640+0000","last_fullsized":"2024-04-26T16:38:25.821640+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:25.821464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:38:25.821464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:28.125673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1537,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435755+0000","last_change":"2024-04-26T16:38:10.136556+0000","last_active":"2024-04-26T16:38:21.435755+0000","last_peered":"2024-04-26T16:38:21.435755+0000","last_clean":"2024-04-26T16:38:21.435755+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:38:21.435755+0000","last_undegraded":"2024-04-26T16:38:21.435755+0000","last_fullsized":"2024-04-26T16:38:21.435755+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:10.136352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:10.136352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.456445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1277,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.128634+0000","last_change":"2024-04-26T16:38:02.229476+0000","last_active":"2024-04-26T16:38:22.128634+0000","last_peered":"2024-04-26T16:38:22.128634+0000","last_clean":"2024-04-26T16:38:22.128634+0000","last_became_active":"2024-04-26T16:35:25.404395+0000","last_became_peered":"2024-04-26T16:35:25.404395+0000","last_unstale":"2024-04-26T16:38:22.128634+0000","last_undegraded":"2024-04-26T16:38:22.128634+0000","last_fullsized":"2024-04-26T16:38:22.128634+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.229233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.229233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.152859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"857'66","reported_seq":2097,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.052495+0000","last_change":"2024-04-26T16:38:03.160319+0000","last_active":"2024-04-26T16:38:25.052495+0000","last_peered":"2024-04-26T16:38:25.052495+0000","last_clean":"2024-04-26T16:38:25.052495+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:38:25.052495+0000","last_undegraded":"2024-04-26T16:38:25.052495+0000","last_fullsized":"2024-04-26T16:38:25.052495+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'66","last_scrub_stamp":"2024-04-26T16:38:03.160063+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.160063+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.536394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1670,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010271+0000","last_change":"2024-04-26T16:35:08.450261+0000","last_active":"2024-04-26T16:38:22.010271+0000","last_peered":"2024-04-26T16:38:22.010271+0000","last_clean":"2024-04-26T16:38:22.010271+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:38:22.010271+0000","last_undegraded":"2024-04-26T16:38:22.010271+0000","last_fullsized":"2024-04-26T16:38:22.010271+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:35:08.450018+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:29:03.080403+0000","last_clean_scrub_stamp":"2024-04-26T16:35:08.450018+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1695,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435359+0000","last_change":"2024-04-26T16:38:02.125681+0000","last_active":"2024-04-26T16:38:21.435359+0000","last_peered":"2024-04-26T16:38:21.435359+0000","last_clean":"2024-04-26T16:38:21.435359+0000","last_became_active":"2024-04-26T16:35:25.411888+0000","last_became_peered":"2024-04-26T16:35:25.411888+0000","last_unstale":"2024-04-26T16:38:21.435359+0000","last_undegraded":"2024-04-26T16:38:21.435359+0000","last_fullsized":"2024-04-26T16:38:21.435359+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.125415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.125415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.366125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1521,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435287+0000","last_change":"2024-04-26T16:37:57.743728+0000","last_active":"2024-04-26T16:38:21.435287+0000","last_peered":"2024-04-26T16:38:21.435287+0000","last_clean":"2024-04-26T16:38:21.435287+0000","last_became_active":"2024-04-26T16:35:25.417017+0000","last_became_peered":"2024-04-26T16:35:25.417017+0000","last_unstale":"2024-04-26T16:38:21.435287+0000","last_undegraded":"2024-04-26T16:38:21.435287+0000","last_fullsized":"2024-04-26T16:38:21.435287+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:57.743476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.743476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.056336+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996439+0000","last_change":"2024-04-26T16:37:58.133197+0000","last_active":"2024-04-26T16:38:21.996439+0000","last_peered":"2024-04-26T16:38:21.996439+0000","last_clean":"2024-04-26T16:38:21.996439+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:38:21.996439+0000","last_undegraded":"2024-04-26T16:38:21.996439+0000","last_fullsized":"2024-04-26T16:38:21.996439+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.132987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.132987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:00.348823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1669,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:26.045970+0000","last_change":"2024-04-26T16:38:26.046010+0000","last_active":"2024-04-26T16:38:26.045970+0000","last_peered":"2024-04-26T16:38:26.045970+0000","last_clean":"2024-04-26T16:38:26.045970+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:38:26.045970+0000","last_undegraded":"2024-04-26T16:38:26.045970+0000","last_fullsized":"2024-04-26T16:38:26.045970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:26.045883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:26.045883+0000","last_clean_scrub_stamp":"2024-04-26T16:38:26.045883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:42.125034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"857'49","reported_seq":1751,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.913957+0000","last_change":"2024-04-26T16:36:58.920449+0000","last_active":"2024-04-26T16:38:21.913957+0000","last_peered":"2024-04-26T16:38:21.913957+0000","last_clean":"2024-04-26T16:38:21.913957+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:38:21.913957+0000","last_undegraded":"2024-04-26T16:38:21.913957+0000","last_fullsized":"2024-04-26T16:38:21.913957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'35","last_scrub_stamp":"2024-04-26T16:36:58.920294+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:36:58.920294+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1697,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436272+0000","last_change":"2024-04-26T16:37:05.007705+0000","last_active":"2024-04-26T16:38:21.436272+0000","last_peered":"2024-04-26T16:38:21.436272+0000","last_clean":"2024-04-26T16:38:21.436272+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:38:21.436272+0000","last_undegraded":"2024-04-26T16:38:21.436272+0000","last_fullsized":"2024-04-26T16:38:21.436272+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:05.007579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.022618+0000","last_clean_scrub_stamp":"2024-04-26T16:37:05.007579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.021319+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1541,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435889+0000","last_change":"2024-04-26T16:38:05.712753+0000","last_active":"2024-04-26T16:38:21.435889+0000","last_peered":"2024-04-26T16:38:21.435889+0000","last_clean":"2024-04-26T16:38:21.435889+0000","last_became_active":"2024-04-26T16:35:25.417111+0000","last_became_peered":"2024-04-26T16:35:25.417111+0000","last_unstale":"2024-04-26T16:38:21.435889+0000","last_undegraded":"2024-04-26T16:38:21.435889+0000","last_fullsized":"2024-04-26T16:38:21.435889+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:05.712520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.712520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.895535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1673,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.004203+0000","last_change":"2024-04-26T16:38:25.004203+0000","last_active":"2024-04-26T16:38:25.004203+0000","last_peered":"2024-04-26T16:38:25.004203+0000","last_clean":"2024-04-26T16:38:25.004203+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:38:25.004203+0000","last_undegraded":"2024-04-26T16:38:25.004203+0000","last_fullsized":"2024-04-26T16:38:25.004203+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:25.003899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:52.080653+0000","last_clean_scrub_stamp":"2024-04-26T16:38:25.003899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:47.092054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":1959,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.008362+0000","last_change":"2024-04-26T16:36:25.188281+0000","last_active":"2024-04-26T16:38:22.008362+0000","last_peered":"2024-04-26T16:38:22.008362+0000","last_clean":"2024-04-26T16:38:22.008362+0000","last_became_active":"2024-04-26T16:35:25.394870+0000","last_became_peered":"2024-04-26T16:35:25.394870+0000","last_unstale":"2024-04-26T16:38:22.008362+0000","last_undegraded":"2024-04-26T16:38:22.008362+0000","last_fullsized":"2024-04-26T16:38:22.008362+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:36:25.187951+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:36:25.187951+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":608,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995289+0000","last_change":"2024-04-26T16:37:57.118532+0000","last_active":"2024-04-26T16:38:21.995289+0000","last_peered":"2024-04-26T16:38:21.995289+0000","last_clean":"2024-04-26T16:38:21.995289+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:38:21.995289+0000","last_undegraded":"2024-04-26T16:38:21.995289+0000","last_fullsized":"2024-04-26T16:38:21.995289+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:57.118297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.118297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.113285+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"857'35","reported_seq":1769,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436017+0000","last_change":"2024-04-26T16:38:11.656964+0000","last_active":"2024-04-26T16:38:21.436017+0000","last_peered":"2024-04-26T16:38:21.436017+0000","last_clean":"2024-04-26T16:38:21.436017+0000","last_became_active":"2024-04-26T16:35:25.397667+0000","last_became_peered":"2024-04-26T16:35:25.397667+0000","last_unstale":"2024-04-26T16:38:21.436017+0000","last_undegraded":"2024-04-26T16:38:21.436017+0000","last_fullsized":"2024-04-26T16:38:21.436017+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'35","last_scrub_stamp":"2024-04-26T16:38:11.656588+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:38:11.656588+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.518037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996706+0000","last_change":"2024-04-26T16:37:02.918875+0000","last_active":"2024-04-26T16:38:21.996706+0000","last_peered":"2024-04-26T16:38:21.996706+0000","last_clean":"2024-04-26T16:38:21.996706+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:38:21.996706+0000","last_undegraded":"2024-04-26T16:38:21.996706+0000","last_fullsized":"2024-04-26T16:38:21.996706+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:02.918675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:37:02.918675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1541,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.434595+0000","last_change":"2024-04-26T16:37:42.746944+0000","last_active":"2024-04-26T16:38:21.434595+0000","last_peered":"2024-04-26T16:38:21.434595+0000","last_clean":"2024-04-26T16:38:21.434595+0000","last_became_active":"2024-04-26T16:35:25.392163+0000","last_became_peered":"2024-04-26T16:35:25.392163+0000","last_unstale":"2024-04-26T16:38:21.434595+0000","last_undegraded":"2024-04-26T16:38:21.434595+0000","last_fullsized":"2024-04-26T16:38:21.434595+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:42.746793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:42.746793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.906091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1704,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.016580+0000","last_change":"2024-04-26T16:38:17.489033+0000","last_active":"2024-04-26T16:38:22.016580+0000","last_peered":"2024-04-26T16:38:22.016580+0000","last_clean":"2024-04-26T16:38:22.016580+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:38:22.016580+0000","last_undegraded":"2024-04-26T16:38:22.016580+0000","last_fullsized":"2024-04-26T16:38:22.016580+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:17.488695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:38:17.488695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:30.166458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1701,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.013387+0000","last_change":"2024-04-26T16:38:11.352575+0000","last_active":"2024-04-26T16:38:22.013387+0000","last_peered":"2024-04-26T16:38:22.013387+0000","last_clean":"2024-04-26T16:38:22.013387+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:38:22.013387+0000","last_undegraded":"2024-04-26T16:38:22.013387+0000","last_fullsized":"2024-04-26T16:38:22.013387+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:11.352344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:39.561528+0000","last_clean_scrub_stamp":"2024-04-26T16:38:11.352344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:31.815779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"857'50","reported_seq":1787,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.434979+0000","last_change":"2024-04-26T16:37:54.196331+0000","last_active":"2024-04-26T16:38:21.434979+0000","last_peered":"2024-04-26T16:38:21.434979+0000","last_clean":"2024-04-26T16:38:21.434979+0000","last_became_active":"2024-04-26T16:35:25.412117+0000","last_became_peered":"2024-04-26T16:35:25.412117+0000","last_unstale":"2024-04-26T16:38:21.434979+0000","last_undegraded":"2024-04-26T16:38:21.434979+0000","last_fullsized":"2024-04-26T16:38:21.434979+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'50","last_scrub_stamp":"2024-04-26T16:37:54.196211+0000","last_deep_scrub":"388'38","last_deep_scrub_stamp":"2024-04-26T16:32:43.426520+0000","last_clean_scrub_stamp":"2024-04-26T16:37:54.196211+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.629419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1658,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010200+0000","last_change":"2024-04-26T16:36:13.193711+0000","last_active":"2024-04-26T16:38:22.010200+0000","last_peered":"2024-04-26T16:38:22.010200+0000","last_clean":"2024-04-26T16:38:22.010200+0000","last_became_active":"2024-04-26T16:35:25.398515+0000","last_became_peered":"2024-04-26T16:35:25.398515+0000","last_unstale":"2024-04-26T16:38:22.010200+0000","last_undegraded":"2024-04-26T16:38:22.010200+0000","last_fullsized":"2024-04-26T16:38:22.010200+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:13.193610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:13.193610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1519,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435910+0000","last_change":"2024-04-26T16:38:12.645178+0000","last_active":"2024-04-26T16:38:21.435910+0000","last_peered":"2024-04-26T16:38:21.435910+0000","last_clean":"2024-04-26T16:38:21.435910+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:38:21.435910+0000","last_undegraded":"2024-04-26T16:38:21.435910+0000","last_fullsized":"2024-04-26T16:38:21.435910+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:12.645018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:12.645018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:42.576598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1677,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.916027+0000","last_change":"2024-04-26T16:36:48.923637+0000","last_active":"2024-04-26T16:38:21.916027+0000","last_peered":"2024-04-26T16:38:21.916027+0000","last_clean":"2024-04-26T16:38:21.916027+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:38:21.916027+0000","last_undegraded":"2024-04-26T16:38:21.916027+0000","last_fullsized":"2024-04-26T16:38:21.916027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:48.923527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:48.923527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"857'47","reported_seq":1773,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436460+0000","last_change":"2024-04-26T16:37:55.775734+0000","last_active":"2024-04-26T16:38:21.436460+0000","last_peered":"2024-04-26T16:38:21.436460+0000","last_clean":"2024-04-26T16:38:21.436460+0000","last_became_active":"2024-04-26T16:35:25.395746+0000","last_became_peered":"2024-04-26T16:35:25.395746+0000","last_unstale":"2024-04-26T16:38:21.436460+0000","last_undegraded":"2024-04-26T16:38:21.436460+0000","last_fullsized":"2024-04-26T16:38:21.436460+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'47","last_scrub_stamp":"2024-04-26T16:37:55.775586+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:37:55.775586+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.155493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":601,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995352+0000","last_change":"2024-04-26T16:36:32.514466+0000","last_active":"2024-04-26T16:38:21.995352+0000","last_peered":"2024-04-26T16:38:21.995352+0000","last_clean":"2024-04-26T16:38:21.995352+0000","last_became_active":"2024-04-26T16:35:25.397940+0000","last_became_peered":"2024-04-26T16:35:25.397940+0000","last_unstale":"2024-04-26T16:38:21.995352+0000","last_undegraded":"2024-04-26T16:38:21.995352+0000","last_fullsized":"2024-04-26T16:38:21.995352+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:32.514129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:36:32.514129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1664,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437163+0000","last_change":"2024-04-26T16:37:46.742109+0000","last_active":"2024-04-26T16:38:21.437163+0000","last_peered":"2024-04-26T16:38:21.437163+0000","last_clean":"2024-04-26T16:38:21.437163+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:38:21.437163+0000","last_undegraded":"2024-04-26T16:38:21.437163+0000","last_fullsized":"2024-04-26T16:38:21.437163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:46.742012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:46.742012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.964801+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1537,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.015584+0000","last_change":"2024-04-26T16:36:59.476993+0000","last_active":"2024-04-26T16:38:22.015584+0000","last_peered":"2024-04-26T16:38:22.015584+0000","last_clean":"2024-04-26T16:38:22.015584+0000","last_became_active":"2024-04-26T16:35:25.390196+0000","last_became_peered":"2024-04-26T16:35:25.390196+0000","last_unstale":"2024-04-26T16:38:22.015584+0000","last_undegraded":"2024-04-26T16:38:22.015584+0000","last_fullsized":"2024-04-26T16:38:22.015584+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:59.476833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.476833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"747'19","reported_seq":1978,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.052496+0000","last_change":"2024-04-26T16:37:57.836348+0000","last_active":"2024-04-26T16:38:25.052496+0000","last_peered":"2024-04-26T16:38:25.052496+0000","last_clean":"2024-04-26T16:38:25.052496+0000","last_became_active":"2024-04-26T16:35:25.404413+0000","last_became_peered":"2024-04-26T16:35:25.404413+0000","last_unstale":"2024-04-26T16:38:25.052496+0000","last_undegraded":"2024-04-26T16:38:25.052496+0000","last_fullsized":"2024-04-26T16:38:25.052496+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'19","last_scrub_stamp":"2024-04-26T16:37:57.836074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:57.836074+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:59.025050+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"89'2","reported_seq":1720,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436910+0000","last_change":"2024-04-26T16:37:48.275884+0000","last_active":"2024-04-26T16:38:21.436910+0000","last_peered":"2024-04-26T16:38:21.436910+0000","last_clean":"2024-04-26T16:38:21.436910+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:38:21.436910+0000","last_undegraded":"2024-04-26T16:38:21.436910+0000","last_fullsized":"2024-04-26T16:38:21.436910+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-04-26T16:37:48.275781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:30.993608+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.275781+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:52.249427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"857'52","reported_seq":720,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.053200+0000","last_change":"2024-04-26T16:38:17.232552+0000","last_active":"2024-04-26T16:38:25.053200+0000","last_peered":"2024-04-26T16:38:25.053200+0000","last_clean":"2024-04-26T16:38:25.053200+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:38:25.053200+0000","last_undegraded":"2024-04-26T16:38:25.053200+0000","last_fullsized":"2024-04-26T16:38:25.053200+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'52","last_scrub_stamp":"2024-04-26T16:38:17.232228+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:38:17.232228+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:36.020038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1705,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.223251+0000","last_change":"2024-04-26T16:38:25.223251+0000","last_active":"2024-04-26T16:38:25.223251+0000","last_peered":"2024-04-26T16:38:25.223251+0000","last_clean":"2024-04-26T16:38:25.223251+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:38:25.223251+0000","last_undegraded":"2024-04-26T16:38:25.223251+0000","last_fullsized":"2024-04-26T16:38:25.223251+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:25.222899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:38:25.222899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.153515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1673,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.913285+0000","last_change":"2024-04-26T16:38:01.824470+0000","last_active":"2024-04-26T16:38:21.913285+0000","last_peered":"2024-04-26T16:38:21.913285+0000","last_clean":"2024-04-26T16:38:21.913285+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:38:21.913285+0000","last_undegraded":"2024-04-26T16:38:21.913285+0000","last_fullsized":"2024-04-26T16:38:21.913285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:01.824258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:01.824258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:18.185258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1535,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:23.833428+0000","last_change":"2024-04-26T16:38:23.833487+0000","last_active":"2024-04-26T16:38:23.833428+0000","last_peered":"2024-04-26T16:38:23.833428+0000","last_clean":"2024-04-26T16:38:23.833428+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:38:23.833428+0000","last_undegraded":"2024-04-26T16:38:23.833428+0000","last_fullsized":"2024-04-26T16:38:23.833428+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:23.833246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:23.833246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:46.920936+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1684,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.013577+0000","last_change":"2024-04-26T16:36:46.698276+0000","last_active":"2024-04-26T16:38:22.013577+0000","last_peered":"2024-04-26T16:38:22.013577+0000","last_clean":"2024-04-26T16:38:22.013577+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:38:22.013577+0000","last_undegraded":"2024-04-26T16:38:22.013577+0000","last_fullsized":"2024-04-26T16:38:22.013577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.698075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.698075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"857'27","reported_seq":1710,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010041+0000","last_change":"2024-04-26T16:37:56.108530+0000","last_active":"2024-04-26T16:38:22.010041+0000","last_peered":"2024-04-26T16:38:22.010041+0000","last_clean":"2024-04-26T16:38:22.010041+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:38:22.010041+0000","last_undegraded":"2024-04-26T16:38:22.010041+0000","last_fullsized":"2024-04-26T16:38:22.010041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:37:56.108290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.108290+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.284265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1716,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436766+0000","last_change":"2024-04-26T16:36:59.986857+0000","last_active":"2024-04-26T16:38:21.436766+0000","last_peered":"2024-04-26T16:38:21.436766+0000","last_clean":"2024-04-26T16:38:21.436766+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:38:21.436766+0000","last_undegraded":"2024-04-26T16:38:21.436766+0000","last_fullsized":"2024-04-26T16:38:21.436766+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:36:59.986744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.986744+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.282717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1656,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436628+0000","last_change":"2024-04-26T16:37:48.766181+0000","last_active":"2024-04-26T16:38:21.436628+0000","last_peered":"2024-04-26T16:38:21.436628+0000","last_clean":"2024-04-26T16:38:21.436628+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:38:21.436628+0000","last_undegraded":"2024-04-26T16:38:21.436628+0000","last_fullsized":"2024-04-26T16:38:21.436628+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:48.765876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.420943+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.765876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.048973+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1535,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.913538+0000","last_change":"2024-04-26T16:38:19.900346+0000","last_active":"2024-04-26T16:38:21.913538+0000","last_peered":"2024-04-26T16:38:21.913538+0000","last_clean":"2024-04-26T16:38:21.913538+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:38:21.913538+0000","last_undegraded":"2024-04-26T16:38:21.913538+0000","last_fullsized":"2024-04-26T16:38:21.913538+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:19.900162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:19.900162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:38.918906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"857'27","reported_seq":1708,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009077+0000","last_change":"2024-04-26T16:36:19.167372+0000","last_active":"2024-04-26T16:38:22.009077+0000","last_peered":"2024-04-26T16:38:22.009077+0000","last_clean":"2024-04-26T16:38:22.009077+0000","last_became_active":"2024-04-26T16:35:25.395993+0000","last_became_peered":"2024-04-26T16:35:25.395993+0000","last_unstale":"2024-04-26T16:38:22.009077+0000","last_undegraded":"2024-04-26T16:38:22.009077+0000","last_fullsized":"2024-04-26T16:38:22.009077+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'19","last_scrub_stamp":"2024-04-26T16:36:19.167132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:19.167132+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1700,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014003+0000","last_change":"2024-04-26T16:38:09.331153+0000","last_active":"2024-04-26T16:38:22.014003+0000","last_peered":"2024-04-26T16:38:22.014003+0000","last_clean":"2024-04-26T16:38:22.014003+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:38:22.014003+0000","last_undegraded":"2024-04-26T16:38:22.014003+0000","last_fullsized":"2024-04-26T16:38:22.014003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.330900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.330900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:25.207505+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1659,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437483+0000","last_change":"2024-04-26T16:37:51.748594+0000","last_active":"2024-04-26T16:38:21.437483+0000","last_peered":"2024-04-26T16:38:21.437483+0000","last_clean":"2024-04-26T16:38:21.437483+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:38:21.437483+0000","last_undegraded":"2024-04-26T16:38:21.437483+0000","last_fullsized":"2024-04-26T16:38:21.437483+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.748496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.748496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.803195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1679,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.015689+0000","last_change":"2024-04-26T16:37:51.423701+0000","last_active":"2024-04-26T16:38:22.015689+0000","last_peered":"2024-04-26T16:38:22.015689+0000","last_clean":"2024-04-26T16:38:22.015689+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:38:22.015689+0000","last_undegraded":"2024-04-26T16:38:22.015689+0000","last_fullsized":"2024-04-26T16:38:22.015689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.423243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:54.807120+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.423243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:54.335512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1545,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.013263+0000","last_change":"2024-04-26T16:37:56.382398+0000","last_active":"2024-04-26T16:38:22.013263+0000","last_peered":"2024-04-26T16:38:22.013263+0000","last_clean":"2024-04-26T16:38:22.013263+0000","last_became_active":"2024-04-26T16:35:25.399182+0000","last_became_peered":"2024-04-26T16:35:25.399182+0000","last_unstale":"2024-04-26T16:38:22.013263+0000","last_undegraded":"2024-04-26T16:38:22.013263+0000","last_fullsized":"2024-04-26T16:38:22.013263+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:56.382182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.382182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.313965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"856'12","reported_seq":1418,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129349+0000","last_change":"2024-04-26T16:37:56.309533+0000","last_active":"2024-04-26T16:38:22.129349+0000","last_peered":"2024-04-26T16:38:22.129349+0000","last_clean":"2024-04-26T16:38:22.129349+0000","last_became_active":"2024-04-26T16:35:25.404028+0000","last_became_peered":"2024-04-26T16:35:25.404028+0000","last_unstale":"2024-04-26T16:38:22.129349+0000","last_undegraded":"2024-04-26T16:38:22.129349+0000","last_fullsized":"2024-04-26T16:38:22.129349+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'12","last_scrub_stamp":"2024-04-26T16:37:56.309129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.309129+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.607105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1263,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129795+0000","last_change":"2024-04-26T16:38:18.352107+0000","last_active":"2024-04-26T16:38:22.129795+0000","last_peered":"2024-04-26T16:38:22.129795+0000","last_clean":"2024-04-26T16:38:22.129795+0000","last_became_active":"2024-04-26T16:35:25.403847+0000","last_became_peered":"2024-04-26T16:35:25.403847+0000","last_unstale":"2024-04-26T16:38:22.129795+0000","last_undegraded":"2024-04-26T16:38:22.129795+0000","last_fullsized":"2024-04-26T16:38:22.129795+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:18.351875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:38:18.351875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:47.112344+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995930+0000","last_change":"2024-04-26T16:36:41.687047+0000","last_active":"2024-04-26T16:38:21.995930+0000","last_peered":"2024-04-26T16:38:21.995930+0000","last_clean":"2024-04-26T16:38:21.995930+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:38:21.995930+0000","last_undegraded":"2024-04-26T16:38:21.995930+0000","last_fullsized":"2024-04-26T16:38:21.995930+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:41.686919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:41.686919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1657,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009828+0000","last_change":"2024-04-26T16:36:33.166398+0000","last_active":"2024-04-26T16:38:22.009828+0000","last_peered":"2024-04-26T16:38:22.009828+0000","last_clean":"2024-04-26T16:38:22.009828+0000","last_became_active":"2024-04-26T16:35:25.379432+0000","last_became_peered":"2024-04-26T16:35:25.379432+0000","last_unstale":"2024-04-26T16:38:22.009828+0000","last_undegraded":"2024-04-26T16:38:22.009828+0000","last_fullsized":"2024-04-26T16:38:22.009828+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:33.166179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.166179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1540,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436224+0000","last_change":"2024-04-26T16:37:43.955898+0000","last_active":"2024-04-26T16:38:21.436224+0000","last_peered":"2024-04-26T16:38:21.436224+0000","last_clean":"2024-04-26T16:38:21.436224+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:38:21.436224+0000","last_undegraded":"2024-04-26T16:38:21.436224+0000","last_fullsized":"2024-04-26T16:38:21.436224+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:43.955684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:43.955684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:12.284132+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"857'27","reported_seq":1724,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014317+0000","last_change":"2024-04-26T16:38:22.014440+0000","last_active":"2024-04-26T16:38:22.014317+0000","last_peered":"2024-04-26T16:38:22.014317+0000","last_clean":"2024-04-26T16:38:22.014317+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:38:22.014317+0000","last_undegraded":"2024-04-26T16:38:22.014317+0000","last_fullsized":"2024-04-26T16:38:22.014317+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:38:22.014118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:22.014118+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:39.811894+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.994936+0000","last_change":"2024-04-26T16:36:49.734509+0000","last_active":"2024-04-26T16:38:21.994936+0000","last_peered":"2024-04-26T16:38:21.994936+0000","last_clean":"2024-04-26T16:38:21.994936+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:38:21.994936+0000","last_undegraded":"2024-04-26T16:38:21.994936+0000","last_fullsized":"2024-04-26T16:38:21.994936+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:49.734276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:49.734276+0000","last_clean_scrub_stamp":"2024-04-26T16:36:49.734276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":687,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996448+0000","last_change":"2024-04-26T16:37:56.088969+0000","last_active":"2024-04-26T16:38:21.996448+0000","last_peered":"2024-04-26T16:38:21.996448+0000","last_clean":"2024-04-26T16:38:21.996448+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:38:21.996448+0000","last_undegraded":"2024-04-26T16:38:21.996448+0000","last_fullsized":"2024-04-26T16:38:21.996448+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:37:56.088660+0000","last_deep_scrub":"101'12","last_deep_scrub_stamp":"2024-04-26T16:19:51.756984+0000","last_clean_scrub_stamp":"2024-04-26T16:37:56.088660+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:09.991987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1671,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.915856+0000","last_change":"2024-04-26T16:37:53.920778+0000","last_active":"2024-04-26T16:38:21.915856+0000","last_peered":"2024-04-26T16:38:21.915856+0000","last_clean":"2024-04-26T16:38:21.915856+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:38:21.915856+0000","last_undegraded":"2024-04-26T16:38:21.915856+0000","last_fullsized":"2024-04-26T16:38:21.915856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:53.920538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:53.920538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.535913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.994886+0000","last_change":"2024-04-26T16:36:54.762255+0000","last_active":"2024-04-26T16:38:21.994886+0000","last_peered":"2024-04-26T16:38:21.994886+0000","last_clean":"2024-04-26T16:38:21.994886+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:38:21.994886+0000","last_undegraded":"2024-04-26T16:38:21.994886+0000","last_fullsized":"2024-04-26T16:38:21.994886+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:54.762119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:36:54.762119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"857'25","reported_seq":1717,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.914275+0000","last_change":"2024-04-26T16:38:02.824020+0000","last_active":"2024-04-26T16:38:21.914275+0000","last_peered":"2024-04-26T16:38:21.914275+0000","last_clean":"2024-04-26T16:38:21.914275+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:38:21.914275+0000","last_undegraded":"2024-04-26T16:38:21.914275+0000","last_fullsized":"2024-04-26T16:38:21.914275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:38:02.823785+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.823785+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.431260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1691,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:25.839724+0000","last_change":"2024-04-26T16:38:25.839783+0000","last_active":"2024-04-26T16:38:25.839724+0000","last_peered":"2024-04-26T16:38:25.839724+0000","last_clean":"2024-04-26T16:38:25.839724+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:38:25.839724+0000","last_undegraded":"2024-04-26T16:38:25.839724+0000","last_fullsized":"2024-04-26T16:38:25.839724+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:25.839574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_clean_scrub_stamp":"2024-04-26T16:38:25.839574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:50.969135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"86'11","reported_seq":1921,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436653+0000","last_change":"2024-04-26T16:37:44.755462+0000","last_active":"2024-04-26T16:38:21.436653+0000","last_peered":"2024-04-26T16:38:21.436653+0000","last_clean":"2024-04-26T16:38:21.436653+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:38:21.436653+0000","last_undegraded":"2024-04-26T16:38:21.436653+0000","last_fullsized":"2024-04-26T16:38:21.436653+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'11","last_scrub_stamp":"2024-04-26T16:37:44.755315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.755315+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:55.270439+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1699,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437034+0000","last_change":"2024-04-26T16:38:05.800827+0000","last_active":"2024-04-26T16:38:21.437034+0000","last_peered":"2024-04-26T16:38:21.437034+0000","last_clean":"2024-04-26T16:38:21.437034+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:38:21.437034+0000","last_undegraded":"2024-04-26T16:38:21.437034+0000","last_fullsized":"2024-04-26T16:38:21.437034+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:05.800618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.800618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.334366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1510,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010768+0000","last_change":"2024-04-26T16:38:13.084550+0000","last_active":"2024-04-26T16:38:22.010768+0000","last_peered":"2024-04-26T16:38:22.010768+0000","last_clean":"2024-04-26T16:38:22.010768+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:38:22.010768+0000","last_undegraded":"2024-04-26T16:38:22.010768+0000","last_fullsized":"2024-04-26T16:38:22.010768+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:13.084343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:13.084343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.917769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"857'25","reported_seq":1701,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.011030+0000","last_change":"2024-04-26T16:38:08.098586+0000","last_active":"2024-04-26T16:38:22.011030+0000","last_peered":"2024-04-26T16:38:22.011030+0000","last_clean":"2024-04-26T16:38:22.011030+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:38:22.011030+0000","last_undegraded":"2024-04-26T16:38:22.011030+0000","last_fullsized":"2024-04-26T16:38:22.011030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:38:08.098198+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:16.092465+0000","last_clean_scrub_stamp":"2024-04-26T16:38:08.098198+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:13.219884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1337,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129560+0000","last_change":"2024-04-26T16:37:45.265963+0000","last_active":"2024-04-26T16:38:22.129560+0000","last_peered":"2024-04-26T16:38:22.129560+0000","last_clean":"2024-04-26T16:38:22.129560+0000","last_became_active":"2024-04-26T16:35:25.401229+0000","last_became_peered":"2024-04-26T16:35:25.401229+0000","last_unstale":"2024-04-26T16:38:22.129560+0000","last_undegraded":"2024-04-26T16:38:22.129560+0000","last_fullsized":"2024-04-26T16:38:22.129560+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:45.265860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:12.898432+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.265860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:07.529405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1667,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436542+0000","last_change":"2024-04-26T16:37:49.748432+0000","last_active":"2024-04-26T16:38:21.436542+0000","last_peered":"2024-04-26T16:38:21.436542+0000","last_clean":"2024-04-26T16:38:21.436542+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:38:21.436542+0000","last_undegraded":"2024-04-26T16:38:21.436542+0000","last_fullsized":"2024-04-26T16:38:21.436542+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:49.748209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:49.748209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:49.909641+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1700,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.913212+0000","last_change":"2024-04-26T16:38:20.888367+0000","last_active":"2024-04-26T16:38:21.913212+0000","last_peered":"2024-04-26T16:38:21.913212+0000","last_clean":"2024-04-26T16:38:21.913212+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:38:21.913212+0000","last_undegraded":"2024-04-26T16:38:21.913212+0000","last_fullsized":"2024-04-26T16:38:21.913212+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:20.888123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:20.888123+0000","last_clean_scrub_stamp":"2024-04-26T16:38:20.888123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:30.990022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1154,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129714+0000","last_change":"2024-04-26T16:37:43.256055+0000","last_active":"2024-04-26T16:38:22.129714+0000","last_peered":"2024-04-26T16:38:22.129714+0000","last_clean":"2024-04-26T16:38:22.129714+0000","last_became_active":"2024-04-26T16:35:25.409902+0000","last_became_peered":"2024-04-26T16:35:25.409902+0000","last_unstale":"2024-04-26T16:38:22.129714+0000","last_undegraded":"2024-04-26T16:38:22.129714+0000","last_fullsized":"2024-04-26T16:38:22.129714+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:43.255960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:37:43.255960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:11.195362+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"857'30","reported_seq":1742,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436385+0000","last_change":"2024-04-26T16:37:49.783263+0000","last_active":"2024-04-26T16:38:21.436385+0000","last_peered":"2024-04-26T16:38:21.436385+0000","last_clean":"2024-04-26T16:38:21.436385+0000","last_became_active":"2024-04-26T16:35:25.383698+0000","last_became_peered":"2024-04-26T16:35:25.383698+0000","last_unstale":"2024-04-26T16:38:21.436385+0000","last_undegraded":"2024-04-26T16:38:21.436385+0000","last_fullsized":"2024-04-26T16:38:21.436385+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:37:49.783106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:49.783106+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:50.844745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1202,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435510+0000","last_change":"2024-04-26T16:38:00.194757+0000","last_active":"2024-04-26T16:38:21.435510+0000","last_peered":"2024-04-26T16:38:21.435510+0000","last_clean":"2024-04-26T16:38:21.435510+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:38:21.435510+0000","last_undegraded":"2024-04-26T16:38:21.435510+0000","last_fullsized":"2024-04-26T16:38:21.435510+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:00.194512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.194512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:08.614121+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1691,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436815+0000","last_change":"2024-04-26T16:37:01.057309+0000","last_active":"2024-04-26T16:38:21.436815+0000","last_peered":"2024-04-26T16:38:21.436815+0000","last_clean":"2024-04-26T16:38:21.436815+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:38:21.436815+0000","last_undegraded":"2024-04-26T16:38:21.436815+0000","last_fullsized":"2024-04-26T16:38:21.436815+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:01.057091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:37:01.057091+0000","last_clean_scrub_stamp":"2024-04-26T16:37:01.057091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1667,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435412+0000","last_change":"2024-04-26T16:36:44.583891+0000","last_active":"2024-04-26T16:38:21.435412+0000","last_peered":"2024-04-26T16:38:21.435412+0000","last_clean":"2024-04-26T16:38:21.435412+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:38:21.435412+0000","last_undegraded":"2024-04-26T16:38:21.435412+0000","last_fullsized":"2024-04-26T16:38:21.435412+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:44.583680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:36:44.583680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1536,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.016408+0000","last_change":"2024-04-26T16:37:00.520805+0000","last_active":"2024-04-26T16:38:22.016408+0000","last_peered":"2024-04-26T16:38:22.016408+0000","last_clean":"2024-04-26T16:38:22.016408+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:38:22.016408+0000","last_undegraded":"2024-04-26T16:38:22.016408+0000","last_fullsized":"2024-04-26T16:38:22.016408+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.520675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.520675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:22.348302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"857'18","reported_seq":1751,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.915896+0000","last_change":"2024-04-26T16:37:44.980167+0000","last_active":"2024-04-26T16:38:21.915896+0000","last_peered":"2024-04-26T16:38:21.915896+0000","last_clean":"2024-04-26T16:38:21.915896+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:38:21.915896+0000","last_undegraded":"2024-04-26T16:38:21.915896+0000","last_fullsized":"2024-04-26T16:38:21.915896+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'18","last_scrub_stamp":"2024-04-26T16:37:44.980066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.980066+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.790501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":608,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996192+0000","last_change":"2024-04-26T16:38:19.245827+0000","last_active":"2024-04-26T16:38:21.996192+0000","last_peered":"2024-04-26T16:38:21.996192+0000","last_clean":"2024-04-26T16:38:21.996192+0000","last_became_active":"2024-04-26T16:35:25.389184+0000","last_became_peered":"2024-04-26T16:35:25.389184+0000","last_unstale":"2024-04-26T16:38:21.996192+0000","last_undegraded":"2024-04-26T16:38:21.996192+0000","last_fullsized":"2024-04-26T16:38:21.996192+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:19.245611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:19.245611+0000","last_clean_scrub_stamp":"2024-04-26T16:38:19.245611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.171876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1303,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129237+0000","last_change":"2024-04-26T16:38:10.273089+0000","last_active":"2024-04-26T16:38:22.129237+0000","last_peered":"2024-04-26T16:38:22.129237+0000","last_clean":"2024-04-26T16:38:22.129237+0000","last_became_active":"2024-04-26T16:35:25.405230+0000","last_became_peered":"2024-04-26T16:35:25.405230+0000","last_unstale":"2024-04-26T16:38:22.129237+0000","last_undegraded":"2024-04-26T16:38:22.129237+0000","last_fullsized":"2024-04-26T16:38:22.129237+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:10.272810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:10.272810+0000","last_clean_scrub_stamp":"2024-04-26T16:38:10.272810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.052304+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1647,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009971+0000","last_change":"2024-04-26T16:37:00.148921+0000","last_active":"2024-04-26T16:38:22.009971+0000","last_peered":"2024-04-26T16:38:22.009971+0000","last_clean":"2024-04-26T16:38:22.009971+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:38:22.009971+0000","last_undegraded":"2024-04-26T16:38:22.009971+0000","last_fullsized":"2024-04-26T16:38:22.009971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:00.148735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:00.148735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1522,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010708+0000","last_change":"2024-04-26T16:36:38.157649+0000","last_active":"2024-04-26T16:38:22.010708+0000","last_peered":"2024-04-26T16:38:22.010708+0000","last_clean":"2024-04-26T16:38:22.010708+0000","last_became_active":"2024-04-26T16:35:25.394016+0000","last_became_peered":"2024-04-26T16:35:25.394016+0000","last_unstale":"2024-04-26T16:38:22.010708+0000","last_undegraded":"2024-04-26T16:38:22.010708+0000","last_fullsized":"2024-04-26T16:38:22.010708+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:38.157418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:39.702030+0000","last_clean_scrub_stamp":"2024-04-26T16:36:38.157418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"856'21","reported_seq":626,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996088+0000","last_change":"2024-04-26T16:36:44.718941+0000","last_active":"2024-04-26T16:38:21.996088+0000","last_peered":"2024-04-26T16:38:21.996088+0000","last_clean":"2024-04-26T16:38:21.996088+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:38:21.996088+0000","last_undegraded":"2024-04-26T16:38:21.996088+0000","last_fullsized":"2024-04-26T16:38:21.996088+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'15","last_scrub_stamp":"2024-04-26T16:36:44.718541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:36:44.718541+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1306,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.130662+0000","last_change":"2024-04-26T16:37:44.221946+0000","last_active":"2024-04-26T16:38:22.130662+0000","last_peered":"2024-04-26T16:38:22.130662+0000","last_clean":"2024-04-26T16:38:22.130662+0000","last_became_active":"2024-04-26T16:35:25.406825+0000","last_became_peered":"2024-04-26T16:35:25.406825+0000","last_unstale":"2024-04-26T16:38:22.130662+0000","last_undegraded":"2024-04-26T16:38:22.130662+0000","last_fullsized":"2024-04-26T16:38:22.130662+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:44.221823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:19:00.145597+0000","last_clean_scrub_stamp":"2024-04-26T16:37:44.221823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:13.401315+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1661,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009336+0000","last_change":"2024-04-26T16:36:15.209969+0000","last_active":"2024-04-26T16:38:22.009336+0000","last_peered":"2024-04-26T16:38:22.009336+0000","last_clean":"2024-04-26T16:38:22.009336+0000","last_became_active":"2024-04-26T16:35:25.378783+0000","last_became_peered":"2024-04-26T16:35:25.378783+0000","last_unstale":"2024-04-26T16:38:22.009336+0000","last_undegraded":"2024-04-26T16:38:22.009336+0000","last_fullsized":"2024-04-26T16:38:22.009336+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:15.209875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:15.209875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1665,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436728+0000","last_change":"2024-04-26T16:37:55.748337+0000","last_active":"2024-04-26T16:38:21.436728+0000","last_peered":"2024-04-26T16:38:21.436728+0000","last_clean":"2024-04-26T16:38:21.436728+0000","last_became_active":"2024-04-26T16:35:25.410836+0000","last_became_peered":"2024-04-26T16:35:25.410836+0000","last_unstale":"2024-04-26T16:38:21.436728+0000","last_undegraded":"2024-04-26T16:38:21.436728+0000","last_fullsized":"2024-04-26T16:38:21.436728+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:55.748010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:55.748010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:06.141796+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1535,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.014003+0000","last_change":"2024-04-26T16:37:59.386367+0000","last_active":"2024-04-26T16:38:22.014003+0000","last_peered":"2024-04-26T16:38:22.014003+0000","last_clean":"2024-04-26T16:38:22.014003+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:38:22.014003+0000","last_undegraded":"2024-04-26T16:38:22.014003+0000","last_fullsized":"2024-04-26T16:38:22.014003+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:59.386150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.386150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:16.277947+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"857'46","reported_seq":1759,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.434932+0000","last_change":"2024-04-26T16:37:04.034823+0000","last_active":"2024-04-26T16:38:21.434932+0000","last_peered":"2024-04-26T16:38:21.434932+0000","last_clean":"2024-04-26T16:38:21.434932+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:38:21.434932+0000","last_undegraded":"2024-04-26T16:38:21.434932+0000","last_fullsized":"2024-04-26T16:38:21.434932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'34","last_scrub_stamp":"2024-04-26T16:37:04.034569+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:37:04.034569+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.994284+0000","last_change":"2024-04-26T16:37:52.063624+0000","last_active":"2024-04-26T16:38:21.994284+0000","last_peered":"2024-04-26T16:38:21.994284+0000","last_clean":"2024-04-26T16:38:21.994284+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:38:21.994284+0000","last_undegraded":"2024-04-26T16:38:21.994284+0000","last_fullsized":"2024-04-26T16:38:21.994284+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.063411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.063411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.562434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"747'16","reported_seq":1502,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.130413+0000","last_change":"2024-04-26T16:38:05.198183+0000","last_active":"2024-04-26T16:38:22.130413+0000","last_peered":"2024-04-26T16:38:22.130413+0000","last_clean":"2024-04-26T16:38:22.130413+0000","last_became_active":"2024-04-26T16:35:25.403416+0000","last_became_peered":"2024-04-26T16:35:25.403416+0000","last_unstale":"2024-04-26T16:38:22.130413+0000","last_undegraded":"2024-04-26T16:38:22.130413+0000","last_fullsized":"2024-04-26T16:38:22.130413+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:38:05.197963+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:38:05.197963+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.854698+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"829'2","reported_seq":1670,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010495+0000","last_change":"2024-04-26T16:36:59.155190+0000","last_active":"2024-04-26T16:38:22.010495+0000","last_peered":"2024-04-26T16:38:22.010495+0000","last_clean":"2024-04-26T16:38:22.010495+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:38:22.010495+0000","last_undegraded":"2024-04-26T16:38:22.010495+0000","last_fullsized":"2024-04-26T16:38:22.010495+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'2","last_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_deep_scrub":"829'2","last_deep_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_clean_scrub_stamp":"2024-04-26T16:36:59.155048+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:27.337428+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1537,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.914713+0000","last_change":"2024-04-26T16:37:52.886432+0000","last_active":"2024-04-26T16:38:21.914713+0000","last_peered":"2024-04-26T16:38:21.914713+0000","last_clean":"2024-04-26T16:38:21.914713+0000","last_became_active":"2024-04-26T16:35:25.409763+0000","last_became_peered":"2024-04-26T16:35:25.409763+0000","last_unstale":"2024-04-26T16:38:21.914713+0000","last_undegraded":"2024-04-26T16:38:21.914713+0000","last_fullsized":"2024-04-26T16:38:21.914713+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:52.886311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:52.886311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:17.121107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"856'21","reported_seq":1705,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.523785+0000","last_change":"2024-04-26T16:38:22.523902+0000","last_active":"2024-04-26T16:38:22.523785+0000","last_peered":"2024-04-26T16:38:22.523785+0000","last_clean":"2024-04-26T16:38:22.523785+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:38:22.523785+0000","last_undegraded":"2024-04-26T16:38:22.523785+0000","last_fullsized":"2024-04-26T16:38:22.523785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'21","last_scrub_stamp":"2024-04-26T16:38:22.523529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:22.523529+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:38.499552+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1679,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435578+0000","last_change":"2024-04-26T16:37:59.723665+0000","last_active":"2024-04-26T16:38:21.435578+0000","last_peered":"2024-04-26T16:38:21.435578+0000","last_clean":"2024-04-26T16:38:21.435578+0000","last_became_active":"2024-04-26T16:35:25.417030+0000","last_became_peered":"2024-04-26T16:35:25.417030+0000","last_unstale":"2024-04-26T16:38:21.435578+0000","last_undegraded":"2024-04-26T16:38:21.435578+0000","last_fullsized":"2024-04-26T16:38:21.435578+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:37:59.723438+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:20:53.090830+0000","last_clean_scrub_stamp":"2024-04-26T16:37:59.723438+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.920638+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1693,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.872737+0000","last_change":"2024-04-26T16:38:22.872799+0000","last_active":"2024-04-26T16:38:22.872737+0000","last_peered":"2024-04-26T16:38:22.872737+0000","last_clean":"2024-04-26T16:38:22.872737+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:38:22.872737+0000","last_undegraded":"2024-04-26T16:38:22.872737+0000","last_fullsized":"2024-04-26T16:38:22.872737+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:22.872578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:22.872578+0000","last_clean_scrub_stamp":"2024-04-26T16:38:22.872578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:38.452737+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1695,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435923+0000","last_change":"2024-04-26T16:36:57.199063+0000","last_active":"2024-04-26T16:38:21.435923+0000","last_peered":"2024-04-26T16:38:21.435923+0000","last_clean":"2024-04-26T16:38:21.435923+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:38:21.435923+0000","last_undegraded":"2024-04-26T16:38:21.435923+0000","last_fullsized":"2024-04-26T16:38:21.435923+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:57.198785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.198785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995045+0000","last_change":"2024-04-26T16:38:04.121917+0000","last_active":"2024-04-26T16:38:21.995045+0000","last_peered":"2024-04-26T16:38:21.995045+0000","last_clean":"2024-04-26T16:38:21.995045+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:38:21.995045+0000","last_undegraded":"2024-04-26T16:38:21.995045+0000","last_fullsized":"2024-04-26T16:38:21.995045+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:04.121774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.121774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.071499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"857'11","reported_seq":1697,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.915328+0000","last_change":"2024-04-26T16:36:57.942537+0000","last_active":"2024-04-26T16:38:21.915328+0000","last_peered":"2024-04-26T16:38:21.915328+0000","last_clean":"2024-04-26T16:38:21.915328+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:38:21.915328+0000","last_undegraded":"2024-04-26T16:38:21.915328+0000","last_fullsized":"2024-04-26T16:38:21.915328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'9","last_scrub_stamp":"2024-04-26T16:36:57.942209+0000","last_deep_scrub":"388'9","last_deep_scrub_stamp":"2024-04-26T16:36:57.942209+0000","last_clean_scrub_stamp":"2024-04-26T16:36:57.942209+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1363,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.128622+0000","last_change":"2024-04-26T16:37:47.358920+0000","last_active":"2024-04-26T16:38:22.128622+0000","last_peered":"2024-04-26T16:38:22.128622+0000","last_clean":"2024-04-26T16:38:22.128622+0000","last_became_active":"2024-04-26T16:35:25.404997+0000","last_became_peered":"2024-04-26T16:35:25.404997+0000","last_unstale":"2024-04-26T16:38:22.128622+0000","last_undegraded":"2024-04-26T16:38:22.128622+0000","last_fullsized":"2024-04-26T16:38:22.128622+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:47.358713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.358713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:57.165878+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1695,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435142+0000","last_change":"2024-04-26T16:38:09.125742+0000","last_active":"2024-04-26T16:38:21.435142+0000","last_peered":"2024-04-26T16:38:21.435142+0000","last_clean":"2024-04-26T16:38:21.435142+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:38:21.435142+0000","last_undegraded":"2024-04-26T16:38:21.435142+0000","last_fullsized":"2024-04-26T16:38:21.435142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.125384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.125384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:15.580879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1711,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437151+0000","last_change":"2024-04-26T16:37:51.818589+0000","last_active":"2024-04-26T16:38:21.437151+0000","last_peered":"2024-04-26T16:38:21.437151+0000","last_clean":"2024-04-26T16:38:21.437151+0000","last_became_active":"2024-04-26T16:35:25.383286+0000","last_became_peered":"2024-04-26T16:35:25.383286+0000","last_unstale":"2024-04-26T16:38:21.437151+0000","last_undegraded":"2024-04-26T16:38:21.437151+0000","last_fullsized":"2024-04-26T16:38:21.437151+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.818319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:15.936392+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.818319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:19.894110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1511,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.008812+0000","last_change":"2024-04-26T16:38:00.069058+0000","last_active":"2024-04-26T16:38:22.008812+0000","last_peered":"2024-04-26T16:38:22.008812+0000","last_clean":"2024-04-26T16:38:22.008812+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:38:22.008812+0000","last_undegraded":"2024-04-26T16:38:22.008812+0000","last_fullsized":"2024-04-26T16:38:22.008812+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:00.068758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:38:00.068758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:24.541601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"857'21","reported_seq":1729,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435947+0000","last_change":"2024-04-26T16:38:20.744401+0000","last_active":"2024-04-26T16:38:21.435947+0000","last_peered":"2024-04-26T16:38:21.435947+0000","last_clean":"2024-04-26T16:38:21.435947+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:38:21.435947+0000","last_undegraded":"2024-04-26T16:38:21.435947+0000","last_fullsized":"2024-04-26T16:38:21.435947+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'21","last_scrub_stamp":"2024-04-26T16:38:20.744142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:20.744142+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:49.302780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1677,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.436355+0000","last_change":"2024-04-26T16:38:03.698985+0000","last_active":"2024-04-26T16:38:21.436355+0000","last_peered":"2024-04-26T16:38:21.436355+0000","last_clean":"2024-04-26T16:38:21.436355+0000","last_became_active":"2024-04-26T16:35:25.392415+0000","last_became_peered":"2024-04-26T16:35:25.392415+0000","last_unstale":"2024-04-26T16:38:21.436355+0000","last_undegraded":"2024-04-26T16:38:21.436355+0000","last_fullsized":"2024-04-26T16:38:21.436355+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:03.698766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:03.698766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:23.722484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1691,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.766936+0000","last_change":"2024-04-26T16:38:22.766991+0000","last_active":"2024-04-26T16:38:22.766936+0000","last_peered":"2024-04-26T16:38:22.766936+0000","last_clean":"2024-04-26T16:38:22.766936+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:38:22.766936+0000","last_undegraded":"2024-04-26T16:38:22.766936+0000","last_fullsized":"2024-04-26T16:38:22.766936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:22.766790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_clean_scrub_stamp":"2024-04-26T16:38:22.766790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.995208+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1679,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437427+0000","last_change":"2024-04-26T16:37:58.738918+0000","last_active":"2024-04-26T16:38:21.437427+0000","last_peered":"2024-04-26T16:38:21.437427+0000","last_clean":"2024-04-26T16:38:21.437427+0000","last_became_active":"2024-04-26T16:35:25.410430+0000","last_became_peered":"2024-04-26T16:35:25.410430+0000","last_unstale":"2024-04-26T16:38:21.437427+0000","last_undegraded":"2024-04-26T16:38:21.437427+0000","last_fullsized":"2024-04-26T16:38:21.437427+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:58.738705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:58.738705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.841581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1535,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.015691+0000","last_change":"2024-04-26T16:37:47.396592+0000","last_active":"2024-04-26T16:38:22.015691+0000","last_peered":"2024-04-26T16:38:22.015691+0000","last_clean":"2024-04-26T16:38:22.015691+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:38:22.015691+0000","last_undegraded":"2024-04-26T16:38:22.015691+0000","last_fullsized":"2024-04-26T16:38:22.015691+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:47.396490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:47.396490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:53.967350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"857'29","reported_seq":629,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.996118+0000","last_change":"2024-04-26T16:36:33.541569+0000","last_active":"2024-04-26T16:38:21.996118+0000","last_peered":"2024-04-26T16:38:21.996118+0000","last_clean":"2024-04-26T16:38:21.996118+0000","last_became_active":"2024-04-26T16:35:25.382813+0000","last_became_peered":"2024-04-26T16:35:25.382813+0000","last_unstale":"2024-04-26T16:38:21.996118+0000","last_undegraded":"2024-04-26T16:38:21.996118+0000","last_fullsized":"2024-04-26T16:38:21.996118+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"388'23","last_scrub_stamp":"2024-04-26T16:36:33.541439+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:36:33.541439+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1688,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009494+0000","last_change":"2024-04-26T16:38:04.132311+0000","last_active":"2024-04-26T16:38:22.009494+0000","last_peered":"2024-04-26T16:38:22.009494+0000","last_clean":"2024-04-26T16:38:22.009494+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:38:22.009494+0000","last_undegraded":"2024-04-26T16:38:22.009494+0000","last_fullsized":"2024-04-26T16:38:22.009494+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:04.132193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:04.132193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:05.515649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":599,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.994842+0000","last_change":"2024-04-26T16:36:40.657677+0000","last_active":"2024-04-26T16:38:21.994842+0000","last_peered":"2024-04-26T16:38:21.994842+0000","last_clean":"2024-04-26T16:38:21.994842+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:38:21.994842+0000","last_undegraded":"2024-04-26T16:38:21.994842+0000","last_fullsized":"2024-04-26T16:38:21.994842+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:40.657538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:40.657538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1679,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.913547+0000","last_change":"2024-04-26T16:38:12.853887+0000","last_active":"2024-04-26T16:38:21.913547+0000","last_peered":"2024-04-26T16:38:21.913547+0000","last_clean":"2024-04-26T16:38:21.913547+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:38:21.913547+0000","last_undegraded":"2024-04-26T16:38:21.913547+0000","last_fullsized":"2024-04-26T16:38:21.913547+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:12.853670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.855045+0000","last_clean_scrub_stamp":"2024-04-26T16:38:12.853670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:40.929680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1142,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.129175+0000","last_change":"2024-04-26T16:37:51.285062+0000","last_active":"2024-04-26T16:38:22.129175+0000","last_peered":"2024-04-26T16:38:22.129175+0000","last_clean":"2024-04-26T16:38:22.129175+0000","last_became_active":"2024-04-26T16:35:25.401906+0000","last_became_peered":"2024-04-26T16:35:25.401906+0000","last_unstale":"2024-04-26T16:38:22.129175+0000","last_undegraded":"2024-04-26T16:38:22.129175+0000","last_fullsized":"2024-04-26T16:38:22.129175+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:51.284847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:37:51.284847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:14.463961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"857'39","reported_seq":644,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995544+0000","last_change":"2024-04-26T16:37:45.123276+0000","last_active":"2024-04-26T16:38:21.995544+0000","last_peered":"2024-04-26T16:38:21.995544+0000","last_clean":"2024-04-26T16:38:21.995544+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:38:21.995544+0000","last_undegraded":"2024-04-26T16:38:21.995544+0000","last_fullsized":"2024-04-26T16:38:21.995544+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'39","last_scrub_stamp":"2024-04-26T16:37:45.122923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.122923+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:02.778170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1677,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.010718+0000","last_change":"2024-04-26T16:36:23.142745+0000","last_active":"2024-04-26T16:38:22.010718+0000","last_peered":"2024-04-26T16:38:22.010718+0000","last_clean":"2024-04-26T16:38:22.010718+0000","last_became_active":"2024-04-26T16:35:25.389963+0000","last_became_peered":"2024-04-26T16:35:25.389963+0000","last_unstale":"2024-04-26T16:38:22.010718+0000","last_undegraded":"2024-04-26T16:38:22.010718+0000","last_fullsized":"2024-04-26T16:38:22.010718+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:23.142443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:36:23.142443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1651,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.009669+0000","last_change":"2024-04-26T16:36:46.231728+0000","last_active":"2024-04-26T16:38:22.009669+0000","last_peered":"2024-04-26T16:38:22.009669+0000","last_clean":"2024-04-26T16:38:22.009669+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:38:22.009669+0000","last_undegraded":"2024-04-26T16:38:22.009669+0000","last_fullsized":"2024-04-26T16:38:22.009669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:46.231628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:36:46.231628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":611,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.994440+0000","last_change":"2024-04-26T16:37:48.095046+0000","last_active":"2024-04-26T16:38:21.994440+0000","last_peered":"2024-04-26T16:38:21.994440+0000","last_clean":"2024-04-26T16:38:21.994440+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:38:21.994440+0000","last_undegraded":"2024-04-26T16:38:21.994440+0000","last_fullsized":"2024-04-26T16:38:21.994440+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:37:48.094883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:37:48.094883+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:52.909485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1533,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:22.011258+0000","last_change":"2024-04-26T16:36:22.139010+0000","last_active":"2024-04-26T16:38:22.011258+0000","last_peered":"2024-04-26T16:38:22.011258+0000","last_clean":"2024-04-26T16:38:22.011258+0000","last_became_active":"2024-04-26T16:35:25.388177+0000","last_became_peered":"2024-04-26T16:35:25.388177+0000","last_unstale":"2024-04-26T16:38:22.011258+0000","last_undegraded":"2024-04-26T16:38:22.011258+0000","last_fullsized":"2024-04-26T16:38:22.011258+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:36:22.138796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:36:22.138796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"857'27","reported_seq":1777,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.915909+0000","last_change":"2024-04-26T16:38:08.788411+0000","last_active":"2024-04-26T16:38:21.915909+0000","last_peered":"2024-04-26T16:38:21.915909+0000","last_clean":"2024-04-26T16:38:21.915909+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:38:21.915909+0000","last_undegraded":"2024-04-26T16:38:21.915909+0000","last_fullsized":"2024-04-26T16:38:21.915909+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:38:08.788157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:08.788157+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:20.504129+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1697,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437154+0000","last_change":"2024-04-26T16:38:18.565114+0000","last_active":"2024-04-26T16:38:21.437154+0000","last_peered":"2024-04-26T16:38:21.437154+0000","last_clean":"2024-04-26T16:38:21.437154+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:38:21.437154+0000","last_undegraded":"2024-04-26T16:38:21.437154+0000","last_fullsized":"2024-04-26T16:38:21.437154+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:18.564887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:38:18.564887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:40.546098+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1707,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435053+0000","last_change":"2024-04-26T16:38:16.162394+0000","last_active":"2024-04-26T16:38:21.435053+0000","last_peered":"2024-04-26T16:38:21.435053+0000","last_clean":"2024-04-26T16:38:21.435053+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:38:21.435053+0000","last_undegraded":"2024-04-26T16:38:21.435053+0000","last_fullsized":"2024-04-26T16:38:21.435053+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:16.162173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.136177+0000","last_clean_scrub_stamp":"2024-04-26T16:38:16.162173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:39.174742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1699,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435756+0000","last_change":"2024-04-26T16:38:14.731024+0000","last_active":"2024-04-26T16:38:21.435756+0000","last_peered":"2024-04-26T16:38:21.435756+0000","last_clean":"2024-04-26T16:38:21.435756+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:38:21.435756+0000","last_undegraded":"2024-04-26T16:38:21.435756+0000","last_fullsized":"2024-04-26T16:38:21.435756+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:14.730813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:55.083331+0000","last_clean_scrub_stamp":"2024-04-26T16:38:14.730813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.405891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":607,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.995862+0000","last_change":"2024-04-26T16:38:01.112600+0000","last_active":"2024-04-26T16:38:21.995862+0000","last_peered":"2024-04-26T16:38:21.995862+0000","last_clean":"2024-04-26T16:38:21.995862+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:38:21.995862+0000","last_undegraded":"2024-04-26T16:38:21.995862+0000","last_fullsized":"2024-04-26T16:38:21.995862+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:01.112389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:16.569721+0000","last_clean_scrub_stamp":"2024-04-26T16:38:01.112389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:22.253547+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"857'30","reported_seq":1728,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.437341+0000","last_change":"2024-04-26T16:38:15.633443+0000","last_active":"2024-04-26T16:38:21.437341+0000","last_peered":"2024-04-26T16:38:21.437341+0000","last_clean":"2024-04-26T16:38:21.437341+0000","last_became_active":"2024-04-26T16:35:25.399680+0000","last_became_peered":"2024-04-26T16:35:25.399680+0000","last_unstale":"2024-04-26T16:38:21.437341+0000","last_undegraded":"2024-04-26T16:38:21.437341+0000","last_fullsized":"2024-04-26T16:38:21.437341+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:38:15.633152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:38:15.633152+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:33.147835+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1693,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.916183+0000","last_change":"2024-04-26T16:38:09.828067+0000","last_active":"2024-04-26T16:38:21.916183+0000","last_peered":"2024-04-26T16:38:21.916183+0000","last_clean":"2024-04-26T16:38:21.916183+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:38:21.916183+0000","last_undegraded":"2024-04-26T16:38:21.916183+0000","last_fullsized":"2024-04-26T16:38:21.916183+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:09.827896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:38:09.827896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:37.537447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1693,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435409+0000","last_change":"2024-04-26T16:37:45.893079+0000","last_active":"2024-04-26T16:38:21.435409+0000","last_peered":"2024-04-26T16:38:21.435409+0000","last_clean":"2024-04-26T16:38:21.435409+0000","last_became_active":"2024-04-26T16:35:25.396616+0000","last_became_peered":"2024-04-26T16:35:25.396616+0000","last_unstale":"2024-04-26T16:38:21.435409+0000","last_undegraded":"2024-04-26T16:38:21.435409+0000","last_fullsized":"2024-04-26T16:38:21.435409+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:37:45.892786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:37:45.892786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:38:49.656756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1665,"reported_epoch":877,"state":"active+clean","last_fresh":"2024-04-26T16:38:21.435794+0000","last_change":"2024-04-26T16:38:02.658129+0000","last_active":"2024-04-26T16:38:21.435794+0000","last_peered":"2024-04-26T16:38:21.435794+0000","last_clean":"2024-04-26T16:38:21.435794+0000","last_became_active":"2024-04-26T16:35:25.392061+0000","last_became_peered":"2024-04-26T16:35:25.392061+0000","last_unstale":"2024-04-26T16:38:21.435794+0000","last_undegraded":"2024-04-26T16:38:21.435794+0000","last_fullsized":"2024-04-26T16:38:21.435794+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:38:02.657915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:38:02.657915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:27.431423+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1234,"num_read_kb":1229,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743686,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112448,"kb_used_data":4920,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93632704,"statfs":{"total":95995035648,"available":95879888896,"internally_reserved":0,"allocated":5038080,"data_stored":2630601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":746,"seq":3204045602853,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":38548,"kb_used_data":4888,"kb_used_omap":23,"kb_used_meta":33384,"kb_avail":93706604,"statfs":{"total":95995035648,"available":95955562496,"internally_reserved":0,"allocated":5005312,"data_stored":2627835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23760,"internal_metadata":34186032},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575172,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":103536,"kb_used_data":4328,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93641616,"statfs":{"total":95995035648,"available":95889014784,"internally_reserved":0,"allocated":4431872,"data_stored":2038799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738702,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108284,"kb_used_data":4916,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93636868,"statfs":{"total":95995035648,"available":95884152832,"internally_reserved":0,"allocated":5033984,"data_stored":2630029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411641,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107680,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93637472,"statfs":{"total":95995035648,"available":95884771328,"internally_reserved":0,"allocated":4415488,"data_stored":2038065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084590,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":129824,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":125502,"kb_avail":93615328,"statfs":{"total":95995035648,"available":95862095872,"internally_reserved":0,"allocated":4415488,"data_stored":2037867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":128514504},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855652,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107952,"kb_used_data":4340,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93637200,"statfs":{"total":95995035648,"available":95884492800,"internally_reserved":0,"allocated":4444160,"data_stored":2038623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463194,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93792,"kb_used_data":4312,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93651360,"statfs":{"total":95995035648,"available":95898992640,"internally_reserved":0,"allocated":4415488,"data_stored":2037877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:38:30.820 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-04-26T16:38:30.820 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-04-26T16:38:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.379346+0000 osd.3 (osd.3) 521 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:38:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.379346+0000 osd.3 (osd.3) 521 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:38:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.382366+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:38:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.382366+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:38:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:30.063508+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:38:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:30.063508+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:38:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:30.065448+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:38:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:30 smithi190 bash[23868]: cluster 2024-04-26T16:38:30.065448+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:38:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.379346+0000 osd.3 (osd.3) 521 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:38:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.379346+0000 osd.3 (osd.3) 521 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:38:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.382366+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:38:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.382366+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:30.063508+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:30.063508+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:30.065448+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[21463]: cluster 2024-04-26T16:38:30.065448+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.379346+0000 osd.3 (osd.3) 521 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.379346+0000 osd.3 (osd.3) 521 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.382366+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.382366+0000 osd.3 (osd.3) 522 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:30.063508+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:30.063508+0000 osd.5 (osd.5) 350 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:30.065448+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:38:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:30 smithi050 bash[29234]: cluster 2024-04-26T16:38:30.065448+0000 osd.5 (osd.5) 351 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:38:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: audit 2024-04-26T16:38:29.865749+0000 mgr.y (mgr.24401) 807 : audit [DBG] from='client.41762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: audit 2024-04-26T16:38:29.865749+0000 mgr.y (mgr.24401) 807 : audit [DBG] from='client.41762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.984271+0000 mgr.y (mgr.24401) 808 : cluster [DBG] pgmap v1383: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: cluster 2024-04-26T16:38:29.984271+0000 mgr.y (mgr.24401) 808 : cluster [DBG] pgmap v1383: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.491836+0000 osd.2 (osd.2) 435 : cluster [DBG] 11.d scrub starts 2024-04-26T16:38:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.491836+0000 osd.2 (osd.2) 435 : cluster [DBG] 11.d scrub starts 2024-04-26T16:38:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.495416+0000 osd.2 (osd.2) 436 : cluster [DBG] 11.d scrub ok 2024-04-26T16:38:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:31 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.495416+0000 osd.2 (osd.2) 436 : cluster [DBG] 11.d scrub ok 2024-04-26T16:38:32.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: audit 2024-04-26T16:38:29.865749+0000 mgr.y (mgr.24401) 807 : audit [DBG] from='client.41762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:32.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: audit 2024-04-26T16:38:29.865749+0000 mgr.y (mgr.24401) 807 : audit [DBG] from='client.41762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.984271+0000 mgr.y (mgr.24401) 808 : cluster [DBG] pgmap v1383: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: cluster 2024-04-26T16:38:29.984271+0000 mgr.y (mgr.24401) 808 : cluster [DBG] pgmap v1383: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.491836+0000 osd.2 (osd.2) 435 : cluster [DBG] 11.d scrub starts 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.491836+0000 osd.2 (osd.2) 435 : cluster [DBG] 11.d scrub starts 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.495416+0000 osd.2 (osd.2) 436 : cluster [DBG] 11.d scrub ok 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.495416+0000 osd.2 (osd.2) 436 : cluster [DBG] 11.d scrub ok 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: audit 2024-04-26T16:38:29.865749+0000 mgr.y (mgr.24401) 807 : audit [DBG] from='client.41762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: audit 2024-04-26T16:38:29.865749+0000 mgr.y (mgr.24401) 807 : audit [DBG] from='client.41762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.984271+0000 mgr.y (mgr.24401) 808 : cluster [DBG] pgmap v1383: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: cluster 2024-04-26T16:38:29.984271+0000 mgr.y (mgr.24401) 808 : cluster [DBG] pgmap v1383: 164 pgs: 164 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:32.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.491836+0000 osd.2 (osd.2) 435 : cluster [DBG] 11.d scrub starts 2024-04-26T16:38:32.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.491836+0000 osd.2 (osd.2) 435 : cluster [DBG] 11.d scrub starts 2024-04-26T16:38:32.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.495416+0000 osd.2 (osd.2) 436 : cluster [DBG] 11.d scrub ok 2024-04-26T16:38:32.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:31 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.495416+0000 osd.2 (osd.2) 436 : cluster [DBG] 11.d scrub ok 2024-04-26T16:38:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.071874+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.071874+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.075015+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.075015+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.378776+0000 osd.3 (osd.3) 523 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.378776+0000 osd.3 (osd.3) 523 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.384615+0000 osd.3 (osd.3) 524 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.384615+0000 osd.3 (osd.3) 524 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.044760+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.044760+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:38:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.049158+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:38:33.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:32 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.049158+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:38:33.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.071874+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.071874+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.075015+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.075015+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.378776+0000 osd.3 (osd.3) 523 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.378776+0000 osd.3 (osd.3) 523 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.384615+0000 osd.3 (osd.3) 524 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.384615+0000 osd.3 (osd.3) 524 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.044760+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.044760+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.049158+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:38:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.049158+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.071874+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.071874+0000 osd.7 (osd.7) 466 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.075015+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.075015+0000 osd.7 (osd.7) 467 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.378776+0000 osd.3 (osd.3) 523 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.378776+0000 osd.3 (osd.3) 523 : cluster [DBG] 3.7 deep-scrub starts 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.384615+0000 osd.3 (osd.3) 524 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.384615+0000 osd.3 (osd.3) 524 : cluster [DBG] 3.7 deep-scrub ok 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.044760+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.044760+0000 osd.7 (osd.7) 468 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.049158+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:38:33.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:32 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.049158+0000 osd.7 (osd.7) 469 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:38:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.986011+0000 mgr.y (mgr.24401) 809 : cluster [DBG] pgmap v1384: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:31.986011+0000 mgr.y (mgr.24401) 809 : cluster [DBG] pgmap v1384: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.450380+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.450380+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.453026+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:32.453026+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.514608+0000 osd.2 (osd.2) 437 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.514608+0000 osd.2 (osd.2) 437 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.517026+0000 osd.2 (osd.2) 438 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:38:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:33 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.517026+0000 osd.2 (osd.2) 438 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:38:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.986011+0000 mgr.y (mgr.24401) 809 : cluster [DBG] pgmap v1384: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:31.986011+0000 mgr.y (mgr.24401) 809 : cluster [DBG] pgmap v1384: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.450380+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:38:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.450380+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.453026+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:32.453026+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.514608+0000 osd.2 (osd.2) 437 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.514608+0000 osd.2 (osd.2) 437 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.517026+0000 osd.2 (osd.2) 438 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.517026+0000 osd.2 (osd.2) 438 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.986011+0000 mgr.y (mgr.24401) 809 : cluster [DBG] pgmap v1384: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:31.986011+0000 mgr.y (mgr.24401) 809 : cluster [DBG] pgmap v1384: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.450380+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.450380+0000 osd.6 (osd.6) 57 : cluster [DBG] 2.7 deep-scrub starts 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.453026+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:38:34.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:32.453026+0000 osd.6 (osd.6) 58 : cluster [DBG] 2.7 deep-scrub ok 2024-04-26T16:38:34.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.514608+0000 osd.2 (osd.2) 437 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:38:34.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.514608+0000 osd.2 (osd.2) 437 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:38:34.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.517026+0000 osd.2 (osd.2) 438 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:38:34.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:33 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.517026+0000 osd.2 (osd.2) 438 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:38:34.793 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:38:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:38:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.395135+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.395135+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.397911+0000 osd.1 (osd.1) 474 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.397911+0000 osd.1 (osd.1) 474 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.786897+0000 osd.4 (osd.4) 447 : cluster [DBG] 2.f scrub starts 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.786897+0000 osd.4 (osd.4) 447 : cluster [DBG] 2.f scrub starts 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.790013+0000 osd.4 (osd.4) 448 : cluster [DBG] 2.f scrub ok 2024-04-26T16:38:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:34 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.790013+0000 osd.4 (osd.4) 448 : cluster [DBG] 2.f scrub ok 2024-04-26T16:38:35.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.395135+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.395135+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.397911+0000 osd.1 (osd.1) 474 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.397911+0000 osd.1 (osd.1) 474 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.786897+0000 osd.4 (osd.4) 447 : cluster [DBG] 2.f scrub starts 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.786897+0000 osd.4 (osd.4) 447 : cluster [DBG] 2.f scrub starts 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.790013+0000 osd.4 (osd.4) 448 : cluster [DBG] 2.f scrub ok 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.790013+0000 osd.4 (osd.4) 448 : cluster [DBG] 2.f scrub ok 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.395135+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.395135+0000 osd.1 (osd.1) 473 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:38:35.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.397911+0000 osd.1 (osd.1) 474 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:38:35.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.397911+0000 osd.1 (osd.1) 474 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:38:35.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.786897+0000 osd.4 (osd.4) 447 : cluster [DBG] 2.f scrub starts 2024-04-26T16:38:35.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.786897+0000 osd.4 (osd.4) 447 : cluster [DBG] 2.f scrub starts 2024-04-26T16:38:35.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.790013+0000 osd.4 (osd.4) 448 : cluster [DBG] 2.f scrub ok 2024-04-26T16:38:35.135 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:34 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.790013+0000 osd.4 (osd.4) 448 : cluster [DBG] 2.f scrub ok 2024-04-26T16:38:35.516 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:38:35.616 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.986936+0000 mgr.y (mgr.24401) 810 : cluster [DBG] pgmap v1385: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:33.986936+0000 mgr.y (mgr.24401) 810 : cluster [DBG] pgmap v1385: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:34.827391+0000 osd.0 (osd.0) 507 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:34.827391+0000 osd.0 (osd.0) 507 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:34.832275+0000 osd.0 (osd.0) 508 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:34.832275+0000 osd.0 (osd.0) 508 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:35.396185+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:35.396185+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:35.398918+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:38:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:35 smithi190 bash[23868]: cluster 2024-04-26T16:38:35.398918+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:38:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.986936+0000 mgr.y (mgr.24401) 810 : cluster [DBG] pgmap v1385: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:33.986936+0000 mgr.y (mgr.24401) 810 : cluster [DBG] pgmap v1385: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:34.827391+0000 osd.0 (osd.0) 507 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:38:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:34.827391+0000 osd.0 (osd.0) 507 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:38:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:34.832275+0000 osd.0 (osd.0) 508 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:38:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:34.832275+0000 osd.0 (osd.0) 508 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:35.396185+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:35.396185+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:35.398918+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[29234]: cluster 2024-04-26T16:38:35.398918+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.986936+0000 mgr.y (mgr.24401) 810 : cluster [DBG] pgmap v1385: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:33.986936+0000 mgr.y (mgr.24401) 810 : cluster [DBG] pgmap v1385: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:34.827391+0000 osd.0 (osd.0) 507 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:34.827391+0000 osd.0 (osd.0) 507 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:34.832275+0000 osd.0 (osd.0) 508 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:34.832275+0000 osd.0 (osd.0) 508 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:35.396185+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:38:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:35.396185+0000 osd.1 (osd.1) 475 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:38:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:35.398918+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:38:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:35 smithi050 bash[21463]: cluster 2024-04-26T16:38:35.398918+0000 osd.1 (osd.1) 476 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:38:36.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:36 smithi190 bash[23868]: cluster 2024-04-26T16:38:36.361893+0000 osd.1 (osd.1) 477 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:38:36.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:36 smithi190 bash[23868]: cluster 2024-04-26T16:38:36.361893+0000 osd.1 (osd.1) 477 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:38:36.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:36 smithi190 bash[23868]: cluster 2024-04-26T16:38:36.366549+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:38:36.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:36 smithi190 bash[23868]: cluster 2024-04-26T16:38:36.366549+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:38:37.122 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[29234]: cluster 2024-04-26T16:38:36.361893+0000 osd.1 (osd.1) 477 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[29234]: cluster 2024-04-26T16:38:36.361893+0000 osd.1 (osd.1) 477 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[29234]: cluster 2024-04-26T16:38:36.366549+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[29234]: cluster 2024-04-26T16:38:36.366549+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[21463]: cluster 2024-04-26T16:38:36.361893+0000 osd.1 (osd.1) 477 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[21463]: cluster 2024-04-26T16:38:36.361893+0000 osd.1 (osd.1) 477 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[21463]: cluster 2024-04-26T16:38:36.366549+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:38:37.123 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:36 smithi050 bash[21463]: cluster 2024-04-26T16:38:36.366549+0000 osd.1 (osd.1) 478 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:38:37.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:38:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:35.987828+0000 mgr.y (mgr.24401) 811 : cluster [DBG] pgmap v1386: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:35.987828+0000 mgr.y (mgr.24401) 811 : cluster [DBG] pgmap v1386: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.079970+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.079970+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.082477+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.082477+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.385951+0000 osd.1 (osd.1) 479 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.385951+0000 osd.1 (osd.1) 479 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.388260+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.388260+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: audit 2024-04-26T16:38:37.397340+0000 mgr.y (mgr.24401) 812 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:37 smithi190 bash[23868]: audit 2024-04-26T16:38:37.397340+0000 mgr.y (mgr.24401) 812 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:35.987828+0000 mgr.y (mgr.24401) 811 : cluster [DBG] pgmap v1386: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:35.987828+0000 mgr.y (mgr.24401) 811 : cluster [DBG] pgmap v1386: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.079970+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:38:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.079970+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:38:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.082477+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.082477+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.385951+0000 osd.1 (osd.1) 479 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.385951+0000 osd.1 (osd.1) 479 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.388260+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.388260+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: audit 2024-04-26T16:38:37.397340+0000 mgr.y (mgr.24401) 812 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[21463]: audit 2024-04-26T16:38:37.397340+0000 mgr.y (mgr.24401) 812 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:35.987828+0000 mgr.y (mgr.24401) 811 : cluster [DBG] pgmap v1386: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:35.987828+0000 mgr.y (mgr.24401) 811 : cluster [DBG] pgmap v1386: 164 pgs: 164 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.079970+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:38:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.079970+0000 osd.5 (osd.5) 352 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.082477+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.082477+0000 osd.5 (osd.5) 353 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.385951+0000 osd.1 (osd.1) 479 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.385951+0000 osd.1 (osd.1) 479 : cluster [DBG] 6.4 deep-scrub starts 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.388260+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.388260+0000 osd.1 (osd.1) 480 : cluster [DBG] 6.4 deep-scrub ok 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: audit 2024-04-26T16:38:37.397340+0000 mgr.y (mgr.24401) 812 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:37 smithi050 bash[29234]: audit 2024-04-26T16:38:37.397340+0000 mgr.y (mgr.24401) 812 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:38.440 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:38:38.440 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-04-26T16:38:38.440 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-04-26T16:38:38.440 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-04-26T16:38:38.440 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:38:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.851237+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:38:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.851237+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:38:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.854838+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:38:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.854838+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.872700+0000 osd.2 (osd.2) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.872700+0000 osd.2 (osd.2) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.873107+0000 osd.2 (osd.2) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.873107+0000 osd.2 (osd.2) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.989556+0000 mgr.y (mgr.24401) 813 : cluster [DBG] pgmap v1387: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:37.989556+0000 mgr.y (mgr.24401) 813 : cluster [DBG] pgmap v1387: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.097846+0000 osd.5 (osd.5) 354 : cluster [DBG] 6.a scrub starts 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.097846+0000 osd.5 (osd.5) 354 : cluster [DBG] 6.a scrub starts 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.100030+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub ok 2024-04-26T16:38:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:38 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.100030+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub ok 2024-04-26T16:38:39.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-04-26T16:38:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.851237+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:38:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.851237+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:38:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.854838+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:38:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.854838+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:38:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.872700+0000 osd.2 (osd.2) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.872700+0000 osd.2 (osd.2) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.873107+0000 osd.2 (osd.2) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.873107+0000 osd.2 (osd.2) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.989556+0000 mgr.y (mgr.24401) 813 : cluster [DBG] pgmap v1387: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:37.989556+0000 mgr.y (mgr.24401) 813 : cluster [DBG] pgmap v1387: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.097846+0000 osd.5 (osd.5) 354 : cluster [DBG] 6.a scrub starts 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.097846+0000 osd.5 (osd.5) 354 : cluster [DBG] 6.a scrub starts 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.100030+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub ok 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.100030+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub ok 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.851237+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:38:39.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.851237+0000 osd.0 (osd.0) 509 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.854838+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.854838+0000 osd.0 (osd.0) 510 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.872700+0000 osd.2 (osd.2) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.872700+0000 osd.2 (osd.2) 439 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.873107+0000 osd.2 (osd.2) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.873107+0000 osd.2 (osd.2) 440 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.989556+0000 mgr.y (mgr.24401) 813 : cluster [DBG] pgmap v1387: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:37.989556+0000 mgr.y (mgr.24401) 813 : cluster [DBG] pgmap v1387: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.097846+0000 osd.5 (osd.5) 354 : cluster [DBG] 6.a scrub starts 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.097846+0000 osd.5 (osd.5) 354 : cluster [DBG] 6.a scrub starts 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.100030+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub ok 2024-04-26T16:38:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:38 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.100030+0000 osd.5 (osd.5) 355 : cluster [DBG] 6.a scrub ok 2024-04-26T16:38:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 -- 172.21.15.50:0/1586509209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef20103ea0 msgr2=0x7fef201102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 --2- 172.21.15.50:0/1586509209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef20103ea0 0x7fef201102c0 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7fef1800b0a0 tx=0x7fef1800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 -- 172.21.15.50:0/1586509209 shutdown_connections 2024-04-26T16:38:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 --2- 172.21.15.50:0/1586509209 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef20103ea0 0x7fef201102c0 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.362 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 --2- 172.21.15.50:0/1586509209 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20109480 0x7fef20109860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 --2- 172.21.15.50:0/1586509209 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef201034e0 0x7fef20103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 -- 172.21.15.50:0/1586509209 >> 172.21.15.50:0/1586509209 conn(0x7fef2007a5e0 msgr2=0x7fef2007a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 -- 172.21.15.50:0/1586509209 shutdown_connections 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.362+0000 7fef2597b700 1 -- 172.21.15.50:0/1586509209 wait complete. 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 librados: starting msgr at 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 librados: starting objecter 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 Processor -- start 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 -- start start 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 librados: setting wanted keys 2024-04-26T16:38:39.363 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 librados: calling monclient init 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef201034e0 0x7fef20199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 0x7fef20199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef20109480 0x7fef2019d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fef20112a70 con 0x7fef20103ea0 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fef201128f0 con 0x7fef20109480 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fef20112bf0 con 0x7fef201034e0 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 0x7fef20199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1e7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 0x7fef20199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:34914/0 (socket says 172.21.15.50:34914) 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1e7fc700 1 -- 172.21.15.50:0/3121062121 learned_addr learned my addr 172.21.15.50:0/3121062121 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef201034e0 0x7fef20199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1f7fe700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef20109480 0x7fef2019d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:39.365 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 -- 172.21.15.50:0/3121062121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef20109480 msgr2=0x7fef2019d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef20109480 0x7fef2019d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 -- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 msgr2=0x7fef20199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 0x7fef20199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 -- 172.21.15.50:0/3121062121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef2019e060 con 0x7fef201034e0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1e7fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 0x7fef20199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef1effd700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef201034e0 0x7fef20199050 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7fef1000e9f0 tx=0x7fef1000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef24979700 10 client.?.objecter ms_handle_connect 0x7fef201034e0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef24979700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1000cb70 con 0x7fef201034e0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fef1000ee90 con 0x7fef201034e0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 -- 172.21.15.50:0/3121062121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef2019e350 con 0x7fef201034e0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 10 client.41768.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 -- 172.21.15.50:0/3121062121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fef201a5bd0 con 0x7fef201034e0 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef2597b700 1 librados: init done 2024-04-26T16:38:39.366 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef10003b80 con 0x7fef201034e0 2024-04-26T16:38:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef04005120 con 0x7fef201034e0 2024-04-26T16:38:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.366+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fef10003d20 con 0x7fef201034e0 2024-04-26T16:38:39.367 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fef0c071f80 0x7fef0c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(879..879 src has 827..879) ==== 8707+0+0 (secure 0 0 0) 0x7fef100917f0 con 0x7fef201034e0 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 10 client.41768.objecter ms_dispatch 0x7fef20001550 osd_map(879..879 src has 827..879) 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 3 client.41768.objecter handle_osd_map got epochs [879,879] > 0 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 3 client.41768.objecter handle_osd_map decoding full epoch 879 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef1e7fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fef0c071f80 0x7fef0c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 20 client.41768.objecter dump_active .. 0 homeless 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef24979700 10 client.41768.objecter ms_handle_connect 0x7fef0c071f80 2024-04-26T16:38:39.368 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.370+0000 7fef1e7fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fef0c071f80 0x7fef0c074440 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 crypto rx=0x7fef2007ae50 tx=0x7fef08005f90 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:39.371 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.374+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fef1005d440 con 0x7fef201034e0 2024-04-26T16:38:39.590 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.594+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} v 0) -- 0x7fef04005410 con 0x7fef201034e0 2024-04-26T16:38:39.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.846+0000 7fef24979700 1 -- 172.21.15.50:0/3121062121 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]=0 set pool 1 pg_num to 11 v880) ==== 129+0+0 (secure 0 0 0) 0x7fef100610f0 con 0x7fef201034e0 2024-04-26T16:38:39.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 10 librados: watch_flush enter 2024-04-26T16:38:39.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 10 librados: watch_flush exit 2024-04-26T16:38:39.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 20 client.41768.objecter shutdown clearing up homeless session... 2024-04-26T16:38:39.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 10 client.41768.objecter successfully canceled tick 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fef0c071f80 msgr2=0x7fef0c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fef0c071f80 0x7fef0c074440 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 crypto rx=0x7fef2007ae50 tx=0x7fef08005f90 comp rx=0 tx=0).stop 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef201034e0 msgr2=0x7fef20199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef201034e0 0x7fef20199050 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7fef1000e9f0 tx=0x7fef1000ed00 comp rx=0 tx=0).stop 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 shutdown_connections 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fef0c071f80 0x7fef0c074440 unknown :-1 s=CLOSED pgs=3853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fef20109480 0x7fef2019d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fef201034e0 0x7fef20199050 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 --2- 172.21.15.50:0/3121062121 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fef20103ea0 0x7fef20199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 >> 172.21.15.50:0/3121062121 conn(0x7fef2007a5e0 msgr2=0x7fef20104f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 shutdown_connections 2024-04-26T16:38:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 -- 172.21.15.50:0/3121062121 wait complete. 2024-04-26T16:38:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:39.850+0000 7fef167fc700 1 librados: shutdown 2024-04-26T16:38:39.850 INFO:teuthology.orchestra.run.smithi050.stderr:set pool 1 pg_num to 11 2024-04-26T16:38:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.600876+0000 osd.2 (osd.2) 441 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:38:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.600876+0000 osd.2 (osd.2) 441 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:38:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.603296+0000 osd.2 (osd.2) 442 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:38:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.603296+0000 osd.2 (osd.2) 442 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:38:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.838595+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: cluster 2024-04-26T16:38:38.838595+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: audit 2024-04-26T16:38:39.596051+0000 mon.c (mon.2) 915 : audit [INF] from='client.? 172.21.15.50:0/3121062121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: audit 2024-04-26T16:38:39.596051+0000 mon.c (mon.2) 915 : audit [INF] from='client.? 172.21.15.50:0/3121062121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: audit 2024-04-26T16:38:39.596705+0000 mon.a (mon.0) 3965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[21463]: audit 2024-04-26T16:38:39.596705+0000 mon.a (mon.0) 3965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.600876+0000 osd.2 (osd.2) 441 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.600876+0000 osd.2 (osd.2) 441 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.603296+0000 osd.2 (osd.2) 442 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.603296+0000 osd.2 (osd.2) 442 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.838595+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: cluster 2024-04-26T16:38:38.838595+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-04-26T16:38:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: audit 2024-04-26T16:38:39.596051+0000 mon.c (mon.2) 915 : audit [INF] from='client.? 172.21.15.50:0/3121062121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: audit 2024-04-26T16:38:39.596051+0000 mon.c (mon.2) 915 : audit [INF] from='client.? 172.21.15.50:0/3121062121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: audit 2024-04-26T16:38:39.596705+0000 mon.a (mon.0) 3965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:39 smithi050 bash[29234]: audit 2024-04-26T16:38:39.596705+0000 mon.a (mon.0) 3965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.600876+0000 osd.2 (osd.2) 441 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.600876+0000 osd.2 (osd.2) 441 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.603296+0000 osd.2 (osd.2) 442 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.603296+0000 osd.2 (osd.2) 442 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.838595+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: cluster 2024-04-26T16:38:38.838595+0000 mon.a (mon.0) 3964 : cluster [DBG] osdmap e879: 8 total, 8 up, 8 in 2024-04-26T16:38:40.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: audit 2024-04-26T16:38:39.596051+0000 mon.c (mon.2) 915 : audit [INF] from='client.? 172.21.15.50:0/3121062121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: audit 2024-04-26T16:38:39.596051+0000 mon.c (mon.2) 915 : audit [INF] from='client.? 172.21.15.50:0/3121062121' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: audit 2024-04-26T16:38:39.596705+0000 mon.a (mon.0) 3965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:40.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:39 smithi190 bash[23868]: audit 2024-04-26T16:38:39.596705+0000 mon.a (mon.0) 3965 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.478584+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.478584+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.478868+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.478868+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.587401+0000 osd.2 (osd.2) 443 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.587401+0000 osd.2 (osd.2) 443 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.594207+0000 osd.2 (osd.2) 444 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.594207+0000 osd.2 (osd.2) 444 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: audit 2024-04-26T16:38:39.839324+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: audit 2024-04-26T16:38:39.839324+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-26T16:38:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.847526+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.847526+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.990274+0000 mgr.y (mgr.24401) 814 : cluster [DBG] pgmap v1390: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.990274+0000 mgr.y (mgr.24401) 814 : cluster [DBG] pgmap v1390: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: audit 2024-04-26T16:38:39.990810+0000 mon.c (mon.2) 916 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: audit 2024-04-26T16:38:39.990810+0000 mon.c (mon.2) 916 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: audit 2024-04-26T16:38:39.991611+0000 mon.a (mon.0) 3968 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: audit 2024-04-26T16:38:39.991611+0000 mon.a (mon.0) 3968 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.993781+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.993781+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.996232+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[21463]: cluster 2024-04-26T16:38:39.996232+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:38:41.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.478584+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.478584+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.478868+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.478868+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.587401+0000 osd.2 (osd.2) 443 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.587401+0000 osd.2 (osd.2) 443 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.594207+0000 osd.2 (osd.2) 444 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.594207+0000 osd.2 (osd.2) 444 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: audit 2024-04-26T16:38:39.839324+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: audit 2024-04-26T16:38:39.839324+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.847526+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.847526+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.990274+0000 mgr.y (mgr.24401) 814 : cluster [DBG] pgmap v1390: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.990274+0000 mgr.y (mgr.24401) 814 : cluster [DBG] pgmap v1390: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: audit 2024-04-26T16:38:39.990810+0000 mon.c (mon.2) 916 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: audit 2024-04-26T16:38:39.990810+0000 mon.c (mon.2) 916 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: audit 2024-04-26T16:38:39.991611+0000 mon.a (mon.0) 3968 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: audit 2024-04-26T16:38:39.991611+0000 mon.a (mon.0) 3968 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.993781+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.993781+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.996232+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:38:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:40 smithi050 bash[29234]: cluster 2024-04-26T16:38:39.996232+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.478584+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.478584+0000 osd.0 (osd.0) 511 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.478868+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.478868+0000 osd.0 (osd.0) 512 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.587401+0000 osd.2 (osd.2) 443 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.587401+0000 osd.2 (osd.2) 443 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.594207+0000 osd.2 (osd.2) 444 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.594207+0000 osd.2 (osd.2) 444 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: audit 2024-04-26T16:38:39.839324+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: audit 2024-04-26T16:38:39.839324+0000 mon.a (mon.0) 3966 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.847526+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-04-26T16:38:41.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.847526+0000 mon.a (mon.0) 3967 : cluster [DBG] osdmap e880: 8 total, 8 up, 8 in 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.990274+0000 mgr.y (mgr.24401) 814 : cluster [DBG] pgmap v1390: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.990274+0000 mgr.y (mgr.24401) 814 : cluster [DBG] pgmap v1390: 164 pgs: 164 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: audit 2024-04-26T16:38:39.990810+0000 mon.c (mon.2) 916 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: audit 2024-04-26T16:38:39.990810+0000 mon.c (mon.2) 916 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: audit 2024-04-26T16:38:39.991611+0000 mon.a (mon.0) 3968 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: audit 2024-04-26T16:38:39.991611+0000 mon.a (mon.0) 3968 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.993781+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.993781+0000 osd.7 (osd.7) 470 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.996232+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:38:41.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:40 smithi190 bash[23868]: cluster 2024-04-26T16:38:39.996232+0000 osd.7 (osd.7) 471 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:38:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[21463]: audit 2024-04-26T16:38:40.842096+0000 mon.a (mon.0) 3969 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:38:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[21463]: audit 2024-04-26T16:38:40.842096+0000 mon.a (mon.0) 3969 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:38:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[21463]: cluster 2024-04-26T16:38:40.852385+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-04-26T16:38:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[21463]: cluster 2024-04-26T16:38:40.852385+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-04-26T16:38:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[29234]: audit 2024-04-26T16:38:40.842096+0000 mon.a (mon.0) 3969 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:38:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[29234]: audit 2024-04-26T16:38:40.842096+0000 mon.a (mon.0) 3969 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:38:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[29234]: cluster 2024-04-26T16:38:40.852385+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-04-26T16:38:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:41 smithi050 bash[29234]: cluster 2024-04-26T16:38:40.852385+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-04-26T16:38:42.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:41 smithi190 bash[23868]: audit 2024-04-26T16:38:40.842096+0000 mon.a (mon.0) 3969 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:38:42.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:41 smithi190 bash[23868]: audit 2024-04-26T16:38:40.842096+0000 mon.a (mon.0) 3969 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-04-26T16:38:42.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:41 smithi190 bash[23868]: cluster 2024-04-26T16:38:40.852385+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-04-26T16:38:42.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:41 smithi190 bash[23868]: cluster 2024-04-26T16:38:40.852385+0000 mon.a (mon.0) 3970 : cluster [DBG] osdmap e881: 8 total, 8 up, 8 in 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.875499+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.875499+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.991198+0000 mgr.y (mgr.24401) 815 : cluster [DBG] pgmap v1393: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.991198+0000 mgr.y (mgr.24401) 815 : cluster [DBG] pgmap v1393: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: audit 2024-04-26T16:38:41.991901+0000 mon.c (mon.2) 917 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: audit 2024-04-26T16:38:41.991901+0000 mon.c (mon.2) 917 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.993469+0000 osd.7 (osd.7) 472 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.993469+0000 osd.7 (osd.7) 472 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: audit 2024-04-26T16:38:41.994132+0000 mon.a (mon.0) 3972 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: audit 2024-04-26T16:38:41.994132+0000 mon.a (mon.0) 3972 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.998059+0000 osd.7 (osd.7) 473 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:41.998059+0000 osd.7 (osd.7) 473 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:42.091671+0000 osd.5 (osd.5) 356 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:42.091671+0000 osd.5 (osd.5) 356 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:42.097113+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[21463]: cluster 2024-04-26T16:38:42.097113+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.875499+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.875499+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.991198+0000 mgr.y (mgr.24401) 815 : cluster [DBG] pgmap v1393: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.991198+0000 mgr.y (mgr.24401) 815 : cluster [DBG] pgmap v1393: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: audit 2024-04-26T16:38:41.991901+0000 mon.c (mon.2) 917 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: audit 2024-04-26T16:38:41.991901+0000 mon.c (mon.2) 917 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.993469+0000 osd.7 (osd.7) 472 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.993469+0000 osd.7 (osd.7) 472 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: audit 2024-04-26T16:38:41.994132+0000 mon.a (mon.0) 3972 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: audit 2024-04-26T16:38:41.994132+0000 mon.a (mon.0) 3972 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.998059+0000 osd.7 (osd.7) 473 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:41.998059+0000 osd.7 (osd.7) 473 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:42.091671+0000 osd.5 (osd.5) 356 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:42.091671+0000 osd.5 (osd.5) 356 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:42.097113+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:38:43.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:42 smithi050 bash[29234]: cluster 2024-04-26T16:38:42.097113+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:38:43.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.875499+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.875499+0000 mon.a (mon.0) 3971 : cluster [DBG] osdmap e882: 8 total, 8 up, 8 in 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.991198+0000 mgr.y (mgr.24401) 815 : cluster [DBG] pgmap v1393: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.991198+0000 mgr.y (mgr.24401) 815 : cluster [DBG] pgmap v1393: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: audit 2024-04-26T16:38:41.991901+0000 mon.c (mon.2) 917 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: audit 2024-04-26T16:38:41.991901+0000 mon.c (mon.2) 917 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.993469+0000 osd.7 (osd.7) 472 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.993469+0000 osd.7 (osd.7) 472 : cluster [DBG] 1.0 deep-scrub starts 2024-04-26T16:38:43.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: audit 2024-04-26T16:38:41.994132+0000 mon.a (mon.0) 3972 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: audit 2024-04-26T16:38:41.994132+0000 mon.a (mon.0) 3972 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.998059+0000 osd.7 (osd.7) 473 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:41.998059+0000 osd.7 (osd.7) 473 : cluster [DBG] 1.0 deep-scrub ok 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:42.091671+0000 osd.5 (osd.5) 356 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:42.091671+0000 osd.5 (osd.5) 356 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:42.097113+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:38:43.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:42 smithi190 bash[23868]: cluster 2024-04-26T16:38:42.097113+0000 osd.5 (osd.5) 357 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:38:43.916 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.b/config 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: audit 2024-04-26T16:38:42.877232+0000 mon.a (mon.0) 3973 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: audit 2024-04-26T16:38:42.877232+0000 mon.a (mon.0) 3973 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: cluster 2024-04-26T16:38:42.887970+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: cluster 2024-04-26T16:38:42.887970+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.107074+0000 osd.5 (osd.5) 358 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.107074+0000 osd.5 (osd.5) 358 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:38:44.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.109487+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:38:44.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.109487+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:38:44.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: audit 2024-04-26T16:38:43.169225+0000 mon.c (mon.2) 918 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:44.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:43 smithi190 bash[23868]: audit 2024-04-26T16:38:43.169225+0000 mon.c (mon.2) 918 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:44.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: audit 2024-04-26T16:38:42.877232+0000 mon.a (mon.0) 3973 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:38:44.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: audit 2024-04-26T16:38:42.877232+0000 mon.a (mon.0) 3973 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:38:44.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: cluster 2024-04-26T16:38:42.887970+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: cluster 2024-04-26T16:38:42.887970+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.107074+0000 osd.5 (osd.5) 358 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.107074+0000 osd.5 (osd.5) 358 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.109487+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.109487+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: audit 2024-04-26T16:38:43.169225+0000 mon.c (mon.2) 918 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[21463]: audit 2024-04-26T16:38:43.169225+0000 mon.c (mon.2) 918 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: audit 2024-04-26T16:38:42.877232+0000 mon.a (mon.0) 3973 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: audit 2024-04-26T16:38:42.877232+0000 mon.a (mon.0) 3973 : audit [INF] from='mgr.24401 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: cluster 2024-04-26T16:38:42.887970+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-04-26T16:38:44.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: cluster 2024-04-26T16:38:42.887970+0000 mon.a (mon.0) 3974 : cluster [DBG] osdmap e883: 8 total, 8 up, 8 in 2024-04-26T16:38:44.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.107074+0000 osd.5 (osd.5) 358 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:38:44.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.107074+0000 osd.5 (osd.5) 358 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:38:44.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.109487+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:38:44.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.109487+0000 osd.5 (osd.5) 359 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:38:44.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: audit 2024-04-26T16:38:43.169225+0000 mon.c (mon.2) 918 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:44.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:43 smithi050 bash[29234]: audit 2024-04-26T16:38:43.169225+0000 mon.c (mon.2) 918 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:44.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:38:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:38:45.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:44 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.887009+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-04-26T16:38:45.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:44 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.887009+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-04-26T16:38:45.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:44 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.992187+0000 mgr.y (mgr.24401) 816 : cluster [DBG] pgmap v1396: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:38:45.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:44 smithi190 bash[23868]: cluster 2024-04-26T16:38:43.992187+0000 mgr.y (mgr.24401) 816 : cluster [DBG] pgmap v1396: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:38:45.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:44 smithi190 bash[23868]: cluster 2024-04-26T16:38:44.890332+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-04-26T16:38:45.194 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:44 smithi190 bash[23868]: cluster 2024-04-26T16:38:44.890332+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-04-26T16:38:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.887009+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-04-26T16:38:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.887009+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-04-26T16:38:45.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.992187+0000 mgr.y (mgr.24401) 816 : cluster [DBG] pgmap v1396: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:38:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21463]: cluster 2024-04-26T16:38:43.992187+0000 mgr.y (mgr.24401) 816 : cluster [DBG] pgmap v1396: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:38:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21463]: cluster 2024-04-26T16:38:44.890332+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-04-26T16:38:45.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[21463]: cluster 2024-04-26T16:38:44.890332+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-04-26T16:38:45.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.887009+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-04-26T16:38:45.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.887009+0000 mon.a (mon.0) 3975 : cluster [DBG] osdmap e884: 8 total, 8 up, 8 in 2024-04-26T16:38:45.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.992187+0000 mgr.y (mgr.24401) 816 : cluster [DBG] pgmap v1396: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:38:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[29234]: cluster 2024-04-26T16:38:43.992187+0000 mgr.y (mgr.24401) 816 : cluster [DBG] pgmap v1396: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-26T16:38:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[29234]: cluster 2024-04-26T16:38:44.890332+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-04-26T16:38:45.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:44 smithi050 bash[29234]: cluster 2024-04-26T16:38:44.890332+0000 mon.a (mon.0) 3976 : cluster [DBG] osdmap e885: 8 total, 8 up, 8 in 2024-04-26T16:38:45.788 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-04-26T16:38:45.788 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-04-26T16:38:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:45 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.192479+0000 osd.5 (osd.5) 360 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:38:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:45 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.192479+0000 osd.5 (osd.5) 360 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:38:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:45 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.199925+0000 osd.5 (osd.5) 361 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:38:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:45 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.199925+0000 osd.5 (osd.5) 361 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:38:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.192479+0000 osd.5 (osd.5) 360 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:38:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.192479+0000 osd.5 (osd.5) 360 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:38:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.199925+0000 osd.5 (osd.5) 361 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:38:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.199925+0000 osd.5 (osd.5) 361 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:38:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.192479+0000 osd.5 (osd.5) 360 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:38:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.192479+0000 osd.5 (osd.5) 360 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:38:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.199925+0000 osd.5 (osd.5) 361 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:38:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:45 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.199925+0000 osd.5 (osd.5) 361 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:38:46.610 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-04-26T16:38:46.610 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3594 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94191258305536 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e746)", 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:24.918471+0000", 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 201.69468125200001, 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.49752239700000001, 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:46.611 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3594 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918471+0000", 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918471+0000", 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918473+0000", 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8840000000000001e-06 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918480+0000", 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7770000000000002e-06 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918493+0000", 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2112e-05 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918504+0000", 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.115e-05 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918575+0000", 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1373999999999998e-05 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:24.918586+0000", 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0910000000000001e-05 2024-04-26T16:38:46.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.404269+0000", 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48568317100000002 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.404449+0000", 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000179734 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415993+0000", 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011544285 2024-04-26T16:38:46.615 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(5.3 epoch 747/746 rep_tid 1377 entries 747'19 (0'0) error 5:c609908c:::notify.5:head by client.14391.0:3593 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:25.405710+0000", 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 201.207442544, 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047988400000000004, 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.405710+0000", 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.405710+0000", 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.405710+0000", 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.04e-07 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.405717+0000", 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9700000000000002e-06 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.405729+0000", 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2707999999999999e-05 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.405743+0000", 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3552e-05 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.406119+0000", 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00037608500000000002 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.410508+0000", 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0043893209999999998 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(5.9 epoch 747/746 rep_tid 2409 entries 747'16 (0'0) error 5:93e5b521:::notify.7:head by client.14391.0:3595 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:25.414128+0000", 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 201.19902381899999, 2024-04-26T16:38:46.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032123009999999999, 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414128+0000", 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414128+0000", 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414129+0000", 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2500000000000001e-07 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414135+0000", 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8449999999999997e-06 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414149+0000", 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3692e-05 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.414164+0000", 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4385e-05 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.416507+0000", 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023436429999999999 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.417341+0000", 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00083341100000000005 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.622 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.41513.0:1698 332.10 e747/746 332:0b536d6c:test-rados-api-smithi050-73372-129::foo:head v 747'2, mlcod=0'0)", 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:25.415838+0000", 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 201.19731438299999, 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0017677089999999999, 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415838+0000", 2024-04-26T16:38:46.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415838+0000", 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415838+0000", 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.63e-07 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415844+0000", 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4430000000000002e-06 2024-04-26T16:38:46.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415851+0000", 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0350000000000004e-06 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.415857+0000", 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6490000000000002e-06 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.416661+0000", 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00080355699999999999 2024-04-26T16:38:46.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.416690+0000", 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9226000000000001e-05 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.417564+0000", 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000873484 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:25.417605+0000", 2024-04-26T16:38:46.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1851999999999998e-05 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2180 334.8 334:10000000:%01:::head [pgnls start_epoch 770] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e770)", 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:50.455540+0000", 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 176.157612486, 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055825149999999997, 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.627 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2180 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455540+0000", 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455540+0000", 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455543+0000", 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5470000000000001e-06 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455564+0000", 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1016e-05 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455604+0000", 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9861999999999999e-05 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455632+0000", 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7576000000000002e-05 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455814+0000", 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000182652 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.455825+0000", 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0776e-05 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.460718+0000", 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048926159999999998 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.460809+0000", 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1514999999999994e-05 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.631 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:50.461122+0000", 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031295500000000001 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2214 334.8 334:10000000::::head [pgnls start_epoch 772] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e772)", 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:52.376862+0000", 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 174.23629023500001, 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025585260000000002, 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.632 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2214 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376862+0000", 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376862+0000", 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376864+0000", 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.17e-06 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376874+0000", 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0135999999999999e-05 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376896+0000", 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1413999999999999e-05 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376905+0000", 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.577e-06 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376981+0000", 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5545999999999998e-05 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.376987+0000", 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.852e-06 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.635 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.379178+0000", 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021913620000000001 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.379224+0000", 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6465000000000003e-05 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.379420+0000", 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000196004 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2276 334.d 334:b3765058:test-rados-api-smithi050-73372-133::foo:head [delete] snapc 0=[] ondisk+write+ignore_cache+known_if_redirected+supports_pool_eio e772)", 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:52.421596+0000", 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 174.19155635800001, 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001759477, 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2276 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.637 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421596+0000", 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421596+0000", 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421598+0000", 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6699999999999998e-06 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421612+0000", 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3349999999999999e-05 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421645+0000", 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3081999999999999e-05 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421663+0000", 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7839999999999999e-05 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421775+0000", 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112377 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.421937+0000", 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000162065 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-04-26T16:38:46.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.422071+0000", 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000133893 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.422590+0000", 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051865099999999996 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.423024+0000", 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043454399999999998 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-04-26T16:38:46.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.423151+0000", 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012638699999999999 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.423262+0000", 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011137300000000001 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:52.423355+0000", 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3245000000000006e-05 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.642 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(323.0s2 778/746 ECSubWrite(tid=1129, reqid=osd.7.69:192, at_version=0'0, trim_to=0'0, roll_forward_to=778'13))", 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:58.437245+0000", 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 168.17590722700001, 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016605369999999999, 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.437245+0000", 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.437245+0000", 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.437246+0000", 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4899999999999999e-06 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.437299+0000", 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2748999999999999e-05 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.437327+0000", 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7676000000000001e-05 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.437351+0000", 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3935999999999999e-05 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438107+0000", 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00075614300000000003 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438121+0000", 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3793e-05 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438828+0000", 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070734499999999996 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:58.438905+0000", 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7404999999999995e-05 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.646 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3665 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94191258305536 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e774)", 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:35:59.939638+0000", 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 166.67351386600001, 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001718628, 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3665 2024-04-26T16:38:46.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.939638+0000", 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.939638+0000", 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.939642+0000", 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8659999999999999e-06 2024-04-26T16:38:46.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.939696+0000", 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4342000000000002e-05 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.939754+0000", 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7828999999999998e-05 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.939782+0000", 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7529e-05 2024-04-26T16:38:46.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.940893+0000", 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001111131 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.941090+0000", 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019720499999999999 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:35:59.941357+0000", 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026672600000000001 2024-04-26T16:38:46.650 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2338 335.16 335:68000000::::head [pgnls start_epoch 780] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e780)", 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:00.462643+0000", 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 166.15050954500001, 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060816749999999999, 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.651 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2338 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462643+0000", 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462643+0000", 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462647+0000", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3470000000000001e-06 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462668+0000", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0735999999999999e-05 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462716+0000", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8769e-05 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462744+0000", 2024-04-26T16:38:46.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7495999999999999e-05 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462918+0000", 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017383500000000001 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.462929+0000", 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1736000000000001e-05 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.468443+0000", 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055131930000000004 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.468499+0000", 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6295000000000002e-05 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:00.468724+0000", 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022526800000000001 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.655 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDECSubOpWrite(323.3s1 786/746 ECSubWrite(tid=1552, reqid=osd.4.35:148, at_version=786'1, trim_to=0'0, roll_forward_to=0'0))", 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:06.452903+0000", 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 160.16024883, 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.01104339, 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.452903+0000", 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.656 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.452903+0000", 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.452904+0000", 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0299999999999998e-07 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458141+0000", 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052367109999999998 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.657 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458175+0000", 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4656000000000002e-05 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458204+0000", 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8762e-05 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458613+0000", 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00040894999999999999 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.658 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_started", 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.458635+0000", 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2388999999999999e-05 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_committed", 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.463866+0000", 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005230434 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:06.463947+0000", 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0785000000000007e-05 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.659 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2429 336.8 336:10000000::::head [pgnls start_epoch 788] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e788)", 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:08.441159+0000", 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 158.17199288399999, 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067891849999999997, 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2429 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.660 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441159+0000", 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441159+0000", 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441163+0000", 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7900000000000001e-06 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.661 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441183+0000", 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0253000000000001e-05 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441224+0000", 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0395999999999997e-05 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441251+0000", 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7651999999999999e-05 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.662 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441412+0000", 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016113299999999999 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.441424+0000", 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1902999999999999e-05 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.447492+0000", 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060675729999999997 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.663 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.447591+0000", 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8839999999999996e-05 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:08.447948+0000", 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035764500000000002 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(osd.4.0:1507 338.6 e805/798 338:60000000:.ceph-internal::hit_set_338.6_archive_2024-04-26 16%3a36%3a28.520426Z_2024-04-26 16%3a36%3a31.532821Z:head v 805'8, has_updated_hit_set_history, mlcod=805'6)", 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:31.533743+0000", 2024-04-26T16:38:46.664 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 135.07940930000001, 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001761023, 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533743+0000", 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533743+0000", 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.665 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533745+0000", 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2639999999999998e-06 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533764+0000", 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9063e-05 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533799+0000", 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4688999999999999e-05 2024-04-26T16:38:46.666 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533815+0000", 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6458999999999999e-05 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.533972+0000", 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015689600000000001 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.534051+0000", 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9307999999999994e-05 2024-04-26T16:38:46.667 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.535380+0000", 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00132836 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:31.535504+0000", 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123984 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.668 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(323.7s1,QREQUEST,e:808,from:5(0),reservation_nonce:3)", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:37.651325+0000", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 128.96182666600001, 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0067472490000000003, 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651325+0000", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651325+0000", 2024-04-26T16:38:46.669 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651327+0000", 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0229999999999999e-06 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651343+0000", 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5446000000000001e-05 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651374+0000", 2024-04-26T16:38:46.670 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1390000000000003e-05 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651389+0000", 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4902000000000001e-05 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651501+0000", 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011144799999999999 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.651513+0000", 2024-04-26T16:38:46.671 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2706e-05 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.657976+0000", 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006462293 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.658007+0000", 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1186000000000003e-05 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:37.658073+0000", 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5854999999999999e-05 2024-04-26T16:38:46.672 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:2655 338.0 338:00000000::::head [pgnls start_epoch 809] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e809)", 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:38.605702+0000", 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 128.00744988100001, 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057759600000000001, 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2655 2024-04-26T16:38:46.673 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.605702+0000", 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.605702+0000", 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.674 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.605706+0000", 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.07e-06 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.605730+0000", 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3711999999999999e-05 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.605784+0000", 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4388999999999997e-05 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.675 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.605815+0000", 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0226000000000002e-05 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.606020+0000", 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000205031 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.606037+0000", 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7496999999999999e-05 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.676 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.610929+0000", 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004891645 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.611028+0000", 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9178000000000005e-05 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.611478+0000", 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045021200000000002 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(osd.4.0:1517 338.6 e809/798 338:60000000:.ceph-internal::hit_set_338.6_archive_2024-04-26 16%3a36%3a34.549141Z_2024-04-26 16%3a36%3a37.562255Z:head v 809'22, has_updated_hit_set_history, mlcod=807'17)", 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:38.607162+0000", 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 128.005989871, 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014691470000000001, 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607162+0000", 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607162+0000", 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607163+0000", 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5300000000000004e-07 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607172+0000", 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0129999999999999e-06 2024-04-26T16:38:46.679 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607194+0000", 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.196e-05 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607213+0000", 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9466e-05 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607363+0000", 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015019999999999999 2024-04-26T16:38:46.680 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.607433+0000", 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9701000000000002e-05 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.608505+0000", 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010716969999999999 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:38.608631+0000", 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012645700000000001 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.681 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3756 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94191258305536 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:36:44.975763+0000", 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 121.637389093, 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016382230000000001, 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3756 2024-04-26T16:38:46.682 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.975763+0000", 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.975763+0000", 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.975766+0000", 2024-04-26T16:38:46.683 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1130000000000001e-06 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.975793+0000", 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7322e-05 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.975830+0000", 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7005999999999998e-05 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.975856+0000", 2024-04-26T16:38:46.684 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5389999999999999e-05 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.976781+0000", 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092568399999999997 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.977038+0000", 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000256607 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:36:44.977401+0000", 2024-04-26T16:38:46.685 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036310100000000001 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.41513.0:3367 341.f 341:f0000000:test-rados-api-smithi050-73372-147:::head [pgnls start_epoch 839] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e839)", 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:37:15.487314+0000", 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 91.125837927000006, 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041411140000000004, 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.41513", 2024-04-26T16:38:46.686 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2980882740", 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3367 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487314+0000", 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487314+0000", 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.687 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487317+0000", 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.23e-06 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487357+0000", 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0102999999999999e-05 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487396+0000", 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8695999999999997e-05 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.688 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487415+0000", 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8936e-05 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487553+0000", 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013747299999999999 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.487563+0000", 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.079e-05 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.689 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.491269+0000", 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003705294 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.491293+0000", 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4389e-05 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:15.491455+0000", 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016220300000000001 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14391.0:3838 5.1f 5:f95f44c2:::notify.0:head [watch ping cookie 94191258305536 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:37:25.007903+0000", 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 81.605249032000003, 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016319419999999999, 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14391", 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.50:0/2760378525", 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 3838 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.691 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.007903+0000", 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.007903+0000", 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.007905+0000", 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3949999999999999e-06 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.692 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.007917+0000", 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1929e-05 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.007954+0000", 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6808999999999999e-05 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.007971+0000", 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7244e-05 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.693 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.008933+0000", 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00096202500000000003 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.009174+0000", 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024090099999999999 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:37:25.009535+0000", 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000360639 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.694 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDScrubReserve(6.5,QREQUEST,e:879,from:7,reservation_nonce:5)", 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-04-26T16:38:39.990162+0000", 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 6.6229900940000004, 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001860663, 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "continuous": false, 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.990162+0000", 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.695 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.990162+0000", 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.990164+0000", 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9970000000000001e-06 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.990177+0000", 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3456e-05 2024-04-26T16:38:46.696 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.990207+0000", 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9303e-05 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.990225+0000", 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8612000000000001e-05 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.991885+0000", 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016594380000000001 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.697 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.991930+0000", 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5411999999999998e-05 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-04-26T16:38:39.992023+0000", 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2445e-05 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-04-26T16:38:46.698 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-04-26T16:38:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:46 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.919660+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-04-26T16:38:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:46 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.919660+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-04-26T16:38:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:46 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.992723+0000 mgr.y (mgr.24401) 817 : cluster [DBG] pgmap v1399: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:46 smithi190 bash[23868]: cluster 2024-04-26T16:38:45.992723+0000 mgr.y (mgr.24401) 817 : cluster [DBG] pgmap v1399: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:47.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.919660+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-04-26T16:38:47.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.919660+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-04-26T16:38:47.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.992723+0000 mgr.y (mgr.24401) 817 : cluster [DBG] pgmap v1399: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:47.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[21463]: cluster 2024-04-26T16:38:45.992723+0000 mgr.y (mgr.24401) 817 : cluster [DBG] pgmap v1399: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:47.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.919660+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-04-26T16:38:47.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.919660+0000 mon.a (mon.0) 3977 : cluster [DBG] osdmap e886: 8 total, 8 up, 8 in 2024-04-26T16:38:47.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.992723+0000 mgr.y (mgr.24401) 817 : cluster [DBG] pgmap v1399: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:47.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:46 smithi050 bash[29234]: cluster 2024-04-26T16:38:45.992723+0000 mgr.y (mgr.24401) 817 : cluster [DBG] pgmap v1399: 174 pgs: 1 peering, 10 unknown, 163 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:47.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:38:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:46.429429+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:46.429429+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:46.429975+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:46.429975+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:46.949800+0000 mon.a (mon.0) 3978 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:46.949800+0000 mon.a (mon.0) 3978 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.245009+0000 osd.5 (osd.5) 362 : cluster [DBG] 1.a scrub starts 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.245009+0000 osd.5 (osd.5) 362 : cluster [DBG] 1.a scrub starts 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.247455+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub ok 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.247455+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub ok 2024-04-26T16:38:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: audit 2024-04-26T16:38:47.405397+0000 mgr.y (mgr.24401) 818 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:48.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: audit 2024-04-26T16:38:47.405397+0000 mgr.y (mgr.24401) 818 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:48.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.467001+0000 osd.1 (osd.1) 481 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:38:48.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.467001+0000 osd.1 (osd.1) 481 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:38:48.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.470310+0000 osd.1 (osd.1) 482 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:38:48.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:47 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.470310+0000 osd.1 (osd.1) 482 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:46.429429+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:46.429429+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:46.429975+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:46.429975+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:46.949800+0000 mon.a (mon.0) 3978 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:46.949800+0000 mon.a (mon.0) 3978 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.245009+0000 osd.5 (osd.5) 362 : cluster [DBG] 1.a scrub starts 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.245009+0000 osd.5 (osd.5) 362 : cluster [DBG] 1.a scrub starts 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.247455+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub ok 2024-04-26T16:38:48.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.247455+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub ok 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: audit 2024-04-26T16:38:47.405397+0000 mgr.y (mgr.24401) 818 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: audit 2024-04-26T16:38:47.405397+0000 mgr.y (mgr.24401) 818 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.467001+0000 osd.1 (osd.1) 481 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.467001+0000 osd.1 (osd.1) 481 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.470310+0000 osd.1 (osd.1) 482 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.470310+0000 osd.1 (osd.1) 482 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:46.429429+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:46.429429+0000 osd.7 (osd.7) 474 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:46.429975+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:46.429975+0000 osd.7 (osd.7) 475 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:46.949800+0000 mon.a (mon.0) 3978 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-04-26T16:38:48.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:46.949800+0000 mon.a (mon.0) 3978 : cluster [DBG] osdmap e887: 8 total, 8 up, 8 in 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.245009+0000 osd.5 (osd.5) 362 : cluster [DBG] 1.a scrub starts 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.245009+0000 osd.5 (osd.5) 362 : cluster [DBG] 1.a scrub starts 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.247455+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub ok 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.247455+0000 osd.5 (osd.5) 363 : cluster [DBG] 1.a scrub ok 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: audit 2024-04-26T16:38:47.405397+0000 mgr.y (mgr.24401) 818 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: audit 2024-04-26T16:38:47.405397+0000 mgr.y (mgr.24401) 818 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.467001+0000 osd.1 (osd.1) 481 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.467001+0000 osd.1 (osd.1) 481 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.470310+0000 osd.1 (osd.1) 482 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:38:48.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:47 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.470310+0000 osd.1 (osd.1) 482 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:38:49.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.303339+0000 osd.3 (osd.3) 525 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.303339+0000 osd.3 (osd.3) 525 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.306773+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.306773+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.330729+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.330729+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.333094+0000 osd.6 (osd.6) 60 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.333094+0000 osd.6 (osd.6) 60 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.883049+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.883049+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.885781+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.885781+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:38:49.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.907437+0000 osd.4 (osd.4) 449 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:38:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.907437+0000 osd.4 (osd.4) 449 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:38:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.910190+0000 osd.4 (osd.4) 450 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:38:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.910190+0000 osd.4 (osd.4) 450 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:38:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.993428+0000 mgr.y (mgr.24401) 819 : cluster [DBG] pgmap v1401: 174 pgs: 8 peering, 2 unknown, 164 active+clean; 530 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T16:38:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:47.993428+0000 mgr.y (mgr.24401) 819 : cluster [DBG] pgmap v1401: 174 pgs: 8 peering, 2 unknown, 164 active+clean; 530 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T16:38:49.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.202612+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:38:49.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.202612+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:38:49.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.205579+0000 osd.5 (osd.5) 365 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:38:49.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:48 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.205579+0000 osd.5 (osd.5) 365 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.303339+0000 osd.3 (osd.3) 525 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.303339+0000 osd.3 (osd.3) 525 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.306773+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.306773+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.330729+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.330729+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:38:49.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.333094+0000 osd.6 (osd.6) 60 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.333094+0000 osd.6 (osd.6) 60 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.883049+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.883049+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.885781+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.885781+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.907437+0000 osd.4 (osd.4) 449 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.907437+0000 osd.4 (osd.4) 449 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.910190+0000 osd.4 (osd.4) 450 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.910190+0000 osd.4 (osd.4) 450 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.993428+0000 mgr.y (mgr.24401) 819 : cluster [DBG] pgmap v1401: 174 pgs: 8 peering, 2 unknown, 164 active+clean; 530 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T16:38:49.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:47.993428+0000 mgr.y (mgr.24401) 819 : cluster [DBG] pgmap v1401: 174 pgs: 8 peering, 2 unknown, 164 active+clean; 530 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.202612+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.202612+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.205579+0000 osd.5 (osd.5) 365 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.205579+0000 osd.5 (osd.5) 365 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.303339+0000 osd.3 (osd.3) 525 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.303339+0000 osd.3 (osd.3) 525 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.306773+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.306773+0000 osd.3 (osd.3) 526 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.330729+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.330729+0000 osd.6 (osd.6) 59 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.333094+0000 osd.6 (osd.6) 60 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:38:49.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.333094+0000 osd.6 (osd.6) 60 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.883049+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.883049+0000 osd.7 (osd.7) 476 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.885781+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.885781+0000 osd.7 (osd.7) 477 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.907437+0000 osd.4 (osd.4) 449 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.907437+0000 osd.4 (osd.4) 449 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.910190+0000 osd.4 (osd.4) 450 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.910190+0000 osd.4 (osd.4) 450 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.993428+0000 mgr.y (mgr.24401) 819 : cluster [DBG] pgmap v1401: 174 pgs: 8 peering, 2 unknown, 164 active+clean; 530 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:47.993428+0000 mgr.y (mgr.24401) 819 : cluster [DBG] pgmap v1401: 174 pgs: 8 peering, 2 unknown, 164 active+clean; 530 KiB data, 773 MiB used, 714 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.202612+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:38:49.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.202612+0000 osd.5 (osd.5) 364 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:38:49.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.205579+0000 osd.5 (osd.5) 365 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:38:49.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:48 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.205579+0000 osd.5 (osd.5) 365 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:38:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.373650+0000 osd.6 (osd.6) 61 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:38:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.373650+0000 osd.6 (osd.6) 61 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.376509+0000 osd.6 (osd.6) 62 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.376509+0000 osd.6 (osd.6) 62 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.862627+0000 osd.7 (osd.7) 478 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.862627+0000 osd.7 (osd.7) 478 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.865037+0000 osd.7 (osd.7) 479 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:48.865037+0000 osd.7 (osd.7) 479 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:49.236556+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:49.236556+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:49.240171+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:38:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:49 smithi190 bash[23868]: cluster 2024-04-26T16:38:49.240171+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:38:50.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.373650+0000 osd.6 (osd.6) 61 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:38:50.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.373650+0000 osd.6 (osd.6) 61 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:38:50.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.376509+0000 osd.6 (osd.6) 62 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:38:50.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.376509+0000 osd.6 (osd.6) 62 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.862627+0000 osd.7 (osd.7) 478 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.862627+0000 osd.7 (osd.7) 478 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.865037+0000 osd.7 (osd.7) 479 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:48.865037+0000 osd.7 (osd.7) 479 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:49.236556+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:49.236556+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:49.240171+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[29234]: cluster 2024-04-26T16:38:49.240171+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.373650+0000 osd.6 (osd.6) 61 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.373650+0000 osd.6 (osd.6) 61 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.376509+0000 osd.6 (osd.6) 62 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:38:50.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.376509+0000 osd.6 (osd.6) 62 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.862627+0000 osd.7 (osd.7) 478 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.862627+0000 osd.7 (osd.7) 478 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.865037+0000 osd.7 (osd.7) 479 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:48.865037+0000 osd.7 (osd.7) 479 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:49.236556+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:49.236556+0000 osd.5 (osd.5) 366 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:49.240171+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:38:50.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:49 smithi050 bash[21463]: cluster 2024-04-26T16:38:49.240171+0000 osd.5 (osd.5) 367 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:38:50.598 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:38:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:50 smithi190 bash[23868]: cluster 2024-04-26T16:38:49.994399+0000 mgr.y (mgr.24401) 820 : cluster [DBG] pgmap v1402: 174 pgs: 8 peering, 166 active+clean; 425 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s, 0 objects/s recovering 2024-04-26T16:38:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:50 smithi190 bash[23868]: cluster 2024-04-26T16:38:49.994399+0000 mgr.y (mgr.24401) 820 : cluster [DBG] pgmap v1402: 174 pgs: 8 peering, 166 active+clean; 425 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s, 0 objects/s recovering 2024-04-26T16:38:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:50 smithi190 bash[23868]: cluster 2024-04-26T16:38:50.262887+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:38:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:50 smithi190 bash[23868]: cluster 2024-04-26T16:38:50.262887+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:38:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:50 smithi190 bash[23868]: cluster 2024-04-26T16:38:50.265188+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:38:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:50 smithi190 bash[23868]: cluster 2024-04-26T16:38:50.265188+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[29234]: cluster 2024-04-26T16:38:49.994399+0000 mgr.y (mgr.24401) 820 : cluster [DBG] pgmap v1402: 174 pgs: 8 peering, 166 active+clean; 425 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s, 0 objects/s recovering 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[29234]: cluster 2024-04-26T16:38:49.994399+0000 mgr.y (mgr.24401) 820 : cluster [DBG] pgmap v1402: 174 pgs: 8 peering, 166 active+clean; 425 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s, 0 objects/s recovering 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[29234]: cluster 2024-04-26T16:38:50.262887+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[29234]: cluster 2024-04-26T16:38:50.262887+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[29234]: cluster 2024-04-26T16:38:50.265188+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[29234]: cluster 2024-04-26T16:38:50.265188+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:38:51.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[21463]: cluster 2024-04-26T16:38:49.994399+0000 mgr.y (mgr.24401) 820 : cluster [DBG] pgmap v1402: 174 pgs: 8 peering, 166 active+clean; 425 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s, 0 objects/s recovering 2024-04-26T16:38:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[21463]: cluster 2024-04-26T16:38:49.994399+0000 mgr.y (mgr.24401) 820 : cluster [DBG] pgmap v1402: 174 pgs: 8 peering, 166 active+clean; 425 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s, 0 objects/s recovering 2024-04-26T16:38:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[21463]: cluster 2024-04-26T16:38:50.262887+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:38:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[21463]: cluster 2024-04-26T16:38:50.262887+0000 osd.5 (osd.5) 368 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:38:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[21463]: cluster 2024-04-26T16:38:50.265188+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:38:51.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:50 smithi050 bash[21463]: cluster 2024-04-26T16:38:50.265188+0000 osd.5 (osd.5) 369 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:38:53.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:51.996207+0000 mgr.y (mgr.24401) 821 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:51.996207+0000 mgr.y (mgr.24401) 821 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.241932+0000 osd.3 (osd.3) 527 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.241932+0000 osd.3 (osd.3) 527 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.242253+0000 osd.3 (osd.3) 528 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.242253+0000 osd.3 (osd.3) 528 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.289907+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.289907+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.292021+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.292021+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:38:53.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.395159+0000 osd.1 (osd.1) 483 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:38:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.395159+0000 osd.1 (osd.1) 483 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:38:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.398253+0000 osd.1 (osd.1) 484 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:38:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.398253+0000 osd.1 (osd.1) 484 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:38:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.464848+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-04-26T16:38:53.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:53 smithi190 bash[23868]: cluster 2024-04-26T16:38:52.464848+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-04-26T16:38:53.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:51.996207+0000 mgr.y (mgr.24401) 821 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:53.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:51.996207+0000 mgr.y (mgr.24401) 821 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:53.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.241932+0000 osd.3 (osd.3) 527 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.241932+0000 osd.3 (osd.3) 527 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.242253+0000 osd.3 (osd.3) 528 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.242253+0000 osd.3 (osd.3) 528 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.289907+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.289907+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.292021+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.292021+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.395159+0000 osd.1 (osd.1) 483 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.395159+0000 osd.1 (osd.1) 483 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.398253+0000 osd.1 (osd.1) 484 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.398253+0000 osd.1 (osd.1) 484 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.464848+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-04-26T16:38:53.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[21463]: cluster 2024-04-26T16:38:52.464848+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:51.996207+0000 mgr.y (mgr.24401) 821 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:51.996207+0000 mgr.y (mgr.24401) 821 : cluster [DBG] pgmap v1403: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.241932+0000 osd.3 (osd.3) 527 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.241932+0000 osd.3 (osd.3) 527 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.242253+0000 osd.3 (osd.3) 528 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.242253+0000 osd.3 (osd.3) 528 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.289907+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.289907+0000 osd.5 (osd.5) 370 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.292021+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.292021+0000 osd.5 (osd.5) 371 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.395159+0000 osd.1 (osd.1) 483 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.395159+0000 osd.1 (osd.1) 483 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:38:53.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.398253+0000 osd.1 (osd.1) 484 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:38:53.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.398253+0000 osd.1 (osd.1) 484 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:38:53.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.464848+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-04-26T16:38:53.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:53 smithi050 bash[29234]: cluster 2024-04-26T16:38:52.464848+0000 mon.a (mon.0) 3979 : cluster [DBG] osdmap e888: 8 total, 8 up, 8 in 2024-04-26T16:38:54.151 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 -- 172.21.15.50:0/1927538848 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006bbf0 msgr2=0x7f3ad006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1927538848 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006bbf0 0x7f3ad006bfd0 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7f3ac000b0d0 tx=0x7f3ac000b3e0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 -- 172.21.15.50:0/1927538848 shutdown_connections 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1927538848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad0110000 0x7f3ad01123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1927538848 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006bbf0 0x7f3ad006bfd0 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1927538848 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad006c510 0x7f3ad010fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 -- 172.21.15.50:0/1927538848 >> 172.21.15.50:0/1927538848 conn(0x7f3ad0078230 msgr2=0x7f3ad0078640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 -- 172.21.15.50:0/1927538848 shutdown_connections 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.150+0000 7f3ad8193700 1 -- 172.21.15.50:0/1927538848 wait complete. 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 librados: starting msgr at 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 librados: starting objecter 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 Processor -- start 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- start start 2024-04-26T16:38:54.152 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 librados: setting wanted keys 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 librados: calling monclient init 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad006bbf0 0x7f3ad019b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006c510 0x7f3ad019b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 0x7f3ad019fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad0114e30 con 0x7f3ad0110000 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3ad0114cb0 con 0x7f3ad006bbf0 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3ad0114fb0 con 0x7f3ad006c510 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad6730700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 0x7f3ad019fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad6730700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 0x7f3ad019fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46540/0 (socket says 172.21.15.50:46540) 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad6730700 1 -- 172.21.15.50:0/1520437036 learned_addr learned my addr 172.21.15.50:0/1520437036 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006c510 0x7f3ad019b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad5f2f700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad006bbf0 0x7f3ad019b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 -- 172.21.15.50:0/1520437036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad006bbf0 msgr2=0x7f3ad019b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad006bbf0 0x7f3ad019b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 -- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 msgr2=0x7f3ad019fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 0x7f3ad019fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 -- 172.21.15.50:0/1520437036 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad01a0440 con 0x7f3ad006c510 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad5f2f700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad006bbf0 0x7f3ad019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad6730700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 0x7f3ad019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad572e700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006c510 0x7f3ad019b970 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 10 client.?.objecter ms_handle_connect 0x7f3ad006c510 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac400cbd0 con 0x7f3ad006c510 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3ac400cd70 con 0x7f3ad006c510 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ad01a0730 con 0x7f3ad006c510 2024-04-26T16:38:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ac4018420 con 0x7f3ad006c510 2024-04-26T16:38:54.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 10 client.41774.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ad01a7fb0 con 0x7f3ad006c510 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 librados: init done 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ad006a110 con 0x7f3ad006c510 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3ac4010430 con 0x7f3ad006c510 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.154+0000 7f3aceffd700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3ab8071f80 0x7f3ab8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(888..888 src has 827..888) ==== 8707+0+0 (secure 0 0 0) 0x7f3ac4091900 con 0x7f3ad006c510 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3aceffd700 10 client.41774.objecter ms_dispatch 0x7f3ad0001550 osd_map(888..888 src has 827..888) 2024-04-26T16:38:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3ad5f2f700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3ab8071f80 0x7f3ab8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:38:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3aceffd700 3 client.41774.objecter handle_osd_map got epochs [888,888] > 0 2024-04-26T16:38:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3aceffd700 3 client.41774.objecter handle_osd_map decoding full epoch 888 2024-04-26T16:38:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3aceffd700 20 client.41774.objecter dump_active .. 0 homeless 2024-04-26T16:38:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3aceffd700 10 client.41774.objecter ms_handle_connect 0x7f3ab8071f80 2024-04-26T16:38:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.158+0000 7f3ad5f2f700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3ab8071f80 0x7f3ab8074440 secure :-1 s=READY pgs=3854 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009e60 tx=0x7f3ac000b040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:38:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.162+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3ac405d550 con 0x7f3ad006c510 2024-04-26T16:38:54.382 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.386+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} v 0) -- 0x7f3ad019c790 con 0x7f3ad006c510 2024-04-26T16:38:54.476 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.478+0000 7f3aceffd700 1 -- 172.21.15.50:0/1520437036 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]=0 set pool 1 pgp_num to 11 v889) ==== 131+0+0 (secure 0 0 0) 0x7f3ac4061200 con 0x7f3ad006c510 2024-04-26T16:38:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 10 librados: watch_flush enter 2024-04-26T16:38:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 10 librados: watch_flush exit 2024-04-26T16:38:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 20 client.41774.objecter shutdown clearing up homeless session... 2024-04-26T16:38:54.480 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 10 client.41774.objecter successfully canceled tick 2024-04-26T16:38:54.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3ab8071f80 msgr2=0x7f3ab8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3ab8071f80 0x7f3ab8074440 secure :-1 s=READY pgs=3854 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009e60 tx=0x7f3ac000b040 comp rx=0 tx=0).stop 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006c510 msgr2=0x7f3ad019b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006c510 0x7f3ad019b970 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f3ac400e9f0 tx=0x7f3ac400ed00 comp rx=0 tx=0).stop 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 shutdown_connections 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3ab8071f80 0x7f3ab8074440 unknown :-1 s=CLOSED pgs=3854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ad006bbf0 0x7f3ad019b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3ad0110000 0x7f3ad019fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 --2- 172.21.15.50:0/1520437036 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3ad006c510 0x7f3ad019b970 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 >> 172.21.15.50:0/1520437036 conn(0x7f3ad0078230 msgr2=0x7f3ad0111820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 shutdown_connections 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 -- 172.21.15.50:0/1520437036 wait complete. 2024-04-26T16:38:54.482 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:38:54.482+0000 7f3ad8193700 1 librados: shutdown 2024-04-26T16:38:54.483 INFO:teuthology.orchestra.run.smithi050.stderr:set pool 1 pgp_num to 11 2024-04-26T16:38:54.773 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:38:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:38:54.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: cluster 2024-04-26T16:38:53.887711+0000 osd.7 (osd.7) 480 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:38:54.773 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: cluster 2024-04-26T16:38:53.887711+0000 osd.7 (osd.7) 480 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: cluster 2024-04-26T16:38:53.893761+0000 osd.7 (osd.7) 481 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: cluster 2024-04-26T16:38:53.893761+0000 osd.7 (osd.7) 481 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: audit 2024-04-26T16:38:54.387966+0000 mon.c (mon.2) 919 : audit [INF] from='client.? 172.21.15.50:0/1520437036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: audit 2024-04-26T16:38:54.387966+0000 mon.c (mon.2) 919 : audit [INF] from='client.? 172.21.15.50:0/1520437036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: audit 2024-04-26T16:38:54.388711+0000 mon.a (mon.0) 3980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[29234]: audit 2024-04-26T16:38:54.388711+0000 mon.a (mon.0) 3980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: cluster 2024-04-26T16:38:53.887711+0000 osd.7 (osd.7) 480 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: cluster 2024-04-26T16:38:53.887711+0000 osd.7 (osd.7) 480 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: cluster 2024-04-26T16:38:53.893761+0000 osd.7 (osd.7) 481 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: cluster 2024-04-26T16:38:53.893761+0000 osd.7 (osd.7) 481 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: audit 2024-04-26T16:38:54.387966+0000 mon.c (mon.2) 919 : audit [INF] from='client.? 172.21.15.50:0/1520437036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.774 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: audit 2024-04-26T16:38:54.387966+0000 mon.c (mon.2) 919 : audit [INF] from='client.? 172.21.15.50:0/1520437036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: audit 2024-04-26T16:38:54.388711+0000 mon.a (mon.0) 3980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:54 smithi050 bash[21463]: audit 2024-04-26T16:38:54.388711+0000 mon.a (mon.0) 3980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: cluster 2024-04-26T16:38:53.887711+0000 osd.7 (osd.7) 480 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:38:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: cluster 2024-04-26T16:38:53.887711+0000 osd.7 (osd.7) 480 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:38:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: cluster 2024-04-26T16:38:53.893761+0000 osd.7 (osd.7) 481 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:38:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: cluster 2024-04-26T16:38:53.893761+0000 osd.7 (osd.7) 481 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:38:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: audit 2024-04-26T16:38:54.387966+0000 mon.c (mon.2) 919 : audit [INF] from='client.? 172.21.15.50:0/1520437036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: audit 2024-04-26T16:38:54.387966+0000 mon.c (mon.2) 919 : audit [INF] from='client.? 172.21.15.50:0/1520437036' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: audit 2024-04-26T16:38:54.388711+0000 mon.a (mon.0) 3980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:54.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:54 smithi190 bash[23868]: audit 2024-04-26T16:38:54.388711+0000 mon.a (mon.0) 3980 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-04-26T16:38:55.369 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:53.996797+0000 mgr.y (mgr.24401) 822 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:53.996797+0000 mgr.y (mgr.24401) 822 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.255165+0000 osd.5 (osd.5) 372 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.255165+0000 osd.5 (osd.5) 372 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.258283+0000 osd.5 (osd.5) 373 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.258283+0000 osd.5 (osd.5) 373 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.366274+0000 osd.3 (osd.3) 529 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.366274+0000 osd.3 (osd.3) 529 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:38:55.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.368665+0000 osd.3 (osd.3) 530 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.368665+0000 osd.3 (osd.3) 530 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: audit 2024-04-26T16:38:54.468309+0000 mon.a (mon.0) 3981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: audit 2024-04-26T16:38:54.468309+0000 mon.a (mon.0) 3981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.478762+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.478762+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.881521+0000 osd.7 (osd.7) 482 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.881521+0000 osd.7 (osd.7) 482 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.885044+0000 osd.7 (osd.7) 483 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[21463]: cluster 2024-04-26T16:38:54.885044+0000 osd.7 (osd.7) 483 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:53.996797+0000 mgr.y (mgr.24401) 822 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:55.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:53.996797+0000 mgr.y (mgr.24401) 822 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.255165+0000 osd.5 (osd.5) 372 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.255165+0000 osd.5 (osd.5) 372 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.258283+0000 osd.5 (osd.5) 373 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.258283+0000 osd.5 (osd.5) 373 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.366274+0000 osd.3 (osd.3) 529 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.366274+0000 osd.3 (osd.3) 529 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.368665+0000 osd.3 (osd.3) 530 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.368665+0000 osd.3 (osd.3) 530 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: audit 2024-04-26T16:38:54.468309+0000 mon.a (mon.0) 3981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: audit 2024-04-26T16:38:54.468309+0000 mon.a (mon.0) 3981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.478762+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.478762+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-04-26T16:38:55.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.881521+0000 osd.7 (osd.7) 482 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:38:55.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.881521+0000 osd.7 (osd.7) 482 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:38:55.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.885044+0000 osd.7 (osd.7) 483 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:38:55.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:55 smithi050 bash[29234]: cluster 2024-04-26T16:38:54.885044+0000 osd.7 (osd.7) 483 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:38:55.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:53.996797+0000 mgr.y (mgr.24401) 822 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:53.996797+0000 mgr.y (mgr.24401) 822 : cluster [DBG] pgmap v1405: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.255165+0000 osd.5 (osd.5) 372 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.255165+0000 osd.5 (osd.5) 372 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.258283+0000 osd.5 (osd.5) 373 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.258283+0000 osd.5 (osd.5) 373 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.366274+0000 osd.3 (osd.3) 529 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.366274+0000 osd.3 (osd.3) 529 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.368665+0000 osd.3 (osd.3) 530 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.368665+0000 osd.3 (osd.3) 530 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:38:55.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: audit 2024-04-26T16:38:54.468309+0000 mon.a (mon.0) 3981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: audit 2024-04-26T16:38:54.468309+0000 mon.a (mon.0) 3981 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.478762+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.478762+0000 mon.a (mon.0) 3982 : cluster [DBG] osdmap e889: 8 total, 8 up, 8 in 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.881521+0000 osd.7 (osd.7) 482 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.881521+0000 osd.7 (osd.7) 482 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.885044+0000 osd.7 (osd.7) 483 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:38:55.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:55 smithi190 bash[23868]: cluster 2024-04-26T16:38:54.885044+0000 osd.7 (osd.7) 483 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:38:56.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.353378+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.353378+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.359656+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.359656+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.428632+0000 osd.2 (osd.2) 445 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.428632+0000 osd.2 (osd.2) 445 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.430553+0000 osd.2 (osd.2) 446 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.430553+0000 osd.2 (osd.2) 446 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.920252+0000 osd.7 (osd.7) 484 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.920252+0000 osd.7 (osd.7) 484 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.922766+0000 osd.7 (osd.7) 485 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.922766+0000 osd.7 (osd.7) 485 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.984719+0000 osd.4 (osd.4) 451 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:38:56.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.984719+0000 osd.4 (osd.4) 451 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:38:56.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.987283+0000 osd.4 (osd.4) 452 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:38:56.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:56 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.987283+0000 osd.4 (osd.4) 452 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.353378+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.353378+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.359656+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.359656+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.428632+0000 osd.2 (osd.2) 445 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.428632+0000 osd.2 (osd.2) 445 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.430553+0000 osd.2 (osd.2) 446 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.430553+0000 osd.2 (osd.2) 446 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:38:56.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.920252+0000 osd.7 (osd.7) 484 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.920252+0000 osd.7 (osd.7) 484 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.922766+0000 osd.7 (osd.7) 485 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.922766+0000 osd.7 (osd.7) 485 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.984719+0000 osd.4 (osd.4) 451 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.984719+0000 osd.4 (osd.4) 451 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.987283+0000 osd.4 (osd.4) 452 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.987283+0000 osd.4 (osd.4) 452 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:38:56.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.353378+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.353378+0000 osd.6 (osd.6) 63 : cluster [DBG] 3.a scrub starts 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.359656+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.359656+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.a scrub ok 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.428632+0000 osd.2 (osd.2) 445 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.428632+0000 osd.2 (osd.2) 445 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.430553+0000 osd.2 (osd.2) 446 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.430553+0000 osd.2 (osd.2) 446 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.920252+0000 osd.7 (osd.7) 484 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.920252+0000 osd.7 (osd.7) 484 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:38:56.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.922766+0000 osd.7 (osd.7) 485 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:38:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.922766+0000 osd.7 (osd.7) 485 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:38:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.984719+0000 osd.4 (osd.4) 451 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:38:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.984719+0000 osd.4 (osd.4) 451 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:38:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.987283+0000 osd.4 (osd.4) 452 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:38:56.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:56 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.987283+0000 osd.4 (osd.4) 452 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:38:57.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:38:57.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.997525+0000 mgr.y (mgr.24401) 823 : cluster [DBG] pgmap v1407: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[23868]: cluster 2024-04-26T16:38:55.997525+0000 mgr.y (mgr.24401) 823 : cluster [DBG] pgmap v1407: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[23868]: cluster 2024-04-26T16:38:56.472223+0000 osd.2 (osd.2) 447 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:38:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[23868]: cluster 2024-04-26T16:38:56.472223+0000 osd.2 (osd.2) 447 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:38:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[23868]: cluster 2024-04-26T16:38:56.480760+0000 osd.2 (osd.2) 448 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:38:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:57 smithi190 bash[23868]: cluster 2024-04-26T16:38:56.480760+0000 osd.2 (osd.2) 448 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.997525+0000 mgr.y (mgr.24401) 823 : cluster [DBG] pgmap v1407: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[29234]: cluster 2024-04-26T16:38:55.997525+0000 mgr.y (mgr.24401) 823 : cluster [DBG] pgmap v1407: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[29234]: cluster 2024-04-26T16:38:56.472223+0000 osd.2 (osd.2) 447 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[29234]: cluster 2024-04-26T16:38:56.472223+0000 osd.2 (osd.2) 447 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[29234]: cluster 2024-04-26T16:38:56.480760+0000 osd.2 (osd.2) 448 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[29234]: cluster 2024-04-26T16:38:56.480760+0000 osd.2 (osd.2) 448 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:38:57.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.997525+0000 mgr.y (mgr.24401) 823 : cluster [DBG] pgmap v1407: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[21463]: cluster 2024-04-26T16:38:55.997525+0000 mgr.y (mgr.24401) 823 : cluster [DBG] pgmap v1407: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-04-26T16:38:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[21463]: cluster 2024-04-26T16:38:56.472223+0000 osd.2 (osd.2) 447 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:38:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[21463]: cluster 2024-04-26T16:38:56.472223+0000 osd.2 (osd.2) 447 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:38:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[21463]: cluster 2024-04-26T16:38:56.480760+0000 osd.2 (osd.2) 448 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:38:57.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:57 smithi050 bash[21463]: cluster 2024-04-26T16:38:56.480760+0000 osd.2 (osd.2) 448 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:38:58.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.119469+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.119469+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.119921+0000 osd.5 (osd.5) 375 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.119921+0000 osd.5 (osd.5) 375 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: audit 2024-04-26T16:38:57.416726+0000 mgr.y (mgr.24401) 824 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: audit 2024-04-26T16:38:57.416726+0000 mgr.y (mgr.24401) 824 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.504262+0000 mon.a (mon.0) 3983 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.504262+0000 mon.a (mon.0) 3983 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.993746+0000 osd.4 (osd.4) 453 : cluster [DBG] 3.14 deep-scrub starts 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.993746+0000 osd.4 (osd.4) 453 : cluster [DBG] 3.14 deep-scrub starts 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:58.004194+0000 osd.4 (osd.4) 454 : cluster [DBG] 3.14 deep-scrub ok 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: cluster 2024-04-26T16:38:58.004194+0000 osd.4 (osd.4) 454 : cluster [DBG] 3.14 deep-scrub ok 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: audit 2024-04-26T16:38:58.202694+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:58.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:58 smithi190 bash[23868]: audit 2024-04-26T16:38:58.202694+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:58.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.119469+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:58.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.119469+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.119921+0000 osd.5 (osd.5) 375 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.119921+0000 osd.5 (osd.5) 375 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: audit 2024-04-26T16:38:57.416726+0000 mgr.y (mgr.24401) 824 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: audit 2024-04-26T16:38:57.416726+0000 mgr.y (mgr.24401) 824 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.504262+0000 mon.a (mon.0) 3983 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.504262+0000 mon.a (mon.0) 3983 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.993746+0000 osd.4 (osd.4) 453 : cluster [DBG] 3.14 deep-scrub starts 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.993746+0000 osd.4 (osd.4) 453 : cluster [DBG] 3.14 deep-scrub starts 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:58.004194+0000 osd.4 (osd.4) 454 : cluster [DBG] 3.14 deep-scrub ok 2024-04-26T16:38:58.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: cluster 2024-04-26T16:38:58.004194+0000 osd.4 (osd.4) 454 : cluster [DBG] 3.14 deep-scrub ok 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: audit 2024-04-26T16:38:58.202694+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[21463]: audit 2024-04-26T16:38:58.202694+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.119469+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.119469+0000 osd.5 (osd.5) 374 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.119921+0000 osd.5 (osd.5) 375 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.119921+0000 osd.5 (osd.5) 375 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: audit 2024-04-26T16:38:57.416726+0000 mgr.y (mgr.24401) 824 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: audit 2024-04-26T16:38:57.416726+0000 mgr.y (mgr.24401) 824 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.504262+0000 mon.a (mon.0) 3983 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-04-26T16:38:58.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.504262+0000 mon.a (mon.0) 3983 : cluster [DBG] osdmap e890: 8 total, 8 up, 8 in 2024-04-26T16:38:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.993746+0000 osd.4 (osd.4) 453 : cluster [DBG] 3.14 deep-scrub starts 2024-04-26T16:38:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.993746+0000 osd.4 (osd.4) 453 : cluster [DBG] 3.14 deep-scrub starts 2024-04-26T16:38:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:58.004194+0000 osd.4 (osd.4) 454 : cluster [DBG] 3.14 deep-scrub ok 2024-04-26T16:38:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: cluster 2024-04-26T16:38:58.004194+0000 osd.4 (osd.4) 454 : cluster [DBG] 3.14 deep-scrub ok 2024-04-26T16:38:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: audit 2024-04-26T16:38:58.202694+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:58.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:58 smithi050 bash[29234]: audit 2024-04-26T16:38:58.202694+0000 mon.c (mon.2) 920 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:38:59.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:59 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.998270+0000 mgr.y (mgr.24401) 825 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:59.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:59 smithi190 bash[23868]: cluster 2024-04-26T16:38:57.998270+0000 mgr.y (mgr.24401) 825 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:59.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:59 smithi190 bash[23868]: cluster 2024-04-26T16:38:58.338728+0000 osd.3 (osd.3) 531 : cluster [DBG] 6.b scrub starts 2024-04-26T16:38:59.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:59 smithi190 bash[23868]: cluster 2024-04-26T16:38:58.338728+0000 osd.3 (osd.3) 531 : cluster [DBG] 6.b scrub starts 2024-04-26T16:38:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:59 smithi190 bash[23868]: cluster 2024-04-26T16:38:58.341365+0000 osd.3 (osd.3) 532 : cluster [DBG] 6.b scrub ok 2024-04-26T16:38:59.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:38:59 smithi190 bash[23868]: cluster 2024-04-26T16:38:58.341365+0000 osd.3 (osd.3) 532 : cluster [DBG] 6.b scrub ok 2024-04-26T16:38:59.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.998270+0000 mgr.y (mgr.24401) 825 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:59.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[21463]: cluster 2024-04-26T16:38:57.998270+0000 mgr.y (mgr.24401) 825 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:59.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[21463]: cluster 2024-04-26T16:38:58.338728+0000 osd.3 (osd.3) 531 : cluster [DBG] 6.b scrub starts 2024-04-26T16:38:59.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[21463]: cluster 2024-04-26T16:38:58.338728+0000 osd.3 (osd.3) 531 : cluster [DBG] 6.b scrub starts 2024-04-26T16:38:59.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[21463]: cluster 2024-04-26T16:38:58.341365+0000 osd.3 (osd.3) 532 : cluster [DBG] 6.b scrub ok 2024-04-26T16:38:59.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[21463]: cluster 2024-04-26T16:38:58.341365+0000 osd.3 (osd.3) 532 : cluster [DBG] 6.b scrub ok 2024-04-26T16:38:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.998270+0000 mgr.y (mgr.24401) 825 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[29234]: cluster 2024-04-26T16:38:57.998270+0000 mgr.y (mgr.24401) 825 : cluster [DBG] pgmap v1409: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:38:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[29234]: cluster 2024-04-26T16:38:58.338728+0000 osd.3 (osd.3) 531 : cluster [DBG] 6.b scrub starts 2024-04-26T16:38:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[29234]: cluster 2024-04-26T16:38:58.338728+0000 osd.3 (osd.3) 531 : cluster [DBG] 6.b scrub starts 2024-04-26T16:38:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[29234]: cluster 2024-04-26T16:38:58.341365+0000 osd.3 (osd.3) 532 : cluster [DBG] 6.b scrub ok 2024-04-26T16:38:59.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:38:59 smithi050 bash[29234]: cluster 2024-04-26T16:38:58.341365+0000 osd.3 (osd.3) 532 : cluster [DBG] 6.b scrub ok 2024-04-26T16:39:00.162 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:39:01.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:59.966300+0000 osd.0 (osd.0) 513 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:39:01.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:59.966300+0000 osd.0 (osd.0) 513 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:59.969848+0000 osd.0 (osd.0) 514 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:59.969848+0000 osd.0 (osd.0) 514 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:59.999295+0000 mgr.y (mgr.24401) 826 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:38:59.999295+0000 mgr.y (mgr.24401) 826 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.290351+0000 osd.3 (osd.3) 533 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.290351+0000 osd.3 (osd.3) 533 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.292092+0000 osd.3 (osd.3) 534 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.292092+0000 osd.3 (osd.3) 534 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.359263+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.359263+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.361685+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:39:01.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[21463]: cluster 2024-04-26T16:39:00.361685+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:59.966300+0000 osd.0 (osd.0) 513 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:59.966300+0000 osd.0 (osd.0) 513 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:59.969848+0000 osd.0 (osd.0) 514 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:59.969848+0000 osd.0 (osd.0) 514 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:59.999295+0000 mgr.y (mgr.24401) 826 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:38:59.999295+0000 mgr.y (mgr.24401) 826 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.290351+0000 osd.3 (osd.3) 533 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.290351+0000 osd.3 (osd.3) 533 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.292092+0000 osd.3 (osd.3) 534 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.292092+0000 osd.3 (osd.3) 534 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.359263+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.359263+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:39:01.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.361685+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:39:01.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:01 smithi050 bash[29234]: cluster 2024-04-26T16:39:00.361685+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:39:01.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:59.966300+0000 osd.0 (osd.0) 513 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:39:01.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:59.966300+0000 osd.0 (osd.0) 513 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:39:01.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:59.969848+0000 osd.0 (osd.0) 514 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:59.969848+0000 osd.0 (osd.0) 514 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:59.999295+0000 mgr.y (mgr.24401) 826 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:38:59.999295+0000 mgr.y (mgr.24401) 826 : cluster [DBG] pgmap v1410: 174 pgs: 174 active+clean; 583 KiB data, 768 MiB used, 714 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.290351+0000 osd.3 (osd.3) 533 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.290351+0000 osd.3 (osd.3) 533 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.292092+0000 osd.3 (osd.3) 534 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.292092+0000 osd.3 (osd.3) 534 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.359263+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.359263+0000 osd.5 (osd.5) 376 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.361685+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:39:01.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:01 smithi190 bash[23868]: cluster 2024-04-26T16:39:00.361685+0000 osd.5 (osd.5) 377 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:39:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:01.470384+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:39:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:01.470384+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:39:02.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:01.473056+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:01.473056+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:02.067170+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.e scrub starts 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:02.067170+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.e scrub starts 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:02.070147+0000 osd.4 (osd.4) 456 : cluster [DBG] 5.e scrub ok 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[29234]: cluster 2024-04-26T16:39:02.070147+0000 osd.4 (osd.4) 456 : cluster [DBG] 5.e scrub ok 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:01.470384+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:01.470384+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:01.473056+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:01.473056+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:02.067170+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.e scrub starts 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:02.067170+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.e scrub starts 2024-04-26T16:39:02.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:02.070147+0000 osd.4 (osd.4) 456 : cluster [DBG] 5.e scrub ok 2024-04-26T16:39:02.632 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:02 smithi050 bash[21463]: cluster 2024-04-26T16:39:02.070147+0000 osd.4 (osd.4) 456 : cluster [DBG] 5.e scrub ok 2024-04-26T16:39:02.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:01.470384+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:39:02.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:01.470384+0000 osd.6 (osd.6) 65 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:39:02.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:01.473056+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:39:02.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:01.473056+0000 osd.6 (osd.6) 66 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:39:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:02.067170+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.e scrub starts 2024-04-26T16:39:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:02.067170+0000 osd.4 (osd.4) 455 : cluster [DBG] 5.e scrub starts 2024-04-26T16:39:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:02.070147+0000 osd.4 (osd.4) 456 : cluster [DBG] 5.e scrub ok 2024-04-26T16:39:02.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:02 smithi190 bash[23868]: cluster 2024-04-26T16:39:02.070147+0000 osd.4 (osd.4) 456 : cluster [DBG] 5.e scrub ok 2024-04-26T16:39:03.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- 172.21.15.50:0/2843004434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3878101a10 msgr2=0x7f3878101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- 172.21.15.50:0/2843004434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3878101a10 0x7f3878101e70 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7f386c009990 tx=0x7f386c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.496 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- 172.21.15.50:0/2843004434 shutdown_connections 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- 172.21.15.50:0/2843004434 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38781023b0 0x7f38781102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- 172.21.15.50:0/2843004434 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781094b0 0x7f3878109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- 172.21.15.50:0/2843004434 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3878101a10 0x7f3878101e70 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- 172.21.15.50:0/2843004434 >> 172.21.15.50:0/2843004434 conn(0x7f3878078990 msgr2=0x7f38780ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- 172.21.15.50:0/2843004434 shutdown_connections 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- 172.21.15.50:0/2843004434 wait complete. 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 librados: starting msgr at 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 librados: starting objecter 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 Processor -- start 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- start start 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 librados: setting wanted keys 2024-04-26T16:39:03.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 librados: calling monclient init 2024-04-26T16:39:03.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3878101a10 0x7f3878199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38781023b0 0x7f3878199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38781094b0 0x7f387819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3878112af0 con 0x7f38781023b0 2024-04-26T16:39:03.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3878112970 con 0x7f38781094b0 2024-04-26T16:39:03.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f387debb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3878112c70 con 0x7f3878101a10 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f38777fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3878101a10 0x7f3878199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38781023b0 0x7f3878199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f38777fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3878101a10 0x7f3878199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56738/0 (socket says 172.21.15.50:56738) 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f38777fe700 1 -- 172.21.15.50:0/91954250 learned_addr learned my addr 172.21.15.50:0/91954250 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38781094b0 0x7f387819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3878101a10 msgr2=0x7f3878199130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3878101a10 0x7f3878199130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38781094b0 msgr2=0x7f387819da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38781094b0 0x7f387819da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387819e140 con 0x7f38781023b0 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f38777fe700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3878101a10 0x7f3878199130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38781094b0 0x7f387819da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3876ffd700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38781023b0 0x7f3878199670 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f386c0059d0 tx=0x7f386c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3874ff9700 10 client.?.objecter ms_handle_connect 0x7f38781023b0 2024-04-26T16:39:03.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.497+0000 7f3874ff9700 10 client.?.objecter resend_mon_ops 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c027070 con 0x7f38781023b0 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f386c022470 con 0x7f38781023b0 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c017650 con 0x7f38781023b0 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387819e3d0 con 0x7f38781023b0 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 10 client.31161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38781a5cb0 con 0x7f38781023b0 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 1 librados: init done 2024-04-26T16:39:03.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3878104d30 con 0x7f38781023b0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f386c02a070 con 0x7f38781023b0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3860071f80 0x7f3860074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(890..890 src has 827..890) ==== 8707+0+0 (secure 0 0 0) 0x7f386c0931c0 con 0x7f38781023b0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 10 client.31161.objecter ms_dispatch 0x7f3878001550 osd_map(890..890 src has 827..890) 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 3 client.31161.objecter handle_osd_map got epochs [890,890] > 0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 3 client.31161.objecter handle_osd_map decoding full epoch 890 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f38777fe700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3860071f80 0x7f3860074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 20 client.31161.objecter dump_active .. 0 homeless 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7f386c031030 con 0x7f38781023b0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 0x7f387810a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f387819a490 con 0x7f387810a210 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f38777fe700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3860071f80 0x7f3860074440 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7f3868011550 tx=0x7f3868009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f3860071f80 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 0x7f387810a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 2: {"prefix": "get_command_descriptions"}) -- 0x7f387810ab30 con 0x7f387810a210 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 0x7f387810a5f0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3870008d30 tx=0x7f387000d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f387810a210 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 10 client.31161.objecter resend_mon_ops 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.501+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387000dd60 con 0x7f387810a210 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.505+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3870018070 con 0x7f387810a210 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.505+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 msgr2=0x7f387810a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.505+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 0x7f387810a5f0 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3870008d30 tx=0x7f387000d3c0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.505+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 msgr2=0x7f387810a5f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:39:03.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.505+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810a210 0x7f387810a5f0 secure :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f3870008d30 tx=0x7f387000d3c0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.509+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 0x7f38781a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.509+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f387810b120 con 0x7f387810ad40 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.509+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 0x7f38781a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- command(tid 3: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f38781aa3e0 con 0x7f387810ad40 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f387810ad40 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 10 client.31161.objecter resend_mon_ops 2024-04-26T16:39:03.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 0x7f38781a9ea0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3870008d30 tx=0x7f3870004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38700138f0 con 0x7f387810ad40 2024-04-26T16:39:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.0 v2:172.21.15.50:3300/0 2 ==== command_reply(tid 3: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f38700139b0 con 0x7f387810ad40 2024-04-26T16:39:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 msgr2=0x7f38781a9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 0x7f38781a9ea0 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3870008d30 tx=0x7f3870004ad0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 msgr2=0x7f38781a9ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:39:03.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f387810ad40 0x7f38781a9ea0 secure :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f3870008d30 tx=0x7f3870004ad0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.512 INFO:teuthology.orchestra.run.smithi050.stdout:mon.a: {} 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 0x7f387802d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f387802dc60 con 0x7f387802d310 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 0x7f387802d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 4: {"prefix": "get_command_descriptions"}) -- 0x7f387802de10 con 0x7f387802d310 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 0x7f387802d720 secure :-1 s=READY pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7f387810b2a0 tx=0x7f38700079e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f387802d310 2024-04-26T16:39:03.513 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.513+0000 7f3874ff9700 10 client.31161.objecter resend_mon_ops 2024-04-26T16:39:03.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870034030 con 0x7f387802d310 2024-04-26T16:39:03.514 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 4: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3870026440 con 0x7f387802d310 2024-04-26T16:39:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 msgr2=0x7f387802d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 0x7f387802d720 secure :-1 s=READY pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7f387810b2a0 tx=0x7f38700079e0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 msgr2=0x7f387802d720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:39:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f387802d310 0x7f387802d720 secure :-1 s=CLOSED pgs=4147 cs=0 l=1 rev1=1 crypto rx=0x7f387810b2a0 tx=0x7f38700079e0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.515 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.517+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 reap_dead start 2024-04-26T16:39:03.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 0x7f387819da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f38780757d0 con 0x7f3878199130 2024-04-26T16:39:03.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 0x7f387819da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- command(tid 5: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f38781a5440 con 0x7f3878199130 2024-04-26T16:39:03.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 0x7f387819da00 secure :-1 s=READY pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f387802dde0 tx=0x7f3870027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f3878199130 2024-04-26T16:39:03.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.525+0000 7f3874ff9700 10 client.31161.objecter resend_mon_ops 2024-04-26T16:39:03.526 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870034070 con 0x7f3878199130 2024-04-26T16:39:03.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.1 v2:172.21.15.190:3300/0 2 ==== command_reply(tid 5: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f3870026dc0 con 0x7f3878199130 2024-04-26T16:39:03.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 msgr2=0x7f387819da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 0x7f387819da00 secure :-1 s=READY pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f387802dde0 tx=0x7f3870027000 comp rx=0 tx=0).stop 2024-04-26T16:39:03.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 msgr2=0x7f387819da00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:39:03.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3878199130 0x7f387819da00 secure :-1 s=CLOSED pgs=4148 cs=0 l=1 rev1=1 crypto rx=0x7f387802dde0 tx=0x7f3870027000 comp rx=0 tx=0).stop 2024-04-26T16:39:03.529 INFO:teuthology.orchestra.run.smithi050.stdout:mon.b: {} 2024-04-26T16:39:03.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 0x7f387802e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f38781a5a30 con 0x7f38781a5650 2024-04-26T16:39:03.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.529+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 0x7f387802e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 6: {"prefix": "get_command_descriptions"}) -- 0x7f387802e560 con 0x7f38781a5650 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 0x7f387802e020 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f387000e010 tx=0x7f3870021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f38781a5650 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 10 client.31161.objecter resend_mon_ops 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870004bb0 con 0x7f38781a5650 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 6: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f38700265f0 con 0x7f38781a5650 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 msgr2=0x7f387802e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 0x7f387802e020 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f387000e010 tx=0x7f3870021000 comp rx=0 tx=0).stop 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 msgr2=0x7f387802e020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:39:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.533+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a5650 0x7f387802e020 secure :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f387000e010 tx=0x7f3870021000 comp rx=0 tx=0).stop 2024-04-26T16:39:03.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 0x7f38781a4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:03.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f38781a5a30 con 0x7f38781a3ca0 2024-04-26T16:39:03.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 0x7f38781a4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:03.538 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- command(tid 7: {"prefix": "injectargs", "injected_args": ["--mon_osd_down_out_interval=600"]}) -- 0x7f387802e770 con 0x7f38781a3ca0 2024-04-26T16:39:03.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3877fff700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 0x7f38781a4080 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f387001f010 tx=0x7f3870034000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:03.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 10 client.31161.objecter ms_handle_connect 0x7f38781a3ca0 2024-04-26T16:39:03.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 10 client.31161.objecter resend_mon_ops 2024-04-26T16:39:03.539 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870055870 con 0x7f38781a3ca0 2024-04-26T16:39:03.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 <== mon.2 v2:172.21.15.50:3301/0 2 ==== command_reply(tid 7: 0 mon_osd_down_out_interval = '' (not observed, change may require restart) ) ==== 82+0+3 (secure 0 0 0) 0x7f3870055670 con 0x7f38781a3ca0 2024-04-26T16:39:03.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 msgr2=0x7f38781a4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 0x7f38781a4080 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f387001f010 tx=0x7f3870034000 comp rx=0 tx=0).stop 2024-04-26T16:39:03.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 msgr2=0x7f38781a4080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T16:39:03.540 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3874ff9700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f38781a3ca0 0x7f38781a4080 secure :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f387001f010 tx=0x7f3870034000 comp rx=0 tx=0).stop 2024-04-26T16:39:03.540 INFO:teuthology.orchestra.run.smithi050.stdout:mon.c: {} 2024-04-26T16:39:03.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 10 librados: watch_flush enter 2024-04-26T16:39:03.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 10 librados: watch_flush exit 2024-04-26T16:39:03.541 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 20 client.31161.objecter shutdown clearing up homeless session... 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 10 client.31161.objecter successfully canceled tick 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3860071f80 msgr2=0x7f3860074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3860071f80 0x7f3860074440 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7f3868011550 tx=0x7f3868009260 comp rx=0 tx=0).stop 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38781023b0 msgr2=0x7f3878199670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 --2- 172.21.15.50:0/91954250 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f38781023b0 0x7f3878199670 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f386c0059d0 tx=0x7f386c0047b0 comp rx=0 tx=0).stop 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f3877fff700 1 -- 172.21.15.50:0/91954250 reap_dead start 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 shutdown_connections 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 >> 172.21.15.50:0/91954250 conn(0x7f3878078990 msgr2=0x7f3878105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 shutdown_connections 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 -- 172.21.15.50:0/91954250 wait complete. 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:03.541+0000 7f387debb700 1 librados: shutdown 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T16:39:03.542 INFO:teuthology.orchestra.run.smithi050.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-04-26T16:39:03.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[21463]: cluster 2024-04-26T16:39:02.001250+0000 mgr.y (mgr.24401) 827 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:03.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[21463]: cluster 2024-04-26T16:39:02.001250+0000 mgr.y (mgr.24401) 827 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:03.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[21463]: audit 2024-04-26T16:39:03.504511+0000 mon.a (mon.0) 3984 : audit [DBG] from='client.? 172.21.15.50:0/91954250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:39:03.568 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[21463]: audit 2024-04-26T16:39:03.504511+0000 mon.a (mon.0) 3984 : audit [DBG] from='client.? 172.21.15.50:0/91954250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:39:03.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[29234]: cluster 2024-04-26T16:39:02.001250+0000 mgr.y (mgr.24401) 827 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:03.568 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[29234]: cluster 2024-04-26T16:39:02.001250+0000 mgr.y (mgr.24401) 827 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:03.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[29234]: audit 2024-04-26T16:39:03.504511+0000 mon.a (mon.0) 3984 : audit [DBG] from='client.? 172.21.15.50:0/91954250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:39:03.569 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:03 smithi050 bash[29234]: audit 2024-04-26T16:39:03.504511+0000 mon.a (mon.0) 3984 : audit [DBG] from='client.? 172.21.15.50:0/91954250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:39:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:03 smithi190 bash[23868]: cluster 2024-04-26T16:39:02.001250+0000 mgr.y (mgr.24401) 827 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:03 smithi190 bash[23868]: cluster 2024-04-26T16:39:02.001250+0000 mgr.y (mgr.24401) 827 : cluster [DBG] pgmap v1411: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:03 smithi190 bash[23868]: audit 2024-04-26T16:39:03.504511+0000 mon.a (mon.0) 3984 : audit [DBG] from='client.? 172.21.15.50:0/91954250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:39:03.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:03 smithi190 bash[23868]: audit 2024-04-26T16:39:03.504511+0000 mon.a (mon.0) 3984 : audit [DBG] from='client.? 172.21.15.50:0/91954250' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-26T16:39:04.430 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 0 1 2024-04-26T16:39:04.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:39:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:39:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:39:05.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:05 smithi190 bash[23868]: cluster 2024-04-26T16:39:04.002273+0000 mgr.y (mgr.24401) 828 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:39:05.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:05 smithi190 bash[23868]: cluster 2024-04-26T16:39:04.002273+0000 mgr.y (mgr.24401) 828 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:39:05.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:05 smithi190 bash[23868]: cluster 2024-04-26T16:39:04.997950+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:39:05.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:05 smithi190 bash[23868]: cluster 2024-04-26T16:39:04.997950+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:39:05.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:05 smithi190 bash[23868]: cluster 2024-04-26T16:39:05.000218+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:39:05.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:05 smithi190 bash[23868]: cluster 2024-04-26T16:39:05.000218+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:39:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[21463]: cluster 2024-04-26T16:39:04.002273+0000 mgr.y (mgr.24401) 828 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:39:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[21463]: cluster 2024-04-26T16:39:04.002273+0000 mgr.y (mgr.24401) 828 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:39:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[21463]: cluster 2024-04-26T16:39:04.997950+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:39:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[21463]: cluster 2024-04-26T16:39:04.997950+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:39:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[21463]: cluster 2024-04-26T16:39:05.000218+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:39:05.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[21463]: cluster 2024-04-26T16:39:05.000218+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:39:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[29234]: cluster 2024-04-26T16:39:04.002273+0000 mgr.y (mgr.24401) 828 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:39:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[29234]: cluster 2024-04-26T16:39:04.002273+0000 mgr.y (mgr.24401) 828 : cluster [DBG] pgmap v1412: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:39:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[29234]: cluster 2024-04-26T16:39:04.997950+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:39:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[29234]: cluster 2024-04-26T16:39:04.997950+0000 osd.7 (osd.7) 486 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:39:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[29234]: cluster 2024-04-26T16:39:05.000218+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:39:05.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:05 smithi050 bash[29234]: cluster 2024-04-26T16:39:05.000218+0000 osd.7 (osd.7) 487 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:39:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:06 smithi190 bash[23868]: cluster 2024-04-26T16:39:05.235357+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:39:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:06 smithi190 bash[23868]: cluster 2024-04-26T16:39:05.235357+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:39:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:06 smithi190 bash[23868]: cluster 2024-04-26T16:39:05.239784+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:39:06.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:06 smithi190 bash[23868]: cluster 2024-04-26T16:39:05.239784+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[21463]: cluster 2024-04-26T16:39:05.235357+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[21463]: cluster 2024-04-26T16:39:05.235357+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[21463]: cluster 2024-04-26T16:39:05.239784+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[21463]: cluster 2024-04-26T16:39:05.239784+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[29234]: cluster 2024-04-26T16:39:05.235357+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[29234]: cluster 2024-04-26T16:39:05.235357+0000 osd.5 (osd.5) 378 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:39:06.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[29234]: cluster 2024-04-26T16:39:05.239784+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:39:06.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:06 smithi050 bash[29234]: cluster 2024-04-26T16:39:05.239784+0000 osd.5 (osd.5) 379 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:39:07.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:39:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[23868]: cluster 2024-04-26T16:39:06.003053+0000 mgr.y (mgr.24401) 829 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[23868]: cluster 2024-04-26T16:39:06.003053+0000 mgr.y (mgr.24401) 829 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[23868]: cluster 2024-04-26T16:39:06.910233+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:39:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[23868]: cluster 2024-04-26T16:39:06.910233+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:39:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[23868]: cluster 2024-04-26T16:39:06.913824+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:39:07.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:07 smithi190 bash[23868]: cluster 2024-04-26T16:39:06.913824+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:39:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[21463]: cluster 2024-04-26T16:39:06.003053+0000 mgr.y (mgr.24401) 829 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[21463]: cluster 2024-04-26T16:39:06.003053+0000 mgr.y (mgr.24401) 829 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[21463]: cluster 2024-04-26T16:39:06.910233+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:39:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[21463]: cluster 2024-04-26T16:39:06.910233+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:39:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[21463]: cluster 2024-04-26T16:39:06.913824+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:39:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[21463]: cluster 2024-04-26T16:39:06.913824+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:39:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[29234]: cluster 2024-04-26T16:39:06.003053+0000 mgr.y (mgr.24401) 829 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[29234]: cluster 2024-04-26T16:39:06.003053+0000 mgr.y (mgr.24401) 829 : cluster [DBG] pgmap v1413: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[29234]: cluster 2024-04-26T16:39:06.910233+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:39:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[29234]: cluster 2024-04-26T16:39:06.910233+0000 osd.0 (osd.0) 515 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:39:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[29234]: cluster 2024-04-26T16:39:06.913824+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:39:07.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:07 smithi050 bash[29234]: cluster 2024-04-26T16:39:06.913824+0000 osd.0 (osd.0) 516 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:39:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.201383+0000 osd.3 (osd.3) 535 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:39:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.201383+0000 osd.3 (osd.3) 535 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:39:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.203564+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.203564+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: audit 2024-04-26T16:39:07.428163+0000 mgr.y (mgr.24401) 830 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: audit 2024-04-26T16:39:07.428163+0000 mgr.y (mgr.24401) 830 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.477476+0000 osd.2 (osd.2) 449 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.477476+0000 osd.2 (osd.2) 449 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.480083+0000 osd.2 (osd.2) 450 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.480083+0000 osd.2 (osd.2) 450 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.940040+0000 osd.4 (osd.4) 457 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.940040+0000 osd.4 (osd.4) 457 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.942321+0000 osd.4 (osd.4) 458 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.942321+0000 osd.4 (osd.4) 458 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.974324+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.974324+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:39:08.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.976761+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:39:08.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:08 smithi190 bash[23868]: cluster 2024-04-26T16:39:07.976761+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.201383+0000 osd.3 (osd.3) 535 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.201383+0000 osd.3 (osd.3) 535 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.203564+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.203564+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: audit 2024-04-26T16:39:07.428163+0000 mgr.y (mgr.24401) 830 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: audit 2024-04-26T16:39:07.428163+0000 mgr.y (mgr.24401) 830 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.477476+0000 osd.2 (osd.2) 449 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.477476+0000 osd.2 (osd.2) 449 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.480083+0000 osd.2 (osd.2) 450 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:39:08.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.480083+0000 osd.2 (osd.2) 450 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.940040+0000 osd.4 (osd.4) 457 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.940040+0000 osd.4 (osd.4) 457 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.942321+0000 osd.4 (osd.4) 458 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.942321+0000 osd.4 (osd.4) 458 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.974324+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.974324+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.976761+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[21463]: cluster 2024-04-26T16:39:07.976761+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.201383+0000 osd.3 (osd.3) 535 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:39:08.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.201383+0000 osd.3 (osd.3) 535 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.203564+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.203564+0000 osd.3 (osd.3) 536 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: audit 2024-04-26T16:39:07.428163+0000 mgr.y (mgr.24401) 830 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: audit 2024-04-26T16:39:07.428163+0000 mgr.y (mgr.24401) 830 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.477476+0000 osd.2 (osd.2) 449 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.477476+0000 osd.2 (osd.2) 449 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.480083+0000 osd.2 (osd.2) 450 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.480083+0000 osd.2 (osd.2) 450 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.940040+0000 osd.4 (osd.4) 457 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.940040+0000 osd.4 (osd.4) 457 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.942321+0000 osd.4 (osd.4) 458 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.942321+0000 osd.4 (osd.4) 458 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.974324+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.974324+0000 osd.7 (osd.7) 488 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:39:08.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.976761+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:39:08.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:08 smithi050 bash[29234]: cluster 2024-04-26T16:39:07.976761+0000 osd.7 (osd.7) 489 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:39:09.228 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:39:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:09 smithi190 bash[23868]: cluster 2024-04-26T16:39:08.005008+0000 mgr.y (mgr.24401) 831 : cluster [DBG] pgmap v1414: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-26T16:39:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:09 smithi190 bash[23868]: cluster 2024-04-26T16:39:08.005008+0000 mgr.y (mgr.24401) 831 : cluster [DBG] pgmap v1414: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-26T16:39:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:09 smithi050 bash[21463]: cluster 2024-04-26T16:39:08.005008+0000 mgr.y (mgr.24401) 831 : cluster [DBG] pgmap v1414: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-26T16:39:09.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:09 smithi050 bash[21463]: cluster 2024-04-26T16:39:08.005008+0000 mgr.y (mgr.24401) 831 : cluster [DBG] pgmap v1414: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-26T16:39:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:09 smithi050 bash[29234]: cluster 2024-04-26T16:39:08.005008+0000 mgr.y (mgr.24401) 831 : cluster [DBG] pgmap v1414: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-26T16:39:09.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:09 smithi050 bash[29234]: cluster 2024-04-26T16:39:08.005008+0000 mgr.y (mgr.24401) 831 : cluster [DBG] pgmap v1414: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:09.350665+0000 osd.1 (osd.1) 485 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:09.350665+0000 osd.1 (osd.1) 485 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:09.352412+0000 osd.1 (osd.1) 486 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:09.352412+0000 osd.1 (osd.1) 486 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.120688+0000 osd.5 (osd.5) 380 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.120688+0000 osd.5 (osd.5) 380 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:39:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.124229+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:39:10.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:10 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.124229+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:39:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:09.350665+0000 osd.1 (osd.1) 485 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:39:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:09.350665+0000 osd.1 (osd.1) 485 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:39:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:09.352412+0000 osd.1 (osd.1) 486 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:39:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:09.352412+0000 osd.1 (osd.1) 486 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:39:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.120688+0000 osd.5 (osd.5) 380 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:39:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.120688+0000 osd.5 (osd.5) 380 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.124229+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.124229+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:09.350665+0000 osd.1 (osd.1) 485 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:09.350665+0000 osd.1 (osd.1) 485 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:09.352412+0000 osd.1 (osd.1) 486 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:09.352412+0000 osd.1 (osd.1) 486 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.120688+0000 osd.5 (osd.5) 380 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.120688+0000 osd.5 (osd.5) 380 : cluster [DBG] 5.15 deep-scrub starts 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.124229+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:39:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:10 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.124229+0000 osd.5 (osd.5) 381 : cluster [DBG] 5.15 deep-scrub ok 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.006016+0000 mgr.y (mgr.24401) 832 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.006016+0000 mgr.y (mgr.24401) 832 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.456145+0000 osd.6 (osd.6) 67 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.456145+0000 osd.6 (osd.6) 67 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.458536+0000 osd.6 (osd.6) 68 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.458536+0000 osd.6 (osd.6) 68 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:39:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.914512+0000 osd.7 (osd.7) 490 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:39:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.914512+0000 osd.7 (osd.7) 490 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:39:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.917131+0000 osd.7 (osd.7) 491 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:39:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:11 smithi190 bash[23868]: cluster 2024-04-26T16:39:10.917131+0000 osd.7 (osd.7) 491 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.006016+0000 mgr.y (mgr.24401) 832 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.006016+0000 mgr.y (mgr.24401) 832 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.456145+0000 osd.6 (osd.6) 67 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.456145+0000 osd.6 (osd.6) 67 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.458536+0000 osd.6 (osd.6) 68 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.458536+0000 osd.6 (osd.6) 68 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:39:11.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.914512+0000 osd.7 (osd.7) 490 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.914512+0000 osd.7 (osd.7) 490 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.917131+0000 osd.7 (osd.7) 491 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[29234]: cluster 2024-04-26T16:39:10.917131+0000 osd.7 (osd.7) 491 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.006016+0000 mgr.y (mgr.24401) 832 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.006016+0000 mgr.y (mgr.24401) 832 : cluster [DBG] pgmap v1415: 174 pgs: 174 active+clean; 583 KiB data, 772 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.456145+0000 osd.6 (osd.6) 67 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.456145+0000 osd.6 (osd.6) 67 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.458536+0000 osd.6 (osd.6) 68 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:39:11.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.458536+0000 osd.6 (osd.6) 68 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:39:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.914512+0000 osd.7 (osd.7) 490 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:39:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.914512+0000 osd.7 (osd.7) 490 : cluster [DBG] 11.1e deep-scrub starts 2024-04-26T16:39:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.917131+0000 osd.7 (osd.7) 491 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:39:11.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:11 smithi050 bash[21463]: cluster 2024-04-26T16:39:10.917131+0000 osd.7 (osd.7) 491 : cluster [DBG] 11.1e deep-scrub ok 2024-04-26T16:39:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.242301+0000 osd.3 (osd.3) 537 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:39:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.242301+0000 osd.3 (osd.3) 537 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:39:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.244701+0000 osd.3 (osd.3) 538 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:39:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.244701+0000 osd.3 (osd.3) 538 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:39:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.407466+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:39:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.407466+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:39:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.409711+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:39:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:12 smithi190 bash[23868]: cluster 2024-04-26T16:39:11.409711+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.242301+0000 osd.3 (osd.3) 537 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.242301+0000 osd.3 (osd.3) 537 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.244701+0000 osd.3 (osd.3) 538 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.244701+0000 osd.3 (osd.3) 538 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.407466+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.407466+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.409711+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:39:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[29234]: cluster 2024-04-26T16:39:11.409711+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.242301+0000 osd.3 (osd.3) 537 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.242301+0000 osd.3 (osd.3) 537 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.244701+0000 osd.3 (osd.3) 538 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.244701+0000 osd.3 (osd.3) 538 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.407466+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.407466+0000 osd.6 (osd.6) 69 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.409711+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:39:12.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:12 smithi050 bash[21463]: cluster 2024-04-26T16:39:11.409711+0000 osd.6 (osd.6) 70 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/1065910928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 msgr2=0x7f850010ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- 172.21.15.50:0/1065910928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850010ba20 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f84fc009990 tx=0x7f84fc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/1065910928 shutdown_connections 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- 172.21.15.50:0/1065910928 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 0x7f8500112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- 172.21.15.50:0/1065910928 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850010ba20 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- 172.21.15.50:0/1065910928 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8500105640 0x7f8500105ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/1065910928 >> 172.21.15.50:0/1065910928 conn(0x7f8500101130 msgr2=0x7f8500103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:12.928 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/1065910928 shutdown_connections 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/1065910928 wait complete. 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 librados: starting msgr at 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 librados: starting objecter 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 Processor -- start 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- start start 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 librados: setting wanted keys 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 librados: calling monclient init 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8500105640 0x7f850019b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 0x7f850019b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:12.929 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850019fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8500114d40 con 0x7f850010b640 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8500114bc0 con 0x7f8500106000 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8500114ec0 con 0x7f8500105640 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850019fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850019fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:45498/0 (socket says 172.21.15.50:45498) 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 -- 172.21.15.50:0/2102945466 learned_addr learned my addr 172.21.15.50:0/2102945466 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8505b6c700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8500105640 0x7f850019b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850536b700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 0x7f850019b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 -- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8500105640 msgr2=0x7f850019b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8500105640 0x7f850019b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:12.930 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 -- 172.21.15.50:0/2102945466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 msgr2=0x7f850019b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 0x7f850019b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 -- 172.21.15.50:0/2102945466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85001a0280 con 0x7f850010b640 2024-04-26T16:39:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850536b700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 0x7f850019b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:39:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f850636d700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850019fb40 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f84f8007d70 tx=0x7f84f800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f84f2ffd700 10 client.?.objecter ms_handle_connect 0x7f850010b640 2024-04-26T16:39:12.931 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f84f2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8016020 con 0x7f850010b640 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f84f800ad40 con 0x7f850010b640 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85001a0570 con 0x7f850010b640 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f8011380 con 0x7f850010b640 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 10 client.31173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85001a7df0 con 0x7f850010b640 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 librados: init done 2024-04-26T16:39:12.933 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.929+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850010dbd0 con 0x7f850010b640 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f84f8019020 con 0x7f850010b640 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f84ec071f80 0x7f84ec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(890..890 src has 827..890) ==== 8707+0+0 (secure 0 0 0) 0x7f84f80916c0 con 0x7f850010b640 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 10 client.31173.objecter ms_dispatch 0x7f8500001550 osd_map(890..890 src has 827..890) 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 3 client.31173.objecter handle_osd_map got epochs [890,890] > 0 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 3 client.31173.objecter handle_osd_map decoding full epoch 890 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f8505b6c700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f84ec071f80 0x7f84ec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 20 client.31173.objecter dump_active .. 0 homeless 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f84f2ffd700 10 client.31173.objecter ms_handle_connect 0x7f84ec071f80 2024-04-26T16:39:12.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.933+0000 7f8505b6c700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f84ec071f80 0x7f84ec074440 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00b2e0 tx=0x7f84fc005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:12.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:12.937+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f84f8020020 con 0x7f850010b640 2024-04-26T16:39:13.176 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.177+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 0, "weight": 1.0} v 0) -- 0x7f850019c5d0 con 0x7f850010b640 2024-04-26T16:39:13.566 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.569+0000 7f84f2ffd700 1 -- 172.21.15.50:0/2102945466 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 0, "weight": 1.0}]=0 reweighted osd.0 to 1 (10000) v891) ==== 113+0+0 (secure 0 0 0) 0x7f84f805d310 con 0x7f850010b640 2024-04-26T16:39:13.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 10 librados: watch_flush enter 2024-04-26T16:39:13.574 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 10 librados: watch_flush exit 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 20 client.31173.objecter shutdown clearing up homeless session... 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 10 client.31173.objecter successfully canceled tick 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f84ec071f80 msgr2=0x7f84ec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f84ec071f80 0x7f84ec074440 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 crypto rx=0x7f84fc00b2e0 tx=0x7f84fc005960 comp rx=0 tx=0).stop 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 msgr2=0x7f850019fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850019fb40 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f84f8007d70 tx=0x7f84f800a340 comp rx=0 tx=0).stop 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 shutdown_connections 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f84ec071f80 0x7f84ec074440 unknown :-1 s=CLOSED pgs=3856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f850010b640 0x7f850019fb40 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8500105640 0x7f850019b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 --2- 172.21.15.50:0/2102945466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8500106000 0x7f850019b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 >> 172.21.15.50:0/2102945466 conn(0x7f8500101130 msgr2=0x7f8500101a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 shutdown_connections 2024-04-26T16:39:13.575 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 -- 172.21.15.50:0/2102945466 wait complete. 2024-04-26T16:39:13.576 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:13.573+0000 7f8507dd0700 1 librados: shutdown 2024-04-26T16:39:13.576 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.0 to 1 (10000) 2024-04-26T16:39:13.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.007965+0000 mgr.y (mgr.24401) 833 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:13.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.007965+0000 mgr.y (mgr.24401) 833 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:13.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.262869+0000 osd.3 (osd.3) 539 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:39:13.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.262869+0000 osd.3 (osd.3) 539 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:39:13.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.265432+0000 osd.3 (osd.3) 540 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:39:13.854 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.265432+0000 osd.3 (osd.3) 540 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.908057+0000 osd.4 (osd.4) 459 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.908057+0000 osd.4 (osd.4) 459 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.913898+0000 osd.4 (osd.4) 460 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: cluster 2024-04-26T16:39:12.913898+0000 osd.4 (osd.4) 460 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: audit 2024-04-26T16:39:13.181516+0000 mon.a (mon.0) 3985 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: audit 2024-04-26T16:39:13.181516+0000 mon.a (mon.0) 3985 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: audit 2024-04-26T16:39:13.236538+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[21463]: audit 2024-04-26T16:39:13.236538+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.007965+0000 mgr.y (mgr.24401) 833 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.007965+0000 mgr.y (mgr.24401) 833 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:13.855 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.262869+0000 osd.3 (osd.3) 539 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.262869+0000 osd.3 (osd.3) 539 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.265432+0000 osd.3 (osd.3) 540 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.265432+0000 osd.3 (osd.3) 540 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.908057+0000 osd.4 (osd.4) 459 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.908057+0000 osd.4 (osd.4) 459 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.913898+0000 osd.4 (osd.4) 460 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: cluster 2024-04-26T16:39:12.913898+0000 osd.4 (osd.4) 460 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: audit 2024-04-26T16:39:13.181516+0000 mon.a (mon.0) 3985 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: audit 2024-04-26T16:39:13.181516+0000 mon.a (mon.0) 3985 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: audit 2024-04-26T16:39:13.236538+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:13.856 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:13 smithi050 bash[29234]: audit 2024-04-26T16:39:13.236538+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:14.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.007965+0000 mgr.y (mgr.24401) 833 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:14.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.007965+0000 mgr.y (mgr.24401) 833 : cluster [DBG] pgmap v1416: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:14.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.262869+0000 osd.3 (osd.3) 539 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:39:14.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.262869+0000 osd.3 (osd.3) 539 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.265432+0000 osd.3 (osd.3) 540 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.265432+0000 osd.3 (osd.3) 540 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.908057+0000 osd.4 (osd.4) 459 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.908057+0000 osd.4 (osd.4) 459 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.913898+0000 osd.4 (osd.4) 460 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: cluster 2024-04-26T16:39:12.913898+0000 osd.4 (osd.4) 460 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: audit 2024-04-26T16:39:13.181516+0000 mon.a (mon.0) 3985 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: audit 2024-04-26T16:39:13.181516+0000 mon.a (mon.0) 3985 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: audit 2024-04-26T16:39:13.236538+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:14.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:13 smithi190 bash[23868]: audit 2024-04-26T16:39:13.236538+0000 mon.c (mon.2) 921 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:14.477 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 0 1 2024-04-26T16:39:14.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:39:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:39:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.230164+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:39:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.230164+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:39:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.233049+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:39:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.233049+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:39:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: audit 2024-04-26T16:39:13.570898+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:14.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: audit 2024-04-26T16:39:13.570898+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.577861+0000 mon.a (mon.0) 3987 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.577861+0000 mon.a (mon.0) 3987 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.906016+0000 osd.7 (osd.7) 492 : cluster [DBG] 2.b scrub starts 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.906016+0000 osd.7 (osd.7) 492 : cluster [DBG] 2.b scrub starts 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.910786+0000 osd.7 (osd.7) 493 : cluster [DBG] 2.b scrub ok 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.910786+0000 osd.7 (osd.7) 493 : cluster [DBG] 2.b scrub ok 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.925560+0000 osd.4 (osd.4) 461 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.925560+0000 osd.4 (osd.4) 461 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.928203+0000 osd.4 (osd.4) 462 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[21463]: cluster 2024-04-26T16:39:13.928203+0000 osd.4 (osd.4) 462 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:39:14.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.230164+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.230164+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.233049+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.233049+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: audit 2024-04-26T16:39:13.570898+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: audit 2024-04-26T16:39:13.570898+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.577861+0000 mon.a (mon.0) 3987 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.577861+0000 mon.a (mon.0) 3987 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.906016+0000 osd.7 (osd.7) 492 : cluster [DBG] 2.b scrub starts 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.906016+0000 osd.7 (osd.7) 492 : cluster [DBG] 2.b scrub starts 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.910786+0000 osd.7 (osd.7) 493 : cluster [DBG] 2.b scrub ok 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.910786+0000 osd.7 (osd.7) 493 : cluster [DBG] 2.b scrub ok 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.925560+0000 osd.4 (osd.4) 461 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.925560+0000 osd.4 (osd.4) 461 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:39:14.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.928203+0000 osd.4 (osd.4) 462 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:39:14.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:14 smithi050 bash[29234]: cluster 2024-04-26T16:39:13.928203+0000 osd.4 (osd.4) 462 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:39:15.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.230164+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:39:15.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.230164+0000 osd.3 (osd.3) 541 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:39:15.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.233049+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.233049+0000 osd.3 (osd.3) 542 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: audit 2024-04-26T16:39:13.570898+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: audit 2024-04-26T16:39:13.570898+0000 mon.a (mon.0) 3986 : audit [INF] from='client.? 172.21.15.50:0/2102945466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.577861+0000 mon.a (mon.0) 3987 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.577861+0000 mon.a (mon.0) 3987 : cluster [DBG] osdmap e891: 8 total, 8 up, 8 in 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.906016+0000 osd.7 (osd.7) 492 : cluster [DBG] 2.b scrub starts 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.906016+0000 osd.7 (osd.7) 492 : cluster [DBG] 2.b scrub starts 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.910786+0000 osd.7 (osd.7) 493 : cluster [DBG] 2.b scrub ok 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.910786+0000 osd.7 (osd.7) 493 : cluster [DBG] 2.b scrub ok 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.925560+0000 osd.4 (osd.4) 461 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.925560+0000 osd.4 (osd.4) 461 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.928203+0000 osd.4 (osd.4) 462 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:39:15.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:14 smithi190 bash[23868]: cluster 2024-04-26T16:39:13.928203+0000 osd.4 (osd.4) 462 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:39:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[21463]: cluster 2024-04-26T16:39:14.008768+0000 mgr.y (mgr.24401) 834 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[21463]: cluster 2024-04-26T16:39:14.008768+0000 mgr.y (mgr.24401) 834 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[21463]: cluster 2024-04-26T16:39:14.914593+0000 osd.7 (osd.7) 494 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:39:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[21463]: cluster 2024-04-26T16:39:14.914593+0000 osd.7 (osd.7) 494 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:39:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[21463]: cluster 2024-04-26T16:39:14.920194+0000 osd.7 (osd.7) 495 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:39:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[21463]: cluster 2024-04-26T16:39:14.920194+0000 osd.7 (osd.7) 495 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:39:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[29234]: cluster 2024-04-26T16:39:14.008768+0000 mgr.y (mgr.24401) 834 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[29234]: cluster 2024-04-26T16:39:14.008768+0000 mgr.y (mgr.24401) 834 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[29234]: cluster 2024-04-26T16:39:14.914593+0000 osd.7 (osd.7) 494 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:39:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[29234]: cluster 2024-04-26T16:39:14.914593+0000 osd.7 (osd.7) 494 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:39:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[29234]: cluster 2024-04-26T16:39:14.920194+0000 osd.7 (osd.7) 495 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:39:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:15 smithi050 bash[29234]: cluster 2024-04-26T16:39:14.920194+0000 osd.7 (osd.7) 495 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:39:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:15 smithi190 bash[23868]: cluster 2024-04-26T16:39:14.008768+0000 mgr.y (mgr.24401) 834 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:15 smithi190 bash[23868]: cluster 2024-04-26T16:39:14.008768+0000 mgr.y (mgr.24401) 834 : cluster [DBG] pgmap v1418: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:15 smithi190 bash[23868]: cluster 2024-04-26T16:39:14.914593+0000 osd.7 (osd.7) 494 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:39:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:15 smithi190 bash[23868]: cluster 2024-04-26T16:39:14.914593+0000 osd.7 (osd.7) 494 : cluster [DBG] 3.d deep-scrub starts 2024-04-26T16:39:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:15 smithi190 bash[23868]: cluster 2024-04-26T16:39:14.920194+0000 osd.7 (osd.7) 495 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:39:16.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:15 smithi190 bash[23868]: cluster 2024-04-26T16:39:14.920194+0000 osd.7 (osd.7) 495 : cluster [DBG] 3.d deep-scrub ok 2024-04-26T16:39:16.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.213345+0000 osd.3 (osd.3) 543 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.213345+0000 osd.3 (osd.3) 543 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.216221+0000 osd.3 (osd.3) 544 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.216221+0000 osd.3 (osd.3) 544 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.380806+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.380806+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.383223+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.383223+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.875282+0000 osd.7 (osd.7) 496 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.875282+0000 osd.7 (osd.7) 496 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.877812+0000 osd.7 (osd.7) 497 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:15.877812+0000 osd.7 (osd.7) 497 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:39:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.570474+0000 osd.2 (osd.2) 451 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.570474+0000 osd.2 (osd.2) 451 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.573895+0000 osd.2 (osd.2) 452 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.573895+0000 osd.2 (osd.2) 452 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.213345+0000 osd.3 (osd.3) 543 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.213345+0000 osd.3 (osd.3) 543 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.216221+0000 osd.3 (osd.3) 544 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.216221+0000 osd.3 (osd.3) 544 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.380806+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.380806+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.383223+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.383223+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:39:16.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.875282+0000 osd.7 (osd.7) 496 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.875282+0000 osd.7 (osd.7) 496 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.877812+0000 osd.7 (osd.7) 497 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:15.877812+0000 osd.7 (osd.7) 497 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.570474+0000 osd.2 (osd.2) 451 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.570474+0000 osd.2 (osd.2) 451 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.573895+0000 osd.2 (osd.2) 452 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:39:16.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:16 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.573895+0000 osd.2 (osd.2) 452 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:39:17.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.213345+0000 osd.3 (osd.3) 543 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:39:17.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.213345+0000 osd.3 (osd.3) 543 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.216221+0000 osd.3 (osd.3) 544 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.216221+0000 osd.3 (osd.3) 544 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.380806+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.380806+0000 osd.6 (osd.6) 71 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.383223+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.383223+0000 osd.6 (osd.6) 72 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.875282+0000 osd.7 (osd.7) 496 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.875282+0000 osd.7 (osd.7) 496 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.877812+0000 osd.7 (osd.7) 497 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:15.877812+0000 osd.7 (osd.7) 497 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.570474+0000 osd.2 (osd.2) 451 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.570474+0000 osd.2 (osd.2) 451 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:39:17.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.573895+0000 osd.2 (osd.2) 452 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:39:17.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:16 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.573895+0000 osd.2 (osd.2) 452 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:39:17.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:39:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.009557+0000 mgr.y (mgr.24401) 835 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.009557+0000 mgr.y (mgr.24401) 835 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.924402+0000 osd.7 (osd.7) 498 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:39:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.924402+0000 osd.7 (osd.7) 498 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:39:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.926719+0000 osd.7 (osd.7) 499 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:39:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:17 smithi190 bash[23868]: cluster 2024-04-26T16:39:16.926719+0000 osd.7 (osd.7) 499 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:39:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.009557+0000 mgr.y (mgr.24401) 835 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.009557+0000 mgr.y (mgr.24401) 835 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.924402+0000 osd.7 (osd.7) 498 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:39:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.924402+0000 osd.7 (osd.7) 498 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:39:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.926719+0000 osd.7 (osd.7) 499 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[21463]: cluster 2024-04-26T16:39:16.926719+0000 osd.7 (osd.7) 499 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.009557+0000 mgr.y (mgr.24401) 835 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.009557+0000 mgr.y (mgr.24401) 835 : cluster [DBG] pgmap v1419: 174 pgs: 174 active+clean; 583 KiB data, 776 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.924402+0000 osd.7 (osd.7) 498 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.924402+0000 osd.7 (osd.7) 498 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.926719+0000 osd.7 (osd.7) 499 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:39:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:17 smithi050 bash[29234]: cluster 2024-04-26T16:39:16.926719+0000 osd.7 (osd.7) 499 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:39:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[21463]: cluster 2024-04-26T16:39:17.326079+0000 osd.1 (osd.1) 487 : cluster [DBG] 11.0 deep-scrub starts 2024-04-26T16:39:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[21463]: cluster 2024-04-26T16:39:17.326079+0000 osd.1 (osd.1) 487 : cluster [DBG] 11.0 deep-scrub starts 2024-04-26T16:39:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[21463]: cluster 2024-04-26T16:39:17.328343+0000 osd.1 (osd.1) 488 : cluster [DBG] 11.0 deep-scrub ok 2024-04-26T16:39:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[21463]: cluster 2024-04-26T16:39:17.328343+0000 osd.1 (osd.1) 488 : cluster [DBG] 11.0 deep-scrub ok 2024-04-26T16:39:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[21463]: audit 2024-04-26T16:39:17.437279+0000 mgr.y (mgr.24401) 836 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[21463]: audit 2024-04-26T16:39:17.437279+0000 mgr.y (mgr.24401) 836 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[29234]: cluster 2024-04-26T16:39:17.326079+0000 osd.1 (osd.1) 487 : cluster [DBG] 11.0 deep-scrub starts 2024-04-26T16:39:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[29234]: cluster 2024-04-26T16:39:17.326079+0000 osd.1 (osd.1) 487 : cluster [DBG] 11.0 deep-scrub starts 2024-04-26T16:39:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[29234]: cluster 2024-04-26T16:39:17.328343+0000 osd.1 (osd.1) 488 : cluster [DBG] 11.0 deep-scrub ok 2024-04-26T16:39:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[29234]: cluster 2024-04-26T16:39:17.328343+0000 osd.1 (osd.1) 488 : cluster [DBG] 11.0 deep-scrub ok 2024-04-26T16:39:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[29234]: audit 2024-04-26T16:39:17.437279+0000 mgr.y (mgr.24401) 836 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:18 smithi050 bash[29234]: audit 2024-04-26T16:39:17.437279+0000 mgr.y (mgr.24401) 836 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:18 smithi190 bash[23868]: cluster 2024-04-26T16:39:17.326079+0000 osd.1 (osd.1) 487 : cluster [DBG] 11.0 deep-scrub starts 2024-04-26T16:39:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:18 smithi190 bash[23868]: cluster 2024-04-26T16:39:17.326079+0000 osd.1 (osd.1) 487 : cluster [DBG] 11.0 deep-scrub starts 2024-04-26T16:39:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:18 smithi190 bash[23868]: cluster 2024-04-26T16:39:17.328343+0000 osd.1 (osd.1) 488 : cluster [DBG] 11.0 deep-scrub ok 2024-04-26T16:39:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:18 smithi190 bash[23868]: cluster 2024-04-26T16:39:17.328343+0000 osd.1 (osd.1) 488 : cluster [DBG] 11.0 deep-scrub ok 2024-04-26T16:39:19.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:18 smithi190 bash[23868]: audit 2024-04-26T16:39:17.437279+0000 mgr.y (mgr.24401) 836 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:19.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:18 smithi190 bash[23868]: audit 2024-04-26T16:39:17.437279+0000 mgr.y (mgr.24401) 836 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:19.281 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:39:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.011484+0000 mgr.y (mgr.24401) 837 : cluster [DBG] pgmap v1420: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.011484+0000 mgr.y (mgr.24401) 837 : cluster [DBG] pgmap v1420: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.206717+0000 osd.3 (osd.3) 545 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:39:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.206717+0000 osd.3 (osd.3) 545 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:39:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.211185+0000 osd.3 (osd.3) 546 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.211185+0000 osd.3 (osd.3) 546 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.288969+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.288969+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.289401+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.289401+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.626710+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:18.626710+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.542653+0000 osd.2 (osd.2) 453 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.542653+0000 osd.2 (osd.2) 453 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.544981+0000 osd.2 (osd.2) 454 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.544981+0000 osd.2 (osd.2) 454 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:39:19.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.011484+0000 mgr.y (mgr.24401) 837 : cluster [DBG] pgmap v1420: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.011484+0000 mgr.y (mgr.24401) 837 : cluster [DBG] pgmap v1420: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.206717+0000 osd.3 (osd.3) 545 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.206717+0000 osd.3 (osd.3) 545 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.211185+0000 osd.3 (osd.3) 546 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.211185+0000 osd.3 (osd.3) 546 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.288969+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.288969+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.289401+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.289401+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.626710+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:18.626710+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.542653+0000 osd.2 (osd.2) 453 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:39:19.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.542653+0000 osd.2 (osd.2) 453 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:39:19.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.544981+0000 osd.2 (osd.2) 454 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:39:19.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:19 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.544981+0000 osd.2 (osd.2) 454 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:39:20.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.011484+0000 mgr.y (mgr.24401) 837 : cluster [DBG] pgmap v1420: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:20.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.011484+0000 mgr.y (mgr.24401) 837 : cluster [DBG] pgmap v1420: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.206717+0000 osd.3 (osd.3) 545 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.206717+0000 osd.3 (osd.3) 545 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.211185+0000 osd.3 (osd.3) 546 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.211185+0000 osd.3 (osd.3) 546 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.288969+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.288969+0000 osd.6 (osd.6) 73 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.289401+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.289401+0000 osd.6 (osd.6) 74 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.626710+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:18.626710+0000 mon.a (mon.0) 3988 : cluster [DBG] osdmap e892: 8 total, 8 up, 8 in 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.542653+0000 osd.2 (osd.2) 453 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.542653+0000 osd.2 (osd.2) 453 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:39:20.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.544981+0000 osd.2 (osd.2) 454 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:39:20.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:19 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.544981+0000 osd.2 (osd.2) 454 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.339295+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.339295+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.341627+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[29234]: cluster 2024-04-26T16:39:19.341627+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.339295+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.339295+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:39:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.341627+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:39:20.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:20 smithi050 bash[21463]: cluster 2024-04-26T16:39:19.341627+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:39:21.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:20 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.339295+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:39:21.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:20 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.339295+0000 osd.1 (osd.1) 489 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:39:21.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:20 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.341627+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:39:21.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:20 smithi190 bash[23868]: cluster 2024-04-26T16:39:19.341627+0000 osd.1 (osd.1) 490 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:39:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:20.012245+0000 mgr.y (mgr.24401) 838 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:20.012245+0000 mgr.y (mgr.24401) 838 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:20.992143+0000 osd.7 (osd.7) 500 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:39:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:20.992143+0000 osd.7 (osd.7) 500 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:39:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:20.994431+0000 osd.7 (osd.7) 501 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:39:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:20.994431+0000 osd.7 (osd.7) 501 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:21.464833+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[21463]: cluster 2024-04-26T16:39:21.464833+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:20.012245+0000 mgr.y (mgr.24401) 838 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:20.012245+0000 mgr.y (mgr.24401) 838 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:20.992143+0000 osd.7 (osd.7) 500 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:20.992143+0000 osd.7 (osd.7) 500 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:20.994431+0000 osd.7 (osd.7) 501 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:20.994431+0000 osd.7 (osd.7) 501 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:21.464833+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-04-26T16:39:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:21 smithi050 bash[29234]: cluster 2024-04-26T16:39:21.464833+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:20.012245+0000 mgr.y (mgr.24401) 838 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:20.012245+0000 mgr.y (mgr.24401) 838 : cluster [DBG] pgmap v1422: 174 pgs: 174 active+clean; 583 KiB data, 780 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:20.992143+0000 osd.7 (osd.7) 500 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:20.992143+0000 osd.7 (osd.7) 500 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:20.994431+0000 osd.7 (osd.7) 501 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:20.994431+0000 osd.7 (osd.7) 501 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:21.464833+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-04-26T16:39:22.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:21 smithi190 bash[23868]: cluster 2024-04-26T16:39:21.464833+0000 mon.a (mon.0) 3989 : cluster [DBG] osdmap e893: 8 total, 8 up, 8 in 2024-04-26T16:39:22.984 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- 172.21.15.50:0/4099865745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb04006b130 msgr2=0x7fb04006b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4099865745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb04006b130 0x7fb04006b590 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009990 tx=0x7fb02c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- 172.21.15.50:0/4099865745 shutdown_connections 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4099865745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb0401094a0 0x7fb040109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4099865745 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006bad0 0x7fb0401102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4099865745 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb04006b130 0x7fb04006b590 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- 172.21.15.50:0/4099865745 >> 172.21.15.50:0/4099865745 conn(0x7fb0400792d0 msgr2=0x7fb0400796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- 172.21.15.50:0/4099865745 shutdown_connections 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- 172.21.15.50:0/4099865745 wait complete. 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 librados: starting msgr at 2024-04-26T16:39:22.985 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 librados: starting objecter 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 Processor -- start 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- start start 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 librados: setting wanted keys 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 librados: calling monclient init 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006b130 0x7fb0401991e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04006bad0 0x7fb040199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0401094a0 0x7fb04019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb040112ae0 con 0x7fb0401094a0 2024-04-26T16:39:22.987 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb040112960 con 0x7fb04006bad0 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb046dd7700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb040112c60 con 0x7fb04006b130 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb044b73700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006b130 0x7fb0401991e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb045374700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0401094a0 0x7fb04019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb045374700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0401094a0 0x7fb04019dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:46132/0 (socket says 172.21.15.50:46132) 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb045374700 1 -- 172.21.15.50:0/4166738851 learned_addr learned my addr 172.21.15.50:0/4166738851 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.985+0000 7fb03ffff700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04006bad0 0x7fb040199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb044b73700 1 -- 172.21.15.50:0/4166738851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04006bad0 msgr2=0x7fb040199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb044b73700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04006bad0 0x7fb040199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb044b73700 1 -- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0401094a0 msgr2=0x7fb04019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb044b73700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0401094a0 0x7fb04019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb044b73700 1 -- 172.21.15.50:0/4166738851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04019e1f0 con 0x7fb04006b130 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb044b73700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006b130 0x7fb0401991e0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d950 tx=0x7fb03400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:22.988 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 10 client.?.objecter ms_handle_connect 0x7fb04006b130 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034016070 con 0x7fb04006b130 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fb03400fcf0 con 0x7fb04006b130 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04019e4e0 con 0x7fb04006b130 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb034010d40 con 0x7fb04006b130 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb046dd7700 10 client.41798.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401a5d60 con 0x7fb04006b130 2024-04-26T16:39:22.989 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb046dd7700 1 librados: init done 2024-04-26T16:39:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04006a110 con 0x7fb04006b130 2024-04-26T16:39:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fb034019070 con 0x7fb04006b130 2024-04-26T16:39:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb030071f80 0x7fb030074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(893..893 src has 827..893) ==== 8707+0+0 (secure 0 0 0) 0x7fb0340910e0 con 0x7fb04006b130 2024-04-26T16:39:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 10 client.41798.objecter ms_dispatch 0x7fb040001550 osd_map(893..893 src has 827..893) 2024-04-26T16:39:22.990 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 3 client.41798.objecter handle_osd_map got epochs [893,893] > 0 2024-04-26T16:39:22.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 3 client.41798.objecter handle_osd_map decoding full epoch 893 2024-04-26T16:39:22.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03ffff700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb030071f80 0x7fb030074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:22.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 20 client.41798.objecter dump_active .. 0 homeless 2024-04-26T16:39:22.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03dffb700 10 client.41798.objecter ms_handle_connect 0x7fb030071f80 2024-04-26T16:39:22.991 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.989+0000 7fb03ffff700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb030071f80 0x7fb030074440 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00b2e0 tx=0x7fb02c005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:22.994 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:22.993+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fb03405cdb0 con 0x7fb04006b130 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:21.992597+0000 osd.5 (osd.5) 382 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:21.992597+0000 osd.5 (osd.5) 382 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:21.994892+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:21.994892+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.007658+0000 osd.7 (osd.7) 502 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.007658+0000 osd.7 (osd.7) 502 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:39:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.010021+0000 osd.7 (osd.7) 503 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:39:23.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:22 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.010021+0000 osd.7 (osd.7) 503 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:21.992597+0000 osd.5 (osd.5) 382 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:21.992597+0000 osd.5 (osd.5) 382 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:21.994892+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:21.994892+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.007658+0000 osd.7 (osd.7) 502 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.007658+0000 osd.7 (osd.7) 502 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.010021+0000 osd.7 (osd.7) 503 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:39:23.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.010021+0000 osd.7 (osd.7) 503 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:21.992597+0000 osd.5 (osd.5) 382 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:21.992597+0000 osd.5 (osd.5) 382 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:21.994892+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:21.994892+0000 osd.5 (osd.5) 383 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.007658+0000 osd.7 (osd.7) 502 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.007658+0000 osd.7 (osd.7) 502 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.010021+0000 osd.7 (osd.7) 503 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:39:23.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:22 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.010021+0000 osd.7 (osd.7) 503 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:39:23.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.233+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} v 0) -- 0x7fb0401a5ff0 con 0x7fb04006b130 2024-04-26T16:39:23.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.645+0000 7fb03dffb700 1 -- 172.21.15.50:0/4166738851 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]=0 set osd.0 primary-affinity to 1 (8655362) v894) ==== 133+0+0 (secure 0 0 0) 0x7fb034058030 con 0x7fb04006b130 2024-04-26T16:39:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 10 librados: watch_flush enter 2024-04-26T16:39:23.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 10 librados: watch_flush exit 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 20 client.41798.objecter shutdown clearing up homeless session... 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 10 client.41798.objecter successfully canceled tick 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb030071f80 msgr2=0x7fb030074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb030071f80 0x7fb030074440 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00b2e0 tx=0x7fb02c005960 comp rx=0 tx=0).stop 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006b130 msgr2=0x7fb0401991e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006b130 0x7fb0401991e0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d950 tx=0x7fb03400dc60 comp rx=0 tx=0).stop 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 shutdown_connections 2024-04-26T16:39:23.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fb030071f80 0x7fb030074440 secure :-1 s=CLOSED pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7fb02c00b2e0 tx=0x7fb02c005960 comp rx=0 tx=0).stop 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb0401094a0 0x7fb04019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb04006b130 0x7fb0401991e0 secure :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d950 tx=0x7fb03400dc60 comp rx=0 tx=0).stop 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 --2- 172.21.15.50:0/4166738851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb04006bad0 0x7fb040199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.649+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 >> 172.21.15.50:0/4166738851 conn(0x7fb0400792d0 msgr2=0x7fb040100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.653+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 shutdown_connections 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.653+0000 7fb046dd7700 1 -- 172.21.15.50:0/4166738851 wait complete. 2024-04-26T16:39:23.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:23.653+0000 7fb046dd7700 1 librados: shutdown 2024-04-26T16:39:23.651 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-04-26T16:39:23.962 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.013972+0000 mgr.y (mgr.24401) 839 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:23.962 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.013972+0000 mgr.y (mgr.24401) 839 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:23.962 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.280918+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:39:23.962 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.280918+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:39:23.962 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.283490+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.283490+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.346826+0000 osd.1 (osd.1) 491 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.346826+0000 osd.1 (osd.1) 491 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.349153+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.349153+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.479533+0000 osd.6 (osd.6) 75 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.479533+0000 osd.6 (osd.6) 75 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.482140+0000 osd.6 (osd.6) 76 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.482140+0000 osd.6 (osd.6) 76 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.848396+0000 osd.4 (osd.4) 463 : cluster [DBG] 6.e scrub starts 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.848396+0000 osd.4 (osd.4) 463 : cluster [DBG] 6.e scrub starts 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.850507+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.e scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: cluster 2024-04-26T16:39:22.850507+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.e scrub ok 2024-04-26T16:39:23.963 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: audit 2024-04-26T16:39:23.238194+0000 mon.c (mon.2) 922 : audit [INF] from='client.? 172.21.15.50:0/4166738851' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: audit 2024-04-26T16:39:23.238194+0000 mon.c (mon.2) 922 : audit [INF] from='client.? 172.21.15.50:0/4166738851' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: audit 2024-04-26T16:39:23.238690+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[21463]: audit 2024-04-26T16:39:23.238690+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.013972+0000 mgr.y (mgr.24401) 839 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.013972+0000 mgr.y (mgr.24401) 839 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.280918+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.280918+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.283490+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.283490+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.346826+0000 osd.1 (osd.1) 491 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:39:23.964 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.346826+0000 osd.1 (osd.1) 491 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.349153+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.349153+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.479533+0000 osd.6 (osd.6) 75 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.479533+0000 osd.6 (osd.6) 75 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.482140+0000 osd.6 (osd.6) 76 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.482140+0000 osd.6 (osd.6) 76 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.848396+0000 osd.4 (osd.4) 463 : cluster [DBG] 6.e scrub starts 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.848396+0000 osd.4 (osd.4) 463 : cluster [DBG] 6.e scrub starts 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.850507+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.e scrub ok 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: cluster 2024-04-26T16:39:22.850507+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.e scrub ok 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: audit 2024-04-26T16:39:23.238194+0000 mon.c (mon.2) 922 : audit [INF] from='client.? 172.21.15.50:0/4166738851' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: audit 2024-04-26T16:39:23.238194+0000 mon.c (mon.2) 922 : audit [INF] from='client.? 172.21.15.50:0/4166738851' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: audit 2024-04-26T16:39:23.238690+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:23.965 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:23 smithi050 bash[29234]: audit 2024-04-26T16:39:23.238690+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.013972+0000 mgr.y (mgr.24401) 839 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.013972+0000 mgr.y (mgr.24401) 839 : cluster [DBG] pgmap v1424: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.280918+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.280918+0000 osd.3 (osd.3) 547 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.283490+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.283490+0000 osd.3 (osd.3) 548 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.346826+0000 osd.1 (osd.1) 491 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.346826+0000 osd.1 (osd.1) 491 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.349153+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.349153+0000 osd.1 (osd.1) 492 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.479533+0000 osd.6 (osd.6) 75 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:39:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.479533+0000 osd.6 (osd.6) 75 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.482140+0000 osd.6 (osd.6) 76 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.482140+0000 osd.6 (osd.6) 76 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.848396+0000 osd.4 (osd.4) 463 : cluster [DBG] 6.e scrub starts 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.848396+0000 osd.4 (osd.4) 463 : cluster [DBG] 6.e scrub starts 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.850507+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.e scrub ok 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: cluster 2024-04-26T16:39:22.850507+0000 osd.4 (osd.4) 464 : cluster [DBG] 6.e scrub ok 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: audit 2024-04-26T16:39:23.238194+0000 mon.c (mon.2) 922 : audit [INF] from='client.? 172.21.15.50:0/4166738851' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: audit 2024-04-26T16:39:23.238194+0000 mon.c (mon.2) 922 : audit [INF] from='client.? 172.21.15.50:0/4166738851' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: audit 2024-04-26T16:39:23.238690+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:24.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:23 smithi190 bash[23868]: audit 2024-04-26T16:39:23.238690+0000 mon.a (mon.0) 3990 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-04-26T16:39:24.527 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 1 1 2024-04-26T16:39:24.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:39:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:39:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: cluster 2024-04-26T16:39:23.460302+0000 osd.6 (osd.6) 77 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:39:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: cluster 2024-04-26T16:39:23.460302+0000 osd.6 (osd.6) 77 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:39:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: cluster 2024-04-26T16:39:23.466657+0000 osd.6 (osd.6) 78 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:39:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: cluster 2024-04-26T16:39:23.466657+0000 osd.6 (osd.6) 78 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: audit 2024-04-26T16:39:23.638255+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: audit 2024-04-26T16:39:23.638255+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: cluster 2024-04-26T16:39:23.645805+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[21463]: cluster 2024-04-26T16:39:23.645805+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: cluster 2024-04-26T16:39:23.460302+0000 osd.6 (osd.6) 77 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: cluster 2024-04-26T16:39:23.460302+0000 osd.6 (osd.6) 77 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: cluster 2024-04-26T16:39:23.466657+0000 osd.6 (osd.6) 78 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: cluster 2024-04-26T16:39:23.466657+0000 osd.6 (osd.6) 78 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: audit 2024-04-26T16:39:23.638255+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: audit 2024-04-26T16:39:23.638255+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: cluster 2024-04-26T16:39:23.645805+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-04-26T16:39:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:24 smithi050 bash[29234]: cluster 2024-04-26T16:39:23.645805+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-04-26T16:39:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: cluster 2024-04-26T16:39:23.460302+0000 osd.6 (osd.6) 77 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:39:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: cluster 2024-04-26T16:39:23.460302+0000 osd.6 (osd.6) 77 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:39:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: cluster 2024-04-26T16:39:23.466657+0000 osd.6 (osd.6) 78 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:39:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: cluster 2024-04-26T16:39:23.466657+0000 osd.6 (osd.6) 78 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:39:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: audit 2024-04-26T16:39:23.638255+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: audit 2024-04-26T16:39:23.638255+0000 mon.a (mon.0) 3991 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-04-26T16:39:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: cluster 2024-04-26T16:39:23.645805+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-04-26T16:39:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:24 smithi190 bash[23868]: cluster 2024-04-26T16:39:23.645805+0000 mon.a (mon.0) 3992 : cluster [DBG] osdmap e894: 8 total, 8 up, 8 in 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.014745+0000 mgr.y (mgr.24401) 840 : cluster [DBG] pgmap v1426: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.014745+0000 mgr.y (mgr.24401) 840 : cluster [DBG] pgmap v1426: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.420233+0000 osd.6 (osd.6) 79 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.420233+0000 osd.6 (osd.6) 79 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.424197+0000 osd.6 (osd.6) 80 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.424197+0000 osd.6 (osd.6) 80 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:39:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.829395+0000 osd.4 (osd.4) 465 : cluster [DBG] 11.e scrub starts 2024-04-26T16:39:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.829395+0000 osd.4 (osd.4) 465 : cluster [DBG] 11.e scrub starts 2024-04-26T16:39:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.832371+0000 osd.4 (osd.4) 466 : cluster [DBG] 11.e scrub ok 2024-04-26T16:39:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:25 smithi190 bash[23868]: cluster 2024-04-26T16:39:24.832371+0000 osd.4 (osd.4) 466 : cluster [DBG] 11.e scrub ok 2024-04-26T16:39:26.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.014745+0000 mgr.y (mgr.24401) 840 : cluster [DBG] pgmap v1426: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:26.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.014745+0000 mgr.y (mgr.24401) 840 : cluster [DBG] pgmap v1426: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:26.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.420233+0000 osd.6 (osd.6) 79 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:39:26.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.420233+0000 osd.6 (osd.6) 79 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.424197+0000 osd.6 (osd.6) 80 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.424197+0000 osd.6 (osd.6) 80 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.829395+0000 osd.4 (osd.4) 465 : cluster [DBG] 11.e scrub starts 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.829395+0000 osd.4 (osd.4) 465 : cluster [DBG] 11.e scrub starts 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.832371+0000 osd.4 (osd.4) 466 : cluster [DBG] 11.e scrub ok 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[21463]: cluster 2024-04-26T16:39:24.832371+0000 osd.4 (osd.4) 466 : cluster [DBG] 11.e scrub ok 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.014745+0000 mgr.y (mgr.24401) 840 : cluster [DBG] pgmap v1426: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.014745+0000 mgr.y (mgr.24401) 840 : cluster [DBG] pgmap v1426: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.420233+0000 osd.6 (osd.6) 79 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.420233+0000 osd.6 (osd.6) 79 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.424197+0000 osd.6 (osd.6) 80 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:39:26.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.424197+0000 osd.6 (osd.6) 80 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:39:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.829395+0000 osd.4 (osd.4) 465 : cluster [DBG] 11.e scrub starts 2024-04-26T16:39:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.829395+0000 osd.4 (osd.4) 465 : cluster [DBG] 11.e scrub starts 2024-04-26T16:39:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.832371+0000 osd.4 (osd.4) 466 : cluster [DBG] 11.e scrub ok 2024-04-26T16:39:26.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:25 smithi050 bash[29234]: cluster 2024-04-26T16:39:24.832371+0000 osd.4 (osd.4) 466 : cluster [DBG] 11.e scrub ok 2024-04-26T16:39:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.019940+0000 osd.0 (osd.0) 517 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:39:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.019940+0000 osd.0 (osd.0) 517 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:39:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.023176+0000 osd.0 (osd.0) 518 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:39:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.023176+0000 osd.0 (osd.0) 518 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:39:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.858450+0000 osd.4 (osd.4) 467 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.858450+0000 osd.4 (osd.4) 467 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.863966+0000 osd.4 (osd.4) 468 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:25.863966+0000 osd.4 (osd.4) 468 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.010340+0000 osd.7 (osd.7) 504 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.010340+0000 osd.7 (osd.7) 504 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.013389+0000 osd.7 (osd.7) 505 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.013389+0000 osd.7 (osd.7) 505 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.106163+0000 osd.5 (osd.5) 384 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.106163+0000 osd.5 (osd.5) 384 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.112285+0000 osd.5 (osd.5) 385 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.112285+0000 osd.5 (osd.5) 385 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.468180+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-04-26T16:39:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:26 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.468180+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-04-26T16:39:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.019940+0000 osd.0 (osd.0) 517 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:39:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.019940+0000 osd.0 (osd.0) 517 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:39:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.023176+0000 osd.0 (osd.0) 518 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:39:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.023176+0000 osd.0 (osd.0) 518 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:39:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.858450+0000 osd.4 (osd.4) 467 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:39:27.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.858450+0000 osd.4 (osd.4) 467 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.863966+0000 osd.4 (osd.4) 468 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:25.863966+0000 osd.4 (osd.4) 468 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.010340+0000 osd.7 (osd.7) 504 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.010340+0000 osd.7 (osd.7) 504 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.013389+0000 osd.7 (osd.7) 505 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.013389+0000 osd.7 (osd.7) 505 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.106163+0000 osd.5 (osd.5) 384 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.106163+0000 osd.5 (osd.5) 384 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.112285+0000 osd.5 (osd.5) 385 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.112285+0000 osd.5 (osd.5) 385 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.468180+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.468180+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-04-26T16:39:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.019940+0000 osd.0 (osd.0) 517 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.019940+0000 osd.0 (osd.0) 517 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.023176+0000 osd.0 (osd.0) 518 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.023176+0000 osd.0 (osd.0) 518 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.858450+0000 osd.4 (osd.4) 467 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.858450+0000 osd.4 (osd.4) 467 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.863966+0000 osd.4 (osd.4) 468 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:25.863966+0000 osd.4 (osd.4) 468 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.010340+0000 osd.7 (osd.7) 504 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.010340+0000 osd.7 (osd.7) 504 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.013389+0000 osd.7 (osd.7) 505 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.013389+0000 osd.7 (osd.7) 505 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.106163+0000 osd.5 (osd.5) 384 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.106163+0000 osd.5 (osd.5) 384 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.112285+0000 osd.5 (osd.5) 385 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.112285+0000 osd.5 (osd.5) 385 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:39:27.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.468180+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-04-26T16:39:27.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:26 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.468180+0000 mon.a (mon.0) 3993 : cluster [DBG] osdmap e895: 8 total, 8 up, 8 in 2024-04-26T16:39:27.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:39:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.015697+0000 mgr.y (mgr.24401) 841 : cluster [DBG] pgmap v1427: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.015697+0000 mgr.y (mgr.24401) 841 : cluster [DBG] pgmap v1427: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.024978+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.024978+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.027531+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.027531+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.335824+0000 osd.3 (osd.3) 549 : cluster [DBG] 11.a scrub starts 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.335824+0000 osd.3 (osd.3) 549 : cluster [DBG] 11.a scrub starts 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.338431+0000 osd.3 (osd.3) 550 : cluster [DBG] 11.a scrub ok 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.338431+0000 osd.3 (osd.3) 550 : cluster [DBG] 11.a scrub ok 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.475008+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.475008+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.490756+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:39:27.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.490756+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.586586+0000 osd.2 (osd.2) 455 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.586586+0000 osd.2 (osd.2) 455 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.590372+0000 osd.2 (osd.2) 456 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.590372+0000 osd.2 (osd.2) 456 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: audit 2024-04-26T16:39:26.829788+0000 mon.c (mon.2) 923 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: audit 2024-04-26T16:39:26.829788+0000 mon.c (mon.2) 923 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.867890+0000 osd.4 (osd.4) 469 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.867890+0000 osd.4 (osd.4) 469 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.870089+0000 osd.4 (osd.4) 470 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.870089+0000 osd.4 (osd.4) 470 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.011526+0000 osd.7 (osd.7) 506 : cluster [DBG] 11.f scrub starts 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.011526+0000 osd.7 (osd.7) 506 : cluster [DBG] 11.f scrub starts 2024-04-26T16:39:27.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.017966+0000 osd.7 (osd.7) 507 : cluster [DBG] 11.f scrub ok 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.017966+0000 osd.7 (osd.7) 507 : cluster [DBG] 11.f scrub ok 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.066767+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.066767+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.068974+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.068974+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: audit 2024-04-26T16:39:27.668949+0000 mon.c (mon.2) 924 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:27.827 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:27 smithi190 bash[23868]: audit 2024-04-26T16:39:27.668949+0000 mon.c (mon.2) 924 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:28.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.015697+0000 mgr.y (mgr.24401) 841 : cluster [DBG] pgmap v1427: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.015697+0000 mgr.y (mgr.24401) 841 : cluster [DBG] pgmap v1427: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.024978+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.024978+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.027531+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.027531+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.335824+0000 osd.3 (osd.3) 549 : cluster [DBG] 11.a scrub starts 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.335824+0000 osd.3 (osd.3) 549 : cluster [DBG] 11.a scrub starts 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.338431+0000 osd.3 (osd.3) 550 : cluster [DBG] 11.a scrub ok 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.338431+0000 osd.3 (osd.3) 550 : cluster [DBG] 11.a scrub ok 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.475008+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.475008+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.490756+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.490756+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:39:28.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.586586+0000 osd.2 (osd.2) 455 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.586586+0000 osd.2 (osd.2) 455 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.590372+0000 osd.2 (osd.2) 456 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.590372+0000 osd.2 (osd.2) 456 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: audit 2024-04-26T16:39:26.829788+0000 mon.c (mon.2) 923 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: audit 2024-04-26T16:39:26.829788+0000 mon.c (mon.2) 923 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.867890+0000 osd.4 (osd.4) 469 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.867890+0000 osd.4 (osd.4) 469 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.870089+0000 osd.4 (osd.4) 470 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.870089+0000 osd.4 (osd.4) 470 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.011526+0000 osd.7 (osd.7) 506 : cluster [DBG] 11.f scrub starts 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.011526+0000 osd.7 (osd.7) 506 : cluster [DBG] 11.f scrub starts 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.017966+0000 osd.7 (osd.7) 507 : cluster [DBG] 11.f scrub ok 2024-04-26T16:39:28.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.017966+0000 osd.7 (osd.7) 507 : cluster [DBG] 11.f scrub ok 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.015697+0000 mgr.y (mgr.24401) 841 : cluster [DBG] pgmap v1427: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.015697+0000 mgr.y (mgr.24401) 841 : cluster [DBG] pgmap v1427: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.024978+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.024978+0000 osd.0 (osd.0) 519 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.027531+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.027531+0000 osd.0 (osd.0) 520 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.335824+0000 osd.3 (osd.3) 549 : cluster [DBG] 11.a scrub starts 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.335824+0000 osd.3 (osd.3) 549 : cluster [DBG] 11.a scrub starts 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.338431+0000 osd.3 (osd.3) 550 : cluster [DBG] 11.a scrub ok 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.338431+0000 osd.3 (osd.3) 550 : cluster [DBG] 11.a scrub ok 2024-04-26T16:39:28.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.475008+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.475008+0000 osd.6 (osd.6) 81 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.490756+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.490756+0000 osd.6 (osd.6) 82 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.586586+0000 osd.2 (osd.2) 455 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.586586+0000 osd.2 (osd.2) 455 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.590372+0000 osd.2 (osd.2) 456 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.590372+0000 osd.2 (osd.2) 456 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: audit 2024-04-26T16:39:26.829788+0000 mon.c (mon.2) 923 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: audit 2024-04-26T16:39:26.829788+0000 mon.c (mon.2) 923 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.867890+0000 osd.4 (osd.4) 469 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.867890+0000 osd.4 (osd.4) 469 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.870089+0000 osd.4 (osd.4) 470 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.870089+0000 osd.4 (osd.4) 470 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:39:28.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.011526+0000 osd.7 (osd.7) 506 : cluster [DBG] 11.f scrub starts 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.011526+0000 osd.7 (osd.7) 506 : cluster [DBG] 11.f scrub starts 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.017966+0000 osd.7 (osd.7) 507 : cluster [DBG] 11.f scrub ok 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.017966+0000 osd.7 (osd.7) 507 : cluster [DBG] 11.f scrub ok 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.066767+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.066767+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.068974+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.068974+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: audit 2024-04-26T16:39:27.668949+0000 mon.c (mon.2) 924 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:28.136 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[21463]: audit 2024-04-26T16:39:27.668949+0000 mon.c (mon.2) 924 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:28.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.066767+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:39:28.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.066767+0000 osd.5 (osd.5) 386 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:39:28.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.068974+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:39:28.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.068974+0000 osd.5 (osd.5) 387 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:39:28.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: audit 2024-04-26T16:39:27.668949+0000 mon.c (mon.2) 924 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:28.137 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:27 smithi050 bash[29234]: audit 2024-04-26T16:39:27.668949+0000 mon.c (mon.2) 924 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:39:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.998397+0000 osd.0 (osd.0) 521 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:39:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:26.998397+0000 osd.0 (osd.0) 521 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.002287+0000 osd.0 (osd.0) 522 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.002287+0000 osd.0 (osd.0) 522 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:27.443589+0000 mgr.y (mgr.24401) 842 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:27.443589+0000 mgr.y (mgr.24401) 842 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:27.670356+0000 mon.c (mon.2) 925 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:27.670356+0000 mon.c (mon.2) 925 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:27.685735+0000 mon.a (mon.0) 3994 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:27.685735+0000 mon.a (mon.0) 3994 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.835698+0000 osd.4 (osd.4) 471 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.835698+0000 osd.4 (osd.4) 471 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.837853+0000 osd.4 (osd.4) 472 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:39:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.837853+0000 osd.4 (osd.4) 472 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:39:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.992328+0000 osd.7 (osd.7) 508 : cluster [DBG] 2.1c deep-scrub starts 2024-04-26T16:39:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.992328+0000 osd.7 (osd.7) 508 : cluster [DBG] 2.1c deep-scrub starts 2024-04-26T16:39:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.994577+0000 osd.7 (osd.7) 509 : cluster [DBG] 2.1c deep-scrub ok 2024-04-26T16:39:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: cluster 2024-04-26T16:39:27.994577+0000 osd.7 (osd.7) 509 : cluster [DBG] 2.1c deep-scrub ok 2024-04-26T16:39:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:28.269177+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:29.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:28 smithi190 bash[23868]: audit 2024-04-26T16:39:28.269177+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:29.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.998397+0000 osd.0 (osd.0) 521 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:39:29.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:26.998397+0000 osd.0 (osd.0) 521 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.002287+0000 osd.0 (osd.0) 522 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.002287+0000 osd.0 (osd.0) 522 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:27.443589+0000 mgr.y (mgr.24401) 842 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:27.443589+0000 mgr.y (mgr.24401) 842 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:27.670356+0000 mon.c (mon.2) 925 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:27.670356+0000 mon.c (mon.2) 925 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:27.685735+0000 mon.a (mon.0) 3994 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:27.685735+0000 mon.a (mon.0) 3994 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.835698+0000 osd.4 (osd.4) 471 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.835698+0000 osd.4 (osd.4) 471 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.837853+0000 osd.4 (osd.4) 472 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.837853+0000 osd.4 (osd.4) 472 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:39:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.992328+0000 osd.7 (osd.7) 508 : cluster [DBG] 2.1c deep-scrub starts 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.992328+0000 osd.7 (osd.7) 508 : cluster [DBG] 2.1c deep-scrub starts 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.994577+0000 osd.7 (osd.7) 509 : cluster [DBG] 2.1c deep-scrub ok 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: cluster 2024-04-26T16:39:27.994577+0000 osd.7 (osd.7) 509 : cluster [DBG] 2.1c deep-scrub ok 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:28.269177+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[29234]: audit 2024-04-26T16:39:28.269177+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.998397+0000 osd.0 (osd.0) 521 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:26.998397+0000 osd.0 (osd.0) 521 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.002287+0000 osd.0 (osd.0) 522 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.002287+0000 osd.0 (osd.0) 522 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:27.443589+0000 mgr.y (mgr.24401) 842 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:27.443589+0000 mgr.y (mgr.24401) 842 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:27.670356+0000 mon.c (mon.2) 925 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:27.670356+0000 mon.c (mon.2) 925 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:39:29.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:27.685735+0000 mon.a (mon.0) 3994 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:27.685735+0000 mon.a (mon.0) 3994 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.835698+0000 osd.4 (osd.4) 471 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.835698+0000 osd.4 (osd.4) 471 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.837853+0000 osd.4 (osd.4) 472 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.837853+0000 osd.4 (osd.4) 472 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.992328+0000 osd.7 (osd.7) 508 : cluster [DBG] 2.1c deep-scrub starts 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.992328+0000 osd.7 (osd.7) 508 : cluster [DBG] 2.1c deep-scrub starts 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.994577+0000 osd.7 (osd.7) 509 : cluster [DBG] 2.1c deep-scrub ok 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: cluster 2024-04-26T16:39:27.994577+0000 osd.7 (osd.7) 509 : cluster [DBG] 2.1c deep-scrub ok 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:28.269177+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:29.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:28 smithi050 bash[21463]: audit 2024-04-26T16:39:28.269177+0000 mon.c (mon.2) 926 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:29.312 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.017608+0000 mgr.y (mgr.24401) 843 : cluster [DBG] pgmap v1429: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.017608+0000 mgr.y (mgr.24401) 843 : cluster [DBG] pgmap v1429: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.590841+0000 osd.2 (osd.2) 457 : cluster [DBG] 5.d scrub starts 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.590841+0000 osd.2 (osd.2) 457 : cluster [DBG] 5.d scrub starts 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.594461+0000 osd.2 (osd.2) 458 : cluster [DBG] 5.d scrub ok 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.594461+0000 osd.2 (osd.2) 458 : cluster [DBG] 5.d scrub ok 2024-04-26T16:39:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.955704+0000 osd.7 (osd.7) 510 : cluster [DBG] 3.e scrub starts 2024-04-26T16:39:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.955704+0000 osd.7 (osd.7) 510 : cluster [DBG] 3.e scrub starts 2024-04-26T16:39:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.959821+0000 osd.7 (osd.7) 511 : cluster [DBG] 3.e scrub ok 2024-04-26T16:39:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:29 smithi190 bash[23868]: cluster 2024-04-26T16:39:28.959821+0000 osd.7 (osd.7) 511 : cluster [DBG] 3.e scrub ok 2024-04-26T16:39:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.017608+0000 mgr.y (mgr.24401) 843 : cluster [DBG] pgmap v1429: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.017608+0000 mgr.y (mgr.24401) 843 : cluster [DBG] pgmap v1429: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.590841+0000 osd.2 (osd.2) 457 : cluster [DBG] 5.d scrub starts 2024-04-26T16:39:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.590841+0000 osd.2 (osd.2) 457 : cluster [DBG] 5.d scrub starts 2024-04-26T16:39:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.594461+0000 osd.2 (osd.2) 458 : cluster [DBG] 5.d scrub ok 2024-04-26T16:39:30.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.594461+0000 osd.2 (osd.2) 458 : cluster [DBG] 5.d scrub ok 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.955704+0000 osd.7 (osd.7) 510 : cluster [DBG] 3.e scrub starts 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.955704+0000 osd.7 (osd.7) 510 : cluster [DBG] 3.e scrub starts 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.959821+0000 osd.7 (osd.7) 511 : cluster [DBG] 3.e scrub ok 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[29234]: cluster 2024-04-26T16:39:28.959821+0000 osd.7 (osd.7) 511 : cluster [DBG] 3.e scrub ok 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.017608+0000 mgr.y (mgr.24401) 843 : cluster [DBG] pgmap v1429: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.017608+0000 mgr.y (mgr.24401) 843 : cluster [DBG] pgmap v1429: 174 pgs: 1 active+clean+scrubbing+deep, 173 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.590841+0000 osd.2 (osd.2) 457 : cluster [DBG] 5.d scrub starts 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.590841+0000 osd.2 (osd.2) 457 : cluster [DBG] 5.d scrub starts 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.594461+0000 osd.2 (osd.2) 458 : cluster [DBG] 5.d scrub ok 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.594461+0000 osd.2 (osd.2) 458 : cluster [DBG] 5.d scrub ok 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.955704+0000 osd.7 (osd.7) 510 : cluster [DBG] 3.e scrub starts 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.955704+0000 osd.7 (osd.7) 510 : cluster [DBG] 3.e scrub starts 2024-04-26T16:39:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.959821+0000 osd.7 (osd.7) 511 : cluster [DBG] 3.e scrub ok 2024-04-26T16:39:30.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:29 smithi050 bash[21463]: cluster 2024-04-26T16:39:28.959821+0000 osd.7 (osd.7) 511 : cluster [DBG] 3.e scrub ok 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.870671+0000 osd.4 (osd.4) 473 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.870671+0000 osd.4 (osd.4) 473 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.872987+0000 osd.4 (osd.4) 474 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.872987+0000 osd.4 (osd.4) 474 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.946909+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.946909+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:39:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.951311+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:39:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:30 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.951311+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.870671+0000 osd.4 (osd.4) 473 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.870671+0000 osd.4 (osd.4) 473 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.872987+0000 osd.4 (osd.4) 474 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.872987+0000 osd.4 (osd.4) 474 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.946909+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.946909+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:39:31.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.951311+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.951311+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.870671+0000 osd.4 (osd.4) 473 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.870671+0000 osd.4 (osd.4) 473 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.872987+0000 osd.4 (osd.4) 474 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.872987+0000 osd.4 (osd.4) 474 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.946909+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.946909+0000 osd.7 (osd.7) 512 : cluster [DBG] 3.15 deep-scrub starts 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.951311+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:39:31.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:30 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.951311+0000 osd.7 (osd.7) 513 : cluster [DBG] 3.15 deep-scrub ok 2024-04-26T16:39:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.992231+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-04-26T16:39:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:29.992231+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.003473+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.003473+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.018559+0000 mgr.y (mgr.24401) 844 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.018559+0000 mgr.y (mgr.24401) 844 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.392123+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.392123+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.394713+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:39:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.394713+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:39:32.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.905199+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:39:32.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.905199+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:39:32.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.907263+0000 osd.4 (osd.4) 476 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:39:32.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:30.907263+0000 osd.4 (osd.4) 476 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:39:32.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.471270+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-04-26T16:39:32.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:31 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.471270+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-04-26T16:39:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.992231+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-04-26T16:39:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:29.992231+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-04-26T16:39:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.003473+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-04-26T16:39:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.003473+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-04-26T16:39:32.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.018559+0000 mgr.y (mgr.24401) 844 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.018559+0000 mgr.y (mgr.24401) 844 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.392123+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.392123+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.394713+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.394713+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.905199+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.905199+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.907263+0000 osd.4 (osd.4) 476 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:39:32.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:30.907263+0000 osd.4 (osd.4) 476 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.471270+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.471270+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.992231+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:29.992231+0000 osd.0 (osd.0) 523 : cluster [DBG] 3.1f deep-scrub starts 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.003473+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.003473+0000 osd.0 (osd.0) 524 : cluster [DBG] 3.1f deep-scrub ok 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.018559+0000 mgr.y (mgr.24401) 844 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.018559+0000 mgr.y (mgr.24401) 844 : cluster [DBG] pgmap v1430: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:32.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.392123+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.392123+0000 osd.6 (osd.6) 83 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.394713+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.394713+0000 osd.6 (osd.6) 84 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.905199+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.905199+0000 osd.4 (osd.4) 475 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.907263+0000 osd.4 (osd.4) 476 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:39:32.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:30.907263+0000 osd.4 (osd.4) 476 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:39:32.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.471270+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-04-26T16:39:32.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:31 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.471270+0000 mon.a (mon.0) 3995 : cluster [DBG] osdmap e896: 8 total, 8 up, 8 in 2024-04-26T16:39:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/1148734859 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103e30 msgr2=0x7fbb70110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/1148734859 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103e30 0x7fbb70110290 secure :-1 s=READY pgs=2782 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009990 tx=0x7fbb68009ca0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/1148734859 shutdown_connections 2024-04-26T16:39:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/1148734859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb701094c0 0x7fbb701098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/1148734859 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103e30 0x7fbb70110290 unknown :-1 s=CLOSED pgs=2782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/1148734859 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb70103470 0x7fbb701038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/1148734859 >> 172.21.15.50:0/1148734859 conn(0x7fbb7007a580 msgr2=0x7fbb7007a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/1148734859 shutdown_connections 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/1148734859 wait complete. 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 librados: starting msgr at 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 librados: starting objecter 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 Processor -- start 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- start start 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 librados: setting wanted keys 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 librados: calling monclient init 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 0x7fbb70199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb70103e30 0x7fbb70199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb701094c0 0x7fbb7019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:33.062 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70112b60 con 0x7fbb701094c0 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701129e0 con 0x7fbb70103e30 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fbb70112ce0 con 0x7fbb70103470 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 0x7fbb70199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 0x7fbb70199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:59046/0 (socket says 172.21.15.50:59046) 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 -- 172.21.15.50:0/3640645877 learned_addr learned my addr 172.21.15.50:0/3640645877 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6ed9d700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb701094c0 0x7fbb7019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6dd9b700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb70103e30 0x7fbb70199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 -- 172.21.15.50:0/3640645877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb70103e30 msgr2=0x7fbb70199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb70103e30 0x7fbb70199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 -- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb701094c0 msgr2=0x7fbb7019db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb701094c0 0x7fbb7019db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 -- 172.21.15.50:0/3640645877 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7019e250 con 0x7fbb70103470 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6dd9b700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb70103e30 0x7fbb70199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:39:33.063 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb6e59c700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 0x7fbb70199240 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000ba50 tx=0x7fbb6000bd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:33.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb677fe700 10 client.?.objecter ms_handle_connect 0x7fbb70103470 2024-04-26T16:39:33.064 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb677fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60019070 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fbb6000ce10 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb60014e30 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7019e540 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 10 client.41804.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb701a5dc0 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.061+0000 7fbb74aa9700 1 librados: init done 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb50005120 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fbb6001c070 con 0x7fbb70103470 2024-04-26T16:39:33.065 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbb5c071f80 0x7fbb5c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(896..896 src has 827..896) ==== 8707+0+0 (secure 0 0 0) 0x7fbb60090990 con 0x7fbb70103470 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb6dd9b700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbb5c071f80 0x7fbb5c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 10 client.41804.objecter ms_dispatch 0x7fbb70001550 osd_map(896..896 src has 827..896) 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 3 client.41804.objecter handle_osd_map got epochs [896,896] > 0 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 3 client.41804.objecter handle_osd_map decoding full epoch 896 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 20 client.41804.objecter dump_active .. 0 homeless 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb6dd9b700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbb5c071f80 0x7fbb5c074440 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fbb70108f80 tx=0x7fbb58009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:33.066 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.065+0000 7fbb677fe700 10 client.41804.objecter ms_handle_connect 0x7fbb5c071f80 2024-04-26T16:39:33.068 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.069+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fbb60060310 con 0x7fbb70103470 2024-04-26T16:39:33.073 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[50598]: logger=cleanup t=2024-04-26T16:39:32.688432153Z level=info msg="Completed cleanup jobs" duration=55.789728ms 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.663528+0000 osd.2 (osd.2) 459 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.663528+0000 osd.2 (osd.2) 459 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.666094+0000 osd.2 (osd.2) 460 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.666094+0000 osd.2 (osd.2) 460 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.916248+0000 osd.7 (osd.7) 514 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.916248+0000 osd.7 (osd.7) 514 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:39:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.919111+0000 osd.7 (osd.7) 515 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:39:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:32 smithi190 bash[23868]: cluster 2024-04-26T16:39:31.919111+0000 osd.7 (osd.7) 515 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.663528+0000 osd.2 (osd.2) 459 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.663528+0000 osd.2 (osd.2) 459 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.666094+0000 osd.2 (osd.2) 460 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.666094+0000 osd.2 (osd.2) 460 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.916248+0000 osd.7 (osd.7) 514 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.916248+0000 osd.7 (osd.7) 514 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.919111+0000 osd.7 (osd.7) 515 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[29234]: cluster 2024-04-26T16:39:31.919111+0000 osd.7 (osd.7) 515 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.663528+0000 osd.2 (osd.2) 459 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:39:33.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.663528+0000 osd.2 (osd.2) 459 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:39:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.666094+0000 osd.2 (osd.2) 460 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:39:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.666094+0000 osd.2 (osd.2) 460 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:39:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.916248+0000 osd.7 (osd.7) 514 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:39:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.916248+0000 osd.7 (osd.7) 514 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:39:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.919111+0000 osd.7 (osd.7) 515 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:39:33.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:32 smithi050 bash[21463]: cluster 2024-04-26T16:39:31.919111+0000 osd.7 (osd.7) 515 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:39:33.309 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.309+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 1, "weight": 1.0} v 0) -- 0x7fbb50005410 con 0x7fbb70103470 2024-04-26T16:39:33.722 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.721+0000 7fbb677fe700 1 -- 172.21.15.50:0/3640645877 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 1, "weight": 1.0}]=0 reweighted osd.1 to 1 (10000) v897) ==== 113+0+0 (secure 0 0 0) 0x7fbb60064b40 con 0x7fbb70103470 2024-04-26T16:39:33.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 10 librados: watch_flush enter 2024-04-26T16:39:33.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 10 librados: watch_flush exit 2024-04-26T16:39:33.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 20 client.41804.objecter shutdown clearing up homeless session... 2024-04-26T16:39:33.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 10 client.41804.objecter successfully canceled tick 2024-04-26T16:39:33.726 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbb5c071f80 msgr2=0x7fbb5c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:33.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbb5c071f80 0x7fbb5c074440 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fbb70108f80 tx=0x7fbb58009260 comp rx=0 tx=0).stop 2024-04-26T16:39:33.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 msgr2=0x7fbb70199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:33.727 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 0x7fbb70199240 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7fbb6000ba50 tx=0x7fbb6000bd60 comp rx=0 tx=0).stop 2024-04-26T16:39:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 shutdown_connections 2024-04-26T16:39:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fbb5c071f80 0x7fbb5c074440 unknown :-1 s=CLOSED pgs=3858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fbb701094c0 0x7fbb7019db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.728 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fbb70103470 0x7fbb70199240 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 --2- 172.21.15.50:0/3640645877 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb70103e30 0x7fbb70199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 >> 172.21.15.50:0/3640645877 conn(0x7fbb7007a580 msgr2=0x7fbb70101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 shutdown_connections 2024-04-26T16:39:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 -- 172.21.15.50:0/3640645877 wait complete. 2024-04-26T16:39:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:33.729+0000 7fbb74aa9700 1 librados: shutdown 2024-04-26T16:39:33.729 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.1 to 1 (10000) 2024-04-26T16:39:34.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.020324+0000 mgr.y (mgr.24401) 845 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:34.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.020324+0000 mgr.y (mgr.24401) 845 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:34.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.428466+0000 osd.6 (osd.6) 85 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:39:34.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.428466+0000 osd.6 (osd.6) 85 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:39:34.027 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.433768+0000 osd.6 (osd.6) 86 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.433768+0000 osd.6 (osd.6) 86 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.703070+0000 osd.2 (osd.2) 461 : cluster [DBG] 2.c scrub starts 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.703070+0000 osd.2 (osd.2) 461 : cluster [DBG] 2.c scrub starts 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.706223+0000 osd.2 (osd.2) 462 : cluster [DBG] 2.c scrub ok 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.706223+0000 osd.2 (osd.2) 462 : cluster [DBG] 2.c scrub ok 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: audit 2024-04-26T16:39:33.314547+0000 mon.c (mon.2) 927 : audit [INF] from='client.? 172.21.15.50:0/3640645877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: audit 2024-04-26T16:39:33.314547+0000 mon.c (mon.2) 927 : audit [INF] from='client.? 172.21.15.50:0/3640645877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: audit 2024-04-26T16:39:33.315200+0000 mon.a (mon.0) 3996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[29234]: audit 2024-04-26T16:39:33.315200+0000 mon.a (mon.0) 3996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.020324+0000 mgr.y (mgr.24401) 845 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.020324+0000 mgr.y (mgr.24401) 845 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:34.028 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.428466+0000 osd.6 (osd.6) 85 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.428466+0000 osd.6 (osd.6) 85 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.433768+0000 osd.6 (osd.6) 86 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.433768+0000 osd.6 (osd.6) 86 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.703070+0000 osd.2 (osd.2) 461 : cluster [DBG] 2.c scrub starts 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.703070+0000 osd.2 (osd.2) 461 : cluster [DBG] 2.c scrub starts 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.706223+0000 osd.2 (osd.2) 462 : cluster [DBG] 2.c scrub ok 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.706223+0000 osd.2 (osd.2) 462 : cluster [DBG] 2.c scrub ok 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: audit 2024-04-26T16:39:33.314547+0000 mon.c (mon.2) 927 : audit [INF] from='client.? 172.21.15.50:0/3640645877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: audit 2024-04-26T16:39:33.314547+0000 mon.c (mon.2) 927 : audit [INF] from='client.? 172.21.15.50:0/3640645877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: audit 2024-04-26T16:39:33.315200+0000 mon.a (mon.0) 3996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.029 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:33 smithi050 bash[21463]: audit 2024-04-26T16:39:33.315200+0000 mon.a (mon.0) 3996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.020324+0000 mgr.y (mgr.24401) 845 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.020324+0000 mgr.y (mgr.24401) 845 : cluster [DBG] pgmap v1432: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.428466+0000 osd.6 (osd.6) 85 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:39:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.428466+0000 osd.6 (osd.6) 85 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.433768+0000 osd.6 (osd.6) 86 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.433768+0000 osd.6 (osd.6) 86 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.703070+0000 osd.2 (osd.2) 461 : cluster [DBG] 2.c scrub starts 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.703070+0000 osd.2 (osd.2) 461 : cluster [DBG] 2.c scrub starts 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.706223+0000 osd.2 (osd.2) 462 : cluster [DBG] 2.c scrub ok 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.706223+0000 osd.2 (osd.2) 462 : cluster [DBG] 2.c scrub ok 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: audit 2024-04-26T16:39:33.314547+0000 mon.c (mon.2) 927 : audit [INF] from='client.? 172.21.15.50:0/3640645877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: audit 2024-04-26T16:39:33.314547+0000 mon.c (mon.2) 927 : audit [INF] from='client.? 172.21.15.50:0/3640645877' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: audit 2024-04-26T16:39:33.315200+0000 mon.a (mon.0) 3996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:33 smithi190 bash[23868]: audit 2024-04-26T16:39:33.315200+0000 mon.a (mon.0) 3996 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:34.681 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:39:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:39:34.686 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 1 1 2024-04-26T16:39:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.989853+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:39:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.989853+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:39:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.991573+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:32.991573+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.273606+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.273606+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.276899+0000 osd.3 (osd.3) 552 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.276899+0000 osd.3 (osd.3) 552 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.330304+0000 osd.1 (osd.1) 493 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.330304+0000 osd.1 (osd.1) 493 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.333623+0000 osd.1 (osd.1) 494 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.333623+0000 osd.1 (osd.1) 494 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: audit 2024-04-26T16:39:33.715351+0000 mon.a (mon.0) 3997 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: audit 2024-04-26T16:39:33.715351+0000 mon.a (mon.0) 3997 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.724734+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-04-26T16:39:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:33.724734+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-04-26T16:39:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.127393+0000 osd.5 (osd.5) 388 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:39:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.127393+0000 osd.5 (osd.5) 388 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:39:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.130555+0000 osd.5 (osd.5) 389 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:39:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:34 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.130555+0000 osd.5 (osd.5) 389 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:39:35.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.989853+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.989853+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.991573+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:32.991573+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.273606+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.273606+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.276899+0000 osd.3 (osd.3) 552 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.276899+0000 osd.3 (osd.3) 552 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.330304+0000 osd.1 (osd.1) 493 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.330304+0000 osd.1 (osd.1) 493 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.333623+0000 osd.1 (osd.1) 494 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.333623+0000 osd.1 (osd.1) 494 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:39:35.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: audit 2024-04-26T16:39:33.715351+0000 mon.a (mon.0) 3997 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: audit 2024-04-26T16:39:33.715351+0000 mon.a (mon.0) 3997 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.724734+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:33.724734+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.127393+0000 osd.5 (osd.5) 388 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.127393+0000 osd.5 (osd.5) 388 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.130555+0000 osd.5 (osd.5) 389 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.130555+0000 osd.5 (osd.5) 389 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:39:35.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.989853+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.989853+0000 osd.0 (osd.0) 525 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.991573+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:32.991573+0000 osd.0 (osd.0) 526 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.273606+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.273606+0000 osd.3 (osd.3) 551 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.276899+0000 osd.3 (osd.3) 552 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.276899+0000 osd.3 (osd.3) 552 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.330304+0000 osd.1 (osd.1) 493 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.330304+0000 osd.1 (osd.1) 493 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:39:35.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.333623+0000 osd.1 (osd.1) 494 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.333623+0000 osd.1 (osd.1) 494 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: audit 2024-04-26T16:39:33.715351+0000 mon.a (mon.0) 3997 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: audit 2024-04-26T16:39:33.715351+0000 mon.a (mon.0) 3997 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.724734+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:33.724734+0000 mon.a (mon.0) 3998 : cluster [DBG] osdmap e897: 8 total, 8 up, 8 in 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.127393+0000 osd.5 (osd.5) 388 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.127393+0000 osd.5 (osd.5) 388 : cluster [DBG] 11.16 deep-scrub starts 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.130555+0000 osd.5 (osd.5) 389 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:39:35.135 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:34 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.130555+0000 osd.5 (osd.5) 389 : cluster [DBG] 11.16 deep-scrub ok 2024-04-26T16:39:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.021092+0000 mgr.y (mgr.24401) 846 : cluster [DBG] pgmap v1434: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.021092+0000 mgr.y (mgr.24401) 846 : cluster [DBG] pgmap v1434: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.299703+0000 osd.1 (osd.1) 495 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:39:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.299703+0000 osd.1 (osd.1) 495 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:39:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.302617+0000 osd.1 (osd.1) 496 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:39:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.302617+0000 osd.1 (osd.1) 496 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:39:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.906413+0000 osd.4 (osd.4) 477 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:39:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.906413+0000 osd.4 (osd.4) 477 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:39:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.908721+0000 osd.4 (osd.4) 478 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:39:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:35 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.908721+0000 osd.4 (osd.4) 478 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:39:36.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.021092+0000 mgr.y (mgr.24401) 846 : cluster [DBG] pgmap v1434: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:36.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.021092+0000 mgr.y (mgr.24401) 846 : cluster [DBG] pgmap v1434: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.299703+0000 osd.1 (osd.1) 495 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.299703+0000 osd.1 (osd.1) 495 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.302617+0000 osd.1 (osd.1) 496 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.302617+0000 osd.1 (osd.1) 496 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.906413+0000 osd.4 (osd.4) 477 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.906413+0000 osd.4 (osd.4) 477 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.908721+0000 osd.4 (osd.4) 478 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.908721+0000 osd.4 (osd.4) 478 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.021092+0000 mgr.y (mgr.24401) 846 : cluster [DBG] pgmap v1434: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.021092+0000 mgr.y (mgr.24401) 846 : cluster [DBG] pgmap v1434: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.299703+0000 osd.1 (osd.1) 495 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:39:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.299703+0000 osd.1 (osd.1) 495 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:39:36.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.302617+0000 osd.1 (osd.1) 496 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:39:36.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.302617+0000 osd.1 (osd.1) 496 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:39:36.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.906413+0000 osd.4 (osd.4) 477 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:39:36.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.906413+0000 osd.4 (osd.4) 477 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:39:36.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.908721+0000 osd.4 (osd.4) 478 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:39:36.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:35 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.908721+0000 osd.4 (osd.4) 478 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:39:37.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.976366+0000 osd.0 (osd.0) 527 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:39:37.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.976366+0000 osd.0 (osd.0) 527 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:39:37.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.978451+0000 osd.0 (osd.0) 528 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:34.978451+0000 osd.0 (osd.0) 528 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.409448+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.409448+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.412819+0000 osd.6 (osd.6) 88 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.412819+0000 osd.6 (osd.6) 88 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.858791+0000 osd.7 (osd.7) 516 : cluster [DBG] 3.f scrub starts 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.858791+0000 osd.7 (osd.7) 516 : cluster [DBG] 3.f scrub starts 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.863715+0000 osd.7 (osd.7) 517 : cluster [DBG] 3.f scrub ok 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:35.863715+0000 osd.7 (osd.7) 517 : cluster [DBG] 3.f scrub ok 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.156297+0000 osd.4 (osd.4) 479 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.156297+0000 osd.4 (osd.4) 479 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.156726+0000 osd.4 (osd.4) 480 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:37.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.156726+0000 osd.4 (osd.4) 480 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:37.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.474039+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-04-26T16:39:37.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:36 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.474039+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.976366+0000 osd.0 (osd.0) 527 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.976366+0000 osd.0 (osd.0) 527 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.978451+0000 osd.0 (osd.0) 528 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:34.978451+0000 osd.0 (osd.0) 528 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.409448+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.409448+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.412819+0000 osd.6 (osd.6) 88 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.412819+0000 osd.6 (osd.6) 88 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.858791+0000 osd.7 (osd.7) 516 : cluster [DBG] 3.f scrub starts 2024-04-26T16:39:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.858791+0000 osd.7 (osd.7) 516 : cluster [DBG] 3.f scrub starts 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.863715+0000 osd.7 (osd.7) 517 : cluster [DBG] 3.f scrub ok 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:35.863715+0000 osd.7 (osd.7) 517 : cluster [DBG] 3.f scrub ok 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.156297+0000 osd.4 (osd.4) 479 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.156297+0000 osd.4 (osd.4) 479 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.156726+0000 osd.4 (osd.4) 480 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.156726+0000 osd.4 (osd.4) 480 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.474039+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.474039+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.976366+0000 osd.0 (osd.0) 527 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.976366+0000 osd.0 (osd.0) 527 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.978451+0000 osd.0 (osd.0) 528 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:39:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:34.978451+0000 osd.0 (osd.0) 528 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.409448+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.409448+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.412819+0000 osd.6 (osd.6) 88 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.412819+0000 osd.6 (osd.6) 88 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.858791+0000 osd.7 (osd.7) 516 : cluster [DBG] 3.f scrub starts 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.858791+0000 osd.7 (osd.7) 516 : cluster [DBG] 3.f scrub starts 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.863715+0000 osd.7 (osd.7) 517 : cluster [DBG] 3.f scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:35.863715+0000 osd.7 (osd.7) 517 : cluster [DBG] 3.f scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.156297+0000 osd.4 (osd.4) 479 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.156297+0000 osd.4 (osd.4) 479 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.156726+0000 osd.4 (osd.4) 480 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.156726+0000 osd.4 (osd.4) 480 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.474039+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-04-26T16:39:37.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:36 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.474039+0000 mon.a (mon.0) 3999 : cluster [DBG] osdmap e898: 8 total, 8 up, 8 in 2024-04-26T16:39:37.750 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:39:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.021884+0000 mgr.y (mgr.24401) 847 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.021884+0000 mgr.y (mgr.24401) 847 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.551080+0000 osd.1 (osd.1) 497 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.551080+0000 osd.1 (osd.1) 497 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.551569+0000 osd.1 (osd.1) 498 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:38.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.551569+0000 osd.1 (osd.1) 498 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.818833+0000 osd.7 (osd.7) 518 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:39:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.818833+0000 osd.7 (osd.7) 518 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:39:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.824457+0000 osd.7 (osd.7) 519 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:39:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:36.824457+0000 osd.7 (osd.7) 519 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:39:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.484449+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-04-26T16:39:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:37 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.484449+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-04-26T16:39:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.021884+0000 mgr.y (mgr.24401) 847 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.021884+0000 mgr.y (mgr.24401) 847 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.551080+0000 osd.1 (osd.1) 497 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.551080+0000 osd.1 (osd.1) 497 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.551569+0000 osd.1 (osd.1) 498 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.551569+0000 osd.1 (osd.1) 498 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.818833+0000 osd.7 (osd.7) 518 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.818833+0000 osd.7 (osd.7) 518 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.824457+0000 osd.7 (osd.7) 519 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:36.824457+0000 osd.7 (osd.7) 519 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.484449+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.484449+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-04-26T16:39:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.021884+0000 mgr.y (mgr.24401) 847 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.021884+0000 mgr.y (mgr.24401) 847 : cluster [DBG] pgmap v1435: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.551080+0000 osd.1 (osd.1) 497 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.551080+0000 osd.1 (osd.1) 497 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.551569+0000 osd.1 (osd.1) 498 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.551569+0000 osd.1 (osd.1) 498 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.818833+0000 osd.7 (osd.7) 518 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.818833+0000 osd.7 (osd.7) 518 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.824457+0000 osd.7 (osd.7) 519 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:36.824457+0000 osd.7 (osd.7) 519 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.484449+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-04-26T16:39:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:37 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.484449+0000 mon.a (mon.0) 4000 : cluster [DBG] osdmap e899: 8 total, 8 up, 8 in 2024-04-26T16:39:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: audit 2024-04-26T16:39:37.454588+0000 mgr.y (mgr.24401) 848 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: audit 2024-04-26T16:39:37.454588+0000 mgr.y (mgr.24401) 848 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.852393+0000 osd.7 (osd.7) 520 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:39:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.852393+0000 osd.7 (osd.7) 520 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:39:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.856389+0000 osd.7 (osd.7) 521 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.856389+0000 osd.7 (osd.7) 521 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.903262+0000 osd.4 (osd.4) 481 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.903262+0000 osd.4 (osd.4) 481 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.908854+0000 osd.4 (osd.4) 482 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.908854+0000 osd.4 (osd.4) 482 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.111916+0000 osd.5 (osd.5) 390 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.111916+0000 osd.5 (osd.5) 390 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.118476+0000 osd.5 (osd.5) 391 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:39:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:38 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.118476+0000 osd.5 (osd.5) 391 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:39:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: audit 2024-04-26T16:39:37.454588+0000 mgr.y (mgr.24401) 848 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: audit 2024-04-26T16:39:37.454588+0000 mgr.y (mgr.24401) 848 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.852393+0000 osd.7 (osd.7) 520 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.852393+0000 osd.7 (osd.7) 520 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.856389+0000 osd.7 (osd.7) 521 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.856389+0000 osd.7 (osd.7) 521 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.903262+0000 osd.4 (osd.4) 481 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.903262+0000 osd.4 (osd.4) 481 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.908854+0000 osd.4 (osd.4) 482 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.908854+0000 osd.4 (osd.4) 482 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.111916+0000 osd.5 (osd.5) 390 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.111916+0000 osd.5 (osd.5) 390 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.118476+0000 osd.5 (osd.5) 391 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.118476+0000 osd.5 (osd.5) 391 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:39:39.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: audit 2024-04-26T16:39:37.454588+0000 mgr.y (mgr.24401) 848 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: audit 2024-04-26T16:39:37.454588+0000 mgr.y (mgr.24401) 848 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.852393+0000 osd.7 (osd.7) 520 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.852393+0000 osd.7 (osd.7) 520 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.856389+0000 osd.7 (osd.7) 521 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.856389+0000 osd.7 (osd.7) 521 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.903262+0000 osd.4 (osd.4) 481 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.903262+0000 osd.4 (osd.4) 481 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.908854+0000 osd.4 (osd.4) 482 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.908854+0000 osd.4 (osd.4) 482 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.111916+0000 osd.5 (osd.5) 390 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.111916+0000 osd.5 (osd.5) 390 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.118476+0000 osd.5 (osd.5) 391 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:39:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:38 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.118476+0000 osd.5 (osd.5) 391 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:39:39.489 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:39:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.983235+0000 osd.0 (osd.0) 529 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:39:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.983235+0000 osd.0 (osd.0) 529 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:39:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.985562+0000 osd.0 (osd.0) 530 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:39:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:37.985562+0000 osd.0 (osd.0) 530 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:39:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.022779+0000 mgr.y (mgr.24401) 849 : cluster [DBG] pgmap v1438: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.022779+0000 mgr.y (mgr.24401) 849 : cluster [DBG] pgmap v1438: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.886520+0000 osd.7 (osd.7) 522 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.886520+0000 osd.7 (osd.7) 522 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.888828+0000 osd.7 (osd.7) 523 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:38.888828+0000 osd.7 (osd.7) 523 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:39.129506+0000 osd.5 (osd.5) 392 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:39.129506+0000 osd.5 (osd.5) 392 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:39.132089+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:39:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:39 smithi190 bash[23868]: cluster 2024-04-26T16:39:39.132089+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:39:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.983235+0000 osd.0 (osd.0) 529 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:39:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.983235+0000 osd.0 (osd.0) 529 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:39:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.985562+0000 osd.0 (osd.0) 530 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:39:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:37.985562+0000 osd.0 (osd.0) 530 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:39:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.022779+0000 mgr.y (mgr.24401) 849 : cluster [DBG] pgmap v1438: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.022779+0000 mgr.y (mgr.24401) 849 : cluster [DBG] pgmap v1438: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.886520+0000 osd.7 (osd.7) 522 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.886520+0000 osd.7 (osd.7) 522 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.888828+0000 osd.7 (osd.7) 523 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:38.888828+0000 osd.7 (osd.7) 523 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:39.129506+0000 osd.5 (osd.5) 392 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:39.129506+0000 osd.5 (osd.5) 392 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:39.132089+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[29234]: cluster 2024-04-26T16:39:39.132089+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.983235+0000 osd.0 (osd.0) 529 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:39:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.983235+0000 osd.0 (osd.0) 529 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.985562+0000 osd.0 (osd.0) 530 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:37.985562+0000 osd.0 (osd.0) 530 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.022779+0000 mgr.y (mgr.24401) 849 : cluster [DBG] pgmap v1438: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.022779+0000 mgr.y (mgr.24401) 849 : cluster [DBG] pgmap v1438: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.886520+0000 osd.7 (osd.7) 522 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.886520+0000 osd.7 (osd.7) 522 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.888828+0000 osd.7 (osd.7) 523 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:38.888828+0000 osd.7 (osd.7) 523 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:39.129506+0000 osd.5 (osd.5) 392 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:39.129506+0000 osd.5 (osd.5) 392 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:39.132089+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:39:40.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:39 smithi050 bash[21463]: cluster 2024-04-26T16:39:39.132089+0000 osd.5 (osd.5) 393 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:39:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:40.023949+0000 mgr.y (mgr.24401) 850 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-04-26T16:39:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:40.023949+0000 mgr.y (mgr.24401) 850 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-04-26T16:39:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:40.212750+0000 osd.1 (osd.1) 499 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:39:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:40.212750+0000 osd.1 (osd.1) 499 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:39:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:40.219813+0000 osd.1 (osd.1) 500 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:39:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:40.219813+0000 osd.1 (osd.1) 500 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:39:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.144203+0000 osd.5 (osd.5) 394 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:39:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.144203+0000 osd.5 (osd.5) 394 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:39:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.149414+0000 osd.5 (osd.5) 395 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:39:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.149414+0000 osd.5 (osd.5) 395 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:39:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.474620+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-04-26T16:39:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:41 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.474620+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-04-26T16:39:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:40.023949+0000 mgr.y (mgr.24401) 850 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-04-26T16:39:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:40.023949+0000 mgr.y (mgr.24401) 850 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-04-26T16:39:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:40.212750+0000 osd.1 (osd.1) 499 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:39:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:40.212750+0000 osd.1 (osd.1) 499 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:39:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:40.219813+0000 osd.1 (osd.1) 500 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:39:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:40.219813+0000 osd.1 (osd.1) 500 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.144203+0000 osd.5 (osd.5) 394 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.144203+0000 osd.5 (osd.5) 394 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.149414+0000 osd.5 (osd.5) 395 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.149414+0000 osd.5 (osd.5) 395 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.474620+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.474620+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:40.023949+0000 mgr.y (mgr.24401) 850 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:40.023949+0000 mgr.y (mgr.24401) 850 : cluster [DBG] pgmap v1439: 174 pgs: 174 active+clean; 583 KiB data, 781 MiB used, 714 GiB / 715 GiB avail; 811 B/s rd, 0 op/s 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:40.212750+0000 osd.1 (osd.1) 499 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:40.212750+0000 osd.1 (osd.1) 499 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:39:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:40.219813+0000 osd.1 (osd.1) 500 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:40.219813+0000 osd.1 (osd.1) 500 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.144203+0000 osd.5 (osd.5) 394 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.144203+0000 osd.5 (osd.5) 394 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.149414+0000 osd.5 (osd.5) 395 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.149414+0000 osd.5 (osd.5) 395 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.474620+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-04-26T16:39:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:41 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.474620+0000 mon.a (mon.0) 4001 : cluster [DBG] osdmap e900: 8 total, 8 up, 8 in 2024-04-26T16:39:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:42 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.216351+0000 osd.1 (osd.1) 501 : cluster [DBG] 11.c scrub starts 2024-04-26T16:39:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:42 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.216351+0000 osd.1 (osd.1) 501 : cluster [DBG] 11.c scrub starts 2024-04-26T16:39:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:42 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.219389+0000 osd.1 (osd.1) 502 : cluster [DBG] 11.c scrub ok 2024-04-26T16:39:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:42 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.219389+0000 osd.1 (osd.1) 502 : cluster [DBG] 11.c scrub ok 2024-04-26T16:39:43.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.216351+0000 osd.1 (osd.1) 501 : cluster [DBG] 11.c scrub starts 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.216351+0000 osd.1 (osd.1) 501 : cluster [DBG] 11.c scrub starts 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.219389+0000 osd.1 (osd.1) 502 : cluster [DBG] 11.c scrub ok 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.219389+0000 osd.1 (osd.1) 502 : cluster [DBG] 11.c scrub ok 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.216351+0000 osd.1 (osd.1) 501 : cluster [DBG] 11.c scrub starts 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.216351+0000 osd.1 (osd.1) 501 : cluster [DBG] 11.c scrub starts 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.219389+0000 osd.1 (osd.1) 502 : cluster [DBG] 11.c scrub ok 2024-04-26T16:39:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:42 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.219389+0000 osd.1 (osd.1) 502 : cluster [DBG] 11.c scrub ok 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- 172.21.15.50:0/865547673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8105660 msgr2=0x7f8bc8105ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/865547673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8105660 0x7f8bc8105ae0 secure :-1 s=READY pgs=2542 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009990 tx=0x7f8bb8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- 172.21.15.50:0/865547673 shutdown_connections 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/865547673 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bc810b660 0x7f8bc810ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/865547673 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc8106020 0x7f8bc8112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/865547673 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8105660 0x7f8bc8105ae0 unknown :-1 s=CLOSED pgs=2542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- 172.21.15.50:0/865547673 >> 172.21.15.50:0/865547673 conn(0x7f8bc8101150 msgr2=0x7f8bc8103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- 172.21.15.50:0/865547673 shutdown_connections 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- 172.21.15.50:0/865547673 wait complete. 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 librados: starting msgr at 2024-04-26T16:39:43.169 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 librados: starting objecter 2024-04-26T16:39:43.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 Processor -- start 2024-04-26T16:39:43.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- start start 2024-04-26T16:39:43.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 librados: setting wanted keys 2024-04-26T16:39:43.170 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 librados: calling monclient init 2024-04-26T16:39:43.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bc8105660 0x7f8bc819b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:43.171 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8106020 0x7f8bc819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 0x7f8bc819fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc8114e10 con 0x7f8bc8106020 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc8114c90 con 0x7f8bc8105660 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcfa16700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc8114f90 con 0x7f8bc810b660 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcdfb3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 0x7f8bc819fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bccfb1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8106020 0x7f8bc819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcdfb3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 0x7f8bc819fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:54900/0 (socket says 172.21.15.50:54900) 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcdfb3700 1 -- 172.21.15.50:0/1715183025 learned_addr learned my addr 172.21.15.50:0/1715183025 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.168+0000 7f8bcd7b2700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bc8105660 0x7f8bc819b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bccfb1700 1 -- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 msgr2=0x7f8bc819fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bccfb1700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 0x7f8bc819fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bccfb1700 1 -- 172.21.15.50:0/1715183025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bc8105660 msgr2=0x7f8bc819b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bccfb1700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bc8105660 0x7f8bc819b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bccfb1700 1 -- 172.21.15.50:0/1715183025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc81a03b0 con 0x7f8bc8106020 2024-04-26T16:39:43.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcdfb3700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 0x7f8bc819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bccfb1700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8106020 0x7f8bc819b8e0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800b2e0 tx=0x7f8bb8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 10 client.?.objecter ms_handle_connect 0x7f8bc8106020 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb8026070 con 0x7f8bc8106020 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8bb8021410 con 0x7f8bc8106020 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc81a0640 con 0x7f8bc8106020 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80173f0 con 0x7f8bc8106020 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcfa16700 10 client.31191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc81a7f20 con 0x7f8bc8106020 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcfa16700 1 librados: init done 2024-04-26T16:39:43.173 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc8106da0 con 0x7f8bc8106020 2024-04-26T16:39:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8bb8029070 con 0x7f8bc8106020 2024-04-26T16:39:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8bb4071f80 0x7f8bb4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(900..900 src has 827..900) ==== 8707+0+0 (secure 0 0 0) 0x7f8bb8092300 con 0x7f8bc8106020 2024-04-26T16:39:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 10 client.31191.objecter ms_dispatch 0x7f8bc8001550 osd_map(900..900 src has 827..900) 2024-04-26T16:39:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 3 client.31191.objecter handle_osd_map got epochs [900,900] > 0 2024-04-26T16:39:43.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 3 client.31191.objecter handle_osd_map decoding full epoch 900 2024-04-26T16:39:43.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcd7b2700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8bb4071f80 0x7f8bb4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:43.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 20 client.31191.objecter dump_active .. 0 homeless 2024-04-26T16:39:43.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bbe7fc700 10 client.31191.objecter ms_handle_connect 0x7f8bb4071f80 2024-04-26T16:39:43.175 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.172+0000 7f8bcd7b2700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8bb4071f80 0x7f8bb4074440 secure :-1 s=READY pgs=3859 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005950 tx=0x7f8bc400b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:43.177 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.176+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f8bb8013070 con 0x7f8bc8106020 2024-04-26T16:39:43.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.420+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} v 0) -- 0x7f8bc819c700 con 0x7f8bc8106020 2024-04-26T16:39:43.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.788+0000 7f8bbe7fc700 1 -- 172.21.15.50:0/1715183025 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]=0 set osd.1 primary-affinity to 1 (8655362) v901) ==== 133+0+0 (secure 0 0 0) 0x7f8bb8061c80 con 0x7f8bc8106020 2024-04-26T16:39:43.793 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.792+0000 7f8bcfa16700 10 librados: watch_flush enter 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 10 librados: watch_flush exit 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 20 client.31191.objecter shutdown clearing up homeless session... 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 10 client.31191.objecter successfully canceled tick 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8bb4071f80 msgr2=0x7f8bb4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8bb4071f80 0x7f8bb4074440 secure :-1 s=READY pgs=3859 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4005950 tx=0x7f8bc400b410 comp rx=0 tx=0).stop 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8106020 msgr2=0x7f8bc819b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8106020 0x7f8bc819b8e0 secure :-1 s=READY pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800b2e0 tx=0x7f8bb8009580 comp rx=0 tx=0).stop 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 shutdown_connections 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8bb4071f80 0x7f8bb4074440 unknown :-1 s=CLOSED pgs=3859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8bc8105660 0x7f8bc819b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8bc810b660 0x7f8bc819fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:43.798 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 --2- 172.21.15.50:0/1715183025 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8bc8106020 0x7f8bc819b8e0 secure :-1 s=CLOSED pgs=2543 cs=0 l=1 rev1=1 crypto rx=0x7f8bb800b2e0 tx=0x7f8bb8009580 comp rx=0 tx=0).stop 2024-04-26T16:39:43.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.796+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 >> 172.21.15.50:0/1715183025 conn(0x7f8bc8101150 msgr2=0x7f8bc8101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:43.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.800+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 shutdown_connections 2024-04-26T16:39:43.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.800+0000 7f8bcfa16700 1 -- 172.21.15.50:0/1715183025 wait complete. 2024-04-26T16:39:43.799 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:43.800+0000 7f8bcfa16700 1 librados: shutdown 2024-04-26T16:39:43.799 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-04-26T16:39:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.979491+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:39:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.979491+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:39:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.982423+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:41.982423+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.025674+0000 mgr.y (mgr.24401) 851 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.025674+0000 mgr.y (mgr.24401) 851 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.233425+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.233425+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.235781+0000 osd.1 (osd.1) 504 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.235781+0000 osd.1 (osd.1) 504 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.277619+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.277619+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.279624+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.279624+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.995189+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:39:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.995189+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:39:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.998383+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:39:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: cluster 2024-04-26T16:39:42.998383+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:39:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: audit 2024-04-26T16:39:43.301592+0000 mon.c (mon.2) 928 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: audit 2024-04-26T16:39:43.301592+0000 mon.c (mon.2) 928 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: audit 2024-04-26T16:39:43.425184+0000 mon.a (mon.0) 4002 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:43 smithi190 bash[23868]: audit 2024-04-26T16:39:43.425184+0000 mon.a (mon.0) 4002 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:44.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.979491+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:39:44.083 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.979491+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.982423+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:41.982423+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.025674+0000 mgr.y (mgr.24401) 851 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.025674+0000 mgr.y (mgr.24401) 851 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.233425+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.233425+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.235781+0000 osd.1 (osd.1) 504 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:39:44.084 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.235781+0000 osd.1 (osd.1) 504 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.277619+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.277619+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.279624+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.279624+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.995189+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.995189+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.998383+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: cluster 2024-04-26T16:39:42.998383+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: audit 2024-04-26T16:39:43.301592+0000 mon.c (mon.2) 928 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: audit 2024-04-26T16:39:43.301592+0000 mon.c (mon.2) 928 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: audit 2024-04-26T16:39:43.425184+0000 mon.a (mon.0) 4002 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:44.085 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[21463]: audit 2024-04-26T16:39:43.425184+0000 mon.a (mon.0) 4002 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:44.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.979491+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:39:44.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.979491+0000 osd.0 (osd.0) 531 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:39:44.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.982423+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:39:44.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:41.982423+0000 osd.0 (osd.0) 532 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:39:44.086 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.025674+0000 mgr.y (mgr.24401) 851 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.025674+0000 mgr.y (mgr.24401) 851 : cluster [DBG] pgmap v1441: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.233425+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.233425+0000 osd.1 (osd.1) 503 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.235781+0000 osd.1 (osd.1) 504 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.235781+0000 osd.1 (osd.1) 504 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.277619+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.277619+0000 osd.3 (osd.3) 553 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.279624+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.279624+0000 osd.3 (osd.3) 554 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.995189+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.995189+0000 osd.7 (osd.7) 524 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.998383+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:39:44.087 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: cluster 2024-04-26T16:39:42.998383+0000 osd.7 (osd.7) 525 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:39:44.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: audit 2024-04-26T16:39:43.301592+0000 mon.c (mon.2) 928 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:44.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: audit 2024-04-26T16:39:43.301592+0000 mon.c (mon.2) 928 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:44.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: audit 2024-04-26T16:39:43.425184+0000 mon.a (mon.0) 4002 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:44.088 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:43 smithi050 bash[29234]: audit 2024-04-26T16:39:43.425184+0000 mon.a (mon.0) 4002 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-04-26T16:39:44.722 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:39:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:39:44.723 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 2 1 2024-04-26T16:39:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:44 smithi190 bash[23868]: audit 2024-04-26T16:39:43.794016+0000 mon.a (mon.0) 4003 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:44 smithi190 bash[23868]: audit 2024-04-26T16:39:43.794016+0000 mon.a (mon.0) 4003 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:44 smithi190 bash[23868]: cluster 2024-04-26T16:39:43.801619+0000 mon.a (mon.0) 4004 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-04-26T16:39:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:44 smithi190 bash[23868]: cluster 2024-04-26T16:39:43.801619+0000 mon.a (mon.0) 4004 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-04-26T16:39:45.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[21463]: audit 2024-04-26T16:39:43.794016+0000 mon.a (mon.0) 4003 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:45.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[21463]: audit 2024-04-26T16:39:43.794016+0000 mon.a (mon.0) 4003 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:45.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[21463]: cluster 2024-04-26T16:39:43.801619+0000 mon.a (mon.0) 4004 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-04-26T16:39:45.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[21463]: cluster 2024-04-26T16:39:43.801619+0000 mon.a (mon.0) 4004 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-04-26T16:39:45.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[29234]: audit 2024-04-26T16:39:43.794016+0000 mon.a (mon.0) 4003 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:45.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[29234]: audit 2024-04-26T16:39:43.794016+0000 mon.a (mon.0) 4003 : audit [INF] from='client.? 172.21.15.50:0/1715183025' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-04-26T16:39:45.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[29234]: cluster 2024-04-26T16:39:43.801619+0000 mon.a (mon.0) 4004 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-04-26T16:39:45.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:44 smithi050 bash[29234]: cluster 2024-04-26T16:39:43.801619+0000 mon.a (mon.0) 4004 : cluster [DBG] osdmap e901: 8 total, 8 up, 8 in 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:43.990114+0000 osd.0 (osd.0) 533 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:43.990114+0000 osd.0 (osd.0) 533 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:43.995530+0000 osd.0 (osd.0) 534 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:43.995530+0000 osd.0 (osd.0) 534 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:44.026606+0000 mgr.y (mgr.24401) 852 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:44.026606+0000 mgr.y (mgr.24401) 852 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:44.647342+0000 osd.2 (osd.2) 463 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:39:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:44.647342+0000 osd.2 (osd.2) 463 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:39:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:44.649715+0000 osd.2 (osd.2) 464 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:39:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:45 smithi190 bash[23868]: cluster 2024-04-26T16:39:44.649715+0000 osd.2 (osd.2) 464 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:39:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:43.990114+0000 osd.0 (osd.0) 533 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:39:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:43.990114+0000 osd.0 (osd.0) 533 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:39:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:43.995530+0000 osd.0 (osd.0) 534 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:39:46.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:43.995530+0000 osd.0 (osd.0) 534 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:44.026606+0000 mgr.y (mgr.24401) 852 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:44.026606+0000 mgr.y (mgr.24401) 852 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:44.647342+0000 osd.2 (osd.2) 463 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:44.647342+0000 osd.2 (osd.2) 463 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:44.649715+0000 osd.2 (osd.2) 464 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[29234]: cluster 2024-04-26T16:39:44.649715+0000 osd.2 (osd.2) 464 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:43.990114+0000 osd.0 (osd.0) 533 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:43.990114+0000 osd.0 (osd.0) 533 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:43.995530+0000 osd.0 (osd.0) 534 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:43.995530+0000 osd.0 (osd.0) 534 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:39:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:44.026606+0000 mgr.y (mgr.24401) 852 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-26T16:39:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:44.026606+0000 mgr.y (mgr.24401) 852 : cluster [DBG] pgmap v1443: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-26T16:39:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:44.647342+0000 osd.2 (osd.2) 463 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:39:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:44.647342+0000 osd.2 (osd.2) 463 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:39:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:44.649715+0000 osd.2 (osd.2) 464 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:39:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:45 smithi050 bash[21463]: cluster 2024-04-26T16:39:44.649715+0000 osd.2 (osd.2) 464 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:45.259888+0000 osd.1 (osd.1) 505 : cluster [DBG] 2.a scrub starts 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:45.259888+0000 osd.1 (osd.1) 505 : cluster [DBG] 2.a scrub starts 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:45.263228+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.a scrub ok 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:45.263228+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.a scrub ok 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.097884+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.097884+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.100282+0000 osd.5 (osd.5) 397 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:39:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.100282+0000 osd.5 (osd.5) 397 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:39:47.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.479084+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-04-26T16:39:47.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:46 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.479084+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-04-26T16:39:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:45.259888+0000 osd.1 (osd.1) 505 : cluster [DBG] 2.a scrub starts 2024-04-26T16:39:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:45.259888+0000 osd.1 (osd.1) 505 : cluster [DBG] 2.a scrub starts 2024-04-26T16:39:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:45.263228+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.a scrub ok 2024-04-26T16:39:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:45.263228+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.a scrub ok 2024-04-26T16:39:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.097884+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.097884+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.100282+0000 osd.5 (osd.5) 397 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.100282+0000 osd.5 (osd.5) 397 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.479084+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.479084+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:45.259888+0000 osd.1 (osd.1) 505 : cluster [DBG] 2.a scrub starts 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:45.259888+0000 osd.1 (osd.1) 505 : cluster [DBG] 2.a scrub starts 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:45.263228+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.a scrub ok 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:45.263228+0000 osd.1 (osd.1) 506 : cluster [DBG] 2.a scrub ok 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.097884+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.097884+0000 osd.5 (osd.5) 396 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:39:47.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.100282+0000 osd.5 (osd.5) 397 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:39:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.100282+0000 osd.5 (osd.5) 397 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:39:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.479084+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-04-26T16:39:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:46 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.479084+0000 mon.a (mon.0) 4005 : cluster [DBG] osdmap e902: 8 total, 8 up, 8 in 2024-04-26T16:39:47.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:39:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.027478+0000 mgr.y (mgr.24401) 853 : cluster [DBG] pgmap v1444: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:46.027478+0000 mgr.y (mgr.24401) 853 : cluster [DBG] pgmap v1444: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.012984+0000 osd.7 (osd.7) 526 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:39:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.012984+0000 osd.7 (osd.7) 526 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:39:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.017576+0000 osd.7 (osd.7) 527 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.017576+0000 osd.7 (osd.7) 527 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.054176+0000 osd.5 (osd.5) 398 : cluster [DBG] 3.c scrub starts 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.054176+0000 osd.5 (osd.5) 398 : cluster [DBG] 3.c scrub starts 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.058464+0000 osd.5 (osd.5) 399 : cluster [DBG] 3.c scrub ok 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.058464+0000 osd.5 (osd.5) 399 : cluster [DBG] 3.c scrub ok 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.288447+0000 osd.1 (osd.1) 507 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.288447+0000 osd.1 (osd.1) 507 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.290722+0000 osd.1 (osd.1) 508 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:39:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:47 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.290722+0000 osd.1 (osd.1) 508 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:39:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.027478+0000 mgr.y (mgr.24401) 853 : cluster [DBG] pgmap v1444: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:46.027478+0000 mgr.y (mgr.24401) 853 : cluster [DBG] pgmap v1444: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.012984+0000 osd.7 (osd.7) 526 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:39:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.012984+0000 osd.7 (osd.7) 526 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.017576+0000 osd.7 (osd.7) 527 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.017576+0000 osd.7 (osd.7) 527 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.054176+0000 osd.5 (osd.5) 398 : cluster [DBG] 3.c scrub starts 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.054176+0000 osd.5 (osd.5) 398 : cluster [DBG] 3.c scrub starts 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.058464+0000 osd.5 (osd.5) 399 : cluster [DBG] 3.c scrub ok 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.058464+0000 osd.5 (osd.5) 399 : cluster [DBG] 3.c scrub ok 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.288447+0000 osd.1 (osd.1) 507 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.288447+0000 osd.1 (osd.1) 507 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.290722+0000 osd.1 (osd.1) 508 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.290722+0000 osd.1 (osd.1) 508 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:39:48.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.027478+0000 mgr.y (mgr.24401) 853 : cluster [DBG] pgmap v1444: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:46.027478+0000 mgr.y (mgr.24401) 853 : cluster [DBG] pgmap v1444: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.012984+0000 osd.7 (osd.7) 526 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.012984+0000 osd.7 (osd.7) 526 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.017576+0000 osd.7 (osd.7) 527 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.017576+0000 osd.7 (osd.7) 527 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.054176+0000 osd.5 (osd.5) 398 : cluster [DBG] 3.c scrub starts 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.054176+0000 osd.5 (osd.5) 398 : cluster [DBG] 3.c scrub starts 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.058464+0000 osd.5 (osd.5) 399 : cluster [DBG] 3.c scrub ok 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.058464+0000 osd.5 (osd.5) 399 : cluster [DBG] 3.c scrub ok 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.288447+0000 osd.1 (osd.1) 507 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.288447+0000 osd.1 (osd.1) 507 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:39:48.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.290722+0000 osd.1 (osd.1) 508 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:39:48.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:47 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.290722+0000 osd.1 (osd.1) 508 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.294335+0000 osd.3 (osd.3) 555 : cluster [DBG] 6.c deep-scrub starts 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.294335+0000 osd.3 (osd.3) 555 : cluster [DBG] 6.c deep-scrub starts 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.296940+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.c deep-scrub ok 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.296940+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.c deep-scrub ok 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.392820+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.392820+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.395283+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.395283+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: audit 2024-04-26T16:39:47.457138+0000 mgr.y (mgr.24401) 854 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: audit 2024-04-26T16:39:47.457138+0000 mgr.y (mgr.24401) 854 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.937644+0000 osd.0 (osd.0) 535 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.937644+0000 osd.0 (osd.0) 535 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:39:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.940253+0000 osd.0 (osd.0) 536 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:39:49.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:48 smithi190 bash[23868]: cluster 2024-04-26T16:39:47.940253+0000 osd.0 (osd.0) 536 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:39:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.294335+0000 osd.3 (osd.3) 555 : cluster [DBG] 6.c deep-scrub starts 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.294335+0000 osd.3 (osd.3) 555 : cluster [DBG] 6.c deep-scrub starts 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.296940+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.c deep-scrub ok 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.296940+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.c deep-scrub ok 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.392820+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.392820+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.395283+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.395283+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: audit 2024-04-26T16:39:47.457138+0000 mgr.y (mgr.24401) 854 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: audit 2024-04-26T16:39:47.457138+0000 mgr.y (mgr.24401) 854 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.937644+0000 osd.0 (osd.0) 535 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.937644+0000 osd.0 (osd.0) 535 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.940253+0000 osd.0 (osd.0) 536 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:39:49.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[21463]: cluster 2024-04-26T16:39:47.940253+0000 osd.0 (osd.0) 536 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.294335+0000 osd.3 (osd.3) 555 : cluster [DBG] 6.c deep-scrub starts 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.294335+0000 osd.3 (osd.3) 555 : cluster [DBG] 6.c deep-scrub starts 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.296940+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.c deep-scrub ok 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.296940+0000 osd.3 (osd.3) 556 : cluster [DBG] 6.c deep-scrub ok 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.392820+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.392820+0000 osd.6 (osd.6) 89 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.395283+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.395283+0000 osd.6 (osd.6) 90 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: audit 2024-04-26T16:39:47.457138+0000 mgr.y (mgr.24401) 854 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:49.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: audit 2024-04-26T16:39:47.457138+0000 mgr.y (mgr.24401) 854 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:49.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.937644+0000 osd.0 (osd.0) 535 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:39:49.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.937644+0000 osd.0 (osd.0) 535 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:39:49.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.940253+0000 osd.0 (osd.0) 536 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:39:49.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:48 smithi050 bash[29234]: cluster 2024-04-26T16:39:47.940253+0000 osd.0 (osd.0) 536 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:39:49.526 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[21463]: cluster 2024-04-26T16:39:48.029341+0000 mgr.y (mgr.24401) 855 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[21463]: cluster 2024-04-26T16:39:48.029341+0000 mgr.y (mgr.24401) 855 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[21463]: cluster 2024-04-26T16:39:48.634502+0000 osd.2 (osd.2) 465 : cluster [DBG] 5.b scrub starts 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[21463]: cluster 2024-04-26T16:39:48.634502+0000 osd.2 (osd.2) 465 : cluster [DBG] 5.b scrub starts 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[21463]: cluster 2024-04-26T16:39:48.636745+0000 osd.2 (osd.2) 466 : cluster [DBG] 5.b scrub ok 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[21463]: cluster 2024-04-26T16:39:48.636745+0000 osd.2 (osd.2) 466 : cluster [DBG] 5.b scrub ok 2024-04-26T16:39:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[29234]: cluster 2024-04-26T16:39:48.029341+0000 mgr.y (mgr.24401) 855 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[29234]: cluster 2024-04-26T16:39:48.029341+0000 mgr.y (mgr.24401) 855 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[29234]: cluster 2024-04-26T16:39:48.634502+0000 osd.2 (osd.2) 465 : cluster [DBG] 5.b scrub starts 2024-04-26T16:39:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[29234]: cluster 2024-04-26T16:39:48.634502+0000 osd.2 (osd.2) 465 : cluster [DBG] 5.b scrub starts 2024-04-26T16:39:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[29234]: cluster 2024-04-26T16:39:48.636745+0000 osd.2 (osd.2) 466 : cluster [DBG] 5.b scrub ok 2024-04-26T16:39:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:49 smithi050 bash[29234]: cluster 2024-04-26T16:39:48.636745+0000 osd.2 (osd.2) 466 : cluster [DBG] 5.b scrub ok 2024-04-26T16:39:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:49 smithi190 bash[23868]: cluster 2024-04-26T16:39:48.029341+0000 mgr.y (mgr.24401) 855 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:49 smithi190 bash[23868]: cluster 2024-04-26T16:39:48.029341+0000 mgr.y (mgr.24401) 855 : cluster [DBG] pgmap v1446: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-04-26T16:39:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:49 smithi190 bash[23868]: cluster 2024-04-26T16:39:48.634502+0000 osd.2 (osd.2) 465 : cluster [DBG] 5.b scrub starts 2024-04-26T16:39:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:49 smithi190 bash[23868]: cluster 2024-04-26T16:39:48.634502+0000 osd.2 (osd.2) 465 : cluster [DBG] 5.b scrub starts 2024-04-26T16:39:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:49 smithi190 bash[23868]: cluster 2024-04-26T16:39:48.636745+0000 osd.2 (osd.2) 466 : cluster [DBG] 5.b scrub ok 2024-04-26T16:39:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:49 smithi190 bash[23868]: cluster 2024-04-26T16:39:48.636745+0000 osd.2 (osd.2) 466 : cluster [DBG] 5.b scrub ok 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.082697+0000 osd.7 (osd.7) 528 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.082697+0000 osd.7 (osd.7) 528 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.086015+0000 osd.7 (osd.7) 529 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.086015+0000 osd.7 (osd.7) 529 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.808836+0000 osd.4 (osd.4) 483 : cluster [DBG] 3.1a deep-scrub starts 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.808836+0000 osd.4 (osd.4) 483 : cluster [DBG] 3.1a deep-scrub starts 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.818000+0000 osd.4 (osd.4) 484 : cluster [DBG] 3.1a deep-scrub ok 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.818000+0000 osd.4 (osd.4) 484 : cluster [DBG] 3.1a deep-scrub ok 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.030301+0000 mgr.y (mgr.24401) 856 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.030301+0000 mgr.y (mgr.24401) 856 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.249471+0000 osd.1 (osd.1) 509 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.249471+0000 osd.1 (osd.1) 509 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:39:51.077 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.251846+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.251846+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.082697+0000 osd.7 (osd.7) 528 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.082697+0000 osd.7 (osd.7) 528 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.086015+0000 osd.7 (osd.7) 529 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.086015+0000 osd.7 (osd.7) 529 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.808836+0000 osd.4 (osd.4) 483 : cluster [DBG] 3.1a deep-scrub starts 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.808836+0000 osd.4 (osd.4) 483 : cluster [DBG] 3.1a deep-scrub starts 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.818000+0000 osd.4 (osd.4) 484 : cluster [DBG] 3.1a deep-scrub ok 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.818000+0000 osd.4 (osd.4) 484 : cluster [DBG] 3.1a deep-scrub ok 2024-04-26T16:39:51.078 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.030301+0000 mgr.y (mgr.24401) 856 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:51.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.030301+0000 mgr.y (mgr.24401) 856 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:51.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.249471+0000 osd.1 (osd.1) 509 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:39:51.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.249471+0000 osd.1 (osd.1) 509 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:39:51.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.251846+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:39:51.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:50 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.251846+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.082697+0000 osd.7 (osd.7) 528 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.082697+0000 osd.7 (osd.7) 528 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.086015+0000 osd.7 (osd.7) 529 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.086015+0000 osd.7 (osd.7) 529 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.808836+0000 osd.4 (osd.4) 483 : cluster [DBG] 3.1a deep-scrub starts 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.808836+0000 osd.4 (osd.4) 483 : cluster [DBG] 3.1a deep-scrub starts 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.818000+0000 osd.4 (osd.4) 484 : cluster [DBG] 3.1a deep-scrub ok 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.818000+0000 osd.4 (osd.4) 484 : cluster [DBG] 3.1a deep-scrub ok 2024-04-26T16:39:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.030301+0000 mgr.y (mgr.24401) 856 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.030301+0000 mgr.y (mgr.24401) 856 : cluster [DBG] pgmap v1447: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.249471+0000 osd.1 (osd.1) 509 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:39:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.249471+0000 osd.1 (osd.1) 509 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:39:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.251846+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:39:51.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:50 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.251846+0000 osd.1 (osd.1) 510 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.991024+0000 osd.0 (osd.0) 537 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.991024+0000 osd.0 (osd.0) 537 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.994181+0000 osd.0 (osd.0) 538 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:49.994181+0000 osd.0 (osd.0) 538 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.847907+0000 osd.4 (osd.4) 485 : cluster [DBG] 2.f scrub starts 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.847907+0000 osd.4 (osd.4) 485 : cluster [DBG] 2.f scrub starts 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.851144+0000 osd.4 (osd.4) 486 : cluster [DBG] 2.f scrub ok 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:50.851144+0000 osd.4 (osd.4) 486 : cluster [DBG] 2.f scrub ok 2024-04-26T16:39:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.024581+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.024581+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.027580+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.027580+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.991024+0000 osd.0 (osd.0) 537 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.991024+0000 osd.0 (osd.0) 537 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.994181+0000 osd.0 (osd.0) 538 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:49.994181+0000 osd.0 (osd.0) 538 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.847907+0000 osd.4 (osd.4) 485 : cluster [DBG] 2.f scrub starts 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.847907+0000 osd.4 (osd.4) 485 : cluster [DBG] 2.f scrub starts 2024-04-26T16:39:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.851144+0000 osd.4 (osd.4) 486 : cluster [DBG] 2.f scrub ok 2024-04-26T16:39:52.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:50.851144+0000 osd.4 (osd.4) 486 : cluster [DBG] 2.f scrub ok 2024-04-26T16:39:52.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.024581+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:39:52.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.024581+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:39:52.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.027580+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:39:52.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:51 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.027580+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:39:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.991024+0000 osd.0 (osd.0) 537 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:39:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.991024+0000 osd.0 (osd.0) 537 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:39:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.994181+0000 osd.0 (osd.0) 538 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:39:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:49.994181+0000 osd.0 (osd.0) 538 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:39:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.847907+0000 osd.4 (osd.4) 485 : cluster [DBG] 2.f scrub starts 2024-04-26T16:39:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.847907+0000 osd.4 (osd.4) 485 : cluster [DBG] 2.f scrub starts 2024-04-26T16:39:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.851144+0000 osd.4 (osd.4) 486 : cluster [DBG] 2.f scrub ok 2024-04-26T16:39:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:50.851144+0000 osd.4 (osd.4) 486 : cluster [DBG] 2.f scrub ok 2024-04-26T16:39:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.024581+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:39:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.024581+0000 osd.7 (osd.7) 530 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:39:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.027580+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:39:52.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:51 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.027580+0000 osd.7 (osd.7) 531 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:39:53.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.341844+0000 osd.6 (osd.6) 91 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:39:53.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.341844+0000 osd.6 (osd.6) 91 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:39:53.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.344499+0000 osd.6 (osd.6) 92 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:39:53.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.344499+0000 osd.6 (osd.6) 92 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:39:53.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.819467+0000 osd.7 (osd.7) 532 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:53.117 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.819467+0000 osd.7 (osd.7) 532 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.820000+0000 osd.7 (osd.7) 533 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:51.820000+0000 osd.7 (osd.7) 533 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.032298+0000 mgr.y (mgr.24401) 857 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.032298+0000 mgr.y (mgr.24401) 857 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.062777+0000 osd.5 (osd.5) 400 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.062777+0000 osd.5 (osd.5) 400 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.067099+0000 osd.5 (osd.5) 401 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:39:53.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.067099+0000 osd.5 (osd.5) 401 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.483609+0000 mon.a (mon.0) 4006 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[29234]: cluster 2024-04-26T16:39:52.483609+0000 mon.a (mon.0) 4006 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.341844+0000 osd.6 (osd.6) 91 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.341844+0000 osd.6 (osd.6) 91 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.344499+0000 osd.6 (osd.6) 92 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.344499+0000 osd.6 (osd.6) 92 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:39:53.119 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.819467+0000 osd.7 (osd.7) 532 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:53.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.819467+0000 osd.7 (osd.7) 532 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:53.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.820000+0000 osd.7 (osd.7) 533 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:53.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:51.820000+0000 osd.7 (osd.7) 533 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:53.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.032298+0000 mgr.y (mgr.24401) 857 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:53.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.032298+0000 mgr.y (mgr.24401) 857 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:53.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.062777+0000 osd.5 (osd.5) 400 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:39:53.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.062777+0000 osd.5 (osd.5) 400 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:39:53.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.067099+0000 osd.5 (osd.5) 401 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:39:53.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.067099+0000 osd.5 (osd.5) 401 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:39:53.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.483609+0000 mon.a (mon.0) 4006 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-04-26T16:39:53.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:52 smithi050 bash[21463]: cluster 2024-04-26T16:39:52.483609+0000 mon.a (mon.0) 4006 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-04-26T16:39:53.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/1404678717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006bb60 msgr2=0x7f92e006bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:53.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- 172.21.15.50:0/1404678717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006bb60 0x7f92e006bfe0 secure :-1 s=READY pgs=2544 cs=0 l=1 rev1=1 crypto rx=0x7f92d00098f0 tx=0x7f92d0009c00 comp rx=0 tx=0).stop 2024-04-26T16:39:53.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/1404678717 shutdown_connections 2024-04-26T16:39:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- 172.21.15.50:0/1404678717 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e01095c0 0x7f92e01099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- 172.21.15.50:0/1404678717 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e006c520 0x7f92e0110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- 172.21.15.50:0/1404678717 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006bb60 0x7f92e006bfe0 unknown :-1 s=CLOSED pgs=2544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/1404678717 >> 172.21.15.50:0/1404678717 conn(0x7f92e0079120 msgr2=0x7f92e0079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:53.286 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/1404678717 shutdown_connections 2024-04-26T16:39:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/1404678717 wait complete. 2024-04-26T16:39:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 librados: starting msgr at 2024-04-26T16:39:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 librados: starting objecter 2024-04-26T16:39:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 Processor -- start 2024-04-26T16:39:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- start start 2024-04-26T16:39:53.287 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 librados: setting wanted keys 2024-04-26T16:39:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 librados: calling monclient init 2024-04-26T16:39:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e006bb60 0x7f92e01026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 0x7f92e0102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e01095c0 0x7f92e0103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:53.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f92e0112b50 con 0x7f92e006c520 2024-04-26T16:39:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f92e01129d0 con 0x7f92e006bb60 2024-04-26T16:39:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f92e0112cd0 con 0x7f92e01095c0 2024-04-26T16:39:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e59a0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e01095c0 0x7f92e0103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 0x7f92e0102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:53.289 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 0x7f92e0102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52602/0 (socket says 172.21.15.50:52602) 2024-04-26T16:39:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e59a0700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e01095c0 0x7f92e0103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56448/0 (socket says 172.21.15.50:56448) 2024-04-26T16:39:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 -- 172.21.15.50:0/2636320418 learned_addr learned my addr 172.21.15.50:0/2636320418 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:39:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 -- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e01095c0 msgr2=0x7f92e0103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e01095c0 0x7f92e0103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 -- 172.21.15.50:0/2636320418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e006bb60 msgr2=0x7f92e01026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:53.290 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e519f700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e006bb60 0x7f92e01026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e006bb60 0x7f92e01026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 -- 172.21.15.50:0/2636320418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92e01a76c0 con 0x7f92e006c520 2024-04-26T16:39:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e519f700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e006bb60 0x7f92e01026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:39:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e499e700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 0x7f92e0102c30 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009230 tx=0x7f92d000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:53.291 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92d67fc700 10 client.?.objecter ms_handle_connect 0x7f92e006c520 2024-04-26T16:39:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92d67fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:39:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d000b9d0 con 0x7f92e006c520 2024-04-26T16:39:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f92d000bb70 con 0x7f92e006c520 2024-04-26T16:39:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92e01a7950 con 0x7f92e006c520 2024-04-26T16:39:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92d0004d50 con 0x7f92e006c520 2024-04-26T16:39:53.292 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 10 client.31197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:39:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92e01a7da0 con 0x7f92e006c520 2024-04-26T16:39:53.293 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.284+0000 7f92e7403700 1 librados: init done 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92e006a110 con 0x7f92e006c520 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f92d0005040 con 0x7f92e006c520 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f92cc071f80 0x7f92cc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(903..903 src has 827..903) ==== 8707+0+0 (secure 0 0 0) 0x7f92d0092900 con 0x7f92e006c520 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 10 client.31197.objecter ms_dispatch 0x7f92e0001550 osd_map(903..903 src has 827..903) 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 3 client.31197.objecter handle_osd_map got epochs [903,903] > 0 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 3 client.31197.objecter handle_osd_map decoding full epoch 903 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92e519f700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f92cc071f80 0x7f92cc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 20 client.31197.objecter dump_active .. 0 homeless 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92e519f700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f92cc071f80 0x7f92cc074440 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009b50 tx=0x7f92dc009550 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.288+0000 7f92d67fc700 10 client.31197.objecter ms_handle_connect 0x7f92cc071f80 2024-04-26T16:39:53.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.292+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f92d0013030 con 0x7f92e006c520 2024-04-26T16:39:53.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.341844+0000 osd.6 (osd.6) 91 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:39:53.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.341844+0000 osd.6 (osd.6) 91 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:39:53.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.344499+0000 osd.6 (osd.6) 92 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.344499+0000 osd.6 (osd.6) 92 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.819467+0000 osd.7 (osd.7) 532 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.819467+0000 osd.7 (osd.7) 532 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.820000+0000 osd.7 (osd.7) 533 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:51.820000+0000 osd.7 (osd.7) 533 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.032298+0000 mgr.y (mgr.24401) 857 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.032298+0000 mgr.y (mgr.24401) 857 : cluster [DBG] pgmap v1448: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.062777+0000 osd.5 (osd.5) 400 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.062777+0000 osd.5 (osd.5) 400 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.067099+0000 osd.5 (osd.5) 401 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.067099+0000 osd.5 (osd.5) 401 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:39:53.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.483609+0000 mon.a (mon.0) 4006 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-04-26T16:39:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:52 smithi190 bash[23868]: cluster 2024-04-26T16:39:52.483609+0000 mon.a (mon.0) 4006 : cluster [DBG] osdmap e903: 8 total, 8 up, 8 in 2024-04-26T16:39:53.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.536+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 2, "weight": 1.0} v 0) -- 0x7f92e01a7f70 con 0x7f92e006c520 2024-04-26T16:39:53.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.860+0000 7f92d67fc700 1 -- 172.21.15.50:0/2636320418 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 2, "weight": 1.0}]=0 reweighted osd.2 to 1 (10000) v904) ==== 113+0+0 (secure 0 0 0) 0x7f92d005e550 con 0x7f92e006c520 2024-04-26T16:39:53.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 10 librados: watch_flush enter 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 10 librados: watch_flush exit 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 20 client.31197.objecter shutdown clearing up homeless session... 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 10 client.31197.objecter successfully canceled tick 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f92cc071f80 msgr2=0x7f92cc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f92cc071f80 0x7f92cc074440 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f92dc009b50 tx=0x7f92dc009550 comp rx=0 tx=0).stop 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 msgr2=0x7f92e0102c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 0x7f92e0102c30 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7f92d0009230 tx=0x7f92d000fa00 comp rx=0 tx=0).stop 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 shutdown_connections 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f92cc071f80 0x7f92cc074440 unknown :-1 s=CLOSED pgs=3860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92e006bb60 0x7f92e01026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f92e01095c0 0x7f92e0103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 --2- 172.21.15.50:0/2636320418 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f92e006c520 0x7f92e0102c30 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:39:53.869 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 >> 172.21.15.50:0/2636320418 conn(0x7f92e0079120 msgr2=0x7f92e0100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:39:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 shutdown_connections 2024-04-26T16:39:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 -- 172.21.15.50:0/2636320418 wait complete. 2024-04-26T16:39:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:39:53.868+0000 7f92e7403700 1 librados: shutdown 2024-04-26T16:39:53.870 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.2 to 1 (10000) 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[29234]: cluster 2024-04-26T16:39:53.238052+0000 osd.1 (osd.1) 511 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[29234]: cluster 2024-04-26T16:39:53.238052+0000 osd.1 (osd.1) 511 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[29234]: cluster 2024-04-26T16:39:53.239794+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[29234]: cluster 2024-04-26T16:39:53.239794+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[29234]: audit 2024-04-26T16:39:53.539746+0000 mon.a (mon.0) 4007 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[29234]: audit 2024-04-26T16:39:53.539746+0000 mon.a (mon.0) 4007 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:39:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[21463]: cluster 2024-04-26T16:39:53.238052+0000 osd.1 (osd.1) 511 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:39:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[21463]: cluster 2024-04-26T16:39:53.238052+0000 osd.1 (osd.1) 511 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:39:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[21463]: cluster 2024-04-26T16:39:53.239794+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:39:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[21463]: cluster 2024-04-26T16:39:53.239794+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:39:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[21463]: audit 2024-04-26T16:39:53.539746+0000 mon.a (mon.0) 4007 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:39:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:53 smithi050 bash[21463]: audit 2024-04-26T16:39:53.539746+0000 mon.a (mon.0) 4007 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:39:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:53 smithi190 bash[23868]: cluster 2024-04-26T16:39:53.238052+0000 osd.1 (osd.1) 511 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:39:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:53 smithi190 bash[23868]: cluster 2024-04-26T16:39:53.238052+0000 osd.1 (osd.1) 511 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:39:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:53 smithi190 bash[23868]: cluster 2024-04-26T16:39:53.239794+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:39:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:53 smithi190 bash[23868]: cluster 2024-04-26T16:39:53.239794+0000 osd.1 (osd.1) 512 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:39:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:53 smithi190 bash[23868]: audit 2024-04-26T16:39:53.539746+0000 mon.a (mon.0) 4007 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:39:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:53 smithi190 bash[23868]: audit 2024-04-26T16:39:53.539746+0000 mon.a (mon.0) 4007 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:39:54.630 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:39:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:39:54.839 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 2 1 2024-04-26T16:39:55.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: audit 2024-04-26T16:39:53.864994+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:39:55.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: audit 2024-04-26T16:39:53.864994+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:53.873025+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:53.873025+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.033447+0000 mgr.y (mgr.24401) 858 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.033447+0000 mgr.y (mgr.24401) 858 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.072260+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.072260+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.077015+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.077015+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: audit 2024-04-26T16:39:53.864994+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:39:55.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: audit 2024-04-26T16:39:53.864994+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:53.873025+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:53.873025+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.033447+0000 mgr.y (mgr.24401) 858 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.033447+0000 mgr.y (mgr.24401) 858 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.072260+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.072260+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.077015+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-04-26T16:39:55.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:54 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.077015+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: audit 2024-04-26T16:39:53.864994+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: audit 2024-04-26T16:39:53.864994+0000 mon.a (mon.0) 4008 : audit [INF] from='client.? 172.21.15.50:0/2636320418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:53.873025+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:53.873025+0000 mon.a (mon.0) 4009 : cluster [DBG] osdmap e904: 8 total, 8 up, 8 in 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.033447+0000 mgr.y (mgr.24401) 858 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.033447+0000 mgr.y (mgr.24401) 858 : cluster [DBG] pgmap v1451: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.072260+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-04-26T16:39:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.072260+0000 osd.5 (osd.5) 402 : cluster [DBG] 6.d scrub starts 2024-04-26T16:39:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.077015+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-04-26T16:39:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:54 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.077015+0000 osd.5 (osd.5) 403 : cluster [DBG] 6.d scrub ok 2024-04-26T16:39:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.105154+0000 osd.7 (osd.7) 534 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:39:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.105154+0000 osd.7 (osd.7) 534 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.111628+0000 osd.7 (osd.7) 535 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.111628+0000 osd.7 (osd.7) 535 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.637757+0000 osd.2 (osd.2) 467 : cluster [DBG] 11.d scrub starts 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.637757+0000 osd.2 (osd.2) 467 : cluster [DBG] 11.d scrub starts 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.641818+0000 osd.2 (osd.2) 468 : cluster [DBG] 11.d scrub ok 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.641818+0000 osd.2 (osd.2) 468 : cluster [DBG] 11.d scrub ok 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.036214+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.f scrub starts 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.036214+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.f scrub starts 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.038718+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.f scrub ok 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.038718+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.f scrub ok 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.105154+0000 osd.7 (osd.7) 534 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:39:56.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.105154+0000 osd.7 (osd.7) 534 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.111628+0000 osd.7 (osd.7) 535 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.111628+0000 osd.7 (osd.7) 535 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.637757+0000 osd.2 (osd.2) 467 : cluster [DBG] 11.d scrub starts 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.637757+0000 osd.2 (osd.2) 467 : cluster [DBG] 11.d scrub starts 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.641818+0000 osd.2 (osd.2) 468 : cluster [DBG] 11.d scrub ok 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.641818+0000 osd.2 (osd.2) 468 : cluster [DBG] 11.d scrub ok 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.036214+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.f scrub starts 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.036214+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.f scrub starts 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.038718+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.f scrub ok 2024-04-26T16:39:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:55 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.038718+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.f scrub ok 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.105154+0000 osd.7 (osd.7) 534 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.105154+0000 osd.7 (osd.7) 534 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.111628+0000 osd.7 (osd.7) 535 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.111628+0000 osd.7 (osd.7) 535 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.637757+0000 osd.2 (osd.2) 467 : cluster [DBG] 11.d scrub starts 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.637757+0000 osd.2 (osd.2) 467 : cluster [DBG] 11.d scrub starts 2024-04-26T16:39:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.641818+0000 osd.2 (osd.2) 468 : cluster [DBG] 11.d scrub ok 2024-04-26T16:39:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.641818+0000 osd.2 (osd.2) 468 : cluster [DBG] 11.d scrub ok 2024-04-26T16:39:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.036214+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.f scrub starts 2024-04-26T16:39:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.036214+0000 osd.5 (osd.5) 404 : cluster [DBG] 5.f scrub starts 2024-04-26T16:39:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.038718+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.f scrub ok 2024-04-26T16:39:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:55 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.038718+0000 osd.5 (osd.5) 405 : cluster [DBG] 5.f scrub ok 2024-04-26T16:39:57.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.982402+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:39:57.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.982402+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.985335+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:54.985335+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.130528+0000 osd.7 (osd.7) 536 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.130528+0000 osd.7 (osd.7) 536 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.132753+0000 osd.7 (osd.7) 537 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.132753+0000 osd.7 (osd.7) 537 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.369698+0000 osd.3 (osd.3) 557 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.369698+0000 osd.3 (osd.3) 557 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.375250+0000 osd.3 (osd.3) 558 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:55.375250+0000 osd.3 (osd.3) 558 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.034237+0000 mgr.y (mgr.24401) 859 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.034237+0000 mgr.y (mgr.24401) 859 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.081568+0000 osd.5 (osd.5) 406 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.081568+0000 osd.5 (osd.5) 406 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:39:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.084068+0000 osd.5 (osd.5) 407 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:39:57.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.084068+0000 osd.5 (osd.5) 407 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:39:57.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.225502+0000 osd.1 (osd.1) 513 : cluster [DBG] 5.c scrub starts 2024-04-26T16:39:57.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.225502+0000 osd.1 (osd.1) 513 : cluster [DBG] 5.c scrub starts 2024-04-26T16:39:57.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.227969+0000 osd.1 (osd.1) 514 : cluster [DBG] 5.c scrub ok 2024-04-26T16:39:57.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:56 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.227969+0000 osd.1 (osd.1) 514 : cluster [DBG] 5.c scrub ok 2024-04-26T16:39:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.982402+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.982402+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.985335+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:54.985335+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.130528+0000 osd.7 (osd.7) 536 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.130528+0000 osd.7 (osd.7) 536 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.132753+0000 osd.7 (osd.7) 537 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.132753+0000 osd.7 (osd.7) 537 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.369698+0000 osd.3 (osd.3) 557 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.369698+0000 osd.3 (osd.3) 557 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.375250+0000 osd.3 (osd.3) 558 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:55.375250+0000 osd.3 (osd.3) 558 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.034237+0000 mgr.y (mgr.24401) 859 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.034237+0000 mgr.y (mgr.24401) 859 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.081568+0000 osd.5 (osd.5) 406 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.081568+0000 osd.5 (osd.5) 406 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.084068+0000 osd.5 (osd.5) 407 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.084068+0000 osd.5 (osd.5) 407 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.225502+0000 osd.1 (osd.1) 513 : cluster [DBG] 5.c scrub starts 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.225502+0000 osd.1 (osd.1) 513 : cluster [DBG] 5.c scrub starts 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.227969+0000 osd.1 (osd.1) 514 : cluster [DBG] 5.c scrub ok 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.227969+0000 osd.1 (osd.1) 514 : cluster [DBG] 5.c scrub ok 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.982402+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.982402+0000 osd.0 (osd.0) 539 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:39:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.985335+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:54.985335+0000 osd.0 (osd.0) 540 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.130528+0000 osd.7 (osd.7) 536 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.130528+0000 osd.7 (osd.7) 536 : cluster [DBG] 5.4 deep-scrub starts 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.132753+0000 osd.7 (osd.7) 537 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.132753+0000 osd.7 (osd.7) 537 : cluster [DBG] 5.4 deep-scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.369698+0000 osd.3 (osd.3) 557 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.369698+0000 osd.3 (osd.3) 557 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.375250+0000 osd.3 (osd.3) 558 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:55.375250+0000 osd.3 (osd.3) 558 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.034237+0000 mgr.y (mgr.24401) 859 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.034237+0000 mgr.y (mgr.24401) 859 : cluster [DBG] pgmap v1452: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.081568+0000 osd.5 (osd.5) 406 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.081568+0000 osd.5 (osd.5) 406 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.084068+0000 osd.5 (osd.5) 407 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:39:57.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.084068+0000 osd.5 (osd.5) 407 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:39:57.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.225502+0000 osd.1 (osd.1) 513 : cluster [DBG] 5.c scrub starts 2024-04-26T16:39:57.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.225502+0000 osd.1 (osd.1) 513 : cluster [DBG] 5.c scrub starts 2024-04-26T16:39:57.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.227969+0000 osd.1 (osd.1) 514 : cluster [DBG] 5.c scrub ok 2024-04-26T16:39:57.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:56 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.227969+0000 osd.1 (osd.1) 514 : cluster [DBG] 5.c scrub ok 2024-04-26T16:39:57.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.972313+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.972313+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.185204+0000 osd.1 (osd.1) 515 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.185204+0000 osd.1 (osd.1) 515 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.187599+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.187599+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: audit 2024-04-26T16:39:57.465120+0000 mgr.y (mgr.24401) 860 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:57 smithi190 bash[23868]: audit 2024-04-26T16:39:57.465120+0000 mgr.y (mgr.24401) 860 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.972313+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.972313+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.185204+0000 osd.1 (osd.1) 515 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.185204+0000 osd.1 (osd.1) 515 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.187599+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.187599+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:39:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: audit 2024-04-26T16:39:57.465120+0000 mgr.y (mgr.24401) 860 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[21463]: audit 2024-04-26T16:39:57.465120+0000 mgr.y (mgr.24401) 860 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.972313+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.972313+0000 osd.0 (osd.0) 541 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.185204+0000 osd.1 (osd.1) 515 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.185204+0000 osd.1 (osd.1) 515 : cluster [DBG] 2.d deep-scrub starts 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.187599+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.187599+0000 osd.1 (osd.1) 516 : cluster [DBG] 2.d deep-scrub ok 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: audit 2024-04-26T16:39:57.465120+0000 mgr.y (mgr.24401) 860 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:57 smithi050 bash[29234]: audit 2024-04-26T16:39:57.465120+0000 mgr.y (mgr.24401) 860 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:39:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.974857+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:39:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:56.974857+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:39:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.716352+0000 osd.2 (osd.2) 469 : cluster [DBG] 5.a scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.716352+0000 osd.2 (osd.2) 469 : cluster [DBG] 5.a scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.718880+0000 osd.2 (osd.2) 470 : cluster [DBG] 5.a scrub ok 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.718880+0000 osd.2 (osd.2) 470 : cluster [DBG] 5.a scrub ok 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.949424+0000 osd.0 (osd.0) 543 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.949424+0000 osd.0 (osd.0) 543 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.952766+0000 osd.0 (osd.0) 544 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:57.952766+0000 osd.0 (osd.0) 544 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.036149+0000 mgr.y (mgr.24401) 861 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.036149+0000 mgr.y (mgr.24401) 861 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.053963+0000 osd.7 (osd.7) 538 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.053963+0000 osd.7 (osd.7) 538 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.058371+0000 osd.7 (osd.7) 539 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.058371+0000 osd.7 (osd.7) 539 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.108770+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:39:59.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.108770+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:39:59.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.111144+0000 osd.5 (osd.5) 409 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:39:59.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: cluster 2024-04-26T16:39:58.111144+0000 osd.5 (osd.5) 409 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:39:59.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: audit 2024-04-26T16:39:58.333369+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:59.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:58 smithi190 bash[23868]: audit 2024-04-26T16:39:58.333369+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.974857+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:56.974857+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.716352+0000 osd.2 (osd.2) 469 : cluster [DBG] 5.a scrub starts 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.716352+0000 osd.2 (osd.2) 469 : cluster [DBG] 5.a scrub starts 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.718880+0000 osd.2 (osd.2) 470 : cluster [DBG] 5.a scrub ok 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.718880+0000 osd.2 (osd.2) 470 : cluster [DBG] 5.a scrub ok 2024-04-26T16:39:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.949424+0000 osd.0 (osd.0) 543 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.949424+0000 osd.0 (osd.0) 543 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.952766+0000 osd.0 (osd.0) 544 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:57.952766+0000 osd.0 (osd.0) 544 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.036149+0000 mgr.y (mgr.24401) 861 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.036149+0000 mgr.y (mgr.24401) 861 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.053963+0000 osd.7 (osd.7) 538 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.053963+0000 osd.7 (osd.7) 538 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.058371+0000 osd.7 (osd.7) 539 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.058371+0000 osd.7 (osd.7) 539 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.108770+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.108770+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.111144+0000 osd.5 (osd.5) 409 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: cluster 2024-04-26T16:39:58.111144+0000 osd.5 (osd.5) 409 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:39:59.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: audit 2024-04-26T16:39:58.333369+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[21463]: audit 2024-04-26T16:39:58.333369+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.974857+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:56.974857+0000 osd.0 (osd.0) 542 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.716352+0000 osd.2 (osd.2) 469 : cluster [DBG] 5.a scrub starts 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.716352+0000 osd.2 (osd.2) 469 : cluster [DBG] 5.a scrub starts 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.718880+0000 osd.2 (osd.2) 470 : cluster [DBG] 5.a scrub ok 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.718880+0000 osd.2 (osd.2) 470 : cluster [DBG] 5.a scrub ok 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.949424+0000 osd.0 (osd.0) 543 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.949424+0000 osd.0 (osd.0) 543 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.952766+0000 osd.0 (osd.0) 544 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:57.952766+0000 osd.0 (osd.0) 544 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:39:59.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.036149+0000 mgr.y (mgr.24401) 861 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.036149+0000 mgr.y (mgr.24401) 861 : cluster [DBG] pgmap v1453: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.053963+0000 osd.7 (osd.7) 538 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.053963+0000 osd.7 (osd.7) 538 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.058371+0000 osd.7 (osd.7) 539 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.058371+0000 osd.7 (osd.7) 539 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.108770+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.108770+0000 osd.5 (osd.5) 408 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.111144+0000 osd.5 (osd.5) 409 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: cluster 2024-04-26T16:39:58.111144+0000 osd.5 (osd.5) 409 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: audit 2024-04-26T16:39:58.333369+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:59.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:58 smithi050 bash[29234]: audit 2024-04-26T16:39:58.333369+0000 mon.c (mon.2) 929 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:39:59.632 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.061113+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.061113+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.063469+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.063469+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.108606+0000 osd.5 (osd.5) 410 : cluster [DBG] 1.a scrub starts 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.108606+0000 osd.5 (osd.5) 410 : cluster [DBG] 1.a scrub starts 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.111080+0000 osd.5 (osd.5) 411 : cluster [DBG] 1.a scrub ok 2024-04-26T16:40:00.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[29234]: cluster 2024-04-26T16:39:59.111080+0000 osd.5 (osd.5) 411 : cluster [DBG] 1.a scrub ok 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.061113+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.061113+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.063469+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.063469+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.108606+0000 osd.5 (osd.5) 410 : cluster [DBG] 1.a scrub starts 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.108606+0000 osd.5 (osd.5) 410 : cluster [DBG] 1.a scrub starts 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.111080+0000 osd.5 (osd.5) 411 : cluster [DBG] 1.a scrub ok 2024-04-26T16:40:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:39:59 smithi050 bash[21463]: cluster 2024-04-26T16:39:59.111080+0000 osd.5 (osd.5) 411 : cluster [DBG] 1.a scrub ok 2024-04-26T16:40:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.061113+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:40:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.061113+0000 osd.7 (osd.7) 540 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:40:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.063469+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:40:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.063469+0000 osd.7 (osd.7) 541 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:40:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.108606+0000 osd.5 (osd.5) 410 : cluster [DBG] 1.a scrub starts 2024-04-26T16:40:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.108606+0000 osd.5 (osd.5) 410 : cluster [DBG] 1.a scrub starts 2024-04-26T16:40:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.111080+0000 osd.5 (osd.5) 411 : cluster [DBG] 1.a scrub ok 2024-04-26T16:40:00.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:39:59 smithi190 bash[23868]: cluster 2024-04-26T16:39:59.111080+0000 osd.5 (osd.5) 411 : cluster [DBG] 1.a scrub ok 2024-04-26T16:40:01.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000134+0000 mon.a (mon.0) 4010 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000134+0000 mon.a (mon.0) 4010 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000191+0000 mon.a (mon.0) 4011 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.314 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000191+0000 mon.a (mon.0) 4011 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000242+0000 mon.a (mon.0) 4012 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000242+0000 mon.a (mon.0) 4012 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000307+0000 mon.a (mon.0) 4013 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.000307+0000 mon.a (mon.0) 4013 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.037082+0000 mgr.y (mgr.24401) 862 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.037082+0000 mgr.y (mgr.24401) 862 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.064006+0000 osd.7 (osd.7) 542 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.064006+0000 osd.7 (osd.7) 542 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.066500+0000 osd.7 (osd.7) 543 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.066500+0000 osd.7 (osd.7) 543 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.080796+0000 osd.5 (osd.5) 412 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.080796+0000 osd.5 (osd.5) 412 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.083121+0000 osd.5 (osd.5) 413 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.083121+0000 osd.5 (osd.5) 413 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:40:01.315 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.164267+0000 osd.1 (osd.1) 517 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.164267+0000 osd.1 (osd.1) 517 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.166727+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.166727+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000134+0000 mon.a (mon.0) 4010 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000134+0000 mon.a (mon.0) 4010 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000191+0000 mon.a (mon.0) 4011 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000191+0000 mon.a (mon.0) 4011 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000242+0000 mon.a (mon.0) 4012 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000242+0000 mon.a (mon.0) 4012 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000307+0000 mon.a (mon.0) 4013 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:40:01.316 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.000307+0000 mon.a (mon.0) 4013 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.037082+0000 mgr.y (mgr.24401) 862 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.037082+0000 mgr.y (mgr.24401) 862 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.064006+0000 osd.7 (osd.7) 542 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.064006+0000 osd.7 (osd.7) 542 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.066500+0000 osd.7 (osd.7) 543 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.066500+0000 osd.7 (osd.7) 543 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.080796+0000 osd.5 (osd.5) 412 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.080796+0000 osd.5 (osd.5) 412 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.083121+0000 osd.5 (osd.5) 413 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.083121+0000 osd.5 (osd.5) 413 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.164267+0000 osd.1 (osd.1) 517 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.164267+0000 osd.1 (osd.1) 517 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.166727+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:40:01.317 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.166727+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:40:01.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000134+0000 mon.a (mon.0) 4010 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000134+0000 mon.a (mon.0) 4010 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000191+0000 mon.a (mon.0) 4011 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000191+0000 mon.a (mon.0) 4011 : cluster [WRN] [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-04-26T16:40:01.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000242+0000 mon.a (mon.0) 4012 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:40:01.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000242+0000 mon.a (mon.0) 4012 : cluster [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000307+0000 mon.a (mon.0) 4013 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.000307+0000 mon.a (mon.0) 4013 : cluster [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.037082+0000 mgr.y (mgr.24401) 862 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.037082+0000 mgr.y (mgr.24401) 862 : cluster [DBG] pgmap v1454: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.064006+0000 osd.7 (osd.7) 542 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.064006+0000 osd.7 (osd.7) 542 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.066500+0000 osd.7 (osd.7) 543 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.066500+0000 osd.7 (osd.7) 543 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.080796+0000 osd.5 (osd.5) 412 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.080796+0000 osd.5 (osd.5) 412 : cluster [DBG] 2.12 deep-scrub starts 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.083121+0000 osd.5 (osd.5) 413 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.083121+0000 osd.5 (osd.5) 413 : cluster [DBG] 2.12 deep-scrub ok 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.164267+0000 osd.1 (osd.1) 517 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:40:01.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.164267+0000 osd.1 (osd.1) 517 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:40:01.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.166727+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:40:01.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.166727+0000 osd.1 (osd.1) 518 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:40:02.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.192784+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:40:02.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.192784+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:40:02.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.195523+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:40:02.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:00.195523+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:40:02.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.023718+0000 osd.7 (osd.7) 544 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:40:02.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.023718+0000 osd.7 (osd.7) 544 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:40:02.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.026670+0000 osd.7 (osd.7) 545 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:40:02.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.026670+0000 osd.7 (osd.7) 545 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:40:02.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.198884+0000 osd.1 (osd.1) 519 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:40:02.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.198884+0000 osd.1 (osd.1) 519 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:40:02.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.201960+0000 osd.1 (osd.1) 520 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:40:02.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.201960+0000 osd.1 (osd.1) 520 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:40:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.192784+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:40:02.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.192784+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.195523+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:00.195523+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.023718+0000 osd.7 (osd.7) 544 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.023718+0000 osd.7 (osd.7) 544 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.026670+0000 osd.7 (osd.7) 545 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.026670+0000 osd.7 (osd.7) 545 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.198884+0000 osd.1 (osd.1) 519 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.198884+0000 osd.1 (osd.1) 519 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.201960+0000 osd.1 (osd.1) 520 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.201960+0000 osd.1 (osd.1) 520 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:40:02.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.192784+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.192784+0000 osd.6 (osd.6) 93 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.195523+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:00.195523+0000 osd.6 (osd.6) 94 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.023718+0000 osd.7 (osd.7) 544 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.023718+0000 osd.7 (osd.7) 544 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.026670+0000 osd.7 (osd.7) 545 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.026670+0000 osd.7 (osd.7) 545 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.198884+0000 osd.1 (osd.1) 519 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.198884+0000 osd.1 (osd.1) 519 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.201960+0000 osd.1 (osd.1) 520 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:40:02.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.201960+0000 osd.1 (osd.1) 520 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:40:03.231 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/590541882 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf481025b0 msgr2=0x7fdf48102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- 172.21.15.50:0/590541882 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf481025b0 0x7fdf48102990 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009fc0 tx=0x7fdf3800b330 comp rx=0 tx=0).stop 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/590541882 shutdown_connections 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- 172.21.15.50:0/590541882 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf48107640 0x7fdf4810f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- 172.21.15.50:0/590541882 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf481025b0 0x7fdf48102990 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- 172.21.15.50:0/590541882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf48102f60 0x7fdf48106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/590541882 >> 172.21.15.50:0/590541882 conn(0x7fdf480792d0 msgr2=0x7fdf480796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/590541882 shutdown_connections 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/590541882 wait complete. 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 librados: starting msgr at 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 librados: starting objecter 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 Processor -- start 2024-04-26T16:40:03.232 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- start start 2024-04-26T16:40:03.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 librados: setting wanted keys 2024-04-26T16:40:03.233 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 librados: calling monclient init 2024-04-26T16:40:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf481025b0 0x7fdf48199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 0x7fdf481997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf48107640 0x7fdf4819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fdf481119d0 con 0x7fdf48107640 2024-04-26T16:40:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fdf48111850 con 0x7fdf481025b0 2024-04-26T16:40:03.234 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fdf48111b50 con 0x7fdf48102f60 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 0x7fdf481997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf477fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf48107640 0x7fdf4819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 0x7fdf481997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:40872/0 (socket says 172.21.15.50:40872) 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 -- 172.21.15.50:0/2268035874 learned_addr learned my addr 172.21.15.50:0/2268035874 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf46ffd700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf481025b0 0x7fdf48199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 -- 172.21.15.50:0/2268035874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf481025b0 msgr2=0x7fdf48199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf481025b0 0x7fdf48199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 -- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf48107640 msgr2=0x7fdf4819db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf48107640 0x7fdf4819db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 -- 172.21.15.50:0/2268035874 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4819e240 con 0x7fdf48102f60 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf467fc700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 0x7fdf481997c0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000b500 tx=0x7fdf3000b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4c87e700 10 client.?.objecter ms_handle_connect 0x7fdf48102f60 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4c87e700 10 client.?.objecter resend_mon_ops 2024-04-26T16:40:03.235 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30015020 con 0x7fdf48102f60 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fdf30011c90 con 0x7fdf48102f60 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf30010b40 con 0x7fdf48102f60 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4819e530 con 0x7fdf48102f60 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 10 client.41822.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf481a5db0 con 0x7fdf48102f60 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.232+0000 7fdf4d880700 1 librados: init done 2024-04-26T16:40:03.236 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4806a110 con 0x7fdf48102f60 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fdf30018020 con 0x7fdf48102f60 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdf34071f80 0x7fdf34074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(904..904 src has 854..904) ==== 8707+0+0 (secure 0 0 0) 0x7fdf30090f00 con 0x7fdf48102f60 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 10 client.41822.objecter ms_dispatch 0x7fdf48001550 osd_map(904..904 src has 854..904) 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 3 client.41822.objecter handle_osd_map got epochs [904,904] > 0 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 3 client.41822.objecter handle_osd_map decoding full epoch 904 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf46ffd700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdf34071f80 0x7fdf34074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 20 client.41822.objecter dump_active .. 0 homeless 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf46ffd700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdf34071f80 0x7fdf34074440 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009790 tx=0x7fdf380096c0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:03.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.236+0000 7fdf4c87e700 10 client.41822.objecter ms_handle_connect 0x7fdf34071f80 2024-04-26T16:40:03.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.240+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fdf3001f020 con 0x7fdf48102f60 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.784754+0000 osd.2 (osd.2) 471 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.784754+0000 osd.2 (osd.2) 471 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.787526+0000 osd.2 (osd.2) 472 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.787526+0000 osd.2 (osd.2) 472 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.873641+0000 osd.0 (osd.0) 545 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.873641+0000 osd.0 (osd.0) 545 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.875976+0000 osd.0 (osd.0) 546 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:01.875976+0000 osd.0 (osd.0) 546 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:40:03.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:02.038196+0000 mgr.y (mgr.24401) 863 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:03.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:02.038196+0000 mgr.y (mgr.24401) 863 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:03.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:02.135803+0000 osd.5 (osd.5) 414 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:40:03.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:02.135803+0000 osd.5 (osd.5) 414 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:40:03.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:02.139452+0000 osd.5 (osd.5) 415 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:40:03.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:02 smithi190 bash[23868]: cluster 2024-04-26T16:40:02.139452+0000 osd.5 (osd.5) 415 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.784754+0000 osd.2 (osd.2) 471 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.784754+0000 osd.2 (osd.2) 471 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.787526+0000 osd.2 (osd.2) 472 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.787526+0000 osd.2 (osd.2) 472 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.873641+0000 osd.0 (osd.0) 545 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.873641+0000 osd.0 (osd.0) 545 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.875976+0000 osd.0 (osd.0) 546 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:40:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:01.875976+0000 osd.0 (osd.0) 546 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:02.038196+0000 mgr.y (mgr.24401) 863 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:02.038196+0000 mgr.y (mgr.24401) 863 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:02.135803+0000 osd.5 (osd.5) 414 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:02.135803+0000 osd.5 (osd.5) 414 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:02.139452+0000 osd.5 (osd.5) 415 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[21463]: cluster 2024-04-26T16:40:02.139452+0000 osd.5 (osd.5) 415 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.784754+0000 osd.2 (osd.2) 471 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.784754+0000 osd.2 (osd.2) 471 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.787526+0000 osd.2 (osd.2) 472 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.787526+0000 osd.2 (osd.2) 472 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:40:03.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.873641+0000 osd.0 (osd.0) 545 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.873641+0000 osd.0 (osd.0) 545 : cluster [DBG] 6.9 deep-scrub starts 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.875976+0000 osd.0 (osd.0) 546 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:01.875976+0000 osd.0 (osd.0) 546 : cluster [DBG] 6.9 deep-scrub ok 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:02.038196+0000 mgr.y (mgr.24401) 863 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:02.038196+0000 mgr.y (mgr.24401) 863 : cluster [DBG] pgmap v1455: 174 pgs: 174 active+clean; 583 KiB data, 782 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:02.135803+0000 osd.5 (osd.5) 414 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:02.135803+0000 osd.5 (osd.5) 414 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:02.139452+0000 osd.5 (osd.5) 415 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:40:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:02 smithi050 bash[29234]: cluster 2024-04-26T16:40:02.139452+0000 osd.5 (osd.5) 415 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:40:03.481 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.480+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} v 0) -- 0x7fdf4819a590 con 0x7fdf48102f60 2024-04-26T16:40:03.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.936+0000 7fdf4c87e700 1 -- 172.21.15.50:0/2268035874 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]=0 set osd.2 primary-affinity to 1 (8655362) v905) ==== 133+0+0 (secure 0 0 0) 0x7fdf30060800 con 0x7fdf48102f60 2024-04-26T16:40:03.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 10 librados: watch_flush enter 2024-04-26T16:40:03.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 10 librados: watch_flush exit 2024-04-26T16:40:03.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 20 client.41822.objecter shutdown clearing up homeless session... 2024-04-26T16:40:03.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 10 client.41822.objecter successfully canceled tick 2024-04-26T16:40:03.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdf34071f80 msgr2=0x7fdf34074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:03.941 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdf34071f80 0x7fdf34074440 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7fdf38009790 tx=0x7fdf380096c0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 msgr2=0x7fdf481997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 0x7fdf481997c0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000b500 tx=0x7fdf3000b810 comp rx=0 tx=0).stop 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 shutdown_connections 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fdf34071f80 0x7fdf34074440 unknown :-1 s=CLOSED pgs=3861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdf481025b0 0x7fdf48199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fdf48107640 0x7fdf4819db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 --2- 172.21.15.50:0/2268035874 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fdf48102f60 0x7fdf481997c0 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 >> 172.21.15.50:0/2268035874 conn(0x7fdf480792d0 msgr2=0x7fdf4806bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:03.942 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 shutdown_connections 2024-04-26T16:40:03.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 -- 172.21.15.50:0/2268035874 wait complete. 2024-04-26T16:40:03.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:03.940+0000 7fdf4d880700 1 librados: shutdown 2024-04-26T16:40:03.943 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.032796+0000 osd.2 (osd.2) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.032796+0000 osd.2 (osd.2) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.033161+0000 osd.2 (osd.2) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.033161+0000 osd.2 (osd.2) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: audit 2024-04-26T16:40:03.486382+0000 mon.c (mon.2) 930 : audit [INF] from='client.? 172.21.15.50:0/2268035874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: audit 2024-04-26T16:40:03.486382+0000 mon.c (mon.2) 930 : audit [INF] from='client.? 172.21.15.50:0/2268035874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: audit 2024-04-26T16:40:03.486940+0000 mon.a (mon.0) 4014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.222 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[29234]: audit 2024-04-26T16:40:03.486940+0000 mon.a (mon.0) 4014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.032796+0000 osd.2 (osd.2) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.032796+0000 osd.2 (osd.2) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.033161+0000 osd.2 (osd.2) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.033161+0000 osd.2 (osd.2) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: audit 2024-04-26T16:40:03.486382+0000 mon.c (mon.2) 930 : audit [INF] from='client.? 172.21.15.50:0/2268035874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: audit 2024-04-26T16:40:03.486382+0000 mon.c (mon.2) 930 : audit [INF] from='client.? 172.21.15.50:0/2268035874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: audit 2024-04-26T16:40:03.486940+0000 mon.a (mon.0) 4014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.223 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:03 smithi050 bash[21463]: audit 2024-04-26T16:40:03.486940+0000 mon.a (mon.0) 4014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.032796+0000 osd.2 (osd.2) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.032796+0000 osd.2 (osd.2) 473 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.033161+0000 osd.2 (osd.2) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.033161+0000 osd.2 (osd.2) 474 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: audit 2024-04-26T16:40:03.486382+0000 mon.c (mon.2) 930 : audit [INF] from='client.? 172.21.15.50:0/2268035874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: audit 2024-04-26T16:40:03.486382+0000 mon.c (mon.2) 930 : audit [INF] from='client.? 172.21.15.50:0/2268035874' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: audit 2024-04-26T16:40:03.486940+0000 mon.a (mon.0) 4014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:03 smithi190 bash[23868]: audit 2024-04-26T16:40:03.486940+0000 mon.a (mon.0) 4014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-04-26T16:40:04.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:40:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:40:04.878 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 3 1 2024-04-26T16:40:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.251144+0000 osd.3 (osd.3) 559 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:40:05.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.251144+0000 osd.3 (osd.3) 559 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:40:05.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.252958+0000 osd.3 (osd.3) 560 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:40:05.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.252958+0000 osd.3 (osd.3) 560 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:40:05.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.784115+0000 osd.2 (osd.2) 475 : cluster [DBG] 6.f scrub starts 2024-04-26T16:40:05.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.784115+0000 osd.2 (osd.2) 475 : cluster [DBG] 6.f scrub starts 2024-04-26T16:40:05.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.786730+0000 osd.2 (osd.2) 476 : cluster [DBG] 6.f scrub ok 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.786730+0000 osd.2 (osd.2) 476 : cluster [DBG] 6.f scrub ok 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: audit 2024-04-26T16:40:03.932429+0000 mon.a (mon.0) 4015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: audit 2024-04-26T16:40:03.932429+0000 mon.a (mon.0) 4015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.940217+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:03.940217+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.038757+0000 mgr.y (mgr.24401) 864 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.038757+0000 mgr.y (mgr.24401) 864 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.251144+0000 osd.3 (osd.3) 559 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.251144+0000 osd.3 (osd.3) 559 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:40:05.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.252958+0000 osd.3 (osd.3) 560 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.252958+0000 osd.3 (osd.3) 560 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.784115+0000 osd.2 (osd.2) 475 : cluster [DBG] 6.f scrub starts 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.784115+0000 osd.2 (osd.2) 475 : cluster [DBG] 6.f scrub starts 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.786730+0000 osd.2 (osd.2) 476 : cluster [DBG] 6.f scrub ok 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.786730+0000 osd.2 (osd.2) 476 : cluster [DBG] 6.f scrub ok 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: audit 2024-04-26T16:40:03.932429+0000 mon.a (mon.0) 4015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: audit 2024-04-26T16:40:03.932429+0000 mon.a (mon.0) 4015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.940217+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:03.940217+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.038757+0000 mgr.y (mgr.24401) 864 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:05.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:04 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.038757+0000 mgr.y (mgr.24401) 864 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.251144+0000 osd.3 (osd.3) 559 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.251144+0000 osd.3 (osd.3) 559 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.252958+0000 osd.3 (osd.3) 560 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.252958+0000 osd.3 (osd.3) 560 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.784115+0000 osd.2 (osd.2) 475 : cluster [DBG] 6.f scrub starts 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.784115+0000 osd.2 (osd.2) 475 : cluster [DBG] 6.f scrub starts 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.786730+0000 osd.2 (osd.2) 476 : cluster [DBG] 6.f scrub ok 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.786730+0000 osd.2 (osd.2) 476 : cluster [DBG] 6.f scrub ok 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: audit 2024-04-26T16:40:03.932429+0000 mon.a (mon.0) 4015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:40:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: audit 2024-04-26T16:40:03.932429+0000 mon.a (mon.0) 4015 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-04-26T16:40:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.940217+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2024-04-26T16:40:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:03.940217+0000 mon.a (mon.0) 4016 : cluster [DBG] osdmap e905: 8 total, 8 up, 8 in 2024-04-26T16:40:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.038757+0000 mgr.y (mgr.24401) 864 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:04 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.038757+0000 mgr.y (mgr.24401) 864 : cluster [DBG] pgmap v1457: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.567197+0000 osd.0 (osd.0) 547 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.567197+0000 osd.0 (osd.0) 547 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.567502+0000 osd.0 (osd.0) 548 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.567502+0000 osd.0 (osd.0) 548 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.822154+0000 osd.2 (osd.2) 477 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.822154+0000 osd.2 (osd.2) 477 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.824884+0000 osd.2 (osd.2) 478 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:40:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.824884+0000 osd.2 (osd.2) 478 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:40:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.945764+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2024-04-26T16:40:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:04.945764+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2024-04-26T16:40:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.283974+0000 osd.1 (osd.1) 521 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:40:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.283974+0000 osd.1 (osd.1) 521 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:40:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.286221+0000 osd.1 (osd.1) 522 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:40:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:05 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.286221+0000 osd.1 (osd.1) 522 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:40:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.567197+0000 osd.0 (osd.0) 547 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.567197+0000 osd.0 (osd.0) 547 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.567502+0000 osd.0 (osd.0) 548 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.567502+0000 osd.0 (osd.0) 548 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.822154+0000 osd.2 (osd.2) 477 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.822154+0000 osd.2 (osd.2) 477 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.824884+0000 osd.2 (osd.2) 478 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.824884+0000 osd.2 (osd.2) 478 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.945764+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:04.945764+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.283974+0000 osd.1 (osd.1) 521 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.283974+0000 osd.1 (osd.1) 521 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.286221+0000 osd.1 (osd.1) 522 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.286221+0000 osd.1 (osd.1) 522 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:40:06.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.567197+0000 osd.0 (osd.0) 547 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.567197+0000 osd.0 (osd.0) 547 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.567502+0000 osd.0 (osd.0) 548 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.567502+0000 osd.0 (osd.0) 548 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.822154+0000 osd.2 (osd.2) 477 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.822154+0000 osd.2 (osd.2) 477 : cluster [DBG] 11.2 deep-scrub starts 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.824884+0000 osd.2 (osd.2) 478 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.824884+0000 osd.2 (osd.2) 478 : cluster [DBG] 11.2 deep-scrub ok 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.945764+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:04.945764+0000 mon.a (mon.0) 4017 : cluster [DBG] osdmap e906: 8 total, 8 up, 8 in 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.283974+0000 osd.1 (osd.1) 521 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.283974+0000 osd.1 (osd.1) 521 : cluster [DBG] 5.12 deep-scrub starts 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.286221+0000 osd.1 (osd.1) 522 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:40:06.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:05 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.286221+0000 osd.1 (osd.1) 522 : cluster [DBG] 5.12 deep-scrub ok 2024-04-26T16:40:07.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:40:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.987993+0000 osd.0 (osd.0) 549 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:40:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.987993+0000 osd.0 (osd.0) 549 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:40:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.992089+0000 osd.0 (osd.0) 550 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:40:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[23868]: cluster 2024-04-26T16:40:05.992089+0000 osd.0 (osd.0) 550 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:40:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[23868]: cluster 2024-04-26T16:40:06.039516+0000 mgr.y (mgr.24401) 865 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:07.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:07 smithi190 bash[23868]: cluster 2024-04-26T16:40:06.039516+0000 mgr.y (mgr.24401) 865 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.987993+0000 osd.0 (osd.0) 549 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.987993+0000 osd.0 (osd.0) 549 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.992089+0000 osd.0 (osd.0) 550 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[29234]: cluster 2024-04-26T16:40:05.992089+0000 osd.0 (osd.0) 550 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[29234]: cluster 2024-04-26T16:40:06.039516+0000 mgr.y (mgr.24401) 865 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[29234]: cluster 2024-04-26T16:40:06.039516+0000 mgr.y (mgr.24401) 865 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:07.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.987993+0000 osd.0 (osd.0) 549 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:40:07.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.987993+0000 osd.0 (osd.0) 549 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:40:07.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.992089+0000 osd.0 (osd.0) 550 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:40:07.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[21463]: cluster 2024-04-26T16:40:05.992089+0000 osd.0 (osd.0) 550 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:40:07.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[21463]: cluster 2024-04-26T16:40:06.039516+0000 mgr.y (mgr.24401) 865 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:07.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:07 smithi050 bash[21463]: cluster 2024-04-26T16:40:06.039516+0000 mgr.y (mgr.24401) 865 : cluster [DBG] pgmap v1459: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.137691+0000 osd.5 (osd.5) 416 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.137691+0000 osd.5 (osd.5) 416 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.140568+0000 osd.5 (osd.5) 417 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.140568+0000 osd.5 (osd.5) 417 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: audit 2024-04-26T16:40:07.469070+0000 mgr.y (mgr.24401) 866 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: audit 2024-04-26T16:40:07.469070+0000 mgr.y (mgr.24401) 866 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.941905+0000 osd.0 (osd.0) 551 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.941905+0000 osd.0 (osd.0) 551 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.946706+0000 osd.0 (osd.0) 552 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:40:08.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:08 smithi190 bash[23868]: cluster 2024-04-26T16:40:07.946706+0000 osd.0 (osd.0) 552 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:40:08.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.137691+0000 osd.5 (osd.5) 416 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:40:08.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.137691+0000 osd.5 (osd.5) 416 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:40:08.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.140568+0000 osd.5 (osd.5) 417 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.140568+0000 osd.5 (osd.5) 417 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: audit 2024-04-26T16:40:07.469070+0000 mgr.y (mgr.24401) 866 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: audit 2024-04-26T16:40:07.469070+0000 mgr.y (mgr.24401) 866 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.941905+0000 osd.0 (osd.0) 551 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.941905+0000 osd.0 (osd.0) 551 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.946706+0000 osd.0 (osd.0) 552 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:40:08.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[29234]: cluster 2024-04-26T16:40:07.946706+0000 osd.0 (osd.0) 552 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:40:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.137691+0000 osd.5 (osd.5) 416 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:40:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.137691+0000 osd.5 (osd.5) 416 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:40:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.140568+0000 osd.5 (osd.5) 417 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:40:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.140568+0000 osd.5 (osd.5) 417 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:40:08.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: audit 2024-04-26T16:40:07.469070+0000 mgr.y (mgr.24401) 866 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:08.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: audit 2024-04-26T16:40:07.469070+0000 mgr.y (mgr.24401) 866 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:08.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.941905+0000 osd.0 (osd.0) 551 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:40:08.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.941905+0000 osd.0 (osd.0) 551 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:40:08.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.946706+0000 osd.0 (osd.0) 552 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:40:08.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:08 smithi050 bash[21463]: cluster 2024-04-26T16:40:07.946706+0000 osd.0 (osd.0) 552 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:40:09.688 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.041319+0000 mgr.y (mgr.24401) 867 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.041319+0000 mgr.y (mgr.24401) 867 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.161079+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.161079+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.163964+0000 osd.1 (osd.1) 524 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.163964+0000 osd.1 (osd.1) 524 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.177202+0000 osd.3 (osd.3) 561 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:40:09.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.177202+0000 osd.3 (osd.3) 561 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.178580+0000 osd.5 (osd.5) 418 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.178580+0000 osd.5 (osd.5) 418 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.181547+0000 osd.5 (osd.5) 419 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.181547+0000 osd.5 (osd.5) 419 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.182210+0000 osd.3 (osd.3) 562 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.182210+0000 osd.3 (osd.3) 562 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.999528+0000 osd.7 (osd.7) 546 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:08.999528+0000 osd.7 (osd.7) 546 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.001961+0000 osd.7 (osd.7) 547 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:40:09.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:09 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.001961+0000 osd.7 (osd.7) 547 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.041319+0000 mgr.y (mgr.24401) 867 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.041319+0000 mgr.y (mgr.24401) 867 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.161079+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.161079+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.163964+0000 osd.1 (osd.1) 524 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.163964+0000 osd.1 (osd.1) 524 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.177202+0000 osd.3 (osd.3) 561 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.177202+0000 osd.3 (osd.3) 561 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.178580+0000 osd.5 (osd.5) 418 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.178580+0000 osd.5 (osd.5) 418 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.181547+0000 osd.5 (osd.5) 419 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:40:09.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.181547+0000 osd.5 (osd.5) 419 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.182210+0000 osd.3 (osd.3) 562 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.182210+0000 osd.3 (osd.3) 562 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.999528+0000 osd.7 (osd.7) 546 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:08.999528+0000 osd.7 (osd.7) 546 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.001961+0000 osd.7 (osd.7) 547 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.001961+0000 osd.7 (osd.7) 547 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.041319+0000 mgr.y (mgr.24401) 867 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.041319+0000 mgr.y (mgr.24401) 867 : cluster [DBG] pgmap v1460: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.161079+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.161079+0000 osd.1 (osd.1) 523 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:40:09.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.163964+0000 osd.1 (osd.1) 524 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.163964+0000 osd.1 (osd.1) 524 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.177202+0000 osd.3 (osd.3) 561 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.177202+0000 osd.3 (osd.3) 561 : cluster [DBG] 3.b deep-scrub starts 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.178580+0000 osd.5 (osd.5) 418 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.178580+0000 osd.5 (osd.5) 418 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.181547+0000 osd.5 (osd.5) 419 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.181547+0000 osd.5 (osd.5) 419 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.182210+0000 osd.3 (osd.3) 562 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.182210+0000 osd.3 (osd.3) 562 : cluster [DBG] 3.b deep-scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.999528+0000 osd.7 (osd.7) 546 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:08.999528+0000 osd.7 (osd.7) 546 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.001961+0000 osd.7 (osd.7) 547 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:40:09.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:09 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.001961+0000 osd.7 (osd.7) 547 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.165389+0000 osd.3 (osd.3) 563 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.165389+0000 osd.3 (osd.3) 563 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.170427+0000 osd.3 (osd.3) 564 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.170427+0000 osd.3 (osd.3) 564 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.016659+0000 osd.7 (osd.7) 548 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.016659+0000 osd.7 (osd.7) 548 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.022785+0000 osd.7 (osd.7) 549 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:40:10.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:10 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.022785+0000 osd.7 (osd.7) 549 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.165389+0000 osd.3 (osd.3) 563 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.165389+0000 osd.3 (osd.3) 563 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.170427+0000 osd.3 (osd.3) 564 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.170427+0000 osd.3 (osd.3) 564 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.016659+0000 osd.7 (osd.7) 548 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.016659+0000 osd.7 (osd.7) 548 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.022785+0000 osd.7 (osd.7) 549 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.022785+0000 osd.7 (osd.7) 549 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:40:10.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.165389+0000 osd.3 (osd.3) 563 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.165389+0000 osd.3 (osd.3) 563 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.170427+0000 osd.3 (osd.3) 564 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.170427+0000 osd.3 (osd.3) 564 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.016659+0000 osd.7 (osd.7) 548 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.016659+0000 osd.7 (osd.7) 548 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.022785+0000 osd.7 (osd.7) 549 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:40:10.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:10 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.022785+0000 osd.7 (osd.7) 549 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:40:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.898006+0000 osd.2 (osd.2) 479 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:40:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.898006+0000 osd.2 (osd.2) 479 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:40:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.905246+0000 osd.2 (osd.2) 480 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:40:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:09.905246+0000 osd.2 (osd.2) 480 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:40:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.042239+0000 mgr.y (mgr.24401) 868 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.042239+0000 mgr.y (mgr.24401) 868 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.118749+0000 osd.3 (osd.3) 565 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.118749+0000 osd.3 (osd.3) 565 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.122735+0000 osd.3 (osd.3) 566 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:10.122735+0000 osd.3 (osd.3) 566 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.052636+0000 osd.7 (osd.7) 550 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.052636+0000 osd.7 (osd.7) 550 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.055193+0000 osd.7 (osd.7) 551 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.055193+0000 osd.7 (osd.7) 551 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.898006+0000 osd.2 (osd.2) 479 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.898006+0000 osd.2 (osd.2) 479 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.905246+0000 osd.2 (osd.2) 480 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:40:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:09.905246+0000 osd.2 (osd.2) 480 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.042239+0000 mgr.y (mgr.24401) 868 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.042239+0000 mgr.y (mgr.24401) 868 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.118749+0000 osd.3 (osd.3) 565 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.118749+0000 osd.3 (osd.3) 565 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.122735+0000 osd.3 (osd.3) 566 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:10.122735+0000 osd.3 (osd.3) 566 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.052636+0000 osd.7 (osd.7) 550 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.052636+0000 osd.7 (osd.7) 550 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.055193+0000 osd.7 (osd.7) 551 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:40:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:11 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.055193+0000 osd.7 (osd.7) 551 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.898006+0000 osd.2 (osd.2) 479 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.898006+0000 osd.2 (osd.2) 479 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.905246+0000 osd.2 (osd.2) 480 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:09.905246+0000 osd.2 (osd.2) 480 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.042239+0000 mgr.y (mgr.24401) 868 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.042239+0000 mgr.y (mgr.24401) 868 : cluster [DBG] pgmap v1461: 174 pgs: 174 active+clean; 583 KiB data, 786 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.118749+0000 osd.3 (osd.3) 565 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.118749+0000 osd.3 (osd.3) 565 : cluster [DBG] 3.18 scrub starts 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.122735+0000 osd.3 (osd.3) 566 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:10.122735+0000 osd.3 (osd.3) 566 : cluster [DBG] 3.18 scrub ok 2024-04-26T16:40:11.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.052636+0000 osd.7 (osd.7) 550 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:40:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.052636+0000 osd.7 (osd.7) 550 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:40:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.055193+0000 osd.7 (osd.7) 551 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:40:11.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:11 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.055193+0000 osd.7 (osd.7) 551 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:40:12.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:12 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.859246+0000 osd.4 (osd.4) 487 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:40:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:12 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.859246+0000 osd.4 (osd.4) 487 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:40:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:12 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.861849+0000 osd.4 (osd.4) 488 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:40:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:12 smithi190 bash[23868]: cluster 2024-04-26T16:40:11.861849+0000 osd.4 (osd.4) 488 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.859246+0000 osd.4 (osd.4) 487 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.859246+0000 osd.4 (osd.4) 487 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.861849+0000 osd.4 (osd.4) 488 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[29234]: cluster 2024-04-26T16:40:11.861849+0000 osd.4 (osd.4) 488 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.859246+0000 osd.4 (osd.4) 487 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.859246+0000 osd.4 (osd.4) 487 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.861849+0000 osd.4 (osd.4) 488 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:40:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:12 smithi050 bash[21463]: cluster 2024-04-26T16:40:11.861849+0000 osd.4 (osd.4) 488 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:40:13.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/4150853674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 msgr2=0x7f24f4110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- 172.21.15.50:0/4150853674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4110310 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00b0a0 tx=0x7f24ec00b3b0 comp rx=0 tx=0).stop 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/4150853674 shutdown_connections 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- 172.21.15.50:0/4150853674 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4110310 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- 172.21.15.50:0/4150853674 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24f41094d0 0x7f24f41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- 172.21.15.50:0/4150853674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f406bbd0 0x7f24f406c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/4150853674 >> 172.21.15.50:0/4150853674 conn(0x7f24f40793c0 msgr2=0x7f24f40797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/4150853674 shutdown_connections 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/4150853674 wait complete. 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 librados: starting msgr at 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 librados: starting objecter 2024-04-26T16:40:13.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 Processor -- start 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- start start 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 librados: setting wanted keys 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 librados: calling monclient init 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f406bbd0 0x7f24f41990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24f41094d0 0x7f24f419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f24f4112b20 con 0x7f24f406c590 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f24f41129a0 con 0x7f24f406bbd0 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f24f4112ca0 con 0x7f24f41094d0 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f8de3700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24f41094d0 0x7f24f419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:49262/0 (socket says 172.21.15.50:49262) 2024-04-26T16:40:13.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 -- 172.21.15.50:0/3039868057 learned_addr learned my addr 172.21.15.50:0/3039868057 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:40:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f3fff700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f406bbd0 0x7f24f41990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 -- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24f41094d0 msgr2=0x7f24f419d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24f41094d0 0x7f24f419d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 -- 172.21.15.50:0/3039868057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f406bbd0 msgr2=0x7f24f41990e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f406bbd0 0x7f24f41990e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:13.283 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 -- 172.21.15.50:0/3039868057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f419e0f0 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f37fe700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4199620 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f24e800d950 tx=0x7f24e800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f17fa700 10 client.?.objecter ms_handle_connect 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f17fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8016070 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f419e3e0 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f24e8009da0 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8010e60 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 10 client.31209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f41a5c60 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.280+0000 7f24fa846700 1 librados: init done 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f406a110 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f24e8019070 con 0x7f24f406c590 2024-04-26T16:40:13.284 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f24dc071f80 0x7f24dc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(906..906 src has 854..906) ==== 8707+0+0 (secure 0 0 0) 0x7f24e8091350 con 0x7f24f406c590 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 10 client.31209.objecter ms_dispatch 0x7f24f4001550 osd_map(906..906 src has 854..906) 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 3 client.31209.objecter handle_osd_map got epochs [906,906] > 0 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 3 client.31209.objecter handle_osd_map decoding full epoch 906 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f3fff700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f24dc071f80 0x7f24dc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 20 client.31209.objecter dump_active .. 0 homeless 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f17fa700 10 client.31209.objecter ms_handle_connect 0x7f24dc071f80 2024-04-26T16:40:13.285 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.284+0000 7f24f3fff700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f24dc071f80 0x7f24dc074440 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009bd0 tx=0x7f24e4009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:13.288 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.288+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f24e805cfa0 con 0x7f24f406c590 2024-04-26T16:40:13.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:13.532+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 3, "weight": 1.0} v 0) -- 0x7f24f41a5e30 con 0x7f24f406c590 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.043421+0000 mgr.y (mgr.24401) 869 : cluster [DBG] pgmap v1462: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.043421+0000 mgr.y (mgr.24401) 869 : cluster [DBG] pgmap v1462: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.843337+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.843337+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.845803+0000 osd.4 (osd.4) 490 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.845803+0000 osd.4 (osd.4) 490 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.954427+0000 osd.0 (osd.0) 553 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.954427+0000 osd.0 (osd.0) 553 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.958316+0000 osd.0 (osd.0) 554 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.958316+0000 osd.0 (osd.0) 554 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:40:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: audit 2024-04-26T16:40:13.365236+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:13 smithi190 bash[23868]: audit 2024-04-26T16:40:13.365236+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.043421+0000 mgr.y (mgr.24401) 869 : cluster [DBG] pgmap v1462: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.043421+0000 mgr.y (mgr.24401) 869 : cluster [DBG] pgmap v1462: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.843337+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.843337+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.845803+0000 osd.4 (osd.4) 490 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.845803+0000 osd.4 (osd.4) 490 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:40:13.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.954427+0000 osd.0 (osd.0) 553 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.954427+0000 osd.0 (osd.0) 553 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.958316+0000 osd.0 (osd.0) 554 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.958316+0000 osd.0 (osd.0) 554 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: audit 2024-04-26T16:40:13.365236+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[21463]: audit 2024-04-26T16:40:13.365236+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.043421+0000 mgr.y (mgr.24401) 869 : cluster [DBG] pgmap v1462: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.043421+0000 mgr.y (mgr.24401) 869 : cluster [DBG] pgmap v1462: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.843337+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.843337+0000 osd.4 (osd.4) 489 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:40:13.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.845803+0000 osd.4 (osd.4) 490 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.845803+0000 osd.4 (osd.4) 490 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.954427+0000 osd.0 (osd.0) 553 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.954427+0000 osd.0 (osd.0) 553 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.958316+0000 osd.0 (osd.0) 554 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.958316+0000 osd.0 (osd.0) 554 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: audit 2024-04-26T16:40:13.365236+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:13 smithi050 bash[29234]: audit 2024-04-26T16:40:13.365236+0000 mon.c (mon.2) 931 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:14.525 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.524+0000 7f24f17fa700 1 -- 172.21.15.50:0/3039868057 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 3, "weight": 1.0}]=0 reweighted osd.3 to 1 (10000) v907) ==== 113+0+0 (secure 0 0 0) 0x7f24e8060c50 con 0x7f24f406c590 2024-04-26T16:40:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.528+0000 7f24fa846700 10 librados: watch_flush enter 2024-04-26T16:40:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.528+0000 7f24fa846700 10 librados: watch_flush exit 2024-04-26T16:40:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.528+0000 7f24fa846700 20 client.31209.objecter shutdown clearing up homeless session... 2024-04-26T16:40:14.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.532+0000 7f24fa846700 10 client.31209.objecter successfully canceled tick 2024-04-26T16:40:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f24dc071f80 msgr2=0x7f24dc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f24dc071f80 0x7f24dc074440 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009bd0 tx=0x7f24e4009260 comp rx=0 tx=0).stop 2024-04-26T16:40:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 msgr2=0x7f24f4199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:14.536 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4199620 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7f24e800d950 tx=0x7f24e800dc60 comp rx=0 tx=0).stop 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 shutdown_connections 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f24dc071f80 0x7f24dc074440 unknown :-1 s=CLOSED pgs=3862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24f406bbd0 0x7f24f41990e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f24f41094d0 0x7f24f419d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 --2- 172.21.15.50:0/3039868057 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f24f406c590 0x7f24f4199620 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 >> 172.21.15.50:0/3039868057 conn(0x7f24f40793c0 msgr2=0x7f24f4100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 shutdown_connections 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 -- 172.21.15.50:0/3039868057 wait complete. 2024-04-26T16:40:14.537 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:14.536+0000 7f24fa846700 1 librados: shutdown 2024-04-26T16:40:14.538 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.3 to 1 (10000) 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.953577+0000 osd.2 (osd.2) 481 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.953577+0000 osd.2 (osd.2) 481 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.957235+0000 osd.2 (osd.2) 482 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:12.957235+0000 osd.2 (osd.2) 482 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.100913+0000 osd.3 (osd.3) 567 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.100913+0000 osd.3 (osd.3) 567 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.102496+0000 osd.3 (osd.3) 568 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.102496+0000 osd.3 (osd.3) 568 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:40:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.125503+0000 osd.1 (osd.1) 525 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.125503+0000 osd.1 (osd.1) 525 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.127821+0000 osd.1 (osd.1) 526 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.127821+0000 osd.1 (osd.1) 526 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: audit 2024-04-26T16:40:13.535991+0000 mon.a (mon.0) 4018 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: audit 2024-04-26T16:40:13.535991+0000 mon.a (mon.0) 4018 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.817584+0000 osd.4 (osd.4) 491 : cluster [DBG] 5.e scrub starts 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.817584+0000 osd.4 (osd.4) 491 : cluster [DBG] 5.e scrub starts 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.820578+0000 osd.4 (osd.4) 492 : cluster [DBG] 5.e scrub ok 2024-04-26T16:40:14.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:14 smithi190 bash[23868]: cluster 2024-04-26T16:40:13.820578+0000 osd.4 (osd.4) 492 : cluster [DBG] 5.e scrub ok 2024-04-26T16:40:14.828 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:40:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:40:14.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.953577+0000 osd.2 (osd.2) 481 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:40:14.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.953577+0000 osd.2 (osd.2) 481 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:40:14.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.957235+0000 osd.2 (osd.2) 482 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:40:14.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:12.957235+0000 osd.2 (osd.2) 482 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:40:14.829 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.100913+0000 osd.3 (osd.3) 567 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.100913+0000 osd.3 (osd.3) 567 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.102496+0000 osd.3 (osd.3) 568 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.102496+0000 osd.3 (osd.3) 568 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.125503+0000 osd.1 (osd.1) 525 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.125503+0000 osd.1 (osd.1) 525 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.127821+0000 osd.1 (osd.1) 526 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.127821+0000 osd.1 (osd.1) 526 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: audit 2024-04-26T16:40:13.535991+0000 mon.a (mon.0) 4018 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: audit 2024-04-26T16:40:13.535991+0000 mon.a (mon.0) 4018 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.817584+0000 osd.4 (osd.4) 491 : cluster [DBG] 5.e scrub starts 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.817584+0000 osd.4 (osd.4) 491 : cluster [DBG] 5.e scrub starts 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.820578+0000 osd.4 (osd.4) 492 : cluster [DBG] 5.e scrub ok 2024-04-26T16:40:14.830 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[29234]: cluster 2024-04-26T16:40:13.820578+0000 osd.4 (osd.4) 492 : cluster [DBG] 5.e scrub ok 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.953577+0000 osd.2 (osd.2) 481 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.953577+0000 osd.2 (osd.2) 481 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.957235+0000 osd.2 (osd.2) 482 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:12.957235+0000 osd.2 (osd.2) 482 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.100913+0000 osd.3 (osd.3) 567 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.100913+0000 osd.3 (osd.3) 567 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.102496+0000 osd.3 (osd.3) 568 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.102496+0000 osd.3 (osd.3) 568 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.125503+0000 osd.1 (osd.1) 525 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:40:14.831 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.125503+0000 osd.1 (osd.1) 525 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.127821+0000 osd.1 (osd.1) 526 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.127821+0000 osd.1 (osd.1) 526 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: audit 2024-04-26T16:40:13.535991+0000 mon.a (mon.0) 4018 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: audit 2024-04-26T16:40:13.535991+0000 mon.a (mon.0) 4018 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.817584+0000 osd.4 (osd.4) 491 : cluster [DBG] 5.e scrub starts 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.817584+0000 osd.4 (osd.4) 491 : cluster [DBG] 5.e scrub starts 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.820578+0000 osd.4 (osd.4) 492 : cluster [DBG] 5.e scrub ok 2024-04-26T16:40:14.832 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:14 smithi050 bash[21463]: cluster 2024-04-26T16:40:13.820578+0000 osd.4 (osd.4) 492 : cluster [DBG] 5.e scrub ok 2024-04-26T16:40:15.434 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 3 1 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.044329+0000 mgr.y (mgr.24401) 870 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.044329+0000 mgr.y (mgr.24401) 870 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.088689+0000 osd.5 (osd.5) 420 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.088689+0000 osd.5 (osd.5) 420 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.089128+0000 osd.5 (osd.5) 421 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.089128+0000 osd.5 (osd.5) 421 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: audit 2024-04-26T16:40:14.530213+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: audit 2024-04-26T16:40:14.530213+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.537259+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2024-04-26T16:40:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:14.537259+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2024-04-26T16:40:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:15.031392+0000 osd.7 (osd.7) 552 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:40:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:15.031392+0000 osd.7 (osd.7) 552 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:40:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:15.035153+0000 osd.7 (osd.7) 553 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:40:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:15 smithi190 bash[23868]: cluster 2024-04-26T16:40:15.035153+0000 osd.7 (osd.7) 553 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:40:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.044329+0000 mgr.y (mgr.24401) 870 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.044329+0000 mgr.y (mgr.24401) 870 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.088689+0000 osd.5 (osd.5) 420 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.088689+0000 osd.5 (osd.5) 420 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.089128+0000 osd.5 (osd.5) 421 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.089128+0000 osd.5 (osd.5) 421 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: audit 2024-04-26T16:40:14.530213+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: audit 2024-04-26T16:40:14.530213+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.537259+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:14.537259+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:15.031392+0000 osd.7 (osd.7) 552 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:15.031392+0000 osd.7 (osd.7) 552 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:15.035153+0000 osd.7 (osd.7) 553 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[21463]: cluster 2024-04-26T16:40:15.035153+0000 osd.7 (osd.7) 553 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:40:15.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.044329+0000 mgr.y (mgr.24401) 870 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.044329+0000 mgr.y (mgr.24401) 870 : cluster [DBG] pgmap v1463: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.088689+0000 osd.5 (osd.5) 420 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.088689+0000 osd.5 (osd.5) 420 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.089128+0000 osd.5 (osd.5) 421 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.089128+0000 osd.5 (osd.5) 421 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: audit 2024-04-26T16:40:14.530213+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: audit 2024-04-26T16:40:14.530213+0000 mon.a (mon.0) 4019 : audit [INF] from='client.? 172.21.15.50:0/3039868057' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.537259+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:14.537259+0000 mon.a (mon.0) 4020 : cluster [DBG] osdmap e907: 8 total, 8 up, 8 in 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:15.031392+0000 osd.7 (osd.7) 552 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:15.031392+0000 osd.7 (osd.7) 552 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:15.035153+0000 osd.7 (osd.7) 553 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:40:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:15 smithi050 bash[29234]: cluster 2024-04-26T16:40:15.035153+0000 osd.7 (osd.7) 553 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:40:16.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:16 smithi190 bash[23868]: cluster 2024-04-26T16:40:16.025237+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:40:16.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:16 smithi190 bash[23868]: cluster 2024-04-26T16:40:16.025237+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:40:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:16 smithi190 bash[23868]: cluster 2024-04-26T16:40:16.027556+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:40:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:16 smithi190 bash[23868]: cluster 2024-04-26T16:40:16.027556+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[29234]: cluster 2024-04-26T16:40:16.025237+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[29234]: cluster 2024-04-26T16:40:16.025237+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[29234]: cluster 2024-04-26T16:40:16.027556+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[29234]: cluster 2024-04-26T16:40:16.027556+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[21463]: cluster 2024-04-26T16:40:16.025237+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[21463]: cluster 2024-04-26T16:40:16.025237+0000 osd.7 (osd.7) 554 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[21463]: cluster 2024-04-26T16:40:16.027556+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:40:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:16 smithi050 bash[21463]: cluster 2024-04-26T16:40:16.027556+0000 osd.7 (osd.7) 555 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:40:17.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:17 smithi190 bash[23868]: cluster 2024-04-26T16:40:16.045094+0000 mgr.y (mgr.24401) 871 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:17 smithi190 bash[23868]: cluster 2024-04-26T16:40:16.045094+0000 mgr.y (mgr.24401) 871 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:17.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:40:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:40:17.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:17 smithi050 bash[29234]: cluster 2024-04-26T16:40:16.045094+0000 mgr.y (mgr.24401) 871 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:17.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:17 smithi050 bash[29234]: cluster 2024-04-26T16:40:16.045094+0000 mgr.y (mgr.24401) 871 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:17 smithi050 bash[21463]: cluster 2024-04-26T16:40:16.045094+0000 mgr.y (mgr.24401) 871 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:17 smithi050 bash[21463]: cluster 2024-04-26T16:40:16.045094+0000 mgr.y (mgr.24401) 871 : cluster [DBG] pgmap v1465: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:17.072122+0000 osd.0 (osd.0) 555 : cluster [DBG] 5.3 deep-scrub starts 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:17.072122+0000 osd.0 (osd.0) 555 : cluster [DBG] 5.3 deep-scrub starts 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:17.078046+0000 osd.0 (osd.0) 556 : cluster [DBG] 5.3 deep-scrub ok 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:17.078046+0000 osd.0 (osd.0) 556 : cluster [DBG] 5.3 deep-scrub ok 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: audit 2024-04-26T16:40:17.476931+0000 mgr.y (mgr.24401) 872 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: audit 2024-04-26T16:40:17.476931+0000 mgr.y (mgr.24401) 872 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.032960+0000 osd.7 (osd.7) 556 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.032960+0000 osd.7 (osd.7) 556 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.035378+0000 osd.7 (osd.7) 557 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.035378+0000 osd.7 (osd.7) 557 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.051274+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a deep-scrub starts 2024-04-26T16:40:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.051274+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a deep-scrub starts 2024-04-26T16:40:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.066510+0000 osd.6 (osd.6) 96 : cluster [DBG] 3.a deep-scrub ok 2024-04-26T16:40:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:18 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.066510+0000 osd.6 (osd.6) 96 : cluster [DBG] 3.a deep-scrub ok 2024-04-26T16:40:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:17.072122+0000 osd.0 (osd.0) 555 : cluster [DBG] 5.3 deep-scrub starts 2024-04-26T16:40:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:17.072122+0000 osd.0 (osd.0) 555 : cluster [DBG] 5.3 deep-scrub starts 2024-04-26T16:40:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:17.078046+0000 osd.0 (osd.0) 556 : cluster [DBG] 5.3 deep-scrub ok 2024-04-26T16:40:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:17.078046+0000 osd.0 (osd.0) 556 : cluster [DBG] 5.3 deep-scrub ok 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: audit 2024-04-26T16:40:17.476931+0000 mgr.y (mgr.24401) 872 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: audit 2024-04-26T16:40:17.476931+0000 mgr.y (mgr.24401) 872 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.032960+0000 osd.7 (osd.7) 556 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.032960+0000 osd.7 (osd.7) 556 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.035378+0000 osd.7 (osd.7) 557 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.035378+0000 osd.7 (osd.7) 557 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.051274+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a deep-scrub starts 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.051274+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a deep-scrub starts 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.066510+0000 osd.6 (osd.6) 96 : cluster [DBG] 3.a deep-scrub ok 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.066510+0000 osd.6 (osd.6) 96 : cluster [DBG] 3.a deep-scrub ok 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:17.072122+0000 osd.0 (osd.0) 555 : cluster [DBG] 5.3 deep-scrub starts 2024-04-26T16:40:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:17.072122+0000 osd.0 (osd.0) 555 : cluster [DBG] 5.3 deep-scrub starts 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:17.078046+0000 osd.0 (osd.0) 556 : cluster [DBG] 5.3 deep-scrub ok 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:17.078046+0000 osd.0 (osd.0) 556 : cluster [DBG] 5.3 deep-scrub ok 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: audit 2024-04-26T16:40:17.476931+0000 mgr.y (mgr.24401) 872 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: audit 2024-04-26T16:40:17.476931+0000 mgr.y (mgr.24401) 872 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.032960+0000 osd.7 (osd.7) 556 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.032960+0000 osd.7 (osd.7) 556 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.035378+0000 osd.7 (osd.7) 557 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.035378+0000 osd.7 (osd.7) 557 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.051274+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a deep-scrub starts 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.051274+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a deep-scrub starts 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.066510+0000 osd.6 (osd.6) 96 : cluster [DBG] 3.a deep-scrub ok 2024-04-26T16:40:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:18 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.066510+0000 osd.6 (osd.6) 96 : cluster [DBG] 3.a deep-scrub ok 2024-04-26T16:40:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:19 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.047124+0000 mgr.y (mgr.24401) 873 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:19.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:19 smithi190 bash[23868]: cluster 2024-04-26T16:40:18.047124+0000 mgr.y (mgr.24401) 873 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:19 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.047124+0000 mgr.y (mgr.24401) 873 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:19 smithi050 bash[21463]: cluster 2024-04-26T16:40:18.047124+0000 mgr.y (mgr.24401) 873 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:19 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.047124+0000 mgr.y (mgr.24401) 873 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:19.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:19 smithi050 bash[29234]: cluster 2024-04-26T16:40:18.047124+0000 mgr.y (mgr.24401) 873 : cluster [DBG] pgmap v1466: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:20.234 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:40:20.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.090979+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.090979+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.096416+0000 osd.3 (osd.3) 570 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.096416+0000 osd.3 (osd.3) 570 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.659909+0000 osd.3 (osd.3) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.659909+0000 osd.3 (osd.3) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.660396+0000 osd.3 (osd.3) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:20 smithi190 bash[23868]: cluster 2024-04-26T16:40:19.660396+0000 osd.3 (osd.3) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.090979+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.090979+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.096416+0000 osd.3 (osd.3) 570 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.096416+0000 osd.3 (osd.3) 570 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.659909+0000 osd.3 (osd.3) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.659909+0000 osd.3 (osd.3) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.660396+0000 osd.3 (osd.3) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[21463]: cluster 2024-04-26T16:40:19.660396+0000 osd.3 (osd.3) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.090979+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.090979+0000 osd.3 (osd.3) 569 : cluster [DBG] 3.8 deep-scrub starts 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.096416+0000 osd.3 (osd.3) 570 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.096416+0000 osd.3 (osd.3) 570 : cluster [DBG] 3.8 deep-scrub ok 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.659909+0000 osd.3 (osd.3) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.659909+0000 osd.3 (osd.3) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.660396+0000 osd.3 (osd.3) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:20.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:20 smithi050 bash[29234]: cluster 2024-04-26T16:40:19.660396+0000 osd.3 (osd.3) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.048203+0000 mgr.y (mgr.24401) 874 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.048203+0000 mgr.y (mgr.24401) 874 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.574479+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.574479+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.964889+0000 osd.4 (osd.4) 493 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.964889+0000 osd.4 (osd.4) 493 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.967466+0000 osd.4 (osd.4) 494 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:20.967466+0000 osd.4 (osd.4) 494 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.038553+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.038553+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:40:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.042028+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:40:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:21 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.042028+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.048203+0000 mgr.y (mgr.24401) 874 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.048203+0000 mgr.y (mgr.24401) 874 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.574479+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.574479+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.964889+0000 osd.4 (osd.4) 493 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.964889+0000 osd.4 (osd.4) 493 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.967466+0000 osd.4 (osd.4) 494 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:40:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:20.967466+0000 osd.4 (osd.4) 494 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.038553+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.038553+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.042028+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.042028+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.048203+0000 mgr.y (mgr.24401) 874 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.048203+0000 mgr.y (mgr.24401) 874 : cluster [DBG] pgmap v1467: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.574479+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.574479+0000 mon.a (mon.0) 4021 : cluster [DBG] osdmap e908: 8 total, 8 up, 8 in 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.964889+0000 osd.4 (osd.4) 493 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.964889+0000 osd.4 (osd.4) 493 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.967466+0000 osd.4 (osd.4) 494 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:40:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:20.967466+0000 osd.4 (osd.4) 494 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:40:21.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.038553+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:40:21.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.038553+0000 osd.6 (osd.6) 97 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:40:21.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.042028+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:40:21.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:21 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.042028+0000 osd.6 (osd.6) 98 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.977201+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b scrub starts 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.977201+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b scrub starts 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.979692+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b scrub ok 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:21.979692+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b scrub ok 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.105370+0000 osd.5 (osd.5) 422 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.105370+0000 osd.5 (osd.5) 422 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.108366+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:40:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:22 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.108366+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.977201+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b scrub starts 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.977201+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b scrub starts 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.979692+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b scrub ok 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:21.979692+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b scrub ok 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.105370+0000 osd.5 (osd.5) 422 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.105370+0000 osd.5 (osd.5) 422 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.108366+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.108366+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:40:22.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.977201+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b scrub starts 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.977201+0000 osd.7 (osd.7) 558 : cluster [DBG] 2.b scrub starts 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.979692+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b scrub ok 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:21.979692+0000 osd.7 (osd.7) 559 : cluster [DBG] 2.b scrub ok 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.105370+0000 osd.5 (osd.5) 422 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.105370+0000 osd.5 (osd.5) 422 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.108366+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:40:22.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:22 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.108366+0000 osd.5 (osd.5) 423 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2201411959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 msgr2=0x7f9d3410f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2201411959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d3410f1c0 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7f9d2c00b1a0 tx=0x7f9d2c00b4b0 comp rx=0 tx=0).stop 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2201411959 shutdown_connections 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2201411959 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d3410f1c0 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2201411959 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34108380 0x7f9d34108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2201411959 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d34102310 0x7f9d34102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2201411959 >> 172.21.15.50:0/2201411959 conn(0x7f9d34079290 msgr2=0x7f9d340796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2201411959 shutdown_connections 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2201411959 wait complete. 2024-04-26T16:40:23.779 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 librados: starting msgr at 2024-04-26T16:40:23.780 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 librados: starting objecter 2024-04-26T16:40:23.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 Processor -- start 2024-04-26T16:40:23.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- start start 2024-04-26T16:40:23.781 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 librados: setting wanted keys 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 librados: calling monclient init 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 0x7f9d34196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d341973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d34108380 0x7f9d3419b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9d34110940 con 0x7f9d34102cd0 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f9d341107c0 con 0x7f9d34108380 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9d34110ac0 con 0x7f9d34102310 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d341973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 0x7f9d34196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 0x7f9d34196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:51186/0 (socket says 172.21.15.50:51186) 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d337fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d341973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56134/0 (socket says 172.21.15.50:56134) 2024-04-26T16:40:23.782 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 -- 172.21.15.50:0/2737949368 learned_addr learned my addr 172.21.15.50:0/2737949368 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d38b5a700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d34108380 0x7f9d3419b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 -- 172.21.15.50:0/2737949368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d34108380 msgr2=0x7f9d3419b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d34108380 0x7f9d3419b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 -- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 msgr2=0x7f9d341973f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d341973f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 -- 172.21.15.50:0/2737949368 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d3419bec0 con 0x7f9d34102310 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d337fe700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d341973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d33fff700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 0x7f9d34196eb0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f9d2400e9a0 tx=0x7f9d2400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d317fa700 10 client.?.objecter ms_handle_connect 0x7f9d34102310 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d317fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d2400cbd0 con 0x7f9d34102310 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9d2400cd70 con 0x7f9d34102310 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d3419c1b0 con 0x7f9d34102310 2024-04-26T16:40:23.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d24018420 con 0x7f9d34102310 2024-04-26T16:40:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 10 client.41834.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:40:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d341a3a30 con 0x7f9d34102310 2024-04-26T16:40:23.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.780+0000 7f9d3a5bd700 1 librados: init done 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d3406a110 con 0x7f9d34102310 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9d24010430 con 0x7f9d34102310 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9d1c071f80 0x7f9d1c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(908..908 src has 854..908) ==== 8707+0+0 (secure 0 0 0) 0x7f9d24091a00 con 0x7f9d34102310 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 10 client.41834.objecter ms_dispatch 0x7f9d34001550 osd_map(908..908 src has 854..908) 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 3 client.41834.objecter handle_osd_map got epochs [908,908] > 0 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 3 client.41834.objecter handle_osd_map decoding full epoch 908 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d337fe700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9d1c071f80 0x7f9d1c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 20 client.41834.objecter dump_active .. 0 homeless 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d317fa700 10 client.41834.objecter ms_handle_connect 0x7f9d1c071f80 2024-04-26T16:40:23.785 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.784+0000 7f9d337fe700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9d1c071f80 0x7f9d1c074440 secure :-1 s=READY pgs=3863 cs=0 l=1 rev1=1 crypto rx=0x7f9d34079900 tx=0x7f9d2800b410 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:23.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:23.788+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f9d2405d650 con 0x7f9d34102310 2024-04-26T16:40:23.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:23 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.050132+0000 mgr.y (mgr.24401) 875 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:23.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:23 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.050132+0000 mgr.y (mgr.24401) 875 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:23 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.050132+0000 mgr.y (mgr.24401) 875 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:23 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.050132+0000 mgr.y (mgr.24401) 875 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:23 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.050132+0000 mgr.y (mgr.24401) 875 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:23 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.050132+0000 mgr.y (mgr.24401) 875 : cluster [DBG] pgmap v1469: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:24.026 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.028+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} v 0) -- 0x7f9d3402e490 con 0x7f9d34102310 2024-04-26T16:40:24.592 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.592+0000 7f9d317fa700 1 -- 172.21.15.50:0/2737949368 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]=0 set osd.3 primary-affinity to 1 (8655362) v909) ==== 133+0+0 (secure 0 0 0) 0x7f9d24061300 con 0x7f9d34102310 2024-04-26T16:40:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 10 librados: watch_flush enter 2024-04-26T16:40:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 10 librados: watch_flush exit 2024-04-26T16:40:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 20 client.41834.objecter shutdown clearing up homeless session... 2024-04-26T16:40:24.595 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 10 client.41834.objecter successfully canceled tick 2024-04-26T16:40:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9d1c071f80 msgr2=0x7f9d1c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9d1c071f80 0x7f9d1c074440 secure :-1 s=READY pgs=3863 cs=0 l=1 rev1=1 crypto rx=0x7f9d34079900 tx=0x7f9d2800b410 comp rx=0 tx=0).stop 2024-04-26T16:40:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 msgr2=0x7f9d34196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 0x7f9d34196eb0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f9d2400e9a0 tx=0x7f9d2400ecb0 comp rx=0 tx=0).stop 2024-04-26T16:40:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 shutdown_connections 2024-04-26T16:40:24.596 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9d1c071f80 0x7f9d1c074440 unknown :-1 s=CLOSED pgs=3863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9d34108380 0x7f9d3419b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f9d34102310 0x7f9d34196eb0 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 --2- 172.21.15.50:0/2737949368 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f9d34102cd0 0x7f9d341973f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 >> 172.21.15.50:0/2737949368 conn(0x7f9d34079290 msgr2=0x7f9d341010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 shutdown_connections 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 -- 172.21.15.50:0/2737949368 wait complete. 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:24.596+0000 7f9d3a5bd700 1 librados: shutdown 2024-04-26T16:40:24.597 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-04-26T16:40:24.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:40:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.999800+0000 osd.1 (osd.1) 527 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: cluster 2024-04-26T16:40:22.999800+0000 osd.1 (osd.1) 527 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: cluster 2024-04-26T16:40:23.002031+0000 osd.1 (osd.1) 528 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: cluster 2024-04-26T16:40:23.002031+0000 osd.1 (osd.1) 528 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: audit 2024-04-26T16:40:24.031716+0000 mon.c (mon.2) 932 : audit [INF] from='client.? 172.21.15.50:0/2737949368' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: audit 2024-04-26T16:40:24.031716+0000 mon.c (mon.2) 932 : audit [INF] from='client.? 172.21.15.50:0/2737949368' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: audit 2024-04-26T16:40:24.032369+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[29234]: audit 2024-04-26T16:40:24.032369+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.999800+0000 osd.1 (osd.1) 527 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: cluster 2024-04-26T16:40:22.999800+0000 osd.1 (osd.1) 527 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: cluster 2024-04-26T16:40:23.002031+0000 osd.1 (osd.1) 528 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: cluster 2024-04-26T16:40:23.002031+0000 osd.1 (osd.1) 528 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: audit 2024-04-26T16:40:24.031716+0000 mon.c (mon.2) 932 : audit [INF] from='client.? 172.21.15.50:0/2737949368' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: audit 2024-04-26T16:40:24.031716+0000 mon.c (mon.2) 932 : audit [INF] from='client.? 172.21.15.50:0/2737949368' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: audit 2024-04-26T16:40:24.032369+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:24 smithi050 bash[21463]: audit 2024-04-26T16:40:24.032369+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:25.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.999800+0000 osd.1 (osd.1) 527 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:40:25.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: cluster 2024-04-26T16:40:22.999800+0000 osd.1 (osd.1) 527 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:40:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: cluster 2024-04-26T16:40:23.002031+0000 osd.1 (osd.1) 528 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:40:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: cluster 2024-04-26T16:40:23.002031+0000 osd.1 (osd.1) 528 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:40:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: audit 2024-04-26T16:40:24.031716+0000 mon.c (mon.2) 932 : audit [INF] from='client.? 172.21.15.50:0/2737949368' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: audit 2024-04-26T16:40:24.031716+0000 mon.c (mon.2) 932 : audit [INF] from='client.? 172.21.15.50:0/2737949368' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: audit 2024-04-26T16:40:24.032369+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:24 smithi190 bash[23868]: audit 2024-04-26T16:40:24.032369+0000 mon.a (mon.0) 4022 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-04-26T16:40:25.616 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 4 1 2024-04-26T16:40:25.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:23.950185+0000 osd.2 (osd.2) 483 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:40:25.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:23.950185+0000 osd.2 (osd.2) 483 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:40:25.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:23.952019+0000 osd.2 (osd.2) 484 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:40:25.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:23.952019+0000 osd.2 (osd.2) 484 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:40:25.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:24.050645+0000 mgr.y (mgr.24401) 876 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:25.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:24.050645+0000 mgr.y (mgr.24401) 876 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: audit 2024-04-26T16:40:24.585963+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: audit 2024-04-26T16:40:24.585963+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:24.595365+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:24.595365+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.023597+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.023597+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.026349+0000 osd.5 (osd.5) 425 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.026349+0000 osd.5 (osd.5) 425 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.043070+0000 osd.4 (osd.4) 495 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.043070+0000 osd.4 (osd.4) 495 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.045837+0000 osd.4 (osd.4) 496 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.045837+0000 osd.4 (osd.4) 496 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:40:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:23.950185+0000 osd.2 (osd.2) 483 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:23.950185+0000 osd.2 (osd.2) 483 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:23.952019+0000 osd.2 (osd.2) 484 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:23.952019+0000 osd.2 (osd.2) 484 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:24.050645+0000 mgr.y (mgr.24401) 876 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:24.050645+0000 mgr.y (mgr.24401) 876 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: audit 2024-04-26T16:40:24.585963+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: audit 2024-04-26T16:40:24.585963+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:24.595365+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:24.595365+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.023597+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.023597+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.026349+0000 osd.5 (osd.5) 425 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.026349+0000 osd.5 (osd.5) 425 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:40:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.043070+0000 osd.4 (osd.4) 495 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:40:25.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.043070+0000 osd.4 (osd.4) 495 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:40:25.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.045837+0000 osd.4 (osd.4) 496 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:40:25.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:25 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.045837+0000 osd.4 (osd.4) 496 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:23.950185+0000 osd.2 (osd.2) 483 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:23.950185+0000 osd.2 (osd.2) 483 : cluster [DBG] 6.14 deep-scrub starts 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:23.952019+0000 osd.2 (osd.2) 484 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:23.952019+0000 osd.2 (osd.2) 484 : cluster [DBG] 6.14 deep-scrub ok 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:24.050645+0000 mgr.y (mgr.24401) 876 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:24.050645+0000 mgr.y (mgr.24401) 876 : cluster [DBG] pgmap v1470: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: audit 2024-04-26T16:40:24.585963+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: audit 2024-04-26T16:40:24.585963+0000 mon.a (mon.0) 4023 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:24.595365+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:24.595365+0000 mon.a (mon.0) 4024 : cluster [DBG] osdmap e909: 8 total, 8 up, 8 in 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.023597+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.023597+0000 osd.5 (osd.5) 424 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.026349+0000 osd.5 (osd.5) 425 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.026349+0000 osd.5 (osd.5) 425 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.043070+0000 osd.4 (osd.4) 495 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.043070+0000 osd.4 (osd.4) 495 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.045837+0000 osd.4 (osd.4) 496 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:40:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:25 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.045837+0000 osd.4 (osd.4) 496 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.107384+0000 osd.3 (osd.3) 573 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.107384+0000 osd.3 (osd.3) 573 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.114068+0000 osd.3 (osd.3) 574 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.114068+0000 osd.3 (osd.3) 574 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.114848+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.114848+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.117344+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:40:26.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[29234]: cluster 2024-04-26T16:40:25.117344+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.107384+0000 osd.3 (osd.3) 573 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.107384+0000 osd.3 (osd.3) 573 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.114068+0000 osd.3 (osd.3) 574 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.114068+0000 osd.3 (osd.3) 574 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.114848+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.114848+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.117344+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:40:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:26 smithi050 bash[21463]: cluster 2024-04-26T16:40:25.117344+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:40:27.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.107384+0000 osd.3 (osd.3) 573 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:40:27.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.107384+0000 osd.3 (osd.3) 573 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:40:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.114068+0000 osd.3 (osd.3) 574 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:40:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.114068+0000 osd.3 (osd.3) 574 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:40:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.114848+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:40:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.114848+0000 osd.6 (osd.6) 99 : cluster [DBG] 2.14 deep-scrub starts 2024-04-26T16:40:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.117344+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:40:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:26 smithi190 bash[23868]: cluster 2024-04-26T16:40:25.117344+0000 osd.6 (osd.6) 100 : cluster [DBG] 2.14 deep-scrub ok 2024-04-26T16:40:27.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.051505+0000 mgr.y (mgr.24401) 877 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:27.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.051505+0000 mgr.y (mgr.24401) 877 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.138160+0000 osd.3 (osd.3) 575 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.138160+0000 osd.3 (osd.3) 575 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.142566+0000 osd.3 (osd.3) 576 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.142566+0000 osd.3 (osd.3) 576 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.014874+0000 osd.5 (osd.5) 426 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.014874+0000 osd.5 (osd.5) 426 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.017566+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:40:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.017566+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:40:27.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:40:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.051505+0000 mgr.y (mgr.24401) 877 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.051505+0000 mgr.y (mgr.24401) 877 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.138160+0000 osd.3 (osd.3) 575 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.138160+0000 osd.3 (osd.3) 575 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.142566+0000 osd.3 (osd.3) 576 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.142566+0000 osd.3 (osd.3) 576 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.014874+0000 osd.5 (osd.5) 426 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.014874+0000 osd.5 (osd.5) 426 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.017566+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:40:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.017566+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.051505+0000 mgr.y (mgr.24401) 877 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.051505+0000 mgr.y (mgr.24401) 877 : cluster [DBG] pgmap v1472: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.138160+0000 osd.3 (osd.3) 575 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.138160+0000 osd.3 (osd.3) 575 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.142566+0000 osd.3 (osd.3) 576 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.142566+0000 osd.3 (osd.3) 576 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.014874+0000 osd.5 (osd.5) 426 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.014874+0000 osd.5 (osd.5) 426 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.017566+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:40:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:27 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.017566+0000 osd.5 (osd.5) 427 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:40:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.895949+0000 osd.2 (osd.2) 485 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:40:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.895949+0000 osd.2 (osd.2) 485 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:40:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.899352+0000 osd.2 (osd.2) 486 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:40:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:26.899352+0000 osd.2 (osd.2) 486 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.137633+0000 osd.3 (osd.3) 577 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.137633+0000 osd.3 (osd.3) 577 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.140695+0000 osd.3 (osd.3) 578 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.140695+0000 osd.3 (osd.3) 578 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:27.483282+0000 mgr.y (mgr.24401) 878 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:27.483282+0000 mgr.y (mgr.24401) 878 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:27.774913+0000 mon.c (mon.2) 933 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:27.774913+0000 mon.c (mon.2) 933 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:28.357180+0000 mon.a (mon.0) 4025 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:28.357180+0000 mon.a (mon.0) 4025 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:28.370799+0000 mon.a (mon.0) 4026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:28.370799+0000 mon.a (mon.0) 4026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:28.394134+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:28.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[21463]: audit 2024-04-26T16:40:28.394134+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.895949+0000 osd.2 (osd.2) 485 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.895949+0000 osd.2 (osd.2) 485 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.899352+0000 osd.2 (osd.2) 486 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:26.899352+0000 osd.2 (osd.2) 486 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.137633+0000 osd.3 (osd.3) 577 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.137633+0000 osd.3 (osd.3) 577 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.140695+0000 osd.3 (osd.3) 578 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.140695+0000 osd.3 (osd.3) 578 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:27.483282+0000 mgr.y (mgr.24401) 878 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:27.483282+0000 mgr.y (mgr.24401) 878 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:27.774913+0000 mon.c (mon.2) 933 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:28.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:27.774913+0000 mon.c (mon.2) 933 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:28.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:28.357180+0000 mon.a (mon.0) 4025 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:28.357180+0000 mon.a (mon.0) 4025 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:28.370799+0000 mon.a (mon.0) 4026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:28.370799+0000 mon.a (mon.0) 4026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:28.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:28.394134+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:28.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:28 smithi050 bash[29234]: audit 2024-04-26T16:40:28.394134+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.895949+0000 osd.2 (osd.2) 485 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:40:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.895949+0000 osd.2 (osd.2) 485 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:40:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.899352+0000 osd.2 (osd.2) 486 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:40:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:26.899352+0000 osd.2 (osd.2) 486 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:40:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.137633+0000 osd.3 (osd.3) 577 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:40:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.137633+0000 osd.3 (osd.3) 577 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.140695+0000 osd.3 (osd.3) 578 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.140695+0000 osd.3 (osd.3) 578 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:27.483282+0000 mgr.y (mgr.24401) 878 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:27.483282+0000 mgr.y (mgr.24401) 878 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:27.774913+0000 mon.c (mon.2) 933 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:27.774913+0000 mon.c (mon.2) 933 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:28.357180+0000 mon.a (mon.0) 4025 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:28.357180+0000 mon.a (mon.0) 4025 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:28.370799+0000 mon.a (mon.0) 4026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:28.370799+0000 mon.a (mon.0) 4026 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:28.394134+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:28 smithi190 bash[23868]: audit 2024-04-26T16:40:28.394134+0000 mon.c (mon.2) 934 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.946140+0000 osd.2 (osd.2) 487 : cluster [DBG] 2.e scrub starts 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.946140+0000 osd.2 (osd.2) 487 : cluster [DBG] 2.e scrub starts 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.948181+0000 osd.2 (osd.2) 488 : cluster [DBG] 2.e scrub ok 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:27.948181+0000 osd.2 (osd.2) 488 : cluster [DBG] 2.e scrub ok 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:28.053323+0000 mgr.y (mgr.24401) 879 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:28.053323+0000 mgr.y (mgr.24401) 879 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.078917+0000 osd.6 (osd.6) 101 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:40:29.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.078917+0000 osd.6 (osd.6) 101 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.081517+0000 osd.6 (osd.6) 102 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.081517+0000 osd.6 (osd.6) 102 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.946140+0000 osd.2 (osd.2) 487 : cluster [DBG] 2.e scrub starts 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.946140+0000 osd.2 (osd.2) 487 : cluster [DBG] 2.e scrub starts 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.948181+0000 osd.2 (osd.2) 488 : cluster [DBG] 2.e scrub ok 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:27.948181+0000 osd.2 (osd.2) 488 : cluster [DBG] 2.e scrub ok 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:28.053323+0000 mgr.y (mgr.24401) 879 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:28.053323+0000 mgr.y (mgr.24401) 879 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.078917+0000 osd.6 (osd.6) 101 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.078917+0000 osd.6 (osd.6) 101 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.081517+0000 osd.6 (osd.6) 102 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:40:29.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:29 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.081517+0000 osd.6 (osd.6) 102 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:40:30.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.946140+0000 osd.2 (osd.2) 487 : cluster [DBG] 2.e scrub starts 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.946140+0000 osd.2 (osd.2) 487 : cluster [DBG] 2.e scrub starts 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.948181+0000 osd.2 (osd.2) 488 : cluster [DBG] 2.e scrub ok 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:27.948181+0000 osd.2 (osd.2) 488 : cluster [DBG] 2.e scrub ok 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:28.053323+0000 mgr.y (mgr.24401) 879 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:28.053323+0000 mgr.y (mgr.24401) 879 : cluster [DBG] pgmap v1473: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.078917+0000 osd.6 (osd.6) 101 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.078917+0000 osd.6 (osd.6) 101 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.081517+0000 osd.6 (osd.6) 102 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:40:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:29 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.081517+0000 osd.6 (osd.6) 102 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:40:30.404 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:40:30.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.080070+0000 osd.3 (osd.3) 579 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.080070+0000 osd.3 (osd.3) 579 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.082508+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[29234]: cluster 2024-04-26T16:40:29.082508+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.080070+0000 osd.3 (osd.3) 579 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.080070+0000 osd.3 (osd.3) 579 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.082508+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:40:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:30 smithi050 bash[21463]: cluster 2024-04-26T16:40:29.082508+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:40:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:30 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.080070+0000 osd.3 (osd.3) 579 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:40:31.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:30 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.080070+0000 osd.3 (osd.3) 579 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:40:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:30 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.082508+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:40:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:30 smithi190 bash[23868]: cluster 2024-04-26T16:40:29.082508+0000 osd.3 (osd.3) 580 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:40:31.873 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.054279+0000 mgr.y (mgr.24401) 880 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:31.873 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.054279+0000 mgr.y (mgr.24401) 880 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.262774+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.262774+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.265451+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.265451+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.013725+0000 osd.5 (osd.5) 428 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.013725+0000 osd.5 (osd.5) 428 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.016839+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.016839+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.054279+0000 mgr.y (mgr.24401) 880 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.054279+0000 mgr.y (mgr.24401) 880 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:31.874 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.262774+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.262774+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.265451+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.265451+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.013725+0000 osd.5 (osd.5) 428 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.013725+0000 osd.5 (osd.5) 428 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.016839+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:40:31.875 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:31 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.016839+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:40:32.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.054279+0000 mgr.y (mgr.24401) 880 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.054279+0000 mgr.y (mgr.24401) 880 : cluster [DBG] pgmap v1474: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.262774+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.262774+0000 osd.0 (osd.0) 557 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.265451+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.265451+0000 osd.0 (osd.0) 558 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.013725+0000 osd.5 (osd.5) 428 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.013725+0000 osd.5 (osd.5) 428 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.016839+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:40:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:31 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.016839+0000 osd.5 (osd.5) 429 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.989947+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 deep-scrub starts 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.989947+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 deep-scrub starts 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.992455+0000 osd.1 (osd.1) 530 : cluster [DBG] 2.15 deep-scrub ok 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:30.992455+0000 osd.1 (osd.1) 530 : cluster [DBG] 2.15 deep-scrub ok 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.917370+0000 osd.2 (osd.2) 489 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.917370+0000 osd.2 (osd.2) 489 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.919250+0000 osd.2 (osd.2) 490 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[21463]: cluster 2024-04-26T16:40:31.919250+0000 osd.2 (osd.2) 490 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:40:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.989947+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 deep-scrub starts 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.989947+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 deep-scrub starts 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.992455+0000 osd.1 (osd.1) 530 : cluster [DBG] 2.15 deep-scrub ok 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:30.992455+0000 osd.1 (osd.1) 530 : cluster [DBG] 2.15 deep-scrub ok 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.917370+0000 osd.2 (osd.2) 489 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.917370+0000 osd.2 (osd.2) 489 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.919250+0000 osd.2 (osd.2) 490 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:40:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:32 smithi050 bash[29234]: cluster 2024-04-26T16:40:31.919250+0000 osd.2 (osd.2) 490 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:40:33.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.989947+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 deep-scrub starts 2024-04-26T16:40:33.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.989947+0000 osd.1 (osd.1) 529 : cluster [DBG] 2.15 deep-scrub starts 2024-04-26T16:40:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.992455+0000 osd.1 (osd.1) 530 : cluster [DBG] 2.15 deep-scrub ok 2024-04-26T16:40:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:30.992455+0000 osd.1 (osd.1) 530 : cluster [DBG] 2.15 deep-scrub ok 2024-04-26T16:40:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.917370+0000 osd.2 (osd.2) 489 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:40:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.917370+0000 osd.2 (osd.2) 489 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:40:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.919250+0000 osd.2 (osd.2) 490 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:40:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:32 smithi190 bash[23868]: cluster 2024-04-26T16:40:31.919250+0000 osd.2 (osd.2) 490 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 -- 172.21.15.50:0/4076181837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41034d0 msgr2=0x7f2ad4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 --2- 172.21.15.50:0/4076181837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41034d0 0x7f2ad4103950 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009990 tx=0x7f2ac8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 -- 172.21.15.50:0/4076181837 shutdown_connections 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 --2- 172.21.15.50:0/4076181837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad4103e90 0x7f2ad41102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 --2- 172.21.15.50:0/4076181837 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ad41094f0 0x7f2ad41098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 --2- 172.21.15.50:0/4076181837 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41034d0 0x7f2ad4103950 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 -- 172.21.15.50:0/4076181837 >> 172.21.15.50:0/4076181837 conn(0x7f2ad407a500 msgr2=0x7f2ad407a910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:33.838 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.835+0000 7f2ad9697700 1 -- 172.21.15.50:0/4076181837 shutdown_connections 2024-04-26T16:40:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- 172.21.15.50:0/4076181837 wait complete. 2024-04-26T16:40:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 librados: starting msgr at 2024-04-26T16:40:33.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 librados: starting objecter 2024-04-26T16:40:33.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 Processor -- start 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- start start 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 librados: setting wanted keys 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 librados: calling monclient init 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41034d0 0x7f2ad4075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ad4103e90 0x7f2ad4075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 0x7f2ad40761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4112ad0 con 0x7f2ad41094f0 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad4112950 con 0x7f2ad41034d0 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f2ad4112c50 con 0x7f2ad4103e90 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 0x7f2ad40761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad27fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ad4103e90 0x7f2ad4075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 0x7f2ad40761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:56892/0 (socket says 172.21.15.50:56892) 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 -- 172.21.15.50:0/487660970 learned_addr learned my addr 172.21.15.50:0/487660970 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:40:33.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad2ffd700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41034d0 0x7f2ad4075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 -- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ad4103e90 msgr2=0x7f2ad4075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ad4103e90 0x7f2ad4075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 -- 172.21.15.50:0/487660970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41034d0 msgr2=0x7f2ad4075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41034d0 0x7f2ad4075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 -- 172.21.15.50:0/487660970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad41a76a0 con 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad37fe700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 0x7f2ad40761a0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f2acc007ee0 tx=0x7f2acc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2abbfff700 10 client.?.objecter ms_handle_connect 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2abbfff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc004bb0 con 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f2acc004d50 con 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc005720 con 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad41a7990 con 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 10 client.31221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad41a7ef0 con 0x7f2ad41094f0 2024-04-26T16:40:33.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 librados: init done 2024-04-26T16:40:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad406a110 con 0x7f2ad41094f0 2024-04-26T16:40:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.839+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f2acc01a020 con 0x7f2ad41094f0 2024-04-26T16:40:33.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2abc071f80 0x7f2abc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(909..909 src has 854..909) ==== 8707+0+0 (secure 0 0 0) 0x7f2acc090e90 con 0x7f2ad41094f0 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 10 client.31221.objecter ms_dispatch 0x7f2ad4001550 osd_map(909..909 src has 854..909) 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 3 client.31221.objecter handle_osd_map got epochs [909,909] > 0 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 3 client.31221.objecter handle_osd_map decoding full epoch 909 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2ad2ffd700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2abc071f80 0x7f2abc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 20 client.31221.objecter dump_active .. 0 homeless 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2ad2ffd700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2abc071f80 0x7f2abc074440 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009be0 tx=0x7f2ac4009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:33.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.843+0000 7f2abbfff700 10 client.31221.objecter ms_handle_connect 0x7f2abc071f80 2024-04-26T16:40:33.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:33.847+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f2acc010070 con 0x7f2ad41094f0 2024-04-26T16:40:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:33 smithi190 bash[23868]: cluster 2024-04-26T16:40:32.056134+0000 mgr.y (mgr.24401) 881 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:33 smithi190 bash[23868]: cluster 2024-04-26T16:40:32.056134+0000 mgr.y (mgr.24401) 881 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:33 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.047677+0000 osd.5 (osd.5) 430 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:40:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:33 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.047677+0000 osd.5 (osd.5) 430 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:40:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:33 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.053966+0000 osd.5 (osd.5) 431 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:40:34.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:33 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.053966+0000 osd.5 (osd.5) 431 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:40:34.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.079+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 4, "weight": 1.0} v 0) -- 0x7f2ad41a8180 con 0x7f2ad41094f0 2024-04-26T16:40:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[21463]: cluster 2024-04-26T16:40:32.056134+0000 mgr.y (mgr.24401) 881 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:34.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[21463]: cluster 2024-04-26T16:40:32.056134+0000 mgr.y (mgr.24401) 881 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.047677+0000 osd.5 (osd.5) 430 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.047677+0000 osd.5 (osd.5) 430 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.053966+0000 osd.5 (osd.5) 431 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.053966+0000 osd.5 (osd.5) 431 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[29234]: cluster 2024-04-26T16:40:32.056134+0000 mgr.y (mgr.24401) 881 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[29234]: cluster 2024-04-26T16:40:32.056134+0000 mgr.y (mgr.24401) 881 : cluster [DBG] pgmap v1475: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.047677+0000 osd.5 (osd.5) 430 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.047677+0000 osd.5 (osd.5) 430 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.053966+0000 osd.5 (osd.5) 431 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:40:34.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:33 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.053966+0000 osd.5 (osd.5) 431 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:40:34.718 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:40:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:40:34.729 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.727+0000 7f2abbfff700 1 -- 172.21.15.50:0/487660970 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 4, "weight": 1.0}]=0 reweighted osd.4 to 1 (10000) v910) ==== 113+0+0 (secure 0 0 0) 0x7f2acc05cb60 con 0x7f2ad41094f0 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 10 librados: watch_flush enter 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 10 librados: watch_flush exit 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 20 client.31221.objecter shutdown clearing up homeless session... 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 10 client.31221.objecter successfully canceled tick 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2abc071f80 msgr2=0x7f2abc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2abc071f80 0x7f2abc074440 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4009be0 tx=0x7f2ac4009260 comp rx=0 tx=0).stop 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 msgr2=0x7f2ad40761a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:34.736 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 0x7f2ad40761a0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f2acc007ee0 tx=0x7f2acc00d3b0 comp rx=0 tx=0).stop 2024-04-26T16:40:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 shutdown_connections 2024-04-26T16:40:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2abc071f80 0x7f2abc074440 unknown :-1 s=CLOSED pgs=3864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2ad41034d0 0x7f2ad4075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f2ad41094f0 0x7f2ad40761a0 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:34.737 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 --2- 172.21.15.50:0/487660970 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f2ad4103e90 0x7f2ad4075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.735+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 >> 172.21.15.50:0/487660970 conn(0x7f2ad407a500 msgr2=0x7f2ad4101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.739+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 shutdown_connections 2024-04-26T16:40:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.739+0000 7f2ad9697700 1 -- 172.21.15.50:0/487660970 wait complete. 2024-04-26T16:40:34.738 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:34.739+0000 7f2ad9697700 1 librados: shutdown 2024-04-26T16:40:34.739 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.4 to 1 (10000) 2024-04-26T16:40:35.060 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.040901+0000 osd.3 (osd.3) 581 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.040901+0000 osd.3 (osd.3) 581 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.044505+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:33.044505+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:33.721350+0000 mon.a (mon.0) 4027 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:33.721350+0000 mon.a (mon.0) 4027 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:33.727886+0000 mon.a (mon.0) 4028 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:33.727886+0000 mon.a (mon.0) 4028 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.022020+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.022020+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.024222+0000 osd.5 (osd.5) 433 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.024222+0000 osd.5 (osd.5) 433 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.052376+0000 osd.4 (osd.4) 497 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.052376+0000 osd.4 (osd.4) 497 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:40:35.061 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.054247+0000 osd.4 (osd.4) 498 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.054247+0000 osd.4 (osd.4) 498 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.067195+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.067195+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.068306+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.068306+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:34.086574+0000 mon.a (mon.0) 4029 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:34.086574+0000 mon.a (mon.0) 4029 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:34.485081+0000 mon.a (mon.0) 4030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:34.485081+0000 mon.a (mon.0) 4030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:34.501991+0000 mon.a (mon.0) 4031 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[21463]: audit 2024-04-26T16:40:34.501991+0000 mon.a (mon.0) 4031 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.062 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.040901+0000 osd.3 (osd.3) 581 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.040901+0000 osd.3 (osd.3) 581 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.044505+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:33.044505+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:33.721350+0000 mon.a (mon.0) 4027 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:33.721350+0000 mon.a (mon.0) 4027 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:33.727886+0000 mon.a (mon.0) 4028 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:33.727886+0000 mon.a (mon.0) 4028 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.022020+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.022020+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.024222+0000 osd.5 (osd.5) 433 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.024222+0000 osd.5 (osd.5) 433 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.052376+0000 osd.4 (osd.4) 497 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.052376+0000 osd.4 (osd.4) 497 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.054247+0000 osd.4 (osd.4) 498 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.054247+0000 osd.4 (osd.4) 498 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:40:35.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.067195+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.067195+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.068306+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.068306+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:34.086574+0000 mon.a (mon.0) 4029 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:34.086574+0000 mon.a (mon.0) 4029 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:34.485081+0000 mon.a (mon.0) 4030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:34.485081+0000 mon.a (mon.0) 4030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:34.501991+0000 mon.a (mon.0) 4031 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:34 smithi050 bash[29234]: audit 2024-04-26T16:40:34.501991+0000 mon.a (mon.0) 4031 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.040901+0000 osd.3 (osd.3) 581 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.040901+0000 osd.3 (osd.3) 581 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.044505+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:33.044505+0000 osd.3 (osd.3) 582 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:33.721350+0000 mon.a (mon.0) 4027 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:33.721350+0000 mon.a (mon.0) 4027 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:33.727886+0000 mon.a (mon.0) 4028 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:33.727886+0000 mon.a (mon.0) 4028 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.022020+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:40:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.022020+0000 osd.5 (osd.5) 432 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.024222+0000 osd.5 (osd.5) 433 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.024222+0000 osd.5 (osd.5) 433 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.052376+0000 osd.4 (osd.4) 497 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.052376+0000 osd.4 (osd.4) 497 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.054247+0000 osd.4 (osd.4) 498 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.054247+0000 osd.4 (osd.4) 498 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.067195+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.067195+0000 osd.6 (osd.6) 103 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.068306+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.068306+0000 osd.6 (osd.6) 104 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:34.086574+0000 mon.a (mon.0) 4029 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:34.086574+0000 mon.a (mon.0) 4029 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:34.485081+0000 mon.a (mon.0) 4030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:34.485081+0000 mon.a (mon.0) 4030 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:34.501991+0000 mon.a (mon.0) 4031 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:34 smithi190 bash[23868]: audit 2024-04-26T16:40:34.501991+0000 mon.a (mon.0) 4031 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:35.702 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 4 1 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.057043+0000 mgr.y (mgr.24401) 882 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.057043+0000 mgr.y (mgr.24401) 882 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:34.734243+0000 mon.a (mon.0) 4032 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:34.734243+0000 mon.a (mon.0) 4032 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.742013+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.742013+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.977585+0000 osd.5 (osd.5) 434 : cluster [DBG] 2.18 deep-scrub starts 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.977585+0000 osd.5 (osd.5) 434 : cluster [DBG] 2.18 deep-scrub starts 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.980856+0000 osd.5 (osd.5) 435 : cluster [DBG] 2.18 deep-scrub ok 2024-04-26T16:40:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: cluster 2024-04-26T16:40:34.980856+0000 osd.5 (osd.5) 435 : cluster [DBG] 2.18 deep-scrub ok 2024-04-26T16:40:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:35.193863+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:35.193863+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:35.195934+0000 mon.c (mon.2) 936 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:35.195934+0000 mon.c (mon.2) 936 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:35.207305+0000 mon.a (mon.0) 4034 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:35 smithi190 bash[23868]: audit 2024-04-26T16:40:35.207305+0000 mon.a (mon.0) 4034 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.057043+0000 mgr.y (mgr.24401) 882 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.057043+0000 mgr.y (mgr.24401) 882 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:34.734243+0000 mon.a (mon.0) 4032 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:34.734243+0000 mon.a (mon.0) 4032 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.742013+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.742013+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.977585+0000 osd.5 (osd.5) 434 : cluster [DBG] 2.18 deep-scrub starts 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.977585+0000 osd.5 (osd.5) 434 : cluster [DBG] 2.18 deep-scrub starts 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.980856+0000 osd.5 (osd.5) 435 : cluster [DBG] 2.18 deep-scrub ok 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: cluster 2024-04-26T16:40:34.980856+0000 osd.5 (osd.5) 435 : cluster [DBG] 2.18 deep-scrub ok 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:35.193863+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:35.193863+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:35.195934+0000 mon.c (mon.2) 936 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:35.195934+0000 mon.c (mon.2) 936 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:35.207305+0000 mon.a (mon.0) 4034 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[29234]: audit 2024-04-26T16:40:35.207305+0000 mon.a (mon.0) 4034 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.057043+0000 mgr.y (mgr.24401) 882 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.057043+0000 mgr.y (mgr.24401) 882 : cluster [DBG] pgmap v1476: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:34.734243+0000 mon.a (mon.0) 4032 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:34.734243+0000 mon.a (mon.0) 4032 : audit [INF] from='client.? 172.21.15.50:0/487660970' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.742013+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.742013+0000 mon.a (mon.0) 4033 : cluster [DBG] osdmap e910: 8 total, 8 up, 8 in 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.977585+0000 osd.5 (osd.5) 434 : cluster [DBG] 2.18 deep-scrub starts 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.977585+0000 osd.5 (osd.5) 434 : cluster [DBG] 2.18 deep-scrub starts 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.980856+0000 osd.5 (osd.5) 435 : cluster [DBG] 2.18 deep-scrub ok 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: cluster 2024-04-26T16:40:34.980856+0000 osd.5 (osd.5) 435 : cluster [DBG] 2.18 deep-scrub ok 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:35.193863+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:35.193863+0000 mon.c (mon.2) 935 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:35.195934+0000 mon.c (mon.2) 936 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:35.195934+0000 mon.c (mon.2) 936 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:35.207305+0000 mon.a (mon.0) 4034 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:36.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:35 smithi050 bash[21463]: audit 2024-04-26T16:40:35.207305+0000 mon.a (mon.0) 4034 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:40:37.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:36.057823+0000 mgr.y (mgr.24401) 883 : cluster [DBG] pgmap v1478: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:36.057823+0000 mgr.y (mgr.24401) 883 : cluster [DBG] pgmap v1478: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:36.930206+0000 osd.5 (osd.5) 436 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:36.930206+0000 osd.5 (osd.5) 436 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:36.932692+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:36.932692+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.038217+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.038217+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.041539+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:40:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:37 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.041539+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:36.057823+0000 mgr.y (mgr.24401) 883 : cluster [DBG] pgmap v1478: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:36.057823+0000 mgr.y (mgr.24401) 883 : cluster [DBG] pgmap v1478: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:36.930206+0000 osd.5 (osd.5) 436 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:36.930206+0000 osd.5 (osd.5) 436 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:36.932692+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:36.932692+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.038217+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:40:38.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.038217+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.041539+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.041539+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:36.057823+0000 mgr.y (mgr.24401) 883 : cluster [DBG] pgmap v1478: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:36.057823+0000 mgr.y (mgr.24401) 883 : cluster [DBG] pgmap v1478: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:36.930206+0000 osd.5 (osd.5) 436 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:36.930206+0000 osd.5 (osd.5) 436 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:36.932692+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:36.932692+0000 osd.5 (osd.5) 437 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.038217+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.038217+0000 osd.6 (osd.6) 105 : cluster [DBG] 3.10 deep-scrub starts 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.041539+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:40:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:37 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.041539+0000 osd.6 (osd.6) 106 : cluster [DBG] 3.10 deep-scrub ok 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.282096+0000 osd.0 (osd.0) 559 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.282096+0000 osd.0 (osd.0) 559 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.284666+0000 osd.0 (osd.0) 560 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.284666+0000 osd.0 (osd.0) 560 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: audit 2024-04-26T16:40:37.492929+0000 mgr.y (mgr.24401) 884 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: audit 2024-04-26T16:40:37.492929+0000 mgr.y (mgr.24401) 884 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:38.029137+0000 osd.6 (osd.6) 107 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:38.029137+0000 osd.6 (osd.6) 107 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:38.031686+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:40:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:38 smithi190 bash[23868]: cluster 2024-04-26T16:40:38.031686+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.282096+0000 osd.0 (osd.0) 559 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.282096+0000 osd.0 (osd.0) 559 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.284666+0000 osd.0 (osd.0) 560 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.284666+0000 osd.0 (osd.0) 560 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: audit 2024-04-26T16:40:37.492929+0000 mgr.y (mgr.24401) 884 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: audit 2024-04-26T16:40:37.492929+0000 mgr.y (mgr.24401) 884 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:38.029137+0000 osd.6 (osd.6) 107 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:38.029137+0000 osd.6 (osd.6) 107 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:38.031686+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:40:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[29234]: cluster 2024-04-26T16:40:38.031686+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.282096+0000 osd.0 (osd.0) 559 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.282096+0000 osd.0 (osd.0) 559 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.284666+0000 osd.0 (osd.0) 560 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.284666+0000 osd.0 (osd.0) 560 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: audit 2024-04-26T16:40:37.492929+0000 mgr.y (mgr.24401) 884 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: audit 2024-04-26T16:40:37.492929+0000 mgr.y (mgr.24401) 884 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:38.029137+0000 osd.6 (osd.6) 107 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:38.029137+0000 osd.6 (osd.6) 107 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:38.031686+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:40:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:38 smithi050 bash[21463]: cluster 2024-04-26T16:40:38.031686+0000 osd.6 (osd.6) 108 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:40:40.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:39 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.965045+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:40:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:39 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.965045+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:40:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:39 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.967630+0000 osd.1 (osd.1) 532 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:40:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:39 smithi190 bash[23868]: cluster 2024-04-26T16:40:37.967630+0000 osd.1 (osd.1) 532 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:40:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:39 smithi190 bash[23868]: cluster 2024-04-26T16:40:38.059757+0000 mgr.y (mgr.24401) 885 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:39 smithi190 bash[23868]: cluster 2024-04-26T16:40:38.059757+0000 mgr.y (mgr.24401) 885 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.965045+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.965045+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.967630+0000 osd.1 (osd.1) 532 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[29234]: cluster 2024-04-26T16:40:37.967630+0000 osd.1 (osd.1) 532 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[29234]: cluster 2024-04-26T16:40:38.059757+0000 mgr.y (mgr.24401) 885 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[29234]: cluster 2024-04-26T16:40:38.059757+0000 mgr.y (mgr.24401) 885 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.965045+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.965045+0000 osd.1 (osd.1) 531 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:40:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.967630+0000 osd.1 (osd.1) 532 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:40:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[21463]: cluster 2024-04-26T16:40:37.967630+0000 osd.1 (osd.1) 532 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:40:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[21463]: cluster 2024-04-26T16:40:38.059757+0000 mgr.y (mgr.24401) 885 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:39 smithi050 bash[21463]: cluster 2024-04-26T16:40:38.059757+0000 mgr.y (mgr.24401) 885 : cluster [DBG] pgmap v1479: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:40.506 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:39.039006+0000 osd.3 (osd.3) 583 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:39.039006+0000 osd.3 (osd.3) 583 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:39.041597+0000 osd.3 (osd.3) 584 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:39.041597+0000 osd.3 (osd.3) 584 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.094365+0000 osd.4 (osd.4) 499 : cluster [DBG] 11.e scrub starts 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.094365+0000 osd.4 (osd.4) 499 : cluster [DBG] 11.e scrub starts 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.097418+0000 osd.4 (osd.4) 500 : cluster [DBG] 11.e scrub ok 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.097418+0000 osd.4 (osd.4) 500 : cluster [DBG] 11.e scrub ok 2024-04-26T16:40:40.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:39.039006+0000 osd.3 (osd.3) 583 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:39.039006+0000 osd.3 (osd.3) 583 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:39.041597+0000 osd.3 (osd.3) 584 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:39.041597+0000 osd.3 (osd.3) 584 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.094365+0000 osd.4 (osd.4) 499 : cluster [DBG] 11.e scrub starts 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.094365+0000 osd.4 (osd.4) 499 : cluster [DBG] 11.e scrub starts 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.097418+0000 osd.4 (osd.4) 500 : cluster [DBG] 11.e scrub ok 2024-04-26T16:40:40.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:40 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.097418+0000 osd.4 (osd.4) 500 : cluster [DBG] 11.e scrub ok 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:39.039006+0000 osd.3 (osd.3) 583 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:39.039006+0000 osd.3 (osd.3) 583 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:39.041597+0000 osd.3 (osd.3) 584 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:39.041597+0000 osd.3 (osd.3) 584 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.094365+0000 osd.4 (osd.4) 499 : cluster [DBG] 11.e scrub starts 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.094365+0000 osd.4 (osd.4) 499 : cluster [DBG] 11.e scrub starts 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.097418+0000 osd.4 (osd.4) 500 : cluster [DBG] 11.e scrub ok 2024-04-26T16:40:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:40 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.097418+0000 osd.4 (osd.4) 500 : cluster [DBG] 11.e scrub ok 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.060675+0000 mgr.y (mgr.24401) 886 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.060675+0000 mgr.y (mgr.24401) 886 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.993465+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.993465+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.996716+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:40.996716+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.046466+0000 osd.5 (osd.5) 438 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.046466+0000 osd.5 (osd.5) 438 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.048676+0000 osd.5 (osd.5) 439 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:40:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:41 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.048676+0000 osd.5 (osd.5) 439 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.060675+0000 mgr.y (mgr.24401) 886 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.060675+0000 mgr.y (mgr.24401) 886 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.993465+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.993465+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.996716+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:40.996716+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.046466+0000 osd.5 (osd.5) 438 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.046466+0000 osd.5 (osd.5) 438 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.048676+0000 osd.5 (osd.5) 439 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.048676+0000 osd.5 (osd.5) 439 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:40:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.060675+0000 mgr.y (mgr.24401) 886 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.060675+0000 mgr.y (mgr.24401) 886 : cluster [DBG] pgmap v1480: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.993465+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.993465+0000 osd.6 (osd.6) 109 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.996716+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:40.996716+0000 osd.6 (osd.6) 110 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.046466+0000 osd.5 (osd.5) 438 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.046466+0000 osd.5 (osd.5) 438 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.048676+0000 osd.5 (osd.5) 439 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:40:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:41 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.048676+0000 osd.5 (osd.5) 439 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:40:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.019378+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:40:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.019378+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:40:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.021783+0000 osd.3 (osd.3) 586 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:40:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.021783+0000 osd.3 (osd.3) 586 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.853788+0000 osd.2 (osd.2) 491 : cluster [DBG] 2.c scrub starts 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.853788+0000 osd.2 (osd.2) 491 : cluster [DBG] 2.c scrub starts 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.855556+0000 osd.2 (osd.2) 492 : cluster [DBG] 2.c scrub ok 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:41.855556+0000 osd.2 (osd.2) 492 : cluster [DBG] 2.c scrub ok 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.010843+0000 osd.7 (osd.7) 560 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.010843+0000 osd.7 (osd.7) 560 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.013613+0000 osd.7 (osd.7) 561 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:40:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:42 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.013613+0000 osd.7 (osd.7) 561 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.019378+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.019378+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.021783+0000 osd.3 (osd.3) 586 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.021783+0000 osd.3 (osd.3) 586 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.853788+0000 osd.2 (osd.2) 491 : cluster [DBG] 2.c scrub starts 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.853788+0000 osd.2 (osd.2) 491 : cluster [DBG] 2.c scrub starts 2024-04-26T16:40:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.855556+0000 osd.2 (osd.2) 492 : cluster [DBG] 2.c scrub ok 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:41.855556+0000 osd.2 (osd.2) 492 : cluster [DBG] 2.c scrub ok 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.010843+0000 osd.7 (osd.7) 560 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.010843+0000 osd.7 (osd.7) 560 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.013613+0000 osd.7 (osd.7) 561 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.013613+0000 osd.7 (osd.7) 561 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.019378+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.019378+0000 osd.3 (osd.3) 585 : cluster [DBG] 6.b deep-scrub starts 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.021783+0000 osd.3 (osd.3) 586 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.021783+0000 osd.3 (osd.3) 586 : cluster [DBG] 6.b deep-scrub ok 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.853788+0000 osd.2 (osd.2) 491 : cluster [DBG] 2.c scrub starts 2024-04-26T16:40:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.853788+0000 osd.2 (osd.2) 491 : cluster [DBG] 2.c scrub starts 2024-04-26T16:40:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.855556+0000 osd.2 (osd.2) 492 : cluster [DBG] 2.c scrub ok 2024-04-26T16:40:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:41.855556+0000 osd.2 (osd.2) 492 : cluster [DBG] 2.c scrub ok 2024-04-26T16:40:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.010843+0000 osd.7 (osd.7) 560 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:40:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.010843+0000 osd.7 (osd.7) 560 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:40:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.013613+0000 osd.7 (osd.7) 561 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:40:43.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:42 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.013613+0000 osd.7 (osd.7) 561 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.062667+0000 mgr.y (mgr.24401) 887 : cluster [DBG] pgmap v1481: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.062667+0000 mgr.y (mgr.24401) 887 : cluster [DBG] pgmap v1481: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.979129+0000 osd.7 (osd.7) 562 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.979129+0000 osd.7 (osd.7) 562 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.981358+0000 osd.7 (osd.7) 563 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:42.981358+0000 osd.7 (osd.7) 563 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.007235+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.007235+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.009724+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:40:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.009724+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:40:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: audit 2024-04-26T16:40:43.412640+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:43 smithi190 bash[23868]: audit 2024-04-26T16:40:43.412640+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/1144948965 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f5010b660 msgr2=0x7f7f5010ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- 172.21.15.50:0/1144948965 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f5010b660 0x7f7f5010ba40 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0098f0 tx=0x7f7f4c009c00 comp rx=0 tx=0).stop 2024-04-26T16:40:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/1144948965 shutdown_connections 2024-04-26T16:40:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- 172.21.15.50:0/1144948965 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f50106020 0x7f7f50112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.127 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- 172.21.15.50:0/1144948965 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f5010b660 0x7f7f5010ba40 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- 172.21.15.50:0/1144948965 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50105660 0x7f7f50105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/1144948965 >> 172.21.15.50:0/1144948965 conn(0x7f7f50101150 msgr2=0x7f7f50103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/1144948965 shutdown_connections 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/1144948965 wait complete. 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 librados: starting msgr at 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 librados: starting objecter 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 Processor -- start 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- start start 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 librados: setting wanted keys 2024-04-26T16:40:44.128 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 librados: calling monclient init 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 0x7f7f5019b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 0x7f7f5019b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f5010b660 0x7f7f5019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50114d80 con 0x7f7f50106020 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7f50114c00 con 0x7f7f5010b660 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7f50114f00 con 0x7f7f50105660 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f553ff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 0x7f7f5019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 0x7f7f5019b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 0x7f7f5019b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:47514/0 (socket says 172.21.15.50:47514) 2024-04-26T16:40:44.130 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 -- 172.21.15.50:0/658112622 learned_addr learned my addr 172.21.15.50:0/658112622 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f553ff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 0x7f7f5019b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:35084/0 (socket says 172.21.15.50:35084) 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f55c00700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f5010b660 0x7f7f5019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 -- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 msgr2=0x7f7f5019b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 0x7f7f5019b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 -- 172.21.15.50:0/658112622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f5010b660 msgr2=0x7f7f5019fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f5010b660 0x7f7f5019fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 -- 172.21.15.50:0/658112622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f501a0340 con 0x7f7f50106020 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f553ff700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 0x7f7f5019b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f55c00700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f5010b660 0x7f7f5019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f54bfe700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 0x7f7f5019b870 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000e9c0 tx=0x7f7f4000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f467fc700 10 client.?.objecter ms_handle_connect 0x7f7f50106020 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f467fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000cbd0 con 0x7f7f50106020 2024-04-26T16:40:44.131 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7f4000cd70 con 0x7f7f50106020 2024-04-26T16:40:44.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f501a0630 con 0x7f7f50106020 2024-04-26T16:40:44.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40005400 con 0x7f7f50106020 2024-04-26T16:40:44.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 10 client.31227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:40:44.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f501a7eb0 con 0x7f7f50106020 2024-04-26T16:40:44.132 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.127+0000 7f7f57663700 1 librados: init done 2024-04-26T16:40:44.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.062667+0000 mgr.y (mgr.24401) 887 : cluster [DBG] pgmap v1481: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:44.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.062667+0000 mgr.y (mgr.24401) 887 : cluster [DBG] pgmap v1481: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:44.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.979129+0000 osd.7 (osd.7) 562 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:40:44.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.979129+0000 osd.7 (osd.7) 562 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.981358+0000 osd.7 (osd.7) 563 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:42.981358+0000 osd.7 (osd.7) 563 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.007235+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.007235+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.009724+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.009724+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: audit 2024-04-26T16:40:43.412640+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[29234]: audit 2024-04-26T16:40:43.412640+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:44.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.062667+0000 mgr.y (mgr.24401) 887 : cluster [DBG] pgmap v1481: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.062667+0000 mgr.y (mgr.24401) 887 : cluster [DBG] pgmap v1481: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.979129+0000 osd.7 (osd.7) 562 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.979129+0000 osd.7 (osd.7) 562 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.981358+0000 osd.7 (osd.7) 563 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:42.981358+0000 osd.7 (osd.7) 563 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.007235+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.007235+0000 osd.6 (osd.6) 111 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.009724+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.009724+0000 osd.6 (osd.6) 112 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: audit 2024-04-26T16:40:43.412640+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:44.134 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:43 smithi050 bash[21463]: audit 2024-04-26T16:40:43.412640+0000 mon.c (mon.2) 937 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:44.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f5010dbf0 con 0x7f7f50106020 2024-04-26T16:40:44.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7f400055a0 con 0x7f7f50106020 2024-04-26T16:40:44.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7f3c071f80 0x7f7f3c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:44.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(911..911 src has 854..911) ==== 8707+0+0 (secure 0 0 0) 0x7f7f40091960 con 0x7f7f50106020 2024-04-26T16:40:44.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 10 client.31227.objecter ms_dispatch 0x7f7f50001550 osd_map(911..911 src has 854..911) 2024-04-26T16:40:44.135 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 3 client.31227.objecter handle_osd_map got epochs [911,911] > 0 2024-04-26T16:40:44.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 3 client.31227.objecter handle_osd_map decoding full epoch 911 2024-04-26T16:40:44.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f553ff700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7f3c071f80 0x7f7f3c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:44.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 20 client.31227.objecter dump_active .. 0 homeless 2024-04-26T16:40:44.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f467fc700 10 client.31227.objecter ms_handle_connect 0x7f7f3c071f80 2024-04-26T16:40:44.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.131+0000 7f7f553ff700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7f3c071f80 0x7f7f3c074440 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0098c0 tx=0x7f7f4c019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:44.136 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.135+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7f4005d550 con 0x7f7f50106020 2024-04-26T16:40:44.372 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.371+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} v 0) -- 0x7f7f5019c690 con 0x7f7f50106020 2024-04-26T16:40:44.797 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.795+0000 7f7f467fc700 1 -- 172.21.15.50:0/658112622 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]=0 set osd.4 primary-affinity to 1 (8655362) v912) ==== 133+0+0 (secure 0 0 0) 0x7f7f40061200 con 0x7f7f50106020 2024-04-26T16:40:44.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 10 librados: watch_flush enter 2024-04-26T16:40:44.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 10 librados: watch_flush exit 2024-04-26T16:40:44.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 20 client.31227.objecter shutdown clearing up homeless session... 2024-04-26T16:40:44.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 10 client.31227.objecter successfully canceled tick 2024-04-26T16:40:44.805 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7f3c071f80 msgr2=0x7f7f3c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:44.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7f3c071f80 0x7f7f3c074440 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0098c0 tx=0x7f7f4c019040 comp rx=0 tx=0).stop 2024-04-26T16:40:44.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 msgr2=0x7f7f5019b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:44.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.803+0000 7f7f57663700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 0x7f7f5019b870 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000e9c0 tx=0x7f7f4000ecd0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.806 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 shutdown_connections 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7f3c071f80 0x7f7f3c074440 unknown :-1 s=CLOSED pgs=3865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f5010b660 0x7f7f5019fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7f50105660 0x7f7f5019b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 --2- 172.21.15.50:0/658112622 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7f50106020 0x7f7f5019b870 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 >> 172.21.15.50:0/658112622 conn(0x7f7f50101150 msgr2=0x7f7f50101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 shutdown_connections 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 -- 172.21.15.50:0/658112622 wait complete. 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:44.807+0000 7f7f57663700 1 librados: shutdown 2024-04-26T16:40:44.807 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-04-26T16:40:44.811 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:40:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:40:45.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.682691+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.682691+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.683081+0000 osd.6 (osd.6) 114 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.683081+0000 osd.6 (osd.6) 114 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.806899+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:43.806899+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.024933+0000 osd.7 (osd.7) 564 : cluster [DBG] 11.f deep-scrub starts 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.024933+0000 osd.7 (osd.7) 564 : cluster [DBG] 11.f deep-scrub starts 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.032580+0000 osd.7 (osd.7) 565 : cluster [DBG] 11.f deep-scrub ok 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.032580+0000 osd.7 (osd.7) 565 : cluster [DBG] 11.f deep-scrub ok 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: audit 2024-04-26T16:40:44.378361+0000 mon.a (mon.0) 4036 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[29234]: audit 2024-04-26T16:40:44.378361+0000 mon.a (mon.0) 4036 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:45.068 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.682691+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.682691+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.683081+0000 osd.6 (osd.6) 114 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.683081+0000 osd.6 (osd.6) 114 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.806899+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:43.806899+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.024933+0000 osd.7 (osd.7) 564 : cluster [DBG] 11.f deep-scrub starts 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.024933+0000 osd.7 (osd.7) 564 : cluster [DBG] 11.f deep-scrub starts 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.032580+0000 osd.7 (osd.7) 565 : cluster [DBG] 11.f deep-scrub ok 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.032580+0000 osd.7 (osd.7) 565 : cluster [DBG] 11.f deep-scrub ok 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: audit 2024-04-26T16:40:44.378361+0000 mon.a (mon.0) 4036 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:45.069 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:44 smithi050 bash[21463]: audit 2024-04-26T16:40:44.378361+0000 mon.a (mon.0) 4036 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.682691+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.682691+0000 osd.6 (osd.6) 113 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.683081+0000 osd.6 (osd.6) 114 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:45.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.683081+0000 osd.6 (osd.6) 114 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.806899+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:43.806899+0000 mon.a (mon.0) 4035 : cluster [DBG] osdmap e911: 8 total, 8 up, 8 in 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.024933+0000 osd.7 (osd.7) 564 : cluster [DBG] 11.f deep-scrub starts 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.024933+0000 osd.7 (osd.7) 564 : cluster [DBG] 11.f deep-scrub starts 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.032580+0000 osd.7 (osd.7) 565 : cluster [DBG] 11.f deep-scrub ok 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.032580+0000 osd.7 (osd.7) 565 : cluster [DBG] 11.f deep-scrub ok 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: audit 2024-04-26T16:40:44.378361+0000 mon.a (mon.0) 4036 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:44 smithi190 bash[23868]: audit 2024-04-26T16:40:44.378361+0000 mon.a (mon.0) 4036 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-04-26T16:40:45.743 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 5 1 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.063199+0000 mgr.y (mgr.24401) 888 : cluster [DBG] pgmap v1483: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.063199+0000 mgr.y (mgr.24401) 888 : cluster [DBG] pgmap v1483: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.635507+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.635507+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.635892+0000 osd.1 (osd.1) 534 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.635892+0000 osd.1 (osd.1) 534 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: audit 2024-04-26T16:40:44.802311+0000 mon.a (mon.0) 4037 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: audit 2024-04-26T16:40:44.802311+0000 mon.a (mon.0) 4037 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.809273+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2024-04-26T16:40:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.809273+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.985842+0000 osd.5 (osd.5) 440 : cluster [DBG] 11.7 deep-scrub starts 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.985842+0000 osd.5 (osd.5) 440 : cluster [DBG] 11.7 deep-scrub starts 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.992554+0000 osd.5 (osd.5) 441 : cluster [DBG] 11.7 deep-scrub ok 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:44.992554+0000 osd.5 (osd.5) 441 : cluster [DBG] 11.7 deep-scrub ok 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.123770+0000 osd.4 (osd.4) 501 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.123770+0000 osd.4 (osd.4) 501 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.124181+0000 osd.4 (osd.4) 502 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:45 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.124181+0000 osd.4 (osd.4) 502 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.063199+0000 mgr.y (mgr.24401) 888 : cluster [DBG] pgmap v1483: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.063199+0000 mgr.y (mgr.24401) 888 : cluster [DBG] pgmap v1483: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.635507+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.635507+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.635892+0000 osd.1 (osd.1) 534 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.635892+0000 osd.1 (osd.1) 534 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: audit 2024-04-26T16:40:44.802311+0000 mon.a (mon.0) 4037 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: audit 2024-04-26T16:40:44.802311+0000 mon.a (mon.0) 4037 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.809273+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2024-04-26T16:40:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.809273+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.985842+0000 osd.5 (osd.5) 440 : cluster [DBG] 11.7 deep-scrub starts 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.985842+0000 osd.5 (osd.5) 440 : cluster [DBG] 11.7 deep-scrub starts 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.992554+0000 osd.5 (osd.5) 441 : cluster [DBG] 11.7 deep-scrub ok 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:44.992554+0000 osd.5 (osd.5) 441 : cluster [DBG] 11.7 deep-scrub ok 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.123770+0000 osd.4 (osd.4) 501 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.123770+0000 osd.4 (osd.4) 501 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.124181+0000 osd.4 (osd.4) 502 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.124181+0000 osd.4 (osd.4) 502 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.063199+0000 mgr.y (mgr.24401) 888 : cluster [DBG] pgmap v1483: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.063199+0000 mgr.y (mgr.24401) 888 : cluster [DBG] pgmap v1483: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:46.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.635507+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.635507+0000 osd.1 (osd.1) 533 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.635892+0000 osd.1 (osd.1) 534 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.635892+0000 osd.1 (osd.1) 534 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: audit 2024-04-26T16:40:44.802311+0000 mon.a (mon.0) 4037 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: audit 2024-04-26T16:40:44.802311+0000 mon.a (mon.0) 4037 : audit [INF] from='client.? 172.21.15.50:0/658112622' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.809273+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.809273+0000 mon.a (mon.0) 4038 : cluster [DBG] osdmap e912: 8 total, 8 up, 8 in 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.985842+0000 osd.5 (osd.5) 440 : cluster [DBG] 11.7 deep-scrub starts 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.985842+0000 osd.5 (osd.5) 440 : cluster [DBG] 11.7 deep-scrub starts 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.992554+0000 osd.5 (osd.5) 441 : cluster [DBG] 11.7 deep-scrub ok 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:44.992554+0000 osd.5 (osd.5) 441 : cluster [DBG] 11.7 deep-scrub ok 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.123770+0000 osd.4 (osd.4) 501 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.123770+0000 osd.4 (osd.4) 501 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:40:46.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.124181+0000 osd.4 (osd.4) 502 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:46.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:45 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.124181+0000 osd.4 (osd.4) 502 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:40:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.073157+0000 osd.3 (osd.3) 587 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.073157+0000 osd.3 (osd.3) 587 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.075480+0000 osd.3 (osd.3) 588 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.075480+0000 osd.3 (osd.3) 588 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.835254+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:45.835254+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.132305+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.e scrub starts 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.132305+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.e scrub starts 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.136292+0000 osd.4 (osd.4) 504 : cluster [DBG] 6.e scrub ok 2024-04-26T16:40:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.136292+0000 osd.4 (osd.4) 504 : cluster [DBG] 6.e scrub ok 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.073157+0000 osd.3 (osd.3) 587 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.073157+0000 osd.3 (osd.3) 587 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.075480+0000 osd.3 (osd.3) 588 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.075480+0000 osd.3 (osd.3) 588 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.835254+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:45.835254+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.132305+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.e scrub starts 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.132305+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.e scrub starts 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.136292+0000 osd.4 (osd.4) 504 : cluster [DBG] 6.e scrub ok 2024-04-26T16:40:47.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:46 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.136292+0000 osd.4 (osd.4) 504 : cluster [DBG] 6.e scrub ok 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.073157+0000 osd.3 (osd.3) 587 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.073157+0000 osd.3 (osd.3) 587 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.075480+0000 osd.3 (osd.3) 588 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.075480+0000 osd.3 (osd.3) 588 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.835254+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:45.835254+0000 mon.a (mon.0) 4039 : cluster [DBG] osdmap e913: 8 total, 8 up, 8 in 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.132305+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.e scrub starts 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.132305+0000 osd.4 (osd.4) 503 : cluster [DBG] 6.e scrub starts 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.136292+0000 osd.4 (osd.4) 504 : cluster [DBG] 6.e scrub ok 2024-04-26T16:40:47.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:46 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.136292+0000 osd.4 (osd.4) 504 : cluster [DBG] 6.e scrub ok 2024-04-26T16:40:47.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.063941+0000 mgr.y (mgr.24401) 889 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.063941+0000 mgr.y (mgr.24401) 889 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.094765+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.094765+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.097914+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.097914+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.982133+0000 osd.2 (osd.2) 493 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:40:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.982133+0000 osd.2 (osd.2) 493 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.983729+0000 osd.2 (osd.2) 494 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: cluster 2024-04-26T16:40:46.983729+0000 osd.2 (osd.2) 494 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: audit 2024-04-26T16:40:47.504438+0000 mgr.y (mgr.24401) 890 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[21463]: audit 2024-04-26T16:40:47.504438+0000 mgr.y (mgr.24401) 890 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.063941+0000 mgr.y (mgr.24401) 889 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.063941+0000 mgr.y (mgr.24401) 889 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.094765+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.094765+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.097914+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.097914+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.982133+0000 osd.2 (osd.2) 493 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:40:48.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.982133+0000 osd.2 (osd.2) 493 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:40:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.983729+0000 osd.2 (osd.2) 494 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:40:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: cluster 2024-04-26T16:40:46.983729+0000 osd.2 (osd.2) 494 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:40:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: audit 2024-04-26T16:40:47.504438+0000 mgr.y (mgr.24401) 890 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:48.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:47 smithi050 bash[29234]: audit 2024-04-26T16:40:47.504438+0000 mgr.y (mgr.24401) 890 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.063941+0000 mgr.y (mgr.24401) 889 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.063941+0000 mgr.y (mgr.24401) 889 : cluster [DBG] pgmap v1486: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.094765+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.094765+0000 osd.7 (osd.7) 566 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.097914+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.097914+0000 osd.7 (osd.7) 567 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.982133+0000 osd.2 (osd.2) 493 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.982133+0000 osd.2 (osd.2) 493 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.983729+0000 osd.2 (osd.2) 494 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:40:48.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: cluster 2024-04-26T16:40:46.983729+0000 osd.2 (osd.2) 494 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:40:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: audit 2024-04-26T16:40:47.504438+0000 mgr.y (mgr.24401) 890 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:48.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:47 smithi190 bash[23868]: audit 2024-04-26T16:40:47.504438+0000 mgr.y (mgr.24401) 890 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:49.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:48 smithi050 bash[21463]: cluster 2024-04-26T16:40:48.065871+0000 mgr.y (mgr.24401) 891 : cluster [DBG] pgmap v1487: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:48 smithi050 bash[21463]: cluster 2024-04-26T16:40:48.065871+0000 mgr.y (mgr.24401) 891 : cluster [DBG] pgmap v1487: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:49.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:48 smithi050 bash[29234]: cluster 2024-04-26T16:40:48.065871+0000 mgr.y (mgr.24401) 891 : cluster [DBG] pgmap v1487: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:49.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:48 smithi050 bash[29234]: cluster 2024-04-26T16:40:48.065871+0000 mgr.y (mgr.24401) 891 : cluster [DBG] pgmap v1487: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:49.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:48 smithi190 bash[23868]: cluster 2024-04-26T16:40:48.065871+0000 mgr.y (mgr.24401) 891 : cluster [DBG] pgmap v1487: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:49.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:48 smithi190 bash[23868]: cluster 2024-04-26T16:40:48.065871+0000 mgr.y (mgr.24401) 891 : cluster [DBG] pgmap v1487: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:48.142408+0000 osd.0 (osd.0) 561 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:48.142408+0000 osd.0 (osd.0) 561 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:48.147185+0000 osd.0 (osd.0) 562 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:48.147185+0000 osd.0 (osd.0) 562 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.032872+0000 osd.6 (osd.6) 115 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.032872+0000 osd.6 (osd.6) 115 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.035267+0000 osd.6 (osd.6) 116 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.035267+0000 osd.6 (osd.6) 116 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.088298+0000 osd.5 (osd.5) 442 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.088298+0000 osd.5 (osd.5) 442 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:40:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.093132+0000 osd.5 (osd.5) 443 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.093132+0000 osd.5 (osd.5) 443 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.094727+0000 osd.4 (osd.4) 505 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.094727+0000 osd.4 (osd.4) 505 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.100973+0000 osd.4 (osd.4) 506 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.100973+0000 osd.4 (osd.4) 506 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:48.142408+0000 osd.0 (osd.0) 561 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:48.142408+0000 osd.0 (osd.0) 561 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:48.147185+0000 osd.0 (osd.0) 562 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:48.147185+0000 osd.0 (osd.0) 562 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.032872+0000 osd.6 (osd.6) 115 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.032872+0000 osd.6 (osd.6) 115 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.035267+0000 osd.6 (osd.6) 116 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:40:50.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.035267+0000 osd.6 (osd.6) 116 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.088298+0000 osd.5 (osd.5) 442 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.088298+0000 osd.5 (osd.5) 442 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.093132+0000 osd.5 (osd.5) 443 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.093132+0000 osd.5 (osd.5) 443 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.094727+0000 osd.4 (osd.4) 505 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.094727+0000 osd.4 (osd.4) 505 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.100973+0000 osd.4 (osd.4) 506 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:40:50.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:49 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.100973+0000 osd.4 (osd.4) 506 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:48.142408+0000 osd.0 (osd.0) 561 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:48.142408+0000 osd.0 (osd.0) 561 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:48.147185+0000 osd.0 (osd.0) 562 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:48.147185+0000 osd.0 (osd.0) 562 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.032872+0000 osd.6 (osd.6) 115 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.032872+0000 osd.6 (osd.6) 115 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.035267+0000 osd.6 (osd.6) 116 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.035267+0000 osd.6 (osd.6) 116 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.088298+0000 osd.5 (osd.5) 442 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:40:50.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.088298+0000 osd.5 (osd.5) 442 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:40:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.093132+0000 osd.5 (osd.5) 443 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:40:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.093132+0000 osd.5 (osd.5) 443 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:40:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.094727+0000 osd.4 (osd.4) 505 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:40:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.094727+0000 osd.4 (osd.4) 505 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:40:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.100973+0000 osd.4 (osd.4) 506 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:40:50.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:49 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.100973+0000 osd.4 (osd.4) 506 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:40:50.556 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.050128+0000 osd.3 (osd.3) 589 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.050128+0000 osd.3 (osd.3) 589 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.052550+0000 osd.3 (osd.3) 590 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.052550+0000 osd.3 (osd.3) 590 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.960697+0000 osd.1 (osd.1) 535 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.960697+0000 osd.1 (osd.1) 535 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.964358+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:49.964358+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:50.066710+0000 mgr.y (mgr.24401) 892 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:40:51.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[29234]: cluster 2024-04-26T16:40:50.066710+0000 mgr.y (mgr.24401) 892 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.050128+0000 osd.3 (osd.3) 589 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.050128+0000 osd.3 (osd.3) 589 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.052550+0000 osd.3 (osd.3) 590 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.052550+0000 osd.3 (osd.3) 590 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.960697+0000 osd.1 (osd.1) 535 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.960697+0000 osd.1 (osd.1) 535 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.964358+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:49.964358+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:50.066710+0000 mgr.y (mgr.24401) 892 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:40:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:50 smithi050 bash[21463]: cluster 2024-04-26T16:40:50.066710+0000 mgr.y (mgr.24401) 892 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.050128+0000 osd.3 (osd.3) 589 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.050128+0000 osd.3 (osd.3) 589 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.052550+0000 osd.3 (osd.3) 590 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.052550+0000 osd.3 (osd.3) 590 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.960697+0000 osd.1 (osd.1) 535 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.960697+0000 osd.1 (osd.1) 535 : cluster [DBG] 2.6 deep-scrub starts 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.964358+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:49.964358+0000 osd.1 (osd.1) 536 : cluster [DBG] 2.6 deep-scrub ok 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:50.066710+0000 mgr.y (mgr.24401) 892 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:40:51.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:50 smithi190 bash[23868]: cluster 2024-04-26T16:40:50.066710+0000 mgr.y (mgr.24401) 892 : cluster [DBG] pgmap v1488: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:50.050773+0000 osd.3 (osd.3) 591 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:50.050773+0000 osd.3 (osd.3) 591 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:50.054655+0000 osd.3 (osd.3) 592 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:50.054655+0000 osd.3 (osd.3) 592 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.032895+0000 osd.4 (osd.4) 507 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.032895+0000 osd.4 (osd.4) 507 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.035273+0000 osd.4 (osd.4) 508 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.035273+0000 osd.4 (osd.4) 508 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:40:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:50.050773+0000 osd.3 (osd.3) 591 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:50.050773+0000 osd.3 (osd.3) 591 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:50.054655+0000 osd.3 (osd.3) 592 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:50.054655+0000 osd.3 (osd.3) 592 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.032895+0000 osd.4 (osd.4) 507 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.032895+0000 osd.4 (osd.4) 507 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.035273+0000 osd.4 (osd.4) 508 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:40:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:51 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.035273+0000 osd.4 (osd.4) 508 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:50.050773+0000 osd.3 (osd.3) 591 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:50.050773+0000 osd.3 (osd.3) 591 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:50.054655+0000 osd.3 (osd.3) 592 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:50.054655+0000 osd.3 (osd.3) 592 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.032895+0000 osd.4 (osd.4) 507 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.032895+0000 osd.4 (osd.4) 507 : cluster [DBG] 6.1e deep-scrub starts 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.035273+0000 osd.4 (osd.4) 508 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:40:52.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:51 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.035273+0000 osd.4 (osd.4) 508 : cluster [DBG] 6.1e deep-scrub ok 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.012975+0000 osd.3 (osd.3) 593 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.012975+0000 osd.3 (osd.3) 593 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.014766+0000 osd.3 (osd.3) 594 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.014766+0000 osd.3 (osd.3) 594 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.147351+0000 osd.0 (osd.0) 563 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.147351+0000 osd.0 (osd.0) 563 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.149827+0000 osd.0 (osd.0) 564 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:40:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.149827+0000 osd.0 (osd.0) 564 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.994607+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.994607+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.996249+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:51.996249+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:52.067934+0000 mgr.y (mgr.24401) 893 : cluster [DBG] pgmap v1489: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[29234]: cluster 2024-04-26T16:40:52.067934+0000 mgr.y (mgr.24401) 893 : cluster [DBG] pgmap v1489: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.012975+0000 osd.3 (osd.3) 593 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.012975+0000 osd.3 (osd.3) 593 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.014766+0000 osd.3 (osd.3) 594 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.014766+0000 osd.3 (osd.3) 594 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:40:53.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.147351+0000 osd.0 (osd.0) 563 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.147351+0000 osd.0 (osd.0) 563 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.149827+0000 osd.0 (osd.0) 564 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.149827+0000 osd.0 (osd.0) 564 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.994607+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.994607+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.996249+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:51.996249+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:52.067934+0000 mgr.y (mgr.24401) 893 : cluster [DBG] pgmap v1489: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:53.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:52 smithi050 bash[21463]: cluster 2024-04-26T16:40:52.067934+0000 mgr.y (mgr.24401) 893 : cluster [DBG] pgmap v1489: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.012975+0000 osd.3 (osd.3) 593 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.012975+0000 osd.3 (osd.3) 593 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.014766+0000 osd.3 (osd.3) 594 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.014766+0000 osd.3 (osd.3) 594 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.147351+0000 osd.0 (osd.0) 563 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.147351+0000 osd.0 (osd.0) 563 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.149827+0000 osd.0 (osd.0) 564 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:40:53.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.149827+0000 osd.0 (osd.0) 564 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:40:53.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.994607+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:40:53.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.994607+0000 osd.1 (osd.1) 537 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:40:53.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.996249+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:40:53.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:51.996249+0000 osd.1 (osd.1) 538 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:40:53.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:52.067934+0000 mgr.y (mgr.24401) 893 : cluster [DBG] pgmap v1489: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:53.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:52 smithi190 bash[23868]: cluster 2024-04-26T16:40:52.067934+0000 mgr.y (mgr.24401) 893 : cluster [DBG] pgmap v1489: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 -- 172.21.15.50:0/3487345540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6254101a10 msgr2=0x7f6254101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 --2- 172.21.15.50:0/3487345540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6254101a10 0x7f6254101e70 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f623c009990 tx=0x7f623c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 -- 172.21.15.50:0/3487345540 shutdown_connections 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 --2- 172.21.15.50:0/3487345540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62541094b0 0x7f6254109890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 --2- 172.21.15.50:0/3487345540 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f62541023b0 0x7f62541102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 --2- 172.21.15.50:0/3487345540 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6254101a10 0x7f6254101e70 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:54.153 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 -- 172.21.15.50:0/3487345540 >> 172.21.15.50:0/3487345540 conn(0x7f6254078990 msgr2=0x7f62540ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 -- 172.21.15.50:0/3487345540 shutdown_connections 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 -- 172.21.15.50:0/3487345540 wait complete. 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 librados: starting msgr at 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 librados: starting objecter 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 Processor -- start 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 -- start start 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 librados: setting wanted keys 2024-04-26T16:40:54.154 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.151+0000 7f625a34b700 1 librados: calling monclient init 2024-04-26T16:40:54.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 0x7f6254199170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:54.155 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62541023b0 0x7f62541996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62541094b0 0x7f625419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6254112af0 con 0x7f62541094b0 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6254112970 con 0x7f62541023b0 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6254112c70 con 0x7f6254101a10 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 0x7f6254199170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62588e8700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62541094b0 0x7f625419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 0x7f6254199170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:48948/0 (socket says 172.21.15.50:48948) 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 -- 172.21.15.50:0/2115940507 learned_addr learned my addr 172.21.15.50:0/2115940507 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62537fe700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62541023b0 0x7f62541996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 -- 172.21.15.50:0/2115940507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62541023b0 msgr2=0x7f62541996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62541023b0 0x7f62541996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 -- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62541094b0 msgr2=0x7f625419d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:54.156 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62541094b0 0x7f625419d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 -- 172.21.15.50:0/2115940507 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f625419e130 con 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62588e8700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62541094b0 0x7f625419d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f6253fff700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 0x7f6254199170 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f62440060b0 tx=0x7f624400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 10 client.?.objecter ms_handle_connect 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f624401a070 con 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f624400ddd0 con 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 -- 172.21.15.50:0/2115940507 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f625419e420 con 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6244014600 con 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 10 client.41852.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 -- 172.21.15.50:0/2115940507 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62541a5ca0 con 0x7f6254101a10 2024-04-26T16:40:54.157 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f625a34b700 1 librados: init done 2024-04-26T16:40:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6238005120 con 0x7f6254101a10 2024-04-26T16:40:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f624401d070 con 0x7f6254101a10 2024-04-26T16:40:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6240071f80 0x7f6240074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:40:54.158 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(913..913 src has 854..913) ==== 8707+0+0 (secure 0 0 0) 0x7f6244090f60 con 0x7f6254101a10 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 10 client.41852.objecter ms_dispatch 0x7f6254001550 osd_map(913..913 src has 854..913) 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 3 client.41852.objecter handle_osd_map got epochs [913,913] > 0 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 3 client.41852.objecter handle_osd_map decoding full epoch 913 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62537fe700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6240071f80 0x7f6240074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 20 client.41852.objecter dump_active .. 0 homeless 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62537fe700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6240071f80 0x7f6240074440 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7f623c00b730 tx=0x7f623c01a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:40:54.159 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.155+0000 7f62517fa700 10 client.41852.objecter ms_handle_connect 0x7f6240071f80 2024-04-26T16:40:54.161 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.163+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6244010070 con 0x7f6254101a10 2024-04-26T16:40:54.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:54 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.068746+0000 osd.3 (osd.3) 595 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:40:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:54 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.068746+0000 osd.3 (osd.3) 595 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:40:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:54 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.071896+0000 osd.3 (osd.3) 596 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:40:54.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:54 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.071896+0000 osd.3 (osd.3) 596 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:40:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.068746+0000 osd.3 (osd.3) 595 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:40:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.068746+0000 osd.3 (osd.3) 595 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:40:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.071896+0000 osd.3 (osd.3) 596 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:40:54.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.071896+0000 osd.3 (osd.3) 596 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:40:54.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.068746+0000 osd.3 (osd.3) 595 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:40:54.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.068746+0000 osd.3 (osd.3) 595 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:40:54.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.071896+0000 osd.3 (osd.3) 596 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:40:54.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.071896+0000 osd.3 (osd.3) 596 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:40:54.396 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:54.395+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 5, "weight": 1.0} v 0) -- 0x7f6238005410 con 0x7f6254101a10 2024-04-26T16:40:54.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:40:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:40:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:40:55.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.039+0000 7f62517fa700 1 -- 172.21.15.50:0/2115940507 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 5, "weight": 1.0}]=0 reweighted osd.5 to 1 (10000) v914) ==== 113+0+0 (secure 0 0 0) 0x7f6244060860 con 0x7f6254101a10 2024-04-26T16:40:55.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 10 librados: watch_flush enter 2024-04-26T16:40:55.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 10 librados: watch_flush exit 2024-04-26T16:40:55.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 20 client.41852.objecter shutdown clearing up homeless session... 2024-04-26T16:40:55.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 10 client.41852.objecter successfully canceled tick 2024-04-26T16:40:55.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6240071f80 msgr2=0x7f6240074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:55.044 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6240071f80 0x7f6240074440 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7f623c00b730 tx=0x7f623c01a040 comp rx=0 tx=0).stop 2024-04-26T16:40:55.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 msgr2=0x7f6254199170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:40:55.045 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 0x7f6254199170 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f62440060b0 tx=0x7f624400d3b0 comp rx=0 tx=0).stop 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 shutdown_connections 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6240071f80 0x7f6240074440 unknown :-1 s=CLOSED pgs=3866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f62541094b0 0x7f625419d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6254101a10 0x7f6254199170 secure :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f62440060b0 tx=0x7f624400d3b0 comp rx=0 tx=0).stop 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 --2- 172.21.15.50:0/2115940507 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62541023b0 0x7f62541996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.043+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 >> 172.21.15.50:0/2115940507 conn(0x7f6254078990 msgr2=0x7f6254106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:40:55.046 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.047+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 shutdown_connections 2024-04-26T16:40:55.049 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.047+0000 7f624affd700 1 -- 172.21.15.50:0/2115940507 wait complete. 2024-04-26T16:40:55.050 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:40:55.051+0000 7f624affd700 1 librados: shutdown 2024-04-26T16:40:55.050 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.5 to 1 (10000) 2024-04-26T16:40:55.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.124712+0000 osd.7 (osd.7) 568 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.124712+0000 osd.7 (osd.7) 568 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.129841+0000 osd.7 (osd.7) 569 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.129841+0000 osd.7 (osd.7) 569 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.917953+0000 osd.1 (osd.1) 539 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.917953+0000 osd.1 (osd.1) 539 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.922826+0000 osd.1 (osd.1) 540 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:53.922826+0000 osd.1 (osd.1) 540 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.055935+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.055935+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.060612+0000 osd.4 (osd.4) 510 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.060612+0000 osd.4 (osd.4) 510 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.068472+0000 mgr.y (mgr.24401) 894 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.068472+0000 mgr.y (mgr.24401) 894 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.107318+0000 osd.2 (osd.2) 495 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.107318+0000 osd.2 (osd.2) 495 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:40:55.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.109079+0000 osd.2 (osd.2) 496 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.109079+0000 osd.2 (osd.2) 496 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.187326+0000 osd.0 (osd.0) 565 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.187326+0000 osd.0 (osd.0) 565 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.188961+0000 osd.0 (osd.0) 566 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.188961+0000 osd.0 (osd.0) 566 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: audit 2024-04-26T16:40:54.402544+0000 mon.c (mon.2) 938 : audit [INF] from='client.? 172.21.15.50:0/2115940507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: audit 2024-04-26T16:40:54.402544+0000 mon.c (mon.2) 938 : audit [INF] from='client.? 172.21.15.50:0/2115940507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: audit 2024-04-26T16:40:54.402938+0000 mon.a (mon.0) 4040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:55 smithi190 bash[23868]: audit 2024-04-26T16:40:54.402938+0000 mon.a (mon.0) 4040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.124712+0000 osd.7 (osd.7) 568 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:40:55.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.124712+0000 osd.7 (osd.7) 568 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:40:55.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.129841+0000 osd.7 (osd.7) 569 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.129841+0000 osd.7 (osd.7) 569 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.917953+0000 osd.1 (osd.1) 539 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.917953+0000 osd.1 (osd.1) 539 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.922826+0000 osd.1 (osd.1) 540 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:53.922826+0000 osd.1 (osd.1) 540 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.055935+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.055935+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.060612+0000 osd.4 (osd.4) 510 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.060612+0000 osd.4 (osd.4) 510 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.068472+0000 mgr.y (mgr.24401) 894 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.068472+0000 mgr.y (mgr.24401) 894 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.107318+0000 osd.2 (osd.2) 495 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.107318+0000 osd.2 (osd.2) 495 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.109079+0000 osd.2 (osd.2) 496 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.109079+0000 osd.2 (osd.2) 496 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:40:55.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.187326+0000 osd.0 (osd.0) 565 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.187326+0000 osd.0 (osd.0) 565 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.188961+0000 osd.0 (osd.0) 566 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.188961+0000 osd.0 (osd.0) 566 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: audit 2024-04-26T16:40:54.402544+0000 mon.c (mon.2) 938 : audit [INF] from='client.? 172.21.15.50:0/2115940507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: audit 2024-04-26T16:40:54.402544+0000 mon.c (mon.2) 938 : audit [INF] from='client.? 172.21.15.50:0/2115940507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: audit 2024-04-26T16:40:54.402938+0000 mon.a (mon.0) 4040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[21463]: audit 2024-04-26T16:40:54.402938+0000 mon.a (mon.0) 4040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.124712+0000 osd.7 (osd.7) 568 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.124712+0000 osd.7 (osd.7) 568 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.129841+0000 osd.7 (osd.7) 569 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.129841+0000 osd.7 (osd.7) 569 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:40:55.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.917953+0000 osd.1 (osd.1) 539 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.917953+0000 osd.1 (osd.1) 539 : cluster [DBG] 3.0 deep-scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.922826+0000 osd.1 (osd.1) 540 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:53.922826+0000 osd.1 (osd.1) 540 : cluster [DBG] 3.0 deep-scrub ok 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.055935+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.055935+0000 osd.4 (osd.4) 509 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.060612+0000 osd.4 (osd.4) 510 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.060612+0000 osd.4 (osd.4) 510 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.068472+0000 mgr.y (mgr.24401) 894 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.068472+0000 mgr.y (mgr.24401) 894 : cluster [DBG] pgmap v1490: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.107318+0000 osd.2 (osd.2) 495 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.107318+0000 osd.2 (osd.2) 495 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.109079+0000 osd.2 (osd.2) 496 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.109079+0000 osd.2 (osd.2) 496 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.187326+0000 osd.0 (osd.0) 565 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.187326+0000 osd.0 (osd.0) 565 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:40:55.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.188961+0000 osd.0 (osd.0) 566 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:40:55.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.188961+0000 osd.0 (osd.0) 566 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:40:55.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: audit 2024-04-26T16:40:54.402544+0000 mon.c (mon.2) 938 : audit [INF] from='client.? 172.21.15.50:0/2115940507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: audit 2024-04-26T16:40:54.402544+0000 mon.c (mon.2) 938 : audit [INF] from='client.? 172.21.15.50:0/2115940507' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: audit 2024-04-26T16:40:54.402938+0000 mon.a (mon.0) 4040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:55.384 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:55 smithi050 bash[29234]: audit 2024-04-26T16:40:54.402938+0000 mon.a (mon.0) 4040 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:40:56.125 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 5 1 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.134200+0000 osd.7 (osd.7) 570 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.134200+0000 osd.7 (osd.7) 570 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.136345+0000 osd.7 (osd.7) 571 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.136345+0000 osd.7 (osd.7) 571 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.992279+0000 osd.5 (osd.5) 444 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.992279+0000 osd.5 (osd.5) 444 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.995464+0000 osd.5 (osd.5) 445 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:54.995464+0000 osd.5 (osd.5) 445 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: audit 2024-04-26T16:40:55.037690+0000 mon.a (mon.0) 4041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: audit 2024-04-26T16:40:55.037690+0000 mon.a (mon.0) 4041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:40:56.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.045309+0000 mon.a (mon.0) 4042 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2024-04-26T16:40:56.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:56 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.045309+0000 mon.a (mon.0) 4042 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2024-04-26T16:40:56.379 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.134200+0000 osd.7 (osd.7) 570 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.134200+0000 osd.7 (osd.7) 570 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.136345+0000 osd.7 (osd.7) 571 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.136345+0000 osd.7 (osd.7) 571 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.992279+0000 osd.5 (osd.5) 444 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.992279+0000 osd.5 (osd.5) 444 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.995464+0000 osd.5 (osd.5) 445 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:54.995464+0000 osd.5 (osd.5) 445 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: audit 2024-04-26T16:40:55.037690+0000 mon.a (mon.0) 4041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: audit 2024-04-26T16:40:55.037690+0000 mon.a (mon.0) 4041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.045309+0000 mon.a (mon.0) 4042 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2024-04-26T16:40:56.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.045309+0000 mon.a (mon.0) 4042 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.134200+0000 osd.7 (osd.7) 570 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.134200+0000 osd.7 (osd.7) 570 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.136345+0000 osd.7 (osd.7) 571 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.136345+0000 osd.7 (osd.7) 571 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.992279+0000 osd.5 (osd.5) 444 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.992279+0000 osd.5 (osd.5) 444 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.995464+0000 osd.5 (osd.5) 445 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:54.995464+0000 osd.5 (osd.5) 445 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: audit 2024-04-26T16:40:55.037690+0000 mon.a (mon.0) 4041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: audit 2024-04-26T16:40:55.037690+0000 mon.a (mon.0) 4041 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.045309+0000 mon.a (mon.0) 4042 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2024-04-26T16:40:56.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:56 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.045309+0000 mon.a (mon.0) 4042 : cluster [DBG] osdmap e914: 8 total, 8 up, 8 in 2024-04-26T16:40:57.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.095068+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:40:57.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.095068+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.098829+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.098829+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.099754+0000 osd.6 (osd.6) 117 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.099754+0000 osd.6 (osd.6) 117 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.104896+0000 osd.6 (osd.6) 118 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:55.104896+0000 osd.6 (osd.6) 118 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.025165+0000 osd.5 (osd.5) 446 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.025165+0000 osd.5 (osd.5) 446 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.030994+0000 osd.5 (osd.5) 447 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.030994+0000 osd.5 (osd.5) 447 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.061314+0000 osd.4 (osd.4) 511 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.061314+0000 osd.4 (osd.4) 511 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.063716+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.063716+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:40:57.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.069194+0000 mgr.y (mgr.24401) 895 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:57.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.069194+0000 mgr.y (mgr.24401) 895 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.095068+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.095068+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.098829+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.098829+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.099754+0000 osd.6 (osd.6) 117 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.099754+0000 osd.6 (osd.6) 117 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.104896+0000 osd.6 (osd.6) 118 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:55.104896+0000 osd.6 (osd.6) 118 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.025165+0000 osd.5 (osd.5) 446 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.025165+0000 osd.5 (osd.5) 446 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.030994+0000 osd.5 (osd.5) 447 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:40:57.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.030994+0000 osd.5 (osd.5) 447 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.061314+0000 osd.4 (osd.4) 511 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.061314+0000 osd.4 (osd.4) 511 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.063716+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.063716+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.069194+0000 mgr.y (mgr.24401) 895 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.069194+0000 mgr.y (mgr.24401) 895 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.095068+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.095068+0000 osd.7 (osd.7) 572 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.098829+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.098829+0000 osd.7 (osd.7) 573 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:40:57.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.099754+0000 osd.6 (osd.6) 117 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.099754+0000 osd.6 (osd.6) 117 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.104896+0000 osd.6 (osd.6) 118 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:55.104896+0000 osd.6 (osd.6) 118 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.025165+0000 osd.5 (osd.5) 446 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.025165+0000 osd.5 (osd.5) 446 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.030994+0000 osd.5 (osd.5) 447 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.030994+0000 osd.5 (osd.5) 447 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.061314+0000 osd.4 (osd.4) 511 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.061314+0000 osd.4 (osd.4) 511 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.063716+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.063716+0000 osd.4 (osd.4) 512 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.069194+0000 mgr.y (mgr.24401) 895 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:57.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:57 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.069194+0000 mgr.y (mgr.24401) 895 : cluster [DBG] pgmap v1492: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:57.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:40:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.122670+0000 osd.6 (osd.6) 119 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.122670+0000 osd.6 (osd.6) 119 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.127503+0000 osd.6 (osd.6) 120 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:56.127503+0000 osd.6 (osd.6) 120 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.020498+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.020498+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.022864+0000 osd.4 (osd.4) 514 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:40:58.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.022864+0000 osd.4 (osd.4) 514 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:40:58.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: audit 2024-04-26T16:40:57.512982+0000 mgr.y (mgr.24401) 896 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:58.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: audit 2024-04-26T16:40:57.512982+0000 mgr.y (mgr.24401) 896 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:58.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.818794+0000 osd.1 (osd.1) 541 : cluster [DBG] 11.c scrub starts 2024-04-26T16:40:58.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:58 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.818794+0000 osd.1 (osd.1) 541 : cluster [DBG] 11.c scrub starts 2024-04-26T16:40:58.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.122670+0000 osd.6 (osd.6) 119 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:40:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.122670+0000 osd.6 (osd.6) 119 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:40:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.127503+0000 osd.6 (osd.6) 120 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:40:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:56.127503+0000 osd.6 (osd.6) 120 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:40:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.020498+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:40:58.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.020498+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.022864+0000 osd.4 (osd.4) 514 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.022864+0000 osd.4 (osd.4) 514 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: audit 2024-04-26T16:40:57.512982+0000 mgr.y (mgr.24401) 896 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: audit 2024-04-26T16:40:57.512982+0000 mgr.y (mgr.24401) 896 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.818794+0000 osd.1 (osd.1) 541 : cluster [DBG] 11.c scrub starts 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.818794+0000 osd.1 (osd.1) 541 : cluster [DBG] 11.c scrub starts 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.122670+0000 osd.6 (osd.6) 119 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:40:58.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.122670+0000 osd.6 (osd.6) 119 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.127503+0000 osd.6 (osd.6) 120 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:56.127503+0000 osd.6 (osd.6) 120 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.020498+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.020498+0000 osd.4 (osd.4) 513 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.022864+0000 osd.4 (osd.4) 514 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.022864+0000 osd.4 (osd.4) 514 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: audit 2024-04-26T16:40:57.512982+0000 mgr.y (mgr.24401) 896 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: audit 2024-04-26T16:40:57.512982+0000 mgr.y (mgr.24401) 896 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.818794+0000 osd.1 (osd.1) 541 : cluster [DBG] 11.c scrub starts 2024-04-26T16:40:58.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:58 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.818794+0000 osd.1 (osd.1) 541 : cluster [DBG] 11.c scrub starts 2024-04-26T16:40:59.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:59 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.822006+0000 osd.1 (osd.1) 542 : cluster [DBG] 11.c scrub ok 2024-04-26T16:40:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:59 smithi190 bash[23868]: cluster 2024-04-26T16:40:57.822006+0000 osd.1 (osd.1) 542 : cluster [DBG] 11.c scrub ok 2024-04-26T16:40:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:59 smithi190 bash[23868]: cluster 2024-04-26T16:40:58.071353+0000 mgr.y (mgr.24401) 897 : cluster [DBG] pgmap v1493: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:59 smithi190 bash[23868]: cluster 2024-04-26T16:40:58.071353+0000 mgr.y (mgr.24401) 897 : cluster [DBG] pgmap v1493: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:59 smithi190 bash[23868]: audit 2024-04-26T16:40:58.445732+0000 mon.c (mon.2) 939 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:59.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:40:59 smithi190 bash[23868]: audit 2024-04-26T16:40:58.445732+0000 mon.c (mon.2) 939 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.822006+0000 osd.1 (osd.1) 542 : cluster [DBG] 11.c scrub ok 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[29234]: cluster 2024-04-26T16:40:57.822006+0000 osd.1 (osd.1) 542 : cluster [DBG] 11.c scrub ok 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[29234]: cluster 2024-04-26T16:40:58.071353+0000 mgr.y (mgr.24401) 897 : cluster [DBG] pgmap v1493: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[29234]: cluster 2024-04-26T16:40:58.071353+0000 mgr.y (mgr.24401) 897 : cluster [DBG] pgmap v1493: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[29234]: audit 2024-04-26T16:40:58.445732+0000 mon.c (mon.2) 939 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[29234]: audit 2024-04-26T16:40:58.445732+0000 mon.c (mon.2) 939 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.822006+0000 osd.1 (osd.1) 542 : cluster [DBG] 11.c scrub ok 2024-04-26T16:40:59.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[21463]: cluster 2024-04-26T16:40:57.822006+0000 osd.1 (osd.1) 542 : cluster [DBG] 11.c scrub ok 2024-04-26T16:40:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[21463]: cluster 2024-04-26T16:40:58.071353+0000 mgr.y (mgr.24401) 897 : cluster [DBG] pgmap v1493: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[21463]: cluster 2024-04-26T16:40:58.071353+0000 mgr.y (mgr.24401) 897 : cluster [DBG] pgmap v1493: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:40:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[21463]: audit 2024-04-26T16:40:58.445732+0000 mon.c (mon.2) 939 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:40:59.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:40:59 smithi050 bash[21463]: audit 2024-04-26T16:40:58.445732+0000 mon.c (mon.2) 939 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:00.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:58.956023+0000 osd.5 (osd.5) 448 : cluster [DBG] 2.3 deep-scrub starts 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:58.956023+0000 osd.5 (osd.5) 448 : cluster [DBG] 2.3 deep-scrub starts 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:58.958575+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.3 deep-scrub ok 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:58.958575+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.3 deep-scrub ok 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.000628+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.000628+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.004845+0000 osd.4 (osd.4) 516 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:41:00.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:00 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.004845+0000 osd.4 (osd.4) 516 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:58.956023+0000 osd.5 (osd.5) 448 : cluster [DBG] 2.3 deep-scrub starts 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:58.956023+0000 osd.5 (osd.5) 448 : cluster [DBG] 2.3 deep-scrub starts 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:58.958575+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.3 deep-scrub ok 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:58.958575+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.3 deep-scrub ok 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.000628+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.000628+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.004845+0000 osd.4 (osd.4) 516 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.004845+0000 osd.4 (osd.4) 516 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:41:00.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:58.956023+0000 osd.5 (osd.5) 448 : cluster [DBG] 2.3 deep-scrub starts 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:58.956023+0000 osd.5 (osd.5) 448 : cluster [DBG] 2.3 deep-scrub starts 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:58.958575+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.3 deep-scrub ok 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:58.958575+0000 osd.5 (osd.5) 449 : cluster [DBG] 2.3 deep-scrub ok 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.000628+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.000628+0000 osd.4 (osd.4) 515 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.004845+0000 osd.4 (osd.4) 516 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:41:00.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:00 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.004845+0000 osd.4 (osd.4) 516 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:41:00.915 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.139345+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.139345+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.142115+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.142115+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.187772+0000 osd.7 (osd.7) 574 : cluster [DBG] 3.e scrub starts 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.187772+0000 osd.7 (osd.7) 574 : cluster [DBG] 3.e scrub starts 2024-04-26T16:41:01.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.191595+0000 osd.7 (osd.7) 575 : cluster [DBG] 3.e scrub ok 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:40:59.191595+0000 osd.7 (osd.7) 575 : cluster [DBG] 3.e scrub ok 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.005732+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.005732+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.008388+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.008388+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.035955+0000 osd.3 (osd.3) 597 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.035955+0000 osd.3 (osd.3) 597 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.038855+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.038855+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.072260+0000 mgr.y (mgr.24401) 898 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:01.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.072260+0000 mgr.y (mgr.24401) 898 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.139345+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:41:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.139345+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:41:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.142115+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:41:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.142115+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:41:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.187772+0000 osd.7 (osd.7) 574 : cluster [DBG] 3.e scrub starts 2024-04-26T16:41:01.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.187772+0000 osd.7 (osd.7) 574 : cluster [DBG] 3.e scrub starts 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.191595+0000 osd.7 (osd.7) 575 : cluster [DBG] 3.e scrub ok 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:40:59.191595+0000 osd.7 (osd.7) 575 : cluster [DBG] 3.e scrub ok 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.005732+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.005732+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.008388+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.008388+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.035955+0000 osd.3 (osd.3) 597 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.035955+0000 osd.3 (osd.3) 597 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.038855+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.038855+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.072260+0000 mgr.y (mgr.24401) 898 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:01.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.072260+0000 mgr.y (mgr.24401) 898 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.139345+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.139345+0000 osd.6 (osd.6) 121 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.142115+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.142115+0000 osd.6 (osd.6) 122 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.187772+0000 osd.7 (osd.7) 574 : cluster [DBG] 3.e scrub starts 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.187772+0000 osd.7 (osd.7) 574 : cluster [DBG] 3.e scrub starts 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.191595+0000 osd.7 (osd.7) 575 : cluster [DBG] 3.e scrub ok 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:40:59.191595+0000 osd.7 (osd.7) 575 : cluster [DBG] 3.e scrub ok 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.005732+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.005732+0000 osd.5 (osd.5) 450 : cluster [DBG] 6.d deep-scrub starts 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.008388+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.008388+0000 osd.5 (osd.5) 451 : cluster [DBG] 6.d deep-scrub ok 2024-04-26T16:41:01.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.035955+0000 osd.3 (osd.3) 597 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:41:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.035955+0000 osd.3 (osd.3) 597 : cluster [DBG] 5.14 scrub starts 2024-04-26T16:41:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.038855+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:41:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.038855+0000 osd.3 (osd.3) 598 : cluster [DBG] 5.14 scrub ok 2024-04-26T16:41:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.072260+0000 mgr.y (mgr.24401) 898 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:01.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.072260+0000 mgr.y (mgr.24401) 898 : cluster [DBG] pgmap v1494: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:02.370 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.185931+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.185931+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.188374+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.188374+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.981362+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.981362+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.983763+0000 osd.4 (osd.4) 518 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:00.983763+0000 osd.4 (osd.4) 518 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.015108+0000 osd.5 (osd.5) 452 : cluster [DBG] 1.a scrub starts 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.015108+0000 osd.5 (osd.5) 452 : cluster [DBG] 1.a scrub starts 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.017583+0000 osd.5 (osd.5) 453 : cluster [DBG] 1.a scrub ok 2024-04-26T16:41:02.371 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.017583+0000 osd.5 (osd.5) 453 : cluster [DBG] 1.a scrub ok 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.185931+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.185931+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.188374+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.188374+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.981362+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.981362+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.983763+0000 osd.4 (osd.4) 518 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:00.983763+0000 osd.4 (osd.4) 518 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.015108+0000 osd.5 (osd.5) 452 : cluster [DBG] 1.a scrub starts 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.015108+0000 osd.5 (osd.5) 452 : cluster [DBG] 1.a scrub starts 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.017583+0000 osd.5 (osd.5) 453 : cluster [DBG] 1.a scrub ok 2024-04-26T16:41:02.372 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:02 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.017583+0000 osd.5 (osd.5) 453 : cluster [DBG] 1.a scrub ok 2024-04-26T16:41:02.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.185931+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:41:02.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.185931+0000 osd.6 (osd.6) 123 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:41:02.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.188374+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:41:02.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.188374+0000 osd.6 (osd.6) 124 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:41:02.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.981362+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.981362+0000 osd.4 (osd.4) 517 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.983763+0000 osd.4 (osd.4) 518 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:00.983763+0000 osd.4 (osd.4) 518 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.015108+0000 osd.5 (osd.5) 452 : cluster [DBG] 1.a scrub starts 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.015108+0000 osd.5 (osd.5) 452 : cluster [DBG] 1.a scrub starts 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.017583+0000 osd.5 (osd.5) 453 : cluster [DBG] 1.a scrub ok 2024-04-26T16:41:02.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:02 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.017583+0000 osd.5 (osd.5) 453 : cluster [DBG] 1.a scrub ok 2024-04-26T16:41:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.205624+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:41:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.205624+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:41:03.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.208048+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:01.208048+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.034703+0000 osd.3 (osd.3) 599 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.034703+0000 osd.3 (osd.3) 599 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.038037+0000 osd.3 (osd.3) 600 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.038037+0000 osd.3 (osd.3) 600 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.074046+0000 mgr.y (mgr.24401) 899 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:03.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.074046+0000 mgr.y (mgr.24401) 899 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.205624+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:41:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.205624+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:41:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.208048+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:41:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:01.208048+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:41:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.034703+0000 osd.3 (osd.3) 599 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:41:03.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.034703+0000 osd.3 (osd.3) 599 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:41:03.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.038037+0000 osd.3 (osd.3) 600 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:41:03.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.038037+0000 osd.3 (osd.3) 600 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:41:03.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.074046+0000 mgr.y (mgr.24401) 899 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:03.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:03 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.074046+0000 mgr.y (mgr.24401) 899 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.205624+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.205624+0000 osd.7 (osd.7) 576 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.208048+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:01.208048+0000 osd.7 (osd.7) 577 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.034703+0000 osd.3 (osd.3) 599 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.034703+0000 osd.3 (osd.3) 599 : cluster [DBG] 2.19 deep-scrub starts 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.038037+0000 osd.3 (osd.3) 600 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.038037+0000 osd.3 (osd.3) 600 : cluster [DBG] 2.19 deep-scrub ok 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.074046+0000 mgr.y (mgr.24401) 899 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:03.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:03 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.074046+0000 mgr.y (mgr.24401) 899 : cluster [DBG] pgmap v1495: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.251430+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.251430+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.256126+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:02.256126+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.023220+0000 osd.5 (osd.5) 454 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.023220+0000 osd.5 (osd.5) 454 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.027944+0000 osd.5 (osd.5) 455 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.027944+0000 osd.5 (osd.5) 455 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.043800+0000 osd.3 (osd.3) 601 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.043800+0000 osd.3 (osd.3) 601 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.046442+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:41:04.340 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21463]: cluster 2024-04-26T16:41:03.046442+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.251430+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.251430+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.256126+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:02.256126+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.023220+0000 osd.5 (osd.5) 454 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.023220+0000 osd.5 (osd.5) 454 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.027944+0000 osd.5 (osd.5) 455 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.027944+0000 osd.5 (osd.5) 455 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.043800+0000 osd.3 (osd.3) 601 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.043800+0000 osd.3 (osd.3) 601 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.046442+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:41:04.341 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[29234]: cluster 2024-04-26T16:41:03.046442+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 -- 172.21.15.50:0/2377572084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 msgr2=0x7fb000110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 --2- 172.21.15.50:0/2377572084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000110310 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7faff8009990 tx=0x7faff8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 -- 172.21.15.50:0/2377572084 shutdown_connections 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 --2- 172.21.15.50:0/2377572084 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000109600 0x7fb0001099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 --2- 172.21.15.50:0/2377572084 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000110310 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 --2- 172.21.15.50:0/2377572084 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000101e50 0x7fb0001022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:04.497 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 -- 172.21.15.50:0/2377572084 >> 172.21.15.50:0/2377572084 conn(0x7fb000077590 msgr2=0x7fb0000779a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:04.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 -- 172.21.15.50:0/2377572084 shutdown_connections 2024-04-26T16:41:04.498 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 -- 172.21.15.50:0/2377572084 wait complete. 2024-04-26T16:41:04.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 librados: starting msgr at 2024-04-26T16:41:04.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.495+0000 7fb00626f700 1 librados: starting objecter 2024-04-26T16:41:04.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 Processor -- start 2024-04-26T16:41:04.499 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- start start 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 librados: setting wanted keys 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 librados: calling monclient init 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000101e50 0x7fb000199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000109600 0x7fb00019da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fb000112b00 con 0x7fb000109600 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fb000112980 con 0x7fb000101e50 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fb000112c80 con 0x7fb000102810 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fafff7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000109600 0x7fb00019da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fafff7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:33886/0 (socket says 172.21.15.50:33886) 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fafff7fe700 1 -- 172.21.15.50:0/560660867 learned_addr learned my addr 172.21.15.50:0/560660867 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:41:04.500 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffffff700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000101e50 0x7fb000199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 -- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 msgr2=0x7fb000199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 -- 172.21.15.50:0/560660867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000101e50 msgr2=0x7fb000199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000101e50 0x7fb000199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 -- 172.21.15.50:0/560660867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00019e150 con 0x7fb000109600 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffffff700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000101e50 0x7fb000199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00480c700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000109600 0x7fb00019da10 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7faff8006010 tx=0x7faff80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 10 client.?.objecter ms_handle_connect 0x7fb000109600 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff800fa90 con 0x7fb000109600 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00019e3e0 con 0x7fb000109600 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7faff8004c90 con 0x7fb000109600 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8005370 con 0x7fb000109600 2024-04-26T16:41:04.501 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 10 client.31239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:41:04.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0001a5cc0 con 0x7fb000109600 2024-04-26T16:41:04.502 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 librados: init done 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00006a110 con 0x7fb000109600 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7faff8009d10 con 0x7fb000109600 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.499+0000 7faffd7fa700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fafec071f80 0x7fafec074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(914..914 src has 864..914) ==== 8707+0+0 (secure 0 0 0) 0x7faff80924c0 con 0x7fb000109600 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffd7fa700 10 client.31239.objecter ms_dispatch 0x7fb000001550 osd_map(914..914 src has 864..914) 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffd7fa700 3 client.31239.objecter handle_osd_map got epochs [914,914] > 0 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffffff700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fafec071f80 0x7fafec074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffd7fa700 3 client.31239.objecter handle_osd_map decoding full epoch 914 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffd7fa700 20 client.31239.objecter dump_active .. 0 homeless 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffffff700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fafec071f80 0x7fafec074440 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7faff0009b50 tx=0x7faff0009550 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:04.503 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.503+0000 7faffd7fa700 10 client.31239.objecter ms_handle_connect 0x7fafec071f80 2024-04-26T16:41:04.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.507+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7faff8013070 con 0x7fb000109600 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.251430+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.251430+0000 osd.6 (osd.6) 125 : cluster [DBG] 11.3 scrub starts 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.256126+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:02.256126+0000 osd.6 (osd.6) 126 : cluster [DBG] 11.3 scrub ok 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.023220+0000 osd.5 (osd.5) 454 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.023220+0000 osd.5 (osd.5) 454 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:41:04.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.027944+0000 osd.5 (osd.5) 455 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:41:04.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.027944+0000 osd.5 (osd.5) 455 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:41:04.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.043800+0000 osd.3 (osd.3) 601 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:41:04.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.043800+0000 osd.3 (osd.3) 601 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:41:04.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.046442+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:41:04.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:04 smithi190 bash[23868]: cluster 2024-04-26T16:41:03.046442+0000 osd.3 (osd.3) 602 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:41:04.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:41:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:41:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:41:04.745 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:04.743+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} v 0) -- 0x7fb0001a5e90 con 0x7fb000109600 2024-04-26T16:41:05.098 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.099+0000 7faffd7fa700 1 -- 172.21.15.50:0/560660867 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]=0 set osd.5 primary-affinity to 1 (8655362) v915) ==== 133+0+0 (secure 0 0 0) 0x7faff805e110 con 0x7fb000109600 2024-04-26T16:41:05.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.103+0000 7fb00626f700 10 librados: watch_flush enter 2024-04-26T16:41:05.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.103+0000 7fb00626f700 10 librados: watch_flush exit 2024-04-26T16:41:05.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.103+0000 7fb00626f700 20 client.31239.objecter shutdown clearing up homeless session... 2024-04-26T16:41:05.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.103+0000 7fb00626f700 10 client.31239.objecter successfully canceled tick 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fafec071f80 msgr2=0x7fafec074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fafec071f80 0x7fafec074440 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7faff0009b50 tx=0x7faff0009550 comp rx=0 tx=0).stop 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000109600 msgr2=0x7fb00019da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000109600 0x7fb00019da10 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7faff8006010 tx=0x7faff80047b0 comp rx=0 tx=0).stop 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 shutdown_connections 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fafec071f80 0x7fafec074440 unknown :-1 s=CLOSED pgs=3867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb000101e50 0x7fb000199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fb000109600 0x7fb00019da10 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 --2- 172.21.15.50:0/560660867 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fb000102810 0x7fb000199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:05.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 >> 172.21.15.50:0/560660867 conn(0x7fb000077590 msgr2=0x7fb000100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:05.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 shutdown_connections 2024-04-26T16:41:05.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 -- 172.21.15.50:0/560660867 wait complete. 2024-04-26T16:41:05.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:05.107+0000 7fb00626f700 1 librados: shutdown 2024-04-26T16:41:05.109 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-04-26T16:41:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.075009+0000 mgr.y (mgr.24401) 900 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.075009+0000 mgr.y (mgr.24401) 900 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.134459+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:41:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.134459+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:41:05.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.136986+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.136986+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: audit 2024-04-26T16:41:04.750740+0000 mon.a (mon.0) 4043 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: audit 2024-04-26T16:41:04.750740+0000 mon.a (mon.0) 4043 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.784633+0000 osd.1 (osd.1) 543 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.784633+0000 osd.1 (osd.1) 543 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.786782+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[21463]: cluster 2024-04-26T16:41:04.786782+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.075009+0000 mgr.y (mgr.24401) 900 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.075009+0000 mgr.y (mgr.24401) 900 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:05.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.134459+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:41:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.134459+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:41:05.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.136986+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.136986+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: audit 2024-04-26T16:41:04.750740+0000 mon.a (mon.0) 4043 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: audit 2024-04-26T16:41:04.750740+0000 mon.a (mon.0) 4043 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.784633+0000 osd.1 (osd.1) 543 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.784633+0000 osd.1 (osd.1) 543 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.786782+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:41:05.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:05 smithi050 bash[29234]: cluster 2024-04-26T16:41:04.786782+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.075009+0000 mgr.y (mgr.24401) 900 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.075009+0000 mgr.y (mgr.24401) 900 : cluster [DBG] pgmap v1496: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.134459+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.134459+0000 osd.7 (osd.7) 578 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.136986+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.136986+0000 osd.7 (osd.7) 579 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: audit 2024-04-26T16:41:04.750740+0000 mon.a (mon.0) 4043 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: audit 2024-04-26T16:41:04.750740+0000 mon.a (mon.0) 4043 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-04-26T16:41:05.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.784633+0000 osd.1 (osd.1) 543 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:41:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.784633+0000 osd.1 (osd.1) 543 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:41:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.786782+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:41:05.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:05 smithi190 bash[23868]: cluster 2024-04-26T16:41:04.786782+0000 osd.1 (osd.1) 544 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:41:05.995 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 7 1 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[21463]: audit 2024-04-26T16:41:05.103353+0000 mon.a (mon.0) 4044 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[21463]: audit 2024-04-26T16:41:05.103353+0000 mon.a (mon.0) 4044 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[21463]: cluster 2024-04-26T16:41:05.114944+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[21463]: cluster 2024-04-26T16:41:05.114944+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[29234]: audit 2024-04-26T16:41:05.103353+0000 mon.a (mon.0) 4044 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[29234]: audit 2024-04-26T16:41:05.103353+0000 mon.a (mon.0) 4044 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[29234]: cluster 2024-04-26T16:41:05.114944+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2024-04-26T16:41:06.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:06 smithi050 bash[29234]: cluster 2024-04-26T16:41:05.114944+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2024-04-26T16:41:06.573 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:06 smithi190 bash[23868]: audit 2024-04-26T16:41:05.103353+0000 mon.a (mon.0) 4044 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:41:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:06 smithi190 bash[23868]: audit 2024-04-26T16:41:05.103353+0000 mon.a (mon.0) 4044 : audit [INF] from='client.? 172.21.15.50:0/560660867' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-04-26T16:41:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:06 smithi190 bash[23868]: cluster 2024-04-26T16:41:05.114944+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2024-04-26T16:41:06.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:06 smithi190 bash[23868]: cluster 2024-04-26T16:41:05.114944+0000 mon.a (mon.0) 4045 : cluster [DBG] osdmap e915: 8 total, 8 up, 8 in 2024-04-26T16:41:07.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:05.906156+0000 osd.4 (osd.4) 519 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:41:07.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:05.906156+0000 osd.4 (osd.4) 519 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:41:07.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:05.908515+0000 osd.4 (osd.4) 520 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:41:07.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:05.908515+0000 osd.4 (osd.4) 520 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:41:07.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:06.041749+0000 osd.5 (osd.5) 456 : cluster [DBG] 5.f scrub starts 2024-04-26T16:41:07.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:06.041749+0000 osd.5 (osd.5) 456 : cluster [DBG] 5.f scrub starts 2024-04-26T16:41:07.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:06.044029+0000 osd.5 (osd.5) 457 : cluster [DBG] 5.f scrub ok 2024-04-26T16:41:07.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:06.044029+0000 osd.5 (osd.5) 457 : cluster [DBG] 5.f scrub ok 2024-04-26T16:41:07.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:06.075890+0000 mgr.y (mgr.24401) 901 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:07.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[29234]: cluster 2024-04-26T16:41:06.075890+0000 mgr.y (mgr.24401) 901 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:05.906156+0000 osd.4 (osd.4) 519 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:05.906156+0000 osd.4 (osd.4) 519 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:05.908515+0000 osd.4 (osd.4) 520 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:05.908515+0000 osd.4 (osd.4) 520 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:06.041749+0000 osd.5 (osd.5) 456 : cluster [DBG] 5.f scrub starts 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:06.041749+0000 osd.5 (osd.5) 456 : cluster [DBG] 5.f scrub starts 2024-04-26T16:41:07.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:06.044029+0000 osd.5 (osd.5) 457 : cluster [DBG] 5.f scrub ok 2024-04-26T16:41:07.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:06.044029+0000 osd.5 (osd.5) 457 : cluster [DBG] 5.f scrub ok 2024-04-26T16:41:07.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:06.075890+0000 mgr.y (mgr.24401) 901 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:07.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:07 smithi050 bash[21463]: cluster 2024-04-26T16:41:06.075890+0000 mgr.y (mgr.24401) 901 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:07.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:05.906156+0000 osd.4 (osd.4) 519 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:05.906156+0000 osd.4 (osd.4) 519 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:05.908515+0000 osd.4 (osd.4) 520 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:05.908515+0000 osd.4 (osd.4) 520 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:06.041749+0000 osd.5 (osd.5) 456 : cluster [DBG] 5.f scrub starts 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:06.041749+0000 osd.5 (osd.5) 456 : cluster [DBG] 5.f scrub starts 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:06.044029+0000 osd.5 (osd.5) 457 : cluster [DBG] 5.f scrub ok 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:06.044029+0000 osd.5 (osd.5) 457 : cluster [DBG] 5.f scrub ok 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:06.075890+0000 mgr.y (mgr.24401) 901 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:07.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[23868]: cluster 2024-04-26T16:41:06.075890+0000 mgr.y (mgr.24401) 901 : cluster [DBG] pgmap v1498: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:07.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:41:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:41:08.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.128925+0000 osd.0 (osd.0) 567 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:41:08.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.128925+0000 osd.0 (osd.0) 567 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:41:08.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.132878+0000 osd.0 (osd.0) 568 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:41:08.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.132878+0000 osd.0 (osd.0) 568 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.181810+0000 osd.2 (osd.2) 497 : cluster [DBG] 5.d scrub starts 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.181810+0000 osd.2 (osd.2) 497 : cluster [DBG] 5.d scrub starts 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.184974+0000 osd.2 (osd.2) 498 : cluster [DBG] 5.d scrub ok 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.184974+0000 osd.2 (osd.2) 498 : cluster [DBG] 5.d scrub ok 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: audit 2024-04-26T16:41:07.521232+0000 mgr.y (mgr.24401) 902 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: audit 2024-04-26T16:41:07.521232+0000 mgr.y (mgr.24401) 902 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.757794+0000 osd.1 (osd.1) 545 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.757794+0000 osd.1 (osd.1) 545 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.763227+0000 osd.1 (osd.1) 546 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[21463]: cluster 2024-04-26T16:41:07.763227+0000 osd.1 (osd.1) 546 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:41:08.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.128925+0000 osd.0 (osd.0) 567 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.128925+0000 osd.0 (osd.0) 567 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.132878+0000 osd.0 (osd.0) 568 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.132878+0000 osd.0 (osd.0) 568 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.181810+0000 osd.2 (osd.2) 497 : cluster [DBG] 5.d scrub starts 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.181810+0000 osd.2 (osd.2) 497 : cluster [DBG] 5.d scrub starts 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.184974+0000 osd.2 (osd.2) 498 : cluster [DBG] 5.d scrub ok 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.184974+0000 osd.2 (osd.2) 498 : cluster [DBG] 5.d scrub ok 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: audit 2024-04-26T16:41:07.521232+0000 mgr.y (mgr.24401) 902 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: audit 2024-04-26T16:41:07.521232+0000 mgr.y (mgr.24401) 902 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.757794+0000 osd.1 (osd.1) 545 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.757794+0000 osd.1 (osd.1) 545 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:41:08.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.763227+0000 osd.1 (osd.1) 546 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:41:08.383 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:08 smithi050 bash[29234]: cluster 2024-04-26T16:41:07.763227+0000 osd.1 (osd.1) 546 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.128925+0000 osd.0 (osd.0) 567 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.128925+0000 osd.0 (osd.0) 567 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.132878+0000 osd.0 (osd.0) 568 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.132878+0000 osd.0 (osd.0) 568 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.181810+0000 osd.2 (osd.2) 497 : cluster [DBG] 5.d scrub starts 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.181810+0000 osd.2 (osd.2) 497 : cluster [DBG] 5.d scrub starts 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.184974+0000 osd.2 (osd.2) 498 : cluster [DBG] 5.d scrub ok 2024-04-26T16:41:08.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.184974+0000 osd.2 (osd.2) 498 : cluster [DBG] 5.d scrub ok 2024-04-26T16:41:08.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: audit 2024-04-26T16:41:07.521232+0000 mgr.y (mgr.24401) 902 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:08.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: audit 2024-04-26T16:41:07.521232+0000 mgr.y (mgr.24401) 902 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:08.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.757794+0000 osd.1 (osd.1) 545 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:41:08.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.757794+0000 osd.1 (osd.1) 545 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:41:08.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.763227+0000 osd.1 (osd.1) 546 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:41:08.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:08 smithi190 bash[23868]: cluster 2024-04-26T16:41:07.763227+0000 osd.1 (osd.1) 546 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:41:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.077799+0000 mgr.y (mgr.24401) 903 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.077799+0000 mgr.y (mgr.24401) 903 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.151280+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d scrub starts 2024-04-26T16:41:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.151280+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d scrub starts 2024-04-26T16:41:09.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.156325+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.156325+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.163812+0000 osd.0 (osd.0) 569 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.163812+0000 osd.0 (osd.0) 569 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.166104+0000 osd.0 (osd.0) 570 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.166104+0000 osd.0 (osd.0) 570 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.189890+0000 osd.2 (osd.2) 499 : cluster [DBG] 5.a scrub starts 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.189890+0000 osd.2 (osd.2) 499 : cluster [DBG] 5.a scrub starts 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.192118+0000 osd.2 (osd.2) 500 : cluster [DBG] 5.a scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.192118+0000 osd.2 (osd.2) 500 : cluster [DBG] 5.a scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.661138+0000 osd.0 (osd.0) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.661138+0000 osd.0 (osd.0) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.661437+0000 osd.0 (osd.0) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:09.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:09 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.661437+0000 osd.0 (osd.0) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:09.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.077799+0000 mgr.y (mgr.24401) 903 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:09.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.077799+0000 mgr.y (mgr.24401) 903 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:09.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.151280+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d scrub starts 2024-04-26T16:41:09.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.151280+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d scrub starts 2024-04-26T16:41:09.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.156325+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.156325+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.163812+0000 osd.0 (osd.0) 569 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.163812+0000 osd.0 (osd.0) 569 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.166104+0000 osd.0 (osd.0) 570 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.166104+0000 osd.0 (osd.0) 570 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.189890+0000 osd.2 (osd.2) 499 : cluster [DBG] 5.a scrub starts 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.189890+0000 osd.2 (osd.2) 499 : cluster [DBG] 5.a scrub starts 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.192118+0000 osd.2 (osd.2) 500 : cluster [DBG] 5.a scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.192118+0000 osd.2 (osd.2) 500 : cluster [DBG] 5.a scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.661138+0000 osd.0 (osd.0) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.661138+0000 osd.0 (osd.0) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.661437+0000 osd.0 (osd.0) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:09.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.661437+0000 osd.0 (osd.0) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.077799+0000 mgr.y (mgr.24401) 903 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.077799+0000 mgr.y (mgr.24401) 903 : cluster [DBG] pgmap v1499: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.151280+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d scrub starts 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.151280+0000 osd.7 (osd.7) 580 : cluster [DBG] 3.d scrub starts 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.156325+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d scrub ok 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.156325+0000 osd.7 (osd.7) 581 : cluster [DBG] 3.d scrub ok 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.163812+0000 osd.0 (osd.0) 569 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.163812+0000 osd.0 (osd.0) 569 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.166104+0000 osd.0 (osd.0) 570 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.166104+0000 osd.0 (osd.0) 570 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.189890+0000 osd.2 (osd.2) 499 : cluster [DBG] 5.a scrub starts 2024-04-26T16:41:09.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.189890+0000 osd.2 (osd.2) 499 : cluster [DBG] 5.a scrub starts 2024-04-26T16:41:09.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.192118+0000 osd.2 (osd.2) 500 : cluster [DBG] 5.a scrub ok 2024-04-26T16:41:09.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.192118+0000 osd.2 (osd.2) 500 : cluster [DBG] 5.a scrub ok 2024-04-26T16:41:09.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.661138+0000 osd.0 (osd.0) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:09.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.661138+0000 osd.0 (osd.0) 571 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:09.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.661437+0000 osd.0 (osd.0) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:09.633 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:09 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.661437+0000 osd.0 (osd.0) 572 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.911065+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.911065+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.915121+0000 osd.4 (osd.4) 522 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:08.915121+0000 osd.4 (osd.4) 522 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.037638+0000 osd.3 (osd.3) 603 : cluster [DBG] 11.a scrub starts 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.037638+0000 osd.3 (osd.3) 603 : cluster [DBG] 11.a scrub starts 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.040067+0000 osd.3 (osd.3) 604 : cluster [DBG] 11.a scrub ok 2024-04-26T16:41:10.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.040067+0000 osd.3 (osd.3) 604 : cluster [DBG] 11.a scrub ok 2024-04-26T16:41:10.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.132921+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2024-04-26T16:41:10.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.132921+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2024-04-26T16:41:10.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.752072+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:41:10.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.752072+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:41:10.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.754618+0000 osd.1 (osd.1) 548 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:41:10.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:10 smithi190 bash[23868]: cluster 2024-04-26T16:41:09.754618+0000 osd.1 (osd.1) 548 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.911065+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.911065+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.915121+0000 osd.4 (osd.4) 522 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:08.915121+0000 osd.4 (osd.4) 522 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.037638+0000 osd.3 (osd.3) 603 : cluster [DBG] 11.a scrub starts 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.037638+0000 osd.3 (osd.3) 603 : cluster [DBG] 11.a scrub starts 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.040067+0000 osd.3 (osd.3) 604 : cluster [DBG] 11.a scrub ok 2024-04-26T16:41:10.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.040067+0000 osd.3 (osd.3) 604 : cluster [DBG] 11.a scrub ok 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.132921+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.132921+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.752072+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.752072+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.754618+0000 osd.1 (osd.1) 548 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[21463]: cluster 2024-04-26T16:41:09.754618+0000 osd.1 (osd.1) 548 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.911065+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.911065+0000 osd.4 (osd.4) 521 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.915121+0000 osd.4 (osd.4) 522 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:08.915121+0000 osd.4 (osd.4) 522 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.037638+0000 osd.3 (osd.3) 603 : cluster [DBG] 11.a scrub starts 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.037638+0000 osd.3 (osd.3) 603 : cluster [DBG] 11.a scrub starts 2024-04-26T16:41:10.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.040067+0000 osd.3 (osd.3) 604 : cluster [DBG] 11.a scrub ok 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.040067+0000 osd.3 (osd.3) 604 : cluster [DBG] 11.a scrub ok 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.132921+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.132921+0000 mon.a (mon.0) 4046 : cluster [DBG] osdmap e916: 8 total, 8 up, 8 in 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.752072+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.752072+0000 osd.1 (osd.1) 547 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.754618+0000 osd.1 (osd.1) 548 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:41:10.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:10 smithi050 bash[29234]: cluster 2024-04-26T16:41:09.754618+0000 osd.1 (osd.1) 548 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:41:10.792 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:41:11.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.078560+0000 mgr.y (mgr.24401) 904 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:41:11.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.078560+0000 mgr.y (mgr.24401) 904 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:41:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.128361+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:41:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.128361+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:41:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.132747+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:41:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.132747+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:41:11.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.202573+0000 osd.2 (osd.2) 501 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:41:11.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.202573+0000 osd.2 (osd.2) 501 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:41:11.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.205020+0000 osd.2 (osd.2) 502 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:41:11.576 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:11 smithi190 bash[23868]: cluster 2024-04-26T16:41:10.205020+0000 osd.2 (osd.2) 502 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:41:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.078560+0000 mgr.y (mgr.24401) 904 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:41:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.078560+0000 mgr.y (mgr.24401) 904 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:41:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.128361+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:41:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.128361+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:41:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.132747+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:41:11.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.132747+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.202573+0000 osd.2 (osd.2) 501 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.202573+0000 osd.2 (osd.2) 501 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.205020+0000 osd.2 (osd.2) 502 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[21463]: cluster 2024-04-26T16:41:10.205020+0000 osd.2 (osd.2) 502 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.078560+0000 mgr.y (mgr.24401) 904 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.078560+0000 mgr.y (mgr.24401) 904 : cluster [DBG] pgmap v1501: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.128361+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.128361+0000 osd.0 (osd.0) 573 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.132747+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:41:11.631 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.132747+0000 osd.0 (osd.0) 574 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:41:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.202573+0000 osd.2 (osd.2) 501 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:41:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.202573+0000 osd.2 (osd.2) 501 : cluster [DBG] 2.1 scrub starts 2024-04-26T16:41:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.205020+0000 osd.2 (osd.2) 502 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:41:11.632 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:11 smithi050 bash[29234]: cluster 2024-04-26T16:41:10.205020+0000 osd.2 (osd.2) 502 : cluster [DBG] 2.1 scrub ok 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.035209+0000 osd.3 (osd.3) 605 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.035209+0000 osd.3 (osd.3) 605 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.037700+0000 osd.3 (osd.3) 606 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.037700+0000 osd.3 (osd.3) 606 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.069471+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.069471+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.071677+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.071677+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:41:12.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.929779+0000 osd.4 (osd.4) 523 : cluster [DBG] 2.f scrub starts 2024-04-26T16:41:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.929779+0000 osd.4 (osd.4) 523 : cluster [DBG] 2.f scrub starts 2024-04-26T16:41:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.933018+0000 osd.4 (osd.4) 524 : cluster [DBG] 2.f scrub ok 2024-04-26T16:41:12.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:12 smithi190 bash[23868]: cluster 2024-04-26T16:41:11.933018+0000 osd.4 (osd.4) 524 : cluster [DBG] 2.f scrub ok 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.035209+0000 osd.3 (osd.3) 605 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.035209+0000 osd.3 (osd.3) 605 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.037700+0000 osd.3 (osd.3) 606 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.037700+0000 osd.3 (osd.3) 606 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.069471+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.069471+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.071677+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.071677+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.929779+0000 osd.4 (osd.4) 523 : cluster [DBG] 2.f scrub starts 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.929779+0000 osd.4 (osd.4) 523 : cluster [DBG] 2.f scrub starts 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.933018+0000 osd.4 (osd.4) 524 : cluster [DBG] 2.f scrub ok 2024-04-26T16:41:12.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[21463]: cluster 2024-04-26T16:41:11.933018+0000 osd.4 (osd.4) 524 : cluster [DBG] 2.f scrub ok 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.035209+0000 osd.3 (osd.3) 605 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.035209+0000 osd.3 (osd.3) 605 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.037700+0000 osd.3 (osd.3) 606 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.037700+0000 osd.3 (osd.3) 606 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.069471+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.069471+0000 osd.5 (osd.5) 458 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.071677+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.071677+0000 osd.5 (osd.5) 459 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.929779+0000 osd.4 (osd.4) 523 : cluster [DBG] 2.f scrub starts 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.929779+0000 osd.4 (osd.4) 523 : cluster [DBG] 2.f scrub starts 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.933018+0000 osd.4 (osd.4) 524 : cluster [DBG] 2.f scrub ok 2024-04-26T16:41:12.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:12 smithi050 bash[29234]: cluster 2024-04-26T16:41:11.933018+0000 osd.4 (osd.4) 524 : cluster [DBG] 2.f scrub ok 2024-04-26T16:41:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.080462+0000 mgr.y (mgr.24401) 905 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.080462+0000 mgr.y (mgr.24401) 905 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.134464+0000 osd.0 (osd.0) 575 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:41:13.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.134464+0000 osd.0 (osd.0) 575 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.137740+0000 osd.0 (osd.0) 576 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.137740+0000 osd.0 (osd.0) 576 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.296298+0000 osd.7 (osd.7) 582 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.296298+0000 osd.7 (osd.7) 582 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.296811+0000 osd.7 (osd.7) 583 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.296811+0000 osd.7 (osd.7) 583 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.526215+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.526215+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.699494+0000 osd.1 (osd.1) 549 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.699494+0000 osd.1 (osd.1) 549 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.701740+0000 osd.1 (osd.1) 550 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.701740+0000 osd.1 (osd.1) 550 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.962842+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:41:13.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.962842+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:41:13.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.966084+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:41:13.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: cluster 2024-04-26T16:41:12.966084+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:41:13.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: audit 2024-04-26T16:41:13.479328+0000 mon.c (mon.2) 940 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:13.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:13 smithi190 bash[23868]: audit 2024-04-26T16:41:13.479328+0000 mon.c (mon.2) 940 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.080462+0000 mgr.y (mgr.24401) 905 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.080462+0000 mgr.y (mgr.24401) 905 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.134464+0000 osd.0 (osd.0) 575 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.134464+0000 osd.0 (osd.0) 575 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.137740+0000 osd.0 (osd.0) 576 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.137740+0000 osd.0 (osd.0) 576 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.296298+0000 osd.7 (osd.7) 582 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.296298+0000 osd.7 (osd.7) 582 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.296811+0000 osd.7 (osd.7) 583 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.296811+0000 osd.7 (osd.7) 583 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.526215+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2024-04-26T16:41:13.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.526215+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.699494+0000 osd.1 (osd.1) 549 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.699494+0000 osd.1 (osd.1) 549 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.701740+0000 osd.1 (osd.1) 550 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.701740+0000 osd.1 (osd.1) 550 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.962842+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.962842+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.966084+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: cluster 2024-04-26T16:41:12.966084+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: audit 2024-04-26T16:41:13.479328+0000 mon.c (mon.2) 940 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[21463]: audit 2024-04-26T16:41:13.479328+0000 mon.c (mon.2) 940 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:13.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.080462+0000 mgr.y (mgr.24401) 905 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.080462+0000 mgr.y (mgr.24401) 905 : cluster [DBG] pgmap v1502: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.134464+0000 osd.0 (osd.0) 575 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.134464+0000 osd.0 (osd.0) 575 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.137740+0000 osd.0 (osd.0) 576 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.137740+0000 osd.0 (osd.0) 576 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.296298+0000 osd.7 (osd.7) 582 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.296298+0000 osd.7 (osd.7) 582 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.296811+0000 osd.7 (osd.7) 583 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.296811+0000 osd.7 (osd.7) 583 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.526215+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.526215+0000 mon.a (mon.0) 4047 : cluster [DBG] osdmap e917: 8 total, 8 up, 8 in 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.699494+0000 osd.1 (osd.1) 549 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.699494+0000 osd.1 (osd.1) 549 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.701740+0000 osd.1 (osd.1) 550 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:41:13.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.701740+0000 osd.1 (osd.1) 550 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:41:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.962842+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:41:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.962842+0000 osd.3 (osd.3) 607 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:41:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.966084+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:41:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: cluster 2024-04-26T16:41:12.966084+0000 osd.3 (osd.3) 608 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:41:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: audit 2024-04-26T16:41:13.479328+0000 mon.c (mon.2) 940 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:13.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:13 smithi050 bash[29234]: audit 2024-04-26T16:41:13.479328+0000 mon.c (mon.2) 940 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:14.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/792375801 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8201094b0 msgr2=0x7fd820109890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- 172.21.15.50:0/792375801 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8201094b0 0x7fd820109890 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fd810009990 tx=0x7fd810009ca0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/792375801 shutdown_connections 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- 172.21.15.50:0/792375801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201023b0 0x7fd8201102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- 172.21.15.50:0/792375801 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8201094b0 0x7fd820109890 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- 172.21.15.50:0/792375801 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd820101a10 0x7fd820101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/792375801 >> 172.21.15.50:0/792375801 conn(0x7fd820078990 msgr2=0x7fd8200ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/792375801 shutdown_connections 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/792375801 wait complete. 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 librados: starting msgr at 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 librados: starting objecter 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 Processor -- start 2024-04-26T16:41:14.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- start start 2024-04-26T16:41:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 librados: setting wanted keys 2024-04-26T16:41:14.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 librados: calling monclient init 2024-04-26T16:41:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 0x7fd82019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8201023b0 0x7fd82019b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201094b0 0x7fd82019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd820112ac0 con 0x7fd8201023b0 2024-04-26T16:41:14.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd820112940 con 0x7fd8201094b0 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd820112c40 con 0x7fd820101a10 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 0x7fd82019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 0x7fd82019b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56350/0 (socket says 172.21.15.50:56350) 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 -- 172.21.15.50:0/4086536591 learned_addr learned my addr 172.21.15.50:0/4086536591 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd8177fe700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8201023b0 0x7fd82019b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd824ddf700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201094b0 0x7fd82019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 -- 172.21.15.50:0/4086536591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201094b0 msgr2=0x7fd82019fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201094b0 0x7fd82019fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 -- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8201023b0 msgr2=0x7fd82019b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8201023b0 0x7fd82019b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 -- 172.21.15.50:0/4086536591 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8201a02f0 con 0x7fd820101a10 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd8177fe700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8201023b0 0x7fd82019b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:41:14.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd824ddf700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201094b0 0x7fd82019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81ffff700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 0x7fd82019b2e0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fd8100047b0 tx=0x7fd81000bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81dffb700 10 client.?.objecter ms_handle_connect 0x7fd820101a10 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81dffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810026070 con 0x7fd820101a10 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd8100047e0 con 0x7fd820101a10 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8201a0580 con 0x7fd820101a10 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd810005330 con 0x7fd820101a10 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 10 client.41864.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8201a7e60 con 0x7fd820101a10 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.263+0000 7fd826842700 1 librados: init done 2024-04-26T16:41:14.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd820104d30 con 0x7fd820101a10 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd810029070 con 0x7fd820101a10 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd80c071f80 0x7fd80c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(917..917 src has 864..917) ==== 8707+0+0 (secure 0 0 0) 0x7fd810092190 con 0x7fd820101a10 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 10 client.41864.objecter ms_dispatch 0x7fd820001550 osd_map(917..917 src has 864..917) 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 3 client.41864.objecter handle_osd_map got epochs [917,917] > 0 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 3 client.41864.objecter handle_osd_map decoding full epoch 917 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd8177fe700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd80c071f80 0x7fd80c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:14.265 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 20 client.41864.objecter dump_active .. 0 homeless 2024-04-26T16:41:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd8177fe700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd80c071f80 0x7fd80c074440 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7fd808009680 tx=0x7fd808009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:14.266 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.267+0000 7fd81dffb700 10 client.41864.objecter ms_handle_connect 0x7fd80c071f80 2024-04-26T16:41:14.268 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.271+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd810013070 con 0x7fd820101a10 2024-04-26T16:41:14.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.511+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd reweight", "id": 7, "weight": 1.0} v 0) -- 0x7fd8201a8030 con 0x7fd820101a10 2024-04-26T16:41:14.524 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.527+0000 7fd81dffb700 1 -- 172.21.15.50:0/4086536591 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 7, "weight": 1.0}]=0 reweighted osd.7 to 1 (10000) v918) ==== 113+0+0 (secure 0 0 0) 0x7fd81005df00 con 0x7fd820101a10 2024-04-26T16:41:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 10 librados: watch_flush enter 2024-04-26T16:41:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 10 librados: watch_flush exit 2024-04-26T16:41:14.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 20 client.41864.objecter shutdown clearing up homeless session... 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 10 client.41864.objecter successfully canceled tick 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd80c071f80 msgr2=0x7fd80c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd80c071f80 0x7fd80c074440 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7fd808009680 tx=0x7fd808009260 comp rx=0 tx=0).stop 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 msgr2=0x7fd82019b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 0x7fd82019b2e0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7fd8100047b0 tx=0x7fd81000bfb0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 shutdown_connections 2024-04-26T16:41:14.528 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd80c071f80 0x7fd80c074440 unknown :-1 s=CLOSED pgs=3868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8201094b0 0x7fd82019fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd820101a10 0x7fd82019b2e0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 --2- 172.21.15.50:0/4086536591 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8201023b0 0x7fd82019b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.531+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 >> 172.21.15.50:0/4086536591 conn(0x7fd820078990 msgr2=0x7fd8201061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:14.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.535+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 shutdown_connections 2024-04-26T16:41:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.535+0000 7fd826842700 1 -- 172.21.15.50:0/4086536591 wait complete. 2024-04-26T16:41:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:14.535+0000 7fd826842700 1 librados: shutdown 2024-04-26T16:41:14.530 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.7 to 1 (10000) 2024-04-26T16:41:14.779 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:41:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:41:14.779 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.120021+0000 osd.0 (osd.0) 577 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.120021+0000 osd.0 (osd.0) 577 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.122068+0000 osd.0 (osd.0) 578 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.122068+0000 osd.0 (osd.0) 578 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.230784+0000 osd.2 (osd.2) 503 : cluster [DBG] 5.b scrub starts 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.230784+0000 osd.2 (osd.2) 503 : cluster [DBG] 5.b scrub starts 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.232320+0000 osd.2 (osd.2) 504 : cluster [DBG] 5.b scrub ok 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: cluster 2024-04-26T16:41:13.232320+0000 osd.2 (osd.2) 504 : cluster [DBG] 5.b scrub ok 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: audit 2024-04-26T16:41:14.511764+0000 mon.c (mon.2) 941 : audit [INF] from='client.? 172.21.15.50:0/4086536591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: audit 2024-04-26T16:41:14.511764+0000 mon.c (mon.2) 941 : audit [INF] from='client.? 172.21.15.50:0/4086536591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: audit 2024-04-26T16:41:14.512264+0000 mon.a (mon.0) 4048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.780 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[29234]: audit 2024-04-26T16:41:14.512264+0000 mon.a (mon.0) 4048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.120021+0000 osd.0 (osd.0) 577 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.120021+0000 osd.0 (osd.0) 577 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.122068+0000 osd.0 (osd.0) 578 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.122068+0000 osd.0 (osd.0) 578 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.230784+0000 osd.2 (osd.2) 503 : cluster [DBG] 5.b scrub starts 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.230784+0000 osd.2 (osd.2) 503 : cluster [DBG] 5.b scrub starts 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.232320+0000 osd.2 (osd.2) 504 : cluster [DBG] 5.b scrub ok 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: cluster 2024-04-26T16:41:13.232320+0000 osd.2 (osd.2) 504 : cluster [DBG] 5.b scrub ok 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: audit 2024-04-26T16:41:14.511764+0000 mon.c (mon.2) 941 : audit [INF] from='client.? 172.21.15.50:0/4086536591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: audit 2024-04-26T16:41:14.511764+0000 mon.c (mon.2) 941 : audit [INF] from='client.? 172.21.15.50:0/4086536591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: audit 2024-04-26T16:41:14.512264+0000 mon.a (mon.0) 4048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.781 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:14 smithi050 bash[21463]: audit 2024-04-26T16:41:14.512264+0000 mon.a (mon.0) 4048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.120021+0000 osd.0 (osd.0) 577 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.120021+0000 osd.0 (osd.0) 577 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.122068+0000 osd.0 (osd.0) 578 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.122068+0000 osd.0 (osd.0) 578 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.230784+0000 osd.2 (osd.2) 503 : cluster [DBG] 5.b scrub starts 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.230784+0000 osd.2 (osd.2) 503 : cluster [DBG] 5.b scrub starts 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.232320+0000 osd.2 (osd.2) 504 : cluster [DBG] 5.b scrub ok 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: cluster 2024-04-26T16:41:13.232320+0000 osd.2 (osd.2) 504 : cluster [DBG] 5.b scrub ok 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: audit 2024-04-26T16:41:14.511764+0000 mon.c (mon.2) 941 : audit [INF] from='client.? 172.21.15.50:0/4086536591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: audit 2024-04-26T16:41:14.511764+0000 mon.c (mon.2) 941 : audit [INF] from='client.? 172.21.15.50:0/4086536591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: audit 2024-04-26T16:41:14.512264+0000 mon.a (mon.0) 4048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:14.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:14 smithi190 bash[23868]: audit 2024-04-26T16:41:14.512264+0000 mon.a (mon.0) 4048 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:15.403 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 7 1 2024-04-26T16:41:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.004872+0000 osd.3 (osd.3) 609 : cluster [DBG] 3.b scrub starts 2024-04-26T16:41:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.004872+0000 osd.3 (osd.3) 609 : cluster [DBG] 3.b scrub starts 2024-04-26T16:41:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.008868+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.b scrub ok 2024-04-26T16:41:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.008868+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.b scrub ok 2024-04-26T16:41:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.081398+0000 mgr.y (mgr.24401) 906 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:15.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.081398+0000 mgr.y (mgr.24401) 906 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.261117+0000 osd.7 (osd.7) 584 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.261117+0000 osd.7 (osd.7) 584 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.264378+0000 osd.7 (osd.7) 585 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.264378+0000 osd.7 (osd.7) 585 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: audit 2024-04-26T16:41:14.520430+0000 mon.a (mon.0) 4049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: audit 2024-04-26T16:41:14.520430+0000 mon.a (mon.0) 4049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.527514+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.527514+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.724091+0000 osd.1 (osd.1) 551 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.724091+0000 osd.1 (osd.1) 551 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.727345+0000 osd.1 (osd.1) 552 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:41:15.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:15 smithi190 bash[23868]: cluster 2024-04-26T16:41:14.727345+0000 osd.1 (osd.1) 552 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:41:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.004872+0000 osd.3 (osd.3) 609 : cluster [DBG] 3.b scrub starts 2024-04-26T16:41:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.004872+0000 osd.3 (osd.3) 609 : cluster [DBG] 3.b scrub starts 2024-04-26T16:41:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.008868+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.b scrub ok 2024-04-26T16:41:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.008868+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.b scrub ok 2024-04-26T16:41:15.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.081398+0000 mgr.y (mgr.24401) 906 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.081398+0000 mgr.y (mgr.24401) 906 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.261117+0000 osd.7 (osd.7) 584 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.261117+0000 osd.7 (osd.7) 584 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.264378+0000 osd.7 (osd.7) 585 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.264378+0000 osd.7 (osd.7) 585 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: audit 2024-04-26T16:41:14.520430+0000 mon.a (mon.0) 4049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: audit 2024-04-26T16:41:14.520430+0000 mon.a (mon.0) 4049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.527514+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.527514+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.724091+0000 osd.1 (osd.1) 551 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.724091+0000 osd.1 (osd.1) 551 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.727345+0000 osd.1 (osd.1) 552 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:41:15.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[21463]: cluster 2024-04-26T16:41:14.727345+0000 osd.1 (osd.1) 552 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.004872+0000 osd.3 (osd.3) 609 : cluster [DBG] 3.b scrub starts 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.004872+0000 osd.3 (osd.3) 609 : cluster [DBG] 3.b scrub starts 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.008868+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.b scrub ok 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.008868+0000 osd.3 (osd.3) 610 : cluster [DBG] 3.b scrub ok 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.081398+0000 mgr.y (mgr.24401) 906 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.081398+0000 mgr.y (mgr.24401) 906 : cluster [DBG] pgmap v1504: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.261117+0000 osd.7 (osd.7) 584 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.261117+0000 osd.7 (osd.7) 584 : cluster [DBG] 2.5 deep-scrub starts 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.264378+0000 osd.7 (osd.7) 585 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.264378+0000 osd.7 (osd.7) 585 : cluster [DBG] 2.5 deep-scrub ok 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: audit 2024-04-26T16:41:14.520430+0000 mon.a (mon.0) 4049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: audit 2024-04-26T16:41:14.520430+0000 mon.a (mon.0) 4049 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:15.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.527514+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2024-04-26T16:41:15.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.527514+0000 mon.a (mon.0) 4050 : cluster [DBG] osdmap e918: 8 total, 8 up, 8 in 2024-04-26T16:41:15.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.724091+0000 osd.1 (osd.1) 551 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:41:15.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.724091+0000 osd.1 (osd.1) 551 : cluster [DBG] 5.19 deep-scrub starts 2024-04-26T16:41:15.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.727345+0000 osd.1 (osd.1) 552 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:41:15.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:15 smithi050 bash[29234]: cluster 2024-04-26T16:41:14.727345+0000 osd.1 (osd.1) 552 : cluster [DBG] 5.19 deep-scrub ok 2024-04-26T16:41:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.055366+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:41:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.055366+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:41:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.057971+0000 osd.5 (osd.5) 461 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:41:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.057971+0000 osd.5 (osd.5) 461 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:41:16.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.063563+0000 osd.0 (osd.0) 579 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.063563+0000 osd.0 (osd.0) 579 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.066569+0000 osd.0 (osd.0) 580 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.066569+0000 osd.0 (osd.0) 580 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.221734+0000 osd.5 (osd.5) 462 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.221734+0000 osd.5 (osd.5) 462 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.222170+0000 osd.5 (osd.5) 463 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.222170+0000 osd.5 (osd.5) 463 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.549752+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.549752+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.677592+0000 osd.1 (osd.1) 553 : cluster [DBG] 2.a scrub starts 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.677592+0000 osd.1 (osd.1) 553 : cluster [DBG] 2.a scrub starts 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.679162+0000 osd.1 (osd.1) 554 : cluster [DBG] 2.a scrub ok 2024-04-26T16:41:16.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:16 smithi190 bash[23868]: cluster 2024-04-26T16:41:15.679162+0000 osd.1 (osd.1) 554 : cluster [DBG] 2.a scrub ok 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.055366+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.055366+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.057971+0000 osd.5 (osd.5) 461 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.057971+0000 osd.5 (osd.5) 461 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.063563+0000 osd.0 (osd.0) 579 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.063563+0000 osd.0 (osd.0) 579 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.066569+0000 osd.0 (osd.0) 580 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:41:16.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.066569+0000 osd.0 (osd.0) 580 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.221734+0000 osd.5 (osd.5) 462 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.221734+0000 osd.5 (osd.5) 462 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.222170+0000 osd.5 (osd.5) 463 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.222170+0000 osd.5 (osd.5) 463 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.549752+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.549752+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.677592+0000 osd.1 (osd.1) 553 : cluster [DBG] 2.a scrub starts 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.677592+0000 osd.1 (osd.1) 553 : cluster [DBG] 2.a scrub starts 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.679162+0000 osd.1 (osd.1) 554 : cluster [DBG] 2.a scrub ok 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[21463]: cluster 2024-04-26T16:41:15.679162+0000 osd.1 (osd.1) 554 : cluster [DBG] 2.a scrub ok 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.055366+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:41:16.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.055366+0000 osd.5 (osd.5) 460 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.057971+0000 osd.5 (osd.5) 461 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.057971+0000 osd.5 (osd.5) 461 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.063563+0000 osd.0 (osd.0) 579 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.063563+0000 osd.0 (osd.0) 579 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.066569+0000 osd.0 (osd.0) 580 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.066569+0000 osd.0 (osd.0) 580 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.221734+0000 osd.5 (osd.5) 462 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.221734+0000 osd.5 (osd.5) 462 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.222170+0000 osd.5 (osd.5) 463 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.222170+0000 osd.5 (osd.5) 463 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.549752+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.549752+0000 mon.a (mon.0) 4051 : cluster [DBG] osdmap e919: 8 total, 8 up, 8 in 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.677592+0000 osd.1 (osd.1) 553 : cluster [DBG] 2.a scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.677592+0000 osd.1 (osd.1) 553 : cluster [DBG] 2.a scrub starts 2024-04-26T16:41:16.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.679162+0000 osd.1 (osd.1) 554 : cluster [DBG] 2.a scrub ok 2024-04-26T16:41:16.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:16 smithi050 bash[29234]: cluster 2024-04-26T16:41:15.679162+0000 osd.1 (osd.1) 554 : cluster [DBG] 2.a scrub ok 2024-04-26T16:41:17.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:41:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[23868]: cluster 2024-04-26T16:41:16.082226+0000 mgr.y (mgr.24401) 907 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[23868]: cluster 2024-04-26T16:41:16.082226+0000 mgr.y (mgr.24401) 907 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[23868]: cluster 2024-04-26T16:41:16.259054+0000 osd.7 (osd.7) 586 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:41:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[23868]: cluster 2024-04-26T16:41:16.259054+0000 osd.7 (osd.7) 586 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:41:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[23868]: cluster 2024-04-26T16:41:16.261420+0000 osd.7 (osd.7) 587 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:41:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:17 smithi190 bash[23868]: cluster 2024-04-26T16:41:16.261420+0000 osd.7 (osd.7) 587 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[21463]: cluster 2024-04-26T16:41:16.082226+0000 mgr.y (mgr.24401) 907 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[21463]: cluster 2024-04-26T16:41:16.082226+0000 mgr.y (mgr.24401) 907 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[21463]: cluster 2024-04-26T16:41:16.259054+0000 osd.7 (osd.7) 586 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[21463]: cluster 2024-04-26T16:41:16.259054+0000 osd.7 (osd.7) 586 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[21463]: cluster 2024-04-26T16:41:16.261420+0000 osd.7 (osd.7) 587 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[21463]: cluster 2024-04-26T16:41:16.261420+0000 osd.7 (osd.7) 587 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[29234]: cluster 2024-04-26T16:41:16.082226+0000 mgr.y (mgr.24401) 907 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[29234]: cluster 2024-04-26T16:41:16.082226+0000 mgr.y (mgr.24401) 907 : cluster [DBG] pgmap v1507: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[29234]: cluster 2024-04-26T16:41:16.259054+0000 osd.7 (osd.7) 586 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:41:17.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[29234]: cluster 2024-04-26T16:41:16.259054+0000 osd.7 (osd.7) 586 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:41:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[29234]: cluster 2024-04-26T16:41:16.261420+0000 osd.7 (osd.7) 587 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:41:17.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:17 smithi050 bash[29234]: cluster 2024-04-26T16:41:16.261420+0000 osd.7 (osd.7) 587 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:41:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:18 smithi190 bash[23868]: audit 2024-04-26T16:41:17.532880+0000 mgr.y (mgr.24401) 908 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:18 smithi190 bash[23868]: audit 2024-04-26T16:41:17.532880+0000 mgr.y (mgr.24401) 908 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:18 smithi190 bash[23868]: cluster 2024-04-26T16:41:17.721627+0000 osd.1 (osd.1) 555 : cluster [DBG] 2.d scrub starts 2024-04-26T16:41:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:18 smithi190 bash[23868]: cluster 2024-04-26T16:41:17.721627+0000 osd.1 (osd.1) 555 : cluster [DBG] 2.d scrub starts 2024-04-26T16:41:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:18 smithi190 bash[23868]: cluster 2024-04-26T16:41:17.724107+0000 osd.1 (osd.1) 556 : cluster [DBG] 2.d scrub ok 2024-04-26T16:41:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:18 smithi190 bash[23868]: cluster 2024-04-26T16:41:17.724107+0000 osd.1 (osd.1) 556 : cluster [DBG] 2.d scrub ok 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[21463]: audit 2024-04-26T16:41:17.532880+0000 mgr.y (mgr.24401) 908 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[21463]: audit 2024-04-26T16:41:17.532880+0000 mgr.y (mgr.24401) 908 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[21463]: cluster 2024-04-26T16:41:17.721627+0000 osd.1 (osd.1) 555 : cluster [DBG] 2.d scrub starts 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[21463]: cluster 2024-04-26T16:41:17.721627+0000 osd.1 (osd.1) 555 : cluster [DBG] 2.d scrub starts 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[21463]: cluster 2024-04-26T16:41:17.724107+0000 osd.1 (osd.1) 556 : cluster [DBG] 2.d scrub ok 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[21463]: cluster 2024-04-26T16:41:17.724107+0000 osd.1 (osd.1) 556 : cluster [DBG] 2.d scrub ok 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[29234]: audit 2024-04-26T16:41:17.532880+0000 mgr.y (mgr.24401) 908 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:18.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[29234]: audit 2024-04-26T16:41:17.532880+0000 mgr.y (mgr.24401) 908 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[29234]: cluster 2024-04-26T16:41:17.721627+0000 osd.1 (osd.1) 555 : cluster [DBG] 2.d scrub starts 2024-04-26T16:41:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[29234]: cluster 2024-04-26T16:41:17.721627+0000 osd.1 (osd.1) 555 : cluster [DBG] 2.d scrub starts 2024-04-26T16:41:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[29234]: cluster 2024-04-26T16:41:17.724107+0000 osd.1 (osd.1) 556 : cluster [DBG] 2.d scrub ok 2024-04-26T16:41:18.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:18 smithi050 bash[29234]: cluster 2024-04-26T16:41:17.724107+0000 osd.1 (osd.1) 556 : cluster [DBG] 2.d scrub ok 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.084220+0000 mgr.y (mgr.24401) 909 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.084220+0000 mgr.y (mgr.24401) 909 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.282842+0000 osd.7 (osd.7) 588 : cluster [DBG] 11.1d deep-scrub starts 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.282842+0000 osd.7 (osd.7) 588 : cluster [DBG] 11.1d deep-scrub starts 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.286797+0000 osd.7 (osd.7) 589 : cluster [DBG] 11.1d deep-scrub ok 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.286797+0000 osd.7 (osd.7) 589 : cluster [DBG] 11.1d deep-scrub ok 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.299459+0000 osd.2 (osd.2) 505 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.299459+0000 osd.2 (osd.2) 505 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:41:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.302037+0000 osd.2 (osd.2) 506 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:41:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:19 smithi190 bash[23868]: cluster 2024-04-26T16:41:18.302037+0000 osd.2 (osd.2) 506 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.084220+0000 mgr.y (mgr.24401) 909 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.084220+0000 mgr.y (mgr.24401) 909 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.282842+0000 osd.7 (osd.7) 588 : cluster [DBG] 11.1d deep-scrub starts 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.282842+0000 osd.7 (osd.7) 588 : cluster [DBG] 11.1d deep-scrub starts 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.286797+0000 osd.7 (osd.7) 589 : cluster [DBG] 11.1d deep-scrub ok 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.286797+0000 osd.7 (osd.7) 589 : cluster [DBG] 11.1d deep-scrub ok 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.299459+0000 osd.2 (osd.2) 505 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.299459+0000 osd.2 (osd.2) 505 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.302037+0000 osd.2 (osd.2) 506 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:41:19.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[21463]: cluster 2024-04-26T16:41:18.302037+0000 osd.2 (osd.2) 506 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.084220+0000 mgr.y (mgr.24401) 909 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.084220+0000 mgr.y (mgr.24401) 909 : cluster [DBG] pgmap v1508: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.282842+0000 osd.7 (osd.7) 588 : cluster [DBG] 11.1d deep-scrub starts 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.282842+0000 osd.7 (osd.7) 588 : cluster [DBG] 11.1d deep-scrub starts 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.286797+0000 osd.7 (osd.7) 589 : cluster [DBG] 11.1d deep-scrub ok 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.286797+0000 osd.7 (osd.7) 589 : cluster [DBG] 11.1d deep-scrub ok 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.299459+0000 osd.2 (osd.2) 505 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.299459+0000 osd.2 (osd.2) 505 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.302037+0000 osd.2 (osd.2) 506 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:41:19.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:19 smithi050 bash[29234]: cluster 2024-04-26T16:41:18.302037+0000 osd.2 (osd.2) 506 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:41:20.204 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:41:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:20 smithi190 bash[23868]: cluster 2024-04-26T16:41:19.040840+0000 osd.0 (osd.0) 581 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:41:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:20 smithi190 bash[23868]: cluster 2024-04-26T16:41:19.040840+0000 osd.0 (osd.0) 581 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:41:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:20 smithi190 bash[23868]: cluster 2024-04-26T16:41:19.043316+0000 osd.0 (osd.0) 582 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:41:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:20 smithi190 bash[23868]: cluster 2024-04-26T16:41:19.043316+0000 osd.0 (osd.0) 582 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:41:20.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[21463]: cluster 2024-04-26T16:41:19.040840+0000 osd.0 (osd.0) 581 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:41:20.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[21463]: cluster 2024-04-26T16:41:19.040840+0000 osd.0 (osd.0) 581 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:41:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[21463]: cluster 2024-04-26T16:41:19.043316+0000 osd.0 (osd.0) 582 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:41:20.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[21463]: cluster 2024-04-26T16:41:19.043316+0000 osd.0 (osd.0) 582 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:41:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[29234]: cluster 2024-04-26T16:41:19.040840+0000 osd.0 (osd.0) 581 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:41:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[29234]: cluster 2024-04-26T16:41:19.040840+0000 osd.0 (osd.0) 581 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:41:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[29234]: cluster 2024-04-26T16:41:19.043316+0000 osd.0 (osd.0) 582 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:41:20.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:20 smithi050 bash[29234]: cluster 2024-04-26T16:41:19.043316+0000 osd.0 (osd.0) 582 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.042573+0000 osd.0 (osd.0) 583 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.042573+0000 osd.0 (osd.0) 583 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.045160+0000 osd.0 (osd.0) 584 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.045160+0000 osd.0 (osd.0) 584 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.085225+0000 mgr.y (mgr.24401) 910 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.085225+0000 mgr.y (mgr.24401) 910 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-26T16:41:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.276293+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.276293+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.280560+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.280560+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.732697+0000 osd.1 (osd.1) 557 : cluster [DBG] 5.c scrub starts 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.732697+0000 osd.1 (osd.1) 557 : cluster [DBG] 5.c scrub starts 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.735580+0000 osd.1 (osd.1) 558 : cluster [DBG] 5.c scrub ok 2024-04-26T16:41:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:21 smithi190 bash[23868]: cluster 2024-04-26T16:41:20.735580+0000 osd.1 (osd.1) 558 : cluster [DBG] 5.c scrub ok 2024-04-26T16:41:21.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.042573+0000 osd.0 (osd.0) 583 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:41:21.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.042573+0000 osd.0 (osd.0) 583 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:41:21.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.045160+0000 osd.0 (osd.0) 584 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:41:21.870 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.045160+0000 osd.0 (osd.0) 584 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.085225+0000 mgr.y (mgr.24401) 910 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.085225+0000 mgr.y (mgr.24401) 910 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.276293+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.276293+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.280560+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.280560+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.732697+0000 osd.1 (osd.1) 557 : cluster [DBG] 5.c scrub starts 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.732697+0000 osd.1 (osd.1) 557 : cluster [DBG] 5.c scrub starts 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.735580+0000 osd.1 (osd.1) 558 : cluster [DBG] 5.c scrub ok 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[21463]: cluster 2024-04-26T16:41:20.735580+0000 osd.1 (osd.1) 558 : cluster [DBG] 5.c scrub ok 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.042573+0000 osd.0 (osd.0) 583 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:41:21.871 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.042573+0000 osd.0 (osd.0) 583 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.045160+0000 osd.0 (osd.0) 584 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.045160+0000 osd.0 (osd.0) 584 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.085225+0000 mgr.y (mgr.24401) 910 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.085225+0000 mgr.y (mgr.24401) 910 : cluster [DBG] pgmap v1509: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.276293+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.276293+0000 osd.7 (osd.7) 590 : cluster [DBG] 3.f deep-scrub starts 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.280560+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.280560+0000 osd.7 (osd.7) 591 : cluster [DBG] 3.f deep-scrub ok 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.732697+0000 osd.1 (osd.1) 557 : cluster [DBG] 5.c scrub starts 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.732697+0000 osd.1 (osd.1) 557 : cluster [DBG] 5.c scrub starts 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.735580+0000 osd.1 (osd.1) 558 : cluster [DBG] 5.c scrub ok 2024-04-26T16:41:21.872 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:21 smithi050 bash[29234]: cluster 2024-04-26T16:41:20.735580+0000 osd.1 (osd.1) 558 : cluster [DBG] 5.c scrub ok 2024-04-26T16:41:22.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[21463]: cluster 2024-04-26T16:41:21.016069+0000 osd.0 (osd.0) 585 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[21463]: cluster 2024-04-26T16:41:21.016069+0000 osd.0 (osd.0) 585 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[21463]: cluster 2024-04-26T16:41:21.018525+0000 osd.0 (osd.0) 586 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[21463]: cluster 2024-04-26T16:41:21.018525+0000 osd.0 (osd.0) 586 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[29234]: cluster 2024-04-26T16:41:21.016069+0000 osd.0 (osd.0) 585 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[29234]: cluster 2024-04-26T16:41:21.016069+0000 osd.0 (osd.0) 585 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[29234]: cluster 2024-04-26T16:41:21.018525+0000 osd.0 (osd.0) 586 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:41:22.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:22 smithi050 bash[29234]: cluster 2024-04-26T16:41:21.018525+0000 osd.0 (osd.0) 586 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:41:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:22 smithi190 bash[23868]: cluster 2024-04-26T16:41:21.016069+0000 osd.0 (osd.0) 585 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:41:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:22 smithi190 bash[23868]: cluster 2024-04-26T16:41:21.016069+0000 osd.0 (osd.0) 585 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:41:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:22 smithi190 bash[23868]: cluster 2024-04-26T16:41:21.018525+0000 osd.0 (osd.0) 586 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:41:23.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:22 smithi190 bash[23868]: cluster 2024-04-26T16:41:21.018525+0000 osd.0 (osd.0) 586 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:41:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 -- 172.21.15.50:0/277063193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075650 msgr2=0x7f7c94075a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 --2- 172.21.15.50:0/277063193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075650 0x7f7c94075a30 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c009990 tx=0x7f7c7c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:41:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 -- 172.21.15.50:0/277063193 shutdown_connections 2024-04-26T16:41:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 --2- 172.21.15.50:0/277063193 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c9410fe50 0x7f7c94112240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 --2- 172.21.15.50:0/277063193 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075650 0x7f7c94075a30 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:23.677 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 --2- 172.21.15.50:0/277063193 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c94075f70 0x7f7c9410f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 -- 172.21.15.50:0/277063193 >> 172.21.15.50:0/277063193 conn(0x7f7c9407a5d0 msgr2=0x7f7c9407a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 -- 172.21.15.50:0/277063193 shutdown_connections 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 -- 172.21.15.50:0/277063193 wait complete. 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 librados: starting msgr at 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 librados: starting objecter 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 Processor -- start 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 -- start start 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 librados: setting wanted keys 2024-04-26T16:41:23.678 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.678+0000 7f7c9883a700 1 librados: calling monclient init 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 0x7f7c9419b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075f70 0x7f7c9419b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c9410fe50 0x7f7c9419fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94114db0 con 0x7f7c94075f70 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f7c94114c30 con 0x7f7c9410fe50 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7c94114f30 con 0x7f7c94075650 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 0x7f7c9419b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c91d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075f70 0x7f7c9419b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 0x7f7c9419b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55816/0 (socket says 172.21.15.50:55816) 2024-04-26T16:41:23.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 -- 172.21.15.50:0/997427435 learned_addr learned my addr 172.21.15.50:0/997427435 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 -- 172.21.15.50:0/997427435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c9410fe50 msgr2=0x7f7c9419fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c9410fe50 0x7f7c9419fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 -- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075f70 msgr2=0x7f7c9419b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075f70 0x7f7c9419b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 -- 172.21.15.50:0/997427435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c941a03e0 con 0x7f7c94075650 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c91d9b700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075f70 0x7f7c9419b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9259c700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 0x7f7c9419b3d0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0057a0 tx=0x7f7c7c0057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 10 client.?.objecter ms_handle_connect 0x7f7c94075650 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c026070 con 0x7f7c94075650 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7c7c021410 con 0x7f7c94075650 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c941a0670 con 0x7f7c94075650 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0173f0 con 0x7f7c94075650 2024-04-26T16:41:23.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 10 client.41870.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:41:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c941a7f50 con 0x7f7c94075650 2024-04-26T16:41:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 librados: init done 2024-04-26T16:41:23.682 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c94076d90 con 0x7f7c94075650 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7c7c029070 con 0x7f7c94075650 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7c80071f80 0x7f7c80074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(920..920 src has 864..920) ==== 8707+0+0 (secure 0 0 0) 0x7f7c7c092470 con 0x7f7c94075650 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 10 client.41870.objecter ms_dispatch 0x7f7c94001550 osd_map(920..920 src has 864..920) 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 3 client.41870.objecter handle_osd_map got epochs [920,920] > 0 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c8b7fe700 3 client.41870.objecter handle_osd_map decoding full epoch 920 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.682+0000 7f7c91d9b700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7c80071f80 0x7f7c80074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.686+0000 7f7c8b7fe700 20 client.41870.objecter dump_active .. 0 homeless 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.686+0000 7f7c8b7fe700 10 client.41870.objecter ms_handle_connect 0x7f7c80071f80 2024-04-26T16:41:23.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.686+0000 7f7c91d9b700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7c80071f80 0x7f7c80074440 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7f7c94103140 tx=0x7f7c8400a300 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:23.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.690+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f7c7c013070 con 0x7f7c94075650 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.087119+0000 mgr.y (mgr.24401) 911 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.087119+0000 mgr.y (mgr.24401) 911 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.339677+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.339677+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.344223+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.344223+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.814006+0000 osd.1 (osd.1) 559 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.814006+0000 osd.1 (osd.1) 559 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.816578+0000 osd.1 (osd.1) 560 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:41:23.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.816578+0000 osd.1 (osd.1) 560 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.117925+0000 osd.5 (osd.5) 464 : cluster [DBG] 6.a scrub starts 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.117925+0000 osd.5 (osd.5) 464 : cluster [DBG] 6.a scrub starts 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.120399+0000 osd.5 (osd.5) 465 : cluster [DBG] 6.a scrub ok 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.120399+0000 osd.5 (osd.5) 465 : cluster [DBG] 6.a scrub ok 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.087119+0000 mgr.y (mgr.24401) 911 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.087119+0000 mgr.y (mgr.24401) 911 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.339677+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.339677+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.344223+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.344223+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.814006+0000 osd.1 (osd.1) 559 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.814006+0000 osd.1 (osd.1) 559 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:41:23.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.816578+0000 osd.1 (osd.1) 560 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:41:23.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.816578+0000 osd.1 (osd.1) 560 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:41:23.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.117925+0000 osd.5 (osd.5) 464 : cluster [DBG] 6.a scrub starts 2024-04-26T16:41:23.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.117925+0000 osd.5 (osd.5) 464 : cluster [DBG] 6.a scrub starts 2024-04-26T16:41:23.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.120399+0000 osd.5 (osd.5) 465 : cluster [DBG] 6.a scrub ok 2024-04-26T16:41:23.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:23 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.120399+0000 osd.5 (osd.5) 465 : cluster [DBG] 6.a scrub ok 2024-04-26T16:41:23.925 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:23.926+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} v 0) -- 0x7f7c9406a110 con 0x7f7c94075650 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.087119+0000 mgr.y (mgr.24401) 911 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.087119+0000 mgr.y (mgr.24401) 911 : cluster [DBG] pgmap v1510: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.339677+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.339677+0000 osd.7 (osd.7) 592 : cluster [DBG] 3.13 scrub starts 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.344223+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.344223+0000 osd.7 (osd.7) 593 : cluster [DBG] 3.13 scrub ok 2024-04-26T16:41:24.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.814006+0000 osd.1 (osd.1) 559 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.814006+0000 osd.1 (osd.1) 559 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.816578+0000 osd.1 (osd.1) 560 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.816578+0000 osd.1 (osd.1) 560 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.117925+0000 osd.5 (osd.5) 464 : cluster [DBG] 6.a scrub starts 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.117925+0000 osd.5 (osd.5) 464 : cluster [DBG] 6.a scrub starts 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.120399+0000 osd.5 (osd.5) 465 : cluster [DBG] 6.a scrub ok 2024-04-26T16:41:24.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:23 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.120399+0000 osd.5 (osd.5) 465 : cluster [DBG] 6.a scrub ok 2024-04-26T16:41:24.605 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.606+0000 7f7c8b7fe700 1 -- 172.21.15.50:0/997427435 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]=0 set osd.7 primary-affinity to 1 (8655362) v921) ==== 133+0+0 (secure 0 0 0) 0x7f7c7c061d70 con 0x7f7c94075650 2024-04-26T16:41:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.610+0000 7f7c9883a700 10 librados: watch_flush enter 2024-04-26T16:41:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.610+0000 7f7c9883a700 10 librados: watch_flush exit 2024-04-26T16:41:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.610+0000 7f7c9883a700 20 client.41870.objecter shutdown clearing up homeless session... 2024-04-26T16:41:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.610+0000 7f7c9883a700 10 client.41870.objecter successfully canceled tick 2024-04-26T16:41:24.609 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.610+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7c80071f80 msgr2=0x7f7c80074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.610+0000 7f7c9883a700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7c80071f80 0x7f7c80074440 secure :-1 s=READY pgs=3869 cs=0 l=1 rev1=1 crypto rx=0x7f7c94103140 tx=0x7f7c8400a300 comp rx=0 tx=0).stop 2024-04-26T16:41:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 msgr2=0x7f7c9419b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 0x7f7c9419b3d0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c0057a0 tx=0x7f7c7c0057d0 comp rx=0 tx=0).stop 2024-04-26T16:41:24.610 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 shutdown_connections 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7c80071f80 0x7f7c80074440 unknown :-1 s=CLOSED pgs=3869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7c9410fe50 0x7f7c9419fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f7c94075650 0x7f7c9419b3d0 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 --2- 172.21.15.50:0/997427435 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f7c94075f70 0x7f7c9419b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 >> 172.21.15.50:0/997427435 conn(0x7f7c9407a5d0 msgr2=0x7f7c94100820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 shutdown_connections 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 -- 172.21.15.50:0/997427435 wait complete. 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:24.614+0000 7f7c9883a700 1 librados: shutdown 2024-04-26T16:41:24.611 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-04-26T16:41:24.877 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:41:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.990505+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.990505+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.994755+0000 osd.3 (osd.3) 612 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:22.994755+0000 osd.3 (osd.3) 612 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.055670+0000 osd.3 (osd.3) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.055670+0000 osd.3 (osd.3) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.056118+0000 osd.3 (osd.3) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.056118+0000 osd.3 (osd.3) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.598468+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2024-04-26T16:41:24.878 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.598468+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: audit 2024-04-26T16:41:23.931255+0000 mon.c (mon.2) 942 : audit [INF] from='client.? 172.21.15.50:0/997427435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: audit 2024-04-26T16:41:23.931255+0000 mon.c (mon.2) 942 : audit [INF] from='client.? 172.21.15.50:0/997427435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: audit 2024-04-26T16:41:23.931989+0000 mon.a (mon.0) 4053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[29234]: audit 2024-04-26T16:41:23.931989+0000 mon.a (mon.0) 4053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.990505+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.990505+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.994755+0000 osd.3 (osd.3) 612 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:22.994755+0000 osd.3 (osd.3) 612 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.055670+0000 osd.3 (osd.3) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:24.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.055670+0000 osd.3 (osd.3) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.056118+0000 osd.3 (osd.3) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.056118+0000 osd.3 (osd.3) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.598468+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.598468+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: audit 2024-04-26T16:41:23.931255+0000 mon.c (mon.2) 942 : audit [INF] from='client.? 172.21.15.50:0/997427435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: audit 2024-04-26T16:41:23.931255+0000 mon.c (mon.2) 942 : audit [INF] from='client.? 172.21.15.50:0/997427435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: audit 2024-04-26T16:41:23.931989+0000 mon.a (mon.0) 4053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:24 smithi050 bash[21463]: audit 2024-04-26T16:41:23.931989+0000 mon.a (mon.0) 4053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.990505+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:41:25.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.990505+0000 osd.3 (osd.3) 611 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.994755+0000 osd.3 (osd.3) 612 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:22.994755+0000 osd.3 (osd.3) 612 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.055670+0000 osd.3 (osd.3) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.055670+0000 osd.3 (osd.3) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.056118+0000 osd.3 (osd.3) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.056118+0000 osd.3 (osd.3) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.598468+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.598468+0000 mon.a (mon.0) 4052 : cluster [DBG] osdmap e920: 8 total, 8 up, 8 in 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: audit 2024-04-26T16:41:23.931255+0000 mon.c (mon.2) 942 : audit [INF] from='client.? 172.21.15.50:0/997427435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: audit 2024-04-26T16:41:23.931255+0000 mon.c (mon.2) 942 : audit [INF] from='client.? 172.21.15.50:0/997427435' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: audit 2024-04-26T16:41:23.931989+0000 mon.a (mon.0) 4053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:25.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:24 smithi190 bash[23868]: audit 2024-04-26T16:41:23.931989+0000 mon.a (mon.0) 4053 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-04-26T16:41:25.535 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd reweight 6 1 2024-04-26T16:41:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.860191+0000 osd.1 (osd.1) 561 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:41:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.860191+0000 osd.1 (osd.1) 561 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:41:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.863668+0000 osd.1 (osd.1) 562 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:41:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:23.863668+0000 osd.1 (osd.1) 562 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:41:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.018325+0000 osd.3 (osd.3) 615 : cluster [DBG] 6.c scrub starts 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.018325+0000 osd.3 (osd.3) 615 : cluster [DBG] 6.c scrub starts 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.021193+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.c scrub ok 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.021193+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.c scrub ok 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.087871+0000 mgr.y (mgr.24401) 912 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.087871+0000 mgr.y (mgr.24401) 912 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: audit 2024-04-26T16:41:24.597717+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: audit 2024-04-26T16:41:24.597717+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.606465+0000 mon.a (mon.0) 4055 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.606465+0000 mon.a (mon.0) 4055 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.801555+0000 osd.2 (osd.2) 507 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.801555+0000 osd.2 (osd.2) 507 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.801854+0000 osd.2 (osd.2) 508 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[21463]: cluster 2024-04-26T16:41:24.801854+0000 osd.2 (osd.2) 508 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.860191+0000 osd.1 (osd.1) 561 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.860191+0000 osd.1 (osd.1) 561 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.863668+0000 osd.1 (osd.1) 562 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:23.863668+0000 osd.1 (osd.1) 562 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.018325+0000 osd.3 (osd.3) 615 : cluster [DBG] 6.c scrub starts 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.018325+0000 osd.3 (osd.3) 615 : cluster [DBG] 6.c scrub starts 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.021193+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.c scrub ok 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.021193+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.c scrub ok 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.087871+0000 mgr.y (mgr.24401) 912 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.087871+0000 mgr.y (mgr.24401) 912 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: audit 2024-04-26T16:41:24.597717+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: audit 2024-04-26T16:41:24.597717+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.606465+0000 mon.a (mon.0) 4055 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2024-04-26T16:41:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.606465+0000 mon.a (mon.0) 4055 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2024-04-26T16:41:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.801555+0000 osd.2 (osd.2) 507 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.801555+0000 osd.2 (osd.2) 507 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.801854+0000 osd.2 (osd.2) 508 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:25 smithi050 bash[29234]: cluster 2024-04-26T16:41:24.801854+0000 osd.2 (osd.2) 508 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.860191+0000 osd.1 (osd.1) 561 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:41:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.860191+0000 osd.1 (osd.1) 561 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:41:26.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.863668+0000 osd.1 (osd.1) 562 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:23.863668+0000 osd.1 (osd.1) 562 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.018325+0000 osd.3 (osd.3) 615 : cluster [DBG] 6.c scrub starts 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.018325+0000 osd.3 (osd.3) 615 : cluster [DBG] 6.c scrub starts 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.021193+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.c scrub ok 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.021193+0000 osd.3 (osd.3) 616 : cluster [DBG] 6.c scrub ok 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.087871+0000 mgr.y (mgr.24401) 912 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.087871+0000 mgr.y (mgr.24401) 912 : cluster [DBG] pgmap v1512: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: audit 2024-04-26T16:41:24.597717+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: audit 2024-04-26T16:41:24.597717+0000 mon.a (mon.0) 4054 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.606465+0000 mon.a (mon.0) 4055 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.606465+0000 mon.a (mon.0) 4055 : cluster [DBG] osdmap e921: 8 total, 8 up, 8 in 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.801555+0000 osd.2 (osd.2) 507 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.801555+0000 osd.2 (osd.2) 507 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.801854+0000 osd.2 (osd.2) 508 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:26.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:25 smithi190 bash[23868]: cluster 2024-04-26T16:41:24.801854+0000 osd.2 (osd.2) 508 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:41:26.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.297805+0000 osd.7 (osd.7) 594 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:41:26.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.297805+0000 osd.7 (osd.7) 594 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:41:26.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.300301+0000 osd.7 (osd.7) 595 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:41:26.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.300301+0000 osd.7 (osd.7) 595 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:41:26.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.630987+0000 mon.a (mon.0) 4056 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2024-04-26T16:41:26.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.630987+0000 mon.a (mon.0) 4056 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.905174+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.905174+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.908728+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:25.908728+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:26.152646+0000 osd.5 (osd.5) 466 : cluster [DBG] 3.c scrub starts 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:26.152646+0000 osd.5 (osd.5) 466 : cluster [DBG] 3.c scrub starts 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:26.156996+0000 osd.5 (osd.5) 467 : cluster [DBG] 3.c scrub ok 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[21463]: cluster 2024-04-26T16:41:26.156996+0000 osd.5 (osd.5) 467 : cluster [DBG] 3.c scrub ok 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.297805+0000 osd.7 (osd.7) 594 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:41:26.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.297805+0000 osd.7 (osd.7) 594 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.300301+0000 osd.7 (osd.7) 595 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.300301+0000 osd.7 (osd.7) 595 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.630987+0000 mon.a (mon.0) 4056 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.630987+0000 mon.a (mon.0) 4056 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.905174+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.905174+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.908728+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:25.908728+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:26.152646+0000 osd.5 (osd.5) 466 : cluster [DBG] 3.c scrub starts 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:26.152646+0000 osd.5 (osd.5) 466 : cluster [DBG] 3.c scrub starts 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:26.156996+0000 osd.5 (osd.5) 467 : cluster [DBG] 3.c scrub ok 2024-04-26T16:41:26.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:26 smithi050 bash[29234]: cluster 2024-04-26T16:41:26.156996+0000 osd.5 (osd.5) 467 : cluster [DBG] 3.c scrub ok 2024-04-26T16:41:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.297805+0000 osd.7 (osd.7) 594 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:41:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.297805+0000 osd.7 (osd.7) 594 : cluster [DBG] 1.9 scrub starts 2024-04-26T16:41:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.300301+0000 osd.7 (osd.7) 595 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:41:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.300301+0000 osd.7 (osd.7) 595 : cluster [DBG] 1.9 scrub ok 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.630987+0000 mon.a (mon.0) 4056 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.630987+0000 mon.a (mon.0) 4056 : cluster [DBG] osdmap e922: 8 total, 8 up, 8 in 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.905174+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.905174+0000 osd.0 (osd.0) 587 : cluster [DBG] 3.1b scrub starts 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.908728+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:25.908728+0000 osd.0 (osd.0) 588 : cluster [DBG] 3.1b scrub ok 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:26.152646+0000 osd.5 (osd.5) 466 : cluster [DBG] 3.c scrub starts 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:26.152646+0000 osd.5 (osd.5) 466 : cluster [DBG] 3.c scrub starts 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:26.156996+0000 osd.5 (osd.5) 467 : cluster [DBG] 3.c scrub ok 2024-04-26T16:41:27.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:26 smithi190 bash[23868]: cluster 2024-04-26T16:41:26.156996+0000 osd.5 (osd.5) 467 : cluster [DBG] 3.c scrub ok 2024-04-26T16:41:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:27 smithi190 bash[23868]: cluster 2024-04-26T16:41:26.088748+0000 mgr.y (mgr.24401) 913 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:27.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:27 smithi190 bash[23868]: cluster 2024-04-26T16:41:26.088748+0000 mgr.y (mgr.24401) 913 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:27.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:41:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:41:28.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:27 smithi050 bash[21463]: cluster 2024-04-26T16:41:26.088748+0000 mgr.y (mgr.24401) 913 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:28.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:27 smithi050 bash[21463]: cluster 2024-04-26T16:41:26.088748+0000 mgr.y (mgr.24401) 913 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:28.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:27 smithi050 bash[29234]: cluster 2024-04-26T16:41:26.088748+0000 mgr.y (mgr.24401) 913 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:28.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:27 smithi050 bash[29234]: cluster 2024-04-26T16:41:26.088748+0000 mgr.y (mgr.24401) 913 : cluster [DBG] pgmap v1515: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.014669+0000 osd.3 (osd.3) 617 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:41:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.014669+0000 osd.3 (osd.3) 617 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:41:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.024190+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:41:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.024190+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: audit 2024-04-26T16:41:27.544347+0000 mgr.y (mgr.24401) 914 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: audit 2024-04-26T16:41:27.544347+0000 mgr.y (mgr.24401) 914 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.828739+0000 osd.4 (osd.4) 525 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.828739+0000 osd.4 (osd.4) 525 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.831414+0000 osd.4 (osd.4) 526 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:27.831414+0000 osd.4 (osd.4) 526 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.145644+0000 osd.5 (osd.5) 468 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.145644+0000 osd.5 (osd.5) 468 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.148173+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.148173+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: audit 2024-04-26T16:41:28.514998+0000 mon.c (mon.2) 943 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:28 smithi190 bash[23868]: audit 2024-04-26T16:41:28.514998+0000 mon.c (mon.2) 943 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.014669+0000 osd.3 (osd.3) 617 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:41:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.014669+0000 osd.3 (osd.3) 617 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:41:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.024190+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:41:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.024190+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:41:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: audit 2024-04-26T16:41:27.544347+0000 mgr.y (mgr.24401) 914 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:29.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: audit 2024-04-26T16:41:27.544347+0000 mgr.y (mgr.24401) 914 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.828739+0000 osd.4 (osd.4) 525 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.828739+0000 osd.4 (osd.4) 525 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.831414+0000 osd.4 (osd.4) 526 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:27.831414+0000 osd.4 (osd.4) 526 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.145644+0000 osd.5 (osd.5) 468 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.145644+0000 osd.5 (osd.5) 468 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.148173+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.148173+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: audit 2024-04-26T16:41:28.514998+0000 mon.c (mon.2) 943 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[21463]: audit 2024-04-26T16:41:28.514998+0000 mon.c (mon.2) 943 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:29.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.014669+0000 osd.3 (osd.3) 617 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.014669+0000 osd.3 (osd.3) 617 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.024190+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.024190+0000 osd.3 (osd.3) 618 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: audit 2024-04-26T16:41:27.544347+0000 mgr.y (mgr.24401) 914 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: audit 2024-04-26T16:41:27.544347+0000 mgr.y (mgr.24401) 914 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.828739+0000 osd.4 (osd.4) 525 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.828739+0000 osd.4 (osd.4) 525 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.831414+0000 osd.4 (osd.4) 526 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:27.831414+0000 osd.4 (osd.4) 526 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.145644+0000 osd.5 (osd.5) 468 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.145644+0000 osd.5 (osd.5) 468 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.148173+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:41:29.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.148173+0000 osd.5 (osd.5) 469 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:41:29.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: audit 2024-04-26T16:41:28.514998+0000 mon.c (mon.2) 943 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:29.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:28 smithi050 bash[29234]: audit 2024-04-26T16:41:28.514998+0000 mon.c (mon.2) 943 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.090620+0000 mgr.y (mgr.24401) 915 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.090620+0000 mgr.y (mgr.24401) 915 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.920811+0000 osd.0 (osd.0) 589 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:41:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.920811+0000 osd.0 (osd.0) 589 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:41:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.925803+0000 osd.0 (osd.0) 590 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:41:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:28.925803+0000 osd.0 (osd.0) 590 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:41:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:29.126378+0000 osd.5 (osd.5) 470 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:41:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:29.126378+0000 osd.5 (osd.5) 470 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:41:30.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:29.129469+0000 osd.5 (osd.5) 471 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:41:30.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:29 smithi190 bash[23868]: cluster 2024-04-26T16:41:29.129469+0000 osd.5 (osd.5) 471 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.090620+0000 mgr.y (mgr.24401) 915 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.090620+0000 mgr.y (mgr.24401) 915 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.920811+0000 osd.0 (osd.0) 589 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.920811+0000 osd.0 (osd.0) 589 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.925803+0000 osd.0 (osd.0) 590 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:28.925803+0000 osd.0 (osd.0) 590 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:29.126378+0000 osd.5 (osd.5) 470 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:29.126378+0000 osd.5 (osd.5) 470 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:41:30.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:29.129469+0000 osd.5 (osd.5) 471 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[21463]: cluster 2024-04-26T16:41:29.129469+0000 osd.5 (osd.5) 471 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.090620+0000 mgr.y (mgr.24401) 915 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.090620+0000 mgr.y (mgr.24401) 915 : cluster [DBG] pgmap v1516: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.920811+0000 osd.0 (osd.0) 589 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.920811+0000 osd.0 (osd.0) 589 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.925803+0000 osd.0 (osd.0) 590 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:28.925803+0000 osd.0 (osd.0) 590 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:29.126378+0000 osd.5 (osd.5) 470 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:41:30.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:29.126378+0000 osd.5 (osd.5) 470 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:41:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:29.129469+0000 osd.5 (osd.5) 471 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:41:30.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:29 smithi050 bash[29234]: cluster 2024-04-26T16:41:29.129469+0000 osd.5 (osd.5) 471 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:41:30.338 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.091632+0000 mgr.y (mgr.24401) 916 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.091632+0000 mgr.y (mgr.24401) 916 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.167966+0000 osd.7 (osd.7) 596 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.167966+0000 osd.7 (osd.7) 596 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.170289+0000 osd.7 (osd.7) 597 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.170289+0000 osd.7 (osd.7) 597 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.059772+0000 osd.5 (osd.5) 472 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.059772+0000 osd.5 (osd.5) 472 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:41:31.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.063353+0000 osd.5 (osd.5) 473 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.063353+0000 osd.5 (osd.5) 473 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.091632+0000 mgr.y (mgr.24401) 916 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.091632+0000 mgr.y (mgr.24401) 916 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.167966+0000 osd.7 (osd.7) 596 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.167966+0000 osd.7 (osd.7) 596 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.170289+0000 osd.7 (osd.7) 597 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.170289+0000 osd.7 (osd.7) 597 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.059772+0000 osd.5 (osd.5) 472 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:41:31.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.059772+0000 osd.5 (osd.5) 472 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:41:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.063353+0000 osd.5 (osd.5) 473 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:41:31.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:31 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.063353+0000 osd.5 (osd.5) 473 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:41:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.091632+0000 mgr.y (mgr.24401) 916 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-26T16:41:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.091632+0000 mgr.y (mgr.24401) 916 : cluster [DBG] pgmap v1517: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 788 B/s rd, 0 op/s 2024-04-26T16:41:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.167966+0000 osd.7 (osd.7) 596 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:41:32.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.167966+0000 osd.7 (osd.7) 596 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:41:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.170289+0000 osd.7 (osd.7) 597 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:41:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.170289+0000 osd.7 (osd.7) 597 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:41:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.059772+0000 osd.5 (osd.5) 472 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:41:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.059772+0000 osd.5 (osd.5) 472 : cluster [DBG] 3.16 scrub starts 2024-04-26T16:41:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.063353+0000 osd.5 (osd.5) 473 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:41:32.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:31 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.063353+0000 osd.5 (osd.5) 473 : cluster [DBG] 3.16 scrub ok 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.960584+0000 osd.1 (osd.1) 563 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.960584+0000 osd.1 (osd.1) 563 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.962998+0000 osd.1 (osd.1) 564 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:30.962998+0000 osd.1 (osd.1) 564 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.139155+0000 osd.2 (osd.2) 509 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.139155+0000 osd.2 (osd.2) 509 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.147053+0000 osd.2 (osd.2) 510 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:41:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.147053+0000 osd.2 (osd.2) 510 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.960584+0000 osd.1 (osd.1) 563 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.960584+0000 osd.1 (osd.1) 563 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.962998+0000 osd.1 (osd.1) 564 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:30.962998+0000 osd.1 (osd.1) 564 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.139155+0000 osd.2 (osd.2) 509 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.139155+0000 osd.2 (osd.2) 509 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.147053+0000 osd.2 (osd.2) 510 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:41:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:32 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.147053+0000 osd.2 (osd.2) 510 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.960584+0000 osd.1 (osd.1) 563 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.960584+0000 osd.1 (osd.1) 563 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.962998+0000 osd.1 (osd.1) 564 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:30.962998+0000 osd.1 (osd.1) 564 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.139155+0000 osd.2 (osd.2) 509 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.139155+0000 osd.2 (osd.2) 509 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.147053+0000 osd.2 (osd.2) 510 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:41:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:32 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.147053+0000 osd.2 (osd.2) 510 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/2999167346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 msgr2=0x7fd8d8110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/2999167346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8110300 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009990 tx=0x7fd8d0009ca0 comp rx=0 tx=0).stop 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/2999167346 shutdown_connections 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/2999167346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8d81094c0 0x7fd8d81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/2999167346 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8110300 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/2999167346 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 0x7fd8d806bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/2999167346 >> 172.21.15.50:0/2999167346 conn(0x7fd8d80792d0 msgr2=0x7fd8d80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/2999167346 shutdown_connections 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/2999167346 wait complete. 2024-04-26T16:41:33.783 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 librados: starting msgr at 2024-04-26T16:41:33.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 librados: starting objecter 2024-04-26T16:41:33.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 Processor -- start 2024-04-26T16:41:33.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- start start 2024-04-26T16:41:33.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 librados: setting wanted keys 2024-04-26T16:41:33.784 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 librados: calling monclient init 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 0x7fd8d81027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8d81094c0 0x7fd8d8103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8112af0 con 0x7fd8d806c490 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d8112970 con 0x7fd8d81094c0 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d8112c70 con 0x7fd8d806baf0 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:53864/0 (socket says 172.21.15.50:53864) 2024-04-26T16:41:33.786 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 -- 172.21.15.50:0/576822803 learned_addr learned my addr 172.21.15.50:0/576822803 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d659c700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 0x7fd8d81027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d6d9d700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8d81094c0 0x7fd8d8103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 -- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 msgr2=0x7fd8d81027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 0x7fd8d81027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 -- 172.21.15.50:0/576822803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8d81094c0 msgr2=0x7fd8d8103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8d81094c0 0x7fd8d8103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 -- 172.21.15.50:0/576822803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d81a76c0 con 0x7fd8d806c490 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d659c700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 0x7fd8d81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8d5d9b700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8102d10 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e9c0 tx=0x7fd8cc00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8c77fe700 10 client.?.objecter ms_handle_connect 0x7fd8d806c490 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8c77fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:41:33.787 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00cbd0 con 0x7fd8d806c490 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd8cc00cd70 con 0x7fd8d806c490 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d81a79b0 con 0x7fd8d806c490 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc005570 con 0x7fd8d806c490 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 10 client.31257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d81a7ef0 con 0x7fd8d806c490 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.786+0000 7fd8dcc8e700 1 librados: init done 2024-04-26T16:41:33.788 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d8104b70 con 0x7fd8d806c490 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd8cc004750 con 0x7fd8d806c490 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd8c0071f80 0x7fd8c0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(922..922 src has 864..922) ==== 8707+0+0 (secure 0 0 0) 0x7fd8cc091b20 con 0x7fd8d806c490 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 10 client.31257.objecter ms_dispatch 0x7fd8d8001550 osd_map(922..922 src has 864..922) 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 3 client.31257.objecter handle_osd_map got epochs [922,922] > 0 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 3 client.31257.objecter handle_osd_map decoding full epoch 922 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8d659c700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd8c0071f80 0x7fd8c0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 20 client.31257.objecter dump_active .. 0 homeless 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8c77fe700 10 client.31257.objecter ms_handle_connect 0x7fd8c0071f80 2024-04-26T16:41:33.789 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.790+0000 7fd8d659c700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd8c0071f80 0x7fd8c0074440 secure :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009d50 tx=0x7fd8c8009040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:33.792 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:33.794+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fd8cc05d7f0 con 0x7fd8d806c490 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.953358+0000 osd.1 (osd.1) 565 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.953358+0000 osd.1 (osd.1) 565 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.955971+0000 osd.1 (osd.1) 566 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:31.955971+0000 osd.1 (osd.1) 566 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.093503+0000 mgr.y (mgr.24401) 917 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.093503+0000 mgr.y (mgr.24401) 917 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.219514+0000 osd.7 (osd.7) 598 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.219514+0000 osd.7 (osd.7) 598 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.221819+0000 osd.7 (osd.7) 599 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:41:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.221819+0000 osd.7 (osd.7) 599 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.437392+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.437392+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.439808+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.439808+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.944013+0000 osd.3 (osd.3) 619 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.944013+0000 osd.3 (osd.3) 619 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.945724+0000 osd.0 (osd.0) 591 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.945724+0000 osd.0 (osd.0) 591 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.946949+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.946949+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.948151+0000 osd.0 (osd.0) 592 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.948151+0000 osd.0 (osd.0) 592 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:33.117831+0000 osd.2 (osd.2) 511 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:33.117831+0000 osd.2 (osd.2) 511 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:41:33.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:33.120126+0000 osd.2 (osd.2) 512 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[29234]: cluster 2024-04-26T16:41:33.120126+0000 osd.2 (osd.2) 512 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.953358+0000 osd.1 (osd.1) 565 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.953358+0000 osd.1 (osd.1) 565 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.955971+0000 osd.1 (osd.1) 566 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:31.955971+0000 osd.1 (osd.1) 566 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.093503+0000 mgr.y (mgr.24401) 917 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.093503+0000 mgr.y (mgr.24401) 917 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.219514+0000 osd.7 (osd.7) 598 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.219514+0000 osd.7 (osd.7) 598 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.221819+0000 osd.7 (osd.7) 599 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:41:33.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.221819+0000 osd.7 (osd.7) 599 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.437392+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.437392+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.439808+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.439808+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.944013+0000 osd.3 (osd.3) 619 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.944013+0000 osd.3 (osd.3) 619 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.945724+0000 osd.0 (osd.0) 591 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.945724+0000 osd.0 (osd.0) 591 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.946949+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.946949+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.948151+0000 osd.0 (osd.0) 592 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.948151+0000 osd.0 (osd.0) 592 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:33.117831+0000 osd.2 (osd.2) 511 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:41:33.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:33.117831+0000 osd.2 (osd.2) 511 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:41:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:33.120126+0000 osd.2 (osd.2) 512 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:41:33.884 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:33 smithi050 bash[21463]: cluster 2024-04-26T16:41:33.120126+0000 osd.2 (osd.2) 512 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:41:34.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.038+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd reweight", "id": 6, "weight": 1.0} v 0) -- 0x7fd8d81a8180 con 0x7fd8d806c490 2024-04-26T16:41:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.953358+0000 osd.1 (osd.1) 565 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.953358+0000 osd.1 (osd.1) 565 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.955971+0000 osd.1 (osd.1) 566 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:31.955971+0000 osd.1 (osd.1) 566 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.093503+0000 mgr.y (mgr.24401) 917 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.093503+0000 mgr.y (mgr.24401) 917 : cluster [DBG] pgmap v1518: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.219514+0000 osd.7 (osd.7) 598 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.219514+0000 osd.7 (osd.7) 598 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.221819+0000 osd.7 (osd.7) 599 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.221819+0000 osd.7 (osd.7) 599 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:41:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.437392+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.437392+0000 osd.6 (osd.6) 127 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.439808+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.439808+0000 osd.6 (osd.6) 128 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.944013+0000 osd.3 (osd.3) 619 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.944013+0000 osd.3 (osd.3) 619 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.945724+0000 osd.0 (osd.0) 591 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.945724+0000 osd.0 (osd.0) 591 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.946949+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.946949+0000 osd.3 (osd.3) 620 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.948151+0000 osd.0 (osd.0) 592 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:41:34.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.948151+0000 osd.0 (osd.0) 592 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:41:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:33.117831+0000 osd.2 (osd.2) 511 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:41:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:33.117831+0000 osd.2 (osd.2) 511 : cluster [DBG] 6.f deep-scrub starts 2024-04-26T16:41:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:33.120126+0000 osd.2 (osd.2) 512 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:41:34.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:33 smithi190 bash[23868]: cluster 2024-04-26T16:41:33.120126+0000 osd.2 (osd.2) 512 : cluster [DBG] 6.f deep-scrub ok 2024-04-26T16:41:34.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.670+0000 7fd8c77fe700 1 -- 172.21.15.50:0/576822803 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd reweight", "id": 6, "weight": 1.0}]=0 reweighted osd.6 to 1 (10000) v923) ==== 113+0+0 (secure 0 0 0) 0x7fd8cc0614a0 con 0x7fd8d806c490 2024-04-26T16:41:34.672 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 10 librados: watch_flush enter 2024-04-26T16:41:34.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 10 librados: watch_flush exit 2024-04-26T16:41:34.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 20 client.31257.objecter shutdown clearing up homeless session... 2024-04-26T16:41:34.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 10 client.31257.objecter successfully canceled tick 2024-04-26T16:41:34.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd8c0071f80 msgr2=0x7fd8c0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:34.673 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd8c0071f80 0x7fd8c0074440 secure :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009d50 tx=0x7fd8c8009040 comp rx=0 tx=0).stop 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 msgr2=0x7fd8d8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.674+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8102d10 secure :-1 s=READY pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e9c0 tx=0x7fd8cc00ed80 comp rx=0 tx=0).stop 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 shutdown_connections 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd8c0071f80 0x7fd8c0074440 unknown :-1 s=CLOSED pgs=3870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd8d81094c0 0x7fd8d8103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd8d806baf0 0x7fd8d81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 --2- 172.21.15.50:0/576822803 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd8d806c490 0x7fd8d8102d10 secure :-1 s=CLOSED pgs=2556 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00e9c0 tx=0x7fd8cc00ed80 comp rx=0 tx=0).stop 2024-04-26T16:41:34.674 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 >> 172.21.15.50:0/576822803 conn(0x7fd8d80792d0 msgr2=0x7fd8d8100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:34.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 shutdown_connections 2024-04-26T16:41:34.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 -- 172.21.15.50:0/576822803 wait complete. 2024-04-26T16:41:34.675 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:34.678+0000 7fd8dcc8e700 1 librados: shutdown 2024-04-26T16:41:34.675 INFO:teuthology.orchestra.run.smithi050.stderr:reweighted osd.6 to 1 (10000) 2024-04-26T16:41:34.676 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:41:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:41:34.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.966595+0000 osd.1 (osd.1) 567 : cluster [DBG] 4.2 deep-scrub starts 2024-04-26T16:41:34.968 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.966595+0000 osd.1 (osd.1) 567 : cluster [DBG] 4.2 deep-scrub starts 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.972337+0000 osd.1 (osd.1) 568 : cluster [DBG] 4.2 deep-scrub ok 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:32.972337+0000 osd.1 (osd.1) 568 : cluster [DBG] 4.2 deep-scrub ok 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: audit 2024-04-26T16:41:34.040293+0000 mon.a (mon.0) 4057 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: audit 2024-04-26T16:41:34.040293+0000 mon.a (mon.0) 4057 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.051918+0000 osd.5 (osd.5) 474 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.051918+0000 osd.5 (osd.5) 474 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.054226+0000 osd.5 (osd.5) 475 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.054226+0000 osd.5 (osd.5) 475 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.162575+0000 osd.2 (osd.2) 513 : cluster [DBG] 11.d scrub starts 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.162575+0000 osd.2 (osd.2) 513 : cluster [DBG] 11.d scrub starts 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.166716+0000 osd.2 (osd.2) 514 : cluster [DBG] 11.d scrub ok 2024-04-26T16:41:34.969 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.166716+0000 osd.2 (osd.2) 514 : cluster [DBG] 11.d scrub ok 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.966595+0000 osd.1 (osd.1) 567 : cluster [DBG] 4.2 deep-scrub starts 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.966595+0000 osd.1 (osd.1) 567 : cluster [DBG] 4.2 deep-scrub starts 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.972337+0000 osd.1 (osd.1) 568 : cluster [DBG] 4.2 deep-scrub ok 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:32.972337+0000 osd.1 (osd.1) 568 : cluster [DBG] 4.2 deep-scrub ok 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: audit 2024-04-26T16:41:34.040293+0000 mon.a (mon.0) 4057 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: audit 2024-04-26T16:41:34.040293+0000 mon.a (mon.0) 4057 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.051918+0000 osd.5 (osd.5) 474 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.051918+0000 osd.5 (osd.5) 474 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.054226+0000 osd.5 (osd.5) 475 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.054226+0000 osd.5 (osd.5) 475 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.162575+0000 osd.2 (osd.2) 513 : cluster [DBG] 11.d scrub starts 2024-04-26T16:41:34.970 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.162575+0000 osd.2 (osd.2) 513 : cluster [DBG] 11.d scrub starts 2024-04-26T16:41:34.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.166716+0000 osd.2 (osd.2) 514 : cluster [DBG] 11.d scrub ok 2024-04-26T16:41:34.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:34 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.166716+0000 osd.2 (osd.2) 514 : cluster [DBG] 11.d scrub ok 2024-04-26T16:41:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.966595+0000 osd.1 (osd.1) 567 : cluster [DBG] 4.2 deep-scrub starts 2024-04-26T16:41:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.966595+0000 osd.1 (osd.1) 567 : cluster [DBG] 4.2 deep-scrub starts 2024-04-26T16:41:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.972337+0000 osd.1 (osd.1) 568 : cluster [DBG] 4.2 deep-scrub ok 2024-04-26T16:41:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:32.972337+0000 osd.1 (osd.1) 568 : cluster [DBG] 4.2 deep-scrub ok 2024-04-26T16:41:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: audit 2024-04-26T16:41:34.040293+0000 mon.a (mon.0) 4057 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: audit 2024-04-26T16:41:34.040293+0000 mon.a (mon.0) 4057 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.051918+0000 osd.5 (osd.5) 474 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:41:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.051918+0000 osd.5 (osd.5) 474 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:41:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.054226+0000 osd.5 (osd.5) 475 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:41:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.054226+0000 osd.5 (osd.5) 475 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:41:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.162575+0000 osd.2 (osd.2) 513 : cluster [DBG] 11.d scrub starts 2024-04-26T16:41:35.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.162575+0000 osd.2 (osd.2) 513 : cluster [DBG] 11.d scrub starts 2024-04-26T16:41:35.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.166716+0000 osd.2 (osd.2) 514 : cluster [DBG] 11.d scrub ok 2024-04-26T16:41:35.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:34 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.166716+0000 osd.2 (osd.2) 514 : cluster [DBG] 11.d scrub ok 2024-04-26T16:41:35.640 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd primary-affinity 6 1 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.094412+0000 mgr.y (mgr.24401) 918 : cluster [DBG] pgmap v1519: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.094412+0000 mgr.y (mgr.24401) 918 : cluster [DBG] pgmap v1519: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: audit 2024-04-26T16:41:34.671901+0000 mon.a (mon.0) 4058 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: audit 2024-04-26T16:41:34.671901+0000 mon.a (mon.0) 4058 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.677358+0000 mon.a (mon.0) 4059 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: cluster 2024-04-26T16:41:34.677358+0000 mon.a (mon.0) 4059 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2024-04-26T16:41:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: audit 2024-04-26T16:41:35.300382+0000 mon.c (mon.2) 944 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:35 smithi190 bash[23868]: audit 2024-04-26T16:41:35.300382+0000 mon.c (mon.2) 944 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.094412+0000 mgr.y (mgr.24401) 918 : cluster [DBG] pgmap v1519: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.094412+0000 mgr.y (mgr.24401) 918 : cluster [DBG] pgmap v1519: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: audit 2024-04-26T16:41:34.671901+0000 mon.a (mon.0) 4058 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: audit 2024-04-26T16:41:34.671901+0000 mon.a (mon.0) 4058 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.677358+0000 mon.a (mon.0) 4059 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: cluster 2024-04-26T16:41:34.677358+0000 mon.a (mon.0) 4059 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: audit 2024-04-26T16:41:35.300382+0000 mon.c (mon.2) 944 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[29234]: audit 2024-04-26T16:41:35.300382+0000 mon.c (mon.2) 944 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:36.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.094412+0000 mgr.y (mgr.24401) 918 : cluster [DBG] pgmap v1519: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.094412+0000 mgr.y (mgr.24401) 918 : cluster [DBG] pgmap v1519: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: audit 2024-04-26T16:41:34.671901+0000 mon.a (mon.0) 4058 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: audit 2024-04-26T16:41:34.671901+0000 mon.a (mon.0) 4058 : audit [INF] from='client.? 172.21.15.50:0/576822803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.677358+0000 mon.a (mon.0) 4059 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: cluster 2024-04-26T16:41:34.677358+0000 mon.a (mon.0) 4059 : cluster [DBG] osdmap e923: 8 total, 8 up, 8 in 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: audit 2024-04-26T16:41:35.300382+0000 mon.c (mon.2) 944 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:35 smithi050 bash[21463]: audit 2024-04-26T16:41:35.300382+0000 mon.c (mon.2) 944 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:41:37.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:35.912592+0000 osd.0 (osd.0) 593 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:35.912592+0000 osd.0 (osd.0) 593 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:35.914699+0000 osd.0 (osd.0) 594 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:35.914699+0000 osd.0 (osd.0) 594 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: audit 2024-04-26T16:41:36.033427+0000 mon.c (mon.2) 945 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: audit 2024-04-26T16:41:36.033427+0000 mon.c (mon.2) 945 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: audit 2024-04-26T16:41:36.035009+0000 mon.c (mon.2) 946 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: audit 2024-04-26T16:41:36.035009+0000 mon.c (mon.2) 946 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: audit 2024-04-26T16:41:36.044431+0000 mon.a (mon.0) 4060 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: audit 2024-04-26T16:41:36.044431+0000 mon.a (mon.0) 4060 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.099425+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.099425+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.101706+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:41:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:36 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.101706+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:35.912592+0000 osd.0 (osd.0) 593 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:35.912592+0000 osd.0 (osd.0) 593 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:35.914699+0000 osd.0 (osd.0) 594 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:35.914699+0000 osd.0 (osd.0) 594 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: audit 2024-04-26T16:41:36.033427+0000 mon.c (mon.2) 945 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: audit 2024-04-26T16:41:36.033427+0000 mon.c (mon.2) 945 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: audit 2024-04-26T16:41:36.035009+0000 mon.c (mon.2) 946 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: audit 2024-04-26T16:41:36.035009+0000 mon.c (mon.2) 946 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: audit 2024-04-26T16:41:36.044431+0000 mon.a (mon.0) 4060 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: audit 2024-04-26T16:41:36.044431+0000 mon.a (mon.0) 4060 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.099425+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.099425+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.101706+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.101706+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:35.912592+0000 osd.0 (osd.0) 593 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:35.912592+0000 osd.0 (osd.0) 593 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:35.914699+0000 osd.0 (osd.0) 594 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:35.914699+0000 osd.0 (osd.0) 594 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: audit 2024-04-26T16:41:36.033427+0000 mon.c (mon.2) 945 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: audit 2024-04-26T16:41:36.033427+0000 mon.c (mon.2) 945 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:41:37.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: audit 2024-04-26T16:41:36.035009+0000 mon.c (mon.2) 946 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: audit 2024-04-26T16:41:36.035009+0000 mon.c (mon.2) 946 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: audit 2024-04-26T16:41:36.044431+0000 mon.a (mon.0) 4060 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: audit 2024-04-26T16:41:36.044431+0000 mon.a (mon.0) 4060 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.099425+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.099425+0000 osd.5 (osd.5) 476 : cluster [DBG] 1.1 scrub starts 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.101706+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:41:37.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:36 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.101706+0000 osd.5 (osd.5) 477 : cluster [DBG] 1.1 scrub ok 2024-04-26T16:41:37.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:41:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.030366+0000 osd.3 (osd.3) 621 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:41:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.030366+0000 osd.3 (osd.3) 621 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:41:37.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.034074+0000 osd.3 (osd.3) 622 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.034074+0000 osd.3 (osd.3) 622 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.095177+0000 mgr.y (mgr.24401) 919 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.095177+0000 mgr.y (mgr.24401) 919 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.264717+0000 osd.7 (osd.7) 600 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.264717+0000 osd.7 (osd.7) 600 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.268180+0000 osd.7 (osd.7) 601 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.268180+0000 osd.7 (osd.7) 601 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.428531+0000 osd.6 (osd.6) 129 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.428531+0000 osd.6 (osd.6) 129 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.430946+0000 osd.6 (osd.6) 130 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.430946+0000 osd.6 (osd.6) 130 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.807591+0000 osd.4 (osd.4) 527 : cluster [DBG] 5.e scrub starts 2024-04-26T16:41:37.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.807591+0000 osd.4 (osd.4) 527 : cluster [DBG] 5.e scrub starts 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.810485+0000 osd.4 (osd.4) 528 : cluster [DBG] 5.e scrub ok 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.810485+0000 osd.4 (osd.4) 528 : cluster [DBG] 5.e scrub ok 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.951886+0000 osd.0 (osd.0) 595 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.951886+0000 osd.0 (osd.0) 595 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.955449+0000 osd.0 (osd.0) 596 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:36.955449+0000 osd.0 (osd.0) 596 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.106493+0000 osd.5 (osd.5) 478 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.106493+0000 osd.5 (osd.5) 478 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.109081+0000 osd.5 (osd.5) 479 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:41:37.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:37 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.109081+0000 osd.5 (osd.5) 479 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:41:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.030366+0000 osd.3 (osd.3) 621 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:41:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.030366+0000 osd.3 (osd.3) 621 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:41:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.034074+0000 osd.3 (osd.3) 622 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.034074+0000 osd.3 (osd.3) 622 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.095177+0000 mgr.y (mgr.24401) 919 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.095177+0000 mgr.y (mgr.24401) 919 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.264717+0000 osd.7 (osd.7) 600 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.264717+0000 osd.7 (osd.7) 600 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.268180+0000 osd.7 (osd.7) 601 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.268180+0000 osd.7 (osd.7) 601 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.428531+0000 osd.6 (osd.6) 129 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.428531+0000 osd.6 (osd.6) 129 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.430946+0000 osd.6 (osd.6) 130 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.430946+0000 osd.6 (osd.6) 130 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.807591+0000 osd.4 (osd.4) 527 : cluster [DBG] 5.e scrub starts 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.807591+0000 osd.4 (osd.4) 527 : cluster [DBG] 5.e scrub starts 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.810485+0000 osd.4 (osd.4) 528 : cluster [DBG] 5.e scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.810485+0000 osd.4 (osd.4) 528 : cluster [DBG] 5.e scrub ok 2024-04-26T16:41:38.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.951886+0000 osd.0 (osd.0) 595 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.951886+0000 osd.0 (osd.0) 595 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.955449+0000 osd.0 (osd.0) 596 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:36.955449+0000 osd.0 (osd.0) 596 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.106493+0000 osd.5 (osd.5) 478 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.106493+0000 osd.5 (osd.5) 478 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.109081+0000 osd.5 (osd.5) 479 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.109081+0000 osd.5 (osd.5) 479 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.030366+0000 osd.3 (osd.3) 621 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.030366+0000 osd.3 (osd.3) 621 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.034074+0000 osd.3 (osd.3) 622 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.034074+0000 osd.3 (osd.3) 622 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.095177+0000 mgr.y (mgr.24401) 919 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:38.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.095177+0000 mgr.y (mgr.24401) 919 : cluster [DBG] pgmap v1521: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.264717+0000 osd.7 (osd.7) 600 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.264717+0000 osd.7 (osd.7) 600 : cluster [DBG] 11.5 scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.268180+0000 osd.7 (osd.7) 601 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.268180+0000 osd.7 (osd.7) 601 : cluster [DBG] 11.5 scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.428531+0000 osd.6 (osd.6) 129 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.428531+0000 osd.6 (osd.6) 129 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.430946+0000 osd.6 (osd.6) 130 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.430946+0000 osd.6 (osd.6) 130 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.807591+0000 osd.4 (osd.4) 527 : cluster [DBG] 5.e scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.807591+0000 osd.4 (osd.4) 527 : cluster [DBG] 5.e scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.810485+0000 osd.4 (osd.4) 528 : cluster [DBG] 5.e scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.810485+0000 osd.4 (osd.4) 528 : cluster [DBG] 5.e scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.951886+0000 osd.0 (osd.0) 595 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.951886+0000 osd.0 (osd.0) 595 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.955449+0000 osd.0 (osd.0) 596 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:41:38.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:36.955449+0000 osd.0 (osd.0) 596 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:41:38.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.106493+0000 osd.5 (osd.5) 478 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:41:38.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.106493+0000 osd.5 (osd.5) 478 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:41:38.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.109081+0000 osd.5 (osd.5) 479 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:41:38.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:37 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.109081+0000 osd.5 (osd.5) 479 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:41:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:38 smithi190 bash[23868]: audit 2024-04-26T16:41:37.555929+0000 mgr.y (mgr.24401) 920 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:38 smithi190 bash[23868]: audit 2024-04-26T16:41:37.555929+0000 mgr.y (mgr.24401) 920 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:38 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.076239+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.1c deep-scrub starts 2024-04-26T16:41:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:38 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.076239+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.1c deep-scrub starts 2024-04-26T16:41:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:38 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.088847+0000 osd.5 (osd.5) 481 : cluster [DBG] 3.1c deep-scrub ok 2024-04-26T16:41:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:38 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.088847+0000 osd.5 (osd.5) 481 : cluster [DBG] 3.1c deep-scrub ok 2024-04-26T16:41:39.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[21463]: audit 2024-04-26T16:41:37.555929+0000 mgr.y (mgr.24401) 920 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[21463]: audit 2024-04-26T16:41:37.555929+0000 mgr.y (mgr.24401) 920 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.076239+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.1c deep-scrub starts 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.076239+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.1c deep-scrub starts 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.088847+0000 osd.5 (osd.5) 481 : cluster [DBG] 3.1c deep-scrub ok 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.088847+0000 osd.5 (osd.5) 481 : cluster [DBG] 3.1c deep-scrub ok 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[29234]: audit 2024-04-26T16:41:37.555929+0000 mgr.y (mgr.24401) 920 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[29234]: audit 2024-04-26T16:41:37.555929+0000 mgr.y (mgr.24401) 920 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.076239+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.1c deep-scrub starts 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.076239+0000 osd.5 (osd.5) 480 : cluster [DBG] 3.1c deep-scrub starts 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.088847+0000 osd.5 (osd.5) 481 : cluster [DBG] 3.1c deep-scrub ok 2024-04-26T16:41:39.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:38 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.088847+0000 osd.5 (osd.5) 481 : cluster [DBG] 3.1c deep-scrub ok 2024-04-26T16:41:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.930270+0000 osd.1 (osd.1) 569 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:41:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.930270+0000 osd.1 (osd.1) 569 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:41:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.932281+0000 osd.1 (osd.1) 570 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:41:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:37.932281+0000 osd.1 (osd.1) 570 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.096976+0000 mgr.y (mgr.24401) 921 : cluster [DBG] pgmap v1522: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.096976+0000 mgr.y (mgr.24401) 921 : cluster [DBG] pgmap v1522: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.301487+0000 osd.7 (osd.7) 602 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.301487+0000 osd.7 (osd.7) 602 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.305428+0000 osd.7 (osd.7) 603 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.305428+0000 osd.7 (osd.7) 603 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.386247+0000 osd.6 (osd.6) 131 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.386247+0000 osd.6 (osd.6) 131 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.388417+0000 osd.6 (osd.6) 132 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:38.388417+0000 osd.6 (osd.6) 132 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.206001+0000 osd.2 (osd.2) 515 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.206001+0000 osd.2 (osd.2) 515 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.208380+0000 osd.2 (osd.2) 516 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:41:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:39 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.208380+0000 osd.2 (osd.2) 516 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.930270+0000 osd.1 (osd.1) 569 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.930270+0000 osd.1 (osd.1) 569 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.932281+0000 osd.1 (osd.1) 570 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:37.932281+0000 osd.1 (osd.1) 570 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.096976+0000 mgr.y (mgr.24401) 921 : cluster [DBG] pgmap v1522: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.096976+0000 mgr.y (mgr.24401) 921 : cluster [DBG] pgmap v1522: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.301487+0000 osd.7 (osd.7) 602 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.301487+0000 osd.7 (osd.7) 602 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:41:40.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.305428+0000 osd.7 (osd.7) 603 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.305428+0000 osd.7 (osd.7) 603 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.386247+0000 osd.6 (osd.6) 131 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.386247+0000 osd.6 (osd.6) 131 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.388417+0000 osd.6 (osd.6) 132 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:38.388417+0000 osd.6 (osd.6) 132 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.206001+0000 osd.2 (osd.2) 515 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.206001+0000 osd.2 (osd.2) 515 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.208380+0000 osd.2 (osd.2) 516 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.208380+0000 osd.2 (osd.2) 516 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.930270+0000 osd.1 (osd.1) 569 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.930270+0000 osd.1 (osd.1) 569 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:41:40.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.932281+0000 osd.1 (osd.1) 570 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:37.932281+0000 osd.1 (osd.1) 570 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.096976+0000 mgr.y (mgr.24401) 921 : cluster [DBG] pgmap v1522: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.096976+0000 mgr.y (mgr.24401) 921 : cluster [DBG] pgmap v1522: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.301487+0000 osd.7 (osd.7) 602 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.301487+0000 osd.7 (osd.7) 602 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.305428+0000 osd.7 (osd.7) 603 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.305428+0000 osd.7 (osd.7) 603 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.386247+0000 osd.6 (osd.6) 131 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.386247+0000 osd.6 (osd.6) 131 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.388417+0000 osd.6 (osd.6) 132 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:38.388417+0000 osd.6 (osd.6) 132 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.206001+0000 osd.2 (osd.2) 515 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.206001+0000 osd.2 (osd.2) 515 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:41:40.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.208380+0000 osd.2 (osd.2) 516 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:41:40.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:39 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.208380+0000 osd.2 (osd.2) 516 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:41:40.428 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:41:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.035847+0000 osd.3 (osd.3) 623 : cluster [DBG] 2.1e deep-scrub starts 2024-04-26T16:41:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.035847+0000 osd.3 (osd.3) 623 : cluster [DBG] 2.1e deep-scrub starts 2024-04-26T16:41:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.038233+0000 osd.3 (osd.3) 624 : cluster [DBG] 2.1e deep-scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.038233+0000 osd.3 (osd.3) 624 : cluster [DBG] 2.1e deep-scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.320487+0000 osd.7 (osd.7) 604 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.320487+0000 osd.7 (osd.7) 604 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.323603+0000 osd.7 (osd.7) 605 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.323603+0000 osd.7 (osd.7) 605 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.370578+0000 osd.6 (osd.6) 133 : cluster [DBG] 3.a scrub starts 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.370578+0000 osd.6 (osd.6) 133 : cluster [DBG] 3.a scrub starts 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.375937+0000 osd.6 (osd.6) 134 : cluster [DBG] 3.a scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:39.375937+0000 osd.6 (osd.6) 134 : cluster [DBG] 3.a scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.166326+0000 osd.2 (osd.2) 517 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.166326+0000 osd.2 (osd.2) 517 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.169394+0000 osd.2 (osd.2) 518 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:41:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:40 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.169394+0000 osd.2 (osd.2) 518 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:41:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.035847+0000 osd.3 (osd.3) 623 : cluster [DBG] 2.1e deep-scrub starts 2024-04-26T16:41:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.035847+0000 osd.3 (osd.3) 623 : cluster [DBG] 2.1e deep-scrub starts 2024-04-26T16:41:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.038233+0000 osd.3 (osd.3) 624 : cluster [DBG] 2.1e deep-scrub ok 2024-04-26T16:41:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.038233+0000 osd.3 (osd.3) 624 : cluster [DBG] 2.1e deep-scrub ok 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.320487+0000 osd.7 (osd.7) 604 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.320487+0000 osd.7 (osd.7) 604 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.323603+0000 osd.7 (osd.7) 605 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.323603+0000 osd.7 (osd.7) 605 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.370578+0000 osd.6 (osd.6) 133 : cluster [DBG] 3.a scrub starts 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.370578+0000 osd.6 (osd.6) 133 : cluster [DBG] 3.a scrub starts 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.375937+0000 osd.6 (osd.6) 134 : cluster [DBG] 3.a scrub ok 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:39.375937+0000 osd.6 (osd.6) 134 : cluster [DBG] 3.a scrub ok 2024-04-26T16:41:41.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.166326+0000 osd.2 (osd.2) 517 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.166326+0000 osd.2 (osd.2) 517 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.169394+0000 osd.2 (osd.2) 518 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.169394+0000 osd.2 (osd.2) 518 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.035847+0000 osd.3 (osd.3) 623 : cluster [DBG] 2.1e deep-scrub starts 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.035847+0000 osd.3 (osd.3) 623 : cluster [DBG] 2.1e deep-scrub starts 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.038233+0000 osd.3 (osd.3) 624 : cluster [DBG] 2.1e deep-scrub ok 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.038233+0000 osd.3 (osd.3) 624 : cluster [DBG] 2.1e deep-scrub ok 2024-04-26T16:41:41.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.320487+0000 osd.7 (osd.7) 604 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.320487+0000 osd.7 (osd.7) 604 : cluster [DBG] 5.9 scrub starts 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.323603+0000 osd.7 (osd.7) 605 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.323603+0000 osd.7 (osd.7) 605 : cluster [DBG] 5.9 scrub ok 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.370578+0000 osd.6 (osd.6) 133 : cluster [DBG] 3.a scrub starts 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.370578+0000 osd.6 (osd.6) 133 : cluster [DBG] 3.a scrub starts 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.375937+0000 osd.6 (osd.6) 134 : cluster [DBG] 3.a scrub ok 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:39.375937+0000 osd.6 (osd.6) 134 : cluster [DBG] 3.a scrub ok 2024-04-26T16:41:41.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.166326+0000 osd.2 (osd.2) 517 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:41:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.166326+0000 osd.2 (osd.2) 517 : cluster [DBG] 11.15 deep-scrub starts 2024-04-26T16:41:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.169394+0000 osd.2 (osd.2) 518 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:41:41.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:40 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.169394+0000 osd.2 (osd.2) 518 : cluster [DBG] 11.15 deep-scrub ok 2024-04-26T16:41:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:41 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.097927+0000 mgr.y (mgr.24401) 922 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:41 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.097927+0000 mgr.y (mgr.24401) 922 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:41 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.343837+0000 osd.7 (osd.7) 606 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:41:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:41 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.343837+0000 osd.7 (osd.7) 606 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:41:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:41 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.348414+0000 osd.7 (osd.7) 607 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:41:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:41 smithi190 bash[23868]: cluster 2024-04-26T16:41:40.348414+0000 osd.7 (osd.7) 607 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:41:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.097927+0000 mgr.y (mgr.24401) 922 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.097927+0000 mgr.y (mgr.24401) 922 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.343837+0000 osd.7 (osd.7) 606 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:41:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.343837+0000 osd.7 (osd.7) 606 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:41:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.348414+0000 osd.7 (osd.7) 607 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:41:42.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[21463]: cluster 2024-04-26T16:41:40.348414+0000 osd.7 (osd.7) 607 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:41:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.097927+0000 mgr.y (mgr.24401) 922 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.097927+0000 mgr.y (mgr.24401) 922 : cluster [DBG] pgmap v1523: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.343837+0000 osd.7 (osd.7) 606 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:41:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.343837+0000 osd.7 (osd.7) 606 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:41:42.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.348414+0000 osd.7 (osd.7) 607 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:41:42.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:41 smithi050 bash[29234]: cluster 2024-04-26T16:41:40.348414+0000 osd.7 (osd.7) 607 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:41:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.052807+0000 osd.3 (osd.3) 625 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:41:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.052807+0000 osd.3 (osd.3) 625 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:41:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.057442+0000 osd.3 (osd.3) 626 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:41:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.057442+0000 osd.3 (osd.3) 626 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.393602+0000 osd.7 (osd.7) 608 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.393602+0000 osd.7 (osd.7) 608 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.398185+0000 osd.7 (osd.7) 609 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:41.398185+0000 osd.7 (osd.7) 609 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:42.145296+0000 osd.2 (osd.2) 519 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:42.145296+0000 osd.2 (osd.2) 519 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:42.147959+0000 osd.2 (osd.2) 520 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:41:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:42 smithi190 bash[23868]: cluster 2024-04-26T16:41:42.147959+0000 osd.2 (osd.2) 520 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.052807+0000 osd.3 (osd.3) 625 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.052807+0000 osd.3 (osd.3) 625 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.057442+0000 osd.3 (osd.3) 626 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.057442+0000 osd.3 (osd.3) 626 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.393602+0000 osd.7 (osd.7) 608 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.393602+0000 osd.7 (osd.7) 608 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.398185+0000 osd.7 (osd.7) 609 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:41.398185+0000 osd.7 (osd.7) 609 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:42.145296+0000 osd.2 (osd.2) 519 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:42.145296+0000 osd.2 (osd.2) 519 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:42.147959+0000 osd.2 (osd.2) 520 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:41:43.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[21463]: cluster 2024-04-26T16:41:42.147959+0000 osd.2 (osd.2) 520 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.052807+0000 osd.3 (osd.3) 625 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.052807+0000 osd.3 (osd.3) 625 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.057442+0000 osd.3 (osd.3) 626 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.057442+0000 osd.3 (osd.3) 626 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.393602+0000 osd.7 (osd.7) 608 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.393602+0000 osd.7 (osd.7) 608 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.398185+0000 osd.7 (osd.7) 609 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:41.398185+0000 osd.7 (osd.7) 609 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:42.145296+0000 osd.2 (osd.2) 519 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:42.145296+0000 osd.2 (osd.2) 519 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:42.147959+0000 osd.2 (osd.2) 520 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:41:43.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:42 smithi050 bash[29234]: cluster 2024-04-26T16:41:42.147959+0000 osd.2 (osd.2) 520 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:41:44.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/2645839941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c105640 msgr2=0x7f423c105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- 172.21.15.50:0/2645839941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c105640 0x7f423c105ac0 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7f422c0098f0 tx=0x7f422c009c00 comp rx=0 tx=0).stop 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/2645839941 shutdown_connections 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- 172.21.15.50:0/2645839941 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f423c106000 0x7f423c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- 172.21.15.50:0/2645839941 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c10b640 0x7f423c10ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- 172.21.15.50:0/2645839941 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c105640 0x7f423c105ac0 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/2645839941 >> 172.21.15.50:0/2645839941 conn(0x7f423c101130 msgr2=0x7f423c103550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/2645839941 shutdown_connections 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/2645839941 wait complete. 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 librados: starting msgr at 2024-04-26T16:41:44.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 librados: starting objecter 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 Processor -- start 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- start start 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 librados: setting wanted keys 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 librados: calling monclient init 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f423c105640 0x7f423c19b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c106000 0x7f423c19b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c10b640 0x7f423c19fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f423c114d40 con 0x7f423c10b640 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f423c114bc0 con 0x7f423c105640 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f423c114ec0 con 0x7f423c106000 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4239d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c106000 0x7f423c19b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4239d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c106000 0x7f423c19b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55646/0 (socket says 172.21.15.50:55646) 2024-04-26T16:41:44.033 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4239d9b700 1 -- 172.21.15.50:0/3927697638 learned_addr learned my addr 172.21.15.50:0/3927697638 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:41:44.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c10b640 0x7f423c19fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:44.034 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423a59c700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f423c105640 0x7f423c19b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 -- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c106000 msgr2=0x7f423c19b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c106000 0x7f423c19b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 -- 172.21.15.50:0/3927697638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f423c105640 msgr2=0x7f423c19b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f423c105640 0x7f423c19b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.035 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 -- 172.21.15.50:0/3927697638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423c1a0330 con 0x7f423c10b640 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f423ad9d700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c10b640 0x7f423c19fbf0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4234007f10 tx=0x7f423400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f42337fe700 10 client.?.objecter ms_handle_connect 0x7f423c10b640 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f42337fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423400ddd0 con 0x7f423c10b640 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423c1a0620 con 0x7f423c10b640 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4234009e20 con 0x7f423c10b640 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4234005a40 con 0x7f423c10b640 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 10 client.31263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:41:44.036 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423c1a7ea0 con 0x7f423c10b640 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.034+0000 7f4240d2e700 1 librados: init done 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423c06a110 con 0x7f423c10b640 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4234005020 con 0x7f423c10b640 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4228071f80 0x7f4228074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(923..923 src has 864..923) ==== 8707+0+0 (secure 0 0 0) 0x7f4234091110 con 0x7f423c10b640 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 10 client.31263.objecter ms_dispatch 0x7f423c001550 osd_map(923..923 src has 864..923) 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 3 client.31263.objecter handle_osd_map got epochs [923,923] > 0 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 3 client.31263.objecter handle_osd_map decoding full epoch 923 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f423a59c700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4228071f80 0x7f4228074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 20 client.31263.objecter dump_active .. 0 homeless 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f42337fe700 10 client.31263.objecter ms_handle_connect 0x7f4228071f80 2024-04-26T16:41:44.037 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.038+0000 7f423a59c700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4228071f80 0x7f4228074440 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f4224005fd0 tx=0x7f4224009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:44.040 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.042+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4234010070 con 0x7f423c10b640 2024-04-26T16:41:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:43 smithi190 bash[23868]: cluster 2024-04-26T16:41:42.099778+0000 mgr.y (mgr.24401) 923 : cluster [DBG] pgmap v1524: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:43 smithi190 bash[23868]: cluster 2024-04-26T16:41:42.099778+0000 mgr.y (mgr.24401) 923 : cluster [DBG] pgmap v1524: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:43 smithi190 bash[23868]: audit 2024-04-26T16:41:43.548793+0000 mon.c (mon.2) 947 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:44.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:43 smithi190 bash[23868]: audit 2024-04-26T16:41:43.548793+0000 mon.c (mon.2) 947 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:44.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[21463]: cluster 2024-04-26T16:41:42.099778+0000 mgr.y (mgr.24401) 923 : cluster [DBG] pgmap v1524: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:44.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[21463]: cluster 2024-04-26T16:41:42.099778+0000 mgr.y (mgr.24401) 923 : cluster [DBG] pgmap v1524: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:44.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[21463]: audit 2024-04-26T16:41:43.548793+0000 mon.c (mon.2) 947 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:44.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[21463]: audit 2024-04-26T16:41:43.548793+0000 mon.c (mon.2) 947 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:44.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[29234]: cluster 2024-04-26T16:41:42.099778+0000 mgr.y (mgr.24401) 923 : cluster [DBG] pgmap v1524: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:44.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[29234]: cluster 2024-04-26T16:41:42.099778+0000 mgr.y (mgr.24401) 923 : cluster [DBG] pgmap v1524: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:44.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[29234]: audit 2024-04-26T16:41:43.548793+0000 mon.c (mon.2) 947 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:44.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:43 smithi050 bash[29234]: audit 2024-04-26T16:41:43.548793+0000 mon.c (mon.2) 947 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:44.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.282+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} v 0) -- 0x7f423c02e490 con 0x7f423c10b640 2024-04-26T16:41:44.718 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.722+0000 7f42337fe700 1 -- 172.21.15.50:0/3927697638 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]=0 set osd.6 primary-affinity to 1 (8655362) v924) ==== 133+0+0 (secure 0 0 0) 0x7f423405cde0 con 0x7f423c10b640 2024-04-26T16:41:44.723 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 10 librados: watch_flush enter 2024-04-26T16:41:44.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 10 librados: watch_flush exit 2024-04-26T16:41:44.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 20 client.31263.objecter shutdown clearing up homeless session... 2024-04-26T16:41:44.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 10 client.31263.objecter successfully canceled tick 2024-04-26T16:41:44.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4228071f80 msgr2=0x7f4228074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:44.724 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4228071f80 0x7f4228074440 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f4224005fd0 tx=0x7f4224009310 comp rx=0 tx=0).stop 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c10b640 msgr2=0x7f423c19fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c10b640 0x7f423c19fbf0 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f4234007f10 tx=0x7f423400d3b0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 shutdown_connections 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4228071f80 0x7f4228074440 unknown :-1 s=CLOSED pgs=3871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f423c105640 0x7f423c19b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f423c10b640 0x7f423c19fbf0 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 --2- 172.21.15.50:0/3927697638 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f423c106000 0x7f423c19b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 >> 172.21.15.50:0/3927697638 conn(0x7f423c101130 msgr2=0x7f423c101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 shutdown_connections 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 -- 172.21.15.50:0/3927697638 wait complete. 2024-04-26T16:41:44.725 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:44.726+0000 7f4240d2e700 1 librados: shutdown 2024-04-26T16:41:44.726 INFO:teuthology.orchestra.run.smithi050.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-04-26T16:41:44.729 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:41:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:41:45.000 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.006667+0000 osd.3 (osd.3) 627 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.006667+0000 osd.3 (osd.3) 627 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.013297+0000 osd.3 (osd.3) 628 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.013297+0000 osd.3 (osd.3) 628 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.437617+0000 osd.7 (osd.7) 610 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.437617+0000 osd.7 (osd.7) 610 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.439969+0000 osd.7 (osd.7) 611 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: cluster 2024-04-26T16:41:43.439969+0000 osd.7 (osd.7) 611 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: audit 2024-04-26T16:41:44.285838+0000 mon.a (mon.0) 4061 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[21463]: audit 2024-04-26T16:41:44.285838+0000 mon.a (mon.0) 4061 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.006667+0000 osd.3 (osd.3) 627 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.006667+0000 osd.3 (osd.3) 627 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:41:45.001 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.013297+0000 osd.3 (osd.3) 628 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.013297+0000 osd.3 (osd.3) 628 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.437617+0000 osd.7 (osd.7) 610 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.437617+0000 osd.7 (osd.7) 610 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.439969+0000 osd.7 (osd.7) 611 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: cluster 2024-04-26T16:41:43.439969+0000 osd.7 (osd.7) 611 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: audit 2024-04-26T16:41:44.285838+0000 mon.a (mon.0) 4061 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:45.002 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:44 smithi050 bash[29234]: audit 2024-04-26T16:41:44.285838+0000 mon.a (mon.0) 4061 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.006667+0000 osd.3 (osd.3) 627 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:41:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.006667+0000 osd.3 (osd.3) 627 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:41:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.013297+0000 osd.3 (osd.3) 628 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:41:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.013297+0000 osd.3 (osd.3) 628 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:41:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.437617+0000 osd.7 (osd.7) 610 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:41:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.437617+0000 osd.7 (osd.7) 610 : cluster [DBG] 6.5 scrub starts 2024-04-26T16:41:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.439969+0000 osd.7 (osd.7) 611 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:41:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: cluster 2024-04-26T16:41:43.439969+0000 osd.7 (osd.7) 611 : cluster [DBG] 6.5 scrub ok 2024-04-26T16:41:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: audit 2024-04-26T16:41:44.285838+0000 mon.a (mon.0) 4061 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:45.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:44 smithi190 bash[23868]: audit 2024-04-26T16:41:44.285838+0000 mon.a (mon.0) 4061 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-04-26T16:41:45.646 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-04-26T16:41:45.646 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-04-26T16:41:45.646 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-04-26T16:41:45.646 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-04-26T16:41:45.646 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-26T16:41:45.647 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T16:41:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:44.100669+0000 mgr.y (mgr.24401) 924 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:44.100669+0000 mgr.y (mgr.24401) 924 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: audit 2024-04-26T16:41:44.723521+0000 mon.a (mon.0) 4062 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: audit 2024-04-26T16:41:44.723521+0000 mon.a (mon.0) 4062 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:44.728447+0000 mon.a (mon.0) 4063 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2024-04-26T16:41:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:44.728447+0000 mon.a (mon.0) 4063 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2024-04-26T16:41:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.157098+0000 osd.2 (osd.2) 521 : cluster [DBG] 11.11 deep-scrub starts 2024-04-26T16:41:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.157098+0000 osd.2 (osd.2) 521 : cluster [DBG] 11.11 deep-scrub starts 2024-04-26T16:41:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.163010+0000 osd.2 (osd.2) 522 : cluster [DBG] 11.11 deep-scrub ok 2024-04-26T16:41:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:45 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.163010+0000 osd.2 (osd.2) 522 : cluster [DBG] 11.11 deep-scrub ok 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:44.100669+0000 mgr.y (mgr.24401) 924 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:44.100669+0000 mgr.y (mgr.24401) 924 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: audit 2024-04-26T16:41:44.723521+0000 mon.a (mon.0) 4062 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: audit 2024-04-26T16:41:44.723521+0000 mon.a (mon.0) 4062 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:44.728447+0000 mon.a (mon.0) 4063 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:44.728447+0000 mon.a (mon.0) 4063 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.157098+0000 osd.2 (osd.2) 521 : cluster [DBG] 11.11 deep-scrub starts 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.157098+0000 osd.2 (osd.2) 521 : cluster [DBG] 11.11 deep-scrub starts 2024-04-26T16:41:46.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.163010+0000 osd.2 (osd.2) 522 : cluster [DBG] 11.11 deep-scrub ok 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.163010+0000 osd.2 (osd.2) 522 : cluster [DBG] 11.11 deep-scrub ok 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:44.100669+0000 mgr.y (mgr.24401) 924 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:44.100669+0000 mgr.y (mgr.24401) 924 : cluster [DBG] pgmap v1525: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: audit 2024-04-26T16:41:44.723521+0000 mon.a (mon.0) 4062 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: audit 2024-04-26T16:41:44.723521+0000 mon.a (mon.0) 4062 : audit [INF] from='client.? 172.21.15.50:0/3927697638' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:44.728447+0000 mon.a (mon.0) 4063 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:44.728447+0000 mon.a (mon.0) 4063 : cluster [DBG] osdmap e924: 8 total, 8 up, 8 in 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.157098+0000 osd.2 (osd.2) 521 : cluster [DBG] 11.11 deep-scrub starts 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.157098+0000 osd.2 (osd.2) 521 : cluster [DBG] 11.11 deep-scrub starts 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.163010+0000 osd.2 (osd.2) 522 : cluster [DBG] 11.11 deep-scrub ok 2024-04-26T16:41:46.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:45 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.163010+0000 osd.2 (osd.2) 522 : cluster [DBG] 11.11 deep-scrub ok 2024-04-26T16:41:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.458103+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:41:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.458103+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:41:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.460595+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:41:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:45.460595+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:41:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.084569+0000 osd.5 (osd.5) 482 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:41:47.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.084569+0000 osd.5 (osd.5) 482 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:41:47.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.088303+0000 osd.5 (osd.5) 483 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:41:47.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:46 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.088303+0000 osd.5 (osd.5) 483 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:41:47.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.458103+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.458103+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.460595+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:45.460595+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.084569+0000 osd.5 (osd.5) 482 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.084569+0000 osd.5 (osd.5) 482 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.088303+0000 osd.5 (osd.5) 483 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.088303+0000 osd.5 (osd.5) 483 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.458103+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.458103+0000 osd.7 (osd.7) 612 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:41:47.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.460595+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:41:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:45.460595+0000 osd.7 (osd.7) 613 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:41:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.084569+0000 osd.5 (osd.5) 482 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:41:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.084569+0000 osd.5 (osd.5) 482 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:41:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.088303+0000 osd.5 (osd.5) 483 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:41:47.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:46 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.088303+0000 osd.5 (osd.5) 483 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:41:47.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:41:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.101455+0000 mgr.y (mgr.24401) 925 : cluster [DBG] pgmap v1527: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.101455+0000 mgr.y (mgr.24401) 925 : cluster [DBG] pgmap v1527: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.710399+0000 osd.4 (osd.4) 529 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:41:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.710399+0000 osd.4 (osd.4) 529 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:41:47.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.712712+0000 osd.4 (osd.4) 530 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:41:47.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:47 smithi190 bash[23868]: cluster 2024-04-26T16:41:46.712712+0000 osd.4 (osd.4) 530 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:41:48.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.101455+0000 mgr.y (mgr.24401) 925 : cluster [DBG] pgmap v1527: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.101455+0000 mgr.y (mgr.24401) 925 : cluster [DBG] pgmap v1527: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.710399+0000 osd.4 (osd.4) 529 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.710399+0000 osd.4 (osd.4) 529 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.712712+0000 osd.4 (osd.4) 530 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[21463]: cluster 2024-04-26T16:41:46.712712+0000 osd.4 (osd.4) 530 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.101455+0000 mgr.y (mgr.24401) 925 : cluster [DBG] pgmap v1527: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.101455+0000 mgr.y (mgr.24401) 925 : cluster [DBG] pgmap v1527: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.710399+0000 osd.4 (osd.4) 529 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.710399+0000 osd.4 (osd.4) 529 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.712712+0000 osd.4 (osd.4) 530 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:41:48.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:47 smithi050 bash[29234]: cluster 2024-04-26T16:41:46.712712+0000 osd.4 (osd.4) 530 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:41:49.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: audit 2024-04-26T16:41:47.565828+0000 mgr.y (mgr.24401) 926 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:49.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: audit 2024-04-26T16:41:47.565828+0000 mgr.y (mgr.24401) 926 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:49.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.751645+0000 osd.4 (osd.4) 531 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:41:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.751645+0000 osd.4 (osd.4) 531 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:41:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.754109+0000 osd.4 (osd.4) 532 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:41:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.754109+0000 osd.4 (osd.4) 532 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:41:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.990343+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:41:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:48 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.990343+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:41:49.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: audit 2024-04-26T16:41:47.565828+0000 mgr.y (mgr.24401) 926 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: audit 2024-04-26T16:41:47.565828+0000 mgr.y (mgr.24401) 926 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.751645+0000 osd.4 (osd.4) 531 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.751645+0000 osd.4 (osd.4) 531 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.754109+0000 osd.4 (osd.4) 532 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.754109+0000 osd.4 (osd.4) 532 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.990343+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.990343+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: audit 2024-04-26T16:41:47.565828+0000 mgr.y (mgr.24401) 926 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:49.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: audit 2024-04-26T16:41:47.565828+0000 mgr.y (mgr.24401) 926 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:49.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.751645+0000 osd.4 (osd.4) 531 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:41:49.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.751645+0000 osd.4 (osd.4) 531 : cluster [DBG] 5.18 scrub starts 2024-04-26T16:41:49.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.754109+0000 osd.4 (osd.4) 532 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:41:49.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.754109+0000 osd.4 (osd.4) 532 : cluster [DBG] 5.18 scrub ok 2024-04-26T16:41:49.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.990343+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:41:49.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:48 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.990343+0000 osd.3 (osd.3) 629 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:41:50.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.993709+0000 osd.3 (osd.3) 630 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:41:50.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:47.993709+0000 osd.3 (osd.3) 630 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:41:50.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:48.103407+0000 mgr.y (mgr.24401) 927 : cluster [DBG] pgmap v1528: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:48.103407+0000 mgr.y (mgr.24401) 927 : cluster [DBG] pgmap v1528: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:48.365040+0000 osd.7 (osd.7) 614 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:41:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:48.365040+0000 osd.7 (osd.7) 614 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:41:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:48.367492+0000 osd.7 (osd.7) 615 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:41:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:49 smithi190 bash[23868]: cluster 2024-04-26T16:41:48.367492+0000 osd.7 (osd.7) 615 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.993709+0000 osd.3 (osd.3) 630 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:47.993709+0000 osd.3 (osd.3) 630 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:48.103407+0000 mgr.y (mgr.24401) 927 : cluster [DBG] pgmap v1528: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:48.103407+0000 mgr.y (mgr.24401) 927 : cluster [DBG] pgmap v1528: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:48.365040+0000 osd.7 (osd.7) 614 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:48.365040+0000 osd.7 (osd.7) 614 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:48.367492+0000 osd.7 (osd.7) 615 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[29234]: cluster 2024-04-26T16:41:48.367492+0000 osd.7 (osd.7) 615 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:41:50.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.993709+0000 osd.3 (osd.3) 630 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:47.993709+0000 osd.3 (osd.3) 630 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:48.103407+0000 mgr.y (mgr.24401) 927 : cluster [DBG] pgmap v1528: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:48.103407+0000 mgr.y (mgr.24401) 927 : cluster [DBG] pgmap v1528: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:48.365040+0000 osd.7 (osd.7) 614 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:48.365040+0000 osd.7 (osd.7) 614 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:48.367492+0000 osd.7 (osd.7) 615 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:41:50.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:49 smithi050 bash[21463]: cluster 2024-04-26T16:41:48.367492+0000 osd.7 (osd.7) 615 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:41:50.444 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:41:51.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.386701+0000 osd.7 (osd.7) 616 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.386701+0000 osd.7 (osd.7) 616 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.389176+0000 osd.7 (osd.7) 617 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.389176+0000 osd.7 (osd.7) 617 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.781752+0000 osd.4 (osd.4) 533 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.781752+0000 osd.4 (osd.4) 533 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.784153+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:49.784153+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.002661+0000 osd.5 (osd.5) 484 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.002661+0000 osd.5 (osd.5) 484 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:41:51.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.005824+0000 osd.5 (osd.5) 485 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:41:51.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:50 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.005824+0000 osd.5 (osd.5) 485 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:41:51.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.386701+0000 osd.7 (osd.7) 616 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:41:51.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.386701+0000 osd.7 (osd.7) 616 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:41:51.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.389176+0000 osd.7 (osd.7) 617 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:41:51.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.389176+0000 osd.7 (osd.7) 617 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.781752+0000 osd.4 (osd.4) 533 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.781752+0000 osd.4 (osd.4) 533 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.784153+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:49.784153+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.002661+0000 osd.5 (osd.5) 484 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.002661+0000 osd.5 (osd.5) 484 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:41:51.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.005824+0000 osd.5 (osd.5) 485 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:41:51.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.005824+0000 osd.5 (osd.5) 485 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:41:51.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.386701+0000 osd.7 (osd.7) 616 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:41:51.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.386701+0000 osd.7 (osd.7) 616 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:41:51.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.389176+0000 osd.7 (osd.7) 617 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:41:51.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.389176+0000 osd.7 (osd.7) 617 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:41:51.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.781752+0000 osd.4 (osd.4) 533 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:41:51.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.781752+0000 osd.4 (osd.4) 533 : cluster [DBG] 4.1 scrub starts 2024-04-26T16:41:51.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.784153+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:41:51.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:49.784153+0000 osd.4 (osd.4) 534 : cluster [DBG] 4.1 scrub ok 2024-04-26T16:41:51.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.002661+0000 osd.5 (osd.5) 484 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:41:51.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.002661+0000 osd.5 (osd.5) 484 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:41:51.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.005824+0000 osd.5 (osd.5) 485 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:41:51.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:50 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.005824+0000 osd.5 (osd.5) 485 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:41:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.063393+0000 osd.0 (osd.0) 597 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.063393+0000 osd.0 (osd.0) 597 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.065969+0000 osd.0 (osd.0) 598 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.065969+0000 osd.0 (osd.0) 598 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.104312+0000 mgr.y (mgr.24401) 928 : cluster [DBG] pgmap v1529: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.104312+0000 mgr.y (mgr.24401) 928 : cluster [DBG] pgmap v1529: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.424192+0000 osd.7 (osd.7) 618 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.424192+0000 osd.7 (osd.7) 618 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.426580+0000 osd.7 (osd.7) 619 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.426580+0000 osd.7 (osd.7) 619 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.957398+0000 osd.3 (osd.3) 631 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.957398+0000 osd.3 (osd.3) 631 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.960198+0000 osd.3 (osd.3) 632 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:41:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.960198+0000 osd.3 (osd.3) 632 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:41:52.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.974092+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:41:52.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.974092+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:41:52.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.977335+0000 osd.5 (osd.5) 487 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:41:52.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:51 smithi190 bash[23868]: cluster 2024-04-26T16:41:50.977335+0000 osd.5 (osd.5) 487 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:41:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.063393+0000 osd.0 (osd.0) 597 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.063393+0000 osd.0 (osd.0) 597 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.065969+0000 osd.0 (osd.0) 598 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.065969+0000 osd.0 (osd.0) 598 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.104312+0000 mgr.y (mgr.24401) 928 : cluster [DBG] pgmap v1529: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.104312+0000 mgr.y (mgr.24401) 928 : cluster [DBG] pgmap v1529: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.424192+0000 osd.7 (osd.7) 618 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.424192+0000 osd.7 (osd.7) 618 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.426580+0000 osd.7 (osd.7) 619 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.426580+0000 osd.7 (osd.7) 619 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.957398+0000 osd.3 (osd.3) 631 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.957398+0000 osd.3 (osd.3) 631 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.960198+0000 osd.3 (osd.3) 632 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.960198+0000 osd.3 (osd.3) 632 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:41:52.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.974092+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.974092+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.977335+0000 osd.5 (osd.5) 487 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[21463]: cluster 2024-04-26T16:41:50.977335+0000 osd.5 (osd.5) 487 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.063393+0000 osd.0 (osd.0) 597 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.063393+0000 osd.0 (osd.0) 597 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.065969+0000 osd.0 (osd.0) 598 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.065969+0000 osd.0 (osd.0) 598 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.104312+0000 mgr.y (mgr.24401) 928 : cluster [DBG] pgmap v1529: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.104312+0000 mgr.y (mgr.24401) 928 : cluster [DBG] pgmap v1529: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.424192+0000 osd.7 (osd.7) 618 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.424192+0000 osd.7 (osd.7) 618 : cluster [DBG] 2.b deep-scrub starts 2024-04-26T16:41:52.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.426580+0000 osd.7 (osd.7) 619 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.426580+0000 osd.7 (osd.7) 619 : cluster [DBG] 2.b deep-scrub ok 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.957398+0000 osd.3 (osd.3) 631 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.957398+0000 osd.3 (osd.3) 631 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.960198+0000 osd.3 (osd.3) 632 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.960198+0000 osd.3 (osd.3) 632 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.974092+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.974092+0000 osd.5 (osd.5) 486 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.977335+0000 osd.5 (osd.5) 487 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:41:52.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:51 smithi050 bash[29234]: cluster 2024-04-26T16:41:50.977335+0000 osd.5 (osd.5) 487 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:41:53.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.934+0000 7f20727da700 1 -- 172.21.15.50:0/2990921830 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103e30 msgr2=0x7f206c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:53.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.934+0000 7f20727da700 1 --2- 172.21.15.50:0/2990921830 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103e30 0x7f206c110290 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f2064009f90 tx=0x7f206400b330 comp rx=0 tx=0).stop 2024-04-26T16:41:53.934 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/2990921830 shutdown_connections 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 --2- 172.21.15.50:0/2990921830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f206c1094c0 0x7f206c1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 --2- 172.21.15.50:0/2990921830 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103e30 0x7f206c110290 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 --2- 172.21.15.50:0/2990921830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103470 0x7f206c1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/2990921830 >> 172.21.15.50:0/2990921830 conn(0x7f206c07a580 msgr2=0x7f206c07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/2990921830 shutdown_connections 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/2990921830 wait complete. 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 librados: starting msgr at 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 librados: starting objecter 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 Processor -- start 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- start start 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 librados: setting wanted keys 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 librados: calling monclient init 2024-04-26T16:41:53.935 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 0x7f206c1990a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 0x7f206c1995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f206c1094c0 0x7f206c19d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f206c112a60 con 0x7f206c103e30 2024-04-26T16:41:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f206c1128e0 con 0x7f206c1094c0 2024-04-26T16:41:53.936 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f206c112be0 con 0x7f206c103470 2024-04-26T16:41:53.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206b7fe700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 0x7f206c1995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:53.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 0x7f206c1990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:53.937 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 0x7f206c1990a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:46060/0 (socket says 172.21.15.50:46060) 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 -- 172.21.15.50:0/3662855182 learned_addr learned my addr 172.21.15.50:0/3662855182 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206b7fe700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 0x7f206c1995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:51070/0 (socket says 172.21.15.50:51070) 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f2070d77700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f206c1094c0 0x7f206c19d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 -- 172.21.15.50:0/3662855182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f206c1094c0 msgr2=0x7f206c19d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f206c1094c0 0x7f206c19d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 -- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 msgr2=0x7f206c1995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 0x7f206c1995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 -- 172.21.15.50:0/3662855182 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c19e0b0 con 0x7f206c103470 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206b7fe700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 0x7f206c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f206bfff700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 0x7f206c1990a0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f205c00d980 tx=0x7f205c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20697fa700 10 client.?.objecter ms_handle_connect 0x7f206c103470 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20697fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c017070 con 0x7f206c103470 2024-04-26T16:41:53.938 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f205c005020 con 0x7f206c103470 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c19e3a0 con 0x7f206c103470 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c009990 con 0x7f206c103470 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 10 client.41888.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c1a5c20 con 0x7f206c103470 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 librados: init done 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f206c06a110 con 0x7f206c103470 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.938+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f205c004a30 con 0x7f206c103470 2024-04-26T16:41:53.939 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2054071f80 0x7f2054074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(924..924 src has 874..924) ==== 8707+0+0 (secure 0 0 0) 0x7f205c0915b0 con 0x7f206c103470 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 10 client.41888.objecter ms_dispatch 0x7f206c001550 osd_map(924..924 src has 874..924) 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 3 client.41888.objecter handle_osd_map got epochs [924,924] > 0 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f206b7fe700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2054071f80 0x7f2054074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 3 client.41888.objecter handle_osd_map decoding full epoch 924 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 20 client.41888.objecter dump_active .. 0 homeless 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f20697fa700 10 client.41888.objecter ms_handle_connect 0x7f2054071f80 2024-04-26T16:41:53.940 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.942+0000 7f206b7fe700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2054071f80 0x7f2054074440 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f2060009750 tx=0x7f2060009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:41:53.943 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:53.946+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f205c020020 con 0x7f206c103470 2024-04-26T16:41:54.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:52.106166+0000 mgr.y (mgr.24401) 929 : cluster [DBG] pgmap v1530: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:54.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:52.106166+0000 mgr.y (mgr.24401) 929 : cluster [DBG] pgmap v1530: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:54.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:52.939862+0000 osd.3 (osd.3) 633 : cluster [DBG] 11.b scrub starts 2024-04-26T16:41:54.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:52.939862+0000 osd.3 (osd.3) 633 : cluster [DBG] 11.b scrub starts 2024-04-26T16:41:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:52.946918+0000 osd.3 (osd.3) 634 : cluster [DBG] 11.b scrub ok 2024-04-26T16:41:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:52.946918+0000 osd.3 (osd.3) 634 : cluster [DBG] 11.b scrub ok 2024-04-26T16:41:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.068061+0000 osd.2 (osd.2) 523 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:41:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.068061+0000 osd.2 (osd.2) 523 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:41:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.070942+0000 osd.2 (osd.2) 524 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:41:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:53 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.070942+0000 osd.2 (osd.2) 524 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:41:54.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:52.106166+0000 mgr.y (mgr.24401) 929 : cluster [DBG] pgmap v1530: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:52.106166+0000 mgr.y (mgr.24401) 929 : cluster [DBG] pgmap v1530: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:52.939862+0000 osd.3 (osd.3) 633 : cluster [DBG] 11.b scrub starts 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:52.939862+0000 osd.3 (osd.3) 633 : cluster [DBG] 11.b scrub starts 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:52.946918+0000 osd.3 (osd.3) 634 : cluster [DBG] 11.b scrub ok 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:52.946918+0000 osd.3 (osd.3) 634 : cluster [DBG] 11.b scrub ok 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.068061+0000 osd.2 (osd.2) 523 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.068061+0000 osd.2 (osd.2) 523 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.070942+0000 osd.2 (osd.2) 524 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.070942+0000 osd.2 (osd.2) 524 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:41:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:52.106166+0000 mgr.y (mgr.24401) 929 : cluster [DBG] pgmap v1530: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:52.106166+0000 mgr.y (mgr.24401) 929 : cluster [DBG] pgmap v1530: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:52.939862+0000 osd.3 (osd.3) 633 : cluster [DBG] 11.b scrub starts 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:52.939862+0000 osd.3 (osd.3) 633 : cluster [DBG] 11.b scrub starts 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:52.946918+0000 osd.3 (osd.3) 634 : cluster [DBG] 11.b scrub ok 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:52.946918+0000 osd.3 (osd.3) 634 : cluster [DBG] 11.b scrub ok 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.068061+0000 osd.2 (osd.2) 523 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.068061+0000 osd.2 (osd.2) 523 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.070942+0000 osd.2 (osd.2) 524 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:41:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:53 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.070942+0000 osd.2 (osd.2) 524 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:41:54.172 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.174+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f206c1a5df0 con 0x7f206c103470 2024-04-26T16:41:54.174 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.178+0000 7f20697fa700 1 -- 172.21.15.50:0/3662855182 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v924) ==== 74+0+26252 (secure 0 0 0) 0x7f205c05d200 con 0x7f206c103470 2024-04-26T16:41:54.175 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:41:54.175 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":924,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","created":"2024-04-26T16:07:04.494213+0000","modified":"2024-04-26T16:41:44.720694+0000","last_up_change":"2024-04-26T16:35:24.350639+0000","last_in_change":"2024-04-26T16:32:08.394374+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":112,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T16:12:30.809607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"889","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"881","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-26T16:17:27.029484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-26T16:17:29.124965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-26T16:17:30.749319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-26T16:17:31.064350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-26T16:17:33.192127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-26T16:23:28.050102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b46ea8aa-cd97-425b-90e0-1e70af46b8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":885,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6803","nonce":310950989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6805","nonce":310950989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6809","nonce":310950989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6807","nonce":310950989}]},"public_addr":"172.21.15.50:6803/310950989","cluster_addr":"172.21.15.50:6805/310950989","heartbeat_back_addr":"172.21.15.50:6809/310950989","heartbeat_front_addr":"172.21.15.50:6807/310950989","state":["exists","up"]},{"osd":1,"uuid":"7042f8e0-f5d0-48d2-9cd6-d9445e326eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6811","nonce":3516275849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6813","nonce":3516275849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6817","nonce":3516275849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6815","nonce":3516275849}]},"public_addr":"172.21.15.50:6811/3516275849","cluster_addr":"172.21.15.50:6813/3516275849","heartbeat_back_addr":"172.21.15.50:6817/3516275849","heartbeat_front_addr":"172.21.15.50:6815/3516275849","state":["exists","up"]},{"osd":2,"uuid":"33069daa-7fd8-4e9c-aafe-89030f84aad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":864,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6819","nonce":2835380668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6821","nonce":2835380668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6825","nonce":2835380668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6823","nonce":2835380668}]},"public_addr":"172.21.15.50:6819/2835380668","cluster_addr":"172.21.15.50:6821/2835380668","heartbeat_back_addr":"172.21.15.50:6825/2835380668","heartbeat_front_addr":"172.21.15.50:6823/2835380668","state":["exists","up"]},{"osd":3,"uuid":"43b6aab5-9b42-4f97-a224-b2872de5cb03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6827","nonce":2541695077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6829","nonce":2541695077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6833","nonce":2541695077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6831","nonce":2541695077}]},"public_addr":"172.21.15.50:6827/2541695077","cluster_addr":"172.21.15.50:6829/2541695077","heartbeat_back_addr":"172.21.15.50:6833/2541695077","heartbeat_front_addr":"172.21.15.50:6831/2541695077","state":["exists","up"]},{"osd":4,"uuid":"f7220491-e9bf-4188-b203-322e9908558c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6801","nonce":2207965503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6803","nonce":2207965503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6807","nonce":2207965503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6805","nonce":2207965503}]},"public_addr":"172.21.15.190:6801/2207965503","cluster_addr":"172.21.15.190:6803/2207965503","heartbeat_back_addr":"172.21.15.190:6807/2207965503","heartbeat_front_addr":"172.21.15.190:6805/2207965503","state":["exists","up"]},{"osd":5,"uuid":"ed5c0044-658b-4dfb-8c61-727de6bdd587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6809","nonce":3373187844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6811","nonce":3373187844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6815","nonce":3373187844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6813","nonce":3373187844}]},"public_addr":"172.21.15.190:6809/3373187844","cluster_addr":"172.21.15.190:6811/3373187844","heartbeat_back_addr":"172.21.15.190:6815/3373187844","heartbeat_front_addr":"172.21.15.190:6813/3373187844","state":["exists","up"]},{"osd":6,"uuid":"94111b7d-7887-4a3a-983b-394294735d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":432,"last_clean_end":627,"up_from":746,"up_thru":886,"down_at":629,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6817","nonce":338770817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6819","nonce":338770817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6823","nonce":338770817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6821","nonce":338770817}]},"public_addr":"172.21.15.190:6817/338770817","cluster_addr":"172.21.15.190:6819/338770817","heartbeat_back_addr":"172.21.15.190:6823/338770817","heartbeat_front_addr":"172.21.15.190:6821/338770817","state":["exists","up"]},{"osd":7,"uuid":"abbd95e3-07b4-4235-9101-a4dea0f35bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6825","nonce":2856617796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6827","nonce":2856617796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6831","nonce":2856617796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6829","nonce":2856617796}]},"public_addr":"172.21.15.190:6825/2856617796","cluster_addr":"172.21.15.190:6827/2856617796","heartbeat_back_addr":"172.21.15.190:6831/2856617796","heartbeat_front_addr":"172.21.15.190:6829/2856617796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:08.661446+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:40:44.635897+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:24.801858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:23.056124+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:40:45.124185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:15.222175+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:40:43.683087+0000","dead_epoch":627},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:12.296816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.9","mappings":[{"from":7,"to":2}]},{"pgid":"3.f","mappings":[{"from":4,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/2324264713":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/542146552":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3454723725":"2024-04-27T16:07:37.705234+0000","172.21.15.50:0/2548717623":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2903276422":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6801/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2979962403":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/71201882":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/3973398204":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/1172169336":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3444006067":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/577039843":"2024-04-27T16:18:25.317826+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":1,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":4}]},{"pool":40,"snaps":[{"begin":1,"length":1}]},{"pool":72,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":108,"snaps":[{"begin":1,"length":1}]},{"pool":128,"snaps":[{"begin":1,"length":2}]},{"pool":143,"snaps":[{"begin":3,"length":1}]},{"pool":150,"snaps":[{"begin":2,"length":3}]},{"pool":157,"snaps":[{"begin":2,"length":1}]},{"pool":158,"snaps":[{"begin":2,"length":2}]},{"pool":168,"snaps":[{"begin":2,"length":3}]},{"pool":173,"snaps":[{"begin":3,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":4}]},{"pool":179,"snaps":[{"begin":2,"length":2}]},{"pool":189,"snaps":[{"begin":2,"length":1}]},{"pool":195,"snaps":[{"begin":2,"length":3}]},{"pool":207,"snaps":[{"begin":2,"length":6}]},{"pool":215,"snaps":[{"begin":7,"length":1}]},{"pool":237,"snaps":[{"begin":8,"length":1}]},{"pool":247,"snaps":[{"begin":10,"length":1}]},{"pool":253,"snaps":[{"begin":11,"length":2}]},{"pool":266,"snaps":[{"begin":14,"length":1}]},{"pool":326,"snaps":[{"begin":2,"length":1}]},{"pool":327,"snaps":[{"begin":3,"length":1}]},{"pool":330,"snaps":[{"begin":4,"length":1}]},{"pool":334,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 10 librados: watch_flush enter 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 10 librados: watch_flush exit 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 20 client.41888.objecter shutdown clearing up homeless session... 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 10 client.41888.objecter successfully canceled tick 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2054071f80 msgr2=0x7f2054074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2054071f80 0x7f2054074440 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f2060009750 tx=0x7f2060009310 comp rx=0 tx=0).stop 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 msgr2=0x7f206c1990a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 0x7f206c1990a0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f205c00d980 tx=0x7f205c00dc90 comp rx=0 tx=0).stop 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 shutdown_connections 2024-04-26T16:41:54.179 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f2054071f80 0x7f2054074440 unknown :-1 s=CLOSED pgs=3872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f206c1094c0 0x7f206c19d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f206c103470 0x7f206c1990a0 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 --2- 172.21.15.50:0/3662855182 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f206c103e30 0x7f206c1995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 >> 172.21.15.50:0/3662855182 conn(0x7f206c07a580 msgr2=0x7f206c101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 shutdown_connections 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 -- 172.21.15.50:0/3662855182 wait complete. 2024-04-26T16:41:54.180 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:41:54.182+0000 7f20727da700 1 librados: shutdown 2024-04-26T16:41:54.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:41:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.015656+0000 osd.0 (osd.0) 599 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.015656+0000 osd.0 (osd.0) 599 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.018840+0000 osd.0 (osd.0) 600 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:53.018840+0000 osd.0 (osd.0) 600 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.002403+0000 osd.5 (osd.5) 488 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.002403+0000 osd.5 (osd.5) 488 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:41:55.035 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.004662+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.004662+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: audit 2024-04-26T16:41:54.178173+0000 mon.c (mon.2) 948 : audit [DBG] from='client.? 172.21.15.50:0/3662855182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[21463]: audit 2024-04-26T16:41:54.178173+0000 mon.c (mon.2) 948 : audit [DBG] from='client.? 172.21.15.50:0/3662855182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.015656+0000 osd.0 (osd.0) 599 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.015656+0000 osd.0 (osd.0) 599 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.018840+0000 osd.0 (osd.0) 600 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:53.018840+0000 osd.0 (osd.0) 600 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.002403+0000 osd.5 (osd.5) 488 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:41:55.036 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.002403+0000 osd.5 (osd.5) 488 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:41:55.037 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.004662+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:41:55.037 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.004662+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:41:55.037 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: audit 2024-04-26T16:41:54.178173+0000 mon.c (mon.2) 948 : audit [DBG] from='client.? 172.21.15.50:0/3662855182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:41:55.037 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:54 smithi050 bash[29234]: audit 2024-04-26T16:41:54.178173+0000 mon.c (mon.2) 948 : audit [DBG] from='client.? 172.21.15.50:0/3662855182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:41:55.038 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-26T16:41:55.039 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.015656+0000 osd.0 (osd.0) 599 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.015656+0000 osd.0 (osd.0) 599 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.018840+0000 osd.0 (osd.0) 600 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:53.018840+0000 osd.0 (osd.0) 600 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.002403+0000 osd.5 (osd.5) 488 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.002403+0000 osd.5 (osd.5) 488 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:41:55.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.004662+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:41:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.004662+0000 osd.5 (osd.5) 489 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:41:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: audit 2024-04-26T16:41:54.178173+0000 mon.c (mon.2) 948 : audit [DBG] from='client.? 172.21.15.50:0/3662855182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:41:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:54 smithi190 bash[23868]: audit 2024-04-26T16:41:54.178173+0000 mon.c (mon.2) 948 : audit [DBG] from='client.? 172.21.15.50:0/3662855182' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:41:56.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.107042+0000 mgr.y (mgr.24401) 930 : cluster [DBG] pgmap v1531: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.107042+0000 mgr.y (mgr.24401) 930 : cluster [DBG] pgmap v1531: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.481206+0000 osd.7 (osd.7) 620 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.481206+0000 osd.7 (osd.7) 620 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.483605+0000 osd.7 (osd.7) 621 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.483605+0000 osd.7 (osd.7) 621 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.740345+0000 osd.4 (osd.4) 535 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.740345+0000 osd.4 (osd.4) 535 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.746413+0000 osd.4 (osd.4) 536 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.746413+0000 osd.4 (osd.4) 536 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.968158+0000 osd.3 (osd.3) 635 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.968158+0000 osd.3 (osd.3) 635 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:41:56.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.970632+0000 osd.3 (osd.3) 636 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:41:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:54.970632+0000 osd.3 (osd.3) 636 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:41:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.062212+0000 osd.2 (osd.2) 525 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:41:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.062212+0000 osd.2 (osd.2) 525 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:41:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.064508+0000 osd.2 (osd.2) 526 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:41:56.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:55 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.064508+0000 osd.2 (osd.2) 526 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:41:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.107042+0000 mgr.y (mgr.24401) 930 : cluster [DBG] pgmap v1531: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.107042+0000 mgr.y (mgr.24401) 930 : cluster [DBG] pgmap v1531: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.481206+0000 osd.7 (osd.7) 620 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:41:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.481206+0000 osd.7 (osd.7) 620 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:41:56.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.483605+0000 osd.7 (osd.7) 621 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.483605+0000 osd.7 (osd.7) 621 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.740345+0000 osd.4 (osd.4) 535 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.740345+0000 osd.4 (osd.4) 535 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.746413+0000 osd.4 (osd.4) 536 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.746413+0000 osd.4 (osd.4) 536 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.968158+0000 osd.3 (osd.3) 635 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.968158+0000 osd.3 (osd.3) 635 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.970632+0000 osd.3 (osd.3) 636 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:54.970632+0000 osd.3 (osd.3) 636 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.062212+0000 osd.2 (osd.2) 525 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.062212+0000 osd.2 (osd.2) 525 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.064508+0000 osd.2 (osd.2) 526 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:41:56.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.064508+0000 osd.2 (osd.2) 526 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.107042+0000 mgr.y (mgr.24401) 930 : cluster [DBG] pgmap v1531: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.107042+0000 mgr.y (mgr.24401) 930 : cluster [DBG] pgmap v1531: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.481206+0000 osd.7 (osd.7) 620 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.481206+0000 osd.7 (osd.7) 620 : cluster [DBG] 2.1d scrub starts 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.483605+0000 osd.7 (osd.7) 621 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.483605+0000 osd.7 (osd.7) 621 : cluster [DBG] 2.1d scrub ok 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.740345+0000 osd.4 (osd.4) 535 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.740345+0000 osd.4 (osd.4) 535 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.746413+0000 osd.4 (osd.4) 536 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.746413+0000 osd.4 (osd.4) 536 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:41:56.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.968158+0000 osd.3 (osd.3) 635 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.968158+0000 osd.3 (osd.3) 635 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.970632+0000 osd.3 (osd.3) 636 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:54.970632+0000 osd.3 (osd.3) 636 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.062212+0000 osd.2 (osd.2) 525 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.062212+0000 osd.2 (osd.2) 525 : cluster [DBG] 2.e deep-scrub starts 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.064508+0000 osd.2 (osd.2) 526 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:41:56.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:55 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.064508+0000 osd.2 (osd.2) 526 : cluster [DBG] 2.e deep-scrub ok 2024-04-26T16:41:57.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:41:57.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.947855+0000 osd.1 (osd.1) 571 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:41:57.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.947855+0000 osd.1 (osd.1) 571 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:41:57.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.950183+0000 osd.1 (osd.1) 572 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.950183+0000 osd.1 (osd.1) 572 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.997546+0000 osd.3 (osd.3) 637 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.997546+0000 osd.3 (osd.3) 637 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.999973+0000 osd.3 (osd.3) 638 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:55.999973+0000 osd.3 (osd.3) 638 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.009012+0000 osd.0 (osd.0) 601 : cluster [DBG] 6.0 deep-scrub starts 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.009012+0000 osd.0 (osd.0) 601 : cluster [DBG] 6.0 deep-scrub starts 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.011716+0000 osd.0 (osd.0) 602 : cluster [DBG] 6.0 deep-scrub ok 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.011716+0000 osd.0 (osd.0) 602 : cluster [DBG] 6.0 deep-scrub ok 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.107798+0000 mgr.y (mgr.24401) 931 : cluster [DBG] pgmap v1532: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.107798+0000 mgr.y (mgr.24401) 931 : cluster [DBG] pgmap v1532: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.976672+0000 osd.3 (osd.3) 639 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.976672+0000 osd.3 (osd.3) 639 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:41:57.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.981388+0000 osd.3 (osd.3) 640 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:41:57.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:57 smithi190 bash[23868]: cluster 2024-04-26T16:41:56.981388+0000 osd.3 (osd.3) 640 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.947855+0000 osd.1 (osd.1) 571 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.947855+0000 osd.1 (osd.1) 571 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.950183+0000 osd.1 (osd.1) 572 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.950183+0000 osd.1 (osd.1) 572 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.997546+0000 osd.3 (osd.3) 637 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.997546+0000 osd.3 (osd.3) 637 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.999973+0000 osd.3 (osd.3) 638 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:55.999973+0000 osd.3 (osd.3) 638 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:41:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.009012+0000 osd.0 (osd.0) 601 : cluster [DBG] 6.0 deep-scrub starts 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.009012+0000 osd.0 (osd.0) 601 : cluster [DBG] 6.0 deep-scrub starts 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.011716+0000 osd.0 (osd.0) 602 : cluster [DBG] 6.0 deep-scrub ok 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.011716+0000 osd.0 (osd.0) 602 : cluster [DBG] 6.0 deep-scrub ok 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.107798+0000 mgr.y (mgr.24401) 931 : cluster [DBG] pgmap v1532: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.107798+0000 mgr.y (mgr.24401) 931 : cluster [DBG] pgmap v1532: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.976672+0000 osd.3 (osd.3) 639 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.976672+0000 osd.3 (osd.3) 639 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.981388+0000 osd.3 (osd.3) 640 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[21463]: cluster 2024-04-26T16:41:56.981388+0000 osd.3 (osd.3) 640 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.947855+0000 osd.1 (osd.1) 571 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:41:58.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.947855+0000 osd.1 (osd.1) 571 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.950183+0000 osd.1 (osd.1) 572 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.950183+0000 osd.1 (osd.1) 572 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.997546+0000 osd.3 (osd.3) 637 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.997546+0000 osd.3 (osd.3) 637 : cluster [DBG] 6.11 scrub starts 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.999973+0000 osd.3 (osd.3) 638 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:55.999973+0000 osd.3 (osd.3) 638 : cluster [DBG] 6.11 scrub ok 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.009012+0000 osd.0 (osd.0) 601 : cluster [DBG] 6.0 deep-scrub starts 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.009012+0000 osd.0 (osd.0) 601 : cluster [DBG] 6.0 deep-scrub starts 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.011716+0000 osd.0 (osd.0) 602 : cluster [DBG] 6.0 deep-scrub ok 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.011716+0000 osd.0 (osd.0) 602 : cluster [DBG] 6.0 deep-scrub ok 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.107798+0000 mgr.y (mgr.24401) 931 : cluster [DBG] pgmap v1532: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.107798+0000 mgr.y (mgr.24401) 931 : cluster [DBG] pgmap v1532: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 899 B/s rd, 0 op/s 2024-04-26T16:41:58.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.976672+0000 osd.3 (osd.3) 639 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:41:58.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.976672+0000 osd.3 (osd.3) 639 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:41:58.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.981388+0000 osd.3 (osd.3) 640 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:41:58.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:57 smithi050 bash[29234]: cluster 2024-04-26T16:41:56.981388+0000 osd.3 (osd.3) 640 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:41:59.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:57.151015+0000 osd.6 (osd.6) 135 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:41:59.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:57.151015+0000 osd.6 (osd.6) 135 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:41:59.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:57.154209+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:41:59.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:57.154209+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: audit 2024-04-26T16:41:57.577359+0000 mgr.y (mgr.24401) 932 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: audit 2024-04-26T16:41:57.577359+0000 mgr.y (mgr.24401) 932 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.030844+0000 osd.5 (osd.5) 490 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.030844+0000 osd.5 (osd.5) 490 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.033130+0000 osd.5 (osd.5) 491 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.033130+0000 osd.5 (osd.5) 491 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: audit 2024-04-26T16:41:58.570046+0000 mon.c (mon.2) 949 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:58 smithi190 bash[23868]: audit 2024-04-26T16:41:58.570046+0000 mon.c (mon.2) 949 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:57.151015+0000 osd.6 (osd.6) 135 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:57.151015+0000 osd.6 (osd.6) 135 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:57.154209+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:57.154209+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: audit 2024-04-26T16:41:57.577359+0000 mgr.y (mgr.24401) 932 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: audit 2024-04-26T16:41:57.577359+0000 mgr.y (mgr.24401) 932 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.030844+0000 osd.5 (osd.5) 490 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.030844+0000 osd.5 (osd.5) 490 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.033130+0000 osd.5 (osd.5) 491 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.033130+0000 osd.5 (osd.5) 491 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: audit 2024-04-26T16:41:58.570046+0000 mon.c (mon.2) 949 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:59.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[29234]: audit 2024-04-26T16:41:58.570046+0000 mon.c (mon.2) 949 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:57.151015+0000 osd.6 (osd.6) 135 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:57.151015+0000 osd.6 (osd.6) 135 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:57.154209+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:57.154209+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: audit 2024-04-26T16:41:57.577359+0000 mgr.y (mgr.24401) 932 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: audit 2024-04-26T16:41:57.577359+0000 mgr.y (mgr.24401) 932 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.030844+0000 osd.5 (osd.5) 490 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.030844+0000 osd.5 (osd.5) 490 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.033130+0000 osd.5 (osd.5) 491 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:41:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.033130+0000 osd.5 (osd.5) 491 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:41:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: audit 2024-04-26T16:41:58.570046+0000 mon.c (mon.2) 949 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:59.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:58 smithi050 bash[21463]: audit 2024-04-26T16:41:58.570046+0000 mon.c (mon.2) 949 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:41:59.839 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:00.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.029891+0000 osd.0 (osd.0) 603 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.029891+0000 osd.0 (osd.0) 603 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.032363+0000 osd.0 (osd.0) 604 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.032363+0000 osd.0 (osd.0) 604 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.109462+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 deep-scrub starts 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.109462+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 deep-scrub starts 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.109726+0000 mgr.y (mgr.24401) 933 : cluster [DBG] pgmap v1533: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.109726+0000 mgr.y (mgr.24401) 933 : cluster [DBG] pgmap v1533: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.112396+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 deep-scrub ok 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.112396+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 deep-scrub ok 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.414138+0000 osd.7 (osd.7) 622 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.414138+0000 osd.7 (osd.7) 622 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:42:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.420167+0000 osd.7 (osd.7) 623 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:42:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.420167+0000 osd.7 (osd.7) 623 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:42:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.952381+0000 osd.3 (osd.3) 641 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:42:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.952381+0000 osd.3 (osd.3) 641 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:42:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.954923+0000 osd.3 (osd.3) 642 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:42:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:41:59 smithi190 bash[23868]: cluster 2024-04-26T16:41:58.954923+0000 osd.3 (osd.3) 642 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.029891+0000 osd.0 (osd.0) 603 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.029891+0000 osd.0 (osd.0) 603 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.032363+0000 osd.0 (osd.0) 604 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.032363+0000 osd.0 (osd.0) 604 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.109462+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 deep-scrub starts 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.109462+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 deep-scrub starts 2024-04-26T16:42:00.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.109726+0000 mgr.y (mgr.24401) 933 : cluster [DBG] pgmap v1533: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.109726+0000 mgr.y (mgr.24401) 933 : cluster [DBG] pgmap v1533: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.112396+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 deep-scrub ok 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.112396+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 deep-scrub ok 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.414138+0000 osd.7 (osd.7) 622 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.414138+0000 osd.7 (osd.7) 622 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.420167+0000 osd.7 (osd.7) 623 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.420167+0000 osd.7 (osd.7) 623 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.952381+0000 osd.3 (osd.3) 641 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.952381+0000 osd.3 (osd.3) 641 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.954923+0000 osd.3 (osd.3) 642 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:42:00.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[21463]: cluster 2024-04-26T16:41:58.954923+0000 osd.3 (osd.3) 642 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.029891+0000 osd.0 (osd.0) 603 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.029891+0000 osd.0 (osd.0) 603 : cluster [DBG] 6.10 scrub starts 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.032363+0000 osd.0 (osd.0) 604 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.032363+0000 osd.0 (osd.0) 604 : cluster [DBG] 6.10 scrub ok 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.109462+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 deep-scrub starts 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.109462+0000 osd.6 (osd.6) 137 : cluster [DBG] 1.6 deep-scrub starts 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.109726+0000 mgr.y (mgr.24401) 933 : cluster [DBG] pgmap v1533: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.109726+0000 mgr.y (mgr.24401) 933 : cluster [DBG] pgmap v1533: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.112396+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 deep-scrub ok 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.112396+0000 osd.6 (osd.6) 138 : cluster [DBG] 1.6 deep-scrub ok 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.414138+0000 osd.7 (osd.7) 622 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.414138+0000 osd.7 (osd.7) 622 : cluster [DBG] 11.17 scrub starts 2024-04-26T16:42:00.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.420167+0000 osd.7 (osd.7) 623 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:42:00.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.420167+0000 osd.7 (osd.7) 623 : cluster [DBG] 11.17 scrub ok 2024-04-26T16:42:00.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.952381+0000 osd.3 (osd.3) 641 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:42:00.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.952381+0000 osd.3 (osd.3) 641 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:42:00.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.954923+0000 osd.3 (osd.3) 642 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:42:00.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:41:59 smithi050 bash[29234]: cluster 2024-04-26T16:41:58.954923+0000 osd.3 (osd.3) 642 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:42:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.905054+0000 osd.3 (osd.3) 643 : cluster [DBG] 6.b scrub starts 2024-04-26T16:42:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.905054+0000 osd.3 (osd.3) 643 : cluster [DBG] 6.b scrub starts 2024-04-26T16:42:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.907443+0000 osd.3 (osd.3) 644 : cluster [DBG] 6.b scrub ok 2024-04-26T16:42:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.907443+0000 osd.3 (osd.3) 644 : cluster [DBG] 6.b scrub ok 2024-04-26T16:42:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.053431+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:42:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.053431+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:42:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.059124+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:42:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.059124+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:42:01.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.905054+0000 osd.3 (osd.3) 643 : cluster [DBG] 6.b scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.905054+0000 osd.3 (osd.3) 643 : cluster [DBG] 6.b scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.907443+0000 osd.3 (osd.3) 644 : cluster [DBG] 6.b scrub ok 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.907443+0000 osd.3 (osd.3) 644 : cluster [DBG] 6.b scrub ok 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.053431+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.053431+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.059124+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.059124+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.905054+0000 osd.3 (osd.3) 643 : cluster [DBG] 6.b scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.905054+0000 osd.3 (osd.3) 643 : cluster [DBG] 6.b scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.907443+0000 osd.3 (osd.3) 644 : cluster [DBG] 6.b scrub ok 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.907443+0000 osd.3 (osd.3) 644 : cluster [DBG] 6.b scrub ok 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.053431+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:42:01.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.053431+0000 osd.5 (osd.5) 492 : cluster [DBG] 3.5 scrub starts 2024-04-26T16:42:01.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.059124+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:42:01.077 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.059124+0000 osd.5 (osd.5) 493 : cluster [DBG] 3.5 scrub ok 2024-04-26T16:42:02.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.968500+0000 osd.1 (osd.1) 573 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.968500+0000 osd.1 (osd.1) 573 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.970496+0000 osd.1 (osd.1) 574 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:41:59.970496+0000 osd.1 (osd.1) 574 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.110692+0000 mgr.y (mgr.24401) 934 : cluster [DBG] pgmap v1534: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.110692+0000 mgr.y (mgr.24401) 934 : cluster [DBG] pgmap v1534: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.117916+0000 osd.6 (osd.6) 139 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.117916+0000 osd.6 (osd.6) 139 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.120174+0000 osd.6 (osd.6) 140 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:00.120174+0000 osd.6 (osd.6) 140 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.031630+0000 osd.2 (osd.2) 527 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.031630+0000 osd.2 (osd.2) 527 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.034183+0000 osd.2 (osd.2) 528 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:42:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:01 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.034183+0000 osd.2 (osd.2) 528 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.968500+0000 osd.1 (osd.1) 573 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.968500+0000 osd.1 (osd.1) 573 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.970496+0000 osd.1 (osd.1) 574 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:41:59.970496+0000 osd.1 (osd.1) 574 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.110692+0000 mgr.y (mgr.24401) 934 : cluster [DBG] pgmap v1534: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.110692+0000 mgr.y (mgr.24401) 934 : cluster [DBG] pgmap v1534: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.117916+0000 osd.6 (osd.6) 139 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.117916+0000 osd.6 (osd.6) 139 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.120174+0000 osd.6 (osd.6) 140 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:00.120174+0000 osd.6 (osd.6) 140 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.031630+0000 osd.2 (osd.2) 527 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:42:02.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.031630+0000 osd.2 (osd.2) 527 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.034183+0000 osd.2 (osd.2) 528 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.034183+0000 osd.2 (osd.2) 528 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.968500+0000 osd.1 (osd.1) 573 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.968500+0000 osd.1 (osd.1) 573 : cluster [DBG] 5.1d scrub starts 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.970496+0000 osd.1 (osd.1) 574 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:41:59.970496+0000 osd.1 (osd.1) 574 : cluster [DBG] 5.1d scrub ok 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.110692+0000 mgr.y (mgr.24401) 934 : cluster [DBG] pgmap v1534: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.110692+0000 mgr.y (mgr.24401) 934 : cluster [DBG] pgmap v1534: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.117916+0000 osd.6 (osd.6) 139 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.117916+0000 osd.6 (osd.6) 139 : cluster [DBG] 2.17 deep-scrub starts 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.120174+0000 osd.6 (osd.6) 140 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:42:02.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:00.120174+0000 osd.6 (osd.6) 140 : cluster [DBG] 2.17 deep-scrub ok 2024-04-26T16:42:02.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.031630+0000 osd.2 (osd.2) 527 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:42:02.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.031630+0000 osd.2 (osd.2) 527 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:42:02.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.034183+0000 osd.2 (osd.2) 528 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:42:02.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:01 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.034183+0000 osd.2 (osd.2) 528 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:42:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:02 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.168228+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:42:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:02 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.168228+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:42:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:02 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.170835+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:42:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:02 smithi190 bash[23868]: cluster 2024-04-26T16:42:01.170835+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:42:03.108 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.168228+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:42:03.108 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.168228+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:42:03.108 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.170835+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:42:03.108 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[21463]: cluster 2024-04-26T16:42:01.170835+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:42:03.109 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.168228+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:42:03.109 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.168228+0000 osd.6 (osd.6) 141 : cluster [DBG] 2.1a scrub starts 2024-04-26T16:42:03.109 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.170835+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:42:03.109 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:02 smithi050 bash[29234]: cluster 2024-04-26T16:42:01.170835+0000 osd.6 (osd.6) 142 : cluster [DBG] 2.1a scrub ok 2024-04-26T16:42:03.295 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- 172.21.15.50:0/53811023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 msgr2=0x7f942406bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- 172.21.15.50:0/53811023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f942406bfe0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f9414009990 tx=0x7f9414009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- 172.21.15.50:0/53811023 shutdown_connections 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- 172.21.15.50:0/53811023 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94241095c0 0x7f94241099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- 172.21.15.50:0/53811023 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 0x7f9424110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- 172.21.15.50:0/53811023 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f942406bfe0 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- 172.21.15.50:0/53811023 >> 172.21.15.50:0/53811023 conn(0x7f9424079120 msgr2=0x7f9424079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- 172.21.15.50:0/53811023 shutdown_connections 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- 172.21.15.50:0/53811023 wait complete. 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 librados: starting msgr at 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 librados: starting objecter 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 Processor -- start 2024-04-26T16:42:03.296 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- start start 2024-04-26T16:42:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 librados: setting wanted keys 2024-04-26T16:42:03.297 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 librados: calling monclient init 2024-04-26T16:42:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f94241026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 0x7f9424102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94241095c0 0x7f9424103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f9424112b50 con 0x7f942406bb60 2024-04-26T16:42:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f94241129d0 con 0x7f94241095c0 2024-04-26T16:42:03.298 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f9424112cd0 con 0x7f942406c520 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f94241026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429629700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 0x7f9424102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f94241026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:50296/0 (socket says 172.21.15.50:50296) 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 -- 172.21.15.50:0/1178303145 learned_addr learned my addr 172.21.15.50:0/1178303145 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942a62b700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94241095c0 0x7f9424103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 -- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 msgr2=0x7f9424102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 0x7f9424102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 -- 172.21.15.50:0/1178303145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94241095c0 msgr2=0x7f9424103170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94241095c0 0x7f9424103170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 -- 172.21.15.50:0/1178303145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94241a7660 con 0x7f942406bb60 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429629700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 0x7f9424102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f9429e2a700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f94241026f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f942000d950 tx=0x7f942000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f941affd700 10 client.?.objecter ms_handle_connect 0x7f942406bb60 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f941affd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:03.299 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420016070 con 0x7f942406bb60 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94241a7950 con 0x7f942406bb60 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9420009da0 con 0x7f942406bb60 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.298+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9420010e60 con 0x7f942406bb60 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f942c08e700 10 client.31275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94241a7eb0 con 0x7f942406bb60 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f942c08e700 1 librados: init done 2024-04-26T16:42:03.300 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f942406a110 con 0x7f942406bb60 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9420019070 con 0x7f942406bb60 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9410071f80 0x7f9410074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(924..924 src has 874..924) ==== 8707+0+0 (secure 0 0 0) 0x7f94200912a0 con 0x7f942406bb60 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 10 client.31275.objecter ms_dispatch 0x7f9424001550 osd_map(924..924 src has 874..924) 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 3 client.31275.objecter handle_osd_map got epochs [924,924] > 0 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 3 client.31275.objecter handle_osd_map decoding full epoch 924 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f9429629700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9410071f80 0x7f9410074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 20 client.31275.objecter dump_active .. 0 homeless 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f9429629700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9410071f80 0x7f9410074440 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7f94140098e0 tx=0x7f9414005960 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:03.301 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.302+0000 7f941affd700 10 client.31275.objecter ms_handle_connect 0x7f9410071f80 2024-04-26T16:42:03.304 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.306+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f942005cf70 con 0x7f942406bb60 2024-04-26T16:42:03.527 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.530+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9424103c20 con 0x7f942406bb60 2024-04-26T16:42:03.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f941affd700 1 -- 172.21.15.50:0/1178303145 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v924) ==== 74+0+26252 (secure 0 0 0) 0x7f9420060c20 con 0x7f942406bb60 2024-04-26T16:42:03.531 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:42:03.531 INFO:teuthology.orchestra.run.smithi050.stdout:{"epoch":924,"fsid":"cf170a7a-03e6-11ef-bc93-c7b262605968","created":"2024-04-26T16:07:04.494213+0000","modified":"2024-04-26T16:41:44.720694+0000","last_up_change":"2024-04-26T16:35:24.350639+0000","last_in_change":"2024-04-26T16:32:08.394374+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":112,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":345,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T16:12:30.809607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"889","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"881","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-04-26T16:17:27.029484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-04-26T16:17:29.124965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"datapool","create_time":"2024-04-26T16:17:30.749319+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":89,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-04-26T16:17:31.064350+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-04-26T16:17:33.192127+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-04-26T16:23:28.050102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"142","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b46ea8aa-cd97-425b-90e0-1e70af46b8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":885,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6802","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6803","nonce":310950989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6804","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6805","nonce":310950989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6808","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6809","nonce":310950989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6806","nonce":310950989},{"type":"v1","addr":"172.21.15.50:6807","nonce":310950989}]},"public_addr":"172.21.15.50:6803/310950989","cluster_addr":"172.21.15.50:6805/310950989","heartbeat_back_addr":"172.21.15.50:6809/310950989","heartbeat_front_addr":"172.21.15.50:6807/310950989","state":["exists","up"]},{"osd":1,"uuid":"7042f8e0-f5d0-48d2-9cd6-d9445e326eed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6810","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6811","nonce":3516275849}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6812","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6813","nonce":3516275849}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6816","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6817","nonce":3516275849}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6814","nonce":3516275849},{"type":"v1","addr":"172.21.15.50:6815","nonce":3516275849}]},"public_addr":"172.21.15.50:6811/3516275849","cluster_addr":"172.21.15.50:6813/3516275849","heartbeat_back_addr":"172.21.15.50:6817/3516275849","heartbeat_front_addr":"172.21.15.50:6815/3516275849","state":["exists","up"]},{"osd":2,"uuid":"33069daa-7fd8-4e9c-aafe-89030f84aad2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":864,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6818","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6819","nonce":2835380668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6820","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6821","nonce":2835380668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6824","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6825","nonce":2835380668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6822","nonce":2835380668},{"type":"v1","addr":"172.21.15.50:6823","nonce":2835380668}]},"public_addr":"172.21.15.50:6819/2835380668","cluster_addr":"172.21.15.50:6821/2835380668","heartbeat_back_addr":"172.21.15.50:6825/2835380668","heartbeat_front_addr":"172.21.15.50:6823/2835380668","state":["exists","up"]},{"osd":3,"uuid":"43b6aab5-9b42-4f97-a224-b2872de5cb03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6826","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6827","nonce":2541695077}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6828","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6829","nonce":2541695077}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6832","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6833","nonce":2541695077}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.50:6830","nonce":2541695077},{"type":"v1","addr":"172.21.15.50:6831","nonce":2541695077}]},"public_addr":"172.21.15.50:6827/2541695077","cluster_addr":"172.21.15.50:6829/2541695077","heartbeat_back_addr":"172.21.15.50:6833/2541695077","heartbeat_front_addr":"172.21.15.50:6831/2541695077","state":["exists","up"]},{"osd":4,"uuid":"f7220491-e9bf-4188-b203-322e9908558c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6801","nonce":2207965503}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6803","nonce":2207965503}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6807","nonce":2207965503}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":2207965503},{"type":"v1","addr":"172.21.15.190:6805","nonce":2207965503}]},"public_addr":"172.21.15.190:6801/2207965503","cluster_addr":"172.21.15.190:6803/2207965503","heartbeat_back_addr":"172.21.15.190:6807/2207965503","heartbeat_front_addr":"172.21.15.190:6805/2207965503","state":["exists","up"]},{"osd":5,"uuid":"ed5c0044-658b-4dfb-8c61-727de6bdd587","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6809","nonce":3373187844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6811","nonce":3373187844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6815","nonce":3373187844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":3373187844},{"type":"v1","addr":"172.21.15.190:6813","nonce":3373187844}]},"public_addr":"172.21.15.190:6809/3373187844","cluster_addr":"172.21.15.190:6811/3373187844","heartbeat_back_addr":"172.21.15.190:6815/3373187844","heartbeat_front_addr":"172.21.15.190:6813/3373187844","state":["exists","up"]},{"osd":6,"uuid":"94111b7d-7887-4a3a-983b-394294735d0e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":432,"last_clean_end":627,"up_from":746,"up_thru":886,"down_at":629,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6817","nonce":338770817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6819","nonce":338770817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6823","nonce":338770817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":338770817},{"type":"v1","addr":"172.21.15.190:6821","nonce":338770817}]},"public_addr":"172.21.15.190:6817/338770817","cluster_addr":"172.21.15.190:6819/338770817","heartbeat_back_addr":"172.21.15.190:6823/338770817","heartbeat_front_addr":"172.21.15.190:6821/338770817","state":["exists","up"]},{"osd":7,"uuid":"abbd95e3-07b4-4235-9101-a4dea0f35bb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":886,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6825","nonce":2856617796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6827","nonce":2856617796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6831","nonce":2856617796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2856617796},{"type":"v1","addr":"172.21.15.190:6829","nonce":2856617796}]},"public_addr":"172.21.15.190:6825/2856617796","cluster_addr":"172.21.15.190:6827/2856617796","heartbeat_back_addr":"172.21.15.190:6831/2856617796","heartbeat_front_addr":"172.21.15.190:6829/2856617796","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:08.661446+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:40:44.635897+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:24.801858+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:23.056124+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:40:45.124185+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:15.222175+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:40:43.683087+0000","dead_epoch":627},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T16:41:12.296816+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.9","mappings":[{"from":7,"to":2}]},{"pgid":"3.f","mappings":[{"from":4,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.50:0/2324264713":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/542146552":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/127073210":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:6801/126446279":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3454723725":"2024-04-27T16:07:37.705234+0000","172.21.15.50:0/2548717623":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6800/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2903276422":"2024-04-27T16:07:37.705234+0000","172.21.15.50:6801/685732569":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/2979962403":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/71201882":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/3973398204":"2024-04-27T16:08:14.593126+0000","172.21.15.50:0/1172169336":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/3444006067":"2024-04-27T16:18:25.317826+0000","172.21.15.50:0/577039843":"2024-04-27T16:18:25.317826+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":16,"snaps":[{"begin":1,"length":4}]},{"pool":20,"snaps":[{"begin":3,"length":4}]},{"pool":40,"snaps":[{"begin":1,"length":1}]},{"pool":72,"snaps":[{"begin":1,"length":1}]},{"pool":99,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":108,"snaps":[{"begin":1,"length":1}]},{"pool":128,"snaps":[{"begin":1,"length":2}]},{"pool":143,"snaps":[{"begin":3,"length":1}]},{"pool":150,"snaps":[{"begin":2,"length":3}]},{"pool":157,"snaps":[{"begin":2,"length":1}]},{"pool":158,"snaps":[{"begin":2,"length":2}]},{"pool":168,"snaps":[{"begin":2,"length":3}]},{"pool":173,"snaps":[{"begin":3,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":4}]},{"pool":179,"snaps":[{"begin":2,"length":2}]},{"pool":189,"snaps":[{"begin":2,"length":1}]},{"pool":195,"snaps":[{"begin":2,"length":3}]},{"pool":207,"snaps":[{"begin":2,"length":6}]},{"pool":215,"snaps":[{"begin":7,"length":1}]},{"pool":237,"snaps":[{"begin":8,"length":1}]},{"pool":247,"snaps":[{"begin":10,"length":1}]},{"pool":253,"snaps":[{"begin":11,"length":2}]},{"pool":266,"snaps":[{"begin":14,"length":1}]},{"pool":326,"snaps":[{"begin":2,"length":1}]},{"pool":327,"snaps":[{"begin":3,"length":1}]},{"pool":330,"snaps":[{"begin":4,"length":1}]},{"pool":334,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 10 librados: watch_flush enter 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 10 librados: watch_flush exit 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 20 client.31275.objecter shutdown clearing up homeless session... 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 10 client.31275.objecter successfully canceled tick 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9410071f80 msgr2=0x7f9410074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9410071f80 0x7f9410074440 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7f94140098e0 tx=0x7f9414005960 comp rx=0 tx=0).stop 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 msgr2=0x7f94241026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:03.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f94241026f0 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f942000d950 tx=0x7f942000dc60 comp rx=0 tx=0).stop 2024-04-26T16:42:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 shutdown_connections 2024-04-26T16:42:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f9410071f80 0x7f9410074440 unknown :-1 s=CLOSED pgs=3873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f94241095c0 0x7f9424103170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f942406bb60 0x7f94241026f0 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.534+0000 7f942c08e700 1 --2- 172.21.15.50:0/1178303145 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f942406c520 0x7f9424102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:03.534 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.538+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 >> 172.21.15.50:0/1178303145 conn(0x7f9424079120 msgr2=0x7f94241070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:03.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.538+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 shutdown_connections 2024-04-26T16:42:03.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.538+0000 7f942c08e700 1 -- 172.21.15.50:0/1178303145 wait complete. 2024-04-26T16:42:03.535 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:03.538+0000 7f942c08e700 1 librados: shutdown 2024-04-26T16:42:04.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.112003+0000 mgr.y (mgr.24401) 935 : cluster [DBG] pgmap v1535: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:04.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.112003+0000 mgr.y (mgr.24401) 935 : cluster [DBG] pgmap v1535: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.299067+0000 osd.7 (osd.7) 624 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.299067+0000 osd.7 (osd.7) 624 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.301487+0000 osd.7 (osd.7) 625 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.301487+0000 osd.7 (osd.7) 625 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.903744+0000 osd.3 (osd.3) 645 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.903744+0000 osd.3 (osd.3) 645 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.906100+0000 osd.3 (osd.3) 646 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.906100+0000 osd.3 (osd.3) 646 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: audit 2024-04-26T16:42:03.533401+0000 mon.a (mon.0) 4064 : audit [DBG] from='client.? 172.21.15.50:0/1178303145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:42:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:03 smithi190 bash[23868]: audit 2024-04-26T16:42:03.533401+0000 mon.a (mon.0) 4064 : audit [DBG] from='client.? 172.21.15.50:0/1178303145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.112003+0000 mgr.y (mgr.24401) 935 : cluster [DBG] pgmap v1535: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.112003+0000 mgr.y (mgr.24401) 935 : cluster [DBG] pgmap v1535: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.299067+0000 osd.7 (osd.7) 624 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.299067+0000 osd.7 (osd.7) 624 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.301487+0000 osd.7 (osd.7) 625 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.301487+0000 osd.7 (osd.7) 625 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.903744+0000 osd.3 (osd.3) 645 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:42:04.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.903744+0000 osd.3 (osd.3) 645 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.906100+0000 osd.3 (osd.3) 646 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.906100+0000 osd.3 (osd.3) 646 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: audit 2024-04-26T16:42:03.533401+0000 mon.a (mon.0) 4064 : audit [DBG] from='client.? 172.21.15.50:0/1178303145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[21463]: audit 2024-04-26T16:42:03.533401+0000 mon.a (mon.0) 4064 : audit [DBG] from='client.? 172.21.15.50:0/1178303145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.112003+0000 mgr.y (mgr.24401) 935 : cluster [DBG] pgmap v1535: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.112003+0000 mgr.y (mgr.24401) 935 : cluster [DBG] pgmap v1535: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.299067+0000 osd.7 (osd.7) 624 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.299067+0000 osd.7 (osd.7) 624 : cluster [DBG] 5.1a scrub starts 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.301487+0000 osd.7 (osd.7) 625 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:42:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.301487+0000 osd.7 (osd.7) 625 : cluster [DBG] 5.1a scrub ok 2024-04-26T16:42:04.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.903744+0000 osd.3 (osd.3) 645 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:42:04.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.903744+0000 osd.3 (osd.3) 645 : cluster [DBG] 5.17 scrub starts 2024-04-26T16:42:04.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.906100+0000 osd.3 (osd.3) 646 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:42:04.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.906100+0000 osd.3 (osd.3) 646 : cluster [DBG] 5.17 scrub ok 2024-04-26T16:42:04.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: audit 2024-04-26T16:42:03.533401+0000 mon.a (mon.0) 4064 : audit [DBG] from='client.? 172.21.15.50:0/1178303145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:42:04.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:03 smithi050 bash[29234]: audit 2024-04-26T16:42:03.533401+0000 mon.a (mon.0) 4064 : audit [DBG] from='client.? 172.21.15.50:0/1178303145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-26T16:42:04.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-26T16:42:04.348 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-26T16:42:04.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-26T16:42:04.349 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-26T16:42:04.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-26T16:42:04.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-26T16:42:04.350 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-26T16:42:04.351 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-26T16:42:04.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:42:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:42:05.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.985826+0000 osd.1 (osd.1) 575 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.985826+0000 osd.1 (osd.1) 575 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.988286+0000 osd.1 (osd.1) 576 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:02.988286+0000 osd.1 (osd.1) 576 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.019362+0000 osd.5 (osd.5) 494 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.019362+0000 osd.5 (osd.5) 494 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.022261+0000 osd.5 (osd.5) 495 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.022261+0000 osd.5 (osd.5) 495 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.985826+0000 osd.1 (osd.1) 575 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.985826+0000 osd.1 (osd.1) 575 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:42:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.988286+0000 osd.1 (osd.1) 576 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:42:05.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:02.988286+0000 osd.1 (osd.1) 576 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:42:05.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.019362+0000 osd.5 (osd.5) 494 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:42:05.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.019362+0000 osd.5 (osd.5) 494 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:42:05.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.022261+0000 osd.5 (osd.5) 495 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:42:05.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:04 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.022261+0000 osd.5 (osd.5) 495 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:42:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.985826+0000 osd.1 (osd.1) 575 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:42:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.985826+0000 osd.1 (osd.1) 575 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:42:05.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.988286+0000 osd.1 (osd.1) 576 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:42:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:02.988286+0000 osd.1 (osd.1) 576 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:42:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.019362+0000 osd.5 (osd.5) 494 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:42:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.019362+0000 osd.5 (osd.5) 494 : cluster [DBG] 11.16 scrub starts 2024-04-26T16:42:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.022261+0000 osd.5 (osd.5) 495 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:42:05.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:04 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.022261+0000 osd.5 (osd.5) 495 : cluster [DBG] 11.16 scrub ok 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.112902+0000 mgr.y (mgr.24401) 936 : cluster [DBG] pgmap v1536: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.112902+0000 mgr.y (mgr.24401) 936 : cluster [DBG] pgmap v1536: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.138194+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.138194+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.141588+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.141588+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.985200+0000 osd.3 (osd.3) 647 : cluster [DBG] 1.4 deep-scrub starts 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.985200+0000 osd.3 (osd.3) 647 : cluster [DBG] 1.4 deep-scrub starts 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.989113+0000 osd.3 (osd.3) 648 : cluster [DBG] 1.4 deep-scrub ok 2024-04-26T16:42:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[21463]: cluster 2024-04-26T16:42:04.989113+0000 osd.3 (osd.3) 648 : cluster [DBG] 1.4 deep-scrub ok 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.112902+0000 mgr.y (mgr.24401) 936 : cluster [DBG] pgmap v1536: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.112902+0000 mgr.y (mgr.24401) 936 : cluster [DBG] pgmap v1536: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.138194+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.138194+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.141588+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.141588+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.985200+0000 osd.3 (osd.3) 647 : cluster [DBG] 1.4 deep-scrub starts 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.985200+0000 osd.3 (osd.3) 647 : cluster [DBG] 1.4 deep-scrub starts 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.989113+0000 osd.3 (osd.3) 648 : cluster [DBG] 1.4 deep-scrub ok 2024-04-26T16:42:06.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:05 smithi050 bash[29234]: cluster 2024-04-26T16:42:04.989113+0000 osd.3 (osd.3) 648 : cluster [DBG] 1.4 deep-scrub ok 2024-04-26T16:42:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.112902+0000 mgr.y (mgr.24401) 936 : cluster [DBG] pgmap v1536: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.112902+0000 mgr.y (mgr.24401) 936 : cluster [DBG] pgmap v1536: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.138194+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:42:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.138194+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:42:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.141588+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:42:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.141588+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:42:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.985200+0000 osd.3 (osd.3) 647 : cluster [DBG] 1.4 deep-scrub starts 2024-04-26T16:42:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.985200+0000 osd.3 (osd.3) 647 : cluster [DBG] 1.4 deep-scrub starts 2024-04-26T16:42:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.989113+0000 osd.3 (osd.3) 648 : cluster [DBG] 1.4 deep-scrub ok 2024-04-26T16:42:06.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:05 smithi190 bash[23868]: cluster 2024-04-26T16:42:04.989113+0000 osd.3 (osd.3) 648 : cluster [DBG] 1.4 deep-scrub ok 2024-04-26T16:42:07.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.162242+0000 osd.6 (osd.6) 145 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:42:07.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.162242+0000 osd.6 (osd.6) 145 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.164900+0000 osd.6 (osd.6) 146 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.164900+0000 osd.6 (osd.6) 146 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.714632+0000 osd.4 (osd.4) 537 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.714632+0000 osd.4 (osd.4) 537 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.717233+0000 osd.4 (osd.4) 538 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[29234]: cluster 2024-04-26T16:42:05.717233+0000 osd.4 (osd.4) 538 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.162242+0000 osd.6 (osd.6) 145 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.162242+0000 osd.6 (osd.6) 145 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.164900+0000 osd.6 (osd.6) 146 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.164900+0000 osd.6 (osd.6) 146 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.714632+0000 osd.4 (osd.4) 537 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:42:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.714632+0000 osd.4 (osd.4) 537 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:42:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.717233+0000 osd.4 (osd.4) 538 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:42:07.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:06 smithi050 bash[21463]: cluster 2024-04-26T16:42:05.717233+0000 osd.4 (osd.4) 538 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:42:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.162242+0000 osd.6 (osd.6) 145 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:42:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.162242+0000 osd.6 (osd.6) 145 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:42:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.164900+0000 osd.6 (osd.6) 146 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:42:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.164900+0000 osd.6 (osd.6) 146 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:42:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.714632+0000 osd.4 (osd.4) 537 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:42:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.714632+0000 osd.4 (osd.4) 537 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:42:07.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.717233+0000 osd.4 (osd.4) 538 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:42:07.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:06 smithi190 bash[23868]: cluster 2024-04-26T16:42:05.717233+0000 osd.4 (osd.4) 538 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:42:07.854 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:42:08.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:06.113678+0000 mgr.y (mgr.24401) 937 : cluster [DBG] pgmap v1537: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:06.113678+0000 mgr.y (mgr.24401) 937 : cluster [DBG] pgmap v1537: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:06.753539+0000 osd.4 (osd.4) 539 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:06.753539+0000 osd.4 (osd.4) 539 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:06.759737+0000 osd.4 (osd.4) 540 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:06.759737+0000 osd.4 (osd.4) 540 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.053889+0000 osd.1 (osd.1) 577 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.053889+0000 osd.1 (osd.1) 577 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.054275+0000 osd.1 (osd.1) 578 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.054275+0000 osd.1 (osd.1) 578 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:08.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:06.113678+0000 mgr.y (mgr.24401) 937 : cluster [DBG] pgmap v1537: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:06.113678+0000 mgr.y (mgr.24401) 937 : cluster [DBG] pgmap v1537: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:06.753539+0000 osd.4 (osd.4) 539 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:06.753539+0000 osd.4 (osd.4) 539 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:06.759737+0000 osd.4 (osd.4) 540 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:06.759737+0000 osd.4 (osd.4) 540 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.053889+0000 osd.1 (osd.1) 577 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.053889+0000 osd.1 (osd.1) 577 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.054275+0000 osd.1 (osd.1) 578 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:07 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.054275+0000 osd.1 (osd.1) 578 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:06.113678+0000 mgr.y (mgr.24401) 937 : cluster [DBG] pgmap v1537: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:06.113678+0000 mgr.y (mgr.24401) 937 : cluster [DBG] pgmap v1537: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:08.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:06.753539+0000 osd.4 (osd.4) 539 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:06.753539+0000 osd.4 (osd.4) 539 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:06.759737+0000 osd.4 (osd.4) 540 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:06.759737+0000 osd.4 (osd.4) 540 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.053889+0000 osd.1 (osd.1) 577 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.053889+0000 osd.1 (osd.1) 577 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.054275+0000 osd.1 (osd.1) 578 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:08.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:07 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.054275+0000 osd.1 (osd.1) 578 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.028484+0000 osd.3 (osd.3) 649 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.028484+0000 osd.3 (osd.3) 649 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.031585+0000 osd.3 (osd.3) 650 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.031585+0000 osd.3 (osd.3) 650 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.156190+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.156190+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:42:09.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.158557+0000 osd.6 (osd.6) 148 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.158557+0000 osd.6 (osd.6) 148 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.433340+0000 osd.7 (osd.7) 626 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.433340+0000 osd.7 (osd.7) 626 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.435655+0000 osd.7 (osd.7) 627 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.435655+0000 osd.7 (osd.7) 627 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: audit 2024-04-26T16:42:07.588895+0000 mgr.y (mgr.24401) 938 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: audit 2024-04-26T16:42:07.588895+0000 mgr.y (mgr.24401) 938 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.757801+0000 osd.4 (osd.4) 541 : cluster [DBG] 11.e scrub starts 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.757801+0000 osd.4 (osd.4) 541 : cluster [DBG] 11.e scrub starts 2024-04-26T16:42:09.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.760896+0000 osd.4 (osd.4) 542 : cluster [DBG] 11.e scrub ok 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.760896+0000 osd.4 (osd.4) 542 : cluster [DBG] 11.e scrub ok 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.856361+0000 mon.a (mon.0) 4065 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:07.856361+0000 mon.a (mon.0) 4065 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.114589+0000 mgr.y (mgr.24401) 939 : cluster [DBG] pgmap v1539: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.114589+0000 mgr.y (mgr.24401) 939 : cluster [DBG] pgmap v1539: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.028484+0000 osd.3 (osd.3) 649 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.028484+0000 osd.3 (osd.3) 649 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.031585+0000 osd.3 (osd.3) 650 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.031585+0000 osd.3 (osd.3) 650 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.156190+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:42:09.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.156190+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.158557+0000 osd.6 (osd.6) 148 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.158557+0000 osd.6 (osd.6) 148 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.433340+0000 osd.7 (osd.7) 626 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.433340+0000 osd.7 (osd.7) 626 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.435655+0000 osd.7 (osd.7) 627 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.435655+0000 osd.7 (osd.7) 627 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: audit 2024-04-26T16:42:07.588895+0000 mgr.y (mgr.24401) 938 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: audit 2024-04-26T16:42:07.588895+0000 mgr.y (mgr.24401) 938 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.757801+0000 osd.4 (osd.4) 541 : cluster [DBG] 11.e scrub starts 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.757801+0000 osd.4 (osd.4) 541 : cluster [DBG] 11.e scrub starts 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.760896+0000 osd.4 (osd.4) 542 : cluster [DBG] 11.e scrub ok 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.760896+0000 osd.4 (osd.4) 542 : cluster [DBG] 11.e scrub ok 2024-04-26T16:42:09.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.856361+0000 mon.a (mon.0) 4065 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2024-04-26T16:42:09.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:07.856361+0000 mon.a (mon.0) 4065 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2024-04-26T16:42:09.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.114589+0000 mgr.y (mgr.24401) 939 : cluster [DBG] pgmap v1539: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:09.134 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:08 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.114589+0000 mgr.y (mgr.24401) 939 : cluster [DBG] pgmap v1539: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:09.264 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.265 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.266 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.268 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.269 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.269 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.270 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:09.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.028484+0000 osd.3 (osd.3) 649 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.028484+0000 osd.3 (osd.3) 649 : cluster [DBG] 5.0 scrub starts 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.031585+0000 osd.3 (osd.3) 650 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.031585+0000 osd.3 (osd.3) 650 : cluster [DBG] 5.0 scrub ok 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.156190+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.156190+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.158557+0000 osd.6 (osd.6) 148 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.158557+0000 osd.6 (osd.6) 148 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.433340+0000 osd.7 (osd.7) 626 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.433340+0000 osd.7 (osd.7) 626 : cluster [DBG] 11.4 scrub starts 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.435655+0000 osd.7 (osd.7) 627 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.435655+0000 osd.7 (osd.7) 627 : cluster [DBG] 11.4 scrub ok 2024-04-26T16:42:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: audit 2024-04-26T16:42:07.588895+0000 mgr.y (mgr.24401) 938 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: audit 2024-04-26T16:42:07.588895+0000 mgr.y (mgr.24401) 938 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.757801+0000 osd.4 (osd.4) 541 : cluster [DBG] 11.e scrub starts 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.757801+0000 osd.4 (osd.4) 541 : cluster [DBG] 11.e scrub starts 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.760896+0000 osd.4 (osd.4) 542 : cluster [DBG] 11.e scrub ok 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.760896+0000 osd.4 (osd.4) 542 : cluster [DBG] 11.e scrub ok 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.856361+0000 mon.a (mon.0) 4065 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:07.856361+0000 mon.a (mon.0) 4065 : cluster [DBG] osdmap e925: 8 total, 8 up, 8 in 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.114589+0000 mgr.y (mgr.24401) 939 : cluster [DBG] pgmap v1539: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:08 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.114589+0000 mgr.y (mgr.24401) 939 : cluster [DBG] pgmap v1539: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:10.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.415214+0000 osd.7 (osd.7) 628 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.415214+0000 osd.7 (osd.7) 628 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.417665+0000 osd.7 (osd.7) 629 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.417665+0000 osd.7 (osd.7) 629 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.449749+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.449749+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.450166+0000 osd.6 (osd.6) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.450166+0000 osd.6 (osd.6) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.755735+0000 osd.4 (osd.4) 543 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.755735+0000 osd.4 (osd.4) 543 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.758240+0000 osd.4 (osd.4) 544 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.758240+0000 osd.4 (osd.4) 544 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:42:10.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.882008+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[21463]: cluster 2024-04-26T16:42:08.882008+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.415214+0000 osd.7 (osd.7) 628 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.415214+0000 osd.7 (osd.7) 628 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.417665+0000 osd.7 (osd.7) 629 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.417665+0000 osd.7 (osd.7) 629 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.449749+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.449749+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.450166+0000 osd.6 (osd.6) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.450166+0000 osd.6 (osd.6) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.755735+0000 osd.4 (osd.4) 543 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.755735+0000 osd.4 (osd.4) 543 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:42:10.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.758240+0000 osd.4 (osd.4) 544 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:42:10.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.758240+0000 osd.4 (osd.4) 544 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:42:10.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.882008+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2024-04-26T16:42:10.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:09 smithi050 bash[29234]: cluster 2024-04-26T16:42:08.882008+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2024-04-26T16:42:10.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.415214+0000 osd.7 (osd.7) 628 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:42:10.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.415214+0000 osd.7 (osd.7) 628 : cluster [DBG] 11.1e scrub starts 2024-04-26T16:42:10.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.417665+0000 osd.7 (osd.7) 629 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:42:10.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.417665+0000 osd.7 (osd.7) 629 : cluster [DBG] 11.1e scrub ok 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.449749+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.449749+0000 osd.6 (osd.6) 149 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.450166+0000 osd.6 (osd.6) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.450166+0000 osd.6 (osd.6) 150 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.755735+0000 osd.4 (osd.4) 543 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.755735+0000 osd.4 (osd.4) 543 : cluster [DBG] 6.1e scrub starts 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.758240+0000 osd.4 (osd.4) 544 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.758240+0000 osd.4 (osd.4) 544 : cluster [DBG] 6.1e scrub ok 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.882008+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2024-04-26T16:42:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:09 smithi190 bash[23868]: cluster 2024-04-26T16:42:08.882008+0000 mon.a (mon.0) 4066 : cluster [DBG] osdmap e926: 8 total, 8 up, 8 in 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.412028+0000 osd.7 (osd.7) 630 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.412028+0000 osd.7 (osd.7) 630 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.417698+0000 osd.7 (osd.7) 631 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.417698+0000 osd.7 (osd.7) 631 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.751541+0000 osd.4 (osd.4) 545 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.751541+0000 osd.4 (osd.4) 545 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.751960+0000 osd.4 (osd.4) 546 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.751960+0000 osd.4 (osd.4) 546 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.890922+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:09.890922+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.094951+0000 osd.2 (osd.2) 529 : cluster [DBG] 2.c scrub starts 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.094951+0000 osd.2 (osd.2) 529 : cluster [DBG] 2.c scrub starts 2024-04-26T16:42:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.097413+0000 osd.2 (osd.2) 530 : cluster [DBG] 2.c scrub ok 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.097413+0000 osd.2 (osd.2) 530 : cluster [DBG] 2.c scrub ok 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.115428+0000 mgr.y (mgr.24401) 940 : cluster [DBG] pgmap v1542: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.115428+0000 mgr.y (mgr.24401) 940 : cluster [DBG] pgmap v1542: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.818833+0000 osd.1 (osd.1) 579 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.818833+0000 osd.1 (osd.1) 579 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.822011+0000 osd.1 (osd.1) 580 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:42:11.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:10 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.822011+0000 osd.1 (osd.1) 580 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.412028+0000 osd.7 (osd.7) 630 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.412028+0000 osd.7 (osd.7) 630 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.417698+0000 osd.7 (osd.7) 631 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.417698+0000 osd.7 (osd.7) 631 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.751541+0000 osd.4 (osd.4) 545 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.751541+0000 osd.4 (osd.4) 545 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.751960+0000 osd.4 (osd.4) 546 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.751960+0000 osd.4 (osd.4) 546 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.890922+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:09.890922+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.094951+0000 osd.2 (osd.2) 529 : cluster [DBG] 2.c scrub starts 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.094951+0000 osd.2 (osd.2) 529 : cluster [DBG] 2.c scrub starts 2024-04-26T16:42:11.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.097413+0000 osd.2 (osd.2) 530 : cluster [DBG] 2.c scrub ok 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.097413+0000 osd.2 (osd.2) 530 : cluster [DBG] 2.c scrub ok 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.115428+0000 mgr.y (mgr.24401) 940 : cluster [DBG] pgmap v1542: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.115428+0000 mgr.y (mgr.24401) 940 : cluster [DBG] pgmap v1542: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.818833+0000 osd.1 (osd.1) 579 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.818833+0000 osd.1 (osd.1) 579 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.822011+0000 osd.1 (osd.1) 580 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.822011+0000 osd.1 (osd.1) 580 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.412028+0000 osd.7 (osd.7) 630 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.412028+0000 osd.7 (osd.7) 630 : cluster [DBG] 6.1c scrub starts 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.417698+0000 osd.7 (osd.7) 631 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.417698+0000 osd.7 (osd.7) 631 : cluster [DBG] 6.1c scrub ok 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.751541+0000 osd.4 (osd.4) 545 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:11.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.751541+0000 osd.4 (osd.4) 545 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.751960+0000 osd.4 (osd.4) 546 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.751960+0000 osd.4 (osd.4) 546 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.890922+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:09.890922+0000 mon.a (mon.0) 4067 : cluster [DBG] osdmap e927: 8 total, 8 up, 8 in 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.094951+0000 osd.2 (osd.2) 529 : cluster [DBG] 2.c scrub starts 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.094951+0000 osd.2 (osd.2) 529 : cluster [DBG] 2.c scrub starts 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.097413+0000 osd.2 (osd.2) 530 : cluster [DBG] 2.c scrub ok 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.097413+0000 osd.2 (osd.2) 530 : cluster [DBG] 2.c scrub ok 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.115428+0000 mgr.y (mgr.24401) 940 : cluster [DBG] pgmap v1542: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.115428+0000 mgr.y (mgr.24401) 940 : cluster [DBG] pgmap v1542: 174 pgs: 174 active+clean; 583 KiB data, 784 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.818833+0000 osd.1 (osd.1) 579 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.818833+0000 osd.1 (osd.1) 579 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.822011+0000 osd.1 (osd.1) 580 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:42:11.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:10 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.822011+0000 osd.1 (osd.1) 580 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:42:12.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.123769+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:42:12.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.123769+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:42:12.102 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.128721+0000 osd.6 (osd.6) 152 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:42:12.103 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[29234]: cluster 2024-04-26T16:42:10.128721+0000 osd.6 (osd.6) 152 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:42:12.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.123769+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:42:12.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.123769+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:42:12.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.128721+0000 osd.6 (osd.6) 152 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:42:12.103 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:11 smithi050 bash[21463]: cluster 2024-04-26T16:42:10.128721+0000 osd.6 (osd.6) 152 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:42:12.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:11 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.123769+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:42:12.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:11 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.123769+0000 osd.6 (osd.6) 151 : cluster [DBG] 1.7 deep-scrub starts 2024-04-26T16:42:12.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:11 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.128721+0000 osd.6 (osd.6) 152 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:42:12.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:11 smithi190 bash[23868]: cluster 2024-04-26T16:42:10.128721+0000 osd.6 (osd.6) 152 : cluster [DBG] 1.7 deep-scrub ok 2024-04-26T16:42:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.096889+0000 osd.3 (osd.3) 651 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:42:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.096889+0000 osd.3 (osd.3) 651 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:42:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.099517+0000 osd.3 (osd.3) 652 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:42:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.099517+0000 osd.3 (osd.3) 652 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.397268+0000 osd.7 (osd.7) 632 : cluster [DBG] 11.f scrub starts 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.397268+0000 osd.7 (osd.7) 632 : cluster [DBG] 11.f scrub starts 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.404876+0000 osd.7 (osd.7) 633 : cluster [DBG] 11.f scrub ok 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.404876+0000 osd.7 (osd.7) 633 : cluster [DBG] 11.f scrub ok 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.918274+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.918274+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.920700+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:11.920700+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:12.116420+0000 mgr.y (mgr.24401) 941 : cluster [DBG] pgmap v1543: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:42:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:12 smithi190 bash[23868]: cluster 2024-04-26T16:42:12.116420+0000 mgr.y (mgr.24401) 941 : cluster [DBG] pgmap v1543: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:42:13.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.096889+0000 osd.3 (osd.3) 651 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:42:13.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.096889+0000 osd.3 (osd.3) 651 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:42:13.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.099517+0000 osd.3 (osd.3) 652 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:42:13.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.099517+0000 osd.3 (osd.3) 652 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:42:13.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.397268+0000 osd.7 (osd.7) 632 : cluster [DBG] 11.f scrub starts 2024-04-26T16:42:13.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.397268+0000 osd.7 (osd.7) 632 : cluster [DBG] 11.f scrub starts 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.404876+0000 osd.7 (osd.7) 633 : cluster [DBG] 11.f scrub ok 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.404876+0000 osd.7 (osd.7) 633 : cluster [DBG] 11.f scrub ok 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.918274+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.918274+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.920700+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:11.920700+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:12.116420+0000 mgr.y (mgr.24401) 941 : cluster [DBG] pgmap v1543: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[21463]: cluster 2024-04-26T16:42:12.116420+0000 mgr.y (mgr.24401) 941 : cluster [DBG] pgmap v1543: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.096889+0000 osd.3 (osd.3) 651 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.096889+0000 osd.3 (osd.3) 651 : cluster [DBG] 6.1f scrub starts 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.099517+0000 osd.3 (osd.3) 652 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.099517+0000 osd.3 (osd.3) 652 : cluster [DBG] 6.1f scrub ok 2024-04-26T16:42:13.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.397268+0000 osd.7 (osd.7) 632 : cluster [DBG] 11.f scrub starts 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.397268+0000 osd.7 (osd.7) 632 : cluster [DBG] 11.f scrub starts 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.404876+0000 osd.7 (osd.7) 633 : cluster [DBG] 11.f scrub ok 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.404876+0000 osd.7 (osd.7) 633 : cluster [DBG] 11.f scrub ok 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.918274+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.918274+0000 osd.5 (osd.5) 496 : cluster [DBG] 2.16 scrub starts 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.920700+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:11.920700+0000 osd.5 (osd.5) 497 : cluster [DBG] 2.16 scrub ok 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:12.116420+0000 mgr.y (mgr.24401) 941 : cluster [DBG] pgmap v1543: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:42:13.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:12 smithi050 bash[29234]: cluster 2024-04-26T16:42:12.116420+0000 mgr.y (mgr.24401) 941 : cluster [DBG] pgmap v1543: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-04-26T16:42:14.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:13 smithi050 bash[21463]: audit 2024-04-26T16:42:13.602756+0000 mon.c (mon.2) 950 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:14.300 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:13 smithi050 bash[21463]: audit 2024-04-26T16:42:13.602756+0000 mon.c (mon.2) 950 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:14.300 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:13 smithi050 bash[29234]: audit 2024-04-26T16:42:13.602756+0000 mon.c (mon.2) 950 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:14.300 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:13 smithi050 bash[29234]: audit 2024-04-26T16:42:13.602756+0000 mon.c (mon.2) 950 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:14.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:13 smithi190 bash[23868]: audit 2024-04-26T16:42:13.602756+0000 mon.c (mon.2) 950 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:14.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:13 smithi190 bash[23868]: audit 2024-04-26T16:42:13.602756+0000 mon.c (mon.2) 950 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:14.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:42:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:42:15.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21463]: cluster 2024-04-26T16:42:13.422028+0000 osd.7 (osd.7) 634 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:42:15.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21463]: cluster 2024-04-26T16:42:13.422028+0000 osd.7 (osd.7) 634 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21463]: cluster 2024-04-26T16:42:13.425909+0000 osd.7 (osd.7) 635 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21463]: cluster 2024-04-26T16:42:13.425909+0000 osd.7 (osd.7) 635 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.117312+0000 mgr.y (mgr.24401) 942 : cluster [DBG] pgmap v1544: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.117312+0000 mgr.y (mgr.24401) 942 : cluster [DBG] pgmap v1544: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[29234]: cluster 2024-04-26T16:42:13.422028+0000 osd.7 (osd.7) 634 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[29234]: cluster 2024-04-26T16:42:13.422028+0000 osd.7 (osd.7) 634 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[29234]: cluster 2024-04-26T16:42:13.425909+0000 osd.7 (osd.7) 635 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[29234]: cluster 2024-04-26T16:42:13.425909+0000 osd.7 (osd.7) 635 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.117312+0000 mgr.y (mgr.24401) 942 : cluster [DBG] pgmap v1544: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:42:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:14 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.117312+0000 mgr.y (mgr.24401) 942 : cluster [DBG] pgmap v1544: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:42:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:14 smithi190 bash[23868]: cluster 2024-04-26T16:42:13.422028+0000 osd.7 (osd.7) 634 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:42:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:14 smithi190 bash[23868]: cluster 2024-04-26T16:42:13.422028+0000 osd.7 (osd.7) 634 : cluster [DBG] 2.1c scrub starts 2024-04-26T16:42:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:14 smithi190 bash[23868]: cluster 2024-04-26T16:42:13.425909+0000 osd.7 (osd.7) 635 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:42:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:14 smithi190 bash[23868]: cluster 2024-04-26T16:42:13.425909+0000 osd.7 (osd.7) 635 : cluster [DBG] 2.1c scrub ok 2024-04-26T16:42:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:14 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.117312+0000 mgr.y (mgr.24401) 942 : cluster [DBG] pgmap v1544: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:42:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:14 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.117312+0000 mgr.y (mgr.24401) 942 : cluster [DBG] pgmap v1544: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.092964+0000 osd.0 (osd.0) 605 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.092964+0000 osd.0 (osd.0) 605 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.096342+0000 osd.0 (osd.0) 606 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.096342+0000 osd.0 (osd.0) 606 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.411837+0000 osd.7 (osd.7) 636 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.411837+0000 osd.7 (osd.7) 636 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.414391+0000 osd.7 (osd.7) 637 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:14.414391+0000 osd.7 (osd.7) 637 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:15.042206+0000 osd.2 (osd.2) 531 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:15.042206+0000 osd.2 (osd.2) 531 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:42:16.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:15.044679+0000 osd.2 (osd.2) 532 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[29234]: cluster 2024-04-26T16:42:15.044679+0000 osd.2 (osd.2) 532 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.092964+0000 osd.0 (osd.0) 605 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.092964+0000 osd.0 (osd.0) 605 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.096342+0000 osd.0 (osd.0) 606 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.096342+0000 osd.0 (osd.0) 606 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.411837+0000 osd.7 (osd.7) 636 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.411837+0000 osd.7 (osd.7) 636 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.414391+0000 osd.7 (osd.7) 637 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:14.414391+0000 osd.7 (osd.7) 637 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:15.042206+0000 osd.2 (osd.2) 531 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:15.042206+0000 osd.2 (osd.2) 531 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:42:16.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:15.044679+0000 osd.2 (osd.2) 532 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:42:16.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:15 smithi050 bash[21463]: cluster 2024-04-26T16:42:15.044679+0000 osd.2 (osd.2) 532 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:42:16.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.092964+0000 osd.0 (osd.0) 605 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:42:16.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.092964+0000 osd.0 (osd.0) 605 : cluster [DBG] 11.6 scrub starts 2024-04-26T16:42:16.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.096342+0000 osd.0 (osd.0) 606 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:42:16.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.096342+0000 osd.0 (osd.0) 606 : cluster [DBG] 11.6 scrub ok 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.411837+0000 osd.7 (osd.7) 636 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.411837+0000 osd.7 (osd.7) 636 : cluster [DBG] 2.8 scrub starts 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.414391+0000 osd.7 (osd.7) 637 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:14.414391+0000 osd.7 (osd.7) 637 : cluster [DBG] 2.8 scrub ok 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:15.042206+0000 osd.2 (osd.2) 531 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:15.042206+0000 osd.2 (osd.2) 531 : cluster [DBG] 5.6 scrub starts 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:15.044679+0000 osd.2 (osd.2) 532 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:42:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:15 smithi190 bash[23868]: cluster 2024-04-26T16:42:15.044679+0000 osd.2 (osd.2) 532 : cluster [DBG] 5.6 scrub ok 2024-04-26T16:42:17.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.118109+0000 mgr.y (mgr.24401) 943 : cluster [DBG] pgmap v1545: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:17.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.118109+0000 mgr.y (mgr.24401) 943 : cluster [DBG] pgmap v1545: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:17.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.178883+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:42:17.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.178883+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:42:17.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.181513+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:42:17.775 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.181513+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.428695+0000 osd.7 (osd.7) 638 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.428695+0000 osd.7 (osd.7) 638 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.431641+0000 osd.7 (osd.7) 639 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[21463]: cluster 2024-04-26T16:42:16.431641+0000 osd.7 (osd.7) 639 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.118109+0000 mgr.y (mgr.24401) 943 : cluster [DBG] pgmap v1545: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.118109+0000 mgr.y (mgr.24401) 943 : cluster [DBG] pgmap v1545: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.178883+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.178883+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.181513+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.181513+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.428695+0000 osd.7 (osd.7) 638 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:42:17.776 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.428695+0000 osd.7 (osd.7) 638 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:42:17.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.431641+0000 osd.7 (osd.7) 639 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:42:17.777 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:17 smithi050 bash[29234]: cluster 2024-04-26T16:42:16.431641+0000 osd.7 (osd.7) 639 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:42:17.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.118109+0000 mgr.y (mgr.24401) 943 : cluster [DBG] pgmap v1545: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.118109+0000 mgr.y (mgr.24401) 943 : cluster [DBG] pgmap v1545: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.178883+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.178883+0000 osd.6 (osd.6) 153 : cluster [DBG] 2.11 scrub starts 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.181513+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.181513+0000 osd.6 (osd.6) 154 : cluster [DBG] 2.11 scrub ok 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.428695+0000 osd.7 (osd.7) 638 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:42:17.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.428695+0000 osd.7 (osd.7) 638 : cluster [DBG] 6.8 scrub starts 2024-04-26T16:42:17.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.431641+0000 osd.7 (osd.7) 639 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:42:17.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:17 smithi190 bash[23868]: cluster 2024-04-26T16:42:16.431641+0000 osd.7 (osd.7) 639 : cluster [DBG] 6.8 scrub ok 2024-04-26T16:42:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:18 smithi190 bash[23868]: cluster 2024-04-26T16:42:17.121983+0000 osd.3 (osd.3) 653 : cluster [DBG] 11.a scrub starts 2024-04-26T16:42:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:18 smithi190 bash[23868]: cluster 2024-04-26T16:42:17.121983+0000 osd.3 (osd.3) 653 : cluster [DBG] 11.a scrub starts 2024-04-26T16:42:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:18 smithi190 bash[23868]: cluster 2024-04-26T16:42:17.124339+0000 osd.3 (osd.3) 654 : cluster [DBG] 11.a scrub ok 2024-04-26T16:42:18.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:18 smithi190 bash[23868]: cluster 2024-04-26T16:42:17.124339+0000 osd.3 (osd.3) 654 : cluster [DBG] 11.a scrub ok 2024-04-26T16:42:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:18 smithi190 bash[23868]: audit 2024-04-26T16:42:17.600476+0000 mgr.y (mgr.24401) 944 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:18.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:18 smithi190 bash[23868]: audit 2024-04-26T16:42:17.600476+0000 mgr.y (mgr.24401) 944 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:18.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[21463]: cluster 2024-04-26T16:42:17.121983+0000 osd.3 (osd.3) 653 : cluster [DBG] 11.a scrub starts 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[21463]: cluster 2024-04-26T16:42:17.121983+0000 osd.3 (osd.3) 653 : cluster [DBG] 11.a scrub starts 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[21463]: cluster 2024-04-26T16:42:17.124339+0000 osd.3 (osd.3) 654 : cluster [DBG] 11.a scrub ok 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[21463]: cluster 2024-04-26T16:42:17.124339+0000 osd.3 (osd.3) 654 : cluster [DBG] 11.a scrub ok 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[21463]: audit 2024-04-26T16:42:17.600476+0000 mgr.y (mgr.24401) 944 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[21463]: audit 2024-04-26T16:42:17.600476+0000 mgr.y (mgr.24401) 944 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[29234]: cluster 2024-04-26T16:42:17.121983+0000 osd.3 (osd.3) 653 : cluster [DBG] 11.a scrub starts 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[29234]: cluster 2024-04-26T16:42:17.121983+0000 osd.3 (osd.3) 653 : cluster [DBG] 11.a scrub starts 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[29234]: cluster 2024-04-26T16:42:17.124339+0000 osd.3 (osd.3) 654 : cluster [DBG] 11.a scrub ok 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[29234]: cluster 2024-04-26T16:42:17.124339+0000 osd.3 (osd.3) 654 : cluster [DBG] 11.a scrub ok 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[29234]: audit 2024-04-26T16:42:17.600476+0000 mgr.y (mgr.24401) 944 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:18.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:18 smithi050 bash[29234]: audit 2024-04-26T16:42:17.600476+0000 mgr.y (mgr.24401) 944 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.073406+0000 osd.0 (osd.0) 607 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:42:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.073406+0000 osd.0 (osd.0) 607 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:42:19.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.075770+0000 osd.0 (osd.0) 608 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.075770+0000 osd.0 (osd.0) 608 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.119996+0000 mgr.y (mgr.24401) 945 : cluster [DBG] pgmap v1546: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.119996+0000 mgr.y (mgr.24401) 945 : cluster [DBG] pgmap v1546: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.216499+0000 osd.6 (osd.6) 155 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.216499+0000 osd.6 (osd.6) 155 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.219161+0000 osd.6 (osd.6) 156 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:42:19.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:19 smithi190 bash[23868]: cluster 2024-04-26T16:42:18.219161+0000 osd.6 (osd.6) 156 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.073406+0000 osd.0 (osd.0) 607 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.073406+0000 osd.0 (osd.0) 607 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.075770+0000 osd.0 (osd.0) 608 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.075770+0000 osd.0 (osd.0) 608 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.119996+0000 mgr.y (mgr.24401) 945 : cluster [DBG] pgmap v1546: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.119996+0000 mgr.y (mgr.24401) 945 : cluster [DBG] pgmap v1546: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.216499+0000 osd.6 (osd.6) 155 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.216499+0000 osd.6 (osd.6) 155 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.219161+0000 osd.6 (osd.6) 156 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:42:19.898 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[21463]: cluster 2024-04-26T16:42:18.219161+0000 osd.6 (osd.6) 156 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.073406+0000 osd.0 (osd.0) 607 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.073406+0000 osd.0 (osd.0) 607 : cluster [DBG] 5.5 scrub starts 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.075770+0000 osd.0 (osd.0) 608 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.075770+0000 osd.0 (osd.0) 608 : cluster [DBG] 5.5 scrub ok 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.119996+0000 mgr.y (mgr.24401) 945 : cluster [DBG] pgmap v1546: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.119996+0000 mgr.y (mgr.24401) 945 : cluster [DBG] pgmap v1546: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.216499+0000 osd.6 (osd.6) 155 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.216499+0000 osd.6 (osd.6) 155 : cluster [DBG] 11.14 scrub starts 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.219161+0000 osd.6 (osd.6) 156 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:42:19.899 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:19 smithi050 bash[29234]: cluster 2024-04-26T16:42:18.219161+0000 osd.6 (osd.6) 156 : cluster [DBG] 11.14 scrub ok 2024-04-26T16:42:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.011019+0000 osd.1 (osd.1) 581 : cluster [DBG] 11.c scrub starts 2024-04-26T16:42:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.011019+0000 osd.1 (osd.1) 581 : cluster [DBG] 11.c scrub starts 2024-04-26T16:42:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.014037+0000 osd.1 (osd.1) 582 : cluster [DBG] 11.c scrub ok 2024-04-26T16:42:20.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.014037+0000 osd.1 (osd.1) 582 : cluster [DBG] 11.c scrub ok 2024-04-26T16:42:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.064786+0000 osd.2 (osd.2) 533 : cluster [DBG] 5.d scrub starts 2024-04-26T16:42:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.064786+0000 osd.2 (osd.2) 533 : cluster [DBG] 5.d scrub starts 2024-04-26T16:42:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.067540+0000 osd.2 (osd.2) 534 : cluster [DBG] 5.d scrub ok 2024-04-26T16:42:20.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:20 smithi190 bash[23868]: cluster 2024-04-26T16:42:19.067540+0000 osd.2 (osd.2) 534 : cluster [DBG] 5.d scrub ok 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.011019+0000 osd.1 (osd.1) 581 : cluster [DBG] 11.c scrub starts 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.011019+0000 osd.1 (osd.1) 581 : cluster [DBG] 11.c scrub starts 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.014037+0000 osd.1 (osd.1) 582 : cluster [DBG] 11.c scrub ok 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.014037+0000 osd.1 (osd.1) 582 : cluster [DBG] 11.c scrub ok 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.064786+0000 osd.2 (osd.2) 533 : cluster [DBG] 5.d scrub starts 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.064786+0000 osd.2 (osd.2) 533 : cluster [DBG] 5.d scrub starts 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.067540+0000 osd.2 (osd.2) 534 : cluster [DBG] 5.d scrub ok 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[21463]: cluster 2024-04-26T16:42:19.067540+0000 osd.2 (osd.2) 534 : cluster [DBG] 5.d scrub ok 2024-04-26T16:42:21.079 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.011019+0000 osd.1 (osd.1) 581 : cluster [DBG] 11.c scrub starts 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.011019+0000 osd.1 (osd.1) 581 : cluster [DBG] 11.c scrub starts 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.014037+0000 osd.1 (osd.1) 582 : cluster [DBG] 11.c scrub ok 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.014037+0000 osd.1 (osd.1) 582 : cluster [DBG] 11.c scrub ok 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.064786+0000 osd.2 (osd.2) 533 : cluster [DBG] 5.d scrub starts 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.064786+0000 osd.2 (osd.2) 533 : cluster [DBG] 5.d scrub starts 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.067540+0000 osd.2 (osd.2) 534 : cluster [DBG] 5.d scrub ok 2024-04-26T16:42:21.080 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:20 smithi050 bash[29234]: cluster 2024-04-26T16:42:19.067540+0000 osd.2 (osd.2) 534 : cluster [DBG] 5.d scrub ok 2024-04-26T16:42:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.088548+0000 osd.0 (osd.0) 609 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:42:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.088548+0000 osd.0 (osd.0) 609 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:42:21.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.090959+0000 osd.0 (osd.0) 610 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.090959+0000 osd.0 (osd.0) 610 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.120935+0000 mgr.y (mgr.24401) 946 : cluster [DBG] pgmap v1547: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.120935+0000 mgr.y (mgr.24401) 946 : cluster [DBG] pgmap v1547: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.368674+0000 osd.7 (osd.7) 640 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.368674+0000 osd.7 (osd.7) 640 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.372134+0000 osd.7 (osd.7) 641 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:42:21.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:21 smithi190 bash[23868]: cluster 2024-04-26T16:42:20.372134+0000 osd.7 (osd.7) 641 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:42:21.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.088548+0000 osd.0 (osd.0) 609 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.088548+0000 osd.0 (osd.0) 609 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.090959+0000 osd.0 (osd.0) 610 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.090959+0000 osd.0 (osd.0) 610 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.120935+0000 mgr.y (mgr.24401) 946 : cluster [DBG] pgmap v1547: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.120935+0000 mgr.y (mgr.24401) 946 : cluster [DBG] pgmap v1547: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.368674+0000 osd.7 (osd.7) 640 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.368674+0000 osd.7 (osd.7) 640 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.372134+0000 osd.7 (osd.7) 641 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:42:21.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[21463]: cluster 2024-04-26T16:42:20.372134+0000 osd.7 (osd.7) 641 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.088548+0000 osd.0 (osd.0) 609 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.088548+0000 osd.0 (osd.0) 609 : cluster [DBG] 2.1f scrub starts 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.090959+0000 osd.0 (osd.0) 610 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.090959+0000 osd.0 (osd.0) 610 : cluster [DBG] 2.1f scrub ok 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.120935+0000 mgr.y (mgr.24401) 946 : cluster [DBG] pgmap v1547: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.120935+0000 mgr.y (mgr.24401) 946 : cluster [DBG] pgmap v1547: 174 pgs: 174 active+clean; 583 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.368674+0000 osd.7 (osd.7) 640 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.368674+0000 osd.7 (osd.7) 640 : cluster [DBG] 3.15 scrub starts 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.372134+0000 osd.7 (osd.7) 641 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:42:21.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:21 smithi050 bash[29234]: cluster 2024-04-26T16:42:20.372134+0000 osd.7 (osd.7) 641 : cluster [DBG] 3.15 scrub ok 2024-04-26T16:42:22.504 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/1955113077 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 msgr2=0x7f1e5810f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/1955113077 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5810f910 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009990 tx=0x7f1e4c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/1955113077 shutdown_connections 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/1955113077 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e58075650 0x7f1e58075a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/1955113077 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 0x7f1e58112240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/1955113077 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5810f910 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/1955113077 >> 172.21.15.50:0/1955113077 conn(0x7f1e5807a5d0 msgr2=0x7f1e5807a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/1955113077 shutdown_connections 2024-04-26T16:42:22.505 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/1955113077 wait complete. 2024-04-26T16:42:22.506 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 librados: starting msgr at 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 librados: starting objecter 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.505+0000 7f1e5cbb9700 1 Processor -- start 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- start start 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 librados: setting wanted keys 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 librados: calling monclient init 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e58075650 0x7f1e5819b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 0x7f1e5819fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f1e58114e50 con 0x7f1e5810fe50 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f1e58114cd0 con 0x7f1e58075650 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f1e58114fd0 con 0x7f1e58075f70 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:22.507 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e56d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 0x7f1e5819fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5819ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52394/0 (socket says 172.21.15.50:52394) 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e56d9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 0x7f1e5819fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32914/0 (socket says 172.21.15.50:32914) 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 -- 172.21.15.50:0/2496770294 learned_addr learned my addr 172.21.15.50:0/2496770294 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5659c700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e58075650 0x7f1e5819b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 -- 172.21.15.50:0/2496770294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e58075650 msgr2=0x7f1e5819b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1e58075650 0x7f1e5819b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 -- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 msgr2=0x7f1e5819fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 0x7f1e5819fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 -- 172.21.15.50:0/2496770294 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e581a04a0 con 0x7f1e58075f70 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e56d9d700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f1e5810fe50 0x7f1e5819fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e55d9b700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5819ba20 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c005910 tx=0x7f1e4c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 10 client.?.objecter ms_handle_connect 0x7f1e58075f70 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c027070 con 0x7f1e58075f70 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e581a0730 con 0x7f1e58075f70 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f1e4c022470 con 0x7f1e58075f70 2024-04-26T16:42:22.508 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c017650 con 0x7f1e58075f70 2024-04-26T16:42:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 10 client.41900.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e581a8010 con 0x7f1e58075f70 2024-04-26T16:42:22.509 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 librados: init done 2024-04-26T16:42:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 10 client.41900.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 15 client.41900.objecter _session_command_op_assign -1 1 2024-04-26T16:42:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 20 client.41900.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 10 client.41900.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:22.510 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1e58076d90 con 0x7f1e58075f70 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f1e4c02a070 con 0x7f1e58075f70 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1e40071f80 0x7f1e40074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(927..927 src has 874..927) ==== 8707+0+0 (secure 0 0 0) 0x7f1e4c0933a0 con 0x7f1e58075f70 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 10 client.41900.objecter ms_dispatch 0x7f1e58001550 osd_map(927..927 src has 874..927) 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 3 client.41900.objecter handle_osd_map got epochs [927,927] > 0 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 3 client.41900.objecter handle_osd_map decoding full epoch 927 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e5659c700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1e40071f80 0x7f1e40074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.509+0000 7f1e477fe700 10 client.41900.objecter checking command 1 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1e4007dc60 0x7f1e40080100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter _get_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 15 client.41900.objecter _session_command_op_remove -1 1 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter _get_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter get_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.511 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 15 client.41900.objecter _session_command_op_assign 0 1 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 4 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 10 client.41900.objecter _send_command 1 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1e4c010ec0 con 0x7f1e4007dc60 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter dump_active .. 0 homeless 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=927) ==== 24+0+0 (secure 0 0 0) 0x7f1e4c0937f0 con 0x7f1e58075f70 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e56d9d700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1e4007dc60 0x7f1e40080100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 10 client.41900.objecter ms_handle_connect 0x7f1e40071f80 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e5659c700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1e40071f80 0x7f1e40074440 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009d10 tx=0x7f1e48009040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 10 client.41900.objecter ms_handle_connect 0x7f1e4007dc60 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e56d9d700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1e4007dc60 0x7f1e40080100 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== osd.0 v2:172.21.15.50:6802/310950989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1e4c010ec0 con 0x7f1e4007dc60 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 10 client.41900.objecter ms_dispatch 0x7f1e58001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 10 client.41900.objecter _finish_command 1 = system:0 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 4 2024-04-26T16:42:22.512 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.513+0000 7f1e477fe700 15 client.41900.objecter _session_command_op_remove 0 1 2024-04-26T16:42:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 10 client.41900.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:22.529 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 15 client.41900.objecter _session_command_op_assign -1 2 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter _get_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter _get_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 15 client.41900.objecter _session_command_op_remove -1 2 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter get_session s=0x7f1e4007d590 osd=0 3 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 15 client.41900.objecter _session_command_op_assign 0 2 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 4 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 10 client.41900.objecter _send_command 2 2024-04-26T16:42:22.530 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 --> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1e5806a110 con 0x7f1e4007dc60 2024-04-26T16:42:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e477fe700 1 -- 172.21.15.50:0/2496770294 <== osd.0 v2:172.21.15.50:6802/310950989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1e5806a110 con 0x7f1e4007dc60 2024-04-26T16:42:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e477fe700 10 client.41900.objecter ms_dispatch 0x7f1e58001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e477fe700 10 client.41900.objecter _finish_command 2 = system:0 2024-04-26T16:42:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e477fe700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 4 2024-04-26T16:42:22.531 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e477fe700 15 client.41900.objecter _session_command_op_remove 0 2 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 10 librados: watch_flush enter 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 10 librados: watch_flush exit 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 10 client.41900.objecter close_session for osd.0 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1e4007dc60 msgr2=0x7f1e40080100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6802/310950989,v1:172.21.15.50:6803/310950989] conn(0x7f1e4007dc60 0x7f1e40080100 crc :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter put_session s=0x7f1e4007d590 osd=0 1 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 20 client.41900.objecter shutdown clearing up homeless session... 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 10 client.41900.objecter successfully canceled tick 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1e40071f80 msgr2=0x7f1e40074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f1e40071f80 0x7f1e40074440 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009d10 tx=0x7f1e48009040 comp rx=0 tx=0).stop 2024-04-26T16:42:22.532 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 msgr2=0x7f1e5819ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 --2- 172.21.15.50:0/2496770294 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f1e58075f70 0x7f1e5819ba20 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c005910 tx=0x7f1e4c0047b0 comp rx=0 tx=0).stop 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e56d9d700 1 -- 172.21.15.50:0/2496770294 reap_dead start 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 shutdown_connections 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 >> 172.21.15.50:0/2496770294 conn(0x7f1e5807a5d0 msgr2=0x7f1e58101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 shutdown_connections 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 -- 172.21.15.50:0/2496770294 wait complete. 2024-04-26T16:42:22.533 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:22.533+0000 7f1e5cbb9700 1 librados: shutdown 2024-04-26T16:42:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.115013+0000 osd.0 (osd.0) 611 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:42:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.115013+0000 osd.0 (osd.0) 611 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:42:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.117734+0000 osd.0 (osd.0) 612 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:42:22.629 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.117734+0000 osd.0 (osd.0) 612 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.375025+0000 osd.7 (osd.7) 642 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.375025+0000 osd.7 (osd.7) 642 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.377447+0000 osd.7 (osd.7) 643 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.377447+0000 osd.7 (osd.7) 643 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.881493+0000 osd.4 (osd.4) 547 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.881493+0000 osd.4 (osd.4) 547 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.883789+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.883789+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.115013+0000 osd.0 (osd.0) 611 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:42:22.630 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.115013+0000 osd.0 (osd.0) 611 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.117734+0000 osd.0 (osd.0) 612 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.117734+0000 osd.0 (osd.0) 612 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.375025+0000 osd.7 (osd.7) 642 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.375025+0000 osd.7 (osd.7) 642 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.377447+0000 osd.7 (osd.7) 643 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.377447+0000 osd.7 (osd.7) 643 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.881493+0000 osd.4 (osd.4) 547 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.881493+0000 osd.4 (osd.4) 547 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.883789+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:42:22.631 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:22 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.883789+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:42:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.115013+0000 osd.0 (osd.0) 611 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:42:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.115013+0000 osd.0 (osd.0) 611 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:42:22.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.117734+0000 osd.0 (osd.0) 612 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.117734+0000 osd.0 (osd.0) 612 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.375025+0000 osd.7 (osd.7) 642 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.375025+0000 osd.7 (osd.7) 642 : cluster [DBG] 6.3 scrub starts 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.377447+0000 osd.7 (osd.7) 643 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.377447+0000 osd.7 (osd.7) 643 : cluster [DBG] 6.3 scrub ok 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.881493+0000 osd.4 (osd.4) 547 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.881493+0000 osd.4 (osd.4) 547 : cluster [DBG] 6.1a scrub starts 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.883789+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:42:22.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:22 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.883789+0000 osd.4 (osd.4) 548 : cluster [DBG] 6.1a scrub ok 2024-04-26T16:42:23.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.946366+0000 osd.5 (osd.5) 498 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.946366+0000 osd.5 (osd.5) 498 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.952842+0000 osd.5 (osd.5) 499 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:21.952842+0000 osd.5 (osd.5) 499 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.122744+0000 mgr.y (mgr.24401) 947 : cluster [DBG] pgmap v1548: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.122744+0000 mgr.y (mgr.24401) 947 : cluster [DBG] pgmap v1548: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.125775+0000 osd.6 (osd.6) 157 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.125775+0000 osd.6 (osd.6) 157 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.129717+0000 osd.6 (osd.6) 158 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.129717+0000 osd.6 (osd.6) 158 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.877174+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.e scrub starts 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.877174+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.e scrub starts 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.879445+0000 osd.4 (osd.4) 550 : cluster [DBG] 6.e scrub ok 2024-04-26T16:42:23.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:23 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.879445+0000 osd.4 (osd.4) 550 : cluster [DBG] 6.e scrub ok 2024-04-26T16:42:23.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.946366+0000 osd.5 (osd.5) 498 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.946366+0000 osd.5 (osd.5) 498 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.952842+0000 osd.5 (osd.5) 499 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:21.952842+0000 osd.5 (osd.5) 499 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.122744+0000 mgr.y (mgr.24401) 947 : cluster [DBG] pgmap v1548: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.122744+0000 mgr.y (mgr.24401) 947 : cluster [DBG] pgmap v1548: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.125775+0000 osd.6 (osd.6) 157 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.125775+0000 osd.6 (osd.6) 157 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.129717+0000 osd.6 (osd.6) 158 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.129717+0000 osd.6 (osd.6) 158 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.877174+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.e scrub starts 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.877174+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.e scrub starts 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.879445+0000 osd.4 (osd.4) 550 : cluster [DBG] 6.e scrub ok 2024-04-26T16:42:23.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.879445+0000 osd.4 (osd.4) 550 : cluster [DBG] 6.e scrub ok 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.946366+0000 osd.5 (osd.5) 498 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.946366+0000 osd.5 (osd.5) 498 : cluster [DBG] 11.7 scrub starts 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.952842+0000 osd.5 (osd.5) 499 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:21.952842+0000 osd.5 (osd.5) 499 : cluster [DBG] 11.7 scrub ok 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.122744+0000 mgr.y (mgr.24401) 947 : cluster [DBG] pgmap v1548: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.122744+0000 mgr.y (mgr.24401) 947 : cluster [DBG] pgmap v1548: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.125775+0000 osd.6 (osd.6) 157 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.125775+0000 osd.6 (osd.6) 157 : cluster [DBG] 11.1 scrub starts 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.129717+0000 osd.6 (osd.6) 158 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.129717+0000 osd.6 (osd.6) 158 : cluster [DBG] 11.1 scrub ok 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.877174+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.e scrub starts 2024-04-26T16:42:23.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.877174+0000 osd.4 (osd.4) 549 : cluster [DBG] 6.e scrub starts 2024-04-26T16:42:23.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.879445+0000 osd.4 (osd.4) 550 : cluster [DBG] 6.e scrub ok 2024-04-26T16:42:23.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:23 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.879445+0000 osd.4 (osd.4) 550 : cluster [DBG] 6.e scrub ok 2024-04-26T16:42:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.969872+0000 osd.5 (osd.5) 500 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:42:24.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.969872+0000 osd.5 (osd.5) 500 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.975090+0000 osd.5 (osd.5) 501 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:22.975090+0000 osd.5 (osd.5) 501 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.066614+0000 osd.1 (osd.1) 583 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.066614+0000 osd.1 (osd.1) 583 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.068383+0000 osd.1 (osd.1) 584 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.068383+0000 osd.1 (osd.1) 584 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.164537+0000 osd.6 (osd.6) 159 : cluster [DBG] 11.3 deep-scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.164537+0000 osd.6 (osd.6) 159 : cluster [DBG] 11.3 deep-scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.169659+0000 osd.6 (osd.6) 160 : cluster [DBG] 11.3 deep-scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.169659+0000 osd.6 (osd.6) 160 : cluster [DBG] 11.3 deep-scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.915917+0000 osd.4 (osd.4) 551 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.915917+0000 osd.4 (osd.4) 551 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.918403+0000 osd.4 (osd.4) 552 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:42:24.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:24 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.918403+0000 osd.4 (osd.4) 552 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:42:24.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:42:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.969872+0000 osd.5 (osd.5) 500 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.969872+0000 osd.5 (osd.5) 500 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.975090+0000 osd.5 (osd.5) 501 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:22.975090+0000 osd.5 (osd.5) 501 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.066614+0000 osd.1 (osd.1) 583 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.066614+0000 osd.1 (osd.1) 583 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.068383+0000 osd.1 (osd.1) 584 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.068383+0000 osd.1 (osd.1) 584 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.164537+0000 osd.6 (osd.6) 159 : cluster [DBG] 11.3 deep-scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.164537+0000 osd.6 (osd.6) 159 : cluster [DBG] 11.3 deep-scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.169659+0000 osd.6 (osd.6) 160 : cluster [DBG] 11.3 deep-scrub ok 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.169659+0000 osd.6 (osd.6) 160 : cluster [DBG] 11.3 deep-scrub ok 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.915917+0000 osd.4 (osd.4) 551 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:42:24.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.915917+0000 osd.4 (osd.4) 551 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.918403+0000 osd.4 (osd.4) 552 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.918403+0000 osd.4 (osd.4) 552 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.969872+0000 osd.5 (osd.5) 500 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.969872+0000 osd.5 (osd.5) 500 : cluster [DBG] 11.13 scrub starts 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.975090+0000 osd.5 (osd.5) 501 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:22.975090+0000 osd.5 (osd.5) 501 : cluster [DBG] 11.13 scrub ok 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.066614+0000 osd.1 (osd.1) 583 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.066614+0000 osd.1 (osd.1) 583 : cluster [DBG] 6.1d scrub starts 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.068383+0000 osd.1 (osd.1) 584 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.068383+0000 osd.1 (osd.1) 584 : cluster [DBG] 6.1d scrub ok 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.164537+0000 osd.6 (osd.6) 159 : cluster [DBG] 11.3 deep-scrub starts 2024-04-26T16:42:24.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.164537+0000 osd.6 (osd.6) 159 : cluster [DBG] 11.3 deep-scrub starts 2024-04-26T16:42:24.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.169659+0000 osd.6 (osd.6) 160 : cluster [DBG] 11.3 deep-scrub ok 2024-04-26T16:42:24.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.169659+0000 osd.6 (osd.6) 160 : cluster [DBG] 11.3 deep-scrub ok 2024-04-26T16:42:24.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.915917+0000 osd.4 (osd.4) 551 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:42:24.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.915917+0000 osd.4 (osd.4) 551 : cluster [DBG] 6.2 scrub starts 2024-04-26T16:42:24.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.918403+0000 osd.4 (osd.4) 552 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:42:24.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:24 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.918403+0000 osd.4 (osd.4) 552 : cluster [DBG] 6.2 scrub ok 2024-04-26T16:42:25.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.609167+0000 osd.5 (osd.5) 502 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.609167+0000 osd.5 (osd.5) 502 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.609679+0000 osd.5 (osd.5) 503 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:23.609679+0000 osd.5 (osd.5) 503 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.050617+0000 osd.3 (osd.3) 655 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.050617+0000 osd.3 (osd.3) 655 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.051082+0000 osd.3 (osd.3) 656 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.051082+0000 osd.3 (osd.3) 656 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.123885+0000 mgr.y (mgr.24401) 948 : cluster [DBG] pgmap v1549: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.123885+0000 mgr.y (mgr.24401) 948 : cluster [DBG] pgmap v1549: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.171255+0000 osd.2 (osd.2) 535 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:42:25.825 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.171255+0000 osd.2 (osd.2) 535 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.173533+0000 osd.2 (osd.2) 536 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.173533+0000 osd.2 (osd.2) 536 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.353852+0000 osd.7 (osd.7) 644 : cluster [DBG] 3.e scrub starts 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.353852+0000 osd.7 (osd.7) 644 : cluster [DBG] 3.e scrub starts 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.358070+0000 osd.7 (osd.7) 645 : cluster [DBG] 3.e scrub ok 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.358070+0000 osd.7 (osd.7) 645 : cluster [DBG] 3.e scrub ok 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.584474+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.584474+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.959489+0000 osd.4 (osd.4) 553 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.959489+0000 osd.4 (osd.4) 553 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.964271+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:42:25.826 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:25 smithi190 bash[23868]: cluster 2024-04-26T16:42:24.964271+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.609167+0000 osd.5 (osd.5) 502 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.609167+0000 osd.5 (osd.5) 502 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.609679+0000 osd.5 (osd.5) 503 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:23.609679+0000 osd.5 (osd.5) 503 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.050617+0000 osd.3 (osd.3) 655 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.050617+0000 osd.3 (osd.3) 655 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.051082+0000 osd.3 (osd.3) 656 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.051082+0000 osd.3 (osd.3) 656 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.123885+0000 mgr.y (mgr.24401) 948 : cluster [DBG] pgmap v1549: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.123885+0000 mgr.y (mgr.24401) 948 : cluster [DBG] pgmap v1549: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.171255+0000 osd.2 (osd.2) 535 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.171255+0000 osd.2 (osd.2) 535 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.173533+0000 osd.2 (osd.2) 536 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.173533+0000 osd.2 (osd.2) 536 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.353852+0000 osd.7 (osd.7) 644 : cluster [DBG] 3.e scrub starts 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.353852+0000 osd.7 (osd.7) 644 : cluster [DBG] 3.e scrub starts 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.358070+0000 osd.7 (osd.7) 645 : cluster [DBG] 3.e scrub ok 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.358070+0000 osd.7 (osd.7) 645 : cluster [DBG] 3.e scrub ok 2024-04-26T16:42:25.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.584474+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.584474+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.959489+0000 osd.4 (osd.4) 553 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.959489+0000 osd.4 (osd.4) 553 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.964271+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[21463]: cluster 2024-04-26T16:42:24.964271+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.609167+0000 osd.5 (osd.5) 502 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.609167+0000 osd.5 (osd.5) 502 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.609679+0000 osd.5 (osd.5) 503 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:23.609679+0000 osd.5 (osd.5) 503 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.882 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.050617+0000 osd.3 (osd.3) 655 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.050617+0000 osd.3 (osd.3) 655 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.051082+0000 osd.3 (osd.3) 656 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.051082+0000 osd.3 (osd.3) 656 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.123885+0000 mgr.y (mgr.24401) 948 : cluster [DBG] pgmap v1549: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.123885+0000 mgr.y (mgr.24401) 948 : cluster [DBG] pgmap v1549: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.171255+0000 osd.2 (osd.2) 535 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.171255+0000 osd.2 (osd.2) 535 : cluster [DBG] 2.1 deep-scrub starts 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.173533+0000 osd.2 (osd.2) 536 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.173533+0000 osd.2 (osd.2) 536 : cluster [DBG] 2.1 deep-scrub ok 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.353852+0000 osd.7 (osd.7) 644 : cluster [DBG] 3.e scrub starts 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.353852+0000 osd.7 (osd.7) 644 : cluster [DBG] 3.e scrub starts 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.358070+0000 osd.7 (osd.7) 645 : cluster [DBG] 3.e scrub ok 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.358070+0000 osd.7 (osd.7) 645 : cluster [DBG] 3.e scrub ok 2024-04-26T16:42:25.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.584474+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2024-04-26T16:42:25.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.584474+0000 mon.a (mon.0) 4068 : cluster [DBG] osdmap e928: 8 total, 8 up, 8 in 2024-04-26T16:42:25.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.959489+0000 osd.4 (osd.4) 553 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:42:25.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.959489+0000 osd.4 (osd.4) 553 : cluster [DBG] 3.3 scrub starts 2024-04-26T16:42:25.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.964271+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:42:25.884 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:25 smithi050 bash[29234]: cluster 2024-04-26T16:42:24.964271+0000 osd.4 (osd.4) 554 : cluster [DBG] 3.3 scrub ok 2024-04-26T16:42:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/2627459985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006c520 msgr2=0x7f7460110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- 172.21.15.50:0/2627459985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006c520 0x7f7460110350 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f745800b0a0 tx=0x7f745800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/2627459985 shutdown_connections 2024-04-26T16:42:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- 172.21.15.50:0/2627459985 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006c520 0x7f7460110350 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.819 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- 172.21.15.50:0/2627459985 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 0x7f74601099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- 172.21.15.50:0/2627459985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f746006bb60 0x7f746006bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/2627459985 >> 172.21.15.50:0/2627459985 conn(0x7f7460079120 msgr2=0x7f7460079530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/2627459985 shutdown_connections 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/2627459985 wait complete. 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 librados: starting msgr at 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 librados: starting objecter 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 Processor -- start 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- start start 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 librados: setting wanted keys 2024-04-26T16:42:26.820 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 librados: calling monclient init 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006bb60 0x7f74601026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f746006c520 0x7f7460102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 0x7f7460103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f7460112b50 con 0x7f74601095c0 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f74601129d0 con 0x7f746006c520 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f7460112cd0 con 0x7f746006bb60 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745e59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006bb60 0x7f74601026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 0x7f7460103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 0x7f7460103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:32932/0 (socket says 172.21.15.50:32932) 2024-04-26T16:42:26.822 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 -- 172.21.15.50:0/3072613208 learned_addr learned my addr 172.21.15.50:0/3072613208 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745e59c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006bb60 0x7f74601026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:52412/0 (socket says 172.21.15.50:52412) 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745dd9b700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f746006c520 0x7f7460102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 -- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006bb60 msgr2=0x7f74601026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006bb60 0x7f74601026f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 -- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f746006c520 msgr2=0x7f7460102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f746006c520 0x7f7460102c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 -- 172.21.15.50:0/3072613208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74601a75b0 con 0x7f74601095c0 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745e59c700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f746006bb60 0x7f74601026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f745ed9d700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 0x7f7460103170 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f745800b780 tx=0x7f7458004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f744f7fe700 10 client.?.objecter ms_handle_connect 0x7f74601095c0 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f744f7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458021070 con 0x7f74601095c0 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/3072613208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74601a7840 con 0x7f74601095c0 2024-04-26T16:42:26.823 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f7458007b80 con 0x7f74601095c0 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458005040 con 0x7f74601095c0 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 10 client.31287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 -- 172.21.15.50:0/3072613208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74601a7cf0 con 0x7f74601095c0 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.821+0000 7f7464af6700 1 librados: init done 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744d7fa700 10 client.31287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744d7fa700 15 client.31287.objecter _session_command_op_assign -1 1 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744d7fa700 20 client.31287.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744d7fa700 10 client.31287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:26.824 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7444002920 con 0x7f74601095c0 2024-04-26T16:42:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f7458024070 con 0x7f74601095c0 2024-04-26T16:42:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7448071f80 0x7f7448074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:26.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(929..929 src has 874..929) ==== 8707+0+0 (secure 0 0 0) 0x7f745800f020 con 0x7f74601095c0 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 10 client.31287.objecter ms_dispatch 0x7f7460001550 osd_map(929..929 src has 874..929) 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 3 client.31287.objecter handle_osd_map got epochs [929,929] > 0 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 3 client.31287.objecter handle_osd_map decoding full epoch 929 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f745e59c700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7448071f80 0x7f7448074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 10 client.31287.objecter checking command 1 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f744807db30 0x7f744807ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 20 client.31287.objecter _get_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 15 client.31287.objecter _session_command_op_remove -1 1 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 20 client.31287.objecter _get_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 20 client.31287.objecter get_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 15 client.31287.objecter _session_command_op_assign 5 1 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 4 2024-04-26T16:42:26.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 10 client.31287.objecter _send_command 1 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f745801fe80 con 0x7f744807db30 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 20 client.31287.objecter dump_active .. 0 homeless 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=929) ==== 24+0+0 (secure 0 0 0) 0x7f74580972a0 con 0x7f74601095c0 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 10 client.31287.objecter ms_handle_connect 0x7f7448071f80 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f745e59c700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7448071f80 0x7f7448074440 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f7450009680 tx=0x7f7450009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f745dd9b700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f744807db30 0x7f744807ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f744f7fe700 10 client.31287.objecter ms_handle_connect 0x7f744807db30 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.825+0000 7f745dd9b700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f744807db30 0x7f744807ffd0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.829+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== osd.5 v2:172.21.15.190:6808/3373187844 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f745801fe80 con 0x7f744807db30 2024-04-26T16:42:26.827 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.829+0000 7f744f7fe700 10 client.31287.objecter ms_dispatch 0x7f7460001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.829+0000 7f744f7fe700 10 client.31287.objecter _finish_command 1 = system:0 2024-04-26T16:42:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.829+0000 7f744f7fe700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 4 2024-04-26T16:42:26.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.829+0000 7f744f7fe700 15 client.31287.objecter _session_command_op_remove 5 1 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 10 client.31287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 15 client.31287.objecter _session_command_op_assign -1 2 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 20 client.31287.objecter _get_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 20 client.31287.objecter _get_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 15 client.31287.objecter _session_command_op_remove -1 2 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 20 client.31287.objecter get_session s=0x7f744807d460 osd=5 3 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 15 client.31287.objecter _session_command_op_assign 5 2 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 4 2024-04-26T16:42:26.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 10 client.31287.objecter _send_command 2 2024-04-26T16:42:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 --> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7444001220 con 0x7f744807db30 2024-04-26T16:42:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744f7fe700 1 -- 172.21.15.50:0/3072613208 <== osd.5 v2:172.21.15.190:6808/3373187844 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7444001220 con 0x7f744807db30 2024-04-26T16:42:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744f7fe700 10 client.31287.objecter ms_dispatch 0x7f7460001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744f7fe700 10 client.31287.objecter _finish_command 2 = system:0 2024-04-26T16:42:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744f7fe700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 4 2024-04-26T16:42:26.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.845+0000 7f744f7fe700 15 client.31287.objecter _session_command_op_remove 5 2 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 10 librados: watch_flush enter 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 10 librados: watch_flush exit 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 10 client.31287.objecter close_session for osd.5 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f744807db30 msgr2=0x7f744807ffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.190:6808/3373187844,v1:172.21.15.190:6809/3373187844] conn(0x7f744807db30 0x7f744807ffd0 crc :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 20 client.31287.objecter put_session s=0x7f744807d460 osd=5 1 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 20 client.31287.objecter shutdown clearing up homeless session... 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 10 client.31287.objecter successfully canceled tick 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7448071f80 msgr2=0x7f7448074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f7448071f80 0x7f7448074440 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f7450009680 tx=0x7f7450009260 comp rx=0 tx=0).stop 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 msgr2=0x7f7460103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 --2- 172.21.15.50:0/3072613208 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f74601095c0 0x7f7460103170 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f745800b780 tx=0x7f7458004d00 comp rx=0 tx=0).stop 2024-04-26T16:42:26.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f745ed9d700 1 -- 172.21.15.50:0/3072613208 reap_dead start 2024-04-26T16:42:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 shutdown_connections 2024-04-26T16:42:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 >> 172.21.15.50:0/3072613208 conn(0x7f7460079120 msgr2=0x7f746010a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 shutdown_connections 2024-04-26T16:42:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 -- 172.21.15.50:0/3072613208 wait complete. 2024-04-26T16:42:26.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:26.849+0000 7f744d7fa700 1 librados: shutdown 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[21463]: cluster 2024-04-26T16:42:25.023272+0000 osd.3 (osd.3) 657 : cluster [DBG] 5.14 deep-scrub starts 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[21463]: cluster 2024-04-26T16:42:25.023272+0000 osd.3 (osd.3) 657 : cluster [DBG] 5.14 deep-scrub starts 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[21463]: cluster 2024-04-26T16:42:25.030731+0000 osd.3 (osd.3) 658 : cluster [DBG] 5.14 deep-scrub ok 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[21463]: cluster 2024-04-26T16:42:25.030731+0000 osd.3 (osd.3) 658 : cluster [DBG] 5.14 deep-scrub ok 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[29234]: cluster 2024-04-26T16:42:25.023272+0000 osd.3 (osd.3) 657 : cluster [DBG] 5.14 deep-scrub starts 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[29234]: cluster 2024-04-26T16:42:25.023272+0000 osd.3 (osd.3) 657 : cluster [DBG] 5.14 deep-scrub starts 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[29234]: cluster 2024-04-26T16:42:25.030731+0000 osd.3 (osd.3) 658 : cluster [DBG] 5.14 deep-scrub ok 2024-04-26T16:42:26.883 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:26 smithi050 bash[29234]: cluster 2024-04-26T16:42:25.030731+0000 osd.3 (osd.3) 658 : cluster [DBG] 5.14 deep-scrub ok 2024-04-26T16:42:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:26 smithi190 bash[23868]: cluster 2024-04-26T16:42:25.023272+0000 osd.3 (osd.3) 657 : cluster [DBG] 5.14 deep-scrub starts 2024-04-26T16:42:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:26 smithi190 bash[23868]: cluster 2024-04-26T16:42:25.023272+0000 osd.3 (osd.3) 657 : cluster [DBG] 5.14 deep-scrub starts 2024-04-26T16:42:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:26 smithi190 bash[23868]: cluster 2024-04-26T16:42:25.030731+0000 osd.3 (osd.3) 658 : cluster [DBG] 5.14 deep-scrub ok 2024-04-26T16:42:27.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:26 smithi190 bash[23868]: cluster 2024-04-26T16:42:25.030731+0000 osd.3 (osd.3) 658 : cluster [DBG] 5.14 deep-scrub ok 2024-04-26T16:42:27.230 INFO:teuthology.orchestra.run.smithi050.stdout:34359738750 2024-04-26T16:42:27.230 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-26T16:42:27.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.106853+0000 osd.2 (osd.2) 537 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.106853+0000 osd.2 (osd.2) 537 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.107272+0000 osd.2 (osd.2) 538 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.107272+0000 osd.2 (osd.2) 538 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.124644+0000 mgr.y (mgr.24401) 949 : cluster [DBG] pgmap v1551: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.124644+0000 mgr.y (mgr.24401) 949 : cluster [DBG] pgmap v1551: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.245573+0000 osd.2 (osd.2) 539 : cluster [DBG] 5.a scrub starts 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.245573+0000 osd.2 (osd.2) 539 : cluster [DBG] 5.a scrub starts 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.248154+0000 osd.2 (osd.2) 540 : cluster [DBG] 5.a scrub ok 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.248154+0000 osd.2 (osd.2) 540 : cluster [DBG] 5.a scrub ok 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.592593+0000 mon.a (mon.0) 4069 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2024-04-26T16:42:27.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[21463]: cluster 2024-04-26T16:42:26.592593+0000 mon.a (mon.0) 4069 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.106853+0000 osd.2 (osd.2) 537 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.106853+0000 osd.2 (osd.2) 537 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.107272+0000 osd.2 (osd.2) 538 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.107272+0000 osd.2 (osd.2) 538 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.124644+0000 mgr.y (mgr.24401) 949 : cluster [DBG] pgmap v1551: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.124644+0000 mgr.y (mgr.24401) 949 : cluster [DBG] pgmap v1551: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.245573+0000 osd.2 (osd.2) 539 : cluster [DBG] 5.a scrub starts 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.245573+0000 osd.2 (osd.2) 539 : cluster [DBG] 5.a scrub starts 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.248154+0000 osd.2 (osd.2) 540 : cluster [DBG] 5.a scrub ok 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.248154+0000 osd.2 (osd.2) 540 : cluster [DBG] 5.a scrub ok 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.592593+0000 mon.a (mon.0) 4069 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2024-04-26T16:42:27.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:27 smithi050 bash[29234]: cluster 2024-04-26T16:42:26.592593+0000 mon.a (mon.0) 4069 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2024-04-26T16:42:28.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:42:28.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.106853+0000 osd.2 (osd.2) 537 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:28.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.106853+0000 osd.2 (osd.2) 537 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.107272+0000 osd.2 (osd.2) 538 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.107272+0000 osd.2 (osd.2) 538 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.124644+0000 mgr.y (mgr.24401) 949 : cluster [DBG] pgmap v1551: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.124644+0000 mgr.y (mgr.24401) 949 : cluster [DBG] pgmap v1551: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.245573+0000 osd.2 (osd.2) 539 : cluster [DBG] 5.a scrub starts 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.245573+0000 osd.2 (osd.2) 539 : cluster [DBG] 5.a scrub starts 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.248154+0000 osd.2 (osd.2) 540 : cluster [DBG] 5.a scrub ok 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.248154+0000 osd.2 (osd.2) 540 : cluster [DBG] 5.a scrub ok 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.592593+0000 mon.a (mon.0) 4069 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2024-04-26T16:42:28.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:27 smithi190 bash[23868]: cluster 2024-04-26T16:42:26.592593+0000 mon.a (mon.0) 4069 : cluster [DBG] osdmap e929: 8 total, 8 up, 8 in 2024-04-26T16:42:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- 172.21.15.50:0/1660727908 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 msgr2=0x7fde38105a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- 172.21.15.50:0/1660727908 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde38105a90 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0098f0 tx=0x7fde2c009c00 comp rx=0 tx=0).stop 2024-04-26T16:42:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- 172.21.15.50:0/1660727908 shutdown_connections 2024-04-26T16:42:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- 172.21.15.50:0/1660727908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde3810b6a0 0x7fde3810ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.332 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- 172.21.15.50:0/1660727908 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde38105fd0 0x7fde381124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- 172.21.15.50:0/1660727908 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde38105a90 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- 172.21.15.50:0/1660727908 >> 172.21.15.50:0/1660727908 conn(0x7fde38101100 msgr2=0x7fde38103520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- 172.21.15.50:0/1660727908 shutdown_connections 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- 172.21.15.50:0/1660727908 wait complete. 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 librados: starting msgr at 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 librados: starting objecter 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 Processor -- start 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- start start 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 librados: setting wanted keys 2024-04-26T16:42:28.333 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 librados: calling monclient init 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde3819b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde38105fd0 0x7fde3819b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde3810b6a0 0x7fde3819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fde38114d30 con 0x7fde3810b6a0 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fde38114bb0 con 0x7fde38105fd0 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3e929700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fde38114eb0 con 0x7fde38105610 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde37fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde3819b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde3cec6700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde3810b6a0 0x7fde3819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.335 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde37fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde3819b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50648/0 (socket says 172.21.15.50:50648) 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde37fff700 1 -- 172.21.15.50:0/910853027 learned_addr learned my addr 172.21.15.50:0/910853027 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.333+0000 7fde377fe700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde38105fd0 0x7fde3819b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3cec6700 1 -- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 msgr2=0x7fde3819b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3cec6700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde3819b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3cec6700 1 -- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde38105fd0 msgr2=0x7fde3819b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3cec6700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde38105fd0 0x7fde3819b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3cec6700 1 -- 172.21.15.50:0/910853027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde381a0320 con 0x7fde3810b6a0 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde37fff700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fde38105610 0x7fde3819b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde377fe700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fde38105fd0 0x7fde3819b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3cec6700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde3810b6a0 0x7fde3819fbe0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fde30009cd0 tx=0x7fde3000c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 10 client.?.objecter ms_handle_connect 0x7fde3810b6a0 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3000cdc0 con 0x7fde3810b6a0 2024-04-26T16:42:28.336 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fde30003a00 con 0x7fde3810b6a0 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde381a05b0 con 0x7fde3810b6a0 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30004450 con 0x7fde3810b6a0 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 10 client.31293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde381a7e90 con 0x7fde3810b6a0 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 1 librados: init done 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 10 client.31293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 15 client.31293.objecter _session_command_op_assign -1 1 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 20 client.31293.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 10 client.31293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:28.337 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fde38106d50 con 0x7fde3810b6a0 2024-04-26T16:42:28.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fde3001a020 con 0x7fde3810b6a0 2024-04-26T16:42:28.338 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fde20071f80 0x7fde20074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(929..929 src has 874..929) ==== 8707+0+0 (secure 0 0 0) 0x7fde300910b0 con 0x7fde3810b6a0 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 10 client.31293.objecter ms_dispatch 0x7fde38001550 osd_map(929..929 src has 874..929) 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 3 client.31293.objecter handle_osd_map got epochs [929,929] > 0 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 3 client.31293.objecter handle_osd_map decoding full epoch 929 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde37fff700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fde20071f80 0x7fde20074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 10 client.31293.objecter checking command 1 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fde2007dce0 0x7fde20080180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 20 client.31293.objecter _get_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 15 client.31293.objecter _session_command_op_remove -1 1 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 20 client.31293.objecter _get_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 20 client.31293.objecter get_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 15 client.31293.objecter _session_command_op_assign 6 1 2024-04-26T16:42:28.339 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 4 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 10 client.31293.objecter _send_command 1 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde30021e50 con 0x7fde2007dce0 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 20 client.31293.objecter dump_active .. 0 homeless 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=929) ==== 24+0+0 (secure 0 0 0) 0x7fde30098050 con 0x7fde3810b6a0 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde357fa700 10 client.31293.objecter ms_handle_connect 0x7fde20071f80 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde37fff700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fde20071f80 0x7fde20074440 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7fde280104e0 tx=0x7fde2800b450 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.337+0000 7fde377fe700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fde2007dce0 0x7fde20080180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde377fe700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fde2007dce0 0x7fde20080180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:28.340 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde357fa700 10 client.31293.objecter ms_handle_connect 0x7fde2007dce0 2024-04-26T16:42:28.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== osd.6 v2:172.21.15.190:6816/338770817 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde30021e50 con 0x7fde2007dce0 2024-04-26T16:42:28.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde357fa700 10 client.31293.objecter ms_dispatch 0x7fde38001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:28.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde357fa700 10 client.31293.objecter _finish_command 1 = system:0 2024-04-26T16:42:28.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde357fa700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 4 2024-04-26T16:42:28.341 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.341+0000 7fde357fa700 15 client.31293.objecter _session_command_op_remove 6 1 2024-04-26T16:42:28.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 10 client.31293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:28.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 15 client.31293.objecter _session_command_op_assign -1 2 2024-04-26T16:42:28.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 20 client.31293.objecter _get_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.356 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 20 client.31293.objecter _get_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 15 client.31293.objecter _session_command_op_remove -1 2 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 20 client.31293.objecter get_session s=0x7fde2007d610 osd=6 3 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 15 client.31293.objecter _session_command_op_assign 6 2 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 4 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 10 client.31293.objecter _send_command 2 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.357+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 --> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fde3806a110 con 0x7fde2007dce0 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde357fa700 1 -- 172.21.15.50:0/910853027 <== osd.6 v2:172.21.15.190:6816/338770817 2 ==== command_reply(tid 2: 0 ) ==== 8+0+13 (crc 0 0 0) 0x7fde3806a110 con 0x7fde2007dce0 2024-04-26T16:42:28.357 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde357fa700 10 client.31293.objecter ms_dispatch 0x7fde38001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde357fa700 10 client.31293.objecter _finish_command 2 = system:0 2024-04-26T16:42:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde357fa700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 4 2024-04-26T16:42:28.358 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde357fa700 15 client.31293.objecter _session_command_op_remove 6 2 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 10 librados: watch_flush enter 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 10 librados: watch_flush exit 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 10 client.31293.objecter close_session for osd.6 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fde2007dce0 msgr2=0x7fde20080180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.190:6816/338770817,v1:172.21.15.190:6817/338770817] conn(0x7fde2007dce0 0x7fde20080180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 20 client.31293.objecter put_session s=0x7fde2007d610 osd=6 1 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 20 client.31293.objecter shutdown clearing up homeless session... 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 10 client.31293.objecter successfully canceled tick 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fde20071f80 msgr2=0x7fde20074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fde20071f80 0x7fde20074440 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7fde280104e0 tx=0x7fde2800b450 comp rx=0 tx=0).stop 2024-04-26T16:42:28.359 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde3810b6a0 msgr2=0x7fde3819fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 --2- 172.21.15.50:0/910853027 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fde3810b6a0 0x7fde3819fbe0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7fde30009cd0 tx=0x7fde3000c620 comp rx=0 tx=0).stop 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3cec6700 1 -- 172.21.15.50:0/910853027 reap_dead start 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 shutdown_connections 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 >> 172.21.15.50:0/910853027 conn(0x7fde38101100 msgr2=0x7fde38102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 shutdown_connections 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 -- 172.21.15.50:0/910853027 wait complete. 2024-04-26T16:42:28.360 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.361+0000 7fde3e929700 1 librados: shutdown 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 -- 172.21.15.50:0/200321134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a64109490 msgr2=0x7f5a64109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/200321134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a64109490 0x7f5a64109870 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f5a58009990 tx=0x7f5a58009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 -- 172.21.15.50:0/200321134 shutdown_connections 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/200321134 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a6406c360 0x7f5a641102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/200321134 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a64109490 0x7f5a64109870 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/200321134 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a6406b9c0 0x7f5a6406be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 -- 172.21.15.50:0/200321134 >> 172.21.15.50:0/200321134 conn(0x7f5a64079290 msgr2=0x7f5a640796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 -- 172.21.15.50:0/200321134 shutdown_connections 2024-04-26T16:42:28.641 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.641+0000 7f5a6aae1700 1 -- 172.21.15.50:0/200321134 wait complete. 2024-04-26T16:42:28.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 librados: starting msgr at 2024-04-26T16:42:28.643 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 librados: starting objecter 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 Processor -- start 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- start start 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 librados: setting wanted keys 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 librados: calling monclient init 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6406b9c0 0x7f5a641023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a6406c360 0x7f5a64102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a64109490 0x7f5a64102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f5a64112a70 con 0x7f5a6406b9c0 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f5a641128f0 con 0x7f5a6406c360 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f5a64112bf0 con 0x7f5a64109490 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6406b9c0 0x7f5a641023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6406b9c0 0x7f5a641023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:57980/0 (socket says 172.21.15.50:57980) 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 -- 172.21.15.50:0/425409054 learned_addr learned my addr 172.21.15.50:0/425409054 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:28.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6907e700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a64109490 0x7f5a64102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a63fff700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a6406c360 0x7f5a64102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 -- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a64109490 msgr2=0x7f5a64102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f5a64109490 0x7f5a64102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 -- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a6406c360 msgr2=0x7f5a64102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a6406c360 0x7f5a64102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 -- 172.21.15.50:0/425409054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a641a7740 con 0x7f5a6406b9c0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6887d700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6406b9c0 0x7f5a641023e0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f5a580057a0 tx=0x7f5a580057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a61ffb700 10 client.?.objecter ms_handle_connect 0x7f5a6406b9c0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a61ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a58026070 con 0x7f5a6406b9c0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f5a58021410 con 0x7f5a6406b9c0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a641a79d0 con 0x7f5a6406b9c0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 10 client.31299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a641a7eb0 con 0x7f5a6406b9c0 2024-04-26T16:42:28.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 librados: init done 2024-04-26T16:42:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a580173f0 con 0x7f5a6406b9c0 2024-04-26T16:42:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 10 client.31299.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 15 client.31299.objecter _session_command_op_assign -1 1 2024-04-26T16:42:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 20 client.31299.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 10 client.31299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:28.646 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.645+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5a48002920 con 0x7f5a6406b9c0 2024-04-26T16:42:28.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f5a58029070 con 0x7f5a6406b9c0 2024-04-26T16:42:28.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5a54071f80 0x7f5a54074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(929..929 src has 874..929) ==== 8707+0+0 (secure 0 0 0) 0x7f5a58092950 con 0x7f5a6406b9c0 2024-04-26T16:42:28.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter ms_dispatch 0x7f5a64001550 osd_map(929..929 src has 874..929) 2024-04-26T16:42:28.647 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 3 client.31299.objecter handle_osd_map got epochs [929,929] > 0 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a63fff700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5a54071f80 0x7f5a54074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 3 client.31299.objecter handle_osd_map decoding full epoch 929 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter checking command 1 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f5a5407db30 0x7f5a5407ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter _get_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 15 client.31299.objecter _session_command_op_remove -1 1 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter _get_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter get_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 15 client.31299.objecter _session_command_op_assign 4 1 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 4 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter _send_command 1 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a540806a0 con 0x7f5a5407db30 2024-04-26T16:42:28.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter dump_active .. 0 homeless 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=929) ==== 24+0+0 (secure 0 0 0) 0x7f5a58092da0 con 0x7f5a6406b9c0 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a63fff700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5a54071f80 0x7f5a54074440 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7f5a50005950 tx=0x7f5a500058e0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter ms_handle_connect 0x7f5a54071f80 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a6907e700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f5a5407db30 0x7f5a5407ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter ms_handle_connect 0x7f5a5407db30 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a6907e700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f5a5407db30 0x7f5a5407ffd0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== osd.4 v2:172.21.15.190:6800/2207965503 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5a540806a0 con 0x7f5a5407db30 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter ms_dispatch 0x7f5a64001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:28.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 10 client.31299.objecter _finish_command 1 = system:0 2024-04-26T16:42:28.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 4 2024-04-26T16:42:28.650 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.649+0000 7f5a61ffb700 15 client.31299.objecter _session_command_op_remove 4 1 2024-04-26T16:42:28.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 10 client.31299.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:28.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 15 client.31299.objecter _session_command_op_assign -1 2 2024-04-26T16:42:28.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter _get_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.666 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter _get_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 15 client.31299.objecter _session_command_op_remove -1 2 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter get_session s=0x7f5a5407d460 osd=4 3 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 15 client.31299.objecter _session_command_op_assign 4 2 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 4 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 10 client.31299.objecter _send_command 2 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 --> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5a48001240 con 0x7f5a5407db30 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a61ffb700 1 -- 172.21.15.50:0/425409054 <== osd.4 v2:172.21.15.190:6800/2207965503 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5a48001240 con 0x7f5a5407db30 2024-04-26T16:42:28.667 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a61ffb700 10 client.31299.objecter ms_dispatch 0x7f5a64001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:28.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a61ffb700 10 client.31299.objecter _finish_command 2 = system:0 2024-04-26T16:42:28.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a61ffb700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 4 2024-04-26T16:42:28.668 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a61ffb700 15 client.31299.objecter _session_command_op_remove 4 2 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 10 librados: watch_flush enter 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 10 librados: watch_flush exit 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 10 client.31299.objecter close_session for osd.4 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f5a5407db30 msgr2=0x7f5a5407ffd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.190:6800/2207965503,v1:172.21.15.190:6801/2207965503] conn(0x7f5a5407db30 0x7f5a5407ffd0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter put_session s=0x7f5a5407d460 osd=4 1 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 20 client.31299.objecter shutdown clearing up homeless session... 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 10 client.31299.objecter successfully canceled tick 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5a54071f80 msgr2=0x7f5a54074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.669 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f5a54071f80 0x7f5a54074440 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7f5a50005950 tx=0x7f5a500058e0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6406b9c0 msgr2=0x7f5a641023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 --2- 172.21.15.50:0/425409054 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f5a6406b9c0 0x7f5a641023e0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f5a580057a0 tx=0x7f5a580057d0 comp rx=0 tx=0).stop 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6907e700 1 -- 172.21.15.50:0/425409054 reap_dead start 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 shutdown_connections 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 >> 172.21.15.50:0/425409054 conn(0x7f5a64079290 msgr2=0x7f5a641001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 shutdown_connections 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 -- 172.21.15.50:0/425409054 wait complete. 2024-04-26T16:42:28.670 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:28.669+0000 7f5a6aae1700 1 librados: shutdown 2024-04-26T16:42:28.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[29234]: audit 2024-04-26T16:42:27.610605+0000 mgr.y (mgr.24401) 950 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:28.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[29234]: audit 2024-04-26T16:42:27.610605+0000 mgr.y (mgr.24401) 950 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:28.879 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[29234]: cluster 2024-04-26T16:42:27.971439+0000 osd.4 (osd.4) 555 : cluster [DBG] 2.f scrub starts 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[29234]: cluster 2024-04-26T16:42:27.971439+0000 osd.4 (osd.4) 555 : cluster [DBG] 2.f scrub starts 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[29234]: cluster 2024-04-26T16:42:27.974705+0000 osd.4 (osd.4) 556 : cluster [DBG] 2.f scrub ok 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[29234]: cluster 2024-04-26T16:42:27.974705+0000 osd.4 (osd.4) 556 : cluster [DBG] 2.f scrub ok 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[21463]: audit 2024-04-26T16:42:27.610605+0000 mgr.y (mgr.24401) 950 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[21463]: audit 2024-04-26T16:42:27.610605+0000 mgr.y (mgr.24401) 950 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[21463]: cluster 2024-04-26T16:42:27.971439+0000 osd.4 (osd.4) 555 : cluster [DBG] 2.f scrub starts 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[21463]: cluster 2024-04-26T16:42:27.971439+0000 osd.4 (osd.4) 555 : cluster [DBG] 2.f scrub starts 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[21463]: cluster 2024-04-26T16:42:27.974705+0000 osd.4 (osd.4) 556 : cluster [DBG] 2.f scrub ok 2024-04-26T16:42:28.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:28 smithi050 bash[21463]: cluster 2024-04-26T16:42:27.974705+0000 osd.4 (osd.4) 556 : cluster [DBG] 2.f scrub ok 2024-04-26T16:42:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:28 smithi190 bash[23868]: audit 2024-04-26T16:42:27.610605+0000 mgr.y (mgr.24401) 950 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:28 smithi190 bash[23868]: audit 2024-04-26T16:42:27.610605+0000 mgr.y (mgr.24401) 950 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:29.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:28 smithi190 bash[23868]: cluster 2024-04-26T16:42:27.971439+0000 osd.4 (osd.4) 555 : cluster [DBG] 2.f scrub starts 2024-04-26T16:42:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:28 smithi190 bash[23868]: cluster 2024-04-26T16:42:27.971439+0000 osd.4 (osd.4) 555 : cluster [DBG] 2.f scrub starts 2024-04-26T16:42:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:28 smithi190 bash[23868]: cluster 2024-04-26T16:42:27.974705+0000 osd.4 (osd.4) 556 : cluster [DBG] 2.f scrub ok 2024-04-26T16:42:29.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:28 smithi190 bash[23868]: cluster 2024-04-26T16:42:27.974705+0000 osd.4 (osd.4) 556 : cluster [DBG] 2.f scrub ok 2024-04-26T16:42:29.851 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:28.126372+0000 mgr.y (mgr.24401) 951 : cluster [DBG] pgmap v1553: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:28.126372+0000 mgr.y (mgr.24401) 951 : cluster [DBG] pgmap v1553: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: audit 2024-04-26T16:42:28.635739+0000 mon.c (mon.2) 951 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: audit 2024-04-26T16:42:28.635739+0000 mon.c (mon.2) 951 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:28.952506+0000 osd.3 (osd.3) 659 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:28.952506+0000 osd.3 (osd.3) 659 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:28.955943+0000 osd.3 (osd.3) 660 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:28.955943+0000 osd.3 (osd.3) 660 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:42:29.852 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.096815+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.d scrub starts 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.096815+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.d scrub starts 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.099246+0000 osd.5 (osd.5) 505 : cluster [DBG] 6.d scrub ok 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.099246+0000 osd.5 (osd.5) 505 : cluster [DBG] 6.d scrub ok 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:28.126372+0000 mgr.y (mgr.24401) 951 : cluster [DBG] pgmap v1553: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:28.126372+0000 mgr.y (mgr.24401) 951 : cluster [DBG] pgmap v1553: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: audit 2024-04-26T16:42:28.635739+0000 mon.c (mon.2) 951 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: audit 2024-04-26T16:42:28.635739+0000 mon.c (mon.2) 951 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:28.952506+0000 osd.3 (osd.3) 659 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:28.952506+0000 osd.3 (osd.3) 659 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:28.955943+0000 osd.3 (osd.3) 660 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:28.955943+0000 osd.3 (osd.3) 660 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:42:29.853 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.096815+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.d scrub starts 2024-04-26T16:42:29.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.096815+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.d scrub starts 2024-04-26T16:42:29.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.099246+0000 osd.5 (osd.5) 505 : cluster [DBG] 6.d scrub ok 2024-04-26T16:42:29.854 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:29 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.099246+0000 osd.5 (osd.5) 505 : cluster [DBG] 6.d scrub ok 2024-04-26T16:42:30.004 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 -- 172.21.15.50:0/845898263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812810fec0 msgr2=0x7f81281122b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 --2- 172.21.15.50:0/845898263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812810fec0 0x7f81281122b0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7f812000b0a0 tx=0x7f812000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 -- 172.21.15.50:0/845898263 shutdown_connections 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 --2- 172.21.15.50:0/845898263 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812810fec0 0x7f81281122b0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 --2- 172.21.15.50:0/845898263 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f812806b9c0 0x7f812806bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 --2- 172.21.15.50:0/845898263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812806c2e0 0x7f812810f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 -- 172.21.15.50:0/845898263 >> 172.21.15.50:0/845898263 conn(0x7f8128079290 msgr2=0x7f81280796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 -- 172.21.15.50:0/845898263 shutdown_connections 2024-04-26T16:42:30.005 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.005+0000 7f813075f700 1 -- 172.21.15.50:0/845898263 wait complete. 2024-04-26T16:42:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 librados: starting msgr at 2024-04-26T16:42:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 librados: starting objecter 2024-04-26T16:42:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 Processor -- start 2024-04-26T16:42:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- start start 2024-04-26T16:42:30.007 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 librados: setting wanted keys 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 librados: calling monclient init 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812806b9c0 0x7f812819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812806c2e0 0x7f812819b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f812810fec0 0x7f812819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8128114d60 con 0x7f812806b9c0 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8128114be0 con 0x7f812806c2e0 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8128114ee0 con 0x7f812810fec0 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812806b9c0 0x7f812819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812806b9c0 0x7f812819b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:58002/0 (socket says 172.21.15.50:58002) 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 -- 172.21.15.50:0/4251715790 learned_addr learned my addr 172.21.15.50:0/4251715790 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812ecfc700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f812810fec0 0x7f812819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812dcfa700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812806c2e0 0x7f812819b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 -- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f812810fec0 msgr2=0x7f812819fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:30.008 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f812810fec0 0x7f812819fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 -- 172.21.15.50:0/4251715790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812806c2e0 msgr2=0x7f812819b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f812806c2e0 0x7f812819b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 -- 172.21.15.50:0/4251715790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81281a0350 con 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812ecfc700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f812810fec0 0x7f812819fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f812e4fb700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812806b9c0 0x7f812819b340 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f812400e9a0 tx=0x7f812400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f811f7fe700 10 client.?.objecter ms_handle_connect 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f811f7fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812400cbd0 con 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f812400cd70 con 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81281a0640 con 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8124018420 con 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 10 client.31305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81281a7ec0 con 0x7f812806b9c0 2024-04-26T16:42:30.009 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 librados: init done 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 10 client.31305.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 15 client.31305.objecter _session_command_op_assign -1 1 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 20 client.31305.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 10 client.31305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8128102930 con 0x7f812806b9c0 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.009+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8124010430 con 0x7f812806b9c0 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8114071f80 0x7f8114074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(929..929 src has 874..929) ==== 8707+0+0 (secure 0 0 0) 0x7f81240921b0 con 0x7f812806b9c0 2024-04-26T16:42:30.011 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter ms_dispatch 0x7f8128001550 osd_map(929..929 src has 874..929) 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 3 client.31305.objecter handle_osd_map got epochs [929,929] > 0 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 3 client.31305.objecter handle_osd_map decoding full epoch 929 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f812dcfa700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8114071f80 0x7f8114074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter checking command 1 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f811407dce0 0x7f8114080180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter _get_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 15 client.31305.objecter _session_command_op_remove -1 1 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter _get_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter get_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 15 client.31305.objecter _session_command_op_assign 1 1 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 4 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter _send_command 1 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8124019ec0 con 0x7f811407dce0 2024-04-26T16:42:30.012 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter dump_active .. 0 homeless 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=929) ==== 24+0+0 (secure 0 0 0) 0x7f8124092600 con 0x7f812806b9c0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f812ecfc700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f811407dce0 0x7f8114080180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter ms_handle_connect 0x7f8114071f80 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f812dcfa700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8114071f80 0x7f8114074440 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f8118006fd0 tx=0x7f8118008040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter ms_handle_connect 0x7f811407dce0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f812ecfc700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f811407dce0 0x7f8114080180 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== osd.1 v2:172.21.15.50:6810/3516275849 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8124019ec0 con 0x7f811407dce0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter ms_dispatch 0x7f8128001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 10 client.31305.objecter _finish_command 1 = system:0 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 4 2024-04-26T16:42:30.013 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.013+0000 7f811f7fe700 15 client.31305.objecter _session_command_op_remove 1 1 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 10 client.31305.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 15 client.31305.objecter _session_command_op_assign -1 2 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 20 client.31305.objecter _get_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 20 client.31305.objecter _get_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 15 client.31305.objecter _session_command_op_remove -1 2 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 20 client.31305.objecter get_session s=0x7f811407d610 osd=1 3 2024-04-26T16:42:30.029 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 15 client.31305.objecter _session_command_op_assign 1 2 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 4 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 10 client.31305.objecter _send_command 2 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 --> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f812806a110 con 0x7f811407dce0 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f811f7fe700 1 -- 172.21.15.50:0/4251715790 <== osd.1 v2:172.21.15.50:6810/3516275849 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f812806a110 con 0x7f811407dce0 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f811f7fe700 10 client.31305.objecter ms_dispatch 0x7f8128001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f811f7fe700 10 client.31305.objecter _finish_command 2 = system:0 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f811f7fe700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 4 2024-04-26T16:42:30.030 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.029+0000 7f811f7fe700 15 client.31305.objecter _session_command_op_remove 1 2 2024-04-26T16:42:30.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 10 librados: watch_flush enter 2024-04-26T16:42:30.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 10 librados: watch_flush exit 2024-04-26T16:42:30.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 10 client.31305.objecter close_session for osd.1 2024-04-26T16:42:30.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f811407dce0 msgr2=0x7f8114080180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:30.031 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6810/3516275849,v1:172.21.15.50:6811/3516275849] conn(0x7f811407dce0 0x7f8114080180 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 20 client.31305.objecter put_session s=0x7f811407d610 osd=1 1 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 20 client.31305.objecter shutdown clearing up homeless session... 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 10 client.31305.objecter successfully canceled tick 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8114071f80 msgr2=0x7f8114074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8114071f80 0x7f8114074440 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f8118006fd0 tx=0x7f8118008040 comp rx=0 tx=0).stop 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812806b9c0 msgr2=0x7f812819b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 --2- 172.21.15.50:0/4251715790 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f812806b9c0 0x7f812819b340 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f812400e9a0 tx=0x7f812400ecb0 comp rx=0 tx=0).stop 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f812ecfc700 1 -- 172.21.15.50:0/4251715790 reap_dead start 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 shutdown_connections 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 >> 172.21.15.50:0/4251715790 conn(0x7f8128079290 msgr2=0x7f8128111130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 shutdown_connections 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 -- 172.21.15.50:0/4251715790 wait complete. 2024-04-26T16:42:30.032 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:30.033+0000 7f813075f700 1 librados: shutdown 2024-04-26T16:42:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:28.126372+0000 mgr.y (mgr.24401) 951 : cluster [DBG] pgmap v1553: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:30.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:28.126372+0000 mgr.y (mgr.24401) 951 : cluster [DBG] pgmap v1553: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: audit 2024-04-26T16:42:28.635739+0000 mon.c (mon.2) 951 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: audit 2024-04-26T16:42:28.635739+0000 mon.c (mon.2) 951 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:28.952506+0000 osd.3 (osd.3) 659 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:28.952506+0000 osd.3 (osd.3) 659 : cluster [DBG] 2.19 scrub starts 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:28.955943+0000 osd.3 (osd.3) 660 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:28.955943+0000 osd.3 (osd.3) 660 : cluster [DBG] 2.19 scrub ok 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.096815+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.d scrub starts 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.096815+0000 osd.5 (osd.5) 504 : cluster [DBG] 6.d scrub starts 2024-04-26T16:42:30.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.099246+0000 osd.5 (osd.5) 505 : cluster [DBG] 6.d scrub ok 2024-04-26T16:42:30.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:29 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.099246+0000 osd.5 (osd.5) 505 : cluster [DBG] 6.d scrub ok 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.023443+0000 osd.1 (osd.1) 585 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.023443+0000 osd.1 (osd.1) 585 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.025363+0000 osd.1 (osd.1) 586 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.025363+0000 osd.1 (osd.1) 586 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.228196+0000 osd.2 (osd.2) 541 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.228196+0000 osd.2 (osd.2) 541 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.230440+0000 osd.2 (osd.2) 542 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:29.230440+0000 osd.2 (osd.2) 542 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.099821+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.099821+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.104075+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:42:30.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.104075+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.023443+0000 osd.1 (osd.1) 585 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.023443+0000 osd.1 (osd.1) 585 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.025363+0000 osd.1 (osd.1) 586 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.025363+0000 osd.1 (osd.1) 586 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.228196+0000 osd.2 (osd.2) 541 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.228196+0000 osd.2 (osd.2) 541 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.230440+0000 osd.2 (osd.2) 542 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:29.230440+0000 osd.2 (osd.2) 542 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.099821+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.099821+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.104075+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:42:30.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:30 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.104075+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:42:30.889 INFO:teuthology.orchestra.run.smithi050.stdout:201863463243 2024-04-26T16:42:30.889 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-26T16:42:31.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.023443+0000 osd.1 (osd.1) 585 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.023443+0000 osd.1 (osd.1) 585 : cluster [DBG] 2.9 scrub starts 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.025363+0000 osd.1 (osd.1) 586 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.025363+0000 osd.1 (osd.1) 586 : cluster [DBG] 2.9 scrub ok 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.228196+0000 osd.2 (osd.2) 541 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.228196+0000 osd.2 (osd.2) 541 : cluster [DBG] 5.8 scrub starts 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.230440+0000 osd.2 (osd.2) 542 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:29.230440+0000 osd.2 (osd.2) 542 : cluster [DBG] 5.8 scrub ok 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.099821+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.099821+0000 osd.5 (osd.5) 506 : cluster [DBG] 3.1d scrub starts 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.104075+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:42:31.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:30 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.104075+0000 osd.5 (osd.5) 507 : cluster [DBG] 3.1d scrub ok 2024-04-26T16:42:31.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.039947+0000 osd.1 (osd.1) 587 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:42:31.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.039947+0000 osd.1 (osd.1) 587 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:42:31.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.042846+0000 osd.1 (osd.1) 588 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:42:31.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.042846+0000 osd.1 (osd.1) 588 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:42:31.922 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.127147+0000 mgr.y (mgr.24401) 952 : cluster [DBG] pgmap v1554: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:30.127147+0000 mgr.y (mgr.24401) 952 : cluster [DBG] pgmap v1554: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.109978+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.a scrub starts 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.109978+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.a scrub starts 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.112576+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.a scrub ok 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.112576+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.a scrub ok 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.039947+0000 osd.1 (osd.1) 587 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.039947+0000 osd.1 (osd.1) 587 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.042846+0000 osd.1 (osd.1) 588 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.042846+0000 osd.1 (osd.1) 588 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.127147+0000 mgr.y (mgr.24401) 952 : cluster [DBG] pgmap v1554: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:30.127147+0000 mgr.y (mgr.24401) 952 : cluster [DBG] pgmap v1554: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:31.923 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.109978+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.a scrub starts 2024-04-26T16:42:31.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.109978+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.a scrub starts 2024-04-26T16:42:31.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.112576+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.a scrub ok 2024-04-26T16:42:31.924 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:31 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.112576+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.a scrub ok 2024-04-26T16:42:31.981 INFO:teuthology.orchestra.run.smithi050.stdout:150323855701 2024-04-26T16:42:31.981 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-26T16:42:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 -- 172.21.15.50:0/4157584926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 msgr2=0x7fd250110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 --2- 172.21.15.50:0/4157584926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd250110330 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7fd248009f90 tx=0x7fd24800b330 comp rx=0 tx=0).stop 2024-04-26T16:42:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 -- 172.21.15.50:0/4157584926 shutdown_connections 2024-04-26T16:42:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 --2- 172.21.15.50:0/4157584926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd250109560 0x7fd250109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 --2- 172.21.15.50:0/4157584926 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd250110330 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.072 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 --2- 172.21.15.50:0/4157584926 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd2501034c0 0x7fd250103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 -- 172.21.15.50:0/4157584926 >> 172.21.15.50:0/4157584926 conn(0x7fd25007a5d0 msgr2=0x7fd25007a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 -- 172.21.15.50:0/4157584926 shutdown_connections 2024-04-26T16:42:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.073+0000 7fd2584fc700 1 -- 172.21.15.50:0/4157584926 wait complete. 2024-04-26T16:42:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 librados: starting msgr at 2024-04-26T16:42:32.073 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 librados: starting objecter 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 Processor -- start 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- start start 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 librados: setting wanted keys 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 librados: calling monclient init 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2501034c0 0x7fd25019b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd25019b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd250109560 0x7fd25019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fd250076b40 con 0x7fd250109560 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fd2500769c0 con 0x7fd2501034c0 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fd250076cc0 con 0x7fd250103e80 2024-04-26T16:42:32.075 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd25019b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd256a99700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd250109560 0x7fd25019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd25019b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50706/0 (socket says 172.21.15.50:50706) 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 -- 172.21.15.50:0/3213021049 learned_addr learned my addr 172.21.15.50:0/3213021049 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 -- 172.21.15.50:0/3213021049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2501034c0 msgr2=0x7fd25019b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd256298700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2501034c0 0x7fd25019b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2501034c0 0x7fd25019b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 -- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd250109560 msgr2=0x7fd25019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd250109560 0x7fd25019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 -- 172.21.15.50:0/3213021049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2501a0200 con 0x7fd250103e80 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd256298700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2501034c0 0x7fd25019b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd256a99700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fd250109560 0x7fd25019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd255a97700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd25019b730 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fd24000ec10 tx=0x7fd24000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2477fe700 10 client.?.objecter ms_handle_connect 0x7fd250103e80 2024-04-26T16:42:32.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2477fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24000cd40 con 0x7fd250103e80 2024-04-26T16:42:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2501a04f0 con 0x7fd250103e80 2024-04-26T16:42:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 10 client.41930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fd240004480 con 0x7fd250103e80 2024-04-26T16:42:32.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2501a7d70 con 0x7fd250103e80 2024-04-26T16:42:32.077 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.039947+0000 osd.1 (osd.1) 587 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.039947+0000 osd.1 (osd.1) 587 : cluster [DBG] 3.0 scrub starts 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.042846+0000 osd.1 (osd.1) 588 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.042846+0000 osd.1 (osd.1) 588 : cluster [DBG] 3.0 scrub ok 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.127147+0000 mgr.y (mgr.24401) 952 : cluster [DBG] pgmap v1554: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:30.127147+0000 mgr.y (mgr.24401) 952 : cluster [DBG] pgmap v1554: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.109978+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.a scrub starts 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.109978+0000 osd.5 (osd.5) 508 : cluster [DBG] 1.a scrub starts 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.112576+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.a scrub ok 2024-04-26T16:42:32.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:31 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.112576+0000 osd.5 (osd.5) 509 : cluster [DBG] 1.a scrub ok 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 librados: init done 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd240010430 con 0x7fd250103e80 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 10 client.41930.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 15 client.41930.objecter _session_command_op_assign -1 1 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 20 client.41930.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 10 client.41930.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd250104bd0 con 0x7fd250103e80 2024-04-26T16:42:32.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.077+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fd2400105d0 con 0x7fd250103e80 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd23c071f80 0x7fd23c074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(929..929 src has 874..929) ==== 8707+0+0 (secure 0 0 0) 0x7fd240092900 con 0x7fd250103e80 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter ms_dispatch 0x7fd250001550 osd_map(929..929 src has 874..929) 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 3 client.41930.objecter handle_osd_map got epochs [929,929] > 0 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 3 client.41930.objecter handle_osd_map decoding full epoch 929 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter checking command 1 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fd23c07dbe0 0x7fd23c080080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter _get_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd256298700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd23c071f80 0x7fd23c074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 15 client.41930.objecter _session_command_op_remove -1 1 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter _get_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter get_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.080 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 15 client.41930.objecter _session_command_op_assign 2 1 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 4 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter _send_command 1 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd23c080750 con 0x7fd23c07dbe0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter dump_active .. 0 homeless 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_get_version_reply(handle=1 version=929) ==== 24+0+0 (secure 0 0 0) 0x7fd24005a110 con 0x7fd250103e80 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd256a99700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fd23c07dbe0 0x7fd23c080080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter ms_handle_connect 0x7fd23c071f80 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd256298700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd23c071f80 0x7fd23c074440 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7fd24c005fd0 tx=0x7fd24c005e20 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter ms_handle_connect 0x7fd23c07dbe0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd256a99700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fd23c07dbe0 0x7fd23c080080 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== osd.2 v2:172.21.15.50:6818/2835380668 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd23c080750 con 0x7fd23c07dbe0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter ms_dispatch 0x7fd250001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 10 client.41930.objecter _finish_command 1 = system:0 2024-04-26T16:42:32.081 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 4 2024-04-26T16:42:32.082 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.081+0000 7fd2477fe700 15 client.41930.objecter _session_command_op_remove 2 1 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 10 client.41930.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 15 client.41930.objecter _session_command_op_assign -1 2 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter _get_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter _get_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 15 client.41930.objecter _session_command_op_remove -1 2 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter get_session s=0x7fd23c07d510 osd=2 3 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 15 client.41930.objecter _session_command_op_assign 2 2 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 4 2024-04-26T16:42:32.099 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 10 client.41930.objecter _send_command 2 2024-04-26T16:42:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 --> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd25006a110 con 0x7fd23c07dbe0 2024-04-26T16:42:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2477fe700 1 -- 172.21.15.50:0/3213021049 <== osd.2 v2:172.21.15.50:6818/2835380668 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd25006a110 con 0x7fd23c07dbe0 2024-04-26T16:42:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2477fe700 10 client.41930.objecter ms_dispatch 0x7fd250001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2477fe700 10 client.41930.objecter _finish_command 2 = system:0 2024-04-26T16:42:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2477fe700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 4 2024-04-26T16:42:32.100 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2477fe700 15 client.41930.objecter _session_command_op_remove 2 2 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 10 librados: watch_flush enter 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 10 librados: watch_flush exit 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 10 client.41930.objecter close_session for osd.2 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fd23c07dbe0 msgr2=0x7fd23c080080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6818/2835380668,v1:172.21.15.50:6819/2835380668] conn(0x7fd23c07dbe0 0x7fd23c080080 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter put_session s=0x7fd23c07d510 osd=2 1 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 20 client.41930.objecter shutdown clearing up homeless session... 2024-04-26T16:42:32.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 10 client.41930.objecter successfully canceled tick 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd23c071f80 msgr2=0x7fd23c074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fd23c071f80 0x7fd23c074440 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7fd24c005fd0 tx=0x7fd24c005e20 comp rx=0 tx=0).stop 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 msgr2=0x7fd25019b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 --2- 172.21.15.50:0/3213021049 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fd250103e80 0x7fd25019b730 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7fd24000ec10 tx=0x7fd24000ef20 comp rx=0 tx=0).stop 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd256a99700 1 -- 172.21.15.50:0/3213021049 reap_dead start 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 shutdown_connections 2024-04-26T16:42:32.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.101+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 >> 172.21.15.50:0/3213021049 conn(0x7fd25007a5d0 msgr2=0x7fd250107060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:32.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.105+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 shutdown_connections 2024-04-26T16:42:32.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.105+0000 7fd2584fc700 1 -- 172.21.15.50:0/3213021049 wait complete. 2024-04-26T16:42:32.105 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.105+0000 7fd2584fc700 1 librados: shutdown 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4145830060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c1098e0 msgr2=0x7f8f1c110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4145830060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c1098e0 0x7f8f1c110310 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7f8f14009fc0 tx=0x7f8f1400b330 comp rx=0 tx=0).stop 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4145830060 shutdown_connections 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4145830060 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c1098e0 0x7f8f1c110310 unknown :-1 s=CLOSED pgs=2801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4145830060 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1c101800 0x7f8f1c101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4145830060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f1c1021b0 0x7f8f1c109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4145830060 >> 172.21.15.50:0/4145830060 conn(0x7f8f1c078520 msgr2=0x7f8f1c0ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4145830060 shutdown_connections 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4145830060 wait complete. 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 librados: starting msgr at 2024-04-26T16:42:32.257 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 librados: starting objecter 2024-04-26T16:42:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 Processor -- start 2024-04-26T16:42:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 -- start start 2024-04-26T16:42:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 librados: setting wanted keys 2024-04-26T16:42:32.258 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.257+0000 7f8f22d8d700 1 librados: calling monclient init 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c101800 0x7f8f1c103010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f1c1021b0 0x7f8f1c103550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1c1098e0 0x7f8f1c103a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c112ac0 con 0x7f8f1c1098e0 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c112940 con 0x7f8f1c1021b0 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1c112c40 con 0x7f8f1c101800 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f20b29700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c101800 0x7f8f1c103010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1c1098e0 0x7f8f1c103a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.259 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f20b29700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c101800 0x7f8f1c103010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50726/0 (socket says 172.21.15.50:50726) 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f20b29700 1 -- 172.21.15.50:0/4263480830 learned_addr learned my addr 172.21.15.50:0/4263480830 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 -- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c101800 msgr2=0x7f8f1c103010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8f1c101800 0x7f8f1c103010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 -- 172.21.15.50:0/4263480830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f1c1021b0 msgr2=0x7f8f1c103550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f1c1021b0 0x7f8f1c103550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 -- 172.21.15.50:0/4263480830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c1a7600 con 0x7f8f1c1098e0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f2132a700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1c1098e0 0x7f8f1c103a90 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400b6c0 tx=0x7f8f14007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 10 client.?.objecter ms_handle_connect 0x7f8f1c1098e0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14021070 con 0x7f8f1c1098e0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8f14004e20 con 0x7f8f1c1098e0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f14005040 con 0x7f8f1c1098e0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f1c1a7890 con 0x7f8f1c1098e0 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 10 client.31317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:32.260 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f1c1a7cf0 con 0x7f8f1c1098e0 2024-04-26T16:42:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 librados: init done 2024-04-26T16:42:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 10 client.31317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 15 client.31317.objecter _session_command_op_assign -1 1 2024-04-26T16:42:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 20 client.31317.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 10 client.31317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:32.261 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f1c102930 con 0x7f8f1c1098e0 2024-04-26T16:42:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8f14024070 con 0x7f8f1c1098e0 2024-04-26T16:42:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8f04071f80 0x7f8f04074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(929..929 src has 874..929) ==== 8707+0+0 (secure 0 0 0) 0x7f8f1400f020 con 0x7f8f1c1098e0 2024-04-26T16:42:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 10 client.31317.objecter ms_dispatch 0x7f8f1c001550 osd_map(929..929 src has 874..929) 2024-04-26T16:42:32.262 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 3 client.31317.objecter handle_osd_map got epochs [929,929] > 0 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 3 client.31317.objecter handle_osd_map decoding full epoch 929 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f20b29700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8f04071f80 0x7f8f04074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 10 client.31317.objecter checking command 1 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8f0407dc60 0x7f8f04080100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 20 client.31317.objecter _get_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 15 client.31317.objecter _session_command_op_remove -1 1 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 20 client.31317.objecter _get_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 20 client.31317.objecter get_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 15 client.31317.objecter _session_command_op_assign 3 1 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 4 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 10 client.31317.objecter _send_command 1 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f1401fe80 con 0x7f8f0407dc60 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 20 client.31317.objecter dump_active .. 0 homeless 2024-04-26T16:42:32.263 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=929) ==== 24+0+0 (secure 0 0 0) 0x7f8f140972a0 con 0x7f8f1c1098e0 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.261+0000 7f8f1bfff700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8f0407dc60 0x7f8f04080100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 10 client.31317.objecter ms_handle_connect 0x7f8f04071f80 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f20b29700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8f04071f80 0x7f8f04074440 secure :-1 s=READY pgs=3880 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c005d10 tx=0x7f8f0c009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f1bfff700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8f0407dc60 0x7f8f04080100 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 10 client.31317.objecter ms_handle_connect 0x7f8f0407dc60 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== osd.3 v2:172.21.15.50:6826/2541695077 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f1401fe80 con 0x7f8f0407dc60 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 10 client.31317.objecter ms_dispatch 0x7f8f1c001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 10 client.31317.objecter _finish_command 1 = system:0 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 4 2024-04-26T16:42:32.264 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.265+0000 7f8f19ffb700 15 client.31317.objecter _session_command_op_remove 3 1 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 10 client.31317.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 15 client.31317.objecter _session_command_op_assign -1 2 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter _get_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter _get_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 15 client.31317.objecter _session_command_op_remove -1 2 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter get_session s=0x7f8f0407d590 osd=3 3 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 15 client.31317.objecter _session_command_op_assign 3 2 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 4 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 10 client.31317.objecter _send_command 2 2024-04-26T16:42:32.279 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 --> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8f1c06a110 con 0x7f8f0407dc60 2024-04-26T16:42:32.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f19ffb700 1 -- 172.21.15.50:0/4263480830 <== osd.3 v2:172.21.15.50:6826/2541695077 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8f1c06a110 con 0x7f8f0407dc60 2024-04-26T16:42:32.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f19ffb700 10 client.31317.objecter ms_dispatch 0x7f8f1c001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:32.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f19ffb700 10 client.31317.objecter _finish_command 2 = system:0 2024-04-26T16:42:32.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f19ffb700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 4 2024-04-26T16:42:32.280 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f19ffb700 15 client.31317.objecter _session_command_op_remove 3 2 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 10 librados: watch_flush enter 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 10 librados: watch_flush exit 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 10 client.31317.objecter close_session for osd.3 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8f0407dc60 msgr2=0x7f8f04080100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6826/2541695077,v1:172.21.15.50:6827/2541695077] conn(0x7f8f0407dc60 0x7f8f04080100 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter put_session s=0x7f8f0407d590 osd=3 1 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 20 client.31317.objecter shutdown clearing up homeless session... 2024-04-26T16:42:32.281 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 10 client.31317.objecter successfully canceled tick 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8f04071f80 msgr2=0x7f8f04074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8f04071f80 0x7f8f04074440 secure :-1 s=READY pgs=3880 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c005d10 tx=0x7f8f0c009310 comp rx=0 tx=0).stop 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1c1098e0 msgr2=0x7f8f1c103a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 --2- 172.21.15.50:0/4263480830 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8f1c1098e0 0x7f8f1c103a90 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f8f1400b6c0 tx=0x7f8f14007ee0 comp rx=0 tx=0).stop 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f2132a700 1 -- 172.21.15.50:0/4263480830 reap_dead start 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 shutdown_connections 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 >> 172.21.15.50:0/4263480830 conn(0x7f8f1c078520 msgr2=0x7f8f1c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 shutdown_connections 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 -- 172.21.15.50:0/4263480830 wait complete. 2024-04-26T16:42:32.282 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:32.281+0000 7f8f22d8d700 1 librados: shutdown 2024-04-26T16:42:32.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.878276+0000 osd.4 (osd.4) 557 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:42:32.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.878276+0000 osd.4 (osd.4) 557 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.880239+0000 osd.4 (osd.4) 558 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:31.880239+0000 osd.4 (osd.4) 558 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:32.099257+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:32.099257+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:32.101443+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[21463]: cluster 2024-04-26T16:42:32.101443+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.878276+0000 osd.4 (osd.4) 557 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.878276+0000 osd.4 (osd.4) 557 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.880239+0000 osd.4 (osd.4) 558 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:31.880239+0000 osd.4 (osd.4) 558 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:32.099257+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:42:32.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:32.099257+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:42:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:32.101443+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:42:32.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:32 smithi050 bash[29234]: cluster 2024-04-26T16:42:32.101443+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:42:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.878276+0000 osd.4 (osd.4) 557 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:42:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.878276+0000 osd.4 (osd.4) 557 : cluster [DBG] 5.1c scrub starts 2024-04-26T16:42:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.880239+0000 osd.4 (osd.4) 558 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:42:33.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:31.880239+0000 osd.4 (osd.4) 558 : cluster [DBG] 5.1c scrub ok 2024-04-26T16:42:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:32.099257+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:42:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:32.099257+0000 osd.5 (osd.5) 510 : cluster [DBG] 2.2 scrub starts 2024-04-26T16:42:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:32.101443+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:42:33.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:32 smithi190 bash[23868]: cluster 2024-04-26T16:42:32.101443+0000 osd.5 (osd.5) 511 : cluster [DBG] 2.2 scrub ok 2024-04-26T16:42:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[21463]: cluster 2024-04-26T16:42:32.128316+0000 mgr.y (mgr.24401) 953 : cluster [DBG] pgmap v1555: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:33.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[21463]: cluster 2024-04-26T16:42:32.128316+0000 mgr.y (mgr.24401) 953 : cluster [DBG] pgmap v1555: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.102295+0000 osd.5 (osd.5) 512 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.102295+0000 osd.5 (osd.5) 512 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.104773+0000 osd.5 (osd.5) 513 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.104773+0000 osd.5 (osd.5) 513 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[29234]: cluster 2024-04-26T16:42:32.128316+0000 mgr.y (mgr.24401) 953 : cluster [DBG] pgmap v1555: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[29234]: cluster 2024-04-26T16:42:32.128316+0000 mgr.y (mgr.24401) 953 : cluster [DBG] pgmap v1555: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.102295+0000 osd.5 (osd.5) 512 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.102295+0000 osd.5 (osd.5) 512 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.104773+0000 osd.5 (osd.5) 513 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:42:33.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:33 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.104773+0000 osd.5 (osd.5) 513 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:42:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:33 smithi190 bash[23868]: cluster 2024-04-26T16:42:32.128316+0000 mgr.y (mgr.24401) 953 : cluster [DBG] pgmap v1555: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:33 smithi190 bash[23868]: cluster 2024-04-26T16:42:32.128316+0000 mgr.y (mgr.24401) 953 : cluster [DBG] pgmap v1555: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:33 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.102295+0000 osd.5 (osd.5) 512 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:42:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:33 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.102295+0000 osd.5 (osd.5) 512 : cluster [DBG] 5.f deep-scrub starts 2024-04-26T16:42:34.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:33 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.104773+0000 osd.5 (osd.5) 513 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:42:34.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:33 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.104773+0000 osd.5 (osd.5) 513 : cluster [DBG] 5.f deep-scrub ok 2024-04-26T16:42:34.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:42:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:42:34.774 INFO:teuthology.orchestra.run.smithi050.stdout:55834575221 2024-04-26T16:42:34.774 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-26T16:42:34.813 INFO:teuthology.orchestra.run.smithi050.stdout:77309411691 2024-04-26T16:42:34.814 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- 172.21.15.50:0/3676261789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103e80 msgr2=0x7f924c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- 172.21.15.50:0/3676261789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103e80 0x7f924c110330 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f924400b0d0 tx=0x7f924400b3e0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- 172.21.15.50:0/3676261789 shutdown_connections 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- 172.21.15.50:0/3676261789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c109560 0x7f924c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- 172.21.15.50:0/3676261789 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c103e80 0x7f924c110330 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- 172.21.15.50:0/3676261789 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 0x7f924c103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.847 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- 172.21.15.50:0/3676261789 >> 172.21.15.50:0/3676261789 conn(0x7f924c07a5d0 msgr2=0x7f924c07a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- 172.21.15.50:0/3676261789 shutdown_connections 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- 172.21.15.50:0/3676261789 wait complete. 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 librados: starting msgr at 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 librados: starting objecter 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 Processor -- start 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- start start 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 librados: setting wanted keys 2024-04-26T16:42:34.848 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 librados: calling monclient init 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 0x7f924c199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c103e80 0x7f924c1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c109560 0x7f924c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f924c112b40 con 0x7f924c109560 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f924c1129c0 con 0x7f924c103e80 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f924c112cc0 con 0x7f924c1034c0 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9250b1a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 0x7f924c199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c109560 0x7f924c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9250b1a700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 0x7f924c199080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:50758/0 (socket says 172.21.15.50:50758) 2024-04-26T16:42:34.850 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9250b1a700 1 -- 172.21.15.50:0/989880676 learned_addr learned my addr 172.21.15.50:0/989880676 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f924bfff700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c103e80 0x7f924c1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 -- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 msgr2=0x7f924c199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 0x7f924c199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 -- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c103e80 msgr2=0x7f924c1995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f924c103e80 0x7f924c1995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 -- 172.21.15.50:0/989880676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c19e090 con 0x7f924c109560 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9250b1a700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f924c1034c0 0x7f924c199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f925131b700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c109560 0x7f924c19d950 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f924400b0d0 tx=0x7f9244007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9249ffb700 10 client.?.objecter ms_handle_connect 0x7f924c109560 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9249ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924401f030 con 0x7f924c109560 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c19e320 con 0x7f924c109560 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.849+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f9244005050 con 0x7f924c109560 2024-04-26T16:42:34.851 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 10 client.31323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c075650 con 0x7f924c109560 2024-04-26T16:42:34.852 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 1 librados: init done 2024-04-26T16:42:34.852 INFO:teuthology.orchestra.run.smithi050.stdout:3204045602902 2024-04-26T16:42:34.853 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-26T16:42:34.853 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 10 client.31323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 15 client.31323.objecter _session_command_op_assign -1 1 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 20 client.31323.objecter _get_session osd=-1 returning homeless 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 10 client.31323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924401b690 con 0x7f924c109560 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9230002920 con 0x7f924c109560 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f9244022030 con 0x7f924c109560 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f923c071ed0 0x7f923c074390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(930..930 src has 874..930) ==== 8707+0+0 (secure 0 0 0) 0x7f924400f020 con 0x7f924c109560 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 10 client.31323.objecter ms_dispatch 0x7f924c001550 osd_map(930..930 src has 874..930) 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 3 client.31323.objecter handle_osd_map got epochs [930,930] > 0 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 3 client.31323.objecter handle_osd_map decoding full epoch 930 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9250b1a700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f923c071ed0 0x7f923c074390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:34.854 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 10 client.31323.objecter checking command 1 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f923c07dcb0 0x7f923c080150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 20 client.31323.objecter _get_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 15 client.31323.objecter _session_command_op_remove -1 1 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 20 client.31323.objecter _get_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 20 client.31323.objecter get_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 15 client.31323.objecter _session_command_op_assign 7 1 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 4 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 10 client.31323.objecter _send_command 1 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9244005050 con 0x7f923c07dcb0 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 20 client.31323.objecter dump_active .. 0 homeless 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_get_version_reply(handle=1 version=930) ==== 24+0+0 (secure 0 0 0) 0x7f9244005520 con 0x7f924c109560 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9249ffb700 10 client.31323.objecter ms_handle_connect 0x7f923c071ed0 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f9250b1a700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f923c071ed0 0x7f923c074390 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 crypto rx=0x7f9240005950 tx=0x7f924000b500 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:34.855 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.853+0000 7f924bfff700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f923c07dcb0 0x7f923c080150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f924bfff700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f923c07dcb0 0x7f923c080150 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f9249ffb700 10 client.31323.objecter ms_handle_connect 0x7f923c07dcb0 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== osd.7 v2:172.21.15.190:6824/2856617796 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9244005050 con 0x7f923c07dcb0 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f9249ffb700 10 client.31323.objecter ms_dispatch 0x7f924c001550 command_reply(tid 1: 0 ) 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f9249ffb700 10 client.31323.objecter _finish_command 1 = system:0 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f9249ffb700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 4 2024-04-26T16:42:34.856 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.857+0000 7f9249ffb700 15 client.31323.objecter _session_command_op_remove 7 1 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 10 client.31323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 15 client.31323.objecter _session_command_op_assign -1 2 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter _get_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter _get_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 15 client.31323.objecter _session_command_op_remove -1 2 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter get_session s=0x7f923c07d5e0 osd=7 3 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 15 client.31323.objecter _session_command_op_assign 7 2 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 4 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 10 client.31323.objecter _send_command 2 2024-04-26T16:42:34.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 --> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9230001220 con 0x7f923c07dcb0 2024-04-26T16:42:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9249ffb700 1 -- 172.21.15.50:0/989880676 <== osd.7 v2:172.21.15.190:6824/2856617796 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9230001220 con 0x7f923c07dcb0 2024-04-26T16:42:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9249ffb700 10 client.31323.objecter ms_dispatch 0x7f924c001550 command_reply(tid 2: 0 ) 2024-04-26T16:42:34.876 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9249ffb700 10 client.31323.objecter _finish_command 2 = system:0 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9249ffb700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 4 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9249ffb700 15 client.31323.objecter _session_command_op_remove 7 2 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 10 librados: watch_flush enter 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 10 librados: watch_flush exit 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 10 client.31323.objecter close_session for osd.7 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f923c07dcb0 msgr2=0x7f923c080150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.190:6824/2856617796,v1:172.21.15.190:6825/2856617796] conn(0x7f923c07dcb0 0x7f923c080150 crc :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter put_session s=0x7f923c07d5e0 osd=7 1 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 20 client.31323.objecter shutdown clearing up homeless session... 2024-04-26T16:42:34.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 10 client.31323.objecter successfully canceled tick 2024-04-26T16:42:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f923c071ed0 msgr2=0x7f923c074390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f923c071ed0 0x7f923c074390 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 crypto rx=0x7f9240005950 tx=0x7f924000b500 comp rx=0 tx=0).stop 2024-04-26T16:42:34.878 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c109560 msgr2=0x7f924c19d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.877+0000 7f9252d7e700 1 --2- 172.21.15.50:0/989880676 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f924c109560 0x7f924c19d950 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7f924400b0d0 tx=0x7f9244007a20 comp rx=0 tx=0).stop 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.881+0000 7f925131b700 1 -- 172.21.15.50:0/989880676 reap_dead start 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.881+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 shutdown_connections 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.881+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 >> 172.21.15.50:0/989880676 conn(0x7f924c07a5d0 msgr2=0x7f924c10a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.881+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 shutdown_connections 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.881+0000 7f9252d7e700 1 -- 172.21.15.50:0/989880676 wait complete. 2024-04-26T16:42:34.879 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:34.881+0000 7f9252d7e700 1 librados: shutdown 2024-04-26T16:42:34.897 INFO:teuthology.orchestra.run.smithi050.stdout:120259084640 2024-04-26T16:42:34.898 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-26T16:42:34.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.542494+0000 osd.0 (osd.0) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:34.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.542494+0000 osd.0 (osd.0) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:34.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.542854+0000 osd.0 (osd.0) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:34.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.542854+0000 osd.0 (osd.0) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:34.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.680894+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2024-04-26T16:42:34.928 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[29234]: cluster 2024-04-26T16:42:33.680894+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2024-04-26T16:42:34.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.542494+0000 osd.0 (osd.0) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:34.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.542494+0000 osd.0 (osd.0) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:34.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.542854+0000 osd.0 (osd.0) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:34.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.542854+0000 osd.0 (osd.0) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:34.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.680894+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2024-04-26T16:42:34.929 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:34 smithi050 bash[21463]: cluster 2024-04-26T16:42:33.680894+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2024-04-26T16:42:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:34 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.542494+0000 osd.0 (osd.0) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:34 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.542494+0000 osd.0 (osd.0) 613 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:34 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.542854+0000 osd.0 (osd.0) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:34 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.542854+0000 osd.0 (osd.0) 614 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:34 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.680894+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2024-04-26T16:42:35.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:34 smithi190 bash[23868]: cluster 2024-04-26T16:42:33.680894+0000 mon.a (mon.0) 4070 : cluster [DBG] osdmap e930: 8 total, 8 up, 8 in 2024-04-26T16:42:35.863 INFO:teuthology.orchestra.run.smithi050.stdout:296352743736 2024-04-26T16:42:35.863 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-26T16:42:36.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.018072+0000 osd.3 (osd.3) 661 : cluster [DBG] 3.b scrub starts 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.018072+0000 osd.3 (osd.3) 661 : cluster [DBG] 3.b scrub starts 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.022210+0000 osd.3 (osd.3) 662 : cluster [DBG] 3.b scrub ok 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.022210+0000 osd.3 (osd.3) 662 : cluster [DBG] 3.b scrub ok 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.060333+0000 osd.1 (osd.1) 589 : cluster [DBG] 5.c scrub starts 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.060333+0000 osd.1 (osd.1) 589 : cluster [DBG] 5.c scrub starts 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.062616+0000 osd.1 (osd.1) 590 : cluster [DBG] 5.c scrub ok 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.062616+0000 osd.1 (osd.1) 590 : cluster [DBG] 5.c scrub ok 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.129074+0000 mgr.y (mgr.24401) 954 : cluster [DBG] pgmap v1557: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.129074+0000 mgr.y (mgr.24401) 954 : cluster [DBG] pgmap v1557: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.349989+0000 osd.7 (osd.7) 646 : cluster [DBG] 3.f scrub starts 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.349989+0000 osd.7 (osd.7) 646 : cluster [DBG] 3.f scrub starts 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.354803+0000 osd.7 (osd.7) 647 : cluster [DBG] 3.f scrub ok 2024-04-26T16:42:36.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:35 smithi190 bash[23868]: cluster 2024-04-26T16:42:34.354803+0000 osd.7 (osd.7) 647 : cluster [DBG] 3.f scrub ok 2024-04-26T16:42:36.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.018072+0000 osd.3 (osd.3) 661 : cluster [DBG] 3.b scrub starts 2024-04-26T16:42:36.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.018072+0000 osd.3 (osd.3) 661 : cluster [DBG] 3.b scrub starts 2024-04-26T16:42:36.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.022210+0000 osd.3 (osd.3) 662 : cluster [DBG] 3.b scrub ok 2024-04-26T16:42:36.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.022210+0000 osd.3 (osd.3) 662 : cluster [DBG] 3.b scrub ok 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.060333+0000 osd.1 (osd.1) 589 : cluster [DBG] 5.c scrub starts 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.060333+0000 osd.1 (osd.1) 589 : cluster [DBG] 5.c scrub starts 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.062616+0000 osd.1 (osd.1) 590 : cluster [DBG] 5.c scrub ok 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.062616+0000 osd.1 (osd.1) 590 : cluster [DBG] 5.c scrub ok 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.129074+0000 mgr.y (mgr.24401) 954 : cluster [DBG] pgmap v1557: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.129074+0000 mgr.y (mgr.24401) 954 : cluster [DBG] pgmap v1557: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.349989+0000 osd.7 (osd.7) 646 : cluster [DBG] 3.f scrub starts 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.349989+0000 osd.7 (osd.7) 646 : cluster [DBG] 3.f scrub starts 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.354803+0000 osd.7 (osd.7) 647 : cluster [DBG] 3.f scrub ok 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[29234]: cluster 2024-04-26T16:42:34.354803+0000 osd.7 (osd.7) 647 : cluster [DBG] 3.f scrub ok 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.018072+0000 osd.3 (osd.3) 661 : cluster [DBG] 3.b scrub starts 2024-04-26T16:42:36.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.018072+0000 osd.3 (osd.3) 661 : cluster [DBG] 3.b scrub starts 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.022210+0000 osd.3 (osd.3) 662 : cluster [DBG] 3.b scrub ok 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.022210+0000 osd.3 (osd.3) 662 : cluster [DBG] 3.b scrub ok 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.060333+0000 osd.1 (osd.1) 589 : cluster [DBG] 5.c scrub starts 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.060333+0000 osd.1 (osd.1) 589 : cluster [DBG] 5.c scrub starts 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.062616+0000 osd.1 (osd.1) 590 : cluster [DBG] 5.c scrub ok 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.062616+0000 osd.1 (osd.1) 590 : cluster [DBG] 5.c scrub ok 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.129074+0000 mgr.y (mgr.24401) 954 : cluster [DBG] pgmap v1557: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.129074+0000 mgr.y (mgr.24401) 954 : cluster [DBG] pgmap v1557: 174 pgs: 174 active+clean; 583 KiB data, 787 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.349989+0000 osd.7 (osd.7) 646 : cluster [DBG] 3.f scrub starts 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.349989+0000 osd.7 (osd.7) 646 : cluster [DBG] 3.f scrub starts 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.354803+0000 osd.7 (osd.7) 647 : cluster [DBG] 3.f scrub ok 2024-04-26T16:42:36.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:35 smithi050 bash[21463]: cluster 2024-04-26T16:42:34.354803+0000 osd.7 (osd.7) 647 : cluster [DBG] 3.f scrub ok 2024-04-26T16:42:37.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:35.026586+0000 osd.3 (osd.3) 663 : cluster [DBG] 6.c scrub starts 2024-04-26T16:42:37.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:35.026586+0000 osd.3 (osd.3) 663 : cluster [DBG] 6.c scrub starts 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:35.028411+0000 osd.3 (osd.3) 664 : cluster [DBG] 6.c scrub ok 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:35.028411+0000 osd.3 (osd.3) 664 : cluster [DBG] 6.c scrub ok 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.115155+0000 osd.5 (osd.5) 514 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.115155+0000 osd.5 (osd.5) 514 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.117497+0000 osd.5 (osd.5) 515 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.117497+0000 osd.5 (osd.5) 515 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: audit 2024-04-26T16:42:36.131615+0000 mon.c (mon.2) 952 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:37.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:36 smithi190 bash[23868]: audit 2024-04-26T16:42:36.131615+0000 mon.c (mon.2) 952 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:37.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:35.026586+0000 osd.3 (osd.3) 663 : cluster [DBG] 6.c scrub starts 2024-04-26T16:42:37.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:35.026586+0000 osd.3 (osd.3) 663 : cluster [DBG] 6.c scrub starts 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:35.028411+0000 osd.3 (osd.3) 664 : cluster [DBG] 6.c scrub ok 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:35.028411+0000 osd.3 (osd.3) 664 : cluster [DBG] 6.c scrub ok 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.115155+0000 osd.5 (osd.5) 514 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.115155+0000 osd.5 (osd.5) 514 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.117497+0000 osd.5 (osd.5) 515 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.117497+0000 osd.5 (osd.5) 515 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: audit 2024-04-26T16:42:36.131615+0000 mon.c (mon.2) 952 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[29234]: audit 2024-04-26T16:42:36.131615+0000 mon.c (mon.2) 952 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:35.026586+0000 osd.3 (osd.3) 663 : cluster [DBG] 6.c scrub starts 2024-04-26T16:42:37.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:35.026586+0000 osd.3 (osd.3) 663 : cluster [DBG] 6.c scrub starts 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:35.028411+0000 osd.3 (osd.3) 664 : cluster [DBG] 6.c scrub ok 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:35.028411+0000 osd.3 (osd.3) 664 : cluster [DBG] 6.c scrub ok 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.115155+0000 osd.5 (osd.5) 514 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.115155+0000 osd.5 (osd.5) 514 : cluster [DBG] 2.12 scrub starts 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.117497+0000 osd.5 (osd.5) 515 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.117497+0000 osd.5 (osd.5) 515 : cluster [DBG] 2.12 scrub ok 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: audit 2024-04-26T16:42:36.131615+0000 mon.c (mon.2) 952 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:37.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:36 smithi050 bash[21463]: audit 2024-04-26T16:42:36.131615+0000 mon.c (mon.2) 952 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:42:38.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.130080+0000 mgr.y (mgr.24401) 955 : cluster [DBG] pgmap v1558: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.130080+0000 mgr.y (mgr.24401) 955 : cluster [DBG] pgmap v1558: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.239426+0000 osd.0 (osd.0) 615 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.239426+0000 osd.0 (osd.0) 615 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.241325+0000 osd.0 (osd.0) 616 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: cluster 2024-04-26T16:42:36.241325+0000 osd.0 (osd.0) 616 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: audit 2024-04-26T16:42:36.911007+0000 mon.c (mon.2) 953 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: audit 2024-04-26T16:42:36.911007+0000 mon.c (mon.2) 953 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: audit 2024-04-26T16:42:36.912939+0000 mon.c (mon.2) 954 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: audit 2024-04-26T16:42:36.912939+0000 mon.c (mon.2) 954 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:38.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: audit 2024-04-26T16:42:36.924059+0000 mon.a (mon.0) 4071 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:42:38.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:37 smithi190 bash[23868]: audit 2024-04-26T16:42:36.924059+0000 mon.a (mon.0) 4071 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:42:38.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.130080+0000 mgr.y (mgr.24401) 955 : cluster [DBG] pgmap v1558: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.130080+0000 mgr.y (mgr.24401) 955 : cluster [DBG] pgmap v1558: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.239426+0000 osd.0 (osd.0) 615 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.239426+0000 osd.0 (osd.0) 615 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.241325+0000 osd.0 (osd.0) 616 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: cluster 2024-04-26T16:42:36.241325+0000 osd.0 (osd.0) 616 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: audit 2024-04-26T16:42:36.911007+0000 mon.c (mon.2) 953 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: audit 2024-04-26T16:42:36.911007+0000 mon.c (mon.2) 953 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: audit 2024-04-26T16:42:36.912939+0000 mon.c (mon.2) 954 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: audit 2024-04-26T16:42:36.912939+0000 mon.c (mon.2) 954 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: audit 2024-04-26T16:42:36.924059+0000 mon.a (mon.0) 4071 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:42:38.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[21463]: audit 2024-04-26T16:42:36.924059+0000 mon.a (mon.0) 4071 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.130080+0000 mgr.y (mgr.24401) 955 : cluster [DBG] pgmap v1558: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.130080+0000 mgr.y (mgr.24401) 955 : cluster [DBG] pgmap v1558: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.239426+0000 osd.0 (osd.0) 615 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.239426+0000 osd.0 (osd.0) 615 : cluster [DBG] 6.9 scrub starts 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.241325+0000 osd.0 (osd.0) 616 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: cluster 2024-04-26T16:42:36.241325+0000 osd.0 (osd.0) 616 : cluster [DBG] 6.9 scrub ok 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: audit 2024-04-26T16:42:36.911007+0000 mon.c (mon.2) 953 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: audit 2024-04-26T16:42:36.911007+0000 mon.c (mon.2) 953 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: audit 2024-04-26T16:42:36.912939+0000 mon.c (mon.2) 954 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: audit 2024-04-26T16:42:36.912939+0000 mon.c (mon.2) 954 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: audit 2024-04-26T16:42:36.924059+0000 mon.a (mon.0) 4071 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:42:38.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:37 smithi050 bash[29234]: audit 2024-04-26T16:42:36.924059+0000 mon.a (mon.0) 4071 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:42:38.357 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:38.358 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:38.358 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:39.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.128992+0000 osd.1 (osd.1) 591 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.128992+0000 osd.1 (osd.1) 591 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.131458+0000 osd.1 (osd.1) 592 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.131458+0000 osd.1 (osd.1) 592 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.300413+0000 osd.7 (osd.7) 648 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.300413+0000 osd.7 (osd.7) 648 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.300946+0000 osd.7 (osd.7) 649 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.300946+0000 osd.7 (osd.7) 649 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: audit 2024-04-26T16:42:37.622216+0000 mgr.y (mgr.24401) 956 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: audit 2024-04-26T16:42:37.622216+0000 mgr.y (mgr.24401) 956 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.812133+0000 osd.4 (osd.4) 559 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.812133+0000 osd.4 (osd.4) 559 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:42:39.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.816373+0000 osd.4 (osd.4) 560 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:42:39.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.816373+0000 osd.4 (osd.4) 560 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:42:39.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.929164+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2024-04-26T16:42:39.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:38 smithi190 bash[23868]: cluster 2024-04-26T16:42:37.929164+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.128992+0000 osd.1 (osd.1) 591 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.128992+0000 osd.1 (osd.1) 591 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.131458+0000 osd.1 (osd.1) 592 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.131458+0000 osd.1 (osd.1) 592 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.300413+0000 osd.7 (osd.7) 648 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.300413+0000 osd.7 (osd.7) 648 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.300946+0000 osd.7 (osd.7) 649 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.300946+0000 osd.7 (osd.7) 649 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:39.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: audit 2024-04-26T16:42:37.622216+0000 mgr.y (mgr.24401) 956 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: audit 2024-04-26T16:42:37.622216+0000 mgr.y (mgr.24401) 956 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.812133+0000 osd.4 (osd.4) 559 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.812133+0000 osd.4 (osd.4) 559 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.816373+0000 osd.4 (osd.4) 560 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.816373+0000 osd.4 (osd.4) 560 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.929164+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[21463]: cluster 2024-04-26T16:42:37.929164+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.128992+0000 osd.1 (osd.1) 591 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.128992+0000 osd.1 (osd.1) 591 : cluster [DBG] 5.12 scrub starts 2024-04-26T16:42:39.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.131458+0000 osd.1 (osd.1) 592 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.131458+0000 osd.1 (osd.1) 592 : cluster [DBG] 5.12 scrub ok 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.300413+0000 osd.7 (osd.7) 648 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.300413+0000 osd.7 (osd.7) 648 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.300946+0000 osd.7 (osd.7) 649 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.300946+0000 osd.7 (osd.7) 649 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: audit 2024-04-26T16:42:37.622216+0000 mgr.y (mgr.24401) 956 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: audit 2024-04-26T16:42:37.622216+0000 mgr.y (mgr.24401) 956 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.812133+0000 osd.4 (osd.4) 559 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.812133+0000 osd.4 (osd.4) 559 : cluster [DBG] 3.1a scrub starts 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.816373+0000 osd.4 (osd.4) 560 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.816373+0000 osd.4 (osd.4) 560 : cluster [DBG] 3.1a scrub ok 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.929164+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2024-04-26T16:42:39.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:38 smithi050 bash[29234]: cluster 2024-04-26T16:42:37.929164+0000 mon.a (mon.0) 4072 : cluster [DBG] osdmap e931: 8 total, 8 up, 8 in 2024-04-26T16:42:39.376 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:39.377 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:39.377 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:39.377 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[21463]: cluster 2024-04-26T16:42:38.130858+0000 mgr.y (mgr.24401) 957 : cluster [DBG] pgmap v1560: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[21463]: cluster 2024-04-26T16:42:38.130858+0000 mgr.y (mgr.24401) 957 : cluster [DBG] pgmap v1560: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.067585+0000 osd.5 (osd.5) 516 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:42:39.693 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.067585+0000 osd.5 (osd.5) 516 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:42:39.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.069879+0000 osd.5 (osd.5) 517 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:42:39.694 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.069879+0000 osd.5 (osd.5) 517 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:42:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:39 smithi190 bash[23868]: cluster 2024-04-26T16:42:38.130858+0000 mgr.y (mgr.24401) 957 : cluster [DBG] pgmap v1560: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:39 smithi190 bash[23868]: cluster 2024-04-26T16:42:38.130858+0000 mgr.y (mgr.24401) 957 : cluster [DBG] pgmap v1560: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:39 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.067585+0000 osd.5 (osd.5) 516 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:42:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:39 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.067585+0000 osd.5 (osd.5) 516 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:42:40.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:39 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.069879+0000 osd.5 (osd.5) 517 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:42:40.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:39 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.069879+0000 osd.5 (osd.5) 517 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:42:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[29234]: cluster 2024-04-26T16:42:38.130858+0000 mgr.y (mgr.24401) 957 : cluster [DBG] pgmap v1560: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[29234]: cluster 2024-04-26T16:42:38.130858+0000 mgr.y (mgr.24401) 957 : cluster [DBG] pgmap v1560: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.067585+0000 osd.5 (osd.5) 516 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:42:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.067585+0000 osd.5 (osd.5) 516 : cluster [DBG] 2.3 scrub starts 2024-04-26T16:42:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.069879+0000 osd.5 (osd.5) 517 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:42:40.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:39 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.069879+0000 osd.5 (osd.5) 517 : cluster [DBG] 2.3 scrub ok 2024-04-26T16:42:40.395 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:42:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.129166+0000 osd.1 (osd.1) 593 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:42:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.129166+0000 osd.1 (osd.1) 593 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:42:41.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.131662+0000 osd.1 (osd.1) 594 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:42:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:39.131662+0000 osd.1 (osd.1) 594 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:42:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.095020+0000 osd.5 (osd.5) 518 : cluster [DBG] 3.16 deep-scrub starts 2024-04-26T16:42:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.095020+0000 osd.5 (osd.5) 518 : cluster [DBG] 3.16 deep-scrub starts 2024-04-26T16:42:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.103782+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 deep-scrub ok 2024-04-26T16:42:41.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:40 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.103782+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 deep-scrub ok 2024-04-26T16:42:41.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.129166+0000 osd.1 (osd.1) 593 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.129166+0000 osd.1 (osd.1) 593 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.131662+0000 osd.1 (osd.1) 594 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:39.131662+0000 osd.1 (osd.1) 594 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.095020+0000 osd.5 (osd.5) 518 : cluster [DBG] 3.16 deep-scrub starts 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.095020+0000 osd.5 (osd.5) 518 : cluster [DBG] 3.16 deep-scrub starts 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.103782+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 deep-scrub ok 2024-04-26T16:42:41.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.103782+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 deep-scrub ok 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.129166+0000 osd.1 (osd.1) 593 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.129166+0000 osd.1 (osd.1) 593 : cluster [DBG] 6.1 scrub starts 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.131662+0000 osd.1 (osd.1) 594 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:39.131662+0000 osd.1 (osd.1) 594 : cluster [DBG] 6.1 scrub ok 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.095020+0000 osd.5 (osd.5) 518 : cluster [DBG] 3.16 deep-scrub starts 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.095020+0000 osd.5 (osd.5) 518 : cluster [DBG] 3.16 deep-scrub starts 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.103782+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 deep-scrub ok 2024-04-26T16:42:41.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:40 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.103782+0000 osd.5 (osd.5) 519 : cluster [DBG] 3.16 deep-scrub ok 2024-04-26T16:42:42.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.131796+0000 mgr.y (mgr.24401) 958 : cluster [DBG] pgmap v1561: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.131796+0000 mgr.y (mgr.24401) 958 : cluster [DBG] pgmap v1561: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.425240+0000 osd.7 (osd.7) 650 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.425240+0000 osd.7 (osd.7) 650 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.428207+0000 osd.7 (osd.7) 651 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.428207+0000 osd.7 (osd.7) 651 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.866219+0000 osd.4 (osd.4) 561 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.866219+0000 osd.4 (osd.4) 561 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.870164+0000 osd.4 (osd.4) 562 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.870164+0000 osd.4 (osd.4) 562 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.948058+0000 osd.3 (osd.3) 665 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.948058+0000 osd.3 (osd.3) 665 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.950847+0000 osd.3 (osd.3) 666 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:42:42.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:41 smithi190 bash[23868]: cluster 2024-04-26T16:42:40.950847+0000 osd.3 (osd.3) 666 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:42:42.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.131796+0000 mgr.y (mgr.24401) 958 : cluster [DBG] pgmap v1561: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.131796+0000 mgr.y (mgr.24401) 958 : cluster [DBG] pgmap v1561: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.425240+0000 osd.7 (osd.7) 650 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.425240+0000 osd.7 (osd.7) 650 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.428207+0000 osd.7 (osd.7) 651 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.428207+0000 osd.7 (osd.7) 651 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.866219+0000 osd.4 (osd.4) 561 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.866219+0000 osd.4 (osd.4) 561 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.870164+0000 osd.4 (osd.4) 562 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.870164+0000 osd.4 (osd.4) 562 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.948058+0000 osd.3 (osd.3) 665 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.948058+0000 osd.3 (osd.3) 665 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.950847+0000 osd.3 (osd.3) 666 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:42:42.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[29234]: cluster 2024-04-26T16:42:40.950847+0000 osd.3 (osd.3) 666 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.131796+0000 mgr.y (mgr.24401) 958 : cluster [DBG] pgmap v1561: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.131796+0000 mgr.y (mgr.24401) 958 : cluster [DBG] pgmap v1561: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.425240+0000 osd.7 (osd.7) 650 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.425240+0000 osd.7 (osd.7) 650 : cluster [DBG] 2.5 scrub starts 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.428207+0000 osd.7 (osd.7) 651 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.428207+0000 osd.7 (osd.7) 651 : cluster [DBG] 2.5 scrub ok 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.866219+0000 osd.4 (osd.4) 561 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.866219+0000 osd.4 (osd.4) 561 : cluster [DBG] 3.9 scrub starts 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.870164+0000 osd.4 (osd.4) 562 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.870164+0000 osd.4 (osd.4) 562 : cluster [DBG] 3.9 scrub ok 2024-04-26T16:42:42.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.948058+0000 osd.3 (osd.3) 665 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:42:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.948058+0000 osd.3 (osd.3) 665 : cluster [DBG] 11.8 scrub starts 2024-04-26T16:42:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.950847+0000 osd.3 (osd.3) 666 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:42:42.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:41 smithi050 bash[21463]: cluster 2024-04-26T16:42:40.950847+0000 osd.3 (osd.3) 666 : cluster [DBG] 11.8 scrub ok 2024-04-26T16:42:42.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.066597+0000 osd.1 (osd.1) 595 : cluster [DBG] 2.d scrub starts 2024-04-26T16:42:42.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.066597+0000 osd.1 (osd.1) 595 : cluster [DBG] 2.d scrub starts 2024-04-26T16:42:42.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.069186+0000 osd.1 (osd.1) 596 : cluster [DBG] 2.d scrub ok 2024-04-26T16:42:42.840 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.069186+0000 osd.1 (osd.1) 596 : cluster [DBG] 2.d scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.461581+0000 osd.7 (osd.7) 652 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.461581+0000 osd.7 (osd.7) 652 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.463829+0000 osd.7 (osd.7) 653 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.463829+0000 osd.7 (osd.7) 653 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.827936+0000 osd.4 (osd.4) 563 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.827936+0000 osd.4 (osd.4) 563 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.830528+0000 osd.4 (osd.4) 564 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.830528+0000 osd.4 (osd.4) 564 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.909061+0000 osd.3 (osd.3) 667 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.909061+0000 osd.3 (osd.3) 667 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.911963+0000 osd.3 (osd.3) 668 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[29234]: cluster 2024-04-26T16:42:41.911963+0000 osd.3 (osd.3) 668 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:42:42.841 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.066597+0000 osd.1 (osd.1) 595 : cluster [DBG] 2.d scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.066597+0000 osd.1 (osd.1) 595 : cluster [DBG] 2.d scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.069186+0000 osd.1 (osd.1) 596 : cluster [DBG] 2.d scrub ok 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.069186+0000 osd.1 (osd.1) 596 : cluster [DBG] 2.d scrub ok 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.461581+0000 osd.7 (osd.7) 652 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.461581+0000 osd.7 (osd.7) 652 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.463829+0000 osd.7 (osd.7) 653 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.463829+0000 osd.7 (osd.7) 653 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.827936+0000 osd.4 (osd.4) 563 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.827936+0000 osd.4 (osd.4) 563 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.830528+0000 osd.4 (osd.4) 564 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.830528+0000 osd.4 (osd.4) 564 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.909061+0000 osd.3 (osd.3) 667 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:42:42.842 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.909061+0000 osd.3 (osd.3) 667 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:42:42.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.911963+0000 osd.3 (osd.3) 668 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:42:42.843 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:42 smithi050 bash[21463]: cluster 2024-04-26T16:42:41.911963+0000 osd.3 (osd.3) 668 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:42:43.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.066597+0000 osd.1 (osd.1) 595 : cluster [DBG] 2.d scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.066597+0000 osd.1 (osd.1) 595 : cluster [DBG] 2.d scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.069186+0000 osd.1 (osd.1) 596 : cluster [DBG] 2.d scrub ok 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.069186+0000 osd.1 (osd.1) 596 : cluster [DBG] 2.d scrub ok 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.461581+0000 osd.7 (osd.7) 652 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.461581+0000 osd.7 (osd.7) 652 : cluster [DBG] 6.12 scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.463829+0000 osd.7 (osd.7) 653 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.463829+0000 osd.7 (osd.7) 653 : cluster [DBG] 6.12 scrub ok 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.827936+0000 osd.4 (osd.4) 563 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.827936+0000 osd.4 (osd.4) 563 : cluster [DBG] 1.5 scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.830528+0000 osd.4 (osd.4) 564 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.830528+0000 osd.4 (osd.4) 564 : cluster [DBG] 1.5 scrub ok 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.909061+0000 osd.3 (osd.3) 667 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:42:43.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.909061+0000 osd.3 (osd.3) 667 : cluster [DBG] 2.1b scrub starts 2024-04-26T16:42:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.911963+0000 osd.3 (osd.3) 668 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:42:43.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:42 smithi190 bash[23868]: cluster 2024-04-26T16:42:41.911963+0000 osd.3 (osd.3) 668 : cluster [DBG] 2.1b scrub ok 2024-04-26T16:42:43.790 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.034632+0000 osd.1 (osd.1) 597 : cluster [DBG] 2.a scrub starts 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.034632+0000 osd.1 (osd.1) 597 : cluster [DBG] 2.a scrub starts 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.037035+0000 osd.1 (osd.1) 598 : cluster [DBG] 2.a scrub ok 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.037035+0000 osd.1 (osd.1) 598 : cluster [DBG] 2.a scrub ok 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.133601+0000 mgr.y (mgr.24401) 959 : cluster [DBG] pgmap v1562: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.133601+0000 mgr.y (mgr.24401) 959 : cluster [DBG] pgmap v1562: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.230890+0000 osd.0 (osd.0) 617 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.230890+0000 osd.0 (osd.0) 617 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.233887+0000 osd.0 (osd.0) 618 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:42.233887+0000 osd.0 (osd.0) 618 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.068625+0000 osd.2 (osd.2) 543 : cluster [DBG] 5.b scrub starts 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.068625+0000 osd.2 (osd.2) 543 : cluster [DBG] 5.b scrub starts 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.070828+0000 osd.2 (osd.2) 544 : cluster [DBG] 5.b scrub ok 2024-04-26T16:42:43.791 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.070828+0000 osd.2 (osd.2) 544 : cluster [DBG] 5.b scrub ok 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.090111+0000 osd.5 (osd.5) 520 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.090111+0000 osd.5 (osd.5) 520 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.092355+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.092355+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: audit 2024-04-26T16:42:43.650132+0000 mon.c (mon.2) 955 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[21463]: audit 2024-04-26T16:42:43.650132+0000 mon.c (mon.2) 955 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.034632+0000 osd.1 (osd.1) 597 : cluster [DBG] 2.a scrub starts 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.034632+0000 osd.1 (osd.1) 597 : cluster [DBG] 2.a scrub starts 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.037035+0000 osd.1 (osd.1) 598 : cluster [DBG] 2.a scrub ok 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.037035+0000 osd.1 (osd.1) 598 : cluster [DBG] 2.a scrub ok 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.133601+0000 mgr.y (mgr.24401) 959 : cluster [DBG] pgmap v1562: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:43.792 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.133601+0000 mgr.y (mgr.24401) 959 : cluster [DBG] pgmap v1562: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.230890+0000 osd.0 (osd.0) 617 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.230890+0000 osd.0 (osd.0) 617 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.233887+0000 osd.0 (osd.0) 618 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:42.233887+0000 osd.0 (osd.0) 618 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.068625+0000 osd.2 (osd.2) 543 : cluster [DBG] 5.b scrub starts 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.068625+0000 osd.2 (osd.2) 543 : cluster [DBG] 5.b scrub starts 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.070828+0000 osd.2 (osd.2) 544 : cluster [DBG] 5.b scrub ok 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.070828+0000 osd.2 (osd.2) 544 : cluster [DBG] 5.b scrub ok 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.090111+0000 osd.5 (osd.5) 520 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.090111+0000 osd.5 (osd.5) 520 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.092355+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.092355+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: audit 2024-04-26T16:42:43.650132+0000 mon.c (mon.2) 955 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:43.793 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:43 smithi050 bash[29234]: audit 2024-04-26T16:42:43.650132+0000 mon.c (mon.2) 955 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.034632+0000 osd.1 (osd.1) 597 : cluster [DBG] 2.a scrub starts 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.034632+0000 osd.1 (osd.1) 597 : cluster [DBG] 2.a scrub starts 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.037035+0000 osd.1 (osd.1) 598 : cluster [DBG] 2.a scrub ok 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.037035+0000 osd.1 (osd.1) 598 : cluster [DBG] 2.a scrub ok 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.133601+0000 mgr.y (mgr.24401) 959 : cluster [DBG] pgmap v1562: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.133601+0000 mgr.y (mgr.24401) 959 : cluster [DBG] pgmap v1562: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.230890+0000 osd.0 (osd.0) 617 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:42:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.230890+0000 osd.0 (osd.0) 617 : cluster [DBG] 11.19 scrub starts 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.233887+0000 osd.0 (osd.0) 618 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:42.233887+0000 osd.0 (osd.0) 618 : cluster [DBG] 11.19 scrub ok 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.068625+0000 osd.2 (osd.2) 543 : cluster [DBG] 5.b scrub starts 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.068625+0000 osd.2 (osd.2) 543 : cluster [DBG] 5.b scrub starts 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.070828+0000 osd.2 (osd.2) 544 : cluster [DBG] 5.b scrub ok 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.070828+0000 osd.2 (osd.2) 544 : cluster [DBG] 5.b scrub ok 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.090111+0000 osd.5 (osd.5) 520 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.090111+0000 osd.5 (osd.5) 520 : cluster [DBG] 6.7 scrub starts 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.092355+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.092355+0000 osd.5 (osd.5) 521 : cluster [DBG] 6.7 scrub ok 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: audit 2024-04-26T16:42:43.650132+0000 mon.c (mon.2) 955 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:44.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:43 smithi190 bash[23868]: audit 2024-04-26T16:42:43.650132+0000 mon.c (mon.2) 955 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:44.723 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:42:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:42:44.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.026570+0000 osd.1 (osd.1) 599 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:42:44.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.026570+0000 osd.1 (osd.1) 599 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:42:44.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.031882+0000 osd.1 (osd.1) 600 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:42:44.989 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[29234]: cluster 2024-04-26T16:42:43.031882+0000 osd.1 (osd.1) 600 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:42:44.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.026570+0000 osd.1 (osd.1) 599 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:42:44.989 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.026570+0000 osd.1 (osd.1) 599 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:42:44.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.031882+0000 osd.1 (osd.1) 600 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:42:44.990 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:44 smithi050 bash[21463]: cluster 2024-04-26T16:42:43.031882+0000 osd.1 (osd.1) 600 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:42:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:44 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.026570+0000 osd.1 (osd.1) 599 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:42:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:44 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.026570+0000 osd.1 (osd.1) 599 : cluster [DBG] 3.4 scrub starts 2024-04-26T16:42:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:44 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.031882+0000 osd.1 (osd.1) 600 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:42:45.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:44 smithi190 bash[23868]: cluster 2024-04-26T16:42:43.031882+0000 osd.1 (osd.1) 600 : cluster [DBG] 3.4 scrub ok 2024-04-26T16:42:45.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[21463]: cluster 2024-04-26T16:42:44.134302+0000 mgr.y (mgr.24401) 960 : cluster [DBG] pgmap v1563: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:45.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[21463]: cluster 2024-04-26T16:42:44.134302+0000 mgr.y (mgr.24401) 960 : cluster [DBG] pgmap v1563: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:45.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[21463]: cluster 2024-04-26T16:42:45.138641+0000 osd.5 (osd.5) 522 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[21463]: cluster 2024-04-26T16:42:45.138641+0000 osd.5 (osd.5) 522 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[21463]: cluster 2024-04-26T16:42:45.143347+0000 osd.5 (osd.5) 523 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[21463]: cluster 2024-04-26T16:42:45.143347+0000 osd.5 (osd.5) 523 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[29234]: cluster 2024-04-26T16:42:44.134302+0000 mgr.y (mgr.24401) 960 : cluster [DBG] pgmap v1563: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[29234]: cluster 2024-04-26T16:42:44.134302+0000 mgr.y (mgr.24401) 960 : cluster [DBG] pgmap v1563: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[29234]: cluster 2024-04-26T16:42:45.138641+0000 osd.5 (osd.5) 522 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[29234]: cluster 2024-04-26T16:42:45.138641+0000 osd.5 (osd.5) 522 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[29234]: cluster 2024-04-26T16:42:45.143347+0000 osd.5 (osd.5) 523 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:42:45.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:45 smithi050 bash[29234]: cluster 2024-04-26T16:42:45.143347+0000 osd.5 (osd.5) 523 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:42:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:45 smithi190 bash[23868]: cluster 2024-04-26T16:42:44.134302+0000 mgr.y (mgr.24401) 960 : cluster [DBG] pgmap v1563: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:46.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:45 smithi190 bash[23868]: cluster 2024-04-26T16:42:44.134302+0000 mgr.y (mgr.24401) 960 : cluster [DBG] pgmap v1563: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:45 smithi190 bash[23868]: cluster 2024-04-26T16:42:45.138641+0000 osd.5 (osd.5) 522 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:42:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:45 smithi190 bash[23868]: cluster 2024-04-26T16:42:45.138641+0000 osd.5 (osd.5) 522 : cluster [DBG] 3.1c scrub starts 2024-04-26T16:42:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:45 smithi190 bash[23868]: cluster 2024-04-26T16:42:45.143347+0000 osd.5 (osd.5) 523 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:42:46.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:45 smithi190 bash[23868]: cluster 2024-04-26T16:42:45.143347+0000 osd.5 (osd.5) 523 : cluster [DBG] 3.1c scrub ok 2024-04-26T16:42:47.733 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[29234]: cluster 2024-04-26T16:42:46.135221+0000 mgr.y (mgr.24401) 961 : cluster [DBG] pgmap v1564: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:47.734 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[29234]: cluster 2024-04-26T16:42:46.135221+0000 mgr.y (mgr.24401) 961 : cluster [DBG] pgmap v1564: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:48.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:42:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[23868]: cluster 2024-04-26T16:42:46.135221+0000 mgr.y (mgr.24401) 961 : cluster [DBG] pgmap v1564: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:48.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[23868]: cluster 2024-04-26T16:42:46.135221+0000 mgr.y (mgr.24401) 961 : cluster [DBG] pgmap v1564: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[23868]: cluster 2024-04-26T16:42:46.877698+0000 osd.3 (osd.3) 669 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:42:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[23868]: cluster 2024-04-26T16:42:46.877698+0000 osd.3 (osd.3) 669 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:42:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[23868]: cluster 2024-04-26T16:42:46.880046+0000 osd.3 (osd.3) 670 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:42:48.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:47 smithi190 bash[23868]: cluster 2024-04-26T16:42:46.880046+0000 osd.3 (osd.3) 670 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:42:48.075 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[29234]: cluster 2024-04-26T16:42:46.877698+0000 osd.3 (osd.3) 669 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[29234]: cluster 2024-04-26T16:42:46.877698+0000 osd.3 (osd.3) 669 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[29234]: cluster 2024-04-26T16:42:46.880046+0000 osd.3 (osd.3) 670 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[29234]: cluster 2024-04-26T16:42:46.880046+0000 osd.3 (osd.3) 670 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[21463]: cluster 2024-04-26T16:42:46.135221+0000 mgr.y (mgr.24401) 961 : cluster [DBG] pgmap v1564: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[21463]: cluster 2024-04-26T16:42:46.135221+0000 mgr.y (mgr.24401) 961 : cluster [DBG] pgmap v1564: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[21463]: cluster 2024-04-26T16:42:46.877698+0000 osd.3 (osd.3) 669 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[21463]: cluster 2024-04-26T16:42:46.877698+0000 osd.3 (osd.3) 669 : cluster [DBG] 6.1b scrub starts 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[21463]: cluster 2024-04-26T16:42:46.880046+0000 osd.3 (osd.3) 670 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:42:48.076 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:47 smithi050 bash[21463]: cluster 2024-04-26T16:42:46.880046+0000 osd.3 (osd.3) 670 : cluster [DBG] 6.1b scrub ok 2024-04-26T16:42:49.036 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:48 smithi050 bash[21463]: cluster 2024-04-26T16:42:47.187454+0000 osd.0 (osd.0) 619 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:42:49.037 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:48 smithi050 bash[21463]: cluster 2024-04-26T16:42:47.187454+0000 osd.0 (osd.0) 619 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:42:49.037 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:47.189952+0000 osd.0 (osd.0) 620 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:42:49.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:47.187454+0000 osd.0 (osd.0) 619 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:42:49.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:47.187454+0000 osd.0 (osd.0) 619 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:47.189952+0000 osd.0 (osd.0) 620 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:47.189952+0000 osd.0 (osd.0) 620 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: audit 2024-04-26T16:42:47.631013+0000 mgr.y (mgr.24401) 962 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: audit 2024-04-26T16:42:47.631013+0000 mgr.y (mgr.24401) 962 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.203808+0000 osd.2 (osd.2) 545 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.203808+0000 osd.2 (osd.2) 545 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.206964+0000 osd.2 (osd.2) 546 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:42:49.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:48 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.206964+0000 osd.2 (osd.2) 546 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:42:49.426 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:47.189952+0000 osd.0 (osd.0) 620 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:42:49.426 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: audit 2024-04-26T16:42:47.631013+0000 mgr.y (mgr.24401) 962 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:49.426 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: audit 2024-04-26T16:42:47.631013+0000 mgr.y (mgr.24401) 962 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.203808+0000 osd.2 (osd.2) 545 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.203808+0000 osd.2 (osd.2) 545 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.206964+0000 osd.2 (osd.2) 546 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.206964+0000 osd.2 (osd.2) 546 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:47.187454+0000 osd.0 (osd.0) 619 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:47.187454+0000 osd.0 (osd.0) 619 : cluster [DBG] 6.16 scrub starts 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:47.189952+0000 osd.0 (osd.0) 620 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:47.189952+0000 osd.0 (osd.0) 620 : cluster [DBG] 6.16 scrub ok 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: audit 2024-04-26T16:42:47.631013+0000 mgr.y (mgr.24401) 962 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: audit 2024-04-26T16:42:47.631013+0000 mgr.y (mgr.24401) 962 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.203808+0000 osd.2 (osd.2) 545 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:42:49.427 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.203808+0000 osd.2 (osd.2) 545 : cluster [DBG] 11.11 scrub starts 2024-04-26T16:42:49.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.206964+0000 osd.2 (osd.2) 546 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:42:49.428 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.206964+0000 osd.2 (osd.2) 546 : cluster [DBG] 11.11 scrub ok 2024-04-26T16:42:50.064 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.091487+0000 osd.1 (osd.1) 601 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.091487+0000 osd.1 (osd.1) 601 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.094115+0000 osd.1 (osd.1) 602 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.094115+0000 osd.1 (osd.1) 602 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.136697+0000 mgr.y (mgr.24401) 963 : cluster [DBG] pgmap v1565: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.136697+0000 mgr.y (mgr.24401) 963 : cluster [DBG] pgmap v1565: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.147548+0000 osd.0 (osd.0) 621 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.147548+0000 osd.0 (osd.0) 621 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.151962+0000 osd.0 (osd.0) 622 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:48.151962+0000 osd.0 (osd.0) 622 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:49.172369+0000 osd.2 (osd.2) 547 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:49.172369+0000 osd.2 (osd.2) 547 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:49.173782+0000 osd.2 (osd.2) 548 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:42:50.065 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[21463]: cluster 2024-04-26T16:42:49.173782+0000 osd.2 (osd.2) 548 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.091487+0000 osd.1 (osd.1) 601 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.091487+0000 osd.1 (osd.1) 601 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.094115+0000 osd.1 (osd.1) 602 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.094115+0000 osd.1 (osd.1) 602 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.136697+0000 mgr.y (mgr.24401) 963 : cluster [DBG] pgmap v1565: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.136697+0000 mgr.y (mgr.24401) 963 : cluster [DBG] pgmap v1565: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.147548+0000 osd.0 (osd.0) 621 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.147548+0000 osd.0 (osd.0) 621 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.151962+0000 osd.0 (osd.0) 622 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:42:50.066 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:48.151962+0000 osd.0 (osd.0) 622 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:42:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:49.172369+0000 osd.2 (osd.2) 547 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:42:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:49.172369+0000 osd.2 (osd.2) 547 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:42:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:49.173782+0000 osd.2 (osd.2) 548 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:42:50.067 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:49 smithi050 bash[29234]: cluster 2024-04-26T16:42:49.173782+0000 osd.2 (osd.2) 548 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:42:50.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.091487+0000 osd.1 (osd.1) 601 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:42:50.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.091487+0000 osd.1 (osd.1) 601 : cluster [DBG] 5.19 scrub starts 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.094115+0000 osd.1 (osd.1) 602 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.094115+0000 osd.1 (osd.1) 602 : cluster [DBG] 5.19 scrub ok 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.136697+0000 mgr.y (mgr.24401) 963 : cluster [DBG] pgmap v1565: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.136697+0000 mgr.y (mgr.24401) 963 : cluster [DBG] pgmap v1565: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.147548+0000 osd.0 (osd.0) 621 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.147548+0000 osd.0 (osd.0) 621 : cluster [DBG] 3.1f scrub starts 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.151962+0000 osd.0 (osd.0) 622 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:48.151962+0000 osd.0 (osd.0) 622 : cluster [DBG] 3.1f scrub ok 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:49.172369+0000 osd.2 (osd.2) 547 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:49.172369+0000 osd.2 (osd.2) 547 : cluster [DBG] 11.2 scrub starts 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:49.173782+0000 osd.2 (osd.2) 548 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:42:50.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:49 smithi190 bash[23868]: cluster 2024-04-26T16:42:49.173782+0000 osd.2 (osd.2) 548 : cluster [DBG] 11.2 scrub ok 2024-04-26T16:42:52.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.137707+0000 mgr.y (mgr.24401) 964 : cluster [DBG] pgmap v1566: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.137707+0000 mgr.y (mgr.24401) 964 : cluster [DBG] pgmap v1566: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.189781+0000 osd.0 (osd.0) 623 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.189781+0000 osd.0 (osd.0) 623 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.192934+0000 osd.0 (osd.0) 624 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.192934+0000 osd.0 (osd.0) 624 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.231042+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.231042+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.233559+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:42:52.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:51 smithi190 bash[23868]: cluster 2024-04-26T16:42:50.233559+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:42:52.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.137707+0000 mgr.y (mgr.24401) 964 : cluster [DBG] pgmap v1566: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:52.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.137707+0000 mgr.y (mgr.24401) 964 : cluster [DBG] pgmap v1566: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.189781+0000 osd.0 (osd.0) 623 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.189781+0000 osd.0 (osd.0) 623 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.192934+0000 osd.0 (osd.0) 624 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.192934+0000 osd.0 (osd.0) 624 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.231042+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.231042+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.233559+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[21463]: cluster 2024-04-26T16:42:50.233559+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.137707+0000 mgr.y (mgr.24401) 964 : cluster [DBG] pgmap v1566: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:52.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.137707+0000 mgr.y (mgr.24401) 964 : cluster [DBG] pgmap v1566: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.189781+0000 osd.0 (osd.0) 623 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.189781+0000 osd.0 (osd.0) 623 : cluster [DBG] 3.1 scrub starts 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.192934+0000 osd.0 (osd.0) 624 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.192934+0000 osd.0 (osd.0) 624 : cluster [DBG] 3.1 scrub ok 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.231042+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.231042+0000 osd.6 (osd.6) 161 : cluster [DBG] 2.7 scrub starts 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.233559+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:42:52.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:51 smithi050 bash[29234]: cluster 2024-04-26T16:42:50.233559+0000 osd.6 (osd.6) 162 : cluster [DBG] 2.7 scrub ok 2024-04-26T16:42:53.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.182816+0000 osd.5 (osd.5) 524 : cluster [DBG] 3.c scrub starts 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.182816+0000 osd.5 (osd.5) 524 : cluster [DBG] 3.c scrub starts 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.186795+0000 osd.5 (osd.5) 525 : cluster [DBG] 3.c scrub ok 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.186795+0000 osd.5 (osd.5) 525 : cluster [DBG] 3.c scrub ok 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.226812+0000 osd.2 (osd.2) 549 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.226812+0000 osd.2 (osd.2) 549 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.229701+0000 osd.2 (osd.2) 550 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:42:53.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:52 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.229701+0000 osd.2 (osd.2) 550 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:42:53.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.182816+0000 osd.5 (osd.5) 524 : cluster [DBG] 3.c scrub starts 2024-04-26T16:42:53.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.182816+0000 osd.5 (osd.5) 524 : cluster [DBG] 3.c scrub starts 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.186795+0000 osd.5 (osd.5) 525 : cluster [DBG] 3.c scrub ok 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.186795+0000 osd.5 (osd.5) 525 : cluster [DBG] 3.c scrub ok 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.226812+0000 osd.2 (osd.2) 549 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.226812+0000 osd.2 (osd.2) 549 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.229701+0000 osd.2 (osd.2) 550 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.229701+0000 osd.2 (osd.2) 550 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.182816+0000 osd.5 (osd.5) 524 : cluster [DBG] 3.c scrub starts 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.182816+0000 osd.5 (osd.5) 524 : cluster [DBG] 3.c scrub starts 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.186795+0000 osd.5 (osd.5) 525 : cluster [DBG] 3.c scrub ok 2024-04-26T16:42:53.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.186795+0000 osd.5 (osd.5) 525 : cluster [DBG] 3.c scrub ok 2024-04-26T16:42:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.226812+0000 osd.2 (osd.2) 549 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:42:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.226812+0000 osd.2 (osd.2) 549 : cluster [DBG] 11.15 scrub starts 2024-04-26T16:42:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.229701+0000 osd.2 (osd.2) 550 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:42:53.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:52 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.229701+0000 osd.2 (osd.2) 550 : cluster [DBG] 11.15 scrub ok 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/467985436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc108740 msgr2=0x7f07dc110350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/467985436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc108740 0x7f07dc110350 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009fc0 tx=0x7f07d400b330 comp rx=0 tx=0).stop 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/467985436 shutdown_connections 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/467985436 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06bdf0 0x7f07dc06c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/467985436 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc108740 0x7f07dc110350 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/467985436 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07dc06c7a0 0x7f07dc108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/467985436 >> 172.21.15.50:0/467985436 conn(0x7f07dc078370 msgr2=0x7f07dc078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/467985436 shutdown_connections 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/467985436 wait complete. 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 librados: starting msgr at 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 librados: starting objecter 2024-04-26T16:42:53.952 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 Processor -- start 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- start start 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 librados: setting wanted keys 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 librados: calling monclient init 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 0x7f07dc102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06c7a0 0x7f07dc1029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07dc108740 0x7f07dc102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc112ae0 con 0x7f07dc06bdf0 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc112960 con 0x7f07dc06c7a0 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc112c60 con 0x7f07dc108740 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 0x7f07dc102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e1129700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07dc108740 0x7f07dc102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 0x7f07dc102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52270/0 (socket says 172.21.15.50:52270) 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 -- 172.21.15.50:0/2997367009 learned_addr learned my addr 172.21.15.50:0/2997367009 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07dbfff700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06c7a0 0x7f07dc1029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:53.953 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 -- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07dc108740 msgr2=0x7f07dc102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07dc108740 0x7f07dc102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 -- 172.21.15.50:0/2997367009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06c7a0 msgr2=0x7f07dc1029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06c7a0 0x7f07dc1029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 -- 172.21.15.50:0/2997367009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc1a7610 con 0x7f07dc06bdf0 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e0928700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 0x7f07dc102480 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f07d000d950 tx=0x7f07d000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07d9ffb700 10 client.?.objecter ms_handle_connect 0x7f07dc06bdf0 2024-04-26T16:42:53.954 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07d9ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:53.955 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0016070 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f07d0009da0 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc1a7900 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0010e60 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 10 client.31329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc1a7e40 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.952+0000 7f07e2b8c700 1 librados: init done 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07dc06a110 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f07d0019070 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f07cc071f80 0x7f07cc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f07d0091350 con 0x7f07dc06bdf0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 10 client.31329.objecter ms_dispatch 0x7f07dc001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 3 client.31329.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 3 client.31329.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07dbfff700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f07cc071f80 0x7f07cc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:53.956 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 20 client.31329.objecter dump_active .. 0 homeless 2024-04-26T16:42:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07d9ffb700 10 client.31329.objecter ms_handle_connect 0x7f07cc071f80 2024-04-26T16:42:53.957 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.956+0000 7f07dbfff700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f07cc071f80 0x7f07cc074440 secure :-1 s=READY pgs=3882 cs=0 l=1 rev1=1 crypto rx=0x7f07dc109d20 tx=0x7f07c8009310 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:53.960 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:53.960+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f07d005cfa0 con 0x7f07dc06bdf0 2024-04-26T16:42:54.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.139584+0000 mgr.y (mgr.24401) 965 : cluster [DBG] pgmap v1567: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.139584+0000 mgr.y (mgr.24401) 965 : cluster [DBG] pgmap v1567: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.177924+0000 osd.1 (osd.1) 603 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.177924+0000 osd.1 (osd.1) 603 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.180396+0000 osd.1 (osd.1) 604 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.180396+0000 osd.1 (osd.1) 604 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.764784+0000 osd.3 (osd.3) 671 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.764784+0000 osd.3 (osd.3) 671 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.767674+0000 osd.3 (osd.3) 672 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:52.767674+0000 osd.3 (osd.3) 672 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.197836+0000 osd.5 (osd.5) 526 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.197836+0000 osd.5 (osd.5) 526 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.200325+0000 osd.5 (osd.5) 527 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:42:54.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.200325+0000 osd.5 (osd.5) 527 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:42:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.223950+0000 osd.2 (osd.2) 551 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:42:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.223950+0000 osd.2 (osd.2) 551 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:42:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.231654+0000 osd.2 (osd.2) 552 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:42:54.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:53 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.231654+0000 osd.2 (osd.2) 552 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:42:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.139584+0000 mgr.y (mgr.24401) 965 : cluster [DBG] pgmap v1567: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.139584+0000 mgr.y (mgr.24401) 965 : cluster [DBG] pgmap v1567: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.177924+0000 osd.1 (osd.1) 603 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:42:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.177924+0000 osd.1 (osd.1) 603 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:42:54.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.180396+0000 osd.1 (osd.1) 604 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.180396+0000 osd.1 (osd.1) 604 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.764784+0000 osd.3 (osd.3) 671 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.764784+0000 osd.3 (osd.3) 671 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.767674+0000 osd.3 (osd.3) 672 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:52.767674+0000 osd.3 (osd.3) 672 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.197836+0000 osd.5 (osd.5) 526 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.197836+0000 osd.5 (osd.5) 526 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.200325+0000 osd.5 (osd.5) 527 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.200325+0000 osd.5 (osd.5) 527 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.223950+0000 osd.2 (osd.2) 551 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.223950+0000 osd.2 (osd.2) 551 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.231654+0000 osd.2 (osd.2) 552 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:42:54.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.231654+0000 osd.2 (osd.2) 552 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.139584+0000 mgr.y (mgr.24401) 965 : cluster [DBG] pgmap v1567: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.139584+0000 mgr.y (mgr.24401) 965 : cluster [DBG] pgmap v1567: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.177924+0000 osd.1 (osd.1) 603 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.177924+0000 osd.1 (osd.1) 603 : cluster [DBG] 11.0 scrub starts 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.180396+0000 osd.1 (osd.1) 604 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.180396+0000 osd.1 (osd.1) 604 : cluster [DBG] 11.0 scrub ok 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.764784+0000 osd.3 (osd.3) 671 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.764784+0000 osd.3 (osd.3) 671 : cluster [DBG] 3.1e scrub starts 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.767674+0000 osd.3 (osd.3) 672 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:52.767674+0000 osd.3 (osd.3) 672 : cluster [DBG] 3.1e scrub ok 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.197836+0000 osd.5 (osd.5) 526 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.197836+0000 osd.5 (osd.5) 526 : cluster [DBG] 6.a deep-scrub starts 2024-04-26T16:42:54.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.200325+0000 osd.5 (osd.5) 527 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:42:54.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.200325+0000 osd.5 (osd.5) 527 : cluster [DBG] 6.a deep-scrub ok 2024-04-26T16:42:54.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.223950+0000 osd.2 (osd.2) 551 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:42:54.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.223950+0000 osd.2 (osd.2) 551 : cluster [DBG] 11.1f scrub starts 2024-04-26T16:42:54.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.231654+0000 osd.2 (osd.2) 552 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:42:54.133 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:53 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.231654+0000 osd.2 (osd.2) 552 : cluster [DBG] 11.1f scrub ok 2024-04-26T16:42:54.185 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f07dc1a80d0 con 0x7f07dc06bdf0 2024-04-26T16:42:54.186 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07d9ffb700 1 -- 172.21.15.50:0/2997367009 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f07d0060c50 con 0x7f07dc06bdf0 2024-04-26T16:42:54.186 INFO:teuthology.orchestra.run.smithi050.stdout:77309411695 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 10 librados: watch_flush enter 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 10 librados: watch_flush exit 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 20 client.31329.objecter shutdown clearing up homeless session... 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 10 client.31329.objecter successfully canceled tick 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f07cc071f80 msgr2=0x7f07cc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f07cc071f80 0x7f07cc074440 secure :-1 s=READY pgs=3882 cs=0 l=1 rev1=1 crypto rx=0x7f07dc109d20 tx=0x7f07c8009310 comp rx=0 tx=0).stop 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 msgr2=0x7f07dc102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.189 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 0x7f07dc102480 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f07d000d950 tx=0x7f07d000dc60 comp rx=0 tx=0).stop 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 shutdown_connections 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f07cc071f80 0x7f07cc074440 unknown :-1 s=CLOSED pgs=3882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f07dc108740 0x7f07dc102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f07dc06bdf0 0x7f07dc102480 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 --2- 172.21.15.50:0/2997367009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f07dc06c7a0 0x7f07dc1029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 >> 172.21.15.50:0/2997367009 conn(0x7f07dc078370 msgr2=0x7f07dc100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:54.190 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 shutdown_connections 2024-04-26T16:42:54.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 -- 172.21.15.50:0/2997367009 wait complete. 2024-04-26T16:42:54.191 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.188+0000 7f07e2b8c700 1 librados: shutdown 2024-04-26T16:42:54.481 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:42:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:42:54.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- 172.21.15.50:0/941348827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd0101ae0 msgr2=0x7f4cd0101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- 172.21.15.50:0/941348827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd0101ae0 0x7f4cd0101f60 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8009990 tx=0x7f4cb8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- 172.21.15.50:0/941348827 shutdown_connections 2024-04-26T16:42:54.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- 172.21.15.50:0/941348827 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cd01024a0 0x7f4cd010e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.620 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- 172.21.15.50:0/941348827 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4cd0107a90 0x7f4cd0107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- 172.21.15.50:0/941348827 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd0101ae0 0x7f4cd0101f60 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- 172.21.15.50:0/941348827 >> 172.21.15.50:0/941348827 conn(0x7f4cd0078990 msgr2=0x7f4cd00ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- 172.21.15.50:0/941348827 shutdown_connections 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- 172.21.15.50:0/941348827 wait complete. 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 librados: starting msgr at 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 librados: starting objecter 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 Processor -- start 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- start start 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 librados: setting wanted keys 2024-04-26T16:42:54.621 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 librados: calling monclient init 2024-04-26T16:42:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cd0101ae0 0x7f4cd0199160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 0x7f4cd01996a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4cd0107a90 0x7f4cd019da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd0110910 con 0x7f4cd0107a90 2024-04-26T16:42:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd0110790 con 0x7f4cd0101ae0 2024-04-26T16:42:54.623 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cd5399700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd0110a90 con 0x7f4cd01024a0 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cce7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 0x7f4cd01996a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cce7fc700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 0x7f4cd01996a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37446/0 (socket says 172.21.15.50:37446) 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4cce7fc700 1 -- 172.21.15.50:0/3314071258 learned_addr learned my addr 172.21.15.50:0/3314071258 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.620+0000 7f4ccf7fe700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4cd0107a90 0x7f4cd019da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cceffd700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cd0101ae0 0x7f4cd0199160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cce7fc700 1 -- 172.21.15.50:0/3314071258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cd0101ae0 msgr2=0x7f4cd0199160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cce7fc700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cd0101ae0 0x7f4cd0199160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cce7fc700 1 -- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4cd0107a90 msgr2=0x7f4cd019da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cce7fc700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4cd0107a90 0x7f4cd019da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cce7fc700 1 -- 172.21.15.50:0/3314071258 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd019e170 con 0x7f4cd01024a0 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cce7fc700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 0x7f4cd01996a0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8000c00 tx=0x7f4cb8004790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:54.624 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 10 client.?.objecter ms_handle_connect 0x7f4cd01024a0 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8027070 con 0x7f4cd01024a0 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd019e400 con 0x7f4cd01024a0 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4cb8022470 con 0x7f4cd01024a0 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8017650 con 0x7f4cd01024a0 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cd5399700 10 client.41954.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd00fdd30 con 0x7f4cd01024a0 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cd5399700 1 librados: init done 2024-04-26T16:42:54.625 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd01031d0 con 0x7f4cd01024a0 2024-04-26T16:42:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4cb802a070 con 0x7f4cd01024a0 2024-04-26T16:42:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4cbc071f80 0x7f4cbc074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f4cb8092800 con 0x7f4cd01024a0 2024-04-26T16:42:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 10 client.41954.objecter ms_dispatch 0x7f4cd0001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 3 client.41954.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:54.626 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 3 client.41954.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cceffd700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4cbc071f80 0x7f4cbc074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 20 client.41954.objecter dump_active .. 0 homeless 2024-04-26T16:42:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cc7fff700 10 client.41954.objecter ms_handle_connect 0x7f4cbc071f80 2024-04-26T16:42:54.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.624+0000 7f4cceffd700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4cbc071f80 0x7f4cbc074440 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800a510 tx=0x7f4cc800aa30 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:54.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.628+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4cb8013070 con 0x7f4cd01024a0 2024-04-26T16:42:54.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f4cd019a4c0 con 0x7f4cd01024a0 2024-04-26T16:42:54.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cc7fff700 1 -- 172.21.15.50:0/3314071258 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f4cb805e450 con 0x7f4cd01024a0 2024-04-26T16:42:54.860 INFO:teuthology.orchestra.run.smithi050.stdout:150323855706 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 10 librados: watch_flush enter 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 10 librados: watch_flush exit 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 20 client.41954.objecter shutdown clearing up homeless session... 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 10 client.41954.objecter successfully canceled tick 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4cbc071f80 msgr2=0x7f4cbc074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4cbc071f80 0x7f4cbc074440 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800a510 tx=0x7f4cc800aa30 comp rx=0 tx=0).stop 2024-04-26T16:42:54.862 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 msgr2=0x7f4cd01996a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.860+0000 7f4cd5399700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 0x7f4cd01996a0 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f4cb8000c00 tx=0x7f4cb8004790 comp rx=0 tx=0).stop 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 shutdown_connections 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4cbc071f80 0x7f4cbc074440 unknown :-1 s=CLOSED pgs=3883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cd0101ae0 0x7f4cd0199160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4cd0107a90 0x7f4cd019da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 --2- 172.21.15.50:0/3314071258 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4cd01024a0 0x7f4cd01996a0 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 >> 172.21.15.50:0/3314071258 conn(0x7f4cd0078990 msgr2=0x7f4cd01004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 shutdown_connections 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 -- 172.21.15.50:0/3314071258 wait complete. 2024-04-26T16:42:54.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.864+0000 7f4cd5399700 1 librados: shutdown 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- 172.21.15.50:0/814151816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 msgr2=0x7f6d8c108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- 172.21.15.50:0/814151816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c108770 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009990 tx=0x7f6d7c009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- 172.21.15.50:0/814151816 shutdown_connections 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- 172.21.15.50:0/814151816 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d8c102ce0 0x7f6d8c10f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- 172.21.15.50:0/814151816 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c108770 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- 172.21.15.50:0/814151816 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102320 0x7f6d8c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- 172.21.15.50:0/814151816 >> 172.21.15.50:0/814151816 conn(0x7f6d8c0792d0 msgr2=0x7f6d8c0796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- 172.21.15.50:0/814151816 shutdown_connections 2024-04-26T16:42:54.868 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- 172.21.15.50:0/814151816 wait complete. 2024-04-26T16:42:54.870 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 librados: starting msgr at 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 librados: starting objecter 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 Processor -- start 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- start start 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 librados: setting wanted keys 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 librados: calling monclient init 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d8c102320 0x7f6d8c199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 0x7f6d8c19a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c1a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c111990 con 0x7f6d8c108390 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6d8c111810 con 0x7f6d8c102320 2024-04-26T16:42:54.871 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d92a41700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6d8c111b10 con 0x7f6d8c102ce0 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d837fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 0x7f6d8c19a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d90fde700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c1a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d837fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 0x7f6d8c19a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37466/0 (socket says 172.21.15.50:37466) 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.868+0000 7f6d837fe700 1 -- 172.21.15.50:0/1429959758 learned_addr learned my addr 172.21.15.50:0/1429959758 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d8bfff700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d8c102320 0x7f6d8c199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d837fe700 1 -- 172.21.15.50:0/1429959758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d8c102320 msgr2=0x7f6d8c199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d837fe700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d8c102320 0x7f6d8c199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d837fe700 1 -- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 msgr2=0x7f6d8c1a4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d837fe700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c1a4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d837fe700 1 -- 172.21.15.50:0/1429959758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d8c1a54c0 con 0x7f6d8c102ce0 2024-04-26T16:42:54.872 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d90fde700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c1a4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d837fe700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 0x7f6d8c19a440 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400e9a0 tx=0x7f6d7400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 10 client.?.objecter ms_handle_connect 0x7f6d8c102ce0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7400cbd0 con 0x7f6d8c102ce0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d8c1a56f0 con 0x7f6d8c102ce0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6d7400cd70 con 0x7f6d8c102ce0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d74018420 con 0x7f6d8c102ce0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d92a41700 10 client.41960.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d8c1a5c30 con 0x7f6d8c102ce0 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d92a41700 1 librados: init done 2024-04-26T16:42:54.873 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d8c06a110 con 0x7f6d8c102ce0 2024-04-26T16:42:54.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6d74010430 con 0x7f6d8c102ce0 2024-04-26T16:42:54.874 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d78071f80 0x7f6d78074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f6d74091f70 con 0x7f6d8c102ce0 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 10 client.41960.objecter ms_dispatch 0x7f6d8c001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 3 client.41960.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 3 client.41960.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d8bfff700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d78071f80 0x7f6d78074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.872+0000 7f6d89ffb700 20 client.41960.objecter dump_active .. 0 homeless 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.876+0000 7f6d8bfff700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d78071f80 0x7f6d78074440 secure :-1 s=READY pgs=3884 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00b730 tx=0x7f6d7c01a040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:54.875 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.876+0000 7f6d89ffb700 10 client.41960.objecter ms_handle_connect 0x7f6d78071f80 2024-04-26T16:42:54.877 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:54.876+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6d7405dc40 con 0x7f6d8c102ce0 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.183373+0000 osd.1 (osd.1) 605 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.183373+0000 osd.1 (osd.1) 605 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.185860+0000 osd.1 (osd.1) 606 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.185860+0000 osd.1 (osd.1) 606 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.200610+0000 osd.0 (osd.0) 625 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.200610+0000 osd.0 (osd.0) 625 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.202977+0000 osd.0 (osd.0) 626 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.202977+0000 osd.0 (osd.0) 626 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.226400+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.226400+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.228688+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.228688+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:42:55.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.737169+0000 osd.3 (osd.3) 673 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.737169+0000 osd.3 (osd.3) 673 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.741827+0000 osd.3 (osd.3) 674 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:53.741827+0000 osd.3 (osd.3) 674 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: audit 2024-04-26T16:42:54.191741+0000 mon.a (mon.0) 4073 : audit [DBG] from='client.? 172.21.15.50:0/2997367009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: audit 2024-04-26T16:42:54.191741+0000 mon.a (mon.0) 4073 : audit [DBG] from='client.? 172.21.15.50:0/2997367009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.226459+0000 osd.2 (osd.2) 553 : cluster [DBG] 6.f scrub starts 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.226459+0000 osd.2 (osd.2) 553 : cluster [DBG] 6.f scrub starts 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.228032+0000 osd.2 (osd.2) 554 : cluster [DBG] 6.f scrub ok 2024-04-26T16:42:55.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:54 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.228032+0000 osd.2 (osd.2) 554 : cluster [DBG] 6.f scrub ok 2024-04-26T16:42:55.101 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.100+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6d8c1a5e90 con 0x7f6d8c102ce0 2024-04-26T16:42:55.102 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.104+0000 7f6d89ffb700 1 -- 172.21.15.50:0/1429959758 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6d740618f0 con 0x7f6d8c102ce0 2024-04-26T16:42:55.102 INFO:teuthology.orchestra.run.smithi050.stdout:201863463248 2024-04-26T16:42:55.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 10 librados: watch_flush enter 2024-04-26T16:42:55.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 10 librados: watch_flush exit 2024-04-26T16:42:55.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 20 client.41960.objecter shutdown clearing up homeless session... 2024-04-26T16:42:55.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 10 client.41960.objecter successfully canceled tick 2024-04-26T16:42:55.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d78071f80 msgr2=0x7f6d78074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:55.106 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d78071f80 0x7f6d78074440 secure :-1 s=READY pgs=3884 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c00b730 tx=0x7f6d7c01a040 comp rx=0 tx=0).stop 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 msgr2=0x7f6d8c19a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 0x7f6d8c19a440 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f6d7400e9a0 tx=0x7f6d7400ecb0 comp rx=0 tx=0).stop 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 shutdown_connections 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6d78071f80 0x7f6d78074440 unknown :-1 s=CLOSED pgs=3884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d8c102320 0x7f6d8c199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6d8c108390 0x7f6d8c1a4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 --2- 172.21.15.50:0/1429959758 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6d8c102ce0 0x7f6d8c19a440 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:55.107 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 >> 172.21.15.50:0/1429959758 conn(0x7f6d8c0792d0 msgr2=0x7f6d8c100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:55.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 shutdown_connections 2024-04-26T16:42:55.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 -- 172.21.15.50:0/1429959758 wait complete. 2024-04-26T16:42:55.108 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:55.108+0000 7f6d92a41700 1 librados: shutdown 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.183373+0000 osd.1 (osd.1) 605 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.183373+0000 osd.1 (osd.1) 605 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.185860+0000 osd.1 (osd.1) 606 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.185860+0000 osd.1 (osd.1) 606 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.200610+0000 osd.0 (osd.0) 625 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.200610+0000 osd.0 (osd.0) 625 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.202977+0000 osd.0 (osd.0) 626 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.202977+0000 osd.0 (osd.0) 626 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:42:55.118 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.226400+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.226400+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.228688+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.228688+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.737169+0000 osd.3 (osd.3) 673 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.737169+0000 osd.3 (osd.3) 673 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.741827+0000 osd.3 (osd.3) 674 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:53.741827+0000 osd.3 (osd.3) 674 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: audit 2024-04-26T16:42:54.191741+0000 mon.a (mon.0) 4073 : audit [DBG] from='client.? 172.21.15.50:0/2997367009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: audit 2024-04-26T16:42:54.191741+0000 mon.a (mon.0) 4073 : audit [DBG] from='client.? 172.21.15.50:0/2997367009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.226459+0000 osd.2 (osd.2) 553 : cluster [DBG] 6.f scrub starts 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.226459+0000 osd.2 (osd.2) 553 : cluster [DBG] 6.f scrub starts 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.228032+0000 osd.2 (osd.2) 554 : cluster [DBG] 6.f scrub ok 2024-04-26T16:42:55.119 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.228032+0000 osd.2 (osd.2) 554 : cluster [DBG] 6.f scrub ok 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.183373+0000 osd.1 (osd.1) 605 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.183373+0000 osd.1 (osd.1) 605 : cluster [DBG] 1.3 scrub starts 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.185860+0000 osd.1 (osd.1) 606 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.185860+0000 osd.1 (osd.1) 606 : cluster [DBG] 1.3 scrub ok 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.200610+0000 osd.0 (osd.0) 625 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.200610+0000 osd.0 (osd.0) 625 : cluster [DBG] 6.18 scrub starts 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.202977+0000 osd.0 (osd.0) 626 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.202977+0000 osd.0 (osd.0) 626 : cluster [DBG] 6.18 scrub ok 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.226400+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.226400+0000 osd.6 (osd.6) 163 : cluster [DBG] 2.14 scrub starts 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.228688+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:42:55.120 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.228688+0000 osd.6 (osd.6) 164 : cluster [DBG] 2.14 scrub ok 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.737169+0000 osd.3 (osd.3) 673 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.737169+0000 osd.3 (osd.3) 673 : cluster [DBG] 3.7 scrub starts 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.741827+0000 osd.3 (osd.3) 674 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:53.741827+0000 osd.3 (osd.3) 674 : cluster [DBG] 3.7 scrub ok 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: audit 2024-04-26T16:42:54.191741+0000 mon.a (mon.0) 4073 : audit [DBG] from='client.? 172.21.15.50:0/2997367009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: audit 2024-04-26T16:42:54.191741+0000 mon.a (mon.0) 4073 : audit [DBG] from='client.? 172.21.15.50:0/2997367009' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.226459+0000 osd.2 (osd.2) 553 : cluster [DBG] 6.f scrub starts 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.226459+0000 osd.2 (osd.2) 553 : cluster [DBG] 6.f scrub starts 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.228032+0000 osd.2 (osd.2) 554 : cluster [DBG] 6.f scrub ok 2024-04-26T16:42:55.121 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:54 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.228032+0000 osd.2 (osd.2) 554 : cluster [DBG] 6.f scrub ok 2024-04-26T16:42:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.140327+0000 mgr.y (mgr.24401) 966 : cluster [DBG] pgmap v1568: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.140327+0000 mgr.y (mgr.24401) 966 : cluster [DBG] pgmap v1568: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.156202+0000 osd.0 (osd.0) 627 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:42:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.156202+0000 osd.0 (osd.0) 627 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:42:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.160164+0000 osd.0 (osd.0) 628 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:42:56.078 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.160164+0000 osd.0 (osd.0) 628 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.179723+0000 osd.1 (osd.1) 607 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.179723+0000 osd.1 (osd.1) 607 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.181733+0000 osd.1 (osd.1) 608 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: cluster 2024-04-26T16:42:54.181733+0000 osd.1 (osd.1) 608 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: audit 2024-04-26T16:42:54.864639+0000 mon.c (mon.2) 956 : audit [DBG] from='client.? 172.21.15.50:0/3314071258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: audit 2024-04-26T16:42:54.864639+0000 mon.c (mon.2) 956 : audit [DBG] from='client.? 172.21.15.50:0/3314071258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: audit 2024-04-26T16:42:55.107253+0000 mon.c (mon.2) 957 : audit [DBG] from='client.? 172.21.15.50:0/1429959758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:42:56.079 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:55 smithi190 bash[23868]: audit 2024-04-26T16:42:55.107253+0000 mon.c (mon.2) 957 : audit [DBG] from='client.? 172.21.15.50:0/1429959758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:42:56.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.140327+0000 mgr.y (mgr.24401) 966 : cluster [DBG] pgmap v1568: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:56.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.140327+0000 mgr.y (mgr.24401) 966 : cluster [DBG] pgmap v1568: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:56.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.156202+0000 osd.0 (osd.0) 627 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:42:56.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.156202+0000 osd.0 (osd.0) 627 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:42:56.150 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.160164+0000 osd.0 (osd.0) 628 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.160164+0000 osd.0 (osd.0) 628 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.179723+0000 osd.1 (osd.1) 607 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.179723+0000 osd.1 (osd.1) 607 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.181733+0000 osd.1 (osd.1) 608 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: cluster 2024-04-26T16:42:54.181733+0000 osd.1 (osd.1) 608 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: audit 2024-04-26T16:42:54.864639+0000 mon.c (mon.2) 956 : audit [DBG] from='client.? 172.21.15.50:0/3314071258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: audit 2024-04-26T16:42:54.864639+0000 mon.c (mon.2) 956 : audit [DBG] from='client.? 172.21.15.50:0/3314071258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: audit 2024-04-26T16:42:55.107253+0000 mon.c (mon.2) 957 : audit [DBG] from='client.? 172.21.15.50:0/1429959758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[29234]: audit 2024-04-26T16:42:55.107253+0000 mon.c (mon.2) 957 : audit [DBG] from='client.? 172.21.15.50:0/1429959758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:42:56.151 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.140327+0000 mgr.y (mgr.24401) 966 : cluster [DBG] pgmap v1568: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.140327+0000 mgr.y (mgr.24401) 966 : cluster [DBG] pgmap v1568: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.156202+0000 osd.0 (osd.0) 627 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.156202+0000 osd.0 (osd.0) 627 : cluster [DBG] 3.1b deep-scrub starts 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.160164+0000 osd.0 (osd.0) 628 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.160164+0000 osd.0 (osd.0) 628 : cluster [DBG] 3.1b deep-scrub ok 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.179723+0000 osd.1 (osd.1) 607 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.179723+0000 osd.1 (osd.1) 607 : cluster [DBG] 4.2 scrub starts 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.181733+0000 osd.1 (osd.1) 608 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: cluster 2024-04-26T16:42:54.181733+0000 osd.1 (osd.1) 608 : cluster [DBG] 4.2 scrub ok 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: audit 2024-04-26T16:42:54.864639+0000 mon.c (mon.2) 956 : audit [DBG] from='client.? 172.21.15.50:0/3314071258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: audit 2024-04-26T16:42:54.864639+0000 mon.c (mon.2) 956 : audit [DBG] from='client.? 172.21.15.50:0/3314071258' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-26T16:42:56.152 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: audit 2024-04-26T16:42:55.107253+0000 mon.c (mon.2) 957 : audit [DBG] from='client.? 172.21.15.50:0/1429959758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:42:56.153 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:55 smithi050 bash[21463]: audit 2024-04-26T16:42:55.107253+0000 mon.c (mon.2) 957 : audit [DBG] from='client.? 172.21.15.50:0/1429959758' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-26T16:42:56.282 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411691 got 77309411695 for osd.2 2024-04-26T16:42:56.285 DEBUG:teuthology.parallel:result is None 2024-04-26T16:42:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- 172.21.15.50:0/2539871426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc109490 msgr2=0x7f0bdc109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.429 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2539871426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc109490 0x7f0bdc109870 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098f0 tx=0x7f0bd8009c00 comp rx=0 tx=0).stop 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- 172.21.15.50:0/2539871426 shutdown_connections 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2539871426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bdc06c360 0x7f0bdc1102d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2539871426 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc109490 0x7f0bdc109870 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2539871426 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc06b9c0 0x7f0bdc06be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- 172.21.15.50:0/2539871426 >> 172.21.15.50:0/2539871426 conn(0x7f0bdc079290 msgr2=0x7f0bdc0796a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- 172.21.15.50:0/2539871426 shutdown_connections 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- 172.21.15.50:0/2539871426 wait complete. 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 librados: starting msgr at 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 librados: starting objecter 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 Processor -- start 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- start start 2024-04-26T16:42:56.430 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 librados: setting wanted keys 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 librados: calling monclient init 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bdc06b9c0 0x7f0bdc1023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 0x7f0bdc102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc109490 0x7f0bdc102e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc112a70 con 0x7f0bdc06c360 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc1128f0 con 0x7f0bdc06b9c0 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be34cf700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc112bf0 con 0x7f0bdc109490 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 0x7f0bdc102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be1a6c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc109490 0x7f0bdc102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.431 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 0x7f0bdc102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52322/0 (socket says 172.21.15.50:52322) 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 -- 172.21.15.50:0/2754697807 learned_addr learned my addr 172.21.15.50:0/2754697807 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be126b700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bdc06b9c0 0x7f0bdc1023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 -- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc109490 msgr2=0x7f0bdc102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc109490 0x7f0bdc102e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 -- 172.21.15.50:0/2754697807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bdc06b9c0 msgr2=0x7f0bdc1023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bdc06b9c0 0x7f0bdc1023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 -- 172.21.15.50:0/2754697807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc1a76f0 con 0x7f0bdc06c360 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be1a6c700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc109490 0x7f0bdc102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0bd27fc700 10 client.?.objecter ms_handle_connect 0x7f0bdc06c360 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0bd27fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.428+0000 7f0be0a6a700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 0x7f0bdc102920 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00ec10 tx=0x7f0bcc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc00cd40 con 0x7f0bdc06c360 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f0bcc004480 con 0x7f0bdc06c360 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bdc1a79e0 con 0x7f0bdc06c360 2024-04-26T16:42:56.432 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be34cf700 10 client.31347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:56.433 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc010430 con 0x7f0bdc06c360 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bdc1a7f20 con 0x7f0bdc06c360 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be34cf700 1 librados: init done 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f0bcc004020 con 0x7f0bdc06c360 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bdc104b40 con 0x7f0bdc06c360 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bc8071e50 0x7f0bc8074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f0bcc0943c0 con 0x7f0bdc06c360 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 10 client.31347.objecter ms_dispatch 0x7f0bdc001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 3 client.31347.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 3 client.31347.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:56.434 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be126b700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bc8071e50 0x7f0bc8074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 20 client.31347.objecter dump_active .. 0 homeless 2024-04-26T16:42:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0bd27fc700 10 client.31347.objecter ms_handle_connect 0x7f0bc8071e50 2024-04-26T16:42:56.435 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.432+0000 7f0be126b700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bc8071e50 0x7f0bc8074310 secure :-1 s=READY pgs=3885 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009810 tx=0x7f0bd8019040 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:56.436 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.436+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f0bcc05d010 con 0x7f0bdc06c360 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/2214435986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc10fd70 msgr2=0x7f4abc112160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/2214435986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc10fd70 0x7f4abc112160 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b0a0 tx=0x7f4ab400b3b0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/2214435986 shutdown_connections 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/2214435986 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc1043a0 0x7f4abc06bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/2214435986 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc10fd70 0x7f4abc112160 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/2214435986 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc06c090 0x7f4abc06c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/2214435986 >> 172.21.15.50:0/2214435986 conn(0x7f4abc078190 msgr2=0x7f4abc0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/2214435986 shutdown_connections 2024-04-26T16:42:56.483 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/2214435986 wait complete. 2024-04-26T16:42:56.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 librados: starting msgr at 2024-04-26T16:42:56.485 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 librados: starting objecter 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 Processor -- start 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- start start 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 librados: setting wanted keys 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 librados: calling monclient init 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc06c090 0x7f4abc19b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 0x7f4abc19b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc10fd70 0x7f4abc19fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f4abc114d90 con 0x7f4abc1043a0 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f4abc114c10 con 0x7f4abc06c090 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f4abc114f10 con 0x7f4abc10fd70 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 0x7f4abc19b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 0x7f4abc19b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52342/0 (socket says 172.21.15.50:52342) 2024-04-26T16:42:56.486 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 -- 172.21.15.50:0/794885813 learned_addr learned my addr 172.21.15.50:0/794885813 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac13c1700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc10fd70 0x7f4abc19fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac0bc0700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc06c090 0x7f4abc19b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 -- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc10fd70 msgr2=0x7f4abc19fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.487 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc10fd70 0x7f4abc19fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 -- 172.21.15.50:0/794885813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc06c090 msgr2=0x7f4abc19b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc06c090 0x7f4abc19b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 -- 172.21.15.50:0/794885813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4abc1a0250 con 0x7f4abc1043a0 2024-04-26T16:42:56.489 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac13c1700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc10fd70 0x7f4abc19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4abbfff700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 0x7f4abc19b780 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800eb30 tx=0x7f4aa800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ab9ffb700 10 client.?.objecter ms_handle_connect 0x7f4abc1043a0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ab9ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa800cc90 con 0x7f4abc1043a0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f4aa800ce30 con 0x7f4abc1043a0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4abc1a0540 con 0x7f4abc1043a0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa8003d80 con 0x7f4abc1043a0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 10 client.31353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4abc1a7dc0 con 0x7f4abc1043a0 2024-04-26T16:42:56.490 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.484+0000 7f4ac2e24700 1 librados: init done 2024-04-26T16:42:56.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4abc1029a0 con 0x7f4abc1043a0 2024-04-26T16:42:56.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f4aa8003f20 con 0x7f4abc1043a0 2024-04-26T16:42:56.491 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4aac071f80 0x7f4aac074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f4aa80919b0 con 0x7f4abc1043a0 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 10 client.31353.objecter ms_dispatch 0x7f4abc001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 3 client.31353.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 3 client.31353.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ac0bc0700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4aac071f80 0x7f4aac074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 20 client.31353.objecter dump_active .. 0 homeless 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ab9ffb700 10 client.31353.objecter ms_handle_connect 0x7f4aac071f80 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.488+0000 7f4ac0bc0700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4aac071f80 0x7f4aac074440 secure :-1 s=READY pgs=3886 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0005fd0 tx=0x7f4ab000af20 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:56.492 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.492+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f4aa805d680 con 0x7f4abc1043a0 2024-04-26T16:42:56.680 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.680+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f0bdc1038c0 con 0x7f0bdc06c360 2024-04-26T16:42:56.681 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.680+0000 7f0bd27fc700 1 -- 172.21.15.50:0/2754697807 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0bcc060cc0 con 0x7f0bdc06c360 2024-04-26T16:42:56.681 INFO:teuthology.orchestra.run.smithi050.stdout:34359738756 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 10 librados: watch_flush enter 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 10 librados: watch_flush exit 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 20 client.31347.objecter shutdown clearing up homeless session... 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 10 client.31347.objecter successfully canceled tick 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bc8071e50 msgr2=0x7f0bc8074310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bc8071e50 0x7f0bc8074310 secure :-1 s=READY pgs=3885 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009810 tx=0x7f0bd8019040 comp rx=0 tx=0).stop 2024-04-26T16:42:56.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 msgr2=0x7f0bdc102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 0x7f0bdc102920 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00ec10 tx=0x7f0bcc00ef20 comp rx=0 tx=0).stop 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 shutdown_connections 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f0bc8071e50 0x7f0bc8074310 unknown :-1 s=CLOSED pgs=3885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0bdc06b9c0 0x7f0bdc1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f0bdc109490 0x7f0bdc102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 --2- 172.21.15.50:0/2754697807 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f0bdc06c360 0x7f0bdc102920 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 >> 172.21.15.50:0/2754697807 conn(0x7f0bdc079290 msgr2=0x7f0bdc100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 shutdown_connections 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 -- 172.21.15.50:0/2754697807 wait complete. 2024-04-26T16:42:56.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.684+0000 7f0be34cf700 1 librados: shutdown 2024-04-26T16:42:56.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.740+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4abc1a7ff0 con 0x7f4abc1043a0 2024-04-26T16:42:56.739 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.740+0000 7f4ab9ffb700 1 -- 172.21.15.50:0/794885813 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+14 (secure 0 0 0) 0x7f4aa8061330 con 0x7f4abc1043a0 2024-04-26T16:42:56.739 INFO:teuthology.orchestra.run.smithi050.stdout:3204045602908 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 10 librados: watch_flush enter 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 10 librados: watch_flush exit 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 20 client.31353.objecter shutdown clearing up homeless session... 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 10 client.31353.objecter successfully canceled tick 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4aac071f80 msgr2=0x7f4aac074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4aac071f80 0x7f4aac074440 secure :-1 s=READY pgs=3886 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0005fd0 tx=0x7f4ab000af20 comp rx=0 tx=0).stop 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 msgr2=0x7f4abc19b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 0x7f4abc19b780 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f4aa800eb30 tx=0x7f4aa800ee40 comp rx=0 tx=0).stop 2024-04-26T16:42:56.743 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 shutdown_connections 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4aac071f80 0x7f4aac074440 unknown :-1 s=CLOSED pgs=3886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc06c090 0x7f4abc19b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f4abc10fd70 0x7f4abc19fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 --2- 172.21.15.50:0/794885813 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f4abc1043a0 0x7f4abc19b780 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 >> 172.21.15.50:0/794885813 conn(0x7f4abc078190 msgr2=0x7f4abc101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 shutdown_connections 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 -- 172.21.15.50:0/794885813 wait complete. 2024-04-26T16:42:56.744 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.744+0000 7f4ac2e24700 1 librados: shutdown 2024-04-26T16:42:56.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- 172.21.15.50:0/1077799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b01094a0 msgr2=0x7f43b0109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.839 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- 172.21.15.50:0/1077799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b01094a0 0x7f43b0109880 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b0a0 tx=0x7f43a800b3b0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- 172.21.15.50:0/1077799345 shutdown_connections 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- 172.21.15.50:0/1077799345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b006bad0 0x7f43b01102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- 172.21.15.50:0/1077799345 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b01094a0 0x7f43b0109880 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- 172.21.15.50:0/1077799345 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43b006b130 0x7f43b006b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- 172.21.15.50:0/1077799345 >> 172.21.15.50:0/1077799345 conn(0x7f43b00792d0 msgr2=0x7f43b00796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- 172.21.15.50:0/1077799345 shutdown_connections 2024-04-26T16:42:56.840 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- 172.21.15.50:0/1077799345 wait complete. 2024-04-26T16:42:56.841 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 librados: starting msgr at 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 librados: starting objecter 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 Processor -- start 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- start start 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 librados: setting wanted keys 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 librados: calling monclient init 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b006b130 0x7f43b01991d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 0x7f43b0199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43b01094a0 0x7f43b019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0112b10 con 0x7f43b006bad0 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0112990 con 0x7f43b006b130 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43b557c700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f43b0112c90 con 0x7f43b01094a0 2024-04-26T16:42:56.842 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 0x7f43b0199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43af7fe700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43b01094a0 0x7f43b019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 0x7f43b0199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:52364/0 (socket says 172.21.15.50:52364) 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 -- 172.21.15.50:0/1111218821 learned_addr learned my addr 172.21.15.50:0/1111218821 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43aeffd700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b006b130 0x7f43b01991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 -- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43b01094a0 msgr2=0x7f43b019daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43b01094a0 0x7f43b019daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 -- 172.21.15.50:0/1111218821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b006b130 msgr2=0x7f43b01991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b006b130 0x7f43b01991d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 -- 172.21.15.50:0/1111218821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b019e1e0 con 0x7f43b006bad0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f43ae7fc700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 0x7f43b0199710 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f43a400eb30 tx=0x7f43a400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f4397fff700 10 client.?.objecter ms_handle_connect 0x7f43b006bad0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.840+0000 7f4397fff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a400cc90 con 0x7f43b006bad0 2024-04-26T16:42:56.843 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f43a400ce30 con 0x7f43b006bad0 2024-04-26T16:42:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b019e4d0 con 0x7f43b006bad0 2024-04-26T16:42:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4003d80 con 0x7f43b006bad0 2024-04-26T16:42:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43b557c700 10 client.31359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b01a5d50 con 0x7f43b006bad0 2024-04-26T16:42:56.844 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43b557c700 1 librados: init done 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b006a110 con 0x7f43b006bad0 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f43a4003f20 con 0x7f43b006bad0 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f43a4091a40 con 0x7f43b006bad0 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 10 client.31359.objecter ms_dispatch 0x7f43b0001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 3 client.31359.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 3 client.31359.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:56.845 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43aeffd700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 20 client.31359.objecter dump_active .. 0 homeless 2024-04-26T16:42:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f43aeffd700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 secure :-1 s=READY pgs=3887 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b780 tx=0x7f43a8009be0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:56.846 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.844+0000 7f4397fff700 10 client.31359.objecter ms_handle_connect 0x7f4398071f80 2024-04-26T16:42:56.849 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:56.848+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f43a405d690 con 0x7f43b006bad0 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.161554+0000 osd.0 (osd.0) 629 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.161554+0000 osd.0 (osd.0) 629 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.162799+0000 osd.0 (osd.0) 630 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.162799+0000 osd.0 (osd.0) 630 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.356491+0000 osd.7 (osd.7) 654 : cluster [DBG] 3.13 deep-scrub starts 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.356491+0000 osd.7 (osd.7) 654 : cluster [DBG] 3.13 deep-scrub starts 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.366012+0000 osd.7 (osd.7) 655 : cluster [DBG] 3.13 deep-scrub ok 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.366012+0000 osd.7 (osd.7) 655 : cluster [DBG] 3.13 deep-scrub ok 2024-04-26T16:42:57.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.711486+0000 osd.3 (osd.3) 675 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.711486+0000 osd.3 (osd.3) 675 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.716210+0000 osd.3 (osd.3) 676 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: cluster 2024-04-26T16:42:55.716210+0000 osd.3 (osd.3) 676 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: audit 2024-04-26T16:42:56.686786+0000 mon.a (mon.0) 4074 : audit [DBG] from='client.? 172.21.15.50:0/2754697807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: audit 2024-04-26T16:42:56.686786+0000 mon.a (mon.0) 4074 : audit [DBG] from='client.? 172.21.15.50:0/2754697807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: audit 2024-04-26T16:42:56.744967+0000 mon.a (mon.0) 4075 : audit [DBG] from='client.? 172.21.15.50:0/794885813' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:42:57.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:56 smithi190 bash[23868]: audit 2024-04-26T16:42:56.744967+0000 mon.a (mon.0) 4075 : audit [DBG] from='client.? 172.21.15.50:0/794885813' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:42:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.076+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f43b01a5f80 con 0x7f43b006bad0 2024-04-26T16:42:57.076 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.076+0000 7f4397fff700 1 -- 172.21.15.50:0/1111218821 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f43a4061340 con 0x7f43b006bad0 2024-04-26T16:42:57.076 INFO:teuthology.orchestra.run.smithi050.stdout:55834575226 2024-04-26T16:42:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.076+0000 7f43b557c700 10 librados: watch_flush enter 2024-04-26T16:42:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.076+0000 7f43b557c700 10 librados: watch_flush exit 2024-04-26T16:42:57.077 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.076+0000 7f43b557c700 20 client.31359.objecter shutdown clearing up homeless session... 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.076+0000 7f43b557c700 10 client.31359.objecter successfully canceled tick 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 msgr2=0x7f4398074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 secure :-1 s=READY pgs=3887 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b780 tx=0x7f43a8009be0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 msgr2=0x7f43b0199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 0x7f43b0199710 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f43a400eb30 tx=0x7f43a400ee40 comp rx=0 tx=0).stop 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 shutdown_connections 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f4398071f80 0x7f4398074440 unknown :-1 s=CLOSED pgs=3887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f43b006b130 0x7f43b01991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f43b01094a0 0x7f43b019daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 --2- 172.21.15.50:0/1111218821 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f43b006bad0 0x7f43b0199710 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 >> 172.21.15.50:0/1111218821 conn(0x7f43b00792d0 msgr2=0x7f43b0100190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 shutdown_connections 2024-04-26T16:42:57.078 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 -- 172.21.15.50:0/1111218821 wait complete. 2024-04-26T16:42:57.079 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.080+0000 7f43b557c700 1 librados: shutdown 2024-04-26T16:42:57.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.161554+0000 osd.0 (osd.0) 629 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:42:57.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.161554+0000 osd.0 (osd.0) 629 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:42:57.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.162799+0000 osd.0 (osd.0) 630 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:42:57.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.162799+0000 osd.0 (osd.0) 630 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:42:57.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.356491+0000 osd.7 (osd.7) 654 : cluster [DBG] 3.13 deep-scrub starts 2024-04-26T16:42:57.090 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.356491+0000 osd.7 (osd.7) 654 : cluster [DBG] 3.13 deep-scrub starts 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.366012+0000 osd.7 (osd.7) 655 : cluster [DBG] 3.13 deep-scrub ok 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.366012+0000 osd.7 (osd.7) 655 : cluster [DBG] 3.13 deep-scrub ok 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.711486+0000 osd.3 (osd.3) 675 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.711486+0000 osd.3 (osd.3) 675 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.716210+0000 osd.3 (osd.3) 676 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: cluster 2024-04-26T16:42:55.716210+0000 osd.3 (osd.3) 676 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: audit 2024-04-26T16:42:56.686786+0000 mon.a (mon.0) 4074 : audit [DBG] from='client.? 172.21.15.50:0/2754697807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: audit 2024-04-26T16:42:56.686786+0000 mon.a (mon.0) 4074 : audit [DBG] from='client.? 172.21.15.50:0/2754697807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: audit 2024-04-26T16:42:56.744967+0000 mon.a (mon.0) 4075 : audit [DBG] from='client.? 172.21.15.50:0/794885813' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[29234]: audit 2024-04-26T16:42:56.744967+0000 mon.a (mon.0) 4075 : audit [DBG] from='client.? 172.21.15.50:0/794885813' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:42:57.091 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.161554+0000 osd.0 (osd.0) 629 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.161554+0000 osd.0 (osd.0) 629 : cluster [DBG] 5.1e scrub starts 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.162799+0000 osd.0 (osd.0) 630 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.162799+0000 osd.0 (osd.0) 630 : cluster [DBG] 5.1e scrub ok 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.356491+0000 osd.7 (osd.7) 654 : cluster [DBG] 3.13 deep-scrub starts 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.356491+0000 osd.7 (osd.7) 654 : cluster [DBG] 3.13 deep-scrub starts 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.366012+0000 osd.7 (osd.7) 655 : cluster [DBG] 3.13 deep-scrub ok 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.366012+0000 osd.7 (osd.7) 655 : cluster [DBG] 3.13 deep-scrub ok 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.711486+0000 osd.3 (osd.3) 675 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.711486+0000 osd.3 (osd.3) 675 : cluster [DBG] 3.18 deep-scrub starts 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.716210+0000 osd.3 (osd.3) 676 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: cluster 2024-04-26T16:42:55.716210+0000 osd.3 (osd.3) 676 : cluster [DBG] 3.18 deep-scrub ok 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: audit 2024-04-26T16:42:56.686786+0000 mon.a (mon.0) 4074 : audit [DBG] from='client.? 172.21.15.50:0/2754697807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: audit 2024-04-26T16:42:56.686786+0000 mon.a (mon.0) 4074 : audit [DBG] from='client.? 172.21.15.50:0/2754697807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-26T16:42:57.092 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: audit 2024-04-26T16:42:56.744967+0000 mon.a (mon.0) 4075 : audit [DBG] from='client.? 172.21.15.50:0/794885813' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:42:57.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:56 smithi050 bash[21463]: audit 2024-04-26T16:42:56.744967+0000 mon.a (mon.0) 4075 : audit [DBG] from='client.? 172.21.15.50:0/794885813' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 -- 172.21.15.50:0/390377618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc10b660 msgr2=0x7f3fdc10ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/390377618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc10b660 0x7f3fdc10ba40 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8009990 tx=0x7f3fd8009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 -- 172.21.15.50:0/390377618 shutdown_connections 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/390377618 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc106020 0x7f3fdc112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/390377618 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc10b660 0x7f3fdc10ba40 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/390377618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fdc105660 0x7f3fdc105ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 -- 172.21.15.50:0/390377618 >> 172.21.15.50:0/390377618 conn(0x7f3fdc101150 msgr2=0x7f3fdc103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 -- 172.21.15.50:0/390377618 shutdown_connections 2024-04-26T16:42:57.237 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.236+0000 7f3fe3caa700 1 -- 172.21.15.50:0/390377618 wait complete. 2024-04-26T16:42:57.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 librados: starting msgr at 2024-04-26T16:42:57.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 librados: starting objecter 2024-04-26T16:42:57.239 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 Processor -- start 2024-04-26T16:42:57.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- start start 2024-04-26T16:42:57.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 librados: setting wanted keys 2024-04-26T16:42:57.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 librados: calling monclient init 2024-04-26T16:42:57.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 0x7f3fdc19b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:57.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc106020 0x7f3fdc19b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:57.240 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fdc10b660 0x7f3fdc19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:57.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc114dc0 con 0x7f3fdc106020 2024-04-26T16:42:57.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f3fdc114c40 con 0x7f3fdc10b660 2024-04-26T16:42:57.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f3fdc114f40 con 0x7f3fdc105660 2024-04-26T16:42:57.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 0x7f3fdc19b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:57.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 0x7f3fdc19b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:37532/0 (socket says 172.21.15.50:37532) 2024-04-26T16:42:57.241 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 -- 172.21.15.50:0/3342850075 learned_addr learned my addr 172.21.15.50:0/3342850075 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1245700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc106020 0x7f3fdc19b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 -- 172.21.15.50:0/3342850075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fdc10b660 msgr2=0x7f3fdc19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fdc10b660 0x7f3fdc19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 -- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc106020 msgr2=0x7f3fdc19b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc106020 0x7f3fdc19b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 -- 172.21.15.50:0/3342850075 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fdc1a0450 con 0x7f3fdc105660 2024-04-26T16:42:57.242 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1245700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc106020 0x7f3fdc19b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1a46700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 0x7f3fdc19b440 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80057a0 tx=0x7f3fd80057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 10 client.?.objecter ms_handle_connect 0x7f3fdc105660 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8026070 con 0x7f3fdc105660 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fdc1a06e0 con 0x7f3fdc105660 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f3fd8021410 con 0x7f3fdc105660 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80173f0 con 0x7f3fdc105660 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 10 client.41984.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fdc1a7fc0 con 0x7f3fdc105660 2024-04-26T16:42:57.243 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 librados: init done 2024-04-26T16:42:57.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fdc06a110 con 0x7f3fdc105660 2024-04-26T16:42:57.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f3fd8029070 con 0x7f3fdc105660 2024-04-26T16:42:57.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fc8071f80 0x7f3fc8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:57.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f3fd8092380 con 0x7f3fdc105660 2024-04-26T16:42:57.244 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 10 client.41984.objecter ms_dispatch 0x7f3fdc001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 3 client.41984.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fd2ffd700 3 client.41984.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.240+0000 7f3fe1245700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fc8071f80 0x7f3fc8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.244+0000 7f3fd2ffd700 20 client.41984.objecter dump_active .. 0 homeless 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.244+0000 7f3fd2ffd700 10 client.41984.objecter ms_handle_connect 0x7f3fc8071f80 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.244+0000 7f3fe1245700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fc8071f80 0x7f3fc8074440 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc10adb0 tx=0x7f3fcc009190 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:57.245 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.248+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f3fd8013070 con 0x7f3fdc105660 2024-04-26T16:42:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.464+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3fdc1a8190 con 0x7f3fdc105660 2024-04-26T16:42:57.465 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.464+0000 7f3fd2ffd700 1 -- 172.21.15.50:0/3342850075 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3fd8061d00 con 0x7f3fdc105660 2024-04-26T16:42:57.465 INFO:teuthology.orchestra.run.smithi050.stdout:120259084644 2024-04-26T16:42:57.467 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 10 librados: watch_flush enter 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 10 librados: watch_flush exit 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 20 client.41984.objecter shutdown clearing up homeless session... 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 10 client.41984.objecter successfully canceled tick 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fc8071f80 msgr2=0x7f3fc8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fc8071f80 0x7f3fc8074440 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc10adb0 tx=0x7f3fcc009190 comp rx=0 tx=0).stop 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 msgr2=0x7f3fdc19b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 0x7f3fdc19b440 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f3fd80057a0 tx=0x7f3fd80057d0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 shutdown_connections 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f3fc8071f80 0x7f3fc8074440 unknown :-1 s=CLOSED pgs=3888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3fdc10b660 0x7f3fdc19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f3fdc105660 0x7f3fdc19b440 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.468 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 --2- 172.21.15.50:0/3342850075 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f3fdc106020 0x7f3fdc19b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 >> 172.21.15.50:0/3342850075 conn(0x7f3fdc101150 msgr2=0x7f3fdc101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 shutdown_connections 2024-04-26T16:42:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 -- 172.21.15.50:0/3342850075 wait complete. 2024-04-26T16:42:57.469 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:57.468+0000 7f3fe3caa700 1 librados: shutdown 2024-04-26T16:42:58.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:42:58.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: cluster 2024-04-26T16:42:56.141106+0000 mgr.y (mgr.24401) 967 : cluster [DBG] pgmap v1569: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: cluster 2024-04-26T16:42:56.141106+0000 mgr.y (mgr.24401) 967 : cluster [DBG] pgmap v1569: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: audit 2024-04-26T16:42:57.080677+0000 mon.a (mon.0) 4076 : audit [DBG] from='client.? 172.21.15.50:0/1111218821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: audit 2024-04-26T16:42:57.080677+0000 mon.a (mon.0) 4076 : audit [DBG] from='client.? 172.21.15.50:0/1111218821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.210868+0000 osd.5 (osd.5) 528 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.210868+0000 osd.5 (osd.5) 528 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.213109+0000 osd.5 (osd.5) 529 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.213109+0000 osd.5 (osd.5) 529 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: audit 2024-04-26T16:42:57.470686+0000 mon.c (mon.2) 958 : audit [DBG] from='client.? 172.21.15.50:0/3342850075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:42:58.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:57 smithi190 bash[23868]: audit 2024-04-26T16:42:57.470686+0000 mon.c (mon.2) 958 : audit [DBG] from='client.? 172.21.15.50:0/3342850075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:42:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: cluster 2024-04-26T16:42:56.141106+0000 mgr.y (mgr.24401) 967 : cluster [DBG] pgmap v1569: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: cluster 2024-04-26T16:42:56.141106+0000 mgr.y (mgr.24401) 967 : cluster [DBG] pgmap v1569: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: audit 2024-04-26T16:42:57.080677+0000 mon.a (mon.0) 4076 : audit [DBG] from='client.? 172.21.15.50:0/1111218821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:42:58.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: audit 2024-04-26T16:42:57.080677+0000 mon.a (mon.0) 4076 : audit [DBG] from='client.? 172.21.15.50:0/1111218821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.210868+0000 osd.5 (osd.5) 528 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.210868+0000 osd.5 (osd.5) 528 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.213109+0000 osd.5 (osd.5) 529 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.213109+0000 osd.5 (osd.5) 529 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: audit 2024-04-26T16:42:57.470686+0000 mon.c (mon.2) 958 : audit [DBG] from='client.? 172.21.15.50:0/3342850075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[29234]: audit 2024-04-26T16:42:57.470686+0000 mon.c (mon.2) 958 : audit [DBG] from='client.? 172.21.15.50:0/3342850075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: cluster 2024-04-26T16:42:56.141106+0000 mgr.y (mgr.24401) 967 : cluster [DBG] pgmap v1569: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: cluster 2024-04-26T16:42:56.141106+0000 mgr.y (mgr.24401) 967 : cluster [DBG] pgmap v1569: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: audit 2024-04-26T16:42:57.080677+0000 mon.a (mon.0) 4076 : audit [DBG] from='client.? 172.21.15.50:0/1111218821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: audit 2024-04-26T16:42:57.080677+0000 mon.a (mon.0) 4076 : audit [DBG] from='client.? 172.21.15.50:0/1111218821' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-26T16:42:58.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.210868+0000 osd.5 (osd.5) 528 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:42:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.210868+0000 osd.5 (osd.5) 528 : cluster [DBG] 5.1b scrub starts 2024-04-26T16:42:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.213109+0000 osd.5 (osd.5) 529 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:42:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.213109+0000 osd.5 (osd.5) 529 : cluster [DBG] 5.1b scrub ok 2024-04-26T16:42:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: audit 2024-04-26T16:42:57.470686+0000 mon.c (mon.2) 958 : audit [DBG] from='client.? 172.21.15.50:0/3342850075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:42:58.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:57 smithi050 bash[21463]: audit 2024-04-26T16:42:57.470686+0000 mon.c (mon.2) 958 : audit [DBG] from='client.? 172.21.15.50:0/3342850075' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-26T16:42:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- 172.21.15.50:0/1824312808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc103e30 msgr2=0x7f6bcc110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/1824312808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc103e30 0x7f6bcc110290 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009990 tx=0x7f6bc4009ca0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- 172.21.15.50:0/1824312808 shutdown_connections 2024-04-26T16:42:58.627 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/1824312808 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc1094c0 0x7f6bcc1098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/1824312808 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc103e30 0x7f6bcc110290 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/1824312808 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- 172.21.15.50:0/1824312808 >> 172.21.15.50:0/1824312808 conn(0x7f6bcc07a580 msgr2=0x7f6bcc07a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- 172.21.15.50:0/1824312808 shutdown_connections 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- 172.21.15.50:0/1824312808 wait complete. 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 librados: starting msgr at 2024-04-26T16:42:58.628 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 librados: starting objecter 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 Processor -- start 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- start start 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 librados: setting wanted keys 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 librados: calling monclient init 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc103e30 0x7f6bcc075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:58.629 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc1094c0 0x7f6bcc0761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc112ab0 con 0x7f6bcc1094c0 2024-04-26T16:42:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f6bcc112930 con 0x7f6bcc103e30 2024-04-26T16:42:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd42eb700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f6bcc112c30 con 0x7f6bcc103470 2024-04-26T16:42:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd2087700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd2888700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc1094c0 0x7f6bcc0761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:58.630 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd2888700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc1094c0 0x7f6bcc0761a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:33962/0 (socket says 172.21.15.50:33962) 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2888700 1 -- 172.21.15.50:0/3831386855 learned_addr learned my addr 172.21.15.50:0/3831386855 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.624+0000 7f6bd2087700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc075720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:55800/0 (socket says 172.21.15.50:55800) 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd1886700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc103e30 0x7f6bcc075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2087700 1 -- 172.21.15.50:0/3831386855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc103e30 msgr2=0x7f6bcc075c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2087700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc103e30 0x7f6bcc075c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2087700 1 -- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc1094c0 msgr2=0x7f6bcc0761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2087700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc1094c0 0x7f6bcc0761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.631 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2087700 1 -- 172.21.15.50:0/3831386855 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bcc1abae0 con 0x7f6bcc103470 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd1886700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc103e30 0x7f6bcc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd2087700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc075720 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800ada0 tx=0x7f6bc800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 10 client.?.objecter ms_handle_connect 0x7f6bcc103470 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 10 client.?.objecter resend_mon_ops 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8017020 con 0x7f6bcc103470 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f6bc800cde0 con 0x7f6bcc103470 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bcc1abd70 con 0x7f6bcc103470 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8005280 con 0x7f6bcc103470 2024-04-26T16:42:58.632 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd42eb700 10 client.41990.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:42:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bcc1ac1f0 con 0x7f6bcc103470 2024-04-26T16:42:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd42eb700 1 librados: init done 2024-04-26T16:42:58.633 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bcc104b60 con 0x7f6bcc103470 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f6bc800c980 con 0x7f6bcc103470 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6bb8071f80 0x7f6bb8074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f6bc8090e90 con 0x7f6bcc103470 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 10 client.41990.objecter ms_dispatch 0x7f6bcc001550 osd_map(931..931 src has 874..931) 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 3 client.41990.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 3 client.41990.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd1886700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6bb8071f80 0x7f6bb8074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 20 client.41990.objecter dump_active .. 0 homeless 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bc37fe700 10 client.41990.objecter ms_handle_connect 0x7f6bb8071f80 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.628+0000 7f6bd1886700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6bb8071f80 0x7f6bb8074440 secure :-1 s=READY pgs=3889 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc076ee0 tx=0x7f6bbc009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:42:58.634 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.636+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f6bc8021020 con 0x7f6bcc103470 2024-04-26T16:42:58.859 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.860+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6bcc076c50 con 0x7f6bcc103470 2024-04-26T16:42:58.860 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.860+0000 7f6bc37fe700 1 -- 172.21.15.50:0/3831386855 <== mon.2 v2:172.21.15.50:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6bc805cae0 con 0x7f6bcc103470 2024-04-26T16:42:58.860 INFO:teuthology.orchestra.run.smithi050.stdout:296352743741 2024-04-26T16:42:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 10 librados: watch_flush enter 2024-04-26T16:42:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 10 librados: watch_flush exit 2024-04-26T16:42:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 20 client.41990.objecter shutdown clearing up homeless session... 2024-04-26T16:42:58.863 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 10 client.41990.objecter successfully canceled tick 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6bb8071f80 msgr2=0x7f6bb8074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6bb8071f80 0x7f6bb8074440 secure :-1 s=READY pgs=3889 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc076ee0 tx=0x7f6bbc009260 comp rx=0 tx=0).stop 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 msgr2=0x7f6bcc075720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc075720 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f6bc800ada0 tx=0x7f6bc800c330 comp rx=0 tx=0).stop 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 shutdown_connections 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f6bb8071f80 0x7f6bb8074440 unknown :-1 s=CLOSED pgs=3889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f6bcc1094c0 0x7f6bcc0761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f6bcc103470 0x7f6bcc075720 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 --2- 172.21.15.50:0/3831386855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6bcc103e30 0x7f6bcc075c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 >> 172.21.15.50:0/3831386855 conn(0x7f6bcc07a580 msgr2=0x7f6bcc10a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 shutdown_connections 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 -- 172.21.15.50:0/3831386855 wait complete. 2024-04-26T16:42:58.864 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:42:58.864+0000 7f6bd42eb700 1 librados: shutdown 2024-04-26T16:42:59.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.161191+0000 osd.1 (osd.1) 609 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.161191+0000 osd.1 (osd.1) 609 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.163719+0000 osd.1 (osd.1) 610 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.163719+0000 osd.1 (osd.1) 610 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.374468+0000 osd.7 (osd.7) 656 : cluster [DBG] 3.d scrub starts 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.374468+0000 osd.7 (osd.7) 656 : cluster [DBG] 3.d scrub starts 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.379539+0000 osd.7 (osd.7) 657 : cluster [DBG] 3.d scrub ok 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.379539+0000 osd.7 (osd.7) 657 : cluster [DBG] 3.d scrub ok 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: audit 2024-04-26T16:42:57.642097+0000 mgr.y (mgr.24401) 968 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: audit 2024-04-26T16:42:57.642097+0000 mgr.y (mgr.24401) 968 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.818557+0000 osd.4 (osd.4) 565 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.818557+0000 osd.4 (osd.4) 565 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.821573+0000 osd.4 (osd.4) 566 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: cluster 2024-04-26T16:42:57.821573+0000 osd.4 (osd.4) 566 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:42:59.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: audit 2024-04-26T16:42:58.664979+0000 mon.c (mon.2) 959 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:59.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:58 smithi190 bash[23868]: audit 2024-04-26T16:42:58.664979+0000 mon.c (mon.2) 959 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.161191+0000 osd.1 (osd.1) 609 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.161191+0000 osd.1 (osd.1) 609 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.163719+0000 osd.1 (osd.1) 610 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.163719+0000 osd.1 (osd.1) 610 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.374468+0000 osd.7 (osd.7) 656 : cluster [DBG] 3.d scrub starts 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.374468+0000 osd.7 (osd.7) 656 : cluster [DBG] 3.d scrub starts 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.379539+0000 osd.7 (osd.7) 657 : cluster [DBG] 3.d scrub ok 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.379539+0000 osd.7 (osd.7) 657 : cluster [DBG] 3.d scrub ok 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: audit 2024-04-26T16:42:57.642097+0000 mgr.y (mgr.24401) 968 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: audit 2024-04-26T16:42:57.642097+0000 mgr.y (mgr.24401) 968 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:59.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.818557+0000 osd.4 (osd.4) 565 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.818557+0000 osd.4 (osd.4) 565 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.821573+0000 osd.4 (osd.4) 566 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: cluster 2024-04-26T16:42:57.821573+0000 osd.4 (osd.4) 566 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: audit 2024-04-26T16:42:58.664979+0000 mon.c (mon.2) 959 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[21463]: audit 2024-04-26T16:42:58.664979+0000 mon.c (mon.2) 959 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.161191+0000 osd.1 (osd.1) 609 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.161191+0000 osd.1 (osd.1) 609 : cluster [DBG] 3.19 scrub starts 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.163719+0000 osd.1 (osd.1) 610 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.163719+0000 osd.1 (osd.1) 610 : cluster [DBG] 3.19 scrub ok 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.374468+0000 osd.7 (osd.7) 656 : cluster [DBG] 3.d scrub starts 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.374468+0000 osd.7 (osd.7) 656 : cluster [DBG] 3.d scrub starts 2024-04-26T16:42:59.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.379539+0000 osd.7 (osd.7) 657 : cluster [DBG] 3.d scrub ok 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.379539+0000 osd.7 (osd.7) 657 : cluster [DBG] 3.d scrub ok 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: audit 2024-04-26T16:42:57.642097+0000 mgr.y (mgr.24401) 968 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: audit 2024-04-26T16:42:57.642097+0000 mgr.y (mgr.24401) 968 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.818557+0000 osd.4 (osd.4) 565 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.818557+0000 osd.4 (osd.4) 565 : cluster [DBG] 5.e deep-scrub starts 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.821573+0000 osd.4 (osd.4) 566 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: cluster 2024-04-26T16:42:57.821573+0000 osd.4 (osd.4) 566 : cluster [DBG] 5.e deep-scrub ok 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: audit 2024-04-26T16:42:58.664979+0000 mon.c (mon.2) 959 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:42:59.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:58 smithi050 bash[29234]: audit 2024-04-26T16:42:58.664979+0000 mon.c (mon.2) 959 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.143216+0000 mgr.y (mgr.24401) 969 : cluster [DBG] pgmap v1570: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.143216+0000 mgr.y (mgr.24401) 969 : cluster [DBG] pgmap v1570: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.143232+0000 osd.1 (osd.1) 611 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.143232+0000 osd.1 (osd.1) 611 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.146338+0000 osd.1 (osd.1) 612 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.146338+0000 osd.1 (osd.1) 612 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.255247+0000 osd.0 (osd.0) 631 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.255247+0000 osd.0 (osd.0) 631 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:43:00.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.258787+0000 osd.0 (osd.0) 632 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.258787+0000 osd.0 (osd.0) 632 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.334131+0000 osd.7 (osd.7) 658 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.334131+0000 osd.7 (osd.7) 658 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.338527+0000 osd.7 (osd.7) 659 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:58.338527+0000 osd.7 (osd.7) 659 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: audit 2024-04-26T16:42:58.865709+0000 mon.c (mon.2) 960 : audit [DBG] from='client.? 172.21.15.50:0/3831386855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: audit 2024-04-26T16:42:58.865709+0000 mon.c (mon.2) 960 : audit [DBG] from='client.? 172.21.15.50:0/3831386855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.146608+0000 osd.5 (osd.5) 530 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.146608+0000 osd.5 (osd.5) 530 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.149577+0000 osd.5 (osd.5) 531 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.149577+0000 osd.5 (osd.5) 531 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.153911+0000 osd.2 (osd.2) 555 : cluster [DBG] 11.d scrub starts 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.153911+0000 osd.2 (osd.2) 555 : cluster [DBG] 11.d scrub starts 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.157853+0000 osd.2 (osd.2) 556 : cluster [DBG] 11.d scrub ok 2024-04-26T16:43:00.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:42:59 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.157853+0000 osd.2 (osd.2) 556 : cluster [DBG] 11.d scrub ok 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.143216+0000 mgr.y (mgr.24401) 969 : cluster [DBG] pgmap v1570: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.143216+0000 mgr.y (mgr.24401) 969 : cluster [DBG] pgmap v1570: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.143232+0000 osd.1 (osd.1) 611 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.143232+0000 osd.1 (osd.1) 611 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.146338+0000 osd.1 (osd.1) 612 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.146338+0000 osd.1 (osd.1) 612 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.255247+0000 osd.0 (osd.0) 631 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.255247+0000 osd.0 (osd.0) 631 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.258787+0000 osd.0 (osd.0) 632 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.258787+0000 osd.0 (osd.0) 632 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.334131+0000 osd.7 (osd.7) 658 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.334131+0000 osd.7 (osd.7) 658 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:43:00.092 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.338527+0000 osd.7 (osd.7) 659 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:58.338527+0000 osd.7 (osd.7) 659 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: audit 2024-04-26T16:42:58.865709+0000 mon.c (mon.2) 960 : audit [DBG] from='client.? 172.21.15.50:0/3831386855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: audit 2024-04-26T16:42:58.865709+0000 mon.c (mon.2) 960 : audit [DBG] from='client.? 172.21.15.50:0/3831386855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.146608+0000 osd.5 (osd.5) 530 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.146608+0000 osd.5 (osd.5) 530 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.149577+0000 osd.5 (osd.5) 531 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.149577+0000 osd.5 (osd.5) 531 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.153911+0000 osd.2 (osd.2) 555 : cluster [DBG] 11.d scrub starts 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.153911+0000 osd.2 (osd.2) 555 : cluster [DBG] 11.d scrub starts 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.157853+0000 osd.2 (osd.2) 556 : cluster [DBG] 11.d scrub ok 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.157853+0000 osd.2 (osd.2) 556 : cluster [DBG] 11.d scrub ok 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.143216+0000 mgr.y (mgr.24401) 969 : cluster [DBG] pgmap v1570: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:00.093 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.143216+0000 mgr.y (mgr.24401) 969 : cluster [DBG] pgmap v1570: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.143232+0000 osd.1 (osd.1) 611 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.143232+0000 osd.1 (osd.1) 611 : cluster [DBG] 2.15 scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.146338+0000 osd.1 (osd.1) 612 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.146338+0000 osd.1 (osd.1) 612 : cluster [DBG] 2.15 scrub ok 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.255247+0000 osd.0 (osd.0) 631 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.255247+0000 osd.0 (osd.0) 631 : cluster [DBG] 3.17 scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.258787+0000 osd.0 (osd.0) 632 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.258787+0000 osd.0 (osd.0) 632 : cluster [DBG] 3.17 scrub ok 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.334131+0000 osd.7 (osd.7) 658 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.334131+0000 osd.7 (osd.7) 658 : cluster [DBG] 3.11 scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.338527+0000 osd.7 (osd.7) 659 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:58.338527+0000 osd.7 (osd.7) 659 : cluster [DBG] 3.11 scrub ok 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: audit 2024-04-26T16:42:58.865709+0000 mon.c (mon.2) 960 : audit [DBG] from='client.? 172.21.15.50:0/3831386855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: audit 2024-04-26T16:42:58.865709+0000 mon.c (mon.2) 960 : audit [DBG] from='client.? 172.21.15.50:0/3831386855' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.146608+0000 osd.5 (osd.5) 530 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:43:00.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.146608+0000 osd.5 (osd.5) 530 : cluster [DBG] 11.1a scrub starts 2024-04-26T16:43:00.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.149577+0000 osd.5 (osd.5) 531 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:43:00.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.149577+0000 osd.5 (osd.5) 531 : cluster [DBG] 11.1a scrub ok 2024-04-26T16:43:00.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.153911+0000 osd.2 (osd.2) 555 : cluster [DBG] 11.d scrub starts 2024-04-26T16:43:00.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.153911+0000 osd.2 (osd.2) 555 : cluster [DBG] 11.d scrub starts 2024-04-26T16:43:00.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.157853+0000 osd.2 (osd.2) 556 : cluster [DBG] 11.d scrub ok 2024-04-26T16:43:00.095 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:42:59 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.157853+0000 osd.2 (osd.2) 556 : cluster [DBG] 11.d scrub ok 2024-04-26T16:43:00.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855701 got 150323855706 for osd.4 2024-04-26T16:43:00.096 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:00.606 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463243 got 201863463248 for osd.5 2024-04-26T16:43:00.606 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:00.761 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738750 got 34359738756 for osd.0 2024-04-26T16:43:00.762 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:00.879 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.351607+0000 osd.7 (osd.7) 660 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.351607+0000 osd.7 (osd.7) 660 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.354428+0000 osd.7 (osd.7) 661 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.354428+0000 osd.7 (osd.7) 661 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.578573+0000 osd.3 (osd.3) 677 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.578573+0000 osd.3 (osd.3) 677 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.581899+0000 osd.3 (osd.3) 678 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:42:59.581899+0000 osd.3 (osd.3) 678 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.102426+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.102426+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.106266+0000 osd.5 (osd.5) 533 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.106266+0000 osd.5 (osd.5) 533 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:43:00.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.351607+0000 osd.7 (osd.7) 660 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.351607+0000 osd.7 (osd.7) 660 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.354428+0000 osd.7 (osd.7) 661 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.354428+0000 osd.7 (osd.7) 661 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.578573+0000 osd.3 (osd.3) 677 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.578573+0000 osd.3 (osd.3) 677 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.581899+0000 osd.3 (osd.3) 678 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:42:59.581899+0000 osd.3 (osd.3) 678 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.102426+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.102426+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.106266+0000 osd.5 (osd.5) 533 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:43:00.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:00 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.106266+0000 osd.5 (osd.5) 533 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:43:01.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.351607+0000 osd.7 (osd.7) 660 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.351607+0000 osd.7 (osd.7) 660 : cluster [DBG] 11.5 deep-scrub starts 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.354428+0000 osd.7 (osd.7) 661 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.354428+0000 osd.7 (osd.7) 661 : cluster [DBG] 11.5 deep-scrub ok 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.578573+0000 osd.3 (osd.3) 677 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.578573+0000 osd.3 (osd.3) 677 : cluster [DBG] 4.0 scrub starts 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.581899+0000 osd.3 (osd.3) 678 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:42:59.581899+0000 osd.3 (osd.3) 678 : cluster [DBG] 4.0 scrub ok 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.102426+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.102426+0000 osd.5 (osd.5) 532 : cluster [DBG] 6.19 scrub starts 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.106266+0000 osd.5 (osd.5) 533 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:43:01.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:00 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.106266+0000 osd.5 (osd.5) 533 : cluster [DBG] 6.19 scrub ok 2024-04-26T16:43:02.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.144295+0000 mgr.y (mgr.24401) 970 : cluster [DBG] pgmap v1571: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:02.062 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.144295+0000 mgr.y (mgr.24401) 970 : cluster [DBG] pgmap v1571: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.582793+0000 osd.3 (osd.3) 679 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.582793+0000 osd.3 (osd.3) 679 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.585085+0000 osd.3 (osd.3) 680 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:00.585085+0000 osd.3 (osd.3) 680 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.099534+0000 osd.5 (osd.5) 534 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.099534+0000 osd.5 (osd.5) 534 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.102070+0000 osd.5 (osd.5) 535 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.102070+0000 osd.5 (osd.5) 535 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.144295+0000 mgr.y (mgr.24401) 970 : cluster [DBG] pgmap v1571: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.144295+0000 mgr.y (mgr.24401) 970 : cluster [DBG] pgmap v1571: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.582793+0000 osd.3 (osd.3) 679 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.582793+0000 osd.3 (osd.3) 679 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:43:02.063 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.585085+0000 osd.3 (osd.3) 680 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:43:02.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:00.585085+0000 osd.3 (osd.3) 680 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:43:02.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.099534+0000 osd.5 (osd.5) 534 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:43:02.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.099534+0000 osd.5 (osd.5) 534 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:43:02.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.102070+0000 osd.5 (osd.5) 535 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:43:02.064 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:01 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.102070+0000 osd.5 (osd.5) 535 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:43:02.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 3204045602902 got 3204045602908 for osd.6 2024-04-26T16:43:02.071 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:02.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.144295+0000 mgr.y (mgr.24401) 970 : cluster [DBG] pgmap v1571: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:02.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.144295+0000 mgr.y (mgr.24401) 970 : cluster [DBG] pgmap v1571: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:02.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.582793+0000 osd.3 (osd.3) 679 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:43:02.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.582793+0000 osd.3 (osd.3) 679 : cluster [DBG] 5.13 scrub starts 2024-04-26T16:43:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.585085+0000 osd.3 (osd.3) 680 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:43:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:00.585085+0000 osd.3 (osd.3) 680 : cluster [DBG] 5.13 scrub ok 2024-04-26T16:43:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.099534+0000 osd.5 (osd.5) 534 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:43:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.099534+0000 osd.5 (osd.5) 534 : cluster [DBG] 5.16 scrub starts 2024-04-26T16:43:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.102070+0000 osd.5 (osd.5) 535 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:43:02.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:01 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.102070+0000 osd.5 (osd.5) 535 : cluster [DBG] 5.16 scrub ok 2024-04-26T16:43:02.628 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575221 got 55834575226 for osd.1 2024-04-26T16:43:02.629 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:02.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084640 got 120259084644 for osd.3 2024-04-26T16:43:02.632 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.179407+0000 osd.6 (osd.6) 165 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.179407+0000 osd.6 (osd.6) 165 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.181717+0000 osd.6 (osd.6) 166 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.181717+0000 osd.6 (osd.6) 166 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.633836+0000 osd.3 (osd.3) 681 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.633836+0000 osd.3 (osd.3) 681 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.638352+0000 osd.3 (osd.3) 682 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:01.638352+0000 osd.3 (osd.3) 682 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:43:02.880 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.083935+0000 osd.5 (osd.5) 536 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.083935+0000 osd.5 (osd.5) 536 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.086552+0000 osd.5 (osd.5) 537 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.086552+0000 osd.5 (osd.5) 537 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.179407+0000 osd.6 (osd.6) 165 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.179407+0000 osd.6 (osd.6) 165 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.181717+0000 osd.6 (osd.6) 166 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.181717+0000 osd.6 (osd.6) 166 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.633836+0000 osd.3 (osd.3) 681 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.633836+0000 osd.3 (osd.3) 681 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.638352+0000 osd.3 (osd.3) 682 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:01.638352+0000 osd.3 (osd.3) 682 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.083935+0000 osd.5 (osd.5) 536 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.083935+0000 osd.5 (osd.5) 536 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:43:02.881 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.086552+0000 osd.5 (osd.5) 537 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:43:02.882 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:02 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.086552+0000 osd.5 (osd.5) 537 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:43:02.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743736 got 296352743741 for osd.7 2024-04-26T16:43:02.936 DEBUG:teuthology.parallel:result is None 2024-04-26T16:43:02.936 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-04-26T16:43:02.936 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:43:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.179407+0000 osd.6 (osd.6) 165 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:43:03.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.179407+0000 osd.6 (osd.6) 165 : cluster [DBG] 11.1c scrub starts 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.181717+0000 osd.6 (osd.6) 166 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.181717+0000 osd.6 (osd.6) 166 : cluster [DBG] 11.1c scrub ok 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.633836+0000 osd.3 (osd.3) 681 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.633836+0000 osd.3 (osd.3) 681 : cluster [DBG] 11.9 scrub starts 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.638352+0000 osd.3 (osd.3) 682 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:01.638352+0000 osd.3 (osd.3) 682 : cluster [DBG] 11.9 scrub ok 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.083935+0000 osd.5 (osd.5) 536 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.083935+0000 osd.5 (osd.5) 536 : cluster [DBG] 1.1 deep-scrub starts 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.086552+0000 osd.5 (osd.5) 537 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:43:03.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:02 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.086552+0000 osd.5 (osd.5) 537 : cluster [DBG] 1.1 deep-scrub ok 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.004159+0000 osd.1 (osd.1) 613 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.004159+0000 osd.1 (osd.1) 613 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.005974+0000 osd.1 (osd.1) 614 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.005974+0000 osd.1 (osd.1) 614 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.146281+0000 mgr.y (mgr.24401) 971 : cluster [DBG] pgmap v1572: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.146281+0000 mgr.y (mgr.24401) 971 : cluster [DBG] pgmap v1572: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.336433+0000 osd.0 (osd.0) 633 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.336433+0000 osd.0 (osd.0) 633 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.338985+0000 osd.0 (osd.0) 634 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.338985+0000 osd.0 (osd.0) 634 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.397536+0000 osd.7 (osd.7) 662 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:43:04.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.397536+0000 osd.7 (osd.7) 662 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:43:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.400055+0000 osd.7 (osd.7) 663 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:43:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:02.400055+0000 osd.7 (osd.7) 663 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:43:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.200547+0000 osd.2 (osd.2) 557 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:43:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.200547+0000 osd.2 (osd.2) 557 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:43:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.201859+0000 osd.2 (osd.2) 558 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:43:04.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:03 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.201859+0000 osd.2 (osd.2) 558 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:43:04.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.004159+0000 osd.1 (osd.1) 613 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:43:04.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.004159+0000 osd.1 (osd.1) 613 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:43:04.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.005974+0000 osd.1 (osd.1) 614 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.005974+0000 osd.1 (osd.1) 614 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.146281+0000 mgr.y (mgr.24401) 971 : cluster [DBG] pgmap v1572: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.146281+0000 mgr.y (mgr.24401) 971 : cluster [DBG] pgmap v1572: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.336433+0000 osd.0 (osd.0) 633 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.336433+0000 osd.0 (osd.0) 633 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.338985+0000 osd.0 (osd.0) 634 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.338985+0000 osd.0 (osd.0) 634 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.397536+0000 osd.7 (osd.7) 662 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.397536+0000 osd.7 (osd.7) 662 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.400055+0000 osd.7 (osd.7) 663 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:02.400055+0000 osd.7 (osd.7) 663 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.200547+0000 osd.2 (osd.2) 557 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.200547+0000 osd.2 (osd.2) 557 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.201859+0000 osd.2 (osd.2) 558 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:43:04.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.201859+0000 osd.2 (osd.2) 558 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.004159+0000 osd.1 (osd.1) 613 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.004159+0000 osd.1 (osd.1) 613 : cluster [DBG] 6.4 scrub starts 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.005974+0000 osd.1 (osd.1) 614 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.005974+0000 osd.1 (osd.1) 614 : cluster [DBG] 6.4 scrub ok 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.146281+0000 mgr.y (mgr.24401) 971 : cluster [DBG] pgmap v1572: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.146281+0000 mgr.y (mgr.24401) 971 : cluster [DBG] pgmap v1572: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.336433+0000 osd.0 (osd.0) 633 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.336433+0000 osd.0 (osd.0) 633 : cluster [DBG] 2.13 scrub starts 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.338985+0000 osd.0 (osd.0) 634 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.338985+0000 osd.0 (osd.0) 634 : cluster [DBG] 2.13 scrub ok 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.397536+0000 osd.7 (osd.7) 662 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:43:04.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.397536+0000 osd.7 (osd.7) 662 : cluster [DBG] 1.9 deep-scrub starts 2024-04-26T16:43:04.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.400055+0000 osd.7 (osd.7) 663 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:43:04.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:02.400055+0000 osd.7 (osd.7) 663 : cluster [DBG] 1.9 deep-scrub ok 2024-04-26T16:43:04.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.200547+0000 osd.2 (osd.2) 557 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:43:04.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.200547+0000 osd.2 (osd.2) 557 : cluster [DBG] 2.10 scrub starts 2024-04-26T16:43:04.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.201859+0000 osd.2 (osd.2) 558 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:43:04.133 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:03 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.201859+0000 osd.2 (osd.2) 558 : cluster [DBG] 2.10 scrub ok 2024-04-26T16:43:04.813 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:43:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:43:05.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.291537+0000 osd.0 (osd.0) 635 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:43:05.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.291537+0000 osd.0 (osd.0) 635 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:43:05.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.293442+0000 osd.0 (osd.0) 636 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:43:05.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.293442+0000 osd.0 (osd.0) 636 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:43:05.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.877479+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:43:05.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.877479+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:43:05.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.881970+0000 osd.4 (osd.4) 568 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:43:05.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:04 smithi190 bash[23868]: cluster 2024-04-26T16:43:03.881970+0000 osd.4 (osd.4) 568 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.291537+0000 osd.0 (osd.0) 635 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.291537+0000 osd.0 (osd.0) 635 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.293442+0000 osd.0 (osd.0) 636 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.293442+0000 osd.0 (osd.0) 636 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.877479+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.877479+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.881970+0000 osd.4 (osd.4) 568 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[21463]: cluster 2024-04-26T16:43:03.881970+0000 osd.4 (osd.4) 568 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:43:05.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.291537+0000 osd.0 (osd.0) 635 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.291537+0000 osd.0 (osd.0) 635 : cluster [DBG] 6.0 scrub starts 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.293442+0000 osd.0 (osd.0) 636 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.293442+0000 osd.0 (osd.0) 636 : cluster [DBG] 6.0 scrub ok 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.877479+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.877479+0000 osd.4 (osd.4) 567 : cluster [DBG] 3.14 scrub starts 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.881970+0000 osd.4 (osd.4) 568 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:43:05.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:04 smithi050 bash[29234]: cluster 2024-04-26T16:43:03.881970+0000 osd.4 (osd.4) 568 : cluster [DBG] 3.14 scrub ok 2024-04-26T16:43:06.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:05 smithi050 bash[21463]: cluster 2024-04-26T16:43:04.147401+0000 mgr.y (mgr.24401) 972 : cluster [DBG] pgmap v1573: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:06.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:05 smithi050 bash[21463]: cluster 2024-04-26T16:43:04.147401+0000 mgr.y (mgr.24401) 972 : cluster [DBG] pgmap v1573: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:06.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:05 smithi050 bash[29234]: cluster 2024-04-26T16:43:04.147401+0000 mgr.y (mgr.24401) 972 : cluster [DBG] pgmap v1573: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:06.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:05 smithi050 bash[29234]: cluster 2024-04-26T16:43:04.147401+0000 mgr.y (mgr.24401) 972 : cluster [DBG] pgmap v1573: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:05 smithi190 bash[23868]: cluster 2024-04-26T16:43:04.147401+0000 mgr.y (mgr.24401) 972 : cluster [DBG] pgmap v1573: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:06.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:05 smithi190 bash[23868]: cluster 2024-04-26T16:43:04.147401+0000 mgr.y (mgr.24401) 972 : cluster [DBG] pgmap v1573: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:07.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.295716+0000 osd.0 (osd.0) 637 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.295716+0000 osd.0 (osd.0) 637 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.299255+0000 osd.0 (osd.0) 638 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.299255+0000 osd.0 (osd.0) 638 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.295716+0000 osd.0 (osd.0) 637 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.295716+0000 osd.0 (osd.0) 637 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.299255+0000 osd.0 (osd.0) 638 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:43:07.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:06 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.299255+0000 osd.0 (osd.0) 638 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:43:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:06 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.295716+0000 osd.0 (osd.0) 637 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:43:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:06 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.295716+0000 osd.0 (osd.0) 637 : cluster [DBG] 5.3 scrub starts 2024-04-26T16:43:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:06 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.299255+0000 osd.0 (osd.0) 638 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:43:07.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:06 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.299255+0000 osd.0 (osd.0) 638 : cluster [DBG] 5.3 scrub ok 2024-04-26T16:43:07.731 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:43:08.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.941100+0000 osd.1 (osd.1) 615 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.941100+0000 osd.1 (osd.1) 615 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.944021+0000 osd.1 (osd.1) 616 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:05.944021+0000 osd.1 (osd.1) 616 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:06.148233+0000 mgr.y (mgr.24401) 973 : cluster [DBG] pgmap v1574: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:06.148233+0000 mgr.y (mgr.24401) 973 : cluster [DBG] pgmap v1574: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:06.209757+0000 osd.6 (osd.6) 167 : cluster [DBG] 3.a scrub starts 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:06.209757+0000 osd.6 (osd.6) 167 : cluster [DBG] 3.a scrub starts 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:06.214925+0000 osd.6 (osd.6) 168 : cluster [DBG] 3.a scrub ok 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:06.214925+0000 osd.6 (osd.6) 168 : cluster [DBG] 3.a scrub ok 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.163365+0000 osd.2 (osd.2) 559 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.163365+0000 osd.2 (osd.2) 559 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:43:08.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.165763+0000 osd.2 (osd.2) 560 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:43:08.076 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:07 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.165763+0000 osd.2 (osd.2) 560 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.941100+0000 osd.1 (osd.1) 615 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.941100+0000 osd.1 (osd.1) 615 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.944021+0000 osd.1 (osd.1) 616 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:05.944021+0000 osd.1 (osd.1) 616 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:06.148233+0000 mgr.y (mgr.24401) 973 : cluster [DBG] pgmap v1574: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:06.148233+0000 mgr.y (mgr.24401) 973 : cluster [DBG] pgmap v1574: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:06.209757+0000 osd.6 (osd.6) 167 : cluster [DBG] 3.a scrub starts 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:06.209757+0000 osd.6 (osd.6) 167 : cluster [DBG] 3.a scrub starts 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:06.214925+0000 osd.6 (osd.6) 168 : cluster [DBG] 3.a scrub ok 2024-04-26T16:43:08.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:06.214925+0000 osd.6 (osd.6) 168 : cluster [DBG] 3.a scrub ok 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.163365+0000 osd.2 (osd.2) 559 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.163365+0000 osd.2 (osd.2) 559 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.165763+0000 osd.2 (osd.2) 560 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.165763+0000 osd.2 (osd.2) 560 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.941100+0000 osd.1 (osd.1) 615 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.941100+0000 osd.1 (osd.1) 615 : cluster [DBG] 5.1d deep-scrub starts 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.944021+0000 osd.1 (osd.1) 616 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:05.944021+0000 osd.1 (osd.1) 616 : cluster [DBG] 5.1d deep-scrub ok 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:06.148233+0000 mgr.y (mgr.24401) 973 : cluster [DBG] pgmap v1574: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:06.148233+0000 mgr.y (mgr.24401) 973 : cluster [DBG] pgmap v1574: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:06.209757+0000 osd.6 (osd.6) 167 : cluster [DBG] 3.a scrub starts 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:06.209757+0000 osd.6 (osd.6) 167 : cluster [DBG] 3.a scrub starts 2024-04-26T16:43:08.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:06.214925+0000 osd.6 (osd.6) 168 : cluster [DBG] 3.a scrub ok 2024-04-26T16:43:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:06.214925+0000 osd.6 (osd.6) 168 : cluster [DBG] 3.a scrub ok 2024-04-26T16:43:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.163365+0000 osd.2 (osd.2) 559 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:43:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.163365+0000 osd.2 (osd.2) 559 : cluster [DBG] 6.14 scrub starts 2024-04-26T16:43:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.165763+0000 osd.2 (osd.2) 560 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:43:08.132 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:07 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.165763+0000 osd.2 (osd.2) 560 : cluster [DBG] 6.14 scrub ok 2024-04-26T16:43:09.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.432869+0000 osd.7 (osd.7) 664 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:43:09.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.432869+0000 osd.7 (osd.7) 664 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:43:09.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.436782+0000 osd.7 (osd.7) 665 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:43:09.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.436782+0000 osd.7 (osd.7) 665 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:43:09.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: audit 2024-04-26T16:43:07.652607+0000 mgr.y (mgr.24401) 974 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:09.097 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: audit 2024-04-26T16:43:07.652607+0000 mgr.y (mgr.24401) 974 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.697590+0000 osd.3 (osd.3) 683 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.697590+0000 osd.3 (osd.3) 683 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.700235+0000 osd.3 (osd.3) 684 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.700235+0000 osd.3 (osd.3) 684 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.952314+0000 osd.4 (osd.4) 569 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.952314+0000 osd.4 (osd.4) 569 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.954902+0000 osd.4 (osd.4) 570 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:07.954902+0000 osd.4 (osd.4) 570 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.090055+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.090055+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.092436+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:43:09.098 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.092436+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.432869+0000 osd.7 (osd.7) 664 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.432869+0000 osd.7 (osd.7) 664 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.436782+0000 osd.7 (osd.7) 665 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.436782+0000 osd.7 (osd.7) 665 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: audit 2024-04-26T16:43:07.652607+0000 mgr.y (mgr.24401) 974 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: audit 2024-04-26T16:43:07.652607+0000 mgr.y (mgr.24401) 974 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.697590+0000 osd.3 (osd.3) 683 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.697590+0000 osd.3 (osd.3) 683 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.700235+0000 osd.3 (osd.3) 684 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.700235+0000 osd.3 (osd.3) 684 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.952314+0000 osd.4 (osd.4) 569 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.952314+0000 osd.4 (osd.4) 569 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:43:09.099 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.954902+0000 osd.4 (osd.4) 570 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:43:09.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:07.954902+0000 osd.4 (osd.4) 570 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:43:09.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.090055+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:43:09.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.090055+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:43:09.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.092436+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:43:09.100 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:08 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.092436+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.432869+0000 osd.7 (osd.7) 664 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.432869+0000 osd.7 (osd.7) 664 : cluster [DBG] 11.1d scrub starts 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.436782+0000 osd.7 (osd.7) 665 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.436782+0000 osd.7 (osd.7) 665 : cluster [DBG] 11.1d scrub ok 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: audit 2024-04-26T16:43:07.652607+0000 mgr.y (mgr.24401) 974 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: audit 2024-04-26T16:43:07.652607+0000 mgr.y (mgr.24401) 974 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.697590+0000 osd.3 (osd.3) 683 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.697590+0000 osd.3 (osd.3) 683 : cluster [DBG] 6.13 scrub starts 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.700235+0000 osd.3 (osd.3) 684 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.700235+0000 osd.3 (osd.3) 684 : cluster [DBG] 6.13 scrub ok 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.952314+0000 osd.4 (osd.4) 569 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.952314+0000 osd.4 (osd.4) 569 : cluster [DBG] 4.1 deep-scrub starts 2024-04-26T16:43:09.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.954902+0000 osd.4 (osd.4) 570 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:43:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:07.954902+0000 osd.4 (osd.4) 570 : cluster [DBG] 4.1 deep-scrub ok 2024-04-26T16:43:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.090055+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:43:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.090055+0000 osd.5 (osd.5) 538 : cluster [DBG] 2.18 scrub starts 2024-04-26T16:43:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.092436+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:43:09.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:08 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.092436+0000 osd.5 (osd.5) 539 : cluster [DBG] 2.18 scrub ok 2024-04-26T16:43:10.110 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.150104+0000 mgr.y (mgr.24401) 975 : cluster [DBG] pgmap v1575: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.150104+0000 mgr.y (mgr.24401) 975 : cluster [DBG] pgmap v1575: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.474719+0000 osd.7 (osd.7) 666 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.474719+0000 osd.7 (osd.7) 666 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.477042+0000 osd.7 (osd.7) 667 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.477042+0000 osd.7 (osd.7) 667 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.939842+0000 osd.1 (osd.1) 617 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.939842+0000 osd.1 (osd.1) 617 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.942380+0000 osd.1 (osd.1) 618 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[29234]: cluster 2024-04-26T16:43:08.942380+0000 osd.1 (osd.1) 618 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:43:10.111 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.150104+0000 mgr.y (mgr.24401) 975 : cluster [DBG] pgmap v1575: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.150104+0000 mgr.y (mgr.24401) 975 : cluster [DBG] pgmap v1575: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.474719+0000 osd.7 (osd.7) 666 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.474719+0000 osd.7 (osd.7) 666 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.477042+0000 osd.7 (osd.7) 667 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.477042+0000 osd.7 (osd.7) 667 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.939842+0000 osd.1 (osd.1) 617 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.939842+0000 osd.1 (osd.1) 617 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.942380+0000 osd.1 (osd.1) 618 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:43:10.112 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:09 smithi050 bash[21463]: cluster 2024-04-26T16:43:08.942380+0000 osd.1 (osd.1) 618 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:43:10.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.150104+0000 mgr.y (mgr.24401) 975 : cluster [DBG] pgmap v1575: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:10.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.150104+0000 mgr.y (mgr.24401) 975 : cluster [DBG] pgmap v1575: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.474719+0000 osd.7 (osd.7) 666 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.474719+0000 osd.7 (osd.7) 666 : cluster [DBG] 1.2 scrub starts 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.477042+0000 osd.7 (osd.7) 667 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.477042+0000 osd.7 (osd.7) 667 : cluster [DBG] 1.2 scrub ok 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.939842+0000 osd.1 (osd.1) 617 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.939842+0000 osd.1 (osd.1) 617 : cluster [DBG] 2.4 scrub starts 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.942380+0000 osd.1 (osd.1) 618 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:43:10.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:09 smithi190 bash[23868]: cluster 2024-04-26T16:43:08.942380+0000 osd.1 (osd.1) 618 : cluster [DBG] 2.4 scrub ok 2024-04-26T16:43:11.129 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.211733+0000 osd.6 (osd.6) 169 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.211733+0000 osd.6 (osd.6) 169 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.215130+0000 osd.6 (osd.6) 170 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.215130+0000 osd.6 (osd.6) 170 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.519155+0000 osd.7 (osd.7) 668 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.519155+0000 osd.7 (osd.7) 668 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.523483+0000 osd.7 (osd.7) 669 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.523483+0000 osd.7 (osd.7) 669 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.693971+0000 osd.3 (osd.3) 685 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.693971+0000 osd.3 (osd.3) 685 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.698234+0000 osd.3 (osd.3) 686 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:43:11.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[21463]: cluster 2024-04-26T16:43:09.698234+0000 osd.3 (osd.3) 686 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.211733+0000 osd.6 (osd.6) 169 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.211733+0000 osd.6 (osd.6) 169 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.215130+0000 osd.6 (osd.6) 170 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.215130+0000 osd.6 (osd.6) 170 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.519155+0000 osd.7 (osd.7) 668 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.519155+0000 osd.7 (osd.7) 668 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.523483+0000 osd.7 (osd.7) 669 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.523483+0000 osd.7 (osd.7) 669 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.693971+0000 osd.3 (osd.3) 685 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.693971+0000 osd.3 (osd.3) 685 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.698234+0000 osd.3 (osd.3) 686 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:43:11.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:10 smithi050 bash[29234]: cluster 2024-04-26T16:43:09.698234+0000 osd.3 (osd.3) 686 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:43:11.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.211733+0000 osd.6 (osd.6) 169 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.211733+0000 osd.6 (osd.6) 169 : cluster [DBG] 3.10 scrub starts 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.215130+0000 osd.6 (osd.6) 170 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.215130+0000 osd.6 (osd.6) 170 : cluster [DBG] 3.10 scrub ok 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.519155+0000 osd.7 (osd.7) 668 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.519155+0000 osd.7 (osd.7) 668 : cluster [DBG] 1.0 scrub starts 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.523483+0000 osd.7 (osd.7) 669 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.523483+0000 osd.7 (osd.7) 669 : cluster [DBG] 1.0 scrub ok 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.693971+0000 osd.3 (osd.3) 685 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.693971+0000 osd.3 (osd.3) 685 : cluster [DBG] 3.2 scrub starts 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.698234+0000 osd.3 (osd.3) 686 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:43:11.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:10 smithi190 bash[23868]: cluster 2024-04-26T16:43:09.698234+0000 osd.3 (osd.3) 686 : cluster [DBG] 3.2 scrub ok 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/3690292995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 msgr2=0x7f8ba81038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- 172.21.15.50:0/3690292995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba81038f0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009990 tx=0x7f8b98009ca0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/3690292995 shutdown_connections 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- 172.21.15.50:0/3690292995 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba81094c0 0x7f8ba81098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- 172.21.15.50:0/3690292995 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba8103e30 0x7f8ba8110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- 172.21.15.50:0/3690292995 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba81038f0 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/3690292995 >> 172.21.15.50:0/3690292995 conn(0x7f8ba807a580 msgr2=0x7f8ba807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:43:11.415 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/3690292995 shutdown_connections 2024-04-26T16:43:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/3690292995 wait complete. 2024-04-26T16:43:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 librados: starting msgr at 2024-04-26T16:43:11.416 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 librados: starting objecter 2024-04-26T16:43:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 Processor -- start 2024-04-26T16:43:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- start start 2024-04-26T16:43:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 librados: setting wanted keys 2024-04-26T16:43:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 librados: calling monclient init 2024-04-26T16:43:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba8199150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:11.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba8103e30 0x7f8ba8199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba81094c0 0x7f8ba819da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba8112af0 con 0x7f8ba8103e30 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba8112970 con 0x7f8ba81094c0 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba8112c70 con 0x7f8ba8103470 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba8199150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bac832700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba8103e30 0x7f8ba8199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba8199150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:56084/0 (socket says 172.21.15.50:56084) 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 -- 172.21.15.50:0/936746466 learned_addr learned my addr 172.21.15.50:0/936746466 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad834700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba81094c0 0x7f8ba819da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 -- 172.21.15.50:0/936746466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba81094c0 msgr2=0x7f8ba819da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba81094c0 0x7f8ba819da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 -- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba8103e30 msgr2=0x7f8ba8199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:11.418 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba8103e30 0x7f8ba8199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 -- 172.21.15.50:0/936746466 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba819e160 con 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8bad033700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba8199150 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400c9b0 tx=0x7f8ba400ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8b9e7fc700 10 client.?.objecter ms_handle_connect 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8b9e7fc700 10 client.?.objecter resend_mon_ops 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mon.2 v2:172.21.15.50:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4007b00 con 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mon.2 v2:172.21.15.50:3301/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f8ba4007ca0 con 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba819e450 con 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mon.2 v2:172.21.15.50:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4018400 con 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 10 client.41996.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba81a5cd0 con 0x7f8ba8103470 2024-04-26T16:43:11.419 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.416+0000 7f8baf297700 1 librados: init done 2024-04-26T16:43:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba806a110 con 0x7f8ba8103470 2024-04-26T16:43:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mon.2 v2:172.21.15.50:3301/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f8ba400f450 con 0x7f8ba8103470 2024-04-26T16:43:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8b94071f80 0x7f8b94074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mon.2 v2:172.21.15.50:3301/0 5 ==== osd_map(931..931 src has 874..931) ==== 8707+0+0 (secure 0 0 0) 0x7f8ba4091700 con 0x7f8ba8103470 2024-04-26T16:43:11.420 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 10 client.41996.objecter ms_dispatch 0x7f8ba8001550 osd_map(931..931 src has 874..931) 2024-04-26T16:43:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 3 client.41996.objecter handle_osd_map got epochs [931,931] > 0 2024-04-26T16:43:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 3 client.41996.objecter handle_osd_map decoding full epoch 931 2024-04-26T16:43:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8bac832700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8b94071f80 0x7f8b94074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 20 client.41996.objecter dump_active .. 0 homeless 2024-04-26T16:43:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8b9e7fc700 10 client.41996.objecter ms_handle_connect 0x7f8b94071f80 2024-04-26T16:43:11.421 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.420+0000 7f8bac832700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8b94071f80 0x7f8b94074440 secure :-1 s=READY pgs=3890 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800b730 tx=0x7f8b98005fd0 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:43:11.424 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.424+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mon.2 v2:172.21.15.50:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f8ba4014030 con 0x7f8ba8103470 2024-04-26T16:43:11.645 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.644+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ba81022d0 con 0x7f8b94071f80 2024-04-26T16:43:11.676 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.676+0000 7f8b9e7fc700 1 -- 172.21.15.50:0/936746466 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+453793 (secure 0 0 0) 0x7f8b98096cc0 con 0x7f8b94071f80 2024-04-26T16:43:11.680 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:43:11.684 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.680+0000 7f8baf297700 10 librados: watch_flush enter 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.680+0000 7f8baf297700 10 librados: watch_flush exit 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.680+0000 7f8baf297700 20 client.41996.objecter shutdown clearing up homeless session... 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.680+0000 7f8baf297700 10 client.41996.objecter successfully canceled tick 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8b94071f80 msgr2=0x7f8b94074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8b94071f80 0x7f8b94074440 secure :-1 s=READY pgs=3890 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800b730 tx=0x7f8b98005fd0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 msgr2=0x7f8ba8199150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba8199150 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400c9b0 tx=0x7f8ba400ccc0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 shutdown_connections 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f8b94071f80 0x7f8b94074440 unknown :-1 s=CLOSED pgs=3890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ba81094c0 0x7f8ba819da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f8ba8103470 0x7f8ba8199150 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 --2- 172.21.15.50:0/936746466 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f8ba8103e30 0x7f8ba8199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:11.685 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 >> 172.21.15.50:0/936746466 conn(0x7f8ba807a580 msgr2=0x7f8ba8101bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:43:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 shutdown_connections 2024-04-26T16:43:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 -- 172.21.15.50:0/936746466 wait complete. 2024-04-26T16:43:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:11.684+0000 7f8baf297700 1 librados: shutdown 2024-04-26T16:43:11.686 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:43:11.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:11 smithi050 bash[21463]: cluster 2024-04-26T16:43:10.151009+0000 mgr.y (mgr.24401) 976 : cluster [DBG] pgmap v1576: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:11.971 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:11 smithi050 bash[21463]: cluster 2024-04-26T16:43:10.151009+0000 mgr.y (mgr.24401) 976 : cluster [DBG] pgmap v1576: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:11.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:11 smithi050 bash[29234]: cluster 2024-04-26T16:43:10.151009+0000 mgr.y (mgr.24401) 976 : cluster [DBG] pgmap v1576: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:11.972 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:11 smithi050 bash[29234]: cluster 2024-04-26T16:43:10.151009+0000 mgr.y (mgr.24401) 976 : cluster [DBG] pgmap v1576: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:12.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:11 smithi190 bash[23868]: cluster 2024-04-26T16:43:10.151009+0000 mgr.y (mgr.24401) 976 : cluster [DBG] pgmap v1576: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:12.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:11 smithi190 bash[23868]: cluster 2024-04-26T16:43:10.151009+0000 mgr.y (mgr.24401) 976 : cluster [DBG] pgmap v1576: 174 pgs: 174 active+clean; 583 KiB data, 788 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:12.542 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2024-04-26T16:43:10.150491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596512,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3105,"num_read_kb":2857,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":3434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2703,"ondisk_log_size":2703,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":806632,"kb_used_data":20584,"kb_used_omap":34,"kb_used_meta":785949,"kb_avail":749154584,"statfs":{"total":767960285184,"available":767134294016,"internally_reserved":0,"allocated":21078016,"data_stored":7103168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34917,"internal_metadata":804812699},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009115"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":741,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.455322+0000","last_change":"2024-04-26T16:42:21.953084+0000","last_active":"2024-04-26T16:42:38.455322+0000","last_peered":"2024-04-26T16:42:38.455322+0000","last_clean":"2024-04-26T16:42:38.455322+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:42:38.455322+0000","last_undegraded":"2024-04-26T16:42:38.455322+0000","last_fullsized":"2024-04-26T16:42:38.455322+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:21.952854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:44.992566+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.952854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:34.118612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"747'16","reported_seq":2118,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272989+0000","last_change":"2024-04-26T16:41:39.323840+0000","last_active":"2024-04-26T16:43:00.272989+0000","last_peered":"2024-04-26T16:43:00.272989+0000","last_clean":"2024-04-26T16:43:00.272989+0000","last_became_active":"2024-04-26T16:35:25.395438+0000","last_became_peered":"2024-04-26T16:35:25.395438+0000","last_unstale":"2024-04-26T16:43:00.272989+0000","last_undegraded":"2024-04-26T16:43:00.272989+0000","last_fullsized":"2024-04-26T16:43:00.272989+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:41:39.323617+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:41:39.323617+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:08.147518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"857'46","reported_seq":1887,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.452650+0000","last_change":"2024-04-26T16:42:34.355089+0000","last_active":"2024-04-26T16:42:38.452650+0000","last_peered":"2024-04-26T16:42:38.452650+0000","last_clean":"2024-04-26T16:42:38.452650+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:42:38.452650+0000","last_undegraded":"2024-04-26T16:42:38.452650+0000","last_fullsized":"2024-04-26T16:42:38.452650+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:42:34.354814+0000","last_deep_scrub":"857'46","last_deep_scrub_stamp":"2024-04-26T16:41:20.280571+0000","last_clean_scrub_stamp":"2024-04-26T16:42:34.354814+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.859237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1826,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927899+0000","last_change":"2024-04-26T16:41:55.064749+0000","last_active":"2024-04-26T16:42:37.927899+0000","last_peered":"2024-04-26T16:42:37.927899+0000","last_clean":"2024-04-26T16:42:37.927899+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:42:37.927899+0000","last_undegraded":"2024-04-26T16:42:37.927899+0000","last_fullsized":"2024-04-26T16:42:37.927899+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:55.064519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:55.064519+0000","last_clean_scrub_stamp":"2024-04-26T16:41:55.064519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:22.442164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":742,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:53.200576+0000","last_change":"2024-04-26T16:42:53.200576+0000","last_active":"2024-04-26T16:42:53.200576+0000","last_peered":"2024-04-26T16:42:53.200576+0000","last_clean":"2024-04-26T16:42:53.200576+0000","last_became_active":"2024-04-26T16:35:25.382820+0000","last_became_peered":"2024-04-26T16:35:25.382820+0000","last_unstale":"2024-04-26T16:42:53.200576+0000","last_undegraded":"2024-04-26T16:42:53.200576+0000","last_fullsized":"2024-04-26T16:42:53.200576+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.200335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:53.200335+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.200335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.012637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1655,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451387+0000","last_change":"2024-04-26T16:42:07.435892+0000","last_active":"2024-04-26T16:42:38.451387+0000","last_peered":"2024-04-26T16:42:38.451387+0000","last_clean":"2024-04-26T16:42:38.451387+0000","last_became_active":"2024-04-26T16:35:25.399977+0000","last_became_peered":"2024-04-26T16:35:25.399977+0000","last_unstale":"2024-04-26T16:42:38.451387+0000","last_undegraded":"2024-04-26T16:42:38.451387+0000","last_fullsized":"2024-04-26T16:42:38.451387+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:07.435665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.435665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:17.179892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"857'30","reported_seq":768,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:52.187019+0000","last_change":"2024-04-26T16:42:52.187165+0000","last_active":"2024-04-26T16:42:52.187019+0000","last_peered":"2024-04-26T16:42:52.187019+0000","last_clean":"2024-04-26T16:42:52.187019+0000","last_became_active":"2024-04-26T16:35:25.393291+0000","last_became_peered":"2024-04-26T16:35:25.393291+0000","last_unstale":"2024-04-26T16:42:52.187019+0000","last_undegraded":"2024-04-26T16:42:52.187019+0000","last_fullsized":"2024-04-26T16:42:52.187019+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:42:52.186807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.186807+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:12.504553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1830,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:41.069362+0000","last_change":"2024-04-26T16:42:41.069430+0000","last_active":"2024-04-26T16:42:41.069362+0000","last_peered":"2024-04-26T16:42:41.069362+0000","last_clean":"2024-04-26T16:42:41.069362+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:42:41.069362+0000","last_undegraded":"2024-04-26T16:42:41.069362+0000","last_fullsized":"2024-04-26T16:42:41.069362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:41.069198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:57.187612+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.069198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.526843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1826,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.924703+0000","last_change":"2024-04-26T16:42:26.248376+0000","last_active":"2024-04-26T16:42:37.924703+0000","last_peered":"2024-04-26T16:42:37.924703+0000","last_clean":"2024-04-26T16:42:37.924703+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:42:37.924703+0000","last_undegraded":"2024-04-26T16:42:37.924703+0000","last_fullsized":"2024-04-26T16:42:37.924703+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:26.248164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:26.248164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:55.830972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1821,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.447977+0000","last_change":"2024-04-26T16:42:36.241515+0000","last_active":"2024-04-26T16:42:38.447977+0000","last_peered":"2024-04-26T16:42:38.447977+0000","last_clean":"2024-04-26T16:42:38.447977+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:42:38.447977+0000","last_undegraded":"2024-04-26T16:42:38.447977+0000","last_fullsized":"2024-04-26T16:42:38.447977+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:36.241334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:01.875991+0000","last_clean_scrub_stamp":"2024-04-26T16:42:36.241334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.353720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1683,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:59.354597+0000","last_change":"2024-04-26T16:42:59.354653+0000","last_active":"2024-04-26T16:42:59.354597+0000","last_peered":"2024-04-26T16:42:59.354597+0000","last_clean":"2024-04-26T16:42:59.354597+0000","last_became_active":"2024-04-26T16:35:25.385146+0000","last_became_peered":"2024-04-26T16:35:25.385146+0000","last_unstale":"2024-04-26T16:42:59.354597+0000","last_undegraded":"2024-04-26T16:42:59.354597+0000","last_fullsized":"2024-04-26T16:42:59.354597+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:59.354438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:59.354438+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.354438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:11.501488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"857'53","reported_seq":1933,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:57.379716+0000","last_change":"2024-04-26T16:42:57.379802+0000","last_active":"2024-04-26T16:42:57.379716+0000","last_peered":"2024-04-26T16:42:57.379716+0000","last_clean":"2024-04-26T16:42:57.379716+0000","last_became_active":"2024-04-26T16:35:25.387497+0000","last_became_peered":"2024-04-26T16:35:25.387497+0000","last_unstale":"2024-04-26T16:42:57.379716+0000","last_undegraded":"2024-04-26T16:42:57.379716+0000","last_fullsized":"2024-04-26T16:42:57.379716+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'53","last_scrub_stamp":"2024-04-26T16:42:57.379550+0000","last_deep_scrub":"857'53","last_deep_scrub_stamp":"2024-04-26T16:39:14.920205+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.379550+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:16.342719+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1846,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927476+0000","last_change":"2024-04-26T16:42:10.097760+0000","last_active":"2024-04-26T16:42:37.927476+0000","last_peered":"2024-04-26T16:42:37.927476+0000","last_clean":"2024-04-26T16:42:37.927476+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:42:37.927476+0000","last_undegraded":"2024-04-26T16:42:37.927476+0000","last_fullsized":"2024-04-26T16:42:37.927476+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:10.097423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:10.097423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:19.608687+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1838,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:43.070992+0000","last_change":"2024-04-26T16:42:43.071047+0000","last_active":"2024-04-26T16:42:43.070992+0000","last_peered":"2024-04-26T16:42:43.070992+0000","last_clean":"2024-04-26T16:42:43.070992+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:42:43.070992+0000","last_undegraded":"2024-04-26T16:42:43.070992+0000","last_fullsized":"2024-04-26T16:42:43.070992+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:43.070838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:43.070838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:08.938502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1781,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448477+0000","last_change":"2024-04-26T16:42:16.431899+0000","last_active":"2024-04-26T16:42:38.448477+0000","last_peered":"2024-04-26T16:42:38.448477+0000","last_clean":"2024-04-26T16:42:38.448477+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:42:38.448477+0000","last_undegraded":"2024-04-26T16:42:38.448477+0000","last_fullsized":"2024-04-26T16:42:38.448477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:16.431658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:42:16.431658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:20.097666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"101'130","reported_seq":459,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448440+0000","last_change":"2024-04-26T16:41:50.977787+0000","last_active":"2024-04-26T16:42:38.448440+0000","last_peered":"2024-04-26T16:42:38.448440+0000","last_clean":"2024-04-26T16:42:38.448440+0000","last_became_active":"2024-04-26T16:38:46.970801+0000","last_became_peered":"2024-04-26T16:38:46.970801+0000","last_unstale":"2024-04-26T16:42:38.448440+0000","last_undegraded":"2024-04-26T16:42:38.448440+0000","last_fullsized":"2024-04-26T16:42:38.448440+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:41:50.977345+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.977345+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:14.982960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1678,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:49.173873+0000","last_change":"2024-04-26T16:42:49.173899+0000","last_active":"2024-04-26T16:42:49.173873+0000","last_peered":"2024-04-26T16:42:49.173873+0000","last_clean":"2024-04-26T16:42:49.173873+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:42:49.173873+0000","last_undegraded":"2024-04-26T16:42:49.173873+0000","last_fullsized":"2024-04-26T16:42:49.173873+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:49.173787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:04.824897+0000","last_clean_scrub_stamp":"2024-04-26T16:42:49.173787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.341183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"907'60","reported_seq":1584,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:06.215105+0000","last_change":"2024-04-26T16:43:06.215205+0000","last_active":"2024-04-26T16:43:06.215105+0000","last_peered":"2024-04-26T16:43:06.215105+0000","last_clean":"2024-04-26T16:43:06.215105+0000","last_became_active":"2024-04-26T16:35:25.414489+0000","last_became_peered":"2024-04-26T16:35:25.414489+0000","last_unstale":"2024-04-26T16:43:06.215105+0000","last_undegraded":"2024-04-26T16:43:06.215105+0000","last_fullsized":"2024-04-26T16:43:06.215105+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'60","last_scrub_stamp":"2024-04-26T16:43:06.214936+0000","last_deep_scrub":"907'60","last_deep_scrub_stamp":"2024-04-26T16:40:18.066521+0000","last_clean_scrub_stamp":"2024-04-26T16:43:06.214936+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.603610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1805,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.453682+0000","last_change":"2024-04-26T16:41:50.426808+0000","last_active":"2024-04-26T16:42:38.453682+0000","last_peered":"2024-04-26T16:42:38.453682+0000","last_clean":"2024-04-26T16:42:38.453682+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:42:38.453682+0000","last_undegraded":"2024-04-26T16:42:38.453682+0000","last_fullsized":"2024-04-26T16:42:38.453682+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:50.426591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:50.426591+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.426591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:12.697594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1830,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927004+0000","last_change":"2024-04-26T16:42:34.062846+0000","last_active":"2024-04-26T16:42:37.927004+0000","last_peered":"2024-04-26T16:42:37.927004+0000","last_clean":"2024-04-26T16:42:37.927004+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:42:37.927004+0000","last_undegraded":"2024-04-26T16:42:37.927004+0000","last_fullsized":"2024-04-26T16:42:37.927004+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:34.062625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:20.156962+0000","last_clean_scrub_stamp":"2024-04-26T16:42:34.062625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.244190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1822,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:54.228098+0000","last_change":"2024-04-26T16:42:54.228119+0000","last_active":"2024-04-26T16:42:54.228098+0000","last_peered":"2024-04-26T16:42:54.228098+0000","last_clean":"2024-04-26T16:42:54.228098+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:42:54.228098+0000","last_undegraded":"2024-04-26T16:42:54.228098+0000","last_fullsized":"2024-04-26T16:42:54.228098+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:54.228037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:33.120136+0000","last_clean_scrub_stamp":"2024-04-26T16:42:54.228037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:19.253610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'130","reported_seq":2118,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:02.400242+0000","last_change":"2024-04-26T16:43:02.400306+0000","last_active":"2024-04-26T16:43:02.400242+0000","last_peered":"2024-04-26T16:43:02.400242+0000","last_clean":"2024-04-26T16:43:02.400242+0000","last_became_active":"2024-04-26T16:38:46.976273+0000","last_became_peered":"2024-04-26T16:38:46.976273+0000","last_unstale":"2024-04-26T16:43:02.400242+0000","last_undegraded":"2024-04-26T16:43:02.400242+0000","last_fullsized":"2024-04-26T16:43:02.400242+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:43:02.400066+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:43:02.400066+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.400066+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:21.702252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1291,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.455693+0000","last_change":"2024-04-26T16:42:23.169869+0000","last_active":"2024-04-26T16:42:38.455693+0000","last_peered":"2024-04-26T16:42:38.455693+0000","last_clean":"2024-04-26T16:42:38.455693+0000","last_became_active":"2024-04-26T16:35:25.388195+0000","last_became_peered":"2024-04-26T16:35:25.388195+0000","last_unstale":"2024-04-26T16:42:38.455693+0000","last_undegraded":"2024-04-26T16:42:38.455693+0000","last_fullsized":"2024-04-26T16:42:38.455693+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:23.169669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:23.169669+0000","last_clean_scrub_stamp":"2024-04-26T16:42:23.169669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:47.195006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"857'25","reported_seq":1844,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448767+0000","last_change":"2024-04-26T16:42:34.022458+0000","last_active":"2024-04-26T16:42:38.448767+0000","last_peered":"2024-04-26T16:42:38.448767+0000","last_clean":"2024-04-26T16:42:38.448767+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:42:38.448767+0000","last_undegraded":"2024-04-26T16:42:38.448767+0000","last_fullsized":"2024-04-26T16:42:38.448767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:42:34.022223+0000","last_deep_scrub":"857'25","last_deep_scrub_stamp":"2024-04-26T16:40:08.182235+0000","last_clean_scrub_stamp":"2024-04-26T16:42:34.022223+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:38.520984+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1830,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:42.037199+0000","last_change":"2024-04-26T16:42:42.037307+0000","last_active":"2024-04-26T16:42:42.037199+0000","last_peered":"2024-04-26T16:42:42.037199+0000","last_clean":"2024-04-26T16:42:42.037199+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:42:42.037199+0000","last_undegraded":"2024-04-26T16:42:42.037199+0000","last_fullsized":"2024-04-26T16:42:42.037199+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:42.037049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:08.142728+0000","last_clean_scrub_stamp":"2024-04-26T16:42:42.037049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.184688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":2164,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272617+0000","last_change":"2024-04-26T16:42:19.067756+0000","last_active":"2024-04-26T16:43:00.272617+0000","last_peered":"2024-04-26T16:43:00.272617+0000","last_clean":"2024-04-26T16:43:00.272617+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:43:00.272617+0000","last_undegraded":"2024-04-26T16:43:00.272617+0000","last_fullsized":"2024-04-26T16:43:00.272617+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:42:19.067550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:19.067550+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:22.639890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1822,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927607+0000","last_change":"2024-04-26T16:42:22.879668+0000","last_active":"2024-04-26T16:42:37.927607+0000","last_peered":"2024-04-26T16:42:37.927607+0000","last_clean":"2024-04-26T16:42:37.927607+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:42:37.927607+0000","last_undegraded":"2024-04-26T16:42:37.927607+0000","last_fullsized":"2024-04-26T16:42:37.927607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:22.879454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.893722+0000","last_clean_scrub_stamp":"2024-04-26T16:42:22.879454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:28.972997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"101'130","reported_seq":467,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449496+0000","last_change":"2024-04-26T16:42:31.112870+0000","last_active":"2024-04-26T16:42:38.449496+0000","last_peered":"2024-04-26T16:42:38.449496+0000","last_clean":"2024-04-26T16:42:38.449496+0000","last_became_active":"2024-04-26T16:38:46.964902+0000","last_became_peered":"2024-04-26T16:38:46.964902+0000","last_unstale":"2024-04-26T16:42:38.449496+0000","last_undegraded":"2024-04-26T16:42:38.449496+0000","last_fullsized":"2024-04-26T16:42:38.449496+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:31.112594+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:42:31.112594+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:51.566028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1680,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:52.180577+0000","last_change":"2024-04-26T16:42:52.180638+0000","last_active":"2024-04-26T16:42:52.180577+0000","last_peered":"2024-04-26T16:42:52.180577+0000","last_clean":"2024-04-26T16:42:52.180577+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:42:52.180577+0000","last_undegraded":"2024-04-26T16:42:52.180577+0000","last_fullsized":"2024-04-26T16:42:52.180577+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:52.180410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:17.328349+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.180410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.777727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"857'46","reported_seq":1866,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.447259+0000","last_change":"2024-04-26T16:41:43.013726+0000","last_active":"2024-04-26T16:42:38.447259+0000","last_peered":"2024-04-26T16:42:38.447259+0000","last_clean":"2024-04-26T16:42:38.447259+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:42:38.447259+0000","last_undegraded":"2024-04-26T16:42:38.447259+0000","last_fullsized":"2024-04-26T16:42:38.447259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:41:43.013322+0000","last_deep_scrub":"857'46","last_deep_scrub_stamp":"2024-04-26T16:40:19.096428+0000","last_clean_scrub_stamp":"2024-04-26T16:41:43.013322+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:06.958863+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1844,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.926360+0000","last_change":"2024-04-26T16:42:29.025575+0000","last_active":"2024-04-26T16:42:37.926360+0000","last_peered":"2024-04-26T16:42:37.926360+0000","last_clean":"2024-04-26T16:42:37.926360+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:42:37.926360+0000","last_undegraded":"2024-04-26T16:42:37.926360+0000","last_fullsized":"2024-04-26T16:42:37.926360+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:29.025381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:42:29.025381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:33.052455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":2165,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272679+0000","last_change":"2024-04-26T16:42:57.821822+0000","last_active":"2024-04-26T16:43:00.272679+0000","last_peered":"2024-04-26T16:43:00.272679+0000","last_clean":"2024-04-26T16:43:00.272679+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:43:00.272679+0000","last_undegraded":"2024-04-26T16:43:00.272679+0000","last_fullsized":"2024-04-26T16:43:00.272679+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:42:57.821585+0000","last_deep_scrub":"585'14","last_deep_scrub_stamp":"2024-04-26T16:42:57.821585+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.821585+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.592349+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":740,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.455385+0000","last_change":"2024-04-26T16:42:29.099545+0000","last_active":"2024-04-26T16:42:38.455385+0000","last_peered":"2024-04-26T16:42:38.455385+0000","last_clean":"2024-04-26T16:42:38.455385+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:42:38.455385+0000","last_undegraded":"2024-04-26T16:42:38.455385+0000","last_fullsized":"2024-04-26T16:42:38.455385+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:29.099258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:00.008398+0000","last_clean_scrub_stamp":"2024-04-26T16:42:29.099258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.484902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":1312,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.452925+0000","last_change":"2024-04-26T16:42:22.129969+0000","last_active":"2024-04-26T16:42:38.452925+0000","last_peered":"2024-04-26T16:42:38.452925+0000","last_clean":"2024-04-26T16:42:38.452925+0000","last_became_active":"2024-04-26T16:35:25.407637+0000","last_became_peered":"2024-04-26T16:35:25.407637+0000","last_unstale":"2024-04-26T16:42:38.452925+0000","last_undegraded":"2024-04-26T16:42:38.452925+0000","last_fullsized":"2024-04-26T16:42:38.452925+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:42:22.129728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:22.129728+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.552357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"856'29","reported_seq":1868,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:40.870358+0000","last_change":"2024-04-26T16:42:40.870439+0000","last_active":"2024-04-26T16:42:40.870358+0000","last_peered":"2024-04-26T16:42:40.870358+0000","last_clean":"2024-04-26T16:42:40.870358+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:42:40.870358+0000","last_undegraded":"2024-04-26T16:42:40.870358+0000","last_fullsized":"2024-04-26T16:42:40.870358+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'29","last_scrub_stamp":"2024-04-26T16:42:40.870176+0000","last_deep_scrub":"824'23","last_deep_scrub_stamp":"2024-04-26T16:36:53.947636+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.870176+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:45.773872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1809,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449095+0000","last_change":"2024-04-26T16:42:14.414627+0000","last_active":"2024-04-26T16:42:38.449095+0000","last_peered":"2024-04-26T16:42:38.449095+0000","last_clean":"2024-04-26T16:42:38.449095+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:42:38.449095+0000","last_undegraded":"2024-04-26T16:42:38.449095+0000","last_fullsized":"2024-04-26T16:42:38.449095+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:42:14.414402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:14.414402+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.593834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":741,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.455726+0000","last_change":"2024-04-26T16:42:33.105010+0000","last_active":"2024-04-26T16:42:38.455726+0000","last_peered":"2024-04-26T16:42:38.455726+0000","last_clean":"2024-04-26T16:42:38.455726+0000","last_became_active":"2024-04-26T16:35:25.382698+0000","last_became_peered":"2024-04-26T16:35:25.382698+0000","last_unstale":"2024-04-26T16:42:38.455726+0000","last_undegraded":"2024-04-26T16:42:38.455726+0000","last_fullsized":"2024-04-26T16:42:38.455726+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:33.104784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:33.104784+0000","last_clean_scrub_stamp":"2024-04-26T16:42:33.104784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:52.669838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1812,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449466+0000","last_change":"2024-04-26T16:42:35.028571+0000","last_active":"2024-04-26T16:42:38.449466+0000","last_peered":"2024-04-26T16:42:38.449466+0000","last_clean":"2024-04-26T16:42:38.449466+0000","last_became_active":"2024-04-26T16:35:25.416938+0000","last_became_peered":"2024-04-26T16:35:25.416938+0000","last_unstale":"2024-04-26T16:42:38.449466+0000","last_undegraded":"2024-04-26T16:42:38.449466+0000","last_fullsized":"2024-04-26T16:42:38.449466+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:35.028420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:47.296951+0000","last_clean_scrub_stamp":"2024-04-26T16:42:35.028420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.473689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'130","reported_seq":460,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449171+0000","last_change":"2024-04-26T16:42:04.989479+0000","last_active":"2024-04-26T16:42:38.449171+0000","last_peered":"2024-04-26T16:42:38.449171+0000","last_clean":"2024-04-26T16:42:38.449171+0000","last_became_active":"2024-04-26T16:38:46.969591+0000","last_became_peered":"2024-04-26T16:38:46.969591+0000","last_unstale":"2024-04-26T16:42:38.449171+0000","last_undegraded":"2024-04-26T16:42:38.449171+0000","last_fullsized":"2024-04-26T16:42:38.449171+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:04.989127+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:42:04.989127+0000","last_clean_scrub_stamp":"2024-04-26T16:42:04.989127+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:09.030218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1675,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927302+0000","last_change":"2024-04-26T16:42:07.761121+0000","last_active":"2024-04-26T16:42:37.927302+0000","last_peered":"2024-04-26T16:42:37.927302+0000","last_clean":"2024-04-26T16:42:37.927302+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:42:37.927302+0000","last_undegraded":"2024-04-26T16:42:37.927302+0000","last_fullsized":"2024-04-26T16:42:37.927302+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:07.760906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.760906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:21.313085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":2114,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272757+0000","last_change":"2024-04-26T16:42:07.031827+0000","last_active":"2024-04-26T16:43:00.272757+0000","last_peered":"2024-04-26T16:43:00.272757+0000","last_clean":"2024-04-26T16:43:00.272757+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:43:00.272757+0000","last_undegraded":"2024-04-26T16:43:00.272757+0000","last_fullsized":"2024-04-26T16:43:00.272757+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:42:07.031596+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.031596+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:34.357075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1850,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925642+0000","last_change":"2024-04-26T16:41:49.784468+0000","last_active":"2024-04-26T16:42:37.925642+0000","last_peered":"2024-04-26T16:42:37.925642+0000","last_clean":"2024-04-26T16:42:37.925642+0000","last_became_active":"2024-04-26T16:35:25.411443+0000","last_became_peered":"2024-04-26T16:35:25.411443+0000","last_unstale":"2024-04-26T16:42:37.925642+0000","last_undegraded":"2024-04-26T16:42:37.925642+0000","last_fullsized":"2024-04-26T16:42:37.925642+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:49.784162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:31:42.531687+0000","last_clean_scrub_stamp":"2024-04-26T16:41:49.784162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:07.089269+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"857'42","reported_seq":1881,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448041+0000","last_change":"2024-04-26T16:41:53.018973+0000","last_active":"2024-04-26T16:42:38.448041+0000","last_peered":"2024-04-26T16:42:38.448041+0000","last_clean":"2024-04-26T16:42:38.448041+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:42:38.448041+0000","last_undegraded":"2024-04-26T16:42:38.448041+0000","last_fullsized":"2024-04-26T16:42:38.448041+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'42","last_scrub_stamp":"2024-04-26T16:41:53.018846+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:41:53.018846+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:19.676483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1423,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:50.233719+0000","last_change":"2024-04-26T16:42:50.233777+0000","last_active":"2024-04-26T16:42:50.233719+0000","last_peered":"2024-04-26T16:42:50.233719+0000","last_clean":"2024-04-26T16:42:50.233719+0000","last_became_active":"2024-04-26T16:35:25.406996+0000","last_became_peered":"2024-04-26T16:35:25.406996+0000","last_unstale":"2024-04-26T16:42:50.233719+0000","last_undegraded":"2024-04-26T16:42:50.233719+0000","last_fullsized":"2024-04-26T16:42:50.233719+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:50.233569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:32.453036+0000","last_clean_scrub_stamp":"2024-04-26T16:42:50.233569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:58.037597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1800,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.453241+0000","last_change":"2024-04-26T16:42:21.377729+0000","last_active":"2024-04-26T16:42:38.453241+0000","last_peered":"2024-04-26T16:42:38.453241+0000","last_clean":"2024-04-26T16:42:38.453241+0000","last_became_active":"2024-04-26T16:35:25.391371+0000","last_became_peered":"2024-04-26T16:35:25.391371+0000","last_unstale":"2024-04-26T16:42:38.453241+0000","last_undegraded":"2024-04-26T16:42:38.453241+0000","last_fullsized":"2024-04-26T16:42:38.453241+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:21.377458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.377458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.768264+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'130","reported_seq":468,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:41.830760+0000","last_change":"2024-04-26T16:42:41.830760+0000","last_active":"2024-04-26T16:42:41.830760+0000","last_peered":"2024-04-26T16:42:41.830760+0000","last_clean":"2024-04-26T16:42:41.830760+0000","last_became_active":"2024-04-26T16:38:46.975448+0000","last_became_peered":"2024-04-26T16:38:46.975448+0000","last_unstale":"2024-04-26T16:42:41.830760+0000","last_undegraded":"2024-04-26T16:42:41.830760+0000","last_fullsized":"2024-04-26T16:42:41.830760+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:41.830542+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.830542+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.945657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1644,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.453140+0000","last_change":"2024-04-26T16:42:11.405087+0000","last_active":"2024-04-26T16:42:38.453140+0000","last_peered":"2024-04-26T16:42:38.453140+0000","last_clean":"2024-04-26T16:42:38.453140+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:42:38.453140+0000","last_undegraded":"2024-04-26T16:42:38.453140+0000","last_fullsized":"2024-04-26T16:42:38.453140+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:11.404886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:44.032590+0000","last_clean_scrub_stamp":"2024-04-26T16:42:11.404886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:18.443013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1827,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925331+0000","last_change":"2024-04-26T16:41:46.712930+0000","last_active":"2024-04-26T16:42:37.925331+0000","last_peered":"2024-04-26T16:42:37.925331+0000","last_clean":"2024-04-26T16:42:37.925331+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:42:37.925331+0000","last_undegraded":"2024-04-26T16:42:37.925331+0000","last_fullsized":"2024-04-26T16:42:37.925331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:46.712725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:41:46.712725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:12.608975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":3329,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:05.453730+0000","last_change":"2024-04-26T16:42:59.582290+0000","last_active":"2024-04-26T16:43:05.453730+0000","last_peered":"2024-04-26T16:43:05.453730+0000","last_clean":"2024-04-26T16:43:05.453730+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:43:05.453730+0000","last_undegraded":"2024-04-26T16:43:05.453730+0000","last_fullsized":"2024-04-26T16:43:05.453730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:42:59.581912+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.581912+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:27.614297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1514,"num_read_kb":1509,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"857'26","reported_seq":1850,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:53.742001+0000","last_change":"2024-04-26T16:42:53.742082+0000","last_active":"2024-04-26T16:42:53.742001+0000","last_peered":"2024-04-26T16:42:53.742001+0000","last_clean":"2024-04-26T16:42:53.742001+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:42:53.742001+0000","last_undegraded":"2024-04-26T16:42:53.742001+0000","last_fullsized":"2024-04-26T16:42:53.742001+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'26","last_scrub_stamp":"2024-04-26T16:42:53.741837+0000","last_deep_scrub":"857'26","last_deep_scrub_stamp":"2024-04-26T16:38:31.384645+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.741837+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.841741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1842,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.928505+0000","last_change":"2024-04-26T16:42:10.822380+0000","last_active":"2024-04-26T16:42:37.928505+0000","last_peered":"2024-04-26T16:42:37.928505+0000","last_clean":"2024-04-26T16:42:37.928505+0000","last_became_active":"2024-04-26T16:35:25.398461+0000","last_became_peered":"2024-04-26T16:35:25.398461+0000","last_unstale":"2024-04-26T16:42:37.928505+0000","last_undegraded":"2024-04-26T16:42:37.928505+0000","last_fullsized":"2024-04-26T16:42:37.928505+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:42:10.822025+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:40:49.964369+0000","last_clean_scrub_stamp":"2024-04-26T16:42:10.822025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:15.924116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1823,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925747+0000","last_change":"2024-04-26T16:42:23.918628+0000","last_active":"2024-04-26T16:42:37.925747+0000","last_peered":"2024-04-26T16:42:37.925747+0000","last_clean":"2024-04-26T16:42:37.925747+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:42:37.925747+0000","last_undegraded":"2024-04-26T16:42:37.925747+0000","last_fullsized":"2024-04-26T16:42:37.925747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:23.918414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:42:23.918414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:45.335918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"101'130","reported_seq":464,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.456760+0000","last_change":"2024-04-26T16:41:58.112705+0000","last_active":"2024-04-26T16:42:38.456760+0000","last_peered":"2024-04-26T16:42:38.456760+0000","last_clean":"2024-04-26T16:42:38.456760+0000","last_became_active":"2024-04-26T16:38:46.981826+0000","last_became_peered":"2024-04-26T16:38:46.981826+0000","last_unstale":"2024-04-26T16:42:38.456760+0000","last_undegraded":"2024-04-26T16:42:38.456760+0000","last_fullsized":"2024-04-26T16:42:38.456760+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:41:58.112409+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:41:58.112409+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.112409+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:08.062128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1671,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.926660+0000","last_change":"2024-04-26T16:42:19.014379+0000","last_active":"2024-04-26T16:42:37.926660+0000","last_peered":"2024-04-26T16:42:37.926660+0000","last_clean":"2024-04-26T16:42:37.926660+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:42:37.926660+0000","last_undegraded":"2024-04-26T16:42:37.926660+0000","last_fullsized":"2024-04-26T16:42:37.926660+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:19.014047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:19.014047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.074321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1409,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.454889+0000","last_change":"2024-04-26T16:42:07.158773+0000","last_active":"2024-04-26T16:42:38.454889+0000","last_peered":"2024-04-26T16:42:38.454889+0000","last_clean":"2024-04-26T16:42:38.454889+0000","last_became_active":"2024-04-26T16:35:25.404395+0000","last_became_peered":"2024-04-26T16:35:25.404395+0000","last_unstale":"2024-04-26T16:42:38.454889+0000","last_undegraded":"2024-04-26T16:42:38.454889+0000","last_fullsized":"2024-04-26T16:42:38.454889+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:07.158568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.158568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:09.386545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"857'66","reported_seq":2287,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272588+0000","last_change":"2024-04-26T16:42:43.032147+0000","last_active":"2024-04-26T16:43:00.272588+0000","last_peered":"2024-04-26T16:43:00.272588+0000","last_clean":"2024-04-26T16:43:00.272588+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:43:00.272588+0000","last_undegraded":"2024-04-26T16:43:00.272588+0000","last_fullsized":"2024-04-26T16:43:00.272588+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'66","last_scrub_stamp":"2024-04-26T16:42:43.031896+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:42:43.031896+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:03.076663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1811,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:40.428382+0000","last_change":"2024-04-26T16:42:40.428452+0000","last_active":"2024-04-26T16:42:40.428382+0000","last_peered":"2024-04-26T16:42:40.428382+0000","last_clean":"2024-04-26T16:42:40.428382+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:42:40.428382+0000","last_undegraded":"2024-04-26T16:42:40.428382+0000","last_fullsized":"2024-04-26T16:42:40.428382+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:42:40.428219+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:41:14.264394+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.428219+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.465194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1829,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:39.131836+0000","last_change":"2024-04-26T16:42:39.131897+0000","last_active":"2024-04-26T16:42:39.131836+0000","last_peered":"2024-04-26T16:42:39.131836+0000","last_clean":"2024-04-26T16:42:39.131836+0000","last_became_active":"2024-04-26T16:35:25.411888+0000","last_became_peered":"2024-04-26T16:35:25.411888+0000","last_unstale":"2024-04-26T16:42:39.131836+0000","last_undegraded":"2024-04-26T16:42:39.131836+0000","last_fullsized":"2024-04-26T16:42:39.131836+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:39.131672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:39.131672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:39.161851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'130","reported_seq":468,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:53.186092+0000","last_change":"2024-04-26T16:42:53.186092+0000","last_active":"2024-04-26T16:42:53.186092+0000","last_peered":"2024-04-26T16:42:53.186092+0000","last_clean":"2024-04-26T16:42:53.186092+0000","last_became_active":"2024-04-26T16:38:46.960634+0000","last_became_peered":"2024-04-26T16:38:46.960634+0000","last_unstale":"2024-04-26T16:42:53.186092+0000","last_undegraded":"2024-04-26T16:42:53.186092+0000","last_fullsized":"2024-04-26T16:42:53.186092+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:53.185871+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.185871+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:21.123382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1656,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:01.638598+0000","last_change":"2024-04-26T16:43:01.638598+0000","last_active":"2024-04-26T16:43:01.638598+0000","last_peered":"2024-04-26T16:43:01.638598+0000","last_clean":"2024-04-26T16:43:01.638598+0000","last_became_active":"2024-04-26T16:35:25.417017+0000","last_became_peered":"2024-04-26T16:35:25.417017+0000","last_unstale":"2024-04-26T16:43:01.638598+0000","last_undegraded":"2024-04-26T16:43:01.638598+0000","last_fullsized":"2024-04-26T16:43:01.638598+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:01.638360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:43:01.638360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.000991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":741,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.455576+0000","last_change":"2024-04-26T16:41:54.004884+0000","last_active":"2024-04-26T16:42:38.455576+0000","last_peered":"2024-04-26T16:42:38.455576+0000","last_clean":"2024-04-26T16:42:38.455576+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:42:38.455576+0000","last_undegraded":"2024-04-26T16:42:38.455576+0000","last_fullsized":"2024-04-26T16:42:38.455576+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:54.004676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:41:54.004676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:14.538352+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1794,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448834+0000","last_change":"2024-04-26T16:41:38.305650+0000","last_active":"2024-04-26T16:42:38.448834+0000","last_peered":"2024-04-26T16:42:38.448834+0000","last_clean":"2024-04-26T16:42:38.448834+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:42:38.448834+0000","last_undegraded":"2024-04-26T16:42:38.448834+0000","last_fullsized":"2024-04-26T16:42:38.448834+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:38.305438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:26.045883+0000","last_clean_scrub_stamp":"2024-04-26T16:41:38.305438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:04.167826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"857'49","reported_seq":1892,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:50.193019+0000","last_change":"2024-04-26T16:42:50.193055+0000","last_active":"2024-04-26T16:42:50.193019+0000","last_peered":"2024-04-26T16:42:50.193019+0000","last_clean":"2024-04-26T16:42:50.193019+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:42:50.193019+0000","last_undegraded":"2024-04-26T16:42:50.193019+0000","last_fullsized":"2024-04-26T16:42:50.193019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'49","last_scrub_stamp":"2024-04-26T16:42:50.192938+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:42:50.192938+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:57.244039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1840,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:02.006095+0000","last_change":"2024-04-26T16:43:02.006158+0000","last_active":"2024-04-26T16:43:02.006095+0000","last_peered":"2024-04-26T16:43:02.006095+0000","last_clean":"2024-04-26T16:43:02.006095+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:43:02.006095+0000","last_undegraded":"2024-04-26T16:43:02.006095+0000","last_fullsized":"2024-04-26T16:43:02.006095+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:02.005980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:37.388271+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.005980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:16.941966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1676,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448655+0000","last_change":"2024-04-26T16:42:17.124471+0000","last_active":"2024-04-26T16:42:38.448655+0000","last_peered":"2024-04-26T16:42:38.448655+0000","last_clean":"2024-04-26T16:42:38.448655+0000","last_became_active":"2024-04-26T16:35:25.417111+0000","last_became_peered":"2024-04-26T16:35:25.417111+0000","last_unstale":"2024-04-26T16:42:38.448655+0000","last_undegraded":"2024-04-26T16:42:38.448655+0000","last_fullsized":"2024-04-26T16:42:38.448655+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:17.124345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:17.124345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.696793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1798,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.452451+0000","last_change":"2024-04-26T16:41:30.170518+0000","last_active":"2024-04-26T16:42:38.452451+0000","last_peered":"2024-04-26T16:42:38.452451+0000","last_clean":"2024-04-26T16:42:38.452451+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:42:38.452451+0000","last_undegraded":"2024-04-26T16:42:38.452451+0000","last_fullsized":"2024-04-26T16:42:38.452451+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:30.170299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:55.132763+0000","last_clean_scrub_stamp":"2024-04-26T16:41:30.170299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:58.369010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":2103,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448957+0000","last_change":"2024-04-26T16:41:40.348657+0000","last_active":"2024-04-26T16:42:38.448957+0000","last_peered":"2024-04-26T16:42:38.448957+0000","last_clean":"2024-04-26T16:42:38.448957+0000","last_became_active":"2024-04-26T16:38:41.889008+0000","last_became_peered":"2024-04-26T16:38:41.889008+0000","last_unstale":"2024-04-26T16:42:38.448957+0000","last_undegraded":"2024-04-26T16:42:38.448957+0000","last_fullsized":"2024-04-26T16:42:38.448957+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:41:40.348428+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:38:41.998073+0000","last_clean_scrub_stamp":"2024-04-26T16:41:40.348428+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":741,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:39.070049+0000","last_change":"2024-04-26T16:42:39.070105+0000","last_active":"2024-04-26T16:42:39.070049+0000","last_peered":"2024-04-26T16:42:39.070049+0000","last_clean":"2024-04-26T16:42:39.070049+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:42:39.070049+0000","last_undegraded":"2024-04-26T16:42:39.070049+0000","last_fullsized":"2024-04-26T16:42:39.070049+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:39.069892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:58.958586+0000","last_clean_scrub_stamp":"2024-04-26T16:42:39.069892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:52.572104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"857'35","reported_seq":1903,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448724+0000","last_change":"2024-04-26T16:41:56.981654+0000","last_active":"2024-04-26T16:42:38.448724+0000","last_peered":"2024-04-26T16:42:38.448724+0000","last_clean":"2024-04-26T16:42:38.448724+0000","last_became_active":"2024-04-26T16:35:25.397667+0000","last_became_peered":"2024-04-26T16:35:25.397667+0000","last_unstale":"2024-04-26T16:42:38.448724+0000","last_undegraded":"2024-04-26T16:42:38.448724+0000","last_fullsized":"2024-04-26T16:42:38.448724+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'35","last_scrub_stamp":"2024-04-26T16:41:56.981398+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:41:56.981398+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:02.024396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":749,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:43.092576+0000","last_change":"2024-04-26T16:42:43.092576+0000","last_active":"2024-04-26T16:42:43.092576+0000","last_peered":"2024-04-26T16:42:43.092576+0000","last_clean":"2024-04-26T16:42:43.092576+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:42:43.092576+0000","last_undegraded":"2024-04-26T16:42:43.092576+0000","last_fullsized":"2024-04-26T16:42:43.092576+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:43.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:42:43.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.634594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"101'130","reported_seq":2110,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449719+0000","last_change":"2024-04-26T16:41:49.389460+0000","last_active":"2024-04-26T16:42:38.449719+0000","last_peered":"2024-04-26T16:42:38.449719+0000","last_clean":"2024-04-26T16:42:38.449719+0000","last_became_active":"2024-04-26T16:38:46.976486+0000","last_became_peered":"2024-04-26T16:38:46.976486+0000","last_unstale":"2024-04-26T16:42:38.449719+0000","last_undegraded":"2024-04-26T16:42:38.449719+0000","last_fullsized":"2024-04-26T16:42:38.449719+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:41:49.389190+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:41:49.389190+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:53.603278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1676,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:40.951206+0000","last_change":"2024-04-26T16:42:40.951206+0000","last_active":"2024-04-26T16:42:40.951206+0000","last_peered":"2024-04-26T16:42:40.951206+0000","last_clean":"2024-04-26T16:42:40.951206+0000","last_became_active":"2024-04-26T16:35:25.392163+0000","last_became_peered":"2024-04-26T16:35:25.392163+0000","last_unstale":"2024-04-26T16:42:40.951206+0000","last_undegraded":"2024-04-26T16:42:40.951206+0000","last_fullsized":"2024-04-26T16:42:40.951206+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:40.950857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.950857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.095107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1838,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.928724+0000","last_change":"2024-04-26T16:42:15.044914+0000","last_active":"2024-04-26T16:42:37.928724+0000","last_peered":"2024-04-26T16:42:37.928724+0000","last_clean":"2024-04-26T16:42:37.928724+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:42:37.928724+0000","last_undegraded":"2024-04-26T16:42:37.928724+0000","last_fullsized":"2024-04-26T16:42:37.928724+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:15.044693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:42:15.044693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:26.315965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1835,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925062+0000","last_change":"2024-04-26T16:42:24.173831+0000","last_active":"2024-04-26T16:42:37.925062+0000","last_peered":"2024-04-26T16:42:37.925062+0000","last_clean":"2024-04-26T16:42:37.925062+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:42:37.925062+0000","last_undegraded":"2024-04-26T16:42:37.925062+0000","last_fullsized":"2024-04-26T16:42:37.925062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:24.173543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:24.173543+0000","last_clean_scrub_stamp":"2024-04-26T16:42:24.173543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.202289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"857'50","reported_seq":1921,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925338+0000","last_change":"2024-04-26T16:42:30.042965+0000","last_active":"2024-04-26T16:42:37.925338+0000","last_peered":"2024-04-26T16:42:37.925338+0000","last_clean":"2024-04-26T16:42:37.925338+0000","last_became_active":"2024-04-26T16:35:25.412117+0000","last_became_peered":"2024-04-26T16:35:25.412117+0000","last_unstale":"2024-04-26T16:42:37.925338+0000","last_undegraded":"2024-04-26T16:42:37.925338+0000","last_fullsized":"2024-04-26T16:42:37.925338+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'50","last_scrub_stamp":"2024-04-26T16:42:30.042852+0000","last_deep_scrub":"857'50","last_deep_scrub_stamp":"2024-04-26T16:40:53.922832+0000","last_clean_scrub_stamp":"2024-04-26T16:42:30.042852+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.618312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1792,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.455836+0000","last_change":"2024-04-26T16:41:43.440279+0000","last_active":"2024-04-26T16:42:38.455836+0000","last_peered":"2024-04-26T16:42:38.455836+0000","last_clean":"2024-04-26T16:42:38.455836+0000","last_became_active":"2024-04-26T16:35:25.398515+0000","last_became_peered":"2024-04-26T16:35:25.398515+0000","last_unstale":"2024-04-26T16:42:38.455836+0000","last_undegraded":"2024-04-26T16:42:38.455836+0000","last_fullsized":"2024-04-26T16:42:38.455836+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:43.439979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:41:43.439979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:08.172522+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'130","reported_seq":467,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:02.086741+0000","last_change":"2024-04-26T16:43:02.086808+0000","last_active":"2024-04-26T16:43:02.086741+0000","last_peered":"2024-04-26T16:43:02.086741+0000","last_clean":"2024-04-26T16:43:02.086741+0000","last_became_active":"2024-04-26T16:38:46.969238+0000","last_became_peered":"2024-04-26T16:38:46.969238+0000","last_unstale":"2024-04-26T16:43:02.086741+0000","last_undegraded":"2024-04-26T16:43:02.086741+0000","last_fullsized":"2024-04-26T16:43:02.086741+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:43:02.086564+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:43:02.086564+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.086564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:02.565058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1645,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448495+0000","last_change":"2024-04-26T16:41:52.947173+0000","last_active":"2024-04-26T16:42:38.448495+0000","last_peered":"2024-04-26T16:42:38.448495+0000","last_clean":"2024-04-26T16:42:38.448495+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:42:38.448495+0000","last_undegraded":"2024-04-26T16:42:38.448495+0000","last_fullsized":"2024-04-26T16:42:38.448495+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:52.946928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:25.114078+0000","last_clean_scrub_stamp":"2024-04-26T16:41:52.946928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:07.989327+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1818,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.447774+0000","last_change":"2024-04-26T16:42:18.076002+0000","last_active":"2024-04-26T16:42:38.447774+0000","last_peered":"2024-04-26T16:42:38.447774+0000","last_clean":"2024-04-26T16:42:38.447774+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:42:38.447774+0000","last_undegraded":"2024-04-26T16:42:38.447774+0000","last_fullsized":"2024-04-26T16:42:38.447774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:18.075781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:18.075781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:26.483866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"857'47","reported_seq":1907,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.926588+0000","last_change":"2024-04-26T16:42:24.964533+0000","last_active":"2024-04-26T16:42:37.926588+0000","last_peered":"2024-04-26T16:42:37.926588+0000","last_clean":"2024-04-26T16:42:37.926588+0000","last_became_active":"2024-04-26T16:35:25.395746+0000","last_became_peered":"2024-04-26T16:35:25.395746+0000","last_unstale":"2024-04-26T16:42:37.926588+0000","last_undegraded":"2024-04-26T16:42:37.926588+0000","last_fullsized":"2024-04-26T16:42:37.926588+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'47","last_scrub_stamp":"2024-04-26T16:42:24.964285+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:42:24.964285+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:50.639776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":742,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451663+0000","last_change":"2024-04-26T16:42:32.101660+0000","last_active":"2024-04-26T16:42:38.451663+0000","last_peered":"2024-04-26T16:42:38.451663+0000","last_clean":"2024-04-26T16:42:38.451663+0000","last_became_active":"2024-04-26T16:35:25.397940+0000","last_became_peered":"2024-04-26T16:35:25.397940+0000","last_unstale":"2024-04-26T16:42:38.451663+0000","last_undegraded":"2024-04-26T16:42:38.451663+0000","last_fullsized":"2024-04-26T16:42:38.451663+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:32.101453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:42:32.101453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:39.430923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1798,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449288+0000","last_change":"2024-04-26T16:41:50.960451+0000","last_active":"2024-04-26T16:42:38.449288+0000","last_peered":"2024-04-26T16:42:38.449288+0000","last_clean":"2024-04-26T16:42:38.449288+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:42:38.449288+0000","last_undegraded":"2024-04-26T16:42:38.449288+0000","last_fullsized":"2024-04-26T16:42:38.449288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:50.960209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.960209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:16.155974+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'130","reported_seq":472,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.457194+0000","last_change":"2024-04-26T16:42:10.128949+0000","last_active":"2024-04-26T16:42:38.457194+0000","last_peered":"2024-04-26T16:42:38.457194+0000","last_clean":"2024-04-26T16:42:38.457194+0000","last_became_active":"2024-04-26T16:38:46.971719+0000","last_became_peered":"2024-04-26T16:38:46.971719+0000","last_unstale":"2024-04-26T16:42:38.457194+0000","last_undegraded":"2024-04-26T16:42:38.457194+0000","last_fullsized":"2024-04-26T16:42:38.457194+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:10.128732+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:42:10.128732+0000","last_clean_scrub_stamp":"2024-04-26T16:42:10.128732+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.726045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1679,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:59.158002+0000","last_change":"2024-04-26T16:42:59.158057+0000","last_active":"2024-04-26T16:42:59.158002+0000","last_peered":"2024-04-26T16:42:59.158002+0000","last_clean":"2024-04-26T16:42:59.158002+0000","last_became_active":"2024-04-26T16:35:25.390196+0000","last_became_peered":"2024-04-26T16:35:25.390196+0000","last_unstale":"2024-04-26T16:42:59.158002+0000","last_undegraded":"2024-04-26T16:42:59.158002+0000","last_fullsized":"2024-04-26T16:42:59.158002+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:59.157862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.157862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:03.795100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"907'20","reported_seq":2176,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:05.299449+0000","last_change":"2024-04-26T16:43:05.299518+0000","last_active":"2024-04-26T16:43:05.299449+0000","last_peered":"2024-04-26T16:43:05.299449+0000","last_clean":"2024-04-26T16:43:05.299449+0000","last_became_active":"2024-04-26T16:35:25.404413+0000","last_became_peered":"2024-04-26T16:35:25.404413+0000","last_unstale":"2024-04-26T16:43:05.299449+0000","last_undegraded":"2024-04-26T16:43:05.299449+0000","last_fullsized":"2024-04-26T16:43:05.299449+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'20","last_scrub_stamp":"2024-04-26T16:43:05.299269+0000","last_deep_scrub":"907'20","last_deep_scrub_stamp":"2024-04-26T16:40:17.078066+0000","last_clean_scrub_stamp":"2024-04-26T16:43:05.299269+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:34.269320+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"922'3","reported_seq":1863,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:54.181826+0000","last_change":"2024-04-26T16:42:54.181861+0000","last_active":"2024-04-26T16:42:54.181826+0000","last_peered":"2024-04-26T16:42:54.181826+0000","last_clean":"2024-04-26T16:42:54.181826+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:42:54.181826+0000","last_undegraded":"2024-04-26T16:42:54.181826+0000","last_fullsized":"2024-04-26T16:42:54.181826+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"922'3","last_scrub_stamp":"2024-04-26T16:42:54.181739+0000","last_deep_scrub":"922'3","last_deep_scrub_stamp":"2024-04-26T16:41:32.972348+0000","last_clean_scrub_stamp":"2024-04-26T16:42:54.181739+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.692136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"857'52","reported_seq":909,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272781+0000","last_change":"2024-04-26T16:42:00.059382+0000","last_active":"2024-04-26T16:43:00.272781+0000","last_peered":"2024-04-26T16:43:00.272781+0000","last_clean":"2024-04-26T16:43:00.272781+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:43:00.272781+0000","last_undegraded":"2024-04-26T16:43:00.272781+0000","last_fullsized":"2024-04-26T16:43:00.272781+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'52","last_scrub_stamp":"2024-04-26T16:42:00.059136+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:42:00.059136+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:24.646186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1839,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.926830+0000","last_change":"2024-04-26T16:42:02.988520+0000","last_active":"2024-04-26T16:42:37.926830+0000","last_peered":"2024-04-26T16:42:37.926830+0000","last_clean":"2024-04-26T16:42:37.926830+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:42:37.926830+0000","last_undegraded":"2024-04-26T16:42:37.926830+0000","last_fullsized":"2024-04-26T16:42:37.926830+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:02.988298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:42:02.988298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:08.581682+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1814,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:03.293619+0000","last_change":"2024-04-26T16:43:03.293681+0000","last_active":"2024-04-26T16:43:03.293619+0000","last_peered":"2024-04-26T16:43:03.293619+0000","last_clean":"2024-04-26T16:43:03.293619+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:43:03.293619+0000","last_undegraded":"2024-04-26T16:43:03.293619+0000","last_fullsized":"2024-04-26T16:43:03.293619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:03.293464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:56.011726+0000","last_clean_scrub_stamp":"2024-04-26T16:43:03.293464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:15.447055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1669,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.447932+0000","last_change":"2024-04-26T16:42:14.096605+0000","last_active":"2024-04-26T16:42:38.447932+0000","last_peered":"2024-04-26T16:42:38.447932+0000","last_clean":"2024-04-26T16:42:38.447932+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:42:38.447932+0000","last_undegraded":"2024-04-26T16:42:38.447932+0000","last_fullsized":"2024-04-26T16:42:38.447932+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:14.096352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:14.096352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.324217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1826,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.924915+0000","last_change":"2024-04-26T16:42:29.230773+0000","last_active":"2024-04-26T16:42:37.924915+0000","last_peered":"2024-04-26T16:42:37.924915+0000","last_clean":"2024-04-26T16:42:37.924915+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:42:37.924915+0000","last_undegraded":"2024-04-26T16:42:37.924915+0000","last_fullsized":"2024-04-26T16:42:37.924915+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:29.230451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:42:29.230451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.825701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"857'27","reported_seq":1843,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.452767+0000","last_change":"2024-04-26T16:42:24.358367+0000","last_active":"2024-04-26T16:42:38.452767+0000","last_peered":"2024-04-26T16:42:38.452767+0000","last_clean":"2024-04-26T16:42:38.452767+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:42:38.452767+0000","last_undegraded":"2024-04-26T16:42:38.452767+0000","last_fullsized":"2024-04-26T16:42:38.452767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:42:24.358083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:24.358083+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.171772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1858,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927227+0000","last_change":"2024-04-26T16:42:27.974961+0000","last_active":"2024-04-26T16:42:37.927227+0000","last_peered":"2024-04-26T16:42:37.927227+0000","last_clean":"2024-04-26T16:42:37.927227+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:42:37.927227+0000","last_undegraded":"2024-04-26T16:42:37.927227+0000","last_fullsized":"2024-04-26T16:42:37.927227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:42:27.974720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:27.974720+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.614473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1790,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448999+0000","last_change":"2024-04-26T16:41:59.907623+0000","last_active":"2024-04-26T16:42:38.448999+0000","last_peered":"2024-04-26T16:42:38.448999+0000","last_clean":"2024-04-26T16:42:38.448999+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:42:38.448999+0000","last_undegraded":"2024-04-26T16:42:38.448999+0000","last_fullsized":"2024-04-26T16:42:38.448999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:59.907452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:41.021797+0000","last_clean_scrub_stamp":"2024-04-26T16:41:59.907452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:24.017201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1668,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:42.234101+0000","last_change":"2024-04-26T16:42:42.234101+0000","last_active":"2024-04-26T16:42:42.234101+0000","last_peered":"2024-04-26T16:42:42.234101+0000","last_clean":"2024-04-26T16:42:42.234101+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:42:42.234101+0000","last_undegraded":"2024-04-26T16:42:42.234101+0000","last_fullsized":"2024-04-26T16:42:42.234101+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:42.233896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:42.233896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.276863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"857'27","reported_seq":1849,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:58.338732+0000","last_change":"2024-04-26T16:42:58.338869+0000","last_active":"2024-04-26T16:42:58.338732+0000","last_peered":"2024-04-26T16:42:58.338732+0000","last_clean":"2024-04-26T16:42:58.338732+0000","last_became_active":"2024-04-26T16:35:25.395993+0000","last_became_peered":"2024-04-26T16:35:25.395993+0000","last_unstale":"2024-04-26T16:42:58.338732+0000","last_undegraded":"2024-04-26T16:42:58.338732+0000","last_fullsized":"2024-04-26T16:42:58.338732+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:42:58.338541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:58.338541+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:15.479805+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1842,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:03.201975+0000","last_change":"2024-04-26T16:43:03.201975+0000","last_active":"2024-04-26T16:43:03.201975+0000","last_peered":"2024-04-26T16:43:03.201975+0000","last_clean":"2024-04-26T16:43:03.201975+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:43:03.201975+0000","last_undegraded":"2024-04-26T16:43:03.201975+0000","last_fullsized":"2024-04-26T16:43:03.201975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:03.201864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:43:03.201864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:20.965555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1793,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449611+0000","last_change":"2024-04-26T16:42:02.906333+0000","last_active":"2024-04-26T16:42:38.449611+0000","last_peered":"2024-04-26T16:42:38.449611+0000","last_clean":"2024-04-26T16:42:38.449611+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:42:38.449611+0000","last_undegraded":"2024-04-26T16:42:38.449611+0000","last_fullsized":"2024-04-26T16:42:38.449611+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:02.906110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:42:02.906110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:28.025610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1813,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927898+0000","last_change":"2024-04-26T16:41:42.148184+0000","last_active":"2024-04-26T16:42:37.927898+0000","last_peered":"2024-04-26T16:42:37.927898+0000","last_clean":"2024-04-26T16:42:37.927898+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:42:37.927898+0000","last_undegraded":"2024-04-26T16:42:37.927898+0000","last_fullsized":"2024-04-26T16:42:37.927898+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:42.147970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:23.952025+0000","last_clean_scrub_stamp":"2024-04-26T16:41:42.147970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:06.037646+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1679,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.924651+0000","last_change":"2024-04-26T16:42:01.034407+0000","last_active":"2024-04-26T16:42:37.924651+0000","last_peered":"2024-04-26T16:42:37.924651+0000","last_clean":"2024-04-26T16:42:37.924651+0000","last_became_active":"2024-04-26T16:35:25.399182+0000","last_became_peered":"2024-04-26T16:35:25.399182+0000","last_unstale":"2024-04-26T16:42:37.924651+0000","last_undegraded":"2024-04-26T16:42:37.924651+0000","last_fullsized":"2024-04-26T16:42:37.924651+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:01.034193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:01.034193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.517517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"893'14","reported_seq":1552,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.452609+0000","last_change":"2024-04-26T16:41:57.154447+0000","last_active":"2024-04-26T16:42:38.452609+0000","last_peered":"2024-04-26T16:42:38.452609+0000","last_clean":"2024-04-26T16:42:38.452609+0000","last_became_active":"2024-04-26T16:35:25.404028+0000","last_became_peered":"2024-04-26T16:35:25.404028+0000","last_unstale":"2024-04-26T16:42:38.452609+0000","last_undegraded":"2024-04-26T16:42:38.452609+0000","last_fullsized":"2024-04-26T16:42:38.452609+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'14","last_scrub_stamp":"2024-04-26T16:41:57.154219+0000","last_deep_scrub":"893'14","last_deep_scrub_stamp":"2024-04-26T16:40:37.041551+0000","last_clean_scrub_stamp":"2024-04-26T16:41:57.154219+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:04.032709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1395,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.454330+0000","last_change":"2024-04-26T16:42:16.181728+0000","last_active":"2024-04-26T16:42:38.454330+0000","last_peered":"2024-04-26T16:42:38.454330+0000","last_clean":"2024-04-26T16:42:38.454330+0000","last_became_active":"2024-04-26T16:35:25.403847+0000","last_became_peered":"2024-04-26T16:35:25.403847+0000","last_unstale":"2024-04-26T16:42:38.454330+0000","last_undegraded":"2024-04-26T16:42:38.454330+0000","last_fullsized":"2024-04-26T16:42:38.454330+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:16.181526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:42:16.181526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:43.662859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":748,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:01.102239+0000","last_change":"2024-04-26T16:43:01.102297+0000","last_active":"2024-04-26T16:43:01.102239+0000","last_peered":"2024-04-26T16:43:01.102239+0000","last_clean":"2024-04-26T16:43:01.102239+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:43:01.102239+0000","last_undegraded":"2024-04-26T16:43:01.102239+0000","last_fullsized":"2024-04-26T16:43:01.102239+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:01.102083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:43:01.102083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:05.943136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1790,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.453360+0000","last_change":"2024-04-26T16:41:45.460825+0000","last_active":"2024-04-26T16:42:38.453360+0000","last_peered":"2024-04-26T16:42:38.453360+0000","last_clean":"2024-04-26T16:42:38.453360+0000","last_became_active":"2024-04-26T16:35:25.379432+0000","last_became_peered":"2024-04-26T16:35:25.379432+0000","last_unstale":"2024-04-26T16:42:38.453360+0000","last_undegraded":"2024-04-26T16:42:38.453360+0000","last_fullsized":"2024-04-26T16:42:38.453360+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:45.460605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:41:45.460605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.258841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1674,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925968+0000","last_change":"2024-04-26T16:42:06.759963+0000","last_active":"2024-04-26T16:42:37.925968+0000","last_peered":"2024-04-26T16:42:37.925968+0000","last_clean":"2024-04-26T16:42:37.925968+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:42:37.925968+0000","last_undegraded":"2024-04-26T16:42:37.925968+0000","last_fullsized":"2024-04-26T16:42:37.925968+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:06.759748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:06.759748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:25.209723+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"931'32","reported_seq":1862,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:55.366181+0000","last_change":"2024-04-26T16:42:55.366286+0000","last_active":"2024-04-26T16:42:55.366181+0000","last_peered":"2024-04-26T16:42:55.366181+0000","last_clean":"2024-04-26T16:42:55.366181+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:42:55.366181+0000","last_undegraded":"2024-04-26T16:42:55.366181+0000","last_fullsized":"2024-04-26T16:42:55.366181+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'32","last_scrub_stamp":"2024-04-26T16:42:55.366026+0000","last_deep_scrub":"931'32","last_deep_scrub_stamp":"2024-04-26T16:42:55.366026+0000","last_clean_scrub_stamp":"2024-04-26T16:42:55.366026+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.663187+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":748,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451166+0000","last_change":"2024-04-26T16:42:36.117791+0000","last_active":"2024-04-26T16:42:38.451166+0000","last_peered":"2024-04-26T16:42:38.451166+0000","last_clean":"2024-04-26T16:42:38.451166+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:42:38.451166+0000","last_undegraded":"2024-04-26T16:42:38.451166+0000","last_fullsized":"2024-04-26T16:42:38.451166+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:36.117506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:00.083131+0000","last_clean_scrub_stamp":"2024-04-26T16:42:36.117506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.369500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":876,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272751+0000","last_change":"2024-04-26T16:41:50.006070+0000","last_active":"2024-04-26T16:43:00.272751+0000","last_peered":"2024-04-26T16:43:00.272751+0000","last_clean":"2024-04-26T16:43:00.272751+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:43:00.272751+0000","last_undegraded":"2024-04-26T16:43:00.272751+0000","last_fullsized":"2024-04-26T16:43:00.272751+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:41:50.005835+0000","last_deep_scrub":"585'14","last_deep_scrub_stamp":"2024-04-26T16:39:10.124243+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.005835+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:16.684246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1804,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:47.190115+0000","last_change":"2024-04-26T16:42:47.190180+0000","last_active":"2024-04-26T16:42:47.190115+0000","last_peered":"2024-04-26T16:42:47.190115+0000","last_clean":"2024-04-26T16:42:47.190115+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:42:47.190115+0000","last_undegraded":"2024-04-26T16:42:47.190115+0000","last_fullsized":"2024-04-26T16:42:47.190115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:47.189962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:47.189962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:08.768493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":748,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:59.149752+0000","last_change":"2024-04-26T16:42:59.149815+0000","last_active":"2024-04-26T16:42:59.149752+0000","last_peered":"2024-04-26T16:42:59.149752+0000","last_clean":"2024-04-26T16:42:59.149752+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:42:59.149752+0000","last_undegraded":"2024-04-26T16:42:59.149752+0000","last_fullsized":"2024-04-26T16:42:59.149752+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:59.149587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.149587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:02.335080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"857'25","reported_seq":1850,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.447330+0000","last_change":"2024-04-26T16:42:21.117872+0000","last_active":"2024-04-26T16:42:38.447330+0000","last_peered":"2024-04-26T16:42:38.447330+0000","last_clean":"2024-04-26T16:42:38.447330+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:42:38.447330+0000","last_undegraded":"2024-04-26T16:42:38.447330+0000","last_fullsized":"2024-04-26T16:42:38.447330+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:42:21.117739+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.117739+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.191760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1824,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:02.339179+0000","last_change":"2024-04-26T16:43:02.339237+0000","last_active":"2024-04-26T16:43:02.339179+0000","last_peered":"2024-04-26T16:43:02.339179+0000","last_clean":"2024-04-26T16:43:02.339179+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:43:02.339179+0000","last_undegraded":"2024-04-26T16:43:02.339179+0000","last_fullsized":"2024-04-26T16:43:02.339179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:02.338995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.338995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:32.216415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"928'12","reported_seq":2112,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.273096+0000","last_change":"2024-04-26T16:42:25.031164+0000","last_active":"2024-04-26T16:43:00.273096+0000","last_peered":"2024-04-26T16:43:00.273096+0000","last_clean":"2024-04-26T16:43:00.273096+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:43:00.273096+0000","last_undegraded":"2024-04-26T16:43:00.273096+0000","last_fullsized":"2024-04-26T16:43:00.273096+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'12","last_scrub_stamp":"2024-04-26T16:42:25.030750+0000","last_deep_scrub":"928'12","last_deep_scrub_stamp":"2024-04-26T16:42:25.030750+0000","last_clean_scrub_stamp":"2024-04-26T16:42:25.030750+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:35.406231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1833,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.928192+0000","last_change":"2024-04-26T16:42:05.717465+0000","last_active":"2024-04-26T16:42:37.928192+0000","last_peered":"2024-04-26T16:42:37.928192+0000","last_clean":"2024-04-26T16:42:37.928192+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:42:37.928192+0000","last_undegraded":"2024-04-26T16:42:37.928192+0000","last_fullsized":"2024-04-26T16:42:37.928192+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:05.717242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:05.717242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:15.409405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1635,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.456883+0000","last_change":"2024-04-26T16:41:18.287028+0000","last_active":"2024-04-26T16:42:38.456883+0000","last_peered":"2024-04-26T16:42:38.456883+0000","last_clean":"2024-04-26T16:42:38.456883+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:42:38.456883+0000","last_undegraded":"2024-04-26T16:42:38.456883+0000","last_fullsized":"2024-04-26T16:42:38.456883+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:18.286808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:18.286808+0000","last_clean_scrub_stamp":"2024-04-26T16:41:18.286808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:46.491708+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"857'25","reported_seq":1834,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.456711+0000","last_change":"2024-04-26T16:42:20.372382+0000","last_active":"2024-04-26T16:42:38.456711+0000","last_peered":"2024-04-26T16:42:38.456711+0000","last_clean":"2024-04-26T16:42:38.456711+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:42:38.456711+0000","last_undegraded":"2024-04-26T16:42:38.456711+0000","last_fullsized":"2024-04-26T16:42:38.456711+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:42:20.372145+0000","last_deep_scrub":"857'25","last_deep_scrub_stamp":"2024-04-26T16:39:29.951322+0000","last_clean_scrub_stamp":"2024-04-26T16:42:20.372145+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.006718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1477,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:53.228843+0000","last_change":"2024-04-26T16:42:53.228898+0000","last_active":"2024-04-26T16:42:53.228843+0000","last_peered":"2024-04-26T16:42:53.228843+0000","last_clean":"2024-04-26T16:42:53.228843+0000","last_became_active":"2024-04-26T16:35:25.401229+0000","last_became_peered":"2024-04-26T16:35:25.401229+0000","last_unstale":"2024-04-26T16:42:53.228843+0000","last_undegraded":"2024-04-26T16:42:53.228843+0000","last_fullsized":"2024-04-26T16:42:53.228843+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.228700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:25.117353+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.228700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:03.005174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1809,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.585221+0000","last_change":"2024-04-26T16:43:00.585299+0000","last_active":"2024-04-26T16:43:00.585221+0000","last_peered":"2024-04-26T16:43:00.585221+0000","last_clean":"2024-04-26T16:43:00.585221+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:43:00.585221+0000","last_undegraded":"2024-04-26T16:43:00.585221+0000","last_fullsized":"2024-04-26T16:43:00.585221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:00.585093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:43:00.585093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:18.901570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1833,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.446946+0000","last_change":"2024-04-26T16:41:58.032603+0000","last_active":"2024-04-26T16:42:38.446946+0000","last_peered":"2024-04-26T16:42:38.446946+0000","last_clean":"2024-04-26T16:42:38.446946+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:42:38.446946+0000","last_undegraded":"2024-04-26T16:42:38.446946+0000","last_fullsized":"2024-04-26T16:42:38.446946+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:58.032373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:20.888123+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.032373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:14.630750+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1294,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:01.181886+0000","last_change":"2024-04-26T16:43:01.181943+0000","last_active":"2024-04-26T16:43:01.181886+0000","last_peered":"2024-04-26T16:43:01.181886+0000","last_clean":"2024-04-26T16:43:01.181886+0000","last_became_active":"2024-04-26T16:35:25.409902+0000","last_became_peered":"2024-04-26T16:35:25.409902+0000","last_unstale":"2024-04-26T16:43:01.181886+0000","last_undegraded":"2024-04-26T16:43:01.181886+0000","last_fullsized":"2024-04-26T16:43:01.181886+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:01.181727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:43:01.181727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.322241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"890'35","reported_seq":1889,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:03.882139+0000","last_change":"2024-04-26T16:43:03.882214+0000","last_active":"2024-04-26T16:43:03.882139+0000","last_peered":"2024-04-26T16:43:03.882139+0000","last_clean":"2024-04-26T16:43:03.882139+0000","last_became_active":"2024-04-26T16:35:25.383698+0000","last_became_peered":"2024-04-26T16:35:25.383698+0000","last_unstale":"2024-04-26T16:43:03.882139+0000","last_undegraded":"2024-04-26T16:43:03.882139+0000","last_fullsized":"2024-04-26T16:43:03.882139+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"890'35","last_scrub_stamp":"2024-04-26T16:43:03.881981+0000","last_deep_scrub":"890'35","last_deep_scrub_stamp":"2024-04-26T16:38:58.004208+0000","last_clean_scrub_stamp":"2024-04-26T16:43:03.881981+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.857115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1344,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:58.146607+0000","last_change":"2024-04-26T16:42:58.146712+0000","last_active":"2024-04-26T16:42:58.146607+0000","last_peered":"2024-04-26T16:42:58.146607+0000","last_clean":"2024-04-26T16:42:58.146607+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:42:58.146607+0000","last_undegraded":"2024-04-26T16:42:58.146607+0000","last_fullsized":"2024-04-26T16:42:58.146607+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:58.146358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:30.992466+0000","last_clean_scrub_stamp":"2024-04-26T16:42:58.146358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.823685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1833,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927557+0000","last_change":"2024-04-26T16:42:37.131685+0000","last_active":"2024-04-26T16:42:37.927557+0000","last_peered":"2024-04-26T16:42:37.927557+0000","last_clean":"2024-04-26T16:42:37.927557+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:42:37.927557+0000","last_undegraded":"2024-04-26T16:42:37.927557+0000","last_fullsized":"2024-04-26T16:42:37.927557+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:37.131468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:05.286233+0000","last_clean_scrub_stamp":"2024-04-26T16:42:37.131468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:39.833142+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1801,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448957+0000","last_change":"2024-04-26T16:41:56.000192+0000","last_active":"2024-04-26T16:42:38.448957+0000","last_peered":"2024-04-26T16:42:38.448957+0000","last_clean":"2024-04-26T16:42:38.448957+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:42:38.448957+0000","last_undegraded":"2024-04-26T16:42:38.448957+0000","last_fullsized":"2024-04-26T16:42:38.448957+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:55.999984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:41:55.999984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.491430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1678,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:53.231830+0000","last_change":"2024-04-26T16:42:53.231887+0000","last_active":"2024-04-26T16:42:53.231830+0000","last_peered":"2024-04-26T16:42:53.231830+0000","last_clean":"2024-04-26T16:42:53.231830+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:42:53.231830+0000","last_undegraded":"2024-04-26T16:42:53.231830+0000","last_fullsized":"2024-04-26T16:42:53.231830+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.231667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.231667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.628426+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"857'18","reported_seq":1892,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:58.258955+0000","last_change":"2024-04-26T16:42:58.259034+0000","last_active":"2024-04-26T16:42:58.258955+0000","last_peered":"2024-04-26T16:42:58.258955+0000","last_clean":"2024-04-26T16:42:58.258955+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:42:58.258955+0000","last_undegraded":"2024-04-26T16:42:58.258955+0000","last_fullsized":"2024-04-26T16:42:58.258955+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'18","last_scrub_stamp":"2024-04-26T16:42:58.258801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:58.258801+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:25.915658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":741,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451970+0000","last_change":"2024-04-26T16:42:11.921015+0000","last_active":"2024-04-26T16:42:38.451970+0000","last_peered":"2024-04-26T16:42:38.451970+0000","last_clean":"2024-04-26T16:42:38.451970+0000","last_became_active":"2024-04-26T16:35:25.389184+0000","last_became_peered":"2024-04-26T16:35:25.389184+0000","last_unstale":"2024-04-26T16:42:38.451970+0000","last_undegraded":"2024-04-26T16:42:38.451970+0000","last_fullsized":"2024-04-26T16:42:38.451970+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:11.920711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:19.245611+0000","last_clean_scrub_stamp":"2024-04-26T16:42:11.920711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.586442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1435,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.453569+0000","last_change":"2024-04-26T16:42:05.165110+0000","last_active":"2024-04-26T16:42:38.453569+0000","last_peered":"2024-04-26T16:42:38.453569+0000","last_clean":"2024-04-26T16:42:38.453569+0000","last_became_active":"2024-04-26T16:35:25.405230+0000","last_became_peered":"2024-04-26T16:35:25.405230+0000","last_unstale":"2024-04-26T16:42:38.453569+0000","last_undegraded":"2024-04-26T16:42:38.453569+0000","last_fullsized":"2024-04-26T16:42:38.453569+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:05.164910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:10.272810+0000","last_clean_scrub_stamp":"2024-04-26T16:42:05.164910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:07.137356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1780,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:41.464028+0000","last_change":"2024-04-26T16:42:41.464140+0000","last_active":"2024-04-26T16:42:41.464028+0000","last_peered":"2024-04-26T16:42:41.464028+0000","last_clean":"2024-04-26T16:42:41.464028+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:42:41.464028+0000","last_undegraded":"2024-04-26T16:42:41.464028+0000","last_fullsized":"2024-04-26T16:42:41.464028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:41.463839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.463839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.059956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1655,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.453193+0000","last_change":"2024-04-26T16:42:08.417897+0000","last_active":"2024-04-26T16:42:38.453193+0000","last_peered":"2024-04-26T16:42:38.453193+0000","last_clean":"2024-04-26T16:42:38.453193+0000","last_became_active":"2024-04-26T16:35:25.394016+0000","last_became_peered":"2024-04-26T16:35:25.394016+0000","last_unstale":"2024-04-26T16:42:38.453193+0000","last_undegraded":"2024-04-26T16:42:38.453193+0000","last_fullsized":"2024-04-26T16:42:38.453193+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:08.417678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:10.917141+0000","last_clean_scrub_stamp":"2024-04-26T16:42:08.417678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:17.950229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"931'25","reported_seq":771,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:40.103954+0000","last_change":"2024-04-26T16:42:40.104037+0000","last_active":"2024-04-26T16:42:40.103954+0000","last_peered":"2024-04-26T16:42:40.103954+0000","last_clean":"2024-04-26T16:42:40.103954+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:42:40.103954+0000","last_undegraded":"2024-04-26T16:42:40.103954+0000","last_fullsized":"2024-04-26T16:42:40.103954+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'25","last_scrub_stamp":"2024-04-26T16:42:40.103796+0000","last_deep_scrub":"931'25","last_deep_scrub_stamp":"2024-04-26T16:42:40.103796+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.103796+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:57.427688+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1438,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.457736+0000","last_change":"2024-04-26T16:42:00.120392+0000","last_active":"2024-04-26T16:42:38.457736+0000","last_peered":"2024-04-26T16:42:38.457736+0000","last_clean":"2024-04-26T16:42:38.457736+0000","last_became_active":"2024-04-26T16:35:25.406825+0000","last_became_peered":"2024-04-26T16:35:25.406825+0000","last_unstale":"2024-04-26T16:42:38.457736+0000","last_undegraded":"2024-04-26T16:42:38.457736+0000","last_fullsized":"2024-04-26T16:42:38.457736+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:00.120185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:00.120185+0000","last_clean_scrub_stamp":"2024-04-26T16:42:00.120185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:16.717815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1794,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449472+0000","last_change":"2024-04-26T16:41:48.367735+0000","last_active":"2024-04-26T16:42:38.449472+0000","last_peered":"2024-04-26T16:42:38.449472+0000","last_clean":"2024-04-26T16:42:38.449472+0000","last_became_active":"2024-04-26T16:35:25.378783+0000","last_became_peered":"2024-04-26T16:35:25.378783+0000","last_unstale":"2024-04-26T16:42:38.449472+0000","last_undegraded":"2024-04-26T16:42:38.449472+0000","last_fullsized":"2024-04-26T16:42:38.449472+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:48.367505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:41:48.367505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:06.854081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1799,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449128+0000","last_change":"2024-04-26T16:41:58.955287+0000","last_active":"2024-04-26T16:42:38.449128+0000","last_peered":"2024-04-26T16:42:38.449128+0000","last_clean":"2024-04-26T16:42:38.449128+0000","last_became_active":"2024-04-26T16:35:25.410836+0000","last_became_peered":"2024-04-26T16:35:25.410836+0000","last_unstale":"2024-04-26T16:42:38.449128+0000","last_undegraded":"2024-04-26T16:42:38.449128+0000","last_fullsized":"2024-04-26T16:42:38.449128+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:58.954935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.954935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:00.473140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1677,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:48.207195+0000","last_change":"2024-04-26T16:42:48.207321+0000","last_active":"2024-04-26T16:42:48.207195+0000","last_peered":"2024-04-26T16:42:48.207195+0000","last_clean":"2024-04-26T16:42:48.207195+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:42:48.207195+0000","last_undegraded":"2024-04-26T16:42:48.207195+0000","last_fullsized":"2024-04-26T16:42:48.207195+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:48.206977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:45.163028+0000","last_clean_scrub_stamp":"2024-04-26T16:42:48.206977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:59.620325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"857'46","reported_seq":1901,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:57.163792+0000","last_change":"2024-04-26T16:42:57.163825+0000","last_active":"2024-04-26T16:42:57.163792+0000","last_peered":"2024-04-26T16:42:57.163792+0000","last_clean":"2024-04-26T16:42:57.163792+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:42:57.163792+0000","last_undegraded":"2024-04-26T16:42:57.163792+0000","last_fullsized":"2024-04-26T16:42:57.163792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:42:57.163723+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.163723+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:25.733581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":740,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448981+0000","last_change":"2024-04-26T16:41:58.033450+0000","last_active":"2024-04-26T16:42:38.448981+0000","last_peered":"2024-04-26T16:42:38.448981+0000","last_clean":"2024-04-26T16:42:38.448981+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:42:38.448981+0000","last_undegraded":"2024-04-26T16:42:38.448981+0000","last_fullsized":"2024-04-26T16:42:38.448981+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:58.033139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:34.980868+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.033139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:06.212103+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"747'16","reported_seq":1691,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.272817+0000","last_change":"2024-04-26T16:42:04.141842+0000","last_active":"2024-04-26T16:43:00.272817+0000","last_peered":"2024-04-26T16:43:00.272817+0000","last_clean":"2024-04-26T16:43:00.272817+0000","last_became_active":"2024-04-26T16:35:25.403416+0000","last_became_peered":"2024-04-26T16:35:25.403416+0000","last_unstale":"2024-04-26T16:43:00.272817+0000","last_undegraded":"2024-04-26T16:43:00.272817+0000","last_fullsized":"2024-04-26T16:43:00.272817+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:42:04.141601+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:42:04.141601+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:05.717054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"829'2","reported_seq":1803,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.452913+0000","last_change":"2024-04-26T16:42:09.418159+0000","last_active":"2024-04-26T16:42:38.452913+0000","last_peered":"2024-04-26T16:42:38.452913+0000","last_clean":"2024-04-26T16:42:38.452913+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:42:38.452913+0000","last_undegraded":"2024-04-26T16:42:38.452913+0000","last_fullsized":"2024-04-26T16:42:38.452913+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'2","last_scrub_stamp":"2024-04-26T16:42:09.417718+0000","last_deep_scrub":"829'2","last_deep_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_clean_scrub_stamp":"2024-04-26T16:42:09.417718+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:15.920861+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1670,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.447757+0000","last_change":"2024-04-26T16:41:50.066200+0000","last_active":"2024-04-26T16:42:38.447757+0000","last_peered":"2024-04-26T16:42:38.447757+0000","last_clean":"2024-04-26T16:42:38.447757+0000","last_became_active":"2024-04-26T16:35:25.409763+0000","last_became_peered":"2024-04-26T16:35:25.409763+0000","last_unstale":"2024-04-26T16:42:38.447757+0000","last_undegraded":"2024-04-26T16:42:38.447757+0000","last_fullsized":"2024-04-26T16:42:38.447757+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:50.065983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.065983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:14.539280+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"922'25","reported_seq":1843,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:55.716383+0000","last_change":"2024-04-26T16:42:55.716458+0000","last_active":"2024-04-26T16:42:55.716383+0000","last_peered":"2024-04-26T16:42:55.716383+0000","last_clean":"2024-04-26T16:42:55.716383+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:42:55.716383+0000","last_undegraded":"2024-04-26T16:42:55.716383+0000","last_fullsized":"2024-04-26T16:42:55.716383+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"922'25","last_scrub_stamp":"2024-04-26T16:42:55.716222+0000","last_deep_scrub":"922'25","last_deep_scrub_stamp":"2024-04-26T16:42:55.716222+0000","last_clean_scrub_stamp":"2024-04-26T16:42:55.716222+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:05.930413+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1813,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448382+0000","last_change":"2024-04-26T16:42:28.956311+0000","last_active":"2024-04-26T16:42:38.448382+0000","last_peered":"2024-04-26T16:42:38.448382+0000","last_clean":"2024-04-26T16:42:38.448382+0000","last_became_active":"2024-04-26T16:35:25.417030+0000","last_became_peered":"2024-04-26T16:35:25.417030+0000","last_unstale":"2024-04-26T16:42:38.448382+0000","last_undegraded":"2024-04-26T16:42:38.448382+0000","last_fullsized":"2024-04-26T16:42:38.448382+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:42:28.955957+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:41:02.038050+0000","last_clean_scrub_stamp":"2024-04-26T16:42:28.955957+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.682762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1826,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:55.162881+0000","last_change":"2024-04-26T16:42:55.162921+0000","last_active":"2024-04-26T16:42:55.162881+0000","last_peered":"2024-04-26T16:42:55.162881+0000","last_clean":"2024-04-26T16:42:55.162881+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:42:55.162881+0000","last_undegraded":"2024-04-26T16:42:55.162881+0000","last_fullsized":"2024-04-26T16:42:55.162881+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:55.162804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:22.872578+0000","last_clean_scrub_stamp":"2024-04-26T16:42:55.162804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:07.152764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1837,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.929647+0000","last_change":"2024-04-26T16:42:23.068496+0000","last_active":"2024-04-26T16:42:37.929647+0000","last_peered":"2024-04-26T16:42:37.929647+0000","last_clean":"2024-04-26T16:42:37.929647+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:42:37.929647+0000","last_undegraded":"2024-04-26T16:42:37.929647+0000","last_fullsized":"2024-04-26T16:42:37.929647+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:23.068390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:23.068390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:38.094123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":740,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449681+0000","last_change":"2024-04-26T16:42:22.975468+0000","last_active":"2024-04-26T16:42:38.449681+0000","last_peered":"2024-04-26T16:42:38.449681+0000","last_clean":"2024-04-26T16:42:38.449681+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:42:38.449681+0000","last_undegraded":"2024-04-26T16:42:38.449681+0000","last_fullsized":"2024-04-26T16:42:38.449681+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:22.975099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:22.975099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.791244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"857'11","reported_seq":1838,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:54.160342+0000","last_change":"2024-04-26T16:42:54.160426+0000","last_active":"2024-04-26T16:42:54.160342+0000","last_peered":"2024-04-26T16:42:54.160342+0000","last_clean":"2024-04-26T16:42:54.160342+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:42:54.160342+0000","last_undegraded":"2024-04-26T16:42:54.160342+0000","last_fullsized":"2024-04-26T16:42:54.160342+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'11","last_scrub_stamp":"2024-04-26T16:42:54.160177+0000","last_deep_scrub":"857'11","last_deep_scrub_stamp":"2024-04-26T16:42:54.160177+0000","last_clean_scrub_stamp":"2024-04-26T16:42:54.160177+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:55.628445+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1495,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.454743+0000","last_change":"2024-04-26T16:42:01.171043+0000","last_active":"2024-04-26T16:42:38.454743+0000","last_peered":"2024-04-26T16:42:38.454743+0000","last_clean":"2024-04-26T16:42:38.454743+0000","last_became_active":"2024-04-26T16:35:25.404997+0000","last_became_peered":"2024-04-26T16:35:25.404997+0000","last_unstale":"2024-04-26T16:42:38.454743+0000","last_undegraded":"2024-04-26T16:42:38.454743+0000","last_fullsized":"2024-04-26T16:42:38.454743+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:01.170844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:01.170844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.453064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1837,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:05.944221+0000","last_change":"2024-04-26T16:43:05.944284+0000","last_active":"2024-04-26T16:43:05.944221+0000","last_peered":"2024-04-26T16:43:05.944221+0000","last_clean":"2024-04-26T16:43:05.944221+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:43:05.944221+0000","last_undegraded":"2024-04-26T16:43:05.944221+0000","last_fullsized":"2024-04-26T16:43:05.944221+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:05.944031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:43:05.944031+0000","last_clean_scrub_stamp":"2024-04-26T16:43:05.944031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.663257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1845,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.927744+0000","last_change":"2024-04-26T16:42:08.758536+0000","last_active":"2024-04-26T16:42:37.927744+0000","last_peered":"2024-04-26T16:42:37.927744+0000","last_clean":"2024-04-26T16:42:37.927744+0000","last_became_active":"2024-04-26T16:35:25.383286+0000","last_became_peered":"2024-04-26T16:35:25.383286+0000","last_unstale":"2024-04-26T16:42:37.927744+0000","last_undegraded":"2024-04-26T16:42:37.927744+0000","last_fullsized":"2024-04-26T16:42:37.927744+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:08.758249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:51.035286+0000","last_clean_scrub_stamp":"2024-04-26T16:42:08.758249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:34.548556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1636,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.450359+0000","last_change":"2024-04-26T16:41:32.222039+0000","last_active":"2024-04-26T16:42:38.450359+0000","last_peered":"2024-04-26T16:42:38.450359+0000","last_clean":"2024-04-26T16:42:38.450359+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:42:38.450359+0000","last_undegraded":"2024-04-26T16:42:38.450359+0000","last_fullsized":"2024-04-26T16:42:38.450359+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:32.221829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:41:32.221829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:58.624430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"902'25","reported_seq":1867,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925910+0000","last_change":"2024-04-26T16:42:37.816736+0000","last_active":"2024-04-26T16:42:37.925910+0000","last_peered":"2024-04-26T16:42:37.925910+0000","last_clean":"2024-04-26T16:42:37.925910+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:42:37.925910+0000","last_undegraded":"2024-04-26T16:42:37.925910+0000","last_fullsized":"2024-04-26T16:42:37.925910+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'25","last_scrub_stamp":"2024-04-26T16:42:37.816385+0000","last_deep_scrub":"902'25","last_deep_scrub_stamp":"2024-04-26T16:39:49.818013+0000","last_clean_scrub_stamp":"2024-04-26T16:42:37.816385+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:43.671767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1811,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:41.912234+0000","last_change":"2024-04-26T16:42:41.912362+0000","last_active":"2024-04-26T16:42:41.912234+0000","last_peered":"2024-04-26T16:42:41.912234+0000","last_clean":"2024-04-26T16:42:41.912234+0000","last_became_active":"2024-04-26T16:35:25.392415+0000","last_became_peered":"2024-04-26T16:35:25.392415+0000","last_unstale":"2024-04-26T16:42:41.912234+0000","last_undegraded":"2024-04-26T16:42:41.912234+0000","last_fullsized":"2024-04-26T16:42:41.912234+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:41.911982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.911982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:50.543906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1825,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.926670+0000","last_change":"2024-04-26T16:42:31.880452+0000","last_active":"2024-04-26T16:42:37.926670+0000","last_peered":"2024-04-26T16:42:37.926670+0000","last_clean":"2024-04-26T16:42:37.926670+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:42:37.926670+0000","last_undegraded":"2024-04-26T16:42:37.926670+0000","last_fullsized":"2024-04-26T16:42:37.926670+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:31.880251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_clean_scrub_stamp":"2024-04-26T16:42:31.880251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:55.010235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1813,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449506+0000","last_change":"2024-04-26T16:42:11.099774+0000","last_active":"2024-04-26T16:42:38.449506+0000","last_peered":"2024-04-26T16:42:38.449506+0000","last_clean":"2024-04-26T16:42:38.449506+0000","last_became_active":"2024-04-26T16:35:25.410430+0000","last_became_peered":"2024-04-26T16:35:25.410430+0000","last_unstale":"2024-04-26T16:42:38.449506+0000","last_undegraded":"2024-04-26T16:42:38.449506+0000","last_fullsized":"2024-04-26T16:42:38.449506+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:11.099526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:11.099526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:32.370546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1677,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:52.229800+0000","last_change":"2024-04-26T16:42:52.229832+0000","last_active":"2024-04-26T16:42:52.229800+0000","last_peered":"2024-04-26T16:42:52.229800+0000","last_clean":"2024-04-26T16:42:52.229800+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:42:52.229800+0000","last_undegraded":"2024-04-26T16:42:52.229800+0000","last_fullsized":"2024-04-26T16:42:52.229800+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:52.229707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:40.169404+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.229707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:07.097155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"857'29","reported_seq":770,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.454513+0000","last_change":"2024-04-26T16:42:30.104346+0000","last_active":"2024-04-26T16:42:38.454513+0000","last_peered":"2024-04-26T16:42:38.454513+0000","last_clean":"2024-04-26T16:42:38.454513+0000","last_became_active":"2024-04-26T16:35:25.382813+0000","last_became_peered":"2024-04-26T16:35:25.382813+0000","last_unstale":"2024-04-26T16:42:38.454513+0000","last_undegraded":"2024-04-26T16:42:38.454513+0000","last_fullsized":"2024-04-26T16:42:38.454513+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'29","last_scrub_stamp":"2024-04-26T16:42:30.104089+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:42:30.104089+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.977065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1821,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451588+0000","last_change":"2024-04-26T16:42:13.426286+0000","last_active":"2024-04-26T16:42:38.451588+0000","last_peered":"2024-04-26T16:42:38.451588+0000","last_clean":"2024-04-26T16:42:38.451588+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:42:38.451588+0000","last_undegraded":"2024-04-26T16:42:38.451588+0000","last_fullsized":"2024-04-26T16:42:38.451588+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:13.425926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:27.994588+0000","last_clean_scrub_stamp":"2024-04-26T16:42:13.425926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:38.328874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":740,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:57.213311+0000","last_change":"2024-04-26T16:42:57.213375+0000","last_active":"2024-04-26T16:42:57.213311+0000","last_peered":"2024-04-26T16:42:57.213311+0000","last_clean":"2024-04-26T16:42:57.213311+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:42:57.213311+0000","last_undegraded":"2024-04-26T16:42:57.213311+0000","last_fullsized":"2024-04-26T16:42:57.213311+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:57.213120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.213120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:02.009393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1812,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:53.203186+0000","last_change":"2024-04-26T16:42:53.203249+0000","last_active":"2024-04-26T16:42:53.203186+0000","last_peered":"2024-04-26T16:42:53.203186+0000","last_clean":"2024-04-26T16:42:53.203186+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:42:53.203186+0000","last_undegraded":"2024-04-26T16:42:53.203186+0000","last_fullsized":"2024-04-26T16:42:53.203186+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.202986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.855045+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.202986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:15.824975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1274,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.456295+0000","last_change":"2024-04-26T16:42:18.219412+0000","last_active":"2024-04-26T16:42:38.456295+0000","last_peered":"2024-04-26T16:42:38.456295+0000","last_clean":"2024-04-26T16:42:38.456295+0000","last_became_active":"2024-04-26T16:35:25.401906+0000","last_became_peered":"2024-04-26T16:35:25.401906+0000","last_unstale":"2024-04-26T16:42:38.456295+0000","last_undegraded":"2024-04-26T16:42:38.456295+0000","last_fullsized":"2024-04-26T16:42:38.456295+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:18.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:18.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.345015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"923'46","reported_seq":792,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:45.143541+0000","last_change":"2024-04-26T16:42:45.143623+0000","last_active":"2024-04-26T16:42:45.143541+0000","last_peered":"2024-04-26T16:42:45.143541+0000","last_clean":"2024-04-26T16:42:45.143541+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:42:45.143541+0000","last_undegraded":"2024-04-26T16:42:45.143541+0000","last_fullsized":"2024-04-26T16:42:45.143541+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'46","last_scrub_stamp":"2024-04-26T16:42:45.143361+0000","last_deep_scrub":"923'46","last_deep_scrub_stamp":"2024-04-26T16:41:38.088859+0000","last_clean_scrub_stamp":"2024-04-26T16:42:45.143361+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.644961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1810,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.456455+0000","last_change":"2024-04-26T16:41:54.483826+0000","last_active":"2024-04-26T16:42:38.456455+0000","last_peered":"2024-04-26T16:42:38.456455+0000","last_clean":"2024-04-26T16:42:38.456455+0000","last_became_active":"2024-04-26T16:35:25.389963+0000","last_became_peered":"2024-04-26T16:35:25.389963+0000","last_unstale":"2024-04-26T16:42:38.456455+0000","last_undegraded":"2024-04-26T16:42:38.456455+0000","last_fullsized":"2024-04-26T16:42:38.456455+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:54.483615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:41:54.483615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:12.597424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1784,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.450952+0000","last_change":"2024-04-26T16:42:02.301779+0000","last_active":"2024-04-26T16:42:38.450952+0000","last_peered":"2024-04-26T16:42:38.450952+0000","last_clean":"2024-04-26T16:42:38.450952+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:42:38.450952+0000","last_undegraded":"2024-04-26T16:42:38.450952+0000","last_fullsized":"2024-04-26T16:42:38.450952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:02.301500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:02.301500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.509272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":752,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:43:00.106432+0000","last_change":"2024-04-26T16:43:00.106500+0000","last_active":"2024-04-26T16:43:00.106432+0000","last_peered":"2024-04-26T16:43:00.106432+0000","last_clean":"2024-04-26T16:43:00.106432+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:43:00.106432+0000","last_undegraded":"2024-04-26T16:43:00.106432+0000","last_fullsized":"2024-04-26T16:43:00.106432+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:43:00.106277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:43:00.106277+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:12.686057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1666,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451795+0000","last_change":"2024-04-26T16:41:58.420395+0000","last_active":"2024-04-26T16:42:38.451795+0000","last_peered":"2024-04-26T16:42:38.451795+0000","last_clean":"2024-04-26T16:42:38.451795+0000","last_became_active":"2024-04-26T16:35:25.388177+0000","last_became_peered":"2024-04-26T16:35:25.388177+0000","last_unstale":"2024-04-26T16:42:38.451795+0000","last_undegraded":"2024-04-26T16:42:38.451795+0000","last_fullsized":"2024-04-26T16:42:38.451795+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:58.420180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.420180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:12.936590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"895'32","reported_seq":1915,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:48.152136+0000","last_change":"2024-04-26T16:42:48.152221+0000","last_active":"2024-04-26T16:42:48.152136+0000","last_peered":"2024-04-26T16:42:48.152136+0000","last_clean":"2024-04-26T16:42:48.152136+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:42:48.152136+0000","last_undegraded":"2024-04-26T16:42:48.152136+0000","last_fullsized":"2024-04-26T16:42:48.152136+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'32","last_scrub_stamp":"2024-04-26T16:42:48.151977+0000","last_deep_scrub":"895'32","last_deep_scrub_stamp":"2024-04-26T16:39:30.003486+0000","last_clean_scrub_stamp":"2024-04-26T16:42:48.151977+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.631690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1823,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.449272+0000","last_change":"2024-04-26T16:41:39.038458+0000","last_active":"2024-04-26T16:42:38.449272+0000","last_peered":"2024-04-26T16:42:38.449272+0000","last_clean":"2024-04-26T16:42:38.449272+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:42:38.449272+0000","last_undegraded":"2024-04-26T16:42:38.449272+0000","last_fullsized":"2024-04-26T16:42:38.449272+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:39.038243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:39.038243+0000","last_clean_scrub_stamp":"2024-04-26T16:41:39.038243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:06.082571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1841,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:48.094286+0000","last_change":"2024-04-26T16:42:48.094343+0000","last_active":"2024-04-26T16:42:48.094286+0000","last_peered":"2024-04-26T16:42:48.094286+0000","last_clean":"2024-04-26T16:42:48.094286+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:42:48.094286+0000","last_undegraded":"2024-04-26T16:42:48.094286+0000","last_fullsized":"2024-04-26T16:42:48.094286+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:48.094125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:14.727355+0000","last_clean_scrub_stamp":"2024-04-26T16:42:48.094125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:13.646575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1833,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925579+0000","last_change":"2024-04-26T16:42:21.884006+0000","last_active":"2024-04-26T16:42:37.925579+0000","last_peered":"2024-04-26T16:42:37.925579+0000","last_clean":"2024-04-26T16:42:37.925579+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:42:37.925579+0000","last_undegraded":"2024-04-26T16:42:37.925579+0000","last_fullsized":"2024-04-26T16:42:37.925579+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:21.883800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:27.837864+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.883800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:25.296934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":740,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.451227+0000","last_change":"2024-04-26T16:42:04.022526+0000","last_active":"2024-04-26T16:42:38.451227+0000","last_peered":"2024-04-26T16:42:38.451227+0000","last_clean":"2024-04-26T16:42:38.451227+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:42:38.451227+0000","last_undegraded":"2024-04-26T16:42:38.451227+0000","last_fullsized":"2024-04-26T16:42:38.451227+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:04.022275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:34.130566+0000","last_clean_scrub_stamp":"2024-04-26T16:42:04.022275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.463543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"857'30","reported_seq":1862,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:52.767762+0000","last_change":"2024-04-26T16:42:52.767797+0000","last_active":"2024-04-26T16:42:52.767762+0000","last_peered":"2024-04-26T16:42:52.767762+0000","last_clean":"2024-04-26T16:42:52.767762+0000","last_became_active":"2024-04-26T16:35:25.399680+0000","last_became_peered":"2024-04-26T16:35:25.399680+0000","last_unstale":"2024-04-26T16:42:52.767762+0000","last_undegraded":"2024-04-26T16:42:52.767762+0000","last_fullsized":"2024-04-26T16:42:52.767762+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:42:52.767679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.767679+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:14.193285+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1826,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:38.448515+0000","last_change":"2024-04-26T16:42:20.091221+0000","last_active":"2024-04-26T16:42:38.448515+0000","last_peered":"2024-04-26T16:42:38.448515+0000","last_clean":"2024-04-26T16:42:38.448515+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:42:38.448515+0000","last_undegraded":"2024-04-26T16:42:38.448515+0000","last_fullsized":"2024-04-26T16:42:38.448515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:20.090970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:42:20.090970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:35.757867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1827,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:37.925070+0000","last_change":"2024-04-26T16:41:47.754441+0000","last_active":"2024-04-26T16:42:37.925070+0000","last_peered":"2024-04-26T16:42:37.925070+0000","last_clean":"2024-04-26T16:42:37.925070+0000","last_became_active":"2024-04-26T16:35:25.396616+0000","last_became_peered":"2024-04-26T16:35:25.396616+0000","last_unstale":"2024-04-26T16:42:37.925070+0000","last_undegraded":"2024-04-26T16:42:37.925070+0000","last_fullsized":"2024-04-26T16:42:37.925070+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:47.754123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:41:47.754123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:12.999182+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1799,"reported_epoch":931,"state":"active+clean","last_fresh":"2024-04-26T16:42:46.880190+0000","last_change":"2024-04-26T16:42:46.880236+0000","last_active":"2024-04-26T16:42:46.880190+0000","last_peered":"2024-04-26T16:42:46.880190+0000","last_clean":"2024-04-26T16:42:46.880190+0000","last_became_active":"2024-04-26T16:35:25.392061+0000","last_became_peered":"2024-04-26T16:35:25.392061+0000","last_unstale":"2024-04-26T16:42:46.880190+0000","last_undegraded":"2024-04-26T16:42:46.880190+0000","last_fullsized":"2024-04-26T16:42:46.880190+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:46.880057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:42:46.880057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.062162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},"log_size":1128,"ondisk_log_size":1128,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1514,"num_read_kb":1509,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743743,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114380,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630772,"statfs":{"total":95995035648,"available":95877910528,"internally_reserved":0,"allocated":2756608,"data_stored":995847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":746,"seq":3204045602911,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39916,"kb_used_data":2340,"kb_used_omap":23,"kb_used_meta":37544,"kb_avail":93705236,"statfs":{"total":95995035648,"available":95954161664,"internally_reserved":0,"allocated":2396160,"data_stored":665401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23760,"internal_metadata":38445872},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575229,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101568,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93643584,"statfs":{"total":95995035648,"available":95891030016,"internally_reserved":0,"allocated":2416640,"data_stored":666733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738759,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109896,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635256,"statfs":{"total":95995035648,"available":95882502144,"internally_reserved":0,"allocated":2424832,"data_stored":667595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411698,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105972,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639180,"statfs":{"total":95995035648,"available":95886520320,"internally_reserved":0,"allocated":2666496,"data_stored":928687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084647,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132596,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93612556,"statfs":{"total":95995035648,"available":95859257344,"internally_reserved":0,"allocated":2994176,"data_stored":1256169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":132774344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855709,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109900,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635252,"statfs":{"total":95995035648,"available":95882498048,"internally_reserved":0,"allocated":2428928,"data_stored":666557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463251,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92404,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93652748,"statfs":{"total":95995035648,"available":95900413952,"internally_reserved":0,"allocated":2994176,"data_stored":1256179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:43:12.587 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-26T16:43:13.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[29234]: audit 2024-04-26T16:43:11.651345+0000 mgr.y (mgr.24401) 977 : audit [DBG] from='client.41996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[29234]: audit 2024-04-26T16:43:11.651345+0000 mgr.y (mgr.24401) 977 : audit [DBG] from='client.41996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[29234]: cluster 2024-04-26T16:43:11.683123+0000 osd.3 (osd.3) 687 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[29234]: cluster 2024-04-26T16:43:11.683123+0000 osd.3 (osd.3) 687 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[29234]: cluster 2024-04-26T16:43:11.686265+0000 osd.3 (osd.3) 688 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[29234]: cluster 2024-04-26T16:43:11.686265+0000 osd.3 (osd.3) 688 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[21463]: audit 2024-04-26T16:43:11.651345+0000 mgr.y (mgr.24401) 977 : audit [DBG] from='client.41996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[21463]: audit 2024-04-26T16:43:11.651345+0000 mgr.y (mgr.24401) 977 : audit [DBG] from='client.41996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[21463]: cluster 2024-04-26T16:43:11.683123+0000 osd.3 (osd.3) 687 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:43:13.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[21463]: cluster 2024-04-26T16:43:11.683123+0000 osd.3 (osd.3) 687 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:43:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[21463]: cluster 2024-04-26T16:43:11.686265+0000 osd.3 (osd.3) 688 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:43:13.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:12 smithi050 bash[21463]: cluster 2024-04-26T16:43:11.686265+0000 osd.3 (osd.3) 688 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:43:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:12 smithi190 bash[23868]: audit 2024-04-26T16:43:11.651345+0000 mgr.y (mgr.24401) 977 : audit [DBG] from='client.41996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:12 smithi190 bash[23868]: audit 2024-04-26T16:43:11.651345+0000 mgr.y (mgr.24401) 977 : audit [DBG] from='client.41996 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:13.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:12 smithi190 bash[23868]: cluster 2024-04-26T16:43:11.683123+0000 osd.3 (osd.3) 687 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:43:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:12 smithi190 bash[23868]: cluster 2024-04-26T16:43:11.683123+0000 osd.3 (osd.3) 687 : cluster [DBG] 2.1e scrub starts 2024-04-26T16:43:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:12 smithi190 bash[23868]: cluster 2024-04-26T16:43:11.686265+0000 osd.3 (osd.3) 688 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:43:13.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:12 smithi190 bash[23868]: cluster 2024-04-26T16:43:11.686265+0000 osd.3 (osd.3) 688 : cluster [DBG] 2.1e scrub ok 2024-04-26T16:43:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.152848+0000 mgr.y (mgr.24401) 978 : cluster [DBG] pgmap v1577: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.152848+0000 mgr.y (mgr.24401) 978 : cluster [DBG] pgmap v1577: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.563370+0000 osd.7 (osd.7) 670 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:43:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.563370+0000 osd.7 (osd.7) 670 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:43:14.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.565968+0000 osd.7 (osd.7) 671 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.565968+0000 osd.7 (osd.7) 671 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.997985+0000 osd.4 (osd.4) 571 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:12.997985+0000 osd.4 (osd.4) 571 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.000241+0000 osd.4 (osd.4) 572 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.000241+0000 osd.4 (osd.4) 572 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: audit 2024-04-26T16:43:13.679214+0000 mon.c (mon.2) 961 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[21463]: audit 2024-04-26T16:43:13.679214+0000 mon.c (mon.2) 961 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.152848+0000 mgr.y (mgr.24401) 978 : cluster [DBG] pgmap v1577: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.152848+0000 mgr.y (mgr.24401) 978 : cluster [DBG] pgmap v1577: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:14.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.563370+0000 osd.7 (osd.7) 670 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.563370+0000 osd.7 (osd.7) 670 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.565968+0000 osd.7 (osd.7) 671 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.565968+0000 osd.7 (osd.7) 671 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.997985+0000 osd.4 (osd.4) 571 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:12.997985+0000 osd.4 (osd.4) 571 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.000241+0000 osd.4 (osd.4) 572 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.000241+0000 osd.4 (osd.4) 572 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: audit 2024-04-26T16:43:13.679214+0000 mon.c (mon.2) 961 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:14.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:13 smithi050 bash[29234]: audit 2024-04-26T16:43:13.679214+0000 mon.c (mon.2) 961 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:14.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.152848+0000 mgr.y (mgr.24401) 978 : cluster [DBG] pgmap v1577: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.152848+0000 mgr.y (mgr.24401) 978 : cluster [DBG] pgmap v1577: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.563370+0000 osd.7 (osd.7) 670 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.563370+0000 osd.7 (osd.7) 670 : cluster [DBG] 6.15 scrub starts 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.565968+0000 osd.7 (osd.7) 671 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.565968+0000 osd.7 (osd.7) 671 : cluster [DBG] 6.15 scrub ok 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.997985+0000 osd.4 (osd.4) 571 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:12.997985+0000 osd.4 (osd.4) 571 : cluster [DBG] 5.1 scrub starts 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.000241+0000 osd.4 (osd.4) 572 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.000241+0000 osd.4 (osd.4) 572 : cluster [DBG] 5.1 scrub ok 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: audit 2024-04-26T16:43:13.679214+0000 mon.c (mon.2) 961 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:14.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:13 smithi190 bash[23868]: audit 2024-04-26T16:43:13.679214+0000 mon.c (mon.2) 961 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:14.877 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:43:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:43:15.129 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.587909+0000 osd.7 (osd.7) 672 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.587909+0000 osd.7 (osd.7) 672 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.590286+0000 osd.7 (osd.7) 673 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.590286+0000 osd.7 (osd.7) 673 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.978508+0000 osd.4 (osd.4) 573 : cluster [DBG] 5.18 deep-scrub starts 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.978508+0000 osd.4 (osd.4) 573 : cluster [DBG] 5.18 deep-scrub starts 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.981026+0000 osd.4 (osd.4) 574 : cluster [DBG] 5.18 deep-scrub ok 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:13.981026+0000 osd.4 (osd.4) 574 : cluster [DBG] 5.18 deep-scrub ok 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:14.153815+0000 mgr.y (mgr.24401) 979 : cluster [DBG] pgmap v1578: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[29234]: cluster 2024-04-26T16:43:14.153815+0000 mgr.y (mgr.24401) 979 : cluster [DBG] pgmap v1578: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:15.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.587909+0000 osd.7 (osd.7) 672 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.587909+0000 osd.7 (osd.7) 672 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.590286+0000 osd.7 (osd.7) 673 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.590286+0000 osd.7 (osd.7) 673 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.978508+0000 osd.4 (osd.4) 573 : cluster [DBG] 5.18 deep-scrub starts 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.978508+0000 osd.4 (osd.4) 573 : cluster [DBG] 5.18 deep-scrub starts 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.981026+0000 osd.4 (osd.4) 574 : cluster [DBG] 5.18 deep-scrub ok 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:13.981026+0000 osd.4 (osd.4) 574 : cluster [DBG] 5.18 deep-scrub ok 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:14.153815+0000 mgr.y (mgr.24401) 979 : cluster [DBG] pgmap v1578: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:15.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:14 smithi050 bash[21463]: cluster 2024-04-26T16:43:14.153815+0000 mgr.y (mgr.24401) 979 : cluster [DBG] pgmap v1578: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:15.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.587909+0000 osd.7 (osd.7) 672 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.587909+0000 osd.7 (osd.7) 672 : cluster [DBG] 5.4 scrub starts 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.590286+0000 osd.7 (osd.7) 673 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.590286+0000 osd.7 (osd.7) 673 : cluster [DBG] 5.4 scrub ok 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.978508+0000 osd.4 (osd.4) 573 : cluster [DBG] 5.18 deep-scrub starts 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.978508+0000 osd.4 (osd.4) 573 : cluster [DBG] 5.18 deep-scrub starts 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.981026+0000 osd.4 (osd.4) 574 : cluster [DBG] 5.18 deep-scrub ok 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:13.981026+0000 osd.4 (osd.4) 574 : cluster [DBG] 5.18 deep-scrub ok 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:14.153815+0000 mgr.y (mgr.24401) 979 : cluster [DBG] pgmap v1578: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:15.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:14 smithi190 bash[23868]: cluster 2024-04-26T16:43:14.153815+0000 mgr.y (mgr.24401) 979 : cluster [DBG] pgmap v1578: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-26T16:43:16.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:15 smithi190 bash[23868]: cluster 2024-04-26T16:43:14.892585+0000 osd.4 (osd.4) 575 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:15 smithi190 bash[23868]: cluster 2024-04-26T16:43:14.892585+0000 osd.4 (osd.4) 575 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:15 smithi190 bash[23868]: cluster 2024-04-26T16:43:14.893011+0000 osd.4 (osd.4) 576 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:16.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:15 smithi190 bash[23868]: cluster 2024-04-26T16:43:14.893011+0000 osd.4 (osd.4) 576 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[21463]: cluster 2024-04-26T16:43:14.892585+0000 osd.4 (osd.4) 575 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:16.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[21463]: cluster 2024-04-26T16:43:14.892585+0000 osd.4 (osd.4) 575 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:16.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[21463]: cluster 2024-04-26T16:43:14.893011+0000 osd.4 (osd.4) 576 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:16.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[21463]: cluster 2024-04-26T16:43:14.893011+0000 osd.4 (osd.4) 576 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[29234]: cluster 2024-04-26T16:43:14.892585+0000 osd.4 (osd.4) 575 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[29234]: cluster 2024-04-26T16:43:14.892585+0000 osd.4 (osd.4) 575 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[29234]: cluster 2024-04-26T16:43:14.893011+0000 osd.4 (osd.4) 576 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:16.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:15 smithi050 bash[29234]: cluster 2024-04-26T16:43:14.893011+0000 osd.4 (osd.4) 576 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:17.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.333147+0000 osd.0 (osd.0) 639 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.333147+0000 osd.0 (osd.0) 639 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.335288+0000 osd.0 (osd.0) 640 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.335288+0000 osd.0 (osd.0) 640 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.625043+0000 osd.7 (osd.7) 674 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.625043+0000 osd.7 (osd.7) 674 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.627600+0000 osd.7 (osd.7) 675 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.627600+0000 osd.7 (osd.7) 675 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.892274+0000 mon.a (mon.0) 4077 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:15.892274+0000 mon.a (mon.0) 4077 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:16.105131+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:16.105131+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:16.108056+0000 osd.5 (osd.5) 541 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:16.108056+0000 osd.5 (osd.5) 541 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:43:17.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:16.154625+0000 mgr.y (mgr.24401) 980 : cluster [DBG] pgmap v1580: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:17.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:16 smithi190 bash[23868]: cluster 2024-04-26T16:43:16.154625+0000 mgr.y (mgr.24401) 980 : cluster [DBG] pgmap v1580: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:17.378 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:43:17.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.333147+0000 osd.0 (osd.0) 639 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:43:17.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.333147+0000 osd.0 (osd.0) 639 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:43:17.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.335288+0000 osd.0 (osd.0) 640 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:43:17.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.335288+0000 osd.0 (osd.0) 640 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:43:17.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.625043+0000 osd.7 (osd.7) 674 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:43:17.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.625043+0000 osd.7 (osd.7) 674 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.627600+0000 osd.7 (osd.7) 675 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.627600+0000 osd.7 (osd.7) 675 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.892274+0000 mon.a (mon.0) 4077 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:15.892274+0000 mon.a (mon.0) 4077 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:16.105131+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:16.105131+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:16.108056+0000 osd.5 (osd.5) 541 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:16.108056+0000 osd.5 (osd.5) 541 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:16.154625+0000 mgr.y (mgr.24401) 980 : cluster [DBG] pgmap v1580: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[21463]: cluster 2024-04-26T16:43:16.154625+0000 mgr.y (mgr.24401) 980 : cluster [DBG] pgmap v1580: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.333147+0000 osd.0 (osd.0) 639 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.333147+0000 osd.0 (osd.0) 639 : cluster [DBG] 11.10 scrub starts 2024-04-26T16:43:17.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.335288+0000 osd.0 (osd.0) 640 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.335288+0000 osd.0 (osd.0) 640 : cluster [DBG] 11.10 scrub ok 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.625043+0000 osd.7 (osd.7) 674 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.625043+0000 osd.7 (osd.7) 674 : cluster [DBG] 11.12 scrub starts 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.627600+0000 osd.7 (osd.7) 675 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.627600+0000 osd.7 (osd.7) 675 : cluster [DBG] 11.12 scrub ok 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.892274+0000 mon.a (mon.0) 4077 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:15.892274+0000 mon.a (mon.0) 4077 : cluster [DBG] osdmap e932: 8 total, 8 up, 8 in 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:16.105131+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:16.105131+0000 osd.5 (osd.5) 540 : cluster [DBG] 5.7 scrub starts 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:16.108056+0000 osd.5 (osd.5) 541 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:16.108056+0000 osd.5 (osd.5) 541 : cluster [DBG] 5.7 scrub ok 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:16.154625+0000 mgr.y (mgr.24401) 980 : cluster [DBG] pgmap v1580: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:17.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:16 smithi050 bash[29234]: cluster 2024-04-26T16:43:16.154625+0000 mgr.y (mgr.24401) 980 : cluster [DBG] pgmap v1580: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:18.074 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:43:17 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:43:18.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:17 smithi190 bash[23868]: audit 2024-04-26T16:43:17.658106+0000 mgr.y (mgr.24401) 981 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:18.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:17 smithi190 bash[23868]: audit 2024-04-26T16:43:17.658106+0000 mgr.y (mgr.24401) 981 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:18.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:17 smithi050 bash[21463]: audit 2024-04-26T16:43:17.658106+0000 mgr.y (mgr.24401) 981 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:18.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:17 smithi050 bash[21463]: audit 2024-04-26T16:43:17.658106+0000 mgr.y (mgr.24401) 981 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:18.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:17 smithi050 bash[29234]: audit 2024-04-26T16:43:17.658106+0000 mgr.y (mgr.24401) 981 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:18.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:17 smithi050 bash[29234]: audit 2024-04-26T16:43:17.658106+0000 mgr.y (mgr.24401) 981 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:19.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:18 smithi190 bash[23868]: cluster 2024-04-26T16:43:17.322625+0000 osd.0 (osd.0) 641 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:43:19.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:18 smithi190 bash[23868]: cluster 2024-04-26T16:43:17.322625+0000 osd.0 (osd.0) 641 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:43:19.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:18 smithi190 bash[23868]: cluster 2024-04-26T16:43:17.325113+0000 osd.0 (osd.0) 642 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:43:19.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:18 smithi190 bash[23868]: cluster 2024-04-26T16:43:17.325113+0000 osd.0 (osd.0) 642 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:43:19.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:18 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.156671+0000 mgr.y (mgr.24401) 982 : cluster [DBG] pgmap v1581: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:19.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:18 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.156671+0000 mgr.y (mgr.24401) 982 : cluster [DBG] pgmap v1581: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:19.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[21463]: cluster 2024-04-26T16:43:17.322625+0000 osd.0 (osd.0) 641 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:43:19.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[21463]: cluster 2024-04-26T16:43:17.322625+0000 osd.0 (osd.0) 641 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:43:19.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[21463]: cluster 2024-04-26T16:43:17.325113+0000 osd.0 (osd.0) 642 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[21463]: cluster 2024-04-26T16:43:17.325113+0000 osd.0 (osd.0) 642 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.156671+0000 mgr.y (mgr.24401) 982 : cluster [DBG] pgmap v1581: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.156671+0000 mgr.y (mgr.24401) 982 : cluster [DBG] pgmap v1581: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[29234]: cluster 2024-04-26T16:43:17.322625+0000 osd.0 (osd.0) 641 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[29234]: cluster 2024-04-26T16:43:17.322625+0000 osd.0 (osd.0) 641 : cluster [DBG] 6.10 deep-scrub starts 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[29234]: cluster 2024-04-26T16:43:17.325113+0000 osd.0 (osd.0) 642 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[29234]: cluster 2024-04-26T16:43:17.325113+0000 osd.0 (osd.0) 642 : cluster [DBG] 6.10 deep-scrub ok 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.156671+0000 mgr.y (mgr.24401) 982 : cluster [DBG] pgmap v1581: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:19.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:18 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.156671+0000 mgr.y (mgr.24401) 982 : cluster [DBG] pgmap v1581: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:20.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.175566+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:43:20.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.175566+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:43:20.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.178652+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:43:20.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.178652+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:43:20.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.649604+0000 osd.3 (osd.3) 689 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:43:20.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.649604+0000 osd.3 (osd.3) 689 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:43:20.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.653522+0000 osd.3 (osd.3) 690 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:43:20.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:19 smithi190 bash[23868]: cluster 2024-04-26T16:43:18.653522+0000 osd.3 (osd.3) 690 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.175566+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.175566+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.178652+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.178652+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.649604+0000 osd.3 (osd.3) 689 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.649604+0000 osd.3 (osd.3) 689 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.653522+0000 osd.3 (osd.3) 690 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[21463]: cluster 2024-04-26T16:43:18.653522+0000 osd.3 (osd.3) 690 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:43:20.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.175566+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.175566+0000 osd.6 (osd.6) 171 : cluster [DBG] 5.1f scrub starts 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.178652+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.178652+0000 osd.6 (osd.6) 172 : cluster [DBG] 5.1f scrub ok 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.649604+0000 osd.3 (osd.3) 689 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.649604+0000 osd.3 (osd.3) 689 : cluster [DBG] 3.8 scrub starts 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.653522+0000 osd.3 (osd.3) 690 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:43:20.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:19 smithi050 bash[29234]: cluster 2024-04-26T16:43:18.653522+0000 osd.3 (osd.3) 690 : cluster [DBG] 3.8 scrub ok 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 -- 172.21.15.50:0/1004177685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103470 msgr2=0x7fea081038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 --2- 172.21.15.50:0/1004177685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103470 0x7fea081038f0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc009990 tx=0x7fe9fc009ca0 comp rx=0 tx=0).stop 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 -- 172.21.15.50:0/1004177685 shutdown_connections 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 --2- 172.21.15.50:0/1004177685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea08103e30 0x7fea08110290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 --2- 172.21.15.50:0/1004177685 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea081094c0 0x7fea081098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 --2- 172.21.15.50:0/1004177685 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103470 0x7fea081038f0 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:20.825 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 -- 172.21.15.50:0/1004177685 >> 172.21.15.50:0/1004177685 conn(0x7fea0807a580 msgr2=0x7fea0807a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 -- 172.21.15.50:0/1004177685 shutdown_connections 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 -- 172.21.15.50:0/1004177685 wait complete. 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 librados: starting msgr at 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 librados: starting objecter 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.823+0000 7fea0d12d700 1 Processor -- start 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- start start 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 librados: setting wanted keys 2024-04-26T16:43:20.826 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 librados: calling monclient init 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea08103470 0x7fea08075720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 0x7fea08075c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea081094c0 0x7fea080761a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7fea08112ab0 con 0x7fea08103e30 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7fea08112930 con 0x7fea08103470 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7fea08112c30 con 0x7fea081094c0 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 0x7fea08075c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 0x7fea08075c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3300/0 says I am v2:172.21.15.50:40022/0 (socket says 172.21.15.50:40022) 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 -- 172.21.15.50:0/4136591086 learned_addr learned my addr 172.21.15.50:0/4136591086 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0759e700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea081094c0 0x7fea080761a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:20.828 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea06d9d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea08103470 0x7fea08075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 -- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea081094c0 msgr2=0x7fea080761a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea081094c0 0x7fea080761a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 -- 172.21.15.50:0/4136591086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea08103470 msgr2=0x7fea08075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:20.829 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea08103470 0x7fea08075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 -- 172.21.15.50:0/4136591086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea081a7640 con 0x7fea08103e30 2024-04-26T16:43:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0659c700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 0x7fea08075c60 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0057a0 tx=0x7fe9fc0057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:43:20.830 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 10 client.?.objecter ms_handle_connect 0x7fea08103e30 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 10 client.?.objecter resend_mon_ops 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc026070 con 0x7fea08103e30 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7fe9fc021410 con 0x7fea08103e30 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea081a78d0 con 0x7fea08103e30 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9fc0173f0 con 0x7fea08103e30 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 10 client.31383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea081a7e00 con 0x7fea08103e30 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 librados: init done 2024-04-26T16:43:20.831 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea0806a110 con 0x7fea08103e30 2024-04-26T16:43:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7fe9fc029070 con 0x7fea08103e30 2024-04-26T16:43:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe9f0071f80 0x7fe9f0074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(932..932 src has 874..932) ==== 8707+0+0 (secure 0 0 0) 0x7fe9fc092780 con 0x7fea08103e30 2024-04-26T16:43:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 10 client.31383.objecter ms_dispatch 0x7fea08001550 osd_map(932..932 src has 874..932) 2024-04-26T16:43:20.832 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 3 client.31383.objecter handle_osd_map got epochs [932,932] > 0 2024-04-26T16:43:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fe9effff700 3 client.31383.objecter handle_osd_map decoding full epoch 932 2024-04-26T16:43:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.827+0000 7fea06d9d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe9f0071f80 0x7fe9f0074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.831+0000 7fe9effff700 20 client.31383.objecter dump_active .. 0 homeless 2024-04-26T16:43:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.831+0000 7fe9effff700 10 client.31383.objecter ms_handle_connect 0x7fe9f0071f80 2024-04-26T16:43:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.831+0000 7fea06d9d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe9f0071f80 0x7fe9f0074440 secure :-1 s=READY pgs=3891 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009bd0 tx=0x7fe9f8009260 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:43:20.833 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:20.835+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7fe9fc013070 con 0x7fea08103e30 2024-04-26T16:43:21.061 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.059+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 --> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea08102300 con 0x7fe9f0071f80 2024-04-26T16:43:21.104 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.103+0000 7fe9effff700 1 -- 172.21.15.50:0/4136591086 <== mgr.24401 v2:172.21.15.50:6800/584348748 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+453076 (secure 0 0 0) 0x7fea08102300 con 0x7fe9f0071f80 2024-04-26T16:43:21.109 INFO:teuthology.orchestra.run.smithi050.stdout: 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 10 librados: watch_flush enter 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 10 librados: watch_flush exit 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 20 client.31383.objecter shutdown clearing up homeless session... 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 10 client.31383.objecter successfully canceled tick 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe9f0071f80 msgr2=0x7fe9f0074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe9f0071f80 0x7fe9f0074440 secure :-1 s=READY pgs=3891 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8009bd0 tx=0x7fe9f8009260 comp rx=0 tx=0).stop 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 msgr2=0x7fea08075c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 0x7fea08075c60 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0057a0 tx=0x7fe9fc0057d0 comp rx=0 tx=0).stop 2024-04-26T16:43:21.113 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 shutdown_connections 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7fe9f0071f80 0x7fe9f0074440 unknown :-1 s=CLOSED pgs=3891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fea08103470 0x7fea08075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7fea081094c0 0x7fea080761a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 --2- 172.21.15.50:0/4136591086 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7fea08103e30 0x7fea08075c60 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 >> 172.21.15.50:0/4136591086 conn(0x7fea0807a580 msgr2=0x7fea08101bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 shutdown_connections 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 -- 172.21.15.50:0/4136591086 wait complete. 2024-04-26T16:43:21.114 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:21.107+0000 7fea0d12d700 1 librados: shutdown 2024-04-26T16:43:21.115 INFO:teuthology.orchestra.run.smithi050.stderr:dumped all 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:19.994771+0000 osd.5 (osd.5) 542 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:19.994771+0000 osd.5 (osd.5) 542 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:19.998018+0000 osd.5 (osd.5) 543 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:19.998018+0000 osd.5 (osd.5) 543 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.117151+0000 osd.1 (osd.1) 619 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.117151+0000 osd.1 (osd.1) 619 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.117543+0000 osd.1 (osd.1) 620 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.117543+0000 osd.1 (osd.1) 620 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.157629+0000 mgr.y (mgr.24401) 983 : cluster [DBG] pgmap v1582: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.157629+0000 mgr.y (mgr.24401) 983 : cluster [DBG] pgmap v1582: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.174814+0000 osd.2 (osd.2) 561 : cluster [DBG] 2.c scrub starts 2024-04-26T16:43:21.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.174814+0000 osd.2 (osd.2) 561 : cluster [DBG] 2.c scrub starts 2024-04-26T16:43:21.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.177152+0000 osd.2 (osd.2) 562 : cluster [DBG] 2.c scrub ok 2024-04-26T16:43:21.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:20 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.177152+0000 osd.2 (osd.2) 562 : cluster [DBG] 2.c scrub ok 2024-04-26T16:43:21.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:19.994771+0000 osd.5 (osd.5) 542 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:43:21.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:19.994771+0000 osd.5 (osd.5) 542 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:43:21.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:19.998018+0000 osd.5 (osd.5) 543 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:43:21.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:19.998018+0000 osd.5 (osd.5) 543 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:43:21.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.117151+0000 osd.1 (osd.1) 619 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.117151+0000 osd.1 (osd.1) 619 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.117543+0000 osd.1 (osd.1) 620 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.117543+0000 osd.1 (osd.1) 620 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.157629+0000 mgr.y (mgr.24401) 983 : cluster [DBG] pgmap v1582: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.157629+0000 mgr.y (mgr.24401) 983 : cluster [DBG] pgmap v1582: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.174814+0000 osd.2 (osd.2) 561 : cluster [DBG] 2.c scrub starts 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.174814+0000 osd.2 (osd.2) 561 : cluster [DBG] 2.c scrub starts 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.177152+0000 osd.2 (osd.2) 562 : cluster [DBG] 2.c scrub ok 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.177152+0000 osd.2 (osd.2) 562 : cluster [DBG] 2.c scrub ok 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:19.994771+0000 osd.5 (osd.5) 542 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:43:21.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:19.994771+0000 osd.5 (osd.5) 542 : cluster [DBG] 1.8 scrub starts 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:19.998018+0000 osd.5 (osd.5) 543 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:19.998018+0000 osd.5 (osd.5) 543 : cluster [DBG] 1.8 scrub ok 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.117151+0000 osd.1 (osd.1) 619 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.117151+0000 osd.1 (osd.1) 619 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.117543+0000 osd.1 (osd.1) 620 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.117543+0000 osd.1 (osd.1) 620 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.157629+0000 mgr.y (mgr.24401) 983 : cluster [DBG] pgmap v1582: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.157629+0000 mgr.y (mgr.24401) 983 : cluster [DBG] pgmap v1582: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.174814+0000 osd.2 (osd.2) 561 : cluster [DBG] 2.c scrub starts 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.174814+0000 osd.2 (osd.2) 561 : cluster [DBG] 2.c scrub starts 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.177152+0000 osd.2 (osd.2) 562 : cluster [DBG] 2.c scrub ok 2024-04-26T16:43:21.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:20 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.177152+0000 osd.2 (osd.2) 562 : cluster [DBG] 2.c scrub ok 2024-04-26T16:43:22.014 INFO:teuthology.orchestra.run.smithi050.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2024-04-26T16:43:20.931402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596512,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":3115,"num_read_kb":2867,"num_write":1015,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":3434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2703,"ondisk_log_size":2703,"up":522,"acting":522,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":522,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":810892,"kb_used_data":20664,"kb_used_omap":34,"kb_used_meta":790109,"kb_avail":749150324,"statfs":{"total":767960285184,"available":767129931776,"internally_reserved":0,"allocated":21159936,"data_stored":7147953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":34917,"internal_metadata":809072539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.780110"},"pg_stats":[{"pgid":"11.7","version":"0'0","reported_seq":743,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109959+0000","last_change":"2024-04-26T16:42:21.953084+0000","last_active":"2024-04-26T16:43:16.109959+0000","last_peered":"2024-04-26T16:43:16.109959+0000","last_clean":"2024-04-26T16:43:16.109959+0000","last_became_active":"2024-04-26T16:32:09.465706+0000","last_became_peered":"2024-04-26T16:32:09.465706+0000","last_unstale":"2024-04-26T16:43:16.109959+0000","last_undegraded":"2024-04-26T16:43:16.109959+0000","last_fullsized":"2024-04-26T16:43:16.109959+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:21.952854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:44.992566+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.952854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:34.118612+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"747'16","reported_seq":2122,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183206+0000","last_change":"2024-04-26T16:41:39.323840+0000","last_active":"2024-04-26T16:43:16.183206+0000","last_peered":"2024-04-26T16:43:16.183206+0000","last_clean":"2024-04-26T16:43:16.183206+0000","last_became_active":"2024-04-26T16:35:25.395438+0000","last_became_peered":"2024-04-26T16:35:25.395438+0000","last_unstale":"2024-04-26T16:43:16.183206+0000","last_undegraded":"2024-04-26T16:43:16.183206+0000","last_fullsized":"2024-04-26T16:43:16.183206+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:41:39.323617+0000","last_deep_scrub":"433'14","last_deep_scrub_stamp":"2024-04-26T16:29:10.136172+0000","last_clean_scrub_stamp":"2024-04-26T16:41:39.323617+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"857'46","reported_seq":1889,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185916+0000","last_change":"2024-04-26T16:42:34.355089+0000","last_active":"2024-04-26T16:43:16.185916+0000","last_peered":"2024-04-26T16:43:16.185916+0000","last_clean":"2024-04-26T16:43:16.185916+0000","last_became_active":"2024-04-26T16:26:27.132042+0000","last_became_peered":"2024-04-26T16:26:27.132042+0000","last_unstale":"2024-04-26T16:43:16.185916+0000","last_undegraded":"2024-04-26T16:43:16.185916+0000","last_fullsized":"2024-04-26T16:43:16.185916+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:42:34.354814+0000","last_deep_scrub":"857'46","last_deep_scrub_stamp":"2024-04-26T16:41:20.280571+0000","last_clean_scrub_stamp":"2024-04-26T16:42:34.354814+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.859237+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1828,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184040+0000","last_change":"2024-04-26T16:41:55.064749+0000","last_active":"2024-04-26T16:43:16.184040+0000","last_peered":"2024-04-26T16:43:16.184040+0000","last_clean":"2024-04-26T16:43:16.184040+0000","last_became_active":"2024-04-26T16:17:29.010013+0000","last_became_peered":"2024-04-26T16:17:29.010013+0000","last_unstale":"2024-04-26T16:43:16.184040+0000","last_undegraded":"2024-04-26T16:43:16.184040+0000","last_fullsized":"2024-04-26T16:43:16.184040+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:55.064519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:55.064519+0000","last_clean_scrub_stamp":"2024-04-26T16:41:55.064519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:22.442164+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":744,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107743+0000","last_change":"2024-04-26T16:42:53.200576+0000","last_active":"2024-04-26T16:43:16.107743+0000","last_peered":"2024-04-26T16:43:16.107743+0000","last_clean":"2024-04-26T16:43:16.107743+0000","last_became_active":"2024-04-26T16:35:25.382820+0000","last_became_peered":"2024-04-26T16:35:25.382820+0000","last_unstale":"2024-04-26T16:43:16.107743+0000","last_undegraded":"2024-04-26T16:43:16.107743+0000","last_fullsized":"2024-04-26T16:43:16.107743+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.200335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:53.200335+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.200335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.012637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1657,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183841+0000","last_change":"2024-04-26T16:42:07.435892+0000","last_active":"2024-04-26T16:43:16.183841+0000","last_peered":"2024-04-26T16:43:16.183841+0000","last_clean":"2024-04-26T16:43:16.183841+0000","last_became_active":"2024-04-26T16:35:25.399977+0000","last_became_peered":"2024-04-26T16:35:25.399977+0000","last_unstale":"2024-04-26T16:43:16.183841+0000","last_undegraded":"2024-04-26T16:43:16.183841+0000","last_fullsized":"2024-04-26T16:43:16.183841+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:07.435665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.435665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:17.179892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.c","version":"857'30","reported_seq":770,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.108661+0000","last_change":"2024-04-26T16:42:52.187165+0000","last_active":"2024-04-26T16:43:16.108661+0000","last_peered":"2024-04-26T16:43:16.108661+0000","last_clean":"2024-04-26T16:43:16.108661+0000","last_became_active":"2024-04-26T16:35:25.393291+0000","last_became_peered":"2024-04-26T16:35:25.393291+0000","last_unstale":"2024-04-26T16:43:16.108661+0000","last_undegraded":"2024-04-26T16:43:16.108661+0000","last_fullsized":"2024-04-26T16:43:16.108661+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:42:52.186807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.186807+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:12.504553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1832,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.895488+0000","last_change":"2024-04-26T16:42:41.069430+0000","last_active":"2024-04-26T16:43:15.895488+0000","last_peered":"2024-04-26T16:43:15.895488+0000","last_clean":"2024-04-26T16:43:15.895488+0000","last_became_active":"2024-04-26T16:17:28.999644+0000","last_became_peered":"2024-04-26T16:17:28.999644+0000","last_unstale":"2024-04-26T16:43:15.895488+0000","last_undegraded":"2024-04-26T16:43:15.895488+0000","last_fullsized":"2024-04-26T16:43:15.895488+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:41.069198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:57.187612+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.069198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.526843+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1828,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182999+0000","last_change":"2024-04-26T16:42:26.248376+0000","last_active":"2024-04-26T16:43:16.182999+0000","last_peered":"2024-04-26T16:43:16.182999+0000","last_clean":"2024-04-26T16:43:16.182999+0000","last_became_active":"2024-04-26T16:17:33.065179+0000","last_became_peered":"2024-04-26T16:17:33.065179+0000","last_unstale":"2024-04-26T16:43:16.182999+0000","last_undegraded":"2024-04-26T16:43:16.182999+0000","last_fullsized":"2024-04-26T16:43:16.182999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:26.248164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:26.248164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:55.830972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1823,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889439+0000","last_change":"2024-04-26T16:42:36.241515+0000","last_active":"2024-04-26T16:43:15.889439+0000","last_peered":"2024-04-26T16:43:15.889439+0000","last_clean":"2024-04-26T16:43:15.889439+0000","last_became_active":"2024-04-26T16:17:35.070827+0000","last_became_peered":"2024-04-26T16:17:35.070827+0000","last_unstale":"2024-04-26T16:43:15.889439+0000","last_undegraded":"2024-04-26T16:43:15.889439+0000","last_fullsized":"2024-04-26T16:43:15.889439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:36.241334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:01.875991+0000","last_clean_scrub_stamp":"2024-04-26T16:42:36.241334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.353720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1685,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184491+0000","last_change":"2024-04-26T16:42:59.354653+0000","last_active":"2024-04-26T16:43:16.184491+0000","last_peered":"2024-04-26T16:43:16.184491+0000","last_clean":"2024-04-26T16:43:16.184491+0000","last_became_active":"2024-04-26T16:35:25.385146+0000","last_became_peered":"2024-04-26T16:35:25.385146+0000","last_unstale":"2024-04-26T16:43:16.184491+0000","last_undegraded":"2024-04-26T16:43:16.184491+0000","last_fullsized":"2024-04-26T16:43:16.184491+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:59.354438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:59.354438+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.354438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:11.501488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"857'53","reported_seq":1935,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184735+0000","last_change":"2024-04-26T16:42:57.379802+0000","last_active":"2024-04-26T16:43:16.184735+0000","last_peered":"2024-04-26T16:43:16.184735+0000","last_clean":"2024-04-26T16:43:16.184735+0000","last_became_active":"2024-04-26T16:35:25.387497+0000","last_became_peered":"2024-04-26T16:35:25.387497+0000","last_unstale":"2024-04-26T16:43:16.184735+0000","last_undegraded":"2024-04-26T16:43:16.184735+0000","last_fullsized":"2024-04-26T16:43:16.184735+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'53","last_scrub_stamp":"2024-04-26T16:42:57.379550+0000","last_deep_scrub":"857'53","last_deep_scrub_stamp":"2024-04-26T16:39:14.920205+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.379550+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:16.342719+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1848,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183930+0000","last_change":"2024-04-26T16:42:10.097760+0000","last_active":"2024-04-26T16:43:16.183930+0000","last_peered":"2024-04-26T16:43:16.183930+0000","last_clean":"2024-04-26T16:43:16.183930+0000","last_became_active":"2024-04-26T16:32:09.482718+0000","last_became_peered":"2024-04-26T16:32:09.482718+0000","last_unstale":"2024-04-26T16:43:16.183930+0000","last_undegraded":"2024-04-26T16:43:16.183930+0000","last_fullsized":"2024-04-26T16:43:16.183930+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:10.097423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:10.097423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:19.608687+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1840,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183596+0000","last_change":"2024-04-26T16:42:43.071047+0000","last_active":"2024-04-26T16:43:16.183596+0000","last_peered":"2024-04-26T16:43:16.183596+0000","last_clean":"2024-04-26T16:43:16.183596+0000","last_became_active":"2024-04-26T16:32:09.482107+0000","last_became_peered":"2024-04-26T16:32:09.482107+0000","last_unstale":"2024-04-26T16:43:16.183596+0000","last_undegraded":"2024-04-26T16:43:16.183596+0000","last_fullsized":"2024-04-26T16:43:16.183596+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:43.070838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:43.070838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:08.938502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1783,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181128+0000","last_change":"2024-04-26T16:42:16.431899+0000","last_active":"2024-04-26T16:43:16.181128+0000","last_peered":"2024-04-26T16:43:16.181128+0000","last_clean":"2024-04-26T16:43:16.181128+0000","last_became_active":"2024-04-26T16:17:35.081699+0000","last_became_peered":"2024-04-26T16:17:35.081699+0000","last_unstale":"2024-04-26T16:43:16.181128+0000","last_undegraded":"2024-04-26T16:43:16.181128+0000","last_fullsized":"2024-04-26T16:43:16.181128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:16.431658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:20:47.169235+0000","last_clean_scrub_stamp":"2024-04-26T16:42:16.431658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:20.097666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"101'130","reported_seq":461,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.106003+0000","last_change":"2024-04-26T16:41:50.977787+0000","last_active":"2024-04-26T16:43:16.106003+0000","last_peered":"2024-04-26T16:43:16.106003+0000","last_clean":"2024-04-26T16:43:16.106003+0000","last_became_active":"2024-04-26T16:38:46.970801+0000","last_became_peered":"2024-04-26T16:38:46.970801+0000","last_unstale":"2024-04-26T16:43:16.106003+0000","last_undegraded":"2024-04-26T16:43:16.106003+0000","last_fullsized":"2024-04-26T16:43:16.106003+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:41:50.977345+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.977345+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1680,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185838+0000","last_change":"2024-04-26T16:42:49.173899+0000","last_active":"2024-04-26T16:43:16.185838+0000","last_peered":"2024-04-26T16:43:16.185838+0000","last_clean":"2024-04-26T16:43:16.185838+0000","last_became_active":"2024-04-26T16:23:29.183432+0000","last_became_peered":"2024-04-26T16:23:29.183432+0000","last_unstale":"2024-04-26T16:43:16.185838+0000","last_undegraded":"2024-04-26T16:43:16.185838+0000","last_fullsized":"2024-04-26T16:43:16.185838+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:49.173787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:04.824897+0000","last_clean_scrub_stamp":"2024-04-26T16:42:49.173787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.341183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"907'60","reported_seq":1586,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181404+0000","last_change":"2024-04-26T16:43:06.215205+0000","last_active":"2024-04-26T16:43:16.181404+0000","last_peered":"2024-04-26T16:43:16.181404+0000","last_clean":"2024-04-26T16:43:16.181404+0000","last_became_active":"2024-04-26T16:35:25.414489+0000","last_became_peered":"2024-04-26T16:35:25.414489+0000","last_unstale":"2024-04-26T16:43:16.181404+0000","last_undegraded":"2024-04-26T16:43:16.181404+0000","last_fullsized":"2024-04-26T16:43:16.181404+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'60","last_scrub_stamp":"2024-04-26T16:43:06.214936+0000","last_deep_scrub":"907'60","last_deep_scrub_stamp":"2024-04-26T16:40:18.066521+0000","last_clean_scrub_stamp":"2024-04-26T16:43:06.214936+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.603610+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1807,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182745+0000","last_change":"2024-04-26T16:41:50.426808+0000","last_active":"2024-04-26T16:43:16.182745+0000","last_peered":"2024-04-26T16:43:16.182745+0000","last_clean":"2024-04-26T16:43:16.182745+0000","last_became_active":"2024-04-26T16:32:09.458962+0000","last_became_peered":"2024-04-26T16:32:09.458962+0000","last_unstale":"2024-04-26T16:43:16.182745+0000","last_undegraded":"2024-04-26T16:43:16.182745+0000","last_fullsized":"2024-04-26T16:43:16.182745+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:50.426591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:50.426591+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.426591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1832,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.894133+0000","last_change":"2024-04-26T16:42:34.062846+0000","last_active":"2024-04-26T16:43:15.894133+0000","last_peered":"2024-04-26T16:43:15.894133+0000","last_clean":"2024-04-26T16:43:15.894133+0000","last_became_active":"2024-04-26T16:17:33.054616+0000","last_became_peered":"2024-04-26T16:17:33.054616+0000","last_unstale":"2024-04-26T16:43:15.894133+0000","last_undegraded":"2024-04-26T16:43:15.894133+0000","last_fullsized":"2024-04-26T16:43:15.894133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:34.062625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:20.156962+0000","last_clean_scrub_stamp":"2024-04-26T16:42:34.062625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.244190+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1824,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184738+0000","last_change":"2024-04-26T16:42:54.228119+0000","last_active":"2024-04-26T16:43:16.184738+0000","last_peered":"2024-04-26T16:43:16.184738+0000","last_clean":"2024-04-26T16:43:16.184738+0000","last_became_active":"2024-04-26T16:17:35.091645+0000","last_became_peered":"2024-04-26T16:17:35.091645+0000","last_unstale":"2024-04-26T16:43:16.184738+0000","last_undegraded":"2024-04-26T16:43:16.184738+0000","last_fullsized":"2024-04-26T16:43:16.184738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:54.228037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:33.120136+0000","last_clean_scrub_stamp":"2024-04-26T16:42:54.228037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:19.253610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"101'130","reported_seq":2120,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181717+0000","last_change":"2024-04-26T16:43:02.400306+0000","last_active":"2024-04-26T16:43:16.181717+0000","last_peered":"2024-04-26T16:43:16.181717+0000","last_clean":"2024-04-26T16:43:16.181717+0000","last_became_active":"2024-04-26T16:38:46.976273+0000","last_became_peered":"2024-04-26T16:38:46.976273+0000","last_unstale":"2024-04-26T16:43:16.181717+0000","last_undegraded":"2024-04-26T16:43:16.181717+0000","last_fullsized":"2024-04-26T16:43:16.181717+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:43:02.400066+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:43:02.400066+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.400066+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:21.702252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1293,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181835+0000","last_change":"2024-04-26T16:42:23.169869+0000","last_active":"2024-04-26T16:43:16.181835+0000","last_peered":"2024-04-26T16:43:16.181835+0000","last_clean":"2024-04-26T16:43:16.181835+0000","last_became_active":"2024-04-26T16:35:25.388195+0000","last_became_peered":"2024-04-26T16:35:25.388195+0000","last_unstale":"2024-04-26T16:43:16.181835+0000","last_undegraded":"2024-04-26T16:43:16.181835+0000","last_fullsized":"2024-04-26T16:43:16.181835+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:23.169669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:23.169669+0000","last_clean_scrub_stamp":"2024-04-26T16:42:23.169669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:47.195006+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"857'25","reported_seq":1846,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180752+0000","last_change":"2024-04-26T16:42:34.022458+0000","last_active":"2024-04-26T16:43:16.180752+0000","last_peered":"2024-04-26T16:43:16.180752+0000","last_clean":"2024-04-26T16:43:16.180752+0000","last_became_active":"2024-04-26T16:17:31.055811+0000","last_became_peered":"2024-04-26T16:17:31.055811+0000","last_unstale":"2024-04-26T16:43:16.180752+0000","last_undegraded":"2024-04-26T16:43:16.180752+0000","last_fullsized":"2024-04-26T16:43:16.180752+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:42:34.022223+0000","last_deep_scrub":"857'25","last_deep_scrub_stamp":"2024-04-26T16:40:08.182235+0000","last_clean_scrub_stamp":"2024-04-26T16:42:34.022223+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:38.520984+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1832,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890917+0000","last_change":"2024-04-26T16:42:42.037307+0000","last_active":"2024-04-26T16:43:15.890917+0000","last_peered":"2024-04-26T16:43:15.890917+0000","last_clean":"2024-04-26T16:43:15.890917+0000","last_became_active":"2024-04-26T16:17:29.003369+0000","last_became_peered":"2024-04-26T16:17:29.003369+0000","last_unstale":"2024-04-26T16:43:15.890917+0000","last_undegraded":"2024-04-26T16:43:15.890917+0000","last_fullsized":"2024-04-26T16:43:15.890917+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:42.037049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:08.142728+0000","last_clean_scrub_stamp":"2024-04-26T16:42:42.037049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.184688+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"585'13","reported_seq":2168,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183620+0000","last_change":"2024-04-26T16:42:19.067756+0000","last_active":"2024-04-26T16:43:16.183620+0000","last_peered":"2024-04-26T16:43:16.183620+0000","last_clean":"2024-04-26T16:43:16.183620+0000","last_became_active":"2024-04-26T16:32:09.489413+0000","last_became_peered":"2024-04-26T16:32:09.489413+0000","last_unstale":"2024-04-26T16:43:16.183620+0000","last_undegraded":"2024-04-26T16:43:16.183620+0000","last_fullsized":"2024-04-26T16:43:16.183620+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'13","last_scrub_stamp":"2024-04-26T16:42:19.067550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:19.067550+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:22.639890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1824,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890931+0000","last_change":"2024-04-26T16:42:22.879668+0000","last_active":"2024-04-26T16:43:15.890931+0000","last_peered":"2024-04-26T16:43:15.890931+0000","last_clean":"2024-04-26T16:43:15.890931+0000","last_became_active":"2024-04-26T16:17:35.081875+0000","last_became_peered":"2024-04-26T16:17:35.081875+0000","last_unstale":"2024-04-26T16:43:15.890931+0000","last_undegraded":"2024-04-26T16:43:15.890931+0000","last_fullsized":"2024-04-26T16:43:15.890931+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:22.879454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:47.893722+0000","last_clean_scrub_stamp":"2024-04-26T16:42:22.879454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:28.972997+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"101'130","reported_seq":469,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.106390+0000","last_change":"2024-04-26T16:42:31.112870+0000","last_active":"2024-04-26T16:43:16.106390+0000","last_peered":"2024-04-26T16:43:16.106390+0000","last_clean":"2024-04-26T16:43:16.106390+0000","last_became_active":"2024-04-26T16:38:46.964902+0000","last_became_peered":"2024-04-26T16:38:46.964902+0000","last_unstale":"2024-04-26T16:43:16.106390+0000","last_undegraded":"2024-04-26T16:43:16.106390+0000","last_fullsized":"2024-04-26T16:43:16.106390+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:31.112594+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:42:31.112594+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:51.566028+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1682,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890269+0000","last_change":"2024-04-26T16:42:52.180638+0000","last_active":"2024-04-26T16:43:15.890269+0000","last_peered":"2024-04-26T16:43:15.890269+0000","last_clean":"2024-04-26T16:43:15.890269+0000","last_became_active":"2024-04-26T16:23:29.219861+0000","last_became_peered":"2024-04-26T16:23:29.219861+0000","last_unstale":"2024-04-26T16:43:15.890269+0000","last_undegraded":"2024-04-26T16:43:15.890269+0000","last_fullsized":"2024-04-26T16:43:15.890269+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:52.180410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:17.328349+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.180410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.777727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"857'46","reported_seq":1868,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.179901+0000","last_change":"2024-04-26T16:41:43.013726+0000","last_active":"2024-04-26T16:43:16.179901+0000","last_peered":"2024-04-26T16:43:16.179901+0000","last_clean":"2024-04-26T16:43:16.179901+0000","last_became_active":"2024-04-26T16:17:31.048921+0000","last_became_peered":"2024-04-26T16:17:31.048921+0000","last_unstale":"2024-04-26T16:43:16.179901+0000","last_undegraded":"2024-04-26T16:43:16.179901+0000","last_fullsized":"2024-04-26T16:43:16.179901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:41:43.013322+0000","last_deep_scrub":"857'46","last_deep_scrub_stamp":"2024-04-26T16:40:19.096428+0000","last_clean_scrub_stamp":"2024-04-26T16:41:43.013322+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1846,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890002+0000","last_change":"2024-04-26T16:42:29.025575+0000","last_active":"2024-04-26T16:43:15.890002+0000","last_peered":"2024-04-26T16:43:15.890002+0000","last_clean":"2024-04-26T16:43:15.890002+0000","last_became_active":"2024-04-26T16:18:16.612212+0000","last_became_peered":"2024-04-26T16:18:16.612212+0000","last_unstale":"2024-04-26T16:43:15.890002+0000","last_undegraded":"2024-04-26T16:43:15.890002+0000","last_fullsized":"2024-04-26T16:43:15.890002+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:29.025381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:35.051759+0000","last_clean_scrub_stamp":"2024-04-26T16:42:29.025381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:33.052455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"585'14","reported_seq":2169,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890197+0000","last_change":"2024-04-26T16:42:57.821822+0000","last_active":"2024-04-26T16:43:15.890197+0000","last_peered":"2024-04-26T16:43:15.890197+0000","last_clean":"2024-04-26T16:43:15.890197+0000","last_became_active":"2024-04-26T16:32:09.467431+0000","last_became_peered":"2024-04-26T16:32:09.467431+0000","last_unstale":"2024-04-26T16:43:15.890197+0000","last_undegraded":"2024-04-26T16:43:15.890197+0000","last_fullsized":"2024-04-26T16:43:15.890197+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:42:57.821585+0000","last_deep_scrub":"585'14","last_deep_scrub_stamp":"2024-04-26T16:42:57.821585+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.821585+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.592349+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":742,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109233+0000","last_change":"2024-04-26T16:42:29.099545+0000","last_active":"2024-04-26T16:43:16.109233+0000","last_peered":"2024-04-26T16:43:16.109233+0000","last_clean":"2024-04-26T16:43:16.109233+0000","last_became_active":"2024-04-26T16:32:09.470416+0000","last_became_peered":"2024-04-26T16:32:09.470416+0000","last_unstale":"2024-04-26T16:43:16.109233+0000","last_undegraded":"2024-04-26T16:43:16.109233+0000","last_fullsized":"2024-04-26T16:43:16.109233+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:29.099258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:00.008398+0000","last_clean_scrub_stamp":"2024-04-26T16:42:29.099258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.484902+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"141'7","reported_seq":1314,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180238+0000","last_change":"2024-04-26T16:42:22.129969+0000","last_active":"2024-04-26T16:43:16.180238+0000","last_peered":"2024-04-26T16:43:16.180238+0000","last_clean":"2024-04-26T16:43:16.180238+0000","last_became_active":"2024-04-26T16:35:25.407637+0000","last_became_peered":"2024-04-26T16:35:25.407637+0000","last_unstale":"2024-04-26T16:43:16.180238+0000","last_undegraded":"2024-04-26T16:43:16.180238+0000","last_fullsized":"2024-04-26T16:43:16.180238+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'7","last_scrub_stamp":"2024-04-26T16:42:22.129728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:22.129728+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.552357+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"856'29","reported_seq":1870,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888763+0000","last_change":"2024-04-26T16:42:40.870439+0000","last_active":"2024-04-26T16:43:15.888763+0000","last_peered":"2024-04-26T16:43:15.888763+0000","last_clean":"2024-04-26T16:43:15.888763+0000","last_became_active":"2024-04-26T16:17:31.021862+0000","last_became_peered":"2024-04-26T16:17:31.021862+0000","last_unstale":"2024-04-26T16:43:15.888763+0000","last_undegraded":"2024-04-26T16:43:15.888763+0000","last_fullsized":"2024-04-26T16:43:15.888763+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"856'29","last_scrub_stamp":"2024-04-26T16:42:40.870176+0000","last_deep_scrub":"824'23","last_deep_scrub_stamp":"2024-04-26T16:36:53.947636+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.870176+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:45.773872+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"79'1","reported_seq":1811,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181795+0000","last_change":"2024-04-26T16:42:14.414627+0000","last_active":"2024-04-26T16:43:16.181795+0000","last_peered":"2024-04-26T16:43:16.181795+0000","last_clean":"2024-04-26T16:43:16.181795+0000","last_became_active":"2024-04-26T16:17:29.006331+0000","last_became_peered":"2024-04-26T16:17:29.006331+0000","last_unstale":"2024-04-26T16:43:16.181795+0000","last_undegraded":"2024-04-26T16:43:16.181795+0000","last_fullsized":"2024-04-26T16:43:16.181795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'1","last_scrub_stamp":"2024-04-26T16:42:14.414402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:14.414402+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.593834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":743,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.110262+0000","last_change":"2024-04-26T16:42:33.105010+0000","last_active":"2024-04-26T16:43:16.110262+0000","last_peered":"2024-04-26T16:43:16.110262+0000","last_clean":"2024-04-26T16:43:16.110262+0000","last_became_active":"2024-04-26T16:35:25.382698+0000","last_became_peered":"2024-04-26T16:35:25.382698+0000","last_unstale":"2024-04-26T16:43:16.110262+0000","last_undegraded":"2024-04-26T16:43:16.110262+0000","last_fullsized":"2024-04-26T16:43:16.110262+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:33.104784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:33.104784+0000","last_clean_scrub_stamp":"2024-04-26T16:42:33.104784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:52.669838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1814,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181771+0000","last_change":"2024-04-26T16:42:35.028571+0000","last_active":"2024-04-26T16:43:16.181771+0000","last_peered":"2024-04-26T16:43:16.181771+0000","last_clean":"2024-04-26T16:43:16.181771+0000","last_became_active":"2024-04-26T16:35:25.416938+0000","last_became_peered":"2024-04-26T16:35:25.416938+0000","last_unstale":"2024-04-26T16:43:16.181771+0000","last_undegraded":"2024-04-26T16:43:16.181771+0000","last_fullsized":"2024-04-26T16:43:16.181771+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:35.028420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:47.296951+0000","last_clean_scrub_stamp":"2024-04-26T16:42:35.028420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.473689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"101'130","reported_seq":462,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181013+0000","last_change":"2024-04-26T16:42:04.989479+0000","last_active":"2024-04-26T16:43:16.181013+0000","last_peered":"2024-04-26T16:43:16.181013+0000","last_clean":"2024-04-26T16:43:16.181013+0000","last_became_active":"2024-04-26T16:38:46.969591+0000","last_became_peered":"2024-04-26T16:38:46.969591+0000","last_unstale":"2024-04-26T16:43:16.181013+0000","last_undegraded":"2024-04-26T16:43:16.181013+0000","last_fullsized":"2024-04-26T16:43:16.181013+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:04.989127+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:42:04.989127+0000","last_clean_scrub_stamp":"2024-04-26T16:42:04.989127+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1677,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890278+0000","last_change":"2024-04-26T16:42:07.761121+0000","last_active":"2024-04-26T16:43:15.890278+0000","last_peered":"2024-04-26T16:43:15.890278+0000","last_clean":"2024-04-26T16:43:15.890278+0000","last_became_active":"2024-04-26T16:23:29.317332+0000","last_became_peered":"2024-04-26T16:23:29.317332+0000","last_unstale":"2024-04-26T16:43:15.890278+0000","last_undegraded":"2024-04-26T16:43:15.890278+0000","last_fullsized":"2024-04-26T16:43:15.890278+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:07.760906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.760906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:21.313085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"120'12","reported_seq":2118,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180208+0000","last_change":"2024-04-26T16:42:07.031827+0000","last_active":"2024-04-26T16:43:16.180208+0000","last_peered":"2024-04-26T16:43:16.180208+0000","last_clean":"2024-04-26T16:43:16.180208+0000","last_became_active":"2024-04-26T16:17:33.044474+0000","last_became_peered":"2024-04-26T16:17:33.044474+0000","last_unstale":"2024-04-26T16:43:16.180208+0000","last_undegraded":"2024-04-26T16:43:16.180208+0000","last_fullsized":"2024-04-26T16:43:16.180208+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'12","last_scrub_stamp":"2024-04-26T16:42:07.031596+0000","last_deep_scrub":"120'12","last_deep_scrub_stamp":"2024-04-26T16:33:54.179322+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.031596+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:34.357075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1860,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888958+0000","last_change":"2024-04-26T16:43:07.955125+0000","last_active":"2024-04-26T16:43:15.888958+0000","last_peered":"2024-04-26T16:43:15.888958+0000","last_clean":"2024-04-26T16:43:15.888958+0000","last_became_active":"2024-04-26T16:35:25.411443+0000","last_became_peered":"2024-04-26T16:35:25.411443+0000","last_unstale":"2024-04-26T16:43:15.888958+0000","last_undegraded":"2024-04-26T16:43:15.888958+0000","last_fullsized":"2024-04-26T16:43:15.888958+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:07.954915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:43:07.954915+0000","last_clean_scrub_stamp":"2024-04-26T16:43:07.954915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:25.833622+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016166049999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"857'42","reported_seq":1883,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893465+0000","last_change":"2024-04-26T16:41:53.018973+0000","last_active":"2024-04-26T16:43:15.893465+0000","last_peered":"2024-04-26T16:43:15.893465+0000","last_clean":"2024-04-26T16:43:15.893465+0000","last_became_active":"2024-04-26T16:17:31.048446+0000","last_became_peered":"2024-04-26T16:17:31.048446+0000","last_unstale":"2024-04-26T16:43:15.893465+0000","last_undegraded":"2024-04-26T16:43:15.893465+0000","last_fullsized":"2024-04-26T16:43:15.893465+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'42","last_scrub_stamp":"2024-04-26T16:41:53.018846+0000","last_deep_scrub":"107'18","last_deep_scrub_stamp":"2024-04-26T16:21:31.258946+0000","last_clean_scrub_stamp":"2024-04-26T16:41:53.018846+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:19.676483+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1425,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183713+0000","last_change":"2024-04-26T16:42:50.233777+0000","last_active":"2024-04-26T16:43:16.183713+0000","last_peered":"2024-04-26T16:43:16.183713+0000","last_clean":"2024-04-26T16:43:16.183713+0000","last_became_active":"2024-04-26T16:35:25.406996+0000","last_became_peered":"2024-04-26T16:35:25.406996+0000","last_unstale":"2024-04-26T16:43:16.183713+0000","last_undegraded":"2024-04-26T16:43:16.183713+0000","last_fullsized":"2024-04-26T16:43:16.183713+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:50.233569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:32.453036+0000","last_clean_scrub_stamp":"2024-04-26T16:42:50.233569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:58.037597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1802,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183220+0000","last_change":"2024-04-26T16:42:21.377729+0000","last_active":"2024-04-26T16:43:16.183220+0000","last_peered":"2024-04-26T16:43:16.183220+0000","last_clean":"2024-04-26T16:43:16.183220+0000","last_became_active":"2024-04-26T16:35:25.391371+0000","last_became_peered":"2024-04-26T16:35:25.391371+0000","last_unstale":"2024-04-26T16:43:16.183220+0000","last_undegraded":"2024-04-26T16:43:16.183220+0000","last_fullsized":"2024-04-26T16:43:16.183220+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:21.377458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.377458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.768264+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"101'130","reported_seq":470,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889411+0000","last_change":"2024-04-26T16:42:41.830760+0000","last_active":"2024-04-26T16:43:15.889411+0000","last_peered":"2024-04-26T16:43:15.889411+0000","last_clean":"2024-04-26T16:43:15.889411+0000","last_became_active":"2024-04-26T16:38:46.975448+0000","last_became_peered":"2024-04-26T16:38:46.975448+0000","last_unstale":"2024-04-26T16:43:15.889411+0000","last_undegraded":"2024-04-26T16:43:15.889411+0000","last_fullsized":"2024-04-26T16:43:15.889411+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:41.830542+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.830542+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.945657+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1646,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.186096+0000","last_change":"2024-04-26T16:42:11.405087+0000","last_active":"2024-04-26T16:43:16.186096+0000","last_peered":"2024-04-26T16:43:16.186096+0000","last_clean":"2024-04-26T16:43:16.186096+0000","last_became_active":"2024-04-26T16:23:29.343592+0000","last_became_peered":"2024-04-26T16:23:29.343592+0000","last_unstale":"2024-04-26T16:43:16.186096+0000","last_undegraded":"2024-04-26T16:43:16.186096+0000","last_fullsized":"2024-04-26T16:43:16.186096+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:11.404886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:44.032590+0000","last_clean_scrub_stamp":"2024-04-26T16:42:11.404886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:18.443013+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1837,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888692+0000","last_change":"2024-04-26T16:43:13.000459+0000","last_active":"2024-04-26T16:43:15.888692+0000","last_peered":"2024-04-26T16:43:15.888692+0000","last_clean":"2024-04-26T16:43:15.888692+0000","last_became_active":"2024-04-26T16:17:33.060395+0000","last_became_peered":"2024-04-26T16:17:33.060395+0000","last_unstale":"2024-04-26T16:43:15.888692+0000","last_undegraded":"2024-04-26T16:43:15.888692+0000","last_fullsized":"2024-04-26T16:43:15.888692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:13.000251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:33:51.205695+0000","last_clean_scrub_stamp":"2024-04-26T16:43:13.000251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:28.678457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"107'6","reported_seq":3341,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180126+0000","last_change":"2024-04-26T16:42:59.582290+0000","last_active":"2024-04-26T16:43:16.180126+0000","last_peered":"2024-04-26T16:43:16.180126+0000","last_clean":"2024-04-26T16:43:16.180126+0000","last_became_active":"2024-04-26T16:17:32.040545+0000","last_became_peered":"2024-04-26T16:17:32.040545+0000","last_unstale":"2024-04-26T16:43:16.180126+0000","last_undegraded":"2024-04-26T16:43:16.180126+0000","last_fullsized":"2024-04-26T16:43:16.180126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'6","last_scrub_stamp":"2024-04-26T16:42:59.581912+0000","last_deep_scrub":"107'6","last_deep_scrub_stamp":"2024-04-26T16:21:31.994692+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.581912+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:27.614297+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0035983669999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1524,"num_read_kb":1519,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"857'26","reported_seq":1852,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181506+0000","last_change":"2024-04-26T16:42:53.742082+0000","last_active":"2024-04-26T16:43:16.181506+0000","last_peered":"2024-04-26T16:43:16.181506+0000","last_clean":"2024-04-26T16:43:16.181506+0000","last_became_active":"2024-04-26T16:17:31.055887+0000","last_became_peered":"2024-04-26T16:17:31.055887+0000","last_unstale":"2024-04-26T16:43:16.181506+0000","last_undegraded":"2024-04-26T16:43:16.181506+0000","last_fullsized":"2024-04-26T16:43:16.181506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'26","last_scrub_stamp":"2024-04-26T16:42:53.741837+0000","last_deep_scrub":"857'26","last_deep_scrub_stamp":"2024-04-26T16:38:31.384645+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.741837+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.841741+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'2","reported_seq":1844,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.895357+0000","last_change":"2024-04-26T16:42:10.822380+0000","last_active":"2024-04-26T16:43:15.895357+0000","last_peered":"2024-04-26T16:43:15.895357+0000","last_clean":"2024-04-26T16:43:15.895357+0000","last_became_active":"2024-04-26T16:35:25.398461+0000","last_became_peered":"2024-04-26T16:35:25.398461+0000","last_unstale":"2024-04-26T16:43:15.895357+0000","last_undegraded":"2024-04-26T16:43:15.895357+0000","last_fullsized":"2024-04-26T16:43:15.895357+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'2","last_scrub_stamp":"2024-04-26T16:42:10.822025+0000","last_deep_scrub":"103'2","last_deep_scrub_stamp":"2024-04-26T16:40:49.964369+0000","last_clean_scrub_stamp":"2024-04-26T16:42:10.822025+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:15.924116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1825,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889476+0000","last_change":"2024-04-26T16:42:23.918628+0000","last_active":"2024-04-26T16:43:15.889476+0000","last_peered":"2024-04-26T16:43:15.889476+0000","last_clean":"2024-04-26T16:43:15.889476+0000","last_became_active":"2024-04-26T16:17:35.070780+0000","last_became_peered":"2024-04-26T16:17:35.070780+0000","last_unstale":"2024-04-26T16:43:15.889476+0000","last_undegraded":"2024-04-26T16:43:15.889476+0000","last_fullsized":"2024-04-26T16:43:15.889476+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:23.918414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:47.089971+0000","last_clean_scrub_stamp":"2024-04-26T16:42:23.918414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:45.335918+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"101'130","reported_seq":466,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182939+0000","last_change":"2024-04-26T16:41:58.112705+0000","last_active":"2024-04-26T16:43:16.182939+0000","last_peered":"2024-04-26T16:43:16.182939+0000","last_clean":"2024-04-26T16:43:16.182939+0000","last_became_active":"2024-04-26T16:38:46.981826+0000","last_became_peered":"2024-04-26T16:38:46.981826+0000","last_unstale":"2024-04-26T16:43:16.182939+0000","last_undegraded":"2024-04-26T16:43:16.182939+0000","last_fullsized":"2024-04-26T16:43:16.182939+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:41:58.112409+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:41:58.112409+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.112409+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1673,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893180+0000","last_change":"2024-04-26T16:42:19.014379+0000","last_active":"2024-04-26T16:43:15.893180+0000","last_peered":"2024-04-26T16:43:15.893180+0000","last_clean":"2024-04-26T16:43:15.893180+0000","last_became_active":"2024-04-26T16:23:29.285178+0000","last_became_peered":"2024-04-26T16:23:29.285178+0000","last_unstale":"2024-04-26T16:43:15.893180+0000","last_undegraded":"2024-04-26T16:43:15.893180+0000","last_fullsized":"2024-04-26T16:43:15.893180+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:19.014047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:19.014047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.074321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1411,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180878+0000","last_change":"2024-04-26T16:42:07.158773+0000","last_active":"2024-04-26T16:43:16.180878+0000","last_peered":"2024-04-26T16:43:16.180878+0000","last_clean":"2024-04-26T16:43:16.180878+0000","last_became_active":"2024-04-26T16:35:25.404395+0000","last_became_peered":"2024-04-26T16:35:25.404395+0000","last_unstale":"2024-04-26T16:43:16.180878+0000","last_undegraded":"2024-04-26T16:43:16.180878+0000","last_fullsized":"2024-04-26T16:43:16.180878+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:07.158568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:45.092135+0000","last_clean_scrub_stamp":"2024-04-26T16:42:07.158568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"857'66","reported_seq":2291,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893366+0000","last_change":"2024-04-26T16:42:43.032147+0000","last_active":"2024-04-26T16:43:15.893366+0000","last_peered":"2024-04-26T16:43:15.893366+0000","last_clean":"2024-04-26T16:43:15.893366+0000","last_became_active":"2024-04-26T16:32:09.470474+0000","last_became_peered":"2024-04-26T16:32:09.470474+0000","last_unstale":"2024-04-26T16:43:15.893366+0000","last_undegraded":"2024-04-26T16:43:15.893366+0000","last_fullsized":"2024-04-26T16:43:15.893366+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'66","last_scrub_stamp":"2024-04-26T16:42:43.031896+0000","last_deep_scrub":"107'40","last_deep_scrub_stamp":"2024-04-26T16:21:57.094571+0000","last_clean_scrub_stamp":"2024-04-26T16:42:43.031896+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:03.076663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":87,"num_read_kb":72,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"465'2","reported_seq":1813,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184525+0000","last_change":"2024-04-26T16:42:40.428452+0000","last_active":"2024-04-26T16:43:16.184525+0000","last_peered":"2024-04-26T16:43:16.184525+0000","last_clean":"2024-04-26T16:43:16.184525+0000","last_became_active":"2024-04-26T16:17:29.025885+0000","last_became_peered":"2024-04-26T16:17:29.025885+0000","last_unstale":"2024-04-26T16:43:16.184525+0000","last_undegraded":"2024-04-26T16:43:16.184525+0000","last_fullsized":"2024-04-26T16:43:16.184525+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'2","last_scrub_stamp":"2024-04-26T16:42:40.428219+0000","last_deep_scrub":"465'2","last_deep_scrub_stamp":"2024-04-26T16:41:14.264394+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.428219+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.465194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1831,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890278+0000","last_change":"2024-04-26T16:42:39.131897+0000","last_active":"2024-04-26T16:43:15.890278+0000","last_peered":"2024-04-26T16:43:15.890278+0000","last_clean":"2024-04-26T16:43:15.890278+0000","last_became_active":"2024-04-26T16:35:25.411888+0000","last_became_peered":"2024-04-26T16:35:25.411888+0000","last_unstale":"2024-04-26T16:43:15.890278+0000","last_undegraded":"2024-04-26T16:43:15.890278+0000","last_fullsized":"2024-04-26T16:43:15.890278+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:39.131672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:39.131672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:39.161851+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"101'130","reported_seq":470,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.892196+0000","last_change":"2024-04-26T16:42:53.186092+0000","last_active":"2024-04-26T16:43:15.892196+0000","last_peered":"2024-04-26T16:43:15.892196+0000","last_clean":"2024-04-26T16:43:15.892196+0000","last_became_active":"2024-04-26T16:38:46.960634+0000","last_became_peered":"2024-04-26T16:38:46.960634+0000","last_unstale":"2024-04-26T16:43:15.892196+0000","last_undegraded":"2024-04-26T16:43:15.892196+0000","last_fullsized":"2024-04-26T16:43:15.892196+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:53.185871+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.185871+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:21.123382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1658,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182419+0000","last_change":"2024-04-26T16:43:01.638598+0000","last_active":"2024-04-26T16:43:16.182419+0000","last_peered":"2024-04-26T16:43:16.182419+0000","last_clean":"2024-04-26T16:43:16.182419+0000","last_became_active":"2024-04-26T16:35:25.417017+0000","last_became_peered":"2024-04-26T16:35:25.417017+0000","last_unstale":"2024-04-26T16:43:16.182419+0000","last_undegraded":"2024-04-26T16:43:16.182419+0000","last_fullsized":"2024-04-26T16:43:16.182419+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:01.638360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:43:01.638360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.000991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":751,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109965+0000","last_change":"2024-04-26T16:43:16.108609+0000","last_active":"2024-04-26T16:43:16.109965+0000","last_peered":"2024-04-26T16:43:16.109965+0000","last_clean":"2024-04-26T16:43:16.109965+0000","last_became_active":"2024-04-26T16:32:09.473793+0000","last_became_peered":"2024-04-26T16:32:09.473793+0000","last_unstale":"2024-04-26T16:43:16.109965+0000","last_undegraded":"2024-04-26T16:43:16.109965+0000","last_fullsized":"2024-04-26T16:43:16.109965+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:16.108067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:36.191214+0000","last_clean_scrub_stamp":"2024-04-26T16:43:16.108067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:37.350076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1796,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181435+0000","last_change":"2024-04-26T16:41:38.305650+0000","last_active":"2024-04-26T16:43:16.181435+0000","last_peered":"2024-04-26T16:43:16.181435+0000","last_clean":"2024-04-26T16:43:16.181435+0000","last_became_active":"2024-04-26T16:17:29.009637+0000","last_became_peered":"2024-04-26T16:17:29.009637+0000","last_unstale":"2024-04-26T16:43:16.181435+0000","last_undegraded":"2024-04-26T16:43:16.181435+0000","last_fullsized":"2024-04-26T16:43:16.181435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:38.305438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:26.045883+0000","last_clean_scrub_stamp":"2024-04-26T16:41:38.305438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"857'49","reported_seq":1894,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888805+0000","last_change":"2024-04-26T16:42:50.193055+0000","last_active":"2024-04-26T16:43:15.888805+0000","last_peered":"2024-04-26T16:43:15.888805+0000","last_clean":"2024-04-26T16:43:15.888805+0000","last_became_active":"2024-04-26T16:17:31.036476+0000","last_became_peered":"2024-04-26T16:17:31.036476+0000","last_unstale":"2024-04-26T16:43:15.888805+0000","last_undegraded":"2024-04-26T16:43:15.888805+0000","last_fullsized":"2024-04-26T16:43:15.888805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'49","last_scrub_stamp":"2024-04-26T16:42:50.192938+0000","last_deep_scrub":"112'21","last_deep_scrub_stamp":"2024-04-26T16:19:47.424244+0000","last_clean_scrub_stamp":"2024-04-26T16:42:50.192938+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:57.244039+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1842,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.894362+0000","last_change":"2024-04-26T16:43:02.006158+0000","last_active":"2024-04-26T16:43:15.894362+0000","last_peered":"2024-04-26T16:43:15.894362+0000","last_clean":"2024-04-26T16:43:15.894362+0000","last_became_active":"2024-04-26T16:32:09.470480+0000","last_became_peered":"2024-04-26T16:32:09.470480+0000","last_unstale":"2024-04-26T16:43:15.894362+0000","last_undegraded":"2024-04-26T16:43:15.894362+0000","last_fullsized":"2024-04-26T16:43:15.894362+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:02.005980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:37.388271+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.005980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:16.941966+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1678,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180306+0000","last_change":"2024-04-26T16:42:17.124471+0000","last_active":"2024-04-26T16:43:16.180306+0000","last_peered":"2024-04-26T16:43:16.180306+0000","last_clean":"2024-04-26T16:43:16.180306+0000","last_became_active":"2024-04-26T16:35:25.417111+0000","last_became_peered":"2024-04-26T16:35:25.417111+0000","last_unstale":"2024-04-26T16:43:16.180306+0000","last_undegraded":"2024-04-26T16:43:16.180306+0000","last_fullsized":"2024-04-26T16:43:16.180306+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:17.124345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:17.124345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.696793+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1808,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184727+0000","last_change":"2024-04-26T16:43:13.590513+0000","last_active":"2024-04-26T16:43:16.184727+0000","last_peered":"2024-04-26T16:43:16.184727+0000","last_clean":"2024-04-26T16:43:16.184727+0000","last_became_active":"2024-04-26T16:32:09.481709+0000","last_became_peered":"2024-04-26T16:32:09.481709+0000","last_unstale":"2024-04-26T16:43:16.184727+0000","last_undegraded":"2024-04-26T16:43:16.184727+0000","last_fullsized":"2024-04-26T16:43:16.184727+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:13.590301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:55.132763+0000","last_clean_scrub_stamp":"2024-04-26T16:43:13.590301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:38.891309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"101'130","reported_seq":2113,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181910+0000","last_change":"2024-04-26T16:43:09.523727+0000","last_active":"2024-04-26T16:43:16.181910+0000","last_peered":"2024-04-26T16:43:16.181910+0000","last_clean":"2024-04-26T16:43:16.181910+0000","last_became_active":"2024-04-26T16:38:41.889008+0000","last_became_peered":"2024-04-26T16:38:41.889008+0000","last_unstale":"2024-04-26T16:43:16.181910+0000","last_undegraded":"2024-04-26T16:43:16.181910+0000","last_fullsized":"2024-04-26T16:43:16.181910+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:43:09.523498+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:38:41.998073+0000","last_clean_scrub_stamp":"2024-04-26T16:43:09.523498+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:26.000093+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":743,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107386+0000","last_change":"2024-04-26T16:42:39.070105+0000","last_active":"2024-04-26T16:43:16.107386+0000","last_peered":"2024-04-26T16:43:16.107386+0000","last_clean":"2024-04-26T16:43:16.107386+0000","last_became_active":"2024-04-26T16:32:09.479713+0000","last_became_peered":"2024-04-26T16:32:09.479713+0000","last_unstale":"2024-04-26T16:43:16.107386+0000","last_undegraded":"2024-04-26T16:43:16.107386+0000","last_fullsized":"2024-04-26T16:43:16.107386+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:39.069892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:58.958586+0000","last_clean_scrub_stamp":"2024-04-26T16:42:39.069892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:52.572104+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"857'35","reported_seq":1913,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180334+0000","last_change":"2024-04-26T16:43:09.698482+0000","last_active":"2024-04-26T16:43:16.180334+0000","last_peered":"2024-04-26T16:43:16.180334+0000","last_clean":"2024-04-26T16:43:16.180334+0000","last_became_active":"2024-04-26T16:35:25.397667+0000","last_became_peered":"2024-04-26T16:35:25.397667+0000","last_unstale":"2024-04-26T16:43:16.180334+0000","last_undegraded":"2024-04-26T16:43:16.180334+0000","last_fullsized":"2024-04-26T16:43:16.180334+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'35","last_scrub_stamp":"2024-04-26T16:43:09.698244+0000","last_deep_scrub":"124'15","last_deep_scrub_stamp":"2024-04-26T16:21:36.925498+0000","last_clean_scrub_stamp":"2024-04-26T16:43:09.698244+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.037604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":751,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.110254+0000","last_change":"2024-04-26T16:42:43.092576+0000","last_active":"2024-04-26T16:43:16.110254+0000","last_peered":"2024-04-26T16:43:16.110254+0000","last_clean":"2024-04-26T16:43:16.110254+0000","last_became_active":"2024-04-26T16:32:09.465492+0000","last_became_peered":"2024-04-26T16:32:09.465492+0000","last_unstale":"2024-04-26T16:43:16.110254+0000","last_undegraded":"2024-04-26T16:43:16.110254+0000","last_fullsized":"2024-04-26T16:43:16.110254+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:43.092369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:40.165004+0000","last_clean_scrub_stamp":"2024-04-26T16:42:43.092369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.634594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"101'130","reported_seq":2120,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180681+0000","last_change":"2024-04-26T16:43:08.477327+0000","last_active":"2024-04-26T16:43:16.180681+0000","last_peered":"2024-04-26T16:43:16.180681+0000","last_clean":"2024-04-26T16:43:16.180681+0000","last_became_active":"2024-04-26T16:38:46.976486+0000","last_became_peered":"2024-04-26T16:38:46.976486+0000","last_unstale":"2024-04-26T16:43:16.180681+0000","last_undegraded":"2024-04-26T16:43:16.180681+0000","last_fullsized":"2024-04-26T16:43:16.180681+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:43:08.477053+0000","last_deep_scrub":"29'119","last_deep_scrub_stamp":"2024-04-26T16:14:39.826020+0000","last_clean_scrub_stamp":"2024-04-26T16:43:08.477053+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:18.374542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1678,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.179397+0000","last_change":"2024-04-26T16:42:40.951206+0000","last_active":"2024-04-26T16:43:16.179397+0000","last_peered":"2024-04-26T16:43:16.179397+0000","last_clean":"2024-04-26T16:43:16.179397+0000","last_became_active":"2024-04-26T16:35:25.392163+0000","last_became_peered":"2024-04-26T16:35:25.392163+0000","last_unstale":"2024-04-26T16:43:16.179397+0000","last_undegraded":"2024-04-26T16:43:16.179397+0000","last_fullsized":"2024-04-26T16:43:16.179397+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:40.950857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.950857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.095107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1840,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184743+0000","last_change":"2024-04-26T16:42:15.044914+0000","last_active":"2024-04-26T16:43:16.184743+0000","last_peered":"2024-04-26T16:43:16.184743+0000","last_clean":"2024-04-26T16:43:16.184743+0000","last_became_active":"2024-04-26T16:32:09.482725+0000","last_became_peered":"2024-04-26T16:32:09.482725+0000","last_unstale":"2024-04-26T16:43:16.184743+0000","last_undegraded":"2024-04-26T16:43:16.184743+0000","last_fullsized":"2024-04-26T16:43:16.184743+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:15.044693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:34.426484+0000","last_clean_scrub_stamp":"2024-04-26T16:42:15.044693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:26.315965+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1837,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183530+0000","last_change":"2024-04-26T16:42:24.173831+0000","last_active":"2024-04-26T16:43:16.183530+0000","last_peered":"2024-04-26T16:43:16.183530+0000","last_clean":"2024-04-26T16:43:16.183530+0000","last_became_active":"2024-04-26T16:17:29.006373+0000","last_became_peered":"2024-04-26T16:17:29.006373+0000","last_unstale":"2024-04-26T16:43:16.183530+0000","last_undegraded":"2024-04-26T16:43:16.183530+0000","last_fullsized":"2024-04-26T16:43:16.183530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:24.173543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:24.173543+0000","last_clean_scrub_stamp":"2024-04-26T16:42:24.173543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.202289+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"857'50","reported_seq":1923,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889810+0000","last_change":"2024-04-26T16:42:30.042965+0000","last_active":"2024-04-26T16:43:15.889810+0000","last_peered":"2024-04-26T16:43:15.889810+0000","last_clean":"2024-04-26T16:43:15.889810+0000","last_became_active":"2024-04-26T16:35:25.412117+0000","last_became_peered":"2024-04-26T16:35:25.412117+0000","last_unstale":"2024-04-26T16:43:15.889810+0000","last_undegraded":"2024-04-26T16:43:15.889810+0000","last_fullsized":"2024-04-26T16:43:15.889810+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'50","last_scrub_stamp":"2024-04-26T16:42:30.042852+0000","last_deep_scrub":"857'50","last_deep_scrub_stamp":"2024-04-26T16:40:53.922832+0000","last_clean_scrub_stamp":"2024-04-26T16:42:30.042852+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.618312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1794,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183862+0000","last_change":"2024-04-26T16:41:43.440279+0000","last_active":"2024-04-26T16:43:16.183862+0000","last_peered":"2024-04-26T16:43:16.183862+0000","last_clean":"2024-04-26T16:43:16.183862+0000","last_became_active":"2024-04-26T16:35:25.398515+0000","last_became_peered":"2024-04-26T16:35:25.398515+0000","last_unstale":"2024-04-26T16:43:16.183862+0000","last_undegraded":"2024-04-26T16:43:16.183862+0000","last_fullsized":"2024-04-26T16:43:16.183862+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:43.439979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:41:43.439979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"101'130","reported_seq":469,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.106212+0000","last_change":"2024-04-26T16:43:02.086808+0000","last_active":"2024-04-26T16:43:16.106212+0000","last_peered":"2024-04-26T16:43:16.106212+0000","last_clean":"2024-04-26T16:43:16.106212+0000","last_became_active":"2024-04-26T16:38:46.969238+0000","last_became_peered":"2024-04-26T16:38:46.969238+0000","last_unstale":"2024-04-26T16:43:16.106212+0000","last_undegraded":"2024-04-26T16:43:16.106212+0000","last_fullsized":"2024-04-26T16:43:16.106212+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":4,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:43:02.086564+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:43:02.086564+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.086564+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:02.565058+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1647,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180218+0000","last_change":"2024-04-26T16:41:52.947173+0000","last_active":"2024-04-26T16:43:16.180218+0000","last_peered":"2024-04-26T16:43:16.180218+0000","last_clean":"2024-04-26T16:43:16.180218+0000","last_became_active":"2024-04-26T16:23:29.219427+0000","last_became_peered":"2024-04-26T16:23:29.219427+0000","last_unstale":"2024-04-26T16:43:16.180218+0000","last_undegraded":"2024-04-26T16:43:16.180218+0000","last_fullsized":"2024-04-26T16:43:16.180218+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:52.946928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:25.114078+0000","last_clean_scrub_stamp":"2024-04-26T16:41:52.946928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1820,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.892708+0000","last_change":"2024-04-26T16:42:18.076002+0000","last_active":"2024-04-26T16:43:15.892708+0000","last_peered":"2024-04-26T16:43:15.892708+0000","last_clean":"2024-04-26T16:43:15.892708+0000","last_became_active":"2024-04-26T16:17:33.074576+0000","last_became_peered":"2024-04-26T16:17:33.074576+0000","last_unstale":"2024-04-26T16:43:15.892708+0000","last_undegraded":"2024-04-26T16:43:15.892708+0000","last_fullsized":"2024-04-26T16:43:15.892708+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:18.075781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:18.075781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:26.483866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"857'47","reported_seq":1909,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889715+0000","last_change":"2024-04-26T16:42:24.964533+0000","last_active":"2024-04-26T16:43:15.889715+0000","last_peered":"2024-04-26T16:43:15.889715+0000","last_clean":"2024-04-26T16:43:15.889715+0000","last_became_active":"2024-04-26T16:35:25.395746+0000","last_became_peered":"2024-04-26T16:35:25.395746+0000","last_unstale":"2024-04-26T16:43:15.889715+0000","last_undegraded":"2024-04-26T16:43:15.889715+0000","last_fullsized":"2024-04-26T16:43:15.889715+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'47","last_scrub_stamp":"2024-04-26T16:42:24.964285+0000","last_deep_scrub":"388'37","last_deep_scrub_stamp":"2024-04-26T16:31:38.498802+0000","last_clean_scrub_stamp":"2024-04-26T16:42:24.964285+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:50.639776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":744,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107819+0000","last_change":"2024-04-26T16:42:32.101660+0000","last_active":"2024-04-26T16:43:16.107819+0000","last_peered":"2024-04-26T16:43:16.107819+0000","last_clean":"2024-04-26T16:43:16.107819+0000","last_became_active":"2024-04-26T16:35:25.397940+0000","last_became_peered":"2024-04-26T16:35:25.397940+0000","last_unstale":"2024-04-26T16:43:16.107819+0000","last_undegraded":"2024-04-26T16:43:16.107819+0000","last_fullsized":"2024-04-26T16:43:16.107819+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:32.101453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:40.662149+0000","last_clean_scrub_stamp":"2024-04-26T16:42:32.101453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:39.430923+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1800,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181229+0000","last_change":"2024-04-26T16:41:50.960451+0000","last_active":"2024-04-26T16:43:16.181229+0000","last_peered":"2024-04-26T16:43:16.181229+0000","last_clean":"2024-04-26T16:43:16.181229+0000","last_became_active":"2024-04-26T16:17:35.089967+0000","last_became_peered":"2024-04-26T16:17:35.089967+0000","last_unstale":"2024-04-26T16:43:16.181229+0000","last_undegraded":"2024-04-26T16:43:16.181229+0000","last_fullsized":"2024-04-26T16:43:16.181229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:50.960209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.960209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.7","version":"101'130","reported_seq":474,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183154+0000","last_change":"2024-04-26T16:42:10.128949+0000","last_active":"2024-04-26T16:43:16.183154+0000","last_peered":"2024-04-26T16:43:16.183154+0000","last_clean":"2024-04-26T16:43:16.183154+0000","last_became_active":"2024-04-26T16:38:46.971719+0000","last_became_peered":"2024-04-26T16:38:46.971719+0000","last_unstale":"2024-04-26T16:43:16.183154+0000","last_undegraded":"2024-04-26T16:43:16.183154+0000","last_fullsized":"2024-04-26T16:43:16.183154+0000","mapping_epoch":886,"log_start":"0'0","ondisk_log_start":"0'0","created":881,"last_epoch_clean":887,"parent":"0.0","parent_split_bits":3,"last_scrub":"101'130","last_scrub_stamp":"2024-04-26T16:42:10.128732+0000","last_deep_scrub":"101'130","last_deep_scrub_stamp":"2024-04-26T16:42:10.128732+0000","last_clean_scrub_stamp":"2024-04-26T16:42:10.128732+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.726045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1681,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183945+0000","last_change":"2024-04-26T16:42:59.158057+0000","last_active":"2024-04-26T16:43:16.183945+0000","last_peered":"2024-04-26T16:43:16.183945+0000","last_clean":"2024-04-26T16:43:16.183945+0000","last_became_active":"2024-04-26T16:35:25.390196+0000","last_became_peered":"2024-04-26T16:35:25.390196+0000","last_unstale":"2024-04-26T16:43:16.183945+0000","last_undegraded":"2024-04-26T16:43:16.183945+0000","last_fullsized":"2024-04-26T16:43:16.183945+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:59.157862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.157862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:03.795100+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"907'20","reported_seq":2180,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.891625+0000","last_change":"2024-04-26T16:43:05.299518+0000","last_active":"2024-04-26T16:43:15.891625+0000","last_peered":"2024-04-26T16:43:15.891625+0000","last_clean":"2024-04-26T16:43:15.891625+0000","last_became_active":"2024-04-26T16:35:25.404413+0000","last_became_peered":"2024-04-26T16:35:25.404413+0000","last_unstale":"2024-04-26T16:43:15.891625+0000","last_undegraded":"2024-04-26T16:43:15.891625+0000","last_fullsized":"2024-04-26T16:43:15.891625+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"907'20","last_scrub_stamp":"2024-04-26T16:43:05.299269+0000","last_deep_scrub":"907'20","last_deep_scrub_stamp":"2024-04-26T16:40:17.078066+0000","last_clean_scrub_stamp":"2024-04-26T16:43:05.299269+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:34.269320+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"922'3","reported_seq":1865,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.891835+0000","last_change":"2024-04-26T16:42:54.181861+0000","last_active":"2024-04-26T16:43:15.891835+0000","last_peered":"2024-04-26T16:43:15.891835+0000","last_clean":"2024-04-26T16:43:15.891835+0000","last_became_active":"2024-04-26T16:32:09.451522+0000","last_became_peered":"2024-04-26T16:32:09.451522+0000","last_unstale":"2024-04-26T16:43:15.891835+0000","last_undegraded":"2024-04-26T16:43:15.891835+0000","last_fullsized":"2024-04-26T16:43:15.891835+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"922'3","last_scrub_stamp":"2024-04-26T16:42:54.181739+0000","last_deep_scrub":"922'3","last_deep_scrub_stamp":"2024-04-26T16:41:32.972348+0000","last_clean_scrub_stamp":"2024-04-26T16:42:54.181739+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.692136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0021052589999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"857'52","reported_seq":913,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109217+0000","last_change":"2024-04-26T16:42:00.059382+0000","last_active":"2024-04-26T16:43:16.109217+0000","last_peered":"2024-04-26T16:43:16.109217+0000","last_clean":"2024-04-26T16:43:16.109217+0000","last_became_active":"2024-04-26T16:32:09.487654+0000","last_became_peered":"2024-04-26T16:32:09.487654+0000","last_unstale":"2024-04-26T16:43:16.109217+0000","last_undegraded":"2024-04-26T16:43:16.109217+0000","last_fullsized":"2024-04-26T16:43:16.109217+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'52","last_scrub_stamp":"2024-04-26T16:42:00.059136+0000","last_deep_scrub":"329'25","last_deep_scrub_stamp":"2024-04-26T16:26:37.181705+0000","last_clean_scrub_stamp":"2024-04-26T16:42:00.059136+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:24.646186+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1849,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893563+0000","last_change":"2024-04-26T16:43:08.942623+0000","last_active":"2024-04-26T16:43:15.893563+0000","last_peered":"2024-04-26T16:43:15.893563+0000","last_clean":"2024-04-26T16:43:15.893563+0000","last_became_active":"2024-04-26T16:17:29.009456+0000","last_became_peered":"2024-04-26T16:17:29.009456+0000","last_unstale":"2024-04-26T16:43:15.893563+0000","last_undegraded":"2024-04-26T16:43:15.893563+0000","last_fullsized":"2024-04-26T16:43:15.893563+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:08.942391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:23.108346+0000","last_clean_scrub_stamp":"2024-04-26T16:43:08.942391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:17.096559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1816,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.887648+0000","last_change":"2024-04-26T16:43:03.293681+0000","last_active":"2024-04-26T16:43:15.887648+0000","last_peered":"2024-04-26T16:43:15.887648+0000","last_clean":"2024-04-26T16:43:15.887648+0000","last_became_active":"2024-04-26T16:17:35.076938+0000","last_became_peered":"2024-04-26T16:17:35.076938+0000","last_unstale":"2024-04-26T16:43:15.887648+0000","last_undegraded":"2024-04-26T16:43:15.887648+0000","last_fullsized":"2024-04-26T16:43:15.887648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:03.293464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:56.011726+0000","last_clean_scrub_stamp":"2024-04-26T16:43:03.293464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:15.447055+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1671,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893071+0000","last_change":"2024-04-26T16:42:14.096605+0000","last_active":"2024-04-26T16:43:15.893071+0000","last_peered":"2024-04-26T16:43:15.893071+0000","last_clean":"2024-04-26T16:43:15.893071+0000","last_became_active":"2024-04-26T16:23:29.328872+0000","last_became_peered":"2024-04-26T16:23:29.328872+0000","last_unstale":"2024-04-26T16:43:15.893071+0000","last_undegraded":"2024-04-26T16:43:15.893071+0000","last_fullsized":"2024-04-26T16:43:15.893071+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:14.096352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:14.096352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.324217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1828,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182804+0000","last_change":"2024-04-26T16:42:29.230773+0000","last_active":"2024-04-26T16:43:16.182804+0000","last_peered":"2024-04-26T16:43:16.182804+0000","last_clean":"2024-04-26T16:43:16.182804+0000","last_became_active":"2024-04-26T16:17:33.063162+0000","last_became_peered":"2024-04-26T16:17:33.063162+0000","last_unstale":"2024-04-26T16:43:16.182804+0000","last_undegraded":"2024-04-26T16:43:16.182804+0000","last_fullsized":"2024-04-26T16:43:16.182804+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:29.230451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:01.923293+0000","last_clean_scrub_stamp":"2024-04-26T16:42:29.230451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:40.825701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"857'27","reported_seq":1845,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185564+0000","last_change":"2024-04-26T16:42:24.358367+0000","last_active":"2024-04-26T16:43:16.185564+0000","last_peered":"2024-04-26T16:43:16.185564+0000","last_clean":"2024-04-26T16:43:16.185564+0000","last_became_active":"2024-04-26T16:17:31.042886+0000","last_became_peered":"2024-04-26T16:17:31.042886+0000","last_unstale":"2024-04-26T16:43:16.185564+0000","last_undegraded":"2024-04-26T16:43:16.185564+0000","last_fullsized":"2024-04-26T16:43:16.185564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:42:24.358083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:24.358083+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.171772+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":1860,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890610+0000","last_change":"2024-04-26T16:42:27.974961+0000","last_active":"2024-04-26T16:43:15.890610+0000","last_peered":"2024-04-26T16:43:15.890610+0000","last_clean":"2024-04-26T16:43:15.890610+0000","last_became_active":"2024-04-26T16:17:29.010016+0000","last_became_peered":"2024-04-26T16:17:29.010016+0000","last_unstale":"2024-04-26T16:43:15.890610+0000","last_undegraded":"2024-04-26T16:43:15.890610+0000","last_fullsized":"2024-04-26T16:43:15.890610+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'2","last_scrub_stamp":"2024-04-26T16:42:27.974720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:27.974720+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.614473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1792,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181170+0000","last_change":"2024-04-26T16:41:59.907623+0000","last_active":"2024-04-26T16:43:16.181170+0000","last_peered":"2024-04-26T16:43:16.181170+0000","last_clean":"2024-04-26T16:43:16.181170+0000","last_became_active":"2024-04-26T16:17:35.094569+0000","last_became_peered":"2024-04-26T16:17:35.094569+0000","last_unstale":"2024-04-26T16:43:16.181170+0000","last_undegraded":"2024-04-26T16:43:16.181170+0000","last_fullsized":"2024-04-26T16:43:16.181170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:59.907452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:41.021797+0000","last_clean_scrub_stamp":"2024-04-26T16:41:59.907452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:24.017201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1670,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888101+0000","last_change":"2024-04-26T16:42:42.234101+0000","last_active":"2024-04-26T16:43:15.888101+0000","last_peered":"2024-04-26T16:43:15.888101+0000","last_clean":"2024-04-26T16:43:15.888101+0000","last_became_active":"2024-04-26T16:23:29.222144+0000","last_became_peered":"2024-04-26T16:23:29.222144+0000","last_unstale":"2024-04-26T16:43:15.888101+0000","last_undegraded":"2024-04-26T16:43:15.888101+0000","last_fullsized":"2024-04-26T16:43:15.888101+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:42.233896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:42.233896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:53.276863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"857'27","reported_seq":1852,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182417+0000","last_change":"2024-04-26T16:42:58.338869+0000","last_active":"2024-04-26T16:43:16.182417+0000","last_peered":"2024-04-26T16:43:16.182417+0000","last_clean":"2024-04-26T16:43:16.182417+0000","last_became_active":"2024-04-26T16:35:25.395993+0000","last_became_peered":"2024-04-26T16:35:25.395993+0000","last_unstale":"2024-04-26T16:43:16.182417+0000","last_undegraded":"2024-04-26T16:43:16.182417+0000","last_fullsized":"2024-04-26T16:43:16.182417+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'27","last_scrub_stamp":"2024-04-26T16:42:58.338541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:58.338541+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:15.479805+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1844,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182719+0000","last_change":"2024-04-26T16:43:03.201975+0000","last_active":"2024-04-26T16:43:16.182719+0000","last_peered":"2024-04-26T16:43:16.182719+0000","last_clean":"2024-04-26T16:43:16.182719+0000","last_became_active":"2024-04-26T16:17:29.008678+0000","last_became_peered":"2024-04-26T16:17:29.008678+0000","last_unstale":"2024-04-26T16:43:16.182719+0000","last_undegraded":"2024-04-26T16:43:16.182719+0000","last_fullsized":"2024-04-26T16:43:16.182719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:03.201864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:29:07.075499+0000","last_clean_scrub_stamp":"2024-04-26T16:43:03.201864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:20.965555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1795,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181839+0000","last_change":"2024-04-26T16:42:02.906333+0000","last_active":"2024-04-26T16:43:16.181839+0000","last_peered":"2024-04-26T16:43:16.181839+0000","last_clean":"2024-04-26T16:43:16.181839+0000","last_became_active":"2024-04-26T16:17:33.068823+0000","last_became_peered":"2024-04-26T16:17:33.068823+0000","last_unstale":"2024-04-26T16:43:16.181839+0000","last_undegraded":"2024-04-26T16:43:16.181839+0000","last_fullsized":"2024-04-26T16:43:16.181839+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:02.906110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.910474+0000","last_clean_scrub_stamp":"2024-04-26T16:42:02.906110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:28.025610+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1824,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184916+0000","last_change":"2024-04-26T16:43:07.165996+0000","last_active":"2024-04-26T16:43:16.184916+0000","last_peered":"2024-04-26T16:43:16.184916+0000","last_clean":"2024-04-26T16:43:16.184916+0000","last_became_active":"2024-04-26T16:17:35.083822+0000","last_became_peered":"2024-04-26T16:17:35.083822+0000","last_unstale":"2024-04-26T16:43:16.184916+0000","last_undegraded":"2024-04-26T16:43:16.184916+0000","last_fullsized":"2024-04-26T16:43:16.184916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:07.165774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:23.952025+0000","last_clean_scrub_stamp":"2024-04-26T16:43:07.165774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:24.512273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1681,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182459+0000","last_change":"2024-04-26T16:42:01.034407+0000","last_active":"2024-04-26T16:43:16.182459+0000","last_peered":"2024-04-26T16:43:16.182459+0000","last_clean":"2024-04-26T16:43:16.182459+0000","last_became_active":"2024-04-26T16:35:25.399182+0000","last_became_peered":"2024-04-26T16:35:25.399182+0000","last_unstale":"2024-04-26T16:43:16.182459+0000","last_undegraded":"2024-04-26T16:43:16.182459+0000","last_fullsized":"2024-04-26T16:43:16.182459+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:01.034193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:01.034193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.517517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"893'14","reported_seq":1562,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180559+0000","last_change":"2024-04-26T16:43:09.215372+0000","last_active":"2024-04-26T16:43:16.180559+0000","last_peered":"2024-04-26T16:43:16.180559+0000","last_clean":"2024-04-26T16:43:16.180559+0000","last_became_active":"2024-04-26T16:35:25.404028+0000","last_became_peered":"2024-04-26T16:35:25.404028+0000","last_unstale":"2024-04-26T16:43:16.180559+0000","last_undegraded":"2024-04-26T16:43:16.180559+0000","last_fullsized":"2024-04-26T16:43:16.180559+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"893'14","last_scrub_stamp":"2024-04-26T16:43:09.215145+0000","last_deep_scrub":"893'14","last_deep_scrub_stamp":"2024-04-26T16:40:37.041551+0000","last_clean_scrub_stamp":"2024-04-26T16:43:09.215145+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:37.962836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1397,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181336+0000","last_change":"2024-04-26T16:42:16.181728+0000","last_active":"2024-04-26T16:43:16.181336+0000","last_peered":"2024-04-26T16:43:16.181336+0000","last_clean":"2024-04-26T16:43:16.181336+0000","last_became_active":"2024-04-26T16:35:25.403847+0000","last_became_peered":"2024-04-26T16:35:25.403847+0000","last_unstale":"2024-04-26T16:43:16.181336+0000","last_undegraded":"2024-04-26T16:43:16.181336+0000","last_fullsized":"2024-04-26T16:43:16.181336+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:16.181526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:30:28.355929+0000","last_clean_scrub_stamp":"2024-04-26T16:42:16.181526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:43.662859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":750,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109168+0000","last_change":"2024-04-26T16:43:01.102297+0000","last_active":"2024-04-26T16:43:16.109168+0000","last_peered":"2024-04-26T16:43:16.109168+0000","last_clean":"2024-04-26T16:43:16.109168+0000","last_became_active":"2024-04-26T16:32:09.468021+0000","last_became_peered":"2024-04-26T16:32:09.468021+0000","last_unstale":"2024-04-26T16:43:16.109168+0000","last_undegraded":"2024-04-26T16:43:16.109168+0000","last_fullsized":"2024-04-26T16:43:16.109168+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:01.102083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:43:01.102083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:05.943136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1800,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183790+0000","last_change":"2024-04-26T16:43:12.566189+0000","last_active":"2024-04-26T16:43:16.183790+0000","last_peered":"2024-04-26T16:43:16.183790+0000","last_clean":"2024-04-26T16:43:16.183790+0000","last_became_active":"2024-04-26T16:35:25.379432+0000","last_became_peered":"2024-04-26T16:35:25.379432+0000","last_unstale":"2024-04-26T16:43:16.183790+0000","last_undegraded":"2024-04-26T16:43:16.183790+0000","last_fullsized":"2024-04-26T16:43:16.183790+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:12.565977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:43:12.565977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:17.152557+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1676,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889119+0000","last_change":"2024-04-26T16:42:06.759963+0000","last_active":"2024-04-26T16:43:15.889119+0000","last_peered":"2024-04-26T16:43:15.889119+0000","last_clean":"2024-04-26T16:43:15.889119+0000","last_became_active":"2024-04-26T16:23:29.295071+0000","last_became_peered":"2024-04-26T16:23:29.295071+0000","last_unstale":"2024-04-26T16:43:15.889119+0000","last_undegraded":"2024-04-26T16:43:15.889119+0000","last_fullsized":"2024-04-26T16:43:15.889119+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:06.759748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:06.759748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:25.209723+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"931'32","reported_seq":1865,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.183144+0000","last_change":"2024-04-26T16:42:55.366286+0000","last_active":"2024-04-26T16:43:16.183144+0000","last_peered":"2024-04-26T16:43:16.183144+0000","last_clean":"2024-04-26T16:43:16.183144+0000","last_became_active":"2024-04-26T16:26:27.130950+0000","last_became_peered":"2024-04-26T16:26:27.130950+0000","last_unstale":"2024-04-26T16:43:16.183144+0000","last_undegraded":"2024-04-26T16:43:16.183144+0000","last_fullsized":"2024-04-26T16:43:16.183144+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'32","last_scrub_stamp":"2024-04-26T16:42:55.366026+0000","last_deep_scrub":"931'32","last_deep_scrub_stamp":"2024-04-26T16:42:55.366026+0000","last_clean_scrub_stamp":"2024-04-26T16:42:55.366026+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.663187+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":750,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107467+0000","last_change":"2024-04-26T16:42:36.117791+0000","last_active":"2024-04-26T16:43:16.107467+0000","last_peered":"2024-04-26T16:43:16.107467+0000","last_clean":"2024-04-26T16:43:16.107467+0000","last_became_active":"2024-04-26T16:32:09.458746+0000","last_became_peered":"2024-04-26T16:32:09.458746+0000","last_unstale":"2024-04-26T16:43:16.107467+0000","last_undegraded":"2024-04-26T16:43:16.107467+0000","last_fullsized":"2024-04-26T16:43:16.107467+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:36.117506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:00.083131+0000","last_clean_scrub_stamp":"2024-04-26T16:42:36.117506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.369500+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"585'14","reported_seq":880,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.108980+0000","last_change":"2024-04-26T16:41:50.006070+0000","last_active":"2024-04-26T16:43:16.108980+0000","last_peered":"2024-04-26T16:43:16.108980+0000","last_clean":"2024-04-26T16:43:16.108980+0000","last_became_active":"2024-04-26T16:32:09.470734+0000","last_became_peered":"2024-04-26T16:32:09.470734+0000","last_unstale":"2024-04-26T16:43:16.108980+0000","last_undegraded":"2024-04-26T16:43:16.108980+0000","last_fullsized":"2024-04-26T16:43:16.108980+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"585'14","last_scrub_stamp":"2024-04-26T16:41:50.005835+0000","last_deep_scrub":"585'14","last_deep_scrub_stamp":"2024-04-26T16:39:10.124243+0000","last_clean_scrub_stamp":"2024-04-26T16:41:50.005835+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:16.684246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1806,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.894029+0000","last_change":"2024-04-26T16:42:47.190180+0000","last_active":"2024-04-26T16:43:15.894029+0000","last_peered":"2024-04-26T16:43:15.894029+0000","last_clean":"2024-04-26T16:43:15.894029+0000","last_became_active":"2024-04-26T16:17:35.077956+0000","last_became_peered":"2024-04-26T16:17:35.077956+0000","last_unstale":"2024-04-26T16:43:15.894029+0000","last_undegraded":"2024-04-26T16:43:15.894029+0000","last_fullsized":"2024-04-26T16:43:15.894029+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:47.189962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:47.189962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:08.768493+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":750,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107202+0000","last_change":"2024-04-26T16:42:59.149815+0000","last_active":"2024-04-26T16:43:16.107202+0000","last_peered":"2024-04-26T16:43:16.107202+0000","last_clean":"2024-04-26T16:43:16.107202+0000","last_became_active":"2024-04-26T16:32:09.482357+0000","last_became_peered":"2024-04-26T16:32:09.482357+0000","last_unstale":"2024-04-26T16:43:16.107202+0000","last_undegraded":"2024-04-26T16:43:16.107202+0000","last_fullsized":"2024-04-26T16:43:16.107202+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:59.149587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:59.149587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:02.335080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"857'25","reported_seq":1853,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888737+0000","last_change":"2024-04-26T16:42:21.117872+0000","last_active":"2024-04-26T16:43:15.888737+0000","last_peered":"2024-04-26T16:43:15.888737+0000","last_clean":"2024-04-26T16:43:15.888737+0000","last_became_active":"2024-04-26T16:17:31.047171+0000","last_became_peered":"2024-04-26T16:17:31.047171+0000","last_unstale":"2024-04-26T16:43:15.888737+0000","last_undegraded":"2024-04-26T16:43:15.888737+0000","last_fullsized":"2024-04-26T16:43:15.888737+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:42:21.117739+0000","last_deep_scrub":"121'13","last_deep_scrub_stamp":"2024-04-26T16:21:13.384551+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.117739+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.191760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1827,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890748+0000","last_change":"2024-04-26T16:43:02.339237+0000","last_active":"2024-04-26T16:43:15.890748+0000","last_peered":"2024-04-26T16:43:15.890748+0000","last_clean":"2024-04-26T16:43:15.890748+0000","last_became_active":"2024-04-26T16:17:29.000712+0000","last_became_peered":"2024-04-26T16:17:29.000712+0000","last_unstale":"2024-04-26T16:43:15.890748+0000","last_undegraded":"2024-04-26T16:43:15.890748+0000","last_fullsized":"2024-04-26T16:43:15.890748+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:02.338995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:45.880889+0000","last_clean_scrub_stamp":"2024-04-26T16:43:02.338995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:32.216415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"928'12","reported_seq":2117,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181898+0000","last_change":"2024-04-26T16:42:25.031164+0000","last_active":"2024-04-26T16:43:16.181898+0000","last_peered":"2024-04-26T16:43:16.181898+0000","last_clean":"2024-04-26T16:43:16.181898+0000","last_became_active":"2024-04-26T16:17:33.065433+0000","last_became_peered":"2024-04-26T16:17:33.065433+0000","last_unstale":"2024-04-26T16:43:16.181898+0000","last_undegraded":"2024-04-26T16:43:16.181898+0000","last_fullsized":"2024-04-26T16:43:16.181898+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'12","last_scrub_stamp":"2024-04-26T16:42:25.030750+0000","last_deep_scrub":"928'12","last_deep_scrub_stamp":"2024-04-26T16:42:25.030750+0000","last_clean_scrub_stamp":"2024-04-26T16:42:25.030750+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:35.406231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1835,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.891334+0000","last_change":"2024-04-26T16:42:05.717465+0000","last_active":"2024-04-26T16:43:15.891334+0000","last_peered":"2024-04-26T16:43:15.891334+0000","last_clean":"2024-04-26T16:43:15.891334+0000","last_became_active":"2024-04-26T16:32:09.467351+0000","last_became_peered":"2024-04-26T16:32:09.467351+0000","last_unstale":"2024-04-26T16:43:15.891334+0000","last_undegraded":"2024-04-26T16:43:15.891334+0000","last_fullsized":"2024-04-26T16:43:15.891334+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:05.717242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:05.717242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1645,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184997+0000","last_change":"2024-04-26T16:43:07.436998+0000","last_active":"2024-04-26T16:43:16.184997+0000","last_peered":"2024-04-26T16:43:16.184997+0000","last_clean":"2024-04-26T16:43:16.184997+0000","last_became_active":"2024-04-26T16:23:29.247963+0000","last_became_peered":"2024-04-26T16:23:29.247963+0000","last_unstale":"2024-04-26T16:43:16.184997+0000","last_undegraded":"2024-04-26T16:43:16.184997+0000","last_fullsized":"2024-04-26T16:43:16.184997+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:07.436793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:18.286808+0000","last_clean_scrub_stamp":"2024-04-26T16:43:07.436793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:16.203943+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"857'25","reported_seq":1836,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185152+0000","last_change":"2024-04-26T16:42:20.372382+0000","last_active":"2024-04-26T16:43:16.185152+0000","last_peered":"2024-04-26T16:43:16.185152+0000","last_clean":"2024-04-26T16:43:16.185152+0000","last_became_active":"2024-04-26T16:17:31.040299+0000","last_became_peered":"2024-04-26T16:17:31.040299+0000","last_unstale":"2024-04-26T16:43:16.185152+0000","last_undegraded":"2024-04-26T16:43:16.185152+0000","last_fullsized":"2024-04-26T16:43:16.185152+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'25","last_scrub_stamp":"2024-04-26T16:42:20.372145+0000","last_deep_scrub":"857'25","last_deep_scrub_stamp":"2024-04-26T16:39:29.951322+0000","last_clean_scrub_stamp":"2024-04-26T16:42:20.372145+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.006718+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1479,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182718+0000","last_change":"2024-04-26T16:42:53.228898+0000","last_active":"2024-04-26T16:43:16.182718+0000","last_peered":"2024-04-26T16:43:16.182718+0000","last_clean":"2024-04-26T16:43:16.182718+0000","last_became_active":"2024-04-26T16:35:25.401229+0000","last_became_peered":"2024-04-26T16:35:25.401229+0000","last_unstale":"2024-04-26T16:43:16.182718+0000","last_undegraded":"2024-04-26T16:43:16.182718+0000","last_fullsized":"2024-04-26T16:43:16.182718+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.228700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:25.117353+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.228700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:03.005174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1812,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180985+0000","last_change":"2024-04-26T16:43:00.585299+0000","last_active":"2024-04-26T16:43:16.180985+0000","last_peered":"2024-04-26T16:43:16.180985+0000","last_clean":"2024-04-26T16:43:16.180985+0000","last_became_active":"2024-04-26T16:17:33.061089+0000","last_became_peered":"2024-04-26T16:17:33.061089+0000","last_unstale":"2024-04-26T16:43:16.180985+0000","last_undegraded":"2024-04-26T16:43:16.180985+0000","last_fullsized":"2024-04-26T16:43:16.180985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:00.585093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:43:00.585093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:18.901570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1836,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888752+0000","last_change":"2024-04-26T16:41:58.032603+0000","last_active":"2024-04-26T16:43:15.888752+0000","last_peered":"2024-04-26T16:43:15.888752+0000","last_clean":"2024-04-26T16:43:15.888752+0000","last_became_active":"2024-04-26T16:32:09.448873+0000","last_became_peered":"2024-04-26T16:32:09.448873+0000","last_unstale":"2024-04-26T16:43:15.888752+0000","last_undegraded":"2024-04-26T16:43:15.888752+0000","last_fullsized":"2024-04-26T16:43:15.888752+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:58.032373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:20.888123+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.032373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1296,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182730+0000","last_change":"2024-04-26T16:43:01.181943+0000","last_active":"2024-04-26T16:43:16.182730+0000","last_peered":"2024-04-26T16:43:16.182730+0000","last_clean":"2024-04-26T16:43:16.182730+0000","last_became_active":"2024-04-26T16:35:25.409902+0000","last_became_peered":"2024-04-26T16:35:25.409902+0000","last_unstale":"2024-04-26T16:43:16.182730+0000","last_undegraded":"2024-04-26T16:43:16.182730+0000","last_fullsized":"2024-04-26T16:43:16.182730+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:01.181727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:32:46.121113+0000","last_clean_scrub_stamp":"2024-04-26T16:43:01.181727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.322241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"890'35","reported_seq":1892,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889516+0000","last_change":"2024-04-26T16:43:03.882214+0000","last_active":"2024-04-26T16:43:15.889516+0000","last_peered":"2024-04-26T16:43:15.889516+0000","last_clean":"2024-04-26T16:43:15.889516+0000","last_became_active":"2024-04-26T16:35:25.383698+0000","last_became_peered":"2024-04-26T16:35:25.383698+0000","last_unstale":"2024-04-26T16:43:15.889516+0000","last_undegraded":"2024-04-26T16:43:15.889516+0000","last_fullsized":"2024-04-26T16:43:15.889516+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"890'35","last_scrub_stamp":"2024-04-26T16:43:03.881981+0000","last_deep_scrub":"890'35","last_deep_scrub_stamp":"2024-04-26T16:38:58.004208+0000","last_clean_scrub_stamp":"2024-04-26T16:43:03.881981+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.857115+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1346,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.894607+0000","last_change":"2024-04-26T16:42:58.146712+0000","last_active":"2024-04-26T16:43:15.894607+0000","last_peered":"2024-04-26T16:43:15.894607+0000","last_clean":"2024-04-26T16:43:15.894607+0000","last_became_active":"2024-04-26T16:32:09.469985+0000","last_became_peered":"2024-04-26T16:32:09.469985+0000","last_unstale":"2024-04-26T16:43:15.894607+0000","last_undegraded":"2024-04-26T16:43:15.894607+0000","last_fullsized":"2024-04-26T16:43:15.894607+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:58.146358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:30.992466+0000","last_clean_scrub_stamp":"2024-04-26T16:42:58.146358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.823685+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1836,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.891542+0000","last_change":"2024-04-26T16:42:37.131685+0000","last_active":"2024-04-26T16:43:15.891542+0000","last_peered":"2024-04-26T16:43:15.891542+0000","last_clean":"2024-04-26T16:43:15.891542+0000","last_became_active":"2024-04-26T16:32:09.451151+0000","last_became_peered":"2024-04-26T16:32:09.451151+0000","last_unstale":"2024-04-26T16:43:15.891542+0000","last_undegraded":"2024-04-26T16:43:15.891542+0000","last_fullsized":"2024-04-26T16:43:15.891542+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:37.131468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:05.286233+0000","last_clean_scrub_stamp":"2024-04-26T16:42:37.131468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:39.833142+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1804,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.179953+0000","last_change":"2024-04-26T16:41:56.000192+0000","last_active":"2024-04-26T16:43:16.179953+0000","last_peered":"2024-04-26T16:43:16.179953+0000","last_clean":"2024-04-26T16:43:16.179953+0000","last_became_active":"2024-04-26T16:32:09.458910+0000","last_became_peered":"2024-04-26T16:32:09.458910+0000","last_unstale":"2024-04-26T16:43:16.179953+0000","last_undegraded":"2024-04-26T16:43:16.179953+0000","last_fullsized":"2024-04-26T16:43:16.179953+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:55.999984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:18:44.006852+0000","last_clean_scrub_stamp":"2024-04-26T16:41:55.999984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.491430+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1680,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185027+0000","last_change":"2024-04-26T16:42:53.231887+0000","last_active":"2024-04-26T16:43:16.185027+0000","last_peered":"2024-04-26T16:43:16.185027+0000","last_clean":"2024-04-26T16:43:16.185027+0000","last_became_active":"2024-04-26T16:23:29.295045+0000","last_became_peered":"2024-04-26T16:23:29.295045+0000","last_unstale":"2024-04-26T16:43:16.185027+0000","last_undegraded":"2024-04-26T16:43:16.185027+0000","last_fullsized":"2024-04-26T16:43:16.185027+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.231667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.231667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:06.628426+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.17","version":"857'18","reported_seq":1894,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893818+0000","last_change":"2024-04-26T16:42:58.259034+0000","last_active":"2024-04-26T16:43:15.893818+0000","last_peered":"2024-04-26T16:43:15.893818+0000","last_clean":"2024-04-26T16:43:15.893818+0000","last_became_active":"2024-04-26T16:32:09.448370+0000","last_became_peered":"2024-04-26T16:32:09.448370+0000","last_unstale":"2024-04-26T16:43:15.893818+0000","last_undegraded":"2024-04-26T16:43:15.893818+0000","last_fullsized":"2024-04-26T16:43:15.893818+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'18","last_scrub_stamp":"2024-04-26T16:42:58.258801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:58.258801+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:25.915658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":743,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109446+0000","last_change":"2024-04-26T16:42:11.921015+0000","last_active":"2024-04-26T16:43:16.109446+0000","last_peered":"2024-04-26T16:43:16.109446+0000","last_clean":"2024-04-26T16:43:16.109446+0000","last_became_active":"2024-04-26T16:35:25.389184+0000","last_became_peered":"2024-04-26T16:35:25.389184+0000","last_unstale":"2024-04-26T16:43:16.109446+0000","last_undegraded":"2024-04-26T16:43:16.109446+0000","last_fullsized":"2024-04-26T16:43:16.109446+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:11.920711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:19.245611+0000","last_clean_scrub_stamp":"2024-04-26T16:42:11.920711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.586442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1437,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181394+0000","last_change":"2024-04-26T16:42:05.165110+0000","last_active":"2024-04-26T16:43:16.181394+0000","last_peered":"2024-04-26T16:43:16.181394+0000","last_clean":"2024-04-26T16:43:16.181394+0000","last_became_active":"2024-04-26T16:35:25.405230+0000","last_became_peered":"2024-04-26T16:35:25.405230+0000","last_unstale":"2024-04-26T16:43:16.181394+0000","last_undegraded":"2024-04-26T16:43:16.181394+0000","last_fullsized":"2024-04-26T16:43:16.181394+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:05.164910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:10.272810+0000","last_clean_scrub_stamp":"2024-04-26T16:42:05.164910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1783,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182445+0000","last_change":"2024-04-26T16:42:41.464140+0000","last_active":"2024-04-26T16:43:16.182445+0000","last_peered":"2024-04-26T16:43:16.182445+0000","last_clean":"2024-04-26T16:43:16.182445+0000","last_became_active":"2024-04-26T16:17:35.081616+0000","last_became_peered":"2024-04-26T16:17:35.081616+0000","last_unstale":"2024-04-26T16:43:16.182445+0000","last_undegraded":"2024-04-26T16:43:16.182445+0000","last_fullsized":"2024-04-26T16:43:16.182445+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:41.463839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.463839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:42.059956+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1657,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185909+0000","last_change":"2024-04-26T16:42:08.417897+0000","last_active":"2024-04-26T16:43:16.185909+0000","last_peered":"2024-04-26T16:43:16.185909+0000","last_clean":"2024-04-26T16:43:16.185909+0000","last_became_active":"2024-04-26T16:35:25.394016+0000","last_became_peered":"2024-04-26T16:35:25.394016+0000","last_unstale":"2024-04-26T16:43:16.185909+0000","last_undegraded":"2024-04-26T16:43:16.185909+0000","last_fullsized":"2024-04-26T16:43:16.185909+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:08.417678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:10.917141+0000","last_clean_scrub_stamp":"2024-04-26T16:42:08.417678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:17.950229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"931'25","reported_seq":773,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109463+0000","last_change":"2024-04-26T16:42:40.104037+0000","last_active":"2024-04-26T16:43:16.109463+0000","last_peered":"2024-04-26T16:43:16.109463+0000","last_clean":"2024-04-26T16:43:16.109463+0000","last_became_active":"2024-04-26T16:32:09.471462+0000","last_became_peered":"2024-04-26T16:32:09.471462+0000","last_unstale":"2024-04-26T16:43:16.109463+0000","last_undegraded":"2024-04-26T16:43:16.109463+0000","last_fullsized":"2024-04-26T16:43:16.109463+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'25","last_scrub_stamp":"2024-04-26T16:42:40.103796+0000","last_deep_scrub":"931'25","last_deep_scrub_stamp":"2024-04-26T16:42:40.103796+0000","last_clean_scrub_stamp":"2024-04-26T16:42:40.103796+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:57.427688+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1440,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184287+0000","last_change":"2024-04-26T16:42:00.120392+0000","last_active":"2024-04-26T16:43:16.184287+0000","last_peered":"2024-04-26T16:43:16.184287+0000","last_clean":"2024-04-26T16:43:16.184287+0000","last_became_active":"2024-04-26T16:35:25.406825+0000","last_became_peered":"2024-04-26T16:35:25.406825+0000","last_unstale":"2024-04-26T16:43:16.184287+0000","last_undegraded":"2024-04-26T16:43:16.184287+0000","last_fullsized":"2024-04-26T16:43:16.184287+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:00.120185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:42:00.120185+0000","last_clean_scrub_stamp":"2024-04-26T16:42:00.120185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:16.717815+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1797,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182424+0000","last_change":"2024-04-26T16:41:48.367735+0000","last_active":"2024-04-26T16:43:16.182424+0000","last_peered":"2024-04-26T16:43:16.182424+0000","last_clean":"2024-04-26T16:43:16.182424+0000","last_became_active":"2024-04-26T16:35:25.378783+0000","last_became_peered":"2024-04-26T16:35:25.378783+0000","last_unstale":"2024-04-26T16:43:16.182424+0000","last_undegraded":"2024-04-26T16:43:16.182424+0000","last_fullsized":"2024-04-26T16:43:16.182424+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:48.367505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:41:48.367505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1810,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181334+0000","last_change":"2024-04-26T16:43:07.700462+0000","last_active":"2024-04-26T16:43:16.181334+0000","last_peered":"2024-04-26T16:43:16.181334+0000","last_clean":"2024-04-26T16:43:16.181334+0000","last_became_active":"2024-04-26T16:35:25.410836+0000","last_became_peered":"2024-04-26T16:35:25.410836+0000","last_unstale":"2024-04-26T16:43:16.181334+0000","last_undegraded":"2024-04-26T16:43:16.181334+0000","last_fullsized":"2024-04-26T16:43:16.181334+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:07.700245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:43:07.700245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:29.690619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1679,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185320+0000","last_change":"2024-04-26T16:42:48.207321+0000","last_active":"2024-04-26T16:43:16.185320+0000","last_peered":"2024-04-26T16:43:16.185320+0000","last_clean":"2024-04-26T16:43:16.185320+0000","last_became_active":"2024-04-26T16:23:29.295952+0000","last_became_peered":"2024-04-26T16:23:29.295952+0000","last_unstale":"2024-04-26T16:43:16.185320+0000","last_undegraded":"2024-04-26T16:43:16.185320+0000","last_fullsized":"2024-04-26T16:43:16.185320+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:48.206977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:45.163028+0000","last_clean_scrub_stamp":"2024-04-26T16:42:48.206977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:59.620325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"857'46","reported_seq":1903,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.895807+0000","last_change":"2024-04-26T16:42:57.163825+0000","last_active":"2024-04-26T16:43:15.895807+0000","last_peered":"2024-04-26T16:43:15.895807+0000","last_clean":"2024-04-26T16:43:15.895807+0000","last_became_active":"2024-04-26T16:17:31.043660+0000","last_became_peered":"2024-04-26T16:17:31.043660+0000","last_unstale":"2024-04-26T16:43:15.895807+0000","last_undegraded":"2024-04-26T16:43:15.895807+0000","last_fullsized":"2024-04-26T16:43:15.895807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'46","last_scrub_stamp":"2024-04-26T16:42:57.163723+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-04-26T16:20:37.959753+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.163723+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:25.733581+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":750,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.106494+0000","last_change":"2024-04-26T16:43:08.092751+0000","last_active":"2024-04-26T16:43:16.106494+0000","last_peered":"2024-04-26T16:43:16.106494+0000","last_clean":"2024-04-26T16:43:16.106494+0000","last_became_active":"2024-04-26T16:32:09.464120+0000","last_became_peered":"2024-04-26T16:32:09.464120+0000","last_unstale":"2024-04-26T16:43:16.106494+0000","last_undegraded":"2024-04-26T16:43:16.106494+0000","last_fullsized":"2024-04-26T16:43:16.106494+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:08.092446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:34.980868+0000","last_clean_scrub_stamp":"2024-04-26T16:43:08.092446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:13.702428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"747'16","reported_seq":1703,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:18.178818+0000","last_change":"2024-04-26T16:43:18.178893+0000","last_active":"2024-04-26T16:43:18.178818+0000","last_peered":"2024-04-26T16:43:18.178818+0000","last_clean":"2024-04-26T16:43:18.178818+0000","last_became_active":"2024-04-26T16:35:25.403416+0000","last_became_peered":"2024-04-26T16:35:25.403416+0000","last_unstale":"2024-04-26T16:43:18.178818+0000","last_undegraded":"2024-04-26T16:43:18.178818+0000","last_fullsized":"2024-04-26T16:43:18.178818+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"747'16","last_scrub_stamp":"2024-04-26T16:43:18.178663+0000","last_deep_scrub":"329'12","last_deep_scrub_stamp":"2024-04-26T16:26:37.837294+0000","last_clean_scrub_stamp":"2024-04-26T16:43:18.178663+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:37.196520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"829'2","reported_seq":1805,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184971+0000","last_change":"2024-04-26T16:42:09.418159+0000","last_active":"2024-04-26T16:43:16.184971+0000","last_peered":"2024-04-26T16:43:16.184971+0000","last_clean":"2024-04-26T16:43:16.184971+0000","last_became_active":"2024-04-26T16:32:09.482912+0000","last_became_peered":"2024-04-26T16:32:09.482912+0000","last_unstale":"2024-04-26T16:43:16.184971+0000","last_undegraded":"2024-04-26T16:43:16.184971+0000","last_fullsized":"2024-04-26T16:43:16.184971+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"829'2","last_scrub_stamp":"2024-04-26T16:42:09.417718+0000","last_deep_scrub":"829'2","last_deep_scrub_stamp":"2024-04-26T16:36:59.155048+0000","last_clean_scrub_stamp":"2024-04-26T16:42:09.417718+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1680,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889311+0000","last_change":"2024-04-26T16:43:15.335426+0000","last_active":"2024-04-26T16:43:15.889311+0000","last_peered":"2024-04-26T16:43:15.889311+0000","last_clean":"2024-04-26T16:43:15.889311+0000","last_became_active":"2024-04-26T16:35:25.409763+0000","last_became_peered":"2024-04-26T16:35:25.409763+0000","last_unstale":"2024-04-26T16:43:15.889311+0000","last_undegraded":"2024-04-26T16:43:15.889311+0000","last_fullsized":"2024-04-26T16:43:15.889311+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:15.335295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:43:15.335295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:40.229650+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"922'25","reported_seq":1845,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.179477+0000","last_change":"2024-04-26T16:42:55.716458+0000","last_active":"2024-04-26T16:43:16.179477+0000","last_peered":"2024-04-26T16:43:16.179477+0000","last_clean":"2024-04-26T16:43:16.179477+0000","last_became_active":"2024-04-26T16:17:31.048450+0000","last_became_peered":"2024-04-26T16:17:31.048450+0000","last_unstale":"2024-04-26T16:43:16.179477+0000","last_undegraded":"2024-04-26T16:43:16.179477+0000","last_fullsized":"2024-04-26T16:43:16.179477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"922'25","last_scrub_stamp":"2024-04-26T16:42:55.716222+0000","last_deep_scrub":"922'25","last_deep_scrub_stamp":"2024-04-26T16:42:55.716222+0000","last_clean_scrub_stamp":"2024-04-26T16:42:55.716222+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:05.930413+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"109'2","reported_seq":1815,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180542+0000","last_change":"2024-04-26T16:42:28.956311+0000","last_active":"2024-04-26T16:43:16.180542+0000","last_peered":"2024-04-26T16:43:16.180542+0000","last_clean":"2024-04-26T16:43:16.180542+0000","last_became_active":"2024-04-26T16:35:25.417030+0000","last_became_peered":"2024-04-26T16:35:25.417030+0000","last_unstale":"2024-04-26T16:43:16.180542+0000","last_undegraded":"2024-04-26T16:43:16.180542+0000","last_fullsized":"2024-04-26T16:43:16.180542+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'2","last_scrub_stamp":"2024-04-26T16:42:28.955957+0000","last_deep_scrub":"109'2","last_deep_scrub_stamp":"2024-04-26T16:41:02.038050+0000","last_clean_scrub_stamp":"2024-04-26T16:42:28.955957+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.682762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1828,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893487+0000","last_change":"2024-04-26T16:42:55.162921+0000","last_active":"2024-04-26T16:43:15.893487+0000","last_peered":"2024-04-26T16:43:15.893487+0000","last_clean":"2024-04-26T16:43:15.893487+0000","last_became_active":"2024-04-26T16:17:33.072703+0000","last_became_peered":"2024-04-26T16:17:33.072703+0000","last_unstale":"2024-04-26T16:43:15.893487+0000","last_undegraded":"2024-04-26T16:43:15.893487+0000","last_fullsized":"2024-04-26T16:43:15.893487+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:55.162804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:38:22.872578+0000","last_clean_scrub_stamp":"2024-04-26T16:42:55.162804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:07.152764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1839,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.896148+0000","last_change":"2024-04-26T16:42:23.068496+0000","last_active":"2024-04-26T16:43:15.896148+0000","last_peered":"2024-04-26T16:43:15.896148+0000","last_clean":"2024-04-26T16:43:15.896148+0000","last_became_active":"2024-04-26T16:32:09.470454+0000","last_became_peered":"2024-04-26T16:32:09.470454+0000","last_unstale":"2024-04-26T16:43:15.896148+0000","last_undegraded":"2024-04-26T16:43:15.896148+0000","last_fullsized":"2024-04-26T16:43:15.896148+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:23.068390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:23.068390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:38.094123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":742,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107316+0000","last_change":"2024-04-26T16:42:22.975468+0000","last_active":"2024-04-26T16:43:16.107316+0000","last_peered":"2024-04-26T16:43:16.107316+0000","last_clean":"2024-04-26T16:43:16.107316+0000","last_became_active":"2024-04-26T16:32:09.476197+0000","last_became_peered":"2024-04-26T16:32:09.476197+0000","last_unstale":"2024-04-26T16:43:16.107316+0000","last_undegraded":"2024-04-26T16:43:16.107316+0000","last_fullsized":"2024-04-26T16:43:16.107316+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:22.975099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:22.975099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:48.791244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"857'11","reported_seq":1840,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.891133+0000","last_change":"2024-04-26T16:42:54.160426+0000","last_active":"2024-04-26T16:43:15.891133+0000","last_peered":"2024-04-26T16:43:15.891133+0000","last_clean":"2024-04-26T16:43:15.891133+0000","last_became_active":"2024-04-26T16:17:31.038596+0000","last_became_peered":"2024-04-26T16:17:31.038596+0000","last_unstale":"2024-04-26T16:43:15.891133+0000","last_undegraded":"2024-04-26T16:43:15.891133+0000","last_fullsized":"2024-04-26T16:43:15.891133+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'11","last_scrub_stamp":"2024-04-26T16:42:54.160177+0000","last_deep_scrub":"857'11","last_deep_scrub_stamp":"2024-04-26T16:42:54.160177+0000","last_clean_scrub_stamp":"2024-04-26T16:42:54.160177+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:55.628445+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1497,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180603+0000","last_change":"2024-04-26T16:42:01.171043+0000","last_active":"2024-04-26T16:43:16.180603+0000","last_peered":"2024-04-26T16:43:16.180603+0000","last_clean":"2024-04-26T16:43:16.180603+0000","last_became_active":"2024-04-26T16:35:25.404997+0000","last_became_peered":"2024-04-26T16:35:25.404997+0000","last_unstale":"2024-04-26T16:43:16.180603+0000","last_undegraded":"2024-04-26T16:43:16.180603+0000","last_fullsized":"2024-04-26T16:43:16.180603+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:01.170844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:01.170844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:29.453064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1839,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.895380+0000","last_change":"2024-04-26T16:43:05.944284+0000","last_active":"2024-04-26T16:43:15.895380+0000","last_peered":"2024-04-26T16:43:15.895380+0000","last_clean":"2024-04-26T16:43:15.895380+0000","last_became_active":"2024-04-26T16:17:33.056472+0000","last_became_peered":"2024-04-26T16:17:33.056472+0000","last_unstale":"2024-04-26T16:43:15.895380+0000","last_undegraded":"2024-04-26T16:43:15.895380+0000","last_fullsized":"2024-04-26T16:43:15.895380+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:05.944031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:43:05.944031+0000","last_clean_scrub_stamp":"2024-04-26T16:43:05.944031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:10.663257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1847,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.890946+0000","last_change":"2024-04-26T16:42:08.758536+0000","last_active":"2024-04-26T16:43:15.890946+0000","last_peered":"2024-04-26T16:43:15.890946+0000","last_clean":"2024-04-26T16:43:15.890946+0000","last_became_active":"2024-04-26T16:35:25.383286+0000","last_became_peered":"2024-04-26T16:35:25.383286+0000","last_unstale":"2024-04-26T16:43:15.890946+0000","last_undegraded":"2024-04-26T16:43:15.890946+0000","last_fullsized":"2024-04-26T16:43:15.890946+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:08.758249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:40:51.035286+0000","last_clean_scrub_stamp":"2024-04-26T16:42:08.758249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:34.548556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1646,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181150+0000","last_change":"2024-04-26T16:43:15.627816+0000","last_active":"2024-04-26T16:43:16.181150+0000","last_peered":"2024-04-26T16:43:16.181150+0000","last_clean":"2024-04-26T16:43:16.181150+0000","last_became_active":"2024-04-26T16:23:29.355430+0000","last_became_peered":"2024-04-26T16:23:29.355430+0000","last_unstale":"2024-04-26T16:43:16.181150+0000","last_undegraded":"2024-04-26T16:43:16.181150+0000","last_fullsized":"2024-04-26T16:43:16.181150+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:15.627610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:27:44.070141+0000","last_clean_scrub_stamp":"2024-04-26T16:43:15.627610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:31.782644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1a","version":"902'25","reported_seq":1869,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889758+0000","last_change":"2024-04-26T16:42:37.816736+0000","last_active":"2024-04-26T16:43:15.889758+0000","last_peered":"2024-04-26T16:43:15.889758+0000","last_clean":"2024-04-26T16:43:15.889758+0000","last_became_active":"2024-04-26T16:17:31.032272+0000","last_became_peered":"2024-04-26T16:17:31.032272+0000","last_unstale":"2024-04-26T16:43:15.889758+0000","last_undegraded":"2024-04-26T16:43:15.889758+0000","last_fullsized":"2024-04-26T16:43:15.889758+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'25","last_scrub_stamp":"2024-04-26T16:42:37.816385+0000","last_deep_scrub":"902'25","last_deep_scrub_stamp":"2024-04-26T16:39:49.818013+0000","last_clean_scrub_stamp":"2024-04-26T16:42:37.816385+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:43.671767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1813,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180570+0000","last_change":"2024-04-26T16:42:41.912362+0000","last_active":"2024-04-26T16:43:16.180570+0000","last_peered":"2024-04-26T16:43:16.180570+0000","last_clean":"2024-04-26T16:43:16.180570+0000","last_became_active":"2024-04-26T16:35:25.392415+0000","last_became_peered":"2024-04-26T16:35:25.392415+0000","last_unstale":"2024-04-26T16:43:16.180570+0000","last_undegraded":"2024-04-26T16:43:16.180570+0000","last_fullsized":"2024-04-26T16:43:16.180570+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:41.911982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:42:41.911982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:50.543906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1827,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889677+0000","last_change":"2024-04-26T16:42:31.880452+0000","last_active":"2024-04-26T16:43:15.889677+0000","last_peered":"2024-04-26T16:43:15.889677+0000","last_clean":"2024-04-26T16:43:15.889677+0000","last_became_active":"2024-04-26T16:17:33.064270+0000","last_became_peered":"2024-04-26T16:17:33.064270+0000","last_unstale":"2024-04-26T16:43:15.889677+0000","last_undegraded":"2024-04-26T16:43:15.889677+0000","last_fullsized":"2024-04-26T16:43:15.889677+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:31.880251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:36:56.985402+0000","last_clean_scrub_stamp":"2024-04-26T16:42:31.880251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:55.010235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1815,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181566+0000","last_change":"2024-04-26T16:42:11.099774+0000","last_active":"2024-04-26T16:43:16.181566+0000","last_peered":"2024-04-26T16:43:16.181566+0000","last_clean":"2024-04-26T16:43:16.181566+0000","last_became_active":"2024-04-26T16:35:25.410430+0000","last_became_peered":"2024-04-26T16:35:25.410430+0000","last_unstale":"2024-04-26T16:43:16.181566+0000","last_undegraded":"2024-04-26T16:43:16.181566+0000","last_fullsized":"2024-04-26T16:43:16.181566+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:11.099526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:42:11.099526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:32.370546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1679,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184005+0000","last_change":"2024-04-26T16:42:52.229832+0000","last_active":"2024-04-26T16:43:16.184005+0000","last_peered":"2024-04-26T16:43:16.184005+0000","last_clean":"2024-04-26T16:43:16.184005+0000","last_became_active":"2024-04-26T16:23:29.313049+0000","last_became_peered":"2024-04-26T16:23:29.313049+0000","last_unstale":"2024-04-26T16:43:16.184005+0000","last_undegraded":"2024-04-26T16:43:16.184005+0000","last_fullsized":"2024-04-26T16:43:16.184005+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:52.229707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:40.169404+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.229707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:07.097155+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"857'29","reported_seq":772,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.108928+0000","last_change":"2024-04-26T16:42:30.104346+0000","last_active":"2024-04-26T16:43:16.108928+0000","last_peered":"2024-04-26T16:43:16.108928+0000","last_clean":"2024-04-26T16:43:16.108928+0000","last_became_active":"2024-04-26T16:35:25.382813+0000","last_became_peered":"2024-04-26T16:35:25.382813+0000","last_unstale":"2024-04-26T16:43:16.108928+0000","last_undegraded":"2024-04-26T16:43:16.108928+0000","last_fullsized":"2024-04-26T16:43:16.108928+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'29","last_scrub_stamp":"2024-04-26T16:42:30.104089+0000","last_deep_scrub":"126'17","last_deep_scrub_stamp":"2024-04-26T16:21:51.362092+0000","last_clean_scrub_stamp":"2024-04-26T16:42:30.104089+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.977065+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1823,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184101+0000","last_change":"2024-04-26T16:42:13.426286+0000","last_active":"2024-04-26T16:43:16.184101+0000","last_peered":"2024-04-26T16:43:16.184101+0000","last_clean":"2024-04-26T16:43:16.184101+0000","last_became_active":"2024-04-26T16:32:09.484050+0000","last_became_peered":"2024-04-26T16:32:09.484050+0000","last_unstale":"2024-04-26T16:43:16.184101+0000","last_undegraded":"2024-04-26T16:43:16.184101+0000","last_fullsized":"2024-04-26T16:43:16.184101+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:13.425926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:27.994588+0000","last_clean_scrub_stamp":"2024-04-26T16:42:13.425926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:38.328874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":742,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.107109+0000","last_change":"2024-04-26T16:42:57.213375+0000","last_active":"2024-04-26T16:43:16.107109+0000","last_peered":"2024-04-26T16:43:16.107109+0000","last_clean":"2024-04-26T16:43:16.107109+0000","last_became_active":"2024-04-26T16:32:09.476089+0000","last_became_peered":"2024-04-26T16:32:09.476089+0000","last_unstale":"2024-04-26T16:43:16.107109+0000","last_undegraded":"2024-04-26T16:43:16.107109+0000","last_fullsized":"2024-04-26T16:43:16.107109+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:57.213120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:57.213120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:02.009393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1814,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.887980+0000","last_change":"2024-04-26T16:42:53.203249+0000","last_active":"2024-04-26T16:43:15.887980+0000","last_peered":"2024-04-26T16:43:15.887980+0000","last_clean":"2024-04-26T16:43:15.887980+0000","last_became_active":"2024-04-26T16:17:35.079259+0000","last_became_peered":"2024-04-26T16:17:35.079259+0000","last_unstale":"2024-04-26T16:43:15.887980+0000","last_undegraded":"2024-04-26T16:43:15.887980+0000","last_fullsized":"2024-04-26T16:43:15.887980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:53.202986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:35:14.855045+0000","last_clean_scrub_stamp":"2024-04-26T16:42:53.202986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:15.824975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1276,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181904+0000","last_change":"2024-04-26T16:42:18.219412+0000","last_active":"2024-04-26T16:43:16.181904+0000","last_peered":"2024-04-26T16:43:16.181904+0000","last_clean":"2024-04-26T16:43:16.181904+0000","last_became_active":"2024-04-26T16:35:25.401906+0000","last_became_peered":"2024-04-26T16:35:25.401906+0000","last_unstale":"2024-04-26T16:43:16.181904+0000","last_undegraded":"2024-04-26T16:43:16.181904+0000","last_fullsized":"2024-04-26T16:43:16.181904+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:18.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:23:28.083853+0000","last_clean_scrub_stamp":"2024-04-26T16:42:18.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:41.345015+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1c","version":"923'46","reported_seq":794,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.108447+0000","last_change":"2024-04-26T16:42:45.143623+0000","last_active":"2024-04-26T16:43:16.108447+0000","last_peered":"2024-04-26T16:43:16.108447+0000","last_clean":"2024-04-26T16:43:16.108447+0000","last_became_active":"2024-04-26T16:32:09.466151+0000","last_became_peered":"2024-04-26T16:32:09.466151+0000","last_unstale":"2024-04-26T16:43:16.108447+0000","last_undegraded":"2024-04-26T16:43:16.108447+0000","last_fullsized":"2024-04-26T16:43:16.108447+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"923'46","last_scrub_stamp":"2024-04-26T16:42:45.143361+0000","last_deep_scrub":"923'46","last_deep_scrub_stamp":"2024-04-26T16:41:38.088859+0000","last_clean_scrub_stamp":"2024-04-26T16:42:45.143361+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:09.644961+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1812,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.184821+0000","last_change":"2024-04-26T16:41:54.483826+0000","last_active":"2024-04-26T16:43:16.184821+0000","last_peered":"2024-04-26T16:43:16.184821+0000","last_clean":"2024-04-26T16:43:16.184821+0000","last_became_active":"2024-04-26T16:35:25.389963+0000","last_became_peered":"2024-04-26T16:35:25.389963+0000","last_unstale":"2024-04-26T16:43:16.184821+0000","last_undegraded":"2024-04-26T16:43:16.184821+0000","last_fullsized":"2024-04-26T16:43:16.184821+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:54.483615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:27.960068+0000","last_clean_scrub_stamp":"2024-04-26T16:41:54.483615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1786,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181777+0000","last_change":"2024-04-26T16:42:02.301779+0000","last_active":"2024-04-26T16:43:16.181777+0000","last_peered":"2024-04-26T16:43:16.181777+0000","last_clean":"2024-04-26T16:43:16.181777+0000","last_became_active":"2024-04-26T16:17:33.054034+0000","last_became_peered":"2024-04-26T16:17:33.054034+0000","last_unstale":"2024-04-26T16:43:16.181777+0000","last_undegraded":"2024-04-26T16:43:16.181777+0000","last_fullsized":"2024-04-26T16:43:16.181777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:02.301500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:32.003751+0000","last_clean_scrub_stamp":"2024-04-26T16:42:02.301500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:23.509272+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"86'1","reported_seq":754,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.106552+0000","last_change":"2024-04-26T16:43:00.106500+0000","last_active":"2024-04-26T16:43:16.106552+0000","last_peered":"2024-04-26T16:43:16.106552+0000","last_clean":"2024-04-26T16:43:16.106552+0000","last_became_active":"2024-04-26T16:32:09.468674+0000","last_became_peered":"2024-04-26T16:32:09.468674+0000","last_unstale":"2024-04-26T16:43:16.106552+0000","last_undegraded":"2024-04-26T16:43:16.106552+0000","last_fullsized":"2024-04-26T16:43:16.106552+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'1","last_scrub_stamp":"2024-04-26T16:43:00.106277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:34.027340+0000","last_clean_scrub_stamp":"2024-04-26T16:43:00.106277+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:12.686057+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1668,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.185505+0000","last_change":"2024-04-26T16:41:58.420395+0000","last_active":"2024-04-26T16:43:16.185505+0000","last_peered":"2024-04-26T16:43:16.185505+0000","last_clean":"2024-04-26T16:43:16.185505+0000","last_became_active":"2024-04-26T16:35:25.388177+0000","last_became_peered":"2024-04-26T16:35:25.388177+0000","last_unstale":"2024-04-26T16:43:16.185505+0000","last_undegraded":"2024-04-26T16:43:16.185505+0000","last_fullsized":"2024-04-26T16:43:16.185505+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:41:58.420180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:26.748278+0000","last_clean_scrub_stamp":"2024-04-26T16:41:58.420180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"895'32","reported_seq":1917,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.893890+0000","last_change":"2024-04-26T16:42:48.152221+0000","last_active":"2024-04-26T16:43:15.893890+0000","last_peered":"2024-04-26T16:43:15.893890+0000","last_clean":"2024-04-26T16:43:15.893890+0000","last_became_active":"2024-04-26T16:32:09.451242+0000","last_became_peered":"2024-04-26T16:32:09.451242+0000","last_unstale":"2024-04-26T16:43:15.893890+0000","last_undegraded":"2024-04-26T16:43:15.893890+0000","last_fullsized":"2024-04-26T16:43:15.893890+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"895'32","last_scrub_stamp":"2024-04-26T16:42:48.151977+0000","last_deep_scrub":"895'32","last_deep_scrub_stamp":"2024-04-26T16:39:30.003486+0000","last_clean_scrub_stamp":"2024-04-26T16:42:48.151977+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:49.631690+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1833,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.182026+0000","last_change":"2024-04-26T16:43:11.686670+0000","last_active":"2024-04-26T16:43:16.182026+0000","last_peered":"2024-04-26T16:43:16.182026+0000","last_clean":"2024-04-26T16:43:16.182026+0000","last_became_active":"2024-04-26T16:32:09.461605+0000","last_became_peered":"2024-04-26T16:32:09.461605+0000","last_unstale":"2024-04-26T16:43:16.182026+0000","last_undegraded":"2024-04-26T16:43:16.182026+0000","last_fullsized":"2024-04-26T16:43:16.182026+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:11.686285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:39.038243+0000","last_clean_scrub_stamp":"2024-04-26T16:43:11.686285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:34.761405+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1843,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889712+0000","last_change":"2024-04-26T16:42:48.094343+0000","last_active":"2024-04-26T16:43:15.889712+0000","last_peered":"2024-04-26T16:43:15.889712+0000","last_clean":"2024-04-26T16:43:15.889712+0000","last_became_active":"2024-04-26T16:32:09.451174+0000","last_became_peered":"2024-04-26T16:32:09.451174+0000","last_unstale":"2024-04-26T16:43:15.889712+0000","last_undegraded":"2024-04-26T16:43:15.889712+0000","last_fullsized":"2024-04-26T16:43:15.889712+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:48.094125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:41:14.727355+0000","last_clean_scrub_stamp":"2024-04-26T16:42:48.094125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:13.646575+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1835,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.889115+0000","last_change":"2024-04-26T16:42:21.884006+0000","last_active":"2024-04-26T16:43:15.889115+0000","last_peered":"2024-04-26T16:43:15.889115+0000","last_clean":"2024-04-26T16:43:15.889115+0000","last_became_active":"2024-04-26T16:32:09.465678+0000","last_became_peered":"2024-04-26T16:32:09.465678+0000","last_unstale":"2024-04-26T16:43:15.889115+0000","last_undegraded":"2024-04-26T16:43:15.889115+0000","last_fullsized":"2024-04-26T16:43:15.889115+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:21.883800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:27.837864+0000","last_clean_scrub_stamp":"2024-04-26T16:42:21.883800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:25.296934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":742,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.109079+0000","last_change":"2024-04-26T16:42:04.022526+0000","last_active":"2024-04-26T16:43:16.109079+0000","last_peered":"2024-04-26T16:43:16.109079+0000","last_clean":"2024-04-26T16:43:16.109079+0000","last_became_active":"2024-04-26T16:32:09.458796+0000","last_became_peered":"2024-04-26T16:32:09.458796+0000","last_unstale":"2024-04-26T16:43:16.109079+0000","last_undegraded":"2024-04-26T16:43:16.109079+0000","last_fullsized":"2024-04-26T16:43:16.109079+0000","mapping_epoch":584,"log_start":"0'0","ondisk_log_start":"0'0","created":140,"last_epoch_clean":585,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:04.022275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:39:34.130566+0000","last_clean_scrub_stamp":"2024-04-26T16:42:04.022275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:30.463543+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"857'30","reported_seq":1864,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.181904+0000","last_change":"2024-04-26T16:42:52.767797+0000","last_active":"2024-04-26T16:43:16.181904+0000","last_peered":"2024-04-26T16:43:16.181904+0000","last_clean":"2024-04-26T16:43:16.181904+0000","last_became_active":"2024-04-26T16:35:25.399680+0000","last_became_peered":"2024-04-26T16:35:25.399680+0000","last_unstale":"2024-04-26T16:43:16.181904+0000","last_undegraded":"2024-04-26T16:43:16.181904+0000","last_fullsized":"2024-04-26T16:43:16.181904+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"857'30","last_scrub_stamp":"2024-04-26T16:42:52.767679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:17:29.981090+0000","last_clean_scrub_stamp":"2024-04-26T16:42:52.767679+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:14.193285+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1828,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.894151+0000","last_change":"2024-04-26T16:42:20.091221+0000","last_active":"2024-04-26T16:43:15.894151+0000","last_peered":"2024-04-26T16:43:15.894151+0000","last_clean":"2024-04-26T16:43:15.894151+0000","last_became_active":"2024-04-26T16:17:29.010722+0000","last_became_peered":"2024-04-26T16:17:29.010722+0000","last_unstale":"2024-04-26T16:43:15.894151+0000","last_undegraded":"2024-04-26T16:43:15.894151+0000","last_fullsized":"2024-04-26T16:43:15.894151+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:20.090970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:26:37.410252+0000","last_clean_scrub_stamp":"2024-04-26T16:42:20.090970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:35.757867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1837,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:15.888364+0000","last_change":"2024-04-26T16:43:13.981261+0000","last_active":"2024-04-26T16:43:15.888364+0000","last_peered":"2024-04-26T16:43:15.888364+0000","last_clean":"2024-04-26T16:43:15.888364+0000","last_became_active":"2024-04-26T16:35:25.396616+0000","last_became_peered":"2024-04-26T16:35:25.396616+0000","last_unstale":"2024-04-26T16:43:15.888364+0000","last_undegraded":"2024-04-26T16:43:15.888364+0000","last_fullsized":"2024-04-26T16:43:15.888364+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:43:13.981036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:43:13.981036+0000","last_clean_scrub_stamp":"2024-04-26T16:43:13.981036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:44:14.254933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1801,"reported_epoch":932,"state":"active+clean","last_fresh":"2024-04-26T16:43:16.180186+0000","last_change":"2024-04-26T16:42:46.880236+0000","last_active":"2024-04-26T16:43:16.180186+0000","last_peered":"2024-04-26T16:43:16.180186+0000","last_clean":"2024-04-26T16:43:16.180186+0000","last_became_active":"2024-04-26T16:35:25.392061+0000","last_became_peered":"2024-04-26T16:35:25.392061+0000","last_unstale":"2024-04-26T16:43:16.180186+0000","last_undegraded":"2024-04-26T16:43:16.180186+0000","last_fullsized":"2024-04-26T16:43:16.180186+0000","mapping_epoch":746,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":747,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T16:42:46.880057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T16:21:58.228927+0000","last_clean_scrub_stamp":"2024-04-26T16:42:46.880057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:43:56.062162+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":15,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":34,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1548,"num_read_kb":1305,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":2564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},"log_size":1128,"ondisk_log_size":1128,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1524,"num_read_kb":1519,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":69,"seq":296352743745,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114380,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93630772,"statfs":{"total":95995035648,"available":95877910528,"internally_reserved":0,"allocated":2756608,"data_stored":995847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":746,"seq":3204045602913,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":39936,"kb_used_data":2356,"kb_used_omap":23,"kb_used_meta":37544,"kb_avail":93705216,"statfs":{"total":95995035648,"available":95954141184,"internally_reserved":0,"allocated":2412544,"data_stored":674358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23760,"internal_metadata":38445872},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575231,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101588,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93643564,"statfs":{"total":95995035648,"available":95891009536,"internally_reserved":0,"allocated":2433024,"data_stored":675690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738761,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109916,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635236,"statfs":{"total":95995035648,"available":95882481664,"internally_reserved":0,"allocated":2441216,"data_stored":676552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411700,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105972,"kb_used_data":2604,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93639180,"statfs":{"total":95995035648,"available":95886520320,"internally_reserved":0,"allocated":2666496,"data_stored":928687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084649,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132616,"kb_used_data":2940,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93612536,"statfs":{"total":95995035648,"available":95859236864,"internally_reserved":0,"allocated":3010560,"data_stored":1265126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":132774344},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":35,"seq":150323855711,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109920,"kb_used_data":2388,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635232,"statfs":{"total":95995035648,"available":95882477568,"internally_reserved":0,"allocated":2445312,"data_stored":675514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":47,"seq":201863463253,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":96564,"kb_used_data":2924,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93648588,"statfs":{"total":95995035648,"available":95896154112,"internally_reserved":0,"allocated":2994176,"data_stored":1256179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3656,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T16:43:22.020 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-04-26T16:43:22.020 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:a430f9ed84de3008ca5b632b1ed304bac0e753f6 shell --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-04-26T16:43:22.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.205062+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.205062+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.207455+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.207455+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.602249+0000 osd.3 (osd.3) 691 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.602249+0000 osd.3 (osd.3) 691 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.609553+0000 osd.3 (osd.3) 692 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.609553+0000 osd.3 (osd.3) 692 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.942865+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.942865+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2024-04-26T16:43:22.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.976424+0000 osd.1 (osd.1) 621 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.976424+0000 osd.1 (osd.1) 621 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.979239+0000 osd.1 (osd.1) 622 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.979239+0000 osd.1 (osd.1) 622 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.994750+0000 osd.4 (osd.4) 577 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.994750+0000 osd.4 (osd.4) 577 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.997133+0000 osd.4 (osd.4) 578 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: cluster 2024-04-26T16:43:20.997133+0000 osd.4 (osd.4) 578 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: audit 2024-04-26T16:43:21.067393+0000 mgr.y (mgr.24401) 984 : audit [DBG] from='client.31383 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:22.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:21 smithi190 bash[23868]: audit 2024-04-26T16:43:21.067393+0000 mgr.y (mgr.24401) 984 : audit [DBG] from='client.31383 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.205062+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.205062+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.207455+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.207455+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.602249+0000 osd.3 (osd.3) 691 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.602249+0000 osd.3 (osd.3) 691 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.609553+0000 osd.3 (osd.3) 692 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:43:22.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.609553+0000 osd.3 (osd.3) 692 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.942865+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.942865+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.976424+0000 osd.1 (osd.1) 621 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.976424+0000 osd.1 (osd.1) 621 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.979239+0000 osd.1 (osd.1) 622 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.979239+0000 osd.1 (osd.1) 622 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.994750+0000 osd.4 (osd.4) 577 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.994750+0000 osd.4 (osd.4) 577 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.997133+0000 osd.4 (osd.4) 578 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: cluster 2024-04-26T16:43:20.997133+0000 osd.4 (osd.4) 578 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: audit 2024-04-26T16:43:21.067393+0000 mgr.y (mgr.24401) 984 : audit [DBG] from='client.31383 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:22.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[29234]: audit 2024-04-26T16:43:21.067393+0000 mgr.y (mgr.24401) 984 : audit [DBG] from='client.31383 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.205062+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.205062+0000 osd.6 (osd.6) 173 : cluster [DBG] 5.11 scrub starts 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.207455+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.207455+0000 osd.6 (osd.6) 174 : cluster [DBG] 5.11 scrub ok 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.602249+0000 osd.3 (osd.3) 691 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.602249+0000 osd.3 (osd.3) 691 : cluster [DBG] 11.b deep-scrub starts 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.609553+0000 osd.3 (osd.3) 692 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.609553+0000 osd.3 (osd.3) 692 : cluster [DBG] 11.b deep-scrub ok 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.942865+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.942865+0000 mon.a (mon.0) 4078 : cluster [DBG] osdmap e933: 8 total, 8 up, 8 in 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.976424+0000 osd.1 (osd.1) 621 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.976424+0000 osd.1 (osd.1) 621 : cluster [DBG] 2.6 scrub starts 2024-04-26T16:43:22.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.979239+0000 osd.1 (osd.1) 622 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.979239+0000 osd.1 (osd.1) 622 : cluster [DBG] 2.6 scrub ok 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.994750+0000 osd.4 (osd.4) 577 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.994750+0000 osd.4 (osd.4) 577 : cluster [DBG] 6.17 scrub starts 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.997133+0000 osd.4 (osd.4) 578 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: cluster 2024-04-26T16:43:20.997133+0000 osd.4 (osd.4) 578 : cluster [DBG] 6.17 scrub ok 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: audit 2024-04-26T16:43:21.067393+0000 mgr.y (mgr.24401) 984 : audit [DBG] from='client.31383 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:22.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:21 smithi050 bash[21463]: audit 2024-04-26T16:43:21.067393+0000 mgr.y (mgr.24401) 984 : audit [DBG] from='client.31383 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.252179+0000 osd.0 (osd.0) 643 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.252179+0000 osd.0 (osd.0) 643 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.256314+0000 osd.0 (osd.0) 644 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.256314+0000 osd.0 (osd.0) 644 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.277664+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.277664+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.278039+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.278039+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.629595+0000 osd.3 (osd.3) 693 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.629595+0000 osd.3 (osd.3) 693 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.632352+0000 osd.3 (osd.3) 694 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:43:23.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.632352+0000 osd.3 (osd.3) 694 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.950545+0000 mon.a (mon.0) 4079 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:21.950545+0000 mon.a (mon.0) 4079 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.023413+0000 osd.4 (osd.4) 579 : cluster [DBG] 11.e deep-scrub starts 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.023413+0000 osd.4 (osd.4) 579 : cluster [DBG] 11.e deep-scrub starts 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.028815+0000 osd.4 (osd.4) 580 : cluster [DBG] 11.e deep-scrub ok 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.028815+0000 osd.4 (osd.4) 580 : cluster [DBG] 11.e deep-scrub ok 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.158374+0000 mgr.y (mgr.24401) 985 : cluster [DBG] pgmap v1585: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:43:23.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:22 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.158374+0000 mgr.y (mgr.24401) 985 : cluster [DBG] pgmap v1585: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:43:23.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.252179+0000 osd.0 (osd.0) 643 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:43:23.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.252179+0000 osd.0 (osd.0) 643 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:43:23.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.256314+0000 osd.0 (osd.0) 644 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:43:23.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.256314+0000 osd.0 (osd.0) 644 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:43:23.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.277664+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:23.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.277664+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.278039+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.278039+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.629595+0000 osd.3 (osd.3) 693 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.629595+0000 osd.3 (osd.3) 693 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.632352+0000 osd.3 (osd.3) 694 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.632352+0000 osd.3 (osd.3) 694 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.950545+0000 mon.a (mon.0) 4079 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:21.950545+0000 mon.a (mon.0) 4079 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.023413+0000 osd.4 (osd.4) 579 : cluster [DBG] 11.e deep-scrub starts 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.023413+0000 osd.4 (osd.4) 579 : cluster [DBG] 11.e deep-scrub starts 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.028815+0000 osd.4 (osd.4) 580 : cluster [DBG] 11.e deep-scrub ok 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.028815+0000 osd.4 (osd.4) 580 : cluster [DBG] 11.e deep-scrub ok 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.158374+0000 mgr.y (mgr.24401) 985 : cluster [DBG] pgmap v1585: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:43:23.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.158374+0000 mgr.y (mgr.24401) 985 : cluster [DBG] pgmap v1585: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.252179+0000 osd.0 (osd.0) 643 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.252179+0000 osd.0 (osd.0) 643 : cluster [DBG] 3.6 scrub starts 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.256314+0000 osd.0 (osd.0) 644 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.256314+0000 osd.0 (osd.0) 644 : cluster [DBG] 3.6 scrub ok 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.277664+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.277664+0000 osd.6 (osd.6) 175 : cluster [DBG] purged_snaps scrub starts 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.278039+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.278039+0000 osd.6 (osd.6) 176 : cluster [DBG] purged_snaps scrub ok 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.629595+0000 osd.3 (osd.3) 693 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.629595+0000 osd.3 (osd.3) 693 : cluster [DBG] 1.4 scrub starts 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.632352+0000 osd.3 (osd.3) 694 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:43:23.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.632352+0000 osd.3 (osd.3) 694 : cluster [DBG] 1.4 scrub ok 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.950545+0000 mon.a (mon.0) 4079 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:21.950545+0000 mon.a (mon.0) 4079 : cluster [DBG] osdmap e934: 8 total, 8 up, 8 in 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.023413+0000 osd.4 (osd.4) 579 : cluster [DBG] 11.e deep-scrub starts 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.023413+0000 osd.4 (osd.4) 579 : cluster [DBG] 11.e deep-scrub starts 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.028815+0000 osd.4 (osd.4) 580 : cluster [DBG] 11.e deep-scrub ok 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.028815+0000 osd.4 (osd.4) 580 : cluster [DBG] 11.e deep-scrub ok 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.158374+0000 mgr.y (mgr.24401) 985 : cluster [DBG] pgmap v1585: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:43:23.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:22 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.158374+0000 mgr.y (mgr.24401) 985 : cluster [DBG] pgmap v1585: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-26T16:43:24.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.180144+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.180144+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.182849+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:22.182849+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.061196+0000 osd.5 (osd.5) 544 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.061196+0000 osd.5 (osd.5) 544 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.064488+0000 osd.5 (osd.5) 545 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:43:24.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:23 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.064488+0000 osd.5 (osd.5) 545 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.180144+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.180144+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.182849+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:22.182849+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.061196+0000 osd.5 (osd.5) 544 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.061196+0000 osd.5 (osd.5) 544 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.064488+0000 osd.5 (osd.5) 545 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.064488+0000 osd.5 (osd.5) 545 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.180144+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.180144+0000 osd.6 (osd.6) 177 : cluster [DBG] 1.6 scrub starts 2024-04-26T16:43:24.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.182849+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:43:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:22.182849+0000 osd.6 (osd.6) 178 : cluster [DBG] 1.6 scrub ok 2024-04-26T16:43:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.061196+0000 osd.5 (osd.5) 544 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:43:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.061196+0000 osd.5 (osd.5) 544 : cluster [DBG] 5.15 scrub starts 2024-04-26T16:43:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.064488+0000 osd.5 (osd.5) 545 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:43:24.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:23 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.064488+0000 osd.5 (osd.5) 545 : cluster [DBG] 5.15 scrub ok 2024-04-26T16:43:24.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:43:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:43:25.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.219390+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.219390+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.221788+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.221788+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.236721+0000 osd.0 (osd.0) 645 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.236721+0000 osd.0 (osd.0) 645 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.241053+0000 osd.0 (osd.0) 646 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:43:25.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:23.241053+0000 osd.0 (osd.0) 646 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:43:25.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:24.159411+0000 mgr.y (mgr.24401) 986 : cluster [DBG] pgmap v1586: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:25.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:24 smithi190 bash[23868]: cluster 2024-04-26T16:43:24.159411+0000 mgr.y (mgr.24401) 986 : cluster [DBG] pgmap v1586: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.219390+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.219390+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.221788+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.221788+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.236721+0000 osd.0 (osd.0) 645 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.236721+0000 osd.0 (osd.0) 645 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.241053+0000 osd.0 (osd.0) 646 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:23.241053+0000 osd.0 (osd.0) 646 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:24.159411+0000 mgr.y (mgr.24401) 986 : cluster [DBG] pgmap v1586: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:25.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[21463]: cluster 2024-04-26T16:43:24.159411+0000 mgr.y (mgr.24401) 986 : cluster [DBG] pgmap v1586: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.219390+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.219390+0000 osd.6 (osd.6) 179 : cluster [DBG] 5.2 scrub starts 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.221788+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.221788+0000 osd.6 (osd.6) 180 : cluster [DBG] 5.2 scrub ok 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.236721+0000 osd.0 (osd.0) 645 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.236721+0000 osd.0 (osd.0) 645 : cluster [DBG] 3.12 scrub starts 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.241053+0000 osd.0 (osd.0) 646 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:23.241053+0000 osd.0 (osd.0) 646 : cluster [DBG] 3.12 scrub ok 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:24.159411+0000 mgr.y (mgr.24401) 986 : cluster [DBG] pgmap v1586: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:25.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:24 smithi050 bash[29234]: cluster 2024-04-26T16:43:24.159411+0000 mgr.y (mgr.24401) 986 : cluster [DBG] pgmap v1586: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:25 smithi190 bash[23868]: cluster 2024-04-26T16:43:24.225358+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:43:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:25 smithi190 bash[23868]: cluster 2024-04-26T16:43:24.225358+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:43:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:25 smithi190 bash[23868]: cluster 2024-04-26T16:43:24.227815+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:43:26.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:25 smithi190 bash[23868]: cluster 2024-04-26T16:43:24.227815+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:43:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[21463]: cluster 2024-04-26T16:43:24.225358+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:43:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[21463]: cluster 2024-04-26T16:43:24.225358+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:43:26.379 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[21463]: cluster 2024-04-26T16:43:24.227815+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:43:26.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[21463]: cluster 2024-04-26T16:43:24.227815+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:43:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[29234]: cluster 2024-04-26T16:43:24.225358+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:43:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[29234]: cluster 2024-04-26T16:43:24.225358+0000 osd.6 (osd.6) 181 : cluster [DBG] 2.17 scrub starts 2024-04-26T16:43:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[29234]: cluster 2024-04-26T16:43:24.227815+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:43:26.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:25 smithi050 bash[29234]: cluster 2024-04-26T16:43:24.227815+0000 osd.6 (osd.6) 182 : cluster [DBG] 2.17 scrub ok 2024-04-26T16:43:26.812 INFO:teuthology.orchestra.run.smithi050.stderr:Inferring config /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/mon.a/config 2024-04-26T16:43:27.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.004277+0000 osd.4 (osd.4) 581 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:43:27.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.004277+0000 osd.4 (osd.4) 581 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:43:27.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.010591+0000 osd.4 (osd.4) 582 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:43:27.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.010591+0000 osd.4 (osd.4) 582 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:43:27.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.160180+0000 mgr.y (mgr.24401) 987 : cluster [DBG] pgmap v1587: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:43:27.130 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.160180+0000 mgr.y (mgr.24401) 987 : cluster [DBG] pgmap v1587: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.276841+0000 osd.2 (osd.2) 563 : cluster [DBG] 2.e scrub starts 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.276841+0000 osd.2 (osd.2) 563 : cluster [DBG] 2.e scrub starts 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.278989+0000 osd.2 (osd.2) 564 : cluster [DBG] 2.e scrub ok 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[21463]: cluster 2024-04-26T16:43:26.278989+0000 osd.2 (osd.2) 564 : cluster [DBG] 2.e scrub ok 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.004277+0000 osd.4 (osd.4) 581 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.004277+0000 osd.4 (osd.4) 581 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.010591+0000 osd.4 (osd.4) 582 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.010591+0000 osd.4 (osd.4) 582 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.160180+0000 mgr.y (mgr.24401) 987 : cluster [DBG] pgmap v1587: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.160180+0000 mgr.y (mgr.24401) 987 : cluster [DBG] pgmap v1587: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.276841+0000 osd.2 (osd.2) 563 : cluster [DBG] 2.e scrub starts 2024-04-26T16:43:27.131 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.276841+0000 osd.2 (osd.2) 563 : cluster [DBG] 2.e scrub starts 2024-04-26T16:43:27.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.278989+0000 osd.2 (osd.2) 564 : cluster [DBG] 2.e scrub ok 2024-04-26T16:43:27.132 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:26 smithi050 bash[29234]: cluster 2024-04-26T16:43:26.278989+0000 osd.2 (osd.2) 564 : cluster [DBG] 2.e scrub ok 2024-04-26T16:43:27.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.004277+0000 osd.4 (osd.4) 581 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.004277+0000 osd.4 (osd.4) 581 : cluster [DBG] 11.1b scrub starts 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.010591+0000 osd.4 (osd.4) 582 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.010591+0000 osd.4 (osd.4) 582 : cluster [DBG] 11.1b scrub ok 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.160180+0000 mgr.y (mgr.24401) 987 : cluster [DBG] pgmap v1587: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.160180+0000 mgr.y (mgr.24401) 987 : cluster [DBG] pgmap v1587: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.276841+0000 osd.2 (osd.2) 563 : cluster [DBG] 2.e scrub starts 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.276841+0000 osd.2 (osd.2) 563 : cluster [DBG] 2.e scrub starts 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.278989+0000 osd.2 (osd.2) 564 : cluster [DBG] 2.e scrub ok 2024-04-26T16:43:27.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:26 smithi190 bash[23868]: cluster 2024-04-26T16:43:26.278989+0000 osd.2 (osd.2) 564 : cluster [DBG] 2.e scrub ok 2024-04-26T16:43:27.972 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:43:27 smithi190 bash[47603]: debug there is no tcmu-runner data available 2024-04-26T16:43:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:27 smithi050 bash[21463]: audit 2024-04-26T16:43:27.669502+0000 mgr.y (mgr.24401) 988 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:28.228 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:27 smithi050 bash[21463]: audit 2024-04-26T16:43:27.669502+0000 mgr.y (mgr.24401) 988 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:28.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:27 smithi050 bash[29234]: audit 2024-04-26T16:43:27.669502+0000 mgr.y (mgr.24401) 988 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:28.228 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:27 smithi050 bash[29234]: audit 2024-04-26T16:43:27.669502+0000 mgr.y (mgr.24401) 988 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:28.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:27 smithi190 bash[23868]: audit 2024-04-26T16:43:27.669502+0000 mgr.y (mgr.24401) 988 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:28.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:27 smithi190 bash[23868]: audit 2024-04-26T16:43:27.669502+0000 mgr.y (mgr.24401) 988 : audit [DBG] from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-26T16:43:29.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:28 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.084030+0000 osd.5 (osd.5) 546 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.084030+0000 osd.5 (osd.5) 546 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.090098+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.090098+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.161980+0000 mgr.y (mgr.24401) 989 : cluster [DBG] pgmap v1588: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.161980+0000 mgr.y (mgr.24401) 989 : cluster [DBG] pgmap v1588: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.274828+0000 osd.2 (osd.2) 565 : cluster [DBG] 5.d scrub starts 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.274828+0000 osd.2 (osd.2) 565 : cluster [DBG] 5.d scrub starts 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.277001+0000 osd.2 (osd.2) 566 : cluster [DBG] 5.d scrub ok 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.277001+0000 osd.2 (osd.2) 566 : cluster [DBG] 5.d scrub ok 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: audit 2024-04-26T16:43:28.711728+0000 mon.c (mon.2) 962 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:29.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: audit 2024-04-26T16:43:28.711728+0000 mon.c (mon.2) 962 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.084030+0000 osd.5 (osd.5) 546 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.084030+0000 osd.5 (osd.5) 546 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.090098+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.090098+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.161980+0000 mgr.y (mgr.24401) 989 : cluster [DBG] pgmap v1588: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.161980+0000 mgr.y (mgr.24401) 989 : cluster [DBG] pgmap v1588: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:29.380 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.274828+0000 osd.2 (osd.2) 565 : cluster [DBG] 5.d scrub starts 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.274828+0000 osd.2 (osd.2) 565 : cluster [DBG] 5.d scrub starts 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.277001+0000 osd.2 (osd.2) 566 : cluster [DBG] 5.d scrub ok 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.277001+0000 osd.2 (osd.2) 566 : cluster [DBG] 5.d scrub ok 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: audit 2024-04-26T16:43:28.711728+0000 mon.c (mon.2) 962 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[21463]: audit 2024-04-26T16:43:28.711728+0000 mon.c (mon.2) 962 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.084030+0000 osd.5 (osd.5) 546 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.084030+0000 osd.5 (osd.5) 546 : cluster [DBG] 3.5 deep-scrub starts 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.090098+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.090098+0000 osd.5 (osd.5) 547 : cluster [DBG] 3.5 deep-scrub ok 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.161980+0000 mgr.y (mgr.24401) 989 : cluster [DBG] pgmap v1588: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.161980+0000 mgr.y (mgr.24401) 989 : cluster [DBG] pgmap v1588: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-26T16:43:29.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.274828+0000 osd.2 (osd.2) 565 : cluster [DBG] 5.d scrub starts 2024-04-26T16:43:29.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.274828+0000 osd.2 (osd.2) 565 : cluster [DBG] 5.d scrub starts 2024-04-26T16:43:29.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.277001+0000 osd.2 (osd.2) 566 : cluster [DBG] 5.d scrub ok 2024-04-26T16:43:29.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.277001+0000 osd.2 (osd.2) 566 : cluster [DBG] 5.d scrub ok 2024-04-26T16:43:29.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: audit 2024-04-26T16:43:28.711728+0000 mon.c (mon.2) 962 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:29.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:28 smithi050 bash[29234]: audit 2024-04-26T16:43:28.711728+0000 mon.c (mon.2) 962 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-26T16:43:30.251 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.239038+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:43:30.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.239038+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:43:30.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.241523+0000 osd.6 (osd.6) 184 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:43:30.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.241523+0000 osd.6 (osd.6) 184 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:43:30.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.693506+0000 osd.7 (osd.7) 676 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:43:30.252 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.693506+0000 osd.7 (osd.7) 676 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:43:30.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.697586+0000 osd.7 (osd.7) 677 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:43:30.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:28.697586+0000 osd.7 (osd.7) 677 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:43:30.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.025671+0000 osd.4 (osd.4) 583 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:43:30.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.025671+0000 osd.4 (osd.4) 583 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:43:30.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.027806+0000 osd.4 (osd.4) 584 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:43:30.253 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.027806+0000 osd.4 (osd.4) 584 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:43:30.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.239038+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:43:30.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.239038+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:43:30.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.241523+0000 osd.6 (osd.6) 184 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:43:30.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.241523+0000 osd.6 (osd.6) 184 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:43:30.254 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.693506+0000 osd.7 (osd.7) 676 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:43:30.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.693506+0000 osd.7 (osd.7) 676 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:43:30.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.697586+0000 osd.7 (osd.7) 677 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:43:30.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:28.697586+0000 osd.7 (osd.7) 677 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:43:30.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.025671+0000 osd.4 (osd.4) 583 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:43:30.255 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.025671+0000 osd.4 (osd.4) 583 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:43:30.256 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.027806+0000 osd.4 (osd.4) 584 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:43:30.256 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:29 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.027806+0000 osd.4 (osd.4) 584 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:43:30.324 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.239038+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.239038+0000 osd.6 (osd.6) 183 : cluster [DBG] 1.7 scrub starts 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.241523+0000 osd.6 (osd.6) 184 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.241523+0000 osd.6 (osd.6) 184 : cluster [DBG] 1.7 scrub ok 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.693506+0000 osd.7 (osd.7) 676 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.693506+0000 osd.7 (osd.7) 676 : cluster [DBG] 2.0 scrub starts 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.697586+0000 osd.7 (osd.7) 677 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:28.697586+0000 osd.7 (osd.7) 677 : cluster [DBG] 2.0 scrub ok 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.025671+0000 osd.4 (osd.4) 583 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.025671+0000 osd.4 (osd.4) 583 : cluster [DBG] 6.1a deep-scrub starts 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.027806+0000 osd.4 (osd.4) 584 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:43:30.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:29 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.027806+0000 osd.4 (osd.4) 584 : cluster [DBG] 6.1a deep-scrub ok 2024-04-26T16:43:30.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- 172.21.15.50:0/3896489988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806b130 msgr2=0x7f17e806b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:30.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3896489988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806b130 0x7f17e806b590 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7f17d000b0a0 tx=0x7f17d000b3b0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- 172.21.15.50:0/3896489988 shutdown_connections 2024-04-26T16:43:30.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3896489988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e806bad0 0x7f17e81102e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.408 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3896489988 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e81094a0 0x7f17e8109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3896489988 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806b130 0x7f17e806b590 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- 172.21.15.50:0/3896489988 >> 172.21.15.50:0/3896489988 conn(0x7f17e80792d0 msgr2=0x7f17e80796e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- 172.21.15.50:0/3896489988 shutdown_connections 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- 172.21.15.50:0/3896489988 wait complete. 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 librados: starting msgr at 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 librados: starting objecter 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 Processor -- start 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- start start 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 librados: setting wanted keys 2024-04-26T16:43:30.409 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 librados: calling monclient init 2024-04-26T16:43:30.410 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e806b130 0x7f17e81023f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 0x7f17e8102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81094a0 0x7f17e8102e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8112b10 con 0x7f17e806b130 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 -- 0x7f17e8112990 con 0x7f17e81094a0 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17eeb20700 1 -- --> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] -- mon_getmap magic: 0 -- 0x7f17e8112c90 con 0x7f17e806bad0 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17e7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 0x7f17e8102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 --2- >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e806b130 0x7f17e81023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17e7fff700 1 --2- >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 0x7f17e8102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.50:3301/0 says I am v2:172.21.15.50:43030/0 (socket says 172.21.15.50:43030) 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17e7fff700 1 -- 172.21.15.50:0/3319247769 learned_addr learned my addr 172.21.15.50:0/3319247769 (peer_addr_for_me v2:172.21.15.50:0/0) 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ed0bd700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81094a0 0x7f17e8102e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 -- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 msgr2=0x7f17e8102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 0x7f17e8102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.411 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 -- 172.21.15.50:0/3319247769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81094a0 msgr2=0x7f17e8102e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81094a0 0x7f17e8102e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 -- 172.21.15.50:0/3319247769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e81a76a0 con 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17e7fff700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 0x7f17e8102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ed0bd700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81094a0 0x7f17e8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17ec8bc700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e806b130 0x7f17e81023f0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f17d800e9f0 tx=0x7f17d800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17e5ffb700 10 client.?.objecter ms_handle_connect 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.407+0000 7f17e5ffb700 10 client.?.objecter resend_mon_ops 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d800cb70 con 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17e81a7990 con 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 2 ==== config(24 keys) ==== 1016+0+0 (secure 0 0 0) 0x7f17d8004480 con 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d8003bf0 con 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17eeb20700 10 client.31389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17e81a7f20 con 0x7f17e806b130 2024-04-26T16:43:30.412 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17eeb20700 1 librados: init done 2024-04-26T16:43:30.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17e806a110 con 0x7f17e806b130 2024-04-26T16:43:30.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 4 ==== mgrmap(e 22) ==== 93960+0+0 (secure 0 0 0) 0x7f17d8003dd0 con 0x7f17e806b130 2024-04-26T16:43:30.413 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f17d4071f80 0x7f17d4074440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 5 ==== osd_map(934..934 src has 884..934) ==== 8707+0+0 (secure 0 0 0) 0x7f17d80918f0 con 0x7f17e806b130 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e7fff700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f17d4071f80 0x7f17d4074440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 10 client.31389.objecter ms_dispatch 0x7f17e8001550 osd_map(934..934 src has 884..934) 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 3 client.31389.objecter handle_osd_map got epochs [934,934] > 0 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 3 client.31389.objecter handle_osd_map decoding full epoch 934 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 20 client.31389.objecter dump_active .. 0 homeless 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e7fff700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f17d4071f80 0x7f17d4074440 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f17e8079aa0 tx=0x7f17d0006210 comp rx=0 tx=0).ready entity=mgr.24401 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T16:43:30.414 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.411+0000 7f17e5ffb700 10 client.31389.objecter ms_handle_connect 0x7f17d4071f80 2024-04-26T16:43:30.417 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.415+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191733 (secure 0 0 0) 0x7f17d805d540 con 0x7f17e806b130 2024-04-26T16:43:30.638 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.639+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 --> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} v 0) -- 0x7f17e802e490 con 0x7f17e806b130 2024-04-26T16:43:30.644 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.643+0000 7f17e5ffb700 1 -- 172.21.15.50:0/3319247769 <== mon.0 v2:172.21.15.50:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]=0 v33) ==== 118+0+0 (secure 0 0 0) 0x7f17d80611f0 con 0x7f17e806b130 2024-04-26T16:43:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 10 librados: watch_flush enter 2024-04-26T16:43:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 10 librados: watch_flush exit 2024-04-26T16:43:30.648 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 20 client.31389.objecter shutdown clearing up homeless session... 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 10 client.31389.objecter successfully canceled tick 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f17d4071f80 msgr2=0x7f17d4074440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f17d4071f80 0x7f17d4074440 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f17e8079aa0 tx=0x7f17d0006210 comp rx=0 tx=0).stop 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e806b130 msgr2=0x7f17e81023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e806b130 0x7f17e81023f0 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f17d800e9f0 tx=0x7f17d800ed00 comp rx=0 tx=0).stop 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 shutdown_connections 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:6800/584348748,v1:172.21.15.50:6801/584348748] conn(0x7f17d4071f80 0x7f17d4074440 secure :-1 s=CLOSED pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f17e8079aa0 tx=0x7f17d0006210 comp rx=0 tx=0).stop 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17e81094a0 0x7f17e8102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3300/0,v1:172.21.15.50:6789/0] conn(0x7f17e806b130 0x7f17e81023f0 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 --2- 172.21.15.50:0/3319247769 >> [v2:172.21.15.50:3301/0,v1:172.21.15.50:6790/0] conn(0x7f17e806bad0 0x7f17e8102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 >> 172.21.15.50:0/3319247769 conn(0x7f17e80792d0 msgr2=0x7f17e8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 shutdown_connections 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 -- 172.21.15.50:0/3319247769 wait complete. 2024-04-26T16:43:30.649 INFO:teuthology.orchestra.run.smithi050.stderr:2024-04-26T16:43:30.647+0000 7f17eeb20700 1 librados: shutdown 2024-04-26T16:43:30.919 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21717]: [26/Apr/2024:16:43:30] ENGINE Bus STOPPING 2024-04-26T16:43:30.919 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21717]: [26/Apr/2024:16:43:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:43:30.919 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21717]: [26/Apr/2024:16:43:30] ENGINE Bus STOPPED 2024-04-26T16:43:30.919 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21717]: [26/Apr/2024:16:43:30] ENGINE Bus STARTING 2024-04-26T16:43:30.919 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21717]: [26/Apr/2024:16:43:30] ENGINE Serving on http://:::9283 2024-04-26T16:43:30.919 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21717]: [26/Apr/2024:16:43:30] ENGINE Bus STARTED 2024-04-26T16:43:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:30 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.645211+0000 osd.7 (osd.7) 678 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:43:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:30 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.645211+0000 osd.7 (osd.7) 678 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:43:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:30 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.647750+0000 osd.7 (osd.7) 679 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:43:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:30 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.647750+0000 osd.7 (osd.7) 679 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:43:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.735713+0000 osd.3 (osd.3) 695 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:43:31.325 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.735713+0000 osd.3 (osd.3) 695 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.738342+0000 osd.3 (osd.3) 696 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:29.738342+0000 osd.3 (osd.3) 696 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.058497+0000 osd.4 (osd.4) 585 : cluster [DBG] 6.e scrub starts 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.058497+0000 osd.4 (osd.4) 585 : cluster [DBG] 6.e scrub starts 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.060630+0000 osd.4 (osd.4) 586 : cluster [DBG] 6.e scrub ok 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.060630+0000 osd.4 (osd.4) 586 : cluster [DBG] 6.e scrub ok 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.163059+0000 mgr.y (mgr.24401) 990 : cluster [DBG] pgmap v1589: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.163059+0000 mgr.y (mgr.24401) 990 : cluster [DBG] pgmap v1589: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.220374+0000 osd.2 (osd.2) 567 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.220374+0000 osd.2 (osd.2) 567 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.222759+0000 osd.2 (osd.2) 568 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: cluster 2024-04-26T16:43:30.222759+0000 osd.2 (osd.2) 568 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.643943+0000 mon.a (mon.0) 4080 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-26T16:43:31.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.643943+0000 mon.a (mon.0) 4080 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.650022+0000 mon.a (mon.0) 4081 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.650022+0000 mon.a (mon.0) 4081 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.709971+0000 mon.c (mon.2) 963 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.709971+0000 mon.c (mon.2) 963 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.711799+0000 mon.c (mon.2) 964 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.711799+0000 mon.c (mon.2) 964 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.712601+0000 mon.c (mon.2) 965 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.712601+0000 mon.c (mon.2) 965 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.717619+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:43:31.327 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:31 smithi190 bash[23868]: audit 2024-04-26T16:43:30.717619+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:43:31.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.645211+0000 osd.7 (osd.7) 678 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:43:31.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.645211+0000 osd.7 (osd.7) 678 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:43:31.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.647750+0000 osd.7 (osd.7) 679 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:43:31.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.647750+0000 osd.7 (osd.7) 679 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:43:31.380 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.735713+0000 osd.3 (osd.3) 695 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.735713+0000 osd.3 (osd.3) 695 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.738342+0000 osd.3 (osd.3) 696 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:29.738342+0000 osd.3 (osd.3) 696 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.058497+0000 osd.4 (osd.4) 585 : cluster [DBG] 6.e scrub starts 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.058497+0000 osd.4 (osd.4) 585 : cluster [DBG] 6.e scrub starts 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.060630+0000 osd.4 (osd.4) 586 : cluster [DBG] 6.e scrub ok 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.060630+0000 osd.4 (osd.4) 586 : cluster [DBG] 6.e scrub ok 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.163059+0000 mgr.y (mgr.24401) 990 : cluster [DBG] pgmap v1589: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.163059+0000 mgr.y (mgr.24401) 990 : cluster [DBG] pgmap v1589: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.220374+0000 osd.2 (osd.2) 567 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.220374+0000 osd.2 (osd.2) 567 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.222759+0000 osd.2 (osd.2) 568 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: cluster 2024-04-26T16:43:30.222759+0000 osd.2 (osd.2) 568 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:43:31.381 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.643943+0000 mon.a (mon.0) 4080 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.643943+0000 mon.a (mon.0) 4080 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.650022+0000 mon.a (mon.0) 4081 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.650022+0000 mon.a (mon.0) 4081 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.709971+0000 mon.c (mon.2) 963 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.709971+0000 mon.c (mon.2) 963 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.645211+0000 osd.7 (osd.7) 678 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:30 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.645211+0000 osd.7 (osd.7) 678 : cluster [DBG] 5.10 scrub starts 2024-04-26T16:43:31.382 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.647750+0000 osd.7 (osd.7) 679 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.647750+0000 osd.7 (osd.7) 679 : cluster [DBG] 5.10 scrub ok 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.735713+0000 osd.3 (osd.3) 695 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.735713+0000 osd.3 (osd.3) 695 : cluster [DBG] 6.6 scrub starts 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.738342+0000 osd.3 (osd.3) 696 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:29.738342+0000 osd.3 (osd.3) 696 : cluster [DBG] 6.6 scrub ok 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.058497+0000 osd.4 (osd.4) 585 : cluster [DBG] 6.e scrub starts 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.058497+0000 osd.4 (osd.4) 585 : cluster [DBG] 6.e scrub starts 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.060630+0000 osd.4 (osd.4) 586 : cluster [DBG] 6.e scrub ok 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.060630+0000 osd.4 (osd.4) 586 : cluster [DBG] 6.e scrub ok 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.163059+0000 mgr.y (mgr.24401) 990 : cluster [DBG] pgmap v1589: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.163059+0000 mgr.y (mgr.24401) 990 : cluster [DBG] pgmap v1589: 174 pgs: 174 active+clean; 583 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.220374+0000 osd.2 (osd.2) 567 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.220374+0000 osd.2 (osd.2) 567 : cluster [DBG] 11.18 scrub starts 2024-04-26T16:43:31.383 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.222759+0000 osd.2 (osd.2) 568 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: cluster 2024-04-26T16:43:30.222759+0000 osd.2 (osd.2) 568 : cluster [DBG] 11.18 scrub ok 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.643943+0000 mon.a (mon.0) 4080 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.643943+0000 mon.a (mon.0) 4080 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.650022+0000 mon.a (mon.0) 4081 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.650022+0000 mon.a (mon.0) 4081 : audit [INF] from='client.? 172.21.15.50:0/3319247769' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.709971+0000 mon.c (mon.2) 963 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.709971+0000 mon.c (mon.2) 963 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.711799+0000 mon.c (mon.2) 964 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.711799+0000 mon.c (mon.2) 964 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.712601+0000 mon.c (mon.2) 965 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.712601+0000 mon.c (mon.2) 965 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.717619+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:43:31.384 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: audit 2024-04-26T16:43:30.717619+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:43:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.711799+0000 mon.c (mon.2) 964 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.711799+0000 mon.c (mon.2) 964 : audit [DBG] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-26T16:43:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.712601+0000 mon.c (mon.2) 965 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.712601+0000 mon.c (mon.2) 965 : audit [INF] from='mgr.24401 172.21.15.50:0/3014928678' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-26T16:43:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.717619+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:43:31.385 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[29234]: audit 2024-04-26T16:43:30.717619+0000 mon.a (mon.0) 4082 : audit [INF] from='mgr.24401 ' entity='mgr.y' 2024-04-26T16:43:31.516 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-26T16:43:31.525 INFO:tasks.cephadm:Teardown begin 2024-04-26T16:43:31.525 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:43:31.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:43:31.553 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-26T16:43:31.554 DEBUG:teuthology.orchestra.run.smithi050:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T16:43:31.586 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-26T16:43:31.604 INFO:tasks.cephadm:Stopping all daemons... 2024-04-26T16:43:31.604 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-26T16:43:31.604 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a 2024-04-26T16:43:31.784 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 systemd[1]: Stopping Ceph mon.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:32.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: debug 2024-04-26T16:43:31.783+0000 7f60ce7e8700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-26T16:43:32.094 INFO:journalctl@ceph.mon.a.smithi050.stdout:Apr 26 16:43:31 smithi050 bash[21463]: debug 2024-04-26T16:43:31.783+0000 7f60ce7e8700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-26T16:43:32.911 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.a.service' 2024-04-26T16:43:32.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:32.937 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-26T16:43:32.937 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-26T16:43:32.937 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.c 2024-04-26T16:43:33.247 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[21717]: [26/Apr/2024:16:43:33] ENGINE Bus STOPPING 2024-04-26T16:43:33.247 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[21717]: [26/Apr/2024:16:43:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-26T16:43:33.247 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[21717]: [26/Apr/2024:16:43:33] ENGINE Bus STOPPED 2024-04-26T16:43:33.247 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[21717]: [26/Apr/2024:16:43:33] ENGINE Bus STARTING 2024-04-26T16:43:33.247 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:32 smithi050 systemd[1]: Stopping Ceph mon.c for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:33.247 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[29234]: debug 2024-04-26T16:43:33.119+0000 7f77f1f33700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-26T16:43:33.248 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[29234]: debug 2024-04-26T16:43:33.119+0000 7f77f1f33700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-04-26T16:43:33.535 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[21717]: [26/Apr/2024:16:43:33] ENGINE Serving on http://:::9283 2024-04-26T16:43:33.536 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[21717]: [26/Apr/2024:16:43:33] ENGINE Bus STARTED 2024-04-26T16:43:34.233 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.c.service' 2024-04-26T16:43:34.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:33 smithi050 bash[215292]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-mon-c 2024-04-26T16:43:34.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:34 smithi050 bash[215362]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-mon-c 2024-04-26T16:43:34.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:34 smithi050 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-26T16:43:34.242 INFO:journalctl@ceph.mon.c.smithi050.stdout:Apr 26 16:43:34 smithi050 systemd[1]: Stopped Ceph mon.c for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:43:34.260 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:34.260 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-26T16:43:34.261 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-26T16:43:34.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.b 2024-04-26T16:43:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:34 smithi190 systemd[1]: Stopping Ceph mon.b for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:34.574 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:34 smithi190 bash[23868]: debug 2024-04-26T16:43:34.515+0000 7f3887c34700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-04-26T16:43:34.575 INFO:journalctl@ceph.mon.b.smithi190.stdout:Apr 26 16:43:34 smithi190 bash[23868]: debug 2024-04-26T16:43:34.515+0000 7f3887c34700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-04-26T16:43:34.629 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:34 smithi050 bash[21717]: ::ffff:172.21.15.190 - - [26/Apr/2024:16:43:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-04-26T16:43:35.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mon.b.service' 2024-04-26T16:43:35.569 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:35.569 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-26T16:43:35.569 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-26T16:43:35.569 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y 2024-04-26T16:43:35.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:35 smithi050 systemd[1]: Stopping Ceph mgr.y for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:35.879 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:35 smithi050 bash[21717]: debug 2024-04-26T16:43:35.715+0000 7f2c92cb0700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:43:35.880 INFO:journalctl@ceph.mgr.y.smithi050.stdout:Apr 26 16:43:35 smithi050 bash[21717]: debug 2024-04-26T16:43:35.715+0000 7f2c92cb0700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-26T16:43:36.792 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-26T16:43:36.816 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:36.816 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-26T16:43:36.816 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-26T16:43:36.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.x 2024-04-26T16:43:37.170 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Apr 26 16:43:36 smithi190 systemd[1]: Stopping Ceph mgr.x for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:37.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-26T16:43:37.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:37.837 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-26T16:43:37.837 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-26T16:43:37.837 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.0 2024-04-26T16:43:38.130 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:43:37 smithi050 systemd[1]: Stopping Ceph osd.0 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:38.130 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:43:37 smithi050 bash[32273]: debug 2024-04-26T16:43:37.975+0000 7fec9a7f7700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:43:38.130 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:43:37 smithi050 bash[32273]: debug 2024-04-26T16:43:37.975+0000 7fec9a7f7700 -1 osd.0 934 *** Got signal Terminated *** 2024-04-26T16:43:38.130 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:43:37 smithi050 bash[32273]: debug 2024-04-26T16:43:37.975+0000 7fec9a7f7700 -1 osd.0 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:43:44.629 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:43:44 smithi050 bash[215500]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-0 2024-04-26T16:43:44.629 INFO:journalctl@ceph.osd.0.smithi050.stdout:Apr 26 16:43:44 smithi050 bash[215565]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-0 2024-04-26T16:43:46.617 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.0.service' 2024-04-26T16:43:46.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:46.647 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-26T16:43:46.647 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-26T16:43:46.648 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.1 2024-04-26T16:43:46.880 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:43:46 smithi050 systemd[1]: Stopping Ceph osd.1 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:46.880 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:43:46 smithi050 bash[37610]: debug 2024-04-26T16:43:46.811+0000 7fa93bb36700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:43:46.880 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:43:46 smithi050 bash[37610]: debug 2024-04-26T16:43:46.811+0000 7fa93bb36700 -1 osd.1 934 *** Got signal Terminated *** 2024-04-26T16:43:46.880 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:43:46 smithi050 bash[37610]: debug 2024-04-26T16:43:46.811+0000 7fa93bb36700 -1 osd.1 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:43:53.379 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:43:53 smithi050 bash[215700]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-1 2024-04-26T16:43:53.379 INFO:journalctl@ceph.osd.1.smithi050.stdout:Apr 26 16:43:53 smithi050 bash[215766]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-1 2024-04-26T16:43:55.326 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.1.service' 2024-04-26T16:43:55.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:43:55.356 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-26T16:43:55.357 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-26T16:43:55.357 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.2 2024-04-26T16:43:55.630 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:43:55 smithi050 systemd[1]: Stopping Ceph osd.2 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:43:55.630 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:43:55 smithi050 bash[43416]: debug 2024-04-26T16:43:55.542+0000 7f92eb119700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:43:55.630 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:43:55 smithi050 bash[43416]: debug 2024-04-26T16:43:55.542+0000 7f92eb119700 -1 osd.2 934 *** Got signal Terminated *** 2024-04-26T16:43:55.630 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:43:55 smithi050 bash[43416]: debug 2024-04-26T16:43:55.542+0000 7f92eb119700 -1 osd.2 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:44:02.161 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:44:01 smithi050 bash[215897]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-2 2024-04-26T16:44:02.161 INFO:journalctl@ceph.osd.2.smithi050.stdout:Apr 26 16:44:02 smithi050 bash[215960]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-2 2024-04-26T16:44:04.051 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.2.service' 2024-04-26T16:44:04.078 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:04.079 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-26T16:44:04.079 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-26T16:44:04.079 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.3 2024-04-26T16:44:04.380 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:04 smithi050 systemd[1]: Stopping Ceph osd.3 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:04.380 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:04 smithi050 bash[49169]: debug 2024-04-26T16:44:04.266+0000 7f553b965700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:44:04.380 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:04 smithi050 bash[49169]: debug 2024-04-26T16:44:04.266+0000 7f553b965700 -1 osd.3 934 *** Got signal Terminated *** 2024-04-26T16:44:04.380 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:04 smithi050 bash[49169]: debug 2024-04-26T16:44:04.266+0000 7f553b965700 -1 osd.3 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:44:05.520 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:05 smithi190 bash[92888]: debug 2024-04-26T16:44:05.138+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:05.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:05 smithi190 bash[43438]: debug 2024-04-26T16:44:05.518+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:06.324 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:06 smithi190 bash[92888]: debug 2024-04-26T16:44:06.094+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:06.325 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:05 smithi190 bash[27377]: debug 2024-04-26T16:44:05.966+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:06.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:06 smithi190 bash[43438]: debug 2024-04-26T16:44:06.502+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:07.324 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:06 smithi190 bash[27377]: debug 2024-04-26T16:44:06.950+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:07.324 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:07 smithi190 bash[92888]: debug 2024-04-26T16:44:07.086+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:07.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:07 smithi190 bash[43438]: debug 2024-04-26T16:44:07.478+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:08.324 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:07 smithi190 bash[27377]: debug 2024-04-26T16:44:07.910+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:08.324 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:08 smithi190 bash[92888]: debug 2024-04-26T16:44:08.098+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:08.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:07 smithi190 bash[32690]: debug 2024-04-26T16:44:07.942+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:08.379 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:07 smithi050 bash[49169]: debug 2024-04-26T16:44:07.910+0000 7f5533f51700 -1 osd.3 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.876541+0000 front 2024-04-26T16:43:41.876615+0000 (oldest deadline 2024-04-26T16:44:07.776603+0000) 2024-04-26T16:44:08.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:08 smithi190 bash[43438]: debug 2024-04-26T16:44:08.458+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:09.142 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:08 smithi190 bash[27377]: debug 2024-04-26T16:44:08.878+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:09.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:08 smithi190 bash[32690]: debug 2024-04-26T16:44:08.922+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:09.379 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:08 smithi050 bash[49169]: debug 2024-04-26T16:44:08.910+0000 7f5533f51700 -1 osd.3 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.876541+0000 front 2024-04-26T16:43:41.876615+0000 (oldest deadline 2024-04-26T16:44:07.776603+0000) 2024-04-26T16:44:09.448 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:09 smithi190 bash[92888]: debug 2024-04-26T16:44:09.142+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:09.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:09 smithi190 bash[43438]: debug 2024-04-26T16:44:09.450+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:10.166 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:09 smithi190 bash[27377]: debug 2024-04-26T16:44:09.838+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:10.166 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:09 smithi190 bash[32690]: debug 2024-04-26T16:44:09.906+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:10.424 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:10 smithi190 bash[92888]: debug 2024-04-26T16:44:10.170+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:10.635 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:10 smithi050 bash[216092]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-3 2024-04-26T16:44:10.635 INFO:journalctl@ceph.osd.3.smithi050.stdout:Apr 26 16:44:10 smithi050 bash[216154]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-3 2024-04-26T16:44:10.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:10 smithi190 bash[43438]: debug 2024-04-26T16:44:10.426+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:11.206 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:10 smithi190 bash[27377]: debug 2024-04-26T16:44:10.834+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:11.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:10 smithi190 bash[32690]: debug 2024-04-26T16:44:10.898+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:11.574 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:11 smithi190 bash[43438]: debug 2024-04-26T16:44:11.430+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:11.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:11 smithi190 bash[92888]: debug 2024-04-26T16:44:11.206+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:12.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:11 smithi190 bash[27377]: debug 2024-04-26T16:44:11.810+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:12.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:11 smithi190 bash[32690]: debug 2024-04-26T16:44:11.878+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:12.379 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.3.service' 2024-04-26T16:44:12.406 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:12.406 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-26T16:44:12.406 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-26T16:44:12.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.4 2024-04-26T16:44:12.574 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:12 smithi190 systemd[1]: Stopping Ceph osd.4 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:12.575 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[92888]: debug 2024-04-26T16:44:12.242+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:12.575 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[43438]: debug 2024-04-26T16:44:12.406+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:12.824 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[27377]: debug 2024-04-26T16:44:12.634+0000 7f5cd53da700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:44:12.824 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[27377]: debug 2024-04-26T16:44:12.634+0000 7f5cd53da700 -1 osd.4 934 *** Got signal Terminated *** 2024-04-26T16:44:12.824 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[27377]: debug 2024-04-26T16:44:12.634+0000 7f5cd53da700 -1 osd.4 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:44:13.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[27377]: debug 2024-04-26T16:44:12.826+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:13.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:12 smithi190 bash[32690]: debug 2024-04-26T16:44:12.882+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:13.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:13 smithi190 bash[92888]: debug 2024-04-26T16:44:13.290+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:13.574 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:13 smithi190 bash[43438]: debug 2024-04-26T16:44:13.438+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:14.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:13 smithi190 bash[27377]: debug 2024-04-26T16:44:13.810+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:14.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:13 smithi190 bash[32690]: debug 2024-04-26T16:44:13.890+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:14.075 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:13 smithi190 bash[32690]: debug 2024-04-26T16:44:13.890+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:14.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:14 smithi190 bash[92888]: debug 2024-04-26T16:44:14.330+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:14.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:14 smithi190 bash[43438]: debug 2024-04-26T16:44:14.474+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:14.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:14 smithi190 bash[43438]: debug 2024-04-26T16:44:14.474+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:15.324 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:14 smithi190 bash[27377]: debug 2024-04-26T16:44:14.826+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:14 smithi190 bash[32690]: debug 2024-04-26T16:44:14.862+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:14 smithi190 bash[32690]: debug 2024-04-26T16:44:14.862+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:15.823 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[92888]: debug 2024-04-26T16:44:15.374+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:15.823 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[92888]: debug 2024-04-26T16:44:15.374+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:15.823 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[43438]: debug 2024-04-26T16:44:15.478+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:15.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[43438]: debug 2024-04-26T16:44:15.478+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:16.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[27377]: debug 2024-04-26T16:44:15.826+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:16.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[27377]: debug 2024-04-26T16:44:15.826+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:46.727049+0000 front 2024-04-26T16:43:46.727102+0000 (oldest deadline 2024-04-26T16:44:14.927537+0000) 2024-04-26T16:44:16.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[32690]: debug 2024-04-26T16:44:15.826+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:16.075 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:15 smithi190 bash[32690]: debug 2024-04-26T16:44:15.826+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:16.796 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[92888]: debug 2024-04-26T16:44:16.358+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:16.796 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[92888]: debug 2024-04-26T16:44:16.358+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:16.796 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[43438]: debug 2024-04-26T16:44:16.526+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:16.796 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[43438]: debug 2024-04-26T16:44:16.526+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:17.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[27377]: debug 2024-04-26T16:44:16.834+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:40.324989+0000 front 2024-04-26T16:43:40.325267+0000 (oldest deadline 2024-04-26T16:44:05.625112+0000) 2024-04-26T16:44:17.074 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[27377]: debug 2024-04-26T16:44:16.834+0000 7f5cd19ce700 -1 osd.4 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:46.727049+0000 front 2024-04-26T16:43:46.727102+0000 (oldest deadline 2024-04-26T16:44:14.927537+0000) 2024-04-26T16:44:17.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[32690]: debug 2024-04-26T16:44:16.794+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:17.075 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:16 smithi190 bash[32690]: debug 2024-04-26T16:44:16.794+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:17.764 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:17 smithi190 bash[92888]: debug 2024-04-26T16:44:17.358+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:17.765 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:17 smithi190 bash[92888]: debug 2024-04-26T16:44:17.358+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:17.765 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:17 smithi190 bash[43438]: debug 2024-04-26T16:44:17.566+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:17.765 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:17 smithi190 bash[43438]: debug 2024-04-26T16:44:17.566+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:18.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:17 smithi190 bash[32690]: debug 2024-04-26T16:44:17.766+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:18.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:17 smithi190 bash[32690]: debug 2024-04-26T16:44:17.766+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:18.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[92888]: debug 2024-04-26T16:44:18.350+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:18.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[92888]: debug 2024-04-26T16:44:18.350+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:18.964 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[102340]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-4 2024-04-26T16:44:18.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[32690]: debug 2024-04-26T16:44:18.806+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:18.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[32690]: debug 2024-04-26T16:44:18.806+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:18.965 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[43438]: debug 2024-04-26T16:44:18.598+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:18.965 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[43438]: debug 2024-04-26T16:44:18.598+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:19.324 INFO:journalctl@ceph.osd.4.smithi190.stdout:Apr 26 16:44:18 smithi190 bash[102409]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-4 2024-04-26T16:44:19.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:19 smithi190 bash[92888]: debug 2024-04-26T16:44:19.374+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:19.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:19 smithi190 bash[92888]: debug 2024-04-26T16:44:19.374+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:19.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:19 smithi190 bash[32690]: debug 2024-04-26T16:44:19.846+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:19.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:19 smithi190 bash[32690]: debug 2024-04-26T16:44:19.846+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:19.909 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:19 smithi190 bash[43438]: debug 2024-04-26T16:44:19.630+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:19.909 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:19 smithi190 bash[43438]: debug 2024-04-26T16:44:19.630+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:20.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:20 smithi190 bash[92888]: debug 2024-04-26T16:44:20.414+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:20.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:20 smithi190 bash[92888]: debug 2024-04-26T16:44:20.414+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:20.950 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:20 smithi190 bash[32690]: debug 2024-04-26T16:44:20.838+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:20.950 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:20 smithi190 bash[32690]: debug 2024-04-26T16:44:20.838+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:20.950 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:20 smithi190 bash[43438]: debug 2024-04-26T16:44:20.650+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:20.950 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:20 smithi190 bash[43438]: debug 2024-04-26T16:44:20.650+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:20.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.4.service' 2024-04-26T16:44:20.983 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:20.983 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-26T16:44:20.983 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-26T16:44:20.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.5 2024-04-26T16:44:21.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 systemd[1]: Stopping Ceph osd.5 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:21.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[32690]: debug 2024-04-26T16:44:21.138+0000 7f1a4d262700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:44:21.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[32690]: debug 2024-04-26T16:44:21.138+0000 7f1a4d262700 -1 osd.5 934 *** Got signal Terminated *** 2024-04-26T16:44:21.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[32690]: debug 2024-04-26T16:44:21.138+0000 7f1a4d262700 -1 osd.5 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:44:21.671 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[92888]: debug 2024-04-26T16:44:21.406+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:21.671 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[92888]: debug 2024-04-26T16:44:21.406+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:22.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[32690]: debug 2024-04-26T16:44:21.850+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:22.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[32690]: debug 2024-04-26T16:44:21.850+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:22.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[32690]: debug 2024-04-26T16:44:21.850+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:58.172719+0000 front 2024-04-26T16:43:58.172581+0000 (oldest deadline 2024-04-26T16:44:21.672446+0000) 2024-04-26T16:44:22.075 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[43438]: debug 2024-04-26T16:44:21.674+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:22.075 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[43438]: debug 2024-04-26T16:44:21.674+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:22.075 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:21 smithi190 bash[43438]: debug 2024-04-26T16:44:21.674+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:22.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[92888]: debug 2024-04-26T16:44:22.450+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:22.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[92888]: debug 2024-04-26T16:44:22.450+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:22.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[92888]: debug 2024-04-26T16:44:22.450+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:22.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[43438]: debug 2024-04-26T16:44:22.674+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:22.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[43438]: debug 2024-04-26T16:44:22.674+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:22.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[43438]: debug 2024-04-26T16:44:22.674+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:23.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[32690]: debug 2024-04-26T16:44:22.842+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:23.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[32690]: debug 2024-04-26T16:44:22.842+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:23.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:22 smithi190 bash[32690]: debug 2024-04-26T16:44:22.842+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:58.172719+0000 front 2024-04-26T16:43:58.172581+0000 (oldest deadline 2024-04-26T16:44:21.672446+0000) 2024-04-26T16:44:23.817 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[43438]: debug 2024-04-26T16:44:23.658+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:23.818 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[43438]: debug 2024-04-26T16:44:23.658+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:23.818 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[43438]: debug 2024-04-26T16:44:23.658+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:23.818 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[92888]: debug 2024-04-26T16:44:23.442+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:23.818 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[92888]: debug 2024-04-26T16:44:23.442+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:23.818 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[92888]: debug 2024-04-26T16:44:23.442+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:24.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[32690]: debug 2024-04-26T16:44:23.822+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:24.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[32690]: debug 2024-04-26T16:44:23.822+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:24.074 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:23 smithi190 bash[32690]: debug 2024-04-26T16:44:23.822+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:58.172719+0000 front 2024-04-26T16:43:58.172581+0000 (oldest deadline 2024-04-26T16:44:21.672446+0000) 2024-04-26T16:44:24.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[92888]: debug 2024-04-26T16:44:24.470+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:24.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[92888]: debug 2024-04-26T16:44:24.470+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:24.825 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[92888]: debug 2024-04-26T16:44:24.470+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:24.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[43438]: debug 2024-04-26T16:44:24.638+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:24.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[43438]: debug 2024-04-26T16:44:24.638+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:24.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[43438]: debug 2024-04-26T16:44:24.638+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:25.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[32690]: debug 2024-04-26T16:44:24.842+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:25.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[32690]: debug 2024-04-26T16:44:24.842+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:25.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:24 smithi190 bash[32690]: debug 2024-04-26T16:44:24.842+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:58.172719+0000 front 2024-04-26T16:43:58.172581+0000 (oldest deadline 2024-04-26T16:44:21.672446+0000) 2024-04-26T16:44:25.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[92888]: debug 2024-04-26T16:44:25.502+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:25.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[92888]: debug 2024-04-26T16:44:25.502+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:25.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[92888]: debug 2024-04-26T16:44:25.502+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:25.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[43438]: debug 2024-04-26T16:44:25.630+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:25.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[43438]: debug 2024-04-26T16:44:25.630+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:25.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[43438]: debug 2024-04-26T16:44:25.630+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:26.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[32690]: debug 2024-04-26T16:44:25.858+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:42.468506+0000 front 2024-04-26T16:43:42.468557+0000 (oldest deadline 2024-04-26T16:44:07.768580+0000) 2024-04-26T16:44:26.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[32690]: debug 2024-04-26T16:44:25.858+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:51.270364+0000 front 2024-04-26T16:43:51.269882+0000 (oldest deadline 2024-04-26T16:44:13.570055+0000) 2024-04-26T16:44:26.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:25 smithi190 bash[32690]: debug 2024-04-26T16:44:25.858+0000 7f1a49856700 -1 osd.5 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:58.172719+0000 front 2024-04-26T16:43:58.172581+0000 (oldest deadline 2024-04-26T16:44:21.672446+0000) 2024-04-26T16:44:26.742 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:26 smithi190 bash[92888]: debug 2024-04-26T16:44:26.486+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:26.742 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:26 smithi190 bash[92888]: debug 2024-04-26T16:44:26.486+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:26.742 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:26 smithi190 bash[92888]: debug 2024-04-26T16:44:26.486+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:26.742 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:26 smithi190 bash[43438]: debug 2024-04-26T16:44:26.618+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:26.742 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:26 smithi190 bash[43438]: debug 2024-04-26T16:44:26.618+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:26.742 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:26 smithi190 bash[43438]: debug 2024-04-26T16:44:26.618+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:27.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[102543]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-5 2024-04-26T16:44:27.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[102609]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-5 2024-04-26T16:44:27.598 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[92888]: debug 2024-04-26T16:44:27.446+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:27.598 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[92888]: debug 2024-04-26T16:44:27.446+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:27.598 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[92888]: debug 2024-04-26T16:44:27.446+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:28.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[43438]: debug 2024-04-26T16:44:27.602+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:28.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[43438]: debug 2024-04-26T16:44:27.602+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:28.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:27 smithi190 bash[43438]: debug 2024-04-26T16:44:27.602+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:28.800 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:28 smithi190 bash[43438]: debug 2024-04-26T16:44:28.586+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:28.801 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:28 smithi190 bash[43438]: debug 2024-04-26T16:44:28.586+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:28.801 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:28 smithi190 bash[43438]: debug 2024-04-26T16:44:28.586+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:28.801 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:28 smithi190 bash[92888]: debug 2024-04-26T16:44:28.446+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:28.801 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:28 smithi190 bash[92888]: debug 2024-04-26T16:44:28.446+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:28.801 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:28 smithi190 bash[92888]: debug 2024-04-26T16:44:28.446+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:29.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.5.service' 2024-04-26T16:44:29.405 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:29.405 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-26T16:44:29.406 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-26T16:44:29.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[92888]: debug 2024-04-26T16:44:29.433+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[92888]: debug 2024-04-26T16:44:29.433+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[92888]: debug 2024-04-26T16:44:29.433+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 systemd[1]: Stopping Ceph osd.6 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[92888]: debug 2024-04-26T16:44:29.581+0000 7f2071a3e700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[92888]: debug 2024-04-26T16:44:29.581+0000 7f2071a3e700 -1 osd.6 934 *** Got signal Terminated *** 2024-04-26T16:44:29.630 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[92888]: debug 2024-04-26T16:44:29.581+0000 7f2071a3e700 -1 osd.6 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:44:30.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[43438]: debug 2024-04-26T16:44:29.629+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:30.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[43438]: debug 2024-04-26T16:44:29.629+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:30.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:29 smithi190 bash[43438]: debug 2024-04-26T16:44:29.629+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:30.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:30 smithi190 bash[92888]: debug 2024-04-26T16:44:30.425+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:30.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:30 smithi190 bash[92888]: debug 2024-04-26T16:44:30.425+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:30.824 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:30 smithi190 bash[92888]: debug 2024-04-26T16:44:30.425+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:30.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:30 smithi190 bash[43438]: debug 2024-04-26T16:44:30.617+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:30.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:30 smithi190 bash[43438]: debug 2024-04-26T16:44:30.617+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:30.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:30 smithi190 bash[43438]: debug 2024-04-26T16:44:30.617+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:31.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[43438]: debug 2024-04-26T16:44:31.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:31.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[43438]: debug 2024-04-26T16:44:31.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:31.825 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[43438]: debug 2024-04-26T16:44:31.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:31.825 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[92888]: debug 2024-04-26T16:44:31.401+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:31.825 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[92888]: debug 2024-04-26T16:44:31.401+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:31.825 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[92888]: debug 2024-04-26T16:44:31.401+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:31.825 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:31 smithi190 bash[92888]: debug 2024-04-26T16:44:31.401+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:03.755517+0000 front 2024-04-26T16:44:03.755682+0000 (oldest deadline 2024-04-26T16:44:31.355867+0000) 2024-04-26T16:44:32.673 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[92888]: debug 2024-04-26T16:44:32.369+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:32.673 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[92888]: debug 2024-04-26T16:44:32.369+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:32.673 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[92888]: debug 2024-04-26T16:44:32.369+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:32.673 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[92888]: debug 2024-04-26T16:44:32.369+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:03.755517+0000 front 2024-04-26T16:44:03.755682+0000 (oldest deadline 2024-04-26T16:44:31.355867+0000) 2024-04-26T16:44:33.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[43438]: debug 2024-04-26T16:44:32.673+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:33.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[43438]: debug 2024-04-26T16:44:32.673+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:33.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:32 smithi190 bash[43438]: debug 2024-04-26T16:44:32.673+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:33.627 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[92888]: debug 2024-04-26T16:44:33.333+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:33.627 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[92888]: debug 2024-04-26T16:44:33.333+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:33.627 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[92888]: debug 2024-04-26T16:44:33.333+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:33.627 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[92888]: debug 2024-04-26T16:44:33.333+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:03.755517+0000 front 2024-04-26T16:44:03.755682+0000 (oldest deadline 2024-04-26T16:44:31.355867+0000) 2024-04-26T16:44:34.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[43438]: debug 2024-04-26T16:44:33.625+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:34.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[43438]: debug 2024-04-26T16:44:33.625+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:34.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:33 smithi190 bash[43438]: debug 2024-04-26T16:44:33.625+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:34.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[92888]: debug 2024-04-26T16:44:34.321+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.651042+0000 front 2024-04-26T16:43:41.651172+0000 (oldest deadline 2024-04-26T16:44:04.550868+0000) 2024-04-26T16:44:34.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[92888]: debug 2024-04-26T16:44:34.321+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:50.452298+0000 front 2024-04-26T16:43:50.452460+0000 (oldest deadline 2024-04-26T16:44:15.152289+0000) 2024-04-26T16:44:34.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[92888]: debug 2024-04-26T16:44:34.321+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.253637+0000 front 2024-04-26T16:43:59.253838+0000 (oldest deadline 2024-04-26T16:44:22.153485+0000) 2024-04-26T16:44:34.574 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[92888]: debug 2024-04-26T16:44:34.321+0000 7f206e032700 -1 osd.6 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:03.755517+0000 front 2024-04-26T16:44:03.755682+0000 (oldest deadline 2024-04-26T16:44:31.355867+0000) 2024-04-26T16:44:35.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[43438]: debug 2024-04-26T16:44:34.661+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:35.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[43438]: debug 2024-04-26T16:44:34.661+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:35.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:34 smithi190 bash[43438]: debug 2024-04-26T16:44:34.661+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:35.707 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[51340]: ts=2024-04-26T16:44:35.456Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-26T16:44:35.707 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[51340]: ts=2024-04-26T16:44:35.457Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-26T16:44:35.708 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[51340]: ts=2024-04-26T16:44:35.461Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-26T16:44:35.708 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[51340]: ts=2024-04-26T16:44:35.464Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-26T16:44:35.708 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[51340]: ts=2024-04-26T16:44:35.464Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.50:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.50:8765: connect: connection refused" 2024-04-26T16:44:35.970 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[102743]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-6 2024-04-26T16:44:35.970 INFO:journalctl@ceph.osd.6.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[102808]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-6 2024-04-26T16:44:35.971 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[43438]: debug 2024-04-26T16:44:35.709+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:35.971 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[43438]: debug 2024-04-26T16:44:35.709+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:35.971 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[43438]: debug 2024-04-26T16:44:35.709+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:35.971 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:35 smithi190 bash[43438]: debug 2024-04-26T16:44:35.709+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:37.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:36 smithi190 bash[43438]: debug 2024-04-26T16:44:36.697+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:37.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:36 smithi190 bash[43438]: debug 2024-04-26T16:44:36.697+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:37.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:36 smithi190 bash[43438]: debug 2024-04-26T16:44:36.697+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:37.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:36 smithi190 bash[43438]: debug 2024-04-26T16:44:36.697+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:37.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.6.service' 2024-04-26T16:44:37.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:37.779 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-26T16:44:37.780 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-26T16:44:37.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.7 2024-04-26T16:44:37.966 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.681+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:37.966 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.681+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:37.966 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.681+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:37.966 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.681+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:37.966 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 systemd[1]: Stopping Ceph osd.7 for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:38.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.965+0000 7f9b01317700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:44:38.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.965+0000 7f9b01317700 -1 osd.7 934 *** Got signal Terminated *** 2024-04-26T16:44:38.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:37 smithi190 bash[43438]: debug 2024-04-26T16:44:37.965+0000 7f9b01317700 -1 osd.7 934 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-26T16:44:39.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:38 smithi190 bash[43438]: debug 2024-04-26T16:44:38.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:39.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:38 smithi190 bash[43438]: debug 2024-04-26T16:44:38.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:39.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:38 smithi190 bash[43438]: debug 2024-04-26T16:44:38.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:39.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:38 smithi190 bash[43438]: debug 2024-04-26T16:44:38.633+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:40.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:39 smithi190 bash[43438]: debug 2024-04-26T16:44:39.593+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:40.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:39 smithi190 bash[43438]: debug 2024-04-26T16:44:39.593+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:40.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:39 smithi190 bash[43438]: debug 2024-04-26T16:44:39.593+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:40.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:39 smithi190 bash[43438]: debug 2024-04-26T16:44:39.593+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:41.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:40 smithi190 bash[43438]: debug 2024-04-26T16:44:40.597+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:41.074 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:40 smithi190 bash[43438]: debug 2024-04-26T16:44:40.597+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:41.075 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:40 smithi190 bash[43438]: debug 2024-04-26T16:44:40.597+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:41.075 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:40 smithi190 bash[43438]: debug 2024-04-26T16:44:40.597+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:41.075 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:40 smithi190 bash[43438]: debug 2024-04-26T16:44:40.597+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-04-26T16:44:15.105602+0000 front 2024-04-26T16:44:15.105675+0000 (oldest deadline 2024-04-26T16:44:39.805415+0000) 2024-04-26T16:44:41.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:41 smithi190 bash[43438]: debug 2024-04-26T16:44:41.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:41.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:41 smithi190 bash[43438]: debug 2024-04-26T16:44:41.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:41.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:41 smithi190 bash[43438]: debug 2024-04-26T16:44:41.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:41.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:41 smithi190 bash[43438]: debug 2024-04-26T16:44:41.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:41.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:41 smithi190 bash[43438]: debug 2024-04-26T16:44:41.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-04-26T16:44:15.105602+0000 front 2024-04-26T16:44:15.105675+0000 (oldest deadline 2024-04-26T16:44:39.805415+0000) 2024-04-26T16:44:42.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:42 smithi190 bash[43438]: debug 2024-04-26T16:44:42.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6806 osd.0 since back 2024-04-26T16:43:41.100067+0000 front 2024-04-26T16:43:41.100291+0000 (oldest deadline 2024-04-26T16:44:05.199803+0000) 2024-04-26T16:44:42.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:42 smithi190 bash[43438]: debug 2024-04-26T16:44:42.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6814 osd.1 since back 2024-04-26T16:43:49.301485+0000 front 2024-04-26T16:43:49.301877+0000 (oldest deadline 2024-04-26T16:44:14.001456+0000) 2024-04-26T16:44:42.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:42 smithi190 bash[43438]: debug 2024-04-26T16:44:42.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6822 osd.2 since back 2024-04-26T16:43:59.303198+0000 front 2024-04-26T16:43:59.303077+0000 (oldest deadline 2024-04-26T16:44:21.602760+0000) 2024-04-26T16:44:42.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:42 smithi190 bash[43438]: debug 2024-04-26T16:44:42.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.50:6830 osd.3 since back 2024-04-26T16:44:09.205096+0000 front 2024-04-26T16:44:09.204979+0000 (oldest deadline 2024-04-26T16:44:35.104790+0000) 2024-04-26T16:44:42.824 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:42 smithi190 bash[43438]: debug 2024-04-26T16:44:42.553+0000 7f9afd90b700 -1 osd.7 934 heartbeat_check: no reply from 172.21.15.190:6804 osd.4 since back 2024-04-26T16:44:15.105602+0000 front 2024-04-26T16:44:15.105675+0000 (oldest deadline 2024-04-26T16:44:39.805415+0000) 2024-04-26T16:44:44.574 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:44 smithi190 bash[102944]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-7 2024-04-26T16:44:44.574 INFO:journalctl@ceph.osd.7.smithi190.stdout:Apr 26 16:44:44 smithi190 bash[103009]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-osd-7 2024-04-26T16:44:46.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@osd.7.service' 2024-04-26T16:44:46.170 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:46.170 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-26T16:44:46.170 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-04-26T16:44:46.170 DEBUG:teuthology.orchestra.run.smithi050:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@rgw.foo.a 2024-04-26T16:44:46.630 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:44:46 smithi050 systemd[1]: Stopping Ceph rgw.foo.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:46.630 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:44:46 smithi050 bash[53323]: debug 2024-04-26T16:44:46.317+0000 7fbdb3d90700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2024-04-26T16:44:46.630 INFO:journalctl@ceph.rgw.foo.a.smithi050.stdout:Apr 26 16:44:46 smithi050 bash[53323]: debug 2024-04-26T16:44:46.317+0000 7fbdc2619ac0 -1 shutting down 2024-04-26T16:44:57.091 DEBUG:teuthology.orchestra.run.smithi050:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@rgw.foo.a.service' 2024-04-26T16:44:57.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:57.114 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-04-26T16:44:57.114 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-26T16:44:57.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@prometheus.a 2024-04-26T16:44:57.516 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 systemd[1]: Stopping Ceph prometheus.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:44:57.516 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-26T16:44:57.516 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.279Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.281Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.281Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-04-26T16:44:57.517 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Apr 26 16:44:57 smithi190 bash[51340]: ts=2024-04-26T16:44:57.281Z caller=main.go:1147 level=info msg="See you next time!" 2024-04-26T16:44:58.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-26T16:44:58.064 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-26T16:44:58.065 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-26T16:44:58.065 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T16:44:58.237 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:45:03.085 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:02 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.085 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:02 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.341 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.342 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.630 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.630 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.959 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:03.959 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.243 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.243 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.243 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:03 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.243 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.548 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.549 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.880 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:04.880 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: Stopping Ceph node-exporter.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:45:04.880 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:04 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:06.271 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:05 smithi050 bash[216700]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-node-exporter-a 2024-04-26T16:45:06.271 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:06 smithi050 bash[216764]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-node-exporter-a 2024-04-26T16:45:06.271 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:06 smithi050 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-26T16:45:06.271 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:06 smithi050 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-26T16:45:06.271 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:06 smithi050 systemd[1]: Stopped Ceph node-exporter.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:45:06.546 INFO:journalctl@ceph.node-exporter.a.smithi050.stdout:Apr 26 16:45:06 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:06.546 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:06 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:06.547 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:06 smithi050 systemd[1]: Stopping Ceph alertmanager.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:45:06.547 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:06 smithi050 bash[60093]: ts=2024-04-26T16:45:06.493Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-04-26T16:45:07.563 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:07 smithi050 bash[216832]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-alertmanager-a 2024-04-26T16:45:07.563 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:07 smithi050 bash[216894]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-alertmanager-a 2024-04-26T16:45:07.563 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:07 smithi050 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-26T16:45:07.563 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:07 smithi050 systemd[1]: Stopped Ceph alertmanager.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:45:07.563 INFO:journalctl@ceph.alertmanager.a.smithi050.stdout:Apr 26 16:45:07 smithi050 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:08.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-26T16:45:08.240 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:45:13.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.152 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.152 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.526 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.527 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.527 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.777 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.778 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:13.778 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.030 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.030 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:13 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.319 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.319 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.320 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.320 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:14.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:14 smithi190 systemd[1]: Stopping Ceph iscsi.iscsi.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:45:14.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:14 smithi190 bash[47603]: debug Shutdown received 2024-04-26T16:45:25.574 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:25 smithi190 bash[103483]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-iscsi-iscsi-a 2024-04-26T16:45:25.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:25 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-04-26T16:45:25.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:25 smithi190 bash[103552]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-iscsi-iscsi-a 2024-04-26T16:45:25.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:25 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-04-26T16:45:25.575 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:25 smithi190 systemd[1]: Stopped Ceph iscsi.iscsi.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:45:25.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Apr 26 16:45:25 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:25.881 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:25 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:25.882 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:25 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:26.947 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:26 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:26.947 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:26 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:26.947 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:26 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:26.947 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:26 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:27.324 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:26 smithi190 systemd[1]: Stopping Ceph node-exporter.b for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:45:27.990 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:27 smithi190 bash[103752]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-node-exporter-b 2024-04-26T16:45:27.990 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:27 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-26T16:45:27.990 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:27 smithi190 bash[103813]: Error response from daemon: No such container: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-node-exporter-b 2024-04-26T16:45:27.990 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:27 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-26T16:45:27.990 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:27 smithi190 systemd[1]: Stopped Ceph node-exporter.b for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:45:28.260 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Apr 26 16:45:28 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:28.260 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:27 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:28.260 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 systemd[1]: Stopping Ceph grafana.a for cf170a7a-03e6-11ef-bc93-c7b262605968... 2024-04-26T16:45:28.260 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 bash[50598]: logger=server t=2024-04-26T16:45:28.227149562Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-04-26T16:45:28.260 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 bash[50598]: logger=ticker t=2024-04-26T16:45:28.22719836Z level=info msg=stopped last_tick=2024-04-26T16:45:20Z 2024-04-26T16:45:28.260 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 bash[50598]: logger=tracing t=2024-04-26T16:45:28.227220989Z level=info msg="Closing tracing" 2024-04-26T16:45:29.074 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 bash[103881]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-grafana-a 2024-04-26T16:45:29.074 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 bash[103942]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968-grafana-a 2024-04-26T16:45:29.074 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 systemd[1]: ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-26T16:45:29.074 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:28 smithi190 systemd[1]: Stopped Ceph grafana.a for cf170a7a-03e6-11ef-bc93-c7b262605968. 2024-04-26T16:45:29.370 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Apr 26 16:45:29 smithi190 systemd[1]: /etc/systemd/system/ceph-cf170a7a-03e6-11ef-bc93-c7b262605968@.service:23: Unit configured to use KillMode=none. This is unsafe, as it disables systemd's process lifecycle management for the service. Please update your service to use a safer KillMode=, such as 'mixed' or 'control-group'. Support for KillMode=none is deprecated and will eventually be removed. 2024-04-26T16:45:29.635 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:45:29.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-26T16:45:29.664 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-26T16:45:29.667 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/remote/smithi050/crash 2024-04-26T16:45:29.668 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/crash -- . 2024-04-26T16:45:29.705 INFO:teuthology.orchestra.run.smithi050.stderr:tar: /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T16:45:29.705 INFO:teuthology.orchestra.run.smithi050.stderr:tar: Error is not recoverable: exiting now 2024-04-26T16:45:29.708 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/remote/smithi190/crash 2024-04-26T16:45:29.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/crash -- . 2024-04-26T16:45:29.720 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-26T16:45:29.720 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-04-26T16:45:29.721 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-26T16:45:29.722 DEBUG:teuthology.orchestra.run.smithi050:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'reached quota' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2024-04-26T16:45:29.766 INFO:tasks.cephadm:Compressing logs... 2024-04-26T16:45:29.766 DEBUG:teuthology.orchestra.run.smithi050:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T16:45:29.816 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T16:45:29.824 INFO:teuthology.orchestra.run.smithi050.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-26T16:45:29.829 INFO:teuthology.orchestra.run.smithi190.stderr:find: '/var/log/rbd-target-api': No such file or directory 2024-04-26T16:45:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T16:45:29.832 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-26T16:45:29.833 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-26T16:45:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-26T16:45:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-26T16:45:29.834 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T16:45:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-26T16:45:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-26T16:45:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.3.log: 88.6% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T16:45:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log 2024-04-26T16:45:29.836 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T16:45:29.838 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-26T16:45:29.839 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-26T16:45:29.839 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-26T16:45:29.840 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-26T16:45:29.840 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-26T16:45:29.840 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/objectstore_tool.7.log 2024-04-26T16:45:29.841 INFO:teuthology.orchestra.run.smithi190.stderr: 80.5% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-26T16:45:29.841 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-26T16:45:29.842 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.5.log: /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-26T16:45:29.843 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/tcmu-runner.log: 67.5%gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log 2024-04-26T16:45:29.843 INFO:teuthology.orchestra.run.smithi190.stderr: -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-26T16:45:29.843 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-26T16:45:29.843 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T16:45:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.1.log: /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log: 93.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T16:45:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-26T16:45:29.849 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-26T16:45:29.856 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-volume.log: 92.5% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-26T16:45:29.857 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-26T16:45:29.858 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.audit.log: 91.0% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T16:45:29.858 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-26T16:45:29.861 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.b.log: 85.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T16:45:29.862 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-26T16:45:29.864 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.4.log: 95.2% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/objectstore_tool.7.log.gz 2024-04-26T16:45:29.866 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log 2024-04-26T16:45:29.871 INFO:teuthology.orchestra.run.smithi190.stderr: 92.1% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T16:45:29.874 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.audit.log: 90.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-26T16:45:29.878 INFO:teuthology.orchestra.run.smithi050.stderr:/var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log: 91.8% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-26T16:45:29.891 INFO:teuthology.orchestra.run.smithi050.stderr: 95.1% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-26T16:45:30.153 INFO:teuthology.orchestra.run.smithi050.stderr: 93.6% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-04-26T16:45:30.407 INFO:teuthology.orchestra.run.smithi050.stderr: 91.1% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-26T16:45:31.197 INFO:teuthology.orchestra.run.smithi050.stderr: 92.1% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-26T16:45:31.685 INFO:teuthology.orchestra.run.smithi190.stderr: 92.5% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-26T16:45:32.352 INFO:teuthology.orchestra.run.smithi050.stderr: 91.6% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-26T16:45:35.994 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-26T16:45:36.486 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-26T16:45:37.842 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-26T16:45:38.066 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-26T16:45:38.262 INFO:teuthology.orchestra.run.smithi050.stderr: 94.0% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-26T16:45:38.613 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-26T16:45:39.393 INFO:teuthology.orchestra.run.smithi190.stderr: 94.0% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-26T16:45:39.395 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-04-26T16:45:39.395 INFO:teuthology.orchestra.run.smithi190.stderr:real 0m9.575s 2024-04-26T16:45:39.395 INFO:teuthology.orchestra.run.smithi190.stderr:user 0m31.114s 2024-04-26T16:45:39.396 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m1.182s 2024-04-26T16:45:39.424 INFO:teuthology.orchestra.run.smithi050.stderr: 93.9% -- replaced with /var/log/ceph/cf170a7a-03e6-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-26T16:45:39.426 INFO:teuthology.orchestra.run.smithi050.stderr: 2024-04-26T16:45:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:real 0m9.608s 2024-04-26T16:45:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:user 0m38.507s 2024-04-26T16:45:39.427 INFO:teuthology.orchestra.run.smithi050.stderr:sys 0m1.305s 2024-04-26T16:45:39.427 INFO:tasks.cephadm:Archiving logs... 2024-04-26T16:45:39.427 DEBUG:teuthology.misc:Transferring archived files from smithi050:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/remote/smithi050/log 2024-04-26T16:45:39.429 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T16:45:43.671 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/remote/smithi190/log 2024-04-26T16:45:43.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T16:45:47.259 INFO:tasks.cephadm:Removing cluster... 2024-04-26T16:45:47.259 DEBUG:teuthology.orchestra.run.smithi050:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 --force 2024-04-26T16:45:47.432 INFO:teuthology.orchestra.run.smithi050.stdout:Deleting cluster with fsid: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:45:48.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid cf170a7a-03e6-11ef-bc93-c7b262605968 --force 2024-04-26T16:45:49.007 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: cf170a7a-03e6-11ef-bc93-c7b262605968 2024-04-26T16:45:50.370 INFO:tasks.cephadm:Removing cephadm ... 2024-04-26T16:45:50.370 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T16:45:50.376 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-26T16:45:50.382 INFO:tasks.cephadm:Teardown complete 2024-04-26T16:45:50.382 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T16:45:50.392 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T16:45:50.392 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T16:45:50.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T16:45:50.488 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T16:45:50.497 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T16:45:50.497 DEBUG:teuthology.orchestra.run.smithi050:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T16:45:50.499 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T16:45:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T16:45:50.510 INFO:teuthology.orchestra.run.smithi050.stdout:============================================================================== 2024-04-26T16:45:50.510 INFO:teuthology.orchestra.run.smithi050.stdout:+hv01.front.sepi 67.205.162.81 3 u 45 128 377 0.086 -6.024 0.311 2024-04-26T16:45:50.510 INFO:teuthology.orchestra.run.smithi050.stdout:*hv02.front.sepi 63.231.80.2 3 u 119 128 377 0.088 -3.936 0.356 2024-04-26T16:45:50.510 INFO:teuthology.orchestra.run.smithi050.stdout:+hv03.front.sepi 74.6.168.72 3 u 118 128 377 0.099 -1.071 0.506 2024-04-26T16:45:50.510 INFO:teuthology.orchestra.run.smithi050.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-26T16:45:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T16:45:50.512 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================== 2024-04-26T16:45:50.512 INFO:teuthology.orchestra.run.smithi190.stdout:+hv01.front.sepi 67.205.162.81 3 u 56 64 377 0.084 -6.083 0.617 2024-04-26T16:45:50.512 INFO:teuthology.orchestra.run.smithi190.stdout:*hv02.front.sepi 63.231.80.2 3 u 48 64 377 0.079 -5.206 0.841 2024-04-26T16:45:50.512 INFO:teuthology.orchestra.run.smithi190.stdout:+hv03.front.sepi 74.6.168.72 3 u 47 64 377 0.118 -2.526 0.823 2024-04-26T16:45:50.512 INFO:teuthology.orchestra.run.smithi190.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-26T16:45:50.512 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T16:45:50.521 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T16:45:50.521 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T16:45:50.530 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T16:45:50.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T16:45:50.548 INFO:teuthology.task.internal:Duration was 2975.050896 seconds 2024-04-26T16:45:50.548 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T16:45:50.559 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T16:45:50.559 DEBUG:teuthology.orchestra.run.smithi050:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T16:45:50.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T16:45:50.597 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T16:45:50.597 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi050.front.sepia.ceph.com 2024-04-26T16:45:50.598 DEBUG:teuthology.orchestra.run.smithi050:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T16:45:50.651 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-04-26T16:45:50.651 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T16:45:50.661 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T16:45:50.661 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T16:45:50.695 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T16:45:50.864 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T16:45:50.864 DEBUG:teuthology.orchestra.run.smithi050:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T16:45:50.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T16:45:51.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T16:45:51.521 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T16:45:51.521 DEBUG:teuthology.orchestra.run.smithi050:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T16:45:51.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T16:45:51.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T16:45:51.557 DEBUG:teuthology.orchestra.run.smithi050:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T16:45:51.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T16:45:51.582 INFO:teuthology.orchestra.run.smithi050.stdout:kernel.core_pattern = core 2024-04-26T16:45:51.603 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-04-26T16:45:51.697 DEBUG:teuthology.orchestra.run.smithi050:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T16:45:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:45:51.702 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T16:45:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T16:45:51.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T16:45:51.754 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T16:45:51.754 DEBUG:teuthology.misc:Transferring archived files from smithi050:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/remote/smithi050 2024-04-26T16:45:51.755 DEBUG:teuthology.orchestra.run.smithi050:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T16:45:51.828 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-04-26_15:06:39-orch:cephadm-wip-phlogistonjohn-testing-2024-04-25-1915-distro-default-smithi/7674824/remote/smithi190 2024-04-26T16:45:51.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T16:45:51.861 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T16:45:51.861 DEBUG:teuthology.orchestra.run.smithi050:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T16:45:51.864 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T16:45:51.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T16:45:51.917 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T16:45:51.917 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T16:45:51.966 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T16:45:51.966 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T16:45:51.969 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T16:45:51.973 INFO:teuthology.orchestra.run.smithi050.stdout: 658468 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 16:45 /home/ubuntu/cephtest 2024-04-26T16:45:51.975 INFO:teuthology.orchestra.run.smithi190.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 16:45 /home/ubuntu/cephtest 2024-04-26T16:45:51.976 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T16:45:51.986 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T16:45:52.006 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} duration: 2975.0508959293365 flavor: default owner: scheduled_phlogistonjohn@teuthology success: true 2024-04-26T16:45:52.007 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T16:45:52.085 INFO:teuthology.run:pass